diff --git a/CHANGELOG.md b/CHANGELOG.md index b25e44f..513beea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,27 @@ session's. `configs/` changes, so the release needs BOTH signed jobs. name nothing (T1); no source (T2); not one CA certificate (T3). No appliance check: the copy is the agent's own trust file and changes nothing of a BYO owner's host. Tests `PVECACopy` (9, one with the real openssl through the real runner); 7 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-wrapper.txt`). +- **The move of a box that exists, in a safe order** (`internal/pvetrust`, new; wired in `cmd/felhom-agent/main.go` + at the agent's start, after a config bundle install, on a timer — hourly until moved, daily after — and on every + call to Proxmox that fails on trust). (1) the root step makes the copy; (2) ONE real call to Proxmox with the copy + ALONE, while the pinned fingerprint is still the configured trust; (3) only then ONE write of `agent.json` sets + `proxmox.tls.ca_file` and clears `fingerprint` (the agent's own file, rewritten in place after a recovery copy + `agent.json.pre-pve-trust` was parked, read back and checked); (4) the RUNNING client takes the new trust + (`proxmox.Client.SetTLS`, new) — no restart. A failed copy, proof or write changes nothing: the fingerprint stays. + **`proxmox.tls` keeps its order (a set fingerprint wins over a CA file), so the fingerprint is cleared LAST.** An + operator's own CA file, a box with checks switched off and a config that is not a file are never moved. +- **A pinned box whose certificate changed heals by itself:** the failed call starts the same order, the box moves to + the CA copy and the next call works. A refreshed copy (a renewed root CA) is proven and reloaded by the running + client. +- **The cause is named** (`proxmox.TrustProblem`, new): the pinned certificate changed; the CA copy did not sign the + served certificate; a certificate ended; the copy is unreadable; the copy ends within 30 days; the root step gave no + copy; the proof failed. An ERROR line in the journal, and the new `proxmox_trust` stanza of the host report + (`mode`, `problem`, `detail`, `ca_not_after`, `ca_days_left`, `moved_at`). **The hub does not read the stanza yet** + (felhom.eu R-1396), and a box that is fully blind sends no host report at all — that case still shows only as + „host stale" on the hub. +- Tests: `internal/pvetrust` (11, against a real TLS server that changes its certificate and renews its CA), the AST + wiring test `TestMainWiresPVETrust`; 12 + 3 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-agent.txt`; + one mutation is not a rule and says so). ## v0.174.0 (2026-11-14) — the box removes its old kernels (R-1368); security profiles are refreshed after a kernel change (R-1388, R-1301); Go 1.27.4 (R-1391) — RELEASED 2026-11-14 (release-agent.sh from the clean working clone at `cc63dd5`, CI run id 2641; agent sha256 6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92, bundle sha256 7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f) and VOUCHED 2026-11-14 15:21:45 CET diff --git a/REUSE.md b/REUSE.md index 2fcaaee..d9309e6 100644 --- a/REUSE.md +++ b/REUSE.md @@ -100,6 +100,7 @@ | `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). | | `Leg.runKernelCleanup` / `kernelCleanupSkip` / `Leg.KernelCleanupDryRun` + the wrapper's `Kernel.cleanup` | internal/osupdate/kernel.go; configs/felhom-os-apply (mode `kernel-cleanup`) | `kernelCleanupSkip(appliance, trigger, host, pve, kernel, blk, okStep) string` ("" = call it) | R-1368 (`11` §5.11.1): old kernel images are removed by the WRAPPER's rule; the agent decides only WHEN (the night leg, after a healthy host step, never in a night with a kernel step) | The plan names NO package (C1). Never `apt-get autoremove`. Whether a box may do it is the root-owned ring-0 mark, never the hub's ring. The hub report carries NO kernel view and no kernel version (the hub would read it as a kernel step). `--selftest=kernel-cleanup-dry-run` changes nothing. | | `Leg.ProfileRefresh` + the wrapper's `Apply.profile_refresh` | internal/osupdate/profilerefresh.go; configs/felhom-os-apply (mode `profile-refresh`) | `ProfileRefresh(ctx, why, gate, tries, wait) string` (the outcome) | R-1388 (`11` §5.12): load every AppArmor profile from a fresh compile once per kernel the box starts on; restart only a FAILED service whose own program's cached profile was stale | The plan names nothing (F1). A stale copy is told by the CACHED FILE changing at the rewrite — never by the kernel's loaded hash (it differs between a cache load and a fresh load for multi-profile files; `lxc-containers` never compiles the same twice). Called at agent start and after a bundle install, under the heavy-op gate. | +| `pvetrust.Manager` (`Ensure`, `OnCallError`, `Status`) + `proxmox.Client.SetTLS` / `proxmox.TrustProblem` + `Leg.PVECACopy` + the wrapper's `pve_ca_copy` | internal/pvetrust/pvetrust.go; internal/proxmox/{client,trust}.go; internal/osupdate/pvecacopy.go; configs/felhom-os-apply (mode `pve-ca-copy`) | `Ensure(ctx, why) (changed bool)`; `TrustProblem(err) string` | R-1380 (felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA; the move from a pinned certificate; the cause of a trust failure | THE ORDER: the copy, a real call with the copy ALONE, then ONE config write that clears the fingerprint LAST (a set fingerprint wins over `ca_file` in `TLSConfig.build`). Never clear the fingerprint anywhere else. The wrapper's plan names nothing (T1). `agent.json` is written by the agent in place, with a parked recovery copy — the same way as the backup-target move. | | `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in | | `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them | | `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc | diff --git a/cmd/felhom-agent/main.go b/cmd/felhom-agent/main.go index 7ddb961..c62ad08 100644 --- a/cmd/felhom-agent/main.go +++ b/cmd/felhom-agent/main.go @@ -54,6 +54,7 @@ import ( "gitea.dooplex.hu/admin/felhom-agent/internal/poweroff" "gitea.dooplex.hu/admin/felhom-agent/internal/provision" "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" + "gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust" "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" "gitea.dooplex.hu/admin/felhom-agent/internal/restorespace" "gitea.dooplex.hu/admin/felhom-agent/internal/selfheal" @@ -988,6 +989,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int osLeg := newOSLeg(cfg, client, px, logger) collector.SetSystemReporter(&factsReporter{leg: osLeg, guest: firstGuest(px)}) // R-852: the versions desiredSyncer.AddConsumer(osLeg) + // R-1380 (D80, felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA. At this start the root + // step makes the copy, a REAL call proves it while the pinned fingerprint is still the trust, and only then ONE + // write of the config sets ca_file and clears the fingerprint. A call that fails on trust later names the cause and + // runs the same order again (a pinned box whose certificate changed moves right then, with no restart). + pveTrust := newPVETrust(cfg, px, osLeg, logger) + collector.SetProxmoxTrustSource(func() any { return pveTrust.Status() }) + loop.SetCollectErrorHook(pveTrust.OnCallError) // S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface, // execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect. desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf { @@ -1018,6 +1026,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int // network and stays under the hub's 45-minute host_stale (its // alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged. go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait}) + // R-1380: the trust order at this start (the copy, the proof, then the config) — before the first report cycle, so + // a pinned box whose certificate changed while the agent was down is not blind for a cycle. Then on a timer. + pveTrust.Ensure(ctx, "start") + go pveTrust.Run(ctx, time.Hour, 24*time.Hour) // R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute // after start so the guest has settled; a failure is logged and the next start retries. go func() { @@ -1301,6 +1313,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int // R-1388: a new wrapper may know the profile refresh the old one refused — ask again (its own goroutine: // it may wait for the heavy-op gate, and the signed job must not wait with it). go osLeg.ProfileRefresh(daemonCtx, "config bundle installed", profileGate, 12, 5*time.Minute) + // R-1380: a new wrapper may know the CA-copy step the old one refused — run the trust order again. + go pveTrust.Ensure(daemonCtx, "config bundle installed") }} // R-1388 / R-1301 (`11` §5.12): after a start, load the security profiles from a fresh compile when the box runs // another kernel than at the last refresh (the wrapper keeps that record and answers "not needed" otherwise). @@ -3948,6 +3962,29 @@ func (f *selftestFlag) Set(v string) error { return nil } +// newPVETrust wires internal/pvetrust (R-1380): the root step through the OS wrapper, the proof as a throwaway client +// with EXACTLY the trust under test, and the running client's SetTLS. +func newPVETrust(cfg config.Config, px *proxmox.Client, leg *osupdate.Leg, logger *slog.Logger) *pvetrust.Manager { + m := &pvetrust.Manager{ConfigPath: cfg.SourcePath, StateDir: agentStateDir(cfg), Logger: logger, + EnsureCopy: func(ctx context.Context) (pvetrust.Copy, error) { + r, err := leg.PVECACopy(ctx) + return pvetrust.Copy{Path: r.Path, Outcome: r.Outcome, NotAfter: r.NotAfter, DaysLeft: r.DaysLeft}, err + }, + Probe: func(ctx context.Context, t proxmox.TLSConfig) error { + p, err := proxmox.NewClient(proxmox.Config{Endpoint: cfg.Proxmox.Endpoint, Node: cfg.Proxmox.Node, Token: cfg.Proxmox.Token, TLS: t}) + if err != nil { + return err + } + _, err = p.Version(ctx) + return err + }, + Apply: px.SetTLS, + } + m.Start(proxmox.TLSConfig{CAFile: cfg.Proxmox.TLS.CAFile, Fingerprint: cfg.Proxmox.TLS.Fingerprint, + InsecureSkipVerify: cfg.Proxmox.TLS.InsecureSkipVerify}) + return m +} + // newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and // the once-per-night marker live in the agent's own os/ dir. func newOSLeg(cfg config.Config, client *hub.Client, px *proxmox.Client, logger *slog.Logger) *osupdate.Leg { diff --git a/cmd/felhom-agent/pvetrust_wiring_test.go b/cmd/felhom-agent/pvetrust_wiring_test.go new file mode 100644 index 0000000..89cebcb --- /dev/null +++ b/cmd/felhom-agent/pvetrust_wiring_test.go @@ -0,0 +1,62 @@ +package main + +import ( + "go/ast" + "go/parser" + "go/token" + "testing" +) + +// R-1380: the trust move is a component with seams (the root step, the proof, the running client, the report stanza, +// the failed-call hook). A unit-green pvetrust that main.go never starts is the inert-seam defect, so the wiring is +// asserted from the AST: built once, run at start and after a bundle install, on a timer, fed by a failed collect, and +// reported. The same for the profile refresh (R-1388), which had no such test. +func TestMainWiresPVETrust(t *testing.T) { + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, "main.go", nil, 0) + if err != nil { + t.Fatalf("parse main.go: %v", err) + } + seen := map[string]int{} + ast.Inspect(f, func(n ast.Node) bool { + switch node := n.(type) { + case *ast.CallExpr: + sel, ok := node.Fun.(*ast.SelectorExpr) + if !ok { + return true + } + x, _ := sel.X.(*ast.Ident) + switch { + case x != nil && x.Name == "pveTrust" && sel.Sel.Name == "Ensure": + seen["ensure"]++ + case sel.Sel.Name == "SetProxmoxTrustSource": + seen["stanza"]++ + case sel.Sel.Name == "SetCollectErrorHook" && len(node.Args) == 1: + if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "OnCallError" { + seen["hook"]++ + } + case x != nil && x.Name == "osLeg" && sel.Sel.Name == "ProfileRefresh": + seen["profile-refresh"]++ + } + case *ast.GoStmt: + if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" { + if id, ok := sel.X.(*ast.Ident); ok && id.Name == "pveTrust" { + seen["timer"]++ + } + } + case *ast.KeyValueExpr: + if k, ok := node.Key.(*ast.Ident); ok && k.Name == "Apply" { + if v, ok := node.Value.(*ast.SelectorExpr); ok && v.Sel.Name == "SetTLS" { + seen["apply=SetTLS"]++ + } + } + } + return true + }) + want := map[string]int{"ensure": 2, "stanza": 1, "hook": 1, "timer": 1, "apply=SetTLS": 1, "profile-refresh": 2} + for k, n := range want { + if seen[k] < n { + t.Errorf("main.go wires %q %d time(s), want at least %d (the two-call ones: at start and after a bundle install)", k, seen[k], n) + } + } +} diff --git a/internal/hub/collect.go b/internal/hub/collect.go index 69286ca..643286a 100644 --- a/internal/hub/collect.go +++ b/internal/hub/collect.go @@ -124,6 +124,7 @@ type Collector struct { temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp) capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty) leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled) + pveTrust func() any // R-1380: the proxmox_trust stanza (nil = omitted) addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses) wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted) pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted) @@ -209,6 +210,12 @@ func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capabi return c } +// SetProxmoxTrustSource wires the `proxmox_trust` stanza (R-1380). Optional; nil = the stanza is omitted. +func (c *Collector) SetProxmoxTrustSource(f func() any) *Collector { + c.pveTrust = f + return c +} + // SetLeafFingerprint records the served local-API leaf fp (v0.48.0) to ride every host report (the hub // watches it for a re-key). Static per process — set once at startup. "" when the local API is // disabled. Returns the collector for chaining. @@ -391,6 +398,13 @@ func (c *Collector) SetOOBReporter(o OOBReporter) *Collector { // error (no useful report — the cycle skips the POST); a failed per-guest // GuestConfig degrades that guest to status="unknown" without spec but still sends; // a cloudflared probe failure yields status="unknown" and is never fatal. +func (c *Collector) proxmoxTrust() any { + if c.pveTrust == nil { + return nil + } + return c.pveTrust() +} + func (c *Collector) Collect(ctx context.Context) (*HostReport, error) { ns, err := c.px.NodeStatus(ctx) if err != nil { @@ -420,6 +434,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) { Cloudflared: c.cloudflared(ctx), Capabilities: c.capabilities(ctx), LeafFingerprint: c.leafFP, + ProxmoxTrust: c.proxmoxTrust(), Addresses: c.collectAddresses(), System: c.systemInfo(ctx, ns), } diff --git a/internal/hub/loop.go b/internal/hub/loop.go index 50f0457..43d7f6f 100644 --- a/internal/hub/loop.go +++ b/internal/hub/loop.go @@ -70,8 +70,14 @@ type Loop struct { // reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off // path waits on it so the hub has the deliberate-off record before the box goes down. reported func(r *HostReport) + // collectErr (R-1380) sees every failed collect. + collectErr func(ctx context.Context, err error) } +// SetCollectErrorHook wires an observer of a failed collect (R-1380: a call to Proxmox that fails on TRUST starts the +// move to the CA copy, or a reload of a refreshed copy). Optional; it must not block for long. +func (l *Loop) SetCollectErrorHook(f func(ctx context.Context, err error)) { l.collectErr = f } + // SetReportedHook wires the R-998 after-report observer. Optional. func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f } @@ -144,6 +150,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration { report, err := l.collector.Collect(ctx) if err != nil { l.logger.Warn("hub: collect failed; skipping this cycle's report", "err", err) + if l.collectErr != nil { + l.collectErr(ctx, err) + } return current } // Fulfill a pending log-pull: attach the ring tail to THIS report and clear the diff --git a/internal/hub/report.go b/internal/hub/report.go index b5d39fc..ecba28c 100644 --- a/internal/hub/report.go +++ b/internal/hub/report.go @@ -53,6 +53,11 @@ type HostReport struct { // alert. Not a secret (the fp is public; the token is never reported). LeafFingerprint string `json:"leaf_fingerprint"` + // ProxmoxTrust (R-1380, felhom.eu `03` §12a): how the agent trusts Proxmox's certificate (`mode`: ca | fingerprint | + // …), and the CAUSE when something about it is wrong (`problem`, `detail`): the proof with the CA copy failed, the + // copy ends soon, the root step gave no copy. Omitted when not wired. No secret: a mode, a cause, a date. + ProxmoxTrust any `json:"proxmox_trust,omitempty"` + // Addresses are the host's routable addresses, one entry per (interface, address) — the LAN // bridge, the WG tunnel, a tailnet. Added v0.119.0 because the hub could not show a managed // box's IP anywhere: nothing in this report carried one. Non-nil so it marshals as []; diff --git a/internal/osupdate/leg.go b/internal/osupdate/leg.go index 1ac2926..f96f863 100644 --- a/internal/osupdate/leg.go +++ b/internal/osupdate/leg.go @@ -139,7 +139,9 @@ type WrapperReport struct { KernelCleanup json.RawMessage `json:"kernel_cleanup"` // ProfileRefresh (R-1388, `11` §5.12): mode profile-refresh's result {outcome, kernel, rewritten, restarted, seconds}. ProfileRefresh json.RawMessage `json:"profile_refresh"` - OutcomeHint string `json:"outcome_hint"` + // PVECACopy (R-1380, felhom.eu `03` §12a): mode pve-ca-copy's result {outcome, path, sha256, not_after, days_left}. + PVECACopy json.RawMessage `json:"pve_ca_copy"` + OutcomeHint string `json:"outcome_hint"` // R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the // agent process that started the pass. ReleaseID / VMID were always in the report. RunID string `json:"run_id"` diff --git a/internal/osupdate/pvecacopy.go b/internal/osupdate/pvecacopy.go new file mode 100644 index 0000000..1a0f131 --- /dev/null +++ b/internal/osupdate/pvecacopy.go @@ -0,0 +1,45 @@ +package osupdate + +// The root step that keeps a readable copy of Proxmox's root CA for the agent (R-1380, D80, felhom.eu `03` §12a). +// The rule lives in the wrapper (mode pve-ca-copy: one fixed source, one fixed place, a CA certificate only); the plan +// names nothing. internal/pvetrust decides when to ask and what to do with the answer. + +import ( + "context" + "encoding/json" + "fmt" +) + +// PVECACopyPath is where the wrapper puts the copy (felhom-os-apply PVE_CA_COPY) — the agent's `proxmox.tls.ca_file`. +const PVECACopyPath = "/etc/felhom-agent/pve-root-ca.pem" + +// PVECACopyResult is the wrapper's pve_ca_copy object. +type PVECACopyResult struct { + Outcome string `json:"outcome"` // written | same + Path string `json:"path"` + SHA256 string `json:"sha256"` + NotAfter string `json:"not_after"` + DaysLeft int `json:"days_left"` + Replaced bool `json:"replaced"` +} + +// PVECACopy asks the wrapper to make or refresh the copy. An error names a refusal (T1–T3, or R11 from a wrapper that +// does not know the mode yet) or a wrapper that gave no answer. +func (l *Leg) PVECACopy(ctx context.Context) (PVECACopyResult, error) { + wr, err := l.call(ctx, "pveca"+l.now().UTC().Format("150405"), map[string]any{"release_id": "pve-ca-copy", + "layer": LayerHost, "mode": "pve-ca-copy"}) + if err != nil { + return PVECACopyResult{}, err + } + if wr.refused() { + return PVECACopyResult{}, fmt.Errorf("pve-ca-copy refused: %s", wr.Refused) + } + if wr.failed() { + return PVECACopyResult{}, fmt.Errorf("pve-ca-copy failed: %s", wr.Failed) + } + var res PVECACopyResult + if jerr := json.Unmarshal(wr.PVECACopy, &res); jerr != nil || res.Outcome == "" || res.Path == "" { + return PVECACopyResult{}, fmt.Errorf("pve-ca-copy: the wrapper gave no result (%v)", jerr) + } + return res, nil +} diff --git a/internal/proxmox/client.go b/internal/proxmox/client.go index e368c59..7cae0c0 100644 --- a/internal/proxmox/client.go +++ b/internal/proxmox/client.go @@ -10,6 +10,7 @@ import ( "net/http" "net/url" "strings" + "sync" "time" "gitea.dooplex.hu/admin/felhom-agent/internal/httpx" @@ -45,6 +46,39 @@ type Client struct { node string token string http doer + + // tlsMu guards http: SetTLS swaps the transport while calls run (R-1380 — the move from a pinned certificate to the + // box's own root CA, and a refreshed CA copy, take effect with no agent restart). + tlsMu sync.RWMutex + timeout time.Duration +} + +func (c *Client) httpDoer() doer { + c.tlsMu.RLock() + defer c.tlsMu.RUnlock() + return c.http +} + +// SetTLS replaces how this client trusts the Proxmox certificate, for every call that starts after it returns. On an +// error (an unreadable CA file, a bad fingerprint) nothing changes. +func (c *Client) SetTLS(t TLSConfig) error { + tlsCfg, err := t.build() + if err != nil { + return err + } + timeout := c.timeout + if timeout == 0 { + timeout = 30 * time.Second + } + hc := &http.Client{Timeout: timeout, Transport: httpx.NewTransport(tlsCfg, 0)} + c.tlsMu.Lock() + old := c.http + c.http = hc + c.tlsMu.Unlock() + if o, ok := old.(*http.Client); ok && o != nil { + o.CloseIdleConnections() // a kept-alive connection made under the old trust must not serve the next call + } + return nil } // NewClient builds an API client. It validates required config and constructs the @@ -74,10 +108,11 @@ func NewClient(cfg Config) (*Client, error) { Transport: httpx.NewTransport(tlsCfg, 0), } return &Client{ - base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json", - node: cfg.Node, - token: cfg.Token, - http: hc, + base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json", + node: cfg.Node, + token: cfg.Token, + http: hc, + timeout: timeout, }, nil } @@ -125,7 +160,7 @@ func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader req.Header.Set("Content-Type", contentType) } - resp, err := c.http.Do(req) + resp, err := c.httpDoer().Do(req) if err != nil { return fmt.Errorf("proxmox: %s %s: %w", method, path, err) } diff --git a/internal/proxmox/trust.go b/internal/proxmox/trust.go new file mode 100644 index 0000000..62a1450 --- /dev/null +++ b/internal/proxmox/trust.go @@ -0,0 +1,43 @@ +package proxmox + +import ( + "crypto/x509" + "errors" + "strings" +) + +// The causes a call to Proxmox can fail on TRUST (R-1380, felhom.eu `03` §12a). A symptom alarm („the agent channel is +// down") exists; these name the cause. +const ( + TrustPinnedCertChanged = "pinned_cert_changed" // the pinned certificate is not the one Proxmox serves any more + TrustCAMismatch = "ca_mismatch" // the CA file did not sign the served certificate + TrustCertExpired = "cert_expired" // the served certificate (or the CA) has ended + TrustCAFileUnreadable = "ca_file_unreadable" // the CA file is missing or holds no certificate + TrustNameMismatch = "name_mismatch" // the certificate does not name the address the agent calls +) + +// TrustProblem names why err is a TRUST failure, or "" when it is none (a refused connection, a 401, a timeout). +func TrustProblem(err error) string { + if err == nil { + return "" + } + var ua x509.UnknownAuthorityError + var inv x509.CertificateInvalidError + var hn x509.HostnameError + switch { + case strings.Contains(err.Error(), "TLS pin mismatch"): + return TrustPinnedCertChanged + case errors.As(err, &ua): + return TrustCAMismatch + case errors.As(err, &inv): + if inv.Reason == x509.Expired { + return TrustCertExpired + } + return TrustCAMismatch + case errors.As(err, &hn): + return TrustNameMismatch + case strings.Contains(err.Error(), "reading TLS CA file"), strings.Contains(err.Error(), "contained no usable certificates"): + return TrustCAFileUnreadable + } + return "" +} diff --git a/internal/pvetrust/pvetrust.go b/internal/pvetrust/pvetrust.go new file mode 100644 index 0000000..6b9ce0a --- /dev/null +++ b/internal/pvetrust/pvetrust.go @@ -0,0 +1,421 @@ +// Package pvetrust moves the agent's trust in Proxmox from ONE pinned certificate to the box's own root CA, and names +// the cause when a call fails on trust (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a). +// +// WHY. Proxmox makes a new node certificate by itself, at most every two years. An agent that pins the old one is then +// blind: every call fails, no host report reaches the hub, and the only alarm names a symptom. A copy of the box's own +// root CA verifies every certificate that CA signs. +// +// THE ORDER, and it is the whole point (a box must never be left with neither way working): +// +// 1. the root step makes or refreshes the copy (it refuses anything that is not one CA certificate); +// 2. a REAL call to Proxmox is made with the copy ALONE — while the fingerprint still is the configured trust; +// 3. only after that call worked, ONE write of the agent's config sets `ca_file` and clears the fingerprint; +// 4. the running client takes the new trust (no restart). +// +// If 1 or 2 fails nothing is written and the fingerprint stays. If 3 fails the fingerprint stays, in the file and in +// the running client. `proxmox.tls` keeps its order in code — a set fingerprint wins over a CA file — so the +// fingerprint is cleared LAST, in the same write that sets `ca_file`. +// +// WHO WRITES THE CONFIG. The agent itself: `agent.json` is the agent's own file (0600) in a root-owned folder, so it is +// rewritten in place after a recovery copy was parked in the agent's state folder — the same way the backup-target move +// and the PBS seed write it. No root step touches it. +package pvetrust + +import ( + "context" + "encoding/json" + "fmt" + "log/slog" + "os" + "path/filepath" + "sync" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" +) + +// EndsSoonDays: the copy's certificate ends within this many days → the problem „ca_copy_ends_soon". Proxmox renews its +// root CA when less than two years are left, so a healthy box never comes near it. +const EndsSoonDays = 30 + +// Problems this package names beyond proxmox.Trust*. +const ( + ProblemCopyUnavailable = "ca_copy_unavailable" // the root step refused or did not answer + ProblemCopyNoProof = "ca_copy_does_not_verify" // a call with the copy alone failed — the fingerprint stays + ProblemCopyEndsSoon = "ca_copy_ends_soon" + ProblemConfigNotSaved = "config_not_written" // the proof worked but the config could not be written — the fingerprint stays +) + +// Copy is what the root step answered. +type Copy struct { + Path string + Outcome string // written | same + NotAfter string + DaysLeft int +} + +// Status is the stanza that rides the host report (`proxmox_trust`). No secret: a mode, a cause, a date. +type Status struct { + Mode string `json:"mode"` // ca | fingerprint | ca_file_own | insecure | system + Problem string `json:"problem,omitempty"` // "" = none + Detail string `json:"detail,omitempty"` + CANotAfter string `json:"ca_not_after,omitempty"` + CADaysLeft *int `json:"ca_days_left,omitempty"` + MovedAt string `json:"moved_at,omitempty"` // when this box moved from the pinned certificate to the CA copy + CheckedAt string `json:"checked_at,omitempty"` +} + +// Manager holds the trust state of one agent process. +type Manager struct { + // EnsureCopy runs the root step (osupdate.Leg.PVECACopy). + EnsureCopy func(ctx context.Context) (Copy, error) + // Probe makes one real call to Proxmox with EXACTLY this trust (a throwaway client). + Probe func(ctx context.Context, t proxmox.TLSConfig) error + // Apply gives the running client the new trust (proxmox.Client.SetTLS). + Apply func(t proxmox.TLSConfig) error + // ConfigPath is agent.json ("" = the config came from the environment: nothing can be saved, so nothing is moved). + ConfigPath string + StateDir string + Logger *slog.Logger + Now func() time.Time + // MinGap between two runs that a failed call asked for (default 10 minutes). + MinGap time.Duration + + mu sync.Mutex + cur proxmox.TLSConfig + status Status + lastRun time.Time +} + +func (m *Manager) now() time.Time { + if m.Now != nil { + return m.Now() + } + return time.Now() +} + +func (m *Manager) log() *slog.Logger { + if m.Logger != nil { + return m.Logger + } + return slog.Default() +} + +// Start records the trust the process started with. Call once, before Ensure. +func (m *Manager) Start(t proxmox.TLSConfig) { + m.mu.Lock() + defer m.mu.Unlock() + m.cur = t + m.status = Status{Mode: modeOf(t)} +} + +func modeOf(t proxmox.TLSConfig) string { + switch { + case t.InsecureSkipVerify: + return "insecure" + case t.Fingerprint != "": + return "fingerprint" + case t.CAFile != "": + return "ca" + } + return "system" +} + +// Status is the current stanza (nil-safe). +func (m *Manager) Status() *Status { + if m == nil { + return nil + } + m.mu.Lock() + defer m.mu.Unlock() + s := m.status + return &s +} + +// Moved reports whether the box trusts Proxmox by the CA copy (nothing is left to move). +func (m *Manager) Moved() bool { + m.mu.Lock() + defer m.mu.Unlock() + return m.cur.Fingerprint == "" && m.cur.CAFile != "" +} + +// Run asks again on a timer until ctx ends: every `pending` while the box is not moved yet (a wrapper that did not +// know the mode, a Proxmox that did not answer), every `settled` after (a renewed CA refreshes the copy). +func (m *Manager) Run(ctx context.Context, pending, settled time.Duration) { + for { + d := settled + if !m.Moved() { + d = pending + } + select { + case <-ctx.Done(): + return + case <-time.After(d): + m.Ensure(ctx, "timer") + } + } +} + +// Current is the trust the running client uses. +func (m *Manager) Current() proxmox.TLSConfig { + m.mu.Lock() + defer m.mu.Unlock() + return m.cur +} + +// OnCallError is the hook for a failed Proxmox call: when the failure is one of TRUST, it names the cause and runs +// Ensure (at most once per MinGap) — a pinned box whose certificate changed moves to the CA copy right then, and a box +// whose CA copy was refreshed reloads it. A failure of another kind does nothing. +func (m *Manager) OnCallError(ctx context.Context, err error) { + cause := proxmox.TrustProblem(err) + if cause == "" { + return + } + m.mu.Lock() + gap := m.MinGap + if gap <= 0 { + gap = 10 * time.Minute + } + tooSoon := !m.lastRun.IsZero() && m.now().Sub(m.lastRun) < gap + if m.status.Problem == "" { + m.status.Problem, m.status.Detail = cause, causeText(cause) + } + m.mu.Unlock() + m.log().Error("pvetrust: a call to Proxmox failed on TRUST — "+causeText(cause), "cause", cause, "err", err) + if tooSoon { + return + } + m.Ensure(ctx, cause) +} + +func causeText(cause string) string { + switch cause { + case proxmox.TrustPinnedCertChanged: + return "the pinned certificate is not the one Proxmox serves any more (Proxmox made a new certificate)" + case proxmox.TrustCAMismatch: + return "the CA copy did not sign the certificate Proxmox serves (the box's root CA changed, or the copy is not this box's)" + case proxmox.TrustCertExpired: + return "the certificate Proxmox serves, or its CA, has ended" + case proxmox.TrustCAFileUnreadable: + return "the CA copy is missing or holds no certificate" + case proxmox.TrustNameMismatch: + return "the certificate Proxmox serves does not name the address the agent calls" + } + return cause +} + +// Ensure runs the order of the package comment once. why is a log word ("start", "daily", a trust cause). It returns +// true when the running trust changed. +func (m *Manager) Ensure(ctx context.Context, why string) bool { + m.mu.Lock() + cur := m.cur + m.lastRun = m.now() + m.mu.Unlock() + lg := m.log().With("why", why, "mode", modeOf(cur)) + set := func(mode, problem, detail string, c *Copy) { + m.mu.Lock() + defer m.mu.Unlock() + moved := m.status.MovedAt + m.status = Status{Mode: mode, Problem: problem, Detail: detail, MovedAt: moved, CheckedAt: m.now().UTC().Format(time.RFC3339)} + if c != nil { + d := c.DaysLeft + m.status.CANotAfter, m.status.CADaysLeft = c.NotAfter, &d + } + } + if cur.InsecureSkipVerify { + lg.Warn("pvetrust: certificate checks are switched off in the config (insecure_skip_verify) — nothing is moved") + set("insecure", "", "", nil) + return false + } + if m.EnsureCopy == nil || m.Probe == nil { + return false + } + // 1. the copy + c, err := m.EnsureCopy(ctx) + if err != nil { + lg.Warn("pvetrust: the root step gave no CA copy — the trust stays as it is", "err", err) + set(modeOf(cur), ProblemCopyUnavailable, truncate(err.Error(), 240), nil) + return false + } + ca := proxmox.TLSConfig{CAFile: c.Path} + onCopy := cur.Fingerprint == "" && cur.CAFile == c.Path + if cur.Fingerprint == "" && cur.CAFile != "" && cur.CAFile != c.Path { + // an operator's own CA file: never replaced + lg.Info("pvetrust: the config names its own CA file — left as it is", "ca_file", cur.CAFile) + set("ca_file_own", "", "", &c) + return false + } + if onCopy && c.Outcome != "written" && why != proxmox.TrustCAMismatch && why != proxmox.TrustCAFileUnreadable && why != proxmox.TrustCertExpired { + // the box already trusts by the copy and the copy did not change: nothing to prove again + problem, detail := "", "" + if c.DaysLeft < EndsSoonDays { + problem, detail = ProblemCopyEndsSoon, fmt.Sprintf("the CA copy ends in %d day(s), on %s", c.DaysLeft, c.NotAfter) + lg.Error("pvetrust: "+detail, "days_left", c.DaysLeft) + } + set("ca", problem, detail, &c) + return false + } + // 2. the proof: a real call with the copy ALONE — the fingerprint is still the configured trust + pctx, cancel := context.WithTimeout(ctx, 30*time.Second) + perr := m.Probe(pctx, ca) + cancel() + if perr != nil { + cause := proxmox.TrustProblem(perr) + detail := "a call to Proxmox with the CA copy alone failed" + if cause != "" { + detail += ": " + causeText(cause) + } else { + detail += " (not a trust failure: " + truncate(perr.Error(), 160) + ")" + } + if onCopy { + lg.Error("pvetrust: the box trusts Proxmox by the CA copy and the copy does not verify the served certificate", "err", perr) + set("ca", firstNonEmpty(cause, ProblemCopyNoProof), detail, &c) + return false + } + if cause == "" { + // Proxmox did not answer at all (it restarts, the box just booted): nothing is known about the copy, so no + // cause is named; the next run asks again. + lg.Warn("pvetrust: the proof could not be made now (not a trust failure) — the trust stays as it is; asked again later", "err", perr) + set(modeOf(cur), "", "", &c) + return false + } + lg.Error("pvetrust: the proof with the CA copy FAILED — the fingerprint STAYS, nothing was written", "err", perr) + set(modeOf(cur), ProblemCopyNoProof, detail+" — the fingerprint stays", &c) + return false + } + if onCopy { + // the copy was refreshed (or a trust failure asked): the running client reloads it + if m.Apply != nil { + if aerr := m.Apply(ca); aerr != nil { + set("ca", proxmox.TrustCAFileUnreadable, truncate(aerr.Error(), 200), &c) + return false + } + } + lg.Warn("pvetrust: the CA copy was refreshed and proven — the running client uses it now", "outcome", c.Outcome) + set("ca", "", "", &c) + return true + } + // 3. the config: ONE write sets ca_file and clears the fingerprint — only now, after the proof + if m.ConfigPath == "" { + lg.Warn("pvetrust: the proof worked, but the config did not come from a file — nothing can be saved, so the trust stays as it is") + set(modeOf(cur), ProblemConfigNotSaved, "the config is not a file; the move was not made", &c) + return false + } + if werr := m.writeConfig(c.Path); werr != nil { + lg.Error("pvetrust: the proof worked but the config could NOT be written — the fingerprint stays", "err", werr) + set(modeOf(cur), ProblemConfigNotSaved, truncate(werr.Error(), 200), &c) + return false + } + // 4. the running client + if m.Apply != nil { + if aerr := m.Apply(ca); aerr != nil { + lg.Error("pvetrust: the config is written but the running client could not take the CA copy — it keeps the old trust until the next start", "err", aerr) + } + } + m.mu.Lock() + m.cur = ca + m.mu.Unlock() + lg.Warn("pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy", + "ca_file", c.Path, "ca_not_after", c.NotAfter) + set("ca", "", "", &c) + m.mu.Lock() + m.status.MovedAt = m.now().UTC().Format(time.RFC3339) + m.mu.Unlock() + return true +} + +func firstNonEmpty(a, b string) string { + if a != "" { + return a + } + return b +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] +} + +// writeConfig sets proxmox.tls.ca_file and clears proxmox.tls.fingerprint in ONE write of agent.json. Every other key is +// kept byte for byte. A recovery copy is parked in the state folder first; the result is read back and checked. +func (m *Manager) writeConfig(caPath string) error { + raw, err := os.ReadFile(m.ConfigPath) + if err != nil { + return err + } + var doc map[string]json.RawMessage + if err := json.Unmarshal(raw, &doc); err != nil { + return fmt.Errorf("parse %s: %w", m.ConfigPath, err) + } + px := map[string]json.RawMessage{} + if cur, ok := doc["proxmox"]; ok { + if err := json.Unmarshal(cur, &px); err != nil { + return fmt.Errorf("parse the proxmox section: %w", err) + } + } + tlsSec := map[string]json.RawMessage{} + if cur, ok := px["tls"]; ok { + if err := json.Unmarshal(cur, &tlsSec); err != nil { + return fmt.Errorf("parse the proxmox.tls section: %w", err) + } + } + caJSON, _ := json.Marshal(caPath) + tlsSec["ca_file"] = caJSON + tlsSec["fingerprint"] = json.RawMessage(`""`) + b, _ := json.Marshal(tlsSec) + px["tls"] = b + b, _ = json.Marshal(px) + doc["proxmox"] = b + out, err := json.MarshalIndent(doc, "", " ") + if err != nil { + return err + } + out = append(out, '\n') + st, err := os.Stat(m.ConfigPath) + if err != nil { + return err + } + if m.StateDir != "" { + if err := os.MkdirAll(m.StateDir, 0o700); err != nil { + return err + } + if err := os.WriteFile(filepath.Join(m.StateDir, "agent.json.pre-pve-trust"), raw, 0o600); err != nil { + return fmt.Errorf("parking the recovery copy: %w", err) + } + } + // In place (O_TRUNC), not tmp+rename: the folder is root-owned and the file is the agent's (the same reason as the + // backup-target move and the PBS seed). One small write, then fsync. + f, err := os.OpenFile(m.ConfigPath, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm()) + if err != nil { + return err + } + if _, err := f.Write(out); err != nil { + f.Close() + _ = os.WriteFile(m.ConfigPath, raw, st.Mode().Perm()) // put the old bytes back: the fingerprint must not be lost + return err + } + if err := f.Sync(); err != nil { + f.Close() + return err + } + if err := f.Close(); err != nil { + return err + } + // read back: the file must say exactly what was meant + var check struct { + Proxmox struct { + TLS struct { + CAFile string `json:"ca_file"` + Fingerprint string `json:"fingerprint"` + } `json:"tls"` + } `json:"proxmox"` + } + back, err := os.ReadFile(m.ConfigPath) + if err != nil || json.Unmarshal(back, &check) != nil || check.Proxmox.TLS.CAFile != caPath || check.Proxmox.TLS.Fingerprint != "" { + _ = os.WriteFile(m.ConfigPath, raw, st.Mode().Perm()) + return fmt.Errorf("the config did not read back as written (%v) — the old config was put back", err) + } + return nil +} diff --git a/internal/pvetrust/pvetrust_test.go b/internal/pvetrust/pvetrust_test.go new file mode 100644 index 0000000..c745e83 --- /dev/null +++ b/internal/pvetrust/pvetrust_test.go @@ -0,0 +1,489 @@ +package pvetrust + +import ( + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/sha256" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "encoding/hex" + "encoding/json" + "encoding/pem" + "errors" + "fmt" + "math/big" + "net" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" +) + +// ---- a small Proxmox: a root CA that signs node certificates, and a server that can change its certificate ---- + +type pki struct { + t *testing.T + caKey *ecdsa.PrivateKey + caCert *x509.Certificate + caPEM []byte +} + +func newPKI(t *testing.T, cn string) *pki { + t.Helper() + k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + return (&pki{t: t, caKey: k}).renewCA(cn) +} + +// renewCA makes a new CA certificate with the SAME key and name (what Proxmox does when its root CA is renewed). +func (p *pki) renewCA(cn string) *pki { + tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: cn}, + NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true, + KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign} + der, err := x509.CreateCertificate(rand.Reader, tpl, tpl, &p.caKey.PublicKey, p.caKey) + if err != nil { + p.t.Fatal(err) + } + p.caCert, _ = x509.ParseCertificate(der) + p.caPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}) + return p +} + +func (p *pki) leaf() tls.Certificate { + k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: "node"}, + NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(12 * time.Hour), IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}, + KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}} + der, err := x509.CreateCertificate(rand.Reader, tpl, p.caCert, &k.PublicKey, p.caKey) + if err != nil { + p.t.Fatal(err) + } + return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: k} +} + +func fp(c tls.Certificate) string { + s := sha256.Sum256(c.Certificate[0]) + return hex.EncodeToString(s[:]) +} + +type pve struct { + srv *httptest.Server + mu sync.Mutex + cert tls.Certificate +} + +func newPVE(t *testing.T, c tls.Certificate) *pve { + p := &pve{cert: c} + p.srv = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + fmt.Fprint(w, `{"data":{"version":"9.2.2","release":"9.2","repoid":"x"}}`) + })) + // GetConfigForClient, not GetCertificate: httptest adds its own certificate, and for a call by IP address (no + // server name) that one would win over GetCertificate. + p.srv.TLS = &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) { + p.mu.Lock() + defer p.mu.Unlock() + return &tls.Config{Certificates: []tls.Certificate{p.cert}}, nil + }} + p.srv.StartTLS() + t.Cleanup(p.srv.Close) + return p +} + +// newCert is `pveupdate` making a new node certificate: served from the next connection on. +func (p *pve) newCert(c tls.Certificate) { + p.mu.Lock() + p.cert = c + p.mu.Unlock() + p.srv.CloseClientConnections() +} + +// ---- the agent's side ---- + +type box struct { + t *testing.T + m *Manager + px *proxmox.Client + cfgPath string + copyPath string + source []byte // what the root step would copy (nil = the root step refuses) + events []string + applyErr error +} + +func cfgJSON(fingerprint, caFile string) string { + return fmt.Sprintf(`{ + "hub": {"url": "https://hub.example", "host_id": "box-1", "api_key": "KEEP-ME"}, + "proxmox": {"endpoint": "X", "node": "n", "token": "u@pve!t=s", "tls": {"fingerprint": %q, "ca_file": %q, "insecure_skip_verify": false}, "custom": [1, 2, 3]}, + "unknown_key": {"stays": true} +}`, fingerprint, caFile) +} + +func newBox(t *testing.T, srv *pve, start proxmox.TLSConfig, source []byte) *box { + t.Helper() + dir := t.TempDir() + b := &box{t: t, cfgPath: filepath.Join(dir, "agent.json"), copyPath: filepath.Join(dir, "pve-root-ca.pem"), source: source} + if start.CAFile == "COPY" { + start.CAFile = b.copyPath + if err := os.WriteFile(b.copyPath, source, 0o644); err != nil { + t.Fatal(err) + } + } + if err := os.WriteFile(b.cfgPath, []byte(cfgJSON(start.Fingerprint, start.CAFile)), 0o600); err != nil { + t.Fatal(err) + } + pcfg := proxmox.Config{Endpoint: srv.srv.URL, Node: "n", Token: "u@pve!t=s", TLS: start, HTTPTimeout: 5 * time.Second} + px, err := proxmox.NewClient(pcfg) + if err != nil { + t.Fatal(err) + } + b.px = px + b.m = &Manager{ConfigPath: b.cfgPath, StateDir: filepath.Join(dir, "state"), + EnsureCopy: func(context.Context) (Copy, error) { + b.events = append(b.events, "copy") + if b.source == nil { + return Copy{}, errors.New(`pve-ca-copy refused: {"code": "T3", "reason": "not a CA certificate"}`) + } + outcome := "same" + if cur, _ := os.ReadFile(b.copyPath); string(cur) != string(b.source) { + outcome = "written" + if err := os.WriteFile(b.copyPath, b.source, 0o644); err != nil { + t.Fatal(err) + } + } + return Copy{Path: b.copyPath, Outcome: outcome, NotAfter: "2036-08-17T15:44:42Z", DaysLeft: 3560}, nil + }, + Probe: func(ctx context.Context, tl proxmox.TLSConfig) error { + b.events = append(b.events, fmt.Sprintf("probe(fp=%q,ca=%v) while the file says fp=%q", tl.Fingerprint, tl.CAFile != "", b.fileTLS().Fingerprint)) + c := pcfg + c.TLS = tl + p, err := proxmox.NewClient(c) + if err != nil { + return err + } + _, err = p.Version(ctx) + return err + }, + Apply: func(tl proxmox.TLSConfig) error { + b.events = append(b.events, fmt.Sprintf("apply(fp=%q,ca=%v)", tl.Fingerprint, tl.CAFile != "")) + if b.applyErr != nil { + return b.applyErr + } + return b.px.SetTLS(tl) + }, + } + b.m.Start(start) + return b +} + +type fileTLS struct { + Fingerprint string `json:"fingerprint"` + CAFile string `json:"ca_file"` +} + +func (b *box) fileTLS() fileTLS { + var d struct { + Proxmox struct { + TLS fileTLS `json:"tls"` + } `json:"proxmox"` + } + raw, _ := os.ReadFile(b.cfgPath) + if err := json.Unmarshal(raw, &d); err != nil { + b.t.Fatalf("agent.json is not JSON any more: %v\n%s", err, raw) + } + return d.Proxmox.TLS +} + +func (b *box) call() error { + _, err := b.px.Version(context.Background()) + return err +} + +func (b *box) has(prefix string) bool { + for _, e := range b.events { + if strings.HasPrefix(e, prefix) { + return true + } + } + return false +} + +// Scenario: a NEW box — the installer wrote the copy and ca_file, no fingerprint. Nothing is proven again, nothing written. +func TestNewBox_CAOnly_NothingToMove(t *testing.T) { + p := newPKI(t, "PVE CA") + srv := newPVE(t, p.leaf()) + b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM) + before, _ := os.ReadFile(b.cfgPath) + if b.m.Ensure(context.Background(), "start") { + t.Fatal("a box that already trusts by the copy changed its trust") + } + after, _ := os.ReadFile(b.cfgPath) + if string(before) != string(after) || b.has("probe") || b.has("apply") { + t.Fatalf("the config was touched or a proof ran: %v", b.events) + } + if st := b.m.Status(); st.Mode != "ca" || st.Problem != "" || st.CADaysLeft == nil || *st.CADaysLeft != 3560 { + t.Fatalf("status = %+v", st) + } + if err := b.call(); err != nil { + t.Fatalf("the call fails: %v", err) + } +} + +// Scenario: an OLD box moves. The proof runs with the CA copy ALONE while the file still holds the fingerprint; only +// then ONE write sets ca_file and clears the fingerprint; every other key stays; a recovery copy is parked. +// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/A/redproof-agent.txt): write the config +// BEFORE the probe → the order check fails; skip the probe → TestProofFails_TheFingerprintStays fails. +func TestOldBox_MovesAndTheFingerprintIsClearedOnlyAfterTheProof(t *testing.T) { + p := newPKI(t, "PVE CA") + leaf := p.leaf() + srv := newPVE(t, leaf) + b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM) + old, _ := os.ReadFile(b.cfgPath) + if !b.m.Ensure(context.Background(), "start") { + t.Fatalf("the box did not move: %+v %v", b.m.Status(), b.events) + } + want := []string{"copy", fmt.Sprintf("probe(fp=\"\",ca=true) while the file says fp=%q", fp(leaf)), "apply(fp=\"\",ca=true)"} + if strings.Join(b.events, " | ") != strings.Join(want, " | ") { + t.Fatalf("order = %v\nwant %v", b.events, want) + } + if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath { + t.Fatalf("the config's tls = %+v", got) + } + raw, _ := os.ReadFile(b.cfgPath) + for _, keep := range []string{`"KEEP-ME"`, `"unknown_key"`, `"stays"`, `"custom"`, `"u@pve!t=s"`, `"insecure_skip_verify"`} { + if !strings.Contains(string(raw), keep) { + t.Fatalf("the write lost %s:\n%s", keep, raw) + } + } + if parked, _ := os.ReadFile(filepath.Join(b.m.StateDir, "agent.json.pre-pve-trust")); string(parked) != string(old) { + t.Fatal("no recovery copy of the old config was parked") + } + if st := b.m.Status(); st.Mode != "ca" || st.MovedAt == "" || st.Problem != "" { + t.Fatalf("status = %+v", st) + } + // and now Proxmox makes a new certificate: the agent goes on, with no restart and no new client + srv.newCert(p.leaf()) + if err := b.call(); err != nil { + t.Fatalf("after a new Proxmox certificate the moved box fails: %v", err) + } +} + +// Scenario: the proof fails (the copy is another box's CA). The fingerprint stays — in the file, byte for byte, and in +// the running client — and the cause is named. +func TestProofFails_TheFingerprintStays(t *testing.T) { + p := newPKI(t, "PVE CA") + leaf := p.leaf() + srv := newPVE(t, leaf) + other := newPKI(t, "another box") + b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM) + before, _ := os.ReadFile(b.cfgPath) + if b.m.Ensure(context.Background(), "start") { + t.Fatal("the trust changed although the proof failed") + } + after, _ := os.ReadFile(b.cfgPath) + if string(before) != string(after) { + t.Fatal("the config was written although the proof failed") + } + if b.has("apply") { + t.Fatalf("the running client was changed: %v", b.events) + } + if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemCopyNoProof || !strings.Contains(st.Detail, "did not sign") { + t.Fatalf("status = %+v", st) + } + if err := b.call(); err != nil { + t.Fatalf("the fingerprint no longer works: %v", err) + } +} + +// Scenario: Proxmox makes a new certificate on a box that still PINS the old one. The failed call names the cause, the +// box moves to the CA copy right then, and the next call works — no restart. +func TestPinnedBox_NewCertificate_HealsWithNoRestart(t *testing.T) { + p := newPKI(t, "PVE CA") + leaf := p.leaf() + srv := newPVE(t, leaf) + b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM) + srv.newCert(p.leaf()) + err := b.call() + if proxmox.TrustProblem(err) != proxmox.TrustPinnedCertChanged { + t.Fatalf("fixture: want a pin mismatch, got %v", err) + } + b.m.OnCallError(context.Background(), err) + if err := b.call(); err != nil { + t.Fatalf("the box is still blind after the move: %v (%v)", err, b.events) + } + if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath { + t.Fatalf("the config's tls = %+v", got) + } + // a failure that is NOT one of trust does nothing + n := len(b.events) + b.m.OnCallError(context.Background(), errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused")) + if len(b.events) != n { + t.Fatalf("a refused connection started a trust run: %v", b.events[n:]) + } +} + +// Scenario: the root CA itself is renewed with the same key. The copy is refreshed, proven, and the running client +// reloads it; the config is not written again. +func TestCARenewedWithTheSameKey_TheCopyIsRefreshedAndReloaded(t *testing.T) { + p := newPKI(t, "PVE CA") + srv := newPVE(t, p.leaf()) + b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM) + oldCA := string(p.caPEM) + p.renewCA("PVE CA") + srv.newCert(p.leaf()) + b.source = p.caPEM + before, _ := os.ReadFile(b.cfgPath) + if !b.m.Ensure(context.Background(), "daily") { + t.Fatalf("a refreshed copy was not taken: %+v %v", b.m.Status(), b.events) + } + if cur, _ := os.ReadFile(b.copyPath); string(cur) == oldCA || string(cur) != string(p.caPEM) { + t.Fatal("the copy was not refreshed") + } + if !b.has("probe") || !b.has("apply") { + t.Fatalf("a refreshed copy must be proven and reloaded: %v", b.events) + } + if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) { + t.Fatal("the config was written again") + } + if err := b.call(); err != nil { + t.Fatalf("the call fails after the CA's renewal: %v", err) + } +} + +// The root step refuses (the source is not a CA certificate): nothing changes, the cause is named. +func TestRootStepRefuses_NothingChanges(t *testing.T) { + p := newPKI(t, "PVE CA") + leaf := p.leaf() + srv := newPVE(t, leaf) + b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil) + before, _ := os.ReadFile(b.cfgPath) + if b.m.Ensure(context.Background(), "start") || b.has("probe") || b.has("apply") { + t.Fatalf("something ran without a copy: %v", b.events) + } + if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) { + t.Fatal("the config was written") + } + if st := b.m.Status(); st.Problem != ProblemCopyUnavailable || !strings.Contains(st.Detail, "T3") { + t.Fatalf("status = %+v", st) + } +} + +// The proof works but the config cannot be written: the fingerprint stays, in the file and in the running client. +func TestConfigCannotBeWritten_TheFingerprintStays(t *testing.T) { + if os.Geteuid() == 0 { + t.Skip("root writes a read-only file") + } + p := newPKI(t, "PVE CA") + leaf := p.leaf() + srv := newPVE(t, leaf) + b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM) + if err := os.Chmod(b.cfgPath, 0o400); err != nil { + t.Fatal(err) + } + if b.m.Ensure(context.Background(), "start") || b.has("apply") { + t.Fatalf("the trust changed without a saved config: %v", b.events) + } + if got := b.fileTLS(); got.Fingerprint != fp(leaf) { + t.Fatalf("the fingerprint is gone from the file: %+v", got) + } + if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemConfigNotSaved { + t.Fatalf("status = %+v", st) + } +} + +// What is never moved: an operator's own CA file, a box with checks switched off, a config that is not a file. +func TestNeverMoved(t *testing.T) { + p := newPKI(t, "PVE CA") + leaf := p.leaf() + srv := newPVE(t, leaf) + own := filepath.Join(t.TempDir(), "own-ca.pem") + os.WriteFile(own, p.caPEM, 0o644) + for name, start := range map[string]proxmox.TLSConfig{"own CA file": {CAFile: own}, "insecure": {InsecureSkipVerify: true}} { + b := newBox(t, srv, start, p.caPEM) + before, _ := os.ReadFile(b.cfgPath) + if b.m.Ensure(context.Background(), "start") || b.has("apply") { + t.Fatalf("%s: moved (%v)", name, b.events) + } + if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) { + t.Fatalf("%s: the config was written", name) + } + } + b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM) + b.m.ConfigPath = "" + if b.m.Ensure(context.Background(), "start") || b.has("apply") || b.m.Status().Problem != ProblemConfigNotSaved { + t.Fatalf("a config that is not a file was 'moved': %v %+v", b.events, b.m.Status()) + } +} + +// The copy ends soon: the cause is named (the box still works). +func TestCopyEndsSoon(t *testing.T) { + p := newPKI(t, "PVE CA") + srv := newPVE(t, p.leaf()) + b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM) + inner := b.m.EnsureCopy + b.m.EnsureCopy = func(ctx context.Context) (Copy, error) { + c, err := inner(ctx) + c.DaysLeft, c.NotAfter = 12, "2026-11-26T00:00:00Z" + return c, err + } + b.m.Ensure(context.Background(), "daily") + if st := b.m.Status(); st.Problem != ProblemCopyEndsSoon || !strings.Contains(st.Detail, "12 day") { + t.Fatalf("status = %+v", st) + } +} + +// A trust failure asks at most once per MinGap. +func TestOnCallError_Throttled(t *testing.T) { + p := newPKI(t, "PVE CA") + leaf := p.leaf() + srv := newPVE(t, leaf) + other := newPKI(t, "another box") + b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM) + srv.newCert(p.leaf()) + err := b.call() + now := time.Date(2026, 11, 15, 3, 0, 0, 0, time.UTC) + b.m.Now = func() time.Time { return now } + b.m.OnCallError(context.Background(), err) + n := len(b.events) + b.m.OnCallError(context.Background(), err) + if len(b.events) != n { + t.Fatalf("a second failure inside the gap ran again: %v", b.events[n:]) + } + now = now.Add(11 * time.Minute) + b.m.OnCallError(context.Background(), err) + if len(b.events) == n { + t.Fatal("no new run after the gap") + } + if st := b.m.Status(); st.Problem == "" { + t.Fatalf("a blind box names no cause: %+v", st) + } +} + +// Proxmox does not answer at all (it restarts): nothing is known about the copy — no cause is named, nothing is written, +// and the box counts as not moved, so the timer asks again soon. +func TestProofCannotBeMadeNow_NoCauseNamed(t *testing.T) { + p := newPKI(t, "PVE CA") + leaf := p.leaf() + srv := newPVE(t, leaf) + b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM) + b.m.Probe = func(context.Context, proxmox.TLSConfig) error { + return errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused") + } + before, _ := os.ReadFile(b.cfgPath) + if b.m.Ensure(context.Background(), "start") { + t.Fatal("moved without a proof") + } + if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) { + t.Fatal("the config was written") + } + if st := b.m.Status(); st.Problem != "" || st.Mode != "fingerprint" || b.m.Moved() { + t.Fatalf("status = %+v moved=%v", st, b.m.Moved()) + } +}