Files
felhom.eu/documentation/audits/certpin-spike-2026-11-11
admin d4c7e84c22
gates / gates (push) Successful in 3m34s
Golden 0.352.0 vouched; token fix finished; why the N100 keeps no time; certificate pin spike on demo-hp
- Golden 0.352.0: bake, publish, one fresh install, the operator's vouch (R-468).
- No clone holds a login in its address; the credential helper script is in the repository (R-925, R-1390).
- N100 clock: a stale cached AppArmor profile stops chrony (R-1301, new R-1388). Read only.
- Certificate pin spike on demo-hp (R-1380): the break, the proven hand repair (runbook), the CA-file test,
  03-host-agent.md 12a, decision D80.
- Rulings of 13:17 written down (09 decisions 297-302). Register 245 -> 248 (R-1388, R-1389, R-1390).
- Docs, evidence and one helper script. No release, no floor change, no hub deploy.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-11 15:15:20 +01:00
..

The Proxmox certificate changes and the agent's pin does not follow — spike on demo-hp, 2026-11-11

Row R-1380. Box: demo-hp only. No agent code, no release. Files: evidence/ (26 files), tools/ (the scripts). Documents read: architecture/01-topology-and-trust.md, 03-host-agent.md, 04-control-plane-authorization.md; the End-dates list entry DB02 and its evidence/box/. No architecture document named this pin before today (a search of 01, 03 and 04 found no line; now 03-host-agent.md §12a).

What was done, in order (times in UTC)

Time Step Result File
13:49:16 Before served certificate = file = agent's pin (07:B5:72:5D…07:BB, ends 2028-08-19); agent calls answer 200 01, 05
— Who else trusts it port 8006's log: only the agent, from 127.0.0.1 (75,374 lines). No file outside the box holds the fingerprint 02
13:49:23 pvecm updatecerts --force new file (21:D6:B5:E0…84:12); Proxmox still serves the old one, the agent still works 06, 07
13:50:23 systemctl reload pveproxy the new certificate is served about 10 s later 08
13:50:39 The agent first TLS pin mismatch line; every call to Proxmox fails from here 09, 10
13:59:25 Hand repair new fingerprint into agent.json, restart: a working call 2.1 s later 16, 17
13:59:39 CA file in place of the fingerprint the agent connects (2.2 s) 18, 19
13:59:54 Second forced certificate (1F:75:D3:51…0E:74) + reload the agent goes on, no restart, no hand step, 0 TLS lines 20
14:01:02 Control: a CA file that did not sign it x509: certificate signed by unknown authority (so the CA file is really checked) 25
14:02:46 Back to the standard pin = served certificate; 77 of 77 capabilities; hub report at 14:02:53 22, 24, 26

What broke while the pin was wrong (8 min 56 s, by choice)

  • The agent's line: proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint, as a WARN in each loop. It does not say "the certificate changed" and does not say what to do.
  • Failed: the guest list, the storage view, the guest config, the pool read, the node status. So: reconcile (pass failed), the controller supervisor, the guest-power watchdog, the guest network sweep, the stale-lock recovery, and the local API's /storage and /disks (502 to the controller). From the code, not tried: backup, restore, snapshot, start/stop, resize all use the same client, so they fail too.
  • The hub: the agent sent no host report (hub: collect failed; skipping this cycle's report). The Hosts page still read ONLINE 1/2 eleven minutes after the last report. Not measured: when it turns to offline.
  • The alarm: one mail to the operator after 94 s, agent_channel_unknown (warning) — Controller→agent channel down: agentapi: GET /storage: HTTP 502. It names a symptom, not the cause. A recovery event came 39 s after the repair.
  • Stayed up: both guests, Tester 1's VM, all 26 containers in the guest, the enrolled drive.

Three facts the fix must know

  1. Proxmox's own renewal reloads the web service by itself (pveupdate, daily timer): node certificate at most 2 × 365 days, renewed when it ends within 14 days. So on a real box the break comes the same night (03).
  2. The agent cannot read Proxmox's certificate files. It runs as user felhom-agent; /etc/pve/pve-root-ca.pem and the node certificate are 0640 root:www-data. The CA test used a copy in /etc/felhom-agent/ (18). The ruled fix shape ("the agent reads the certificate file on its own host") needs a root step for the same reason.
  3. The fingerprint wins. Config key proxmox.tls.ca_file exists, but it is used only when proxmox.tls.fingerprint is empty (felhom-agent/internal/proxmox/tls.go).

The root CA lasts 10 years (demo-hp: to 2036-08-17). Proxmox renews it when less than two years are left (about August 2034) and keeps its key and its name, so new node certificates still pass a copy of the old CA file until that copy's own end date. Read from Proxmox's code (03), not forced.

The two ways, and the pick → 03-host-agent.md §12a, decision D80 in STATUS.md

CA file (CC's pick): proven here, no agent code. Follow the fingerprint: needs new agent code and a root helper. Both need one root step to reach the boxes that exist.

CC's mistakes in this run

  • The first control ran under 1 s (the waiter matched an older call): void. The repeat met systemd's start limit (4 starts in 2 minutes): the agent was stopped for 92 s (14:01:14 to 14:02:46). The failure hook ran and did nothing ("no pending update"). The proof for the control comes from the 1-second run's own log (23, 24, 25).
  • The scripts rewrote agent.json with a new layout. The content is equal to the saved copy (24: rest equal: True).

How demo-hp was left

Fingerprint pin equal to the served certificate (1F:75:D3:51…0E:74, ends 2028-11-09). Agent active, 0 restarts, calls answer 200. The CA copy and the saved config are removed. The certificate itself is new: that is the one lasting change, and it moves demo-hp's end date from 2028-08-19 to 2028-11-09.