- Golden 0.352.0: bake, publish, one fresh install, the operator's vouch (R-468). - No clone holds a login in its address; the credential helper script is in the repository (R-925, R-1390). - N100 clock: a stale cached AppArmor profile stops chrony (R-1301, new R-1388). Read only. - Certificate pin spike on demo-hp (R-1380): the break, the proven hand repair (runbook), the CA-file test, 03-host-agent.md 12a, decision D80. - Rulings of 13:17 written down (09 decisions 297-302). Register 245 -> 248 (R-1388, R-1389, R-1390). - Docs, evidence and one helper script. No release, no floor change, no hub deploy. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The Proxmox certificate changes and the agent's pin does not follow — spike on demo-hp, 2026-11-11
Row R-1380. Box: demo-hp only. No agent code, no release. Files: evidence/ (26 files), tools/ (the scripts).
Documents read: architecture/01-topology-and-trust.md, 03-host-agent.md, 04-control-plane-authorization.md;
the End-dates list entry DB02 and its evidence/box/. No architecture document named this pin before today
(a search of 01, 03 and 04 found no line; now 03-host-agent.md §12a).
What was done, in order (times in UTC)
| Time | Step | Result | File |
|---|---|---|---|
| 13:49:16 | Before | served certificate = file = agent's pin (07:B5:72:5D…07:BB, ends 2028-08-19); agent calls answer 200 |
01, 05 |
| — | Who else trusts it | port 8006's log: only the agent, from 127.0.0.1 (75,374 lines). No file outside the box holds the fingerprint | 02 |
| 13:49:23 | pvecm updatecerts --force |
new file (21:D6:B5:E0…84:12); Proxmox still serves the old one, the agent still works |
06, 07 |
| 13:50:23 | systemctl reload pveproxy |
the new certificate is served about 10 s later | 08 |
| 13:50:39 | The agent | first TLS pin mismatch line; every call to Proxmox fails from here |
09, 10 |
| 13:59:25 | Hand repair | new fingerprint into agent.json, restart: a working call 2.1 s later |
16, 17 |
| 13:59:39 | CA file in place of the fingerprint | the agent connects (2.2 s) | 18, 19 |
| 13:59:54 | Second forced certificate (1F:75:D3:51…0E:74) + reload |
the agent goes on, no restart, no hand step, 0 TLS lines | 20 |
| 14:01:02 | Control: a CA file that did not sign it | x509: certificate signed by unknown authority (so the CA file is really checked) |
25 |
| 14:02:46 | Back to the standard | pin = served certificate; 77 of 77 capabilities; hub report at 14:02:53 | 22, 24, 26 |
What broke while the pin was wrong (8 min 56 s, by choice)
- The agent's line:
proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint, as a WARN in each loop. It does not say "the certificate changed" and does not say what to do. - Failed: the guest list, the storage view, the guest config, the pool read, the node status. So: reconcile
(
pass failed), the controller supervisor, the guest-power watchdog, the guest network sweep, the stale-lock recovery, and the local API's/storageand/disks(502 to the controller). From the code, not tried: backup, restore, snapshot, start/stop, resize all use the same client, so they fail too. - The hub: the agent sent no host report (
hub: collect failed; skipping this cycle's report). The Hosts page still readONLINE 1/2eleven minutes after the last report. Not measured: when it turns to offline. - The alarm: one mail to the operator after 94 s,
agent_channel_unknown (warning) — Controller→agent channel down: agentapi: GET /storage: HTTP 502. It names a symptom, not the cause. A recovery event came 39 s after the repair. - Stayed up: both guests, Tester 1's VM, all 26 containers in the guest, the enrolled drive.
Three facts the fix must know
- Proxmox's own renewal reloads the web service by itself (
pveupdate, daily timer): node certificate at most 2 × 365 days, renewed when it ends within 14 days. So on a real box the break comes the same night (03). - The agent cannot read Proxmox's certificate files. It runs as user
felhom-agent;/etc/pve/pve-root-ca.pemand the node certificate are0640 root:www-data. The CA test used a copy in/etc/felhom-agent/(18). The ruled fix shape ("the agent reads the certificate file on its own host") needs a root step for the same reason. - The fingerprint wins. Config key
proxmox.tls.ca_fileexists, but it is used only whenproxmox.tls.fingerprintis empty (felhom-agent/internal/proxmox/tls.go).
The root CA lasts 10 years (demo-hp: to 2036-08-17). Proxmox renews it when less than two years are left (about
August 2034) and keeps its key and its name, so new node certificates still pass a copy of the old CA file until
that copy's own end date. Read from Proxmox's code (03), not forced.
The two ways, and the pick → 03-host-agent.md §12a, decision D80 in STATUS.md
CA file (CC's pick): proven here, no agent code. Follow the fingerprint: needs new agent code and a root helper. Both need one root step to reach the boxes that exist.
CC's mistakes in this run
- The first control ran under 1 s (the waiter matched an older call): void. The repeat met systemd's start limit
(4 starts in 2 minutes): the agent was stopped for 92 s (14:01:14 to 14:02:46). The failure hook ran and did
nothing ("no pending update"). The proof for the control comes from the 1-second run's own log (
23,24,25). - The scripts rewrote
agent.jsonwith a new layout. The content is equal to the saved copy (24:rest equal: True).
How demo-hp was left
Fingerprint pin equal to the served certificate (1F:75:D3:51…0E:74, ends 2028-11-09). Agent active, 0 restarts,
calls answer 200. The CA copy and the saved config are removed. The certificate itself is new: that is the one
lasting change, and it moves demo-hp's end date from 2028-08-19 to 2028-11-09.