Golden 0.352.0 vouched; token fix finished; why the N100 keeps no time; certificate pin spike on demo-hp
gates / gates (push) Successful in 3m34s

- Golden 0.352.0: bake, publish, one fresh install, the operator's vouch (R-468).
- No clone holds a login in its address; the credential helper script is in the repository (R-925, R-1390).
- N100 clock: a stale cached AppArmor profile stops chrony (R-1301, new R-1388). Read only.
- Certificate pin spike on demo-hp (R-1380): the break, the proven hand repair (runbook), the CA-file test,
  03-host-agent.md 12a, decision D80.
- Rulings of 13:17 written down (09 decisions 297-302). Register 245 -> 248 (R-1388, R-1389, R-1390).
- Docs, evidence and one helper script. No release, no floor change, no hub deploy.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-11 15:15:20 +01:00
1 parent 978666dde3
commit d4c7e84c22
74 files changed
+3376 -14

No files matched your search

@@ -933,6 +933,45 @@ tunnel token, encrypted storage creds (NFS/CIFS/PBS), and the **live PBS key**.
and the secret footprint that left the controller now concentrate here — which is the whole
argument for §3's root-minimization and a small, auditable agent.
### 12a. How the agent trusts Proxmox's certificate, and what happens when it changes (R-1380)
`[FACT — measured on demo-hp 2026-11-11]` (`documentation/audits/certpin-spike-2026-11-11/README.md`)
- The agent calls Proxmox at `https://127.0.0.1:8006`. It trusts **one exact certificate**: the installer reads the
served certificate and writes its sha256 into `proxmox.tls.fingerprint` (`scripts/felhom-host-install.sh`). The
order in the agent is: skip-verify, then fingerprint, then `proxmox.tls.ca_file`, then the system's roots
(`felhom-agent/internal/proxmox/tls.go`). **A set fingerprint wins over a CA file.**
- Proxmox makes a new node certificate by itself (`pveupdate`, daily timer; `PVE/Cluster/Setup.pm`, pve-cluster
9.1.9): the certificate lasts at most 2 × 365 days and is renewed when it ends within 14 days. `pveupdate` then
reloads the web service, so the new certificate is served the same night.
- From that moment every agent call to Proxmox fails with `TLS pin mismatch`. Measured: no reconcile, no guest list,
no storage view, no host report to the hub; the controller gets 502 from the local API. Guests and apps stay up.
One operator mail names the symptom (`agent_channel_unknown`), none names the cause. The hand repair is
`runbooks/RUNBOOK-agent-proxmox-pin-repair.md` (2.1 s from restart to a working call).
- The agent's user **cannot read** `/etc/pve/pve-root-ca.pem` or the node certificate (`0640 root:www-data`).
- With the fingerprint empty and `ca_file` set to a readable copy of Proxmox's root CA, the agent connected, and it
went on through a second forced certificate with no restart and no hand step. A CA file that did not sign the
certificate was refused (`x509: certificate signed by unknown authority`).
- Proxmox's root CA lasts 3650 days. Its code renews it when less than two years are left, **with the same key and
the same name**; a node certificate is not made from a CA that ends in less than 14 days. Read from the code, not
forced.
`[DESIGN — proposed, not built]` Two ways. The operator's ruling of 2026-11-11 13:17 (`09` §3 decision 297) names the
first; the spike points to the second, so the choice is back with the operator (`STATUS.md` D80).
| | A — the agent follows the fingerprint | B — the agent trusts the box's own root CA (**CC's pick**) |
|---|---|---|
| How | A root helper reads the node certificate after a mismatch and gives the agent the new fingerprint | The installer puts a copy of `/etc/pve/pve-root-ca.pem` where the agent can read it, sets `ca_file`, leaves the fingerprint empty |
| New agent code | yes: the helper, a sudo rule, the re-pin logic, its tests | none for the trust itself (the key exists and is proven) |
| Trust | one exact certificate | every certificate this box's own CA signs; the CA's key is root-only on the same box |
| Next hand step | none, if the helper is right | none until the copy's own end date (10 years from install); the copy must be refreshed after Proxmox renews the CA |
| Boxes that exist | one root step each (the signed config bundle) | the same |
Why B: it is the standard way to trust a private CA, it is proven on a box, and it removes the failure with no new
privileged code. What B still needs before it is built: the root step that refreshes the copy (and a test that an
old copy raises an alarm that names the cause), and a rule for what the agent does if both keys are set.
Whichever way is built, the mismatch must raise an alarm that names the cause.
## 13. Open items / what this unblocks
Resolved here: tunnel placement (host, agent-managed, own systemd service), the
@@ -1265,6 +1265,28 @@ Recorded before the work (`audits/space-list-2026-11-10/README.md` §5 held the
Hetzner and the mail service are read from source and documents only: no hub database read, no account call.
*Operator, 2026-11-10 12:22.*
### 2026-11-11 (13:17, operator, planning chat) — four rulings on the End-dates and Clock lists; the clones; the gap review pauses
Recorded before any build. **None of these is built.** The questions were D76–D79 in `STATUS.md`
(`audits/end-dates-list-2026-11-11/README.md`, `audits/clock-list-2026-11-11/README.md`).
297. **D76 = yes: the agent follows Proxmox's new certificate by itself** — after the failure is seen on demo-hp
(`audits/certpin-spike-2026-11-11/README.md`). **Cost of the other option:** a hand step in a runbook and a
calendar date for each box, for ever. The two ways to build it are in `03-host-agent.md` (the part on the Proxmox
certificate pin). R-1380. *Operator, 2026-11-11 13:17.*
298. **D77 = yes: the product warns 14 days before an end date** (a box's web certificate, a hub token). **Cost of the
other option:** the dates stay in a calendar kept by hand. R-1379. *Operator, 2026-11-11 13:17.*
299. **D78 = yes: a timed job that gives up or is skipped tells the operator — the four cases:** the guest-power
watchdog gave up; a hub daily check was skipped; an operator mail was lost; a signed job was refused. **Cost of
the other option:** the logs stay the only trace. R-1383. *Operator, 2026-11-11 13:17.*
300. **D79 = yes: the hub checks each box's clock and alarms above one minute of error.** **Cost of the other option:**
the repair of one box's time service (R-1301) stays the only guard. R-1384. *Operator, 2026-11-11 13:17.*
301. **The two other clone addresses are fixed the same way as `felhom-agent`'s** (no login in a repository address;
`audits/no-token-in-address-2026-11-11/`). **The token rotation is the operator's and comes later; CC does not ask
again.** R-925. *Operator, 2026-11-11 13:17.*
302. **The gap review pauses after four lists** (crossings, endings, end dates, clock): the operator chose „build”.
The lists „outside services”, „sizes” and „fleet” are not run. *Operator, 2026-11-11 13:17.*
### 2026-11-10 (15:19, operator, planning chat) — five rulings on the Endings list and the system disk
Recorded before any build. **None of these is built.** The questions were D71–D75 in `STATUS.md`