felhom.eu/documentation/audits/SPIKE-agent-pin-mismatch-2026-06-29.md

SPIKE — controller↔agent TLS leaf-pin mismatch (disk ops + quiesce backup dead)

• Date: 2026-06-29
• Class: Diagnostic / read-only. No code, config, cert, service, or version change was made.
• Host: felhom-pve (demo-felhom, 192.168.0.162), controller in LXC 9201.
• Verdict: Hypothesis #1 — the agent local-API leaf was regenerated during the root→non-root agent migration on 2026-06-28 ~08:56 CEST. The controller's pin (and bootstrap.json seed) still reference the pre-migration leaf. A secondary blocker (empty token store) is latent behind the pin and will surface the moment the pin is fixed.
• RESOLVED 2026-06-29 via R1 (RUNBOOK, supervised) — service restored; see the closing §9 Outcome at the bottom.

---

1. Live running versions

Component	Version / image	Notes
Host agent	felhom-agent 0.43.0	unit felhom-agent.service, runs non-root as felhom-agent, --config /etc/felhom-agent/agent.json, ActiveEnterTimestamp = Sun 2026-06-28 08:59:06 CEST
In-guest controller	felhom-controller:0.89.0	LXC 9201, Up ~1h (healthy) at probe time

---

2. The three fingerprints

Label	Value	Source
AGENT_FP (served now)	ced34036440639f48f4b96fb1802e79b2c1b3102eb763d3ba97aea2da1944f9c	on-disk local-api.crt == live wire (s_client) == boot log — all three agree
PIN_FP (controller pins)	60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245	controller.yaml → local_api.fingerprint (data volume in 9201)
SEED_FP (bootstrap seed)	60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245	/etc/felhom-bootstrap/bootstrap.json → local_api.fingerprint

PIN_FP == SEED_FP == 60b5974d… ≠ AGENT_FP == ced34036…. The controller is talking to the right endpoint (both sides 192.168.0.162:8443) and pinning a consistent but now-obsolete leaf.

---

3. Raw evidence

A — what the agent serves (host felhom-pve)

A1 — socket owner on :8443 (single process, not a stale/second daemon → hypothesis #3 ruled out):

LISTEN 192.168.0.162:8443  users:(("felhom-agent",pid=3550072,fd=18))

A2 — unit + boot-log fingerprint history. The leaf fingerprint logged at every boot is stable 60b5974d… from Jun 24 through Jun 27, then flips to ced34036… on Jun 28 08:56:37, same cert path:

Jun 26 19:54:50  local-api leaf ready fingerprint_sha256=60b5974d…  cert=/var/lib/felhom-agent/local-api.crt
Jun 27 …         (controller swaps to 0.86.0 / 0.87.0; leaf still 60b5974d)
Jun 28 08:56:37  local-api leaf ready fingerprint_sha256=ced34036…  cert=/var/lib/felhom-agent/local-api.crt   ← REGENERATED
Jun 28 08:59:06  local-api leaf ready fingerprint_sha256=ced34036…  cert=/var/lib/felhom-agent/local-api.crt   ← current daemon (pid 3550072)

ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json, FragmentPath=/etc/systemd/system/felhom-agent.service, ActiveEnterTimestamp=2026-06-28 08:59:06.

A3 — on-disk leaf (fresh, owned by the new non-root user, mtime = the switchover):

-rw-r--r-- 1 felhom-agent felhom-agent 591 Jun 28 08:56 local-api.crt
-rw------- 1 felhom-agent felhom-agent 227 Jun 28 08:56 local-api.key
-rw------- 1 felhom-agent felhom-agent   0 Jun 28 08:56 local-tokens.log      ← 0 bytes (see §6)
openssl … DER | sha256sum = ced34036440639f48f4b96fb1802e79b2c1b3102eb763d3ba97aea2da1944f9c
serial=85131255A9AE3FD902A5DB12A15D0459  notBefore=Jun 28 05:56:37 2026 GMT  notAfter=Jun 25 05:56:37 2036 GMT

A4 — on-the-wire leaf (live s_client to 192.168.0.162:8443):

ced34036440639f48f4b96fb1802e79b2c1b3102eb763d3ba97aea2da1944f9c

→ wire == disk == boot-log. No stale/second process; one honest agent serving one fresh cert.

A5 — agent config local_api (paths NOT overridden to anything unexpected; defaults):

"local_api": { "enable": true, "listen_addr": "192.168.0.162:8443",
  "cert_file": "/var/lib/felhom-agent/local-api.crt",
  "key_file":  "/var/lib/felhom-agent/local-api.key",
  "token_store": "/var/lib/felhom-agent/local-tokens.log" }

Surviving pre-migration leaf (the restore candidate). A set-aside copy of the old root-era /var/lib exists from the bundle migration test, and it matches the controller pin exactly:

/root/agent-backup-bundle-test/aside-var-lib/local-api.crt
  DER|sha256 = 60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245   ← == PIN_FP == SEED_FP
  notBefore=Jun 10 06:52:37 2026 GMT   serial=8FFA15857DABD903BB53742E6EA2CE46
  (local-api.key present alongside)

B — what the controller pins (LXC 9201)

B1 — controller.yaml local_api (/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml):

local_api:
    endpoint: 192.168.0.162:8443
    fingerprint: 60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245
    token: <redacted — stored out-of-band>

B2 — bootstrap.json seed (/etc/felhom-bootstrap/bootstrap.json):

"local_api": { "endpoint": "192.168.0.162:8443",
  "fingerprint": "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245" }

B3 — controller error log (every 5 min, two distinct call sites):

[quiesce] cycle error: check due: agentapi: GET /backup/due: … TLS pin mismatch: agent leaf SHA-256 does not match the bootstrap fingerprint
[web] disk list via agent failed: agentapi: GET /disks: … TLS pin mismatch: agent leaf SHA-256 does not match the bootstrap fingerprint

C — endpoint sanity

controller.yaml.endpoint = bootstrap.json.endpoint = agent listen_addr = 192.168.0.162:8443. The controller pins the right host; only the leaf value is stale. Endpoint mismatch ruled out.

---

4. Cert-regeneration ↔ agent-restart correlation

The fresh leaf's notBefore (Jun 28 05:56:37 GMT = 08:56:37 CEST, with the code's −1h skew backdate) and the cert/key mtimes (Jun 28 08:56) coincide to the second with the first post-migration leaf ready log line. The agent then settled into its current daemon at 08:59:06. This is not an expiry event (10-year validity) — it is EnsureLeaf taking the generate branch because the non-root agent's data dir (/var/lib/felhom-agent, now owned felhom-agent:felhom-agent) contained no cert/key at first boot under the new user. The old pair was preserved only in the /root/agent-backup-bundle-test/aside-var-lib/ set-aside, so EnsureLeaf minted a new self-signed leaf and persisted it — changing the served fingerprint out from under every already-issued pin.

This is a direct side-effect of the BUNDLE Day-0 / root→non-root agent migration on 2026-06-28.

---

5. Listening-socket owner

Single process: felhom-agent pid 3550072 (the running daemon), bound to 192.168.0.162:8443. No second/stale listener. Hypothesis #3 (stale process serving an old cert) is disproven — and note the direction is the opposite of #3 anyway: the live process serves the new cert, while the old value is the one frozen in the controller's pin.

---

6. Matched matrix row + conclusion

Matrix row #1 — agent leaf regenerated (files were absent at the new non-root path; not a path typo, not an expiry):

A3 == A2 == fresh mtime (cert regenerated 2026-06-28 ~08:56, the agent redeploy/non-root switch); PIN_FP ≠ AGENT_FP; a surviving backup cert matching PIN_FP does exist (…/aside-var-lib/, fp 60b5974d…).

Root cause (high confidence): the root→non-root agent migration on 2026-06-28 reinitialised /var/lib/felhom-agent without carrying over the existing leaf, so the agent generated a new self-signed local-API leaf (ced34036…). The in-guest controller (and its bootstrap.json) still pin the pre-migration leaf (60b5974d…), so the pinned agentapi client fails closed on every call. Because all disk/storage/backup traffic goes through that one pinned client, both the disk UI and the quiesce app-consistent-backup loop are down.

Note — this is NOT a stale controller re-merge (#2). PIN_FP == SEED_FP, so re-running the controller's bootstrap/ensureLocalAPI merge would re-apply the same obsolete 60b5974d… and not help. The divergence originated on the agent side, not from a stale seed re-merge.

Secondary blocker (latent behind the pin) — empty token store

/var/lib/felhom-agent/local-tokens.log is 0 bytes, mtime 2026-06-28 08:56 (reset at the same migration). The controller's per-guest bearer token was minted into the pre-migration store. The TLS pin fails first (handshake), so this is currently masked — but once the pin is corrected the agent will have no token entry for guest 9201 and authenticated calls will 401. This matches the known "local-API 401 until token re-seed after agent restart / stale token map" pattern. Any remedy below must be paired with re-seeding the guest token (re-provision token mint, or restore the old token store), or the symptom merely changes from TLS pin mismatch to 401.

---

7. Remedy options for the operator (NOT executed — operator's choice)

#	Remedy	Effect	Trade-offs / cautions
R1	Restore the pre-migration leaf: copy …/aside-var-lib/local-api.{crt,key} → /var/lib/felhom-agent/, chown felhom-agent:felhom-agent, 0644/0600, restart agent	Agent serves 60b5974d… again → matches existing PIN_FP+SEED_FP; zero controller-side change	Lowest blast radius. Verify the non-root user can read both files; restart now loads (files present) so it mints nothing new. Does not fix the token store.
R2	Re-pin the controller to ced34036…: set controller.yaml.local_api.fingerprint (and bootstrap.json) to AGENT_FP, restart controller	Keeps the freshly-generated leaf; controller trusts the current agent	Edits in-guest config; keeps the "current cert" as canonical, which is the cleaner long-term state if the migration intends a new leaf. Does not fix the token store.
R3	Re-bootstrap / re-provision the controller against the live agent	Re-mints token and re-pins fingerprint from a freshly emitted bootstrap.json	Heaviest; resolves the token-store blocker too — but only correct if a fresh bootstrap.json carrying AGENT_FP is generated (the existing one still holds the stale 60b5974d… and must be regenerated, not reused).

Token store (applies to R1/R2): also re-seed guest 9201's bearer token into /var/lib/felhom-agent/local-tokens.log (re-provision token mint, or restore the old token store from the same aside-var-lib set-aside), else expect a 401 after the pin is fixed.

Follow-up CONFIRMED (2026-06-29) — aside-var-lib holds the matching token store → R1 is one copy

The set-aside does contain the pre-migration token store, and it matches guest 9201's current controller token, so R1 fixes the pin and the token blocker in a single restore:

/root/agent-backup-bundle-test/aside-var-lib/local-tokens.log
  656 bytes, 8 records, mtime 2026-06-26 19:56 (pre-migration)
  effective (last-write-wins) vmids: 9200, 9201
vs live /var/lib/felhom-agent/local-tokens.log : 0 bytes, mtime 2026-06-28 08:56 (reset at migration)

Verified on-host (token plaintext/hashes never left the host; only the verdict was emitted): the agent stores {"v":<vmid>,"h":hex(sha256(token))} JSONL (internal/localapi/tokenstore.go, last-write-wins per vmid). The SHA-256 of the controller's current controller.yaml local_api.token (the line under local_api:, not the unrelated git:/hub: tokens in the same file) is present in the old store under vmid 9201. So:

• old leaf fp 60b5974d… == PIN_FP (and SEED_FP) ✓
• old store's effective 9201 token hash == controller's current local_api.token hash ✓

Therefore the cleanest remedy is R1 extended to three files: restore local-api.crt, local-api.key, and local-tokens.log from …/aside-var-lib/ → /var/lib/felhom-agent/, chown felhom-agent:felhom-agent (crt 0644, key/tokens 0600), restart the agent. The agent then loads (not regenerates) the old leaf and re-indexes the old token store; the controller's existing pin and token both validate, with zero in-guest change. (Caveat: this reinstates the pre-migration leaf 60b5974d… rather than adopting the new ced34036… — fine for restoring service; if the operator wants the new leaf to be canonical, that's R2/R3 instead, both of which still need the token re-seed handled separately.)

Forward-looking: the root→non-root migration runbook should carry the existing local-api.{crt,key} and local-tokens.log into the new data dir (or treat their loss as a re-bootstrap trigger). Regenerating the leaf silently invalidates every issued pin — exactly this incident.

---

8. Confidence + gaps

• Pin root cause: very high. Disk == wire == boot-log all agree; mtime/notBefore pin the regeneration to the migration minute; the matching pre-migration backup leaf proves the old value.
• Token-store secondary blocker: high (inferred, not yet triggered). The 0-byte store at the migration mtime is strong evidence; not proven live because the TLS pin fails before any token is evaluated, and probing it would require an authenticated call (out of scope for read-only). Operator should expect it and plan the token re-seed alongside whichever pin remedy is chosen.
• Resolved (was "not assessed"): …/aside-var-lib/ does hold the matching pre-migration local-tokens.log, and its effective 9201 token hash equals the controller's current local_api.token hash — so R1 restored as three files (crt+key+tokens) fixes pin and token in one copy, zero in-guest change. See §7 follow-up.

No secrets recorded: the per-guest bearer token is referenced as "stored out-of-band"; the leaf fingerprint is not a secret.

---

9. Outcome — R1 executed 2026-06-29 (supervised), service RESTORED

Ran the R1 RUNBOOK (restore the pre-migration leaf + token store from /root/agent-backup-bundle-test/aside-var-lib/ → /var/lib/felhom-agent/, chown felhom-agent, modes 644/600/600, agent restart). Current post-migration state was first backed up to /root/agent-rollback-20260629-150137/ (rollback source, retained).

Gates — all passed:

• Gate 1 (loaded, not regenerated): boot log local-api leaf ready fingerprint_sha256=60b5974d… at 15:02:10 — the agent loaded the restored leaf (ownership/mode correct; no third cert minted).
• Gate 2 (wire): single felhom-agent listener on 192.168.0.162:8443; on-wire leaf fp 60b5974d… == controller PIN_FP.
• Gate 3 (real pinned+auth path): authenticated GET /disks with the controller's local_api.token → HTTP 200, ok=true, 5 disks; a bogus token → 401 (auth genuinely enforced). Token/hashes never left the host.
• Controller corroboration: the first quiesce cycle after the restart (15:05:33 CEST) succeeded end-to-end — "backup due — quiescing 5 stack(s) … backup job … started … snapshotted — resuming app early … unquiescing: restarting 5 stack(s)". No TLS pin mismatch / 401 lines after 15:02. The disk UI path and the quiesce app-consistent-backup loop are both restored.

Restored values: agent now serves leaf 60b5974d… (matches the in-guest pin and bootstrap.json seed) and re-indexed the pre-migration token store (effective 9201 token = the controller's current local_api.token). Zero in-guest change was needed.

Retained (do not delete yet): /root/agent-backup-bundle-test/aside-var-lib/ (only copy of the pinning material until the future-proofing fix lands) and /root/agent-rollback-20260629-150137/ (R1 rollback point).

Still open (carried forward, NOT addressed by R1):

• The root→non-root agent migration regenerates the local-API leaf (and truncates the token store) because it does not carry local-api.{crt,key} + local-tokens.log into the new felhom-agent-owned data dir. Until the migration/Day-0 path preserves these (or treats their loss as a re-bootstrap trigger), any re-migration/host-reprovision will reintroduce this exact mismatch. This restore reinstated the old leaf; it did not change the migration behavior.