felhom.eu/documentation/audits/SPIKE-agent-pin-mismatch-2026-06-29.md

# SPIKE — controller↔agent TLS leaf-pin mismatch (disk ops + quiesce backup dead)

- **Date:** 2026-06-29
- **Class:** Diagnostic / read-only. **No code, config, cert, service, or version change was made.**
- **Host:** felhom-pve (`demo-felhom`, 192.168.0.162), controller in **LXC 9201**.
- **Verdict:** **Hypothesis #1 — the agent local-API leaf was regenerated** during the root→non-root
  agent migration on **2026-06-28 ~08:56 CEST**. The controller's pin (and `bootstrap.json` seed)
  still reference the **pre-migration** leaf. A secondary blocker (empty token store) is latent
  behind the pin and will surface the moment the pin is fixed.
- **RESOLVED 2026-06-29 via R1** (RUNBOOK, supervised) — service restored; see the closing
  **§9 Outcome** at the bottom.

---

## 1. Live running versions

| Component | Version / image | Notes |
|---|---|---|
| Host agent | **felhom-agent 0.43.0** | unit `felhom-agent.service`, runs **non-root** as `felhom-agent`, `--config /etc/felhom-agent/agent.json`, `ActiveEnterTimestamp = Sun 2026-06-28 08:59:06 CEST` |
| In-guest controller | **felhom-controller:0.89.0** | LXC 9201, `Up ~1h (healthy)` at probe time |

---

## 2. The three fingerprints

| Label | Value | Source |
|---|---|---|
| **AGENT_FP** (served now) | `ced34036440639f48f4b96fb1802e79b2c1b3102eb763d3ba97aea2da1944f9c` | on-disk `local-api.crt` **==** live wire (`s_client`) **==** boot log — all three agree |
| **PIN_FP** (controller pins) | `60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245` | `controller.yaml → local_api.fingerprint` (data volume in 9201) |
| **SEED_FP** (bootstrap seed) | `60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245` | `/etc/felhom-bootstrap/bootstrap.json → local_api.fingerprint` |

`PIN_FP == SEED_FP == 60b5974d…` ≠ `AGENT_FP == ced34036…`. The controller is talking to the
**right endpoint** (both sides `192.168.0.162:8443`) and pinning a **consistent but now-obsolete**
leaf.

---

## 3. Raw evidence

### A — what the agent serves (host felhom-pve)

**A1 — socket owner on :8443** (single process, not a stale/second daemon → hypothesis #3 ruled out):
```
LISTEN 192.168.0.162:8443  users:(("felhom-agent",pid=3550072,fd=18))
```

**A2 — unit + boot-log fingerprint history.** The leaf fingerprint logged at every boot is stable
`60b5974d…` from Jun 24 through Jun 27, then **flips to `ced34036…` on Jun 28 08:56:37**, same
cert path:
```
Jun 26 19:54:50  local-api leaf ready fingerprint_sha256=60b5974d…  cert=/var/lib/felhom-agent/local-api.crt
Jun 27 …         (controller swaps to 0.86.0 / 0.87.0; leaf still 60b5974d)
Jun 28 08:56:37  local-api leaf ready fingerprint_sha256=ced34036…  cert=/var/lib/felhom-agent/local-api.crt   ← REGENERATED
Jun 28 08:59:06  local-api leaf ready fingerprint_sha256=ced34036…  cert=/var/lib/felhom-agent/local-api.crt   ← current daemon (pid 3550072)
```
`ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`,
`FragmentPath=/etc/systemd/system/felhom-agent.service`, `ActiveEnterTimestamp=2026-06-28 08:59:06`.

**A3 — on-disk leaf** (fresh, owned by the new non-root user, mtime = the switchover):
```
-rw-r--r-- 1 felhom-agent felhom-agent 591 Jun 28 08:56 local-api.crt
-rw------- 1 felhom-agent felhom-agent 227 Jun 28 08:56 local-api.key
-rw------- 1 felhom-agent felhom-agent   0 Jun 28 08:56 local-tokens.log      ← 0 bytes (see §6)
openssl … DER | sha256sum = ced34036440639f48f4b96fb1802e79b2c1b3102eb763d3ba97aea2da1944f9c
serial=85131255A9AE3FD902A5DB12A15D0459  notBefore=Jun 28 05:56:37 2026 GMT  notAfter=Jun 25 05:56:37 2036 GMT
```

**A4 — on-the-wire leaf** (live `s_client` to 192.168.0.162:8443):
```
ced34036440639f48f4b96fb1802e79b2c1b3102eb763d3ba97aea2da1944f9c
```
→ **wire == disk == boot-log.** No stale/second process; one honest agent serving one fresh cert.

**A5 — agent config `local_api`** (paths NOT overridden to anything unexpected; defaults):
```json
"local_api": { "enable": true, "listen_addr": "192.168.0.162:8443",
  "cert_file": "/var/lib/felhom-agent/local-api.crt",
  "key_file":  "/var/lib/felhom-agent/local-api.key",
  "token_store": "/var/lib/felhom-agent/local-tokens.log" }
```

**Surviving pre-migration leaf (the restore candidate).** A set-aside copy of the old root-era
`/var/lib` exists from the bundle migration test, and it **matches the controller pin exactly**:
```
/root/agent-backup-bundle-test/aside-var-lib/local-api.crt
  DER|sha256 = 60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245   ← == PIN_FP == SEED_FP
  notBefore=Jun 10 06:52:37 2026 GMT   serial=8FFA15857DABD903BB53742E6EA2CE46
  (local-api.key present alongside)
```

### B — what the controller pins (LXC 9201)

**B1 — `controller.yaml` `local_api`** (`/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml`):
```yaml
local_api:
    endpoint: 192.168.0.162:8443
    fingerprint: 60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245
    token: <redacted — stored out-of-band>
```

**B2 — `bootstrap.json` seed** (`/etc/felhom-bootstrap/bootstrap.json`):
```json
"local_api": { "endpoint": "192.168.0.162:8443",
  "fingerprint": "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245" }
```

**B3 — controller error log** (every 5 min, two distinct call sites):
```
[quiesce] cycle error: check due: agentapi: GET /backup/due: … TLS pin mismatch: agent leaf SHA-256 does not match the bootstrap fingerprint
[web] disk list via agent failed: agentapi: GET /disks: … TLS pin mismatch: agent leaf SHA-256 does not match the bootstrap fingerprint
```

### C — endpoint sanity
`controller.yaml.endpoint` = `bootstrap.json.endpoint` = `agent listen_addr` = `192.168.0.162:8443`.
The controller pins the right host; only the leaf value is stale. Endpoint mismatch ruled out.

---

## 4. Cert-regeneration ↔ agent-restart correlation

The fresh leaf's `notBefore` (Jun 28 05:56:37 GMT = **08:56:37 CEST**, with the code's −1h skew
backdate) and the cert/key mtimes (Jun 28 08:56) coincide to the second with the first
post-migration `leaf ready` log line. The agent then settled into its current daemon at 08:59:06.
This is **not** an expiry event (10-year validity) — it is `EnsureLeaf` taking the *generate* branch
because the non-root agent's data dir (`/var/lib/felhom-agent`, now owned `felhom-agent:felhom-agent`)
contained **no** cert/key at first boot under the new user. The old pair was preserved only in the
`/root/agent-backup-bundle-test/aside-var-lib/` set-aside, so `EnsureLeaf` minted a new self-signed
leaf and persisted it — changing the served fingerprint out from under every already-issued pin.

This is a direct side-effect of the **BUNDLE Day-0 / root→non-root agent migration on 2026-06-28**.

---

## 5. Listening-socket owner

Single process: `felhom-agent` pid **3550072** (the running daemon), bound to `192.168.0.162:8443`.
No second/stale listener. Hypothesis #3 (stale process serving an old cert) is **disproven** — and
note the *direction* is the opposite of #3 anyway: the live process serves the **new** cert, while
the **old** value is the one frozen in the controller's pin.

---

## 6. Matched matrix row + conclusion

Matrix row **#1 — agent leaf regenerated** (files were absent at the new non-root path; not a path
typo, not an expiry):

> A3 == A2 == fresh mtime (cert regenerated 2026-06-28 ~08:56, the agent redeploy/non-root switch);
> `PIN_FP` ≠ `AGENT_FP`; a surviving backup cert matching `PIN_FP` **does** exist
> (`…/aside-var-lib/`, fp `60b5974d…`).

**Root cause (high confidence):** the root→non-root agent migration on 2026-06-28 reinitialised
`/var/lib/felhom-agent` without carrying over the existing leaf, so the agent generated a new
self-signed local-API leaf (`ced34036…`). The in-guest controller (and its `bootstrap.json`) still
pin the pre-migration leaf (`60b5974d…`), so the pinned `agentapi` client fails closed on every
call. Because **all** disk/storage/backup traffic goes through that one pinned client, both the disk
UI **and** the quiesce app-consistent-backup loop are down.

**Note — this is NOT a stale controller re-merge (#2).** `PIN_FP == SEED_FP`, so re-running the
controller's bootstrap/`ensureLocalAPI` merge would re-apply the **same obsolete** `60b5974d…` and
not help. The divergence originated on the agent side, not from a stale seed re-merge.

### Secondary blocker (latent behind the pin) — empty token store
`/var/lib/felhom-agent/local-tokens.log` is **0 bytes**, mtime **2026-06-28 08:56** (reset at the
same migration). The controller's per-guest bearer token was minted into the *pre-migration* store.
The TLS pin fails first (handshake), so this is currently masked — but once the pin is corrected the
agent will have **no token entry** for guest 9201 and authenticated calls will **401**. This matches
the known "local-API 401 until token re-seed after agent restart / stale token map" pattern. **Any
remedy below must be paired with re-seeding the guest token** (re-provision token mint, or restore
the old token store), or the symptom merely changes from `TLS pin mismatch` to `401`.

---

## 7. Remedy options for the operator (NOT executed — operator's choice)

| # | Remedy | Effect | Trade-offs / cautions |
|---|---|---|---|
| **R1** | **Restore the pre-migration leaf**: copy `…/aside-var-lib/local-api.{crt,key}` → `/var/lib/felhom-agent/`, `chown felhom-agent:felhom-agent`, `0644/0600`, restart agent | Agent serves `60b5974d…` again → matches existing `PIN_FP`+`SEED_FP`; **zero controller-side change** | Lowest blast radius. Verify the **non-root** user can read both files; restart now *loads* (files present) so it mints nothing new. Does **not** fix the token store. |
| **R2** | **Re-pin the controller** to `ced34036…`: set `controller.yaml.local_api.fingerprint` (and `bootstrap.json`) to AGENT_FP, restart controller | Keeps the freshly-generated leaf; controller trusts the current agent | Edits in-guest config; keeps the "current cert" as canonical, which is the cleaner long-term state if the migration intends a new leaf. Does **not** fix the token store. |
| **R3** | **Re-bootstrap / re-provision the controller** against the live agent | Re-mints token **and** re-pins fingerprint from a freshly emitted `bootstrap.json` | Heaviest; resolves the token-store blocker too — but only correct if a fresh `bootstrap.json` carrying AGENT_FP is generated (the *existing* one still holds the stale `60b5974d…` and must be regenerated, not reused). |

**Token store (applies to R1/R2):** also re-seed guest 9201's bearer token into
`/var/lib/felhom-agent/local-tokens.log` (re-provision token mint, or restore the old token store
from the same `aside-var-lib` set-aside), else expect a `401` after the pin is fixed.

### Follow-up CONFIRMED (2026-06-29) — `aside-var-lib` holds the matching token store → R1 is one copy

The set-aside **does** contain the pre-migration token store, and it matches guest 9201's
*current* controller token, so **R1 fixes the pin and the token blocker in a single restore**:

```
/root/agent-backup-bundle-test/aside-var-lib/local-tokens.log
  656 bytes, 8 records, mtime 2026-06-26 19:56 (pre-migration)
  effective (last-write-wins) vmids: 9200, 9201
vs live /var/lib/felhom-agent/local-tokens.log : 0 bytes, mtime 2026-06-28 08:56 (reset at migration)
```

Verified on-host (token plaintext/hashes never left the host; only the verdict was emitted): the
agent stores `{"v":<vmid>,"h":hex(sha256(token))}` JSONL (`internal/localapi/tokenstore.go`,
last-write-wins per vmid). The SHA-256 of the controller's **current** `controller.yaml`
`local_api.token` (the line under `local_api:`, **not** the unrelated `git:`/`hub:` tokens in the
same file) is **present in the old store under vmid 9201**. So:

- old leaf fp `60b5974d…` **==** `PIN_FP` (and `SEED_FP`) ✓
- old store's effective 9201 token hash **==** controller's current `local_api.token` hash ✓

**Therefore the cleanest remedy is R1 extended to three files:** restore
`local-api.crt`, `local-api.key`, **and** `local-tokens.log` from `…/aside-var-lib/` →
`/var/lib/felhom-agent/`, `chown felhom-agent:felhom-agent` (crt `0644`, key/tokens `0600`),
restart the agent. The agent then *loads* (not regenerates) the old leaf and re-indexes the old
token store; the controller's existing pin **and** token both validate, **with zero in-guest
change**. (Caveat: this reinstates the pre-migration leaf `60b5974d…` rather than adopting the
new `ced34036…` — fine for restoring service; if the operator wants the new leaf to be canonical,
that's R2/R3 instead, both of which still need the token re-seed handled separately.)

**Forward-looking:** the root→non-root migration runbook should **carry the existing
`local-api.{crt,key}` and `local-tokens.log` into the new data dir** (or treat their loss as a
re-bootstrap trigger). Regenerating the leaf silently invalidates every issued pin — exactly this
incident.

---

## 8. Confidence + gaps

- **Pin root cause: very high.** Disk == wire == boot-log all agree; mtime/`notBefore` pin the
  regeneration to the migration minute; the matching pre-migration backup leaf proves the old value.
- **Token-store secondary blocker: high (inferred, not yet triggered).** The 0-byte store at the
  migration mtime is strong evidence; not *proven* live because the TLS pin fails before any token is
  evaluated, and probing it would require an authenticated call (out of scope for read-only). Operator
  should expect it and plan the token re-seed alongside whichever pin remedy is chosen.
- **Resolved (was "not assessed"):** `…/aside-var-lib/` **does** hold the matching pre-migration
  `local-tokens.log`, and its effective 9201 token hash equals the controller's current
  `local_api.token` hash — so R1 restored as three files (crt+key+tokens) fixes pin **and** token in
  one copy, zero in-guest change. See §7 follow-up.

*No secrets recorded: the per-guest bearer token is referenced as "stored out-of-band"; the leaf
fingerprint is not a secret.*

---

## 9. Outcome — R1 executed 2026-06-29 (supervised), service RESTORED

Ran the R1 RUNBOOK (restore the pre-migration leaf **+** token store from
`/root/agent-backup-bundle-test/aside-var-lib/` → `/var/lib/felhom-agent/`, chown
`felhom-agent`, modes 644/600/600, agent restart). Current post-migration state was first backed
up to `/root/agent-rollback-20260629-150137/` (rollback source, retained).

**Gates — all passed:**
- **Gate 1 (loaded, not regenerated):** boot log `local-api leaf ready fingerprint_sha256=60b5974d…`
  at 15:02:10 — the agent **loaded** the restored leaf (ownership/mode correct; no third cert minted).
- **Gate 2 (wire):** single `felhom-agent` listener on `192.168.0.162:8443`; on-wire leaf fp
  `60b5974d…` == controller `PIN_FP`.
- **Gate 3 (real pinned+auth path):** authenticated `GET /disks` with the controller's
  `local_api.token` → **HTTP 200, ok=true, 5 disks**; a bogus token → **401** (auth genuinely
  enforced). Token/hashes never left the host.
- **Controller corroboration:** the first quiesce cycle after the restart (15:05:33 CEST)
  succeeded end-to-end — *"backup due — quiescing 5 stack(s) … backup job … started … snapshotted
  — resuming app early … unquiescing: restarting 5 stack(s)"*. **No** `TLS pin mismatch` / 401
  lines after 15:02. The disk UI path and the quiesce app-consistent-backup loop are both restored.

**Restored values:** agent now serves leaf `60b5974d…` (matches the in-guest pin and
`bootstrap.json` seed) and re-indexed the pre-migration token store (effective 9201 token = the
controller's current `local_api.token`). **Zero in-guest change** was needed.

**Retained (do not delete yet):** `/root/agent-backup-bundle-test/aside-var-lib/` (only copy of the
pinning material until the future-proofing fix lands) and `/root/agent-rollback-20260629-150137/`
(R1 rollback point).

**Still open (carried forward, NOT addressed by R1):**
- The root→non-root agent migration **regenerates the local-API leaf** (and truncates the token
  store) because it does not carry `local-api.{crt,key}` + `local-tokens.log` into the new
  `felhom-agent`-owned data dir. Until the migration/Day-0 path preserves these (or treats their
  loss as a re-bootstrap trigger), any re-migration/host-reprovision will reintroduce this exact
  mismatch. This restore reinstated the **old** leaf; it did not change the migration behavior.