# SPIKE — dedicated `felhom-sshd` instance (own port/config/keys, customer-host coexistence) — 2026-07-05

> **STATUS: COMPLETE — verdict below, with one CAUGHT-AND-FIXED incident (§8) that is itself the
> single most important finding.** Validates a dedicated second sshd instance as Felhom's OOB entry
> point on the demo host `felhom-pve` (node `demo-felhom`, PVE 9.2.2, OpenSSH 10.0p2). All spike
> artifacts removed at cleanup (§13, re-verified to the P0 baseline). No production code shipped.

**Class:** SPIKE (empirical; no product code). **Repos:** felhom.eu (this doc only); felhom-agent +
felhom.eu/scripts read-only for grounding. **Host:** `felhom-pve` (`192.168.0.162`). The offsite
`wg-felhom` tunnel + PBS path were NOT touched (OOB transport was proven separately in
`SPIKE-oob-wg-operator-peer-2026-07-05.md`).

**Verdict (one line):** **GO — a dedicated `felhom-sshd` on a claimed non-22 port, with its own
config/host-key/AuthorizedKeysFile/unit, coexists cleanly with the stock (customer) sshd on :22
(independence proven on all four axes), claims its port robustly (skip/idempotent/loud-exhaustion),
scopes to the tunnel by nft, and uninstalls to a byte-clean baseline — PROVIDED the unit NEVER
declares `RuntimeDirectory=sshd` (which removes the SHARED `/run/sshd` privsep dir on stop and takes
the stock sshd down with it; this spike hit that live, §8, and it is the #1 TASK constraint).**

---

## 0. What the demo box could and could not represent

The box is **Felhom-owned** and single-NAT; the stock `ssh.service` on :22 stood in for "the
customer's existing sshd". Faithfully represented: two independent sshd instances on one host, port
contention, nft scoping, the systemd/privsep mechanics, uninstall removability. NOT represented: a
**real customer's** sshd config (they may run `AllowUsers`, non-default ports, `Match` blocks,
`fail2ban`/`sshguard`, a hardened `sshd_config` that our drop-in model doesn't inherit — we ship a
SEPARATE config so we don't inherit theirs, which is the point), their actual **port map** (our
candidate list must still probe-and-claim against whatever they run), and true **CGNAT**. The
privsep-dir incident (§8) is host-OS-generic (Debian/PVE), so it applies to a customer host
identically.

---

## 1. P0 — ground truth (no changes) — GATE PASSED

```
stock sshd unit  : ssh.service  (Debian; sshd.service/sshd@.service are ALIASES/indirect)
                   MainPID 922, FragmentPath /usr/lib/systemd/system/ssh.service
effective config : port 22 ; ListenAddress 0.0.0.0:22 + [::]:22 ; UsePAM yes ;
                   PermitRootLogin yes ; PasswordAuthentication yes  (Proxmox defaults) ;
                   AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 ;
                   HostKey rsa+ecdsa+ed25519 ; OpenSSH_10.0p2 Debian-7+deb13u4
instance template: /lib/systemd/system/ssh@.service ABSENT at that path; BUT `sshd@.service`
                   (indirect) + `ssh.socket` (disabled) exist — socket-activation route is available.
listeners (ss -tlnp): 22(sshd) 111(rpcbind) 85(pvedaemon,localhost) 25(postfix,localhost)
                   53(dnsmasq) 8443(felhom-agent) 3128(spiceproxy) 8006(pveproxy)
                   → candidate ports 8822/2222/8022/62222 ALL FREE.
pve-firewall     : disabled ;  nft list tables: EMPTY (baseline)
PVE health       : pveproxy + pvedaemon active
```
**Gate:** stock sshd identified (`ssh.service`/PID 922/:22) and never touched thereafter.

---

## 2. P2 — port probe-then-claim under contention — GO (all four behaviours)

Claim routine (`/usr/local/sbin/felhom-sshd-spike-claim.sh`, throwaway) — candidate list
**`8822 2222 8022 62222`**; `is_free()` = nothing in `ss -Htln sport=:N` **AND** a real `python3`
bind succeeds; idempotent re-pick of the persisted port if still free; persists to
`/etc/felhom-sshd-spike/port`.

| Scenario | Result |
|---|---|
| clean | → **8822**, persisted |
| **contention** (a listener bound on 8822, persisted cleared) | → **skips to 2222** |
| **idempotent** (re-run while 8822 still busy) | → **stays 2222** (no thrash) |
| **exhaustion** (all four occupied) | → **rc=3, LOUD stderr** "refusing to fall back to :22 or a random port" — **never silently grabs :22** |
| recovery (all free) | → 8822 |

The exhaustion-must-fail-loud property (a silent :22 grab would be the worst bug) is proven, not
assumed.

**Algorithm (copy-paste, the shape the TASK's Go/agent version should mirror):** ordered candidate
list → for each, `ss -Htln "sport = :$p"` empty **and** an actual `bind()` succeeds → first free
wins → persist + emit; if none free → non-zero exit + explicit error, **no fallback**. Idempotency
gate reads the persisted port first and keeps it if still free.

---

## 3. P1 — dedicated instance, minimal — GO

Working `sshd_config` (`/etc/felhom-sshd-spike/sshd_config`), copy-paste ready:
```
Port <claimed>            # from P2
ListenAddress 0.0.0.0
ListenAddress ::
HostKey /etc/felhom-sshd-spike/ssh_host_ed25519_key   # DEDICATED key (ssh-keygen -t ed25519)
PidFile /run/felhom-sshd-spike.pid
AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u   # OUTSIDE ~/.ssh — see P6
PasswordAuthentication no
PermitRootLogin prohibit-password        # key-only
PubkeyAuthentication yes
UsePAM yes                               # works on PVE with no extra PAM config (service name "sshd")
AllowUsers root felhom-op-spike          # the operator identity/ies only
X11Forwarding no
Subsystem sftp internal-sftp
```
The `sshd -t -f <conf>` validation gate passed. **GO:** `ss -tlnp` showed felhom-sshd on 8822
(PID 4095087) AND stock sshd on :22 (PID 922, UNCHANGED), both LISTEN; `Type=notify` sd_notify'd
cleanly (unit `active`, not just `activating`).

**The `.service` — SAFE FINAL FORM (post-§8-incident):**
```
[Unit]
Description=Felhom dedicated OOB sshd
After=network.target
[Service]
Type=notify
# NEVER "RuntimeDirectory=sshd" — systemd removes /run/sshd on stop, clobbering the SHARED
# privsep dir and breaking the stock sshd (§8). Ensure the dir exists idempotently instead:
ExecStartPre=-/usr/bin/mkdir -p /run/sshd
ExecStartPre=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config
ExecStart=/usr/sbin/sshd -D -f /etc/felhom-sshd-spike/sshd_config
ExecReload=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
```

---

## 4. P3 — customer-owned-Proxmox coexistence (the headline) — GO (four axes, evidenced)

Stock :22 as the customer-sshd stand-in, felhom-sshd on 8822 simultaneously:

| Axis | Evidence |
|---|---|
| **Both accept independently** | felhom-sshd key login (localhost:8822 → `whoami=root`); LAN :22 kept working (CC's own line) |
| **Distinct host keys** | stock ed25519 `SHA256:8Tea…` vs felhom `SHA256:EnC4…` — a client pinning one is never confused by the other (known_hosts hygiene for the operator) |
| **Reload stock → felhom unaffected** | `systemctl reload ssh` → felhom PID **4095087 unchanged**, still listening |
| **Restart felhom → stock unaffected** | `systemctl restart felhom-sshd-spike` → stock PID **922 unchanged**; pveproxy+pvedaemon stayed active; no port collision with 8006/8007/85/111/3128 |

PVE (pveproxy :8006 + pvedaemon) undisturbed throughout; guest 9201 healthy.

---

## 5. P4 — nft scoping to the tunnel — GO (LAN-block + :22-untouched evidenced; tunnel-accept by structure)

Named table `felhom_sshd_spike`, retargeted from the prior spike's proven belt to the felhom port:
```
table inet felhom_sshd_spike {
  chain input { type filter hook input priority -5; policy accept;
    iifname "wg-felhom" tcp dport 8822 ip saddr <operator/32> counter accept   # tunnel, operator only
    iifname "wg-felhom" tcp dport 8822 counter drop                            # tunnel, other src
    tcp dport 8822 iifname != "wg-felhom" counter drop                         # OFF-tunnel: unreachable
  } }
```
- **LAN → 8822: BLOCKED** — a LAN `ssh -p 8822` from CC `Connection timed out`; the off-tunnel drop
  counter incremented (3 pkts). Localhost → 8822 also blocked (lo ≠ wg-felhom), confirming
  tunnel-only.
- **Stock :22 UNTOUCHED** — `nft list ruleset | grep "dport 22"` is EMPTY; LAN :22 kept working
  (the safety line), by construction (the rules match `dport <felhom-port>` only, never 22).
- **Tunnel-accept path:** not re-exercised live here (the safety rail forbade re-establishing the
  operator peer on `wg-felhom` in this spike); it is the **byte-identical belt** to
  `SPIKE-oob-wg-operator-peer-2026-07-05.md` §5, which WAS live-tested against a real operator peer
  (operator→box SSH at ~24 ms, dummy dropped). Structure + prior live proof = confident.

---

## 6. P5 — robustness — GO (with a sharp restart-vs-reload finding)

- **SAFE config-change path — `reload` survives a bad config:** appended a bogus directive →
  `systemctl reload felhom-sshd-spike` **refused** (the `ExecReload=sshd -t` gate fails) → the
  running instance **SURVIVED** (PID unchanged, still listening on 8822 with the last-good config).
  This is the path the TASK must use for config changes.
- **UNSAFE path — `restart` is NOT atomic:** `systemctl restart` with a bad config takes the daemon
  **DOWN** (systemd stops the old instance, then `ExecStartPre=sshd -t` fails so nothing comes back;
  unit ends `failed`/`activating`, PID 0). A bad config therefore must be caught BEFORE a restart
  (validate, then reload — never restart-on-change).
- **Blast-containment (post-fix):** even the unsafe `restart+badconfig` left stock :22 (PID 922) and
  `/run/sshd` INTACT and LAN :22 working — because the SAFE unit (§3) has no `RuntimeDirectory=sshd`.
  (Before the fix, this exact sequence removed `/run/sshd` and took :22 down — §8.)
- **Boot persistence:** `enabled`; host-key fingerprint **STABLE** (`SHA256:EnC4…`) across restart.
- **Late-network independence:** binds `0.0.0.0` (not a late-coming IP), `After=network.target`
  only, **no `wg-felhom` dependency** — comes up regardless of tunnel state.
- **Start-limit gotcha (D1 echo):** repeated failed starts hit systemd's StartLimit → the unit
  refuses further starts until `systemctl reset-failed`. The TASK's health/repair logic must
  `reset-failed` before a deliberate restart (same lesson as the agent self-update spike).

---

## 7. P6 — operator login identity — GO + the clean separation mechanism

Throwaway `felhom-op-spike` (normal login user, `--create-home`, `/bin/bash`) with a scoped sudoers
(`/etc/sudoers.d/felhom-op-spike`: `mkdir -p /run/sshd`, `systemctl restart felhom-sshd-spike`,
`systemctl start felhom-agent` — the repair verbs):
- **Key login via felhom-sshd (8822) works** → `whoami=felhom-op-spike`, scoped `sudo -n` → `SUDO_OK`.
- **NOT reachable via stock :22** → `Permission denied (publickey)`. **Why it's clean:** the operator
  key lives ONLY in felhom-sshd's `AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u`,
  NOT in `~felhom-op-spike/.ssh/authorized_keys` (which doesn't exist). So the customer's sshd —
  which reads `~/.ssh/authorized_keys` — never honours the operator key, **without us having to
  touch the customer's `AllowUsers`/config at all.** This is the load-bearing separation primitive.
- `AllowUsers root felhom-op-spike` on felhom-sshd is the second gate (who may use 8822).

**Operator-identity trade (record-only, for the TASK to decide):**
- **`felhom-op` (dedicated, sudo-scoped)** — clean least-privilege, auditable, isolated to
  felhom-sshd via the AuthorizedKeysFile trick; needs a scoped sudoers for the repair verbs it will
  run (agent restart, wg repair, `mkdir /run/sshd`, etc.). Doesn't collide with the customer's users.
- **`root` (PVE-native)** — deep host repairs (pct, systemctl, storage) need root anyway; PermitRootLogin
  prohibit-password + the tunnel-only nft belt + the dedicated AuthorizedKeysFile still contain it;
  simplest for "the operator must be able to fix anything". Trade: broader blast radius if the
  operator key leaks (mitigated by the tunnel gate + key custody).
- Likely answer: **`felhom-op` with a tightly-scoped sudoers for the known repair actions, escalating
  to root only via those sudo verbs** — but that's a TASK decision, recorded here, not made.

---

## 8. THE INCIDENT — `RuntimeDirectory=sshd` clobbered the shared `/run/sshd` (caught, fixed, root-caused)

**What happened.** The P1 unit was first written with `RuntimeDirectory=sshd`. During the P5
bad-config test, `systemctl restart felhom-sshd-spike` failed at `ExecStartPre` (as designed) and the
unit went `failed`. systemd, cleaning up that unit's runtime directory, **removed `/run/sshd`** — the
compiled-in OpenSSH **privilege-separation directory**, which the **STOCK** sshd on :22 also depends
on. The stock listener (PID 922) stayed up and accepted TCP, but every new session **reset right
after `SSH2_MSG_KEXINIT`** — the textbook signature of a child that cannot initialise privsep. **CC's
LAN SSH — the safety line — went down.**

**Why the "shared RuntimeDirectory is preserved" assumption was wrong.** Debian's stock `ssh.service`
also declares `RuntimeDirectory=sshd`; I expected systemd (v257) to keep `/run/sshd` alive while the
active stock unit referenced it. Empirically it did **not** — the failing unit's cleanup removed the
shared dir. (A full 135 s connection-drain ruled out `MaxStartups`; the cause was structural.)

**Recovery (no shell available; the honest bit).** `:22` was the only shell path and it was the thing
broken; the felhom-sshd port was down (start-limited); the PVE API token cached on the build server
was **stale (401)**; no root password is documented (correctly). The customer **data plane was
unaffected** (`felhom.demo-felhom.eu` 200, pveproxy 200, agent alive) — a *management* lockout only.
Recovery was via the **PVE web console** (operator authenticated the session — CC is prohibited from
entering passwords — then CC drove the node xterm.js shell): `mkdir -p /run/sshd && chmod 0755
/run/sshd` restored stock :22 **instantly, without even restarting the stock sshd** (PID stayed 922).
Confirmed live: `ls /run/sshd` → "No such file or directory" pre-fix; present post-fix.

**The fix in the design (mandatory TASK constraint #1).** A second sshd instance must **NEVER**
declare `RuntimeDirectory=sshd`. Ensure `/run/sshd` exists idempotently instead — the spike used
`ExecStartPre=-/usr/bin/mkdir -p /run/sshd`; the production-clean equivalent is a
`tmpfiles.d` entry (`d /run/sshd 0755 root root -`) so the dir is boot-persistent and owned by
neither sshd unit's lifecycle. With the fixed unit, even `restart`+badconfig (§6) left `/run/sshd`
and :22 intact.

**Secondary lessons from the incident:** (a) the build-server PVE token copy is **stale** — CC's only
non-SSH lever into the host is unusable; a working break-glass path (a valid console credential, or a
second admin key on a different port) should exist BEFORE this feature ships. (b) reset-after-KEXINIT
with the listener up = missing/again-broken `/run/sshd` — a fast diagnostic for the runbook.

---

## 9. Findings that must shape the TASK

1. **NEVER `RuntimeDirectory=sshd` in the felhom-sshd unit (§8).** It removes the shared `/run/sshd`
   on stop/failure and takes the customer's sshd down. Ship a `tmpfiles.d` `/run/sshd` entry (or an
   idempotent `ExecStartPre=mkdir`) instead. This is the #1 safety constraint and it is not obvious.
2. **Config changes go via `reload` (HUP) behind an `ExecReload=sshd -t` gate — never `restart`
   (§6).** Reload keeps the running instance on the last-good config if the new one is bad; restart
   is not atomic and downs the daemon on a failed pre-check. Validate, then reload.
3. **Identity isolation is the `AuthorizedKeysFile` location, not the customer's config (§7).**
   Putting the operator key under `/etc/felhom-sshd/authorized_keys/%u` (outside `~/.ssh`) means the
   customer's sshd never honours it and we never touch their config. This is the clean coexistence
   primitive — carry it into the design.
4. **Port claim: probe-and-claim over an ordered candidate list, fail LOUD on exhaustion (§2).**
   Never silently fall back to :22 or a random port. Persist the chosen port; be idempotent. The
   shell PoC works; the TASK's agent-Go version should mirror the `ss`-empty **and** real-`bind()`
   test.
5. **Ownership: agent-managed, mirroring the `wg-felhom`/`dnsmasq` pattern.** felhom-sshd is ongoing
   reconciled desired-state (port may need re-claim, keys rotate, AllowUsers changes) — the agent's
   job, not the one-shot host-install. Grant a NARROW sudoers slice (install the config + host key to
   `/etc/felhom-sshd/*`, `sshd -t`, `systemctl {reload,restart,enable,disable} felhom-sshd`, the
   `tmpfiles`/`mkdir /run/sshd`). `sshd -t` is the pre-apply gate the way `felhom-peersync` validates
   before applying. Host-install can lay down the STATIC skeleton on a fresh box; the agent reconciles.
6. **Debian's `ssh@.service`/`ssh.socket` (socket-activation) is NOT a cleaner base than a bespoke
   unit here (P0).** A per-connection socket-activated instance still needs `/run/sshd` and gives no
   coexistence benefit; a dedicated always-on `felhom-sshd.service` with its own config is simpler
   and is what the spike validated. Recommend the bespoke unit.
7. **PAM: `UsePAM yes` works on PVE with zero extra config** (the second sshd reuses the `sshd` PAM
   service). No PVE-specific PAM work needed for key-only login. (If the TASK ever enables anything
   PAM-gated like 2FA, revisit — out of scope here.)
8. **Fresh-host vs customer-host is a CONFIG VALUE, not a fork.** The same felhom-sshd instance model
   works whether or not a stock sshd exists on :22 — on a fresh Felhom box it simply coexists with the
   Proxmox-default sshd; on a customer box it coexists with theirs. The only variable is the CLAIMED
   PORT (probe-and-claim handles both). No separate code path; the install just always claims a port
   and never assumes :22 is Felhom's.
9. **Break-glass prerequisite (§8 secondary).** Before this ships, ensure a working out-of-band root
   path independent of the felhom-sshd port itself (a valid PVE console credential and/or a distinct
   recovery key), and keep the build-server PVE token fresh — the stale token left CC with no
   non-SSH lever during the incident.

---

## 10. Cleanup assertion (§13) — VERIFIED to the P0 baseline

Removed exactly the spike set: `felhom-sshd-spike.service` (disable+rm+daemon-reload+reset-failed),
`/etc/felhom-sshd-spike/` (config + host key + authorized_keys), `/etc/sudoers.d/felhom-op-spike`,
the `felhom-op-spike` user + home (`userdel -r`), `/usr/local/sbin/felhom-sshd-spike-claim.sh`,
`/root/felhom-sshd-spike-client{,.pub}`, `/run/felhom-sshd-spike.pid`, `/run/felhom-claim-listener.pid`,
the `felhom_sshd_spike` nft table (both the box belt and the P4 table — removed). Re-verified:
stock `ssh.service` **active, PID 922** (same as P0 — never restarted; the /run/sshd fix alone
restored it), only `:22` listening, `sshd -T` port 22, `felhom-op-spike` gone, unit gone,
`/etc/felhom-sshd-spike` gone, `nft list tables` EMPTY (P0), PVE active, guest 9201 running.
**One intentional deviation from P0, permanent and correct:** `/run/sshd` exists (recreated during
§8 recovery) — it existed at P0 too; only its mtime differs. Leaving it is required.

**No repo/hub/agent/manifest production change** — this commit is docs-only.

---

## 11. Method-bar honored

`wg show dump` never run; `wg-felhom`/PBS untouched. Every mutation a separately-named
unit/dir/file/nft-table, inventoried as created; P7 removed exactly that set. Coexistence and
negative claims carry evidence (PIDs, `ss`, nft counters, session survival, fingerprints), not
"seemed fine". The one surprise (the `/run/sshd` clobber) is recorded as the headline finding, not
massaged. Per-probe verdicts: P0 GATE-PASS, P1 GO, P2 GO, P3 GO, P4 GO, P5 GO (+restart/reload
finding), P6 GO, §8 SURPRISE→FIXED.