felhom.eu/documentation/audits/SPIKE-felhom-sshd-2026-07-05.md

SPIKE — dedicated felhom-sshd instance (own port/config/keys, customer-host coexistence) — 2026-07-05

STATUS: COMPLETE — verdict below, with one CAUGHT-AND-FIXED incident (§8) that is itself the single most important finding. Validates a dedicated second sshd instance as Felhom's OOB entry point on the demo host felhom-pve (node demo-felhom, PVE 9.2.2, OpenSSH 10.0p2). All spike artifacts removed at cleanup (§13, re-verified to the P0 baseline). No production code shipped.

Class: SPIKE (empirical; no product code). Repos: felhom.eu (this doc only); felhom-agent + felhom.eu/scripts read-only for grounding. Host: felhom-pve (192.168.0.162). The offsite wg-felhom tunnel + PBS path were NOT touched (OOB transport was proven separately in SPIKE-oob-wg-operator-peer-2026-07-05.md).

Verdict (one line): GO — a dedicated felhom-sshd on a claimed non-22 port, with its own config/host-key/AuthorizedKeysFile/unit, coexists cleanly with the stock (customer) sshd on :22 (independence proven on all four axes), claims its port robustly (skip/idempotent/loud-exhaustion), scopes to the tunnel by nft, and uninstalls to a byte-clean baseline — PROVIDED the unit NEVER declares RuntimeDirectory=sshd (which removes the SHARED /run/sshd privsep dir on stop and takes the stock sshd down with it; this spike hit that live, §8, and it is the #1 TASK constraint).

---

0. What the demo box could and could not represent

The box is Felhom-owned and single-NAT; the stock ssh.service on :22 stood in for "the customer's existing sshd". Faithfully represented: two independent sshd instances on one host, port contention, nft scoping, the systemd/privsep mechanics, uninstall removability. NOT represented: a real customer's sshd config (they may run AllowUsers, non-default ports, Match blocks, fail2ban/sshguard, a hardened sshd_config that our drop-in model doesn't inherit — we ship a SEPARATE config so we don't inherit theirs, which is the point), their actual port map (our candidate list must still probe-and-claim against whatever they run), and true CGNAT. The privsep-dir incident (§8) is host-OS-generic (Debian/PVE), so it applies to a customer host identically.

---

1. P0 — ground truth (no changes) — GATE PASSED

stock sshd unit  : ssh.service  (Debian; sshd.service/sshd@.service are ALIASES/indirect)
                   MainPID 922, FragmentPath /usr/lib/systemd/system/ssh.service
effective config : port 22 ; ListenAddress 0.0.0.0:22 + [::]:22 ; UsePAM yes ;
                   PermitRootLogin yes ; PasswordAuthentication yes  (Proxmox defaults) ;
                   AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 ;
                   HostKey rsa+ecdsa+ed25519 ; OpenSSH_10.0p2 Debian-7+deb13u4
instance template: /lib/systemd/system/ssh@.service ABSENT at that path; BUT `sshd@.service`
                   (indirect) + `ssh.socket` (disabled) exist — socket-activation route is available.
listeners (ss -tlnp): 22(sshd) 111(rpcbind) 85(pvedaemon,localhost) 25(postfix,localhost)
                   53(dnsmasq) 8443(felhom-agent) 3128(spiceproxy) 8006(pveproxy)
                   → candidate ports 8822/2222/8022/62222 ALL FREE.
pve-firewall     : disabled ;  nft list tables: EMPTY (baseline)
PVE health       : pveproxy + pvedaemon active

Gate: stock sshd identified (ssh.service/PID 922/:22) and never touched thereafter.

---

2. P2 — port probe-then-claim under contention — GO (all four behaviours)

Claim routine (/usr/local/sbin/felhom-sshd-spike-claim.sh, throwaway) — candidate list 8822 2222 8022 62222; is_free() = nothing in ss -Htln sport=:N AND a real python3 bind succeeds; idempotent re-pick of the persisted port if still free; persists to /etc/felhom-sshd-spike/port.

Scenario	Result
clean	→ 8822, persisted
contention (a listener bound on 8822, persisted cleared)	→ skips to 2222
idempotent (re-run while 8822 still busy)	→ stays 2222 (no thrash)
exhaustion (all four occupied)	→ rc=3, LOUD stderr "refusing to fall back to :22 or a random port" — never silently grabs :22
recovery (all free)	→ 8822

The exhaustion-must-fail-loud property (a silent :22 grab would be the worst bug) is proven, not assumed.

Algorithm (copy-paste, the shape the TASK's Go/agent version should mirror): ordered candidate list → for each, ss -Htln "sport = :$p" empty and an actual bind() succeeds → first free wins → persist + emit; if none free → non-zero exit + explicit error, no fallback. Idempotency gate reads the persisted port first and keeps it if still free.

---

3. P1 — dedicated instance, minimal — GO

Working sshd_config (/etc/felhom-sshd-spike/sshd_config), copy-paste ready:

Port <claimed>            # from P2
ListenAddress 0.0.0.0
ListenAddress ::
HostKey /etc/felhom-sshd-spike/ssh_host_ed25519_key   # DEDICATED key (ssh-keygen -t ed25519)
PidFile /run/felhom-sshd-spike.pid
AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u   # OUTSIDE ~/.ssh — see P6
PasswordAuthentication no
PermitRootLogin prohibit-password        # key-only
PubkeyAuthentication yes
UsePAM yes                               # works on PVE with no extra PAM config (service name "sshd")
AllowUsers root felhom-op-spike          # the operator identity/ies only
X11Forwarding no
Subsystem sftp internal-sftp

The sshd -t -f <conf> validation gate passed. GO: ss -tlnp showed felhom-sshd on 8822 (PID 4095087) AND stock sshd on :22 (PID 922, UNCHANGED), both LISTEN; Type=notify sd_notify'd cleanly (unit active, not just activating).

The .service — SAFE FINAL FORM (post-§8-incident):

[Unit]
Description=Felhom dedicated OOB sshd
After=network.target
[Service]
Type=notify
# NEVER "RuntimeDirectory=sshd" — systemd removes /run/sshd on stop, clobbering the SHARED
# privsep dir and breaking the stock sshd (§8). Ensure the dir exists idempotently instead:
ExecStartPre=-/usr/bin/mkdir -p /run/sshd
ExecStartPre=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config
ExecStart=/usr/sbin/sshd -D -f /etc/felhom-sshd-spike/sshd_config
ExecReload=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target

---

4. P3 — customer-owned-Proxmox coexistence (the headline) — GO (four axes, evidenced)

Stock :22 as the customer-sshd stand-in, felhom-sshd on 8822 simultaneously:

Axis	Evidence
Both accept independently	felhom-sshd key login (localhost:8822 → whoami=root); LAN :22 kept working (CC's own line)
Distinct host keys	stock ed25519 SHA256:8Tea… vs felhom SHA256:EnC4… — a client pinning one is never confused by the other (known_hosts hygiene for the operator)
Reload stock → felhom unaffected	systemctl reload ssh → felhom PID 4095087 unchanged, still listening
Restart felhom → stock unaffected	systemctl restart felhom-sshd-spike → stock PID 922 unchanged; pveproxy+pvedaemon stayed active; no port collision with 8006/8007/85/111/3128

PVE (pveproxy :8006 + pvedaemon) undisturbed throughout; guest 9201 healthy.

---

5. P4 — nft scoping to the tunnel — GO (LAN-block + :22-untouched evidenced; tunnel-accept by structure)

Named table felhom_sshd_spike, retargeted from the prior spike's proven belt to the felhom port:

table inet felhom_sshd_spike {
  chain input { type filter hook input priority -5; policy accept;
    iifname "wg-felhom" tcp dport 8822 ip saddr <operator/32> counter accept   # tunnel, operator only
    iifname "wg-felhom" tcp dport 8822 counter drop                            # tunnel, other src
    tcp dport 8822 iifname != "wg-felhom" counter drop                         # OFF-tunnel: unreachable
  } }

• LAN → 8822: BLOCKED — a LAN ssh -p 8822 from CC Connection timed out; the off-tunnel drop counter incremented (3 pkts). Localhost → 8822 also blocked (lo ≠ wg-felhom), confirming tunnel-only.
• Stock :22 UNTOUCHED — nft list ruleset | grep "dport 22" is EMPTY; LAN :22 kept working (the safety line), by construction (the rules match dport <felhom-port> only, never 22).
• Tunnel-accept path: not re-exercised live here (the safety rail forbade re-establishing the operator peer on wg-felhom in this spike); it is the byte-identical belt to SPIKE-oob-wg-operator-peer-2026-07-05.md §5, which WAS live-tested against a real operator peer (operator→box SSH at ~24 ms, dummy dropped). Structure + prior live proof = confident.

---

6. P5 — robustness — GO (with a sharp restart-vs-reload finding)

• SAFE config-change path — reload survives a bad config: appended a bogus directive → systemctl reload felhom-sshd-spike refused (the ExecReload=sshd -t gate fails) → the running instance SURVIVED (PID unchanged, still listening on 8822 with the last-good config). This is the path the TASK must use for config changes.
• UNSAFE path — restart is NOT atomic: systemctl restart with a bad config takes the daemon DOWN (systemd stops the old instance, then ExecStartPre=sshd -t fails so nothing comes back; unit ends failed/activating, PID 0). A bad config therefore must be caught BEFORE a restart (validate, then reload — never restart-on-change).
• Blast-containment (post-fix): even the unsafe restart+badconfig left stock :22 (PID 922) and /run/sshd INTACT and LAN :22 working — because the SAFE unit (§3) has no RuntimeDirectory=sshd. (Before the fix, this exact sequence removed /run/sshd and took :22 down — §8.)
• Boot persistence: enabled; host-key fingerprint STABLE (SHA256:EnC4…) across restart.
• Late-network independence: binds 0.0.0.0 (not a late-coming IP), After=network.target only, no wg-felhom dependency — comes up regardless of tunnel state.
• Start-limit gotcha (D1 echo): repeated failed starts hit systemd's StartLimit → the unit refuses further starts until systemctl reset-failed. The TASK's health/repair logic must reset-failed before a deliberate restart (same lesson as the agent self-update spike).

---

7. P6 — operator login identity — GO + the clean separation mechanism

Throwaway felhom-op-spike (normal login user, --create-home, /bin/bash) with a scoped sudoers (/etc/sudoers.d/felhom-op-spike: mkdir -p /run/sshd, systemctl restart felhom-sshd-spike, systemctl start felhom-agent — the repair verbs):

• Key login via felhom-sshd (8822) works → whoami=felhom-op-spike, scoped sudo -n → SUDO_OK.
• NOT reachable via stock :22 → Permission denied (publickey). Why it's clean: the operator key lives ONLY in felhom-sshd's AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u, NOT in ~felhom-op-spike/.ssh/authorized_keys (which doesn't exist). So the customer's sshd — which reads ~/.ssh/authorized_keys — never honours the operator key, without us having to touch the customer's AllowUsers/config at all. This is the load-bearing separation primitive.
• AllowUsers root felhom-op-spike on felhom-sshd is the second gate (who may use 8822).

Operator-identity trade (record-only, for the TASK to decide):

• felhom-op (dedicated, sudo-scoped) — clean least-privilege, auditable, isolated to felhom-sshd via the AuthorizedKeysFile trick; needs a scoped sudoers for the repair verbs it will run (agent restart, wg repair, mkdir /run/sshd, etc.). Doesn't collide with the customer's users.
• root (PVE-native) — deep host repairs (pct, systemctl, storage) need root anyway; PermitRootLogin prohibit-password + the tunnel-only nft belt + the dedicated AuthorizedKeysFile still contain it; simplest for "the operator must be able to fix anything". Trade: broader blast radius if the operator key leaks (mitigated by the tunnel gate + key custody).
• Likely answer: felhom-op with a tightly-scoped sudoers for the known repair actions, escalating to root only via those sudo verbs — but that's a TASK decision, recorded here, not made.

---

8. THE INCIDENT — RuntimeDirectory=sshd clobbered the shared /run/sshd (caught, fixed, root-caused)

What happened. The P1 unit was first written with RuntimeDirectory=sshd. During the P5 bad-config test, systemctl restart felhom-sshd-spike failed at ExecStartPre (as designed) and the unit went failed. systemd, cleaning up that unit's runtime directory, removed /run/sshd — the compiled-in OpenSSH privilege-separation directory, which the STOCK sshd on :22 also depends on. The stock listener (PID 922) stayed up and accepted TCP, but every new session reset right after SSH2_MSG_KEXINIT — the textbook signature of a child that cannot initialise privsep. CC's LAN SSH — the safety line — went down.

Why the "shared RuntimeDirectory is preserved" assumption was wrong. Debian's stock ssh.service also declares RuntimeDirectory=sshd; I expected systemd (v257) to keep /run/sshd alive while the active stock unit referenced it. Empirically it did not — the failing unit's cleanup removed the shared dir. (A full 135 s connection-drain ruled out MaxStartups; the cause was structural.)

Recovery (no shell available; the honest bit). :22 was the only shell path and it was the thing broken; the felhom-sshd port was down (start-limited); the PVE API token cached on the build server was stale (401); no root password is documented (correctly). The customer data plane was unaffected (felhom.demo-felhom.eu 200, pveproxy 200, agent alive) — a management lockout only. Recovery was via the PVE web console (operator authenticated the session — CC is prohibited from entering passwords — then CC drove the node xterm.js shell): mkdir -p /run/sshd && chmod 0755 /run/sshd restored stock :22 instantly, without even restarting the stock sshd (PID stayed 922). Confirmed live: ls /run/sshd → "No such file or directory" pre-fix; present post-fix.

The fix in the design (mandatory TASK constraint #1). A second sshd instance must NEVER declare RuntimeDirectory=sshd. Ensure /run/sshd exists idempotently instead — the spike used ExecStartPre=-/usr/bin/mkdir -p /run/sshd; the production-clean equivalent is a tmpfiles.d entry (d /run/sshd 0755 root root -) so the dir is boot-persistent and owned by neither sshd unit's lifecycle. With the fixed unit, even restart+badconfig (§6) left /run/sshd and :22 intact.

Secondary lessons from the incident: (a) the build-server PVE token copy is stale — CC's only non-SSH lever into the host is unusable; a working break-glass path (a valid console credential, or a second admin key on a different port) should exist BEFORE this feature ships. (b) reset-after-KEXINIT with the listener up = missing/again-broken /run/sshd — a fast diagnostic for the runbook.

---

9. Findings that must shape the TASK

1. NEVER RuntimeDirectory=sshd in the felhom-sshd unit (§8). It removes the shared /run/sshd on stop/failure and takes the customer's sshd down. Ship a tmpfiles.d /run/sshd entry (or an idempotent ExecStartPre=mkdir) instead. This is the #1 safety constraint and it is not obvious.
2. Config changes go via reload (HUP) behind an ExecReload=sshd -t gate — never restart (§6). Reload keeps the running instance on the last-good config if the new one is bad; restart is not atomic and downs the daemon on a failed pre-check. Validate, then reload.
3. Identity isolation is the AuthorizedKeysFile location, not the customer's config (§7). Putting the operator key under /etc/felhom-sshd/authorized_keys/%u (outside ~/.ssh) means the customer's sshd never honours it and we never touch their config. This is the clean coexistence primitive — carry it into the design.
4. Port claim: probe-and-claim over an ordered candidate list, fail LOUD on exhaustion (§2). Never silently fall back to :22 or a random port. Persist the chosen port; be idempotent. The shell PoC works; the TASK's agent-Go version should mirror the ss-empty and real-bind() test.
5. Ownership: agent-managed, mirroring the wg-felhom/dnsmasq pattern. felhom-sshd is ongoing reconciled desired-state (port may need re-claim, keys rotate, AllowUsers changes) — the agent's job, not the one-shot host-install. Grant a NARROW sudoers slice (install the config + host key to /etc/felhom-sshd/*, sshd -t, systemctl {reload,restart,enable,disable} felhom-sshd, the tmpfiles/mkdir /run/sshd). sshd -t is the pre-apply gate the way felhom-peersync validates before applying. Host-install can lay down the STATIC skeleton on a fresh box; the agent reconciles.
6. Debian's ssh@.service/ssh.socket (socket-activation) is NOT a cleaner base than a bespoke unit here (P0). A per-connection socket-activated instance still needs /run/sshd and gives no coexistence benefit; a dedicated always-on felhom-sshd.service with its own config is simpler and is what the spike validated. Recommend the bespoke unit.
7. PAM: UsePAM yes works on PVE with zero extra config (the second sshd reuses the sshd PAM service). No PVE-specific PAM work needed for key-only login. (If the TASK ever enables anything PAM-gated like 2FA, revisit — out of scope here.)
8. Fresh-host vs customer-host is a CONFIG VALUE, not a fork. The same felhom-sshd instance model works whether or not a stock sshd exists on :22 — on a fresh Felhom box it simply coexists with the Proxmox-default sshd; on a customer box it coexists with theirs. The only variable is the CLAIMED PORT (probe-and-claim handles both). No separate code path; the install just always claims a port and never assumes :22 is Felhom's.
9. Break-glass prerequisite (§8 secondary). Before this ships, ensure a working out-of-band root path independent of the felhom-sshd port itself (a valid PVE console credential and/or a distinct recovery key), and keep the build-server PVE token fresh — the stale token left CC with no non-SSH lever during the incident.

---

10. Cleanup assertion (§13) — VERIFIED to the P0 baseline

Removed exactly the spike set: felhom-sshd-spike.service (disable+rm+daemon-reload+reset-failed), /etc/felhom-sshd-spike/ (config + host key + authorized_keys), /etc/sudoers.d/felhom-op-spike, the felhom-op-spike user + home (userdel -r), /usr/local/sbin/felhom-sshd-spike-claim.sh, /root/felhom-sshd-spike-client{,.pub}, /run/felhom-sshd-spike.pid, /run/felhom-claim-listener.pid, the felhom_sshd_spike nft table (both the box belt and the P4 table — removed). Re-verified: stock ssh.service active, PID 922 (same as P0 — never restarted; the /run/sshd fix alone restored it), only :22 listening, sshd -T port 22, felhom-op-spike gone, unit gone, /etc/felhom-sshd-spike gone, nft list tables EMPTY (P0), PVE active, guest 9201 running. One intentional deviation from P0, permanent and correct: /run/sshd exists (recreated during §8 recovery) — it existed at P0 too; only its mtime differs. Leaving it is required.

No repo/hub/agent/manifest production change — this commit is docs-only.

---

11. Method-bar honored

wg show dump never run; wg-felhom/PBS untouched. Every mutation a separately-named unit/dir/file/nft-table, inventoried as created; P7 removed exactly that set. Coexistence and negative claims carry evidence (PIDs, ss, nft counters, session survival, fingerprints), not "seemed fine". The one surprise (the /run/sshd clobber) is recorded as the headline finding, not massaged. Per-probe verdicts: P0 GATE-PASS, P1 GO, P2 GO, P3 GO, P4 GO, P5 GO (+restart/reload finding), P6 GO, §8 SURPRISE→FIXED.