docs(audit): dedicated felhom-sshd instance spike (2026-07-05) — GO

Validates a second sshd (own port/config/host-key/AuthorizedKeysFile/unit) as
Felhom's OOB entry point, coexisting with the customer's stock sshd on :22:
robust port probe-and-claim (skip/idempotent/loud-exhaustion), four-axis
coexistence independence, tunnel-scoped nft, reload-gate robustness, operator
identity isolated via AuthorizedKeysFile location, clean uninstall.

#1 TASK constraint (caught live): a second sshd unit must NEVER declare
RuntimeDirectory=sshd — it removes the SHARED /run/sshd privsep dir on stop and
takes the stock sshd down (LAN SSH lockout; recovered via PVE console + mkdir
/run/sshd). Use a tmpfiles.d entry instead. All spike artifacts removed; baseline
re-verified.

Docs-only; no code/hub/agent/manifest change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

documentation/audits/SPIKE-felhom-sshd-2026-07-05.md

@@ -0,0 +1,315 @@
+# SPIKE — dedicated `felhom-sshd` instance (own port/config/keys, customer-host coexistence) — 2026-07-05
+
+> **STATUS: COMPLETE — verdict below, with one CAUGHT-AND-FIXED incident (§8) that is itself the
+> single most important finding.** Validates a dedicated second sshd instance as Felhom's OOB entry
+> point on the demo host `felhom-pve` (node `demo-felhom`, PVE 9.2.2, OpenSSH 10.0p2). All spike
+> artifacts removed at cleanup (§13, re-verified to the P0 baseline). No production code shipped.
+
+**Class:** SPIKE (empirical; no product code). **Repos:** felhom.eu (this doc only); felhom-agent +
+felhom.eu/scripts read-only for grounding. **Host:** `felhom-pve` (`192.168.0.162`). The offsite
+`wg-felhom` tunnel + PBS path were NOT touched (OOB transport was proven separately in
+`SPIKE-oob-wg-operator-peer-2026-07-05.md`).
+
+**Verdict (one line):** **GO — a dedicated `felhom-sshd` on a claimed non-22 port, with its own
+config/host-key/AuthorizedKeysFile/unit, coexists cleanly with the stock (customer) sshd on :22
+(independence proven on all four axes), claims its port robustly (skip/idempotent/loud-exhaustion),
+scopes to the tunnel by nft, and uninstalls to a byte-clean baseline — PROVIDED the unit NEVER
+declares `RuntimeDirectory=sshd` (which removes the SHARED `/run/sshd` privsep dir on stop and takes
+the stock sshd down with it; this spike hit that live, §8, and it is the #1 TASK constraint).**
+
+---
+
+## 0. What the demo box could and could not represent
+
+The box is **Felhom-owned** and single-NAT; the stock `ssh.service` on :22 stood in for "the
+customer's existing sshd". Faithfully represented: two independent sshd instances on one host, port
+contention, nft scoping, the systemd/privsep mechanics, uninstall removability. NOT represented: a
+**real customer's** sshd config (they may run `AllowUsers`, non-default ports, `Match` blocks,
+`fail2ban`/`sshguard`, a hardened `sshd_config` that our drop-in model doesn't inherit — we ship a
+SEPARATE config so we don't inherit theirs, which is the point), their actual **port map** (our
+candidate list must still probe-and-claim against whatever they run), and true **CGNAT**. The
+privsep-dir incident (§8) is host-OS-generic (Debian/PVE), so it applies to a customer host
+identically.
+
+---
+
+## 1. P0 — ground truth (no changes) — GATE PASSED
+
+```
+stock sshd unit  : ssh.service  (Debian; sshd.service/sshd@.service are ALIASES/indirect)
+                   MainPID 922, FragmentPath /usr/lib/systemd/system/ssh.service
+effective config : port 22 ; ListenAddress 0.0.0.0:22 + [::]:22 ; UsePAM yes ;
+                   PermitRootLogin yes ; PasswordAuthentication yes  (Proxmox defaults) ;
+                   AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 ;
+                   HostKey rsa+ecdsa+ed25519 ; OpenSSH_10.0p2 Debian-7+deb13u4
+instance template: /lib/systemd/system/ssh@.service ABSENT at that path; BUT `sshd@.service`
+                   (indirect) + `ssh.socket` (disabled) exist — socket-activation route is available.
+listeners (ss -tlnp): 22(sshd) 111(rpcbind) 85(pvedaemon,localhost) 25(postfix,localhost)
+                   53(dnsmasq) 8443(felhom-agent) 3128(spiceproxy) 8006(pveproxy)
+                   → candidate ports 8822/2222/8022/62222 ALL FREE.
+pve-firewall     : disabled ;  nft list tables: EMPTY (baseline)
+PVE health       : pveproxy + pvedaemon active
+```
+**Gate:** stock sshd identified (`ssh.service`/PID 922/:22) and never touched thereafter.
+
+---
+
+## 2. P2 — port probe-then-claim under contention — GO (all four behaviours)
+
+Claim routine (`/usr/local/sbin/felhom-sshd-spike-claim.sh`, throwaway) — candidate list
+**`8822 2222 8022 62222`**; `is_free()` = nothing in `ss -Htln sport=:N` **AND** a real `python3`
+bind succeeds; idempotent re-pick of the persisted port if still free; persists to
+`/etc/felhom-sshd-spike/port`.
+
+| Scenario | Result |
+|---|---|
+| clean | → **8822**, persisted |
+| **contention** (a listener bound on 8822, persisted cleared) | → **skips to 2222** |
+| **idempotent** (re-run while 8822 still busy) | → **stays 2222** (no thrash) |
+| **exhaustion** (all four occupied) | → **rc=3, LOUD stderr** "refusing to fall back to :22 or a random port" — **never silently grabs :22** |
+| recovery (all free) | → 8822 |
+
+The exhaustion-must-fail-loud property (a silent :22 grab would be the worst bug) is proven, not
+assumed.
+
+**Algorithm (copy-paste, the shape the TASK's Go/agent version should mirror):** ordered candidate
+list → for each, `ss -Htln "sport = :$p"` empty **and** an actual `bind()` succeeds → first free
+wins → persist + emit; if none free → non-zero exit + explicit error, **no fallback**. Idempotency
+gate reads the persisted port first and keeps it if still free.
+
+---
+
+## 3. P1 — dedicated instance, minimal — GO
+
+Working `sshd_config` (`/etc/felhom-sshd-spike/sshd_config`), copy-paste ready:
+```
+Port <claimed>            # from P2
+ListenAddress 0.0.0.0
+ListenAddress ::
+HostKey /etc/felhom-sshd-spike/ssh_host_ed25519_key   # DEDICATED key (ssh-keygen -t ed25519)
+PidFile /run/felhom-sshd-spike.pid
+AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u   # OUTSIDE ~/.ssh — see P6
+PasswordAuthentication no
+PermitRootLogin prohibit-password        # key-only
+PubkeyAuthentication yes
+UsePAM yes                               # works on PVE with no extra PAM config (service name "sshd")
+AllowUsers root felhom-op-spike          # the operator identity/ies only
+X11Forwarding no
+Subsystem sftp internal-sftp
+```
+The `sshd -t -f <conf>` validation gate passed. **GO:** `ss -tlnp` showed felhom-sshd on 8822
+(PID 4095087) AND stock sshd on :22 (PID 922, UNCHANGED), both LISTEN; `Type=notify` sd_notify'd
+cleanly (unit `active`, not just `activating`).
+
+**The `.service` — SAFE FINAL FORM (post-§8-incident):**
+```
+[Unit]
+Description=Felhom dedicated OOB sshd
+After=network.target
+[Service]
+Type=notify
+# NEVER "RuntimeDirectory=sshd" — systemd removes /run/sshd on stop, clobbering the SHARED
+# privsep dir and breaking the stock sshd (§8). Ensure the dir exists idempotently instead:
+ExecStartPre=-/usr/bin/mkdir -p /run/sshd
+ExecStartPre=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config
+ExecStart=/usr/sbin/sshd -D -f /etc/felhom-sshd-spike/sshd_config
+ExecReload=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config
+ExecReload=/bin/kill -HUP $MAINPID
+KillMode=process
+Restart=on-failure
+[Install]
+WantedBy=multi-user.target
+```
+
+---
+
+## 4. P3 — customer-owned-Proxmox coexistence (the headline) — GO (four axes, evidenced)
+
+Stock :22 as the customer-sshd stand-in, felhom-sshd on 8822 simultaneously:
+
+| Axis | Evidence |
+|---|---|
+| **Both accept independently** | felhom-sshd key login (localhost:8822 → `whoami=root`); LAN :22 kept working (CC's own line) |
+| **Distinct host keys** | stock ed25519 `SHA256:8Tea…` vs felhom `SHA256:EnC4…` — a client pinning one is never confused by the other (known_hosts hygiene for the operator) |
+| **Reload stock → felhom unaffected** | `systemctl reload ssh` → felhom PID **4095087 unchanged**, still listening |
+| **Restart felhom → stock unaffected** | `systemctl restart felhom-sshd-spike` → stock PID **922 unchanged**; pveproxy+pvedaemon stayed active; no port collision with 8006/8007/85/111/3128 |
+
+PVE (pveproxy :8006 + pvedaemon) undisturbed throughout; guest 9201 healthy.
+
+---
+
+## 5. P4 — nft scoping to the tunnel — GO (LAN-block + :22-untouched evidenced; tunnel-accept by structure)
+
+Named table `felhom_sshd_spike`, retargeted from the prior spike's proven belt to the felhom port:
+```
+table inet felhom_sshd_spike {
+  chain input { type filter hook input priority -5; policy accept;
+    iifname "wg-felhom" tcp dport 8822 ip saddr <operator/32> counter accept   # tunnel, operator only
+    iifname "wg-felhom" tcp dport 8822 counter drop                            # tunnel, other src
+    tcp dport 8822 iifname != "wg-felhom" counter drop                         # OFF-tunnel: unreachable
+  } }
+```
+- **LAN → 8822: BLOCKED** — a LAN `ssh -p 8822` from CC `Connection timed out`; the off-tunnel drop
+  counter incremented (3 pkts). Localhost → 8822 also blocked (lo ≠ wg-felhom), confirming
+  tunnel-only.
+- **Stock :22 UNTOUCHED** — `nft list ruleset | grep "dport 22"` is EMPTY; LAN :22 kept working
+  (the safety line), by construction (the rules match `dport <felhom-port>` only, never 22).
+- **Tunnel-accept path:** not re-exercised live here (the safety rail forbade re-establishing the
+  operator peer on `wg-felhom` in this spike); it is the **byte-identical belt** to
+  `SPIKE-oob-wg-operator-peer-2026-07-05.md` §5, which WAS live-tested against a real operator peer
+  (operator→box SSH at ~24 ms, dummy dropped). Structure + prior live proof = confident.
+
+---
+
+## 6. P5 — robustness — GO (with a sharp restart-vs-reload finding)
+
+- **SAFE config-change path — `reload` survives a bad config:** appended a bogus directive →
+  `systemctl reload felhom-sshd-spike` **refused** (the `ExecReload=sshd -t` gate fails) → the
+  running instance **SURVIVED** (PID unchanged, still listening on 8822 with the last-good config).
+  This is the path the TASK must use for config changes.
+- **UNSAFE path — `restart` is NOT atomic:** `systemctl restart` with a bad config takes the daemon
+  **DOWN** (systemd stops the old instance, then `ExecStartPre=sshd -t` fails so nothing comes back;
+  unit ends `failed`/`activating`, PID 0). A bad config therefore must be caught BEFORE a restart
+  (validate, then reload — never restart-on-change).
+- **Blast-containment (post-fix):** even the unsafe `restart+badconfig` left stock :22 (PID 922) and
+  `/run/sshd` INTACT and LAN :22 working — because the SAFE unit (§3) has no `RuntimeDirectory=sshd`.
+  (Before the fix, this exact sequence removed `/run/sshd` and took :22 down — §8.)
+- **Boot persistence:** `enabled`; host-key fingerprint **STABLE** (`SHA256:EnC4…`) across restart.
+- **Late-network independence:** binds `0.0.0.0` (not a late-coming IP), `After=network.target`
+  only, **no `wg-felhom` dependency** — comes up regardless of tunnel state.
+- **Start-limit gotcha (D1 echo):** repeated failed starts hit systemd's StartLimit → the unit
+  refuses further starts until `systemctl reset-failed`. The TASK's health/repair logic must
+  `reset-failed` before a deliberate restart (same lesson as the agent self-update spike).
+
+---
+
+## 7. P6 — operator login identity — GO + the clean separation mechanism
+
+Throwaway `felhom-op-spike` (normal login user, `--create-home`, `/bin/bash`) with a scoped sudoers
+(`/etc/sudoers.d/felhom-op-spike`: `mkdir -p /run/sshd`, `systemctl restart felhom-sshd-spike`,
+`systemctl start felhom-agent` — the repair verbs):
+- **Key login via felhom-sshd (8822) works** → `whoami=felhom-op-spike`, scoped `sudo -n` → `SUDO_OK`.
+- **NOT reachable via stock :22** → `Permission denied (publickey)`. **Why it's clean:** the operator
+  key lives ONLY in felhom-sshd's `AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u`,
+  NOT in `~felhom-op-spike/.ssh/authorized_keys` (which doesn't exist). So the customer's sshd —
+  which reads `~/.ssh/authorized_keys` — never honours the operator key, **without us having to
+  touch the customer's `AllowUsers`/config at all.** This is the load-bearing separation primitive.
+- `AllowUsers root felhom-op-spike` on felhom-sshd is the second gate (who may use 8822).
+
+**Operator-identity trade (record-only, for the TASK to decide):**
+- **`felhom-op` (dedicated, sudo-scoped)** — clean least-privilege, auditable, isolated to
+  felhom-sshd via the AuthorizedKeysFile trick; needs a scoped sudoers for the repair verbs it will
+  run (agent restart, wg repair, `mkdir /run/sshd`, etc.). Doesn't collide with the customer's users.
+- **`root` (PVE-native)** — deep host repairs (pct, systemctl, storage) need root anyway; PermitRootLogin
+  prohibit-password + the tunnel-only nft belt + the dedicated AuthorizedKeysFile still contain it;
+  simplest for "the operator must be able to fix anything". Trade: broader blast radius if the
+  operator key leaks (mitigated by the tunnel gate + key custody).
+- Likely answer: **`felhom-op` with a tightly-scoped sudoers for the known repair actions, escalating
+  to root only via those sudo verbs** — but that's a TASK decision, recorded here, not made.
+
+---
+
+## 8. THE INCIDENT — `RuntimeDirectory=sshd` clobbered the shared `/run/sshd` (caught, fixed, root-caused)
+
+**What happened.** The P1 unit was first written with `RuntimeDirectory=sshd`. During the P5
+bad-config test, `systemctl restart felhom-sshd-spike` failed at `ExecStartPre` (as designed) and the
+unit went `failed`. systemd, cleaning up that unit's runtime directory, **removed `/run/sshd`** — the
+compiled-in OpenSSH **privilege-separation directory**, which the **STOCK** sshd on :22 also depends
+on. The stock listener (PID 922) stayed up and accepted TCP, but every new session **reset right
+after `SSH2_MSG_KEXINIT`** — the textbook signature of a child that cannot initialise privsep. **CC's
+LAN SSH — the safety line — went down.**
+
+**Why the "shared RuntimeDirectory is preserved" assumption was wrong.** Debian's stock `ssh.service`
+also declares `RuntimeDirectory=sshd`; I expected systemd (v257) to keep `/run/sshd` alive while the
+active stock unit referenced it. Empirically it did **not** — the failing unit's cleanup removed the
+shared dir. (A full 135 s connection-drain ruled out `MaxStartups`; the cause was structural.)
+
+**Recovery (no shell available; the honest bit).** `:22` was the only shell path and it was the thing
+broken; the felhom-sshd port was down (start-limited); the PVE API token cached on the build server
+was **stale (401)**; no root password is documented (correctly). The customer **data plane was
+unaffected** (`felhom.demo-felhom.eu` 200, pveproxy 200, agent alive) — a *management* lockout only.
+Recovery was via the **PVE web console** (operator authenticated the session — CC is prohibited from
+entering passwords — then CC drove the node xterm.js shell): `mkdir -p /run/sshd && chmod 0755
+/run/sshd` restored stock :22 **instantly, without even restarting the stock sshd** (PID stayed 922).
+Confirmed live: `ls /run/sshd` → "No such file or directory" pre-fix; present post-fix.
+
+**The fix in the design (mandatory TASK constraint #1).** A second sshd instance must **NEVER**
+declare `RuntimeDirectory=sshd`. Ensure `/run/sshd` exists idempotently instead — the spike used
+`ExecStartPre=-/usr/bin/mkdir -p /run/sshd`; the production-clean equivalent is a
+`tmpfiles.d` entry (`d /run/sshd 0755 root root -`) so the dir is boot-persistent and owned by
+neither sshd unit's lifecycle. With the fixed unit, even `restart`+badconfig (§6) left `/run/sshd`
+and :22 intact.
+
+**Secondary lessons from the incident:** (a) the build-server PVE token copy is **stale** — CC's only
+non-SSH lever into the host is unusable; a working break-glass path (a valid console credential, or a
+second admin key on a different port) should exist BEFORE this feature ships. (b) reset-after-KEXINIT
+with the listener up = missing/again-broken `/run/sshd` — a fast diagnostic for the runbook.
+
+---
+
+## 9. Findings that must shape the TASK
+
+1. **NEVER `RuntimeDirectory=sshd` in the felhom-sshd unit (§8).** It removes the shared `/run/sshd`
+   on stop/failure and takes the customer's sshd down. Ship a `tmpfiles.d` `/run/sshd` entry (or an
+   idempotent `ExecStartPre=mkdir`) instead. This is the #1 safety constraint and it is not obvious.
+2. **Config changes go via `reload` (HUP) behind an `ExecReload=sshd -t` gate — never `restart`
+   (§6).** Reload keeps the running instance on the last-good config if the new one is bad; restart
+   is not atomic and downs the daemon on a failed pre-check. Validate, then reload.
+3. **Identity isolation is the `AuthorizedKeysFile` location, not the customer's config (§7).**
+   Putting the operator key under `/etc/felhom-sshd/authorized_keys/%u` (outside `~/.ssh`) means the
+   customer's sshd never honours it and we never touch their config. This is the clean coexistence
+   primitive — carry it into the design.
+4. **Port claim: probe-and-claim over an ordered candidate list, fail LOUD on exhaustion (§2).**
+   Never silently fall back to :22 or a random port. Persist the chosen port; be idempotent. The
+   shell PoC works; the TASK's agent-Go version should mirror the `ss`-empty **and** real-`bind()`
+   test.
+5. **Ownership: agent-managed, mirroring the `wg-felhom`/`dnsmasq` pattern.** felhom-sshd is ongoing
+   reconciled desired-state (port may need re-claim, keys rotate, AllowUsers changes) — the agent's
+   job, not the one-shot host-install. Grant a NARROW sudoers slice (install the config + host key to
+   `/etc/felhom-sshd/*`, `sshd -t`, `systemctl {reload,restart,enable,disable} felhom-sshd`, the
+   `tmpfiles`/`mkdir /run/sshd`). `sshd -t` is the pre-apply gate the way `felhom-peersync` validates
+   before applying. Host-install can lay down the STATIC skeleton on a fresh box; the agent reconciles.
+6. **Debian's `ssh@.service`/`ssh.socket` (socket-activation) is NOT a cleaner base than a bespoke
+   unit here (P0).** A per-connection socket-activated instance still needs `/run/sshd` and gives no
+   coexistence benefit; a dedicated always-on `felhom-sshd.service` with its own config is simpler
+   and is what the spike validated. Recommend the bespoke unit.
+7. **PAM: `UsePAM yes` works on PVE with zero extra config** (the second sshd reuses the `sshd` PAM
+   service). No PVE-specific PAM work needed for key-only login. (If the TASK ever enables anything
+   PAM-gated like 2FA, revisit — out of scope here.)
+8. **Fresh-host vs customer-host is a CONFIG VALUE, not a fork.** The same felhom-sshd instance model
+   works whether or not a stock sshd exists on :22 — on a fresh Felhom box it simply coexists with the
+   Proxmox-default sshd; on a customer box it coexists with theirs. The only variable is the CLAIMED
+   PORT (probe-and-claim handles both). No separate code path; the install just always claims a port
+   and never assumes :22 is Felhom's.
+9. **Break-glass prerequisite (§8 secondary).** Before this ships, ensure a working out-of-band root
+   path independent of the felhom-sshd port itself (a valid PVE console credential and/or a distinct
+   recovery key), and keep the build-server PVE token fresh — the stale token left CC with no
+   non-SSH lever during the incident.
+
+---
+
+## 10. Cleanup assertion (§13) — VERIFIED to the P0 baseline
+
+Removed exactly the spike set: `felhom-sshd-spike.service` (disable+rm+daemon-reload+reset-failed),
+`/etc/felhom-sshd-spike/` (config + host key + authorized_keys), `/etc/sudoers.d/felhom-op-spike`,
+the `felhom-op-spike` user + home (`userdel -r`), `/usr/local/sbin/felhom-sshd-spike-claim.sh`,
+`/root/felhom-sshd-spike-client{,.pub}`, `/run/felhom-sshd-spike.pid`, `/run/felhom-claim-listener.pid`,
+the `felhom_sshd_spike` nft table (both the box belt and the P4 table — removed). Re-verified:
+stock `ssh.service` **active, PID 922** (same as P0 — never restarted; the /run/sshd fix alone
+restored it), only `:22` listening, `sshd -T` port 22, `felhom-op-spike` gone, unit gone,
+`/etc/felhom-sshd-spike` gone, `nft list tables` EMPTY (P0), PVE active, guest 9201 running.
+**One intentional deviation from P0, permanent and correct:** `/run/sshd` exists (recreated during
+§8 recovery) — it existed at P0 too; only its mtime differs. Leaving it is required.
+
+**No repo/hub/agent/manifest production change** — this commit is docs-only.
+
+---
+
+## 11. Method-bar honored
+
+`wg show dump` never run; `wg-felhom`/PBS untouched. Every mutation a separately-named
+unit/dir/file/nft-table, inventoried as created; P7 removed exactly that set. Coexistence and
+negative claims carry evidence (PIDs, `ss`, nft counters, session survival, fingerprints), not
+"seemed fine". The one surprise (the `/run/sshd` clobber) is recorded as the headline finding, not
+massaged. Per-probe verdicts: P0 GATE-PASS, P1 GO, P2 GO, P3 GO, P4 GO, P5 GO (+restart/reload
+finding), P6 GO, §8 SURPRISE→FIXED.