diff --git a/documentation/audits/SPIKE-felhom-sshd-2026-07-05.md b/documentation/audits/SPIKE-felhom-sshd-2026-07-05.md new file mode 100644 index 0000000..5d35c7d --- /dev/null +++ b/documentation/audits/SPIKE-felhom-sshd-2026-07-05.md @@ -0,0 +1,315 @@ +# SPIKE — dedicated `felhom-sshd` instance (own port/config/keys, customer-host coexistence) — 2026-07-05 + +> **STATUS: COMPLETE — verdict below, with one CAUGHT-AND-FIXED incident (§8) that is itself the +> single most important finding.** Validates a dedicated second sshd instance as Felhom's OOB entry +> point on the demo host `felhom-pve` (node `demo-felhom`, PVE 9.2.2, OpenSSH 10.0p2). All spike +> artifacts removed at cleanup (§13, re-verified to the P0 baseline). No production code shipped. + +**Class:** SPIKE (empirical; no product code). **Repos:** felhom.eu (this doc only); felhom-agent + +felhom.eu/scripts read-only for grounding. **Host:** `felhom-pve` (`192.168.0.162`). The offsite +`wg-felhom` tunnel + PBS path were NOT touched (OOB transport was proven separately in +`SPIKE-oob-wg-operator-peer-2026-07-05.md`). + +**Verdict (one line):** **GO — a dedicated `felhom-sshd` on a claimed non-22 port, with its own +config/host-key/AuthorizedKeysFile/unit, coexists cleanly with the stock (customer) sshd on :22 +(independence proven on all four axes), claims its port robustly (skip/idempotent/loud-exhaustion), +scopes to the tunnel by nft, and uninstalls to a byte-clean baseline — PROVIDED the unit NEVER +declares `RuntimeDirectory=sshd` (which removes the SHARED `/run/sshd` privsep dir on stop and takes +the stock sshd down with it; this spike hit that live, §8, and it is the #1 TASK constraint).** + +--- + +## 0. What the demo box could and could not represent + +The box is **Felhom-owned** and single-NAT; the stock `ssh.service` on :22 stood in for "the +customer's existing sshd". Faithfully represented: two independent sshd instances on one host, port +contention, nft scoping, the systemd/privsep mechanics, uninstall removability. NOT represented: a +**real customer's** sshd config (they may run `AllowUsers`, non-default ports, `Match` blocks, +`fail2ban`/`sshguard`, a hardened `sshd_config` that our drop-in model doesn't inherit — we ship a +SEPARATE config so we don't inherit theirs, which is the point), their actual **port map** (our +candidate list must still probe-and-claim against whatever they run), and true **CGNAT**. The +privsep-dir incident (§8) is host-OS-generic (Debian/PVE), so it applies to a customer host +identically. + +--- + +## 1. P0 — ground truth (no changes) — GATE PASSED + +``` +stock sshd unit : ssh.service (Debian; sshd.service/sshd@.service are ALIASES/indirect) + MainPID 922, FragmentPath /usr/lib/systemd/system/ssh.service +effective config : port 22 ; ListenAddress 0.0.0.0:22 + [::]:22 ; UsePAM yes ; + PermitRootLogin yes ; PasswordAuthentication yes (Proxmox defaults) ; + AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 ; + HostKey rsa+ecdsa+ed25519 ; OpenSSH_10.0p2 Debian-7+deb13u4 +instance template: /lib/systemd/system/ssh@.service ABSENT at that path; BUT `sshd@.service` + (indirect) + `ssh.socket` (disabled) exist — socket-activation route is available. +listeners (ss -tlnp): 22(sshd) 111(rpcbind) 85(pvedaemon,localhost) 25(postfix,localhost) + 53(dnsmasq) 8443(felhom-agent) 3128(spiceproxy) 8006(pveproxy) + → candidate ports 8822/2222/8022/62222 ALL FREE. +pve-firewall : disabled ; nft list tables: EMPTY (baseline) +PVE health : pveproxy + pvedaemon active +``` +**Gate:** stock sshd identified (`ssh.service`/PID 922/:22) and never touched thereafter. + +--- + +## 2. P2 — port probe-then-claim under contention — GO (all four behaviours) + +Claim routine (`/usr/local/sbin/felhom-sshd-spike-claim.sh`, throwaway) — candidate list +**`8822 2222 8022 62222`**; `is_free()` = nothing in `ss -Htln sport=:N` **AND** a real `python3` +bind succeeds; idempotent re-pick of the persisted port if still free; persists to +`/etc/felhom-sshd-spike/port`. + +| Scenario | Result | +|---|---| +| clean | → **8822**, persisted | +| **contention** (a listener bound on 8822, persisted cleared) | → **skips to 2222** | +| **idempotent** (re-run while 8822 still busy) | → **stays 2222** (no thrash) | +| **exhaustion** (all four occupied) | → **rc=3, LOUD stderr** "refusing to fall back to :22 or a random port" — **never silently grabs :22** | +| recovery (all free) | → 8822 | + +The exhaustion-must-fail-loud property (a silent :22 grab would be the worst bug) is proven, not +assumed. + +**Algorithm (copy-paste, the shape the TASK's Go/agent version should mirror):** ordered candidate +list → for each, `ss -Htln "sport = :$p"` empty **and** an actual `bind()` succeeds → first free +wins → persist + emit; if none free → non-zero exit + explicit error, **no fallback**. Idempotency +gate reads the persisted port first and keeps it if still free. + +--- + +## 3. P1 — dedicated instance, minimal — GO + +Working `sshd_config` (`/etc/felhom-sshd-spike/sshd_config`), copy-paste ready: +``` +Port # from P2 +ListenAddress 0.0.0.0 +ListenAddress :: +HostKey /etc/felhom-sshd-spike/ssh_host_ed25519_key # DEDICATED key (ssh-keygen -t ed25519) +PidFile /run/felhom-sshd-spike.pid +AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u # OUTSIDE ~/.ssh — see P6 +PasswordAuthentication no +PermitRootLogin prohibit-password # key-only +PubkeyAuthentication yes +UsePAM yes # works on PVE with no extra PAM config (service name "sshd") +AllowUsers root felhom-op-spike # the operator identity/ies only +X11Forwarding no +Subsystem sftp internal-sftp +``` +The `sshd -t -f ` validation gate passed. **GO:** `ss -tlnp` showed felhom-sshd on 8822 +(PID 4095087) AND stock sshd on :22 (PID 922, UNCHANGED), both LISTEN; `Type=notify` sd_notify'd +cleanly (unit `active`, not just `activating`). + +**The `.service` — SAFE FINAL FORM (post-§8-incident):** +``` +[Unit] +Description=Felhom dedicated OOB sshd +After=network.target +[Service] +Type=notify +# NEVER "RuntimeDirectory=sshd" — systemd removes /run/sshd on stop, clobbering the SHARED +# privsep dir and breaking the stock sshd (§8). Ensure the dir exists idempotently instead: +ExecStartPre=-/usr/bin/mkdir -p /run/sshd +ExecStartPre=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config +ExecStart=/usr/sbin/sshd -D -f /etc/felhom-sshd-spike/sshd_config +ExecReload=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config +ExecReload=/bin/kill -HUP $MAINPID +KillMode=process +Restart=on-failure +[Install] +WantedBy=multi-user.target +``` + +--- + +## 4. P3 — customer-owned-Proxmox coexistence (the headline) — GO (four axes, evidenced) + +Stock :22 as the customer-sshd stand-in, felhom-sshd on 8822 simultaneously: + +| Axis | Evidence | +|---|---| +| **Both accept independently** | felhom-sshd key login (localhost:8822 → `whoami=root`); LAN :22 kept working (CC's own line) | +| **Distinct host keys** | stock ed25519 `SHA256:8Tea…` vs felhom `SHA256:EnC4…` — a client pinning one is never confused by the other (known_hosts hygiene for the operator) | +| **Reload stock → felhom unaffected** | `systemctl reload ssh` → felhom PID **4095087 unchanged**, still listening | +| **Restart felhom → stock unaffected** | `systemctl restart felhom-sshd-spike` → stock PID **922 unchanged**; pveproxy+pvedaemon stayed active; no port collision with 8006/8007/85/111/3128 | + +PVE (pveproxy :8006 + pvedaemon) undisturbed throughout; guest 9201 healthy. + +--- + +## 5. P4 — nft scoping to the tunnel — GO (LAN-block + :22-untouched evidenced; tunnel-accept by structure) + +Named table `felhom_sshd_spike`, retargeted from the prior spike's proven belt to the felhom port: +``` +table inet felhom_sshd_spike { + chain input { type filter hook input priority -5; policy accept; + iifname "wg-felhom" tcp dport 8822 ip saddr counter accept # tunnel, operator only + iifname "wg-felhom" tcp dport 8822 counter drop # tunnel, other src + tcp dport 8822 iifname != "wg-felhom" counter drop # OFF-tunnel: unreachable + } } +``` +- **LAN → 8822: BLOCKED** — a LAN `ssh -p 8822` from CC `Connection timed out`; the off-tunnel drop + counter incremented (3 pkts). Localhost → 8822 also blocked (lo ≠ wg-felhom), confirming + tunnel-only. +- **Stock :22 UNTOUCHED** — `nft list ruleset | grep "dport 22"` is EMPTY; LAN :22 kept working + (the safety line), by construction (the rules match `dport ` only, never 22). +- **Tunnel-accept path:** not re-exercised live here (the safety rail forbade re-establishing the + operator peer on `wg-felhom` in this spike); it is the **byte-identical belt** to + `SPIKE-oob-wg-operator-peer-2026-07-05.md` §5, which WAS live-tested against a real operator peer + (operator→box SSH at ~24 ms, dummy dropped). Structure + prior live proof = confident. + +--- + +## 6. P5 — robustness — GO (with a sharp restart-vs-reload finding) + +- **SAFE config-change path — `reload` survives a bad config:** appended a bogus directive → + `systemctl reload felhom-sshd-spike` **refused** (the `ExecReload=sshd -t` gate fails) → the + running instance **SURVIVED** (PID unchanged, still listening on 8822 with the last-good config). + This is the path the TASK must use for config changes. +- **UNSAFE path — `restart` is NOT atomic:** `systemctl restart` with a bad config takes the daemon + **DOWN** (systemd stops the old instance, then `ExecStartPre=sshd -t` fails so nothing comes back; + unit ends `failed`/`activating`, PID 0). A bad config therefore must be caught BEFORE a restart + (validate, then reload — never restart-on-change). +- **Blast-containment (post-fix):** even the unsafe `restart+badconfig` left stock :22 (PID 922) and + `/run/sshd` INTACT and LAN :22 working — because the SAFE unit (§3) has no `RuntimeDirectory=sshd`. + (Before the fix, this exact sequence removed `/run/sshd` and took :22 down — §8.) +- **Boot persistence:** `enabled`; host-key fingerprint **STABLE** (`SHA256:EnC4…`) across restart. +- **Late-network independence:** binds `0.0.0.0` (not a late-coming IP), `After=network.target` + only, **no `wg-felhom` dependency** — comes up regardless of tunnel state. +- **Start-limit gotcha (D1 echo):** repeated failed starts hit systemd's StartLimit → the unit + refuses further starts until `systemctl reset-failed`. The TASK's health/repair logic must + `reset-failed` before a deliberate restart (same lesson as the agent self-update spike). + +--- + +## 7. P6 — operator login identity — GO + the clean separation mechanism + +Throwaway `felhom-op-spike` (normal login user, `--create-home`, `/bin/bash`) with a scoped sudoers +(`/etc/sudoers.d/felhom-op-spike`: `mkdir -p /run/sshd`, `systemctl restart felhom-sshd-spike`, +`systemctl start felhom-agent` — the repair verbs): +- **Key login via felhom-sshd (8822) works** → `whoami=felhom-op-spike`, scoped `sudo -n` → `SUDO_OK`. +- **NOT reachable via stock :22** → `Permission denied (publickey)`. **Why it's clean:** the operator + key lives ONLY in felhom-sshd's `AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u`, + NOT in `~felhom-op-spike/.ssh/authorized_keys` (which doesn't exist). So the customer's sshd — + which reads `~/.ssh/authorized_keys` — never honours the operator key, **without us having to + touch the customer's `AllowUsers`/config at all.** This is the load-bearing separation primitive. +- `AllowUsers root felhom-op-spike` on felhom-sshd is the second gate (who may use 8822). + +**Operator-identity trade (record-only, for the TASK to decide):** +- **`felhom-op` (dedicated, sudo-scoped)** — clean least-privilege, auditable, isolated to + felhom-sshd via the AuthorizedKeysFile trick; needs a scoped sudoers for the repair verbs it will + run (agent restart, wg repair, `mkdir /run/sshd`, etc.). Doesn't collide with the customer's users. +- **`root` (PVE-native)** — deep host repairs (pct, systemctl, storage) need root anyway; PermitRootLogin + prohibit-password + the tunnel-only nft belt + the dedicated AuthorizedKeysFile still contain it; + simplest for "the operator must be able to fix anything". Trade: broader blast radius if the + operator key leaks (mitigated by the tunnel gate + key custody). +- Likely answer: **`felhom-op` with a tightly-scoped sudoers for the known repair actions, escalating + to root only via those sudo verbs** — but that's a TASK decision, recorded here, not made. + +--- + +## 8. THE INCIDENT — `RuntimeDirectory=sshd` clobbered the shared `/run/sshd` (caught, fixed, root-caused) + +**What happened.** The P1 unit was first written with `RuntimeDirectory=sshd`. During the P5 +bad-config test, `systemctl restart felhom-sshd-spike` failed at `ExecStartPre` (as designed) and the +unit went `failed`. systemd, cleaning up that unit's runtime directory, **removed `/run/sshd`** — the +compiled-in OpenSSH **privilege-separation directory**, which the **STOCK** sshd on :22 also depends +on. The stock listener (PID 922) stayed up and accepted TCP, but every new session **reset right +after `SSH2_MSG_KEXINIT`** — the textbook signature of a child that cannot initialise privsep. **CC's +LAN SSH — the safety line — went down.** + +**Why the "shared RuntimeDirectory is preserved" assumption was wrong.** Debian's stock `ssh.service` +also declares `RuntimeDirectory=sshd`; I expected systemd (v257) to keep `/run/sshd` alive while the +active stock unit referenced it. Empirically it did **not** — the failing unit's cleanup removed the +shared dir. (A full 135 s connection-drain ruled out `MaxStartups`; the cause was structural.) + +**Recovery (no shell available; the honest bit).** `:22` was the only shell path and it was the thing +broken; the felhom-sshd port was down (start-limited); the PVE API token cached on the build server +was **stale (401)**; no root password is documented (correctly). The customer **data plane was +unaffected** (`felhom.demo-felhom.eu` 200, pveproxy 200, agent alive) — a *management* lockout only. +Recovery was via the **PVE web console** (operator authenticated the session — CC is prohibited from +entering passwords — then CC drove the node xterm.js shell): `mkdir -p /run/sshd && chmod 0755 +/run/sshd` restored stock :22 **instantly, without even restarting the stock sshd** (PID stayed 922). +Confirmed live: `ls /run/sshd` → "No such file or directory" pre-fix; present post-fix. + +**The fix in the design (mandatory TASK constraint #1).** A second sshd instance must **NEVER** +declare `RuntimeDirectory=sshd`. Ensure `/run/sshd` exists idempotently instead — the spike used +`ExecStartPre=-/usr/bin/mkdir -p /run/sshd`; the production-clean equivalent is a +`tmpfiles.d` entry (`d /run/sshd 0755 root root -`) so the dir is boot-persistent and owned by +neither sshd unit's lifecycle. With the fixed unit, even `restart`+badconfig (§6) left `/run/sshd` +and :22 intact. + +**Secondary lessons from the incident:** (a) the build-server PVE token copy is **stale** — CC's only +non-SSH lever into the host is unusable; a working break-glass path (a valid console credential, or a +second admin key on a different port) should exist BEFORE this feature ships. (b) reset-after-KEXINIT +with the listener up = missing/again-broken `/run/sshd` — a fast diagnostic for the runbook. + +--- + +## 9. Findings that must shape the TASK + +1. **NEVER `RuntimeDirectory=sshd` in the felhom-sshd unit (§8).** It removes the shared `/run/sshd` + on stop/failure and takes the customer's sshd down. Ship a `tmpfiles.d` `/run/sshd` entry (or an + idempotent `ExecStartPre=mkdir`) instead. This is the #1 safety constraint and it is not obvious. +2. **Config changes go via `reload` (HUP) behind an `ExecReload=sshd -t` gate — never `restart` + (§6).** Reload keeps the running instance on the last-good config if the new one is bad; restart + is not atomic and downs the daemon on a failed pre-check. Validate, then reload. +3. **Identity isolation is the `AuthorizedKeysFile` location, not the customer's config (§7).** + Putting the operator key under `/etc/felhom-sshd/authorized_keys/%u` (outside `~/.ssh`) means the + customer's sshd never honours it and we never touch their config. This is the clean coexistence + primitive — carry it into the design. +4. **Port claim: probe-and-claim over an ordered candidate list, fail LOUD on exhaustion (§2).** + Never silently fall back to :22 or a random port. Persist the chosen port; be idempotent. The + shell PoC works; the TASK's agent-Go version should mirror the `ss`-empty **and** real-`bind()` + test. +5. **Ownership: agent-managed, mirroring the `wg-felhom`/`dnsmasq` pattern.** felhom-sshd is ongoing + reconciled desired-state (port may need re-claim, keys rotate, AllowUsers changes) — the agent's + job, not the one-shot host-install. Grant a NARROW sudoers slice (install the config + host key to + `/etc/felhom-sshd/*`, `sshd -t`, `systemctl {reload,restart,enable,disable} felhom-sshd`, the + `tmpfiles`/`mkdir /run/sshd`). `sshd -t` is the pre-apply gate the way `felhom-peersync` validates + before applying. Host-install can lay down the STATIC skeleton on a fresh box; the agent reconciles. +6. **Debian's `ssh@.service`/`ssh.socket` (socket-activation) is NOT a cleaner base than a bespoke + unit here (P0).** A per-connection socket-activated instance still needs `/run/sshd` and gives no + coexistence benefit; a dedicated always-on `felhom-sshd.service` with its own config is simpler + and is what the spike validated. Recommend the bespoke unit. +7. **PAM: `UsePAM yes` works on PVE with zero extra config** (the second sshd reuses the `sshd` PAM + service). No PVE-specific PAM work needed for key-only login. (If the TASK ever enables anything + PAM-gated like 2FA, revisit — out of scope here.) +8. **Fresh-host vs customer-host is a CONFIG VALUE, not a fork.** The same felhom-sshd instance model + works whether or not a stock sshd exists on :22 — on a fresh Felhom box it simply coexists with the + Proxmox-default sshd; on a customer box it coexists with theirs. The only variable is the CLAIMED + PORT (probe-and-claim handles both). No separate code path; the install just always claims a port + and never assumes :22 is Felhom's. +9. **Break-glass prerequisite (§8 secondary).** Before this ships, ensure a working out-of-band root + path independent of the felhom-sshd port itself (a valid PVE console credential and/or a distinct + recovery key), and keep the build-server PVE token fresh — the stale token left CC with no + non-SSH lever during the incident. + +--- + +## 10. Cleanup assertion (§13) — VERIFIED to the P0 baseline + +Removed exactly the spike set: `felhom-sshd-spike.service` (disable+rm+daemon-reload+reset-failed), +`/etc/felhom-sshd-spike/` (config + host key + authorized_keys), `/etc/sudoers.d/felhom-op-spike`, +the `felhom-op-spike` user + home (`userdel -r`), `/usr/local/sbin/felhom-sshd-spike-claim.sh`, +`/root/felhom-sshd-spike-client{,.pub}`, `/run/felhom-sshd-spike.pid`, `/run/felhom-claim-listener.pid`, +the `felhom_sshd_spike` nft table (both the box belt and the P4 table — removed). Re-verified: +stock `ssh.service` **active, PID 922** (same as P0 — never restarted; the /run/sshd fix alone +restored it), only `:22` listening, `sshd -T` port 22, `felhom-op-spike` gone, unit gone, +`/etc/felhom-sshd-spike` gone, `nft list tables` EMPTY (P0), PVE active, guest 9201 running. +**One intentional deviation from P0, permanent and correct:** `/run/sshd` exists (recreated during +§8 recovery) — it existed at P0 too; only its mtime differs. Leaving it is required. + +**No repo/hub/agent/manifest production change** — this commit is docs-only. + +--- + +## 11. Method-bar honored + +`wg show dump` never run; `wg-felhom`/PBS untouched. Every mutation a separately-named +unit/dir/file/nft-table, inventoried as created; P7 removed exactly that set. Coexistence and +negative claims carry evidence (PIDs, `ss`, nft counters, session survival, fingerprints), not +"seemed fine". The one surprise (the `/run/sshd` clobber) is recorded as the headline finding, not +massaged. Per-probe verdicts: P0 GATE-PASS, P1 GO, P2 GO, P3 GO, P4 GO, P5 GO (+restart/reload +finding), P6 GO, §8 SURPRISE→FIXED.