docs(audit): dedicated felhom-sshd instance spike (2026-07-05) — GO

Validates a second sshd (own port/config/host-key/AuthorizedKeysFile/unit) as
Felhom's OOB entry point, coexisting with the customer's stock sshd on :22:
robust port probe-and-claim (skip/idempotent/loud-exhaustion), four-axis
coexistence independence, tunnel-scoped nft, reload-gate robustness, operator
identity isolated via AuthorizedKeysFile location, clean uninstall.

#1 TASK constraint (caught live): a second sshd unit must NEVER declare
RuntimeDirectory=sshd — it removes the SHARED /run/sshd privsep dir on stop and
takes the stock sshd down (LAN SSH lockout; recovered via PVE console + mkdir
/run/sshd). Use a tmpfiles.d entry instead. All spike artifacts removed; baseline
re-verified.

Docs-only; no code/hub/agent/manifest change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-05 18:18:37 +02:00
parent 2db92c8837
commit 2f97ce31dd
@@ -0,0 +1,315 @@
# SPIKE — dedicated `felhom-sshd` instance (own port/config/keys, customer-host coexistence) — 2026-07-05
> **STATUS: COMPLETE — verdict below, with one CAUGHT-AND-FIXED incident (§8) that is itself the
> single most important finding.** Validates a dedicated second sshd instance as Felhom's OOB entry
> point on the demo host `felhom-pve` (node `demo-felhom`, PVE 9.2.2, OpenSSH 10.0p2). All spike
> artifacts removed at cleanup (§13, re-verified to the P0 baseline). No production code shipped.
**Class:** SPIKE (empirical; no product code). **Repos:** felhom.eu (this doc only); felhom-agent +
felhom.eu/scripts read-only for grounding. **Host:** `felhom-pve` (`192.168.0.162`). The offsite
`wg-felhom` tunnel + PBS path were NOT touched (OOB transport was proven separately in
`SPIKE-oob-wg-operator-peer-2026-07-05.md`).
**Verdict (one line):** **GO — a dedicated `felhom-sshd` on a claimed non-22 port, with its own
config/host-key/AuthorizedKeysFile/unit, coexists cleanly with the stock (customer) sshd on :22
(independence proven on all four axes), claims its port robustly (skip/idempotent/loud-exhaustion),
scopes to the tunnel by nft, and uninstalls to a byte-clean baseline — PROVIDED the unit NEVER
declares `RuntimeDirectory=sshd` (which removes the SHARED `/run/sshd` privsep dir on stop and takes
the stock sshd down with it; this spike hit that live, §8, and it is the #1 TASK constraint).**
---
## 0. What the demo box could and could not represent
The box is **Felhom-owned** and single-NAT; the stock `ssh.service` on :22 stood in for "the
customer's existing sshd". Faithfully represented: two independent sshd instances on one host, port
contention, nft scoping, the systemd/privsep mechanics, uninstall removability. NOT represented: a
**real customer's** sshd config (they may run `AllowUsers`, non-default ports, `Match` blocks,
`fail2ban`/`sshguard`, a hardened `sshd_config` that our drop-in model doesn't inherit — we ship a
SEPARATE config so we don't inherit theirs, which is the point), their actual **port map** (our
candidate list must still probe-and-claim against whatever they run), and true **CGNAT**. The
privsep-dir incident (§8) is host-OS-generic (Debian/PVE), so it applies to a customer host
identically.
---
## 1. P0 — ground truth (no changes) — GATE PASSED
```
stock sshd unit : ssh.service (Debian; sshd.service/sshd@.service are ALIASES/indirect)
MainPID 922, FragmentPath /usr/lib/systemd/system/ssh.service
effective config : port 22 ; ListenAddress 0.0.0.0:22 + [::]:22 ; UsePAM yes ;
PermitRootLogin yes ; PasswordAuthentication yes (Proxmox defaults) ;
AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 ;
HostKey rsa+ecdsa+ed25519 ; OpenSSH_10.0p2 Debian-7+deb13u4
instance template: /lib/systemd/system/ssh@.service ABSENT at that path; BUT `sshd@.service`
(indirect) + `ssh.socket` (disabled) exist — socket-activation route is available.
listeners (ss -tlnp): 22(sshd) 111(rpcbind) 85(pvedaemon,localhost) 25(postfix,localhost)
53(dnsmasq) 8443(felhom-agent) 3128(spiceproxy) 8006(pveproxy)
→ candidate ports 8822/2222/8022/62222 ALL FREE.
pve-firewall : disabled ; nft list tables: EMPTY (baseline)
PVE health : pveproxy + pvedaemon active
```
**Gate:** stock sshd identified (`ssh.service`/PID 922/:22) and never touched thereafter.
---
## 2. P2 — port probe-then-claim under contention — GO (all four behaviours)
Claim routine (`/usr/local/sbin/felhom-sshd-spike-claim.sh`, throwaway) — candidate list
**`8822 2222 8022 62222`**; `is_free()` = nothing in `ss -Htln sport=:N` **AND** a real `python3`
bind succeeds; idempotent re-pick of the persisted port if still free; persists to
`/etc/felhom-sshd-spike/port`.
| Scenario | Result |
|---|---|
| clean | → **8822**, persisted |
| **contention** (a listener bound on 8822, persisted cleared) | → **skips to 2222** |
| **idempotent** (re-run while 8822 still busy) | → **stays 2222** (no thrash) |
| **exhaustion** (all four occupied) | → **rc=3, LOUD stderr** "refusing to fall back to :22 or a random port" — **never silently grabs :22** |
| recovery (all free) | → 8822 |
The exhaustion-must-fail-loud property (a silent :22 grab would be the worst bug) is proven, not
assumed.
**Algorithm (copy-paste, the shape the TASK's Go/agent version should mirror):** ordered candidate
list → for each, `ss -Htln "sport = :$p"` empty **and** an actual `bind()` succeeds → first free
wins → persist + emit; if none free → non-zero exit + explicit error, **no fallback**. Idempotency
gate reads the persisted port first and keeps it if still free.
---
## 3. P1 — dedicated instance, minimal — GO
Working `sshd_config` (`/etc/felhom-sshd-spike/sshd_config`), copy-paste ready:
```
Port <claimed> # from P2
ListenAddress 0.0.0.0
ListenAddress ::
HostKey /etc/felhom-sshd-spike/ssh_host_ed25519_key # DEDICATED key (ssh-keygen -t ed25519)
PidFile /run/felhom-sshd-spike.pid
AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u # OUTSIDE ~/.ssh — see P6
PasswordAuthentication no
PermitRootLogin prohibit-password # key-only
PubkeyAuthentication yes
UsePAM yes # works on PVE with no extra PAM config (service name "sshd")
AllowUsers root felhom-op-spike # the operator identity/ies only
X11Forwarding no
Subsystem sftp internal-sftp
```
The `sshd -t -f <conf>` validation gate passed. **GO:** `ss -tlnp` showed felhom-sshd on 8822
(PID 4095087) AND stock sshd on :22 (PID 922, UNCHANGED), both LISTEN; `Type=notify` sd_notify'd
cleanly (unit `active`, not just `activating`).
**The `.service` — SAFE FINAL FORM (post-§8-incident):**
```
[Unit]
Description=Felhom dedicated OOB sshd
After=network.target
[Service]
Type=notify
# NEVER "RuntimeDirectory=sshd" — systemd removes /run/sshd on stop, clobbering the SHARED
# privsep dir and breaking the stock sshd (§8). Ensure the dir exists idempotently instead:
ExecStartPre=-/usr/bin/mkdir -p /run/sshd
ExecStartPre=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config
ExecStart=/usr/sbin/sshd -D -f /etc/felhom-sshd-spike/sshd_config
ExecReload=/usr/sbin/sshd -t -f /etc/felhom-sshd-spike/sshd_config
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
```
---
## 4. P3 — customer-owned-Proxmox coexistence (the headline) — GO (four axes, evidenced)
Stock :22 as the customer-sshd stand-in, felhom-sshd on 8822 simultaneously:
| Axis | Evidence |
|---|---|
| **Both accept independently** | felhom-sshd key login (localhost:8822 → `whoami=root`); LAN :22 kept working (CC's own line) |
| **Distinct host keys** | stock ed25519 `SHA256:8Tea…` vs felhom `SHA256:EnC4…` — a client pinning one is never confused by the other (known_hosts hygiene for the operator) |
| **Reload stock → felhom unaffected** | `systemctl reload ssh` → felhom PID **4095087 unchanged**, still listening |
| **Restart felhom → stock unaffected** | `systemctl restart felhom-sshd-spike` → stock PID **922 unchanged**; pveproxy+pvedaemon stayed active; no port collision with 8006/8007/85/111/3128 |
PVE (pveproxy :8006 + pvedaemon) undisturbed throughout; guest 9201 healthy.
---
## 5. P4 — nft scoping to the tunnel — GO (LAN-block + :22-untouched evidenced; tunnel-accept by structure)
Named table `felhom_sshd_spike`, retargeted from the prior spike's proven belt to the felhom port:
```
table inet felhom_sshd_spike {
chain input { type filter hook input priority -5; policy accept;
iifname "wg-felhom" tcp dport 8822 ip saddr <operator/32> counter accept # tunnel, operator only
iifname "wg-felhom" tcp dport 8822 counter drop # tunnel, other src
tcp dport 8822 iifname != "wg-felhom" counter drop # OFF-tunnel: unreachable
} }
```
- **LAN → 8822: BLOCKED** — a LAN `ssh -p 8822` from CC `Connection timed out`; the off-tunnel drop
counter incremented (3 pkts). Localhost → 8822 also blocked (lo ≠ wg-felhom), confirming
tunnel-only.
- **Stock :22 UNTOUCHED** — `nft list ruleset | grep "dport 22"` is EMPTY; LAN :22 kept working
(the safety line), by construction (the rules match `dport <felhom-port>` only, never 22).
- **Tunnel-accept path:** not re-exercised live here (the safety rail forbade re-establishing the
operator peer on `wg-felhom` in this spike); it is the **byte-identical belt** to
`SPIKE-oob-wg-operator-peer-2026-07-05.md` §5, which WAS live-tested against a real operator peer
(operator→box SSH at ~24 ms, dummy dropped). Structure + prior live proof = confident.
---
## 6. P5 — robustness — GO (with a sharp restart-vs-reload finding)
- **SAFE config-change path — `reload` survives a bad config:** appended a bogus directive →
`systemctl reload felhom-sshd-spike` **refused** (the `ExecReload=sshd -t` gate fails) → the
running instance **SURVIVED** (PID unchanged, still listening on 8822 with the last-good config).
This is the path the TASK must use for config changes.
- **UNSAFE path — `restart` is NOT atomic:** `systemctl restart` with a bad config takes the daemon
**DOWN** (systemd stops the old instance, then `ExecStartPre=sshd -t` fails so nothing comes back;
unit ends `failed`/`activating`, PID 0). A bad config therefore must be caught BEFORE a restart
(validate, then reload — never restart-on-change).
- **Blast-containment (post-fix):** even the unsafe `restart+badconfig` left stock :22 (PID 922) and
`/run/sshd` INTACT and LAN :22 working — because the SAFE unit (§3) has no `RuntimeDirectory=sshd`.
(Before the fix, this exact sequence removed `/run/sshd` and took :22 down — §8.)
- **Boot persistence:** `enabled`; host-key fingerprint **STABLE** (`SHA256:EnC4…`) across restart.
- **Late-network independence:** binds `0.0.0.0` (not a late-coming IP), `After=network.target`
only, **no `wg-felhom` dependency** — comes up regardless of tunnel state.
- **Start-limit gotcha (D1 echo):** repeated failed starts hit systemd's StartLimit → the unit
refuses further starts until `systemctl reset-failed`. The TASK's health/repair logic must
`reset-failed` before a deliberate restart (same lesson as the agent self-update spike).
---
## 7. P6 — operator login identity — GO + the clean separation mechanism
Throwaway `felhom-op-spike` (normal login user, `--create-home`, `/bin/bash`) with a scoped sudoers
(`/etc/sudoers.d/felhom-op-spike`: `mkdir -p /run/sshd`, `systemctl restart felhom-sshd-spike`,
`systemctl start felhom-agent` — the repair verbs):
- **Key login via felhom-sshd (8822) works** → `whoami=felhom-op-spike`, scoped `sudo -n``SUDO_OK`.
- **NOT reachable via stock :22** → `Permission denied (publickey)`. **Why it's clean:** the operator
key lives ONLY in felhom-sshd's `AuthorizedKeysFile /etc/felhom-sshd-spike/authorized_keys/%u`,
NOT in `~felhom-op-spike/.ssh/authorized_keys` (which doesn't exist). So the customer's sshd —
which reads `~/.ssh/authorized_keys` — never honours the operator key, **without us having to
touch the customer's `AllowUsers`/config at all.** This is the load-bearing separation primitive.
- `AllowUsers root felhom-op-spike` on felhom-sshd is the second gate (who may use 8822).
**Operator-identity trade (record-only, for the TASK to decide):**
- **`felhom-op` (dedicated, sudo-scoped)** — clean least-privilege, auditable, isolated to
felhom-sshd via the AuthorizedKeysFile trick; needs a scoped sudoers for the repair verbs it will
run (agent restart, wg repair, `mkdir /run/sshd`, etc.). Doesn't collide with the customer's users.
- **`root` (PVE-native)** — deep host repairs (pct, systemctl, storage) need root anyway; PermitRootLogin
prohibit-password + the tunnel-only nft belt + the dedicated AuthorizedKeysFile still contain it;
simplest for "the operator must be able to fix anything". Trade: broader blast radius if the
operator key leaks (mitigated by the tunnel gate + key custody).
- Likely answer: **`felhom-op` with a tightly-scoped sudoers for the known repair actions, escalating
to root only via those sudo verbs** — but that's a TASK decision, recorded here, not made.
---
## 8. THE INCIDENT — `RuntimeDirectory=sshd` clobbered the shared `/run/sshd` (caught, fixed, root-caused)
**What happened.** The P1 unit was first written with `RuntimeDirectory=sshd`. During the P5
bad-config test, `systemctl restart felhom-sshd-spike` failed at `ExecStartPre` (as designed) and the
unit went `failed`. systemd, cleaning up that unit's runtime directory, **removed `/run/sshd`** — the
compiled-in OpenSSH **privilege-separation directory**, which the **STOCK** sshd on :22 also depends
on. The stock listener (PID 922) stayed up and accepted TCP, but every new session **reset right
after `SSH2_MSG_KEXINIT`** — the textbook signature of a child that cannot initialise privsep. **CC's
LAN SSH — the safety line — went down.**
**Why the "shared RuntimeDirectory is preserved" assumption was wrong.** Debian's stock `ssh.service`
also declares `RuntimeDirectory=sshd`; I expected systemd (v257) to keep `/run/sshd` alive while the
active stock unit referenced it. Empirically it did **not** — the failing unit's cleanup removed the
shared dir. (A full 135 s connection-drain ruled out `MaxStartups`; the cause was structural.)
**Recovery (no shell available; the honest bit).** `:22` was the only shell path and it was the thing
broken; the felhom-sshd port was down (start-limited); the PVE API token cached on the build server
was **stale (401)**; no root password is documented (correctly). The customer **data plane was
unaffected** (`felhom.demo-felhom.eu` 200, pveproxy 200, agent alive) — a *management* lockout only.
Recovery was via the **PVE web console** (operator authenticated the session — CC is prohibited from
entering passwords — then CC drove the node xterm.js shell): `mkdir -p /run/sshd && chmod 0755
/run/sshd` restored stock :22 **instantly, without even restarting the stock sshd** (PID stayed 922).
Confirmed live: `ls /run/sshd` → "No such file or directory" pre-fix; present post-fix.
**The fix in the design (mandatory TASK constraint #1).** A second sshd instance must **NEVER**
declare `RuntimeDirectory=sshd`. Ensure `/run/sshd` exists idempotently instead — the spike used
`ExecStartPre=-/usr/bin/mkdir -p /run/sshd`; the production-clean equivalent is a
`tmpfiles.d` entry (`d /run/sshd 0755 root root -`) so the dir is boot-persistent and owned by
neither sshd unit's lifecycle. With the fixed unit, even `restart`+badconfig (§6) left `/run/sshd`
and :22 intact.
**Secondary lessons from the incident:** (a) the build-server PVE token copy is **stale** — CC's only
non-SSH lever into the host is unusable; a working break-glass path (a valid console credential, or a
second admin key on a different port) should exist BEFORE this feature ships. (b) reset-after-KEXINIT
with the listener up = missing/again-broken `/run/sshd` — a fast diagnostic for the runbook.
---
## 9. Findings that must shape the TASK
1. **NEVER `RuntimeDirectory=sshd` in the felhom-sshd unit (§8).** It removes the shared `/run/sshd`
on stop/failure and takes the customer's sshd down. Ship a `tmpfiles.d` `/run/sshd` entry (or an
idempotent `ExecStartPre=mkdir`) instead. This is the #1 safety constraint and it is not obvious.
2. **Config changes go via `reload` (HUP) behind an `ExecReload=sshd -t` gate — never `restart`
(§6).** Reload keeps the running instance on the last-good config if the new one is bad; restart
is not atomic and downs the daemon on a failed pre-check. Validate, then reload.
3. **Identity isolation is the `AuthorizedKeysFile` location, not the customer's config (§7).**
Putting the operator key under `/etc/felhom-sshd/authorized_keys/%u` (outside `~/.ssh`) means the
customer's sshd never honours it and we never touch their config. This is the clean coexistence
primitive — carry it into the design.
4. **Port claim: probe-and-claim over an ordered candidate list, fail LOUD on exhaustion (§2).**
Never silently fall back to :22 or a random port. Persist the chosen port; be idempotent. The
shell PoC works; the TASK's agent-Go version should mirror the `ss`-empty **and** real-`bind()`
test.
5. **Ownership: agent-managed, mirroring the `wg-felhom`/`dnsmasq` pattern.** felhom-sshd is ongoing
reconciled desired-state (port may need re-claim, keys rotate, AllowUsers changes) — the agent's
job, not the one-shot host-install. Grant a NARROW sudoers slice (install the config + host key to
`/etc/felhom-sshd/*`, `sshd -t`, `systemctl {reload,restart,enable,disable} felhom-sshd`, the
`tmpfiles`/`mkdir /run/sshd`). `sshd -t` is the pre-apply gate the way `felhom-peersync` validates
before applying. Host-install can lay down the STATIC skeleton on a fresh box; the agent reconciles.
6. **Debian's `ssh@.service`/`ssh.socket` (socket-activation) is NOT a cleaner base than a bespoke
unit here (P0).** A per-connection socket-activated instance still needs `/run/sshd` and gives no
coexistence benefit; a dedicated always-on `felhom-sshd.service` with its own config is simpler
and is what the spike validated. Recommend the bespoke unit.
7. **PAM: `UsePAM yes` works on PVE with zero extra config** (the second sshd reuses the `sshd` PAM
service). No PVE-specific PAM work needed for key-only login. (If the TASK ever enables anything
PAM-gated like 2FA, revisit — out of scope here.)
8. **Fresh-host vs customer-host is a CONFIG VALUE, not a fork.** The same felhom-sshd instance model
works whether or not a stock sshd exists on :22 — on a fresh Felhom box it simply coexists with the
Proxmox-default sshd; on a customer box it coexists with theirs. The only variable is the CLAIMED
PORT (probe-and-claim handles both). No separate code path; the install just always claims a port
and never assumes :22 is Felhom's.
9. **Break-glass prerequisite (§8 secondary).** Before this ships, ensure a working out-of-band root
path independent of the felhom-sshd port itself (a valid PVE console credential and/or a distinct
recovery key), and keep the build-server PVE token fresh — the stale token left CC with no
non-SSH lever during the incident.
---
## 10. Cleanup assertion (§13) — VERIFIED to the P0 baseline
Removed exactly the spike set: `felhom-sshd-spike.service` (disable+rm+daemon-reload+reset-failed),
`/etc/felhom-sshd-spike/` (config + host key + authorized_keys), `/etc/sudoers.d/felhom-op-spike`,
the `felhom-op-spike` user + home (`userdel -r`), `/usr/local/sbin/felhom-sshd-spike-claim.sh`,
`/root/felhom-sshd-spike-client{,.pub}`, `/run/felhom-sshd-spike.pid`, `/run/felhom-claim-listener.pid`,
the `felhom_sshd_spike` nft table (both the box belt and the P4 table — removed). Re-verified:
stock `ssh.service` **active, PID 922** (same as P0 — never restarted; the /run/sshd fix alone
restored it), only `:22` listening, `sshd -T` port 22, `felhom-op-spike` gone, unit gone,
`/etc/felhom-sshd-spike` gone, `nft list tables` EMPTY (P0), PVE active, guest 9201 running.
**One intentional deviation from P0, permanent and correct:** `/run/sshd` exists (recreated during
§8 recovery) — it existed at P0 too; only its mtime differs. Leaving it is required.
**No repo/hub/agent/manifest production change** — this commit is docs-only.
---
## 11. Method-bar honored
`wg show dump` never run; `wg-felhom`/PBS untouched. Every mutation a separately-named
unit/dir/file/nft-table, inventoried as created; P7 removed exactly that set. Coexistence and
negative claims carry evidence (PIDs, `ss`, nft counters, session survival, fingerprints), not
"seemed fine". The one surprise (the `/run/sshd` clobber) is recorded as the headline finding, not
massaged. Per-probe verdicts: P0 GATE-PASS, P1 GO, P2 GO, P3 GO, P4 GO, P5 GO (+restart/reload
finding), P6 GO, §8 SURPRISE→FIXED.