From 26664d6d51511f2a371539ecc2b7fe04060aa11a Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 5 Jul 2026 23:03:32 +0200 Subject: [PATCH] docs(H1): v0.72.0 CHANGELOG + REPORT + CONTEXT (OOB operator access, live-validated) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CHANGELOG.md | 30 ++++++++++++ CONTEXT.md | 9 ++++ REPORT.md | 130 ++++++++++++++++++++++++++------------------------- 3 files changed, 106 insertions(+), 63 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index caeb9c8..ea77fdf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,33 @@ +## v0.72.0 — OOB operator access: rendered operator /32 + dedicated felhom-sshd + port-adaptive belt + oob health (TASK H1) (2026-07-05) + +The agent half of the merged E1+H1 operator-SSH-access feature (hub half = felhom-hub v0.35.0). +Provenance: `felhom.eu/documentation/audits/SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05.md`. +Live-validated on felhom-pve + the dev endpoint (both spikes' key probes re-run as acceptance). + +- **Operator /32 rendered into wg-felhom** (`internal/wgtunnel/manager.go` `renderConf`/`allowedIPsLine`): + `oob_peer_ip` from the desired-state block is appended to AllowedIPs, deterministically SORTED + (byte-stable conf-hash — no per-tick flap). RENDERED, not a runtime `wg set`, so it survives the + agent's self-heal ([OF-1]; live-proven: tunnel stopped → self-heal → operator SSH still works). +- **Dedicated felhom-sshd** (`internal/felhomsshd/`): a SECOND sshd on a claimed non-22 port + (`[8822,2222,8022,62222]`, LOUD-fail on exhaustion [SF-4]), own config/host-key/AuthorizedKeysFile + (`/etc/felhom-sshd/authorized_keys/%u`, outside ~/.ssh [SF-3])/unit — COEXISTS with the customer's + :22 (never touched). Config: render→`sshd -t`→**reload** (never restart-on-change [SF-2]); operator + authorized_keys from the hub block; `reset-failed`-then-restart heal with a 10-min cooldown, NEVER + restarting onto an invalid config. `configs/felhom-sshd.service` SAFE — **no `RuntimeDirectory=`** + [SF-1]. +- **Port-adaptive belt** (`internal/felhomsshd/belt.go` + `configs/felhom-oob.nft`): a STATIC + `inet felhom_oob` table; the agent mutates ONLY its SETS — `@operator_ips` + `@ssh_port` [trap 4] — + so felhom-sshd's port is reachable ONLY from the operator `/32` over `wg-felhom` (off-tunnel + + box↔box dropped at the host; :22 untouched). Idempotent; a nil/unfetched block never empties it + (no operator lockout). +- **OOB health** (`internal/felhomsshd/health.go`): the additive `oob` heartbeat stanza + (`felhom_sshd_active/port/reachable/config_invalid/operator_peer_configured/operator_key_configured/ + wg_handshake_age_s`) — reaches the hub over HTTPS even with felhom-sshd/tunnel down. `reachable` = + a listener check (the belt blocks a dial); `operator_*_configured` from persistent state + (belt/authorized_keys), accurate immediately after a restart. +- **Sudoers**: `FELHOM_SSHD` (config/authkeys install + `sshd -t/-T` + scoped systemctl) + `FELHOM_OOB` + (nft SET-element ops only — never a rule grant). `oob.enabled` config DEFAULT FALSE. + ## v0.71.0 — management-plane break-glass: privsep-dir watchdog + mgmt_plane health (TASK G1) (2026-07-05) Prerequisite for the felhom-sshd OOB feature (H1). Closes the lockout from diff --git a/CONTEXT.md b/CONTEXT.md index 0916139..a1e87f6 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,15 @@ ## Current +- **v0.72.0** (2026-07-05) — **OOB operator access (merged E1+H1)** — TASK H1, provenance both + `SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`. Operator `/32` RENDERED into wg-felhom + AllowedIPs (survives self-heal, [OF-1]); dedicated `internal/felhomsshd` (port claim + config + render→sshd -t→reload + operator authorized_keys + heal + oob heartbeat stanza); static + `inet felhom_oob` belt (agent mutates SET ELEMENTS ONLY); `configs/felhom-sshd.service` (NO + RuntimeDirectory [SF-1]) + `felhom-oob.nft` + `felhom-op.sudoers`; `FELHOM_SSHD`+`FELHOM_OOB` + grants; `oob.enabled` DEFAULT FALSE. Live on felhom-pve (8822, belt filled, operator SSH as + felhom-op with scoped sudo); hub v0.35.0. Rollback `.bak-0.71.0`. 5 live-found bugs fixed (port + path, self-listen flip-flop, nil-block lockout, reachable-via-dial, operator-configured source). - **v0.71.0** (2026-07-05) — **management-plane break-glass: privsep-dir watchdog + mgmt_plane health** — TASK G1 (prereq for felhom-sshd/H1), provenance `SPIKE-felhom-sshd-2026-07-05` §8. Host artifacts (`configs/felhom-privsep.tmpfiles` + `felhom-mgmt-watchdog.{sh,service,timer}`) make diff --git a/REPORT.md b/REPORT.md index 0cfb8cc..d8e2dcd 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,78 +1,82 @@ -# REPORT — TASK G1: management-plane break-glass (felhom-agent half) — v0.71.0 +# REPORT — TASK H1: OOB operator access (felhom-agent half) — v0.72.0 -**Baseline:** felhom-agent `main` @ `1c75a45` (v0.70.0) → **v0.71.0** (commit `fd4e177`). felhom.eu @ -`2f97ce3`. Trunk-based, direct to `main`. +**Baseline:** felhom-agent `main` @ `4b7c5bf` (v0.71.0) → **v0.72.0** (final binary sha +`f72c2fed…`). felhom.eu @ `a3ee93e` → hub v0.35.0. Trunk-based, direct to `main`. Provenance: BOTH +`SPIKE-felhom-sshd-2026-07-05` [SF-n] and `SPIKE-oob-wg-operator-peer-2026-07-05` [OF-n]. ## What shipped (this repo) -The **prevent + auto-heal + observe** thirds of the break-glass system (the hub vault + surfacing are -the felhom.eu half, hub v0.34.1). Closes the lockout from -`felhom.eu/documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8: a second sshd's -`RuntimeDirectory=sshd` removed the SHARED `/run/sshd` privsep dir → the stock sshd on :22 reset every -session after `SSH2_MSG_KEXINIT`. +- **Operator `/32` rendered into `wg-felhom` AllowedIPs** (`internal/wgtunnel`): `oob_peer_ip` appended + sorted + byte-stable; RENDERED so it survives self-heal ([OF-1]). +- **Dedicated `felhom-sshd`** (`internal/felhomsshd`): claim (loud-fail exhaustion), render→`sshd -t`→ + reload (never restart-on-change [SF-2]), operator authorized_keys outside ~/.ssh [SF-3], heal with + cooldown + never-onto-invalid-config, `configs/felhom-sshd.service` with **no `RuntimeDirectory=`** + [SF-1]. +- **Port-adaptive belt** (`internal/felhomsshd/belt.go` + `configs/felhom-oob.nft`): static table, agent + mutates SET ELEMENTS ONLY (`@operator_ips`/`@ssh_port`) [trap 4]; a nil block never empties it. +- **OOB heartbeat stanza** + `FELHOM_SSHD`/`FELHOM_OOB` sudoers. `oob.enabled` DEFAULT FALSE. -- **Host artifacts** (`configs/`, installed by felhom-host-install): - - `felhom-privsep.tmpfiles` — `d /run/sshd 0755 root root -` (layer 1: boot-persistent, owned by no - unit's lifecycle). - - `felhom-mgmt-watchdog.sh` — the heal action: stat-first recreate `/run/sshd`, `reset-failed` the - stock sshd ONLY when `failed`, write an RFC3339 heal-marker; never restarts the stock sshd, never - touches a healthy dir. shellcheck-clean. - - `felhom-mgmt-watchdog.service` (oneshot) + `.timer` (OnUnitActiveSec=60s, Persistent). **No unit - declares `RuntimeDirectory=`** (the incident cause). -- **Go** (`internal/mgmtplane/`): a read-only `Reporter` (os.Stat `/run/sshd`, read the heal-marker, - TCP-dial sshd:22) → the additive `omitempty` `mgmt_plane` heartbeat stanza - (`privsep_dir_ok, sshd_reachable, healed_recently, privsep_healed_at`). Wired always-on via - `Collector.SetMgmtPlaneReporter` (report.go additive fields, the SelfUpdatePending precedent — no - hub-schema change; the golden contract test uses a nil reporter so the golden stays byte-identical). +## Tests + red-proofs (all green — `go build/vet/test ./...`, 23 packages ok) -## Tests + red-proofs +- renderConf: OOB /32 appended sorted + byte-stable; lower-IP-sorts-first; absent = byte-identical + golden; invalid/v6 rejected. Companion: byte-stability is the anti-flap red-proof. +- claim: clean/contention/idempotent/exhaustion (LOUD-fail is the key negative — non-zero, never :22); + persisted-port kept unconditionally (the self-listen flip-flop fix). +- config: SAFE template, byte-stable, refuses :22, never `RuntimeDirectory`. +- belt: mutate-then-idempotent (zero nft ops on the 2nd sync) + **never touches rules** (trap 4 + asserted); empty operator empties the set. +- heal: NO restart on an invalid config (red-proof) + 10-min cooldown; status reflects the block. +- hub collector + OOB monitor: stanza ingest + transition-based `oob_degraded`/`oob_recovered` (+ the + unconfigured-not-alerted and no-stanza-ignored negatives). Red-proof: neutering the emit fails it. -`go build ./... && go vet ./... && go test ./...` on the build server — **all green**. -- `internal/mgmtplane`: healthy (dir ok, no marker → healed_recently=false); missing dir detected - while sshd_reachable stays true (the "TCP up, sessions broken" trap); marker present → healed_recently - + timestamp; empty/whitespace marker treated as absent (red-proof for a hub warning with no - timestamp); sshd-unreachable reported. -- `internal/hub` collector: nil reporter → `mgmt_plane` omitted; wired → carried through. +## Live validation (felhom-pve + dev endpoint) — the spikes re-run -## Live validation (felhom-pve) — the acceptance +Deployed **manually** (agent self-update needs a pinned operator signer; the box has none — D1's +self-update path was not usable, stated honestly). Endpoint set up first (ip_forward=1 + +per-pair forward chain + operator peer via the hub); then agent v0.72.0 + `--enable-oob` artifacts + +`oob.enabled`. -Deployed **manually** (agent self-update needs an operator-signed op + a pinned signer; the box has -**no signer pinned**, so D1's self-update path was not usable — stated honestly). Build on 180 -(`-X main.version=0.71.0`, sha `b3b616fe…`) → scp via local → `install -m0755` (backup `.bak-0.70.0`) -→ restart. Host artifacts installed + timer enabled; `systemd-analyze verify` + `systemd-tmpfiles ---create` + shellcheck all clean; RuntimeDirectory guard verified (no directive, only the warning -comment). +- **Core** — operator→box SSH as **`felhom-op`** over the tunnel, with a scoped `sudo pct list`. ✓ +- **A [OF-1]** — stopped `wg-quick@wg-felhom` → agent self-healed (+48s) → box AllowedIPs STILL + `{10.77.0.1/32, 10.77.0.250/32}` (the /32 is rendered) → operator SSH survived. Conf-hash stable + (no per-tick flap). ✓ +- **B** — coexistence: stock :22 **PID 922 unchanged** across all felhom-sshd churn; both listening; + distinct host-key fingerprints (stock `8Tea…` vs felhom `EzuJ…`); **felhom-op DENIED on :22** + (`Permission denied (publickey)` — the AuthorizedKeysFile isolation); pveproxy active. ✓ +- **C** — port claim: felhom-sshd on 8822, belt `@ssh_port={8822}` follows; clean/contention/ + idempotent/exhaustion unit-proven (the flip-flop bug was found + fixed live: once claimed the port is + kept unconditionally, since felhom-sshd itself holds it). ✓ +- **D** — belt: LAN→8822 **dropped** (off-tunnel); LAN :22 untouched (safety line); PBS unaffected. ✓ +- **E** — heal: stopped felhom-sshd → agent restored it in ~15s. Config-invalid-no-restart unit-proven + (the agent always renders a valid config, so it self-corrects). ✓ +- **F** — endpoint: operator peer survives peersync; a dummy tunnel peer is **dropped box↔box** + (§4.5 drop counter=4); PBS healthy. ✓ +- **G** — hub: the healthy `oob` stanza `{active:true, port:8822, reachable:true, config_invalid:false, + operator_peer_configured:true, operator_key_configured:true, wg_handshake_age_s:24}` reaches the hub; + `oob_degraded` fired during the real felhom-sshd downtime (E) and `oob_recovered` on recovery. ✓ -**Auto-heal drill (agent STOPPED — the load-bearing proof):** `systemctl stop felhom-agent` → -`rm -rf /run/sshd` → a new `:22` session reproduced the incident (`Connection reset` right after the -banner) → the agent-independent watchdog **recreated `/run/sshd` and a new `:22` session succeeded -again in 30.0 s (measured), with the agent still `inactive`.** Marker written. This proves the heal is -login-free AND agent-independent. - -**End-to-end visibility chain (hub v0.34.1):** the v0.71.0 report carried -`mgmt_plane:{privsep_dir_ok:true, sshd_reachable:true, healed_recently:true, -privsep_healed_at:"2026-07-05T17:15:32Z"}`; the prior healthy report had `healed_recently:false`; the -old v0.70.0 report had no stanza (correctly absent). The hub monitor raised the -**`mgmt_plane_healed` warning** at 17:16:21 ("privsep dir … was missing and was AUTO-HEALED …"). - -**KEXINIT-reset reproduction:** confirmed live in the drill (`Connection reset by 127.0.0.1 port 22` -immediately after the banner exchange, `/run/sshd` absent). +**Bugs found + fixed during live validation** (each committed): (1) port persisted under root-owned +`/etc/felhom-sshd` → moved to the agent StateDir; (2) self-listen flip-flop (isFree(persisted) sees our +own daemon) → keep the persisted port unconditionally; (3) nil-block emptied the belt/authkeys +(operator lockout on restart) → fetched-gate like wgtunnel; (4) `reachable` via a dial always failed +(belt blocks localhost) → listener check; (5) `operator_*_configured` from persistent state (belt + +authorized_keys), not only the in-memory block. ## Cross-repo state -- Agent **v0.71.0** live on felhom-pve; three break-glass layers active + enabled (they STAY — this - is the shipped feature). Rollback: `.bak-0.70.0`. -- Hub **v0.34.1** live; break-glass vault + `mgmt_plane_healed` monitor. felhom.eu commits `05d8181` - (hub+installer), `28ac5f5` (runbook), `012e5f3` (monitor fix + v0.34.1). -- **H1 (felhom-sshd) can now assume `/run/sshd` is guaranteed present** (tmpfiles + watchdog) and must - never declare `RuntimeDirectory=` — the installer refuses any unit that does. +- Agent **v0.72.0** live on felhom-pve (sha `f72c2fed`); felhom-sshd active on 8822, belt filled, + operator `/32` rendered, oob stanza healthy. Rollback `.bak-0.71.0`. Endpoint + operator peer + + felhom-sshd all STAY (shipped). Stock :22 PID 922 unchanged throughout. +- Hub **v0.35.0** live. felhom.eu commits pushed; doc 06 §4.5/§4.6 amended. -## Notes / observations +## Observations not acted on (honest ledger) -- **Secret discipline verified:** the root@pam recovery password appears in NO log/commit (the vault - handler logs username + length only; red-proofed hub-side). -- **Least-privilege console user** (vs root@pam): not probed live (PVE gates the node Shell to - root@pam historically); G1 ships the proven root@pam path — a non-blocking future item. -- **Auto-rotation** of the recovery credential is a noted future item; `--rotate-recovery` re-vaults on - demand. -- Heal cadence: OnUnitActiveSec=60s (measured 30 s this run, mid-cycle); worst case ~60 s. +- **CGNAT** OOB traversal still unproven (the box is single-NAT public-v4, as in both spikes). +- **IPv6/AAAA** out of scope (arc v4-pinned); the standing ep0-AAAA item persists. +- **Operator-key auto-rotation** not implemented (re-`PUT /admin/wg/operator-peer` rotates manually). +- **"Customer network fully down (powered)"** — explicitly OUT OF SCOPE / accepted risk (operator + decision 2026-07-05); the smart-plug is the only mitigation and is not part of this task. +- **`operator_peer_configured` post-restart lag** eliminated by reading the persistent belt (not the + 0600 root-owned wg conf the non-root agent can't read). +- Agent binary **not yet published** to Gitea (manual deploy for validation); a fleet rollout should + `scripts/publish-agent.sh 0.72.0 ` + Day-0 vouch.