Validates a second sshd (own port/config/host-key/AuthorizedKeysFile/unit) as
Felhom's OOB entry point, coexisting with the customer's stock sshd on :22:
robust port probe-and-claim (skip/idempotent/loud-exhaustion), four-axis
coexistence independence, tunnel-scoped nft, reload-gate robustness, operator
identity isolated via AuthorizedKeysFile location, clean uninstall.
#1 TASK constraint (caught live): a second sshd unit must NEVER declare
RuntimeDirectory=sshd — it removes the SHARED /run/sshd privsep dir on stop and
takes the stock sshd down (LAN SSH lockout; recovered via PVE console + mkdir
/run/sshd). Use a tmpfiles.d entry instead. All spike artifacts removed; baseline
re-verified.
Docs-only; no code/hub/agent/manifest change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Validates operator-inbound access over the existing offsite WG arc (doc 06):
operator peer forwarded operator->box only, box sshd gated to the operator /32,
§4.5 box<->box isolation intact (both negatives counter-proven), mutual repair
real (agent self-healed a stopped tunnel in ~15s unaided). One TASK-shaping gap:
the operator /32 must be a RENDERED conf field — a runtime `wg set` is wiped by
the agent's own self-heal. All live-arc changes reverted + baseline re-verified.
Docs-only; no code/hub/agent/manifest change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
felhom-host-install.sh installs felhom-selfupdate-guarded (sh -n), the rollback
unit + the [Unit] start-limit drop-in (daemon-reload) so day-0 boxes get
operator-signed self-update from birth; non-fatal on pre-D1 agent repos.
03-host-agent.md §11 updated to the shipped mechanism (signed op, A/B wrapper,
OnFailure rollback, tuned start-limit backstop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
GO with two design-reshaping surprises: OnFailure= fires on EVERY crash on
systemd 257 (rollback triggers at first post-apply crash; marker-guard +
idempotency mandatory), and deliberate restarts consume start-limit budget.
S1 trap confirmed (default shape never reaches failed — 35 starts/180s);
tuned [Unit] StartLimitIntervalSec=120 + Burst=4 → failed in ~20s;
[Service] placement is an asymmetric half-applied trap; reset-failed required.
S2: all restart variants reliable, detached systemd-run wins. S3: A/B flip
atomic under kill -9, all verbs + guards pass. S4: StateDir + sudoers shapes OK.
Dummy artifacts only; real agent uptime unchanged (verified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Keep-side restore validated live: class-A DB replay, fail-closed data-key
gate + proven non-destruction, resettable-secret discrimination, Tier-2
class-C copy, and guardrails (traversal/missing/single-flight/restart).
Findings: F1 (HIGH) UI restore dead — /api/backup/snapshots has no handler;
F2 no one-click in-place class-C restore; F3 named-volume data never backed
up (DumpAppVolumes* has no caller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
escrow-create -storage felhom-offsite -identity-bundle -directive -upload on
felhom-pve: fresh R (operator-recorded off-transcript) wraps K + WG key + offsite
PBS token; uploaded to hub. Independently verified in host_escrow (demo-felhom-01:
key_fingerprint=offsite b0:fe:2a…, identity blob 499B, directive present,
created_at=now) with zero knowledge of R. S5 (DR consume) now UNBLOCKED.
This re-run also remediated an earlier escrow whose R had leaked into an agent
transcript (last-write-wins overwrote it with the clean-R blob).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Offsite restore-test now runs unattended on the agent-token path (agent v0.68.0):
tier-aware restore-task timeout (pbs→120m, local→10m). The scratch-band
VM.Allocate follow-up was a phantom — the scratch restores INTO /pool/felhom
(ACL already grants it); the earlier 403 was the 10m-timeout consequence. No
ACL/host-install change. doc-06 S4 row + runbook §4b updated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The CGNAT smoke test's OPEN DECISION is made: fleet-wide, permanent,
family-agnostic client MTU 1280 (RFC 8200 IPv6-minimum floor), implemented in
felhom-agent v0.65.0.
- §4.3: OPEN DECISION → made decision + rationale + rejected alternatives
(auto-probe/per-type = fragility for a non-metric; MSS-clamp = unneeded).
"ships MTU 1420" → 1280; corrected "fixed-line unaffected" (DS-Lite ~1452,
PPPoE would also black-hole at 1420 — one floor covers all).
- §4.2: 1280 is family-agnostic so MTU correctness is decoupled from v4/v6
resolution; v4-pin noted as a separate optional determinism item.
- §3.2 wire-field note + §8 S6 row: MTU 1420 → 1280; S6 residual now only the
true-CGNAT-SIM retest.
- REPORT.md overwritten.
Docs-only; the constant change is in felhom-agent v0.65.0.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
S3 appendix CGNAT smoke test run live on felhom-pve (agent v0.64.0) with a
USB-tethered phone. Tunnel established + held its NAT mapping across a 32-min
idle soak on PersistentKeepalive=25 alone (0 stalls, final ping no-bounce).
- §7: open-validation CLOSED; two honest caveats kept (SIM had a public mobile
IPv4 so true CGNAT 100.64/10 not reproduced — retest-when-able; + MTU).
- §4.3: mobile outer path MTU ~1400 black-holed bulk data at the shipped 1420;
MTU 1340 restored the PBS page. OPEN DECISION: permanent vs per-connection.
- §4.4: keepalive 32-min mobile-NAT hold datapoint.
- §4.2: dual-stack v6 trap (wg-quick prefers endpoint AAAA, rides un-NATed v6).
- §8: S6 row CGNAT closure marked done.
- REPORT.md: overwritten with this operational-validation summary.
Runbook: RUNBOOK-s3-cgnat-smoke. Docs-only; no agent/hub/controller code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
<(wg-quick strip ...) hid the strip exit code: a corrupt head file could feed
syncconf partial input that wipes the live peer set with exit 0 (S1 REPORT
finding). Strip now writes a temp file; its failure aborts before wg runs.
Sandbox red-proof: pre-fix shape invoked wg with rc=0 despite strip exit 1;
fixed shape never reaches wg.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Endpoint card + peers table (truncated pubkeys with full-value title attr,
bound peers link to /hosts/<id>); Offsite nav link in all 9 page templates;
render tests for endpoint/peers, empty, and not-configured states.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
POST /hosts/{id}/wg (per-host self-scoped; global = operator/DR path): bind /
re-key-in-place / adopt; generation bump + endpoint push ONLY on real change.
mergeWireguard injects the hub-owned block into served desired-state at READ
time (stored operator blob never touched; fail-safe unmerged on any error;
no-peer = byte-identical pass-through — existing golden test untouched+green).
handleAdminSetDesiredState rejects top-level wireguard (400). Admin DELETE of a
BOUND peer bumps the owning host. NEW golden desired-state-wireguard.golden.json
= the S3 cross-repo contract. Red-proofs a/b/c/d run + reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
All 8 steps run on the dev endpoint; live-run corrections folded in: enterprise-
repo removal after PBS install, proxmox-backup-client is a separate package,
throwaway-token namespace proof, wg-show-dump-leaks-the-private-key warning
(incident: first server key leaked to session log -> rotated on the spot),
AAAA-must-be-::1 + DNS-propagation-lag notes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live S1 validation caught it: a stock multi-hostkey sshd presented ECDSA while
we pin ed25519 → FixedHostKey refused a legitimate server. Regression test with
an in-process dual-hostkey server (fails without the fix — red-proofed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Peersync script: validate-first (jq contract check before ANY state change),
head-file + generated-peers conf model, syncconf-from-tmp then atomic mv (live
conf never diverges in the failure direction), zero-peer payload valid (wipe).
hub.yaml: 0.32.0 image + WG_ENDPOINT_SSH_* env + optional Secret mount so the
pod starts before the runbook's step-6 Secret exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
internal/wgsync: x/crypto/ssh client with ssh.FixedHostKey pin (no insecure
fallback), forced-command exec, ok/applied response contract; Reconciler pushes
the FULL peer list on Trigger or 5-min tick (drift repair by construction).
internal/api/wg.go: PUT/GET /admin/wg/endpoint + POST/DELETE/GET /admin/wg/peers,
global-key-only, pubkey in body (base64 vs URL), sync ok|deferred|disabled.
main.go: WG_ENDPOINT_SSH_* env wiring, disabled-with-INFO when unconfigured.
Groups B/C/D tests incl. in-process SSH server; red-proofs b/c/d run + reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Records the settled transport decisions (plain WG, host-side agent-managed pilot,
one hub-driven endpoint VM running WG+PBS, hub source-of-truth over WireDesiredState,
one datastore + per-customer namespaces, relay-through-DooPlex rejected). Grounded
at file:line vs felhom.eu@bf099f6 + felhom-agent@4ba1b14. Backlog line resolved to
cite spike + design doc. Docs-only; slices are separate future tasks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
P0-P7 probe ladder on real ends (demo-felhom PVE host <-> throwaway Hetzner).
Verdict: GO at H1 — plain host-side WG, box-dials-out to a small public endpoint;
Headscale is a separable fleet layer, not a traversal necessity. Line is plain-NAT
(not CGNAT) with zero IPv6 — recorded honestly; CGNAT vantage deferred to Peti VM 110.
Real 2 GiB worst-case PBS backup over the tunnel at the full home uplink, pin intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Companion to felhom-agent v0.62.0: the stale-lock reaper reads GET /pools/felhom
as its ownership registry. Pool.Allocate does NOT satisfy the read (spike T2).
Idempotent upgrade via --rescope-acl (_ensure_role modifies to the exact set).
Rescope FIRST, agent second. + REUSE §2 errata: gitea-creds is NOT out-of-band
(committed in felhom.secret.yaml, live-consumed — rotation pending).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live-probed on felhom-pve under the PRODUCTION scoped token vs root:
enumeration IS pool-filtered (T1: token sees [9201] of 4 guests); pool read
403s naming Pool.Audit (T2); Pool.Audit@/pool/felhom alone suffices (T3,
throwaway identity, torn down); /cluster/resources withholds the pool field
without Pool.Audit (T7); local ownership records all partial (T5).
Recommendation for the A1 impl spec (now unblocked): Pool.Audit added to
FelhomAgentGuest + GET /pools/felhom cross-check in staleLockController,
fail-safe skip on read failure. Appendix: committed-secrets rotation
micro-runbook (operator follow-up).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
- static/fonts/: the 4 vendored woff2 (byte-copied from
felhom-controller), embedded (embed.go) and served at /static/fonts/
(font/woff2, immutable) mirroring the chart.min.js pattern. No CDN
before, none now.
- templates/icons.html: 12-symbol Lucide sprite partial (icon_sprite),
included at the top of <body> on all 9 pages ({{template}} — the hub
has no shared layout; per-page include is the minimal shared block).
- statusColor now returns v2 semantic tokens (nominal/warn/crit/
neutral) consumed as class suffixes: ok->nominal, warn+stale->warn,
down+fail->crit, pending+disabled->neutral (a not-yet-provisioned or
deliberately paused customer is a normal fleet state), blocked->warn
(intentional operator cut-off, attention-worthy not an outage),
unknown->neutral. The inline style="color: {{statusColor}}" pattern
is dead: dashboard + customer_unified render a class-based
.status-dot-<token>; statusIcon (constant "●") retired from funcmap
and templates.
- Tests (new; the hub web package had no funcmap/template tests):
TestStatusColorTruthTable over the full enumerated status set —
red-proven vs the old implementation (ok returned "#4ade80") — and
TestTemplatesParseWithFuncmap.
admin