F4: ReissueCredentials — explicit operator recovery for consumed-password
dead-ends; resets the labelled resource's password (exactly-1 guard,
red-proofed), stores a fresh one-time secret, bumps ConfigVersion.
New hetznerapi.ResetBoxPassword for the dedicated path.
F2: host-key scan retry-with-backoff (~60s ladder, red-proofed) — first
save survives fresh-subaccount DNS lag.
F5: config form disables submits + shows an in-flight notice (the re-click
bait that caused live F1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
First supervised live run: the ~25s spinner-less offsite save invited a
re-click; the abandoned first request's r.Context() was canceled between
CreateSubaccount and SaveOneTimeSecret, stranding sub-account 268985 with a
password lost forever (consume 404s permanently).
applyOffsite now provisions on context.WithoutCancel + 3-minute absolute
timeout: once the create starts, create->wait->store runs to completion.
Regression test with a ctx-honoring fake that cancels the request context
mid-create; red-proofed against the raw-ctx pre-fix shape (reproduces the
exact live error).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Descriptor.HostFingerprint (SHA256, non-secret), captured at provision via an
x/crypto/ssh keyscan (SSHHostKeyScanner — dials :23, grabs the host key from the
handshake, no ssh binary). Fail-closed: nil scanner or scan failure → error (don't
serve a descriptor the controller can't verify). Pairs with controller v0.106.0
which re-scans + refuses on mismatch (no blind TOFU).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Replaces the static Option-1/2 Setup Command blocks with an interactive,
client-side builder: mode (required radio), cores/memory (required for byo,
optional for appliance), vmid, node, acl-storages (quote-wrapped),
operator-pubkey-file, preserve-state-from, and --dry-run/--preflight-only/
--skip-provision/--allow-new-leaf checkboxes. genFlags()/genUpdate() assemble
a live-updating download-then-run command (never curl|bash) + a local-run
variant, enforcing the script's own rules client-side (mode required; byo
requires caps → shows a warning + no runnable command; appliance hides the
caps requirement; allow-new-leaf shows its leaf-regen warning). Emits ONLY
real host-install v1.12.0 flags; the dangerous/operator-only set
(--force/--rotate-recovery/--enable-oob/--remove-golden/--uninstall/
--adopt-pool/--rescope-acl) is never offered. Graceful static fallback:
the server-rendered Option-1/2 commands keep --customer-id + a --mode
placeholder when JS is off. No framework/CDN/network; ScriptVersion (const,
in sync with SCRIPT_VERSION) drives the header. Render/structure test covers
the control ids, version, fallback, and the excluded-flag absence.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The Setup Command panel rendered the per-customer retrieval passphrase (the
secret that fetches the WHOLE config) in cleartext twice — as #retrieval-pw
text and baked into the Option-3 curl's X-Retrieval-Password header — which
contradicts the panel's own "never on the command line" guidance. Now:
the retrieval password is MASKED by default (bullet run) with Reveal/Hide +
copy-secret controls (value lives in data-secret — the existing reveal
model); the Option-3 debug command carries a <YOUR-RETRIEVAL-PASSWORD>
placeholder, never the secret. Render test asserts the secret is not baked
into any command + is masked by default; red-proof (bake it back) FAILS.
(A zero-secret-in-DOM reveal-on-demand fetch is a noted follow-up, not this task.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
store.GetHostOOBStates parses the agent oob heartbeat stanza. monitor/host_oob:
transition-based oob_degraded/oob_recovered warning (felhom-sshd down while the
operator peer is configured, OR config invalid) — proactive "can the operator get
in right now" signal; unconfigured OOB never alerts. Wired into the 60s sweep.
Non-hollow tests + transitions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
A heal marker is an event, not a baseline: construction seeds pre-existing markers
(startup false-alarm guard) but a newly-observed marker now raises the warning, so
the first auto-heal surfaces (matches the live drill). Added tests for both halves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Endpoint card + peers table (truncated pubkeys with full-value title attr,
bound peers link to /hosts/<id>); Offsite nav link in all 9 page templates;
render tests for endpoint/peers, empty, and not-configured states.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
POST /hosts/{id}/wg (per-host self-scoped; global = operator/DR path): bind /
re-key-in-place / adopt; generation bump + endpoint push ONLY on real change.
mergeWireguard injects the hub-owned block into served desired-state at READ
time (stored operator blob never touched; fail-safe unmerged on any error;
no-peer = byte-identical pass-through — existing golden test untouched+green).
handleAdminSetDesiredState rejects top-level wireguard (400). Admin DELETE of a
BOUND peer bumps the owning host. NEW golden desired-state-wireguard.golden.json
= the S3 cross-repo contract. Red-proofs a/b/c/d run + reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live S1 validation caught it: a stock multi-hostkey sshd presented ECDSA while
we pin ed25519 → FixedHostKey refused a legitimate server. Regression test with
an in-process dual-hostkey server (fails without the fix — red-proofed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
internal/wgsync: x/crypto/ssh client with ssh.FixedHostKey pin (no insecure
fallback), forced-command exec, ok/applied response contract; Reconciler pushes
the FULL peer list on Trigger or 5-min tick (drift repair by construction).
internal/api/wg.go: PUT/GET /admin/wg/endpoint + POST/DELETE/GET /admin/wg/peers,
global-key-only, pubkey in body (base64 vs URL), sync ok|deferred|disabled.
main.go: WG_ENDPOINT_SSH_* env wiring, disabled-with-INFO when unconfigured.
Groups B/C/D tests incl. in-process SSH server; red-proofs b/c/d run + reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
- static/fonts/: the 4 vendored woff2 (byte-copied from
felhom-controller), embedded (embed.go) and served at /static/fonts/
(font/woff2, immutable) mirroring the chart.min.js pattern. No CDN
before, none now.
- templates/icons.html: 12-symbol Lucide sprite partial (icon_sprite),
included at the top of <body> on all 9 pages ({{template}} — the hub
has no shared layout; per-page include is the minimal shared block).
- statusColor now returns v2 semantic tokens (nominal/warn/crit/
neutral) consumed as class suffixes: ok->nominal, warn+stale->warn,
down+fail->crit, pending+disabled->neutral (a not-yet-provisioned or
deliberately paused customer is a normal fleet state), blocked->warn
(intentional operator cut-off, attention-worthy not an outage),
unknown->neutral. The inline style="color: {{statusColor}}" pattern
is dead: dashboard + customer_unified render a class-based
.status-dot-<token>; statusIcon (constant "●") retired from funcmap
and templates.
- Tests (new; the hub web package had no funcmap/template tests):
TestStatusColorTruthTable over the full enumerated status set —
red-proven vs the old implementation (ok returned "#4ade80") — and
TestTemplatesParseWithFuncmap.
Operator picks a version from a Gitea-populated dropdown; the hub reads that
version's sha256 from Gitea itself (files-metadata API, no artifact download) and
vouches it — no hand-copied checksums. New internal/gitea read-only client
(ListVersions + FileSHA256, unit-tested). Configuration UI: version <select>s +
read-only sha display; handleSetArtifacts derives the sha authoritatively and
refuses the save on a Gitea lookup failure. Degrades to manual text entry without
registry creds. go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Part A (hub): move the global-floor + Day-0-artifacts cards from the Customers
page to the Configuration tab; routes → /configuration/{global-floor,artifacts};
redirects + flashes to /configuration. Customers page back to list + Add.
Part B: online setup command on the customer page (download-then-run, passphrase
at prompt, not templated); serve /scripts/ from the website (sparse-checkout +
nginx location) so felhom.eu/scripts/felhom-host-install.sh resolves; script
passphrase prompt reads < /dev/tty (works for pipe-to-bash too).
Part C (script): --vmid auto-detect — default 9201 in use + no --force → pick the
next free id from pct+qm and confirm; explicit --vmid stays die-unless-force.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ProcessEvent routed only warning/error; a critical-severity event was silently dropped.
Now routes warning/error/critical, logs unrecognized severities, and guards a nil
GetNotificationPrefs (which would panic/crash the hub). host_disk_critical emits its
natural critical severity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
New HostDiskChecker on the 60s sweep alerts the operator when a Proxmox host root
filesystem crosses warn (90%) / crit (95%). Born/persistent (a disk already full at
hub restart alerts on cycle 1); distinct host_disk_* event types from the guest disk_*;
critical band maps to severity error (the dispatcher only routes warning/error).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Part A: hub-config operator_email/operator_enabled (committed earlier);
operator + customer email paths proven end-to-end via real Resend sends.
Append the "Findings closed" section to the TESTRUN audit (Part A proof,
F2-a root cause + deploy fix, F2-b fix + live validation, pending §D
supervised reboot) and remove the stale duplicate "Final state" block.
REPORT overwritten. hub/CHANGELOG entry. No hub image change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
The v0.22.0 commit added the checker but the main.go goroutine edit never applied. Live test caught
it (no host_leaf_changed on a leaf regen). Now started on the 60s sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
Constructors seed only healthy hosts; an already-degraded/stale host is left unseeded so the first
Check() emits once (cooldown dedups on hub restart). Born-degraded red-proof + staleness test updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
The controller v0.90.0 channel-health checker pushes agent_channel_* events to /api/v1/event;
the allowedEventTypes allowlist rejected them (HTTP 400). Added the 8 types (operator-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Companion to felhom-agent v0.44.0. New monitor.HostCapabilityChecker (sibling of
HostStalenessChecker) reads the capabilities snapshot from the latest host report and emits
agent_capability_degraded/recovered (operator-only, 1h cooldown) on ok<->degraded transitions
for any Critical capability. store.GetHostCapabilities (MAX(id), no migration). Goldens mirror
the new capabilities field.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Resend send-scoped key was committed in plaintext in manifests/hub.yaml (ConfigMap)
and manifests/felhom.secret.yaml. Rotated to a new key and removed from git.
- hub: new RESEND_API_KEY env override (cmd/hub/main.go), mirrors REGISTRY_TOKEN;
ConfigMap resend_api_key now an empty placeholder; Deployment injects from
Secret/resend-api. Image 0.17.0.
- contact-mailer: secretKeyRef repointed contact-mailer-config -> resend-api.
- felhom.secret.yaml: contact-mailer-config Secret removed; healthchecks
EMAIL_HOST_PASSWORD blanked (workload not deployed).
- documentation/runbooks/secrets.md: out-of-band secret model + create/rotate steps.
Secret/resend-api is created imperatively out-of-band and is NOT committed.
No secret value appears in this repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- cmd/hub/main.go: seed the Day-0 artifact manifest from ARTIFACT_AGENT_VERSION/
ARTIFACT_AGENT_SHA256/ARTIFACT_GOLDEN_VERSION/ARTIFACT_GOLDEN_SHA256 on startup
(only empty fields, so UI edits stick) — same escape hatch the floor uses.
- manifests/hub.yaml: image 0.15.0 -> 0.16.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator sets a minimum controller version (FLOOR), per-customer defaulting to a
global floor; the report ACK returns the effective floor + latest_version so the
controller auto-updates to the floor when below it (latest stays the opt-in button).
- store: min_controller_version column + hub_settings global floor + Effective/
Get/SetGlobal/SetMin resolution + config/env DEFAULT_MIN_CONTROLLER_VERSION
- handler: report ACK {min_controller_version, latest_version}; LatestVersionProvider
- web: global floor editor + per-customer override form + Floor column (English)
- tests: floor resolution + ACK + render; override-precedence red-proof verified
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
New POST /api/v1/host-enroll (handleHostEnroll): X-Retrieval-Password authed,
body {customer_id} -> {host_id, api_key}. Mint-once-reuse (201 first, 200
reuse) so re-running the host-bootstrap never orphans a running agent's key;
auth checked before any mint. Backed by new Store.GetHostByCustomer
(ORDER BY updated_at DESC LIMIT 1, idx_hosts_customer).
GET /config/{id} and global-key POST /admin/hosts left untouched. Exact-match
route (path == "/host-enroll") to avoid the /hosts/ prefix collision.
Tests: host_enroll_test.go (mint/reuse/401-no-mint/404/400) + GetHostByCustomer
store test; companion red-proof verified always-mint fails the reuse assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
Tracks felhom-agent v0.39.0, which dropped role + restic_repo_coord from the host-half
drives[]. Hub reads drives as json.RawMessage (verbatim passthrough) → no store/handler
change needed. Dropped role from the store golden + drHostHalf fixture; re-synced the api
host-report golden byte-identical with the agent copy (it previously lacked dr_recipe
entirely). Added TestAssembleDRRecipe_V1DriveShape regression guard (demonstrated to fail
when the fixture re-adds role, then reverted). No behavior change; redeploy optional.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin