Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.
New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
hub v0.54.0: change operator login password from the Configuration UI
Adds a "Login password" card on /configuration. The password was previously
settable only via the hub-config ConfigMap (auth.password_hash) + redeploy.
- store: hub_settings key operator_password_hash + Get/SetOperatorPasswordHash
- server: passwordHash field -> configPasswordHash (seed); new
effectivePasswordHash() (DB override wins, else seed) is now the single
source for the CSRF gate, RequireAuth, and handleLogin
- POST /configuration/password (handleChangePassword): requires current
password, 8-72 byte new + confirm, bcrypt cost 10, persists DB override;
existing sessions kept valid; ConfigMap stays the break-glass reset path
- UI: current/new/confirm form + inline mismatch pre-check + 6 flashes
- tests + red-proofs: override precedence, happy-path via handleLogin,
wrong-current rejection, mismatch/too-short/no-op, template render
- docs: CHANGELOG, README (auth+config), REUSE, REPORT
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
Adds a form-level debug-mode checkbox to the customer config editor so an
operator can flip the controller's Logging.Level=debug (/debug menu + verbose
log) remotely, without SSH. Form field (not raw-JSON injection) because
handleConfigUpdate rebuilds ConfigJSON from the form on every save; the
config-version bump makes the controller re-pull + self-restart next cycle.
- buildConfigJSON: debug_mode checked -> "logging":{"level":"debug"};
unchecked -> logging key omitted.
- config_form.html: "Hibakeresési mód (fejlesztői)" section + render state.
- configs_debug_test.go: form->JSON both ways; full-path survival test
(debug lands, offsite descriptor unchanged, foreign-key red-proof); render
state; red-proof exercised.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Sibling checker over the controller report's offsite object: quota-fill
warn/crit + the silently-stuck staleness detector (escrowed-only,
red-proofed; nil-safe on pre-v0.109 reports; same-second tie-guard).
SetOffsiteFrozen flips ONLY readonly on the exactly-1 labelled sub-account
(SSH preserved); Freeze/Unfreeze buttons — manual only, never automatic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Additive host_escrow migration; SaveHostEscrow/HostEscrow gain the hash
(NULL-safe for legacy rows); GetEscrowStatusForCustomer joins hosts;
the report ACK gains escrow:{identity_blob_present,restic_pw_sha256,
created_at} (omitted without a row). Contract test mirrors the agent's
v0.79.0 emit struct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
F4: ReissueCredentials — explicit operator recovery for consumed-password
dead-ends; resets the labelled resource's password (exactly-1 guard,
red-proofed), stores a fresh one-time secret, bumps ConfigVersion.
New hetznerapi.ResetBoxPassword for the dedicated path.
F2: host-key scan retry-with-backoff (~60s ladder, red-proofed) — first
save survives fresh-subaccount DNS lag.
F5: config form disables submits + shows an in-flight notice (the re-click
bait that caused live F1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
First supervised live run: the ~25s spinner-less offsite save invited a
re-click; the abandoned first request's r.Context() was canceled between
CreateSubaccount and SaveOneTimeSecret, stranding sub-account 268985 with a
password lost forever (consume 404s permanently).
applyOffsite now provisions on context.WithoutCancel + 3-minute absolute
timeout: once the create starts, create->wait->store runs to completion.
Regression test with a ctx-honoring fake that cancels the request context
mid-create; red-proofed against the raw-ctx pre-fix shape (reproduces the
exact live error).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Descriptor.HostFingerprint (SHA256, non-secret), captured at provision via an
x/crypto/ssh keyscan (SSHHostKeyScanner — dials :23, grabs the host key from the
handshake, no ssh binary). Fail-closed: nil scanner or scan failure → error (don't
serve a descriptor the controller can't verify). Pairs with controller v0.106.0
which re-scans + refuses on mismatch (no blind TOFU).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
A heal marker is an event, not a baseline: construction seeds pre-existing markers
(startup false-alarm guard) but a newly-observed marker now raises the warning, so
the first auto-heal surfaces (matches the live drill). Added tests for both halves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Operator picks a version from a Gitea-populated dropdown; the hub reads that
version's sha256 from Gitea itself (files-metadata API, no artifact download) and
vouches it — no hand-copied checksums. New internal/gitea read-only client
(ListVersions + FileSHA256, unit-tested). Configuration UI: version <select>s +
read-only sha display; handleSetArtifacts derives the sha authoritatively and
refuses the save on a Gitea lookup failure. Degrades to manual text entry without
registry creds. go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Part A (hub): move the global-floor + Day-0-artifacts cards from the Customers
page to the Configuration tab; routes → /configuration/{global-floor,artifacts};
redirects + flashes to /configuration. Customers page back to list + Add.
Part B: online setup command on the customer page (download-then-run, passphrase
at prompt, not templated); serve /scripts/ from the website (sparse-checkout +
nginx location) so felhom.eu/scripts/felhom-host-install.sh resolves; script
passphrase prompt reads < /dev/tty (works for pipe-to-bash too).
Part C (script): --vmid auto-detect — default 9201 in use + no --force → pick the
next free id from pct+qm and confirm; explicit --vmid stays die-unless-force.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ProcessEvent routed only warning/error; a critical-severity event was silently dropped.
Now routes warning/error/critical, logs unrecognized severities, and guards a nil
GetNotificationPrefs (which would panic/crash the hub). host_disk_critical emits its
natural critical severity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
New HostDiskChecker on the 60s sweep alerts the operator when a Proxmox host root
filesystem crosses warn (90%) / crit (95%). Born/persistent (a disk already full at
hub restart alerts on cycle 1); distinct host_disk_* event types from the guest disk_*;
critical band maps to severity error (the dispatcher only routes warning/error).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Part A: hub-config operator_email/operator_enabled (committed earlier);
operator + customer email paths proven end-to-end via real Resend sends.
Append the "Findings closed" section to the TESTRUN audit (Part A proof,
F2-a root cause + deploy fix, F2-b fix + live validation, pending §D
supervised reboot) and remove the stale duplicate "Final state" block.
REPORT overwritten. hub/CHANGELOG entry. No hub image change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
Constructors seed only healthy hosts; an already-degraded/stale host is left unseeded so the first
Check() emits once (cooldown dedups on hub restart). Born-degraded red-proof + staleness test updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
The controller v0.90.0 channel-health checker pushes agent_channel_* events to /api/v1/event;
the allowedEventTypes allowlist rejected them (HTTP 400). Added the 8 types (operator-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Companion to felhom-agent v0.44.0. New monitor.HostCapabilityChecker (sibling of
HostStalenessChecker) reads the capabilities snapshot from the latest host report and emits
agent_capability_degraded/recovered (operator-only, 1h cooldown) on ok<->degraded transitions
for any Critical capability. store.GetHostCapabilities (MAX(id), no migration). Goldens mirror
the new capabilities field.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Resend send-scoped key was committed in plaintext in manifests/hub.yaml (ConfigMap)
and manifests/felhom.secret.yaml. Rotated to a new key and removed from git.
- hub: new RESEND_API_KEY env override (cmd/hub/main.go), mirrors REGISTRY_TOKEN;
ConfigMap resend_api_key now an empty placeholder; Deployment injects from
Secret/resend-api. Image 0.17.0.
- contact-mailer: secretKeyRef repointed contact-mailer-config -> resend-api.
- felhom.secret.yaml: contact-mailer-config Secret removed; healthchecks
EMAIL_HOST_PASSWORD blanked (workload not deployed).
- documentation/runbooks/secrets.md: out-of-band secret model + create/rotate steps.
Secret/resend-api is created imperatively out-of-band and is NOT committed.
No secret value appears in this repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- cmd/hub/main.go: seed the Day-0 artifact manifest from ARTIFACT_AGENT_VERSION/
ARTIFACT_AGENT_SHA256/ARTIFACT_GOLDEN_VERSION/ARTIFACT_GOLDEN_SHA256 on startup
(only empty fields, so UI edits stick) — same escape hatch the floor uses.
- manifests/hub.yaml: image 0.15.0 -> 0.16.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin