Runbook Phase 0b. Values come from the out-of-band felhom-system/storagebox
secret; optional:true so the pod starts without it (hub degrades to offsite
disabled). HETZNER_LOCATION explicit fsn1. NOT synced yet — gated on the
secret holding the DEDICATED-project scoped token (the current one fails the
scope check: it sees the old project's ep0 box, not the new pool box).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
A heal marker is an event, not a baseline: construction seeds pre-existing markers
(startup false-alarm guard) but a newly-observed marker now raises the warning, so
the first auto-heal surfaces (matches the live drill). Added tests for both halves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Peersync script: validate-first (jq contract check before ANY state change),
head-file + generated-peers conf model, syncconf-from-tmp then atomic mv (live
conf never diverges in the failure direction), zero-peer payload valid (wipe).
hub.yaml: 0.32.0 image + WG_ENDPOINT_SSH_* env + optional Secret mount so the
pod starts before the runbook's step-6 Secret exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Part A (hub): move the global-floor + Day-0-artifacts cards from the Customers
page to the Configuration tab; routes → /configuration/{global-floor,artifacts};
redirects + flashes to /configuration. Customers page back to list + Add.
Part B: online setup command on the customer page (download-then-run, passphrase
at prompt, not templated); serve /scripts/ from the website (sparse-checkout +
nginx location) so felhom.eu/scripts/felhom-host-install.sh resolves; script
passphrase prompt reads < /dev/tty (works for pipe-to-bash too).
Part C (script): --vmid auto-detect — default 9201 in use + no --force → pick the
next free id from pct+qm and confirm; explicit --vmid stays die-unless-force.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dispatcher's operator path was gated off (processOperator returns
unless operatorOn && operatorEmail!=""), so the self-health pipeline
stopped one hop short of the inbox. Set the operator recipient
(admin@felhom.eu — the operator's own address, not a secret) and enable.
Config-only; no image bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
Resend send-scoped key was committed in plaintext in manifests/hub.yaml (ConfigMap)
and manifests/felhom.secret.yaml. Rotated to a new key and removed from git.
- hub: new RESEND_API_KEY env override (cmd/hub/main.go), mirrors REGISTRY_TOKEN;
ConfigMap resend_api_key now an empty placeholder; Deployment injects from
Secret/resend-api. Image 0.17.0.
- contact-mailer: secretKeyRef repointed contact-mailer-config -> resend-api.
- felhom.secret.yaml: contact-mailer-config Secret removed; healthchecks
EMAIL_HOST_PASSWORD blanked (workload not deployed).
- documentation/runbooks/secrets.md: out-of-band secret model + create/rotate steps.
Secret/resend-api is created imperatively out-of-band and is NOT committed.
No secret value appears in this repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- cmd/hub/main.go: seed the Day-0 artifact manifest from ARTIFACT_AGENT_VERSION/
ARTIFACT_AGENT_SHA256/ARTIFACT_GOLDEN_VERSION/ARTIFACT_GOLDEN_SHA256 on startup
(only empty fields, so UI edits stick) — same escape hatch the floor uses.
- manifests/hub.yaml: image 0.15.0 -> 0.16.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DR recipe slice (hub half), grounded in SPIKE-dr-recipe-2026-06-16. The hub
receives two additive dr_recipe halves on the existing report paths (agent
storage/guest/PBS on host-report; controller customer/apps on the controller
report), stores them PLAINTEXT in a DEDICATED dr_recipe table keyed by customer
(each half preserves the other), and AssembleDRRecipe stitches them into one
operator-readable recipe (ignore-unknown + version-skew tolerant).
View: a DR-recipe panel on the customer page + GET /customers/{id}/dr-recipe.json
download (operator-auth, no secrets to redact). Plaintext-at-rest is correct —
the recipe is the clean inverse of the retired infra-backup.
Tests: store round-trip (each half preserves the other), assemble-matches-golden,
ignore-unknown + version skew, partial halves, no-secrets sweep. Manifest tag
bumped to v0.13.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase-1 of SPIKE-infra-backup-2026-06-15. The infra-backup mechanism was dead
since slice 8C yet stored plaintext customer secrets at rest (app-secret key,
restic password, Cloudflare tokens) — a zero-knowledge violation — and its
absence made the daily expected_backup_missed email fire for healthy customers.
- Repoint monitor.CheckBackupDeadlines backup half to the agent host-report's
PBS snapshots (+vzdump): alarm only on no-backup / >26h stale / verify failed.
Keep the db_dump half. No host-report → no backup alarm (liveness owns that).
New store.GetLatestHostReportJSON. Tests incl. a companion that fails pre-fix.
- Remove the infra-backup endpoints, store methods/types, and operator panel;
/recovery now returns config_yaml only.
- migrate(): DROP infra_backup_versions/infra_backups + VACUUM (+wal_checkpoint)
to physically reclaim the plaintext pages, gated on table existence.
Flagged out-of-scope: exposed creds need operator rotation; legacy reports table
holds historical plaintext restic_password rows (separate leak, not purged here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin