A failed restore-test was a [WARN] line in the ingest handler and nothing else —
no event, no notification, no gauge. True for the LOCAL tier that was already
being tested, so the loudest DR signal this system produces was inaudible.
Rotating tiers without this would only mean two tiers can fail silently
instead of one.
Two signals, deliberately NOT merged:
restore_test_failed (error) — a run completed and did NOT pass
restore_test_stale (warning) — a tier not PROVEN within its interval
Merging them collapses 'your DR is broken' into 'your DR is unverified', and
the second is the one that quietly becomes the first. The staleness wording
says 'unverified, not known-broken' and a test asserts that phrasing.
Anchored per R-81, not re-derived: a never-proven tier on a newborn box is
UNKNOWN, not FAILED, until the window elapses. This family has made the
opposite mistake three times; this monitor was written straight after the third,
so it copies R-81's structure rather than inventing a fourth shape.
restoreProvenStaleAfter = 7d is derived: oldest-first over two tiers at a 24h
cadence proves each ~every 2 days, so 7d tolerates ~3 missed opportunities and
sits inside the 2-week offsite retention.
Per-tier proof comes from the hub's retained host-report window — the agent
reports only its latest run, so the latest report alone cannot answer 'when was
the OTHER tier last proven?'. Reused R-81's mechanism instead of a wire change.
Both types registered in allowedEventTypes (R-77's inert-seam lesson) and
operator-tier only — no customerMessages entry.
FIXED a time bomb I introduced in Slice C: the restart-blind-window test
hard-coded 2026-07-18T18:31:06Z while comparing against the real clock. Harmless
under one 26h threshold; once the offsite tier got an 8-day limit it passed all
day and began failing at 18:31 UTC, exactly 8 days later. Now relative.
Red-proofs B and D observed. Full suite green (17 packages, rc=0).
Let a customer bind their own freshly-installed appliance without the
operator: operator "Send self-bind link" mints a 7-day tokenized
capability link, emailed (Hungarian, sibling sender) to the customer, who
opens a public /bind/<token> page and proves two factors — the console
pairing code shown on the box screen + their retrieval passphrase — and
the hub stages the bind via the same BindAppliance (provenance
customer_selfbind). The box's ~30s appliance poll delivers.
Viktor's three rulings verbatim: console pairing code (no appliance list
ever rendered), operator-sent tokenized link, 5-attempt lockout ->
"call support". Wrong code == wrong passphrase (one generic failure, no
oracle, both factors compared unconditionally); expiry falls back to
operator-bind unchanged.
THE TRAP: one public prefix /bind/, exempt from auth+CSRF at both /login
gate sites via a single isPublicBindPath predicate (tight trailing-slash
match; ServeMux ..-cleans; handler rejects '/' in token). 9 tests
(Scenarios A-F + F1/F2); 4 red-proofs verified red-then-green (lockout,
oracle, widened-prefix, single-active). GC verdict: no appliance GC ->
the 7-day TTL stands alone. Controller/agent untouched; R-27b deferred.
Green: full hub build/vet/test (17 ok) + bash -n + hub confirm gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.
Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.
- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).
Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).
Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).
- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
+ OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".
Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
The immediate-sync arc covered only operator-initiated desired-state changes;
system-initiated mutations bumped the generation silently, so a freshly onboarded
box waited a full agent tick for state the hub had already minted (observed live at
slice-C onboarding). Wire the existing, live-proven notifiers into every system site
on the correct plane — call-site wiring only, no new mechanism.
Agent plane (poke.Notifier):
- web/pbsdr.go: PBSDRAutoProvision (the observed lag), ReissuePBSDR (also lifts the
pbsdrheal reconciler escalation, zero reconciler changes), handlePBSDRReissue —
each pokes AFTER the successful SetHostDesired, never on a blocked/error path.
- api: new nil-safe Poker seam (PokeHost/PokeAllHosts + SetPoker); handleAdminSetDesiredState
pokes the target host; handleAdminSetOperatorPeer fires PokeAllHosts only when the
fleet generation bump succeeded (fire-after-commit).
- main.go: one poke.Notifier now feeds both planes (SetPoke + SetPoker).
Controller plane (intent.Hub.Bump):
- api/reissueOnReenroll: one nil-guarded bump so a long-polling controller wakes in
seconds instead of on the 15-min cycle.
Deliberate non-sites (unchanged): WG register (undeliverable pre-tunnel — the agent
fast-tick SECONDARY owns it), WG delete (transport removed), pbsdrheal Restage (no
generation bump → the 60s ticker is the pickup path). internal/pbsdrheal byte-unchanged.
Tests: 10 non-hollow tests (web async channel-synchronized fake sender; api synchronous
fake Poker) with explicit zero-count negatives; representative red-proofs per group
(A/B/C/D) run-fail-restored. Green: go build/vet/test all pass.
GET /api/v1/wait long-poll: the box holds an authed hanging GET; the hub
completes it the instant any operator intent bumps that customer's in-memory
generation, then the box fires its ordinary report and the ACK delivers
everything through the unchanged machinery. 240s hold with a 25s heartbeat
newline defeats the nginx 60s proxy_read_timeout with no ingress annotation;
WriteTimeout lifted per-connection via ResponseController.
- internal/intent: per-customer generation counter + waiter registry
(Bump/Wait/Close), coalescing to latest, race-closer, in-memory by design.
Red-proofs: counter-vs-queue + race-closer (run-fail-reverted).
- api/wait.go: the endpoint (per-customer only; global key 400; A cannot see B).
- web bumps after every intent write (fire-after-commit): config CRUD, claim
resend, offsite re-issue/freeze, password regen, block/unblock, floors
(global bumps all config-managed), controller log-tail + log-bundle.
- main.go: one intent hub shared by web+api; Close() before server.Shutdown.
Pairs with controller v0.140.0 (the long-poll client). Grounding:
documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md.
F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path,
single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on
re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty
(red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the
mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events:
claim_reissued_reenroll / offsite_reissued / escrow_stale.
Controller + scripts unchanged (source contradicted both premises): the controller reads escrow
prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4
root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc
F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass.
Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.
New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
The global bearer (api.report_api_key) leaves manifests/hub.yaml: the
ConfigMap field is an empty placeholder; the Deployment injects
REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT
optional - a missing Secret fails Ready rather than booting a hub with an
empty bearer). main.go gains the REPORT_API_KEY env override (the
RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped
(64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing
umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md
backlog, out of scope here).
Ordering (load-bearing): create Secret/report-api with the CURRENT value
BEFORE syncing; this manifest change rides the same sync as the 0.53.0
image (older images ignore the env override). The exposed git-history copy
dies only with the supervised ROTATION - procedure + consumer list in
documentation/runbooks/secrets.md ("Operator/global bearer key"); both
publish runbooks' ROTATION notes now point at that disposition. The
controller repo's example-config copy of the literal is scrubbed
(felhom-controller 51c871a).
Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml ->
gate exit 1 (recorded); restored clean.
Sibling checker over the controller report's offsite object: quota-fill
warn/crit + the silently-stuck staleness detector (escrowed-only,
red-proofed; nil-safe on pre-v0.109 reports; same-second tie-guard).
SetOffsiteFrozen flips ONLY readonly on the exactly-1 labelled sub-account
(SSH preserved); Freeze/Unfreeze buttons — manual only, never automatic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Descriptor.HostFingerprint (SHA256, non-secret), captured at provision via an
x/crypto/ssh keyscan (SSHHostKeyScanner — dials :23, grabs the host key from the
handshake, no ssh binary). Fail-closed: nil scanner or scan failure → error (don't
serve a descriptor the controller can't verify). Pairs with controller v0.106.0
which re-scans + refuses on mismatch (no blind TOFU).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
store.GetHostOOBStates parses the agent oob heartbeat stanza. monitor/host_oob:
transition-based oob_degraded/oob_recovered warning (felhom-sshd down while the
operator peer is configured, OR config invalid) — proactive "can the operator get
in right now" signal; unconfigured OOB never alerts. Wired into the 60s sweep.
Non-hollow tests + transitions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
internal/wgsync: x/crypto/ssh client with ssh.FixedHostKey pin (no insecure
fallback), forced-command exec, ok/applied response contract; Reconciler pushes
the FULL peer list on Trigger or 5-min tick (drift repair by construction).
internal/api/wg.go: PUT/GET /admin/wg/endpoint + POST/DELETE/GET /admin/wg/peers,
global-key-only, pubkey in body (base64 vs URL), sync ok|deferred|disabled.
main.go: WG_ENDPOINT_SSH_* env wiring, disabled-with-INFO when unconfigured.
Groups B/C/D tests incl. in-process SSH server; red-proofs b/c/d run + reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Operator picks a version from a Gitea-populated dropdown; the hub reads that
version's sha256 from Gitea itself (files-metadata API, no artifact download) and
vouches it — no hand-copied checksums. New internal/gitea read-only client
(ListVersions + FileSHA256, unit-tested). Configuration UI: version <select>s +
read-only sha display; handleSetArtifacts derives the sha authoritatively and
refuses the save on a Gitea lookup failure. Degrades to manual text entry without
registry creds. go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New HostDiskChecker on the 60s sweep alerts the operator when a Proxmox host root
filesystem crosses warn (90%) / crit (95%). Born/persistent (a disk already full at
hub restart alerts on cycle 1); distinct host_disk_* event types from the guest disk_*;
critical band maps to severity error (the dispatcher only routes warning/error).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
The v0.22.0 commit added the checker but the main.go goroutine edit never applied. Live test caught
it (no host_leaf_changed on a leaf regen). Now started on the 60s sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
Companion to felhom-agent v0.44.0. New monitor.HostCapabilityChecker (sibling of
HostStalenessChecker) reads the capabilities snapshot from the latest host report and emits
agent_capability_degraded/recovered (operator-only, 1h cooldown) on ok<->degraded transitions
for any Critical capability. store.GetHostCapabilities (MAX(id), no migration). Goldens mirror
the new capabilities field.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Resend send-scoped key was committed in plaintext in manifests/hub.yaml (ConfigMap)
and manifests/felhom.secret.yaml. Rotated to a new key and removed from git.
- hub: new RESEND_API_KEY env override (cmd/hub/main.go), mirrors REGISTRY_TOKEN;
ConfigMap resend_api_key now an empty placeholder; Deployment injects from
Secret/resend-api. Image 0.17.0.
- contact-mailer: secretKeyRef repointed contact-mailer-config -> resend-api.
- felhom.secret.yaml: contact-mailer-config Secret removed; healthchecks
EMAIL_HOST_PASSWORD blanked (workload not deployed).
- documentation/runbooks/secrets.md: out-of-band secret model + create/rotate steps.
Secret/resend-api is created imperatively out-of-band and is NOT committed.
No secret value appears in this repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- cmd/hub/main.go: seed the Day-0 artifact manifest from ARTIFACT_AGENT_VERSION/
ARTIFACT_AGENT_SHA256/ARTIFACT_GOLDEN_VERSION/ARTIFACT_GOLDEN_SHA256 on startup
(only empty fields, so UI edits stick) — same escape hatch the floor uses.
- manifests/hub.yaml: image 0.15.0 -> 0.16.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator sets a minimum controller version (FLOOR), per-customer defaulting to a
global floor; the report ACK returns the effective floor + latest_version so the
controller auto-updates to the floor when below it (latest stays the opt-in button).
- store: min_controller_version column + hub_settings global floor + Effective/
Get/SetGlobal/SetMin resolution + config/env DEFAULT_MIN_CONTROLLER_VERSION
- handler: report ACK {min_controller_version, latest_version}; LatestVersionProvider
- web: global floor editor + per-customer override form + Floor column (English)
- tests: floor resolution + ACK + render; override-precedence red-proof verified
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
- Add Configuration page with "Refresh Assets" button
- Replace seedIfEmpty with seedOrUpdate (SHA-256 compare on startup)
- Translate all Hungarian text on Apps pages to English
- Add Configuration tab to all template navigation
- Expand isAssetFile to match favicon patterns
- Add felhom-logo.svg to website assets for the pipeline
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add internal/assets package that manages app assets (logos, screenshots)
on Hub PVC with automatic seeding from baked-in image copy on first run.
Two new API endpoints: GET /assets/manifest (JSON with SHA-256 checksums)
and GET /assets/file/{name} for controllers to sync assets.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add hubVersion template function via closure in web.New(). Version is
passed from main.go (set via ldflags at build time) and displayed in
the footer of all 6 page templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New "Configurations" section lets operators pre-configure customer settings
in the Hub, then docker-setup.sh can download a ready-made controller.yaml
using just a customer ID and retrieval password.
- Store: customer_configs table with CRUD + per-customer API key lookup
- API: GET /api/v1/config/{id} with X-Retrieval-Password auth
- Auth: per-customer API keys alongside existing global key (backward compatible)
- Web UI: /configs list, create, edit, delete, YAML preview, copy-to-clipboard
- YAML gen: deep-merge controller.yaml.example template with customer overrides
- Template fetcher: background goroutine refreshing template from Gitea repo
- Navigation: Dashboard / Configurations tabs on all pages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hub now tracks controller_url from reports, periodically checks the Gitea
registry for the latest controller image version, and shows a "Trigger Update"
button on the customer detail page that proxies to the controller's self-update
API endpoint using the shared API key.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Hub service receives reports from customer controllers
- SQLite store with 90-day retention and auto-prune
- REST API: POST /api/v1/report, GET /api/v1/customers
- Dark theme dashboard with status overview table
- Customer detail page with system, storage, containers, backup, health
- Bearer token auth for report ingest, bcrypt auth for dashboard
- K8s manifest for felhom-system namespace (Deployment, Service, Ingress, PVC)
- Dockerfile with multi-stage build
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude Code
admin