The immediate-sync arc covered only operator-initiated desired-state changes;
system-initiated mutations bumped the generation silently, so a freshly onboarded
box waited a full agent tick for state the hub had already minted (observed live at
slice-C onboarding). Wire the existing, live-proven notifiers into every system site
on the correct plane — call-site wiring only, no new mechanism.
Agent plane (poke.Notifier):
- web/pbsdr.go: PBSDRAutoProvision (the observed lag), ReissuePBSDR (also lifts the
pbsdrheal reconciler escalation, zero reconciler changes), handlePBSDRReissue —
each pokes AFTER the successful SetHostDesired, never on a blocked/error path.
- api: new nil-safe Poker seam (PokeHost/PokeAllHosts + SetPoker); handleAdminSetDesiredState
pokes the target host; handleAdminSetOperatorPeer fires PokeAllHosts only when the
fleet generation bump succeeded (fire-after-commit).
- main.go: one poke.Notifier now feeds both planes (SetPoke + SetPoker).
Controller plane (intent.Hub.Bump):
- api/reissueOnReenroll: one nil-guarded bump so a long-polling controller wakes in
seconds instead of on the 15-min cycle.
Deliberate non-sites (unchanged): WG register (undeliverable pre-tunnel — the agent
fast-tick SECONDARY owns it), WG delete (transport removed), pbsdrheal Restage (no
generation bump → the 60s ticker is the pickup path). internal/pbsdrheal byte-unchanged.
Tests: 10 non-hollow tests (web async channel-synchronized fake sender; api synchronous
fake Poker) with explicit zero-count negatives; representative red-proofs per group
(A/B/C/D) run-fail-restored. Green: go build/vet/test all pass.
A generic ISO carries NO customer secret. The box registers itself at the hub
as an unclaimed appliance; the operator binds it to a customer; the hub delivers
the customer-id + retrieval passphrase ONCE; day-0 completes via the slice-A path.
Hub (v0.62.0):
- store/appliance.go: appliance_registrations keyed by (uuid, mac_set) — MAC set
is the tiebreaker (duplicate SMBIOS UUIDs); token stored as sha256 only.
Idempotent register (sticky-discard), atomic one-shot delivery, bind/discard.
- api/appliance.go: POST /appliance/register (the one unauth endpoint, per-IP
rate-limited, 256-bit token); GET /appliance/poll (404 no-oracle / 204 unbound
/ 200 deliver-once / 410 delivered). Passphrase read live, never logged.
- web/appliances.go: Hosts-page "Unclaimed appliances" section + BIND (customer
picker, host count display-only) + DISCARD; SSH host-key fingerprints; events.
- Red-proofs: one-shot delivery + register idempotency (both proven red);
404-no-oracle, sticky-discard, bind staging, render. Green + confirm gate.
Scripts (v1.19.0):
- felhom-bootstrap.sh: ONE unit, TWO modes. Direct (env has customer/passphrase)
= slice-A path, byte-identical, only branched around. Pairing (generic) =
register + poll (RestartSec=30 is the poll timer); on delivery write the env
0600 and fall through to direct. Secrets + token shredded on success.
- build-felhom-iso.sh --pairing: generic secret-free ISO, -generic filename,
manifest mode=pairing. profiles/generic.profile (new).
- test/bootstrap-modes.sh: Scenario D (direct = zero appliance calls) + pairing
register/poll + delivery handoff — all green in a debian container.
GET /api/v1/wait long-poll: the box holds an authed hanging GET; the hub
completes it the instant any operator intent bumps that customer's in-memory
generation, then the box fires its ordinary report and the ACK delivers
everything through the unchanged machinery. 240s hold with a 25s heartbeat
newline defeats the nginx 60s proxy_read_timeout with no ingress annotation;
WriteTimeout lifted per-connection via ResponseController.
- internal/intent: per-customer generation counter + waiter registry
(Bump/Wait/Close), coalescing to latest, race-closer, in-memory by design.
Red-proofs: counter-vs-queue + race-closer (run-fail-reverted).
- api/wait.go: the endpoint (per-customer only; global key 400; A cannot see B).
- web bumps after every intent write (fire-after-commit): config CRUD, claim
resend, offsite re-issue/freeze, password regen, block/unblock, floors
(global bumps all config-managed), controller log-tail + log-bundle.
- main.go: one intent hub shared by web+api; Close() before server.Shutdown.
Pairs with controller v0.140.0 (the long-poll client). Grounding:
documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md.
F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path,
single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on
re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty
(red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the
mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events:
claim_reissued_reenroll / offsite_reissued / escrow_stale.
Controller + scripts unchanged (source contradicted both premises): the controller reads escrow
prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4
root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc
F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass.
Additive host_escrow migration; SaveHostEscrow/HostEscrow gain the hash
(NULL-safe for legacy rows); GetEscrowStatusForCustomer joins hosts;
the report ACK gains escrow:{identity_blob_present,restic_pw_sha256,
created_at} (omitted without a row). Contract test mirrors the agent's
v0.79.0 emit struct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Endpoint card + peers table (truncated pubkeys with full-value title attr,
bound peers link to /hosts/<id>); Offsite nav link in all 9 page templates;
render tests for endpoint/peers, empty, and not-configured states.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
POST /hosts/{id}/wg (per-host self-scoped; global = operator/DR path): bind /
re-key-in-place / adopt; generation bump + endpoint push ONLY on real change.
mergeWireguard injects the hub-owned block into served desired-state at READ
time (stored operator blob never touched; fail-safe unmerged on any error;
no-peer = byte-identical pass-through — existing golden test untouched+green).
handleAdminSetDesiredState rejects top-level wireguard (400). Admin DELETE of a
BOUND peer bumps the owning host. NEW golden desired-state-wireguard.golden.json
= the S3 cross-repo contract. Red-proofs a/b/c/d run + reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
internal/wgsync: x/crypto/ssh client with ssh.FixedHostKey pin (no insecure
fallback), forced-command exec, ok/applied response contract; Reconciler pushes
the FULL peer list on Trigger or 5-min tick (drift repair by construction).
internal/api/wg.go: PUT/GET /admin/wg/endpoint + POST/DELETE/GET /admin/wg/peers,
global-key-only, pubkey in body (base64 vs URL), sync ok|deferred|disabled.
main.go: WG_ENDPOINT_SSH_* env wiring, disabled-with-INFO when unconfigured.
Groups B/C/D tests incl. in-process SSH server; red-proofs b/c/d run + reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
New HostDiskChecker on the 60s sweep alerts the operator when a Proxmox host root
filesystem crosses warn (90%) / crit (95%). Born/persistent (a disk already full at
hub restart alerts on cycle 1); distinct host_disk_* event types from the guest disk_*;
critical band maps to severity error (the dispatcher only routes warning/error).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
The controller v0.90.0 channel-health checker pushes agent_channel_* events to /api/v1/event;
the allowedEventTypes allowlist rejected them (HTTP 400). Added the 8 types (operator-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Operator sets a minimum controller version (FLOOR), per-customer defaulting to a
global floor; the report ACK returns the effective floor + latest_version so the
controller auto-updates to the floor when below it (latest stays the opt-in button).
- store: min_controller_version column + hub_settings global floor + Effective/
Get/SetGlobal/SetMin resolution + config/env DEFAULT_MIN_CONTROLLER_VERSION
- handler: report ACK {min_controller_version, latest_version}; LatestVersionProvider
- web: global floor editor + per-customer override form + Floor column (English)
- tests: floor resolution + ACK + render; override-precedence red-proof verified
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
New POST /api/v1/host-enroll (handleHostEnroll): X-Retrieval-Password authed,
body {customer_id} -> {host_id, api_key}. Mint-once-reuse (201 first, 200
reuse) so re-running the host-bootstrap never orphans a running agent's key;
auth checked before any mint. Backed by new Store.GetHostByCustomer
(ORDER BY updated_at DESC LIMIT 1, idx_hosts_customer).
GET /config/{id} and global-key POST /admin/hosts left untouched. Exact-match
route (path == "/host-enroll") to avoid the /hosts/ prefix collision.
Tests: host_enroll_test.go (mint/reuse/401-no-mint/404/400) + GetHostByCustomer
store test; companion red-proof verified always-mint fails the reuse assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
DR recipe slice (hub half), grounded in SPIKE-dr-recipe-2026-06-16. The hub
receives two additive dr_recipe halves on the existing report paths (agent
storage/guest/PBS on host-report; controller customer/apps on the controller
report), stores them PLAINTEXT in a DEDICATED dr_recipe table keyed by customer
(each half preserves the other), and AssembleDRRecipe stitches them into one
operator-readable recipe (ignore-unknown + version-skew tolerant).
View: a DR-recipe panel on the customer page + GET /customers/{id}/dr-recipe.json
download (operator-auth, no secrets to redact). Plaintext-at-rest is correct —
the recipe is the clean inverse of the retired infra-backup.
Tests: store round-trip (each half preserves the other), assemble-matches-golden,
ignore-unknown + version skew, partial halves, no-secrets sweep. Manifest tag
bumped to v0.13.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase-1 of SPIKE-infra-backup-2026-06-15. The infra-backup mechanism was dead
since slice 8C yet stored plaintext customer secrets at rest (app-secret key,
restic password, Cloudflare tokens) — a zero-knowledge violation — and its
absence made the daily expected_backup_missed email fire for healthy customers.
- Repoint monitor.CheckBackupDeadlines backup half to the agent host-report's
PBS snapshots (+vzdump): alarm only on no-backup / >26h stale / verify failed.
Keep the db_dump half. No host-report → no backup alarm (liveness owns that).
New store.GetLatestHostReportJSON. Tests incl. a companion that fails pre-fix.
- Remove the infra-backup endpoints, store methods/types, and operator panel;
/recovery now returns config_yaml only.
- migrate(): DROP infra_backup_versions/infra_backups + VACUUM (+wal_checkpoint)
to physically reclaim the plaintext pages, gated on table existence.
Flagged out-of-scope: exposed creds need operator rotation; legacy reports table
holds historical plaintext restic_password rows (separate leak, not purged here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add DELETE /hosts/{id}/jobs/{job_id} (per-host self-scoped, idempotent) so the
agent clears a job after executing or terminally rejecting it. The hub stores
the operator-signed blobs opaquely (no signing key — cannot forge or open);
the agent verifies + executes. Doc 03 §4/§6/§9 updated (operator-signed path
live; 8C wipe completes; 10B done).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Serve operator intent to authenticated hosts: PUT /admin/hosts/{id}/desired-state
(global key) bumps desired_generation; GET /hosts/{id}/desired-state + /jobs are
per-host self-scoped; the host-report envelope now carries the real generation +
has_signed_ops. New signed_jobs table + store methods. Desired-state stored/served
opaquely (agent owns the schema). Cross-repo golden (envelope + desired-state)
byte-identical with felhom-agent; doc 03 §4/§9 updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase B (hub half) of the restore-test warning fix. The agent v0.7.0 now passes a
restore-test that emitted a benign start advisory (systemd-nesting) and carries the
warning text on the wire.
- hostRestoreTest gains warnings + warnings_recognized mirror fields (omitempty;
absent recognized => false => louder unrecognized path)
- ingest logs [INFO] passed WITH WARNINGS (recognized), [WARN] for unrecognized;
FAILED still [WARN]
- golden restore_tests[0] gains the keys, byte-identical with felhom-agent (sha256
e6999d77...); bidirectional key-set contract test round-trips them
- no dashboard widget: no host-domain dashboard surface exists yet (log+persist only,
as with pbs_snapshots) -- deferred to slice 10
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Accept + persist the now-populated host-report pbs_snapshots. hostPBSSnapshot mirror in
hostReportPayload (persisted via report_json, no schema change); a FAILED PBS verify is
logged prominently (loudest offsite-DR signal). Shared golden updated byte-identical with
felhom-agent; TestHostPBSSnapshot_GoldenContract added. Build/deploy deferred (backward-compatible).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Accept + persist the now-populated host-report backups/restore_tests. Mirror structs in
hostReportPayload; persisted via report_json (no schema change); a FAILED restore-test is
logged prominently (loudest DR signal). Shared golden updated byte-identical with
felhom-agent; bidirectional key-set tests added. Build/deploy deferred (backward-compatible).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Accept + persist the now-populated host-report storage_targets. Minimal — the
authoritative storage manifest is hub-owned (slice 10); this mirrors what the agent
observes.
- hostReportPayload.StorageTargets: full mirror of the agent's hub.StorageTarget
wire contract; persisted verbatim in report_json (no schema change); count +
WARN on disconnected targets.
- shared host-report golden updated with two populated targets; byte-identical with
felhom-agent's copy.
- TestHostStorageTarget_GoldenContract: hub half of the bidirectional key-set test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- handleHostReport: read maxHostReportBytes+1 (4 MiB const) and reject oversize with
413 instead of silent LimitReader truncation. Controller handleReport (1 MiB) is
unchanged. Test asserts 413.
- contract: hub/internal/api/testdata/host-report.golden.json (byte-identical with
felhom-agent's copy) + TestHostReport_GoldenContract drives the real handler and
asserts 200 + denorm + both guests upserted.
- CHANGELOG v0.7.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New infra_backup_versions table with GFS pruning (~14 versions per
customer). Recovery endpoint supports ?version=ID. New /versions API.
Dashboard shows collapsible backup history with app names and disk count.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add internal/assets package that manages app assets (logos, screenshots)
on Hub PVC with automatic seeding from baked-in image copy on first run.
Two new API endpoints: GET /assets/manifest (JSON with SHA-256 checksums)
and GET /assets/file/{name} for controllers to sync assets.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New GET /api/v1/recovery/{customer_id}: returns generated controller.yaml
and infra backup in a single response for disaster recovery.
Auth via X-Retrieval-Password header.
- Report response now includes customer_blocked: true when customer
status is "blocked" — controllers use this to detect standing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace separate config detail and report detail pages with unified
/customers/{id} page showing both config info and live report data
- Add "blocked" status for customers (hidden from dashboard, notifications
suppressed, still accepts reports)
- Dashboard now shows config-only customers as "PENDING" status
- Customers list: all rows link to /customers/{id}, show BLOCKED badge
- New actions: block/unblock, push config to controller, auto-create
config from report data
- /configs/{id} now redirects to /customers/{id}
- Add config-badge CSS classes for MANAGED/MANUAL/BLOCKED badges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New "Configurations" section lets operators pre-configure customer settings
in the Hub, then docker-setup.sh can download a ready-made controller.yaml
using just a customer ID and retrieval password.
- Store: customer_configs table with CRUD + per-customer API key lookup
- API: GET /api/v1/config/{id} with X-Retrieval-Password auth
- Auth: per-customer API keys alongside existing global key (backward compatible)
- Web UI: /configs list, create, edit, delete, YAML preview, copy-to-clipboard
- YAML gen: deep-merge controller.yaml.example template with customer overrides
- Template fetcher: background goroutine refreshing template from Gitea repo
- Navigation: Dashboard / Configurations tabs on all pages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hub now tracks controller_url from reports, periodically checks the Gitea
registry for the latest controller image version, and shows a "Trigger Update"
button on the customer detail page that proxies to the controller's self-update
API endpoint using the shared API key.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
admin