The fleet half of R-39. An ep0 credential re-issue re-keys the SECRET of an existing
token, so token_id, fingerprint, datastore and namespace all come back byte-identical.
The agent re-applies on the descriptor's CONTENT HASH, so a re-issue was invisible to a
converged box: it short-circuited, never consumed the fresh secret, and served a revoked
credential while reporting `applied` — the N100 failure of 2026-07-18.
host_pbs_secrets gains a monotonic per-host `generation`, advanced by every fresh MINT and
by nothing else, stamped into the descriptor as `secret_generation`. That is now the only
field a re-key moves, and it is what re-arms the agent.
DEVIATION FROM SPEC, deliberate: the brief said to return "the new row's id (int64) …
no schema change". There is no row id — host_pbs_secrets is keyed by host_id and UPSERTed
last-write-wins, so a new row never exists, and created_at collides for two mints in the
same second. An additive counter column is the only monotonic source; it uses the repo's
existing idempotent ALTER-TABLE idiom.
RestageHostPBSSecret deliberately does NOT advance it: a re-stage re-arms the SAME secret,
the descriptor content genuinely has not changed, and a bump would cause a pointless agent
refetch loop (that method's own contract says so).
Also corrects a comment that asserted the re-issue refreshes the descriptor "with the NEW
token_id/fingerprint". That is false for a re-key, and believing it is why the descriptor
was never expected to be identical in the first place.
omitempty is load-bearing: a zero generation must not start emitting a new key into every
pre-existing descriptor, which would itself be a fleet-wide spurious re-apply.
Compatibility: agents below 0.91.0 drop the unknown JSON key and behave exactly as today —
inert, not breaking (Scenario C).
Tests: store-level monotonicity + per-host isolation + restage-leaves-it-alone; descriptor
byte-change, omitempty, and sibling-key round-trip; and a FLOW-level test driving
ReissuePBSDR against a fake that models a real re-key. Red-proof run at the assertion
level (not the compiler): commenting out the stamp makes the flow test fail with both
byte-identical blocks printed.
Let a customer bind their own freshly-installed appliance without the
operator: operator "Send self-bind link" mints a 7-day tokenized
capability link, emailed (Hungarian, sibling sender) to the customer, who
opens a public /bind/<token> page and proves two factors — the console
pairing code shown on the box screen + their retrieval passphrase — and
the hub stages the bind via the same BindAppliance (provenance
customer_selfbind). The box's ~30s appliance poll delivers.
Viktor's three rulings verbatim: console pairing code (no appliance list
ever rendered), operator-sent tokenized link, 5-attempt lockout ->
"call support". Wrong code == wrong passphrase (one generic failure, no
oracle, both factors compared unconditionally); expiry falls back to
operator-bind unchanged.
THE TRAP: one public prefix /bind/, exempt from auth+CSRF at both /login
gate sites via a single isPublicBindPath predicate (tight trailing-slash
match; ServeMux ..-cleans; handler rejects '/' in token). 9 tests
(Scenarios A-F + F1/F2); 4 red-proofs verified red-then-green (lockout,
oracle, widened-prefix, single-active). GC verdict: no appliance GC ->
the 7-day TTL stands alone. Controller/agent untouched; R-27b deferred.
Green: full hub build/vet/test (17 ok) + bash -n + hub confirm gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
The immediate-sync arc covered only operator-initiated desired-state changes;
system-initiated mutations bumped the generation silently, so a freshly onboarded
box waited a full agent tick for state the hub had already minted (observed live at
slice-C onboarding). Wire the existing, live-proven notifiers into every system site
on the correct plane — call-site wiring only, no new mechanism.
Agent plane (poke.Notifier):
- web/pbsdr.go: PBSDRAutoProvision (the observed lag), ReissuePBSDR (also lifts the
pbsdrheal reconciler escalation, zero reconciler changes), handlePBSDRReissue —
each pokes AFTER the successful SetHostDesired, never on a blocked/error path.
- api: new nil-safe Poker seam (PokeHost/PokeAllHosts + SetPoker); handleAdminSetDesiredState
pokes the target host; handleAdminSetOperatorPeer fires PokeAllHosts only when the
fleet generation bump succeeded (fire-after-commit).
- main.go: one poke.Notifier now feeds both planes (SetPoke + SetPoker).
Controller plane (intent.Hub.Bump):
- api/reissueOnReenroll: one nil-guarded bump so a long-polling controller wakes in
seconds instead of on the 15-min cycle.
Deliberate non-sites (unchanged): WG register (undeliverable pre-tunnel — the agent
fast-tick SECONDARY owns it), WG delete (transport removed), pbsdrheal Restage (no
generation bump → the 60s ticker is the pickup path). internal/pbsdrheal byte-unchanged.
Tests: 10 non-hollow tests (web async channel-synchronized fake sender; api synchronous
fake Poker) with explicit zero-count negatives; representative red-proofs per group
(A/B/C/D) run-fail-restored. Green: go build/vet/test all pass.
A generic ISO carries NO customer secret. The box registers itself at the hub
as an unclaimed appliance; the operator binds it to a customer; the hub delivers
the customer-id + retrieval passphrase ONCE; day-0 completes via the slice-A path.
Hub (v0.62.0):
- store/appliance.go: appliance_registrations keyed by (uuid, mac_set) — MAC set
is the tiebreaker (duplicate SMBIOS UUIDs); token stored as sha256 only.
Idempotent register (sticky-discard), atomic one-shot delivery, bind/discard.
- api/appliance.go: POST /appliance/register (the one unauth endpoint, per-IP
rate-limited, 256-bit token); GET /appliance/poll (404 no-oracle / 204 unbound
/ 200 deliver-once / 410 delivered). Passphrase read live, never logged.
- web/appliances.go: Hosts-page "Unclaimed appliances" section + BIND (customer
picker, host count display-only) + DISCARD; SSH host-key fingerprints; events.
- Red-proofs: one-shot delivery + register idempotency (both proven red);
404-no-oracle, sticky-discard, bind staging, render. Green + confirm gate.
Scripts (v1.19.0):
- felhom-bootstrap.sh: ONE unit, TWO modes. Direct (env has customer/passphrase)
= slice-A path, byte-identical, only branched around. Pairing (generic) =
register + poll (RestartSec=30 is the poll timer); on delivery write the env
0600 and fall through to direct. Secrets + token shredded on success.
- build-felhom-iso.sh --pairing: generic secret-free ISO, -generic filename,
manifest mode=pairing. profiles/generic.profile (new).
- test/bootstrap-modes.sh: Scenario D (direct = zero appliance calls) + pairing
register/poll + delivery handoff — all green in a debian container.
GET /api/v1/wait long-poll: the box holds an authed hanging GET; the hub
completes it the instant any operator intent bumps that customer's in-memory
generation, then the box fires its ordinary report and the ACK delivers
everything through the unchanged machinery. 240s hold with a 25s heartbeat
newline defeats the nginx 60s proxy_read_timeout with no ingress annotation;
WriteTimeout lifted per-connection via ResponseController.
- internal/intent: per-customer generation counter + waiter registry
(Bump/Wait/Close), coalescing to latest, race-closer, in-memory by design.
Red-proofs: counter-vs-queue + race-closer (run-fail-reverted).
- api/wait.go: the endpoint (per-customer only; global key 400; A cannot see B).
- web bumps after every intent write (fire-after-commit): config CRUD, claim
resend, offsite re-issue/freeze, password regen, block/unblock, floors
(global bumps all config-managed), controller log-tail + log-bundle.
- main.go: one intent hub shared by web+api; Close() before server.Shutdown.
Pairs with controller v0.140.0 (the long-poll client). Grounding:
documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md.
F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path,
single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on
re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty
(red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the
mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events:
claim_reissued_reenroll / offsite_reissued / escrow_stale.
Controller + scripts unchanged (source contradicted both premises): the controller reads escrow
prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4
root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc
F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass.
Additive host_escrow migration; SaveHostEscrow/HostEscrow gain the hash
(NULL-safe for legacy rows); GetEscrowStatusForCustomer joins hosts;
the report ACK gains escrow:{identity_blob_present,restic_pw_sha256,
created_at} (omitted without a row). Contract test mirrors the agent's
v0.79.0 emit struct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
store.GetHostOOBStates parses the agent oob heartbeat stanza. monitor/host_oob:
transition-based oob_degraded/oob_recovered warning (felhom-sshd down while the
operator peer is configured, OR config invalid) — proactive "can the operator get
in right now" signal; unconfigured OOB never alerts. Wired into the 60s sweep.
Non-hollow tests + transitions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Endpoint card + peers table (truncated pubkeys with full-value title attr,
bound peers link to /hosts/<id>); Offsite nav link in all 9 page templates;
render tests for endpoint/peers, empty, and not-configured states.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
POST /hosts/{id}/wg (per-host self-scoped; global = operator/DR path): bind /
re-key-in-place / adopt; generation bump + endpoint push ONLY on real change.
mergeWireguard injects the hub-owned block into served desired-state at READ
time (stored operator blob never touched; fail-safe unmerged on any error;
no-peer = byte-identical pass-through — existing golden test untouched+green).
handleAdminSetDesiredState rejects top-level wireguard (400). Admin DELETE of a
BOUND peer bumps the owning host. NEW golden desired-state-wireguard.golden.json
= the S3 cross-repo contract. Red-proofs a/b/c/d run + reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
internal/wgsync: x/crypto/ssh client with ssh.FixedHostKey pin (no insecure
fallback), forced-command exec, ok/applied response contract; Reconciler pushes
the FULL peer list on Trigger or 5-min tick (drift repair by construction).
internal/api/wg.go: PUT/GET /admin/wg/endpoint + POST/DELETE/GET /admin/wg/peers,
global-key-only, pubkey in body (base64 vs URL), sync ok|deferred|disabled.
main.go: WG_ENDPOINT_SSH_* env wiring, disabled-with-INFO when unconfigured.
Groups B/C/D tests incl. in-process SSH server; red-proofs b/c/d run + reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
New HostDiskChecker on the 60s sweep alerts the operator when a Proxmox host root
filesystem crosses warn (90%) / crit (95%). Born/persistent (a disk already full at
hub restart alerts on cycle 1); distinct host_disk_* event types from the guest disk_*;
critical band maps to severity error (the dispatcher only routes warning/error).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
The controller v0.90.0 channel-health checker pushes agent_channel_* events to /api/v1/event;
the allowedEventTypes allowlist rejected them (HTTP 400). Added the 8 types (operator-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Companion to felhom-agent v0.44.0. New monitor.HostCapabilityChecker (sibling of
HostStalenessChecker) reads the capabilities snapshot from the latest host report and emits
agent_capability_degraded/recovered (operator-only, 1h cooldown) on ok<->degraded transitions
for any Critical capability. store.GetHostCapabilities (MAX(id), no migration). Goldens mirror
the new capabilities field.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Operator sets a minimum controller version (FLOOR), per-customer defaulting to a
global floor; the report ACK returns the effective floor + latest_version so the
controller auto-updates to the floor when below it (latest stays the opt-in button).
- store: min_controller_version column + hub_settings global floor + Effective/
Get/SetGlobal/SetMin resolution + config/env DEFAULT_MIN_CONTROLLER_VERSION
- handler: report ACK {min_controller_version, latest_version}; LatestVersionProvider
- web: global floor editor + per-customer override form + Floor column (English)
- tests: floor resolution + ACK + render; override-precedence red-proof verified
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
New POST /api/v1/host-enroll (handleHostEnroll): X-Retrieval-Password authed,
body {customer_id} -> {host_id, api_key}. Mint-once-reuse (201 first, 200
reuse) so re-running the host-bootstrap never orphans a running agent's key;
auth checked before any mint. Backed by new Store.GetHostByCustomer
(ORDER BY updated_at DESC LIMIT 1, idx_hosts_customer).
GET /config/{id} and global-key POST /admin/hosts left untouched. Exact-match
route (path == "/host-enroll") to avoid the /hosts/ prefix collision.
Tests: host_enroll_test.go (mint/reuse/401-no-mint/404/400) + GetHostByCustomer
store test; companion red-proof verified always-mint fails the reuse assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
Tracks felhom-agent v0.39.0, which dropped role + restic_repo_coord from the host-half
drives[]. Hub reads drives as json.RawMessage (verbatim passthrough) → no store/handler
change needed. Dropped role from the store golden + drHostHalf fixture; re-synced the api
host-report golden byte-identical with the agent copy (it previously lacked dr_recipe
entirely). Added TestAssembleDRRecipe_V1DriveShape regression guard (demonstrated to fail
when the fixture re-adds role, then reverted). No behavior change; redeploy optional.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DR recipe slice (hub half), grounded in SPIKE-dr-recipe-2026-06-16. The hub
receives two additive dr_recipe halves on the existing report paths (agent
storage/guest/PBS on host-report; controller customer/apps on the controller
report), stores them PLAINTEXT in a DEDICATED dr_recipe table keyed by customer
(each half preserves the other), and AssembleDRRecipe stitches them into one
operator-readable recipe (ignore-unknown + version-skew tolerant).
View: a DR-recipe panel on the customer page + GET /customers/{id}/dr-recipe.json
download (operator-auth, no secrets to redact). Plaintext-at-rest is correct —
the recipe is the clean inverse of the retired infra-backup.
Tests: store round-trip (each half preserves the other), assemble-matches-golden,
ignore-unknown + version skew, partial halves, no-secrets sweep. Manifest tag
bumped to v0.13.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase-1 of SPIKE-infra-backup-2026-06-15. The infra-backup mechanism was dead
since slice 8C yet stored plaintext customer secrets at rest (app-secret key,
restic password, Cloudflare tokens) — a zero-knowledge violation — and its
absence made the daily expected_backup_missed email fire for healthy customers.
- Repoint monitor.CheckBackupDeadlines backup half to the agent host-report's
PBS snapshots (+vzdump): alarm only on no-backup / >26h stale / verify failed.
Keep the db_dump half. No host-report → no backup alarm (liveness owns that).
New store.GetLatestHostReportJSON. Tests incl. a companion that fails pre-fix.
- Remove the infra-backup endpoints, store methods/types, and operator panel;
/recovery now returns config_yaml only.
- migrate(): DROP infra_backup_versions/infra_backups + VACUUM (+wal_checkpoint)
to physically reclaim the plaintext pages, gated on table existence.
Flagged out-of-scope: exposed creds need operator rotation; legacy reports table
holds historical plaintext restic_password rows (separate leak, not purged here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add DELETE /hosts/{id}/jobs/{job_id} (per-host self-scoped, idempotent) so the
agent clears a job after executing or terminally rejecting it. The hub stores
the operator-signed blobs opaquely (no signing key — cannot forge or open);
the agent verifies + executes. Doc 03 §4/§6/§9 updated (operator-signed path
live; 8C wipe completes; 10B done).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Serve operator intent to authenticated hosts: PUT /admin/hosts/{id}/desired-state
(global key) bumps desired_generation; GET /hosts/{id}/desired-state + /jobs are
per-host self-scoped; the host-report envelope now carries the real generation +
has_signed_ops. New signed_jobs table + store methods. Desired-state stored/served
opaquely (agent owns the schema). Cross-repo golden (envelope + desired-state)
byte-identical with felhom-agent; doc 03 §4/§9 updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the cross-repo host-report golden byte-identical with felhom-agent
(host.cpu_temp_c). Document GET /host/metrics in doc 03 section 6 and define
slice 9 in the section 9 roadmap. No hub code change / no version bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase B (hub half) of the restore-test warning fix. The agent v0.7.0 now passes a
restore-test that emitted a benign start advisory (systemd-nesting) and carries the
warning text on the wire.
- hostRestoreTest gains warnings + warnings_recognized mirror fields (omitempty;
absent recognized => false => louder unrecognized path)
- ingest logs [INFO] passed WITH WARNINGS (recognized), [WARN] for unrecognized;
FAILED still [WARN]
- golden restore_tests[0] gains the keys, byte-identical with felhom-agent (sha256
e6999d77...); bidirectional key-set contract test round-trips them
- no dashboard widget: no host-domain dashboard surface exists yet (log+persist only,
as with pbs_snapshots) -- deferred to slice 10
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Accept + persist the now-populated host-report pbs_snapshots. hostPBSSnapshot mirror in
hostReportPayload (persisted via report_json, no schema change); a FAILED PBS verify is
logged prominently (loudest offsite-DR signal). Shared golden updated byte-identical with
felhom-agent; TestHostPBSSnapshot_GoldenContract added. Build/deploy deferred (backward-compatible).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Accept + persist the now-populated host-report backups/restore_tests. Mirror structs in
hostReportPayload; persisted via report_json (no schema change); a FAILED restore-test is
logged prominently (loudest DR signal). Shared golden updated byte-identical with
felhom-agent; bidirectional key-set tests added. Build/deploy deferred (backward-compatible).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Accept + persist the now-populated host-report storage_targets. Minimal — the
authoritative storage manifest is hub-owned (slice 10); this mirrors what the agent
observes.
- hostReportPayload.StorageTargets: full mirror of the agent's hub.StorageTarget
wire contract; persisted verbatim in report_json (no schema change); count +
WARN on disconnected targets.
- shared host-report golden updated with two populated targets; byte-identical with
felhom-agent's copy.
- TestHostStorageTarget_GoldenContract: hub half of the bidirectional key-set test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- handleHostReport: read maxHostReportBytes+1 (4 MiB const) and reject oversize with
413 instead of silent LimitReader truncation. Controller handleReport (1 MiB) is
unchanged. Test asserts 413.
- contract: hub/internal/api/testdata/host-report.golden.json (byte-identical with
felhom-agent's copy) + TestHostReport_GoldenContract drives the real handler and
asserts 200 + denorm + both guests upserted.
- CHANGELOG v0.7.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin