e07d90f0f4
Pairs with agent v0.119.0 and is useless without it. A managed box's LAN IP was not shown anywhere in the hub, because nothing reported it — the host report carried no address of any kind. The only IP reachable from the UI at all was the WireGuard one, on /offsite's peer table keyed by pubkey, so an operator could go peer->host and never host->peer, which is the direction anyone actually asks in. The host page grows a Network card: every routable address the box holds, one row per (interface, address), plus a WireGuard row. On demo-felhom that is vmbr0 192.168.0.162/24 and tailscale0 100.70.170.35/32 — with the PVE web console at https://<the LAN address>:8006, the thing the operator wanted and could not get. WireGuard is rendered as TWO facts, deliberately. WGAssignedIP is the hub's own allocation (wg_peers, authoritative desired state); WGConfirmed is whether the box reports actually holding it. Showing the allocation alone would make a peer that was never applied look healthy — the same shape as reading a timestamp that records an attempt as if it recorded a result. The split is keyed on the ALLOCATION, not the interface name: wg-felhom is the agent's current unit name, and a UI keyed on that string would silently mis-render the day it changes. An old agent renders UNKNOWN, never "no addresses". Below agent 0.119.0 the field is absent from the wire, and an absent signal is not a negative result — the page says so and names the version needed. Rendering an empty list there would have stated something false about the host. No new store table and no new ingest path: the report is already stored opaquely and GetWGPeerForHost already existed with no UI consumer. This is parse + render. The report fixture in the tests is the REAL wire — the addresses block copied out of `felhom-agent --selftest=hub` on demo-felhom running 0.119.0. Tests 559 -> 566; four red-proofs (inert view-model, unconditional confirmation, the old-agent branch, and the drift case) each run, observed failing, reverted.
46 KiB
46 KiB
REUSE.md — felhom.eu (hub + website + scripts)
Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid. Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper. Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.
1. Canonical helpers (MUST reuse — do not reinvent)
Report ingest & API auth (hub/internal/api/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Handler).checkAuthCustomer |
hub/internal/api/handler.go (~L94) | (r) (customerID string, isGlobal, ok bool) |
Bearer auth for controller-facing endpoints (global key OR per-customer key) | Global key → ("", true, true): caller must then trust body customer_id. Constant-time compare on global key. |
(*Handler).checkAuthHost |
hub/internal/api/handler.go (~L119) | (r) (hostID, customerID string, isGlobal, ok bool) |
Bearer auth for agent-facing endpoints (global OR per-host key) | Sibling of checkAuthCustomer — do NOT mix the two token namespaces. Global key requires the host row to already exist (see handleHostReport). |
(*Handler).handleEvent + allowedEventTypes |
hub/internal/api/handler.go (~L1115 / ~L1063) | POST /api/v1/event |
The ONLY controller→hub structured-event ingest | Unknown event_type → 400 (add to the map FIRST). Accepted severities: info/warning/error/critical (critical since v0.31.0); anything else coerces to "info" — exact-match lowercase ("Critical" coerces). Tests: hub/internal/api/event_test.go. |
(*Handler).handleHostReport |
hub/internal/api/handler.go (~L464) | POST /api/v1/host-report |
Agent heartbeat ingest: denorm + guest upsert | Body cap via LimitReader; per-host key enforces host_id match (403 on mismatch); received_at is the dead-man's-switch. |
(*Handler).handleConfigRetrieve |
hub/internal/api/handler.go (~L1484) | GET /api/v1/config/{id}, header X-Retrieval-Password |
Canonical password-gated retrieval endpoint | Constant-time compare vs cfg.RetrievalPassword; 404-before-401 ordering. handleArtifactManifest mirrors it EXACTLY — keep them in lockstep. Also the Day-0 claim entry point: calls claimEngine.EnsureIssued + bakes the hash via configgen.Generate(…, claimState). |
claim.Engine |
hub/internal/claim/engine.go | EnsureIssued / Resend / RequestReset / MarkClaimed (all take *store.CustomerConfig) |
Customer-claim code engine (v0.50.0, F-4) | Stores bcrypt(code) ONLY — plaintext lives just in the email send. EnsureIssued is idempotent (never rotates/re-sends an existing row). Wired via api.SetClaimEngine + web.SetClaimEngine; the Mailer seam is *notify.Dispatcher. |
(*Store).RotateClaimCode / GetClaim / MarkClaimed |
hub/internal/store/store.go | claim-state CRUD | customer_claims row (v0.50.0) |
RotateClaimCode bumps generation (single active code) + PRESERVES claimed_at (reset never un-claims); MarkClaimed is set-only. |
configgen.Generate |
hub/internal/configgen/configgen.go (~L16) | (templateYAML string, cfg *store.CustomerConfig, claimState *store.ClaimState) (string, error) |
Generate a customer's controller.yaml | The 3rd arg (nil-safe) bakes web.claim_code_*. The REAL config-retrieve path issues+emails first (EnsureIssued); the preview/DR paths bake read-only via store.GetClaim. |
notify.FormatClaimEmail / (*Dispatcher).SendClaimEmail |
hub/internal/notify/{templates,dispatcher}.go | (kind, customerID, email, domain, code) |
Hungarian claim/reset/claimed emails | kind ∈ claim|reset|claimed. The dispatcher method IS the claim.Mailer. Never log the code. |
writeJSON |
hub/internal/api/dr.go (~L25) | (w, code int, v any) |
JSON responses in api package | Only used in dr.go so far; prefer it over ad-hoc byte-writes for new endpoints. |
(*mailRateLimiter).allow |
hub/internal/api/mail.go (~L48) | (key string) bool |
Per-key token-bucket rate limiting | Refill = perMinute/60 per sec, burst = perMinute; in-memory (lost on restart, accepted). now is injectable for tests. |
Alerting / Resend / dispatcher (hub/internal/notify/, monitor/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Dispatcher).ProcessEvent |
hub/internal/notify/dispatcher.go (~L88) | (customerID, eventType, severity, message, detailsJSON, source) |
THE notification pipeline (operator + customer channels, cooldowns, prefs) | Call in a goroutine (handlers do go d.ProcessEvent(...)). No Resend key → silent no-op. eventType=="test" bypasses prefs/cooldown (and since v0.71.0 also mails the operator). *_recovered routes via the explicit recovery branch BEFORE the severity gate (v0.71.0) — see processRecovery. |
(*Dispatcher).processRecovery |
hub/internal/notify/dispatcher.go (~L160) | internal | node_recovered/host_recovered routing (audit F11) |
Operator always (1 h cooldown); customer iff PAIRED — a customer-channel sent stale/down row newer than the last sent recovery (store.LastCustomerSentAt); enabled_events deliberately ignored for recovery; timestamp ties → no mail (flap-safe). Severity stays info — never "fix" that by widening severityNotifies. |
severityNotifies |
hub/internal/notify/dispatcher.go (~L77) | (severity string) bool |
Deciding whether a severity emails | warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress). Recovery mails exist DESPITE this gate (eventType branch), not through it. |
priorityHeaders |
hub/internal/notify/dispatcher.go (~L56) | (severity string) map[string]string |
High-priority mail-client nudge (audit F14-light) | error/critical → X-Priority: 1 + Importance: high; everything else nil — a warning/info mail must NOT masquerade as urgent (red-proofed). |
sendEmailFn seam / sendEmail |
hub/internal/notify/dispatcher.go (~L33 / ~L300) | func(to, subject, textBody string, headers map[string]string) error |
Test seam for all sends; Resend POST | Signature grew a headers param in v0.71.0 — payload carries "headers" only when non-empty. Tests capture recipient+subject+headers through the seam. |
FormatOperatorEmail / FormatCustomerEmail |
hub/internal/notify/templates.go (~L24 / ~L118) | (...) (subject, body) |
Operator (English) / customer (Hungarian) email bodies | Customer messages come from the customerMessages map — add the Hungarian text when adding an event type. Budapest TZ via package init(). Operator icon is eventType-aware: *_recovered → ✅ (severity is the fallback). |
monitor.EventNotifyFunc |
hub/internal/monitor/staleness.go (~L14) | func(customerID, eventType, severity, message, detailsJSON, source) |
Decoupling checkers from notify; wired to dispatcher.ProcessEvent in main |
May be nil — always nil-check before calling (all checkers do). |
(*Store).LogNotification |
hub/internal/store/store.go (~L433) | (customerID, eventType, severity, message, status, errorMsg, channel) |
Audit trail of every send attempt (sent/failed, per channel) | Log BOTH success and failure (dispatcher does). Since v0.71.0 these rows are also the recovery PAIRING evidence — never prune them casually. |
(*Store).LastCustomerSentAt |
hub/internal/store/store.go (~L815) | (customerID, eventTypes []string) (time.Time, bool, error) |
Pairing-evidence query (max customer-channel sent created_at over types) |
Uses the (customer_id, created_at DESC) index. Empty type list → (zero, false, nil). |
(*Store).SeedNotificationPrefs |
hub/internal/store/store.go (~L850) | (customerID, email, enabledEvents) (seeded bool, err) |
Claim-time prefs seeding (audit F12) | INSERT OR IGNORE — never an upsert (red-proofed); empty email = no-op. Customer edits go through SaveNotificationPrefs, seeds NEVER do. |
App-mail passthrough (hub/internal/mailrelay/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
mailrelay.Sender / (*ResendSMTP).Send |
hub/internal/mailrelay/relay.go (~L24 / ~L50) | Send(ctx, raw []byte, mailFrom, rcptTo) error |
Raw-MIME passthrough to Resend SMTP | Deliberately separate from notify's HTTP path — parse-then-API drops inline CID images (spike-proven). Do NOT "unify" them. Delivery verdict lands at DATA-close. |
mailrelay.FromDomain |
hub/internal/mailrelay/relay.go (~L130) | (raw []byte) (string, error) |
From-HEADER domain extraction for allowlisting | Header domain, not envelope — Resend checks the header. |
Web auth / session / CSRF (hub/internal/web/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Server).RequireAuth |
hub/internal/web/server.go (~L359) | (next http.Handler) http.Handler |
Session-cookie OR Basic-auth gate for all web routes | Empty effective hash disables auth entirely (dev mode). Browsers → /login redirect; JSON-ish requests → 401. |
(*Server).effectivePasswordHash |
hub/internal/web/server.go (~L118) | () string |
THE single source for the operator login hash — call this, never read configPasswordHash |
Precedence: hub_settings DB override (set via Configuration UI) wins, else the hub.yaml auth.password_hash seed. ConfigMap = break-glass reset. Change it via POST /configuration/password (handleChangePassword). |
(*Server).validateCSRF |
hub/internal/web/server.go (~L446) | (r) bool |
CSRF check — enforced centrally in web.ServeHTTP for every non-GET |
No session cookie → returns true (Basic-auth path is exempt). New POST routes get CSRF for free; forms MUST embed csrfField. |
(*Server).csrfField |
hub/internal/web/server.go (~L483) | (r) template.HTML |
Hidden _csrf input for HTML forms |
Pass into template data on every form-rendering handler. |
(*Server).CleanupSessions |
hub/internal/web/server.go (~L110) | (ctx) — goroutine |
Expired-session sweeper | Started once from main; 15-min tick. |
Status tokens & template funcmap (hub/internal/web/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
funcMap in web.New |
hub/internal/web/server.go (~L67) | template.FuncMap{...} |
ALL template helpers (timeAgo, statusColor, json, hubVersion…) |
Add new template funcs HERE, nowhere else. Tested by hub/internal/web/funcmap_test.go + render_test.go. |
inline_confirm_js partial (v0.52.0) |
hub/internal/web/templates/inline_confirm.html | {{template "inline_confirm_js"}} in the page <head> |
Inline "question + Igen/Mégse" confirm for consequential buttons — data-confirm="…" on the button, or felhomConfirm(el, q, onYes) from JS |
NEVER native confirm()/prompt() — OS-modals freeze browser automation (F-16). scripts/hub_confirm_gate.py asserts zero, but is not wired to run (R-29) — so the rule holds only as long as you keep it. Uses requestSubmit so formaction sub-buttons riding a parent form work. NOT for the danger-zone typed-confirm cascade. |
timeAgo |
hub/internal/web/server.go (~L603) | (t time.Time) string |
Human-relative timestamps in UI | — |
statusColor |
hub/internal/web/server.go (~L630) | (status string) string |
Status → design-system-v2 token (nominal/warn/crit/neutral) | Class SUFFIX only, never inline color (D4). Exception-color principle: healthy = blue/neutral. |
(*Server).hostStatus + hostStatusClass/hostStatusLabel |
hub/internal/web/hosts.go (~L16/34/48) | (lastReport *time.Time) string |
Host liveness badge | Uses the SAME threshold as HostStalenessChecker (down = 2× stale) — never invent a second definition. |
parseSQLiteTime |
hub/internal/store/store.go (~L1160) | (s string) time.Time |
Parsing ANY timestamp read from SQLite | modernc/sqlite returns multiple formats; raw time.Parse will intermittently zero out. Always use this. |
compareVersions |
hub/internal/web/server.go (~L571) | (a, b string) int |
X.Y.Z comparisons in web (floor checks, update-available) | Returns 0 on parse error — unparseable compares as "equal" (see §3). |
Host views & lifecycle / offsite endpoints (v0.47.0, hub/internal/web + store)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Server).hostDetailData |
hub/internal/web/hosts.go (~L282) | (host *store.Host, r) map[string]interface{} |
The ONE view-model builder for the shared host_detail_body sub-template (standalone /hosts/{id} + customer Host tab) |
Booleans/counts only for DR/escrow; carries Deletable (= status != "ok") which gates the danger-zone card. Never add a secret field. |
parseHostAddresses + (*Server).hostNetwork / hostNetworkView (v0.85.0) |
hub/internal/web/hosts.go | (reportJSON) []hostAddressView · (host, reportJSON) hostNetworkView |
The host page's Network card: every routable address the box holds + its WireGuard allocation | Needs agent >= 0.119.0 (minAgentForAddresses); below it the wire has no addresses key and the card renders UNKNOWN, never "no addresses" — an absent signal is not a negative result. WireGuard is TWO facts: the hub's allocation (GetWGPeerForHost, authoritative) AND whether the box confirms holding it — the allocation alone cannot distinguish a live tunnel from a peer that was never applied. The WG row is split out by comparing against the ALLOCATION, never by matching the interface name wg-felhom, which is a unit name that can change. |
(*Store).GetHostRecoveryMeta + (*Server).handleHostRevealRecoveryCredential |
hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | (hostID) (*HostRecoveryMeta, error) · POST /hosts/{id}/reveal-recovery-credential |
The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | Use GetHostRecoveryMeta on any page-render path — its struct and its SELECT both omit the secret column, so it cannot leak one; GetHostRecoveryCredential (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE recovery_credential_revealed event via SaveEvent and calls NO dispatcher (the handleRequestLogTail shape). api/handler.go handleAdminGetRecoveryCredential (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
host_detail_body sub-template |
hub/internal/web/templates/host_detail_body.html | {{template "host_detail_body" .}} |
Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with {{.HostID}} (the customer page renders N instances). |
(*Store).ListHostsByCustomer |
hub/internal/store/store.go (~L1620) | (customerID) ([]Host, error) |
A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
(*Server).configFormData (v0.49.0) |
hub/internal/web/configs.go (~L430) | (r, isNew, cfg, overrides, errMsg) configFormView |
The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | overrides=nil → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
config_form_body sub-template (v0.49.0) |
hub/internal/web/templates/config_form_body.html | {{template "config_form_body" <configFormView>}} |
Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after </form> — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight <script>. |
(*Store).CountHostArtifacts / DeleteHost |
hub/internal/store/store.go (~L1640/~L1690) | (hostID) (HostArtifacts, error) / (hostID, deleteEscrow bool) error |
Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (ErrHostEscrowPresent, tx never starts). log_bundles die by scope_id == host_id ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
(*Server).commitCustomerReset (v0.69.0) |
hub/internal/web/customer_reset.go (~L165) | (ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError |
THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the customer_resets journal |
Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). purgeEscrow governs ONLY whether PurgeCustomerResetDBState destroys retained custody: standalone RESET passes the operator's escrow_ack; the DELETE cascade passes false so custody dies exactly once, in its leg 3. Returns a resetLegError carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
(*Store).CustomerResidue / PurgeCustomerResidue (v0.70.0) |
hub/internal/store/customer_delete.go | (customerID) (*CustomerResidue, error) / (customerID) error |
Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | GetCustomers() is REPORT-derived — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared residueQueries list so a table can never be counted-but-not-purged. Includes appliance_registrations + selfbind_tokens (credential-bearing, not telemetry). NEVER touches events, notification_log, host_deletions, customer_resets. |
(*Server).handleCustomerDelete / handleCustomerDeletePreview (v0.69.0) |
hub/internal/web/customer_delete.go | (w, r, customerID) |
THE customer offboarding entry: the guided full-teardown cascade hosts → RESET → purge (R-25b). GET = live inventory JSON for the dialog, POST = the cascade |
There is NO shallow delete path any more — the old handleConfigDelete is gone; do not reintroduce one. Every gate (3 acks, typed customer-id, stale host-count, ONLINE-host refusal) runs BEFORE any write, so a refusal has zero side effects. Leg order is load-bearing twice over: ruling 3 (RESET never sees a host row) and custody purged exactly once, in leg 3. A failed leg retains the journal — a re-run resumes and must pass every gate again. |
(*Store).ListWGEndpoints / DeleteWGEndpoint |
hub/internal/store/wg.go (~L64/~L86) | () ([]WGEndpoint, error) / (endpointID) error |
The /offsite endpoint-management surface | GetWGEndpoint (lowest id, LIMIT 1) stays THE allocation/sync endpoint — do not switch allocator/reconciler/desired-state to the list without the wg_peers.endpoint_id migration arc. Peers-in-subnet guards live in hub/internal/web/offsite.go. |
Artifact manifest / Day-0 trust root
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Store).GetArtifactManifest / SetArtifactManifest |
hub/internal/store/store.go (~L933 / ~L944) | () ArtifactManifest / (m) error |
The DB-backed (hub_settings) Day-0 artifact record | This is the checksum TRUST ROOT the host-bootstrap verifies against — distinct from Gitea, which only stores bytes. |
(*Store).GetOperatorPasswordHash / SetOperatorPasswordHash |
hub/internal/store/store.go (~L1350) | () string / (hash) error |
The DB-backed (hub_settings) operator login password override | Read via Server.effectivePasswordHash(), not directly. "" = no override (config seed authoritative). Store the bcrypt hash, never the plaintext. |
(*Server).handleSetArtifacts + resolveArtifactSHA |
hub/internal/web/configs.go (~L644 / ~L680) | POST /configuration/artifacts |
Operator UI to vouch artifact versions | With a Gitea client the sha is fetched AUTHORITATIVELY (submitted sha ignored); fetch failure refuses the save. Manual sha only in the no-creds fallback. |
(*gitea.Client).ListVersions / FileSHA256 |
hub/internal/gitea/gitea.go (~L47 / ~L72) | (ctx, pkg) ([]string, error) / (ctx, pkg, ver, file) |
Read-only Gitea generic-package metadata | sha comes from package metadata — artifact bytes are never downloaded. Newest-semver-first sort. |
(*Server).artifactChoices |
hub/internal/web/server.go (~L155) | (ctx, pkg, file) []artifactChoice |
Version+sha dropdown data | nil Gitea client / unreachable → nil → UI degrades to manual entry. One bad version drops itself, not the list. |
(*Handler).handleArtifactManifest |
hub/internal/api/handler.go (~L1550) | GET /api/v1/artifacts/{id} |
Serving the vouched set to the bootstrap script | Auth mirrors handleConfigRetrieve exactly. Unset manifest = empty fields, not an error. |
normalizeFloorInput / normalizeSHA256 |
hub/internal/web/configs.go (~L27 / ~L627) | (raw string) (string, bool) |
Validating operator-typed versions / shas | Empty string is VALID (means "clear"). Reuse for any new version/sha form field. |
PBS DR tier / tenantsync (hub/internal/tenantsync/, hub/internal/web/pbsdr.go, hub/internal/api/pbsdr.go)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
tenantsync.Client (Provision/Reissue/Fingerprint) |
hub/internal/tenantsync/client.go | (ctx, customerID) (*Result, error) |
ep0 per-customer PBS tenancy over the pinned-SSH forced-command channel (the wgsync twin) | Result.TokenSecret is transient custody → SaveHostPBSSecret immediately, never log the struct. Error paths NEVER embed stdout (the secret channel) — do not "improve" diagnostics by quoting the response. ErrTokenExists is typed: provision refuses an existing token; re-issue is the explicit path. |
(*Store).SaveHostPBSSecret / ConsumeHostPBSSecret |
hub/internal/store/pbsdr.go | (hostID, value) / (hostID) (string, error) |
HOST-scoped consume-once secret (the one_time_secrets host twin) | Same-tx mark-consumed; re-save resets consumption (re-issue supersedes). The agent consumes via POST /api/v1/hosts/{id}/pbs/consume-token (hub/internal/api/pbsdr.go). |
offsite.DeliveryStateFor (+ DeliveryStatus) |
hub/internal/offsite/delivery.go | (st, customerID) (DeliveryStatus, error) |
THE R-70 offsite last-mile detector — one implementation for every consumer (customer card deliveryViewFor, monitor.OffsiteDeliveryChecker event + R-71c heal) |
Precedence: applied (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + StaleStagedSince flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
(*Store).GetOneTimeSecretInfo / LastEventAt / LatestReportOffsitePresence / CountReportsOffsiteSince |
hub/internal/store/store.go | (customerID) (*OneTimeSecretInfo, error) / (customerID, eventType) (time.Time, error) / … |
Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | GetOneTimeSecretInfo never selects the value column — keep it that way. SetOneTimeSecretTimesForTest is the back-dating seam (PBSDR pattern). |
monitor.OffsiteDeliveryChecker + OffsiteReissuer |
hub/internal/monitor/offsite_delivery.go | NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger) |
R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in maybeHeal: re-reads the secret row at act time and refuses over an UNCONSUMED row — SaveOneTimeSecret clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
monitor.RestoreTestChecker + assessRestoreProven |
hub/internal/monitor/restoretest.go | NewRestoreTestChecker(st, onEvent, logger); .Check() |
R-85: turns a restore-test result into a SIGNAL — it was a [WARN] log line and nothing else, even for the tier already being tested |
TWO event types, never merged: restore_test_failed (broken now, error) vs restore_test_stale (unverified — not known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. Anchored on R-81 (assessRestoreProven reuses backupAssessment/verdict*): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: no customerMessages entry — do not add one without copy review. |
(*Server).applyPBSDR + mergePBSDR/readPBSDR |
hub/internal/web/pbsdr.go | (ctx, r, cfg) error |
The config form's DR-tier section → HOST desired_json pbs_dr descriptor + generation bump |
Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by cfg.DRTier (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
(*Server).pbsdrProvisionAtom + PBSDRAutoProvision |
hub/internal/web/pbsdr.go | (ctx, customerID, host, storageID) (blocked string, err error) / (ctx, customerID) |
The shared fresh-provision cascade atom; the WG-registration hook target (api SetWGRegisteredHook, wired in hub/cmd/hub/main.go when tenantsync is on) |
blocked != "" = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
cfg.DRTier + offsite coupling |
hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field dr_tier (formBool helper). |
pbsdrheal.Reconciler + NewActions |
hub/internal/pbsdrheal/reconciler.go | NewReconciler(st, act, logger) · RestrictToHost(hostID) · Run(ctx)/Trigger() |
PBS-DR SELF-HEAL: re-arms a consumable secret for a box stuck in waiting_secret/consumed_failed after losing its converged marker (re-install/rollback). From SPIKE-pbsdr-selfheal-2026-07-15. |
Primary heal = re-stage the stored secret (no ep0 call, NO generation bump — a bump = agent refetch loop). Escalate to Re-issue only when no stored secret / consumed_failed. NEVER re-run pbsdrProvisionAtom (refuses ErrTokenExists) or blind-timer Re-issue (hash/gen thrash). Converged/disabled/verify_failed/DR-OFF = no-op. Debounce ≥2 distinct reports. PBSDRHEAL_ONLY_HOST scopes a supervised rollout. Fake seam: fakeActions in reconciler_test.go. |
(*Store).RestageHostPBSSecret + PBSDRHealStates |
hub/internal/store/pbsdr.go | (hostID) (restaged bool, err) / () ([]PBSDRHealRow, error) |
The self-heal store primitives: clear consumed_at IFF a row exists (re-arm the SAME value); the fleet work-set query (descriptor enable/provision + latest report pbs_dr.state + id) |
RestageHostPBSSecret does NO insert, NO value change, NO generation bump (restaged=false = no row → caller escalates). PBSDRHealStates mirrors GetHostOOBStates' latest-report-per-host join; malformed JSON → zero values, never an error. |
(*Server).ReissuePBSDR |
hub/internal/web/pbsdr.go | (ctx, customerID) error |
The non-HTTP core of the operator Re-issue button — the self-heal reconciler's escalation seam (satisfies pbsdrheal.Reissuer) |
Reuses tenantsync.Reissue + SaveHostPBSSecret + descriptor bump — NOT a re-run of pbsdrProvisionAtom. Keep in lockstep with the tail of handlePBSDRReissue (which is unchanged; the operator button's 303/400 codes are pinned by tests). |
parseHostCapabilities + capabilityView |
hub/internal/web/hosts.go | (reportJSON) []capabilityView |
Host-page capability chips (ok/degraded/inactive) | inactive (agent v0.86.0) = badge-neutral, NEVER warn/error — disabled ≠ degraded; unknown future statuses fall to the degraded styling (surface, don't hide). capabilitiesNeedDRMigration keys the pre-v1.15.0 migration hint on pbsdr-* + "binary not found". |
Config generation & secrets hygiene (hub/internal/configgen/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
configgen.Generate |
hub/internal/configgen/configgen.go (~L16) | (templateYAML, cfg) (string, error) |
Producing a customer controller.yaml | Programmatic overrides (customer id/hub url/api_key) ALWAYS win over config_json; fresh session secret per generation. |
configgen.RandomHex |
hub/internal/configgen/configgen.go (~L110) | (n int) (string, error) |
crypto/rand hex tokens (API keys, session secrets) | — |
configgen.RandomPassphrase |
hub/internal/configgen/passphrase.go (~L35) | (wordCount int) (string, error) |
Human-dictatable Hungarian passphrases (retrieval passwords) | ~29K-word embedded list; 5 words ≈ 74 bits. |
(*Store).EffectiveMinControllerVersion |
hub/internal/store/store.go (~L960) | (customerID) string |
Resolving the floor that actually applies (per-customer → global) | "" = no floor (Phase 2 inert). |
Assets, store misc, scheduling
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
assets.Manager (New, ServeFile, ReSeed) |
hub/internal/assets/assets.go (~L44/190/122) | seed-dir → PVC sync + manifest | Serving app logos/screenshots to controllers | ServeFile sanitizes to filepath.Base (no traversal). isAssetFile enforces the naming convention. |
copyFile |
hub/internal/assets/assets.go (~L259) | (src, dst) error |
THE atomic file write (tmp + rename) in the hub | Copy this shape for any new on-disk write. |
fileSHA256 |
hub/internal/assets/assets.go (~L244) | (path) (string, error) |
Streaming sha256 of a file | — |
(*Store).SaveEvent |
hub/internal/store/store.go (~L1003) | (...) (int64, error) |
Persisting ANY event (controller or hub source) | Pair with dispatcher/onEvent — saving alone never notifies. |
(*Store).RequestLogTail / GetPendingLogTailRequests / SaveAppLogTail |
hub/internal/store/logtail.go | pending-intent + consume-once fulfillment | THE ACK-flag pull pattern for hub→box requests (copy for any new one) | SaveAppLogTail clears the request in the SAME tx (consume-once) + prunes to last 2 per (customer,app); the hub NEVER connects into a box |
(*Store).RequestLogBundle / PendingLogBundleRequest / SaveLogBundle / PurgeExpiredLogBundles |
hub/internal/store/logbundle.go | component (controller/agent) log pulls — the v0.46.0 sibling of logtail.go | box-component debug-ring pulls; gzip custody, newest-3, 72 h TTL on the 60 s sweep | scope = customer_id (controller/report ACK) vs host_id (agent/heartbeat envelope); SaveLogBundle runs the SECRET GATE fail-closed (blocked flag row, no payload) and clears the request in the same tx; [REDACTED]/checksums pass by design |
upsertAppIssue dismissal/context semantics |
hub/internal/store/telemetry.go | ON CONFLICT CASE guards | Issue dismissal + first-capture-wins context | Un-dismiss ONLY on excluded.last_seen > dismissed_at; context adopted only while stored one is empty — do not "simplify" either CASE (red-proofed) |
store.GuestID |
hub/internal/store/store.go (~L1268) | (hostID string, vmid int) string |
Canonical guest primary key | Never hand-concatenate host+vmid. |
(*Store).GetHostReportsSince + GetFirstHostReportAt + monitor.newestBackupEvidence |
hub/internal/store/store.go, hub/internal/monitor/deadline.go | (customerID, since) ([]HostReportRow, error); (customerID) (time.Time, error); (rows, now) (time.Time, bool) |
Asking "when did the hub last SEE evidence of X?" instead of "what does the latest report say?" — the R-81 anchor. The agent's reporters are point-in-time and forget across a restart; the hub retains ~90 d of host-reports and does not. | The three go together: window scan + first-contact anchor + a bounded lookback (backupEvidenceLookback). Never judge a report-derived absence on the LATEST report alone — that is the bug class R-81 fixed for the third time. The scan early-exits on sufficiently-fresh evidence, so don't reorder rows away from newest-first. |
scheduleDaily |
hub/cmd/hub/main.go (~L449) | (ctx, name, "HH:MM", fn, logger) |
Daily jobs in Europe/Budapest (prune etc.) | Blocking — run as goroutine. parseHM returns 0,0 (midnight) on bad input. |
2. Canonical patterns (copy structure from THE named file)
| Pattern | Canonical file | Key traits |
|---|---|---|
| Monitor checker | hub/internal/monitor/staleness.go | Seed state on construction WITHOUT emitting events; in-memory states map under mutex; periodic Check(); emitTransition = SaveEvent then nil-checked onEvent; cleanup of vanished IDs. HostStaleness/HostDisk/HostLeaf/HostCapability/StorageFill all follow it. |
| API endpoint | hub/internal/api/handler.go ServeHTTP (~L139) + any handler |
Path switch in ServeHTTP; first line of every handler = checkAuth{Customer,Host}; io.LimitReader body cap; typed anonymous payload struct; explicit 4xx strings. |
| Web POST action | hub/internal/web/configs.go handleSetGlobalFloor (~L602) |
CSRF enforced centrally in web ServeHTTP; validate via normalize* helper; POST-redirect-GET with ?flash= token; log INFO on success. |
| Optional dependency injection | hub/internal/web/server.go Set* setters (~L131–148) / api handler SetDispatcher etc. |
Constructor takes hard deps; optional subsystems wired via SetX after construction; nil = graceful degradation (never panic). |
| Seam-injected sender for tests | hub/internal/notify/dispatcher.go sendEmailFn (~L33) |
Function-field defaulting to the real impl in the constructor; tests overwrite it. Same idea: mailRateLimiter.now, mailrelay.Sender fake. |
| Immediate-sync notify (per plane) | agent plane: poke.Notifier (web.Server.poke / api.Handler.poker via SetPoke/SetPoker) — controller plane: intent.Hub.Bump (web.Server.bumpIntent, api.Handler.intentHub) |
EVERY desired-state mutation fires the RIGHT plane's notifier AFTER the successful store write, never on an error path (fire-after-commit). Agent-plane pokes a HOST when its generation moved (SetHostDesired/Bump*HostDesired); controller-plane bumps a CUSTOMER on a controller-visible change. Both receivers COALESCE bursts — add NO hub-side dedup. Deliberate non-sites need a documented reason (undeliverable pre-tunnel, transport removed, or no generation bump → the 60 s ticker is the pickup path). Both seams nil-safe: unset = the ≤15-min cycle still reconciles. Full site inventory: REPORT.md audit table (v0.63.0). |
| Website page | website/index.html | UTF-8 with BOM; shared <nav>/<footer> byte-identical across pages (only class="active" differs); two-tone H1 = <h1>…<span>accent</span></h1>; all styling in website/assets/site.css tokens (:root) — zero embedded <style>; ?v=N cache-bust on site.css/icons.svg; umami snippet; no CDN fonts; no emoji (sprite icons.svg instead). |
| Gate script | scripts/site_gates.py | Byte-level mechanical gates (BOM, emoji codepoint ranges, nav/footer diff, analytics, banned tokens, cache-bust); run python scripts/site_gates.py after ANY website change; non-zero exit on failure. |
| Fetch-validate-install (shell) | scripts/felhom-host-install.sh step_agent_install (~L1108) |
fetch_raw to mktemp → syntax-check (bash -n) → install -m0755 -o root -g root → only then activate; guarded-mkfs wrapper installed BEFORE the sudoers that references it (ordering is the safety property). All mutations through run() (dry-run aware). |
| Install-profile gate (shell) | scripts/felhom-host-install.sh --mode appliance|byo (GL-2, v1.10.0) |
Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, FELHOM_INSTALL_STATE_DIR override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). |
| Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh _uninstall_statement + harness GL4-D (v1.11.0) |
Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by run_uninstall; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain umount only, never -l/-f, never any format op under /mnt/felhom-drives. |
| Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout /website/ + /scripts/, --link=current) + init container waits for first sync; nginx serves current/website; push to main = deployed, no image build. |
| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via secretKeyRef to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). report-api (the operator bearer, v0.53.0) is deliberately NOT optional: — a missing Secret fails Ready instead of booting an unauthenticatable hub. scripts/manifest_bearer_gate.py (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): gitea-creds is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
| Hub deploy (GitOps) | manifests/hub.yaml image: (~L129) |
Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. |
3. Dangerous lookalikes — do NOT reuse
| Trap | Why it bites | Use instead |
|---|---|---|
A plain missed bool for a report-derived absence (hub/internal/monitor/deadline.go) |
Collapsing the three-valued verdict re-introduces one of TWO failure modes: absence→MISSED is the 2026-07-26 cry-wolf (three boxes alarmed at once, one reached a customer channel); absence→OK means a genuinely dead box alarms NEVER, which is strictly worse. Three instances of this class so far: hub v0.12.0, v0.73.0, R-81. | `backupAssessment{verdict: verdictOK |
(*Handler).handleNotify + formatNotificationEmail + sendResendEmail (hub/internal/api/handler.go ~L1289/1624/1589) |
Legacy pre-dispatcher notification trio: no cooldowns, no operator channel, no allowedEventTypes gate, duplicate Hungarian formatter. Controller path is FROZEN until slice-10 cutover. | POST /api/v1/event → Dispatcher.ProcessEvent + notify.Format*Email |
Severity "critical" POSTed to a PRE-v0.31.0 hub |
Fixed in hub v0.31.0 (handleEvent now accepts critical). Older hubs coerce critical → "info", which never notifies — silent alert loss. Case-variants ("Critical") still coerce on every version. |
Against an old hub send warning/error; otherwise lowercase critical is safe |
compareVersions for anything security-ish (hub/internal/web/server.go ~L571) |
Returns 0 (equal) on unparseable input — a garbage version passes a floor check. gitea.compareSemver behaves differently (lexical fallback). |
Validate input with normalizeFloorInput first; then compareVersions is safe |
Inline stringData secrets à la manifests/felhom.secret.yaml |
Commits real credentials to git (healthchecks superuser pw, umami APP_SECRET/POSTGRES_PASSWORD, gitea-creds admin password still live there). | Out-of-band kubectl create secret + secretKeyRef (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md) |
kubectl apply / kubectl set image on manifests/ |
ArgoCD app felhom reverts drift on next sync; live state lies about git. |
Edit manifest in git → push → ArgoCD sync (CLAUDE.md steps 3–5) |
:latest image tag in manifests |
Re-push doesn't change the manifest → no redeploy; Synced/Rollback misreport. | Pinned version tag, bumped per deploy |
| grep/regex hunting emoji in website HTML | Windows grep false-negatives multibyte emoji (proven in D0). | python scripts/site_gates.py (codepoint-range check) |
| Adding a website page without touching site_gates.py | PAGES list (scripts/site_gates.py ~L22) is explicit — an unlisted page is silently ungated (BOM/nav/emoji drift undetected). |
Add the filename to PAGES in the same commit |
4. Seams & interfaces (testing + cross-repo)
| Interface | Defined in | Implemented by | Fakes/tests at |
|---|---|---|---|
mailrelay.Sender |
hub/internal/mailrelay/relay.go (~L24) | ResendSMTP (prod) |
fake sender in hub/internal/api/mail_test.go; hub/internal/mailrelay/relay_test.go |
Dispatcher.sendEmailFn (func seam) |
hub/internal/notify/dispatcher.go (~L33) | (*Dispatcher).sendEmail (Resend HTTP) |
hub/internal/notify/dispatcher_test.go |
monitor.EventNotifyFunc |
hub/internal/monitor/staleness.go (~L14) | closure over Dispatcher.ProcessEvent (main.go) |
hub/internal/monitor tests (captured-events func) |
api.ConfigTemplateProvider |
hub/internal/api/handler.go (~L24) | web.TemplateFetcher (Gitea-pulled controller.yaml template) |
stub providers in api tests |
api.LatestVersionProvider |
hub/internal/api/handler.go (~L31) | web.VersionChecker (registry poll) |
hub/internal/api/config_version_ack_test.go |
mailRateLimiter.now (func seam) |
hub/internal/api/mail.go (~L27) | time.Now |
hub/internal/api/mail_test.go clock injection |
web.tenancyProvisioner |
hub/internal/web/pbsdr.go | *tenantsync.Client (pinned SSH to ep0's felhom-tenantsync) |
fakeTenancy in hub/internal/web/pbsdr_test.go; in-process SSH server in hub/internal/tenantsync/client_test.go |
| Cross-repo: ep0 tenancy surface | scripts/felhom-tenantsync.sh (JSON stdin/stdout forced command) |
installed on ep0 per runbook offsite-endpoint.md §10 | provision/reissue/fingerprint ops; token secret rides stdout ONLY; the peersync script/key are untouched |
| Cross-repo: controller → hub | POST /api/v1/report (frozen) + POST /api/v1/event |
felhom-controller repo | new event types MUST enter allowedEventTypes (hub/internal/api/handler.go ~L1063) or the controller gets 400 |
| Cross-repo: agent → hub | POST /api/v1/host-report, /host-enroll, jobs/desired-state/escrow routes (handler.go ~L145–189) |
felhom-agent repo | hub/internal/api/host_test.go, desired_test.go, escrow_test.go, dr_test.go |
| Cross-repo: Day-0 bootstrap → hub | GET /api/v1/config/{id} + /artifacts/{id} (X-Retrieval-Password) |
scripts/felhom-host-install.sh (fetches + sha256-verifies against the hub-vouched manifest) | hub/internal/api/artifact_test.go |
| Cross-repo: controller ← hub assets | GET /api/v1/assets/manifest + /assets/file/{name} |
felhom-controller pulls app logos/screenshots | assets manifest sha-based change detection |
5. Extension points (where new features plug in)
- New event type: add to
allowedEventTypes(hub/internal/api/handler.go ~L1063) andcustomerMessages(hub/internal/notify/templates.go) and the customer-prefs default list if customer-notifiable. Missing the first = controller POST 400s (the known gotcha). - New monitor checker: copy hub/internal/monitor/staleness.go (§2 pattern); wire in hub/cmd/hub/main.go with an
EventNotifyFunc; severity must be warning/error/critical to notify. - New API route: switch in
api.ServeHTTP(handler.go ~L139); auth helper first line. - New web page/action: switch in
web.ServeHTTP(server.go ~L182) — non-GET gets CSRF automatically; template into hub/internal/web/templates/ (embedded FS, parsed inweb.New); new helpers into the funcMap (server.go ~L67). - New template func: funcMap in web.New only; add a case to hub/internal/web/funcmap_test.go.
- New daily job:
scheduleDailyin hub/cmd/hub/main.go + add pruning topruneAllif data grows. - New site gate: append to scripts/site_gates.py; new website pages go into its
PAGESlist. - New artifact kind (Day-0): consts
pkg*/file*(hub/internal/web/server.go ~L27),ArtifactManifestfields + settings keys (hub/internal/store/store.go ~L905),handleSetArtifacts,artifactManifestResponse(handler.go), and the install script's verify step. - New host-install step:
step_*function in scripts/felhom-host-install.sh usingrun()/fetch_raw/diehelpers; keep dry-run coverage. - New DR-recipe section:
hostHalfShape/appHalfShapeandAssembledRecipe(hub/internal/store/dr_recipe.go) — those shape structs are ALLOW-LISTS, not forward-compat: a section only the emitter knows about is stored intact and silently dropped before the operator downloads it. No error, no log, no red test. That is R-122: the controller emittedoffsite_resticfrom fork-4, all three real customers had it stored, and no delivered recipe ever contained it. Then extendTestAssembleDRRecipe_CarriesEveryEmittedSection(same commit) and, for a host-half section, the agent'sDRRecipeHostHalf+ BOTH copies oftestdata/host-report.golden.json(byte-identical, cross-repo).
6. Known duplication (observed — NOT fixed)
- Resend HTTP sender ×2:
(*Handler).sendResendEmail(hub/internal/api/handler.go ~L1589) ≈(*Dispatcher).sendEmail(hub/internal/notify/dispatcher.go ~L185) — byte-near-identical POST to api.resend.com. Kept because the handler copy serves the frozen legacy /notify path. - Hungarian customer-email formatter ×2:
formatNotificationEmail(hub/internal/api/handler.go ~L1624) vsnotify.FormatCustomerEmail(hub/internal/notify/templates.go ~L118). Legacy vs dispatcher; the legacy one lacks the per-event-type message map. - Semver compare ×2 with DIFFERENT fallback semantics:
web.compareVersions(hub/internal/web/server.go ~L571, parse error → 0) vsgitea.compareSemver(hub/internal/gitea/gitea.go ~L115, parse error → lexical). Documented as deliberate (import-cycle avoidance) in gitea.go, but the behavior drift is not. - Checker-family structural repetition: staleness.go vs host_staleness.go, and host_disk.go vs storage_fill.go (band/bandRank/bandLabel vs bandForPercent) — same skeleton re-implemented per domain; treated as the accepted §2 pattern rather than a defect.
- Duration formatting ×2:
monitor.formatDuration(hub/internal/monitor/staleness.go ~L187) vsweb.timeAgo(hub/internal/web/server.go ~L603) — different audiences (email vs UI) but overlapping logic.