f5c9411e5e
The customer page's Backup card read `Snapshots 0 / Repo Size 0 MB / Integrity
Unknown` for EVERY customer, indefinitely. Measured on demo-hp 2026-08-30 while
that night's controller log said `[offbox] backup OK: 8 app(s) backed up, 67
snapshot(s), 2m14s` and the box held snapshot_count:67, repo_size_bytes:
140829678, stats_known:true.
A card reading "no backups" over a working backup is worse than no card -- the
R-88 direction of failure (degrade to NO BACKUP rather than to UNKNOWN) on the
one screen that answers "is this customer protected?".
The data was never missing. The card rendered the report's `backup` object,
whose snapshot/size/integrity fields have had no producer since slice 8C. The
live numbers are in the `offsite` object, which THIS PACKAGE already reads for
the Offsite page and which monitor.OffsiteChecker already alarms from. Proof the
bytes were arriving: the Offsite page rendered demo-hp's usage as 0.1 GB from
that very object while the Backup card said 0 MB. So this is a render fix over
an existing feed, not a new pipeline.
Not a one-line swap, because snapshot_count:0 means two opposite things --
"holds nothing" and "never measured". R-225 measured that confusion one layer
down. backup_card.go resolves a three-way ruling in Go (a {{if}} chain over
map[string]interface{} float64s cannot keep the absent/zero distinction the card
is entirely about):
no offsite object -> "No off-site data reported", and says explicitly that
this is NOT the same as "no backups"
disabled + state -> names the blocker (needs_credential)
stats_known:false -> em-dash + "never been measured". NEVER 0
stats_known:true -> the real numbers, INCLUDING a real 0
A pre-v0.225.0 controller sends no stats_known -> false -> "unknown". That
direction is pinned: upgrading the hub ahead of the fleet must not report every
un-upgraded customer as having zero backups.
The Integrity row is DELETED, not re-sourced: nothing produces it, the
controller runs no integrity check, and NotifyIntegrityOK/Failed are called from
nowhere.
RED-PROOF: restore the pre-fix card markup -> all four tests fail, reporting 67
and 134.3 MB absent from the rendered page and the Integrity row present. The
tests drive handleCustomerUnified and grep the HTML on purpose: the defect was
the template's choice of source object, so a test one layer below it would have
been green against the shipped bug.
Green gate clean: 18 packages, rc 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM
56 KiB
56 KiB
REUSE.md — felhom.eu (hub + website + scripts)
Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid. Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper. Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.
1. Canonical helpers (MUST reuse — do not reinvent)
Report ingest & API auth (hub/internal/api/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Handler).checkAuthCustomer |
hub/internal/api/handler.go (~L94) | (r) (customerID string, isGlobal, ok bool) |
Bearer auth for controller-facing endpoints (global key OR per-customer key) | Global key → ("", true, true): caller must then trust body customer_id. Constant-time compare on global key. |
(*Handler).checkAuthHost |
hub/internal/api/handler.go (~L119) | (r) (hostID, customerID string, isGlobal, ok bool) |
Bearer auth for agent-facing endpoints (global OR per-host key) | Sibling of checkAuthCustomer — do NOT mix the two token namespaces. Global key requires the host row to already exist (see handleHostReport). |
(*Handler).handleEvent + allowedEventTypes |
hub/internal/api/handler.go (~L1115 / ~L1063) | POST /api/v1/event |
The ONLY controller→hub structured-event ingest | Unknown event_type → 400 (add to the map FIRST). Accepted severities: info/warning/error/critical (critical since v0.31.0); anything else coerces to "info" — exact-match lowercase ("Critical" coerces). Tests: hub/internal/api/event_test.go. |
(*Handler).handleHostReport |
hub/internal/api/handler.go (~L464) | POST /api/v1/host-report |
Agent heartbeat ingest: denorm + guest upsert | Body cap via LimitReader; per-host key enforces host_id match (403 on mismatch); received_at is the dead-man's-switch. |
(*Handler).handleConfigRetrieve |
hub/internal/api/handler.go (~L1484) | GET /api/v1/config/{id}, header X-Retrieval-Password |
Canonical password-gated retrieval endpoint | Constant-time compare vs cfg.RetrievalPassword; 404-before-401 ordering. handleArtifactManifest mirrors it EXACTLY — keep them in lockstep. Also the Day-0 claim entry point: calls claimEngine.EnsureIssued + bakes the hash via configgen.Generate(…, claimState). |
claim.Engine |
hub/internal/claim/engine.go | EnsureIssued / Resend / RequestReset / MarkClaimed (all take *store.CustomerConfig) |
Customer-claim code engine (v0.50.0, F-4) | Stores bcrypt(code) ONLY — plaintext lives just in the email send. EnsureIssued is idempotent (never rotates/re-sends an existing row). Wired via api.SetClaimEngine + web.SetClaimEngine; the Mailer seam is *notify.Dispatcher. |
(*Store).RotateClaimCode / GetClaim / MarkClaimed |
hub/internal/store/store.go | claim-state CRUD | customer_claims row (v0.50.0) |
RotateClaimCode bumps generation (single active code) + PRESERVES claimed_at (reset never un-claims); MarkClaimed is set-only. |
configgen.Generate |
hub/internal/configgen/configgen.go (~L16) | (templateYAML string, cfg *store.CustomerConfig, claimState *store.ClaimState) (string, error) |
Generate a customer's controller.yaml | The 3rd arg (nil-safe) bakes web.claim_code_*. The REAL config-retrieve path issues+emails first (EnsureIssued); the preview/DR paths bake read-only via store.GetClaim. |
notify.FormatClaimEmail / (*Dispatcher).SendClaimEmail |
hub/internal/notify/{templates,dispatcher}.go | (kind, customerID, email, domain, code) |
Hungarian claim/reset/claimed emails | kind ∈ claim|reset|claimed. The dispatcher method IS the claim.Mailer. Never log the code. |
writeJSON |
hub/internal/api/dr.go (~L25) | (w, code int, v any) |
JSON responses in api package | Only used in dr.go so far; prefer it over ad-hoc byte-writes for new endpoints. |
(*mailRateLimiter).allow |
hub/internal/api/mail.go (~L48) | (key string) bool |
Per-key token-bucket rate limiting | Refill = perMinute/60 per sec, burst = perMinute; in-memory (lost on restart, accepted). now is injectable for tests. |
Alerting / Resend / dispatcher (hub/internal/notify/, monitor/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Dispatcher).ProcessEvent |
hub/internal/notify/dispatcher.go (~L88) | (customerID, eventType, severity, message, detailsJSON, source) |
THE notification pipeline (operator + customer channels, cooldowns, prefs) | Call in a goroutine (handlers do go d.ProcessEvent(...)). No Resend key → silent no-op. eventType=="test" bypasses prefs/cooldown (and since v0.71.0 also mails the operator). *_recovered routes via the explicit recovery branch BEFORE the severity gate (v0.71.0) — see processRecovery. |
(*Dispatcher).processRecovery |
hub/internal/notify/dispatcher.go (~L160) | internal | node_recovered/host_recovered routing (audit F11) |
Operator always (1 h cooldown); customer iff PAIRED — a customer-channel sent stale/down row newer than the last sent recovery (store.LastCustomerSentAt); enabled_events deliberately ignored for recovery; timestamp ties → no mail (flap-safe). Severity stays info — never "fix" that by widening severityNotifies. |
severityNotifies |
hub/internal/notify/dispatcher.go (~L77) | (severity string) bool |
Deciding whether a severity emails | warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress). Recovery mails exist DESPITE this gate (eventType branch), not through it. |
priorityHeaders |
hub/internal/notify/dispatcher.go (~L56) | (severity string) map[string]string |
High-priority mail-client nudge (audit F14-light) | error/critical → X-Priority: 1 + Importance: high; everything else nil — a warning/info mail must NOT masquerade as urgent (red-proofed). |
sendEmailFn seam / sendEmail |
hub/internal/notify/dispatcher.go (~L33 / ~L300) | func(to, subject, textBody string, headers map[string]string) error |
Test seam for all sends; Resend POST | Signature grew a headers param in v0.71.0 — payload carries "headers" only when non-empty. Tests capture recipient+subject+headers through the seam. |
FormatOperatorEmail / FormatCustomerEmail |
hub/internal/notify/templates.go (~L24 / ~L118) | (...) (subject, body) |
Operator (English) / customer (Hungarian) email bodies | Customer messages come from the customerMessages map — add the Hungarian text when adding an event type. Budapest TZ via package init(). Operator icon is eventType-aware: *_recovered → ✅ (severity is the fallback). |
monitor.EventNotifyFunc |
hub/internal/monitor/staleness.go (~L14) | func(customerID, eventType, severity, message, detailsJSON, source) |
Decoupling checkers from notify; wired to dispatcher.ProcessEvent in main |
May be nil — always nil-check before calling (all checkers do). |
(*Store).LogNotification |
hub/internal/store/store.go (~L433) | (customerID, eventType, severity, message, status, errorMsg, channel) |
Audit trail of every send attempt (sent/failed, per channel) | Log BOTH success and failure (dispatcher does). Since v0.71.0 these rows are also the recovery PAIRING evidence — never prune them casually. |
(*Store).LastCustomerSentAt |
hub/internal/store/store.go (~L815) | (customerID, eventTypes []string) (time.Time, bool, error) |
Pairing-evidence query (max customer-channel sent created_at over types) |
Uses the (customer_id, created_at DESC) index. Empty type list → (zero, false, nil). |
(*Store).SeedNotificationPrefs |
hub/internal/store/store.go (~L850) | (customerID, email, enabledEvents) (seeded bool, err) |
Claim-time prefs seeding (audit F12) | INSERT OR IGNORE — never an upsert (red-proofed); empty email = no-op. Customer edits go through SaveNotificationPrefs, seeds NEVER do. |
App-mail passthrough (hub/internal/mailrelay/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
mailrelay.Sender / (*ResendSMTP).Send |
hub/internal/mailrelay/relay.go (~L24 / ~L50) | Send(ctx, raw []byte, mailFrom, rcptTo) error |
Raw-MIME passthrough to Resend SMTP | Deliberately separate from notify's HTTP path — parse-then-API drops inline CID images (spike-proven). Do NOT "unify" them. Delivery verdict lands at DATA-close. |
mailrelay.FromDomain |
hub/internal/mailrelay/relay.go (~L130) | (raw []byte) (string, error) |
From-HEADER domain extraction for allowlisting | Header domain, not envelope — Resend checks the header. |
Web auth / session / CSRF (hub/internal/web/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Server).RequireAuth |
hub/internal/web/server.go (~L359) | (next http.Handler) http.Handler |
Session-cookie OR Basic-auth gate for all web routes | Empty effective hash disables auth entirely (dev mode). Browsers → /login redirect; JSON-ish requests → 401. |
(*Server).effectivePasswordHash |
hub/internal/web/server.go (~L118) | () string |
THE single source for the operator login hash — call this, never read configPasswordHash |
Precedence: hub_settings DB override (set via Configuration UI) wins, else the hub.yaml auth.password_hash seed. ConfigMap = break-glass reset. Change it via POST /configuration/password (handleChangePassword). |
(*Server).validateCSRF |
hub/internal/web/server.go (~L446) | (r) bool |
CSRF check — enforced centrally in web.ServeHTTP for every non-GET |
No session cookie → returns true (Basic-auth path is exempt). New POST routes get CSRF for free; forms MUST embed csrfField. |
(*Server).csrfField |
hub/internal/web/server.go (~L483) | (r) template.HTML |
Hidden _csrf input for HTML forms |
Pass into template data on every form-rendering handler. |
(*Server).CleanupSessions |
hub/internal/web/server.go (~L110) | (ctx) — goroutine |
Expired-session sweeper | Started once from main; 15-min tick. |
Status tokens & template funcmap (hub/internal/web/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
funcMap in web.New |
hub/internal/web/server.go (~L67) | template.FuncMap{...} |
ALL template helpers (timeAgo, statusColor, json, hubVersion…) |
Add new template funcs HERE, nowhere else. Tested by hub/internal/web/funcmap_test.go + render_test.go. |
inline_confirm_js partial (v0.52.0) |
hub/internal/web/templates/inline_confirm.html | {{template "inline_confirm_js"}} in the page <head> |
Inline "question + Igen/Mégse" confirm for consequential buttons — data-confirm="…" on the button, or felhomConfirm(el, q, onYes) from JS |
NEVER native confirm()/prompt() — OS-modals freeze browser automation (F-16). scripts/hub_confirm_gate.py asserts zero and IS wired — it is gate 3 of scripts/repo_gates.py, which the pre-push hook and CI both run (R-29 closed the wiring; corrected here 2026-08-06). Uses requestSubmit so formaction sub-buttons riding a parent form work. NOT for the danger-zone typed-confirm cascade. |
timeAgo |
hub/internal/web/server.go (~L603) | (t time.Time) string |
Human-relative timestamps in UI | — |
statusColor |
hub/internal/web/server.go (~L630) | (status string) string |
Status → design-system-v2 token (nominal/warn/crit/neutral) | Class SUFFIX only, never inline color (D4). Exception-color principle: healthy = blue/neutral. |
(*Server).hostStatus + hostStatusClass/hostStatusLabel |
hub/internal/web/hosts.go (~L16/34/48) | (lastReport *time.Time) string |
Host liveness badge | Uses the SAME threshold as HostStalenessChecker (down = 2× stale) — never invent a second definition. |
parseSQLiteTime |
hub/internal/store/store.go (~L1160) | (s string) time.Time |
Parsing ANY timestamp read from SQLite | modernc/sqlite returns multiple formats; raw time.Parse will intermittently zero out. Always use this. |
compareVersions |
hub/internal/web/server.go (~L571) | (a, b string) int |
X.Y.Z comparisons in web (floor checks, update-available) | Returns 0 on parse error — unparseable compares as "equal" (see §3). |
reportBackupCard + backupCardView + fmtBytesAuto (R-331, v0.109.0) |
hub/internal/web/backup_card.go | (reportJSON string) backupCardView / (int64) string |
THE customer-page Backup card — everything it shows about a customer's backups | Reads the report's offsite object, NEVER backup. The backup object's snapshot_count/repo_size_mb/integrity_ok have had no producer since slice 8C and rendering them told every operator every customer had Snapshots 0 (measured on demo-hp over a repo holding 67). Always consult StatsKnown before believing a zero — absent/false means "never measured", NOT "empty", and those are opposite news (R-225 measured the same confusion one layer down). Resolved in Go, not the template, because a {{if}} chain over Report's map[string]interface{} float64s cannot keep the absent/zero distinction the card is entirely about. fmtBytesAuto scales MB/GB/TB — do NOT swap in fmtBytesGB, which renders demo-hp's real 140 829 678 B as 0.1 GB. |
Host views & lifecycle / offsite endpoints (v0.47.0, hub/internal/web + store)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Server).hostDetailData |
hub/internal/web/hosts.go (~L282) | (host *store.Host, r) map[string]interface{} |
The ONE view-model builder for the shared host_detail_body sub-template (standalone /hosts/{id} + customer Host tab) |
Booleans/counts only for DR/escrow; carries Deletable (= status != "ok") which gates the danger-zone card. Never add a secret field. |
parseHostAddresses + (*Server).hostNetwork / hostNetworkView (v0.85.0) |
hub/internal/web/hosts.go | (reportJSON) []hostAddressView · (host, reportJSON) hostNetworkView |
The host page's Network card: every routable address the box holds + its WireGuard allocation | Needs agent >= 0.119.0 (minAgentForAddresses); below it the wire has no addresses key and the card renders UNKNOWN, never "no addresses" — an absent signal is not a negative result. WireGuard is TWO facts: the hub's allocation (GetWGPeerForHost, authoritative) AND whether the box confirms holding it — the allocation alone cannot distinguish a live tunnel from a peer that was never applied. The WG row is split out by comparing against the ALLOCATION, never by matching the interface name wg-felhom, which is a unit name that can change. |
(*Store).GetHostRecoveryMeta + (*Server).handleHostRevealRecoveryCredential |
hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | (hostID) (*HostRecoveryMeta, error) · POST /hosts/{id}/reveal-recovery-credential |
The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | Use GetHostRecoveryMeta on any page-render path — its struct and its SELECT both omit the secret column, so it cannot leak one; GetHostRecoveryCredential (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE recovery_credential_revealed event via SaveEvent and calls NO dispatcher (the handleRequestLogTail shape). api/handler.go handleAdminGetRecoveryCredential (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
host_detail_body sub-template |
hub/internal/web/templates/host_detail_body.html | {{template "host_detail_body" .}} |
Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with {{.HostID}} (the customer page renders N instances). |
(*Store).ListHostsByCustomer |
hub/internal/store/store.go (~L1620) | (customerID) ([]Host, error) |
A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
(*Store).HasEverBoundHost (v0.92.0, R-195) |
hub/internal/store/store.go | (customerID) (bool, error) |
Any verdict that must not fire for a customer with no machine ever bound — "was anything ever expected of this customer" | hosts row OR host_deletions tombstone. NOT a liveness check and never a substitute for one: a box that was bound and went silent returns true and must keep alarming — that is the case any change here breaks first (pinned by TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms). Callers fail OPEN on its error: an unreadable binding must never SUPPRESS an alarm. Do not re-derive this from report presence — store.GetCustomers() (and therefore the staleness checker's down state) is a query over reports, so a never-reported customer has no state at all, which is exactly how the daily false alarm reached david. |
(*Server).configFormData (v0.49.0) |
hub/internal/web/configs.go (~L430) | (r, isNew, cfg, overrides, errMsg) configFormView |
The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | overrides=nil → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
config_form_body sub-template (v0.49.0) |
hub/internal/web/templates/config_form_body.html | {{template "config_form_body" <configFormView>}} |
Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after </form> — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight <script>. |
handleHostEscrowGet + escrowSelfServiceRetrieval |
hub/internal/api/handler.go | GET /hosts/{id}/escrow |
The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. escrowSelfServiceRetrieval is THE single decision point for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. Never merge this with dr.go's handleReEnroll/handleGetRestoreDirective — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by TestEscrowGet_OperatorDRPathUnchanged). Every successful retrieval MUST raise escrow_blob_served before the bytes leave; that audit row is the mitigation the trade rests on. |
|
demoteCurrentEscrowTx (+ (*Store).SaveHostEscrow) |
hub/internal/store/store.go (~L2597/~L2612) | (tx, hostID) (int64, error) / (hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error) |
THE ONE escrow row-copy routine — used by the re-escrow retention AND by DeleteHost's custody demotion; never write a second one |
It must copy BOTH sealed artifacts — blob (K-escrow / PBS key) and identity_blob (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. Ordering it depends on: SaveHostDRBundle writes identity_blob AFTER SaveHostEscrow returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by TestSaveHostEscrow_RetainsIdentityBlob + TestDeleteHost_DemotesIdentityBlob (both callers). prevResticPwSHA feeds R-197's changed-key signal; it is a hash and never leaves the store. |
(*Store).CountHostArtifacts / DeleteHost |
hub/internal/store/store.go (~L1640/~L1690) | (hostID) (HostArtifacts, error) / (hostID, deleteEscrow bool) error |
Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (ErrHostEscrowPresent, tx never starts). log_bundles die by scope_id == host_id ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
(*Server).commitCustomerReset (v0.69.0) |
hub/internal/web/customer_reset.go (~L165) | (ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError |
THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the customer_resets journal |
Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). purgeEscrow governs ONLY whether PurgeCustomerResetDBState destroys retained custody: standalone RESET passes the operator's escrow_ack; the DELETE cascade passes false so custody dies exactly once, in its leg 3. Returns a resetLegError carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
(*Store).CustomerResidue / PurgeCustomerResidue (v0.70.0) |
hub/internal/store/customer_delete.go | (customerID) (*CustomerResidue, error) / (customerID) error |
Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | GetCustomers() is REPORT-derived — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared residueQueries list so a table can never be counted-but-not-purged. Includes appliance_registrations + selfbind_tokens (credential-bearing, not telemetry). NEVER touches events, notification_log, host_deletions, customer_resets. |
(*Server).handleCustomerDelete / handleCustomerDeletePreview (v0.69.0) |
hub/internal/web/customer_delete.go | (w, r, customerID) |
THE customer offboarding entry: the guided full-teardown cascade hosts → RESET → purge (R-25b). GET = live inventory JSON for the dialog, POST = the cascade |
There is NO shallow delete path any more — the old handleConfigDelete is gone; do not reintroduce one. Every gate (3 acks, typed customer-id, stale host-count, ONLINE-host refusal) runs BEFORE any write, so a refusal has zero side effects. Leg order is load-bearing twice over: ruling 3 (RESET never sees a host row) and custody purged exactly once, in leg 3. A failed leg retains the journal — a re-run resumes and must pass every gate again. |
(*Store).ListWGEndpoints / DeleteWGEndpoint |
hub/internal/store/wg.go (~L64/~L86) | () ([]WGEndpoint, error) / (endpointID) error |
The /offsite endpoint-management surface | GetWGEndpoint (lowest id, LIMIT 1) stays THE allocation/sync endpoint — do not switch allocator/reconciler/desired-state to the list without the wg_peers.endpoint_id migration arc. Peers-in-subnet guards live in hub/internal/web/offsite.go. |
Artifact manifest / Day-0 trust root
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
(*Store).GetArtifactManifest / SetArtifactManifest |
hub/internal/store/store.go (~L933 / ~L944) | () ArtifactManifest / (m) error |
The DB-backed (hub_settings) Day-0 artifact record | This is the checksum TRUST ROOT the host-bootstrap verifies against — distinct from Gitea, which only stores bytes. |
(*Store).GetOperatorPasswordHash / SetOperatorPasswordHash |
hub/internal/store/store.go (~L1350) | () string / (hash) error |
The DB-backed (hub_settings) operator login password override | Read via Server.effectivePasswordHash(), not directly. "" = no override (config seed authoritative). Store the bcrypt hash, never the plaintext. |
(*Server).handleSetArtifacts + resolveArtifactSHA |
hub/internal/web/configs.go (~L644 / ~L680) | POST /configuration/artifacts |
Operator UI to vouch artifact versions | With a Gitea client the sha is fetched AUTHORITATIVELY (submitted sha ignored); fetch failure refuses the save. Manual sha only in the no-creds fallback. |
(*gitea.Client).ListVersions / FileSHA256 |
hub/internal/gitea/gitea.go (~L47 / ~L72) | (ctx, pkg) ([]string, error) / (ctx, pkg, ver, file) |
Read-only Gitea generic-package metadata | sha comes from package metadata — artifact bytes are never downloaded. Newest-semver-first sort. |
(*Server).artifactChoices |
hub/internal/web/server.go (~L155) | (ctx, pkg, file) []artifactChoice |
Version+sha dropdown data | nil Gitea client / unreachable → nil → UI degrades to manual entry. One bad version drops itself, not the list. |
(*Handler).handleArtifactManifest |
hub/internal/api/handler.go (~L1550) | GET /api/v1/artifacts/{id} |
Serving the vouched set to the bootstrap script | Auth mirrors handleConfigRetrieve exactly. Unset manifest = empty fields, not an error. |
normalizeFloorInput / normalizeSHA256 |
hub/internal/web/configs.go (~L27 / ~L627) | (raw string) (string, bool) |
Validating operator-typed versions / shas | Empty string is VALID (means "clear"). Reuse for any new version/sha form field. |
PBS DR tier / tenantsync (hub/internal/tenantsync/, hub/internal/web/pbsdr.go, hub/internal/api/pbsdr.go)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
tenantsync.Client (Provision/Reissue/Fingerprint) |
hub/internal/tenantsync/client.go | (ctx, customerID) (*Result, error) |
ep0 per-customer PBS tenancy over the pinned-SSH forced-command channel (the wgsync twin) | Result.TokenSecret is transient custody → SaveHostPBSSecret immediately, never log the struct. Error paths NEVER embed stdout (the secret channel) — do not "improve" diagnostics by quoting the response. ErrTokenExists is typed: provision refuses an existing token; re-issue is the explicit path. |
(*Store).SaveHostPBSSecret / ConsumeHostPBSSecret |
hub/internal/store/pbsdr.go | (hostID, value) / (hostID) (string, error) |
HOST-scoped consume-once secret (the one_time_secrets host twin) | Same-tx mark-consumed; re-save resets consumption (re-issue supersedes). The agent consumes via POST /api/v1/hosts/{id}/pbs/consume-token (hub/internal/api/pbsdr.go). |
offsite.DeliveryStateFor (+ DeliveryStatus) |
hub/internal/offsite/delivery.go | (st, customerID) (DeliveryStatus, error) |
THE R-70 offsite last-mile detector — one implementation for every consumer (customer card deliveryViewFor, monitor.OffsiteDeliveryChecker event + R-71c heal) |
Precedence: applied (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + StaleStagedSince flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
(*Store).GetOneTimeSecretInfo / LastEventAt / LatestReportOffsitePresence / CountReportsOffsiteSince |
hub/internal/store/store.go | (customerID) (*OneTimeSecretInfo, error) / (customerID, eventType) (time.Time, error) / … |
Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | GetOneTimeSecretInfo never selects the value column — keep it that way. SetOneTimeSecretTimesForTest is the back-dating seam (PBSDR pattern). |
monitor.OffsiteDeliveryChecker + OffsiteReissuer |
hub/internal/monitor/offsite_delivery.go | NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger) |
R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in maybeHeal: re-reads the secret row at act time and refuses over an UNCONSUMED row — SaveOneTimeSecret clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
monitor.RestoreTestChecker + assessRestoreProven |
hub/internal/monitor/restoretest.go | NewRestoreTestChecker(st, onEvent, logger); .Check() |
R-85: turns a restore-test result into a SIGNAL — it was a [WARN] log line and nothing else, even for the tier already being tested |
TWO event types, never merged: restore_test_failed (broken now, error) vs restore_test_stale (unverified — not known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. Anchored on R-81 (assessRestoreProven reuses backupAssessment/verdict*): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: no customerMessages entry — do not add one without copy review. R-86 (2026-08-03): the window is PER TIER, not one constant. restoreProvenWindow(tier, observed, ok) = clamp(4 × max(observed, declared), floor 7d, cap 12d), where declared is that tier's own backup-freshness threshold (backupStaleAfter 26 h / offsiteBackupStaleAfter 8 d — reuse those, never a second opinion) and observed comes from observedArchiveIntervals over the retained window. Observation may only WIDEN: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. assessRestoreProven takes the window as an argument and every reason string names it (R-100's corollary). |
(*Server).applyPBSDR + mergePBSDR/readPBSDR |
hub/internal/web/pbsdr.go | (ctx, r, cfg) error |
The config form's DR-tier section → HOST desired_json pbs_dr descriptor + generation bump |
Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by cfg.DRTier (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
(*Server).pbsdrProvisionAtom + PBSDRAutoProvision |
hub/internal/web/pbsdr.go | (ctx, customerID, host, storageID) (blocked string, err error) / (ctx, customerID) |
The shared fresh-provision cascade atom; the WG-registration hook target (api SetWGRegisteredHook, wired in hub/cmd/hub/main.go when tenantsync is on) |
blocked != "" = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
cfg.DRTier + offsite coupling |
hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field dr_tier (formBool helper). |
pbsdrheal.Reconciler + NewActions |
hub/internal/pbsdrheal/reconciler.go | NewReconciler(st, act, logger) · RestrictToHost(hostID) · Run(ctx)/Trigger() |
PBS-DR SELF-HEAL: re-arms a consumable secret for a box stuck in waiting_secret/consumed_failed after losing its converged marker (re-install/rollback). From SPIKE-pbsdr-selfheal-2026-07-15. |
Primary heal = re-stage the stored secret (no ep0 call, NO generation bump — a bump = agent refetch loop). Escalate to Re-issue only when no stored secret / consumed_failed. NEVER re-run pbsdrProvisionAtom (refuses ErrTokenExists) or blind-timer Re-issue (hash/gen thrash). Converged/disabled/verify_failed/DR-OFF = no-op. Debounce ≥2 distinct reports. PBSDRHEAL_ONLY_HOST scopes a supervised rollout. Fake seam: fakeActions in reconciler_test.go. |
(*Store).RestageHostPBSSecret + PBSDRHealStates |
hub/internal/store/pbsdr.go | (hostID) (restaged bool, err) / () ([]PBSDRHealRow, error) |
The self-heal store primitives: clear consumed_at IFF a row exists (re-arm the SAME value); the fleet work-set query (descriptor enable/provision + latest report pbs_dr.state + id) |
RestageHostPBSSecret does NO insert, NO value change, NO generation bump (restaged=false = no row → caller escalates). PBSDRHealStates mirrors GetHostOOBStates' latest-report-per-host join; malformed JSON → zero values, never an error. |
(*Server).ReissuePBSDR |
hub/internal/web/pbsdr.go | (ctx, customerID) error |
The non-HTTP core of the operator Re-issue button — the self-heal reconciler's escalation seam (satisfies pbsdrheal.Reissuer) |
Reuses tenantsync.Reissue + SaveHostPBSSecret + descriptor bump — NOT a re-run of pbsdrProvisionAtom. Keep in lockstep with the tail of handlePBSDRReissue (which is unchanged; the operator button's 303/400 codes are pinned by tests). |
parseHostCapabilities + capabilityView |
hub/internal/web/hosts.go | (reportJSON) []capabilityView |
Host-page capability chips (ok/degraded/inactive) | inactive (agent v0.86.0) = badge-neutral, NEVER warn/error — disabled ≠ degraded; unknown future statuses fall to the degraded styling (surface, don't hide). capabilitiesNeedDRMigration keys the pre-v1.15.0 migration hint on pbsdr-* + "binary not found". |
Config generation & secrets hygiene (hub/internal/configgen/)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
configgen.Generate |
hub/internal/configgen/configgen.go (~L16) | (templateYAML, cfg) (string, error) |
Producing a customer controller.yaml | Programmatic overrides (customer id/hub url/api_key) ALWAYS win over config_json; fresh session secret per generation. |
configgen.RandomHex |
hub/internal/configgen/configgen.go (~L110) | (n int) (string, error) |
crypto/rand hex tokens (API keys, session secrets) | — |
configgen.RandomPassphrase |
hub/internal/configgen/passphrase.go (~L35) | (wordCount int) (string, error) |
Human-dictatable Hungarian passphrases (retrieval passwords) | ~29K-word embedded list; 5 words ≈ 74 bits. |
(*Store).EffectiveMinControllerVersion |
hub/internal/store/store.go (~L960) | (customerID) string |
Resolving the floor that actually applies (per-customer → global) | "" = no floor (Phase 2 inert). |
Assets, store misc, scheduling
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
assets.Manager (New, ServeFile, ReSeed) |
hub/internal/assets/assets.go (~L44/190/122) | seed-dir → PVC sync + manifest | Serving app logos/screenshots to controllers | ServeFile sanitizes to filepath.Base (no traversal). isAssetFile enforces the naming convention. |
copyFile |
hub/internal/assets/assets.go (~L259) | (src, dst) error |
THE atomic file write (tmp + rename) in the hub | Copy this shape for any new on-disk write. |
fileSHA256 |
hub/internal/assets/assets.go (~L244) | (path) (string, error) |
Streaming sha256 of a file | — |
(*Store).SaveEvent |
hub/internal/store/store.go (~L1003) | (...) (int64, error) |
Persisting ANY event (controller or hub source) | Pair with dispatcher/onEvent — saving alone never notifies. |
(*Store).RequestLogTail / GetPendingLogTailRequests / SaveAppLogTail |
hub/internal/store/logtail.go | pending-intent + consume-once fulfillment | THE ACK-flag pull pattern for hub→box requests (copy for any new one) | SaveAppLogTail clears the request in the SAME tx (consume-once) + prunes to last 2 per (customer,app); the hub NEVER connects into a box |
(*Store).RequestLogBundle / PendingLogBundleRequest / SaveLogBundle / PurgeExpiredLogBundles |
hub/internal/store/logbundle.go | component (controller/agent) log pulls — the v0.46.0 sibling of logtail.go | box-component debug-ring pulls; gzip custody, newest-3, 72 h TTL on the 60 s sweep | scope = customer_id (controller/report ACK) vs host_id (agent/heartbeat envelope); SaveLogBundle runs the SECRET GATE fail-closed (blocked flag row, no payload) and clears the request in the same tx; [REDACTED]/checksums pass by design |
upsertAppIssue dismissal/context semantics |
hub/internal/store/telemetry.go | ON CONFLICT CASE guards | Issue dismissal + first-capture-wins context | Un-dismiss ONLY on excluded.last_seen > dismissed_at; context adopted only while stored one is empty — do not "simplify" either CASE (red-proofed) |
store.GuestID |
hub/internal/store/store.go (~L1268) | (hostID string, vmid int) string |
Canonical guest primary key | Never hand-concatenate host+vmid. |
(*Store).GetHostReportsSince + GetFirstHostReportAt + monitor.newestBackupEvidence |
hub/internal/store/store.go, hub/internal/monitor/deadline.go | (customerID, since) ([]HostReportRow, error); (customerID) (time.Time, error); (rows, now) (time.Time, bool) |
Asking "when did the hub last SEE evidence of X?" instead of "what does the latest report say?" — the R-81 anchor. The agent's reporters are point-in-time and forget across a restart; the hub retains ~90 d of host-reports and does not. | The three go together: window scan + first-contact anchor + a bounded lookback (backupEvidenceLookback). Never judge a report-derived absence on the LATEST report alone — that is the bug class R-81 fixed for the third time. The scan early-exits on sufficiently-fresh evidence, so don't reorder rows away from newest-first. |
scheduleDaily |
hub/cmd/hub/main.go (~L449) | (ctx, name, "HH:MM", fn, logger) |
Daily jobs in Europe/Budapest (prune etc.) | Blocking — run as goroutine. parseHM returns 0,0 (midnight) on bad input. |
2. Canonical patterns (copy structure from THE named file)
| Pattern | Canonical file | Key traits |
|---|---|---|
| Monitor checker | hub/internal/monitor/staleness.go | Seed state on construction WITHOUT emitting events; in-memory states map under mutex; periodic Check(); emitTransition = SaveEvent then nil-checked onEvent; cleanup of vanished IDs. HostStaleness/HostDisk/HostLeaf/HostCapability/StorageFill all follow it. |
| API endpoint | hub/internal/api/handler.go ServeHTTP (~L139) + any handler |
Path switch in ServeHTTP; first line of every handler = checkAuth{Customer,Host}; io.LimitReader body cap; typed anonymous payload struct; explicit 4xx strings. |
| Web POST action | hub/internal/web/configs.go handleSetGlobalFloor (~L602) |
CSRF enforced centrally in web ServeHTTP; validate via normalize* helper; POST-redirect-GET with ?flash= token; log INFO on success. |
| Optional dependency injection | hub/internal/web/server.go Set* setters (~L131–148) / api handler SetDispatcher etc. |
Constructor takes hard deps; optional subsystems wired via SetX after construction; nil = graceful degradation (never panic). |
| Seam-injected sender for tests | hub/internal/notify/dispatcher.go sendEmailFn (~L33) |
Function-field defaulting to the real impl in the constructor; tests overwrite it. Same idea: mailRateLimiter.now, mailrelay.Sender fake. |
| Immediate-sync notify (per plane) | agent plane: poke.Notifier (web.Server.poke / api.Handler.poker via SetPoke/SetPoker) — controller plane: intent.Hub.Bump (web.Server.bumpIntent, api.Handler.intentHub) |
EVERY desired-state mutation fires the RIGHT plane's notifier AFTER the successful store write, never on an error path (fire-after-commit). Agent-plane pokes a HOST when its generation moved (SetHostDesired/Bump*HostDesired); controller-plane bumps a CUSTOMER on a controller-visible change. Both receivers COALESCE bursts — add NO hub-side dedup. Deliberate non-sites need a documented reason (undeliverable pre-tunnel, transport removed, or no generation bump → the 60 s ticker is the pickup path). Both seams nil-safe: unset = the ≤15-min cycle still reconciles. Full site inventory: REPORT.md audit table (v0.63.0). |
| Website page | website/index.html | UTF-8 with BOM; shared <nav>/<footer> byte-identical across pages (only class="active" differs); two-tone H1 = <h1>…<span>accent</span></h1>; all styling in website/assets/site.css tokens (:root) — zero embedded <style>; ?v=N cache-bust on site.css/icons.svg; umami snippet; no CDN fonts; no emoji (sprite icons.svg instead). |
| Gate RUNNER (the entry point) | scripts/repo_gates.py | THE one entry point — runs all six gates, streams each gate's own output, non-zero if any fails. --fast = no network, no container runtime. A missing gate script is a FAILURE, never a skip. .githooks/pre-push runs it with --fast; CI re-runs it on every push and mails on failure (R-168). Register a new gate as a row in GATES; do NOT model new work on site_gates.py, which is a gate, not a runner (R-161). |
| Instruction-file gate (shared) | scripts/instructions_gate.py | Registered in all four repos' runners and never copied into a sibling — a copy recreates the drift it detects. Checks CLAUDE.md effective length / version literals / expired TEMPORARY, .claude/rules/*.md scoping, workspace-copy identity, and (check 6) the auto-memory index. Effective text = HTML comments stripped, because they are stripped before injection. Three outcomes are deliberately different: over-limit FAILS, an orphan WARNS, an absent store PASSES printing its reason — and the test asserts on that reason text, since a pass with no reason is indistinguishable from a gate that stopped running. |
| Workspace installer | scripts/install_workspace.py | Lays down the workspace-root CLAUDE.md (canonical source: documentation/runbooks/workspace-CLAUDE.md) and the InstructionsLoaded hook from workspace/hooks/. Unlike install_skills.py the targets are LIVE CONFIG, not disposable: timestamped backup before every write, settings.json MERGED (owns exactly one key, hooks.InstructionsLoaded), a diverged CLAUDE.md reported not silently resolved, an unparseable settings.json REFUSED. Idempotent — a second run writes nothing. |
| Rule-firing report | scripts/rules_report.py | Reads the InstructionsLoaded hook log and cross-references every .claude/rules/*.md in the workspace. The column that matters is the empty one — a rule that has never fired is mis-globbed or dead (built-but-never-wired, applied to instructions). Prints each silent rule's paths: so "wrong glob" is distinguishable from "quiet month". Caveat: the log only covers since the hook was armed. |
| Gate script | scripts/site_gates.py | Byte-level mechanical gates (BOM, emoji codepoint ranges, nav/footer diff, analytics, banned tokens, cache-bust); run python scripts/site_gates.py after ANY website change; non-zero exit on failure. |
| Fetch-validate-install (shell) | scripts/felhom-host-install.sh step_agent_install (~L1108) |
fetch_raw to mktemp → syntax-check (bash -n) → install -m0755 -o root -g root → only then activate; guarded-mkfs wrapper installed BEFORE the sudoers that references it (ordering is the safety property). All mutations through run() (dry-run aware). |
| Install-profile gate (shell) | scripts/felhom-host-install.sh --mode appliance|byo (GL-2, v1.10.0) |
Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, FELHOM_INSTALL_STATE_DIR override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). |
| Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh _uninstall_statement + harness GL4-D (v1.11.0) |
Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by run_uninstall; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain umount only, never -l/-f, never any format op under /mnt/felhom-drives. |
| Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout /website/ + /scripts/, --link=current) + init container waits for first sync; nginx serves current/website; push to main = deployed, no image build. |
| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via secretKeyRef to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). report-api (the operator bearer, v0.53.0) is deliberately NOT optional: — a missing Secret fails Ready instead of booting an unauthenticatable hub. scripts/manifest_bearer_gate.py (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): gitea-creds is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
| Hub deploy (GitOps) | manifests/hub.yaml image: (~L129) |
Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. |
3. Dangerous lookalikes — do NOT reuse
| Trap | Why it bites | Use instead |
|---|---|---|
A plain missed bool for a report-derived absence (hub/internal/monitor/deadline.go) |
Collapsing the three-valued verdict re-introduces one of TWO failure modes: absence→MISSED is the 2026-07-26 cry-wolf (three boxes alarmed at once, one reached a customer channel); absence→OK means a genuinely dead box alarms NEVER, which is strictly worse. Three instances of this class so far: hub v0.12.0, v0.73.0, R-81. | `backupAssessment{verdict: verdictOK |
(*Handler).handleNotify + formatNotificationEmail + sendResendEmail (hub/internal/api/handler.go ~L1289/1624/1589) |
Legacy pre-dispatcher notification trio: no cooldowns, no operator channel, no allowedEventTypes gate, duplicate Hungarian formatter. Controller path is FROZEN until slice-10 cutover. | POST /api/v1/event → Dispatcher.ProcessEvent + notify.Format*Email |
Severity "critical" POSTed to a PRE-v0.31.0 hub |
Fixed in hub v0.31.0 (handleEvent now accepts critical). Older hubs coerce critical → "info", which never notifies — silent alert loss. Case-variants ("Critical") still coerce on every version. |
Against an old hub send warning/error; otherwise lowercase critical is safe |
compareVersions for anything security-ish (hub/internal/web/server.go ~L571) |
Returns 0 (equal) on unparseable input — a garbage version passes a floor check. gitea.compareSemver behaves differently (lexical fallback). |
Validate input with normalizeFloorInput first; then compareVersions is safe |
Inline stringData secrets à la manifests/felhom.secret.yaml |
Commits real credentials to git (healthchecks superuser pw, umami APP_SECRET/POSTGRES_PASSWORD, gitea-creds admin password still live there). | Out-of-band kubectl create secret + secretKeyRef (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md) |
kubectl apply / kubectl set image on manifests/ |
ArgoCD app felhom reverts drift on next sync; live state lies about git. |
Edit manifest in git → push → ArgoCD sync (CLAUDE.md steps 3–5) |
:latest image tag in manifests |
Re-push doesn't change the manifest → no redeploy; Synced/Rollback misreport. | Pinned version tag, bumped per deploy |
The report's backup object for snapshots / repo size / integrity (snapshot_count, repo_size_mb, integrity_ok) |
No producer since slice 8C — the controller's buildBackupReport leaves all four zero deliberately and says so. Rendering them gave every customer Snapshots 0 · Repo Size 0 MB · Integrity Unknown indefinitely (R-331, measured on demo-hp over a repository holding 67 snapshots). integrity_ok is worse than stale: the controller runs no integrity check at all, so it can only ever be "Unknown" or a lie. |
The report's offsite object (backup.OffboxReportStatus) via reportBackupCard — and check stats_known before trusting a zero |
| grep/regex hunting emoji in website HTML | Windows grep false-negatives multibyte emoji (proven in D0). | python scripts/site_gates.py (codepoint-range check) |
| Adding a website page without touching site_gates.py | PAGES list (scripts/site_gates.py ~L22) is explicit — an unlisted page is silently ungated (BOM/nav/emoji drift undetected). |
Add the filename to PAGES in the same commit |
4. Seams & interfaces (testing + cross-repo)
| Interface | Defined in | Implemented by | Fakes/tests at |
|---|---|---|---|
mailrelay.Sender |
hub/internal/mailrelay/relay.go (~L24) | ResendSMTP (prod) |
fake sender in hub/internal/api/mail_test.go; hub/internal/mailrelay/relay_test.go |
Dispatcher.sendEmailFn (func seam) |
hub/internal/notify/dispatcher.go (~L33) | (*Dispatcher).sendEmail (Resend HTTP) |
hub/internal/notify/dispatcher_test.go |
monitor.EventNotifyFunc |
hub/internal/monitor/staleness.go (~L14) | closure over Dispatcher.ProcessEvent (main.go) |
hub/internal/monitor tests (captured-events func) |
api.ConfigTemplateProvider |
hub/internal/api/handler.go (~L24) | web.TemplateFetcher (Gitea-pulled controller.yaml template) |
stub providers in api tests |
api.LatestVersionProvider |
hub/internal/api/handler.go (~L31) | web.VersionChecker (registry poll) |
hub/internal/api/config_version_ack_test.go |
mailRateLimiter.now (func seam) |
hub/internal/api/mail.go (~L27) | time.Now |
hub/internal/api/mail_test.go clock injection |
web.tenancyProvisioner |
hub/internal/web/pbsdr.go | *tenantsync.Client (pinned SSH to ep0's felhom-tenantsync) |
fakeTenancy in hub/internal/web/pbsdr_test.go; in-process SSH server in hub/internal/tenantsync/client_test.go |
| Cross-repo: ep0 tenancy surface | scripts/felhom-tenantsync.sh (JSON stdin/stdout forced command) |
installed on ep0 per runbook offsite-endpoint.md §10 | provision/reissue/fingerprint ops; token secret rides stdout ONLY; the peersync script/key are untouched |
| Cross-repo: controller → hub | POST /api/v1/report (frozen) + POST /api/v1/event |
felhom-controller repo | new event types MUST enter allowedEventTypes (hub/internal/api/handler.go ~L1063) or the controller gets 400 |
| Cross-repo: agent → hub | POST /api/v1/host-report, /host-enroll, jobs/desired-state/escrow routes (handler.go ~L145–189) |
felhom-agent repo | hub/internal/api/host_test.go, desired_test.go, escrow_test.go, dr_test.go |
| Cross-repo: Day-0 bootstrap → hub | GET /api/v1/config/{id} + /artifacts/{id} (X-Retrieval-Password) |
scripts/felhom-host-install.sh (fetches + sha256-verifies against the hub-vouched manifest) | hub/internal/api/artifact_test.go |
| Cross-repo: controller ← hub assets | GET /api/v1/assets/manifest + /assets/file/{name} |
felhom-controller pulls app logos/screenshots | assets manifest sha-based change detection |
5. Extension points (where new features plug in)
- New event type — THREE registers, and which ones depend on the AUDIENCE. Always:
allowedEventTypes(hub/internal/api/handler.go) — missing it means the controller's POST 400s and the event vanishes (the known gotcha). Then decide the audience and stop guessing from the other registers:- Operator-only → add to
notify.operatorOnlyEvents(hub/internal/notify/dispatcher.go) and give it nocustomerMessagesentry. Allowlisting alone does NOT make a type operator-only —FormatCustomerEmailtreats a missingcustomerMessagesentry as a fallback to the raw message, not a block, and the only customer gate is configuration. v0.78.0 asserted the opposite in a comment and shipped the defect (R-97c). Examples:whole_guest_backup_failed,recovery_unit_capture_failed. - Customer-facing with a STATIC message → add a
customerMessagesentry (hub/internal/notify/templates.go) and the controller'ssettings.DefaultEnabledEventsif it should be on by default. - Customer-facing with a DYNAMIC message (the producer builds Hungarian text carrying names/numbers) → deliberately no
customerMessagesentry:FormatCustomerEmailPREFERS the entry over the message, so adding one silently discards the specifics. Examples:offbox_enlarge_blocked,disk_health_degraded, and since v0.89.0disk_warning/disk_critical. - Pin BOTH registers in ONE test (hub/internal/api/recovery_unit_event_test.go is the model) — fixing one and not the other is the realistic mistake, and
notify.IsOperatorOnlyexists so the api package can assert it. - A type in these registers with no PRODUCER is inert.
disk_warning/disk_criticalwere allowlisted, copy'd, default-enabled and checkbox'd from early on, and nothing in any repo emitted them until controller v0.191.0 — grep for an emitter before assuming a type works.
- Operator-only → add to
- New monitor checker: copy hub/internal/monitor/staleness.go (§2 pattern); wire in hub/cmd/hub/main.go with an
EventNotifyFunc; severity must be warning/error/critical to notify. - New API route: switch in
api.ServeHTTP(handler.go ~L139); auth helper first line. - New web page/action: switch in
web.ServeHTTP(server.go ~L182) — non-GET gets CSRF automatically; template into hub/internal/web/templates/ (embedded FS, parsed inweb.New); new helpers into the funcMap (server.go ~L67). - New template func: funcMap in web.New only; add a case to hub/internal/web/funcmap_test.go.
- New daily job:
scheduleDailyin hub/cmd/hub/main.go + add pruning topruneAllif data grows. - New site gate: append to scripts/site_gates.py; new website pages go into its
PAGESlist. - New gate of any kind: add a row to
GATESin scripts/repo_gates.py — a gate reachable only by being named in a CLAUDE.md is the R-29 finding. A new.claude/rules/*.mdneeds apaths:list, and is NOT live in the session that created it — verify from a fresh session (claude -p) and confirm with scripts/rules_report.py, never from the frontmatter. - New artifact kind (Day-0): consts
pkg*/file*(hub/internal/web/server.go ~L27),ArtifactManifestfields + settings keys (hub/internal/store/store.go ~L905),handleSetArtifacts,artifactManifestResponse(handler.go), and the install script's verify step. - New host-install step:
step_*function in scripts/felhom-host-install.sh usingrun()/fetch_raw/diehelpers; keep dry-run coverage. - New DR-recipe section:
hostHalfShape/appHalfShapeandAssembledRecipe(hub/internal/store/dr_recipe.go) — those shape structs are ALLOW-LISTS, not forward-compat: a section only the emitter knows about is stored intact and silently dropped before the operator downloads it. No error, no log, no red test. That is R-122: the controller emittedoffsite_resticfrom fork-4, all three real customers had it stored, and no delivered recipe ever contained it. Then extendTestAssembleDRRecipe_CarriesEveryEmittedSection(same commit) and, for a host-half section, the agent'sDRRecipeHostHalf+ BOTH copies oftestdata/host-report.golden.json(byte-identical, cross-repo).
6. Known duplication (observed — NOT fixed)
- Resend HTTP sender ×2:
(*Handler).sendResendEmail(hub/internal/api/handler.go ~L1589) ≈(*Dispatcher).sendEmail(hub/internal/notify/dispatcher.go ~L185) — byte-near-identical POST to api.resend.com. Kept because the handler copy serves the frozen legacy /notify path. - Hungarian customer-email formatter ×2:
formatNotificationEmail(hub/internal/api/handler.go ~L1624) vsnotify.FormatCustomerEmail(hub/internal/notify/templates.go ~L118). Legacy vs dispatcher; the legacy one lacks the per-event-type message map. - Semver compare ×2 with DIFFERENT fallback semantics:
web.compareVersions(hub/internal/web/server.go ~L571, parse error → 0) vsgitea.compareSemver(hub/internal/gitea/gitea.go ~L115, parse error → lexical). Documented as deliberate (import-cycle avoidance) in gitea.go, but the behavior drift is not. - Checker-family structural repetition: staleness.go vs host_staleness.go, and host_disk.go vs storage_fill.go (band/bandRank/bandLabel vs bandForPercent) — same skeleton re-implemented per domain; treated as the accepted §2 pattern rather than a defect.
- Duration formatting ×2:
monitor.formatDuration(hub/internal/monitor/staleness.go ~L187) vsweb.timeAgo(hub/internal/web/server.go ~L603) — different audiences (email vs UI) but overlapping logic.