# Felhom Hub — Changelog ## v0.37.0 — offsite provisioning SLICE 1: Hetzner Cloud-API client + provisioning core (2026-07-09) Slice 1 of the offsite-provisioning epic. On operator enable, the hub provisions a Hetzner storage-box sub-account (shared) or dedicated box, generates the transient password, stores it one-time-consumable, and serves the non-secret target descriptor to the controller via `ConfigJSON`. Coded against the API shapes measured live in `documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (996d403). The controller apply-bridge (SLICE 2), escrow auto-confirm (SLICE 3), and soft-quota enforcement (SLICE 4) are separate slices. - **`internal/hetznerapi`:** a typed client for the storage-box surface at **`https://api.hetzner.com/v1`** (NOT `api.hetzner.cloud` — the classic Cloud API 404s for storage boxes). Sub-account + box create/reset/access/change_type/delete/list-by-label + `WaitAction` (poll to `success`, bounded). A `CloudAPI` interface + an exported `Fake` so provisioning is unit-tested with **no live Hetzner calls**. Bearer token from an injected func (out-of-band secret; never logged). - **`internal/offsite`:** `Provisioner.ProvisionOffsite` — **idempotent by `label_selector` (felhom-customer=)** (names aren't unique); shared → sub-account on the pool box, dedicated → box; generates a 4-class transient password → `WaitAction` → `Store.SaveOneTimeSecret` → builds the NON-SECRET `Descriptor{enabled,type,host,user,port:23,repo_path:/home/felhom-repo, quota_gb|box_type}`. **Fail-closed:** any API/action error returns without a provisioned resource, one-time password, or descriptor. `MergeDescriptor` merges it under the `offsite` key of `ConfigJSON` (never a secret). - **`internal/store`:** `one_time_secrets` table + `SaveOneTimeSecret` / `ConsumeOneTimeSecret` (single-use, return-and-mark in one tx). The transient password NEVER rides `ConfigJSON`. - **`internal/api`:** `POST /offsite/consume-password/{id}` — serves the one-time password to the authenticated customer (same API-key auth as config-pull) EXACTLY once, then 404s. Never logged. - **`internal/web`:** the config form gains an **Offsite backup** section (enable / type / soft-quota / box type); save → `applyOffsite` provisions (fail-closed: a provisioning error returns 502 and does NOT save) and merges the descriptor → `ConfigVersion` bump → controller re-pulls. Optional dep (`SetOffsiteProvisioner`), wired in `cmd/hub/main.go` from `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID`/`HETZNER_LOCATION`. - Tests (faked Cloud API, no live calls): shared/dedicated provision + descriptor + one-time-password-stored + **password-absent-from-ConfigJSON**; idempotent re-save (no 2nd resource); fail-closed **+ companion red-proof** (swallow the create error → offsite marked enabled despite failure → test fails); one-time consume-once; `WaitAction` success/error/timeout; the consume endpoint (auth + single-use). - **NOT yet live-provisioned** — awaiting the **dedicated-project scoped token** (the current token can delete ep0 — SPIKE §6); a live create is a supervised validation. Unit tests are this slice's proof. ## v0.36.0 — customer page: passphrase hardening + interactive install-command generator (TASK GL-7) (2026-07-09) Two coupled, security-first changes to the operator-facing customer page (`customer_unified.html` + `configs.go`). felhom.eu only; agent + host-install untouched. - **Passphrase hardening (ships the security win).** The per-customer retrieval passphrase was rendered in cleartext twice — the visible `#retrieval-pw` node and baked into the Option-3 debug curl's `X-Retrieval-Password:` header. Now: `#retrieval-pw` renders a masked bullet run by default with **reveal** (`toggleSecret`) + **copy** (`copySecret`) controls, the value carried in `data-secret` (the existing reveal model). The Option-3 command carries a `` placeholder — the secret is NEVER in a copyable command block. (A zero-secret-in-DOM reveal-on-demand fetch is a deliberate future follow-up, not this task.) - **Interactive install-command generator.** The three hard-coded install `` blocks became a client-side builder (vanilla JS — no framework, CDN, or network) that assembles a live-updating command from form controls, emitting ONLY real host-install **v1.12.0** flags in a download-then-run shape (never `curl | bash`). CustomerID is prefilled from the server (`ScriptVersion`/`data-customer-id` via `pageData`); a byo selection requires `--cores`/`--memory` (enforced client-side with a `.gen-req`/`gen-msg` prompt); caps/mode are placeholders, never silent defaults. Graceful **JS-off static fallback**: the Option-1/2 code nodes keep a `--customer-id … --mode ` command. The curated control surface excludes the seven dangerous/operator-only flags (`--force`, `--rotate-recovery`, `--enable-oob`, `--remove-golden`, `--uninstall`, `--adopt-pool`, `--rescope-acl`) — they are never offered as controls. - **`configs.go`:** `const hostInstallVersion = "1.12.0"`; `pageData.ScriptVersion` added + populated. - **Tests** (`render_test.go`): `TestTemplates_PassphraseHardened` (secret NOT in the Option-3 command, placeholder present, masked-by-default bullet run, `data-secret` populated, reveal/copy controls present; red-proof = revert Option-3 to the raw secret → fails) and `TestTemplates_InstallGenerator` (all curated control ids present, script version + `data-customer-id` rendered, static-fallback command present, and none of the seven excluded flags appear page-wide). - **Style** (`style.css`): `.gen-controls`/`.gen-radios`/`.gen-radio`/`.gen-check(s)`/number inputs/ `.gen-msg` — dark palette, 2px radius. ## v0.35.0 — OOB operator access: operator peer + oob_peer_ip/oob_operator_ssh_key + OOB health alert (TASK H1) (2026-07-05) The hub half of the merged E1+H1 operator-SSH-access feature (agent half = felhom-agent v0.72.0). - **Operator OOB peer** (`store/wg_operator.go`): the fleet operator peer as an UNBOUND wg_peers row (host_id '', note operator-oob) at an EXPLICIT /32 (so the endpoint's static forward chain can hardcode it); validated in-subnet/not-reserved/not-taken; last-write-wins rotation. It rides ListWGPeers → peersync pushes it to the endpoint. `PUT/GET /admin/wg/operator-peer` (global key); the PUT also takes an optional `ssh_pubkey` (the operator authorized_keys line, hub_settings) and bumps EVERY host's generation. - **Desired-state** (`api/wg.go` mergeWireguard): when an operator peer exists, the served wireguard block carries `oob_peer_ip` (rendered into the box's AllowedIPs — survives self-heal [OF-1]) and `oob_operator_ssh_key` (agent writes felhom-sshd's authorized_keys). Absent → byte-identical. - **OOB health** (`monitor/host_oob.go`): ingests the agent's `oob` heartbeat stanza and raises a transition-based `oob_degraded`/`oob_recovered` warning (felhom-sshd down while the operator peer is configured, OR config invalid) — the proactive "can the operator get in right now" signal. ## v0.34.1 — mgmt_plane_healed alerts on the FIRST auto-heal (TASK G1 fix) (2026-07-05) The mgmt-plane checker seeded a heal marker silently on first observation (copied from HostLeafChecker's trust-on-first-report). A heal is an EVENT, not a baseline: construction still seeds pre-existing markers (startup false-alarm guard), but a newly-observed marker now raises the warning — so the FIRST auto-heal surfaces, matching the live drill. Added tests for both halves. ## v0.34.0 — break-glass recovery vault + mgmt_plane surfacing (TASK G1) (2026-07-05) The hub half of the management-plane break-glass system (prerequisite for felhom-sshd / H1; agent half = felhom-agent v0.71.0). Closes the recovery gap from `documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8/#9. - **Break-glass credential vault** (`store.host_recovery` + `internal/store/host_recovery.go`): a per-host root@pam console password, stored at rest, operator-retrievable — the human fallback for reaching the PVE web console (pveproxy, a failure domain distinct from sshd) when both the sshd path and the agent-independent auto-heal have failed. `PUT /hosts/{id}/recovery-credential` (SELF-scoped host key — day-0 vaults it) + `GET /admin/hosts/{id}/recovery-credential` (GLOBAL key only — a host key cannot read its own console password back). Secret discipline: never logged (username + length only); red-proofed that the password never reaches the hub log. - **mgmt_plane surfacing** (`internal/monitor/host_mgmtplane.go`, on the 60s sweep): parses the agent's additive `mgmt_plane` heartbeat stanza and raises a `mgmt_plane_healed` WARNING when the watchdog auto-healed a missing `/run/sshd` (new `privsep_healed_at`) — a recurring clobber surfaces BEFORE it becomes a lockout, complementing host_staleness. Trust-on-first-report (seed, then alert on change), mirroring HostLeafChecker. ## v0.33.0 — S2 offsite connectivity: box-facing WG registration + wireguard desired-state block + /offsite UI (2026-07-04) Doc 06 roadmap row S2 (commits `fcf84a0`/`ba52005`/`13203c2`); the S2 architectural decision: the stored `desired_json` stays a pure OPERATOR blob — the WG assignment is HUB-owned state, **merged into the served desired-state at READ time**, never written into the store. - **Store** (`internal/store/wg.go` + `store.go`): `RegisterWGPeerForHost` (idempotent / **re-key-in-place keeps the /32** — stable tunnel addressing across rotation/DR / adopt-unbound S1 rows / typed `ErrWGPubkeyBoundElsewhere` — a key is never silently stolen); partial unique index `idx_wg_peers_host` = one bound peer per host; `BumpHostDesired` bumps ONLY the generation (the merge changes served state, not the blob); `allocateWGPeerTx` extracted from the S1 path behavior-neutrally (S1 tests unmodified). `WGPeer` gains `CreatedAt`. - **API** (`internal/api/wg.go` + `handler.go`): `POST /hosts/{id}/wg` — per-host key SELF-SCOPED (global = operator/DR path); generation bump + endpoint push ONLY on real change (idempotent re-register moves nothing — asserted negatives). `mergeWireguard` injects `{endpoint{dns_name,wg_port,server_pubkey,pbs_tunnel_ip}, pubkey, assigned_ip}` into served desired-state; no peer → **byte-identical pass-through** (the cross-repo golden test passes UNMODIFIED); any merge failure → fail-safe unmerged serve (never 500 the control channel). `handleAdminSetDesiredState` REJECTS a top-level `wireguard` key (400 — an operator copy-paste-PUT can never clobber the hub-owned block). Admin DELETE of a BOUND peer bumps the owning host; unbound deletes move no generation. NEW golden `testdata/desired-state-wireguard.golden.json` = **the S3 cross-repo contract** (agent copy must stay byte-identical). - **UI** (`internal/web/offsite.go` + `templates/offsite.html`): read-only `/offsite` page — endpoint card + peer table (truncated pubkeys, full value in title; bound peers link to `/hosts/`); Offsite nav link in all 9 page templates. Mutations stay on the admin API (UI actions arrive with tunnel health, S3/S6). - **Tests:** Groups A/B/C; five red-proofs run + reverted (self-scope drop, unconditional merge, rejection drop, bump-on-idempotent, script exit-swallow — see scripts/CHANGELOG v1.0.1). Old-agent (v0.63.0) tolerance proven live against the real felhom-pve record. ## v0.32.0 + v0.32.1 — S1 offsite connectivity: WG endpoint record + peer registry + pinned-SSH peer-sync (2026-07-04) The hub side of doc 06's roadmap row S1 (`documentation/architecture/06-offsite-connectivity.md`), resolving the slice-1 design point: **peer-sync = hub pushes over SSH** to a forced-command reconcile script on the endpoint (pull/signed-manifest rejected — weakens immediate revocation; HTTPS push API rejected — a new versioned binary + third public port for nothing). - **Store** (`internal/store/wg.go` + migration in `store.go`, commit `b18f6ae`): `wg_endpoints` (single expected row "ep0") + `wg_peers` (presence = desired state; no status column — that's the S2 host-join). `AddWGPeer` = one tx, idempotent on pubkey, lowest-free-host `/32` allocation skipping network/`pbs_tunnel_ip`/broadcast, `UNIQUE(assigned_ip)` race backstop + one internal retry; typed `ErrWGEndpointUnset`/`ErrWGSubnetExhausted`. - **wgsync** (`internal/wgsync/`, commits `fbeeacb` + `0fa7ea1`): `x/crypto/ssh` push client with `ssh.FixedHostKey` pin (no insecure fallback, ever) **+ `HostKeyAlgorithms` constrained to the pinned key's type** — the live validation caught a stock multi-hostkey sshd presenting ECDSA against the ed25519 pin (legitimate server refused); regression-tested with an in-process dual-hostkey SSH server. Reconciler pushes the FULL peer list (never deltas — drift repair by construction) on `Trigger()` or a 5-min tick; payload `{"version":1,"interface":"wg0","peers":[{pubkey, allowed_ip}]}`, deterministic order. - **API** (`internal/api/wg.go`): `PUT/GET /admin/wg/endpoint`, `POST/DELETE/GET /admin/wg/peers` — GLOBAL key only (the `handleAdminSetDesiredState` gate); pubkey validated 44-b64/32-byte; **DELETE takes the pubkey in the JSON body** (base64 `/`+`+` keep pubkeys out of URL paths); mutation responses carry `sync: ok | deferred: | disabled` — the DB is the source of truth, a failed push defers to the reconciler. - **Wiring** (`cmd/hub/main.go`): `WG_ENDPOINT_SSH_{ADDR,USER,KEY_FILE,HOSTKEY}` env (key from the mounted `Secret/wg-endpoint-ssh`, host key non-secret plain env); any piece missing → `[INFO] WG peer-sync disabled` and mutations still work DB-only. - **Tests:** allocator (exact IPs, freed-IP reuse, /30 exhaustion), API auth/validation with a fake syncer, SSH client against an in-process server (exact payload bytes, stderr surfacing, wrong-host-key refusal, multi-hostkey pin), reconciler (full-list, retry-on-tick, no-mutation drift push, removed-peer-absent negative). Four red-proofs run and reverted (allocator-ignores- rows, gate removal, InsecureIgnoreHostKey, delta-only push) — each failed its test. - **Live-validated** end-to-end on the dev endpoint (`felhom-hetzner`, runbook `documentation/runbooks/offsite-endpoint.md`): add → `wg show` on the box; delete → gone (+404/403 paths); malformed payloads leave wg state byte-identical; endpoint reboot → persisted set + hub push converges; client tunnel `ep0.felhom.eu:443` → PBS login page via the wg0-only 8007 rule; public 8007 unreachable. v0.32.1 = the HostKeyAlgorithms fix (0.32.0 image was already pulled by the cluster; tag kept immutable). ## docs — Felhom skills introduced + CLAUDE.md refresh (2026-07-03) Repo-level docs work alongside v0.31.0 (no hub code in this entry): - **`skills/`** (new, repo root): three versioned Claude Code skills — `felhom-build-deploy` (per-artifact runbooks, all commands verified live), `felhom-ui-design` (v2 tokens + gates), `felhom-testing` (non-hollow doctrine + red-proof procedure). Installed to `~/.claude/skills/` via `scripts/install_skills.py` (junction mode verified). - **CLAUDE.md refresh:** the "Hub — current state (v0.7.x)" narrative (stale by ~23 versions) replaced with a version-free architecture section; standing rule adopted — CLAUDE.md carries NO version-pinned state (that lives in CONTEXT/CHANGELOG/REUSE); skills pointers added. Same rule applied to the sibling repos' CLAUDE.md in their own commits. ## v0.31.0 — critical severity accepted at event ingest + visible in UI (2026-07-03) Fixes the gotcha the REUSE sweep surfaced: `handleEvent` coerced any severity outside {info,warning,error} — including `"critical"` — to `"info"` at ingest, so a controller-POSTed critical event never notified even though the dispatcher (`severityNotifies`, v0.24.0) and `FormatOperatorEmail` already handle critical correctly. - **Ingest (`internal/api/handler.go` `handleEvent`):** `"critical"` added to the severity case list. Unknown values (and case-variants like `"Critical"`) still coerce to `"info"` — the exact-match-lowercase coercion contract is kept and now locked by test. - **Hungarian label (`internal/notify/templates.go`):** `severityLabels["critical"] = "Kritikus hiba"` (was missing — customer emails would have shown the raw English word). - **UI counts:** dashboard consumer (`internal/web/server.go`) gains `EventCriticals`; `dashboard.html` renders the critical badge FIRST in the 24h count chain (guard extended); `customer_unified.html` gains the `{{.}} critical` summary badge before errors. - **style.css:** defines the previously-referenced-but-undefined `.severity-critical` (`--crit`/`--crit-dim` tokens) and `.severity-ok` (neutral, exception-color principle). No other restyle. - **Tests (`internal/api/event_test.go`, new):** critical preserved to store (companion red-proof: shown failing against the pre-fix switch — stored `"info"`); unknown severity → info; unknown event_type → 400 + nothing stored; no-auth → 401. First tests on the /event endpoint. - REUSE.md §1/§3 updated in the same commit (the maintenance rule's first outing). ## docs — REUSE.md introduced (2026-07-03) Cross-repo reuse-map rollout (docs-only, no code change, no version bump). New `REUSE.md` at the repo root covering hub + website + scripts + manifests: canonical helpers (34 rows), patterns (monitor checker, website page, gate script, GitOps deploy), dangerous lookalikes (legacy /notify trio, severity-`critical` coercion at handleEvent ingest, inline stringData secrets, kubectl-apply drift…), seams, extension points, and observed duplication (5 clusters, NOT fixed). New `scripts/reuse_refs_check.py` machine-checks every cited path in all four repos' REUSE.md files. CLAUDE.md gains the REUSE.md pointer + same-commit maintenance rule. ## v0.30.1 — status badge no-wrap (2026-07-02) Found in the authenticated D4 validation pass: multi-word status tags (PENDING in a narrow dashboard column, NO REPORT on hosts) wrapped between the CSS dot and the label. One line: `white-space: nowrap` on `.status-badge`. ## v0.30.0 — TASK-D4: design system v2 re-skin (appearance only) (2026-07-02) Last surface of the design sprint (controller D0/D1, website D3). The hub leaves its Tailwind-slate theme for the canonical navy v2 language. **API surface untouched** (`/api/*` ingestion, artifact manifest, config generation, DR/escrow — `git diff` clean under internal/api + internal/store). - **Fonts (`internal/web/static/fonts/`, embed.go, server.go):** the 4 vendored woff2 (byte-copied from felhom-controller; latin-ext for Hungarian customer names in an English UI), embedded and served at `/static/fonts/` (font/woff2, immutable), mirroring the chart.min.js pattern. No CDN before or after. - **`statusColor` semantic remap (server.go):** returns `nominal/warn/crit/neutral` class tokens instead of raw hex colors — ok→nominal, warn+stale→warn, down+fail→crit, pending+disabled→neutral (a not-yet-provisioned or deliberately paused customer is a normal fleet state), blocked→warn (intentional operator cut-off: attention-worthy, not an outage). The inline `style="color: {{statusColor}}"` pattern is dead (dashboard + customer_unified use class-based `.status-dot-`); `statusIcon` ("●") retired. Truth-table test red-proven vs the old implementation; new template-parse test (neither existed for the hub). - **style.css v2:** navy tokens + @font-face; 2px radius; hairline `--line-soft` table rows (fleet-NOC density kept); `.status-badge` re-expressed as an outline tag + CSS dot per the design-system addendum (ok=blue, warn/blocked/stale=amber+dim, down/fail=red+dim, pending/disabled=quiet neutral with hollow dot); severity badges stay filled amber/red (exceptions stay loud); config badges = filled informational chips in v2; row tint only for warn/down. Two-tone brand H1 (`Felhom Hub`) on all pages; 12-symbol Lucide sprite partial included per page. - **Charts (app_detail):** avg memory `#2EA8F5`, peak `#8E7CE8` (secondary DATA series — not status red), catalog-limit line `#E0A93E` (threshold marker); legend/tick/grid → v2 literals. - customer_unified JS status-message colors → blue-bright/crit; login page inline HTML retinted. - Grep gate: all slate hexes (`#0f172a #1e293b #334155 #60a5fa #4ade80 #facc15 #f87171 #94a3b8 #64748b #475569 #e2e8f0`) at zero across internal/web (non-test). ## v0.29.0 — Day-0 artifact manifest: version dropdowns + auto-derived sha (2026-07-01) Removes the hand-copied sha256 from the Day-0 artifact manifest. The operator now **picks a version** from a dropdown of what's actually in Gitea (olders get pruned), and the hub **reads that version's sha256 from Gitea itself** — no transcription, no stale checksums. Keeps the human-in-the-loop trust gate (the operator still deliberately chooses the version; "latest" is never auto-promoted) while the hub stays the checksum trust root. - **`internal/gitea`** (new): a minimal read-only Gitea packages client — `ListVersions` (generic package versions, newest-semver first) + `FileSHA256` (a version's file sha256 via the files-metadata API, **without downloading** the artifact — important for the ~GB golden). Basic-auth with the registry creds the hub already holds. Unit-tested against an httptest server (filter+sort, preferred file match + fallback, non-200 → error). - **Configuration → Day-0 artifacts:** the two version text inputs are now `