# Felhom Hub — Changelog ## v0.43.1 — Git Sync form hint: credentials are optional (2026-07-10) Pairs with controller v0.112.0 (anonymous registry self-update). The config editor's Git Sync section looked load-bearing; in truth the credentials matter only for a private app catalog — version discovery and self-update work without them since controller v0.112.0. One template hint added (`config_form.html`): "Opcionális — csak privát alkalmazás-katalógushoz. A verziófrissítés enélkül is működik." No behavior change. ## v0.43.0 — Remote app-log diagnostics: copyable issues + error context + on-demand log tails (2026-07-10) Pairs with controller v0.111.0. Motivated live: Peti's CWA NFS issue was visible in Known Issues but tooltip-only unreadable and context-free, and there was no way to see the app's actual logs without box access. The hub still never connects into a guest — everything rides the existing report + ACK. - **Part A — readable, copyable issues** (`templates/app_detail.html`): Known Issues rows are click-to-expand — full message in a wrapping monospace `
` + **Copy** button (clipboard API with
  execCommand fallback), fingerprint/severity/first-last-seen in the body. Tooltip-only truncation killed.
- **Part C — context stored + rendered** (`store/telemetry.go`): `app_log_issues` gains `context`
  (JSON array) + `context_customer` (provenance); `upsertAppIssue` stores context on INSERT and adopts a
  later one ONLY while the stored context is empty (**first capture wins** — stable repro, no churn).
  Rendered in the expanded row as "Context around first occurrence — from ", copyable.
  Nil-safe with pre-v0.111 reports.
- **Part D — on-demand ordered log tail (pull-based)**: per-app **"Request log tail"** button on the
  customer page → `log_tail_requests` row (one active per app; re-click refreshes) + a customer-visible
  `log_tail_requested` event (transparency by default). The report ACK advertises
  `log_tail_requests: [app…]` (same additive omit-when-empty pattern as escrow); the controller's next
  report ships `log_tails` → stored in `app_log_tails` (transient, **last 2 per app kept**) and the
  request is **cleared (consume-once)**. Ordered tail view with line numbers (`log_tail.html`) +
  **Download .log**; tail reads are customer-scoped.
- **Part F fix — the 24h/7d/30d selector now filters Known Issues**: `GetAppIssues` gained the same
  `since` cutoff the Memory Trend uses (it had NO time filter — a 24h view showed 25-day-old rows).
- **Part G — deletion → dismissal**: diagnosis = the delete handler was NOT broken; deletion is futile
  because the controller re-scans its rolling 15-minute window every report and re-upserts a still-
  occurring fingerprint with fresh `last_seen` minutes later. Replaced with `dismissed_at`:
  Dismiss Selected/All (buttons renamed), dismissed rows out of the default view ("Show dismissed"
  toggle), and `upsertAppIssue` un-dismisses ONLY on `excluded.last_seen > dismissed_at` — a re-sent
  old window stays hidden, a genuinely NEW occurrence resurfaces (recurrence never silently swallowed).
- **Part H — per-customer scoping**: `?customer=` on the app detail page filters Known Issues to
  rows whose `affected_customers` contains the id (header shows "filtered: "); the customer
  page's App Telemetry rows link there (the drill-down). The fleet view stays; the expanded row lists
  the affected customers explicitly (linked) and the count column is labeled
  "Occurrences (all customers)".
- **Tests + red-proofs (all four failed exactly as designed, restored green):** dismissal guard dropped
  → old-window re-report resurrected the row → FAIL; range predicate neutered → 10d-old issue visible
  at 24h → FAIL; first-capture-wins dropped → empty-context upsert clobbered stored context → FAIL;
  consume-once DELETE removed → request survived fulfillment (store test + API ACK round-trip both) →
  FAIL. Plus: late-context adoption, warn-no-context, occurrence counting, tail request/fulfill/prune-
  to-2/cross-customer-404, ACK omit-when-empty baseline, render tests (expanded row content, customer
  page sections, ordered tail view + download headers).

## v0.42.0 — Remote "Debug mód" toggle on the customer config editor (2026-07-10)

Lets an operator flip the controller's debug mode (verbose log + the `/debug` menu, which the controller
gates on `Logging.Level=="debug"` / `isDebug()`) **remotely, without SSH** — the support workflow (today:
Peti's box). The config-version bump on save makes the controller re-pull + self-restart on its next
report ACK, so the switch takes effect within a cycle, hands-free.

- **Form field, not raw-JSON injection — on purpose.** `handleConfigUpdate` REBUILDS `ConfigJSON` from the
  form on every save (`buildConfigJSON`), so any foreign key injected straight into the stored JSON is
  dropped on the next save. The toggle is therefore a real form field, which by definition survives every
  save. (The offsite descriptor survives via its own separate provision-merge, untouched by this.)
- **`buildConfigJSON`** (`internal/web/configs.go`): `debug_mode` checked → emits `"logging":{"level":"debug"}`;
  unchecked → the `logging` key is OMITTED entirely (the generated `controller.yaml` default stands — no
  needless `"info"`).
- **Config form** (`templates/config_form.html`): new collapsible **"Hibakeresési mód (fejlesztői)"** section
  with the `debug_mode` checkbox; render state parsed back from `ConfigJSON` (`logging.level=="debug"` → checked).
  No change to the offsite/CF/git leg; no generic raw-JSON editor (deliberately — validated surfaces only).
- **Tests (non-hollow, `configs_debug_test.go`):** form→JSON both ways (checked emits / unchecked omits);
  full-path **survival test** through `handleConfigUpdate` proving the debug key lands, the offsite descriptor
  is byte-for-byte unchanged across save+re-provision, and the **red-proof** that a hand-injected foreign key
  is gone after one save (why the switch must be a form field); render state both ways. Red-proof exercised
  (feature disabled → survival + form tests fail).

## v0.41.0 — SLICE 4: OffsiteChecker (fill + staleness) + operator freeze lever (2026-07-09)

The last build item of the offsite arc (pairs with controller v0.109.0's soft-quota gate + report object).

- **`internal/monitor.OffsiteChecker`** — a SIBLING of StorageFillChecker (same born/persistent,
  escalation-only, recovery-re-arm shape; NOT bolted onto the disk checkers), reading the controller
  report's new `offsite` object. Two signals: **fill** (`repo_size_bytes` vs `quota_gb` at warn 90 / crit
  95 — quota 0 = dedicated, never alerts) and **staleness** (`offsite_stale`, warning): enabled+**escrowed**
  but no run in >48h (or never) — the silently-STUCK detector; a recently-FAILING offsite is not stale
  (`backup_failed` owns that), and pending/disabled targets never alert (normal onboarding — **companion
  red-proof:** dropped the escrowed-only filter → the pending customer alerted → test FAILED). Nil-safe on
  reports without the object (pre-v0.109 controllers). Tie-guard: duplicate same-second latest reports are
  processed once per sweep. Same 60s sweep as the other checkers.
- **Freeze lever (operator, MANUAL only):** `Provisioner.SetOffsiteFrozen` — flips ONLY `readonly` on the
  exactly-1 labelled sub-account via `UpdateSubaccountAccess` (SSH stays on; ambiguity refuses — tested),
  wired to confirm-gated **Freeze/Unfreeze offsite** buttons next to Re-issue (shared model only; dedicated
  is Hetzner-enforced). NEVER automatic — freezing also blocks prune, the customer's only way DOWN from
  over-quota. Route `POST /configs/{id}/offsite-freeze` (`unfreeze=1` reverses); action logged, value-free.

## v0.40.0 — SLICE 3: store the escrow password-hash + serve escrow status in the report ACK (2026-07-09)

The hub-verified escrow auto-confirm chain, hub third (pairs with agent v0.79.0 + controller v0.108.0).
The controller must verify the RIGHT fact — not "a blob exists" but "the blob covers the CURRENT repo
password" — so the hub records WHICH password each escrow covers, as a non-reversible sha256 (a 256-bit
random secret's hash is safe to store/serve; the password itself never reaches the hub).

- `internal/store`: additive migration `ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`
  (NULL on legacy rows — e.g. the demo's — which therefore never auto-confirm; the deprecated manual
  confirm covers them). `HostEscrow.ResticPwSHA256` + `SaveHostEscrow` gains the param (last-write-wins);
  NULL-safe reads via COALESCE. New `GetEscrowStatusForCustomer` (hosts⋈host_escrow; latest-updated wins).
- `internal/api`: `escrowUploadRequest.restic_pw_sha256,omitempty` (the agent emit struct's mirror —
  `TestEscrowUploadContract` updated in lockstep with the agent's half); stored on upload. The **report ACK**
  gains `escrow: {identity_blob_present, restic_pw_sha256, created_at}` — omitted entirely when the customer
  has no escrow row (a fresh customer stays pending silently).
- Tests: hash stored + legacy-upload reads back NULL-safe as ""; ACK carries the object / omits it without a
  row; contract mirror.

## v0.39.0 — offsite hardening: F4 credential re-issue + F2 scan retry + F5 save UX (2026-07-09)

Part of the offsite-provisioning hardening bundle (pairs with controller v0.107.0 + agent v0.78.0); the
sharp edges from the live e2e (`documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`).

- **F4 (pilot-gating) — "Re-issue offsite credentials":** `Provisioner.ReissueCredentials` — the EXPLICIT
  operator recovery for a consumed-password dead-end (fresh-guest DR; consumed-but-failed install). Resets
  the customer's sub-account password (`ResetSubaccountPassword`) or dedicated-box password (new
  `ResetBoxPassword` in `hetznerapi`, client+interface+fake) → stores a FRESH one-time secret → the handler
  re-saves the config unchanged so `ConfigVersion` bumps and the stuck guest's next refresh re-runs the
  bridge. **Hard-scoped:** targets ONLY the resource labelled `felhom-customer=`; refuses unless the
  label lookup finds exactly 1 (ambiguity = refuse, no reset, no secret) **+ companion red-proof** (dropped
  the exactly-1 guard → ambiguous lookup proceeded → test FAILED). NOT implicit rotation — `ProvisionOffsite`
  never calls it. UI: a confirm-gated button on the config form (shown only when provisioned), route
  `POST /configs/{id}/offsite-reissue` (CSRF rides the parent form). The password value is never logged.
- **F2 — host-key scan retry-with-backoff:** a fresh sub-account's DNS lags creation, so the FIRST save
  502'd (`no such host`, live). `scanWithRetry` retries on failure (default ladder 2/4/8/16/30s ≈ 60s total,
  inside applyOffsite's 3-min detached ctx; ctx-abortable; fail-closed past the budget) **+ companion
  red-proof** (disabled the retry loop → DNS-lag save failed → test FAILED). `Provisioner.ScanBackoff`
  injectable for tests.
- **F5 — save UX:** the config form disables its submit buttons and shows an in-flight notice on submit
  (the ~25–60s spinner-less save was the re-click bait that caused F1 live).

## v0.38.1 — offsite provisioning: detach from the client's request context (live finding F1) (2026-07-09)

Found in the first supervised live run: the offsite save takes ~25s (create + wait + host-key scan) with no
UI feedback, the operator re-clicked, the browser abandoned the first request, and `r.Context()` was canceled
**between `CreateSubaccount` and `SaveOneTimeSecret`** — the sub-account was created on Hetzner but its
one-time password was lost forever (the controller's consume 404s permanently; stranded resource).

- `internal/web.applyOffsite`: provisioning now runs on `context.WithoutCancel(r.Context())` with a 3-minute
  absolute timeout — once the create starts, the create→wait→store atom runs to completion even if the client
  disconnects. Fail-closed behavior unchanged (an actual provisioning error still 502s and saves nothing).
- Test `TestApplyOffsite_ClientDisconnectMidProvision` (a ctx-honoring fake cancels the request context
  mid-create): the one-time password must reach the store and the descriptor must merge despite the
  disconnect. **Companion red-proof:** reverted to the raw request ctx → the exact live error
  (`subaccount create action: context canceled`) → test FAILED. Restored.
- Known residuals (recorded, not fixed here): the form has no in-flight spinner/disable (the re-click bait),
  and a concurrent save can still hit Hetzner's box-level HTTP 423 action lock (surfaces as the fail-closed 502).

## v0.38.0 — offsite provisioning SLICE 2 (hub side): capture the box host-key fingerprint (2026-07-09)

Pairs with controller v0.106.0. So the controller can VERIFY the box identity instead of blind-TOFU, the hub
captures the box's SSH host-key fingerprint at provision and serves it in the descriptor.

- `internal/offsite`: `Descriptor.HostFingerprint` (SHA256:…, non-secret). `ProvisionOffsite` now captures it
  after the resource is ready via a `HostKeyScanner` seam (`SSHHostKeyScanner`, x/crypto/ssh — dials port 23
  and grabs the host key from the handshake, no ssh binary needed). **Fail-closed:** a nil scanner or a scan
  failure returns an error (don't serve a descriptor the controller can't verify). The controller re-scans and
  refuses on mismatch (v0.106.0).
- Tests: descriptor carries the fingerprint from a faked scanner; a scan failure fails-closed.

## v0.37.0 — offsite provisioning SLICE 1: Hetzner Cloud-API client + provisioning core (2026-07-09)

Slice 1 of the offsite-provisioning epic. On operator enable, the hub provisions a Hetzner storage-box
sub-account (shared) or dedicated box, generates the transient password, stores it one-time-consumable, and
serves the non-secret target descriptor to the controller via `ConfigJSON`. Coded against the API shapes
measured live in `documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (996d403). The
controller apply-bridge (SLICE 2), escrow auto-confirm (SLICE 3), and soft-quota enforcement (SLICE 4) are
separate slices.

- **`internal/hetznerapi`:** a typed client for the storage-box surface at **`https://api.hetzner.com/v1`**
  (NOT `api.hetzner.cloud` — the classic Cloud API 404s for storage boxes). Sub-account + box
  create/reset/access/change_type/delete/list-by-label + `WaitAction` (poll to `success`, bounded). A
  `CloudAPI` interface + an exported `Fake` so provisioning is unit-tested with **no live Hetzner calls**.
  Bearer token from an injected func (out-of-band secret; never logged).
- **`internal/offsite`:** `Provisioner.ProvisionOffsite` — **idempotent by `label_selector`
  (felhom-customer=)** (names aren't unique); shared → sub-account on the pool box, dedicated → box;
  generates a 4-class transient password → `WaitAction` → `Store.SaveOneTimeSecret` → builds the NON-SECRET
  `Descriptor{enabled,type,host,user,port:23,repo_path:/home/felhom-repo, quota_gb|box_type}`. **Fail-closed:**
  any API/action error returns without a provisioned resource, one-time password, or descriptor.
  `MergeDescriptor` merges it under the `offsite` key of `ConfigJSON` (never a secret).
- **`internal/store`:** `one_time_secrets` table + `SaveOneTimeSecret` / `ConsumeOneTimeSecret` (single-use,
  return-and-mark in one tx). The transient password NEVER rides `ConfigJSON`.
- **`internal/api`:** `POST /offsite/consume-password/{id}` — serves the one-time password to the
  authenticated customer (same API-key auth as config-pull) EXACTLY once, then 404s. Never logged.
- **`internal/web`:** the config form gains an **Offsite backup** section (enable / type / soft-quota / box
  type); save → `applyOffsite` provisions (fail-closed: a provisioning error returns 502 and does NOT save)
  and merges the descriptor → `ConfigVersion` bump → controller re-pulls. Optional dep
  (`SetOffsiteProvisioner`), wired in `cmd/hub/main.go` from `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID`/`HETZNER_LOCATION`.
- Tests (faked Cloud API, no live calls): shared/dedicated provision + descriptor + one-time-password-stored
  + **password-absent-from-ConfigJSON**; idempotent re-save (no 2nd resource); fail-closed **+ companion
  red-proof** (swallow the create error → offsite marked enabled despite failure → test fails); one-time
  consume-once; `WaitAction` success/error/timeout; the consume endpoint (auth + single-use).
- **NOT yet live-provisioned** — awaiting the **dedicated-project scoped token** (the current token can delete
  ep0 — SPIKE §6); a live create is a supervised validation. Unit tests are this slice's proof.

## v0.36.0 — customer page: passphrase hardening + interactive install-command generator (TASK GL-7) (2026-07-09)

Two coupled, security-first changes to the operator-facing customer page (`customer_unified.html` +
`configs.go`). felhom.eu only; agent + host-install untouched.

- **Passphrase hardening (ships the security win).** The per-customer retrieval passphrase was
  rendered in cleartext twice — the visible `#retrieval-pw` node and baked into the Option-3 debug
  curl's `X-Retrieval-Password:` header. Now: `#retrieval-pw` renders a masked bullet run by default
  with **reveal** (`toggleSecret`) + **copy** (`copySecret`) controls, the value carried in
  `data-secret` (the existing reveal model). The Option-3 command carries a
  `` placeholder — the secret is NEVER in a copyable command block. (A
  zero-secret-in-DOM reveal-on-demand fetch is a deliberate future follow-up, not this task.)
- **Interactive install-command generator.** The three hard-coded install `` blocks became a
  client-side builder (vanilla JS — no framework, CDN, or network) that assembles a live-updating
  command from form controls, emitting ONLY real host-install **v1.12.0** flags in a
  download-then-run shape (never `curl | bash`). CustomerID is prefilled from the server
  (`ScriptVersion`/`data-customer-id` via `pageData`); a byo selection requires `--cores`/`--memory`
  (enforced client-side with a `.gen-req`/`gen-msg` prompt); caps/mode are placeholders, never silent
  defaults. Graceful **JS-off static fallback**: the Option-1/2 code nodes keep a
  `--customer-id … --mode ` command. The curated control surface excludes the seven
  dangerous/operator-only flags (`--force`, `--rotate-recovery`, `--enable-oob`, `--remove-golden`,
  `--uninstall`, `--adopt-pool`, `--rescope-acl`) — they are never offered as controls.
- **`configs.go`:** `const hostInstallVersion = "1.12.0"`; `pageData.ScriptVersion` added + populated.
- **Tests** (`render_test.go`): `TestTemplates_PassphraseHardened` (secret NOT in the Option-3
  command, placeholder present, masked-by-default bullet run, `data-secret` populated, reveal/copy
  controls present; red-proof = revert Option-3 to the raw secret → fails) and
  `TestTemplates_InstallGenerator` (all curated control ids present, script version + `data-customer-id`
  rendered, static-fallback command present, and none of the seven excluded flags appear page-wide).
- **Style** (`style.css`): `.gen-controls`/`.gen-radios`/`.gen-radio`/`.gen-check(s)`/number inputs/
  `.gen-msg` — dark palette, 2px radius.

## v0.35.0 — OOB operator access: operator peer + oob_peer_ip/oob_operator_ssh_key + OOB health alert (TASK H1) (2026-07-05)

The hub half of the merged E1+H1 operator-SSH-access feature (agent half = felhom-agent v0.72.0).

- **Operator OOB peer** (`store/wg_operator.go`): the fleet operator peer as an UNBOUND wg_peers row
  (host_id '', note operator-oob) at an EXPLICIT /32 (so the endpoint's static forward chain can
  hardcode it); validated in-subnet/not-reserved/not-taken; last-write-wins rotation. It rides
  ListWGPeers → peersync pushes it to the endpoint. `PUT/GET /admin/wg/operator-peer` (global key);
  the PUT also takes an optional `ssh_pubkey` (the operator authorized_keys line, hub_settings) and
  bumps EVERY host's generation.
- **Desired-state** (`api/wg.go` mergeWireguard): when an operator peer exists, the served wireguard
  block carries `oob_peer_ip` (rendered into the box's AllowedIPs — survives self-heal [OF-1]) and
  `oob_operator_ssh_key` (agent writes felhom-sshd's authorized_keys). Absent → byte-identical.
- **OOB health** (`monitor/host_oob.go`): ingests the agent's `oob` heartbeat stanza and raises a
  transition-based `oob_degraded`/`oob_recovered` warning (felhom-sshd down while the operator peer
  is configured, OR config invalid) — the proactive "can the operator get in right now" signal.

## v0.34.1 — mgmt_plane_healed alerts on the FIRST auto-heal (TASK G1 fix) (2026-07-05)

The mgmt-plane checker seeded a heal marker silently on first observation (copied from HostLeafChecker's
trust-on-first-report). A heal is an EVENT, not a baseline: construction still seeds pre-existing markers
(startup false-alarm guard), but a newly-observed marker now raises the warning — so the FIRST auto-heal
surfaces, matching the live drill. Added tests for both halves.

## v0.34.0 — break-glass recovery vault + mgmt_plane surfacing (TASK G1) (2026-07-05)

The hub half of the management-plane break-glass system (prerequisite for felhom-sshd / H1; agent half
= felhom-agent v0.71.0). Closes the recovery gap from
`documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8/#9.

- **Break-glass credential vault** (`store.host_recovery` + `internal/store/host_recovery.go`): a
  per-host root@pam console password, stored at rest, operator-retrievable — the human fallback for
  reaching the PVE web console (pveproxy, a failure domain distinct from sshd) when both the sshd path
  and the agent-independent auto-heal have failed. `PUT /hosts/{id}/recovery-credential` (SELF-scoped
  host key — day-0 vaults it) + `GET /admin/hosts/{id}/recovery-credential` (GLOBAL key only — a host
  key cannot read its own console password back). Secret discipline: never logged (username + length
  only); red-proofed that the password never reaches the hub log.
- **mgmt_plane surfacing** (`internal/monitor/host_mgmtplane.go`, on the 60s sweep): parses the agent's
  additive `mgmt_plane` heartbeat stanza and raises a `mgmt_plane_healed` WARNING when the watchdog
  auto-healed a missing `/run/sshd` (new `privsep_healed_at`) — a recurring clobber surfaces BEFORE it
  becomes a lockout, complementing host_staleness. Trust-on-first-report (seed, then alert on change),
  mirroring HostLeafChecker.

## v0.33.0 — S2 offsite connectivity: box-facing WG registration + wireguard desired-state block + /offsite UI (2026-07-04)

Doc 06 roadmap row S2 (commits `fcf84a0`/`ba52005`/`13203c2`); the S2 architectural decision:
the stored `desired_json` stays a pure OPERATOR blob — the WG assignment is HUB-owned state,
**merged into the served desired-state at READ time**, never written into the store.

- **Store** (`internal/store/wg.go` + `store.go`): `RegisterWGPeerForHost` (idempotent /
  **re-key-in-place keeps the /32** — stable tunnel addressing across rotation/DR / adopt-unbound
  S1 rows / typed `ErrWGPubkeyBoundElsewhere` — a key is never silently stolen); partial unique
  index `idx_wg_peers_host` = one bound peer per host; `BumpHostDesired` bumps ONLY the
  generation (the merge changes served state, not the blob); `allocateWGPeerTx` extracted from
  the S1 path behavior-neutrally (S1 tests unmodified). `WGPeer` gains `CreatedAt`.
- **API** (`internal/api/wg.go` + `handler.go`): `POST /hosts/{id}/wg` — per-host key
  SELF-SCOPED (global = operator/DR path); generation bump + endpoint push ONLY on real change
  (idempotent re-register moves nothing — asserted negatives). `mergeWireguard` injects
  `{endpoint{dns_name,wg_port,server_pubkey,pbs_tunnel_ip}, pubkey, assigned_ip}` into served
  desired-state; no peer → **byte-identical pass-through** (the cross-repo golden test passes
  UNMODIFIED); any merge failure → fail-safe unmerged serve (never 500 the control channel).
  `handleAdminSetDesiredState` REJECTS a top-level `wireguard` key (400 — an operator
  copy-paste-PUT can never clobber the hub-owned block). Admin DELETE of a BOUND peer bumps the
  owning host; unbound deletes move no generation. NEW golden
  `testdata/desired-state-wireguard.golden.json` = **the S3 cross-repo contract** (agent copy
  must stay byte-identical).
- **UI** (`internal/web/offsite.go` + `templates/offsite.html`): read-only `/offsite` page —
  endpoint card + peer table (truncated pubkeys, full value in title; bound peers link to
  `/hosts/`); Offsite nav link in all 9 page templates. Mutations stay on the admin API
  (UI actions arrive with tunnel health, S3/S6).
- **Tests:** Groups A/B/C; five red-proofs run + reverted (self-scope drop, unconditional
  merge, rejection drop, bump-on-idempotent, script exit-swallow — see scripts/CHANGELOG
  v1.0.1). Old-agent (v0.63.0) tolerance proven live against the real felhom-pve record.

## v0.32.0 + v0.32.1 — S1 offsite connectivity: WG endpoint record + peer registry + pinned-SSH peer-sync (2026-07-04)

The hub side of doc 06's roadmap row S1 (`documentation/architecture/06-offsite-connectivity.md`),
resolving the slice-1 design point: **peer-sync = hub pushes over SSH** to a forced-command
reconcile script on the endpoint (pull/signed-manifest rejected — weakens immediate revocation;
HTTPS push API rejected — a new versioned binary + third public port for nothing).

- **Store** (`internal/store/wg.go` + migration in `store.go`, commit `b18f6ae`): `wg_endpoints`
  (single expected row "ep0") + `wg_peers` (presence = desired state; no status column — that's
  the S2 host-join). `AddWGPeer` = one tx, idempotent on pubkey, lowest-free-host `/32` allocation
  skipping network/`pbs_tunnel_ip`/broadcast, `UNIQUE(assigned_ip)` race backstop + one internal
  retry; typed `ErrWGEndpointUnset`/`ErrWGSubnetExhausted`.
- **wgsync** (`internal/wgsync/`, commits `fbeeacb` + `0fa7ea1`): `x/crypto/ssh` push client with
  `ssh.FixedHostKey` pin (no insecure fallback, ever) **+ `HostKeyAlgorithms` constrained to the
  pinned key's type** — the live validation caught a stock multi-hostkey sshd presenting ECDSA
  against the ed25519 pin (legitimate server refused); regression-tested with an in-process
  dual-hostkey SSH server. Reconciler pushes the FULL peer list (never deltas — drift repair by
  construction) on `Trigger()` or a 5-min tick; payload
  `{"version":1,"interface":"wg0","peers":[{pubkey, allowed_ip}]}`, deterministic order.
- **API** (`internal/api/wg.go`): `PUT/GET /admin/wg/endpoint`, `POST/DELETE/GET /admin/wg/peers`
  — GLOBAL key only (the `handleAdminSetDesiredState` gate); pubkey validated 44-b64/32-byte;
  **DELETE takes the pubkey in the JSON body** (base64 `/`+`+` keep pubkeys out of URL paths);
  mutation responses carry `sync: ok | deferred:  | disabled` — the DB is the source of
  truth, a failed push defers to the reconciler.
- **Wiring** (`cmd/hub/main.go`): `WG_ENDPOINT_SSH_{ADDR,USER,KEY_FILE,HOSTKEY}` env (key from
  the mounted `Secret/wg-endpoint-ssh`, host key non-secret plain env); any piece missing →
  `[INFO] WG peer-sync disabled` and mutations still work DB-only.
- **Tests:** allocator (exact IPs, freed-IP reuse, /30 exhaustion), API auth/validation with a
  fake syncer, SSH client against an in-process server (exact payload bytes, stderr surfacing,
  wrong-host-key refusal, multi-hostkey pin), reconciler (full-list, retry-on-tick, no-mutation
  drift push, removed-peer-absent negative). Four red-proofs run and reverted (allocator-ignores-
  rows, gate removal, InsecureIgnoreHostKey, delta-only push) — each failed its test.
- **Live-validated** end-to-end on the dev endpoint (`felhom-hetzner`, runbook
  `documentation/runbooks/offsite-endpoint.md`): add → `wg show` on the box; delete → gone
  (+404/403 paths); malformed payloads leave wg state byte-identical; endpoint reboot → persisted
  set + hub push converges; client tunnel `ep0.felhom.eu:443` → PBS login page via the wg0-only
  8007 rule; public 8007 unreachable. v0.32.1 = the HostKeyAlgorithms fix (0.32.0 image was
  already pulled by the cluster; tag kept immutable).

## docs — Felhom skills introduced + CLAUDE.md refresh (2026-07-03)

Repo-level docs work alongside v0.31.0 (no hub code in this entry):
- **`skills/`** (new, repo root): three versioned Claude Code skills — `felhom-build-deploy`
  (per-artifact runbooks, all commands verified live), `felhom-ui-design` (v2 tokens + gates),
  `felhom-testing` (non-hollow doctrine + red-proof procedure). Installed to `~/.claude/skills/`
  via `scripts/install_skills.py` (junction mode verified).
- **CLAUDE.md refresh:** the "Hub — current state (v0.7.x)" narrative (stale by ~23 versions)
  replaced with a version-free architecture section; standing rule adopted — CLAUDE.md carries NO
  version-pinned state (that lives in CONTEXT/CHANGELOG/REUSE); skills pointers added. Same rule
  applied to the sibling repos' CLAUDE.md in their own commits.

## v0.31.0 — critical severity accepted at event ingest + visible in UI (2026-07-03)

Fixes the gotcha the REUSE sweep surfaced: `handleEvent` coerced any severity outside
{info,warning,error} — including `"critical"` — to `"info"` at ingest, so a controller-POSTed
critical event never notified even though the dispatcher (`severityNotifies`, v0.24.0) and
`FormatOperatorEmail` already handle critical correctly.

- **Ingest (`internal/api/handler.go` `handleEvent`):** `"critical"` added to the severity case
  list. Unknown values (and case-variants like `"Critical"`) still coerce to `"info"` — the
  exact-match-lowercase coercion contract is kept and now locked by test.
- **Hungarian label (`internal/notify/templates.go`):** `severityLabels["critical"] = "Kritikus hiba"`
  (was missing — customer emails would have shown the raw English word).
- **UI counts:** dashboard consumer (`internal/web/server.go`) gains `EventCriticals`;
  `dashboard.html` renders the critical badge FIRST in the 24h count chain (guard extended);
  `customer_unified.html` gains the `{{.}} critical` summary badge before errors.
- **style.css:** defines the previously-referenced-but-undefined `.severity-critical`
  (`--crit`/`--crit-dim` tokens) and `.severity-ok` (neutral, exception-color principle). No other
  restyle.
- **Tests (`internal/api/event_test.go`, new):** critical preserved to store (companion red-proof:
  shown failing against the pre-fix switch — stored `"info"`); unknown severity → info; unknown
  event_type → 400 + nothing stored; no-auth → 401. First tests on the /event endpoint.
- REUSE.md §1/§3 updated in the same commit (the maintenance rule's first outing).

## docs — REUSE.md introduced (2026-07-03)

Cross-repo reuse-map rollout (docs-only, no code change, no version bump). New `REUSE.md` at the
repo root covering hub + website + scripts + manifests: canonical helpers (34 rows), patterns
(monitor checker, website page, gate script, GitOps deploy), dangerous lookalikes (legacy /notify
trio, severity-`critical` coercion at handleEvent ingest, inline stringData secrets, kubectl-apply
drift…), seams, extension points, and observed duplication (5 clusters, NOT fixed). New
`scripts/reuse_refs_check.py` machine-checks every cited path in all four repos' REUSE.md files.
CLAUDE.md gains the REUSE.md pointer + same-commit maintenance rule.

## v0.30.1 — status badge no-wrap (2026-07-02)

Found in the authenticated D4 validation pass: multi-word status tags (PENDING in a narrow
dashboard column, NO REPORT on hosts) wrapped between the CSS dot and the label. One line:
`white-space: nowrap` on `.status-badge`.


## v0.30.0 — TASK-D4: design system v2 re-skin (appearance only) (2026-07-02)

Last surface of the design sprint (controller D0/D1, website D3). The hub leaves its Tailwind-slate
theme for the canonical navy v2 language. **API surface untouched** (`/api/*` ingestion, artifact
manifest, config generation, DR/escrow — `git diff` clean under internal/api + internal/store).

- **Fonts (`internal/web/static/fonts/`, embed.go, server.go):** the 4 vendored woff2 (byte-copied
  from felhom-controller; latin-ext for Hungarian customer names in an English UI), embedded and
  served at `/static/fonts/` (font/woff2, immutable), mirroring the chart.min.js pattern. No CDN
  before or after.
- **`statusColor` semantic remap (server.go):** returns `nominal/warn/crit/neutral` class tokens
  instead of raw hex colors — ok→nominal, warn+stale→warn, down+fail→crit, pending+disabled→neutral
  (a not-yet-provisioned or deliberately paused customer is a normal fleet state), blocked→warn
  (intentional operator cut-off: attention-worthy, not an outage). The inline
  `style="color: {{statusColor}}"` pattern is dead (dashboard + customer_unified use class-based
  `.status-dot-`); `statusIcon` ("●") retired. Truth-table test red-proven vs the old
  implementation; new template-parse test (neither existed for the hub).
- **style.css v2:** navy tokens + @font-face; 2px radius; hairline `--line-soft` table rows
  (fleet-NOC density kept); `.status-badge` re-expressed as an outline tag + CSS dot per the
  design-system addendum (ok=blue, warn/blocked/stale=amber+dim, down/fail=red+dim,
  pending/disabled=quiet neutral with hollow dot); severity badges stay filled amber/red
  (exceptions stay loud); config badges = filled informational chips in v2; row tint only for
  warn/down. Two-tone brand H1 (`Felhom Hub`) on all pages; 12-symbol Lucide sprite
  partial included per page.
- **Charts (app_detail):** avg memory `#2EA8F5`, peak `#8E7CE8` (secondary DATA series — not
  status red), catalog-limit line `#E0A93E` (threshold marker); legend/tick/grid → v2 literals.
- customer_unified JS status-message colors → blue-bright/crit; login page inline HTML retinted.
- Grep gate: all slate hexes (`#0f172a #1e293b #334155 #60a5fa #4ade80 #facc15 #f87171 #94a3b8
  #64748b #475569 #e2e8f0`) at zero across internal/web (non-test).

## v0.29.0 — Day-0 artifact manifest: version dropdowns + auto-derived sha (2026-07-01)

Removes the hand-copied sha256 from the Day-0 artifact manifest. The operator now **picks a version**
from a dropdown of what's actually in Gitea (olders get pruned), and the hub **reads that version's
sha256 from Gitea itself** — no transcription, no stale checksums. Keeps the human-in-the-loop trust
gate (the operator still deliberately chooses the version; "latest" is never auto-promoted) while the
hub stays the checksum trust root.

- **`internal/gitea`** (new): a minimal read-only Gitea packages client — `ListVersions` (generic
  package versions, newest-semver first) + `FileSHA256` (a version's file sha256 via the files-metadata
  API, **without downloading** the artifact — important for the ~GB golden). Basic-auth with the
  registry creds the hub already holds. Unit-tested against an httptest server (filter+sort, preferred
  file match + fallback, non-200 → error).
- **Configuration → Day-0 artifacts:** the two version text inputs are now `