Commit Graph

128 Commits

Author SHA1 Message Date
admin 7f11cfb36c hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)
Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.

Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.

- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
  path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
  cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
  no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
  neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
  60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
  the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).

Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
2026-07-17 21:13:30 +02:00
admin a4268c3707 hub: offsite pool-box checker logs the pool trend on each refresh (operator visibility) 2026-07-17 20:20:20 +02:00
admin 4bb2df0dc4 hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5)
The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).

Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).

- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
  method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
  15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
  + OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
  Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
  excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
  last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
  defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
  60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
  dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".

Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
2026-07-17 20:15:34 +02:00
admin 30972d8f54 hub v0.63.0 — system-initiated immediacy: wire poke/bump at every mutation site that lacked one
The immediate-sync arc covered only operator-initiated desired-state changes;
system-initiated mutations bumped the generation silently, so a freshly onboarded
box waited a full agent tick for state the hub had already minted (observed live at
slice-C onboarding). Wire the existing, live-proven notifiers into every system site
on the correct plane — call-site wiring only, no new mechanism.

Agent plane (poke.Notifier):
- web/pbsdr.go: PBSDRAutoProvision (the observed lag), ReissuePBSDR (also lifts the
  pbsdrheal reconciler escalation, zero reconciler changes), handlePBSDRReissue —
  each pokes AFTER the successful SetHostDesired, never on a blocked/error path.
- api: new nil-safe Poker seam (PokeHost/PokeAllHosts + SetPoker); handleAdminSetDesiredState
  pokes the target host; handleAdminSetOperatorPeer fires PokeAllHosts only when the
  fleet generation bump succeeded (fire-after-commit).
- main.go: one poke.Notifier now feeds both planes (SetPoke + SetPoker).

Controller plane (intent.Hub.Bump):
- api/reissueOnReenroll: one nil-guarded bump so a long-polling controller wakes in
  seconds instead of on the 15-min cycle.

Deliberate non-sites (unchanged): WG register (undeliverable pre-tunnel — the agent
fast-tick SECONDARY owns it), WG delete (transport removed), pbsdrheal Restage (no
generation bump → the 60s ticker is the pickup path). internal/pbsdrheal byte-unchanged.

Tests: 10 non-hollow tests (web async channel-synchronized fake sender; api synchronous
fake Poker) with explicit zero-count negatives; representative red-proofs per group
(A/B/C/D) run-fail-restored. Green: go build/vet/test all pass.
2026-07-17 17:30:41 +02:00
admin 36c5cd5fdf hub v0.62.0 + scripts v1.19.0 — R-21 slice C: the universal secret-free ISO
A generic ISO carries NO customer secret. The box registers itself at the hub
as an unclaimed appliance; the operator binds it to a customer; the hub delivers
the customer-id + retrieval passphrase ONCE; day-0 completes via the slice-A path.

Hub (v0.62.0):
- store/appliance.go: appliance_registrations keyed by (uuid, mac_set) — MAC set
  is the tiebreaker (duplicate SMBIOS UUIDs); token stored as sha256 only.
  Idempotent register (sticky-discard), atomic one-shot delivery, bind/discard.
- api/appliance.go: POST /appliance/register (the one unauth endpoint, per-IP
  rate-limited, 256-bit token); GET /appliance/poll (404 no-oracle / 204 unbound
  / 200 deliver-once / 410 delivered). Passphrase read live, never logged.
- web/appliances.go: Hosts-page "Unclaimed appliances" section + BIND (customer
  picker, host count display-only) + DISCARD; SSH host-key fingerprints; events.
- Red-proofs: one-shot delivery + register idempotency (both proven red);
  404-no-oracle, sticky-discard, bind staging, render. Green + confirm gate.

Scripts (v1.19.0):
- felhom-bootstrap.sh: ONE unit, TWO modes. Direct (env has customer/passphrase)
  = slice-A path, byte-identical, only branched around. Pairing (generic) =
  register + poll (RestartSec=30 is the poll timer); on delivery write the env
  0600 and fall through to direct. Secrets + token shredded on success.
- build-felhom-iso.sh --pairing: generic secret-free ISO, -generic filename,
  manifest mode=pairing. profiles/generic.profile (new).
- test/bootstrap-modes.sh: Scenario D (direct = zero appliance calls) + pairing
  register/poll + delivery handoff — all green in a debian container.
2026-07-17 15:07:31 +02:00
admin 4009401f46 hub v0.61.0 + felhom-tenantsync v1.1.0: Customer RESET (middle lifecycle tier)
One operator action returns a customer to pre-first-install: all operational
state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret,
claim state, retained escrow custody); identity + basic config + provenance +
events survive. Sits between host delete and customer Delete.

- store/customer_reset.go: customer_resets journal, live inventory, ack-gated
  purge (never touches identity/provenance/events), DeleteClaim.
- claim.ResetToUnclaimed: delete claim row -> fresh code next onboarding.
- offsite.Deprovision (idempotent) + OffsiteIdentifier + ClearProvisionedDescriptor.
- tenantsync.Deprovision + felhom-tenantsync.sh deprovision op (destroys ns +
  backup groups + token; shared user untouched; idempotent).
- web/customer_reset.go: GET reset -> inventory JSON; POST -> orchestration
  (external teardown FIRST, DB purge LAST; refuse-while-hosts; typed-id +
  separate escrow ack). Amber RESET card distinct from red Danger-zone Delete.
- Red-proofs: ack-gate + partial-failure resumability (both proven red);
  store ack-gating + journal round-trip; offsite idempotency + descriptor clear;
  RESET-card render. Green: build + vet + test.
2026-07-17 13:09:04 +02:00
admin 2752e12acc hub v0.60.1: host deletion demotes escrow custody (never destroys) + customer-delete purge point + S6b obsolete
- DeleteHost(deleteEscrow) demotes current host_escrow into host_escrow_superseded (copy-before-delete, same tx), spares existing; one shared demoteCurrentEscrowTx (reused by SaveHostEscrow). F-14 provenance/gate unchanged.
- DeleteCustomerConfig now purges both escrow tables for all the customer's hosts incl. already-deleted (F-14 provenance UNION) — the one true purge point.
- Wording: checkbox/refusal/Danger-zone → demotion. S6b OBSOLETE. Red-proofs TestDeleteHost_Demotes + TestDeleteCustomer_Purges + wording guard.
2026-07-17 11:25:38 +02:00
admin e247dbc1be hub v0.60.0: offsite continuity Part B — superseded-escrow retention (data-first)
- host_escrow_superseded table + SaveHostEscrow retains a different-sha old blob before overwrite (tx); same-sha idempotent (no supersede row); returns superseded bool. ACK/restore read the current row unchanged. CountSuperseded/ListSuperseded; DeleteHost drops retained rows.
- escrow_superseded audit event + operator retained-count on host detail; register offbox_repo_orphaned/reset. Red-proof TestSaveHostEscrow_RetainsSuperseded.
2026-07-17 10:47:50 +02:00
admin eb227486d0 hub v0.59.0: Direction-2a agent-plane immediate-sync poke sender + ep0 felhom-poke surface
- internal/poke: pinned-host-key SSH poke sender (wgsync sibling) + fire-and-forget Notifier (PokeHost/PokeAllHosts). Poke refuses non-WG targets pre-dial; contentless via ep0 forced command to the box WG /32:51822.
- wiring: Server.SetPoke; applyPBSDR pokes the host after each descriptor gen-bump; handleSetArtifacts (MinAgent floor) pokes all hosts. main.go env POKE_SSH_KEY_FILE (reuses peersync endpoint/hostkey).
- scripts/felhom-poke.sh (non-root forced command) + offsite-endpoint.md §11; manifests/hub.yaml Secret/agent-poke + POKE_SSH_KEY_FILE (image tag bump follows the build).
2026-07-16 22:48:15 +02:00
admin 60244727ad feat(hub): Direction-2 immediate-sync wait channel (v0.58.0)
GET /api/v1/wait long-poll: the box holds an authed hanging GET; the hub
completes it the instant any operator intent bumps that customer's in-memory
generation, then the box fires its ordinary report and the ACK delivers
everything through the unchanged machinery. 240s hold with a 25s heartbeat
newline defeats the nginx 60s proxy_read_timeout with no ingress annotation;
WriteTimeout lifted per-connection via ResponseController.

- internal/intent: per-customer generation counter + waiter registry
  (Bump/Wait/Close), coalescing to latest, race-closer, in-memory by design.
  Red-proofs: counter-vs-queue + race-closer (run-fail-reverted).
- api/wait.go: the endpoint (per-customer only; global key 400; A cannot see B).
- web bumps after every intent write (fire-after-commit): config CRUD, claim
  resend, offsite re-issue/freeze, password regen, block/unblock, floors
  (global bumps all config-managed), controller log-tail + log-bundle.
- main.go: one intent hub shared by web+api; Close() before server.Shutdown.

Pairs with controller v0.140.0 (the long-poll client). Grounding:
documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md.
2026-07-16 20:44:22 +02:00
admin 7747a16ff1 feat(hub): v0.57.0 reinstall-of-existing-customer arc — claim/offsite/escrow continuity
F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path,
single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on
re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty
(red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the
mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events:
claim_reissued_reenroll / offsite_reissued / escrow_stale.

Controller + scripts unchanged (source contradicted both premises): the controller reads escrow
prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4
root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc
F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass.
2026-07-16 18:00:13 +02:00
admin 6218e7919d hub v0.56.0: PBS-DR self-heal reconciler (re-stage a consumable secret)
Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.

New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
2026-07-15 18:27:39 +02:00
admin 08fef4872b hub v0.55.0: accept offbox_enlarge_blocked event (Task 3a-fix delivery chain)
allowedEventTypes gains offbox_enlarge_blocked (was 400 at ingestion, dropping the customer email).
Deliberate NON-change: no customerMessages entry — the static map would discard the controller's
dynamic two-number Hungarian message (templates.go:129 fallback is correct). event_test acceptance +
400 red-proof; templates_offbox_test locks the raw-message fallback. manifest bumped to :0.55.0.
2026-07-15 07:50:29 +02:00
admin a1d045079f @
hub v0.54.0: change operator login password from the Configuration UI

Adds a "Login password" card on /configuration. The password was previously
settable only via the hub-config ConfigMap (auth.password_hash) + redeploy.

- store: hub_settings key operator_password_hash + Get/SetOperatorPasswordHash
- server: passwordHash field -> configPasswordHash (seed); new
  effectivePasswordHash() (DB override wins, else seed) is now the single
  source for the CSRF gate, RequireAuth, and handleLogin
- POST /configuration/password (handleChangePassword): requires current
  password, 8-72 byte new + confirm, bcrypt cost 10, persists DB override;
  existing sessions kept valid; ConfigMap stays the break-glass reset path
- UI: current/new/confirm form + inline mismatch pre-check + 6 flashes
- tests + red-proofs: override precedence, happy-path via handleLogin,
  wrong-current rejection, mismatch/too-short/no-op, template render
- docs: CHANGELOG, README (auth+config), REUSE, REPORT

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
2026-07-13 22:46:49 +02:00
admin 375cb082e2 host-install v1.16.0 (FELHOM_ESCROW via canonical sudoers fetch; hub hostInstallVersion synced, no deploy) + RUNBOOK-escrow-ceremony rewrite: controller wizard PRIMARY, CLI operator fallback, F1 threat-model paragraph (R over CF tunnel accepted 2026-07-13), stale-blob/supersede/void semantics 2026-07-13 19:01:46 +02:00
admin 36c72138f1 hub: dead-host roll-up honesty - customer status folds worst expected host
Customer status (dashboard row, /configs list, detail header + strip) is
now worst(controllerDerived, hostStatusOf(each expected host)) via the ONE
staleness definition (Server.hostStatus, hosts.go - shared with the
HostStalenessChecker; no second threshold). Any host down/stale caps the
customer at WARN with a cause chip naming the host ("host down: <id>");
pending (never-reported) hosts worsen only once the customer has reported
(onboarding exclusion). The three previously-inlined controller-status
chains collapse into controllerStatus() (rollup.go). Display + derivation
only - checker alerting untouched.

Live shape pinned (drill-1 / Peti cluster): host down 23h + controller
report minutes old rendered a GREEN row - TestRollup_DeadHostMasking now
fails that exact outcome. Red-proof: short-circuiting foldHostStatus to
controller-only flips C + two D subtests red ("dashboard row is GREEN
over a 23h-dead host").
2026-07-13 14:52:04 +02:00
admin 04861a7ed3 hub: F-14 part 2 — gated auto-Reissue on the PBS-DR enable dead-end
pbsdrProvisionAtom, on tenantsync token_exists: consult the customer's
MOST RECENT host-deletion record. escrow_acked=true -> invoke the
EXISTING tenantsync Reissue op, store the audit event (pbsdr_auto_reissue,
hub-source, "Previous key destroyed (acknowledged deletion) - credentials
re-issued automatically."), proceed to secret+descriptor as a normal
provision. No record / un-acked -> the pre-existing refusal, byte-unchanged
(never-silently-re-key law; operator ruling 2026-07-13).
Scenario A test drives the REAL handler path over a REAL escrow-ack delete;
scenario B pins the exact non-effect (zero Reissue calls, no state).
Red-proof: bypassing the gate flips both B subtests to 303 (silent re-key).
2026-07-13 14:44:42 +02:00
admin 2321077800 hub: F-14 part 1 — host-deletion provenance (host_deletions, in-tx)
Every DeleteHost writes a provenance row INSIDE the cascade tx: host_id,
customer_id, deleted_at, escrow_acked. escrow_acked = ack given over a
PRESENT escrow row (acknowledged destruction, not a vacuous checkbox).
LatestHostDeletion(customer) serves the F-14 gate — newest record only,
so an old acked deletion never whitelists a newer un-acked one. No
backfill by design: pre-record deletions stay on the manual path.
Red-proof: dropping the in-tx INSERT fails TestDeleteHost_ProvenanceRecord
("no deletion record written") + the part-2 scenario-A test.
2026-07-13 14:44:42 +02:00
admin 5d66c62bd6 hub: F-16 — inline two-step confirm partial replaces every native confirm() (offsite/PBS re-issue, freeze, telemetry reset, dismiss-all, regen-password, claim-resend, block/delete, geo-disable); hub_confirm_gate.py enforces zero
Claude-Session: https://claude.ai/code/session_01GzammAMzsJTgpQHqxwM2bC
2026-07-13 08:19:54 +02:00
admin bad9203daa hub: F-15 instant reset codes — reset-request response carries the rotated {code_hash, generation, issued_at} (same shape/guarantee as the report ACK); claim-resent flash now states the operator-resend ACK lag
Claude-Session: https://claude.ai/code/session_01GzammAMzsJTgpQHqxwM2bC
2026-07-13 08:04:53 +02:00
admin 448a68237a hub v0.51.0: DR-tier-by-default — per-customer dr_tier flag (default ON, legacy backfill from reality), cascade stages, WG-registration auto-provision hook, offsite-requires-DR guard (F-6 policy), host-page capability chips (inactive=neutral)
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 20:37:00 +02:00
admin 007946faf4 installer v1.15.0: DR-tier-by-default sweep (F-7 pbs-apply, F-10 age, F-9 wg default-on, F-1 single version + hub const, F-2 dry-run auth, F-8 rotation pointer, ACL felhom-pbs kept) + hostinstall_gates.py
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 19:52:56 +02:00
admin e205a2d6a1 test: claim engine — reset requests never alter claimed state (Scenario C non-DoS; leftover from the claim-arc session)
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 19:38:42 +02:00
admin 3ded6cf7c1 hub v0.50.0: customer-claim arc docs (CHANGELOG, default template claim_code_* fields, REUSE rows)
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 18:45:43 +02:00
admin 6b40eb8619 hub: customer-claim password arc parts 1+2 — code engine, emails, ACK, configgen bake, UI (v0.50.0)
Closes DRILL-day0-vm F-4 hub-side: per-customer claim state (customer_claims,
bcrypt-only custody), the claim engine (issue at real config retrieve = Day-0
bake; first-report issue for live boxes; resend rotates generation; reset
rate-limited 3/day), three Hungarian emails via the dispatcher, report-ACK
claim object {code_hash, generation, issued_at} + set-only claimed ingest,
web.claim_code_* baked into generated controller.yaml, Setup-tab status chip
+ resend button, POST /api/v1/claim/reset-request (self-scoped), claim_lockout
event allowlisted. 13 new tests; full repo green.

Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 18:12:48 +02:00
admin 1d94b1a9e9 hub: scoped auto-refresh + style.css cache-bust + staging rule (v0.48.0 parts 3-4)
- Auto-refresh: the 60s reload fires only while a live tab is active
  (nav data-live-tabs="overview,applications,events,host") AND no form is
  dirty (delegated input/change listener; never reset — a reload clears it).
  Skipped ticks reschedule; a muted (paused) hint shows next to the toggle on
  non-live tabs / dirty forms. Toggle, localStorage key, 60s cadence, and
  default-on behavior unchanged. The refresh script resolves the legacy
  settings→edit hash alias like the tabs script.
- Rider 4a: every template's stylesheet link is /style.css?v={{hubVersion}}
  (the v0.47.0 gotcha: max-age=3600 served stale styling for up to an hour
  after a deploy). Red-proof run: a reverted bare link fails the test.
- Rider 4b: CLAUDE.md standing rule — never git add -A in this repo
  (the 146d165 sweep incident); explicit paths + pull-rebase + one writing
  session per clone.
- Tests: Group C structural pins (attribute read, dirty listeners, alias x2,
  hint element, cadence/key survivors) + Group D cache-bust over six pages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZc5w5jDhFLv6qDC32KN5v
2026-07-12 17:37:04 +02:00
admin 2e03de1e0c hub: merge the customer edit page into the Edit tab (v0.48.0 edit-a, part 2)
- Settings tab renamed Edit; embeds config_form_body (.ConfigForm via the
  builder) + Controller Update + Geo + a new Danger zone card holding the
  relocated Block/Unblock/Delete forms (endpoints + confirm() unchanged).
  All cards are SIBLINGS after </form> — never nested in the config form.
- Customer Info header loses the Edit link and Block/Delete forms; only the
  config-less Create Config action stays.
- GET /configs/{id}/edit is a 302 to /customers/{id}#tab=edit; tabs JS gains
  the settings→edit legacy-hash alias.
- Post-action redirects land back on their tab: update/block/unblock/
  offsite-reissue/offsite-freeze/pbsdr-reissue → #tab=edit, regen-password
  → #tab=setup; delete unchanged (/configs).
- handleConfigUpdate gains the server-side twin of the form's required
  fields; the error path re-renders the STANDALONE page with the SUBMITTED
  overrides (B3 red-proof: nil overrides → typed values reset → test FAILS;
  header red-proof: restored header buttons → count=2 → test FAILS; both run).
- Tests: Group A (panel surface, sibling forms, header cleaned by COUNT),
  Group B (B1 302, B2 create unchanged, B3 typed-values, B4/B5 anchor table).
  Amended pins: customer_tabs_test settings→edit; pbsdr_test postUpdate now
  supplies the required fields + FormRendersState asserts the embedded render.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZc5w5jDhFLv6qDC32KN5v
2026-07-12 17:33:07 +02:00
admin e74014730a hub: extract config_form_body sub-template + configFormData builder (v0.48.0 part 1)
Behavior-neutral extraction (the host_detail_body pattern): the config form's
<form> + in-flight script move to a {{define}} sub-template; config_form.html
keeps the chrome. renderConfigForm/handleConfigNewForm/handleConfigEditForm
now go through the one configFormData builder (nil overrides = parse stored
ConfigJSON). Prepares the customer page Edit tab embed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZc5w5jDhFLv6qDC32KN5v
2026-07-12 17:24:59 +02:00
admin d7b3c82144 hub: accept app_start_failed event (controller fix-3, CAMPAIGN-3) — allowlist + customer message
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 10:09:54 +02:00
admin 0daddcd1c4 hub: offsite multi-endpoint management UI (v0.47.0 part 5)
- /offsite lists ALL wg_endpoints rows as cards (id, address, pubkey, subnet,
  PBS addr, peers-in-subnet count) + add/edit/delete forms
- store: ListWGEndpoints (id order) + DeleteWGEndpoint (plain delete; the
  peers-in-subnet guard lives in the handler where the refusal is built);
  SetWGEndpoint upsert reused, single-expected comment updated
- guards: subnet edit refused 409 while peers sit in the current subnet;
  endpoint delete refused 409 while peers sit in its subnet; full form
  validation (CIDR, pbs ip in subnet, port 1-65535, pubkey, id charset) -> 400
- peer table gains an Endpoint column (first id-ordered subnet match; em dash
  when none); pubkey-change edit gets a type-to-confirm noting pull-based
  convergence
- allocation/reconciler/desired-state STAY lowest-endpoint-id (page notes the
  deferral); GetWGEndpoint semantics untouched
- tests: E1-E6 incl. guard red-proofs; store list/delete test

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vvz1NCu22p8dGkRCpeX9re
2026-07-11 21:35:24 +02:00
admin 068427a729 hub: restore the host-delete escrow-ack gate (red-proof mutation swept by 146d165)
The parallel session's git add -A swept the v0.47.0 Part-4 WIP mid-red-proof:
hosts.go landed with the RED-PROOF 2 mutation (deleteEscrow := true), i.e. the
escrow acknowledgement bypassed. This restores the real gate; the swept store/
template/test files were captured in their correct final state (full go test
green). Red-proofs 1/2/5 all ran and FAILED as required before this point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vvz1NCu22p8dGkRCpeX9re
2026-07-11 21:28:04 +02:00
admin 146d165c26 docs: Q1c GREEN — reboot survival automatic since agent 0.84.0 (feature doc + audit §7 + CONTEXT + REPORT)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 21:25:14 +02:00
admin ae950e5933 hub: shared host_detail_body sub-template + customer Host tab (v0.47.0 part 3)
- host_detail_body.html: {{define}}'d body sections extracted from
  host_detail.html; the standalone page is now chrome + the sub-template
- hosts.go: hostDetailData(host, r) view-model builder extracted from
  handleHostDetail (reused by both surfaces)
- store: ListHostsByCustomer (host_id order; the Host tab is a list by
  design - N hosts for a future HA cluster)
- customer Host tab renders one host_detail_body per host + cross-link;
  empty state when no host is enrolled
- tests: TestTemplates_CustomerHostTab(+_Empty), TestListHostsByCustomer

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vvz1NCu22p8dGkRCpeX9re
2026-07-11 21:17:44 +02:00
admin 9f29bf34c0 hub: fix .data-table btn contrast + customer page hash tabs (v0.47.0 part 1-2)
- style.css: .data-table td a -> :not(.btn) so <a class=btn> keeps the .btn
  palette (View/Download buttons were blue-on-blue invisible)
- customer_unified.html: 8 client-side hash tabs (#tab=...) + sticky summary
  strip; all sections preserved in DOM, hiding is a JS-added body class only
  (no-JS = everything visible); Events tab gets a red error-count badge
- new render tests: TestTemplates_CustomerTabs (+_NoReports)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vvz1NCu22p8dGkRCpeX9re
2026-07-11 21:13:23 +02:00
admin e35b1ae0e6 hub v0.46.0: observability pass — per-box log pulls, bundle custody, 72h TTL + secret gate
log_bundle_requests + log_bundles store (gzip, newest-3, 72h TTL purged on the
60s sweep); SaveLogBundle secret gate fail-closed (blocked flag row, no payload;
REDACTED/checksums pass). Report ACK gains controller_log_requested + ingests
controller_log_tail; heartbeat envelope gains log_tail_requested + ingests
log_tail (consume-once on arrival; pre-0.83 agents stay visibly pending). Host
detail Diagnostics section: request buttons (controller/agent), state rows with
honest latency hints, View/Download endpoint. Red-proofs: gate disabled and
clear-on-arrival removed both FAIL their tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 16:57:47 +02:00
admin bbecf0592e hub v0.45.0: floor-UI separation + effective-floor source + per-box MinAgent conditional floor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 15:33:14 +02:00
admin ce6a56691e hub v0.44.0: PBS DR tier SLICE 1 — felhom-tenantsync surface (script+client) + hub provisioning flow (consume-once host secret, pbs_dr desired-state descriptor, fail-closed + idempotent, re-issue)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-10 20:49:48 +02:00
admin 3e0c327f16 hub v0.43.1: Git Sync form hint — credentials optional, private catalog only (pairs with controller v0.112.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-10 17:53:52 +02:00
admin c084046af0 hub v0.43.0: remote app-log diagnostics — copyable issues + context + on-demand log tails + range/dismissal fixes
- store: app_log_issues gains context/context_customer (first capture wins) + dismissed_at (resurface only on last_seen > dismissed_at); log_tail_requests (pending operator intents, consume-once) + app_log_tails (transient, keep last 2 per app)
- api: /report ingests log_tails (stores + clears the request); ACK advertises log_tail_requests (same additive omit-when-empty pattern as escrow)
- web: Known Issues rows click-to-expand (full copyable message + context with provenance + explicit affected-customers list); Dismiss replaces Delete; period selector now filters issues (F); ?customer= filtered view + customer-page drill-down links (H); per-app Request-log-tail button + pending badge + App Log Tails section + ordered tail view with line numbers + .log download; customer-visible log_tail_requested event
- tests: store (context first-capture/late-adopt, range filter, dismissal old-window vs new-occurrence, tail request/fulfill/prune/scoping), api ACK round-trip, web render (expanded row, customer page sections, tail view + download + cross-customer 404)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-10 16:03:32 +02:00
admin 9f1759132d hub: configgen test pinning logging.level=debug propagation (v0.42.0 e2e leg)
Proves a customer ConfigJSON logging.level=debug deep-merges over the template
default into the generated controller.yaml — the integration leg the remote
debug-mode toggle depends on. Merge target (logging.level: info) verified live
on the demo controller.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-10 15:21:55 +02:00
admin c918baa656 hub v0.42.0: remote "Debug mód" toggle on the customer config editor
Adds a form-level debug-mode checkbox to the customer config editor so an
operator can flip the controller's Logging.Level=debug (/debug menu + verbose
log) remotely, without SSH. Form field (not raw-JSON injection) because
handleConfigUpdate rebuilds ConfigJSON from the form on every save; the
config-version bump makes the controller re-pull + self-restart next cycle.

- buildConfigJSON: debug_mode checked -> "logging":{"level":"debug"};
  unchecked -> logging key omitted.
- config_form.html: "Hibakeresési mód (fejlesztői)" section + render state.
- configs_debug_test.go: form->JSON both ways; full-path survival test
  (debug lands, offsite descriptor unchanged, foreign-key red-proof); render
  state; red-proof exercised.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-10 15:14:43 +02:00
admin fad5573dd3 hub v0.41.0: OffsiteChecker (fill 90/95 + staleness >48h) + operator freeze lever (SLICE 4)
Sibling checker over the controller report's offsite object: quota-fill
warn/crit + the silently-stuck staleness detector (escrowed-only,
red-proofed; nil-safe on pre-v0.109 reports; same-second tie-guard).
SetOffsiteFrozen flips ONLY readonly on the exactly-1 labelled sub-account
(SSH preserved); Freeze/Unfreeze buttons — manual only, never automatic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 23:57:12 +02:00
admin 49d1233391 hub v0.40.0: store escrow restic_pw_sha256 + serve escrow status in the report ACK (SLICE 3)
Additive host_escrow migration; SaveHostEscrow/HostEscrow gain the hash
(NULL-safe for legacy rows); GetEscrowStatusForCustomer joins hosts;
the report ACK gains escrow:{identity_blob_present,restic_pw_sha256,
created_at} (omitted without a row). Contract test mirrors the agent's
v0.79.0 emit struct.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 23:10:29 +02:00
admin 17cc67f7cd hub v0.39.0: offsite hardening — F4 credential re-issue, F2 scan retry, F5 save UX
F4: ReissueCredentials — explicit operator recovery for consumed-password
dead-ends; resets the labelled resource's password (exactly-1 guard,
red-proofed), stores a fresh one-time secret, bumps ConfigVersion.
New hetznerapi.ResetBoxPassword for the dedicated path.
F2: host-key scan retry-with-backoff (~60s ladder, red-proofed) — first
save survives fresh-subaccount DNS lag.
F5: config form disables submits + shows an in-flight notice (the re-click
bait that caused live F1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 22:39:08 +02:00
admin 7c70c545d9 hub v0.38.1: offsite provisioning must survive a client disconnect (live F1)
First supervised live run: the ~25s spinner-less offsite save invited a
re-click; the abandoned first request's r.Context() was canceled between
CreateSubaccount and SaveOneTimeSecret, stranding sub-account 268985 with a
password lost forever (consume 404s permanently).

applyOffsite now provisions on context.WithoutCancel + 3-minute absolute
timeout: once the create starts, create->wait->store runs to completion.
Regression test with a ctx-honoring fake that cancels the request context
mid-create; red-proofed against the raw-ctx pre-fix shape (reproduces the
exact live error).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 21:00:27 +02:00
admin 229650b4ee hub v0.38.0: offsite SLICE 2 — capture the box host-key fingerprint
Descriptor.HostFingerprint (SHA256, non-secret), captured at provision via an
x/crypto/ssh keyscan (SSHHostKeyScanner — dials :23, grabs the host key from the
handshake, no ssh binary). Fail-closed: nil scanner or scan failure → error (don't
serve a descriptor the controller can't verify). Pairs with controller v0.106.0
which re-scans + refuses on mismatch (no blind TOFU).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 19:14:36 +02:00
admin 44ec06b50f hub v0.37.0: offsite provisioning SLICE 1 — Cloud-API client + provisioning core
Hetzner storage-box provisioning against api.hetzner.com/v1 (NOT .cloud).
internal/hetznerapi (typed client + CloudAPI interface + Fake + WaitAction);
internal/offsite (Provisioner.ProvisionOffsite — idempotent by label, shared
sub-account/dedicated box, transient password, non-secret Descriptor,
fail-closed); one_time_secrets store (single-use Save/Consume); POST
/offsite/consume-password/{id} (customer-key auth, once); config-form Offsite
section → applyOffsite (502+no-save on error) → descriptor in ConfigJSON →
version bump. Token/passwords never logged/committed/in ConfigJSON. Tested vs a
faked Cloud API + fail-closed red-proof. NOT yet live-provisioned (needs the
dedicated-project scoped token; current token can delete ep0).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 18:38:24 +02:00
admin 844fbfa749 GL-7 Part 2: install-command generator on the customer page
Replaces the static Option-1/2 Setup Command blocks with an interactive,
client-side builder: mode (required radio), cores/memory (required for byo,
optional for appliance), vmid, node, acl-storages (quote-wrapped),
operator-pubkey-file, preserve-state-from, and --dry-run/--preflight-only/
--skip-provision/--allow-new-leaf checkboxes. genFlags()/genUpdate() assemble
a live-updating download-then-run command (never curl|bash) + a local-run
variant, enforcing the script's own rules client-side (mode required; byo
requires caps → shows a warning + no runnable command; appliance hides the
caps requirement; allow-new-leaf shows its leaf-regen warning). Emits ONLY
real host-install v1.12.0 flags; the dangerous/operator-only set
(--force/--rotate-recovery/--enable-oob/--remove-golden/--uninstall/
--adopt-pool/--rescope-acl) is never offered. Graceful static fallback:
the server-rendered Option-1/2 commands keep --customer-id + a --mode
placeholder when JS is off. No framework/CDN/network; ScriptVersion (const,
in sync with SCRIPT_VERSION) drives the header. Render/structure test covers
the control ids, version, fallback, and the excluded-flag absence.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 08:29:15 +02:00
admin b93dd03f0a GL-7 Part 1: passphrase hardening on the customer page (security)
The Setup Command panel rendered the per-customer retrieval passphrase (the
secret that fetches the WHOLE config) in cleartext twice — as #retrieval-pw
text and baked into the Option-3 curl's X-Retrieval-Password header — which
contradicts the panel's own "never on the command line" guidance. Now:
the retrieval password is MASKED by default (bullet run) with Reveal/Hide +
copy-secret controls (value lives in data-secret — the existing reveal
model); the Option-3 debug command carries a <YOUR-RETRIEVAL-PASSWORD>
placeholder, never the secret. Render test asserts the secret is not baked
into any command + is masked by default; red-proof (bake it back) FAILS.
(A zero-secret-in-DOM reveal-on-demand fetch is a noted follow-up, not this task.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 08:23:41 +02:00
admin f8fc09e5cc feat(hub): OOB access health ingest + degraded alert (H1 Part 4)
store.GetHostOOBStates parses the agent oob heartbeat stanza. monitor/host_oob:
transition-based oob_degraded/oob_recovered warning (felhom-sshd down while the
operator peer is configured, OR config invalid) — proactive "can the operator get
in right now" signal; unconfigured OOB never alerts. Wired into the 60s sweep.
Non-hollow tests + transitions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:30:03 +02:00