From 9b3381be0a2edf58e213248fe5b397f7be5f7a2f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 21 Jul 2026 20:28:06 +0200 Subject: [PATCH] =?UTF-8?q?feat(hub):=20v0.70.0=20=E2=80=94=20a=20deleted?= =?UTF-8?q?=20customer=20actually=20disappears=20(residue=20leg=20+=20ghos?= =?UTF-8?q?t=20cleanup)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted on 07-18 and was still on the Customers list AND still raising offsite_stale (10 events, latest 07-21 17:34, operator email at 19:34) — because GetCustomers() is report-derived and no lifecycle tier ever deleted a report. New leg 3 (residue), before the record purge: reports, app_telemetry, app_log_tails, log_tail_requests, customer_notifications, plus the credential-bearing appliance_registrations and selfbind_tokens. Audit (events, notification_log) and F-14 provenance still survive. Ghost customers are now deletable: 404 means "nothing here", not "no config row". With no config row the offsite descriptor is unknowable, so the Hetzner and descriptor legs record skipped_no_config rather than a bare "skipped". Two more red-proofs. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS --- REPORT.md | 66 +++++++ REUSE.md | 1 + .../architecture/00-capability-map.md | 2 +- documentation/backlog/ROADMAP.md | 2 +- .../runbooks/RUNBOOK-onboarding-draft-v3.md | 6 + hub/CHANGELOG.md | 50 ++++++ hub/internal/store/customer_delete.go | 96 ++++++++++ hub/internal/store/customer_reset.go | 4 +- hub/internal/web/customer_delete.go | 99 ++++++++--- hub/internal/web/customer_delete_test.go | 168 +++++++++++++++++- hub/internal/web/customer_reset.go | 58 ++++-- .../web/templates/customer_unified.html | 12 +- manifests/hub.yaml | 2 +- 13 files changed, 521 insertions(+), 45 deletions(-) create mode 100644 hub/internal/store/customer_delete.go diff --git a/REPORT.md b/REPORT.md index e2b9c72..08421ba 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,5 +1,7 @@ # REPORT — TASK-I: R-25b, customer DELETE becomes the guided full-teardown cascade +**Two releases: v0.69.0 (the cascade) and v0.70.0 (the residue leg + ghost cleanup, found validating v0.69.0 against the live hub).** + **Date:** 2026-07-21 · **Repo:** `felhom.eu` · **Baseline:** `f59aa97` (clean, HEAD == origin/main) **Scope:** hub only — **v0.68.1 → v0.69.0**. No agent / controller / catalog change. @@ -119,3 +121,67 @@ unit-proven; the live leg proves the happy path + external teardown only. The STOP-gated live cascade run above is still **outstanding** — the code is deployed, the scratch customer teardown has not been performed. + +--- + +# Addendum — v0.70.0: a deleted customer actually disappears + +## How it was found + +The operator reported that `demo-vm-felhom` "was deleted but is still here". It had NOT failed: +config row gone, both hosts deleted (07-16, 07-18, escrow acked), `host_escrow` and +`host_escrow_superseded` empty, the 07-18 RESET journal complete with every leg `ok`. + +The customer was still listed because **`store.GetCustomers()` derives the customer list purely from +the REPORT stream**, and no lifecycle tier has ever deleted a report. 502 report rows kept the ghost +alive. + +**This was not cosmetic.** The staleness and offsite checkers iterate the same report-derived list, +so the hub kept raising `offsite_stale` for a customer that no longer exists — **10 events, the most +recent 2026-07-21 17:34, three days after deletion, with an operator email sent at 19:34** (after the +v0.69.0 deploy). Verified by streaming the live `hub.db` out read-only and querying it. + +Two further residue rows are **credential-bearing**, not telemetry: +`appliance_registrations` (a `token_hash` with `status='delivered'`, still bound to the dead customer +— confirmed present for `demo-vm-felhom`) and `selfbind_tokens` (an unconsumed bind token would be a +working path to bind a box to a nonexistent customer). + +## What changed + +- **New leg 3, `residue`** (`store.PurgeCustomerResidue`, one transaction): `reports`, + `app_telemetry`, `app_log_tails`, `log_tail_requests`, `customer_notifications`, + `selfbind_tokens`, `appliance_registrations`. Runs BEFORE the record purge — `customer_configs` is + the identifying descriptor and goes last. The cascade is now `hosts → RESET → residue → purge`. + `events`, `notification_log`, `host_deletions`, `customer_resets` still survive. + The counter and the purge walk **one shared `residueQueries` list**, so a table cannot be + counted-but-not-purged. +- **Ghost customers are deletable.** Both the cascade and its preview used to 404 whenever the config + row was missing — meaning no operator surface could clear a customer deleted by any earlier path. + **404 now means "there is nothing here"** (no config, no host, no residue). With no config row the + offsite descriptor is unknowable, so `commitCustomerReset` records **`skipped_no_config`** for the + Hetzner and descriptor legs — never a bare `skipped`, which would read as "nothing to do". PBS is + id-keyed and idempotent, so it still runs. The dialog labels the ghost case and names the row count. + +## Tests + red-proofs (v0.70.0) + +- `TestDeleteCascade_PurgesResidueAndUnlistsCustomer` — residue zeroed, customer absent from + `GetCustomers()`, appliance registration and self-bind token gone **by name**, audit + F-14 + provenance intact, journal `residue=ok customer_delete=ok`. +- `TestDeleteCascade_GhostCustomerIsDeletable` — the exact `demo-vm-felhom` shape (hosts deleted, + config dropped, residue alive): preview 200 with `has_config:false`, cascade completes, journal + records `skipped_no_config` for hetzner + descriptor. +- `TestDeleteCascade_404WhenNothingRemains`. + +| # | Pre-fix shape restored | Failure observed | +|---|---|---| +| 6 | residue leg removed (the v0.69.0 shape) | `residue after cascade = {Reports:1 AppTelemetry:1 NotificationPrefs:1 SelfBindTokens:1 ApplianceRegistrations:1}` + "customer is STILL on the Customers list" + appliance/self-bind rows outlived their customer | +| 7 | `cfg == nil` 404 restored | `ghost preview = 404, want 200` | + +Full suite green (`go build ./... && go vet ./... && go test ./...`), hub confirm gate OK. + +## Still outstanding + +The STOP-gated live leg. `demo-vm-felhom` is now the natural subject — it is a real ghost, the +operator has authorised it, and clearing it proves the v0.70.0 path end to end on production data. +It cannot prove the **Hetzner** teardown (no config row → `skipped_no_config`), so a scratch customer +with offsite provisioned is still needed for that half. diff --git a/REUSE.md b/REUSE.md index 661f253..eecb3ad 100644 --- a/REUSE.md +++ b/REUSE.md @@ -72,6 +72,7 @@ | `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" }}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `