felhom.eu/REPORT.md

REPORT — TASK-I: R-25b, customer DELETE becomes the guided full-teardown cascade

Two releases: v0.69.0 (the cascade) and v0.70.0 (the residue leg + ghost cleanup, found validating v0.69.0 against the live hub).

Date: 2026-07-21 · Repo: felhom.eu · Baseline: f59aa97 (clean, HEAD == origin/main) Scope: hub only — v0.68.1 → v0.69.0. No agent / controller / catalog change.

What shipped

POST /configs/{id}/delete (same route, new behaviour) is now the guided full-teardown cascade. GET on the same path returns the dialog's live inventory. The shallow handleConfigDelete is gone.

Three legs, fixed order:

1. hosts — every host row via store.DeleteHost(hostID, true); escrow DEMOTED to retained custody, never destroyed. Host-delete's own ONLINE rule is kept: an ONLINE host refuses the whole cascade, checked for every host up front so it never half-runs.
2. reset — the committed RESET sequence verbatim (Hetzner → PBS → claim → descriptor → DB purge) through the newly extracted commitCustomerReset, called with purgeEscrow=false.
3. purge — store.DeleteCustomerConfig: the customer record and all escrow ciphertext.

Two invariants are asserted, not merely commented:

• Ruling 3 by construction — leg 2 can only run after leg 1, so the RESET sequence never sees a host row. The standalone RESET handler's 409 gate is untouched.
• Custody purged exactly ONCE, in leg 3 — leg 1 demotes; leg 2 runs with purgeEscrow=false; leg 3 is the one true purge point (v0.60.1).

Gates, all before any write (a refusal has zero side effects): three acknowledgements (ack_hosts / ack_reset / ack_purge, each exactly 1), the typed customer-id, a stale-preview check (the acknowledged host count must still match live → else 409), and the ONLINE-host refusal. No force flag, no skip flag, no partial-run downgrade.

Resume: a failed leg retains the customer_resets journal row and the HTTP error names the leg. The dialog renders the incomplete journal and offers Resume; a re-run is idempotent and must pass every gate again (acknowledgements are not cached across attempts).

UI: Danger zone → Delete customer… → guided dialog (inventory panel: hosts by name + status, offsite repository identifier, PBS namespace, custody state; three consequence checkboxes; typed customer-id; one submit). Client-side checks are convenience only.

Refactor — standalone RESET behaviour unchanged

handleCustomerReset's committed half became commitCustomerReset(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError. The standalone path is byte-identical to v0.68.1: same leg order, same leg names, same operator-facing messages, same status codes. Its existing suite is untouched and green.

Files

File	Change
hub/internal/web/customer_delete.go	NEW — the cascade + preview
hub/internal/web/customer_delete_test.go	NEW — scenarios A–E
hub/internal/web/customer_reset.go	commitCustomerReset + resetLegError extracted
hub/internal/web/configs.go	handleConfigDelete removed (replaced by a do-not-reintroduce note)
hub/internal/web/server.go	route: GET → preview, POST → cascade
hub/internal/web/templates/customer_unified.html	guided dialog replaces the one-click Delete
hub/internal/web/customer_edit_tab_test.go	the delete redirect case now posts the full acks
hub/CHANGELOG.md, REUSE.md, documentation/backlog/ROADMAP.md, documentation/architecture/00-capability-map.md, documentation/runbooks/RUNBOOK-onboarding-draft-v3.md	docs

Tests + red-proofs

Green gate in hub/: go build ./... && go vet ./... && go test ./... — all green, no flakes.

New coverage (customer_delete_test.go):

• A — leg ORDER, observed from inside leg 2 via a tenancyProvisioner fake whose Deprovision snapshots store state: at that instant hosts = 0 (leg 1 done), the customer row is still present (leg 3 not started), retained custody still present (leg 1 demoted, did not purge). Plus final state, the four journal legs stamped ok, completion stamped, and the customer_deleted audit event surviving the record.
• B — 9 fail-closed gate cases (each missing ack, an ack sent as yes, id mismatch, id absent, host count moved, host count absent, ONLINE host). Each asserts the status code and that the host, the customer row, the current escrow and the retained custody are untouched, and that zero external calls fired, and that no journal row was opened.
• C — resume: an injected PBS failure → 502 naming the leg; hosts gone, customer + custody SURVIVE, journal retained with hosts=ok pbs=failed; re-run converges and completes. Plus: a resume without ack #3 is still refused.
• E — custody: commitCustomerReset(..., purgeEscrow=false) leaves the retained blobs; leg 3 purges them.
• Preview: names the real host/offsite/PBS/custody facts and leaks no secret (one-time password, API keys, escrow blobs all asserted absent).

Five red-proofs run, each failed red with the wrong value visible, then restored (git diff clean):

#	Pre-fix shape restored	Failure observed
1	ack gate disabled	status = 303, want 400 + host deleted, customer deleted, custody destroyed, PBS deprovision fired, journal row opened
2	stale-preview gate weakened to always pass	status = 303, want 409 + the same six non-effect assertions
3	ONLINE-host gate removed	status = 303, want 409 + live host deleted
4	leg order inverted (RESET leg before the host leg)	at the RESET leg the customer still had 1 host(s) — leg 1 must complete FIRST
5	cascade's RESET leg called with purgeEscrow=true	retained blobs after the RESET leg = 0, want 2

Method / limits

Unit-land only so far — the STOP-gated live leg has not been run (see below). Validation method: Go tests against a real SQLite store on t.TempDir() with fakes at the existing tenancyProvisioner seam. No browser is available on DooPlex; the dialog's markup is covered by the existing customer-page render tests (exactly one /configs/{id}/delete form on the page) — a strict click-through remains a manual operator pass.

Not covered by unit tests: the Hetzner offsite Deprovision leg (offsite.Provisioner is a concrete type, no interface seam — same as the standalone RESET suite). That leg is exactly what the live run is for.

STOP — operator-present live leg (NOT yet run)

Create a scratch customer on a throwaway domain, provision offsite only (no host — cheap), run the cascade end-to-end, then verify from outside the hub that the Hetzner repository is gone and the customer row is purged. Never run against Demo Ügyfél, Demo HP, or Peti. Failure paths are unit-proven; the live leg proves the happy path + external teardown only.

Deploy status

LIVE. Code 61dbd87 → main; image gitea.dooplex.hu/admin/felhom-hub:0.69.0 built + pushed; manifests/hub.yaml bumped in 5dcb72a; ArgoCD hard-refresh + deliberate sync → app felhom Synced / Healthy; deploy/hub rolled out on image :0.69.0; startup log [INFO] felhom-hub 0.69.0 starting … Listening on :8080, no errors.

The STOP-gated live cascade run above is still outstanding — the code is deployed, the scratch customer teardown has not been performed.

---

Addendum — v0.70.0: a deleted customer actually disappears

How it was found

The operator reported that demo-vm-felhom "was deleted but is still here". It had NOT failed: config row gone, both hosts deleted (07-16, 07-18, escrow acked), host_escrow and host_escrow_superseded empty, the 07-18 RESET journal complete with every leg ok.

The customer was still listed because store.GetCustomers() derives the customer list purely from the REPORT stream, and no lifecycle tier has ever deleted a report. 502 report rows kept the ghost alive.

This was not cosmetic. The staleness and offsite checkers iterate the same report-derived list, so the hub kept raising offsite_stale for a customer that no longer exists — 10 events, the most recent 2026-07-21 17:34, three days after deletion, with an operator email sent at 19:34 (after the v0.69.0 deploy). Verified by streaming the live hub.db out read-only and querying it.

Two further residue rows are credential-bearing, not telemetry: appliance_registrations (a token_hash with status='delivered', still bound to the dead customer — confirmed present for demo-vm-felhom) and selfbind_tokens (an unconsumed bind token would be a working path to bind a box to a nonexistent customer).

What changed

• New leg 3, residue (store.PurgeCustomerResidue, one transaction): reports, app_telemetry, app_log_tails, log_tail_requests, customer_notifications, selfbind_tokens, appliance_registrations. Runs BEFORE the record purge — customer_configs is the identifying descriptor and goes last. The cascade is now hosts → RESET → residue → purge. events, notification_log, host_deletions, customer_resets still survive. The counter and the purge walk one shared residueQueries list, so a table cannot be counted-but-not-purged.
• Ghost customers are deletable. Both the cascade and its preview used to 404 whenever the config row was missing — meaning no operator surface could clear a customer deleted by any earlier path. 404 now means "there is nothing here" (no config, no host, no residue). With no config row the offsite descriptor is unknowable, so commitCustomerReset records skipped_no_config for the Hetzner and descriptor legs — never a bare skipped, which would read as "nothing to do". PBS is id-keyed and idempotent, so it still runs. The dialog labels the ghost case and names the row count.

Tests + red-proofs (v0.70.0)

• TestDeleteCascade_PurgesResidueAndUnlistsCustomer — residue zeroed, customer absent from GetCustomers(), appliance registration and self-bind token gone by name, audit + F-14 provenance intact, journal residue=ok customer_delete=ok.
• TestDeleteCascade_GhostCustomerIsDeletable — the exact demo-vm-felhom shape (hosts deleted, config dropped, residue alive): preview 200 with has_config:false, cascade completes, journal records skipped_no_config for hetzner + descriptor.
• TestDeleteCascade_404WhenNothingRemains.

#	Pre-fix shape restored	Failure observed
6	residue leg removed (the v0.69.0 shape)	residue after cascade = {Reports:1 AppTelemetry:1 NotificationPrefs:1 SelfBindTokens:1 ApplianceRegistrations:1} + "customer is STILL on the Customers list" + appliance/self-bind rows outlived their customer
7	cfg == nil 404 restored	ghost preview = 404, want 200

Full suite green (go build ./... && go vet ./... && go test ./...), hub confirm gate OK.

Still outstanding

The STOP-gated live leg. demo-vm-felhom is now the natural subject — it is a real ghost, the operator has authorised it, and clearing it proves the v0.70.0 path end to end on production data. It cannot prove the Hetzner teardown (no config row → skipped_no_config), so a scratch customer with offsite provisioned is still needed for that half.