feat(hub): v0.70.0 — a deleted customer actually disappears (residue leg + ghost cleanup)
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted on 07-18 and was still on the Customers list AND still raising offsite_stale (10 events, latest 07-21 17:34, operator email at 19:34) — because GetCustomers() is report-derived and no lifecycle tier ever deleted a report. New leg 3 (residue), before the record purge: reports, app_telemetry, app_log_tails, log_tail_requests, customer_notifications, plus the credential-bearing appliance_registrations and selfbind_tokens. Audit (events, notification_log) and F-14 provenance still survive. Ghost customers are now deletable: 404 means "nothing here", not "no config row". With no config row the offsite descriptor is unknowable, so the Hetzner and descriptor legs record skipped_no_config rather than a bare "skipped". Two more red-proofs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
This commit is contained in:
@@ -1,5 +1,7 @@
|
||||
# REPORT — TASK-I: R-25b, customer DELETE becomes the guided full-teardown cascade
|
||||
|
||||
**Two releases: v0.69.0 (the cascade) and v0.70.0 (the residue leg + ghost cleanup, found validating v0.69.0 against the live hub).**
|
||||
|
||||
**Date:** 2026-07-21 · **Repo:** `felhom.eu` · **Baseline:** `f59aa97` (clean, HEAD == origin/main)
|
||||
**Scope:** hub only — **v0.68.1 → v0.69.0**. No agent / controller / catalog change.
|
||||
|
||||
@@ -119,3 +121,67 @@ unit-proven; the live leg proves the happy path + external teardown only.
|
||||
|
||||
The STOP-gated live cascade run above is still **outstanding** — the code is deployed, the scratch
|
||||
customer teardown has not been performed.
|
||||
|
||||
---
|
||||
|
||||
# Addendum — v0.70.0: a deleted customer actually disappears
|
||||
|
||||
## How it was found
|
||||
|
||||
The operator reported that `demo-vm-felhom` "was deleted but is still here". It had NOT failed:
|
||||
config row gone, both hosts deleted (07-16, 07-18, escrow acked), `host_escrow` and
|
||||
`host_escrow_superseded` empty, the 07-18 RESET journal complete with every leg `ok`.
|
||||
|
||||
The customer was still listed because **`store.GetCustomers()` derives the customer list purely from
|
||||
the REPORT stream**, and no lifecycle tier has ever deleted a report. 502 report rows kept the ghost
|
||||
alive.
|
||||
|
||||
**This was not cosmetic.** The staleness and offsite checkers iterate the same report-derived list,
|
||||
so the hub kept raising `offsite_stale` for a customer that no longer exists — **10 events, the most
|
||||
recent 2026-07-21 17:34, three days after deletion, with an operator email sent at 19:34** (after the
|
||||
v0.69.0 deploy). Verified by streaming the live `hub.db` out read-only and querying it.
|
||||
|
||||
Two further residue rows are **credential-bearing**, not telemetry:
|
||||
`appliance_registrations` (a `token_hash` with `status='delivered'`, still bound to the dead customer
|
||||
— confirmed present for `demo-vm-felhom`) and `selfbind_tokens` (an unconsumed bind token would be a
|
||||
working path to bind a box to a nonexistent customer).
|
||||
|
||||
## What changed
|
||||
|
||||
- **New leg 3, `residue`** (`store.PurgeCustomerResidue`, one transaction): `reports`,
|
||||
`app_telemetry`, `app_log_tails`, `log_tail_requests`, `customer_notifications`,
|
||||
`selfbind_tokens`, `appliance_registrations`. Runs BEFORE the record purge — `customer_configs` is
|
||||
the identifying descriptor and goes last. The cascade is now `hosts → RESET → residue → purge`.
|
||||
`events`, `notification_log`, `host_deletions`, `customer_resets` still survive.
|
||||
The counter and the purge walk **one shared `residueQueries` list**, so a table cannot be
|
||||
counted-but-not-purged.
|
||||
- **Ghost customers are deletable.** Both the cascade and its preview used to 404 whenever the config
|
||||
row was missing — meaning no operator surface could clear a customer deleted by any earlier path.
|
||||
**404 now means "there is nothing here"** (no config, no host, no residue). With no config row the
|
||||
offsite descriptor is unknowable, so `commitCustomerReset` records **`skipped_no_config`** for the
|
||||
Hetzner and descriptor legs — never a bare `skipped`, which would read as "nothing to do". PBS is
|
||||
id-keyed and idempotent, so it still runs. The dialog labels the ghost case and names the row count.
|
||||
|
||||
## Tests + red-proofs (v0.70.0)
|
||||
|
||||
- `TestDeleteCascade_PurgesResidueAndUnlistsCustomer` — residue zeroed, customer absent from
|
||||
`GetCustomers()`, appliance registration and self-bind token gone **by name**, audit + F-14
|
||||
provenance intact, journal `residue=ok customer_delete=ok`.
|
||||
- `TestDeleteCascade_GhostCustomerIsDeletable` — the exact `demo-vm-felhom` shape (hosts deleted,
|
||||
config dropped, residue alive): preview 200 with `has_config:false`, cascade completes, journal
|
||||
records `skipped_no_config` for hetzner + descriptor.
|
||||
- `TestDeleteCascade_404WhenNothingRemains`.
|
||||
|
||||
| # | Pre-fix shape restored | Failure observed |
|
||||
|---|---|---|
|
||||
| 6 | residue leg removed (the v0.69.0 shape) | `residue after cascade = {Reports:1 AppTelemetry:1 NotificationPrefs:1 SelfBindTokens:1 ApplianceRegistrations:1}` + "customer is STILL on the Customers list" + appliance/self-bind rows outlived their customer |
|
||||
| 7 | `cfg == nil` 404 restored | `ghost preview = 404, want 200` |
|
||||
|
||||
Full suite green (`go build ./... && go vet ./... && go test ./...`), hub confirm gate OK.
|
||||
|
||||
## Still outstanding
|
||||
|
||||
The STOP-gated live leg. `demo-vm-felhom` is now the natural subject — it is a real ghost, the
|
||||
operator has authorised it, and clearing it proves the v0.70.0 path end to end on production data.
|
||||
It cannot prove the **Hetzner** teardown (no config row → `skipped_no_config`), so a scratch customer
|
||||
with offsite provisioned is still needed for that half.
|
||||
|
||||
@@ -72,6 +72,7 @@
|
||||
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
||||
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
||||
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
||||
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
|
||||
| `(*Server).handleCustomerDelete` / `handleCustomerDeletePreview` (v0.69.0) | hub/internal/web/customer_delete.go | `(w, r, customerID)` | THE customer offboarding entry: the guided full-teardown cascade `hosts → RESET → purge` (R-25b). GET = live inventory JSON for the dialog, POST = the cascade | There is NO shallow delete path any more — the old `handleConfigDelete` is gone; do not reintroduce one. Every gate (3 acks, typed customer-id, stale host-count, ONLINE-host refusal) runs BEFORE any write, so a refusal has zero side effects. Leg order is load-bearing twice over: ruling 3 (RESET never sees a host row) and custody purged exactly once, in leg 3. A failed leg retains the journal — a re-run resumes and must pass every gate again. |
|
||||
| `(*Store).ListWGEndpoints` / `DeleteWGEndpoint` | hub/internal/store/wg.go (~L64/~L86) | `() ([]WGEndpoint, error)` / `(endpointID) error` | The /offsite endpoint-management surface | `GetWGEndpoint` (lowest id, LIMIT 1) stays THE allocation/sync endpoint — do not switch allocator/reconciler/desired-state to the list without the `wg_peers.endpoint_id` migration arc. Peers-in-subnet guards live in hub/internal/web/offsite.go. |
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
|
||||
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). |
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below |
|
||||
| **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run |
|
||||
| **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run |
|
||||
| Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 |
|
||||
|
||||
## B. Apps & catalog
|
||||
|
||||
@@ -84,7 +84,7 @@
|
||||
| R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-<date>` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
|
||||
| R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/<token>` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Slice 1 PROVEN-LIVE 2026-07-18:** in the rehearsal the operator minted + emailed the link at 16:28:55 and **the customer bound their own box at 16:29:55 with zero failed attempts**, `appliance_bound` carrying source `customer_selfbind`, credential delivered 26 s later with no operator action (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). The new-ISO console banner rendered on the real box. **Slice 2 = console-passphrase bind → R-27c** (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) |
|
||||
| R-27b | **Customer self-bind, second-box flow (controller side).** For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub `/bind/` page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). | M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same `/bind/` public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance |
|
||||
| R-25b | **RULED: customer DELETE becomes a guided full-teardown cascade.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | **SHIPPED hub v0.69.0 (2026-07-21)** | **operator ruling 2026-07-21**: **DELETE subsumes the whole cascade, behind explicit consent.** Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a **typed customer-name confirmation** before the button arms. Internal order is **host-delete → RESET → delete**, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). **Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox.** Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. **It no longer blocks R-3** — the model is decided, so the friend-alpha runbook can be written against it. **IMPLEMENTED per the ruling (TASK-I, hub v0.69.0):** `POST /configs/{id}/delete` now runs `hosts → RESET → purge`; three acks + typed customer-id + a stale-preview check + the ONLINE-host refusal, all gates before any write (zero side effects on refusal); custody purged exactly ONCE in leg 3 (leg 2 runs with `purgeEscrow=false`); ruling-3 preserved BY CONSTRUCTION and asserted from inside leg 2; failed legs retain the journal and the dialog offers Resume. Standalone RESET byte-identical. 5 red-proofs. Offboarding guidance: `runbooks/RUNBOOK-onboarding-draft-v3.md` §G |
|
||||
| R-25b | **RULED: customer DELETE becomes a guided full-teardown cascade.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | **SHIPPED hub v0.69.0 (2026-07-21)** | **operator ruling 2026-07-21**: **DELETE subsumes the whole cascade, behind explicit consent.** Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a **typed customer-name confirmation** before the button arms. Internal order is **host-delete → RESET → delete**, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). **Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox.** Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. **It no longer blocks R-3** — the model is decided, so the friend-alpha runbook can be written against it. **IMPLEMENTED per the ruling (TASK-I, hub v0.69.0):** `POST /configs/{id}/delete` now runs `hosts → RESET → purge`; three acks + typed customer-id + a stale-preview check + the ONLINE-host refusal, all gates before any write (zero side effects on refusal); custody purged exactly ONCE in leg 3 (leg 2 runs with `purgeEscrow=false`); ruling-3 preserved BY CONSTRUCTION and asserted from inside leg 2; failed legs retain the journal and the dialog offers Resume. Standalone RESET byte-identical. 5 red-proofs. Offboarding guidance: `runbooks/RUNBOOK-onboarding-draft-v3.md` §G. **v0.70.0 follow-up (same day, found validating against the live hub):** a completed delete still left the customer on the Customers list and still ALERTING, because `GetCustomers()` is report-derived and no tier ever deleted a report — new **residue** leg (reports/telemetry/log-tails/notif-prefs + the credential-bearing `appliance_registrations`/`selfbind_tokens`), and **ghost customers are now deletable** (404 = nothing here, not no-config-row) |
|
||||
| R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close |
|
||||
| R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). **COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18):** customer zero shrank the guest **11675 → 8192 MB (16:50:22)** and grew it **8192 → 12288 MB (17:02:17)**, each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. `tests/VALIDATION-n100-rehearsal-2026-07-18.md` ledger 9. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED + PROVEN-LIVE** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". |
|
||||
| R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster |
|
||||
|
||||
@@ -195,3 +195,9 @@ Notes for the operator:
|
||||
the error names the leg, and re-opening the dialog offers **Resume**. Completed legs are no-ops.
|
||||
- The **audit event stream survives** the delete — the customer_deleted event and the host-deletion
|
||||
provenance outlive the record, by design.
|
||||
- **A deleted customer disappears from the Customers list immediately** (hub v0.70.0). If one lingers
|
||||
with a `MANUAL` config badge, it is a **ghost** left by a pre-v0.70.0 delete: the record is gone but
|
||||
its report stream survives, which both keeps it on the list and keeps the staleness/offsite checkers
|
||||
emailing you about it. Open **Delete customer…** on it and confirm — the dialog labels it as a ghost
|
||||
and the cascade clears the residue. (With no config row the Hetzner leg cannot be determined and is
|
||||
recorded as `skipped_no_config` — verify the Hetzner side by hand for that customer.)
|
||||
|
||||
@@ -1,5 +1,55 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.70.0 — a deleted customer actually disappears (the ghost + its alerts) (2026-07-21)
|
||||
|
||||
Found while validating v0.69.0 against the live hub, on the operator's report that `demo-vm-felhom`
|
||||
"was deleted but is still here". The delete HAD worked — config row gone, both hosts deleted, escrow
|
||||
tables empty, the RESET journal complete. The customer was still on the list because **`GetCustomers()`
|
||||
builds the Customers list purely from the REPORT stream** (`SELECT ... FROM reports GROUP BY
|
||||
customer_id`), and no lifecycle tier — host delete, RESET or DELETE — has ever deleted a report.
|
||||
|
||||
Not cosmetic: the **staleness and offsite checkers iterate that same report-derived list**, so the
|
||||
hub kept raising `offsite_stale` and kept **emailing the operator about a customer that no longer
|
||||
exists** — 10 events for `demo-vm-felhom`, the last one 3 days after its deletion.
|
||||
|
||||
### Leg 3: residue (new)
|
||||
|
||||
The cascade is now `hosts → RESET → residue → purge`. The residue leg deletes, in one transaction:
|
||||
`reports`, `app_telemetry`, `app_log_tails`, `log_tail_requests`, `customer_notifications` — plus two
|
||||
rows that are not telemetry at all but **credential-bearing**, and outliving their customer is a
|
||||
security defect rather than noise:
|
||||
|
||||
- `appliance_registrations` — a `token_hash` + `status='delivered'` row binding a box to the customer
|
||||
id. Deleting it returns a still-living box to the unclaimed pool on its next registration, which is
|
||||
the correct state for a decommissioned appliance.
|
||||
- `selfbind_tokens` — an unconsumed 7-day bind token is a working path to bind a box to a customer
|
||||
that does not exist.
|
||||
|
||||
It runs BEFORE the record purge on purpose: `customer_configs` is the identifying descriptor and goes
|
||||
last. `events`, `notification_log`, `host_deletions` and `customer_resets` still SURVIVE — the audit
|
||||
trail outlives every lifecycle tier, and that rule is not relaxed here. The counter and the purge walk
|
||||
**one shared table list**, so a table can never be counted-but-not-purged.
|
||||
|
||||
### Ghost customers are deletable
|
||||
|
||||
`handleCustomerDelete` / `handleCustomerDeletePreview` used to 404 whenever the config row was
|
||||
missing — so a customer deleted by any pre-v0.70.0 path could not be cleaned up by ANY operator
|
||||
surface. **404 now means "there is nothing here"** (no config, no host, no residue), not "there is no
|
||||
config row". With no config row the offsite descriptor is unknowable, so `commitCustomerReset` skips
|
||||
the Hetzner and descriptor legs and records **`skipped_no_config`** in the journal — never a bare
|
||||
`skipped`, which would read as "there was nothing to do". PBS is customer-id-keyed and idempotent, so
|
||||
it still runs. The dialog labels the case explicitly as a ghost and names the row count.
|
||||
|
||||
### Tests
|
||||
|
||||
`TestDeleteCascade_PurgesResidueAndUnlistsCustomer` (residue zeroed, customer gone from
|
||||
`GetCustomers()`, appliance + self-bind rows gone BY NAME, audit/provenance intact, journal legs
|
||||
`residue=ok customer_delete=ok`), `TestDeleteCascade_GhostCustomerIsDeletable` (the exact
|
||||
`demo-vm-felhom` shape: preview 200 with `has_config:false`, cascade completes, journal records
|
||||
`skipped_no_config`), `TestDeleteCascade_404WhenNothingRemains`. **Two more red-proofs**: dropping the
|
||||
residue leg leaves 5 residue rows and the customer still listed; restoring the `cfg == nil` 404 makes
|
||||
the ghost preview 404 again. Full suite green.
|
||||
|
||||
## v0.69.0 — customer DELETE becomes the guided full-teardown cascade (R-25b) (2026-07-21)
|
||||
|
||||
Implements the operator ruling of 2026-07-21. The customer page carried two half-truths: **RESET**
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
package store
|
||||
|
||||
import "fmt"
|
||||
|
||||
// Customer DELETE residue (v0.70.0, R-25b follow-up).
|
||||
//
|
||||
// The v0.69.0 cascade tore down hosts, external state and the customer record — and a fully deleted
|
||||
// customer STILL came back on the Customers list, because `GetCustomers()` derives the list purely
|
||||
// from the REPORT stream (`SELECT ... FROM reports ... GROUP BY customer_id`). Nothing in any
|
||||
// lifecycle tier ever deleted a report. Worse than cosmetic: the staleness and offsite checkers
|
||||
// iterate the same report-derived list, so a deleted customer kept raising `offsite_stale` and kept
|
||||
// emailing the operator — observed live on `demo-vm-felhom`, deleted 2026-07-18, still alerting
|
||||
// 2026-07-21.
|
||||
//
|
||||
// Two of these tables are not telemetry at all but CREDENTIAL-BEARING, and outliving their customer
|
||||
// is a security defect rather than noise:
|
||||
//
|
||||
// - `appliance_registrations` — a `token_hash` + `status='delivered'` row binding a box to the
|
||||
// customer id. Deleting the row returns a still-living box to the unclaimed pool on its next
|
||||
// registration, which is exactly the right state for a decommissioned appliance.
|
||||
// - `selfbind_tokens` — an unconsumed 7-day bind token would remain a working path to bind a box
|
||||
// to a customer that no longer exists.
|
||||
//
|
||||
// What deliberately SURVIVES (unchanged from every other tier): `events` (the audit stream),
|
||||
// `notification_log` (the send-attempt audit), `host_deletions` and `customer_resets` (F-14
|
||||
// provenance). The audit trail outlives every lifecycle tier — that rule is not relaxed here.
|
||||
|
||||
// CustomerResidue counts what a customer has left behind OUTSIDE the config row: the report-derived
|
||||
// state that keeps a deleted customer visible and alerting, plus the credential-bearing bindings.
|
||||
// Counts only — never a token, hash or payload.
|
||||
type CustomerResidue struct {
|
||||
Reports int
|
||||
AppTelemetry int
|
||||
AppLogTails int
|
||||
LogTailRequests int
|
||||
NotificationPrefs int
|
||||
SelfBindTokens int
|
||||
ApplianceRegistrations int
|
||||
}
|
||||
|
||||
// Total is the single number the "is there anything left at all?" decision hangs on.
|
||||
func (r *CustomerResidue) Total() int {
|
||||
if r == nil {
|
||||
return 0
|
||||
}
|
||||
return r.Reports + r.AppTelemetry + r.AppLogTails + r.LogTailRequests +
|
||||
r.NotificationPrefs + r.SelfBindTokens + r.ApplianceRegistrations
|
||||
}
|
||||
|
||||
// residueQueries is the ONE list both the counter and the purge walk, so a table can never be
|
||||
// counted-but-not-purged (or purged-but-not-counted) — the two drift silently otherwise.
|
||||
var residueQueries = []struct {
|
||||
table string
|
||||
dst func(*CustomerResidue) *int
|
||||
}{
|
||||
{"reports", func(r *CustomerResidue) *int { return &r.Reports }},
|
||||
{"app_telemetry", func(r *CustomerResidue) *int { return &r.AppTelemetry }},
|
||||
{"app_log_tails", func(r *CustomerResidue) *int { return &r.AppLogTails }},
|
||||
{"log_tail_requests", func(r *CustomerResidue) *int { return &r.LogTailRequests }},
|
||||
{"customer_notifications", func(r *CustomerResidue) *int { return &r.NotificationPrefs }},
|
||||
{"selfbind_tokens", func(r *CustomerResidue) *int { return &r.SelfBindTokens }},
|
||||
{"appliance_registrations", func(r *CustomerResidue) *int { return &r.ApplianceRegistrations }},
|
||||
}
|
||||
|
||||
// CustomerResidue counts the customer's report-derived + binding residue. Read-only.
|
||||
func (s *Store) CustomerResidue(customerID string) (*CustomerResidue, error) {
|
||||
res := &CustomerResidue{}
|
||||
for _, q := range residueQueries {
|
||||
if err := s.db.QueryRow(`SELECT COUNT(*) FROM `+q.table+` WHERE customer_id = ?`, customerID).
|
||||
Scan(q.dst(res)); err != nil {
|
||||
return nil, fmt.Errorf("CustomerResidue %s: count %s: %w", customerID, q.table, err)
|
||||
}
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// PurgeCustomerResidue removes the report-derived state and the credential-bearing bindings in ONE
|
||||
// transaction — the leg that actually makes a deleted customer disappear from the Customers list and
|
||||
// stop alerting. Idempotent: a re-run deletes nothing extra. It NEVER touches events,
|
||||
// notification_log, host_deletions or customer_resets.
|
||||
func (s *Store) PurgeCustomerResidue(customerID string) error {
|
||||
if customerID == "" {
|
||||
return fmt.Errorf("PurgeCustomerResidue: empty customer_id")
|
||||
}
|
||||
tx, err := s.db.Begin()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
for _, q := range residueQueries {
|
||||
if _, err := tx.Exec(`DELETE FROM `+q.table+` WHERE customer_id = ?`, customerID); err != nil {
|
||||
return fmt.Errorf("PurgeCustomerResidue %s: purge %s: %w", customerID, q.table, err)
|
||||
}
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
@@ -13,16 +13,20 @@ package web
|
||||
// (ONLINE refuses; escrow is DEMOTED to retained custody, never destroyed)
|
||||
// leg 2 reset — the committed RESET sequence verbatim (Hetzner FIRST, PBS, claim, descriptor,
|
||||
// DB purge) via commitCustomerReset — with purgeEscrow=FALSE, see below
|
||||
// leg 3 purge — DeleteCustomerConfig: the customer record AND all escrow ciphertext
|
||||
// leg 3 residue — the report stream + credential-bearing bindings (v0.70.0): reports, app
|
||||
// telemetry/log tails, notification prefs, self-bind tokens, appliance
|
||||
// registrations. Without this leg a deleted customer stays on the Customers list
|
||||
// and keeps raising staleness alerts, because GetCustomers() is report-derived
|
||||
// leg 4 purge — DeleteCustomerConfig: the customer record AND all escrow ciphertext
|
||||
//
|
||||
// Nothing here is newly destructive: the cascade only SEQUENCES three operations that already exist,
|
||||
// each keeping its own safety rules. Two invariants are load-bearing:
|
||||
// Nothing here is newly destructive: the cascade only SEQUENCES operations that already exist (plus
|
||||
// the v0.70.0 residue sweep), each keeping its own safety rules. Two invariants are load-bearing:
|
||||
//
|
||||
// - Ruling 3 is preserved BY CONSTRUCTION: leg 2 can only run after leg 1, so the RESET sequence
|
||||
// never sees a host row. The standalone RESET handler's 409 gate is untouched.
|
||||
// - Custody is purged EXACTLY ONCE, in leg 3. Leg 1 demotes (host_escrow → host_escrow_superseded);
|
||||
// - Custody is purged EXACTLY ONCE, in leg 4. Leg 1 demotes (host_escrow → host_escrow_superseded);
|
||||
// leg 2 is called with purgeEscrow=false so PurgeCustomerResetDBState leaves the retained blobs
|
||||
// alone; leg 3's DeleteCustomerConfig is the one true purge point (v0.60.1).
|
||||
// alone; leg 4's DeleteCustomerConfig is the one true purge point (v0.60.1).
|
||||
//
|
||||
// A leg that fails leaves the journal row retained and the error names the leg. A re-run resumes:
|
||||
// leg 1 is a no-op once the hosts are gone, and every leg of the RESET sequence is idempotent.
|
||||
@@ -55,7 +59,8 @@ func readDeleteCascadeAcks(r *http.Request) deleteCascadeAcks {
|
||||
func (a deleteCascadeAcks) complete() bool { return a.Hosts && a.Reset && a.Purge }
|
||||
|
||||
// handleCustomerDeletePreview — GET /configs/{id}/delete. The read-only inventory the guided dialog
|
||||
// renders: the RESET inventory EXTENDED with the host list (ruling 4 applied to all three legs).
|
||||
// renders: the RESET inventory EXTENDED with the host list and the residue counts (ruling 4 applied
|
||||
// to every leg).
|
||||
// Counts, names and booleans only — never a secret, blob or key. Also surfaces an incomplete journal
|
||||
// row so the dialog can offer "Resume".
|
||||
func (s *Server) handleCustomerDeletePreview(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
@@ -65,10 +70,6 @@ func (s *Server) handleCustomerDeletePreview(w http.ResponseWriter, r *http.Requ
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if cfg == nil {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
inv, err := s.store.CustomerResetInventory(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete preview %s: inventory: %v", customerID, err)
|
||||
@@ -81,6 +82,18 @@ func (s *Server) handleCustomerDeletePreview(w http.ResponseWriter, r *http.Requ
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
residue, err := s.store.CustomerResidue(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete preview %s: residue: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
// Ghost customers (config row already gone, residue alive) MUST preview — the dialog is the only
|
||||
// surface that can clear them. 404 only when there is genuinely nothing left.
|
||||
if cfg == nil && len(hosts) == 0 && residue.Total() == 0 {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
hostRows := make([]map[string]any, 0, len(hosts))
|
||||
onlineBlocked := false
|
||||
for i := range hosts {
|
||||
@@ -94,7 +107,10 @@ func (s *Server) handleCustomerDeletePreview(w http.ResponseWriter, r *http.Requ
|
||||
"online": status == "ok",
|
||||
})
|
||||
}
|
||||
offsiteEnabled, offsiteType := offsiteChoice(cfg.ConfigJSON)
|
||||
offsiteEnabled, offsiteType := false, ""
|
||||
if cfg != nil {
|
||||
offsiteEnabled, offsiteType = offsiteChoice(cfg.ConfigJSON)
|
||||
}
|
||||
offsiteName := ""
|
||||
if offsiteEnabled && s.offsite != nil {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
|
||||
@@ -105,6 +121,10 @@ func (s *Server) handleCustomerDeletePreview(w http.ResponseWriter, r *http.Requ
|
||||
offsiteName = n
|
||||
}
|
||||
}
|
||||
customerName := ""
|
||||
if cfg != nil {
|
||||
customerName = cfg.CustomerName
|
||||
}
|
||||
// An incomplete journal row = a cascade that stopped mid-way; the dialog renders it + Resume.
|
||||
var pending map[string]any
|
||||
if cr, jerr := s.store.LatestCustomerReset(customerID); jerr != nil {
|
||||
@@ -119,7 +139,8 @@ func (s *Server) handleCustomerDeletePreview(w http.ResponseWriter, r *http.Requ
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"customer_id": customerID,
|
||||
"customer_name": cfg.CustomerName,
|
||||
"customer_name": customerName,
|
||||
"has_config": cfg != nil, // false = GHOST (config already gone, residue alive)
|
||||
"hosts": hostRows,
|
||||
"host_count": inv.HostCount,
|
||||
"online_host_present": onlineBlocked, // leg 1 refuses; decommission the agent first
|
||||
@@ -132,6 +153,16 @@ func (s *Server) handleCustomerDeletePreview(w http.ResponseWriter, r *http.Requ
|
||||
"offsite_identifier": offsiteName,
|
||||
"pbs_tenancy_configured": s.tenantsync != nil,
|
||||
"pending_journal": pending,
|
||||
"residue_total": residue.Total(),
|
||||
"residue": map[string]int{
|
||||
"reports": residue.Reports,
|
||||
"app_telemetry": residue.AppTelemetry,
|
||||
"app_log_tails": residue.AppLogTails,
|
||||
"log_tail_requests": residue.LogTailRequests,
|
||||
"notification_prefs": residue.NotificationPrefs,
|
||||
"selfbind_tokens": residue.SelfBindTokens,
|
||||
"appliance_registrations": residue.ApplianceRegistrations,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -145,16 +176,26 @@ func (s *Server) handleCustomerDelete(w http.ResponseWriter, r *http.Request, cu
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if cfg == nil {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
hosts, err := s.store.ListHostsByCustomer(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete %s: hosts: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
residue, err := s.store.CustomerResidue(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete %s: residue: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
// 404 means "there is nothing here", not "there is no config row". A GHOST customer — config
|
||||
// already deleted but the report stream (and therefore the Customers list, and therefore the
|
||||
// staleness alerts) still alive — MUST be deletable; before v0.70.0 it 404'd and no operator
|
||||
// surface could clear it.
|
||||
if cfg == nil && len(hosts) == 0 && residue.Total() == 0 {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
// ── Gates (all before any write) ────────────────────────────────────────────────────────────
|
||||
acks := readDeleteCascadeAcks(r)
|
||||
@@ -214,18 +255,36 @@ func (s *Server) handleCustomerDelete(w http.ResponseWriter, r *http.Request, cu
|
||||
_ = s.store.UpdateResetLeg(journalID, "hosts", "ok")
|
||||
|
||||
// ── Leg 2: the committed RESET sequence (external teardown FIRST, DB purge last) ─────────────
|
||||
// purgeEscrow=false — retained custody dies exactly once, in leg 3.
|
||||
if lerr := s.commitCustomerReset(ctx, cfg, journalID, false); lerr != nil {
|
||||
// purgeEscrow=false — retained custody dies exactly once, in leg 4.
|
||||
// cfg may be nil (ghost customer): the Hetzner + descriptor legs then record `skipped_no_config`.
|
||||
if lerr := s.commitCustomerReset(ctx, cfg, customerID, journalID, false); lerr != nil {
|
||||
s.logger.Printf("[ERROR] delete %s: cascade stopped at leg 2 (%s) — journal #%d retained", customerID, lerr.Leg, journalID)
|
||||
http.Error(w, "Delete incomplete at leg 2 (reset/"+lerr.Leg+"): "+lerr.Msg+" The host(s) are already deleted; re-run to resume.", lerr.Status)
|
||||
return
|
||||
}
|
||||
|
||||
// ── Leg 3: the one true purge point — customer record + ALL escrow ciphertext ────────────────
|
||||
// ── Leg 3: residue — the report stream + the credential-bearing bindings ─────────────────────
|
||||
// This is what actually makes the customer DISAPPEAR: `GetCustomers()` builds the Customers list
|
||||
// (and the staleness/offsite checkers' work list) purely from `reports`, so before v0.70.0 a
|
||||
// fully deleted customer stayed visible AND kept emailing the operator. It runs BEFORE the record
|
||||
// purge on purpose — the customer_configs row is the identifying descriptor and goes LAST, and a
|
||||
// crash between the two legs leaves a ghost the cascade can now clean up on a re-run.
|
||||
if perr := s.store.PurgeCustomerResidue(customerID); perr != nil {
|
||||
_ = s.store.UpdateResetLeg(journalID, "residue", "failed")
|
||||
s.logger.Printf("[ERROR] delete %s: residue purge FAILED (journal #%d retained; re-run to resume): %v", customerID, journalID, perr)
|
||||
http.Error(w, "Delete incomplete at leg 3 (residue): the report stream and bindings could not be purged — the customer record is NOT yet removed; re-run to resume. ("+perr.Error()+")", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(journalID, "residue", "ok")
|
||||
s.logger.Printf("[INFO] delete %s: residue purged (reports=%d app_telemetry=%d app_log_tails=%d log_tail_requests=%d notif_prefs=%d selfbind_tokens=%d appliance_registrations=%d)",
|
||||
customerID, residue.Reports, residue.AppTelemetry, residue.AppLogTails, residue.LogTailRequests,
|
||||
residue.NotificationPrefs, residue.SelfBindTokens, residue.ApplianceRegistrations)
|
||||
|
||||
// ── Leg 4: the one true purge point — customer record + ALL escrow ciphertext ────────────────
|
||||
if derr := s.store.DeleteCustomerConfig(customerID); derr != nil {
|
||||
_ = s.store.UpdateResetLeg(journalID, "customer_delete", "failed")
|
||||
s.logger.Printf("[ERROR] delete %s: final purge FAILED (journal #%d retained; re-run to resume): %v", customerID, journalID, derr)
|
||||
http.Error(w, "Delete incomplete at leg 3 (purge): the customer record could not be removed — re-run to resume. ("+derr.Error()+")", http.StatusInternalServerError)
|
||||
http.Error(w, "Delete incomplete at leg 4 (purge): the customer record could not be removed — re-run to resume. ("+derr.Error()+")", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(journalID, "customer_delete", "ok")
|
||||
|
||||
@@ -379,7 +379,7 @@ func TestCommitCustomerReset_PurgeEscrowFlagGovernsCustody(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("journal: %v", err)
|
||||
}
|
||||
if lerr := s.commitCustomerReset(context.Background(), cfg, id, false); lerr != nil {
|
||||
if lerr := s.commitCustomerReset(context.Background(), cfg, "acme", id, false); lerr != nil {
|
||||
t.Fatalf("commitCustomerReset: %v", lerr)
|
||||
}
|
||||
if n := superseded(t, st, "acme"); n != 2 {
|
||||
@@ -419,3 +419,169 @@ func TestDeleteCascadePreview_Inventory(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Residue (v0.70.0): the leg that actually makes a deleted customer DISAPPEAR ─────────────────
|
||||
//
|
||||
// The v0.69.0 cascade left the report stream behind, and `GetCustomers()` builds the Customers list
|
||||
// (and the staleness/offsite checkers' work list) purely from `reports` — so a fully deleted
|
||||
// customer stayed visible AND kept emailing the operator. Observed live on `demo-vm-felhom`:
|
||||
// deleted 2026-07-18, still raising `offsite_stale` on 2026-07-21.
|
||||
|
||||
// seedResidue adds the report-derived state + the credential-bearing bindings to a customer.
|
||||
func seedResidue(t *testing.T, st *store.Store, customerID string) {
|
||||
t.Helper()
|
||||
if err := st.SaveReport(customerID, []byte(`{"health":{"status":"ok"}}`)); err != nil {
|
||||
t.Fatalf("seed report: %v", err)
|
||||
}
|
||||
if err := st.SaveAppTelemetry(customerID, time.Now(), []store.AppTelemetryRecord{
|
||||
{AppName: "immich", DisplayName: "Immich", MemoryCurrentMB: 512},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed telemetry: %v", err)
|
||||
}
|
||||
if err := st.SaveNotificationPrefs(customerID, "t@example.com", []string{"host_down"}, 6); err != nil {
|
||||
t.Fatalf("seed notif prefs: %v", err)
|
||||
}
|
||||
if err := st.MintSelfBindToken(customerID, "tokenhash-"+customerID, time.Hour); err != nil {
|
||||
t.Fatalf("seed selfbind token: %v", err)
|
||||
}
|
||||
// A DELIVERED appliance registration bound to the customer — credential-bearing (token_hash).
|
||||
if _, _, err := st.RegisterAppliance("uuid-"+customerID, "aa:bb", "ssh-ed25519 AAAA", "{}", "apphash-"+customerID, ""); err != nil {
|
||||
t.Fatalf("seed appliance: %v", err)
|
||||
}
|
||||
app, err := st.ApplianceByToken("apphash-" + customerID)
|
||||
if err != nil || app == nil {
|
||||
t.Fatalf("seed appliance lookup: %v (row=%v)", err, app != nil)
|
||||
}
|
||||
if err := st.BindAppliance(app.ID, customerID, "appliance", ""); err != nil {
|
||||
t.Fatalf("bind appliance: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func listedInCustomers(t *testing.T, st *store.Store, customerID string) bool {
|
||||
t.Helper()
|
||||
cs, err := st.GetCustomers()
|
||||
if err != nil {
|
||||
t.Fatalf("GetCustomers: %v", err)
|
||||
}
|
||||
for _, c := range cs {
|
||||
if c.CustomerID == customerID {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// The cascade purges the residue, so the customer leaves the Customers list — and with it the
|
||||
// staleness/offsite checkers' work list. RED-PROOF: drop the residue leg and this FAILS with the
|
||||
// customer still listed and 1 report row alive.
|
||||
func TestDeleteCascade_PurgesResidueAndUnlistsCustomer(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
seedDeletable(t, st, "acme")
|
||||
seedResidue(t, st, "acme")
|
||||
s.SetTenantSync(&orderTenancy{})
|
||||
|
||||
if !listedInCustomers(t, st, "acme") {
|
||||
t.Fatal("precondition: the customer must be listed before the cascade")
|
||||
}
|
||||
res, err := st.CustomerResidue("acme")
|
||||
if err != nil || res.Total() == 0 {
|
||||
t.Fatalf("precondition: residue must exist (err=%v, total=%d)", err, res.Total())
|
||||
}
|
||||
|
||||
if rr := postDelete(t, s, "acme", cascadeForm("acme", 1)); rr.Code != http.StatusSeeOther {
|
||||
t.Fatalf("status = %d, want 303: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
|
||||
after, err := st.CustomerResidue("acme")
|
||||
if err != nil {
|
||||
t.Fatalf("residue: %v", err)
|
||||
}
|
||||
if after.Total() != 0 {
|
||||
t.Errorf("residue after cascade = %+v, want all zero", *after)
|
||||
}
|
||||
if listedInCustomers(t, st, "acme") {
|
||||
t.Error("the customer is STILL on the Customers list after a complete delete — the ghost that " +
|
||||
"kept raising offsite_stale alerts for demo-vm-felhom")
|
||||
}
|
||||
// The credential-bearing rows are gone by NAME, not just by count.
|
||||
if app, _ := st.ApplianceByToken("apphash-acme"); app != nil {
|
||||
t.Error("the appliance registration (token_hash, status=delivered) outlived its customer")
|
||||
}
|
||||
if n, _ := st.CountSelfBindTokens("acme"); n != 0 {
|
||||
t.Errorf("self-bind tokens = %d, want 0 — a live bind path to a deleted customer", n)
|
||||
}
|
||||
// Audit + provenance SURVIVE, exactly as in every other tier.
|
||||
if evs, _ := st.GetRecentEvents("acme", 10); len(evs) == 0 {
|
||||
t.Error("the audit event stream was purged — it must outlive every lifecycle tier")
|
||||
}
|
||||
if d, _ := st.LatestHostDeletion("acme"); d == nil {
|
||||
t.Error("F-14 host-deletion provenance was purged — it must survive")
|
||||
}
|
||||
cr, _ := st.LatestCustomerReset("acme")
|
||||
if cr == nil || cr.Legs["residue"] != "ok" || cr.Legs["customer_delete"] != "ok" {
|
||||
t.Errorf("journal legs = %v, want residue=ok customer_delete=ok", cr)
|
||||
}
|
||||
}
|
||||
|
||||
// A GHOST — config row already gone (a pre-v0.70.0 delete), residue alive. Before v0.70.0 this
|
||||
// 404'd and NO operator surface could clear it. This is the demo-vm-felhom shape exactly.
|
||||
func TestDeleteCascade_GhostCustomerIsDeletable(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
seedDeletable(t, st, "ghost")
|
||||
seedResidue(t, st, "ghost")
|
||||
s.SetTenantSync(&orderTenancy{})
|
||||
// Model the pre-v0.70.0 aftermath: hosts deleted, config row dropped, residue left behind.
|
||||
if err := st.DeleteHost("ghost-01", true); err != nil {
|
||||
t.Fatalf("delete host: %v", err)
|
||||
}
|
||||
if err := st.DeleteCustomerConfig("ghost"); err != nil {
|
||||
t.Fatalf("drop config: %v", err)
|
||||
}
|
||||
if cfg, _ := st.GetCustomerConfig("ghost"); cfg != nil {
|
||||
t.Fatal("precondition: the config row must be gone")
|
||||
}
|
||||
if !listedInCustomers(t, st, "ghost") {
|
||||
t.Fatal("precondition: the ghost must still be listed (that IS the defect)")
|
||||
}
|
||||
|
||||
// The preview must render it rather than 404 — it is the only surface that can clear a ghost.
|
||||
req := httptest.NewRequest("GET", "/configs/ghost/delete", nil)
|
||||
rr := httptest.NewRecorder()
|
||||
s.handleCustomerDeletePreview(rr, req, "ghost")
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("ghost preview = %d, want 200: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if !strings.Contains(rr.Body.String(), `"has_config":false`) {
|
||||
t.Errorf("preview must mark the ghost (has_config=false): %s", rr.Body.String())
|
||||
}
|
||||
|
||||
if rr := postDelete(t, s, "ghost", cascadeForm("ghost", 0)); rr.Code != http.StatusSeeOther {
|
||||
t.Fatalf("ghost cascade = %d, want 303: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if listedInCustomers(t, st, "ghost") {
|
||||
t.Error("the ghost survived its own cleanup")
|
||||
}
|
||||
if res, _ := st.CustomerResidue("ghost"); res.Total() != 0 {
|
||||
t.Errorf("ghost residue = %+v, want all zero", *res)
|
||||
}
|
||||
// With no config row the offsite descriptor is unknowable — the journal must SAY so, never
|
||||
// record a bare "skipped" that reads as "there was nothing to do".
|
||||
cr, _ := st.LatestCustomerReset("ghost")
|
||||
if cr == nil || cr.Legs["hetzner"] != "skipped_no_config" || cr.Legs["descriptor"] != "skipped_no_config" {
|
||||
t.Errorf("journal legs = %v, want hetzner/descriptor = skipped_no_config", cr)
|
||||
}
|
||||
}
|
||||
|
||||
// 404 still means "there is nothing here" — an id with no config, no host and no residue.
|
||||
func TestDeleteCascade_404WhenNothingRemains(t *testing.T) {
|
||||
s, _ := newTestServer(t)
|
||||
req := httptest.NewRequest("GET", "/configs/nobody/delete", nil)
|
||||
rr := httptest.NewRecorder()
|
||||
s.handleCustomerDeletePreview(rr, req, "nobody")
|
||||
if rr.Code != http.StatusNotFound {
|
||||
t.Errorf("preview for an empty id = %d, want 404", rr.Code)
|
||||
}
|
||||
if rr2 := postDelete(t, s, "nobody", cascadeForm("nobody", 0)); rr2.Code != http.StatusNotFound {
|
||||
t.Errorf("cascade for an empty id = %d, want 404", rr2.Code)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -137,7 +137,7 @@ func (s *Server) handleCustomerReset(w http.ResponseWriter, r *http.Request, cus
|
||||
s.logger.Printf("[INFO] customer RESET started for %s (journal #%d, escrow_ack=%t)", customerID, resetID, escrowAck)
|
||||
|
||||
// Standalone RESET purges the retained custody itself, gated by the ack it just checked.
|
||||
if lerr := s.commitCustomerReset(ctx, cfg, resetID, escrowAck); lerr != nil {
|
||||
if lerr := s.commitCustomerReset(ctx, cfg, customerID, resetID, escrowAck); lerr != nil {
|
||||
http.Error(w, lerr.Msg, lerr.Status)
|
||||
return
|
||||
}
|
||||
@@ -193,12 +193,28 @@ func (e *resetLegError) Error() string {
|
||||
//
|
||||
// Behaviour for the standalone caller is byte-identical to v0.68.1 (same order, same leg names, same
|
||||
// messages, same status codes).
|
||||
func (s *Server) commitCustomerReset(ctx context.Context, cfg *store.CustomerConfig, resetID int64, purgeEscrow bool) *resetLegError {
|
||||
customerID := cfg.CustomerID
|
||||
//
|
||||
// GHOST CUSTOMERS (v0.70.0): cfg may be nil — the DELETE cascade also runs against a customer whose
|
||||
// config row is already gone but whose residue is not (a pre-v0.70.0 delete leaves the report stream
|
||||
// behind; see store/customer_delete.go). The standalone RESET handler 404s on a nil config before it
|
||||
// ever gets here, so this path is cascade-only. With no config row the offsite DESCRIPTOR is
|
||||
// unknowable, so the Hetzner leg is skipped and SAYS SO in the journal (`skipped_no_config`) rather
|
||||
// than silently reporting "skipped"; PBS is customer-id-keyed and idempotent, so it still runs.
|
||||
func (s *Server) commitCustomerReset(ctx context.Context, cfg *store.CustomerConfig, customerID string, resetID int64, purgeEscrow bool) *resetLegError {
|
||||
if cfg != nil {
|
||||
customerID = cfg.CustomerID
|
||||
}
|
||||
|
||||
// Leg: Hetzner offsite (repo DATA destroyed). Only when the customer chose an offsite tier.
|
||||
offsiteEnabled, offsiteType := offsiteChoice(cfg.ConfigJSON)
|
||||
if offsiteEnabled && s.offsite != nil {
|
||||
offsiteEnabled, offsiteType := false, ""
|
||||
if cfg != nil {
|
||||
offsiteEnabled, offsiteType = offsiteChoice(cfg.ConfigJSON)
|
||||
}
|
||||
if cfg == nil {
|
||||
// No descriptor to read — never guess a tier, and never let the journal imply "nothing to do".
|
||||
s.logger.Printf("[WARN] %s: no config row — the offsite (Hetzner) teardown CANNOT be determined and is SKIPPED; verify the Hetzner side by hand", customerID)
|
||||
_ = s.store.UpdateResetLeg(resetID, "hetzner", "skipped_no_config")
|
||||
} else if offsiteEnabled && s.offsite != nil {
|
||||
if derr := s.offsite.Deprovision(ctx, customerID, offsiteType); derr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "hetzner", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: hetzner deprovision FAILED (journal #%d retained; re-run to resume): %v", customerID, resetID, derr)
|
||||
@@ -228,7 +244,7 @@ func (s *Server) commitCustomerReset(ctx context.Context, cfg *store.CustomerCon
|
||||
|
||||
// All external legs are ok — now the DB side (publish-last, one leg at a time so the journal
|
||||
// records where a mid-purge crash stopped). Claim → unclaimed (fresh code next onboarding).
|
||||
if s.claimEngine != nil {
|
||||
if s.claimEngine != nil && cfg != nil {
|
||||
if cerr := s.claimEngine.ResetToUnclaimed(cfg); cerr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "claim", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: claim reset failed: %v", customerID, cerr)
|
||||
@@ -244,6 +260,11 @@ func (s *Server) commitCustomerReset(ctx context.Context, cfg *store.CustomerCon
|
||||
|
||||
// Clear the provisioned offsite descriptor (keep the tier CHOICE, drop provisioned host/user/repo/
|
||||
// fingerprint) and re-save → ConfigVersion bump. Identity + basic config survive intact.
|
||||
// Ghost customers have no row to clear OR to re-save — re-saving here would RESURRECT the very
|
||||
// record the cascade is deleting, so the leg is skipped, not "made to work".
|
||||
if cfg == nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "descriptor", "skipped_no_config")
|
||||
} else {
|
||||
newConfigJSON, cerr := offsite.ClearProvisionedDescriptor(cfg.ConfigJSON)
|
||||
if cerr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "descriptor", "failed")
|
||||
@@ -257,6 +278,7 @@ func (s *Server) commitCustomerReset(ctx context.Context, cfg *store.CustomerCon
|
||||
return &resetLegError{Leg: "descriptor", Status: http.StatusInternalServerError, Err: serr, Msg: "Internal error"}
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "descriptor", "ok")
|
||||
}
|
||||
|
||||
// DB purge LAST: retained escrow (ack-gated), one-time secret, DR recipe, log bundles.
|
||||
if perr := s.store.PurgeCustomerResetDBState(customerID, purgeEscrow); perr != nil {
|
||||
|
||||
@@ -893,13 +893,23 @@
|
||||
if (d.dr_recipe_present) dies.push('DR recipe');
|
||||
if (d.one_time_secret) dies.push('one-time password');
|
||||
if (d.claim_present) dies.push('claim state');
|
||||
if (d.residue && d.residue.reports) dies.push(d.residue.reports + ' report row(s)');
|
||||
if (d.residue && d.residue.app_telemetry) dies.push(d.residue.app_telemetry + ' app-telemetry row(s)');
|
||||
if (d.residue && d.residue.appliance_registrations) dies.push('appliance registration (token)');
|
||||
if (d.residue && d.residue.selfbind_tokens) dies.push('self-bind token(s)');
|
||||
if (d.residue && d.residue.notification_prefs) dies.push('notification preferences');
|
||||
dies.push('customer record');
|
||||
var custody = d.superseded_blobs > 0
|
||||
? d.superseded_blobs + ' retained escrow blob(s) + every current host escrow'
|
||||
: 'every current host escrow';
|
||||
inv.innerHTML = '<strong>Will be destroyed:</strong> ' + dies.join(', ') +
|
||||
'. <strong>Custody:</strong> ' + custody + ' (purged in the final leg). ' +
|
||||
'<strong>Survives:</strong> the audit event stream and the deletion provenance.';
|
||||
'<strong>Survives:</strong> the audit event stream, the notification log and the deletion provenance.';
|
||||
if (d.has_config === false) {
|
||||
inv.innerHTML = '<strong style="color: var(--warn)">Ghost customer:</strong> the configuration record ' +
|
||||
'is already gone, but ' + d.residue_total + ' row(s) of report/telemetry state keep it on the ' +
|
||||
'Customers list and keep it raising staleness alerts. This clears it.<br>' + inv.innerHTML;
|
||||
}
|
||||
if (d.online_host_present) {
|
||||
inv.innerHTML += '<br><strong style="color: var(--crit)">Refused:</strong> a host is ONLINE. ' +
|
||||
'Decommission the box first — the cascade never deletes a live host.';
|
||||
|
||||
+1
-1
@@ -125,7 +125,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: hub
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.69.0
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.70.0
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
|
||||
Reference in New Issue
Block a user