feat(hub): v0.69.0 — customer DELETE is the guided full-teardown cascade (R-25b)
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.
Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).
handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
This commit is contained in:
@@ -1,132 +1,116 @@
|
||||
# REPORT — TASK-H: HP node access + docs + today's findings (2026-07-21, evening)
|
||||
# REPORT — TASK-I: R-25b, customer DELETE becomes the guided full-teardown cascade
|
||||
|
||||
Baseline `59226ed`, clean tree. No product code touched; the N100 and guest 9201 were not modified.
|
||||
**Date:** 2026-07-21 · **Repo:** `felhom.eu` · **Baseline:** `f59aa97` (clean, HEAD == origin/main)
|
||||
**Scope:** hub only — **v0.68.1 → v0.69.0**. No agent / controller / catalog change.
|
||||
|
||||
| Part | Status |
|
||||
## What shipped
|
||||
|
||||
`POST /configs/{id}/delete` (same route, new behaviour) is now the guided full-teardown cascade.
|
||||
`GET` on the same path returns the dialog's live inventory. The shallow `handleConfigDelete` is
|
||||
**gone**.
|
||||
|
||||
Three legs, fixed order:
|
||||
|
||||
1. **hosts** — every host row via `store.DeleteHost(hostID, true)`; escrow **DEMOTED** to retained
|
||||
custody, never destroyed. Host-delete's own ONLINE rule is kept: an ONLINE host refuses the whole
|
||||
cascade, checked for every host up front so it never half-runs.
|
||||
2. **reset** — the committed RESET sequence verbatim (Hetzner → PBS → claim → descriptor → DB purge)
|
||||
through the newly extracted `commitCustomerReset`, called with `purgeEscrow=false`.
|
||||
3. **purge** — `store.DeleteCustomerConfig`: the customer record **and all escrow ciphertext**.
|
||||
|
||||
Two invariants are asserted, not merely commented:
|
||||
|
||||
- **Ruling 3 by construction** — leg 2 can only run after leg 1, so the RESET sequence never sees a
|
||||
host row. The standalone RESET handler's 409 gate is untouched.
|
||||
- **Custody purged exactly ONCE, in leg 3** — leg 1 demotes; leg 2 runs with `purgeEscrow=false`;
|
||||
leg 3 is the one true purge point (v0.60.1).
|
||||
|
||||
**Gates, all before any write** (a refusal has zero side effects): three acknowledgements
|
||||
(`ack_hosts` / `ack_reset` / `ack_purge`, each exactly `1`), the typed customer-id, a **stale-preview**
|
||||
check (the acknowledged host count must still match live → else 409), and the ONLINE-host refusal.
|
||||
No force flag, no skip flag, no partial-run downgrade.
|
||||
|
||||
**Resume:** a failed leg retains the `customer_resets` journal row and the HTTP error names the leg.
|
||||
The dialog renders the incomplete journal and offers **Resume**; a re-run is idempotent and must pass
|
||||
every gate again (acknowledgements are not cached across attempts).
|
||||
|
||||
**UI:** Danger zone → **Delete customer…** → guided dialog (inventory panel: hosts by name + status,
|
||||
offsite repository identifier, PBS namespace, custody state; three consequence checkboxes; typed
|
||||
customer-id; one submit). Client-side checks are convenience only.
|
||||
|
||||
## Refactor — standalone RESET behaviour unchanged
|
||||
|
||||
`handleCustomerReset`'s committed half became
|
||||
`commitCustomerReset(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError`. The standalone path
|
||||
is byte-identical to v0.68.1: same leg order, same leg names, same operator-facing messages, same
|
||||
status codes. Its existing suite is untouched and green.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| 1 — tailscale on demo-hp | **DONE**, one item open (key expiry — needs an admin-console toggle) |
|
||||
| 2 — inventory + docs | **DONE** — `documentation/operations/nodes.md` |
|
||||
| 3 — findings filed | **DONE** — **R-59, R-60, R-61** + positive evidence recorded |
|
||||
| `hub/internal/web/customer_delete.go` | NEW — the cascade + preview |
|
||||
| `hub/internal/web/customer_delete_test.go` | NEW — scenarios A–E |
|
||||
| `hub/internal/web/customer_reset.go` | `commitCustomerReset` + `resetLegError` extracted |
|
||||
| `hub/internal/web/configs.go` | `handleConfigDelete` removed (replaced by a do-not-reintroduce note) |
|
||||
| `hub/internal/web/server.go` | route: GET → preview, POST → cascade |
|
||||
| `hub/internal/web/templates/customer_unified.html` | guided dialog replaces the one-click Delete |
|
||||
| `hub/internal/web/customer_edit_tab_test.go` | the delete redirect case now posts the full acks |
|
||||
| `hub/CHANGELOG.md`, `REUSE.md`, `documentation/backlog/ROADMAP.md`, `documentation/architecture/00-capability-map.md`, `documentation/runbooks/RUNBOOK-onboarding-draft-v3.md` | docs |
|
||||
|
||||
## Part 1 — tailscale on demo-hp
|
||||
## Tests + red-proofs
|
||||
|
||||
`tailscale 1.98.9` (Debian trixie apt repo, host package — same shape as `felhom-pve`).
|
||||
Green gate in `hub/`: `go build ./... && go vet ./... && go test ./...` — **all green**, no flakes.
|
||||
|
||||
```
|
||||
demo-hp 100.76.96.79 online direct path 37.191.56.193:45127
|
||||
```
|
||||
New coverage (`customer_delete_test.go`):
|
||||
|
||||
Verified **from DooPlex over the tailnet, no jumphost**: ping ~40 ms, `ssh root@100.76.96.79` works,
|
||||
and the path is **direct**, not a DERP relay. `~/.ssh/config` gains `demo-hp` (tailnet) and
|
||||
`demo-hp-lan` (`192.168.0.87` via `ProxyJump felhom-pve`) as the away-fallback.
|
||||
- **A — leg ORDER**, observed from *inside* leg 2 via a `tenancyProvisioner` fake whose `Deprovision`
|
||||
snapshots store state: at that instant hosts = 0 (leg 1 done), the customer row is still present
|
||||
(leg 3 not started), retained custody still present (leg 1 demoted, did not purge). Plus final
|
||||
state, the four journal legs stamped `ok`, completion stamped, and the `customer_deleted` audit
|
||||
event surviving the record.
|
||||
- **B — 9 fail-closed gate cases** (each missing ack, an ack sent as `yes`, id mismatch, id absent,
|
||||
host count moved, host count absent, ONLINE host). Each asserts the status code **and** that the
|
||||
host, the customer row, the current escrow and the retained custody are untouched, **and** that
|
||||
zero external calls fired, **and** that no journal row was opened.
|
||||
- **C — resume**: an injected PBS failure → 502 naming the leg; hosts gone, customer + custody
|
||||
SURVIVE, journal retained with `hosts=ok pbs=failed`; re-run converges and completes. Plus: a
|
||||
resume without ack #3 is still refused.
|
||||
- **E — custody**: `commitCustomerReset(..., purgeEscrow=false)` leaves the retained blobs; leg 3
|
||||
purges them.
|
||||
- **Preview**: names the real host/offsite/PBS/custody facts and leaks no secret (one-time password,
|
||||
API keys, escrow blobs all asserted absent).
|
||||
|
||||
The pre-auth key was passed as `--authkey=file:<path>` and shredded immediately, so it never entered
|
||||
the box's process list.
|
||||
**Five red-proofs run, each failed red with the wrong value visible, then restored (`git diff` clean):**
|
||||
|
||||
### OPEN — key expiry is NOT disabled
|
||||
|
||||
`demo-hp` expires **2027-01-17**; `felhom-pve` and `dooplex` both have expiry disabled, so this is
|
||||
the one place it diverges from the fleet convention. **I could not do it from here:** disabling
|
||||
per-device key expiry is an admin-console toggle or an API call, and the supplied `tskey-auth-…`
|
||||
pre-auth key cannot drive the API (verified — `GET /api/v2/tailnet/-/devices` returns **401**).
|
||||
|
||||
**One click when convenient:** Tailscale admin → Machines → `demo-hp` → *Disable key expiry*. Or
|
||||
hand me a `tskey-api-…` token and I will do it. Verify with:
|
||||
`ssh demo-hp 'tailscale status --json' | python3 -c "import sys,json;print(json.load(sys.stdin)['Self'].get('KeyExpiry'))"` → should print `None`.
|
||||
|
||||
### A rule I broke, then fixed
|
||||
|
||||
The join omitted `--accept-dns=false`, and MagicDNS immediately rewrote `/etc/resolv.conf` to
|
||||
`nameserver 100.100.100.100` — exactly what `operations/tailscale.md` forbids for a host node.
|
||||
|
||||
Nothing broke *at the vacation site*: there is no pi-hole there, and both `gitea.dooplex.hu` and
|
||||
`hub.felhom.eu` resolve publicly anyway. **The damage would have surfaced silently when the box comes
|
||||
home**, where split-horizon is what makes `gitea.dooplex.hu` resolve to `192.168.0.180` locally — a
|
||||
class of failure that looks like "the network is slow" rather than "DNS is wrong".
|
||||
`tailscale set --accept-dns=false` restored `nameserver 192.168.0.1`; tailnet and agent unaffected.
|
||||
The doc now carries the evidence and the instruction to pass the flag **at join time**.
|
||||
|
||||
## Part 2 — the node inventory
|
||||
|
||||
**`documentation/operations/nodes.md`** (new). The fleet is now two hosts, both agent 0.92.1:
|
||||
`demo-felhom-8363b5` (N100) and `demo-hp-bb76ea` (HP t740). Both are at the vacation site and travel
|
||||
home ~2026-08-02.
|
||||
|
||||
**demo-hp:** HP t740 Thin Client, s/n `8CN944035T`, AMI M42 v01.10 (11/11/2020), Ryzen Embedded
|
||||
V1756B (8 threads), **30 GiB RAM**, PVE 9.2.2 as node `felhom-host`, guest 9201 `demo-hp` running,
|
||||
WireGuard `10.77.0.3/32` up.
|
||||
|
||||
| device | serial | role |
|
||||
| # | Pre-fix shape restored | Failure observed |
|
||||
|---|---|---|
|
||||
| `sda` SanDisk X600 128GB | `182195804614` | system disk — PVE + LVM + guest volumes |
|
||||
| `nvme0n1` Toshiba KXG50 1024GB | `58BS11AFT8MQ` | **PRESENT AND UNENROLLED — do not touch** |
|
||||
| 1 | ack gate disabled | `status = 303, want 400` + host deleted, customer deleted, custody destroyed, PBS deprovision fired, journal row opened |
|
||||
| 2 | stale-preview gate weakened to always pass | `status = 303, want 409` + the same six non-effect assertions |
|
||||
| 3 | ONLINE-host gate removed | `status = 303, want 409` + live host deleted |
|
||||
| 4 | leg order inverted (RESET leg before the host leg) | `at the RESET leg the customer still had 1 host(s) — leg 1 must complete FIRST` |
|
||||
| 5 | cascade's RESET leg called with `purgeEscrow=true` | `retained blobs after the RESET leg = 0, want 2` |
|
||||
|
||||
The NVMe still holds its previous **NTFS** partition, is unmounted, and is in no LVM PV and no ZFS
|
||||
pool — the exact-serial filter did its job on hardware it had never seen. It joins later through the
|
||||
**Tárhely flow, never the installer**.
|
||||
## Method / limits
|
||||
|
||||
**NIC map** (documented in both `nodes.md` and `scripts/iso/README.md`): `enp1s0f0–f3` are the
|
||||
4-port `igb` card with **no carrier and no DHCP** at this site; **`enp2s0f0`** (`r8169`, MAC
|
||||
`7c:d3:0a:77:d9:76`) is the onboard port that works and is now `vmbr0`'s bridge-port. MACs for all
|
||||
five are in the doc.
|
||||
Unit-land only so far — **the STOP-gated live leg has not been run** (see below). Validation method:
|
||||
Go tests against a real SQLite store on `t.TempDir()` with fakes at the existing `tenancyProvisioner`
|
||||
seam. No browser is available on DooPlex; the dialog's markup is covered by the existing
|
||||
customer-page render tests (exactly one `/configs/{id}/delete` form on the page) — a strict
|
||||
click-through remains a manual operator pass.
|
||||
|
||||
**Loader/firmware finding:** the box installed with the **shim** loader and **Secure Boot ENABLED**
|
||||
(`mokutil --sb-state` → `SecureBoot enabled`). That retires an assumption — SB-off was an
|
||||
N100-firmware workaround, not a Felhom requirement.
|
||||
**Not covered by unit tests:** the Hetzner offsite `Deprovision` leg (`offsite.Provisioner` is a
|
||||
concrete type, no interface seam — same as the standalone RESET suite). That leg is exactly what the
|
||||
live run is for.
|
||||
|
||||
**Access, and why it is awkward:** no operator SSH key is baked (the HP profile deliberately left
|
||||
`FELHOM_ROOT_SSH_KEY` blank), so authentication is the **G1 break-glass root password vaulted in the
|
||||
hub** (`host_recovery` row `demo-hp-bb76ea`, set 2026-07-21 16:24 UTC). I retrieved it by streaming
|
||||
`/data/hub.db` out of the hub pod, extracting one field to a 0600 file, and **shredding the copy
|
||||
immediately** — that DB holds every host's secret. Recipe is in the doc. This lockout is R-61.
|
||||
## STOP — operator-present live leg (NOT yet run)
|
||||
|
||||
**The operator-lab exception is documented prominently**, in both `nodes.md` and `tailscale.md`:
|
||||
demo-hp is *customer-shaped* but tailscale is **not** part of that shape. Real customer boxes get the
|
||||
WireGuard tunnel and the H1 OOB path and nothing else. A future product-shape audit finding tailscale
|
||||
here must not conclude the product ships it.
|
||||
Create a scratch customer on a throwaway domain, provision **offsite only** (no host — cheap), run
|
||||
the cascade end-to-end, then verify from **outside** the hub that the Hetzner repository is gone and
|
||||
the customer row is purged. **Never run against Demo Ügyfél, Demo HP, or Peti.** Failure paths are
|
||||
unit-proven; the live leg proves the happy path + external teardown only.
|
||||
|
||||
## Part 3 — findings filed
|
||||
## Deploy status
|
||||
|
||||
**TASK-G Part 3 verified as already filed** — R-58 (assisted disk-picker) exists and includes the
|
||||
abort-screen candidate-table slice. Nothing to complete.
|
||||
|
||||
New rows:
|
||||
|
||||
- **R-59 [P1] — a no-DHCP install must HARD-ABORT.** Instead it baked `192.168.100.2` as a *static*
|
||||
`vmbr0` address and completed: the install "succeeded", the box looked finished, and it could never
|
||||
call home. The worst silent onboarding failure shape there is. The philosophy already exists one
|
||||
layer over — the disk filter refuses loudly and touches nothing when it cannot identify its target;
|
||||
networking should do the same, with the candidate NIC table on screen (same grammar as R-58 slice 1).
|
||||
- **R-60 [P2] — first-boot NIC sweep self-heal.** Hub unreachable ⇒ DHCP across every carrier-bearing
|
||||
NIC before settling. Today's repair was a human moving one cable; a sweep would have healed it
|
||||
unaided. Scoped to first boot and the hub-unreachable condition only — a running box must never
|
||||
re-shuffle its own networking.
|
||||
- **R-61 [P1] — the baked root password must be knowable.** The ISO mints a throwaway hash per build
|
||||
and discards the plaintext, so nobody can reach the console of a box they just installed. Slice 1:
|
||||
emit it into the build REPORT + operator cheat-sheet alongside the sha256. **A fixed well-known
|
||||
password is explicitly rejected** (operator ruling) — a pre-pairing box sits on a stranger's LAN.
|
||||
|
||||
Positive evidence, same session:
|
||||
|
||||
- **R-21 slice C is no longer a one-board result.** The row and the capability-map ISO row now read
|
||||
**PROVEN-LIVE on TWO different boards** (N100 2026-07-18; HP t740 2026-07-21) — virgin hardware,
|
||||
one pass, self-registration → operator bind → day-0 → running guest + agent check-in. The Secure
|
||||
Boot finding is recorded there too.
|
||||
- **Fresh-box floor lift**: the new box came up on golden **0.153.0** and self-updated to the fleet
|
||||
floor **0.156.0** during day-0, unattended. Cited on the publish-train capability-map row — the
|
||||
floor mechanism works on first contact, not only on boxes with history.
|
||||
- **The t740 five-NIC trap** is recorded as a board gotcha in `scripts/iso/README.md`.
|
||||
|
||||
## Observations
|
||||
|
||||
1. **The break-glass path works, and it is also the argument for R-61.** Reaching a box whose root
|
||||
password was never known required a working hub, a working network, and operator tooling — at
|
||||
exactly the moment the reason you want the console is usually that one of those is broken.
|
||||
2. **`apt update` fails on this box** against the PVE **enterprise** repos (401, no subscription) —
|
||||
pre-existing from the install, unrelated to today. I worked around it with a list-scoped
|
||||
`apt-get update` rather than editing the box's repo config. Worth deciding whether `host-install`
|
||||
should switch fresh boxes to the no-subscription repo; left alone deliberately.
|
||||
3. **A heredoc silently ate a piped secret.** `printf … | ssh host 'bash -s' <<'EOF'` sends the
|
||||
*heredoc* as stdin, so the piped key never arrives and `read` consumes script text instead. The
|
||||
working shape is the command as an argument, with stdin free for the secret — then stage it as a
|
||||
file and use `--authkey=file:`. Worth remembering next time a credential has to cross an SSH hop.
|
||||
Code pushed to `main`; hub image `0.69.0` built and pushed; `manifests/hub.yaml` bumped and synced —
|
||||
see the verification lines below.
|
||||
|
||||
@@ -71,6 +71,8 @@
|
||||
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
||||
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
||||
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
||||
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
||||
| `(*Server).handleCustomerDelete` / `handleCustomerDeletePreview` (v0.69.0) | hub/internal/web/customer_delete.go | `(w, r, customerID)` | THE customer offboarding entry: the guided full-teardown cascade `hosts → RESET → purge` (R-25b). GET = live inventory JSON for the dialog, POST = the cascade | There is NO shallow delete path any more — the old `handleConfigDelete` is gone; do not reintroduce one. Every gate (3 acks, typed customer-id, stale host-count, ONLINE-host refusal) runs BEFORE any write, so a refusal has zero side effects. Leg order is load-bearing twice over: ruling 3 (RESET never sees a host row) and custody purged exactly once, in leg 3. A failed leg retains the journal — a re-run resumes and must pass every gate again. |
|
||||
| `(*Store).ListWGEndpoints` / `DeleteWGEndpoint` | hub/internal/store/wg.go (~L64/~L86) | `() ([]WGEndpoint, error)` / `(endpointID) error` | The /offsite endpoint-management surface | `GetWGEndpoint` (lowest id, LIMIT 1) stays THE allocation/sync endpoint — do not switch allocator/reconciler/desired-state to the list without the `wg_peers.endpoint_id` migration arc. Peers-in-subnet guards live in hub/internal/web/offsite.go. |
|
||||
|
||||
### Artifact manifest / Day-0 trust root
|
||||
|
||||
@@ -36,7 +36,8 @@
|
||||
| Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
|
||||
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). |
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **Consistency gap → R-25b:** the Danger-zone DELETE leaves host rows and doesn't run this teardown |
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below |
|
||||
| **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run |
|
||||
| Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 |
|
||||
|
||||
## B. Apps & catalog
|
||||
|
||||
@@ -84,7 +84,7 @@
|
||||
| R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-<date>` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
|
||||
| R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/<token>` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Slice 1 PROVEN-LIVE 2026-07-18:** in the rehearsal the operator minted + emailed the link at 16:28:55 and **the customer bound their own box at 16:29:55 with zero failed attempts**, `appliance_bound` carrying source `customer_selfbind`, credential delivered 26 s later with no operator action (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). The new-ISO console banner rendered on the real box. **Slice 2 = console-passphrase bind → R-27c** (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) |
|
||||
| R-27b | **Customer self-bind, second-box flow (controller side).** For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub `/bind/` page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). | M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same `/bind/` public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance |
|
||||
| R-25b | **RULED: customer DELETE becomes a guided full-teardown cascade.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | **RULED 2026-07-21 — spec to follow; NOT implemented** | **operator ruling 2026-07-21**: **DELETE subsumes the whole cascade, behind explicit consent.** Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a **typed customer-name confirmation** before the button arms. Internal order is **host-delete → RESET → delete**, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). **Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox.** Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. **It no longer blocks R-3** — the model is decided, so the friend-alpha runbook can be written against it |
|
||||
| R-25b | **RULED: customer DELETE becomes a guided full-teardown cascade.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | **SHIPPED hub v0.69.0 (2026-07-21)** | **operator ruling 2026-07-21**: **DELETE subsumes the whole cascade, behind explicit consent.** Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a **typed customer-name confirmation** before the button arms. Internal order is **host-delete → RESET → delete**, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). **Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox.** Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. **It no longer blocks R-3** — the model is decided, so the friend-alpha runbook can be written against it. **IMPLEMENTED per the ruling (TASK-I, hub v0.69.0):** `POST /configs/{id}/delete` now runs `hosts → RESET → purge`; three acks + typed customer-id + a stale-preview check + the ONLINE-host refusal, all gates before any write (zero side effects on refusal); custody purged exactly ONCE in leg 3 (leg 2 runs with `purgeEscrow=false`); ruling-3 preserved BY CONSTRUCTION and asserted from inside leg 2; failed legs retain the journal and the dialog offers Resume. Standalone RESET byte-identical. 5 red-proofs. Offboarding guidance: `runbooks/RUNBOOK-onboarding-draft-v3.md` §G |
|
||||
| R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close |
|
||||
| R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). **COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18):** customer zero shrank the guest **11675 → 8192 MB (16:50:22)** and grew it **8192 → 12288 MB (17:02:17)**, each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. `tests/VALIDATION-n100-rehearsal-2026-07-18.md` ledger 9. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED + PROVEN-LIVE** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". |
|
||||
| R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster |
|
||||
|
||||
@@ -172,3 +172,26 @@ This is the friction lens: the customer's hands on the keyboard, the operator's
|
||||
- Post-rehearsal checklist item: rebuild the golden to the current controller before the FIRST
|
||||
real tester (fresh boxes shouldn't sit in a stale window longer than needed), and re-verify the
|
||||
Day-0 manifest after every controller release during alpha.
|
||||
## G. Offboarding — which lever to pull (hub v0.69.0, R-25b)
|
||||
|
||||
Two levers, and they are not interchangeable:
|
||||
|
||||
- **Ügyfél-visszaállítás (RESET)** — *identity-preserving re-onboarding*. Every operational trace
|
||||
dies (offsite repository, PBS namespace, DR recipe, one-time password, claim state); the customer
|
||||
record, the basic config and the audit history survive. This is Peti's Friday flow: the same
|
||||
customer coming back on new hardware. RESET still **refuses while any host row exists** — delete
|
||||
the host(s) first, on purpose.
|
||||
- **Delete customer… (the DELETE cascade)** — *full teardown, the customer is leaving*. One guided
|
||||
dialog shows the live inventory, takes **three acknowledgements** (1: the host(s) will be deleted,
|
||||
custody demoted not destroyed; 2: the customer will be RESET incl. external teardown and offsite
|
||||
data destruction; 3: the record and ALL escrow ciphertext are purged) plus the **typed
|
||||
customer-id**, then runs `hosts → RESET → purge` in that order.
|
||||
|
||||
Notes for the operator:
|
||||
|
||||
- An **ONLINE host refuses the whole cascade** — decommission the box first (a live agent would get
|
||||
permanent 401s). The dialog says so and disables the button.
|
||||
- If a leg fails (a slow Hetzner call), **nothing is left half-forgotten**: the journal is retained,
|
||||
the error names the leg, and re-opening the dialog offers **Resume**. Completed legs are no-ops.
|
||||
- The **audit event stream survives** the delete — the customer_deleted event and the host-deletion
|
||||
provenance outlive the record, by design.
|
||||
|
||||
@@ -1,5 +1,74 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.69.0 — customer DELETE becomes the guided full-teardown cascade (R-25b) (2026-07-21)
|
||||
|
||||
Implements the operator ruling of 2026-07-21. The customer page carried two half-truths: **RESET**
|
||||
was the real teardown but refused while any host row existed, and **DELETE** quietly removed only
|
||||
the `customer_configs` row (plus escrow custody) — leaving the Hetzner Storage-Box repo, the PBS
|
||||
namespace + credentials, the tunnel/zone plumbing and the host rows behind. DELETE now does what its
|
||||
name promises.
|
||||
|
||||
### The cascade
|
||||
|
||||
`POST /configs/{id}/delete` (same route, new behaviour) runs three legs in a fixed order:
|
||||
|
||||
1. **hosts** — every host row deleted through host-delete's own rules: an **ONLINE host refuses the
|
||||
whole cascade** (checked for every host up front, so it never half-runs) and escrow is **DEMOTED**
|
||||
to retained custody, never destroyed.
|
||||
2. **reset** — the committed RESET sequence verbatim (Hetzner FIRST → PBS → claim → descriptor → DB
|
||||
purge), reached through the newly extracted `commitCustomerReset`.
|
||||
3. **purge** — `DeleteCustomerConfig`: the customer record **and all escrow ciphertext**.
|
||||
|
||||
Nothing here is newly destructive: the cascade only *sequences* three operations that already
|
||||
existed, each keeping its own safety rules. Two invariants are load-bearing and asserted, not merely
|
||||
commented:
|
||||
|
||||
- **Ruling 3 is preserved BY CONSTRUCTION** — leg 2 can only run after leg 1, so the RESET sequence
|
||||
never sees a host row. The standalone RESET handler's 409 gate is untouched.
|
||||
- **Custody is purged EXACTLY ONCE, in leg 3.** Leg 1 demotes; leg 2 is called with
|
||||
`purgeEscrow=false` so `PurgeCustomerResetDBState` leaves retained blobs alone; leg 3 is the one
|
||||
true purge point (v0.60.1). Both are proven from *inside* leg 2 by a fake that observes store state
|
||||
at the moment the PBS deprovision fires.
|
||||
|
||||
### Gates (all before any write — a refused delete has ZERO side effects)
|
||||
|
||||
Three separate acknowledgements (`ack_hosts`, `ack_reset`, `ack_purge`, each must be exactly `1`),
|
||||
the **typed customer-id**, a **stale-preview** check (the acknowledged host count must still match
|
||||
live — otherwise 409 "re-open the dialog"), and the ONLINE-host refusal. No force flag, no skip flag,
|
||||
no partial-run downgrade anywhere in this path.
|
||||
|
||||
### Resume
|
||||
|
||||
A failed leg retains the journal row (`customer_resets`, per-leg status) and the HTTP error **names
|
||||
the leg**. Re-opening the dialog renders the incomplete journal and offers **Resume**; a re-run is
|
||||
idempotent (leg 1 is a no-op once the hosts are gone). The acknowledgements are **not** cached across
|
||||
attempts — a resume passes every gate again.
|
||||
|
||||
### UI
|
||||
|
||||
Danger zone → **Delete customer…** opens a guided dialog: live inventory panel (hosts by name +
|
||||
status, offsite repository identifier, PBS namespace, custody state), the three consequence
|
||||
checkboxes, the typed customer-id field, one submit. Mid-cascade failures render the journal state.
|
||||
The client-side checks are convenience — every gate is enforced server-side.
|
||||
|
||||
### Refactor (standalone RESET behaviour unchanged)
|
||||
|
||||
`handleCustomerReset`'s committed half became `commitCustomerReset(ctx, cfg, resetID, purgeEscrow)`,
|
||||
returning a `resetLegError` (leg name + status + the exact operator-facing message). The standalone
|
||||
path is byte-identical to v0.68.1: same order, same leg names, same messages, same status codes; its
|
||||
suite is untouched and green. The shallow `handleConfigDelete` is **gone** — do not reintroduce a
|
||||
shallow delete path.
|
||||
|
||||
### Tests
|
||||
|
||||
New `internal/web/customer_delete_test.go`: happy-path leg ORDER (observed from inside leg 2), nine
|
||||
fail-closed gate cases each asserting zero mutations *and* zero external calls *and* no journal row,
|
||||
resume-after-external-failure (custody + customer survive the failure, then converge), resume is
|
||||
still gated, the `purgeEscrow` flag's custody semantics, and a preview test asserting the inventory
|
||||
names real things and leaks no secret. **Five red-proofs run** (ack gate, stale-preview gate,
|
||||
ONLINE-host gate, leg order inverted, `purgeEscrow=true`) — all failed red with the wrong value
|
||||
visible, then restored. Full suite green.
|
||||
|
||||
## v0.68.1 — fix the Configuration page layout broken by the wrapper-sha field (2026-07-21)
|
||||
|
||||
The v0.68.0 wrapper-sha256 row wrapped itself in a `<div>`. The artifacts **`<form>` IS the CSS
|
||||
|
||||
@@ -833,18 +833,11 @@ func (s *Server) handleOffsiteFreeze(w http.ResponseWriter, r *http.Request, cus
|
||||
http.Redirect(w, r, "/customers/"+customerID+"?flash="+flash+"#tab=edit", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
// handleConfigDelete deletes a customer config.
|
||||
func (s *Server) handleConfigDelete(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
if err := s.store.DeleteCustomerConfig(customerID); err != nil {
|
||||
s.logger.Printf("[ERROR] Failed to delete config %s: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
s.logger.Printf("[INFO] Customer config deleted: %s", customerID)
|
||||
s.bumpIntent(customerID) // Direction-2: wake any still-holding wait so it completes promptly
|
||||
http.Redirect(w, r, "/configs?flash=deleted", http.StatusSeeOther)
|
||||
}
|
||||
// The shallow customer DELETE that used to live here (a bare DeleteCustomerConfig behind a native
|
||||
// confirm) was REPLACED in v0.69.0 by the guided full-teardown cascade — see web/customer_delete.go
|
||||
// (R-25b). The route is unchanged (POST /configs/{id}/delete); what changed is that it now tears the
|
||||
// hosts, the offsite repo, the PBS namespace and the tunnel/zone down before purging the record,
|
||||
// behind three acknowledgements and a typed customer-id. Do NOT reintroduce a shallow delete path.
|
||||
|
||||
// handleConfigPreview returns the generated YAML for a customer config.
|
||||
func (s *Server) handleConfigPreview(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
package web
|
||||
|
||||
// Customer DELETE cascade (v0.69.0, R-25b — operator ruling 2026-07-21).
|
||||
//
|
||||
// Before v0.69.0 the customer page carried two half-truths: RESET was the real teardown but REFUSED
|
||||
// while any host row existed, and DELETE quietly removed only the customer_configs row (plus escrow
|
||||
// custody) — leaving the Hetzner Storage-Box repo, the PBS namespace + credentials, the tunnel/zone
|
||||
// plumbing and the host rows themselves behind. The ruling makes DELETE what its name promises: ONE
|
||||
// guided flow that shows exactly what exists, takes THREE explicit acknowledgements plus the typed
|
||||
// customer-id, then runs the full teardown in the safe order:
|
||||
//
|
||||
// leg 1 hosts — every host row deleted through the SAME service path as a manual host delete
|
||||
// (ONLINE refuses; escrow is DEMOTED to retained custody, never destroyed)
|
||||
// leg 2 reset — the committed RESET sequence verbatim (Hetzner FIRST, PBS, claim, descriptor,
|
||||
// DB purge) via commitCustomerReset — with purgeEscrow=FALSE, see below
|
||||
// leg 3 purge — DeleteCustomerConfig: the customer record AND all escrow ciphertext
|
||||
//
|
||||
// Nothing here is newly destructive: the cascade only SEQUENCES three operations that already exist,
|
||||
// each keeping its own safety rules. Two invariants are load-bearing:
|
||||
//
|
||||
// - Ruling 3 is preserved BY CONSTRUCTION: leg 2 can only run after leg 1, so the RESET sequence
|
||||
// never sees a host row. The standalone RESET handler's 409 gate is untouched.
|
||||
// - Custody is purged EXACTLY ONCE, in leg 3. Leg 1 demotes (host_escrow → host_escrow_superseded);
|
||||
// leg 2 is called with purgeEscrow=false so PurgeCustomerResetDBState leaves the retained blobs
|
||||
// alone; leg 3's DeleteCustomerConfig is the one true purge point (v0.60.1).
|
||||
//
|
||||
// A leg that fails leaves the journal row retained and the error names the leg. A re-run resumes:
|
||||
// leg 1 is a no-op once the hosts are gone, and every leg of the RESET sequence is idempotent.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// deleteCascadeAcks is the three-acknowledgement gate. Every one is REQUIRED — there is no force or
|
||||
// skip flag anywhere in this file (a missing ack is a refusal, never a downgrade to a partial run).
|
||||
type deleteCascadeAcks struct {
|
||||
Hosts bool // "N host(s) will be deleted — recovery-key custody is demoted, not destroyed"
|
||||
Reset bool // "the customer will be RESET — offsite repo DESTROYED, PBS revoked, tunnel/zone removed"
|
||||
Purge bool // "the customer record and ALL escrow ciphertext are PURGED — unrecoverable"
|
||||
}
|
||||
|
||||
func readDeleteCascadeAcks(r *http.Request) deleteCascadeAcks {
|
||||
return deleteCascadeAcks{
|
||||
Hosts: r.FormValue("ack_hosts") == "1",
|
||||
Reset: r.FormValue("ack_reset") == "1",
|
||||
Purge: r.FormValue("ack_purge") == "1",
|
||||
}
|
||||
}
|
||||
|
||||
func (a deleteCascadeAcks) complete() bool { return a.Hosts && a.Reset && a.Purge }
|
||||
|
||||
// handleCustomerDeletePreview — GET /configs/{id}/delete. The read-only inventory the guided dialog
|
||||
// renders: the RESET inventory EXTENDED with the host list (ruling 4 applied to all three legs).
|
||||
// Counts, names and booleans only — never a secret, blob or key. Also surfaces an incomplete journal
|
||||
// row so the dialog can offer "Resume".
|
||||
func (s *Server) handleCustomerDeletePreview(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
cfg, err := s.store.GetCustomerConfig(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete preview %s: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if cfg == nil {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
inv, err := s.store.CustomerResetInventory(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete preview %s: inventory: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
hosts, err := s.store.ListHostsByCustomer(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete preview %s: hosts: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
hostRows := make([]map[string]any, 0, len(hosts))
|
||||
onlineBlocked := false
|
||||
for i := range hosts {
|
||||
status := s.hostStatus(hosts[i].LastReportAt)
|
||||
if status == "ok" {
|
||||
onlineBlocked = true
|
||||
}
|
||||
hostRows = append(hostRows, map[string]any{
|
||||
"host_id": hosts[i].HostID,
|
||||
"status": status,
|
||||
"online": status == "ok",
|
||||
})
|
||||
}
|
||||
offsiteEnabled, offsiteType := offsiteChoice(cfg.ConfigJSON)
|
||||
offsiteName := ""
|
||||
if offsiteEnabled && s.offsite != nil {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
|
||||
defer cancel()
|
||||
if n, oerr := s.offsite.OffsiteIdentifier(ctx, customerID, offsiteType); oerr != nil {
|
||||
s.logger.Printf("[WARN] delete preview %s: offsite identifier lookup: %v", customerID, oerr)
|
||||
} else {
|
||||
offsiteName = n
|
||||
}
|
||||
}
|
||||
// An incomplete journal row = a cascade that stopped mid-way; the dialog renders it + Resume.
|
||||
var pending map[string]any
|
||||
if cr, jerr := s.store.LatestCustomerReset(customerID); jerr != nil {
|
||||
s.logger.Printf("[WARN] delete preview %s: journal read: %v", customerID, jerr)
|
||||
} else if cr != nil && cr.CompletedAt == nil {
|
||||
pending = map[string]any{
|
||||
"id": cr.ID,
|
||||
"started_at": cr.StartedAt.UTC().Format(time.RFC3339),
|
||||
"legs": cr.Legs,
|
||||
}
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"customer_id": customerID,
|
||||
"customer_name": cfg.CustomerName,
|
||||
"hosts": hostRows,
|
||||
"host_count": inv.HostCount,
|
||||
"online_host_present": onlineBlocked, // leg 1 refuses; decommission the agent first
|
||||
"superseded_blobs": inv.SupersededBlobs,
|
||||
"dr_recipe_present": inv.DRRecipePresent,
|
||||
"one_time_secret": inv.OneTimeSecretPresent,
|
||||
"claim_present": inv.ClaimPresent,
|
||||
"offsite_enabled": offsiteEnabled,
|
||||
"offsite_type": offsiteType,
|
||||
"offsite_identifier": offsiteName,
|
||||
"pbs_tenancy_configured": s.tenantsync != nil,
|
||||
"pending_journal": pending,
|
||||
})
|
||||
}
|
||||
|
||||
// handleCustomerDelete — POST /configs/{id}/delete. The guided full-teardown cascade. EVERY gate is
|
||||
// checked before ANY write or external call: a refused delete leaves ZERO side effects (no host
|
||||
// deleted, no journal row opened, no external call made, no config row touched).
|
||||
func (s *Server) handleCustomerDelete(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||
cfg, err := s.store.GetCustomerConfig(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete %s: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if cfg == nil {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
hosts, err := s.store.ListHostsByCustomer(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete %s: hosts: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// ── Gates (all before any write) ────────────────────────────────────────────────────────────
|
||||
acks := readDeleteCascadeAcks(r)
|
||||
if !acks.complete() {
|
||||
s.logger.Printf("[WARN] delete %s REFUSED: acknowledgements incomplete (hosts=%t reset=%t purge=%t)",
|
||||
customerID, acks.Hosts, acks.Reset, acks.Purge)
|
||||
http.Error(w, "Delete refused: all three acknowledgements are required — nothing was deleted.", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(r.FormValue("confirm_id")) != customerID {
|
||||
s.logger.Printf("[WARN] delete %s REFUSED: typed customer-id mismatch", customerID)
|
||||
http.Error(w, "Delete refused: the typed customer-id does not match — nothing was deleted.", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
// Stale-preview gate: the operator acknowledged a specific host count. If the fleet changed
|
||||
// between opening the dialog and submitting, the acknowledgement no longer describes reality.
|
||||
if expect := strings.TrimSpace(r.FormValue("expect_hosts")); expect == "" || expect != strconv.Itoa(len(hosts)) {
|
||||
s.logger.Printf("[WARN] delete %s REFUSED: stale preview (acknowledged %q host(s), live %d)", customerID, expect, len(hosts))
|
||||
http.Error(w, "Delete refused: the inventory changed since the dialog was opened — re-open it and confirm again. Nothing was deleted.", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
// Leg 1 keeps host-delete's own safety rule: an ONLINE host is never deleted (a live agent would
|
||||
// receive 401s permanently). Checked for EVERY host up front, so the cascade never half-runs.
|
||||
for i := range hosts {
|
||||
if s.hostStatus(hosts[i].LastReportAt) == "ok" {
|
||||
s.logger.Printf("[WARN] delete %s REFUSED: host %s is ONLINE", customerID, hosts[i].HostID)
|
||||
http.Error(w, "Delete refused: host "+hosts[i].HostID+" is ONLINE. Decommission the box first — the cascade never deletes a live host.", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// ── Committed. Detached ctx: once teardown starts it must run to a clean journal state ───────
|
||||
ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 10*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
// escrow_acked=true on the journal row: ack #3 carries the ruling-1 custody-destruction
|
||||
// acknowledgement. The PURGE itself is leg 3's, not the RESET sequence's (see the file header).
|
||||
journalID, err := s.store.StartCustomerReset(customerID, true)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] delete %s: open journal: %v", customerID, err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[INFO] customer DELETE cascade started for %s (journal #%d, %d host(s))", customerID, journalID, len(hosts))
|
||||
|
||||
// ── Leg 1: hosts (demotion, never destruction) ───────────────────────────────────────────────
|
||||
for i := range hosts {
|
||||
hostID := hosts[i].HostID
|
||||
if derr := s.store.DeleteHost(hostID, true); derr != nil {
|
||||
_ = s.store.UpdateResetLeg(journalID, "hosts", "failed")
|
||||
s.logger.Printf("[ERROR] delete %s: host %s delete FAILED (journal #%d retained; re-run to resume): %v", customerID, hostID, journalID, derr)
|
||||
http.Error(w, "Delete incomplete at leg 1 (hosts): removing host "+hostID+" failed — nothing else was touched; re-run to resume. ("+derr.Error()+")", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[INFO] delete %s: host %s deleted (escrow DEMOTED to retained custody)", customerID, hostID)
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(journalID, "hosts", "ok")
|
||||
|
||||
// ── Leg 2: the committed RESET sequence (external teardown FIRST, DB purge last) ─────────────
|
||||
// purgeEscrow=false — retained custody dies exactly once, in leg 3.
|
||||
if lerr := s.commitCustomerReset(ctx, cfg, journalID, false); lerr != nil {
|
||||
s.logger.Printf("[ERROR] delete %s: cascade stopped at leg 2 (%s) — journal #%d retained", customerID, lerr.Leg, journalID)
|
||||
http.Error(w, "Delete incomplete at leg 2 (reset/"+lerr.Leg+"): "+lerr.Msg+" The host(s) are already deleted; re-run to resume.", lerr.Status)
|
||||
return
|
||||
}
|
||||
|
||||
// ── Leg 3: the one true purge point — customer record + ALL escrow ciphertext ────────────────
|
||||
if derr := s.store.DeleteCustomerConfig(customerID); derr != nil {
|
||||
_ = s.store.UpdateResetLeg(journalID, "customer_delete", "failed")
|
||||
s.logger.Printf("[ERROR] delete %s: final purge FAILED (journal #%d retained; re-run to resume): %v", customerID, journalID, derr)
|
||||
http.Error(w, "Delete incomplete at leg 3 (purge): the customer record could not be removed — re-run to resume. ("+derr.Error()+")", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(journalID, "customer_delete", "ok")
|
||||
if ferr := s.store.FinishCustomerReset(journalID); ferr != nil {
|
||||
s.logger.Printf("[WARN] delete %s: journal finish stamp failed (state is complete): %v", customerID, ferr)
|
||||
}
|
||||
|
||||
// Audit event SURVIVES the delete (events are never wiped — the audit trail outlives every
|
||||
// lifecycle tier, and the customer_configs row is gone by now, which is fine: events are keyed
|
||||
// by customer_id, not by a foreign key).
|
||||
msg := "Ügyfél TÖRLÉSE (teljes lebontás): host(ok) törölve, offsite tároló és PBS névtér megsemmisítve, majd az ügyfélrekord és a teljes helyreállítási-kulcs letét véglegesen törölve. Visszafordíthatatlan."
|
||||
if _, eerr := s.store.SaveEvent(customerID, "customer_deleted", "critical", msg, "", "hub"); eerr != nil {
|
||||
s.logger.Printf("[WARN] delete %s: save audit event: %v", customerID, eerr)
|
||||
}
|
||||
|
||||
s.logger.Printf("[INFO] customer DELETE cascade COMPLETE for %s (journal #%d) — full teardown", customerID, journalID)
|
||||
s.bumpIntent(customerID) // Direction-2: wake any still-holding wait so it completes promptly
|
||||
http.Redirect(w, r, "/configs?flash=deleted", http.StatusSeeOther)
|
||||
}
|
||||
@@ -0,0 +1,421 @@
|
||||
package web
|
||||
|
||||
// Customer DELETE cascade (v0.69.0, R-25b) — the load-bearing contracts:
|
||||
//
|
||||
// A) Happy cascade: the legs run in the ORDER hosts → reset → purge. Proven from inside leg 2 (the
|
||||
// PBS deprovision callback): at that instant the host rows are ALREADY gone (leg 1 done) and the
|
||||
// customer row is STILL there (leg 3 not started). Ruling 3 — RESET never runs while a host
|
||||
// exists — is therefore preserved BY CONSTRUCTION and asserted, not merely commented.
|
||||
// B) Gates fail-closed: any missing ack / typed-id mismatch / stale host count / ONLINE host →
|
||||
// 4xx and ZERO mutations (no host deleted, no journal row, no external call, no config touched).
|
||||
// C) Resume: a leg-2 external failure retains the journal and names the leg; the hosts are already
|
||||
// gone and the customer + custody SURVIVE; a re-run resumes and completes without re-demoting.
|
||||
// D) Standalone RESET is untouched (its own suite stays green; here: the cascade's purgeEscrow=false
|
||||
// does not change what a standalone RESET purges).
|
||||
// E) Custody: leg 1 DEMOTES (never purges); the RESET leg with purgeEscrow=false leaves the retained
|
||||
// blobs alone; the purge happens exactly once, in leg 3 (DeleteCustomerConfig).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync"
|
||||
)
|
||||
|
||||
// errCascadeBoom is the injected external-teardown failure (scenario C).
|
||||
var errCascadeBoom = errors.New("pbs unreachable")
|
||||
|
||||
// orderTenancy is a tenancyProvisioner that runs a callback at the exact moment leg 2's PBS
|
||||
// deprovision fires — the observation point that proves the cascade's leg ORDER.
|
||||
type orderTenancy struct {
|
||||
deprovisionCalls int
|
||||
err error
|
||||
onDeprovision func()
|
||||
}
|
||||
|
||||
func (f *orderTenancy) Provision(ctx context.Context, customerID string) (*tenantsync.Result, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (f *orderTenancy) Reissue(ctx context.Context, customerID string) (*tenantsync.Result, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (f *orderTenancy) Deprovision(ctx context.Context, customerID string) (bool, error) {
|
||||
f.deprovisionCalls++
|
||||
if f.onDeprovision != nil {
|
||||
f.onDeprovision()
|
||||
}
|
||||
if f.err != nil {
|
||||
return false, f.err
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// seedDeletable seeds a customer with a full footprint INCLUDING one OFFLINE host that carries a
|
||||
// current escrow blob plus one already-retained (superseded) blob. Returns the host id.
|
||||
func seedDeletable(t *testing.T, st *store.Store, customerID string) string {
|
||||
t.Helper()
|
||||
cfgJSON := `{"offsite":{"enabled":true,"type":"shared","host":"u1-sub3.your-storagebox.de","user":"u1-sub3","port":23,"repo_path":"/home/felhom","quota_gb":100,"host_fingerprint":"SHA256:abc"}}`
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{
|
||||
CustomerID: customerID, CustomerName: "Teszt", Domain: customerID + ".example",
|
||||
Email: "t@example.com", RetrievalPassword: "pw", APIKey: "capi-" + customerID, ConfigJSON: cfgJSON,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed config: %v", err)
|
||||
}
|
||||
hostID := customerID + "-01"
|
||||
long := time.Now().Add(-72 * time.Hour) // far past the stale threshold → status "down", deletable
|
||||
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi-" + hostID, LastReportAt: &long}); err != nil {
|
||||
t.Fatalf("seed host: %v", err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||
t.Fatalf("seed escrow A: %v", err)
|
||||
}
|
||||
// A second save supersedes A → one RETAINED blob + one CURRENT blob before the cascade runs.
|
||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||
t.Fatalf("seed escrow B: %v", err)
|
||||
}
|
||||
if err := st.SaveOneTimeSecret(customerID, "one-time-pw"); err != nil {
|
||||
t.Fatalf("seed one-time secret: %v", err)
|
||||
}
|
||||
if err := st.SaveDRRecipeHostHalf(customerID, hostID, 1, []byte("half")); err != nil {
|
||||
t.Fatalf("seed dr recipe: %v", err)
|
||||
}
|
||||
if _, err := st.RotateClaimCode(customerID, "$2a$10$hashhashhashhashhashha"); err != nil {
|
||||
t.Fatalf("seed claim: %v", err)
|
||||
}
|
||||
return hostID
|
||||
}
|
||||
|
||||
func cascadeForm(customerID string, hostCount int) url.Values {
|
||||
return url.Values{
|
||||
"ack_hosts": {"1"},
|
||||
"ack_reset": {"1"},
|
||||
"ack_purge": {"1"},
|
||||
"confirm_id": {customerID},
|
||||
"expect_hosts": {strconv.Itoa(hostCount)},
|
||||
}
|
||||
}
|
||||
|
||||
func postDelete(t *testing.T, s *Server, customerID string, form url.Values) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest("POST", "/configs/"+customerID+"/delete", strings.NewReader(form.Encode()))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
rr := httptest.NewRecorder()
|
||||
s.handleCustomerDelete(rr, req, customerID)
|
||||
return rr
|
||||
}
|
||||
|
||||
func hostCount(t *testing.T, st *store.Store, customerID string) int {
|
||||
t.Helper()
|
||||
hosts, err := st.ListHostsByCustomer(customerID)
|
||||
if err != nil {
|
||||
t.Fatalf("list hosts: %v", err)
|
||||
}
|
||||
return len(hosts)
|
||||
}
|
||||
|
||||
func configPresent(t *testing.T, st *store.Store, customerID string) bool {
|
||||
t.Helper()
|
||||
cfg, err := st.GetCustomerConfig(customerID)
|
||||
if err != nil {
|
||||
t.Fatalf("get config: %v", err)
|
||||
}
|
||||
return cfg != nil
|
||||
}
|
||||
|
||||
// ── Scenario A: the happy cascade, and the leg ORDER it must run in ─────────────────────────────
|
||||
|
||||
func TestDeleteCascade_HappyPath_LegOrder(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
hostID := seedDeletable(t, st, "acme")
|
||||
|
||||
var (
|
||||
hostsAtResetLeg = -1
|
||||
configAtResetLeg = false
|
||||
custodyAtResetLeg = -1
|
||||
)
|
||||
fake := &orderTenancy{onDeprovision: func() {
|
||||
hostsAtResetLeg = hostCount(t, st, "acme")
|
||||
configAtResetLeg = configPresent(t, st, "acme")
|
||||
custodyAtResetLeg = superseded(t, st, "acme")
|
||||
}}
|
||||
s.SetTenantSync(fake)
|
||||
|
||||
rr := postDelete(t, s, "acme", cascadeForm("acme", 1))
|
||||
if rr.Code != http.StatusSeeOther {
|
||||
t.Fatalf("status = %d, want 303: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if loc := rr.Header().Get("Location"); loc != "/configs?flash=deleted" {
|
||||
t.Errorf("Location = %q, want /configs?flash=deleted", loc)
|
||||
}
|
||||
|
||||
// ORDER, observed from inside leg 2 — this is the assertion the whole cascade hangs on.
|
||||
if fake.deprovisionCalls != 1 {
|
||||
t.Fatalf("PBS deprovision calls = %d, want 1 (leg 2 must run)", fake.deprovisionCalls)
|
||||
}
|
||||
if hostsAtResetLeg != 0 {
|
||||
t.Errorf("at the RESET leg the customer still had %d host(s) — leg 1 must complete FIRST "+
|
||||
"(ruling 3: the RESET sequence never runs while a host row exists)", hostsAtResetLeg)
|
||||
}
|
||||
if !configAtResetLeg {
|
||||
t.Error("at the RESET leg the customer row was already gone — leg 3 must run LAST")
|
||||
}
|
||||
if custodyAtResetLeg == 0 {
|
||||
t.Error("at the RESET leg the retained custody was already gone — leg 1 DEMOTES, it must never purge")
|
||||
}
|
||||
|
||||
// Final state: hosts gone, customer gone, ALL custody gone (leg 3 is the one true purge point).
|
||||
if n := hostCount(t, st, "acme"); n != 0 {
|
||||
t.Errorf("hosts after cascade = %d, want 0", n)
|
||||
}
|
||||
if configPresent(t, st, "acme") {
|
||||
t.Error("customer row survived the cascade")
|
||||
}
|
||||
if esc, err := st.GetHostEscrow(hostID); err != nil || esc != nil {
|
||||
t.Errorf("current escrow survived the cascade (err=%v, row=%v)", err, esc != nil)
|
||||
}
|
||||
if n := superseded(t, st, "acme"); n != 0 {
|
||||
t.Errorf("retained escrow blobs after cascade = %d, want 0 (leg 3 purges custody)", n)
|
||||
}
|
||||
|
||||
// Journal: every leg stamped, completion stamped.
|
||||
cr, err := st.LatestCustomerReset("acme")
|
||||
if err != nil || cr == nil {
|
||||
t.Fatalf("journal: %v (row=%v)", err, cr != nil)
|
||||
}
|
||||
if cr.CompletedAt == nil {
|
||||
t.Error("journal not stamped complete")
|
||||
}
|
||||
for leg, want := range map[string]string{"hosts": "ok", "pbs": "ok", "db_purge": "ok", "customer_delete": "ok"} {
|
||||
if got := cr.Legs[leg]; got != want {
|
||||
t.Errorf("journal leg %q = %q, want %q (legs=%v)", leg, got, want, cr.Legs)
|
||||
}
|
||||
}
|
||||
|
||||
// The audit event SURVIVES the customer row (events are keyed by id, never wiped).
|
||||
evs, err := st.GetRecentEvents("acme", 10)
|
||||
if err != nil {
|
||||
t.Fatalf("events: %v", err)
|
||||
}
|
||||
found := false
|
||||
for _, e := range evs {
|
||||
if e.EventType == "customer_deleted" {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("no customer_deleted audit event survived the cascade (events=%d)", len(evs))
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario B: every gate fails closed, with ZERO mutations ────────────────────────────────────
|
||||
|
||||
func TestDeleteCascade_GatesFailClosed(t *testing.T) {
|
||||
base := func() url.Values { return cascadeForm("acme", 1) }
|
||||
cases := []struct {
|
||||
name string
|
||||
mutate func(url.Values)
|
||||
online bool
|
||||
wantCode int
|
||||
}{
|
||||
{"missing ack 1 (hosts)", func(f url.Values) { f.Del("ack_hosts") }, false, http.StatusBadRequest},
|
||||
{"missing ack 2 (reset)", func(f url.Values) { f.Del("ack_reset") }, false, http.StatusBadRequest},
|
||||
{"missing ack 3 (purge)", func(f url.Values) { f.Del("ack_purge") }, false, http.StatusBadRequest},
|
||||
{"ack sent as something other than 1", func(f url.Values) { f.Set("ack_purge", "yes") }, false, http.StatusBadRequest},
|
||||
{"typed id mismatch", func(f url.Values) { f.Set("confirm_id", "acm") }, false, http.StatusBadRequest},
|
||||
{"typed id absent", func(f url.Values) { f.Del("confirm_id") }, false, http.StatusBadRequest},
|
||||
{"stale preview (host count moved)", func(f url.Values) { f.Set("expect_hosts", "0") }, false, http.StatusConflict},
|
||||
{"stale preview (count absent)", func(f url.Values) { f.Del("expect_hosts") }, false, http.StatusConflict},
|
||||
{"ONLINE host", func(f url.Values) {}, true, http.StatusConflict},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
hostID := seedDeletable(t, st, "acme")
|
||||
if tc.online {
|
||||
// A just-saved report → status "ok" (online), the same way host-delete's own gate sees it.
|
||||
if err := st.SaveHostReport(hostID, "acme", []byte(`{}`), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatalf("make host online: %v", err)
|
||||
}
|
||||
}
|
||||
fake := &orderTenancy{}
|
||||
s.SetTenantSync(fake)
|
||||
custodyBefore := superseded(t, st, "acme")
|
||||
|
||||
f := base()
|
||||
tc.mutate(f)
|
||||
rr := postDelete(t, s, "acme", f)
|
||||
|
||||
if rr.Code != tc.wantCode {
|
||||
t.Errorf("status = %d, want %d: %s", rr.Code, tc.wantCode, rr.Body.String())
|
||||
}
|
||||
// ZERO mutations — the whole point of a fail-closed gate.
|
||||
if n := hostCount(t, st, "acme"); n != 1 {
|
||||
t.Errorf("hosts = %d, want 1 (a refused delete deletes NOTHING)", n)
|
||||
}
|
||||
if !configPresent(t, st, "acme") {
|
||||
t.Error("customer row was deleted by a REFUSED delete")
|
||||
}
|
||||
if esc, _ := st.GetHostEscrow(hostID); esc == nil {
|
||||
t.Error("current escrow was touched by a REFUSED delete")
|
||||
}
|
||||
if n := superseded(t, st, "acme"); n != custodyBefore {
|
||||
t.Errorf("retained custody = %d, want %d (untouched)", n, custodyBefore)
|
||||
}
|
||||
if fake.deprovisionCalls != 0 {
|
||||
t.Errorf("PBS deprovision called %d time(s) on a REFUSED delete — no external call may fire", fake.deprovisionCalls)
|
||||
}
|
||||
if cr, _ := st.LatestCustomerReset("acme"); cr != nil {
|
||||
t.Errorf("a journal row was opened by a REFUSED delete (#%d) — gates run before any write", cr.ID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario C: a mid-cascade external failure is resumable ─────────────────────────────────────
|
||||
|
||||
func TestDeleteCascade_ResumesAfterExternalFailure(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
hostID := seedDeletable(t, st, "acme")
|
||||
fake := &orderTenancy{err: errCascadeBoom}
|
||||
s.SetTenantSync(fake)
|
||||
|
||||
rr := postDelete(t, s, "acme", cascadeForm("acme", 1))
|
||||
if rr.Code != http.StatusBadGateway {
|
||||
t.Fatalf("status = %d, want 502: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if body := rr.Body.String(); !strings.Contains(body, "leg 2") || !strings.Contains(body, "pbs") {
|
||||
t.Errorf("the error must NAME the failed leg, got %q", body)
|
||||
}
|
||||
// Leg 1 completed; legs 2-3 did not. The customer and ALL custody SURVIVE.
|
||||
if n := hostCount(t, st, "acme"); n != 0 {
|
||||
t.Errorf("hosts = %d, want 0 (leg 1 completed before the failure)", n)
|
||||
}
|
||||
if !configPresent(t, st, "acme") {
|
||||
t.Fatal("the customer row was purged despite a failed external leg — the purge must be withheld")
|
||||
}
|
||||
custodyAfterFailure := superseded(t, st, "acme")
|
||||
if custodyAfterFailure == 0 {
|
||||
t.Error("retained custody was destroyed by a FAILED cascade — leg 3 is the only purge point")
|
||||
}
|
||||
cr, err := st.LatestCustomerReset("acme")
|
||||
if err != nil || cr == nil {
|
||||
t.Fatalf("journal retained? err=%v row=%v", err, cr != nil)
|
||||
}
|
||||
if cr.CompletedAt != nil {
|
||||
t.Error("journal stamped complete despite a failed leg")
|
||||
}
|
||||
if cr.Legs["hosts"] != "ok" || cr.Legs["pbs"] != "failed" {
|
||||
t.Errorf("journal legs = %v, want hosts=ok pbs=failed", cr.Legs)
|
||||
}
|
||||
|
||||
// ── Re-run: resumes at leg 2, does NOT re-delete/re-demote hosts, completes. ────────────────
|
||||
fake.err = nil
|
||||
rr2 := postDelete(t, s, "acme", cascadeForm("acme", 0)) // the live host count is now 0
|
||||
if rr2.Code != http.StatusSeeOther {
|
||||
t.Fatalf("resume status = %d, want 303: %s", rr2.Code, rr2.Body.String())
|
||||
}
|
||||
if fake.deprovisionCalls != 2 {
|
||||
t.Errorf("PBS deprovision calls = %d, want 2 (failed + resumed)", fake.deprovisionCalls)
|
||||
}
|
||||
if configPresent(t, st, "acme") {
|
||||
t.Error("customer row survived the resumed cascade")
|
||||
}
|
||||
if esc, _ := st.GetHostEscrow(hostID); esc != nil {
|
||||
t.Error("current escrow survived the resumed cascade")
|
||||
}
|
||||
if n := superseded(t, st, "acme"); n != 0 {
|
||||
t.Errorf("retained custody after resume = %d, want 0", n)
|
||||
}
|
||||
cr2, _ := st.LatestCustomerReset("acme")
|
||||
if cr2 == nil || cr2.CompletedAt == nil {
|
||||
t.Error("the resumed run did not stamp a completed journal")
|
||||
}
|
||||
}
|
||||
|
||||
// A resumed run must still pass every gate — the acknowledgements are not cached across attempts.
|
||||
func TestDeleteCascade_ResumeStillGated(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
seedDeletable(t, st, "acme")
|
||||
fake := &orderTenancy{err: errCascadeBoom}
|
||||
s.SetTenantSync(fake)
|
||||
if rr := postDelete(t, s, "acme", cascadeForm("acme", 1)); rr.Code != http.StatusBadGateway {
|
||||
t.Fatalf("first run status = %d, want 502", rr.Code)
|
||||
}
|
||||
fake.err = nil
|
||||
f := cascadeForm("acme", 0)
|
||||
f.Del("ack_purge")
|
||||
if rr := postDelete(t, s, "acme", f); rr.Code != http.StatusBadRequest {
|
||||
t.Fatalf("resume without ack #3 status = %d, want 400", rr.Code)
|
||||
}
|
||||
if !configPresent(t, st, "acme") {
|
||||
t.Error("an ungated resume purged the customer")
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario E: custody is purged exactly once, in leg 3 ────────────────────────────────────────
|
||||
|
||||
// The cascade calls commitCustomerReset with purgeEscrow=FALSE so the retained custody survives the
|
||||
// RESET leg and dies only in DeleteCustomerConfig. RED-PROOF: pass true here and the first assertion
|
||||
// fails with 0 retained blobs — i.e. the purge would have moved into leg 2.
|
||||
func TestCommitCustomerReset_PurgeEscrowFlagGovernsCustody(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
hostID := seedDeletable(t, st, "acme")
|
||||
if err := st.DeleteHost(hostID, true); err != nil { // leg 1: DEMOTE
|
||||
t.Fatalf("demote: %v", err)
|
||||
}
|
||||
if n := superseded(t, st, "acme"); n != 2 {
|
||||
t.Fatalf("retained blobs after demotion = %d, want 2 (leg 1 demotes, never purges)", n)
|
||||
}
|
||||
cfg, _ := st.GetCustomerConfig("acme")
|
||||
id, err := st.StartCustomerReset("acme", true)
|
||||
if err != nil {
|
||||
t.Fatalf("journal: %v", err)
|
||||
}
|
||||
if lerr := s.commitCustomerReset(context.Background(), cfg, id, false); lerr != nil {
|
||||
t.Fatalf("commitCustomerReset: %v", lerr)
|
||||
}
|
||||
if n := superseded(t, st, "acme"); n != 2 {
|
||||
t.Errorf("retained blobs after the RESET leg = %d, want 2 — the cascade's RESET leg must NOT purge custody", n)
|
||||
}
|
||||
// Leg 3 is the one true purge point.
|
||||
if err := st.DeleteCustomerConfig("acme"); err != nil {
|
||||
t.Fatalf("leg 3: %v", err)
|
||||
}
|
||||
if n := superseded(t, st, "acme"); n != 0 {
|
||||
t.Errorf("retained blobs after leg 3 = %d, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Preview: the dialog's inventory names the real things (never a secret) ──────────────────────
|
||||
|
||||
func TestDeleteCascadePreview_Inventory(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
hostID := seedDeletable(t, st, "acme")
|
||||
s.SetTenantSync(&orderTenancy{})
|
||||
req := httptest.NewRequest("GET", "/configs/acme/delete", nil)
|
||||
rr := httptest.NewRecorder()
|
||||
s.handleCustomerDeletePreview(rr, req, "acme")
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
body := rr.Body.String()
|
||||
for _, want := range []string{hostID, `"host_count":1`, `"online_host_present":false`,
|
||||
`"offsite_enabled":true`, `"pbs_tenancy_configured":true`, `"claim_present":true`, `"superseded_blobs":1`} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("preview missing %s\nbody: %s", want, body)
|
||||
}
|
||||
}
|
||||
for _, secret := range []string{"one-time-pw", "capi-acme", "hapi-", "blobA", "blobB"} {
|
||||
if strings.Contains(body, secret) {
|
||||
t.Errorf("preview leaked %q — counts and names only", secret)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -263,8 +263,11 @@ func TestCustomerActions_RedirectAnchors(t *testing.T) {
|
||||
{"regen-password lands on Setup (its card lives there)", func(t *testing.T, s *Server, rr *httptest.ResponseRecorder) {
|
||||
s.handleConfigRegenPassword(rr, postForm("/configs/c1/regen-password", ""), "c1")
|
||||
}, false, "/customers/c1?flash=password_regenerated#tab=setup"},
|
||||
// v0.69.0 (R-25b): the shallow delete became the guided cascade — same route, same
|
||||
// anchor-free landing, but the three acks + typed id + host-count are now required.
|
||||
{"delete stays anchor-free (leaves the page)", func(t *testing.T, s *Server, rr *httptest.ResponseRecorder) {
|
||||
s.handleConfigDelete(rr, postForm("/configs/c1/delete", ""), "c1")
|
||||
s.handleCustomerDelete(rr, postForm("/configs/c1/delete",
|
||||
"ack_hosts=1&ack_reset=1&ack_purge=1&confirm_id=c1&expect_hosts=0"), "c1")
|
||||
}, false, "/configs?flash=deleted"},
|
||||
}
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// Customer RESET (v0.61.0) — the middle lifecycle tier (host delete < RESET < customer DELETE). One
|
||||
@@ -135,80 +136,11 @@ func (s *Server) handleCustomerReset(w http.ResponseWriter, r *http.Request, cus
|
||||
}
|
||||
s.logger.Printf("[INFO] customer RESET started for %s (journal #%d, escrow_ack=%t)", customerID, resetID, escrowAck)
|
||||
|
||||
// Leg: Hetzner offsite (repo DATA destroyed). Only when the customer chose an offsite tier.
|
||||
offsiteEnabled, offsiteType := offsiteChoice(cfg.ConfigJSON)
|
||||
if offsiteEnabled && s.offsite != nil {
|
||||
if derr := s.offsite.Deprovision(ctx, customerID, offsiteType); derr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "hetzner", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: hetzner deprovision FAILED (journal #%d retained; re-run to resume): %v", customerID, resetID, derr)
|
||||
http.Error(w, "Reset incomplete: the offsite (Hetzner) teardown failed — nothing was purged; re-run to resume. ("+derr.Error()+")", http.StatusBadGateway)
|
||||
// Standalone RESET purges the retained custody itself, gated by the ack it just checked.
|
||||
if lerr := s.commitCustomerReset(ctx, cfg, resetID, escrowAck); lerr != nil {
|
||||
http.Error(w, lerr.Msg, lerr.Status)
|
||||
return
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "hetzner", "ok")
|
||||
s.logger.Printf("[INFO] reset %s: offsite deprovisioned (repo data destroyed)", customerID)
|
||||
} else {
|
||||
_ = s.store.UpdateResetLeg(resetID, "hetzner", "skipped")
|
||||
}
|
||||
|
||||
// Leg: PBS DR tenancy (namespace + backups + token destroyed). The namespace is customer-id-keyed
|
||||
// and survives host deletion, so it is torn down here by id; idempotent when absent.
|
||||
if s.tenantsync != nil {
|
||||
if _, derr := s.tenantsync.Deprovision(ctx, customerID); derr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "pbs", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: PBS deprovision FAILED (journal #%d retained; re-run to resume): %v", customerID, resetID, derr)
|
||||
http.Error(w, "Reset incomplete: the PBS namespace teardown failed — nothing was purged; re-run to resume. ("+derr.Error()+")", http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "pbs", "ok")
|
||||
s.logger.Printf("[INFO] reset %s: PBS tenancy deprovisioned", customerID)
|
||||
} else {
|
||||
_ = s.store.UpdateResetLeg(resetID, "pbs", "skipped")
|
||||
}
|
||||
|
||||
// All external legs are ok — now the DB side (publish-last, one leg at a time so the journal
|
||||
// records where a mid-purge crash stopped). Claim → unclaimed (fresh code next onboarding).
|
||||
if s.claimEngine != nil {
|
||||
if cerr := s.claimEngine.ResetToUnclaimed(cfg); cerr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "claim", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: claim reset failed: %v", customerID, cerr)
|
||||
http.Error(w, "Reset incomplete: the claim reset failed — re-run to resume. ("+cerr.Error()+")", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
} else if derr := s.store.DeleteClaim(customerID); derr != nil { // no engine wired: use the store primitive directly
|
||||
_ = s.store.UpdateResetLeg(resetID, "claim", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: claim delete failed: %v", customerID, derr)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "claim", "ok")
|
||||
|
||||
// Clear the provisioned offsite descriptor (keep the tier CHOICE, drop provisioned host/user/repo/
|
||||
// fingerprint) and re-save → ConfigVersion bump. Identity + basic config survive intact.
|
||||
newConfigJSON, cerr := offsite.ClearProvisionedDescriptor(cfg.ConfigJSON)
|
||||
if cerr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "descriptor", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: clear offsite descriptor: %v", customerID, cerr)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
cfg.ConfigJSON = newConfigJSON
|
||||
if serr := s.store.SaveCustomerConfig(cfg); serr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "descriptor", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: save cleared config: %v", customerID, serr)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "descriptor", "ok")
|
||||
|
||||
// DB purge LAST: retained escrow (ack-gated), one-time secret, DR recipe, log bundles.
|
||||
if perr := s.store.PurgeCustomerResetDBState(customerID, escrowAck); perr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "db_purge", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: DB purge failed: %v", customerID, perr)
|
||||
http.Error(w, "Reset incomplete: the DB purge failed — re-run to resume. ("+perr.Error()+")", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "db_purge", "ok")
|
||||
|
||||
if ferr := s.store.FinishCustomerReset(resetID); ferr != nil {
|
||||
s.logger.Printf("[WARN] reset %s: journal finish stamp failed (state is complete): %v", customerID, ferr)
|
||||
}
|
||||
@@ -231,3 +163,108 @@ func (s *Server) handleCustomerReset(w http.ResponseWriter, r *http.Request, cus
|
||||
s.bumpIntent(customerID) // wake any holding wait so a lingering box sees the cleared state promptly
|
||||
http.Redirect(w, r, "/customers/"+customerID+"?flash=reset_done", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
// resetLegError names the leg of the committed RESET sequence that failed, carrying the exact
|
||||
// operator-facing message + HTTP status the standalone RESET handler has always returned. The DELETE
|
||||
// cascade (v0.69.0, R-25b) reuses the same values so a mid-cascade failure names its leg too.
|
||||
type resetLegError struct {
|
||||
Leg string // journal leg name: hetzner | pbs | claim | descriptor | db_purge
|
||||
Status int
|
||||
Msg string
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *resetLegError) Error() string {
|
||||
if e.Err != nil {
|
||||
return e.Leg + ": " + e.Err.Error()
|
||||
}
|
||||
return e.Leg
|
||||
}
|
||||
|
||||
// commitCustomerReset runs the COMMITTED reset sequence against an already-gated customer: external
|
||||
// teardown FIRST (Hetzner, PBS), then the DB side (claim → descriptor → purge), each leg stamped into
|
||||
// the journal so a failed run is resumable. It deliberately owns no gate, no audit event, no journal
|
||||
// open/close and no redirect — those belong to the caller, because the two callers differ there:
|
||||
//
|
||||
// - standalone RESET (v0.61.0): purgeEscrow = the operator's escrow_ack; the customer survives.
|
||||
// - DELETE cascade (v0.69.0, R-25b): purgeEscrow = FALSE — retained custody is purged exactly ONCE,
|
||||
// in the cascade's final leg (DeleteCustomerConfig, the one true purge point). Purging here too
|
||||
// would split the single custody-destruction point across two legs.
|
||||
//
|
||||
// Behaviour for the standalone caller is byte-identical to v0.68.1 (same order, same leg names, same
|
||||
// messages, same status codes).
|
||||
func (s *Server) commitCustomerReset(ctx context.Context, cfg *store.CustomerConfig, resetID int64, purgeEscrow bool) *resetLegError {
|
||||
customerID := cfg.CustomerID
|
||||
|
||||
// Leg: Hetzner offsite (repo DATA destroyed). Only when the customer chose an offsite tier.
|
||||
offsiteEnabled, offsiteType := offsiteChoice(cfg.ConfigJSON)
|
||||
if offsiteEnabled && s.offsite != nil {
|
||||
if derr := s.offsite.Deprovision(ctx, customerID, offsiteType); derr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "hetzner", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: hetzner deprovision FAILED (journal #%d retained; re-run to resume): %v", customerID, resetID, derr)
|
||||
return &resetLegError{Leg: "hetzner", Status: http.StatusBadGateway, Err: derr,
|
||||
Msg: "Reset incomplete: the offsite (Hetzner) teardown failed — nothing was purged; re-run to resume. (" + derr.Error() + ")"}
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "hetzner", "ok")
|
||||
s.logger.Printf("[INFO] reset %s: offsite deprovisioned (repo data destroyed)", customerID)
|
||||
} else {
|
||||
_ = s.store.UpdateResetLeg(resetID, "hetzner", "skipped")
|
||||
}
|
||||
|
||||
// Leg: PBS DR tenancy (namespace + backups + token destroyed). The namespace is customer-id-keyed
|
||||
// and survives host deletion, so it is torn down here by id; idempotent when absent.
|
||||
if s.tenantsync != nil {
|
||||
if _, derr := s.tenantsync.Deprovision(ctx, customerID); derr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "pbs", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: PBS deprovision FAILED (journal #%d retained; re-run to resume): %v", customerID, resetID, derr)
|
||||
return &resetLegError{Leg: "pbs", Status: http.StatusBadGateway, Err: derr,
|
||||
Msg: "Reset incomplete: the PBS namespace teardown failed — nothing was purged; re-run to resume. (" + derr.Error() + ")"}
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "pbs", "ok")
|
||||
s.logger.Printf("[INFO] reset %s: PBS tenancy deprovisioned", customerID)
|
||||
} else {
|
||||
_ = s.store.UpdateResetLeg(resetID, "pbs", "skipped")
|
||||
}
|
||||
|
||||
// All external legs are ok — now the DB side (publish-last, one leg at a time so the journal
|
||||
// records where a mid-purge crash stopped). Claim → unclaimed (fresh code next onboarding).
|
||||
if s.claimEngine != nil {
|
||||
if cerr := s.claimEngine.ResetToUnclaimed(cfg); cerr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "claim", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: claim reset failed: %v", customerID, cerr)
|
||||
return &resetLegError{Leg: "claim", Status: http.StatusInternalServerError, Err: cerr,
|
||||
Msg: "Reset incomplete: the claim reset failed — re-run to resume. (" + cerr.Error() + ")"}
|
||||
}
|
||||
} else if derr := s.store.DeleteClaim(customerID); derr != nil { // no engine wired: use the store primitive directly
|
||||
_ = s.store.UpdateResetLeg(resetID, "claim", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: claim delete failed: %v", customerID, derr)
|
||||
return &resetLegError{Leg: "claim", Status: http.StatusInternalServerError, Err: derr, Msg: "Internal error"}
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "claim", "ok")
|
||||
|
||||
// Clear the provisioned offsite descriptor (keep the tier CHOICE, drop provisioned host/user/repo/
|
||||
// fingerprint) and re-save → ConfigVersion bump. Identity + basic config survive intact.
|
||||
newConfigJSON, cerr := offsite.ClearProvisionedDescriptor(cfg.ConfigJSON)
|
||||
if cerr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "descriptor", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: clear offsite descriptor: %v", customerID, cerr)
|
||||
return &resetLegError{Leg: "descriptor", Status: http.StatusInternalServerError, Err: cerr, Msg: "Internal error"}
|
||||
}
|
||||
cfg.ConfigJSON = newConfigJSON
|
||||
if serr := s.store.SaveCustomerConfig(cfg); serr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "descriptor", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: save cleared config: %v", customerID, serr)
|
||||
return &resetLegError{Leg: "descriptor", Status: http.StatusInternalServerError, Err: serr, Msg: "Internal error"}
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "descriptor", "ok")
|
||||
|
||||
// DB purge LAST: retained escrow (ack-gated), one-time secret, DR recipe, log bundles.
|
||||
if perr := s.store.PurgeCustomerResetDBState(customerID, purgeEscrow); perr != nil {
|
||||
_ = s.store.UpdateResetLeg(resetID, "db_purge", "failed")
|
||||
s.logger.Printf("[ERROR] reset %s: DB purge failed: %v", customerID, perr)
|
||||
return &resetLegError{Leg: "db_purge", Status: http.StatusInternalServerError, Err: perr,
|
||||
Msg: "Reset incomplete: the DB purge failed — re-run to resume. (" + perr.Error() + ")"}
|
||||
}
|
||||
_ = s.store.UpdateResetLeg(resetID, "db_purge", "ok")
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -494,12 +494,14 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
}
|
||||
case strings.HasPrefix(path, "/configs/") && strings.HasSuffix(path, "/delete"):
|
||||
// Customer DELETE cascade (v0.69.0, R-25b): GET returns the guided dialog's live inventory
|
||||
// (hosts + offsite + PBS + custody + any incomplete journal), POST runs the full teardown.
|
||||
customerID := strings.TrimPrefix(path, "/configs/")
|
||||
customerID = strings.TrimSuffix(customerID, "/delete")
|
||||
if r.Method == http.MethodPost {
|
||||
s.handleConfigDelete(w, r, customerID)
|
||||
s.handleCustomerDelete(w, r, customerID)
|
||||
} else {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
s.handleCustomerDeletePreview(w, r, customerID)
|
||||
}
|
||||
case strings.HasPrefix(path, "/configs/") && strings.HasSuffix(path, "/edit"):
|
||||
customerID := strings.TrimPrefix(path, "/configs/")
|
||||
|
||||
@@ -816,7 +816,7 @@
|
||||
Customer Info header — endpoints and confirm() handlers unchanged. -->
|
||||
<section class="card">
|
||||
<h2>Danger zone</h2>
|
||||
<p class="text-muted">Blocking hides the customer from the Dashboard (reports are still accepted); deleting removes the managed configuration permanently — and permanently removes the retained recovery-key custody (escrow blobs) for this customer's hosts. This is the one true purge point; host deletion only demotes custody, never destroys it.</p>
|
||||
<p class="text-muted">Blocking hides the customer from the Dashboard (reports are still accepted). <strong>Delete customer</strong> is the full offboarding teardown (v0.69.0): it deletes the host(s), then RESETs the customer (offsite repository destroyed, PBS credentials revoked, tunnel and zone removed), then purges the customer record and all escrow ciphertext — including the <strong>retained recovery-key custody</strong> for this customer's hosts. This is the one true purge point; host deletion only demotes custody, never destroys it. Three acknowledgements and the typed customer-id are required. For identity-preserving re-onboarding use <em>Ügyfél-visszaállítás (RESET)</em> above instead.</p>
|
||||
<div style="display: flex; gap: 0.5rem; flex-wrap: wrap; margin-top: 0.5rem;">
|
||||
{{if .IsBlocked}}
|
||||
<form method="POST" action="/customers/{{.CustomerID}}/unblock" style="display:inline">
|
||||
@@ -829,12 +829,109 @@
|
||||
<button type="submit" class="btn btn-outline btn-sm" data-confirm="Block this customer? They will be hidden from the Dashboard.">Block</button>
|
||||
</form>
|
||||
{{end}}
|
||||
<form method="POST" action="/configs/{{.CustomerID}}/delete" style="display:inline">
|
||||
<button type="button" class="btn btn-danger btn-sm" onclick="customerDeleteOpen('{{.CustomerID}}')">Delete customer…</button>
|
||||
</div>
|
||||
|
||||
<!-- Guided full-teardown cascade (v0.69.0, R-25b). The inventory panel is fetched from
|
||||
GET /configs/{id}/delete; the three acknowledgements + typed customer-id are ALSO
|
||||
enforced server-side (this is convenience, never the gate). -->
|
||||
<div id="cust-del-box-{{.CustomerID}}" style="display: none; margin-top: 0.75rem; padding: 0.75rem; border: 1px solid var(--crit); border-radius: var(--radius); max-width: 52em;">
|
||||
<p id="cust-del-inv-{{.CustomerID}}" style="margin: 0 0 0.75rem; font-size: 0.9em;">…</p>
|
||||
<div id="cust-del-journal-{{.CustomerID}}" style="display: none; margin: 0 0 0.75rem; padding: 0.5rem; border: 1px solid var(--warn); font-size: 0.85em;"></div>
|
||||
<label style="display: block; margin: 0 0 0.5rem; font-size: 0.85em;">
|
||||
<input type="checkbox" id="cust-del-ack1-{{.CustomerID}}">
|
||||
<strong>1.</strong> <span id="cust-del-ack1-text-{{.CustomerID}}">The host(s) will be deleted</span> — recovery-key custody is <strong>demoted</strong> to retained custody, not destroyed.
|
||||
</label>
|
||||
<label style="display: block; margin: 0 0 0.5rem; font-size: 0.85em;">
|
||||
<input type="checkbox" id="cust-del-ack2-{{.CustomerID}}">
|
||||
<strong>2.</strong> The customer will be <strong>RESET</strong> — the offsite repository is <strong>DESTROYED</strong>, PBS credentials are revoked, tunnel and zone are removed.
|
||||
</label>
|
||||
<label style="display: block; margin: 0 0 0.75rem; font-size: 0.85em; color: var(--crit);">
|
||||
<input type="checkbox" id="cust-del-ack3-{{.CustomerID}}">
|
||||
<strong>3.</strong> The customer record and <strong>ALL escrow ciphertext</strong> are <strong>PURGED</strong> — unrecoverable.
|
||||
</label>
|
||||
<p style="margin: 0 0 0.4rem; font-size: 0.85em; color: var(--text-2);">Type the customer-id to confirm:</p>
|
||||
<form method="POST" action="/configs/{{.CustomerID}}/delete" id="cust-del-form-{{.CustomerID}}" style="display: flex; gap: 0.5rem; align-items: center; flex-wrap: wrap;">
|
||||
{{.CSRFField}}
|
||||
<button type="submit" class="btn btn-danger btn-sm" data-confirm="Delete configuration for {{.CustomerID}}? This cannot be undone.">Delete</button>
|
||||
<input type="hidden" name="confirm_id" id="cust-del-cid-{{.CustomerID}}" value="">
|
||||
<input type="hidden" name="ack_hosts" id="cust-del-h1-{{.CustomerID}}" value="">
|
||||
<input type="hidden" name="ack_reset" id="cust-del-h2-{{.CustomerID}}" value="">
|
||||
<input type="hidden" name="ack_purge" id="cust-del-h3-{{.CustomerID}}" value="">
|
||||
<input type="hidden" name="expect_hosts" id="cust-del-exp-{{.CustomerID}}" value="">
|
||||
<input type="text" id="cust-del-input-{{.CustomerID}}" placeholder="customer-id…" style="padding: 0.3em 0.5em; width: 16em;">
|
||||
<button type="button" class="btn btn-danger btn-sm" id="cust-del-go-{{.CustomerID}}" onclick="customerDeleteSubmit('{{.CustomerID}}')">Confirm & delete everything</button>
|
||||
<button type="button" class="btn btn-sm btn-outline" onclick="document.getElementById('cust-del-box-{{.CustomerID}}').style.display='none';">Cancel</button>
|
||||
</form>
|
||||
<p id="cust-del-err-{{.CustomerID}}" style="margin: 0.4em 0 0; font-size: 0.8em; color: var(--crit);"></p>
|
||||
</div>
|
||||
</section>
|
||||
<script>
|
||||
function customerDeleteOpen(cid) {
|
||||
var box = document.getElementById('cust-del-box-' + cid);
|
||||
var inv = document.getElementById('cust-del-inv-' + cid);
|
||||
var go = document.getElementById('cust-del-go-' + cid);
|
||||
var jr = document.getElementById('cust-del-journal-' + cid);
|
||||
box.style.display = 'block';
|
||||
inv.textContent = 'Loading inventory…';
|
||||
jr.style.display = 'none';
|
||||
document.getElementById('cust-del-input-' + cid).value = '';
|
||||
document.getElementById('cust-del-err-' + cid).textContent = '';
|
||||
document.getElementById('cust-del-exp-' + cid).value = '';
|
||||
['ack1', 'ack2', 'ack3'].forEach(function(a) { document.getElementById('cust-del-' + a + '-' + cid).checked = false; });
|
||||
go.disabled = false;
|
||||
fetch('/configs/' + encodeURIComponent(cid) + '/delete')
|
||||
.then(function(r){ return r.json(); })
|
||||
.then(function(d){
|
||||
document.getElementById('cust-del-exp-' + cid).value = String(d.host_count);
|
||||
var hostNames = (d.hosts || []).map(function(h){ return h.host_id + ' (' + h.status + ')'; });
|
||||
document.getElementById('cust-del-ack1-text-' + cid).textContent =
|
||||
d.host_count + ' host(s) will be deleted' + (hostNames.length ? ': ' + hostNames.join(', ') : '');
|
||||
var dies = [];
|
||||
if (d.host_count) dies.push(d.host_count + ' host row(s)');
|
||||
if (d.offsite_enabled) dies.push('offsite repository' + (d.offsite_identifier ? ' (' + d.offsite_identifier + ')' : ''));
|
||||
if (d.pbs_tenancy_configured) dies.push('PBS namespace + backups + token');
|
||||
if (d.dr_recipe_present) dies.push('DR recipe');
|
||||
if (d.one_time_secret) dies.push('one-time password');
|
||||
if (d.claim_present) dies.push('claim state');
|
||||
dies.push('customer record');
|
||||
var custody = d.superseded_blobs > 0
|
||||
? d.superseded_blobs + ' retained escrow blob(s) + every current host escrow'
|
||||
: 'every current host escrow';
|
||||
inv.innerHTML = '<strong>Will be destroyed:</strong> ' + dies.join(', ') +
|
||||
'. <strong>Custody:</strong> ' + custody + ' (purged in the final leg). ' +
|
||||
'<strong>Survives:</strong> the audit event stream and the deletion provenance.';
|
||||
if (d.online_host_present) {
|
||||
inv.innerHTML += '<br><strong style="color: var(--crit)">Refused:</strong> a host is ONLINE. ' +
|
||||
'Decommission the box first — the cascade never deletes a live host.';
|
||||
go.disabled = true;
|
||||
}
|
||||
if (d.pending_journal) {
|
||||
var legs = d.pending_journal.legs || {};
|
||||
var parts = Object.keys(legs).map(function(k){ return k + '=' + legs[k]; });
|
||||
jr.innerHTML = '<strong>An earlier cascade stopped mid-way</strong> (journal #' + d.pending_journal.id +
|
||||
', started ' + d.pending_journal.started_at + '). Legs: ' + (parts.length ? parts.join(', ') : 'none recorded') +
|
||||
'. Confirming again RESUMES it — completed legs are no-ops.';
|
||||
jr.style.display = 'block';
|
||||
document.getElementById('cust-del-go-' + cid).textContent = 'Confirm & resume teardown';
|
||||
}
|
||||
})
|
||||
.catch(function(){ inv.textContent = 'Inventory unavailable — the server enforces every gate regardless.'; });
|
||||
}
|
||||
function customerDeleteSubmit(cid) {
|
||||
var err = document.getElementById('cust-del-err-' + cid);
|
||||
var a1 = document.getElementById('cust-del-ack1-' + cid).checked;
|
||||
var a2 = document.getElementById('cust-del-ack2-' + cid).checked;
|
||||
var a3 = document.getElementById('cust-del-ack3-' + cid).checked;
|
||||
if (!a1 || !a2 || !a3) { err.textContent = 'All three acknowledgements are required.'; return; }
|
||||
var typed = document.getElementById('cust-del-input-' + cid).value.trim();
|
||||
if (typed !== cid) { err.textContent = 'The typed customer-id does not match.'; return; }
|
||||
document.getElementById('cust-del-cid-' + cid).value = typed;
|
||||
document.getElementById('cust-del-h1-' + cid).value = '1';
|
||||
document.getElementById('cust-del-h2-' + cid).value = '1';
|
||||
document.getElementById('cust-del-h3-' + cid).value = '1';
|
||||
document.getElementById('cust-del-form-' + cid).submit();
|
||||
}
|
||||
</script>
|
||||
{{end}}
|
||||
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user