docs: hub v0.61.0 Customer RESET — REPORT/CONTEXT + ep0 runbook §10 (tenantsync v1.1.0 deprovision, live-drilled)

This commit is contained in:
2026-07-17 13:15:57 +02:00
parent e144c5e491
commit 7c9647327f
3 changed files with 74 additions and 48 deletions
+16
View File
@@ -3,6 +3,22 @@
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-17 — CUSTOMER RESET SHIPPED: hub v0.61.0 + felhom-tenantsync v1.1.0 (LIVE).** The middle
lifecycle tier (host delete < RESET < customer Delete). One operator action → pre-first-install: all
OPERATIONAL state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state,
retained escrow custody); IDENTITY + basic config (incl. the offsite tier CHOICE) + provenance + events
SURVIVE. Rulings: separate escrow-custody ack; clears claim (fresh code next onboarding); REFUSES while
any host row exists; live-counted confirm inventory. Discipline: external teardown FIRST, DB purge LAST,
every leg idempotent → partial run re-runs from the top (purge withheld until externals ok). New:
`store/customer_reset.go` (journal+inventory+ack-gated purge), `claim.ResetToUnclaimed`,
`offsite.{Deprovision,OffsiteIdentifier,ClearProvisionedDescriptor}`, `tenantsync.Deprovision` +
`felhom-tenantsync.sh` **deprovision** op (v1.1.0, destroys ns+groups+token, shared user untouched),
`web/customer_reset.go` (GET inventory JSON / POST orchestration) + an **amber** RESET card distinct
from the red Danger-zone Delete. Red-proofs: ack-gate + partial-failure resumability (both proven red).
**Live-drilled on ep0** (throwaway `drill-reset-01` with a real backup): deprovision `deleted:true`,
idempotent re-run `deleted:false`, all 3 real tenants survived. Hetzner-delete + the password-gated web
POST covered by tests (offsite Deprovision mirrors live-proven ReissueCredentials). See REPORT.md.
- **2026-07-17 — HOST-DELETE DEMOTES ESCROW: hub v0.60.1 (LIVE).** Closes the v0.60.0 review gap:
`DeleteHost(deleteEscrow=true)` now DEMOTES the current escrow blob into `host_escrow_superseded`
(copy-before-delete, same tx) + spares existing superseded rows — never destroys custody. The
+48 -46
View File
@@ -2,56 +2,58 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## Hub v0.60.1 — host deletion demotes escrow custody (never destroys) + S6b verdict — 2026-07-17
## Hub v0.61.0 + felhom-tenantsync v1.1.0 — Customer RESET (middle lifecycle tier) — 2026-07-17
**Baseline:** hub v0.60.0 (`106c337`) → **v0.60.1** (LIVE, ArgoCD Synced/Healthy). Hub-only; no
controller/agent change. Closes the deletion-path gap found in the v0.60.0 review. Green:
`go build ./... && go vet ./... && go test ./...` + hub confirm gate.
**Commits:** `4009401` (code), `e144c5e` (manifest). **Live:** hub **0.61.0** on k3s (Synced/Healthy);
felhom-tenantsync **v1.1.0** on ep0 (`felhom-hetzner`).
### Scenario A — host delete demotes, never destroys (`store.DeleteHost`)
The v0.60.0 code deleted `host_escrow` + `host_escrow_superseded` on `deleteEscrow=true`. Now it
DEMOTES the current row into `host_escrow_superseded` (copy-BEFORE-delete, same tx) and SPARES existing
superseded rows. Extracted THE one escrow row-copy routine `demoteCurrentEscrowTx` (INSERT…SELECT) and
refactored `SaveHostEscrow` to use it too — one copy routine, not two (rule 2). The F-14 provenance row
+ gate semantics are unchanged (comments reworded: demotion, not destruction). Edge: no escrow row →
unchanged; the `ErrHostEscrowPresent` refusal without the flag is unchanged.
**Red-proof (run-fail-revert):** `TestDeleteHost_DemotesEscrowNeverDestroys` — pre-fix (delete both) →
retained=0 → FAIL; fixed → both blobs (demoted current + spared superseded) retrievable, host gone.
### What shipped
### Scenario B — customer delete is the purge point (`store.DeleteCustomerConfig`)
**What it did before:** ONLY `DELETE FROM customer_configs` — it touched neither hosts nor escrow
(so deleting a customer left hosts + escrow blobs orphaned — an offboarding gap, see Observations).
**Now:** one tx that purges `host_escrow` AND `host_escrow_superseded` for all the customer's hosts —
INCLUDING already-deleted hosts (resolved via the F-14 `host_deletions` provenance `UNION`), so a
host-delete-then-customer-delete ordering leaves nothing orphaned. Danger-zone copy states it.
**Red-proof:** `TestDeleteCustomer_PurgesEscrowCustody` (pre-fix config-only → blobs survive → FAIL).
The **middle lifecycle tier** between *host delete* and *customer Delete*. One operator action returns a
customer to **pre-first-install**: every OPERATIONAL trace dies — offsite repo (Hetzner sub-account/box),
PBS namespace + all backup groups + token, DR recipe, one-time secret, claim state, retained escrow
custody — while **identity + basic config survive** (`customer_configs` incl. the offsite tier CHOICE, all
provenance rows, and the audit-event stream).
### Scenario C — wording (`host_detail_body.html`, `hosts.go`, `customer_unified.html`)
Host-delete escrow checkbox → "Move key escrow to retained custody (required when escrow present) +
remove DR bundle for this host"; the 409 refusal message → "…move it to retained custody…"; the
customer Danger-zone copy → "…permanently removes the retained recovery-key custody (escrow blobs)…
This is the one true purge point; host deletion only demotes custody, never destroys it." Guard test
`TestHostDeleteEscrowLabel_DemotionWording`.
Viktor's rulings, all implemented: (1) destroying retained escrow custody gets its **own separate
acknowledgment**; (2) RESET clears claim state → fresh code next onboarding; (3) RESET **refuses while any
host row exists** (delete hosts first — reset never deletes hosts); (4) the confirm surface shows a
**live-counted** inventory.
### Scenario D — S6b necessity verdict: **OBSOLETE**
**Verdict + landmarks:** re-enrolling an existing host_id upserts cleanly — `store.UpsertHost` is
`INSERT … ON CONFLICT(host_id) DO UPDATE` (`internal/store/store.go:2099`), and
`handleAdminCreateHost` (`internal/api/handler.go:913`) calls it with NO duplicate refusal (it even
mints a fresh api_key). Combined with the v0.57.0 re-enroll arc (claim/offsite/escrow-honesty
auto-fire on re-enrollment, `reissueOnReenroll` `handler.go:1020`), no manual stale-host deletion is
required before re-enroll. Since S6b is not a literal runbook step, the verdict is recorded here + in
ROADMAP R-3 ("re-enrollment upserts cleanly; S6b obsolete; escrow cascade now defined; the rehearsal
live-confirms it"). Scenario A also makes the funnel harmless either way (deletion demotes, not
destroys).
### Discipline
### Deploy verification
`felhom-hub:0.60.1` built + pushed; `manifests/hub.yaml` bumped; ArgoCD Synced/Healthy; rollout
complete (`:0.60.1`). The `demoteCurrentEscrowTx` path is INSERT…SELECT (atomic within the delete tx).
External teardown FIRST, DB purge LAST (publish-last). Every leg idempotent → a partial run is re-run from
the top; a failed external leg is a clean journal entry and the DB purge (which erases the descriptors that
say what still needs tearing down) is **withheld** until every external leg is `ok`. Provenance + events
are never wiped.
### Observations
- **Customer-delete offboarding gap (unchanged this task, by scope):** `DeleteCustomerConfig` still
does NOT delete the customer's HOST rows or the broader offboarding (Hetzner sub-account, WG peer,
Storage-Box data) — only the escrow tables + the config row. That non-escrow lifecycle stays the
delete/re-create rehearsal's question (ROADMAP R-3).
- No pruning added (Viktor's data-first ruling); the ACK assembly + the upload supersede path are
untouched.
### Components
- **store/customer_reset.go**: `customer_resets` journal (per-attempt, per-leg, resumable);
`CustomerResetInventory` (live counts incl. retained blobs via the F-14 `host_deletions` UNION);
`Start/UpdateResetLeg/Finish/LatestCustomerReset`; ack-gated `PurgeCustomerResetDBState`; `DeleteClaim`.
- **claim.ResetToUnclaimed**: deletes the claim row → `EnsureIssued` mints a fresh first code next onboarding.
- **offsite**: `Deprovision` (idempotent label-lookup delete), `OffsiteIdentifier`, `ClearProvisionedDescriptor`.
- **tenantsync + scripts/felhom-tenantsync.sh v1.1.0**: `deprovision` op destroys the namespace + backup
groups + token; shared `felhom@pbs` user untouched; idempotent.
- **web/customer_reset.go**: `GET /configs/{id}/reset` → inventory JSON; `POST` → orchestration (all gates
before any write/external call). A distinct **amber** RESET card + `reset_done` flash.
### Verification
- **Green gate:** `go build ./... && go vet ./... && go test ./...` all pass. Hub confirm gate OK.
- **Red-proofs (run-fail-revert, both proven red then restored):** (a) escrow-ack gate — defeat it →
reset proceeds & destroys blobs → FAIL; (b) partial-failure resumability — don't withhold the purge →
DB purged despite an external failure → FAIL. Plus store ack-gating + journal round-trip; offsite
Deprovision idempotency + descriptor clear; RESET-card render.
- **Live drill on ep0 (throwaway `drill-reset-01`, real PBS data):** provision → **real backup written into
the namespace** → deprovision (`deleted:true`; namespace + backup group + token destroyed) → token/ns
confirmed gone → idempotent re-run (`deleted:false`) → **all 3 real tenants + the shared user survived**.
### Not live-drilled (covered otherwise)
The web RESET POST and the Hetzner sub-account delete are covered by the httptest-driven orchestration
tests + offsite unit tests (the offsite `Deprovision` faithfully mirrors the live-proven
`ReissueCredentials`, swapping reset→delete). The hub UI is operator-password-gated → CC verifies the UI
via render tests, not a live click. A live Hetzner create/delete drill was not run (billed external
service; mirror-of-live-proven + idempotency-tested).
+10 -2
View File
@@ -353,8 +353,16 @@ The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenanc
privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its
OWN keypair + script + sudoers drop-in — **the peersync files are untouched** (one script, one job;
peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth:
`scripts/felhom-tenantsync.sh` (v1.0.0; JSON stdin/stdout, ops `provision`/`reissue`/`fingerprint`,
NO deprovision — data deletion stays a deliberate decision). Hub client: `hub/internal/tenantsync`.
`scripts/felhom-tenantsync.sh` (**v1.1.0**; JSON stdin/stdout, ops
`provision`/`reissue`/`deprovision`/`fingerprint`). Hub client: `hub/internal/tenantsync`.
> **v1.1.0 (2026-07-17, deployed to ep0):** adds the `deprovision` op — the deliberate data-deletion
> the slice-1 header reserved, now driven by the hub-side **customer RESET** (hub v0.61.0, operator
> ack-gated). It destroys the customer's namespace + all backup groups + token
> (`namespace delete --delete-groups true`); the shared `felhom@pbs` user is never touched; idempotent
> (`deleted:false` when nothing was there). To update the on-box script, re-run the install block below
> with the new source (a `.bak-<ver>` is kept). Live-drilled 2026-07-17: provision → real backup →
> deprovision (`deleted:true`, group destroyed) → idempotent re-run → all real tenants survived.
On the build server (the hub's credential; mirror of step 6):
```sh