docs: hub v0.61.0 Customer RESET — REPORT/CONTEXT + ep0 runbook §10 (tenantsync v1.1.0 deprovision, live-drilled)
This commit is contained in:
+16
@@ -3,6 +3,22 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
- **2026-07-17 — CUSTOMER RESET SHIPPED: hub v0.61.0 + felhom-tenantsync v1.1.0 (LIVE).** The middle
|
||||
lifecycle tier (host delete < RESET < customer Delete). One operator action → pre-first-install: all
|
||||
OPERATIONAL state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state,
|
||||
retained escrow custody); IDENTITY + basic config (incl. the offsite tier CHOICE) + provenance + events
|
||||
SURVIVE. Rulings: separate escrow-custody ack; clears claim (fresh code next onboarding); REFUSES while
|
||||
any host row exists; live-counted confirm inventory. Discipline: external teardown FIRST, DB purge LAST,
|
||||
every leg idempotent → partial run re-runs from the top (purge withheld until externals ok). New:
|
||||
`store/customer_reset.go` (journal+inventory+ack-gated purge), `claim.ResetToUnclaimed`,
|
||||
`offsite.{Deprovision,OffsiteIdentifier,ClearProvisionedDescriptor}`, `tenantsync.Deprovision` +
|
||||
`felhom-tenantsync.sh` **deprovision** op (v1.1.0, destroys ns+groups+token, shared user untouched),
|
||||
`web/customer_reset.go` (GET inventory JSON / POST orchestration) + an **amber** RESET card distinct
|
||||
from the red Danger-zone Delete. Red-proofs: ack-gate + partial-failure resumability (both proven red).
|
||||
**Live-drilled on ep0** (throwaway `drill-reset-01` with a real backup): deprovision `deleted:true`,
|
||||
idempotent re-run `deleted:false`, all 3 real tenants survived. Hetzner-delete + the password-gated web
|
||||
POST covered by tests (offsite Deprovision mirrors live-proven ReissueCredentials). See REPORT.md.
|
||||
|
||||
- **2026-07-17 — HOST-DELETE DEMOTES ESCROW: hub v0.60.1 (LIVE).** Closes the v0.60.0 review gap:
|
||||
`DeleteHost(deleteEscrow=true)` now DEMOTES the current escrow blob into `host_escrow_superseded`
|
||||
(copy-before-delete, same tx) + spares existing superseded rows — never destroys custody. The
|
||||
|
||||
@@ -2,56 +2,58 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## Hub v0.60.1 — host deletion demotes escrow custody (never destroys) + S6b verdict — 2026-07-17
|
||||
## Hub v0.61.0 + felhom-tenantsync v1.1.0 — Customer RESET (middle lifecycle tier) — 2026-07-17
|
||||
|
||||
**Baseline:** hub v0.60.0 (`106c337`) → **v0.60.1** (LIVE, ArgoCD Synced/Healthy). Hub-only; no
|
||||
controller/agent change. Closes the deletion-path gap found in the v0.60.0 review. Green:
|
||||
`go build ./... && go vet ./... && go test ./...` + hub confirm gate.
|
||||
**Commits:** `4009401` (code), `e144c5e` (manifest). **Live:** hub **0.61.0** on k3s (Synced/Healthy);
|
||||
felhom-tenantsync **v1.1.0** on ep0 (`felhom-hetzner`).
|
||||
|
||||
### Scenario A — host delete demotes, never destroys (`store.DeleteHost`)
|
||||
The v0.60.0 code deleted `host_escrow` + `host_escrow_superseded` on `deleteEscrow=true`. Now it
|
||||
DEMOTES the current row into `host_escrow_superseded` (copy-BEFORE-delete, same tx) and SPARES existing
|
||||
superseded rows. Extracted THE one escrow row-copy routine `demoteCurrentEscrowTx` (INSERT…SELECT) and
|
||||
refactored `SaveHostEscrow` to use it too — one copy routine, not two (rule 2). The F-14 provenance row
|
||||
+ gate semantics are unchanged (comments reworded: demotion, not destruction). Edge: no escrow row →
|
||||
unchanged; the `ErrHostEscrowPresent` refusal without the flag is unchanged.
|
||||
**Red-proof (run-fail-revert):** `TestDeleteHost_DemotesEscrowNeverDestroys` — pre-fix (delete both) →
|
||||
retained=0 → FAIL; fixed → both blobs (demoted current + spared superseded) retrievable, host gone.
|
||||
### What shipped
|
||||
|
||||
### Scenario B — customer delete is the purge point (`store.DeleteCustomerConfig`)
|
||||
**What it did before:** ONLY `DELETE FROM customer_configs` — it touched neither hosts nor escrow
|
||||
(so deleting a customer left hosts + escrow blobs orphaned — an offboarding gap, see Observations).
|
||||
**Now:** one tx that purges `host_escrow` AND `host_escrow_superseded` for all the customer's hosts —
|
||||
INCLUDING already-deleted hosts (resolved via the F-14 `host_deletions` provenance `UNION`), so a
|
||||
host-delete-then-customer-delete ordering leaves nothing orphaned. Danger-zone copy states it.
|
||||
**Red-proof:** `TestDeleteCustomer_PurgesEscrowCustody` (pre-fix config-only → blobs survive → FAIL).
|
||||
The **middle lifecycle tier** between *host delete* and *customer Delete*. One operator action returns a
|
||||
customer to **pre-first-install**: every OPERATIONAL trace dies — offsite repo (Hetzner sub-account/box),
|
||||
PBS namespace + all backup groups + token, DR recipe, one-time secret, claim state, retained escrow
|
||||
custody — while **identity + basic config survive** (`customer_configs` incl. the offsite tier CHOICE, all
|
||||
provenance rows, and the audit-event stream).
|
||||
|
||||
### Scenario C — wording (`host_detail_body.html`, `hosts.go`, `customer_unified.html`)
|
||||
Host-delete escrow checkbox → "Move key escrow to retained custody (required when escrow present) +
|
||||
remove DR bundle for this host"; the 409 refusal message → "…move it to retained custody…"; the
|
||||
customer Danger-zone copy → "…permanently removes the retained recovery-key custody (escrow blobs)…
|
||||
This is the one true purge point; host deletion only demotes custody, never destroys it." Guard test
|
||||
`TestHostDeleteEscrowLabel_DemotionWording`.
|
||||
Viktor's rulings, all implemented: (1) destroying retained escrow custody gets its **own separate
|
||||
acknowledgment**; (2) RESET clears claim state → fresh code next onboarding; (3) RESET **refuses while any
|
||||
host row exists** (delete hosts first — reset never deletes hosts); (4) the confirm surface shows a
|
||||
**live-counted** inventory.
|
||||
|
||||
### Scenario D — S6b necessity verdict: **OBSOLETE**
|
||||
**Verdict + landmarks:** re-enrolling an existing host_id upserts cleanly — `store.UpsertHost` is
|
||||
`INSERT … ON CONFLICT(host_id) DO UPDATE` (`internal/store/store.go:2099`), and
|
||||
`handleAdminCreateHost` (`internal/api/handler.go:913`) calls it with NO duplicate refusal (it even
|
||||
mints a fresh api_key). Combined with the v0.57.0 re-enroll arc (claim/offsite/escrow-honesty
|
||||
auto-fire on re-enrollment, `reissueOnReenroll` `handler.go:1020`), no manual stale-host deletion is
|
||||
required before re-enroll. Since S6b is not a literal runbook step, the verdict is recorded here + in
|
||||
ROADMAP R-3 ("re-enrollment upserts cleanly; S6b obsolete; escrow cascade now defined; the rehearsal
|
||||
live-confirms it"). Scenario A also makes the funnel harmless either way (deletion demotes, not
|
||||
destroys).
|
||||
### Discipline
|
||||
|
||||
### Deploy verification
|
||||
`felhom-hub:0.60.1` built + pushed; `manifests/hub.yaml` bumped; ArgoCD Synced/Healthy; rollout
|
||||
complete (`:0.60.1`). The `demoteCurrentEscrowTx` path is INSERT…SELECT (atomic within the delete tx).
|
||||
External teardown FIRST, DB purge LAST (publish-last). Every leg idempotent → a partial run is re-run from
|
||||
the top; a failed external leg is a clean journal entry and the DB purge (which erases the descriptors that
|
||||
say what still needs tearing down) is **withheld** until every external leg is `ok`. Provenance + events
|
||||
are never wiped.
|
||||
|
||||
### Observations
|
||||
- **Customer-delete offboarding gap (unchanged this task, by scope):** `DeleteCustomerConfig` still
|
||||
does NOT delete the customer's HOST rows or the broader offboarding (Hetzner sub-account, WG peer,
|
||||
Storage-Box data) — only the escrow tables + the config row. That non-escrow lifecycle stays the
|
||||
delete/re-create rehearsal's question (ROADMAP R-3).
|
||||
- No pruning added (Viktor's data-first ruling); the ACK assembly + the upload supersede path are
|
||||
untouched.
|
||||
### Components
|
||||
|
||||
- **store/customer_reset.go**: `customer_resets` journal (per-attempt, per-leg, resumable);
|
||||
`CustomerResetInventory` (live counts incl. retained blobs via the F-14 `host_deletions` UNION);
|
||||
`Start/UpdateResetLeg/Finish/LatestCustomerReset`; ack-gated `PurgeCustomerResetDBState`; `DeleteClaim`.
|
||||
- **claim.ResetToUnclaimed**: deletes the claim row → `EnsureIssued` mints a fresh first code next onboarding.
|
||||
- **offsite**: `Deprovision` (idempotent label-lookup delete), `OffsiteIdentifier`, `ClearProvisionedDescriptor`.
|
||||
- **tenantsync + scripts/felhom-tenantsync.sh v1.1.0**: `deprovision` op destroys the namespace + backup
|
||||
groups + token; shared `felhom@pbs` user untouched; idempotent.
|
||||
- **web/customer_reset.go**: `GET /configs/{id}/reset` → inventory JSON; `POST` → orchestration (all gates
|
||||
before any write/external call). A distinct **amber** RESET card + `reset_done` flash.
|
||||
|
||||
### Verification
|
||||
|
||||
- **Green gate:** `go build ./... && go vet ./... && go test ./...` all pass. Hub confirm gate OK.
|
||||
- **Red-proofs (run-fail-revert, both proven red then restored):** (a) escrow-ack gate — defeat it →
|
||||
reset proceeds & destroys blobs → FAIL; (b) partial-failure resumability — don't withhold the purge →
|
||||
DB purged despite an external failure → FAIL. Plus store ack-gating + journal round-trip; offsite
|
||||
Deprovision idempotency + descriptor clear; RESET-card render.
|
||||
- **Live drill on ep0 (throwaway `drill-reset-01`, real PBS data):** provision → **real backup written into
|
||||
the namespace** → deprovision (`deleted:true`; namespace + backup group + token destroyed) → token/ns
|
||||
confirmed gone → idempotent re-run (`deleted:false`) → **all 3 real tenants + the shared user survived**.
|
||||
|
||||
### Not live-drilled (covered otherwise)
|
||||
|
||||
The web RESET POST and the Hetzner sub-account delete are covered by the httptest-driven orchestration
|
||||
tests + offsite unit tests (the offsite `Deprovision` faithfully mirrors the live-proven
|
||||
`ReissueCredentials`, swapping reset→delete). The hub UI is operator-password-gated → CC verifies the UI
|
||||
via render tests, not a live click. A live Hetzner create/delete drill was not run (billed external
|
||||
service; mirror-of-live-proven + idempotency-tested).
|
||||
|
||||
@@ -353,8 +353,16 @@ The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenanc
|
||||
privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its
|
||||
OWN keypair + script + sudoers drop-in — **the peersync files are untouched** (one script, one job;
|
||||
peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth:
|
||||
`scripts/felhom-tenantsync.sh` (v1.0.0; JSON stdin/stdout, ops `provision`/`reissue`/`fingerprint`,
|
||||
NO deprovision — data deletion stays a deliberate decision). Hub client: `hub/internal/tenantsync`.
|
||||
`scripts/felhom-tenantsync.sh` (**v1.1.0**; JSON stdin/stdout, ops
|
||||
`provision`/`reissue`/`deprovision`/`fingerprint`). Hub client: `hub/internal/tenantsync`.
|
||||
|
||||
> **v1.1.0 (2026-07-17, deployed to ep0):** adds the `deprovision` op — the deliberate data-deletion
|
||||
> the slice-1 header reserved, now driven by the hub-side **customer RESET** (hub v0.61.0, operator
|
||||
> ack-gated). It destroys the customer's namespace + all backup groups + token
|
||||
> (`namespace delete --delete-groups true`); the shared `felhom@pbs` user is never touched; idempotent
|
||||
> (`deleted:false` when nothing was there). To update the on-box script, re-run the install block below
|
||||
> with the new source (a `.bak-<ver>` is kept). Live-drilled 2026-07-17: provision → real backup →
|
||||
> deprovision (`deleted:true`, group destroyed) → idempotent re-run → all real tenants survived.
|
||||
|
||||
On the build server (the hub's credential; mirror of step 6):
|
||||
```sh
|
||||
|
||||
Reference in New Issue
Block a user