feat(hub): v0.69.0 — customer DELETE is the guided full-teardown cascade (R-25b)
POST /configs/{id}/delete now runs hosts -> RESET -> purge behind three
acknowledgements, a typed customer-id, a stale-preview check and the
ONLINE-host refusal (every gate before any write, so a refusal has zero
side effects). The shallow handleConfigDelete is gone.
Two invariants are asserted, not just commented: ruling 3 is preserved by
construction (leg 2 never sees a host row) and retained escrow custody is
purged exactly once, in leg 3 (leg 2 runs with purgeEscrow=false).
handleCustomerReset's committed half was extracted as commitCustomerReset;
the standalone RESET path is byte-identical to v0.68.1 and its suite is
untouched. Five red-proofs run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
This commit is contained in:
@@ -1,132 +1,116 @@
|
||||
# REPORT — TASK-H: HP node access + docs + today's findings (2026-07-21, evening)
|
||||
# REPORT — TASK-I: R-25b, customer DELETE becomes the guided full-teardown cascade
|
||||
|
||||
Baseline `59226ed`, clean tree. No product code touched; the N100 and guest 9201 were not modified.
|
||||
**Date:** 2026-07-21 · **Repo:** `felhom.eu` · **Baseline:** `f59aa97` (clean, HEAD == origin/main)
|
||||
**Scope:** hub only — **v0.68.1 → v0.69.0**. No agent / controller / catalog change.
|
||||
|
||||
| Part | Status |
|
||||
## What shipped
|
||||
|
||||
`POST /configs/{id}/delete` (same route, new behaviour) is now the guided full-teardown cascade.
|
||||
`GET` on the same path returns the dialog's live inventory. The shallow `handleConfigDelete` is
|
||||
**gone**.
|
||||
|
||||
Three legs, fixed order:
|
||||
|
||||
1. **hosts** — every host row via `store.DeleteHost(hostID, true)`; escrow **DEMOTED** to retained
|
||||
custody, never destroyed. Host-delete's own ONLINE rule is kept: an ONLINE host refuses the whole
|
||||
cascade, checked for every host up front so it never half-runs.
|
||||
2. **reset** — the committed RESET sequence verbatim (Hetzner → PBS → claim → descriptor → DB purge)
|
||||
through the newly extracted `commitCustomerReset`, called with `purgeEscrow=false`.
|
||||
3. **purge** — `store.DeleteCustomerConfig`: the customer record **and all escrow ciphertext**.
|
||||
|
||||
Two invariants are asserted, not merely commented:
|
||||
|
||||
- **Ruling 3 by construction** — leg 2 can only run after leg 1, so the RESET sequence never sees a
|
||||
host row. The standalone RESET handler's 409 gate is untouched.
|
||||
- **Custody purged exactly ONCE, in leg 3** — leg 1 demotes; leg 2 runs with `purgeEscrow=false`;
|
||||
leg 3 is the one true purge point (v0.60.1).
|
||||
|
||||
**Gates, all before any write** (a refusal has zero side effects): three acknowledgements
|
||||
(`ack_hosts` / `ack_reset` / `ack_purge`, each exactly `1`), the typed customer-id, a **stale-preview**
|
||||
check (the acknowledged host count must still match live → else 409), and the ONLINE-host refusal.
|
||||
No force flag, no skip flag, no partial-run downgrade.
|
||||
|
||||
**Resume:** a failed leg retains the `customer_resets` journal row and the HTTP error names the leg.
|
||||
The dialog renders the incomplete journal and offers **Resume**; a re-run is idempotent and must pass
|
||||
every gate again (acknowledgements are not cached across attempts).
|
||||
|
||||
**UI:** Danger zone → **Delete customer…** → guided dialog (inventory panel: hosts by name + status,
|
||||
offsite repository identifier, PBS namespace, custody state; three consequence checkboxes; typed
|
||||
customer-id; one submit). Client-side checks are convenience only.
|
||||
|
||||
## Refactor — standalone RESET behaviour unchanged
|
||||
|
||||
`handleCustomerReset`'s committed half became
|
||||
`commitCustomerReset(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError`. The standalone path
|
||||
is byte-identical to v0.68.1: same leg order, same leg names, same operator-facing messages, same
|
||||
status codes. Its existing suite is untouched and green.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| 1 — tailscale on demo-hp | **DONE**, one item open (key expiry — needs an admin-console toggle) |
|
||||
| 2 — inventory + docs | **DONE** — `documentation/operations/nodes.md` |
|
||||
| 3 — findings filed | **DONE** — **R-59, R-60, R-61** + positive evidence recorded |
|
||||
| `hub/internal/web/customer_delete.go` | NEW — the cascade + preview |
|
||||
| `hub/internal/web/customer_delete_test.go` | NEW — scenarios A–E |
|
||||
| `hub/internal/web/customer_reset.go` | `commitCustomerReset` + `resetLegError` extracted |
|
||||
| `hub/internal/web/configs.go` | `handleConfigDelete` removed (replaced by a do-not-reintroduce note) |
|
||||
| `hub/internal/web/server.go` | route: GET → preview, POST → cascade |
|
||||
| `hub/internal/web/templates/customer_unified.html` | guided dialog replaces the one-click Delete |
|
||||
| `hub/internal/web/customer_edit_tab_test.go` | the delete redirect case now posts the full acks |
|
||||
| `hub/CHANGELOG.md`, `REUSE.md`, `documentation/backlog/ROADMAP.md`, `documentation/architecture/00-capability-map.md`, `documentation/runbooks/RUNBOOK-onboarding-draft-v3.md` | docs |
|
||||
|
||||
## Part 1 — tailscale on demo-hp
|
||||
## Tests + red-proofs
|
||||
|
||||
`tailscale 1.98.9` (Debian trixie apt repo, host package — same shape as `felhom-pve`).
|
||||
Green gate in `hub/`: `go build ./... && go vet ./... && go test ./...` — **all green**, no flakes.
|
||||
|
||||
```
|
||||
demo-hp 100.76.96.79 online direct path 37.191.56.193:45127
|
||||
```
|
||||
New coverage (`customer_delete_test.go`):
|
||||
|
||||
Verified **from DooPlex over the tailnet, no jumphost**: ping ~40 ms, `ssh root@100.76.96.79` works,
|
||||
and the path is **direct**, not a DERP relay. `~/.ssh/config` gains `demo-hp` (tailnet) and
|
||||
`demo-hp-lan` (`192.168.0.87` via `ProxyJump felhom-pve`) as the away-fallback.
|
||||
- **A — leg ORDER**, observed from *inside* leg 2 via a `tenancyProvisioner` fake whose `Deprovision`
|
||||
snapshots store state: at that instant hosts = 0 (leg 1 done), the customer row is still present
|
||||
(leg 3 not started), retained custody still present (leg 1 demoted, did not purge). Plus final
|
||||
state, the four journal legs stamped `ok`, completion stamped, and the `customer_deleted` audit
|
||||
event surviving the record.
|
||||
- **B — 9 fail-closed gate cases** (each missing ack, an ack sent as `yes`, id mismatch, id absent,
|
||||
host count moved, host count absent, ONLINE host). Each asserts the status code **and** that the
|
||||
host, the customer row, the current escrow and the retained custody are untouched, **and** that
|
||||
zero external calls fired, **and** that no journal row was opened.
|
||||
- **C — resume**: an injected PBS failure → 502 naming the leg; hosts gone, customer + custody
|
||||
SURVIVE, journal retained with `hosts=ok pbs=failed`; re-run converges and completes. Plus: a
|
||||
resume without ack #3 is still refused.
|
||||
- **E — custody**: `commitCustomerReset(..., purgeEscrow=false)` leaves the retained blobs; leg 3
|
||||
purges them.
|
||||
- **Preview**: names the real host/offsite/PBS/custody facts and leaks no secret (one-time password,
|
||||
API keys, escrow blobs all asserted absent).
|
||||
|
||||
The pre-auth key was passed as `--authkey=file:<path>` and shredded immediately, so it never entered
|
||||
the box's process list.
|
||||
**Five red-proofs run, each failed red with the wrong value visible, then restored (`git diff` clean):**
|
||||
|
||||
### OPEN — key expiry is NOT disabled
|
||||
|
||||
`demo-hp` expires **2027-01-17**; `felhom-pve` and `dooplex` both have expiry disabled, so this is
|
||||
the one place it diverges from the fleet convention. **I could not do it from here:** disabling
|
||||
per-device key expiry is an admin-console toggle or an API call, and the supplied `tskey-auth-…`
|
||||
pre-auth key cannot drive the API (verified — `GET /api/v2/tailnet/-/devices` returns **401**).
|
||||
|
||||
**One click when convenient:** Tailscale admin → Machines → `demo-hp` → *Disable key expiry*. Or
|
||||
hand me a `tskey-api-…` token and I will do it. Verify with:
|
||||
`ssh demo-hp 'tailscale status --json' | python3 -c "import sys,json;print(json.load(sys.stdin)['Self'].get('KeyExpiry'))"` → should print `None`.
|
||||
|
||||
### A rule I broke, then fixed
|
||||
|
||||
The join omitted `--accept-dns=false`, and MagicDNS immediately rewrote `/etc/resolv.conf` to
|
||||
`nameserver 100.100.100.100` — exactly what `operations/tailscale.md` forbids for a host node.
|
||||
|
||||
Nothing broke *at the vacation site*: there is no pi-hole there, and both `gitea.dooplex.hu` and
|
||||
`hub.felhom.eu` resolve publicly anyway. **The damage would have surfaced silently when the box comes
|
||||
home**, where split-horizon is what makes `gitea.dooplex.hu` resolve to `192.168.0.180` locally — a
|
||||
class of failure that looks like "the network is slow" rather than "DNS is wrong".
|
||||
`tailscale set --accept-dns=false` restored `nameserver 192.168.0.1`; tailnet and agent unaffected.
|
||||
The doc now carries the evidence and the instruction to pass the flag **at join time**.
|
||||
|
||||
## Part 2 — the node inventory
|
||||
|
||||
**`documentation/operations/nodes.md`** (new). The fleet is now two hosts, both agent 0.92.1:
|
||||
`demo-felhom-8363b5` (N100) and `demo-hp-bb76ea` (HP t740). Both are at the vacation site and travel
|
||||
home ~2026-08-02.
|
||||
|
||||
**demo-hp:** HP t740 Thin Client, s/n `8CN944035T`, AMI M42 v01.10 (11/11/2020), Ryzen Embedded
|
||||
V1756B (8 threads), **30 GiB RAM**, PVE 9.2.2 as node `felhom-host`, guest 9201 `demo-hp` running,
|
||||
WireGuard `10.77.0.3/32` up.
|
||||
|
||||
| device | serial | role |
|
||||
| # | Pre-fix shape restored | Failure observed |
|
||||
|---|---|---|
|
||||
| `sda` SanDisk X600 128GB | `182195804614` | system disk — PVE + LVM + guest volumes |
|
||||
| `nvme0n1` Toshiba KXG50 1024GB | `58BS11AFT8MQ` | **PRESENT AND UNENROLLED — do not touch** |
|
||||
| 1 | ack gate disabled | `status = 303, want 400` + host deleted, customer deleted, custody destroyed, PBS deprovision fired, journal row opened |
|
||||
| 2 | stale-preview gate weakened to always pass | `status = 303, want 409` + the same six non-effect assertions |
|
||||
| 3 | ONLINE-host gate removed | `status = 303, want 409` + live host deleted |
|
||||
| 4 | leg order inverted (RESET leg before the host leg) | `at the RESET leg the customer still had 1 host(s) — leg 1 must complete FIRST` |
|
||||
| 5 | cascade's RESET leg called with `purgeEscrow=true` | `retained blobs after the RESET leg = 0, want 2` |
|
||||
|
||||
The NVMe still holds its previous **NTFS** partition, is unmounted, and is in no LVM PV and no ZFS
|
||||
pool — the exact-serial filter did its job on hardware it had never seen. It joins later through the
|
||||
**Tárhely flow, never the installer**.
|
||||
## Method / limits
|
||||
|
||||
**NIC map** (documented in both `nodes.md` and `scripts/iso/README.md`): `enp1s0f0–f3` are the
|
||||
4-port `igb` card with **no carrier and no DHCP** at this site; **`enp2s0f0`** (`r8169`, MAC
|
||||
`7c:d3:0a:77:d9:76`) is the onboard port that works and is now `vmbr0`'s bridge-port. MACs for all
|
||||
five are in the doc.
|
||||
Unit-land only so far — **the STOP-gated live leg has not been run** (see below). Validation method:
|
||||
Go tests against a real SQLite store on `t.TempDir()` with fakes at the existing `tenancyProvisioner`
|
||||
seam. No browser is available on DooPlex; the dialog's markup is covered by the existing
|
||||
customer-page render tests (exactly one `/configs/{id}/delete` form on the page) — a strict
|
||||
click-through remains a manual operator pass.
|
||||
|
||||
**Loader/firmware finding:** the box installed with the **shim** loader and **Secure Boot ENABLED**
|
||||
(`mokutil --sb-state` → `SecureBoot enabled`). That retires an assumption — SB-off was an
|
||||
N100-firmware workaround, not a Felhom requirement.
|
||||
**Not covered by unit tests:** the Hetzner offsite `Deprovision` leg (`offsite.Provisioner` is a
|
||||
concrete type, no interface seam — same as the standalone RESET suite). That leg is exactly what the
|
||||
live run is for.
|
||||
|
||||
**Access, and why it is awkward:** no operator SSH key is baked (the HP profile deliberately left
|
||||
`FELHOM_ROOT_SSH_KEY` blank), so authentication is the **G1 break-glass root password vaulted in the
|
||||
hub** (`host_recovery` row `demo-hp-bb76ea`, set 2026-07-21 16:24 UTC). I retrieved it by streaming
|
||||
`/data/hub.db` out of the hub pod, extracting one field to a 0600 file, and **shredding the copy
|
||||
immediately** — that DB holds every host's secret. Recipe is in the doc. This lockout is R-61.
|
||||
## STOP — operator-present live leg (NOT yet run)
|
||||
|
||||
**The operator-lab exception is documented prominently**, in both `nodes.md` and `tailscale.md`:
|
||||
demo-hp is *customer-shaped* but tailscale is **not** part of that shape. Real customer boxes get the
|
||||
WireGuard tunnel and the H1 OOB path and nothing else. A future product-shape audit finding tailscale
|
||||
here must not conclude the product ships it.
|
||||
Create a scratch customer on a throwaway domain, provision **offsite only** (no host — cheap), run
|
||||
the cascade end-to-end, then verify from **outside** the hub that the Hetzner repository is gone and
|
||||
the customer row is purged. **Never run against Demo Ügyfél, Demo HP, or Peti.** Failure paths are
|
||||
unit-proven; the live leg proves the happy path + external teardown only.
|
||||
|
||||
## Part 3 — findings filed
|
||||
## Deploy status
|
||||
|
||||
**TASK-G Part 3 verified as already filed** — R-58 (assisted disk-picker) exists and includes the
|
||||
abort-screen candidate-table slice. Nothing to complete.
|
||||
|
||||
New rows:
|
||||
|
||||
- **R-59 [P1] — a no-DHCP install must HARD-ABORT.** Instead it baked `192.168.100.2` as a *static*
|
||||
`vmbr0` address and completed: the install "succeeded", the box looked finished, and it could never
|
||||
call home. The worst silent onboarding failure shape there is. The philosophy already exists one
|
||||
layer over — the disk filter refuses loudly and touches nothing when it cannot identify its target;
|
||||
networking should do the same, with the candidate NIC table on screen (same grammar as R-58 slice 1).
|
||||
- **R-60 [P2] — first-boot NIC sweep self-heal.** Hub unreachable ⇒ DHCP across every carrier-bearing
|
||||
NIC before settling. Today's repair was a human moving one cable; a sweep would have healed it
|
||||
unaided. Scoped to first boot and the hub-unreachable condition only — a running box must never
|
||||
re-shuffle its own networking.
|
||||
- **R-61 [P1] — the baked root password must be knowable.** The ISO mints a throwaway hash per build
|
||||
and discards the plaintext, so nobody can reach the console of a box they just installed. Slice 1:
|
||||
emit it into the build REPORT + operator cheat-sheet alongside the sha256. **A fixed well-known
|
||||
password is explicitly rejected** (operator ruling) — a pre-pairing box sits on a stranger's LAN.
|
||||
|
||||
Positive evidence, same session:
|
||||
|
||||
- **R-21 slice C is no longer a one-board result.** The row and the capability-map ISO row now read
|
||||
**PROVEN-LIVE on TWO different boards** (N100 2026-07-18; HP t740 2026-07-21) — virgin hardware,
|
||||
one pass, self-registration → operator bind → day-0 → running guest + agent check-in. The Secure
|
||||
Boot finding is recorded there too.
|
||||
- **Fresh-box floor lift**: the new box came up on golden **0.153.0** and self-updated to the fleet
|
||||
floor **0.156.0** during day-0, unattended. Cited on the publish-train capability-map row — the
|
||||
floor mechanism works on first contact, not only on boxes with history.
|
||||
- **The t740 five-NIC trap** is recorded as a board gotcha in `scripts/iso/README.md`.
|
||||
|
||||
## Observations
|
||||
|
||||
1. **The break-glass path works, and it is also the argument for R-61.** Reaching a box whose root
|
||||
password was never known required a working hub, a working network, and operator tooling — at
|
||||
exactly the moment the reason you want the console is usually that one of those is broken.
|
||||
2. **`apt update` fails on this box** against the PVE **enterprise** repos (401, no subscription) —
|
||||
pre-existing from the install, unrelated to today. I worked around it with a list-scoped
|
||||
`apt-get update` rather than editing the box's repo config. Worth deciding whether `host-install`
|
||||
should switch fresh boxes to the no-subscription repo; left alone deliberately.
|
||||
3. **A heredoc silently ate a piped secret.** `printf … | ssh host 'bash -s' <<'EOF'` sends the
|
||||
*heredoc* as stdin, so the piped key never arrives and `read` consumes script text instead. The
|
||||
working shape is the command as an argument, with stdin free for the secret — then stage it as a
|
||||
file and use `--authkey=file:`. Worth remembering next time a credential has to cross an SSH hop.
|
||||
Code pushed to `main`; hub image `0.69.0` built and pushed; `manifests/hub.yaml` bumped and synced —
|
||||
see the verification lines below.
|
||||
|
||||
Reference in New Issue
Block a user