admin
094e93e828
gates / gates (push) Successful in 13s
All five layers gone, each verified with a positive control that must
survive and does:
VM 324 + 4 disks -> absent (VM 300 drill-r50 remains)
hub: 13 tables at 0, incl. BOTH escrow tables (demo-felhom/demo-hp/peti remain)
Storage Box u629488-sub4 -> gone (sub1/2/3 remain)
ep0 PBS ns finalwalk -> gone (demo-felhom, demo-hp remain)
WireGuard 10.77.0.5 -> gone from the LIVE wg show on ep0, not just
the hub DB (.2/.3/.4/.250 remain)
14.06 GiB reclaimed against 15 G measured before deletion.
R shredded with a planted-copy control: plant -> search finds both ->
shred -> the same search finds 0. The zero was not believed until the
instrument was proven.
R-244 (NEW): a FULL census after the cascade logged COMPLETE full teardown
found 61 rows still matching finalwalk. Four sources are deliberate
provenance; the fifth, app_log_issues (29 rows), is NOT covered by the
residue purge - and it is systematic: c11 40, rewalk 20, part4 24 still
present from the 2026-08-06 teardown, whose ledger recorded zero
occurrences. That claim used a narrower query than a census and does not
hold; the correction is recorded in both the prior ledger and the register
rather than the measurement quietly redone.
No secret material is involved. The table is a fleet-wide aggregate: 12 of
the 29 rows are finalwalk-only orphans, 17 are shared with LIVE customers
and must be de-referenced, not deleted - very likely why the leg was never
written. Not fixed; a cascade change needs its own red-proof.
Lesson, and it is the reusable part: a per-table absence query is not a
census.
125 lines
6.4 KiB
Markdown
125 lines
6.4 KiB
Markdown
# Teardown ledger — the campaign venues (2026-08-06)
|
|
|
|
Three throwaway appliances had accumulated, each with its own storage sub-account, hub record, tunnel
|
|
peer and off-site repository. **Two were destroyed here.** The third (`part4`, VM 323) was kept until
|
|
Parts 1 and 3 had banked their findings, and goes separately.
|
|
|
|
## What was matched, and how
|
|
|
|
Matched on **identity** — customer id → host id → off-site sub-account → wg peer — never on size. The
|
|
hub's own delete-preview (`GET /configs/<id>/delete`) was used as an independent enumeration and
|
|
agreed with the hand-built one in every field.
|
|
|
|
## The cascade is the hub's, not a hand-deletion
|
|
|
|
`POST /configs/<id>/delete` runs the designed teardown: leg 1 hosts (including
|
|
`DELETE FROM wg_peers WHERE host_id = ?`), leg 2 the committed reset with **external teardown FIRST**
|
|
(off-site store + PBS namespace destroyed), leg 3 residue, leg 4 the customer record and **all** escrow
|
|
ciphertext. Its own confirmation: „offsite tároló és PBS névtér megsemmisítve… Visszafordíthatatlan."
|
|
|
|
**Six gates, all of which had to be satisfied deliberately** — three acknowledgements
|
|
(`ack_hosts`/`ack_reset`/`ack_purge`), a type-to-confirm `confirm_id`, an `expect_hosts` race guard,
|
|
and — the one that shaped the whole operation — **the cascade refuses to delete a live host**:
|
|
|
|
> Delete refused: host c11-36d660 is ONLINE. Decommission the box first — the cascade never deletes a
|
|
> live host.
|
|
|
|
There is no hub decommission endpoint; the word appears only in that refusal. So both VMs were
|
|
**stopped** and the hub was allowed to age them past its `stale_threshold` (**30m**, from the deployed
|
|
`hub-config`) before the cascade would accept them. `rewalk` aged out at ~21 min of silence, `c11` at
|
|
~28 min — the two boxes' last reports had landed at different points in their cycles.
|
|
|
|
## Venue 1 — `c11` "Campaign 11"
|
|
|
|
| Layer | Item | After |
|
|
|---|---|---|
|
|
| Machine | `demo-hp` VM **321** `c11-appliance` (200 G + 50 G + 50 G + efidisk, **20 G on disk**) | `qm destroy 321 --purge` — absent from `qm list` |
|
|
| Hub host | `c11-36d660` | 0 occurrences |
|
|
| Hub customer | `c11` (residue 340 rows, 1 superseded blob) | 0 occurrences |
|
|
| Off-site | Storage Box sub-account **`u629488-sub4`** | 0 occurrences |
|
|
| Off-site | `ep0` PBS namespace **`c11`** | gone |
|
|
| Network | WireGuard peer **10.77.0.5** | 0 occurrences |
|
|
|
|
## Venue 2 — `rewalk` "Re-walk R-201"
|
|
|
|
| Layer | Item | After |
|
|
|---|---|---|
|
|
| Machine | `demo-hp` VM **322** `rewalk-appliance` (**17 G on disk**) | `qm destroy 322 --purge` — absent |
|
|
| Hub host | `rewalk-1ab77d` | 0 occurrences |
|
|
| Hub customer | `rewalk` (residue 93 rows) | 0 occurrences |
|
|
| Off-site | Storage Box sub-account **`u629488-sub5`** | 0 occurrences |
|
|
| Off-site | `ep0` PBS namespace **`rewalk`** | gone |
|
|
| Network | WireGuard peer **10.77.0.6** | 0 occurrences |
|
|
|
|
## Space returned, measured
|
|
|
|
`c11-scratch` used: **58,578,256 KiB → 21,228,640 KiB** = **37.3 GB reclaimed**, against 20 G + 17 G
|
|
measured before deletion. Free went 874.8 GB → 912.1 GB.
|
|
|
|
## The positive control
|
|
|
|
An absence proves nothing unless the instrument can still find something. After each cascade the same
|
|
greps were run for **`part4`**, which must survive: host `part4-905bdb` still present (2 rows), wg peer
|
|
**10.77.0.7** still present, `ep0` namespace `part4` still present. So the zero counts above are
|
|
measurements, not a broken query.
|
|
|
|
`ep0` namespaces after both teardowns: `demo-felhom`, `demo-hp`, `part4` — exactly the three that
|
|
should remain. Nothing on `ep0` outside the campaign's own namespaces was touched.
|
|
|
|
## Untouched, as scoped
|
|
|
|
`demo-hp` itself and its guest 9201, `demo-felhom`, `drill-r50` (VM 300), DooPlex, the tester's box,
|
|
and every `ep0` namespace that is not the campaign's own.
|
|
|
|
|
|
## Venue 3 — `part4` "Part 4 - no hand upgrade" (destroyed after the findings were written)
|
|
|
|
Kept deliberately while Parts 1 and 3 read it: it was standing in the exact failed half-B state, and
|
|
tearing it down first would have destroyed the evidence. Destroyed only after the diagnoses were
|
|
committed (`70b6260`).
|
|
|
|
| Layer | Item | After |
|
|
|---|---|---|
|
|
| Machine | `demo-hp` VM **323** `part4-appliance` (**16 G on disk**) | `qm destroy 323 --purge` — absent |
|
|
| Hub host | `part4-905bdb` | 0 occurrences |
|
|
| Hub customer | `part4` (residue 66 rows) | 0 occurrences |
|
|
| Off-site | Storage Box sub-account **`u629488-sub6`** | 0 occurrences |
|
|
| Off-site | `ep0` PBS namespace **`part4`** | gone |
|
|
| Network | WireGuard peer **10.77.0.7** | 0 occurrences |
|
|
|
|
Positive control for this one: **`demo-hp-bb76ea`** must survive the same greps, and does (2 rows).
|
|
|
|
## Final state
|
|
|
|
- `demo-hp` guests: **VM 300 `drill-r50`** and **CT 9201** (demo-hp's own customer guest). All three
|
|
campaign appliances are gone.
|
|
- `ep0` namespaces: **`demo-felhom`, `demo-hp`** — only the two real demo boxes remain.
|
|
- `c11-scratch` used: **58,578,256 KiB → 5,184,912 KiB = 53.4 GB reclaimed** across the three venues
|
|
(20 G + 17 G + 16 G measured beforehand). Free 874.8 GB → 928.2 GB.
|
|
|
|
## Secrets that died with the venues
|
|
|
|
Each cascade purged the customer record **and all escrow ciphertext** (leg 4). The recovery codes for
|
|
these venues therefore no longer open anything: `R_C11_REWALK` and `R_PART4` in the operator's
|
|
credentials file are now inert and can be removed at the operator's convenience. The working copies
|
|
under `~/.config/part4/` on DooPlex were shredded.
|
|
|
|
**One operational note carried over from the walk:** the `part4` retrieval passphrase was leaked into
|
|
a session transcript on 2026-08-06 and rotated at the time; the customer it belonged to no longer
|
|
exists.
|
|
|
|
---
|
|
|
|
## ⚠ CORRECTION, 2026-08-07 — the "0 occurrences" claims above are narrower than they read
|
|
|
|
Verifying the `finalwalk` teardown with a **full census** (every table, every column) rather than a
|
|
per-table query found rows this ledger's method could not see. `app_log_issues` is **not** covered by
|
|
the cascade's residue purge, and it still holds **`c11` 40 rows, `rewalk` 20, `part4` 24** — plus 29
|
|
for `finalwalk`.
|
|
|
|
The tables this ledger *did* check are genuinely at zero; the claim that is wrong is the implied
|
|
completeness of the sweep. **A per-table absence query is not a census.** Filed as **R-244**, with the
|
|
reason the leg is not trivial (the table is a fleet-wide aggregate whose rows are shared with live
|
|
customers, so the fix is a de-reference plus an orphan sweep, not a delete). No secret material is
|
|
involved. Full working: `teardown-finalwalk-2026-08-07.md`.
|