All five layers gone, each verified with a positive control that must
survive and does:
VM 324 + 4 disks -> absent (VM 300 drill-r50 remains)
hub: 13 tables at 0, incl. BOTH escrow tables (demo-felhom/demo-hp/peti remain)
Storage Box u629488-sub4 -> gone (sub1/2/3 remain)
ep0 PBS ns finalwalk -> gone (demo-felhom, demo-hp remain)
WireGuard 10.77.0.5 -> gone from the LIVE wg show on ep0, not just
the hub DB (.2/.3/.4/.250 remain)
14.06 GiB reclaimed against 15 G measured before deletion.
R shredded with a planted-copy control: plant -> search finds both ->
shred -> the same search finds 0. The zero was not believed until the
instrument was proven.
R-244 (NEW): a FULL census after the cascade logged COMPLETE full teardown
found 61 rows still matching finalwalk. Four sources are deliberate
provenance; the fifth, app_log_issues (29 rows), is NOT covered by the
residue purge - and it is systematic: c11 40, rewalk 20, part4 24 still
present from the 2026-08-06 teardown, whose ledger recorded zero
occurrences. That claim used a narrower query than a census and does not
hold; the correction is recorded in both the prior ledger and the register
rather than the measurement quietly redone.
No secret material is involved. The table is a fleet-wide aggregate: 12 of
the 29 rows are finalwalk-only orphans, 17 are shared with LIVE customers
and must be de-referenced, not deleted - very likely why the leg was never
written. Not fixed; a cascade change needs its own red-proof.
Lesson, and it is the reusable part: a per-table absence query is not a
census.
This commit is contained in:
@@ -107,3 +107,18 @@ under `~/.config/part4/` on DooPlex were shredded.
|
||||
**One operational note carried over from the walk:** the `part4` retrieval passphrase was leaked into
|
||||
a session transcript on 2026-08-06 and rotated at the time; the customer it belonged to no longer
|
||||
exists.
|
||||
|
||||
---
|
||||
|
||||
## ⚠ CORRECTION, 2026-08-07 — the "0 occurrences" claims above are narrower than they read
|
||||
|
||||
Verifying the `finalwalk` teardown with a **full census** (every table, every column) rather than a
|
||||
per-table query found rows this ledger's method could not see. `app_log_issues` is **not** covered by
|
||||
the cascade's residue purge, and it still holds **`c11` 40 rows, `rewalk` 20, `part4` 24** — plus 29
|
||||
for `finalwalk`.
|
||||
|
||||
The tables this ledger *did* check are genuinely at zero; the claim that is wrong is the implied
|
||||
completeness of the sweep. **A per-table absence query is not a census.** Filed as **R-244**, with the
|
||||
reason the leg is not trivial (the table is a fleet-wide aggregate whose rows are shared with live
|
||||
customers, so the fix is a de-reference plus an orphan sweep, not a delete). No secret material is
|
||||
involved. Full working: `teardown-finalwalk-2026-08-07.md`.
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
# Teardown ledger — `finalwalk`, the final-walk venue (2026-08-07)
|
||||
|
||||
The last campaign venue. **Kept standing until the R-241 spike had read it and its findings were
|
||||
pushed** (`71c43f87c240`) — it was the only machine standing in the failed state, and tearing it down
|
||||
first would have destroyed the evidence. Destroyed only after that, with the operator's confirmation.
|
||||
|
||||
Predecessor ledger: `teardown-2026-08-06.md` (venues `c11`, `rewalk`, `part4`).
|
||||
|
||||
## What was matched, and how
|
||||
|
||||
Matched on **identity** — customer id → host id → off-site sub-account → PBS namespace → wg peer —
|
||||
**never on size**. On 2026-08-06 an item that measured exactly the expected size turned out to be a
|
||||
working store, which is why size is not a key here.
|
||||
|
||||
The hub's own delete-preview (`GET /configs/finalwalk/delete`) was used as an **independent
|
||||
enumeration** and agreed with the hand-built one in every field:
|
||||
|
||||
```json
|
||||
{"customer_id":"finalwalk","customer_name":"Final walk R-201","host_count":1,
|
||||
"hosts":[{"host_id":"finalwalk-ed05d6","online":true,"status":"ok"}],
|
||||
"offsite_enabled":true,"offsite_identifier":"u629488-sub4","offsite_type":"shared",
|
||||
"pbs_tenancy_configured":true,"claim_present":true,"dr_recipe_present":true,
|
||||
"one_time_secret":true,"superseded_blobs":0,"residue_total":124}
|
||||
```
|
||||
|
||||
`superseded_blobs: 0` matches the hand query on `host_escrow_superseded` (no rows) — consistent with
|
||||
the spike's finding that the box never re-escrowed, which is exactly why its key never matched.
|
||||
|
||||
## The enumeration, before anything was deleted
|
||||
|
||||
| Layer | Item | Identified by |
|
||||
|---|---|---|
|
||||
| **machine** | `demo-hp` VM **324** `finalwalk-appliance` — 4 disks (efidisk + 200 G + 50 G + 50 G), **15 G actual** on `/mnt/nvme-1tb/images/324` | `qm config 324` → `name: finalwalk-appliance` |
|
||||
| **hub** | host `finalwalk-ed05d6`, customer `finalwalk` — **221 rows across 19 tables** (full census, no truncation) | census below |
|
||||
| **off-site** | Storage Box sub-account **`u629488-sub4`**, home `felhom-finalwalk`, id `285071` | Hetzner API, matched on the **home directory naming the customer** |
|
||||
| **off-site** | `ep0` PBS namespace **`finalwalk`** in datastore `felhom-offsite` — 1 CT snapshot (`ct/9201/2026-08-06T22:17:08Z`), 88 K of metadata | `ls /mnt/pbs-datastore/ns` |
|
||||
| **network** | WireGuard peer **10.77.0.5** | `wg_peers.host_id = finalwalk-ed05d6` |
|
||||
|
||||
**Full hub-DB census — every table, every column, matching `finalwalk`** (221 rows / 19 tables):
|
||||
`app_log_issues` 29 · `app_telemetry` 70 · `appliance_registrations` 1 · `customer_claims` 1 ·
|
||||
`customer_configs` 1 · `customer_notifications` 1 · `dr_recipe` 1 · `events` 14 · `guests` 1 ·
|
||||
`host_escrow` 1 · `host_pbs_secrets` 1 · `host_recovery` 1 · `host_reports` 37 · `hosts` 1 ·
|
||||
`notification_log` 13 · `one_time_secrets` 1 · `reports` 45 · `selfbind_tokens` 1 · `wg_peers` 1.
|
||||
|
||||
**`pvesm status` on `demo-hp` BEFORE:**
|
||||
|
||||
```
|
||||
c11-scratch dir active 983379700 KiB total 21010360 KiB used 912342728 KiB avail 2.14%
|
||||
```
|
||||
|
||||
**A note on the `ep0` datastore path.** The live datastore is `/mnt/pbs-datastore` (the 100 GB volume).
|
||||
`/srv/pbs-felhom` still exists on the root filesystem and holds **stale** namespaces
|
||||
(`demo-felhom`, `demo-felhom-01`, `demo-hp`) from before the volume move. It is **not** the live store
|
||||
and was not touched. Reading namespaces from it would have produced a wrong answer in both directions —
|
||||
recorded here because the first listing this session did exactly that.
|
||||
|
||||
## The gate that shaped the operation
|
||||
|
||||
The cascade **refuses to delete a live host** — there is no hub decommission endpoint; the word appears
|
||||
only in the refusal. So VM 324 was **stopped** at **07:23:56Z** (guarded on `qm config 324` reading
|
||||
`name: finalwalk-appliance` first — `demo-hp` also carries a guest 9201) and the hub was allowed to age
|
||||
it past its `stale_threshold` (**30m**, read from the deployed `hub-config`, not assumed) before the
|
||||
cascade would accept it.
|
||||
|
||||
---
|
||||
|
||||
## The cascade
|
||||
|
||||
`POST /configs/finalwalk/delete` with the six gates satisfied deliberately — `ack_hosts`,
|
||||
`ack_reset`, `ack_purge`, `confirm_id=finalwalk` (typed), `expect_hosts=1` (the stale-preview race
|
||||
guard), and the online-host gate lifted by ageing. → **`303 /configs?flash=deleted`**.
|
||||
|
||||
**The hub's own leg-by-leg log, which is the evidence — not the flash:**
|
||||
|
||||
```
|
||||
09:54:02 customer DELETE cascade started for finalwalk (journal #14, 1 host(s))
|
||||
09:54:02 delete finalwalk: host finalwalk-ed05d6 deleted (escrow DEMOTED to retained custody)
|
||||
09:54:10 [offsite] deprovisioned shared sub-account 285071 for finalwalk (repo data destroyed)
|
||||
09:54:10 reset finalwalk: offsite deprovisioned (repo data destroyed)
|
||||
09:54:10 tenantsync: deprovision ok for finalwalk (ns=finalwalk, existed=true)
|
||||
09:54:10 reset finalwalk: PBS tenancy deprovisioned
|
||||
09:54:10 [claim] reset to unclaimed for finalwalk (customer RESET)
|
||||
09:54:11 delete finalwalk: residue purged (reports=48 app_telemetry=73 app_log_tails=0
|
||||
log_tail_requests=0 notif_prefs=1 selfbind_tokens=1 appliance_registrations=1)
|
||||
09:54:11 customer DELETE cascade COMPLETE for finalwalk (journal #14) — full teardown
|
||||
```
|
||||
|
||||
`existed=true` on the PBS leg is the positive observable that it deleted something rather than
|
||||
no-op'ing on an absent namespace. The residue counts sum to **124**, matching the preview exactly.
|
||||
|
||||
Then, guarded a second time on `qm config 324` reading `name: finalwalk-appliance`:
|
||||
`qm destroy 324 --purge` → absent from `qm list`, and `/mnt/nvme-1tb/images/` is now empty.
|
||||
|
||||
## After — every layer, each with a positive control
|
||||
|
||||
| Layer | Item | After | Positive control (must survive, and does) |
|
||||
|---|---|---|---|
|
||||
| machine | VM 324 + 4 disks | **absent** from `qm list`; image dir empty | VM **300** `drill-r50` still listed |
|
||||
| hub | `hosts`, `customer_configs`, `host_escrow`, `host_escrow_superseded`, `host_recovery`, `host_pbs_secrets`, `one_time_secrets`, `dr_recipe`, `customer_claims`, `guests`, `reports`, `host_reports`, `wg_peers` | **0 rows each** | `demo-felhom` 50,823 · `demo-hp` 7,640 · `peti` 1,827 rows still present |
|
||||
| off-site | Storage Box sub-account `u629488-sub4` (id 285071) | **gone** — 3 sub-accounts remain | `u629488-sub1/2/3` (demo-felhom, peti-felhom, demo-hp) all present |
|
||||
| off-site | `ep0` PBS namespace `finalwalk` | **gone** | namespaces `demo-felhom`, `demo-hp` both present |
|
||||
| network | WireGuard peer `10.77.0.5` | **gone from the LIVE `wg show` on ep0**, not merely from the hub DB | peers `10.77.0.2`, `.3`, `.4`, `.250` all still on `wg0` |
|
||||
|
||||
**Space returned, measured.** `c11-scratch` used **21,010,360 → 6,262,060 KiB = 14,748,300 KiB
|
||||
(14.06 GiB)** reclaimed, against the 15 G measured on disk before deletion. Available went
|
||||
912,342,728 → 927,091,028 KiB.
|
||||
|
||||
**`demo-hp`'s own guest 9201 is still running** — the host itself was untouched.
|
||||
|
||||
## ⚠ WHAT IS **NOT** GONE, and it is not what the previous ledger implied
|
||||
|
||||
A full census (every table, every column) after the cascade returns **61 rows still matching
|
||||
`finalwalk`**, not 0. Four of the five sources are **deliberate and correct** — the cascade's own
|
||||
header says *"Provenance/events are NEVER wiped — audit outlives every tier"*:
|
||||
|
||||
| table | rows | verdict |
|
||||
|---|---|---|
|
||||
| `events` | 16 | **by design** — audit |
|
||||
| `notification_log` | 14 | **by design** — audit |
|
||||
| `host_deletions` | 1 | **by design** — the deletion tombstone |
|
||||
| `customer_resets` | 1 | **by design** — the journal row (`#14`, completed) |
|
||||
| **`app_log_issues`** | **29** | **A GAP — see below** |
|
||||
|
||||
**`app_log_issues` is not covered by the residue purge, and it is systematic, not a finalwalk
|
||||
quirk.** Rows still reference every venue ever torn down: **finalwalk 29, `c11` 40, `rewalk` 20,
|
||||
`part4` 24** — the last three from the 2026-08-06 teardown, whose ledger recorded "0 occurrences"
|
||||
for those customers. **That claim was measured with a narrower query than the one used here and does
|
||||
not hold under a full census.** Recording the correction rather than quietly re-measuring.
|
||||
|
||||
**Why it was probably never written, established rather than assumed:** the table is a **fleet-wide
|
||||
aggregate**, keyed on `app_name`+`fingerprint` with an `affected_customers` JSON list. Of the 29 rows,
|
||||
**12 reference only `finalwalk`** (orphans, safely deletable) and **17 are shared with live
|
||||
customers** (`demo-felhom`, `peti-felhom`, …) and **must not be deleted — only de-referenced**. A
|
||||
naive `DELETE … WHERE customer LIKE` would destroy live customers' issue history, so the fix is a
|
||||
de-reference plus an orphan sweep, not a delete.
|
||||
|
||||
**It carries no secret material** — app name, fingerprint, message text, counts and timestamps. What
|
||||
survives is the deleted customer's *identifier* inside an aggregate. Filed as **R-244**; **not fixed
|
||||
here**, because this session's scope is a spike and two operations, and a cascade change needs its own
|
||||
red-proof.
|
||||
|
||||
## Side effect, recorded
|
||||
|
||||
Stopping the box to age it past the staleness gate fired a genuine operator alert at 07:43:11Z —
|
||||
`Staleness: finalwalk ok → stale (node_stale)` and `Operator email sent for finalwalk/node_stale`.
|
||||
Expected, and correct behaviour: the hub cannot know a stop was deliberate. Noted so the mail is not
|
||||
later read as an incident.
|
||||
|
||||
## R, and the other session credentials
|
||||
|
||||
**Shredded, with a planted-copy control** — because a sweep that finds nothing proves nothing until
|
||||
the instrument is shown to work:
|
||||
|
||||
1. A copy of `R` was **planted** at a second path.
|
||||
2. The search found **both** the real `R` (`~/.config/finalwalk/R_finalwalk.txt`) and the plant —
|
||||
the instrument works.
|
||||
3. Both were `shred -u`'d, along with the venue's other session credentials (appliance root, dashboard
|
||||
passwords, claim code, retrieval password) and the working copy of the appliance root password.
|
||||
`~/.config/finalwalk/` was removed.
|
||||
4. The **same** search then returned **0 hits**.
|
||||
|
||||
On 2026-08-05 a sweep pointed at a path that did not exist and its zero meant nothing; on 2026-08-06 a
|
||||
control found a copy nobody had created. This one was proven in both directions before the zero was
|
||||
believed.
|
||||
|
||||
## Untouched, as scoped
|
||||
|
||||
`demo-hp` itself and its guest 9201, `demo-felhom`, `drill-r50` (VM 300), DooPlex, **the parked
|
||||
tester's box (`peti-felhom` / `u629488-sub2`)**, and every `ep0` namespace that is not the venue's own.
|
||||
The stale pre-volume-move tree at `/srv/pbs-felhom` on ep0 was read and left alone — it is not the live
|
||||
datastore and contains no `finalwalk` namespace.
|
||||
Reference in New Issue
Block a user