# Teardown ledger — `finalwalk`, the final-walk venue (2026-08-07)

The last campaign venue. **Kept standing until the R-241 spike had read it and its findings were
pushed** (`71c43f87c240`) — it was the only machine standing in the failed state, and tearing it down
first would have destroyed the evidence. Destroyed only after that, with the operator's confirmation.

Predecessor ledger: `teardown-2026-08-06.md` (venues `c11`, `rewalk`, `part4`).

## What was matched, and how

Matched on **identity** — customer id → host id → off-site sub-account → PBS namespace → wg peer —
**never on size**. On 2026-08-06 an item that measured exactly the expected size turned out to be a
working store, which is why size is not a key here.

The hub's own delete-preview (`GET /configs/finalwalk/delete`) was used as an **independent
enumeration** and agreed with the hand-built one in every field:

```json
{"customer_id":"finalwalk","customer_name":"Final walk R-201","host_count":1,
 "hosts":[{"host_id":"finalwalk-ed05d6","online":true,"status":"ok"}],
 "offsite_enabled":true,"offsite_identifier":"u629488-sub4","offsite_type":"shared",
 "pbs_tenancy_configured":true,"claim_present":true,"dr_recipe_present":true,
 "one_time_secret":true,"superseded_blobs":0,"residue_total":124}
```

`superseded_blobs: 0` matches the hand query on `host_escrow_superseded` (no rows) — consistent with
the spike's finding that the box never re-escrowed, which is exactly why its key never matched.

## The enumeration, before anything was deleted

| Layer | Item | Identified by |
|---|---|---|
| **machine** | `demo-hp` VM **324** `finalwalk-appliance` — 4 disks (efidisk + 200 G + 50 G + 50 G), **15 G actual** on `/mnt/nvme-1tb/images/324` | `qm config 324` → `name: finalwalk-appliance` |
| **hub** | host `finalwalk-ed05d6`, customer `finalwalk` — **221 rows across 19 tables** (full census, no truncation) | census below |
| **off-site** | Storage Box sub-account **`u629488-sub4`**, home `felhom-finalwalk`, id `285071` | Hetzner API, matched on the **home directory naming the customer** |
| **off-site** | `ep0` PBS namespace **`finalwalk`** in datastore `felhom-offsite` — 1 CT snapshot (`ct/9201/2026-08-06T22:17:08Z`), 88 K of metadata | `ls /mnt/pbs-datastore/ns` |
| **network** | WireGuard peer **10.77.0.5** | `wg_peers.host_id = finalwalk-ed05d6` |

**Full hub-DB census — every table, every column, matching `finalwalk`** (221 rows / 19 tables):
`app_log_issues` 29 · `app_telemetry` 70 · `appliance_registrations` 1 · `customer_claims` 1 ·
`customer_configs` 1 · `customer_notifications` 1 · `dr_recipe` 1 · `events` 14 · `guests` 1 ·
`host_escrow` 1 · `host_pbs_secrets` 1 · `host_recovery` 1 · `host_reports` 37 · `hosts` 1 ·
`notification_log` 13 · `one_time_secrets` 1 · `reports` 45 · `selfbind_tokens` 1 · `wg_peers` 1.

**`pvesm status` on `demo-hp` BEFORE:**

```
c11-scratch    dir  active   983379700 KiB total   21010360 KiB used   912342728 KiB avail   2.14%
```

**A note on the `ep0` datastore path.** The live datastore is `/mnt/pbs-datastore` (the 100 GB volume).
`/srv/pbs-felhom` still exists on the root filesystem and holds **stale** namespaces
(`demo-felhom`, `demo-felhom-01`, `demo-hp`) from before the volume move. It is **not** the live store
and was not touched. Reading namespaces from it would have produced a wrong answer in both directions —
recorded here because the first listing this session did exactly that.

## The gate that shaped the operation

The cascade **refuses to delete a live host** — there is no hub decommission endpoint; the word appears
only in the refusal. So VM 324 was **stopped** at **07:23:56Z** (guarded on `qm config 324` reading
`name: finalwalk-appliance` first — `demo-hp` also carries a guest 9201) and the hub was allowed to age
it past its `stale_threshold` (**30m**, read from the deployed `hub-config`, not assumed) before the
cascade would accept it.

---

## The cascade

`POST /configs/finalwalk/delete` with the six gates satisfied deliberately — `ack_hosts`,
`ack_reset`, `ack_purge`, `confirm_id=finalwalk` (typed), `expect_hosts=1` (the stale-preview race
guard), and the online-host gate lifted by ageing. → **`303 /configs?flash=deleted`**.

**The hub's own leg-by-leg log, which is the evidence — not the flash:**

```
09:54:02  customer DELETE cascade started for finalwalk (journal #14, 1 host(s))
09:54:02  delete finalwalk: host finalwalk-ed05d6 deleted (escrow DEMOTED to retained custody)
09:54:10  [offsite] deprovisioned shared sub-account 285071 for finalwalk (repo data destroyed)
09:54:10  reset finalwalk: offsite deprovisioned (repo data destroyed)
09:54:10  tenantsync: deprovision ok for finalwalk (ns=finalwalk, existed=true)
09:54:10  reset finalwalk: PBS tenancy deprovisioned
09:54:10  [claim] reset to unclaimed for finalwalk (customer RESET)
09:54:11  delete finalwalk: residue purged (reports=48 app_telemetry=73 app_log_tails=0
          log_tail_requests=0 notif_prefs=1 selfbind_tokens=1 appliance_registrations=1)
09:54:11  customer DELETE cascade COMPLETE for finalwalk (journal #14) — full teardown
```

`existed=true` on the PBS leg is the positive observable that it deleted something rather than
no-op'ing on an absent namespace. The residue counts sum to **124**, matching the preview exactly.

Then, guarded a second time on `qm config 324` reading `name: finalwalk-appliance`:
`qm destroy 324 --purge` → absent from `qm list`, and `/mnt/nvme-1tb/images/` is now empty.

## After — every layer, each with a positive control

| Layer | Item | After | Positive control (must survive, and does) |
|---|---|---|---|
| machine | VM 324 + 4 disks | **absent** from `qm list`; image dir empty | VM **300** `drill-r50` still listed |
| hub | `hosts`, `customer_configs`, `host_escrow`, `host_escrow_superseded`, `host_recovery`, `host_pbs_secrets`, `one_time_secrets`, `dr_recipe`, `customer_claims`, `guests`, `reports`, `host_reports`, `wg_peers` | **0 rows each** | `demo-felhom` 50,823 · `demo-hp` 7,640 · `peti` 1,827 rows still present |
| off-site | Storage Box sub-account `u629488-sub4` (id 285071) | **gone** — 3 sub-accounts remain | `u629488-sub1/2/3` (demo-felhom, peti-felhom, demo-hp) all present |
| off-site | `ep0` PBS namespace `finalwalk` | **gone** | namespaces `demo-felhom`, `demo-hp` both present |
| network | WireGuard peer `10.77.0.5` | **gone from the LIVE `wg show` on ep0**, not merely from the hub DB | peers `10.77.0.2`, `.3`, `.4`, `.250` all still on `wg0` |

**Space returned, measured.** `c11-scratch` used **21,010,360 → 6,262,060 KiB = 14,748,300 KiB
(14.06 GiB)** reclaimed, against the 15 G measured on disk before deletion. Available went
912,342,728 → 927,091,028 KiB.

**`demo-hp`'s own guest 9201 is still running** — the host itself was untouched.

## ⚠ WHAT IS **NOT** GONE, and it is not what the previous ledger implied

A full census (every table, every column) after the cascade returns **61 rows still matching
`finalwalk`**, not 0. Four of the five sources are **deliberate and correct** — the cascade's own
header says *"Provenance/events are NEVER wiped — audit outlives every tier"*:

| table | rows | verdict |
|---|---|---|
| `events` | 16 | **by design** — audit |
| `notification_log` | 14 | **by design** — audit |
| `host_deletions` | 1 | **by design** — the deletion tombstone |
| `customer_resets` | 1 | **by design** — the journal row (`#14`, completed) |
| **`app_log_issues`** | **29** | **A GAP — see below** |

**`app_log_issues` is not covered by the residue purge, and it is systematic, not a finalwalk
quirk.** Rows still reference every venue ever torn down: **finalwalk 29, `c11` 40, `rewalk` 20,
`part4` 24** — the last three from the 2026-08-06 teardown, whose ledger recorded "0 occurrences"
for those customers. **That claim was measured with a narrower query than the one used here and does
not hold under a full census.** Recording the correction rather than quietly re-measuring.

**Why it was probably never written, established rather than assumed:** the table is a **fleet-wide
aggregate**, keyed on `app_name`+`fingerprint` with an `affected_customers` JSON list. Of the 29 rows,
**12 reference only `finalwalk`** (orphans, safely deletable) and **17 are shared with live
customers** (`demo-felhom`, `peti-felhom`, …) and **must not be deleted — only de-referenced**. A
naive `DELETE … WHERE customer LIKE` would destroy live customers' issue history, so the fix is a
de-reference plus an orphan sweep, not a delete.

**It carries no secret material** — app name, fingerprint, message text, counts and timestamps. What
survives is the deleted customer's *identifier* inside an aggregate. Filed as **R-244**; **not fixed
here**, because this session's scope is a spike and two operations, and a cascade change needs its own
red-proof.

## Side effect, recorded

Stopping the box to age it past the staleness gate fired a genuine operator alert at 07:43:11Z —
`Staleness: finalwalk ok → stale (node_stale)` and `Operator email sent for finalwalk/node_stale`.
Expected, and correct behaviour: the hub cannot know a stop was deliberate. Noted so the mail is not
later read as an incident.

## R, and the other session credentials

**Shredded, with a planted-copy control** — because a sweep that finds nothing proves nothing until
the instrument is shown to work:

1. A copy of `R` was **planted** at a second path.
2. The search found **both** the real `R` (`~/.config/finalwalk/R_finalwalk.txt`) and the plant —
   the instrument works.
3. Both were `shred -u`'d, along with the venue's other session credentials (appliance root, dashboard
   passwords, claim code, retrieval password) and the working copy of the appliance root password.
   `~/.config/finalwalk/` was removed.
4. The **same** search then returned **0 hits**.

On 2026-08-05 a sweep pointed at a path that did not exist and its zero meant nothing; on 2026-08-06 a
control found a copy nobody had created. This one was proven in both directions before the zero was
believed.

## Untouched, as scoped

`demo-hp` itself and its guest 9201, `demo-felhom`, `drill-r50` (VM 300), DooPlex, **the parked
tester's box (`peti-felhom` / `u629488-sub2`)**, and every `ep0` namespace that is not the venue's own.
The stale pre-volume-move tree at `/srv/pbs-felhom` on ep0 was read and left alone — it is not the live
datastore and contains no `finalwalk` namespace.