felhom.eu/documentation/tests/teardown-finalwalk-2026-08-07.md

Teardown ledger — finalwalk, the final-walk venue (2026-08-07)

The last campaign venue. Kept standing until the R-241 spike had read it and its findings were pushed (71c43f87c240) — it was the only machine standing in the failed state, and tearing it down first would have destroyed the evidence. Destroyed only after that, with the operator's confirmation.

Predecessor ledger: teardown-2026-08-06.md (venues c11, rewalk, part4).

What was matched, and how

Matched on identity — customer id → host id → off-site sub-account → PBS namespace → wg peer — never on size. On 2026-08-06 an item that measured exactly the expected size turned out to be a working store, which is why size is not a key here.

The hub's own delete-preview (GET /configs/finalwalk/delete) was used as an independent enumeration and agreed with the hand-built one in every field:

{"customer_id":"finalwalk","customer_name":"Final walk R-201","host_count":1,
 "hosts":[{"host_id":"finalwalk-ed05d6","online":true,"status":"ok"}],
 "offsite_enabled":true,"offsite_identifier":"u629488-sub4","offsite_type":"shared",
 "pbs_tenancy_configured":true,"claim_present":true,"dr_recipe_present":true,
 "one_time_secret":true,"superseded_blobs":0,"residue_total":124}

superseded_blobs: 0 matches the hand query on host_escrow_superseded (no rows) — consistent with the spike's finding that the box never re-escrowed, which is exactly why its key never matched.

The enumeration, before anything was deleted

Layer	Item	Identified by
machine	demo-hp VM 324 finalwalk-appliance — 4 disks (efidisk + 200 G + 50 G + 50 G), 15 G actual on /mnt/nvme-1tb/images/324	qm config 324 → name: finalwalk-appliance
hub	host finalwalk-ed05d6, customer finalwalk — 221 rows across 19 tables (full census, no truncation)	census below
off-site	Storage Box sub-account u629488-sub4, home felhom-finalwalk, id 285071	Hetzner API, matched on the home directory naming the customer
off-site	ep0 PBS namespace finalwalk in datastore felhom-offsite — 1 CT snapshot (ct/9201/2026-08-06T22:17:08Z), 88 K of metadata	ls /mnt/pbs-datastore/ns
network	WireGuard peer 10.77.0.5	wg_peers.host_id = finalwalk-ed05d6

Full hub-DB census — every table, every column, matching finalwalk (221 rows / 19 tables): app_log_issues 29 · app_telemetry 70 · appliance_registrations 1 · customer_claims 1 · customer_configs 1 · customer_notifications 1 · dr_recipe 1 · events 14 · guests 1 · host_escrow 1 · host_pbs_secrets 1 · host_recovery 1 · host_reports 37 · hosts 1 · notification_log 13 · one_time_secrets 1 · reports 45 · selfbind_tokens 1 · wg_peers 1.

pvesm status on demo-hp BEFORE:

c11-scratch    dir  active   983379700 KiB total   21010360 KiB used   912342728 KiB avail   2.14%

A note on the ep0 datastore path. The live datastore is /mnt/pbs-datastore (the 100 GB volume). /srv/pbs-felhom still exists on the root filesystem and holds stale namespaces (demo-felhom, demo-felhom-01, demo-hp) from before the volume move. It is not the live store and was not touched. Reading namespaces from it would have produced a wrong answer in both directions — recorded here because the first listing this session did exactly that.

The gate that shaped the operation

The cascade refuses to delete a live host — there is no hub decommission endpoint; the word appears only in the refusal. So VM 324 was stopped at 07:23:56Z (guarded on qm config 324 reading name: finalwalk-appliance first — demo-hp also carries a guest 9201) and the hub was allowed to age it past its stale_threshold (30m, read from the deployed hub-config, not assumed) before the cascade would accept it.

---

The cascade

POST /configs/finalwalk/delete with the six gates satisfied deliberately — ack_hosts, ack_reset, ack_purge, confirm_id=finalwalk (typed), expect_hosts=1 (the stale-preview race guard), and the online-host gate lifted by ageing. → 303 /configs?flash=deleted.

The hub's own leg-by-leg log, which is the evidence — not the flash:

09:54:02  customer DELETE cascade started for finalwalk (journal #14, 1 host(s))
09:54:02  delete finalwalk: host finalwalk-ed05d6 deleted (escrow DEMOTED to retained custody)
09:54:10  [offsite] deprovisioned shared sub-account 285071 for finalwalk (repo data destroyed)
09:54:10  reset finalwalk: offsite deprovisioned (repo data destroyed)
09:54:10  tenantsync: deprovision ok for finalwalk (ns=finalwalk, existed=true)
09:54:10  reset finalwalk: PBS tenancy deprovisioned
09:54:10  [claim] reset to unclaimed for finalwalk (customer RESET)
09:54:11  delete finalwalk: residue purged (reports=48 app_telemetry=73 app_log_tails=0
          log_tail_requests=0 notif_prefs=1 selfbind_tokens=1 appliance_registrations=1)
09:54:11  customer DELETE cascade COMPLETE for finalwalk (journal #14) — full teardown

existed=true on the PBS leg is the positive observable that it deleted something rather than no-op'ing on an absent namespace. The residue counts sum to 124, matching the preview exactly.

Then, guarded a second time on qm config 324 reading name: finalwalk-appliance: qm destroy 324 --purge → absent from qm list, and /mnt/nvme-1tb/images/ is now empty.

After — every layer, each with a positive control

Layer	Item	After	Positive control (must survive, and does)
machine	VM 324 + 4 disks	absent from qm list; image dir empty	VM 300 drill-r50 still listed
hub	hosts, customer_configs, host_escrow, host_escrow_superseded, host_recovery, host_pbs_secrets, one_time_secrets, dr_recipe, customer_claims, guests, reports, host_reports, wg_peers	0 rows each	demo-felhom 50,823 · demo-hp 7,640 · peti 1,827 rows still present
off-site	Storage Box sub-account u629488-sub4 (id 285071)	gone — 3 sub-accounts remain	u629488-sub1/2/3 (demo-felhom, peti-felhom, demo-hp) all present
off-site	ep0 PBS namespace finalwalk	gone	namespaces demo-felhom, demo-hp both present
network	WireGuard peer 10.77.0.5	gone from the LIVE wg show on ep0, not merely from the hub DB	peers 10.77.0.2, .3, .4, .250 all still on wg0

Space returned, measured. c11-scratch used 21,010,360 → 6,262,060 KiB = 14,748,300 KiB (14.06 GiB) reclaimed, against the 15 G measured on disk before deletion. Available went 912,342,728 → 927,091,028 KiB.

demo-hp's own guest 9201 is still running — the host itself was untouched.

⚠ WHAT IS NOT GONE, and it is not what the previous ledger implied

A full census (every table, every column) after the cascade returns 61 rows still matching finalwalk, not 0. Four of the five sources are deliberate and correct — the cascade's own header says "Provenance/events are NEVER wiped — audit outlives every tier":

table	rows	verdict
events	16	by design — audit
notification_log	14	by design — audit
host_deletions	1	by design — the deletion tombstone
customer_resets	1	by design — the journal row (#14, completed)
app_log_issues	29	A GAP — see below

app_log_issues is not covered by the residue purge, and it is systematic, not a finalwalk quirk. Rows still reference every venue ever torn down: finalwalk 29, c11 40, rewalk 20, part4 24 — the last three from the 2026-08-06 teardown, whose ledger recorded "0 occurrences" for those customers. That claim was measured with a narrower query than the one used here and does not hold under a full census. Recording the correction rather than quietly re-measuring.

Why it was probably never written, established rather than assumed: the table is a fleet-wide aggregate, keyed on app_name+fingerprint with an affected_customers JSON list. Of the 29 rows, 12 reference only finalwalk (orphans, safely deletable) and 17 are shared with live customers (demo-felhom, peti-felhom, …) and must not be deleted — only de-referenced. A naive DELETE … WHERE customer LIKE would destroy live customers' issue history, so the fix is a de-reference plus an orphan sweep, not a delete.

It carries no secret material — app name, fingerprint, message text, counts and timestamps. What survives is the deleted customer's identifier inside an aggregate. Filed as R-244; not fixed here, because this session's scope is a spike and two operations, and a cascade change needs its own red-proof.

Side effect, recorded

Stopping the box to age it past the staleness gate fired a genuine operator alert at 07:43:11Z — Staleness: finalwalk ok → stale (node_stale) and Operator email sent for finalwalk/node_stale. Expected, and correct behaviour: the hub cannot know a stop was deliberate. Noted so the mail is not later read as an incident.

R, and the other session credentials

Shredded, with a planted-copy control — because a sweep that finds nothing proves nothing until the instrument is shown to work:

1. A copy of R was planted at a second path.
2. The search found both the real R (~/.config/finalwalk/R_finalwalk.txt) and the plant — the instrument works.
3. Both were shred -u'd, along with the venue's other session credentials (appliance root, dashboard passwords, claim code, retrieval password) and the working copy of the appliance root password. ~/.config/finalwalk/ was removed.
4. The same search then returned 0 hits.

On 2026-08-05 a sweep pointed at a path that did not exist and its zero meant nothing; on 2026-08-06 a control found a copy nobody had created. This one was proven in both directions before the zero was believed.

Untouched, as scoped

demo-hp itself and its guest 9201, demo-felhom, drill-r50 (VM 300), DooPlex, the parked tester's box (peti-felhom / u629488-sub2), and every ep0 namespace that is not the venue's own. The stale pre-volume-move tree at /srv/pbs-felhom on ep0 was read and left alone — it is not the live datastore and contains no finalwalk namespace.