Files
felhom.eu/documentation/tests/teardown-2026-08-06.md
T
admin 094e93e828
gates / gates (push) Successful in 13s
finalwalk teardown complete; R-244 filed; session report
All five layers gone, each verified with a positive control that must
survive and does:
  VM 324 + 4 disks   -> absent (VM 300 drill-r50 remains)
  hub: 13 tables at 0, incl. BOTH escrow tables (demo-felhom/demo-hp/peti remain)
  Storage Box u629488-sub4 -> gone (sub1/2/3 remain)
  ep0 PBS ns finalwalk     -> gone (demo-felhom, demo-hp remain)
  WireGuard 10.77.0.5      -> gone from the LIVE wg show on ep0, not just
                              the hub DB (.2/.3/.4/.250 remain)
14.06 GiB reclaimed against 15 G measured before deletion.

R shredded with a planted-copy control: plant -> search finds both ->
shred -> the same search finds 0. The zero was not believed until the
instrument was proven.

R-244 (NEW): a FULL census after the cascade logged COMPLETE full teardown
found 61 rows still matching finalwalk. Four sources are deliberate
provenance; the fifth, app_log_issues (29 rows), is NOT covered by the
residue purge - and it is systematic: c11 40, rewalk 20, part4 24 still
present from the 2026-08-06 teardown, whose ledger recorded zero
occurrences. That claim used a narrower query than a census and does not
hold; the correction is recorded in both the prior ledger and the register
rather than the measurement quietly redone.

No secret material is involved. The table is a fleet-wide aggregate: 12 of
the 29 rows are finalwalk-only orphans, 17 are shared with LIVE customers
and must be de-referenced, not deleted - very likely why the leg was never
written. Not fixed; a cascade change needs its own red-proof.

Lesson, and it is the reusable part: a per-table absence query is not a
census.
2026-08-07 09:59:16 +02:00

6.4 KiB

Teardown ledger — the campaign venues (2026-08-06)

Three throwaway appliances had accumulated, each with its own storage sub-account, hub record, tunnel peer and off-site repository. Two were destroyed here. The third (part4, VM 323) was kept until Parts 1 and 3 had banked their findings, and goes separately.

What was matched, and how

Matched on identity — customer id → host id → off-site sub-account → wg peer — never on size. The hub's own delete-preview (GET /configs/<id>/delete) was used as an independent enumeration and agreed with the hand-built one in every field.

The cascade is the hub's, not a hand-deletion

POST /configs/<id>/delete runs the designed teardown: leg 1 hosts (including DELETE FROM wg_peers WHERE host_id = ?), leg 2 the committed reset with external teardown FIRST (off-site store + PBS namespace destroyed), leg 3 residue, leg 4 the customer record and all escrow ciphertext. Its own confirmation: „offsite tároló és PBS névtér megsemmisítve… Visszafordíthatatlan."

Six gates, all of which had to be satisfied deliberately — three acknowledgements (ack_hosts/ack_reset/ack_purge), a type-to-confirm confirm_id, an expect_hosts race guard, and — the one that shaped the whole operation — the cascade refuses to delete a live host:

Delete refused: host c11-36d660 is ONLINE. Decommission the box first — the cascade never deletes a live host.

There is no hub decommission endpoint; the word appears only in that refusal. So both VMs were stopped and the hub was allowed to age them past its stale_threshold (30m, from the deployed hub-config) before the cascade would accept them. rewalk aged out at ~21 min of silence, c11 at ~28 min — the two boxes' last reports had landed at different points in their cycles.

Venue 1 — c11 "Campaign 11"

Layer Item After
Machine demo-hp VM 321 c11-appliance (200 G + 50 G + 50 G + efidisk, 20 G on disk) qm destroy 321 --purge — absent from qm list
Hub host c11-36d660 0 occurrences
Hub customer c11 (residue 340 rows, 1 superseded blob) 0 occurrences
Off-site Storage Box sub-account u629488-sub4 0 occurrences
Off-site ep0 PBS namespace c11 gone
Network WireGuard peer 10.77.0.5 0 occurrences

Venue 2 — rewalk "Re-walk R-201"

Layer Item After
Machine demo-hp VM 322 rewalk-appliance (17 G on disk) qm destroy 322 --purge — absent
Hub host rewalk-1ab77d 0 occurrences
Hub customer rewalk (residue 93 rows) 0 occurrences
Off-site Storage Box sub-account u629488-sub5 0 occurrences
Off-site ep0 PBS namespace rewalk gone
Network WireGuard peer 10.77.0.6 0 occurrences

Space returned, measured

c11-scratch used: 58,578,256 KiB → 21,228,640 KiB = 37.3 GB reclaimed, against 20 G + 17 G measured before deletion. Free went 874.8 GB → 912.1 GB.

The positive control

An absence proves nothing unless the instrument can still find something. After each cascade the same greps were run for part4, which must survive: host part4-905bdb still present (2 rows), wg peer 10.77.0.7 still present, ep0 namespace part4 still present. So the zero counts above are measurements, not a broken query.

ep0 namespaces after both teardowns: demo-felhom, demo-hp, part4 — exactly the three that should remain. Nothing on ep0 outside the campaign's own namespaces was touched.

Untouched, as scoped

demo-hp itself and its guest 9201, demo-felhom, drill-r50 (VM 300), DooPlex, the tester's box, and every ep0 namespace that is not the campaign's own.

Venue 3 — part4 "Part 4 - no hand upgrade" (destroyed after the findings were written)

Kept deliberately while Parts 1 and 3 read it: it was standing in the exact failed half-B state, and tearing it down first would have destroyed the evidence. Destroyed only after the diagnoses were committed (70b6260).

Layer Item After
Machine demo-hp VM 323 part4-appliance (16 G on disk) qm destroy 323 --purge — absent
Hub host part4-905bdb 0 occurrences
Hub customer part4 (residue 66 rows) 0 occurrences
Off-site Storage Box sub-account u629488-sub6 0 occurrences
Off-site ep0 PBS namespace part4 gone
Network WireGuard peer 10.77.0.7 0 occurrences

Positive control for this one: demo-hp-bb76ea must survive the same greps, and does (2 rows).

Final state

  • demo-hp guests: VM 300 drill-r50 and CT 9201 (demo-hp's own customer guest). All three campaign appliances are gone.
  • ep0 namespaces: demo-felhom, demo-hp — only the two real demo boxes remain.
  • c11-scratch used: 58,578,256 KiB → 5,184,912 KiB = 53.4 GB reclaimed across the three venues (20 G + 17 G + 16 G measured beforehand). Free 874.8 GB → 928.2 GB.

Secrets that died with the venues

Each cascade purged the customer record and all escrow ciphertext (leg 4). The recovery codes for these venues therefore no longer open anything: R_C11_REWALK and R_PART4 in the operator's credentials file are now inert and can be removed at the operator's convenience. The working copies under ~/.config/part4/ on DooPlex were shredded.

One operational note carried over from the walk: the part4 retrieval passphrase was leaked into a session transcript on 2026-08-06 and rotated at the time; the customer it belonged to no longer exists.


⚠ CORRECTION, 2026-08-07 — the "0 occurrences" claims above are narrower than they read

Verifying the finalwalk teardown with a full census (every table, every column) rather than a per-table query found rows this ledger's method could not see. app_log_issues is not covered by the cascade's residue purge, and it still holds c11 40 rows, rewalk 20, part4 24 — plus 29 for finalwalk.

The tables this ledger did check are genuinely at zero; the claim that is wrong is the implied completeness of the sweep. A per-table absence query is not a census. Filed as R-244, with the reason the leg is not trivial (the table is a fleet-wide aggregate whose rows are shared with live customers, so the fix is a de-reference plus an orphan sweep, not a delete). No secret material is involved. Full working: teardown-finalwalk-2026-08-07.md.