Campaign 10 closeout Part 2: teardown complete — five layers, each verified gone
Evidence-survival check FIRST: HEAD == origin/main == 7efb7a5, and every artefact
confirmed present in origin/main with git cat-file, including all 27 evidence
files. Nothing died with the rig.
1. VM 311 destroyed by exact VMID. qm list now shows only drill-r50; 311.conf does
not exist; images/311 gone. 126 GB returned - nvme 133G used / 758G avail ->
6.7G / 884G. A first attempt stopped the VM but its destroy never ran, killed by
an unescaped paren in my own echo; caught by checking qm list rather than the
exit code.
2. c10-scratch removed by name. storage.cfg 5 -> 4 entries. felhom-backup shares
the same path and is intact, config unchanged.
3. PBS on DooPlex (Tier 2). Identity confirmed first - the target carried the
comment this campaign set at creation. Datastore felhom-c10, user c10@pbs,
token !box, both ACLs and the on-disk chunkstore all gone; felhom-spike,
felhom@pbs, felhom@pbs!n100 and /mnt/5_hdd/backup/demo-felhom all intact. Only
the campaign's entries disappeared. "user delete" is not a PBS subcommand - it
printed usage and the user survived rc=0; caught by re-reading the user list.
4. Restic subaccount 281530 deleted after asserting home==felhom-campaign10 and
the description names Campaign 10 - the script aborts otherwise. 4 -> 3
subaccounts; demo-felhom, peti-felhom and demo-hp all present.
5. Hub customer c10-soak. The ONLINE gate REFUSED with HTTP 409 for 18 consecutive
attempts, exactly as anticipated (customer_delete.go:98-102, :146). Predicted
deletable at last-report 09:38:19 + 30 min = 10:08:19; measured 303 at 10:08:23.
Verified positively: 0 customer rows, 0 host/appliance rows, GET
/customers/c10-soak -> 404, and the hub logged the DELETE cascade COMPLETE with
residue purged. No /appliances/discard was needed - the appliance record was
bound and went with the cascade. drill-r50 verified INTACT (200, VM 300 present).
All 12 credential files shredded and the directory removed.
Register rows NOT written: app-catalog-felhom.eu has an uncommitted working tree
including scripts/check-volume-persistence.py, so the parallel catalog session is
still open and owns the register. R-156/R-157/R-158 rows and rankings remain owed,
itemised in the doc - and that session's volume-persistence checker looks like
R-156's fix shape, so the two should be reconciled before filing.
This commit is contained in:
@@ -133,3 +133,146 @@ a defect that is not already covered.
|
||||
chain. Recording that a design is wrong is the spike rule working.
|
||||
|
||||
Nothing fixed, per the fences.
|
||||
|
||||
---
|
||||
|
||||
# Part 2 — teardown: five layers, each verified gone
|
||||
|
||||
## 0. Evidence-survival check — done BEFORE anything was deleted
|
||||
|
||||
`HEAD == origin/main == 7efb7a5`, tree clean, and each artefact confirmed present **in `origin/main`**
|
||||
(not merely locally) with `git cat-file -e origin/main:<path>`:
|
||||
|
||||
CAMPAIGN-10-two-storage-soak-2026-07-31.md IN origin/main
|
||||
CAMPAIGN-10-closeout-2026-08-02.md IN origin/main
|
||||
SPIKE-recovery-unit-space-2026-08-02.md IN origin/main
|
||||
REPORT-campaign10.md IN origin/main
|
||||
documentation/tests/campaign10-evidence-2026-07-31/ — 27 files in origin/main
|
||||
|
||||
Everything the campaign and both spikes produced survives the rig.
|
||||
|
||||
## 1. VM 311 on demo-hp — GONE
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| `qm list` | 300 `drill-r50`, **311 `c10-appliance` running** | **300 `drill-r50` only** |
|
||||
| `311.conf` | present | `Configuration file 'nodes/felhom-host/qemu-server/311.conf' does not exist` |
|
||||
| `/mnt/nvme-1tb/images/311` | **127 G** | `No such file or directory` |
|
||||
| `/mnt/nvme-1tb` | 133 G used / 758 G avail | **6.7 G used / 884 G avail** |
|
||||
|
||||
**Space returned: 126 GB.** Deleted by exact VMID with `qm destroy 311 --purge`. `drill-r50` untouched.
|
||||
|
||||
*(A first attempt stopped the VM but its `qm destroy` never ran — an unescaped paren in my own echo
|
||||
made the remote shell exit before it. Caught by checking `qm list` rather than the exit code, and
|
||||
re-run.)*
|
||||
|
||||
## 2. `c10-scratch` dir storage — GONE
|
||||
|
||||
before: dir: local | lvmthin: local-lvm | pbs: felhom-pbs | dir: felhom-backup | dir: c10-scratch (count=5)
|
||||
after : dir: local | lvmthin: local-lvm | pbs: felhom-pbs | dir: felhom-backup (count=4)
|
||||
|
||||
Removed by name (`pvesm remove c10-scratch`). **`felhom-backup` shares the same path
|
||||
(`/mnt/nvme-1tb`) and is intact** — `path /mnt/nvme-1tb, content backup, is_mountpoint 1`, unchanged.
|
||||
|
||||
## 3. PBS datastore + token on DooPlex (Tier 2) — GONE, production untouched
|
||||
|
||||
**Identity confirmed before deleting.** The target carried the comment this campaign set at creation:
|
||||
|
||||
felhom-c10 /mnt/5_hdd/backup/campaign10 "Campaign 10 scratch - DELETE at teardown" <-- target
|
||||
felhom-spike /mnt/5_hdd/backup/demo-felhom (no comment) <-- PRODUCTION
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| datastores | `felhom-c10`, `felhom-spike` | **`felhom-spike` only** |
|
||||
| users | `c10@pbs`, `felhom@pbs`, `root@pam` | **`felhom@pbs`, `root@pam`** |
|
||||
| tokens | `c10@pbs!box`, `felhom@pbs!n100` | **`felhom@pbs!n100`** |
|
||||
| ACLs | 2 × `/datastore/felhom-c10` + 2 × `/datastore/felhom-spike` | **2 × `/datastore/felhom-spike` only** |
|
||||
| on disk | `/mnt/5_hdd/backup/campaign10` | removed; `/mnt/5_hdd/backup/demo-felhom` **INTACT** |
|
||||
|
||||
**Only the campaign's entries disappeared.** *(`user delete` is not a PBS subcommand — it printed
|
||||
usage and the user survived. Caught by re-reading the user list rather than trusting `rc=0`; the
|
||||
correct pair is `user delete-token` then `user remove`.)*
|
||||
|
||||
## 4. Restic subaccount on the Storage Box — GONE, production untouched
|
||||
|
||||
Identity asserted programmatically before the DELETE — the script aborts unless both hold:
|
||||
|
||||
id=281530 username=u629488-sub4 home=felhom-campaign10 desc="Campaign 10 scratch - DELETE at teardown"
|
||||
assert home == "felhom-campaign10" PASSED
|
||||
assert "Campaign 10" in description PASSED
|
||||
|
||||
Its inability to reach any production path was already proven in `isolation/restic-denial.txt`.
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| subaccounts | 4 | **3** |
|
||||
| `281530` (campaign) | present | **absent** |
|
||||
| `273581`/`269130`/`275124` (demo-felhom, **peti-felhom**, demo-hp) | present | **all present** |
|
||||
|
||||
## 5. Hub customer `c10-soak` — GONE, after the ONLINE gate refused
|
||||
|
||||
**The gate refused, exactly as anticipated.** `POST /configs/c10-soak/delete` returned **HTTP 409**
|
||||
for 18 consecutive attempts. Source: `internal/web/customer_delete.go:98-102` sets
|
||||
`onlineBlocked = true` while `hostStatus(LastReportAt) == "ok"`, and `:146` comments
|
||||
*"leg 1 refuses; decommission the agent first"*.
|
||||
|
||||
**When it became deletable — predicted and confirmed.** Last host-report **09:38:19**; `ok` lapses at
|
||||
30 minutes ⇒ deletable from **10:08:19**. Measured:
|
||||
|
||||
09:50:17 attempt 1 -> HTTP 409
|
||||
…
|
||||
10:07:20 attempt 18 -> HTTP 409
|
||||
10:08:23 attempt 19 -> HTTP 303 <-- 4 s after the predicted threshold
|
||||
|
||||
**Verified positively, not by trusting the response:**
|
||||
|
||||
customers list — c10-soak / "Campaign 10" rows: 0
|
||||
hosts+appliances — c10-soak / 55ab3776 rows: 0
|
||||
GET /customers/c10-soak -> HTTP 404
|
||||
hub: delete c10-soak: residue purged (reports=198 app_telemetry=1099 notif_prefs=1
|
||||
selfbind_tokens=1 appliance_registrations=1)
|
||||
hub: customer DELETE cascade COMPLETE for c10-soak (journal #9) — full teardown
|
||||
|
||||
**No separate `/appliances/<id>/discard` was needed**: the appliance record was bound, so it left the
|
||||
unclaimed list at bind time and was purged by the cascade (`appliance_registrations=1`). The task
|
||||
anticipated `/discard`; on this path the cascade covered it.
|
||||
|
||||
**Remaining customers, unchanged:** Demo Ügyfél, Demo HP, Peti Proxmox, and
|
||||
**`R-120 golden 0.186.0 proof` (= `sess-f`) — the pre-existing R-131 orphan, not this campaign's and
|
||||
deliberately not touched.**
|
||||
|
||||
**`drill-r50` (R-93 fixture) verified intact:** `GET /customers/drill-r50` → **HTTP 200**, and VM 300
|
||||
still present and stopped on demo-hp. (It is absent from the dashboard listing because it is DOWN, not
|
||||
because it is gone — checked explicitly rather than inferred from the listing.)
|
||||
|
||||
## 6. Credentials
|
||||
|
||||
All 12 files under `~/.config/campaign10/` **shredded** (`shred -u`) and the directory removed —
|
||||
host/dashboard passwords, API key, app secrets, PBS token, restic password, claim code. The guest-side
|
||||
copies (`/root/c10/`, `/root/c10api.sh`) died with the VM. No campaign processes remain.
|
||||
|
||||
---
|
||||
|
||||
# Register work still owed — NOT done this session, and why
|
||||
|
||||
**The parallel catalog session is OPEN.** `app-catalog-felhom.eu` has an uncommitted working tree:
|
||||
`CLAUDE.md` and `REUSE.md` modified, plus untracked `audits/`, `scripts/check-volume-persistence.py`
|
||||
and `scripts/test_check_volume_persistence.py`. One session owns the register at a time, so **no rows
|
||||
were written**.
|
||||
|
||||
Still owed, for whoever takes the register next:
|
||||
|
||||
| finding | needs |
|
||||
|---|---|
|
||||
| **R-156** | row + rank. papra's data neither persisted nor backed up, reports healthy. *Note: the parallel session's `check-volume-persistence.py` looks like exactly this fix's shape — reconcile before filing.* |
|
||||
| **R-157** | row + rank. bootrecon's start-once sweep, **two mechanisms** (A: Exited/race ~50%; B: zero-container, **silent on every channel**). A settle-condition fix closes A only. |
|
||||
| **R-158** | row + rank — **ranked BELOW R-157 by Q1** (notification gap, not silent failure; visible on `/backups`, absent from `/backups/apps`). |
|
||||
| spike C2 | not a finding: a **dependency chain** (sound predicate → warn→gate → tar-drop), lengthened by Q2. |
|
||||
|
||||
# What did not happen, and why
|
||||
|
||||
- **No register rows written** — parallel session owns it (above).
|
||||
- **Nothing fixed** — R-156/R-157/R-158 and everything Q1/Q2 surfaced are findings only, per the fences.
|
||||
- **No `/appliances/<id>/discard` call** — there was no appliance row left to discard (§5).
|
||||
- **`sess-f` / R-131 orphan not deleted** — not this campaign's record; deliberately out of scope.
|
||||
- **The ~5 GB linearity point** — the earlier spike's brief said skip it; still skipped.
|
||||
|
||||
Reference in New Issue
Block a user