diff --git a/documentation/audits/CAMPAIGN-10-closeout-2026-08-02.md b/documentation/audits/CAMPAIGN-10-closeout-2026-08-02.md index b8525e5..efad066 100644 --- a/documentation/audits/CAMPAIGN-10-closeout-2026-08-02.md +++ b/documentation/audits/CAMPAIGN-10-closeout-2026-08-02.md @@ -133,3 +133,146 @@ a defect that is not already covered. chain. Recording that a design is wrong is the spike rule working. Nothing fixed, per the fences. + +--- + +# Part 2 — teardown: five layers, each verified gone + +## 0. Evidence-survival check — done BEFORE anything was deleted + +`HEAD == origin/main == 7efb7a5`, tree clean, and each artefact confirmed present **in `origin/main`** +(not merely locally) with `git cat-file -e origin/main:`: + + CAMPAIGN-10-two-storage-soak-2026-07-31.md IN origin/main + CAMPAIGN-10-closeout-2026-08-02.md IN origin/main + SPIKE-recovery-unit-space-2026-08-02.md IN origin/main + REPORT-campaign10.md IN origin/main + documentation/tests/campaign10-evidence-2026-07-31/ — 27 files in origin/main + +Everything the campaign and both spikes produced survives the rig. + +## 1. VM 311 on demo-hp — GONE + +| | before | after | +|---|---|---| +| `qm list` | 300 `drill-r50`, **311 `c10-appliance` running** | **300 `drill-r50` only** | +| `311.conf` | present | `Configuration file 'nodes/felhom-host/qemu-server/311.conf' does not exist` | +| `/mnt/nvme-1tb/images/311` | **127 G** | `No such file or directory` | +| `/mnt/nvme-1tb` | 133 G used / 758 G avail | **6.7 G used / 884 G avail** | + +**Space returned: 126 GB.** Deleted by exact VMID with `qm destroy 311 --purge`. `drill-r50` untouched. + +*(A first attempt stopped the VM but its `qm destroy` never ran — an unescaped paren in my own echo +made the remote shell exit before it. Caught by checking `qm list` rather than the exit code, and +re-run.)* + +## 2. `c10-scratch` dir storage — GONE + + before: dir: local | lvmthin: local-lvm | pbs: felhom-pbs | dir: felhom-backup | dir: c10-scratch (count=5) + after : dir: local | lvmthin: local-lvm | pbs: felhom-pbs | dir: felhom-backup (count=4) + +Removed by name (`pvesm remove c10-scratch`). **`felhom-backup` shares the same path +(`/mnt/nvme-1tb`) and is intact** — `path /mnt/nvme-1tb, content backup, is_mountpoint 1`, unchanged. + +## 3. PBS datastore + token on DooPlex (Tier 2) — GONE, production untouched + +**Identity confirmed before deleting.** The target carried the comment this campaign set at creation: + + felhom-c10 /mnt/5_hdd/backup/campaign10 "Campaign 10 scratch - DELETE at teardown" <-- target + felhom-spike /mnt/5_hdd/backup/demo-felhom (no comment) <-- PRODUCTION + +| | before | after | +|---|---|---| +| datastores | `felhom-c10`, `felhom-spike` | **`felhom-spike` only** | +| users | `c10@pbs`, `felhom@pbs`, `root@pam` | **`felhom@pbs`, `root@pam`** | +| tokens | `c10@pbs!box`, `felhom@pbs!n100` | **`felhom@pbs!n100`** | +| ACLs | 2 × `/datastore/felhom-c10` + 2 × `/datastore/felhom-spike` | **2 × `/datastore/felhom-spike` only** | +| on disk | `/mnt/5_hdd/backup/campaign10` | removed; `/mnt/5_hdd/backup/demo-felhom` **INTACT** | + +**Only the campaign's entries disappeared.** *(`user delete` is not a PBS subcommand — it printed +usage and the user survived. Caught by re-reading the user list rather than trusting `rc=0`; the +correct pair is `user delete-token` then `user remove`.)* + +## 4. Restic subaccount on the Storage Box — GONE, production untouched + +Identity asserted programmatically before the DELETE — the script aborts unless both hold: + + id=281530 username=u629488-sub4 home=felhom-campaign10 desc="Campaign 10 scratch - DELETE at teardown" + assert home == "felhom-campaign10" PASSED + assert "Campaign 10" in description PASSED + +Its inability to reach any production path was already proven in `isolation/restic-denial.txt`. + +| | before | after | +|---|---|---| +| subaccounts | 4 | **3** | +| `281530` (campaign) | present | **absent** | +| `273581`/`269130`/`275124` (demo-felhom, **peti-felhom**, demo-hp) | present | **all present** | + +## 5. Hub customer `c10-soak` — GONE, after the ONLINE gate refused + +**The gate refused, exactly as anticipated.** `POST /configs/c10-soak/delete` returned **HTTP 409** +for 18 consecutive attempts. Source: `internal/web/customer_delete.go:98-102` sets +`onlineBlocked = true` while `hostStatus(LastReportAt) == "ok"`, and `:146` comments +*"leg 1 refuses; decommission the agent first"*. + +**When it became deletable — predicted and confirmed.** Last host-report **09:38:19**; `ok` lapses at +30 minutes ⇒ deletable from **10:08:19**. Measured: + + 09:50:17 attempt 1 -> HTTP 409 + … + 10:07:20 attempt 18 -> HTTP 409 + 10:08:23 attempt 19 -> HTTP 303 <-- 4 s after the predicted threshold + +**Verified positively, not by trusting the response:** + + customers list — c10-soak / "Campaign 10" rows: 0 + hosts+appliances — c10-soak / 55ab3776 rows: 0 + GET /customers/c10-soak -> HTTP 404 + hub: delete c10-soak: residue purged (reports=198 app_telemetry=1099 notif_prefs=1 + selfbind_tokens=1 appliance_registrations=1) + hub: customer DELETE cascade COMPLETE for c10-soak (journal #9) — full teardown + +**No separate `/appliances//discard` was needed**: the appliance record was bound, so it left the +unclaimed list at bind time and was purged by the cascade (`appliance_registrations=1`). The task +anticipated `/discard`; on this path the cascade covered it. + +**Remaining customers, unchanged:** Demo Ügyfél, Demo HP, Peti Proxmox, and +**`R-120 golden 0.186.0 proof` (= `sess-f`) — the pre-existing R-131 orphan, not this campaign's and +deliberately not touched.** + +**`drill-r50` (R-93 fixture) verified intact:** `GET /customers/drill-r50` → **HTTP 200**, and VM 300 +still present and stopped on demo-hp. (It is absent from the dashboard listing because it is DOWN, not +because it is gone — checked explicitly rather than inferred from the listing.) + +## 6. Credentials + +All 12 files under `~/.config/campaign10/` **shredded** (`shred -u`) and the directory removed — +host/dashboard passwords, API key, app secrets, PBS token, restic password, claim code. The guest-side +copies (`/root/c10/`, `/root/c10api.sh`) died with the VM. No campaign processes remain. + +--- + +# Register work still owed — NOT done this session, and why + +**The parallel catalog session is OPEN.** `app-catalog-felhom.eu` has an uncommitted working tree: +`CLAUDE.md` and `REUSE.md` modified, plus untracked `audits/`, `scripts/check-volume-persistence.py` +and `scripts/test_check_volume_persistence.py`. One session owns the register at a time, so **no rows +were written**. + +Still owed, for whoever takes the register next: + +| finding | needs | +|---|---| +| **R-156** | row + rank. papra's data neither persisted nor backed up, reports healthy. *Note: the parallel session's `check-volume-persistence.py` looks like exactly this fix's shape — reconcile before filing.* | +| **R-157** | row + rank. bootrecon's start-once sweep, **two mechanisms** (A: Exited/race ~50%; B: zero-container, **silent on every channel**). A settle-condition fix closes A only. | +| **R-158** | row + rank — **ranked BELOW R-157 by Q1** (notification gap, not silent failure; visible on `/backups`, absent from `/backups/apps`). | +| spike C2 | not a finding: a **dependency chain** (sound predicate → warn→gate → tar-drop), lengthened by Q2. | + +# What did not happen, and why + +- **No register rows written** — parallel session owns it (above). +- **Nothing fixed** — R-156/R-157/R-158 and everything Q1/Q2 surfaced are findings only, per the fences. +- **No `/appliances//discard` call** — there was no appliance row left to discard (§5). +- **`sess-f` / R-131 orphan not deleted** — not this campaign's record; deliberately out of scope. +- **The ~5 GB linearity point** — the earlier spike's brief said skip it; still skipped.