felhom.eu/documentation/audits/CAMPAIGN-10-closeout-2026-08-02.md

# Campaign 10 — closeout: two last measurements, then teardown (2026-08-02)

**Companion doc** (my call, per the task) to `CAMPAIGN-10-two-storage-soak-2026-07-31.md` and
`SPIKE-recovery-unit-space-2026-08-02.md`, rather than appending — those two are already long and this
session answers two specific questions and then dismantles the rig.

**Part 1 headline:** Q1 **lowers** R-158's rank — the failure *is* customer-visible, on one page, and it
persists and clears correctly. Q2 **clears** `ValidateDump` of shipping bad dumps, and simultaneously
**kills C2's proposed gate** on its current predicate.

---

## Q1 — when a backup refuses for lack of space, what does the customer see?

Method: ballast `/mnt/sys_drive` to 699 MB free (homebox's unit needs 2.3 GB), run the real backup
through `POST /api/backup/run`, and read every customer-facing surface through the real UI endpoints.

### The five points

| # | Surface | Result |
|---|---|---|
| 1 | **`/backups` overview** | **SHOWS IT** — `✗ Adatmentés sikertelen` |
| 2 | **`/backups/apps`** (per-app) | **shows nothing** — Homebox tier-1 reads `Utolsó: 3 perce` |
| 3 | `/stacks/homebox` (app detail) | **shows nothing** |
| 4 | `/` dashboard · `/launcher` | **show nothing** — no banner |
| 5 | `GET /api/backup/status` | `db_dump.success: **false**` |
| — | hub event / email | **none** (R-158) |

Exact copy, as rendered on `/backups`:

    ✗  Adatmentés sikertelen

### The point-5 question — retry, stay failed, or quietly go green?

**It retries, stays failed, and does not lie.** Measured across three runs:

| run | condition | `status.success` | `/backups` shows failure | unit mtime |
|---|---|---|---|---|
| 1 | constrained | false | **true** | 07:30:50 |
| 2 | still constrained | false | **true (persists)** | 07:30:50 (unchanged) |
| 3 | space freed | **true** | **false (cleared)** | **07:37:38 (fresh)** |

**`/backups/apps` is honest, not deceptive.** Its `Utolsó: 3 perce` tracks the *unit's real mtime*
(07:30:50), not the failed run's timestamp — so it correctly reports the age of the last good unit. It
simply never says the latest attempt failed. **This is NOT the R-156 family**: no surface claims a
fresh backup over a stale unit.

### R-158's rank — LOWERED, with the reason

> **R-158 is a notification gap, not a silent-failure defect. It ranks BELOW R-157.**

Because the customer *can* find out by looking, and the indicator behaves correctly — it persists while
failing and clears on recovery. What is missing is that **nothing pushes it**: no hub event, no email,
no dashboard banner. And the discovery path is poor in a specific way worth fixing: **the per-app
backup page — the natural place to ask "is homebox backed up?" — shows no failure at all**, while the
answer lives one page away on the overview.

R-157 outranks it because R-157's mechanism B leaves a `deployed: true` app **not running**, with the
deadapp check reporting `0 currently down` — silent on *every* channel, including the one the customer
would look at.

---

## Q2 — was `ValidateDump`'s warning correct?

The warning, seen twice:

    [WARN] [backup] ValidateDump: …/rallly-postgres.sql is structurally valid (32 tables) but its
           accounts table has NO rows — the dump may predate the customer's data

### It was factually correct, and the dump was NOT deficient

The live database genuinely had no accounts. Every non-empty table, from `pg_stat_user_tables`:

    _prisma_migrations | 129
    cc_proof           |  82      <-- the campaign's own canary
    instance_settings  |   1
    accounts           |   0

**An empty table proves nothing either way**, so — as the task required — an account was **seeded** and
the backup re-run:

    INSERT users    (id='c10user', …)   -> 1 row
    INSERT accounts (id='c10acct', …)   -> 1 row

Result:

    [INFO] [backup] DB dump: rallly-postgres → rallly-postgres.sql (100.6 KB, 396ms, 32 tables)
    (no ValidateDump line at all — the warning STOPPED)

    grep -c c10acct <dump> = 1        grep -c c10user <dump> = 2
    dump size 102 766 B -> 103 029 B

**Verdict: the validator is sound and no bad dumps are shipping.** It correctly detected an empty
`accounts` table, and the moment real data existed the dump captured it and the warning cleared.

*(Seeded data disclosed: one `users` row and one `accounts` row, both `c10*`-prefixed, on a scratch
customer that this session then deletes.)*

### What it does to C2's ordering — the proposed gate is DEAD on this predicate

C2 proposed dropping the DB volume tar (halving DB-app units, and closing the D5/R-127(b) initdb-skip
password trap) **once `ValidateDump` is promoted from a warning to a gate**. That ordering does not
survive Q2:

> **A fresh appliance legitimately has zero accounts.** Gating on "the `accounts` table has rows" would
> **block the backups of every new customer** until someone registers a user — refusing good backups
> for the entire period a box is most fragile.

The validator's *fact* is right and its *inference* ("may predate the customer's data") is wrong: there
was no data to predate. So the dependency chain is longer than C2 assumed:

1. a **sound gate predicate** first — a comparison against the **live** DB (per-table row counts in the
   dump vs `pg_stat_user_tables`) rather than an absolute expectation about one table;
2. **then** the warn→gate promotion;
3. **only then** the tar-drop.

**Until (1) exists, the DB volume tar stays load-bearing** — not because dumps are bad, but because
nothing can yet *prove* a given dump is good.

---

## Findings

**No new `R-n`.** Register grepped (`documentation/backlog/*.md`) for `ValidateDump`, `accounts table`,
`Adatmentés sikertelen`, and the per-app backup page: no existing rows, and neither Q1 nor Q2 produced
a defect that is not already covered.

- **Q1 does not mint an R** — it *re-ranks* R-158 (filed in the spike) and adds the specific
  improvement worth making: surface the per-app failure on `/backups/apps`, where a customer looks for
  a specific app.
- **Q2 does not mint an R** — it *removes* a proposed change (the gate) and lengthens C2's dependency
  chain. Recording that a design is wrong is the spike rule working.

Nothing fixed, per the fences.

---

# Part 2 — teardown: five layers, each verified gone

## 0. Evidence-survival check — done BEFORE anything was deleted

`HEAD == origin/main == 7efb7a5`, tree clean, and each artefact confirmed present **in `origin/main`**
(not merely locally) with `git cat-file -e origin/main:<path>`:

    CAMPAIGN-10-two-storage-soak-2026-07-31.md   IN origin/main
    CAMPAIGN-10-closeout-2026-08-02.md           IN origin/main
    SPIKE-recovery-unit-space-2026-08-02.md      IN origin/main
    REPORT-campaign10.md                         IN origin/main
    documentation/tests/campaign10-evidence-2026-07-31/  — 27 files in origin/main

Everything the campaign and both spikes produced survives the rig.

## 1. VM 311 on demo-hp — GONE

| | before | after |
|---|---|---|
| `qm list` | 300 `drill-r50`, **311 `c10-appliance` running** | **300 `drill-r50` only** |
| `311.conf` | present | `Configuration file 'nodes/felhom-host/qemu-server/311.conf' does not exist` |
| `/mnt/nvme-1tb/images/311` | **127 G** | `No such file or directory` |
| `/mnt/nvme-1tb` | 133 G used / 758 G avail | **6.7 G used / 884 G avail** |

**Space returned: 126 GB.** Deleted by exact VMID with `qm destroy 311 --purge`. `drill-r50` untouched.

*(A first attempt stopped the VM but its `qm destroy` never ran — an unescaped paren in my own echo
made the remote shell exit before it. Caught by checking `qm list` rather than the exit code, and
re-run.)*

## 2. `c10-scratch` dir storage — GONE

    before: dir: local | lvmthin: local-lvm | pbs: felhom-pbs | dir: felhom-backup | dir: c10-scratch   (count=5)
    after : dir: local | lvmthin: local-lvm | pbs: felhom-pbs | dir: felhom-backup                       (count=4)

Removed by name (`pvesm remove c10-scratch`). **`felhom-backup` shares the same path
(`/mnt/nvme-1tb`) and is intact** — `path /mnt/nvme-1tb, content backup, is_mountpoint 1`, unchanged.

## 3. PBS datastore + token on DooPlex (Tier 2) — GONE, production untouched

**Identity confirmed before deleting.** The target carried the comment this campaign set at creation:

    felhom-c10   /mnt/5_hdd/backup/campaign10   "Campaign 10 scratch - DELETE at teardown"   <-- target
    felhom-spike /mnt/5_hdd/backup/demo-felhom  (no comment)                                 <-- PRODUCTION

| | before | after |
|---|---|---|
| datastores | `felhom-c10`, `felhom-spike` | **`felhom-spike` only** |
| users | `c10@pbs`, `felhom@pbs`, `root@pam` | **`felhom@pbs`, `root@pam`** |
| tokens | `c10@pbs!box`, `felhom@pbs!n100` | **`felhom@pbs!n100`** |
| ACLs | 2 × `/datastore/felhom-c10` + 2 × `/datastore/felhom-spike` | **2 × `/datastore/felhom-spike` only** |
| on disk | `/mnt/5_hdd/backup/campaign10` | removed; `/mnt/5_hdd/backup/demo-felhom` **INTACT** |

**Only the campaign's entries disappeared.** *(`user delete` is not a PBS subcommand — it printed
usage and the user survived. Caught by re-reading the user list rather than trusting `rc=0`; the
correct pair is `user delete-token` then `user remove`.)*

## 4. Restic subaccount on the Storage Box — GONE, production untouched

Identity asserted programmatically before the DELETE — the script aborts unless both hold:

    id=281530 username=u629488-sub4 home=felhom-campaign10 desc="Campaign 10 scratch - DELETE at teardown"
    assert home == "felhom-campaign10"      PASSED
    assert "Campaign 10" in description     PASSED

Its inability to reach any production path was already proven in `isolation/restic-denial.txt`.

| | before | after |
|---|---|---|
| subaccounts | 4 | **3** |
| `281530` (campaign) | present | **absent** |
| `273581`/`269130`/`275124` (demo-felhom, **peti-felhom**, demo-hp) | present | **all present** |

## 5. Hub customer `c10-soak` — GONE, after the ONLINE gate refused

**The gate refused, exactly as anticipated.** `POST /configs/c10-soak/delete` returned **HTTP 409**
for 18 consecutive attempts. Source: `internal/web/customer_delete.go:98-102` sets
`onlineBlocked = true` while `hostStatus(LastReportAt) == "ok"`, and `:146` comments
*"leg 1 refuses; decommission the agent first"*.

**When it became deletable — predicted and confirmed.** Last host-report **09:38:19**; `ok` lapses at
30 minutes ⇒ deletable from **10:08:19**. Measured:

    09:50:17  attempt  1 -> HTTP 409
    …
    10:07:20  attempt 18 -> HTTP 409
    10:08:23  attempt 19 -> HTTP 303      <-- 4 s after the predicted threshold

**Verified positively, not by trusting the response:**

    customers list  — c10-soak / "Campaign 10" rows: 0
    hosts+appliances — c10-soak / 55ab3776 rows:      0
    GET /customers/c10-soak                        -> HTTP 404
    hub: delete c10-soak: residue purged (reports=198 app_telemetry=1099 notif_prefs=1
         selfbind_tokens=1 appliance_registrations=1)
    hub: customer DELETE cascade COMPLETE for c10-soak (journal #9) — full teardown

**No separate `/appliances/<id>/discard` was needed**: the appliance record was bound, so it left the
unclaimed list at bind time and was purged by the cascade (`appliance_registrations=1`). The task
anticipated `/discard`; on this path the cascade covered it.

**Remaining customers, unchanged:** Demo Ügyfél, Demo HP, Peti Proxmox, and
**`R-120 golden 0.186.0 proof` (= `sess-f`) — the pre-existing R-131 orphan, not this campaign's and
deliberately not touched.**

**`drill-r50` (R-93 fixture) verified intact:** `GET /customers/drill-r50` → **HTTP 200**, and VM 300
still present and stopped on demo-hp. (It is absent from the dashboard listing because it is DOWN, not
because it is gone — checked explicitly rather than inferred from the listing.)

## 6. Credentials

All 12 files under `~/.config/campaign10/` **shredded** (`shred -u`) and the directory removed —
host/dashboard passwords, API key, app secrets, PBS token, restic password, claim code. The guest-side
copies (`/root/c10/`, `/root/c10api.sh`) died with the VM. No campaign processes remain.

---

# Register work still owed — NOT done this session, and why

**The parallel catalog session is OPEN.** `app-catalog-felhom.eu` has an uncommitted working tree:
`CLAUDE.md` and `REUSE.md` modified, plus untracked `audits/`, `scripts/check-volume-persistence.py`
and `scripts/test_check_volume_persistence.py`. One session owns the register at a time, so **no rows
were written**.

Still owed, for whoever takes the register next:

| finding | needs |
|---|---|
| **R-156** | row + rank. papra's data neither persisted nor backed up, reports healthy. *Note: the parallel session's `check-volume-persistence.py` looks like exactly this fix's shape — reconcile before filing.* |
| **R-157** | row + rank. bootrecon's start-once sweep, **two mechanisms** (A: Exited/race ~50%; B: zero-container, **silent on every channel**). A settle-condition fix closes A only. |
| **R-158** | row + rank — **ranked BELOW R-157 by Q1** (notification gap, not silent failure; visible on `/backups`, absent from `/backups/apps`). |
| spike C2 | not a finding: a **dependency chain** (sound predicate → warn→gate → tar-drop), lengthened by Q2. |

# What did not happen, and why

- **No register rows written** — parallel session owns it (above).
- **Nothing fixed** — R-156/R-157/R-158 and everything Q1/Q2 surfaced are findings only, per the fences.
- **No `/appliances/<id>/discard` call** — there was no appliance row left to discard (§5).
- **`sess-f` / R-131 orphan not deleted** — not this campaign's record; deliberately out of scope.
- **The ~5 GB linearity point** — the earlier spike's brief said skip it; still skipped.