felhom.eu/documentation/audits/CAMPAIGN-10-closeout-2026-08-02.md

Campaign 10 — closeout: two last measurements, then teardown (2026-08-02)

Companion doc (my call, per the task) to CAMPAIGN-10-two-storage-soak-2026-07-31.md and SPIKE-recovery-unit-space-2026-08-02.md, rather than appending — those two are already long and this session answers two specific questions and then dismantles the rig.

Part 1 headline: Q1 lowers R-158's rank — the failure is customer-visible, on one page, and it persists and clears correctly. Q2 clears ValidateDump of shipping bad dumps, and simultaneously kills C2's proposed gate on its current predicate.

---

Q1 — when a backup refuses for lack of space, what does the customer see?

Method: ballast /mnt/sys_drive to 699 MB free (homebox's unit needs 2.3 GB), run the real backup through POST /api/backup/run, and read every customer-facing surface through the real UI endpoints.

The five points

#	Surface	Result
1	/backups overview	SHOWS IT — ✗ Adatmentés sikertelen
2	/backups/apps (per-app)	shows nothing — Homebox tier-1 reads Utolsó: 3 perce
3	/stacks/homebox (app detail)	shows nothing
4	/ dashboard · /launcher	show nothing — no banner
5	GET /api/backup/status	db_dump.success: **false**
—	hub event / email	none (R-158)

Exact copy, as rendered on /backups:

✗  Adatmentés sikertelen

The point-5 question — retry, stay failed, or quietly go green?

It retries, stays failed, and does not lie. Measured across three runs:

run	condition	status.success	/backups shows failure	unit mtime
1	constrained	false	true	07:30:50
2	still constrained	false	true (persists)	07:30:50 (unchanged)
3	space freed	true	false (cleared)	07:37:38 (fresh)

/backups/apps is honest, not deceptive. Its Utolsó: 3 perce tracks the unit's real mtime (07:30:50), not the failed run's timestamp — so it correctly reports the age of the last good unit. It simply never says the latest attempt failed. This is NOT the R-156 family: no surface claims a fresh backup over a stale unit.

R-158's rank — LOWERED, with the reason

R-158 is a notification gap, not a silent-failure defect. It ranks BELOW R-157.

Because the customer can find out by looking, and the indicator behaves correctly — it persists while failing and clears on recovery. What is missing is that nothing pushes it: no hub event, no email, no dashboard banner. And the discovery path is poor in a specific way worth fixing: the per-app backup page — the natural place to ask "is homebox backed up?" — shows no failure at all, while the answer lives one page away on the overview.

R-157 outranks it because R-157's mechanism B leaves a deployed: true app not running, with the deadapp check reporting 0 currently down — silent on every channel, including the one the customer would look at.

---

Q2 — was ValidateDump's warning correct?

The warning, seen twice:

[WARN] [backup] ValidateDump: …/rallly-postgres.sql is structurally valid (32 tables) but its
       accounts table has NO rows — the dump may predate the customer's data

It was factually correct, and the dump was NOT deficient

The live database genuinely had no accounts. Every non-empty table, from pg_stat_user_tables:

_prisma_migrations | 129
cc_proof           |  82      <-- the campaign's own canary
instance_settings  |   1
accounts           |   0

An empty table proves nothing either way, so — as the task required — an account was seeded and the backup re-run:

INSERT users    (id='c10user', …)   -> 1 row
INSERT accounts (id='c10acct', …)   -> 1 row

Result:

[INFO] [backup] DB dump: rallly-postgres → rallly-postgres.sql (100.6 KB, 396ms, 32 tables)
(no ValidateDump line at all — the warning STOPPED)

grep -c c10acct <dump> = 1        grep -c c10user <dump> = 2
dump size 102 766 B -> 103 029 B

Verdict: the validator is sound and no bad dumps are shipping. It correctly detected an empty accounts table, and the moment real data existed the dump captured it and the warning cleared.

(Seeded data disclosed: one users row and one accounts row, both c10*-prefixed, on a scratch customer that this session then deletes.)

What it does to C2's ordering — the proposed gate is DEAD on this predicate

C2 proposed dropping the DB volume tar (halving DB-app units, and closing the D5/R-127(b) initdb-skip password trap) once ValidateDump is promoted from a warning to a gate. That ordering does not survive Q2:

A fresh appliance legitimately has zero accounts. Gating on "the accounts table has rows" would block the backups of every new customer until someone registers a user — refusing good backups for the entire period a box is most fragile.

The validator's fact is right and its inference ("may predate the customer's data") is wrong: there was no data to predate. So the dependency chain is longer than C2 assumed:

1. a sound gate predicate first — a comparison against the live DB (per-table row counts in the dump vs pg_stat_user_tables) rather than an absolute expectation about one table;
2. then the warn→gate promotion;
3. only then the tar-drop.

Until (1) exists, the DB volume tar stays load-bearing — not because dumps are bad, but because nothing can yet prove a given dump is good.

---

Findings

No new R-n. Register grepped (documentation/backlog/*.md) for ValidateDump, accounts table, Adatmentés sikertelen, and the per-app backup page: no existing rows, and neither Q1 nor Q2 produced a defect that is not already covered.

• Q1 does not mint an R — it re-ranks R-158 (filed in the spike) and adds the specific improvement worth making: surface the per-app failure on /backups/apps, where a customer looks for a specific app.
• Q2 does not mint an R — it removes a proposed change (the gate) and lengthens C2's dependency chain. Recording that a design is wrong is the spike rule working.

Nothing fixed, per the fences.

---

Part 2 — teardown: five layers, each verified gone

0. Evidence-survival check — done BEFORE anything was deleted

HEAD == origin/main == 7efb7a5, tree clean, and each artefact confirmed present in origin/main (not merely locally) with git cat-file -e origin/main:<path>:

CAMPAIGN-10-two-storage-soak-2026-07-31.md   IN origin/main
CAMPAIGN-10-closeout-2026-08-02.md           IN origin/main
SPIKE-recovery-unit-space-2026-08-02.md      IN origin/main
REPORT-campaign10.md                         IN origin/main
documentation/tests/campaign10-evidence-2026-07-31/  — 27 files in origin/main

Everything the campaign and both spikes produced survives the rig.

1. VM 311 on demo-hp — GONE

	before	after
qm list	300 drill-r50, 311 c10-appliance running	300 drill-r50 only
311.conf	present	Configuration file 'nodes/felhom-host/qemu-server/311.conf' does not exist
/mnt/nvme-1tb/images/311	127 G	No such file or directory
/mnt/nvme-1tb	133 G used / 758 G avail	6.7 G used / 884 G avail

Space returned: 126 GB. Deleted by exact VMID with qm destroy 311 --purge. drill-r50 untouched.

(A first attempt stopped the VM but its qm destroy never ran — an unescaped paren in my own echo made the remote shell exit before it. Caught by checking qm list rather than the exit code, and re-run.)

2. c10-scratch dir storage — GONE

before: dir: local | lvmthin: local-lvm | pbs: felhom-pbs | dir: felhom-backup | dir: c10-scratch   (count=5)
after : dir: local | lvmthin: local-lvm | pbs: felhom-pbs | dir: felhom-backup                       (count=4)

Removed by name (pvesm remove c10-scratch). felhom-backup shares the same path (/mnt/nvme-1tb) and is intact — path /mnt/nvme-1tb, content backup, is_mountpoint 1, unchanged.

3. PBS datastore + token on DooPlex (Tier 2) — GONE, production untouched

Identity confirmed before deleting. The target carried the comment this campaign set at creation:

felhom-c10   /mnt/5_hdd/backup/campaign10   "Campaign 10 scratch - DELETE at teardown"   <-- target
felhom-spike /mnt/5_hdd/backup/demo-felhom  (no comment)                                 <-- PRODUCTION

	before	after
datastores	felhom-c10, felhom-spike	felhom-spike only
users	c10@pbs, felhom@pbs, root@pam	felhom@pbs, root@pam
tokens	c10@pbs!box, felhom@pbs!n100	felhom@pbs!n100
ACLs	2 × /datastore/felhom-c10 + 2 × /datastore/felhom-spike	2 × /datastore/felhom-spike only
on disk	/mnt/5_hdd/backup/campaign10	removed; /mnt/5_hdd/backup/demo-felhom INTACT

Only the campaign's entries disappeared. (user delete is not a PBS subcommand — it printed usage and the user survived. Caught by re-reading the user list rather than trusting rc=0; the correct pair is user delete-token then user remove.)

4. Restic subaccount on the Storage Box — GONE, production untouched

Identity asserted programmatically before the DELETE — the script aborts unless both hold:

id=281530 username=u629488-sub4 home=felhom-campaign10 desc="Campaign 10 scratch - DELETE at teardown"
assert home == "felhom-campaign10"      PASSED
assert "Campaign 10" in description     PASSED

Its inability to reach any production path was already proven in isolation/restic-denial.txt.

	before	after
subaccounts	4	3
281530 (campaign)	present	absent
273581/269130/275124 (demo-felhom, peti-felhom, demo-hp)	present	all present

5. Hub customer c10-soak — GONE, after the ONLINE gate refused

The gate refused, exactly as anticipated. POST /configs/c10-soak/delete returned HTTP 409 for 18 consecutive attempts. Source: internal/web/customer_delete.go:98-102 sets onlineBlocked = true while hostStatus(LastReportAt) == "ok", and :146 comments "leg 1 refuses; decommission the agent first".

When it became deletable — predicted and confirmed. Last host-report 09:38:19; ok lapses at 30 minutes ⇒ deletable from 10:08:19. Measured:

09:50:17  attempt  1 -> HTTP 409
…
10:07:20  attempt 18 -> HTTP 409
10:08:23  attempt 19 -> HTTP 303      <-- 4 s after the predicted threshold

Verified positively, not by trusting the response:

customers list  — c10-soak / "Campaign 10" rows: 0
hosts+appliances — c10-soak / 55ab3776 rows:      0
GET /customers/c10-soak                        -> HTTP 404
hub: delete c10-soak: residue purged (reports=198 app_telemetry=1099 notif_prefs=1
     selfbind_tokens=1 appliance_registrations=1)
hub: customer DELETE cascade COMPLETE for c10-soak (journal #9) — full teardown

No separate /appliances/<id>/discard was needed: the appliance record was bound, so it left the unclaimed list at bind time and was purged by the cascade (appliance_registrations=1). The task anticipated /discard; on this path the cascade covered it.

Remaining customers, unchanged: Demo Ügyfél, Demo HP, Peti Proxmox, and R-120 golden 0.186.0 proof (= sess-f) — the pre-existing R-131 orphan, not this campaign's and deliberately not touched.

drill-r50 (R-93 fixture) verified intact: GET /customers/drill-r50 → HTTP 200, and VM 300 still present and stopped on demo-hp. (It is absent from the dashboard listing because it is DOWN, not because it is gone — checked explicitly rather than inferred from the listing.)

6. Credentials

All 12 files under ~/.config/campaign10/ shredded (shred -u) and the directory removed — host/dashboard passwords, API key, app secrets, PBS token, restic password, claim code. The guest-side copies (/root/c10/, /root/c10api.sh) died with the VM. No campaign processes remain.

---

Register work still owed — NOT done this session, and why

The parallel catalog session is OPEN. app-catalog-felhom.eu has an uncommitted working tree: CLAUDE.md and REUSE.md modified, plus untracked audits/, scripts/check-volume-persistence.py and scripts/test_check_volume_persistence.py. One session owns the register at a time, so no rows were written.

Still owed, for whoever takes the register next:

finding	needs
R-156	row + rank. papra's data neither persisted nor backed up, reports healthy. Note: the parallel session's check-volume-persistence.py looks like exactly this fix's shape — reconcile before filing.
R-157	row + rank. bootrecon's start-once sweep, two mechanisms (A: Exited/race ~50%; B: zero-container, silent on every channel). A settle-condition fix closes A only.
R-158	row + rank — ranked BELOW R-157 by Q1 (notification gap, not silent failure; visible on /backups, absent from /backups/apps).
spike C2	not a finding: a dependency chain (sound predicate → warn→gate → tar-drop), lengthened by Q2.

What did not happen, and why

• No register rows written — parallel session owns it (above).
• Nothing fixed — R-156/R-157/R-158 and everything Q1/Q2 surfaced are findings only, per the fences.
• No /appliances/<id>/discard call — there was no appliance row left to discard (§5).
• sess-f / R-131 orphan not deleted — not this campaign's record; deliberately out of scope.
• The ~5 GB linearity point — the earlier spike's brief said skip it; still skipped.