diff --git a/documentation/audits/evidence-chaos-night-2026-09-17/ep0-baseline.txt b/documentation/audits/evidence-chaos-night-2026-09-17/ep0-baseline.txt new file mode 100644 index 00000000..16bbde7e --- /dev/null +++ b/documentation/audits/evidence-chaos-night-2026-09-17/ep0-baseline.txt @@ -0,0 +1,17 @@ +# ep0 off-site store - BASELINE taken 2026-09-17T00:17:15Z, READ ONLY +# Purpose: the teardown must show the off-site backups STAYED and nothing was removed. +# One listing at the end cannot prove that. This is the "before" half of the pair. +# Nothing was written, nothing pruned, and the .chunks store was deliberately not walked. + +host felhom-hetzner (167.233.158.164, reached from DooPlex) +datastore felhom-offsite -> /mnt/pbs-datastore + +namespaces demo-felhom, demo-hp, tester-1 +backup groups ns/demo-felhom/ct/9201 + ns/demo-hp/ct/9201 + ns/tester-1/ct/9201 <- tonight's box (tester-1-022354) +snapshots 2 in EACH group, 6 in total +disk /dev/sdb 98G total, 16G used, 83G available, 16% used + +At teardown this listing is repeated. The expected result is that every group and snapshot above is +still present; the tester-1 group may have MORE snapshots if a backup ran, and must never have fewer. diff --git a/documentation/audits/evidence-chaos-night-2026-09-17/phase2-readiness.txt b/documentation/audits/evidence-chaos-night-2026-09-17/phase2-readiness.txt new file mode 100644 index 00000000..5ddf28d4 --- /dev/null +++ b/documentation/audits/evidence-chaos-night-2026-09-17/phase2-readiness.txt @@ -0,0 +1,29 @@ +# Phase 2 readiness, checked DURING round 12 without touching the box under test +# 2026-09-17T00:15-00:16Z + +## Scratch guest 9202 is ready to receive an off-site restore + pct list 9201 demo-hp (running), 9202 demo-hp-scratch (running) + 9202 containers felhom-controller Up 5 hours (healthy), filebrowser Up 3 days, traefik Up 3 days + restic 0.14.0, INSIDE the controller container (as the design requires - every path it + touches must be under /mnt) + off-site target NOT configured on 9202 + +## The constraint that follows, found before it could waste time at 05:00 +To restore one DB-backed app from off-site onto 9202, 9202 must be pointed at the BOX'S restic +repository - which needs that repository's address and its password. The password is minted per +guest, so 9202 cannot derive it; it has to be read from the box and handed over. + +## A DELIBERATE DECISION: I am not reading that from the box yet, and why +Round 12 is the closing CONTROL round. Its entire value is that nothing was done to the box while +it ran. Authenticated reads against the controller during it would add load and log lines to the one +round whose meaning is "nothing happened". The read costs nothing to defer and the round cannot be +re-run, so it waits until round 12 closes. +Recorded here rather than left as an unstated habit, because a control round that was quietly +touched is worth nothing and nobody could tell afterwards. + +## My own instrument slipped again getting this far (the eighth) +The first version of this check drowned in perl locale warnings from `pct`, and my `head -12` cut +the output before the real lines were ever reached. It returned an empty marker block that looked +like "9202 has no containers". The fix was LC_ALL=C on the remote plus filtering the warning lines, +and not truncating before the marker. Same class as every other instrument fault tonight: a probe +that can drop its own result silently is not a measurement.