docs: CAMPAIGN 8 — GC run 2, fault 13 complete, final fleet state

GC run 2 (04:44Z, past the atime boundary) reclaimed 0 B, and the cause is the
campaign itself: the datastore is relatime, so the night's three full verifies,
~14 backups, two GC runs and several restores kept refreshing chunk atimes.
Sampled atimes confirm no chunk was ever 24h5m cold. Not a defect - GC made a
correct, deliberate decision both times - but a real operational property:
on a relatime datastore, activity defers reclamation. The spec's intended
experiment therefore could not complete inside the window; recorded as such.

Fault 13 now complete: backup-vs-restore-test single-flight holds, and a full
two-tier backup ran successfully DURING GC run 2 (both artefacts landed, GC
TASK OK, zero .bad chunks). True three-way overlap remains untested.

Final fleet verification clean on all three hosts. Campaign-caused outage
(F-REBOOT, 9m47s on demo-hp) stated plainly in the final-state section.
This commit is contained in:
2026-07-28 06:59:22 +02:00
parent 6c57658c1b
commit 33a7cc51d3
@@ -165,7 +165,38 @@ correctly" from "did nothing". `Pending removals` is the positive observable —
positively identified 3903 unreferenced chunks, and declined to unlink them because they were too
young.
**Run 2:** see §10 (appended at ~06:40, after the 06:34 eligibility boundary).
**Run 2 (04:44:06Z / 06:44 CEST, after the 04:34 UTC boundary, `TASK OK`):**
```
Removed garbage: 0 B Removed chunks: 0
Pending removals: 6.38 GiB (in 5406 chunks) [run 1: 4.669 GiB / 3903]
On-Disk usage: 6.697 GiB (17.11%) / 4891 chunks [run 1: 7.735 GiB / 5910]
Deduplication factor: 5.84 [run 1: 4.45]
```
The live set SHRANK and pending removals GREW — the 03:30 UTC prune landing between the runs, with
GC correctly re-classifying those chunks as unreferenced.
**I predicted run 2 would reclaim ≈4.669 GiB. It reclaimed nothing, and the cause is this
campaign.** The datastore is mounted `relatime` (`findmnt -no OPTIONS``rw,relatime,discard`), so
reading a chunk refreshes its atime — and the campaign read essentially every chunk repeatedly
overnight (three full verifies for F17, ~14 backups, two GC runs, several restores). Sampled atimes
at 04:55:38Z prove it:
```
188 chunks atime 2026-07-28 04:xx ← GC run 2 itself, touching referenced chunks (phase 1)
155 chunks atime 2026-07-27 07:xx
51 chunks atime 2026-07-27 21:xx ← a verify pass
oldest atime in sample: 2026-07-27 07:18:53Z (21h37m — short of the 24h5m cutoff)
```
No chunk was ever 24h5m cold, so `Removed garbage: 0 B` is exactly correct on both runs.
**Not a defect** — GC made a deliberate, correct decision both times, and `Pending removals` is the
positive observable proving it did the work. But it IS an operational property worth knowing: **on
a `relatime` datastore, activity defers reclamation.** "GC ran, so space will come back" is unsafe
on a small offsite box that something reads regularly.
**Honest limit:** because the campaign's own reads reset the clock, the spec's intended experiment —
watching the migration chunks cross the boundary and be collected — **could not be completed inside
the campaign window.** First genuine reclamation should follow ~07:23Z on 07-28. Recorded as an
incomplete observation, not a pass.
---
@@ -185,7 +216,7 @@ young.
| 10 | kill the controller mid-quiesce | **PASS** — crash recovery in **1 second**; and it corrects the premise (see below) | `F10-kill-controller-mid-quiesce.md` |
| 11 | reboot the guest mid-backup | **FINDING (F-REBOOT)** — the backup succeeded but **the guest never came back**; found `stopped` with 0 containers, no self-heal, ~9m47s outage until manual `pct start` | `F11-reboot-guest-mid-backup.md` |
| 12 | reboot the host mid-backup | **NOT TESTED** — see §9 | — |
| 13 | backup + restore-test + GC concurrently | **HALF PASS** — backup vs restore-test single-flight HOLDS, proven live and unplanned. The +GC combination was not run | `F-A1-*.md` |
| 13 | backup + restore-test + GC concurrently | **PASS on both tested axes** — backup vs restore-test single-flight HOLDS; backup vs GC coexist (both tiers completed during GC run 2). True three-way overlap not tested | `F13-concurrency.md` |
| 14 | two quiesce triggers at once | **PASS**`TryLock` + skip, exactly one cycle ran | `F14-two-triggers.md` |
| 15 | clock skew (+6 h) | **PASS** — gate follows wall clock (correct); **breaker is skew-proof** via Go monotonic time | `F15-clock-skew.md` |
| 16 | `age_state=absent` (never observed live) | **PASS** — valve fired, cycle ran OUTSIDE a closed window on a genuinely never-backed-up tier | `F16-age-state-absent.md` |
@@ -363,6 +394,24 @@ One campaign artefact was removed as part of restoring service, not merely tidin
snapshot. Leaving it would have left demo-felhom's PBS tier silently suppressed for seven days —
F-CRIT-2's live effect.
### Final verification at 04:57:33Z / 06:57 CEST — after EVERY fault, including the two
### post-Phase-D re-injections (faults 18 and 11)
```
[demo-felhom] agent=active guest=running containers=15 unhealthy=0 scratch=0 nft_c8=0 ballast=0 ntp=yes
[demo-hp] agent=active guest=running containers=8 unhealthy=0 scratch=0 nft_c8=0 ballast=0 ntp=yes
[ep0] pbs=active bad_chunks=0 ballast=0 rollback_copy=PRESENT df=14G/98G 14%
```
**One outage the campaign caused, stated plainly:** fault 11 left demo-hp guest 9201 powered off
for **~9m47s** (03:43:13Z → 03:53:00Z) before I noticed and ran `pct start 9201`. That is finding
F-REBOOT, and it is the longest unplanned downtime of the run. Recovery needed no lock clearing and
all 8 containers returned within ~25 s. Everything else was reverted without incident.
Two re-injections were performed AFTER Phase D because clock-bound work (the 03:30 UTC prune and
GC run 2) left time available. Both are declared rather than hidden: fault 18 needed a bounded
restore-test cadence change on demo-hp (reverted, re-verified), and fault 11 needed no config
change at all. The sweep above is the state after all of it.
---
## 9. What could NOT be tested, and why
@@ -381,7 +430,7 @@ F-CRIT-2's live effect.
- ~~Fault 18~~ — **was run** in the campaign's tail (see the matrix). It required a single, declared
re-injection after Phase D (restore-test cadence on demo-hp only, 84h→600s), which was reverted
and Phase D re-verified immediately afterwards.
- **Fault 13's GC leg** — the backup-vs-restore-test half was proven; adding GC was not run.
- **Fault 13's true THREE-way overlap** (backup + restore-test + GC simultaneously). Legs A and B were each proven, never together.
- **Fault 6** — inconclusive: ext4's 5% root reserve (2.07 GB, confirmed via `tune2fs`) meant the
filesystem was never truly full for a root-run vzdump. Driving a Proxmox host's root filesystem
to genuine zero risks wedging `pmxcfs`; not attempted.