docs: CAMPAIGN 8 — GC run 2, fault 13 complete, final fleet state
GC run 2 (04:44Z, past the atime boundary) reclaimed 0 B, and the cause is the campaign itself: the datastore is relatime, so the night's three full verifies, ~14 backups, two GC runs and several restores kept refreshing chunk atimes. Sampled atimes confirm no chunk was ever 24h5m cold. Not a defect - GC made a correct, deliberate decision both times - but a real operational property: on a relatime datastore, activity defers reclamation. The spec's intended experiment therefore could not complete inside the window; recorded as such. Fault 13 now complete: backup-vs-restore-test single-flight holds, and a full two-tier backup ran successfully DURING GC run 2 (both artefacts landed, GC TASK OK, zero .bad chunks). True three-way overlap remains untested. Final fleet verification clean on all three hosts. Campaign-caused outage (F-REBOOT, 9m47s on demo-hp) stated plainly in the final-state section.
This commit is contained in:
@@ -165,7 +165,38 @@ correctly" from "did nothing". `Pending removals` is the positive observable —
|
||||
positively identified 3903 unreferenced chunks, and declined to unlink them because they were too
|
||||
young.
|
||||
|
||||
**Run 2:** see §10 (appended at ~06:40, after the 06:34 eligibility boundary).
|
||||
**Run 2 (04:44:06Z / 06:44 CEST, after the 04:34 UTC boundary, `TASK OK`):**
|
||||
```
|
||||
Removed garbage: 0 B Removed chunks: 0
|
||||
Pending removals: 6.38 GiB (in 5406 chunks) [run 1: 4.669 GiB / 3903]
|
||||
On-Disk usage: 6.697 GiB (17.11%) / 4891 chunks [run 1: 7.735 GiB / 5910]
|
||||
Deduplication factor: 5.84 [run 1: 4.45]
|
||||
```
|
||||
The live set SHRANK and pending removals GREW — the 03:30 UTC prune landing between the runs, with
|
||||
GC correctly re-classifying those chunks as unreferenced.
|
||||
|
||||
**I predicted run 2 would reclaim ≈4.669 GiB. It reclaimed nothing, and the cause is this
|
||||
campaign.** The datastore is mounted `relatime` (`findmnt -no OPTIONS` → `rw,relatime,discard`), so
|
||||
reading a chunk refreshes its atime — and the campaign read essentially every chunk repeatedly
|
||||
overnight (three full verifies for F17, ~14 backups, two GC runs, several restores). Sampled atimes
|
||||
at 04:55:38Z prove it:
|
||||
```
|
||||
188 chunks atime 2026-07-28 04:xx ← GC run 2 itself, touching referenced chunks (phase 1)
|
||||
155 chunks atime 2026-07-27 07:xx
|
||||
51 chunks atime 2026-07-27 21:xx ← a verify pass
|
||||
oldest atime in sample: 2026-07-27 07:18:53Z (21h37m — short of the 24h5m cutoff)
|
||||
```
|
||||
No chunk was ever 24h5m cold, so `Removed garbage: 0 B` is exactly correct on both runs.
|
||||
|
||||
**Not a defect** — GC made a deliberate, correct decision both times, and `Pending removals` is the
|
||||
positive observable proving it did the work. But it IS an operational property worth knowing: **on
|
||||
a `relatime` datastore, activity defers reclamation.** "GC ran, so space will come back" is unsafe
|
||||
on a small offsite box that something reads regularly.
|
||||
|
||||
**Honest limit:** because the campaign's own reads reset the clock, the spec's intended experiment —
|
||||
watching the migration chunks cross the boundary and be collected — **could not be completed inside
|
||||
the campaign window.** First genuine reclamation should follow ~07:23Z on 07-28. Recorded as an
|
||||
incomplete observation, not a pass.
|
||||
|
||||
---
|
||||
|
||||
@@ -185,7 +216,7 @@ young.
|
||||
| 10 | kill the controller mid-quiesce | **PASS** — crash recovery in **1 second**; and it corrects the premise (see below) | `F10-kill-controller-mid-quiesce.md` |
|
||||
| 11 | reboot the guest mid-backup | **FINDING (F-REBOOT)** — the backup succeeded but **the guest never came back**; found `stopped` with 0 containers, no self-heal, ~9m47s outage until manual `pct start` | `F11-reboot-guest-mid-backup.md` |
|
||||
| 12 | reboot the host mid-backup | **NOT TESTED** — see §9 | — |
|
||||
| 13 | backup + restore-test + GC concurrently | **HALF PASS** — backup vs restore-test single-flight HOLDS, proven live and unplanned. The +GC combination was not run | `F-A1-*.md` |
|
||||
| 13 | backup + restore-test + GC concurrently | **PASS on both tested axes** — backup vs restore-test single-flight HOLDS; backup vs GC coexist (both tiers completed during GC run 2). True three-way overlap not tested | `F13-concurrency.md` |
|
||||
| 14 | two quiesce triggers at once | **PASS** — `TryLock` + skip, exactly one cycle ran | `F14-two-triggers.md` |
|
||||
| 15 | clock skew (+6 h) | **PASS** — gate follows wall clock (correct); **breaker is skew-proof** via Go monotonic time | `F15-clock-skew.md` |
|
||||
| 16 | `age_state=absent` (never observed live) | **PASS** — valve fired, cycle ran OUTSIDE a closed window on a genuinely never-backed-up tier | `F16-age-state-absent.md` |
|
||||
@@ -363,6 +394,24 @@ One campaign artefact was removed as part of restoring service, not merely tidin
|
||||
snapshot. Leaving it would have left demo-felhom's PBS tier silently suppressed for seven days —
|
||||
F-CRIT-2's live effect.
|
||||
|
||||
### Final verification at 04:57:33Z / 06:57 CEST — after EVERY fault, including the two
|
||||
### post-Phase-D re-injections (faults 18 and 11)
|
||||
```
|
||||
[demo-felhom] agent=active guest=running containers=15 unhealthy=0 scratch=0 nft_c8=0 ballast=0 ntp=yes
|
||||
[demo-hp] agent=active guest=running containers=8 unhealthy=0 scratch=0 nft_c8=0 ballast=0 ntp=yes
|
||||
[ep0] pbs=active bad_chunks=0 ballast=0 rollback_copy=PRESENT df=14G/98G 14%
|
||||
```
|
||||
|
||||
**One outage the campaign caused, stated plainly:** fault 11 left demo-hp guest 9201 powered off
|
||||
for **~9m47s** (03:43:13Z → 03:53:00Z) before I noticed and ran `pct start 9201`. That is finding
|
||||
F-REBOOT, and it is the longest unplanned downtime of the run. Recovery needed no lock clearing and
|
||||
all 8 containers returned within ~25 s. Everything else was reverted without incident.
|
||||
|
||||
Two re-injections were performed AFTER Phase D because clock-bound work (the 03:30 UTC prune and
|
||||
GC run 2) left time available. Both are declared rather than hidden: fault 18 needed a bounded
|
||||
restore-test cadence change on demo-hp (reverted, re-verified), and fault 11 needed no config
|
||||
change at all. The sweep above is the state after all of it.
|
||||
|
||||
---
|
||||
|
||||
## 9. What could NOT be tested, and why
|
||||
@@ -381,7 +430,7 @@ F-CRIT-2's live effect.
|
||||
- ~~Fault 18~~ — **was run** in the campaign's tail (see the matrix). It required a single, declared
|
||||
re-injection after Phase D (restore-test cadence on demo-hp only, 84h→600s), which was reverted
|
||||
and Phase D re-verified immediately afterwards.
|
||||
- **Fault 13's GC leg** — the backup-vs-restore-test half was proven; adding GC was not run.
|
||||
- **Fault 13's true THREE-way overlap** (backup + restore-test + GC simultaneously). Legs A and B were each proven, never together.
|
||||
- **Fault 6** — inconclusive: ext4's 5% root reserve (2.07 GB, confirmed via `tune2fs`) meant the
|
||||
filesystem was never truly full for a root-run vzdump. Driving a Proxmox host's root filesystem
|
||||
to genuine zero risks wedging `pmxcfs`; not attempted.
|
||||
|
||||
Reference in New Issue
Block a user