F-LEAK's first fix (pool adoption) was refuted live and is recorded as such. R-100 is
F-CRIT-2's defect class on the hub for the restic tier: isStale reads only LastRun, which
the controller writes unconditionally on failure. Investigated, not fixed.
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three
of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).
Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md:
a comment asserting an invariant needs a test pinning it, or it is a wish — with
all six shipped-false-guarantee instances catalogued, and the corollary that a
test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM
(does suppression expire?).
F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only
plausibly-complete entries (measured 1 MiB floor), verified by replaying
campaign fault 2 on demo-hp in both directions.
Files two follow-ups:
- R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last
(dry-run kept 2 real + the phantom), so there is no retention/data-loss bug —
but they accumulate one per aborted upload.
- F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by
this fix.
GC run 2 (04:44Z, past the atime boundary) reclaimed 0 B, and the cause is the
campaign itself: the datastore is relatime, so the night's three full verifies,
~14 backups, two GC runs and several restores kept refreshing chunk atimes.
Sampled atimes confirm no chunk was ever 24h5m cold. Not a defect - GC made a
correct, deliberate decision both times - but a real operational property:
on a relatime datastore, activity defers reclamation. The spec's intended
experiment therefore could not complete inside the window; recorded as such.
Fault 13 now complete: backup-vs-restore-test single-flight holds, and a full
two-tier backup ran successfully DURING GC run 2 (both artefacts landed, GC
TASK OK, zero .bad chunks). True three-way overlap remains untested.
Final fleet verification clean on all three hosts. Campaign-caused outage
(F-REBOOT, 9m47s on demo-hp) stated plainly in the final-state section.
Tail-of-campaign additions after the Phase D revert (both re-injections declared):
- fault 18 (delete a snapshot mid restore-test): detection PASS, and it ROOT-CAUSES
F-LEAK — a failed restore-test cannot destroy its own scratch guest (403,
missing VM.Allocate; the agent token is pool-scoped and a failed restore never
joins the felhom pool)
- fault 11 (guest reboot mid-backup): new finding F-REBOOT — the backup succeeds
but the guest never comes back; ~9m47s outage until a manual pct start
Two evidence corrections, both self-inflicted tooling errors:
- pgrep -cf <pattern> matches its own ssh command line, which invalidated fault
11's first two injections and put one unsound line in fault 9 (withdrawn; that
finding stands on the controller's own job state)
- ep0 runs Etc/UTC, so its 03:30 prune fires at 05:30 CEST — nearly misread as a
broken prune job
Nine findings now, still two HIGH. Fleet healthy.
Unattended 10h run against demo-felhom, demo-hp and ep0. No production code
changed; findings recorded and ranked, not fixed inline.
8 findings, 2 HIGH — both in the system's ability to report that a backup did
NOT happen:
- F-CRIT-1: an app failing to restart after a quiesce never alarms (invariant
I1 in main.go:1213 is false for the failed-restart path)
- F-CRIT-2: a failed offsite backup leaves a phantom snapshot that resets the
tier's freshness clock (NewestArchiveTime has no completeness check)
Retires several never-validated items, including R-87 (first restic restore
round-trip, byte-verified), the full R-88 backoff ladder, age_state=absent,
and the crash-recovery unquiesce under a real SIGKILL.
peti-felhom untouched; ep0 rollback copy intact; fleet healthy at end.
admin