CAMPAIGN-11 Phase 4: the soak passed, with two absences answered honestly
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
All five daily jobs fired exactly once and completed on time. The 04:15 offsite run produced snapshot_count 1 -> 2, last_status ok, 24.9s — unprompted, on a box rebuilt twice and set aside four hours earlier. Nothing on the must-not list fired, including offsiteheal: that is R-218's NEGATIVE control passing (a box with a target must not declare), which does not substitute for its positive half, still owed. Investigated and DISPROVED: tier2-backup completing in 118ms looked like a silent no-op; the copy is real and on the backup drive (818.5 KB verified). Correction to my own pre-registration: backup_run_digest is a TEST FILENAME, not an event type. The real one is backup_run_failures, a failures digest whose silence on a clean night is correct — reporting it as a miss would have been a finding I invented. What survives: the offsite run emits no hub event while both lesser tiers announce success; failures and staleness are covered elsewhere, so it is a consistency wrinkle, recorded not filed. Two absences answered rather than assumed: the restore-test's silence was PRE-REGISTERED as correct (6h eval, 24h settle); the agent's whole-guest tier is left explicitly UNRESOLVED because routine local-api calls are not logged at INFO — a 5h search returns 0 on a box that demonstrably served such calls, so the instrument cannot answer. No product code changed.
This commit is contained in:
@@ -705,3 +705,131 @@ backup: restore-test scheduler starting (per-archive due-check) eval_interval=6
|
||||
- Whether the agent's local tier fires tonight depends on when its 24 h cadence last elapsed on a box
|
||||
built today.
|
||||
|
||||
|
||||
## Phase 4 — results, against the pre-registration
|
||||
|
||||
**Window observed: 2026-08-05 23:56 → 2026-08-06 04:35 CEST.** The venue was not touched: no endpoint
|
||||
driven, no service restarted, nothing configured. **30 five-minute samples** of the box's own
|
||||
`settings.json` and the hub's host row, plus a full log census at the end.
|
||||
|
||||
### What fired — and every daily job fired exactly once, on time
|
||||
|
||||
Census taken with **no `tail`** (the documented "tail silently drops rows" trap — the first checkpoint
|
||||
hit it and was re-run):
|
||||
|
||||
| job | scheduled | ran | completed | evidence |
|
||||
|---|---|---|---|---|
|
||||
| `db-dump` | 02:30 | **1** | **1** | `00:30:00Z Running` → `00:30:05Z completed (took 5.237s)` |
|
||||
| `tier2-backup` | 03:30 | **1** | **1** | `01:30:00Z` → `completed (118ms)` |
|
||||
| `fill-watch` | 03:30 | **1** | **1** | `01:30:00Z` → `completed (0s)` |
|
||||
| `metrics-prune` | 04:00 | **1** | **1** | — |
|
||||
| **`offbox-backup`** | **04:15** | **1** | **1** | `02:15:00Z Running` → `02:15:24Z completed (took 24.884s)` |
|
||||
|
||||
**The one this campaign cares about, verbatim:**
|
||||
|
||||
```
|
||||
02:15:00 [scheduler] Running job: offbox-backup
|
||||
02:15:00 [offbox] backup run started (1 app(s) toggled)
|
||||
02:15:05 [offbox] pre-push dump leg completed in 5.203s — snapshot pair is coherent
|
||||
02:15:14 [offbox] backed up calibre-web (…/backups/primary/calibre-web, 1 mandatory path(s))
|
||||
02:15:24 [offbox] backup OK: 1 app(s) backed up, 2 snapshot(s), 18s
|
||||
```
|
||||
|
||||
**Pre-registered prediction met exactly:** `snapshot_count` **1 → 2**, `last_status: ok`,
|
||||
`repo_size_bytes` 27 019 → **51 206**, `last_success 2026-08-06T02:15:24Z`. **The off-site tier runs
|
||||
itself, unprompted, on a box that was rebuilt twice and had its repository set aside hours earlier.**
|
||||
|
||||
Two events reached the hub, and both are the tier announcing success:
|
||||
|
||||
```
|
||||
02:30:05 Event from c11: db_dump_completed (info) — Adatbázis mentés elkészült
|
||||
03:30:00 Event from c11: crossdrive_completed (info) — Másodlagos mentés elkészült: calibre-web
|
||||
```
|
||||
|
||||
### Investigated and DISPROVED — `tier2-backup` completing in 118 ms
|
||||
|
||||
118 ms looked like a scheduled backup that silently no-ops. **It is not.** The copy is real and on the
|
||||
disk:
|
||||
|
||||
```
|
||||
[backup] Tier 2 copied calibre-web → /mnt/felhom-drives/mentes/backups/secondary/calibre-web
|
||||
(818.5 KB, 1 leg(s), 0s)
|
||||
[backup] Tier 2 run complete: 1 app(s) processed (incl. volume-only — F6)
|
||||
|
||||
/mnt/felhom-drives/mentes/backups/secondary/calibre-web ← present on the BACKUP drive
|
||||
```
|
||||
|
||||
388 KB of source on local NVMe in 118 ms is honest. **No finding.**
|
||||
|
||||
### What must NOT have fired — and none of it did
|
||||
|
||||
Counted over the whole window, box side and hub side:
|
||||
|
||||
| | box | hub |
|
||||
|---|---|---|
|
||||
| `needs_credential` | **0** | — |
|
||||
| `offsiteheal` / `offsite_selfheal_restaged` | **0** | **0** |
|
||||
| `offbox_repo_orphaned` | **0** in-window | **0** |
|
||||
| `offsite_repo_key_changed` | — | **0** |
|
||||
| `escrow blob SERVED` | — | **0** |
|
||||
| `host_stale` / `host_recovered` | — | **0** |
|
||||
| any controller self-update | **0** | — |
|
||||
| `Operator email sent` for c11 | — | **0** |
|
||||
|
||||
> **§4.2's negative control PASSES.** A box that HAS an off-site target did **not** declare
|
||||
> `needs_credential` even once in five hours, and the hub's self-heal never fired. R-218's fix is not
|
||||
> over-firing in the other direction. This does not substitute for its positive half, which is still
|
||||
> owed.
|
||||
|
||||
The three `orphan` hits in the box's log were placed in time rather than counted blind: all three are
|
||||
**Phase 2's own set-aside at 21:30–21:32Z**, before the window opened. **Zero after.**
|
||||
|
||||
### A correction to my own pre-registration — item 4 was wrong
|
||||
|
||||
I pre-registered *"a `backup_run_digest` event reaches the hub after a backup run"*. **There is no such
|
||||
event type.** `backup_run_digest` is a **test filename** (`internal/notify/backup_run_digest_test.go`),
|
||||
which is where I took it from — a census of the hub's `allowedEventTypes` shows the real one is
|
||||
**`backup_run_failures`**, a *failures* digest. **Its silence on a clean night is correct**, and
|
||||
reporting it as a miss would have been a finding invented by my own bad reading.
|
||||
|
||||
**One real observation survives that correction.** The off-site run emitted **no hub event at all**,
|
||||
while both lesser tiers announced themselves (`db_dump_completed`, `crossdrive_completed`). So the
|
||||
operator's event stream carries a nightly "DB dump done" and "second copy done" and **nothing** when
|
||||
the tier the whole campaign is about succeeds. **Mitigations exist and are named rather than assumed:**
|
||||
failures are covered by `backup_run_failures`, and off-site staleness by the hub's own tier deadline
|
||||
monitor (`deadline_tiers.go`, `offsiteBackupStaleAfter = 8 days`). **A consistency wrinkle, not a blind
|
||||
spot** — recorded, not filed.
|
||||
|
||||
### The two genuinely uncertain items, resolved honestly
|
||||
|
||||
- **The restore-test did NOT run — and that is correct, as pre-registered.** `eval_interval=6h` with
|
||||
the scheduler armed 23:29:43, so the first evaluation is ~05:29; and `settle=24h` on archives hours
|
||||
old. **Predicted not to run, did not run.** Had this not been pre-registered it would have read as a
|
||||
gap.
|
||||
- **The agent's whole-guest tier did NOT run, and whether it should have is NOT RESOLVABLE with the
|
||||
instruments available.** Stated as an open question rather than scored either way:
|
||||
- Measured: the box reports **`0 backups`** in every host-report all night (03:59:47, 04:14:47), and
|
||||
no whole-guest backup, vzdump, snapshot or quiesce line appears in the agent journal.
|
||||
- **The agent is emphatically alive** — 67 251 journal lines since 00:00, **2 091 of them from
|
||||
`felhom-agent` itself**, and it was polling the guest at 04:17:57.
|
||||
- **But the instruments cannot answer the question.** Routine local-api requests are not logged at
|
||||
INFO: a search for `local-api` over five hours returns **0** on a box that demonstrably served
|
||||
`local-api: recovering the offsite repository password` earlier in the session. So "0 `/backup/due`
|
||||
polls" is **not evidence that no poll happened**. The hub's deadline monitor is equally invisible —
|
||||
a census for `deadline|backup_missing|tier` across all customers returns **0** at INFO.
|
||||
- **What can be said:** no alarm fired for c11 all night other than F11's own, and the tier is armed
|
||||
with a 24 h cadence on a box whose local tier has never run. **Whether a first-ever whole-guest
|
||||
backup was due and skipped, or correctly not yet due, needs an instrument that does not exist
|
||||
today.** → recorded as an open question, not as a finding.
|
||||
|
||||
### Phase 4 verdict
|
||||
|
||||
**The machine ran a full scheduled cycle correctly and unaided.** Five of five daily jobs fired on
|
||||
time and completed; the off-site tier produced a second snapshot by itself; the two local tiers
|
||||
reported success to the operator; and **nothing fired that should not have** — including the
|
||||
self-heal, which is R-218's negative control.
|
||||
|
||||
**The one thing a soak like this cannot do is prove the absence of an alarm was right**, and where that
|
||||
was true it is said so: the restore-test's silence was predicted in advance, and the whole-guest
|
||||
tier's silence is left explicitly unresolved rather than counted as a pass.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user