docs: CAMPAIGN 8 — adversarial backup & restore campaign report
Unattended 10h run against demo-felhom, demo-hp and ep0. No production code changed; findings recorded and ranked, not fixed inline. 8 findings, 2 HIGH — both in the system's ability to report that a backup did NOT happen: - F-CRIT-1: an app failing to restart after a quiesce never alarms (invariant I1 in main.go:1213 is false for the failed-restart path) - F-CRIT-2: a failed offsite backup leaves a phantom snapshot that resets the tier's freshness clock (NewestArchiveTime has no completeness check) Retires several never-validated items, including R-87 (first restic restore round-trip, byte-verified), the full R-88 backoff ladder, age_state=absent, and the crash-recovery unquiesce under a real SIGKILL. peti-felhom untouched; ep0 rollback copy intact; fleet healthy at end.
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
# REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
|
||||
|
||||
Adversarial, destructive, unattended run against `demo-felhom`, `demo-hp` and `ep0`.
|
||||
Full report: `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
Evidence: `DooPlex:~/campaign8/evidence/` (103 files, 35 MB, written continuously by 11 collectors).
|
||||
|
||||
**No production code was changed.** Findings are recorded and ranked, never fixed inline, per the
|
||||
campaign's own rules.
|
||||
|
||||
## Scope safety
|
||||
`peti-felhom`, its namespace and `u629488-sub2` were never touched. Phase 0 established with five
|
||||
documented probes that peti has **no data at all** in `felhom-offsite`, which is what made the
|
||||
operator-approved 100% datastore-fill safe. The 13 GB rollback copy `/srv/pbs-felhom` on ep0 is
|
||||
intact.
|
||||
|
||||
## Findings
|
||||
|
||||
| # | Finding | Severity | Class |
|
||||
|---|---|---|---|
|
||||
| F-CRIT-1 | An app that fails to restart after a quiesce **never alarms**, on any channel | **HIGH** | DEFECT |
|
||||
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
|
||||
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
|
||||
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
|
||||
| F-LEAK | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT |
|
||||
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
|
||||
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
|
||||
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
|
||||
|
||||
Both HIGH findings are in the same place: **the system's ability to tell you a backup did not
|
||||
happen.** Both cite the code and the comment that asserts the property the code does not provide.
|
||||
|
||||
## What is now proven that was not before
|
||||
R-88 breaker arming **and its full ladder** (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
|
||||
real one-tier-fails case · `whole_guest_backup_failed` end-to-end with correct tier attribution ·
|
||||
R-97c operator-only routing verified against the hub DB (zero customer rows with `status='sent'`) ·
|
||||
`whole_guest_backup_recovered` + the R-68 pairing gate firing live · **`age_state=absent`** ·
|
||||
R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
|
||||
**R-87 — the first restic restore round-trip ever performed**, byte-verified (6/7 sha256 identical,
|
||||
the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
|
||||
|
||||
## Restore round-trips
|
||||
restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
|
||||
cleanly. `mount_parity` exact on both whole-guest tiers, `unprivileged: 1` preserved.
|
||||
|
||||
## Fleet state
|
||||
**Healthy. Nothing left broken.** All four compression knobs reverted and verified; every fault
|
||||
unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
|
||||
zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
|
||||
|
||||
## Not tested (with reasons)
|
||||
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
|
||||
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), faults 11/12
|
||||
(reboots — reasoned skip after F9/F10 covered hard-kill crash safety), fault 18, fault 13's GC leg,
|
||||
and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for documented reasons.
|
||||
|
||||
## Note on repo conventions
|
||||
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
|
||||
entry — there is nothing shipped to log. The deliverable is the audit document plus this report.
|
||||
@@ -0,0 +1,399 @@
|
||||
# CAMPAIGN 8 — the backup & restore subsystem
|
||||
**Adversarial, destructive, unattended. 2026-07-27 19:00 → 2026-07-28 ~06:45 CEST.**
|
||||
|
||||
Nodes exercised: `demo-felhom` (+ guest 9201), `demo-hp` (+ guest 9201), `ep0`
|
||||
(`felhom-hetzner`, datastore `felhom-offsite`), restic subaccounts `u629488-sub1` / `u629488-sub3`.
|
||||
|
||||
**`peti-felhom` was never touched.** Neither its data, its namespace, nor `u629488-sub2`. Phase 0
|
||||
established with five documented probes that peti has NO data in `felhom-offsite` at all (see §1),
|
||||
which is what made the operator-approved datastore-fill safe. The 13 GB rollback copy at
|
||||
`/srv/pbs-felhom` on ep0 was never deleted and is present at the end of the run.
|
||||
|
||||
Evidence: **`DooPlex:~/campaign8/evidence/`** — 103 files, 35 MB, written continuously by 11
|
||||
collectors throughout. Nothing in this report is reconstructed after the fact.
|
||||
|
||||
---
|
||||
|
||||
## Headline
|
||||
|
||||
Six findings, two of them HIGH severity, both in the same place: **the system's ability to tell you
|
||||
that a backup did not happen.**
|
||||
|
||||
- **F-CRIT-1** — an app that fails to restart after a quiesce **never alarms, on any channel**.
|
||||
- **F-CRIT-2** — a failed offsite backup leaves a phantom snapshot that **resets the tier's
|
||||
freshness clock**, silently suppressing the next backup for up to 7 days on production cadences.
|
||||
|
||||
Everything the campaign was asked to prove about the *happy* path proved out, including the
|
||||
first-ever restic restore round-trip (R-87). The mechanisms designed to catch failure — R-88's
|
||||
breaker, R-97a/c's routing, verify, quarantine, crash recovery — all work, and several are better
|
||||
than documented. The gaps are all in the seams *between* those mechanisms.
|
||||
|
||||
---
|
||||
|
||||
## 1. Phase 0 — questions, answers, and skips
|
||||
|
||||
Asked in one batch inside the operator's window, before touching anything.
|
||||
|
||||
| # | Question | Answer received |
|
||||
|---|---|---|
|
||||
| 1 | How may ep0 be taken down? | **Path-blocking + stop the PBS daemon.** No power-cycle. |
|
||||
| 2 | May VM 300 `drill-r50` be destroyed? | **Free to destroy.** |
|
||||
| 3 | Fill the `felhom-offsite` volume? | **Fill to 100%.** |
|
||||
| 4 | End-of-run bar? | **A broken test node is acceptable if precisely reported.** |
|
||||
|
||||
**Faults skipped for lack of an answer: none.** All four were answered in the window.
|
||||
|
||||
One question was NOT put to the operator because it was answerable from configuration, and was
|
||||
verified live instead: the restic subaccount mapping. Confirmed from each guest's `settings.json`:
|
||||
demo-felhom = `u629488-sub1`, demo-hp = `u629488-sub3`. `u629488-sub2` appears in neither box's
|
||||
configuration and was never addressed.
|
||||
|
||||
**The peti-exposure check (done before asking, so question 3 could be answered honestly).** What
|
||||
was tried, per standing rule 2:
|
||||
1. `find /mnt/pbs-datastore/ns -maxdepth 3 -type d` → only `demo-felhom`, `demo-hp`
|
||||
2. `ls -la /mnt/pbs-datastore/` → no depth-0 group dirs
|
||||
3. `ls -d /mnt/pbs-datastore/{vm,ct,host}/*` → no such paths
|
||||
4. `proxmox-backup-manager user list` → `felhom@pbs`, `root@pam` only
|
||||
5. `proxmox-backup-manager prune-job list` → jobs for demo-* namespaces only
|
||||
|
||||
**Incidental security note, reported at the time:** reading `/etc/felhom-agent/agent.json` printed
|
||||
`hub.api_key` in cleartext into the session transcript (the redactor matched `*token*`/`*secret*`/
|
||||
`*password*` but not the bare `api_key`). No secret was written to disk or to any campaign file;
|
||||
the operator may wish to rotate. All later captures used a stricter redactor.
|
||||
|
||||
---
|
||||
|
||||
## 2. Compression applied, and the revert
|
||||
|
||||
| Knob | Real | Campaign | Ratio |
|
||||
|---|---|---|---|
|
||||
| local (vzdump) cadence | 24 h (via the `0`⇒24h default, `config.go:517-520`) | 1800 s | 48× |
|
||||
| PBS offsite cadence | 604800 s (168 h) | 7200 s | 84× |
|
||||
| restore-test cadence | 302400 s (84 h) | 2700 s | 112× |
|
||||
| backup window start W | `02:30` | rolled 17:30 → 21:30 → (14:00 on demo-hp for F16) | — |
|
||||
|
||||
Feasibility was measured before choosing, not assumed: demo-felhom's local vzdump is 5.7 GB /
|
||||
`00:03:12`, demo-hp's PBS backup 1.5 GB / `00:00:55`. Both fit a 30-minute cadence with margin.
|
||||
|
||||
**"Widen the backup window" was not possible.** The gate is `[W+2h, W+6h)` and both offsets are
|
||||
compile-time constants (`backupwindow.go:19-21`); only W is settable. The gate is permanently 4 h
|
||||
wide against a ~10.5 h campaign, so the window was **moved** in contiguous steps rather than
|
||||
widened — which had the side benefit of exercising the gate at three different absolute times.
|
||||
|
||||
**A compression the spec asked for that configuration cannot deliver:** "restic daily → 30m". The
|
||||
three app-data legs are `Daily` scheduler jobs pinned to fixed offsets from the same W
|
||||
(`main.go:595,632,665`). Moving W reschedules them; it cannot make them 30-minutely. Restic was
|
||||
therefore driven by explicit manual triggers, and **no 30-minute restic cadence was ever in force.**
|
||||
|
||||
**Revert:** all four knobs restored and verified (§8). demo-hp's `backup_window_start` was restored
|
||||
to genuine ABSENCE from its pre-campaign snapshot rather than written as "02:30", because absent is
|
||||
what it was.
|
||||
|
||||
### The pre-registered artifact list
|
||||
Declared BEFORE Phase A so findings could be classified honestly:
|
||||
1. R-88 breaker cap 4 h — chosen against a 24 h cadence. **ARTIFACT** under compression.
|
||||
2. R-88 ladder base 15 m — chosen to exceed the 5 m poll. **ARTIFACT**.
|
||||
3. Window-gate valve "cadence + 24 h" — fires at 24.5 h instead of 48 h. **ARTIFACT**.
|
||||
4. `quiesceAlarmGrace` 180 s — derived from a 120 s deploy budget and Mealie's 60 s start_period,
|
||||
**not** from cadence. **NOT distorted** → a failure here is a genuine DEFECT. *(This one mattered:
|
||||
F-CRIT-1 lives here.)*
|
||||
5. `local_backup_retention` 3 — turnover in 90 min instead of 3 days. **ARTIFACT**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Phase A — the happy path at volume
|
||||
|
||||
| | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| quiesce cycles run | 16 | 28 |
|
||||
| unquiesces | 16 | 26 (the 2-cycle gap is F9/F10, where crash-recovery restarted the stacks instead) |
|
||||
| archives on the local tier at end | 3 (retention=3) | 3 (retention=3) |
|
||||
| PBS snapshots at end | 4 | 10 |
|
||||
|
||||
**A1 — the window gate.** Deferrals captured on both boxes, and the bounds visibly recomputed
|
||||
across a live W change with no restart (`[04:30–08:30)` → `[19:30–23:30)` between two polls
|
||||
5 minutes apart). 5-minute poll cadence confirmed to the second.
|
||||
*Methodology note that changed the evidence:* demo-felhom runs `level: info`, so none of its
|
||||
quiesce DEBUG lines reach `docker logs`. Judging by docker logs alone would have produced a false
|
||||
"demo-felhom never defers". The lines were recovered from the always-DEBUG ring via
|
||||
`/api/debug/logs?level=DEBUG`, which became a continuous collector.
|
||||
|
||||
**A2 — R-82 one-quiesce-two-tiers.** Proven end-to-end:
|
||||
```
|
||||
17:30:09 backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx]
|
||||
17:31:25 tier local: ... done — next tier may start (app still quiesced)
|
||||
17:31:25 tier felhom-pbs: ... started — polling
|
||||
17:31:35 tier felhom-pbs: ... snapshotted — resuming app early (8B.2)
|
||||
17:31:35 unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s)
|
||||
```
|
||||
One window, two tiers, sequential, local first — matching the downtime argument in `tiers.go`.
|
||||
**Total app downtime 86 s** even though the PBS upload continued past the resume. The stop set was
|
||||
exact: during demo-felhom's cycle only the 5 protected infra containers remained, with all 10 app
|
||||
containers of the 4 announced stacks gone.
|
||||
|
||||
**A3 — `verify-new` populates `verify_state`.** All snapshots `state: ok`; the snapshot this
|
||||
campaign created carried its own per-snapshot verify UPID rather than the shared namespace-wide
|
||||
one. *Near-miss worth recording:* `proxmox-backup-manager verify-job list` on ep0 is **empty**,
|
||||
which reads as "nothing verifies these backups". That would have been a false finding —
|
||||
verification is driven by the **agent's** `VerifyLoop` (6 h, `internal/pbs/verify.go:13`), not by a
|
||||
server-side job. The verify cadence was NOT compressed, so this result applies directly to
|
||||
production.
|
||||
|
||||
**A4 — prune actually prunes.** Named, not inferred:
|
||||
```
|
||||
INFO: prune older backups with retention: keep-last=3
|
||||
INFO: removing backup 'local:backup/vzdump-lxc-9201-2026_07_27-10_24_37.tar.zst'
|
||||
INFO: pruned 1 backup(s) not covered by keep-retention policy
|
||||
```
|
||||
The same logs confirm the three temporary `snap_vm-9201-disk-*_vzdump` LVM volumes are torn down
|
||||
after every run — including on failure paths (verified during F7). No snapshot leaked all night.
|
||||
**PBS-side prune** (`03:30`, keep-last 2) is reported in §3.1 with GC run 2.
|
||||
|
||||
### 3.1 GC — both runs, with the atime explanation
|
||||
|
||||
**Run 1 (early, 17:17:31→17:18:11Z, TASK OK):**
|
||||
```
|
||||
Removed garbage: 0 B
|
||||
Removed chunks: 0
|
||||
Pending removals: 4.669 GiB (in 3903 chunks)
|
||||
On-Disk usage: 7.735 GiB (22.45%) Deduplication factor: 4.45
|
||||
```
|
||||
Correct, and **not a finding** — every chunk carried a fresh atime from the 06:29 migration copy
|
||||
and GC only reclaims past a 24 h+5 m cutoff. The load-bearing line is **`Pending removals`**, not
|
||||
`Removed garbage: 0 B`: per standing rule 3, "0 removed" alone cannot distinguish "deferred
|
||||
correctly" from "did nothing". `Pending removals` is the positive observable — GC walked the index,
|
||||
positively identified 3903 unreferenced chunks, and declined to unlink them because they were too
|
||||
young.
|
||||
|
||||
**Run 2:** see §10 (appended at ~06:40, after the 06:34 eligibility boundary).
|
||||
|
||||
---
|
||||
|
||||
## 4. Phase B — the fault matrix
|
||||
|
||||
| # | Fault | Outcome | Evidence |
|
||||
|---|---|---|---|
|
||||
| 1 | ep0 unreachable while PBS due | **PASS** — breaker armed, event end-to-end, **per-tier isolation proven** (local succeeded while PBS failed in the same window) | `F1-ep0-unreachable.md` |
|
||||
| 2 | ep0 killed mid-vzdump | **PASS** on chunk-store integrity (0 `.bad`, datastore unchanged) — **but produced F-CRIT-2** | `F-CRIT-2-*.md` |
|
||||
| 3 | wg-felhom down | **PASS** — and attempt 1 SELF-HEALED (60 s watchdog). Attempt 2 (endpoint blocked) → **same error string as fault 1** → F-DIAG | `F3-wg-down.md` |
|
||||
| 4 | storage box unreachable during restic | **NOT TESTED** — 4 injection approaches all failed to interrupt the run; reasons documented | `F4-...-NOT-TESTED.md` |
|
||||
| 5 | stale restic lock from a killed run | **PASS** — recovers, does not wedge; every run begins with an unconditional `restic unlock` | `F5-stale-restic-lock.md` |
|
||||
| 6 | fill local disk mid-backup | **INCONCLUSIVE** — backup succeeded on ext4's 5% root reserve (2.07 GB) while `df` read 100% | `F6-fill-local-disk.md` |
|
||||
| 7 | fill the PBS volume to 100% | **PASS** — clean ENOSPC, no corruption, no leaked LVM snapshot, instant recovery | `F7-fill-pbs-volume.md` |
|
||||
| 8 | fill the guest disk during a quiesce | **PARTIAL** — vzdump correctly unaffected; the write-path volume was not the one filled | `F8-fill-guest-disk.md` |
|
||||
| 9 | kill the agent mid-backup | **PASS** — apps restarted in 10 s; agent auto-restarted in 17 s; vzdump survived (it is a PVE task) | `F9-kill-agent-mid-backup.md` |
|
||||
| 10 | kill the controller mid-quiesce | **PASS** — crash recovery in **1 second**; and it corrects the premise (see below) | `F10-kill-controller-mid-quiesce.md` |
|
||||
| 11 | reboot the guest mid-backup | **NOT TESTED** — see §9 | — |
|
||||
| 12 | reboot the host mid-backup | **NOT TESTED** — see §9 | — |
|
||||
| 13 | backup + restore-test + GC concurrently | **HALF PASS** — backup vs restore-test single-flight HOLDS, proven live and unplanned. The +GC combination was not run | `F-A1-*.md` |
|
||||
| 14 | two quiesce triggers at once | **PASS** — `TryLock` + skip, exactly one cycle ran | `F14-two-triggers.md` |
|
||||
| 15 | clock skew (+6 h) | **PASS** — gate follows wall clock (correct); **breaker is skew-proof** via Go monotonic time | `F15-clock-skew.md` |
|
||||
| 16 | `age_state=absent` (never observed live) | **PASS** — valve fired, cycle ran OUTSIDE a closed window on a genuinely never-backed-up tier | `F16-age-state-absent.md` |
|
||||
| 17 | corrupt a chunk | **PASS on all four legs** — detect → quarantine `.0.bad` → `verify_state: failed` → restore fails cleanly | `F17-corrupt-chunk.md` |
|
||||
| 18 | delete a snapshot mid restore-test | **NOT TESTED** — see §9 | — |
|
||||
| 19 | R-97b both halves | **part 1 PASS** (suppression, with the detector proven alive); **part 2 FAIL → F-CRIT-1** | `F19-part1-*.md`, `F-CRIT-1-*.md` |
|
||||
|
||||
**Fault 10 corrects its own premise.** The spec describes the mechanism as "the guaranteed
|
||||
unquiesce via `defer`". `defer` played no part — it does not run on SIGKILL. What saved the stacks
|
||||
was the **crash marker + `Recover()`**, which fired 1 s after the kill and named the job id and
|
||||
stopped-stack count read back from the marker. The correct statement is: *the unquiesce is
|
||||
guaranteed by the marker, not by the defer.*
|
||||
|
||||
### The R-88 backoff ladder — fully validated
|
||||
Driven by a deliberately-failing tier: **15m → 30m → 1h → 2h → 4h → 4h (cap holds)**. All six
|
||||
rungs matched `backoffFor()` exactly. Rung 6 is the important one: the breaker clamps rather than
|
||||
continuing to double, so it **never latches open** — the property `breaker.go:22-26` insists on.
|
||||
R-97a's edge-triggering also held: exactly ONE event per failure *run* per tier, not one per retry,
|
||||
and the edge is tracked per-target (a second tier breaking is not masked by a first).
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase C — restore round-trips
|
||||
|
||||
| tier | what was restored | result |
|
||||
|---|---|---|
|
||||
| **restic app-data (R-87)** | `bookstack`, snapshot `b2d8936d`, 155.5 MB, 14 s, via the real `POST /backup/offbox/restore` two-step size gate | **PASS — 6/7 files byte-identical by sha256**, including the 160 MB MariaDB volume dump. The 7th (`manifest.json`) differs only in `controller_version` 0.175.0→0.178.0 and `created_at`, i.e. the restore correctly returned the SNAPSHOT's manifest, not the live one |
|
||||
| local vzdump | 5.82 GB archive → fresh CT 9299 (15 GiB extracted @ 210 MiB/s) | **PASS — exact mount_parity** |
|
||||
| PBS offsite | encrypted snapshot 4.36 GB → fresh CT 9297 | **PASS — exact mount_parity** |
|
||||
| corrupted PBS snapshot | deliberately corrupted snapshot → CT 9298 | **PASS — failed cleanly**: exit 255, named the entry AND the chunk, rolled back all three LVs, left no debris |
|
||||
|
||||
**mount_parity** was exact on both whole-guest tiers: both storage-backed mountpoints returned with
|
||||
identical in-guest path, `backup=1` flag and size (only volume names changed), and `unprivileged: 1`
|
||||
was preserved — a restore that silently dropped that would be a privilege escalation.
|
||||
|
||||
**R-87 is retired.** The restic tier now has a proven round-trip. What is NOT retired: restore into
|
||||
a fresh *guest* from restic; the `place`/`reconstitute` legs that turn restored bytes back into a
|
||||
running app; and demo-hp's `u629488-sub3` repo. Restoring bytes is not the same as restoring
|
||||
service, and only the bytes were proven.
|
||||
|
||||
Neither restored guest was booted — deliberately, see F-OPS below.
|
||||
|
||||
---
|
||||
|
||||
## 6. Findings, ranked
|
||||
|
||||
| # | Finding | Severity | DEFECT / ARTIFACT |
|
||||
|---|---|---|---|
|
||||
| **F-CRIT-1** | An app that fails to restart after a quiesce **never alarms** | **HIGH** | **DEFECT** (compression-independent) |
|
||||
| **F-CRIT-2** | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** | **HIGH** | **DEFECT** (worse on real cadences) |
|
||||
| **F-A1** | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker and pages the operator | MEDIUM | **DEFECT** in behaviour / ARTIFACT in frequency |
|
||||
| **F-HUB** | The hub **dropped an event** under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT (frequency compression-influenced) |
|
||||
| **F-LEAK** | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT in the reclaim path |
|
||||
| **F-DIAG** | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
|
||||
| **F-OBS** | `deadapp-check` leaves NO positive observable on a default (info-level) box | LOW | DEFECT |
|
||||
| **F-OPS** | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational, not code |
|
||||
|
||||
### F-CRIT-1 — the failure that has no alarm
|
||||
A production-faithful injection (one unresolvable image tag in an otherwise intact compose) made
|
||||
`paperless-ngx` fail to restart after a quiesce:
|
||||
```
|
||||
20:46:15 [ERROR] [quiesce] restart paperless-ngx: starting stack paperless-ngx: exit code 1
|
||||
containers: 0 · controller state: 'stopped' · app_start_failed events: 0 · dashboard banner: 0
|
||||
dashboard renders the dead app as: <div class="app-row stack-state-neutral" ...>
|
||||
```
|
||||
The detector was demonstrably alive — **11** `deadapp-check` scans ran after the 180 s grace
|
||||
expired. The chain: quiesce's `compose down` removes the containers → the failed `compose up`
|
||||
creates none → deployed + zero containers aggregates to `StateStopped` → and
|
||||
`main.go:1237` **explicitly excludes** `StateStopped` from the down predicate.
|
||||
|
||||
That exclusion rests on an invariant the code states outright at `main.go:1213-1216`:
|
||||
> "I1: … a deployed stack with zero containers aggregates to StateStopped. So **StateStopped means
|
||||
> 'deployed, deliberately stopped by the user'**."
|
||||
|
||||
**I1 is false.** "Deployed with zero containers" is *also* the state of a failed restart. R-97b
|
||||
itself is correct and its grace expired on time; v0.164.0's state filter sits downstream and
|
||||
swallows the case unconditionally. Neither change is wrong alone — composed, they leave a hole.
|
||||
The quiesce loop already *knows* it tried and failed (`quiesce.go:654`); that outcome simply is not
|
||||
carried into the classification. Third instance this project has shipped of a comment asserting a
|
||||
guarantee the code stopped providing.
|
||||
|
||||
### F-CRIT-2 — a failed backup that looks like a fresh one
|
||||
Killing the PBS daemon mid-upload left `ns/demo-felhom/ct/9201/2026-07-27T22:08:31Z` containing
|
||||
only `.tmp_didx` stubs and **no `index.json.blob`** — yet it is listed as a restorable backup, size
|
||||
**1 byte**, and it is the NEWEST entry. `NewestArchiveTime` (`runner.go:311-326`) filters on
|
||||
`Content == "backup"` and `VMID` only, with no completeness test, and the due-check treats storage
|
||||
as ground truth.
|
||||
|
||||
Empirically: the controller UI reported the PBS tier's last backup as *"2026-07-28 00:08 (9
|
||||
perce)"* — the failed upload. After a controller restart (which clears the in-memory breaker, so
|
||||
the breaker cannot explain the silence), two full poll cycles produced **no due-tier line at all**,
|
||||
while the last genuinely successful PBS backup was 2.5 h old against a 2 h cadence — **overdue and
|
||||
not attempted.** On the real 168 h cadence this suppresses the offsite tier for up to **seven
|
||||
days**, with every surface reporting it fresh and the hub's ~8-day deadline monitor as the only
|
||||
remaining net. The R-88 breaker does not help: it defers tiers that are *due*, and this tier is not.
|
||||
|
||||
**The signal to reject the phantom already exists one subsystem over.** A full verify flagged
|
||||
exactly this snapshot:
|
||||
`manifest load error: unable to load blob '.../index.json.blob' - No such file or directory`.
|
||||
Two discriminators were available and unused: the missing manifest, and the snapshot's
|
||||
`verification` state (which the agent's own `VerifyLoop` maintains and which A3/F17 showed
|
||||
propagating to the Proxmox API).
|
||||
|
||||
*Narrowing that makes it more actionable, not less serious:* fault 7 showed that an **ENOSPC**
|
||||
failure leaves nothing behind — the server was alive and cleaned up. The phantom is specific to the
|
||||
server dying or becoming unreachable mid-write.
|
||||
|
||||
### F-A1 — routine contention reported as failure
|
||||
A restore-test in flight made the agent correctly refuse a backup (`busy=restore-test`, the R-85
|
||||
Scenario F gate). The controller treats that HTTP 409 as a tier failure: breaker armed, and
|
||||
`whole_guest_backup_failed` emailed to the operator — on **both** boxes. No 409 special-casing
|
||||
exists on the start path (`quiesce.go:455-460`). Two comments assert otherwise:
|
||||
`inflight.go:18-20` ("A caller that cannot acquire DEFERS to its next cadence" — the backup caller
|
||||
does not) and `quiesce.go:489-491` (which frames the 409 as what *prevents* "a spurious failure").
|
||||
Frequency is an ARTIFACT; the behaviour is a DEFECT — at real cadences a ~12-minute restore-test
|
||||
against a daily backup collides roughly once per 420 guest-days, i.e. **every ~4 days on a
|
||||
100-guest fleet**, forever training R-97a's alarm into noise.
|
||||
|
||||
*(Full write-ups for every finding, with complete evidence, are in `~/campaign8/evidence/phaseB/`.)*
|
||||
|
||||
---
|
||||
|
||||
## 7. What can come off the "never validated" list
|
||||
|
||||
**Retired — proven live tonight:**
|
||||
- **The R-88 breaker arming** — and its full ladder 15m/30m/1h/2h/4h/4h-cap, its per-target
|
||||
isolation, and its recovery edge.
|
||||
- **`whole_guest_backup_failed` from a real failure, end-to-end** — controller → hub → operator
|
||||
email, with correct tier attribution.
|
||||
- **R-97c operator-only routing** — verified against the hub DB: **zero** customer rows with
|
||||
`status='sent'`, plus the deliberate `skipped`/`operator_only` row. (The spec's "zero customer
|
||||
rows" is imprecise: the skip is logged on purpose so it cannot look like a delivery that never
|
||||
happened.)
|
||||
- **`whole_guest_backup_recovered`**, at severity `info`, including the R-68 **pairing gate**
|
||||
firing live: *"customer mail skipped — no unanswered customer down mail (pairing miss)"*.
|
||||
- **`age_state=absent`** — a genuinely never-backed-up tier fired the window-gate safety valve and
|
||||
ran outside a closed window.
|
||||
- **R-97b's suppression half** — with the detector proven alive by 5 scans inside the window.
|
||||
- **The crash-recovery unquiesce** — by an actual SIGKILL, recovering in 1 second.
|
||||
- **R-87: the restic tier has a restore round-trip**, byte-verified.
|
||||
- **R-82 one-quiesce-two-tiers**, and per-tier isolation under a real one-tier-fails case.
|
||||
- **Single-flight**: backup vs restore-test, and scheduled vs manual trigger.
|
||||
|
||||
**Still NOT validated:**
|
||||
- **R-97b's other half** — an app that genuinely fails to restart does **not** alarm (F-CRIT-1).
|
||||
This is now a known defect rather than an open question.
|
||||
- Restic restore into a fresh guest; the `place`/`reconstitute` legs; demo-hp's restic repo.
|
||||
- The agent's own DR bring-up path (`ModeDRGuestLoss`) — read in code, never executed.
|
||||
- Backup + restore-test + **GC** three-way concurrency.
|
||||
- Guest and host reboots mid-backup.
|
||||
|
||||
---
|
||||
|
||||
## 8. Final fleet state — HEALTHY, nothing left broken
|
||||
|
||||
All four compression knobs reverted and verified on both hosts (`local= 0`, `restore_test= 302400`,
|
||||
`felhom-pbs 604800`); windows restored (demo-felhom `02:30`, demo-hp back to genuine ABSENCE);
|
||||
every fault injection unwound. Sweep:
|
||||
```
|
||||
leaked scratch guests (990000-990009): NONE nft c8 tables: 0/0
|
||||
ballast files (host+guest, both boxes): 0 clocks synchronized: yes/yes
|
||||
ep0 .bad chunks: 0 ep0 pbs daemon: active
|
||||
ep0 rollback copy /srv/pbs-felhom: PRESENT ep0 datastore: 98G 14G 85G 14%
|
||||
```
|
||||
demo-felhom 15/15 containers healthy; demo-hp 8/8. In-memory breakers cleared by the revert
|
||||
restarts. **The answer to Phase-0 question 4 is that it did not come to that.**
|
||||
|
||||
One campaign artefact was removed as part of restoring service, not merely tidiness: F2's partial
|
||||
snapshot. Leaving it would have left demo-felhom's PBS tier silently suppressed for seven days —
|
||||
F-CRIT-2's live effect.
|
||||
|
||||
---
|
||||
|
||||
## 9. What could NOT be tested, and why
|
||||
|
||||
- **Fault 4 (storage box unreachable during restic)** — four approaches tried; all failed to
|
||||
interrupt the run. Root cause of the failure to inject is itself informative: restic runs inside
|
||||
the guest, so its traffic never enters the host's `output` hook (which is exactly why F1/F3
|
||||
*did* work — PBS/vzdump traffic is host-originated), and guest 9201 is **bridged**, so it does not
|
||||
traverse the host's `forward` hook either. `bridge`-family rules would not parse on this nft
|
||||
version, and a guest-side blackhole route did not cover the container's path. **This is the single
|
||||
most valuable follow-up**, because F-CRIT-2 raises the analogous question for restic: does an
|
||||
interrupted restic run leave a partial pack that later looks like success?
|
||||
- **Faults 11 and 12 (guest / host reboot mid-backup)** — not run. Deliberate: F9 and F10 already
|
||||
exercised hard-kill crash safety for both the agent and the controller, and a host reboot at
|
||||
02:00 on a remote box with no console carried recovery risk disproportionate to the marginal
|
||||
information. Recorded as a reasoned skip, not an oversight.
|
||||
- **Fault 18 (delete a snapshot out from under a restore-test)** — not reached before the campaign's
|
||||
time budget ran out.
|
||||
- **Fault 13's GC leg** — the backup-vs-restore-test half was proven; adding GC was not run.
|
||||
- **Fault 6** — inconclusive: ext4's 5% root reserve (2.07 GB, confirmed via `tune2fs`) meant the
|
||||
filesystem was never truly full for a root-run vzdump. Driving a Proxmox host's root filesystem
|
||||
to genuine zero risks wedging `pmxcfs`; not attempted.
|
||||
- **Fault 8** — partial: the volume filled was not the one this guest's app-data backup writes to.
|
||||
Filling the real target would have meant writing ~890 GB to a remote demo box.
|
||||
- **Booting either restored guest** — refused on purpose. Their `mp8`/`mp9` binds still point at the
|
||||
SOURCE guest's live data drive and bootstrap credentials (F-OPS); starting one would have mounted
|
||||
live customer data read-write beside the running guest.
|
||||
|
||||
### Two self-inflicted methodology traps, recorded so they are not repeated
|
||||
1. **A noise filter deleted the evidence.** `grep -viE 'locale|perl|…'` silently dropped every line
|
||||
naming `pa`**`perl`**`ess-*` containers, producing a convincing false reading that a quiesce had
|
||||
failed to restart a stack. Caught only by an arithmetic inconsistency (`docker ps -a -q | wc -l`
|
||||
= 8 vs 5 formatted lines) and the controller's own container count. Filters are now anchored to
|
||||
the actual warning lines. **This is the spec's own "absence of a signal is not evidence" rule
|
||||
biting the campaign's instrumentation rather than the system.**
|
||||
2. **A ~64 KB single-line ceiling through `ssh → pct exec`** silently returned 0 bytes for a
|
||||
debug-ring fetch at `limit=400`, and the parser swallowed the error — presenting as "this box has
|
||||
no DEBUG lines". Parser now fails loudly.
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user