40 KiB
CAMPAIGN 8 — the backup & restore subsystem
Adversarial, destructive, unattended. 2026-07-27 19:00 → 2026-07-28 ~06:45 CEST.
Nodes exercised: demo-felhom (+ guest 9201), demo-hp (+ guest 9201), ep0
(felhom-hetzner, datastore felhom-offsite), restic subaccounts u629488-sub1 / u629488-sub3.
peti-felhom was never touched. Neither its data, its namespace, nor u629488-sub2. Phase 0
established with five documented probes that peti has NO data in felhom-offsite at all (see §1),
which is what made the operator-approved datastore-fill safe. The 13 GB rollback copy at
/srv/pbs-felhom on ep0 was never deleted and is present at the end of the run.
Evidence: DooPlex:~/campaign8/evidence/ — 103 files, 35 MB, written continuously by 11
collectors throughout. Nothing in this report is reconstructed after the fact.
Headline
Nine findings, two of them HIGH severity, both in the same place: the system's ability to tell you that a backup did not happen.
- F-CRIT-1 — an app that fails to restart after a quiesce never alarms, on any channel.
- F-CRIT-2 — a failed offsite backup leaves a phantom snapshot that resets the tier's freshness clock, silently suppressing the next backup for up to 7 days on production cadences.
Everything the campaign was asked to prove about the happy path proved out, including the first-ever restic restore round-trip (R-87). The mechanisms designed to catch failure — R-88's breaker, R-97a/c's routing, verify, quarantine, crash recovery — all work, and several are better than documented. The gaps are all in the seams between those mechanisms.
1. Phase 0 — questions, answers, and skips
Asked in one batch inside the operator's window, before touching anything.
| # | Question | Answer received |
|---|---|---|
| 1 | How may ep0 be taken down? | Path-blocking + stop the PBS daemon. No power-cycle. |
| 2 | May VM 300 drill-r50 be destroyed? |
Free to destroy. |
| 3 | Fill the felhom-offsite volume? |
Fill to 100%. |
| 4 | End-of-run bar? | A broken test node is acceptable if precisely reported. |
Faults skipped for lack of an answer: none. All four were answered in the window.
One question was NOT put to the operator because it was answerable from configuration, and was
verified live instead: the restic subaccount mapping. Confirmed from each guest's settings.json:
demo-felhom = u629488-sub1, demo-hp = u629488-sub3. u629488-sub2 appears in neither box's
configuration and was never addressed.
The peti-exposure check (done before asking, so question 3 could be answered honestly). What was tried, per standing rule 2:
find /mnt/pbs-datastore/ns -maxdepth 3 -type d→ onlydemo-felhom,demo-hpls -la /mnt/pbs-datastore/→ no depth-0 group dirsls -d /mnt/pbs-datastore/{vm,ct,host}/*→ no such pathsproxmox-backup-manager user list→felhom@pbs,root@pamonlyproxmox-backup-manager prune-job list→ jobs for demo-* namespaces only
Incidental security note, reported at the time: reading /etc/felhom-agent/agent.json printed
hub.api_key in cleartext into the session transcript (the redactor matched *token*/*secret*/
*password* but not the bare api_key). No secret was written to disk or to any campaign file;
the operator may wish to rotate. All later captures used a stricter redactor.
2. Compression applied, and the revert
| Knob | Real | Campaign | Ratio |
|---|---|---|---|
| local (vzdump) cadence | 24 h (via the 0⇒24h default, config.go:517-520) |
1800 s | 48× |
| PBS offsite cadence | 604800 s (168 h) | 7200 s | 84× |
| restore-test cadence | 302400 s (84 h) | 2700 s | 112× |
| backup window start W | 02:30 |
rolled 17:30 → 21:30 → (14:00 on demo-hp for F16) | — |
Feasibility was measured before choosing, not assumed: demo-felhom's local vzdump is 5.7 GB /
00:03:12, demo-hp's PBS backup 1.5 GB / 00:00:55. Both fit a 30-minute cadence with margin.
"Widen the backup window" was not possible. The gate is [W+2h, W+6h) and both offsets are
compile-time constants (backupwindow.go:19-21); only W is settable. The gate is permanently 4 h
wide against a ~10.5 h campaign, so the window was moved in contiguous steps rather than
widened — which had the side benefit of exercising the gate at three different absolute times.
A compression the spec asked for that configuration cannot deliver: "restic daily → 30m". The
three app-data legs are Daily scheduler jobs pinned to fixed offsets from the same W
(main.go:595,632,665). Moving W reschedules them; it cannot make them 30-minutely. Restic was
therefore driven by explicit manual triggers, and no 30-minute restic cadence was ever in force.
Revert: all four knobs restored and verified (§8). demo-hp's backup_window_start was restored
to genuine ABSENCE from its pre-campaign snapshot rather than written as "02:30", because absent is
what it was.
The pre-registered artifact list
Declared BEFORE Phase A so findings could be classified honestly:
- R-88 breaker cap 4 h — chosen against a 24 h cadence. ARTIFACT under compression.
- R-88 ladder base 15 m — chosen to exceed the 5 m poll. ARTIFACT.
- Window-gate valve "cadence + 24 h" — fires at 24.5 h instead of 48 h. ARTIFACT.
quiesceAlarmGrace180 s — derived from a 120 s deploy budget and Mealie's 60 s start_period, not from cadence. NOT distorted → a failure here is a genuine DEFECT. (This one mattered: F-CRIT-1 lives here.)local_backup_retention3 — turnover in 90 min instead of 3 days. ARTIFACT.
3. Phase A — the happy path at volume
| demo-felhom | demo-hp | |
|---|---|---|
| quiesce cycles run | 16 | 28 |
| unquiesces | 16 | 26 (the 2-cycle gap is F9/F10, where crash-recovery restarted the stacks instead) |
| archives on the local tier at end | 3 (retention=3) | 3 (retention=3) |
| PBS snapshots at end | 4 | 10 |
A1 — the window gate. Deferrals captured on both boxes, and the bounds visibly recomputed
across a live W change with no restart ([04:30–08:30) → [19:30–23:30) between two polls
5 minutes apart). 5-minute poll cadence confirmed to the second.
Methodology note that changed the evidence: demo-felhom runs level: info, so none of its
quiesce DEBUG lines reach docker logs. Judging by docker logs alone would have produced a false
"demo-felhom never defers". The lines were recovered from the always-DEBUG ring via
/api/debug/logs?level=DEBUG, which became a continuous collector.
A2 — R-82 one-quiesce-two-tiers. Proven end-to-end:
17:30:09 backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx]
17:31:25 tier local: ... done — next tier may start (app still quiesced)
17:31:25 tier felhom-pbs: ... started — polling
17:31:35 tier felhom-pbs: ... snapshotted — resuming app early (8B.2)
17:31:35 unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s)
One window, two tiers, sequential, local first — matching the downtime argument in tiers.go.
Total app downtime 86 s even though the PBS upload continued past the resume. The stop set was
exact: during demo-felhom's cycle only the 5 protected infra containers remained, with all 10 app
containers of the 4 announced stacks gone.
A3 — verify-new populates verify_state. All snapshots state: ok; the snapshot this
campaign created carried its own per-snapshot verify UPID rather than the shared namespace-wide
one. Near-miss worth recording: proxmox-backup-manager verify-job list on ep0 is empty,
which reads as "nothing verifies these backups". That would have been a false finding —
verification is driven by the agent's VerifyLoop (6 h, internal/pbs/verify.go:13), not by a
server-side job. The verify cadence was NOT compressed, so this result applies directly to
production.
A4 — prune actually prunes. Named, not inferred:
INFO: prune older backups with retention: keep-last=3
INFO: removing backup 'local:backup/vzdump-lxc-9201-2026_07_27-10_24_37.tar.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policy
The same logs confirm the three temporary snap_vm-9201-disk-*_vzdump LVM volumes are torn down
after every run — including on failure paths (verified during F7). No snapshot leaked all night.
PBS-side prune (03:30, keep-last 2) is reported in §3.1 with GC run 2.
3.1 GC — both runs, with the atime explanation
Run 1 (early, 17:17:31→17:18:11Z, TASK OK):
Removed garbage: 0 B
Removed chunks: 0
Pending removals: 4.669 GiB (in 3903 chunks)
On-Disk usage: 7.735 GiB (22.45%) Deduplication factor: 4.45
Correct, and not a finding — every chunk carried a fresh atime from the 06:29 migration copy
and GC only reclaims past a 24 h+5 m cutoff. The load-bearing line is Pending removals, not
Removed garbage: 0 B: per standing rule 3, "0 removed" alone cannot distinguish "deferred
correctly" from "did nothing". Pending removals is the positive observable — GC walked the index,
positively identified 3903 unreferenced chunks, and declined to unlink them because they were too
young.
Run 2 (04:44:06Z / 06:44 CEST, after the 04:34 UTC boundary, TASK OK):
Removed garbage: 0 B Removed chunks: 0
Pending removals: 6.38 GiB (in 5406 chunks) [run 1: 4.669 GiB / 3903]
On-Disk usage: 6.697 GiB (17.11%) / 4891 chunks [run 1: 7.735 GiB / 5910]
Deduplication factor: 5.84 [run 1: 4.45]
The live set SHRANK and pending removals GREW — the 03:30 UTC prune landing between the runs, with GC correctly re-classifying those chunks as unreferenced.
I predicted run 2 would reclaim ≈4.669 GiB. It reclaimed nothing, and the cause is this
campaign. The datastore is mounted relatime (findmnt -no OPTIONS → rw,relatime,discard), so
reading a chunk refreshes its atime — and the campaign read essentially every chunk repeatedly
overnight (three full verifies for F17, ~14 backups, two GC runs, several restores). Sampled atimes
at 04:55:38Z prove it:
188 chunks atime 2026-07-28 04:xx ← GC run 2 itself, touching referenced chunks (phase 1)
155 chunks atime 2026-07-27 07:xx
51 chunks atime 2026-07-27 21:xx ← a verify pass
oldest atime in sample: 2026-07-27 07:18:53Z (21h37m — short of the 24h5m cutoff)
No chunk was ever 24h5m cold, so Removed garbage: 0 B is exactly correct on both runs.
Not a defect — GC made a deliberate, correct decision both times, and Pending removals is the
positive observable proving it did the work. But it IS an operational property worth knowing: on
a relatime datastore, activity defers reclamation. "GC ran, so space will come back" is unsafe
on a small offsite box that something reads regularly.
Honest limit: because the campaign's own reads reset the clock, the spec's intended experiment — watching the migration chunks cross the boundary and be collected — could not be completed inside the campaign window. First genuine reclamation should follow ~07:23Z on 07-28. Recorded as an incomplete observation, not a pass.
4. Phase B — the fault matrix
| # | Fault | Outcome | Evidence |
|---|---|---|---|
| 1 | ep0 unreachable while PBS due | PASS — breaker armed, event end-to-end, per-tier isolation proven (local succeeded while PBS failed in the same window) | F1-ep0-unreachable.md |
| 2 | ep0 killed mid-vzdump | PASS on chunk-store integrity (0 .bad, datastore unchanged) — but produced F-CRIT-2 |
F-CRIT-2-*.md |
| 3 | wg-felhom down | PASS — and attempt 1 SELF-HEALED (60 s watchdog). Attempt 2 (endpoint blocked) → same error string as fault 1 → F-DIAG | F3-wg-down.md |
| 4 | storage box unreachable during restic | NOT TESTED — 4 injection approaches all failed to interrupt the run; reasons documented | F4-...-NOT-TESTED.md |
| 5 | stale restic lock from a killed run | PASS — recovers, does not wedge; every run begins with an unconditional restic unlock |
F5-stale-restic-lock.md |
| 6 | fill local disk mid-backup | INCONCLUSIVE — backup succeeded on ext4's 5% root reserve (2.07 GB) while df read 100% |
F6-fill-local-disk.md |
| 7 | fill the PBS volume to 100% | PASS — clean ENOSPC, no corruption, no leaked LVM snapshot, instant recovery | F7-fill-pbs-volume.md |
| 8 | fill the guest disk during a quiesce | PARTIAL — vzdump correctly unaffected; the write-path volume was not the one filled | F8-fill-guest-disk.md |
| 9 | kill the agent mid-backup | PASS — apps restarted in 10 s; agent auto-restarted in 17 s. (One evidence line withdrawn — see the correction note) | F9-kill-agent-mid-backup.md |
| 10 | kill the controller mid-quiesce | PASS — crash recovery in 1 second; and it corrects the premise (see below) | F10-kill-controller-mid-quiesce.md |
| 11 | reboot the guest mid-backup | FINDING (F-REBOOT) — the backup succeeded but the guest never came back; found stopped with 0 containers, no self-heal, ~9m47s outage until manual pct start |
F11-reboot-guest-mid-backup.md |
| 12 | reboot the host mid-backup | NOT TESTED — see §9 | — |
| 13 | backup + restore-test + GC concurrently | PASS on both tested axes — backup vs restore-test single-flight HOLDS; backup vs GC coexist (both tiers completed during GC run 2). True three-way overlap not tested | F13-concurrency.md |
| 14 | two quiesce triggers at once | PASS — TryLock + skip, exactly one cycle ran |
F14-two-triggers.md |
| 15 | clock skew (+6 h) | PASS — gate follows wall clock (correct); breaker is skew-proof via Go monotonic time | F15-clock-skew.md |
| 16 | age_state=absent (never observed live) |
PASS — valve fired, cycle ran OUTSIDE a closed window on a genuinely never-backed-up tier | F16-age-state-absent.md |
| 17 | corrupt a chunk | PASS on all four legs — detect → quarantine .0.bad → verify_state: failed → restore fails cleanly |
F17-corrupt-chunk.md |
| 18 | delete a snapshot mid restore-test | PASS on detection — restore-test failed loudly, did NOT advance the proven-restorable timestamp; and it root-caused F-LEAK | F18-delete-snapshot-mid-restore-test.md |
| 19 | R-97b both halves | part 1 PASS (suppression, with the detector proven alive); part 2 FAIL → F-CRIT-1 | F19-part1-*.md, F-CRIT-1-*.md |
Fault 10 corrects its own premise. The spec describes the mechanism as "the guaranteed
unquiesce via defer". defer played no part — it does not run on SIGKILL. What saved the stacks
was the crash marker + Recover(), which fired 1 s after the kill and named the job id and
stopped-stack count read back from the marker. The correct statement is: the unquiesce is
guaranteed by the marker, not by the defer.
The R-88 backoff ladder — fully validated
Driven by a deliberately-failing tier: 15m → 30m → 1h → 2h → 4h → 4h (cap holds). All six
rungs matched backoffFor() exactly. Rung 6 is the important one: the breaker clamps rather than
continuing to double, so it never latches open — the property breaker.go:22-26 insists on.
R-97a's edge-triggering also held: exactly ONE event per failure run per tier, not one per retry,
and the edge is tracked per-target (a second tier breaking is not masked by a first).
5. Phase C — restore round-trips
| tier | what was restored | result |
|---|---|---|
| restic app-data (R-87) | bookstack, snapshot b2d8936d, 155.5 MB, 14 s, via the real POST /backup/offbox/restore two-step size gate |
PASS — 6/7 files byte-identical by sha256, including the 160 MB MariaDB volume dump. The 7th (manifest.json) differs only in controller_version 0.175.0→0.178.0 and created_at, i.e. the restore correctly returned the SNAPSHOT's manifest, not the live one |
| local vzdump | 5.82 GB archive → fresh CT 9299 (15 GiB extracted @ 210 MiB/s) | PASS — exact mount_parity |
| PBS offsite | encrypted snapshot 4.36 GB → fresh CT 9297 | PASS — exact mount_parity |
| corrupted PBS snapshot | deliberately corrupted snapshot → CT 9298 | PASS — failed cleanly: exit 255, named the entry AND the chunk, rolled back all three LVs, left no debris |
mount_parity was exact on both whole-guest tiers: both storage-backed mountpoints returned with
identical in-guest path, backup=1 flag and size (only volume names changed), and unprivileged: 1
was preserved — a restore that silently dropped that would be a privilege escalation.
R-87 is retired. The restic tier now has a proven round-trip. What is NOT retired: restore into
a fresh guest from restic; the place/reconstitute legs that turn restored bytes back into a
running app; and demo-hp's u629488-sub3 repo. Restoring bytes is not the same as restoring
service, and only the bytes were proven.
Neither restored guest was booted — deliberately, see F-OPS below.
6. Findings, ranked
| # | Finding | Severity | DEFECT / ARTIFACT |
|---|---|---|---|
| DEFECT — closed | |||
| DEFECT — closed | |||
| DEFECT — closed | |||
| F-HUB | The hub dropped an event under concurrent load (SQLITE_BUSY), no retry, cause unnamed |
MEDIUM | DEFECT (frequency compression-influenced) |
| F-LEAK | A failed restore-test cannot destroy its own scratch guest (403 VM.Allocate); leaks are never reclaimed and the 10-slot VMID band shrinks silently |
MEDIUM | DEFECT (root-caused by fault 18) |
| F-REBOOT | A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
| F-OBS | deadapp-check leaves NO positive observable on a default (info-level) box |
LOW | DEFECT |
| F-OPS | A manual pct restore inherits the source guest's binds (live data drive + another guest's credentials) |
LOW | Operational, not code |
F-CRIT-1 — the failure that has no alarm
A production-faithful injection (one unresolvable image tag in an otherwise intact compose) made
paperless-ngx fail to restart after a quiesce:
20:46:15 [ERROR] [quiesce] restart paperless-ngx: starting stack paperless-ngx: exit code 1
containers: 0 · controller state: 'stopped' · app_start_failed events: 0 · dashboard banner: 0
dashboard renders the dead app as: <div class="app-row stack-state-neutral" ...>
The detector was demonstrably alive — 11 deadapp-check scans ran after the 180 s grace
expired. The chain: quiesce's compose down removes the containers → the failed compose up
creates none → deployed + zero containers aggregates to StateStopped → and
main.go:1237 explicitly excludes StateStopped from the down predicate.
That exclusion rests on an invariant the code states outright at main.go:1213-1216:
"I1: … a deployed stack with zero containers aggregates to StateStopped. So StateStopped means 'deployed, deliberately stopped by the user'."
I1 is false. "Deployed with zero containers" is also the state of a failed restart. R-97b
itself is correct and its grace expired on time; v0.164.0's state filter sits downstream and
swallows the case unconditionally. Neither change is wrong alone — composed, they leave a hole.
The quiesce loop already knows it tried and failed (quiesce.go:654); that outcome simply is not
carried into the classification. Third instance this project has shipped of a comment asserting a
guarantee the code stopped providing.
F-CRIT-2 — a failed backup that looks like a fresh one
Killing the PBS daemon mid-upload left ns/demo-felhom/ct/9201/2026-07-27T22:08:31Z containing
only .tmp_didx stubs and no index.json.blob — yet it is listed as a restorable backup, size
1 byte, and it is the NEWEST entry. NewestArchiveTime (runner.go:311-326) filters on
Content == "backup" and VMID only, with no completeness test, and the due-check treats storage
as ground truth.
Empirically: the controller UI reported the PBS tier's last backup as "2026-07-28 00:08 (9 perce)" — the failed upload. After a controller restart (which clears the in-memory breaker, so the breaker cannot explain the silence), two full poll cycles produced no due-tier line at all, while the last genuinely successful PBS backup was 2.5 h old against a 2 h cadence — overdue and not attempted. On the real 168 h cadence this suppresses the offsite tier for up to seven days, with every surface reporting it fresh and the hub's ~8-day deadline monitor as the only remaining net. The R-88 breaker does not help: it defers tiers that are due, and this tier is not.
The signal to reject the phantom already exists one subsystem over. A full verify flagged
exactly this snapshot:
manifest load error: unable to load blob '.../index.json.blob' - No such file or directory.
Two discriminators were available and unused: the missing manifest, and the snapshot's
verification state (which the agent's own VerifyLoop maintains and which A3/F17 showed
propagating to the Proxmox API).
Narrowing that makes it more actionable, not less serious: fault 7 showed that an ENOSPC failure leaves nothing behind — the server was alive and cleaned up. The phantom is specific to the server dying or becoming unreachable mid-write.
F-A1 — routine contention reported as failure
A restore-test in flight made the agent correctly refuse a backup (busy=restore-test, the R-85
Scenario F gate). The controller treats that HTTP 409 as a tier failure: breaker armed, and
whole_guest_backup_failed emailed to the operator — on both boxes. No 409 special-casing
exists on the start path (quiesce.go:455-460). Two comments assert otherwise:
inflight.go:18-20 ("A caller that cannot acquire DEFERS to its next cadence" — the backup caller
does not) and quiesce.go:489-491 (which frames the 409 as what prevents "a spurious failure").
Frequency is an ARTIFACT; the behaviour is a DEFECT — at real cadences a ~12-minute restore-test
against a daily backup collides roughly once per 420 guest-days, i.e. every ~4 days on a
100-guest fleet, forever training R-97a's alarm into noise.
F-CRIT-1 and F-A1 — FIXED (controller v0.179.0, 2026-07-28)
F-CRIT-1, both causes: restartAll now returns the stacks that failed, and classifyRunStates
lifts the StateStopped whitelist for exactly those (Loop.FailedRestarts()). I1 is restated
truthfully — StateStopped means "deployed with zero containers", which is EITHER a user stop OR a
failed quiesce restart, and only the loop's own knowledge separates them.
F-A1: HTTP 409 is contention, not failure — typed *StatusError on POST, mapped to
quiesce.ErrTierBusy at the adapter. The tier stays DUE and is dropped before anything stops
(contentionRetryAfter 15m), and contention outliving the agent's own 120m PBS restore-test ceiling
raises a BLOCKED signal at 3h so a wedged gate is never silent.
Replayed live on demo-hp with the hub DB as arbiter: the failed restart alarmed 9 s after grace
expiry with the banner naming (stopped); a deliberate user stop stayed silent through 9 dead-app
scans; a 409 produced 0 operator emails (8 → 8) while a real transport failure produced 1
(8 → 9). Six red-proofs. Detail: felhom-controller/REPORT.md.
All three of Campaign 8's HIGH/MEDIUM alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).
F-CRIT-2 — FIXED (agent v0.106.0, 2026-07-28)
NewestArchiveTime now counts only plausibly complete entries, via a measured size floor
(minPlausibleArchiveBytes = 1 MiB; smallest real backup on the fleet is 612,397,450 B, so 584x
headroom, asserted by a test). Undecidable ⇒ not counted. Size is the only tier-agnostic
discriminator: verification and encrypted are absent on every local dir archive and on a good
PBS snapshot until verify-new catches up, so gating on either would have rejected 100% of local
backups and caused fleet-wide backup thrash.
Verified by replaying fault 2 against the fixed agent on demo-hp: the phantom was created, rejected,
and announced once; the tier was correctly reported DUE and backed up; and the inverse case showed
no re-backup thrash (with 91 scheduler ticks proving the loop was alive). Detail:
felhom-agent/REPORT.md.
Also settled: server-side prune does NOT count phantoms toward keep-last — a dry-run retained
two real snapshots PLUS the phantom — so the feared retention/data-loss scenario does not occur. It
never removes them either; filed as R-99.
F-REBOOT — FIXED (agent v0.107.0, 2026-07-28)
A periodic guest-power watchdog (internal/localapi/guestpower.go, 60 s) starts a guest that is
onboot:1, stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
RecoverStaleLockedGuests miss fault 11: that recovery acts only on a guest holding a stale vzdump
lock (fault 11's guest was unlocked) and runs once at agent startup (fault 11's guest went down
while the agent was already up).
onboot is the deliberate-stop discriminator, and it is not invented here — it is already what the
stale-lock path uses for this decision, it is 0 on scratch/golden guests, and it is what pve-guests
itself consults at host boot, so the agent agrees with the platform rather than keeping a second
private definition of "should be running". Retry is bounded (3 attempts, 1m/2m/4m) and then escalates
once; an unbounded silent retry loop would be the over-correction.
Replayed live on demo-hp: a guest stopped out from under the agent came back unattended in 120 s,
against the 587 s the finding's original incident needed a human. The lock-deferral guard fired
live during the replay. Scenario B proven on the same box — an onboot:0 guest was left stopped
across the whole window.
F-LEAK — FIXED (agent v0.110.0 + host-install v1.21.0, 2026-07-28) — after TWO refuted attempts
Three attempts, two refuted by live test. Recorded in full because both wrong answers looked right and one of them shipped.
The cause is structural rather than a missing privilege in the role: FelhomAgentGuest is granted at
/pool/felhom, and a guest joins that pool only when its restore completes, so a failed
restore-test leaves a guest that exists, is in no pool, and is out of the token's reach entirely.
Attempt 1 — adopt the stranded guest into the pool, then retry (agent v0.107.0, shipped). Fired
exactly as designed and PVE refused it: PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing privilege ...). PUT /pools/{pool} also requires VM.Allocate on the VM being added —
pool membership cannot bootstrap its own authority. Removed in v0.108.0.
Attempt 2 — grant the role per-path at /vms/990000..990009 (host-install v1.21.0). It works, and
the live A/B proved it: grant removed → 403 (/vms/990000, VM.Allocate), guest still present; grant
restored → 200 UPID:...vzdestroy, guest gone. But it is durable for exactly one use per slot. The
post-destroy check showed /vms/990000 with 0 grants, and PVE's own source says why: the destroy
path calls AccessControl::remove_vm_access($vmid) (API2/LXC.pm:906), which deletes every ACL at
/vms/<vmid> (AccessControl.pm:1898). The grant is consumed by the operation it authorises — after
ten teardowns the band is ungranted and the defect returns. A per-VM ACL is the wrong tool; the pool
grant is durable only because it lives on a pool path, which remove_vm_access does not touch.
Attempt 3 — route the destroy through the authority that already performs the create (agent
v0.110.0). Privileged.DestroyScratchLXC is the fourth root-fenced exception, and it is fenced
harder than the other three, in three places none of which is consumed by use:
| layer | enforcement |
|---|---|
| sudoers | /usr/sbin/pct destroy 99000[0-9] --purge — sudo matches the vmid literally |
DestroyScratchLXC |
re-checks the band; refuses an unconfigured or inverted band rather than defaulting |
teardownScratch |
acts only on this journal entry's own scratch provenance |
The API destroy is still tried first and stays the normal path; the v1.21.0 band ACLs stay provisioned so the common case needs no privileged call at all. This is the fallback that makes teardown deterministic instead of once-per-slot.
The fence, proven live by asking sudo rather than executing (executing the negative case against a live guest is precisely what must not happen):
| command | sudo verdict |
|---|---|
pct destroy 990000 / 990005 / 990009 --purge |
PERMITTED |
pct destroy 9201 --purge (the live customer guest) |
REFUSED |
pct destroy 9100 / 9999 / 1 --purge |
REFUSED |
pct destroy 990010 --purge (one past the band) |
REFUSED |
pct start 990000 / pct set 990000 -memory 512 |
REFUSED — the grant is this one op, not this vmid |
Also proven at the API seam, and worth keeping because it establishes the shape of the refusal: a
DELETE of the non-existent 990010 returned 403, not "does not exist" — PVE evaluates
permission before existence, so these are genuine authorization refusals rather than artifacts.
Granting VM.Allocate at /vms was rejected throughout: it would authorise destroying every guest on
the box, including a co-tenant's.
F-OBS — FIXED (controller v0.180.0, 2026-07-28)
deadapp-check now emits a summary line at INFO every 20th scan (10 min at its 30 s cadence),
carrying scans-since-boot, stacks evaluated, and how many are currently down. The original silence
came from its per-cycle line going through Scheduler.dbg(), which is gated on logging.level==debug
and so was never produced on a default box — meaning "no alarms" was indistinguishable from "the
detector never ran", the exact fallacy this project has a standing rule against. A line per run was
not an option either (2880/day is what made silence attractive), so the cadence is asserted by test in
both directions: it must not flood, and it must stay frequent enough to expose a stalled detector
inside the 180 s alarm grace it feeds.
(Full write-ups for every finding, with complete evidence, are in ~/campaign8/evidence/phaseB/.)
6b. Follow-up investigation, 2026-07-28 — R-100 (filed, deliberately NOT fixed)
A restic offsite tier that fails every night never goes stale on the hub
This is F-CRIT-2's defect class, one layer up and on the other tier — a failed run resetting the freshness clock — and it was found by asking whether the F-CRIT-2 shape existed anywhere else.
Two halves, each verified in the source rather than inferred:
- Controller.
o.LastRun = time.Now()is set unconditionally incontroller/internal/backup/offbox.go:716, outside therunErrbranch. The failure is recorded faithfully — but in a different field:o.LastStatus = "error"at :725. - Hub.
isStale()reads onlyoff.LastRun(hub/internal/monitor/offsite.go:120,:127,:131). It never consultsLastStatus.
So a nightly restic run that fails every night keeps LastRun fresh, isStale is permanently false,
and the staleness alarm never fires — while no successful offsite backup has occurred at all.
Scope of the silence, stated precisely. LastStatus does reach the hub: it is parsed into the
report struct and only logged (offsite.go:270). It drives no checker and no notification. The
controller's own guest UI does surface LastStatus="error", so the failure is visible to someone who
looks; what is missing is the push — the operator's fleet-wide alarm plane is silent, which is the
plane that matters for an unattended appliance.
Not fixed, per this task's investigation-only scope. Filed as R-100; the fix direction is to
gate staleness on the last successful run rather than the last attempt, which is precisely what
F-CRIT-2's NewestArchiveTime fix did for the PBS tier.
7. What can come off the "never validated" list
Retired — proven live tonight:
- The R-88 breaker arming — and its full ladder 15m/30m/1h/2h/4h/4h-cap, its per-target isolation, and its recovery edge.
whole_guest_backup_failedfrom a real failure, end-to-end — controller → hub → operator email, with correct tier attribution.- R-97c operator-only routing — verified against the hub DB: zero customer rows with
status='sent', plus the deliberateskipped/operator_onlyrow. (The spec's "zero customer rows" is imprecise: the skip is logged on purpose so it cannot look like a delivery that never happened.) whole_guest_backup_recovered, at severityinfo, including the R-68 pairing gate firing live: "customer mail skipped — no unanswered customer down mail (pairing miss)".age_state=absent— a genuinely never-backed-up tier fired the window-gate safety valve and ran outside a closed window.- R-97b's suppression half — with the detector proven alive by 5 scans inside the window.
- The crash-recovery unquiesce — by an actual SIGKILL, recovering in 1 second.
- R-87: the restic tier has a restore round-trip, byte-verified.
- R-82 one-quiesce-two-tiers, and per-tier isolation under a real one-tier-fails case.
- Single-flight: backup vs restore-test, and scheduled vs manual trigger.
- A restore-test whose source vanishes fails loudly and does NOT falsely advance the tier's proven-restorable timestamp (fault 18).
- R-89's write-only box grant — the box was correctly refused when it tried to delete its own
offsite snapshot (
missing Datastore.Modify|Datastore.Prune). A compromised box cannot destroy its own backups.
Still NOT validated:
- R-97b's other half — an app that genuinely fails to restart does not alarm (F-CRIT-1). This is now a known defect rather than an open question.
- Restic restore into a fresh guest; the
place/reconstitutelegs; demo-hp's restic repo. - The agent's own DR bring-up path (
ModeDRGuestLoss) — read in code, never executed. - Backup + restore-test + GC three-way concurrency.
- Guest and host reboots mid-backup.
8. Final fleet state — HEALTHY, nothing left broken
All four compression knobs reverted and verified on both hosts (local= 0, restore_test= 302400,
felhom-pbs 604800); windows restored (demo-felhom 02:30, demo-hp back to genuine ABSENCE);
every fault injection unwound. Sweep:
leaked scratch guests (990000-990009): NONE nft c8 tables: 0/0
ballast files (host+guest, both boxes): 0 clocks synchronized: yes/yes
ep0 .bad chunks: 0 ep0 pbs daemon: active
ep0 rollback copy /srv/pbs-felhom: PRESENT ep0 datastore: 98G 14G 85G 14%
demo-felhom 15/15 containers healthy; demo-hp 8/8. In-memory breakers cleared by the revert restarts. The answer to Phase-0 question 4 is that it did not come to that.
One campaign artefact was removed as part of restoring service, not merely tidiness: F2's partial snapshot. Leaving it would have left demo-felhom's PBS tier silently suppressed for seven days — F-CRIT-2's live effect.
Final verification at 04:57:33Z / 06:57 CEST — after EVERY fault, including the two
post-Phase-D re-injections (faults 18 and 11)
[demo-felhom] agent=active guest=running containers=15 unhealthy=0 scratch=0 nft_c8=0 ballast=0 ntp=yes
[demo-hp] agent=active guest=running containers=8 unhealthy=0 scratch=0 nft_c8=0 ballast=0 ntp=yes
[ep0] pbs=active bad_chunks=0 ballast=0 rollback_copy=PRESENT df=14G/98G 14%
One outage the campaign caused, stated plainly: fault 11 left demo-hp guest 9201 powered off
for ~9m47s (03:43:13Z → 03:53:00Z) before I noticed and ran pct start 9201. That is finding
F-REBOOT, and it is the longest unplanned downtime of the run. Recovery needed no lock clearing and
all 8 containers returned within ~25 s. Everything else was reverted without incident.
Two re-injections were performed AFTER Phase D because clock-bound work (the 03:30 UTC prune and GC run 2) left time available. Both are declared rather than hidden: fault 18 needed a bounded restore-test cadence change on demo-hp (reverted, re-verified), and fault 11 needed no config change at all. The sweep above is the state after all of it.
9. What could NOT be tested, and why
- Fault 4 (storage box unreachable during restic) — four approaches tried; all failed to
interrupt the run. Root cause of the failure to inject is itself informative: restic runs inside
the guest, so its traffic never enters the host's
outputhook (which is exactly why F1/F3 did work — PBS/vzdump traffic is host-originated), and guest 9201 is bridged, so it does not traverse the host'sforwardhook either.bridge-family rules would not parse on this nft version, and a guest-side blackhole route did not cover the container's path. This is the single most valuable follow-up, because F-CRIT-2 raises the analogous question for restic: does an interrupted restic run leave a partial pack that later looks like success? - Fault 12 (host reboot mid-backup) — not run. Deliberate: F9 and F10 already exercised hard-kill crash safety for both the agent and the controller, and rebooting a remote Proxmox host with no console access carried recovery risk disproportionate to the marginal information.
Fault 18— was run in the campaign's tail (see the matrix). It required a single, declared re-injection after Phase D (restore-test cadence on demo-hp only, 84h→600s), which was reverted and Phase D re-verified immediately afterwards.- Fault 13's true THREE-way overlap (backup + restore-test + GC simultaneously). Legs A and B were each proven, never together.
- Fault 6 — inconclusive: ext4's 5% root reserve (2.07 GB, confirmed via
tune2fs) meant the filesystem was never truly full for a root-run vzdump. Driving a Proxmox host's root filesystem to genuine zero risks wedgingpmxcfs; not attempted. - Fault 8 — partial: the volume filled was not the one this guest's app-data backup writes to. Filling the real target would have meant writing ~890 GB to a remote demo box.
- Booting either restored guest — refused on purpose. Their
mp8/mp9binds still point at the SOURCE guest's live data drive and bootstrap credentials (F-OPS); starting one would have mounted live customer data read-write beside the running guest.
Two self-inflicted methodology traps, recorded so they are not repeated
- A noise filter deleted the evidence.
grep -viE 'locale|perl|…'silently dropped every line namingpaperless-*containers, producing a convincing false reading that a quiesce had failed to restart a stack. Caught only by an arithmetic inconsistency (docker ps -a -q | wc -l= 8 vs 5 formatted lines) and the controller's own container count. Filters are now anchored to the actual warning lines. This is the spec's own "absence of a signal is not evidence" rule biting the campaign's instrumentation rather than the system. - A ~64 KB single-line ceiling through
ssh → pct execsilently returned 0 bytes for a debug-ring fetch atlimit=400, and the parser swallowed the error — presenting as "this box has no DEBUG lines". Parser now fails loudly. pgrep -cf <pattern>matches its own shell when the pattern appears in thesshcommand line thatpgrep -finspects.pgrep -cf vzdump→ 1 whileps -eo args | grep -c '[/]usr/bin/vzdump'→ 0. This invalidated fault 11's injection outright and put one unsound line into fault 9's evidence (withdrawn; that finding stands on the controller's own job state instead). Drive such triggers off the controller's job log, which is authoritative and cannot self-match.
All three traps share a shape: the instrument agreed with the hypothesis for the wrong reason. Two were caught only because a second, independent source disagreed — a container count, and a job-state log. Where no second source existed, the campaign reports NOT TESTED rather than a pass.