admin
92cebb8c95
gates / gates (push) Successful in 11s
Measured live on demo-hp 2026-08-30 (controller 0.223.0): the nightly db-dump
and offbox-backup legs stop each stack ~13s to tar its volumes while the
deadapp-check job scans every 30s, so the scan caught whichever stack was
mid-cycle and pushed app_start_failed to the customer. 61 e-mails about apps
that were never broken.
The defect is not a missing mechanism. quiesce/suppress.go solved exactly this
in v0.179.0 and works -- but classifyRunStates read only the quiesce loop's set,
and that loop covers the WHOLE-GUEST backup. The per-app legs stop stacks
through Manager.DumpAppVolumesSafe, which registered with nothing. Two
mechanisms stop apps on purpose; only one told the alarm. Fifth instance of the
"seam built but never wired" class, and the first where the unwired half was a
consumer.
The suppression now rides AppStopGuard, which already brackets every deliberate
stop in the product (Begin before the stop, End after a successful restart) at
all three call sites, and which main.go hands as ONE object to the backup
manager and the exporter. scanDeployedAppRunStates takes the union of both sets.
All three per-app stop paths are covered, not only the reported nightly one.
It cannot latch -- End() runs only on a restart that SUCCEEDED, so unlike the
quiesce loop an open-ended hold is a real hazard here:
1. ReleaseFailed drops the entry IMMEDIATELY on a restart that broke, wired at
every failure path, so the app alarms on the next scan;
2. Begin REPLACES the set (one marker file = one operation);
3. appStopMaxHold (6h) caps a hold nothing released, logged at WARN.
Grace is 180s, deliberately quiesce's own constant and derivation. Suppression
is NOT persisted: after a crash the guard holds nothing and a down app must
alarm. ReleaseFailed keeps the durable crash marker; a test pins that.
Three companion red-proofs, each printing the pre-fix value (REPORT.md section 5):
- drop markStopped from Begin -> "suppressed at stop = map[]"
- drop ReleaseFailed from the dump -> "map[bookstack:true] after a restart that FAILED"
- pass nil instead of appStopGuard -> the AST wiring test fails
The third is load-bearing: the component was never the broken part, so a suite
that only injected it would have been green against the shipped defect.
Green gate clean: go build + go vet + go test ./... -- 28 packages, rc 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM
180 lines
9.7 KiB
Markdown
180 lines
9.7 KiB
Markdown
# REPORT — R-330: the nightly backup alarmed about the apps it was holding down
|
|
|
|
**Controller v0.224.0 · 2026-08-30 · implemented on DooPlex, diagnosed live on `demo-hp`**
|
|
|
|
This report covers Option A of a two-part request. Option B (the hub Backup card that always reads
|
|
`Snapshots 0`) is a separate, still-open defect and is described in §7.
|
|
|
|
---
|
|
|
|
## 1. What was reported
|
|
|
|
61 e-mails, arriving in two bursts every night from both demo boxes:
|
|
|
|
```
|
|
[Felhom] demo-hp: app_start_failed
|
|
Severity: warning
|
|
Time: 2026-08-30 02:30 CEST
|
|
Message: Telepített alkalmazás nem fut: Docmost
|
|
```
|
|
|
|
## 2. What was actually happening
|
|
|
|
**Nothing was broken.** Both boxes were healthy at every check:
|
|
|
|
| | `demo-felhom` (N100) | `demo-hp` (HP t740) |
|
|
|---|---|---|
|
|
| host uptime | 20 d | 8 d 23 h |
|
|
| agent | 0.130.0, `active` | 0.130.0, `active` |
|
|
| controller | 0.223.0 | 0.223.0 |
|
|
| containers | 1/1 | **16/16, all `healthy`** |
|
|
| SMART, all disks | PASSED | PASSED |
|
|
| `journalctl -u felhom-agent -p warning`, 3 days | no entries | no entries |
|
|
| hub health | `ok` | `ok` |
|
|
|
|
The alarms are the box's own backup. Both bursts line up exactly with the nightly legs
|
|
(`backupwindow`: DB dump at W, tier-2 at W+60m, off-box at W+105m; default W = 02:30 CEST):
|
|
|
|
| leg | fired (UTC) | = CEST | events pushed |
|
|
|---|---|---|---|
|
|
| `db-dump` | 00:30 | 02:30 | Docmost, Paperless-ngx, RomM |
|
|
| `tier2-backup` | 01:30 | 03:30 | — |
|
|
| `offbox-backup` | 02:15 | 04:15 | Docmost, Paperless-ngx |
|
|
|
|
Evidence, from the guest's own controller log (all copied off the box before any change):
|
|
|
|
```
|
|
00:30:01 backup.go:868 [INFO] [backup] Stopping bookstack for safe volume dump
|
|
00:30:06 manager.go:1129 [INFO] [stacks] Stack bookstack stopped successfully (took 4.4s)
|
|
00:30:14 manager.go:1054 [INFO] [stacks] Stack bookstack started successfully (took 6.2s)
|
|
00:30:26 notifier.go:234 [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: Docmost
|
|
```
|
|
|
|
`DumpAppVolumesSafe` stops a stack (`docker compose down`), tars its volumes and starts it again —
|
|
**~13 s per stack, measured** — while the `deadapp-check` scheduler job runs every **30 s**. The scan
|
|
caught whichever stack was mid-cycle.
|
|
|
|
**The positive observable that proves the boxes were fine** (standing rule 3 — an absent alarm is not
|
|
evidence): every dead-app scan across the other 23 hours logged `8 deployed app(s) evaluated,
|
|
0 currently down`, and the same night's off-site run logged
|
|
`[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s`.
|
|
|
|
## 3. Root cause
|
|
|
|
**A mechanism that exists, works, and was never consulted.** `quiesce/suppress.go` solved exactly this
|
|
problem in v0.179.0 (R-97b). But `classifyRunStates` read only `quiesce.Loop.SuppressedStacks()`, and
|
|
the quiesce loop covers the **whole-guest** (vzdump/PBS) backup. The **per-app** legs stop stacks
|
|
through `Manager.DumpAppVolumesSafe`, which registered with no suppressor at all.
|
|
|
|
Two mechanisms in this product stop a customer's app on purpose. Only one told the alarm.
|
|
|
|
This is the **"seam built but never wired"** class, fifth instance — and the first where the unwired
|
|
half was a *consumer* rather than a producer. That distinction is why the tests below include an AST
|
|
wiring check: a test that injected the suppressor directly would have passed against the shipped bug.
|
|
|
|
## 4. The fix
|
|
|
|
`internal/backup/appstop_suppress.go` (new) puts the suppression on **`AppStopGuard`**, which already
|
|
brackets every deliberate stop in the product (`Begin` before the stop, `End` after a successful
|
|
restart) at all three call sites — volume dump, off-site reconstitute, `.fab` export — and which
|
|
`main.go` hands as ONE object to both the backup manager and the exporter. The fact the alarm needs
|
|
already lived there with exactly one writer; a fourth registry beside it would have been drift.
|
|
|
|
`scanDeployedAppRunStates` now passes `unionSuppressed(q.SuppressedStacks(), g.SuppressedStacks())`.
|
|
**All three per-app stop paths are fixed by the one change**, not only the nightly leg that was
|
|
reported — a `.fab` export and an off-site restore stop an app the same way and would alarm the same
|
|
way.
|
|
|
|
### It must never latch — the harder half
|
|
|
|
Permanent suppression trades a loud false alarm for a silent real one (F-CRIT-1, R-88 Scenario D).
|
|
`End()` runs **only on a restart that succeeded**, so an open-ended hold is a genuine hazard here in a
|
|
way it is not for the quiesce loop, which always releases. Three independent guards:
|
|
|
|
1. **`ReleaseFailed`** — a restart attempted and broken drops the entry **immediately**; the app
|
|
alarms on the very next scan, with no delay at all. Wired at every failure path (volume dump,
|
|
`restartStack` in the off-site reconstitution, the exporter's restart defer — the exporter's seam
|
|
interface grew the method rather than the exporter keeping separate bookkeeping).
|
|
2. **`Begin` replaces the set** — one marker file is one operation, so a set stranded by an operation
|
|
that died mid-window cannot survive into a later one.
|
|
3. **`appStopMaxHold` = 6 h** — a backstop for a hold nothing released, logged at WARN when it fires.
|
|
|
|
The post-restart grace is **180 s, the same constant and derivation as `quiesce.quiesceAlarmGrace`**.
|
|
Two windows over one alarm that disagreed on how long a restart takes would be a bug waiting to be
|
|
found on whichever path used the shorter one.
|
|
|
|
The suppression is **deliberately not persisted**: after a crash the guard holds nothing, `Recover()`
|
|
either brings the apps back or leaves them genuinely down, and a down app must alarm. `ReleaseFailed`
|
|
drops the suppression and **keeps** the durable crash marker — the two are independent, and a test
|
|
pins that.
|
|
|
|
## 5. Tests, and the red-proofs
|
|
|
|
`internal/backup/appstop_suppress_test.go` drives the **real** `DumpAppVolumesSafe` (not `Begin`
|
|
directly) and asserts the suppression set the dead-app scanner actually reads — the consequence, not
|
|
a log line.
|
|
|
|
| test | pins |
|
|
|---|---|
|
|
| `TestVolumeDump_SuppressesTheAlarmForTheAppItIsHolding` | the fix, through the production path |
|
|
| `TestVolumeDump_SuppressionExpiresSoARealOutageStillAlarms` | the window is a bounded delay, never a lost alarm |
|
|
| `TestVolumeDump_FailedRestartAlarmsImmediately` | anti-latch #1, and that the crash marker survives |
|
|
| `TestSuppression_CannotOutliveTheBackstop` | anti-latch #3 |
|
|
| `TestBeginReplacesThePreviousOperationsSet` | anti-latch #2 |
|
|
| `TestSuppression_FailedBeginSuppressesNothing` | a refused stop suppresses nothing |
|
|
| `TestHeldAppIsNotReportedDown` | the consequence: held app silent, genuinely exited app still alarms |
|
|
| `TestScanDeployedAppRunStatesIsGivenTheAppStopGuard` | the wiring, by AST walk |
|
|
|
|
**Three companion red-proofs were run, and each printed the pre-fix value:**
|
|
|
|
1. Delete `g.markStopped(stackNames)` from `Begin` →
|
|
`suppressed at stop = map[], want bookstack` — the exact shape that produced the e-mails.
|
|
(`TestVolumeDump_SuppressionExpires…` and `TestSuppression_CannotOutlive…` also went red.)
|
|
2. Delete `m.appStop.ReleaseFailed(stackName)` from the volume dump →
|
|
`suppressed = map[bookstack:true] after a restart that FAILED`.
|
|
3. Pass `nil` instead of `appStopGuard` in `main.go` → the AST wiring test failed.
|
|
|
|
Each was restored immediately and `git diff` verified clean afterwards. Red-proof 3 is the load-bearing
|
|
one: the component was never the broken part, so a suite that only injected it would have been green
|
|
against the shipped defect.
|
|
|
|
**Green gate:** `go build ./... && go vet ./... && go test ./...` in `felhom-controller/controller/` —
|
|
clean, no failures.
|
|
|
|
## 6. Not done, and why
|
|
|
|
- **No live deploy yet.** The build/deploy of 0.224.0 to the two boxes is the next step and is
|
|
reported separately; this report covers the change and its unit-land proof only.
|
|
- **The `restore-hold` path** (`offbox_reconstitute.go`, an app deliberately held down after a failed
|
|
replay) calls `End()`, so it gets the 180 s grace and then alarms. That is **today's behaviour plus
|
|
180 s** and is deliberate: the app really is down, the customer should learn that, and the hold has
|
|
its own operator notification (`restoreHoldNotify`) besides.
|
|
- **`HeldStacks()` was left alone.** It reads the marker from disk for the boot reconciler and covers
|
|
"held right now" but not the post-restart grace — which is precisely the window R-97b proved is
|
|
needed. The new in-memory set is a superset for alarm purposes; the durable one stays the recovery
|
|
record.
|
|
|
|
## 7. Found while diagnosing — a second, still-open defect (Option B)
|
|
|
|
**The hub's customer Backup card is inert for every customer.** It reads
|
|
`Snapshots 0 · Repo Size 0 MB · Integrity Unknown` while the same box's log says
|
|
`[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s)`.
|
|
|
|
`hub/internal/web/templates/customer_unified.html:284` renders `snapshot_count` / `repo_size_mb` /
|
|
`integrity_ok` from the report. Those fields are **declared** in
|
|
`controller/internal/report/types.go:105-108` and **assigned nowhere** — a repo-wide grep finds only
|
|
the declaration. The controller tracks the real numbers in `internal/backup/offbox.go` (`SnapshotCount`,
|
|
line 1043) and serves them on its own API (`internal/web/offbox_handlers.go:295`); they are simply
|
|
never copied into the hub report.
|
|
|
|
**Until it is fixed, that card must not be read as evidence of a missing backup.** It is a two-repo
|
|
change (controller report builder + hub) and is Option B of this request.
|
|
|
|
## 8. Also observed (not changed)
|
|
|
|
- **`ssh demo-hp` no longer works** — the tailnet peer `100.76.96.79` has been offline 8 days
|
|
(`tailscale status`: `offline, last seen 8d ago`). The box is reachable on the home LAN as
|
|
`ssh hp` → `192.168.0.104`, which is what every command in this report used.
|
|
- **`drill-r50-0a4f9a` still appears in the hub host list as `DOWN`** — a leftover record from the
|
|
R-50 drill whose rig was torn down; not a live box.
|