admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Campaign 9: the Tier-2 restore paths are PROVEN; 3 defects filed, none fixed

Browse Source 
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.

Three defects, recorded not fixed:
  C9-F1 (HIGH)   the Tier-2 restore button is offered for apps it can never restore, takes a
                 real outage, and reports "nothing was missing" — indistinguishable from a
                 genuine result, while 156 MB of that app's data sits unread in the same copy.
  C9-F2 (HIGH)   an app in a crash loop never alarms on any channel; StateRestarting is in no
                 down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
  C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
                 is dead until a human unlocks and the operator is told "unknown reason".
                 This answers Phase C item 8.

Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.

Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
This commit is contained in:
authentik Default Admin
2026-07-28 18:24:06 +02:00
parent f47b0a61d7
commit 955083c0fc
4 changed files with 981 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+68 -130
View File
@@ -1,147 +1,85 @@
# REPORT — closing the backup arc: seam sweep + live outage validation (2026-07-27)
# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
**Overwritten** per the standing rule. **No version shipped** — the sweep found no defect, so the
witnesses are compile-time-only source changes riding the next release. Agent `9842c52`→`0236553`,
controller `ca013c8`→`8f46495`. Live fleet unchanged: hub v0.79.0, agent v0.105.0, controller v0.178.0.
**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
and findings are recorded, never fixed inline. Full write-up:
`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
## Part 1 — the seam table
Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
Every optional-capability type assertion across all four repos. `StackDataProvider`, `BackupReporter`
and `RestoreTestReporter` are **not** in this table: they are interface-typed struct *fields*, which
the compiler already enforces at assignment. The dangerous shape is `x.(Interface)`.
## The headline — two never-proven restore paths are now proven
| Interface | Asserted at | On failure | Silent? | Witness now |
|---|---|---|---|---|
| `quiesce.TieredBackend` | `tiers.go:109` (`resolveDueTiers`) | falls back to the untargeted single-tier path | **SILENT** | ✅ moved to production |
| `quiesce.TieredBackend` | `quiesce.go:375` (`allTiersForManualRun`) | manual run covers only the untargeted tier | **SILENT** | ✅ same witness |
| `quiesce.TieredBackend` | `tiers.go:191`, `:199` | **unchecked — would panic** | no (loud) | ✅ same witness |
| `localapi.BackupArchiveLister` | `server.go:1155` (`newestArchiveOn`) | degrades to `archiveAbsent` = pre-R-84 in-memory-only | **SILENT** | ✅ added |
| `agentapi.AgentVersionReporter` | `features.go:157` | version gate → live probe | silent, benign (both decide correctly) | ✅ added |
| anonymous `GuestMemory` iface | `features.go:86` | returns `errNoMemoryProbe` → `SupportUnknown` | **no** — distinct error | n/a (anonymous) |
Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
and CSRF token, on live hardware.
**Was `TieredBackend` the only one, or the first one? It was the first one.**
`BackupArchiveLister` has the identical shape and a worse blast radius: a silent degrade to
`archiveAbsent` is the R-84 bug returning — a cold store after a restart reading as "no backup ever" —
with nothing in any log. It had **no witness of any kind**, and in fact **zero compile-time witnesses
existed in production code anywhere in either repo** before today.
**No defect found.** Every implementation satisfies its interface today, so per the task's rule there
is no version bump and no deploy. What was missing was the *guard*, not the behaviour.
**The guard is proven, not assumed.** Breaking `BackupRunner.NewestArchiveTime`'s signature now fails
`go build`:
```
cmd/felhom-agent/seam_witnesses.go:28:38: *backup.BackupRunner does not implement
localapi.BackupArchiveLister (wrong type for method NewestArchiveTime)
```
Before the witness, that same break compiled and vetted clean. The controller's witness also **moved
out of a `_test.go` file** — a test-file witness fires on `go test`/`go vet` but not on `go build`
alone, which is exactly the path the R-88 Part 2 near-miss would have taken.
## Part 2 — the live outage
Pre-flight cleared: 0 tasks in flight on ep0, 0 backup/restore processes on either box (counted by
`COMM`, since `pgrep -f` self-matches and gave a false `1` first), 0 scratch guests, no quiesce marker.
### The headline — `age_state=unknown` on real hardware, first ever
```json
{"vmid":9201,"due":true,
"reason":"backup storage unreadable and no in-memory record — age UNKNOWN, treating as due",
"target":"felhom-pbs","age_state":"unknown"}
```
Identical on both boxes. The transition is timestamped on demo-felhom:
```
18:35:30 felhom-pbs/content 200 ← before power-off
18:39:26 felhom-pbs/content 500 ← after
18:39:11 local/content 200 ← local tier, same window, unaffected
```
Agent journal, both boxes: `could not read the backup storage for the due-check … target=felhom-pbs`
— the degrade path genuinely executed. **Positive observables throughout**, per standing rule 3: an
empty quiesce log alone would have been equally consistent with a dead loop.
### The behaviour it gates
demo-felhom's `local` tier was **genuinely due** (25.8 h) for the whole window, so the due set was
`[local(known), felhom-pbs(unknown)]` → `oldestAge` nil → `valveLicensed` false → **defer**. This is
the 09:02 incident's exact condition replayed with the fix in place.
| | demo-felhom | demo-hp |
|---|---|---|
| App containers running | **10**, unchanged | **3**, unchanged |
| Quiesce cycles | **0** | **0** |
| Quiesce marker | absent | absent |
| vzdump attempted | none since 07:13 | none since 09:37 |
**Not one app stack was stopped.** Per-tier isolation demonstrated on the same box in the same
minute: `local` → 200 with a real age, `felhom-pbs` → 500 → unknown.
### Item 4 — conditions did not arise, and that is confirmed, not assumed
`felhom-pbs` was not due on either box (28 h / 7 h against a 168 h cadence), so no backup ran, nothing
failed, **the R-88 breaker did not arm**, and **no `whole_guest_backup_failed` travelled**. I did not
force one. Confirmed positively: the newest hub event is `16:10:33`, *before* the window opened at
16:24, and 0 events exist in the window.
## Part 2.2 — post-boot
| Check | Result |
| proof | result |
|---|---|
| Mount | `/mnt/pbs-datastore ext4 rw,relatime,discard` — **relatime present, `noatime` ABSENT** ✅ |
| Swap | `/swapfile 4G` back — **survived its first reboot** (it was added and verified without one) ✅ |
| PBS services | `proxmox-backup` + `proxmox-backup-proxy` both active ✅ |
| Datastore | `felhom-offsite → /mnt/pbs-datastore`, online ✅ |
| Snapshots | demo-felhom **2**, demo-hp **3** — **exact match**, same timestamps ✅ |
| Schedules | `gc-schedule sun 04:30`, `verify-new 1`, 2 prune jobs ✅ |
| Filesystem | `EXT4-fs (sda): recovery complete` — journal replayed after the unclean power-off, then mounted r/w; **0** I/O or EXT4 errors ✅ |
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
### The unplanned finding: the volume changed device name
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
at their last-backup state.**
```
before: volume=/dev/sdb root=/dev/sda1
after: volume=/dev/sda root=/dev/sdb1
```
## Findings — 3 defects, ranked (none fixed)
The kernel enumerated the disks in the opposite order across the reboot. The mount survived only
because fstab uses `/dev/disk/by-id/scsi-0HC_Volume_106469259`. **A `/dev/sdb` entry would have failed
to mount — or mounted the root disk's device node.** That choice was load-bearing and had never been
tested; this reboot tested it.
| # | finding | severity |
|---|---|---|
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
### Recovery
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
so live.
Both boxes returned to `age_state=known` on both tiers within ~60 s of boot; `felhom-pbs` on
demo-felhom back to `due=false, within cadence window, age_seconds=102102`. No breaker was armed, so
none needed clearing; no failure event fired, so no recovery event was expected — and none appeared.
## Mechanisms confirmed working, live
## Part 3 — CX33
R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
**Not available even with the server powered off**, so the powered-on theory is settled: it is the
Cost-Optimized line's "Limited availability", not the power state. No rescale happened. R-90 stays
`BLOCKED`/`WATCHING` with the 4 GiB swap as the interim — now a *proven* interim, since it survived a
reboot. Escape hatch if 8 GB becomes urgent: the **Regular Performance (CPX)** / **General Purpose
(CCX)** lines carry no availability warning, at higher cost. Operator's call.
## Where it stopped, and what remains
## Comes off the never-live-validated list
Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
- **`age_state=unknown` observed on real hardware** — the path R-88 Part 2 exists for.
- **The window gate refusing an unknown-driven valve, with a genuinely due tier present** — zero app
stacks stopped.
- **Per-tier isolation under a real offsite outage.**
- **The by-id mount surviving a reboot** (and surviving a device rename, which was not anticipated).
- **The 4 GiB swap surviving a reboot.**
- **Datastore integrity across an unclean power-off** — journal replay, exact snapshot match.
**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time
`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
method next time.
## Honestly cannot come off it
**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
- **The R-88 breaker has still never armed on real hardware.** No backup was due, so none failed.
- **`whole_guest_backup_failed` has never travelled from a REAL failure** — only from the induced
POST on 2026-07-27 15:04:57. The quiesce→notify link remains unit-proven.
- **R-97b's app-alarm suppression has never fired live** — it needs a real quiesce cycle.
- **`age_state=absent` has never been observed live** — it needs a genuinely fresh tier.
- **The legacy-agent path (Scenario C) is now unobservable on this fleet** — both agents are v0.105.0.
- **The first in-window scheduled backup** lands 2026-07-28 at the earliest.
## Recovery
Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
controller restart (by design, not persisted). Clearing it would have cost another app outage for no
benefit.
**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
`/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.