admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: R-86 closed and proven live; ep0 recorded as protected; R-185/186/187 filed
gates / gates (push) Successful in 8s
Details

Browse Source 
- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal
  reading is never true on a daily tier); R-87 re-ranked UP because R-86 built
  most of what it waited for; R-185 (the agent cannot list demo-felhom's host
  backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's
  sha is not reproducible from its tag), R-187 (R-115's publish leg had never
  actually run) filed. R-184 was the highest ID in use.
- ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row
  itself proposed — which would have been the never-fires version.
- 07-backup-architecture: new contract section — restore-testing is per ARCHIVE
  GENERATION, with the trap and what did not change (S-1).
- 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on
  the 635 s due-triggered offsite run, with the restart and teardown evidence.
- CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and
  S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines).
  Numbered 17/18 because S-14 and S-15 were already duplicated in the file.
- STATUS: rewritten for the operator, trimmed back to one screen.
This commit is contained in:
authentik Default Admin
2026-08-03 15:28:32 +02:00
parent ff2655cf19
commit c102832892
7 changed files with 214 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+101 -159
View File
@@ -1,188 +1,130 @@
# REPORT — R-182: one email per backup run, nothing dropped without a trace + ep0 rescaled (R-90, R-86)
# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)
**Date:** 2026-08-03 · **Repos:** `felhom-controller` **v0.194.0** (`88897a2`), `felhom.eu` hub
**v0.90.0 → v0.90.1** (`dd40f85`, `f21e7ca`)
**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1**
(`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not
after it. Ruling recorded: **ep0 is Tier 2, protected**.
## 1. Baselines — re-read on arrival
## 1. Baselines, re-read on arrival
| Repo | @ arrival | Version | Matched §1? |
| Repo | `main` @ commit | Version | Matched §1? |
|---|---|---|---|
| `felhom-controller` | `db0d4b129d3b` | v0.193.1 | **yes** |
| `felhom.eu` | `7dc1744eece3` | hub v0.89.0 | **yes** — my local clone was one commit behind origin (another session had deleted 24 stale `REPORT-*.md` files); pulled before starting |
| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** |
## 2. §3's safety property — confirmed at file:line
The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in
`REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and
`REPORT.md` is overwritten every session, so the per-repo history under-reported what was running.
`hub/internal/monitor/deadline.go:396` raises `expected_backup_missed` and `:417` raises
`expected_dbdump_missed`, both from `assessBackupFreshness(reportJSON,…)` and `GetEventsByType` — the
box's **report freshness and stored events**, never from an email. So the hub notices a run that never
finished **independently of any mail the controller chooses to send**, and a digest's silence still
means "the run finished and found nothing wrong". **The whole design rests on this; weakening that
check re-opens a silent-failure path.**
## 2. Why Part 2 could not ship later
## 3. Files
The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved
weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment
derived that number *from the cadence R-86 removes*.
**Hub:** `internal/notify/dispatcher.go` (suppression row, `cooldownRunSuffix`, `recordOnlyEvents`,
`operatorOnlyEvents`) · `internal/notify/templates.go` (digest renderer, `trimRepeatedUsage`) ·
`internal/api/handler.go` (allowlist) · `internal/store/store.go` (status doc) · two new test files ·
`recovery_unit_dispatch_test.go` (updated, deliberately) · `hub/CHANGELOG.md` · `manifests/hub.yaml`
The brief said a weekly tier would "sit exactly on that line". That is literally true, and the
arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the
current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false
by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late,
one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
**Docs:** `CONTEXT.md` (S-16) · `STATUS.md` · `documentation/architecture/07-backup-architecture.md`
(the alerting contract, S-1) · `00-capability-map.md` · `backlog/{OPEN-ITEMS,ROADMAP}.md` · five
runbooks corrected · two audit/evidence documents annotated
## 4. Tests and red-proofs
`go build && go vet && go test ./...` in `hub/` → **17 packages ok, rc=0**. `repo_gates.py --fast` →
all 5 gates OK.
| # | Mutation | Result |
|---|---|---|
| D | the suppression row removed (back to the bare `return`) | **RED** — `want 1 'suppressed' operator row, got 0` |
| E | `cooldownRunSuffix` removed from the key | **RED** — `operator mails = 1, want 2`; the manual run's digest is swallowed |
| G | the digest removed from `operatorOnlyEvents` | **RED** — and it showed **a customer actually receiving the operator digest**, demonstrated rather than argued |
| A | (controller) all four `noteFailure` feeds removed | **RED** |
| I | (controller) the `main.go` seam commented out | **DID NOT FAIL FIRST TIME** — see below |
**Red-proof I is the one worth reading.** The AST test walked the backup package but not `main.go`, so
the seam could be disconnected with the whole suite green. Per §9.11 that is not a pass: **the test was
fixed and the mutation re-run**, at which point it failed correctly. The string `SetRunSummaryNotify`
is still present in the commented-out file, which is exactly why this is an AST walk.
**One passing test was changed, deliberately.**
`TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer` required the OPERATOR to be emailed a per-app
capture failure — correct when that event was the only signal, wrong now that it is the record and the
digest is the notification. Its **customer-safety claim is unchanged** and is why the test still
exists; the operator assertion is inverted with the reasoning written in place, and it now asserts the
`recorded` row instead. R-158's guarantee **moved rather than weakened**.
## 5. The live run
Demo-hp guest 9201, filled for real to **241 MB free / 100% used** (thin pool held **30.78 → 30.78**).
Triggered through `POST /api/debug/backup/dbdump` — the exact endpoint the debug UI button calls.
**The hub's `notification_log`, quoted:**
## 3. The window as derived
```
11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "opengist" …
11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "privatebin" …
11:50:45 | backup_run_failures | operator | sent | | 2 of 2 apps failed to back up in this manual run: opengist, …
11:50:45 | backup_run_failures | customer | skipped | operator_only | 2 of 2 apps failed …
11:51:00 | disk_critical | operator | suppressed| operator cooldown 1h, key=demo-hp:disk_critical | …
11:51:00 | disk_critical | operator | sent | | …
interval = declaredArchiveInterval(tier) // host 26h, offsite 8d
if observedOK && observed > interval { interval = observed } // observation may only WIDEN
window = clamp(4 * interval, floor 7d, cap 12d)
```
Two failures → **both recorded**, **one** email naming **both**, customer refused with the reason
logged. Against the measured previous behaviour: two failures → one email naming one app, one leaving
no trace.
- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same
tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience.
- **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very
thresholds the backup-freshness checker already judges those tiers against. Not a second opinion:
if the two checkers disagreed about how often a tier receives an archive, one would be alarming on
the other's model.
- **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful
`backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both*
arrays, and classifying by array membership would attribute an offsite archive to the host tier).
Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
- **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before.
- **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is
never called stale against an archive PBS has already pruned.
**Scenario D proved itself on an event I had not planned.** `disk_critical` alarmed on two
filesystems; the second was collapsed by the cooldown, and that collapse is now **visible with its
key**. Yesterday it would have left nothing at all — which is the entire finding, demonstrated by
accident on a different event type.
| tier | interval used | window | proof age in health | verdict |
|---|---|---|---|---|
| daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before |
| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin |
| newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes |
**Scenario E:** a second run in the same hour (11:51:28) produced a second digest.
**Scenario B:** after freeing the space, the run completed with 2 volume dumps and **no** digest line.
## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box
## 6. The email as shipped
demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and
`2026-07-28T04:49:43Z` — **8 h 54 m apart**, because one is a healing artefact and the other a real
weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor
lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm
this whole task exists to prevent would have returned within a week, on the box it had just shipped
to.**
```
SUBJECT: [Felhom] 🔴 demo-hp: 2 of 2 apps failed to back up (manual run)
Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means
nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real
information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared
rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a
signal meaning *unverified* — *broken now* is `restore_test_failed`, immediate and untouched.
Customer: demo-hp
Event: backup_run_failures
Severity: error
Time: 2026-08-03 13:52 CEST
Message: 2 of 2 apps failed to back up in this manual run: opengist, privatebin
## 5. Kept, because it was earned
FAILED: 2 of 2 apps attempted in this manual run.
- **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged).
- **The stale signal stays edge-triggered** (`staleStates` untouched).
- **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken,
the other that it is unverified, and the second is the one that quietly becomes the first.
- **Every reason string now states the window it was judged against** — R-100's corollary: when a
verdict changes what it counts from, the alarm text must change with it, or an operator reads
"limit 168h" under a tier judged at 288 h and dismisses a true alarm.
- **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window
and to see two generations of a weekly tier.
opengist whole app (refused before any write) refused: backing up this app would leave the
filesystem below the reserve (reserve: 97% used
or 1.0 GiB free; the filesystem is already below
it, before this app's estimated 178.0 KB write)
privatebin whole app (refused before any write) refused: … estimated 2.5 KB write
## 6. Tests and red-proofs
Filesystem: /mnt/sys_drive — 65.0/68.7 GB used (95%), 0.2 GB free
| Test | Asserts | Mutation | Observed |
|---|---|---|---|
| `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL** — `week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` |
| `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval` → `observed > 0` | **FAIL** — `window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` |
| `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass |
| `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass |
| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged |
Every failure above is also recorded individually in the notification log,
whether or not this mail was sent.
**A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular
weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on
exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while
looking like proof. The jitter is what makes it a test — and it is also the truth about the old
constant.
Dashboard: https://hub.felhom.eu/customers/demo-hp
```
## 7. Deployment
**v0.90.1 came from reading exactly this.** In the first rendering every app row also ended with
`— /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free` — the same clause the `Filesystem:` line
already carries. On two apps that is untidy; down a list of a dozen it is forty characters twelve
times, pushing the part that *differs* off a phone screen at 07:00. The reserve's message is unchanged
(it is right for a single-app alert); the digest trims the duplicate when rendering, and only an exact
`— <target path>:` suffix.
## 7. §8.1's table, in practice
The run refused **both** deployed apps on the reserve → both in the digest. **No** deliberate skips
occurred in the live run (no drive was disconnected or decommissioned on that box), so that row of the
table was exercised by test only — `TestRunSummary_DeliberateSkipsAreNotFailures`, which marks one app
disconnected and one decommissioned through the real `settings` store and asserts **no digest at all**.
## 8. ep0 — measured, then corrected
| | measured on the box |
| Step | Evidence |
|---|---|
| hostname | `felhom-hetzner` |
| RAM | **7757 MB (8 GB)** — was 3.8 |
| vCPU | **4** |
| swapfile | **survived the resize** — `/swapfile`, 4 GiB, `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`) |
| local disk | `/` 38 G, 58% used — **UNCHANGED**, as expected for a CPU/RAM resize |
| datastore volume | `/mnt/pbs-datastore` 98 G, 8% — unaffected |
| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` |
| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` |
| ArgoCD deliberate sync | `sync=Synced health=Healthy` |
| Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` |
| Startup log | checkers initialized; `Listening on :8080` |
The swapfile was **checked rather than assumed**: a resize is a stop/start, so "the interim lever is
still in place" was an assumption until measured.
## 8. Part 3 — the ep0 ruling, recorded
**Corrected** (five places, three of which the spec's list did not name — found by searching):
`RUNBOOK-ep0-datastore-volume-2026-07-27.md:6` · `RUNBOOK-pbs-prune-serverside-2026-07-27.md:232,:310`
· `runbooks/offsite-endpoint.md:8,:34` · **`runbooks/target-selection.md:101`**.
`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"*
for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table
names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**.
`CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex,
Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores,
prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the
ordinary off-site read a restore-test performs remains permitted** — which this session then exercised.
**Annotated, body untouched** (they record what was true when written, and that is their value):
`audits/SPIKE-connectivity-wireguard-2026-07-03.md:13` ·
`tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md:142`.
## 9. Registers and docs
**No disk figure was touched anywhere** — the 40 GB local disk did not change.
## 9. Register
| ID | Outcome |
|---|---|
| **R-182** | **CLOSED — SHIPPED**, with the live evidence above |
| **R-90** | **CLOSED** — ep0 rescaled to a CX33; measured on the box, swapfile confirmed surviving |
| **R-86** | **READY, unblocked and re-ranked** — endpoint headroom no longer sets the restore-test cadence. Stated honestly: 8 GB is comfortable, not unbounded (the original OOM was a 14.46 GB restore), so the cadence should still be paced — just not by fear of the endpoint |
**No new IDs were minted**, so no grep was needed. R-184 remains the highest.
**`target-selection.md`'s ep0 protection question is STILL OPEN and deliberately untouched** —
*"D-d did not name ep0 either way. Confirm it explicitly."* That is the operator's to settle, not CC's.
## 10. Deployed
- Controller **0.194.0** on demo-hp guest 9201 — `Up (healthy)`.
- Hub **0.90.1** — ArgoCD `Synced` / `Healthy`, image `gitea.dooplex.hu/admin/felhom-hub:0.90.1`,
startup log clean (`Listening on :8080`, deadline-check scheduled for 05:00 CEST).
## 11. Teardown
Fill file removed; guest helper scripts and the credential file `shred`-ed; `pct fstrim 9201` returned
63.3 GiB; thin pool **30.76**; the guest is back to 1.9 G used / 64 G free. The scratch email renderer
under `hub/tmprender/` was deleted after use. The hub DB copy is scratch-only.
## 12. Observations — NOT acted on
1. **Only two apps are deployed on the demo box**, so "several apps" was 2 of 2 — exactly the measured
case, but a five-app demonstration would be stronger.
2. **`FormatOperatorEmail` puts an emoji in every operator subject.** Part 3 asked for none; the icon
comes from the shared formatter and changing it globally would alter every other operator mail and
its tests. The digest's own copy has no emoji.
3. **A `recorded` row is written for record-only events on the `operator` channel.** It is not an
operator *notification*, so the channel label is slightly overloaded — readable today, worth a
thought if a third channel ever appears.
4. **The customer-channel `skipped/operator_only` row no longer appears for the per-app event**,
because record-only returns before the customer leg. That is correct, but anyone grepping history
for that pattern will see it stop on 2026-08-03.
- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row.
- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific.
- **R-185 / R-186 / R-187 → filed** (grep established 185–187 free; R-184 was the highest in use).
- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself
proposed** — which was the never-fires version.
- `STATUS.md` rewritten for the operator, trimmed back to one screen.
- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test
row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key)
and S-18 (ep0).