Compare commits
23 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 51871a7ea6 | |||
| 33f0ab2f33 | |||
| 2584dfb938 | |||
| 185661062a | |||
| 967ddf2f44 | |||
| d692c46db0 | |||
| b93ee06abc | |||
| e3187c86d5 | |||
| 311dc06c13 | |||
| 688470c945 | |||
| 7a5694341d | |||
| 81d4438630 | |||
| c9a3e48b21 | |||
| 0779c5db85 | |||
| 0f8f084817 | |||
| c102832892 | |||
| ff2655cf19 | |||
| 046df303b6 | |||
| 687fedd8ee | |||
| 323f45a5ef | |||
| e34b614e5b | |||
| f21e7caed1 | |||
| dd40f85bb8 |
+169
@@ -17,6 +17,148 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
|
||||
(2026-08-04). Scope: those two boxes, not the fleet.**
|
||||
|
||||
Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite
|
||||
**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest,
|
||||
booted, verified and destroyed itself.
|
||||
|
||||
*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE
|
||||
tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and
|
||||
not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an
|
||||
archive there to test.
|
||||
|
||||
*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven
|
||||
sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next
|
||||
evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together,
|
||||
unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path
|
||||
carrying a host-tier entry for the first time.
|
||||
|
||||
*The asymmetry worth remembering:* a host-tier restore is **83–109 s**; an offsite one is
|
||||
**300–540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove.
|
||||
|
||||
**S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
|
||||
question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**
|
||||
|
||||
*The defect.* On both demo boxes the agent's token had `FelhomAgentStore` on `local`, `local-lvm` and
|
||||
`felhom-pbs` and **not** on `felhom-backup` — the storage the same installer configured as
|
||||
`local_backup_target`. The content API answered `{"data":[]}` through the token while root listed
|
||||
three archives. `pickForThisRun` skipped the tier as *"no settled archive yet"*, which is exactly
|
||||
what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.
|
||||
|
||||
*The rule.* The permission question has a definite answer where the listing does not. `Permissions`
|
||||
reads `/access/permissions?path=/storage/<target>` **as the agent's own token** — asking as root
|
||||
answers a different question and always says yes — and one `capability.Status` per configured tier
|
||||
reports it. The probed set comes from `BackupTiers()`, never a fixed list: a hardcoded probe list is
|
||||
the defect reproduced inside the fix.
|
||||
|
||||
*The measured trap, because the obvious reading is wrong.* An ungranted path answers **neither empty
|
||||
nor 403**: it carries the privileges inherited from the box-wide `/` grant
|
||||
(`Sys.Audit, SDN.Use, Datastore.Audit`). Testing for path-presence, or for `Datastore.Audit`, reports
|
||||
a blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**, and re-measuring is required
|
||||
before that constant is ever changed.
|
||||
|
||||
*Criticality, weighed once.* Critical, because the hub alerts only on critical and a non-critical
|
||||
entry would ride the report and alert nobody. **Except** the `local` fallback target, which
|
||||
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
|
||||
does not page — turning an ordinary documented setup into an alert is how a signal becomes something
|
||||
an operator archives unread. It never consults content, so it cannot alarm on a newborn tier by
|
||||
construction, and it never reports ok when it could not ask.
|
||||
|
||||
**S-22 — the installer's Scenario-F arm must finish the job, not just leave the definition alone
|
||||
(2026-08-03, R-185).** `configure_backup_target` has two arms. Case A creates the storage and grants
|
||||
in the same breath. The reuse arm — *"the target already exists"* — returned **without granting**, and
|
||||
that, not `PVE_STORAGES`, is where the drift came from: a box whose target pre-dated the install
|
||||
(the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token
|
||||
could not read. The reuse arm now ensures the ACL through the same guarded wrapper. **Scenario F is
|
||||
unviolated** — the storage DEFINITION is untouched, and granting the role the agent is supposed to
|
||||
have on the target this script is about to write into `agent.json` is finishing the job, not
|
||||
retargeting the box. `$BACKUP_TARGET_ID` stays OUT of `PVE_STORAGES`: that list is granted a step
|
||||
before the target is resolved, and `--acl-storages` entries are preflight-checked for existence.
|
||||
A gate asserts every arm that resolves the target also grants on it.
|
||||
|
||||
**S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
|
||||
the design (2026-08-03, R-189; agent v0.122.0).**
|
||||
|
||||
*The rule.* Only successful restore-tests are written to `RestoreTestState`, and that state is what
|
||||
the host report carries after a restart. Failures live only in the in-memory `backup.Store`.
|
||||
|
||||
*Why, in one line each.* A **success suppresses future work** — under R-86's per-archive due-check a
|
||||
proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it
|
||||
believes, for a whole archive generation (a week on the offsite tier). A **failure causes future
|
||||
work** — a failing tier stays due and is retried at the next evaluation, so a lost failure heals
|
||||
itself within one interval, while a *persisted* failure would outlive the fault it describes.
|
||||
|
||||
*What the report does with the two.* The collector merges them: **one entry per tier, newest by
|
||||
`TestedAt` wins**. A fresh failure therefore beats a stored success (the failure is the news and
|
||||
exists nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier can
|
||||
never appear twice — the hub would read that as two tests.
|
||||
|
||||
*It refuses to lie.* A persisted record missing the archive **or** the tier produces **no entry**: an
|
||||
unproven tier reading as proven would be worse than the defect this closes. Run mechanics (scratch
|
||||
VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be.
|
||||
**Migration consequence, seen live:** a pre-R-189 record has no tier, so upgrading does not
|
||||
retroactively make an old proof visible to the hub; the tier's next real proof fills it in.
|
||||
|
||||
**S-20 — the release order is build → tag LOCALLY → publish → push tag, and every step protects
|
||||
something (2026-08-03, R-188 + R-186).**
|
||||
|
||||
The tag is created before the publish, so the build and the tag describe the same commit. It is
|
||||
**pushed** after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
|
||||
package made `check-published-versions.py` correctly fail a *correct* release — measured on roughly
|
||||
every second release, and R-168 mails those failures to the operator.
|
||||
|
||||
The invariant the old order protected is **asserted directly instead**: the gate now also refuses a
|
||||
**published version with no tag**, as a bounded probe (frontier + patch gaps) that prints its own
|
||||
coverage, because the package listing api is 401 without a token and absence cannot be enumerated.
|
||||
A half-done release is loud: publish-then-failed-push dies naming the recovery command, and a failed
|
||||
publish deletes the local-only tag so a retry is clean.
|
||||
|
||||
**A released binary is independently verifiable** — `-trimpath -buildvcs=false` means the same source
|
||||
yields the same bytes with or without the tag; the verification command lives in
|
||||
`felhom-agent/CLAUDE.md`. Both build paths (`release-agent.sh` and `publish-agent.sh`'s fallback) use
|
||||
identical flags: they differed by `CGO_ENABLED=0` and produced binaries 74 KB apart for one version.
|
||||
|
||||
**S-17 — restore-testing is PER ARCHIVE GENERATION, and the hub's staleness window follows each
|
||||
tier's own rhythm (2026-08-03, R-86; agent v0.121.0 + hub v0.91.0).**
|
||||
|
||||
*The rule.* Let **A** be the newest archive on a tier that has settled for at least the settle lag
|
||||
(24 h). The tier is **DUE** when A exists and **A has not already been proven**. The daemon-start
|
||||
ticker survives only as the **evaluation interval** (6 h). A daily tier is proved daily on yesterday's
|
||||
archive; a weekly tier weekly on its own; a tier with no archive is UNKNOWN, never a fault.
|
||||
|
||||
*The trap, written down so it is not reintroduced.* The literal reading of R-86 — *"due when the
|
||||
newest archive is ≥24 h old"* — is **never true on a daily tier**, because a new archive resets the
|
||||
newest-archive age to zero long before it reaches the lag. It would have switched restore-testing off
|
||||
for the tier that matters most, silently. Red-proved at 0 runs over 5 simulated days
|
||||
(`felhom-agent/internal/backup/restoretest_due_test.go`).
|
||||
|
||||
*What the state holds now.* `RestoreTestState` records **which archive** was proven, not just when a
|
||||
tier passed — a timestamp cannot answer *"have we proven THIS archive"*. A pre-R-86 file keeps its
|
||||
time (rotation ordering survives a deploy) and yields no proven archive, so each tier is due exactly
|
||||
once after the upgrade.
|
||||
|
||||
*The old config key.* `backup.restore_test_cadence_seconds` is DEPRECATED. **Negative still disables**
|
||||
verbatim; a positive value now seeds the **settle lag** only, and the daemon WARNs once at start-up
|
||||
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
|
||||
(default 24 h). It is deliberately NOT carried into the evaluation interval.
|
||||
|
||||
*The hub half is not optional.* `restoreProvenStaleAfter` was a flat 7 days **derived from the cadence
|
||||
R-86 removes**, and a healthy weekly tier's proof age reaches EXACTLY 168 h just before its next
|
||||
proof — it sat ON the line. `restoreProvenWindow(tier, observed, ok)` now takes the tier's own
|
||||
observed archive interval × 4 generations, floored at 7 days, capped at 12 days (strictly inside the
|
||||
2-week offsite retention), falling back to the tier's **declared** rhythm (`backupStaleAfter` 26 h /
|
||||
`offsiteBackupStaleAfter` 8 d — the backup-freshness checker's own thresholds) when history is too
|
||||
short to observe one. Shipping Part 1 alone would have produced a nightly false alarm.
|
||||
|
||||
**S-18 — `ep0` is Tier 2, PROTECTED (operator ruling, 2026-08-03).** D-d named two protected machines
|
||||
and did not name ep0 either way; `runbooks/target-selection.md` carried the question in writing for
|
||||
two days. The ruling **extends D-d's protected list to three machines**: DooPlex, Peti's cluster,
|
||||
**ep0**. It is a classification, not a new set of prohibitions — destroying datastores, prune jobs,
|
||||
tunnel config or nftables rules was already forbidden by what it would destroy, and the ordinary
|
||||
off-site READ a restore-test performs remains permitted.
|
||||
|
||||
**S-13 — the `mp1` merge landed, and the variant was chosen on measurement (2026-08-03, R-165 / D-a).**
|
||||
The appliance's two data volumes are one. **Variant V-c**: the volume mounts at the NEUTRAL path
|
||||
`/var/lib/felhom`, and both `/var/lib/docker` and `/mnt/sys_drive` are binds of subdirectories of it.
|
||||
@@ -123,6 +265,33 @@ Two rulings, one shape: something became live because someone pushed, not becaus
|
||||
that bumps a version, before publishing — and a gate that fails on the normal path is one people
|
||||
learn to ignore.
|
||||
|
||||
**S-16 — a backup run NOTIFIES ONCE and RECORDS ALWAYS, and those are different things
|
||||
(2026-08-03, R-182; controller v0.194.0 + hub v0.90.0/.1).** Measured: nine per-app capture failures
|
||||
reached the hub, two were mailed, seven were dropped by a cooldown whose key carries no app
|
||||
identifier — *before* `LogNotification`, so they left no row anywhere.
|
||||
|
||||
- **The record:** `recovery_unit_capture_failed`, per app, unconditionally, now routed
|
||||
**record-only** by the hub (`recordOnlyEvents`) — stored and logged every time, never competing
|
||||
for an e-mail slot.
|
||||
- **The notification:** `backup_run_failures`, ONE per run, listing every failed app with its leg
|
||||
and reason plus the counts and free space. **A clean run emits nothing.**
|
||||
- **A suppressed operator event now leaves a `suppressed` row** carrying the key that suppressed it —
|
||||
for every operator type. *"We chose not to e-mail you"* and *"nothing happened"* must never look
|
||||
identical; that is the whole finding, stated as a rule.
|
||||
- **The periodic sweep gets a digest too, with NO `run_id`**, so it stays under the ordinary hourly
|
||||
cooldown. Without it the sweep's failures would be recorded and never notified — a new silence
|
||||
created while closing one. A real run's digest carries a unique `run_id` precisely so the cooldown
|
||||
can never collapse a manual run into the nightly one.
|
||||
- **Why the silence is safe:** the hub's deadline check raises `expected_backup_missed` from report
|
||||
freshness, independently of any mail the box sends (`monitor/deadline.go:396,417`). **If that check
|
||||
is ever weakened, this design loses its footing.**
|
||||
- **Not taken, and why:** putting `app` in the cooldown key. It fixes the swallowing by producing one
|
||||
mail per failing app — a dozen on a full disk.
|
||||
|
||||
**ep0 was rescaled by the operator to a CX33 (2026-08-03): 4 vCPU, 8 GB RAM, measured on the box, and
|
||||
the 4 GiB swapfile survived. The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only, so no disk
|
||||
figure in any runbook needed correcting. That closed **R-90** and unblocked **R-86**.
|
||||
|
||||
**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
|
||||
Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
|
||||
CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
|
||||
|
||||
@@ -1,200 +1,173 @@
|
||||
# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed
|
||||
# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
|
||||
|
||||
**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**)
|
||||
**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.**
|
||||
**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
|
||||
and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
|
||||
box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
|
||||
`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
|
||||
source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
|
||||
|
||||
## 1. Baselines — re-read on arrival, both matched §1
|
||||
**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
|
||||
|
||||
| Repo | @ arrival | Version | Result |
|
||||
|---|---|---|---|
|
||||
| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` |
|
||||
| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only |
|
||||
---
|
||||
|
||||
## 2. Part 0 — the R-182 measurement, and it REVERSED the row
|
||||
## 1. Preconditions
|
||||
|
||||
Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed
|
||||
at the sending end. Measured at the **receiving end**, it is the opposite.
|
||||
|
||||
Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db`
|
||||
alone would have read stale data, the exact trap this project recorded before), freshness confirmed by
|
||||
the newest `notification_log` row post-dating the session.
|
||||
|
||||
**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.**
|
||||
|
||||
| time | apps refused (events in) | operator emails out |
|
||||
| # | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| 06:40:03 | privatebin, opengist | **opengist only** |
|
||||
| 08:59:46/47 | opengist, privatebin | **privatebin only** |
|
||||
| 08:59:59 | privatebin, opengist | **none** |
|
||||
| 09:03:00 | opengist | **none** |
|
||||
| 09:07:06 | privatebin, opengist | **none** |
|
||||
| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
|
||||
| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
|
||||
| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
|
||||
| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
|
||||
| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
|
||||
| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
|
||||
| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
|
||||
|
||||
**Cause, confirmed at source:** the operator cooldown key is
|
||||
`customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded).
|
||||
`RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key
|
||||
holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the
|
||||
next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any
|
||||
channel** and cannot be audited afterwards.
|
||||
|
||||
This is **R-97a's failure mode in a second event type**; that row's own comment states it
|
||||
(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was
|
||||
written narrow on purpose; `recovery_unit_capture_failed` simply never opted in.
|
||||
|
||||
**A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app,
|
||||
HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the
|
||||
operator was told about each app* — which is false. The gap between an accepted event and a sent
|
||||
email is the whole of this row.
|
||||
|
||||
**Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape.
|
||||
|
||||
## 3. Probes
|
||||
|
||||
| | Question | Method | Verdict |
|
||||
|---|---|---|---|
|
||||
| **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) |
|
||||
| **P-B** | does Gitea serve `raw/tag/<tag>/<path>`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` |
|
||||
| **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** |
|
||||
|
||||
**Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in
|
||||
~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone
|
||||
(`git ls-remote --tags` → 0 at the time).
|
||||
|
||||
## 4. §8.2's three channels — enumerated
|
||||
|
||||
| Channel | Before | After | |
|
||||
|---|---|---|---|
|
||||
| 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs |
|
||||
| 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below |
|
||||
| 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission |
|
||||
|
||||
**Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.**
|
||||
There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so
|
||||
no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"*
|
||||
was impossible for them. Raised before building; operator ruled to pin them to **the agent version
|
||||
being installed**, which the installer already resolves from the hub manifest and already sha-verifies.
|
||||
That is strictly better than any installer-derived tag: binary and configs now come from one ref.
|
||||
|
||||
**Channel 3 needed no change because the URL never carried a ref** —
|
||||
`https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So
|
||||
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template
|
||||
change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose.
|
||||
|
||||
## 5. The tag convention
|
||||
|
||||
- **Shape:** `installer-v<SCRIPT_VERSION>` in `felhom.eu` (prefixed so it cannot be read as a hub,
|
||||
agent, controller or golden version); `v<semver>` in `felhom-agent` (that repo versions one thing).
|
||||
**No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time,
|
||||
and channel 1's ref lives only in the manifest.
|
||||
- **Publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (sidecar *and*
|
||||
init container), commit, sync.
|
||||
- **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**.
|
||||
|
||||
## 6. Scenario A — proven by HTTP
|
||||
|
||||
A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and
|
||||
three sync periods were allowed to pass so "unchanged" means "had every chance to change":
|
||||
## 2. The due verdicts before the run, quoted
|
||||
|
||||
```
|
||||
website tree (main): .worktrees/6a82719… <- ADVANCED to the new commit
|
||||
scripts tree (tag): .worktrees/bee6848… <- STAYED
|
||||
sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
|
||||
sha256 after push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
|
||||
marker present at the served URL? 0
|
||||
https://felhom.eu/ -> HTTP 200
|
||||
demo-felhom tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst"
|
||||
reason: newest settled archive … has not been proven; nothing proven on this tier yet
|
||||
tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z"
|
||||
reason: … has not been proven (last proven archive was a different one)
|
||||
|
||||
demo-hp tier=felhom-backup due=true archive="…2026_08_02-04_49_29.tar.zst"
|
||||
tier=felhom-pbs due=true archive="…2026-07-28T19:19:45Z"
|
||||
```
|
||||
|
||||
Both halves of the split in one observation: the site still tracks `main`, the installer does not.
|
||||
**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
|
||||
|
||||
## 7. Scenario B — publish and rollback, both directions
|
||||
## 3. The runs — all four SCHEDULED, none triggered
|
||||
|
||||
| act | result |
|
||||
|---|---|
|
||||
| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** |
|
||||
| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** |
|
||||
| box | tier | due at | archive | result |
|
||||
|---|---|---|---|---|
|
||||
| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
|
||||
| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
|
||||
| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
|
||||
| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
|
||||
|
||||
`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved
|
||||
to `main`'s head — a **byte no-op**, verified by the served sha not changing.
|
||||
**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
|
||||
only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
|
||||
which start nothing.
|
||||
|
||||
## 8. Files, commits, tags
|
||||
**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
|
||||
proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
|
||||
evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
|
||||
is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
|
||||
|
||||
**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b`
|
||||
(marker removed), plus the docs commit below.
|
||||
`scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` ·
|
||||
`manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` ·
|
||||
`documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md`
|
||||
**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The
|
||||
tier an ordinary recovery uses is also the cheapest to prove.
|
||||
|
||||
**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT).
|
||||
`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** ·
|
||||
`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md`
|
||||
## 4. What the runs left behind
|
||||
|
||||
**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and
|
||||
`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from;
|
||||
`configs/` is byte-identical there and at `main`, so nothing depended on the choice).
|
||||
**Persisted state — v3, naming the archive, the tier and what was verified:**
|
||||
|
||||
## 9. Tests and red-proofs
|
||||
```json
|
||||
demo-felhom felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
|
||||
proven_at 2026-08-03T22:56:45Z}
|
||||
felhom-pbs → {archive …2026-07-28T04:49:43Z, tier pbs, proven_at 2026-08-04T05:04:21Z}
|
||||
demo-hp felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
|
||||
felhom-pbs → {archive …2026-07-28T19:19:45Z, tier pbs, proven_at 2026-08-04T06:10:39Z}
|
||||
```
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `felhom.eu` `repo_gates.py --fast` | all 5 gates OK |
|
||||
| `felhom-agent` `go build ./... && go vet ./...` | OK |
|
||||
| `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
|
||||
| `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) |
|
||||
| `agent_gates.py` (full) | both OK |
|
||||
**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
|
||||
latest report holds **two** entries, one per tier; the `local` one can only have come from the
|
||||
persisted state, because the in-memory store held only that morning's offsite run:
|
||||
|
||||
**Red-proofs, each demonstrated failing then restored:**
|
||||
```
|
||||
demo-felhom tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
|
||||
tier=pbs pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z tested_at=2026-08-04T05:04:21Z
|
||||
demo-hp tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
|
||||
```
|
||||
|
||||
| # | Mutation | Result |
|
||||
**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
|
||||
|
||||
```
|
||||
tier=felhom-pbs due=false proven="…2026-07-28T04:49:43Z"
|
||||
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
|
||||
```
|
||||
|
||||
…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
|
||||
That is not a defect: it is "proved daily, on its own archive", visible one day later.
|
||||
|
||||
## 5. Teardown — three layers, per box
|
||||
|
||||
| layer | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired |
|
||||
| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it |
|
||||
| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state |
|
||||
| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 |
|
||||
| F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 |
|
||||
| the machine | `pct list` → **0** entries for 990000 | **0** |
|
||||
| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
|
||||
| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
|
||||
|
||||
**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit*
|
||||
`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also
|
||||
retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was
|
||||
this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API
|
||||
token, and the only credential available (`~/.docker/config.json`) is a registry password that the API
|
||||
rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.
|
||||
Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
|
||||
|
||||
## 10. No version bumps, nothing built
|
||||
## 6. What the run surfaced — three findings, two of them corrections to my own record
|
||||
|
||||
`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file
|
||||
touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's
|
||||
no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or
|
||||
golden was built.
|
||||
### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
|
||||
|
||||
## 11. Register
|
||||
A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
|
||||
2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
|
||||
~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
|
||||
(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry.
|
||||
Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before.
|
||||
**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
|
||||
and v0.123.0's probe detects the *state* but says nothing about the *transition*.
|
||||
|
||||
| ID | Outcome |
|
||||
|---|---|
|
||||
| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed |
|
||||
| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) |
|
||||
| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described |
|
||||
| **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs |
|
||||
| **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released |
|
||||
### R-191 (new) — every weekly offsite backup reports FAILED although it worked
|
||||
|
||||
**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each;
|
||||
all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0
|
||||
hits anywhere and remains free.
|
||||
demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
|
||||
incrementally) and the job then failed on the prune —
|
||||
`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
|
||||
errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
|
||||
(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
|
||||
the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
|
||||
must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
|
||||
— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
|
||||
(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
|
||||
namespaces.
|
||||
|
||||
## 12. Observations — noticed, documented, NOT acted on
|
||||
### Two corrections to yesterday's record
|
||||
|
||||
1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs
|
||||
either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails
|
||||
closed where the mistake is made, needs no new credential).
|
||||
2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`,
|
||||
so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside
|
||||
R-182 because it is what made that row take a day to get the right way round.
|
||||
3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it
|
||||
means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
|
||||
4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a
|
||||
network gate is ever added there, that workflow needs the same change the agent's just got.
|
||||
1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
|
||||
**six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
|
||||
`whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
|
||||
said backups kept landing because writes go through a root path — wrong, and now corrected in the
|
||||
runbook and on the row.
|
||||
2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
|
||||
I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
|
||||
since the R-86 work — and one **more** since, below.
|
||||
|
||||
## 13. Teardown
|
||||
**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
|
||||
2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
|
||||
host tier had never been proven, and the hub said so on its own, without being asked. It has not
|
||||
re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
|
||||
|
||||
Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag
|
||||
`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from
|
||||
`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl
|
||||
pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.
|
||||
## 7. The capability map
|
||||
|
||||
The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
|
||||
boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
|
||||
teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
|
||||
is untested and untouched, and one box proving something does not make it a fleet property.
|
||||
|
||||
## 8. Registers
|
||||
|
||||
- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
|
||||
correction in §6.
|
||||
- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
|
||||
the highest in use).
|
||||
- `ROADMAP.md` holds none of these rows, so nothing to collapse.
|
||||
- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
|
||||
"What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
|
||||
|
||||
## 9. Observations — noticed, NOT acted on
|
||||
|
||||
- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
|
||||
returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
|
||||
warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
|
||||
the restore pointed at `/mnt/nvme-1tb` instead.
|
||||
- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
|
||||
none". Nothing depends on that assumption now, but the operations notes still carry it.
|
||||
- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
|
||||
role — a 2.7× difference worth understanding before either is used to size anything.
|
||||
- **Both waiters this session produced no output** despite the runs completing; the evidence was
|
||||
gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
|
||||
instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
|
||||
the hub's database, not on the waiters.
|
||||
|
||||
@@ -104,7 +104,7 @@
|
||||
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
|
||||
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
|
||||
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
|
||||
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. |
|
||||
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
|
||||
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
|
||||
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
|
||||
| `cfg.DRTier` + offsite coupling | hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field `dr_tier` (formBool helper). |
|
||||
|
||||
@@ -14,114 +14,76 @@ A blank machine boots the Felhom disc, installs itself unattended, and is claime
|
||||
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
||||
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
||||
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
||||
restore files and app data from the drive alone. Proven end to end on real hardware.
|
||||
|
||||
**Apps come back after a power cut.** The machine tells an app the customer switched off from one
|
||||
that simply did not come back, and waits for the system to finish starting before deciding instead of
|
||||
glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every
|
||||
time, and an app switched off deliberately stayed off every time.
|
||||
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
|
||||
box six times, everything returned every time, and an app switched off deliberately stayed off.
|
||||
Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
|
||||
**The off-site copy can be erased by the machine that made it** — the credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
*(R-95, R-87)*
|
||||
|
||||
**A full disk tells you about ONE app and silently swallows the rest.** Yesterday this was written
|
||||
down the wrong way round — as *too many* emails. Measuring the receiving end reversed it: of nine
|
||||
refusals the machine reported today, **two emails were sent**. When two apps are refused in the same
|
||||
second you are told about one of them, and the other leaves no trace anywhere — not an email, not
|
||||
even a line in the log saying it was held back. So a second app can be going unbacked-up while you
|
||||
have already been told the problem is handled. It is the same fault we fixed once before for
|
||||
whole-machine backups, in a second place that never opted into the fix. *(R-182)*
|
||||
- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on
|
||||
3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued,
|
||||
the machine picked the new password up in 15 seconds and reattached to the same repository, and you
|
||||
escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly
|
||||
once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild.
|
||||
The other machine survived the same rebuild only because an unused password happened to be waiting
|
||||
for it. *(R-193)*
|
||||
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
|
||||
this declines without saying why. The message says the password was never applied; it was, on
|
||||
23 July, and worked for eleven days. *(R-192)*
|
||||
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
|
||||
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
|
||||
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
|
||||
setting per machine fixes it. *(R-191)*
|
||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap.
|
||||
*(R-95, R-87)*
|
||||
|
||||
## What shipped recently
|
||||
|
||||
**Pushing the installer no longer publishes it.** The script that runs as root on a brand-new
|
||||
machine was copied from the main branch and served within thirty seconds, so pushing it *was*
|
||||
publishing it, with no staging and no way back but another push. It now comes from a **labelled**
|
||||
version: publishing is moving the label, and undoing it is moving the label back — about half a
|
||||
minute, no deploy. The website is untouched by this and still updates in thirty seconds, because a
|
||||
typo fix must never need a release. Proven by actually doing it: a real push changed nothing that
|
||||
anyone downloads, moving the label published it, moving it back restored the previous bytes exactly.
|
||||
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
|
||||
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
|
||||
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
|
||||
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
|
||||
never been tested on either machine. Both also proved their off-site copy on the same night, one
|
||||
after the other rather than at once, which is the machine deciding for itself what to do first.
|
||||
*(closes the last open half of R-86/R-185)*
|
||||
- **A backup copy the machine was never allowed to read — and could not tell you about**, on both
|
||||
demo machines. The permission was one command; the silence was the real fault, and the machine now
|
||||
checks whether it may read each copy it depends on and says so when it may not. *(R-185)*
|
||||
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
||||
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
||||
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
||||
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
|
||||
written to disk with the result and survives. This was caught happening, not predicted: a real
|
||||
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
|
||||
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
|
||||
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
||||
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
||||
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
||||
|
||||
**The catch that would have made it cosmetic was found and covered.** While it runs, the installer
|
||||
fetches sixteen more files — not nine, and from the *agent's* repository, not the website's. They now
|
||||
come from the same version of the agent the machine is installing. That closed a real fault nobody
|
||||
had noticed: a new machine was getting the agent's tested program and its untested settings files, in
|
||||
one install, from two different places. *(R-110, R-183)*
|
||||
|
||||
**Releasing the agent now publishes it, in one command.** Putting a built agent where a new machine
|
||||
can download it was a step someone had to remember, and it was forgotten three times in five days —
|
||||
the last time leaving both demo machines running a version nobody could download, so a rebuild would
|
||||
have quietly installed the *older* one and reported success. There is now one command that builds,
|
||||
labels, publishes and then **downloads it back to check** — and a check that refuses to stay quiet if
|
||||
a released version cannot actually be fetched. Proven by making CI fail on purpose and then go green
|
||||
again on the same code. *(R-115)*
|
||||
|
||||
**The backup partition is gone and both demo machines run on the new shape** — wiped, rebuilt and
|
||||
taken through the whole customer journey on 3 August, by two deliberately different routes so the disk
|
||||
shape and the delivery route are both proven. The space a backup can use went from 19 GB to 65 GB on
|
||||
the small machine and 45 GB to 233 GB on the big one. Their previous demo apps and data are gone; that
|
||||
was the point of a wipe, and you approved it. *(R-165, R-178)*
|
||||
|
||||
**What replaced the wall now watches the right moment.** The wall was quietly keeping a runaway
|
||||
backup from eating the space the machine needs to run. As first built, that replacement was checked
|
||||
too late — the big write happened first, unchecked — while still promising your last good copy was
|
||||
untouched. Fixed and proven on 3 August: the machine decides once, per app, **before it writes
|
||||
anything**, and that one answer covers all three steps, so a refused app writes nothing, is not
|
||||
restarted, and the promise is now literally true. It also stopped being blind to size. Nothing is ever
|
||||
deleted to make room. *(R-181)*
|
||||
|
||||
**The last of the three apps that never saved their data is fixed.** Installed nowhere, so nothing was
|
||||
stranded — checked on both demo machines and in the fleet list rather than assumed. Proven by the check
|
||||
that caught it, run in both directions: it clears the fixed version and still convicts the old one.
|
||||
*(R-156)*
|
||||
|
||||
**A filling disk warns the customer before anything breaks, and a failed backup reaches you** — the
|
||||
customer while there is still room to act, naming the drive and the space left; you when one app's
|
||||
backup fails, with the disk figures. The customer is deliberately not told about the second: they can
|
||||
free space, but they can do nothing about a failed backup. Both proven by filling a real disk. There
|
||||
are two rules and not one because the serious warning fired on free space while the disk was only 91%
|
||||
full — a percentage alone would have missed it. *(R-167, R-158)*
|
||||
|
||||
**The checks have two nets and the second emails you.** Every repository has one command that runs all
|
||||
its checks, before every push. That one can be skipped, so the build server runs them again and emails
|
||||
you on failure. It cannot *stop* a change — everything goes straight to the main copy with no review
|
||||
step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** nothing outstanding from today — the reserve, the last unsaved app, and both of your
|
||||
decisions are all built and proven.
|
||||
- **Next:** the alert that tells you about one app and swallows the second *(R-182)*.
|
||||
- **After:** the off-site copy that the machine making it can still erase *(R-95, R-87)*.
|
||||
- **Now:** nothing outstanding.
|
||||
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
||||
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
||||
*(R-87)*
|
||||
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **Nothing else.** Both decisions you took on 3 August are now built and proven. One small question
|
||||
will come back later: the automatic check cannot see which version you have told machines to
|
||||
install, only which ones exist — closing that either needs a password given to the build server or
|
||||
a check inside the hub itself. Filed, not urgent. *(R-184)*
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
machines to install, only which ones exist. Closing that needs either a password given to the build
|
||||
server or a check inside the hub itself. *(R-184)*
|
||||
- **Nothing else.**
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-03** — Publishing became something you do rather than something that happens: the
|
||||
installer and the agent both moved onto labelled versions with a way back, and a check now refuses
|
||||
a release nobody can download. Earlier the same day: the reserve now guards the step that fills the
|
||||
disk and its promise is true, and the last app whose data was never saved is fixed. All proven on
|
||||
real machines, not just in tests.
|
||||
Earlier the same day: both demo machines wiped and rebuilt from the new base image and taken through
|
||||
set-up → install an app → back it up → restore it, with the backup space ceiling gone and measured.
|
||||
|
||||
- **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in
|
||||
a mode where reading a page cannot block a machine's status update; a one-word difference meant that
|
||||
setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found:
|
||||
the hub's own database is in no automatic backup** — it holds every machine's emergency password.
|
||||
Filed, not yet fixed.
|
||||
|
||||
- **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were
|
||||
failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this
|
||||
machine and the tester's box are protected, every other box may be broken or reinstalled freely.
|
||||
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
||||
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
||||
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
||||
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
||||
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
||||
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -132,6 +132,41 @@ executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by const
|
||||
(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
|
||||
(INV Part D1).
|
||||
|
||||
### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03)
|
||||
|
||||
**[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to
|
||||
fire on an interval started at daemon start. It no longer does. The rule is:
|
||||
|
||||
> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
|
||||
> The tier is **DUE** when **A** exists and **A has not already been proven**.
|
||||
|
||||
So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather
|
||||
than the process's uptime:
|
||||
|
||||
| tier rhythm | what is proved, and when |
|
||||
|---|---|
|
||||
| daily (host tier) | yesterday's archive, once a day |
|
||||
| weekly (offsite tier) | last week's archive, once a week |
|
||||
| newborn (no archive yet) | nothing — **UNKNOWN, never a fault** |
|
||||
|
||||
**The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest
|
||||
archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive
|
||||
age to zero long before it reaches the lag — so the literal reading silently switches restore-testing
|
||||
off for the tier that matters most.
|
||||
|
||||
What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its
|
||||
journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on
|
||||
success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker
|
||||
remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one
|
||||
due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN).
|
||||
|
||||
**The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very
|
||||
cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval
|
||||
just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay
|
||||
tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed
|
||||
archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite
|
||||
retention), and falling back to the tier's declared rhythm when history is too short to observe one.
|
||||
|
||||
### Why the split is right, stated once
|
||||
|
||||
**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
|
||||
@@ -574,6 +609,30 @@ because that was a different filesystem. On a merged box it can. Decision **B2**
|
||||
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
|
||||
only destroy a different app's only local copy.
|
||||
|
||||
**WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
|
||||
The two are deliberately different mechanisms, because conflating them is how seven failures went
|
||||
missing on 2026-08-03 without leaving a trace.
|
||||
|
||||
| | Record | Notification |
|
||||
|---|---|---|
|
||||
| what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
|
||||
| when | every time, unconditionally | at the end of a run, **only if something failed** |
|
||||
| gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
|
||||
| where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
|
||||
|
||||
- **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
|
||||
is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
|
||||
REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
|
||||
design; weakening it re-opens a silent-failure path.
|
||||
- **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
|
||||
Deciding not to tell someone is itself an event worth recording.
|
||||
- **Deliberate skips are not failures** and never appear in the digest — a disconnected or
|
||||
decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
|
||||
operator stops reading.
|
||||
- **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
|
||||
the hour, because someone pressing the button is actively trying to get a backup. The periodic
|
||||
capture sweep is capped by the ordinary hourly cooldown.
|
||||
|
||||
**THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
|
||||
**per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
|
||||
before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
|
||||
@@ -796,7 +855,7 @@ does **not** hold as written. → **R-108**
|
||||
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
|
||||
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
|
||||
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
|
||||
| R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written |
|
||||
| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
|
||||
| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
|
||||
|
||||
### 10.3 Divergences that are documented elsewhere and are not re-opened here
|
||||
@@ -862,7 +921,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
|
||||
| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
|
||||
| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
|
||||
| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
|
||||
| unattended restore-test across tiers | **IMPLEMENTED** (rotation not observed across consecutive cadences) | `00-capability-map.md:41`; LIVE per-tier timestamps this session |
|
||||
| unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) |
|
||||
| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
|
||||
| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
|
||||
| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
**Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only);
|
||||
felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`,
|
||||
`internal/hub/cloudflared.go`, `internal/escrow/identity.go`).
|
||||
**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`,
|
||||
**Probe ends:** `felhom-hetzner` = Hetzner CX23 (**note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged**), Debian 13.4, public IPv4 `167.233.158.164`,
|
||||
global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** =
|
||||
the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven
|
||||
over the existing LAN SSH path; the tunnel itself always dialed **out**.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -144,7 +144,7 @@
|
||||
| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
|
||||
| R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. |
|
||||
| R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
|
||||
| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
|
||||
| R-86 | ~~**Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup**~~ | S | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03 (agent v0.121.0 + hub v0.91.0)** | **The reasoning that produced the row still holds and is worth keeping:** R-85 scheduled on a free-running INTERVAL with oldest-proven-first rotation, which cannot express *"the day after the PBS backup"* — any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. The operator's ruling was weekly per tier, ~1 day after that tier produced a new backup, so the gap between an archive being written and being proven stays small and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **What shipped is that ruling, with one correction the row's own wording would have got wrong:** the shape written here — *"test tier X once its newest archive is ~24 h old and not yet proven"* — is NEVER true on a DAILY tier, because a new archive resets the newest-archive age to zero before it reaches 24 h. The rule that works is *the newest archive that HAS settled 24 h, and has not been proven*: `A = newest archive older than the settle lag`, due iff A exists and A ≠ the proven archive. **The interim is retired:** `restore_test_cadence_seconds = 302400` (3.5 d) is no longer the mechanism; that key now only disables (negative) or seeds the settle lag. Alignment is no longer approximated by a cadence — it is the schedule. Full detail: `OPEN-ITEMS.md` R-86, `07-backup-architecture.md` §3 |
|
||||
| R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
|
||||
| R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | **Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change)** — **Part 1** added the failure breaker: consecutive failures tracked per TARGET, backoff `15m→30m→1h→2h→4h` capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to `TriggerNow`, and `stillRunning` is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. **Part 2 is agent-side and needs a wire change** — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely.
|
||||
|
||||
@@ -175,7 +175,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
|
||||
| R-162 | **`docker diff` is the gate's only witness and its failure mode is quiet** | XS | WATCHING — 2026-08-02 | A limitation, not a defect. The gate's power is `docker diff` excluding mounted paths; on a driver where it is unsupported or lies, the gate degrades to mount-occupancy + writability **and would not say so**. It fails closed (the canary self-test stops reporting BROKEN and the gate then refuses to report), but the message blames the prober rather than the driver. Revisit only if a non-overlay driver ships |
|
||||
| R-163 | **`mp1` is RETENTION, not staging — and it is sized as if it were neither** | M | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | Settled by reading: a unit is the KEPT copy on the app's **own** drive (`backup.go:245-255`), falling back to the system SSD for driveless apps — *"the SSD-only system-data fallback"* (`appbackup/paths.go:26-27`); **no post-copy deletion**, the only prune is F5 residue-on-old-drives (`backup.go:1053-1112`). So `mp1` **20 G** retains every driveless app's unit while `mp0` permits **50 G** of volumes, and a DB app's unit is up to **~2×** its data (measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to 0 (`felhom-agent/.../main.go:178`), not derived from the drive; demo-hp's real guest ships 50G/20G. **`mp1` gates the whole chain** — Tier-2 mirrors the unit *"(always)"* (`tier2.go:302,368`) and Tier-3 carries it. Bounded: a unit is volume tars + DB dumps only, **never `mp8` userdata** (`recovery_unit.go:20-25`). **No number proposed.** → `07-backup-architecture.md` §7.5 |
|
||||
| R-164 | **C2's chain — the DB volume tar cannot be dropped until a sound dump predicate exists** | S | BLOCKED — on the predicate (2026-08-02) | The unit holds a volume tar **and** a SQL dump and the restore uses both: the dump is authoritative and replayed after the tar so it WINS (F17), with only the DB service up (R-47) — `restore_unit.go:262-266`. Dropping the DB tar would halve DB-app units and close R-127(b)'s initdb-skip trap. **The obvious gate is dead, measured:** `ValidateDump`'s empty-`accounts` warning was **correct** (the DB truly had 0 rows; seeding one stopped the warning and put the row in the dump) — but **a fresh appliance legitimately has zero accounts**, so gating on it blocks every new customer's first backup. Order: sound predicate (dump vs **live** per-table counts) → warn→gate → tar-drop. Pairs with **R-127** |
|
||||
| R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |
|
||||
| R-90 | ~~**ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86**~~ | S | **CLOSED 2026-08-03 — the operator rescaled ep0 to a CX33.** MEASURED on the box, not read from an invoice: `Mem: 7757` MB (**8 GB**, was 3.8), `nproc` **4**, and the 4 GiB swapfile added 2026-07-27 **survived the resize** and is active (checked because a resize is a stop/start, so "still there" was an assumption until measured). **The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only — so no disk figure in any runbook needed correcting. Unblocks R-86 | Runbooks corrected; SPIKE/evidence documents ANNOTATED not revised. `target-selection.md`'s "D-d did not name ep0 either way" stays open — it is the operator's |
|
||||
| R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` |
|
||||
| R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect |
|
||||
| R-93 | **`drill-r50` is both a blocked customer and the only drift fixture** | XS | idea — 2026-07-27 | The drill customer is blocked in the hub (so it stops alarming) yet it is also the only record exercising the endpoint-drift path R-77 added. Blocking hides it from `GetActiveCustomerIDs`, so the fixture it provides is silently inert — a monitor with no live subject reads exactly like a monitor that passes. Decide: retire it and build a synthetic fixture, or unblock it and silence per-customer instead (the operator has a per-alert silencing feature planned). Related to the R-50 drill VM, now shut down |
|
||||
|
||||
@@ -3,7 +3,8 @@
|
||||
**Class:** supervised operational run. **No repo version bump** — the only commits are this record
|
||||
and the capacity note. **Nothing was deleted.**
|
||||
|
||||
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg.
|
||||
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner **CX33 (4 vCPU / 8 GB RAM)**, Nuremberg.
|
||||
> **Rescaled 2026-08-03** from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. **The 40 GB local disk did NOT change** — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize.
|
||||
**Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom` → **`/mnt/pbs-datastore`** (name unchanged).
|
||||
**Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.
|
||||
|
||||
|
||||
@@ -229,7 +229,9 @@ as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as
|
||||
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
|
||||
minutes of ep0 CPU per weekly backup.
|
||||
|
||||
> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> Watch item (**superseded 2026-08-03**: ep0 is now a **CX33, 8 GB RAM**, and it HAS a 4 GiB swapfile
|
||||
> which survived the resize — so the pressure below is much reduced, though the shape of the concern
|
||||
> stands). As written: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
|
||||
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
|
||||
|
||||
@@ -307,7 +309,7 @@ Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13
|
||||
watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
|
||||
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
|
||||
it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
|
||||
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
|
||||
5. ~~**ep0 has no swap** (3.7 GB CX23)~~ — **corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile.** See the volume runbook's OOM for the original incident.
|
||||
|
||||
## 11. Observations
|
||||
|
||||
|
||||
@@ -431,7 +431,7 @@ label. Filed under E-2.
|
||||
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
|
||||
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
|
||||
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
|
||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. |
|
||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
|
||||
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
|
||||
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
|
||||
| 8 | The honest **single-drive label**. |
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
> firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM
|
||||
> re-creates the endpoint from nothing (that is the DR story, step 8).
|
||||
>
|
||||
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23, Debian 13,
|
||||
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled
|
||||
> to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13,
|
||||
> `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a
|
||||
> later re-run of this runbook on a production VM.
|
||||
>
|
||||
@@ -31,7 +32,7 @@ Parameters used throughout (adjust for a new endpoint):
|
||||
points at nothing (live-run finding). Home-resolver propagation can lag public DNS by
|
||||
minutes — a client-side `wg-quick up` that fails to resolve right after record creation
|
||||
just needs a retry.
|
||||
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway CX23), not
|
||||
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not
|
||||
any production box.
|
||||
|
||||
## 1. Base (on the box, as root)
|
||||
|
||||
@@ -29,20 +29,23 @@ prohibition as covering the act it names and nothing more.
|
||||
|---|---|---|
|
||||
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
|
||||
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
|
||||
|
||||
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
|
||||
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
|
||||
thing under test, the source of truth for it, and the backups, at once.
|
||||
|
||||
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
|
||||
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
|
||||
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
|
||||
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
|
||||
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
|
||||
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
|
||||
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
|
||||
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
|
||||
**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
|
||||
not name ep0 either way, so this page carried the question in writing for two days and read it the
|
||||
narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
|
||||
protected list to three machines**: DooPlex, Peti's cluster, ep0.
|
||||
|
||||
The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
|
||||
real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
|
||||
prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
|
||||
ruling makes the classification say so plainly instead of leaving each session to re-derive it.
|
||||
**Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
|
||||
never a drill target. The Hetzner Storage Boxes ride the same reasoning.
|
||||
|
||||
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
|
||||
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
|
||||
@@ -95,11 +98,15 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
|
||||
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
|
||||
enrolled host. No access route from DooPlex, and nothing here needs one.
|
||||
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
|
||||
|
||||
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
||||
path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
|
||||
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
|
||||
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
|
||||
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
|
||||
is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
|
||||
3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
|
||||
**The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
|
||||
(R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
|
||||
customer documents and photos, on a credential that can still delete (R-95).
|
||||
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
|
||||
produced a false "unreachable" verdict (standing rule 2).
|
||||
|
||||
@@ -139,7 +139,7 @@ is not reachable, for two reasons that are each **already-recorded deliberate po
|
||||
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
|
||||
|
||||
So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
|
||||
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off* →
|
||||
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90 — **note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off* →
|
||||
no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
|
||||
|
||||
Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the
|
||||
|
||||
@@ -1,3 +1,132 @@
|
||||
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
|
||||
|
||||
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
|
||||
offsite tier holds two retained snapshots — `2026-07-27T19:55:41Z` and `2026-07-28T04:49:43Z` —
|
||||
**8 h 54 m apart**, because one is a healing artefact and the other a real weekly run. The mean-gap
|
||||
estimator therefore reads a **weekly** tier as nine-hourly: ×4 gives 36 h, the 7-day floor lifts it to
|
||||
168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm this whole
|
||||
task exists to prevent would have returned within a week, on the box it had just shipped to.**
|
||||
|
||||
`restoreProvenWindow` now takes `max(observed, declared)`. Observation refines a tier's rhythm
|
||||
**upward** and is ignored downward, which is right on its own terms and not merely cautious: a gap
|
||||
SHORTER than the declared rhythm is routine and means nothing — a retry, a manual run, a heal, a
|
||||
catch-up after an outage — while a gap LONGER than it is real information, saying this tier genuinely
|
||||
receives archives less often than the model assumes and its window must widen or it alarms.
|
||||
|
||||
**The cost, stated rather than hidden:** a tier that truly runs faster than its declared rhythm gets a
|
||||
wider window than it strictly needs, i.e. a slower `restore_test_stale` signal. That is the right
|
||||
direction for a signal whose message is *"unverified"*. *"Broken now"* is `restore_test_failed`, which
|
||||
is immediate and untouched.
|
||||
|
||||
Three live-derived cases added to `TestRestoreProvenWindow_Contract`, including the exact 8 h 54 m
|
||||
gap measured on the box; red-proved by restoring the tighten-too branch
|
||||
(`window(pbs, observed=8h54m) = 168h, want 288h`).
|
||||
|
||||
## v0.91.0 — a tier's staleness window learns the tier's own rhythm (2026-08-03, R-86 Part 2)
|
||||
|
||||
**This ships WITH the agent's v0.121.0, not after it.** The agent now proves a tier once per ARCHIVE
|
||||
GENERATION rather than on a 24h timer, so a tier backed up weekly is proved weekly — correctly, and
|
||||
in perfect health. `restoreProvenStaleAfter` was a flat 7 days, and its own comment derived that
|
||||
number from the cadence R-86 removes:
|
||||
|
||||
> *"the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
|
||||
> proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."*
|
||||
|
||||
A weekly tier proved weekly reaches a proof age of **exactly one interval** just before its next
|
||||
proof — 168h against a 168h window, false by a hair — so it did not merely sit near that line, it sat
|
||||
ON it, and any ordinary delay (a late backup, one deferral behind a running backup) tipped it into a
|
||||
nightly alarm about a system that is working. Shipping the agent alone would have converted the
|
||||
improvement into a false alarm.
|
||||
|
||||
**The window is now per tier**, `restoreProvenWindow(tier, observed, ok)`:
|
||||
|
||||
- **the tier's own archive interval**, observed from the host-reports the hub already holds —
|
||||
`pbs_snapshots[]` plus successful `backups[]` attributed by TARGET TYPE (the slice-A.4 rule: a
|
||||
PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an
|
||||
offsite archive to the host tier);
|
||||
- **× 4 generations** — the settle generation plus ~3 missed opportunities, deliberately the same
|
||||
tolerance the flat constant expressed. The change is to the RHYTHM, not to the patience;
|
||||
- **floored at 7 days** — the old constant kept as a floor, so no tier is judged more tightly than
|
||||
before;
|
||||
- **capped at 12 days** — strictly inside the 2-week offsite retention with two days to spare, so a
|
||||
tier is never called stale against an archive PBS has already pruned;
|
||||
- **falling back to the DECLARED rhythm** when a box's history is too short to observe one:
|
||||
`backupStaleAfter` (26h) for the host tier and `offsiteBackupStaleAfter` (8d) for the offsite tier
|
||||
— the very thresholds the backup-freshness checker already judges those tiers against. A fresh box
|
||||
with one snapshot has no observable interval, and falling back to the FLOOR there would recreate
|
||||
the false alarm on exactly the tier this task is about.
|
||||
|
||||
**Kept, because both were earned:** absence is UNKNOWN until an anchored window has passed (R-81's
|
||||
structure, untouched), and the stale signal stays edge-triggered. `restore_test_failed` and
|
||||
`restore_test_stale` remain DISTINCT — one says your recovery is broken, the other says it is
|
||||
unverified, and the second is the one that quietly becomes the first.
|
||||
|
||||
**Every reason string now states the window it was judged against** — R-100's corollary: when a
|
||||
verdict changes what it counts from, the alarm text has to change with it, or an operator reads
|
||||
"limit 168h" under a tier actually judged at 288h and dismisses a true alarm.
|
||||
|
||||
**The window READ is unchanged in cost** (14 days), which is both enough to find proof inside the
|
||||
widest window and enough to see two generations of a weekly tier.
|
||||
|
||||
## v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (2026-08-03, R-182)
|
||||
|
||||
**Backfilled 2026-08-03 (R-86 session).** This version was built, deployed and recorded in `REPORT.md`
|
||||
and the R-182 row, but never given a CHANGELOG entry — and `REPORT.md` is overwritten every session,
|
||||
so the per-repo history under-reported what was actually running. The deployed image has been
|
||||
`felhom-hub:0.90.1` since `f21e7ca`.
|
||||
|
||||
Found by reading the first REAL digest, not by design: every app row ended with the same usage clause
|
||||
the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list
|
||||
of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone
|
||||
screen at 07:00 — the only moment that mail has to work.
|
||||
|
||||
The reserve's refusal message is authored for a single-app alert where naming the filesystem is
|
||||
right, so the message is unchanged and the DIGEST trims the duplicate when rendering.
|
||||
`trimRepeatedUsage` removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is
|
||||
untouched and a reason that is nothing but the usage clause is left alone rather than emptied.
|
||||
|
||||
Also inverted the operator half of `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer`: it
|
||||
required the operator to be e-mailed a per-app capture failure, which was correct when that event was
|
||||
the only signal and is wrong now that it is the RECORD and the digest is the notification. The
|
||||
customer-safety claim is unchanged — R-158's guarantee MOVED, it did not weaken.
|
||||
|
||||
## v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (2026-08-03, R-182)
|
||||
|
||||
**The smallest change on the board with the largest effect on trust: `processOperator`'s cooldown no
|
||||
longer returns bare.** It used to drop the event *before* `LogNotification`, so a suppressed operator
|
||||
alert and an event that never happened were indistinguishable — from the operator's side **and from
|
||||
the hub's own records**. Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two
|
||||
were mailed, and **seven left no row anywhere**. That is why the defect took a day to get the right
|
||||
way round: there was nothing to read.
|
||||
|
||||
A suppressed operator event now writes a `suppressed` row carrying the message and **the key that
|
||||
suppressed it**, so the collision is readable without reading code. This applies to **every** operator
|
||||
event, not only the one that exposed it. It deliberately does **not** change the cooldown's duration
|
||||
or semantics — it makes the drop visible, not absent.
|
||||
|
||||
**`backup_run_failures` — the per-run digest.** One operator mail at the end of a backup run listing
|
||||
every app that failed, its leg and its reason, with the counts and the target filesystem's free
|
||||
space. Added to `allowedEventTypes` **and** to `operatorOnlyEvents` — allowlisting alone does not make
|
||||
an event operator-only, and `FormatCustomerEmail` falls back to the raw English message rather than
|
||||
blocking. A test demonstrates a customer with the type in their enabled list receiving nothing.
|
||||
|
||||
**`recordOnlyEvents` — a third routing class.** Types that are STORED and RECORDED but never mailed.
|
||||
`recovery_unit_capture_failed` moves here: it is the durable per-failure record, and the digest is the
|
||||
notification. Deliberately a register rather than downgrading the severity to `info`, which would have
|
||||
had the same routing effect while relabelling a genuine failure as informational in the events table,
|
||||
the operator UI and every historical query.
|
||||
|
||||
**`cooldownRunSuffix` — the run discriminator.** A sibling of `cooldownTierSuffix` rather than a
|
||||
branch inside it, so `tier` keeps byte-identical semantics and R-97a's tests are untouched. It makes
|
||||
the cooldown effectively inert for the digest, **which is the intent**: a digest is already
|
||||
rate-limited by construction, one per run and only when something failed, so there is nothing for a
|
||||
timer to collapse — while the periodic refresh sweep sends **no** `run_id` and therefore stays under
|
||||
the ordinary hourly cooldown.
|
||||
|
||||
**The e-mail is rendered as a list, not a JSON blob** — the one operator mail with a variable-length
|
||||
payload, and a dozen apps on one line is unreadable on a phone at 07:00. An absent space reading
|
||||
renders as *unavailable*, never as zeros.
|
||||
|
||||
## v0.89.0 — the two halves of decision D-c (2026-08-02, R-167 · R-158)
|
||||
|
||||
**Decision D-c routes two new signals to two different audiences, and the hub is where that routing
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
package api
|
||||
|
||||
import "testing"
|
||||
|
||||
// R-182 — the backup run digest needs the allowlist half, and NOT the customerMessages half.
|
||||
//
|
||||
// A new event type is a pair of register entries, and each half fails differently. For this type the
|
||||
// pair is unusual and that is the point:
|
||||
//
|
||||
// - missing from `allowedEventTypes` → POST /event returns 400 and the digest VANISHES,
|
||||
// which would rebuild the exact silence R-182 exists to end;
|
||||
// - PRESENT in `customerMessages` → the customer would be e-mailed, in Hungarian, a
|
||||
// list of which apps' backups failed and why — operator detail they can take no action on. So
|
||||
// this type must be allowlisted and must NOT have a customer message.
|
||||
//
|
||||
// The customer-facing half of decision D-c is the FILL WARNING, which fires before this and is
|
||||
// actionable (free space, delete files, add a drive). This is the operator's half.
|
||||
//
|
||||
// Operator-only routing itself is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a
|
||||
// customerMessages entry — that assumption shipped in v0.78.0 and was wrong, because
|
||||
// FormatCustomerEmail falls back to the raw message. It is pinned in
|
||||
// `internal/notify/backup_run_digest_test.go`, which demonstrates a customer with the type in their
|
||||
// enabled list receiving nothing.
|
||||
func TestBackupRunDigestIsAllowlisted(t *testing.T) {
|
||||
if !allowedEventTypes["backup_run_failures"] {
|
||||
t.Fatal("backup_run_failures must be in allowedEventTypes, or POST /event 400s and the " +
|
||||
"whole run digest is dropped at the door — the silence R-182 was filed against")
|
||||
}
|
||||
}
|
||||
|
||||
// The per-app event is the RECORD and must not be removed while the digest is the notification.
|
||||
// Deleting it would make the digest the only trace, and a digest that fails to send would then take
|
||||
// the record with it — the coupling R-182's fix exists to break.
|
||||
func TestPerAppCaptureEventStaysAllowlisted(t *testing.T) {
|
||||
if !allowedEventTypes["recovery_unit_capture_failed"] {
|
||||
t.Fatal("recovery_unit_capture_failed was removed from allowedEventTypes — it is the " +
|
||||
"durable per-failure RECORD, and the digest is only the notification; the operator " +
|
||||
"register and every historical query depend on it")
|
||||
}
|
||||
}
|
||||
@@ -1586,6 +1586,11 @@ var allowedEventTypes = map[string]bool{
|
||||
// OPERATOR-ONLY IS ENFORCED BY `notify.operatorOnlyEvents` — see the paragraph above. This entry
|
||||
// alone does NOT make it operator-only.
|
||||
"recovery_unit_capture_failed": true,
|
||||
// R-182. The per-run backup digest: one event at the end of a run, listing every app whose
|
||||
// backup failed or was refused. Allowlisting it is NOT what keeps it away from customers —
|
||||
// `notify.operatorOnlyEvents` is (see the comment there); both entries ship together and
|
||||
// `backup_run_digest_event_test.go` pins the pair.
|
||||
"backup_run_failures": true,
|
||||
|
||||
// Controller-pushed events
|
||||
"controller_started": true,
|
||||
|
||||
@@ -32,15 +32,104 @@ import (
|
||||
// is a NEW monitor written straight after the third, so it copies R-81's verdict structure rather
|
||||
// than re-deriving it. A tier never proven on a newborn box is UNKNOWN, never FAILED.
|
||||
|
||||
// restoreProvenStaleAfter is how long a tier may go unproven before it is called stale.
|
||||
// ── HOW LONG MAY A TIER GO UNPROVEN? (R-86 Part 2) ───────────────────────────────────────────
|
||||
//
|
||||
// Derivation, not a guess: the restore-test cadence is 24h and rotation is oldest-first across two
|
||||
// tiers, so each tier is proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive
|
||||
// missed opportunities before alarming — loud enough to matter, quiet enough not to fire on one
|
||||
// skipped cycle (a deferral behind a long backup is normal, not a fault). It is also comfortably
|
||||
// inside the 2-week offsite retention (operator ruling 2026-07-26), so a tier is never reported
|
||||
// stale against an archive that is about to be pruned anyway.
|
||||
const restoreProvenStaleAfter = 7 * 24 * time.Hour
|
||||
// This was one flat constant, 7 days, and its comment derived that number like this:
|
||||
//
|
||||
// "the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
|
||||
// proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."
|
||||
//
|
||||
// **That premise is exactly what R-86 removed.** The agent no longer tests on an interval at all: a
|
||||
// tier is tested once per ARCHIVE GENERATION — when it holds a settled archive that has not been
|
||||
// proven. A tier backed up weekly is therefore proved weekly, by design and in perfect health, and
|
||||
// against a flat 7-day window it would sit on the line and alarm every night about a system that is
|
||||
// working. Shipping the agent's half alone would have converted the improvement into a false alarm,
|
||||
// which is why the two ship together.
|
||||
//
|
||||
// The window is now derived from **the tier's own backup rhythm**, which the hub can observe from
|
||||
// the reports it already receives, and it keeps everything the constant had earned:
|
||||
//
|
||||
// - absence is UNKNOWN until an anchored window has passed (R-81's structure, untouched);
|
||||
// - the signal stays edge-triggered;
|
||||
// - it never exceeds the offsite retention, so a tier is never called stale against an archive
|
||||
// that is about to be pruned;
|
||||
// - and it is never TIGHTER than the 7 days that were already tolerated.
|
||||
const (
|
||||
// restoreProvenGenerations is how many archive generations may pass unproven before alarming.
|
||||
// 4 = the settle lag's own generation plus ~3 missed opportunities — deliberately the same
|
||||
// tolerance the flat constant expressed, so the change is to the RHYTHM, not to the patience.
|
||||
restoreProvenGenerations = 4
|
||||
|
||||
// restoreProvenWindowFloor is the shortest window that may be applied to any tier. It is the
|
||||
// old constant, kept as a FLOOR rather than deleted: a daily tier computes 4 days from its own
|
||||
// rhythm, and tightening a live threshold is not what this task is for. A deferral behind a
|
||||
// long backup is normal, not a fault.
|
||||
restoreProvenWindowFloor = 7 * 24 * time.Hour
|
||||
|
||||
// restoreProvenWindowCap keeps the window strictly inside the 2-week offsite retention
|
||||
// (operator ruling 2026-07-26) with two days to spare. Beyond it the hub would be judging a
|
||||
// tier against an archive PBS has already pruned — an alarm nobody can act on, and the bound
|
||||
// the old constant respected in its own way.
|
||||
restoreProvenWindowCap = 12 * 24 * time.Hour
|
||||
|
||||
// restoreWindowRead is how far back the hub reads host-reports for this check: far enough to
|
||||
// find proof anywhere inside the widest window, and to see at least two archive generations of
|
||||
// a WEEKLY tier so its rhythm is observable at all.
|
||||
restoreWindowRead = 2 * restoreProvenWindowFloor
|
||||
)
|
||||
|
||||
// declaredArchiveInterval is the rhythm the hub ALREADY attributes to a tier — the same thresholds
|
||||
// the backup-freshness checker judges it against (deadline.go / deadline_tiers.go). It is the
|
||||
// fallback when a box's history is too short to observe a rhythm, and it is the right fallback
|
||||
// precisely because it is not a second opinion: if these two checkers disagreed about how often a
|
||||
// tier is expected to receive an archive, one of them would be alarming on the other's model.
|
||||
//
|
||||
// It is stated per RESTORE tier name ("local"/"pbs" — what the agent reports as source_tier), which
|
||||
// is the same split the backup tiers use under different names ("host"/"offsite").
|
||||
func declaredArchiveInterval(tier string) time.Duration {
|
||||
if tier == "pbs" {
|
||||
return offsiteBackupStaleAfter // 8 days: the weekly cadence plus a day of headroom
|
||||
}
|
||||
return backupStaleAfter // 26 hours: the daily cadence plus headroom
|
||||
}
|
||||
|
||||
// restoreProvenWindow is how long THIS tier may go unproven, given its observed archive interval.
|
||||
//
|
||||
// observedOK=false means the box's retained history did not contain two archive generations for
|
||||
// this tier, so the declared rhythm is used. That fallback matters most for exactly the tier this
|
||||
// task is about: a fresh box with a weekly offsite tier has one snapshot and no observable
|
||||
// interval, and falling back to the FLOOR there would recreate the false alarm.
|
||||
//
|
||||
// OBSERVATION MAY ONLY WIDEN, NEVER TIGHTEN — and this is not caution, it is a live measurement.
|
||||
// On demo-felhom (2026-08-03) the offsite tier's two retained snapshots are `2026-07-27T19:55:41Z`
|
||||
// and `2026-07-28T04:49:43Z`: **8 h 54 m apart**, because one is a healing artefact and the other a
|
||||
// real weekly run. A mean-gap estimate therefore reads a WEEKLY tier as nine-hourly, ×4 gives 36 h,
|
||||
// the floor lifts it to 7 days — and a weekly tier proved weekly reaches ~8.25 days of proof age, so
|
||||
// the false alarm this whole task exists to prevent would have returned within a week, on the very
|
||||
// box it shipped to.
|
||||
//
|
||||
// The asymmetry is right on its own terms too. A gap SHORTER than the declared rhythm is routine and
|
||||
// means nothing — a retry, a manual run, a heal, a catch-up after an outage. A gap LONGER than the
|
||||
// declared rhythm is real information: this tier genuinely receives archives less often than the
|
||||
// model says, and its window must widen or it alarms. So observation refines the rhythm upward and
|
||||
// is ignored downward. The cost is stated plainly: a tier that truly runs FASTER than its declared
|
||||
// rhythm gets a wider window than it strictly needs, i.e. a slower stale signal. That is the right
|
||||
// direction for a signal whose message is "unverified" — "broken NOW" is `restore_test_failed`, and
|
||||
// that one is immediate and unaffected.
|
||||
func restoreProvenWindow(tier string, observed time.Duration, observedOK bool) time.Duration {
|
||||
interval := declaredArchiveInterval(tier)
|
||||
if observedOK && observed > interval {
|
||||
interval = observed
|
||||
}
|
||||
w := time.Duration(restoreProvenGenerations) * interval
|
||||
if w < restoreProvenWindowFloor {
|
||||
w = restoreProvenWindowFloor
|
||||
}
|
||||
if w > restoreProvenWindowCap {
|
||||
w = restoreProvenWindowCap
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// Event types. Operator-tier only — see the dispatcher note in RestoreTestChecker.
|
||||
const (
|
||||
@@ -156,12 +245,13 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
return
|
||||
}
|
||||
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-2*restoreProvenStaleAfter))
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-restoreWindowRead))
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] restore-test check: window read failed for %s: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
proven := lastProvenPerTier(rows)
|
||||
intervals := observedArchiveIntervals(rows)
|
||||
|
||||
first, ferr := c.store.GetFirstHostReportAt(customerID)
|
||||
if ferr != nil {
|
||||
@@ -170,7 +260,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
|
||||
for _, tier := range tiers {
|
||||
v := assessRestoreProven(tier, proven[tier], first, now)
|
||||
observed, observedOK := intervals[tier]
|
||||
window := restoreProvenWindow(tier, observed, observedOK)
|
||||
v := assessRestoreProven(tier, proven[tier], first, now, window)
|
||||
key := customerID + "|" + tier
|
||||
c.mu.Lock()
|
||||
prev := c.staleStates[key]
|
||||
@@ -192,37 +284,120 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
}
|
||||
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now injected) so the policy is unit-tested —
|
||||
// the property that made R-81 provable, kept deliberately.
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
|
||||
// unit-tested — the property that made R-81 provable, kept deliberately.
|
||||
//
|
||||
// no proof, anchor NOT elapsed → UNKNOWN (newborn box; never an alarm)
|
||||
// no proof, anchor elapsed → MISSED
|
||||
// proof older than the window → MISSED
|
||||
// otherwise → OK
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time) backupAssessment {
|
||||
//
|
||||
// `window` is now the TIER'S OWN (R-86 Part 2) rather than one constant for every tier, and every
|
||||
// reason string states the window it was judged against. That is R-100's corollary applied here:
|
||||
// when a verdict changes what it counts from, the alarm text has to change with it, or an operator
|
||||
// reads "limit 168h" under a tier that was actually judged at 288h and dismisses a true alarm.
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time, window time.Duration) backupAssessment {
|
||||
if provenAt.IsZero() {
|
||||
if firstReportAt.IsZero() {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: never restore-proven, and no first-contact anchor to defer against", tier)}
|
||||
}
|
||||
watched := now.Sub(firstReportAt)
|
||||
if watched <= restoreProvenStaleAfter {
|
||||
if watched <= window {
|
||||
return backupAssessment{verdict: verdictUnknown,
|
||||
reason: fmt.Sprintf("%s tier: not restore-proven yet, but only watching for %s (grace %s since first contact %s) — newborn, not a fault",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter, firstReportAt.Format(time.RFC3339))}
|
||||
tier, watched.Round(time.Hour), window, firstReportAt.Format(time.RFC3339))}
|
||||
}
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), window)}
|
||||
}
|
||||
if age := now.Sub(provenAt); age > restoreProvenStaleAfter {
|
||||
if age := now.Sub(provenAt); age > window {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), window)}
|
||||
}
|
||||
return backupAssessment{verdict: verdictOK}
|
||||
}
|
||||
|
||||
// observedArchiveIntervals estimates how often each RESTORE tier actually receives an archive, from
|
||||
// the host-reports the hub already holds. Keyed by restore-tier name ("local"/"pbs").
|
||||
//
|
||||
// Evidence is every distinct archive timestamp in the window: `pbs_snapshots[]` for the offsite
|
||||
// tier (PBS enumerates its whole retention in each report, so one report usually settles the
|
||||
// question) and successful `backups[]` records attributed by TARGET TYPE for both tiers — the
|
||||
// slice-A.4 rule, because a PBS-targeted vzdump appears in BOTH arrays and classifying by array
|
||||
// membership would attribute an offsite archive to the host tier.
|
||||
//
|
||||
// The estimate is the MEAN gap: (newest − oldest) / (generations − 1). It needs two generations;
|
||||
// with fewer, ok=false and the caller falls back to the declared rhythm. It is deliberately crude,
|
||||
// and can afford to be: restoreProvenWindow clamps the result between a 7-day floor and a 12-day
|
||||
// cap, so the only discrimination this has to get right is "roughly daily" versus "several days or
|
||||
// slower" — which is exactly the distinction that turns a healthy weekly tier into a false alarm.
|
||||
func observedArchiveIntervals(rows []store.HostReportRow) map[string]time.Duration {
|
||||
seen := map[string]map[int64]struct{}{ // tier → set of archive unix times
|
||||
"local": {},
|
||||
"pbs": {},
|
||||
}
|
||||
add := func(tier string, t time.Time) {
|
||||
if t.IsZero() {
|
||||
return
|
||||
}
|
||||
seen[tier][t.UTC().Unix()] = struct{}{}
|
||||
}
|
||||
|
||||
for _, r := range rows {
|
||||
var hr hostReportBackups
|
||||
if json.Unmarshal([]byte(r.ReportJSON), &hr) != nil {
|
||||
continue // one malformed retained report must not blind the scan
|
||||
}
|
||||
pbs := pbsTargetSet(hr)
|
||||
for _, ps := range hr.PBSSnapshots {
|
||||
if t, ok := parseBackupTime(ps.BackupTime); ok {
|
||||
add("pbs", t)
|
||||
}
|
||||
}
|
||||
for _, b := range hr.Backups {
|
||||
if !b.Success {
|
||||
continue
|
||||
}
|
||||
t, ok := parseBackupTime(b.StartedAt)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if pbs[b.TargetID] {
|
||||
add("pbs", t)
|
||||
} else {
|
||||
add("local", t)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
out := map[string]time.Duration{}
|
||||
for tier, set := range seen {
|
||||
if len(set) < 2 {
|
||||
continue // not observable — the caller uses the declared rhythm
|
||||
}
|
||||
var oldest, newest int64
|
||||
first := true
|
||||
for ts := range set {
|
||||
if first || ts < oldest {
|
||||
oldest = ts
|
||||
}
|
||||
if first || ts > newest {
|
||||
newest = ts
|
||||
}
|
||||
first = false
|
||||
}
|
||||
span := time.Duration(newest-oldest) * time.Second
|
||||
if span <= 0 {
|
||||
continue
|
||||
}
|
||||
out[tier] = span / time.Duration(len(set)-1)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// expectedRestoreTiers names the tiers this box actually HAS, so a box without an offsite tier is
|
||||
// never reported stale for one. Same gate as Slice C's `expected`, and for the same reason: without
|
||||
// it every box lacking a tier would alarm once the anchor elapsed — absence-is-not-failure,
|
||||
|
||||
@@ -182,26 +182,33 @@ func TestRestoreTest_NewbornDoesNotAlarm(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The boundary, pinned by name so a refactor has to delete an obviously-named contract.
|
||||
// The boundary, pinned by name so a refactor has to delete an obviously-named contract. R-86 made
|
||||
// the limit per-tier, so the anchor is now measured against THE TIER'S OWN window — here the local
|
||||
// tier's, which clamps to the 7-day floor and so keeps this contract numerically identical to the
|
||||
// one the flat constant expressed.
|
||||
func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
window := restoreProvenWindow("local", 24*time.Hour, true)
|
||||
if window != restoreProvenWindowFloor {
|
||||
t.Fatalf("precondition: a daily local tier must clamp to the floor; got %s", window)
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
watched time.Duration
|
||||
wantMissed bool
|
||||
}{
|
||||
{"newborn, 1h", time.Hour, false},
|
||||
{"just inside", restoreProvenStaleAfter - time.Minute, false},
|
||||
{"exactly at the limit", restoreProvenStaleAfter, false},
|
||||
{"just outside", restoreProvenStaleAfter + time.Minute, true},
|
||||
{"just inside", window - time.Minute, false},
|
||||
{"exactly at the limit", window, false},
|
||||
{"just outside", window + time.Minute, true},
|
||||
{"long past", 30 * 24 * time.Hour, true},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := assessRestoreProven("pbs", time.Time{}, now.Add(-c.watched), now)
|
||||
got := assessRestoreProven("local", time.Time{}, now.Add(-c.watched), now, window)
|
||||
if got.missed() != c.wantMissed {
|
||||
t.Fatalf("CONTRACT VIOLATED: unproven for %s (limit %s) → missed=%v, want %v (reason %q)",
|
||||
c.watched, restoreProvenStaleAfter, got.missed(), c.wantMissed, got.reason)
|
||||
c.watched, window, got.missed(), c.wantMissed, got.reason)
|
||||
}
|
||||
if !c.wantMissed && got.verdict != verdictUnknown {
|
||||
t.Fatalf("a deferred tier must be UNKNOWN (visible), not OK; got verdict=%d", got.verdict)
|
||||
@@ -215,15 +222,17 @@ func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.
|
||||
func TestRestoreTest_StaleIsSeparateFromFailure(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
stale := assessRestoreProven("pbs", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now)
|
||||
// A DAILY tier judged on its own rhythm: the window clamps to the 7-day floor.
|
||||
daily := restoreProvenWindow("local", 24*time.Hour, true)
|
||||
stale := assessRestoreProven("local", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
|
||||
if !stale.missed() {
|
||||
t.Fatalf("a tier last proven 9 days ago (limit %s) must be stale; got %q", restoreProvenStaleAfter, stale.reason)
|
||||
t.Fatalf("a daily tier last proven 9 days ago (limit %s) must be stale; got %q", daily, stale.reason)
|
||||
}
|
||||
// The wording must not read as "broken" — that is the other signal.
|
||||
if !strings.Contains(stale.reason, "unverified, not known-broken") {
|
||||
t.Fatalf("staleness must say UNVERIFIED, not broken — merging the two is the thing this avoids; got %q", stale.reason)
|
||||
}
|
||||
fresh := assessRestoreProven("pbs", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now)
|
||||
fresh := assessRestoreProven("local", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
|
||||
if fresh.verdict != verdictOK {
|
||||
t.Fatalf("a tier proven 2 days ago is fine; got verdict=%d reason=%q", fresh.verdict, fresh.reason)
|
||||
}
|
||||
@@ -283,3 +292,208 @@ func boolStr(b bool) string {
|
||||
}
|
||||
return "false"
|
||||
}
|
||||
|
||||
// ── SCENARIO G — a healthy WEEKLY tier is never reported stale (R-86 Part 2) ─────────────────
|
||||
//
|
||||
// This is the test that pins the false alarm this change would otherwise have CREATED. The agent
|
||||
// now proves a tier once per archive generation, so a weekly offsite tier is proved weekly — in
|
||||
// perfect health. Against the old flat 7-day window it would sit on the line and alarm every night.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): pin the window flat, as it was —
|
||||
//
|
||||
// - window := restoreProvenWindow(tier, observed, observedOK)
|
||||
// - window := restoreProvenWindowFloor // the pre-R-86 flat 7 days
|
||||
//
|
||||
// → --- FAIL: TestRestoreTest_HealthyWeeklyTierIsNeverStale
|
||||
//
|
||||
// week 0: a weekly tier proved on its own archive must never be stale (proof age 172h0m0s,
|
||||
// window 168h0m0s); verdict=2 reason="pbs tier: last successful restore-test was 172h0m0s ago
|
||||
// (limit 168h0m0s, this tier's own backup rhythm) — the tier is unverified, not known-broken"
|
||||
//
|
||||
// Restored. The mutation is one line because the whole of Part 2 is one decision: whose rhythm.
|
||||
//
|
||||
// NOTE, because it is the finding this test nearly hid: the FIRST version of this fixture had NO
|
||||
// jitter, and it PASSED under the mutation. A perfectly regular weekly tier's proof age reaches
|
||||
// EXACTLY 168h just before the next proof, and `age > window` is false by a hair — a hollow test
|
||||
// that would have shipped Part 1 and its false alarm together. The jitter below is what makes this
|
||||
// a test, and it is also the truth about the old constant: a healthy weekly tier did not merely sit
|
||||
// near the line, it sat ON it, so any ordinary delay tipped it over.
|
||||
func TestRestoreTest_HealthyWeeklyTierIsNeverStale(t *testing.T) {
|
||||
start := time.Date(2026, 6, 1, 3, 0, 0, 0, time.UTC)
|
||||
firstContact := start.Add(-24 * time.Hour)
|
||||
|
||||
// The observable rhythm of a weekly tier, as the hub would compute it from the reports. No
|
||||
// assertion about the window ITSELF here on purpose: that is the mechanism, and it is pinned in
|
||||
// TestRestoreProvenWindow_Contract. What this test asserts is the CONSEQUENCE — does the alarm
|
||||
// fire? — because R-97b proved a mechanism and shipped a broken consequence anyway.
|
||||
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
|
||||
|
||||
// Walk several weeks of a HEALTHY tier, with the jitter a real one has: the backup does not land
|
||||
// to the second, and a restore-test can be deferred one evaluation behind a running backup.
|
||||
//
|
||||
// The jitter is the point. A perfectly regular weekly tier's proof reaches an age of EXACTLY one
|
||||
// interval (168h) just before the next proof, and against a flat 168h window `age > window` is
|
||||
// false by a hair — so a regular fixture would pass against the very constant this task must
|
||||
// change, and prove nothing. That is the brief's "sits exactly on that line": every real-world
|
||||
// delay pushes it over, and the alarm is about a system that is working.
|
||||
settle, evalLatency := 24*time.Hour, 6*time.Hour
|
||||
archiveLate := []time.Duration{0, 4 * time.Hour, 2 * time.Hour, 6 * time.Hour, 0, 3 * time.Hour}
|
||||
deferred := []time.Duration{0, 0, 6 * time.Hour, 0, 0, 6 * time.Hour} // one evaluation behind a backup
|
||||
|
||||
archiveAt := func(week int) time.Time {
|
||||
return start.AddDate(0, 0, 7*week).Add(archiveLate[week])
|
||||
}
|
||||
provenAt := func(week int) time.Time {
|
||||
return archiveAt(week).Add(settle + evalLatency + deferred[week])
|
||||
}
|
||||
|
||||
var worst time.Duration
|
||||
for week := 0; week+1 < len(archiveLate); week++ {
|
||||
// The widest the proof's age ever gets: the instant before the NEXT week's proof lands.
|
||||
now := provenAt(week + 1).Add(-time.Second)
|
||||
age := now.Sub(provenAt(week))
|
||||
if age > worst {
|
||||
worst = age
|
||||
}
|
||||
v := assessRestoreProven("pbs", provenAt(week), firstContact, now, weekly)
|
||||
if v.verdict != verdictOK {
|
||||
t.Fatalf("week %d: a weekly tier proved on its own archive must never be stale (proof age %s, window %s); verdict=%d reason=%q",
|
||||
week, age.Round(time.Hour), weekly, v.verdict, v.reason)
|
||||
}
|
||||
}
|
||||
// The fixture must actually EXERCISE the boundary — a jitter-free walk would sit at exactly one
|
||||
// interval and pass against a flat 7-day window, which is the hollow version of this test.
|
||||
if worst <= restoreProvenWindowFloor {
|
||||
t.Fatalf("this fixture never exceeds the old flat window (worst proof age %s) — it cannot detect the defect it exists for", worst)
|
||||
}
|
||||
}
|
||||
|
||||
// ...and a weekly tier that genuinely STOPS being proved must still alarm. A window that never
|
||||
// fires is not a fix, it is a deletion.
|
||||
func TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
|
||||
|
||||
v := assessRestoreProven("pbs", now.Add(-weekly-time.Hour), now.Add(-90*24*time.Hour), now, weekly)
|
||||
if !v.missed() {
|
||||
t.Fatalf("a weekly tier unproven for longer than its own window MUST alarm; got verdict=%d reason=%q", v.verdict, v.reason)
|
||||
}
|
||||
if !strings.Contains(v.reason, weekly.String()) {
|
||||
t.Fatalf("the alarm must state the window it was judged against (R-100's corollary); got %q", v.reason)
|
||||
}
|
||||
}
|
||||
|
||||
// The window's own contract: derived from the tier's rhythm, floored, capped, and never dependent
|
||||
// on an unobservable history for the tier that would suffer most from a wrong answer.
|
||||
func TestRestoreProvenWindow_Contract(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
tier string
|
||||
observed time.Duration
|
||||
observedOK bool
|
||||
want time.Duration
|
||||
}{
|
||||
{"daily local clamps to the floor", "local", 24 * time.Hour, true, restoreProvenWindowFloor},
|
||||
{"weekly pbs widens", "pbs", 7 * 24 * time.Hour, true, restoreProvenWindowCap},
|
||||
{"3-day tier sits between", "pbs", 72 * time.Hour, true, 12 * 24 * time.Hour},
|
||||
{"unobservable local falls back to its declared rhythm", "local", 0, false, restoreProvenWindowFloor},
|
||||
{"unobservable pbs falls back WIDE, not to the floor", "pbs", 0, false, restoreProvenWindowCap},
|
||||
{"a nonsense zero interval is ignored", "pbs", 0, true, restoreProvenWindowCap},
|
||||
// MEASURED ON THE LIVE BOX, and the reason observation may only WIDEN. demo-felhom's two
|
||||
// retained PBS snapshots sit 8h54m apart (one is a healing artefact), so a mean-gap estimate
|
||||
// reads a WEEKLY tier as nine-hourly. Taking that at face value gives 4x9h = 36h → the 7-day
|
||||
// floor → and a weekly tier proved weekly (~8.25d of proof age) alarms within a week of this
|
||||
// shipping, on the box it shipped to.
|
||||
{"a short observed gap must NOT tighten a weekly tier", "pbs", 8*time.Hour + 54*time.Minute, true, restoreProvenWindowCap},
|
||||
{"a short observed gap must not tighten the host tier either", "local", 30 * time.Minute, true, restoreProvenWindowFloor},
|
||||
// ...but a tier that genuinely runs SLOWER than its declared rhythm still widens.
|
||||
{"a genuinely slower tier widens", "local", 4 * 24 * time.Hour, true, restoreProvenWindowCap},
|
||||
}
|
||||
// The relationship Part 1 depends on: a weekly tier's window must be WIDER than a daily tier's,
|
||||
// or proving weekly (which is now correct behaviour) alarms on itself.
|
||||
if restoreProvenWindow("pbs", 7*24*time.Hour, true) <= restoreProvenWindow("local", 24*time.Hour, true) {
|
||||
t.Fatal("a weekly tier must earn a wider window than a daily one — otherwise R-86's agent half alarms about itself")
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := restoreProvenWindow(c.tier, c.observed, c.observedOK)
|
||||
if got != c.want {
|
||||
t.Fatalf("window(%s, observed=%s ok=%v) = %s, want %s", c.tier, c.observed, c.observedOK, got, c.want)
|
||||
}
|
||||
if got < restoreProvenWindowFloor || got > restoreProvenWindowCap {
|
||||
t.Fatalf("every window must stay inside [%s, %s]; got %s", restoreProvenWindowFloor, restoreProvenWindowCap, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The rhythm must be OBSERVED from the reports, not assumed — including the slice-A.4 rule that a
|
||||
// PBS-targeted vzdump appears in both arrays and must be attributed by TARGET TYPE.
|
||||
func TestObservedArchiveIntervals_FromReports(t *testing.T) {
|
||||
base := time.Date(2026, 7, 1, 2, 0, 0, 0, time.UTC)
|
||||
mk := func(localAt []time.Time, pbsAt []time.Time) string {
|
||||
type stg struct{ Name, Type, Content string }
|
||||
type bk struct {
|
||||
TargetID string `json:"target_id"`
|
||||
Success bool `json:"success"`
|
||||
StartedAt string `json:"started_at"`
|
||||
}
|
||||
type snap struct {
|
||||
BackupTime string `json:"backup_time"`
|
||||
}
|
||||
payload := struct {
|
||||
StorageTargets []struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
} `json:"storage_targets"`
|
||||
Backups []bk `json:"backups"`
|
||||
PBSSnapshots []snap `json:"pbs_snapshots"`
|
||||
}{}
|
||||
payload.StorageTargets = append(payload.StorageTargets, struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
}{"felhom-backup", "dir", "backup"}, struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
}{"felhom-pbs", "pbs", "backup"})
|
||||
for _, at := range localAt {
|
||||
payload.Backups = append(payload.Backups, bk{"felhom-backup", true, at.Format(time.RFC3339)})
|
||||
}
|
||||
for _, at := range pbsAt {
|
||||
// The SAME archive appears as a vzdump record AND as a snapshot — slice A.4.
|
||||
payload.Backups = append(payload.Backups, bk{"felhom-pbs", true, at.Format(time.RFC3339)})
|
||||
payload.PBSSnapshots = append(payload.PBSSnapshots, snap{at.Format(time.RFC3339)})
|
||||
}
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
rows := []store.HostReportRow{
|
||||
{ReportJSON: mk(
|
||||
[]time.Time{base, base.AddDate(0, 0, 1), base.AddDate(0, 0, 2)},
|
||||
[]time.Time{base, base.AddDate(0, 0, 7)},
|
||||
)},
|
||||
{ReportJSON: `{{{malformed`}, // must not blind the scan
|
||||
}
|
||||
|
||||
got := observedArchiveIntervals(rows)
|
||||
if d, ok := got["local"]; !ok || d != 24*time.Hour {
|
||||
t.Fatalf("a daily host tier must be observed as ~24h; got %s ok=%v", d, ok)
|
||||
}
|
||||
if d, ok := got["pbs"]; !ok || d != 7*24*time.Hour {
|
||||
t.Fatalf("a weekly offsite tier must be observed as ~7d — and its vzdump record must NOT be "+
|
||||
"counted into the host tier (slice A.4); got %s ok=%v", d, ok)
|
||||
}
|
||||
|
||||
// One generation is not a rhythm: unobservable, so the caller falls back to the declared one.
|
||||
single := []store.HostReportRow{{ReportJSON: mk(nil, []time.Time{base})}}
|
||||
if d, ok := observedArchiveIntervals(single)["pbs"]; ok {
|
||||
t.Fatalf("one archive cannot yield an interval; got %s", d)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,301 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-182 — one e-mail per backup run, and nothing dropped without a trace.
|
||||
//
|
||||
// MEASURED, NOT SUPPOSED. On 2026-08-03 nine `recovery_unit_capture_failed` events reached the hub
|
||||
// and TWO operator e-mails went out. The operator cooldown key is
|
||||
// `customerID + ":" + eventType + cooldownTierSuffix(details)`, that event carries `app` but no
|
||||
// `tier`, so the key held no app identifier: the first refused app took the hour's slot and every
|
||||
// other app's failure was discarded — **before `LogNotification`**, so it left no row on any channel
|
||||
// and could not be found afterwards.
|
||||
//
|
||||
// The operator ruled against the obvious fix (putting `app` in the key), because on a full disk that
|
||||
// is one e-mail per app. These pin the shape that replaced it.
|
||||
|
||||
// ── Scenario D — a suppressed operator event leaves a trace ───────────────────────────────────────
|
||||
|
||||
// The bare `return` at the cooldown is the whole reason this defect took a day to get the right way
|
||||
// round: there was nothing to read. A drop must be as visible in the record as a send.
|
||||
//
|
||||
// DELIBERATELY EXERCISED ON A DIFFERENT EVENT TYPE than the one that exposed the defect.
|
||||
// `recovery_unit_capture_failed` is now record-only and never reaches the cooldown at all, so using
|
||||
// it here would prove nothing. `whole_guest_backup_failed` is an ordinary operator event, and using
|
||||
// it pins §2.1's actual claim: the suppression row applies to EVERY operator event, not only the one
|
||||
// that happened to be measured.
|
||||
func TestSuppressedOperatorEvent_LeavesARow(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
// Two events of the SAME type with no discriminator — the second must be suppressed.
|
||||
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "opengist failed", `{"app":"opengist"}`, "controller")
|
||||
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "privatebin failed", `{"app":"privatebin"}`, "controller")
|
||||
|
||||
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
|
||||
t.Fatalf("operator mails = %d, want 1 — the premise of this test is that the second IS suppressed", got)
|
||||
}
|
||||
|
||||
rows, err := st.GetRecentNotifications("c1", 50)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var sentRows, suppressed []store2Row
|
||||
for _, r := range rows {
|
||||
if r.Channel != "operator" || r.EventType != "whole_guest_backup_failed" {
|
||||
continue
|
||||
}
|
||||
switch r.Status {
|
||||
case "sent":
|
||||
sentRows = append(sentRows, store2Row{r.Status, r.Message, r.ErrorMessage})
|
||||
case "suppressed":
|
||||
suppressed = append(suppressed, store2Row{r.Status, r.Message, r.ErrorMessage})
|
||||
}
|
||||
}
|
||||
if len(sentRows) != 1 {
|
||||
t.Fatalf("want 1 'sent' operator row, got %d", len(sentRows))
|
||||
}
|
||||
if len(suppressed) != 1 {
|
||||
t.Fatalf("want 1 'suppressed' operator row, got %d — a cooldown drop that writes NOTHING is "+
|
||||
"indistinguishable from an event that never happened, which is exactly how seven "+
|
||||
"failures went missing on 2026-08-03", len(suppressed))
|
||||
}
|
||||
// The row must name the app that was dropped, or it records that something was suppressed
|
||||
// without recording WHAT — half a fix.
|
||||
if !strings.Contains(suppressed[0].message, "privatebin") {
|
||||
t.Fatalf("the suppressed row does not name the dropped event: %q", suppressed[0].message)
|
||||
}
|
||||
// And it must carry the key, so the reason it collided is readable without reading code.
|
||||
if !strings.Contains(suppressed[0].errMsg, "key=") {
|
||||
t.Fatalf("the suppressed row does not carry the cooldown key: %q", suppressed[0].errMsg)
|
||||
}
|
||||
}
|
||||
|
||||
type store2Row struct{ status, message, errMsg string }
|
||||
|
||||
// ── Scenario E — two runs in a day each report ───────────────────────────────────────────────────
|
||||
|
||||
// The operator ruled explicitly on this: someone pressing the backup button is actively trying to
|
||||
// get a backup, and finding out tomorrow would be worse than an extra mail in a rare case.
|
||||
func TestTwoRunsInAnHour_BothReport(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
nightly := `{"run_id":"run-a","run_kind":"nightly","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
|
||||
manual := `{"run_id":"run-b","run_kind":"manual","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
|
||||
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", nightly, "controller")
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", manual, "controller")
|
||||
|
||||
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 2 {
|
||||
t.Fatalf("operator mails = %d, want 2 — the 1-hour cooldown swallowed the manual run's "+
|
||||
"digest, which is the fix reappearing one level up: press the button, the run fails, "+
|
||||
"and you are told nothing because the machine already wrote this hour", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The run discriminator must be NARROW, exactly like its `tier` sibling — empty unless the producer
|
||||
// opts in — or every existing event type's cooldown silently stops collapsing anything.
|
||||
func TestCooldownRunSuffix_EmptyForEverythingElse(t *testing.T) {
|
||||
cases := []struct{ name, details string }{
|
||||
{"no details", ""},
|
||||
{"details without run_id", `{"app":"opengist","error":"boom"}`},
|
||||
{"empty run_id value", `{"run_id":""}`},
|
||||
{"malformed json", `{{{nope`},
|
||||
{"run_id mentioned in a STRING, not as a key", `{"error":"the run_id: abc failed"}`},
|
||||
{"null details", `null`},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := cooldownRunSuffix(c.details); got != "" {
|
||||
t.Errorf("%s: suffix must be EMPTY so every other type's cooldown is unchanged, got %q", c.name, got)
|
||||
}
|
||||
}
|
||||
if got := cooldownRunSuffix(`{"run_id":"run-a"}`); got != ":run-a" {
|
||||
t.Fatalf("suffix should be the run id, got %q", got)
|
||||
}
|
||||
// The two suffixes must not interfere: a tier event still keys on its tier and nothing else.
|
||||
if got := cooldownTierSuffix(`{"tier":"felhom-pbs"}`) + cooldownRunSuffix(`{"tier":"felhom-pbs"}`); got != ":felhom-pbs" {
|
||||
t.Fatalf("a tier-only event's key changed to %q — R-97a's behaviour must be byte-identical", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario G — the customer never receives the digest ──────────────────────────────────────────
|
||||
|
||||
// v0.78.0 asserted in a COMMENT that a type with no `customerMessages` entry structurally cannot
|
||||
// reach a customer. It can: FormatCustomerEmail falls back to the raw English message and the only
|
||||
// customer gate is configuration. So this is demonstrated, not argued.
|
||||
func TestDigest_IsOperatorOnly_EvenWithAWideEnabledList(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
// A customer who has enabled EVERYTHING, including this type by name.
|
||||
if err := st.SaveNotificationPrefs("c1", "cust@example.com",
|
||||
[]string{"backup_run_failures", "node_down", "disk_warning"}, 6); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
details := `{"run_id":"run-a","run_kind":"nightly","failed":1,"attempted":3,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"}]}`
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", "1 of 3 apps failed", details, "controller")
|
||||
|
||||
if got := mailsFor(*sent, "cust@example.com"); len(got) != 0 {
|
||||
t.Fatalf("the CUSTOMER received an operator digest (%d mails) — a list of which apps' "+
|
||||
"backups failed is not something they can act on, and the raw body is English", len(got))
|
||||
}
|
||||
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
|
||||
t.Fatalf("operator mails = %d, want 1", got)
|
||||
}
|
||||
if !operatorOnlyEvents["backup_run_failures"] {
|
||||
t.Fatal("backup_run_failures is not in operatorOnlyEvents — allowlisting alone does NOT " +
|
||||
"keep it from a customer; that assumption shipped once and was wrong (v0.78.0)")
|
||||
}
|
||||
}
|
||||
|
||||
// ── Part 3 — the e-mail a person actually reads ──────────────────────────────────────────────────
|
||||
|
||||
func TestDigestEmail_ListsAppsLegsAndReasons(t *testing.T) {
|
||||
details := `{"run_id":"run-a","run_kind":"nightly","failed":3,"attempted":40,` +
|
||||
`"target_path":"/mnt/sys_drive","used_gb":64.3,"avail_gb":0.9,"total_gb":68.7,` +
|
||||
`"used_percent":94,"space_known":true,"apps":[` +
|
||||
`{"app":"opengist","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
|
||||
`{"app":"privatebin","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
|
||||
`{"app":"immich","leg":"database dump","reason":"pg_dump: connection refused"}]}`
|
||||
|
||||
subject, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error",
|
||||
"3 of 40 apps failed to back up", details)
|
||||
|
||||
// The subject must carry the counts: the operator's first decision is made from it alone.
|
||||
for _, want := range []string{"demo-hp", "3 of 40", "nightly"} {
|
||||
if !strings.Contains(subject, want) {
|
||||
t.Errorf("subject %q missing %q", subject, want)
|
||||
}
|
||||
}
|
||||
// Every app, its leg and its reason.
|
||||
for _, want := range []string{
|
||||
"opengist", "privatebin", "immich",
|
||||
"volume dump", "database dump",
|
||||
"below the reserve", "pg_dump: connection refused",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("body missing %q:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
// The counts and the free space, so "one broken app" and "a full disk" read differently.
|
||||
if !strings.Contains(body, "3 of 40") {
|
||||
t.Errorf("body does not carry the failed-of-attempted count:\n%s", body)
|
||||
}
|
||||
if !strings.Contains(body, "0.9 GB free") {
|
||||
t.Errorf("body does not carry the free space:\n%s", body)
|
||||
}
|
||||
// It must NOT be a JSON blob.
|
||||
if strings.Contains(body, `"apps":[`) {
|
||||
t.Errorf("the digest rendered as raw JSON — unreadable on a phone at 07:00:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// An absent space reading must render as unavailable, never as zeros: "0 GB free" and "we could not
|
||||
// look" are opposite diagnoses, and the operator acts differently on each.
|
||||
func TestDigestEmail_UnknownSpaceIsNotZero(t *testing.T) {
|
||||
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/x",` +
|
||||
`"space_known":false,"apps":[{"app":"a","leg":"capture","reason":"boom"}]}`
|
||||
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
|
||||
if strings.Contains(body, "0.0 GB free") {
|
||||
t.Fatalf("an unreadable filesystem rendered as zeros:\n%s", body)
|
||||
}
|
||||
if !strings.Contains(body, "unavailable") {
|
||||
t.Fatalf("an unreadable filesystem must say so:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A payload that cannot be parsed must still produce a mail — degraded, never swallowed.
|
||||
func TestDigestEmail_UnparseableDetailsStillMails(t *testing.T) {
|
||||
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "something failed", `{{{`)
|
||||
if body == "" || !strings.Contains(body, "something failed") {
|
||||
t.Fatalf("an unparseable digest lost the mail entirely:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario C — every failure is RECORDED, e-mailed or not ──────────────────────────────────────
|
||||
|
||||
// The per-app event is the record; the digest is the notification. The record must not inherit the
|
||||
// notification's conditions — no cooldown, no preferences, no dependence on a mail going out.
|
||||
func TestPerAppFailure_IsRecordedButNotMailed(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
sent := captureSeam(d)
|
||||
|
||||
apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"}
|
||||
for _, a := range apps {
|
||||
d.ProcessEvent("c1", "recovery_unit_capture_failed", "error",
|
||||
"Recovery unit capture FAILED for \""+a+"\"", `{"app":"`+a+`"}`, "controller")
|
||||
}
|
||||
|
||||
// NOT mailed — the digest is the notification.
|
||||
if got := len(*sent); got != 0 {
|
||||
t.Fatalf("%d mail(s) sent for per-app failures — they are the RECORD; one mail per app on a "+
|
||||
"full disk is the volume problem wearing the correctness problem's clothes, which is "+
|
||||
"exactly what the operator ruled against", got)
|
||||
}
|
||||
|
||||
// But ALL FIVE recorded — this is the assertion yesterday's defect would have failed: nine
|
||||
// arrived, two were mailed, seven left no row anywhere.
|
||||
rows, err := st.GetRecentNotifications("c1", 50)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, r := range rows {
|
||||
if r.EventType == "recovery_unit_capture_failed" && r.Status == "recorded" {
|
||||
for _, a := range apps {
|
||||
if strings.Contains(r.Message, a) {
|
||||
seen[a] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(seen) != len(apps) {
|
||||
t.Fatalf("only %d of %d per-app failures were recorded (%v) — a failure that produced no row "+
|
||||
"anywhere is the measured defect of 2026-08-03", len(seen), len(apps), seen)
|
||||
}
|
||||
}
|
||||
|
||||
// The per-app reason must not repeat the filesystem figures the digest already prints once. Reviewed
|
||||
// as copy against the first real digest, not designed in the abstract.
|
||||
func TestDigestEmail_ReasonDoesNotRepeatTheUsageLine(t *testing.T) {
|
||||
reason := "refused: below the reserve (reserve: 97% used or 1.0 GiB free) — /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free"
|
||||
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/sys_drive",` +
|
||||
`"used_gb":65,"avail_gb":0.2,"total_gb":68.7,"used_percent":95,"space_known":true,` +
|
||||
`"apps":[{"app":"opengist","leg":"whole app","reason":"` + reason + `"}]}`
|
||||
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
|
||||
|
||||
// The figures appear ONCE, on the Filesystem line — not again on every app row.
|
||||
if strings.Count(body, "65.0/68.7 GB used") != 1 {
|
||||
t.Fatalf("the usage clause appears %d times; it must appear once, on its own line — repeated "+
|
||||
"down a list of a dozen apps it pushes the part that DIFFERS off a phone screen:\n%s",
|
||||
strings.Count(body, "65.0/68.7 GB used"), body)
|
||||
}
|
||||
// But the reason itself survives — trimming must not eat the diagnosis.
|
||||
if !strings.Contains(body, "below the reserve") {
|
||||
t.Fatalf("the reason was trimmed away entirely:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A reason naming a DIFFERENT path, or none, must be left completely alone.
|
||||
func TestTrimRepeatedUsage_LeavesUnrelatedReasonsAlone(t *testing.T) {
|
||||
for _, c := range []struct{ reason, target string }{
|
||||
{"pg_dump: connection refused", "/mnt/sys_drive"},
|
||||
{"tar failed — /mnt/other: 1/2 GB used (50%), 1 GB free", "/mnt/sys_drive"},
|
||||
{"boom", ""},
|
||||
{"", "/mnt/sys_drive"},
|
||||
} {
|
||||
if got := trimRepeatedUsage(c.reason, c.target); got != c.reason {
|
||||
t.Errorf("reason %q (target %q) was altered to %q", c.reason, c.target, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -116,6 +116,18 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta
|
||||
return
|
||||
}
|
||||
|
||||
// R-182: record-only types are written down and never mailed. Placed BEFORE the severity gate
|
||||
// so the row is written whatever the severity — the record must not inherit the notification's
|
||||
// conditions, which is the coupling this whole finding is about.
|
||||
if recordOnlyEvents[eventType] {
|
||||
if err := d.store.LogNotification(customerID, eventType, severity, message, "recorded",
|
||||
"record-only: the per-run digest (backup_run_failures) carries the notification", "operator"); err != nil {
|
||||
d.logger.Printf("[WARN] Failed to record %s for %s: %v", eventType, customerID, err)
|
||||
}
|
||||
d.logger.Printf("[INFO] Recorded (not mailed) %s for %s — the run digest is the notification", eventType, customerID)
|
||||
return
|
||||
}
|
||||
|
||||
// warning / error / critical trigger notifications. "info" is an intentional non-notify (status/
|
||||
// recovery events). Anything else is UNRECOGNIZED — log it (don't silently drop), so a bad severity
|
||||
// surfaces instead of vanishing (the felhom-pve-class lesson: a critical event must never be lost).
|
||||
@@ -260,15 +272,67 @@ func cooldownTierSuffix(detailsJSON string) string {
|
||||
return ":" + d.Tier
|
||||
}
|
||||
|
||||
// cooldownRunSuffix returns ":"+run_id when the event's details carry a non-empty `run_id`, else "".
|
||||
//
|
||||
// R-182. `cooldownTierSuffix`'s sibling, and deliberately a SEPARATE function rather than an extra
|
||||
// branch inside it: `tier` keeps byte-identical semantics for every type that uses it, so R-97a's
|
||||
// behaviour and its tests are untouched by this.
|
||||
//
|
||||
// WHY A BACKUP RUN NEEDS ONE. The run digest describes ONE RUN, and a box can have two in a day —
|
||||
// the nightly one and a manual one the operator triggered *because* something looked wrong. With no
|
||||
// run-scoped discriminator the 1-hour cooldown would swallow the second, which is the failure this
|
||||
// row exists to fix, reappearing one level up: the operator presses the button, the run fails, and
|
||||
// they are told nothing because the machine already wrote that hour.
|
||||
//
|
||||
// IT MAKES THE COOLDOWN EFFECTIVELY INERT FOR THIS TYPE, AND THAT IS THE INTENT, NOT AN OVERSIGHT.
|
||||
// A digest is already rate-limited by construction — one per run, emitted only when something
|
||||
// failed — so there is nothing for a timer to collapse. The cooldown protects against a repeating
|
||||
// identical alert; a digest cannot repeat, because each run is a different run.
|
||||
//
|
||||
// NARROW, LIKE ITS SIBLING: empty unless the producer opts in by sending a `run_id`, so no existing
|
||||
// event type's cooldown behaviour changes.
|
||||
func cooldownRunSuffix(detailsJSON string) string {
|
||||
if detailsJSON == "" || !strings.Contains(detailsJSON, "\"run_id\"") {
|
||||
return ""
|
||||
}
|
||||
var d struct {
|
||||
RunID string `json:"run_id"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || d.RunID == "" {
|
||||
return ""
|
||||
}
|
||||
return ":" + d.RunID
|
||||
}
|
||||
|
||||
func (d *Dispatcher) processOperator(customerID, eventType, severity, message, detailsJSON, source string) {
|
||||
if !d.operatorOn || d.operatorEmail == "" {
|
||||
return
|
||||
}
|
||||
|
||||
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON)
|
||||
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON) + cooldownRunSuffix(detailsJSON)
|
||||
d.mu.Lock()
|
||||
if last, ok := d.opCooldowns[cooldownKey]; ok && time.Since(last) < 1*time.Hour {
|
||||
d.mu.Unlock()
|
||||
// R-182: RECORD THE SUPPRESSION. This used to be a bare `return` — the event was dropped
|
||||
// before any LogNotification, so a cooldown drop and an event that never happened were
|
||||
// indistinguishable from the operator's side AND from the hub's own records.
|
||||
//
|
||||
// Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two emails were
|
||||
// sent, and the other seven left NO ROW ON ANY CHANNEL. The defect that hid was serious —
|
||||
// the cooldown key carries no app identifier, so the first refused app took the slot and
|
||||
// every other app's failure that hour was discarded — but the reason it took a day to find
|
||||
// the right way round is this line: there was nothing to read.
|
||||
//
|
||||
// "We chose not to e-mail you" and "nothing happened" must never look identical. This
|
||||
// applies to EVERY operator event, not only the one that exposed it. It makes the drop
|
||||
// visible; it deliberately does NOT change the cooldown's duration or semantics.
|
||||
if err := d.store.LogNotification(customerID, eventType, severity, message,
|
||||
"suppressed", "operator cooldown 1h, key="+cooldownKey, "operator"); err != nil {
|
||||
d.logger.Printf("[WARN] Failed to record suppressed operator notification for %s/%s: %v",
|
||||
customerID, eventType, err)
|
||||
}
|
||||
d.logger.Printf("[INFO] Operator email suppressed for %s/%s — cooldown (key=%s)",
|
||||
customerID, eventType, cooldownKey)
|
||||
return
|
||||
}
|
||||
d.opCooldowns[cooldownKey] = time.Now()
|
||||
@@ -285,6 +349,33 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "operator")
|
||||
}
|
||||
|
||||
// recordOnlyEvents are STORED and RECORDED but never e-mailed, on either channel.
|
||||
//
|
||||
// R-182. The distinction this register exists to make is the whole of that finding: **the record and
|
||||
// the notification are different things.** A per-app backup failure must always be written down —
|
||||
// every time, unconditionally, regardless of cooldowns, preferences or whether any mail went out —
|
||||
// and it must NOT compete for an e-mail slot, because the per-run digest
|
||||
// (`backup_run_failures`) is what a person is meant to read.
|
||||
//
|
||||
// Before this, `recovery_unit_capture_failed` was both at once, and it did neither well: on
|
||||
// 2026-08-03 nine of them arrived, two were e-mailed, and the other seven were dropped by the
|
||||
// 1-hour cooldown BEFORE anything was written down. So the operator was told about one app, the
|
||||
// other apps' failures were discarded, and nothing anywhere recorded that a choice had been made.
|
||||
//
|
||||
// WHY A REGISTER AND NOT severity "info". Downgrading the severity would have the same routing
|
||||
// effect — `severityNotifies` drops info — but it would also relabel a genuine failure as
|
||||
// informational in the events table, the operator UI and every historical query, and it would
|
||||
// silently drop the X-Priority handling if the type were ever promoted back. This says what it
|
||||
// means: not silent, not urgent, RECORDED.
|
||||
//
|
||||
// IT IS NOT A WAY TO MUTE THINGS. A type belongs here only when something else carries its
|
||||
// notification. Adding one with no digest behind it rebuilds the silence R-182 was filed against.
|
||||
var recordOnlyEvents = map[string]bool{
|
||||
// The per-app Tier-1 capture/refusal failure. Its notification is the run digest, which lists
|
||||
// every failed app in one mail; this row is the durable per-failure record behind it.
|
||||
"recovery_unit_capture_failed": true,
|
||||
}
|
||||
|
||||
// operatorOnlyEvents are event types that must NEVER reach a customer, whatever their preferences say.
|
||||
//
|
||||
// R-97c. This register exists because the guarantee it provides was previously ASSERTED IN A COMMENT
|
||||
@@ -321,6 +412,15 @@ var operatorOnlyEvents = map[string]bool{
|
||||
// figures, the raw error). The customer's half of D-c is the FILL WARNING, which fires BEFORE
|
||||
// this and is actionable: free space, delete files, add a drive.
|
||||
"recovery_unit_capture_failed": true,
|
||||
// R-182. The per-run backup digest. It is the same class as the line above and for the same
|
||||
// reason — a customer can act on a full disk (that is the fill warning, which fires first and
|
||||
// IS customer-facing) but not on a list of which apps' backups failed and why. It also carries
|
||||
// operator-grade detail: per-app leg names, raw refusal reasons and byte figures.
|
||||
//
|
||||
// Listed here rather than relying on the absence of a `customerMessages` entry, which is NOT a
|
||||
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
|
||||
// once (v0.78.0) and the comment above records it.
|
||||
"backup_run_failures": true,
|
||||
}
|
||||
|
||||
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
|
||||
|
||||
@@ -39,33 +39,45 @@ func TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The operator must still get it: the register mutes the customer channel, not the signal.
|
||||
gotOperator := false
|
||||
// R-182 CHANGED WHAT THIS ASSERTS, DELIBERATELY, AND THE OLD ASSERTION IS WORTH KEEPING IN VIEW.
|
||||
//
|
||||
// Until 2026-08-03 this test required the OPERATOR to be e-mailed here, on the grounds that "the
|
||||
// alert is the whole point of R-158". That was right when this event was the only signal, and it
|
||||
// is wrong now: measured, nine of these arrived at the hub and two were mailed, because the
|
||||
// operator cooldown key carries no app identifier — so as an alarm it told the operator about one
|
||||
// app and threw the rest away.
|
||||
//
|
||||
// The type is now RECORD-ONLY: written down every time, never mailed. R-158's guarantee — the
|
||||
// operator learns WHICH app failed and WHY — is not weakened, it MOVED: the per-run digest
|
||||
// `backup_run_failures` carries every failed app in one mail, and is pinned by
|
||||
// backup_run_digest_test.go. The customer safety claim below is untouched and is the reason this
|
||||
// test still exists.
|
||||
for _, to := range rec.to {
|
||||
if to == "operator@felhom.eu" {
|
||||
gotOperator = true
|
||||
t.Fatal("the operator was e-mailed a PER-APP capture failure — this type is the record " +
|
||||
"now, not the alarm. One mail per failing app on a full disk is a dozen mails, which " +
|
||||
"is the volume problem the operator ruled against; the digest is the notification")
|
||||
}
|
||||
}
|
||||
if !gotOperator {
|
||||
t.Fatal("the operator was not notified of a recovery-unit capture failure — the alert is the " +
|
||||
"whole point of R-158 and it went nowhere")
|
||||
}
|
||||
|
||||
// The skip must be VISIBLE. An absent log row is equally consistent with "correctly skipped" and
|
||||
// "the dispatcher never ran" — the positive observable is the row itself (standing rule 3).
|
||||
// The RECORD must exist, always. It is what makes the digest trustworthy: if the digest is ever
|
||||
// lost, delayed or suppressed, the failures are still individually written down. An absent row is
|
||||
// equally consistent with "correctly not mailed" and "the dispatcher never ran" — the positive
|
||||
// observable is the row itself (standing rule 3).
|
||||
logs, err := st.GetRecentNotifications("c1", 20)
|
||||
if err != nil {
|
||||
t.Fatalf("GetRecentNotifications: %v", err)
|
||||
}
|
||||
found := false
|
||||
for _, l := range logs {
|
||||
if l.Channel == "customer" && l.Status == "skipped" && strings.Contains(l.ErrorMessage, "operator_only") {
|
||||
if l.EventType == "recovery_unit_capture_failed" && l.Status == "recorded" &&
|
||||
strings.Contains(l.ErrorMessage, "record-only") {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("the customer skip is not logged as skipped/operator_only — it is indistinguishable "+
|
||||
"from a delivery that never happened; got %d row(s)", len(logs))
|
||||
t.Fatalf("the per-app failure left no 'recorded' row — a failure that is neither mailed nor "+
|
||||
"written down is exactly the 2026-08-03 defect, rebuilt; got %d row(s)", len(logs))
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -41,6 +42,17 @@ Severity: %s
|
||||
Time: %s
|
||||
Message: %s`, customerID, eventType, severity, now, message)
|
||||
|
||||
// R-182: the backup run digest gets a rendered list instead of a raw JSON blob. It is the one
|
||||
// operator mail that carries a VARIABLE-LENGTH payload, and a dozen apps as one line of JSON is
|
||||
// unreadable on a phone at 07:00, which is the only time it matters.
|
||||
if eventType == "backup_run_failures" {
|
||||
if rendered, sub, ok := renderBackupRunFailures(customerID, detailsJSON); ok {
|
||||
return sub, body + rendered + fmt.Sprintf("\n\nDashboard: https://hub.felhom.eu/customers/%s", customerID)
|
||||
}
|
||||
// Unparseable details fall through to the raw form below rather than losing the mail. A
|
||||
// digest that renders badly still tells the operator something; a swallowed one does not.
|
||||
}
|
||||
|
||||
if detailsJSON != "" && detailsJSON != "{}" {
|
||||
body += fmt.Sprintf("\nDetails: %s", detailsJSON)
|
||||
}
|
||||
@@ -282,3 +294,115 @@ Ha nem te kérted ezt, hagyd figyelmen kívül ezt az e-mailt.
|
||||
Felhom.eu`, link)
|
||||
return subject, body
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────────────────────────────
|
||||
// R-182 — the backup run digest
|
||||
// ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
// backupRunFailure is one app's failed leg within a run.
|
||||
type backupRunFailure struct {
|
||||
App string `json:"app"`
|
||||
Leg string `json:"leg"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// backupRunDetails is the digest payload the controller sends.
|
||||
type backupRunDetails struct {
|
||||
RunID string `json:"run_id"`
|
||||
RunKind string `json:"run_kind"`
|
||||
Failed int `json:"failed"`
|
||||
Attempted int `json:"attempted"`
|
||||
TargetPath string `json:"target_path"`
|
||||
UsedGB float64 `json:"used_gb"`
|
||||
AvailGB float64 `json:"avail_gb"`
|
||||
TotalGB float64 `json:"total_gb"`
|
||||
UsedPercent float64 `json:"used_percent"`
|
||||
SpaceKnown bool `json:"space_known"`
|
||||
Apps []backupRunFailure `json:"apps"`
|
||||
}
|
||||
|
||||
// renderBackupRunFailures turns the digest details into an operator-readable block and a subject
|
||||
// that says the count without being opened. Returns ok=false when the payload cannot be parsed or
|
||||
// names no apps, so the caller can fall back to the raw rendering rather than mail an empty list.
|
||||
//
|
||||
// THE SUCCESS COUNT IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
|
||||
// night; the list alone cannot tell them apart, and the operator's first decision — get up now, or
|
||||
// look after coffee — depends entirely on which it is.
|
||||
func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bool) {
|
||||
if detailsJSON == "" {
|
||||
return "", "", false
|
||||
}
|
||||
var d backupRunDetails
|
||||
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || len(d.Apps) == 0 {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
kind := d.RunKind
|
||||
if kind == "" {
|
||||
kind = "backup"
|
||||
}
|
||||
subject := fmt.Sprintf("[Felhom] 🔴 %s: %d of %d apps failed to back up (%s run)",
|
||||
customerID, d.Failed, d.Attempted, kind)
|
||||
|
||||
// Column-align the app names so the leg and reason line up and the block scans vertically.
|
||||
width := 0
|
||||
for _, a := range d.Apps {
|
||||
if len(a.App) > width {
|
||||
width = len(a.App)
|
||||
}
|
||||
}
|
||||
legWidth := 0
|
||||
for _, a := range d.Apps {
|
||||
if len(a.Leg) > legWidth {
|
||||
legWidth = len(a.Leg)
|
||||
}
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "\n\nFAILED: %d of %d apps attempted in this %s run.\n\n", d.Failed, d.Attempted, kind)
|
||||
for _, a := range d.Apps {
|
||||
reason := trimRepeatedUsage(a.Reason, d.TargetPath)
|
||||
if reason == "" {
|
||||
reason = "(no reason recorded)"
|
||||
}
|
||||
fmt.Fprintf(&b, " %-*s %-*s %s\n", width, a.App, legWidth, a.Leg, reason)
|
||||
}
|
||||
|
||||
// The space figures answer "is this one broken app or a full disk" before the reasons are read.
|
||||
// An absent reading renders as unavailable, never as zeros — "0 GB free" and "we could not look"
|
||||
// are opposite diagnoses (the UnitSpace rule, same reasoning, other side of the wire).
|
||||
if d.SpaceKnown {
|
||||
fmt.Fprintf(&b, "\nFilesystem: %s — %.1f/%.1f GB used (%.0f%%), %.1f GB free\n",
|
||||
d.TargetPath, d.UsedGB, d.TotalGB, d.UsedPercent, d.AvailGB)
|
||||
} else {
|
||||
fmt.Fprintf(&b, "\nFilesystem: %s — usage unavailable (the filesystem could not be read)\n", d.TargetPath)
|
||||
}
|
||||
|
||||
b.WriteString("\nEvery failure above is also recorded individually in the notification log,\n")
|
||||
b.WriteString("whether or not this mail was sent.")
|
||||
return b.String(), subject, true
|
||||
}
|
||||
|
||||
// trimRepeatedUsage strips the trailing "— /path: X/Y GB used (Z%), W GB free" clause from a per-app
|
||||
// reason, because the digest prints those figures ONCE for the whole run on its own line.
|
||||
//
|
||||
// This is a copy fix, and it was made after reading the first real digest rather than from the
|
||||
// design. The reserve's refusal message is authored for a single-app alert, where naming the
|
||||
// filesystem is exactly right; repeated down a list of a dozen apps it is the same forty characters
|
||||
// twelve times, and it pushes the part that differs off the right-hand edge of a phone screen at
|
||||
// 07:00 — which is the only moment this mail has to work.
|
||||
//
|
||||
// It trims ONLY an exact "— <target path>:" suffix, so a reason that mentions a different path, or
|
||||
// none, is left completely alone. A reason that is nothing but the usage clause is left alone too:
|
||||
// removing everything would turn a bad line into an empty one.
|
||||
func trimRepeatedUsage(reason, targetPath string) string {
|
||||
if reason == "" || targetPath == "" {
|
||||
return reason
|
||||
}
|
||||
marker := " — " + targetPath + ":"
|
||||
i := strings.LastIndex(reason, marker)
|
||||
if i <= 0 {
|
||||
return reason
|
||||
}
|
||||
return strings.TrimSpace(reason[:i])
|
||||
}
|
||||
|
||||
@@ -815,7 +815,7 @@ type NotificationLogEntry struct {
|
||||
EventType string
|
||||
Severity string
|
||||
Message string
|
||||
Status string // "sent", "skipped", "failed"
|
||||
Status string // "sent", "skipped", "failed", "suppressed" (R-182: a cooldown drop, recorded rather than silent)
|
||||
ErrorMessage string
|
||||
Channel string // "operator" or "customer"
|
||||
CreatedAt time.Time
|
||||
|
||||
+1
-1
@@ -125,7 +125,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: hub
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.89.0
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.91.1
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
|
||||
@@ -324,7 +324,7 @@ spec:
|
||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||
args:
|
||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||
- --ref=installer-v1.23.0
|
||||
- --ref=installer-v1.24.0
|
||||
- --root=/git-scripts
|
||||
- --link=current
|
||||
- --period=30s
|
||||
@@ -369,7 +369,7 @@ spec:
|
||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||
args:
|
||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||
- --ref=installer-v1.23.0
|
||||
- --ref=installer-v1.24.0
|
||||
- --root=/git-scripts
|
||||
- --link=current
|
||||
- --one-time
|
||||
|
||||
@@ -1,3 +1,55 @@
|
||||
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
|
||||
|
||||
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
|
||||
so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the
|
||||
box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing
|
||||
Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the
|
||||
operator's inbox. Every week, on both boxes, about a backup that had already succeeded.
|
||||
|
||||
**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay
|
||||
write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was
|
||||
not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes
|
||||
`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already
|
||||
reads as "never prune from the box" — no agent change needed.
|
||||
|
||||
**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and
|
||||
`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||
and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing
|
||||
`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling
|
||||
the client-side prune without that check would have traded a weekly false alarm for unbounded growth.
|
||||
|
||||
A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back
|
||||
quietly. The local tier's retention is untouched.
|
||||
|
||||
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
|
||||
|
||||
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
|
||||
storage and calls `felhom-backup-target-apply grant` in the same breath — a box that builds its own
|
||||
target has always been fine. The **Scenario-F arm** — *"the target already exists, leave it exactly as
|
||||
it is"* — returned without ever granting.
|
||||
|
||||
So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
|
||||
surviving a reinstall) ended up with `local_backup_target: felhom-backup` while its token held
|
||||
`FelhomAgentStore` on only `local`, `local-lvm` and `felhom-pbs`. Measured on **both** demo boxes
|
||||
2026-08-03: the content API answers `{"data":[]}` through the agent's token while root lists three
|
||||
archives. That tier was invisible to the agent and never restore-tested — and nothing said so,
|
||||
because an empty listing is also what a brand-new tier returns.
|
||||
|
||||
The reuse arm now ensures the ACL through the same guarded wrapper, so both arms leave the box in the
|
||||
same state. **Scenario F is unviolated:** the storage DEFINITION is still untouched — granting the
|
||||
role the agent is supposed to have on the target this script is about to write into `agent.json` is
|
||||
finishing the job, not retargeting the box. `pveum acl modify` is idempotent, so a box that already
|
||||
has the grant is unchanged and a box whose token was rotated gets it back.
|
||||
|
||||
`$BACKUP_TARGET_ID` is deliberately **not** added to `PVE_STORAGES`, and the comment now says why: that
|
||||
list is granted in step 4/5, before the target has been resolved in step 6, and `--acl-storages`
|
||||
entries are preflight-checked for existence. The grant belongs with the resolution, which is where it
|
||||
already was for a newly created target.
|
||||
|
||||
**A gate now asserts it** (`hostinstall_gates.py`): every arm of `configure_backup_target` that
|
||||
resolves the target must also grant on it. Red-proved by reverting the reuse arm — `resolves the
|
||||
backup target in 2 place(s) but grants in only 1`.
|
||||
|
||||
## v1.23.0 — the installer is published, not pushed (2026-08-03, R-110 + R-183)
|
||||
|
||||
**Two channels moved off `main` in the same change, because either one left behind makes the other
|
||||
|
||||
@@ -184,7 +184,7 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_VERSION="1.23.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||
SCRIPT_VERSION="1.25.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||
# The hub used to carry a copy for its Setup tab; R-94 DELETED it
|
||||
# (2026-08-02) because the hub cannot know which version a box runs —
|
||||
# the Setup command fetches this script at run time. scripts/
|
||||
@@ -307,6 +307,14 @@ PVE_PRIVS_STORE="Datastore.Allocate Datastore.AllocateSpace"
|
||||
PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"
|
||||
# Storages the agent reads/writes (archive+dump=local, restore=local-lvm, offsite DR=felhom-pbs). The
|
||||
# offsite felhom-pbs MUST be included or the agent's DR backup 403s (SPIKE residual #1). --acl-storages overrides.
|
||||
#
|
||||
# `$BACKUP_TARGET_ID` is deliberately NOT in this list, and the reason matters because the obvious
|
||||
# tidy-up is to add it (R-185, 2026-08-03). This list is granted in STEP 4/5, before
|
||||
# configure_backup_target has run in STEP 6 — so at this point the target may not exist yet, and
|
||||
# --acl-storages entries are preflight-checked for existence. The target's grant therefore belongs
|
||||
# with the target's RESOLUTION, where it already is for a freshly created one, and now also for a
|
||||
# pre-existing one. Adding it here would grant on a storage that may not exist and would still leave
|
||||
# the resolution path as the single owner of that decision, split across two places.
|
||||
PVE_STORAGES=(local local-lvm felhom-pbs)
|
||||
# E-2: the whole-guest backup target storage id, and what configure_backup_target resolved to.
|
||||
# BACKUP_TARGET_RESOLVED feeds agent.json's local_backup_target; "local" means DEGRADED (Case B).
|
||||
@@ -652,8 +660,31 @@ configure_backup_target() {
|
||||
# existing id is an error, and repointing a live target is exactly the silent retarget this
|
||||
# whole arc closes.
|
||||
if pvesm status --storage "$BACKUP_TARGET_ID" >/dev/null 2>&1; then
|
||||
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving it exactly as it is (Scenario F)"
|
||||
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving its DEFINITION exactly as it is (Scenario F)"
|
||||
BACKUP_TARGET_RESOLVED="$BACKUP_TARGET_ID"
|
||||
# R-185: …but STILL ensure the ACL. "The storage already exists" says nothing about whether
|
||||
# the agent may READ it, and this early return is where the two came apart.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES, measured on both demo boxes 2026-08-03. The CASE A path below
|
||||
# creates the storage and grants in the same breath, so a box that built its own target is
|
||||
# fine. A box whose target ALREADY existed — created by the vzdump-target-move runbook, or
|
||||
# surviving a reinstall — returned here and never granted. The result: `local_backup_target`
|
||||
# pointed at `felhom-backup` while the token held FelhomAgentStore only on local, local-lvm
|
||||
# and felhom-pbs, so the API answered `{"data":[]}` for that storage while root saw three
|
||||
# archives. The tier was invisible to the agent and never restore-tested, and nothing said so
|
||||
# — because an empty listing is also what a brand-new tier returns.
|
||||
#
|
||||
# Scenario F is UNVIOLATED: the storage definition is still untouched. Granting a role the
|
||||
# agent is supposed to have on the target this same script is about to write into
|
||||
# agent.json is not "touching the box's target", it is finishing the job. `pveum acl modify`
|
||||
# is idempotent, so a box that already has the grant is unchanged, and a box that had its
|
||||
# token rotated gets it back.
|
||||
if $DRY_RUN; then
|
||||
log_dry "felhom-backup-target-apply grant $BACKUP_TARGET_ID # R-185: ACL on a pre-existing target"
|
||||
else
|
||||
/usr/local/sbin/felhom-backup-target-apply grant "$BACKUP_TARGET_ID" \
|
||||
|| die "backup target grant failed on the pre-existing target — the agent could not read its own backup tier (R-185)"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
local mp
|
||||
@@ -2448,8 +2479,30 @@ for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agen
|
||||
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
|
||||
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
|
||||
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
|
||||
# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator
|
||||
# ruling 2026-07-26).
|
||||
# would guarantee that one of them is wrong.
|
||||
#
|
||||
# THE OFFSITE TIER CARRIES NO CLIENT-SIDE RETENTION, AND THAT IS THE POINT (R-191, 2026-08-04).
|
||||
# It used to be written `keep_last: 2` ("two weeks", operator ruling 2026-07-26). **R-89 then moved
|
||||
# offsite pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay write-only,
|
||||
# deliberately, so that a box can never delete its own offsite history. The 2026-07-26 ruling was not
|
||||
# reversed; where it is ENFORCED moved, and this value did not follow.
|
||||
#
|
||||
# The consequence was weekly and wrong in the worst direction: vzdump UPLOADED the snapshot fine and
|
||||
# then failed the whole job on the prune the token is refused —
|
||||
# `prune 'ct/9201': permission check failed - missing Datastore.Modify|Datastore.Prune`
|
||||
# → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the operator's inbox. Every week, on
|
||||
# both boxes, about a backup that had already succeeded. A tier that cries wolf weekly is a tier
|
||||
# whose real failure nobody will see.
|
||||
#
|
||||
# VERIFIED BEFORE CHANGING IT (2026-08-04, read-only on ep0): prune jobs `prune-demo-felhom` and
|
||||
# `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||
# and have run EVERY DAY since 2026-07-27 — 18 tasks, all `status=OK`, the newest showing
|
||||
# `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Retention
|
||||
# happens; it happens THERE. **If that ever stops being true, this zero is unbounded growth** — check
|
||||
# ep0's prune jobs before assuming the offsite tier is retained.
|
||||
#
|
||||
# `keep_last: 0` means "never prune from the box" and is the value the agent's own guard reads
|
||||
# (allowPBSPrune = !primary && keep_last > 0), so no agent change is needed to honour it.
|
||||
#
|
||||
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
|
||||
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
|
||||
@@ -2460,7 +2513,7 @@ base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_sec
|
||||
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
|
||||
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
|
||||
# writing to an offsite datastore.
|
||||
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]})
|
||||
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":0}]})
|
||||
base.setdefault('local_api', {})
|
||||
base['local_api'].setdefault('enable', True)
|
||||
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
|
||||
|
||||
@@ -202,6 +202,63 @@ if wp is not None:
|
||||
fail("manifests/webpage.yaml no longer tracks main for the website — pinning the SITE to "
|
||||
"the installer tag turns every copy edit into a release")
|
||||
|
||||
# ── R-185: every path that RESOLVES the backup target must also grant on it ──────────────────
|
||||
#
|
||||
# THE DEFECT THIS WOULD HAVE CAUGHT, measured on both demo boxes 2026-08-03. `configure_backup_target`
|
||||
# has two arms. The CASE A arm creates the storage and grants in the same breath. The Scenario-F arm —
|
||||
# "the target already exists, leave it alone" — returned WITHOUT granting, so a box whose target
|
||||
# pre-dated the install pointed `local_backup_target` at a storage its own token could not read. The
|
||||
# API answered `{"data":[]}` while root saw three archives, and nothing said so, because an empty
|
||||
# listing is also what a brand-new tier returns.
|
||||
#
|
||||
# The assertion is deliberately about the FUNCTION, not about PVE_STORAGES: the target's grant belongs
|
||||
# with the target's resolution (PVE_STORAGES is granted a step earlier, before the target exists), so
|
||||
# what must hold is that no arm of that function can resolve a target and skip the grant.
|
||||
fn = re.search(r'^configure_backup_target\(\)\s*\{(.*?)^\}', src, re.S | re.M)
|
||||
if not fn:
|
||||
fail("cannot find configure_backup_target() — the backup-target ACL assertion cannot run, and a "
|
||||
"check that cannot run must never report OK (R-185)")
|
||||
else:
|
||||
body = fn.group(1)
|
||||
resolutions = len(re.findall(r'BACKUP_TARGET_RESOLVED="\$BACKUP_TARGET_ID"', body))
|
||||
grants = len(re.findall(r'felhom-backup-target-apply grant', body))
|
||||
if resolutions == 0:
|
||||
fail("configure_backup_target no longer resolves BACKUP_TARGET_ID anywhere — re-read it")
|
||||
elif grants >= resolutions:
|
||||
ok("every arm that resolves the backup target also grants on it (%d resolution(s), %d grant(s))"
|
||||
% (resolutions, grants))
|
||||
else:
|
||||
fail("configure_backup_target resolves the backup target in %d place(s) but grants in only %d "
|
||||
"— an arm resolves a target the agent may not READ. That is R-185: the tier's archives are "
|
||||
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
|
||||
"like a brand-new tier." % (resolutions, grants))
|
||||
|
||||
# ── R-191: the OFFSITE tier must not arm a client-side prune ─────────────────────────────────
|
||||
#
|
||||
# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay
|
||||
# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a
|
||||
# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator
|
||||
# was told the offsite backup had failed when it had succeeded.
|
||||
#
|
||||
# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched
|
||||
# and must stay untouched — it prunes correctly and is allowed to.
|
||||
m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S)
|
||||
if not m:
|
||||
fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention "
|
||||
"assertion cannot run, and a check that cannot run must never report OK (R-191)")
|
||||
else:
|
||||
targets = m.group(1)
|
||||
kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets)
|
||||
if not kl:
|
||||
fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 "
|
||||
"(R-191: 0 means 'never prune from the box'; absent is not the same statement)")
|
||||
elif kl.group(1) != "0":
|
||||
fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite "
|
||||
"pruning server-side to ep0 and box tokens are write-only, so every weekly run will "
|
||||
"upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1))
|
||||
else:
|
||||
ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)")
|
||||
|
||||
print()
|
||||
if fails:
|
||||
print("hostinstall gates: %d FAILURE(S)" % len(fails))
|
||||
|
||||
Reference in New Issue
Block a user