23 Commits

Author SHA1 Message Date
admin 51871a7ea6 installer 1.25.0: the off-site tier stops asking to prune (R-191)
gates / gates (push) Successful in 8s
Every weekly off-site run uploaded successfully and then failed the job on a prune
the box's token is deliberately refused — R-89 moved off-site pruning server-side
to ep0 and box tokens stay write-only. The 2026-07-26 'two weeks' ruling was not
reversed; where it is enforced moved, and keep_last: 2 did not follow. Now 0,
which the agent's existing guard already reads as 'never prune from the box'.

Verified read-only on ep0 before changing it: both namespaces have a prune job at
03:30 keep-last 2 that has run every day since 2026-07-27 — 18 tasks, all OK, the
newest keeping exactly two. Without that check this would have traded a weekly
false alarm for unbounded growth.

A gate asserts the offsite tier carries no client-side prune. The local tier is
untouched.
2026-08-04 09:46:09 +02:00
admin 33f0ab2f33 R-193: demo-hp offsite restored on the box; escrow correction recorded
gates / gates (push) Successful in 7s
Re-issue through the designed endpoint: fresh password staged 07:11:51, settle-gate
GO, consumed 15s later at 07:12:06, offsite configured at 07:12:09 against the SAME
sub-account (275124) and repo path. Both boxes now escrowed.

CORRECTION to this session's own reading: the escrow did NOT recover by itself.
escrow_state went pending -> escrowed 15s after the apply and I inferred automation;
the operator had run the ceremony. It needed a human on both boxes — demo-hp at
07:16:02, demo-felhom at 07:20:28 (its escrow had been pending since its own rebuild
the day before). A 15-second state change is not evidence of automation, which is the
same class as reading an absent log line as success.

Snapshot continuity is NOT established: both boxes report snapshot_count 0, but the
run-history keys are ABSENT rather than zeroed — a controller that has never run an
offbox backup, not an empty repo. The 04:15 run decides it.
2026-08-04 09:23:08 +02:00
admin 2584dfb938 R-193: a guest rebuild silently drops the offsite tier; R-192 cause established
gates / gates (push) Successful in 8s
Operator confirms no hub-side offsite config change, so the regression was not an
action. Evidence: demo-hp's controller went 0.187.0 -> 0.192.0 at 06:12:18 with a
new config hash and the agent re-keyed its leaf three minutes earlier — a guest
rebuild. The last pre-rebuild report shows the tier fully healthy: escrowed, last
success 02:16:39Z, 15 snapshots, 40.9 MB. No offsite object in the 108 reports
since.

Mechanism: the restic credential is delivered once. demo-hp consumed its secret on
2026-07-23; the rebuilt controller has no copy and no way to request another.
demo-felhom survived the SAME rebuild only because its secret was still unconsumed
— it consumed it four seconds after its config hash changed and was reporting
offsite again 76 seconds later. That difference was luck, not design.

Also sharpens R-192: the self-heal's guard refuses when any report since the
consume carried an offbox target, but that query reads the OLDEST 500 reports —
all of which predate the rebuild. Healthy history before a rebuild is not evidence
the credential still works, which is why the automation that exists for this case
declined to act.
2026-08-04 09:05:53 +02:00
admin 185661062a STATUS: back to one screen
gates / gates (push) Successful in 7s
2026-08-04 08:58:04 +02:00
admin 967ddf2f44 R-192: offsite_delivery_stuck states the opposite of what it measured
gates / gates (push) Successful in 7s
demo-hp has been emailing daily since 2026-08-03 06:12 UTC that '500 reports
since carry no offbox target — the credential is likely burned'. Measured against
the hub's own data: all 500 of those reports DO carry one. The credential was
consumed 2026-07-23 and applied successfully; the controller reported an offsite
object continuously until 2026-08-03 05:59:21, then it vanished at 06:12:19 and
has been absent for 108 reports. A regressed apply, not a burn.

Two defects. maybeEmitStuck builds its message from ReportsSinceConsume while
hardcoding 'carry no offbox target' and never consults OffsiteReportsSinceConsume,
so the recommended action targets a failure that did not occur. And maybeHeal
refuses SILENTLY on exactly that field ('regressed-apply shape -> operator's
call', a bare return), so there is a daily email with the wrong story, no heal,
and nothing saying why the heal declined.

The underlying condition is real: that box currently has no off-site copy of its
customer app data, since 08:12 CEST on 2026-08-03. What removed it is not
established.
2026-08-04 08:57:53 +02:00
admin d692c46db0 RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes
gates / gates (push) Successful in 7s
Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite
540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest,
booted, verified and destroyed itself; zero 990000 guests or volumes afterwards
and both local-lvm figures returned to their pre-run values.

Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for
the first time: never-proven sorted first, each box took its HOST tier, deferred
the offsite one, and picked it up on the next evaluation six hours later. The
host-tier proofs reached the hub through R-189's merge — demo-felhom's report
carries two tiers, and the local one can only have come from disk.

The capability map's optimistic half is cashed, with its scope stated: these two
boxes, not the fleet.

Surfaced and filed rather than fixed:
- R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with
  a reinstall, any logged pveum activity and any cluster-log entry ruled out.
- R-191: every weekly offsite backup uploads successfully and then fails the job
  on a prune the box is deliberately not allowed to do (R-89 moved it
  server-side; both boxes still arm keep_last=2).

Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the
write path (six, with the hub raising whole_guest_backup_failed at the first), and
my earlier "no restore_test_* events" was produced by grepping a 404 page.
2026-08-04 08:13:43 +02:00
admin b93ee06abc R-190 filed; two corrections to yesterday's R-185 record
gates / gates (push) Successful in 8s
CORRECTION 1 — the runbook annotation and the R-185 row both said the drift did
not surface as a 403 because writes go through a root path. That is WRONG.
demo-felhom's local-api backup jobs 403'd six times between 09:24 and 17:34 CEST
on exactly that storage and privilege, and the hub raised whole_guest_backup_failed
at the first with edge-triggering suppressing the rest. The impact was not only an
unreadable tier: the agent's own whole-guest backups to it were failing.

CORRECTION 2 — on this box the grant was LOST, not never issued. A vzdump by the
agent's token to that storage completed OK at 04:44:50 the same morning; the first
403 is 09:24:56. Ruled out by measurement: a host reinstall (uptime 12 days), any
pveum/ACL/user.cfg activity in syslog 04:00-10:00, any ACL entry in the cluster
log. Correlated but not established: guest 9201 was reprovisioned nine minutes
before the first failure.

R-190 files the unexplained disappearance, and notes that the new store-grant
probe detects the STATE but says nothing about the TRANSITION.
2026-08-03 20:11:09 +02:00
admin e3187c86d5 docs: R-185 closed — the silence as well as the grant
gates / gates (push) Successful in 8s
- OPEN-ITEMS: R-185 closed with the measurement, the corrected root cause (the
  installer's Scenario-F reuse arm, not PVE_STORAGES), and the live sequence.
  Records that demo-hp carried the same drift and was fixed too.
- capability map: the whole-guest row's HOST-tier half was OPTIMISTIC and now
  says so — that tier was not merely unproven, it was unprovable on both demo
  boxes, and every live proof cited was on the offsite tier.
- vzdump-target-move runbook: its item 5 predicted this; annotated (not
  rewritten) with what actually happened — the create arm did grant, the reuse
  arm did not, and it surfaced as a silent unreadable tier rather than the 403
  the item expected, because vzdump writes through a root path.
- CONTEXT: S-21 (an empty listing cannot distinguish forbidden from newborn; the
  measured trap that an ungranted path answers with INHERITED privileges) and
  S-22 (the Scenario-F arm must finish the job).
- STATUS: rewritten for the operator, back to one screen.
2026-08-03 19:02:25 +02:00
admin 311dc06c13 manifests: /scripts/ syncs installer-v1.23.0 -> installer-v1.24.0 (R-185)
gates / gates (push) Successful in 8s
Both refs — the git-sync sidecar and its init container. A fresh pod must not
serve a different installer from a running one.
2026-08-03 18:59:05 +02:00
admin 688470c945 installer 1.24.0: a PRE-EXISTING backup target is granted too (R-185)
gates / gates (push) Successful in 7s
configure_backup_target has two arms and only one granted. Case A creates the
storage and grants in the same breath; the Scenario-F arm ('the target already
exists') returned without granting. A box whose felhom-backup pre-dated the
install therefore pointed local_backup_target at a storage its own token could
not read — measured on BOTH demo boxes: {"data":[]} through the token while root
lists three archives. That tier was never restore-tested and nothing said so,
because an empty listing is also what a brand-new tier returns.

The reuse arm now ensures the ACL through the same guarded wrapper. Scenario F is
unviolated: the storage DEFINITION is untouched, and pveum acl modify is
idempotent. BACKUP_TARGET_ID is deliberately NOT added to PVE_STORAGES — that
list is granted a step before the target is resolved, and --acl-storages entries
are preflight-checked for existence; the comment now says so.

A gate asserts it: every arm that resolves the target must also grant on it.
Red-proved by reverting the arm.
2026-08-03 18:58:49 +02:00
admin 7a5694341d REPORT: point at the agent report for the later 2026-08-03 session (docs-only here)
gates / gates (push) Successful in 9s
2026-08-03 16:49:17 +02:00
admin 81d4438630 docs: R-189, R-188 and R-186 closed (agent v0.122.0); R-185 stays open
gates / gates (push) Successful in 8s
- OPEN-ITEMS: all three rows closed with their measurements. R-189 records the
  migration consequence seen live (a pre-R-189 record has no tier and is not
  reportable, so upgrading does not retroactively make an old proof visible).
  R-188 records both directions demonstrated and the fixture cleaned up. R-186
  carries the before/after sha values including the CGO_ENABLED discrepancy.
  The priority list states explicitly that R-185 is the one left open from that
  group and that this session did not touch it.
- capability map: the restore-proof row now says the evidence path itself had a
  gap — the proof reached the hub only because no restart intervened — and what
  closed it.
- CONTEXT: S-19 (a proof is durable and reportable, a failure is neither, and
  why) and S-20 (the release ordering, what each step protects, and how a binary
  is verified).
- STATUS: rewritten for the operator, back to one screen.
2026-08-03 16:46:49 +02:00
admin c9a3e48b21 R-189: a passing restore-test can be invisible to the hub — observed live
gates / gates (push) Successful in 8s
The agent reports restore_tests[] from an IN-MEMORY store ('lost on restart; the
cadence re-populates'), and the hub report interval is 900s. A restart inside
that window drops the result. That used to self-heal within 24h because the next
cadence re-tested the tier; under per-archive due-ness the agent will NOT re-test
a proven archive, so the hub can stay ignorant until the next archive generation
— a week on the offsite tier.

Observed today: the 15:25:14 PASS on felhom-pbs (a real 14.5 GB offsite restore)
reached no host-report at all, because the agent was restarted 2m43s later for
the v0.121.1 deploy.

The persisted proof already exists — RestoreTestState.Snapshot() even carries the
comment 'for the host-report gauge' and has NO production caller: a seam built and
never wired, and an invariant asserted in a comment with no test, in one method.
2026-08-03 15:35:17 +02:00
admin 0779c5db85 REUSE: the restore-proven window is per tier, and observation may only widen it (R-86)
gates / gates (push) Successful in 7s
2026-08-03 15:33:19 +02:00
admin 0f8f084817 R-188: a correct agent release emails a CI failure ~50% of the time
gates / gates (push) Successful in 8s
on: [push] fires the gates workflow on the TAG push too, and release-agent.sh
pushes the tag before publishing (deliberately). CI can therefore run the
published-versions gate inside the window where the tag exists and the package
does not, and correctly reports 'every released agent version must be
INSTALLABLE' for a release that completes seconds later.

Measured across two releases in one session: v0.121.0 -> runs 12/13 (success,
failure) on the same sha; v0.121.1 -> runs 17/18 (failure, success) on the same
sha; and one pair both green — a race, not a rule.

It matters because R-168 made CI email on failure so a red gate cannot be missed.
A signal that cries wolf on every second correct release is how that mail becomes
something you archive unread.
2026-08-03 15:29:59 +02:00
admin c102832892 docs: R-86 closed and proven live; ep0 recorded as protected; R-185/186/187 filed
gates / gates (push) Successful in 8s
- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal
  reading is never true on a daily tier); R-87 re-ranked UP because R-86 built
  most of what it waited for; R-185 (the agent cannot list demo-felhom's host
  backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's
  sha is not reproducible from its tag), R-187 (R-115's publish leg had never
  actually run) filed. R-184 was the highest ID in use.
- ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row
  itself proposed — which would have been the never-fires version.
- 07-backup-architecture: new contract section — restore-testing is per ARCHIVE
  GENERATION, with the trap and what did not change (S-1).
- 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on
  the 635 s due-triggered offsite run, with the restart and teardown evidence.
- CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and
  S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines).
  Numbered 17/18 because S-14 and S-15 were already duplicated in the file.
- STATUS: rewritten for the operator, trimmed back to one screen.
2026-08-03 15:28:32 +02:00
admin ff2655cf19 manifests: hub 0.91.0 -> 0.91.1 (R-86: observation may only widen a tier's window)
gates / gates (push) Successful in 7s
2026-08-03 15:18:00 +02:00
admin 046df303b6 hub v0.91.1 — observation may only WIDEN a tier's window, never tighten it (R-86)
gates / gates (push) Successful in 7s
Found by checking v0.91.0 against the live box, not by review. demo-felhom's two
retained PBS snapshots sit 8h54m apart (one is a healing artefact), so the
mean-gap estimator reads a WEEKLY tier as nine-hourly: x4 = 36h, the 7-day floor
lifts it to 168h, and a weekly tier proved weekly reaches ~8.25d of proof age.
The false alarm this task exists to prevent would have returned within a week, on
the box it had just shipped to.

restoreProvenWindow now takes max(observed, declared). A gap SHORTER than the
declared rhythm is routine and means nothing (a retry, a manual run, a heal, a
catch-up); a gap LONGER than it is real information. Cost stated: a tier running
faster than its declared rhythm gets a slower stale signal — the right direction
for a signal that means 'unverified', since 'broken now' is a different event.
2026-08-03 15:16:59 +02:00
admin 687fedd8ee manifests: hub 0.90.1 -> 0.91.0 (R-86 Part 2, per-tier restore-proven window)
gates / gates (push) Successful in 7s
2026-08-03 15:07:52 +02:00
admin 323f45a5ef hub v0.91.0 — the staleness window learns each tier's own rhythm (R-86 Part 2)
gates / gates (push) Successful in 7s
Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.

restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
  (pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
  2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
  backup-freshness checker already uses) when history is too short to observe
  one; falling back to the FLOOR would recreate the false alarm on a fresh box

Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).

Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and
records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.
2026-08-03 15:03:35 +02:00
admin e34b614e5b docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected
gates / gates (push) Successful in 7s
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The
hub's notification_log for the run reads: two per-app failures RECORDED, one
digest SENT naming both, and the customer channel SKIPPED with operator_only.
Against the measured previous behaviour — two failures, one email naming one
app, one leaving no trace anywhere.

Scenario D proved itself on an event I had not planned: disk_critical alarmed on
two filesystems, the second was collapsed by the cooldown, and that collapse is
now visible WITH ITS KEY. Yesterday it would have left nothing at all.

A gap the spec did not anticipate is recorded with its fix: the per-app event
also fires from the periodic sweep, outside any run, so making it record-only
would have created a NEW silence. The sweep emits a digest too, with no run_id,
so it stays under the ordinary hourly cooldown.

ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile
SURVIVED the resize and is active (checked, because a resize is a stop/start).
The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere.

Five documents corrected — three of which the task's list did not name, found by
searching. Two audit/evidence documents ANNOTATED, body untouched: they record
what was true when written and that is their value.

R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable,
not unbounded — the original OOM was a 14.46 GB restore — so the restore-test
cadence should still be paced, just not by fear of the endpoint.

target-selection.md's "D-d did not name ep0 either way" is deliberately left
standing. It is the operator's question, not CC's.

STATUS.md 127 -> 83 lines, items rather than sentences.
2026-08-03 14:00:54 +02:00
admin f21e7caed1 hub v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (R-182)
gates / gates (push) Successful in 7s
Found by reading the first REAL digest, not by design. Every app row ended with
the same usage clause the mail already prints once on its own Filesystem line.
On a two-app box that is untidy; down a list of a dozen it is the same forty
characters twelve times, pushing the part that DIFFERS off a phone screen at
07:00 — the only moment this mail has to work.

The reserve's refusal message is authored for a single-app alert where naming
the filesystem is right, so the message is unchanged; the digest trims the
duplicate when rendering. trimRepeatedUsage removes ONLY an exact
"— <target path>:" suffix, so an unrelated reason is untouched and a reason that
is nothing but the usage clause is left alone rather than emptied.

Also updates TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer, which
required the OPERATOR to be emailed a per-app capture failure. That was correct
when the event was the only signal and is wrong now that it is the record and
the digest is the notification. Its customer-safety claim is unchanged and is
why the test still exists; the operator assertion is inverted with the reasoning
written in place, and R-158's guarantee is shown to have MOVED, not weakened.
2026-08-03 13:54:02 +02:00
admin dd40f85bb8 hub v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (R-182)
gates / gates (push) Successful in 7s
processOperator's cooldown no longer returns bare. It dropped the event BEFORE
LogNotification, so a suppressed operator alert and an event that never happened
were indistinguishable — from the operator's side and from the hub's own records.
Measured 2026-08-03: nine recovery_unit_capture_failed events arrived, two were
mailed, seven left no row anywhere. That is why the defect took a day to get the
right way round: there was nothing to read.

A suppressed operator event now writes a `suppressed` row carrying the message
and the key that suppressed it. This applies to EVERY operator event, not only
the one that exposed it. It does NOT change the cooldown's duration or semantics.

backup_run_failures: the per-run digest. In allowedEventTypes AND in
operatorOnlyEvents — allowlisting alone does not make an event operator-only,
and FormatCustomerEmail falls back to the raw English message rather than
blocking. A test demonstrates a customer with the type enabled receiving nothing.

recordOnlyEvents: a third routing class — stored and recorded, never mailed.
recovery_unit_capture_failed moves here: it is the record, the digest is the
notification. A register rather than downgrading severity to info, which would
relabel a genuine failure as informational everywhere it is queried.

cooldownRunSuffix: a sibling of cooldownTierSuffix, not a branch inside it, so
tier keeps byte-identical semantics and R-97a's tests are untouched. It makes
the cooldown effectively inert for the digest, which is the intent — a digest is
already rate-limited by construction; the refresh sweep sends no run_id and so
stays under the ordinary hourly cooldown.

The email renders as a list, not a JSON blob. An absent space reading renders as
unavailable, never as zeros.
2026-08-03 13:46:48 +02:00
30 changed files with 1800 additions and 345 deletions
+169
View File
@@ -17,6 +17,148 @@
## Standing rulings
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
(2026-08-04). Scope: those two boxes, not the fleet.**
Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite
**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest,
booted, verified and destroyed itself.
*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE
tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and
not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an
archive there to test.
*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven
sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next
evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together,
unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path
carrying a host-tier entry for the first time.
*The asymmetry worth remembering:* a host-tier restore is **83109 s**; an offsite one is
**300540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove.
**S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**
*The defect.* On both demo boxes the agent's token had `FelhomAgentStore` on `local`, `local-lvm` and
`felhom-pbs` and **not** on `felhom-backup` — the storage the same installer configured as
`local_backup_target`. The content API answered `{"data":[]}` through the token while root listed
three archives. `pickForThisRun` skipped the tier as *"no settled archive yet"*, which is exactly
what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.
*The rule.* The permission question has a definite answer where the listing does not. `Permissions`
reads `/access/permissions?path=/storage/<target>` **as the agent's own token** — asking as root
answers a different question and always says yes — and one `capability.Status` per configured tier
reports it. The probed set comes from `BackupTiers()`, never a fixed list: a hardcoded probe list is
the defect reproduced inside the fix.
*The measured trap, because the obvious reading is wrong.* An ungranted path answers **neither empty
nor 403**: it carries the privileges inherited from the box-wide `/` grant
(`Sys.Audit, SDN.Use, Datastore.Audit`). Testing for path-presence, or for `Datastore.Audit`, reports
a blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**, and re-measuring is required
before that constant is ever changed.
*Criticality, weighed once.* Critical, because the hub alerts only on critical and a non-critical
entry would ride the report and alert nobody. **Except** the `local` fallback target, which
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
does not page — turning an ordinary documented setup into an alert is how a signal becomes something
an operator archives unread. It never consults content, so it cannot alarm on a newborn tier by
construction, and it never reports ok when it could not ask.
**S-22 — the installer's Scenario-F arm must finish the job, not just leave the definition alone
(2026-08-03, R-185).** `configure_backup_target` has two arms. Case A creates the storage and grants
in the same breath. The reuse arm — *"the target already exists"* — returned **without granting**, and
that, not `PVE_STORAGES`, is where the drift came from: a box whose target pre-dated the install
(the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token
could not read. The reuse arm now ensures the ACL through the same guarded wrapper. **Scenario F is
unviolated** — the storage DEFINITION is untouched, and granting the role the agent is supposed to
have on the target this script is about to write into `agent.json` is finishing the job, not
retargeting the box. `$BACKUP_TARGET_ID` stays OUT of `PVE_STORAGES`: that list is granted a step
before the target is resolved, and `--acl-storages` entries are preflight-checked for existence.
A gate asserts every arm that resolves the target also grants on it.
**S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
the design (2026-08-03, R-189; agent v0.122.0).**
*The rule.* Only successful restore-tests are written to `RestoreTestState`, and that state is what
the host report carries after a restart. Failures live only in the in-memory `backup.Store`.
*Why, in one line each.* A **success suppresses future work** — under R-86's per-archive due-check a
proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it
believes, for a whole archive generation (a week on the offsite tier). A **failure causes future
work** — a failing tier stays due and is retried at the next evaluation, so a lost failure heals
itself within one interval, while a *persisted* failure would outlive the fault it describes.
*What the report does with the two.* The collector merges them: **one entry per tier, newest by
`TestedAt` wins**. A fresh failure therefore beats a stored success (the failure is the news and
exists nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier can
never appear twice — the hub would read that as two tests.
*It refuses to lie.* A persisted record missing the archive **or** the tier produces **no entry**: an
unproven tier reading as proven would be worse than the defect this closes. Run mechanics (scratch
VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be.
**Migration consequence, seen live:** a pre-R-189 record has no tier, so upgrading does not
retroactively make an old proof visible to the hub; the tier's next real proof fills it in.
**S-20 — the release order is build → tag LOCALLY → publish → push tag, and every step protects
something (2026-08-03, R-188 + R-186).**
The tag is created before the publish, so the build and the tag describe the same commit. It is
**pushed** after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
package made `check-published-versions.py` correctly fail a *correct* release — measured on roughly
every second release, and R-168 mails those failures to the operator.
The invariant the old order protected is **asserted directly instead**: the gate now also refuses a
**published version with no tag**, as a bounded probe (frontier + patch gaps) that prints its own
coverage, because the package listing api is 401 without a token and absence cannot be enumerated.
A half-done release is loud: publish-then-failed-push dies naming the recovery command, and a failed
publish deletes the local-only tag so a retry is clean.
**A released binary is independently verifiable**`-trimpath -buildvcs=false` means the same source
yields the same bytes with or without the tag; the verification command lives in
`felhom-agent/CLAUDE.md`. Both build paths (`release-agent.sh` and `publish-agent.sh`'s fallback) use
identical flags: they differed by `CGO_ENABLED=0` and produced binaries 74 KB apart for one version.
**S-17 — restore-testing is PER ARCHIVE GENERATION, and the hub's staleness window follows each
tier's own rhythm (2026-08-03, R-86; agent v0.121.0 + hub v0.91.0).**
*The rule.* Let **A** be the newest archive on a tier that has settled for at least the settle lag
(24 h). The tier is **DUE** when A exists and **A has not already been proven**. The daemon-start
ticker survives only as the **evaluation interval** (6 h). A daily tier is proved daily on yesterday's
archive; a weekly tier weekly on its own; a tier with no archive is UNKNOWN, never a fault.
*The trap, written down so it is not reintroduced.* The literal reading of R-86 — *"due when the
newest archive is ≥24 h old"* — is **never true on a daily tier**, because a new archive resets the
newest-archive age to zero long before it reaches the lag. It would have switched restore-testing off
for the tier that matters most, silently. Red-proved at 0 runs over 5 simulated days
(`felhom-agent/internal/backup/restoretest_due_test.go`).
*What the state holds now.* `RestoreTestState` records **which archive** was proven, not just when a
tier passed — a timestamp cannot answer *"have we proven THIS archive"*. A pre-R-86 file keeps its
time (rotation ordering survives a deploy) and yields no proven archive, so each tier is due exactly
once after the upgrade.
*The old config key.* `backup.restore_test_cadence_seconds` is DEPRECATED. **Negative still disables**
verbatim; a positive value now seeds the **settle lag** only, and the daemon WARNs once at start-up
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
(default 24 h). It is deliberately NOT carried into the evaluation interval.
*The hub half is not optional.* `restoreProvenStaleAfter` was a flat 7 days **derived from the cadence
R-86 removes**, and a healthy weekly tier's proof age reaches EXACTLY 168 h just before its next
proof — it sat ON the line. `restoreProvenWindow(tier, observed, ok)` now takes the tier's own
observed archive interval × 4 generations, floored at 7 days, capped at 12 days (strictly inside the
2-week offsite retention), falling back to the tier's **declared** rhythm (`backupStaleAfter` 26 h /
`offsiteBackupStaleAfter` 8 d — the backup-freshness checker's own thresholds) when history is too
short to observe one. Shipping Part 1 alone would have produced a nightly false alarm.
**S-18 — `ep0` is Tier 2, PROTECTED (operator ruling, 2026-08-03).** D-d named two protected machines
and did not name ep0 either way; `runbooks/target-selection.md` carried the question in writing for
two days. The ruling **extends D-d's protected list to three machines**: DooPlex, Peti's cluster,
**ep0**. It is a classification, not a new set of prohibitions — destroying datastores, prune jobs,
tunnel config or nftables rules was already forbidden by what it would destroy, and the ordinary
off-site READ a restore-test performs remains permitted.
**S-13 — the `mp1` merge landed, and the variant was chosen on measurement (2026-08-03, R-165 / D-a).**
The appliance's two data volumes are one. **Variant V-c**: the volume mounts at the NEUTRAL path
`/var/lib/felhom`, and both `/var/lib/docker` and `/mnt/sys_drive` are binds of subdirectories of it.
@@ -123,6 +265,33 @@ Two rulings, one shape: something became live because someone pushed, not becaus
that bumps a version, before publishing — and a gate that fails on the normal path is one people
learn to ignore.
**S-16 — a backup run NOTIFIES ONCE and RECORDS ALWAYS, and those are different things
(2026-08-03, R-182; controller v0.194.0 + hub v0.90.0/.1).** Measured: nine per-app capture failures
reached the hub, two were mailed, seven were dropped by a cooldown whose key carries no app
identifier — *before* `LogNotification`, so they left no row anywhere.
- **The record:** `recovery_unit_capture_failed`, per app, unconditionally, now routed
**record-only** by the hub (`recordOnlyEvents`) — stored and logged every time, never competing
for an e-mail slot.
- **The notification:** `backup_run_failures`, ONE per run, listing every failed app with its leg
and reason plus the counts and free space. **A clean run emits nothing.**
- **A suppressed operator event now leaves a `suppressed` row** carrying the key that suppressed it —
for every operator type. *"We chose not to e-mail you"* and *"nothing happened"* must never look
identical; that is the whole finding, stated as a rule.
- **The periodic sweep gets a digest too, with NO `run_id`**, so it stays under the ordinary hourly
cooldown. Without it the sweep's failures would be recorded and never notified — a new silence
created while closing one. A real run's digest carries a unique `run_id` precisely so the cooldown
can never collapse a manual run into the nightly one.
- **Why the silence is safe:** the hub's deadline check raises `expected_backup_missed` from report
freshness, independently of any mail the box sends (`monitor/deadline.go:396,417`). **If that check
is ever weakened, this design loses its footing.**
- **Not taken, and why:** putting `app` in the cooldown key. It fixes the swallowing by producing one
mail per failing app — a dozen on a full disk.
**ep0 was rescaled by the operator to a CX33 (2026-08-03): 4 vCPU, 8 GB RAM, measured on the box, and
the 4 GiB swapfile survived. The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only, so no disk
figure in any runbook needed correcting. That closed **R-90** and unblocked **R-86**.
**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
+140 -167
View File
@@ -1,200 +1,173 @@
# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed
# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**)
**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.**
**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
## 1. Baselines — re-read on arrival, both matched §1
**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
| Repo | @ arrival | Version | Result |
|---|---|---|---|
| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` |
| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only |
---
## 2. Part 0 — the R-182 measurement, and it REVERSED the row
## 1. Preconditions
Filed yesterday as *"the reserve re-alerts on every status refresh"***too many** alerts, observed
at the sending end. Measured at the **receiving end**, it is the opposite.
Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db`
alone would have read stale data, the exact trap this project recorded before), freshness confirmed by
the newest `notification_log` row post-dating the session.
**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.**
| time | apps refused (events in) | operator emails out |
| # | demo-felhom | demo-hp |
|---|---|---|
| 06:40:03 | privatebin, opengist | **opengist only** |
| 08:59:46/47 | opengist, privatebin | **privatebin only** |
| 08:59:59 | privatebin, opengist | **none** |
| 09:03:00 | opengist | **none** |
| 09:07:06 | privatebin, opengist | **none** |
| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
**Cause, confirmed at source:** the operator cooldown key is
`customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded).
`RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key
holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the
next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any
channel** and cannot be audited afterwards.
This is **R-97a's failure mode in a second event type**; that row's own comment states it
(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was
written narrow on purpose; `recovery_unit_capture_failed` simply never opted in.
**A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app,
HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the
operator was told about each app* — which is false. The gap between an accepted event and a sent
email is the whole of this row.
**Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape.
## 3. Probes
| | Question | Method | Verdict |
|---|---|---|---|
| **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves**`update required … local:fb65202 remote:8360f94``updated successfully`, one period (~20 s) |
| **P-B** | does Gitea serve `raw/tag/<tag>/<path>`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` |
| **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** |
**Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in
~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone
(`git ls-remote --tags` → 0 at the time).
## 4. §8.2's three channels — enumerated
| Channel | Before | After | |
|---|---|---|---|
| 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED**`webpage.yaml` split into two syncs |
| 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below |
| 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission |
**Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.**
There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so
no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"*
was impossible for them. Raised before building; operator ruled to pin them to **the agent version
being installed**, which the installer already resolves from the hub manifest and already sha-verifies.
That is strictly better than any installer-derived tag: binary and configs now come from one ref.
**Channel 3 needed no change because the URL never carried a ref** —
`https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template
change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose.
## 5. The tag convention
- **Shape:** `installer-v<SCRIPT_VERSION>` in `felhom.eu` (prefixed so it cannot be read as a hub,
agent, controller or golden version); `v<semver>` in `felhom-agent` (that repo versions one thing).
**No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time,
and channel 1's ref lives only in the manifest.
- **Publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (sidecar *and*
init container), commit, sync.
- **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**.
## 6. Scenario A — proven by HTTP
A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and
three sync periods were allowed to pass so "unchanged" means "had every chance to change":
## 2. The due verdicts before the run, quoted
```
website tree (main): .worktrees/6a82719… <- ADVANCED to the new commit
scripts tree (tag): .worktrees/bee6848… <- STAYED
sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
sha256 after push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
marker present at the served URL? 0
https://felhom.eu/ -> HTTP 200
demo-felhom tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst"
reason: newest settled archive … has not been proven; nothing proven on this tier yet
tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z"
reason: … has not been proven (last proven archive was a different one)
demo-hp tier=felhom-backup due=true archive="…2026_08_02-04_49_29.tar.zst"
tier=felhom-pbs due=true archive="…2026-07-28T19:19:45Z"
```
Both halves of the split in one observation: the site still tracks `main`, the installer does not.
**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
## 7. Scenario B — publish and rollback, both directions
## 3. The runs — all four SCHEDULED, none triggered
| act | result |
|---|---|
| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** |
| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` **exactly** the pre-publish sha; **marker gone** |
| box | tier | due at | archive | result |
|---|---|---|---|---|
| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved
to `main`'s head — a **byte no-op**, verified by the served sha not changing.
**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
which start nothing.
## 8. Files, commits, tags
**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b`
(marker removed), plus the docs commit below.
`scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` ·
`manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` ·
`documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md`
**The asymmetry worth keeping:** a host-tier restore is **83109 s**; an offsite one **300540 s**. The
tier an ordinary recovery uses is also the cheapest to prove.
**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT).
`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** ·
`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md`
## 4. What the runs left behind
**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and
`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from;
`configs/` is byte-identical there and at `main`, so nothing depended on the choice).
**Persisted state — v3, naming the archive, the tier and what was verified:**
## 9. Tests and red-proofs
```json
demo-felhom felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
proven_at 2026-08-03T22:56:45Z}
felhom-pbs → {archive …2026-07-28T04:49:43Z, tier pbs, proven_at 2026-08-04T05:04:21Z}
demo-hp felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
felhom-pbs → {archive …2026-07-28T19:19:45Z, tier pbs, proven_at 2026-08-04T06:10:39Z}
```
| Check | Result |
|---|---|
| `felhom.eu` `repo_gates.py --fast` | all 5 gates OK |
| `felhom-agent` `go build ./... && go vet ./...` | OK |
| `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
| `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) |
| `agent_gates.py` (full) | both OK |
**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
latest report holds **two** entries, one per tier; the `local` one can only have come from the
persisted state, because the in-memory store held only that morning's offsite run:
**Red-proofs, each demonstrated failing then restored:**
```
demo-felhom tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
tier=pbs pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z tested_at=2026-08-04T05:04:21Z
demo-hp tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
```
| # | Mutation | Result |
**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
```
tier=felhom-pbs due=false proven="…2026-07-28T04:49:43Z"
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
```
…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
That is not a defect: it is "proved daily, on its own archive", visible one day later.
## 5. Teardown — three layers, per box
| layer | demo-felhom | demo-hp |
|---|---|---|
| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired |
| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it |
| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state |
| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 |
| F | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 |
| the machine | `pct list` **0** entries for 990000 | **0** |
| the host | `lvs` **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit*
`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also
retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was
this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API
token, and the only credential available (`~/.docker/config.json`) is a registry password that the API
rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.
Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
## 10. No version bumps, nothing built
## 6. What the run surfaced — three findings, two of them corrections to my own record
`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file
touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's
no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or
golden was built.
### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
## 11. Register
A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:0010:00, any cluster-log ACL entry.
Correlated but not established: guest 9201 was reprovisioned at 09:1509:19, nine minutes before.
**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
and v0.123.0's probe detects the *state* but says nothing about the *transition*.
| ID | Outcome |
|---|---|
| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed |
| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) |
| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described |
| **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs |
| **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released |
### R-191 (new) — every weekly offsite backup reports FAILED although it worked
**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md`**0 rows** each;
all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0
hits anywhere and remains free.
demo-felhom, 06:4906:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
incrementally) and the job then failed on the prune —
`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
namespaces.
## 12. Observations — noticed, documented, NOT acted on
### Two corrections to yesterday's record
1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs
either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails
closed where the mistake is made, needs no new credential).
2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`,
so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside
R-182 because it is what made that row take a day to get the right way round.
3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it
means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a
network gate is ever added there, that workflow needs the same change the agent's just got.
1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
**six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
`whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
said backups kept landing because writes go through a root path — wrong, and now corrected in the
runbook and on the row.
2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
since the R-86 work — and one **more** since, below.
## 13. Teardown
**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
host tier had never been proven, and the hub said so on its own, without being asked. It has not
re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag
`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from
`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl
pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.
## 7. The capability map
The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
is untested and untouched, and one box proving something does not make it a fleet property.
## 8. Registers
- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
correction in §6.
- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
the highest in use).
- `ROADMAP.md` holds none of these rows, so nothing to collapse.
- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
"What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
## 9. Observations — noticed, NOT acted on
- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
the restore pointed at `/mnt/nvme-1tb` instead.
- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
none". Nothing depends on that assumption now, but the operations notes still carry it.
- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
role — a 2.7× difference worth understanding before either is used to size anything.
- **Both waiters this session produced no output** despite the runs completing; the evidence was
gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
the hub's database, not on the waiters.
+1 -1
View File
@@ -104,7 +104,7 @@
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. |
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
| `cfg.DRTier` + offsite coupling | hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field `dr_tier` (formBool helper). |
+55 -93
View File
@@ -14,114 +14,76 @@ A blank machine boots the Felhom disc, installs itself unattended, and is claime
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
home network, and open apps from a launcher or a shared link. Backups run on their own to three
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
restore files and app data from the drive alone. Proven end to end on real hardware.
**Apps come back after a power cut.** The machine tells an app the customer switched off from one
that simply did not come back, and waits for the system to finish starting before deciding instead of
glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every
time, and an app switched off deliberately stayed off every time.
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
box six times, everything returned every time, and an app switched off deliberately stayed off.
Proven end to end on real hardware.
## What's broken
**The off-site copy can be erased by the machine that made it** — the credential that writes it can
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
*(R-95, R-87)*
**A full disk tells you about ONE app and silently swallows the rest.** Yesterday this was written
down the wrong way round — as *too many* emails. Measuring the receiving end reversed it: of nine
refusals the machine reported today, **two emails were sent**. When two apps are refused in the same
second you are told about one of them, and the other leaves no trace anywhere — not an email, not
even a line in the log saying it was held back. So a second app can be going unbacked-up while you
have already been told the problem is handled. It is the same fault we fixed once before for
whole-machine backups, in a second place that never opted into the fix. *(R-182)*
- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on
3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued,
the machine picked the new password up in 15 seconds and reattached to the same repository, and you
escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly
once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild.
The other machine survived the same rebuild only because an unused password happened to be waiting
for it. *(R-193)*
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
this declines without saying why. The message says the password was never applied; it was, on
23 July, and worked for eleven days. *(R-192)*
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
setting per machine fixes it. *(R-191)*
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
also delete it. A daily snapshot is armed as a stopgap.
*(R-95, R-87)*
## What shipped recently
**Pushing the installer no longer publishes it.** The script that runs as root on a brand-new
machine was copied from the main branch and served within thirty seconds, so pushing it *was*
publishing it, with no staging and no way back but another push. It now comes from a **labelled**
version: publishing is moving the label, and undoing it is moving the label back — about half a
minute, no deploy. The website is untouched by this and still updates in thirty seconds, because a
typo fix must never need a release. Proven by actually doing it: a real push changed nothing that
anyone downloads, moving the label published it, moving it back restored the previous bytes exactly.
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
never been tested on either machine. Both also proved their off-site copy on the same night, one
after the other rather than at once, which is the machine deciding for itself what to do first.
*(closes the last open half of R-86/R-185)*
- **A backup copy the machine was never allowed to read — and could not tell you about**, on both
demo machines. The permission was one command; the silence was the real fault, and the machine now
checks whether it may read each copy it depends on and says so when it may not. *(R-185)*
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
written to disk with the result and survives. This was caught happening, not predicted: a real
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
**The catch that would have made it cosmetic was found and covered.** While it runs, the installer
fetches sixteen more files — not nine, and from the *agent's* repository, not the website's. They now
come from the same version of the agent the machine is installing. That closed a real fault nobody
had noticed: a new machine was getting the agent's tested program and its untested settings files, in
one install, from two different places. *(R-110, R-183)*
**Releasing the agent now publishes it, in one command.** Putting a built agent where a new machine
can download it was a step someone had to remember, and it was forgotten three times in five days —
the last time leaving both demo machines running a version nobody could download, so a rebuild would
have quietly installed the *older* one and reported success. There is now one command that builds,
labels, publishes and then **downloads it back to check** — and a check that refuses to stay quiet if
a released version cannot actually be fetched. Proven by making CI fail on purpose and then go green
again on the same code. *(R-115)*
**The backup partition is gone and both demo machines run on the new shape** — wiped, rebuilt and
taken through the whole customer journey on 3 August, by two deliberately different routes so the disk
shape and the delivery route are both proven. The space a backup can use went from 19 GB to 65 GB on
the small machine and 45 GB to 233 GB on the big one. Their previous demo apps and data are gone; that
was the point of a wipe, and you approved it. *(R-165, R-178)*
**What replaced the wall now watches the right moment.** The wall was quietly keeping a runaway
backup from eating the space the machine needs to run. As first built, that replacement was checked
too late — the big write happened first, unchecked — while still promising your last good copy was
untouched. Fixed and proven on 3 August: the machine decides once, per app, **before it writes
anything**, and that one answer covers all three steps, so a refused app writes nothing, is not
restarted, and the promise is now literally true. It also stopped being blind to size. Nothing is ever
deleted to make room. *(R-181)*
**The last of the three apps that never saved their data is fixed.** Installed nowhere, so nothing was
stranded — checked on both demo machines and in the fleet list rather than assumed. Proven by the check
that caught it, run in both directions: it clears the fixed version and still convicts the old one.
*(R-156)*
**A filling disk warns the customer before anything breaks, and a failed backup reaches you** — the
customer while there is still room to act, naming the drive and the space left; you when one app's
backup fails, with the disk figures. The customer is deliberately not told about the second: they can
free space, but they can do nothing about a failed backup. Both proven by filling a real disk. There
are two rules and not one because the serious warning fired on free space while the disk was only 91%
full — a percentage alone would have missed it. *(R-167, R-158)*
**The checks have two nets and the second emails you.** Every repository has one command that runs all
its checks, before every push. That one can be skipped, so the build server runs them again and emails
you on failure. It cannot *stop* a change — everything goes straight to the main copy with no review
step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)*
## What we're working on
- **Now:** nothing outstanding from today — the reserve, the last unsaved app, and both of your
decisions are all built and proven.
- **Next:** the alert that tells you about one app and swallows the second *(R-182)*.
- **After:** the off-site copy that the machine making it can still erase *(R-95, R-87)*.
- **Now:** nothing outstanding.
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
*(R-87)*
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
## Waiting on you
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
session log; nothing suggests anyone else saw it. *(R-132)*
- **Nothing else.** Both decisions you took on 3 August are now built and proven. One small question
will come back later: the automatic check cannot see which version you have told machines to
install, only which ones exist — closing that either needs a password given to the build server or
a check inside the hub itself. Filed, not urgent. *(R-184)*
- **One small question, not urgent.** The automatic check cannot see which version you have told
machines to install, only which ones exist. Closing that needs either a password given to the build
server or a check inside the hub itself. *(R-184)*
- **Nothing else.**
## Changed since last update
- **2026-08-03**Publishing became something you do rather than something that happens: the
installer and the agent both moved onto labelled versions with a way back, and a check now refuses
a release nobody can download. Earlier the same day: the reserve now guards the step that fills the
disk and its promise is true, and the last app whose data was never saved is fixed. All proven on
real machines, not just in tests.
Earlier the same day: both demo machines wiped and rebuilt from the new base image and taken through
set-up → install an app → back it up → restore it, with the backup space ceiling gone and measured.
- **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in
a mode where reading a page cannot block a machine's status update; a one-word difference meant that
setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found:
the hub's own database is in no automatic backup** — it holds every machine's emergency password.
Filed, not yet fixed.
- **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were
failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this
machine and the tester's box are protected, every other box may be broken or reinstalled freely.
- **2026-08-04**Both demo machines proved their on-machine backup restores, on their own,
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
that could vanish on a restart (seen happening), a release that emailed a failure for a release
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
File diff suppressed because one or more lines are too long
@@ -132,6 +132,41 @@ executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by const
(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
(INV Part D1).
### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03)
**[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to
fire on an interval started at daemon start. It no longer does. The rule is:
> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
> The tier is **DUE** when **A** exists and **A has not already been proven**.
So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather
than the process's uptime:
| tier rhythm | what is proved, and when |
|---|---|
| daily (host tier) | yesterday's archive, once a day |
| weekly (offsite tier) | last week's archive, once a week |
| newborn (no archive yet) | nothing — **UNKNOWN, never a fault** |
**The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest
archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive
age to zero long before it reaches the lag — so the literal reading silently switches restore-testing
off for the tier that matters most.
What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its
journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on
success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker
remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one
due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN).
**The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very
cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval
just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay
tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed
archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite
retention), and falling back to the tier's declared rhythm when history is too short to observe one.
### Why the split is right, stated once
**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
@@ -574,6 +609,30 @@ because that was a different filesystem. On a merged box it can. Decision **B2**
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
only destroy a different app's only local copy.
**WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
The two are deliberately different mechanisms, because conflating them is how seven failures went
missing on 2026-08-03 without leaving a trace.
| | Record | Notification |
|---|---|---|
| what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
| when | every time, unconditionally | at the end of a run, **only if something failed** |
| gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
| where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
- **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
design; weakening it re-opens a silent-failure path.
- **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
Deciding not to tell someone is itself an event worth recording.
- **Deliberate skips are not failures** and never appear in the digest — a disconnected or
decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
operator stops reading.
- **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
the hour, because someone pressing the button is actively trying to get a backup. The periodic
capture sweep is capped by the ordinary hourly cooldown.
**THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
**per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
@@ -796,7 +855,7 @@ does **not** hold as written. → **R-108**
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
| R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written |
| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
### 10.3 Divergences that are documented elsewhere and are not re-opened here
@@ -862,7 +921,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
| unattended restore-test across tiers | **IMPLEMENTED** (rotation not observed across consecutive cadences) | `00-capability-map.md:41`; LIVE per-tier timestamps this session |
| unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) |
| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
@@ -10,7 +10,7 @@
**Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only);
felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`,
`internal/hub/cloudflared.go`, `internal/escrow/identity.go`).
**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`,
**Probe ends:** `felhom-hetzner` = Hetzner CX23 (**note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged**), Debian 13.4, public IPv4 `167.233.158.164`,
global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** =
the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven
over the existing LAN SSH path; the tunnel itself always dialed **out**.
File diff suppressed because one or more lines are too long
+2 -2
View File
@@ -144,7 +144,7 @@
| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs``felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
| R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. |
| R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
| R-86 | ~~**Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup**~~ | S | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03 (agent v0.121.0 + hub v0.91.0)** | **The reasoning that produced the row still holds and is worth keeping:** R-85 scheduled on a free-running INTERVAL with oldest-proven-first rotation, which cannot express *"the day after the PBS backup"* any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. The operator's ruling was weekly per tier, ~1 day after that tier produced a new backup, so the gap between an archive being written and being proven stays small and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **What shipped is that ruling, with one correction the row's own wording would have got wrong:** the shape written here — *"test tier X once its newest archive is ~24 h old and not yet proven"* — is NEVER true on a DAILY tier, because a new archive resets the newest-archive age to zero before it reaches 24 h. The rule that works is *the newest archive that HAS settled 24 h, and has not been proven*: `A = newest archive older than the settle lag`, due iff A exists and A ≠ the proven archive. **The interim is retired:** `restore_test_cadence_seconds = 302400` (3.5 d) is no longer the mechanism; that key now only disables (negative) or seeds the settle lag. Alignment is no longer approximated by a cadence — it is the schedule. Full detail: `OPEN-ITEMS.md` R-86, `07-backup-architecture.md` §3 |
| R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
| R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | **Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change)****Part 1** added the failure breaker: consecutive failures tracked per TARGET, backoff `15m→30m→1h→2h→4h` capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to `TriggerNow`, and `stillRunning` is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. **Part 2 is agent-side and needs a wire change** — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely.
@@ -175,7 +175,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
| R-162 | **`docker diff` is the gate's only witness and its failure mode is quiet** | XS | WATCHING — 2026-08-02 | A limitation, not a defect. The gate's power is `docker diff` excluding mounted paths; on a driver where it is unsupported or lies, the gate degrades to mount-occupancy + writability **and would not say so**. It fails closed (the canary self-test stops reporting BROKEN and the gate then refuses to report), but the message blames the prober rather than the driver. Revisit only if a non-overlay driver ships |
| R-163 | **`mp1` is RETENTION, not staging — and it is sized as if it were neither** | M | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | Settled by reading: a unit is the KEPT copy on the app's **own** drive (`backup.go:245-255`), falling back to the system SSD for driveless apps — *"the SSD-only system-data fallback"* (`appbackup/paths.go:26-27`); **no post-copy deletion**, the only prune is F5 residue-on-old-drives (`backup.go:1053-1112`). So `mp1` **20 G** retains every driveless app's unit while `mp0` permits **50 G** of volumes, and a DB app's unit is up to **~2×** its data (measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to 0 (`felhom-agent/.../main.go:178`), not derived from the drive; demo-hp's real guest ships 50G/20G. **`mp1` gates the whole chain** — Tier-2 mirrors the unit *"(always)"* (`tier2.go:302,368`) and Tier-3 carries it. Bounded: a unit is volume tars + DB dumps only, **never `mp8` userdata** (`recovery_unit.go:20-25`). **No number proposed.**`07-backup-architecture.md` §7.5 |
| R-164 | **C2's chain — the DB volume tar cannot be dropped until a sound dump predicate exists** | S | BLOCKED — on the predicate (2026-08-02) | The unit holds a volume tar **and** a SQL dump and the restore uses both: the dump is authoritative and replayed after the tar so it WINS (F17), with only the DB service up (R-47) — `restore_unit.go:262-266`. Dropping the DB tar would halve DB-app units and close R-127(b)'s initdb-skip trap. **The obvious gate is dead, measured:** `ValidateDump`'s empty-`accounts` warning was **correct** (the DB truly had 0 rows; seeding one stopped the warning and put the row in the dump) — but **a fresh appliance legitimately has zero accounts**, so gating on it blocks every new customer's first backup. Order: sound predicate (dump vs **live** per-table counts) → warn→gate → tar-drop. Pairs with **R-127** |
| R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile`**swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |
| R-90 | ~~**ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86**~~ | S | **CLOSED 2026-08-03 — the operator rescaled ep0 to a CX33.** MEASURED on the box, not read from an invoice: `Mem: 7757` MB (**8 GB**, was 3.8), `nproc` **4**, and the 4 GiB swapfile added 2026-07-27 **survived the resize** and is active (checked because a resize is a stop/start, so "still there" was an assumption until measured). **The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only — so no disk figure in any runbook needed correcting. Unblocks R-86 | Runbooks corrected; SPIKE/evidence documents ANNOTATED not revised. `target-selection.md`'s "D-d did not name ep0 either way" stays open — it is the operator's |
| R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` |
| R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect |
| R-93 | **`drill-r50` is both a blocked customer and the only drift fixture** | XS | idea — 2026-07-27 | The drill customer is blocked in the hub (so it stops alarming) yet it is also the only record exercising the endpoint-drift path R-77 added. Blocking hides it from `GetActiveCustomerIDs`, so the fixture it provides is silently inert — a monitor with no live subject reads exactly like a monitor that passes. Decide: retire it and build a synthetic fixture, or unblock it and silence per-customer instead (the operator has a per-alert silencing feature planned). Related to the R-50 drill VM, now shut down |
@@ -3,7 +3,8 @@
**Class:** supervised operational run. **No repo version bump** — the only commits are this record
and the capacity note. **Nothing was deleted.**
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg.
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner **CX33 (4 vCPU / 8 GB RAM)**, Nuremberg.
> **Rescaled 2026-08-03** from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. **The 40 GB local disk did NOT change** — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize.
**Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom`**`/mnt/pbs-datastore`** (name unchanged).
**Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.
@@ -229,7 +229,9 @@ as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
minutes of ep0 CPU per weekly backup.
> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
> Watch item (**superseded 2026-08-03**: ep0 is now a **CX33, 8 GB RAM**, and it HAS a 4 GiB swapfile
> which survived the resize — so the pressure below is much reduced, though the shape of the concern
> stands). As written: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
@@ -307,7 +309,7 @@ Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13
watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
5. ~~**ep0 has no swap** (3.7 GB CX23)~~**corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile.** See the volume runbook's OOM for the original incident.
## 11. Observations
@@ -431,7 +431,7 @@ label. Filed under E-2.
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. |
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
| 8 | The honest **single-drive label**. |
+3 -2
View File
@@ -5,7 +5,8 @@
> firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM
> re-creates the endpoint from nothing (that is the DR story, step 8).
>
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23, Debian 13,
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled
> to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13,
> `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a
> later re-run of this runbook on a production VM.
>
@@ -31,7 +32,7 @@ Parameters used throughout (adjust for a new endpoint):
points at nothing (live-run finding). Home-resolver propagation can lag public DNS by
minutes — a client-side `wg-quick up` that fails to resolve right after record creation
just needs a retry.
- [ ] Sanity: `ssh root@167.233.158.164 hostname``felhom-hetzner` (the throwaway CX23), not
- [ ] Sanity: `ssh root@167.233.158.164 hostname``felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not
any production box.
## 1. Base (on the box, as root)
+19 -12
View File
@@ -29,20 +29,23 @@ prohibition as covering the act it names and nothing more.
|---|---|---|
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
thing under test, the source of truth for it, and the backups, at once.
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
not name ep0 either way, so this page carried the question in writing for two days and read it the
narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
protected list to three machines**: DooPlex, Peti's cluster, ep0.
The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
ruling makes the classification say so plainly instead of leaving each session to re-derive it.
**Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
never a drill target. The Hetzner Storage Boxes ride the same reasoning.
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
@@ -95,11 +98,15 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
enrolled host. No access route from DooPlex, and nothing here needs one.
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
**The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
(R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
customer documents and photos, on a credential that can still delete (R-95).
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
produced a false "unreachable" verdict (standing rule 2).
@@ -139,7 +139,7 @@ is not reachable, for two reasons that are each **already-recorded deliberate po
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off*
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90**note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off*
no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the
+129
View File
@@ -1,3 +1,132 @@
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
offsite tier holds two retained snapshots — `2026-07-27T19:55:41Z` and `2026-07-28T04:49:43Z`
**8 h 54 m apart**, because one is a healing artefact and the other a real weekly run. The mean-gap
estimator therefore reads a **weekly** tier as nine-hourly: ×4 gives 36 h, the 7-day floor lifts it to
168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm this whole
task exists to prevent would have returned within a week, on the box it had just shipped to.**
`restoreProvenWindow` now takes `max(observed, declared)`. Observation refines a tier's rhythm
**upward** and is ignored downward, which is right on its own terms and not merely cautious: a gap
SHORTER than the declared rhythm is routine and means nothing — a retry, a manual run, a heal, a
catch-up after an outage — while a gap LONGER than it is real information, saying this tier genuinely
receives archives less often than the model assumes and its window must widen or it alarms.
**The cost, stated rather than hidden:** a tier that truly runs faster than its declared rhythm gets a
wider window than it strictly needs, i.e. a slower `restore_test_stale` signal. That is the right
direction for a signal whose message is *"unverified"*. *"Broken now"* is `restore_test_failed`, which
is immediate and untouched.
Three live-derived cases added to `TestRestoreProvenWindow_Contract`, including the exact 8 h 54 m
gap measured on the box; red-proved by restoring the tighten-too branch
(`window(pbs, observed=8h54m) = 168h, want 288h`).
## v0.91.0 — a tier's staleness window learns the tier's own rhythm (2026-08-03, R-86 Part 2)
**This ships WITH the agent's v0.121.0, not after it.** The agent now proves a tier once per ARCHIVE
GENERATION rather than on a 24h timer, so a tier backed up weekly is proved weekly — correctly, and
in perfect health. `restoreProvenStaleAfter` was a flat 7 days, and its own comment derived that
number from the cadence R-86 removes:
> *"the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
> proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."*
A weekly tier proved weekly reaches a proof age of **exactly one interval** just before its next
proof — 168h against a 168h window, false by a hair — so it did not merely sit near that line, it sat
ON it, and any ordinary delay (a late backup, one deferral behind a running backup) tipped it into a
nightly alarm about a system that is working. Shipping the agent alone would have converted the
improvement into a false alarm.
**The window is now per tier**, `restoreProvenWindow(tier, observed, ok)`:
- **the tier's own archive interval**, observed from the host-reports the hub already holds —
`pbs_snapshots[]` plus successful `backups[]` attributed by TARGET TYPE (the slice-A.4 rule: a
PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an
offsite archive to the host tier);
- **× 4 generations** — the settle generation plus ~3 missed opportunities, deliberately the same
tolerance the flat constant expressed. The change is to the RHYTHM, not to the patience;
- **floored at 7 days** — the old constant kept as a floor, so no tier is judged more tightly than
before;
- **capped at 12 days** — strictly inside the 2-week offsite retention with two days to spare, so a
tier is never called stale against an archive PBS has already pruned;
- **falling back to the DECLARED rhythm** when a box's history is too short to observe one:
`backupStaleAfter` (26h) for the host tier and `offsiteBackupStaleAfter` (8d) for the offsite tier
— the very thresholds the backup-freshness checker already judges those tiers against. A fresh box
with one snapshot has no observable interval, and falling back to the FLOOR there would recreate
the false alarm on exactly the tier this task is about.
**Kept, because both were earned:** absence is UNKNOWN until an anchored window has passed (R-81's
structure, untouched), and the stale signal stays edge-triggered. `restore_test_failed` and
`restore_test_stale` remain DISTINCT — one says your recovery is broken, the other says it is
unverified, and the second is the one that quietly becomes the first.
**Every reason string now states the window it was judged against** — R-100's corollary: when a
verdict changes what it counts from, the alarm text has to change with it, or an operator reads
"limit 168h" under a tier actually judged at 288h and dismisses a true alarm.
**The window READ is unchanged in cost** (14 days), which is both enough to find proof inside the
widest window and enough to see two generations of a weekly tier.
## v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (2026-08-03, R-182)
**Backfilled 2026-08-03 (R-86 session).** This version was built, deployed and recorded in `REPORT.md`
and the R-182 row, but never given a CHANGELOG entry — and `REPORT.md` is overwritten every session,
so the per-repo history under-reported what was actually running. The deployed image has been
`felhom-hub:0.90.1` since `f21e7ca`.
Found by reading the first REAL digest, not by design: every app row ended with the same usage clause
the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list
of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone
screen at 07:00 — the only moment that mail has to work.
The reserve's refusal message is authored for a single-app alert where naming the filesystem is
right, so the message is unchanged and the DIGEST trims the duplicate when rendering.
`trimRepeatedUsage` removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is
untouched and a reason that is nothing but the usage clause is left alone rather than emptied.
Also inverted the operator half of `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer`: it
required the operator to be e-mailed a per-app capture failure, which was correct when that event was
the only signal and is wrong now that it is the RECORD and the digest is the notification. The
customer-safety claim is unchanged — R-158's guarantee MOVED, it did not weaken.
## v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (2026-08-03, R-182)
**The smallest change on the board with the largest effect on trust: `processOperator`'s cooldown no
longer returns bare.** It used to drop the event *before* `LogNotification`, so a suppressed operator
alert and an event that never happened were indistinguishable — from the operator's side **and from
the hub's own records**. Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two
were mailed, and **seven left no row anywhere**. That is why the defect took a day to get the right
way round: there was nothing to read.
A suppressed operator event now writes a `suppressed` row carrying the message and **the key that
suppressed it**, so the collision is readable without reading code. This applies to **every** operator
event, not only the one that exposed it. It deliberately does **not** change the cooldown's duration
or semantics — it makes the drop visible, not absent.
**`backup_run_failures` — the per-run digest.** One operator mail at the end of a backup run listing
every app that failed, its leg and its reason, with the counts and the target filesystem's free
space. Added to `allowedEventTypes` **and** to `operatorOnlyEvents` — allowlisting alone does not make
an event operator-only, and `FormatCustomerEmail` falls back to the raw English message rather than
blocking. A test demonstrates a customer with the type in their enabled list receiving nothing.
**`recordOnlyEvents` — a third routing class.** Types that are STORED and RECORDED but never mailed.
`recovery_unit_capture_failed` moves here: it is the durable per-failure record, and the digest is the
notification. Deliberately a register rather than downgrading the severity to `info`, which would have
had the same routing effect while relabelling a genuine failure as informational in the events table,
the operator UI and every historical query.
**`cooldownRunSuffix` — the run discriminator.** A sibling of `cooldownTierSuffix` rather than a
branch inside it, so `tier` keeps byte-identical semantics and R-97a's tests are untouched. It makes
the cooldown effectively inert for the digest, **which is the intent**: a digest is already
rate-limited by construction, one per run and only when something failed, so there is nothing for a
timer to collapse — while the periodic refresh sweep sends **no** `run_id` and therefore stays under
the ordinary hourly cooldown.
**The e-mail is rendered as a list, not a JSON blob** — the one operator mail with a variable-length
payload, and a dozen apps on one line is unreadable on a phone at 07:00. An absent space reading
renders as *unavailable*, never as zeros.
## v0.89.0 — the two halves of decision D-c (2026-08-02, R-167 · R-158)
**Decision D-c routes two new signals to two different audiences, and the hub is where that routing
@@ -0,0 +1,40 @@
package api
import "testing"
// R-182 — the backup run digest needs the allowlist half, and NOT the customerMessages half.
//
// A new event type is a pair of register entries, and each half fails differently. For this type the
// pair is unusual and that is the point:
//
// - missing from `allowedEventTypes` → POST /event returns 400 and the digest VANISHES,
// which would rebuild the exact silence R-182 exists to end;
// - PRESENT in `customerMessages` → the customer would be e-mailed, in Hungarian, a
// list of which apps' backups failed and why — operator detail they can take no action on. So
// this type must be allowlisted and must NOT have a customer message.
//
// The customer-facing half of decision D-c is the FILL WARNING, which fires before this and is
// actionable (free space, delete files, add a drive). This is the operator's half.
//
// Operator-only routing itself is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a
// customerMessages entry — that assumption shipped in v0.78.0 and was wrong, because
// FormatCustomerEmail falls back to the raw message. It is pinned in
// `internal/notify/backup_run_digest_test.go`, which demonstrates a customer with the type in their
// enabled list receiving nothing.
func TestBackupRunDigestIsAllowlisted(t *testing.T) {
if !allowedEventTypes["backup_run_failures"] {
t.Fatal("backup_run_failures must be in allowedEventTypes, or POST /event 400s and the " +
"whole run digest is dropped at the door — the silence R-182 was filed against")
}
}
// The per-app event is the RECORD and must not be removed while the digest is the notification.
// Deleting it would make the digest the only trace, and a digest that fails to send would then take
// the record with it — the coupling R-182's fix exists to break.
func TestPerAppCaptureEventStaysAllowlisted(t *testing.T) {
if !allowedEventTypes["recovery_unit_capture_failed"] {
t.Fatal("recovery_unit_capture_failed was removed from allowedEventTypes — it is the " +
"durable per-failure RECORD, and the digest is only the notification; the operator " +
"register and every historical query depend on it")
}
}
+5
View File
@@ -1586,6 +1586,11 @@ var allowedEventTypes = map[string]bool{
// OPERATOR-ONLY IS ENFORCED BY `notify.operatorOnlyEvents` — see the paragraph above. This entry
// alone does NOT make it operator-only.
"recovery_unit_capture_failed": true,
// R-182. The per-run backup digest: one event at the end of a run, listing every app whose
// backup failed or was refused. Allowlisting it is NOT what keeps it away from customers —
// `notify.operatorOnlyEvents` is (see the comment there); both entries ship together and
// `backup_run_digest_event_test.go` pins the pair.
"backup_run_failures": true,
// Controller-pushed events
"controller_started": true,
+195 -20
View File
@@ -32,15 +32,104 @@ import (
// is a NEW monitor written straight after the third, so it copies R-81's verdict structure rather
// than re-deriving it. A tier never proven on a newborn box is UNKNOWN, never FAILED.
// restoreProvenStaleAfter is how long a tier may go unproven before it is called stale.
// ── HOW LONG MAY A TIER GO UNPROVEN? (R-86 Part 2) ───────────────────────────────────────────
//
// Derivation, not a guess: the restore-test cadence is 24h and rotation is oldest-first across two
// tiers, so each tier is proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive
// missed opportunities before alarming — loud enough to matter, quiet enough not to fire on one
// skipped cycle (a deferral behind a long backup is normal, not a fault). It is also comfortably
// inside the 2-week offsite retention (operator ruling 2026-07-26), so a tier is never reported
// stale against an archive that is about to be pruned anyway.
const restoreProvenStaleAfter = 7 * 24 * time.Hour
// This was one flat constant, 7 days, and its comment derived that number like this:
//
// "the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
// proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."
//
// **That premise is exactly what R-86 removed.** The agent no longer tests on an interval at all: a
// tier is tested once per ARCHIVE GENERATION — when it holds a settled archive that has not been
// proven. A tier backed up weekly is therefore proved weekly, by design and in perfect health, and
// against a flat 7-day window it would sit on the line and alarm every night about a system that is
// working. Shipping the agent's half alone would have converted the improvement into a false alarm,
// which is why the two ship together.
//
// The window is now derived from **the tier's own backup rhythm**, which the hub can observe from
// the reports it already receives, and it keeps everything the constant had earned:
//
// - absence is UNKNOWN until an anchored window has passed (R-81's structure, untouched);
// - the signal stays edge-triggered;
// - it never exceeds the offsite retention, so a tier is never called stale against an archive
// that is about to be pruned;
// - and it is never TIGHTER than the 7 days that were already tolerated.
const (
// restoreProvenGenerations is how many archive generations may pass unproven before alarming.
// 4 = the settle lag's own generation plus ~3 missed opportunities — deliberately the same
// tolerance the flat constant expressed, so the change is to the RHYTHM, not to the patience.
restoreProvenGenerations = 4
// restoreProvenWindowFloor is the shortest window that may be applied to any tier. It is the
// old constant, kept as a FLOOR rather than deleted: a daily tier computes 4 days from its own
// rhythm, and tightening a live threshold is not what this task is for. A deferral behind a
// long backup is normal, not a fault.
restoreProvenWindowFloor = 7 * 24 * time.Hour
// restoreProvenWindowCap keeps the window strictly inside the 2-week offsite retention
// (operator ruling 2026-07-26) with two days to spare. Beyond it the hub would be judging a
// tier against an archive PBS has already pruned — an alarm nobody can act on, and the bound
// the old constant respected in its own way.
restoreProvenWindowCap = 12 * 24 * time.Hour
// restoreWindowRead is how far back the hub reads host-reports for this check: far enough to
// find proof anywhere inside the widest window, and to see at least two archive generations of
// a WEEKLY tier so its rhythm is observable at all.
restoreWindowRead = 2 * restoreProvenWindowFloor
)
// declaredArchiveInterval is the rhythm the hub ALREADY attributes to a tier — the same thresholds
// the backup-freshness checker judges it against (deadline.go / deadline_tiers.go). It is the
// fallback when a box's history is too short to observe a rhythm, and it is the right fallback
// precisely because it is not a second opinion: if these two checkers disagreed about how often a
// tier is expected to receive an archive, one of them would be alarming on the other's model.
//
// It is stated per RESTORE tier name ("local"/"pbs" — what the agent reports as source_tier), which
// is the same split the backup tiers use under different names ("host"/"offsite").
func declaredArchiveInterval(tier string) time.Duration {
if tier == "pbs" {
return offsiteBackupStaleAfter // 8 days: the weekly cadence plus a day of headroom
}
return backupStaleAfter // 26 hours: the daily cadence plus headroom
}
// restoreProvenWindow is how long THIS tier may go unproven, given its observed archive interval.
//
// observedOK=false means the box's retained history did not contain two archive generations for
// this tier, so the declared rhythm is used. That fallback matters most for exactly the tier this
// task is about: a fresh box with a weekly offsite tier has one snapshot and no observable
// interval, and falling back to the FLOOR there would recreate the false alarm.
//
// OBSERVATION MAY ONLY WIDEN, NEVER TIGHTEN — and this is not caution, it is a live measurement.
// On demo-felhom (2026-08-03) the offsite tier's two retained snapshots are `2026-07-27T19:55:41Z`
// and `2026-07-28T04:49:43Z`: **8 h 54 m apart**, because one is a healing artefact and the other a
// real weekly run. A mean-gap estimate therefore reads a WEEKLY tier as nine-hourly, ×4 gives 36 h,
// the floor lifts it to 7 days — and a weekly tier proved weekly reaches ~8.25 days of proof age, so
// the false alarm this whole task exists to prevent would have returned within a week, on the very
// box it shipped to.
//
// The asymmetry is right on its own terms too. A gap SHORTER than the declared rhythm is routine and
// means nothing — a retry, a manual run, a heal, a catch-up after an outage. A gap LONGER than the
// declared rhythm is real information: this tier genuinely receives archives less often than the
// model says, and its window must widen or it alarms. So observation refines the rhythm upward and
// is ignored downward. The cost is stated plainly: a tier that truly runs FASTER than its declared
// rhythm gets a wider window than it strictly needs, i.e. a slower stale signal. That is the right
// direction for a signal whose message is "unverified" — "broken NOW" is `restore_test_failed`, and
// that one is immediate and unaffected.
func restoreProvenWindow(tier string, observed time.Duration, observedOK bool) time.Duration {
interval := declaredArchiveInterval(tier)
if observedOK && observed > interval {
interval = observed
}
w := time.Duration(restoreProvenGenerations) * interval
if w < restoreProvenWindowFloor {
w = restoreProvenWindowFloor
}
if w > restoreProvenWindowCap {
w = restoreProvenWindowCap
}
return w
}
// Event types. Operator-tier only — see the dispatcher note in RestoreTestChecker.
const (
@@ -156,12 +245,13 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
return
}
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-2*restoreProvenStaleAfter))
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-restoreWindowRead))
if err != nil {
c.logger.Printf("[WARN] restore-test check: window read failed for %s: %v", customerID, err)
return
}
proven := lastProvenPerTier(rows)
intervals := observedArchiveIntervals(rows)
first, ferr := c.store.GetFirstHostReportAt(customerID)
if ferr != nil {
@@ -170,7 +260,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
}
for _, tier := range tiers {
v := assessRestoreProven(tier, proven[tier], first, now)
observed, observedOK := intervals[tier]
window := restoreProvenWindow(tier, observed, observedOK)
v := assessRestoreProven(tier, proven[tier], first, now, window)
key := customerID + "|" + tier
c.mu.Lock()
prev := c.staleStates[key]
@@ -192,37 +284,120 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
}
}
// assessRestoreProven is the per-tier verdict. PURE (now injected) so the policy is unit-tested —
// the property that made R-81 provable, kept deliberately.
// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
// unit-tested — the property that made R-81 provable, kept deliberately.
//
// no proof, anchor NOT elapsed → UNKNOWN (newborn box; never an alarm)
// no proof, anchor elapsed → MISSED
// proof older than the window → MISSED
// otherwise → OK
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time) backupAssessment {
//
// `window` is now the TIER'S OWN (R-86 Part 2) rather than one constant for every tier, and every
// reason string states the window it was judged against. That is R-100's corollary applied here:
// when a verdict changes what it counts from, the alarm text has to change with it, or an operator
// reads "limit 168h" under a tier that was actually judged at 288h and dismisses a true alarm.
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time, window time.Duration) backupAssessment {
if provenAt.IsZero() {
if firstReportAt.IsZero() {
return backupAssessment{verdict: verdictMissed,
reason: fmt.Sprintf("%s tier: never restore-proven, and no first-contact anchor to defer against", tier)}
}
watched := now.Sub(firstReportAt)
if watched <= restoreProvenStaleAfter {
if watched <= window {
return backupAssessment{verdict: verdictUnknown,
reason: fmt.Sprintf("%s tier: not restore-proven yet, but only watching for %s (grace %s since first contact %s) — newborn, not a fault",
tier, watched.Round(time.Hour), restoreProvenStaleAfter, firstReportAt.Format(time.RFC3339))}
tier, watched.Round(time.Hour), window, firstReportAt.Format(time.RFC3339))}
}
return backupAssessment{verdict: verdictMissed,
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s) — the tier is unverified, not known-broken",
tier, watched.Round(time.Hour), restoreProvenStaleAfter)}
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
tier, watched.Round(time.Hour), window)}
}
if age := now.Sub(provenAt); age > restoreProvenStaleAfter {
if age := now.Sub(provenAt); age > window {
return backupAssessment{verdict: verdictMissed,
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s) — the tier is unverified, not known-broken",
tier, age.Round(time.Hour), restoreProvenStaleAfter)}
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
tier, age.Round(time.Hour), window)}
}
return backupAssessment{verdict: verdictOK}
}
// observedArchiveIntervals estimates how often each RESTORE tier actually receives an archive, from
// the host-reports the hub already holds. Keyed by restore-tier name ("local"/"pbs").
//
// Evidence is every distinct archive timestamp in the window: `pbs_snapshots[]` for the offsite
// tier (PBS enumerates its whole retention in each report, so one report usually settles the
// question) and successful `backups[]` records attributed by TARGET TYPE for both tiers — the
// slice-A.4 rule, because a PBS-targeted vzdump appears in BOTH arrays and classifying by array
// membership would attribute an offsite archive to the host tier.
//
// The estimate is the MEAN gap: (newest oldest) / (generations 1). It needs two generations;
// with fewer, ok=false and the caller falls back to the declared rhythm. It is deliberately crude,
// and can afford to be: restoreProvenWindow clamps the result between a 7-day floor and a 12-day
// cap, so the only discrimination this has to get right is "roughly daily" versus "several days or
// slower" — which is exactly the distinction that turns a healthy weekly tier into a false alarm.
func observedArchiveIntervals(rows []store.HostReportRow) map[string]time.Duration {
seen := map[string]map[int64]struct{}{ // tier → set of archive unix times
"local": {},
"pbs": {},
}
add := func(tier string, t time.Time) {
if t.IsZero() {
return
}
seen[tier][t.UTC().Unix()] = struct{}{}
}
for _, r := range rows {
var hr hostReportBackups
if json.Unmarshal([]byte(r.ReportJSON), &hr) != nil {
continue // one malformed retained report must not blind the scan
}
pbs := pbsTargetSet(hr)
for _, ps := range hr.PBSSnapshots {
if t, ok := parseBackupTime(ps.BackupTime); ok {
add("pbs", t)
}
}
for _, b := range hr.Backups {
if !b.Success {
continue
}
t, ok := parseBackupTime(b.StartedAt)
if !ok {
continue
}
if pbs[b.TargetID] {
add("pbs", t)
} else {
add("local", t)
}
}
}
out := map[string]time.Duration{}
for tier, set := range seen {
if len(set) < 2 {
continue // not observable — the caller uses the declared rhythm
}
var oldest, newest int64
first := true
for ts := range set {
if first || ts < oldest {
oldest = ts
}
if first || ts > newest {
newest = ts
}
first = false
}
span := time.Duration(newest-oldest) * time.Second
if span <= 0 {
continue
}
out[tier] = span / time.Duration(len(set)-1)
}
return out
}
// expectedRestoreTiers names the tiers this box actually HAS, so a box without an offsite tier is
// never reported stale for one. Same gate as Slice C's `expected`, and for the same reason: without
// it every box lacking a tier would alarm once the anchor elapsed — absence-is-not-failure,
+223 -9
View File
@@ -182,26 +182,33 @@ func TestRestoreTest_NewbornDoesNotAlarm(t *testing.T) {
}
}
// The boundary, pinned by name so a refactor has to delete an obviously-named contract.
// The boundary, pinned by name so a refactor has to delete an obviously-named contract. R-86 made
// the limit per-tier, so the anchor is now measured against THE TIER'S OWN window — here the local
// tier's, which clamps to the 7-day floor and so keeps this contract numerically identical to the
// one the flat constant expressed.
func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.T) {
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
window := restoreProvenWindow("local", 24*time.Hour, true)
if window != restoreProvenWindowFloor {
t.Fatalf("precondition: a daily local tier must clamp to the floor; got %s", window)
}
cases := []struct {
name string
watched time.Duration
wantMissed bool
}{
{"newborn, 1h", time.Hour, false},
{"just inside", restoreProvenStaleAfter - time.Minute, false},
{"exactly at the limit", restoreProvenStaleAfter, false},
{"just outside", restoreProvenStaleAfter + time.Minute, true},
{"just inside", window - time.Minute, false},
{"exactly at the limit", window, false},
{"just outside", window + time.Minute, true},
{"long past", 30 * 24 * time.Hour, true},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
got := assessRestoreProven("pbs", time.Time{}, now.Add(-c.watched), now)
got := assessRestoreProven("local", time.Time{}, now.Add(-c.watched), now, window)
if got.missed() != c.wantMissed {
t.Fatalf("CONTRACT VIOLATED: unproven for %s (limit %s) → missed=%v, want %v (reason %q)",
c.watched, restoreProvenStaleAfter, got.missed(), c.wantMissed, got.reason)
c.watched, window, got.missed(), c.wantMissed, got.reason)
}
if !c.wantMissed && got.verdict != verdictUnknown {
t.Fatalf("a deferred tier must be UNKNOWN (visible), not OK; got verdict=%d", got.verdict)
@@ -215,15 +222,17 @@ func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.
func TestRestoreTest_StaleIsSeparateFromFailure(t *testing.T) {
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
stale := assessRestoreProven("pbs", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now)
// A DAILY tier judged on its own rhythm: the window clamps to the 7-day floor.
daily := restoreProvenWindow("local", 24*time.Hour, true)
stale := assessRestoreProven("local", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
if !stale.missed() {
t.Fatalf("a tier last proven 9 days ago (limit %s) must be stale; got %q", restoreProvenStaleAfter, stale.reason)
t.Fatalf("a daily tier last proven 9 days ago (limit %s) must be stale; got %q", daily, stale.reason)
}
// The wording must not read as "broken" — that is the other signal.
if !strings.Contains(stale.reason, "unverified, not known-broken") {
t.Fatalf("staleness must say UNVERIFIED, not broken — merging the two is the thing this avoids; got %q", stale.reason)
}
fresh := assessRestoreProven("pbs", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now)
fresh := assessRestoreProven("local", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
if fresh.verdict != verdictOK {
t.Fatalf("a tier proven 2 days ago is fine; got verdict=%d reason=%q", fresh.verdict, fresh.reason)
}
@@ -283,3 +292,208 @@ func boolStr(b bool) string {
}
return "false"
}
// ── SCENARIO G — a healthy WEEKLY tier is never reported stale (R-86 Part 2) ─────────────────
//
// This is the test that pins the false alarm this change would otherwise have CREATED. The agent
// now proves a tier once per archive generation, so a weekly offsite tier is proved weekly — in
// perfect health. Against the old flat 7-day window it would sit on the line and alarm every night.
//
// COMPANION RED-PROOF (observed 2026-08-03): pin the window flat, as it was —
//
// - window := restoreProvenWindow(tier, observed, observedOK)
// - window := restoreProvenWindowFloor // the pre-R-86 flat 7 days
//
// → --- FAIL: TestRestoreTest_HealthyWeeklyTierIsNeverStale
//
// week 0: a weekly tier proved on its own archive must never be stale (proof age 172h0m0s,
// window 168h0m0s); verdict=2 reason="pbs tier: last successful restore-test was 172h0m0s ago
// (limit 168h0m0s, this tier's own backup rhythm) — the tier is unverified, not known-broken"
//
// Restored. The mutation is one line because the whole of Part 2 is one decision: whose rhythm.
//
// NOTE, because it is the finding this test nearly hid: the FIRST version of this fixture had NO
// jitter, and it PASSED under the mutation. A perfectly regular weekly tier's proof age reaches
// EXACTLY 168h just before the next proof, and `age > window` is false by a hair — a hollow test
// that would have shipped Part 1 and its false alarm together. The jitter below is what makes this
// a test, and it is also the truth about the old constant: a healthy weekly tier did not merely sit
// near the line, it sat ON it, so any ordinary delay tipped it over.
func TestRestoreTest_HealthyWeeklyTierIsNeverStale(t *testing.T) {
start := time.Date(2026, 6, 1, 3, 0, 0, 0, time.UTC)
firstContact := start.Add(-24 * time.Hour)
// The observable rhythm of a weekly tier, as the hub would compute it from the reports. No
// assertion about the window ITSELF here on purpose: that is the mechanism, and it is pinned in
// TestRestoreProvenWindow_Contract. What this test asserts is the CONSEQUENCE — does the alarm
// fire? — because R-97b proved a mechanism and shipped a broken consequence anyway.
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
// Walk several weeks of a HEALTHY tier, with the jitter a real one has: the backup does not land
// to the second, and a restore-test can be deferred one evaluation behind a running backup.
//
// The jitter is the point. A perfectly regular weekly tier's proof reaches an age of EXACTLY one
// interval (168h) just before the next proof, and against a flat 168h window `age > window` is
// false by a hair — so a regular fixture would pass against the very constant this task must
// change, and prove nothing. That is the brief's "sits exactly on that line": every real-world
// delay pushes it over, and the alarm is about a system that is working.
settle, evalLatency := 24*time.Hour, 6*time.Hour
archiveLate := []time.Duration{0, 4 * time.Hour, 2 * time.Hour, 6 * time.Hour, 0, 3 * time.Hour}
deferred := []time.Duration{0, 0, 6 * time.Hour, 0, 0, 6 * time.Hour} // one evaluation behind a backup
archiveAt := func(week int) time.Time {
return start.AddDate(0, 0, 7*week).Add(archiveLate[week])
}
provenAt := func(week int) time.Time {
return archiveAt(week).Add(settle + evalLatency + deferred[week])
}
var worst time.Duration
for week := 0; week+1 < len(archiveLate); week++ {
// The widest the proof's age ever gets: the instant before the NEXT week's proof lands.
now := provenAt(week + 1).Add(-time.Second)
age := now.Sub(provenAt(week))
if age > worst {
worst = age
}
v := assessRestoreProven("pbs", provenAt(week), firstContact, now, weekly)
if v.verdict != verdictOK {
t.Fatalf("week %d: a weekly tier proved on its own archive must never be stale (proof age %s, window %s); verdict=%d reason=%q",
week, age.Round(time.Hour), weekly, v.verdict, v.reason)
}
}
// The fixture must actually EXERCISE the boundary — a jitter-free walk would sit at exactly one
// interval and pass against a flat 7-day window, which is the hollow version of this test.
if worst <= restoreProvenWindowFloor {
t.Fatalf("this fixture never exceeds the old flat window (worst proof age %s) — it cannot detect the defect it exists for", worst)
}
}
// ...and a weekly tier that genuinely STOPS being proved must still alarm. A window that never
// fires is not a fix, it is a deletion.
func TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms(t *testing.T) {
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
v := assessRestoreProven("pbs", now.Add(-weekly-time.Hour), now.Add(-90*24*time.Hour), now, weekly)
if !v.missed() {
t.Fatalf("a weekly tier unproven for longer than its own window MUST alarm; got verdict=%d reason=%q", v.verdict, v.reason)
}
if !strings.Contains(v.reason, weekly.String()) {
t.Fatalf("the alarm must state the window it was judged against (R-100's corollary); got %q", v.reason)
}
}
// The window's own contract: derived from the tier's rhythm, floored, capped, and never dependent
// on an unobservable history for the tier that would suffer most from a wrong answer.
func TestRestoreProvenWindow_Contract(t *testing.T) {
cases := []struct {
name string
tier string
observed time.Duration
observedOK bool
want time.Duration
}{
{"daily local clamps to the floor", "local", 24 * time.Hour, true, restoreProvenWindowFloor},
{"weekly pbs widens", "pbs", 7 * 24 * time.Hour, true, restoreProvenWindowCap},
{"3-day tier sits between", "pbs", 72 * time.Hour, true, 12 * 24 * time.Hour},
{"unobservable local falls back to its declared rhythm", "local", 0, false, restoreProvenWindowFloor},
{"unobservable pbs falls back WIDE, not to the floor", "pbs", 0, false, restoreProvenWindowCap},
{"a nonsense zero interval is ignored", "pbs", 0, true, restoreProvenWindowCap},
// MEASURED ON THE LIVE BOX, and the reason observation may only WIDEN. demo-felhom's two
// retained PBS snapshots sit 8h54m apart (one is a healing artefact), so a mean-gap estimate
// reads a WEEKLY tier as nine-hourly. Taking that at face value gives 4x9h = 36h → the 7-day
// floor → and a weekly tier proved weekly (~8.25d of proof age) alarms within a week of this
// shipping, on the box it shipped to.
{"a short observed gap must NOT tighten a weekly tier", "pbs", 8*time.Hour + 54*time.Minute, true, restoreProvenWindowCap},
{"a short observed gap must not tighten the host tier either", "local", 30 * time.Minute, true, restoreProvenWindowFloor},
// ...but a tier that genuinely runs SLOWER than its declared rhythm still widens.
{"a genuinely slower tier widens", "local", 4 * 24 * time.Hour, true, restoreProvenWindowCap},
}
// The relationship Part 1 depends on: a weekly tier's window must be WIDER than a daily tier's,
// or proving weekly (which is now correct behaviour) alarms on itself.
if restoreProvenWindow("pbs", 7*24*time.Hour, true) <= restoreProvenWindow("local", 24*time.Hour, true) {
t.Fatal("a weekly tier must earn a wider window than a daily one — otherwise R-86's agent half alarms about itself")
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
got := restoreProvenWindow(c.tier, c.observed, c.observedOK)
if got != c.want {
t.Fatalf("window(%s, observed=%s ok=%v) = %s, want %s", c.tier, c.observed, c.observedOK, got, c.want)
}
if got < restoreProvenWindowFloor || got > restoreProvenWindowCap {
t.Fatalf("every window must stay inside [%s, %s]; got %s", restoreProvenWindowFloor, restoreProvenWindowCap, got)
}
})
}
}
// The rhythm must be OBSERVED from the reports, not assumed — including the slice-A.4 rule that a
// PBS-targeted vzdump appears in both arrays and must be attributed by TARGET TYPE.
func TestObservedArchiveIntervals_FromReports(t *testing.T) {
base := time.Date(2026, 7, 1, 2, 0, 0, 0, time.UTC)
mk := func(localAt []time.Time, pbsAt []time.Time) string {
type stg struct{ Name, Type, Content string }
type bk struct {
TargetID string `json:"target_id"`
Success bool `json:"success"`
StartedAt string `json:"started_at"`
}
type snap struct {
BackupTime string `json:"backup_time"`
}
payload := struct {
StorageTargets []struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
} `json:"storage_targets"`
Backups []bk `json:"backups"`
PBSSnapshots []snap `json:"pbs_snapshots"`
}{}
payload.StorageTargets = append(payload.StorageTargets, struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
}{"felhom-backup", "dir", "backup"}, struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
}{"felhom-pbs", "pbs", "backup"})
for _, at := range localAt {
payload.Backups = append(payload.Backups, bk{"felhom-backup", true, at.Format(time.RFC3339)})
}
for _, at := range pbsAt {
// The SAME archive appears as a vzdump record AND as a snapshot — slice A.4.
payload.Backups = append(payload.Backups, bk{"felhom-pbs", true, at.Format(time.RFC3339)})
payload.PBSSnapshots = append(payload.PBSSnapshots, snap{at.Format(time.RFC3339)})
}
b, err := json.Marshal(payload)
if err != nil {
t.Fatal(err)
}
return string(b)
}
rows := []store.HostReportRow{
{ReportJSON: mk(
[]time.Time{base, base.AddDate(0, 0, 1), base.AddDate(0, 0, 2)},
[]time.Time{base, base.AddDate(0, 0, 7)},
)},
{ReportJSON: `{{{malformed`}, // must not blind the scan
}
got := observedArchiveIntervals(rows)
if d, ok := got["local"]; !ok || d != 24*time.Hour {
t.Fatalf("a daily host tier must be observed as ~24h; got %s ok=%v", d, ok)
}
if d, ok := got["pbs"]; !ok || d != 7*24*time.Hour {
t.Fatalf("a weekly offsite tier must be observed as ~7d — and its vzdump record must NOT be "+
"counted into the host tier (slice A.4); got %s ok=%v", d, ok)
}
// One generation is not a rhythm: unobservable, so the caller falls back to the declared one.
single := []store.HostReportRow{{ReportJSON: mk(nil, []time.Time{base})}}
if d, ok := observedArchiveIntervals(single)["pbs"]; ok {
t.Fatalf("one archive cannot yield an interval; got %s", d)
}
}
@@ -0,0 +1,301 @@
package notify
import (
"io"
"log"
"strings"
"testing"
)
// R-182 — one e-mail per backup run, and nothing dropped without a trace.
//
// MEASURED, NOT SUPPOSED. On 2026-08-03 nine `recovery_unit_capture_failed` events reached the hub
// and TWO operator e-mails went out. The operator cooldown key is
// `customerID + ":" + eventType + cooldownTierSuffix(details)`, that event carries `app` but no
// `tier`, so the key held no app identifier: the first refused app took the hour's slot and every
// other app's failure was discarded — **before `LogNotification`**, so it left no row on any channel
// and could not be found afterwards.
//
// The operator ruled against the obvious fix (putting `app` in the key), because on a full disk that
// is one e-mail per app. These pin the shape that replaced it.
// ── Scenario D — a suppressed operator event leaves a trace ───────────────────────────────────────
// The bare `return` at the cooldown is the whole reason this defect took a day to get the right way
// round: there was nothing to read. A drop must be as visible in the record as a send.
//
// DELIBERATELY EXERCISED ON A DIFFERENT EVENT TYPE than the one that exposed the defect.
// `recovery_unit_capture_failed` is now record-only and never reaches the cooldown at all, so using
// it here would prove nothing. `whole_guest_backup_failed` is an ordinary operator event, and using
// it pins §2.1's actual claim: the suppression row applies to EVERY operator event, not only the one
// that happened to be measured.
func TestSuppressedOperatorEvent_LeavesARow(t *testing.T) {
st := newDispStore(t)
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
// Two events of the SAME type with no discriminator — the second must be suppressed.
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "opengist failed", `{"app":"opengist"}`, "controller")
d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "privatebin failed", `{"app":"privatebin"}`, "controller")
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
t.Fatalf("operator mails = %d, want 1 — the premise of this test is that the second IS suppressed", got)
}
rows, err := st.GetRecentNotifications("c1", 50)
if err != nil {
t.Fatal(err)
}
var sentRows, suppressed []store2Row
for _, r := range rows {
if r.Channel != "operator" || r.EventType != "whole_guest_backup_failed" {
continue
}
switch r.Status {
case "sent":
sentRows = append(sentRows, store2Row{r.Status, r.Message, r.ErrorMessage})
case "suppressed":
suppressed = append(suppressed, store2Row{r.Status, r.Message, r.ErrorMessage})
}
}
if len(sentRows) != 1 {
t.Fatalf("want 1 'sent' operator row, got %d", len(sentRows))
}
if len(suppressed) != 1 {
t.Fatalf("want 1 'suppressed' operator row, got %d — a cooldown drop that writes NOTHING is "+
"indistinguishable from an event that never happened, which is exactly how seven "+
"failures went missing on 2026-08-03", len(suppressed))
}
// The row must name the app that was dropped, or it records that something was suppressed
// without recording WHAT — half a fix.
if !strings.Contains(suppressed[0].message, "privatebin") {
t.Fatalf("the suppressed row does not name the dropped event: %q", suppressed[0].message)
}
// And it must carry the key, so the reason it collided is readable without reading code.
if !strings.Contains(suppressed[0].errMsg, "key=") {
t.Fatalf("the suppressed row does not carry the cooldown key: %q", suppressed[0].errMsg)
}
}
type store2Row struct{ status, message, errMsg string }
// ── Scenario E — two runs in a day each report ───────────────────────────────────────────────────
// The operator ruled explicitly on this: someone pressing the backup button is actively trying to
// get a backup, and finding out tomorrow would be worse than an extra mail in a rare case.
func TestTwoRunsInAnHour_BothReport(t *testing.T) {
st := newDispStore(t)
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
nightly := `{"run_id":"run-a","run_kind":"nightly","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
manual := `{"run_id":"run-b","run_kind":"manual","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", nightly, "controller")
d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", manual, "controller")
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 2 {
t.Fatalf("operator mails = %d, want 2 — the 1-hour cooldown swallowed the manual run's "+
"digest, which is the fix reappearing one level up: press the button, the run fails, "+
"and you are told nothing because the machine already wrote this hour", got)
}
}
// The run discriminator must be NARROW, exactly like its `tier` sibling — empty unless the producer
// opts in — or every existing event type's cooldown silently stops collapsing anything.
func TestCooldownRunSuffix_EmptyForEverythingElse(t *testing.T) {
cases := []struct{ name, details string }{
{"no details", ""},
{"details without run_id", `{"app":"opengist","error":"boom"}`},
{"empty run_id value", `{"run_id":""}`},
{"malformed json", `{{{nope`},
{"run_id mentioned in a STRING, not as a key", `{"error":"the run_id: abc failed"}`},
{"null details", `null`},
}
for _, c := range cases {
if got := cooldownRunSuffix(c.details); got != "" {
t.Errorf("%s: suffix must be EMPTY so every other type's cooldown is unchanged, got %q", c.name, got)
}
}
if got := cooldownRunSuffix(`{"run_id":"run-a"}`); got != ":run-a" {
t.Fatalf("suffix should be the run id, got %q", got)
}
// The two suffixes must not interfere: a tier event still keys on its tier and nothing else.
if got := cooldownTierSuffix(`{"tier":"felhom-pbs"}`) + cooldownRunSuffix(`{"tier":"felhom-pbs"}`); got != ":felhom-pbs" {
t.Fatalf("a tier-only event's key changed to %q — R-97a's behaviour must be byte-identical", got)
}
}
// ── Scenario G — the customer never receives the digest ──────────────────────────────────────────
// v0.78.0 asserted in a COMMENT that a type with no `customerMessages` entry structurally cannot
// reach a customer. It can: FormatCustomerEmail falls back to the raw English message and the only
// customer gate is configuration. So this is demonstrated, not argued.
func TestDigest_IsOperatorOnly_EvenWithAWideEnabledList(t *testing.T) {
st := newDispStore(t)
// A customer who has enabled EVERYTHING, including this type by name.
if err := st.SaveNotificationPrefs("c1", "cust@example.com",
[]string{"backup_run_failures", "node_down", "disk_warning"}, 6); err != nil {
t.Fatal(err)
}
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
details := `{"run_id":"run-a","run_kind":"nightly","failed":1,"attempted":3,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"}]}`
d.ProcessEvent("c1", "backup_run_failures", "error", "1 of 3 apps failed", details, "controller")
if got := mailsFor(*sent, "cust@example.com"); len(got) != 0 {
t.Fatalf("the CUSTOMER received an operator digest (%d mails) — a list of which apps' "+
"backups failed is not something they can act on, and the raw body is English", len(got))
}
if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
t.Fatalf("operator mails = %d, want 1", got)
}
if !operatorOnlyEvents["backup_run_failures"] {
t.Fatal("backup_run_failures is not in operatorOnlyEvents — allowlisting alone does NOT " +
"keep it from a customer; that assumption shipped once and was wrong (v0.78.0)")
}
}
// ── Part 3 — the e-mail a person actually reads ──────────────────────────────────────────────────
func TestDigestEmail_ListsAppsLegsAndReasons(t *testing.T) {
details := `{"run_id":"run-a","run_kind":"nightly","failed":3,"attempted":40,` +
`"target_path":"/mnt/sys_drive","used_gb":64.3,"avail_gb":0.9,"total_gb":68.7,` +
`"used_percent":94,"space_known":true,"apps":[` +
`{"app":"opengist","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
`{"app":"privatebin","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
`{"app":"immich","leg":"database dump","reason":"pg_dump: connection refused"}]}`
subject, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error",
"3 of 40 apps failed to back up", details)
// The subject must carry the counts: the operator's first decision is made from it alone.
for _, want := range []string{"demo-hp", "3 of 40", "nightly"} {
if !strings.Contains(subject, want) {
t.Errorf("subject %q missing %q", subject, want)
}
}
// Every app, its leg and its reason.
for _, want := range []string{
"opengist", "privatebin", "immich",
"volume dump", "database dump",
"below the reserve", "pg_dump: connection refused",
} {
if !strings.Contains(body, want) {
t.Errorf("body missing %q:\n%s", want, body)
}
}
// The counts and the free space, so "one broken app" and "a full disk" read differently.
if !strings.Contains(body, "3 of 40") {
t.Errorf("body does not carry the failed-of-attempted count:\n%s", body)
}
if !strings.Contains(body, "0.9 GB free") {
t.Errorf("body does not carry the free space:\n%s", body)
}
// It must NOT be a JSON blob.
if strings.Contains(body, `"apps":[`) {
t.Errorf("the digest rendered as raw JSON — unreadable on a phone at 07:00:\n%s", body)
}
}
// An absent space reading must render as unavailable, never as zeros: "0 GB free" and "we could not
// look" are opposite diagnoses, and the operator acts differently on each.
func TestDigestEmail_UnknownSpaceIsNotZero(t *testing.T) {
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/x",` +
`"space_known":false,"apps":[{"app":"a","leg":"capture","reason":"boom"}]}`
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
if strings.Contains(body, "0.0 GB free") {
t.Fatalf("an unreadable filesystem rendered as zeros:\n%s", body)
}
if !strings.Contains(body, "unavailable") {
t.Fatalf("an unreadable filesystem must say so:\n%s", body)
}
}
// A payload that cannot be parsed must still produce a mail — degraded, never swallowed.
func TestDigestEmail_UnparseableDetailsStillMails(t *testing.T) {
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "something failed", `{{{`)
if body == "" || !strings.Contains(body, "something failed") {
t.Fatalf("an unparseable digest lost the mail entirely:\n%s", body)
}
}
// ── Scenario C — every failure is RECORDED, e-mailed or not ──────────────────────────────────────
// The per-app event is the record; the digest is the notification. The record must not inherit the
// notification's conditions — no cooldown, no preferences, no dependence on a mail going out.
func TestPerAppFailure_IsRecordedButNotMailed(t *testing.T) {
st := newDispStore(t)
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"}
for _, a := range apps {
d.ProcessEvent("c1", "recovery_unit_capture_failed", "error",
"Recovery unit capture FAILED for \""+a+"\"", `{"app":"`+a+`"}`, "controller")
}
// NOT mailed — the digest is the notification.
if got := len(*sent); got != 0 {
t.Fatalf("%d mail(s) sent for per-app failures — they are the RECORD; one mail per app on a "+
"full disk is the volume problem wearing the correctness problem's clothes, which is "+
"exactly what the operator ruled against", got)
}
// But ALL FIVE recorded — this is the assertion yesterday's defect would have failed: nine
// arrived, two were mailed, seven left no row anywhere.
rows, err := st.GetRecentNotifications("c1", 50)
if err != nil {
t.Fatal(err)
}
seen := map[string]bool{}
for _, r := range rows {
if r.EventType == "recovery_unit_capture_failed" && r.Status == "recorded" {
for _, a := range apps {
if strings.Contains(r.Message, a) {
seen[a] = true
}
}
}
}
if len(seen) != len(apps) {
t.Fatalf("only %d of %d per-app failures were recorded (%v) — a failure that produced no row "+
"anywhere is the measured defect of 2026-08-03", len(seen), len(apps), seen)
}
}
// The per-app reason must not repeat the filesystem figures the digest already prints once. Reviewed
// as copy against the first real digest, not designed in the abstract.
func TestDigestEmail_ReasonDoesNotRepeatTheUsageLine(t *testing.T) {
reason := "refused: below the reserve (reserve: 97% used or 1.0 GiB free) — /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free"
details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/sys_drive",` +
`"used_gb":65,"avail_gb":0.2,"total_gb":68.7,"used_percent":95,"space_known":true,` +
`"apps":[{"app":"opengist","leg":"whole app","reason":"` + reason + `"}]}`
_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
// The figures appear ONCE, on the Filesystem line — not again on every app row.
if strings.Count(body, "65.0/68.7 GB used") != 1 {
t.Fatalf("the usage clause appears %d times; it must appear once, on its own line — repeated "+
"down a list of a dozen apps it pushes the part that DIFFERS off a phone screen:\n%s",
strings.Count(body, "65.0/68.7 GB used"), body)
}
// But the reason itself survives — trimming must not eat the diagnosis.
if !strings.Contains(body, "below the reserve") {
t.Fatalf("the reason was trimmed away entirely:\n%s", body)
}
}
// A reason naming a DIFFERENT path, or none, must be left completely alone.
func TestTrimRepeatedUsage_LeavesUnrelatedReasonsAlone(t *testing.T) {
for _, c := range []struct{ reason, target string }{
{"pg_dump: connection refused", "/mnt/sys_drive"},
{"tar failed — /mnt/other: 1/2 GB used (50%), 1 GB free", "/mnt/sys_drive"},
{"boom", ""},
{"", "/mnt/sys_drive"},
} {
if got := trimRepeatedUsage(c.reason, c.target); got != c.reason {
t.Errorf("reason %q (target %q) was altered to %q", c.reason, c.target, got)
}
}
}
+101 -1
View File
@@ -116,6 +116,18 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta
return
}
// R-182: record-only types are written down and never mailed. Placed BEFORE the severity gate
// so the row is written whatever the severity — the record must not inherit the notification's
// conditions, which is the coupling this whole finding is about.
if recordOnlyEvents[eventType] {
if err := d.store.LogNotification(customerID, eventType, severity, message, "recorded",
"record-only: the per-run digest (backup_run_failures) carries the notification", "operator"); err != nil {
d.logger.Printf("[WARN] Failed to record %s for %s: %v", eventType, customerID, err)
}
d.logger.Printf("[INFO] Recorded (not mailed) %s for %s — the run digest is the notification", eventType, customerID)
return
}
// warning / error / critical trigger notifications. "info" is an intentional non-notify (status/
// recovery events). Anything else is UNRECOGNIZED — log it (don't silently drop), so a bad severity
// surfaces instead of vanishing (the felhom-pve-class lesson: a critical event must never be lost).
@@ -260,15 +272,67 @@ func cooldownTierSuffix(detailsJSON string) string {
return ":" + d.Tier
}
// cooldownRunSuffix returns ":"+run_id when the event's details carry a non-empty `run_id`, else "".
//
// R-182. `cooldownTierSuffix`'s sibling, and deliberately a SEPARATE function rather than an extra
// branch inside it: `tier` keeps byte-identical semantics for every type that uses it, so R-97a's
// behaviour and its tests are untouched by this.
//
// WHY A BACKUP RUN NEEDS ONE. The run digest describes ONE RUN, and a box can have two in a day —
// the nightly one and a manual one the operator triggered *because* something looked wrong. With no
// run-scoped discriminator the 1-hour cooldown would swallow the second, which is the failure this
// row exists to fix, reappearing one level up: the operator presses the button, the run fails, and
// they are told nothing because the machine already wrote that hour.
//
// IT MAKES THE COOLDOWN EFFECTIVELY INERT FOR THIS TYPE, AND THAT IS THE INTENT, NOT AN OVERSIGHT.
// A digest is already rate-limited by construction — one per run, emitted only when something
// failed — so there is nothing for a timer to collapse. The cooldown protects against a repeating
// identical alert; a digest cannot repeat, because each run is a different run.
//
// NARROW, LIKE ITS SIBLING: empty unless the producer opts in by sending a `run_id`, so no existing
// event type's cooldown behaviour changes.
func cooldownRunSuffix(detailsJSON string) string {
if detailsJSON == "" || !strings.Contains(detailsJSON, "\"run_id\"") {
return ""
}
var d struct {
RunID string `json:"run_id"`
}
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || d.RunID == "" {
return ""
}
return ":" + d.RunID
}
func (d *Dispatcher) processOperator(customerID, eventType, severity, message, detailsJSON, source string) {
if !d.operatorOn || d.operatorEmail == "" {
return
}
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON)
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON) + cooldownRunSuffix(detailsJSON)
d.mu.Lock()
if last, ok := d.opCooldowns[cooldownKey]; ok && time.Since(last) < 1*time.Hour {
d.mu.Unlock()
// R-182: RECORD THE SUPPRESSION. This used to be a bare `return` — the event was dropped
// before any LogNotification, so a cooldown drop and an event that never happened were
// indistinguishable from the operator's side AND from the hub's own records.
//
// Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two emails were
// sent, and the other seven left NO ROW ON ANY CHANNEL. The defect that hid was serious —
// the cooldown key carries no app identifier, so the first refused app took the slot and
// every other app's failure that hour was discarded — but the reason it took a day to find
// the right way round is this line: there was nothing to read.
//
// "We chose not to e-mail you" and "nothing happened" must never look identical. This
// applies to EVERY operator event, not only the one that exposed it. It makes the drop
// visible; it deliberately does NOT change the cooldown's duration or semantics.
if err := d.store.LogNotification(customerID, eventType, severity, message,
"suppressed", "operator cooldown 1h, key="+cooldownKey, "operator"); err != nil {
d.logger.Printf("[WARN] Failed to record suppressed operator notification for %s/%s: %v",
customerID, eventType, err)
}
d.logger.Printf("[INFO] Operator email suppressed for %s/%s — cooldown (key=%s)",
customerID, eventType, cooldownKey)
return
}
d.opCooldowns[cooldownKey] = time.Now()
@@ -285,6 +349,33 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "operator")
}
// recordOnlyEvents are STORED and RECORDED but never e-mailed, on either channel.
//
// R-182. The distinction this register exists to make is the whole of that finding: **the record and
// the notification are different things.** A per-app backup failure must always be written down —
// every time, unconditionally, regardless of cooldowns, preferences or whether any mail went out —
// and it must NOT compete for an e-mail slot, because the per-run digest
// (`backup_run_failures`) is what a person is meant to read.
//
// Before this, `recovery_unit_capture_failed` was both at once, and it did neither well: on
// 2026-08-03 nine of them arrived, two were e-mailed, and the other seven were dropped by the
// 1-hour cooldown BEFORE anything was written down. So the operator was told about one app, the
// other apps' failures were discarded, and nothing anywhere recorded that a choice had been made.
//
// WHY A REGISTER AND NOT severity "info". Downgrading the severity would have the same routing
// effect — `severityNotifies` drops info — but it would also relabel a genuine failure as
// informational in the events table, the operator UI and every historical query, and it would
// silently drop the X-Priority handling if the type were ever promoted back. This says what it
// means: not silent, not urgent, RECORDED.
//
// IT IS NOT A WAY TO MUTE THINGS. A type belongs here only when something else carries its
// notification. Adding one with no digest behind it rebuilds the silence R-182 was filed against.
var recordOnlyEvents = map[string]bool{
// The per-app Tier-1 capture/refusal failure. Its notification is the run digest, which lists
// every failed app in one mail; this row is the durable per-failure record behind it.
"recovery_unit_capture_failed": true,
}
// operatorOnlyEvents are event types that must NEVER reach a customer, whatever their preferences say.
//
// R-97c. This register exists because the guarantee it provides was previously ASSERTED IN A COMMENT
@@ -321,6 +412,15 @@ var operatorOnlyEvents = map[string]bool{
// figures, the raw error). The customer's half of D-c is the FILL WARNING, which fires BEFORE
// this and is actionable: free space, delete files, add a drive.
"recovery_unit_capture_failed": true,
// R-182. The per-run backup digest. It is the same class as the line above and for the same
// reason — a customer can act on a full disk (that is the fill warning, which fires first and
// IS customer-facing) but not on a list of which apps' backups failed and why. It also carries
// operator-grade detail: per-app leg names, raw refusal reasons and byte figures.
//
// Listed here rather than relying on the absence of a `customerMessages` entry, which is NOT a
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
// once (v0.78.0) and the comment above records it.
"backup_run_failures": true,
}
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
@@ -39,33 +39,45 @@ func TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer(t *testing.T) {
}
}
// The operator must still get it: the register mutes the customer channel, not the signal.
gotOperator := false
// R-182 CHANGED WHAT THIS ASSERTS, DELIBERATELY, AND THE OLD ASSERTION IS WORTH KEEPING IN VIEW.
//
// Until 2026-08-03 this test required the OPERATOR to be e-mailed here, on the grounds that "the
// alert is the whole point of R-158". That was right when this event was the only signal, and it
// is wrong now: measured, nine of these arrived at the hub and two were mailed, because the
// operator cooldown key carries no app identifier — so as an alarm it told the operator about one
// app and threw the rest away.
//
// The type is now RECORD-ONLY: written down every time, never mailed. R-158's guarantee — the
// operator learns WHICH app failed and WHY — is not weakened, it MOVED: the per-run digest
// `backup_run_failures` carries every failed app in one mail, and is pinned by
// backup_run_digest_test.go. The customer safety claim below is untouched and is the reason this
// test still exists.
for _, to := range rec.to {
if to == "operator@felhom.eu" {
gotOperator = true
t.Fatal("the operator was e-mailed a PER-APP capture failure — this type is the record " +
"now, not the alarm. One mail per failing app on a full disk is a dozen mails, which " +
"is the volume problem the operator ruled against; the digest is the notification")
}
}
if !gotOperator {
t.Fatal("the operator was not notified of a recovery-unit capture failure — the alert is the " +
"whole point of R-158 and it went nowhere")
}
// The skip must be VISIBLE. An absent log row is equally consistent with "correctly skipped" and
// "the dispatcher never ran" — the positive observable is the row itself (standing rule 3).
// The RECORD must exist, always. It is what makes the digest trustworthy: if the digest is ever
// lost, delayed or suppressed, the failures are still individually written down. An absent row is
// equally consistent with "correctly not mailed" and "the dispatcher never ran" — the positive
// observable is the row itself (standing rule 3).
logs, err := st.GetRecentNotifications("c1", 20)
if err != nil {
t.Fatalf("GetRecentNotifications: %v", err)
}
found := false
for _, l := range logs {
if l.Channel == "customer" && l.Status == "skipped" && strings.Contains(l.ErrorMessage, "operator_only") {
if l.EventType == "recovery_unit_capture_failed" && l.Status == "recorded" &&
strings.Contains(l.ErrorMessage, "record-only") {
found = true
}
}
if !found {
t.Fatalf("the customer skip is not logged as skipped/operator_only — it is indistinguishable "+
"from a delivery that never happened; got %d row(s)", len(logs))
t.Fatalf("the per-app failure left no 'recorded' row — a failure that is neither mailed nor "+
"written down is exactly the 2026-08-03 defect, rebuilt; got %d row(s)", len(logs))
}
}
+124
View File
@@ -1,6 +1,7 @@
package notify
import (
"encoding/json"
"fmt"
"strings"
"time"
@@ -41,6 +42,17 @@ Severity: %s
Time: %s
Message: %s`, customerID, eventType, severity, now, message)
// R-182: the backup run digest gets a rendered list instead of a raw JSON blob. It is the one
// operator mail that carries a VARIABLE-LENGTH payload, and a dozen apps as one line of JSON is
// unreadable on a phone at 07:00, which is the only time it matters.
if eventType == "backup_run_failures" {
if rendered, sub, ok := renderBackupRunFailures(customerID, detailsJSON); ok {
return sub, body + rendered + fmt.Sprintf("\n\nDashboard: https://hub.felhom.eu/customers/%s", customerID)
}
// Unparseable details fall through to the raw form below rather than losing the mail. A
// digest that renders badly still tells the operator something; a swallowed one does not.
}
if detailsJSON != "" && detailsJSON != "{}" {
body += fmt.Sprintf("\nDetails: %s", detailsJSON)
}
@@ -282,3 +294,115 @@ Ha nem te kérted ezt, hagyd figyelmen kívül ezt az e-mailt.
Felhom.eu`, link)
return subject, body
}
// ──────────────────────────────────────────────────────────────────────
// R-182 — the backup run digest
// ──────────────────────────────────────────────────────────────────────
// backupRunFailure is one app's failed leg within a run.
type backupRunFailure struct {
App string `json:"app"`
Leg string `json:"leg"`
Reason string `json:"reason"`
}
// backupRunDetails is the digest payload the controller sends.
type backupRunDetails struct {
RunID string `json:"run_id"`
RunKind string `json:"run_kind"`
Failed int `json:"failed"`
Attempted int `json:"attempted"`
TargetPath string `json:"target_path"`
UsedGB float64 `json:"used_gb"`
AvailGB float64 `json:"avail_gb"`
TotalGB float64 `json:"total_gb"`
UsedPercent float64 `json:"used_percent"`
SpaceKnown bool `json:"space_known"`
Apps []backupRunFailure `json:"apps"`
}
// renderBackupRunFailures turns the digest details into an operator-readable block and a subject
// that says the count without being opened. Returns ok=false when the payload cannot be parsed or
// names no apps, so the caller can fall back to the raw rendering rather than mail an empty list.
//
// THE SUCCESS COUNT IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
// night; the list alone cannot tell them apart, and the operator's first decision — get up now, or
// look after coffee — depends entirely on which it is.
func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bool) {
if detailsJSON == "" {
return "", "", false
}
var d backupRunDetails
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || len(d.Apps) == 0 {
return "", "", false
}
kind := d.RunKind
if kind == "" {
kind = "backup"
}
subject := fmt.Sprintf("[Felhom] 🔴 %s: %d of %d apps failed to back up (%s run)",
customerID, d.Failed, d.Attempted, kind)
// Column-align the app names so the leg and reason line up and the block scans vertically.
width := 0
for _, a := range d.Apps {
if len(a.App) > width {
width = len(a.App)
}
}
legWidth := 0
for _, a := range d.Apps {
if len(a.Leg) > legWidth {
legWidth = len(a.Leg)
}
}
var b strings.Builder
fmt.Fprintf(&b, "\n\nFAILED: %d of %d apps attempted in this %s run.\n\n", d.Failed, d.Attempted, kind)
for _, a := range d.Apps {
reason := trimRepeatedUsage(a.Reason, d.TargetPath)
if reason == "" {
reason = "(no reason recorded)"
}
fmt.Fprintf(&b, " %-*s %-*s %s\n", width, a.App, legWidth, a.Leg, reason)
}
// The space figures answer "is this one broken app or a full disk" before the reasons are read.
// An absent reading renders as unavailable, never as zeros — "0 GB free" and "we could not look"
// are opposite diagnoses (the UnitSpace rule, same reasoning, other side of the wire).
if d.SpaceKnown {
fmt.Fprintf(&b, "\nFilesystem: %s — %.1f/%.1f GB used (%.0f%%), %.1f GB free\n",
d.TargetPath, d.UsedGB, d.TotalGB, d.UsedPercent, d.AvailGB)
} else {
fmt.Fprintf(&b, "\nFilesystem: %s — usage unavailable (the filesystem could not be read)\n", d.TargetPath)
}
b.WriteString("\nEvery failure above is also recorded individually in the notification log,\n")
b.WriteString("whether or not this mail was sent.")
return b.String(), subject, true
}
// trimRepeatedUsage strips the trailing "— /path: X/Y GB used (Z%), W GB free" clause from a per-app
// reason, because the digest prints those figures ONCE for the whole run on its own line.
//
// This is a copy fix, and it was made after reading the first real digest rather than from the
// design. The reserve's refusal message is authored for a single-app alert, where naming the
// filesystem is exactly right; repeated down a list of a dozen apps it is the same forty characters
// twelve times, and it pushes the part that differs off the right-hand edge of a phone screen at
// 07:00 — which is the only moment this mail has to work.
//
// It trims ONLY an exact "— <target path>:" suffix, so a reason that mentions a different path, or
// none, is left completely alone. A reason that is nothing but the usage clause is left alone too:
// removing everything would turn a bad line into an empty one.
func trimRepeatedUsage(reason, targetPath string) string {
if reason == "" || targetPath == "" {
return reason
}
marker := " — " + targetPath + ":"
i := strings.LastIndex(reason, marker)
if i <= 0 {
return reason
}
return strings.TrimSpace(reason[:i])
}
+1 -1
View File
@@ -815,7 +815,7 @@ type NotificationLogEntry struct {
EventType string
Severity string
Message string
Status string // "sent", "skipped", "failed"
Status string // "sent", "skipped", "failed", "suppressed" (R-182: a cooldown drop, recorded rather than silent)
ErrorMessage string
Channel string // "operator" or "customer"
CreatedAt time.Time
+1 -1
View File
@@ -125,7 +125,7 @@ spec:
spec:
containers:
- name: hub
image: gitea.dooplex.hu/admin/felhom-hub:0.89.0
image: gitea.dooplex.hu/admin/felhom-hub:0.91.1
ports:
- containerPort: 8080
name: http
+2 -2
View File
@@ -324,7 +324,7 @@ spec:
image: registry.k8s.io/git-sync/git-sync:v4.4.0
args:
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
- --ref=installer-v1.23.0
- --ref=installer-v1.24.0
- --root=/git-scripts
- --link=current
- --period=30s
@@ -369,7 +369,7 @@ spec:
image: registry.k8s.io/git-sync/git-sync:v4.4.0
args:
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
- --ref=installer-v1.23.0
- --ref=installer-v1.24.0
- --root=/git-scripts
- --link=current
- --one-time
+52
View File
@@ -1,3 +1,55 @@
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the
box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing
Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the
operator's inbox. Every week, on both boxes, about a backup that had already succeeded.
**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay
write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was
not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes
`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already
reads as "never prune from the box" — no agent change needed.
**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and
`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing
`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling
the client-side prune without that check would have traded a weekly false alarm for unbounded growth.
A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back
quietly. The local tier's retention is untouched.
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
storage and calls `felhom-backup-target-apply grant` in the same breath — a box that builds its own
target has always been fine. The **Scenario-F arm** — *"the target already exists, leave it exactly as
it is"* — returned without ever granting.
So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
surviving a reinstall) ended up with `local_backup_target: felhom-backup` while its token held
`FelhomAgentStore` on only `local`, `local-lvm` and `felhom-pbs`. Measured on **both** demo boxes
2026-08-03: the content API answers `{"data":[]}` through the agent's token while root lists three
archives. That tier was invisible to the agent and never restore-tested — and nothing said so,
because an empty listing is also what a brand-new tier returns.
The reuse arm now ensures the ACL through the same guarded wrapper, so both arms leave the box in the
same state. **Scenario F is unviolated:** the storage DEFINITION is still untouched — granting the
role the agent is supposed to have on the target this script is about to write into `agent.json` is
finishing the job, not retargeting the box. `pveum acl modify` is idempotent, so a box that already
has the grant is unchanged and a box whose token was rotated gets it back.
`$BACKUP_TARGET_ID` is deliberately **not** added to `PVE_STORAGES`, and the comment now says why: that
list is granted in step 4/5, before the target has been resolved in step 6, and `--acl-storages`
entries are preflight-checked for existence. The grant belongs with the resolution, which is where it
already was for a newly created target.
**A gate now asserts it** (`hostinstall_gates.py`): every arm of `configure_backup_target` that
resolves the target must also grant on it. Red-proved by reverting the reuse arm — `resolves the
backup target in 2 place(s) but grants in only 1`.
## v1.23.0 — the installer is published, not pushed (2026-08-03, R-110 + R-183)
**Two channels moved off `main` in the same change, because either one left behind makes the other
+58 -5
View File
@@ -184,7 +184,7 @@
set -euo pipefail
SCRIPT_VERSION="1.23.0" # the SINGLE version source (F-1): -h and the run banners follow it.
SCRIPT_VERSION="1.25.0" # the SINGLE version source (F-1): -h and the run banners follow it.
# The hub used to carry a copy for its Setup tab; R-94 DELETED it
# (2026-08-02) because the hub cannot know which version a box runs —
# the Setup command fetches this script at run time. scripts/
@@ -307,6 +307,14 @@ PVE_PRIVS_STORE="Datastore.Allocate Datastore.AllocateSpace"
PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"
# Storages the agent reads/writes (archive+dump=local, restore=local-lvm, offsite DR=felhom-pbs). The
# offsite felhom-pbs MUST be included or the agent's DR backup 403s (SPIKE residual #1). --acl-storages overrides.
#
# `$BACKUP_TARGET_ID` is deliberately NOT in this list, and the reason matters because the obvious
# tidy-up is to add it (R-185, 2026-08-03). This list is granted in STEP 4/5, before
# configure_backup_target has run in STEP 6 — so at this point the target may not exist yet, and
# --acl-storages entries are preflight-checked for existence. The target's grant therefore belongs
# with the target's RESOLUTION, where it already is for a freshly created one, and now also for a
# pre-existing one. Adding it here would grant on a storage that may not exist and would still leave
# the resolution path as the single owner of that decision, split across two places.
PVE_STORAGES=(local local-lvm felhom-pbs)
# E-2: the whole-guest backup target storage id, and what configure_backup_target resolved to.
# BACKUP_TARGET_RESOLVED feeds agent.json's local_backup_target; "local" means DEGRADED (Case B).
@@ -652,8 +660,31 @@ configure_backup_target() {
# existing id is an error, and repointing a live target is exactly the silent retarget this
# whole arc closes.
if pvesm status --storage "$BACKUP_TARGET_ID" >/dev/null 2>&1; then
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving it exactly as it is (Scenario F)"
log_skip " backup target '$BACKUP_TARGET_ID' already exists — leaving its DEFINITION exactly as it is (Scenario F)"
BACKUP_TARGET_RESOLVED="$BACKUP_TARGET_ID"
# R-185: …but STILL ensure the ACL. "The storage already exists" says nothing about whether
# the agent may READ it, and this early return is where the two came apart.
#
# THE DEFECT THIS CLOSES, measured on both demo boxes 2026-08-03. The CASE A path below
# creates the storage and grants in the same breath, so a box that built its own target is
# fine. A box whose target ALREADY existed — created by the vzdump-target-move runbook, or
# surviving a reinstall — returned here and never granted. The result: `local_backup_target`
# pointed at `felhom-backup` while the token held FelhomAgentStore only on local, local-lvm
# and felhom-pbs, so the API answered `{"data":[]}` for that storage while root saw three
# archives. The tier was invisible to the agent and never restore-tested, and nothing said so
# — because an empty listing is also what a brand-new tier returns.
#
# Scenario F is UNVIOLATED: the storage definition is still untouched. Granting a role the
# agent is supposed to have on the target this same script is about to write into
# agent.json is not "touching the box's target", it is finishing the job. `pveum acl modify`
# is idempotent, so a box that already has the grant is unchanged, and a box that had its
# token rotated gets it back.
if $DRY_RUN; then
log_dry "felhom-backup-target-apply grant $BACKUP_TARGET_ID # R-185: ACL on a pre-existing target"
else
/usr/local/sbin/felhom-backup-target-apply grant "$BACKUP_TARGET_ID" \
|| die "backup target grant failed on the pre-existing target — the agent could not read its own backup tier (R-185)"
fi
return 0
fi
local mp
@@ -2448,8 +2479,30 @@ for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agen
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator
# ruling 2026-07-26).
# would guarantee that one of them is wrong.
#
# THE OFFSITE TIER CARRIES NO CLIENT-SIDE RETENTION, AND THAT IS THE POINT (R-191, 2026-08-04).
# It used to be written `keep_last: 2` ("two weeks", operator ruling 2026-07-26). **R-89 then moved
# offsite pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay write-only,
# deliberately, so that a box can never delete its own offsite history. The 2026-07-26 ruling was not
# reversed; where it is ENFORCED moved, and this value did not follow.
#
# The consequence was weekly and wrong in the worst direction: vzdump UPLOADED the snapshot fine and
# then failed the whole job on the prune the token is refused —
# `prune 'ct/9201': permission check failed - missing Datastore.Modify|Datastore.Prune`
# → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the operator's inbox. Every week, on
# both boxes, about a backup that had already succeeded. A tier that cries wolf weekly is a tier
# whose real failure nobody will see.
#
# VERIFIED BEFORE CHANGING IT (2026-08-04, read-only on ep0): prune jobs `prune-demo-felhom` and
# `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
# and have run EVERY DAY since 2026-07-27 — 18 tasks, all `status=OK`, the newest showing
# `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Retention
# happens; it happens THERE. **If that ever stops being true, this zero is unbounded growth** — check
# ep0's prune jobs before assuming the offsite tier is retained.
#
# `keep_last: 0` means "never prune from the box" and is the value the agent's own guard reads
# (allowPBSPrune = !primary && keep_last > 0), so no agent change is needed to honour it.
#
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
@@ -2460,7 +2513,7 @@ base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_sec
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
# writing to an offsite datastore.
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]})
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":0}]})
base.setdefault('local_api', {})
base['local_api'].setdefault('enable', True)
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
+57
View File
@@ -202,6 +202,63 @@ if wp is not None:
fail("manifests/webpage.yaml no longer tracks main for the website — pinning the SITE to "
"the installer tag turns every copy edit into a release")
# ── R-185: every path that RESOLVES the backup target must also grant on it ──────────────────
#
# THE DEFECT THIS WOULD HAVE CAUGHT, measured on both demo boxes 2026-08-03. `configure_backup_target`
# has two arms. The CASE A arm creates the storage and grants in the same breath. The Scenario-F arm —
# "the target already exists, leave it alone" — returned WITHOUT granting, so a box whose target
# pre-dated the install pointed `local_backup_target` at a storage its own token could not read. The
# API answered `{"data":[]}` while root saw three archives, and nothing said so, because an empty
# listing is also what a brand-new tier returns.
#
# The assertion is deliberately about the FUNCTION, not about PVE_STORAGES: the target's grant belongs
# with the target's resolution (PVE_STORAGES is granted a step earlier, before the target exists), so
# what must hold is that no arm of that function can resolve a target and skip the grant.
fn = re.search(r'^configure_backup_target\(\)\s*\{(.*?)^\}', src, re.S | re.M)
if not fn:
fail("cannot find configure_backup_target() — the backup-target ACL assertion cannot run, and a "
"check that cannot run must never report OK (R-185)")
else:
body = fn.group(1)
resolutions = len(re.findall(r'BACKUP_TARGET_RESOLVED="\$BACKUP_TARGET_ID"', body))
grants = len(re.findall(r'felhom-backup-target-apply grant', body))
if resolutions == 0:
fail("configure_backup_target no longer resolves BACKUP_TARGET_ID anywhere — re-read it")
elif grants >= resolutions:
ok("every arm that resolves the backup target also grants on it (%d resolution(s), %d grant(s))"
% (resolutions, grants))
else:
fail("configure_backup_target resolves the backup target in %d place(s) but grants in only %d "
"— an arm resolves a target the agent may not READ. That is R-185: the tier's archives are "
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
"like a brand-new tier." % (resolutions, grants))
# ── R-191: the OFFSITE tier must not arm a client-side prune ─────────────────────────────────
#
# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay
# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a
# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator
# was told the offsite backup had failed when it had succeeded.
#
# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched
# and must stay untouched — it prunes correctly and is allowed to.
m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S)
if not m:
fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention "
"assertion cannot run, and a check that cannot run must never report OK (R-191)")
else:
targets = m.group(1)
kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets)
if not kl:
fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 "
"(R-191: 0 means 'never prune from the box'; absent is not the same statement)")
elif kl.group(1) != "0":
fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite "
"pruning server-side to ep0 and box tokens are write-only, so every weekly run will "
"upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1))
else:
ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)")
print()
if fails:
print("hostinstall gates: %d FAILURE(S)" % len(fails))