admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

R-85 close-out: session REPORT + R-86/R-87 queued

Browse Source 
R-86 (NEXT, operator ruling 2026-07-27): backup-ALIGNED restore-test scheduling
— test a tier ~1 day after ITS OWN backup. R-85 schedules on a free-running
interval, which cannot express 'the day after the PBS backup': any fixed offset
drifts, so alignment would be luck. Shape: trigger from the tier's own last
successful backup rather than a clock. Interim in force: 302400s (3.5d), which
lands each tier ~weekly — the cadence half of the ruling, not the alignment half.

R-87: the restic app-data offsite tier is NEVER restore-tested. R-85 covers
whole-guest vzdump tiers only; the agent has no restic surface. That is arguably
the tier that matters most — the only one that survives losing the box AND
carries the customer's app data, since the whole-guest snapshot excludes the
bind-mounted drives. Exactly the state PBS was in before R-85.

REPORT.md: the full R-80 -> R-85 arc, including a section on the seven mistakes
I made and the two recurring shapes behind them (inferring behaviour from an
artifact instead of the code that consumes it; reading a result without its exit
code). Records demo-felhom's restore-test as IN FLIGHT at close, with the manual
recovery step if the deferred restart watcher does not complete.

Hub gate green (17 packages, rc=0).
This commit is contained in:
Claude Code
2026-07-27 08:11:19 +02:00
parent 73889e9fdf
commit 6d11c54a50
2 changed files with 222 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+220 -169
View File
@@ -1,194 +1,245 @@
# REPORT — R-82: the backup target split (2026-07-26)
# REPORT — R-80 → R-85: from a false alarm to an audible DR tier (2026-07-26 → 27)
Local **daily** + offsite **weekly**, made expressible at all. Spans four artifacts:
agent **v0.97.0 → v0.103.0**, controller **v0.174.0 → v0.175.0**, hub **v0.76.0**,
host-install **1.20.0**.
One long session. It began as a diagnostic into a nightly alarm and ended with the offsite DR tier
being **restore-proven unattended, with its failure audible**. Five roadmap items, four artifacts.
Phase-0 gates: `documentation/audits/SPIKE-r82-phase0-2026-07-26.md`.
| Repo | From → To |
|---|---|
| felhom-agent | v0.96.0 → **v0.104.0** |
| felhom-controller | v0.173.0 → **v0.175.0** |
| felhom.eu hub | v0.74.0 → **v0.77.0** |
| host-install | 1.19.0 → **1.20.0** |
All live on demo-felhom and demo-hp.
---
## 1. What was wrong
## 1. The arc, in one line each
`BackupTarget()` returned ONE string and `BackupCadence()` ONE 24 h window, so "local daily **and**
PBS weekly" could not be said. The consequence was not theoretical: the DR tier reported `applied`
since 2026-07-21 while demo-felhom held **one** snapshot (2026-07-18, a healing artifact) and demo-hp
held **zero, ever**. R-39 was "applied and dead"; this was **applied and empty** — the same shape,
quieter, and it would have surfaced first at a real restore.
## 2. Phase 0 — three gates
| Gate | Verdict |
| Item | What it was |
|---|---|
| P0.1 what is exposed for 7 days | **weekly CONFIRMED.** The only 7-day-exposed state is the non-SMB half of `settings.json`. `encryption.key` and the offbox credentials are **stable files unchanged since first boot**, so a week-old copy is byte-identical — that was the risk that could have overturned it |
| P0.2 the `pvesm status` 0/0/0 anomaly | **RESOLVED, benign.** PBS returns HTTP 200 with zeroed usage to a namespace-scoped token (`DatastoreBackup`, not `Datastore.Audit`). Ground truth via the hub's ep0 `df`: the datastore is real and writable |
| P0.3 capacity | **STOP raised; operator ruled to proceed and grow later.** 37.2 GB total. Per-tenant encryption means **no cross-customer dedup** |
| **R-80** | Diagnose `expected_backup_missed` firing on three boxes. **The premise was wrong on both counts** — it fired ONCE, not nightly, and no real external customer was notified |
| **R-81** | Fix the CLASS behind it: absence of a signal treated as evidence of failure. Third instance |
| **R-82** | The backup target split — "local daily + PBS weekly" was **not expressible at all**, which is why the DR tier was `applied` and empty |
| **R-84** | An agent restart no longer triggers a redundant backup |
| **R-85** | The DR tier is restore-tested **unattended**, and its failure is **heard** |
**Capacity, now measured rather than bracketed:** the second weekly snapshot cost **+2.7 GB on disk**
against 14.46 GB logical (~81 % dedup). Weekly top-ups are cheap; **first** snapshots are not — one
customer at two retained snapshots costs ~13.5 GB, so the 80 % warn arrives at roughly the **first**
additional customer, not the second as I first estimated. Recorded in `07-backup-architecture.md`
§9.1.
---
## 3. What shipped
## 2. R-80 — the diagnosis, and what it actually found
- **Agent** — `backup_targets[]`: each tier carries its **own** cadence, retention and wait bound
(`keep-last=3` is three DAYS on a daily tier and three WEEKS on a weekly one; one shared knob
guarantees one of them is wrong). `/backup/due?target=` judges a tier against **its own** newest
successful backup. `GET /backup/tiers` is the controller's capability probe. One runner per tier.
- **Controller** — every due tier collected up front and run in **ONE quiesce window**. Two cycles on
the weekly night would mean two app outages for one night's work. The app stays quiesced until the
**last** tier snapshots, so every tier is app-consistent.
- **Hub** — per-tier thresholds (host 26 h, offsite 8 d), preserving R-81's three-valued verdicts,
anchored absence and distinct reason strings. Classification is by **target type**
(`target_id` → `storage_targets[].name` → `.type`), never by array membership.
- **Installer** — a fresh box defaults to local-daily + offsite-weekly; an unprovisioned tier
**defers** rather than firing at a storage that does not exist.
The alarm fired at 2026-07-26 03:00 UTC on demo-felhom, demo-hp and drill-r50. The brief described it
as nightly, on three customers, one reaching a customer channel with a 7.3-day staleness claim.
**The untargeted local-API contract is frozen.** No `?target=` ⇒ the primary tier, same response
**bytes** (`Target` is `omitempty` and stays empty). An old controller cannot tell the new agent from
the old one; a new controller against an old agent degrades on a 404 probe, logs once, and **still
takes the backup**.
**It fired exactly once.** The check ran and was correctly silent on every prior night. The one
customer-channel mail went to the operator's own mailbox; `peti-felhom` — the only real external
customer — did not fire at all. And the 7.3 days was the age of the **PBS** snapshot, reached only as
a fallback: the local tier had run on 07-24, 07-25 and 07-26.
## 4. Operator rulings (2026-07-26), all implemented
**Root cause:** the agent's backup store is in-memory (`"lost on restart; the cadence re-populates"`).
The R-50 fleet restart at 12:44 UTC emptied `backups[]`; the next backup landed at 07:03 the following
morning; the 03:00 check fell in that ~18 h blind window and read empty as *no backup exists*.
| Ruling | Implementation |
**One real finding stood:** the PBS/offsite tier had **no schedule at all**. demo-felhom held one
snapshot from a healing artifact; demo-hp held **zero, ever**, while reporting `applied` since 07-21.
---
## 3. R-81 — fixing the class, not the instance
`assessBackupFreshness` collapsed *absence of records* into *failure*. It now returns
**OK / UNKNOWN / MISSED**: absence is UNKNOWN until it outlives a window anchored at first contact.
The anchor was **free** — a Phase-0 probe found the hub already retains 90 days of host-reports, so
`GetHostReportsSince` + `newestBackupEvidence` answer *"when did I last SEE evidence of a backup?"*
rather than *"what does the latest report say?"*. No agent change, no new state.
**Not silence.** A genuinely dead box must still alarm — that is the half the naive fix breaks, and
`TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms` is what makes the suppression safe.
The invariant is now written at the head of the function naming all three instances (v0.12.0,
v0.73.0, R-81), pinned by a boundary test whose name says what it protects.
---
## 4. R-82 — the target split
`BackupTarget()` returned ONE string and `BackupCadence()` ONE 24 h window.
**Phase 0 gates:** weekly CONFIRMED (the only 7-day-exposed state is the non-SMB half of
`settings.json`; `encryption.key` and the offbox credentials are **stable files unchanged since first
boot**, so a week-old copy is byte-identical). The `pvesm status` 0/0/0 anomaly resolved as a
namespace-scoped-token reporting artifact. Capacity raised a **STOP**, which the operator ruled past.
**Measured, not bracketed:** the second weekly snapshot cost **+2.7 GB** against 14.46 GB logical
(~81 % dedup). Weekly top-ups are cheap; **first** snapshots are not. Per-tenant encryption means
**no cross-customer dedup**, so the 80 % warn arrives at roughly the first additional customer.
**Shipped:** per-tier cadence/retention/wait-bound (agent); **one quiesce window for both tiers**
(controller — two cycles would mean two app outages for one night's work); per-tier thresholds
(hub, host 26 h / offsite 8 d); fresh-install default (installer).
### Four defects found by RUNNING it, not reviewing it
1. **v0.98.0** — a 41-minute backup recorded `success:false` at 30 minutes **while still running**,
then completed `TASK OK`. Worse than "didn't happen": the tier stays due and the retry hits the
guest lock.
2. **v0.100.0** — the restore tier read from the configured target, not the archive. **A silent
regression of the S4.1 fix** — the mechanism was never removed; its *input* changed when
`local_backup_target` was retargeted to `local`.
3. **v0.101.0** — a leaked scratch kept `onboot: 1`; a host reboot would have started a clone of the
live guest.
4. **v0.102.0** — a tier fired at a storage that does not exist yet.
---
## 5. R-84 — ground truth instead of memory
Three redundant local backups ran in one afternoon of deploys, because a restart empties the store.
On the offsite tier that is a wasted multi-hour upload after **every** agent deploy.
Fixed by **asking the storage**, not persisting the record: a pruned archive correctly stops counting,
where a persisted record would keep claiming a backup that no longer exists. Proven live on both
boxes with the store cold.
---
## 6. R-85 — unattended, and audible
Three defects, each verified at source: the scheduler only ever saw `BackupTarget()`; the spec was
**frozen at daemon start** (an immediately-invoked function — also a latent staleness bug); and a
failed restore-test was a `[WARN]` line with **no event, no notification, no gauge** — true for the
local tier that *was* being tested.
**Ruling (operator): oldest-first.** Never-proven sorts first, which is where the offsite tier starts.
Rotation credit **only on success**, or a permanently failing tier looks freshly proven and stops
being retried. State persisted — unlike R-84 there is no ground truth, because a restore-test destroys
its scratch and leaves no artifact.
**`backup.InFlight`** — a host-wide one-heavy-operation gate. Not a lock concern (the scratch VMID
never touches the live guest's vzdump lock) but a **LINK** one: an offsite restore PULLS multi-GB over
the tunnel a backup PUSHES one. Callers **defer**; they never cancel.
**Two hub signals, never merged:** `restore_test_failed` (broken now) and `restore_test_stale`
(*unverified*, not known-broken). Anchored on R-81, operator-tier only.
---
## 7. Live evidence
**The restore round-trip PASSED** (demo-hp, manual):
```
source_tier: pbs pass: true verified: boot+running mount_parity: ok 4m5s, clean teardown
```
`mount_parity` is the non-hollow half — a boot-only verify cannot see a missing data volume.
**The multi-tier quiesce ran** through the real UI endpoint (authed + CSRF):
exactly **ONE stop/start pair with BOTH backups inside it**, app quiesced through the non-last tier,
early resume on the last tier's snapshot. **Downtime 1m27s for both tiers.**
**Rotation proven unattended** (demo-hp):
```
tier selected (oldest-proven first) target=felhom-pbs ← never-proven sorted first
scheduled restore-test passed duration_s=194.8
tier selected (oldest-proven first) target=local ← rotated
```
**The signal fired on its first sweep — on real faults:**
```
[ERROR] drill-r50 Restore-test FAILED on the local tier … HTTP 403 missing privilege VM.Backup
[WARN] demo-felhom pbs tier: NEVER restore-proven in 205h (limit 168h) — unverified, not known-broken
[INFO] demo-hp pbs tier: watching 133h of a 168h grace — newborn, not a fault
```
The first two are genuine, previously-invisible failures. All notifications went to the **operator**
channel; the DB confirms no customer routing.
**demo-hp's first ever offsite backup landed** — 4.25 GB into a verifiably empty namespace.
---
## 8. Mistakes I made, and what they cost
Recorded because the pattern matters more than any one of them.
1. **I claimed the restore-test would break the control plane** by booting a network-conflicting
clone. **Wrong** — it link-downs every NIC before boot, and that is unit-tested. I read a config
artifact and inferred behaviour without reading the code that consumes it, *then escalated before
finishing the check*. I also disabled a safety mechanism on that basis; it is re-enabled.
2. **I pushed one commit red** — read `packages ok: 28` and missed `rc=1` in the same command. Five
of my own tests were failing.
3. **I planted a time bomb in Slice C** — a test hard-coded the incident timestamp
`2026-07-18T18:31:06Z` while comparing against the real clock. It passed all day and began failing
at exactly 18:31 UTC, 8 days later. A test that passes at commit time and fails hours later is
worse than one that fails immediately.
4. **I left `REPORT.md` claiming "not deployed"** 26 minutes before deploying, and did not update it.
5. **Two bad observables** while verifying the controller: read a UTC timestamp as host time, then
checked for log lines the agent does not emit at INFO. Both produced confident wrong readings.
6. **I told the operator blocking would not silence the new alerts.** Wrong — blocked customers are
excluded by `GetActiveCustomerIDs` already.
7. **My own spec's §5.3 was wrong** — it said to force a failure with `--selftest`, but the selftest
runs in a separate process and never reaches a host-report.
The recurring shape: **inferring behaviour from an artifact instead of reading the code path that
consumes it**, and **reading a result without reading its exit code**. The host=CEST /
component=UTC mismatch caught me twice in one session.
---
## 9. State at close
| | |
|---|---|
| Two weeks of offsite backups | `keep_last=2`; the blanket PBS-prune refusal scoped to *additional* tiers with an explicit setting — the primary keeps the absolute refusal, because its target **and** retention both default and could prune the DR by accident |
| Grow the datastore before any real tester | recorded in `07` §9.1; no action taken |
| First backup runs as long as needed; nothing else starts until done | wait bound → 12 h (measured ~5 h for a first full snapshot); **one backup at a time per guest** — a second tier gets a 409 naming the busy tier, with no job id it could mistake for its own; a tier overrunning the quiesce bound defers the rest |
| Drill box is temporary | dropped from the rollout |
| Restore test, then next slice | done — see §6 |
| Backups | local daily + offsite weekly, both boxes |
| Offsite retention | 2 weeks (`keep_last=2`) |
| Restore-test | **rotating both tiers, unattended**; interim cadence 3.5 d → each tier ~weekly |
| Signals | failure + staleness, operator-only |
| drill-r50 | VM 300 **stopped**, `onboot=0`. Operator will block the customer |
## 5. Four defects found by RUNNING it, not reviewing it
### ⏳ IN FLIGHT AT CLOSE — check on this
1. **30-minute wait bound vs a 41-minute backup** (v0.98.0). The agent recorded `success:false`
**while the vzdump was still running**, and it later completed `TASK OK`. Not "the backup didn't
happen" but worse: the tier stays permanently due and the retry collides with the guest lock.
2. **The restore tier read from the configured target, not the archive** (v0.100.0). A `felhom-pbs:`
archive was classified `local` and got the 10-minute bound against a 14.46 GB WAN restore, failing
at 600 s. **This was a silent regression of the S4.1 fix** — the mechanism was never removed, its
*input* changed when `local_backup_target` was retargeted to `local`. The lesson is not "add a
timeout" (one was already there) but that a fix keyed on *"the configured target"* stops holding
the moment more than one target exists. Recorded in `06-offsite-connectivity.md`.
3. **A leaked scratch guest kept `onboot: 1`** (v0.101.0) — a host reboot would have started a clone
of the live guest. Now `onboot=0` is set **at restore time**, because "after" is the path that
leaks.
4. **A tier fires at a not-yet-provisioned storage** (v0.102.0) — would have quiesced the apps and
failed every cadence on a fresh box until DR provisioning.
**demo-felhom's offsite restore-test is still running.** Its archive is 14.46 GB, so expect roughly
two hours. Scratch guest `990000` is on the box while it runs; a clean finish tears it down.
## 6. Live validation
Its agent still has the **compressed 10-minute cadence in memory** — the reverted 3.5-day value is on
disk and takes effect at the next restart. A watcher is set to restart the agent once the test
finishes. **If that watcher did not complete, restart `felhom-agent` on demo-felhom manually** or the
box keeps restore-testing every 10 minutes.
**demo-felhom** — the first real PBS-targeted backup: **`TASK OK`, 41 minutes, 14.46 GB snapshot**,
and it **restored cleanly** (`vzrestore: stopped OK`, all volumes back). Both tiers armed and
verified over the real local API; the untargeted response confirmed byte-identical.
When it passes, demo-felhom's `restore_test_stale` clears on its own.
**demo-hp** — reached via the documented break-glass path; binary and config backed up first.
Its **first ever** PBS backup **landed**: 4.25 GB into a namespace that was verifiably empty.
---
**THE RESTORE ROUND-TRIP PASSED** — the bar for calling a tier real:
## 10. NOT done — explicitly
```
source_archive : felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z
source_tier : pbs <- the v0.100.0 fix; the earlier attempt said "local" and died at 600s
pass : true
verified : boot+running
mount_parity : ok <- mp0=/var/lib/docker 50G, mp1=/mnt/sys_drive 20G, mp8/mp9 stand-ins
duration : 4m5s (restore + boot + verify + teardown)
```
`mount_parity` is the non-hollow half: a boot-only verify cannot see a missing data volume. The
scratch guest tore down **cleanly** — no 403, no leak — which confirms `06-offsite-connectivity.md`'s
reading that the teardown 403 was a **phantom** (a consequence of the short timeout, not an ACL gap),
and corrects my earlier framing of it as a standing privilege gap. Afterwards: scratch band empty,
thin pool back to its exact pre-restore figure, live guest running, snapshot intact.
**THE MULTI-TIER QUIESCE RAN LIVE**, through the real UI endpoint (`POST /api/guest-backup/trigger`,
session auth + CSRF — the exact call the "Mentés most" button makes):
```
17:01:39 manual backup requested — quiescing now
17:01:39 backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx] <- ONE stop
17:01:46 tier local: backup job ... started
17:02:56 tier local: ... done — next tier may start (app still quiesced) <- app stays DOWN
17:02:56 tier felhom-pbs: backup job backup-9201-felhom-pbs-... started
17:03:06 tier felhom-pbs: ... snapshotted — resuming app early (8B.2)
17:03:06 unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s) <- ONE start
```
**Exactly one stop/start pair with both backups inside it** — the assertion that matters, since
"both backups ran" would also pass against an implementation that quiesces twice. Tier order was
local-first/PBS-last as designed, the app stayed quiesced through the non-last tier (app-consistency
preserved on the DR tier), and it resumed at the **last** tier's snapshot rather than waiting for the
upload. **Total app downtime 1m27s for both tiers**; paperless came back healthy.
**Hub Slice C replayed against the live DB before deploying:**
```
demo-felhom host=07-26T14:38Z offsite=07-26T12:21Z -> OK
demo-hp host=07-26T07:06Z offsite=none -> UNKNOWN (119h of a 192h grace)
drill-r50 host=none offsite=not expected -> MISSED (no evidence in 29h)
```
**No customer email results from the deploy.** demo-hp defers correctly and will alarm in ~3 days if
its offsite tier stays empty — the true finding arriving on schedule, not a false alarm.
## 7. A correction I had to make mid-arc
I reported that the restore-test would boot a scratch guest carrying the live guest's MAC, static
island IP and hostname, and so would break the controller→agent link. **That was wrong.**
`RunRestoreTest` step 2 link-downs **every** interface before the guest is started, and it is
unit-tested. I read a restored config artifact, inferred the boot behaviour from it, and escalated
before reading the code path that consumes it. I also disabled the scheduled restore-test on that
basis, which was an unnecessary reduction in safety coverage; it is re-enabled.
The residual hazard was real but far narrower — it needed the restore to fail *before* the link-down
step, which is what defect 1 caused — and that is what v0.101.0 fixes.
Separately, `06-offsite-connectivity.md` records that the teardown `403` I flagged as a standing
privilege gap was **already diagnosed in S4.1 as a phantom**: it is a consequence of the short
timeout, not an ACL problem. With the timeout fixed the guest is pool-associated by teardown time.
## 8. Tests
| Repo | Result |
|---|---|
| felhom-agent | `build/vet/test` rc=0, **29 packages** |
| felhom-controller | `build/vet/test` rc=0, **27 packages** |
| felhom.eu (hub) | `build/vet/test` rc=0, **17 packages** |
Red-proofs observed and restored for every mandatory scenario: old-controller compat, new-controller
degrade (the hollow version asserts "no error" while silently skipping the backup), the both-due
night (**the COUNT is the assertion** — asserting only "both ran" passes against a double-quiesce),
the merged threshold, the per-tier wait bound, the `onboot` override, and the overrun defer.
**A process failure worth recording:** I ran the agent suite and committed in the same command, read
`packages ok: 28`, and pushed **without reading `rc=1`**. Five of my own Slice A tests were failing —
a harness artifact, not a product bug, but the commit went out red. Fixed in `13ca2d9`. This is the
exact exit-code trap recorded twice earlier in this arc.
## 9. NOT done — explicitly
1. **The offsite tier is never AUTOMATICALLY restore-tested.** The scheduled restore-test picks
candidates from a runner built on the primary target, so it can never select a PBS archive. The
**manual/selftest** path is now proven end-to-end; the **unattended** one is not. Needs a per-tick
spec.
2. **The hub infers "PBS ⇒ weekly" from storage TYPE.** `defaultBackupTarget` is `felhom-pbs`, so a
box that never sets `local_backup_target` would run PBS as its **daily** tier and be judged
against 8 days — seven days of blindness. No box is in that shape today. The real fix is the agent
reporting each tier's actual cadence.
3. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a
genuine gate, not an oversight.
5. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
1. **The restic (app-data offsite) tier is NEVER restore-tested** → **R-87**. R-85 covers whole-guest
vzdump tiers only. This is arguably the tier that matters most: the only one that survives losing
the box **and** carries the customer's app data (the whole-guest snapshot excludes the
bind-mounted drives). It is exactly the state PBS was in yesterday.
2. **Restore-tests are interval-scheduled, not backup-aligned** → **R-86** (operator ruling
2026-07-27: ~1 day after that tier's own backup).
3. **Rotation observed on demo-hp only**, and under a compressed cadence. It proves the rotation
logic, not the production interval.
4. **The hub infers "PBS ⇒ weekly" from storage TYPE**, not a reported cadence. No box is in the
wrong shape today.
5. **The installer-default fleet flip** awaits a full weekly cycle.
6. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
ratification is Viktor's review of the §10 list.
7. **The customer-facing Hungarian copy still overstates scope** — unchanged since R-80 flagged it.
8. **Capacity:** the datastore still needs growing before the first real customer.
## 10. Observations
---
- The `felhom-pbs` PVE storage will permanently show **0 %** in the PVE UI (namespace-scoped token).
Operators must read fill from the hub's PBS-DR gauge. Worth a runbook line.
- demo-felhom's guest grew **9.74 → 14.46 GB logical in eight days**. Probably one-off from app
## 11. Observations
- **drill-r50's 403** (`missing privilege VM.Backup at /vms/9100`) is real and was invisible. The box
is being retired, so it needs a decision — fix or accept — not silence.
- **A blocked customer is hidden from the Dashboard and from every checker**, and it is reversible
(`configs.go` sets status back to `active`, with a UI affordance). Blocking is a blunt instrument
though: it silences *everything*, which is why the operator is building per-alert silencing.
- **The PVE UI will permanently show `felhom-pbs` at 0 %** (namespace-scoped token). Read fill from
the hub's PBS-DR gauge.
- **demo-felhom's guest grew 9.74 → 14.46 GB logical in eight days.** Probably one-off from app
testing, but if it is a rate the capacity sizing changes quickly.
- demo-felhom was enabled **before** demo-hp, out of the specified rollout order, because Slice A
could not be validated otherwise.
- The customer-facing Hungarian copy still overstates scope ("A mai biztonsági mentés nem készült el
a határidőig!" covers only the host/PBS tier). Unchanged; flagged since the R-80 diagnostic.
- **Three of the five items in this arc were the same bug class** — absence of a signal read as
evidence, or a correct mechanism whose input changed underneath it. Both are now written into the
code as invariants rather than left as lore.