3696188636
hub 0.81.0, agent 0.112.0, controller 0.184.1 live on BOTH demo boxes. E-2c: eject/decommission of the backup-target drive refused 409 on both boxes, drives unmoved. E-2b: the never-called disconnect seam is wired, with the target case raising the specific backup_target_absent. Records the keying bug caught before deploy (a.Path is the GUEST path, so the target branch was unreachable -- 0.184.0 superseded, never deployed) and states plainly that backup_target_absent is NOT proven end-to-end live: proving it needs a live enrolled drive to go absent. Parts 2/3/4 and E-2a remain open; Peti risk stays parked.
112 lines
6.4 KiB
Markdown
112 lines
6.4 KiB
Markdown
# REPORT — E-2 increment 1: the backup target is now defended and audible (2026-07-29)
|
|
|
|
**Overwritten** per the standing rule. **E-2 is PARTIALLY shipped** — this increment is Parts 1 + 5
|
|
plus findings E-2b/E-2c. Parts 2, 3, 4 and E-2a remain open (see §6).
|
|
|
|
**Live fleet after this run:** hub **0.81.0**, agent **0.112.0** (both boxes), controller **0.184.1**
|
|
(both boxes). `peti-felhom` deliberately untouched.
|
|
|
|
---
|
|
|
|
## 1. Why this order
|
|
|
|
The operator reordered the plan, correctly: **E-2b and E-2c protect a configuration that exists
|
|
right now; E-2a builds for boxes that do not.** Both demo boxes are configured, Peti is parked, and
|
|
no new installs are happening this week — so Part 3 has no user yet, while E-1's new configuration
|
|
had a drive that could be ejected without warning and no event when it went absent.
|
|
|
|
## 2. E-2c — the backup drive could be ejected out from under the backup
|
|
|
|
`RoleForStorage` types a `local-dir` on a non-system device as **user-data**, so the pre-existing
|
|
eject role gate PASSED the drive E-1 had just made the vzdump target. `POST /disks/eject` on
|
|
`/mnt/nvme-1tb` or `/mnt/hdd_1` would have **succeeded silently**, taking the only local whole-guest
|
|
backup with it while the box kept reporting a configured tier.
|
|
|
|
Fixed in agent v0.111.0/0.112.0: `handleDiskEject` and `handleDiskDecommission` refuse `409` when the
|
|
mount backs a configured tier, naming the storage and the remedy — the op is *ordered*, not
|
|
forbidden.
|
|
|
|
**Deliberately NOT a role reclassification.** Making `RoleForStorage` return `RoleBackup` would
|
|
refuse every legitimate eject of the customer's own data drive, because on both boxes that drive
|
|
**is** the target. `TestEjectStillAllowedOnANonTargetDrive` pins that, and the over-correction is
|
|
red-proofed.
|
|
|
|
**Live on both boxes:**
|
|
|
|
```
|
|
demo-hp felhom-backup mount=/mnt/nvme-1tb role=user-data backup_target=True
|
|
demo-felhom felhom-backup mount=/mnt/hdd_1 role=user-data backup_target=True
|
|
```
|
|
|
|
That single line is the exposure: `role=user-data` (which the gate permits) yet `backup_target=True`.
|
|
Both ejects returned **409** with the full message; both decommissions **409**; both drives still
|
|
mounted with `dump/` intact — the refusal had no side effect. `/var/lib/vz` is still refused by the
|
|
**pre-existing** role gate (`role: system`), proving the new gate is narrow.
|
|
|
|
## 3. E-2b — the alarm that was defined and never called
|
|
|
|
`NotifyStorageDisconnected` and `NotifyStorageReconnected` were registered in `allowedEventTypes`, in
|
|
`DefaultEnabledEvents`, and given Hungarian hub copy — and **called from nowhere**. A drive going
|
|
absent produced stopped apps, a `[WARN]` log and a UI badge, then silence on every channel. A drive
|
|
that is *only* a backup target has no apps to stop, so it was silent twice over. Fifth instance of
|
|
this class; found by Phase 0, not by a failure.
|
|
|
|
Now wired in `ReconcileDriveGates`, with the target case raising the more specific
|
|
`backup_target_absent` (error) and recovering as `backup_target_restored` (info — the pairing-gated
|
|
pattern; `severityNotifies` NOT widened). Never both: two mails for one event trains people to
|
|
ignore the channel.
|
|
|
|
## 4. The bug I introduced and caught before deploy
|
|
|
|
`ReconcileDriveGates` resolves the target as `isTarget[a.Path]`, and **`a.Path` is the registered
|
|
GUEST path** (`/mnt/felhom-drives/<name>`), not the agent's host `MountPath`. I had keyed the map on
|
|
`MountPath` alone, so the backup-target branch was **unreachable** — every absent drive, the target
|
|
included, would have fallen through to the generic event.
|
|
|
|
It would have looked wired, passed its own unit tests, and shipped silently wrong on exactly the
|
|
drive it exists for. Caught by tracing `a.Path` back to its source instead of assuming. Fixed in
|
|
v0.184.1 (both keys, mirroring `planDriveGates`), red-proofed, and the **0.184.0 image is superseded
|
|
and was never deployed**.
|
|
|
|
## 5. Tests and red-proofs
|
|
|
|
| Red-proof | Observed failure |
|
|
|---|---|
|
|
| hub allowlist entry removed | `backup_target_absent must be in allowedEventTypes, or POST /event 400s…` |
|
|
| hub Hungarian copy shortened | `must state the consequence…, got: "Meghajtó hiányzik."` |
|
|
| C — auto-elevation on register | `registering drives assigned the backup target "/mnt/hdd_1"` |
|
|
| E-2c — eject guard removed | `eject of the backup-target drive SUCCEEDED (200)` |
|
|
| E-2c over-correction | the gate blocked `/mnt/spare` (a non-target) |
|
|
| keying reverted to MountPath | `not resolvable by its GUEST path — …the branch would never fire` |
|
|
|
|
Every mutation was **verified to have landed** before the result was trusted. The first attempt
|
|
silently no-op'd (gofmt had realigned the map) and the test "passed" — a false proof.
|
|
|
|
Controller notify tests observe the **wire** (a real `Notifier` against an `httptest` hub), not a
|
|
mock, because the failure class is "nothing arrives".
|
|
|
|
## 6. What is NOT done
|
|
|
|
- **Part 2** (installer Case A/B), **Part 3** (the offer + the agent-side move), **Part 4** (the
|
|
degraded Hungarian banner). Phase 0 inverted their emphasis: the installer has **no drive-enrollment
|
|
step**, so at install the common case is system-drive-only and **Part 3 is the primary path**.
|
|
- **E-2a** — the guarded wrapper. The agent holds neither `Datastore.Allocate` at `/storage` nor
|
|
`Permissions.Modify`, and its sudoers has no `pvesm`/`pveum`. Use the wrapper pattern; **do not
|
|
widen the agent's PVE role.**
|
|
- **`backup_target_absent` is not proven END-TO-END live.** Its components are: the controller emits
|
|
the right type/severity (proven at the wire), the hub allowlists it (unit-pinned, image deployed),
|
|
and the keying is pinned. Proving the whole chain needs a live drive to actually go absent, which
|
|
means unmounting an enrolled drive carrying customer data and the guest bind on a remote box —
|
|
the same thing declined in E-1, for the same reason.
|
|
- Full drive-loss recovery (physical access) — carried forward from E-1.
|
|
|
|
## 7. Observations
|
|
|
|
- **`peti-felhom` untouched.** Its whole-guest backup still shares a device with its guest, so a
|
|
drive failure there is offsite-only recovery. Accepted until the operator's reinstall; re-evaluate
|
|
if that slips past ~2026-09-01.
|
|
- The fleet was briefly split (demo-hp updated first). Left that way it would have protected one of
|
|
two identically-exposed boxes, so demo-felhom was rolled to the same proven artifacts.
|
|
- `StorageTarget.Role` carries a documented vocabulary including `vzdump-target` and is always `""`.
|
|
Checked — it is a deliberate unpopulated field ("hub-owned, slice 10"), **not** a sixth dead seam.
|