E-2 increment 1: report + close E-2b/E-2c as shipped and proven live
hub 0.81.0, agent 0.112.0, controller 0.184.1 live on BOTH demo boxes. E-2c: eject/decommission of the backup-target drive refused 409 on both boxes, drives unmoved. E-2b: the never-called disconnect seam is wired, with the target case raising the specific backup_target_absent. Records the keying bug caught before deploy (a.Path is the GUEST path, so the target branch was unreachable -- 0.184.0 superseded, never deployed) and states plainly that backup_target_absent is NOT proven end-to-end live: proving it needs a live enrolled drive to go absent. Parts 2/3/4 and E-2a remain open; Peti risk stays parked.
This commit is contained in:
@@ -1,96 +1,111 @@
|
||||
# REPORT — the local whole-guest backup moved off the guest's own device (2026-07-28)
|
||||
# REPORT — E-2 increment 1: the backup target is now defended and audible (2026-07-29)
|
||||
|
||||
**Overwritten** per the standing rule (the prior C9-F1/C9-F2 text stays in git history).
|
||||
**Supervised operational run — no code, no version bump.** Fleet unchanged: hub v0.80.0,
|
||||
agent v0.110.0, controller v0.183.0. `peti-felhom` untouched. **Nothing was deleted.**
|
||||
**Overwritten** per the standing rule. **E-2 is PARTIALLY shipped** — this increment is Parts 1 + 5
|
||||
plus findings E-2b/E-2c. Parts 2, 3, 4 and E-2a remain open (see §6).
|
||||
|
||||
Record: `documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`.
|
||||
**Live fleet after this run:** hub **0.81.0**, agent **0.112.0** (both boxes), controller **0.184.1**
|
||||
(both boxes). `peti-felhom` deliberately untouched.
|
||||
|
||||
---
|
||||
|
||||
## What changed
|
||||
## 1. Why this order
|
||||
|
||||
On both demo boxes the primary backup tier moved from `local` — a `dir` storage on `/var/lib/vz`,
|
||||
i.e. **the same physical device as the guest itself** — to **`felhom-backup`**, a `dir` storage on
|
||||
the secondary drive.
|
||||
The operator reordered the plan, correctly: **E-2b and E-2c protect a configuration that exists
|
||||
right now; E-2a builds for boxes that do not.** Both demo boxes are configured, Peti is parked, and
|
||||
no new installs are happening this week — so Part 3 has no user yet, while E-1's new configuration
|
||||
had a drive that could be ejected without warning and no event when it went absent.
|
||||
|
||||
| | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| Target drive | `/dev/sdb` USB HDD → `/mnt/hdd_1` | `/dev/nvme0n1` → `/mnt/nvme-1tb` |
|
||||
| Durable id | `uuid:47a3361a-91e0-4831-a69d-27f540ed3f48` | `uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae` |
|
||||
| Archive proven | 5,957,878,962 B | 2,256,044,492 B |
|
||||
| Restore-test | `pass`, **`mount_parity: ok`**, 1 m 24 s | `pass`, **`mount_parity: ok`**, 1 m 54 s |
|
||||
## 2. E-2c — the backup drive could be ejected out from under the backup
|
||||
|
||||
Three steps per box: `pvesm add dir … --is_mountpoint 1`, two `pveum acl modify`, and a one-line
|
||||
`local_backup_target` edit in `agent.json`.
|
||||
`RoleForStorage` types a `local-dir` on a non-system device as **user-data**, so the pre-existing
|
||||
eject role gate PASSED the drive E-1 had just made the vzdump target. `POST /disks/eject` on
|
||||
`/mnt/nvme-1tb` or `/mnt/hdd_1` would have **succeeded silently**, taking the only local whole-guest
|
||||
backup with it while the box kept reporting a configured tier.
|
||||
|
||||
**Drive loss is now locally recoverable in principle on both boxes** — the whole-guest archive lives
|
||||
on different hardware from the guest, and a restore from it boots and passes mount parity. The proof
|
||||
that closes matrix row 4 (pull the system drive and restore for real) needs physical access and is
|
||||
outstanding.
|
||||
Fixed in agent v0.111.0/0.112.0: `handleDiskEject` and `handleDiskDecommission` refuse `409` when the
|
||||
mount backs a configured tier, naming the storage and the remedy — the op is *ordered*, not
|
||||
forbidden.
|
||||
|
||||
## Phase 0 verdict
|
||||
**Deliberately NOT a role reclassification.** Making `RoleForStorage` return `RoleBackup` would
|
||||
refuse every legitimate eject of the customer's own data drive, because on both boxes that drive
|
||||
**is** the target. `TestEjectStillAllowedOnANonTargetDrive` pins that, and the over-correction is
|
||||
red-proofed.
|
||||
|
||||
**The target is configuration, not converged — the runbook held and the §3 STOP did not fire.**
|
||||
Exactly one writer of `agent.json` exists (`pbsdr.seedEscrowStorageID`); it touches only
|
||||
`escrow.pbs_storage_id` and preserves unknown keys verbatim via a `map[string]json.RawMessage`
|
||||
read-modify-write. Nothing on the hub pushes agent config. No consumer hardcodes `"local"` on the
|
||||
backup path — runners are one-per-tier, `NewestArchiveTime` reads its own target, and
|
||||
`restoreTierForArchive` classifies from the archive, not from config.
|
||||
**Live on both boxes:**
|
||||
|
||||
## What the run found
|
||||
```
|
||||
demo-hp felhom-backup mount=/mnt/nvme-1tb role=user-data backup_target=True
|
||||
demo-felhom felhom-backup mount=/mnt/hdd_1 role=user-data backup_target=True
|
||||
```
|
||||
|
||||
- **F-1 (before any command).** The storage `path` must **be** the drive's mountpoint. A
|
||||
subdirectory fails `exactMount` (`internal/storage/observe.go:321`), so the target would report
|
||||
`disconnected` **permanently** and its durable id would degrade off the filesystem UUID.
|
||||
- **F-2 (before any command).** `--is_mountpoint 1` is load-bearing. **Proven live:** an unguarded
|
||||
`dir` storage on a non-mounted path reports `active`, advertises the **root filesystem's** free
|
||||
space, and had already created `dump/` on `/dev/mapper/pve-root` — a silent retarget onto the
|
||||
exact device this change exists to escape. The guarded one refuses outright.
|
||||
- **F-3 (found by the first real backup).** `FelhomAgentStore` is granted **per storage path**; a
|
||||
new target without its own grant 403s every backup. **`felhom-host-install.sh` must issue it for
|
||||
new installs**, or a new box ships with a tier that fails on its first run.
|
||||
- **R-109 (new).** The DR recipe records **no backup target** — and each box now carries two
|
||||
`content=backup` dir storages, one live and one holding frozen 2026-07-28 archives.
|
||||
- **R-105 narrowed and traced.** `dr_recipe.host_half.drives` was `[]` fleet-wide with the cause
|
||||
untraced. Cause: the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them.
|
||||
Both demo boxes now populate `drives`, and the backup drives gained SMART reporting. R-105's other
|
||||
two fields are untouched.
|
||||
That single line is the exposure: `role=user-data` (which the gate permits) yet `backup_target=True`.
|
||||
Both ejects returned **409** with the full message; both decommissions **409**; both drives still
|
||||
mounted with `dump/` intact — the refusal had no side effect. `/var/lib/vz` is still refused by the
|
||||
**pre-existing** role gate (`role: system`), proving the new gate is narrow.
|
||||
|
||||
## Absent-drive behaviour (Part 3)
|
||||
## 3. E-2b — the alarm that was defined and never called
|
||||
|
||||
Today: **fail loudly, no silent retarget.** The PVE half is live-proven with throwaway storages (no
|
||||
live drive was unmounted). The agent half is source-traced: `targetStoragePresent` checks name
|
||||
presence only, never `Reachable`, so the tier stays DUE, the controller quiesces, vzdump is refused,
|
||||
and the run fails and alarms.
|
||||
`NotifyStorageDisconnected` and `NotifyStorageReconnected` were registered in `allowedEventTypes`, in
|
||||
`DefaultEnabledEvents`, and given Hungarian hub copy — and **called from nowhere**. A drive going
|
||||
absent produced stopped apps, a `[WARN]` log and a UI badge, then silence on every channel. A drive
|
||||
that is *only* a backup target has no apps to stop, so it was silent twice over. Fifth instance of
|
||||
this class; found by Phase 0, not by a failure.
|
||||
|
||||
**This is not §6's intended design** (fall back to the system drive and alarm). There is no fallback
|
||||
at all, so an absent drive means no local backup until a human intervenes. Filed as **E-2**, together
|
||||
with the honest single-drive label — a one-drive box protects against corruption only, and two
|
||||
drives is effectively a hardware requirement for drive-loss protection.
|
||||
Now wired in `ReconcileDriveGates`, with the target case raising the more specific
|
||||
`backup_target_absent` (error) and recovering as `backup_target_restored` (info — the pairing-gated
|
||||
pattern; `severityNotifies` NOT widened). Never both: two mails for one event trains people to
|
||||
ignore the channel.
|
||||
|
||||
## One operational error, reported in full
|
||||
## 4. The bug I introduced and caught before deploy
|
||||
|
||||
**The agent was restarted on demo-hp with a `felhom-pbs` backup in flight**, against §4.3. The
|
||||
in-flight check was done before the first restart and not repeated before the second. It produced a
|
||||
**spurious tier failure** (`context canceled` while waiting) — the F-A1 class the project already
|
||||
fixed once.
|
||||
`ReconcileDriveGates` resolves the target as `isTarget[a.Path]`, and **`a.Path` is the registered
|
||||
GUEST path** (`/mnt/felhom-drives/<name>`), not the agent's host `MountPath`. I had keyed the map on
|
||||
`MountPath` alone, so the backup-target branch was **unreachable** — every absent drive, the target
|
||||
included, would have fallen through to the generic event.
|
||||
|
||||
**The backup had not failed:** the PVE task returned OK and the PBS snapshot `2026-07-28T19:19:45Z`
|
||||
is 6,264,034,053 B. The spurious failure **reached no channel** — the agent died with its in-memory
|
||||
record and R-84 ground truth superseded it; the controller's event trail for the window shows only
|
||||
the real 403 failure and its recovery. The design absorbed it; the mistake was still a mistake, and
|
||||
on a slower tier the same slip could have aborted a multi-hour WAN upload.
|
||||
It would have looked wired, passed its own unit tests, and shipped silently wrong on exactly the
|
||||
drive it exists for. Caught by tracing `a.Path` back to its source instead of assuming. Fixed in
|
||||
v0.184.1 (both keys, mirroring `planDriveGates`), red-proofed, and the **0.184.0 image is superseded
|
||||
and was never deployed**.
|
||||
|
||||
Also recorded: `pgrep -f vzdump` self-matches a polling script's own command line and is not a safe
|
||||
in-flight check — use the PVE task list.
|
||||
## 5. Tests and red-proofs
|
||||
|
||||
## State at close
|
||||
| Red-proof | Observed failure |
|
||||
|---|---|
|
||||
| hub allowlist entry removed | `backup_target_absent must be in allowedEventTypes, or POST /event 400s…` |
|
||||
| hub Hungarian copy shortened | `must state the consequence…, got: "Meghajtó hiányzik."` |
|
||||
| C — auto-elevation on register | `registering drives assigned the backup target "/mnt/hdd_1"` |
|
||||
| E-2c — eject guard removed | `eject of the backup-target drive SUCCEEDED (200)` |
|
||||
| E-2c over-correction | the gate blocked `/mnt/spare` (a non-target) |
|
||||
| keying reverted to MountPath | `not resolvable by its GUEST path — …the branch would never fire` |
|
||||
|
||||
Both boxes healthy. Primary and offsite tiers `due=false` with `age_state=known` on both; breakers
|
||||
clear; agents `active`; no thrash and no spurious staleness. Target drives at 1–2 % used with SMART
|
||||
`PASSED`. The old archives (18 GB demo-felhom, 5.2 GB demo-hp) are **left in place** as the rollback
|
||||
and as the only evidence of what the previous configuration produced.
|
||||
Every mutation was **verified to have landed** before the result was trusted. The first attempt
|
||||
silently no-op'd (gofmt had realigned the map) and the test "passed" — a false proof.
|
||||
|
||||
**Outstanding:** full drive-loss recovery (physical access), and the agent half of the absent-drive
|
||||
behaviour (needs a drive unmount that would break the guest bind on a remote box).
|
||||
Controller notify tests observe the **wire** (a real `Notifier` against an `httptest` hub), not a
|
||||
mock, because the failure class is "nothing arrives".
|
||||
|
||||
## 6. What is NOT done
|
||||
|
||||
- **Part 2** (installer Case A/B), **Part 3** (the offer + the agent-side move), **Part 4** (the
|
||||
degraded Hungarian banner). Phase 0 inverted their emphasis: the installer has **no drive-enrollment
|
||||
step**, so at install the common case is system-drive-only and **Part 3 is the primary path**.
|
||||
- **E-2a** — the guarded wrapper. The agent holds neither `Datastore.Allocate` at `/storage` nor
|
||||
`Permissions.Modify`, and its sudoers has no `pvesm`/`pveum`. Use the wrapper pattern; **do not
|
||||
widen the agent's PVE role.**
|
||||
- **`backup_target_absent` is not proven END-TO-END live.** Its components are: the controller emits
|
||||
the right type/severity (proven at the wire), the hub allowlists it (unit-pinned, image deployed),
|
||||
and the keying is pinned. Proving the whole chain needs a live drive to actually go absent, which
|
||||
means unmounting an enrolled drive carrying customer data and the guest bind on a remote box —
|
||||
the same thing declined in E-1, for the same reason.
|
||||
- Full drive-loss recovery (physical access) — carried forward from E-1.
|
||||
|
||||
## 7. Observations
|
||||
|
||||
- **`peti-felhom` untouched.** Its whole-guest backup still shares a device with its guest, so a
|
||||
drive failure there is offsite-only recovery. Accepted until the operator's reinstall; re-evaluate
|
||||
if that slips past ~2026-09-01.
|
||||
- The fleet was briefly split (demo-hp updated first). Left that way it would have protected one of
|
||||
two identically-exposed boxes, so demo-felhom was rolled to the same proven artifacts.
|
||||
- `StorageTarget.Role` carries a documented vocabulary including `vzdump-target` and is always `""`.
|
||||
Checked — it is a deliberate unpopulated field ("hub-owned, slice 10"), **not** a sixth dead seam.
|
||||
|
||||
@@ -41,8 +41,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — |
|
||||
| **E-2** | **IN PROGRESS 2026-07-29.** SHIPPED: hub v0.81.0 (`backup_target_absent`/`_restored` in allowedEventTypes + Hungarian customerMessages, 3 tests, red-proofed) and controller Part 1 (`StoragePath.BackupTarget` role + `SetBackupTarget`/`ClearBackupTarget`, 5 invariant tests, red-proof C). **REMAINING: Part 2 installer Case A/B, Part 3 the offer + the agent-side move, Part 4 the degraded banner, Part 5 the controller half of the signal, red-proofs E/F, live validation.** | **READY (M) — partially shipped** | — | **Phase 0 inverted the emphasis: the installer has NO drive-enrollment step (`/mnt/felhom-drives` is agent-installed at runtime), so at install time the common case is system-drive-only and Part 3 (drive added later) is the PRIMARY path, not Case A.** Scope + rationale: `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Flips matrix row 4 | CC |
|
||||
| **E-2a** | **The target move needs a root-fenced wrapper — the agent cannot do it.** Creating a PVE `dir` storage needs `Datastore.Allocate` at `/storage` and the ACL grant needs `Permissions.Modify`; the agent holds neither by design (blast-radius containment), and its sudoers is a literal-argument `Cmnd_Alias` model with **no `pvesm` and no `pveum`**. The established pattern is a guarded wrapper (`felhom-mkfs-guarded`, `felhom-pbs-apply` — the latter already has the exact `grant <id>` dual-grant verb E-2 needs) | READY (M) | blocks E-2 Part 3 | Add `felhom-backup-target-apply` (create-dir-storage with `is_mountpoint 1` + grant) + its literal sudoers rule + installer install step, then the agent endpoint. **Do NOT widen the agent's PVE role instead** | CC |
|
||||
| **E-2b** | **`NotifyStorageDisconnected`/`NotifyStorageReconnected` are defined and called NOWHERE** — a drive going absent emits no event on any channel. Detection works (`SetDisconnected`, apps stopped, `[WARN]` log, UI badge); the notification is dead code. Verified against the gitignored-`cmd/` trap with a positive control | **READY (S) — real silence** | — | Fifth instance of seam-built-but-never-wired. Hub side is fully plumbed already (allowedEventTypes + Hungarian message + DefaultEnabledEvents), so wiring needs **no hub change**. A backup-target-only drive has no apps to stop → today it is entirely silent | CC |
|
||||
| **E-2c** | **E-1 put the whole-guest backups on a drive the customer can self-serve eject.** `RoleForStorage` returns `RoleUserData` for a `local-dir` on a non-system device, so `POST /disks/eject` permits ejecting `/mnt/nvme-1tb` / `/mnt/hdd_1` — now the backup targets | READY (S) | — | Do NOT simply reclassify the drive `RoleBackup`: on both demo boxes it is ALSO the enrolled user-data drive, so that would block legitimate ejects (over-correction). Guard the eject specifically when the mount is the assigned backup target, and make it surface the consequence | CC |
|
||||
| **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:** `a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — |
|
||||
| **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — |
|
||||
| **PETI** | **`peti-felhom` deliberately NOT migrated.** Its whole-guest backup still shares a device with its guest, so a drive failure there is **offsite-only recovery** | **ACCEPTED RISK — parked** | operator's next visit (tester reinstalling from scratch) | **Accepted until the reinstall; re-evaluate if that slips past ~2026-09-01.** Do not migrate, do not touch | operator |
|
||||
| **R-109** | **The DR recipe records no backup target.** It lists every storage's name/type/content but never which one holds the local archives — and each demo box now carries TWO `content=backup` dir storages, `felhom-backup` (live) and `local` (frozen 2026-07-28 archives) | READY (XS) | — | Add the resolved `BackupTarget()` to the host-half. Third recipe-completeness defect beside R-105/R-106 | CC |
|
||||
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
|
||||
|
||||
Reference in New Issue
Block a user