Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c5daa9480d | |||
| d2053e7d47 | |||
| 7d4e564758 | |||
| 9b749ee7ad | |||
| 7ae3081a76 | |||
| 7ecd63cd61 |
@@ -44,8 +44,12 @@ it **first** in the morning note. A decision you cannot write in that shape is o
|
||||
Throwaway apps only; the standing apps and `bentopdf` stay.
|
||||
5. **Evidence off the machine at the end of each phase**, before any revert (R-320).
|
||||
6. **One release per repo per session**, with a CHANGELOG entry (controller: with its `MinAgent`
|
||||
line), REPORT overwritten, floor raised to deliver it. **No golden unless a drill or fresh install
|
||||
needs one** (the waiver, R-468). **No `--no-verify`.**
|
||||
line) and REPORT overwritten. **A controller release needs no golden** (operator ruling 2026-10-15
|
||||
11:26, `09` §3 decision 250; decision 7 stands): push the image only when it is ready for EVERY box (the
|
||||
household's update button offers the highest tag); raise the floor, MinAgent declared, only after it ran
|
||||
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
|
||||
on all three — not at night unless the night brief says so. **Agent releases still stop for the
|
||||
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
|
||||
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
|
||||
Prose is not a record.
|
||||
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
|
||||
|
||||
@@ -1,3 +1,78 @@
|
||||
## Unreleased (2026-10-15, afternoon) — the shared rule file; no code change
|
||||
|
||||
- `.claude/rules/unprompted-work.md` §3 item 6: **a controller release needs no golden** — the floor carries it after the
|
||||
three test boxes; agent releases still wait for the operator's vouch; the golden is weekly (operator ruling 2026-10-15
|
||||
11:26, `09` §3 decision 250; the same morning's „bake first", decision 247, reversed). Same text in all five copies.
|
||||
|
||||
## Unreleased (2026-10-15) — the shared rule file; no code change
|
||||
|
||||
- `.claude/rules/unprompted-work.md` §3 item 6: **bake first** — a controller release is released only when its golden is
|
||||
baked; bake → the operator vouches → the operator raises the floor; never ask for a floor for a version with no golden
|
||||
(operator ruling 2026-10-15, `09` §3 decision 247, superseding decision 7). Same text in all five copies.
|
||||
|
||||
## v0.156.1 — R-1008 by process liveness, not the clock; 0.156.0 superseded — do not vouch it (2026-10-14)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `83c7da27d457b63ba8a4cde5a8717b3e5f9db9f83a46e63f38902dc7471bffb9`, bundle
|
||||
`5a678def792c70ec1cbbd198c14542161fac91d11a69bd0ce23dd86469e96ec3` (tag `v0.156.1` = `7ae3081`). Step bundle `0.156.1-step1`
|
||||
sha256 `ac83ea41507616d4c1056cccfd35de1410ff01e0296f4e64ff925207a43a8490` (the 0.155.0 bundle with only `felhom-os-apply`
|
||||
replaced), published as package version `0.156.1-step1`. **Not vouched** (the operator's act). Delivery as 0.156.0: per box
|
||||
`agent_update 0.156.1` → `agent_config_update 0.156.1-step1` → `agent_config_update 0.156.1`.
|
||||
|
||||
- **0.156.0 is superseded — never vouch it.** A background security review of the pushed commit flagged „fail-open state
|
||||
drift": 0.156.0 called a listed vzdump task dead when its wall-clock start was before the kernel's `btime`. `btime` is
|
||||
derived from the wall clock and moves when the clock is corrected after the boot (an NTP step after a power cut), so a
|
||||
backup started between the boot and the correction could look „from before the boot" — and the stale-lock pass would
|
||||
clear a LIVE backup's lock.
|
||||
- **Now:** `taskAlive` is Proxmox's own rule (`PVE::ProcFSTools::check_process_running`, read on demo-hp): the UPID's pid and
|
||||
pstart against `/proc/<pid>/stat` — same start tick (field 22) and not a zombie. No clock is involved. Fails safe (running)
|
||||
when the UPID does not parse, the stat cannot be read, or this agent cannot see root's processes at all (PID 1 not visible:
|
||||
a `hidepid` /proc or `ProtectProc=` would hide every worker — demo-hp: `/proc` plain, `ProtectProc=default`).
|
||||
- Tests: worker gone, pid reused, zombie, live worker, a „clock step" (wall-clock start absurdly early, worker alive → still
|
||||
running), doubt, hidden /proc — each red-proofed; 19 guards in all (`power-cut-fixes-2026-10-14/redproof.txt`).
|
||||
- R-1006 (the cleanup) is unchanged; its „before this boot" rule still reads `btime`, but a drifted `btime` cannot remove a
|
||||
live partial: the Go side asks this liveness rule, and the root helper refuses while any vzdump runs or anything holds
|
||||
the file open.
|
||||
|
||||
## v0.156.0 — after a power cut: the dead half-written backup is removed (R-1006); a backup task from before the boot is not „in flight" (R-1008) (2026-10-14)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `8db2278838dcb733d5201fba84b75a829b9adebdf899c89dd7de0939ec10d51a`, bundle
|
||||
`a21605217966ebc953d55a35157a9eb0243ca542c50dbd94fc9fa67257f27369` (tag `v0.156.0` = `a2f3aee`). Step bundle `0.156.0-step1` sha256
|
||||
`8be2bb8f3d54bff8e7140cf99714664df31adecf13aa721597323fb2b9d43db5` (the 0.155.0 bundle `795f2ab4…` with only `felhom-os-apply`
|
||||
replaced; `scripts/build-step-bundle.py`), published as package version `0.156.0-step1`. **Not vouched** (the operator's act).
|
||||
|
||||
**Delivery: the agent binary AND the config bundle, and the bundle ADDS a path** (`/usr/local/sbin/felhom-vzdump-partial-clean`)
|
||||
— so per box: signed `agent_update 0.156.0` → `agent_config_update 0.156.0-step1` (the step bundle) → `agent_config_update
|
||||
0.156.0`. Evidence: felhom.eu `documentation/audits/power-cut-fixes-2026-10-14/` (the spike C1–C4, the red-proofs).
|
||||
|
||||
- **R-1008 — one in-flight answer.** `staleLockController.BackupRunning` (the ONE implementation all four callers reach:
|
||||
the start-up stale-lock pass, the guest-power watchdog, the household power-off guard, the controller supervisor) now
|
||||
answers through `vzdumpInFlight`: a vzdump task for the vmid that started at or after the host's boot. Proxmox keeps a
|
||||
dead task in `tasks?source=active` for ~30–60 s after a power cut (measured, C3) — the agent starts inside that window
|
||||
and called it „genuinely in-flight". A task from before the boot is dead; logged once (vmid, UPID, start, boot). The
|
||||
start time comes from the list or, if absent, the UPID; an unreadable boot time or start time FAILS SAFE (running), as
|
||||
before. The boot time is `poweroff.ProcBtime` (no third /proc/stat parser).
|
||||
- **R-1006 — the half-written backup is removed.** New `internal/vzpartial`: on a `dir` backup target it finds exactly
|
||||
`vzdump-(lxc|qemu)-<vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp)` of THAT guest (never a finished archive, a log,
|
||||
another guest, a symlink, a wrong kind), and — only when no vzdump of the guest is in flight (R-1008) and the entry is
|
||||
dead (modified before this boot, or untouched for 120 s) — asks the root helper to remove it. Runs at agent start (after
|
||||
the stale-lock pass, every owned guest × every backup tier) and before every backup's space preflight
|
||||
(`BackupRunner.SetBeforeBackup`). Any doubt leaves the entry and logs why.
|
||||
- **The root helper `configs/felhom-vzdump-partial-clean`** (config bundle; sudoers `FELHOM_VZPARTIAL`, one exact line: a
|
||||
storage id, a vmid, one half-written name) re-checks everything as root: a `dir` storage from `/etc/pve/storage.cfg`; the
|
||||
name of that vmid; not a symlink; directly in `<path>/dump`; **no vzdump on the host at all** (a job for several guests
|
||||
carries no vmid in its task name); **no process holds the entry open** (C2: after a killed worker, `tar` and `zstd` kept
|
||||
writing into the `.tar.dat` for 4 min 40 s, none named „vzdump"); dead (before this boot, or quiet 120 s).
|
||||
- **Not changed, by evidence (C4):** the space preflight's sizing (R-1006 part 2) — on „No space left" vzdump removes its
|
||||
own partial on both its paths (rsync and tar); only a power cut leaves one.
|
||||
- **Tests:** `stalelock_inflight_test.go` (the helper, the production controller, three callers end to end, a structural
|
||||
pin that every `BackupRunning` call goes through `s.staleLock` and there is one implementation), `vzpartial_test.go`
|
||||
(removes; keeps in flight / unknown / non-dir / other names / recent / wrong kind / symlink / a helper refusal — each keep
|
||||
with its positive control), `r685_space_test.go` (the cleanup runs BEFORE the preflight), the helper's own 21 Python
|
||||
tests (`configs/test_felhom_vzdump_partial_clean.py`, run by the Go suite), 8 sudoers injection cases. **17 guards
|
||||
red-proofed** by mutation (`power-cut-fixes-2026-10-14/redproof.txt`).
|
||||
- **Fixed without a row:** `internal/localapi/guestbind.go` and `controllersupervisor_test.go` were not gofmt-clean on
|
||||
`main` (an import order and a struct alignment) — formatted.
|
||||
|
||||
## v0.155.0 — the household can turn the machine off: POST /host/poweroff (R-998); the escrow reply race fixed (2026-10-13)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `bb339e944eff4aa6d10ae09158e0943de2e5c3777f52bf993c796d57e46800c3`, bundle
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# REPORT — v0.156.0: after a power cut the box clears up (R-1006, R-1008) — built, not yet released (2026-10-14)
|
||||
# REPORT — v0.156.1: after a power cut the box clears up (R-1006, R-1008) (2026-10-14)
|
||||
|
||||
Evidence: felhom.eu `documentation/audits/power-cut-fixes-2026-10-14/` — the spike (C1 the night's leftover, C2 a killed
|
||||
worker, C3 a real power cut, C4 a full disk), the 17 red-proofs.
|
||||
@@ -14,6 +14,14 @@ worker, C3 a real power cut, C4 a full disk), the 17 red-proofs.
|
||||
- On „No space left" vzdump removes its own partial (both paths): the preflight's sizing stays. But a full host root breaks
|
||||
LVM's own metadata writes (a half-removed snapshot; the janitor cannot clear it — new row).
|
||||
|
||||
## 0.156.0 is superseded — do not vouch it
|
||||
|
||||
A background security review of the pushed 0.156.0 flagged „fail-open state drift": its in-flight rule compared a task's
|
||||
wall-clock start with the kernel's `btime`, which moves when the clock is corrected after the boot (an NTP step after a
|
||||
power cut) — a live backup could then look „from before the boot" and its lock be cleared. 0.156.1 uses Proxmox's own
|
||||
liveness rule instead (the UPID's pid + pstart against `/proc/<pid>/stat` start tick and state; no clock), and fails safe
|
||||
when this agent cannot see root's processes (PID 1 not visible).
|
||||
|
||||
## What changed
|
||||
|
||||
See `CHANGELOG.md` v0.156.0: `vzdumpInFlight` behind the one `BackupRunning`; `internal/vzpartial` at start-up and before
|
||||
@@ -26,5 +34,5 @@ config bundle tests (55) and the helper's tests (21) pass. 17 mutations, 17 red.
|
||||
|
||||
## Next
|
||||
|
||||
Release (tag, publish, step bundle), the operator's vouch + floor, then the live proof on demo-hp (a cut mid-backup) and
|
||||
Released: tag `v0.156.0`, binary sha256 `8db22788…`, bundle `a2160521…`, step bundle `0.156.0-step1` `8be2bb8f…`. Then the operator's vouch + floor, then the live proof on demo-hp (a cut mid-backup) and
|
||||
delivery to Tester 1 and the N100.
|
||||
|
||||
@@ -41,6 +41,7 @@
|
||||
| `storage.ResolveStorageDevice` | internal/storage/durable_device.go | `ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)` | re-resolve enrolled STORAGE drives (remount) | `uuid:` scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node |
|
||||
| `deriveDurableID` | internal/storage/durableid.go | `deriveDurableID(typ, s, backingDevice, uuid) string` | storage-target durable id (DR re-attach key) | Deterministic per type; `uuid:` for usb/local-dir; PBS id includes `#<fingerprint>` |
|
||||
| `SystemDisks` / `isSystemBacked` / `RoleForStorage` / `RoleForRawDevice` | internal/storage/role.go | `RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole` | protection-tier classification | Fails safe to `system` (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied |
|
||||
| `storage.IdentityResolver` / `Relation` / `MountFor` | internal/storage/identity.go | `HostPath(path) PlaceIdentity`, `Device(dev) PlaceIdentity`, `Relation(a, b) (different, proven, reason)` | physical-disk identity of a storage place (felhom.eu `16-storage-physical-identity.md` §3, v0.157.0) | Reads /sys + /run/udev/data + /proc/mounts only (no block I/O). Judge a drive ONLY on its own mount (`localapi.ownMountIdentity`) — never the containing fs, or an absent drive reads as the root disk. A shared disk key is "same, proven" even when unknown. Pinned by identity_test.go (fixture trees) + samedisk_test.go |
|
||||
| `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process |
|
||||
| `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` |
|
||||
|
||||
@@ -158,6 +159,7 @@
|
||||
| `proxmox.Runner` | internal/proxmox/privileged.go | `*ExecRunner` (direct/sudo) | `mockRunner` internal/proxmox/mock_test.go; runner fakes in storage tests |
|
||||
| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
|
||||
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
|
||||
| `storage.IdentityHost` | internal/storage/identity.go | `*SysfsIdentityHost` | fixture /sys + udev trees in identity_test.go (`newFx`); `fakeIdentHost` in internal/localapi/samedisk_test.go. A SEPARATE interface from HostReader on purpose — adding methods to HostReader would break every existing fake. |
|
||||
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
|
||||
| `lanresolver.hostRoot` + `dnsmasqUnitPaths` (data seam, R-317) | internal/lanresolver/lanresolver.go | prod `hostRoot = "/"`; probe = the `dnsmasq` package's systemd UNIT, never `/usr/sbin/dnsmasq` (owned by `dnsmasq-base`) | internal/lanresolver/ensure_dnsmasq_test.go — fixture root tree + recording `proxmox.Runner`; the REAL `os.Stat` probe and `EnsureDnsmasq` run. `TestEnsureDnsmasq_ProductionProbeIsTheUnit` pins the production wiring |
|
||||
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
||||
|
||||
@@ -777,6 +777,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
hostReader := storage.NewProcHostReader()
|
||||
hostOps := newHostOps(cfg, logger)
|
||||
observer := storage.NewObserver(px, hostReader, hostOps, logger)
|
||||
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
|
||||
// Backup + restore-test state store (slice 6): holds the latest backup-per-target +
|
||||
// latest restore-test result; the collector reads it via the BackupReporter /
|
||||
// RestoreTestReporter seams; the cadence scheduler writes it.
|
||||
@@ -2000,6 +2001,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
||||
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
|
||||
Identity: storage.NewIdentityResolver(), // v0.157.0: `16` §3 identity on /disks + POST /disks/same-disk
|
||||
Tokens: tokens,
|
||||
BackupCadence: cfg.Backup.BackupCadence(),
|
||||
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
|
||||
@@ -2188,6 +2190,7 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger)
|
||||
return 1
|
||||
}
|
||||
observer := storage.NewObserver(px, storage.NewProcHostReader(), newHostOps(cfg, logger), logger)
|
||||
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
|
||||
// Wire the LIVE PBS reporter here too (v0.39.0): selftest=hub is a separate one-shot process — no
|
||||
// verify loop runs — so a nil reporter previously yielded pbs_snapshots:[] and an absent dr_recipe
|
||||
// pbs coord. The live reporter lists snapshots directly (fresh store, last-known-good fallback) so
|
||||
@@ -2244,6 +2247,7 @@ func runSelftestStorage(ctx context.Context, cfg config.Config, logger *slog.Log
|
||||
hostReader := storage.NewProcHostReader()
|
||||
hostOps := newHostOps(cfg, logger)
|
||||
observer := storage.NewObserver(px, hostReader, hostOps, logger)
|
||||
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
|
||||
|
||||
octx, cancel := context.WithTimeout(ctx, 60*time.Second)
|
||||
fmt.Printf("=== felhom-agent %s selftest=storage ===\n", version)
|
||||
|
||||
@@ -39,11 +39,14 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
|
||||
State: StorageStateAttached, Reachable: true, ClassHint: "fast",
|
||||
ThinPool: &ThinPoolFill{DataUsedFraction: 0.42},
|
||||
Smart: SmartSummary{Health: SmartUnknown},
|
||||
// v0.157.0 (`16` §3): the identity keys ride on both rows so the key-set guard sees them.
|
||||
Kind: "system", PhysicalDisks: []string{"serial:ata:SanDisk_X600:182195804614"}, Identity: "proven",
|
||||
},
|
||||
{
|
||||
Name: "usb-backup", Type: StorageTypeUSB, DurableID: "uuid:x",
|
||||
State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000,
|
||||
Smart: SmartSummary{Health: SmartUnknown},
|
||||
Kind: "disk", PhysicalDisks: []string{"serial:usb:External:000000000000"}, Identity: "unknown", IdentityReason: "usb_bridge",
|
||||
},
|
||||
// A pbs target so the recipe's pbs coord has a storage.cfg row to resolve its namespace
|
||||
// from (R-106). Without one the fixture produces namespace_state=unknown while the golden
|
||||
|
||||
@@ -431,6 +431,15 @@ type StorageTarget struct {
|
||||
// metadata fill is null until Phase B's privileged `lvs` read.
|
||||
ThinPool *ThinPoolFill `json:"thin_pool,omitempty"`
|
||||
|
||||
// Physical-disk identity (felhom.eu architecture/16-storage-physical-identity.md §3, agent v0.157.0):
|
||||
// the place's kind, the keys of the whole physical disks under it, and whether the agent can prove
|
||||
// them (proven | unknown + identity_reason vm|network|usb_bridge|weak_wwn|no_serial). Agent-computed
|
||||
// from /sys + udev only; empty when not computed (an absent device, an older agent). Additive.
|
||||
Kind string `json:"kind,omitempty"`
|
||||
PhysicalDisks []string `json:"physical_disks,omitempty"`
|
||||
Identity string `json:"identity,omitempty"`
|
||||
IdentityReason string `json:"identity_reason,omitempty"`
|
||||
|
||||
// Smart is the disk-health summary, populated in Phase B via smartctl (sudoers). In
|
||||
// Phase A it is {health:"UNKNOWN", <counters null>} — the keys are committed now so the
|
||||
// contract is stable across both phases.
|
||||
|
||||
@@ -46,6 +46,9 @@
|
||||
"backing_device": "",
|
||||
"class_hint": "fast",
|
||||
"role": "",
|
||||
"kind": "system",
|
||||
"physical_disks": ["serial:ata:SanDisk_X600:182195804614"],
|
||||
"identity": "proven",
|
||||
"thin_pool": { "data_used_fraction": 0.42, "metadata_used_fraction": null },
|
||||
"smart": {
|
||||
"health": "UNKNOWN",
|
||||
@@ -74,6 +77,10 @@
|
||||
"backing_device": "/dev/sdb1",
|
||||
"class_hint": "slow",
|
||||
"role": "",
|
||||
"kind": "disk",
|
||||
"physical_disks": ["serial:usb:External:000000000000"],
|
||||
"identity": "unknown",
|
||||
"identity_reason": "usb_bridge",
|
||||
"smart": {
|
||||
"health": "UNKNOWN",
|
||||
"temperature_c": null,
|
||||
|
||||
@@ -188,6 +188,13 @@ type DiskInfo struct {
|
||||
// device that exposes no SMART (USB bridge, unread) is ABSENT, not a misleading zero-value UNKNOWN;
|
||||
// the controller feature-detects presence and renders "Nincs adat" when nil (never alarms).
|
||||
Smart *hub.SmartSummary `json:"smart,omitempty"`
|
||||
// Physical-disk identity (v0.157.0, felhom.eu architecture/16-storage-physical-identity.md §3) — the
|
||||
// same agent-computed verdict the hub report carries. Empty when not computed (an absent drive, no
|
||||
// resolver). Additive; an older controller ignores it.
|
||||
Kind string `json:"kind,omitempty"`
|
||||
PhysicalDisks []string `json:"physical_disks,omitempty"`
|
||||
Identity string `json:"identity,omitempty"`
|
||||
IdentityReason string `json:"identity_reason,omitempty"`
|
||||
}
|
||||
|
||||
// handleDisks lists the host's drives + data-bearing flags (read-only/benign).
|
||||
@@ -227,6 +234,8 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
// really is the target. Reported here so the controller can name the drive in an
|
||||
// absent-target alarm and hide the destructive controls the agent would refuse anyway.
|
||||
BackupTarget: t.Name == primaryTargetID,
|
||||
// `16` §3: copied from the observer, which computed it from /sys + udev.
|
||||
Kind: t.Kind, PhysicalDisks: t.PhysicalDisks, Identity: t.Identity, IdentityReason: t.IdentityReason,
|
||||
}
|
||||
// Intermediary model: the stable in-guest path + whether felhom-data is bound under the parent.
|
||||
// Only user-data /mnt/<name> drives have a guest path (system/backup mounts never cross in).
|
||||
@@ -398,6 +407,10 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
di.Smart = &sm
|
||||
}
|
||||
}
|
||||
// `16` §3: a registry drive's identity, judged on its OWN mount only (never root's).
|
||||
if p, kind, ok := s.ownMountIdentity(d.MountPath); ok {
|
||||
di.Kind, di.PhysicalDisks, di.Identity, di.IdentityReason = kind, p.PhysicalDisks, p.Identity, p.Reason
|
||||
}
|
||||
// R-118: statfs ONLY while the drive's device is present. With the device gone the raw
|
||||
// mountpoint reverts to a bare directory on the ROOT filesystem, and statfs would report
|
||||
// pve-root's size as this drive's (measured: a 4 GB drive advertising 46 GiB). Same trap
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"path"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// POST /disks/same-disk (v0.157.0) — felhom.eu architecture/16-storage-physical-identity.md §4.
|
||||
//
|
||||
// The controller asks whether two of ITS paths sit on the same physical disk; the agent answers from its
|
||||
// own /sys + udev read, never from the caller's claim (`16` §10). A guest path is mapped only to a place
|
||||
// the agent itself made:
|
||||
// - /mnt/felhom-drives/<name>[/…] → the host mount at exactly /mnt/felhom-drives/<name> (the drive's
|
||||
// felhom-data bind, or a network share's mount). A path whose <name> is not mounted there (an absent
|
||||
// drive: the bare directory on the root disk) is UNMAPPED — never "the system disk".
|
||||
// - /mnt/sys_drive, /var/lib/felhom, /var/lib/docker [/…] → the guest's data volume (mp0, mp=/var/lib/felhom;
|
||||
// or a legacy mp at that exact path) → the PVE storage holding it → that storage's identity.
|
||||
// - anything else → unmapped.
|
||||
// Answer: {different, proven, reason, a, b}. Act on `proven` first: different+proven = a counted second
|
||||
// copy; same_disk+proven = not a second copy; proven=false = unknown (reason vm|network|usb_bridge|
|
||||
// weak_wwn|no_serial|unmapped). Pinned by samedisk_test.go.
|
||||
|
||||
type sameDiskRequest struct {
|
||||
VMID int `json:"vmid,omitempty"`
|
||||
A string `json:"a"`
|
||||
B string `json:"b"`
|
||||
}
|
||||
|
||||
// SamePlace is one side of the answer.
|
||||
type SamePlace struct {
|
||||
Path string `json:"path"`
|
||||
Mapped bool `json:"mapped"`
|
||||
Kind string `json:"kind,omitempty"`
|
||||
PhysicalDisks []string `json:"physical_disks,omitempty"`
|
||||
Identity string `json:"identity,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// SameDiskResponse is the wire answer.
|
||||
type SameDiskResponse struct {
|
||||
Different bool `json:"different"`
|
||||
Proven bool `json:"proven"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
A SamePlace `json:"a"`
|
||||
B SamePlace `json:"b"`
|
||||
}
|
||||
|
||||
// sysDataPaths are the guest paths that live on the guest's one data volume (golden v3: mp0 at
|
||||
// /var/lib/felhom, its docker/ and sys_drive/ folders bound in the guest — 07 §7.5.1).
|
||||
var sysDataPaths = []string{"/var/lib/felhom", "/mnt/sys_drive", "/var/lib/docker"}
|
||||
|
||||
func (s *Server) handleSameDisk(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
if s.ident == nil {
|
||||
writeErr(w, http.StatusServiceUnavailable, "disk identity not configured on this host")
|
||||
return
|
||||
}
|
||||
var req sameDiskRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||
return
|
||||
}
|
||||
if !path.IsAbs(req.A) || !path.IsAbs(req.B) {
|
||||
writeErr(w, http.StatusBadRequest, "a and b must be absolute guest paths")
|
||||
return
|
||||
}
|
||||
a := s.mapGuestPlace(r.Context(), vmid, req.A)
|
||||
b := s.mapGuestPlace(r.Context(), vmid, req.B)
|
||||
resp := SameDiskResponse{A: a.view, B: b.view}
|
||||
resp.Different, resp.Proven, resp.Reason = storage.Relation(a.id, b.id)
|
||||
s.logger.Info("local-api: same-disk", "vmid", vmid, "a", req.A, "b", req.B,
|
||||
"different", resp.Different, "proven", resp.Proven, "reason", resp.Reason)
|
||||
writeOK(w, resp)
|
||||
}
|
||||
|
||||
type mappedPlace struct {
|
||||
view SamePlace
|
||||
id storage.PlaceIdentity
|
||||
}
|
||||
|
||||
func (s *Server) mapGuestPlace(ctx context.Context, vmid int, guestPath string) mappedPlace {
|
||||
p := path.Clean(guestPath)
|
||||
out := mappedPlace{view: SamePlace{Path: p}}
|
||||
set := func(kind string, id storage.PlaceIdentity) mappedPlace {
|
||||
if !id.Computed() {
|
||||
return out
|
||||
}
|
||||
out.id = id
|
||||
out.view.Mapped, out.view.Kind = true, kind
|
||||
out.view.PhysicalDisks, out.view.Identity, out.view.Reason = id.PhysicalDisks, id.Identity, id.Reason
|
||||
return out
|
||||
}
|
||||
if strings.HasPrefix(p, StableParentDir+"/") {
|
||||
name := strings.SplitN(strings.TrimPrefix(p, StableParentDir+"/"), "/", 2)[0]
|
||||
if id, kind, ok := s.ownMountIdentity(StableParentDir + "/" + name); ok {
|
||||
return set(kind, id)
|
||||
}
|
||||
return out
|
||||
}
|
||||
for _, root := range sysDataPaths {
|
||||
if p != root && !strings.HasPrefix(p, root+"/") {
|
||||
continue
|
||||
}
|
||||
id := s.guestDataVolumeIdentity(ctx, vmid, root)
|
||||
return set(storage.PlaceKindSystem, id)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ownMountIdentity judges a HOST path only when it is ITSELF a mountpoint (an enrolled drive's raw mount
|
||||
// or its felhom-data bind, a network share). A path that is not its own mount — an absent drive, whose
|
||||
// directory now sits on the root disk — is not computed, never root's identity (the R-118 trap).
|
||||
func (s *Server) ownMountIdentity(hostPath string) (storage.PlaceIdentity, string, bool) {
|
||||
if s.ident == nil || hostPath == "" {
|
||||
return storage.PlaceIdentity{}, "", false
|
||||
}
|
||||
mounts, err := s.hostReader().Mounts()
|
||||
if err != nil {
|
||||
return storage.PlaceIdentity{}, "", false
|
||||
}
|
||||
m, ok := storage.MountFor(mounts, hostPath)
|
||||
if !ok || path.Clean(m.MountPoint) != path.Clean(hostPath) {
|
||||
return storage.PlaceIdentity{}, "", false
|
||||
}
|
||||
id := s.ident.HostPath(hostPath)
|
||||
if !id.Computed() {
|
||||
return id, "", false
|
||||
}
|
||||
kind := storage.PlaceKindDisk
|
||||
switch {
|
||||
case id.Reason == storage.ReasonNetwork:
|
||||
kind = storage.PlaceKindNetwork
|
||||
case strings.HasPrefix(m.Device, "/dev/") && !strings.HasPrefix(m.Device, "/dev/mapper/") && isPartitionPath(m.Device):
|
||||
kind = storage.PlaceKindPartition
|
||||
}
|
||||
return id, kind, true
|
||||
}
|
||||
|
||||
var rePartitionName = regexp.MustCompile(`^(nvme[0-9]+n[0-9]+p[0-9]+|(sd|vd|hd)[a-z]+[0-9]+)$`)
|
||||
|
||||
func isPartitionPath(dev string) bool { return rePartitionName.MatchString(path.Base(dev)) }
|
||||
|
||||
// guestDataVolumeIdentity finds the guest mountpoint that holds root (mp=/var/lib/felhom on golden v3, or a
|
||||
// legacy mp at that exact path) and returns the identity of the PVE storage that holds its volume.
|
||||
func (s *Server) guestDataVolumeIdentity(ctx context.Context, vmid int, root string) storage.PlaceIdentity {
|
||||
if s.guests == nil || s.storage == nil {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
cfg, err := s.guests.GuestConfig(ctx, vmid)
|
||||
if err != nil {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
storeID, fallback := "", ""
|
||||
for _, spec := range cfg.MountPoints() {
|
||||
st, mp, _ := parseMount(spec)
|
||||
if strings.HasPrefix(st, "/") {
|
||||
continue // a host bind (mp8, mp9), not a volume
|
||||
}
|
||||
switch mp {
|
||||
case root:
|
||||
storeID = st // a legacy mp at exactly this path wins
|
||||
case "/var/lib/felhom":
|
||||
fallback = st // golden v3: the one data volume
|
||||
}
|
||||
}
|
||||
if storeID == "" {
|
||||
storeID = fallback
|
||||
}
|
||||
if storeID == "" {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
targets, err := s.storage.Observe(ctx)
|
||||
if err != nil {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
for _, t := range targets {
|
||||
if t.Name == storeID {
|
||||
return identityOfTarget(t)
|
||||
}
|
||||
}
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
|
||||
func identityOfTarget(t hub.StorageTarget) storage.PlaceIdentity {
|
||||
if t.Identity == "" {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
disks := t.PhysicalDisks
|
||||
if disks == nil {
|
||||
disks = []string{}
|
||||
}
|
||||
return storage.PlaceIdentity{PhysicalDisks: disks, Identity: t.Identity, Reason: t.IdentityReason}
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// POST /disks/same-disk (`16` §4) — the controller's Tier-2 question, answered from the agent's own read.
|
||||
|
||||
type fakeIdentHost struct {
|
||||
mounts []storage.Mount
|
||||
parent map[string]string
|
||||
props map[string]map[string]string
|
||||
virt string
|
||||
}
|
||||
|
||||
func (f *fakeIdentHost) Mounts() ([]storage.Mount, error) { return f.mounts, nil }
|
||||
func (f *fakeIdentHost) KernelName(dev string) (string, bool) {
|
||||
if len(dev) > 5 && dev[:5] == "/dev/" {
|
||||
return dev[5:], true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
func (f *fakeIdentHost) Parent(n string) (string, bool) { p, ok := f.parent[n]; return p, ok }
|
||||
func (f *fakeIdentHost) Slaves(string) []string { return nil }
|
||||
func (f *fakeIdentHost) LoopBackingFile(string) (string, bool) { return "", false }
|
||||
func (f *fakeIdentHost) UdevProps(n string) map[string]string { return f.props[n] }
|
||||
func (f *fakeIdentHost) Virt() string { return f.virt }
|
||||
|
||||
// sameDiskRig: guest 8200 (token A) with mp0 on local-lvm (the system SSD, proven) + the given mounts.
|
||||
func sameDiskRig(t *testing.T, host *fakeIdentHost, localLVM hub.StorageTarget) http.Handler {
|
||||
t.Helper()
|
||||
fg := &fakeGuestsCfg{mounts: map[int]map[string]string{8200: {
|
||||
"mp0": "local-lvm:vm-8200-disk-1,mp=/var/lib/felhom,backup=1,size=70G",
|
||||
"mp8": "/mnt/felhom-drives,mp=/mnt/felhom-drives",
|
||||
}}}
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: fg, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{localLVM}}, Tokens: staticTokens{"A": 8200},
|
||||
HostReader: fakeHostReader{mounts: host.mounts},
|
||||
Identity: &storage.IdentityResolver{Host: host},
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
return srv.Handler()
|
||||
}
|
||||
|
||||
var provenSSD = hub.StorageTarget{Name: "local-lvm", Type: hub.StorageTypeLVMThin, Kind: "system",
|
||||
PhysicalDisks: []string{"serial:ata:SanDisk:1"}, Identity: "proven"}
|
||||
|
||||
func baseHost() *fakeIdentHost {
|
||||
return &fakeIdentHost{
|
||||
virt: "none",
|
||||
parent: map[string]string{"sdc1": "sdc", "sdc2": "sdc"},
|
||||
props: map[string]map[string]string{
|
||||
"sdb": {"ID_BUS": "ata", "ID_MODEL": "TOSHIBA", "ID_SERIAL_SHORT": "65NOP3HDT"},
|
||||
"sdc": {"ID_BUS": "usb", "ID_MODEL": "Portable", "ID_SERIAL_SHORT": "NAAB7C21"},
|
||||
},
|
||||
mounts: []storage.Mount{
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"},
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/mnt/felhom-drives", FSType: "ext4"},
|
||||
{Device: "/dev/sdb", MountPoint: "/mnt/felhom-drives/hdd_1", FSType: "ext4"},
|
||||
{Device: "/dev/sdc1", MountPoint: "/mnt/felhom-drives/a", FSType: "ext4"},
|
||||
{Device: "/dev/sdc2", MountPoint: "/mnt/felhom-drives/b", FSType: "ext4"},
|
||||
{Device: "//nas/films", MountPoint: "/mnt/felhom-drives/nas", FSType: "cifs"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func sameDisk(t *testing.T, h http.Handler, a, b string) (SameDiskResponse, int) {
|
||||
t.Helper()
|
||||
body, _ := json.Marshal(map[string]string{"a": a, "b": b})
|
||||
w := do(t, h, "POST", "/disks/same-disk", "A", string(body))
|
||||
var env struct {
|
||||
Data SameDiskResponse `json:"data"`
|
||||
}
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &env)
|
||||
return env.Data, w.Code
|
||||
}
|
||||
|
||||
func TestSameDisk_DriveVsSystemData_DifferentProven(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
got, code := sameDisk(t, h, "/mnt/felhom-drives/hdd_1/felhom-data/paperless", "/mnt/sys_drive")
|
||||
if code != http.StatusOK || !got.Different || !got.Proven || got.Reason != "" {
|
||||
t.Fatalf("drive vs sys_drive: want different+proven, got %d %+v", code, got)
|
||||
}
|
||||
if got.B.Kind != "system" || got.A.Kind != "disk" {
|
||||
t.Fatalf("kinds: got a=%q b=%q", got.A.Kind, got.B.Kind)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_TwoPartitionsOfOneDisk_SameProven(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
got, _ := sameDisk(t, h, "/mnt/felhom-drives/a", "/mnt/felhom-drives/b/x")
|
||||
if got.Different || !got.Proven || got.Reason != "same_disk" || got.A.Kind != "partition" {
|
||||
t.Fatalf("two partitions: want same+proven, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_InsideAVM_NotProven(t *testing.T) {
|
||||
host := baseHost()
|
||||
host.virt = "kvm"
|
||||
h := sameDiskRig(t, host, hub.StorageTarget{Name: "local-lvm", Identity: "unknown", IdentityReason: "vm",
|
||||
PhysicalDisks: []string{"serial:scsi:QEMU_HARDDISK:drive-scsi0"}})
|
||||
got, _ := sameDisk(t, h, "/mnt/felhom-drives/hdd_1", "/mnt/sys_drive")
|
||||
if got.Proven || got.Reason != "vm" {
|
||||
t.Fatalf("inside a VM: want not proven (vm), got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_NetworkAndUnmapped(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
if got, _ := sameDisk(t, h, "/mnt/felhom-drives/nas", "/mnt/sys_drive"); got.Proven || got.Reason != "network" {
|
||||
t.Fatalf("a NAS: want not proven (network), got %+v", got)
|
||||
}
|
||||
// An ABSENT drive: its directory sits on the root disk — must be unmapped, never "the system disk".
|
||||
if got, _ := sameDisk(t, h, "/mnt/felhom-drives/gone", "/mnt/sys_drive"); got.Proven || got.Reason != "unmapped" || got.A.Mapped {
|
||||
t.Fatalf("an absent drive: want unmapped, got %+v", got)
|
||||
}
|
||||
if got, _ := sameDisk(t, h, "/opt/elsewhere", "/mnt/sys_drive"); got.Proven || got.Reason != "unmapped" {
|
||||
t.Fatalf("an unknown guest path: want unmapped, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_ScopedAndValidated(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
if w := do(t, h, "POST", "/disks/same-disk", "A", `{"vmid":9300,"a":"/mnt/sys_drive","b":"/mnt/sys_drive"}`); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("another guest's vmid: got %d want 403", w.Code)
|
||||
}
|
||||
if w := do(t, h, "POST", "/disks/same-disk", "A", `{"a":"relative","b":"/mnt/sys_drive"}`); w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("a relative path: got %d want 400", w.Code)
|
||||
}
|
||||
if w := do(t, h, "POST", "/disks/same-disk", "", `{"a":"/x","b":"/y"}`); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: got %d want 401", w.Code)
|
||||
}
|
||||
}
|
||||
@@ -220,6 +220,10 @@ type Options struct {
|
||||
// ROLE (it backs SystemDisks for the eject role-gate + the /disks role hints). OPTIONAL — when nil
|
||||
// it defaults to the production *storage.ProcHostReader. Injectable so the role-gate is testable.
|
||||
HostReader storage.HostReader
|
||||
// Identity computes physical-disk identity (felhom.eu architecture/16-storage-physical-identity.md §3)
|
||||
// for the /disks rows and POST /disks/same-disk. OPTIONAL — when nil, /disks carries no identity and
|
||||
// same-disk answers 503 "not configured" (the controller then keeps its old st_dev rule).
|
||||
Identity *storage.IdentityResolver
|
||||
// HostMetrics serves GET /host/metrics (slice 9) — host-wide health (cpu%/mem/load/uptime/
|
||||
// cpu-temp) + per-storage capacity, host-wide and token-authed (one-customer-per-host). When
|
||||
// nil the endpoint reports "not configured" (host still reports/reconciles).
|
||||
@@ -325,7 +329,8 @@ type Server struct {
|
||||
// guestPowerSweeps counts completed guest-power sweeps, for the liveness observable. Touched only
|
||||
// from GuestPowerTick, which the ticker calls serially.
|
||||
guestPowerSweeps int
|
||||
host storage.HostReader // role classification source (optional; defaults to ProcHostReader)
|
||||
host storage.HostReader // role classification source (optional; defaults to ProcHostReader)
|
||||
ident *storage.IdentityResolver // `16` §3 physical-disk identity (optional)
|
||||
|
||||
hostMetrics HostMetricsProvider // slice 9 (optional)
|
||||
hostID string // slice 10B: for the data-bearing-format pending-op hint
|
||||
@@ -480,6 +485,7 @@ func NewServer(o Options) (*Server, error) {
|
||||
formatJobs: o.FormatJobs,
|
||||
staleLock: o.StaleLock,
|
||||
host: o.HostReader,
|
||||
ident: o.Identity,
|
||||
hostMetrics: o.HostMetrics,
|
||||
hostID: o.HostID,
|
||||
agentVersion: o.AgentVersion,
|
||||
@@ -546,6 +552,7 @@ func (s *Server) Handler() http.Handler {
|
||||
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
||||
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
|
||||
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
|
||||
mux.HandleFunc("POST /disks/same-disk", s.withGuest(s.handleSameDisk)) // `16` §4 (v0.157.0)
|
||||
mux.HandleFunc("POST /disks/assign", s.withGuest(s.handleDiskAssign))
|
||||
mux.HandleFunc("POST /disks/eject", s.withGuest(s.handleDiskEject))
|
||||
mux.HandleFunc("POST /disks/decommission", s.withGuest(s.handleDiskDecommission))
|
||||
|
||||
@@ -4,10 +4,12 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
@@ -144,9 +146,8 @@ type staleLockController struct {
|
||||
runner proxmox.Runner
|
||||
pool string // ownership registry: only members of this PVE pool are ever scanned (A1)
|
||||
logger *slog.Logger // the one success-path scan-summary line; nil = silent
|
||||
// bootTime is the host's boot time (unix seconds) and whether it could be read (R-1008). nil = the
|
||||
// kernel's btime (poweroff.ProcBtime — the one reader of /proc/stat btime this agent shares).
|
||||
bootTime func() (int64, bool)
|
||||
// procDir is where taskAlive reads /proc/<pid>/stat (R-1008); "" = /proc. Tests point it at a fixture.
|
||||
procDir string
|
||||
deadMu sync.Mutex
|
||||
deadSeen map[string]bool // UPIDs of dead tasks already logged (once per task)
|
||||
}
|
||||
@@ -220,29 +221,28 @@ func (c *staleLockController) Lock(ctx context.Context, vmid int) (string, bool,
|
||||
// recovery, the guest-power watchdog, the household power-off guard, the controller supervisor): every one of
|
||||
// them reaches this method through the StaleLockController interface. R-1008: Proxmox keeps a task in
|
||||
// `tasks?source=active` after a power cut — nobody is left to end it (measured on demo-hp 2026-10-14: still
|
||||
// listed RUNNING after the boot) — so a task that STARTED BEFORE THIS BOOT is dead, not in flight.
|
||||
// listed RUNNING ~30–60 s after the boot) — so a listed task whose worker process is gone is dead, not in flight.
|
||||
func (c *staleLockController) BackupRunning(ctx context.Context, vmid int) (bool, error) {
|
||||
tasks, err := c.px.ListRunningTasks(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
bt := c.bootTime
|
||||
if bt == nil {
|
||||
bt = procBootTime
|
||||
proc := c.procDir
|
||||
if proc == "" {
|
||||
proc = "/proc"
|
||||
}
|
||||
boot, ok := bt()
|
||||
running, dead := vzdumpInFlight(tasks, vmid, boot, ok)
|
||||
running, dead := vzdumpInFlight(tasks, vmid, proc)
|
||||
for _, t := range dead {
|
||||
c.logDeadOnce(t, vmid, boot)
|
||||
c.logDeadOnce(t, vmid)
|
||||
}
|
||||
return running, nil
|
||||
}
|
||||
|
||||
// vzdumpInFlight decides, from the node's active-task list, whether a vzdump for vmid is in flight. A matching
|
||||
// task is in flight when it started at or after the host's boot; one that started before the boot is dead
|
||||
// (returned in dead, for the log). FAIL SAFE on doubt — the old answer: an unreadable boot time, or a task whose
|
||||
// start time is unknown, counts as in flight. Pure; pinned by stalelock_inflight_test.go.
|
||||
func vzdumpInFlight(tasks []proxmox.TaskStatus, vmid int, boot int64, bootOK bool) (bool, []proxmox.TaskStatus) {
|
||||
// vzdumpInFlight decides, from the node's active-task list, whether a vzdump for vmid is in flight: a matching
|
||||
// task is in flight while its worker process lives (taskAlive). A task whose worker is gone is dead (returned in dead,
|
||||
// for the log). FAIL SAFE on doubt — the old answer: a UPID that does not parse, or a /proc entry that cannot be read,
|
||||
// counts as in flight. Pure over procDir; pinned by stalelock_inflight_test.go.
|
||||
func vzdumpInFlight(tasks []proxmox.TaskStatus, vmid int, procDir string) (bool, []proxmox.TaskStatus) {
|
||||
id := strconv.Itoa(vmid)
|
||||
var dead []proxmox.TaskStatus
|
||||
for _, t := range tasks {
|
||||
@@ -250,11 +250,8 @@ func vzdumpInFlight(tasks []proxmox.TaskStatus, vmid int, boot int64, bootOK boo
|
||||
if t.Type != "vzdump" || t.ID != id {
|
||||
continue
|
||||
}
|
||||
if !bootOK || boot <= 0 {
|
||||
return true, nil
|
||||
}
|
||||
start := taskStart(t)
|
||||
if start <= 0 || start >= boot {
|
||||
alive, known := taskAlive(procDir, t.UPID)
|
||||
if !known || alive {
|
||||
return true, nil
|
||||
}
|
||||
dead = append(dead, t)
|
||||
@@ -262,24 +259,47 @@ func vzdumpInFlight(tasks []proxmox.TaskStatus, vmid int, boot int64, bootOK boo
|
||||
return false, dead
|
||||
}
|
||||
|
||||
// taskStart is the task's start time: the list's starttime field, else the one encoded in its UPID; 0 = unknown.
|
||||
func taskStart(t proxmox.TaskStatus) int64 {
|
||||
if t.StartTime > 0 {
|
||||
return t.StartTime
|
||||
// taskAlive is Proxmox's own liveness rule for a task (PVE::ProcFSTools::check_process_running): the UPID names the
|
||||
// worker's pid and its start time in clock ticks since boot (pstart); the task lives while /proc/<pid>/stat exists with
|
||||
// that same start time (field 22). No wall clock is involved — a clock step after the boot (an NTP correction after a
|
||||
// power cut) cannot make a live task look dead (the 2026-10-14 security review of the first, btime-based version).
|
||||
// known=false: the UPID does not parse, or /proc/<pid>/stat exists but cannot be read or parsed.
|
||||
func taskAlive(procDir, upid string) (alive, known bool) {
|
||||
u, err := proxmox.ParseUPID(upid)
|
||||
if err != nil || u.PID == 0 {
|
||||
return false, false
|
||||
}
|
||||
if u, err := proxmox.ParseUPID(t.UPID); err == nil {
|
||||
return int64(u.StartTime)
|
||||
raw, err := os.ReadFile(filepath.Join(procDir, strconv.FormatUint(u.PID, 10), "stat"))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
// „no such process" is only an answer if this agent can see other users' processes at all (a /proc mounted
|
||||
// hidepid=, or a unit with ProtectProc=, hides root's workers — every task would then look dead). PID 1 is
|
||||
// root's; not seeing it means not knowing.
|
||||
if _, err1 := os.Stat(filepath.Join(procDir, "1", "stat")); err1 != nil {
|
||||
return false, false
|
||||
}
|
||||
return false, true // no such process in this boot: the worker is gone
|
||||
}
|
||||
return false, false
|
||||
}
|
||||
return 0
|
||||
st := string(raw)
|
||||
i := strings.LastIndexByte(st, ')') // the comm field may hold spaces and parentheses
|
||||
if i < 0 {
|
||||
return false, false
|
||||
}
|
||||
f := strings.Fields(st[i+1:]) // f[0] is field 3 (state), so field 22 (starttime) is f[19]
|
||||
if len(f) < 20 {
|
||||
return false, false
|
||||
}
|
||||
start, err := strconv.ParseUint(f[19], 10, 64)
|
||||
if err != nil {
|
||||
return false, false
|
||||
}
|
||||
// Proxmox's rule exactly: the start tick matches (a pstart of 0 is not checked) and the process is not a zombie.
|
||||
return (u.PStart == 0 || start == u.PStart) && f[0] != "Z", true
|
||||
}
|
||||
|
||||
// procBootTime reads the kernel's boot time through poweroff.ProcBtime (no second /proc/stat parser).
|
||||
func procBootTime() (int64, bool) {
|
||||
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
|
||||
return b, err == nil && b > 0
|
||||
}
|
||||
|
||||
func (c *staleLockController) logDeadOnce(t proxmox.TaskStatus, vmid int, boot int64) {
|
||||
func (c *staleLockController) logDeadOnce(t proxmox.TaskStatus, vmid int) {
|
||||
c.deadMu.Lock()
|
||||
defer c.deadMu.Unlock()
|
||||
if c.deadSeen == nil {
|
||||
@@ -289,8 +309,8 @@ func (c *staleLockController) logDeadOnce(t proxmox.TaskStatus, vmid int, boot i
|
||||
return
|
||||
}
|
||||
c.deadSeen[t.UPID] = true
|
||||
c.logger.Warn("backup-in-flight: a vzdump task from before this boot is listed as active — it is dead, not running",
|
||||
"vmid", vmid, "upid", t.UPID, "task_start", taskStart(t), "host_boot", boot)
|
||||
c.logger.Warn("backup-in-flight: a vzdump task is listed as active but its worker process is gone — it is dead, not running",
|
||||
"vmid", vmid, "upid", t.UPID)
|
||||
}
|
||||
|
||||
func (c *staleLockController) HasVzdumpSnapshot(ctx context.Context, vmid int) (bool, error) {
|
||||
|
||||
@@ -15,86 +15,157 @@ import (
|
||||
)
|
||||
|
||||
// R-1008: after a power cut Proxmox keeps the dead vzdump task in `tasks?source=active` (measured on demo-hp
|
||||
// 2026-10-14: still RUNNING after the boot, started 136 s before it). A task that started before this boot is
|
||||
// dead, not in flight. These tests pin the helper (vzdumpInFlight), the production controller, and three of the
|
||||
// four callers end to end; the fourth (the controller supervisor) by the structural pin at the bottom.
|
||||
// 2026-10-14: still RUNNING ~30–60 s after the boot). A task whose worker process is gone is dead, not in flight.
|
||||
// Liveness is Proxmox's own rule (pid + pstart vs /proc/<pid>/stat field 22), never the wall clock: the first version
|
||||
// compared start time with btime and a clock step after the boot could make a live task look dead (security review
|
||||
// 2026-10-14). These tests pin the helper, the production controller, three callers end to end; the fourth (the
|
||||
// controller supervisor) by the structural pin at the bottom.
|
||||
|
||||
const testBoot = int64(1791967481) // the C3 boot (power-cut-fixes-2026-10-14)
|
||||
// A worker: pid 0x1E6EEC = 1994476, pstart 0x1B4A0D = 1788429 (UPID hex fields).
|
||||
const (
|
||||
testPID = 1994476
|
||||
testPStart = 1788429
|
||||
)
|
||||
|
||||
func vz(vmid string, start int64) proxmox.TaskStatus {
|
||||
return proxmox.TaskStatus{UPID: "UPID:demo-hp:001E6EEC:00000001:6ACF4071:vzdump:" + vmid + ":felhom-agent@pve!agent:",
|
||||
Type: "vzdump", ID: vmid, Status: "RUNNING", StartTime: start}
|
||||
func vz(vmid string) proxmox.TaskStatus {
|
||||
return proxmox.TaskStatus{UPID: "UPID:demo-hp:001E6EEC:001B4A0D:6ACF4071:vzdump:" + vmid + ":felhom-agent@pve!agent:",
|
||||
Type: "vzdump", ID: vmid, Status: "RUNNING", StartTime: 1791967345}
|
||||
}
|
||||
|
||||
// RED-PROOF: make vzdumpInFlight return true for every matching task (drop the `start >= boot` test) → this fails.
|
||||
func TestVzdumpInFlight_PreBootTaskIsDead(t *testing.T) {
|
||||
running, dead := vzdumpInFlight([]proxmox.TaskStatus{vz("9201", testBoot-136)}, 9201, testBoot, true)
|
||||
if running || len(dead) != 1 {
|
||||
t.Fatalf("a task from before the boot must be dead: running=%v dead=%d", running, len(dead))
|
||||
// procFixture writes <dir>/<pid>/stat with the given start ticks (comm with a space and a parenthesis, as real ones can).
|
||||
func procFixture(t *testing.T, pid int, start int) string {
|
||||
return procFixtureState(t, pid, start, "S")
|
||||
}
|
||||
|
||||
func procFixtureState(t *testing.T, pid int, start int, state string) string {
|
||||
t.Helper()
|
||||
d := t.TempDir()
|
||||
// PID 1 is always visible on a real host; procFixtureHidden leaves it out
|
||||
if err := os.MkdirAll(filepath.Join(d, "1"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVzdumpInFlight_PostBootTaskRuns(t *testing.T) {
|
||||
for _, start := range []int64{testBoot, testBoot + 600} {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201", start)}, 9201, testBoot, true); !running {
|
||||
t.Fatalf("a task started at %d (boot %d) must be in flight", start, testBoot)
|
||||
if err := os.WriteFile(filepath.Join(d, "1", "stat"), []byte("1 (systemd) S 0 1 1 0 -1 0 0 0 0 0 0 0 0 0 20 0 1 0 2 0 0\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pid > 0 {
|
||||
p := filepath.Join(d, itoaT(pid))
|
||||
if err := os.MkdirAll(p, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f := []string{itoaT(pid), "(task UPID:x (vzdump))", state}
|
||||
for i := 4; i <= 21; i++ {
|
||||
f = append(f, "0")
|
||||
}
|
||||
f = append(f, itoaT(start), "0", "0")
|
||||
if err := os.WriteFile(filepath.Join(p, "stat"), []byte(strings.Join(f, " ")+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the `!bootOK` early return → a pre-boot task with an unreadable boot is called dead → fails.
|
||||
func TestVzdumpInFlight_UnreadableBootFailsSafe(t *testing.T) {
|
||||
// the boot value is a real one on purpose: only bootOK=false may make this „running"
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201", testBoot-136)}, 9201, testBoot, false); !running {
|
||||
t.Fatal("an unreadable boot time must count as running (fail safe)")
|
||||
// RED-PROOF: make taskAlive return (true, true) for a missing /proc entry → the dead task runs → fails.
|
||||
func TestVzdumpInFlight_WorkerGoneIsDead(t *testing.T) {
|
||||
running, dead := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, 0, 0))
|
||||
if running || len(dead) != 1 {
|
||||
t.Fatalf("a task whose worker is gone must be dead: running=%v dead=%d", running, len(dead))
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: make taskStart ignore the UPID → StartTime 0 counts as unknown → running; the first case fails.
|
||||
func TestVzdumpInFlight_StartFromUPIDThenUnknownFailsSafe(t *testing.T) {
|
||||
tk := vz("9201", 0) // no starttime in the list: the UPID's 6ACF4071 = 1791967345 (before the boot)
|
||||
if running, dead := vzdumpInFlight([]proxmox.TaskStatus{tk}, 9201, testBoot, true); running || len(dead) != 1 {
|
||||
t.Fatalf("the UPID's start time must be used: running=%v", running)
|
||||
// RED-PROOF: drop the `start == u.PStart` comparison (alive whenever the pid exists) → a reused pid runs → fails.
|
||||
func TestVzdumpInFlight_ReusedPidIsDead(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, testPID, 42)); running {
|
||||
t.Fatal("the pid exists but started at another tick: another process, the task is dead")
|
||||
}
|
||||
tk.UPID = "garbage"
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{tk}, 9201, testBoot, true); !running {
|
||||
t.Fatal("an unknown start time must count as running (fail safe)")
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the `f[0] != "Z"` test → a zombie worker counts as running → fails.
|
||||
func TestVzdumpInFlight_ZombieIsDead(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixtureState(t, testPID, testPStart, "Z")); running {
|
||||
t.Fatal("a zombie worker is not running (Proxmox's own rule)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVzdumpInFlight_LiveWorkerRuns(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, testPID, testPStart)); !running {
|
||||
t.Fatal("the worker lives (same pid, same start tick): in flight")
|
||||
}
|
||||
}
|
||||
|
||||
// The reason for the rewrite: the wall clock plays no part. A live task whose list start time lies „before the boot"
|
||||
// (a clock step after the boot) is still running.
|
||||
func TestVzdumpInFlight_ClockStepCannotKillALiveTask(t *testing.T) {
|
||||
tk := vz("9201")
|
||||
tk.StartTime = 1 // absurdly early by the wall clock
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{tk}, 9201, procFixture(t, testPID, testPStart)); !running {
|
||||
t.Fatal("a live worker must count as running whatever its wall-clock start says")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: make an unparsable UPID or unreadable stat „known dead" → fails.
|
||||
func TestVzdumpInFlight_DoubtFailsSafe(t *testing.T) {
|
||||
bad := vz("9201")
|
||||
bad.UPID = "garbage"
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{bad}, 9201, procFixture(t, 0, 0)); !running {
|
||||
t.Fatal("an unparsable UPID must count as running (fail safe)")
|
||||
}
|
||||
d := procFixture(t, testPID, testPStart)
|
||||
if err := os.WriteFile(filepath.Join(d, itoaT(testPID), "stat"), []byte("garbage"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, d); !running {
|
||||
t.Fatal("an unparsable stat must count as running (fail safe)")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the PID-1 visibility check → a hidden /proc makes the live task „gone" → fails.
|
||||
func TestVzdumpInFlight_HiddenProcFailsSafe(t *testing.T) {
|
||||
d := t.TempDir() // nothing visible: not even PID 1 (hidepid / ProtectProc)
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, d); !running {
|
||||
t.Fatal("a /proc that hides other users' processes must count as running (fail safe)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVzdumpInFlight_OtherGuestOrTypeIgnored(t *testing.T) {
|
||||
tasks := []proxmox.TaskStatus{vz("9202", testBoot+5), {Type: "vzsnapshot", ID: "9201", StartTime: testBoot + 5}}
|
||||
if running, dead := vzdumpInFlight(tasks, 9201, testBoot, true); running || len(dead) != 0 {
|
||||
tasks := []proxmox.TaskStatus{vz("9202"), {UPID: vz("9201").UPID, Type: "vzsnapshot", ID: "9201"}}
|
||||
if running, dead := vzdumpInFlight(tasks, 9201, procFixture(t, testPID, testPStart)); running || len(dead) != 0 {
|
||||
t.Fatalf("other guests' and other types' tasks are not 9201's backup: running=%v dead=%d", running, len(dead))
|
||||
}
|
||||
}
|
||||
|
||||
func inflightCtrl(api *fakeStaleLockAPI, runner *recordingRunner, logger *slog.Logger) *staleLockController {
|
||||
func inflightCtrl(api *fakeStaleLockAPI, runner *recordingRunner, logger *slog.Logger, proc string) *staleLockController {
|
||||
c := NewStaleLockController(api, runner, "felhom", logger).(*staleLockController)
|
||||
c.bootTime = func() (int64, bool) { return testBoot, true }
|
||||
c.procDir = proc
|
||||
return c
|
||||
}
|
||||
|
||||
// The production controller answers through the helper, and logs a dead task ONCE.
|
||||
// RED-PROOF: make BackupRunning ignore bootTime (the pre-R-1008 body) → returns true → fails.
|
||||
// RED-PROOF: make BackupRunning ignore the helper (any listed task runs) → fails.
|
||||
func TestStaleLockController_BackupRunningIgnoresDeadTask(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201", testBoot-136)}}
|
||||
c := inflightCtrl(api, &recordingRunner{}, slog.New(slog.NewTextHandler(&buf, nil)))
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")}}
|
||||
c := inflightCtrl(api, &recordingRunner{}, slog.New(slog.NewTextHandler(&buf, nil)), procFixture(t, 0, 0))
|
||||
for i := 0; i < 3; i++ {
|
||||
if running, err := c.BackupRunning(context.Background(), 9201); err != nil || running {
|
||||
t.Fatalf("dead pre-boot task reported running=%v err=%v", running, err)
|
||||
t.Fatalf("dead task reported running=%v err=%v", running, err)
|
||||
}
|
||||
}
|
||||
if n := strings.Count(buf.String(), "backup-in-flight: a vzdump task from before this boot"); n != 1 {
|
||||
if n := strings.Count(buf.String(), "backup-in-flight: a vzdump task is listed as active but its worker process is gone"); n != 1 {
|
||||
t.Fatalf("the dead task must be logged exactly once, got %d", n)
|
||||
}
|
||||
api.tasks = []proxmox.TaskStatus{vz("9201", testBoot+30)}
|
||||
c.procDir = procFixture(t, testPID, testPStart)
|
||||
if running, _ := c.BackupRunning(context.Background(), 9201); !running {
|
||||
t.Fatal("a post-boot task must be running")
|
||||
t.Fatal("a live worker must be running")
|
||||
}
|
||||
}
|
||||
|
||||
func fixtureFor(t *testing.T, alive bool) string {
|
||||
if alive {
|
||||
return procFixture(t, testPID, testPStart)
|
||||
}
|
||||
return procFixture(t, 0, 0)
|
||||
}
|
||||
|
||||
func poolOf(vmids ...int) proxmox.PoolInfo {
|
||||
var p proxmox.PoolInfo
|
||||
raw := `{"members":[`
|
||||
@@ -113,16 +184,16 @@ func poolOf(vmids ...int) proxmox.PoolInfo {
|
||||
// „genuinely in-flight — leaving the lock").
|
||||
func TestCaller_StartupStaleLockClearsAfterPowerCut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
start int64
|
||||
alive bool
|
||||
cleared bool
|
||||
}{{testBoot - 136, true}, {testBoot + 30, false}} {
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201", tc.start)},
|
||||
}{{false, true}, {true, false}} {
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")},
|
||||
cfg: map[int]proxmox.GuestConfig{9201: staleCfg("backup", true)}}
|
||||
runner := &recordingRunner{}
|
||||
s := &Server{staleLock: inflightCtrl(api, runner, nil), logger: slog.New(slog.NewTextHandler(discardW{}, nil))}
|
||||
s := &Server{staleLock: inflightCtrl(api, runner, nil, fixtureFor(t, tc.alive)), logger: slog.New(slog.NewTextHandler(discardW{}, nil))}
|
||||
s.recoverOneStaleLock(context.Background(), proxmox.Guest{VMID: 9201, Status: "running"})
|
||||
if runner.unlocked("9201") != tc.cleared {
|
||||
t.Fatalf("task start %d: unlocked=%v, want %v", tc.start, runner.unlocked("9201"), tc.cleared)
|
||||
t.Fatalf("worker alive=%v: unlocked=%v, want %v", tc.alive, runner.unlocked("9201"), tc.cleared)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -130,15 +201,15 @@ func TestCaller_StartupStaleLockClearsAfterPowerCut(t *testing.T) {
|
||||
// Caller 2 — the guest-power watchdog starts a stopped guest despite a dead pre-boot vzdump task.
|
||||
func TestCaller_GuestPowerStartsGuestAfterPowerCut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
start int64
|
||||
alive bool
|
||||
started bool
|
||||
}{{testBoot - 136, true}, {testBoot + 30, false}} {
|
||||
}{{false, true}, {true, false}} {
|
||||
api := &fakeStaleLockAPI{lxc: []proxmox.Guest{{VMID: 9201, Status: "stopped"}}, pool: poolOf(9201),
|
||||
tasks: []proxmox.TaskStatus{vz("9201", tc.start)}, cfg: map[int]proxmox.GuestConfig{9201: staleCfg("", true)}}
|
||||
s := gpServer(t, inflightCtrl(api, &recordingRunner{}, nil), nil)
|
||||
tasks: []proxmox.TaskStatus{vz("9201")}, cfg: map[int]proxmox.GuestConfig{9201: staleCfg("", true)}}
|
||||
s := gpServer(t, inflightCtrl(api, &recordingRunner{}, nil, fixtureFor(t, tc.alive)), nil)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if (len(api.started) == 1) != tc.started {
|
||||
t.Fatalf("task start %d: started=%v, want %v", tc.start, api.started, tc.started)
|
||||
t.Fatalf("worker alive=%v: started=%v, want %v", tc.alive, api.started, tc.started)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -146,13 +217,13 @@ func TestCaller_GuestPowerStartsGuestAfterPowerCut(t *testing.T) {
|
||||
// Caller 3 — the household power-off guard does not refuse „backup running" for a dead task.
|
||||
func TestCaller_PowerOffJobBlockerIgnoresDeadTask(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
start int64
|
||||
alive bool
|
||||
want string
|
||||
}{{testBoot - 136, ""}, {testBoot + 30, PowerOffBackup}} {
|
||||
}{{false, ""}, {true, PowerOffBackup}} {
|
||||
rig := newPowerRig(t, 0)
|
||||
rig.srv.staleLock = inflightCtrl(&fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201", tc.start)}}, &recordingRunner{}, nil)
|
||||
rig.srv.staleLock = inflightCtrl(&fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")}}, &recordingRunner{}, nil, fixtureFor(t, tc.alive))
|
||||
if code, _ := rig.srv.jobBlocker(context.Background(), 9201); code != tc.want {
|
||||
t.Fatalf("task start %d: jobBlocker=%q, want %q", tc.start, code, tc.want)
|
||||
t.Fatalf("worker alive=%v: jobBlocker=%q, want %q", tc.alive, code, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -189,7 +260,7 @@ func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
|
||||
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
|
||||
}
|
||||
b, _ := os.ReadFile("stalelock.go")
|
||||
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, boot, ok)") {
|
||||
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
|
||||
t.Error("staleLockController.BackupRunning no longer answers through vzdumpInFlight")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,445 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Physical-disk identity of a storage place — felhom.eu architecture/16-storage-physical-identity.md §3.
|
||||
//
|
||||
// A storage place (a PVE storage, an enrolled drive, the guest's data volume, a network share) is
|
||||
// reduced to the PHYSICAL whole disks under it — partition → parent disk, device-mapper / LVM thin →
|
||||
// sysfs slaves (recursive), loop → its backing file → the filesystem holding that file — and each disk
|
||||
// is named by a key the agent trusts: `eui:` for NVMe, `serial:<bus>:<model>:<serial>` from udev,
|
||||
// `wwn:` only when no serial exists and the WWN carries a vendor (OUI ≠ 000000).
|
||||
//
|
||||
// The verdict is AGENT-COMPUTED evidence (`16` §10): it is read from /sys, /run/udev/data and
|
||||
// /proc/mounts only, never from a caller's claim, so a compromised controller cannot make two places
|
||||
// "different disks". `proven` means every disk under the place has a key the agent trusts; anything
|
||||
// else is `unknown` with ONE reason, chosen by precedence (vm > network > usb_bridge > weak_wwn >
|
||||
// no_serial). Nothing is cached: every call reads the current topology, so a disk swapped in the same
|
||||
// bay is re-judged on the next read (the "right before each Tier-2 run" rule is the caller calling).
|
||||
//
|
||||
// Pinned by identity_test.go (fixture /sys + udev trees, one per §3 case) — the spike that proved the
|
||||
// walk on real devices: felhom.eu documentation/audits/storage-identity-2026-10-15/.
|
||||
|
||||
const (
|
||||
IdentityProven = "proven"
|
||||
IdentityUnknown = "unknown"
|
||||
|
||||
ReasonVM = "vm"
|
||||
ReasonNetwork = "network"
|
||||
ReasonUSBBridge = "usb_bridge"
|
||||
ReasonWeakWWN = "weak_wwn"
|
||||
ReasonNoSerial = "no_serial"
|
||||
|
||||
PlaceKindSystem = "system"
|
||||
PlaceKindInternal = "internal"
|
||||
PlaceKindDisk = "disk"
|
||||
PlaceKindPartition = "partition"
|
||||
PlaceKindNetwork = "network"
|
||||
)
|
||||
|
||||
// reasonRank orders the unknown reasons: the first one a place hits, by this precedence, is reported.
|
||||
var reasonRank = map[string]int{ReasonVM: 0, ReasonNetwork: 1, ReasonUSBBridge: 2, ReasonWeakWWN: 3, ReasonNoSerial: 4}
|
||||
|
||||
// PlaceIdentity is what a storage place reports.
|
||||
type PlaceIdentity struct {
|
||||
PhysicalDisks []string // disk keys, sorted, deduplicated
|
||||
Identity string // proven | unknown | "" (not computed — e.g. the device is absent)
|
||||
Reason string // set only when Identity == unknown
|
||||
}
|
||||
|
||||
// Computed reports whether a verdict was reached at all.
|
||||
func (p PlaceIdentity) Computed() bool { return p.Identity != "" }
|
||||
|
||||
// IdentityHost is the read surface the resolver needs. SysfsIdentityHost is production; tests use the
|
||||
// same struct pointed at fixture trees.
|
||||
type IdentityHost interface {
|
||||
Mounts() ([]Mount, error)
|
||||
// KernelName maps a /dev path (symlinks allowed: /dev/mapper/x, /dev/pve/y) to its kernel block name.
|
||||
KernelName(dev string) (string, bool)
|
||||
// Parent returns the parent disk's kernel name when name is a partition.
|
||||
Parent(name string) (string, bool)
|
||||
// Slaves lists /sys/class/block/<name>/slaves.
|
||||
Slaves(name string) []string
|
||||
// LoopBackingFile returns a loop device's backing file.
|
||||
LoopBackingFile(name string) (string, bool)
|
||||
// UdevProps returns the udev E: properties of a block device.
|
||||
UdevProps(name string) map[string]string
|
||||
// Virt returns the VM technology ("none" on bare metal).
|
||||
Virt() string
|
||||
}
|
||||
|
||||
// SysfsIdentityHost reads the real host. Every path is a field so tests can point it at fixtures.
|
||||
type SysfsIdentityHost struct {
|
||||
SysClass string // default /sys/class/block
|
||||
UdevData string // default /run/udev/data
|
||||
ProcMounts string // default /proc/mounts
|
||||
DevResolve func(string) (string, error) // default filepath.EvalSymlinks
|
||||
VirtFn func() string // default `systemd-detect-virt --vm`, read once
|
||||
|
||||
virtOnce sync.Once
|
||||
virt string
|
||||
}
|
||||
|
||||
// NewSysfsIdentityHost returns the production reader.
|
||||
func NewSysfsIdentityHost() *SysfsIdentityHost { return &SysfsIdentityHost{} }
|
||||
|
||||
func (h *SysfsIdentityHost) sysClass() string {
|
||||
if h.SysClass != "" {
|
||||
return h.SysClass
|
||||
}
|
||||
return "/sys/class/block"
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Mounts() ([]Mount, error) {
|
||||
return (&ProcHostReader{ProcMounts: h.ProcMounts}).Mounts()
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) KernelName(dev string) (string, bool) {
|
||||
if !strings.HasPrefix(dev, "/dev/") {
|
||||
return "", false
|
||||
}
|
||||
resolve := h.DevResolve
|
||||
if resolve == nil {
|
||||
resolve = filepath.EvalSymlinks
|
||||
}
|
||||
real := dev
|
||||
if r, err := resolve(dev); err == nil {
|
||||
real = r
|
||||
}
|
||||
name := filepath.Base(real)
|
||||
if _, err := os.Stat(filepath.Join(h.sysClass(), name)); err != nil {
|
||||
return "", false
|
||||
}
|
||||
return name, true
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Parent(name string) (string, bool) {
|
||||
p := filepath.Join(h.sysClass(), name)
|
||||
if _, err := os.Stat(filepath.Join(p, "partition")); err != nil {
|
||||
return "", false
|
||||
}
|
||||
real, err := filepath.EvalSymlinks(p)
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
return filepath.Base(filepath.Dir(real)), true
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Slaves(name string) []string {
|
||||
entries, err := os.ReadDir(filepath.Join(h.sysClass(), name, "slaves"))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) LoopBackingFile(name string) (string, bool) {
|
||||
if !strings.HasPrefix(name, "loop") {
|
||||
return "", false
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(h.sysClass(), name, "loop", "backing_file"))
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
f := strings.TrimSpace(string(b))
|
||||
return f, f != ""
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) UdevProps(name string) map[string]string {
|
||||
out := map[string]string{}
|
||||
devB, err := os.ReadFile(filepath.Join(h.sysClass(), name, "dev"))
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
dir := h.UdevData
|
||||
if dir == "" {
|
||||
dir = "/run/udev/data"
|
||||
}
|
||||
f, err := os.Open(filepath.Join(dir, "b"+strings.TrimSpace(string(devB))))
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if !strings.HasPrefix(line, "E:") {
|
||||
continue
|
||||
}
|
||||
k, v, ok := strings.Cut(line[2:], "=")
|
||||
if ok {
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Virt() string {
|
||||
h.virtOnce.Do(func() {
|
||||
if h.VirtFn != nil {
|
||||
h.virt = h.VirtFn()
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
// --vm: a container is not a reason (the agent runs on the host); exit 1 + "none" on metal.
|
||||
out, _ := exec.CommandContext(ctx, "systemd-detect-virt", "--vm").Output()
|
||||
h.virt = strings.TrimSpace(string(out))
|
||||
if h.virt == "" {
|
||||
h.virt = "unknown"
|
||||
}
|
||||
})
|
||||
return h.virt
|
||||
}
|
||||
|
||||
// IdentityResolver turns devices and host paths into PlaceIdentity.
|
||||
type IdentityResolver struct {
|
||||
Host IdentityHost
|
||||
}
|
||||
|
||||
// NewIdentityResolver returns a resolver over the real host.
|
||||
func NewIdentityResolver() *IdentityResolver {
|
||||
return &IdentityResolver{Host: NewSysfsIdentityHost()}
|
||||
}
|
||||
|
||||
var networkFSTypes = map[string]bool{
|
||||
"nfs": true, "nfs4": true, "cifs": true, "smb3": true, "fuse.sshfs": true, "9p": true, "autofs": true, "ceph": true, "glusterfs": true,
|
||||
}
|
||||
|
||||
// NetworkPlace is the fixed verdict for a network share or a remote store.
|
||||
func NetworkPlace() PlaceIdentity {
|
||||
return PlaceIdentity{PhysicalDisks: []string{}, Identity: IdentityUnknown, Reason: ReasonNetwork}
|
||||
}
|
||||
|
||||
// MountFor returns the mount that holds path (longest mountpoint prefix; the LAST entry wins on an equal
|
||||
// mountpoint, which is the one stacked on top — an automount's real filesystem over its autofs trigger).
|
||||
func MountFor(mounts []Mount, path string) (Mount, bool) {
|
||||
p := filepath.Clean(path)
|
||||
var best Mount
|
||||
found := false
|
||||
for _, m := range mounts {
|
||||
mp := filepath.Clean(m.MountPoint)
|
||||
if p != mp && !strings.HasPrefix(p, strings.TrimSuffix(mp, "/")+"/") {
|
||||
continue
|
||||
}
|
||||
if !found || len(mp) >= len(filepath.Clean(best.MountPoint)) {
|
||||
best, found = m, true
|
||||
}
|
||||
}
|
||||
return best, found
|
||||
}
|
||||
|
||||
// HostPath resolves the filesystem holding a HOST path.
|
||||
func (r *IdentityResolver) HostPath(path string) PlaceIdentity {
|
||||
return r.hostPath(path, 0)
|
||||
}
|
||||
|
||||
func (r *IdentityResolver) hostPath(path string, depth int) PlaceIdentity {
|
||||
if r == nil || r.Host == nil || depth > 4 {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
mounts, err := r.Host.Mounts()
|
||||
if err != nil {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
m, ok := MountFor(mounts, path)
|
||||
if !ok {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
if networkFSTypes[m.FSType] || !strings.HasPrefix(m.Device, "/dev/") {
|
||||
return NetworkPlace()
|
||||
}
|
||||
return r.device(m.Device, depth)
|
||||
}
|
||||
|
||||
// Device resolves a block device (a /dev path, symlinks allowed).
|
||||
func (r *IdentityResolver) Device(dev string) PlaceIdentity {
|
||||
return r.device(dev, 0)
|
||||
}
|
||||
|
||||
func (r *IdentityResolver) device(dev string, depth int) PlaceIdentity {
|
||||
if r == nil || r.Host == nil || dev == "" {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
name, ok := r.Host.KernelName(dev)
|
||||
if !ok {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
keys := map[string]bool{}
|
||||
reason := ""
|
||||
if !r.walk(name, depth, map[string]bool{}, keys, &reason) {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
out := PlaceIdentity{PhysicalDisks: sortedKeys(keys), Identity: IdentityProven}
|
||||
if reason != "" {
|
||||
out.Identity, out.Reason = IdentityUnknown, reason
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// walk adds the keys of every physical disk under name; ok=false when a branch cannot be grounded.
|
||||
func (r *IdentityResolver) walk(name string, depth int, seen, keys map[string]bool, reason *string) bool {
|
||||
if seen[name] {
|
||||
return true // a diamond (a thin pool's tmeta and tdata on one partition): already counted
|
||||
}
|
||||
if len(seen) > 64 {
|
||||
return false
|
||||
}
|
||||
seen[name] = true
|
||||
if parent, ok := r.Host.Parent(name); ok {
|
||||
return r.walk(parent, depth, seen, keys, reason)
|
||||
}
|
||||
if file, ok := r.Host.LoopBackingFile(name); ok {
|
||||
p := r.hostPath(file, depth+1)
|
||||
if !p.Computed() {
|
||||
return false
|
||||
}
|
||||
for _, k := range p.PhysicalDisks {
|
||||
keys[k] = true
|
||||
}
|
||||
noteReason(reason, p.Reason)
|
||||
return true
|
||||
}
|
||||
if slaves := r.Host.Slaves(name); len(slaves) > 0 {
|
||||
for _, s := range slaves {
|
||||
if !r.walk(s, depth, seen, keys, reason) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
if strings.HasPrefix(name, "dm-") || strings.HasPrefix(name, "md") || strings.HasPrefix(name, "loop") {
|
||||
return false // a virtual device with nothing beneath it cannot be grounded
|
||||
}
|
||||
key, why := r.diskKey(name)
|
||||
keys[key] = true
|
||||
noteReason(reason, why)
|
||||
return true
|
||||
}
|
||||
|
||||
func noteReason(cur *string, why string) {
|
||||
if why == "" {
|
||||
return
|
||||
}
|
||||
if *cur == "" || reasonRank[why] < reasonRank[*cur] {
|
||||
*cur = why
|
||||
}
|
||||
}
|
||||
|
||||
// diskKey names one whole disk and says why it is not trusted ("" when it is).
|
||||
func (r *IdentityResolver) diskKey(name string) (key, reason string) {
|
||||
p := r.Host.UdevProps(name)
|
||||
serial := strings.TrimSpace(p["ID_SERIAL_SHORT"])
|
||||
model := strings.TrimSpace(p["ID_MODEL"])
|
||||
bus := strings.TrimSpace(p["ID_BUS"])
|
||||
wwn := strings.ToLower(strings.TrimSpace(p["ID_WWN"]))
|
||||
if bus == "" && strings.HasPrefix(name, "nvme") {
|
||||
bus = "nvme"
|
||||
}
|
||||
usb := bus == "usb" || strings.Contains(p["DEVPATH"], "/usb")
|
||||
switch {
|
||||
case strings.HasPrefix(name, "nvme") && (strings.HasPrefix(wwn, "eui.") || strings.HasPrefix(wwn, "nvme.")):
|
||||
key = "eui:" + strings.TrimPrefix(strings.TrimPrefix(wwn, "eui."), "nvme.")
|
||||
case serial != "":
|
||||
key = "serial:" + bus + ":" + model + ":" + serial
|
||||
if usb && FakeBridgeSerial(serial) {
|
||||
reason = ReasonUSBBridge
|
||||
}
|
||||
case wwn != "" && WWNHasVendor(wwn):
|
||||
key = "wwn:" + wwn
|
||||
case wwn != "":
|
||||
key, reason = "wwn:"+wwn, ReasonWeakWWN
|
||||
default:
|
||||
key, reason = "dev:"+name, ReasonNoSerial
|
||||
}
|
||||
if v := r.Host.Virt(); v != "none" {
|
||||
reason = ReasonVM // inside a VM a virtual disk says nothing about the host disk (spike cases 5–6)
|
||||
}
|
||||
return key, reason
|
||||
}
|
||||
|
||||
// WWNHasVendor reports whether an NAA-5/6 WWN carries a registered vendor (OUI ≠ 000000). Other NAA
|
||||
// formats are taken as they are. Measured: the N100's AirDisk SSD reports 0x5000000123456f1f.
|
||||
func WWNHasVendor(wwn string) bool {
|
||||
h := strings.TrimPrefix(strings.ToLower(wwn), "0x")
|
||||
if len(h) < 7 {
|
||||
return false
|
||||
}
|
||||
if h[0] == '5' || h[0] == '6' {
|
||||
return h[1:7] != "000000"
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// FakeBridgeSerial reports a serial that a USB bridge reports for any disk: too short, one repeated
|
||||
// character, or a known placeholder. UNPROVEN on a real enclosure (R-1077) — the rule is the design's.
|
||||
func FakeBridgeSerial(s string) bool {
|
||||
s = strings.ToUpper(strings.TrimSpace(s))
|
||||
if len(s) < 6 {
|
||||
return true
|
||||
}
|
||||
if strings.Count(s, s[:1]) == len(s) {
|
||||
return true
|
||||
}
|
||||
switch s {
|
||||
case "123456789", "1234567890", "123456789ABC", "0123456789ABCDEF", "1234567890ABCDEF", "DEADBEEF":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func sortedKeys(m map[string]bool) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// Relation compares two places (`16` §4). A SHARED disk key is a positive fact, so two places on one disk
|
||||
// are "same, proven" even when that disk is unknown (inside a VM, two parts of one virtual disk). Otherwise
|
||||
// proven is true only when both verdicts are proven; different then means "no disk in common". When proven
|
||||
// is false, different only says the known keys do not overlap — callers must act on proven first.
|
||||
// reason: "" (proven, different), "same_disk", "unmapped" (a side has no verdict), or the unknown reason.
|
||||
func Relation(a, b PlaceIdentity) (different, proven bool, reason string) {
|
||||
if !a.Computed() || !b.Computed() {
|
||||
return false, false, "unmapped"
|
||||
}
|
||||
set := map[string]bool{}
|
||||
for _, k := range a.PhysicalDisks {
|
||||
set[k] = true
|
||||
}
|
||||
shared := false
|
||||
for _, k := range b.PhysicalDisks {
|
||||
if set[k] {
|
||||
shared = true
|
||||
}
|
||||
}
|
||||
if shared {
|
||||
return false, true, "same_disk"
|
||||
}
|
||||
if a.Identity == IdentityProven && b.Identity == IdentityProven {
|
||||
return true, true, ""
|
||||
}
|
||||
why := a.Reason
|
||||
if why == "" || (b.Reason != "" && reasonRank[b.Reason] < reasonRank[why]) {
|
||||
why = b.Reason
|
||||
}
|
||||
return true, false, why
|
||||
}
|
||||
@@ -0,0 +1,257 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Fixture /sys/class/block + /run/udev/data + /proc/mounts trees, one per `16` §3 case. The tree mimics the
|
||||
// kernel: /sys/devices/.../<disk>/<part> real dirs (a partition carries a "partition" file), /sys/class/block
|
||||
// symlinks, a "dev" file per node, udev "E:" lines keyed by major:minor. The cases are the ones the spike
|
||||
// measured on real devices (felhom.eu documentation/audits/storage-identity-2026-10-15/).
|
||||
|
||||
type fx struct {
|
||||
t *testing.T
|
||||
root string
|
||||
minor int
|
||||
mounts []string
|
||||
links map[string]string // /dev path -> kernel name
|
||||
virt string
|
||||
}
|
||||
|
||||
func newFx(t *testing.T) *fx {
|
||||
t.Helper()
|
||||
r := t.TempDir()
|
||||
for _, d := range []string{"sys/class/block", "sys/devices", "udev"} {
|
||||
if err := os.MkdirAll(filepath.Join(r, d), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return &fx{t: t, root: r, links: map[string]string{}, virt: "none"}
|
||||
}
|
||||
|
||||
func (f *fx) write(p, s string) {
|
||||
f.t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte(s), 0o644); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// node makes a block node. parent "" → a top-level device. udev lines are "KEY=VAL".
|
||||
func (f *fx) node(name, parent string, udev ...string) {
|
||||
f.t.Helper()
|
||||
real := filepath.Join(f.root, "sys/devices", name)
|
||||
if parent != "" {
|
||||
pr, _ := filepath.EvalSymlinks(filepath.Join(f.root, "sys/class/block", parent))
|
||||
real = filepath.Join(pr, name)
|
||||
f.write(filepath.Join(real, "partition"), "1\n")
|
||||
}
|
||||
f.minor++
|
||||
dev := "8:" + strconv.Itoa(f.minor)
|
||||
f.write(filepath.Join(real, "dev"), dev+"\n")
|
||||
if err := os.Symlink(real, filepath.Join(f.root, "sys/class/block", name)); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
var b strings.Builder
|
||||
for _, u := range udev {
|
||||
b.WriteString("E:" + u + "\n")
|
||||
}
|
||||
f.write(filepath.Join(f.root, "udev", "b"+dev), b.String())
|
||||
f.links["/dev/"+name] = name
|
||||
}
|
||||
|
||||
func (f *fx) slaves(name string, slaves ...string) {
|
||||
f.t.Helper()
|
||||
real, _ := filepath.EvalSymlinks(filepath.Join(f.root, "sys/class/block", name))
|
||||
for _, s := range slaves {
|
||||
if err := os.MkdirAll(filepath.Join(real, "slaves", s), 0o755); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fx) loop(name, file string) {
|
||||
real, _ := filepath.EvalSymlinks(filepath.Join(f.root, "sys/class/block", name))
|
||||
f.write(filepath.Join(real, "loop", "backing_file"), file+"\n")
|
||||
}
|
||||
|
||||
func (f *fx) mount(dev, mp, fstype string) {
|
||||
f.mounts = append(f.mounts, dev+" "+mp+" "+fstype+" rw 0 0")
|
||||
}
|
||||
|
||||
func (f *fx) resolver() *IdentityResolver {
|
||||
f.write(filepath.Join(f.root, "mounts"), strings.Join(f.mounts, "\n")+"\n")
|
||||
return &IdentityResolver{Host: &SysfsIdentityHost{
|
||||
SysClass: filepath.Join(f.root, "sys/class/block"),
|
||||
UdevData: filepath.Join(f.root, "udev"),
|
||||
ProcMounts: filepath.Join(f.root, "mounts"),
|
||||
DevResolve: func(p string) (string, error) {
|
||||
if n, ok := f.links[p]; ok {
|
||||
return "/dev/" + n, nil
|
||||
}
|
||||
return p, nil
|
||||
},
|
||||
VirtFn: func() string { return f.virt },
|
||||
}}
|
||||
}
|
||||
|
||||
var sanDisk = []string{"ID_BUS=ata", "ID_MODEL=SanDisk_X600", "ID_SERIAL_SHORT=182195804614", "ID_WWN=0x5001b448b60a9d6e"}
|
||||
|
||||
// oneDiskLVM builds the PVE default layout on one SATA SSD: sda3 → pve-root, pve-data tmeta/tdata → tpool →
|
||||
// a guest volume (the shape of all three test boxes).
|
||||
func oneDiskLVM(f *fx) {
|
||||
f.node("sda", "", sanDisk...)
|
||||
f.node("sda3", "sda")
|
||||
for _, d := range []string{"dm-1", "dm-2", "dm-3", "dm-4", "dm-6"} {
|
||||
f.node(d, "")
|
||||
}
|
||||
f.slaves("dm-1", "sda3")
|
||||
f.slaves("dm-2", "sda3")
|
||||
f.slaves("dm-3", "sda3")
|
||||
f.slaves("dm-4", "dm-2", "dm-3")
|
||||
f.slaves("dm-6", "dm-4")
|
||||
f.links["/dev/mapper/pve-root"] = "dm-1"
|
||||
f.links["/dev/mapper/pve-data-tpool"] = "dm-4"
|
||||
f.links["/dev/pve/vm-9201-disk-1"] = "dm-6"
|
||||
f.mount("/dev/mapper/pve-root", "/", "ext4")
|
||||
}
|
||||
|
||||
func TestIdentity_OneDiskLVM_AllPlacesSameProvenDisk(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
r := f.resolver()
|
||||
want := []string{"serial:ata:SanDisk_X600:182195804614"}
|
||||
for _, p := range []PlaceIdentity{r.HostPath("/var/lib/vz"), r.Device("/dev/mapper/pve-data-tpool"), r.Device("/dev/pve/vm-9201-disk-1")} {
|
||||
if p.Identity != IdentityProven || !reflect.DeepEqual(p.PhysicalDisks, want) {
|
||||
t.Fatalf("want proven %v, got %+v", want, p)
|
||||
}
|
||||
}
|
||||
diff, proven, why := Relation(r.HostPath("/var/lib/vz"), r.Device("/dev/pve/vm-9201-disk-1"))
|
||||
if diff || !proven || why != "same_disk" {
|
||||
t.Fatalf("root vs guest volume: want same+proven, got diff=%v proven=%v why=%q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_TwoDisks_ProvenDifferent(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.node("sdb", "", "ID_BUS=ata", "ID_MODEL=TOSHIBA_MQ04ABF100", "ID_SERIAL_SHORT=65NOP3HDT", "ID_WWN=0x5000039ddb108568", "DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-3/block/sdb")
|
||||
f.mount("/dev/sdb", "/mnt/hdd_1", "ext4")
|
||||
f.mount("/dev/sdb", "/mnt/felhom-drives/hdd_1", "ext4") // the felhom-data bind shows the source device
|
||||
r := f.resolver()
|
||||
drive := r.HostPath("/mnt/felhom-drives/hdd_1/userdata")
|
||||
if drive.Identity != IdentityProven || drive.PhysicalDisks[0] != "serial:ata:TOSHIBA_MQ04ABF100:65NOP3HDT" {
|
||||
t.Fatalf("USB drive passing its identity: want proven, got %+v", drive)
|
||||
}
|
||||
diff, proven, why := Relation(r.Device("/dev/pve/vm-9201-disk-1"), drive)
|
||||
if !diff || !proven || why != "" {
|
||||
t.Fatalf("system vs drive: want different+proven, got %v %v %q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_TwoPartitionsOneDisk_Same(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("sdc", "", "ID_BUS=usb", "ID_MODEL=Portable", "ID_SERIAL_SHORT=NAAB7C21")
|
||||
f.node("sdc1", "sdc")
|
||||
f.node("sdc2", "sdc")
|
||||
f.mount("/dev/sdc1", "/mnt/a", "ext4")
|
||||
f.mount("/dev/sdc2", "/mnt/b", "ext4")
|
||||
r := f.resolver()
|
||||
diff, proven, why := Relation(r.HostPath("/mnt/a"), r.HostPath("/mnt/b"))
|
||||
if diff || !proven || why != "same_disk" {
|
||||
t.Fatalf("two partitions of one disk: want same+proven, got %v %v %q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_LoopPartitionsOnAFile_ResolveToTheFilesDisk(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("nvme0n1", "", "ID_MODEL=KXG50PNV1T02", "ID_SERIAL_SHORT=58BS11AFT8MQ", "ID_WWN=eui.000000000000001000080d020046c07b")
|
||||
f.mount("/dev/nvme0n1", "/mnt/hdd_1", "ext4")
|
||||
f.node("loop2", "")
|
||||
f.loop("loop2", "/mnt/hdd_1/scratch.img")
|
||||
f.node("loop2p1", "loop2") // the partition step must run BEFORE the loop step (the spike's probe bug)
|
||||
f.mount("/dev/loop2p1", "/tmp/scrA", "ext4")
|
||||
r := f.resolver()
|
||||
p := r.HostPath("/tmp/scrA")
|
||||
if p.Identity != IdentityProven || !reflect.DeepEqual(p.PhysicalDisks, []string{"eui:000000000000001000080d020046c07b"}) {
|
||||
t.Fatalf("loop partition: want the NVMe's eui, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_WWNWithoutVendor(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("sda", "", "ID_BUS=ata", "ID_MODEL=AirDisk_512GB_SSD", "ID_SERIAL_SHORT=QDF922W009654S30EX", "ID_WWN=0x5000000123456f1f")
|
||||
f.node("sdb", "", "ID_BUS=ata", "ID_MODEL=Cheap", "ID_WWN=0x5000000123456f1f")
|
||||
r := f.resolver()
|
||||
if p := r.Device("/dev/sda"); p.Identity != IdentityProven || p.PhysicalDisks[0] != "serial:ata:AirDisk_512GB_SSD:QDF922W009654S30EX" {
|
||||
t.Fatalf("a real serial wins over a vendor-less WWN: got %+v", p)
|
||||
}
|
||||
if p := r.Device("/dev/sdb"); p.Identity != IdentityUnknown || p.Reason != ReasonWeakWWN {
|
||||
t.Fatalf("no serial + vendor-less WWN: want unknown/weak_wwn, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_NoSerial(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("vdb", "", "ID_MODEL=")
|
||||
if p := f.resolver().Device("/dev/vdb"); p.Identity != IdentityUnknown || p.Reason != ReasonNoSerial {
|
||||
t.Fatalf("want unknown/no_serial, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_InsideAVM_EverythingUnknown(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.virt = "kvm"
|
||||
f.node("sdb", "", "ID_BUS=scsi", "ID_MODEL=QEMU_HARDDISK", "ID_SERIAL_SHORT=drive-scsi1")
|
||||
f.mount("/dev/sdb", "/mnt/felhom-drives/adatlemez", "ext4")
|
||||
r := f.resolver()
|
||||
a, b := r.Device("/dev/pve/vm-9201-disk-1"), r.HostPath("/mnt/felhom-drives/adatlemez")
|
||||
for _, p := range []PlaceIdentity{a, b} {
|
||||
if p.Identity != IdentityUnknown || p.Reason != ReasonVM {
|
||||
t.Fatalf("inside a VM: want unknown/vm, got %+v", p)
|
||||
}
|
||||
}
|
||||
diff, proven, why := Relation(a, b)
|
||||
if proven || why != ReasonVM || !diff {
|
||||
t.Fatalf("two virtual disks: want not proven (vm), got %v %v %q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_USBBridgeSerial(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("sdd", "", "ID_BUS=usb", "ID_MODEL=External_USB3.0", "ID_SERIAL_SHORT=000000000000")
|
||||
f.node("sde", "", "ID_BUS=usb", "ID_MODEL=External_USB3.0", "ID_SERIAL_SHORT=20231106A1B2")
|
||||
r := f.resolver()
|
||||
if p := r.Device("/dev/sdd"); p.Identity != IdentityUnknown || p.Reason != ReasonUSBBridge {
|
||||
t.Fatalf("a bridge placeholder serial: want unknown/usb_bridge, got %+v", p)
|
||||
}
|
||||
if p := r.Device("/dev/sde"); p.Identity != IdentityProven {
|
||||
t.Fatalf("a USB disk with a plausible serial: want proven, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_NetworkShareAndUnmapped(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.mount("//192.168.0.123/TesztFilmek", "/mnt/felhom-drives/omv-filmek", "cifs")
|
||||
r := f.resolver()
|
||||
n := r.HostPath("/mnt/felhom-drives/omv-filmek/x")
|
||||
if n.Identity != IdentityUnknown || n.Reason != ReasonNetwork || len(n.PhysicalDisks) != 0 {
|
||||
t.Fatalf("network share: want unknown/network with no disks, got %+v", n)
|
||||
}
|
||||
_, proven, why := Relation(r.HostPath("/"), n)
|
||||
if proven || why != ReasonNetwork {
|
||||
t.Fatalf("system vs network: want not proven (network), got %v %q", proven, why)
|
||||
}
|
||||
if _, proven, why := Relation(PlaceIdentity{}, n); proven || why != "unmapped" {
|
||||
t.Fatalf("an uncomputed side: want unmapped, got %v %q", proven, why)
|
||||
}
|
||||
}
|
||||
@@ -40,6 +40,10 @@ type Observer struct {
|
||||
// R-672 (v0.133.0): a thin pool crossing thinPoolAlarmFraction (data OR metadata) requests an
|
||||
// out-of-band host report at once, so the hub's storage-fill alarm sees it in seconds instead of at
|
||||
// the next 15-minute report. Rising edge per pool; re-armed below thinPoolRearmFraction.
|
||||
// ident (agent v0.157.0, `16` §3) computes each target's physical-disk identity. nil → the fields stay
|
||||
// empty (the pre-0.157 shape), which every consumer reads as "not computed".
|
||||
ident *IdentityResolver
|
||||
|
||||
highMu sync.Mutex
|
||||
high map[string]bool
|
||||
onThinHigh func()
|
||||
@@ -50,6 +54,51 @@ const (
|
||||
thinPoolRearmFraction = 0.85
|
||||
)
|
||||
|
||||
// SetIdentity wires the physical-disk identity resolver (main: NewIdentityResolver()).
|
||||
func (o *Observer) SetIdentity(r *IdentityResolver) { o.ident = r }
|
||||
|
||||
// placeIdentity fills a target's identity fields (`16` §3). Network and remote stores are fixed
|
||||
// "unknown/network"; a dir storage is judged only on its OWN mount (the same rule as backing_device —
|
||||
// never the containing filesystem, so an absent drive is "not computed", never the root disk); the
|
||||
// builtin local and an lvmthin pool are the system disk(s) they sit on.
|
||||
func (o *Observer) placeIdentity(tgt *hub.StorageTarget, s proxmox.Storage) {
|
||||
if o.ident == nil {
|
||||
return
|
||||
}
|
||||
var p PlaceIdentity
|
||||
switch tgt.Type {
|
||||
case hub.StorageTypeNFS, hub.StorageTypeCIFS, hub.StorageTypePBS:
|
||||
tgt.Kind = PlaceKindNetwork
|
||||
p = NetworkPlace()
|
||||
case hub.StorageTypeLocal:
|
||||
tgt.Kind = PlaceKindSystem
|
||||
if s.Path != "" {
|
||||
p = o.ident.HostPath(s.Path)
|
||||
}
|
||||
case hub.StorageTypeLVMThin:
|
||||
tgt.Kind = PlaceKindSystem
|
||||
if s.VGName != "" && s.ThinPool != "" {
|
||||
p = o.ident.Device("/dev/mapper/" + dmEscape(s.VGName) + "-" + dmEscape(s.ThinPool) + "-tpool")
|
||||
}
|
||||
default:
|
||||
if tgt.BackingDevice == "" {
|
||||
return // not its own mount (absent, or a dir on root): not computed
|
||||
}
|
||||
tgt.Kind = PlaceKindDisk
|
||||
if _, known := wholeDiskOf(tgt.BackingDevice); known && !reWholeDisk.MatchString(tgt.BackingDevice) {
|
||||
tgt.Kind = PlaceKindPartition
|
||||
}
|
||||
p = o.ident.Device(tgt.BackingDevice)
|
||||
}
|
||||
if !p.Computed() {
|
||||
return
|
||||
}
|
||||
tgt.PhysicalDisks, tgt.Identity, tgt.IdentityReason = p.PhysicalDisks, p.Identity, p.Reason
|
||||
}
|
||||
|
||||
// dmEscape doubles dashes the way device-mapper names an LV (pve + vm-1-disk-0 → pve-vm--1--disk--0).
|
||||
func dmEscape(s string) string { return strings.ReplaceAll(s, "-", "--") }
|
||||
|
||||
// SetThinHighTrigger wires the out-of-band report request (main: the storage trigger channel).
|
||||
func (o *Observer) SetThinHighTrigger(f func()) { o.onThinHigh = f }
|
||||
|
||||
@@ -303,6 +352,9 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
|
||||
o.noteThinFill(s.Storage+"/data", frac)
|
||||
}
|
||||
|
||||
// Physical-disk identity (`16` §3) — after typ/backing are settled; never from a caller's claim.
|
||||
o.placeIdentity(&tgt, s)
|
||||
|
||||
// SMART-only device hint (v0.95.0): a dir-storage that lives INSIDE a shared filesystem (the
|
||||
// builtin `local` on the LVM root) has empty backing by design, yet we can still read the PHYSICAL
|
||||
// disk's SMART by resolving its containing filesystem. Gated to catDir + no own backing + reachable,
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// The report rows carry the `16` §3 identity per storage type: local + lvmthin → the system disk; a dir
|
||||
// on its own partition → that disk, kind "partition"; nfs/pbs → network; an UNMOUNTED dir → not computed
|
||||
// (never the root disk — the same rule as backing_device).
|
||||
func TestObserve_FillsPhysicalDiskIdentity(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.node("sdb", "", "ID_BUS=ata", "ID_MODEL=TOSHIBA_MQ04ABF100", "ID_SERIAL_SHORT=65NOP3HDT")
|
||||
f.node("sdb1", "sdb")
|
||||
f.mount("/dev/sdb1", "/mnt/usb-backup", "ext4")
|
||||
res := f.resolver()
|
||||
|
||||
api := &fakeStorageAPI{
|
||||
node: "demo",
|
||||
cluster: []proxmox.Storage{
|
||||
{Storage: "local", Type: "dir", Path: "/var/lib/vz"},
|
||||
{Storage: "local-lvm", Type: "lvmthin", VGName: "pve", ThinPool: "data"},
|
||||
{Storage: "usb-backup", Type: "dir", Path: "/mnt/usb-backup"},
|
||||
{Storage: "gone", Type: "dir", Path: "/mnt/gone"},
|
||||
{Storage: "nfs-arch", Type: "nfs", Server: "10.0.0.5", Export: "/x"},
|
||||
},
|
||||
nodeSt: []proxmox.Storage{
|
||||
{Storage: "local", Type: "dir", Path: "/var/lib/vz", Active: 1},
|
||||
{Storage: "local-lvm", Type: "lvmthin", Active: 1},
|
||||
{Storage: "usb-backup", Type: "dir", Path: "/mnt/usb-backup", Active: 1},
|
||||
{Storage: "gone", Type: "dir", Path: "/mnt/gone", Active: 1},
|
||||
{Storage: "nfs-arch", Type: "nfs", Active: 1},
|
||||
},
|
||||
}
|
||||
host := &fakeHostReader{
|
||||
mounts: []Mount{
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"},
|
||||
{Device: "/dev/sdb1", MountPoint: "/mnt/usb-backup", FSType: "ext4"},
|
||||
},
|
||||
exists: map[string]bool{"/dev/sdb1": true, "/dev/mapper/pve-root": true},
|
||||
}
|
||||
ob := NewObserver(api, host, nil, quietLogger())
|
||||
ob.SetIdentity(res)
|
||||
got, err := ob.Observe(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := byName(got)
|
||||
ssd := []string{"serial:ata:SanDisk_X600:182195804614"}
|
||||
for _, n := range []string{"local", "local-lvm"} {
|
||||
if tg := m[n]; tg.Kind != "system" || tg.Identity != "proven" || !reflect.DeepEqual(tg.PhysicalDisks, ssd) {
|
||||
t.Errorf("%s: want system/proven on the SSD, got kind=%q id=%q disks=%v", n, tg.Kind, tg.Identity, tg.PhysicalDisks)
|
||||
}
|
||||
}
|
||||
if tg := m["usb-backup"]; tg.Kind != "partition" || tg.Identity != "proven" || tg.PhysicalDisks[0] != "serial:ata:TOSHIBA_MQ04ABF100:65NOP3HDT" {
|
||||
t.Errorf("usb-backup: got %+v", tg)
|
||||
}
|
||||
if tg := m["nfs-arch"]; tg.Kind != "network" || tg.IdentityReason != "network" {
|
||||
t.Errorf("nfs: got kind=%q reason=%q", tg.Kind, tg.IdentityReason)
|
||||
}
|
||||
if tg := m["gone"]; tg.Identity != "" || len(tg.PhysicalDisks) != 0 {
|
||||
t.Errorf("an unmounted dir must not be computed (never root's disk), got %+v", tg)
|
||||
}
|
||||
// Without a resolver the fields stay empty (the pre-0.157 shape).
|
||||
plain, _ := NewObserver(api, host, nil, quietLogger()).Observe(context.Background())
|
||||
if tg := byName(plain)["local"]; tg.Identity != "" || tg.Kind != "" {
|
||||
t.Errorf("no resolver: want empty identity, got %+v", tg)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user