Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 7ae3081a76 | |||
| 7ecd63cd61 | |||
| a2f3aeeeff | |||
| e7d4ecd607 | |||
| 10ca63aa32 | |||
| 68a2130681 | |||
| bdf25f019c | |||
| c6d9435316 | |||
| f38b03cdb8 | |||
| 3403c87c80 | |||
| 86adbcaf8a | |||
| 24ea960229 | |||
| cff6fb5a45 | |||
| db25b469ba | |||
| b228b44f0e | |||
| 91b9405c81 | |||
| 4c69c25b48 | |||
| c9013bb47d | |||
| 2d1e5d0774 | |||
| f09c53efc1 | |||
| 92d647a6f6 |
@@ -73,3 +73,19 @@ that no longer exists. Each edit is named in the report (file, line, before, aft
|
||||
operator's word: loosen a safety rule, a fence, a „never", a protected machine, a secret rule, or a review step; or
|
||||
remove a rule. When in doubt, it is a rule change, and it goes to the operator. If Claude Code's own permission check
|
||||
asks before such an edit, wait for the operator's click; if it refuses, record that and file the exact line.
|
||||
|
||||
## 6. The system poster stays true
|
||||
|
||||
**A session that changes a fact listed in `architecture/felhom-system-poster.facts.md` (a machine, a
|
||||
role, a traffic path, a backup tier, a time, a retention, a key, a known gap) updates that file in
|
||||
the same commit.** If the change is text only, it also edits the matching text in
|
||||
`felhom-system-poster.html`. If it needs a new drawing, it adds the line
|
||||
**"System poster needs a refresh: <what changed>"** to `STATUS.md`'s "waiting on the operator" list.
|
||||
**The session report names which of the three it did.**
|
||||
|
||||
Why the three-way split: the poster is a drawing made in Claude Design, and **nothing in this
|
||||
repository renders it** — unlike `where-felhom-stands.html`, which has `render_stands.py`. So the
|
||||
facts file is the source of truth and the drawing trails it. `scripts/poster_facts_gate.py` warns
|
||||
when the facts file has a newer commit than the poster; it **never fails a push**, because a refresh
|
||||
needs the operator and another tool, and a gate nobody can clear is a gate people learn to route
|
||||
around.
|
||||
|
||||
+133
-2
@@ -1,4 +1,135 @@
|
||||
## Unreleased — part of v0.152.0: the kernel lane (R-836; `09` §3 decisions 164, 172; `11` §5.11) (2026-10-07)
|
||||
## v0.156.0 — after a power cut: the dead half-written backup is removed (R-1006); a backup task from before the boot is not „in flight" (R-1008) (2026-10-14)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `8db2278838dcb733d5201fba84b75a829b9adebdf899c89dd7de0939ec10d51a`, bundle
|
||||
`a21605217966ebc953d55a35157a9eb0243ca542c50dbd94fc9fa67257f27369` (tag `v0.156.0` = `a2f3aee`). Step bundle `0.156.0-step1` sha256
|
||||
`8be2bb8f3d54bff8e7140cf99714664df31adecf13aa721597323fb2b9d43db5` (the 0.155.0 bundle `795f2ab4…` with only `felhom-os-apply`
|
||||
replaced; `scripts/build-step-bundle.py`), published as package version `0.156.0-step1`. **Not vouched** (the operator's act).
|
||||
|
||||
**Delivery: the agent binary AND the config bundle, and the bundle ADDS a path** (`/usr/local/sbin/felhom-vzdump-partial-clean`)
|
||||
— so per box: signed `agent_update 0.156.0` → `agent_config_update 0.156.0-step1` (the step bundle) → `agent_config_update
|
||||
0.156.0`. Evidence: felhom.eu `documentation/audits/power-cut-fixes-2026-10-14/` (the spike C1–C4, the red-proofs).
|
||||
|
||||
- **R-1008 — one in-flight answer.** `staleLockController.BackupRunning` (the ONE implementation all four callers reach:
|
||||
the start-up stale-lock pass, the guest-power watchdog, the household power-off guard, the controller supervisor) now
|
||||
answers through `vzdumpInFlight`: a vzdump task for the vmid that started at or after the host's boot. Proxmox keeps a
|
||||
dead task in `tasks?source=active` for ~30–60 s after a power cut (measured, C3) — the agent starts inside that window
|
||||
and called it „genuinely in-flight". A task from before the boot is dead; logged once (vmid, UPID, start, boot). The
|
||||
start time comes from the list or, if absent, the UPID; an unreadable boot time or start time FAILS SAFE (running), as
|
||||
before. The boot time is `poweroff.ProcBtime` (no third /proc/stat parser).
|
||||
- **R-1006 — the half-written backup is removed.** New `internal/vzpartial`: on a `dir` backup target it finds exactly
|
||||
`vzdump-(lxc|qemu)-<vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp)` of THAT guest (never a finished archive, a log,
|
||||
another guest, a symlink, a wrong kind), and — only when no vzdump of the guest is in flight (R-1008) and the entry is
|
||||
dead (modified before this boot, or untouched for 120 s) — asks the root helper to remove it. Runs at agent start (after
|
||||
the stale-lock pass, every owned guest × every backup tier) and before every backup's space preflight
|
||||
(`BackupRunner.SetBeforeBackup`). Any doubt leaves the entry and logs why.
|
||||
- **The root helper `configs/felhom-vzdump-partial-clean`** (config bundle; sudoers `FELHOM_VZPARTIAL`, one exact line: a
|
||||
storage id, a vmid, one half-written name) re-checks everything as root: a `dir` storage from `/etc/pve/storage.cfg`; the
|
||||
name of that vmid; not a symlink; directly in `<path>/dump`; **no vzdump on the host at all** (a job for several guests
|
||||
carries no vmid in its task name); **no process holds the entry open** (C2: after a killed worker, `tar` and `zstd` kept
|
||||
writing into the `.tar.dat` for 4 min 40 s, none named „vzdump"); dead (before this boot, or quiet 120 s).
|
||||
- **Not changed, by evidence (C4):** the space preflight's sizing (R-1006 part 2) — on „No space left" vzdump removes its
|
||||
own partial on both its paths (rsync and tar); only a power cut leaves one.
|
||||
- **Tests:** `stalelock_inflight_test.go` (the helper, the production controller, three callers end to end, a structural
|
||||
pin that every `BackupRunning` call goes through `s.staleLock` and there is one implementation), `vzpartial_test.go`
|
||||
(removes; keeps in flight / unknown / non-dir / other names / recent / wrong kind / symlink / a helper refusal — each keep
|
||||
with its positive control), `r685_space_test.go` (the cleanup runs BEFORE the preflight), the helper's own 21 Python
|
||||
tests (`configs/test_felhom_vzdump_partial_clean.py`, run by the Go suite), 8 sudoers injection cases. **17 guards
|
||||
red-proofed** by mutation (`power-cut-fixes-2026-10-14/redproof.txt`).
|
||||
- **Fixed without a row:** `internal/localapi/guestbind.go` and `controllersupervisor_test.go` were not gofmt-clean on
|
||||
`main` (an import order and a struct alignment) — formatted.
|
||||
|
||||
## v0.155.0 — the household can turn the machine off: POST /host/poweroff (R-998); the escrow reply race fixed (2026-10-13)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `bb339e944eff4aa6d10ae09158e0943de2e5c3777f52bf993c796d57e46800c3`, bundle
|
||||
`795f2ab41661e993e3fc4ce65df4f11ce94280d94871ab2e5cf1993ae3fbff7d` (tag `v0.155.0`). **Not vouched** (the operator's act).
|
||||
**The bundle changes `felhom-agent.sudoers`** (no new path — no step bundle): delivery is a signed `agent_update` +
|
||||
`agent_config_update`. Design: felhom.eu `documentation/architecture/14-host-power-off.md`.
|
||||
|
||||
- **`POST /host/poweroff`** (local API, `withGuest`-scoped like `/guest/reboot`) — the controller's „A gép kikapcsolása"
|
||||
(controller v0.310.0). Refused (409, a stable `reason`) in the first 10 min after boot; a second time in one boot
|
||||
(`internal/poweroff`, a file keyed by `/proc/stat` btime); while ANY other guest or VM runs on the host (read from
|
||||
`/proc`: `lxc-start -n N`, `kvm -id N` — demo-hp's scratch guest and Tester 1 VM block it); while the heavy-operation
|
||||
gate is held (backup, restore test, OS step, sweep, trim), a backup job, a controller swap, a drive format, a NAS verify
|
||||
or an escrow ceremony runs, or Proxmox runs a vzdump of the guest. Every check fails closed.
|
||||
- On acceptance: the record first; the heavy gate TAKEN and kept; a host report carrying `deliberate_off.requested_at`
|
||||
accepted by the hub (bounded 60 s, `Loop.SetReportedHook`); then before each step the jobs and other guests are
|
||||
checked again — a job waits (≤ 15 min), another guest cancels at once, a job still running after the wait cancels
|
||||
(gate released, box stays on); `pct shutdown <vmid> --timeout 120`; `systemctl poweroff`.
|
||||
- **Sudoers:** `FELHOM_POWEROFF` = `/usr/bin/systemctl poweroff`; `pct ^shutdown [0-9]+ --timeout [0-9]+$` joins
|
||||
`FELHOM_GUESTHOOK`; nine injection cases (`--forceStop`, `poweroff --force`, `reboot`, `halt`, `kexec`…).
|
||||
**New `TestEveryAliasIsGranted`:** an alias missing from the grant line passed the coverage check before.
|
||||
- **Host report:** `deliberate_off` (omitempty; the cross-repo golden unchanged). Hub v0.149.0 holds its silence alarms
|
||||
for 14 days on it.
|
||||
- **Fixed without a row:** `go test -race` found a data race in the escrow ceremony start reply (it read `job.Phase`
|
||||
while the job goroutine wrote it); the reply fields are copied first. Red before, green 3/3 after.
|
||||
- Security review (three passes the same evening): other guests, a gate not held, fail-open unwired checks, jobs and
|
||||
guests starting after acceptance — each fixed and red-proofed (`hostpoweroff_test.go`, 11 tests).
|
||||
|
||||
## v0.154.0 — no OS leg after a household press; the after-boot kernel report carries the real ring (R-899); no „wrong code" when older packages were not checked (R-304); the last backup survives a restart in the host report; the Secure Boot meta-package leaves the Proxmox lane (2026-10-09)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `36a54ad20e896cd61d7b9944e2a521f4209c2b76940d8f1827af05b645463226`, bundle
|
||||
`93487989f50d4a3039a05cec0f91ec478d62489a414705e49403e9b1340d20e0` (tag `v0.154.0` = `db25b46`).
|
||||
|
||||
**Delivery: the agent binary AND the config bundle** — the wrapper `felhom-os-apply` changed (2026-10-09).
|
||||
|
||||
- **The hub's backup evidence survives a restart (2026-10-09, found in the kernel-night read-back).** The host report's
|
||||
`backups` list came only from memory, which is empty after an agent restart. On demo-felhom the night backup landed
|
||||
02:40 UTC and the kernel step restarted the host at 02:44 — no report fell in between, two nights running — so the hub
|
||||
alarmed „host tier: newest backup is 48h old" at 03:00 after a good backup. The report now adds R-894's saved newest
|
||||
success per tier and guest (`backup-success-state.json`, ONE instance shared with the local API) where memory holds no
|
||||
success at or after it. Only successes are saved, so a failure is never hidden. `hub.KnownBackupReporter`,
|
||||
`BackupSuccessState.KnownBackupSuccesses`; tests `TestCollectBackups_SavedSuccessSurvivesARestart` (red-proved: the
|
||||
restart case reported nothing), `TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen`, and
|
||||
`TestR894_LastKnownBackupsIsWiredIntoTheDaemon` now also asserts the collector gets the same instance (red-proved).
|
||||
- **`proxmox-secure-boot-support` is a boot-chain package (2026-10-09).** On demo-hp (Secure Boot on) its upgrade pulled
|
||||
`shim-signed-common`; in the ring-0 Proxmox plan that refused the whole step (R6) and skipped the night's kernel step,
|
||||
after the household had been told „tonight". It joins `HOST_SLOW_RE` with shim and GRUB (they stay out of every lane,
|
||||
as before). `test_ring0_pending_pve_leaves_the_secure_boot_meta_out` (red-proved).
|
||||
|
||||
- **R-899 (operator ruling 2026-10-08, option A):** `POST /backup?trigger=manual` (a „Mentés most" press, sent by the
|
||||
controller from its next release) runs no OS leg after it: the leg belongs to the night, after the night's own copy.
|
||||
Before, a press ran the leg at once (in the day; on 2026-10-07 08:51 demo-hp skipped it only by the 20-hour rule). A
|
||||
request without the parameter (an older controller, the scheduled path) behaves as before.
|
||||
`internal/localapi/server.go`; `TestAfterPrimaryBackup` gained two sub-cases (red-proved: ignoring `trigger`, the leg
|
||||
ran once after a press).
|
||||
- **R-304 — „wrong code" only when every earlier package was tried.** `POST /escrow/recover-offsite-password` answered
|
||||
400 („the recovery code did not open the sealed bundle") whenever the current package and the TRIED earlier packages
|
||||
refused the code — even when the hub withheld earlier packages (rows with no key material, rows over its serve cap), a
|
||||
served package was malformed, the 6-attempt cap stopped the loop, or the retained list could not be read at all. Now
|
||||
those cases answer **424** with `older_unchecked` (the count, -1 = unknown) and a sentence that does not call the code
|
||||
wrong. `escrow.ErrRetainedUnchecked` / `RetainedUncheckedError`; the retained fetcher's second value is now every
|
||||
withheld package (unopenable + truncated + malformed). Tests `TestR304_*` (real age crypto; red-proved: without the
|
||||
check, four cases returned the wrong-code error) and a 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus`.
|
||||
An older controller maps the unknown 424 to its neutral „we do not know why" sentence.
|
||||
- **The „ring 1" label after a boot:** in the first second after a reboot the agent has not fetched the hub's block, and
|
||||
the after-boot kernel reports (`judging`, `good`, `revert`, `fell_back`…) said ring 1 on a ring-0 box (demo-felhom,
|
||||
2026-10-08 night). Now they read the fetched block, else the block the daemon saved on disk before the reboot (R-866),
|
||||
else ring 1 as before. A label only — the hub's approval reads its own ring list. `kernelReportRing`,
|
||||
`TestKernelReportRing_BeforeFirstFetch` (red-proved: ring 1, want 0).
|
||||
|
||||
## v0.153.0 — ring 0 stages exactly the told kernel (R-898; `09` §3 decision 176) (2026-10-07)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `b204ebe6d65944f43b70dcfa4ae0dfb90da38c92e2ba6a38a1826e488eeb6630`, bundle
|
||||
`6db216275eb0dd39188d93481a2045998a69e8cb7838ad908a58d65d9a6a9b57` (tag `v0.153.0` = `2d1e5d0`). Delivered (binary only — the
|
||||
bundle files are unchanged from 0.152.0) to demo-hp, demo-felhom, Tester 1 on 2026-10-07 19:03.
|
||||
|
||||
**Delivery: the agent binary only** — no root file changed (the wrapper is unchanged; its tests gained two cases).
|
||||
|
||||
- `internal/osupdate/kernel.go`: ring 0's night kernel step stages EXACTLY the kernel the household was told about
|
||||
(select `listed`, `KernelSet(kver)` = the series meta-package and the signed image at the kernel's own version) instead
|
||||
of "whatever is pending tonight". Seen 2026-10-07: demo-felhom was told about 7.0.14-20 while its sources offered
|
||||
7.0.14-22 by night — the old code staged `pending-kernel` and the wrapper refused it (R23), losing the night. A told
|
||||
version that is no longer installable is refused by the wrapper before any change (R7) and the hub tells the household
|
||||
again for the newer kernel (hub v0.143.1). Tests `TestKernel_Ring0ToldNightStagesThenReboots` (red-proved against the
|
||||
old select), `TestKernelSet`; wrapper `test_ring0_listed_installs_the_told_kernel_not_the_newest`,
|
||||
`test_ring0_told_kernel_gone_is_refused_before_any_change`.
|
||||
|
||||
## v0.152.0 — the kernel lane (R-836; `09` §3 decisions 164, 172; `11` §5.11) (2026-10-07)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `95ff42208e36ba49b6e2b09a97e81a6fa11562ecc8042f1ed18d378b8b1f88b1`,
|
||||
config bundle sha256 `f0c2cec374b711b3c131c955012b33fd0ad495337d049b7a743c3eef9e85c20b` (tag `v0.152.0` = `d03ab7f`).
|
||||
Step bundle `0.152.0-step1` sha256 `0b71d32b054cf3b7ade0234ffcbb0df159901f542cde540adaee411db466f48e` (the 0.151.0 bundle
|
||||
with only `felhom-os-apply` replaced; `scripts/build-step-bundle.py`), published as package version `0.152.0-step1`.
|
||||
|
||||
**Delivery: agent binary, then the STEP bundle `0.152.0-step1`, then the bundle `0.152.0`** — the bundle ADDS two paths
|
||||
(the GRUB generators), and an installed `felhom-os-apply` refuses a path its own table lacks (R16, R-880).
|
||||
@@ -34,7 +165,7 @@ candidate 2 (`audits/kernel-spike-2026-10-07/`), with option C on the one-shot e
|
||||
signed `os_kernel_step` staged (`KernelStepExecutor`: stage only, under the heavy-op gate). The hub hears `staged`
|
||||
BEFORE the reboot. At every start `KernelAfterBoot`: on the new kernel it JUDGES the boot — `KernelVerdict` = the
|
||||
host health rule (`11` §8.2) AND the box reached the hub (the `judging` report itself) — for 20 minutes (measured:
|
||||
everything healthy 68 s after the reboot on demo-felhom, 272 s on demo-hp; under the hub's 30-minute `host_stale`).
|
||||
everything healthy 68 s after the reboot on demo-felhom, 272 s on demo-hp; under the hub's 45-minute `host_stale` (`alerting.stale_threshold`)).
|
||||
Healthy → `kernel-good`, outcome `applied`; not healthy → outcome `health_failed`, then ONE `kernel-revert`.
|
||||
Tests: `TestKernel*` (13); red-proofs in the same file.
|
||||
- `internal/hub`: `WireOSUpdate.Kernel` {kver, tonight, notified_at}. `internal/reconcile`: `os_kernel_step` is
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-08 (day) — UNRELEASED on main, ships with tomorrow's release (decision 178).** R-899: `POST /backup?trigger=manual`
|
||||
> runs no OS leg (`localapi/server.go`); after-boot kernel reports carry the saved block's ring (`kernelReportRing`). R-304:
|
||||
> `escrow.ErrRetainedUnchecked` → HTTP 424 `older_unchecked` when not every earlier package was tried (withheld, caps,
|
||||
> malformed, unreadable list). Binary-only delivery; no wrapper or root file changed.
|
||||
|
||||
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
|
||||
|
||||
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
|
||||
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
|
||||
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
|
||||
|
||||
@@ -1,9 +1,38 @@
|
||||
# REPORT — v0.151.0 released and delivered (2026-10-07 day)
|
||||
# REPORT — v0.156.1: after a power cut the box clears up (R-1006, R-1008) (2026-10-14)
|
||||
|
||||
On the operator's word (`09` §3 decisions 163, 165, 168, 169). sha `0464354f…`, bundle `bacd1d17…`, tag `v0.151.0` = `dd7cdc0`.
|
||||
Delivered binary first, then the bundle (its sudoers drops the `tee` grant 0.150.0 used), to demo-hp, demo-felhom and
|
||||
Tester 1 — probe 68/68 on each. **Vouch refused by the hub** (`golden_behind_fleet`): new installs keep 0.150.0 until the
|
||||
weekly golden. Carries: the Proxmox package lane + `/etc/pve` write gate (R-812 A — proven on demo-felhom: 65 packages,
|
||||
70 s, healthy), the `controller-image` root verb (R-861 a — on demo-hp an `alpine` ref is refused; a managed swap not yet
|
||||
seen), anchored felhom-op lines (B2), the other-key archive ledger (R-366), the `-directive` flag removed (R-105). Evidence
|
||||
`felhom.eu/documentation/audits/day-2026-10-07/`. Shared rule file: decision 162 line added.
|
||||
Evidence: felhom.eu `documentation/audits/power-cut-fixes-2026-10-14/` — the spike (C1 the night's leftover, C2 a killed
|
||||
worker, C3 a real power cut, C4 a full disk), the 17 red-proofs.
|
||||
|
||||
## What the spike showed
|
||||
|
||||
- A power cut mid-backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` + `.tmp/` (root, in `/var/lib/vz/dump`); Proxmox never
|
||||
lists it as a backup; nothing removes it; every later backup is skipped „not enough space".
|
||||
- The dead vzdump task stays in Proxmox's active list ~30–60 s after the boot — long enough for the agent's start-up pass
|
||||
to call it „genuinely in flight" (R-1008). Short, so R-1008's harm is small.
|
||||
- A killed worker leaves `tar`/`zstd` writing into the `.tar.dat` for minutes, none named „vzdump" — the reason the root
|
||||
helper checks open files, not only process names.
|
||||
- On „No space left" vzdump removes its own partial (both paths): the preflight's sizing stays. But a full host root breaks
|
||||
LVM's own metadata writes (a half-removed snapshot; the janitor cannot clear it — new row).
|
||||
|
||||
## 0.156.0 is superseded — do not vouch it
|
||||
|
||||
A background security review of the pushed 0.156.0 flagged „fail-open state drift": its in-flight rule compared a task's
|
||||
wall-clock start with the kernel's `btime`, which moves when the clock is corrected after the boot (an NTP step after a
|
||||
power cut) — a live backup could then look „from before the boot" and its lock be cleared. 0.156.1 uses Proxmox's own
|
||||
liveness rule instead (the UPID's pid + pstart against `/proc/<pid>/stat` start tick and state; no clock), and fails safe
|
||||
when this agent cannot see root's processes (PID 1 not visible).
|
||||
|
||||
## What changed
|
||||
|
||||
See `CHANGELOG.md` v0.156.0: `vzdumpInFlight` behind the one `BackupRunning`; `internal/vzpartial` at start-up and before
|
||||
each backup; the root helper `felhom-vzdump-partial-clean` + one sudoers line + the bundle entry.
|
||||
|
||||
## Gates
|
||||
|
||||
`go build ./... && go vet` clean; `go test ./... -count=1` rc 0; `agent_gates.py` all OK; `visudo -cf` parsed OK; the
|
||||
config bundle tests (55) and the helper's tests (21) pass. 17 mutations, 17 red.
|
||||
|
||||
## Next
|
||||
|
||||
Released: tag `v0.156.0`, binary sha256 `8db22788…`, bundle `a2160521…`, step bundle `0.156.0-step1` `8be2bb8f…`. Then the operator's vouch + floor, then the live proof on demo-hp (a cut mid-backup) and
|
||||
delivery to Tester 1 and the N100.
|
||||
|
||||
@@ -50,6 +50,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbs"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poke"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
@@ -58,6 +59,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/vzpartial"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/wgtunnel"
|
||||
)
|
||||
|
||||
@@ -833,7 +835,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
|
||||
// already carries the gated view — v0.86.0.)
|
||||
collector.SetCapabilityProber(probeAll)
|
||||
// R-998 (doc 14): the household power-off record of THIS boot rides every report until the box is down.
|
||||
powerOffState := poweroff.New(filepath.Join(cfg.OOB.WithDefaults().StateDir, "poweroff.json"), poweroff.ProcBtime)
|
||||
collector.SetDeliberateOffSource(func() *hub.DeliberateOff {
|
||||
if r := powerOffState.Current(); r != nil {
|
||||
return &hub.DeliberateOff{RequestedAt: r.RequestedAt.UTC().Format(time.RFC3339)}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
loop := hub.NewLoop(collector, client, time.Duration(hcfg.PollSeconds)*time.Second, logger)
|
||||
// R-998: the power-off path waits until the hub ACCEPTED a report carrying the record.
|
||||
powerOffAck := make(chan struct{}, 1)
|
||||
loop.SetReportedHook(func(r *hub.HostReport) {
|
||||
if r != nil && r.DeliberateOff != nil {
|
||||
select {
|
||||
case powerOffAck <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
})
|
||||
interval := time.Duration(hcfg.PollSeconds) * time.Second
|
||||
// Heartbeat log-pull (v0.83.0): when the control envelope requests it, the NEXT
|
||||
// heartbeat carries the debug ring's tail (newest-kept, byte-capped in the loop).
|
||||
@@ -881,7 +901,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// judge it (the host health rule + the hub reached) for KernelJudgeWait, then make it the default or revert ONCE.
|
||||
// The wait: measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`) — every container healthy 68 s after the
|
||||
// reboot on demo-felhom and 272 s on demo-hp (the hub reached at 63 s / 189 s); 20 minutes leaves room for a slow
|
||||
// network and stays under the hub's 30-minute host_stale. On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
|
||||
// network and stays under the hub's 45-minute host_stale (its
|
||||
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
|
||||
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
|
||||
|
||||
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
|
||||
@@ -1179,6 +1200,34 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localSrv != nil {
|
||||
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
|
||||
pwMode := proxmox.RunnerMode(cfg.Privileged.Mode)
|
||||
if pwMode == "" {
|
||||
pwMode = proxmox.RunnerSudo
|
||||
}
|
||||
localSrv.SetHostPowerOff(localapi.HostPowerOffConfig{
|
||||
State: powerOffState,
|
||||
Power: localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: pwMode, SudoPath: cfg.Privileged.SudoPath}, logger),
|
||||
ReportNow: func(ctx context.Context) bool {
|
||||
select {
|
||||
case storageTrigger <- struct{}{}:
|
||||
default: // a report is already queued; it runs after the record was written
|
||||
}
|
||||
select {
|
||||
case <-powerOffAck:
|
||||
return true
|
||||
case <-ctx.Done():
|
||||
return false
|
||||
}
|
||||
},
|
||||
Uptime: func() (time.Duration, bool) {
|
||||
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
|
||||
if err != nil || b <= 0 {
|
||||
return 0, false
|
||||
}
|
||||
return time.Since(time.Unix(b, 0)), true
|
||||
},
|
||||
})
|
||||
localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) {
|
||||
// Let the controller finish bringing its apps back after the backup, then run (still under the gate).
|
||||
select {
|
||||
@@ -1484,6 +1533,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// the dangling snapshot → start iff onboot). Runs BEFORE the backup loop starts, so a present
|
||||
// backup lock is stale by definition (guarded by a no-vzdump-running check; fail-safe otherwise).
|
||||
localSrv.RecoverStaleLockedGuests(ctx)
|
||||
// R-1006: then remove a dead half-written backup a power cut left (the lock recovery first: same order as a human).
|
||||
localSrv.SweepVzdumpPartials(ctx)
|
||||
// F-REBOOT: the startup recovery above only covers a guest left LOCKED by an interrupted
|
||||
// backup. A guest that simply ends up stopped-and-unlocked (a `pct reboot` whose shutdown
|
||||
// half completed and whose start half never fired — Campaign 8 fault 11, 9m47s of total
|
||||
@@ -1844,6 +1895,8 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
}
|
||||
apiTiers := make([]localapi.BackupTier, 0, len(backupTiers))
|
||||
var runner *backup.BackupRunner
|
||||
var tierRunners []*backup.BackupRunner
|
||||
var tierTargets []string
|
||||
for _, t := range backupTiers {
|
||||
prune := ""
|
||||
if t.KeepLast > 0 {
|
||||
@@ -1856,6 +1909,8 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
if t.Primary {
|
||||
runner = r
|
||||
}
|
||||
tierRunners = append(tierRunners, r)
|
||||
tierTargets = append(tierTargets, t.TargetID)
|
||||
apiTiers = append(apiTiers, localapi.BackupTier{
|
||||
TargetID: t.TargetID,
|
||||
Cadence: t.Cadence,
|
||||
@@ -1903,11 +1958,15 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
return nil, 0, ferr
|
||||
}
|
||||
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
|
||||
// R-304: every package the hub holds and the code will NOT be tried against — no key material,
|
||||
// over the hub's cap, or malformed here. A refusal may call the code wrong only when this is 0.
|
||||
withheld := resp.UnopenableCount + resp.TruncatedCount
|
||||
for _, p := range resp.Packages {
|
||||
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
|
||||
if derr != nil || len(blob) == 0 {
|
||||
// One malformed package must not sink the rest — the customer's code may open a
|
||||
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
|
||||
withheld++
|
||||
continue
|
||||
}
|
||||
out = append(out, escrow.RetainedBlob{
|
||||
@@ -1917,9 +1976,13 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
Index: p.Index,
|
||||
})
|
||||
}
|
||||
return out, resp.UnopenableCount, nil
|
||||
return out, withheld, nil
|
||||
},
|
||||
}
|
||||
// R-894: ONE instance — the local API writes it, the host report reads it (2026-10-09: a restart right
|
||||
// after the night backup erased the hub's evidence of it).
|
||||
lastKnownBackups := backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json"))
|
||||
collector.SetKnownBackupReporter(lastKnownBackups)
|
||||
srv, err := localapi.NewServer(localapi.Options{
|
||||
EscrowRecovery: escrowRecoverer,
|
||||
ListenAddr: cfg.LocalAPI.ListenAddr,
|
||||
@@ -1932,7 +1995,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
Store: store,
|
||||
// R-894: the newest success per tier on disk — the due-check's fallback when the storage cannot be
|
||||
// read right after a restart. Same state dir as restore-test-state.json.
|
||||
LastKnownBackups: backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json")),
|
||||
LastKnownBackups: lastKnownBackups,
|
||||
Storage: observer,
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
||||
@@ -1976,6 +2039,26 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
logger.Warn("daemon: local-api disabled (server build)", "err", err)
|
||||
return nil
|
||||
}
|
||||
// R-1006: a dead half-written vzdump (a power cut mid-backup) is removed at start-up and before every backup of the
|
||||
// guest on that target, through the root helper; the in-flight answer is R-1008's (a task from before the boot is dead).
|
||||
cleaner := &vzpartial.Cleaner{
|
||||
API: px,
|
||||
Runner: &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath},
|
||||
InFlight: srv.VzdumpInFlight,
|
||||
Boot: func() (int64, bool) {
|
||||
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
|
||||
return b, err == nil && b > 0
|
||||
},
|
||||
Logger: logger,
|
||||
}
|
||||
for _, r := range tierRunners {
|
||||
r.SetBeforeBackup(func(ctx context.Context, storage string, vmid int) { cleaner.Clean(ctx, storage, vmid) })
|
||||
}
|
||||
srv.SetPartialSweep(func(ctx context.Context, vmid int) {
|
||||
for _, t := range tierTargets {
|
||||
cleaner.Clean(ctx, t, vmid)
|
||||
}
|
||||
})
|
||||
return srv
|
||||
}
|
||||
|
||||
|
||||
@@ -12,12 +12,18 @@ import (
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): delete the `LastKnownBackups:` line from buildLocalAPIServer → this
|
||||
// fails with "localapi.Options in buildLocalAPIServer has no LastKnownBackups field". Restored.
|
||||
//
|
||||
// 2026-10-09: the SAME instance also feeds the host report (collector.SetKnownBackupReporter), so a
|
||||
// restart right after a backup no longer erases the hub's evidence of it. The field may name a local
|
||||
// variable; the variable must be built by backup.NewBackupSuccessState and be the one handed to the
|
||||
// collector. RED-PROOF (observed): drop the SetKnownBackupReporter call → "not handed to the collector".
|
||||
func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
|
||||
_, f := parseMain(t)
|
||||
if !callsWithin(f, "main")["runDaemon"] || !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
|
||||
t.Fatal("main → runDaemon → buildLocalAPIServer is broken — the path this test asserts is not the live one")
|
||||
}
|
||||
var field, built bool
|
||||
var fieldVar, handed string
|
||||
for _, d := range f.Decls {
|
||||
fd, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
||||
@@ -45,6 +51,28 @@ func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
|
||||
if callsIn(kv.Value)["backup.NewBackupSuccessState"] {
|
||||
built = true
|
||||
}
|
||||
if id, ok := kv.Value.(*ast.Ident); ok {
|
||||
fieldVar = id.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
// a local variable: built by NewBackupSuccessState, and handed to the collector
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
switch x := n.(type) {
|
||||
case *ast.AssignStmt:
|
||||
for i, l := range x.Lhs {
|
||||
if id, ok := l.(*ast.Ident); ok && fieldVar != "" && id.Name == fieldVar && i < len(x.Rhs) &&
|
||||
callsIn(x.Rhs[i])["backup.NewBackupSuccessState"] {
|
||||
built = true
|
||||
}
|
||||
}
|
||||
case *ast.CallExpr:
|
||||
if fn, ok := x.Fun.(*ast.SelectorExpr); ok && fn.Sel.Name == "SetKnownBackupReporter" && len(x.Args) == 1 {
|
||||
if id, ok := x.Args[0].(*ast.Ident); ok {
|
||||
handed = id.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
@@ -56,6 +84,9 @@ func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
|
||||
if !built {
|
||||
t.Fatal("LastKnownBackups is not built by backup.NewBackupSuccessState")
|
||||
}
|
||||
if handed == "" || handed != fieldVar {
|
||||
t.Fatalf("the saved backups (%q) are not handed to the collector (SetKnownBackupReporter got %q)", fieldVar, handed)
|
||||
}
|
||||
}
|
||||
|
||||
func callsIn(n ast.Node) map[string]bool {
|
||||
|
||||
@@ -80,7 +80,21 @@ Cmnd_Alias FELHOM_DNSMASQ = \
|
||||
Cmnd_Alias FELHOM_GUESTHOOK = \
|
||||
/usr/sbin/pct ^set [0-9]+ --hookscript local\:snippets/felhom-guest-hook\.sh$, \
|
||||
/usr/sbin/pct ^set [0-9]+ --delete mp[0-9]+$, \
|
||||
/usr/sbin/pct ^reboot [0-9]+$
|
||||
/usr/sbin/pct ^reboot [0-9]+$, \
|
||||
/usr/sbin/pct ^shutdown [0-9]+ --timeout [0-9]+$
|
||||
|
||||
# Host power-off (R-998, doc 14): the household's „A gép kikapcsolása" — guest shutdown above, then this one fixed
|
||||
# vector. Guarded agent-side (10 min after boot, one per boot, never during a backup/restore/update/drive job).
|
||||
Cmnd_Alias FELHOM_POWEROFF = \
|
||||
/usr/bin/systemctl poweroff
|
||||
|
||||
# Dead half-written vzdump removal (R-1006, power-cut-fixes-2026-10-14): a power cut mid-backup leaves
|
||||
# vzdump-lxc-<vmid>-<date>.tar.dat + .tmp/ on the dir storage; nothing removed them and every later backup was skipped
|
||||
# „not enough space". ONE root helper (configs/felhom-vzdump-partial-clean, from the config bundle) re-checks everything
|
||||
# as root — a dir storage from storage.cfg, the exact name of THAT vmid, not in use, dead — and refuses otherwise. The
|
||||
# arguments here are its exact shape (storage id, vmid, one half-written name); the vmid-in-name match is the helper's.
|
||||
Cmnd_Alias FELHOM_VZPARTIAL = \
|
||||
/usr/local/sbin/felhom-vzdump-partial-clean ^[A-Za-z][A-Za-z0-9_-]{0\,63} [1-9][0-9]{2\,8} vzdump-(lxc|qemu)-[1-9][0-9]{2\,8}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$
|
||||
|
||||
# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
|
||||
# /mnt/felhom-drives (self-bind + make-shared) and binds/unbinds each drive's felhom-data namespace UNDERNEATH it so the
|
||||
@@ -314,4 +328,4 @@ Cmnd_Alias FELHOM_GUESTNET = \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$
|
||||
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY, FELHOM_POWEROFF, FELHOM_VZPARTIAL
|
||||
|
||||
@@ -93,8 +93,13 @@ OOMCHECK_TIMEOUTS = {"image": 10, "clock": 5, "run": 30, "inspect": 10, "events"
|
||||
OOMCHECK_SETTLE = 2
|
||||
# Kernel, boot and firmware packages are the SLOW lane on the host whatever their origin (`11` C3, §5.2): a host
|
||||
# reboot is needed for them to take effect, and a bad one can stop the box from booting.
|
||||
# proxmox-secure-boot-support is the Secure Boot meta-package: its only job is to pull shim-signed and the signed GRUB,
|
||||
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
|
||||
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
|
||||
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
|
||||
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
|
||||
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)")
|
||||
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
|
||||
r"proxmox-secure-boot-support)")
|
||||
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
|
||||
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
|
||||
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
|
||||
@@ -184,6 +189,8 @@ BUNDLE_FILES = [
|
||||
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
|
||||
("/usr/local/sbin/felhom-pbs-apply", "felhom-pbs-apply", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-backup-target-apply", "felhom-backup-target-apply", 0o755, "bash", "replace"),
|
||||
# R-1006 (agent v0.156.0): the only delete of backup files — a dead half-written vzdump, re-checked as root.
|
||||
("/usr/local/sbin/felhom-vzdump-partial-clean", "felhom-vzdump-partial-clean", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-os-apply", "felhom-os-apply", 0o755, "python", "replace"),
|
||||
("/usr/local/sbin/felhom-crash-guard", "felhom-crash-guard", 0o755, "python", "replace"),
|
||||
# R-861 (agent v0.146.0): the content checker for every agent-staged file a root program reads, and the two FIXED
|
||||
|
||||
Executable
+99
@@ -0,0 +1,99 @@
|
||||
#!/bin/bash
|
||||
# felhom-vzdump-partial-clean — remove ONE dead, half-written vzdump backup (R-1006). The only delete the agent's
|
||||
# sudoers allows for backup files. Installed as /usr/local/sbin/felhom-vzdump-partial-clean (config bundle).
|
||||
#
|
||||
# felhom-vzdump-partial-clean <storage> <vmid> <name>
|
||||
#
|
||||
# A power cut during the night backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` (+ its `.tmp/` directory) on the
|
||||
# storage; nothing removes it, every later backup is then skipped „not enough space", and a full host root breaks
|
||||
# LVM's own metadata writes (power-cut-fixes-2026-10-14 C4). This helper removes such a leftover — and NOTHING else.
|
||||
# It trusts nothing the agent says; every rule is checked here, as root, and any doubt REFUSES (exit 1):
|
||||
# 1. the arguments: a storage id, a vmid, and a name that is exactly
|
||||
# vzdump-(lxc|qemu)-<THAT vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp) — never a finished archive
|
||||
# (.tar.zst, .vma.zst, …), never a .log/.notes, never another vmid;
|
||||
# 2. the storage: its section in /etc/pve/storage.cfg is `dir: <storage>` with a `path`; the target is
|
||||
# <path>/dump/<name>, not a symlink, and its real directory IS <path>/dump;
|
||||
# 3. not in use: no vzdump runs on the host at all (a task `…:vzdump:…` or a `vzdump` command — a job for several
|
||||
# guests carries no vmid in its task name), and no
|
||||
# process holds the target (or, for the .tmp directory, anything under it) open or as its working directory
|
||||
# (C2: after a killed worker, tar and zstd kept writing into the .tar.dat for 4 min 40 s, none named „vzdump");
|
||||
# 4. dead: last modified before this boot (`btime` in /proc/stat), or untouched for at least QUIET_S seconds.
|
||||
# Prints one line on success: „removed <name> <bytes> B (<why>)". Pinned by configs/test_felhom_vzdump_partial_clean.py.
|
||||
set -u
|
||||
STORAGE_CFG=/etc/pve/storage.cfg
|
||||
PROC=/proc
|
||||
QUIET_S=120
|
||||
|
||||
die() { echo "felhom-vzdump-partial-clean: REFUSED: $*" >&2; exit 1; }
|
||||
|
||||
storage_path() { # $1 storage id → prints the dir storage's path, or nothing
|
||||
awk -v want="$1" '
|
||||
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (id == want); next }
|
||||
in_s && /^[ \t]+path[ \t]/ { if (type == "dir") { sub(/^[ \t]+path[ \t]+/, ""); sub(/[ \t]+$/, ""); print; exit } }
|
||||
' "$STORAGE_CFG"
|
||||
}
|
||||
|
||||
storage_type() {
|
||||
awk -v want="$1" '/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); id = h[2]; sub(/[ \t]+$/, "", id); if (id == want) { print h[1]; exit } }' "$STORAGE_CFG"
|
||||
}
|
||||
|
||||
boot_time() { awk '$1 == "btime" { print $2 }' "$PROC/stat"; }
|
||||
|
||||
newest_mtime() { # a file's mtime, or the newest mtime of anything in a directory (the dir itself included)
|
||||
if [[ -d "$1" ]]; then find "$1" -xdev -printf '%T@\n' 2>/dev/null | sort -n | tail -1 | cut -d. -f1
|
||||
else stat -c %Y "$1"; fi
|
||||
}
|
||||
|
||||
busy_reason() { # $1 vmid, $2 target → prints why it is in use, or nothing
|
||||
local vmid=$1 target=$2 p link cmd
|
||||
for p in "$PROC"/[0-9]*; do
|
||||
[[ -r "$p/cmdline" ]] || continue
|
||||
cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)
|
||||
# ANY vzdump on the host refuses: a job for several guests runs as `task UPID:…:vzdump::…` (no vmid in it), and
|
||||
# its tar may be between this guest's files. Waiting costs one backup run; deleting a live one costs the backup.
|
||||
if [[ "$cmd" == *":vzdump:"* ]] || [[ "$cmd" =~ (^|/)vzdump([[:space:]]|$) ]]; then
|
||||
echo "a vzdump runs on this host (pid ${p##*/})"; return
|
||||
fi
|
||||
for link in "$p/cwd" "$p"/fd/*; do
|
||||
local to
|
||||
to=$(readlink "$link" 2>/dev/null) || continue
|
||||
if [[ "$to" == "$target" || "$to" == "$target/"* ]]; then
|
||||
echo "pid ${p##*/} has it open"; return
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
local storage=${1:-} vmid=${2:-} name=${3:-}
|
||||
[[ $# -eq 3 ]] || die "usage: felhom-vzdump-partial-clean <storage> <vmid> <name>"
|
||||
[[ "$storage" =~ ^[A-Za-z][A-Za-z0-9_-]{0,63}$ ]] || die "bad storage id"
|
||||
[[ "$vmid" =~ ^[1-9][0-9]{2,8}$ ]] || die "bad vmid"
|
||||
[[ "$name" =~ ^vzdump-(lxc|qemu)-${vmid}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$ ]] \
|
||||
|| die "name is not a half-written vzdump of guest $vmid: $name"
|
||||
[[ "$(storage_type "$storage")" == "dir" ]] || die "storage $storage is not a dir storage"
|
||||
local path; path=$(storage_path "$storage")
|
||||
[[ "$path" == /* ]] || die "storage $storage has no path"
|
||||
local dump="$path/dump" target="$path/dump/$name"
|
||||
[[ -e "$target" ]] || die "no such entry: $target"
|
||||
[[ ! -L "$target" ]] || die "a symlink: $target"
|
||||
[[ "$(realpath -e "$dump")" == "$(realpath -e "$(dirname "$target")")" ]] || die "not directly in $dump"
|
||||
if [[ "$name" == *.tmp ]]; then [[ -d "$target" ]] || die ".tmp is not a directory"
|
||||
else [[ -f "$target" ]] || die "not a regular file"; fi
|
||||
local busy; busy=$(busy_reason "$vmid" "$target")
|
||||
[[ -z "$busy" ]] || die "in use: $busy"
|
||||
local bt mt now why
|
||||
bt=$(boot_time); mt=$(newest_mtime "$target"); now=$(date +%s)
|
||||
[[ "$bt" =~ ^[0-9]+$ && "$mt" =~ ^[0-9]+$ ]] || die "cannot read the boot time or the modification time"
|
||||
if (( mt < bt )); then why="written before this boot"
|
||||
elif (( now - mt >= QUIET_S )); then why="untouched for $(( now - mt )) s, no vzdump for $vmid, nothing holds it"
|
||||
else die "modified $(( now - mt )) s ago — may still be written"; fi
|
||||
local size; size=$(du -sb "$target" 2>/dev/null | cut -f1)
|
||||
if [[ -d "$target" ]]; then rm -rf --one-file-system -- "$target"; else rm -f -- "$target"; fi
|
||||
[[ ! -e "$target" ]] || die "remove failed: $target"
|
||||
echo "removed $name ${size:-?} B ($why)"
|
||||
}
|
||||
|
||||
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
@@ -1469,6 +1469,23 @@ class PVELane(unittest.TestCase):
|
||||
"pending-pve: installed, Proxmox-origin, never kernel/boot/firmware, never Debian or other origins")
|
||||
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3")
|
||||
|
||||
# 2026-10-09, demo-hp (Secure Boot on): proxmox-secure-boot-support's upgrade pulls shim-signed-common; in the pve
|
||||
# plan that refused the whole step (R6) and skipped the night's kernel step. It is a boot-chain package: left out.
|
||||
# RED-PROOF: drop proxmox-secure-boot-support from HOST_SLOW_RE -> it is in the plan -> this test fails.
|
||||
def test_ring0_pending_pve_leaves_the_secure_boot_meta_out(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["select"], f.plan["packages"] = "pending-pve", []
|
||||
f.installed.update({"proxmox-secure-boot-support": "9.0.1"})
|
||||
f.pending_sim = [
|
||||
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst proxmox-secure-boot-support [9.0.1] (9.0.2 Proxmox Debian Repository:stable [all])",
|
||||
"Inst shim-signed-common [1.48+pmx1+16.1-1+pmx1] (1.51+pmx1+16.1-2+pmx1 Proxmox Debian Repository:stable [all])",
|
||||
]
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
|
||||
|
||||
def test_select_pending_pve_needs_the_pve_layer(self):
|
||||
f = Fake()
|
||||
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
|
||||
@@ -1838,6 +1855,33 @@ class KernelLane(unittest.TestCase):
|
||||
m.origins = {"proxmox-kernel-7.0": DEB}
|
||||
self.refused(m, "R2")
|
||||
|
||||
# R-898: ring 0 stages EXACTLY the told kernel (select listed, the set derived from it) — even when the sources
|
||||
# offer a newer one by night; a told version that can no longer be installed is refused BEFORE any change (R7).
|
||||
def test_ring0_listed_installs_the_told_kernel_not_the_newest(self):
|
||||
f = kfake()
|
||||
f.live["proxmox-kernel-7.0"] = {"7.0.14-20", "7.0.14-22", "7.0.2-6"}
|
||||
f.live["proxmox-kernel-7.0.14-20-pve-signed"] = {"7.0.14-20"}
|
||||
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
|
||||
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
|
||||
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["authority"], "ring0")
|
||||
self.assertEqual(f.env, {"felhom_next": "7.0.14-20-pve"})
|
||||
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.14-20")
|
||||
self.assertNotIn("7.0.14-22-pve", f.boot)
|
||||
|
||||
def test_ring0_told_kernel_gone_is_refused_before_any_change(self):
|
||||
f = kfake()
|
||||
f.live["proxmox-kernel-7.0"] = {"7.0.14-22"} # 7.0.14-20 is no longer in the archive
|
||||
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
|
||||
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
|
||||
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
|
||||
rep = self.refused(f, "R7")
|
||||
self.assertIsNone(f.env)
|
||||
self.assertNotIn(osapply.KERNEL_DEFAULT_CFG, f.tree, "refused before the default was even pinned")
|
||||
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.2-6")
|
||||
|
||||
def test_nothing_pending_changes_nothing(self):
|
||||
f = kfake(kernel_pending=[])
|
||||
rc, rep = run(f)
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for configs/felhom-vzdump-partial-clean (R-1006): it removes a DEAD half-written vzdump backup and refuses
|
||||
everything else. Each refusal has a positive control: the same setup without the one condition is removed — so a
|
||||
test cannot pass because the helper refuses everything. A fake storage.cfg and a fake /proc; nothing real is touched.
|
||||
Run: python3 configs/test_felhom_vzdump_partial_clean.py (also run by internal/backup's Go test)
|
||||
"""
|
||||
import os, pathlib, subprocess, tempfile, time, unittest
|
||||
|
||||
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-vzdump-partial-clean"
|
||||
BOOT = 1_791_967_481
|
||||
NAME = "vzdump-lxc-9201-2026_10_14-10_42_25"
|
||||
|
||||
|
||||
class Box:
|
||||
"""A temp tree: a dir storage `local` (path vz), an lvmthin `local-lvm`, a dir storage `other`, and a fake /proc."""
|
||||
|
||||
def __init__(self):
|
||||
self.t = tempfile.TemporaryDirectory()
|
||||
r = pathlib.Path(self.t.name)
|
||||
self.vz, self.other, self.nas, self.proc = r / "vz", r / "other", r / "nas", r / "proc"
|
||||
for d in (self.vz / "dump", self.other / "dump", self.nas / "dump", self.proc):
|
||||
d.mkdir(parents=True)
|
||||
(self.proc / "stat").write_text(f"cpu 1 2 3\nbtime {BOOT}\n")
|
||||
self.cfg = r / "storage.cfg"
|
||||
self.cfg.write_text(f"dir: local\n\tpath {self.vz}\n\tcontent iso,backup\n\n"
|
||||
f"lvmthin: local-lvm\n\tthinpool data\n\tvgname pve\n\n"
|
||||
f"dir: other\n\tpath {self.other}\n\tcontent backup\n\n"
|
||||
f"nfs: nas\n\tpath {self.nas}\n\tserver 192.0.2.1\n\texport /x\n\tcontent backup\n")
|
||||
self.pid = 1000
|
||||
|
||||
def partial(self, name=NAME + ".tar.dat", mtime=BOOT - 100, where=None, size=4096):
|
||||
p = (where or self.vz / "dump") / name
|
||||
if name.endswith(".tmp"):
|
||||
(p / "etc" / "vzdump").mkdir(parents=True)
|
||||
(p / "etc" / "vzdump" / "pct.conf").write_text("x")
|
||||
for q in [p / "etc" / "vzdump" / "pct.conf", p / "etc" / "vzdump", p / "etc", p]:
|
||||
os.utime(q, (mtime, mtime))
|
||||
else:
|
||||
p.write_bytes(b"\0" * size)
|
||||
os.utime(p, (mtime, mtime))
|
||||
return p
|
||||
|
||||
def process(self, cmdline, fds=(), cwd=None):
|
||||
self.pid += 1
|
||||
d = self.proc / str(self.pid)
|
||||
(d / "fd").mkdir(parents=True)
|
||||
(d / "cmdline").write_bytes(b"\0".join(a.encode() for a in cmdline) + b"\0")
|
||||
for i, f in enumerate(fds):
|
||||
os.symlink(str(f), d / "fd" / str(i + 3))
|
||||
os.symlink(str(cwd or "/"), d / "cwd")
|
||||
|
||||
def run(self, storage, vmid, name, quiet=120):
|
||||
script = (f'source "{HELPER}"; STORAGE_CFG="{self.cfg}"; PROC="{self.proc}"; QUIET_S={quiet}; '
|
||||
f'main "$@"')
|
||||
r = subprocess.run(["bash", "-c", script, "x", storage, str(vmid), name], capture_output=True, text=True)
|
||||
return r.returncode, r.stdout + r.stderr
|
||||
|
||||
|
||||
class Removes(unittest.TestCase):
|
||||
def test_dead_tar_dat_before_boot(self):
|
||||
b = Box(); p = b.partial()
|
||||
rc, out = b.run("local", 9201, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("written before this boot", out)
|
||||
|
||||
def test_dead_tmp_dir_before_boot(self):
|
||||
b = Box(); p = b.partial(NAME + ".tmp")
|
||||
rc, out = b.run("local", 9201, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
|
||||
|
||||
def test_quiet_partial_after_boot(self):
|
||||
b = Box(); p = b.partial(mtime=int(time.time()) - 300)
|
||||
rc, out = b.run("local", 9201, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("untouched", out)
|
||||
|
||||
def test_qemu_vma_dat(self):
|
||||
b = Box(); p = b.partial("vzdump-qemu-341-2026_10_14-10_42_25.vma.dat")
|
||||
rc, out = b.run("local", 341, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
|
||||
|
||||
|
||||
class Keeps(unittest.TestCase):
|
||||
"""Each case: (setup that must be REFUSED, the same setup that must be REMOVED)."""
|
||||
|
||||
def pair(self, refuse, control, quiet=120):
|
||||
b = Box(); target, args = refuse(b)
|
||||
rc, out = b.run(*args, quiet=quiet)
|
||||
self.assertNotEqual(rc, 0, f"must refuse: {out}")
|
||||
self.assertIn("REFUSED", out)
|
||||
self.assertTrue(os.path.lexists(target), "a refused entry must still exist")
|
||||
c = Box(); ctarget, cargs = control(c)
|
||||
crc, cout = c.run(*cargs, quiet=quiet)
|
||||
self.assertEqual(crc, 0, f"positive control must be removed: {cout}")
|
||||
self.assertFalse(os.path.lexists(ctarget))
|
||||
|
||||
def test_complete_archive_of_same_vmid(self):
|
||||
self.pair(lambda b: (b.partial(NAME + ".tar.zst"), ("local", 9201, NAME + ".tar.zst")),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_log_and_notes(self):
|
||||
for suffix in (".log", ".tar.zst.notes"):
|
||||
self.pair(lambda b, s=suffix: (b.partial(NAME + s), ("local", 9201, NAME + s)),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_partial_of_another_vmid(self):
|
||||
other = "vzdump-lxc-9202-2026_10_14-10_42_25.tar.dat"
|
||||
self.pair(lambda b: (b.partial(other), ("local", 9201, other)),
|
||||
lambda b: (b.partial(other), ("local", 9202, other)))
|
||||
|
||||
def test_partial_on_a_non_dir_storage(self):
|
||||
self.pair(lambda b: (b.partial(), ("local-lvm", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_partial_on_a_network_storage_with_a_path(self):
|
||||
# an nfs storage HAS a path: only the „dir only" rule (both its guards) can refuse it
|
||||
self.pair(lambda b: (b.partial(where=b.nas / "dump"), ("nas", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_partial_on_another_storage_than_named(self):
|
||||
self.pair(lambda b: (b.partial(where=b.other / "dump"), ("local", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_unknown_storage(self):
|
||||
self.pair(lambda b: (b.partial(), ("nosuch", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_vzdump_task_in_flight(self):
|
||||
def refuse(b):
|
||||
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzdump:9201:felhom-agent@pve!agent:"]); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzrestore:9202:root@pam:"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_any_vzdump_job_even_without_the_vmid(self):
|
||||
# a job for several guests: `vzdump::` — and a vzdump for ANOTHER guest — both refuse
|
||||
for cmd in (["task UPID:demo-hp:001:002:6ACF4071:vzdump::root@pam:"],
|
||||
["task UPID:demo-hp:001:002:6ACF4071:vzdump:9202:felhom-agent@pve!agent:"]):
|
||||
def refuse(b, c=cmd):
|
||||
p = b.partial(); b.process(c); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzstart:9202:root@pam:"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_vzdump_command_for_the_vmid(self):
|
||||
def refuse(b):
|
||||
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdump", "9201", "--storage", "local"]); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdumpfoo", "9201"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_file_held_open_by_an_orphan_writer(self):
|
||||
def refuse(b):
|
||||
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[p]); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[b.vz / "dump" / "something-else"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_tmp_dir_with_a_process_inside(self):
|
||||
def refuse(b):
|
||||
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=p / "etc"); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=b.vz); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_recently_written_after_boot(self):
|
||||
self.pair(lambda b: (b.partial(mtime=int(time.time()) - 10), ("local", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(mtime=int(time.time()) - 300), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_symlink(self):
|
||||
def refuse(b):
|
||||
real = b.partial(where=b.vz)
|
||||
link = b.vz / "dump" / (NAME + ".tar.dat")
|
||||
os.symlink(real, link)
|
||||
return link, ("local", 9201, link.name)
|
||||
# quiet=0: the link was made just now, so only the symlink rule may refuse it
|
||||
self.pair(refuse, lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")), quiet=0)
|
||||
|
||||
def test_near_miss_names(self):
|
||||
for bad in (NAME + ".tar.dat.bak", "vzdump-lxc-92011-2026_10_14-10_42_25.tar.dat", NAME + ".tar.da",
|
||||
"x" + NAME + ".tar.dat", NAME + ".TAR.DAT", "vzdump-lxc-9201-2026_10_14-10_42.tar.dat"):
|
||||
self.pair(lambda b, n=bad: (b.partial(n), ("local", 9201, n)),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_path_traversal_and_bad_args(self):
|
||||
b = Box(); p = b.partial()
|
||||
for args in (("local", 9201, "../dump/" + p.name), ("local", 9201, ""), ("loc al", 9201, p.name),
|
||||
("local", "9201x", p.name), ("local", 12, p.name)):
|
||||
rc, out = b.run(*args)
|
||||
self.assertNotEqual(rc, 0, f"{args}: {out}")
|
||||
self.assertTrue(p.exists())
|
||||
|
||||
def test_wrong_kind(self):
|
||||
def refuse(b): # a FILE named .tmp
|
||||
p = b.vz / "dump" / (NAME + ".tmp"); p.write_bytes(b"x"); os.utime(p, (BOOT - 100, BOOT - 100)); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, lambda b: (b.partial(NAME + ".tmp"), ("local", 9201, NAME + ".tmp")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -103,6 +103,29 @@ func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Tim
|
||||
return e.at, ok
|
||||
}
|
||||
|
||||
// KnownBackupSuccesses returns every saved success as a host-report record (the collector's
|
||||
// KnownBackupReporter, so the hub keeps its evidence across a restart). Only the tier, guest and start
|
||||
// time are known; the archive name is not saved and stays empty. Sorted for a stable report.
|
||||
func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := make([]hub.Backup, 0, len(s.last))
|
||||
for _, e := range s.last {
|
||||
out = append(out, hub.Backup{TargetID: e.target, VMID: e.vmid, Success: true, CrashConsistent: true,
|
||||
StartedAt: e.at.Format(time.RFC3339), UncoveredVolumes: []string{}})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].TargetID != out[j].TargetID {
|
||||
return out[i].TargetID < out[j].TargetID
|
||||
}
|
||||
return out[i].VMID < out[j].VMID
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *BackupSuccessState) saveLocked() error {
|
||||
entries := make([]backupSuccessJSON, 0, len(s.last))
|
||||
for _, e := range s.last {
|
||||
|
||||
@@ -57,3 +57,20 @@ func TestBackupSuccessState_CorruptFileIsEmpty(t *testing.T) {
|
||||
t.Fatal("a corrupt file must read as nothing known")
|
||||
}
|
||||
}
|
||||
|
||||
// The saved successes come back as host-report records: success, tier, guest, start time (R-894 → the
|
||||
// host report, 2026-10-09). A new state read from the same file gives the same records (the restart case).
|
||||
func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
s := NewBackupSuccessState(path)
|
||||
if err := s.RecordBackupSuccess("felhom-backup", hub.Backup{VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := NewBackupSuccessState(path).KnownBackupSuccesses()
|
||||
if len(got) != 1 || !got[0].Success || got[0].TargetID != "felhom-backup" || got[0].VMID != 9201 || got[0].StartedAt != "2026-10-09T02:40:16Z" {
|
||||
t.Fatalf("got %+v", got)
|
||||
}
|
||||
if got[0].UncoveredVolumes == nil {
|
||||
t.Fatal("uncovered_volumes must marshal as [], not null")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -81,3 +81,28 @@ func TestR685_FailsOpen(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestR1006_LeftoverCleanupRunsBeforeThePreflight — the night of 2026-10-14: a 12.2 GB half-written archive from a power
|
||||
// cut held the space, so every later backup was skipped. The cleanup step runs BEFORE the preflight reads the free space:
|
||||
// here it „removes" the leftover (free 4 → 16 GiB) and the backup then fits and runs.
|
||||
//
|
||||
// RED-PROOF: drop the r.beforeBackup call from backup() (or move it after spaceFits) → the backup is skipped → fails.
|
||||
func TestR1006_LeftoverCleanupRunsBeforeThePreflight(t *testing.T) {
|
||||
api := spaceAPI(4*gib, 8182759056, "dir")
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
var got []string
|
||||
r.SetBeforeBackup(func(_ context.Context, storage string, vmid int) {
|
||||
got = append(got, storage)
|
||||
if vmid != 9201 {
|
||||
t.Errorf("cleanup asked for vmid %d", vmid)
|
||||
}
|
||||
api.storages[0].Avail = 16 * gib
|
||||
})
|
||||
_, _ = r.Backup(context.Background(), 9201)
|
||||
if len(got) != 1 || got[0] != "local" {
|
||||
t.Fatalf("the cleanup must run once for this target, got %v", got)
|
||||
}
|
||||
if len(api.vzdumps) != 1 {
|
||||
t.Fatalf("after the cleanup freed the space the backup must run, vzdumps=%d", len(api.vzdumps))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -58,8 +58,11 @@ type BackupRunner struct {
|
||||
// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
|
||||
// there is a deliberate act.
|
||||
allowPBSPrune bool
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// beforeBackup runs just before the space preflight (R-1006): it removes a dead half-written vzdump of this guest on
|
||||
// this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing.
|
||||
beforeBackup func(ctx context.Context, storage string, vmid int)
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
|
||||
// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
|
||||
// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
|
||||
@@ -102,6 +105,11 @@ func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode,
|
||||
now: func() time.Time { return time.Now().UTC() }}
|
||||
}
|
||||
|
||||
// SetBeforeBackup sets the step run just before the space preflight (R-1006: the dead-leftover cleanup).
|
||||
func (r *BackupRunner) SetBeforeBackup(fn func(ctx context.Context, storage string, vmid int)) {
|
||||
r.beforeBackup = fn
|
||||
}
|
||||
|
||||
// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
|
||||
// applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a
|
||||
// separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type
|
||||
@@ -183,6 +191,9 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
|
||||
// R-685 (v0.134.0): will the new archive FIT on a local target? Asked before anything runs, so a
|
||||
// target that cannot hold it is a named SKIP with the numbers, not a nightly "No space left on
|
||||
// device" that only the vzdump log explains (demo-hp, every night from 2026-09-23 — R-684).
|
||||
if r.beforeBackup != nil {
|
||||
r.beforeBackup(ctx, r.target, vmid)
|
||||
}
|
||||
if ok, why := r.spaceFits(ctx, vmid); !ok {
|
||||
rec.Error = BackupSkipNoSpacePrefix + why
|
||||
rec.DurationSeconds = time.Since(start).Seconds()
|
||||
|
||||
@@ -115,6 +115,9 @@ var manifest = []Capability{
|
||||
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
|
||||
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
|
||||
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
|
||||
{"guest-shutdown", "household power-off: guest shutdown (R-998)", "/usr/sbin/pct", []string{"shutdown", "9201", "--timeout", "120"}, false, ""},
|
||||
{"host-poweroff", "household power-off: host power-off (R-998)", "/usr/bin/systemctl", []string{"poweroff"}, false, ""},
|
||||
{"vzdump-partial-clean", "dead half-written vzdump removal (R-1006)", "/usr/local/sbin/felhom-vzdump-partial-clean", []string{"local", "9201", "vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat"}, false, ""},
|
||||
|
||||
// ---- LAN split-horizon resolver (dnsmasq) ----
|
||||
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
|
||||
|
||||
@@ -311,3 +311,39 @@ func TestWGCapabilityCriticality(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEveryAliasIsGranted closes a gap found while adding FELHOM_POWEROFF (R-998): the coverage check above reads
|
||||
// every Cmnd_Alias body whether or not the alias appears on the grant line, so a manifest command whose alias was
|
||||
// never GRANTED still passed — and failed with „a password is required" on the box. Every defined alias must be
|
||||
// named on the `felhom-agent ALL=(root) NOPASSWD:` line. Red-proof: drop FELHOM_POWEROFF from that line.
|
||||
func TestEveryAliasIsGranted(t *testing.T) {
|
||||
data, err := os.ReadFile(sudoersPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read sudoers: %v", err)
|
||||
}
|
||||
var defined []string
|
||||
granted := map[string]bool{}
|
||||
grantLines := 0
|
||||
for _, ln := range strings.Split(string(data), "\n") {
|
||||
tr := strings.TrimSpace(ln)
|
||||
if strings.HasPrefix(tr, "Cmnd_Alias ") {
|
||||
if f := strings.Fields(tr); len(f) >= 2 {
|
||||
defined = append(defined, f[1])
|
||||
}
|
||||
}
|
||||
if strings.HasPrefix(tr, "felhom-agent ") && strings.Contains(tr, "NOPASSWD:") {
|
||||
grantLines++
|
||||
for _, a := range strings.Split(tr[strings.Index(tr, "NOPASSWD:")+len("NOPASSWD:"):], ",") {
|
||||
granted[strings.TrimSpace(a)] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if grantLines == 0 || len(defined) < 10 {
|
||||
t.Fatalf("parsed %d grant line(s), %d aliases — parser broke", grantLines, len(defined))
|
||||
}
|
||||
for _, a := range defined {
|
||||
if !granted[a] {
|
||||
t.Errorf("Cmnd_Alias %s is defined but never granted to felhom-agent", a)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -53,6 +53,25 @@ var r861Injections = []string{
|
||||
"/usr/sbin/pct exec 9201 -- tee /etc/felhom-controller-image",
|
||||
"/usr/local/sbin/felhom-priv-apply controller-image 9201 9202",
|
||||
"/usr/local/sbin/felhom-priv-apply controller-image 9201;id",
|
||||
// R-998 (host power-off): the two new grants take exactly their fixed shape — no extra option, no other target
|
||||
"/usr/sbin/pct shutdown 9201 --timeout 120 --forceStop 1",
|
||||
"/usr/sbin/pct shutdown 9201 --forceStop 1 --timeout 120",
|
||||
"/usr/sbin/pct shutdown 9201",
|
||||
"/usr/sbin/pct shutdown 9201 --timeout -1",
|
||||
"/usr/bin/systemctl poweroff --force",
|
||||
"/usr/bin/systemctl poweroff -ff",
|
||||
"/usr/bin/systemctl reboot",
|
||||
"/usr/bin/systemctl halt",
|
||||
"/usr/bin/systemctl kexec",
|
||||
// R-1006: the partial-cleanup helper takes exactly one storage, one vmid, one half-written name.
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.zst",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.log",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 ../../etc/vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat /etc/shadow",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat;id",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local;id 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean --help",
|
||||
}
|
||||
|
||||
func TestSudoersRefusesTheR861Injections(t *testing.T) {
|
||||
|
||||
+50
-10
@@ -59,8 +59,26 @@ var (
|
||||
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
|
||||
// which is the one fact the customer needs to recognise it.
|
||||
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
|
||||
// ErrRetainedUnchecked — the code did NOT open the current package, and NOT every earlier package the hub
|
||||
// holds for this host was tried (R-304, 2026-10-08): the retained list could not be fetched, the hub
|
||||
// withheld rows (over its serve cap, or rows with no key material), a package was malformed, or the
|
||||
// attempt cap stopped the loop. So „the code is wrong" is NOT known — it may be right for a package
|
||||
// nobody tried. Distinct from a mistype for exactly the R-224 reason: never accuse the customer of
|
||||
// something we did not check.
|
||||
ErrRetainedUnchecked = errors.New("escrow: the recovery code did not open the current sealed package, and some earlier packages were NOT checked")
|
||||
)
|
||||
|
||||
// RetainedUncheckedError wraps ErrRetainedUnchecked with how many earlier packages went unchecked
|
||||
// (-1 = unknown: the retained list itself could not be read). No secret.
|
||||
type RetainedUncheckedError struct {
|
||||
Unchecked int
|
||||
}
|
||||
|
||||
func (e *RetainedUncheckedError) Error() string {
|
||||
return fmt.Sprintf("%s (unchecked=%d)", ErrRetainedUnchecked.Error(), e.Unchecked)
|
||||
}
|
||||
func (e *RetainedUncheckedError) Unwrap() error { return ErrRetainedUnchecked }
|
||||
|
||||
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
|
||||
// carries no secret — not the code, not the bundle, not the repository password.
|
||||
type RetainedMatch struct {
|
||||
@@ -102,8 +120,10 @@ type RetainedBlob struct {
|
||||
}
|
||||
|
||||
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
|
||||
// slice is a clean "none". R-311.
|
||||
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error)
|
||||
// slice is a clean "none". R-311. `withheld` counts the earlier packages the hub holds that are NOT in
|
||||
// blobs — rows with no key material, rows over the hub's serve cap, and packages dropped as malformed
|
||||
// (R-304): each is a package the code was never tried against.
|
||||
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, withheld int, err error)
|
||||
|
||||
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
||||
type OffsiteKeyRecoverer struct {
|
||||
@@ -154,9 +174,14 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
|
||||
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
|
||||
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
|
||||
// about our own incuriosity, read by the customer as a statement about their code.
|
||||
if m, ok := r.tryRetained(ctx, recoveryCode); ok {
|
||||
m, ok, unchecked := r.tryRetained(ctx, recoveryCode)
|
||||
if ok {
|
||||
return "", &RetainedOpenedError{Match: m}
|
||||
}
|
||||
// R-304: only when EVERY earlier package the hub holds was tried may this stay a wrong code.
|
||||
if unchecked != 0 {
|
||||
return "", &RetainedUncheckedError{Unchecked: unchecked}
|
||||
}
|
||||
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
||||
}
|
||||
if bundle.ResticRepoPassword == "" {
|
||||
@@ -173,23 +198,38 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
|
||||
// function breaking is the behaviour we had before it existed.
|
||||
//
|
||||
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
|
||||
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) {
|
||||
//
|
||||
// R-304 (2026-10-08): it also returns how many earlier packages were NOT tried — `withheld` from the hub, plus
|
||||
// the ones past the attempt cap — or -1 when the retained list could not be read at all. A nil FetchRetained
|
||||
// (an agent wired without the lookup) reports 0: the pre-R-311 refusal, unchanged.
|
||||
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool, int) {
|
||||
if r.FetchRetained == nil {
|
||||
return RetainedMatch{}, false
|
||||
return RetainedMatch{}, false, 0
|
||||
}
|
||||
blobs, _, err := r.FetchRetained(ctx)
|
||||
if err != nil || len(blobs) == 0 {
|
||||
return RetainedMatch{}, false
|
||||
blobs, withheld, err := r.FetchRetained(ctx)
|
||||
if err != nil {
|
||||
return RetainedMatch{}, false, -1
|
||||
}
|
||||
if withheld < 0 {
|
||||
withheld = 0
|
||||
}
|
||||
if len(blobs) == 0 {
|
||||
return RetainedMatch{}, false, withheld
|
||||
}
|
||||
limit := r.MaxRetainedTried
|
||||
if limit <= 0 {
|
||||
limit = defaultMaxRetainedTried
|
||||
}
|
||||
unchecked := withheld
|
||||
if len(blobs) > limit {
|
||||
unchecked += len(blobs) - limit
|
||||
}
|
||||
for i, rb := range blobs {
|
||||
if i >= limit {
|
||||
break
|
||||
}
|
||||
if len(rb.Blob) == 0 {
|
||||
unchecked++
|
||||
continue
|
||||
}
|
||||
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
|
||||
@@ -204,7 +244,7 @@ func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode strin
|
||||
// correct and must be told so — but the history behind it still cannot be reopened, and
|
||||
// saying otherwise would be a promise this path cannot keep.
|
||||
HasResticPassword: bundle.ResticRepoPassword != "",
|
||||
}, true
|
||||
}, true, 0
|
||||
}
|
||||
return RetainedMatch{}, false
|
||||
return RetainedMatch{}, false, unchecked
|
||||
}
|
||||
|
||||
@@ -115,8 +115,9 @@ func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The
|
||||
// worst outcome of this feature breaking is the behaviour we had before it.
|
||||
// FAIL-SAFE — if the retained lookup itself fails, the lookup's error never reaches the customer and no
|
||||
// retained package is claimed. R-304 (2026-10-08) changed what stands instead: not the wrong-code refusal (the
|
||||
// earlier packages were never tried, so „wrong" is not known) but ErrRetainedUnchecked with Unchecked = -1.
|
||||
//
|
||||
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
|
||||
// gets a new, unexplained failure mode → this FAILS.
|
||||
@@ -140,6 +141,10 @@ func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
|
||||
if containsStr(err.Error(), "hub exploded") {
|
||||
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
|
||||
}
|
||||
var ue *RetainedUncheckedError
|
||||
if !errors.As(err, &ue) || ue.Unchecked != -1 {
|
||||
t.Errorf("err = %v, want RetainedUncheckedError{-1} — the earlier packages were never tried (R-304)", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
|
||||
@@ -228,3 +233,72 @@ func containsStr(hay, needle string) bool {
|
||||
return false
|
||||
})()
|
||||
}
|
||||
|
||||
// ── R-304 (2026-10-08) — „wrong code" only when every earlier package was tried ──────────────────────
|
||||
//
|
||||
// The consequence asserted: is the refusal a WRONG CODE (the only error the screen may answer with „check your
|
||||
// typing")? It may be only when the hub withheld nothing and every served package was tried.
|
||||
//
|
||||
// RED-PROOF: make tryRetained return 0 for `unchecked` → the three „unchecked" cases return the plain refusal
|
||||
// → they FAIL; the all-tried case keeps passing.
|
||||
func TestR304_WrongCodeOnlyWhenEveryEarlierPackageWasTried(t *testing.T) {
|
||||
ensureAge(t)
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "4444567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
other := sealBundle(t, IdentityBundle{ResticRepoPassword: "5555567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
const code = "a code that opens nothing whatsoever in this test"
|
||||
cases := []struct {
|
||||
name string
|
||||
blobs int
|
||||
withheld int
|
||||
limit int
|
||||
unchecked int // 0 = must be a plain wrong code
|
||||
}{
|
||||
{"all tried, nothing withheld", 2, 0, 6, 0},
|
||||
{"the hub withheld rows (no key material / over its cap)", 1, 3, 6, 3},
|
||||
{"only withheld rows, nothing served", 0, 2, 6, 2},
|
||||
{"the attempt cap stopped the loop", 5, 0, 2, 3},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
blobs := make([]RetainedBlob, 0, c.blobs)
|
||||
for i := 0; i < c.blobs; i++ {
|
||||
blobs = append(blobs, RetainedBlob{Blob: other, SupersededAt: "2026-08-01 00:00:00", Index: i})
|
||||
}
|
||||
_, err := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
|
||||
return blobs, c.withheld, nil
|
||||
},
|
||||
MaxRetainedTried: c.limit,
|
||||
}.RecoverOffsiteRepoPassword(context.Background(), code)
|
||||
if err == nil {
|
||||
t.Fatal("a code that opens nothing succeeded")
|
||||
}
|
||||
var ue *RetainedUncheckedError
|
||||
isUnchecked := errors.As(err, &ue)
|
||||
if c.unchecked == 0 {
|
||||
if isUnchecked {
|
||||
t.Fatalf("every earlier package was tried — this IS a wrong code, got %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if !isUnchecked || ue.Unchecked != c.unchecked || !errors.Is(err, ErrRetainedUnchecked) {
|
||||
t.Fatalf("err = %v, want RetainedUncheckedError{%d} — never call the code wrong when packages were not tried", err, c.unchecked)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A malformed (empty) served package counts as not tried.
|
||||
func TestR304_EmptyServedPackageCountsAsUnchecked(t *testing.T) {
|
||||
ensureAge(t)
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "6666567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
_, err := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: nil, SupersededAt: "2026-08-01 00:00:00"}),
|
||||
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
|
||||
var ue *RetainedUncheckedError
|
||||
if !errors.As(err, &ue) || ue.Unchecked != 1 {
|
||||
t.Fatalf("err = %v, want RetainedUncheckedError{1}", err)
|
||||
}
|
||||
}
|
||||
|
||||
+60
-5
@@ -66,6 +66,13 @@ type ProvenRestoreTestReporter interface {
|
||||
ProvenRestoreTests(ctx context.Context) []RestoreTest
|
||||
}
|
||||
|
||||
// KnownBackupReporter is the newest SUCCESSFUL backup per tier and guest kept on disk (R-894's
|
||||
// backup-success-state.json). collectBackups folds it into the report so a restart does not erase the
|
||||
// hub's evidence of a backup that ran minutes before it (the kernel-night shape, 2026-10-09).
|
||||
type KnownBackupReporter interface {
|
||||
KnownBackupSuccesses() []Backup
|
||||
}
|
||||
|
||||
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
|
||||
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
|
||||
type PBSReporter interface {
|
||||
@@ -105,6 +112,7 @@ type Collector struct {
|
||||
backups BackupReporter
|
||||
restoreTests RestoreTestReporter
|
||||
provenTests ProvenRestoreTestReporter
|
||||
knownBackups KnownBackupReporter // R-894 file → host report after a restart (nil → in-memory only)
|
||||
pbs PBSReporter
|
||||
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||
@@ -115,6 +123,7 @@ type Collector struct {
|
||||
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
||||
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
|
||||
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
|
||||
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
||||
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
|
||||
@@ -247,6 +256,12 @@ func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector
|
||||
return c
|
||||
}
|
||||
|
||||
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
|
||||
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
|
||||
c.deliberateOff = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
|
||||
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
|
||||
type SelfUpdateReporter interface {
|
||||
@@ -423,6 +438,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
if c.oob != nil {
|
||||
report.OOB = c.oob.OOBStatus(ctx)
|
||||
}
|
||||
// R-998: a household power-off accepted in this boot rides every report until the box is down.
|
||||
if c.deliberateOff != nil {
|
||||
report.DeliberateOff = c.deliberateOff()
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
@@ -576,14 +595,50 @@ func (c *Collector) collectStorage(ctx context.Context) []StorageTarget {
|
||||
// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state
|
||||
// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil)
|
||||
// list so the collection always marshals as [].
|
||||
//
|
||||
// The saved successes (SetKnownBackupReporter) are added for each tier and guest the in-memory list has
|
||||
// no success for at or after the saved time. Why: the in-memory list is empty after an agent restart,
|
||||
// and the hub's backup-freshness check reads only what the reports carried. Measured 2026-10-09 on
|
||||
// demo-felhom: the night backup landed 02:40 UTC, the kernel step restarted the host at 02:44, no report
|
||||
// fell in between, so two kernel nights in a row left no trace and the hub alarmed „newest backup is 48h
|
||||
// old" at 03:00. Only SUCCESSES are saved, so a failure is never hidden and never invented.
|
||||
// Pinned by TestCollectBackups_SavedSuccessSurvivesARestart.
|
||||
func (c *Collector) collectBackups(ctx context.Context) []Backup {
|
||||
if c.backups == nil {
|
||||
return []Backup{}
|
||||
out := []Backup{}
|
||||
if c.backups != nil {
|
||||
if b := c.backups.Backups(ctx); b != nil {
|
||||
out = append(out, b...)
|
||||
}
|
||||
}
|
||||
if b := c.backups.Backups(ctx); b != nil {
|
||||
return b
|
||||
if c.knownBackups == nil {
|
||||
return out
|
||||
}
|
||||
return []Backup{}
|
||||
for _, k := range c.knownBackups.KnownBackupSuccesses() {
|
||||
kt, err := time.Parse(time.RFC3339, k.StartedAt)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
covered := false
|
||||
for _, b := range out {
|
||||
if !b.Success || b.TargetID != k.TargetID || b.VMID != k.VMID {
|
||||
continue
|
||||
}
|
||||
if bt, err := time.Parse(time.RFC3339, b.StartedAt); err == nil && !bt.Before(kt) {
|
||||
covered = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !covered {
|
||||
out = append(out, k)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SetKnownBackupReporter wires the R-894 saved successes into the host report (nil-safe → in-memory only).
|
||||
func (c *Collector) SetKnownBackupReporter(r KnownBackupReporter) *Collector {
|
||||
c.knownBackups = r
|
||||
return c
|
||||
}
|
||||
|
||||
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type fakeMemBackups []Backup
|
||||
|
||||
func (f fakeMemBackups) Backups(context.Context) []Backup { return f }
|
||||
|
||||
type fakeKnownBackups []Backup
|
||||
|
||||
func (f fakeKnownBackups) KnownBackupSuccesses() []Backup { return f }
|
||||
|
||||
// The consequence, not the mechanism: after a restart (empty in-memory list) the host report still carries
|
||||
// the night's backup, so the hub's freshness check sees it. Measured 2026-10-09 on demo-felhom: without this
|
||||
// the report carried backups: [] and the hub alarmed „newest backup is 48h old" an hour after a good backup.
|
||||
// RED-PROOF: return before the saved-success loop in collectBackups → the first case reports nothing → fails.
|
||||
func TestCollectBackups_SavedSuccessSurvivesARestart(t *testing.T) {
|
||||
saved := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}
|
||||
|
||||
t.Run("restart: memory empty, the saved success is reported", func(t *testing.T) {
|
||||
c := &Collector{backups: fakeMemBackups(nil), knownBackups: fakeKnownBackups{saved}}
|
||||
got := c.collectBackups(context.Background())
|
||||
if len(got) != 1 || got[0].StartedAt != saved.StartedAt || !got[0].Success || got[0].TargetID != "felhom-backup" {
|
||||
t.Fatalf("want the saved success, got %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("memory holds the same or a newer success: no second entry", func(t *testing.T) {
|
||||
mem := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z", Archive: "a"}
|
||||
c := &Collector{backups: fakeMemBackups{mem}, knownBackups: fakeKnownBackups{saved}}
|
||||
if got := c.collectBackups(context.Background()); len(got) != 1 || got[0].Archive != "a" {
|
||||
t.Fatalf("want only the in-memory record, got %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a newer FAILURE in memory never hides the saved success, and is kept", func(t *testing.T) {
|
||||
fail := Backup{TargetID: "felhom-backup", VMID: 9201, Success: false, StartedAt: "2026-10-10T02:40:00Z", Error: "x"}
|
||||
c := &Collector{backups: fakeMemBackups{fail}, knownBackups: fakeKnownBackups{saved}}
|
||||
got := c.collectBackups(context.Background())
|
||||
if len(got) != 2 || got[0].Success || !got[1].Success {
|
||||
t.Fatalf("want the failure and the saved success, got %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("another tier's success does not cover this tier", func(t *testing.T) {
|
||||
other := Backup{TargetID: "felhom-pbs", VMID: 9201, Success: true, StartedAt: "2026-10-10T00:00:00Z"}
|
||||
c := &Collector{backups: fakeMemBackups{other}, knownBackups: fakeKnownBackups{saved}}
|
||||
if got := c.collectBackups(context.Background()); len(got) != 2 {
|
||||
t.Fatalf("want both tiers, got %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("no saved state wired: the in-memory list as before, never nil", func(t *testing.T) {
|
||||
c := &Collector{}
|
||||
if got := c.collectBackups(context.Background()); got == nil || len(got) != 0 {
|
||||
t.Fatalf("want an empty non-nil list, got %#v", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-998: the power-off path waits on SetReportedHook, so the hook must fire only for a report the hub
|
||||
// ACCEPTED — a failed push must not let the box power off believing the hub knows.
|
||||
func TestLoop_ReportedHookOnlyAfterAcceptedReport(t *testing.T) {
|
||||
var cn, rn int32
|
||||
rep := &HostReport{DeliberateOff: &DeliberateOff{RequestedAt: "2026-10-13T18:00:00Z"}}
|
||||
loop := NewLoop(&fakeCollector{report: rep, n: &cn},
|
||||
&fakeReporter{env: &ControlEnvelope{}, errSeq: []error{errors.New("hub 5xx")}, n: &rn},
|
||||
900*time.Second, quietLogger())
|
||||
var seen []*HostReport
|
||||
loop.SetReportedHook(func(r *HostReport) { seen = append(seen, r) })
|
||||
|
||||
loop.cycle(context.Background(), 900*time.Second) // the push fails
|
||||
if len(seen) != 0 {
|
||||
t.Fatalf("hook fired for a refused report: %d", len(seen))
|
||||
}
|
||||
loop.cycle(context.Background(), 900*time.Second) // the push is accepted
|
||||
if len(seen) != 1 || seen[0].DeliberateOff == nil || seen[0].DeliberateOff.RequestedAt != "2026-10-13T18:00:00Z" {
|
||||
t.Fatalf("hook after an accepted report = %+v", seen)
|
||||
}
|
||||
}
|
||||
@@ -66,8 +66,15 @@ type Loop struct {
|
||||
// retry, no duplicate shipping). Loop state is single-goroutine (cycle only).
|
||||
logTailSource func(maxBytes int) []string
|
||||
logTailPending bool
|
||||
|
||||
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
|
||||
// path waits on it so the hub has the deliberate-off record before the box goes down.
|
||||
reported func(r *HostReport)
|
||||
}
|
||||
|
||||
// SetReportedHook wires the R-998 after-report observer. Optional.
|
||||
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
|
||||
|
||||
// logTailMaxBytes caps the heartbeat log tail (newest lines kept).
|
||||
const logTailMaxBytes = 128 * 1024
|
||||
|
||||
@@ -156,6 +163,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
|
||||
l.logger.Warn("hub: report failed; keeping current interval", "err", err)
|
||||
return current
|
||||
}
|
||||
if l.reported != nil {
|
||||
l.reported(report)
|
||||
}
|
||||
if report.LogTail != nil {
|
||||
// Transparency: the pull is visible in the box's own log (and thus in the ring).
|
||||
l.logger.Info("operator log pull served", "component", "agent", "lines", len(report.LogTail.Lines))
|
||||
|
||||
@@ -144,6 +144,12 @@ type HostReport struct {
|
||||
// in the steady state, so the cross-repo host-report golden stays byte-stable.
|
||||
LogTail *LogTail `json:"log_tail,omitempty"`
|
||||
|
||||
// DeliberateOff (R-998, doc 14): the household asked this box to power off, in THIS boot. Present on
|
||||
// every report from the request until the box goes down; absent again after the next boot — that
|
||||
// absence is how the hub learns the box is back. The hub holds its silence alarms while it is set
|
||||
// (14 days). `omitempty`: absent in the steady state, so the cross-repo host-report golden stays stable.
|
||||
DeliberateOff *DeliberateOff `json:"deliberate_off,omitempty"`
|
||||
|
||||
// OOB is the operator-access health stanza (TASK H1). It answers the operator's question — "can I
|
||||
// get into this box right now, and if not, why" — from the hub: felhom-sshd up + on which port,
|
||||
// locally reachable, the tunnel handshake age (the OOB path rides wg-felhom), whether the operator
|
||||
@@ -758,3 +764,8 @@ type ForeignKeyArchives struct {
|
||||
Oldest string `json:"oldest"`
|
||||
Newest string `json:"newest"`
|
||||
}
|
||||
|
||||
// DeliberateOff is the household-requested power-off record (R-998).
|
||||
type DeliberateOff struct {
|
||||
RequestedAt string `json:"requested_at"` // RFC3339 UTC, the agent's clock
|
||||
}
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
|
||||
// and the gate sub-case fails.
|
||||
func TestAfterPrimaryBackup(t *testing.T) {
|
||||
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
|
||||
run := func(t *testing.T, failErr, path string) (calls []int, gateHeld bool) {
|
||||
gate := &backup.InFlight{}
|
||||
b := &fakeBackups{failErr: failErr}
|
||||
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
|
||||
@@ -34,7 +34,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
||||
done <- struct{}{}
|
||||
})
|
||||
h := srv.Handler()
|
||||
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
|
||||
if do(t, h, "POST", path, "A", "").Code != http.StatusAccepted {
|
||||
t.Fatal("POST /backup not accepted")
|
||||
}
|
||||
select {
|
||||
@@ -47,7 +47,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
||||
return calls, gateHeld
|
||||
}
|
||||
t.Run("success runs the leg under the gate", func(t *testing.T) {
|
||||
calls, held := run(t, "")
|
||||
calls, held := run(t, "", "/backup")
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
|
||||
}
|
||||
@@ -56,8 +56,21 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
||||
}
|
||||
})
|
||||
t.Run("a failed backup runs nothing", func(t *testing.T) {
|
||||
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
|
||||
if calls, _ := run(t, "vzdump exploded", "/backup"); len(calls) != 0 {
|
||||
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
|
||||
}
|
||||
})
|
||||
// R-899: a household press is not the night's backup — no OS leg after it. Same fake, same successful backup as
|
||||
// the first sub-case; only the query differs. Red-proof: make handleBackup ignore `trigger` and this sub-case
|
||||
// fails with the leg run once.
|
||||
t.Run("a manual press runs nothing", func(t *testing.T) {
|
||||
if calls, _ := run(t, "", "/backup?trigger=manual"); len(calls) != 0 {
|
||||
t.Fatalf("the OS leg ran after a manual press: %v", calls)
|
||||
}
|
||||
})
|
||||
t.Run("the scheduled path with the new query still runs the leg", func(t *testing.T) {
|
||||
if calls, _ := run(t, "", "/backup?trigger=night"); len(calls) != 1 {
|
||||
t.Fatalf("the leg ran %d time(s) after a non-manual backup, want 1", len(calls))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -19,10 +19,10 @@ import (
|
||||
// again", and each guard is pinned by the case where acting would be wrong.
|
||||
|
||||
type supExec struct {
|
||||
mu sync.Mutex
|
||||
status map[int]string // docker .State.Status per vmid; "" = container absent
|
||||
inspectErr error // non-nil = pct exec itself failed (unknown)
|
||||
restarts map[int]int
|
||||
mu sync.Mutex
|
||||
status map[int]string // docker .State.Status per vmid; "" = container absent
|
||||
inspectErr error // non-nil = pct exec itself failed (unknown)
|
||||
restarts map[int]int
|
||||
// onRestart, when set, is the status the container reaches after a restart (a crash-looper
|
||||
// stays "exited").
|
||||
onRestart string
|
||||
@@ -249,7 +249,6 @@ func TestControllerSupervisorStanza_WireShape(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ---- R-539 (operator ruling 3 of 2026-09-16): the SLOW crash loop ----------------------------------
|
||||
|
||||
// supKillOnce kills the controller and lets the supervisor restart it (two confirming sweeps), then
|
||||
|
||||
@@ -226,6 +226,9 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
// The reply's fields are read HERE, before the job goroutine may write job.Phase (a data race the race
|
||||
// detector caught, 2026-10-13 — the reply could show the finished phase or a torn value).
|
||||
jobID, phase := job.JobID, job.Phase
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
@@ -274,7 +277,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
|
||||
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
|
||||
}()
|
||||
s.escrowDone = done // tests wait on it; production polls the status endpoint
|
||||
writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": job.JobID, "phase": job.Phase}, "")
|
||||
writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": jobID, "phase": phase}, "")
|
||||
}
|
||||
|
||||
// handleEscrowCeremonyStatus is GET /escrow/ceremony/status: phase + the NON-SECRET summary.
|
||||
|
||||
@@ -127,6 +127,22 @@ func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Req
|
||||
"retained_has_restic_pw": match.HasResticPassword,
|
||||
},
|
||||
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
|
||||
// ── R-304 (2026-10-08) — NOT EVERY EARLIER PACKAGE WAS CHECKED. ─────────────────────────
|
||||
//
|
||||
// The current package refused the code, no retained package opened it — and at least one earlier
|
||||
// package the hub holds was never tried (or the list could not be read). Saying „the code is
|
||||
// wrong" here would claim a check that did not happen. 424 (Failed Dependency): the verdict
|
||||
// depends on packages we could not try. The controller classifies on the status, never on this
|
||||
// sentence; an older controller maps an unknown status to its neutral „we do not know why".
|
||||
case errors.Is(err, escrow.ErrRetainedUnchecked):
|
||||
n := -1
|
||||
var ue *escrow.RetainedUncheckedError
|
||||
if errors.As(err, &ue) {
|
||||
n = ue.Unchecked
|
||||
}
|
||||
s.logger.Warn("local-api: offsite key recovery: the code did not open the current package and earlier packages were NOT all checked — not reported as a wrong code (R-304)", "vmid", vmid, "unchecked", n)
|
||||
writeStatus(w, http.StatusFailedDependency, false, map[string]any{"older_unchecked": n},
|
||||
"the recovery code did not open the current sealed package, and earlier packages the hub holds were not all checked — the code may belong to one of them; nothing was written")
|
||||
case errors.Is(err, escrow.ErrNoResticPassword):
|
||||
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
||||
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
||||
|
||||
@@ -54,6 +54,13 @@ func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
|
||||
wantStatus: 404,
|
||||
mustNotSay: []string{"did not open"},
|
||||
},
|
||||
{
|
||||
// R-304: earlier packages were not all tried — never „did not open the sealed bundle" (the wrong-code words).
|
||||
name: "earlier packages not all checked — not a wrong code",
|
||||
err: &escrow.RetainedUncheckedError{Unchecked: 2},
|
||||
wantStatus: 424,
|
||||
mustNotSay: []string{"did not open the sealed bundle", "could not be fetched"},
|
||||
},
|
||||
{
|
||||
name: "the bundle predates the repository-password field",
|
||||
err: escrow.ErrNoResticPassword,
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
@@ -104,6 +105,26 @@ func (b *GuestBinder) RebootGuest(ctx context.Context, vmid int) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// ShutdownGuest stops the guest cleanly with a time limit (R-998 host power-off, step 2). `pct shutdown`
|
||||
// blocks until the guest is down or the limit passes; then it fails and the guest is still running.
|
||||
func (b *GuestBinder) ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error {
|
||||
secs := int(timeout / time.Second)
|
||||
if secs < 1 {
|
||||
secs = 1
|
||||
}
|
||||
b.logger.Warn("host-poweroff: shutting the guest down", "vmid", vmid, "timeout_s", secs)
|
||||
if err := b.run(ctx, "pct", "shutdown", strconv.Itoa(vmid), "--timeout", strconv.Itoa(secs)); err != nil {
|
||||
return fmt.Errorf("host-poweroff: pct shutdown %d: %w", vmid, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PowerOffHost powers the Proxmox host off (R-998, step 3). The one fixed vector `/usr/bin/systemctl poweroff`.
|
||||
func (b *GuestBinder) PowerOffHost(ctx context.Context) error {
|
||||
b.logger.Warn("host-poweroff: powering the host off")
|
||||
return b.run(ctx, "/usr/bin/systemctl", "poweroff")
|
||||
}
|
||||
|
||||
func (b *GuestBinder) run(ctx context.Context, name string, args ...string) error {
|
||||
_, stderr, err := b.runner.Run(ctx, name, args...)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
)
|
||||
|
||||
// ── POST /host/poweroff (R-998, documentation/architecture/14-host-power-off.md) ──────────────────────
|
||||
//
|
||||
// The household's „A gép kikapcsolása": shut THIS guest down cleanly, then power the whole host off. The
|
||||
// first local-API route whose effect reaches beyond the caller's own guest — a compromised controller can
|
||||
// now switch the box off, and nobody can switch it on remotely. The mitigations are the limits below:
|
||||
//
|
||||
// - refused in the first 10 minutes after boot (no power-off loop);
|
||||
// - at most ONE accepted request per boot (poweroff.State, persisted, keyed by the boot);
|
||||
// - refused while a backup, restore test, OS update step, controller update, drive format, NAS verify
|
||||
// or escrow ceremony runs, or a vzdump of the guest is in flight — each named in the 409;
|
||||
// - refused while ANY OTHER guest or VM runs on the host (security review 2026-10-13): one guest's
|
||||
// controller must not switch off another household's guest, a scratch guest or a test VM (demo-hp
|
||||
// carries the Tester 1 VM). Read from /proc (`lxc-start -n N`, `kvm -id N`), fail closed;
|
||||
// - the heavy-operation gate is TAKEN on acceptance and never given back (the box is going down), so no
|
||||
// backup, restore test or OS step can start between the checks and the power-off (security review).
|
||||
// Released only if the power-off command itself fails.
|
||||
//
|
||||
// Every check fails CLOSED: an unwired gate, an unreadable task list or /proc refuses.
|
||||
// On acceptance the record is written FIRST, the hub is given a report carrying it (bounded wait), and
|
||||
// only then: guest shutdown (time-limited), host power-off. Pinned by hostpoweroff_test.go.
|
||||
|
||||
// HostPowerer is the privileged half: both run as fixed sudo vectors (configs/felhom-agent.sudoers).
|
||||
type HostPowerer interface {
|
||||
ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error
|
||||
PowerOffHost(ctx context.Context) error
|
||||
}
|
||||
|
||||
// HostPowerOffConfig wires the route. A nil config leaves it answering 503.
|
||||
type HostPowerOffConfig struct {
|
||||
State *poweroff.State
|
||||
Power HostPowerer
|
||||
ReportNow func(ctx context.Context) bool // push a report carrying the record; true = the hub accepted it
|
||||
Uptime func() (time.Duration, bool) // time since host boot; false = unknown (refuse)
|
||||
Running func() ([]int, error) // vmids of every running guest and VM on the host; default /proc
|
||||
MinUptime time.Duration // default 10 min
|
||||
GuestStop time.Duration // pct shutdown --timeout; default 120 s
|
||||
ReportMax time.Duration // longest wait for the hub; default 60 s
|
||||
JobWait time.Duration // longest wait for a job that started after acceptance; default 15 min
|
||||
JobPoll time.Duration // how often that wait looks again; default 10 s
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
// Refusal codes — the controller maps each to a household sentence. Stable wire values.
|
||||
const (
|
||||
PowerOffTooSoon = "too_soon_after_boot"
|
||||
PowerOffAlreadyAsked = "already_requested_this_boot"
|
||||
PowerOffBackup = "backup_running"
|
||||
PowerOffRestoreTest = "restore_test_running"
|
||||
PowerOffOSUpdate = "os_update_running"
|
||||
PowerOffCtrlUpdate = "controller_update_running"
|
||||
PowerOffDrive = "drive_operation_running"
|
||||
PowerOffEscrow = "escrow_ceremony_running"
|
||||
PowerOffBusy = "host_busy"
|
||||
PowerOffOtherGuests = "other_guests_running"
|
||||
PowerOffUnknown = "state_unknown"
|
||||
)
|
||||
|
||||
// SetHostPowerOff enables POST /host/poweroff.
|
||||
func (s *Server) SetHostPowerOff(c HostPowerOffConfig) {
|
||||
if c.MinUptime == 0 {
|
||||
c.MinUptime = 10 * time.Minute
|
||||
}
|
||||
if c.GuestStop == 0 {
|
||||
c.GuestStop = 120 * time.Second
|
||||
}
|
||||
if c.ReportMax == 0 {
|
||||
c.ReportMax = 60 * time.Second
|
||||
}
|
||||
if c.Now == nil {
|
||||
c.Now = time.Now
|
||||
}
|
||||
if c.JobWait == 0 {
|
||||
c.JobWait = 15 * time.Minute
|
||||
}
|
||||
if c.JobPoll == 0 {
|
||||
c.JobPoll = 10 * time.Second
|
||||
}
|
||||
if c.Running == nil {
|
||||
c.Running = func() ([]int, error) { return runningGuestsFromProc("/proc") }
|
||||
}
|
||||
s.powerOff = &c
|
||||
}
|
||||
|
||||
type hostPowerOffRequest struct {
|
||||
VMID int `json:"vmid"`
|
||||
}
|
||||
|
||||
func (s *Server) handleHostPowerOff(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
c := s.powerOff
|
||||
if c == nil || c.State == nil || c.Power == nil {
|
||||
writeErr(w, http.StatusServiceUnavailable, "host power-off not configured on this host")
|
||||
return
|
||||
}
|
||||
if r.ContentLength != 0 {
|
||||
var req hostPowerOffRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||
return
|
||||
}
|
||||
}
|
||||
// One decision at a time: two requests racing must not both pass the guards before either marks.
|
||||
s.powerOffMu.Lock()
|
||||
defer s.powerOffMu.Unlock()
|
||||
if code, detail := s.powerOffBlocker(r.Context(), vmid); code != "" {
|
||||
s.logger.Info("local-api: host power-off refused", "vmid", vmid, "reason", code, "detail", detail)
|
||||
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": detail}, "host power-off refused: "+code)
|
||||
return
|
||||
}
|
||||
// Take the heavy-operation gate for good: nothing heavy may start between these checks and the power-off.
|
||||
release, busy, free := s.inFlight.TryAcquire("host-poweroff")
|
||||
if !free {
|
||||
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": PowerOffBusy, "detail": busy}, "host power-off refused: "+PowerOffBusy)
|
||||
return
|
||||
}
|
||||
rec, err := c.State.Mark(c.Now(), vmid)
|
||||
if err != nil {
|
||||
release()
|
||||
code := PowerOffUnknown
|
||||
if errors.Is(err, poweroff.ErrAlreadyThisBoot) {
|
||||
code = PowerOffAlreadyAsked
|
||||
}
|
||||
s.logger.Warn("local-api: host power-off refused (record)", "vmid", vmid, "reason", code, "err", err)
|
||||
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": err.Error()}, "host power-off refused: "+code)
|
||||
return
|
||||
}
|
||||
s.logger.Warn("local-api: HOST POWER-OFF accepted (household request)", "vmid", vmid,
|
||||
"requested_at", rec.RequestedAt.Format(time.RFC3339))
|
||||
base := s.baseCtx
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
go s.runHostPowerOff(base, vmid, release)
|
||||
writeStatus(w, http.StatusAccepted, true, map[string]any{
|
||||
"vmid": vmid, "powering_off": true, "requested_at": rec.RequestedAt.Format(time.RFC3339),
|
||||
}, "")
|
||||
}
|
||||
|
||||
// runHostPowerOff: hub first (bounded), then the guest (time-limited), then the host. A failed guest
|
||||
// shutdown does not stop the power-off — the household asked for it, and Proxmox's own shutdown stops
|
||||
// any guest still running (pve-guests) before the host goes down.
|
||||
func (s *Server) runHostPowerOff(base context.Context, vmid int, release func()) {
|
||||
c := s.powerOff
|
||||
if c.ReportNow != nil {
|
||||
ctx, cancel := context.WithTimeout(base, c.ReportMax)
|
||||
ok := c.ReportNow(ctx)
|
||||
cancel()
|
||||
s.logger.Warn("host-poweroff: step 1/3 — hub told", "vmid", vmid, "hub_accepted", ok)
|
||||
}
|
||||
if !s.waitJobsClear(base, vmid, "before the guest shutdown") {
|
||||
release()
|
||||
return
|
||||
}
|
||||
start := time.Now()
|
||||
ctx, cancel := context.WithTimeout(base, c.GuestStop+60*time.Second)
|
||||
err := c.Power.ShutdownGuest(ctx, vmid, c.GuestStop)
|
||||
cancel()
|
||||
if err != nil {
|
||||
s.logger.Error("host-poweroff: step 2/3 — guest shutdown failed; powering off anyway", "vmid", vmid,
|
||||
"took", time.Since(start).Round(time.Second), "err", err)
|
||||
} else {
|
||||
s.logger.Warn("host-poweroff: step 2/3 — guest stopped", "vmid", vmid, "took", time.Since(start).Round(time.Second))
|
||||
}
|
||||
if !s.waitJobsClear(base, vmid, "before the host power-off") {
|
||||
release()
|
||||
return
|
||||
}
|
||||
s.logger.Warn("host-poweroff: step 3/3 — powering the host off")
|
||||
ctx, cancel = context.WithTimeout(base, 60*time.Second)
|
||||
defer cancel()
|
||||
if err := c.Power.PowerOffHost(ctx); err != nil {
|
||||
s.logger.Error("host-poweroff: host power-off command failed — heavy-operation gate released", "err", err)
|
||||
release()
|
||||
}
|
||||
}
|
||||
|
||||
// powerOffBlocker returns the first reason to refuse ("" = none) and a detail for the log/operator.
|
||||
func (s *Server) powerOffBlocker(ctx context.Context, vmid int) (string, string) {
|
||||
c := s.powerOff
|
||||
if c.State.Current() != nil {
|
||||
return PowerOffAlreadyAsked, "a power-off was already accepted in this boot"
|
||||
}
|
||||
if c.Uptime == nil {
|
||||
return PowerOffUnknown, "uptime not wired"
|
||||
}
|
||||
up, ok := c.Uptime()
|
||||
if !ok {
|
||||
return PowerOffUnknown, "cannot read the host uptime"
|
||||
}
|
||||
if up < c.MinUptime {
|
||||
return PowerOffTooSoon, "host up " + up.Round(time.Second).String() + ", minimum " + c.MinUptime.String()
|
||||
}
|
||||
if s.inFlight == nil || s.staleLock == nil {
|
||||
return PowerOffUnknown, "the heavy-operation gate or the Proxmox task reader is not wired"
|
||||
}
|
||||
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
|
||||
return code, detail
|
||||
}
|
||||
if busy := s.inFlight.Busy(); busy != "" {
|
||||
switch {
|
||||
case strings.HasPrefix(busy, "backup:"):
|
||||
return PowerOffBackup, busy
|
||||
case busy == "restore-test":
|
||||
return PowerOffRestoreTest, busy
|
||||
case strings.HasPrefix(busy, "os-"):
|
||||
return PowerOffOSUpdate, busy
|
||||
default:
|
||||
return PowerOffBusy, busy
|
||||
}
|
||||
}
|
||||
return s.jobBlocker(ctx, vmid)
|
||||
}
|
||||
|
||||
// jobBlocker is the part of the checks that can change after acceptance — a backup job, a controller update, a drive
|
||||
// or NAS job, an escrow ceremony, a Proxmox vzdump. runHostPowerOff asks it again before each step (security review
|
||||
// 2026-10-13: the checks ran only once). The heavy-operation gate is not here: the power-off holds it.
|
||||
func (s *Server) jobBlocker(ctx context.Context, vmid int) (string, string) {
|
||||
s.jobsMu.Lock()
|
||||
for k, j := range s.jobs {
|
||||
if j != nil && backupInFlight(j.Phase) {
|
||||
s.jobsMu.Unlock()
|
||||
return PowerOffBackup, "backup job " + j.JobID + " on target " + k.target
|
||||
}
|
||||
}
|
||||
s.jobsMu.Unlock()
|
||||
s.swapMu.Lock()
|
||||
for v, on := range s.swapInFlight {
|
||||
if on {
|
||||
s.swapMu.Unlock()
|
||||
return PowerOffCtrlUpdate, "controller swap in flight for guest " + strconv.Itoa(v)
|
||||
}
|
||||
}
|
||||
s.swapMu.Unlock()
|
||||
if s.formatJobs != nil {
|
||||
if j := s.formatJobs.get(); j != nil && j.Phase == formatPhaseRunning {
|
||||
return PowerOffDrive, "drive format running"
|
||||
}
|
||||
}
|
||||
s.netVerifyMu.Lock()
|
||||
nv := s.netVerifyCur != nil && s.netVerifyCur.Phase == netVerifyPhaseRunning
|
||||
s.netVerifyMu.Unlock()
|
||||
if nv {
|
||||
return PowerOffDrive, "network storage verify running"
|
||||
}
|
||||
s.escrowMu.Lock()
|
||||
esc := s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning
|
||||
s.escrowMu.Unlock()
|
||||
if esc {
|
||||
return PowerOffEscrow, "escrow ceremony running"
|
||||
}
|
||||
vz, err := s.staleLock.BackupRunning(ctx, vmid)
|
||||
if err != nil {
|
||||
return PowerOffUnknown, "cannot list running Proxmox tasks: " + err.Error()
|
||||
}
|
||||
if vz {
|
||||
return PowerOffBackup, "vzdump running for guest " + strconv.Itoa(vmid)
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// runningGuestsFromProc lists the vmids of running containers (`lxc-start … -n N`) and VMs (`kvm -id N`) from
|
||||
// the process table — readable by the non-root agent, unlike /run/qemu-server (measured on demo-hp 2026-10-13).
|
||||
// An unreadable /proc is an error (fail closed); an unreadable single process is skipped (it exited).
|
||||
func runningGuestsFromProc(procDir string) ([]int, error) {
|
||||
ents, err := os.ReadDir(procDir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
seen := map[int]bool{}
|
||||
var out []int
|
||||
for _, e := range ents {
|
||||
if _, err := strconv.Atoi(e.Name()); err != nil {
|
||||
continue
|
||||
}
|
||||
raw, err := os.ReadFile(filepath.Join(procDir, e.Name(), "cmdline"))
|
||||
if err != nil || len(raw) == 0 {
|
||||
continue
|
||||
}
|
||||
args := strings.Split(strings.TrimRight(string(raw), "\x00"), "\x00")
|
||||
flag := ""
|
||||
switch filepath.Base(args[0]) {
|
||||
case "lxc-start":
|
||||
flag = "-n"
|
||||
case "kvm", "qemu-system-x86_64":
|
||||
flag = "-id"
|
||||
default:
|
||||
continue
|
||||
}
|
||||
for i := 1; i+1 < len(args); i++ {
|
||||
if args[i] == flag {
|
||||
if v, err := strconv.Atoi(args[i+1]); err == nil && !seen[v] {
|
||||
seen[v] = true
|
||||
out = append(out, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// otherGuestsBlocker refuses while any guest or VM other than the caller's runs (or the list cannot be read).
|
||||
func (s *Server) otherGuestsBlocker(vmid int) (string, string) {
|
||||
running, err := s.powerOff.Running()
|
||||
if err != nil {
|
||||
return PowerOffUnknown, "cannot list the running guests: " + err.Error()
|
||||
}
|
||||
var others []string
|
||||
for _, v := range running {
|
||||
if v != vmid {
|
||||
others = append(others, strconv.Itoa(v))
|
||||
}
|
||||
}
|
||||
if len(others) > 0 {
|
||||
return PowerOffOtherGuests, "other guests or VMs running: " + strings.Join(others, ", ")
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// waitJobsClear waits until jobBlocker finds nothing, at most JobWait. false = still busy (or unreadable) at the end:
|
||||
// the power-off is CANCELLED — the box stays on, the gate is released, the log says why. Better a box that did not
|
||||
// switch off than a backup, update or format cut in the middle.
|
||||
func (s *Server) waitJobsClear(base context.Context, vmid int, when string) bool {
|
||||
c := s.powerOff
|
||||
ctx, cancel := context.WithTimeout(base, c.JobWait)
|
||||
defer cancel()
|
||||
for {
|
||||
// Another guest or VM started after acceptance (security review 2026-10-13): no waiting — cancel at once.
|
||||
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
|
||||
s.logger.Error("host-poweroff: CANCELLED — another guest is running now; the box stays on", "when", when,
|
||||
"reason", code, "detail", detail)
|
||||
return false
|
||||
}
|
||||
code, detail := s.jobBlocker(ctx, vmid)
|
||||
if code == "" {
|
||||
return true
|
||||
}
|
||||
s.logger.Warn("host-poweroff: waiting — a job started after the request", "when", when, "reason", code, "detail", detail)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
s.logger.Error("host-poweroff: CANCELLED — still busy after the wait; the box stays on", "when", when,
|
||||
"reason", code, "detail", detail, "waited", c.JobWait)
|
||||
return false
|
||||
case <-time.After(c.JobPoll):
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,353 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// fakePower records the power-off steps in order.
|
||||
type fakePower struct {
|
||||
mu sync.Mutex
|
||||
steps []string
|
||||
}
|
||||
|
||||
func (f *fakePower) ShutdownGuest(_ context.Context, vmid int, timeout time.Duration) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.steps = append(f.steps, "shutdown:"+itoaT(vmid)+":"+timeout.String())
|
||||
return nil
|
||||
}
|
||||
func (f *fakePower) PowerOffHost(context.Context) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.steps = append(f.steps, "poweroff")
|
||||
return nil
|
||||
}
|
||||
func (f *fakePower) got() []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]string(nil), f.steps...)
|
||||
}
|
||||
|
||||
func itoaT(n int) string { b, _ := json.Marshal(n); return string(b) }
|
||||
|
||||
type fakeVzdump struct{ running bool }
|
||||
|
||||
func (f fakeVzdump) Guests(context.Context) ([]proxmox.Guest, error) { return nil, nil }
|
||||
func (f fakeVzdump) Lock(context.Context, int) (string, bool, error) { return "", true, nil }
|
||||
func (f fakeVzdump) BackupRunning(context.Context, int) (bool, error) { return f.running, nil }
|
||||
func (f fakeVzdump) Unlock(context.Context, int) error { return nil }
|
||||
func (f fakeVzdump) Start(context.Context, int) error { return nil }
|
||||
func (f fakeVzdump) HasVzdumpSnapshot(context.Context, int) (bool, error) { return false, nil }
|
||||
func (f fakeVzdump) DeleteVzdumpSnapshot(context.Context, int) error { return nil }
|
||||
|
||||
type powerRig struct {
|
||||
srv *Server
|
||||
h http.Handler
|
||||
power *fakePower
|
||||
state *poweroff.State
|
||||
report *[]string
|
||||
}
|
||||
|
||||
func newPowerRig(t *testing.T, uptime time.Duration) *powerRig {
|
||||
t.Helper()
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
srv.inFlight = &backup.InFlight{}
|
||||
srv.staleLock = fakeVzdump{}
|
||||
p := &fakePower{}
|
||||
st := poweroff.New(filepath.Join(t.TempDir(), "poweroff.json"), func() string { return "1700000000" })
|
||||
var reports []string
|
||||
srv.SetHostPowerOff(HostPowerOffConfig{
|
||||
State: st,
|
||||
Power: p,
|
||||
ReportNow: func(context.Context) bool {
|
||||
// The record must already exist when the hub is told — that is the point of telling it.
|
||||
if st.Current() == nil {
|
||||
reports = append(reports, "report-without-record")
|
||||
} else if len(p.got()) != 0 {
|
||||
reports = append(reports, "report-after-"+p.got()[0])
|
||||
} else {
|
||||
reports = append(reports, "report")
|
||||
}
|
||||
return true
|
||||
},
|
||||
Uptime: func() (time.Duration, bool) { return uptime, true },
|
||||
Running: func() ([]int, error) { return []int{8200}, nil },
|
||||
Now: func() time.Time { return testNow },
|
||||
})
|
||||
return &powerRig{srv: srv, h: srv.Handler(), power: p, state: st, report: &reports}
|
||||
}
|
||||
|
||||
func (r *powerRig) waitSteps(t *testing.T, n int) []string {
|
||||
t.Helper()
|
||||
for i := 0; i < 200 && len(r.power.got()) < n; i++ {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
return r.power.got()
|
||||
}
|
||||
|
||||
func reasonOf(t *testing.T, body []byte) string {
|
||||
t.Helper()
|
||||
var resp struct {
|
||||
Data struct {
|
||||
Reason string `json:"reason"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
t.Fatalf("body %s: %v", body, err)
|
||||
}
|
||||
return resp.Data.Reason
|
||||
}
|
||||
|
||||
// The whole path: 202, the record written, the hub told FIRST, then guest shutdown (120 s), then host off.
|
||||
func TestHostPowerOff_AcceptedOrder(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusAccepted {
|
||||
t.Fatalf("got %d want 202 (%s)", w.Code, w.Body.String())
|
||||
}
|
||||
steps := r.waitSteps(t, 2)
|
||||
want := []string{"shutdown:8200:2m0s", "poweroff"}
|
||||
if len(steps) != 2 || steps[0] != want[0] || steps[1] != want[1] {
|
||||
t.Fatalf("steps = %v, want %v", steps, want)
|
||||
}
|
||||
// The heavy-operation gate stays taken: no backup can start between the checks and the power-off.
|
||||
if _, busy, ok := r.srv.inFlight.TryAcquire("backup:primary"); ok || busy != "host-poweroff" {
|
||||
t.Fatalf("after acceptance a backup could take the gate (ok=%v busy=%q)", ok, busy)
|
||||
}
|
||||
if len(*r.report) != 1 || (*r.report)[0] != "report" {
|
||||
t.Fatalf("hub report = %v, want one report carrying the record", *r.report)
|
||||
}
|
||||
if rec := r.state.Current(); rec == nil || rec.VMID != 8200 {
|
||||
t.Fatalf("record = %+v", rec)
|
||||
}
|
||||
}
|
||||
|
||||
// Refused during a backup — the shared heavy-operation gate held by a backup tier.
|
||||
func TestHostPowerOff_RefusedDuringBackup(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
release, _, ok := r.srv.inFlight.TryAcquire("backup:primary")
|
||||
if !ok {
|
||||
t.Fatal("could not hold the gate")
|
||||
}
|
||||
defer release()
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 0 {
|
||||
t.Fatalf("power steps ran during a backup: %v", s)
|
||||
}
|
||||
if r.state.Current() != nil {
|
||||
t.Fatal("a refused request must not use up the boot's one request")
|
||||
}
|
||||
// The same guard names a restore test and an OS step.
|
||||
release()
|
||||
for gate, want := range map[string]string{"restore-test": PowerOffRestoreTest, "os-pve-step": PowerOffOSUpdate} {
|
||||
rel, _, _ := r.srv.inFlight.TryAcquire(gate)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
rel()
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != want {
|
||||
t.Fatalf("%s: got %d %s, want 409 %s", gate, w.Code, w.Body.String(), want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A vzdump of the guest in flight (a Proxmox task, not the agent's own gate) refuses too.
|
||||
func TestHostPowerOff_RefusedDuringVzdump(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
r.srv.staleLock = fakeVzdump{running: true}
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
|
||||
}
|
||||
}
|
||||
|
||||
// One accepted request per boot: the second is refused and nothing runs twice.
|
||||
func TestHostPowerOff_SecondRequestSameBootRefused(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("first: %d", w.Code)
|
||||
}
|
||||
r.waitSteps(t, 2)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffAlreadyAsked {
|
||||
t.Fatalf("second: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffAlreadyAsked)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 2 {
|
||||
t.Fatalf("steps = %v, want exactly one run", s)
|
||||
}
|
||||
}
|
||||
|
||||
// Refused in the first 10 minutes after boot; an unknown uptime refuses too (fail closed).
|
||||
func TestHostPowerOff_TooSoonAfterBoot(t *testing.T) {
|
||||
r := newPowerRig(t, 9*time.Minute)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffTooSoon {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffTooSoon)
|
||||
}
|
||||
r.srv.powerOff.Uptime = func() (time.Duration, bool) { return 0, false }
|
||||
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
|
||||
t.Fatalf("unknown uptime: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
|
||||
}
|
||||
}
|
||||
|
||||
// Not configured → 503; another guest's vmid in the body → 403; no token → 401.
|
||||
func TestHostPowerOff_ScopeAndConfig(t *testing.T) {
|
||||
plain := newTestServer(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
if w := do(t, plain, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("unconfigured: %d", w.Code)
|
||||
}
|
||||
r := newPowerRig(t, time.Hour)
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", `{"vmid":9300}`); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("cross-guest: %d", w.Code)
|
||||
}
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "", ""); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: %d", w.Code)
|
||||
}
|
||||
if r.state.Current() != nil || len(r.power.got()) != 0 {
|
||||
t.Fatal("a refused request changed state")
|
||||
}
|
||||
}
|
||||
|
||||
// Another guest or VM on the host refuses: one controller must not switch off someone else's guest (security
|
||||
// review 2026-10-13; demo-hp runs a scratch guest and the Tester 1 VM). An unreadable process list refuses too.
|
||||
func TestHostPowerOff_RefusedWhileOtherGuestsRun(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
r.srv.powerOff.Running = func() ([]int, error) { return []int{8200, 9202, 341}, nil }
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffOtherGuests {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffOtherGuests)
|
||||
}
|
||||
r.srv.powerOff.Running = func() ([]int, error) { return nil, errors.New("no /proc") }
|
||||
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
|
||||
t.Fatalf("unreadable: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
|
||||
}
|
||||
if len(r.power.got()) != 0 || r.state.Current() != nil {
|
||||
t.Fatal("a refused request changed state")
|
||||
}
|
||||
}
|
||||
|
||||
// Fail closed: an unwired vzdump reader refuses instead of skipping the check.
|
||||
func TestHostPowerOff_FailsClosedWithoutTaskReader(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
r.srv.staleLock = nil
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
|
||||
}
|
||||
}
|
||||
|
||||
// The /proc reader finds containers and VMs by their real command lines (as measured on demo-hp).
|
||||
func TestRunningGuestsFromProc(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
write := func(pid, cmd string) {
|
||||
_ = os.MkdirAll(filepath.Join(dir, pid), 0o755)
|
||||
_ = os.WriteFile(filepath.Join(dir, pid, "cmdline"), []byte(strings.ReplaceAll(cmd, " ", "\x00")+"\x00"), 0o644)
|
||||
}
|
||||
write("1927", "/usr/bin/lxc-start -F -n 9201")
|
||||
write("1900", "/usr/bin/lxc-start -F -n 9202")
|
||||
write("1823", "/usr/bin/kvm -id 341 -name night1004-tester1 -no-shutdown")
|
||||
write("974", "/usr/libexec/lxc/lxc-monitord --daemon")
|
||||
write("50", "/usr/bin/bash -c echo -n 5")
|
||||
_ = os.MkdirAll(filepath.Join(dir, "self"), 0o755)
|
||||
got, err := runningGuestsFromProc(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sort.Ints(got)
|
||||
if len(got) != 3 || got[0] != 341 || got[1] != 9201 || got[2] != 9202 {
|
||||
t.Fatalf("got %v, want [341 9201 9202]", got)
|
||||
}
|
||||
if _, err := runningGuestsFromProc(filepath.Join(dir, "missing")); err == nil {
|
||||
t.Fatal("an unreadable /proc must be an error")
|
||||
}
|
||||
}
|
||||
|
||||
// switchVzdump is a vzdump reader the test can flip while the power-off runs.
|
||||
type switchVzdump struct {
|
||||
fakeVzdump
|
||||
on *atomic.Bool
|
||||
}
|
||||
|
||||
func (f switchVzdump) BackupRunning(context.Context, int) (bool, error) { return f.on.Load(), nil }
|
||||
|
||||
// A job that starts AFTER acceptance (security review 2026-10-13): the power-off waits; if the job is still running
|
||||
// when the wait ends, the power-off is cancelled — no guest shutdown, no host power-off, the gate given back.
|
||||
func TestHostPowerOff_JobStartedAfterAcceptance(t *testing.T) {
|
||||
// (a) it clears during the wait → the power-off goes ahead after it
|
||||
r := newPowerRig(t, time.Hour)
|
||||
var on atomic.Bool
|
||||
r.srv.staleLock = switchVzdump{on: &on}
|
||||
r.srv.powerOff.JobWait, r.srv.powerOff.JobPoll = 2*time.Second, 2*time.Millisecond
|
||||
r.srv.powerOff.ReportNow = func(context.Context) bool { on.Store(true); return true } // a vzdump starts now
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
time.Sleep(30 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 0 {
|
||||
t.Fatalf("a step ran while the vzdump ran: %v", s)
|
||||
}
|
||||
on.Store(false)
|
||||
if s := r.waitSteps(t, 2); len(s) != 2 {
|
||||
t.Fatalf("after the vzdump ended the steps = %v, want shutdown + poweroff", s)
|
||||
}
|
||||
|
||||
// (b) it never clears → cancelled, nothing switched off, the heavy gate free again
|
||||
r2 := newPowerRig(t, time.Hour)
|
||||
var on2 atomic.Bool
|
||||
r2.srv.staleLock = switchVzdump{on: &on2}
|
||||
r2.srv.powerOff.JobWait, r2.srv.powerOff.JobPoll = 40*time.Millisecond, 2*time.Millisecond
|
||||
r2.srv.powerOff.ReportNow = func(context.Context) bool { on2.Store(true); return true }
|
||||
if w := do(t, r2.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("accept: %d", w.Code)
|
||||
}
|
||||
time.Sleep(150 * time.Millisecond)
|
||||
if s := r2.power.got(); len(s) != 0 {
|
||||
t.Fatalf("cancelled power-off still ran: %v", s)
|
||||
}
|
||||
if busy := r2.srv.inFlight.Busy(); busy != "" {
|
||||
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
|
||||
}
|
||||
}
|
||||
|
||||
// A guest or VM started AFTER acceptance cancels the power-off at once (security review 2026-10-13).
|
||||
func TestHostPowerOff_OtherGuestStartedAfterAcceptance(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
var started atomic.Bool
|
||||
r.srv.powerOff.Running = func() ([]int, error) {
|
||||
if started.Load() {
|
||||
return []int{8200, 341}, nil
|
||||
}
|
||||
return []int{8200}, nil
|
||||
}
|
||||
r.srv.powerOff.ReportNow = func(context.Context) bool { started.Store(true); return true } // a VM starts now
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 0 {
|
||||
t.Fatalf("power-off ran although a VM started: %v", s)
|
||||
}
|
||||
if busy := r.srv.inFlight.Busy(); busy != "" {
|
||||
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
|
||||
}
|
||||
}
|
||||
@@ -313,10 +313,11 @@ type Server struct {
|
||||
// offsite repository password. OPTIONAL — nil (no hub client configured) makes
|
||||
// POST /escrow/recover-offsite-password answer 503 rather than pretending.
|
||||
escrowRecovery EscrowRecoverer
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||
partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional)
|
||||
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
|
||||
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
|
||||
guestPower map[int]guestPowerState
|
||||
@@ -394,6 +395,10 @@ type Server struct {
|
||||
// detached pipeline runs through (tests inject; production defaults set in NewServer).
|
||||
netVerifyMu sync.Mutex
|
||||
netVerifyCur *netVerifyJob
|
||||
|
||||
// R-998: POST /host/poweroff (nil = not configured → 503); one guard-and-mark decision at a time.
|
||||
powerOff *HostPowerOffConfig
|
||||
powerOffMu sync.Mutex
|
||||
// netTrigger performs the mount-waking directory read through the automount path (Q1).
|
||||
netTrigger func(where string) error
|
||||
// netMounted judges mount success from /proc/mounts — the §8 truth source (never readability).
|
||||
@@ -550,6 +555,8 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
|
||||
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
|
||||
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
|
||||
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
|
||||
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
|
||||
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
|
||||
// (live cgroup apply, no reboot). Self-scoped; the agent enforces every bound fresh per request.
|
||||
mux.HandleFunc("GET /guest/memory", s.withGuest(s.handleGuestMemory))
|
||||
@@ -825,6 +832,10 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
return
|
||||
}
|
||||
key := backupJobKey{vmid: vmid, target: tier.TargetID}
|
||||
// R-899 (operator ruling 2026-10-08): a household press („Mentés most") is not the night's backup. A controller
|
||||
// that knows sends `trigger=manual`; then the OS leg does not follow (it belongs to the night, after the night's
|
||||
// own copy). An older controller sends nothing and keeps the old behaviour.
|
||||
manual := r.URL.Query().Get("trigger") == "manual"
|
||||
|
||||
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
|
||||
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
|
||||
@@ -926,7 +937,9 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
}
|
||||
s.finishJob(key, jobID, b)
|
||||
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
|
||||
if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
|
||||
if b.Success && tier.Primary && s.afterPrimaryBackup != nil && manual {
|
||||
s.logger.Info("local-api: no OS leg after a manual backup — it follows the night's own backup (R-899)", "vmid", vmid, "job", jobID)
|
||||
} else if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
|
||||
s.afterPrimaryBackup(base, vmid)
|
||||
}
|
||||
}()
|
||||
|
||||
@@ -4,7 +4,11 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
@@ -142,6 +146,10 @@ type staleLockController struct {
|
||||
runner proxmox.Runner
|
||||
pool string // ownership registry: only members of this PVE pool are ever scanned (A1)
|
||||
logger *slog.Logger // the one success-path scan-summary line; nil = silent
|
||||
// procDir is where taskAlive reads /proc/<pid>/stat (R-1008); "" = /proc. Tests point it at a fixture.
|
||||
procDir string
|
||||
deadMu sync.Mutex
|
||||
deadSeen map[string]bool // UPIDs of dead tasks already logged (once per task)
|
||||
}
|
||||
|
||||
// staleLockAPI is the subset of *proxmox.Client the controller uses (kept narrow for clarity/testing).
|
||||
@@ -209,19 +217,100 @@ func (c *staleLockController) Lock(ctx context.Context, vmid int) (string, bool,
|
||||
return cfg.Lock(), cfg.OnBoot(), nil
|
||||
}
|
||||
|
||||
// BackupRunning is THE answer to „is a vzdump in flight for vmid" for all four callers (startup stale-lock
|
||||
// recovery, the guest-power watchdog, the household power-off guard, the controller supervisor): every one of
|
||||
// them reaches this method through the StaleLockController interface. R-1008: Proxmox keeps a task in
|
||||
// `tasks?source=active` after a power cut — nobody is left to end it (measured on demo-hp 2026-10-14: still
|
||||
// listed RUNNING ~30–60 s after the boot) — so a listed task whose worker process is gone is dead, not in flight.
|
||||
func (c *staleLockController) BackupRunning(ctx context.Context, vmid int) (bool, error) {
|
||||
tasks, err := c.px.ListRunningTasks(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
proc := c.procDir
|
||||
if proc == "" {
|
||||
proc = "/proc"
|
||||
}
|
||||
running, dead := vzdumpInFlight(tasks, vmid, proc)
|
||||
for _, t := range dead {
|
||||
c.logDeadOnce(t, vmid)
|
||||
}
|
||||
return running, nil
|
||||
}
|
||||
|
||||
// vzdumpInFlight decides, from the node's active-task list, whether a vzdump for vmid is in flight: a matching
|
||||
// task is in flight while its worker process lives (taskAlive). A task whose worker is gone is dead (returned in dead,
|
||||
// for the log). FAIL SAFE on doubt — the old answer: a UPID that does not parse, or a /proc entry that cannot be read,
|
||||
// counts as in flight. Pure over procDir; pinned by stalelock_inflight_test.go.
|
||||
func vzdumpInFlight(tasks []proxmox.TaskStatus, vmid int, procDir string) (bool, []proxmox.TaskStatus) {
|
||||
id := strconv.Itoa(vmid)
|
||||
var dead []proxmox.TaskStatus
|
||||
for _, t := range tasks {
|
||||
// A single-guest vzdump task carries the vmid in its ID field.
|
||||
if t.Type == "vzdump" && t.ID == id {
|
||||
if t.Type != "vzdump" || t.ID != id {
|
||||
continue
|
||||
}
|
||||
alive, known := taskAlive(procDir, t.UPID)
|
||||
if !known || alive {
|
||||
return true, nil
|
||||
}
|
||||
dead = append(dead, t)
|
||||
}
|
||||
return false, nil
|
||||
return false, dead
|
||||
}
|
||||
|
||||
// taskAlive is Proxmox's own liveness rule for a task (PVE::ProcFSTools::check_process_running): the UPID names the
|
||||
// worker's pid and its start time in clock ticks since boot (pstart); the task lives while /proc/<pid>/stat exists with
|
||||
// that same start time (field 22). No wall clock is involved — a clock step after the boot (an NTP correction after a
|
||||
// power cut) cannot make a live task look dead (the 2026-10-14 security review of the first, btime-based version).
|
||||
// known=false: the UPID does not parse, or /proc/<pid>/stat exists but cannot be read or parsed.
|
||||
func taskAlive(procDir, upid string) (alive, known bool) {
|
||||
u, err := proxmox.ParseUPID(upid)
|
||||
if err != nil || u.PID == 0 {
|
||||
return false, false
|
||||
}
|
||||
raw, err := os.ReadFile(filepath.Join(procDir, strconv.FormatUint(u.PID, 10), "stat"))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
// „no such process" is only an answer if this agent can see other users' processes at all (a /proc mounted
|
||||
// hidepid=, or a unit with ProtectProc=, hides root's workers — every task would then look dead). PID 1 is
|
||||
// root's; not seeing it means not knowing.
|
||||
if _, err1 := os.Stat(filepath.Join(procDir, "1", "stat")); err1 != nil {
|
||||
return false, false
|
||||
}
|
||||
return false, true // no such process in this boot: the worker is gone
|
||||
}
|
||||
return false, false
|
||||
}
|
||||
st := string(raw)
|
||||
i := strings.LastIndexByte(st, ')') // the comm field may hold spaces and parentheses
|
||||
if i < 0 {
|
||||
return false, false
|
||||
}
|
||||
f := strings.Fields(st[i+1:]) // f[0] is field 3 (state), so field 22 (starttime) is f[19]
|
||||
if len(f) < 20 {
|
||||
return false, false
|
||||
}
|
||||
start, err := strconv.ParseUint(f[19], 10, 64)
|
||||
if err != nil {
|
||||
return false, false
|
||||
}
|
||||
// Proxmox's rule exactly: the start tick matches (a pstart of 0 is not checked) and the process is not a zombie.
|
||||
return (u.PStart == 0 || start == u.PStart) && f[0] != "Z", true
|
||||
}
|
||||
|
||||
func (c *staleLockController) logDeadOnce(t proxmox.TaskStatus, vmid int) {
|
||||
c.deadMu.Lock()
|
||||
defer c.deadMu.Unlock()
|
||||
if c.deadSeen == nil {
|
||||
c.deadSeen = map[string]bool{}
|
||||
}
|
||||
if c.deadSeen[t.UPID] || c.logger == nil {
|
||||
return
|
||||
}
|
||||
c.deadSeen[t.UPID] = true
|
||||
c.logger.Warn("backup-in-flight: a vzdump task is listed as active but its worker process is gone — it is dead, not running",
|
||||
"vmid", vmid, "upid", t.UPID)
|
||||
}
|
||||
|
||||
func (c *staleLockController) HasVzdumpSnapshot(ctx context.Context, vmid int) (bool, error) {
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1008: after a power cut Proxmox keeps the dead vzdump task in `tasks?source=active` (measured on demo-hp
|
||||
// 2026-10-14: still RUNNING ~30–60 s after the boot). A task whose worker process is gone is dead, not in flight.
|
||||
// Liveness is Proxmox's own rule (pid + pstart vs /proc/<pid>/stat field 22), never the wall clock: the first version
|
||||
// compared start time with btime and a clock step after the boot could make a live task look dead (security review
|
||||
// 2026-10-14). These tests pin the helper, the production controller, three callers end to end; the fourth (the
|
||||
// controller supervisor) by the structural pin at the bottom.
|
||||
|
||||
// A worker: pid 0x1E6EEC = 1994476, pstart 0x1B4A0D = 1788429 (UPID hex fields).
|
||||
const (
|
||||
testPID = 1994476
|
||||
testPStart = 1788429
|
||||
)
|
||||
|
||||
func vz(vmid string) proxmox.TaskStatus {
|
||||
return proxmox.TaskStatus{UPID: "UPID:demo-hp:001E6EEC:001B4A0D:6ACF4071:vzdump:" + vmid + ":felhom-agent@pve!agent:",
|
||||
Type: "vzdump", ID: vmid, Status: "RUNNING", StartTime: 1791967345}
|
||||
}
|
||||
|
||||
// procFixture writes <dir>/<pid>/stat with the given start ticks (comm with a space and a parenthesis, as real ones can).
|
||||
func procFixture(t *testing.T, pid int, start int) string {
|
||||
return procFixtureState(t, pid, start, "S")
|
||||
}
|
||||
|
||||
func procFixtureState(t *testing.T, pid int, start int, state string) string {
|
||||
t.Helper()
|
||||
d := t.TempDir()
|
||||
// PID 1 is always visible on a real host; procFixtureHidden leaves it out
|
||||
if err := os.MkdirAll(filepath.Join(d, "1"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(d, "1", "stat"), []byte("1 (systemd) S 0 1 1 0 -1 0 0 0 0 0 0 0 0 0 20 0 1 0 2 0 0\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pid > 0 {
|
||||
p := filepath.Join(d, itoaT(pid))
|
||||
if err := os.MkdirAll(p, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f := []string{itoaT(pid), "(task UPID:x (vzdump))", state}
|
||||
for i := 4; i <= 21; i++ {
|
||||
f = append(f, "0")
|
||||
}
|
||||
f = append(f, itoaT(start), "0", "0")
|
||||
if err := os.WriteFile(filepath.Join(p, "stat"), []byte(strings.Join(f, " ")+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// RED-PROOF: make taskAlive return (true, true) for a missing /proc entry → the dead task runs → fails.
|
||||
func TestVzdumpInFlight_WorkerGoneIsDead(t *testing.T) {
|
||||
running, dead := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, 0, 0))
|
||||
if running || len(dead) != 1 {
|
||||
t.Fatalf("a task whose worker is gone must be dead: running=%v dead=%d", running, len(dead))
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the `start == u.PStart` comparison (alive whenever the pid exists) → a reused pid runs → fails.
|
||||
func TestVzdumpInFlight_ReusedPidIsDead(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, testPID, 42)); running {
|
||||
t.Fatal("the pid exists but started at another tick: another process, the task is dead")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the `f[0] != "Z"` test → a zombie worker counts as running → fails.
|
||||
func TestVzdumpInFlight_ZombieIsDead(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixtureState(t, testPID, testPStart, "Z")); running {
|
||||
t.Fatal("a zombie worker is not running (Proxmox's own rule)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVzdumpInFlight_LiveWorkerRuns(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, testPID, testPStart)); !running {
|
||||
t.Fatal("the worker lives (same pid, same start tick): in flight")
|
||||
}
|
||||
}
|
||||
|
||||
// The reason for the rewrite: the wall clock plays no part. A live task whose list start time lies „before the boot"
|
||||
// (a clock step after the boot) is still running.
|
||||
func TestVzdumpInFlight_ClockStepCannotKillALiveTask(t *testing.T) {
|
||||
tk := vz("9201")
|
||||
tk.StartTime = 1 // absurdly early by the wall clock
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{tk}, 9201, procFixture(t, testPID, testPStart)); !running {
|
||||
t.Fatal("a live worker must count as running whatever its wall-clock start says")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: make an unparsable UPID or unreadable stat „known dead" → fails.
|
||||
func TestVzdumpInFlight_DoubtFailsSafe(t *testing.T) {
|
||||
bad := vz("9201")
|
||||
bad.UPID = "garbage"
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{bad}, 9201, procFixture(t, 0, 0)); !running {
|
||||
t.Fatal("an unparsable UPID must count as running (fail safe)")
|
||||
}
|
||||
d := procFixture(t, testPID, testPStart)
|
||||
if err := os.WriteFile(filepath.Join(d, itoaT(testPID), "stat"), []byte("garbage"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, d); !running {
|
||||
t.Fatal("an unparsable stat must count as running (fail safe)")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the PID-1 visibility check → a hidden /proc makes the live task „gone" → fails.
|
||||
func TestVzdumpInFlight_HiddenProcFailsSafe(t *testing.T) {
|
||||
d := t.TempDir() // nothing visible: not even PID 1 (hidepid / ProtectProc)
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, d); !running {
|
||||
t.Fatal("a /proc that hides other users' processes must count as running (fail safe)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVzdumpInFlight_OtherGuestOrTypeIgnored(t *testing.T) {
|
||||
tasks := []proxmox.TaskStatus{vz("9202"), {UPID: vz("9201").UPID, Type: "vzsnapshot", ID: "9201"}}
|
||||
if running, dead := vzdumpInFlight(tasks, 9201, procFixture(t, testPID, testPStart)); running || len(dead) != 0 {
|
||||
t.Fatalf("other guests' and other types' tasks are not 9201's backup: running=%v dead=%d", running, len(dead))
|
||||
}
|
||||
}
|
||||
|
||||
func inflightCtrl(api *fakeStaleLockAPI, runner *recordingRunner, logger *slog.Logger, proc string) *staleLockController {
|
||||
c := NewStaleLockController(api, runner, "felhom", logger).(*staleLockController)
|
||||
c.procDir = proc
|
||||
return c
|
||||
}
|
||||
|
||||
// The production controller answers through the helper, and logs a dead task ONCE.
|
||||
// RED-PROOF: make BackupRunning ignore the helper (any listed task runs) → fails.
|
||||
func TestStaleLockController_BackupRunningIgnoresDeadTask(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")}}
|
||||
c := inflightCtrl(api, &recordingRunner{}, slog.New(slog.NewTextHandler(&buf, nil)), procFixture(t, 0, 0))
|
||||
for i := 0; i < 3; i++ {
|
||||
if running, err := c.BackupRunning(context.Background(), 9201); err != nil || running {
|
||||
t.Fatalf("dead task reported running=%v err=%v", running, err)
|
||||
}
|
||||
}
|
||||
if n := strings.Count(buf.String(), "backup-in-flight: a vzdump task is listed as active but its worker process is gone"); n != 1 {
|
||||
t.Fatalf("the dead task must be logged exactly once, got %d", n)
|
||||
}
|
||||
c.procDir = procFixture(t, testPID, testPStart)
|
||||
if running, _ := c.BackupRunning(context.Background(), 9201); !running {
|
||||
t.Fatal("a live worker must be running")
|
||||
}
|
||||
}
|
||||
|
||||
func fixtureFor(t *testing.T, alive bool) string {
|
||||
if alive {
|
||||
return procFixture(t, testPID, testPStart)
|
||||
}
|
||||
return procFixture(t, 0, 0)
|
||||
}
|
||||
|
||||
func poolOf(vmids ...int) proxmox.PoolInfo {
|
||||
var p proxmox.PoolInfo
|
||||
raw := `{"members":[`
|
||||
for i, v := range vmids {
|
||||
if i > 0 {
|
||||
raw += ","
|
||||
}
|
||||
raw += `{"vmid":` + itoaT(v) + `,"type":"lxc"}`
|
||||
}
|
||||
raw += `]}`
|
||||
_ = json.Unmarshal([]byte(raw), &p)
|
||||
return p
|
||||
}
|
||||
|
||||
// Caller 1 — the startup stale-lock recovery clears the lock a power cut left (the night of 2026-10-14 it said
|
||||
// „genuinely in-flight — leaving the lock").
|
||||
func TestCaller_StartupStaleLockClearsAfterPowerCut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
alive bool
|
||||
cleared bool
|
||||
}{{false, true}, {true, false}} {
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")},
|
||||
cfg: map[int]proxmox.GuestConfig{9201: staleCfg("backup", true)}}
|
||||
runner := &recordingRunner{}
|
||||
s := &Server{staleLock: inflightCtrl(api, runner, nil, fixtureFor(t, tc.alive)), logger: slog.New(slog.NewTextHandler(discardW{}, nil))}
|
||||
s.recoverOneStaleLock(context.Background(), proxmox.Guest{VMID: 9201, Status: "running"})
|
||||
if runner.unlocked("9201") != tc.cleared {
|
||||
t.Fatalf("worker alive=%v: unlocked=%v, want %v", tc.alive, runner.unlocked("9201"), tc.cleared)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Caller 2 — the guest-power watchdog starts a stopped guest despite a dead pre-boot vzdump task.
|
||||
func TestCaller_GuestPowerStartsGuestAfterPowerCut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
alive bool
|
||||
started bool
|
||||
}{{false, true}, {true, false}} {
|
||||
api := &fakeStaleLockAPI{lxc: []proxmox.Guest{{VMID: 9201, Status: "stopped"}}, pool: poolOf(9201),
|
||||
tasks: []proxmox.TaskStatus{vz("9201")}, cfg: map[int]proxmox.GuestConfig{9201: staleCfg("", true)}}
|
||||
s := gpServer(t, inflightCtrl(api, &recordingRunner{}, nil, fixtureFor(t, tc.alive)), nil)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if (len(api.started) == 1) != tc.started {
|
||||
t.Fatalf("worker alive=%v: started=%v, want %v", tc.alive, api.started, tc.started)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Caller 3 — the household power-off guard does not refuse „backup running" for a dead task.
|
||||
func TestCaller_PowerOffJobBlockerIgnoresDeadTask(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
alive bool
|
||||
want string
|
||||
}{{false, ""}, {true, PowerOffBackup}} {
|
||||
rig := newPowerRig(t, 0)
|
||||
rig.srv.staleLock = inflightCtrl(&fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")}}, &recordingRunner{}, nil, fixtureFor(t, tc.alive))
|
||||
if code, _ := rig.srv.jobBlocker(context.Background(), 9201); code != tc.want {
|
||||
t.Fatalf("worker alive=%v: jobBlocker=%q, want %q", tc.alive, code, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Caller 4 (the supervisor), the R-1006 cleanup (VzdumpInFlight) and the rest — structural pin: every non-test BackupRunning call in this package goes through
|
||||
// s.staleLock (whose one production implementation is staleLockController, which uses vzdumpInFlight), and no
|
||||
// other production type implements BackupRunning. If someone adds a second path, this fails.
|
||||
func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
|
||||
files, _ := filepath.Glob("*.go")
|
||||
call := regexp.MustCompile(`([A-Za-z_.]+)\.BackupRunning\(`)
|
||||
impl := regexp.MustCompile(`func \([a-z]+ \*?([A-Za-z]+)\) BackupRunning\(`)
|
||||
callers := 0
|
||||
for _, f := range files {
|
||||
if strings.HasSuffix(f, "_test.go") {
|
||||
continue
|
||||
}
|
||||
b, err := os.ReadFile(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, m := range call.FindAllStringSubmatch(string(b), -1) {
|
||||
if m[1] != "s.staleLock" {
|
||||
t.Errorf("%s: BackupRunning called on %q, not s.staleLock", f, m[1])
|
||||
}
|
||||
callers++
|
||||
}
|
||||
for _, m := range impl.FindAllStringSubmatch(string(b), -1) {
|
||||
if m[1] != "staleLockController" {
|
||||
t.Errorf("%s: a second production BackupRunning on %s", f, m[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
if callers != 5 {
|
||||
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
|
||||
}
|
||||
b, _ := os.ReadFile("stalelock.go")
|
||||
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
|
||||
t.Error("staleLockController.BackupRunning no longer answers through vzdumpInFlight")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package localapi
|
||||
|
||||
import "context"
|
||||
|
||||
// R-1006: the half-written vzdump cleanup's two needs from the Server, and its start-up sweep. The cleaner itself is
|
||||
// internal/vzpartial; the root helper is configs/felhom-vzdump-partial-clean.
|
||||
|
||||
// VzdumpInFlight is the one in-flight answer (staleLockController.BackupRunning, R-1008) for the cleanup. No stale-lock
|
||||
// controller wired → true: fail safe, nothing is removed.
|
||||
func (s *Server) VzdumpInFlight(ctx context.Context, vmid int) (bool, error) {
|
||||
if s.staleLock == nil {
|
||||
return true, nil
|
||||
}
|
||||
return s.staleLock.BackupRunning(ctx, vmid)
|
||||
}
|
||||
|
||||
// SetPartialSweep sets the start-up sweep: called with every owned guest (the felhom pool, ownership proven).
|
||||
func (s *Server) SetPartialSweep(fn func(ctx context.Context, vmid int)) { s.partialSweep = fn }
|
||||
|
||||
// SweepVzdumpPartials runs the sweep once for every owned guest. No-op when unwired.
|
||||
func (s *Server) SweepVzdumpPartials(ctx context.Context) {
|
||||
if s.partialSweep == nil || s.staleLock == nil {
|
||||
return
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
s.logger.Warn("vzdump-partial: guest list unavailable — skipping the start-up sweep", "err", err)
|
||||
return
|
||||
}
|
||||
for _, g := range guests {
|
||||
s.partialSweep(ctx, g.VMID)
|
||||
}
|
||||
}
|
||||
@@ -33,7 +33,7 @@ const OpKernelStep = "os_kernel_step"
|
||||
|
||||
// DefaultKernelJudgeWait is how long a one-shot boot may take to come back healthy before the agent reverts it ONCE.
|
||||
// Measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`): every container healthy 68 s after a reboot on demo-felhom,
|
||||
// 272 s on demo-hp; 20 minutes stays under the hub's 30-minute host_stale (a box that never comes back alarms after it).
|
||||
// 272 s on demo-hp; 20 minutes stays under the hub's 45-minute host_stale (a box that never comes back alarms after it).
|
||||
const DefaultKernelJudgeWait = 20 * time.Minute
|
||||
|
||||
var kverRE = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$`)
|
||||
@@ -51,6 +51,24 @@ type KernelView struct {
|
||||
VMID int `json:"vmid"` // the customer guest the step was staged for (the health rule's guest)
|
||||
}
|
||||
|
||||
// kernelReportRing is the ring an after-boot report carries. In the first second after a boot the agent has not
|
||||
// fetched the hub's block yet, and Block() then answers ring 1 — so a ring-0 box's „judging" report said ring 1
|
||||
// (seen on demo-felhom, 2026-10-08 night, `audits/kernel-night-2026-10-07/readback/`). Order: the fetched block; the
|
||||
// block the daemon saved on disk before the reboot (R-866); ring 1 as before. A label only: the hub's approval reads its
|
||||
// own ring list. Pinned by TestKernelReportRing_BeforeFirstFetch.
|
||||
func (l *Leg) kernelReportRing() int {
|
||||
l.mu.Lock()
|
||||
fetched := l.block
|
||||
l.mu.Unlock()
|
||||
if fetched != nil {
|
||||
return fetched.Ring
|
||||
}
|
||||
if b, _, ok := LoadSavedBlock(l.planDir()); ok && b != nil {
|
||||
return b.Ring
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
func parseKernel(raw json.RawMessage) KernelView {
|
||||
var v KernelView
|
||||
_ = json.Unmarshal(raw, &v)
|
||||
@@ -119,8 +137,12 @@ func (l *Leg) runKernel(ctx context.Context, runID string, vmid int, trigger str
|
||||
lg.Info("osupdate: kernel step skipped — ring 1 boots only a kernel a signed os_kernel_step staged", "phase", st.Phase, "staged", st.To, "want", want)
|
||||
return Report{}
|
||||
default:
|
||||
// R-898: EXACTLY the kernel the household was told about — never "whatever is pending tonight" (the sources can
|
||||
// offer a newer one by night; the step then refused, R23, and the night was lost). A version no longer
|
||||
// installable is refused by the wrapper before any change (R7) and the hub tells the household again.
|
||||
wr, cerr := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": "ring0-" + runID,
|
||||
"select": "pending-kernel", "expect_kver": want, "run_id": runID, "trigger": trigger, "ring": blk.Ring}))
|
||||
"select": "listed", "packages": KernelSet(want), "expect_kver": want, "run_id": runID, "trigger": trigger,
|
||||
"ring": blk.Ring}))
|
||||
rep.unsent = reportFile(l.planDir(), runID, LayerKernel, "apply")
|
||||
rep.Kernel = rawOrNil(wr.Kernel)
|
||||
switch {
|
||||
@@ -186,7 +208,7 @@ func (l *Leg) KernelAfterBoot(ctx context.Context, vmid int, j KernelJudge) Repo
|
||||
if vmid <= 0 {
|
||||
vmid = v.VMID // after a boot the guest may not run yet — the step's own record names it
|
||||
}
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, Mode: "kernel-boot",
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-boot",
|
||||
ReleaseID: v.To, Kernel: rawOrNil(wr.Kernel)}
|
||||
switch wr.KernelEvent {
|
||||
case "fell_back":
|
||||
@@ -236,7 +258,7 @@ func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelV
|
||||
for {
|
||||
if !hubReached && l.Hub != nil {
|
||||
// the hub's reachability IS this report reaching it (and the operator sees the box is back on the new kernel)
|
||||
body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid,
|
||||
body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||
Mode: "kernel-boot", ReleaseID: v.To, Outcome: "judging", Kernel: mustRaw(v)})
|
||||
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
if err := l.Hub.PostOSReport(rctx, body); err == nil {
|
||||
@@ -276,7 +298,7 @@ func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelV
|
||||
return Report{}
|
||||
}
|
||||
// not healthy by the deadline: tell the hub (best effort), then ONE self-revert into the old kernel
|
||||
rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid,
|
||||
rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "health_failed", HealthReason: why + " — reverting to " + v.From,
|
||||
Kernel: mustRaw(v)})
|
||||
lg.Error("osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel", "reason", why,
|
||||
@@ -285,7 +307,7 @@ func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelV
|
||||
if err != nil || wr.refused() || wr.failed() {
|
||||
lg.Error("osupdate: kernel self-revert did not start — the box stays on the new kernel; the operator decides",
|
||||
"err", err, "refused", string(firstRaw(wr.Refused, wr.Failed)))
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid,
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "revert_failed", Refused: firstRaw(wr.Refused, wr.Failed),
|
||||
HealthReason: "the self-revert did not start"})
|
||||
}
|
||||
@@ -294,7 +316,7 @@ func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelV
|
||||
|
||||
func (l *Leg) kernelGood(ctx context.Context, runID string, vmid int, v KernelView, start time.Time, lg *slog.Logger) Report {
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-good", vmid, nil))
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, Mode: "kernel-good",
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-good",
|
||||
ReleaseID: v.To}
|
||||
switch {
|
||||
case err != nil:
|
||||
@@ -322,6 +344,21 @@ func truncate(s string, n int) string {
|
||||
return s[:n]
|
||||
}
|
||||
|
||||
// KernelSet is the package set that installs exactly kver (R-898; the hub's kernelSet, field-exact): the series
|
||||
// meta-package and the signed image, both at the kernel's own version. Proxmox keeps old kernel versions in its archive.
|
||||
// nil for a string that is not a kernel version.
|
||||
func KernelSet(kver string) []Package {
|
||||
m := kverSeriesRE.FindStringSubmatch(kver)
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
v := kver[:len(kver)-len("-pve")]
|
||||
return []Package{{Name: "proxmox-kernel-" + m[1], Version: v, Origin: PVEOrigin},
|
||||
{Name: "proxmox-kernel-" + kver + "-signed", Version: v, Origin: PVEOrigin}}
|
||||
}
|
||||
|
||||
var kverSeriesRE = regexp.MustCompile(`^([0-9]+\.[0-9]+)\.[0-9]+-[0-9]+-pve$`)
|
||||
|
||||
// KernelStepParams are a signed os_kernel_step's params: the exact kernel set (the wrapper compares it with the plan).
|
||||
type KernelStepParams struct {
|
||||
ReleaseID string `json:"release_id"`
|
||||
|
||||
@@ -54,8 +54,12 @@ func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
|
||||
ap = x
|
||||
}
|
||||
}
|
||||
if ap["select"] != "pending-kernel" || ap["expect_kver"] != kNew || ap["lane"] != "slow" {
|
||||
t.Fatalf("stage plan = %v", ap)
|
||||
// R-898: EXACTLY the told kernel — the listed set derived from it, never "pending" (red before the fix: select was
|
||||
// pending-kernel, so a newer kernel in the sources by night was refused R23 and the night was lost)
|
||||
pk, _ := json.Marshal(ap["packages"])
|
||||
if ap["select"] != "listed" || ap["expect_kver"] != kNew || ap["lane"] != "slow" ||
|
||||
string(pk) != `[{"name":"proxmox-kernel-7.0","origin":"Proxmox Debian Repository","version":"7.0.14-22"},{"name":"proxmox-kernel-7.0.14-22-pve-signed","origin":"Proxmox Debian Repository","version":"7.0.14-22"}]` {
|
||||
t.Fatalf("stage plan = %v (packages %s)", ap, pk)
|
||||
}
|
||||
if p.Kernel.Outcome != "staged" || !p.Kernel.Healthy {
|
||||
t.Fatalf("kernel report = %+v", p.Kernel)
|
||||
@@ -312,3 +316,13 @@ func TestKernel_KeptStageReportIsStaged(t *testing.T) {
|
||||
t.Fatalf("kept = %+v", rep)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelSet(t *testing.T) {
|
||||
if got := KernelSet("7.0.14-20-pve"); len(got) != 2 || got[0].Name != "proxmox-kernel-7.0" || got[0].Version != "7.0.14-20" ||
|
||||
got[1].Name != "proxmox-kernel-7.0.14-20-pve-signed" {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
if KernelSet("7.0; reboot") != nil || KernelSet("") != nil {
|
||||
t.Fatal("a non-kernel string must give no set")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// After a boot the agent has not fetched the hub's block yet; the after-boot kernel report must still carry the
|
||||
// box's real ring (seen 2026-10-08: a ring-0 box's „judging" report said ring 1). Red-proof: return Block().Ring
|
||||
// from kernelReportRing and the saved-block case fails with 1.
|
||||
func TestKernelReportRing_BeforeFirstFetch(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
l := &Leg{PlanDir: dir}
|
||||
if got := l.kernelReportRing(); got != 1 {
|
||||
t.Fatalf("nothing fetched, nothing saved: ring %d, want 1 (the old default)", got)
|
||||
}
|
||||
l.saveBlock(&hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
l2 := &Leg{PlanDir: dir} // a fresh daemon after the reboot: no block fetched yet
|
||||
if got := l2.kernelReportRing(); got != 0 {
|
||||
t.Fatalf("ring-0 block saved before the reboot: ring %d, want 0", got)
|
||||
}
|
||||
l2.SetBlock(&hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
if got := l2.kernelReportRing(); got != 1 {
|
||||
t.Fatalf("a fetched block wins over the saved one: ring %d, want 1", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, SavedBlockFile)); err != nil {
|
||||
t.Fatalf("saved block missing: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
// Package poweroff keeps the box's record of a household-requested power-off (R-998, doc 14).
|
||||
//
|
||||
// One small JSON file in the agent's state dir. A record belongs to the boot it was made in (the
|
||||
// kernel's btime from /proc/stat), so:
|
||||
// - "at most one accepted request per boot": Mark refuses when this boot already has a record;
|
||||
// - the host report carries the record only while the box is still in that boot (Current), and
|
||||
// the next boot's reports carry nothing — that absence is how the hub learns the box is back.
|
||||
//
|
||||
// Pinned by state_test.go.
|
||||
package poweroff
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrAlreadyThisBoot is Mark's refusal: a power-off was already accepted in the current boot.
|
||||
var ErrAlreadyThisBoot = errors.New("a power-off was already accepted in this boot")
|
||||
|
||||
// ErrBootUnknown is Mark's refusal when the boot cannot be identified — fail closed: without it the
|
||||
// per-boot limit could not hold.
|
||||
var ErrBootUnknown = errors.New("cannot read the host's boot time")
|
||||
|
||||
// Record is one accepted request.
|
||||
type Record struct {
|
||||
RequestedAt time.Time `json:"requested_at"`
|
||||
BootBtime string `json:"boot_btime"` // /proc/stat btime of the boot it was accepted in
|
||||
VMID int `json:"vmid"` // the guest whose controller asked
|
||||
}
|
||||
|
||||
// State is the persisted record.
|
||||
type State struct {
|
||||
path string
|
||||
btime func() string
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// New returns the state at path. btime returns the current boot's /proc/stat btime ("" = unknown).
|
||||
func New(path string, btime func() string) *State {
|
||||
if btime == nil {
|
||||
btime = ProcBtime
|
||||
}
|
||||
return &State{path: path, btime: btime}
|
||||
}
|
||||
|
||||
// Current is the record made in THIS boot, or nil (none, another boot's, unreadable).
|
||||
func (s *State) Current() *Record {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
r := s.load()
|
||||
b := s.btime()
|
||||
if r == nil || b == "" || r.BootBtime != b {
|
||||
return nil
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// Mark records an accepted request for this boot. It refuses a second one in the same boot.
|
||||
func (s *State) Mark(now time.Time, vmid int) (*Record, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
b := s.btime()
|
||||
if b == "" {
|
||||
return nil, ErrBootUnknown
|
||||
}
|
||||
if r := s.load(); r != nil && r.BootBtime == b {
|
||||
return nil, ErrAlreadyThisBoot
|
||||
}
|
||||
r := &Record{RequestedAt: now.UTC().Truncate(time.Second), BootBtime: b, VMID: vmid}
|
||||
data, _ := json.Marshal(r)
|
||||
if err := os.MkdirAll(filepath.Dir(s.path), 0o750); err != nil {
|
||||
return nil, fmt.Errorf("poweroff: state dir: %w", err)
|
||||
}
|
||||
tmp := s.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||
return nil, fmt.Errorf("poweroff: write: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmp, s.path); err != nil {
|
||||
return nil, fmt.Errorf("poweroff: rename: %w", err)
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (s *State) load() *Record {
|
||||
data, err := os.ReadFile(s.path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var r Record
|
||||
if json.Unmarshal(data, &r) != nil || r.BootBtime == "" {
|
||||
return nil
|
||||
}
|
||||
return &r
|
||||
}
|
||||
|
||||
// ProcBtime reads the boot time (seconds since the epoch, as text) from /proc/stat; "" when unreadable.
|
||||
func ProcBtime() string {
|
||||
data, err := os.ReadFile("/proc/stat")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
for _, line := range strings.Split(string(data), "\n") {
|
||||
if strings.HasPrefix(line, "btime ") {
|
||||
return strings.TrimSpace(strings.TrimPrefix(line, "btime "))
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package poweroff
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// One accepted request per boot; the record is "current" only in the boot it was made in.
|
||||
func TestOnePerBootAndCurrentFollowsTheBoot(t *testing.T) {
|
||||
boot := "1000"
|
||||
s := New(filepath.Join(t.TempDir(), "poweroff.json"), func() string { return boot })
|
||||
if s.Current() != nil {
|
||||
t.Fatal("no record yet, Current must be nil")
|
||||
}
|
||||
now := time.Date(2026, 10, 13, 18, 0, 0, 0, time.UTC)
|
||||
if _, err := s.Mark(now, 9201); err != nil {
|
||||
t.Fatalf("first Mark: %v", err)
|
||||
}
|
||||
if r := s.Current(); r == nil || !r.RequestedAt.Equal(now) || r.VMID != 9201 {
|
||||
t.Fatalf("Current after Mark = %+v", r)
|
||||
}
|
||||
if _, err := s.Mark(now.Add(time.Minute), 9201); !errors.Is(err, ErrAlreadyThisBoot) {
|
||||
t.Fatalf("second Mark in the same boot: err = %v, want ErrAlreadyThisBoot", err)
|
||||
}
|
||||
boot = "2000" // the machine was switched on again
|
||||
if s.Current() != nil {
|
||||
t.Fatal("a record from the previous boot must not be current")
|
||||
}
|
||||
if _, err := s.Mark(now.Add(time.Hour), 9201); err != nil {
|
||||
t.Fatalf("Mark in a new boot: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownBootFailsClosed(t *testing.T) {
|
||||
s := New(filepath.Join(t.TempDir(), "p.json"), func() string { return "" })
|
||||
if _, err := s.Mark(time.Now(), 1); !errors.Is(err, ErrBootUnknown) {
|
||||
t.Fatalf("err = %v, want ErrBootUnknown", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,208 @@
|
||||
// Package vzpartial removes a dead, half-written vzdump backup of a guest from a local dir storage (R-1006).
|
||||
//
|
||||
// A power cut during the night's whole-guest backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` and its `.tmp/` directory
|
||||
// on the target (demo-hp 2026-10-14: 12.2 GB on the host root). Nothing removes them: Proxmox does not list them as
|
||||
// backups, every later backup is skipped „not enough space", and a host root that fills up breaks LVM's own metadata
|
||||
// writes (audits/power-cut-fixes-2026-10-14 C4). This package finds such a leftover and asks the root helper
|
||||
// /usr/local/sbin/felhom-vzdump-partial-clean to remove it; the helper re-checks every rule as root.
|
||||
//
|
||||
// Rules here (the helper repeats them, and adds the open-file check the agent cannot do):
|
||||
// - only a `dir` storage; only names of the exact vzdump pattern for that vmid (NamePattern) — never a finished
|
||||
// archive, a log, another guest;
|
||||
// - only when no vzdump for that vmid is in flight (the R-1008 answer: a task from before the boot is not in flight);
|
||||
// - only when dead: last modified before this boot, or untouched for QuietFor.
|
||||
//
|
||||
// Any doubt leaves the entry and logs why. Pinned by vzpartial_test.go.
|
||||
package vzpartial
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// Helper is the root helper's installed path (configs/felhom-vzdump-partial-clean, delivered by the config bundle).
|
||||
const Helper = "/usr/local/sbin/felhom-vzdump-partial-clean"
|
||||
|
||||
// QuietFor is how long an entry written in THIS boot must be untouched before it counts as dead (the helper's QUIET_S).
|
||||
const QuietFor = 120 * time.Second
|
||||
|
||||
// NamePattern matches the half-written entries of one guest: the archive being written (.tar.dat / .vma.dat) and the
|
||||
// temporary directory (.tmp). The timestamp is vzdump's own (host local time).
|
||||
func NamePattern(vmid int) *regexp.Regexp {
|
||||
return regexp.MustCompile(`^vzdump-(lxc|qemu)-` + strconv.Itoa(vmid) +
|
||||
`-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$`)
|
||||
}
|
||||
|
||||
// Entry is one leftover found in the dump directory.
|
||||
type Entry struct {
|
||||
Name string
|
||||
Dir bool
|
||||
ModTime time.Time
|
||||
Size int64
|
||||
}
|
||||
|
||||
// Candidates lists the entries of dumpDir that match vmid's half-written pattern. A missing directory is no error.
|
||||
func Candidates(dumpDir string, vmid int) ([]Entry, error) {
|
||||
ents, err := os.ReadDir(dumpDir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
re := NamePattern(vmid)
|
||||
var out []Entry
|
||||
for _, e := range ents {
|
||||
if !re.MatchString(e.Name()) || e.Type()&os.ModeSymlink != 0 {
|
||||
continue
|
||||
}
|
||||
info, err := e.Info()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if strings.HasSuffix(e.Name(), ".tmp") != info.IsDir() {
|
||||
continue // a .tmp that is not a directory, or an archive name that is one: not vzdump's
|
||||
}
|
||||
mt := info.ModTime()
|
||||
if info.IsDir() {
|
||||
mt = newestUnder(filepath.Join(dumpDir, e.Name()), mt)
|
||||
}
|
||||
out = append(out, Entry{Name: e.Name(), Dir: info.IsDir(), ModTime: mt, Size: info.Size()})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func newestUnder(dir string, mt time.Time) time.Time {
|
||||
_ = filepath.WalkDir(dir, func(_ string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
if i, e := d.Info(); e == nil && i.ModTime().After(mt) {
|
||||
mt = i.ModTime()
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return mt
|
||||
}
|
||||
|
||||
// Dead decides whether an entry is dead: written before this boot, or untouched for QuietFor. bootOK=false (the boot
|
||||
// time could not be read) leaves only the quiet rule.
|
||||
func Dead(e Entry, boot time.Time, bootOK bool, now time.Time) (bool, string) {
|
||||
if bootOK && e.ModTime.Before(boot) {
|
||||
return true, "written before this boot"
|
||||
}
|
||||
if age := now.Sub(e.ModTime); age >= QuietFor {
|
||||
return true, "untouched for " + age.Round(time.Second).String()
|
||||
}
|
||||
return false, "modified " + now.Sub(e.ModTime).Round(time.Second).String() + " ago — may still be written"
|
||||
}
|
||||
|
||||
// StorageAPI is the one Proxmox read the cleaner needs (the storage's type and path).
|
||||
type StorageAPI interface {
|
||||
ListStorage(ctx context.Context) ([]proxmox.Storage, error)
|
||||
}
|
||||
|
||||
// Cleaner removes dead leftovers. All fields but Logger and Now are required; a nil Cleaner is a no-op.
|
||||
type Cleaner struct {
|
||||
API StorageAPI
|
||||
Runner proxmox.Runner // the fenced sudo runner
|
||||
InFlight func(ctx context.Context, vmid int) (bool, error)
|
||||
Boot func() (int64, bool)
|
||||
Logger *slog.Logger
|
||||
Now func() time.Time
|
||||
// dumpDir maps a storage path to its dump directory (tests); nil = <path>/dump.
|
||||
dumpDir func(path string) string
|
||||
}
|
||||
|
||||
// Result is one removal or one entry left in place.
|
||||
type Result struct {
|
||||
Entry Entry
|
||||
Removed bool
|
||||
Why string
|
||||
}
|
||||
|
||||
// Clean removes the dead leftovers of vmid on storage. It never returns an error for „nothing to do"; every entry it
|
||||
// finds is logged with what was done and why.
|
||||
func (c *Cleaner) Clean(ctx context.Context, storage string, vmid int) []Result {
|
||||
if c == nil || c.API == nil || c.Runner == nil || c.InFlight == nil || c.Boot == nil {
|
||||
return nil
|
||||
}
|
||||
log := c.Logger
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
now := time.Now
|
||||
if c.Now != nil {
|
||||
now = c.Now
|
||||
}
|
||||
stores, err := c.API.ListStorage(ctx)
|
||||
if err != nil {
|
||||
log.Warn("vzdump-partial: storage list unreadable — not looking for leftovers", "storage", storage, "err", err)
|
||||
return nil
|
||||
}
|
||||
var st *proxmox.Storage
|
||||
for i := range stores {
|
||||
if stores[i].Storage == storage {
|
||||
st = &stores[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if st == nil || st.Type != "dir" || st.Path == "" {
|
||||
return nil // PBS, LVM, NFS …: vzdump does not leave a .tar.dat there that this agent may remove
|
||||
}
|
||||
dump := filepath.Join(st.Path, "dump")
|
||||
if c.dumpDir != nil {
|
||||
dump = c.dumpDir(st.Path)
|
||||
}
|
||||
ents, err := Candidates(dump, vmid)
|
||||
if err != nil {
|
||||
log.Warn("vzdump-partial: dump directory unreadable", "dir", dump, "err", err)
|
||||
return nil
|
||||
}
|
||||
if len(ents) == 0 {
|
||||
return nil
|
||||
}
|
||||
var out []Result
|
||||
inFlight, err := c.InFlight(ctx, vmid)
|
||||
if err != nil || inFlight {
|
||||
why := "a backup of this guest is in flight"
|
||||
if err != nil {
|
||||
why = "cannot tell whether a backup is in flight: " + err.Error()
|
||||
}
|
||||
for _, e := range ents {
|
||||
log.Info("vzdump-partial: leaving a half-written backup", "storage", storage, "vmid", vmid, "name", e.Name, "why", why)
|
||||
out = append(out, Result{Entry: e, Why: why})
|
||||
}
|
||||
return out
|
||||
}
|
||||
b, bootOK := c.Boot()
|
||||
boot := time.Unix(b, 0)
|
||||
for _, e := range ents {
|
||||
dead, why := Dead(e, boot, bootOK, now())
|
||||
if !dead {
|
||||
log.Info("vzdump-partial: leaving a half-written backup", "storage", storage, "vmid", vmid, "name", e.Name, "why", why)
|
||||
out = append(out, Result{Entry: e, Why: why})
|
||||
continue
|
||||
}
|
||||
stdout, stderr, err := c.Runner.Run(ctx, Helper, storage, strconv.Itoa(vmid), e.Name)
|
||||
if err != nil {
|
||||
why = fmt.Sprintf("the root helper refused: %s", strings.TrimSpace(string(stderr)))
|
||||
log.Warn("vzdump-partial: leaving a half-written backup", "storage", storage, "vmid", vmid, "name", e.Name, "why", why, "err", err)
|
||||
out = append(out, Result{Entry: e, Why: why})
|
||||
continue
|
||||
}
|
||||
log.Warn("vzdump-partial: removed a dead half-written backup", "storage", storage, "vmid", vmid, "name", e.Name,
|
||||
"bytes", e.Size, "why", why, "helper", strings.TrimSpace(string(stdout)))
|
||||
out = append(out, Result{Entry: e, Removed: true, Why: why})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package vzpartial
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
const vmid = 9201
|
||||
const name = "vzdump-lxc-9201-2026_10_14-10_42_25"
|
||||
|
||||
var boot = time.Unix(1791967481, 0)
|
||||
|
||||
type fakeAPI struct{ stores []proxmox.Storage }
|
||||
|
||||
func (f fakeAPI) ListStorage(context.Context) ([]proxmox.Storage, error) { return f.stores, nil }
|
||||
|
||||
type fakeRunner struct {
|
||||
calls [][]string
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *fakeRunner) Run(_ context.Context, n string, a ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{n}, a...))
|
||||
if r.err != nil {
|
||||
return nil, []byte("REFUSED: in use"), r.err
|
||||
}
|
||||
return []byte("removed"), nil, nil
|
||||
}
|
||||
func (r *fakeRunner) RunStdin(ctx context.Context, _ io.Reader, n string, a ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, n, a...)
|
||||
}
|
||||
|
||||
type rig struct {
|
||||
dump string
|
||||
runner *fakeRunner
|
||||
c *Cleaner
|
||||
}
|
||||
|
||||
func newRig(t *testing.T, storeType string, inFlight bool, inErr error) *rig {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
dump := filepath.Join(root, "dump")
|
||||
if err := os.MkdirAll(dump, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := &rig{dump: dump, runner: &fakeRunner{}}
|
||||
r.c = &Cleaner{
|
||||
API: fakeAPI{stores: []proxmox.Storage{{Storage: "local", Type: storeType, Path: root}}},
|
||||
Runner: r.runner,
|
||||
InFlight: func(context.Context, int) (bool, error) { return inFlight, inErr },
|
||||
Boot: func() (int64, bool) { return boot.Unix(), true },
|
||||
Now: func() time.Time { return boot.Add(5 * time.Minute) },
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *rig) put(t *testing.T, n string, dir bool, mt time.Time) {
|
||||
t.Helper()
|
||||
p := filepath.Join(r.dump, n)
|
||||
if dir {
|
||||
if err := os.MkdirAll(filepath.Join(p, "etc"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chtimes(filepath.Join(p, "etc"), mt, mt)
|
||||
} else if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chtimes(p, mt, mt)
|
||||
}
|
||||
|
||||
func (r *rig) asked() []string {
|
||||
var out []string
|
||||
for _, c := range r.runner.calls {
|
||||
if len(c) == 4 && c[0] == Helper && c[1] == "local" && c[2] == "9201" {
|
||||
out = append(out, c[3])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// RED-PROOF: remove the `bootOK && e.ModTime.Before(boot)` rule in Dead → a pre-boot entry modified 1 min before the
|
||||
// boot is still „untouched for 6m" — so the test uses a 1-min-old-at-boot entry AND a Now 30 s after the boot.
|
||||
func TestClean_RemovesDeadPartialAndTmp(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.c.Now = func() time.Time { return boot.Add(30 * time.Second) }
|
||||
r.put(t, name+".tar.dat", false, boot.Add(-time.Minute))
|
||||
r.put(t, name+".tmp", true, boot.Add(-time.Minute))
|
||||
res := r.c.Clean(context.Background(), "local", vmid)
|
||||
if got := r.asked(); len(got) != 2 {
|
||||
t.Fatalf("both dead entries must go to the helper, got %v", got)
|
||||
}
|
||||
for _, x := range res {
|
||||
if !x.Removed || x.Why != "written before this boot" {
|
||||
t.Fatalf("%+v", x)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClean_RemovesQuietPartialOfThisBoot(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.put(t, name+".tar.dat", false, boot.Add(time.Minute)) // 4 min before Now
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if len(r.asked()) != 1 {
|
||||
t.Fatal("a partial untouched for QuietFor must be removed")
|
||||
}
|
||||
}
|
||||
|
||||
// Each keep has its positive control (the dead pre-boot partial in the same rig IS asked for).
|
||||
func TestClean_Keeps(t *testing.T) {
|
||||
cases := map[string]func(t *testing.T) (*rig, string){
|
||||
// RED-PROOF: drop the InFlight check in Clean → the in-flight partial is asked for → fails.
|
||||
"in flight": func(t *testing.T) (*rig, string) { return newRig(t, "dir", true, nil), name + ".tar.dat" },
|
||||
"in-flight unknown": func(t *testing.T) (*rig, string) {
|
||||
return newRig(t, "dir", false, errors.New("api down")), name + ".tar.dat"
|
||||
},
|
||||
// RED-PROOF: drop the `st.Type != "dir"` test → an lvmthin target is cleaned → fails.
|
||||
"not a dir storage": func(t *testing.T) (*rig, string) { return newRig(t, "lvmthin", false, nil), name + ".tar.dat" },
|
||||
}
|
||||
for what, mk := range cases {
|
||||
r, n := mk(t)
|
||||
r.put(t, n, false, boot.Add(-time.Hour))
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if len(r.asked()) != 0 {
|
||||
t.Fatalf("%s: must keep, helper asked for %v", what, r.asked())
|
||||
}
|
||||
ctl := newRig(t, "dir", false, nil)
|
||||
ctl.put(t, n, false, boot.Add(-time.Hour))
|
||||
ctl.c.Clean(context.Background(), "local", vmid)
|
||||
if len(ctl.asked()) != 1 {
|
||||
t.Fatalf("%s: positive control not removed", what)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: weaken NamePattern to `vzdump-.*` → the finished archive / log / other guest's partial are asked for.
|
||||
func TestClean_OnlyThisGuestsHalfWrittenNames(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
old := boot.Add(-time.Hour)
|
||||
for _, n := range []string{name + ".tar.zst", name + ".log", name + ".tar.zst.notes", "vzdump-lxc-9202-2026_10_14-10_42_25.tar.dat",
|
||||
"vzdump-lxc-92011-2026_10_14-10_42_25.tar.dat", name + ".tar.dat.bak", name + ".tar.da", "x" + name + ".tar.dat"} {
|
||||
r.put(t, n, false, old)
|
||||
}
|
||||
r.put(t, name+".tar.dat", false, old) // the positive control in the same directory
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if got := r.asked(); len(got) != 1 || got[0] != name+".tar.dat" {
|
||||
t.Fatalf("only %s.tar.dat may be asked for, got %v", name, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClean_KeepsRecentAndWrongKindAndSymlink(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.put(t, name+".tar.dat", false, boot.Add(4*time.Minute)) // 1 min before Now: may still be written
|
||||
r.put(t, "vzdump-lxc-9201-2026_10_14-11_00_00.tmp", false, boot.Add(-time.Hour)) // a FILE named .tmp
|
||||
r.put(t, "vzdump-lxc-9201-2026_10_14-13_00_00.tar.dat", false, boot.Add(-time.Hour)) // positive control
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if got := r.asked(); len(got) != 1 || got[0] != "vzdump-lxc-9201-2026_10_14-13_00_00.tar.dat" {
|
||||
t.Fatalf("only the positive control may be asked for, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the ModeSymlink skip in Candidates → the symlink (quiet: Now is an hour ahead) is asked for → fails.
|
||||
func TestClean_KeepsSymlink(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.c.Now = func() time.Time { return time.Now().Add(time.Hour) }
|
||||
target := filepath.Join(r.dump, "elsewhere")
|
||||
_ = os.WriteFile(target, []byte("x"), 0o644)
|
||||
_ = os.Symlink(target, filepath.Join(r.dump, "vzdump-lxc-9201-2026_10_14-12_00_00.tar.dat"))
|
||||
r.put(t, "vzdump-lxc-9201-2026_10_14-13_00_00.tar.dat", false, boot.Add(-time.Hour)) // positive control
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if got := r.asked(); len(got) != 1 || got[0] != "vzdump-lxc-9201-2026_10_14-13_00_00.tar.dat" {
|
||||
t.Fatalf("only the positive control may be asked for, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClean_HelperRefusalIsNotARemoval(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.runner.err = errors.New("exit status 1")
|
||||
r.put(t, name+".tar.dat", false, boot.Add(-time.Hour))
|
||||
res := r.c.Clean(context.Background(), "local", vmid)
|
||||
if len(res) != 1 || res[0].Removed {
|
||||
t.Fatalf("a refused helper call must not count as removed: %+v", res)
|
||||
}
|
||||
}
|
||||
|
||||
// The root helper's own tests (configs/test_felhom_vzdump_partial_clean.py) run with the Go suite.
|
||||
func TestRootHelperPythonSuite(t *testing.T) {
|
||||
py, err := exec.LookPath("python3")
|
||||
if err != nil {
|
||||
t.Skip("python3 not found")
|
||||
}
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
script := filepath.Join(filepath.Dir(file), "..", "..", "configs", "test_felhom_vzdump_partial_clean.py")
|
||||
out, err := exec.Command(py, script).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("helper suite failed: %v\n%s", err, out)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user