Compare commits
34 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 65c1994d3b | |||
| 6545cccff7 | |||
| eb38d78945 | |||
| 4a1c07b288 | |||
| 5c3b218822 | |||
| e15d1d30ea | |||
| 344c4f9030 | |||
| c52d591851 | |||
| 37b99d73b3 | |||
| dd8b2ace02 | |||
| 383522450a | |||
| 1a8bed11e7 | |||
| 98f3cdd76a | |||
| 888a98571b | |||
| 912f7c127e | |||
| e569ec6c5f | |||
| 52ed52c131 | |||
| c5daa9480d | |||
| d2053e7d47 | |||
| 7d4e564758 | |||
| 9b749ee7ad | |||
| 7ae3081a76 | |||
| 7ecd63cd61 | |||
| a2f3aeeeff | |||
| e7d4ecd607 | |||
| 10ca63aa32 | |||
| 68a2130681 | |||
| bdf25f019c | |||
| c6d9435316 | |||
| f38b03cdb8 | |||
| 3403c87c80 | |||
| 86adbcaf8a | |||
| 24ea960229 | |||
| cff6fb5a45 |
@@ -44,8 +44,12 @@ it **first** in the morning note. A decision you cannot write in that shape is o
|
||||
Throwaway apps only; the standing apps and `bentopdf` stay.
|
||||
5. **Evidence off the machine at the end of each phase**, before any revert (R-320).
|
||||
6. **One release per repo per session**, with a CHANGELOG entry (controller: with its `MinAgent`
|
||||
line), REPORT overwritten, floor raised to deliver it. **No golden unless a drill or fresh install
|
||||
needs one** (the waiver, R-468). **No `--no-verify`.**
|
||||
line) and REPORT overwritten. **A controller release needs no golden** (operator ruling 2026-10-15
|
||||
11:26, `09` §3 decision 250; decision 7 stands): push the image only when it is ready for EVERY box (the
|
||||
household's update button offers the highest tag); raise the floor, MinAgent declared, only after it ran
|
||||
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
|
||||
on all three — not at night unless the night brief says so. **Agent releases still stop for the
|
||||
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
|
||||
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
|
||||
Prose is not a record.
|
||||
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
|
||||
@@ -73,3 +77,19 @@ that no longer exists. Each edit is named in the report (file, line, before, aft
|
||||
operator's word: loosen a safety rule, a fence, a „never", a protected machine, a secret rule, or a review step; or
|
||||
remove a rule. When in doubt, it is a rule change, and it goes to the operator. If Claude Code's own permission check
|
||||
asks before such an edit, wait for the operator's click; if it refuses, record that and file the exact line.
|
||||
|
||||
## 6. The system poster stays true
|
||||
|
||||
**A session that changes a fact listed in `architecture/felhom-system-poster.facts.md` (a machine, a
|
||||
role, a traffic path, a backup tier, a time, a retention, a key, a known gap) updates that file in
|
||||
the same commit.** If the change is text only, it also edits the matching text in
|
||||
`felhom-system-poster.html`. If it needs a new drawing, it adds the line
|
||||
**"System poster needs a refresh: <what changed>"** to `STATUS.md`'s "waiting on the operator" list.
|
||||
**The session report names which of the three it did.**
|
||||
|
||||
Why the three-way split: the poster is a drawing made in Claude Design, and **nothing in this
|
||||
repository renders it** — unlike `where-felhom-stands.html`, which has `render_stands.py`. So the
|
||||
facts file is the source of truth and the drawing trails it. `scripts/poster_facts_gate.py` warns
|
||||
when the facts file has a newer commit than the poster; it **never fails a push**, because a refresh
|
||||
needs the operator and another tool, and a gate nobody can clear is a gate people learn to route
|
||||
around.
|
||||
|
||||
@@ -16,7 +16,16 @@
|
||||
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
|
||||
# layer up. That is the last step, and it runs ONLY on failure.
|
||||
name: gates
|
||||
on: [push]
|
||||
# R-1129: `workflow_dispatch` is the manual PROOF that the failure mail can be sent. A push run is
|
||||
# unchanged by it: the proof step below is skipped on a push, and `force_alarm` is then empty.
|
||||
on:
|
||||
push:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
force_alarm:
|
||||
description: "Fail this run on purpose so the alarm step sends its real mail (a proof, not a gate result)"
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
jobs:
|
||||
gates:
|
||||
@@ -57,6 +66,22 @@ jobs:
|
||||
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
|
||||
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
|
||||
|
||||
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
|
||||
# Runs ONLY on a manual dispatch, never on a push. It prints the key's LENGTH, never the key,
|
||||
# so "CI can see the secret" is an observable rather than an assumption. With
|
||||
# force_alarm=true it then fails on purpose so the alarm step below sends its real mail;
|
||||
# the commit itself stays green, because its push run skips this step.
|
||||
if: github.event_name == 'workflow_dispatch'
|
||||
env:
|
||||
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
|
||||
FORCE_ALARM: ${{ github.event.inputs.force_alarm }}
|
||||
run: |
|
||||
python3 -c 'import os;print("RESEND_API_KEY length: %d" % len(os.environ.get("RESEND_API_KEY","")))'
|
||||
if [ "$FORCE_ALARM" = "true" ]; then
|
||||
echo "force_alarm=true - failing on purpose so the alarm step runs (R-1129 proof)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Alarm on failure
|
||||
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
|
||||
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
|
||||
@@ -70,6 +95,8 @@ jobs:
|
||||
if: failure()
|
||||
env:
|
||||
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
|
||||
# Empty on a push, so a push run's mail is exactly what it was before R-1129.
|
||||
FORCE_ALARM: ${{ github.event.inputs.force_alarm }}
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import json, os, sys, urllib.request, urllib.error
|
||||
@@ -87,7 +114,9 @@ jobs:
|
||||
body = json.dumps({
|
||||
"from": "Felhom CI <monitoring@felhom.eu>",
|
||||
"to": ["admin@felhom.eu"],
|
||||
"subject": "[felhom CI] gates FAILED in %s" % repo,
|
||||
"subject": ("[felhom CI] FORCED TEST, not a real failure - gates FAILED in %s"
|
||||
if os.environ.get("FORCE_ALARM") == "true"
|
||||
else "[felhom CI] gates FAILED in %s") % repo,
|
||||
"text": (
|
||||
"The gate entry point exited non-zero.\n\n"
|
||||
"Repository : %s\n"
|
||||
|
||||
+263
-1
@@ -1,4 +1,266 @@
|
||||
## Unreleased (2026-10-08) — no OS leg after a household press; the after-boot kernel report carries the real ring (R-899); no „wrong code" when older packages were not checked (R-304); the last backup survives a restart in the host report; the Secure Boot meta-package leaves the Proxmox lane
|
||||
## Unreleased — the orphan thin check runs only after a failed backup (R-1012 follow-up) (2026-10-17)
|
||||
|
||||
**Not released, not vouched; the next vouched agent release carries it.** v0.159.0/v0.159.1 ran the R-1012 orphan check
|
||||
before EVERY snapshot-mode whole-guest backup, and that check takes a short metadata snapshot of the live thin pool.
|
||||
Now `BackupRunner` runs it only when the snapshot step has failed (the vzdump start or its task) — the moment the
|
||||
answer is needed. The finding is still appended in plain words to the failed backup's error, exactly as before; a
|
||||
successful backup never runs the check and keeps a clean record; stop mode and an unwired runner are unchanged. The
|
||||
`thin-orphan-check` capability text says „after a failed whole-guest backup". Tests
|
||||
`TestBackup_SnapshotFailure_NamesTheOrphan` (now: exactly 1 call, after the failure) and
|
||||
`TestBackup_ThinCheck_SilentWhenNotApplicable` (now: 0 calls on a successful backup), red-proved against the v0.159.1
|
||||
`runner.go`: failure legs „thin check ran 2 times, want 1", success leg `calls=1`. No root file, sudoers or bundle
|
||||
change. (Installer side, felhom.eu, same night: `felhom-host-install.sh --rotate-recovery-only`, R-1141.)
|
||||
**Owed by the release that carries it (not yet proven live):** one green snapshot backup with NO `felhom-thin-check`
|
||||
call in the journal, and one failed snapshot backup whose error carries the finding.
|
||||
|
||||
## v0.159.1 — the signed decommission prunes the guest-bind record (found by v0.159.0's own live proof) (2026-10-17)
|
||||
|
||||
**v0.159.0 is superseded — do not vouch it.** Everything of v0.159.0 below, plus: `signedjobs.DecommissionExecutor`
|
||||
drops the drive from every guest's bind record (`SetGuestBinds`, wired to the agent's `GuestBindStore`), as the
|
||||
self-serve decommission always did. Without it a drive decommissioned by an operator-signed job and then forgotten
|
||||
(R-1124) left a record the 20-second guest-bind re-assert warned about every tick („reconcile: skipping non-enrolled
|
||||
drive") — seen on demo-hp minutes after the R-1124 proof. Test `TestDecommissionExecutor_PrunesTheGuestBindRecord`,
|
||||
red-proved. Released: tag `v0.159.1`, sha256 `de8f6ad9…aff9`, bundle `2af3d27d…7cd6` (the same root files as v0.159.0).
|
||||
**Not vouched.** Delivery to a box on the v0.158.1 bundle: `agent_update 0.159.1` → `agent_config_update 0.159.0-step1`
|
||||
(sha `1d65edb3…d6cf`, published) → `agent_config_update 0.159.1` (the bundle adds `felhom-thin-check`, R16).
|
||||
|
||||
## v0.159.0 — forget a decommissioned drive that is gone (R-1124); name an orphan thin device in a backup failure (R-1012); Go 1.27.2 (R-972) (2026-10-17)
|
||||
|
||||
**Superseded by v0.159.1 — do not vouch.** Task: „agent release, the catalog allow-list, break-glass rotation, off-site
|
||||
history check" (felhom.eu `REPORT-agent-allowlist-rotation-2026-10-17.md`).
|
||||
|
||||
- **Go 1.27.2 + x/crypto v0.58.0** (R-972, `37b99d7`): the first release that carries them (v0.158.1 was built with
|
||||
go1.26.0).
|
||||
- **R-1124 — a decommissioned drive that is gone is forgotten.** `SudoHostOps.ForgetRetiredDrives` (agent start and the
|
||||
20-second storage tick): for a Felhom drive unit whose intent is `decommissioned`, whose `/dev/disk/by-uuid` link is
|
||||
gone and which is not mounted, `systemctl disable` + the new root verb `felhom-priv-apply unit-forget <unit>` +
|
||||
`daemon-reload`. The unit is gone, so a later boot no longer waits for the drive and the agent stops listing it (no
|
||||
„absent by UUID" line every 20 s). A decommissioned drive still plugged in KEEPS its unit and raw mount, so the
|
||||
one-click re-enroll (H3) still works. `unit-forget` re-checks as root: a felhom by-UUID drive unit (not a network
|
||||
unit), disabled, its `Where` not mounted, its by-UUID link absent — otherwise refused. Sudoers `FELHOM_MOUNT` gains the
|
||||
one verb. Tests `TestForgetRetiredDrives_*` and `configs/test_felhom_priv_apply.py` `UnitForget`, red-proved.
|
||||
**Live on demo-hp:** a loop-backed test drive enrolled through the local API, decommissioned by a signed job (kept
|
||||
its unit while present), unplugged → 8 s later the unit and its wants link were removed and `GET /disks` dropped it.
|
||||
- **R-1012 — an orphan thin device is NAMED in the backup failure, never removed.** New READ-ONLY root helper
|
||||
`felhom-thin-check <vg> <pool>` (config bundle): LVM's thin ids before and after, the pool's ids through a metadata
|
||||
snapshot it reserves and releases (refuses — exit 3 — when someone else holds one; releases on every exit; sends no
|
||||
other pool message). `internal/thincheck`: an orphan = in the pool and in neither LVM list, on two reads 3 s apart;
|
||||
fails open. `BackupRunner.SetThinCheck`: before a snapshot-mode backup (a WARN when found) and again after a failure —
|
||||
the finding is appended in plain words to the error the controller mails to the operator, pointing at the new
|
||||
runbook `whole-guest-backup-orphan-thin.md`. Sudoers `FELHOM_THINCHECK`, capability `thin-orphan-check`. Tests
|
||||
`internal/thincheck`, `TestBackup_SnapshotFailure_NamesTheOrphan`, `configs/test_felhom_thin_check.py`, red-proved.
|
||||
**Live on demo-hp:** on a scratch thin pool (a loop file on `/mnt/hdd_1`, never `pve/data`) the helper listed a made
|
||||
orphan (device 7) and refused a snapshot held by someone else; on `pve/data` it read 1, 2 = LVM's and released; a
|
||||
normal whole-guest backup of 9201 ran the check first and finished green.
|
||||
- Released: tag `v0.159.0`, sha256 `3cfd0a25…edfed5`, bundle `40663c4f…6d681`; step bundle `0.159.0-step1`.
|
||||
|
||||
## CI only — a manual proof that the failure mail can be sent (R-1129, 2026-10-17)
|
||||
|
||||
No agent release; no binary changes.
|
||||
|
||||
`.gitea/workflows/gates.yml` gains a manual `workflow_dispatch` trigger with one input, `force_alarm` (default false), and
|
||||
one step that runs ONLY on a dispatch: it prints the length of `RESEND_API_KEY` (never the key) and, with
|
||||
`force_alarm=true`, fails on purpose so the existing alarm step sends its real mail; the mail's subject then starts
|
||||
„FORCED TEST, not a real failure". A push run is unchanged: the new step is skipped and the subject is the old one. Same
|
||||
change in all four repos' workflows. Purpose: prove the failure mail now goes out after the operator set the key
|
||||
(every failed run until today ended „ALARM FAILED: RESEND_API_KEY is empty").
|
||||
|
||||
## scripts only — the decoy suite is a gate of the full run (R-1116, 2026-10-17)
|
||||
|
||||
No agent release; no binary changes. `scripts/test_gate_decoys.py` is now the `gate-decoys` row of `agent_gates.py`:
|
||||
the full run (which CI runs on every push), not `--fast` — it takes ~20 s in the workspace (measured: 30 cases; ~5 s in a
|
||||
CI-shaped shallow clone, also green). It was in no runner, the shape that let the catalog's R-1115 fail unseen for a
|
||||
night. It plants only in scratch clones and asks a fake Gitea on 127.0.0.1; it does not call the runner (no
|
||||
recursion). Declared in its own `COVERS`; `test_agent_gates.py` pins it (registered once, full run only, named by
|
||||
`--fast`'s SKIPPED line). Red-proved: a reuse-refs FACT case flipped to expect rc 0 turned the full run red
|
||||
(`CONVICTED: gate-decoys`); restored, green. CLAUDE.md's `--fast` sentence names it.
|
||||
|
||||
## unreleased — Go 1.27.2 and golang.org/x/crypto v0.58.0 (R-972, 2026-10-16)
|
||||
|
||||
**Not released, not deployed** — agent releases wait for the operator's vouch; the next vouched release carries this.
|
||||
`go.mod`: `go 1.27.2` (was 1.25.0; the shipped 0.158.1 binary was built with go1.26.0), x/crypto v0.58.0 (was v0.52.0),
|
||||
x/sys v0.49.0. `go build`, `go vet`, the full `go test ./...` and `agent_gates.py --fast` green on Go 1.27.2.
|
||||
`release-agent.sh` builds with the toolchain `go.mod` names (GOTOOLCHAIN=auto), so no script change.
|
||||
|
||||
## scripts only — `release-agent.sh` refuses when the registry cannot say the version is new (R-1098, 2026-10-16)
|
||||
|
||||
No agent release; no binary changes. `scripts/release-agent.sh` already refused a version whose tag exists or whose
|
||||
package answers 200. It let everything else through — a registry it could not reach (curl prints `000`) or a 500 read
|
||||
as „new". Now only a clean `404` proves the version is new; anything else refuses with the code it got. New
|
||||
`scripts/test_release_push_once.sh` (the real script, a fake `curl` and `go` on PATH; published / absent / unreachable /
|
||||
server error / tag exists), red-proved: with the 404 rule removed, the unreachable and server-error cases fail. Same rule
|
||||
as the controller `build.sh` and, since today, felhom.eu `scripts/build-hub.sh`.
|
||||
|
||||
## v0.158.1 — security review of v0.158.0: the mkfs wrapper trusts no sibling claim; the internal-place helper fails closed (2026-10-16)
|
||||
|
||||
**v0.158.0 is superseded — do not vouch it.** A security review of `912f7c1` found two defects; both fixed here, each test
|
||||
seen failing against the 0.158.0 script:
|
||||
- **`felhom-mkfs-guarded`**: 0.158.0 let a format through when a SIBLING part was mounted at /mnt/<name> AND under
|
||||
/mnt/felhom-drives („Felhom's own drive"). That corroboration is a mount the agent can make itself, so a compromised agent
|
||||
could unlock a host owner's disk — the exact case this root wrapper exists for. Removed: any mount of any node outside
|
||||
/mnt/felhom-drives refuses again (now read from the kernel's mount table for every node; an unlistable disk refuses).
|
||||
Consequence: the free part next to an in-use Felhom drive cannot be formatted — `CandidatePartition.Formattable` is false
|
||||
for every part of a disk with a mounted node, and the controller does not offer it (a part with a filesystem can still be
|
||||
USED). Pinned by configs/test_felhom_mkfs_guarded.py + partitions_test.go.
|
||||
- **`felhom-internal-place`**: a failed `lvs` listing read as „nothing promised" (a too-large place) and a `blkid` error read
|
||||
as „no signature" (the only gate before mkfs). Now: a failed listing, a listing without the pool, or any blkid exit other
|
||||
than 0/2 REFUSES. Three new tests, all failing against 0.158.0.
|
||||
- Released: tag `v0.158.1` (code `98f3cdd`), sha256 `20264df6…84b76a1`, bundle `c23c3d35…cb1c41b`. **Not vouched.**
|
||||
|
||||
## v0.158.0 — the internal data place; partitions as places; add_internal_place (2026-10-16)
|
||||
|
||||
Design: felhom.eu `architecture/16-storage-physical-identity.md` §5, §6, §7, §9 (operator rulings D3–D5; the D3 sizes
|
||||
ruled 2026-10-16 after the space check, felhom.eu `audits/storage-identity-buildB-2026-10-15/SPACE-CHECK.md`).
|
||||
|
||||
- **The internal data place („Belső tárhely")** (`internal/localapi/internalplace.go`): when the record
|
||||
`/var/lib/felhom-agent/internal-place.json` names a filesystem, the agent mounts it at `/mnt/belso` by fs-UUID like an
|
||||
enrolled drive (startup + every 20 s tick) and records it `enrolled`; `GET /disks` reports it with kind `internal`.
|
||||
It never takes `/mnt/belso` from a household drive, and eject/decommission/format of it are refused (device-mapper on
|
||||
the system disk). Its identity is the system disk's, so build A never counts a copy to `sys_drive` as a second copy.
|
||||
- **Root helper `configs/felhom-internal-place`** (new in the bundle): `plan|create <planned GiB>` — the ruled size rule
|
||||
(reserve max(16 GiB, 10 %) ≤ 48 GiB, made only when ≥ 20 GiB, nothing promised beyond pool − reserve), the only
|
||||
lvcreate/mkfs of a system-disk volume; formats only a `felhom-belso` with no signature. Run by the installer (1.33.0)
|
||||
and by the new signed op.
|
||||
- **Signed op `add_internal_place`** (old one-disk boxes, `16` §9): no parameters (the size is the rule's), destructive
|
||||
class (operator key). Runs the helper with `create 0`, writes the record, mounts.
|
||||
- **Partitions as places** (`internal/storage/partitions.go`): `GET /disks/candidates` lists each partition of a free
|
||||
disk as empty / filesystem / in_use. A data-bearing part's wipe id binds disk id + part number + fs-UUID
|
||||
(`byid:<…-partN>+fsuuid:<uuid>`); both must name the same device at format time.
|
||||
- **`felhom-mkfs-guarded`** (REVERTED in v0.158.1): a SIBLING part that is Felhom's own drive no longer blocked formatting
|
||||
the free part next to it. First tests of the wrapper: `configs/test_felhom_mkfs_guarded.py`.
|
||||
- `configs/` changed (sudoers `FELHOM_INTERNALPLACE`, the helper, the wrapper, the bundle list) — **a config-bundle job
|
||||
is needed** with the agent update. **Not vouched** (the operator's act).
|
||||
- Released: tag `v0.158.0` (code `912f7c1`), sha256 `a3589d1d…2caba`, bundle `17066273…fb5c76`. **Superseded by v0.158.1 —
|
||||
do not vouch** (its mkfs wrapper trusted a sibling claim the agent can fake; its helper failed open).
|
||||
|
||||
## v0.157.0 — every storage place knows its physical disk(s); POST /disks/same-disk (2026-10-15)
|
||||
|
||||
Design: felhom.eu `architecture/16-storage-physical-identity.md` §3, §4, §10 (operator rulings D1–D6, 2026-10-15). Spike:
|
||||
felhom.eu `audits/storage-identity-2026-10-15/`.
|
||||
|
||||
- **Identity of a storage place** (`internal/storage/identity.go`): the physical disk(s) under it, from `/sys`, `/run/udev/data`
|
||||
and `/proc/mounts` only — partition → disk, device-mapper / LVM thin → slaves, loop → backing file → its disk. Keys:
|
||||
`eui:` (NVMe), `serial:<bus>:<model>:<serial>`, `wwn:` only without a serial and only with a vendor (OUI ≠ 000000).
|
||||
`proven`, or `unknown` with one reason: `vm` (systemd-detect-virt), `network`, `usb_bridge` (a placeholder serial — the rule
|
||||
is the design's, UNPROVEN on a real enclosure, R-1077), `weak_wwn`, `no_serial`. No block I/O.
|
||||
- **Reported:** `GET /disks` rows and the hub report's `storage_targets` gain `kind`, `physical_disks`, `identity`,
|
||||
`identity_reason` (additive; the golden and the hub's copy change together). A dir storage is judged only on its OWN mount — an
|
||||
absent drive is "not computed", never the root disk.
|
||||
- **`POST /disks/same-disk`** `{a, b}` (guest paths) → `{different, proven, reason, a, b}`. Maps `/mnt/felhom-drives/<name>` to
|
||||
the host mount at exactly that path, and `/mnt/sys_drive`, `/var/lib/felhom`, `/var/lib/docker` to the guest's data volume's
|
||||
storage. Anything else, or an absent drive → `unmapped`. A shared disk key is "same, proven" even when the disk is unknown.
|
||||
Scoped like every `/disks` call.
|
||||
- Tests: fixture `/sys` + udev trees for every spike case (one disk LVM thin, two disks, two partitions, a loop on a file, NVMe eui,
|
||||
vendor-less WWN, a VM, a USB placeholder serial, a network share) and the same-disk call (equal, different, VM, NAS, absent
|
||||
drive, scope); red-proofed (the LVM "diamond" bug was seen failing first; the VM, WWN, own-mount and sys_drive rules broken on purpose).
|
||||
- `configs/` unchanged — no config-bundle job needed. **Not vouched** (the operator's act).
|
||||
|
||||
## Unreleased (2026-10-15, afternoon) — the shared rule file; no code change
|
||||
|
||||
- `.claude/rules/unprompted-work.md` §3 item 6: **a controller release needs no golden** — the floor carries it after the
|
||||
three test boxes; agent releases still wait for the operator's vouch; the golden is weekly (operator ruling 2026-10-15
|
||||
11:26, `09` §3 decision 250; the same morning's „bake first", decision 247, reversed). Same text in all five copies.
|
||||
|
||||
## Unreleased (2026-10-15) — the shared rule file; no code change
|
||||
|
||||
- `.claude/rules/unprompted-work.md` §3 item 6: **bake first** — a controller release is released only when its golden is
|
||||
baked; bake → the operator vouches → the operator raises the floor; never ask for a floor for a version with no golden
|
||||
(operator ruling 2026-10-15, `09` §3 decision 247, superseding decision 7). Same text in all five copies.
|
||||
|
||||
## v0.156.1 — R-1008 by process liveness, not the clock; 0.156.0 superseded — do not vouch it (2026-10-14)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `83c7da27d457b63ba8a4cde5a8717b3e5f9db9f83a46e63f38902dc7471bffb9`, bundle
|
||||
`5a678def792c70ec1cbbd198c14542161fac91d11a69bd0ce23dd86469e96ec3` (tag `v0.156.1` = `7ae3081`). Step bundle `0.156.1-step1`
|
||||
sha256 `ac83ea41507616d4c1056cccfd35de1410ff01e0296f4e64ff925207a43a8490` (the 0.155.0 bundle with only `felhom-os-apply`
|
||||
replaced), published as package version `0.156.1-step1`. **Not vouched** (the operator's act). Delivery as 0.156.0: per box
|
||||
`agent_update 0.156.1` → `agent_config_update 0.156.1-step1` → `agent_config_update 0.156.1`.
|
||||
|
||||
- **0.156.0 is superseded — never vouch it.** A background security review of the pushed commit flagged „fail-open state
|
||||
drift": 0.156.0 called a listed vzdump task dead when its wall-clock start was before the kernel's `btime`. `btime` is
|
||||
derived from the wall clock and moves when the clock is corrected after the boot (an NTP step after a power cut), so a
|
||||
backup started between the boot and the correction could look „from before the boot" — and the stale-lock pass would
|
||||
clear a LIVE backup's lock.
|
||||
- **Now:** `taskAlive` is Proxmox's own rule (`PVE::ProcFSTools::check_process_running`, read on demo-hp): the UPID's pid and
|
||||
pstart against `/proc/<pid>/stat` — same start tick (field 22) and not a zombie. No clock is involved. Fails safe (running)
|
||||
when the UPID does not parse, the stat cannot be read, or this agent cannot see root's processes at all (PID 1 not visible:
|
||||
a `hidepid` /proc or `ProtectProc=` would hide every worker — demo-hp: `/proc` plain, `ProtectProc=default`).
|
||||
- Tests: worker gone, pid reused, zombie, live worker, a „clock step" (wall-clock start absurdly early, worker alive → still
|
||||
running), doubt, hidden /proc — each red-proofed; 19 guards in all (`power-cut-fixes-2026-10-14/redproof.txt`).
|
||||
- R-1006 (the cleanup) is unchanged; its „before this boot" rule still reads `btime`, but a drifted `btime` cannot remove a
|
||||
live partial: the Go side asks this liveness rule, and the root helper refuses while any vzdump runs or anything holds
|
||||
the file open.
|
||||
|
||||
## v0.156.0 — after a power cut: the dead half-written backup is removed (R-1006); a backup task from before the boot is not „in flight" (R-1008) (2026-10-14)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `8db2278838dcb733d5201fba84b75a829b9adebdf899c89dd7de0939ec10d51a`, bundle
|
||||
`a21605217966ebc953d55a35157a9eb0243ca542c50dbd94fc9fa67257f27369` (tag `v0.156.0` = `a2f3aee`). Step bundle `0.156.0-step1` sha256
|
||||
`8be2bb8f3d54bff8e7140cf99714664df31adecf13aa721597323fb2b9d43db5` (the 0.155.0 bundle `795f2ab4…` with only `felhom-os-apply`
|
||||
replaced; `scripts/build-step-bundle.py`), published as package version `0.156.0-step1`. **Not vouched** (the operator's act).
|
||||
|
||||
**Delivery: the agent binary AND the config bundle, and the bundle ADDS a path** (`/usr/local/sbin/felhom-vzdump-partial-clean`)
|
||||
— so per box: signed `agent_update 0.156.0` → `agent_config_update 0.156.0-step1` (the step bundle) → `agent_config_update
|
||||
0.156.0`. Evidence: felhom.eu `documentation/audits/power-cut-fixes-2026-10-14/` (the spike C1–C4, the red-proofs).
|
||||
|
||||
- **R-1008 — one in-flight answer.** `staleLockController.BackupRunning` (the ONE implementation all four callers reach:
|
||||
the start-up stale-lock pass, the guest-power watchdog, the household power-off guard, the controller supervisor) now
|
||||
answers through `vzdumpInFlight`: a vzdump task for the vmid that started at or after the host's boot. Proxmox keeps a
|
||||
dead task in `tasks?source=active` for ~30–60 s after a power cut (measured, C3) — the agent starts inside that window
|
||||
and called it „genuinely in-flight". A task from before the boot is dead; logged once (vmid, UPID, start, boot). The
|
||||
start time comes from the list or, if absent, the UPID; an unreadable boot time or start time FAILS SAFE (running), as
|
||||
before. The boot time is `poweroff.ProcBtime` (no third /proc/stat parser).
|
||||
- **R-1006 — the half-written backup is removed.** New `internal/vzpartial`: on a `dir` backup target it finds exactly
|
||||
`vzdump-(lxc|qemu)-<vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp)` of THAT guest (never a finished archive, a log,
|
||||
another guest, a symlink, a wrong kind), and — only when no vzdump of the guest is in flight (R-1008) and the entry is
|
||||
dead (modified before this boot, or untouched for 120 s) — asks the root helper to remove it. Runs at agent start (after
|
||||
the stale-lock pass, every owned guest × every backup tier) and before every backup's space preflight
|
||||
(`BackupRunner.SetBeforeBackup`). Any doubt leaves the entry and logs why.
|
||||
- **The root helper `configs/felhom-vzdump-partial-clean`** (config bundle; sudoers `FELHOM_VZPARTIAL`, one exact line: a
|
||||
storage id, a vmid, one half-written name) re-checks everything as root: a `dir` storage from `/etc/pve/storage.cfg`; the
|
||||
name of that vmid; not a symlink; directly in `<path>/dump`; **no vzdump on the host at all** (a job for several guests
|
||||
carries no vmid in its task name); **no process holds the entry open** (C2: after a killed worker, `tar` and `zstd` kept
|
||||
writing into the `.tar.dat` for 4 min 40 s, none named „vzdump"); dead (before this boot, or quiet 120 s).
|
||||
- **Not changed, by evidence (C4):** the space preflight's sizing (R-1006 part 2) — on „No space left" vzdump removes its
|
||||
own partial on both its paths (rsync and tar); only a power cut leaves one.
|
||||
- **Tests:** `stalelock_inflight_test.go` (the helper, the production controller, three callers end to end, a structural
|
||||
pin that every `BackupRunning` call goes through `s.staleLock` and there is one implementation), `vzpartial_test.go`
|
||||
(removes; keeps in flight / unknown / non-dir / other names / recent / wrong kind / symlink / a helper refusal — each keep
|
||||
with its positive control), `r685_space_test.go` (the cleanup runs BEFORE the preflight), the helper's own 21 Python
|
||||
tests (`configs/test_felhom_vzdump_partial_clean.py`, run by the Go suite), 8 sudoers injection cases. **17 guards
|
||||
red-proofed** by mutation (`power-cut-fixes-2026-10-14/redproof.txt`).
|
||||
- **Fixed without a row:** `internal/localapi/guestbind.go` and `controllersupervisor_test.go` were not gofmt-clean on
|
||||
`main` (an import order and a struct alignment) — formatted.
|
||||
|
||||
## v0.155.0 — the household can turn the machine off: POST /host/poweroff (R-998); the escrow reply race fixed (2026-10-13)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `bb339e944eff4aa6d10ae09158e0943de2e5c3777f52bf993c796d57e46800c3`, bundle
|
||||
`795f2ab41661e993e3fc4ce65df4f11ce94280d94871ab2e5cf1993ae3fbff7d` (tag `v0.155.0`). **Not vouched** (the operator's act).
|
||||
**The bundle changes `felhom-agent.sudoers`** (no new path — no step bundle): delivery is a signed `agent_update` +
|
||||
`agent_config_update`. Design: felhom.eu `documentation/architecture/14-host-power-off.md`.
|
||||
|
||||
- **`POST /host/poweroff`** (local API, `withGuest`-scoped like `/guest/reboot`) — the controller's „A gép kikapcsolása"
|
||||
(controller v0.310.0). Refused (409, a stable `reason`) in the first 10 min after boot; a second time in one boot
|
||||
(`internal/poweroff`, a file keyed by `/proc/stat` btime); while ANY other guest or VM runs on the host (read from
|
||||
`/proc`: `lxc-start -n N`, `kvm -id N` — demo-hp's scratch guest and Tester 1 VM block it); while the heavy-operation
|
||||
gate is held (backup, restore test, OS step, sweep, trim), a backup job, a controller swap, a drive format, a NAS verify
|
||||
or an escrow ceremony runs, or Proxmox runs a vzdump of the guest. Every check fails closed.
|
||||
- On acceptance: the record first; the heavy gate TAKEN and kept; a host report carrying `deliberate_off.requested_at`
|
||||
accepted by the hub (bounded 60 s, `Loop.SetReportedHook`); then before each step the jobs and other guests are
|
||||
checked again — a job waits (≤ 15 min), another guest cancels at once, a job still running after the wait cancels
|
||||
(gate released, box stays on); `pct shutdown <vmid> --timeout 120`; `systemctl poweroff`.
|
||||
- **Sudoers:** `FELHOM_POWEROFF` = `/usr/bin/systemctl poweroff`; `pct ^shutdown [0-9]+ --timeout [0-9]+$` joins
|
||||
`FELHOM_GUESTHOOK`; nine injection cases (`--forceStop`, `poweroff --force`, `reboot`, `halt`, `kexec`…).
|
||||
**New `TestEveryAliasIsGranted`:** an alias missing from the grant line passed the coverage check before.
|
||||
- **Host report:** `deliberate_off` (omitempty; the cross-repo golden unchanged). Hub v0.149.0 holds its silence alarms
|
||||
for 14 days on it.
|
||||
- **Fixed without a row:** `go test -race` found a data race in the escrow ceremony start reply (it read `job.Phase`
|
||||
while the job goroutine wrote it); the reply fields are copied first. Red before, green 3/3 after.
|
||||
- Security review (three passes the same evening): other guests, a gate not held, fail-open unwired checks, jobs and
|
||||
guests starting after acceptance — each fixed and red-proofed (`hostpoweroff_test.go`, 11 tests).
|
||||
|
||||
## v0.154.0 — no OS leg after a household press; the after-boot kernel report carries the real ring (R-899); no „wrong code" when older packages were not checked (R-304); the last backup survives a restart in the host report; the Secure Boot meta-package leaves the Proxmox lane (2026-10-09)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `36a54ad20e896cd61d7b9944e2a521f4209c2b76940d8f1827af05b645463226`, bundle
|
||||
`93487989f50d4a3039a05cec0f91ec478d62489a414705e49403e9b1340d20e0` (tag `v0.154.0` = `db25b46`).
|
||||
|
||||
**Delivery: the agent binary AND the config bundle** — the wrapper `felhom-os-apply` changed (2026-10-09).
|
||||
|
||||
|
||||
@@ -56,7 +56,8 @@ This is in the core because breaching it is how this component stops being audit
|
||||
gate in its `GATES` table (that table is the list); the shared ones — `reuse_refs_check`,
|
||||
`instructions_gate`, `observations_gate` — are the copies in `felhom.eu/scripts/`, never copied into
|
||||
this repo (a copy recreates the drift they detect; an absent sibling clone FAILS). `--fast` selects the
|
||||
gates touching no network and no container runtime, and skips `published` (network), naming it. **A missing gate is a FAILURE, never a skip.**
|
||||
gates touching no network and no container runtime, and skips `published` (network) and `gate-decoys` (the decoy
|
||||
suite, ~20 s; R-1116), naming them — CI runs the full set. **A missing gate is a FAILURE, never a skip.**
|
||||
|
||||
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
|
||||
**per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
|
||||
|
||||
+17
@@ -1,5 +1,22 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-17 — v0.159.1 RELEASED (0.159.0 superseded, do not vouch).** Go 1.27.2; R-1124 a decommissioned drive that is
|
||||
> gone is forgotten (`ForgetRetiredDrives`, root verb `unit-forget`); R-1012 the read-only `felhom-thin-check` names an
|
||||
> orphan thin device in a failed backup's text; the signed decommission prunes the guest-bind record. Bundle adds a path →
|
||||
> step bundle `0.159.0-step1`. Proven on demo-hp; delivered by signed jobs on the operator's yes. Vouch: the operator's.
|
||||
|
||||
> **2026-10-16 — v0.158.1 RELEASED and VOUCHED (storage identity build B; 0.158.0 superseded, do not vouch).** The internal
|
||||
> data place (root helper `felhom-internal-place`, the record in the state dir, mounted at /mnt/belso like a drive), the
|
||||
> signed `add_internal_place`, partitions in `/disks/candidates` (`formattable` only while nothing on the disk is mounted),
|
||||
> the partition wipe id `byid:<…-partN>+fsuuid:<uuid>`, the strict mkfs wrapper. Delivered through step bundle
|
||||
> `0.158.1-step1` (a bundle that adds a path, R16). Next: R-1093 (an empty part's probe reads the parent's partition table).
|
||||
|
||||
> **2026-10-15 (night) — v0.157.0 RELEASED, NOT VOUCHED (storage identity build A, felhom.eu `16` §13).** Every storage
|
||||
> place carries `kind`, `physical_disks`, `identity` (+ reason) in `GET /disks` and the hub report; `POST /disks/same-disk`
|
||||
> answers the controller (v0.317.0, MinAgent 0.157.0). Computed on every read, never cached. Delivered by signed jobs to
|
||||
> Tester 1, demo-hp, N100 only. Next: the operator vouches 0.157.0 → the global floor 0.317.0 / MinAgent 0.157.0. Build B
|
||||
> (partition candidates, the internal place) waits for a task file (R-1079).
|
||||
|
||||
> **2026-10-08 (day) — UNRELEASED on main, ships with tomorrow's release (decision 178).** R-899: `POST /backup?trigger=manual`
|
||||
> runs no OS leg (`localapi/server.go`); after-boot kernel reports carry the saved block's ring (`kernelReportRing`). R-304:
|
||||
> `escrow.ErrRetainedUnchecked` → HTTP 424 `older_unchecked` when not every earlier package was tried (withheld, caps,
|
||||
|
||||
@@ -1,16 +1,41 @@
|
||||
# REPORT — agent, 2026-10-08 (day): R-899 and R-304 on main, unreleased
|
||||
# REPORT — the orphan thin check runs only after a failed backup (unreleased) (2026-10-17, night)
|
||||
|
||||
**No release, no delivery today** (kernel night 8→9; `09` §3 decision 178). Binary-only; ships with tomorrow's release.
|
||||
Written in ASD-STE100 Simplified Technical English. Architecture read: felhom.eu `documentation/architecture/03-host-agent.md`
|
||||
(the `FELHOM_THINCHECK` row), `runbooks/whole-guest-backup-orphan-thin.md`, R-1012's closing note.
|
||||
|
||||
| Item | Commit | Tests | Red-proof |
|
||||
|---|---|---|---|
|
||||
| R-899 — `trigger=manual` (a press) runs no OS leg | `4c69c25` | `TestAfterPrimaryBackup` + 2 sub-cases | ignoring `trigger` → the leg ran once after a press (`[8200]`) |
|
||||
| „ring 1" label in after-boot kernel reports (seen 2026-10-08 night, demo-felhom) | `4c69c25` | `TestKernelReportRing_BeforeFirstFetch` | `Block().Ring` → `ring 1, want 0` |
|
||||
| R-304 — 424 `older_unchecked` instead of „did not open the sealed bundle" when earlier packages were not all tried | `91b9405` | `TestR304_*` (real age crypto), 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus` | check off → four cases returned the wrong-code error |
|
||||
## In plain words
|
||||
|
||||
**Read before the change (read-only, demo-hp):** on 2026-10-07 08:51 the morning press DID reach the OS leg
|
||||
(`osupdate: skipped — already ran tonight`, saved by the 20-hour rule only); `os-update-block.json` on demo-hp holds
|
||||
`ring 0` (the fallback the label fix reads).
|
||||
- Before tonight the agent looked for an orphan thin device before **every** whole-guest backup. To look, it takes a
|
||||
short metadata snapshot of the live disk pool. Now it looks **only when the backup's snapshot step has failed** — the
|
||||
only moment somebody needs the answer. The operator's failure mail still names the orphan, in the same words.
|
||||
- **Not released. Not vouched.** The next agent release that the operator vouches carries it. Nothing was copied to a
|
||||
box.
|
||||
|
||||
**Gates:** `go build/vet/test ./...` rc 0; `agent_gates.py --fast` rc 0. **CI:** job 1529 (`4c69c25`) success; job 1530
|
||||
(`91b9405`) success.
|
||||
## What changed
|
||||
|
||||
- `internal/backup/runner.go`: the pre-run check (and its WARN line) is gone; `explainFailure` (after a vzdump start
|
||||
failure or a task failure, snapshot mode only) is now the only caller. Comments say so.
|
||||
- `internal/capability/manifest.go`: the `thin-orphan-check` text says „after a failed whole-guest backup".
|
||||
- `internal/backup/r1012_thincheck_test.go`: failure → exactly ONE call and the finding in the error; success → ZERO
|
||||
calls and a clean record; stop mode → zero calls; unwired → error unchanged.
|
||||
|
||||
## Red-proof
|
||||
|
||||
The new tests, run against the v0.159.1 `runner.go` (`git show HEAD:…`), then the fix restored:
|
||||
|
||||
| test | failed with |
|
||||
|---|---|
|
||||
| `TestBackup_SnapshotFailure_NamesTheOrphan/{task_fails,start_fails}` | `thin check ran 2 times, want 1 (after the failure only)` |
|
||||
| `TestBackup_ThinCheck_SilentWhenNotApplicable` | `success: … Success:true … calls=1 (want no check on a successful backup, clean record)` |
|
||||
|
||||
Restored → both PASS; `git diff` on `runner.go` = the fix only.
|
||||
|
||||
## Gates
|
||||
|
||||
`go build ./... && go vet ./... && go test ./...` and `python3 scripts/agent_gates.py` — results in the session report.
|
||||
|
||||
## Not live-validated
|
||||
|
||||
No agent was built or delivered (no release tonight). The changed path is the failure path; the next release's live
|
||||
proof should make one snapshot backup succeed (helper NOT called: no `felhom-thin-check` line in the agent journal
|
||||
and none in the sudo log for that run) and one fail (the finding in the failure text).
|
||||
|
||||
@@ -15,13 +15,14 @@
|
||||
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
|
||||
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
|
||||
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
|
||||
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key \| controller-image <vmid>` (the last reads the ref on stdin, R-861 (a) A1) | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
|
||||
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| unit-forget <name> (v0.159.0, R-1124) \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key \| controller-image <vmid>` (the last reads the ref on stdin, R-861 (a) A1) | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
|
||||
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
|
||||
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
|
||||
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
|
||||
| `osupdate.Leg.SendUnsent` / `lockPass` (v0.144.0, R-868) | internal/osupdate/unsent.go | `(ctx) int` | an OS-pass report the agent never sent (killed mid-pass): the wrapper keeps `report-<run>-<layer>-apply.json` beside the plan; the agent deletes it once the hub has it | any new caller that runs an apply pass must hold `lockPass` (flock, across processes) — the sender must never take a running pass's copy |
|
||||
| `osupdate.LoadSavedBlock` (v0.144.0, R-866) | internal/osupdate/leg.go | `(planDir) (block, savedAt, ok)` | the hub's newest os_update block as the daemon last received it (`os-update-block.json`) | the debug pass uses it ONLY when the hub cannot be reached, and says so in its header; no saved block → no pass |
|
||||
| `dpkg_state()` / `DPKG_STATE_SCRIPT` (v0.145.0, R-876) | configs/felhom-os-apply | `audit, journal = self.dpkg_state()` | dpkg's state in ONE call: `--audit` AND the update journal | never gate a repair on `--audit` alone — a crash leaves only the journal (measured); keep it one call (R-845) |
|
||||
| `thincheck.Checker.Check` + `Describe` (v0.159.0, R-1012) | internal/thincheck/thincheck.go | `Check(ctx, vmid) []Finding` | name an ORPHAN thin device (pool holds it, LVM does not) in a whole-guest backup failure — `BackupRunner.SetThinCheck` | READ-ONLY by design: the root helper `felhom-thin-check <vg> <pool>` sends only reserve/release_metadata_snap; an orphan needs two reads; fails open (never stops a backup). The repair stays human |
|
||||
| `guesthook.SnippetReady` / `Register` (v0.146.0, R-861) | internal/guesthook/install.go | `SnippetReady(path) error` | is the pre-start hook (a FIXED file from the bundle) in place — register only then | The agent never installs the hook (Proxmox runs it as root); a missing hookscript stops a guest start, so never `Register` without `SnippetReady` |
|
||||
|
||||
### Disk / format safety (role gates, durable IDs, format guards)
|
||||
@@ -41,6 +42,11 @@
|
||||
| `storage.ResolveStorageDevice` | internal/storage/durable_device.go | `ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)` | re-resolve enrolled STORAGE drives (remount) | `uuid:` scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node |
|
||||
| `deriveDurableID` | internal/storage/durableid.go | `deriveDurableID(typ, s, backingDevice, uuid) string` | storage-target durable id (DR re-attach key) | Deterministic per type; `uuid:` for usb/local-dir; PBS id includes `#<fingerprint>` |
|
||||
| `SystemDisks` / `isSystemBacked` / `RoleForStorage` / `RoleForRawDevice` | internal/storage/role.go | `RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole` | protection-tier classification | Fails safe to `system` (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied |
|
||||
| `storage.IdentityResolver` / `Relation` / `MountFor` | internal/storage/identity.go | `HostPath(path) PlaceIdentity`, `Device(dev) PlaceIdentity`, `Relation(a, b) (different, proven, reason)` | physical-disk identity of a storage place (felhom.eu `16-storage-physical-identity.md` §3, v0.157.0) | Reads /sys + /run/udev/data + /proc/mounts only (no block I/O). Judge a drive ONLY on its own mount (`localapi.ownMountIdentity`) — never the containing fs, or an absent drive reads as the root disk. A shared disk key is "same, proven" even when unknown. Pinned by identity_test.go (fixture trees) + samedisk_test.go |
|
||||
| `localapi.Server.EnsureInternalPlace` / `InternalPlaceRecord` | internal/localapi/internalplace.go | `EnsureInternalPlace(ctx)` (startup + 20 s tick); `ReadInternalPlace` / `WriteInternalPlace` | the one-disk box's internal data place (felhom.eu `16` §5, v0.158.0): mounts it at /mnt/belso by fs-UUID like an enrolled drive, records it `enrolled`; the controller binds + registers it | Never takes /mnt/belso from a household drive (a foreign mount, or a unit naming another UUID, waits). Never formats/ejects (dm on the system disk → role system). The ONLY maker is the root helper `configs/felhom-internal-place` (installer + signed `add_internal_place`). Pinned by internalplace_test.go |
|
||||
| `configs/felhom-internal-place` | configs/ (bundle → /usr/local/sbin) | `plan\|create <planned GiB>` | the ruled size rule (reserve max(16, 10 %) ≤ 48 GiB; ≥ 20 GiB or nothing) + the only lvcreate/mkfs of a system-disk volume | Formats only a felhom-belso with NO signature; refuses a foreign one; never resizes/removes. Sudoers: `^(plan\|create) 0$` only. Pinned by configs/test_felhom_internal_place.py |
|
||||
| `storage.candidatePartitions` / `CandidatePartition` | internal/storage/partitions.go | `CandidateDisk.Partitions` | partitions as places (`16` §6): empty / filesystem / in_use per part of a free disk; `Formattable` only while no node of the disk is mounted | A part with children or a member type is never listed; a raw /mnt mount counts as Felhom's only when corroborated (felhomOwnedMounts) — for LISTING only: the root mkfs wrapper trusts no such claim (v0.158.1). Pinned by partitions_test.go |
|
||||
| partition wipe id `byid:<…-partN>+fsuuid:<uuid>` | internal/storage/durable_device.go | `DeviceDurableID` / `ResolveDurableDevice` | a data-bearing PART binds disk id + part number + filesystem (`16` §6) | Both halves must resolve to the SAME device or the wipe refuses. Whole disks keep the plain id. Pinned by partition_durable_test.go |
|
||||
| `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process |
|
||||
| `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` |
|
||||
|
||||
@@ -51,6 +57,7 @@
|
||||
| `SudoHostOps.EnsureMount` | internal/storage/hostops.go | `EnsureMount(ctx, MountSpec) error` | persistent by-UUID systemd .mount | Validate→render→stage→`install`→`enable --now`; idempotent |
|
||||
| `SudoHostOps.Unmount` | internal/storage/hostops.go | `Unmount(ctx, where) error` | detach a mount unit | DESTRUCTIVE — caller MUST have gated it; does not self-authorize |
|
||||
| `SudoHostOps.ReassertEnrolledMounts` | internal/storage/hostops.go | `ReassertEnrolledMounts(ctx)` | reboot remount (re-resolve by UUID) | Re-asserts unless mounted AND enabled (`shouldReassertMount`); skips absent UUIDs |
|
||||
| `SudoHostOps.ForgetRetiredDrives` (v0.159.0, R-1124) | internal/storage/forget_retired.go | `ForgetRetiredDrives(ctx, IntentReader) int` | drop the host mount unit of a DECOMMISSIONED drive that is GONE (start-up + the 20 s tick) | Only decommissioned + absent + unmounted; a present decommissioned drive keeps its unit (H3 re-enroll). The file goes only through `felhom-priv-apply unit-forget`, which re-checks all three as root |
|
||||
| `GuestBinder.AttachDrive` / `DetachDrive` | internal/localapi/intermediary.go | `AttachDrive(ctx, vmid, where) (guestPath, err)` | live drive hot-swap under `/mnt/felhom-drives` | Normalizes to EXACTLY ONE bind via `countHostMounts` (converges double-binds); force re-bind when guest can't see it |
|
||||
| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
|
||||
| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
|
||||
@@ -158,6 +165,7 @@
|
||||
| `proxmox.Runner` | internal/proxmox/privileged.go | `*ExecRunner` (direct/sudo) | `mockRunner` internal/proxmox/mock_test.go; runner fakes in storage tests |
|
||||
| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
|
||||
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
|
||||
| `storage.IdentityHost` | internal/storage/identity.go | `*SysfsIdentityHost` | fixture /sys + udev trees in identity_test.go (`newFx`); `fakeIdentHost` in internal/localapi/samedisk_test.go. A SEPARATE interface from HostReader on purpose — adding methods to HostReader would break every existing fake. |
|
||||
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
|
||||
| `lanresolver.hostRoot` + `dnsmasqUnitPaths` (data seam, R-317) | internal/lanresolver/lanresolver.go | prod `hostRoot = "/"`; probe = the `dnsmasq` package's systemd UNIT, never `/usr/sbin/dnsmasq` (owned by `dnsmasq-base`) | internal/lanresolver/ensure_dnsmasq_test.go — fixture root tree + recording `proxmox.Runner`; the REAL `os.Stat` probe and `EnsureDnsmasq` run. `TestEnsureDnsmasq_ProductionProbeIsTheUnit` pins the production wiring |
|
||||
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
||||
|
||||
+116
-9
@@ -50,6 +50,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbs"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poke"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
@@ -58,6 +59,8 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/thincheck"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/vzpartial"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/wgtunnel"
|
||||
)
|
||||
|
||||
@@ -775,6 +778,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
hostReader := storage.NewProcHostReader()
|
||||
hostOps := newHostOps(cfg, logger)
|
||||
observer := storage.NewObserver(px, hostReader, hostOps, logger)
|
||||
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
|
||||
// Backup + restore-test state store (slice 6): holds the latest backup-per-target +
|
||||
// latest restore-test result; the collector reads it via the BackupReporter /
|
||||
// RestoreTestReporter seams; the cadence scheduler writes it.
|
||||
@@ -833,7 +837,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
|
||||
// already carries the gated view — v0.86.0.)
|
||||
collector.SetCapabilityProber(probeAll)
|
||||
// R-998 (doc 14): the household power-off record of THIS boot rides every report until the box is down.
|
||||
powerOffState := poweroff.New(filepath.Join(cfg.OOB.WithDefaults().StateDir, "poweroff.json"), poweroff.ProcBtime)
|
||||
collector.SetDeliberateOffSource(func() *hub.DeliberateOff {
|
||||
if r := powerOffState.Current(); r != nil {
|
||||
return &hub.DeliberateOff{RequestedAt: r.RequestedAt.UTC().Format(time.RFC3339)}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
loop := hub.NewLoop(collector, client, time.Duration(hcfg.PollSeconds)*time.Second, logger)
|
||||
// R-998: the power-off path waits until the hub ACCEPTED a report carrying the record.
|
||||
powerOffAck := make(chan struct{}, 1)
|
||||
loop.SetReportedHook(func(r *hub.HostReport) {
|
||||
if r != nil && r.DeliberateOff != nil {
|
||||
select {
|
||||
case powerOffAck <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
})
|
||||
interval := time.Duration(hcfg.PollSeconds) * time.Second
|
||||
// Heartbeat log-pull (v0.83.0): when the control envelope requests it, the NEXT
|
||||
// heartbeat carries the debug ring's tail (newest-kept, byte-capped in the loop).
|
||||
@@ -938,10 +960,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// state. Gates the watchdog's self-heal re-mount to ENROLLED drives, and the local API records
|
||||
// enroll/eject into it. Open failure → ungated legacy remount (logged) so the daemon still runs.
|
||||
var intentReader storage.IntentReader
|
||||
intentStateDir := "/var/lib/felhom-agent"
|
||||
if cfg.Authz.NonceStorePath != "" {
|
||||
intentStateDir = filepath.Dir(cfg.Authz.NonceStorePath) // share the agent's durable state dir
|
||||
}
|
||||
intentStateDir := agentStateDir(cfg)
|
||||
intentStore, ierr := storage.OpenIntentStore(filepath.Join(intentStateDir, "drive-intents.json"))
|
||||
if ierr != nil {
|
||||
logger.Warn("storage: intent store unavailable — self-heal runs UNGATED (legacy remount-any)", "err", ierr)
|
||||
@@ -1073,6 +1092,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
decommIntent = intentStore
|
||||
}
|
||||
decommExec := signedjobs.NewDecommissionExecutor(decommIntent, logger)
|
||||
if guestBindStore != nil { // typed-nil guard, as above
|
||||
decommExec.SetGuestBinds(guestBindStore) // v0.159.1: prune the bind record like the self-serve decommission
|
||||
}
|
||||
|
||||
// Agent self-update (TASK D1): the agent_update executor downloads the operator-signed binary,
|
||||
// verifies it against the SIGNED sha256, and hands it to the root guarded wrapper (A/B flip +
|
||||
@@ -1147,7 +1169,11 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||
}}
|
||||
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, kernelExec, bundleExec}, cfg.Hub.HostID, logger)
|
||||
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
|
||||
// to the local API server built below).
|
||||
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
|
||||
localapi.InternalPlaceFileIn(intentStateDir), logger)
|
||||
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, kernelExec, bundleExec, internalPlaceExec}, cfg.Hub.HostID, logger)
|
||||
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
|
||||
|
||||
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
|
||||
@@ -1180,6 +1206,35 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localSrv != nil {
|
||||
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
|
||||
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
|
||||
pwMode := proxmox.RunnerMode(cfg.Privileged.Mode)
|
||||
if pwMode == "" {
|
||||
pwMode = proxmox.RunnerSudo
|
||||
}
|
||||
localSrv.SetHostPowerOff(localapi.HostPowerOffConfig{
|
||||
State: powerOffState,
|
||||
Power: localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: pwMode, SudoPath: cfg.Privileged.SudoPath}, logger),
|
||||
ReportNow: func(ctx context.Context) bool {
|
||||
select {
|
||||
case storageTrigger <- struct{}{}:
|
||||
default: // a report is already queued; it runs after the record was written
|
||||
}
|
||||
select {
|
||||
case <-powerOffAck:
|
||||
return true
|
||||
case <-ctx.Done():
|
||||
return false
|
||||
}
|
||||
},
|
||||
Uptime: func() (time.Duration, bool) {
|
||||
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
|
||||
if err != nil || b <= 0 {
|
||||
return 0, false
|
||||
}
|
||||
return time.Since(time.Unix(b, 0)), true
|
||||
},
|
||||
})
|
||||
localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) {
|
||||
// Let the controller finish bringing its apps back after the backup, then run (still under the gate).
|
||||
select {
|
||||
@@ -1370,7 +1425,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// filesystem UUID and re-mounts (idempotent `enable --now`) so a letter reshuffle is a no-op.
|
||||
// Type-asserted (the concrete op exposes it; the interface stays lean).
|
||||
mountReasserter := hostOps // concrete *storage.SudoHostOps — exposes ReassertEnrolledMounts
|
||||
// v0.158.0 (felhom.eu `16` §5): the internal data place is mounted like an enrolled drive — first, so the
|
||||
// re-assert below sees its unit.
|
||||
localSrv.EnsureInternalPlace(ctx)
|
||||
if mountReasserter != nil {
|
||||
// R-1124: a decommissioned drive that is gone loses its host mount unit first (start-up and every tick), so
|
||||
// no later boot waits for it and the re-assert below no longer lists it.
|
||||
mountReasserter.ForgetRetiredDrives(ctx, intentReader)
|
||||
mountReasserter.ReassertEnrolledMounts(ctx)
|
||||
}
|
||||
// F9: on startup (the host's bring-up/reconcile trigger), re-assert any enrolled guest data-drive
|
||||
@@ -1407,7 +1468,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
case <-t.C:
|
||||
// Re-assert mounts first (handles a USB that enumerated late after a host reboot, or a
|
||||
// unit re-disabled at runtime), then re-assert the guest binds against the now-live mounts.
|
||||
localSrv.EnsureInternalPlace(ctx) // `16` §5: a record written since the last tick (the signed job)
|
||||
if mountReasserter != nil {
|
||||
mountReasserter.ForgetRetiredDrives(ctx, intentReader) // R-1124: decommissioned + gone → unit removed
|
||||
mountReasserter.ReassertEnrolledMounts(ctx)
|
||||
}
|
||||
localSrv.ReassertGuestBinds(ctx)
|
||||
@@ -1485,6 +1548,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// the dangling snapshot → start iff onboot). Runs BEFORE the backup loop starts, so a present
|
||||
// backup lock is stale by definition (guarded by a no-vzdump-running check; fail-safe otherwise).
|
||||
localSrv.RecoverStaleLockedGuests(ctx)
|
||||
// R-1006: then remove a dead half-written backup a power cut left (the lock recovery first: same order as a human).
|
||||
localSrv.SweepVzdumpPartials(ctx)
|
||||
// F-REBOOT: the startup recovery above only covers a guest left LOCKED by an interrupted
|
||||
// backup. A guest that simply ends up stopped-and-unlocked (a `pct reboot` whose shutdown
|
||||
// half completed and whose start half never fired — Campaign 8 fault 11, 9m47s of total
|
||||
@@ -1845,6 +1910,8 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
}
|
||||
apiTiers := make([]localapi.BackupTier, 0, len(backupTiers))
|
||||
var runner *backup.BackupRunner
|
||||
var tierRunners []*backup.BackupRunner
|
||||
var tierTargets []string
|
||||
for _, t := range backupTiers {
|
||||
prune := ""
|
||||
if t.KeepLast > 0 {
|
||||
@@ -1857,6 +1924,8 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
if t.Primary {
|
||||
runner = r
|
||||
}
|
||||
tierRunners = append(tierRunners, r)
|
||||
tierTargets = append(tierTargets, t.TargetID)
|
||||
apiTiers = append(apiTiers, localapi.BackupTier{
|
||||
TargetID: t.TargetID,
|
||||
Cadence: t.Cadence,
|
||||
@@ -1946,6 +2015,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
||||
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
|
||||
Identity: storage.NewIdentityResolver(), // v0.157.0: `16` §3 identity on /disks + POST /disks/same-disk
|
||||
Tokens: tokens,
|
||||
BackupCadence: cfg.Backup.BackupCadence(),
|
||||
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
|
||||
@@ -1967,10 +2037,12 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
// F2-b: recover a guest left with a stale vzdump lock by a reboot-during-backup. Reads + start
|
||||
// go through the API client; the `pct unlock` is the one fenced root-CLI op (no API equivalent).
|
||||
// A1 (v0.62.0): the scan is restricted to felhom-pool members (ownership proven, not assumed).
|
||||
StaleLock: localapi.NewStaleLockController(px, &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, reconcile.DefaultPool, logger),
|
||||
Intent: intent, // slice 10 P3: record enroll/eject intent for self-heal
|
||||
GuestBinds: guestBinds, // F9: per-guest bind record for the startup re-assert
|
||||
FormatJobs: formatJobs, // F20-BUG3: detached-format job record + restart recovery
|
||||
StaleLock: localapi.NewStaleLockController(px, &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, reconcile.DefaultPool, logger),
|
||||
Intent: intent, // slice 10 P3: record enroll/eject intent for self-heal
|
||||
// v0.158.0 (felhom.eu `16` §5): the internal data place record, beside the intent store.
|
||||
InternalPlaceFile: localapi.InternalPlaceFileIn(agentStateDir(cfg)),
|
||||
GuestBinds: guestBinds, // F9: per-guest bind record for the startup re-assert
|
||||
FormatJobs: formatJobs, // F20-BUG3: detached-format job record + restart recovery
|
||||
|
||||
// Host metrics (slice 9): the shared collector serves GET /host/metrics — a fresh host +
|
||||
// per-storage view to the customer's monitoring page (reuses the slice-4 collector).
|
||||
@@ -1985,6 +2057,30 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
logger.Warn("daemon: local-api disabled (server build)", "err", err)
|
||||
return nil
|
||||
}
|
||||
// R-1006: a dead half-written vzdump (a power cut mid-backup) is removed at start-up and before every backup of the
|
||||
// guest on that target, through the root helper; the in-flight answer is R-1008's (a task from before the boot is dead).
|
||||
cleaner := &vzpartial.Cleaner{
|
||||
API: px,
|
||||
Runner: &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath},
|
||||
InFlight: srv.VzdumpInFlight,
|
||||
Boot: func() (int64, bool) {
|
||||
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
|
||||
return b, err == nil && b > 0
|
||||
},
|
||||
Logger: logger,
|
||||
}
|
||||
// R-1012: an orphan thin device (a full host root during a snapshot) is NAMED in the backup failure the controller
|
||||
// mails to the operator, through the read-only root helper. Never removed — the repair stays a human step.
|
||||
thinChecker := &thincheck.Checker{API: px, Runner: &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, Logger: logger}
|
||||
for _, r := range tierRunners {
|
||||
r.SetBeforeBackup(func(ctx context.Context, storage string, vmid int) { cleaner.Clean(ctx, storage, vmid) })
|
||||
r.SetThinCheck(func(ctx context.Context, vmid int) string { return thincheck.Describe(thinChecker.Check(ctx, vmid)) })
|
||||
}
|
||||
srv.SetPartialSweep(func(ctx context.Context, vmid int) {
|
||||
for _, t := range tierTargets {
|
||||
cleaner.Clean(ctx, t, vmid)
|
||||
}
|
||||
})
|
||||
return srv
|
||||
}
|
||||
|
||||
@@ -2114,6 +2210,7 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger)
|
||||
return 1
|
||||
}
|
||||
observer := storage.NewObserver(px, storage.NewProcHostReader(), newHostOps(cfg, logger), logger)
|
||||
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
|
||||
// Wire the LIVE PBS reporter here too (v0.39.0): selftest=hub is a separate one-shot process — no
|
||||
// verify loop runs — so a nil reporter previously yielded pbs_snapshots:[] and an absent dr_recipe
|
||||
// pbs coord. The live reporter lists snapshots directly (fresh store, last-known-good fallback) so
|
||||
@@ -2170,6 +2267,7 @@ func runSelftestStorage(ctx context.Context, cfg config.Config, logger *slog.Log
|
||||
hostReader := storage.NewProcHostReader()
|
||||
hostOps := newHostOps(cfg, logger)
|
||||
observer := storage.NewObserver(px, hostReader, hostOps, logger)
|
||||
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
|
||||
|
||||
octx, cancel := context.WithTimeout(ctx, 60*time.Second)
|
||||
fmt.Printf("=== felhom-agent %s selftest=storage ===\n", version)
|
||||
@@ -3827,3 +3925,12 @@ func (f *factsReporter) SystemFacts(ctx context.Context) (int, json.RawMessage,
|
||||
f.facts, f.err = f.leg.Facts(ctx, f.vmid)
|
||||
return f.vmid, f.facts, f.err
|
||||
}
|
||||
|
||||
// agentStateDir is the agent's durable state dir: beside the nonce store when one is configured, else the default.
|
||||
// The intent store, the guest-bind record and the internal place record (v0.158.0) all live here.
|
||||
func agentStateDir(cfg config.Config) string {
|
||||
if cfg.Authz.NonceStorePath != "" {
|
||||
return filepath.Dir(cfg.Authz.NonceStorePath)
|
||||
}
|
||||
return "/var/lib/felhom-agent"
|
||||
}
|
||||
|
||||
@@ -43,7 +43,7 @@ func main() {
|
||||
|
||||
func run() error {
|
||||
var (
|
||||
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update")
|
||||
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update | add_internal_place")
|
||||
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
|
||||
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
|
||||
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
|
||||
|
||||
@@ -25,6 +25,7 @@
|
||||
|
||||
Cmnd_Alias FELHOM_MOUNT = \
|
||||
/usr/local/sbin/felhom-priv-apply ^unit mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$, \
|
||||
/usr/local/sbin/felhom-priv-apply ^unit-forget mnt-[A-Za-z0-9_.\\-]+\.mount$, \
|
||||
/usr/bin/systemctl daemon-reload, \
|
||||
/usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
|
||||
/usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
|
||||
@@ -80,7 +81,34 @@ Cmnd_Alias FELHOM_DNSMASQ = \
|
||||
Cmnd_Alias FELHOM_GUESTHOOK = \
|
||||
/usr/sbin/pct ^set [0-9]+ --hookscript local\:snippets/felhom-guest-hook\.sh$, \
|
||||
/usr/sbin/pct ^set [0-9]+ --delete mp[0-9]+$, \
|
||||
/usr/sbin/pct ^reboot [0-9]+$
|
||||
/usr/sbin/pct ^reboot [0-9]+$, \
|
||||
/usr/sbin/pct ^shutdown [0-9]+ --timeout [0-9]+$
|
||||
|
||||
# Host power-off (R-998, doc 14): the household's „A gép kikapcsolása" — guest shutdown above, then this one fixed
|
||||
# vector. Guarded agent-side (10 min after boot, one per boot, never during a backup/restore/update/drive job).
|
||||
Cmnd_Alias FELHOM_POWEROFF = \
|
||||
/usr/bin/systemctl poweroff
|
||||
|
||||
# Dead half-written vzdump removal (R-1006, power-cut-fixes-2026-10-14): a power cut mid-backup leaves
|
||||
# vzdump-lxc-<vmid>-<date>.tar.dat + .tmp/ on the dir storage; nothing removed them and every later backup was skipped
|
||||
# „not enough space". ONE root helper (configs/felhom-vzdump-partial-clean, from the config bundle) re-checks everything
|
||||
# as root — a dir storage from storage.cfg, the exact name of THAT vmid, not in use, dead — and refuses otherwise. The
|
||||
# arguments here are its exact shape (storage id, vmid, one half-written name); the vmid-in-name match is the helper's.
|
||||
Cmnd_Alias FELHOM_VZPARTIAL = \
|
||||
/usr/local/sbin/felhom-vzdump-partial-clean ^[A-Za-z][A-Za-z0-9_-]{0\,63} [1-9][0-9]{2\,8} vzdump-(lxc|qemu)-[1-9][0-9]{2\,8}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$
|
||||
|
||||
# Internal data place („Belső tárhely", felhom.eu architecture/16 §5, §7; agent v0.158.0). ONE root helper
|
||||
# (configs/felhom-internal-place, from the config bundle) creates <vg>/felhom-belso in the local-lvm thin pool by the
|
||||
# ruled size rule and formats ONLY that volume, ONLY when it carries no signature. The agent calls it from the signed
|
||||
# `add_internal_place` job (an old box), always with 0 GiB planned: its guest's volumes already exist. `plan` changes
|
||||
# nothing. The installer runs the helper as root itself (a new box).
|
||||
# R-1012 (agent v0.159.0): the READ-ONLY orphan-thin check before and after a whole-guest backup — lists the pool's thin
|
||||
# device ids (through a metadata snapshot it reserves and releases) next to LVM's. It sends no other pool message; the
|
||||
# repair stays a human step. Pinned by configs/test_felhom_thin_check.py.
|
||||
Cmnd_Alias FELHOM_THINCHECK = \
|
||||
/usr/local/sbin/felhom-thin-check ^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0\,63} [A-Za-z0-9_+][A-Za-z0-9_.+-]{0\,63}$
|
||||
Cmnd_Alias FELHOM_INTERNALPLACE = \
|
||||
/usr/local/sbin/felhom-internal-place ^(plan|create) 0$
|
||||
|
||||
# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
|
||||
# /mnt/felhom-drives (self-bind + make-shared) and binds/unbinds each drive's felhom-data namespace UNDERNEATH it so the
|
||||
@@ -314,4 +342,4 @@ Cmnd_Alias FELHOM_GUESTNET = \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$
|
||||
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY, FELHOM_POWEROFF, FELHOM_VZPARTIAL, FELHOM_INTERNALPLACE, FELHOM_THINCHECK
|
||||
|
||||
Executable
+118
@@ -0,0 +1,118 @@
|
||||
#!/bin/bash
|
||||
# felhom-internal-place — make the one-disk box's internal data place („Belső tárhely"), felhom.eu
|
||||
# architecture/16-storage-physical-identity.md §5, §7 (sizes ruled by the operator 2026-10-16). Installed as
|
||||
# /usr/local/sbin/felhom-internal-place (config bundle). Called by felhom-host-install.sh (a new box) and by the agent's
|
||||
# signed `add_internal_place` job (an old box) — ONE implementation of the size rule and of the only mkfs it does.
|
||||
#
|
||||
# felhom-internal-place plan <planned GiB> → prints the numbers, changes nothing
|
||||
# felhom-internal-place create <planned GiB> → makes the place (or finds it), prints the result
|
||||
#
|
||||
# <planned GiB> is thin space still to be promised to the guest (rootfs + data volume) that does not exist yet:
|
||||
# the installer passes 32 + the data volume; an old box passes 0 (its guest's volumes are already counted).
|
||||
#
|
||||
# The rule (16 §7, ruled): reserve = max(16 GiB, 10 % of the pool), at most 48 GiB; internal = pool − reserve −
|
||||
# every thin volume already promised in the pool − <planned>, made only when ≥ 20 GiB. So nothing in the pool is ever
|
||||
# promised beyond pool − reserve: a pool at 100 % stops every volume on the box.
|
||||
#
|
||||
# It trusts nothing it is told. Every rule is checked here, as root, and any doubt REFUSES (exit 1):
|
||||
# - the pool is the `lvmthin: local-lvm` section of /etc/pve/storage.cfg (its vgname/thinpool), nothing else;
|
||||
# - the only volume it creates, formats or reads is <vg>/felhom-belso;
|
||||
# - it formats ONLY a felhom-belso that carries no signature at all (blkid -p finds nothing) — the one it just made,
|
||||
# or one a crash left between lvcreate and mkfs; any other content REFUSES; it never resizes or removes anything.
|
||||
# Output, one line: „created uuid=<fs-uuid> gib=<n>", „formatted uuid=… gib=…" (a crash left it empty), „existing uuid=<fs-uuid> gib=<n>", „skipped <why>" (exit 0),
|
||||
# or „plan pool=… reserve=… promised=… planned=… internal=…". Pinned by configs/test_felhom_internal_place.py.
|
||||
set -u
|
||||
STORAGE_CFG=${FELHOM_STORAGE_CFG:-/etc/pve/storage.cfg}
|
||||
LV_NAME=felhom-belso
|
||||
LABEL=felhom-belso
|
||||
MIN_GIB=20
|
||||
|
||||
die() { echo "felhom-internal-place: REFUSED: $*" >&2; exit 1; }
|
||||
|
||||
pool_of_local_lvm() { # prints "<vg> <thinpool>" of the lvmthin storage local-lvm, or nothing
|
||||
awk '
|
||||
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (type == "lvmthin" && id == "local-lvm"); next }
|
||||
in_s && /^[ \t]+vgname[ \t]/ { vg = $2 }
|
||||
in_s && /^[ \t]+thinpool[ \t]/ { tp = $2 }
|
||||
END { if (vg != "" && tp != "") print vg, tp }
|
||||
' "$STORAGE_CFG"
|
||||
}
|
||||
|
||||
gib_floor() { echo $(( $1 / 1073741824 )); }
|
||||
|
||||
[[ $# -eq 2 ]] || die "usage: felhom-internal-place plan|create <planned GiB>"
|
||||
MODE=$1; PLANNED=$2
|
||||
[[ "$MODE" == plan || "$MODE" == create ]] || die "mode must be plan or create"
|
||||
[[ "$PLANNED" =~ ^(0|[1-9][0-9]{0,5})$ ]] || die "planned GiB must be a whole number"
|
||||
|
||||
read -r VG POOL < <(pool_of_local_lvm) || true
|
||||
[[ -n "${VG:-}" && -n "${POOL:-}" ]] || die "no lvmthin storage local-lvm in $STORAGE_CFG"
|
||||
[[ "$VG" =~ ^[A-Za-z0-9_.+-]+$ && "$POOL" =~ ^[A-Za-z0-9_.+-]+$ ]] || die "unexpected vg/pool name"
|
||||
|
||||
pool_b=$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$POOL" 2>/dev/null | tr -d ' ')
|
||||
[[ "$pool_b" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$POOL"
|
||||
pool_gib=$(gib_floor "$pool_b")
|
||||
|
||||
# Every thin volume promised in the pool (its virtual size) — except the place itself and vzdump's short-lived
|
||||
# snapshots (snap_*_vzdump), which exist only during a whole-box copy and are what the reserve is for.
|
||||
# FAIL CLOSED: a failed or empty listing must never read as „nothing promised" (that would size the place too big).
|
||||
listing=$(lvs --noheadings --nosuffix --units b -o lv_name,pool_lv,lv_size "$VG" 2>/dev/null) || die "cannot list the volumes of $VG"
|
||||
pool_seen=false; promised_b=0; have_lv=false
|
||||
while read -r name pl size; do
|
||||
[[ -n "${name:-}" ]] || continue
|
||||
[[ "$name" == "$POOL" ]] && pool_seen=true
|
||||
[[ "$pl" == "$POOL" ]] || continue
|
||||
if [[ "$name" == "$LV_NAME" ]]; then have_lv=true; continue; fi
|
||||
[[ "$name" =~ ^snap_.*_vzdump$ ]] && continue
|
||||
[[ "$size" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$name"
|
||||
promised_b=$(( promised_b + size ))
|
||||
done <<< "$listing"
|
||||
$pool_seen || die "the listing of $VG does not contain the pool $POOL — refusing to guess what is promised"
|
||||
promised_gib=$(( (promised_b + 1073741823) / 1073741824 )) # rounded UP: never promise a partial GiB twice
|
||||
|
||||
reserve=$(( pool_gib / 10 )); (( reserve < 16 )) && reserve=16; (( reserve > 48 )) && reserve=48
|
||||
internal=$(( pool_gib - reserve - promised_gib - PLANNED ))
|
||||
|
||||
if [[ "$MODE" == plan ]]; then
|
||||
echo "plan pool=$pool_gib reserve=$reserve promised=$promised_gib planned=$PLANNED internal=$internal exists=$have_lv"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
DEV=/dev/$VG/$LV_NAME
|
||||
is_block() { [[ "$(stat -L -c %F "$1" 2>/dev/null)" == "block special file" ]]; }
|
||||
fs_uuid() { blkid -p -s UUID -o value "$DEV" 2>/dev/null; }
|
||||
fs_type() { blkid -p -s TYPE -o value "$DEV" 2>/dev/null; }
|
||||
fs_label() { blkid -p -s LABEL -o value "$DEV" 2>/dev/null; }
|
||||
# has_signature: FAIL CLOSED. blkid -p exits 0 when it found something and 2 when it found NOTHING; any other result (an
|
||||
# error, an ambivalent probe, blkid missing) must never read as „empty" — that is the only gate in front of mkfs.
|
||||
has_signature() {
|
||||
local rc=0
|
||||
blkid -p -o value "$DEV" >/dev/null 2>&1 || rc=$?
|
||||
case $rc in
|
||||
0) return 0 ;;
|
||||
2) return 1 ;;
|
||||
*) die "cannot probe $DEV for a signature (blkid exit $rc) — refusing" ;;
|
||||
esac
|
||||
}
|
||||
lv_gib() { gib_floor "$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$LV_NAME" 2>/dev/null | tr -d ' ')"; }
|
||||
|
||||
if $have_lv; then
|
||||
is_block "$DEV" || die "$DEV is not an active block device"
|
||||
t=$(fs_type)
|
||||
if [[ -n "$t" ]]; then
|
||||
[[ "$t" == ext4 && "$(fs_label)" == "$LABEL" ]] || die "$DEV holds something that is not Felhom's ($t) — left alone"
|
||||
echo "existing uuid=$(fs_uuid) gib=$(lv_gib)"; exit 0
|
||||
fi
|
||||
# A felhom-belso with NO signature: a crash between lvcreate and mkfs. Nothing can be on it — format it.
|
||||
has_signature && die "$DEV carries a signature blkid cannot name — left alone"
|
||||
else
|
||||
(( internal >= MIN_GIB )) || { echo "skipped only $internal GiB free for it (pool $pool_gib, reserve $reserve, promised $promised_gib, planned $PLANNED; needs $MIN_GIB)"; exit 0; }
|
||||
lvcreate -q -y -V "${internal}G" -T "$VG/$POOL" -n "$LV_NAME" >/dev/null || die "lvcreate failed"
|
||||
is_block "$DEV" || die "$DEV did not appear after lvcreate"
|
||||
has_signature && die "the new $DEV already carries a signature — left alone"
|
||||
made=created
|
||||
fi
|
||||
mkfs.ext4 -q -L "$LABEL" "$DEV" >/dev/null || die "mkfs.ext4 failed on $DEV"
|
||||
u=$(fs_uuid)
|
||||
[[ "$u" =~ ^[0-9a-fA-F-]{36}$ ]] || die "no filesystem UUID after mkfs on $DEV"
|
||||
echo "${made:-formatted} uuid=$u gib=$(lv_gib)"
|
||||
@@ -58,14 +58,26 @@ fi
|
||||
|
||||
# 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under
|
||||
# /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed.
|
||||
while read -r mp; do
|
||||
[[ -z "$mp" ]] && continue
|
||||
case "$mp" in
|
||||
/mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
|
||||
*) die "device (or a partition) is mounted at $mp ($dev)" ;;
|
||||
esac
|
||||
done < <(lsblk -nro MOUNTPOINT "$whole" 2>/dev/null || true)
|
||||
|
||||
# Agent v0.158.0: read from the kernel's mount table for EVERY node of the disk (lsblk's MOUNTPOINT shows one mount
|
||||
# per node; a bind elsewhere hid behind it), and a disk whose nodes cannot be listed REFUSES (it passed before).
|
||||
# Deliberately NO exemption for a sibling that is Felhom's own enrolled drive (raw /mnt/<name> + /mnt/felhom-drives):
|
||||
# the only corroboration available here is a mount the agent itself can make, so trusting it would let a compromised
|
||||
# agent unlock a host owner's disk. Consequence (felhom.eu `16` §13): the free part next to an in-use Felhom drive
|
||||
# cannot be formatted; the dashboard does not offer it.
|
||||
PROC_MOUNTS=/proc/mounts
|
||||
node_mounts() { awk -v d="$1" '$1 == d { print $2 }' "$PROC_MOUNTS"; }
|
||||
disk_nodes="$(lsblk -nrpo NAME "$whole" 2>/dev/null || true)"
|
||||
[[ -n "$disk_nodes" ]] || die "cannot list the nodes of $whole — refusing"
|
||||
while read -r node; do
|
||||
[[ -z "$node" ]] && continue
|
||||
while read -r mp; do
|
||||
[[ -z "$mp" ]] && continue
|
||||
case "$mp" in
|
||||
/mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
|
||||
*) die "device (or a partition) is mounted at $mp ($node)" ;;
|
||||
esac
|
||||
done < <(node_mounts "$node")
|
||||
done <<< "$disk_nodes"
|
||||
# 4) read-only device (audit D1): a device the kernel marks RO is never a formattable data disk.
|
||||
wbase="${whole#/dev/}"
|
||||
rof="/sys/block/$wbase/ro"
|
||||
|
||||
@@ -189,6 +189,12 @@ BUNDLE_FILES = [
|
||||
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
|
||||
("/usr/local/sbin/felhom-pbs-apply", "felhom-pbs-apply", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-backup-target-apply", "felhom-backup-target-apply", 0o755, "bash", "replace"),
|
||||
# R-1006 (agent v0.156.0): the only delete of backup files — a dead half-written vzdump, re-checked as root.
|
||||
("/usr/local/sbin/felhom-vzdump-partial-clean", "felhom-vzdump-partial-clean", 0o755, "bash", "replace"),
|
||||
# R-1012 (agent v0.159.0): the read-only orphan-thin check around a whole-guest backup (names an orphan, never removes).
|
||||
("/usr/local/sbin/felhom-thin-check", "felhom-thin-check", 0o755, "bash", "replace"),
|
||||
# felhom.eu architecture/16 §5 (agent v0.158.0): the internal data place — the only lvcreate/mkfs of a system-disk volume.
|
||||
("/usr/local/sbin/felhom-internal-place", "felhom-internal-place", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-os-apply", "felhom-os-apply", 0o755, "python", "replace"),
|
||||
("/usr/local/sbin/felhom-crash-guard", "felhom-crash-guard", 0o755, "python", "replace"),
|
||||
# R-861 (agent v0.146.0): the content checker for every agent-staged file a root program reads, and the two FIXED
|
||||
|
||||
@@ -13,6 +13,8 @@ longer name the destination: each verb has a fixed source and a fixed (or strict
|
||||
|
||||
Verbs (each one sudoers line, exact-match pattern):
|
||||
unit <name> /var/lib/felhom-agent/units/<name> -> /etc/systemd/system/<name> (.mount | .automount)
|
||||
unit-forget <name> REMOVES /etc/systemd/system/<name> (R-1124): only a felhom by-UUID drive .mount unit that is
|
||||
disabled, not mounted, and whose drive is absent (its /dev/disk/by-uuid link is gone)
|
||||
dnsmasq <tmp> <name> /tmp/felhom-resolver-<digits>.conf -> /etc/dnsmasq.d/felhom-<...>.conf
|
||||
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
|
||||
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
|
||||
@@ -41,7 +43,7 @@ WG_SRC, WG_DEST = STATE + "/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"
|
||||
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
|
||||
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
|
||||
MAX_BYTES = 64 * 1024
|
||||
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key", "controller-image")
|
||||
VERBS = ("unit", "unit-forget", "dnsmasq", "wg", "sshd-config", "sshd-key", "controller-image")
|
||||
# R-861 (a) A1 (`09` §3 decision 165): the SAME pattern as the agent's controllerImageRe (internal/localapi/
|
||||
# controllerswap.go) — a compromised agent cannot hand the guest's bootstrap any other image. Pinned by
|
||||
# configs/test_felhom_priv_apply.py ControllerImage.
|
||||
@@ -51,6 +53,11 @@ CONTROLLER_IMAGE_MAX = 256
|
||||
VMID_RE = re.compile(r"^[0-9]{1,9}$")
|
||||
|
||||
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
|
||||
DRIVE_UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.mount$")
|
||||
DRIVE_UNIT_MARKER = "Managed by felhom-agent"
|
||||
NET_UNIT_MARKER = "felhom network storage"
|
||||
WANTS_DIR = UNIT_DIR + "/multi-user.target.wants"
|
||||
BY_UUID_DIR = "/dev/disk/by-uuid"
|
||||
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
|
||||
DNSMASQ_NAME_RE = re.compile(r"^felhom-[a-z0-9][a-z0-9._-]*\.conf$")
|
||||
SEG = r"[A-Za-z0-9_-][A-Za-z0-9_.-]*"
|
||||
@@ -135,6 +142,36 @@ class Host:
|
||||
def read_stdin(self, limit):
|
||||
return sys.stdin.buffer.read(limit + 1)
|
||||
|
||||
# ---- unit-forget (R-1124) ----
|
||||
def read_installed(self, path):
|
||||
"""The installed unit: None when absent; Refused when it is not a regular file (a symlink is never followed)."""
|
||||
try:
|
||||
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
except OSError as e:
|
||||
raise Refused("F1", f"cannot open {path}: {e.strerror}")
|
||||
with os.fdopen(fd, "rb") as f:
|
||||
if not stat.S_ISREG(os.fstat(f.fileno()).st_mode):
|
||||
raise Refused("F1", f"{path} is not a regular file")
|
||||
return f.read(MAX_BYTES + 1)
|
||||
|
||||
def lexists(self, path):
|
||||
return os.path.lexists(path)
|
||||
|
||||
def mountpoints(self):
|
||||
"""Every mount point in /proc/self/mountinfo (field 5, octal escapes decoded)."""
|
||||
out = set()
|
||||
with open("/proc/self/mountinfo", "r", encoding="utf-8", errors="replace") as f:
|
||||
for line in f:
|
||||
parts = line.split(" ")
|
||||
if len(parts) > 4:
|
||||
out.add(re.sub(r"\\([0-7]{3})", lambda m: chr(int(m.group(1), 8)), parts[4]))
|
||||
return out
|
||||
|
||||
def remove(self, path):
|
||||
os.remove(path)
|
||||
|
||||
def write_guest_image(self, vmid, data):
|
||||
"""As root: `pct exec <vmid> -- tee <the fixed file>` with the checked ref on stdin (no shell)."""
|
||||
r = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", CONTROLLER_IMAGE_FILE],
|
||||
@@ -424,6 +461,45 @@ def controller_image(rest, host):
|
||||
return 0
|
||||
|
||||
|
||||
def unit_forget(rest, host):
|
||||
"""R-1124: remove ONE felhom drive mount unit that the agent has already disabled, whose drive is gone. A drive that
|
||||
is still present or still mounted keeps its unit — this verb can only forget, never unmount or take a drive away."""
|
||||
try:
|
||||
if len(rest) != 1 or not DRIVE_UNIT_NAME_RE.match(rest[0]) or "/" in rest[0]:
|
||||
raise Refused("A1", "usage: felhom-priv-apply unit-forget mnt-<escaped path>.mount")
|
||||
name = rest[0]
|
||||
path = os.path.join(UNIT_DIR, name)
|
||||
data = host.read_installed(path)
|
||||
if data is None:
|
||||
host.log(f"felhom-priv-apply: ABSENT unit-forget {path} (nothing to remove)")
|
||||
return 0
|
||||
text = text_of(data, path)
|
||||
if DRIVE_UNIT_MARKER not in text or NET_UNIT_MARKER in text:
|
||||
raise Refused("F2", f"{name} is not a felhom drive unit")
|
||||
check_unit(name, text)
|
||||
s = parse_ini(text, name)
|
||||
what, where = s["Mount"].get("What", ""), s["Mount"]["Where"]
|
||||
mu = re.match(r"^/dev/disk/by-uuid/(.+)$", what)
|
||||
if not mu:
|
||||
raise Refused("F2", f"{name} is not a by-UUID drive unit")
|
||||
if host.lexists(os.path.join(WANTS_DIR, name)):
|
||||
raise Refused("F3", f"{name} is still enabled (disable it first)")
|
||||
if where in host.mountpoints():
|
||||
raise Refused("F4", f"{where} is mounted")
|
||||
if host.lexists(os.path.join(BY_UUID_DIR, mu.group(1))):
|
||||
raise Refused("F5", f"the drive of {name} is present (filesystem UUID {mu.group(1)})")
|
||||
except Refused as e:
|
||||
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] unit-forget {' '.join(rest)[:120]}: {e.reason}")
|
||||
return 2 if e.rule == "A1" else 3
|
||||
try:
|
||||
host.remove(path)
|
||||
except OSError as e:
|
||||
host.log(f"felhom-priv-apply: FAILED unit-forget {path}: {e}")
|
||||
return 4
|
||||
host.log(f"felhom-priv-apply: REMOVED unit-forget {path} (where={where})")
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv, host=None):
|
||||
host = host or Host()
|
||||
if argv == ["--self-check"]:
|
||||
@@ -445,6 +521,8 @@ def main(argv, host=None):
|
||||
return 0
|
||||
if argv and argv[0] == "controller-image":
|
||||
return controller_image(argv[1:], host)
|
||||
if argv and argv[0] == "unit-forget":
|
||||
return unit_forget(argv[1:], host)
|
||||
try:
|
||||
verb, src, dest, mode, checker = plan(argv)
|
||||
data = host.read_source(src)
|
||||
|
||||
Executable
+74
@@ -0,0 +1,74 @@
|
||||
#!/bin/bash
|
||||
# felhom-thin-check — list the thin device ids a thin pool holds next to the ones LVM knows (R-1012). READ-ONLY: it
|
||||
# names an orphan, it never removes one. Installed as /usr/local/sbin/felhom-thin-check (config bundle).
|
||||
#
|
||||
# felhom-thin-check <vg> <pool>
|
||||
#
|
||||
# A full host root while vzdump takes its snapshot can leave a thin snapshot inside the pool that LVM no longer knows
|
||||
# (demo-hp 2026-10-14: `thin_ls` device 4, LVM only 1 and 2); from then on every snapshot backup of that guest fails
|
||||
# („Creation of new snapshot 4 of device 2 failed") and nothing said why. The repair (`dmsetup message … delete <id>`)
|
||||
# stays a HUMAN step, decided by the operator — this helper only reads.
|
||||
#
|
||||
# Reading the live pool's device list needs a metadata snapshot: the kernel's documented way to read metadata while
|
||||
# the pool is in use (`reserve_metadata_snap`, `thin_ls --metadata-snap`, `release_metadata_snap`). It changes no data
|
||||
# and no mapping; it is held for the length of one `thin_ls -o DEV` (well under a second on these pools). Rules:
|
||||
# 1. the arguments are an LVM name pair; <vg>/<pool> is a thin pool (lv_attr starts with `t`);
|
||||
# 2. only one run at a time (a lock file); a metadata snapshot that someone ELSE holds is never touched — the run
|
||||
# refuses (exit 3) and does not release it;
|
||||
# 3. LVM's list is read before AND after the pool's, so a volume created in between is not reported as an orphan
|
||||
# (the caller treats an id as an orphan only when it is in the pool and in neither LVM list).
|
||||
# Output, one record per line: lvm-before <id> <lv> | pool <id> | lvm-after <id> <lv> | ok
|
||||
# Exit: 0 read, 1 refused (arguments / not a thin pool), 3 a metadata snapshot is already held or cannot be reserved,
|
||||
# 4 the pool could not be read. Pinned by configs/test_felhom_thin_check.py.
|
||||
set -u
|
||||
LVS=/usr/sbin/lvs
|
||||
DMSETUP=/usr/sbin/dmsetup
|
||||
THIN_LS=/usr/sbin/thin_ls
|
||||
LOCK=/run/lock/felhom-thin-check.lock
|
||||
NAME_RE='^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0,63}$'
|
||||
|
||||
die() { echo "felhom-thin-check: REFUSED: $1" >&2; exit "${2:-1}"; } # $1 the reason, $2 the exit code
|
||||
|
||||
lvm_ids() { # $1 tag → "<tag> <id> <lv>" for every thin volume of this pool
|
||||
"$LVS" --noheadings --separator , -o lv_name,thin_id --select "pool_lv=$POOL" -- "$VG" 2>/dev/null |
|
||||
awk -F, -v tag="$1" '{ gsub(/ /, ""); if ($2 ~ /^[0-9]+$/) print tag, $2, $1 }'
|
||||
}
|
||||
|
||||
release() { "$DMSETUP" message "$TPOOL" 0 release_metadata_snap >/dev/null 2>&1; }
|
||||
|
||||
main() {
|
||||
[[ $# -eq 2 ]] || die "usage: felhom-thin-check <vg> <pool>"
|
||||
VG=$1 POOL=$2
|
||||
local attr held ids
|
||||
[[ "$VG" =~ $NAME_RE && "$POOL" =~ $NAME_RE ]] || die "not an LVM name pair"
|
||||
|
||||
attr=$("$LVS" --noheadings -o lv_attr -- "$VG/$POOL" 2>/dev/null | tr -d ' ')
|
||||
[[ "$attr" == t* ]] || die "$VG/$POOL is not a thin pool"
|
||||
|
||||
DM="${VG//-/--}-${POOL//-/--}"
|
||||
TPOOL="$DM-tpool"
|
||||
TMETA="/dev/mapper/${DM}_tmeta"
|
||||
|
||||
exec 9>"$LOCK" || die "cannot open the lock file" 4
|
||||
flock -w 30 9 || die "another check holds the lock" 3
|
||||
|
||||
# Field 7 of the thin-pool status is the held metadata root: "-" when no metadata snapshot exists.
|
||||
held=$("$DMSETUP" status "$TPOOL" 2>/dev/null | awk '$3 == "thin-pool" { print $7 }')
|
||||
[[ -n "$held" ]] || die "cannot read the status of $TPOOL" 4
|
||||
[[ "$held" == "-" ]] || die "a metadata snapshot of $TPOOL is already held (by someone else) — not touched" 3
|
||||
|
||||
lvm_ids lvm-before
|
||||
"$DMSETUP" message "$TPOOL" 0 reserve_metadata_snap >/dev/null 2>&1 || die "could not reserve a metadata snapshot of $TPOOL" 3
|
||||
trap release EXIT
|
||||
trap 'release; trap - EXIT; exit 4' INT TERM HUP
|
||||
if ! ids=$("$THIN_LS" --metadata-snap --no-headers -o DEV "$TMETA" 2>/dev/null); then
|
||||
die "thin_ls could not read $TMETA" 4
|
||||
fi
|
||||
release
|
||||
trap - EXIT INT TERM HUP
|
||||
awk '$1 ~ /^[0-9]+$/ { print "pool", $1 }' <<<"$ids"
|
||||
lvm_ids lvm-after
|
||||
echo ok
|
||||
}
|
||||
|
||||
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then main "$@"; fi
|
||||
Executable
+99
@@ -0,0 +1,99 @@
|
||||
#!/bin/bash
|
||||
# felhom-vzdump-partial-clean — remove ONE dead, half-written vzdump backup (R-1006). The only delete the agent's
|
||||
# sudoers allows for backup files. Installed as /usr/local/sbin/felhom-vzdump-partial-clean (config bundle).
|
||||
#
|
||||
# felhom-vzdump-partial-clean <storage> <vmid> <name>
|
||||
#
|
||||
# A power cut during the night backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` (+ its `.tmp/` directory) on the
|
||||
# storage; nothing removes it, every later backup is then skipped „not enough space", and a full host root breaks
|
||||
# LVM's own metadata writes (power-cut-fixes-2026-10-14 C4). This helper removes such a leftover — and NOTHING else.
|
||||
# It trusts nothing the agent says; every rule is checked here, as root, and any doubt REFUSES (exit 1):
|
||||
# 1. the arguments: a storage id, a vmid, and a name that is exactly
|
||||
# vzdump-(lxc|qemu)-<THAT vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp) — never a finished archive
|
||||
# (.tar.zst, .vma.zst, …), never a .log/.notes, never another vmid;
|
||||
# 2. the storage: its section in /etc/pve/storage.cfg is `dir: <storage>` with a `path`; the target is
|
||||
# <path>/dump/<name>, not a symlink, and its real directory IS <path>/dump;
|
||||
# 3. not in use: no vzdump runs on the host at all (a task `…:vzdump:…` or a `vzdump` command — a job for several
|
||||
# guests carries no vmid in its task name), and no
|
||||
# process holds the target (or, for the .tmp directory, anything under it) open or as its working directory
|
||||
# (C2: after a killed worker, tar and zstd kept writing into the .tar.dat for 4 min 40 s, none named „vzdump");
|
||||
# 4. dead: last modified before this boot (`btime` in /proc/stat), or untouched for at least QUIET_S seconds.
|
||||
# Prints one line on success: „removed <name> <bytes> B (<why>)". Pinned by configs/test_felhom_vzdump_partial_clean.py.
|
||||
set -u
|
||||
STORAGE_CFG=/etc/pve/storage.cfg
|
||||
PROC=/proc
|
||||
QUIET_S=120
|
||||
|
||||
die() { echo "felhom-vzdump-partial-clean: REFUSED: $*" >&2; exit 1; }
|
||||
|
||||
storage_path() { # $1 storage id → prints the dir storage's path, or nothing
|
||||
awk -v want="$1" '
|
||||
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (id == want); next }
|
||||
in_s && /^[ \t]+path[ \t]/ { if (type == "dir") { sub(/^[ \t]+path[ \t]+/, ""); sub(/[ \t]+$/, ""); print; exit } }
|
||||
' "$STORAGE_CFG"
|
||||
}
|
||||
|
||||
storage_type() {
|
||||
awk -v want="$1" '/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); id = h[2]; sub(/[ \t]+$/, "", id); if (id == want) { print h[1]; exit } }' "$STORAGE_CFG"
|
||||
}
|
||||
|
||||
boot_time() { awk '$1 == "btime" { print $2 }' "$PROC/stat"; }
|
||||
|
||||
newest_mtime() { # a file's mtime, or the newest mtime of anything in a directory (the dir itself included)
|
||||
if [[ -d "$1" ]]; then find "$1" -xdev -printf '%T@\n' 2>/dev/null | sort -n | tail -1 | cut -d. -f1
|
||||
else stat -c %Y "$1"; fi
|
||||
}
|
||||
|
||||
busy_reason() { # $1 vmid, $2 target → prints why it is in use, or nothing
|
||||
local vmid=$1 target=$2 p link cmd
|
||||
for p in "$PROC"/[0-9]*; do
|
||||
[[ -r "$p/cmdline" ]] || continue
|
||||
cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)
|
||||
# ANY vzdump on the host refuses: a job for several guests runs as `task UPID:…:vzdump::…` (no vmid in it), and
|
||||
# its tar may be between this guest's files. Waiting costs one backup run; deleting a live one costs the backup.
|
||||
if [[ "$cmd" == *":vzdump:"* ]] || [[ "$cmd" =~ (^|/)vzdump([[:space:]]|$) ]]; then
|
||||
echo "a vzdump runs on this host (pid ${p##*/})"; return
|
||||
fi
|
||||
for link in "$p/cwd" "$p"/fd/*; do
|
||||
local to
|
||||
to=$(readlink "$link" 2>/dev/null) || continue
|
||||
if [[ "$to" == "$target" || "$to" == "$target/"* ]]; then
|
||||
echo "pid ${p##*/} has it open"; return
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
local storage=${1:-} vmid=${2:-} name=${3:-}
|
||||
[[ $# -eq 3 ]] || die "usage: felhom-vzdump-partial-clean <storage> <vmid> <name>"
|
||||
[[ "$storage" =~ ^[A-Za-z][A-Za-z0-9_-]{0,63}$ ]] || die "bad storage id"
|
||||
[[ "$vmid" =~ ^[1-9][0-9]{2,8}$ ]] || die "bad vmid"
|
||||
[[ "$name" =~ ^vzdump-(lxc|qemu)-${vmid}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$ ]] \
|
||||
|| die "name is not a half-written vzdump of guest $vmid: $name"
|
||||
[[ "$(storage_type "$storage")" == "dir" ]] || die "storage $storage is not a dir storage"
|
||||
local path; path=$(storage_path "$storage")
|
||||
[[ "$path" == /* ]] || die "storage $storage has no path"
|
||||
local dump="$path/dump" target="$path/dump/$name"
|
||||
[[ -e "$target" ]] || die "no such entry: $target"
|
||||
[[ ! -L "$target" ]] || die "a symlink: $target"
|
||||
[[ "$(realpath -e "$dump")" == "$(realpath -e "$(dirname "$target")")" ]] || die "not directly in $dump"
|
||||
if [[ "$name" == *.tmp ]]; then [[ -d "$target" ]] || die ".tmp is not a directory"
|
||||
else [[ -f "$target" ]] || die "not a regular file"; fi
|
||||
local busy; busy=$(busy_reason "$vmid" "$target")
|
||||
[[ -z "$busy" ]] || die "in use: $busy"
|
||||
local bt mt now why
|
||||
bt=$(boot_time); mt=$(newest_mtime "$target"); now=$(date +%s)
|
||||
[[ "$bt" =~ ^[0-9]+$ && "$mt" =~ ^[0-9]+$ ]] || die "cannot read the boot time or the modification time"
|
||||
if (( mt < bt )); then why="written before this boot"
|
||||
elif (( now - mt >= QUIET_S )); then why="untouched for $(( now - mt )) s, no vzdump for $vmid, nothing holds it"
|
||||
else die "modified $(( now - mt )) s ago — may still be written"; fi
|
||||
local size; size=$(du -sb "$target" 2>/dev/null | cut -f1)
|
||||
if [[ -d "$target" ]]; then rm -rf --one-file-system -- "$target"; else rm -f -- "$target"; fi
|
||||
[[ ! -e "$target" ]] || die "remove failed: $target"
|
||||
echo "removed $name ${size:-?} B ($why)"
|
||||
}
|
||||
|
||||
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
@@ -0,0 +1,255 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for configs/felhom-internal-place (felhom.eu architecture/16 §5, §7 — sizes ruled 2026-10-16).
|
||||
|
||||
The helper is run for real under bash; `lvs`, `lvcreate`, `blkid`, `mkfs.ext4` and `stat` are PATH stubs that read and
|
||||
write one JSON state file, so nothing on this machine is touched. Every refusal has a positive control: the same setup
|
||||
without the one condition goes through — so a test cannot pass because the helper refuses everything.
|
||||
Run: python3 configs/test_felhom_internal_place.py (also run by the Go suite, internal/internalplace)
|
||||
"""
|
||||
import json, os, pathlib, stat, subprocess, tempfile, unittest
|
||||
|
||||
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-internal-place"
|
||||
GIB = 1 << 30
|
||||
|
||||
STUB_LVS = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
st = json.load(open(os.environ["IP_STATE"]))
|
||||
if st.get("lvs_fail_listing") and "lv_name,pool_lv,lv_size" in sys.argv: sys.exit(5)
|
||||
args = [a for a in sys.argv[1:] if not a.startswith("-") and a not in ("b",)]
|
||||
fields = sys.argv[sys.argv.index("-o") + 1].split(",")
|
||||
target = args[-1]
|
||||
listing = target == st["vg"]
|
||||
for lv in st["lvs"]:
|
||||
full = st["vg"] + "/" + lv["name"]
|
||||
if listing and st.get("hide_pool_in_listing") and lv.get("pool", "") == "":
|
||||
continue # a listing that silently lost the pool row (the pool's own size stays readable)
|
||||
if listing or target == full:
|
||||
row = {"lv_name": lv["name"], "pool_lv": lv.get("pool", ""), "lv_size": str(lv["size"])}
|
||||
print(" " + " ".join(row[f] for f in fields))
|
||||
'''
|
||||
STUB_LVCREATE = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
p = os.environ["IP_STATE"]; st = json.load(open(p))
|
||||
a = sys.argv[1:]
|
||||
size = int(a[a.index("-V") + 1].rstrip("G")) * (1 << 30)
|
||||
vg, pool = a[a.index("-T") + 1].split("/")
|
||||
st["lvs"].append({"name": a[a.index("-n") + 1], "pool": pool, "size": size, "active": True})
|
||||
st.setdefault("calls", []).append("lvcreate " + " ".join(a))
|
||||
if st.get("dirty_new"): st["fs"]["felhom-belso"] = {"TYPE": "", "RAW": "junk"}
|
||||
json.dump(st, open(p, "w"))
|
||||
'''
|
||||
STUB_BLKID = r'''#!/usr/bin/env python3
|
||||
# like blkid -p: exit 0 when something is found, 2 when nothing is; st["blkid_rc"] forces an error exit
|
||||
import json, os, sys
|
||||
st = json.load(open(os.environ["IP_STATE"]))
|
||||
if st.get("blkid_rc"): sys.exit(st["blkid_rc"])
|
||||
fs = st["fs"].get(os.path.basename(sys.argv[-1]), {})
|
||||
if "-s" in sys.argv:
|
||||
v = fs.get(sys.argv[sys.argv.index("-s") + 1], "")
|
||||
if v: print(v)
|
||||
sys.exit(0 if v else 2)
|
||||
vals = [fs.get(k, "") for k in ("TYPE", "LABEL", "UUID", "RAW") if fs.get(k)]
|
||||
if vals: print("\n".join(vals))
|
||||
sys.exit(0 if vals else 2)
|
||||
'''
|
||||
STUB_MKFS = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
p = os.environ["IP_STATE"]; st = json.load(open(p))
|
||||
dev = os.path.basename(sys.argv[-1])
|
||||
st["fs"][dev] = {"TYPE": "ext4", "LABEL": sys.argv[sys.argv.index("-L") + 1], "UUID": "0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c"}
|
||||
st.setdefault("calls", []).append("mkfs.ext4 " + " ".join(sys.argv[1:]))
|
||||
json.dump(st, open(p, "w"))
|
||||
'''
|
||||
STUB_STAT = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
st = json.load(open(os.environ["IP_STATE"]))
|
||||
name = os.path.basename(sys.argv[-1])
|
||||
if any(lv["name"] == name and lv.get("active", True) for lv in st["lvs"]): print("block special file")
|
||||
else: sys.exit(1)
|
||||
'''
|
||||
|
||||
CFG = "dir: local\n\tpath /var/lib/vz\n\tcontent iso,backup\n\nlvmthin: local-lvm\n\tthinpool data\n\tvgname pve\n\tcontent rootdir,images\n"
|
||||
|
||||
|
||||
class Box:
|
||||
def __init__(self, pool_gib, lvs=(), fs=None, cfg=CFG, **extra):
|
||||
self.t = tempfile.TemporaryDirectory()
|
||||
r = pathlib.Path(self.t.name)
|
||||
self.bin = r / "bin"; self.bin.mkdir()
|
||||
for name, body in (("lvs", STUB_LVS), ("lvcreate", STUB_LVCREATE), ("blkid", STUB_BLKID),
|
||||
("mkfs.ext4", STUB_MKFS), ("stat", STUB_STAT)):
|
||||
f = self.bin / name; f.write_text(body); f.chmod(0o755)
|
||||
self.cfg = r / "storage.cfg"; self.cfg.write_text(cfg)
|
||||
self.state = r / "state.json"
|
||||
st = {"vg": "pve", "lvs": [{"name": "data", "pool": "", "size": pool_gib * GIB}] + [dict(l) for l in lvs],
|
||||
"fs": fs or {}}
|
||||
st.update(extra)
|
||||
self.state.write_text(json.dumps(st))
|
||||
|
||||
def run(self, *args):
|
||||
env = {"PATH": f"{self.bin}:/usr/bin:/bin", "IP_STATE": str(self.state), "FELHOM_STORAGE_CFG": str(self.cfg)}
|
||||
p = subprocess.run(["bash", str(HELPER), *args], env=env, capture_output=True, text=True)
|
||||
return p.returncode, p.stdout.strip(), p.stderr.strip()
|
||||
|
||||
def st(self):
|
||||
return json.loads(self.state.read_text())
|
||||
|
||||
def calls(self):
|
||||
return self.st().get("calls", [])
|
||||
|
||||
def close(self):
|
||||
self.t.cleanup()
|
||||
|
||||
|
||||
def thin(name, gib, pool="data"):
|
||||
return {"name": name, "pool": pool, "size": gib * GIB}
|
||||
|
||||
|
||||
class Sizes(unittest.TestCase):
|
||||
"""The operator's table (SPACE-CHECK §4, ruled 2026-10-16), as the helper computes it on a fresh box."""
|
||||
|
||||
def plan(self, pool, planned, lvs=()):
|
||||
b = Box(pool, lvs)
|
||||
try:
|
||||
rc, out, err = b.run("plan", str(planned))
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertEqual(b.calls(), [], "plan must change nothing")
|
||||
return dict(kv.split("=") for kv in out.split()[1:])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_the_ruled_table(self):
|
||||
# disk → pool (GiB), data volume (planned = 32 + mp0), expected internal
|
||||
for pool, mp0, reserve, internal in ((140, 50, 16, 42), (349, 100, 34, 183), (800, 130, 48, 590), (1720, 130, 48, 1510)):
|
||||
got = self.plan(pool, 32 + mp0)
|
||||
self.assertEqual((int(got["reserve"]), int(got["internal"])), (reserve, internal), (pool, got))
|
||||
|
||||
def test_promised_volumes_count_but_vzdump_snapshots_and_other_pools_do_not(self):
|
||||
got = self.plan(349, 0, [thin("vm-9201-disk-0", 32), thin("vm-9201-disk-1", 250),
|
||||
thin("snap_vm-9201-disk-1_vzdump", 250), thin("vm-500-disk-0", 40, pool="other")])
|
||||
self.assertEqual((int(got["promised"]), int(got["internal"])), (282, 33), got)
|
||||
|
||||
|
||||
class Create(unittest.TestCase):
|
||||
def test_a_fresh_box_gets_the_place(self):
|
||||
b = Box(349)
|
||||
try:
|
||||
rc, out, err = b.run("create", "132")
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertRegex(out, r"^created uuid=[0-9a-f-]{36} gib=183$")
|
||||
self.assertEqual(len([c for c in b.calls() if c.startswith("lvcreate")]), 1)
|
||||
self.assertIn("-V 183G -T pve/data -n felhom-belso", b.calls()[0])
|
||||
self.assertIn("-L felhom-belso", b.calls()[1])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_too_little_space_makes_nothing(self):
|
||||
b = Box(53) # demo-hp's 128 GB disk
|
||||
try:
|
||||
rc, out, err = b.run("create", str(32 + 50))
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertTrue(out.startswith("skipped "), out)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_an_existing_place_is_found_never_formatted(self):
|
||||
fs = {"felhom-belso": {"TYPE": "ext4", "LABEL": "felhom-belso", "UUID": "11111111-2222-3333-4444-555555555555"}}
|
||||
b = Box(349, [thin("felhom-belso", 183)], fs)
|
||||
try:
|
||||
rc, out, err = b.run("create", "0")
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertEqual(out, "existing uuid=11111111-2222-3333-4444-555555555555 gib=183")
|
||||
self.assertEqual(b.calls(), [], "an existing place must never be created or formatted again")
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_foreign_filesystem_on_the_name_is_refused(self):
|
||||
# positive control: test_an_existing_place_is_found_never_formatted (same LV, Felhom's label) goes through
|
||||
for fs in ({"TYPE": "xfs", "LABEL": "felhom-belso", "UUID": "x"}, {"TYPE": "ext4", "LABEL": "films", "UUID": "y"}):
|
||||
b = Box(349, [thin("felhom-belso", 183)], {"felhom-belso": fs})
|
||||
try:
|
||||
rc, out, err = b.run("create", "0")
|
||||
self.assertEqual(rc, 1, (fs, out))
|
||||
self.assertIn("REFUSED", err)
|
||||
self.assertEqual(b.calls(), [], fs)
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_crash_between_lvcreate_and_mkfs_is_finished(self):
|
||||
b = Box(349, [thin("felhom-belso", 183)], {})
|
||||
try:
|
||||
rc, out, err = b.run("create", "0")
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertRegex(out, r"^formatted uuid=")
|
||||
self.assertEqual([c.split()[0] for c in b.calls()], ["mkfs.ext4"])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_an_unnamed_signature_is_refused(self):
|
||||
b = Box(349, [thin("felhom-belso", 183)], {"felhom-belso": {"RAW": "something"}})
|
||||
try:
|
||||
rc, out, err = b.run("create", "0")
|
||||
self.assertEqual(rc, 1, out)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_new_volume_that_is_not_empty_is_not_formatted(self):
|
||||
b = Box(349, dirty_new=True)
|
||||
try:
|
||||
rc, out, err = b.run("create", "132")
|
||||
self.assertEqual(rc, 1, out)
|
||||
self.assertNotIn("mkfs", " ".join(b.calls()))
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_bad_arguments_and_no_local_lvm_are_refused(self):
|
||||
b = Box(349, cfg="dir: local\n\tpath /var/lib/vz\n")
|
||||
try:
|
||||
self.assertEqual(b.run("create", "0")[0], 1)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
b = Box(349)
|
||||
try:
|
||||
for args in (("create", "-1"), ("create", "1x"), ("wipe", "0"), ("create",)):
|
||||
self.assertEqual(b.run(*args)[0], 1, args)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
|
||||
class FailsClosed(unittest.TestCase):
|
||||
"""A read that fails must never read as „empty" or „nothing promised". Positive control: Create (same setups)."""
|
||||
|
||||
def test_a_failed_volume_listing_creates_nothing(self):
|
||||
b = Box(349, lvs_fail_listing=True)
|
||||
try:
|
||||
rc, out, err = b.run("create", "132")
|
||||
self.assertEqual(rc, 1, out)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_listing_without_the_pool_creates_nothing(self):
|
||||
b = Box(349, [thin("vm-9201-disk-1", 250)], hide_pool_in_listing=True)
|
||||
try:
|
||||
self.assertEqual(b.run("create", "132")[0], 1)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_blkid_error_never_reads_as_empty(self):
|
||||
for setup in ({"lvs": [thin("felhom-belso", 183)], "fs": {}}, {}):
|
||||
b = Box(349, **({"lvs": setup["lvs"], "fs": setup["fs"]} if setup else {}), blkid_rc=4)
|
||||
try:
|
||||
rc, out, err = b.run("create", "0" if setup else "132")
|
||||
self.assertEqual(rc, 1, (setup, out))
|
||||
self.assertNotIn("mkfs", " ".join(b.calls()), setup)
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for configs/felhom-mkfs-guarded.sh — the ONLY mkfs path the agent's sudoers permits.
|
||||
|
||||
First tests of this wrapper (agent v0.158.0), written for partitions as places (felhom.eu architecture/16 §6). Every
|
||||
mount of every node of the disk outside /mnt/felhom-drives refuses — INCLUDING a sibling that is Felhom's own drive
|
||||
(its raw /mnt/<name> mount): the agent can make that corroborating mount itself, so the root wrapper must not trust
|
||||
it (security review of 912f7c1). An unlistable disk refuses.
|
||||
|
||||
HOW IT RUNS ANYWHERE: the wrapper is copied with five literal host paths replaced (each replacement is asserted, so a
|
||||
drifted wrapper fails here instead of silently testing nothing): /proc/mounts → a fake mount table, the block-device
|
||||
test → true (no real /dev nodes here), the pvs candidates → a stub, /sys/block → a fake tree, the mkfs binaries → a
|
||||
recorder. `lsblk` is a PATH stub answering the three calls the wrapper makes from one JSON disk description.
|
||||
Every refusal has a positive control (the same disk without the one condition is formatted).
|
||||
Run: python3 configs/test_felhom_mkfs_guarded.py (also run by the Go suite, internal/storage)
|
||||
"""
|
||||
import json, os, pathlib, subprocess, tempfile, unittest
|
||||
|
||||
WRAPPER = pathlib.Path(__file__).resolve().parent / "felhom-mkfs-guarded.sh"
|
||||
|
||||
STUB_LSBLK = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
st = json.load(open(os.environ["MG_STATE"]))
|
||||
a = sys.argv[1:]; dev = a[-1]
|
||||
nodes = st["nodes"] # [{"name": "/dev/sdb", "pk": "", "fstype": ""}, ...]
|
||||
if a[0] == "-ndo" and a[1] == "PKNAME":
|
||||
for n in nodes:
|
||||
if n["name"] == dev and n.get("pk"): print(n["pk"])
|
||||
elif a[0] == "-nrpo" and a[1] == "NAME":
|
||||
if not st.get("lsblk_empty"):
|
||||
for n in nodes: print(n["name"])
|
||||
elif a[0] == "-nro" and a[1] == "FSTYPE":
|
||||
for n in nodes: print(n.get("fstype", ""))
|
||||
elif a[0] == "-nro" and a[1] == "MOUNTPOINT": # the pre-v0.158.0 wrapper's call: one mountpoint per node, like lsblk
|
||||
mt = [l.split()[:2] for l in open(os.environ["MG_MOUNTS"]) if l.strip()]
|
||||
for n in nodes:
|
||||
mps = [m for d, m in mt if d == n["name"]]
|
||||
print(mps[0] if mps else "")
|
||||
'''
|
||||
STUB_MKFS = '#!/bin/sh\necho "$0 $*" >> "$MG_CALLS"\n'
|
||||
STUB_PVS = '#!/bin/sh\nprintf "%s\\n" $MG_PVS\n'
|
||||
|
||||
|
||||
class Disk:
|
||||
def __init__(self, nodes, mounts, ro="0", pvs="", lsblk_empty=False):
|
||||
self.t = tempfile.TemporaryDirectory()
|
||||
r = pathlib.Path(self.t.name)
|
||||
self.bin = r / "bin"; self.bin.mkdir()
|
||||
(self.bin / "lsblk").write_text(STUB_LSBLK); (self.bin / "lsblk").chmod(0o755)
|
||||
for n in ("mkfs.ext4", "mkfs.xfs"):
|
||||
(self.bin / n).write_text(STUB_MKFS); (self.bin / n).chmod(0o755)
|
||||
(self.bin / "pvs").write_text(STUB_PVS); (self.bin / "pvs").chmod(0o755)
|
||||
self.state = r / "state.json"
|
||||
self.state.write_text(json.dumps({"nodes": nodes, "lsblk_empty": lsblk_empty}))
|
||||
self.mounts = r / "mounts"
|
||||
self.mounts.write_text("".join(f"{d} {m} ext4 rw 0 0\n" for d, m in mounts))
|
||||
self.sys = r / "sys"
|
||||
for n in nodes:
|
||||
if not n.get("pk"):
|
||||
d = self.sys / os.path.basename(n["name"]); d.mkdir(parents=True); (d / "ro").write_text(ro + "\n")
|
||||
self.calls = r / "calls"
|
||||
self.pvs = pvs
|
||||
src = WRAPPER.read_text()
|
||||
subs = [
|
||||
("/proc/mounts", str(self.mounts)),
|
||||
('[[ -b "$dev" ]]', "true"),
|
||||
("for c in /usr/sbin/pvs /sbin/pvs; do", f"for c in {self.bin}/pvs; do"),
|
||||
('rof="/sys/block/', f'rof="{self.sys}/'),
|
||||
("exec /usr/sbin/mkfs.ext4", f"exec {self.bin}/mkfs.ext4"),
|
||||
("exec /usr/sbin/mkfs.xfs ", f"exec {self.bin}/mkfs.xfs "),
|
||||
]
|
||||
for a, b in subs:
|
||||
assert a in src, f"the wrapper no longer contains {a!r} — update this test's substitutions"
|
||||
src = src.replace(a, b)
|
||||
self.copy = r / "mkfs-guarded"; self.copy.write_text(src)
|
||||
|
||||
def run(self, dev, fstype="ext4"):
|
||||
env = {"PATH": f"{self.bin}:/usr/bin:/bin", "MG_STATE": str(self.state), "MG_CALLS": str(self.calls),
|
||||
"MG_MOUNTS": str(self.mounts),
|
||||
"MG_PVS": self.pvs}
|
||||
p = subprocess.run(["bash", str(self.copy), dev, fstype], env=env, capture_output=True, text=True)
|
||||
made = self.calls.read_text().split("\n")[0] if self.calls.exists() else ""
|
||||
return p.returncode, p.stderr.strip(), made
|
||||
|
||||
def close(self):
|
||||
self.t.cleanup()
|
||||
|
||||
|
||||
def disk_sdb(sdb1_fstype="ext4", sdb2_fstype=""):
|
||||
return [{"name": "/dev/sdb", "pk": "", "fstype": ""},
|
||||
{"name": "/dev/sdb1", "pk": "sdb", "fstype": sdb1_fstype},
|
||||
{"name": "/dev/sdb2", "pk": "sdb", "fstype": sdb2_fstype}]
|
||||
|
||||
|
||||
ROOT = ("/dev/sda2", "/")
|
||||
OWN_SDB1 = [("/dev/sdb1", "/mnt/hdd_1"), ("/dev/sdb1", "/mnt/felhom-drives/hdd_1")]
|
||||
|
||||
|
||||
class Partitions(unittest.TestCase):
|
||||
def check(self, mounts, dev="/dev/sdb2", nodes=None, want_ok=True, **kw):
|
||||
d = Disk(nodes or disk_sdb(), [ROOT] + mounts, **kw)
|
||||
try:
|
||||
rc, err, made = d.run(dev)
|
||||
if want_ok:
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertTrue(made.endswith(f"mkfs.ext4 -F {dev}"), made)
|
||||
else:
|
||||
self.assertEqual(rc, 1, (mounts, made))
|
||||
self.assertIn("REFUSED", err)
|
||||
self.assertEqual(made, "", "a refused format must not run mkfs")
|
||||
return err
|
||||
finally:
|
||||
d.close()
|
||||
|
||||
def test_a_free_part_on_an_unmounted_disk_is_formatted(self):
|
||||
self.check([]) # positive control for every refusal below
|
||||
|
||||
def test_the_free_part_next_to_felhoms_own_drive_is_refused(self):
|
||||
# The agent could make the „corroborating" managed mount itself: the root wrapper does not trust it.
|
||||
self.check(OWN_SDB1, want_ok=False)
|
||||
|
||||
def test_a_sibling_mounted_raw_only_is_refused(self):
|
||||
self.check([("/dev/sdb1", "/mnt/hdd_1")], want_ok=False)
|
||||
|
||||
def test_a_sibling_mounted_elsewhere_is_refused_even_with_a_managed_mount(self):
|
||||
for elsewhere in ("/srv/data", "/media/usb", "/mnt/a/b", "/home/x"):
|
||||
self.check([("/dev/sdb1", elsewhere), ("/dev/sdb1", "/mnt/felhom-drives/hdd_1")], want_ok=False)
|
||||
|
||||
def test_the_target_itself_mounted_raw_is_refused(self):
|
||||
self.check([("/dev/sdb2", "/mnt/films"), ("/dev/sdb2", "/mnt/felhom-drives/films")], want_ok=False)
|
||||
|
||||
def test_the_target_mounted_only_under_felhom_drives_is_formatted(self):
|
||||
self.check([("/dev/sdb2", "/mnt/felhom-drives/films")])
|
||||
|
||||
def test_an_unlistable_disk_is_refused(self):
|
||||
self.check([], want_ok=False, lsblk_empty=True)
|
||||
|
||||
def test_the_catastrophic_cases_still_refuse(self):
|
||||
self.check([("/dev/sdb1", "/")], want_ok=False) # system disk
|
||||
self.check([], nodes=disk_sdb(sdb1_fstype="LVM2_member"), want_ok=False) # a member signature
|
||||
self.check([], pvs="/dev/sdb1", want_ok=False) # an LVM PV (pvs)
|
||||
self.check([], ro="1", want_ok=False) # read-only
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -376,5 +376,67 @@ class ControllerImage(unittest.TestCase):
|
||||
self.assertIn("controller-image", buf.getvalue())
|
||||
|
||||
|
||||
class UnitForget(unittest.TestCase):
|
||||
"""R-1124: the agent forgets a decommissioned drive whose disk is gone — the root half removes its unit file, and
|
||||
ONLY a felhom by-UUID drive unit that is disabled, not mounted, and whose /dev/disk/by-uuid link is absent.
|
||||
RED-PROOF: on the pre-R-1124 wrapper `unit-forget` is not a verb (A1 usage, rc 2) — the removal case fails."""
|
||||
NAME = "mnt-hdd_1.mount"
|
||||
PATH = "/etc/systemd/system/mnt-hdd_1.mount"
|
||||
UUID_LINK = "/dev/disk/by-uuid/91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae"
|
||||
|
||||
def host(self, text=LOCAL_UNIT, kind="file", links=(), mounted=()):
|
||||
h = FakeHost()
|
||||
h.installed = {} if text is None else {self.PATH: (kind, text.encode())}
|
||||
h.links, h.mounted, h.removed = set(links), set(mounted), []
|
||||
h.read_installed = lambda p: (None if p not in h.installed else
|
||||
(_ for _ in ()).throw(pa.Refused("F1", f"{p} is not a regular file"))
|
||||
if h.installed[p][0] != "file" else h.installed[p][1])
|
||||
h.lexists = lambda p: p in h.links
|
||||
h.mountpoints = lambda: set(h.mounted)
|
||||
h.remove = lambda p: h.removed.append(p)
|
||||
return h
|
||||
|
||||
def test_an_absent_disabled_unmounted_drive_unit_is_removed(self):
|
||||
h = self.host()
|
||||
self.assertEqual(run(h, "unit-forget", self.NAME), 0, h.logs)
|
||||
self.assertEqual(h.removed, [self.PATH])
|
||||
|
||||
def test_already_gone_is_fine(self):
|
||||
h = self.host(text=None)
|
||||
self.assertEqual(run(h, "unit-forget", self.NAME), 0, h.logs)
|
||||
self.assertEqual(h.removed, [])
|
||||
|
||||
def refused(self, h, rule, *argv):
|
||||
rc = run(h, *(argv or ("unit-forget", self.NAME)))
|
||||
self.assertIn(rc, (2, 3), h.logs)
|
||||
self.assertEqual(h.removed, [], h.logs)
|
||||
self.assertTrue(any(f"[{rule}]" in l for l in h.logs), h.logs)
|
||||
|
||||
def test_F5_a_present_drive_keeps_its_unit(self):
|
||||
self.refused(self.host(links=[self.UUID_LINK]), "F5")
|
||||
|
||||
def test_F4_a_mounted_drive_keeps_its_unit(self):
|
||||
self.refused(self.host(mounted=["/mnt/hdd_1"]), "F4")
|
||||
|
||||
def test_F3_an_enabled_unit_is_not_removed(self):
|
||||
self.refused(self.host(links=["/etc/systemd/system/multi-user.target.wants/" + self.NAME]), "F3")
|
||||
|
||||
def test_F2_a_network_unit_or_a_foreign_unit_is_not_removed(self):
|
||||
self.refused(self.host(text=LOCAL_UNIT.replace("Managed by felhom-agent", "Written by someone")), "F2")
|
||||
net = LOCAL_UNIT.replace("# Managed by felhom-agent", "# Managed by felhom-agent felhom network storage")
|
||||
self.refused(self.host(text=net), "F2")
|
||||
|
||||
def test_F1_a_symlinked_unit_is_not_followed(self):
|
||||
self.refused(self.host(kind="symlink"), "F1")
|
||||
|
||||
def test_A1_names(self):
|
||||
for argv in (("unit-forget", "mnt-hdd_1.automount"), ("unit-forget", "sshd.service"),
|
||||
("unit-forget", "../mnt-hdd_1.mount"), ("unit-forget",), ("unit-forget", self.NAME, "x")):
|
||||
self.refused(self.host(), "A1", *argv)
|
||||
|
||||
def test_U3_a_unit_whose_name_lies_about_where_is_not_removed(self):
|
||||
self.refused(self.host(text=LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/mnt/other")), "U3")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for configs/felhom-thin-check (R-1012): it lists the thin pool's device ids next to LVM's and NEVER removes
|
||||
anything. Fake `lvs`, `dmsetup` and `thin_ls` record every call; nothing real is touched. Each refusal has a positive
|
||||
control (the healthy read), so a test cannot pass because the helper refuses everything.
|
||||
Run: python3 configs/test_felhom_thin_check.py (also run by internal/thincheck's Go test)
|
||||
"""
|
||||
import os, pathlib, re, subprocess, tempfile, unittest
|
||||
|
||||
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-thin-check"
|
||||
|
||||
|
||||
class Box:
|
||||
def __init__(self, pool_ids=(1, 2), lvm=(("vm-9201-disk-0", 1), ("vm-9201-disk-1", 2)), attr="twi-aotz--",
|
||||
held="-", reserve_rc=0, thin_ls_rc=0):
|
||||
self.t = tempfile.TemporaryDirectory()
|
||||
r = pathlib.Path(self.t.name)
|
||||
self.log = r / "calls"
|
||||
self.log.write_text("")
|
||||
lvm_lines = "\\n".join(f" {n},{i}" for n, i in lvm)
|
||||
pool_lines = "\\n".join(str(i) for i in pool_ids)
|
||||
self.bins = {}
|
||||
for name, body in {
|
||||
"lvs": f'''case "$*" in *lv_attr*) echo " {attr}";; *thin_id*) printf "{lvm_lines}\\n";; esac''',
|
||||
"dmsetup": f'''case "$1" in
|
||||
status) echo "0 112975872 thin-pool 585 6860/262144 517080/882624 {held} rw discard_passdown queue_if_no_space - 1024";;
|
||||
message) [ "$4" = reserve_metadata_snap ] && exit {reserve_rc}; exit 0;;
|
||||
esac''',
|
||||
"thin_ls": f'''[ {thin_ls_rc} -ne 0 ] && exit {thin_ls_rc}; printf "{pool_lines}\\n"''',
|
||||
}.items():
|
||||
p = r / name
|
||||
p.write_text(f'#!/bin/bash\necho "{name} $*" >> "{self.log}"\n{body}\n')
|
||||
p.chmod(0o755)
|
||||
self.bins[name] = p
|
||||
self.lock = r / "lock"
|
||||
|
||||
def run(self, *args):
|
||||
script = (f'source "{HELPER}"; LVS="{self.bins["lvs"]}"; DMSETUP="{self.bins["dmsetup"]}"; '
|
||||
f'THIN_LS="{self.bins["thin_ls"]}"; LOCK="{self.lock}"; main "$@"')
|
||||
r = subprocess.run(["bash", "-c", script, "x", *args], capture_output=True, text=True)
|
||||
return r.returncode, r.stdout, r.stderr
|
||||
|
||||
def calls(self):
|
||||
return self.log.read_text().splitlines()
|
||||
|
||||
|
||||
def messages(calls):
|
||||
return [c.split()[-1] for c in calls if c.startswith("dmsetup message")]
|
||||
|
||||
|
||||
class Reads(unittest.TestCase):
|
||||
def test_healthy_pool_lists_both_sides_and_releases(self):
|
||||
b = Box()
|
||||
rc, out, err = b.run("pve", "data")
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertEqual(out.splitlines(), ["lvm-before 1 vm-9201-disk-0", "lvm-before 2 vm-9201-disk-1",
|
||||
"pool 1", "pool 2",
|
||||
"lvm-after 1 vm-9201-disk-0", "lvm-after 2 vm-9201-disk-1", "ok"])
|
||||
self.assertEqual(messages(b.calls()), ["reserve_metadata_snap", "release_metadata_snap"])
|
||||
self.assertIn("thin_ls --metadata-snap --no-headers -o DEV /dev/mapper/pve-data_tmeta", b.calls())
|
||||
|
||||
def test_the_2026_10_14_orphan_is_listed(self):
|
||||
rc, out, _ = Box(pool_ids=(1, 2, 4)).run("pve", "data")
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertIn("pool 4", out.splitlines())
|
||||
|
||||
def test_dashed_names_map_to_the_device_mapper_name(self):
|
||||
b = Box()
|
||||
rc, _, err = b.run("felhom-scratch", "thin-pool")
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertIn("dmsetup status felhom--scratch-thin--pool-tpool", b.calls())
|
||||
|
||||
|
||||
class Refuses(unittest.TestCase):
|
||||
def refused(self, box, rc_want, *args):
|
||||
rc, out, err = box.run(*args)
|
||||
self.assertEqual(rc, rc_want, (out, err))
|
||||
self.assertNotIn("ok", out.splitlines())
|
||||
return box.calls()
|
||||
|
||||
def test_arguments(self):
|
||||
for args in (("pve",), ("pve", "data", "x"), ("pve;id", "data"), ("-pve", "data"), ("pve", "../data"),
|
||||
("pve", "da ta")):
|
||||
calls = self.refused(Box(), 1, *args)
|
||||
self.assertEqual(calls, [], args)
|
||||
|
||||
def test_not_a_thin_pool(self):
|
||||
calls = self.refused(Box(attr="Vwi-aotz--"), 1, "pve", "vm-9201-disk-1")
|
||||
self.assertEqual(messages(calls), [])
|
||||
|
||||
def test_a_snapshot_someone_else_holds_is_never_touched(self):
|
||||
b = Box(held="1234")
|
||||
rc, _, err = b.run("pve", "data")
|
||||
self.assertEqual(rc, 3)
|
||||
self.assertEqual(messages(b.calls()), []) # neither reserve nor release
|
||||
# the reason line ends with the reason, not the exit code (found live on demo-hp 2026-10-17)
|
||||
self.assertTrue(err.strip().endswith("not touched"), err)
|
||||
|
||||
def test_reserve_failure_does_not_release(self):
|
||||
calls = self.refused(Box(reserve_rc=1), 3, "pve", "data")
|
||||
self.assertEqual(messages(calls), ["reserve_metadata_snap"])
|
||||
|
||||
def test_thin_ls_failure_still_releases(self):
|
||||
calls = self.refused(Box(thin_ls_rc=1), 4, "pve", "data")
|
||||
self.assertEqual(messages(calls), ["reserve_metadata_snap", "release_metadata_snap"])
|
||||
|
||||
|
||||
class NeverDeletes(unittest.TestCase):
|
||||
def test_the_only_pool_messages_are_reserve_and_release(self):
|
||||
src = HELPER.read_text()
|
||||
sent = re.findall(r'"\$DMSETUP" message "\$TPOOL" 0 (\S+)', src)
|
||||
self.assertEqual(sorted(set(sent)), ["release_metadata_snap", "reserve_metadata_snap"])
|
||||
for word in ("delete", "lvremove", "thin_repair", "create_thin", "create_snap", " rm "):
|
||||
code = "\n".join(l for l in src.splitlines() if not l.lstrip().startswith("#"))
|
||||
self.assertNotIn(word, code, word)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for configs/felhom-vzdump-partial-clean (R-1006): it removes a DEAD half-written vzdump backup and refuses
|
||||
everything else. Each refusal has a positive control: the same setup without the one condition is removed — so a
|
||||
test cannot pass because the helper refuses everything. A fake storage.cfg and a fake /proc; nothing real is touched.
|
||||
Run: python3 configs/test_felhom_vzdump_partial_clean.py (also run by internal/backup's Go test)
|
||||
"""
|
||||
import os, pathlib, subprocess, tempfile, time, unittest
|
||||
|
||||
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-vzdump-partial-clean"
|
||||
BOOT = 1_791_967_481
|
||||
NAME = "vzdump-lxc-9201-2026_10_14-10_42_25"
|
||||
|
||||
|
||||
class Box:
|
||||
"""A temp tree: a dir storage `local` (path vz), an lvmthin `local-lvm`, a dir storage `other`, and a fake /proc."""
|
||||
|
||||
def __init__(self):
|
||||
self.t = tempfile.TemporaryDirectory()
|
||||
r = pathlib.Path(self.t.name)
|
||||
self.vz, self.other, self.nas, self.proc = r / "vz", r / "other", r / "nas", r / "proc"
|
||||
for d in (self.vz / "dump", self.other / "dump", self.nas / "dump", self.proc):
|
||||
d.mkdir(parents=True)
|
||||
(self.proc / "stat").write_text(f"cpu 1 2 3\nbtime {BOOT}\n")
|
||||
self.cfg = r / "storage.cfg"
|
||||
self.cfg.write_text(f"dir: local\n\tpath {self.vz}\n\tcontent iso,backup\n\n"
|
||||
f"lvmthin: local-lvm\n\tthinpool data\n\tvgname pve\n\n"
|
||||
f"dir: other\n\tpath {self.other}\n\tcontent backup\n\n"
|
||||
f"nfs: nas\n\tpath {self.nas}\n\tserver 192.0.2.1\n\texport /x\n\tcontent backup\n")
|
||||
self.pid = 1000
|
||||
|
||||
def partial(self, name=NAME + ".tar.dat", mtime=BOOT - 100, where=None, size=4096):
|
||||
p = (where or self.vz / "dump") / name
|
||||
if name.endswith(".tmp"):
|
||||
(p / "etc" / "vzdump").mkdir(parents=True)
|
||||
(p / "etc" / "vzdump" / "pct.conf").write_text("x")
|
||||
for q in [p / "etc" / "vzdump" / "pct.conf", p / "etc" / "vzdump", p / "etc", p]:
|
||||
os.utime(q, (mtime, mtime))
|
||||
else:
|
||||
p.write_bytes(b"\0" * size)
|
||||
os.utime(p, (mtime, mtime))
|
||||
return p
|
||||
|
||||
def process(self, cmdline, fds=(), cwd=None):
|
||||
self.pid += 1
|
||||
d = self.proc / str(self.pid)
|
||||
(d / "fd").mkdir(parents=True)
|
||||
(d / "cmdline").write_bytes(b"\0".join(a.encode() for a in cmdline) + b"\0")
|
||||
for i, f in enumerate(fds):
|
||||
os.symlink(str(f), d / "fd" / str(i + 3))
|
||||
os.symlink(str(cwd or "/"), d / "cwd")
|
||||
|
||||
def run(self, storage, vmid, name, quiet=120):
|
||||
script = (f'source "{HELPER}"; STORAGE_CFG="{self.cfg}"; PROC="{self.proc}"; QUIET_S={quiet}; '
|
||||
f'main "$@"')
|
||||
r = subprocess.run(["bash", "-c", script, "x", storage, str(vmid), name], capture_output=True, text=True)
|
||||
return r.returncode, r.stdout + r.stderr
|
||||
|
||||
|
||||
class Removes(unittest.TestCase):
|
||||
def test_dead_tar_dat_before_boot(self):
|
||||
b = Box(); p = b.partial()
|
||||
rc, out = b.run("local", 9201, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("written before this boot", out)
|
||||
|
||||
def test_dead_tmp_dir_before_boot(self):
|
||||
b = Box(); p = b.partial(NAME + ".tmp")
|
||||
rc, out = b.run("local", 9201, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
|
||||
|
||||
def test_quiet_partial_after_boot(self):
|
||||
b = Box(); p = b.partial(mtime=int(time.time()) - 300)
|
||||
rc, out = b.run("local", 9201, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("untouched", out)
|
||||
|
||||
def test_qemu_vma_dat(self):
|
||||
b = Box(); p = b.partial("vzdump-qemu-341-2026_10_14-10_42_25.vma.dat")
|
||||
rc, out = b.run("local", 341, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
|
||||
|
||||
|
||||
class Keeps(unittest.TestCase):
|
||||
"""Each case: (setup that must be REFUSED, the same setup that must be REMOVED)."""
|
||||
|
||||
def pair(self, refuse, control, quiet=120):
|
||||
b = Box(); target, args = refuse(b)
|
||||
rc, out = b.run(*args, quiet=quiet)
|
||||
self.assertNotEqual(rc, 0, f"must refuse: {out}")
|
||||
self.assertIn("REFUSED", out)
|
||||
self.assertTrue(os.path.lexists(target), "a refused entry must still exist")
|
||||
c = Box(); ctarget, cargs = control(c)
|
||||
crc, cout = c.run(*cargs, quiet=quiet)
|
||||
self.assertEqual(crc, 0, f"positive control must be removed: {cout}")
|
||||
self.assertFalse(os.path.lexists(ctarget))
|
||||
|
||||
def test_complete_archive_of_same_vmid(self):
|
||||
self.pair(lambda b: (b.partial(NAME + ".tar.zst"), ("local", 9201, NAME + ".tar.zst")),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_log_and_notes(self):
|
||||
for suffix in (".log", ".tar.zst.notes"):
|
||||
self.pair(lambda b, s=suffix: (b.partial(NAME + s), ("local", 9201, NAME + s)),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_partial_of_another_vmid(self):
|
||||
other = "vzdump-lxc-9202-2026_10_14-10_42_25.tar.dat"
|
||||
self.pair(lambda b: (b.partial(other), ("local", 9201, other)),
|
||||
lambda b: (b.partial(other), ("local", 9202, other)))
|
||||
|
||||
def test_partial_on_a_non_dir_storage(self):
|
||||
self.pair(lambda b: (b.partial(), ("local-lvm", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_partial_on_a_network_storage_with_a_path(self):
|
||||
# an nfs storage HAS a path: only the „dir only" rule (both its guards) can refuse it
|
||||
self.pair(lambda b: (b.partial(where=b.nas / "dump"), ("nas", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_partial_on_another_storage_than_named(self):
|
||||
self.pair(lambda b: (b.partial(where=b.other / "dump"), ("local", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_unknown_storage(self):
|
||||
self.pair(lambda b: (b.partial(), ("nosuch", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_vzdump_task_in_flight(self):
|
||||
def refuse(b):
|
||||
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzdump:9201:felhom-agent@pve!agent:"]); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzrestore:9202:root@pam:"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_any_vzdump_job_even_without_the_vmid(self):
|
||||
# a job for several guests: `vzdump::` — and a vzdump for ANOTHER guest — both refuse
|
||||
for cmd in (["task UPID:demo-hp:001:002:6ACF4071:vzdump::root@pam:"],
|
||||
["task UPID:demo-hp:001:002:6ACF4071:vzdump:9202:felhom-agent@pve!agent:"]):
|
||||
def refuse(b, c=cmd):
|
||||
p = b.partial(); b.process(c); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzstart:9202:root@pam:"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_vzdump_command_for_the_vmid(self):
|
||||
def refuse(b):
|
||||
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdump", "9201", "--storage", "local"]); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdumpfoo", "9201"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_file_held_open_by_an_orphan_writer(self):
|
||||
def refuse(b):
|
||||
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[p]); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[b.vz / "dump" / "something-else"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_tmp_dir_with_a_process_inside(self):
|
||||
def refuse(b):
|
||||
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=p / "etc"); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=b.vz); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_recently_written_after_boot(self):
|
||||
self.pair(lambda b: (b.partial(mtime=int(time.time()) - 10), ("local", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(mtime=int(time.time()) - 300), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_symlink(self):
|
||||
def refuse(b):
|
||||
real = b.partial(where=b.vz)
|
||||
link = b.vz / "dump" / (NAME + ".tar.dat")
|
||||
os.symlink(real, link)
|
||||
return link, ("local", 9201, link.name)
|
||||
# quiet=0: the link was made just now, so only the symlink rule may refuse it
|
||||
self.pair(refuse, lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")), quiet=0)
|
||||
|
||||
def test_near_miss_names(self):
|
||||
for bad in (NAME + ".tar.dat.bak", "vzdump-lxc-92011-2026_10_14-10_42_25.tar.dat", NAME + ".tar.da",
|
||||
"x" + NAME + ".tar.dat", NAME + ".TAR.DAT", "vzdump-lxc-9201-2026_10_14-10_42.tar.dat"):
|
||||
self.pair(lambda b, n=bad: (b.partial(n), ("local", 9201, n)),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_path_traversal_and_bad_args(self):
|
||||
b = Box(); p = b.partial()
|
||||
for args in (("local", 9201, "../dump/" + p.name), ("local", 9201, ""), ("loc al", 9201, p.name),
|
||||
("local", "9201x", p.name), ("local", 12, p.name)):
|
||||
rc, out = b.run(*args)
|
||||
self.assertNotEqual(rc, 0, f"{args}: {out}")
|
||||
self.assertTrue(p.exists())
|
||||
|
||||
def test_wrong_kind(self):
|
||||
def refuse(b): # a FILE named .tmp
|
||||
p = b.vz / "dump" / (NAME + ".tmp"); p.write_bytes(b"x"); os.utime(p, (BOOT - 100, BOOT - 100)); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, lambda b: (b.partial(NAME + ".tmp"), ("local", 9201, NAME + ".tmp")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,7 +1,8 @@
|
||||
module gitea.dooplex.hu/admin/felhom-agent
|
||||
|
||||
go 1.25.0
|
||||
go 1.27.2
|
||||
|
||||
require golang.org/x/crypto v0.52.0
|
||||
|
||||
require golang.org/x/sys v0.45.0 // indirect
|
||||
require (
|
||||
golang.org/x/crypto v0.58.0
|
||||
golang.org/x/sys v0.49.0
|
||||
)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
|
||||
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
|
||||
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
|
||||
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
|
||||
golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
|
||||
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
|
||||
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
|
||||
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
||||
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
|
||||
golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1012: a snapshot backup that fails carries the orphan finding in its error — the text the controller mails to
|
||||
// the operator (`whole_guest_backup_failed`). The check runs ONCE, after the failure — never before the run.
|
||||
// RED-PROOF (P1/redproof-thincheck.txt): with explainFailure's append removed this test fails. RED-PROOF (2026-10-17
|
||||
// night, felhom-agent REPORT.md): with the v0.159.0 pre-run check restored, calls = 2 and this test fails.
|
||||
func TestBackup_SnapshotFailure_NamesTheOrphan(t *testing.T) {
|
||||
const why = "probable cause: thin pool pve/data holds device 4 that LVM does not know"
|
||||
for name, api := range map[string]*fakeBackupAPI{
|
||||
"task fails": {vzdumpUPID: "UPID:x", waitErr: errors.New("task failed: job errors")},
|
||||
"start fails": {vzdumpErr: errors.New("vzdump: 500")},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
r := NewBackupRunner(api, "local", "", "", "", quiet())
|
||||
calls := 0
|
||||
r.SetThinCheck(func(context.Context, int) string { calls++; return why })
|
||||
rec, err := r.Backup(context.Background(), 9201)
|
||||
if err == nil || rec.Success {
|
||||
t.Fatalf("want a failure, got %+v", rec)
|
||||
}
|
||||
if !strings.Contains(rec.Error, why) {
|
||||
t.Fatalf("the failure does not name the orphan: %q", rec.Error)
|
||||
}
|
||||
if calls != 1 {
|
||||
t.Fatalf("thin check ran %d times, want 1 (after the failure only)", calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A successful backup never runs the check (it takes a metadata snapshot of the live pool, and nobody needs the answer)
|
||||
// and keeps a clean record; a stop-mode backup takes no snapshot, so the check never runs; an unwired runner behaves
|
||||
// exactly as before. RED-PROOF (2026-10-17 night): with the v0.159.0 pre-run check restored, the success leg fails
|
||||
// with calls=1.
|
||||
func TestBackup_ThinCheck_SilentWhenNotApplicable(t *testing.T) {
|
||||
ok := &fakeBackupAPI{vzdumpUPID: "UPID:x",
|
||||
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-x", Content: "backup", VMID: 9201, Size: 100, CTime: 1}}}
|
||||
r := NewBackupRunner(ok, "local", "", "", "", quiet())
|
||||
calls := 0
|
||||
r.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
|
||||
rec, err := r.Backup(context.Background(), 9201)
|
||||
if err != nil || !rec.Success || rec.Error != "" || calls != 0 {
|
||||
t.Fatalf("success: err=%v rec=%+v calls=%d (want no check on a successful backup, clean record)", err, rec, calls)
|
||||
}
|
||||
|
||||
stop := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", proxmox.ModeStop, "", "", quiet())
|
||||
calls = 0
|
||||
stop.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
|
||||
rec, _ = stop.Backup(context.Background(), 9201)
|
||||
if calls != 0 || strings.Contains(rec.Error, "probable cause") {
|
||||
t.Fatalf("stop mode: calls=%d error=%q", calls, rec.Error)
|
||||
}
|
||||
|
||||
plain := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", "", "", "", quiet())
|
||||
if rec, _ := plain.Backup(context.Background(), 9201); rec.Error != "boom" {
|
||||
t.Fatalf("unwired: error = %q, want the task error unchanged", rec.Error)
|
||||
}
|
||||
}
|
||||
@@ -81,3 +81,28 @@ func TestR685_FailsOpen(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestR1006_LeftoverCleanupRunsBeforeThePreflight — the night of 2026-10-14: a 12.2 GB half-written archive from a power
|
||||
// cut held the space, so every later backup was skipped. The cleanup step runs BEFORE the preflight reads the free space:
|
||||
// here it „removes" the leftover (free 4 → 16 GiB) and the backup then fits and runs.
|
||||
//
|
||||
// RED-PROOF: drop the r.beforeBackup call from backup() (or move it after spaceFits) → the backup is skipped → fails.
|
||||
func TestR1006_LeftoverCleanupRunsBeforeThePreflight(t *testing.T) {
|
||||
api := spaceAPI(4*gib, 8182759056, "dir")
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
var got []string
|
||||
r.SetBeforeBackup(func(_ context.Context, storage string, vmid int) {
|
||||
got = append(got, storage)
|
||||
if vmid != 9201 {
|
||||
t.Errorf("cleanup asked for vmid %d", vmid)
|
||||
}
|
||||
api.storages[0].Avail = 16 * gib
|
||||
})
|
||||
_, _ = r.Backup(context.Background(), 9201)
|
||||
if len(got) != 1 || got[0] != "local" {
|
||||
t.Fatalf("the cleanup must run once for this target, got %v", got)
|
||||
}
|
||||
if len(api.vzdumps) != 1 {
|
||||
t.Fatalf("after the cleanup freed the space the backup must run, vzdumps=%d", len(api.vzdumps))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -58,8 +58,16 @@ type BackupRunner struct {
|
||||
// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
|
||||
// there is a deliberate act.
|
||||
allowPBSPrune bool
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// beforeBackup runs just before the space preflight (R-1006): it removes a dead half-written vzdump of this guest on
|
||||
// this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing.
|
||||
beforeBackup func(ctx context.Context, storage string, vmid int)
|
||||
// thinCheck (R-1012) names an orphan thin device in the guest's pools in plain words ("" = none / not checked). It
|
||||
// runs ONLY when a snapshot-mode vzdump has failed (start or task), so the failure the controller mails to the
|
||||
// operator says why; a successful backup never runs it (it takes a metadata snapshot of the live pool). It never
|
||||
// removes anything. nil = not wired.
|
||||
thinCheck func(ctx context.Context, vmid int) string
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
|
||||
// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
|
||||
// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
|
||||
@@ -102,6 +110,27 @@ func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode,
|
||||
now: func() time.Time { return time.Now().UTC() }}
|
||||
}
|
||||
|
||||
// SetBeforeBackup sets the step run just before the space preflight (R-1006: the dead-leftover cleanup).
|
||||
func (r *BackupRunner) SetBeforeBackup(fn func(ctx context.Context, storage string, vmid int)) {
|
||||
r.beforeBackup = fn
|
||||
}
|
||||
|
||||
// SetThinCheck wires the R-1012 orphan check (see BackupRunner.thinCheck).
|
||||
func (r *BackupRunner) SetThinCheck(fn func(ctx context.Context, vmid int) string) { r.thinCheck = fn }
|
||||
|
||||
// explainFailure appends the orphan finding to a failed snapshot-mode backup's error — the only place the check runs, so
|
||||
// it sees the pool as the failure left it. Bounded, so a stuck helper cannot hold the job.
|
||||
func (r *BackupRunner) explainFailure(ctx context.Context, vmid int, rec *hub.Backup) {
|
||||
if r.thinCheck == nil || r.mode != proxmox.ModeSnapshot {
|
||||
return
|
||||
}
|
||||
cctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute)
|
||||
defer cancel()
|
||||
if why := r.thinCheck(cctx, vmid); why != "" {
|
||||
rec.Error = rec.Error + " — " + why
|
||||
}
|
||||
}
|
||||
|
||||
// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
|
||||
// applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a
|
||||
// separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type
|
||||
@@ -183,6 +212,9 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
|
||||
// R-685 (v0.134.0): will the new archive FIT on a local target? Asked before anything runs, so a
|
||||
// target that cannot hold it is a named SKIP with the numbers, not a nightly "No space left on
|
||||
// device" that only the vzdump log explains (demo-hp, every night from 2026-09-23 — R-684).
|
||||
if r.beforeBackup != nil {
|
||||
r.beforeBackup(ctx, r.target, vmid)
|
||||
}
|
||||
if ok, why := r.spaceFits(ctx, vmid); !ok {
|
||||
rec.Error = BackupSkipNoSpacePrefix + why
|
||||
rec.DurationSeconds = time.Since(start).Seconds()
|
||||
@@ -190,12 +222,15 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
|
||||
return rec, fmt.Errorf("backup: %s", rec.Error)
|
||||
}
|
||||
|
||||
// R-1012: no orphan check here. It takes a metadata snapshot of the live pool, and its answer is only needed when the
|
||||
// snapshot step has failed — explainFailure runs it then (pinned by TestBackup_ThinCheck_SilentWhenNotApplicable).
|
||||
upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
|
||||
VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
|
||||
PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
|
||||
})
|
||||
if err != nil {
|
||||
rec.Error = err.Error()
|
||||
r.explainFailure(ctx, vmid, &rec)
|
||||
rec.DurationSeconds = time.Since(start).Seconds()
|
||||
return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err)
|
||||
}
|
||||
@@ -209,6 +244,7 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
|
||||
}
|
||||
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil {
|
||||
rec.Error = err.Error()
|
||||
r.explainFailure(ctx, vmid, &rec)
|
||||
rec.DurationSeconds = time.Since(start).Seconds()
|
||||
return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
|
||||
}
|
||||
|
||||
@@ -91,9 +91,11 @@ var manifest = []Capability{
|
||||
{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
|
||||
{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
|
||||
{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
|
||||
{"thin-orphan-check", "orphan thin device check after a failed whole-guest backup (read-only, R-1012)", "/usr/local/sbin/felhom-thin-check", []string{"pve", "data"}, false, ""},
|
||||
|
||||
// ---- Storage mount units (watchdog re-mount) ----
|
||||
{"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""},
|
||||
{"mount-unit-forget", "decommissioned-drive unit removal (root check, R-1124)", "/usr/local/sbin/felhom-priv-apply", []string{"unit-forget", "mnt-felhom\\x2dx.mount"}, false, ""},
|
||||
{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
|
||||
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
|
||||
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
|
||||
@@ -115,6 +117,11 @@ var manifest = []Capability{
|
||||
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
|
||||
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
|
||||
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
|
||||
{"guest-shutdown", "household power-off: guest shutdown (R-998)", "/usr/sbin/pct", []string{"shutdown", "9201", "--timeout", "120"}, false, ""},
|
||||
{"host-poweroff", "household power-off: host power-off (R-998)", "/usr/bin/systemctl", []string{"poweroff"}, false, ""},
|
||||
{"vzdump-partial-clean", "dead half-written vzdump removal (R-1006)", "/usr/local/sbin/felhom-vzdump-partial-clean", []string{"local", "9201", "vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat"}, false, ""},
|
||||
{"internal-place-plan", "internal data place: the size plan (16 §7)", "/usr/local/sbin/felhom-internal-place", []string{"plan", "0"}, false, ""},
|
||||
{"internal-place-create", "internal data place: create on an old box (signed add_internal_place)", "/usr/local/sbin/felhom-internal-place", []string{"create", "0"}, false, ""},
|
||||
|
||||
// ---- LAN split-horizon resolver (dnsmasq) ----
|
||||
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
|
||||
|
||||
@@ -311,3 +311,39 @@ func TestWGCapabilityCriticality(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEveryAliasIsGranted closes a gap found while adding FELHOM_POWEROFF (R-998): the coverage check above reads
|
||||
// every Cmnd_Alias body whether or not the alias appears on the grant line, so a manifest command whose alias was
|
||||
// never GRANTED still passed — and failed with „a password is required" on the box. Every defined alias must be
|
||||
// named on the `felhom-agent ALL=(root) NOPASSWD:` line. Red-proof: drop FELHOM_POWEROFF from that line.
|
||||
func TestEveryAliasIsGranted(t *testing.T) {
|
||||
data, err := os.ReadFile(sudoersPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read sudoers: %v", err)
|
||||
}
|
||||
var defined []string
|
||||
granted := map[string]bool{}
|
||||
grantLines := 0
|
||||
for _, ln := range strings.Split(string(data), "\n") {
|
||||
tr := strings.TrimSpace(ln)
|
||||
if strings.HasPrefix(tr, "Cmnd_Alias ") {
|
||||
if f := strings.Fields(tr); len(f) >= 2 {
|
||||
defined = append(defined, f[1])
|
||||
}
|
||||
}
|
||||
if strings.HasPrefix(tr, "felhom-agent ") && strings.Contains(tr, "NOPASSWD:") {
|
||||
grantLines++
|
||||
for _, a := range strings.Split(tr[strings.Index(tr, "NOPASSWD:")+len("NOPASSWD:"):], ",") {
|
||||
granted[strings.TrimSpace(a)] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if grantLines == 0 || len(defined) < 10 {
|
||||
t.Fatalf("parsed %d grant line(s), %d aliases — parser broke", grantLines, len(defined))
|
||||
}
|
||||
for _, a := range defined {
|
||||
if !granted[a] {
|
||||
t.Errorf("Cmnd_Alias %s is defined but never granted to felhom-agent", a)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -53,6 +53,30 @@ var r861Injections = []string{
|
||||
"/usr/sbin/pct exec 9201 -- tee /etc/felhom-controller-image",
|
||||
"/usr/local/sbin/felhom-priv-apply controller-image 9201 9202",
|
||||
"/usr/local/sbin/felhom-priv-apply controller-image 9201;id",
|
||||
// R-998 (host power-off): the two new grants take exactly their fixed shape — no extra option, no other target
|
||||
"/usr/sbin/pct shutdown 9201 --timeout 120 --forceStop 1",
|
||||
"/usr/sbin/pct shutdown 9201 --forceStop 1 --timeout 120",
|
||||
"/usr/sbin/pct shutdown 9201",
|
||||
"/usr/sbin/pct shutdown 9201 --timeout -1",
|
||||
"/usr/bin/systemctl poweroff --force",
|
||||
"/usr/bin/systemctl poweroff -ff",
|
||||
"/usr/bin/systemctl reboot",
|
||||
"/usr/bin/systemctl halt",
|
||||
"/usr/bin/systemctl kexec",
|
||||
// R-1006: the partial-cleanup helper takes exactly one storage, one vmid, one half-written name.
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.zst",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.log",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 ../../etc/vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat /etc/shadow",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat;id",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local;id 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean --help",
|
||||
"/usr/local/sbin/felhom-internal-place create 132",
|
||||
"/usr/local/sbin/felhom-internal-place create 0 extra",
|
||||
"/usr/local/sbin/felhom-internal-place create 0;id",
|
||||
"/usr/local/sbin/felhom-internal-place wipe 0",
|
||||
"/usr/local/sbin/felhom-internal-place create",
|
||||
}
|
||||
|
||||
func TestSudoersRefusesTheR861Injections(t *testing.T) {
|
||||
|
||||
@@ -123,6 +123,7 @@ type Collector struct {
|
||||
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
||||
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
|
||||
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
|
||||
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
||||
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
|
||||
@@ -255,6 +256,12 @@ func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector
|
||||
return c
|
||||
}
|
||||
|
||||
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
|
||||
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
|
||||
c.deliberateOff = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
|
||||
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
|
||||
type SelfUpdateReporter interface {
|
||||
@@ -431,6 +438,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
if c.oob != nil {
|
||||
report.OOB = c.oob.OOBStatus(ctx)
|
||||
}
|
||||
// R-998: a household power-off accepted in this boot rides every report until the box is down.
|
||||
if c.deliberateOff != nil {
|
||||
report.DeliberateOff = c.deliberateOff()
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -39,11 +39,14 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
|
||||
State: StorageStateAttached, Reachable: true, ClassHint: "fast",
|
||||
ThinPool: &ThinPoolFill{DataUsedFraction: 0.42},
|
||||
Smart: SmartSummary{Health: SmartUnknown},
|
||||
// v0.157.0 (`16` §3): the identity keys ride on both rows so the key-set guard sees them.
|
||||
Kind: "system", PhysicalDisks: []string{"serial:ata:SanDisk_X600:182195804614"}, Identity: "proven",
|
||||
},
|
||||
{
|
||||
Name: "usb-backup", Type: StorageTypeUSB, DurableID: "uuid:x",
|
||||
State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000,
|
||||
Smart: SmartSummary{Health: SmartUnknown},
|
||||
Kind: "disk", PhysicalDisks: []string{"serial:usb:External:000000000000"}, Identity: "unknown", IdentityReason: "usb_bridge",
|
||||
},
|
||||
// A pbs target so the recipe's pbs coord has a storage.cfg row to resolve its namespace
|
||||
// from (R-106). Without one the fixture produces namespace_state=unknown while the golden
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-998: the power-off path waits on SetReportedHook, so the hook must fire only for a report the hub
|
||||
// ACCEPTED — a failed push must not let the box power off believing the hub knows.
|
||||
func TestLoop_ReportedHookOnlyAfterAcceptedReport(t *testing.T) {
|
||||
var cn, rn int32
|
||||
rep := &HostReport{DeliberateOff: &DeliberateOff{RequestedAt: "2026-10-13T18:00:00Z"}}
|
||||
loop := NewLoop(&fakeCollector{report: rep, n: &cn},
|
||||
&fakeReporter{env: &ControlEnvelope{}, errSeq: []error{errors.New("hub 5xx")}, n: &rn},
|
||||
900*time.Second, quietLogger())
|
||||
var seen []*HostReport
|
||||
loop.SetReportedHook(func(r *HostReport) { seen = append(seen, r) })
|
||||
|
||||
loop.cycle(context.Background(), 900*time.Second) // the push fails
|
||||
if len(seen) != 0 {
|
||||
t.Fatalf("hook fired for a refused report: %d", len(seen))
|
||||
}
|
||||
loop.cycle(context.Background(), 900*time.Second) // the push is accepted
|
||||
if len(seen) != 1 || seen[0].DeliberateOff == nil || seen[0].DeliberateOff.RequestedAt != "2026-10-13T18:00:00Z" {
|
||||
t.Fatalf("hook after an accepted report = %+v", seen)
|
||||
}
|
||||
}
|
||||
@@ -66,8 +66,15 @@ type Loop struct {
|
||||
// retry, no duplicate shipping). Loop state is single-goroutine (cycle only).
|
||||
logTailSource func(maxBytes int) []string
|
||||
logTailPending bool
|
||||
|
||||
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
|
||||
// path waits on it so the hub has the deliberate-off record before the box goes down.
|
||||
reported func(r *HostReport)
|
||||
}
|
||||
|
||||
// SetReportedHook wires the R-998 after-report observer. Optional.
|
||||
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
|
||||
|
||||
// logTailMaxBytes caps the heartbeat log tail (newest lines kept).
|
||||
const logTailMaxBytes = 128 * 1024
|
||||
|
||||
@@ -156,6 +163,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
|
||||
l.logger.Warn("hub: report failed; keeping current interval", "err", err)
|
||||
return current
|
||||
}
|
||||
if l.reported != nil {
|
||||
l.reported(report)
|
||||
}
|
||||
if report.LogTail != nil {
|
||||
// Transparency: the pull is visible in the box's own log (and thus in the ring).
|
||||
l.logger.Info("operator log pull served", "component", "agent", "lines", len(report.LogTail.Lines))
|
||||
|
||||
@@ -144,6 +144,12 @@ type HostReport struct {
|
||||
// in the steady state, so the cross-repo host-report golden stays byte-stable.
|
||||
LogTail *LogTail `json:"log_tail,omitempty"`
|
||||
|
||||
// DeliberateOff (R-998, doc 14): the household asked this box to power off, in THIS boot. Present on
|
||||
// every report from the request until the box goes down; absent again after the next boot — that
|
||||
// absence is how the hub learns the box is back. The hub holds its silence alarms while it is set
|
||||
// (14 days). `omitempty`: absent in the steady state, so the cross-repo host-report golden stays stable.
|
||||
DeliberateOff *DeliberateOff `json:"deliberate_off,omitempty"`
|
||||
|
||||
// OOB is the operator-access health stanza (TASK H1). It answers the operator's question — "can I
|
||||
// get into this box right now, and if not, why" — from the hub: felhom-sshd up + on which port,
|
||||
// locally reachable, the tunnel handshake age (the OOB path rides wg-felhom), whether the operator
|
||||
@@ -425,6 +431,15 @@ type StorageTarget struct {
|
||||
// metadata fill is null until Phase B's privileged `lvs` read.
|
||||
ThinPool *ThinPoolFill `json:"thin_pool,omitempty"`
|
||||
|
||||
// Physical-disk identity (felhom.eu architecture/16-storage-physical-identity.md §3, agent v0.157.0):
|
||||
// the place's kind, the keys of the whole physical disks under it, and whether the agent can prove
|
||||
// them (proven | unknown + identity_reason vm|network|usb_bridge|weak_wwn|no_serial). Agent-computed
|
||||
// from /sys + udev only; empty when not computed (an absent device, an older agent). Additive.
|
||||
Kind string `json:"kind,omitempty"`
|
||||
PhysicalDisks []string `json:"physical_disks,omitempty"`
|
||||
Identity string `json:"identity,omitempty"`
|
||||
IdentityReason string `json:"identity_reason,omitempty"`
|
||||
|
||||
// Smart is the disk-health summary, populated in Phase B via smartctl (sudoers). In
|
||||
// Phase A it is {health:"UNKNOWN", <counters null>} — the keys are committed now so the
|
||||
// contract is stable across both phases.
|
||||
@@ -758,3 +773,8 @@ type ForeignKeyArchives struct {
|
||||
Oldest string `json:"oldest"`
|
||||
Newest string `json:"newest"`
|
||||
}
|
||||
|
||||
// DeliberateOff is the household-requested power-off record (R-998).
|
||||
type DeliberateOff struct {
|
||||
RequestedAt string `json:"requested_at"` // RFC3339 UTC, the agent's clock
|
||||
}
|
||||
|
||||
@@ -46,6 +46,9 @@
|
||||
"backing_device": "",
|
||||
"class_hint": "fast",
|
||||
"role": "",
|
||||
"kind": "system",
|
||||
"physical_disks": ["serial:ata:SanDisk_X600:182195804614"],
|
||||
"identity": "proven",
|
||||
"thin_pool": { "data_used_fraction": 0.42, "metadata_used_fraction": null },
|
||||
"smart": {
|
||||
"health": "UNKNOWN",
|
||||
@@ -74,6 +77,10 @@
|
||||
"backing_device": "/dev/sdb1",
|
||||
"class_hint": "slow",
|
||||
"role": "",
|
||||
"kind": "disk",
|
||||
"physical_disks": ["serial:usb:External:000000000000"],
|
||||
"identity": "unknown",
|
||||
"identity_reason": "usb_bridge",
|
||||
"smart": {
|
||||
"health": "UNKNOWN",
|
||||
"temperature_c": null,
|
||||
|
||||
@@ -19,10 +19,10 @@ import (
|
||||
// again", and each guard is pinned by the case where acting would be wrong.
|
||||
|
||||
type supExec struct {
|
||||
mu sync.Mutex
|
||||
status map[int]string // docker .State.Status per vmid; "" = container absent
|
||||
inspectErr error // non-nil = pct exec itself failed (unknown)
|
||||
restarts map[int]int
|
||||
mu sync.Mutex
|
||||
status map[int]string // docker .State.Status per vmid; "" = container absent
|
||||
inspectErr error // non-nil = pct exec itself failed (unknown)
|
||||
restarts map[int]int
|
||||
// onRestart, when set, is the status the container reaches after a restart (a crash-looper
|
||||
// stays "exited").
|
||||
onRestart string
|
||||
@@ -249,7 +249,6 @@ func TestControllerSupervisorStanza_WireShape(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ---- R-539 (operator ruling 3 of 2026-09-16): the SLOW crash loop ----------------------------------
|
||||
|
||||
// supKillOnce kills the controller and lets the supervisor restart it (two confirming sweeps), then
|
||||
|
||||
@@ -188,6 +188,13 @@ type DiskInfo struct {
|
||||
// device that exposes no SMART (USB bridge, unread) is ABSENT, not a misleading zero-value UNKNOWN;
|
||||
// the controller feature-detects presence and renders "Nincs adat" when nil (never alarms).
|
||||
Smart *hub.SmartSummary `json:"smart,omitempty"`
|
||||
// Physical-disk identity (v0.157.0, felhom.eu architecture/16-storage-physical-identity.md §3) — the
|
||||
// same agent-computed verdict the hub report carries. Empty when not computed (an absent drive, no
|
||||
// resolver). Additive; an older controller ignores it.
|
||||
Kind string `json:"kind,omitempty"`
|
||||
PhysicalDisks []string `json:"physical_disks,omitempty"`
|
||||
Identity string `json:"identity,omitempty"`
|
||||
IdentityReason string `json:"identity_reason,omitempty"`
|
||||
}
|
||||
|
||||
// handleDisks lists the host's drives + data-bearing flags (read-only/benign).
|
||||
@@ -227,6 +234,8 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
// really is the target. Reported here so the controller can name the drive in an
|
||||
// absent-target alarm and hide the destructive controls the agent would refuse anyway.
|
||||
BackupTarget: t.Name == primaryTargetID,
|
||||
// `16` §3: copied from the observer, which computed it from /sys + udev.
|
||||
Kind: t.Kind, PhysicalDisks: t.PhysicalDisks, Identity: t.Identity, IdentityReason: t.IdentityReason,
|
||||
}
|
||||
// Intermediary model: the stable in-guest path + whether felhom-data is bound under the parent.
|
||||
// Only user-data /mnt/<name> drives have a guest path (system/backup mounts never cross in).
|
||||
@@ -398,6 +407,10 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
di.Smart = &sm
|
||||
}
|
||||
}
|
||||
// `16` §3: a registry drive's identity, judged on its OWN mount only (never root's).
|
||||
if p, kind, ok := s.ownMountIdentity(d.MountPath); ok {
|
||||
di.Kind, di.PhysicalDisks, di.Identity, di.IdentityReason = kind, p.PhysicalDisks, p.Identity, p.Reason
|
||||
}
|
||||
// R-118: statfs ONLY while the drive's device is present. With the device gone the raw
|
||||
// mountpoint reverts to a bare directory on the ROOT filesystem, and statfs would report
|
||||
// pve-root's size as this drive's (measured: a 4 GB drive advertising 46 GiB). Same trap
|
||||
|
||||
@@ -226,6 +226,9 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
// The reply's fields are read HERE, before the job goroutine may write job.Phase (a data race the race
|
||||
// detector caught, 2026-10-13 — the reply could show the finished phase or a torn value).
|
||||
jobID, phase := job.JobID, job.Phase
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
@@ -274,7 +277,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
|
||||
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
|
||||
}()
|
||||
s.escrowDone = done // tests wait on it; production polls the status endpoint
|
||||
writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": job.JobID, "phase": job.Phase}, "")
|
||||
writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": jobID, "phase": phase}, "")
|
||||
}
|
||||
|
||||
// handleEscrowCeremonyStatus is GET /escrow/ceremony/status: phase + the NON-SECRET summary.
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
@@ -104,6 +105,26 @@ func (b *GuestBinder) RebootGuest(ctx context.Context, vmid int) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// ShutdownGuest stops the guest cleanly with a time limit (R-998 host power-off, step 2). `pct shutdown`
|
||||
// blocks until the guest is down or the limit passes; then it fails and the guest is still running.
|
||||
func (b *GuestBinder) ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error {
|
||||
secs := int(timeout / time.Second)
|
||||
if secs < 1 {
|
||||
secs = 1
|
||||
}
|
||||
b.logger.Warn("host-poweroff: shutting the guest down", "vmid", vmid, "timeout_s", secs)
|
||||
if err := b.run(ctx, "pct", "shutdown", strconv.Itoa(vmid), "--timeout", strconv.Itoa(secs)); err != nil {
|
||||
return fmt.Errorf("host-poweroff: pct shutdown %d: %w", vmid, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PowerOffHost powers the Proxmox host off (R-998, step 3). The one fixed vector `/usr/bin/systemctl poweroff`.
|
||||
func (b *GuestBinder) PowerOffHost(ctx context.Context) error {
|
||||
b.logger.Warn("host-poweroff: powering the host off")
|
||||
return b.run(ctx, "/usr/bin/systemctl", "poweroff")
|
||||
}
|
||||
|
||||
func (b *GuestBinder) run(ctx context.Context, name string, args ...string) error {
|
||||
_, stderr, err := b.runner.Run(ctx, name, args...)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
)
|
||||
|
||||
// ── POST /host/poweroff (R-998, documentation/architecture/14-host-power-off.md) ──────────────────────
|
||||
//
|
||||
// The household's „A gép kikapcsolása": shut THIS guest down cleanly, then power the whole host off. The
|
||||
// first local-API route whose effect reaches beyond the caller's own guest — a compromised controller can
|
||||
// now switch the box off, and nobody can switch it on remotely. The mitigations are the limits below:
|
||||
//
|
||||
// - refused in the first 10 minutes after boot (no power-off loop);
|
||||
// - at most ONE accepted request per boot (poweroff.State, persisted, keyed by the boot);
|
||||
// - refused while a backup, restore test, OS update step, controller update, drive format, NAS verify
|
||||
// or escrow ceremony runs, or a vzdump of the guest is in flight — each named in the 409;
|
||||
// - refused while ANY OTHER guest or VM runs on the host (security review 2026-10-13): one guest's
|
||||
// controller must not switch off another household's guest, a scratch guest or a test VM (demo-hp
|
||||
// carries the Tester 1 VM). Read from /proc (`lxc-start -n N`, `kvm -id N`), fail closed;
|
||||
// - the heavy-operation gate is TAKEN on acceptance and never given back (the box is going down), so no
|
||||
// backup, restore test or OS step can start between the checks and the power-off (security review).
|
||||
// Released only if the power-off command itself fails.
|
||||
//
|
||||
// Every check fails CLOSED: an unwired gate, an unreadable task list or /proc refuses.
|
||||
// On acceptance the record is written FIRST, the hub is given a report carrying it (bounded wait), and
|
||||
// only then: guest shutdown (time-limited), host power-off. Pinned by hostpoweroff_test.go.
|
||||
|
||||
// HostPowerer is the privileged half: both run as fixed sudo vectors (configs/felhom-agent.sudoers).
|
||||
type HostPowerer interface {
|
||||
ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error
|
||||
PowerOffHost(ctx context.Context) error
|
||||
}
|
||||
|
||||
// HostPowerOffConfig wires the route. A nil config leaves it answering 503.
|
||||
type HostPowerOffConfig struct {
|
||||
State *poweroff.State
|
||||
Power HostPowerer
|
||||
ReportNow func(ctx context.Context) bool // push a report carrying the record; true = the hub accepted it
|
||||
Uptime func() (time.Duration, bool) // time since host boot; false = unknown (refuse)
|
||||
Running func() ([]int, error) // vmids of every running guest and VM on the host; default /proc
|
||||
MinUptime time.Duration // default 10 min
|
||||
GuestStop time.Duration // pct shutdown --timeout; default 120 s
|
||||
ReportMax time.Duration // longest wait for the hub; default 60 s
|
||||
JobWait time.Duration // longest wait for a job that started after acceptance; default 15 min
|
||||
JobPoll time.Duration // how often that wait looks again; default 10 s
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
// Refusal codes — the controller maps each to a household sentence. Stable wire values.
|
||||
const (
|
||||
PowerOffTooSoon = "too_soon_after_boot"
|
||||
PowerOffAlreadyAsked = "already_requested_this_boot"
|
||||
PowerOffBackup = "backup_running"
|
||||
PowerOffRestoreTest = "restore_test_running"
|
||||
PowerOffOSUpdate = "os_update_running"
|
||||
PowerOffCtrlUpdate = "controller_update_running"
|
||||
PowerOffDrive = "drive_operation_running"
|
||||
PowerOffEscrow = "escrow_ceremony_running"
|
||||
PowerOffBusy = "host_busy"
|
||||
PowerOffOtherGuests = "other_guests_running"
|
||||
PowerOffUnknown = "state_unknown"
|
||||
)
|
||||
|
||||
// SetHostPowerOff enables POST /host/poweroff.
|
||||
func (s *Server) SetHostPowerOff(c HostPowerOffConfig) {
|
||||
if c.MinUptime == 0 {
|
||||
c.MinUptime = 10 * time.Minute
|
||||
}
|
||||
if c.GuestStop == 0 {
|
||||
c.GuestStop = 120 * time.Second
|
||||
}
|
||||
if c.ReportMax == 0 {
|
||||
c.ReportMax = 60 * time.Second
|
||||
}
|
||||
if c.Now == nil {
|
||||
c.Now = time.Now
|
||||
}
|
||||
if c.JobWait == 0 {
|
||||
c.JobWait = 15 * time.Minute
|
||||
}
|
||||
if c.JobPoll == 0 {
|
||||
c.JobPoll = 10 * time.Second
|
||||
}
|
||||
if c.Running == nil {
|
||||
c.Running = func() ([]int, error) { return runningGuestsFromProc("/proc") }
|
||||
}
|
||||
s.powerOff = &c
|
||||
}
|
||||
|
||||
type hostPowerOffRequest struct {
|
||||
VMID int `json:"vmid"`
|
||||
}
|
||||
|
||||
func (s *Server) handleHostPowerOff(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
c := s.powerOff
|
||||
if c == nil || c.State == nil || c.Power == nil {
|
||||
writeErr(w, http.StatusServiceUnavailable, "host power-off not configured on this host")
|
||||
return
|
||||
}
|
||||
if r.ContentLength != 0 {
|
||||
var req hostPowerOffRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||
return
|
||||
}
|
||||
}
|
||||
// One decision at a time: two requests racing must not both pass the guards before either marks.
|
||||
s.powerOffMu.Lock()
|
||||
defer s.powerOffMu.Unlock()
|
||||
if code, detail := s.powerOffBlocker(r.Context(), vmid); code != "" {
|
||||
s.logger.Info("local-api: host power-off refused", "vmid", vmid, "reason", code, "detail", detail)
|
||||
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": detail}, "host power-off refused: "+code)
|
||||
return
|
||||
}
|
||||
// Take the heavy-operation gate for good: nothing heavy may start between these checks and the power-off.
|
||||
release, busy, free := s.inFlight.TryAcquire("host-poweroff")
|
||||
if !free {
|
||||
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": PowerOffBusy, "detail": busy}, "host power-off refused: "+PowerOffBusy)
|
||||
return
|
||||
}
|
||||
rec, err := c.State.Mark(c.Now(), vmid)
|
||||
if err != nil {
|
||||
release()
|
||||
code := PowerOffUnknown
|
||||
if errors.Is(err, poweroff.ErrAlreadyThisBoot) {
|
||||
code = PowerOffAlreadyAsked
|
||||
}
|
||||
s.logger.Warn("local-api: host power-off refused (record)", "vmid", vmid, "reason", code, "err", err)
|
||||
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": err.Error()}, "host power-off refused: "+code)
|
||||
return
|
||||
}
|
||||
s.logger.Warn("local-api: HOST POWER-OFF accepted (household request)", "vmid", vmid,
|
||||
"requested_at", rec.RequestedAt.Format(time.RFC3339))
|
||||
base := s.baseCtx
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
go s.runHostPowerOff(base, vmid, release)
|
||||
writeStatus(w, http.StatusAccepted, true, map[string]any{
|
||||
"vmid": vmid, "powering_off": true, "requested_at": rec.RequestedAt.Format(time.RFC3339),
|
||||
}, "")
|
||||
}
|
||||
|
||||
// runHostPowerOff: hub first (bounded), then the guest (time-limited), then the host. A failed guest
|
||||
// shutdown does not stop the power-off — the household asked for it, and Proxmox's own shutdown stops
|
||||
// any guest still running (pve-guests) before the host goes down.
|
||||
func (s *Server) runHostPowerOff(base context.Context, vmid int, release func()) {
|
||||
c := s.powerOff
|
||||
if c.ReportNow != nil {
|
||||
ctx, cancel := context.WithTimeout(base, c.ReportMax)
|
||||
ok := c.ReportNow(ctx)
|
||||
cancel()
|
||||
s.logger.Warn("host-poweroff: step 1/3 — hub told", "vmid", vmid, "hub_accepted", ok)
|
||||
}
|
||||
if !s.waitJobsClear(base, vmid, "before the guest shutdown") {
|
||||
release()
|
||||
return
|
||||
}
|
||||
start := time.Now()
|
||||
ctx, cancel := context.WithTimeout(base, c.GuestStop+60*time.Second)
|
||||
err := c.Power.ShutdownGuest(ctx, vmid, c.GuestStop)
|
||||
cancel()
|
||||
if err != nil {
|
||||
s.logger.Error("host-poweroff: step 2/3 — guest shutdown failed; powering off anyway", "vmid", vmid,
|
||||
"took", time.Since(start).Round(time.Second), "err", err)
|
||||
} else {
|
||||
s.logger.Warn("host-poweroff: step 2/3 — guest stopped", "vmid", vmid, "took", time.Since(start).Round(time.Second))
|
||||
}
|
||||
if !s.waitJobsClear(base, vmid, "before the host power-off") {
|
||||
release()
|
||||
return
|
||||
}
|
||||
s.logger.Warn("host-poweroff: step 3/3 — powering the host off")
|
||||
ctx, cancel = context.WithTimeout(base, 60*time.Second)
|
||||
defer cancel()
|
||||
if err := c.Power.PowerOffHost(ctx); err != nil {
|
||||
s.logger.Error("host-poweroff: host power-off command failed — heavy-operation gate released", "err", err)
|
||||
release()
|
||||
}
|
||||
}
|
||||
|
||||
// powerOffBlocker returns the first reason to refuse ("" = none) and a detail for the log/operator.
|
||||
func (s *Server) powerOffBlocker(ctx context.Context, vmid int) (string, string) {
|
||||
c := s.powerOff
|
||||
if c.State.Current() != nil {
|
||||
return PowerOffAlreadyAsked, "a power-off was already accepted in this boot"
|
||||
}
|
||||
if c.Uptime == nil {
|
||||
return PowerOffUnknown, "uptime not wired"
|
||||
}
|
||||
up, ok := c.Uptime()
|
||||
if !ok {
|
||||
return PowerOffUnknown, "cannot read the host uptime"
|
||||
}
|
||||
if up < c.MinUptime {
|
||||
return PowerOffTooSoon, "host up " + up.Round(time.Second).String() + ", minimum " + c.MinUptime.String()
|
||||
}
|
||||
if s.inFlight == nil || s.staleLock == nil {
|
||||
return PowerOffUnknown, "the heavy-operation gate or the Proxmox task reader is not wired"
|
||||
}
|
||||
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
|
||||
return code, detail
|
||||
}
|
||||
if busy := s.inFlight.Busy(); busy != "" {
|
||||
switch {
|
||||
case strings.HasPrefix(busy, "backup:"):
|
||||
return PowerOffBackup, busy
|
||||
case busy == "restore-test":
|
||||
return PowerOffRestoreTest, busy
|
||||
case strings.HasPrefix(busy, "os-"):
|
||||
return PowerOffOSUpdate, busy
|
||||
default:
|
||||
return PowerOffBusy, busy
|
||||
}
|
||||
}
|
||||
return s.jobBlocker(ctx, vmid)
|
||||
}
|
||||
|
||||
// jobBlocker is the part of the checks that can change after acceptance — a backup job, a controller update, a drive
|
||||
// or NAS job, an escrow ceremony, a Proxmox vzdump. runHostPowerOff asks it again before each step (security review
|
||||
// 2026-10-13: the checks ran only once). The heavy-operation gate is not here: the power-off holds it.
|
||||
func (s *Server) jobBlocker(ctx context.Context, vmid int) (string, string) {
|
||||
s.jobsMu.Lock()
|
||||
for k, j := range s.jobs {
|
||||
if j != nil && backupInFlight(j.Phase) {
|
||||
s.jobsMu.Unlock()
|
||||
return PowerOffBackup, "backup job " + j.JobID + " on target " + k.target
|
||||
}
|
||||
}
|
||||
s.jobsMu.Unlock()
|
||||
s.swapMu.Lock()
|
||||
for v, on := range s.swapInFlight {
|
||||
if on {
|
||||
s.swapMu.Unlock()
|
||||
return PowerOffCtrlUpdate, "controller swap in flight for guest " + strconv.Itoa(v)
|
||||
}
|
||||
}
|
||||
s.swapMu.Unlock()
|
||||
if s.formatJobs != nil {
|
||||
if j := s.formatJobs.get(); j != nil && j.Phase == formatPhaseRunning {
|
||||
return PowerOffDrive, "drive format running"
|
||||
}
|
||||
}
|
||||
s.netVerifyMu.Lock()
|
||||
nv := s.netVerifyCur != nil && s.netVerifyCur.Phase == netVerifyPhaseRunning
|
||||
s.netVerifyMu.Unlock()
|
||||
if nv {
|
||||
return PowerOffDrive, "network storage verify running"
|
||||
}
|
||||
s.escrowMu.Lock()
|
||||
esc := s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning
|
||||
s.escrowMu.Unlock()
|
||||
if esc {
|
||||
return PowerOffEscrow, "escrow ceremony running"
|
||||
}
|
||||
vz, err := s.staleLock.BackupRunning(ctx, vmid)
|
||||
if err != nil {
|
||||
return PowerOffUnknown, "cannot list running Proxmox tasks: " + err.Error()
|
||||
}
|
||||
if vz {
|
||||
return PowerOffBackup, "vzdump running for guest " + strconv.Itoa(vmid)
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// runningGuestsFromProc lists the vmids of running containers (`lxc-start … -n N`) and VMs (`kvm -id N`) from
|
||||
// the process table — readable by the non-root agent, unlike /run/qemu-server (measured on demo-hp 2026-10-13).
|
||||
// An unreadable /proc is an error (fail closed); an unreadable single process is skipped (it exited).
|
||||
func runningGuestsFromProc(procDir string) ([]int, error) {
|
||||
ents, err := os.ReadDir(procDir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
seen := map[int]bool{}
|
||||
var out []int
|
||||
for _, e := range ents {
|
||||
if _, err := strconv.Atoi(e.Name()); err != nil {
|
||||
continue
|
||||
}
|
||||
raw, err := os.ReadFile(filepath.Join(procDir, e.Name(), "cmdline"))
|
||||
if err != nil || len(raw) == 0 {
|
||||
continue
|
||||
}
|
||||
args := strings.Split(strings.TrimRight(string(raw), "\x00"), "\x00")
|
||||
flag := ""
|
||||
switch filepath.Base(args[0]) {
|
||||
case "lxc-start":
|
||||
flag = "-n"
|
||||
case "kvm", "qemu-system-x86_64":
|
||||
flag = "-id"
|
||||
default:
|
||||
continue
|
||||
}
|
||||
for i := 1; i+1 < len(args); i++ {
|
||||
if args[i] == flag {
|
||||
if v, err := strconv.Atoi(args[i+1]); err == nil && !seen[v] {
|
||||
seen[v] = true
|
||||
out = append(out, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// otherGuestsBlocker refuses while any guest or VM other than the caller's runs (or the list cannot be read).
|
||||
func (s *Server) otherGuestsBlocker(vmid int) (string, string) {
|
||||
running, err := s.powerOff.Running()
|
||||
if err != nil {
|
||||
return PowerOffUnknown, "cannot list the running guests: " + err.Error()
|
||||
}
|
||||
var others []string
|
||||
for _, v := range running {
|
||||
if v != vmid {
|
||||
others = append(others, strconv.Itoa(v))
|
||||
}
|
||||
}
|
||||
if len(others) > 0 {
|
||||
return PowerOffOtherGuests, "other guests or VMs running: " + strings.Join(others, ", ")
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// waitJobsClear waits until jobBlocker finds nothing, at most JobWait. false = still busy (or unreadable) at the end:
|
||||
// the power-off is CANCELLED — the box stays on, the gate is released, the log says why. Better a box that did not
|
||||
// switch off than a backup, update or format cut in the middle.
|
||||
func (s *Server) waitJobsClear(base context.Context, vmid int, when string) bool {
|
||||
c := s.powerOff
|
||||
ctx, cancel := context.WithTimeout(base, c.JobWait)
|
||||
defer cancel()
|
||||
for {
|
||||
// Another guest or VM started after acceptance (security review 2026-10-13): no waiting — cancel at once.
|
||||
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
|
||||
s.logger.Error("host-poweroff: CANCELLED — another guest is running now; the box stays on", "when", when,
|
||||
"reason", code, "detail", detail)
|
||||
return false
|
||||
}
|
||||
code, detail := s.jobBlocker(ctx, vmid)
|
||||
if code == "" {
|
||||
return true
|
||||
}
|
||||
s.logger.Warn("host-poweroff: waiting — a job started after the request", "when", when, "reason", code, "detail", detail)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
s.logger.Error("host-poweroff: CANCELLED — still busy after the wait; the box stays on", "when", when,
|
||||
"reason", code, "detail", detail, "waited", c.JobWait)
|
||||
return false
|
||||
case <-time.After(c.JobPoll):
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,353 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// fakePower records the power-off steps in order.
|
||||
type fakePower struct {
|
||||
mu sync.Mutex
|
||||
steps []string
|
||||
}
|
||||
|
||||
func (f *fakePower) ShutdownGuest(_ context.Context, vmid int, timeout time.Duration) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.steps = append(f.steps, "shutdown:"+itoaT(vmid)+":"+timeout.String())
|
||||
return nil
|
||||
}
|
||||
func (f *fakePower) PowerOffHost(context.Context) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.steps = append(f.steps, "poweroff")
|
||||
return nil
|
||||
}
|
||||
func (f *fakePower) got() []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]string(nil), f.steps...)
|
||||
}
|
||||
|
||||
func itoaT(n int) string { b, _ := json.Marshal(n); return string(b) }
|
||||
|
||||
type fakeVzdump struct{ running bool }
|
||||
|
||||
func (f fakeVzdump) Guests(context.Context) ([]proxmox.Guest, error) { return nil, nil }
|
||||
func (f fakeVzdump) Lock(context.Context, int) (string, bool, error) { return "", true, nil }
|
||||
func (f fakeVzdump) BackupRunning(context.Context, int) (bool, error) { return f.running, nil }
|
||||
func (f fakeVzdump) Unlock(context.Context, int) error { return nil }
|
||||
func (f fakeVzdump) Start(context.Context, int) error { return nil }
|
||||
func (f fakeVzdump) HasVzdumpSnapshot(context.Context, int) (bool, error) { return false, nil }
|
||||
func (f fakeVzdump) DeleteVzdumpSnapshot(context.Context, int) error { return nil }
|
||||
|
||||
type powerRig struct {
|
||||
srv *Server
|
||||
h http.Handler
|
||||
power *fakePower
|
||||
state *poweroff.State
|
||||
report *[]string
|
||||
}
|
||||
|
||||
func newPowerRig(t *testing.T, uptime time.Duration) *powerRig {
|
||||
t.Helper()
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
srv.inFlight = &backup.InFlight{}
|
||||
srv.staleLock = fakeVzdump{}
|
||||
p := &fakePower{}
|
||||
st := poweroff.New(filepath.Join(t.TempDir(), "poweroff.json"), func() string { return "1700000000" })
|
||||
var reports []string
|
||||
srv.SetHostPowerOff(HostPowerOffConfig{
|
||||
State: st,
|
||||
Power: p,
|
||||
ReportNow: func(context.Context) bool {
|
||||
// The record must already exist when the hub is told — that is the point of telling it.
|
||||
if st.Current() == nil {
|
||||
reports = append(reports, "report-without-record")
|
||||
} else if len(p.got()) != 0 {
|
||||
reports = append(reports, "report-after-"+p.got()[0])
|
||||
} else {
|
||||
reports = append(reports, "report")
|
||||
}
|
||||
return true
|
||||
},
|
||||
Uptime: func() (time.Duration, bool) { return uptime, true },
|
||||
Running: func() ([]int, error) { return []int{8200}, nil },
|
||||
Now: func() time.Time { return testNow },
|
||||
})
|
||||
return &powerRig{srv: srv, h: srv.Handler(), power: p, state: st, report: &reports}
|
||||
}
|
||||
|
||||
func (r *powerRig) waitSteps(t *testing.T, n int) []string {
|
||||
t.Helper()
|
||||
for i := 0; i < 200 && len(r.power.got()) < n; i++ {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
return r.power.got()
|
||||
}
|
||||
|
||||
func reasonOf(t *testing.T, body []byte) string {
|
||||
t.Helper()
|
||||
var resp struct {
|
||||
Data struct {
|
||||
Reason string `json:"reason"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
t.Fatalf("body %s: %v", body, err)
|
||||
}
|
||||
return resp.Data.Reason
|
||||
}
|
||||
|
||||
// The whole path: 202, the record written, the hub told FIRST, then guest shutdown (120 s), then host off.
|
||||
func TestHostPowerOff_AcceptedOrder(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusAccepted {
|
||||
t.Fatalf("got %d want 202 (%s)", w.Code, w.Body.String())
|
||||
}
|
||||
steps := r.waitSteps(t, 2)
|
||||
want := []string{"shutdown:8200:2m0s", "poweroff"}
|
||||
if len(steps) != 2 || steps[0] != want[0] || steps[1] != want[1] {
|
||||
t.Fatalf("steps = %v, want %v", steps, want)
|
||||
}
|
||||
// The heavy-operation gate stays taken: no backup can start between the checks and the power-off.
|
||||
if _, busy, ok := r.srv.inFlight.TryAcquire("backup:primary"); ok || busy != "host-poweroff" {
|
||||
t.Fatalf("after acceptance a backup could take the gate (ok=%v busy=%q)", ok, busy)
|
||||
}
|
||||
if len(*r.report) != 1 || (*r.report)[0] != "report" {
|
||||
t.Fatalf("hub report = %v, want one report carrying the record", *r.report)
|
||||
}
|
||||
if rec := r.state.Current(); rec == nil || rec.VMID != 8200 {
|
||||
t.Fatalf("record = %+v", rec)
|
||||
}
|
||||
}
|
||||
|
||||
// Refused during a backup — the shared heavy-operation gate held by a backup tier.
|
||||
func TestHostPowerOff_RefusedDuringBackup(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
release, _, ok := r.srv.inFlight.TryAcquire("backup:primary")
|
||||
if !ok {
|
||||
t.Fatal("could not hold the gate")
|
||||
}
|
||||
defer release()
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 0 {
|
||||
t.Fatalf("power steps ran during a backup: %v", s)
|
||||
}
|
||||
if r.state.Current() != nil {
|
||||
t.Fatal("a refused request must not use up the boot's one request")
|
||||
}
|
||||
// The same guard names a restore test and an OS step.
|
||||
release()
|
||||
for gate, want := range map[string]string{"restore-test": PowerOffRestoreTest, "os-pve-step": PowerOffOSUpdate} {
|
||||
rel, _, _ := r.srv.inFlight.TryAcquire(gate)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
rel()
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != want {
|
||||
t.Fatalf("%s: got %d %s, want 409 %s", gate, w.Code, w.Body.String(), want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A vzdump of the guest in flight (a Proxmox task, not the agent's own gate) refuses too.
|
||||
func TestHostPowerOff_RefusedDuringVzdump(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
r.srv.staleLock = fakeVzdump{running: true}
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
|
||||
}
|
||||
}
|
||||
|
||||
// One accepted request per boot: the second is refused and nothing runs twice.
|
||||
func TestHostPowerOff_SecondRequestSameBootRefused(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("first: %d", w.Code)
|
||||
}
|
||||
r.waitSteps(t, 2)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffAlreadyAsked {
|
||||
t.Fatalf("second: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffAlreadyAsked)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 2 {
|
||||
t.Fatalf("steps = %v, want exactly one run", s)
|
||||
}
|
||||
}
|
||||
|
||||
// Refused in the first 10 minutes after boot; an unknown uptime refuses too (fail closed).
|
||||
func TestHostPowerOff_TooSoonAfterBoot(t *testing.T) {
|
||||
r := newPowerRig(t, 9*time.Minute)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffTooSoon {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffTooSoon)
|
||||
}
|
||||
r.srv.powerOff.Uptime = func() (time.Duration, bool) { return 0, false }
|
||||
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
|
||||
t.Fatalf("unknown uptime: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
|
||||
}
|
||||
}
|
||||
|
||||
// Not configured → 503; another guest's vmid in the body → 403; no token → 401.
|
||||
func TestHostPowerOff_ScopeAndConfig(t *testing.T) {
|
||||
plain := newTestServer(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
if w := do(t, plain, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("unconfigured: %d", w.Code)
|
||||
}
|
||||
r := newPowerRig(t, time.Hour)
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", `{"vmid":9300}`); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("cross-guest: %d", w.Code)
|
||||
}
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "", ""); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: %d", w.Code)
|
||||
}
|
||||
if r.state.Current() != nil || len(r.power.got()) != 0 {
|
||||
t.Fatal("a refused request changed state")
|
||||
}
|
||||
}
|
||||
|
||||
// Another guest or VM on the host refuses: one controller must not switch off someone else's guest (security
|
||||
// review 2026-10-13; demo-hp runs a scratch guest and the Tester 1 VM). An unreadable process list refuses too.
|
||||
func TestHostPowerOff_RefusedWhileOtherGuestsRun(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
r.srv.powerOff.Running = func() ([]int, error) { return []int{8200, 9202, 341}, nil }
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffOtherGuests {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffOtherGuests)
|
||||
}
|
||||
r.srv.powerOff.Running = func() ([]int, error) { return nil, errors.New("no /proc") }
|
||||
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
|
||||
t.Fatalf("unreadable: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
|
||||
}
|
||||
if len(r.power.got()) != 0 || r.state.Current() != nil {
|
||||
t.Fatal("a refused request changed state")
|
||||
}
|
||||
}
|
||||
|
||||
// Fail closed: an unwired vzdump reader refuses instead of skipping the check.
|
||||
func TestHostPowerOff_FailsClosedWithoutTaskReader(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
r.srv.staleLock = nil
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
|
||||
}
|
||||
}
|
||||
|
||||
// The /proc reader finds containers and VMs by their real command lines (as measured on demo-hp).
|
||||
func TestRunningGuestsFromProc(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
write := func(pid, cmd string) {
|
||||
_ = os.MkdirAll(filepath.Join(dir, pid), 0o755)
|
||||
_ = os.WriteFile(filepath.Join(dir, pid, "cmdline"), []byte(strings.ReplaceAll(cmd, " ", "\x00")+"\x00"), 0o644)
|
||||
}
|
||||
write("1927", "/usr/bin/lxc-start -F -n 9201")
|
||||
write("1900", "/usr/bin/lxc-start -F -n 9202")
|
||||
write("1823", "/usr/bin/kvm -id 341 -name night1004-tester1 -no-shutdown")
|
||||
write("974", "/usr/libexec/lxc/lxc-monitord --daemon")
|
||||
write("50", "/usr/bin/bash -c echo -n 5")
|
||||
_ = os.MkdirAll(filepath.Join(dir, "self"), 0o755)
|
||||
got, err := runningGuestsFromProc(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sort.Ints(got)
|
||||
if len(got) != 3 || got[0] != 341 || got[1] != 9201 || got[2] != 9202 {
|
||||
t.Fatalf("got %v, want [341 9201 9202]", got)
|
||||
}
|
||||
if _, err := runningGuestsFromProc(filepath.Join(dir, "missing")); err == nil {
|
||||
t.Fatal("an unreadable /proc must be an error")
|
||||
}
|
||||
}
|
||||
|
||||
// switchVzdump is a vzdump reader the test can flip while the power-off runs.
|
||||
type switchVzdump struct {
|
||||
fakeVzdump
|
||||
on *atomic.Bool
|
||||
}
|
||||
|
||||
func (f switchVzdump) BackupRunning(context.Context, int) (bool, error) { return f.on.Load(), nil }
|
||||
|
||||
// A job that starts AFTER acceptance (security review 2026-10-13): the power-off waits; if the job is still running
|
||||
// when the wait ends, the power-off is cancelled — no guest shutdown, no host power-off, the gate given back.
|
||||
func TestHostPowerOff_JobStartedAfterAcceptance(t *testing.T) {
|
||||
// (a) it clears during the wait → the power-off goes ahead after it
|
||||
r := newPowerRig(t, time.Hour)
|
||||
var on atomic.Bool
|
||||
r.srv.staleLock = switchVzdump{on: &on}
|
||||
r.srv.powerOff.JobWait, r.srv.powerOff.JobPoll = 2*time.Second, 2*time.Millisecond
|
||||
r.srv.powerOff.ReportNow = func(context.Context) bool { on.Store(true); return true } // a vzdump starts now
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
time.Sleep(30 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 0 {
|
||||
t.Fatalf("a step ran while the vzdump ran: %v", s)
|
||||
}
|
||||
on.Store(false)
|
||||
if s := r.waitSteps(t, 2); len(s) != 2 {
|
||||
t.Fatalf("after the vzdump ended the steps = %v, want shutdown + poweroff", s)
|
||||
}
|
||||
|
||||
// (b) it never clears → cancelled, nothing switched off, the heavy gate free again
|
||||
r2 := newPowerRig(t, time.Hour)
|
||||
var on2 atomic.Bool
|
||||
r2.srv.staleLock = switchVzdump{on: &on2}
|
||||
r2.srv.powerOff.JobWait, r2.srv.powerOff.JobPoll = 40*time.Millisecond, 2*time.Millisecond
|
||||
r2.srv.powerOff.ReportNow = func(context.Context) bool { on2.Store(true); return true }
|
||||
if w := do(t, r2.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("accept: %d", w.Code)
|
||||
}
|
||||
time.Sleep(150 * time.Millisecond)
|
||||
if s := r2.power.got(); len(s) != 0 {
|
||||
t.Fatalf("cancelled power-off still ran: %v", s)
|
||||
}
|
||||
if busy := r2.srv.inFlight.Busy(); busy != "" {
|
||||
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
|
||||
}
|
||||
}
|
||||
|
||||
// A guest or VM started AFTER acceptance cancels the power-off at once (security review 2026-10-13).
|
||||
func TestHostPowerOff_OtherGuestStartedAfterAcceptance(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
var started atomic.Bool
|
||||
r.srv.powerOff.Running = func() ([]int, error) {
|
||||
if started.Load() {
|
||||
return []int{8200, 341}, nil
|
||||
}
|
||||
return []int{8200}, nil
|
||||
}
|
||||
r.srv.powerOff.ReportNow = func(context.Context) bool { started.Store(true); return true } // a VM starts now
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 0 {
|
||||
t.Fatalf("power-off ran although a VM started: %v", s)
|
||||
}
|
||||
if busy := r.srv.inFlight.Busy(); busy != "" {
|
||||
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// The internal data place („Belső tárhely", agent v0.158.0) — felhom.eu architecture/16-storage-physical-identity.md
|
||||
// §5, §7. On a one-disk box the root helper configs/felhom-internal-place makes a second thin volume in local-lvm
|
||||
// (<vg>/felhom-belso, ext4, label felhom-belso). The installer (a new box) or the signed add_internal_place job (an old
|
||||
// box) then writes InternalPlaceRecord to the agent's state dir. From then on the agent treats the volume exactly like
|
||||
// an enrolled drive: a Felhom mount unit by fs-UUID at /mnt/belso (EnsureMount), the `enrolled` intent, and — when the
|
||||
// controller asks with its own token (POST /disks/guest-attach) — the bind at /mnt/felhom-drives/belso. So it is a
|
||||
// HOST mount under the guest's mp8 parent: not in the guest config, not in vzdump, not in PBS (`16` §5).
|
||||
//
|
||||
// What it is NOT: a drive the household can format, wipe, eject or decommission. Its device is a device-mapper volume
|
||||
// on the system disk, so the role gate (roleForMountPath → system) and the format path (ValidateBlockDevice refuses
|
||||
// dm) refuse all four — pinned by internalplace_test.go. Its identity is the system disk's (build A), so a Tier-2 copy
|
||||
// between it and sys_drive is never a second copy.
|
||||
const (
|
||||
InternalPlaceName = "belso"
|
||||
InternalPlaceWhere = "/mnt/belso"
|
||||
// InternalPlaceFile is the record's name inside the agent's state dir.
|
||||
InternalPlaceFile = "internal-place.json"
|
||||
)
|
||||
|
||||
var reFSUUID = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||
|
||||
// InternalPlaceRecord is the state file the maker writes once the volume exists and is formatted.
|
||||
type InternalPlaceRecord struct {
|
||||
FSUUID string `json:"fs_uuid"`
|
||||
GiB int `json:"gib,omitempty"`
|
||||
MadeBy string `json:"made_by,omitempty"` // "installer" | "signed-job"
|
||||
MadeAt string `json:"made_at,omitempty"`
|
||||
}
|
||||
|
||||
// ReadInternalPlace reads the record; ok=false when there is none (the common case: a box with no internal place).
|
||||
func ReadInternalPlace(file string) (InternalPlaceRecord, bool, error) {
|
||||
var rec InternalPlaceRecord
|
||||
b, err := os.ReadFile(file)
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return rec, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return rec, false, err
|
||||
}
|
||||
if err := json.Unmarshal(b, &rec); err != nil {
|
||||
return rec, false, fmt.Errorf("internal place record %s: %w", file, err)
|
||||
}
|
||||
if !reFSUUID.MatchString(rec.FSUUID) {
|
||||
return rec, false, fmt.Errorf("internal place record %s: fs_uuid %q is not a filesystem UUID", file, rec.FSUUID)
|
||||
}
|
||||
return rec, true, nil
|
||||
}
|
||||
|
||||
// WriteInternalPlace writes the record atomically (0600).
|
||||
func WriteInternalPlace(file string, rec InternalPlaceRecord) error {
|
||||
if !reFSUUID.MatchString(rec.FSUUID) {
|
||||
return fmt.Errorf("internal place: fs_uuid %q is not a filesystem UUID", rec.FSUUID)
|
||||
}
|
||||
b, err := json.MarshalIndent(rec, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := file + ".tmp"
|
||||
if err := os.WriteFile(tmp, append(b, '\n'), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, file)
|
||||
}
|
||||
|
||||
// internalPlaceState memoises what EnsureInternalPlace last saw, so the 20-second tick logs a state CHANGE once and
|
||||
// does no work on the steady state.
|
||||
type internalPlaceState struct {
|
||||
mu sync.Mutex
|
||||
done string // the fs-UUID that is mounted and recorded enrolled
|
||||
lastWhy string // the last reason it could not be ensured (logged once per change)
|
||||
}
|
||||
|
||||
// internalPlaceUUID returns the configured internal place's fs-UUID, or "" when this box has none (or the record is
|
||||
// unreadable — logged by EnsureInternalPlace).
|
||||
func (s *Server) internalPlaceUUID() string {
|
||||
if s.internalPlaceFile == "" {
|
||||
return ""
|
||||
}
|
||||
rec, ok, err := ReadInternalPlace(s.internalPlaceFile)
|
||||
if err != nil || !ok {
|
||||
return ""
|
||||
}
|
||||
return rec.FSUUID
|
||||
}
|
||||
|
||||
// isInternalPlace reports whether a host path is this box's internal place: its raw mount (/mnt/belso) or its bind
|
||||
// under the guests' shared parent (/mnt/felhom-drives/belso — the path POST /disks/same-disk judges).
|
||||
func (s *Server) isInternalPlace(hostPath string) bool {
|
||||
p := path.Clean(hostPath)
|
||||
return (p == InternalPlaceWhere || p == StableParentDir+"/"+InternalPlaceName) && s.internalPlaceUUID() != ""
|
||||
}
|
||||
|
||||
// EnsureInternalPlace mounts the internal place on the host at /mnt/belso by fs-UUID and records it `enrolled`,
|
||||
// when this box has one. Idempotent and cheap on the steady state; safe on the 20-second tick. It never formats,
|
||||
// never unmounts, and never mounts anything at /mnt/belso that is not the recorded UUID: a household drive that
|
||||
// happens to be named „belso" keeps its mount, and the internal place waits (logged once).
|
||||
func (s *Server) EnsureInternalPlace(ctx context.Context) {
|
||||
if s.internalPlaceFile == "" {
|
||||
return
|
||||
}
|
||||
rec, ok, err := ReadInternalPlace(s.internalPlaceFile)
|
||||
if err != nil {
|
||||
s.internalPlaceNote("record unreadable: " + err.Error())
|
||||
return
|
||||
}
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
uuid := rec.FSUUID
|
||||
st := &s.ipState
|
||||
st.mu.Lock()
|
||||
done := st.done == uuid
|
||||
st.mu.Unlock()
|
||||
|
||||
// What is at /mnt/belso now, if anything?
|
||||
mountedUUID, mounted := "", false
|
||||
if mounts, merr := s.hostReader().Mounts(); merr == nil {
|
||||
for _, m := range mounts {
|
||||
if path.Clean(m.MountPoint) == InternalPlaceWhere {
|
||||
mounted = true
|
||||
if u, okU := s.hostReader().ResolveUUID(m.Device); okU {
|
||||
mountedUUID = u
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if mounted && mountedUUID != "" && mountedUUID != uuid {
|
||||
s.internalPlaceNote("another filesystem (uuid " + mountedUUID + ") is mounted at " + InternalPlaceWhere + " — the internal place waits")
|
||||
return
|
||||
}
|
||||
// A Felhom unit for /mnt/belso that names another UUID (a household drive called „belso", unplugged): never
|
||||
// overwrite it — that would silently take the drive's name and mount point.
|
||||
if s.driveTargets != nil {
|
||||
if drives, derr := s.driveTargets.Known(ctx); derr == nil {
|
||||
for _, d := range drives {
|
||||
if path.Clean(d.MountPath) == InternalPlaceWhere && d.UUID != "" && d.UUID != uuid {
|
||||
s.internalPlaceNote("a drive unit for " + InternalPlaceWhere + " names another uuid (" + d.UUID + ") — the internal place waits")
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if mounted && mountedUUID == uuid {
|
||||
if !done {
|
||||
s.recordIntent(ctx, InternalPlaceWhere, "enrolled")
|
||||
st.mu.Lock()
|
||||
st.done, st.lastWhy = uuid, ""
|
||||
st.mu.Unlock()
|
||||
s.logger.Info("internal place: mounted and enrolled", "where", InternalPlaceWhere, "fs_uuid", uuid, "gib", rec.GiB)
|
||||
}
|
||||
return
|
||||
}
|
||||
if s.disks == nil {
|
||||
s.internalPlaceNote("disk management is not configured on this host")
|
||||
return
|
||||
}
|
||||
if s.resolveStorageDevice != nil {
|
||||
if _, rerr := s.resolveStorageDevice("uuid:" + uuid); rerr != nil {
|
||||
s.internalPlaceNote("its volume is not present (uuid " + uuid + ")")
|
||||
return
|
||||
}
|
||||
}
|
||||
start := time.Now()
|
||||
if err := s.disks.EnsureMount(ctx, storage.MountSpec{Name: InternalPlaceName, UUID: uuid, Where: InternalPlaceWhere, FSType: "ext4"}); err != nil {
|
||||
s.internalPlaceNote("mount failed: " + err.Error())
|
||||
return
|
||||
}
|
||||
s.recordIntent(ctx, InternalPlaceWhere, "enrolled")
|
||||
st.mu.Lock()
|
||||
st.done, st.lastWhy = uuid, ""
|
||||
st.mu.Unlock()
|
||||
s.logger.Info("internal place: mounted on the host", "where", InternalPlaceWhere, "fs_uuid", uuid, "gib", rec.GiB,
|
||||
"duration_ms", time.Since(start).Milliseconds())
|
||||
}
|
||||
|
||||
func (s *Server) internalPlaceNote(why string) {
|
||||
st := &s.ipState
|
||||
st.mu.Lock()
|
||||
changed := st.lastWhy != why
|
||||
st.lastWhy, st.done = why, ""
|
||||
st.mu.Unlock()
|
||||
if changed {
|
||||
s.logger.Warn("internal place: not ensured", "why", why)
|
||||
} else {
|
||||
s.logger.Debug("internal place: still not ensured", "why", why)
|
||||
}
|
||||
}
|
||||
|
||||
// InternalPlaceFileIn is the record's path inside a state dir.
|
||||
func InternalPlaceFileIn(stateDir string) string { return filepath.Join(stateDir, InternalPlaceFile) }
|
||||
@@ -0,0 +1,249 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// The internal data place (felhom.eu architecture/16 §5, agent v0.158.0): mounted like an enrolled drive, but never
|
||||
// at the cost of a household drive, never formatted, ejected or decommissioned, and the same disk as sys_drive.
|
||||
|
||||
const ipUUID = "0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c"
|
||||
const ipDev = "/dev/mapper/pve-felhom--belso"
|
||||
|
||||
// ipHost is a HostReader whose mount table the test changes, and whose by-uuid map it controls.
|
||||
type ipHost struct {
|
||||
mu sync.Mutex
|
||||
mounts []storage.Mount
|
||||
uuids map[string]string // device → fs-UUID
|
||||
}
|
||||
|
||||
func (h *ipHost) Mounts() ([]storage.Mount, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
return append([]storage.Mount(nil), h.mounts...), nil
|
||||
}
|
||||
func (h *ipHost) ResolveUUID(dev string) (string, bool) { u, ok := h.uuids[dev]; return u, ok }
|
||||
func (h *ipHost) DeviceExists(string) bool { return true }
|
||||
func (h *ipHost) Rotational(string) (bool, bool) { return false, false }
|
||||
func (h *ipHost) Removable(string) (bool, bool) { return false, false }
|
||||
func (h *ipHost) BlockSlaves(string) ([]string, bool) { return nil, false }
|
||||
|
||||
func newIPHost(extra ...storage.Mount) *ipHost {
|
||||
return &ipHost{
|
||||
// root on a whole-disk partition (as sysOnSDA): the system disk resolves, so a USB drive is user-data.
|
||||
mounts: append([]storage.Mount{{Device: "/dev/sda1", MountPoint: "/", FSType: "ext4"}}, extra...),
|
||||
uuids: map[string]string{ipDev: ipUUID, "/dev/sdb1": "99999999-aaaa-bbbb-cccc-dddddddddddd"},
|
||||
}
|
||||
}
|
||||
|
||||
type ipRig struct {
|
||||
srv *Server
|
||||
disks *fakeDiskOps
|
||||
intent *fakeIntent
|
||||
}
|
||||
|
||||
func newIPRig(t *testing.T, host *ipHost, known []storage.KnownTarget, record bool) ipRig {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, InternalPlaceFile)
|
||||
if record {
|
||||
if err := WriteInternalPlace(file, InternalPlaceRecord{FSUUID: ipUUID, GiB: 183, MadeBy: "installer"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
d, in := &fakeDiskOps{}, newFakeIntent()
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{}, Tokens: staticTokens{"A": 8200}, Disks: d, DiskGate: &fakeGate{},
|
||||
HostReader: host, Intent: in, DriveTargets: fakeKnownTargets{drives: known},
|
||||
InternalPlaceFile: file, Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.resolveStorageDevice = func(id string) (string, error) {
|
||||
if id == "uuid:"+ipUUID {
|
||||
return "/dev/dm-7", nil
|
||||
}
|
||||
return "", errors.New("absent")
|
||||
}
|
||||
return ipRig{srv, d, in}
|
||||
}
|
||||
|
||||
func TestInternalPlace_NoRecord_DoesNothing(t *testing.T) {
|
||||
r := newIPRig(t, newIPHost(), nil, false)
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 0 {
|
||||
t.Fatalf("no record: no mount may be made, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalPlace_Record_MountsByUUIDAtMntBelso(t *testing.T) {
|
||||
r := newIPRig(t, newIPHost(), nil, true)
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 1 {
|
||||
t.Fatalf("want exactly one EnsureMount, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
got := r.disks.mountCalls[0]
|
||||
if got.UUID != ipUUID || got.Where != InternalPlaceWhere || got.FSType != "ext4" || got.Name != InternalPlaceName {
|
||||
t.Fatalf("EnsureMount spec: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalPlace_AlreadyMounted_EnrolledOnceNoRemount(t *testing.T) {
|
||||
host := newIPHost(storage.Mount{Device: ipDev, MountPoint: InternalPlaceWhere, FSType: "ext4"})
|
||||
r := newIPRig(t, host, nil, true)
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 0 {
|
||||
t.Fatalf("already mounted: no EnsureMount, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
if r.intent.Get("uuid:"+ipUUID) != storage.IntentEnrolled {
|
||||
t.Fatalf("the internal place must be recorded enrolled")
|
||||
}
|
||||
}
|
||||
|
||||
// A household drive called „belso" keeps its mount point: the internal place never takes it.
|
||||
// Positive control: TestInternalPlace_Record_MountsByUUIDAtMntBelso (the same record, /mnt/belso free).
|
||||
func TestInternalPlace_NeverTakesAHouseholdDriveNamedBelso(t *testing.T) {
|
||||
mounted := newIPHost(storage.Mount{Device: "/dev/sdb1", MountPoint: InternalPlaceWhere, FSType: "ext4"})
|
||||
r := newIPRig(t, mounted, nil, true)
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 0 || r.intent.Get("uuid:"+ipUUID) != "" {
|
||||
t.Fatalf("a foreign filesystem at /mnt/belso: nothing may be mounted or recorded, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
unplugged := []storage.KnownTarget{{Name: "belso", UUID: "99999999-aaaa-bbbb-cccc-dddddddddddd", MountPath: InternalPlaceWhere, MountBacked: true}}
|
||||
r2 := newIPRig(t, newIPHost(), unplugged, true)
|
||||
r2.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r2.disks.mountCalls) != 0 {
|
||||
t.Fatalf("an unplugged household drive's unit for /mnt/belso must not be overwritten, got %+v", r2.disks.mountCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalPlace_VolumeAbsent_NoMount(t *testing.T) {
|
||||
r := newIPRig(t, newIPHost(), nil, true)
|
||||
r.srv.resolveStorageDevice = func(string) (string, error) { return "", errors.New("absent") }
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 0 {
|
||||
t.Fatalf("absent volume: no mount, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
}
|
||||
|
||||
// The household cannot eject or decommission it: its device is a device-mapper volume on the system disk.
|
||||
// Positive control: the same calls for a whole-disk USB drive (/dev/sdb1) are not refused by role.
|
||||
func TestInternalPlace_EjectAndDecommissionRefused(t *testing.T) {
|
||||
host := newIPHost(
|
||||
storage.Mount{Device: ipDev, MountPoint: InternalPlaceWhere, FSType: "ext4"},
|
||||
storage.Mount{Device: "/dev/sdb1", MountPoint: "/mnt/hdd_1", FSType: "ext4"},
|
||||
)
|
||||
r := newIPRig(t, host, nil, true)
|
||||
h := r.srv.Handler()
|
||||
for _, route := range []string{"/disks/eject", "/disks/decommission"} {
|
||||
if w := do(t, h, "POST", route, "A", `{"where":"/mnt/belso"}`); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("%s /mnt/belso: want 403, got %d %s", route, w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
if w := do(t, h, "POST", "/disks/eject", "A", `{"where":"/mnt/hdd_1"}`); w.Code == http.StatusForbidden {
|
||||
t.Fatalf("positive control: a USB drive's eject must not be refused by role, got %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
for _, u := range r.disks.unmountCalls {
|
||||
if u == InternalPlaceWhere {
|
||||
t.Fatalf("the internal place must never be unmounted, got %+v", r.disks.unmountCalls)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The consequence `16` §5 needs: a copy between the internal place and sys_drive is the SAME disk (build A then never
|
||||
// counts it), and the place is reported as kind internal.
|
||||
type ipIdentHost struct{ fakeIdentHost }
|
||||
|
||||
func (h *ipIdentHost) KernelName(dev string) (string, bool) {
|
||||
if dev == ipDev {
|
||||
return "dm-7", true
|
||||
}
|
||||
return h.fakeIdentHost.KernelName(dev)
|
||||
}
|
||||
func (h *ipIdentHost) Slaves(n string) []string {
|
||||
if n == "dm-7" {
|
||||
return []string{"dm-3"} // felhom-belso → the thin pool
|
||||
}
|
||||
if n == "dm-3" {
|
||||
return []string{"sda3"} // the pool → the system disk's LVM partition
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestInternalPlace_IsTheSystemDiskAndKindInternal(t *testing.T) {
|
||||
ih := &ipIdentHost{fakeIdentHost{
|
||||
virt: "none",
|
||||
parent: map[string]string{"sda3": "sda"},
|
||||
props: map[string]map[string]string{"sda": {"ID_BUS": "ata", "ID_MODEL": "SanDisk", "ID_SERIAL_SHORT": "1"}},
|
||||
mounts: []storage.Mount{
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"},
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/mnt/felhom-drives", FSType: "ext4"},
|
||||
{Device: ipDev, MountPoint: InternalPlaceWhere, FSType: "ext4"},
|
||||
{Device: ipDev, MountPoint: "/mnt/felhom-drives/belso", FSType: "ext4"},
|
||||
},
|
||||
}}
|
||||
fg := &fakeGuestsCfg{mounts: map[int]map[string]string{8200: {
|
||||
"mp0": "local-lvm:vm-8200-disk-1,mp=/var/lib/felhom,backup=1,size=100G",
|
||||
"mp8": "/mnt/felhom-drives,mp=/mnt/felhom-drives",
|
||||
}}}
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, InternalPlaceFile)
|
||||
if err := WriteInternalPlace(file, InternalPlaceRecord{FSUUID: ipUUID}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: fg, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{provenSSD}}, Tokens: staticTokens{"A": 8200},
|
||||
HostReader: fakeHostReader{mounts: ih.mounts}, Identity: &storage.IdentityResolver{Host: ih},
|
||||
InternalPlaceFile: file, Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
got, code := sameDisk(t, srv.Handler(), "/mnt/felhom-drives/belso/felhom-data/jellyfin", "/mnt/sys_drive")
|
||||
if code != http.StatusOK || got.Different || !got.Proven || got.Reason != "same_disk" {
|
||||
t.Fatalf("internal place vs sys_drive: want same disk, proven; got %d %+v", code, got)
|
||||
}
|
||||
if got.A.Kind != storage.PlaceKindInternal {
|
||||
t.Fatalf("kind: want %q, got %q", storage.PlaceKindInternal, got.A.Kind)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalPlaceRecord_RefusesANonUUID(t *testing.T) {
|
||||
file := filepath.Join(t.TempDir(), InternalPlaceFile)
|
||||
if err := WriteInternalPlace(file, InternalPlaceRecord{FSUUID: "../../etc"}); err == nil {
|
||||
t.Fatal("a non-UUID must not be written")
|
||||
}
|
||||
}
|
||||
|
||||
// The root helper's own tests (configs/test_felhom_internal_place.py — the ruled size table, never formatting what
|
||||
// holds anything) run with the Go suite.
|
||||
func TestInternalPlaceHelperPythonSuite(t *testing.T) {
|
||||
py, err := exec.LookPath("python3")
|
||||
if err != nil {
|
||||
t.Skip("python3 not found")
|
||||
}
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
script := filepath.Join(filepath.Dir(file), "..", "..", "configs", "test_felhom_internal_place.py")
|
||||
out, err := exec.Command(py, script).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("helper suite failed: %v\n%s", err, out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"path"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// POST /disks/same-disk (v0.157.0) — felhom.eu architecture/16-storage-physical-identity.md §4.
|
||||
//
|
||||
// The controller asks whether two of ITS paths sit on the same physical disk; the agent answers from its
|
||||
// own /sys + udev read, never from the caller's claim (`16` §10). A guest path is mapped only to a place
|
||||
// the agent itself made:
|
||||
// - /mnt/felhom-drives/<name>[/…] → the host mount at exactly /mnt/felhom-drives/<name> (the drive's
|
||||
// felhom-data bind, or a network share's mount). A path whose <name> is not mounted there (an absent
|
||||
// drive: the bare directory on the root disk) is UNMAPPED — never "the system disk".
|
||||
// - /mnt/sys_drive, /var/lib/felhom, /var/lib/docker [/…] → the guest's data volume (mp0, mp=/var/lib/felhom;
|
||||
// or a legacy mp at that exact path) → the PVE storage holding it → that storage's identity.
|
||||
// - anything else → unmapped.
|
||||
// Answer: {different, proven, reason, a, b}. Act on `proven` first: different+proven = a counted second
|
||||
// copy; same_disk+proven = not a second copy; proven=false = unknown (reason vm|network|usb_bridge|
|
||||
// weak_wwn|no_serial|unmapped). Pinned by samedisk_test.go.
|
||||
|
||||
type sameDiskRequest struct {
|
||||
VMID int `json:"vmid,omitempty"`
|
||||
A string `json:"a"`
|
||||
B string `json:"b"`
|
||||
}
|
||||
|
||||
// SamePlace is one side of the answer.
|
||||
type SamePlace struct {
|
||||
Path string `json:"path"`
|
||||
Mapped bool `json:"mapped"`
|
||||
Kind string `json:"kind,omitempty"`
|
||||
PhysicalDisks []string `json:"physical_disks,omitempty"`
|
||||
Identity string `json:"identity,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// SameDiskResponse is the wire answer.
|
||||
type SameDiskResponse struct {
|
||||
Different bool `json:"different"`
|
||||
Proven bool `json:"proven"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
A SamePlace `json:"a"`
|
||||
B SamePlace `json:"b"`
|
||||
}
|
||||
|
||||
// sysDataPaths are the guest paths that live on the guest's one data volume (golden v3: mp0 at
|
||||
// /var/lib/felhom, its docker/ and sys_drive/ folders bound in the guest — 07 §7.5.1).
|
||||
var sysDataPaths = []string{"/var/lib/felhom", "/mnt/sys_drive", "/var/lib/docker"}
|
||||
|
||||
func (s *Server) handleSameDisk(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
if s.ident == nil {
|
||||
writeErr(w, http.StatusServiceUnavailable, "disk identity not configured on this host")
|
||||
return
|
||||
}
|
||||
var req sameDiskRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||
return
|
||||
}
|
||||
if !path.IsAbs(req.A) || !path.IsAbs(req.B) {
|
||||
writeErr(w, http.StatusBadRequest, "a and b must be absolute guest paths")
|
||||
return
|
||||
}
|
||||
a := s.mapGuestPlace(r.Context(), vmid, req.A)
|
||||
b := s.mapGuestPlace(r.Context(), vmid, req.B)
|
||||
resp := SameDiskResponse{A: a.view, B: b.view}
|
||||
resp.Different, resp.Proven, resp.Reason = storage.Relation(a.id, b.id)
|
||||
s.logger.Info("local-api: same-disk", "vmid", vmid, "a", req.A, "b", req.B,
|
||||
"different", resp.Different, "proven", resp.Proven, "reason", resp.Reason)
|
||||
writeOK(w, resp)
|
||||
}
|
||||
|
||||
type mappedPlace struct {
|
||||
view SamePlace
|
||||
id storage.PlaceIdentity
|
||||
}
|
||||
|
||||
func (s *Server) mapGuestPlace(ctx context.Context, vmid int, guestPath string) mappedPlace {
|
||||
p := path.Clean(guestPath)
|
||||
out := mappedPlace{view: SamePlace{Path: p}}
|
||||
set := func(kind string, id storage.PlaceIdentity) mappedPlace {
|
||||
if !id.Computed() {
|
||||
return out
|
||||
}
|
||||
out.id = id
|
||||
out.view.Mapped, out.view.Kind = true, kind
|
||||
out.view.PhysicalDisks, out.view.Identity, out.view.Reason = id.PhysicalDisks, id.Identity, id.Reason
|
||||
return out
|
||||
}
|
||||
if strings.HasPrefix(p, StableParentDir+"/") {
|
||||
name := strings.SplitN(strings.TrimPrefix(p, StableParentDir+"/"), "/", 2)[0]
|
||||
if id, kind, ok := s.ownMountIdentity(StableParentDir + "/" + name); ok {
|
||||
return set(kind, id)
|
||||
}
|
||||
return out
|
||||
}
|
||||
for _, root := range sysDataPaths {
|
||||
if p != root && !strings.HasPrefix(p, root+"/") {
|
||||
continue
|
||||
}
|
||||
id := s.guestDataVolumeIdentity(ctx, vmid, root)
|
||||
return set(storage.PlaceKindSystem, id)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ownMountIdentity judges a HOST path only when it is ITSELF a mountpoint (an enrolled drive's raw mount
|
||||
// or its felhom-data bind, a network share). A path that is not its own mount — an absent drive, whose
|
||||
// directory now sits on the root disk — is not computed, never root's identity (the R-118 trap).
|
||||
func (s *Server) ownMountIdentity(hostPath string) (storage.PlaceIdentity, string, bool) {
|
||||
if s.ident == nil || hostPath == "" {
|
||||
return storage.PlaceIdentity{}, "", false
|
||||
}
|
||||
mounts, err := s.hostReader().Mounts()
|
||||
if err != nil {
|
||||
return storage.PlaceIdentity{}, "", false
|
||||
}
|
||||
m, ok := storage.MountFor(mounts, hostPath)
|
||||
if !ok || path.Clean(m.MountPoint) != path.Clean(hostPath) {
|
||||
return storage.PlaceIdentity{}, "", false
|
||||
}
|
||||
id := s.ident.HostPath(hostPath)
|
||||
if !id.Computed() {
|
||||
return id, "", false
|
||||
}
|
||||
kind := storage.PlaceKindDisk
|
||||
switch {
|
||||
case s.isInternalPlace(hostPath): // v0.158.0 `16` §5: the one-disk box's data place, on the system disk
|
||||
kind = storage.PlaceKindInternal
|
||||
case id.Reason == storage.ReasonNetwork:
|
||||
kind = storage.PlaceKindNetwork
|
||||
case strings.HasPrefix(m.Device, "/dev/") && !strings.HasPrefix(m.Device, "/dev/mapper/") && isPartitionPath(m.Device):
|
||||
kind = storage.PlaceKindPartition
|
||||
}
|
||||
return id, kind, true
|
||||
}
|
||||
|
||||
var rePartitionName = regexp.MustCompile(`^(nvme[0-9]+n[0-9]+p[0-9]+|(sd|vd|hd)[a-z]+[0-9]+)$`)
|
||||
|
||||
func isPartitionPath(dev string) bool { return rePartitionName.MatchString(path.Base(dev)) }
|
||||
|
||||
// guestDataVolumeIdentity finds the guest mountpoint that holds root (mp=/var/lib/felhom on golden v3, or a
|
||||
// legacy mp at that exact path) and returns the identity of the PVE storage that holds its volume.
|
||||
func (s *Server) guestDataVolumeIdentity(ctx context.Context, vmid int, root string) storage.PlaceIdentity {
|
||||
if s.guests == nil || s.storage == nil {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
cfg, err := s.guests.GuestConfig(ctx, vmid)
|
||||
if err != nil {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
storeID, fallback := "", ""
|
||||
for _, spec := range cfg.MountPoints() {
|
||||
st, mp, _ := parseMount(spec)
|
||||
if strings.HasPrefix(st, "/") {
|
||||
continue // a host bind (mp8, mp9), not a volume
|
||||
}
|
||||
switch mp {
|
||||
case root:
|
||||
storeID = st // a legacy mp at exactly this path wins
|
||||
case "/var/lib/felhom":
|
||||
fallback = st // golden v3: the one data volume
|
||||
}
|
||||
}
|
||||
if storeID == "" {
|
||||
storeID = fallback
|
||||
}
|
||||
if storeID == "" {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
targets, err := s.storage.Observe(ctx)
|
||||
if err != nil {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
for _, t := range targets {
|
||||
if t.Name == storeID {
|
||||
return identityOfTarget(t)
|
||||
}
|
||||
}
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
|
||||
func identityOfTarget(t hub.StorageTarget) storage.PlaceIdentity {
|
||||
if t.Identity == "" {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
disks := t.PhysicalDisks
|
||||
if disks == nil {
|
||||
disks = []string{}
|
||||
}
|
||||
return storage.PlaceIdentity{PhysicalDisks: disks, Identity: t.Identity, Reason: t.IdentityReason}
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// POST /disks/same-disk (`16` §4) — the controller's Tier-2 question, answered from the agent's own read.
|
||||
|
||||
type fakeIdentHost struct {
|
||||
mounts []storage.Mount
|
||||
parent map[string]string
|
||||
props map[string]map[string]string
|
||||
virt string
|
||||
}
|
||||
|
||||
func (f *fakeIdentHost) Mounts() ([]storage.Mount, error) { return f.mounts, nil }
|
||||
func (f *fakeIdentHost) KernelName(dev string) (string, bool) {
|
||||
if len(dev) > 5 && dev[:5] == "/dev/" {
|
||||
return dev[5:], true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
func (f *fakeIdentHost) Parent(n string) (string, bool) { p, ok := f.parent[n]; return p, ok }
|
||||
func (f *fakeIdentHost) Slaves(string) []string { return nil }
|
||||
func (f *fakeIdentHost) LoopBackingFile(string) (string, bool) { return "", false }
|
||||
func (f *fakeIdentHost) UdevProps(n string) map[string]string { return f.props[n] }
|
||||
func (f *fakeIdentHost) Virt() string { return f.virt }
|
||||
|
||||
// sameDiskRig: guest 8200 (token A) with mp0 on local-lvm (the system SSD, proven) + the given mounts.
|
||||
func sameDiskRig(t *testing.T, host *fakeIdentHost, localLVM hub.StorageTarget) http.Handler {
|
||||
t.Helper()
|
||||
fg := &fakeGuestsCfg{mounts: map[int]map[string]string{8200: {
|
||||
"mp0": "local-lvm:vm-8200-disk-1,mp=/var/lib/felhom,backup=1,size=70G",
|
||||
"mp8": "/mnt/felhom-drives,mp=/mnt/felhom-drives",
|
||||
}}}
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: fg, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{localLVM}}, Tokens: staticTokens{"A": 8200},
|
||||
HostReader: fakeHostReader{mounts: host.mounts},
|
||||
Identity: &storage.IdentityResolver{Host: host},
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
return srv.Handler()
|
||||
}
|
||||
|
||||
var provenSSD = hub.StorageTarget{Name: "local-lvm", Type: hub.StorageTypeLVMThin, Kind: "system",
|
||||
PhysicalDisks: []string{"serial:ata:SanDisk:1"}, Identity: "proven"}
|
||||
|
||||
func baseHost() *fakeIdentHost {
|
||||
return &fakeIdentHost{
|
||||
virt: "none",
|
||||
parent: map[string]string{"sdc1": "sdc", "sdc2": "sdc"},
|
||||
props: map[string]map[string]string{
|
||||
"sdb": {"ID_BUS": "ata", "ID_MODEL": "TOSHIBA", "ID_SERIAL_SHORT": "65NOP3HDT"},
|
||||
"sdc": {"ID_BUS": "usb", "ID_MODEL": "Portable", "ID_SERIAL_SHORT": "NAAB7C21"},
|
||||
},
|
||||
mounts: []storage.Mount{
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"},
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/mnt/felhom-drives", FSType: "ext4"},
|
||||
{Device: "/dev/sdb", MountPoint: "/mnt/felhom-drives/hdd_1", FSType: "ext4"},
|
||||
{Device: "/dev/sdc1", MountPoint: "/mnt/felhom-drives/a", FSType: "ext4"},
|
||||
{Device: "/dev/sdc2", MountPoint: "/mnt/felhom-drives/b", FSType: "ext4"},
|
||||
{Device: "//nas/films", MountPoint: "/mnt/felhom-drives/nas", FSType: "cifs"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func sameDisk(t *testing.T, h http.Handler, a, b string) (SameDiskResponse, int) {
|
||||
t.Helper()
|
||||
body, _ := json.Marshal(map[string]string{"a": a, "b": b})
|
||||
w := do(t, h, "POST", "/disks/same-disk", "A", string(body))
|
||||
var env struct {
|
||||
Data SameDiskResponse `json:"data"`
|
||||
}
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &env)
|
||||
return env.Data, w.Code
|
||||
}
|
||||
|
||||
func TestSameDisk_DriveVsSystemData_DifferentProven(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
got, code := sameDisk(t, h, "/mnt/felhom-drives/hdd_1/felhom-data/paperless", "/mnt/sys_drive")
|
||||
if code != http.StatusOK || !got.Different || !got.Proven || got.Reason != "" {
|
||||
t.Fatalf("drive vs sys_drive: want different+proven, got %d %+v", code, got)
|
||||
}
|
||||
if got.B.Kind != "system" || got.A.Kind != "disk" {
|
||||
t.Fatalf("kinds: got a=%q b=%q", got.A.Kind, got.B.Kind)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_TwoPartitionsOfOneDisk_SameProven(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
got, _ := sameDisk(t, h, "/mnt/felhom-drives/a", "/mnt/felhom-drives/b/x")
|
||||
if got.Different || !got.Proven || got.Reason != "same_disk" || got.A.Kind != "partition" {
|
||||
t.Fatalf("two partitions: want same+proven, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_InsideAVM_NotProven(t *testing.T) {
|
||||
host := baseHost()
|
||||
host.virt = "kvm"
|
||||
h := sameDiskRig(t, host, hub.StorageTarget{Name: "local-lvm", Identity: "unknown", IdentityReason: "vm",
|
||||
PhysicalDisks: []string{"serial:scsi:QEMU_HARDDISK:drive-scsi0"}})
|
||||
got, _ := sameDisk(t, h, "/mnt/felhom-drives/hdd_1", "/mnt/sys_drive")
|
||||
if got.Proven || got.Reason != "vm" {
|
||||
t.Fatalf("inside a VM: want not proven (vm), got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_NetworkAndUnmapped(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
if got, _ := sameDisk(t, h, "/mnt/felhom-drives/nas", "/mnt/sys_drive"); got.Proven || got.Reason != "network" {
|
||||
t.Fatalf("a NAS: want not proven (network), got %+v", got)
|
||||
}
|
||||
// An ABSENT drive: its directory sits on the root disk — must be unmapped, never "the system disk".
|
||||
if got, _ := sameDisk(t, h, "/mnt/felhom-drives/gone", "/mnt/sys_drive"); got.Proven || got.Reason != "unmapped" || got.A.Mapped {
|
||||
t.Fatalf("an absent drive: want unmapped, got %+v", got)
|
||||
}
|
||||
if got, _ := sameDisk(t, h, "/opt/elsewhere", "/mnt/sys_drive"); got.Proven || got.Reason != "unmapped" {
|
||||
t.Fatalf("an unknown guest path: want unmapped, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_ScopedAndValidated(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
if w := do(t, h, "POST", "/disks/same-disk", "A", `{"vmid":9300,"a":"/mnt/sys_drive","b":"/mnt/sys_drive"}`); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("another guest's vmid: got %d want 403", w.Code)
|
||||
}
|
||||
if w := do(t, h, "POST", "/disks/same-disk", "A", `{"a":"relative","b":"/mnt/sys_drive"}`); w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("a relative path: got %d want 400", w.Code)
|
||||
}
|
||||
if w := do(t, h, "POST", "/disks/same-disk", "", `{"a":"/x","b":"/y"}`); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: got %d want 401", w.Code)
|
||||
}
|
||||
}
|
||||
+62
-42
@@ -208,6 +208,9 @@ type Options struct {
|
||||
// Intent records drive enroll/eject intent for the self-heal watchdog (slice 10 P3). OPTIONAL —
|
||||
// when nil, no intent is recorded (self-heal runs ungated).
|
||||
Intent IntentRecorder
|
||||
// InternalPlaceFile (v0.158.0, felhom.eu `16` §5) is the path of the internal data place record
|
||||
// (InternalPlaceRecord). OPTIONAL — "" means this host has no internal place and EnsureInternalPlace is a no-op.
|
||||
InternalPlaceFile string
|
||||
// GuestBinds persists which user-data drives (by durable-id) are enrolled into each guest, so the
|
||||
// startup re-assert (ReassertGuestBinds) can restore a bind that a re-provision dropped (F9).
|
||||
// OPTIONAL — when nil, guest binds are not recorded and the startup re-assert is a no-op.
|
||||
@@ -220,6 +223,10 @@ type Options struct {
|
||||
// ROLE (it backs SystemDisks for the eject role-gate + the /disks role hints). OPTIONAL — when nil
|
||||
// it defaults to the production *storage.ProcHostReader. Injectable so the role-gate is testable.
|
||||
HostReader storage.HostReader
|
||||
// Identity computes physical-disk identity (felhom.eu architecture/16-storage-physical-identity.md §3)
|
||||
// for the /disks rows and POST /disks/same-disk. OPTIONAL — when nil, /disks carries no identity and
|
||||
// same-disk answers 503 "not configured" (the controller then keeps its old st_dev rule).
|
||||
Identity *storage.IdentityResolver
|
||||
// HostMetrics serves GET /host/metrics (slice 9) — host-wide health (cpu%/mem/load/uptime/
|
||||
// cpu-temp) + per-storage capacity, host-wide and token-authed (one-customer-per-host). When
|
||||
// nil the endpoint reports "not configured" (host still reports/reconciles).
|
||||
@@ -312,11 +319,14 @@ type Server struct {
|
||||
// identity blob from the hub, unseal it with the customer's recovery code, return ONLY the
|
||||
// offsite repository password. OPTIONAL — nil (no hub client configured) makes
|
||||
// POST /escrow/recover-offsite-password answer 503 rather than pretending.
|
||||
escrowRecovery EscrowRecoverer
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||
escrowRecovery EscrowRecoverer
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
internalPlaceFile string // v0.158.0 `16` §5 (optional)
|
||||
ipState internalPlaceState // EnsureInternalPlace's memo
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||
partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional)
|
||||
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
|
||||
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
|
||||
guestPower map[int]guestPowerState
|
||||
@@ -324,7 +334,8 @@ type Server struct {
|
||||
// guestPowerSweeps counts completed guest-power sweeps, for the liveness observable. Touched only
|
||||
// from GuestPowerTick, which the ticker calls serially.
|
||||
guestPowerSweeps int
|
||||
host storage.HostReader // role classification source (optional; defaults to ProcHostReader)
|
||||
host storage.HostReader // role classification source (optional; defaults to ProcHostReader)
|
||||
ident *storage.IdentityResolver // `16` §3 physical-disk identity (optional)
|
||||
|
||||
hostMetrics HostMetricsProvider // slice 9 (optional)
|
||||
hostID string // slice 10B: for the data-bearing-format pending-op hint
|
||||
@@ -394,6 +405,10 @@ type Server struct {
|
||||
// detached pipeline runs through (tests inject; production defaults set in NewServer).
|
||||
netVerifyMu sync.Mutex
|
||||
netVerifyCur *netVerifyJob
|
||||
|
||||
// R-998: POST /host/poweroff (nil = not configured → 503); one guard-and-mark decision at a time.
|
||||
powerOff *HostPowerOffConfig
|
||||
powerOffMu sync.Mutex
|
||||
// netTrigger performs the mount-waking directory read through the automount path (Q1).
|
||||
netTrigger func(where string) error
|
||||
// netMounted judges mount success from /proc/mounts — the §8 truth source (never readability).
|
||||
@@ -445,42 +460,44 @@ func NewServer(o Options) (*Server, error) {
|
||||
cadence = defaultBackupCadence
|
||||
}
|
||||
s := &Server{
|
||||
addr: o.ListenAddr,
|
||||
cert: o.Cert,
|
||||
guests: o.Guests,
|
||||
backups: o.Backups,
|
||||
store: o.Store,
|
||||
storage: o.Storage,
|
||||
driveTargets: o.DriveTargets,
|
||||
smart: o.Smart,
|
||||
tokens: o.Tokens,
|
||||
cadence: cadence,
|
||||
logger: o.Logger,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
disks: o.Disks,
|
||||
diskGate: o.DiskGate,
|
||||
guestList: o.Guests2,
|
||||
guestAttach: o.GuestAttach,
|
||||
mem: o.Memory,
|
||||
netStorage: o.NetStorage,
|
||||
privileged: o.Privileged,
|
||||
configPath: o.ConfigPath,
|
||||
stateDir: o.StateDir,
|
||||
netMountRoot: storage.NetworkMountRoot,
|
||||
smbCredsDir: o.SmbCredsDir,
|
||||
escrowStagePath: o.EscrowStagePath,
|
||||
escrowRecovery: o.EscrowRecovery,
|
||||
intent: o.Intent,
|
||||
guestBinds: o.GuestBinds,
|
||||
formatJobs: o.FormatJobs,
|
||||
staleLock: o.StaleLock,
|
||||
host: o.HostReader,
|
||||
hostMetrics: o.HostMetrics,
|
||||
hostID: o.HostID,
|
||||
agentVersion: o.AgentVersion,
|
||||
logRing: o.LogRing,
|
||||
jobs: map[backupJobKey]*backupJob{},
|
||||
swapInFlight: map[int]bool{},
|
||||
addr: o.ListenAddr,
|
||||
cert: o.Cert,
|
||||
guests: o.Guests,
|
||||
backups: o.Backups,
|
||||
store: o.Store,
|
||||
storage: o.Storage,
|
||||
driveTargets: o.DriveTargets,
|
||||
smart: o.Smart,
|
||||
tokens: o.Tokens,
|
||||
cadence: cadence,
|
||||
logger: o.Logger,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
disks: o.Disks,
|
||||
diskGate: o.DiskGate,
|
||||
guestList: o.Guests2,
|
||||
guestAttach: o.GuestAttach,
|
||||
mem: o.Memory,
|
||||
netStorage: o.NetStorage,
|
||||
privileged: o.Privileged,
|
||||
configPath: o.ConfigPath,
|
||||
stateDir: o.StateDir,
|
||||
netMountRoot: storage.NetworkMountRoot,
|
||||
smbCredsDir: o.SmbCredsDir,
|
||||
escrowStagePath: o.EscrowStagePath,
|
||||
escrowRecovery: o.EscrowRecovery,
|
||||
intent: o.Intent,
|
||||
internalPlaceFile: o.InternalPlaceFile,
|
||||
guestBinds: o.GuestBinds,
|
||||
formatJobs: o.FormatJobs,
|
||||
staleLock: o.StaleLock,
|
||||
host: o.HostReader,
|
||||
ident: o.Identity,
|
||||
hostMetrics: o.HostMetrics,
|
||||
hostID: o.HostID,
|
||||
agentVersion: o.AgentVersion,
|
||||
logRing: o.LogRing,
|
||||
jobs: map[backupJobKey]*backupJob{},
|
||||
swapInFlight: map[int]bool{},
|
||||
}
|
||||
// R-82 tier resolution. Options.BackupTiers is authoritative when supplied; otherwise ONE tier
|
||||
// is synthesized from Backups + BackupCadence — the pre-R-82 shape, so every existing caller
|
||||
@@ -541,6 +558,7 @@ func (s *Server) Handler() http.Handler {
|
||||
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
||||
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
|
||||
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
|
||||
mux.HandleFunc("POST /disks/same-disk", s.withGuest(s.handleSameDisk)) // `16` §4 (v0.157.0)
|
||||
mux.HandleFunc("POST /disks/assign", s.withGuest(s.handleDiskAssign))
|
||||
mux.HandleFunc("POST /disks/eject", s.withGuest(s.handleDiskEject))
|
||||
mux.HandleFunc("POST /disks/decommission", s.withGuest(s.handleDiskDecommission))
|
||||
@@ -550,6 +568,8 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
|
||||
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
|
||||
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
|
||||
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
|
||||
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
|
||||
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
|
||||
// (live cgroup apply, no reboot). Self-scoped; the agent enforces every bound fresh per request.
|
||||
mux.HandleFunc("GET /guest/memory", s.withGuest(s.handleGuestMemory))
|
||||
|
||||
@@ -4,7 +4,11 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
@@ -142,6 +146,10 @@ type staleLockController struct {
|
||||
runner proxmox.Runner
|
||||
pool string // ownership registry: only members of this PVE pool are ever scanned (A1)
|
||||
logger *slog.Logger // the one success-path scan-summary line; nil = silent
|
||||
// procDir is where taskAlive reads /proc/<pid>/stat (R-1008); "" = /proc. Tests point it at a fixture.
|
||||
procDir string
|
||||
deadMu sync.Mutex
|
||||
deadSeen map[string]bool // UPIDs of dead tasks already logged (once per task)
|
||||
}
|
||||
|
||||
// staleLockAPI is the subset of *proxmox.Client the controller uses (kept narrow for clarity/testing).
|
||||
@@ -209,19 +217,100 @@ func (c *staleLockController) Lock(ctx context.Context, vmid int) (string, bool,
|
||||
return cfg.Lock(), cfg.OnBoot(), nil
|
||||
}
|
||||
|
||||
// BackupRunning is THE answer to „is a vzdump in flight for vmid" for all four callers (startup stale-lock
|
||||
// recovery, the guest-power watchdog, the household power-off guard, the controller supervisor): every one of
|
||||
// them reaches this method through the StaleLockController interface. R-1008: Proxmox keeps a task in
|
||||
// `tasks?source=active` after a power cut — nobody is left to end it (measured on demo-hp 2026-10-14: still
|
||||
// listed RUNNING ~30–60 s after the boot) — so a listed task whose worker process is gone is dead, not in flight.
|
||||
func (c *staleLockController) BackupRunning(ctx context.Context, vmid int) (bool, error) {
|
||||
tasks, err := c.px.ListRunningTasks(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
proc := c.procDir
|
||||
if proc == "" {
|
||||
proc = "/proc"
|
||||
}
|
||||
running, dead := vzdumpInFlight(tasks, vmid, proc)
|
||||
for _, t := range dead {
|
||||
c.logDeadOnce(t, vmid)
|
||||
}
|
||||
return running, nil
|
||||
}
|
||||
|
||||
// vzdumpInFlight decides, from the node's active-task list, whether a vzdump for vmid is in flight: a matching
|
||||
// task is in flight while its worker process lives (taskAlive). A task whose worker is gone is dead (returned in dead,
|
||||
// for the log). FAIL SAFE on doubt — the old answer: a UPID that does not parse, or a /proc entry that cannot be read,
|
||||
// counts as in flight. Pure over procDir; pinned by stalelock_inflight_test.go.
|
||||
func vzdumpInFlight(tasks []proxmox.TaskStatus, vmid int, procDir string) (bool, []proxmox.TaskStatus) {
|
||||
id := strconv.Itoa(vmid)
|
||||
var dead []proxmox.TaskStatus
|
||||
for _, t := range tasks {
|
||||
// A single-guest vzdump task carries the vmid in its ID field.
|
||||
if t.Type == "vzdump" && t.ID == id {
|
||||
if t.Type != "vzdump" || t.ID != id {
|
||||
continue
|
||||
}
|
||||
alive, known := taskAlive(procDir, t.UPID)
|
||||
if !known || alive {
|
||||
return true, nil
|
||||
}
|
||||
dead = append(dead, t)
|
||||
}
|
||||
return false, nil
|
||||
return false, dead
|
||||
}
|
||||
|
||||
// taskAlive is Proxmox's own liveness rule for a task (PVE::ProcFSTools::check_process_running): the UPID names the
|
||||
// worker's pid and its start time in clock ticks since boot (pstart); the task lives while /proc/<pid>/stat exists with
|
||||
// that same start time (field 22). No wall clock is involved — a clock step after the boot (an NTP correction after a
|
||||
// power cut) cannot make a live task look dead (the 2026-10-14 security review of the first, btime-based version).
|
||||
// known=false: the UPID does not parse, or /proc/<pid>/stat exists but cannot be read or parsed.
|
||||
func taskAlive(procDir, upid string) (alive, known bool) {
|
||||
u, err := proxmox.ParseUPID(upid)
|
||||
if err != nil || u.PID == 0 {
|
||||
return false, false
|
||||
}
|
||||
raw, err := os.ReadFile(filepath.Join(procDir, strconv.FormatUint(u.PID, 10), "stat"))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
// „no such process" is only an answer if this agent can see other users' processes at all (a /proc mounted
|
||||
// hidepid=, or a unit with ProtectProc=, hides root's workers — every task would then look dead). PID 1 is
|
||||
// root's; not seeing it means not knowing.
|
||||
if _, err1 := os.Stat(filepath.Join(procDir, "1", "stat")); err1 != nil {
|
||||
return false, false
|
||||
}
|
||||
return false, true // no such process in this boot: the worker is gone
|
||||
}
|
||||
return false, false
|
||||
}
|
||||
st := string(raw)
|
||||
i := strings.LastIndexByte(st, ')') // the comm field may hold spaces and parentheses
|
||||
if i < 0 {
|
||||
return false, false
|
||||
}
|
||||
f := strings.Fields(st[i+1:]) // f[0] is field 3 (state), so field 22 (starttime) is f[19]
|
||||
if len(f) < 20 {
|
||||
return false, false
|
||||
}
|
||||
start, err := strconv.ParseUint(f[19], 10, 64)
|
||||
if err != nil {
|
||||
return false, false
|
||||
}
|
||||
// Proxmox's rule exactly: the start tick matches (a pstart of 0 is not checked) and the process is not a zombie.
|
||||
return (u.PStart == 0 || start == u.PStart) && f[0] != "Z", true
|
||||
}
|
||||
|
||||
func (c *staleLockController) logDeadOnce(t proxmox.TaskStatus, vmid int) {
|
||||
c.deadMu.Lock()
|
||||
defer c.deadMu.Unlock()
|
||||
if c.deadSeen == nil {
|
||||
c.deadSeen = map[string]bool{}
|
||||
}
|
||||
if c.deadSeen[t.UPID] || c.logger == nil {
|
||||
return
|
||||
}
|
||||
c.deadSeen[t.UPID] = true
|
||||
c.logger.Warn("backup-in-flight: a vzdump task is listed as active but its worker process is gone — it is dead, not running",
|
||||
"vmid", vmid, "upid", t.UPID)
|
||||
}
|
||||
|
||||
func (c *staleLockController) HasVzdumpSnapshot(ctx context.Context, vmid int) (bool, error) {
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1008: after a power cut Proxmox keeps the dead vzdump task in `tasks?source=active` (measured on demo-hp
|
||||
// 2026-10-14: still RUNNING ~30–60 s after the boot). A task whose worker process is gone is dead, not in flight.
|
||||
// Liveness is Proxmox's own rule (pid + pstart vs /proc/<pid>/stat field 22), never the wall clock: the first version
|
||||
// compared start time with btime and a clock step after the boot could make a live task look dead (security review
|
||||
// 2026-10-14). These tests pin the helper, the production controller, three callers end to end; the fourth (the
|
||||
// controller supervisor) by the structural pin at the bottom.
|
||||
|
||||
// A worker: pid 0x1E6EEC = 1994476, pstart 0x1B4A0D = 1788429 (UPID hex fields).
|
||||
const (
|
||||
testPID = 1994476
|
||||
testPStart = 1788429
|
||||
)
|
||||
|
||||
func vz(vmid string) proxmox.TaskStatus {
|
||||
return proxmox.TaskStatus{UPID: "UPID:demo-hp:001E6EEC:001B4A0D:6ACF4071:vzdump:" + vmid + ":felhom-agent@pve!agent:",
|
||||
Type: "vzdump", ID: vmid, Status: "RUNNING", StartTime: 1791967345}
|
||||
}
|
||||
|
||||
// procFixture writes <dir>/<pid>/stat with the given start ticks (comm with a space and a parenthesis, as real ones can).
|
||||
func procFixture(t *testing.T, pid int, start int) string {
|
||||
return procFixtureState(t, pid, start, "S")
|
||||
}
|
||||
|
||||
func procFixtureState(t *testing.T, pid int, start int, state string) string {
|
||||
t.Helper()
|
||||
d := t.TempDir()
|
||||
// PID 1 is always visible on a real host; procFixtureHidden leaves it out
|
||||
if err := os.MkdirAll(filepath.Join(d, "1"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(d, "1", "stat"), []byte("1 (systemd) S 0 1 1 0 -1 0 0 0 0 0 0 0 0 0 20 0 1 0 2 0 0\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pid > 0 {
|
||||
p := filepath.Join(d, itoaT(pid))
|
||||
if err := os.MkdirAll(p, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f := []string{itoaT(pid), "(task UPID:x (vzdump))", state}
|
||||
for i := 4; i <= 21; i++ {
|
||||
f = append(f, "0")
|
||||
}
|
||||
f = append(f, itoaT(start), "0", "0")
|
||||
if err := os.WriteFile(filepath.Join(p, "stat"), []byte(strings.Join(f, " ")+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// RED-PROOF: make taskAlive return (true, true) for a missing /proc entry → the dead task runs → fails.
|
||||
func TestVzdumpInFlight_WorkerGoneIsDead(t *testing.T) {
|
||||
running, dead := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, 0, 0))
|
||||
if running || len(dead) != 1 {
|
||||
t.Fatalf("a task whose worker is gone must be dead: running=%v dead=%d", running, len(dead))
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the `start == u.PStart` comparison (alive whenever the pid exists) → a reused pid runs → fails.
|
||||
func TestVzdumpInFlight_ReusedPidIsDead(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, testPID, 42)); running {
|
||||
t.Fatal("the pid exists but started at another tick: another process, the task is dead")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the `f[0] != "Z"` test → a zombie worker counts as running → fails.
|
||||
func TestVzdumpInFlight_ZombieIsDead(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixtureState(t, testPID, testPStart, "Z")); running {
|
||||
t.Fatal("a zombie worker is not running (Proxmox's own rule)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVzdumpInFlight_LiveWorkerRuns(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, testPID, testPStart)); !running {
|
||||
t.Fatal("the worker lives (same pid, same start tick): in flight")
|
||||
}
|
||||
}
|
||||
|
||||
// The reason for the rewrite: the wall clock plays no part. A live task whose list start time lies „before the boot"
|
||||
// (a clock step after the boot) is still running.
|
||||
func TestVzdumpInFlight_ClockStepCannotKillALiveTask(t *testing.T) {
|
||||
tk := vz("9201")
|
||||
tk.StartTime = 1 // absurdly early by the wall clock
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{tk}, 9201, procFixture(t, testPID, testPStart)); !running {
|
||||
t.Fatal("a live worker must count as running whatever its wall-clock start says")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: make an unparsable UPID or unreadable stat „known dead" → fails.
|
||||
func TestVzdumpInFlight_DoubtFailsSafe(t *testing.T) {
|
||||
bad := vz("9201")
|
||||
bad.UPID = "garbage"
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{bad}, 9201, procFixture(t, 0, 0)); !running {
|
||||
t.Fatal("an unparsable UPID must count as running (fail safe)")
|
||||
}
|
||||
d := procFixture(t, testPID, testPStart)
|
||||
if err := os.WriteFile(filepath.Join(d, itoaT(testPID), "stat"), []byte("garbage"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, d); !running {
|
||||
t.Fatal("an unparsable stat must count as running (fail safe)")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the PID-1 visibility check → a hidden /proc makes the live task „gone" → fails.
|
||||
func TestVzdumpInFlight_HiddenProcFailsSafe(t *testing.T) {
|
||||
d := t.TempDir() // nothing visible: not even PID 1 (hidepid / ProtectProc)
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, d); !running {
|
||||
t.Fatal("a /proc that hides other users' processes must count as running (fail safe)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVzdumpInFlight_OtherGuestOrTypeIgnored(t *testing.T) {
|
||||
tasks := []proxmox.TaskStatus{vz("9202"), {UPID: vz("9201").UPID, Type: "vzsnapshot", ID: "9201"}}
|
||||
if running, dead := vzdumpInFlight(tasks, 9201, procFixture(t, testPID, testPStart)); running || len(dead) != 0 {
|
||||
t.Fatalf("other guests' and other types' tasks are not 9201's backup: running=%v dead=%d", running, len(dead))
|
||||
}
|
||||
}
|
||||
|
||||
func inflightCtrl(api *fakeStaleLockAPI, runner *recordingRunner, logger *slog.Logger, proc string) *staleLockController {
|
||||
c := NewStaleLockController(api, runner, "felhom", logger).(*staleLockController)
|
||||
c.procDir = proc
|
||||
return c
|
||||
}
|
||||
|
||||
// The production controller answers through the helper, and logs a dead task ONCE.
|
||||
// RED-PROOF: make BackupRunning ignore the helper (any listed task runs) → fails.
|
||||
func TestStaleLockController_BackupRunningIgnoresDeadTask(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")}}
|
||||
c := inflightCtrl(api, &recordingRunner{}, slog.New(slog.NewTextHandler(&buf, nil)), procFixture(t, 0, 0))
|
||||
for i := 0; i < 3; i++ {
|
||||
if running, err := c.BackupRunning(context.Background(), 9201); err != nil || running {
|
||||
t.Fatalf("dead task reported running=%v err=%v", running, err)
|
||||
}
|
||||
}
|
||||
if n := strings.Count(buf.String(), "backup-in-flight: a vzdump task is listed as active but its worker process is gone"); n != 1 {
|
||||
t.Fatalf("the dead task must be logged exactly once, got %d", n)
|
||||
}
|
||||
c.procDir = procFixture(t, testPID, testPStart)
|
||||
if running, _ := c.BackupRunning(context.Background(), 9201); !running {
|
||||
t.Fatal("a live worker must be running")
|
||||
}
|
||||
}
|
||||
|
||||
func fixtureFor(t *testing.T, alive bool) string {
|
||||
if alive {
|
||||
return procFixture(t, testPID, testPStart)
|
||||
}
|
||||
return procFixture(t, 0, 0)
|
||||
}
|
||||
|
||||
func poolOf(vmids ...int) proxmox.PoolInfo {
|
||||
var p proxmox.PoolInfo
|
||||
raw := `{"members":[`
|
||||
for i, v := range vmids {
|
||||
if i > 0 {
|
||||
raw += ","
|
||||
}
|
||||
raw += `{"vmid":` + itoaT(v) + `,"type":"lxc"}`
|
||||
}
|
||||
raw += `]}`
|
||||
_ = json.Unmarshal([]byte(raw), &p)
|
||||
return p
|
||||
}
|
||||
|
||||
// Caller 1 — the startup stale-lock recovery clears the lock a power cut left (the night of 2026-10-14 it said
|
||||
// „genuinely in-flight — leaving the lock").
|
||||
func TestCaller_StartupStaleLockClearsAfterPowerCut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
alive bool
|
||||
cleared bool
|
||||
}{{false, true}, {true, false}} {
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")},
|
||||
cfg: map[int]proxmox.GuestConfig{9201: staleCfg("backup", true)}}
|
||||
runner := &recordingRunner{}
|
||||
s := &Server{staleLock: inflightCtrl(api, runner, nil, fixtureFor(t, tc.alive)), logger: slog.New(slog.NewTextHandler(discardW{}, nil))}
|
||||
s.recoverOneStaleLock(context.Background(), proxmox.Guest{VMID: 9201, Status: "running"})
|
||||
if runner.unlocked("9201") != tc.cleared {
|
||||
t.Fatalf("worker alive=%v: unlocked=%v, want %v", tc.alive, runner.unlocked("9201"), tc.cleared)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Caller 2 — the guest-power watchdog starts a stopped guest despite a dead pre-boot vzdump task.
|
||||
func TestCaller_GuestPowerStartsGuestAfterPowerCut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
alive bool
|
||||
started bool
|
||||
}{{false, true}, {true, false}} {
|
||||
api := &fakeStaleLockAPI{lxc: []proxmox.Guest{{VMID: 9201, Status: "stopped"}}, pool: poolOf(9201),
|
||||
tasks: []proxmox.TaskStatus{vz("9201")}, cfg: map[int]proxmox.GuestConfig{9201: staleCfg("", true)}}
|
||||
s := gpServer(t, inflightCtrl(api, &recordingRunner{}, nil, fixtureFor(t, tc.alive)), nil)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if (len(api.started) == 1) != tc.started {
|
||||
t.Fatalf("worker alive=%v: started=%v, want %v", tc.alive, api.started, tc.started)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Caller 3 — the household power-off guard does not refuse „backup running" for a dead task.
|
||||
func TestCaller_PowerOffJobBlockerIgnoresDeadTask(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
alive bool
|
||||
want string
|
||||
}{{false, ""}, {true, PowerOffBackup}} {
|
||||
rig := newPowerRig(t, 0)
|
||||
rig.srv.staleLock = inflightCtrl(&fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")}}, &recordingRunner{}, nil, fixtureFor(t, tc.alive))
|
||||
if code, _ := rig.srv.jobBlocker(context.Background(), 9201); code != tc.want {
|
||||
t.Fatalf("worker alive=%v: jobBlocker=%q, want %q", tc.alive, code, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Caller 4 (the supervisor), the R-1006 cleanup (VzdumpInFlight) and the rest — structural pin: every non-test BackupRunning call in this package goes through
|
||||
// s.staleLock (whose one production implementation is staleLockController, which uses vzdumpInFlight), and no
|
||||
// other production type implements BackupRunning. If someone adds a second path, this fails.
|
||||
func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
|
||||
files, _ := filepath.Glob("*.go")
|
||||
call := regexp.MustCompile(`([A-Za-z_.]+)\.BackupRunning\(`)
|
||||
impl := regexp.MustCompile(`func \([a-z]+ \*?([A-Za-z]+)\) BackupRunning\(`)
|
||||
callers := 0
|
||||
for _, f := range files {
|
||||
if strings.HasSuffix(f, "_test.go") {
|
||||
continue
|
||||
}
|
||||
b, err := os.ReadFile(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, m := range call.FindAllStringSubmatch(string(b), -1) {
|
||||
if m[1] != "s.staleLock" {
|
||||
t.Errorf("%s: BackupRunning called on %q, not s.staleLock", f, m[1])
|
||||
}
|
||||
callers++
|
||||
}
|
||||
for _, m := range impl.FindAllStringSubmatch(string(b), -1) {
|
||||
if m[1] != "staleLockController" {
|
||||
t.Errorf("%s: a second production BackupRunning on %s", f, m[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
if callers != 5 {
|
||||
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
|
||||
}
|
||||
b, _ := os.ReadFile("stalelock.go")
|
||||
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
|
||||
t.Error("staleLockController.BackupRunning no longer answers through vzdumpInFlight")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package localapi
|
||||
|
||||
import "context"
|
||||
|
||||
// R-1006: the half-written vzdump cleanup's two needs from the Server, and its start-up sweep. The cleaner itself is
|
||||
// internal/vzpartial; the root helper is configs/felhom-vzdump-partial-clean.
|
||||
|
||||
// VzdumpInFlight is the one in-flight answer (staleLockController.BackupRunning, R-1008) for the cleanup. No stale-lock
|
||||
// controller wired → true: fail safe, nothing is removed.
|
||||
func (s *Server) VzdumpInFlight(ctx context.Context, vmid int) (bool, error) {
|
||||
if s.staleLock == nil {
|
||||
return true, nil
|
||||
}
|
||||
return s.staleLock.BackupRunning(ctx, vmid)
|
||||
}
|
||||
|
||||
// SetPartialSweep sets the start-up sweep: called with every owned guest (the felhom pool, ownership proven).
|
||||
func (s *Server) SetPartialSweep(fn func(ctx context.Context, vmid int)) { s.partialSweep = fn }
|
||||
|
||||
// SweepVzdumpPartials runs the sweep once for every owned guest. No-op when unwired.
|
||||
func (s *Server) SweepVzdumpPartials(ctx context.Context) {
|
||||
if s.partialSweep == nil || s.staleLock == nil {
|
||||
return
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
s.logger.Warn("vzdump-partial: guest list unavailable — skipping the start-up sweep", "err", err)
|
||||
return
|
||||
}
|
||||
for _, g := range guests {
|
||||
s.partialSweep(ctx, g.VMID)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
// Package poweroff keeps the box's record of a household-requested power-off (R-998, doc 14).
|
||||
//
|
||||
// One small JSON file in the agent's state dir. A record belongs to the boot it was made in (the
|
||||
// kernel's btime from /proc/stat), so:
|
||||
// - "at most one accepted request per boot": Mark refuses when this boot already has a record;
|
||||
// - the host report carries the record only while the box is still in that boot (Current), and
|
||||
// the next boot's reports carry nothing — that absence is how the hub learns the box is back.
|
||||
//
|
||||
// Pinned by state_test.go.
|
||||
package poweroff
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrAlreadyThisBoot is Mark's refusal: a power-off was already accepted in the current boot.
|
||||
var ErrAlreadyThisBoot = errors.New("a power-off was already accepted in this boot")
|
||||
|
||||
// ErrBootUnknown is Mark's refusal when the boot cannot be identified — fail closed: without it the
|
||||
// per-boot limit could not hold.
|
||||
var ErrBootUnknown = errors.New("cannot read the host's boot time")
|
||||
|
||||
// Record is one accepted request.
|
||||
type Record struct {
|
||||
RequestedAt time.Time `json:"requested_at"`
|
||||
BootBtime string `json:"boot_btime"` // /proc/stat btime of the boot it was accepted in
|
||||
VMID int `json:"vmid"` // the guest whose controller asked
|
||||
}
|
||||
|
||||
// State is the persisted record.
|
||||
type State struct {
|
||||
path string
|
||||
btime func() string
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// New returns the state at path. btime returns the current boot's /proc/stat btime ("" = unknown).
|
||||
func New(path string, btime func() string) *State {
|
||||
if btime == nil {
|
||||
btime = ProcBtime
|
||||
}
|
||||
return &State{path: path, btime: btime}
|
||||
}
|
||||
|
||||
// Current is the record made in THIS boot, or nil (none, another boot's, unreadable).
|
||||
func (s *State) Current() *Record {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
r := s.load()
|
||||
b := s.btime()
|
||||
if r == nil || b == "" || r.BootBtime != b {
|
||||
return nil
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// Mark records an accepted request for this boot. It refuses a second one in the same boot.
|
||||
func (s *State) Mark(now time.Time, vmid int) (*Record, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
b := s.btime()
|
||||
if b == "" {
|
||||
return nil, ErrBootUnknown
|
||||
}
|
||||
if r := s.load(); r != nil && r.BootBtime == b {
|
||||
return nil, ErrAlreadyThisBoot
|
||||
}
|
||||
r := &Record{RequestedAt: now.UTC().Truncate(time.Second), BootBtime: b, VMID: vmid}
|
||||
data, _ := json.Marshal(r)
|
||||
if err := os.MkdirAll(filepath.Dir(s.path), 0o750); err != nil {
|
||||
return nil, fmt.Errorf("poweroff: state dir: %w", err)
|
||||
}
|
||||
tmp := s.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||
return nil, fmt.Errorf("poweroff: write: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmp, s.path); err != nil {
|
||||
return nil, fmt.Errorf("poweroff: rename: %w", err)
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (s *State) load() *Record {
|
||||
data, err := os.ReadFile(s.path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var r Record
|
||||
if json.Unmarshal(data, &r) != nil || r.BootBtime == "" {
|
||||
return nil
|
||||
}
|
||||
return &r
|
||||
}
|
||||
|
||||
// ProcBtime reads the boot time (seconds since the epoch, as text) from /proc/stat; "" when unreadable.
|
||||
func ProcBtime() string {
|
||||
data, err := os.ReadFile("/proc/stat")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
for _, line := range strings.Split(string(data), "\n") {
|
||||
if strings.HasPrefix(line, "btime ") {
|
||||
return strings.TrimSpace(strings.TrimPrefix(line, "btime "))
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package poweroff
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// One accepted request per boot; the record is "current" only in the boot it was made in.
|
||||
func TestOnePerBootAndCurrentFollowsTheBoot(t *testing.T) {
|
||||
boot := "1000"
|
||||
s := New(filepath.Join(t.TempDir(), "poweroff.json"), func() string { return boot })
|
||||
if s.Current() != nil {
|
||||
t.Fatal("no record yet, Current must be nil")
|
||||
}
|
||||
now := time.Date(2026, 10, 13, 18, 0, 0, 0, time.UTC)
|
||||
if _, err := s.Mark(now, 9201); err != nil {
|
||||
t.Fatalf("first Mark: %v", err)
|
||||
}
|
||||
if r := s.Current(); r == nil || !r.RequestedAt.Equal(now) || r.VMID != 9201 {
|
||||
t.Fatalf("Current after Mark = %+v", r)
|
||||
}
|
||||
if _, err := s.Mark(now.Add(time.Minute), 9201); !errors.Is(err, ErrAlreadyThisBoot) {
|
||||
t.Fatalf("second Mark in the same boot: err = %v, want ErrAlreadyThisBoot", err)
|
||||
}
|
||||
boot = "2000" // the machine was switched on again
|
||||
if s.Current() != nil {
|
||||
t.Fatal("a record from the previous boot must not be current")
|
||||
}
|
||||
if _, err := s.Mark(now.Add(time.Hour), 9201); err != nil {
|
||||
t.Fatalf("Mark in a new boot: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownBootFailsClosed(t *testing.T) {
|
||||
s := New(filepath.Join(t.TempDir(), "p.json"), func() string { return "" })
|
||||
if _, err := s.Mark(time.Now(), 1); !errors.Is(err, ErrBootUnknown) {
|
||||
t.Fatalf("err = %v, want ErrBootUnknown", err)
|
||||
}
|
||||
}
|
||||
@@ -64,6 +64,11 @@ const (
|
||||
// The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units).
|
||||
// Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself.
|
||||
ClassAgentConfigUpdate OpClass = "agent_config_update"
|
||||
|
||||
// The internal data place on an OLD one-disk box (agent v0.158.0, felhom.eu `16` §9): thin space on the system
|
||||
// disk, made by the root helper felhom-internal-place. Destructive-class (signed, operational key): it is an
|
||||
// operator act, never a household button, and no agent-internal provenance may start it.
|
||||
ClassAddInternalPlace OpClass = "add_internal_place"
|
||||
)
|
||||
|
||||
// Disposition is the classifier verdict.
|
||||
@@ -132,7 +137,7 @@ func Classify(class OpClass, prov Provenance) Disposition {
|
||||
return Destructive
|
||||
case ClassKeyRotation:
|
||||
return Destructive
|
||||
case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassOSKernelStep, ClassAgentConfigUpdate:
|
||||
case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassOSKernelStep, ClassAgentConfigUpdate, ClassAddInternalPlace:
|
||||
// Never benign — no agent-internal provenance can make replacing the agent binary
|
||||
// unsigned-safe (a compromised process must not be able to self-bless an update).
|
||||
return Destructive
|
||||
|
||||
@@ -39,9 +39,21 @@ type decommissionParams struct {
|
||||
// none is wanted, because a drive can legitimately be decommissioned while physically ABSENT.
|
||||
type DecommissionExecutor struct {
|
||||
intent IntentDecommissioner
|
||||
binds GuestBindPruner // optional (v0.159.1): drop the drive from every guest's bind record, as the self-serve path does
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
// GuestBindPruner is the per-guest enrolled-bind record (*localapi.GuestBindStore). The self-serve decommission prunes
|
||||
// the drive from it; the signed one did not, so a forgotten drive (R-1124) left a record the 20-second guest-bind
|
||||
// re-assert warned about forever ("skipping non-enrolled drive"), measured on demo-hp 2026-10-17.
|
||||
type GuestBindPruner interface {
|
||||
Guests() map[int][]string
|
||||
Remove(vmid int, durableID string) error
|
||||
}
|
||||
|
||||
// SetGuestBinds wires the bind record (nil-safe: no record, nothing pruned).
|
||||
func (d *DecommissionExecutor) SetGuestBinds(b GuestBindPruner) { d.binds = b }
|
||||
|
||||
// NewDecommissionExecutor wires the executor to the agent's intent store.
|
||||
func NewDecommissionExecutor(intent IntentDecommissioner, logger *slog.Logger) *DecommissionExecutor {
|
||||
if logger == nil {
|
||||
@@ -75,6 +87,20 @@ func (d *DecommissionExecutor) Execute(_ context.Context, op string, params json
|
||||
if err := d.intent.SetDecommissioned(p.DurableID); err != nil {
|
||||
return fmt.Errorf("decommission: record intent for %q: %w", p.DurableID, err)
|
||||
}
|
||||
if d.binds != nil {
|
||||
for vmid, ids := range d.binds.Guests() {
|
||||
for _, id := range ids {
|
||||
if id != p.DurableID {
|
||||
continue
|
||||
}
|
||||
if err := d.binds.Remove(vmid, id); err != nil {
|
||||
d.logger.Warn("decommission: guest-bind record not pruned", "vmid", vmid, "durable_id", id, "err", err)
|
||||
} else {
|
||||
d.logger.Info("decommission: drive removed from the guest's bind record", "vmid", vmid, "durable_id", id)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
d.logger.Warn("decommission: operator-signed decommission complete (drive will never auto-mount)", "durable_id", p.DurableID)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -89,3 +89,37 @@ func TestExecutorChain_DispatchesToOwner(t *testing.T) {
|
||||
t.Errorf("want ErrNoExecutor for an unowned op, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
type fakeBinds struct{ m map[int][]string }
|
||||
|
||||
func (f *fakeBinds) Guests() map[int][]string { return f.m }
|
||||
func (f *fakeBinds) Remove(vmid int, id string) error {
|
||||
var kept []string
|
||||
for _, x := range f.m[vmid] {
|
||||
if x != id {
|
||||
kept = append(kept, x)
|
||||
}
|
||||
}
|
||||
f.m[vmid] = kept
|
||||
return nil
|
||||
}
|
||||
|
||||
// v0.159.1: the signed decommission drops the drive from EVERY guest's bind record, as the self-serve path does — and
|
||||
// only that drive. RED-PROOF (felhom.eu audits/agent-allowlist-rotation-2026-10-17/P1/redproof-bindprune.txt): with
|
||||
// the prune loop removed, the record still holds the drive and this test fails.
|
||||
func TestDecommissionExecutor_PrunesTheGuestBindRecord(t *testing.T) {
|
||||
fd := &fakeDecommissioner{}
|
||||
fb := &fakeBinds{m: map[int][]string{9201: {"uuid:keep", "uuid:gone"}, 9202: {"uuid:gone"}}}
|
||||
ex := NewDecommissionExecutor(fd, nil)
|
||||
ex.SetGuestBinds(fb)
|
||||
params, _ := json.Marshal(decommissionParams{DurableID: "uuid:gone"})
|
||||
if err := ex.Execute(context.Background(), opDecommission, params); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := fb.m[9201]; len(got) != 1 || got[0] != "uuid:keep" {
|
||||
t.Fatalf("9201 record = %v, want [uuid:keep]", got)
|
||||
}
|
||||
if got := fb.m[9202]; len(got) != 0 {
|
||||
t.Fatalf("9202 record = %v, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
package signedjobs
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// opAddInternalPlace is the op class this executor serves (mirrors reconcile.ClassAddInternalPlace).
|
||||
const opAddInternalPlace = "add_internal_place"
|
||||
|
||||
// InternalPlaceHelper is the root helper's path (configs/felhom-internal-place, from the config bundle).
|
||||
const InternalPlaceHelper = "/usr/local/sbin/felhom-internal-place"
|
||||
|
||||
var reHelperResult = regexp.MustCompile(`^(created|existing|formatted) uuid=([0-9a-fA-F-]{36}) gib=([0-9]+)$`)
|
||||
|
||||
// InternalPlaceExecutor adds the internal data place to an OLD one-disk box (felhom.eu architecture/16 §9: „one-disk
|
||||
// boxes get no internal place by themselves; adding one later is a signed operator job, not a household button").
|
||||
// It runs the SAME root helper the installer runs — `felhom-internal-place create 0` (0 GiB planned: the guest's
|
||||
// volumes already exist and are counted) — writes the record the agent mounts from, and asks the local API to mount
|
||||
// it now. Every safety rule (the ruled size, never format what holds anything) lives in the helper, as root.
|
||||
//
|
||||
// The op carries no parameters: the size is the rule's, never the signer's. Destructive-class by default (the
|
||||
// classifier's fail-safe; ClassAddInternalPlace documents it): it takes thin space on the system disk, so only an
|
||||
// operator signature may start it.
|
||||
type InternalPlaceExecutor struct {
|
||||
runner proxmox.Runner
|
||||
recordFile string
|
||||
ensure func(context.Context) // late-bound: the local API server is built after the jobs runner
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
// NewInternalPlaceExecutor wires the executor. ensure may be nil at construction and set later through SetEnsure.
|
||||
func NewInternalPlaceExecutor(runner proxmox.Runner, recordFile string, logger *slog.Logger) *InternalPlaceExecutor {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
return &InternalPlaceExecutor{runner: runner, recordFile: recordFile, logger: logger}
|
||||
}
|
||||
|
||||
// SetEnsure late-binds the mount step (localapi.Server.EnsureInternalPlace).
|
||||
func (e *InternalPlaceExecutor) SetEnsure(f func(context.Context)) { e.ensure = f }
|
||||
|
||||
// Execute implements signedjobs.Executor for add_internal_place.
|
||||
func (e *InternalPlaceExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
|
||||
if op != opAddInternalPlace {
|
||||
return ErrNoExecutor
|
||||
}
|
||||
if p := bytes.TrimSpace(params); len(p) > 0 && !bytes.Equal(p, []byte("{}")) && !bytes.Equal(p, []byte("null")) {
|
||||
return fmt.Errorf("add_internal_place: takes no parameters (the size is the ruled rule's), got %s", p)
|
||||
}
|
||||
if e.runner == nil || e.recordFile == "" {
|
||||
return fmt.Errorf("add_internal_place: not wired on this host (no privileged runner or state dir)")
|
||||
}
|
||||
start := time.Now()
|
||||
e.logger.Warn("add_internal_place: executing operator-signed job (the internal data place, `16` §9)")
|
||||
out, stderr, err := e.runner.Run(ctx, InternalPlaceHelper, "create", "0")
|
||||
line := strings.TrimSpace(string(out))
|
||||
if err != nil {
|
||||
return fmt.Errorf("add_internal_place: helper refused: %v: %s", err, strings.TrimSpace(string(stderr)))
|
||||
}
|
||||
if strings.HasPrefix(line, "skipped ") {
|
||||
// Not an error the operator can retry away: the pool has no room by the ruled rule. Fail the job with the
|
||||
// helper's own numbers so the hub shows why.
|
||||
return fmt.Errorf("add_internal_place: %s", line)
|
||||
}
|
||||
m := reHelperResult.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
return fmt.Errorf("add_internal_place: unexpected helper output %q", line)
|
||||
}
|
||||
gib, _ := strconv.Atoi(m[3])
|
||||
rec := localapi.InternalPlaceRecord{FSUUID: m[2], GiB: gib, MadeBy: "signed-job", MadeAt: time.Now().UTC().Format(time.RFC3339)}
|
||||
if err := localapi.WriteInternalPlace(e.recordFile, rec); err != nil {
|
||||
return fmt.Errorf("add_internal_place: writing the record: %w", err)
|
||||
}
|
||||
if e.ensure != nil {
|
||||
e.ensure(ctx)
|
||||
}
|
||||
e.logger.Warn("add_internal_place: complete", "result", m[1], "fs_uuid", m[2], "gib", gib,
|
||||
"duration_ms", time.Since(start).Milliseconds())
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
package signedjobs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// add_internal_place (felhom.eu architecture/16 §9, agent v0.158.0): the operator's job for an OLD one-disk box.
|
||||
|
||||
type ipRunner struct {
|
||||
out, stderr string
|
||||
err error
|
||||
calls [][]string
|
||||
}
|
||||
|
||||
func (r *ipRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
return []byte(r.out), []byte(r.stderr), r.err
|
||||
}
|
||||
func (r *ipRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
func ipExec(t *testing.T, r *ipRunner) (*InternalPlaceExecutor, string, *int) {
|
||||
t.Helper()
|
||||
file := filepath.Join(t.TempDir(), localapi.InternalPlaceFile)
|
||||
e := NewInternalPlaceExecutor(r, file, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
n := 0
|
||||
e.SetEnsure(func(context.Context) { n++ })
|
||||
return e, file, &n
|
||||
}
|
||||
|
||||
func TestAddInternalPlace_RunsTheHelperWritesTheRecordAndMounts(t *testing.T) {
|
||||
r := &ipRunner{out: "created uuid=0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c gib=33\n"}
|
||||
e, file, ensured := ipExec(t, r)
|
||||
if err := e.Execute(context.Background(), "add_internal_place", []byte(`{}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(r.calls) != 1 || strings.Join(r.calls[0], " ") != InternalPlaceHelper+" create 0" {
|
||||
t.Fatalf("the helper must be called exactly as `create 0`, got %v", r.calls)
|
||||
}
|
||||
rec, ok, err := localapi.ReadInternalPlace(file)
|
||||
if err != nil || !ok || rec.FSUUID != "0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c" || rec.GiB != 33 || rec.MadeBy != "signed-job" {
|
||||
t.Fatalf("record: %+v ok=%v err=%v", rec, ok, err)
|
||||
}
|
||||
if *ensured != 1 {
|
||||
t.Fatalf("the mount must be asked for once, got %d", *ensured)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddInternalPlace_NoRoomOrRefusalOrGarbage_WritesNothing(t *testing.T) {
|
||||
for name, r := range map[string]*ipRunner{
|
||||
"skipped": {out: "skipped only 7 GiB free for it (pool 53, reserve 16, promised 102, planned 0; needs 20)"},
|
||||
"refused": {err: errors.New("exit status 1"), stderr: "felhom-internal-place: REFUSED: holds something"},
|
||||
"garbage": {out: "created uuid=../../etc gib=1"},
|
||||
} {
|
||||
e, file, ensured := ipExec(t, r)
|
||||
if err := e.Execute(context.Background(), "add_internal_place", nil); err == nil {
|
||||
t.Fatalf("%s: want an error", name)
|
||||
}
|
||||
if _, statErr := os.Stat(file); !os.IsNotExist(statErr) || *ensured != 0 {
|
||||
t.Fatalf("%s: no record and no mount may follow, stat=%v ensured=%d", name, statErr, *ensured)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddInternalPlace_TakesNoParametersAndIgnoresOtherOps(t *testing.T) {
|
||||
r := &ipRunner{out: "created uuid=0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c gib=33"}
|
||||
e, _, _ := ipExec(t, r)
|
||||
if err := e.Execute(context.Background(), "add_internal_place", []byte(`{"gib":900}`)); err == nil || len(r.calls) != 0 {
|
||||
t.Fatalf("a size from the signer must be refused before the helper runs, err=%v calls=%v", err, r.calls)
|
||||
}
|
||||
if err := e.Execute(context.Background(), "decommission", []byte(`{}`)); !errors.Is(err, ErrNoExecutor) {
|
||||
t.Fatalf("another op: want ErrNoExecutor, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddInternalPlace_IsSignatureOnly(t *testing.T) {
|
||||
if got := reconcile.Classify(reconcile.ClassAddInternalPlace, reconcile.Provenance{}); got != reconcile.Destructive {
|
||||
t.Fatalf("add_internal_place must need an operator signature, classified %v", got)
|
||||
}
|
||||
if string(reconcile.ClassAddInternalPlace) != opAddInternalPlace {
|
||||
t.Fatalf("op string drift: %q vs %q", reconcile.ClassAddInternalPlace, opAddInternalPlace)
|
||||
}
|
||||
}
|
||||
@@ -13,14 +13,16 @@ import (
|
||||
// (guarded, Impl-1) and/or mounts + binds. Blank disks are "initialize"-only; disks already carrying a
|
||||
// mountable FS are ALSO "attach" candidates.
|
||||
type CandidateDisk struct {
|
||||
Device string `json:"device"` // whole disk, e.g. /dev/sdd
|
||||
SizeBytes int64 `json:"size_bytes"` // 0 when unreadable
|
||||
Device string `json:"device"` // whole disk, e.g. /dev/sdd
|
||||
SizeBytes int64 `json:"size_bytes"` // 0 when unreadable
|
||||
Model string `json:"model,omitempty"`
|
||||
FSType string `json:"fstype,omitempty"` // first filesystem found ("" = blank)
|
||||
DataBearing bool `json:"data_bearing"` // has any FS / partition (→ wipe-confirm downstream)
|
||||
Mountable bool `json:"mountable"` // carries an ext4/xfs FS the agent can mount as-is
|
||||
FSType string `json:"fstype,omitempty"` // first filesystem found ("" = blank)
|
||||
DataBearing bool `json:"data_bearing"` // has any FS / partition (→ wipe-confirm downstream)
|
||||
Mountable bool `json:"mountable"` // carries an ext4/xfs FS the agent can mount as-is
|
||||
MountSource string `json:"mount_source,omitempty"` // the node to mount for attach (e.g. /dev/sdd1)
|
||||
DurableID string `json:"durable_id,omitempty"` // "uuid:<fs-uuid>" if it has an FS; "" for a blank disk
|
||||
// Partitions (v0.158.0, felhom.eu `16` §6): each partition of this disk as its own place — see partitions.go.
|
||||
Partitions []CandidatePartition `json:"partitions,omitempty"`
|
||||
}
|
||||
|
||||
// mountableFSTypes are the filesystems the agent can mount as-is (EnsureMount) → an "attach" candidate.
|
||||
@@ -83,6 +85,8 @@ func (h *SudoHostOps) buildCandidate(dev, name string) CandidateDisk {
|
||||
if len(nodes) > 1 {
|
||||
c.DataBearing = true
|
||||
}
|
||||
c.Partitions = candidatePartitions(lout, name, readSysPartSize, readSysPartNumber, h.host.ResolveUUID,
|
||||
felhomOwnedMounts(dev, nodes, h.mountTable))
|
||||
}
|
||||
}
|
||||
return c
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
@@ -24,8 +25,13 @@ var devDiskRoot = "/dev/disk"
|
||||
const (
|
||||
durableByID = "byid:"
|
||||
durableByUUID = "byuuid:"
|
||||
// partFSUUIDSep joins a partition's by-id link and its fs-UUID in one wipe id (v0.158.0, `16` §6).
|
||||
partFSUUIDSep = "+fsuuid:"
|
||||
)
|
||||
|
||||
// rePartLink matches a by-id link of a PARTITION (udev appends -part<N> to the disk's link).
|
||||
var rePartLink = regexp.MustCompile(`-part[1-9][0-9]{0,2}$`)
|
||||
|
||||
// byIDPriority ranks /dev/disk/by-id link prefixes most-stable-first: wwn (hardware world-wide
|
||||
// name) > nvme-eui (NVMe EUI) > nvme-/ata-/scsi-/usb- (model+serial). dm-/lvm-/md- names are
|
||||
// excluded (they are mapper constructs, not the physical disk we want to bind a wipe to).
|
||||
@@ -41,6 +47,14 @@ func DeviceDurableID(device string) (string, error) {
|
||||
}
|
||||
// 1. best by-id link pointing at this device.
|
||||
if name := bestByIDLink(target); name != "" {
|
||||
// v0.158.0 (felhom.eu `16` §6): a PARTITION's link already carries the disk id + the part number
|
||||
// (wwn-…-part2). When the part holds a filesystem, its fs-UUID is bound too, so a part that was reformatted —
|
||||
// or another disk's part2 that took the link — derives a different id and the wipe refuses.
|
||||
if rePartLink.MatchString(name) {
|
||||
if uuid := byUUIDFor(target); uuid != "" {
|
||||
return durableByID + name + partFSUUIDSep + uuid, nil
|
||||
}
|
||||
}
|
||||
return durableByID + name, nil
|
||||
}
|
||||
// 2. fallback: a filesystem UUID symlink.
|
||||
@@ -56,10 +70,29 @@ func ResolveDurableDevice(durableID string) (string, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(durableID, durableByID):
|
||||
name := strings.TrimPrefix(durableID, durableByID)
|
||||
fsUUID := ""
|
||||
if i := strings.Index(name, partFSUUIDSep); i >= 0 {
|
||||
name, fsUUID = name[:i], name[i+len(partFSUUIDSep):]
|
||||
if !rePartLink.MatchString(name) || !safeLinkName(fsUUID) {
|
||||
return "", fmt.Errorf("storage: unsafe durable id %q", durableID)
|
||||
}
|
||||
}
|
||||
if !safeLinkName(name) {
|
||||
return "", fmt.Errorf("storage: unsafe durable id %q", durableID)
|
||||
}
|
||||
return filepath.EvalSymlinks(filepath.Join(devDiskRoot, "by-id", name))
|
||||
dev, err := filepath.EvalSymlinks(filepath.Join(devDiskRoot, "by-id", name))
|
||||
if err != nil || fsUUID == "" {
|
||||
return dev, err
|
||||
}
|
||||
// Both halves must name the SAME device now (`16` §6): the part by its disk id + number, and its filesystem.
|
||||
byFS, ferr := filepath.EvalSymlinks(filepath.Join(devDiskRoot, "by-uuid", fsUUID))
|
||||
if ferr != nil {
|
||||
return "", fmt.Errorf("storage: durable id %q: its filesystem %s no longer resolves: %w", durableID, fsUUID, ferr)
|
||||
}
|
||||
if byFS != dev {
|
||||
return "", fmt.Errorf("storage: durable id %q: the part (%s) and its filesystem (%s) are different devices now", durableID, dev, byFS)
|
||||
}
|
||||
return dev, nil
|
||||
case strings.HasPrefix(durableID, durableByUUID):
|
||||
uuid := strings.TrimPrefix(durableID, durableByUUID)
|
||||
if !safeLinkName(uuid) {
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ForgetRetiredDrives (R-1124) removes the host mount unit of every DECOMMISSIONED drive that is physically gone, so a
|
||||
// later host boot no longer waits for it and the agent stops listing it (RegistryKnownTargets and
|
||||
// ReassertEnrolledMounts both enumerate the units — no unit, no entry, no „absent by UUID" line every 20 s).
|
||||
//
|
||||
// Only a drive that is BOTH decommissioned (the operator's or the customer's permanent removal) AND absent (its
|
||||
// /dev/disk/by-uuid link does not resolve) AND not mounted is forgotten. A decommissioned drive that is still plugged in
|
||||
// keeps its unit and its raw host mount — that is what makes the one-click re-enroll (H3) work, and decommission stays a
|
||||
// logical retire (`03` §storage). The intent record stays `decommissioned`: a forgotten drive that comes back is a new
|
||||
// candidate, never auto-mounted. The root half (`felhom-priv-apply unit-forget`) re-checks every one of these conditions
|
||||
// and refuses otherwise. Called at agent start, on the 20-second storage tick, and right after a decommission. Best
|
||||
// effort per unit; returns how many units it removed. Pinned by TestForgetRetiredDrives_*.
|
||||
func (h *SudoHostOps) ForgetRetiredDrives(ctx context.Context, intent IntentReader) int {
|
||||
if intent == nil {
|
||||
return 0 // no intent store → we cannot know a drive was retired; forget nothing
|
||||
}
|
||||
entries, err := os.ReadDir(h.unitDir)
|
||||
if err != nil {
|
||||
h.logger.Warn("storage: forget retired drives — cannot read unit dir", "dir", h.unitDir, "err", err)
|
||||
return 0
|
||||
}
|
||||
mounted := map[string]bool{}
|
||||
if ms, merr := h.host.Mounts(); merr == nil {
|
||||
for _, m := range ms {
|
||||
mounted[m.MountPoint] = true
|
||||
}
|
||||
} else {
|
||||
h.logger.Warn("storage: forget retired drives — mount table unreadable; forgetting nothing", "err", merr)
|
||||
return 0
|
||||
}
|
||||
removed := 0
|
||||
for _, e := range entries {
|
||||
name := e.Name()
|
||||
if !strings.HasSuffix(name, ".mount") {
|
||||
continue
|
||||
}
|
||||
data, rerr := os.ReadFile(filepath.Join(h.unitDir, name))
|
||||
if rerr != nil {
|
||||
continue
|
||||
}
|
||||
spec, ok := parseFelhomMountUnit(string(data))
|
||||
if !ok {
|
||||
continue // not one of our by-UUID drive units (network units are refused by the parser)
|
||||
}
|
||||
durableID := "uuid:" + spec.UUID
|
||||
if intent.Get(durableID) != IntentDecommissioned {
|
||||
continue
|
||||
}
|
||||
if mounted[spec.Where] {
|
||||
h.logger.Debug("storage: decommissioned drive still mounted — keeping its unit", "where", spec.Where)
|
||||
continue
|
||||
}
|
||||
if _, derr := ResolveStorageDevice(durableID); derr == nil {
|
||||
h.logger.Debug("storage: decommissioned drive is present — keeping its unit (re-enroll stays one click)", "where", spec.Where)
|
||||
continue
|
||||
}
|
||||
if h.unitEnabled(name) {
|
||||
if err := h.run(ctx, h.bins.Systemctl, "disable", "--", name); err != nil {
|
||||
h.logger.Warn("storage: forget retired drive — disable failed", "where", spec.Where, "unit", name, "err", err)
|
||||
continue
|
||||
}
|
||||
}
|
||||
if err := h.run(ctx, h.bins.PrivApply, "unit-forget", name); err != nil {
|
||||
h.logger.Warn("storage: forget retired drive — unit removal refused or failed", "where", spec.Where, "unit", name, "err", err)
|
||||
continue
|
||||
}
|
||||
removed++
|
||||
h.logger.Info("storage: forgot a decommissioned drive that is gone — its mount unit is removed",
|
||||
"name", spec.Name, "where", spec.Where, "uuid", spec.UUID, "unit", name)
|
||||
}
|
||||
if removed > 0 {
|
||||
if err := h.run(ctx, h.bins.Systemctl, "daemon-reload"); err != nil {
|
||||
h.logger.Warn("storage: daemon-reload after forgetting retired drives failed", "err", err)
|
||||
}
|
||||
}
|
||||
return removed
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// forgetRunner records every privileged call and plays the root half of `felhom-priv-apply unit-forget`: the unit file
|
||||
// disappears from the unit dir, so the CONSEQUENCE (the drive is no longer listed) is observable, not just the calls.
|
||||
type forgetRunner struct {
|
||||
unitDir string
|
||||
calls [][]string
|
||||
}
|
||||
|
||||
func (r *forgetRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
func (r *forgetRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
if len(args) == 2 && args[0] == "unit-forget" {
|
||||
_ = os.Remove(filepath.Join(r.unitDir, args[1]))
|
||||
}
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
type forgetCase struct {
|
||||
intent DriveIntent
|
||||
present bool // the by-uuid link resolves
|
||||
mounted bool
|
||||
enabled bool // the wants-symlink exists
|
||||
mountErr bool
|
||||
}
|
||||
|
||||
const forgetUUID = "4ebfd36d-87d3-4b63-839e-a5a710fc799a"
|
||||
|
||||
func runForget(t *testing.T, c forgetCase, intent IntentReader) (int, *forgetRunner, *RegistryKnownTargets, string) {
|
||||
t.Helper()
|
||||
unitDir := t.TempDir()
|
||||
unit := "mnt-media_a.mount"
|
||||
writeUnit(t, unitDir, unit, renderMountUnit(MountSpec{Name: forgetUUID, UUID: forgetUUID, Where: "/mnt/media_a", FSType: "ext4"}))
|
||||
if c.enabled {
|
||||
if err := os.MkdirAll(filepath.Join(unitDir, "multi-user.target.wants"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink(filepath.Join(unitDir, unit), filepath.Join(unitDir, "multi-user.target.wants", unit)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
devRoot := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(devRoot, "by-uuid"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if c.present {
|
||||
dev := filepath.Join(devRoot, "sdz1")
|
||||
if err := os.WriteFile(dev, nil, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink(dev, filepath.Join(devRoot, "by-uuid", forgetUUID)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
withDevDiskRoot(t, devRoot)
|
||||
hr := &fakeHostReader{}
|
||||
if c.mounted {
|
||||
hr.mounts = []Mount{{MountPoint: "/mnt/media_a", FSType: "ext4"}}
|
||||
}
|
||||
if c.mountErr {
|
||||
hr.mountsErr = os.ErrPermission
|
||||
}
|
||||
rr := &forgetRunner{unitDir: unitDir}
|
||||
ops := NewSudoHostOps(SudoHostOpsConfig{Runner: rr, Bins: Binaries{}.withDefaults(), UnitDir: unitDir,
|
||||
StageDir: t.TempDir(), Host: hr, Logger: quietLogger()})
|
||||
if intent == nil {
|
||||
intent = &fakeIntent{m: map[string]DriveIntent{"uuid:" + forgetUUID: c.intent}}
|
||||
}
|
||||
n := ops.ForgetRetiredDrives(context.Background(), intent)
|
||||
return n, rr, NewRegistryKnownTargets(unitDir, intent, quietLogger()), unitDir
|
||||
}
|
||||
|
||||
// The R-1124 case: decommissioned, then unplugged. The unit is disabled and removed through the root checker, systemd
|
||||
// is reloaded, and the CONSEQUENCE — the agent no longer lists the drive — holds. RED-PROOF (P1/redproof-forget.txt):
|
||||
// with ForgetRetiredDrives' body replaced by `return 0` this test fails (the drive is still listed).
|
||||
func TestForgetRetiredDrives_DecommissionedAndGone_UnitRemovedAndNotListed(t *testing.T) {
|
||||
n, rr, known, _ := runForget(t, forgetCase{intent: IntentDecommissioned, enabled: true}, nil)
|
||||
if n != 1 {
|
||||
t.Fatalf("removed = %d, want 1 (calls %v)", n, rr.calls)
|
||||
}
|
||||
pa := Binaries{}.withDefaults()
|
||||
want := [][]string{
|
||||
{pa.Systemctl, "disable", "--", "mnt-media_a.mount"},
|
||||
{pa.PrivApply, "unit-forget", "mnt-media_a.mount"},
|
||||
{pa.Systemctl, "daemon-reload"},
|
||||
}
|
||||
if !reflect.DeepEqual(rr.calls, want) {
|
||||
t.Fatalf("calls = %v\nwant %v", rr.calls, want)
|
||||
}
|
||||
got, err := known.Known(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("the forgotten drive is still listed: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Already disabled (the Tester 1 shape after a manual disable): no disable call, the removal still runs.
|
||||
func TestForgetRetiredDrives_AlreadyDisabled_NoDisableCall(t *testing.T) {
|
||||
n, rr, _, _ := runForget(t, forgetCase{intent: IntentDecommissioned}, nil)
|
||||
pa := Binaries{}.withDefaults()
|
||||
want := [][]string{{pa.PrivApply, "unit-forget", "mnt-media_a.mount"}, {pa.Systemctl, "daemon-reload"}}
|
||||
if n != 1 || !reflect.DeepEqual(rr.calls, want) {
|
||||
t.Fatalf("n=%d calls=%v want %v", n, rr.calls, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Everything that must KEEP its unit: a decommissioned drive still plugged in (re-enroll is one click — H3), one still
|
||||
// mounted, an enrolled or ejected drive that is merely absent (it comes back), a drive with no recorded intent, and an
|
||||
// unreadable mount table. No privileged call at all, and the drive stays listed where it was listed.
|
||||
// RED-PROOF: with the present-drive check removed, the "present" case calls unit-forget and fails.
|
||||
func TestForgetRetiredDrives_KeepsEveryOtherUnit(t *testing.T) {
|
||||
for name, c := range map[string]forgetCase{
|
||||
"decommissioned but present": {intent: IntentDecommissioned, present: true, enabled: true},
|
||||
"decommissioned but mounted": {intent: IntentDecommissioned, mounted: true},
|
||||
"enrolled and absent": {intent: IntentEnrolled},
|
||||
"ejected and absent": {intent: IntentEjected},
|
||||
"no intent": {intent: IntentNew},
|
||||
"mount table unreadable": {intent: IntentDecommissioned, mountErr: true},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
n, rr, _, unitDir := runForget(t, c, nil)
|
||||
if n != 0 || len(rr.calls) != 0 {
|
||||
t.Fatalf("n=%d calls=%v — this unit must be kept", n, rr.calls)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(unitDir, "mnt-media_a.mount")); err != nil {
|
||||
t.Fatalf("unit file gone: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// No intent store on the host (open failure): nothing is known to be retired, so nothing is forgotten.
|
||||
func TestForgetRetiredDrives_NilIntent_ForgetsNothing(t *testing.T) {
|
||||
unitDir := t.TempDir()
|
||||
writeUnit(t, unitDir, "mnt-media_a.mount", renderMountUnit(MountSpec{Name: "x", UUID: forgetUUID, Where: "/mnt/media_a"}))
|
||||
rr := &forgetRunner{unitDir: unitDir}
|
||||
ops := NewSudoHostOps(SudoHostOpsConfig{Runner: rr, UnitDir: unitDir, StageDir: t.TempDir(), Host: &fakeHostReader{}, Logger: quietLogger()})
|
||||
if n := ops.ForgetRetiredDrives(context.Background(), nil); n != 0 || len(rr.calls) != 0 {
|
||||
t.Fatalf("n=%d calls=%v", n, rr.calls)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,445 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Physical-disk identity of a storage place — felhom.eu architecture/16-storage-physical-identity.md §3.
|
||||
//
|
||||
// A storage place (a PVE storage, an enrolled drive, the guest's data volume, a network share) is
|
||||
// reduced to the PHYSICAL whole disks under it — partition → parent disk, device-mapper / LVM thin →
|
||||
// sysfs slaves (recursive), loop → its backing file → the filesystem holding that file — and each disk
|
||||
// is named by a key the agent trusts: `eui:` for NVMe, `serial:<bus>:<model>:<serial>` from udev,
|
||||
// `wwn:` only when no serial exists and the WWN carries a vendor (OUI ≠ 000000).
|
||||
//
|
||||
// The verdict is AGENT-COMPUTED evidence (`16` §10): it is read from /sys, /run/udev/data and
|
||||
// /proc/mounts only, never from a caller's claim, so a compromised controller cannot make two places
|
||||
// "different disks". `proven` means every disk under the place has a key the agent trusts; anything
|
||||
// else is `unknown` with ONE reason, chosen by precedence (vm > network > usb_bridge > weak_wwn >
|
||||
// no_serial). Nothing is cached: every call reads the current topology, so a disk swapped in the same
|
||||
// bay is re-judged on the next read (the "right before each Tier-2 run" rule is the caller calling).
|
||||
//
|
||||
// Pinned by identity_test.go (fixture /sys + udev trees, one per §3 case) — the spike that proved the
|
||||
// walk on real devices: felhom.eu documentation/audits/storage-identity-2026-10-15/.
|
||||
|
||||
const (
|
||||
IdentityProven = "proven"
|
||||
IdentityUnknown = "unknown"
|
||||
|
||||
ReasonVM = "vm"
|
||||
ReasonNetwork = "network"
|
||||
ReasonUSBBridge = "usb_bridge"
|
||||
ReasonWeakWWN = "weak_wwn"
|
||||
ReasonNoSerial = "no_serial"
|
||||
|
||||
PlaceKindSystem = "system"
|
||||
PlaceKindInternal = "internal"
|
||||
PlaceKindDisk = "disk"
|
||||
PlaceKindPartition = "partition"
|
||||
PlaceKindNetwork = "network"
|
||||
)
|
||||
|
||||
// reasonRank orders the unknown reasons: the first one a place hits, by this precedence, is reported.
|
||||
var reasonRank = map[string]int{ReasonVM: 0, ReasonNetwork: 1, ReasonUSBBridge: 2, ReasonWeakWWN: 3, ReasonNoSerial: 4}
|
||||
|
||||
// PlaceIdentity is what a storage place reports.
|
||||
type PlaceIdentity struct {
|
||||
PhysicalDisks []string // disk keys, sorted, deduplicated
|
||||
Identity string // proven | unknown | "" (not computed — e.g. the device is absent)
|
||||
Reason string // set only when Identity == unknown
|
||||
}
|
||||
|
||||
// Computed reports whether a verdict was reached at all.
|
||||
func (p PlaceIdentity) Computed() bool { return p.Identity != "" }
|
||||
|
||||
// IdentityHost is the read surface the resolver needs. SysfsIdentityHost is production; tests use the
|
||||
// same struct pointed at fixture trees.
|
||||
type IdentityHost interface {
|
||||
Mounts() ([]Mount, error)
|
||||
// KernelName maps a /dev path (symlinks allowed: /dev/mapper/x, /dev/pve/y) to its kernel block name.
|
||||
KernelName(dev string) (string, bool)
|
||||
// Parent returns the parent disk's kernel name when name is a partition.
|
||||
Parent(name string) (string, bool)
|
||||
// Slaves lists /sys/class/block/<name>/slaves.
|
||||
Slaves(name string) []string
|
||||
// LoopBackingFile returns a loop device's backing file.
|
||||
LoopBackingFile(name string) (string, bool)
|
||||
// UdevProps returns the udev E: properties of a block device.
|
||||
UdevProps(name string) map[string]string
|
||||
// Virt returns the VM technology ("none" on bare metal).
|
||||
Virt() string
|
||||
}
|
||||
|
||||
// SysfsIdentityHost reads the real host. Every path is a field so tests can point it at fixtures.
|
||||
type SysfsIdentityHost struct {
|
||||
SysClass string // default /sys/class/block
|
||||
UdevData string // default /run/udev/data
|
||||
ProcMounts string // default /proc/mounts
|
||||
DevResolve func(string) (string, error) // default filepath.EvalSymlinks
|
||||
VirtFn func() string // default `systemd-detect-virt --vm`, read once
|
||||
|
||||
virtOnce sync.Once
|
||||
virt string
|
||||
}
|
||||
|
||||
// NewSysfsIdentityHost returns the production reader.
|
||||
func NewSysfsIdentityHost() *SysfsIdentityHost { return &SysfsIdentityHost{} }
|
||||
|
||||
func (h *SysfsIdentityHost) sysClass() string {
|
||||
if h.SysClass != "" {
|
||||
return h.SysClass
|
||||
}
|
||||
return "/sys/class/block"
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Mounts() ([]Mount, error) {
|
||||
return (&ProcHostReader{ProcMounts: h.ProcMounts}).Mounts()
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) KernelName(dev string) (string, bool) {
|
||||
if !strings.HasPrefix(dev, "/dev/") {
|
||||
return "", false
|
||||
}
|
||||
resolve := h.DevResolve
|
||||
if resolve == nil {
|
||||
resolve = filepath.EvalSymlinks
|
||||
}
|
||||
real := dev
|
||||
if r, err := resolve(dev); err == nil {
|
||||
real = r
|
||||
}
|
||||
name := filepath.Base(real)
|
||||
if _, err := os.Stat(filepath.Join(h.sysClass(), name)); err != nil {
|
||||
return "", false
|
||||
}
|
||||
return name, true
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Parent(name string) (string, bool) {
|
||||
p := filepath.Join(h.sysClass(), name)
|
||||
if _, err := os.Stat(filepath.Join(p, "partition")); err != nil {
|
||||
return "", false
|
||||
}
|
||||
real, err := filepath.EvalSymlinks(p)
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
return filepath.Base(filepath.Dir(real)), true
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Slaves(name string) []string {
|
||||
entries, err := os.ReadDir(filepath.Join(h.sysClass(), name, "slaves"))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) LoopBackingFile(name string) (string, bool) {
|
||||
if !strings.HasPrefix(name, "loop") {
|
||||
return "", false
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(h.sysClass(), name, "loop", "backing_file"))
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
f := strings.TrimSpace(string(b))
|
||||
return f, f != ""
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) UdevProps(name string) map[string]string {
|
||||
out := map[string]string{}
|
||||
devB, err := os.ReadFile(filepath.Join(h.sysClass(), name, "dev"))
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
dir := h.UdevData
|
||||
if dir == "" {
|
||||
dir = "/run/udev/data"
|
||||
}
|
||||
f, err := os.Open(filepath.Join(dir, "b"+strings.TrimSpace(string(devB))))
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if !strings.HasPrefix(line, "E:") {
|
||||
continue
|
||||
}
|
||||
k, v, ok := strings.Cut(line[2:], "=")
|
||||
if ok {
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Virt() string {
|
||||
h.virtOnce.Do(func() {
|
||||
if h.VirtFn != nil {
|
||||
h.virt = h.VirtFn()
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
// --vm: a container is not a reason (the agent runs on the host); exit 1 + "none" on metal.
|
||||
out, _ := exec.CommandContext(ctx, "systemd-detect-virt", "--vm").Output()
|
||||
h.virt = strings.TrimSpace(string(out))
|
||||
if h.virt == "" {
|
||||
h.virt = "unknown"
|
||||
}
|
||||
})
|
||||
return h.virt
|
||||
}
|
||||
|
||||
// IdentityResolver turns devices and host paths into PlaceIdentity.
|
||||
type IdentityResolver struct {
|
||||
Host IdentityHost
|
||||
}
|
||||
|
||||
// NewIdentityResolver returns a resolver over the real host.
|
||||
func NewIdentityResolver() *IdentityResolver {
|
||||
return &IdentityResolver{Host: NewSysfsIdentityHost()}
|
||||
}
|
||||
|
||||
var networkFSTypes = map[string]bool{
|
||||
"nfs": true, "nfs4": true, "cifs": true, "smb3": true, "fuse.sshfs": true, "9p": true, "autofs": true, "ceph": true, "glusterfs": true,
|
||||
}
|
||||
|
||||
// NetworkPlace is the fixed verdict for a network share or a remote store.
|
||||
func NetworkPlace() PlaceIdentity {
|
||||
return PlaceIdentity{PhysicalDisks: []string{}, Identity: IdentityUnknown, Reason: ReasonNetwork}
|
||||
}
|
||||
|
||||
// MountFor returns the mount that holds path (longest mountpoint prefix; the LAST entry wins on an equal
|
||||
// mountpoint, which is the one stacked on top — an automount's real filesystem over its autofs trigger).
|
||||
func MountFor(mounts []Mount, path string) (Mount, bool) {
|
||||
p := filepath.Clean(path)
|
||||
var best Mount
|
||||
found := false
|
||||
for _, m := range mounts {
|
||||
mp := filepath.Clean(m.MountPoint)
|
||||
if p != mp && !strings.HasPrefix(p, strings.TrimSuffix(mp, "/")+"/") {
|
||||
continue
|
||||
}
|
||||
if !found || len(mp) >= len(filepath.Clean(best.MountPoint)) {
|
||||
best, found = m, true
|
||||
}
|
||||
}
|
||||
return best, found
|
||||
}
|
||||
|
||||
// HostPath resolves the filesystem holding a HOST path.
|
||||
func (r *IdentityResolver) HostPath(path string) PlaceIdentity {
|
||||
return r.hostPath(path, 0)
|
||||
}
|
||||
|
||||
func (r *IdentityResolver) hostPath(path string, depth int) PlaceIdentity {
|
||||
if r == nil || r.Host == nil || depth > 4 {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
mounts, err := r.Host.Mounts()
|
||||
if err != nil {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
m, ok := MountFor(mounts, path)
|
||||
if !ok {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
if networkFSTypes[m.FSType] || !strings.HasPrefix(m.Device, "/dev/") {
|
||||
return NetworkPlace()
|
||||
}
|
||||
return r.device(m.Device, depth)
|
||||
}
|
||||
|
||||
// Device resolves a block device (a /dev path, symlinks allowed).
|
||||
func (r *IdentityResolver) Device(dev string) PlaceIdentity {
|
||||
return r.device(dev, 0)
|
||||
}
|
||||
|
||||
func (r *IdentityResolver) device(dev string, depth int) PlaceIdentity {
|
||||
if r == nil || r.Host == nil || dev == "" {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
name, ok := r.Host.KernelName(dev)
|
||||
if !ok {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
keys := map[string]bool{}
|
||||
reason := ""
|
||||
if !r.walk(name, depth, map[string]bool{}, keys, &reason) {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
out := PlaceIdentity{PhysicalDisks: sortedKeys(keys), Identity: IdentityProven}
|
||||
if reason != "" {
|
||||
out.Identity, out.Reason = IdentityUnknown, reason
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// walk adds the keys of every physical disk under name; ok=false when a branch cannot be grounded.
|
||||
func (r *IdentityResolver) walk(name string, depth int, seen, keys map[string]bool, reason *string) bool {
|
||||
if seen[name] {
|
||||
return true // a diamond (a thin pool's tmeta and tdata on one partition): already counted
|
||||
}
|
||||
if len(seen) > 64 {
|
||||
return false
|
||||
}
|
||||
seen[name] = true
|
||||
if parent, ok := r.Host.Parent(name); ok {
|
||||
return r.walk(parent, depth, seen, keys, reason)
|
||||
}
|
||||
if file, ok := r.Host.LoopBackingFile(name); ok {
|
||||
p := r.hostPath(file, depth+1)
|
||||
if !p.Computed() {
|
||||
return false
|
||||
}
|
||||
for _, k := range p.PhysicalDisks {
|
||||
keys[k] = true
|
||||
}
|
||||
noteReason(reason, p.Reason)
|
||||
return true
|
||||
}
|
||||
if slaves := r.Host.Slaves(name); len(slaves) > 0 {
|
||||
for _, s := range slaves {
|
||||
if !r.walk(s, depth, seen, keys, reason) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
if strings.HasPrefix(name, "dm-") || strings.HasPrefix(name, "md") || strings.HasPrefix(name, "loop") {
|
||||
return false // a virtual device with nothing beneath it cannot be grounded
|
||||
}
|
||||
key, why := r.diskKey(name)
|
||||
keys[key] = true
|
||||
noteReason(reason, why)
|
||||
return true
|
||||
}
|
||||
|
||||
func noteReason(cur *string, why string) {
|
||||
if why == "" {
|
||||
return
|
||||
}
|
||||
if *cur == "" || reasonRank[why] < reasonRank[*cur] {
|
||||
*cur = why
|
||||
}
|
||||
}
|
||||
|
||||
// diskKey names one whole disk and says why it is not trusted ("" when it is).
|
||||
func (r *IdentityResolver) diskKey(name string) (key, reason string) {
|
||||
p := r.Host.UdevProps(name)
|
||||
serial := strings.TrimSpace(p["ID_SERIAL_SHORT"])
|
||||
model := strings.TrimSpace(p["ID_MODEL"])
|
||||
bus := strings.TrimSpace(p["ID_BUS"])
|
||||
wwn := strings.ToLower(strings.TrimSpace(p["ID_WWN"]))
|
||||
if bus == "" && strings.HasPrefix(name, "nvme") {
|
||||
bus = "nvme"
|
||||
}
|
||||
usb := bus == "usb" || strings.Contains(p["DEVPATH"], "/usb")
|
||||
switch {
|
||||
case strings.HasPrefix(name, "nvme") && (strings.HasPrefix(wwn, "eui.") || strings.HasPrefix(wwn, "nvme.")):
|
||||
key = "eui:" + strings.TrimPrefix(strings.TrimPrefix(wwn, "eui."), "nvme.")
|
||||
case serial != "":
|
||||
key = "serial:" + bus + ":" + model + ":" + serial
|
||||
if usb && FakeBridgeSerial(serial) {
|
||||
reason = ReasonUSBBridge
|
||||
}
|
||||
case wwn != "" && WWNHasVendor(wwn):
|
||||
key = "wwn:" + wwn
|
||||
case wwn != "":
|
||||
key, reason = "wwn:"+wwn, ReasonWeakWWN
|
||||
default:
|
||||
key, reason = "dev:"+name, ReasonNoSerial
|
||||
}
|
||||
if v := r.Host.Virt(); v != "none" {
|
||||
reason = ReasonVM // inside a VM a virtual disk says nothing about the host disk (spike cases 5–6)
|
||||
}
|
||||
return key, reason
|
||||
}
|
||||
|
||||
// WWNHasVendor reports whether an NAA-5/6 WWN carries a registered vendor (OUI ≠ 000000). Other NAA
|
||||
// formats are taken as they are. Measured: the N100's AirDisk SSD reports 0x5000000123456f1f.
|
||||
func WWNHasVendor(wwn string) bool {
|
||||
h := strings.TrimPrefix(strings.ToLower(wwn), "0x")
|
||||
if len(h) < 7 {
|
||||
return false
|
||||
}
|
||||
if h[0] == '5' || h[0] == '6' {
|
||||
return h[1:7] != "000000"
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// FakeBridgeSerial reports a serial that a USB bridge reports for any disk: too short, one repeated
|
||||
// character, or a known placeholder. UNPROVEN on a real enclosure (R-1077) — the rule is the design's.
|
||||
func FakeBridgeSerial(s string) bool {
|
||||
s = strings.ToUpper(strings.TrimSpace(s))
|
||||
if len(s) < 6 {
|
||||
return true
|
||||
}
|
||||
if strings.Count(s, s[:1]) == len(s) {
|
||||
return true
|
||||
}
|
||||
switch s {
|
||||
case "123456789", "1234567890", "123456789ABC", "0123456789ABCDEF", "1234567890ABCDEF", "DEADBEEF":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func sortedKeys(m map[string]bool) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// Relation compares two places (`16` §4). A SHARED disk key is a positive fact, so two places on one disk
|
||||
// are "same, proven" even when that disk is unknown (inside a VM, two parts of one virtual disk). Otherwise
|
||||
// proven is true only when both verdicts are proven; different then means "no disk in common". When proven
|
||||
// is false, different only says the known keys do not overlap — callers must act on proven first.
|
||||
// reason: "" (proven, different), "same_disk", "unmapped" (a side has no verdict), or the unknown reason.
|
||||
func Relation(a, b PlaceIdentity) (different, proven bool, reason string) {
|
||||
if !a.Computed() || !b.Computed() {
|
||||
return false, false, "unmapped"
|
||||
}
|
||||
set := map[string]bool{}
|
||||
for _, k := range a.PhysicalDisks {
|
||||
set[k] = true
|
||||
}
|
||||
shared := false
|
||||
for _, k := range b.PhysicalDisks {
|
||||
if set[k] {
|
||||
shared = true
|
||||
}
|
||||
}
|
||||
if shared {
|
||||
return false, true, "same_disk"
|
||||
}
|
||||
if a.Identity == IdentityProven && b.Identity == IdentityProven {
|
||||
return true, true, ""
|
||||
}
|
||||
why := a.Reason
|
||||
if why == "" || (b.Reason != "" && reasonRank[b.Reason] < reasonRank[why]) {
|
||||
why = b.Reason
|
||||
}
|
||||
return true, false, why
|
||||
}
|
||||
@@ -0,0 +1,257 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Fixture /sys/class/block + /run/udev/data + /proc/mounts trees, one per `16` §3 case. The tree mimics the
|
||||
// kernel: /sys/devices/.../<disk>/<part> real dirs (a partition carries a "partition" file), /sys/class/block
|
||||
// symlinks, a "dev" file per node, udev "E:" lines keyed by major:minor. The cases are the ones the spike
|
||||
// measured on real devices (felhom.eu documentation/audits/storage-identity-2026-10-15/).
|
||||
|
||||
type fx struct {
|
||||
t *testing.T
|
||||
root string
|
||||
minor int
|
||||
mounts []string
|
||||
links map[string]string // /dev path -> kernel name
|
||||
virt string
|
||||
}
|
||||
|
||||
func newFx(t *testing.T) *fx {
|
||||
t.Helper()
|
||||
r := t.TempDir()
|
||||
for _, d := range []string{"sys/class/block", "sys/devices", "udev"} {
|
||||
if err := os.MkdirAll(filepath.Join(r, d), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return &fx{t: t, root: r, links: map[string]string{}, virt: "none"}
|
||||
}
|
||||
|
||||
func (f *fx) write(p, s string) {
|
||||
f.t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte(s), 0o644); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// node makes a block node. parent "" → a top-level device. udev lines are "KEY=VAL".
|
||||
func (f *fx) node(name, parent string, udev ...string) {
|
||||
f.t.Helper()
|
||||
real := filepath.Join(f.root, "sys/devices", name)
|
||||
if parent != "" {
|
||||
pr, _ := filepath.EvalSymlinks(filepath.Join(f.root, "sys/class/block", parent))
|
||||
real = filepath.Join(pr, name)
|
||||
f.write(filepath.Join(real, "partition"), "1\n")
|
||||
}
|
||||
f.minor++
|
||||
dev := "8:" + strconv.Itoa(f.minor)
|
||||
f.write(filepath.Join(real, "dev"), dev+"\n")
|
||||
if err := os.Symlink(real, filepath.Join(f.root, "sys/class/block", name)); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
var b strings.Builder
|
||||
for _, u := range udev {
|
||||
b.WriteString("E:" + u + "\n")
|
||||
}
|
||||
f.write(filepath.Join(f.root, "udev", "b"+dev), b.String())
|
||||
f.links["/dev/"+name] = name
|
||||
}
|
||||
|
||||
func (f *fx) slaves(name string, slaves ...string) {
|
||||
f.t.Helper()
|
||||
real, _ := filepath.EvalSymlinks(filepath.Join(f.root, "sys/class/block", name))
|
||||
for _, s := range slaves {
|
||||
if err := os.MkdirAll(filepath.Join(real, "slaves", s), 0o755); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fx) loop(name, file string) {
|
||||
real, _ := filepath.EvalSymlinks(filepath.Join(f.root, "sys/class/block", name))
|
||||
f.write(filepath.Join(real, "loop", "backing_file"), file+"\n")
|
||||
}
|
||||
|
||||
func (f *fx) mount(dev, mp, fstype string) {
|
||||
f.mounts = append(f.mounts, dev+" "+mp+" "+fstype+" rw 0 0")
|
||||
}
|
||||
|
||||
func (f *fx) resolver() *IdentityResolver {
|
||||
f.write(filepath.Join(f.root, "mounts"), strings.Join(f.mounts, "\n")+"\n")
|
||||
return &IdentityResolver{Host: &SysfsIdentityHost{
|
||||
SysClass: filepath.Join(f.root, "sys/class/block"),
|
||||
UdevData: filepath.Join(f.root, "udev"),
|
||||
ProcMounts: filepath.Join(f.root, "mounts"),
|
||||
DevResolve: func(p string) (string, error) {
|
||||
if n, ok := f.links[p]; ok {
|
||||
return "/dev/" + n, nil
|
||||
}
|
||||
return p, nil
|
||||
},
|
||||
VirtFn: func() string { return f.virt },
|
||||
}}
|
||||
}
|
||||
|
||||
var sanDisk = []string{"ID_BUS=ata", "ID_MODEL=SanDisk_X600", "ID_SERIAL_SHORT=182195804614", "ID_WWN=0x5001b448b60a9d6e"}
|
||||
|
||||
// oneDiskLVM builds the PVE default layout on one SATA SSD: sda3 → pve-root, pve-data tmeta/tdata → tpool →
|
||||
// a guest volume (the shape of all three test boxes).
|
||||
func oneDiskLVM(f *fx) {
|
||||
f.node("sda", "", sanDisk...)
|
||||
f.node("sda3", "sda")
|
||||
for _, d := range []string{"dm-1", "dm-2", "dm-3", "dm-4", "dm-6"} {
|
||||
f.node(d, "")
|
||||
}
|
||||
f.slaves("dm-1", "sda3")
|
||||
f.slaves("dm-2", "sda3")
|
||||
f.slaves("dm-3", "sda3")
|
||||
f.slaves("dm-4", "dm-2", "dm-3")
|
||||
f.slaves("dm-6", "dm-4")
|
||||
f.links["/dev/mapper/pve-root"] = "dm-1"
|
||||
f.links["/dev/mapper/pve-data-tpool"] = "dm-4"
|
||||
f.links["/dev/pve/vm-9201-disk-1"] = "dm-6"
|
||||
f.mount("/dev/mapper/pve-root", "/", "ext4")
|
||||
}
|
||||
|
||||
func TestIdentity_OneDiskLVM_AllPlacesSameProvenDisk(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
r := f.resolver()
|
||||
want := []string{"serial:ata:SanDisk_X600:182195804614"}
|
||||
for _, p := range []PlaceIdentity{r.HostPath("/var/lib/vz"), r.Device("/dev/mapper/pve-data-tpool"), r.Device("/dev/pve/vm-9201-disk-1")} {
|
||||
if p.Identity != IdentityProven || !reflect.DeepEqual(p.PhysicalDisks, want) {
|
||||
t.Fatalf("want proven %v, got %+v", want, p)
|
||||
}
|
||||
}
|
||||
diff, proven, why := Relation(r.HostPath("/var/lib/vz"), r.Device("/dev/pve/vm-9201-disk-1"))
|
||||
if diff || !proven || why != "same_disk" {
|
||||
t.Fatalf("root vs guest volume: want same+proven, got diff=%v proven=%v why=%q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_TwoDisks_ProvenDifferent(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.node("sdb", "", "ID_BUS=ata", "ID_MODEL=TOSHIBA_MQ04ABF100", "ID_SERIAL_SHORT=65NOP3HDT", "ID_WWN=0x5000039ddb108568", "DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-3/block/sdb")
|
||||
f.mount("/dev/sdb", "/mnt/hdd_1", "ext4")
|
||||
f.mount("/dev/sdb", "/mnt/felhom-drives/hdd_1", "ext4") // the felhom-data bind shows the source device
|
||||
r := f.resolver()
|
||||
drive := r.HostPath("/mnt/felhom-drives/hdd_1/userdata")
|
||||
if drive.Identity != IdentityProven || drive.PhysicalDisks[0] != "serial:ata:TOSHIBA_MQ04ABF100:65NOP3HDT" {
|
||||
t.Fatalf("USB drive passing its identity: want proven, got %+v", drive)
|
||||
}
|
||||
diff, proven, why := Relation(r.Device("/dev/pve/vm-9201-disk-1"), drive)
|
||||
if !diff || !proven || why != "" {
|
||||
t.Fatalf("system vs drive: want different+proven, got %v %v %q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_TwoPartitionsOneDisk_Same(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("sdc", "", "ID_BUS=usb", "ID_MODEL=Portable", "ID_SERIAL_SHORT=NAAB7C21")
|
||||
f.node("sdc1", "sdc")
|
||||
f.node("sdc2", "sdc")
|
||||
f.mount("/dev/sdc1", "/mnt/a", "ext4")
|
||||
f.mount("/dev/sdc2", "/mnt/b", "ext4")
|
||||
r := f.resolver()
|
||||
diff, proven, why := Relation(r.HostPath("/mnt/a"), r.HostPath("/mnt/b"))
|
||||
if diff || !proven || why != "same_disk" {
|
||||
t.Fatalf("two partitions of one disk: want same+proven, got %v %v %q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_LoopPartitionsOnAFile_ResolveToTheFilesDisk(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("nvme0n1", "", "ID_MODEL=KXG50PNV1T02", "ID_SERIAL_SHORT=58BS11AFT8MQ", "ID_WWN=eui.000000000000001000080d020046c07b")
|
||||
f.mount("/dev/nvme0n1", "/mnt/hdd_1", "ext4")
|
||||
f.node("loop2", "")
|
||||
f.loop("loop2", "/mnt/hdd_1/scratch.img")
|
||||
f.node("loop2p1", "loop2") // the partition step must run BEFORE the loop step (the spike's probe bug)
|
||||
f.mount("/dev/loop2p1", "/tmp/scrA", "ext4")
|
||||
r := f.resolver()
|
||||
p := r.HostPath("/tmp/scrA")
|
||||
if p.Identity != IdentityProven || !reflect.DeepEqual(p.PhysicalDisks, []string{"eui:000000000000001000080d020046c07b"}) {
|
||||
t.Fatalf("loop partition: want the NVMe's eui, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_WWNWithoutVendor(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("sda", "", "ID_BUS=ata", "ID_MODEL=AirDisk_512GB_SSD", "ID_SERIAL_SHORT=QDF922W009654S30EX", "ID_WWN=0x5000000123456f1f")
|
||||
f.node("sdb", "", "ID_BUS=ata", "ID_MODEL=Cheap", "ID_WWN=0x5000000123456f1f")
|
||||
r := f.resolver()
|
||||
if p := r.Device("/dev/sda"); p.Identity != IdentityProven || p.PhysicalDisks[0] != "serial:ata:AirDisk_512GB_SSD:QDF922W009654S30EX" {
|
||||
t.Fatalf("a real serial wins over a vendor-less WWN: got %+v", p)
|
||||
}
|
||||
if p := r.Device("/dev/sdb"); p.Identity != IdentityUnknown || p.Reason != ReasonWeakWWN {
|
||||
t.Fatalf("no serial + vendor-less WWN: want unknown/weak_wwn, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_NoSerial(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("vdb", "", "ID_MODEL=")
|
||||
if p := f.resolver().Device("/dev/vdb"); p.Identity != IdentityUnknown || p.Reason != ReasonNoSerial {
|
||||
t.Fatalf("want unknown/no_serial, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_InsideAVM_EverythingUnknown(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.virt = "kvm"
|
||||
f.node("sdb", "", "ID_BUS=scsi", "ID_MODEL=QEMU_HARDDISK", "ID_SERIAL_SHORT=drive-scsi1")
|
||||
f.mount("/dev/sdb", "/mnt/felhom-drives/adatlemez", "ext4")
|
||||
r := f.resolver()
|
||||
a, b := r.Device("/dev/pve/vm-9201-disk-1"), r.HostPath("/mnt/felhom-drives/adatlemez")
|
||||
for _, p := range []PlaceIdentity{a, b} {
|
||||
if p.Identity != IdentityUnknown || p.Reason != ReasonVM {
|
||||
t.Fatalf("inside a VM: want unknown/vm, got %+v", p)
|
||||
}
|
||||
}
|
||||
diff, proven, why := Relation(a, b)
|
||||
if proven || why != ReasonVM || !diff {
|
||||
t.Fatalf("two virtual disks: want not proven (vm), got %v %v %q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_USBBridgeSerial(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("sdd", "", "ID_BUS=usb", "ID_MODEL=External_USB3.0", "ID_SERIAL_SHORT=000000000000")
|
||||
f.node("sde", "", "ID_BUS=usb", "ID_MODEL=External_USB3.0", "ID_SERIAL_SHORT=20231106A1B2")
|
||||
r := f.resolver()
|
||||
if p := r.Device("/dev/sdd"); p.Identity != IdentityUnknown || p.Reason != ReasonUSBBridge {
|
||||
t.Fatalf("a bridge placeholder serial: want unknown/usb_bridge, got %+v", p)
|
||||
}
|
||||
if p := r.Device("/dev/sde"); p.Identity != IdentityProven {
|
||||
t.Fatalf("a USB disk with a plausible serial: want proven, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_NetworkShareAndUnmapped(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.mount("//192.168.0.123/TesztFilmek", "/mnt/felhom-drives/omv-filmek", "cifs")
|
||||
r := f.resolver()
|
||||
n := r.HostPath("/mnt/felhom-drives/omv-filmek/x")
|
||||
if n.Identity != IdentityUnknown || n.Reason != ReasonNetwork || len(n.PhysicalDisks) != 0 {
|
||||
t.Fatalf("network share: want unknown/network with no disks, got %+v", n)
|
||||
}
|
||||
_, proven, why := Relation(r.HostPath("/"), n)
|
||||
if proven || why != ReasonNetwork {
|
||||
t.Fatalf("system vs network: want not proven (network), got %v %q", proven, why)
|
||||
}
|
||||
if _, proven, why := Relation(PlaceIdentity{}, n); proven || why != "unmapped" {
|
||||
t.Fatalf("an uncomputed side: want unmapped, got %v %q", proven, why)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The root mkfs wrapper's own tests (configs/test_felhom_mkfs_guarded.py, agent v0.158.0 — a sibling partition that
|
||||
// is Felhom's own drive no longer blocks formatting the part next to it; every other mount still refuses) run with
|
||||
// the Go suite.
|
||||
func TestMkfsGuardedPythonSuite(t *testing.T) {
|
||||
py, err := exec.LookPath("python3")
|
||||
if err != nil {
|
||||
t.Skip("python3 not found")
|
||||
}
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
script := filepath.Join(filepath.Dir(file), "..", "..", "configs", "test_felhom_mkfs_guarded.py")
|
||||
out, err := exec.Command(py, script).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("wrapper suite failed: %v\n%s", err, out)
|
||||
}
|
||||
}
|
||||
@@ -40,6 +40,10 @@ type Observer struct {
|
||||
// R-672 (v0.133.0): a thin pool crossing thinPoolAlarmFraction (data OR metadata) requests an
|
||||
// out-of-band host report at once, so the hub's storage-fill alarm sees it in seconds instead of at
|
||||
// the next 15-minute report. Rising edge per pool; re-armed below thinPoolRearmFraction.
|
||||
// ident (agent v0.157.0, `16` §3) computes each target's physical-disk identity. nil → the fields stay
|
||||
// empty (the pre-0.157 shape), which every consumer reads as "not computed".
|
||||
ident *IdentityResolver
|
||||
|
||||
highMu sync.Mutex
|
||||
high map[string]bool
|
||||
onThinHigh func()
|
||||
@@ -50,6 +54,51 @@ const (
|
||||
thinPoolRearmFraction = 0.85
|
||||
)
|
||||
|
||||
// SetIdentity wires the physical-disk identity resolver (main: NewIdentityResolver()).
|
||||
func (o *Observer) SetIdentity(r *IdentityResolver) { o.ident = r }
|
||||
|
||||
// placeIdentity fills a target's identity fields (`16` §3). Network and remote stores are fixed
|
||||
// "unknown/network"; a dir storage is judged only on its OWN mount (the same rule as backing_device —
|
||||
// never the containing filesystem, so an absent drive is "not computed", never the root disk); the
|
||||
// builtin local and an lvmthin pool are the system disk(s) they sit on.
|
||||
func (o *Observer) placeIdentity(tgt *hub.StorageTarget, s proxmox.Storage) {
|
||||
if o.ident == nil {
|
||||
return
|
||||
}
|
||||
var p PlaceIdentity
|
||||
switch tgt.Type {
|
||||
case hub.StorageTypeNFS, hub.StorageTypeCIFS, hub.StorageTypePBS:
|
||||
tgt.Kind = PlaceKindNetwork
|
||||
p = NetworkPlace()
|
||||
case hub.StorageTypeLocal:
|
||||
tgt.Kind = PlaceKindSystem
|
||||
if s.Path != "" {
|
||||
p = o.ident.HostPath(s.Path)
|
||||
}
|
||||
case hub.StorageTypeLVMThin:
|
||||
tgt.Kind = PlaceKindSystem
|
||||
if s.VGName != "" && s.ThinPool != "" {
|
||||
p = o.ident.Device("/dev/mapper/" + dmEscape(s.VGName) + "-" + dmEscape(s.ThinPool) + "-tpool")
|
||||
}
|
||||
default:
|
||||
if tgt.BackingDevice == "" {
|
||||
return // not its own mount (absent, or a dir on root): not computed
|
||||
}
|
||||
tgt.Kind = PlaceKindDisk
|
||||
if _, known := wholeDiskOf(tgt.BackingDevice); known && !reWholeDisk.MatchString(tgt.BackingDevice) {
|
||||
tgt.Kind = PlaceKindPartition
|
||||
}
|
||||
p = o.ident.Device(tgt.BackingDevice)
|
||||
}
|
||||
if !p.Computed() {
|
||||
return
|
||||
}
|
||||
tgt.PhysicalDisks, tgt.Identity, tgt.IdentityReason = p.PhysicalDisks, p.Identity, p.Reason
|
||||
}
|
||||
|
||||
// dmEscape doubles dashes the way device-mapper names an LV (pve + vm-1-disk-0 → pve-vm--1--disk--0).
|
||||
func dmEscape(s string) string { return strings.ReplaceAll(s, "-", "--") }
|
||||
|
||||
// SetThinHighTrigger wires the out-of-band report request (main: the storage trigger channel).
|
||||
func (o *Observer) SetThinHighTrigger(f func()) { o.onThinHigh = f }
|
||||
|
||||
@@ -303,6 +352,9 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
|
||||
o.noteThinFill(s.Storage+"/data", frac)
|
||||
}
|
||||
|
||||
// Physical-disk identity (`16` §3) — after typ/backing are settled; never from a caller's claim.
|
||||
o.placeIdentity(&tgt, s)
|
||||
|
||||
// SMART-only device hint (v0.95.0): a dir-storage that lives INSIDE a shared filesystem (the
|
||||
// builtin `local` on the LVM root) has empty backing by design, yet we can still read the PHYSICAL
|
||||
// disk's SMART by resolving its containing filesystem. Gated to catDir + no own backing + reachable,
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// The report rows carry the `16` §3 identity per storage type: local + lvmthin → the system disk; a dir
|
||||
// on its own partition → that disk, kind "partition"; nfs/pbs → network; an UNMOUNTED dir → not computed
|
||||
// (never the root disk — the same rule as backing_device).
|
||||
func TestObserve_FillsPhysicalDiskIdentity(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.node("sdb", "", "ID_BUS=ata", "ID_MODEL=TOSHIBA_MQ04ABF100", "ID_SERIAL_SHORT=65NOP3HDT")
|
||||
f.node("sdb1", "sdb")
|
||||
f.mount("/dev/sdb1", "/mnt/usb-backup", "ext4")
|
||||
res := f.resolver()
|
||||
|
||||
api := &fakeStorageAPI{
|
||||
node: "demo",
|
||||
cluster: []proxmox.Storage{
|
||||
{Storage: "local", Type: "dir", Path: "/var/lib/vz"},
|
||||
{Storage: "local-lvm", Type: "lvmthin", VGName: "pve", ThinPool: "data"},
|
||||
{Storage: "usb-backup", Type: "dir", Path: "/mnt/usb-backup"},
|
||||
{Storage: "gone", Type: "dir", Path: "/mnt/gone"},
|
||||
{Storage: "nfs-arch", Type: "nfs", Server: "10.0.0.5", Export: "/x"},
|
||||
},
|
||||
nodeSt: []proxmox.Storage{
|
||||
{Storage: "local", Type: "dir", Path: "/var/lib/vz", Active: 1},
|
||||
{Storage: "local-lvm", Type: "lvmthin", Active: 1},
|
||||
{Storage: "usb-backup", Type: "dir", Path: "/mnt/usb-backup", Active: 1},
|
||||
{Storage: "gone", Type: "dir", Path: "/mnt/gone", Active: 1},
|
||||
{Storage: "nfs-arch", Type: "nfs", Active: 1},
|
||||
},
|
||||
}
|
||||
host := &fakeHostReader{
|
||||
mounts: []Mount{
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"},
|
||||
{Device: "/dev/sdb1", MountPoint: "/mnt/usb-backup", FSType: "ext4"},
|
||||
},
|
||||
exists: map[string]bool{"/dev/sdb1": true, "/dev/mapper/pve-root": true},
|
||||
}
|
||||
ob := NewObserver(api, host, nil, quietLogger())
|
||||
ob.SetIdentity(res)
|
||||
got, err := ob.Observe(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := byName(got)
|
||||
ssd := []string{"serial:ata:SanDisk_X600:182195804614"}
|
||||
for _, n := range []string{"local", "local-lvm"} {
|
||||
if tg := m[n]; tg.Kind != "system" || tg.Identity != "proven" || !reflect.DeepEqual(tg.PhysicalDisks, ssd) {
|
||||
t.Errorf("%s: want system/proven on the SSD, got kind=%q id=%q disks=%v", n, tg.Kind, tg.Identity, tg.PhysicalDisks)
|
||||
}
|
||||
}
|
||||
if tg := m["usb-backup"]; tg.Kind != "partition" || tg.Identity != "proven" || tg.PhysicalDisks[0] != "serial:ata:TOSHIBA_MQ04ABF100:65NOP3HDT" {
|
||||
t.Errorf("usb-backup: got %+v", tg)
|
||||
}
|
||||
if tg := m["nfs-arch"]; tg.Kind != "network" || tg.IdentityReason != "network" {
|
||||
t.Errorf("nfs: got kind=%q reason=%q", tg.Kind, tg.IdentityReason)
|
||||
}
|
||||
if tg := m["gone"]; tg.Identity != "" || len(tg.PhysicalDisks) != 0 {
|
||||
t.Errorf("an unmounted dir must not be computed (never root's disk), got %+v", tg)
|
||||
}
|
||||
// Without a resolver the fields stay empty (the pre-0.157 shape).
|
||||
plain, _ := NewObserver(api, host, nil, quietLogger()).Observe(context.Background())
|
||||
if tg := byName(plain)["local"]; tg.Identity != "" || tg.Kind != "" {
|
||||
t.Errorf("no resolver: want empty identity, got %+v", tg)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The data-bearing partition path binds the disk id + the part number + the fs-UUID (felhom.eu architecture/16 §6,
|
||||
// agent v0.158.0). The wipe re-resolve (localapi antiRetargetResolveExpect) formats only when resolve(id) succeeds AND
|
||||
// DeviceDurableID(that device) == id — the round trip these tests pin, in the three ways a target can move.
|
||||
|
||||
type partFixture struct{ base, root string }
|
||||
|
||||
func newPartFixture(t *testing.T) partFixture {
|
||||
t.Helper()
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("symlink fixtures run on Linux")
|
||||
}
|
||||
f := partFixture{base: t.TempDir()}
|
||||
f.root = filepath.Join(f.base, "disk")
|
||||
for _, d := range []string{"sdb", "sdb1", "sdb2", "sdc2"} {
|
||||
if err := os.WriteFile(filepath.Join(f.base, d), []byte("x"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
os.MkdirAll(filepath.Join(f.root, "by-id"), 0o755)
|
||||
os.MkdirAll(filepath.Join(f.root, "by-uuid"), 0o755)
|
||||
withDevDiskRoot(t, f.root)
|
||||
return f
|
||||
}
|
||||
|
||||
func (f partFixture) link(t *testing.T, sub, name, dev string) {
|
||||
t.Helper()
|
||||
p := filepath.Join(f.root, sub, name)
|
||||
os.Remove(p)
|
||||
if err := os.Symlink(filepath.Join(f.base, dev), p); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func (f partFixture) dev(name string) string { return filepath.Join(f.base, name) }
|
||||
|
||||
func (f partFixture) standard(t *testing.T) {
|
||||
f.link(t, "by-id", "wwn-0x5000aaaa", "sdb")
|
||||
f.link(t, "by-id", "ata-TOSHIBA_MQ04_65NOP-part1", "sdb1")
|
||||
f.link(t, "by-id", "wwn-0x5000aaaa-part1", "sdb1")
|
||||
f.link(t, "by-id", "wwn-0x5000aaaa-part2", "sdb2")
|
||||
f.link(t, "by-uuid", "1111-films", "sdb2")
|
||||
}
|
||||
|
||||
func roundTrip(id string) (string, bool) {
|
||||
dev, err := ResolveDurableDevice(id)
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
again, err := DeviceDurableID(dev)
|
||||
return dev, err == nil && again == id
|
||||
}
|
||||
|
||||
func TestPartitionWipeID_BindsDiskPartAndFilesystem(t *testing.T) {
|
||||
f := newPartFixture(t)
|
||||
f.standard(t)
|
||||
id, err := DeviceDurableID(f.dev("sdb2"))
|
||||
if err != nil || id != "byid:wwn-0x5000aaaa-part2+fsuuid:1111-films" {
|
||||
t.Fatalf("a part with a filesystem: want disk id + part + fs-UUID, got %q %v", id, err)
|
||||
}
|
||||
if dev, ok := roundTrip(id); !ok || dev != f.dev("sdb2") {
|
||||
t.Fatalf("the unchanged part must round-trip, got %q %v", dev, ok)
|
||||
}
|
||||
// A blank part (no filesystem) and a whole disk keep the plain by-id binding.
|
||||
if id1, _ := DeviceDurableID(f.dev("sdb1")); id1 != "byid:wwn-0x5000aaaa-part1" {
|
||||
t.Fatalf("a blank part: want the plain part link, got %q", id1)
|
||||
}
|
||||
if idDisk, _ := DeviceDurableID(f.dev("sdb")); idDisk != "byid:wwn-0x5000aaaa" {
|
||||
t.Fatalf("a whole disk keeps today's id, got %q", idDisk)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPartitionWipeID_ReformattedPartRefuses(t *testing.T) {
|
||||
f := newPartFixture(t)
|
||||
f.standard(t)
|
||||
id, _ := DeviceDurableID(f.dev("sdb2"))
|
||||
os.Remove(filepath.Join(f.root, "by-uuid", "1111-films"))
|
||||
f.link(t, "by-uuid", "2222-other", "sdb2") // someone reformatted the part since the household confirmed
|
||||
if _, ok := roundTrip(id); ok {
|
||||
t.Fatalf("a reformatted part must not round-trip (the wipe would hit a different filesystem)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPartitionWipeID_LinkMovedToAnotherDiskRefuses(t *testing.T) {
|
||||
f := newPartFixture(t)
|
||||
f.standard(t)
|
||||
id, _ := DeviceDurableID(f.dev("sdb2"))
|
||||
f.link(t, "by-id", "wwn-0x5000aaaa-part2", "sdc2") // a different node took the part link; the fs is still sdb2
|
||||
if _, err := ResolveDurableDevice(id); err == nil || !strings.Contains(err.Error(), "different devices") {
|
||||
t.Fatalf("the part link and its filesystem on different devices must refuse, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPartitionWipeID_RefusesAForgedCompound(t *testing.T) {
|
||||
f := newPartFixture(t)
|
||||
f.standard(t)
|
||||
f.link(t, "by-uuid", "3333-whole", "sdb") // a filesystem straight on the whole disk: both halves name sdb
|
||||
for _, id := range []string{"byid:wwn-0x5000aaaa+fsuuid:3333-whole", "byid:wwn-0x5000aaaa-part2+fsuuid:../x"} {
|
||||
if _, err := ResolveDurableDevice(id); err == nil {
|
||||
t.Fatalf("%q must be refused", id)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Partitions as places (agent v0.158.0) — felhom.eu architecture/16-storage-physical-identity.md §6.
|
||||
//
|
||||
// A disk that passed the WHOLE-disk unclaimed filter (classifyClaim: not the system disk, no LVM/ZFS/RAID/LUKS/swap
|
||||
// member on any node, no foreign mount on any node) also lists its partitions, each with its own state:
|
||||
//
|
||||
// empty no filesystem, no children → „format this part" (self-serve: the agent's probe says blank) —
|
||||
// only while NO part of the disk is mounted (Formattable)
|
||||
// filesystem ext4/xfs (mountable) or another fs → „use this part" when mountable; „format this part" only through
|
||||
// the customer-confirmed data-bearing path (the agent's own probe)
|
||||
// in_use already Felhom's own drive → shown, never offered (it is a registered drive already)
|
||||
//
|
||||
// A partition with children (LVM, a mapper) is not listed. The design's separate „a filesystem with no files" state is
|
||||
// folded into `filesystem` on purpose: telling the two apart needs a root read-only mount, and the data-bearing path's
|
||||
// confirmation is the safer default for both (`16` §13). Creating, deleting or resizing a partition: never here (D5).
|
||||
// The format binding is the partition's own by-id link (…-partN: the disk id + the partition number) via
|
||||
// DeviceDurableID, re-resolved before mkfs (wipe_reresolve.go); the attach key is its fs-UUID.
|
||||
type CandidatePartition struct {
|
||||
Device string `json:"device"` // e.g. /dev/sdb2
|
||||
Number int `json:"number,omitempty"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
FSType string `json:"fstype,omitempty"`
|
||||
State string `json:"state"` // empty | filesystem | in_use
|
||||
Mountable bool `json:"mountable"`
|
||||
DurableID string `json:"durable_id,omitempty"` // uuid:<fs-uuid> — the attach key ("" when empty)
|
||||
MountedAt string `json:"mounted_at,omitempty"` // in_use: Felhom's own mount
|
||||
// Formattable is false when ANY node of the disk is mounted (another part in use): the root mkfs wrapper refuses
|
||||
// every mount outside /mnt/felhom-drives on the disk and deliberately does not trust a sibling that claims to be
|
||||
// Felhom's own (a compromised agent could fake that). Such a part can still be USED (attach erases nothing).
|
||||
Formattable bool `json:"formattable"`
|
||||
}
|
||||
|
||||
const (
|
||||
PartEmpty = "empty"
|
||||
PartFilesystem = "filesystem"
|
||||
PartInUse = "in_use"
|
||||
)
|
||||
|
||||
// candidatePartitions is the pure lister: the whole disk's lsblk tree → its partitions. sizeOf/numberOf read /sys,
|
||||
// uuidOf the by-uuid links, owned is the set of mountpoints corroborated as Felhom's own (felhomOwnedMounts).
|
||||
func candidatePartitions(lsblkJSON []byte, wholeName string, sizeOf func(string) int64, numberOf func(string) int,
|
||||
uuidOf func(string) (string, bool), owned map[string]bool) []CandidatePartition {
|
||||
var doc struct {
|
||||
BlockDevices []lsblkDev `json:"blockdevices"`
|
||||
}
|
||||
if json.Unmarshal(lsblkJSON, &doc) != nil {
|
||||
return nil
|
||||
}
|
||||
var out []CandidatePartition
|
||||
for _, d := range doc.BlockDevices {
|
||||
if d.Name != wholeName {
|
||||
continue
|
||||
}
|
||||
diskMounted := d.MountPoint != ""
|
||||
var walk func(n lsblkDev)
|
||||
walk = func(n lsblkDev) {
|
||||
if n.MountPoint != "" {
|
||||
diskMounted = true
|
||||
}
|
||||
for _, c := range n.Children {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
for _, c := range d.Children {
|
||||
walk(c)
|
||||
}
|
||||
for _, p := range d.Children {
|
||||
if len(p.Children) > 0 || memberFSTypes[p.FSType] {
|
||||
continue // LVM/mapper on it, or a member: not a place (the disk filter refuses these anyway)
|
||||
}
|
||||
c := CandidatePartition{Device: "/dev/" + p.Name, Number: numberOf(p.Name), SizeBytes: sizeOf(p.Name), FSType: p.FSType}
|
||||
switch {
|
||||
case p.MountPoint != "" && (underFelhomDrives(p.MountPoint) || owned[p.MountPoint]):
|
||||
c.State, c.MountedAt = PartInUse, p.MountPoint
|
||||
case p.MountPoint != "":
|
||||
continue // mounted by something else: never offered (the disk filter refuses the disk anyway)
|
||||
case p.FSType == "":
|
||||
c.State = PartEmpty
|
||||
default:
|
||||
c.State = PartFilesystem
|
||||
c.Mountable = mountableFSTypes[p.FSType]
|
||||
if u, ok := uuidOf(c.Device); ok && u != "" {
|
||||
c.DurableID = "uuid:" + u
|
||||
}
|
||||
}
|
||||
c.Formattable = c.State != PartInUse && !diskMounted
|
||||
out = append(out, c)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// readSysPartNumber reads /sys/class/block/<part>/partition (0 when unreadable).
|
||||
func readSysPartNumber(name string) int {
|
||||
b, err := os.ReadFile(filepath.Join("/sys/class/block", name, "partition"))
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.Atoi(strings.TrimSpace(string(b)))
|
||||
return n
|
||||
}
|
||||
|
||||
// readSysPartSize reads a partition's size in bytes from /sys/class/block/<part>/size (512-byte sectors).
|
||||
func readSysPartSize(name string) int64 {
|
||||
b, err := os.ReadFile(filepath.Join("/sys/class/block", name, "size"))
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
s, err := strconv.ParseInt(strings.TrimSpace(string(b)), 10, 64)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return s * 512
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Partitions as places (felhom.eu architecture/16 §6, agent v0.158.0): every partition of a free disk with its own
|
||||
// state; Felhom's own drive shown but never offered; anything foreign, layered or a member never listed.
|
||||
|
||||
const partTree = `{"blockdevices":[
|
||||
{"name":"sdb","fstype":null,"mountpoint":null,"children":[
|
||||
{"name":"sdb1","fstype":"ext4","mountpoint":"/mnt/felhom-drives/hdd_1"},
|
||||
{"name":"sdb2","fstype":null,"mountpoint":null},
|
||||
{"name":"sdb3","fstype":"ntfs","mountpoint":null},
|
||||
{"name":"sdb4","fstype":"xfs","mountpoint":null},
|
||||
{"name":"sdb5","fstype":"ext4","mountpoint":"/mnt/films"},
|
||||
{"name":"sdb6","fstype":"ext4","mountpoint":"/srv/data"},
|
||||
{"name":"sdb7","fstype":"LVM2_member","mountpoint":null,"children":[{"name":"vg-lv","fstype":"ext4","mountpoint":null}]},
|
||||
{"name":"sdb8","fstype":"crypto_LUKS","mountpoint":null}
|
||||
]},
|
||||
{"name":"sdc","fstype":null,"mountpoint":null,"children":[{"name":"sdc1","fstype":null,"mountpoint":null}]}
|
||||
]}`
|
||||
|
||||
func listParts(owned map[string]bool) map[string]CandidatePartition {
|
||||
got := candidatePartitions([]byte(partTree), "sdb",
|
||||
func(n string) int64 { return int64(len(n)) << 30 },
|
||||
func(n string) int { return int(n[len(n)-1] - '0') },
|
||||
func(dev string) (string, bool) { return "uuid-of-" + strings.TrimPrefix(dev, "/dev/"), true },
|
||||
owned)
|
||||
m := map[string]CandidatePartition{}
|
||||
for _, p := range got {
|
||||
m[p.Device] = p
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func TestPartitions_StatesPerPartition(t *testing.T) {
|
||||
m := listParts(map[string]bool{"/mnt/films": true})
|
||||
want := map[string]string{"/dev/sdb1": PartInUse, "/dev/sdb2": PartEmpty, "/dev/sdb3": PartFilesystem,
|
||||
"/dev/sdb4": PartFilesystem, "/dev/sdb5": PartInUse}
|
||||
if len(m) != len(want) {
|
||||
t.Fatalf("want exactly %d listed partitions, got %+v", len(want), m)
|
||||
}
|
||||
for dev, st := range want {
|
||||
if m[dev].State != st {
|
||||
t.Fatalf("%s: want %s, got %+v", dev, st, m[dev])
|
||||
}
|
||||
}
|
||||
if m["/dev/sdb2"].DurableID != "" || m["/dev/sdb2"].Mountable {
|
||||
t.Fatalf("an empty part has no attach key: %+v", m["/dev/sdb2"])
|
||||
}
|
||||
if !m["/dev/sdb4"].Mountable || m["/dev/sdb4"].DurableID != "uuid:uuid-of-sdb4" || m["/dev/sdb4"].Number != 4 {
|
||||
t.Fatalf("an xfs part is usable by its fs-UUID: %+v", m["/dev/sdb4"])
|
||||
}
|
||||
if m["/dev/sdb3"].Mountable {
|
||||
t.Fatalf("ntfs is a filesystem the agent does not mount as-is: %+v", m["/dev/sdb3"])
|
||||
}
|
||||
}
|
||||
|
||||
// A raw /mnt/<name> mount counts as Felhom's own ONLY when corroborated (felhomOwnedMounts); else it is foreign and
|
||||
// the part is not listed. Positive control: TestPartitions_StatesPerPartition (the same mount, corroborated).
|
||||
func TestPartitions_UncorroboratedMountIsNotOffered(t *testing.T) {
|
||||
m := listParts(nil)
|
||||
if _, ok := m["/dev/sdb5"]; ok {
|
||||
t.Fatalf("an uncorroborated /mnt/films mount must not be listed, got %+v", m["/dev/sdb5"])
|
||||
}
|
||||
for _, dev := range []string{"/dev/sdb6", "/dev/sdb7", "/dev/sdb8", "/dev/sdc1"} {
|
||||
if _, ok := m[dev]; ok {
|
||||
t.Fatalf("%s must never be listed (foreign mount, LVM, LUKS, another disk)", dev)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The root wrapper refuses any format on a disk with a mounted node (and trusts no „it is Felhom's own" claim), so a
|
||||
// part next to a mounted one is never offered for formatting — only for use. Positive control: an unmounted disk.
|
||||
func TestPartitions_FormattableOnlyWhenNothingOnTheDiskIsMounted(t *testing.T) {
|
||||
m := listParts(map[string]bool{"/mnt/films": true})
|
||||
for dev, p := range m {
|
||||
if p.Formattable {
|
||||
t.Fatalf("%s: a part of a disk with a mounted part must not be formattable: %+v", dev, p)
|
||||
}
|
||||
}
|
||||
free := `{"blockdevices":[{"name":"sdc","fstype":null,"mountpoint":null,"children":[
|
||||
{"name":"sdc1","fstype":null,"mountpoint":null},{"name":"sdc2","fstype":"ext4","mountpoint":null}]}]}`
|
||||
got := candidatePartitions([]byte(free), "sdc", func(string) int64 { return 1 << 30 }, func(string) int { return 1 },
|
||||
func(string) (string, bool) { return "u", true }, nil)
|
||||
if len(got) != 2 || !got[0].Formattable || !got[1].Formattable {
|
||||
t.Fatalf("parts of an unmounted disk are formattable, got %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
// Package thincheck finds an ORPHAN thin device: one the LVM thin pool holds that LVM itself no longer knows (R-1012).
|
||||
//
|
||||
// A full host root while vzdump takes its storage snapshot can leave such a device behind (demo-hp 2026-10-14: the
|
||||
// pool held devices 1, 2 and 4; LVM knew 1 and 2). From then on every snapshot backup of that guest fails at the
|
||||
// snapshot step („Creation of new snapshot 4 of device 2 failed") and nothing in the product said why. This package
|
||||
// NAMES the orphan, in plain words, in the backup's failure — it never removes one: the repair
|
||||
// (`dmsetup message <pool>-tpool 0 "delete <id>"`) stays a human step, decided by the operator (`03` whole-guest
|
||||
// backup; runbook `whole-guest-backup-orphan-thin.md`).
|
||||
//
|
||||
// The device list comes from the root helper `felhom-thin-check <vg> <pool>` (config bundle), which reads the pool
|
||||
// through a metadata snapshot and LVM's list before and after. An id is an orphan only when the pool holds it and
|
||||
// NEITHER LVM list names it, on two reads a few seconds apart — so a volume being created at that moment is never
|
||||
// reported.
|
||||
package thincheck
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// HelperPath is where the config bundle installs the root helper.
|
||||
const HelperPath = "/usr/local/sbin/felhom-thin-check"
|
||||
|
||||
// Pool names one LVM thin pool.
|
||||
type Pool struct{ VG, Pool string }
|
||||
|
||||
func (p Pool) String() string { return p.VG + "/" + p.Pool }
|
||||
|
||||
// Reading is one helper run, parsed.
|
||||
type Reading struct {
|
||||
PoolIDs []int
|
||||
LVMBefore map[int]string
|
||||
LVMAfter map[int]string
|
||||
}
|
||||
|
||||
// Parse reads the helper's output. ok=false unless the closing "ok" line is there (a cut-off run is no reading).
|
||||
func Parse(out string) (Reading, bool) {
|
||||
r := Reading{LVMBefore: map[int]string{}, LVMAfter: map[int]string{}}
|
||||
done := false
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
f := strings.Fields(line)
|
||||
switch {
|
||||
case len(f) == 1 && f[0] == "ok":
|
||||
done = true
|
||||
case len(f) == 2 && f[0] == "pool":
|
||||
if id, err := strconv.Atoi(f[1]); err == nil {
|
||||
r.PoolIDs = append(r.PoolIDs, id)
|
||||
}
|
||||
case len(f) == 3 && (f[0] == "lvm-before" || f[0] == "lvm-after"):
|
||||
id, err := strconv.Atoi(f[1])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if f[0] == "lvm-before" {
|
||||
r.LVMBefore[id] = f[2]
|
||||
} else {
|
||||
r.LVMAfter[id] = f[2]
|
||||
}
|
||||
}
|
||||
}
|
||||
return r, done
|
||||
}
|
||||
|
||||
// Orphans returns the ids the pool holds that neither LVM list names, sorted.
|
||||
func (r Reading) Orphans() []int {
|
||||
var out []int
|
||||
for _, id := range r.PoolIDs {
|
||||
_, b := r.LVMBefore[id]
|
||||
_, a := r.LVMAfter[id]
|
||||
if !b && !a {
|
||||
out = append(out, id)
|
||||
}
|
||||
}
|
||||
sort.Ints(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// Runner is the privileged exec surface (proxmox.ExecRunner in sudo mode).
|
||||
type Runner interface {
|
||||
Run(ctx context.Context, name string, args ...string) ([]byte, []byte, error)
|
||||
}
|
||||
|
||||
// StorageLister is the read the checker needs to map a guest's volumes to their thin pools (*proxmox.Client).
|
||||
type StorageLister interface {
|
||||
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
|
||||
ListStorage(ctx context.Context) ([]proxmox.Storage, error)
|
||||
}
|
||||
|
||||
// Checker runs the check for one guest's pools.
|
||||
type Checker struct {
|
||||
API StorageLister
|
||||
Runner Runner
|
||||
Helper string // "" → HelperPath
|
||||
Pause time.Duration // between the two reads when the first finds an orphan; 0 → 3 s
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
// Finding is one pool with orphans.
|
||||
type Finding struct {
|
||||
Pool Pool
|
||||
Orphans []int
|
||||
}
|
||||
|
||||
// PoolsOf maps the guest's volume-backed disks (rootfs, mpN) to the lvmthin pools they live on.
|
||||
func (c *Checker) PoolsOf(ctx context.Context, vmid int) ([]Pool, error) {
|
||||
cfg, err := c.API.GuestConfig(ctx, vmid)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
stores, err := c.API.ListStorage(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
thin := map[string]Pool{}
|
||||
for _, s := range stores {
|
||||
if s.Type == "lvmthin" && s.VGName != "" && s.ThinPool != "" {
|
||||
thin[s.Storage] = Pool{VG: s.VGName, Pool: s.ThinPool}
|
||||
}
|
||||
}
|
||||
vols := []string{cfg.RootFS}
|
||||
mps := cfg.MountPoints()
|
||||
keys := make([]string, 0, len(mps))
|
||||
for k := range mps {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
vols = append(vols, mps[k])
|
||||
}
|
||||
seen := map[Pool]bool{}
|
||||
var out []Pool
|
||||
for _, v := range vols {
|
||||
store, _, ok := strings.Cut(strings.Split(v, ",")[0], ":")
|
||||
if !ok || strings.HasPrefix(store, "/") {
|
||||
continue // a bind mount (a path) is no volume
|
||||
}
|
||||
if p, isThin := thin[store]; isThin && !seen[p] {
|
||||
seen[p] = true
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// read runs the helper once.
|
||||
func (c *Checker) read(ctx context.Context, p Pool) (Reading, error) {
|
||||
helper := c.Helper
|
||||
if helper == "" {
|
||||
helper = HelperPath
|
||||
}
|
||||
out, stderr, err := c.Runner.Run(ctx, helper, p.VG, p.Pool)
|
||||
if err != nil {
|
||||
return Reading{}, fmt.Errorf("felhom-thin-check %s: %w: %s", p, err, strings.TrimSpace(string(stderr)))
|
||||
}
|
||||
r, ok := Parse(string(out))
|
||||
if !ok {
|
||||
return Reading{}, fmt.Errorf("felhom-thin-check %s: the output has no closing ok line", p)
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// Check returns the pools of vmid that hold an orphan (two reads, intersected). A pool that cannot be read is logged
|
||||
// and left out: the check FAILS OPEN — it explains a failure, it never stops a backup.
|
||||
func (c *Checker) Check(ctx context.Context, vmid int) []Finding {
|
||||
logger := c.Logger
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
pools, err := c.PoolsOf(ctx, vmid)
|
||||
if err != nil {
|
||||
logger.Warn("thincheck: cannot map the guest's disks to thin pools — not checked", "vmid", vmid, "err", err)
|
||||
return nil
|
||||
}
|
||||
pause := c.Pause
|
||||
if pause <= 0 {
|
||||
pause = 3 * time.Second
|
||||
}
|
||||
var found []Finding
|
||||
for _, p := range pools {
|
||||
first, err := c.read(ctx, p)
|
||||
if err != nil {
|
||||
logger.Warn("thincheck: thin pool not read — not checked", "vmid", vmid, "pool", p.String(), "err", err)
|
||||
continue
|
||||
}
|
||||
o1 := first.Orphans()
|
||||
if len(o1) == 0 {
|
||||
logger.Debug("thincheck: thin pool and LVM agree", "vmid", vmid, "pool", p.String(), "devices", len(first.PoolIDs))
|
||||
continue
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return found
|
||||
case <-time.After(pause):
|
||||
}
|
||||
second, err := c.read(ctx, p)
|
||||
if err != nil {
|
||||
logger.Warn("thincheck: second read failed — not reported", "vmid", vmid, "pool", p.String(), "err", err)
|
||||
continue
|
||||
}
|
||||
in2 := map[int]bool{}
|
||||
for _, id := range second.Orphans() {
|
||||
in2[id] = true
|
||||
}
|
||||
var both []int
|
||||
for _, id := range o1 {
|
||||
if in2[id] {
|
||||
both = append(both, id)
|
||||
}
|
||||
}
|
||||
if len(both) > 0 {
|
||||
found = append(found, Finding{Pool: p, Orphans: both})
|
||||
logger.Warn("thincheck: ORPHAN thin device(s) in the pool — LVM does not know them; NOT removed (a human step)",
|
||||
"vmid", vmid, "pool", p.String(), "orphans", both)
|
||||
}
|
||||
}
|
||||
return found
|
||||
}
|
||||
|
||||
// Describe is the operator sentence for a backup failure. Plain words; names the pool and the ids; says nothing was
|
||||
// removed and where the repair is written down.
|
||||
func Describe(f []Finding) string {
|
||||
if len(f) == 0 {
|
||||
return ""
|
||||
}
|
||||
var parts []string
|
||||
for _, x := range f {
|
||||
ids := make([]string, len(x.Orphans))
|
||||
for i, id := range x.Orphans {
|
||||
ids[i] = strconv.Itoa(id)
|
||||
}
|
||||
parts = append(parts, fmt.Sprintf("thin pool %s holds device %s that LVM does not know", x.Pool, strings.Join(ids, ", ")))
|
||||
}
|
||||
return "probable cause: " + strings.Join(parts, "; ") +
|
||||
" (an orphan snapshot, usually left when the host disk filled during a backup). Snapshot backups on this pool " +
|
||||
"fail until an operator removes it; Felhom did not remove anything. Repair: runbook whole-guest-backup-orphan-thin.md"
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
package thincheck
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// The helper's output on demo-hp 2026-10-14, before the repair: the pool held 1, 2 and 4; LVM knew 1 and 2
|
||||
// (audits/power-cut-fixes-2026-10-14/E/thin-ls.txt).
|
||||
const orphanOut = `lvm-before 1 vm-9201-disk-0
|
||||
lvm-before 2 vm-9201-disk-1
|
||||
pool 1
|
||||
pool 2
|
||||
pool 4
|
||||
lvm-after 1 vm-9201-disk-0
|
||||
lvm-after 2 vm-9201-disk-1
|
||||
ok
|
||||
`
|
||||
|
||||
const healthyOut = `lvm-before 1 vm-9201-disk-0
|
||||
lvm-before 2 vm-9201-disk-1
|
||||
pool 1
|
||||
pool 2
|
||||
lvm-after 1 vm-9201-disk-0
|
||||
lvm-after 2 vm-9201-disk-1
|
||||
ok
|
||||
`
|
||||
|
||||
func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
|
||||
|
||||
func TestParse_TheOrphanOf20261014(t *testing.T) {
|
||||
r, ok := Parse(orphanOut)
|
||||
if !ok {
|
||||
t.Fatal("a complete reading was not accepted")
|
||||
}
|
||||
if got := r.Orphans(); !reflect.DeepEqual(got, []int{4}) {
|
||||
t.Fatalf("orphans = %v, want [4]", got)
|
||||
}
|
||||
h, _ := Parse(healthyOut)
|
||||
if got := h.Orphans(); len(got) != 0 {
|
||||
t.Fatalf("a healthy pool reported orphans %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A volume LVM created WHILE the pool was read appears only in the after-list: never an orphan.
|
||||
func TestOrphans_AVolumeCreatedDuringTheReadIsNotAnOrphan(t *testing.T) {
|
||||
r, _ := Parse(strings.Replace(orphanOut, "lvm-after 2 vm-9201-disk-1\n", "lvm-after 2 vm-9201-disk-1\nlvm-after 4 snap_vm-9201-disk-1_vzdump\n", 1))
|
||||
if got := r.Orphans(); len(got) != 0 {
|
||||
t.Fatalf("orphans = %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A run cut off before its closing "ok" (the helper died, the output is partial) is no reading at all.
|
||||
func TestParse_NoClosingLineIsNoReading(t *testing.T) {
|
||||
if _, ok := Parse(strings.TrimSuffix(orphanOut, "ok\n")); ok {
|
||||
t.Fatal("a cut-off run was accepted as a reading")
|
||||
}
|
||||
}
|
||||
|
||||
type fakeAPI struct {
|
||||
cfg proxmox.GuestConfig
|
||||
stores []proxmox.Storage
|
||||
}
|
||||
|
||||
func (f *fakeAPI) GuestConfig(context.Context, int) (proxmox.GuestConfig, error) { return f.cfg, nil }
|
||||
func (f *fakeAPI) ListStorage(context.Context) ([]proxmox.Storage, error) { return f.stores, nil }
|
||||
|
||||
func guest9201() *fakeAPI {
|
||||
extra := map[string]json.RawMessage{
|
||||
"mp0": json.RawMessage(`"local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=20G"`),
|
||||
"mp8": json.RawMessage(`"/mnt/felhom-drives,mp=/mnt/felhom-drives"`),
|
||||
"mp9": json.RawMessage(`"local:9201/vm-9201-disk-2.raw,mp=/x"`),
|
||||
}
|
||||
return &fakeAPI{
|
||||
cfg: proxmox.GuestConfig{RootFS: "local-lvm:vm-9201-disk-0,size=8G", Extra: extra},
|
||||
stores: []proxmox.Storage{
|
||||
{Storage: "local", Type: "dir", Path: "/var/lib/vz"},
|
||||
{Storage: "local-lvm", Type: "lvmthin", VGName: "pve", ThinPool: "data"},
|
||||
{Storage: "felhom-pbs", Type: "pbs"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
type seqRunner struct {
|
||||
outs []string
|
||||
errs []error
|
||||
calls [][]string
|
||||
}
|
||||
|
||||
func (r *seqRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
i := len(r.calls) - 1
|
||||
var err error
|
||||
if i < len(r.errs) {
|
||||
err = r.errs[i]
|
||||
}
|
||||
if i < len(r.outs) {
|
||||
return []byte(r.outs[i]), nil, err
|
||||
}
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
func TestPoolsOf_RootfsAndVolumeMountpointsOnly(t *testing.T) {
|
||||
c := &Checker{API: guest9201(), Runner: &seqRunner{}, Logger: quiet()}
|
||||
pools, err := c.PoolsOf(context.Background(), 9201)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(pools, []Pool{{VG: "pve", Pool: "data"}}) {
|
||||
t.Fatalf("pools = %v, want [pve/data] once (bind mounts and dir volumes are not thin)", pools)
|
||||
}
|
||||
}
|
||||
|
||||
// The consequence: an orphan seen on BOTH reads becomes a finding whose sentence names the pool and the id, and says
|
||||
// nothing was removed. The only command ever run is the read-only helper with <vg> <pool>.
|
||||
// RED-PROOF (P1/redproof-thincheck.txt): with Orphans() returning nil this test fails.
|
||||
func TestCheck_OrphanOnBothReads_IsNamed(t *testing.T) {
|
||||
rr := &seqRunner{outs: []string{orphanOut, orphanOut}}
|
||||
c := &Checker{API: guest9201(), Runner: rr, Pause: 1, Logger: quiet()}
|
||||
f := c.Check(context.Background(), 9201)
|
||||
if len(f) != 1 || f[0].Pool.String() != "pve/data" || !reflect.DeepEqual(f[0].Orphans, []int{4}) {
|
||||
t.Fatalf("findings = %+v", f)
|
||||
}
|
||||
want := [][]string{{HelperPath, "pve", "data"}, {HelperPath, "pve", "data"}}
|
||||
if !reflect.DeepEqual(rr.calls, want) {
|
||||
t.Fatalf("calls = %v, want only the read-only helper twice", rr.calls)
|
||||
}
|
||||
d := Describe(f)
|
||||
for _, s := range []string{"thin pool pve/data holds device 4 that LVM does not know", "Felhom did not remove anything",
|
||||
"whole-guest-backup-orphan-thin.md"} {
|
||||
if !strings.Contains(d, s) {
|
||||
t.Fatalf("Describe = %q, missing %q", d, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An id that is gone on the second read (a volume mid-creation) is not reported; a healthy pool is read ONCE.
|
||||
func TestCheck_TransientOrHealthy_NothingReported(t *testing.T) {
|
||||
rr := &seqRunner{outs: []string{orphanOut, healthyOut}}
|
||||
if f := (&Checker{API: guest9201(), Runner: rr, Pause: 1, Logger: quiet()}).Check(context.Background(), 9201); len(f) != 0 {
|
||||
t.Fatalf("a transient id was reported: %+v", f)
|
||||
}
|
||||
rr = &seqRunner{outs: []string{healthyOut}}
|
||||
if f := (&Checker{API: guest9201(), Runner: rr, Logger: quiet()}).Check(context.Background(), 9201); len(f) != 0 || len(rr.calls) != 1 {
|
||||
t.Fatalf("healthy: findings %+v, calls %v", f, rr.calls)
|
||||
}
|
||||
if Describe(nil) != "" {
|
||||
t.Fatal("no finding must describe as empty")
|
||||
}
|
||||
}
|
||||
|
||||
// The helper refusing (a metadata snapshot held by someone else, exit 3) or failing: no finding, no panic — the check
|
||||
// fails open; it explains a failure, it never stops a backup.
|
||||
func TestCheck_HelperRefuses_FailsOpen(t *testing.T) {
|
||||
rr := &seqRunner{errs: []error{errors.New("exit status 3")}}
|
||||
if f := (&Checker{API: guest9201(), Runner: rr, Logger: quiet()}).Check(context.Background(), 9201); len(f) != 0 {
|
||||
t.Fatalf("findings = %+v", f)
|
||||
}
|
||||
}
|
||||
|
||||
// The root helper's own tests (configs/test_felhom_thin_check.py) run with the Go suite.
|
||||
func TestRootHelperPythonSuite(t *testing.T) {
|
||||
py, err := exec.LookPath("python3")
|
||||
if err != nil {
|
||||
t.Skip("python3 not found")
|
||||
}
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
script := filepath.Join(filepath.Dir(file), "..", "..", "configs", "test_felhom_thin_check.py")
|
||||
out, err := exec.Command(py, script).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("helper suite failed: %v\n%s", err, out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,208 @@
|
||||
// Package vzpartial removes a dead, half-written vzdump backup of a guest from a local dir storage (R-1006).
|
||||
//
|
||||
// A power cut during the night's whole-guest backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` and its `.tmp/` directory
|
||||
// on the target (demo-hp 2026-10-14: 12.2 GB on the host root). Nothing removes them: Proxmox does not list them as
|
||||
// backups, every later backup is skipped „not enough space", and a host root that fills up breaks LVM's own metadata
|
||||
// writes (audits/power-cut-fixes-2026-10-14 C4). This package finds such a leftover and asks the root helper
|
||||
// /usr/local/sbin/felhom-vzdump-partial-clean to remove it; the helper re-checks every rule as root.
|
||||
//
|
||||
// Rules here (the helper repeats them, and adds the open-file check the agent cannot do):
|
||||
// - only a `dir` storage; only names of the exact vzdump pattern for that vmid (NamePattern) — never a finished
|
||||
// archive, a log, another guest;
|
||||
// - only when no vzdump for that vmid is in flight (the R-1008 answer: a task from before the boot is not in flight);
|
||||
// - only when dead: last modified before this boot, or untouched for QuietFor.
|
||||
//
|
||||
// Any doubt leaves the entry and logs why. Pinned by vzpartial_test.go.
|
||||
package vzpartial
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// Helper is the root helper's installed path (configs/felhom-vzdump-partial-clean, delivered by the config bundle).
|
||||
const Helper = "/usr/local/sbin/felhom-vzdump-partial-clean"
|
||||
|
||||
// QuietFor is how long an entry written in THIS boot must be untouched before it counts as dead (the helper's QUIET_S).
|
||||
const QuietFor = 120 * time.Second
|
||||
|
||||
// NamePattern matches the half-written entries of one guest: the archive being written (.tar.dat / .vma.dat) and the
|
||||
// temporary directory (.tmp). The timestamp is vzdump's own (host local time).
|
||||
func NamePattern(vmid int) *regexp.Regexp {
|
||||
return regexp.MustCompile(`^vzdump-(lxc|qemu)-` + strconv.Itoa(vmid) +
|
||||
`-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$`)
|
||||
}
|
||||
|
||||
// Entry is one leftover found in the dump directory.
|
||||
type Entry struct {
|
||||
Name string
|
||||
Dir bool
|
||||
ModTime time.Time
|
||||
Size int64
|
||||
}
|
||||
|
||||
// Candidates lists the entries of dumpDir that match vmid's half-written pattern. A missing directory is no error.
|
||||
func Candidates(dumpDir string, vmid int) ([]Entry, error) {
|
||||
ents, err := os.ReadDir(dumpDir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
re := NamePattern(vmid)
|
||||
var out []Entry
|
||||
for _, e := range ents {
|
||||
if !re.MatchString(e.Name()) || e.Type()&os.ModeSymlink != 0 {
|
||||
continue
|
||||
}
|
||||
info, err := e.Info()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if strings.HasSuffix(e.Name(), ".tmp") != info.IsDir() {
|
||||
continue // a .tmp that is not a directory, or an archive name that is one: not vzdump's
|
||||
}
|
||||
mt := info.ModTime()
|
||||
if info.IsDir() {
|
||||
mt = newestUnder(filepath.Join(dumpDir, e.Name()), mt)
|
||||
}
|
||||
out = append(out, Entry{Name: e.Name(), Dir: info.IsDir(), ModTime: mt, Size: info.Size()})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func newestUnder(dir string, mt time.Time) time.Time {
|
||||
_ = filepath.WalkDir(dir, func(_ string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
if i, e := d.Info(); e == nil && i.ModTime().After(mt) {
|
||||
mt = i.ModTime()
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return mt
|
||||
}
|
||||
|
||||
// Dead decides whether an entry is dead: written before this boot, or untouched for QuietFor. bootOK=false (the boot
|
||||
// time could not be read) leaves only the quiet rule.
|
||||
func Dead(e Entry, boot time.Time, bootOK bool, now time.Time) (bool, string) {
|
||||
if bootOK && e.ModTime.Before(boot) {
|
||||
return true, "written before this boot"
|
||||
}
|
||||
if age := now.Sub(e.ModTime); age >= QuietFor {
|
||||
return true, "untouched for " + age.Round(time.Second).String()
|
||||
}
|
||||
return false, "modified " + now.Sub(e.ModTime).Round(time.Second).String() + " ago — may still be written"
|
||||
}
|
||||
|
||||
// StorageAPI is the one Proxmox read the cleaner needs (the storage's type and path).
|
||||
type StorageAPI interface {
|
||||
ListStorage(ctx context.Context) ([]proxmox.Storage, error)
|
||||
}
|
||||
|
||||
// Cleaner removes dead leftovers. All fields but Logger and Now are required; a nil Cleaner is a no-op.
|
||||
type Cleaner struct {
|
||||
API StorageAPI
|
||||
Runner proxmox.Runner // the fenced sudo runner
|
||||
InFlight func(ctx context.Context, vmid int) (bool, error)
|
||||
Boot func() (int64, bool)
|
||||
Logger *slog.Logger
|
||||
Now func() time.Time
|
||||
// dumpDir maps a storage path to its dump directory (tests); nil = <path>/dump.
|
||||
dumpDir func(path string) string
|
||||
}
|
||||
|
||||
// Result is one removal or one entry left in place.
|
||||
type Result struct {
|
||||
Entry Entry
|
||||
Removed bool
|
||||
Why string
|
||||
}
|
||||
|
||||
// Clean removes the dead leftovers of vmid on storage. It never returns an error for „nothing to do"; every entry it
|
||||
// finds is logged with what was done and why.
|
||||
func (c *Cleaner) Clean(ctx context.Context, storage string, vmid int) []Result {
|
||||
if c == nil || c.API == nil || c.Runner == nil || c.InFlight == nil || c.Boot == nil {
|
||||
return nil
|
||||
}
|
||||
log := c.Logger
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
now := time.Now
|
||||
if c.Now != nil {
|
||||
now = c.Now
|
||||
}
|
||||
stores, err := c.API.ListStorage(ctx)
|
||||
if err != nil {
|
||||
log.Warn("vzdump-partial: storage list unreadable — not looking for leftovers", "storage", storage, "err", err)
|
||||
return nil
|
||||
}
|
||||
var st *proxmox.Storage
|
||||
for i := range stores {
|
||||
if stores[i].Storage == storage {
|
||||
st = &stores[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if st == nil || st.Type != "dir" || st.Path == "" {
|
||||
return nil // PBS, LVM, NFS …: vzdump does not leave a .tar.dat there that this agent may remove
|
||||
}
|
||||
dump := filepath.Join(st.Path, "dump")
|
||||
if c.dumpDir != nil {
|
||||
dump = c.dumpDir(st.Path)
|
||||
}
|
||||
ents, err := Candidates(dump, vmid)
|
||||
if err != nil {
|
||||
log.Warn("vzdump-partial: dump directory unreadable", "dir", dump, "err", err)
|
||||
return nil
|
||||
}
|
||||
if len(ents) == 0 {
|
||||
return nil
|
||||
}
|
||||
var out []Result
|
||||
inFlight, err := c.InFlight(ctx, vmid)
|
||||
if err != nil || inFlight {
|
||||
why := "a backup of this guest is in flight"
|
||||
if err != nil {
|
||||
why = "cannot tell whether a backup is in flight: " + err.Error()
|
||||
}
|
||||
for _, e := range ents {
|
||||
log.Info("vzdump-partial: leaving a half-written backup", "storage", storage, "vmid", vmid, "name", e.Name, "why", why)
|
||||
out = append(out, Result{Entry: e, Why: why})
|
||||
}
|
||||
return out
|
||||
}
|
||||
b, bootOK := c.Boot()
|
||||
boot := time.Unix(b, 0)
|
||||
for _, e := range ents {
|
||||
dead, why := Dead(e, boot, bootOK, now())
|
||||
if !dead {
|
||||
log.Info("vzdump-partial: leaving a half-written backup", "storage", storage, "vmid", vmid, "name", e.Name, "why", why)
|
||||
out = append(out, Result{Entry: e, Why: why})
|
||||
continue
|
||||
}
|
||||
stdout, stderr, err := c.Runner.Run(ctx, Helper, storage, strconv.Itoa(vmid), e.Name)
|
||||
if err != nil {
|
||||
why = fmt.Sprintf("the root helper refused: %s", strings.TrimSpace(string(stderr)))
|
||||
log.Warn("vzdump-partial: leaving a half-written backup", "storage", storage, "vmid", vmid, "name", e.Name, "why", why, "err", err)
|
||||
out = append(out, Result{Entry: e, Why: why})
|
||||
continue
|
||||
}
|
||||
log.Warn("vzdump-partial: removed a dead half-written backup", "storage", storage, "vmid", vmid, "name", e.Name,
|
||||
"bytes", e.Size, "why", why, "helper", strings.TrimSpace(string(stdout)))
|
||||
out = append(out, Result{Entry: e, Removed: true, Why: why})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package vzpartial
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
const vmid = 9201
|
||||
const name = "vzdump-lxc-9201-2026_10_14-10_42_25"
|
||||
|
||||
var boot = time.Unix(1791967481, 0)
|
||||
|
||||
type fakeAPI struct{ stores []proxmox.Storage }
|
||||
|
||||
func (f fakeAPI) ListStorage(context.Context) ([]proxmox.Storage, error) { return f.stores, nil }
|
||||
|
||||
type fakeRunner struct {
|
||||
calls [][]string
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *fakeRunner) Run(_ context.Context, n string, a ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{n}, a...))
|
||||
if r.err != nil {
|
||||
return nil, []byte("REFUSED: in use"), r.err
|
||||
}
|
||||
return []byte("removed"), nil, nil
|
||||
}
|
||||
func (r *fakeRunner) RunStdin(ctx context.Context, _ io.Reader, n string, a ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, n, a...)
|
||||
}
|
||||
|
||||
type rig struct {
|
||||
dump string
|
||||
runner *fakeRunner
|
||||
c *Cleaner
|
||||
}
|
||||
|
||||
func newRig(t *testing.T, storeType string, inFlight bool, inErr error) *rig {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
dump := filepath.Join(root, "dump")
|
||||
if err := os.MkdirAll(dump, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := &rig{dump: dump, runner: &fakeRunner{}}
|
||||
r.c = &Cleaner{
|
||||
API: fakeAPI{stores: []proxmox.Storage{{Storage: "local", Type: storeType, Path: root}}},
|
||||
Runner: r.runner,
|
||||
InFlight: func(context.Context, int) (bool, error) { return inFlight, inErr },
|
||||
Boot: func() (int64, bool) { return boot.Unix(), true },
|
||||
Now: func() time.Time { return boot.Add(5 * time.Minute) },
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *rig) put(t *testing.T, n string, dir bool, mt time.Time) {
|
||||
t.Helper()
|
||||
p := filepath.Join(r.dump, n)
|
||||
if dir {
|
||||
if err := os.MkdirAll(filepath.Join(p, "etc"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chtimes(filepath.Join(p, "etc"), mt, mt)
|
||||
} else if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chtimes(p, mt, mt)
|
||||
}
|
||||
|
||||
func (r *rig) asked() []string {
|
||||
var out []string
|
||||
for _, c := range r.runner.calls {
|
||||
if len(c) == 4 && c[0] == Helper && c[1] == "local" && c[2] == "9201" {
|
||||
out = append(out, c[3])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// RED-PROOF: remove the `bootOK && e.ModTime.Before(boot)` rule in Dead → a pre-boot entry modified 1 min before the
|
||||
// boot is still „untouched for 6m" — so the test uses a 1-min-old-at-boot entry AND a Now 30 s after the boot.
|
||||
func TestClean_RemovesDeadPartialAndTmp(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.c.Now = func() time.Time { return boot.Add(30 * time.Second) }
|
||||
r.put(t, name+".tar.dat", false, boot.Add(-time.Minute))
|
||||
r.put(t, name+".tmp", true, boot.Add(-time.Minute))
|
||||
res := r.c.Clean(context.Background(), "local", vmid)
|
||||
if got := r.asked(); len(got) != 2 {
|
||||
t.Fatalf("both dead entries must go to the helper, got %v", got)
|
||||
}
|
||||
for _, x := range res {
|
||||
if !x.Removed || x.Why != "written before this boot" {
|
||||
t.Fatalf("%+v", x)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClean_RemovesQuietPartialOfThisBoot(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.put(t, name+".tar.dat", false, boot.Add(time.Minute)) // 4 min before Now
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if len(r.asked()) != 1 {
|
||||
t.Fatal("a partial untouched for QuietFor must be removed")
|
||||
}
|
||||
}
|
||||
|
||||
// Each keep has its positive control (the dead pre-boot partial in the same rig IS asked for).
|
||||
func TestClean_Keeps(t *testing.T) {
|
||||
cases := map[string]func(t *testing.T) (*rig, string){
|
||||
// RED-PROOF: drop the InFlight check in Clean → the in-flight partial is asked for → fails.
|
||||
"in flight": func(t *testing.T) (*rig, string) { return newRig(t, "dir", true, nil), name + ".tar.dat" },
|
||||
"in-flight unknown": func(t *testing.T) (*rig, string) {
|
||||
return newRig(t, "dir", false, errors.New("api down")), name + ".tar.dat"
|
||||
},
|
||||
// RED-PROOF: drop the `st.Type != "dir"` test → an lvmthin target is cleaned → fails.
|
||||
"not a dir storage": func(t *testing.T) (*rig, string) { return newRig(t, "lvmthin", false, nil), name + ".tar.dat" },
|
||||
}
|
||||
for what, mk := range cases {
|
||||
r, n := mk(t)
|
||||
r.put(t, n, false, boot.Add(-time.Hour))
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if len(r.asked()) != 0 {
|
||||
t.Fatalf("%s: must keep, helper asked for %v", what, r.asked())
|
||||
}
|
||||
ctl := newRig(t, "dir", false, nil)
|
||||
ctl.put(t, n, false, boot.Add(-time.Hour))
|
||||
ctl.c.Clean(context.Background(), "local", vmid)
|
||||
if len(ctl.asked()) != 1 {
|
||||
t.Fatalf("%s: positive control not removed", what)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: weaken NamePattern to `vzdump-.*` → the finished archive / log / other guest's partial are asked for.
|
||||
func TestClean_OnlyThisGuestsHalfWrittenNames(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
old := boot.Add(-time.Hour)
|
||||
for _, n := range []string{name + ".tar.zst", name + ".log", name + ".tar.zst.notes", "vzdump-lxc-9202-2026_10_14-10_42_25.tar.dat",
|
||||
"vzdump-lxc-92011-2026_10_14-10_42_25.tar.dat", name + ".tar.dat.bak", name + ".tar.da", "x" + name + ".tar.dat"} {
|
||||
r.put(t, n, false, old)
|
||||
}
|
||||
r.put(t, name+".tar.dat", false, old) // the positive control in the same directory
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if got := r.asked(); len(got) != 1 || got[0] != name+".tar.dat" {
|
||||
t.Fatalf("only %s.tar.dat may be asked for, got %v", name, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClean_KeepsRecentAndWrongKindAndSymlink(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.put(t, name+".tar.dat", false, boot.Add(4*time.Minute)) // 1 min before Now: may still be written
|
||||
r.put(t, "vzdump-lxc-9201-2026_10_14-11_00_00.tmp", false, boot.Add(-time.Hour)) // a FILE named .tmp
|
||||
r.put(t, "vzdump-lxc-9201-2026_10_14-13_00_00.tar.dat", false, boot.Add(-time.Hour)) // positive control
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if got := r.asked(); len(got) != 1 || got[0] != "vzdump-lxc-9201-2026_10_14-13_00_00.tar.dat" {
|
||||
t.Fatalf("only the positive control may be asked for, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the ModeSymlink skip in Candidates → the symlink (quiet: Now is an hour ahead) is asked for → fails.
|
||||
func TestClean_KeepsSymlink(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.c.Now = func() time.Time { return time.Now().Add(time.Hour) }
|
||||
target := filepath.Join(r.dump, "elsewhere")
|
||||
_ = os.WriteFile(target, []byte("x"), 0o644)
|
||||
_ = os.Symlink(target, filepath.Join(r.dump, "vzdump-lxc-9201-2026_10_14-12_00_00.tar.dat"))
|
||||
r.put(t, "vzdump-lxc-9201-2026_10_14-13_00_00.tar.dat", false, boot.Add(-time.Hour)) // positive control
|
||||
r.c.Clean(context.Background(), "local", vmid)
|
||||
if got := r.asked(); len(got) != 1 || got[0] != "vzdump-lxc-9201-2026_10_14-13_00_00.tar.dat" {
|
||||
t.Fatalf("only the positive control may be asked for, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClean_HelperRefusalIsNotARemoval(t *testing.T) {
|
||||
r := newRig(t, "dir", false, nil)
|
||||
r.runner.err = errors.New("exit status 1")
|
||||
r.put(t, name+".tar.dat", false, boot.Add(-time.Hour))
|
||||
res := r.c.Clean(context.Background(), "local", vmid)
|
||||
if len(res) != 1 || res[0].Removed {
|
||||
t.Fatalf("a refused helper call must not count as removed: %+v", res)
|
||||
}
|
||||
}
|
||||
|
||||
// The root helper's own tests (configs/test_felhom_vzdump_partial_clean.py) run with the Go suite.
|
||||
func TestRootHelperPythonSuite(t *testing.T) {
|
||||
py, err := exec.LookPath("python3")
|
||||
if err != nil {
|
||||
t.Skip("python3 not found")
|
||||
}
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
script := filepath.Join(filepath.Dir(file), "..", "..", "configs", "test_felhom_vzdump_partial_clean.py")
|
||||
out, err := exec.Command(py, script).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("helper suite failed: %v\n%s", err, out)
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,9 @@ Gates (all must pass; **non-zero exit on any failure**):
|
||||
or because someone is offline on a train. CI runs the FULL set for exactly this
|
||||
reason: it is the machine that can afford a network check, and it is the half
|
||||
that emails when something is wrong.
|
||||
… gate-decoys the decoy suite (test_gate_decoys.py): every gate here still convicts its decoys
|
||||
(R-1116). ~20 s in the workspace, so the full run only: not the hook, but CI
|
||||
runs the full set on every push. Scratch clones and a local fake Gitea only.
|
||||
|
||||
WHY THIS FILE EXISTS, WITH ONE GATE (2026-08-02, R-29 leg (b)).
|
||||
|
||||
@@ -58,6 +61,12 @@ GATES = [
|
||||
("release-complete", os.path.join(ROOT, "scripts", "check-release-complete.py"), [], True),
|
||||
# R-389 — a REPORT.md observation with no register row behind it. Fast: stdlib file reads.
|
||||
("observations", SHARED_OBSERVATIONS, [ROOT], True),
|
||||
# R-1116 (2026-10-17) — the gates' OWN decoy suite. It was in no runner, the shape that let the catalog's
|
||||
# R-1115 fail unseen for a night. Full run only: ~20 s in the workspace (measured, 30 cases; ~5 s in a
|
||||
# CI-shaped clone, where the shared gates have fewer siblings to search), too slow for the hook — and CI
|
||||
# runs the FULL set, so every push is still judged by it. It plants only in scratch clones and asks a fake
|
||||
# Gitea on 127.0.0.1 (never the network, never the real tree); it does not call this runner (no recursion).
|
||||
("gate-decoys", os.path.join(ROOT, "scripts", "test_gate_decoys.py"), [], False),
|
||||
]
|
||||
|
||||
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
|
||||
|
||||
@@ -69,6 +69,10 @@ fi
|
||||
existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
|
||||
[[ "$existing" != "200" ]] || die "version $VERSION is ALREADY PUBLISHED — bump the version instead"
|
||||
# R-1098 (2026-10-16): only a clean 404 proves the version is new. Until now a registry that could not be reached
|
||||
# (curl prints 000) or answered anything else passed this check — „I could not ask" is not that proof.
|
||||
# Test: scripts/test_release_push_once.sh.
|
||||
[[ "$existing" == "404" ]] || die "could not establish whether $VERSION is published (registry answered HTTP '${existing:-none}') — refusing"
|
||||
|
||||
# ── 3. Build ────────────────────────────────────────────────────────────────────────────────────
|
||||
BIN="$(mktemp -t felhom-agent-XXXXXX)"
|
||||
|
||||
@@ -57,6 +57,20 @@ class AgentGatesTest(unittest.TestCase):
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
def test_decoy_suite_is_a_full_run_gate(self):
|
||||
"""R-1116: the decoy suite is a gate of the FULL run (which CI runs on every push), and --fast
|
||||
names it as skipped (~20 s, too slow for the hook)."""
|
||||
sys.dont_write_bytecode = True # no __pycache__ left in scripts/ (not gitignored here)
|
||||
sys.path.insert(0, os.path.join(ROOT, "scripts"))
|
||||
import agent_gates as mod
|
||||
rows = [g for g in mod.GATES if g[0] == "gate-decoys"]
|
||||
self.assertEqual(len(rows), 1, "gate-decoys is not registered exactly once")
|
||||
self.assertEqual(os.path.basename(rows[0][1]), "test_gate_decoys.py")
|
||||
self.assertFalse(rows[0][3], "gate-decoys must not be --fast")
|
||||
self.assertEqual([g[0] for g in mod.GATES if not g[3]], ["published", "gate-decoys"])
|
||||
self.assertIn("--fast SKIPPED (deliberate periodic runs, never in a hook): published, gate-decoys",
|
||||
self.out)
|
||||
|
||||
def test_unknown_argument_is_rejected(self):
|
||||
p = subprocess.run([sys.executable, ENTRY, "--nope"], cwd=ROOT,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||
|
||||
@@ -66,6 +66,9 @@ COVERS = {
|
||||
"inside an HTML comment (R-426)",
|
||||
"observations": "R-419 in THIS repo's REPORT.md: an Observations note SAYING it carries no marker - vs the "
|
||||
"two genuine markers (R-426)",
|
||||
"gate-decoys": "this suite, run as a gate of agent_gates.py's full run, which CI runs on every push (R-1116): "
|
||||
"every case above is a decoy of its own gate, and ONE case that stops convicting turns the run "
|
||||
"red - red-proved 2026-10-17 by flipping a reuse-refs FACT case to expect rc 0",
|
||||
}
|
||||
|
||||
fails = []
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env bash
|
||||
# test_release_push_once.sh — release-agent.sh releases a version only when the registry says it is NEW (R-1098).
|
||||
#
|
||||
# The REAL release-agent.sh runs from a copy inside a scratch git repo (so its tag check reads scratch tags), with
|
||||
# RELEASE_ALLOW_DIRTY=1 (the clean-tree gate has its own history) and a fake `curl` and `go` first on PATH: curl answers
|
||||
# the package probe as FAKE_MODE says, and `go` stops the run right after the check, so nothing is built or published.
|
||||
# published (200) -> refused, "ALREADY PUBLISHED"
|
||||
# absent (404) -> passes the check, reaches the build
|
||||
# unreachable (000) -> refused, "could not establish"
|
||||
# server error (500) -> refused, "could not establish"
|
||||
# the tag already exists -> refused, "already exists"
|
||||
# Run: bash scripts/test_release_push_once.sh (exit 0 all cases right · 1 a case wrong)
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
WS="$(mktemp -d)"; trap 'rm -rf "${WS:?}"' EXIT
|
||||
fails=0; ran=0
|
||||
mkdir -p "$WS/bin" "$WS/repo/scripts"
|
||||
cp "$HERE/release-agent.sh" "$WS/repo/scripts/release-agent.sh"
|
||||
(cd "$WS/repo" && git init -q -b main && git -c user.name=t -c user.email=t@t commit -q --allow-empty -m init)
|
||||
cat > "$WS/bin/curl" <<'FAKE'
|
||||
#!/bin/bash
|
||||
case "${FAKE_MODE}" in
|
||||
published) printf 200; exit 0 ;;
|
||||
absent) printf 404; exit 22 ;;
|
||||
down) printf 000; exit 7 ;;
|
||||
error) printf 500; exit 22 ;;
|
||||
esac
|
||||
FAKE
|
||||
printf '#!/bin/sh\necho "FAKE-GO-REACHED $*"; exit 1\n' > "$WS/bin/go"
|
||||
chmod +x "$WS/bin/curl" "$WS/bin/go"
|
||||
run() { # name, mode, expected rc, required fragment, forbidden fragment
|
||||
local out rc
|
||||
out=$(cd "$WS/repo" && FAKE_MODE="$2" RELEASE_ALLOW_DIRTY=1 GITEA_USER=u GITEA_TOKEN=t PATH="$WS/bin:$PATH" \
|
||||
bash scripts/release-agent.sh 9.9.9 2>&1); rc=$?
|
||||
ran=$((ran+1))
|
||||
if [[ "$rc" != "$3" ]] || ! grep -qi -- "$4" <<<"$out" || { [[ -n "$5" ]] && grep -qi -- "$5" <<<"$out"; }; then
|
||||
fails=$((fails+1)); echo " FAIL $1 (exit $rc)"; echo "$out" | tail -4 | sed 's/^/ /'; return
|
||||
fi
|
||||
echo " ok $1"
|
||||
}
|
||||
echo "release-agent.sh — a version is released once, and only when the registry says it is new"
|
||||
run "published (200) -> refused" published 1 "ALREADY PUBLISHED" "FAKE-GO-REACHED"
|
||||
run "absent (404) -> proceeds" absent 1 "FAKE-GO-REACHED" "could not establish"
|
||||
run "unreachable (000) -> refused" down 1 "could not establish" "FAKE-GO-REACHED"
|
||||
run "server error (500) -> refused" error 1 "could not establish" "FAKE-GO-REACHED"
|
||||
(cd "$WS/repo" && git tag v9.9.9)
|
||||
run "the tag exists -> refused" absent 1 "already exists" "FAKE-GO-REACHED"
|
||||
echo
|
||||
if [[ $fails -eq 0 ]]; then echo "all $ran cases behaved"; exit 0; fi
|
||||
echo "$fails of $ran cases FAILED"; exit 1
|
||||
Reference in New Issue
Block a user