Compare commits
61 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 98f3cdd76a | |||
| 888a98571b | |||
| 912f7c127e | |||
| e569ec6c5f | |||
| 52ed52c131 | |||
| c5daa9480d | |||
| d2053e7d47 | |||
| 7d4e564758 | |||
| 9b749ee7ad | |||
| 7ae3081a76 | |||
| 7ecd63cd61 | |||
| a2f3aeeeff | |||
| e7d4ecd607 | |||
| 10ca63aa32 | |||
| 68a2130681 | |||
| bdf25f019c | |||
| c6d9435316 | |||
| f38b03cdb8 | |||
| 3403c87c80 | |||
| 86adbcaf8a | |||
| 24ea960229 | |||
| cff6fb5a45 | |||
| db25b469ba | |||
| b228b44f0e | |||
| 91b9405c81 | |||
| 4c69c25b48 | |||
| c9013bb47d | |||
| 2d1e5d0774 | |||
| f09c53efc1 | |||
| 92d647a6f6 | |||
| d03ab7f1f5 | |||
| b17d1c597d | |||
| 0ae01dfb52 | |||
| dd7cdc09e7 | |||
| 7b0a8b234b | |||
| ce1a4b4758 | |||
| ac90169a5d | |||
| 154d6dcaa9 | |||
| 2f7072050f | |||
| 85e799f360 | |||
| 3a72a4811b | |||
| a165d53c88 | |||
| adaf86ad57 | |||
| 74b5eae5b0 | |||
| 7e82f325b8 | |||
| acccb66bd3 | |||
| de812bc027 | |||
| 3e8ebeb96c | |||
| cefdc731a4 | |||
| e56dcb8a4c | |||
| f277e619e2 | |||
| 386f51edc6 | |||
| 130e3ed882 | |||
| be398f92e8 | |||
| ee71abd1d4 | |||
| 37e98f452b | |||
| b2b82ae828 | |||
| b78a0ff3ac | |||
| 96047453cb | |||
| 4bf5db6875 | |||
| 87977ff40a |
@@ -21,6 +21,7 @@ fast, and wrong.
|
||||
|
||||
This rule used to be duplicated verbatim in felhom-agent/CLAUDE.md with a note explaining that
|
||||
felhom.eu/CLAUDE.md "does not load in an agent-only session". That reasoning was correct before
|
||||
path-scoped rules existed. The single source is now felhom.eu/CLAUDE.md "Code quality rules"; this
|
||||
file is the scoped copy that loads exactly where health checks are written. (2026-08-06)
|
||||
path-scoped rules existed. Deliberate scoped copies now live in felhom.eu/.claude/rules/hub.md and
|
||||
felhom-controller/.claude/rules/gates.md (hub.md's comment names them); none is the single source. This
|
||||
file is the copy that loads exactly where agent health checks are written. (2026-08-06; corrected 2026-10-06)
|
||||
-->
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
unconditional: true
|
||||
---
|
||||
# Unprompted work — rules for any session without a task file
|
||||
|
||||
> Goal sessions, nightly sessions, "work the register" sessions. **A session that starts from
|
||||
> `/goal` or a standing brief inherits these rules exactly as it inherits the gates.** They are the
|
||||
> part of `PROMPT-TEMPLATE.md` that a task file used to carry and a goal does not. Same wording lives
|
||||
> in `felhom.eu`, `felhom-controller`, `felhom-agent` and `app-catalog-felhom.eu` `.claude/rules/`, and in the workspace
|
||||
> root's unversioned `.claude/rules/`; change all five or none.
|
||||
|
||||
## 1. What you may pick up on your own
|
||||
|
||||
- A register row **you or another CC session filed**, with owner CC, at P3 or a bounded P2, that
|
||||
needs **no operator decision**, touches **no customer data by design**, and introduces **no
|
||||
mechanism nobody has measured**. Smallest first.
|
||||
- A defect you find while exercising the product, filed as a row **before** you fix it — **unless it is small**:
|
||||
a small finding is fixed in the session and never filed (the size rule, `OPEN-ITEMS.md` „How a row is filed").
|
||||
- Hygiene: register compression, stale citations, rows with no owner, documents that contradict
|
||||
live source.
|
||||
|
||||
**Not yours, ever, without a task file or an operator word:** money; anything that changes risk to
|
||||
customer data; anything that changes a promise the product makes to a customer; anything that
|
||||
reverses a documented design decision (`documentation/architecture/` — a design decision is not a
|
||||
defect, R-370); anything on DooPlex or ep0; baking or vouching a golden; promoting a
|
||||
catalog version; a new external dependency; **a hub image build or hub deploy in a session the operator does not
|
||||
attend** (operator ruling 2026-10-07, `09` §3 decision 162).
|
||||
|
||||
## 2. When you may decide instead of ask (operator grant, 2026-09-14)
|
||||
|
||||
You may take a decision yourself when **all** of these hold: the architecture folder and the register
|
||||
give a clear direction; your choice follows that direction; it is reversible without customer-data
|
||||
risk; and you can write it in the `09-update-architecture.md` §3 shape — one answerable sentence, the
|
||||
options, what each costs, why this one. **Then record it** as a dated decision in `CONTEXT.md` and
|
||||
the owning architecture document, tagged *decided by CC unattended — operator may reverse*, and put
|
||||
it **first** in the morning note. A decision you cannot write in that shape is one you do not take.
|
||||
|
||||
## 3. The discipline a task file used to carry
|
||||
|
||||
1. **Baselines first.** Read each repo's `main` hash and version from live source before touching it.
|
||||
2. **Read the architecture document for the area, and name it** in the report, before any claim.
|
||||
3. **Red-proof every correctness fix.** A test never seen failing has not been shown to test anything.
|
||||
4. **Live-validate on a Tier-0 box** through the endpoints the UI invokes. `demo-hp` is `ssh hp`.
|
||||
Throwaway apps only; the standing apps and `bentopdf` stay.
|
||||
5. **Evidence off the machine at the end of each phase**, before any revert (R-320).
|
||||
6. **One release per repo per session**, with a CHANGELOG entry (controller: with its `MinAgent`
|
||||
line) and REPORT overwritten. **A controller release needs no golden** (operator ruling 2026-10-15
|
||||
11:26, `09` §3 decision 250; decision 7 stands): push the image only when it is ready for EVERY box (the
|
||||
household's update button offers the highest tag); raise the floor, MinAgent declared, only after it ran
|
||||
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
|
||||
on all three — not at night unless the night brief says so. **Agent releases still stop for the
|
||||
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
|
||||
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
|
||||
Prose is not a record.
|
||||
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
|
||||
9. **Never leave a half-state.** If time runs out, revert to clean and say what was reverted.
|
||||
10. **Teardown, three layers, stated** — machine, host, hub — or "provisioned nothing".
|
||||
11. **Every helper prompt carries the brief's fences in full** (operator ruling 2026-10-07). A helper session (a
|
||||
subagent, a fork, a workflow agent) gets the brief's fence list word for word — every protected machine, every
|
||||
„no", every delivery and Docker limit — not a summary and not „the usual fences". Earned on 2026-10-06 night: two
|
||||
helpers whose prompts carried only part of the fences ran `docker volume prune` on the bench and a Docker-using
|
||||
gate on DooPlex.
|
||||
|
||||
## 4. The morning note
|
||||
|
||||
One screen, plain language, in this order: **decisions you took** (§2) first; what you exercised;
|
||||
what broke and whether you fixed it; rows opened and closed with the register size before and after;
|
||||
what needs the operator, each with what happens if they do nothing. No file paths, no function
|
||||
names, no row numbers as the subject of a sentence.
|
||||
|
||||
## 5. Instruction files
|
||||
|
||||
**Instruction files (`CLAUDE.md`, `.claude/rules/*`) are kept true by the session that finds them wrong**
|
||||
(operator ruling 2026-10-06, `09` §3 decision 150). A session MAY, without asking: correct a stale fact (a command, a
|
||||
count, a version, a path, a description of what a gate does), add a fact it proved, and remove a reference to something
|
||||
that no longer exists. Each edit is named in the report (file, line, before, after, why). A session MAY NOT, without the
|
||||
operator's word: loosen a safety rule, a fence, a „never", a protected machine, a secret rule, or a review step; or
|
||||
remove a rule. When in doubt, it is a rule change, and it goes to the operator. If Claude Code's own permission check
|
||||
asks before such an edit, wait for the operator's click; if it refuses, record that and file the exact line.
|
||||
|
||||
## 6. The system poster stays true
|
||||
|
||||
**A session that changes a fact listed in `architecture/felhom-system-poster.facts.md` (a machine, a
|
||||
role, a traffic path, a backup tier, a time, a retention, a key, a known gap) updates that file in
|
||||
the same commit.** If the change is text only, it also edits the matching text in
|
||||
`felhom-system-poster.html`. If it needs a new drawing, it adds the line
|
||||
**"System poster needs a refresh: <what changed>"** to `STATUS.md`'s "waiting on the operator" list.
|
||||
**The session report names which of the three it did.**
|
||||
|
||||
Why the three-way split: the poster is a drawing made in Claude Design, and **nothing in this
|
||||
repository renders it** — unlike `where-felhom-stands.html`, which has `render_stands.py`. So the
|
||||
facts file is the source of truth and the drawing trails it. `scripts/poster_facts_gate.py` warns
|
||||
when the facts file has a newer commit than the poster; it **never fails a push**, because a refresh
|
||||
needs the operator and another tool, and a gate nobody can clear is a gate people learn to route
|
||||
around.
|
||||
+358
@@ -1,5 +1,363 @@
|
||||
## v0.158.0 — the internal data place; partitions as places; add_internal_place (2026-10-16)
|
||||
|
||||
Design: felhom.eu `architecture/16-storage-physical-identity.md` §5, §6, §7, §9 (operator rulings D3–D5; the D3 sizes
|
||||
ruled 2026-10-16 after the space check, felhom.eu `audits/storage-identity-buildB-2026-10-15/SPACE-CHECK.md`).
|
||||
|
||||
- **The internal data place („Belső tárhely")** (`internal/localapi/internalplace.go`): when the record
|
||||
`/var/lib/felhom-agent/internal-place.json` names a filesystem, the agent mounts it at `/mnt/belso` by fs-UUID like an
|
||||
enrolled drive (startup + every 20 s tick) and records it `enrolled`; `GET /disks` reports it with kind `internal`.
|
||||
It never takes `/mnt/belso` from a household drive, and eject/decommission/format of it are refused (device-mapper on
|
||||
the system disk). Its identity is the system disk's, so build A never counts a copy to `sys_drive` as a second copy.
|
||||
- **Root helper `configs/felhom-internal-place`** (new in the bundle): `plan|create <planned GiB>` — the ruled size rule
|
||||
(reserve max(16 GiB, 10 %) ≤ 48 GiB, made only when ≥ 20 GiB, nothing promised beyond pool − reserve), the only
|
||||
lvcreate/mkfs of a system-disk volume; formats only a `felhom-belso` with no signature. Run by the installer (1.33.0)
|
||||
and by the new signed op.
|
||||
- **Signed op `add_internal_place`** (old one-disk boxes, `16` §9): no parameters (the size is the rule's), destructive
|
||||
class (operator key). Runs the helper with `create 0`, writes the record, mounts.
|
||||
- **Partitions as places** (`internal/storage/partitions.go`): `GET /disks/candidates` lists each partition of a free
|
||||
disk as empty / filesystem / in_use. A data-bearing part's wipe id binds disk id + part number + fs-UUID
|
||||
(`byid:<…-partN>+fsuuid:<uuid>`); both must name the same device at format time.
|
||||
- **`felhom-mkfs-guarded`**: a SIBLING part that is Felhom's own drive (mounted raw at /mnt/<name> AND under
|
||||
/mnt/felhom-drives) no longer blocks formatting the free part next to it; every other mount still refuses; an
|
||||
unlistable disk now refuses (it passed before). First tests of the wrapper: `configs/test_felhom_mkfs_guarded.py`.
|
||||
- `configs/` changed (sudoers `FELHOM_INTERNALPLACE`, the helper, the wrapper, the bundle list) — **a config-bundle job
|
||||
is needed** with the agent update. **Not vouched** (the operator's act).
|
||||
- Released: tag `v0.158.0` (code `912f7c1`), sha256 `a3589d1d…2caba`, bundle `17066273…fb5c76`.
|
||||
|
||||
## v0.157.0 — every storage place knows its physical disk(s); POST /disks/same-disk (2026-10-15)
|
||||
|
||||
Design: felhom.eu `architecture/16-storage-physical-identity.md` §3, §4, §10 (operator rulings D1–D6, 2026-10-15). Spike:
|
||||
felhom.eu `audits/storage-identity-2026-10-15/`.
|
||||
|
||||
- **Identity of a storage place** (`internal/storage/identity.go`): the physical disk(s) under it, from `/sys`, `/run/udev/data`
|
||||
and `/proc/mounts` only — partition → disk, device-mapper / LVM thin → slaves, loop → backing file → its disk. Keys:
|
||||
`eui:` (NVMe), `serial:<bus>:<model>:<serial>`, `wwn:` only without a serial and only with a vendor (OUI ≠ 000000).
|
||||
`proven`, or `unknown` with one reason: `vm` (systemd-detect-virt), `network`, `usb_bridge` (a placeholder serial — the rule
|
||||
is the design's, UNPROVEN on a real enclosure, R-1077), `weak_wwn`, `no_serial`. No block I/O.
|
||||
- **Reported:** `GET /disks` rows and the hub report's `storage_targets` gain `kind`, `physical_disks`, `identity`,
|
||||
`identity_reason` (additive; the golden and the hub's copy change together). A dir storage is judged only on its OWN mount — an
|
||||
absent drive is "not computed", never the root disk.
|
||||
- **`POST /disks/same-disk`** `{a, b}` (guest paths) → `{different, proven, reason, a, b}`. Maps `/mnt/felhom-drives/<name>` to
|
||||
the host mount at exactly that path, and `/mnt/sys_drive`, `/var/lib/felhom`, `/var/lib/docker` to the guest's data volume's
|
||||
storage. Anything else, or an absent drive → `unmapped`. A shared disk key is "same, proven" even when the disk is unknown.
|
||||
Scoped like every `/disks` call.
|
||||
- Tests: fixture `/sys` + udev trees for every spike case (one disk LVM thin, two disks, two partitions, a loop on a file, NVMe eui,
|
||||
vendor-less WWN, a VM, a USB placeholder serial, a network share) and the same-disk call (equal, different, VM, NAS, absent
|
||||
drive, scope); red-proofed (the LVM "diamond" bug was seen failing first; the VM, WWN, own-mount and sys_drive rules broken on purpose).
|
||||
- `configs/` unchanged — no config-bundle job needed. **Not vouched** (the operator's act).
|
||||
|
||||
## Unreleased (2026-10-15, afternoon) — the shared rule file; no code change
|
||||
|
||||
- `.claude/rules/unprompted-work.md` §3 item 6: **a controller release needs no golden** — the floor carries it after the
|
||||
three test boxes; agent releases still wait for the operator's vouch; the golden is weekly (operator ruling 2026-10-15
|
||||
11:26, `09` §3 decision 250; the same morning's „bake first", decision 247, reversed). Same text in all five copies.
|
||||
|
||||
## Unreleased (2026-10-15) — the shared rule file; no code change
|
||||
|
||||
- `.claude/rules/unprompted-work.md` §3 item 6: **bake first** — a controller release is released only when its golden is
|
||||
baked; bake → the operator vouches → the operator raises the floor; never ask for a floor for a version with no golden
|
||||
(operator ruling 2026-10-15, `09` §3 decision 247, superseding decision 7). Same text in all five copies.
|
||||
|
||||
## v0.156.1 — R-1008 by process liveness, not the clock; 0.156.0 superseded — do not vouch it (2026-10-14)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `83c7da27d457b63ba8a4cde5a8717b3e5f9db9f83a46e63f38902dc7471bffb9`, bundle
|
||||
`5a678def792c70ec1cbbd198c14542161fac91d11a69bd0ce23dd86469e96ec3` (tag `v0.156.1` = `7ae3081`). Step bundle `0.156.1-step1`
|
||||
sha256 `ac83ea41507616d4c1056cccfd35de1410ff01e0296f4e64ff925207a43a8490` (the 0.155.0 bundle with only `felhom-os-apply`
|
||||
replaced), published as package version `0.156.1-step1`. **Not vouched** (the operator's act). Delivery as 0.156.0: per box
|
||||
`agent_update 0.156.1` → `agent_config_update 0.156.1-step1` → `agent_config_update 0.156.1`.
|
||||
|
||||
- **0.156.0 is superseded — never vouch it.** A background security review of the pushed commit flagged „fail-open state
|
||||
drift": 0.156.0 called a listed vzdump task dead when its wall-clock start was before the kernel's `btime`. `btime` is
|
||||
derived from the wall clock and moves when the clock is corrected after the boot (an NTP step after a power cut), so a
|
||||
backup started between the boot and the correction could look „from before the boot" — and the stale-lock pass would
|
||||
clear a LIVE backup's lock.
|
||||
- **Now:** `taskAlive` is Proxmox's own rule (`PVE::ProcFSTools::check_process_running`, read on demo-hp): the UPID's pid and
|
||||
pstart against `/proc/<pid>/stat` — same start tick (field 22) and not a zombie. No clock is involved. Fails safe (running)
|
||||
when the UPID does not parse, the stat cannot be read, or this agent cannot see root's processes at all (PID 1 not visible:
|
||||
a `hidepid` /proc or `ProtectProc=` would hide every worker — demo-hp: `/proc` plain, `ProtectProc=default`).
|
||||
- Tests: worker gone, pid reused, zombie, live worker, a „clock step" (wall-clock start absurdly early, worker alive → still
|
||||
running), doubt, hidden /proc — each red-proofed; 19 guards in all (`power-cut-fixes-2026-10-14/redproof.txt`).
|
||||
- R-1006 (the cleanup) is unchanged; its „before this boot" rule still reads `btime`, but a drifted `btime` cannot remove a
|
||||
live partial: the Go side asks this liveness rule, and the root helper refuses while any vzdump runs or anything holds
|
||||
the file open.
|
||||
|
||||
## v0.156.0 — after a power cut: the dead half-written backup is removed (R-1006); a backup task from before the boot is not „in flight" (R-1008) (2026-10-14)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `8db2278838dcb733d5201fba84b75a829b9adebdf899c89dd7de0939ec10d51a`, bundle
|
||||
`a21605217966ebc953d55a35157a9eb0243ca542c50dbd94fc9fa67257f27369` (tag `v0.156.0` = `a2f3aee`). Step bundle `0.156.0-step1` sha256
|
||||
`8be2bb8f3d54bff8e7140cf99714664df31adecf13aa721597323fb2b9d43db5` (the 0.155.0 bundle `795f2ab4…` with only `felhom-os-apply`
|
||||
replaced; `scripts/build-step-bundle.py`), published as package version `0.156.0-step1`. **Not vouched** (the operator's act).
|
||||
|
||||
**Delivery: the agent binary AND the config bundle, and the bundle ADDS a path** (`/usr/local/sbin/felhom-vzdump-partial-clean`)
|
||||
— so per box: signed `agent_update 0.156.0` → `agent_config_update 0.156.0-step1` (the step bundle) → `agent_config_update
|
||||
0.156.0`. Evidence: felhom.eu `documentation/audits/power-cut-fixes-2026-10-14/` (the spike C1–C4, the red-proofs).
|
||||
|
||||
- **R-1008 — one in-flight answer.** `staleLockController.BackupRunning` (the ONE implementation all four callers reach:
|
||||
the start-up stale-lock pass, the guest-power watchdog, the household power-off guard, the controller supervisor) now
|
||||
answers through `vzdumpInFlight`: a vzdump task for the vmid that started at or after the host's boot. Proxmox keeps a
|
||||
dead task in `tasks?source=active` for ~30–60 s after a power cut (measured, C3) — the agent starts inside that window
|
||||
and called it „genuinely in-flight". A task from before the boot is dead; logged once (vmid, UPID, start, boot). The
|
||||
start time comes from the list or, if absent, the UPID; an unreadable boot time or start time FAILS SAFE (running), as
|
||||
before. The boot time is `poweroff.ProcBtime` (no third /proc/stat parser).
|
||||
- **R-1006 — the half-written backup is removed.** New `internal/vzpartial`: on a `dir` backup target it finds exactly
|
||||
`vzdump-(lxc|qemu)-<vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp)` of THAT guest (never a finished archive, a log,
|
||||
another guest, a symlink, a wrong kind), and — only when no vzdump of the guest is in flight (R-1008) and the entry is
|
||||
dead (modified before this boot, or untouched for 120 s) — asks the root helper to remove it. Runs at agent start (after
|
||||
the stale-lock pass, every owned guest × every backup tier) and before every backup's space preflight
|
||||
(`BackupRunner.SetBeforeBackup`). Any doubt leaves the entry and logs why.
|
||||
- **The root helper `configs/felhom-vzdump-partial-clean`** (config bundle; sudoers `FELHOM_VZPARTIAL`, one exact line: a
|
||||
storage id, a vmid, one half-written name) re-checks everything as root: a `dir` storage from `/etc/pve/storage.cfg`; the
|
||||
name of that vmid; not a symlink; directly in `<path>/dump`; **no vzdump on the host at all** (a job for several guests
|
||||
carries no vmid in its task name); **no process holds the entry open** (C2: after a killed worker, `tar` and `zstd` kept
|
||||
writing into the `.tar.dat` for 4 min 40 s, none named „vzdump"); dead (before this boot, or quiet 120 s).
|
||||
- **Not changed, by evidence (C4):** the space preflight's sizing (R-1006 part 2) — on „No space left" vzdump removes its
|
||||
own partial on both its paths (rsync and tar); only a power cut leaves one.
|
||||
- **Tests:** `stalelock_inflight_test.go` (the helper, the production controller, three callers end to end, a structural
|
||||
pin that every `BackupRunning` call goes through `s.staleLock` and there is one implementation), `vzpartial_test.go`
|
||||
(removes; keeps in flight / unknown / non-dir / other names / recent / wrong kind / symlink / a helper refusal — each keep
|
||||
with its positive control), `r685_space_test.go` (the cleanup runs BEFORE the preflight), the helper's own 21 Python
|
||||
tests (`configs/test_felhom_vzdump_partial_clean.py`, run by the Go suite), 8 sudoers injection cases. **17 guards
|
||||
red-proofed** by mutation (`power-cut-fixes-2026-10-14/redproof.txt`).
|
||||
- **Fixed without a row:** `internal/localapi/guestbind.go` and `controllersupervisor_test.go` were not gofmt-clean on
|
||||
`main` (an import order and a struct alignment) — formatted.
|
||||
|
||||
## v0.155.0 — the household can turn the machine off: POST /host/poweroff (R-998); the escrow reply race fixed (2026-10-13)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `bb339e944eff4aa6d10ae09158e0943de2e5c3777f52bf993c796d57e46800c3`, bundle
|
||||
`795f2ab41661e993e3fc4ce65df4f11ce94280d94871ab2e5cf1993ae3fbff7d` (tag `v0.155.0`). **Not vouched** (the operator's act).
|
||||
**The bundle changes `felhom-agent.sudoers`** (no new path — no step bundle): delivery is a signed `agent_update` +
|
||||
`agent_config_update`. Design: felhom.eu `documentation/architecture/14-host-power-off.md`.
|
||||
|
||||
- **`POST /host/poweroff`** (local API, `withGuest`-scoped like `/guest/reboot`) — the controller's „A gép kikapcsolása"
|
||||
(controller v0.310.0). Refused (409, a stable `reason`) in the first 10 min after boot; a second time in one boot
|
||||
(`internal/poweroff`, a file keyed by `/proc/stat` btime); while ANY other guest or VM runs on the host (read from
|
||||
`/proc`: `lxc-start -n N`, `kvm -id N` — demo-hp's scratch guest and Tester 1 VM block it); while the heavy-operation
|
||||
gate is held (backup, restore test, OS step, sweep, trim), a backup job, a controller swap, a drive format, a NAS verify
|
||||
or an escrow ceremony runs, or Proxmox runs a vzdump of the guest. Every check fails closed.
|
||||
- On acceptance: the record first; the heavy gate TAKEN and kept; a host report carrying `deliberate_off.requested_at`
|
||||
accepted by the hub (bounded 60 s, `Loop.SetReportedHook`); then before each step the jobs and other guests are
|
||||
checked again — a job waits (≤ 15 min), another guest cancels at once, a job still running after the wait cancels
|
||||
(gate released, box stays on); `pct shutdown <vmid> --timeout 120`; `systemctl poweroff`.
|
||||
- **Sudoers:** `FELHOM_POWEROFF` = `/usr/bin/systemctl poweroff`; `pct ^shutdown [0-9]+ --timeout [0-9]+$` joins
|
||||
`FELHOM_GUESTHOOK`; nine injection cases (`--forceStop`, `poweroff --force`, `reboot`, `halt`, `kexec`…).
|
||||
**New `TestEveryAliasIsGranted`:** an alias missing from the grant line passed the coverage check before.
|
||||
- **Host report:** `deliberate_off` (omitempty; the cross-repo golden unchanged). Hub v0.149.0 holds its silence alarms
|
||||
for 14 days on it.
|
||||
- **Fixed without a row:** `go test -race` found a data race in the escrow ceremony start reply (it read `job.Phase`
|
||||
while the job goroutine wrote it); the reply fields are copied first. Red before, green 3/3 after.
|
||||
- Security review (three passes the same evening): other guests, a gate not held, fail-open unwired checks, jobs and
|
||||
guests starting after acceptance — each fixed and red-proofed (`hostpoweroff_test.go`, 11 tests).
|
||||
|
||||
## v0.154.0 — no OS leg after a household press; the after-boot kernel report carries the real ring (R-899); no „wrong code" when older packages were not checked (R-304); the last backup survives a restart in the host report; the Secure Boot meta-package leaves the Proxmox lane (2026-10-09)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `36a54ad20e896cd61d7b9944e2a521f4209c2b76940d8f1827af05b645463226`, bundle
|
||||
`93487989f50d4a3039a05cec0f91ec478d62489a414705e49403e9b1340d20e0` (tag `v0.154.0` = `db25b46`).
|
||||
|
||||
**Delivery: the agent binary AND the config bundle** — the wrapper `felhom-os-apply` changed (2026-10-09).
|
||||
|
||||
- **The hub's backup evidence survives a restart (2026-10-09, found in the kernel-night read-back).** The host report's
|
||||
`backups` list came only from memory, which is empty after an agent restart. On demo-felhom the night backup landed
|
||||
02:40 UTC and the kernel step restarted the host at 02:44 — no report fell in between, two nights running — so the hub
|
||||
alarmed „host tier: newest backup is 48h old" at 03:00 after a good backup. The report now adds R-894's saved newest
|
||||
success per tier and guest (`backup-success-state.json`, ONE instance shared with the local API) where memory holds no
|
||||
success at or after it. Only successes are saved, so a failure is never hidden. `hub.KnownBackupReporter`,
|
||||
`BackupSuccessState.KnownBackupSuccesses`; tests `TestCollectBackups_SavedSuccessSurvivesARestart` (red-proved: the
|
||||
restart case reported nothing), `TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen`, and
|
||||
`TestR894_LastKnownBackupsIsWiredIntoTheDaemon` now also asserts the collector gets the same instance (red-proved).
|
||||
- **`proxmox-secure-boot-support` is a boot-chain package (2026-10-09).** On demo-hp (Secure Boot on) its upgrade pulled
|
||||
`shim-signed-common`; in the ring-0 Proxmox plan that refused the whole step (R6) and skipped the night's kernel step,
|
||||
after the household had been told „tonight". It joins `HOST_SLOW_RE` with shim and GRUB (they stay out of every lane,
|
||||
as before). `test_ring0_pending_pve_leaves_the_secure_boot_meta_out` (red-proved).
|
||||
|
||||
- **R-899 (operator ruling 2026-10-08, option A):** `POST /backup?trigger=manual` (a „Mentés most" press, sent by the
|
||||
controller from its next release) runs no OS leg after it: the leg belongs to the night, after the night's own copy.
|
||||
Before, a press ran the leg at once (in the day; on 2026-10-07 08:51 demo-hp skipped it only by the 20-hour rule). A
|
||||
request without the parameter (an older controller, the scheduled path) behaves as before.
|
||||
`internal/localapi/server.go`; `TestAfterPrimaryBackup` gained two sub-cases (red-proved: ignoring `trigger`, the leg
|
||||
ran once after a press).
|
||||
- **R-304 — „wrong code" only when every earlier package was tried.** `POST /escrow/recover-offsite-password` answered
|
||||
400 („the recovery code did not open the sealed bundle") whenever the current package and the TRIED earlier packages
|
||||
refused the code — even when the hub withheld earlier packages (rows with no key material, rows over its serve cap), a
|
||||
served package was malformed, the 6-attempt cap stopped the loop, or the retained list could not be read at all. Now
|
||||
those cases answer **424** with `older_unchecked` (the count, -1 = unknown) and a sentence that does not call the code
|
||||
wrong. `escrow.ErrRetainedUnchecked` / `RetainedUncheckedError`; the retained fetcher's second value is now every
|
||||
withheld package (unopenable + truncated + malformed). Tests `TestR304_*` (real age crypto; red-proved: without the
|
||||
check, four cases returned the wrong-code error) and a 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus`.
|
||||
An older controller maps the unknown 424 to its neutral „we do not know why" sentence.
|
||||
- **The „ring 1" label after a boot:** in the first second after a reboot the agent has not fetched the hub's block, and
|
||||
the after-boot kernel reports (`judging`, `good`, `revert`, `fell_back`…) said ring 1 on a ring-0 box (demo-felhom,
|
||||
2026-10-08 night). Now they read the fetched block, else the block the daemon saved on disk before the reboot (R-866),
|
||||
else ring 1 as before. A label only — the hub's approval reads its own ring list. `kernelReportRing`,
|
||||
`TestKernelReportRing_BeforeFirstFetch` (red-proved: ring 1, want 0).
|
||||
|
||||
## v0.153.0 — ring 0 stages exactly the told kernel (R-898; `09` §3 decision 176) (2026-10-07)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `b204ebe6d65944f43b70dcfa4ae0dfb90da38c92e2ba6a38a1826e488eeb6630`, bundle
|
||||
`6db216275eb0dd39188d93481a2045998a69e8cb7838ad908a58d65d9a6a9b57` (tag `v0.153.0` = `2d1e5d0`). Delivered (binary only — the
|
||||
bundle files are unchanged from 0.152.0) to demo-hp, demo-felhom, Tester 1 on 2026-10-07 19:03.
|
||||
|
||||
**Delivery: the agent binary only** — no root file changed (the wrapper is unchanged; its tests gained two cases).
|
||||
|
||||
- `internal/osupdate/kernel.go`: ring 0's night kernel step stages EXACTLY the kernel the household was told about
|
||||
(select `listed`, `KernelSet(kver)` = the series meta-package and the signed image at the kernel's own version) instead
|
||||
of "whatever is pending tonight". Seen 2026-10-07: demo-felhom was told about 7.0.14-20 while its sources offered
|
||||
7.0.14-22 by night — the old code staged `pending-kernel` and the wrapper refused it (R23), losing the night. A told
|
||||
version that is no longer installable is refused by the wrapper before any change (R7) and the hub tells the household
|
||||
again for the newer kernel (hub v0.143.1). Tests `TestKernel_Ring0ToldNightStagesThenReboots` (red-proved against the
|
||||
old select), `TestKernelSet`; wrapper `test_ring0_listed_installs_the_told_kernel_not_the_newest`,
|
||||
`test_ring0_told_kernel_gone_is_refused_before_any_change`.
|
||||
|
||||
## v0.152.0 — the kernel lane (R-836; `09` §3 decisions 164, 172; `11` §5.11) (2026-10-07)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `95ff42208e36ba49b6e2b09a97e81a6fa11562ecc8042f1ed18d378b8b1f88b1`,
|
||||
config bundle sha256 `f0c2cec374b711b3c131c955012b33fd0ad495337d049b7a743c3eef9e85c20b` (tag `v0.152.0` = `d03ab7f`).
|
||||
Step bundle `0.152.0-step1` sha256 `0b71d32b054cf3b7ade0234ffcbb0df159901f542cde540adaee411db466f48e` (the 0.151.0 bundle
|
||||
with only `felhom-os-apply` replaced; `scripts/build-step-bundle.py`), published as package version `0.152.0-step1`.
|
||||
|
||||
**Delivery: agent binary, then the STEP bundle `0.152.0-step1`, then the bundle `0.152.0`** — the bundle ADDS two paths
|
||||
(the GRUB generators), and an installed `felhom-os-apply` refuses a path its own table lacks (R16, R-880).
|
||||
|
||||
A new kernel boots ONCE; if it crashes the box comes back on the old kernel by itself; it becomes the default only after
|
||||
a healthy boot; a booted-but-unhealthy kernel is reverted ONCE by the agent with no person. Built on the spike's
|
||||
candidate 2 (`audits/kernel-spike-2026-10-07/`), with option C on the one-shot entry.
|
||||
|
||||
- `configs/felhom-grub-oneshot.sh` → `/etc/grub.d/01_felhom_oneshot` (bundle): reads `felhom_next` from a GRUB env block
|
||||
on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and sets the default to that kernel's
|
||||
one-shot entry only when the name is an installed kernel. No vfat ESP → prints nothing.
|
||||
- `configs/felhom-grub-oneshot-entries.sh` → `/etc/grub.d/42_felhom_oneshot` (bundle): one entry per installed kernel,
|
||||
id `felhom-oneshot-<ver>`, the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`
|
||||
(option C). Sorted after `10_linux`: never entry 0, never the default.
|
||||
- `configs/felhom-os-apply`: layer `kernel` (lane slow; an appliance; authority = a signed `os_kernel_step` or the
|
||||
root-owned ring-0 mark). Modes: `apply` STAGES (select `pending-kernel` or a signed `listed` set; `expect_kver` = the
|
||||
kernel the household was told about): pins the GRUB default to the RUNNING kernel in
|
||||
`/etc/default/grub.d/zz-felhom-kernel-default.cfg` and proves it from grub.cfg, installs, proves the default did not
|
||||
move and the one-shot entry exists, writes the flag; never reboots. `kernel-reboot` (a staged step only),
|
||||
`kernel-boot` (judging | fell_back | self_reverted | revert_failed), `kernel-good` (the new kernel becomes the
|
||||
default, proved), `kernel-revert` (ONE per step; refused when the default is not the old kernel), `kernel-cancel`,
|
||||
`kernel-status`. New refusals: R20 (the box cannot do a one-shot: not UEFI, no vfat ESP, a separate /boot, GRUB
|
||||
without fat/loadenv, the generators missing, a hand pin), R21 (the crash guard tripped or an unclean boot in its
|
||||
window), R22 (the phase does not allow the mode; never two steps within 20 h), R23 (not exactly one newer kernel, or
|
||||
not the one signed / told). State `/var/lib/felhom-kernel/state.json`. Facts carry `kernel_lane`; the next-boot
|
||||
kernel reads the flag and the grub.cfg default. Tests: `KernelLane` (27), red-proof
|
||||
`felhom.eu/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`.
|
||||
- `configs/felhom-crash-guard` unchanged; `KernelStepCannotLeaveTheBoxOff` (3 tests) pins that a step's planned reboot,
|
||||
one crash and one self-revert add ONE unclean boot (a panic before userspace adds none), so the box cannot stay off.
|
||||
- `internal/osupdate/kernel.go`: the night leg ends with the kernel step — after a healthy host step (and a healthy
|
||||
Proxmox step when one ran), trigger `night` only, on a night the hub's `os_update.kernel` block marks `tonight` (the
|
||||
household was mailed the day before — no mail, no step). Ring 0 stages + reboots; ring 1 reboots only a kernel a
|
||||
signed `os_kernel_step` staged (`KernelStepExecutor`: stage only, under the heavy-op gate). The hub hears `staged`
|
||||
BEFORE the reboot. At every start `KernelAfterBoot`: on the new kernel it JUDGES the boot — `KernelVerdict` = the
|
||||
host health rule (`11` §8.2) AND the box reached the hub (the `judging` report itself) — for 20 minutes (measured:
|
||||
everything healthy 68 s after the reboot on demo-felhom, 272 s on demo-hp; under the hub's 45-minute `host_stale` (`alerting.stale_threshold`)).
|
||||
Healthy → `kernel-good`, outcome `applied`; not healthy → outcome `health_failed`, then ONE `kernel-revert`.
|
||||
Tests: `TestKernel*` (13); red-proofs in the same file.
|
||||
- `internal/hub`: `WireOSUpdate.Kernel` {kver, tonight, notified_at}. `internal/reconcile`: `os_kernel_step` is
|
||||
destructive-class. `cmd/felhom-opsign`: the op is listed.
|
||||
|
||||
## v0.151.0 — the agent can no longer hand the guest any image; the Proxmox package lane; the other-key archives reported; the DR directive retired (R-861, R-812 A, R-366, R-105; `09` §3 163, 165, 168, 169) (2026-10-07)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `0464354f2cdf452a7c5d2a74d9191fe91415fcfa244154480d26d5b30e10b194`
|
||||
config bundle sha256 `bacd1d175a9392bc1755341d01a106abbd72aba48ab575e7a32dd819d8f5da4c` (tag `v0.151.0` = `dd7cdc0`).
|
||||
**Deliver the binary FIRST, then the bundle:** the bundle's sudoers removes the `tee` grant the 0.150.0 binary still uses.
|
||||
No path added (26 → 26), so no step bundle.
|
||||
|
||||
### Part of v0.151.0 — the agent can no longer hand the guest any image; felhom-op's pct lines are exact (R-861 (a) A1, (b) B2; `09` §3 decision 165)
|
||||
|
||||
**Delivery order: agent binary FIRST, then the config bundle.** The new sudoers drops the agent's in-guest `tee`
|
||||
grant; an older binary still calls `tee`, so a bundle that lands before the binary would stop managed controller
|
||||
updates (and the old binary's capability probe would read `controllerswap-write` degraded). No bundle path is added
|
||||
(`felhom-priv-apply` and `/etc/sudoers.d/felhom-op` are already bundle files), so no step bundle.
|
||||
|
||||
- `configs/felhom-priv-apply`: new verb `controller-image <vmid>` — reads the ref on stdin (≤ 256 bytes, ASCII, one
|
||||
optional trailing newline), requires `^gitea\.dooplex\.hu/admin/felhom-controller:[0-9]+\.[0-9]+\.[0-9]+$` (the agent's
|
||||
own `controllerImageRe`), then runs `pct exec <vmid> -- tee /etc/felhom-controller-image` AS ROOT; refusal rule `I1`
|
||||
(rc 3), a bad vmid `A1` (rc 2); listed in `--self-check`.
|
||||
- `configs/felhom-agent.sudoers` `FELHOM_CONTROLLERSWAP`: `pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$`
|
||||
REMOVED; `/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$` added.
|
||||
- `internal/localapi`: `GuestExecutor.GuestExecStdin` replaced by `WriteControllerImage`; `GuestBinder.WriteControllerImage`
|
||||
pipes `ref\n` to the verb through the fenced runner; the swap's `writeImage` calls it. Capability `controllerswap-write`
|
||||
now probes the verb.
|
||||
- `configs/felhom-op.sudoers` (B2, hygiene): `pct start|stop|unlock [0-9]*` → `pct ^start [0-9]+$` etc. (the glob's `*`
|
||||
matched spaces: `pct stop 9201 --skiplock 1` passed).
|
||||
- Tests: `ControllerImage` (5, `configs/test_felhom_priv_apply.py`), `TestSudoersRefusesTheR861Injections` (+3 lines),
|
||||
`TestSudoersAllowsTheControllerImageVerb`, `TestFelhomOpSudoersPctIsExact`, `TestR861_WriteControllerImageUsesTheRootVerb`,
|
||||
`TestControllerSwap_WriteViaRootVerb_NoShell`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/C/`.
|
||||
- `README.md`: the controller-swap paragraph described the removed `tee` path — corrected.
|
||||
### Part of v0.151.0 — the Proxmox package lane (R-812 option A, `09` §3 decision 163)
|
||||
|
||||
**MinAgent impact: none** (a new layer; an older hub ignores the pve report). **The bundle carries the new
|
||||
`felhom-os-apply` — deliver it with the binary** (signed `agent_update`, then signed `agent_config_update`).
|
||||
|
||||
- `configs/felhom-os-apply`: new layer `pve`, lane `slow` only — the host's Proxmox USERSPACE packages: origin `Proxmox Debian Repository` only (R2), never a kernel / boot / firmware / microcode name (R14, `HOST_SLOW_RE` — the kernel is R-836's lane), no removal (R4), no undo (R5), a new package only from `PVE_NEW_ALLOW` (`proxmox-firewall-data`, measured on demo-felhom; R6 otherwise), an appliance only (R12), authority = a signed `os_pve_step` or the root-owned ring-0 mark (R3). Select `pending-pve` (ring 0): installed Proxmox-origin packages with a pending upgrade. The report carries `pve_manager` (pveversion after the step).
|
||||
- `internal/pvegate` (new): the agent's own writes to /etc/pve wait while a pve step runs (pmxcfs restarts); the step waits for writes in flight (bounded, 2 min — then it fails and does not run). Wired at `proxmox.Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile).
|
||||
- `internal/osupdate`: `LayerPVE`; the night leg runs the pve step in ring 0 after a healthy host step (an appliance; ring 1 never in the night leg); `PVEHealthVerdict` = the host rule + every running container keeps its id + pveversion reads the installed pve-manager; the pve report carries Proxmox userspace only (the hub's candidate set). `PVEStepExecutor` (signed `os_pve_step`, ring 1, under the heavy-op gate and the /etc/pve gate); `reconcile.ClassOSPVEStep` (destructive-class); `felhom-opsign -op os_pve_step` (params by `-params`).
|
||||
- Tests: wrapper `PVELane` (17; red first — the `pve-manager` plan was refused R12 on the old code), `pvegate` (5), `TestPVEGate_*` + `TestWritesEtcPVE`, `TestPVE_*`, `TestPVEHealthVerdict`, `TestPVEStepExecutor_*`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/B/`.
|
||||
### Part of v0.151.0
|
||||
|
||||
- R-366 slice 2 (`09` §3 decision 168): the restore-test pick records, per tier, the archives it skipped as written with another key (count, oldest, newest — no key material) in a `ForeignKeyLedger`; the host report carries it as `foreign_key_archives.tiers` (the stanza absent until a tier was evaluated since start, `tiers: []` when none — no null on the wire, the report contract forbids it). The hub turns a change into one operator line. Tests `TestR366_PickRecordsArchivesWrittenWithAnotherKey`, `TestR366_EvaluatedWithNoneIsAnEmptyList` (red-proved, `felhom.eu/documentation/audits/day-2026-10-07/E/`).
|
||||
- R-105 option A (`09` §3 decision 169): the `--selftest=escrow-create -directive <file>` flag and the escrow upload's `directive` field are removed — nothing read the directive; the DR path reads the recipe, tenantsync and the escrow blob. The hub ignores a `directive` from an older agent.
|
||||
|
||||
## v0.150.0 — the Docker step proves the engine reports a memory kill; after a restart the agent remembers the last backup per tier; three more SMART counters on the wire (R-528, R-894, R-330; `09` §3 decisions 157, 161) (2026-10-07)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `a23d1c9085bc7fd4fc48fe0327f6504aa83e6331510fb4a3d23e042dddb26f9c`
|
||||
config bundle sha256 `88456b386d9b1027bd22861cac8c23df004bf9fd9f67644d6595bfca8c94498e` (tag `v0.150.0` = `3a72a48`).
|
||||
**The bundle carries the new `felhom-os-apply` (the memory-kill check) — deliver it with the binary:** signed
|
||||
`agent_update`, then signed `agent_config_update`. No path added (26 → 26), so no step bundle.
|
||||
|
||||
### Part of v0.150.0 (2026-10-06 night, later) — after a restart the agent remembers the last backup per tier (R-894); three more SMART counters on the wire (R-330)
|
||||
|
||||
Ships with the memory-kill check below as v0.150.0, AFTER the 2026-10-07 night read-back. Nothing delivered tonight.
|
||||
|
||||
- **The defect (measured 2026-10-05 on demo-hp):** the agent restarted at 04:57; at 06:25 the off-site storage answered *Can't connect*; the per-tier backup record is in memory only, so the due-check fell back to an EMPTY record and the 7-day tier (last copy 4 days old) read DUE; the controller asked and vzdump failed.
|
||||
- New `internal/backup/backup_state.go` `BackupSuccessState`: the newest SUCCESSFUL backup per tier and guest, on disk (`<oob state dir>/backup-success-state.json`, atomic tmp+rename, 0600). Only successes are written; a corrupt file reads as nothing known.
|
||||
- `internal/localapi` `handleBackupDue`: when the tier's storage CANNOT be read, the saved copy stands in for the in-memory record. A fresh copy → not due („… (storage unreadable — age from the last success saved on disk)"); a copy older than the cadence → DUE; no copy → the old answer (DUE, age unknown). A storage that answers stays the ground truth: an archive absent there is due even when the file remembers one.
|
||||
- Wired in `buildLocalAPIServer` (`LastKnownBackups`); the local API's backup job saves each success.
|
||||
- Tests: `TestBackupDue_R894_*` (restart = a new server and a new state from the same file; fresh / old / none / storage answers / failed backup not saved), `TestBackupSuccessState_*`, `TestR894_LastKnownBackupsIsWiredIntoTheDaemon` (AST). Four red-proofs observed (`felhom.eu/documentation/audits/night-burndown-2026-10-06/s4/`).
|
||||
- **R-330 (disk health Phase 2, the wire only):** the SMART summary carries three more SATA raw counters — `reported_uncorrect` (187), `command_timeout` (188, carried as the vendor reports it; some pack several counters), `udma_crc_errors` (199). Pointer + omitempty: an attribute the drive does not report is OMITTED (unknown), never 0. No verdict reads them yet. Tests `TestParseSMART_R330_*` (two red-proofs, `felhom.eu/documentation/audits/night-burndown-2026-10-06/r330/`).
|
||||
|
||||
### Part of v0.150.0 (2026-10-06 night) — the Docker step proves the engine reports a memory kill (`09` §3 decision 157, R-528)
|
||||
|
||||
To be released as v0.150.0 with its config bundle AFTER the 2026-10-07 night read-back (the night of 2026-10-06 runs v0.149.0 on purpose).
|
||||
|
||||
- `configs/felhom-os-apply`: after a docker-layer APPLY (after `health_after`) the wrapper runs `oom_check()`: a throwaway container from the image the running controller uses (`--pull never`, `--network none`, no volume, label `felhom.oomcheck=1`, 64 MB cap) asks for one 200 MB block; „pass" only when `OOMKilled=true` AND the `oom` event; it waits 2 s and reads the events window to the guest's epoch + 1 (measured: a window closed in the same second missed the event); the container is always removed. Reported as `oom_check`; it never changes the step's outcome or health. A wrapper-only mode `oom-check` runs the check alone (no apt, no engine change), by hand as root.
|
||||
- `internal/osupdate`: `WrapperReport` and `Report` carry `oom_check` verbatim, on the normal pass and on the kept-copy path (R-868).
|
||||
- `configs/test_felhom_os_apply.py`: its `unittest.main()` sat in the middle of the file, so 11 tests (UnsentReport, SaveReportOnDisk, AgentDiesMidPass, CrashLeftTheJournal) never ran — moved to the end; all pass.
|
||||
- Tests: the OOMCheck class (pass, OOMKilled=false, no event, unreadable image, removal on an inspect error, not on other layers or in health mode, the events window after the settle wait, mode oom-check alone and its refusals); TestDocker_OOMCheckReachesTheHubUnchanged, TestR868_KeptCopyCarriesTheOOMCheck. 12 red-proofs in `felhom.eu/documentation/audits/readback-2026-10-07/F/`.
|
||||
|
||||
### Part of v0.150.0 (2026-10-06 evening) — the shared rule file (`09` §3 decision 152); no code change
|
||||
|
||||
- `.claude/rules/unprompted-work.md` added, byte-identical to the copies in felhom.eu, felhom-controller, app-catalog-felhom.eu and the workspace root (checked with `diff` against the controller's copy and one md5 across all five). Its copies line names five copies.
|
||||
|
||||
### Part of v0.150.0 (2026-10-06 afternoon) — instruction files kept true (`09` §3 decision 150); no code change
|
||||
|
||||
- `CLAUDE.md` „Gates — ONE entry point": the runner runs every gate in its `GATES` table (five: three shared, `published`, `release-complete`); `--fast` skips `published` (network). It said two gates and „all of them".
|
||||
- `CLAUDE.md`: the decoy gate and its audit are named with their `felhom.eu/` prefix (they do not exist in this repo).
|
||||
- `.claude/rules/health-checks.md` (comment): the health-check rule's copies live in felhom.eu `hub.md` and the controller's `gates.md`; it named felhom.eu `CLAUDE.md` „Code quality rules", which holds no such rule.
|
||||
|
||||
## v0.149.0 — a weekly disk trim of each customer guest, the crash-boot fact for the controller, the phantom WARN names its runbook (R-444, R-856, R-99; operator rulings `09` §3 139, 143, 140) (2026-10-06)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `6bcae9c2eb5d97e8285316583870059835793893299e291891a53a4ce505585f`
|
||||
config bundle sha256 `e182c82dcf4a67faa3bcb74dbe4ffa7b06e0b27dc8451cb7574d6339ce91ad66` (tag `v0.149.0` = `f277e61`).
|
||||
**The bundle carries the new sudoers rule for the trim (`FELHOM_FSTRIM`) — deliver it with the binary:** signed
|
||||
`agent_update`, then signed `agent_config_update`.
|
||||
|
||||
- R-856 (`09` §3 decision 143): new local-API route `GET /host/crash-guard` — passes the host crash guard's last-boot record (present, last_boot_at, last_boot_unclean, tripped) from /var/lib/felhom-crash-guard/state.json to the controller, which waits ~15 min with app mails after a crash boot. Read-only, no Proxmox call, guest-token authed; a missing/unreadable/garbled file answers 200 present:false (never an error page). An older agent answers 404, which the controller reads as unknown (normal 90 s grace) — no controller MinAgent raise needed.
|
||||
- R-444 (`09` §3 decision 139): weekly guest disk trim. New sudoers alias FELHOM_FSTRIM with ONE exact rule `/usr/sbin/pct ^fstrim [0-9]+$` (rides the signed config bundle; decoys pinned by TestSudoersFstrimRuleIsExact) and capability guest-fstrim (non-critical). New internal/fstrim job: each owned RUNNING guest gets `pct fstrim <vmid>` once a week - due Wednesday from 10:00 host-local, starts only 10:00-20:59 (never the 01:00-06:59 night), holds the one-heavy-op gate so it never runs beside a backup or restore-test (busy -> deferred to the next hourly tick; a box that was off catches up at its next daytime hour); a failed trim WARNs and is retried at most 3 times that week; bytes parsed from `pct fstrim`'s "(N bytes) trimmed" lines; positive log `fstrim: guest N trimmed X GiB in Ys`; last result per guest persisted in <state_dir>/guest-disk-trim.json and reported as the new omitempty host-report stanza `guest_disk_trim`. Opt-out: agent.json "disk_trim": {"disable": true}.
|
||||
- R-99 (`09` §3 decision 140): the agent's WARN for a PBS archive below the 1 MiB plausibility floor now ends with the pointer to the sanctioned cleanup (`documentation/runbooks/pbs-phantom-cleanup.md`); detection only — nothing is deleted automatically. Dir-storage archives keep the old text.
|
||||
|
||||
## unreleased
|
||||
|
||||
- R-426: scripts/test_gate_decoys.py (new) — the published gate judged against a fake Gitea (127.0.0.1, via GITEA_BASE; never the real registry): 11 cases; COVERS published — `felhom-agent/published` leaves the decoy-coverage EXEMPT list.
|
||||
- R-426: release-complete gate — 10 decoy cases (scratch clone + scratch bare origin + fake Gitea); COVERS release-complete — `felhom-agent/release-complete` leaves the decoy-coverage EXEMPT list.
|
||||
- R-426: the shared reuse-refs/instructions/observations gates get agent-side decoys (9 cases on a scratch clone of this repo); COVERS reuse-refs, instructions, observations — three `felhom-agent/*` entries leave the decoy-coverage EXEMPT list.
|
||||
- **Fixed without a row:** `configs/test_felhom_config_bundle.py` read the two ISO first-boot files that installer 1.32.0 now NAMES under KEPT (R-275) as files the installer writes — `go test ./internal/osupdate` was red on DooPlex from 21:25 to 01:55 (felhom.eu `85de3f9b`); they are listed with why. Test only; agent v0.148.0's code is unaffected.
|
||||
|
||||
## v0.148.0 — the host report names the running binary's sha; the format answer carries the new filesystem's UUID (burn-down night: R-349, R-25 agent halves) (2026-10-06)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `3e68a0870e0e2ce262cb4819294edddeb0a73e8c558a31611a20329a6d9ee283`
|
||||
config bundle sha256 `a6fa4f589d184b58c9911303bd087e300be1e75b3647e4302c9594df6989c4de` (tag `v0.148.0` = `861d32a`).
|
||||
Delivery order as for v0.147.0: signed `agent_update`, then signed `agent_config_update`.
|
||||
|
||||
- **R-349:** the host report carries `agent_sha256`, the sha256 of the running agent binary (read once from
|
||||
`/proc/self/exe`; empty = unknown), so a hand-built binary under the vouched version name becomes visible. The hub
|
||||
comparison is a separate hub change. Test `TestCollect_AgentSHA256IsTheRunningBinary`; red-proved.
|
||||
|
||||
@@ -52,11 +52,11 @@ This is in the core because breaching it is how this component stops being audit
|
||||
|
||||
## Gates — ONE entry point
|
||||
|
||||
**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs this
|
||||
repo's gates — `reuse_refs_check` and `instructions_gate`, both the **shared** copies in
|
||||
`felhom.eu/scripts/`, never copied into this repo (a copy recreates the drift they detect; an absent
|
||||
sibling clone FAILS). `--fast` selects the gates touching no network and no container runtime; today
|
||||
that is all of them. **A missing gate is a FAILURE, never a skip.**
|
||||
**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs every
|
||||
gate in its `GATES` table (that table is the list); the shared ones — `reuse_refs_check`,
|
||||
`instructions_gate`, `observations_gate` — are the copies in `felhom.eu/scripts/`, never copied into
|
||||
this repo (a copy recreates the drift they detect; an absent sibling clone FAILS). `--fast` selects the
|
||||
gates touching no network and no container runtime, and skips `published` (network), naming it. **A missing gate is a FAILURE, never a skip.**
|
||||
|
||||
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
|
||||
**per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
|
||||
@@ -104,8 +104,8 @@ the mechanism are exempt.
|
||||
|
||||
**A gate ships with a decoy test that has been seen to fail (R-421).** A decoy is the LABEL without
|
||||
the FACT — a directory with the right name and no bake log, a note whose prose mentions the marker it
|
||||
lacks. `scripts/decoy_coverage_gate.py` refuses a new gate that has neither a decoy nor a named
|
||||
lacks. `felhom.eu/scripts/decoy_coverage_gate.py` (run by felhom.eu's `repo_gates.py`, for all four repos) refuses a new gate that has neither a decoy nor a named
|
||||
exemption carrying its row. The four shapes, the 2026-09-01 sweep that fooled 16 of 29 gates, and the
|
||||
decoys withdrawn as illegitimate: `documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and
|
||||
decoys withdrawn as illegitimate: `felhom.eu/documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and
|
||||
`felhom-controller/.claude/rules/gates.md`. **Scope is a fact too** — prefer `os.walk` over
|
||||
`os.listdir`, and a glob over a hand-maintained list.
|
||||
|
||||
+13
@@ -1,5 +1,18 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-15 (night) — v0.157.0 RELEASED, NOT VOUCHED (storage identity build A, felhom.eu `16` §13).** Every storage
|
||||
> place carries `kind`, `physical_disks`, `identity` (+ reason) in `GET /disks` and the hub report; `POST /disks/same-disk`
|
||||
> answers the controller (v0.317.0, MinAgent 0.157.0). Computed on every read, never cached. Delivered by signed jobs to
|
||||
> Tester 1, demo-hp, N100 only. Next: the operator vouches 0.157.0 → the global floor 0.317.0 / MinAgent 0.157.0. Build B
|
||||
> (partition candidates, the internal place) waits for a task file (R-1079).
|
||||
|
||||
> **2026-10-08 (day) — UNRELEASED on main, ships with tomorrow's release (decision 178).** R-899: `POST /backup?trigger=manual`
|
||||
> runs no OS leg (`localapi/server.go`); after-boot kernel reports carry the saved block's ring (`kernelReportRing`). R-304:
|
||||
> `escrow.ErrRetainedUnchecked` → HTTP 424 `older_unchecked` when not every earlier package was tried (withheld, caps,
|
||||
> malformed, unreadable list). Binary-only delivery; no wrapper or root file changed.
|
||||
|
||||
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
|
||||
|
||||
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
|
||||
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
|
||||
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
|
||||
|
||||
@@ -44,13 +44,14 @@ unnoticed until a user hit them. `internal/capability` makes that loud:
|
||||
(`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
|
||||
— the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
|
||||
|
||||
**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
|
||||
(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
|
||||
**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin` →
|
||||
`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
|
||||
grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
|
||||
target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
|
||||
dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.
|
||||
**Controller-swap under non-root (v0.45.0; the write since R-861 (a) A1).** The agent-owned controller image swap
|
||||
(`internal/localapi/controllerswap.go`) no longer shells out. The write goes on **stdin** to the ROOT verb
|
||||
`felhom-priv-apply controller-image <vmid>` (`GuestBinder.WriteControllerImage` → `Runner.RunStdin`, the same fenced
|
||||
`sudo -n` runner), which re-checks the ref against our registry + repository + an x.y.z tag and writes
|
||||
`/etc/felhom-controller-image` inside the guest itself; the agent has no in-guest `tee` grant any more (before, a
|
||||
compromised agent could feed any image — sudo cannot see stdin). Its grants live in the `FELHOM_CONTROLLERSWAP` sudoers
|
||||
alias (read-only or fixed-target) and in the capability manifest (Critical), so a dropped grant is a build failure + a
|
||||
live degraded signal. No general `pct exec` is granted.
|
||||
|
||||
## The `storage` package — observe + watchdog (slice 5)
|
||||
|
||||
|
||||
@@ -1,10 +1,43 @@
|
||||
# REPORT — agent v0.147.0 (2026-10-05, burn-down round 2)
|
||||
# REPORT — v0.157.0: every storage place knows its physical disk(s); POST /disks/same-disk (2026-10-15)
|
||||
|
||||
Full session report: `felhom.eu/REPORT-burndown2-2026-10-05.md`. Baseline `d833163` (v0.146.1). Code commit `f1b9b41`
|
||||
(CI job 1365 success), tag `v0.147.0`, binary sha256 `642c4d19…`, bundle sha256 `326527d0…` (verified by download).
|
||||
Written in ASD-STE100 Simplified Technical English. Design read first: felhom.eu `architecture/16-storage-physical-identity.md`
|
||||
(§3, §4, §8, §10; ACCEPTED 2026-10-15, §13 = what was built), `03-host-agent.md`. Task: storage identity build A (operator
|
||||
rulings D1–D6). Controller counterpart: felhom-controller v0.317.0 (MinAgent 0.157.0); hub v0.154.0 shows the fields.
|
||||
|
||||
Rows: R-124 (recipe root namespace = ""), R-118 (no root size for an absent drive), R-269 (rotated-out token rejected at
|
||||
once), R-317 (dnsmasq install probed by its unit). Tests + red-proofs: `felhom.eu/documentation/audits/burndown2-2026-10-05/`.
|
||||
`go build/vet/test ./...` green; `agent_gates.py --fast` green after the release (release-complete needs the tag).
|
||||
## What changed
|
||||
|
||||
Delivery: see the session report (vouch, signed jobs per box, the hub System page afterwards).
|
||||
- `internal/storage/identity.go`: the physical disk(s) under a place, from `/sys`, `/run/udev/data`, `/proc/mounts` only
|
||||
(no block I/O). `proven`, or `unknown` with `vm` / `network` / `usb_bridge` / `weak_wwn` / `no_serial`.
|
||||
- `GET /disks` rows and the hub report's `storage_targets` gain `kind`, `physical_disks`, `identity`, `identity_reason`.
|
||||
- `POST /disks/same-disk` `{a, b}` → `{different, proven, reason, a, b}`; scoped like every `/disks` call; a path the agent
|
||||
did not mount itself is `unmapped`.
|
||||
- `configs/` unchanged.
|
||||
|
||||
## Tests and red-proofs
|
||||
|
||||
Fixture `/sys` + udev trees for every spike case and five same-disk handler tests. Seen failing first: the LVM thin
|
||||
„diamond" (tmeta/tdata under one pool) read as a failure — fixed by counting a seen node once. Broken on purpose and seen
|
||||
failing: the VM rule, the vendor-less WWN rule, the own-mount rule (an absent drive), the `sys_drive` mapping.
|
||||
`python3 scripts/agent_gates.py`: all OK.
|
||||
|
||||
## Release and delivery
|
||||
|
||||
- `scripts/release-agent.sh 0.157.0`: tag `v0.157.0`, sha256 `addd4041…0aef`, bundle `0604bc60…397a`.
|
||||
- Delivered by signed `agent_update` jobs to Tester 1, demo-hp and the N100 (the three test boxes). **NOT VOUCHED** — the
|
||||
operator's act. Until the vouch, the global floor cannot declare MinAgent 0.157.0.
|
||||
|
||||
## Live proof (method: the agent's local API on each PVE host, with the guest's own token)
|
||||
|
||||
| Box | `GET /disks` | same-disk: drive vs system data |
|
||||
|---|---|---|
|
||||
| demo-hp | `local`/`local-lvm` proven (SATA SSD), NVMe drive proven (`eui:`), PBS `network` | different, proven |
|
||||
| N100 | system proven (SATA SSD serial), USB drive proven (the bridge passes the real serial) | different, proven |
|
||||
| Tester 1 (VM) | every place `unknown/vm` | not proven (`vm`) |
|
||||
|
||||
On all three: system data vs `/var/lib/docker` → same disk, proven; a drive name that is not mounted → `unmapped`. The hub
|
||||
stored the same fields (read from its database with the WAL). Evidence: felhom.eu `audits/storage-identity-buildA-2026-10-15/`.
|
||||
|
||||
## Observations
|
||||
|
||||
1. FILED: R-1083 — scratch box 9202 has no `local_api` in its `bootstrap.json`, so it cannot use any agent call.
|
||||
2. NOT-A-FINDING: `usb_bridge` is built from the design's rule and was not seen on a real enclosure; that is R-1077, already open.
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
|
||||
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
|
||||
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
|
||||
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key` | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
|
||||
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key \| controller-image <vmid>` (the last reads the ref on stdin, R-861 (a) A1) | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
|
||||
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
|
||||
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
|
||||
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
|
||||
@@ -41,6 +41,11 @@
|
||||
| `storage.ResolveStorageDevice` | internal/storage/durable_device.go | `ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)` | re-resolve enrolled STORAGE drives (remount) | `uuid:` scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node |
|
||||
| `deriveDurableID` | internal/storage/durableid.go | `deriveDurableID(typ, s, backingDevice, uuid) string` | storage-target durable id (DR re-attach key) | Deterministic per type; `uuid:` for usb/local-dir; PBS id includes `#<fingerprint>` |
|
||||
| `SystemDisks` / `isSystemBacked` / `RoleForStorage` / `RoleForRawDevice` | internal/storage/role.go | `RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole` | protection-tier classification | Fails safe to `system` (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied |
|
||||
| `storage.IdentityResolver` / `Relation` / `MountFor` | internal/storage/identity.go | `HostPath(path) PlaceIdentity`, `Device(dev) PlaceIdentity`, `Relation(a, b) (different, proven, reason)` | physical-disk identity of a storage place (felhom.eu `16-storage-physical-identity.md` §3, v0.157.0) | Reads /sys + /run/udev/data + /proc/mounts only (no block I/O). Judge a drive ONLY on its own mount (`localapi.ownMountIdentity`) — never the containing fs, or an absent drive reads as the root disk. A shared disk key is "same, proven" even when unknown. Pinned by identity_test.go (fixture trees) + samedisk_test.go |
|
||||
| `localapi.Server.EnsureInternalPlace` / `InternalPlaceRecord` | internal/localapi/internalplace.go | `EnsureInternalPlace(ctx)` (startup + 20 s tick); `ReadInternalPlace` / `WriteInternalPlace` | the one-disk box's internal data place (felhom.eu `16` §5, v0.158.0): mounts it at /mnt/belso by fs-UUID like an enrolled drive, records it `enrolled`; the controller binds + registers it | Never takes /mnt/belso from a household drive (a foreign mount, or a unit naming another UUID, waits). Never formats/ejects (dm on the system disk → role system). The ONLY maker is the root helper `configs/felhom-internal-place` (installer + signed `add_internal_place`). Pinned by internalplace_test.go |
|
||||
| `configs/felhom-internal-place` | configs/ (bundle → /usr/local/sbin) | `plan\|create <planned GiB>` | the ruled size rule (reserve max(16, 10 %) ≤ 48 GiB; ≥ 20 GiB or nothing) + the only lvcreate/mkfs of a system-disk volume | Formats only a felhom-belso with NO signature; refuses a foreign one; never resizes/removes. Sudoers: `^(plan\|create) 0$` only. Pinned by configs/test_felhom_internal_place.py |
|
||||
| `storage.candidatePartitions` / `CandidatePartition` | internal/storage/partitions.go | `CandidateDisk.Partitions` | partitions as places (`16` §6): empty / filesystem / in_use per part of a free disk; `Formattable` only while no node of the disk is mounted | A part with children or a member type is never listed; a raw /mnt mount counts as Felhom's only when corroborated (felhomOwnedMounts) — for LISTING only: the root mkfs wrapper trusts no such claim (v0.158.1). Pinned by partitions_test.go |
|
||||
| partition wipe id `byid:<…-partN>+fsuuid:<uuid>` | internal/storage/durable_device.go | `DeviceDurableID` / `ResolveDurableDevice` | a data-bearing PART binds disk id + part number + filesystem (`16` §6) | Both halves must resolve to the SAME device or the wipe refuses. Whole disks keep the plain id. Pinned by partition_durable_test.go |
|
||||
| `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process |
|
||||
| `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` |
|
||||
|
||||
@@ -88,6 +93,8 @@
|
||||
|
||||
| Symbol | File | Short signature | Use for | Gotchas |
|
||||
|---|---|---|---|---|
|
||||
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
|
||||
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
|
||||
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
|
||||
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
||||
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
||||
@@ -121,7 +128,7 @@
|
||||
| Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only |
|
||||
| Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set |
|
||||
| Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction |
|
||||
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
|
||||
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`; R-856 `crashGuardStatePath` — GET /host/crash-guard's state file, internal/localapi/crashguard.go) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
|
||||
| `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) |
|
||||
| `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool** — `BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs |
|
||||
| `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted` ⇒ **quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s |
|
||||
@@ -156,17 +163,20 @@
|
||||
| `proxmox.Runner` | internal/proxmox/privileged.go | `*ExecRunner` (direct/sudo) | `mockRunner` internal/proxmox/mock_test.go; runner fakes in storage tests |
|
||||
| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
|
||||
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
|
||||
| `storage.IdentityHost` | internal/storage/identity.go | `*SysfsIdentityHost` | fixture /sys + udev trees in identity_test.go (`newFx`); `fakeIdentHost` in internal/localapi/samedisk_test.go. A SEPARATE interface from HostReader on purpose — adding methods to HostReader would break every existing fake. |
|
||||
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
|
||||
| `lanresolver.hostRoot` + `dnsmasqUnitPaths` (data seam, R-317) | internal/lanresolver/lanresolver.go | prod `hostRoot = "/"`; probe = the `dnsmasq` package's systemd UNIT, never `/usr/sbin/dnsmasq` (owned by `dnsmasq-base`) | internal/lanresolver/ensure_dnsmasq_test.go — fixture root tree + recording `proxmox.Runner`; the REAL `os.Stat` probe and `EnsureDnsmasq` run. `TestEnsureDnsmasq_ProductionProbeIsTheUnit` pins the production wiring |
|
||||
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
||||
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
|
||||
| `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct |
|
||||
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
|
||||
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
|
||||
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
|
||||
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
|
||||
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
|
||||
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
|
||||
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
|
||||
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
|
||||
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec; the image write via `felhom-priv-apply controller-image`) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
|
||||
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
|
||||
| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
|
||||
| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-444: the weekly trim has the guestnet shape (component + reporter seam + goroutine), so its wiring is asserted
|
||||
// from the AST like TestMainWiresGuestNetWatchdog — a unit-green trim job that main.go never starts is the inert-seam
|
||||
// defect. It must also share the ONE heavy-op gate (heavyOps), or it could run beside a backup.
|
||||
func TestMainWiresGuestDiskTrim(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
var constructedWithGate, reporterWired, started bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
switch node := n.(type) {
|
||||
case *ast.CallExpr:
|
||||
if fn, ok := node.Fun.(*ast.SelectorExpr); ok {
|
||||
switch fn.Sel.Name {
|
||||
case "New":
|
||||
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "fstrim" && len(node.Args) >= 3 {
|
||||
if id, ok := node.Args[2].(*ast.Ident); ok && id.Name == "heavyOps" {
|
||||
constructedWithGate = true
|
||||
}
|
||||
}
|
||||
case "SetGuestDiskTrimReporter":
|
||||
reporterWired = true
|
||||
}
|
||||
}
|
||||
case *ast.GoStmt:
|
||||
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
|
||||
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "diskTrim" {
|
||||
started = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !constructedWithGate {
|
||||
t.Error("main.go never calls fstrim.New(..., heavyOps, ...) — no trim job, or one outside the heavy-op gate")
|
||||
}
|
||||
if !reporterWired {
|
||||
t.Error("main.go never calls collector.SetGuestDiskTrimReporter — the guest_disk_trim stanza never reaches the hub")
|
||||
}
|
||||
if !started {
|
||||
t.Error("main.go never starts the trim job with `go diskTrim.Run(ctx)`")
|
||||
}
|
||||
}
|
||||
+221
-74
@@ -38,6 +38,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/fasttick"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/felhomsshd"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/fstrim"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/guesthook"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/guestnet"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
@@ -49,6 +50,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbs"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poke"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
@@ -57,6 +59,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/vzpartial"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/wgtunnel"
|
||||
)
|
||||
|
||||
@@ -134,39 +137,38 @@ func main() {
|
||||
return
|
||||
}
|
||||
var (
|
||||
cfgPath string
|
||||
selftest selftestFlag
|
||||
vmid int
|
||||
watch time.Duration
|
||||
archive string
|
||||
mode string
|
||||
hostname string
|
||||
keep bool
|
||||
rootfsGrow int
|
||||
dataVolGrow int
|
||||
dataVolMount string
|
||||
sysDataGrow int
|
||||
sysDataMount string
|
||||
cores int
|
||||
memoryMB int
|
||||
pbsStorage string
|
||||
paperkey bool
|
||||
offline bool
|
||||
upload bool
|
||||
custID string
|
||||
custDomain string
|
||||
custName string
|
||||
custEmail string
|
||||
hubPassword string
|
||||
blobPath string
|
||||
expectedFP string
|
||||
keyDest string
|
||||
installWGKey bool
|
||||
idBundlePath string
|
||||
directivePath string
|
||||
swapImage string
|
||||
outputMode string
|
||||
showVersion bool
|
||||
cfgPath string
|
||||
selftest selftestFlag
|
||||
vmid int
|
||||
watch time.Duration
|
||||
archive string
|
||||
mode string
|
||||
hostname string
|
||||
keep bool
|
||||
rootfsGrow int
|
||||
dataVolGrow int
|
||||
dataVolMount string
|
||||
sysDataGrow int
|
||||
sysDataMount string
|
||||
cores int
|
||||
memoryMB int
|
||||
pbsStorage string
|
||||
paperkey bool
|
||||
offline bool
|
||||
upload bool
|
||||
custID string
|
||||
custDomain string
|
||||
custName string
|
||||
custEmail string
|
||||
hubPassword string
|
||||
blobPath string
|
||||
expectedFP string
|
||||
keyDest string
|
||||
installWGKey bool
|
||||
idBundlePath string
|
||||
swapImage string
|
||||
outputMode string
|
||||
showVersion bool
|
||||
)
|
||||
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
|
||||
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `restore-test-due` = READ-ONLY: print the per-tier due verdict the scheduler would act on, with its cost; `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
|
||||
@@ -197,7 +199,6 @@ func main() {
|
||||
flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)")
|
||||
flag.BoolVar(&installWGKey, "install-wg-key", false, "for --selftest=identity-consume: ALSO install the recovered wg_private_key into wgtunnel's key file (S5 DR; create-only, refuses to overwrite)")
|
||||
flag.StringVar(&idBundlePath, "identity-bundle", "", "for --selftest=escrow-create: a 0600 JSON file {tunnel_token,pbs_token} to ALSO escrow under R (10D)")
|
||||
flag.StringVar(&directivePath, "directive", "", "for --selftest=escrow-create: a JSON file with the non-secret DR directive (pbs repo/ns, expected fingerprint, tunnel id)")
|
||||
flag.StringVar(&custID, "customer-id", "", "for --selftest=provision: the customer id — the hub config-pull target, baked into the guest's bootstrap")
|
||||
flag.StringVar(&hubPassword, "hub-password", "", "for --selftest=provision: the customer's hub RETRIEVAL PASSPHRASE (SECRET) — baked into bootstrap.json so the controller pulls its config (and the customer-scoped hub key) from the hub. The customer must already exist in the hub.")
|
||||
flag.StringVar(&swapImage, "image", "", "for --selftest=controller-swap: the target controller image ref (gitea.dooplex.hu/admin/felhom-controller:<semver>) — must already be pulled in the guest")
|
||||
@@ -268,7 +269,7 @@ func main() {
|
||||
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount, Cores: cores, MemoryMB: memoryMB},
|
||||
}))
|
||||
case "escrow-create":
|
||||
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath, outputMode))
|
||||
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, outputMode))
|
||||
case "escrow-consume":
|
||||
os.Exit(runSelftestEscrowConsume(context.Background(), logger, blobPath, expectedFP, keyDest))
|
||||
case "identity-consume":
|
||||
@@ -776,6 +777,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
hostReader := storage.NewProcHostReader()
|
||||
hostOps := newHostOps(cfg, logger)
|
||||
observer := storage.NewObserver(px, hostReader, hostOps, logger)
|
||||
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
|
||||
// Backup + restore-test state store (slice 6): holds the latest backup-per-target +
|
||||
// latest restore-test result; the collector reads it via the BackupReporter /
|
||||
// RestoreTestReporter seams; the cadence scheduler writes it.
|
||||
@@ -791,6 +793,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
pbsTargets := pbsTargetsFromPVE(cfg, px, logger)
|
||||
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)
|
||||
collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
|
||||
// R-366 slice 2: the restore-test's ledger of archives written with another key → the host report.
|
||||
foreignKeys := backup.NewForeignKeyLedger()
|
||||
collector.SetForeignKeyArchiveReporter(foreignKeys)
|
||||
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target
|
||||
// Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent
|
||||
// depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the
|
||||
@@ -831,7 +836,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
|
||||
// already carries the gated view — v0.86.0.)
|
||||
collector.SetCapabilityProber(probeAll)
|
||||
// R-998 (doc 14): the household power-off record of THIS boot rides every report until the box is down.
|
||||
powerOffState := poweroff.New(filepath.Join(cfg.OOB.WithDefaults().StateDir, "poweroff.json"), poweroff.ProcBtime)
|
||||
collector.SetDeliberateOffSource(func() *hub.DeliberateOff {
|
||||
if r := powerOffState.Current(); r != nil {
|
||||
return &hub.DeliberateOff{RequestedAt: r.RequestedAt.UTC().Format(time.RFC3339)}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
loop := hub.NewLoop(collector, client, time.Duration(hcfg.PollSeconds)*time.Second, logger)
|
||||
// R-998: the power-off path waits until the hub ACCEPTED a report carrying the record.
|
||||
powerOffAck := make(chan struct{}, 1)
|
||||
loop.SetReportedHook(func(r *hub.HostReport) {
|
||||
if r != nil && r.DeliberateOff != nil {
|
||||
select {
|
||||
case powerOffAck <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
})
|
||||
interval := time.Duration(hcfg.PollSeconds) * time.Second
|
||||
// Heartbeat log-pull (v0.83.0): when the control envelope requests it, the NEXT
|
||||
// heartbeat carries the debug ring's tail (newest-kept, byte-capped in the loop).
|
||||
@@ -875,6 +898,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) {
|
||||
logger.Info("osupdate: sent kept report(s)", "count", n)
|
||||
})
|
||||
// R-836 (`09` §3 decision 172): what became of a kernel step across this boot; on a one-shot boot of a new kernel,
|
||||
// judge it (the host health rule + the hub reached) for KernelJudgeWait, then make it the default or revert ONCE.
|
||||
// The wait: measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`) — every container healthy 68 s after the
|
||||
// reboot on demo-felhom and 272 s on demo-hp (the hub reached at 63 s / 189 s); 20 minutes leaves room for a slow
|
||||
// network and stays under the hub's 45-minute host_stale (its
|
||||
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
|
||||
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
|
||||
|
||||
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
|
||||
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
|
||||
@@ -929,10 +959,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// state. Gates the watchdog's self-heal re-mount to ENROLLED drives, and the local API records
|
||||
// enroll/eject into it. Open failure → ungated legacy remount (logged) so the daemon still runs.
|
||||
var intentReader storage.IntentReader
|
||||
intentStateDir := "/var/lib/felhom-agent"
|
||||
if cfg.Authz.NonceStorePath != "" {
|
||||
intentStateDir = filepath.Dir(cfg.Authz.NonceStorePath) // share the agent's durable state dir
|
||||
}
|
||||
intentStateDir := agentStateDir(cfg)
|
||||
intentStore, ierr := storage.OpenIntentStore(filepath.Join(intentStateDir, "drive-intents.json"))
|
||||
if ierr != nil {
|
||||
logger.Warn("storage: intent store unavailable — self-heal runs UNGATED (legacy remount-any)", "err", ierr)
|
||||
@@ -1016,7 +1043,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// with the local API so a backup and a restore-test can never run together.
|
||||
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
||||
heavyOps := &backup.InFlight{}
|
||||
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
|
||||
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, foreignKeys, logger)
|
||||
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
|
||||
// holds only this process's latest run, and under per-archive due-ness the agent will not
|
||||
// re-test an archive it has already proven — so without this the hub can report a tier unproven
|
||||
@@ -1108,6 +1135,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
return release, nil
|
||||
}}
|
||||
// R-812 option A: a signed Proxmox package step (`11` §5.10) — ring 1; under the heavy-op gate and the /etc/pve gate.
|
||||
pveExec := osupdate.PVEStepExecutor{Leg: osLeg, Guest: firstGuest(px),
|
||||
Gate: func(ctx context.Context) (func(), error) {
|
||||
release, busy, ok := heavyOps.TryAcquire("os-pve-step")
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("busy: %s", busy)
|
||||
}
|
||||
return release, nil
|
||||
}}
|
||||
// R-836 (`09` §3 decision 172): a signed kernel step STAGES a kernel on a ring-1 box (install + the one-shot flag,
|
||||
// never a reboot — the night leg reboots it on a night the household was told about); under the heavy-op gate.
|
||||
kernelExec := osupdate.KernelStepExecutor{Leg: osLeg, Guest: firstGuest(px),
|
||||
Gate: func(ctx context.Context) (func(), error) {
|
||||
release, busy, ok := heavyOps.TryAcquire("os-kernel-step")
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("busy: %s", busy)
|
||||
}
|
||||
return release, nil
|
||||
}}
|
||||
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
|
||||
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
|
||||
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
|
||||
@@ -1119,7 +1165,11 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||
}}
|
||||
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, bundleExec}, cfg.Hub.HostID, logger)
|
||||
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
|
||||
// to the local API server built below).
|
||||
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
|
||||
localapi.InternalPlaceFileIn(intentStateDir), logger)
|
||||
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, kernelExec, bundleExec, internalPlaceExec}, cfg.Hub.HostID, logger)
|
||||
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
|
||||
|
||||
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
|
||||
@@ -1152,6 +1202,35 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localSrv != nil {
|
||||
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
|
||||
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
|
||||
pwMode := proxmox.RunnerMode(cfg.Privileged.Mode)
|
||||
if pwMode == "" {
|
||||
pwMode = proxmox.RunnerSudo
|
||||
}
|
||||
localSrv.SetHostPowerOff(localapi.HostPowerOffConfig{
|
||||
State: powerOffState,
|
||||
Power: localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: pwMode, SudoPath: cfg.Privileged.SudoPath}, logger),
|
||||
ReportNow: func(ctx context.Context) bool {
|
||||
select {
|
||||
case storageTrigger <- struct{}{}:
|
||||
default: // a report is already queued; it runs after the record was written
|
||||
}
|
||||
select {
|
||||
case <-powerOffAck:
|
||||
return true
|
||||
case <-ctx.Done():
|
||||
return false
|
||||
}
|
||||
},
|
||||
Uptime: func() (time.Duration, bool) {
|
||||
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
|
||||
if err != nil || b <= 0 {
|
||||
return 0, false
|
||||
}
|
||||
return time.Since(time.Unix(b, 0)), true
|
||||
},
|
||||
})
|
||||
localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) {
|
||||
// Let the controller finish bringing its apps back after the backup, then run (still under the gate).
|
||||
select {
|
||||
@@ -1342,6 +1421,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// filesystem UUID and re-mounts (idempotent `enable --now`) so a letter reshuffle is a no-op.
|
||||
// Type-asserted (the concrete op exposes it; the interface stays lean).
|
||||
mountReasserter := hostOps // concrete *storage.SudoHostOps — exposes ReassertEnrolledMounts
|
||||
// v0.158.0 (felhom.eu `16` §5): the internal data place is mounted like an enrolled drive — first, so the
|
||||
// re-assert below sees its unit.
|
||||
localSrv.EnsureInternalPlace(ctx)
|
||||
if mountReasserter != nil {
|
||||
mountReasserter.ReassertEnrolledMounts(ctx)
|
||||
}
|
||||
@@ -1379,6 +1461,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
case <-t.C:
|
||||
// Re-assert mounts first (handles a USB that enumerated late after a host reboot, or a
|
||||
// unit re-disabled at runtime), then re-assert the guest binds against the now-live mounts.
|
||||
localSrv.EnsureInternalPlace(ctx) // `16` §5: a record written since the last tick (the signed job)
|
||||
if mountReasserter != nil {
|
||||
mountReasserter.ReassertEnrolledMounts(ctx)
|
||||
}
|
||||
@@ -1457,6 +1540,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// the dangling snapshot → start iff onboot). Runs BEFORE the backup loop starts, so a present
|
||||
// backup lock is stale by definition (guarded by a no-vzdump-running check; fail-safe otherwise).
|
||||
localSrv.RecoverStaleLockedGuests(ctx)
|
||||
// R-1006: then remove a dead half-written backup a power cut left (the lock recovery first: same order as a human).
|
||||
localSrv.SweepVzdumpPartials(ctx)
|
||||
// F-REBOOT: the startup recovery above only covers a guest left LOCKED by an interrupted
|
||||
// backup. A guest that simply ends up stopped-and-unlocked (a `pct reboot` whose shutdown
|
||||
// half completed and whose start half never fired — Campaign 8 fault 11, 9m47s of total
|
||||
@@ -1479,6 +1564,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
go runJanitor(ctx, jd)
|
||||
}
|
||||
// R-444 (`09` §3 decision 139): the weekly guest disk trim — `pct fstrim <vmid>` of every owned, running guest,
|
||||
// Wednesday from 10:00 local, daytime only, under the one-heavy-op gate. Not part of the errc fan-out: a trim job
|
||||
// must never be able to bring the agent down.
|
||||
if cfg.DiskTrim.Enabled() {
|
||||
dtMode := proxmox.RunnerMode(cfg.Privileged.Mode)
|
||||
if dtMode == "" {
|
||||
dtMode = proxmox.RunnerSudo
|
||||
}
|
||||
dtRunner := &proxmox.ExecRunner{Mode: dtMode, SudoPath: cfg.Privileged.SudoPath}
|
||||
dtGuests := localapi.NewStaleLockController(px, dtRunner, reconcile.DefaultPool, logger)
|
||||
if dtGuests != nil {
|
||||
diskTrim := fstrim.New(dtRunner, dtGuests, heavyOps,
|
||||
filepath.Join(cfg.OOB.WithDefaults().StateDir, "guest-disk-trim.json"), logger)
|
||||
collector.SetGuestDiskTrimReporter(diskTrim)
|
||||
go diskTrim.Run(ctx)
|
||||
}
|
||||
} else {
|
||||
logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)")
|
||||
}
|
||||
if lanLoop != nil {
|
||||
lanServers = 1
|
||||
go func() { errc <- lanLoop.Run(ctx) }()
|
||||
@@ -1680,7 +1784,7 @@ func primaryBackupTargetOf(cfg config.Config) func() hub.ConfiguredBackupTarget
|
||||
// disables the cadence (returns a scheduler that just waits) when the cadence is off or the
|
||||
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
|
||||
// machinery still works on-demand via --selftest=restore-test.
|
||||
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, logger *slog.Logger) *backup.Scheduler {
|
||||
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, foreign *backup.ForeignKeyLedger, logger *slog.Logger) *backup.Scheduler {
|
||||
// R-86: this is the EVALUATION interval, not the trigger. What decides a test happens is the
|
||||
// per-archive due-check in internal/backup/restoretest_due.go.
|
||||
cadence := cfg.Backup.RestoreTestEvalInterval()
|
||||
@@ -1699,6 +1803,9 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
|
||||
min, max := cfg.Backup.ScratchBand()
|
||||
target := cfg.Backup.BackupTarget()
|
||||
runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger)
|
||||
if foreign != nil {
|
||||
runner.SetForeignKeyLedger(foreign) // R-366 slice 2: the pick records archives written with another key
|
||||
}
|
||||
// Every configured tier is a rotation candidate, not just the primary.
|
||||
cfgTiers, _ := cfg.Backup.BackupTiers() // warnings already logged where the tiers are armed
|
||||
tierIDs := make([]string, 0, len(cfgTiers))
|
||||
@@ -1795,6 +1902,8 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
}
|
||||
apiTiers := make([]localapi.BackupTier, 0, len(backupTiers))
|
||||
var runner *backup.BackupRunner
|
||||
var tierRunners []*backup.BackupRunner
|
||||
var tierTargets []string
|
||||
for _, t := range backupTiers {
|
||||
prune := ""
|
||||
if t.KeepLast > 0 {
|
||||
@@ -1807,6 +1916,8 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
if t.Primary {
|
||||
runner = r
|
||||
}
|
||||
tierRunners = append(tierRunners, r)
|
||||
tierTargets = append(tierTargets, t.TargetID)
|
||||
apiTiers = append(apiTiers, localapi.BackupTier{
|
||||
TargetID: t.TargetID,
|
||||
Cadence: t.Cadence,
|
||||
@@ -1854,11 +1965,15 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
return nil, 0, ferr
|
||||
}
|
||||
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
|
||||
// R-304: every package the hub holds and the code will NOT be tried against — no key material,
|
||||
// over the hub's cap, or malformed here. A refusal may call the code wrong only when this is 0.
|
||||
withheld := resp.UnopenableCount + resp.TruncatedCount
|
||||
for _, p := range resp.Packages {
|
||||
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
|
||||
if derr != nil || len(blob) == 0 {
|
||||
// One malformed package must not sink the rest — the customer's code may open a
|
||||
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
|
||||
withheld++
|
||||
continue
|
||||
}
|
||||
out = append(out, escrow.RetainedBlob{
|
||||
@@ -1868,9 +1983,13 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
Index: p.Index,
|
||||
})
|
||||
}
|
||||
return out, resp.UnopenableCount, nil
|
||||
return out, withheld, nil
|
||||
},
|
||||
}
|
||||
// R-894: ONE instance — the local API writes it, the host report reads it (2026-10-09: a restart right
|
||||
// after the night backup erased the hub's evidence of it).
|
||||
lastKnownBackups := backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json"))
|
||||
collector.SetKnownBackupReporter(lastKnownBackups)
|
||||
srv, err := localapi.NewServer(localapi.Options{
|
||||
EscrowRecovery: escrowRecoverer,
|
||||
ListenAddr: cfg.LocalAPI.ListenAddr,
|
||||
@@ -1881,12 +2000,16 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary
|
||||
InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F)
|
||||
Store: store,
|
||||
Storage: observer,
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
||||
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
|
||||
Tokens: tokens,
|
||||
BackupCadence: cfg.Backup.BackupCadence(),
|
||||
// R-894: the newest success per tier on disk — the due-check's fallback when the storage cannot be
|
||||
// read right after a restart. Same state dir as restore-test-state.json.
|
||||
LastKnownBackups: lastKnownBackups,
|
||||
Storage: observer,
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
||||
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
|
||||
Identity: storage.NewIdentityResolver(), // v0.157.0: `16` §3 identity on /disks + POST /disks/same-disk
|
||||
Tokens: tokens,
|
||||
BackupCadence: cfg.Backup.BackupCadence(),
|
||||
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
|
||||
Disks: hostOps,
|
||||
DiskGate: storageGateAdapter{gate: gate, hostID: cfg.Hub.HostID},
|
||||
@@ -1906,10 +2029,12 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
// F2-b: recover a guest left with a stale vzdump lock by a reboot-during-backup. Reads + start
|
||||
// go through the API client; the `pct unlock` is the one fenced root-CLI op (no API equivalent).
|
||||
// A1 (v0.62.0): the scan is restricted to felhom-pool members (ownership proven, not assumed).
|
||||
StaleLock: localapi.NewStaleLockController(px, &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, reconcile.DefaultPool, logger),
|
||||
Intent: intent, // slice 10 P3: record enroll/eject intent for self-heal
|
||||
GuestBinds: guestBinds, // F9: per-guest bind record for the startup re-assert
|
||||
FormatJobs: formatJobs, // F20-BUG3: detached-format job record + restart recovery
|
||||
StaleLock: localapi.NewStaleLockController(px, &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, reconcile.DefaultPool, logger),
|
||||
Intent: intent, // slice 10 P3: record enroll/eject intent for self-heal
|
||||
// v0.158.0 (felhom.eu `16` §5): the internal data place record, beside the intent store.
|
||||
InternalPlaceFile: localapi.InternalPlaceFileIn(agentStateDir(cfg)),
|
||||
GuestBinds: guestBinds, // F9: per-guest bind record for the startup re-assert
|
||||
FormatJobs: formatJobs, // F20-BUG3: detached-format job record + restart recovery
|
||||
|
||||
// Host metrics (slice 9): the shared collector serves GET /host/metrics — a fresh host +
|
||||
// per-storage view to the customer's monitoring page (reuses the slice-4 collector).
|
||||
@@ -1924,6 +2049,26 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
logger.Warn("daemon: local-api disabled (server build)", "err", err)
|
||||
return nil
|
||||
}
|
||||
// R-1006: a dead half-written vzdump (a power cut mid-backup) is removed at start-up and before every backup of the
|
||||
// guest on that target, through the root helper; the in-flight answer is R-1008's (a task from before the boot is dead).
|
||||
cleaner := &vzpartial.Cleaner{
|
||||
API: px,
|
||||
Runner: &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath},
|
||||
InFlight: srv.VzdumpInFlight,
|
||||
Boot: func() (int64, bool) {
|
||||
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
|
||||
return b, err == nil && b > 0
|
||||
},
|
||||
Logger: logger,
|
||||
}
|
||||
for _, r := range tierRunners {
|
||||
r.SetBeforeBackup(func(ctx context.Context, storage string, vmid int) { cleaner.Clean(ctx, storage, vmid) })
|
||||
}
|
||||
srv.SetPartialSweep(func(ctx context.Context, vmid int) {
|
||||
for _, t := range tierTargets {
|
||||
cleaner.Clean(ctx, t, vmid)
|
||||
}
|
||||
})
|
||||
return srv
|
||||
}
|
||||
|
||||
@@ -2053,6 +2198,7 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger)
|
||||
return 1
|
||||
}
|
||||
observer := storage.NewObserver(px, storage.NewProcHostReader(), newHostOps(cfg, logger), logger)
|
||||
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
|
||||
// Wire the LIVE PBS reporter here too (v0.39.0): selftest=hub is a separate one-shot process — no
|
||||
// verify loop runs — so a nil reporter previously yielded pbs_snapshots:[] and an absent dr_recipe
|
||||
// pbs coord. The live reporter lists snapshots directly (fresh store, last-known-good fallback) so
|
||||
@@ -2109,6 +2255,7 @@ func runSelftestStorage(ctx context.Context, cfg config.Config, logger *slog.Log
|
||||
hostReader := storage.NewProcHostReader()
|
||||
hostOps := newHostOps(cfg, logger)
|
||||
observer := storage.NewObserver(px, hostReader, hostOps, logger)
|
||||
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
|
||||
|
||||
octx, cancel := context.WithTimeout(ctx, 60*time.Second)
|
||||
fmt.Printf("=== felhom-agent %s selftest=storage ===\n", version)
|
||||
@@ -2241,7 +2388,7 @@ func runSelftestRestoreTestDue(ctx context.Context, cfg config.Config, logger *s
|
||||
return 1
|
||||
}
|
||||
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
||||
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, logger)
|
||||
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, nil, logger)
|
||||
|
||||
fmt.Printf("eval_interval=%s settle=%s\n", cfg.Backup.RestoreTestEvalInterval(), cfg.Backup.RestoreTestSettle())
|
||||
start := time.Now()
|
||||
@@ -2667,7 +2814,6 @@ type escrowCeremonyOpts struct {
|
||||
offline bool
|
||||
upload bool
|
||||
identityBundlePath string
|
||||
directivePath string
|
||||
}
|
||||
|
||||
// escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the
|
||||
@@ -2726,11 +2872,10 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
|
||||
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("PBS key for %q not found (%s): %v", storage, keyPath, err)}
|
||||
}
|
||||
|
||||
// Slice 10D.1: optionally ALSO wrap the identity bundle under the same R, and carry the non-secret
|
||||
// directive for the hub. The bundle file is a 0600 secret (tunnel/pbs tokens); the directive is
|
||||
// non-secret (pbs repo/ns, expected fingerprint, tunnel id).
|
||||
// Slice 10D.1: optionally ALSO wrap the identity bundle under the same R. The bundle file is a 0600 secret
|
||||
// (tunnel/pbs tokens). The non-secret "directive" that used to ride along is retired (R-105, `09` §3 decision 169:
|
||||
// nothing read it; the DR path reads the recipe, tenantsync and the escrow blob).
|
||||
var identity *escrow.IdentityBundle
|
||||
var directive json.RawMessage
|
||||
if opts.identityBundlePath != "" {
|
||||
raw, err := os.ReadFile(opts.identityBundlePath)
|
||||
if err != nil {
|
||||
@@ -2741,11 +2886,6 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
|
||||
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("identity bundle is not valid JSON {tunnel_token,pbs_token}: %v", err)}
|
||||
}
|
||||
identity = &b
|
||||
if opts.directivePath != "" {
|
||||
if d, err := os.ReadFile(opts.directivePath); err == nil && json.Valid(d) {
|
||||
directive = d
|
||||
}
|
||||
}
|
||||
}
|
||||
// S3: auto-inject the offsite WG private key into the escrowed identity when the key file
|
||||
// exists — a NEW escrow run should always capture the live tunnel identity. Field NAME only
|
||||
@@ -2824,7 +2964,7 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
|
||||
ResticPwSealed: resticStaged,
|
||||
}
|
||||
if opts.upload {
|
||||
if err := uploadEscrowBlob(ctx, cfg, res, directive, resticPwSHA256); err != nil {
|
||||
if err := uploadEscrowBlob(ctx, cfg, res, resticPwSHA256); err != nil {
|
||||
// R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets
|
||||
// the text shell keep the pre-extraction order (R surfaced, THEN the failure).
|
||||
return out, &escrowCeremonyErr{kind: "upload", err: err}
|
||||
@@ -2881,7 +3021,7 @@ func printEscrowTextRBlock(out *escrowCeremonyOutcome) {
|
||||
// NOTHING else there; every human/info line goes to stderr; failures exit non-zero with no
|
||||
// partial JSON. This is the controller-driven ceremony's parse surface (spike §2.3: the text
|
||||
// banner is positionally brittle).
|
||||
func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slog.Logger, storage string, paperkey, offline, upload bool, identityBundlePath, directivePath, outputMode string) int {
|
||||
func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slog.Logger, storage string, paperkey, offline, upload bool, identityBundlePath, outputMode string) int {
|
||||
switch outputMode {
|
||||
case "", "text", "json":
|
||||
default:
|
||||
@@ -2899,7 +3039,7 @@ func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slo
|
||||
|
||||
out, cerr := escrowCeremony(ctx, cfg, logger, escrowCeremonyOpts{
|
||||
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
|
||||
identityBundlePath: identityBundlePath, directivePath: directivePath,
|
||||
identityBundlePath: identityBundlePath,
|
||||
})
|
||||
if cerr != nil {
|
||||
switch cerr.kind {
|
||||
@@ -3082,20 +3222,19 @@ type escrowUploadRequest struct {
|
||||
BlobB64 string `json:"blob_b64"` // base64 of the opaque R-wrapped blob (ciphertext)
|
||||
KeyFingerprint string `json:"key_fingerprint"` // for operator display only
|
||||
Posture string `json:"posture"` // e.g. "zero_knowledge"
|
||||
// Slice 10D.1 — optional DR bundle (identity escrow + non-secret directive). Omitted in slice-7.
|
||||
IdentityBlobB64 string `json:"identity_blob_b64,omitempty"`
|
||||
DirectiveJSON json.RawMessage `json:"directive,omitempty"`
|
||||
CreatedAt string `json:"created_at"` // RFC3339
|
||||
// Slice 10D.1 — optional identity escrow. Omitted in slice-7. (The `directive` is retired — R-105.)
|
||||
IdentityBlobB64 string `json:"identity_blob_b64,omitempty"`
|
||||
CreatedAt string `json:"created_at"` // RFC3339
|
||||
// SLICE 3 — sha256 hex of the offsite restic repo password sealed in the identity blob (present only
|
||||
// when a staged password was folded in). Non-reversible hash of a 256-bit random secret — safe to
|
||||
// store/serve; lets the controller VERIFY "the escrow covers the CURRENT key" and auto-confirm.
|
||||
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
|
||||
}
|
||||
|
||||
// uploadEscrowBlob PUTs the opaque blob (and, for 10D, the identity blob + non-secret directive) to
|
||||
// uploadEscrowBlob PUTs the opaque blob (and, for 10D, the identity blob) to
|
||||
// the hub, authed with the per-host key. The hub stores ciphertext + non-secret fields; no usable
|
||||
// secret leaves the agent.
|
||||
func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateResult, directive json.RawMessage, resticPwSHA256 string) error {
|
||||
func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateResult, resticPwSHA256 string) error {
|
||||
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
|
||||
return fmt.Errorf("hub not configured (url/host_id/api_key)")
|
||||
}
|
||||
@@ -3108,7 +3247,6 @@ func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateR
|
||||
}
|
||||
if len(res.IdentityBlob) > 0 {
|
||||
upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob)
|
||||
upReq.DirectiveJSON = directive
|
||||
}
|
||||
body, _ := json.Marshal(upReq)
|
||||
url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow"
|
||||
@@ -3775,3 +3913,12 @@ func (f *factsReporter) SystemFacts(ctx context.Context) (int, json.RawMessage,
|
||||
f.facts, f.err = f.leg.Facts(ctx, f.vmid)
|
||||
return f.vmid, f.facts, f.err
|
||||
}
|
||||
|
||||
// agentStateDir is the agent's durable state dir: beside the nonce store when one is configured, else the default.
|
||||
// The intent store, the guest-bind record and the internal place record (v0.158.0) all live here.
|
||||
func agentStateDir(cfg config.Config) string {
|
||||
if cfg.Authz.NonceStorePath != "" {
|
||||
return filepath.Dir(cfg.Authz.NonceStorePath)
|
||||
}
|
||||
return "/var/lib/felhom-agent"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-894 — the on-disk backup record is WIRED on the daemon path (the built-but-never-wired class).
|
||||
// main → runDaemon → buildLocalAPIServer, and inside it the localapi.Options literal carries
|
||||
// LastKnownBackups built by backup.NewBackupSuccessState. An AST walk, not a string match, for the
|
||||
// reasons in escrow_recover_wiring_test.go.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): delete the `LastKnownBackups:` line from buildLocalAPIServer → this
|
||||
// fails with "localapi.Options in buildLocalAPIServer has no LastKnownBackups field". Restored.
|
||||
//
|
||||
// 2026-10-09: the SAME instance also feeds the host report (collector.SetKnownBackupReporter), so a
|
||||
// restart right after a backup no longer erases the hub's evidence of it. The field may name a local
|
||||
// variable; the variable must be built by backup.NewBackupSuccessState and be the one handed to the
|
||||
// collector. RED-PROOF (observed): drop the SetKnownBackupReporter call → "not handed to the collector".
|
||||
func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
|
||||
_, f := parseMain(t)
|
||||
if !callsWithin(f, "main")["runDaemon"] || !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
|
||||
t.Fatal("main → runDaemon → buildLocalAPIServer is broken — the path this test asserts is not the live one")
|
||||
}
|
||||
var field, built bool
|
||||
var fieldVar, handed string
|
||||
for _, d := range f.Decls {
|
||||
fd, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
cl, ok := n.(*ast.CompositeLit)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := cl.Type.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if pkg, _ := sel.X.(*ast.Ident); pkg == nil || pkg.Name+"."+sel.Sel.Name != "localapi.Options" {
|
||||
return true
|
||||
}
|
||||
for _, el := range cl.Elts {
|
||||
kv, ok := el.(*ast.KeyValueExpr)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "LastKnownBackups" {
|
||||
field = true
|
||||
if callsIn(kv.Value)["backup.NewBackupSuccessState"] {
|
||||
built = true
|
||||
}
|
||||
if id, ok := kv.Value.(*ast.Ident); ok {
|
||||
fieldVar = id.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
// a local variable: built by NewBackupSuccessState, and handed to the collector
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
switch x := n.(type) {
|
||||
case *ast.AssignStmt:
|
||||
for i, l := range x.Lhs {
|
||||
if id, ok := l.(*ast.Ident); ok && fieldVar != "" && id.Name == fieldVar && i < len(x.Rhs) &&
|
||||
callsIn(x.Rhs[i])["backup.NewBackupSuccessState"] {
|
||||
built = true
|
||||
}
|
||||
}
|
||||
case *ast.CallExpr:
|
||||
if fn, ok := x.Fun.(*ast.SelectorExpr); ok && fn.Sel.Name == "SetKnownBackupReporter" && len(x.Args) == 1 {
|
||||
if id, ok := x.Args[0].(*ast.Ident); ok {
|
||||
handed = id.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
if !field {
|
||||
t.Fatal("localapi.Options in buildLocalAPIServer has no LastKnownBackups field")
|
||||
}
|
||||
if !built {
|
||||
t.Fatal("LastKnownBackups is not built by backup.NewBackupSuccessState")
|
||||
}
|
||||
if handed == "" || handed != fieldVar {
|
||||
t.Fatalf("the saved backups (%q) are not handed to the collector (SetKnownBackupReporter got %q)", fieldVar, handed)
|
||||
}
|
||||
}
|
||||
|
||||
func callsIn(n ast.Node) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
ast.Inspect(n, func(n ast.Node) bool {
|
||||
if ce, ok := n.(*ast.CallExpr); ok {
|
||||
if fn, ok := ce.Fun.(*ast.SelectorExpr); ok {
|
||||
if x, ok := fn.X.(*ast.Ident); ok {
|
||||
out[x.Name+"."+fn.Sel.Name] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
return out
|
||||
}
|
||||
@@ -43,7 +43,7 @@ func main() {
|
||||
|
||||
func run() error {
|
||||
var (
|
||||
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | agent_config_update")
|
||||
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update | add_internal_place")
|
||||
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
|
||||
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
|
||||
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
|
||||
|
||||
@@ -80,7 +80,29 @@ Cmnd_Alias FELHOM_DNSMASQ = \
|
||||
Cmnd_Alias FELHOM_GUESTHOOK = \
|
||||
/usr/sbin/pct ^set [0-9]+ --hookscript local\:snippets/felhom-guest-hook\.sh$, \
|
||||
/usr/sbin/pct ^set [0-9]+ --delete mp[0-9]+$, \
|
||||
/usr/sbin/pct ^reboot [0-9]+$
|
||||
/usr/sbin/pct ^reboot [0-9]+$, \
|
||||
/usr/sbin/pct ^shutdown [0-9]+ --timeout [0-9]+$
|
||||
|
||||
# Host power-off (R-998, doc 14): the household's „A gép kikapcsolása" — guest shutdown above, then this one fixed
|
||||
# vector. Guarded agent-side (10 min after boot, one per boot, never during a backup/restore/update/drive job).
|
||||
Cmnd_Alias FELHOM_POWEROFF = \
|
||||
/usr/bin/systemctl poweroff
|
||||
|
||||
# Dead half-written vzdump removal (R-1006, power-cut-fixes-2026-10-14): a power cut mid-backup leaves
|
||||
# vzdump-lxc-<vmid>-<date>.tar.dat + .tmp/ on the dir storage; nothing removed them and every later backup was skipped
|
||||
# „not enough space". ONE root helper (configs/felhom-vzdump-partial-clean, from the config bundle) re-checks everything
|
||||
# as root — a dir storage from storage.cfg, the exact name of THAT vmid, not in use, dead — and refuses otherwise. The
|
||||
# arguments here are its exact shape (storage id, vmid, one half-written name); the vmid-in-name match is the helper's.
|
||||
Cmnd_Alias FELHOM_VZPARTIAL = \
|
||||
/usr/local/sbin/felhom-vzdump-partial-clean ^[A-Za-z][A-Za-z0-9_-]{0\,63} [1-9][0-9]{2\,8} vzdump-(lxc|qemu)-[1-9][0-9]{2\,8}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$
|
||||
|
||||
# Internal data place („Belső tárhely", felhom.eu architecture/16 §5, §7; agent v0.158.0). ONE root helper
|
||||
# (configs/felhom-internal-place, from the config bundle) creates <vg>/felhom-belso in the local-lvm thin pool by the
|
||||
# ruled size rule and formats ONLY that volume, ONLY when it carries no signature. The agent calls it from the signed
|
||||
# `add_internal_place` job (an old box), always with 0 GiB planned: its guest's volumes already exist. `plan` changes
|
||||
# nothing. The installer runs the helper as root itself (a new box).
|
||||
Cmnd_Alias FELHOM_INTERNALPLACE = \
|
||||
/usr/local/sbin/felhom-internal-place ^(plan|create) 0$
|
||||
|
||||
# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
|
||||
# /mnt/felhom-drives (self-bind + make-shared) and binds/unbinds each drive's felhom-data namespace UNDERNEATH it so the
|
||||
@@ -111,15 +133,17 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
|
||||
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
|
||||
# + container across spaces, spike-confirmed)
|
||||
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
|
||||
# tee <FIXED image file> — WRITE the ref; content is fed on STDIN (no shell, no interpolation),
|
||||
# the agent strict-validates the ref (controllerImageRe) before the write.
|
||||
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
|
||||
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
|
||||
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
|
||||
# compromised agent could hand the guest's bootstrap ANY image (sudo cannot see stdin).
|
||||
# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
|
||||
Cmnd_Alias FELHOM_CONTROLLERSWAP = \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/felhom-controller-image$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$
|
||||
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
|
||||
|
||||
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
|
||||
# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
|
||||
@@ -129,6 +153,14 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
|
||||
Cmnd_Alias FELHOM_STALELOCK = \
|
||||
/usr/sbin/pct ^unlock [0-9]+$
|
||||
|
||||
# Weekly guest disk trim (R-444, operator ruling `09` §3 decision 139). A thin pool only ever grows from blocks the
|
||||
# guest has FREED: `fstrim` inside the unprivileged container is refused (FITRIM: Operation not permitted), so the host
|
||||
# trims the guest's mounts. Measured on demo-hp 2026-10-06: `pct fstrim 9201` rc 0 in 24.4 s, pool 65.53 % -> 33.40 %,
|
||||
# apps kept answering. ONE exact pattern: a vmid and nothing else — no `--ignore-mountpoints`, no second argument
|
||||
# (pinned: TestSudoersFstrimRuleIsExact). The agent runs it on a weekly daytime timer under the heavy-op gate.
|
||||
Cmnd_Alias FELHOM_FSTRIM = \
|
||||
/usr/sbin/pct ^fstrim [0-9]+$
|
||||
|
||||
# Restore-test scratch teardown (F-LEAK, Campaign 8, v0.110.0). A restore-test whose restore FAILS
|
||||
# leaves a scratch guest the API token CANNOT destroy: `FelhomAgentGuest` is granted at /pool/felhom and
|
||||
# a guest joins that pool only when its restore COMPLETES, so a failed restore leaves a pool-less guest
|
||||
@@ -304,4 +336,4 @@ Cmnd_Alias FELHOM_GUESTNET = \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$
|
||||
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY, FELHOM_POWEROFF, FELHOM_VZPARTIAL, FELHOM_INTERNALPLACE
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
#!/bin/sh
|
||||
# /etc/grub.d/42_felhom_oneshot — the kernel lane's one-shot ENTRIES (R-836, `09` §3 decision 172, `11` §5.11).
|
||||
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it.
|
||||
#
|
||||
# One menu entry per installed Proxmox kernel, id `felhom-oneshot-<version>`, booted ONLY when 01_felhom_oneshot found
|
||||
# the flag naming it. It is the normal entry plus option C (decision 172): softlockup_panic=1 hardlockup_panic=1
|
||||
# hung_task_panic=1 panic=10 — a lockup the kernel can detect becomes a panic, and a panic restarts the box in 10 s into
|
||||
# the default (the old kernel). A true dead freeze still needs a person (spike candidate 3 failed on all three boxes).
|
||||
# It sorts AFTER 10_linux, so it is never entry 0 and never the default.
|
||||
#
|
||||
# No vfat ESP at /boot/efi → prints nothing (no flag can name these entries).
|
||||
set -e
|
||||
prefix="/usr"
|
||||
exec_prefix="/usr"
|
||||
datarootdir="/usr/share"
|
||||
. "$datarootdir/grub/grub-mkconfig_lib"
|
||||
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
|
||||
[ -n "$esp_uuid" ] || exit 0
|
||||
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
|
||||
[ -n "$kernels" ] || exit 0
|
||||
case "${GRUB_DEVICE}" in
|
||||
/dev/mapper/*|/dev/dm-*|"") root_arg="root=${GRUB_DEVICE}" ;;
|
||||
*) if [ -n "${GRUB_DEVICE_UUID}" ]; then root_arg="root=UUID=${GRUB_DEVICE_UUID}"; else root_arg="root=${GRUB_DEVICE}"; fi ;;
|
||||
esac
|
||||
[ -n "${GRUB_DEVICE}" ] || root_arg="root=$(findmnt -n -o SOURCE /)"
|
||||
rel=$(make_system_path_relative_to_its_root /boot)
|
||||
prep=$(prepare_grub_to_access_device "$(${grub_probe:-grub-probe} --target=device /boot)" | sed 's/^/ /')
|
||||
for k in $kernels; do
|
||||
[ -f "/boot/initrd.img-$k" ] || continue
|
||||
cat <<EOF
|
||||
menuentry 'Felhom one-shot: $k' --class proxmox --id felhom-oneshot-$k {
|
||||
insmod gzio
|
||||
$prep
|
||||
echo 'Loading Linux $k (felhom one-shot) ...'
|
||||
linux $rel/vmlinuz-$k $root_arg ro ${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
|
||||
initrd $rel/initrd.img-$k
|
||||
}
|
||||
EOF
|
||||
done
|
||||
@@ -0,0 +1,36 @@
|
||||
#!/bin/sh
|
||||
# /etc/grub.d/01_felhom_oneshot — the kernel lane's ONE-SHOT boot (R-836, `09` §3 decisions 164 + 172, `11` §5.11).
|
||||
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it; it prints GRUB script.
|
||||
#
|
||||
# At boot, GRUB reads `felhom_next` from an environment block on the ESP (vfat — GRUB can rewrite a file there; it
|
||||
# cannot on the LVM /boot, R-836), CLEARS it, and — only if it names an installed kernel — boots that kernel's one-shot
|
||||
# entry (42_felhom_oneshot) instead of the default. The next boot uses the default again whatever happens: a new kernel
|
||||
# that panics comes back on the old one by itself. felhom-os-apply writes the flag (mode apply) and never the default
|
||||
# for a new kernel. Measured on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
|
||||
# `audits/kernel-spike-2026-10-07/` (candidate 2).
|
||||
#
|
||||
# No vfat ESP at /boot/efi, or no Proxmox kernel → prints nothing (the box boots exactly as before).
|
||||
set -e
|
||||
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
|
||||
[ -n "$esp_uuid" ] || exit 0
|
||||
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
|
||||
[ -n "$kernels" ] || exit 0
|
||||
cat <<EOF
|
||||
# felhom kernel lane: a one-shot kernel named on the ESP, read and cleared before the menu
|
||||
insmod part_gpt
|
||||
insmod fat
|
||||
search --no-floppy --fs-uuid --set=felhom_esp $esp_uuid
|
||||
if [ -f (\$felhom_esp)/EFI/felhom/oneshot.env ]; then
|
||||
load_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
|
||||
if [ "\${felhom_next}" ]; then
|
||||
set felhom_boot="\${felhom_next}"
|
||||
set felhom_next=
|
||||
save_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
|
||||
EOF
|
||||
for k in $kernels; do
|
||||
printf ' if [ "${felhom_boot}" = "%s" ]; then set default="felhom-oneshot-%s"; fi\n' "$k" "$k"
|
||||
done
|
||||
cat <<EOF
|
||||
fi
|
||||
fi
|
||||
EOF
|
||||
Executable
+118
@@ -0,0 +1,118 @@
|
||||
#!/bin/bash
|
||||
# felhom-internal-place — make the one-disk box's internal data place („Belső tárhely"), felhom.eu
|
||||
# architecture/16-storage-physical-identity.md §5, §7 (sizes ruled by the operator 2026-10-16). Installed as
|
||||
# /usr/local/sbin/felhom-internal-place (config bundle). Called by felhom-host-install.sh (a new box) and by the agent's
|
||||
# signed `add_internal_place` job (an old box) — ONE implementation of the size rule and of the only mkfs it does.
|
||||
#
|
||||
# felhom-internal-place plan <planned GiB> → prints the numbers, changes nothing
|
||||
# felhom-internal-place create <planned GiB> → makes the place (or finds it), prints the result
|
||||
#
|
||||
# <planned GiB> is thin space still to be promised to the guest (rootfs + data volume) that does not exist yet:
|
||||
# the installer passes 32 + the data volume; an old box passes 0 (its guest's volumes are already counted).
|
||||
#
|
||||
# The rule (16 §7, ruled): reserve = max(16 GiB, 10 % of the pool), at most 48 GiB; internal = pool − reserve −
|
||||
# every thin volume already promised in the pool − <planned>, made only when ≥ 20 GiB. So nothing in the pool is ever
|
||||
# promised beyond pool − reserve: a pool at 100 % stops every volume on the box.
|
||||
#
|
||||
# It trusts nothing it is told. Every rule is checked here, as root, and any doubt REFUSES (exit 1):
|
||||
# - the pool is the `lvmthin: local-lvm` section of /etc/pve/storage.cfg (its vgname/thinpool), nothing else;
|
||||
# - the only volume it creates, formats or reads is <vg>/felhom-belso;
|
||||
# - it formats ONLY a felhom-belso that carries no signature at all (blkid -p finds nothing) — the one it just made,
|
||||
# or one a crash left between lvcreate and mkfs; any other content REFUSES; it never resizes or removes anything.
|
||||
# Output, one line: „created uuid=<fs-uuid> gib=<n>", „formatted uuid=… gib=…" (a crash left it empty), „existing uuid=<fs-uuid> gib=<n>", „skipped <why>" (exit 0),
|
||||
# or „plan pool=… reserve=… promised=… planned=… internal=…". Pinned by configs/test_felhom_internal_place.py.
|
||||
set -u
|
||||
STORAGE_CFG=${FELHOM_STORAGE_CFG:-/etc/pve/storage.cfg}
|
||||
LV_NAME=felhom-belso
|
||||
LABEL=felhom-belso
|
||||
MIN_GIB=20
|
||||
|
||||
die() { echo "felhom-internal-place: REFUSED: $*" >&2; exit 1; }
|
||||
|
||||
pool_of_local_lvm() { # prints "<vg> <thinpool>" of the lvmthin storage local-lvm, or nothing
|
||||
awk '
|
||||
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (type == "lvmthin" && id == "local-lvm"); next }
|
||||
in_s && /^[ \t]+vgname[ \t]/ { vg = $2 }
|
||||
in_s && /^[ \t]+thinpool[ \t]/ { tp = $2 }
|
||||
END { if (vg != "" && tp != "") print vg, tp }
|
||||
' "$STORAGE_CFG"
|
||||
}
|
||||
|
||||
gib_floor() { echo $(( $1 / 1073741824 )); }
|
||||
|
||||
[[ $# -eq 2 ]] || die "usage: felhom-internal-place plan|create <planned GiB>"
|
||||
MODE=$1; PLANNED=$2
|
||||
[[ "$MODE" == plan || "$MODE" == create ]] || die "mode must be plan or create"
|
||||
[[ "$PLANNED" =~ ^(0|[1-9][0-9]{0,5})$ ]] || die "planned GiB must be a whole number"
|
||||
|
||||
read -r VG POOL < <(pool_of_local_lvm) || true
|
||||
[[ -n "${VG:-}" && -n "${POOL:-}" ]] || die "no lvmthin storage local-lvm in $STORAGE_CFG"
|
||||
[[ "$VG" =~ ^[A-Za-z0-9_.+-]+$ && "$POOL" =~ ^[A-Za-z0-9_.+-]+$ ]] || die "unexpected vg/pool name"
|
||||
|
||||
pool_b=$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$POOL" 2>/dev/null | tr -d ' ')
|
||||
[[ "$pool_b" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$POOL"
|
||||
pool_gib=$(gib_floor "$pool_b")
|
||||
|
||||
# Every thin volume promised in the pool (its virtual size) — except the place itself and vzdump's short-lived
|
||||
# snapshots (snap_*_vzdump), which exist only during a whole-box copy and are what the reserve is for.
|
||||
# FAIL CLOSED: a failed or empty listing must never read as „nothing promised" (that would size the place too big).
|
||||
listing=$(lvs --noheadings --nosuffix --units b -o lv_name,pool_lv,lv_size "$VG" 2>/dev/null) || die "cannot list the volumes of $VG"
|
||||
pool_seen=false; promised_b=0; have_lv=false
|
||||
while read -r name pl size; do
|
||||
[[ -n "${name:-}" ]] || continue
|
||||
[[ "$name" == "$POOL" ]] && pool_seen=true
|
||||
[[ "$pl" == "$POOL" ]] || continue
|
||||
if [[ "$name" == "$LV_NAME" ]]; then have_lv=true; continue; fi
|
||||
[[ "$name" =~ ^snap_.*_vzdump$ ]] && continue
|
||||
[[ "$size" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$name"
|
||||
promised_b=$(( promised_b + size ))
|
||||
done <<< "$listing"
|
||||
$pool_seen || die "the listing of $VG does not contain the pool $POOL — refusing to guess what is promised"
|
||||
promised_gib=$(( (promised_b + 1073741823) / 1073741824 )) # rounded UP: never promise a partial GiB twice
|
||||
|
||||
reserve=$(( pool_gib / 10 )); (( reserve < 16 )) && reserve=16; (( reserve > 48 )) && reserve=48
|
||||
internal=$(( pool_gib - reserve - promised_gib - PLANNED ))
|
||||
|
||||
if [[ "$MODE" == plan ]]; then
|
||||
echo "plan pool=$pool_gib reserve=$reserve promised=$promised_gib planned=$PLANNED internal=$internal exists=$have_lv"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
DEV=/dev/$VG/$LV_NAME
|
||||
is_block() { [[ "$(stat -L -c %F "$1" 2>/dev/null)" == "block special file" ]]; }
|
||||
fs_uuid() { blkid -p -s UUID -o value "$DEV" 2>/dev/null; }
|
||||
fs_type() { blkid -p -s TYPE -o value "$DEV" 2>/dev/null; }
|
||||
fs_label() { blkid -p -s LABEL -o value "$DEV" 2>/dev/null; }
|
||||
# has_signature: FAIL CLOSED. blkid -p exits 0 when it found something and 2 when it found NOTHING; any other result (an
|
||||
# error, an ambivalent probe, blkid missing) must never read as „empty" — that is the only gate in front of mkfs.
|
||||
has_signature() {
|
||||
local rc=0
|
||||
blkid -p -o value "$DEV" >/dev/null 2>&1 || rc=$?
|
||||
case $rc in
|
||||
0) return 0 ;;
|
||||
2) return 1 ;;
|
||||
*) die "cannot probe $DEV for a signature (blkid exit $rc) — refusing" ;;
|
||||
esac
|
||||
}
|
||||
lv_gib() { gib_floor "$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$LV_NAME" 2>/dev/null | tr -d ' ')"; }
|
||||
|
||||
if $have_lv; then
|
||||
is_block "$DEV" || die "$DEV is not an active block device"
|
||||
t=$(fs_type)
|
||||
if [[ -n "$t" ]]; then
|
||||
[[ "$t" == ext4 && "$(fs_label)" == "$LABEL" ]] || die "$DEV holds something that is not Felhom's ($t) — left alone"
|
||||
echo "existing uuid=$(fs_uuid) gib=$(lv_gib)"; exit 0
|
||||
fi
|
||||
# A felhom-belso with NO signature: a crash between lvcreate and mkfs. Nothing can be on it — format it.
|
||||
has_signature && die "$DEV carries a signature blkid cannot name — left alone"
|
||||
else
|
||||
(( internal >= MIN_GIB )) || { echo "skipped only $internal GiB free for it (pool $pool_gib, reserve $reserve, promised $promised_gib, planned $PLANNED; needs $MIN_GIB)"; exit 0; }
|
||||
lvcreate -q -y -V "${internal}G" -T "$VG/$POOL" -n "$LV_NAME" >/dev/null || die "lvcreate failed"
|
||||
is_block "$DEV" || die "$DEV did not appear after lvcreate"
|
||||
has_signature && die "the new $DEV already carries a signature — left alone"
|
||||
made=created
|
||||
fi
|
||||
mkfs.ext4 -q -L "$LABEL" "$DEV" >/dev/null || die "mkfs.ext4 failed on $DEV"
|
||||
u=$(fs_uuid)
|
||||
[[ "$u" =~ ^[0-9a-fA-F-]{36}$ ]] || die "no filesystem UUID after mkfs on $DEV"
|
||||
echo "${made:-formatted} uuid=$u gib=$(lv_gib)"
|
||||
@@ -58,14 +58,26 @@ fi
|
||||
|
||||
# 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under
|
||||
# /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed.
|
||||
while read -r mp; do
|
||||
[[ -z "$mp" ]] && continue
|
||||
case "$mp" in
|
||||
/mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
|
||||
*) die "device (or a partition) is mounted at $mp ($dev)" ;;
|
||||
esac
|
||||
done < <(lsblk -nro MOUNTPOINT "$whole" 2>/dev/null || true)
|
||||
|
||||
# Agent v0.158.0: read from the kernel's mount table for EVERY node of the disk (lsblk's MOUNTPOINT shows one mount
|
||||
# per node; a bind elsewhere hid behind it), and a disk whose nodes cannot be listed REFUSES (it passed before).
|
||||
# Deliberately NO exemption for a sibling that is Felhom's own enrolled drive (raw /mnt/<name> + /mnt/felhom-drives):
|
||||
# the only corroboration available here is a mount the agent itself can make, so trusting it would let a compromised
|
||||
# agent unlock a host owner's disk. Consequence (felhom.eu `16` §13): the free part next to an in-use Felhom drive
|
||||
# cannot be formatted; the dashboard does not offer it.
|
||||
PROC_MOUNTS=/proc/mounts
|
||||
node_mounts() { awk -v d="$1" '$1 == d { print $2 }' "$PROC_MOUNTS"; }
|
||||
disk_nodes="$(lsblk -nrpo NAME "$whole" 2>/dev/null || true)"
|
||||
[[ -n "$disk_nodes" ]] || die "cannot list the nodes of $whole — refusing"
|
||||
while read -r node; do
|
||||
[[ -z "$node" ]] && continue
|
||||
while read -r mp; do
|
||||
[[ -z "$mp" ]] && continue
|
||||
case "$mp" in
|
||||
/mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
|
||||
*) die "device (or a partition) is mounted at $mp ($node)" ;;
|
||||
esac
|
||||
done < <(node_mounts "$node")
|
||||
done <<< "$disk_nodes"
|
||||
# 4) read-only device (audit D1): a device the kernel marks RO is never a formattable data disk.
|
||||
wbase="${whole#/dev/}"
|
||||
rof="/sys/block/$wbase/ro"
|
||||
|
||||
@@ -16,8 +16,10 @@ Cmnd_Alias FELHOM_OP_REPAIR = \
|
||||
/usr/bin/systemctl reset-failed felhom-sshd, \
|
||||
/usr/bin/systemctl restart felhom-sshd, \
|
||||
/usr/sbin/pct list, \
|
||||
/usr/sbin/pct start [0-9]*, \
|
||||
/usr/sbin/pct stop [0-9]*, \
|
||||
/usr/sbin/pct unlock [0-9]*
|
||||
/usr/sbin/pct ^start [0-9]+$, \
|
||||
/usr/sbin/pct ^stop [0-9]+$, \
|
||||
/usr/sbin/pct ^unlock [0-9]+$
|
||||
|
||||
# R-861 (b) B2 (`09` §3 decision 165, hygiene): one numeric vmid per pct verb, anchored — the old glob `[0-9]*` also
|
||||
# matched spaces, so `pct stop 9201 --skiplock 1` passed. Pinned by TestFelhomOpSudoersPctIsExact.
|
||||
felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR
|
||||
|
||||
+661
-19
@@ -21,6 +21,11 @@
|
||||
# or, for an unsigned ring-0 step, the root-owned TRUST_FILE saying `"ring0_slow_lane": true` (set by hand on the demo
|
||||
# boxes only). The agent's own config is NOT trusted for either: the agent can write it. A Docker step also needs
|
||||
# `live-restore` ON (R15) — without it every container restarts.
|
||||
# Agent v0.151.0 (R-812 option A, `09` §3 decision 163) adds the layer "pve": the HOST's Proxmox USERSPACE packages,
|
||||
# lane "slow" only, origin "Proxmox Debian Repository" only, never a kernel / boot / firmware / microcode name (R14 —
|
||||
# the kernel is R-836's lane), no removal, no undo, a NEW package only from PVE_NEW_ALLOW, an appliance only (R12), and
|
||||
# the same authority as the Docker step (R3: a signed `os_pve_step`, or the root-owned ring-0 mark). Select
|
||||
# "pending-pve" (ring 0): every installed Proxmox-origin package with a pending upgrade, minus HOST_SLOW_RE.
|
||||
#
|
||||
# Modes (plan field "mode"):
|
||||
# inventory `apt-get update`, then report what is installed (with origin), what is pending, and health.
|
||||
@@ -32,6 +37,14 @@
|
||||
# held packages, kernel taint, the crash guard; guest Debian, Docker engine, containerd, live-restore.
|
||||
# live-restore-on (v0.142.0, layer guest) the ONE-TIME step of `09` decision 87: merge `"live-restore": true`
|
||||
# into the guest's /etc/docker/daemon.json and `systemctl reload docker`. NEVER a restart (R-835).
|
||||
# oom-check (R-528, `09` decision 157; layer docker, lane slow) ONLY the memory-kill check below: no apt, no engine
|
||||
# change, no authority needed. Wrapper-only — the agent never writes this plan; run it by hand as root.
|
||||
# R-528 (`09` decision 157): a docker-layer APPLY also runs `oom_check()` after health_after and reports it as
|
||||
# "oom_check": {"result": "pass"|"fail"|"error", "oom_killed": bool, "oom_event": bool, "exit_code": int|null,
|
||||
# "image": str|null, "detail": str}
|
||||
# — one throwaway container (the controller's own image, no network / volume / port, 64 MB cap) is made to exceed its
|
||||
# memory; "pass" only when the engine says OOMKilled=true AND emits the `oom` event. It never changes the step's
|
||||
# outcome or health: the hub decides whether the engine set can be approved. Pinned by the OOMCheck tests.
|
||||
# Output: log lines on stderr and the journal (tag felhom-os-apply); the LAST stdout line is
|
||||
# OSAPPLY-REPORT <one JSON object>
|
||||
# which is what the agent parses. Exit 0 = done; 2 = refused (nothing changed); 3 = failed during install.
|
||||
@@ -68,10 +81,25 @@ JOURNAL_MARK = "@@FELHOM-DPKG-JOURNAL@@"
|
||||
DPKG_STATE_SCRIPT = "dpkg --audit; echo " + JOURNAL_MARK + "; ls -A /var/lib/dpkg/updates 2>/dev/null; true"
|
||||
# The installer's ROOT-OWNED record (felhom-host-install.sh `state_set mode`); the agent cannot write it.
|
||||
INSTALL_STATE = "/var/lib/felhom-install/state.json"
|
||||
# R-528: the memory-kill check (oom_check). One 200 MB block under a 64 MB cap: measured on Docker 29.8.2 to be
|
||||
# OOM-killed with OOMKilled=true and an `oom` event. Every call is bounded: timeouts (the clock read twice) + the
|
||||
# settle wait stay within 90 s.
|
||||
OOMCHECK_PREFIX = "felhom-oomcheck-"
|
||||
OOMCHECK_SCRIPT = "dd if=/dev/zero of=/dev/null bs=200M count=1"
|
||||
OOMCHECK_TIMEOUTS = {"image": 10, "clock": 5, "run": 30, "inspect": 10, "events": 10, "rm": 15}
|
||||
# Measured on demo-hp 9201 (Docker 29.8.2, 2026-10-06, audits/readback-2026-10-07/F/F1, F2): an `--until` taken right
|
||||
# after the run MISSED the oom event although OOMKilled=true; after a 2 s wait and `--until` = guest epoch + 1 it is
|
||||
# seen. Pinned by test_events_window_ends_after_the_settle_wait.
|
||||
OOMCHECK_SETTLE = 2
|
||||
# Kernel, boot and firmware packages are the SLOW lane on the host whatever their origin (`11` C3, §5.2): a host
|
||||
# reboot is needed for them to take effect, and a bad one can stop the box from booting.
|
||||
# proxmox-secure-boot-support is the Secure Boot meta-package: its only job is to pull shim-signed and the signed GRUB,
|
||||
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
|
||||
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
|
||||
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
|
||||
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
|
||||
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)")
|
||||
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
|
||||
r"proxmox-secure-boot-support)")
|
||||
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
|
||||
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
|
||||
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
|
||||
@@ -82,6 +110,12 @@ HOST_SERVICES = ["pveproxy", "pvedaemon", "pvestatd", "pve-cluster", "felhom-age
|
||||
DOCKER_NAMES = ("containerd.io", "docker-buildx-plugin", "docker-ce", "docker-ce-cli", "docker-ce-rootless-extras",
|
||||
"docker-compose-plugin")
|
||||
DOCKER_ORIGIN = "Docker CE"
|
||||
# The Proxmox package lane (R-812 option A): the origin apt prints for download.proxmox.com, and the ONLY new packages
|
||||
# a pve step may add (measured on demo-felhom 2026-10-07: a full upgrade adds proxmox-firewall-data and the kernel;
|
||||
# the kernel is refused by R14 whatever this list says).
|
||||
PVE_ORIGIN = "Proxmox Debian Repository"
|
||||
PVE_NEW_ALLOW = ("proxmox-firewall-data",)
|
||||
PVE_SIGNED_OP = "os_pve_step"
|
||||
# ROOT-OWNED trust anchors (the installer writes them; the demo boxes got them by hand, R-840). Never the agent's config.
|
||||
TRUST_FILE = "/etc/felhom/os-trust.json" # {"host_id": "...", "ring0_slow_lane": false}
|
||||
TRUST_SIGNERS = "/etc/felhom/operator-signers" # ssh allowed_signers: <key_id> namespaces="felhom-op-v1" <key>
|
||||
@@ -95,6 +129,32 @@ DAEMON_JSON = "/etc/docker/daemon.json"
|
||||
# wrapper restarts exactly the containers that mount one of these paths (never the apps, never the engine).
|
||||
DOCKER_SOCKETS = ("/var/run/docker.sock", "/run/docker.sock")
|
||||
CRASH_GUARD_STATE = "/var/lib/felhom-crash-guard/state.json"
|
||||
# ---------- the kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11) ----------
|
||||
# A new kernel boots ONCE through a flag in a GRUB environment block on the ESP (vfat — GRUB can rewrite it there; on
|
||||
# the LVM /boot it cannot, R-836). The bundle's two GRUB generators read and clear the flag (01_felhom_oneshot) and
|
||||
# give each installed kernel a one-shot entry with the lockup-to-panic options (42_felhom_oneshot, option C). The GRUB
|
||||
# default is the kernel the box RUNS, pinned in KERNEL_DEFAULT_CFG; only "kernel-good" (after a healthy one-shot boot)
|
||||
# moves it to the new kernel. Measured in the spike on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
|
||||
# `audits/kernel-spike-2026-10-07/`.
|
||||
KERNEL_OP = "os_kernel_step"
|
||||
KERNEL_STATE = "/var/lib/felhom-kernel/state.json" # 0644 root: the step's phase (the agent reads it)
|
||||
KERNEL_DEFAULT_CFG = "/etc/default/grub.d/zz-felhom-kernel-default.cfg" # sourced last: GRUB_DEFAULT = this kernel
|
||||
ONESHOT_SNIPPET = "/etc/grub.d/01_felhom_oneshot" # bundle-owned: read + clear the flag, pick the entry
|
||||
ONESHOT_ENTRIES = "/etc/grub.d/42_felhom_oneshot" # bundle-owned: the one-shot entries (option C options)
|
||||
GRUB_CFG = "/boot/grub/grub.cfg"
|
||||
ESP_MOUNT = "/boot/efi"
|
||||
ONESHOT_ENV = ESP_MOUNT + "/EFI/felhom/oneshot.env"
|
||||
ONESHOT_ARGS = "softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10"
|
||||
KERNEL_PIN_FILE = "/etc/kernel/proxmox-boot-pin" # an operator's `proxmox-boot-tool kernel pin` — never fought
|
||||
KVER_RE = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$")
|
||||
KERNEL_IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)(-signed)?$")
|
||||
# the packages a kernel step may UPGRADE (never add): the kernel series meta-package, the default-kernel meta, the boot
|
||||
# helper and the firmware the kernel loads. Everything else in HOST_SLOW_RE (grub, shim, microcode, efibootmgr) stays out.
|
||||
KERNEL_UPGRADE_RE = re.compile(r"^(proxmox-default-kernel|proxmox-kernel-[0-9]+\.[0-9]+|proxmox-kernel-helper|pve-firmware)$")
|
||||
KERNEL_MIN_GAP = 20 * 3600 # never two kernel steps in one night
|
||||
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel")
|
||||
# phases: staged → oneshot → judging → good | reverting → self_reverted | revert_failed; oneshot → fell_back; staged → cancelled
|
||||
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
||||
|
||||
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
|
||||
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
|
||||
@@ -129,6 +189,10 @@ BUNDLE_FILES = [
|
||||
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
|
||||
("/usr/local/sbin/felhom-pbs-apply", "felhom-pbs-apply", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-backup-target-apply", "felhom-backup-target-apply", 0o755, "bash", "replace"),
|
||||
# R-1006 (agent v0.156.0): the only delete of backup files — a dead half-written vzdump, re-checked as root.
|
||||
("/usr/local/sbin/felhom-vzdump-partial-clean", "felhom-vzdump-partial-clean", 0o755, "bash", "replace"),
|
||||
# felhom.eu architecture/16 §5 (agent v0.158.0): the internal data place — the only lvcreate/mkfs of a system-disk volume.
|
||||
("/usr/local/sbin/felhom-internal-place", "felhom-internal-place", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-os-apply", "felhom-os-apply", 0o755, "python", "replace"),
|
||||
("/usr/local/sbin/felhom-crash-guard", "felhom-crash-guard", 0o755, "python", "replace"),
|
||||
# R-861 (agent v0.146.0): the content checker for every agent-staged file a root program reads, and the two FIXED
|
||||
@@ -141,6 +205,10 @@ BUNDLE_FILES = [
|
||||
("/etc/systemd/system/felhom-crash-guard-check.service", "felhom-crash-guard-check.service", 0o644, "unit", "replace"),
|
||||
("/etc/systemd/system/felhom-crash-guard-check.timer", "felhom-crash-guard-check.timer", 0o644, "unit", "replace"),
|
||||
("/etc/felhom/crash-guard.conf", "crash-guard.conf", 0o644, "plain", "if-absent"),
|
||||
# the kernel lane's two GRUB generators (R-836, `11` §5.11): they take effect at the next update-grub, which the
|
||||
# first kernel step runs itself; with no flag on the ESP they change nothing about how the box boots.
|
||||
("/etc/grub.d/01_felhom_oneshot", "felhom-grub-oneshot.sh", 0o755, "sh", "replace"),
|
||||
("/etc/grub.d/42_felhom_oneshot", "felhom-grub-oneshot-entries.sh", 0o755, "sh", "replace"),
|
||||
("/etc/systemd/system/felhom-agent.service", "felhom-agent.service", 0o644, "agent-unit", "replace"),
|
||||
("/etc/systemd/system/felhom-agent-rollback.service", "felhom-agent-rollback.service", 0o644, "unit", "replace"),
|
||||
("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", "felhom-agent-limits.conf", 0o644, "dropin", "replace"),
|
||||
@@ -405,7 +473,8 @@ class Apply:
|
||||
|
||||
def check_plan(self, plan):
|
||||
mode = plan.get("mode", "apply")
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update"):
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \
|
||||
+ KERNEL_MODES:
|
||||
raise Refused("R11", f"unknown mode {mode!r}")
|
||||
if mode == "agent_update":
|
||||
if plan.get("layer") != "host":
|
||||
@@ -416,21 +485,31 @@ class Apply:
|
||||
raise Refused("R11", "bundle is a host-layer mode")
|
||||
return mode, "host", 0, "bundle"
|
||||
layer = plan.get("layer")
|
||||
if layer not in ("guest", "host", "docker"):
|
||||
raise Refused("R12", f"layer {layer!r} is not guest, host or docker")
|
||||
if layer not in ("guest", "host", "docker", "pve", "kernel"):
|
||||
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
|
||||
if (mode in KERNEL_MODES) != (layer == "kernel" and mode not in ("apply", "health")):
|
||||
raise Refused("R11", f"mode {mode!r} does not fit layer {layer!r}")
|
||||
lane = plan.get("lane", "fast")
|
||||
if layer == "docker" and lane != "slow":
|
||||
raise Refused("R3", "the Docker engine is the slow lane (`11` §5.8); a fast-lane Docker plan is refused")
|
||||
if layer != "docker" and lane != "fast":
|
||||
if layer == "pve" and lane != "slow":
|
||||
raise Refused("R3", "the Proxmox packages are the slow lane (`11` §5.10); a fast-lane pve plan is refused")
|
||||
if layer == "kernel" and lane != "slow":
|
||||
raise Refused("R3", "the kernel is the slow lane (`11` §5.11); a fast-lane kernel plan is refused")
|
||||
if layer not in ("docker", "pve", "kernel") and lane != "fast":
|
||||
raise Refused("R3", f"the {layer} layer has no slow lane in this release (kernel, Proxmox: `11` §8 step 6)")
|
||||
if mode == "facts" and layer != "host":
|
||||
raise Refused("R11", "facts is a host-layer mode (it reads the host and the guest)")
|
||||
if mode == "live-restore-on" and layer != "guest":
|
||||
raise Refused("R11", "live-restore-on is a guest-layer mode")
|
||||
if plan.get("undo") and layer != "docker":
|
||||
if mode == "oom-check" and layer != "docker":
|
||||
raise Refused("R11", "oom-check is a docker-layer mode (it checks the guest's Docker engine)")
|
||||
if plan.get("undo") and layer != "docker": # the pve layer has no undo in this release (R-812 option A)
|
||||
raise Refused("R5", "an undo (downgrade) exists only for the Docker layer, inside a signed job")
|
||||
vmid = plan.get("vmid")
|
||||
if not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
|
||||
if layer == "kernel" and mode in KERNEL_MODES and mode != "kernel-reboot" and vmid == 0:
|
||||
pass # after a boot the guest may not run (that is what is judged); these modes never touch it
|
||||
elif not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
|
||||
raise Refused("R11", f"vmid must be a positive integer, got {vmid!r}")
|
||||
rid = plan.get("release_id", "")
|
||||
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
|
||||
@@ -438,17 +517,24 @@ class Apply:
|
||||
if plan.get("allow_new"):
|
||||
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
|
||||
select = plan.get("select", "listed")
|
||||
if select not in ("listed", "pending-fast", "pending-docker"):
|
||||
if select not in ("listed", "pending-fast", "pending-docker", "pending-pve", "pending-kernel"):
|
||||
raise Refused("R11", f"unknown select {select!r}")
|
||||
if (select == "pending-docker") != (layer == "docker" and select != "listed"):
|
||||
if select == "pending-docker" or layer == "docker":
|
||||
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
||||
if (select == "pending-pve") != (layer == "pve" and select != "listed"):
|
||||
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
||||
if (select == "pending-kernel") != (layer == "kernel" and select != "listed"):
|
||||
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
||||
ek = plan.get("expect_kver")
|
||||
if ek is not None and (layer != "kernel" or not isinstance(ek, str) or not KVER_RE.match(ek)):
|
||||
raise Refused("R11", f"expect_kver {ek!r} is not a kernel version of the kernel layer")
|
||||
pk = plan.get("packages", [])
|
||||
if not isinstance(pk, list):
|
||||
raise Refused("R11", "packages must be a list")
|
||||
if mode == "apply" and select == "listed" and not pk:
|
||||
raise Refused("R11", "packages must be a non-empty list in apply mode (select listed)")
|
||||
if select in ("pending-fast", "pending-docker") and pk:
|
||||
if select in ("pending-fast", "pending-docker", "pending-pve", "pending-kernel") and pk:
|
||||
raise Refused("R11", f"select {select} takes no package list")
|
||||
seen = set()
|
||||
for e in pk:
|
||||
@@ -468,6 +554,19 @@ class Apply:
|
||||
continue
|
||||
if n in DOCKER_NAMES:
|
||||
raise Refused("R2", f"{n} is a Docker package — the slow lane (`11` §5.8), never in a {layer} plan")
|
||||
if layer == "kernel":
|
||||
if o != PVE_ORIGIN:
|
||||
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the kernel layer)")
|
||||
if not (KERNEL_UPGRADE_RE.match(n) or KERNEL_IMAGE_RE.match(n)):
|
||||
raise Refused("R23", f"{n} is not a kernel package (a kernel image, the kernel meta-packages, "
|
||||
f"proxmox-kernel-helper or pve-firmware)")
|
||||
continue
|
||||
if layer == "pve":
|
||||
if o != PVE_ORIGIN:
|
||||
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the pve layer)")
|
||||
if HOST_SLOW_RE.match(n):
|
||||
raise Refused("R14", f"{n} is a kernel / boot / firmware package — never the pve lane (R-836)")
|
||||
continue
|
||||
if o not in FAST_ORIGINS:
|
||||
raise Refused("R2", f"{n}: origin {o!r} is not Debian / Debian-Security (the fast lane, `11` C3)")
|
||||
if layer == "host" and HOST_SLOW_RE.match(n):
|
||||
@@ -610,12 +709,13 @@ class Apply:
|
||||
now = self.r.now()
|
||||
self.r.write_nonces({k: v for k, v in seen.items() if v > now})
|
||||
|
||||
def docker_authority(self, plan):
|
||||
"""R3 for the docker layer: returns (who, undo). A signed job binds the EXACT package list and the undo flag."""
|
||||
def docker_authority(self, plan, op_name=SIGNED_OP):
|
||||
"""R3 for the docker (and, op_name os_pve_step, the pve) layer: returns (who, undo). A signed job binds the EXACT
|
||||
package list and the undo flag."""
|
||||
trust = self.load_trust()
|
||||
signed = plan.get("signed")
|
||||
if signed:
|
||||
params = self.verify_signed(signed, trust)
|
||||
params = self.verify_signed(signed, trust, op_name=op_name)
|
||||
want = sorted(f"{e.get('name')}={e.get('version')}" for e in params.get("packages") or [])
|
||||
got = sorted(f"{e['name']}={e['version']}" for e in plan.get("packages", []))
|
||||
if not want or want != got:
|
||||
@@ -629,7 +729,7 @@ class Apply:
|
||||
raise Refused("R3", "an undo (downgrade) needs a signed operator job")
|
||||
if trust.get("ring0_slow_lane") is True:
|
||||
return "ring0", False
|
||||
raise Refused("R3", "a Docker step needs a signed operator job (ring 1) or this box's root-owned ring-0 mark")
|
||||
raise Refused("R3", f"a {self.layer} slow-lane step needs a signed operator job (ring 1) or this box's root-owned ring-0 mark")
|
||||
|
||||
def live_restore(self):
|
||||
rc, out, _ = self.g(["docker", "info", "--format", "{{.LiveRestoreEnabled}}"], timeout=60)
|
||||
@@ -717,6 +817,17 @@ class Apply:
|
||||
return v or "unknown"
|
||||
h = {"debian": first(["cat", "/etc/debian_version"]), "kernel_running": first(["uname", "-r"])}
|
||||
h["kernel_next_boot"], h["kernel_next_boot_source"] = self.kernel_next_boot()
|
||||
try:
|
||||
# the kernel lane (R-836): the default and the one-shot flag, read from grub.cfg and the ESP themselves
|
||||
kv = Kernel(self, {}).view()
|
||||
kv["setup_problems"] = Kernel(self, {}).setup_problems()
|
||||
h["kernel_lane"] = kv
|
||||
if kv.get("flag"):
|
||||
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["flag"], "felhom one-shot flag (once; then the default)"
|
||||
elif kv.get("default") not in (None, "unknown"):
|
||||
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["default"], "grub.cfg default"
|
||||
except Exception as e: # never cost the System page its other facts
|
||||
h["kernel_lane"] = {"error": str(e)[:200]}
|
||||
rc, out, _ = self.r.host(["apt-mark", "showhold"], 60)
|
||||
h["held"] = sorted(out.split()) if rc == 0 else None
|
||||
try:
|
||||
@@ -775,8 +886,8 @@ class Apply:
|
||||
|
||||
# ---------- target helpers ----------
|
||||
def x(self, argv, timeout=1800):
|
||||
"""Run in the TARGET layer: the guest via pct exec, or the host directly."""
|
||||
if self.layer == "host":
|
||||
"""Run in the TARGET layer: the guest via pct exec, or the host directly (host, pve and kernel)."""
|
||||
if self.layer in ("host", "pve", "kernel"):
|
||||
return self.r.host(argv, timeout)
|
||||
return self.r.guest(self.vmid, argv, timeout) # guest and docker both live in the customer guest
|
||||
|
||||
@@ -871,7 +982,7 @@ class Apply:
|
||||
def restart_needed(self):
|
||||
"""Processes still mapping deleted files, OUTSIDE containers (C11). Guest: outside docker; host: outside the
|
||||
LXC guests (the host's /proc shows guest processes too)."""
|
||||
skip = RESTART_SKIP_CGROUP["host" if self.layer == "host" else "guest"]
|
||||
skip = RESTART_SKIP_CGROUP["host" if self.layer in ("host", "pve", "kernel") else "guest"]
|
||||
script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
|
||||
'grep -q "%s" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done' % skip)
|
||||
rc, out, _ = self.x(["sh", "-c", script], timeout=120)
|
||||
@@ -945,13 +1056,24 @@ class Apply:
|
||||
return Bundle(self).from_plan(plan)
|
||||
if self.mode == "agent_update":
|
||||
return self.agent_update(plan)
|
||||
if self.layer == "host":
|
||||
if self.layer in ("host", "pve", "kernel"):
|
||||
self.check_appliance()
|
||||
if self.layer == "kernel" and self.mode != "health":
|
||||
# the kernel lane's own modes (R-836): most of them run while the guest is still starting after a boot, so
|
||||
# the guest check is the stage's and the reboot's own (Kernel.run), not every mode's
|
||||
return Kernel(self, plan).run()
|
||||
self.check_guest(self.vmid)
|
||||
log = self.r.log
|
||||
if self.mode == "live-restore-on":
|
||||
return self.live_restore_on()
|
||||
if self.mode == "oom-check":
|
||||
# R-528: the check alone — no apt, no engine change; check_guest above still applies.
|
||||
self.report["oom_check"] = self.oom_check()
|
||||
return 0
|
||||
self.who, self.allow_downgrade = ("fast", False)
|
||||
if self.layer == "pve" and self.mode == "apply":
|
||||
self.who, self.allow_downgrade = self.docker_authority(plan, op_name=PVE_SIGNED_OP)
|
||||
self.report["authority"] = self.who
|
||||
if self.layer == "docker" and self.mode == "apply":
|
||||
self.who, self.allow_downgrade = self.docker_authority(plan)
|
||||
if self.live_restore() != "true":
|
||||
@@ -964,7 +1086,7 @@ class Apply:
|
||||
log(f"os-apply: START release={plan.get('release_id')} layer={self.layer}" +
|
||||
(f":{self.vmid}" if self.layer != "host" else "") +
|
||||
f" lane={plan.get('lane', 'fast')} mode={self.mode} select={self.select} packages={len(plan.get('packages', []))}" +
|
||||
(f" authority={self.who}{' UNDO' if self.allow_downgrade else ''}" if self.layer == "docker" else ""))
|
||||
(f" authority={self.who}{' UNDO' if self.allow_downgrade else ''}" if self.layer in ("docker", "pve") else ""))
|
||||
if self.apt_lock_held():
|
||||
raise Refused("R9", f"another apt/dpkg holds the lock on the {self.layer}")
|
||||
self.report["health_before"] = self.health()
|
||||
@@ -988,10 +1110,91 @@ class Apply:
|
||||
if self.layer == "docker":
|
||||
rc_v, out_v, _ = self.g(["docker", "version", "--format", "{{.Server.Version}}"], timeout=60)
|
||||
self.report["docker_engine"] = out_v.strip() if rc_v == 0 and out_v.strip() else "unknown"
|
||||
if self.layer == "pve":
|
||||
self.report["pve_manager"] = self.pve_manager()
|
||||
self.report["reboot_scanned"] = "reboot_needed" in self.report
|
||||
self.report["health_after"] = self.health()
|
||||
if self.layer == "docker" and self.mode == "apply":
|
||||
# R-528 (`09` decision 157): does the engine report a memory kill? Reported only — never the outcome.
|
||||
self.report["oom_check"] = self.oom_check()
|
||||
return 0
|
||||
|
||||
# ---------- R-528: the memory-kill check ----------
|
||||
def oom_check(self):
|
||||
"""Run one throwaway container over its memory cap in the guest and read what the engine says about it.
|
||||
Never raises: any failure becomes result "error". The container is ALWAYS removed (finally), and a failed
|
||||
removal is named in the detail."""
|
||||
t = OOMCHECK_TIMEOUTS
|
||||
res = {"result": "error", "oom_killed": False, "oom_event": False, "exit_code": None, "image": None, "detail": ""}
|
||||
log = self.r.log
|
||||
try:
|
||||
rc, out, err = self.g(["docker", "inspect", "-f", "{{.Config.Image}}", "felhom-controller"], timeout=t["image"])
|
||||
except Exception as e:
|
||||
rc, out, err = -1, "", str(e)
|
||||
img = out.strip() if rc == 0 else ""
|
||||
if not img or any(c.isspace() for c in img):
|
||||
res["detail"] = f"the controller's image could not be read (rc={rc}): {(err or out).strip()[:200]}"
|
||||
log(f"os-apply: OOM-CHECK error — {res['detail']}")
|
||||
return res
|
||||
res["image"] = img
|
||||
name = f"{OOMCHECK_PREFIX}{os.getpid()}-{os.urandom(4).hex()}"
|
||||
notes = []
|
||||
try:
|
||||
t0 = self.guest_epoch()
|
||||
if t0 is None:
|
||||
raise RuntimeError("the guest clock could not be read")
|
||||
rrc, rout, rerr = self.g(["docker", "run", "--name", name, "--pull", "never", "--network", "none",
|
||||
"--memory", "64m", "--memory-swap", "64m", "--label", "felhom.oomcheck=1",
|
||||
"--entrypoint", "sh", img, "-c", OOMCHECK_SCRIPT], timeout=t["run"])
|
||||
self.r.sleep(OOMCHECK_SETTLE) # the engine publishes the oom event a moment after the run returns (F1/F2)
|
||||
t1 = self.guest_epoch()
|
||||
if t1 is None:
|
||||
t1 = t0 + t["run"] + OOMCHECK_SETTLE + 1
|
||||
irc, iout, ierr = self.g(["docker", "inspect", "-f", "{{.State.OOMKilled}} {{.State.ExitCode}}", name], timeout=t["inspect"])
|
||||
if irc != 0:
|
||||
raise RuntimeError(f"the check container could not be inspected (run rc={rrc}: {(rerr or rout).strip()[:120]}; "
|
||||
f"inspect rc={irc}: {(ierr or iout).strip()[:120]})")
|
||||
f = iout.split()
|
||||
res["oom_killed"] = bool(f) and f[0] == "true"
|
||||
try:
|
||||
res["exit_code"] = int(f[1]) if len(f) > 1 else None
|
||||
except ValueError:
|
||||
res["exit_code"] = None
|
||||
erc, eout, eerr = self.g(["docker", "events", "--since", str(t0 - 1), "--until", str(t1 + 1),
|
||||
"--filter", f"container={name}", "--filter", "event=oom",
|
||||
"--format", "{{.Action}}"], timeout=t["events"])
|
||||
if erc != 0:
|
||||
notes.append(f"the event read failed (rc={erc}): {(eerr or eout).strip()[:120]}")
|
||||
res["oom_event"] = erc == 0 and any(l.strip() == "oom" for l in eout.splitlines())
|
||||
if res["oom_killed"] and res["oom_event"]:
|
||||
res["result"] = "pass"
|
||||
notes.insert(0, "the engine reported the memory kill: OOMKilled=true and the oom event")
|
||||
else:
|
||||
res["result"] = "fail"
|
||||
miss = [w for w, ok in (("OOMKilled=true", res["oom_killed"]), ("the oom event", res["oom_event"])) if not ok]
|
||||
notes.insert(0, f"the engine did not report the memory kill: missing {' and '.join(miss)} (exit code {res['exit_code']})")
|
||||
except Exception as e:
|
||||
res["result"] = "error"
|
||||
notes.insert(0, f"the check could not finish: {type(e).__name__}: {str(e)[:200]}")
|
||||
finally:
|
||||
try:
|
||||
mrc, mout, merr = self.g(["docker", "rm", "-f", name], timeout=t["rm"])
|
||||
if mrc != 0 and "no such container" not in (merr + mout).lower():
|
||||
notes.append(f"the check container {name} could not be removed (rc={mrc}): {(merr or mout).strip()[:120]}")
|
||||
except Exception as e:
|
||||
notes.append(f"the check container {name} could not be removed: {type(e).__name__}: {str(e)[:120]}")
|
||||
res["detail"] = "; ".join(notes)
|
||||
log(f"os-apply: OOM-CHECK result={res['result']} oom_killed={res['oom_killed']} oom_event={res['oom_event']} "
|
||||
f"exit={res['exit_code']} image={img} — {res['detail']}")
|
||||
return res
|
||||
|
||||
def guest_epoch(self):
|
||||
rc, out, _ = self.g(["date", "+%s"], timeout=OOMCHECK_TIMEOUTS["clock"])
|
||||
try:
|
||||
return int(out.strip()) if rc == 0 else None
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
def dpkg_state(self):
|
||||
"""`dpkg --audit` AND dpkg's update journal, in ONE call (R-876, agent v0.145.0). A crash in the middle of an
|
||||
install can leave `/var/lib/dpkg/updates/` non-empty while `--audit` reads clean — measured on demo-hp
|
||||
@@ -1058,10 +1261,26 @@ class Apply:
|
||||
return [{"name": p["name"], "version": p["to"], "origin": DOCKER_ORIGIN} for p in pend
|
||||
if p["from"] is not None and p["name"] in DOCKER_NAMES and self.origin_name(p["origin"]) == {DOCKER_ORIGIN}]
|
||||
|
||||
def pending_pve(self):
|
||||
"""Ring 0 (select pending-pve): the newest pending version of each INSTALLED Proxmox-origin package, never a
|
||||
kernel / boot / firmware / microcode name (HOST_SLOW_RE, R-836's lane), never another origin."""
|
||||
rc, pend, remv, _ = self.simulate(["dist-upgrade"])
|
||||
return [{"name": p["name"], "version": p["to"], "origin": PVE_ORIGIN} for p in pend
|
||||
if p["from"] is not None and not HOST_SLOW_RE.match(p["name"]) and p["name"] not in DOCKER_NAMES
|
||||
and self.origin_name(p["origin"]) == {PVE_ORIGIN}]
|
||||
|
||||
def pve_manager(self):
|
||||
"""pveversion's pve-manager version ("unknown" when it cannot be read)."""
|
||||
rc, out, _ = self.r.host(["pveversion"], 60)
|
||||
m = re.match(r"^pve-manager/([^/\s]+)", out.strip()) if rc == 0 else None
|
||||
return m.group(1) if m else "unknown"
|
||||
|
||||
def origin_ok(self, origin):
|
||||
o = self.origin_name(origin)
|
||||
if self.layer == "docker":
|
||||
return o == {DOCKER_ORIGIN}
|
||||
if self.layer == "pve":
|
||||
return o == {PVE_ORIGIN}
|
||||
return bool(o & set(FAST_ORIGINS))
|
||||
|
||||
def apply(self, plan):
|
||||
@@ -1070,6 +1289,8 @@ class Apply:
|
||||
packages = plan["packages"]
|
||||
elif self.select == "pending-docker":
|
||||
packages = self.pending_docker()
|
||||
elif self.select == "pending-pve":
|
||||
packages = self.pending_pve()
|
||||
else:
|
||||
packages = self.pending_fast()
|
||||
cmp_op = "ne" if self.allow_downgrade else "gt"
|
||||
@@ -1118,6 +1339,11 @@ class Apply:
|
||||
raise Refused("R4", f"the plan would remove {', '.join(remv[:5])}")
|
||||
want = dict(upgrade)
|
||||
for p in sim:
|
||||
if p["from"] is None and self.layer == "pve" and p["name"] in PVE_NEW_ALLOW and self.origin_ok(p["origin"]) \
|
||||
and not HOST_SLOW_RE.match(p["name"]):
|
||||
self.r.log(f"os-apply: NEW {p['name']}={p['to']} (on the pve lane's allow-list)")
|
||||
self.report.setdefault("added", []).append({"name": p["name"], "version": p["to"]})
|
||||
continue
|
||||
if p["from"] is None:
|
||||
raise Refused("R6", f"the plan would add a package that is not installed: {p['name']}")
|
||||
if p["name"] not in want:
|
||||
@@ -1128,7 +1354,7 @@ class Apply:
|
||||
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
|
||||
if not self.origin_ok(p["origin"]):
|
||||
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not the {self.layer} layer's origin")
|
||||
if self.layer == "host" and HOST_SLOW_RE.match(p["name"]):
|
||||
if self.layer in ("host", "pve") and HOST_SLOW_RE.match(p["name"]):
|
||||
raise Refused("R14", f"{p['name']} is a kernel / boot / firmware package — the host's slow lane")
|
||||
need = self.download_bytes(args)
|
||||
free = self.free_bytes()
|
||||
@@ -1211,6 +1437,422 @@ class Apply:
|
||||
self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||
|
||||
|
||||
def _iso(t):
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(t))
|
||||
|
||||
|
||||
def _parse_iso(s):
|
||||
try:
|
||||
return calendar.timegm(time.strptime(s, "%Y-%m-%dT%H:%M:%SZ"))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
class Kernel:
|
||||
"""The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). Refusal codes: R3 (authority), R4 (removal),
|
||||
R6 (a package outside the step), R8 (space), R9 (locks), R12 (appliance), R20 (the box's boot setup cannot do a
|
||||
one-shot), R21 (the crash guard is tripped or saw an unclean boot within its window), R22 (the step's phase does not
|
||||
allow this mode), R23 (the kernel set is not one exact new kernel).
|
||||
|
||||
Modes (plan "mode", layer "kernel", lane "slow"):
|
||||
apply STAGE: install the kernel set, keep the GRUB default on the kernel the box runs, write the flag.
|
||||
Never reboots. select "pending-kernel" (ring 0, the root-owned mark) or "listed" (a signed
|
||||
os_kernel_step). expect_kver: the kernel the hub told the household about — any other is R23.
|
||||
kernel-reboot a STAGED step's reboot (the night leg, after the household was told): phase oneshot, then reboot.
|
||||
kernel-boot after a boot: what became of the step (judging | fell_back | self_reverted | revert_failed).
|
||||
kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default.
|
||||
kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default).
|
||||
kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved).
|
||||
kernel-status read only."""
|
||||
|
||||
def __init__(self, apply, plan):
|
||||
self.a, self.r, self.plan = apply, apply.r, plan
|
||||
self.report = apply.report
|
||||
self.log = apply.r.log
|
||||
|
||||
# ---------- reading the box ----------
|
||||
def running(self):
|
||||
rc, out, _ = self.r.host(["uname", "-r"], 30)
|
||||
v = out.strip() if rc == 0 else ""
|
||||
return v if KVER_RE.match(v) else ""
|
||||
|
||||
def state(self):
|
||||
try:
|
||||
s = json.loads(self.r.read_file(KERNEL_STATE))
|
||||
return s if isinstance(s, dict) else {}
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
|
||||
def save_state(self, s):
|
||||
s["updated_at"] = _iso(self.r.now())
|
||||
self.r.put_file(KERNEL_STATE, (json.dumps(s, indent=2, sort_keys=True) + "\n").encode(), 0o644)
|
||||
|
||||
def flag(self):
|
||||
"""The one-shot flag: the kernel named, "" when the env block holds none, None when there is no env block."""
|
||||
rc, out, _ = self.r.host(["grub-editenv", ONESHOT_ENV, "list"], 30)
|
||||
if rc != 0:
|
||||
return None
|
||||
for l in out.splitlines():
|
||||
if l.startswith("felhom_next="):
|
||||
return l.split("=", 1)[1].strip()
|
||||
return ""
|
||||
|
||||
def grub_cfg(self):
|
||||
try:
|
||||
return self.r.read_file(GRUB_CFG)
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
@staticmethod
|
||||
def default_kver(cfg):
|
||||
"""The kernel grub.cfg boots by default (00_header's `set default=`), or "unknown"."""
|
||||
m = re.search(r'^\s*set default="(?:gnulinux-advanced-[^>"]*>)?gnulinux-([0-9][^"]*?-pve)-advanced-[^"]*"', cfg, re.M)
|
||||
return m.group(1) if m else "unknown"
|
||||
|
||||
@staticmethod
|
||||
def entry_id(cfg, kver):
|
||||
"""The GRUB_DEFAULT value that names kver's normal entry, read from grub.cfg itself (10_linux's ids)."""
|
||||
sub = re.search(r"\$menuentry_id_option '(gnulinux-advanced-[^']+)'", cfg)
|
||||
m = re.search(r"\$menuentry_id_option '(gnulinux-" + re.escape(kver) + r"-advanced-[^']+)'", cfg)
|
||||
if not m:
|
||||
return None
|
||||
return f"{sub.group(1)}>{m.group(1)}" if sub else m.group(1)
|
||||
|
||||
def setup_problems(self):
|
||||
"""R20: why this box cannot do a one-shot boot (empty = it can). Measured shape: UEFI, a vfat ESP at /boot/efi,
|
||||
/boot on the root filesystem, GRUB with fat + loadenv, the bundle's two generators, no hand pin."""
|
||||
why = []
|
||||
if not self.r.lexists("/sys/firmware/efi"):
|
||||
why.append("the box does not boot UEFI")
|
||||
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "FSTYPE", ESP_MOUNT], 30)
|
||||
if rc != 0 or out.strip() != "vfat":
|
||||
why.append(f"{ESP_MOUNT} is not a mounted vfat ESP ({out.strip() or 'not mounted'})")
|
||||
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "TARGET", "/boot"], 30)
|
||||
if rc == 0 and out.strip():
|
||||
why.append("/boot is a separate filesystem (the one-shot entries assume /boot on the root filesystem)")
|
||||
for m in ("fat", "loadenv"):
|
||||
if not self.r.lexists(f"/usr/lib/grub/x86_64-efi/{m}.mod"):
|
||||
why.append(f"GRUB has no {m} module")
|
||||
for p in (ONESHOT_SNIPPET, ONESHOT_ENTRIES):
|
||||
if not self.r.lexists(p):
|
||||
why.append(f"{p} is missing (the config bundle installs it)")
|
||||
if self.r.lexists(KERNEL_PIN_FILE):
|
||||
why.append("a kernel is pinned by hand (proxmox-boot-tool kernel pin) — the lane never fights it")
|
||||
return why
|
||||
|
||||
def guard(self):
|
||||
try:
|
||||
return json.loads(self.r.read_file(CRASH_GUARD_STATE))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
def check_guard(self):
|
||||
"""R21: a kernel step only on a box whose crash guard is armed and saw no unclean boot within its window — so
|
||||
the step's own reboots (clean), one crash and one self-revert (clean) can never reach the 3rd unclean boot that
|
||||
leaves the box off (`11` §5.9). Pinned by test_felhom_crash_guard KernelStepCannotLeaveTheBoxOff."""
|
||||
g = self.guard()
|
||||
if not isinstance(g, dict):
|
||||
raise Refused("R21", f"no crash guard state ({CRASH_GUARD_STATE}) — a kernel step needs the guard")
|
||||
if g.get("tripped") or not g.get("armed"):
|
||||
raise Refused("R21", "the crash guard is tripped — no kernel step until it re-arms")
|
||||
if (g.get("unclean_boots_in_window") or 0) > 0:
|
||||
raise Refused("R21", f"{g.get('unclean_boots_in_window')} unclean boot(s) within the guard's window — wait")
|
||||
|
||||
def view(self, st=None, cfg=None):
|
||||
st = self.state() if st is None else st
|
||||
cfg = self.grub_cfg() if cfg is None else cfg
|
||||
return {"running": self.running() or "unknown", "default": self.default_kver(cfg), "flag": self.flag(),
|
||||
"phase": st.get("phase", "none"), "from": st.get("from"), "to": st.get("to"),
|
||||
"step_id": st.get("step_id"), "self_revert_used": bool(st.get("self_revert_used")), "vmid": st.get("vmid"),
|
||||
"staged_at": st.get("staged_at"), "rebooted_at": st.get("rebooted_at"),
|
||||
"result_at": st.get("result_at"), "reason": st.get("reason")}
|
||||
|
||||
# ---------- writing the box ----------
|
||||
def write_default(self, kver):
|
||||
"""Pin the GRUB default to kver's normal entry, regenerate grub.cfg, and PROVE it (the default read back)."""
|
||||
cfg = self.grub_cfg()
|
||||
eid = self.entry_id(cfg, kver)
|
||||
if not eid:
|
||||
raise Refused("R20", f"grub.cfg has no normal entry for {kver}")
|
||||
body = ("# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily\n"
|
||||
f'GRUB_DEFAULT="{eid}"\n')
|
||||
self.r.put_file(KERNEL_DEFAULT_CFG, body.encode(), 0o644)
|
||||
rc, out, err = self.r.host(["update-grub"], 300)
|
||||
got = self.default_kver(self.grub_cfg())
|
||||
if rc != 0 or got != kver:
|
||||
raise Refused("R20", f"update-grub rc={rc}: the default reads {got}, not {kver}: {(out + err).strip()[-200:]}")
|
||||
self.log(f"os-apply: KERNEL default = {kver} (proved from grub.cfg)")
|
||||
|
||||
def set_flag(self, kver):
|
||||
self.r.host(["mkdir", "-p", os.path.dirname(ONESHOT_ENV)], 30)
|
||||
if self.flag() is None:
|
||||
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "create"], 30)
|
||||
if rc != 0:
|
||||
raise Refused("R20", f"cannot create the one-shot env block on the ESP: {(out + err).strip()[-200:]}")
|
||||
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "set", f"felhom_next={kver}"], 30)
|
||||
if rc != 0 or self.flag() != kver:
|
||||
raise Refused("R20", f"the one-shot flag did not read back as {kver}: {(out + err).strip()[-200:]}")
|
||||
|
||||
def clear_flag(self):
|
||||
if self.flag():
|
||||
self.r.host(["grub-editenv", ONESHOT_ENV, "unset", "felhom_next"], 30)
|
||||
|
||||
def reboot(self, why):
|
||||
self.log(f"os-apply: KERNEL REBOOT — {why}")
|
||||
rc, out, err = self.r.host(["systemctl", "reboot"], 60)
|
||||
self.report["reboot_rc"] = rc
|
||||
if rc != 0:
|
||||
self.report["failed"] = {"rc": 3, "step": "reboot", "reason": (out + err).strip()[-200:]}
|
||||
return 3
|
||||
return 0
|
||||
|
||||
# ---------- the modes ----------
|
||||
def run(self):
|
||||
mode = self.a.mode
|
||||
self.report["kernel_mode"] = mode
|
||||
if mode == "kernel-status":
|
||||
v = self.view()
|
||||
v["setup_problems"] = self.setup_problems()
|
||||
self.report["kernel"] = v
|
||||
return 0
|
||||
fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot,
|
||||
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel}[mode]
|
||||
rc = fn()
|
||||
self.report["kernel"] = self.view()
|
||||
return rc
|
||||
|
||||
def phase_is(self, st, *phases):
|
||||
if st.get("phase") not in phases:
|
||||
raise Refused("R22", f"the kernel step is {st.get('phase', 'none')!r}, not {' or '.join(phases)} — "
|
||||
f"{self.a.mode} does not apply")
|
||||
|
||||
def stage(self):
|
||||
a = self.a
|
||||
st = self.state()
|
||||
if st.get("phase") in KERNEL_ACTIVE:
|
||||
raise Refused("R22", f"a kernel step is already {st['phase']} ({st.get('from')} -> {st.get('to')})")
|
||||
last = _parse_iso(st.get("staged_at"))
|
||||
if last is not None and self.r.now() - last < KERNEL_MIN_GAP:
|
||||
raise Refused("R22", "a kernel step was staged within the last 20 hours — never two in one night")
|
||||
why = self.setup_problems()
|
||||
if why:
|
||||
raise Refused("R20", "; ".join(why))
|
||||
self.check_guard()
|
||||
a.check_guest(a.vmid)
|
||||
who, _ = a.docker_authority(self.plan, op_name=KERNEL_OP)
|
||||
self.report["authority"] = who
|
||||
old = self.running()
|
||||
if not old:
|
||||
raise Refused("R20", "the running kernel is not a Proxmox kernel version")
|
||||
self.log(f"os-apply: START release={self.plan.get('release_id')} layer=kernel lane=slow mode=apply "
|
||||
f"select={a.select} authority={who} running={old}")
|
||||
if a.apt_lock_held():
|
||||
raise Refused("R9", "another apt/dpkg holds the lock on the host")
|
||||
self.report["health_before"] = a.health()
|
||||
a.repair()
|
||||
rc, out, err = a.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||
if rc != 0:
|
||||
raise Refused("R7", f"apt-get update failed on the host: {(out + err).strip().splitlines()[-1:]}")
|
||||
inst = a.installed()
|
||||
if a.select == "pending-kernel":
|
||||
_, pend, _, _ = a.simulate(["dist-upgrade"])
|
||||
want = [(p["name"], p["to"]) for p in pend if p["from"] is not None and KERNEL_UPGRADE_RE.match(p["name"])
|
||||
and a.origin_name(p["origin"]) == {PVE_ORIGIN}]
|
||||
else:
|
||||
want = [(e["name"], e["version"]) for e in self.plan["packages"]]
|
||||
# upgrades of installed names (never a downgrade), and at most the listed new kernel image
|
||||
args, upg = [], {}
|
||||
for n, v in want:
|
||||
if n in inst:
|
||||
if a.dpkg_cmp(v, "gt", inst[n]):
|
||||
upg[n] = v
|
||||
args.append(f"{n}={v}")
|
||||
elif KERNEL_IMAGE_RE.match(n):
|
||||
args.append(f"{n}={v}")
|
||||
else:
|
||||
raise Refused("R6", f"{n} is not installed and is not a kernel image")
|
||||
if not args:
|
||||
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
|
||||
self.log("os-apply: DONE rc=0 upgraded=0 (no pending kernel)")
|
||||
return 0
|
||||
rc, sim, remv, text = a.simulate(["install", "--no-install-recommends"] + args)
|
||||
if rc != 0:
|
||||
raise Refused("R7", "the simulation failed: " + (text.strip().splitlines()[-1] if text.strip() else ""))
|
||||
if remv:
|
||||
raise Refused("R4", f"the kernel step would remove {', '.join(remv[:5])}")
|
||||
images = []
|
||||
listed = dict(want)
|
||||
for p in sim:
|
||||
if a.origin_name(p["origin"]) != {PVE_ORIGIN}:
|
||||
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not {PVE_ORIGIN!r}")
|
||||
m = KERNEL_IMAGE_RE.match(p["name"])
|
||||
if p["from"] is None:
|
||||
if not m:
|
||||
raise Refused("R6", f"the kernel step would add {p['name']}, which is not a kernel image")
|
||||
if a.select == "listed" and listed.get(p["name"]) != p["to"]:
|
||||
raise Refused("R23", f"the kernel step would add {p['name']}={p['to']}, not the signed set")
|
||||
images.append((m.group(1), p["to"]))
|
||||
continue
|
||||
if p["name"] not in upg or p["to"] != upg[p["name"]]:
|
||||
raise Refused("R6", f"the kernel step would touch {p['name']} ({p['to']}), which is not in the step")
|
||||
if not a.dpkg_cmp(p["to"], "gt", p["from"]):
|
||||
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
|
||||
if len(images) > 1:
|
||||
raise Refused("R23", f"the kernel step would add {len(images)} kernels — one at a time")
|
||||
# the target kernel: the new image, else the series meta-package's version (its image is already installed)
|
||||
if images:
|
||||
new = images[0][0]
|
||||
if images[0][1] + "-pve" != new:
|
||||
raise Refused("R23", f"the image version {images[0][1]} does not name the kernel {new}")
|
||||
else:
|
||||
metas = [(n, v) for n, v in upg.items() if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)]
|
||||
if len(metas) != 1:
|
||||
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
|
||||
self.log("os-apply: DONE rc=0 upgraded=0 (the pending set names no kernel to boot)")
|
||||
return 0
|
||||
new = metas[0][1] + "-pve"
|
||||
if not KVER_RE.match(new) or not a.dpkg_cmp(new[:-4], "gt", old[:-4]):
|
||||
raise Refused("R23", f"the kernel {new} is not newer than the running {old}")
|
||||
ek = self.plan.get("expect_kver")
|
||||
if ek and ek != new:
|
||||
raise Refused("R23", f"the step would boot {new}, but the household was told about {ek}")
|
||||
need = a.download_bytes(["install", "--no-install-recommends"] + args)
|
||||
free = a.free_bytes()
|
||||
if free >= 0 and free < max(MIN_FREE, 3 * need):
|
||||
raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)")
|
||||
# 1. the default = the kernel the box RUNS (it booted healthily), proved from grub.cfg BEFORE the install
|
||||
default_before = self.default_kver(self.grub_cfg())
|
||||
self.write_default(old)
|
||||
# 2. install (the kernel's own postinst runs update-grub; our default file keeps the default on `old`)
|
||||
t0 = time.time()
|
||||
rc, out, err = a.x(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + ["install", "--no-install-recommends"] + args)
|
||||
a.x(["apt-get", "clean"])
|
||||
if rc != 0:
|
||||
_, aud, _ = a.x(["dpkg", "--audit"])
|
||||
self.report["failed"] = {"rc": rc, "step": "install", "dpkg_audit": (aud.strip().splitlines() or ["clean"])[0],
|
||||
"tail": (out + err).strip().splitlines()[-3:]}
|
||||
self.log(f"os-apply: FAILED rc={rc} step=install (the default stays {old}; no flag written)")
|
||||
return 3
|
||||
self.report["upgraded"] = [{"name": x.split("=", 1)[0], "version": x.split("=", 1)[1]} for x in args]
|
||||
self.report["seconds"] = round(time.time() - t0, 1)
|
||||
# 3. prove: the image is there, the default is still `old`, the one-shot entry for `new` exists
|
||||
cfg = self.grub_cfg()
|
||||
if f"felhom-oneshot-{new}" not in cfg or self.default_kver(cfg) != old:
|
||||
self.r.host(["update-grub"], 300)
|
||||
cfg = self.grub_cfg()
|
||||
for f in (f"/boot/vmlinuz-{new}", f"/boot/initrd.img-{new}"):
|
||||
if not self.r.lexists(f):
|
||||
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"{f} is missing after the install"}
|
||||
return 3
|
||||
if f"felhom-oneshot-{new}" not in cfg or "felhom_next" not in cfg:
|
||||
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"grub.cfg has no one-shot entry for {new}"}
|
||||
return 3
|
||||
if self.default_kver(cfg) != old:
|
||||
self.report["failed"] = {"rc": 3, "step": "verify",
|
||||
"reason": f"the install moved the default to {self.default_kver(cfg)} — no flag written"}
|
||||
return 3
|
||||
# 4. the flag — the ONLY thing that makes the next boot use `new`, and only once
|
||||
self.set_flag(new)
|
||||
self.save_state({"phase": "staged", "step_id": self.plan.get("release_id"), "from": old, "to": new, "vmid": a.vmid,
|
||||
"staged_at": _iso(self.r.now()), "authority": who, "default_before": default_before,
|
||||
"packages": self.report["upgraded"], "self_revert_used": False})
|
||||
self.report["reboot_needed"] = True
|
||||
self.log(f"os-apply: KERNEL STAGED {old} -> {new} (default {old}, one-shot flag {new}); upgraded={len(args)} "
|
||||
f"seconds={self.report['seconds']}")
|
||||
return 0
|
||||
|
||||
def reboot_staged(self):
|
||||
st = self.state()
|
||||
self.phase_is(st, "staged")
|
||||
if self.flag() != st.get("to"):
|
||||
raise Refused("R22", f"the one-shot flag reads {self.flag()!r}, not {st.get('to')!r}")
|
||||
if self.running() != st.get("from"):
|
||||
raise Refused("R22", f"the box runs {self.running()!r}, not the step's old kernel {st.get('from')!r}")
|
||||
cfg = self.grub_cfg()
|
||||
if self.default_kver(cfg) != st["from"] or f"felhom-oneshot-{st['to']}" not in cfg:
|
||||
raise Refused("R20", "grub.cfg no longer keeps the old default with a one-shot entry for the new kernel")
|
||||
if self.setup_problems():
|
||||
raise Refused("R20", "; ".join(self.setup_problems()))
|
||||
self.check_guard()
|
||||
self.a.check_guest(self.a.vmid)
|
||||
st["health_before"] = self.a.health()
|
||||
st["phase"], st["rebooted_at"] = "oneshot", _iso(self.r.now())
|
||||
self.save_state(st)
|
||||
return self.reboot(f"one-shot boot of {st['to']} (the default stays {st['from']})")
|
||||
|
||||
def after_boot(self):
|
||||
st = self.state()
|
||||
ph, run = st.get("phase"), self.running()
|
||||
old, new = st.get("from"), st.get("to")
|
||||
event = "none"
|
||||
if ph in ("oneshot", "staged", "judging") and run == new and new:
|
||||
if ph != "judging":
|
||||
st["phase"], st["judging_since"], event = "judging", _iso(self.r.now()), "judging"
|
||||
else:
|
||||
event = "judging"
|
||||
elif ph in ("oneshot", "judging") and run == old:
|
||||
# the new kernel did not come up, or crashed: GRUB already booted the default (the old kernel)
|
||||
self.clear_flag()
|
||||
st["phase"], st["result_at"], event = "fell_back", _iso(self.r.now()), "fell_back"
|
||||
st["reason"] = "the box came back on the old kernel by itself (the new one did not boot, or crashed)"
|
||||
elif ph == "reverting" and run == old:
|
||||
st["phase"], st["result_at"], event = "self_reverted", _iso(self.r.now()), "self_reverted"
|
||||
elif ph == "reverting" and run == new:
|
||||
st["phase"], st["result_at"], event = "revert_failed", _iso(self.r.now()), "revert_failed"
|
||||
st["reason"] = "the self-revert came back on the NEW kernel — never retried (one self-revert per step)"
|
||||
if event not in ("none",) and st.get("phase") != ph:
|
||||
self.save_state(st)
|
||||
self.log(f"os-apply: KERNEL AFTER-BOOT {ph} -> {st['phase']} running={run} ({old} -> {new})")
|
||||
self.report["kernel_event"] = event
|
||||
if event == "judging":
|
||||
self.report["health_before"] = st.get("health_before") # the agent judges the boot against it
|
||||
return 0
|
||||
|
||||
def good(self):
|
||||
st = self.state()
|
||||
self.phase_is(st, "judging")
|
||||
if self.running() != st.get("to"):
|
||||
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
|
||||
try:
|
||||
self.write_default(st["to"])
|
||||
except Refused:
|
||||
# put the old default back — the box must never be left without a proved default
|
||||
self.write_default(st["from"])
|
||||
raise
|
||||
self.clear_flag()
|
||||
st["phase"], st["result_at"] = "good", _iso(self.r.now())
|
||||
self.save_state(st)
|
||||
self.log(f"os-apply: KERNEL GOOD {st['to']} is the default now (was {st['from']})")
|
||||
return 0
|
||||
|
||||
def revert(self):
|
||||
st = self.state()
|
||||
self.phase_is(st, "judging")
|
||||
if st.get("self_revert_used"):
|
||||
raise Refused("R22", "this kernel step already used its one self-revert")
|
||||
if self.running() != st.get("to"):
|
||||
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
|
||||
self.clear_flag()
|
||||
cfg = self.grub_cfg()
|
||||
if self.default_kver(cfg) != st.get("from"):
|
||||
raise Refused("R20", f"the GRUB default reads {self.default_kver(cfg)}, not the old {st.get('from')} — "
|
||||
f"no self-revert into an unknown kernel")
|
||||
reason = self.plan.get("reason")
|
||||
st["reason"] = reason[:300] if isinstance(reason, str) else "the one-shot boot was not healthy"
|
||||
st["self_revert_used"], st["phase"], st["reverted_at"] = True, "reverting", _iso(self.r.now())
|
||||
self.save_state(st)
|
||||
return self.reboot(f"self-revert to {st['from']}: {st['reason']}")
|
||||
|
||||
def cancel(self):
|
||||
st = self.state()
|
||||
self.phase_is(st, "staged")
|
||||
self.clear_flag()
|
||||
st["phase"], st["result_at"], st["reason"] = "cancelled", _iso(self.r.now()), "cancelled before the reboot"
|
||||
self.save_state(st)
|
||||
self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)")
|
||||
return 0
|
||||
|
||||
|
||||
class Bundle:
|
||||
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
|
||||
signed unit. Every check runs before the first write; a failed write or a failed self-check puts every previous
|
||||
|
||||
@@ -17,6 +17,8 @@ Verbs (each one sudoers line, exact-match pattern):
|
||||
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
|
||||
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
|
||||
sshd-key /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op -> /etc/felhom-sshd/authorized_keys/felhom-op
|
||||
controller-image <vmid> the ref on STDIN -> /etc/felhom-controller-image INSIDE guest <vmid> (R-861 (a) A1): only
|
||||
our registry + our repository + an x.y.z tag; the agent no longer has a `tee` grant
|
||||
--self-check prints "felhom-priv-apply ok verbs=..." (the bundle's self-check)
|
||||
|
||||
Exit codes: 0 installed (or already identical), 2 usage, 3 refused (content or source), 4 install failed.
|
||||
@@ -39,7 +41,14 @@ WG_SRC, WG_DEST = STATE + "/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"
|
||||
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
|
||||
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
|
||||
MAX_BYTES = 64 * 1024
|
||||
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key")
|
||||
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key", "controller-image")
|
||||
# R-861 (a) A1 (`09` §3 decision 165): the SAME pattern as the agent's controllerImageRe (internal/localapi/
|
||||
# controllerswap.go) — a compromised agent cannot hand the guest's bootstrap any other image. Pinned by
|
||||
# configs/test_felhom_priv_apply.py ControllerImage.
|
||||
CONTROLLER_IMAGE_RE = re.compile(r"^gitea\.dooplex\.hu/admin/felhom-controller:[0-9]+\.[0-9]+\.[0-9]+$")
|
||||
CONTROLLER_IMAGE_FILE = "/etc/felhom-controller-image"
|
||||
CONTROLLER_IMAGE_MAX = 256
|
||||
VMID_RE = re.compile(r"^[0-9]{1,9}$")
|
||||
|
||||
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
|
||||
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
|
||||
@@ -123,6 +132,16 @@ class Host:
|
||||
pass
|
||||
raise
|
||||
|
||||
def read_stdin(self, limit):
|
||||
return sys.stdin.buffer.read(limit + 1)
|
||||
|
||||
def write_guest_image(self, vmid, data):
|
||||
"""As root: `pct exec <vmid> -- tee <the fixed file>` with the checked ref on stdin (no shell)."""
|
||||
r = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", CONTROLLER_IMAGE_FILE],
|
||||
input=data, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=60)
|
||||
if r.returncode != 0:
|
||||
raise OSError(f"pct exec {vmid} tee exited {r.returncode}: {r.stderr.decode(errors='replace').strip()[:200]}")
|
||||
|
||||
def log(self, line):
|
||||
print(line, file=sys.stderr)
|
||||
try:
|
||||
@@ -377,6 +396,34 @@ def plan(argv):
|
||||
raise Refused("A1", f"wrong arguments for {v}")
|
||||
|
||||
|
||||
def controller_image(rest, host):
|
||||
"""R-861 (a) A1: read the ref on stdin, check it, write it INSIDE the guest as root."""
|
||||
try:
|
||||
if len(rest) != 1 or not VMID_RE.match(rest[0]):
|
||||
raise Refused("A1", "usage: felhom-priv-apply controller-image <vmid> (the ref on stdin)")
|
||||
raw = host.read_stdin(CONTROLLER_IMAGE_MAX)
|
||||
if len(raw) > CONTROLLER_IMAGE_MAX:
|
||||
raise Refused("I1", f"the image ref is longer than {CONTROLLER_IMAGE_MAX} bytes")
|
||||
try:
|
||||
text = raw.decode("ascii")
|
||||
except UnicodeDecodeError:
|
||||
raise Refused("I1", "the image ref is not ASCII")
|
||||
ref = text[:-1] if text.endswith("\n") else text
|
||||
if not CONTROLLER_IMAGE_RE.match(ref) or "\n" in ref:
|
||||
raise Refused("I1", "the image ref is not gitea.dooplex.hu/admin/felhom-controller:<x.y.z>")
|
||||
except Refused as e:
|
||||
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] controller-image {' '.join(rest)[:40]}: {e.reason}")
|
||||
return 2 if e.rule == "A1" else 3
|
||||
vmid = int(rest[0])
|
||||
try:
|
||||
host.write_guest_image(vmid, (ref + "\n").encode())
|
||||
except (OSError, subprocess.SubprocessError) as e:
|
||||
host.log(f"felhom-priv-apply: FAILED controller-image {vmid}: {e}")
|
||||
return 4
|
||||
host.log(f"felhom-priv-apply: WROTE controller-image {vmid} {ref}")
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv, host=None):
|
||||
host = host or Host()
|
||||
if argv == ["--self-check"]:
|
||||
@@ -396,6 +443,8 @@ def main(argv, host=None):
|
||||
return 3
|
||||
print("OK")
|
||||
return 0
|
||||
if argv and argv[0] == "controller-image":
|
||||
return controller_image(argv[1:], host)
|
||||
try:
|
||||
verb, src, dest, mode, checker = plan(argv)
|
||||
data = host.read_source(src)
|
||||
|
||||
Executable
+99
@@ -0,0 +1,99 @@
|
||||
#!/bin/bash
|
||||
# felhom-vzdump-partial-clean — remove ONE dead, half-written vzdump backup (R-1006). The only delete the agent's
|
||||
# sudoers allows for backup files. Installed as /usr/local/sbin/felhom-vzdump-partial-clean (config bundle).
|
||||
#
|
||||
# felhom-vzdump-partial-clean <storage> <vmid> <name>
|
||||
#
|
||||
# A power cut during the night backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` (+ its `.tmp/` directory) on the
|
||||
# storage; nothing removes it, every later backup is then skipped „not enough space", and a full host root breaks
|
||||
# LVM's own metadata writes (power-cut-fixes-2026-10-14 C4). This helper removes such a leftover — and NOTHING else.
|
||||
# It trusts nothing the agent says; every rule is checked here, as root, and any doubt REFUSES (exit 1):
|
||||
# 1. the arguments: a storage id, a vmid, and a name that is exactly
|
||||
# vzdump-(lxc|qemu)-<THAT vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp) — never a finished archive
|
||||
# (.tar.zst, .vma.zst, …), never a .log/.notes, never another vmid;
|
||||
# 2. the storage: its section in /etc/pve/storage.cfg is `dir: <storage>` with a `path`; the target is
|
||||
# <path>/dump/<name>, not a symlink, and its real directory IS <path>/dump;
|
||||
# 3. not in use: no vzdump runs on the host at all (a task `…:vzdump:…` or a `vzdump` command — a job for several
|
||||
# guests carries no vmid in its task name), and no
|
||||
# process holds the target (or, for the .tmp directory, anything under it) open or as its working directory
|
||||
# (C2: after a killed worker, tar and zstd kept writing into the .tar.dat for 4 min 40 s, none named „vzdump");
|
||||
# 4. dead: last modified before this boot (`btime` in /proc/stat), or untouched for at least QUIET_S seconds.
|
||||
# Prints one line on success: „removed <name> <bytes> B (<why>)". Pinned by configs/test_felhom_vzdump_partial_clean.py.
|
||||
set -u
|
||||
STORAGE_CFG=/etc/pve/storage.cfg
|
||||
PROC=/proc
|
||||
QUIET_S=120
|
||||
|
||||
die() { echo "felhom-vzdump-partial-clean: REFUSED: $*" >&2; exit 1; }
|
||||
|
||||
storage_path() { # $1 storage id → prints the dir storage's path, or nothing
|
||||
awk -v want="$1" '
|
||||
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (id == want); next }
|
||||
in_s && /^[ \t]+path[ \t]/ { if (type == "dir") { sub(/^[ \t]+path[ \t]+/, ""); sub(/[ \t]+$/, ""); print; exit } }
|
||||
' "$STORAGE_CFG"
|
||||
}
|
||||
|
||||
storage_type() {
|
||||
awk -v want="$1" '/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); id = h[2]; sub(/[ \t]+$/, "", id); if (id == want) { print h[1]; exit } }' "$STORAGE_CFG"
|
||||
}
|
||||
|
||||
boot_time() { awk '$1 == "btime" { print $2 }' "$PROC/stat"; }
|
||||
|
||||
newest_mtime() { # a file's mtime, or the newest mtime of anything in a directory (the dir itself included)
|
||||
if [[ -d "$1" ]]; then find "$1" -xdev -printf '%T@\n' 2>/dev/null | sort -n | tail -1 | cut -d. -f1
|
||||
else stat -c %Y "$1"; fi
|
||||
}
|
||||
|
||||
busy_reason() { # $1 vmid, $2 target → prints why it is in use, or nothing
|
||||
local vmid=$1 target=$2 p link cmd
|
||||
for p in "$PROC"/[0-9]*; do
|
||||
[[ -r "$p/cmdline" ]] || continue
|
||||
cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)
|
||||
# ANY vzdump on the host refuses: a job for several guests runs as `task UPID:…:vzdump::…` (no vmid in it), and
|
||||
# its tar may be between this guest's files. Waiting costs one backup run; deleting a live one costs the backup.
|
||||
if [[ "$cmd" == *":vzdump:"* ]] || [[ "$cmd" =~ (^|/)vzdump([[:space:]]|$) ]]; then
|
||||
echo "a vzdump runs on this host (pid ${p##*/})"; return
|
||||
fi
|
||||
for link in "$p/cwd" "$p"/fd/*; do
|
||||
local to
|
||||
to=$(readlink "$link" 2>/dev/null) || continue
|
||||
if [[ "$to" == "$target" || "$to" == "$target/"* ]]; then
|
||||
echo "pid ${p##*/} has it open"; return
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
local storage=${1:-} vmid=${2:-} name=${3:-}
|
||||
[[ $# -eq 3 ]] || die "usage: felhom-vzdump-partial-clean <storage> <vmid> <name>"
|
||||
[[ "$storage" =~ ^[A-Za-z][A-Za-z0-9_-]{0,63}$ ]] || die "bad storage id"
|
||||
[[ "$vmid" =~ ^[1-9][0-9]{2,8}$ ]] || die "bad vmid"
|
||||
[[ "$name" =~ ^vzdump-(lxc|qemu)-${vmid}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$ ]] \
|
||||
|| die "name is not a half-written vzdump of guest $vmid: $name"
|
||||
[[ "$(storage_type "$storage")" == "dir" ]] || die "storage $storage is not a dir storage"
|
||||
local path; path=$(storage_path "$storage")
|
||||
[[ "$path" == /* ]] || die "storage $storage has no path"
|
||||
local dump="$path/dump" target="$path/dump/$name"
|
||||
[[ -e "$target" ]] || die "no such entry: $target"
|
||||
[[ ! -L "$target" ]] || die "a symlink: $target"
|
||||
[[ "$(realpath -e "$dump")" == "$(realpath -e "$(dirname "$target")")" ]] || die "not directly in $dump"
|
||||
if [[ "$name" == *.tmp ]]; then [[ -d "$target" ]] || die ".tmp is not a directory"
|
||||
else [[ -f "$target" ]] || die "not a regular file"; fi
|
||||
local busy; busy=$(busy_reason "$vmid" "$target")
|
||||
[[ -z "$busy" ]] || die "in use: $busy"
|
||||
local bt mt now why
|
||||
bt=$(boot_time); mt=$(newest_mtime "$target"); now=$(date +%s)
|
||||
[[ "$bt" =~ ^[0-9]+$ && "$mt" =~ ^[0-9]+$ ]] || die "cannot read the boot time or the modification time"
|
||||
if (( mt < bt )); then why="written before this boot"
|
||||
elif (( now - mt >= QUIET_S )); then why="untouched for $(( now - mt )) s, no vzdump for $vmid, nothing holds it"
|
||||
else die "modified $(( now - mt )) s ago — may still be written"; fi
|
||||
local size; size=$(du -sb "$target" 2>/dev/null | cut -f1)
|
||||
if [[ -d "$target" ]]; then rm -rf --one-file-system -- "$target"; else rm -f -- "$target"; fi
|
||||
[[ ! -e "$target" ]] || die "remove failed: $target"
|
||||
echo "removed $name ${size:-?} B ($why)"
|
||||
}
|
||||
|
||||
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
@@ -546,7 +546,10 @@ class Builder(unittest.TestCase):
|
||||
r"/etc/felhom/[a-z.-]+)", text))
|
||||
agent_writes = {"/usr/local/sbin/felhom-shared-parent", "/etc/systemd/system/felhom-shared-parent.service"}
|
||||
trust = {osapply.TRUST_FILE, osapply.TRUST_SIGNERS, osapply.TRUST_SIGNERS + ".tmp", osapply.BUNDLE_RECORD}
|
||||
missing = sorted(p for p in found if p not in osapply.BUNDLE_DESTS and p not in agent_writes | trust)
|
||||
# Written by the appliance ISO's first boot (felhom.eu scripts/iso/felhom-bootstrap.sh), never by the installer:
|
||||
# since installer 1.32.0 (R-275) the uninstall only NAMES them under KEPT.
|
||||
iso_writes = {"/etc/felhom/.bootstrap-done", "/etc/felhom/appliance-pairing-code"}
|
||||
missing = sorted(p for p in found if p not in osapply.BUNDLE_DESTS and p not in agent_writes | trust | iso_writes)
|
||||
self.assertEqual(missing, [], "the installer writes these root files, but the bundle does not carry them")
|
||||
# the limits drop-in is named through $AGENT_UNIT in the installer
|
||||
self.assertIn("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", osapply.BUNDLE_DESTS)
|
||||
|
||||
@@ -139,5 +139,62 @@ class Guard(unittest.TestCase):
|
||||
self.assertEqual(mode, 0o644)
|
||||
|
||||
|
||||
class KernelStepCannotLeaveTheBoxOff(unittest.TestCase):
|
||||
"""R-836 / `11` §5.11 Part B 4: a kernel step's planned reboot, one crash and one self-revert cannot add up to the
|
||||
box staying off. The wrapper starts a step only when the guard is armed with NO unclean boot in its window
|
||||
(felhom-os-apply Kernel.check_guard, R21); the planned reboot and the self-revert are orderly (`systemctl reboot` —
|
||||
the clean-stop marker); so the step adds at most ONE unclean boot, and the box stays off only after the LIMIT-th
|
||||
(3rd) within the hour. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
|
||||
|
||||
def setUp(self):
|
||||
self.d = tempfile.TemporaryDirectory()
|
||||
self.e = FakeEnv(self.d.name)
|
||||
cg.main(["x", "boot"], self.e)
|
||||
s = self.e.state()
|
||||
self.assertTrue(s["armed"])
|
||||
self.assertEqual(s["unclean_boots_in_window"], 0, "the wrapper's precondition (R21)")
|
||||
|
||||
def tearDown(self):
|
||||
self.d.cleanup()
|
||||
|
||||
def planned(self, minutes):
|
||||
cg.main(["x", "clean-stop"], self.e)
|
||||
self.e.t += minutes * 60
|
||||
cg.main(["x", "boot"], self.e)
|
||||
|
||||
def crash(self, minutes):
|
||||
self.e.t += minutes * 60
|
||||
cg.main(["x", "boot"], self.e)
|
||||
|
||||
def test_planned_reboot_one_crash_one_self_revert(self):
|
||||
self.planned(2) # the step's one-shot reboot (orderly)
|
||||
self.crash(3) # the new kernel crashes after the guard ran; the box restarts (panic=10)
|
||||
self.planned(2) # the self-revert (orderly)
|
||||
s = self.e.state()
|
||||
self.assertFalse(s["tripped"], s)
|
||||
self.assertTrue(s["armed"])
|
||||
self.assertEqual(self.e.panic(), 10, "the box still restarts after a crash")
|
||||
self.assertEqual(s["unclean_boots_in_window"], 1, "the step added exactly one unclean boot")
|
||||
|
||||
def test_a_panic_before_userspace_is_not_even_counted(self):
|
||||
# the one-shot kernel panics before the guard's unit runs (measured: rdinit= and init= missing): the planned
|
||||
# reboot's clean-stop marker is still there when the old kernel boots, so this boot counts as clean.
|
||||
cg.main(["x", "clean-stop"], self.e)
|
||||
self.e.t += 120 # the panicking boot: no userspace, the guard never ran
|
||||
cg.main(["x", "boot"], self.e)
|
||||
s = self.e.state()
|
||||
self.assertEqual(s["unclean_boots_in_window"], 0, s)
|
||||
self.assertEqual(self.e.panic(), 10)
|
||||
|
||||
def test_the_box_stays_off_only_after_two_more_crashes_than_the_step_makes(self):
|
||||
self.planned(2)
|
||||
self.crash(3) # the step's one crash
|
||||
self.planned(2) # the self-revert
|
||||
self.crash(5) # an UNRELATED crash within the hour: the guard trips (the 3rd would leave it off)
|
||||
s = self.e.state()
|
||||
self.assertTrue(s["tripped"])
|
||||
self.assertEqual(s["unclean_boots_in_window"], 2, "two unclean boots: one from the step, one not")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,255 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for configs/felhom-internal-place (felhom.eu architecture/16 §5, §7 — sizes ruled 2026-10-16).
|
||||
|
||||
The helper is run for real under bash; `lvs`, `lvcreate`, `blkid`, `mkfs.ext4` and `stat` are PATH stubs that read and
|
||||
write one JSON state file, so nothing on this machine is touched. Every refusal has a positive control: the same setup
|
||||
without the one condition goes through — so a test cannot pass because the helper refuses everything.
|
||||
Run: python3 configs/test_felhom_internal_place.py (also run by the Go suite, internal/internalplace)
|
||||
"""
|
||||
import json, os, pathlib, stat, subprocess, tempfile, unittest
|
||||
|
||||
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-internal-place"
|
||||
GIB = 1 << 30
|
||||
|
||||
STUB_LVS = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
st = json.load(open(os.environ["IP_STATE"]))
|
||||
if st.get("lvs_fail_listing") and "lv_name,pool_lv,lv_size" in sys.argv: sys.exit(5)
|
||||
args = [a for a in sys.argv[1:] if not a.startswith("-") and a not in ("b",)]
|
||||
fields = sys.argv[sys.argv.index("-o") + 1].split(",")
|
||||
target = args[-1]
|
||||
listing = target == st["vg"]
|
||||
for lv in st["lvs"]:
|
||||
full = st["vg"] + "/" + lv["name"]
|
||||
if listing and st.get("hide_pool_in_listing") and lv.get("pool", "") == "":
|
||||
continue # a listing that silently lost the pool row (the pool's own size stays readable)
|
||||
if listing or target == full:
|
||||
row = {"lv_name": lv["name"], "pool_lv": lv.get("pool", ""), "lv_size": str(lv["size"])}
|
||||
print(" " + " ".join(row[f] for f in fields))
|
||||
'''
|
||||
STUB_LVCREATE = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
p = os.environ["IP_STATE"]; st = json.load(open(p))
|
||||
a = sys.argv[1:]
|
||||
size = int(a[a.index("-V") + 1].rstrip("G")) * (1 << 30)
|
||||
vg, pool = a[a.index("-T") + 1].split("/")
|
||||
st["lvs"].append({"name": a[a.index("-n") + 1], "pool": pool, "size": size, "active": True})
|
||||
st.setdefault("calls", []).append("lvcreate " + " ".join(a))
|
||||
if st.get("dirty_new"): st["fs"]["felhom-belso"] = {"TYPE": "", "RAW": "junk"}
|
||||
json.dump(st, open(p, "w"))
|
||||
'''
|
||||
STUB_BLKID = r'''#!/usr/bin/env python3
|
||||
# like blkid -p: exit 0 when something is found, 2 when nothing is; st["blkid_rc"] forces an error exit
|
||||
import json, os, sys
|
||||
st = json.load(open(os.environ["IP_STATE"]))
|
||||
if st.get("blkid_rc"): sys.exit(st["blkid_rc"])
|
||||
fs = st["fs"].get(os.path.basename(sys.argv[-1]), {})
|
||||
if "-s" in sys.argv:
|
||||
v = fs.get(sys.argv[sys.argv.index("-s") + 1], "")
|
||||
if v: print(v)
|
||||
sys.exit(0 if v else 2)
|
||||
vals = [fs.get(k, "") for k in ("TYPE", "LABEL", "UUID", "RAW") if fs.get(k)]
|
||||
if vals: print("\n".join(vals))
|
||||
sys.exit(0 if vals else 2)
|
||||
'''
|
||||
STUB_MKFS = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
p = os.environ["IP_STATE"]; st = json.load(open(p))
|
||||
dev = os.path.basename(sys.argv[-1])
|
||||
st["fs"][dev] = {"TYPE": "ext4", "LABEL": sys.argv[sys.argv.index("-L") + 1], "UUID": "0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c"}
|
||||
st.setdefault("calls", []).append("mkfs.ext4 " + " ".join(sys.argv[1:]))
|
||||
json.dump(st, open(p, "w"))
|
||||
'''
|
||||
STUB_STAT = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
st = json.load(open(os.environ["IP_STATE"]))
|
||||
name = os.path.basename(sys.argv[-1])
|
||||
if any(lv["name"] == name and lv.get("active", True) for lv in st["lvs"]): print("block special file")
|
||||
else: sys.exit(1)
|
||||
'''
|
||||
|
||||
CFG = "dir: local\n\tpath /var/lib/vz\n\tcontent iso,backup\n\nlvmthin: local-lvm\n\tthinpool data\n\tvgname pve\n\tcontent rootdir,images\n"
|
||||
|
||||
|
||||
class Box:
|
||||
def __init__(self, pool_gib, lvs=(), fs=None, cfg=CFG, **extra):
|
||||
self.t = tempfile.TemporaryDirectory()
|
||||
r = pathlib.Path(self.t.name)
|
||||
self.bin = r / "bin"; self.bin.mkdir()
|
||||
for name, body in (("lvs", STUB_LVS), ("lvcreate", STUB_LVCREATE), ("blkid", STUB_BLKID),
|
||||
("mkfs.ext4", STUB_MKFS), ("stat", STUB_STAT)):
|
||||
f = self.bin / name; f.write_text(body); f.chmod(0o755)
|
||||
self.cfg = r / "storage.cfg"; self.cfg.write_text(cfg)
|
||||
self.state = r / "state.json"
|
||||
st = {"vg": "pve", "lvs": [{"name": "data", "pool": "", "size": pool_gib * GIB}] + [dict(l) for l in lvs],
|
||||
"fs": fs or {}}
|
||||
st.update(extra)
|
||||
self.state.write_text(json.dumps(st))
|
||||
|
||||
def run(self, *args):
|
||||
env = {"PATH": f"{self.bin}:/usr/bin:/bin", "IP_STATE": str(self.state), "FELHOM_STORAGE_CFG": str(self.cfg)}
|
||||
p = subprocess.run(["bash", str(HELPER), *args], env=env, capture_output=True, text=True)
|
||||
return p.returncode, p.stdout.strip(), p.stderr.strip()
|
||||
|
||||
def st(self):
|
||||
return json.loads(self.state.read_text())
|
||||
|
||||
def calls(self):
|
||||
return self.st().get("calls", [])
|
||||
|
||||
def close(self):
|
||||
self.t.cleanup()
|
||||
|
||||
|
||||
def thin(name, gib, pool="data"):
|
||||
return {"name": name, "pool": pool, "size": gib * GIB}
|
||||
|
||||
|
||||
class Sizes(unittest.TestCase):
|
||||
"""The operator's table (SPACE-CHECK §4, ruled 2026-10-16), as the helper computes it on a fresh box."""
|
||||
|
||||
def plan(self, pool, planned, lvs=()):
|
||||
b = Box(pool, lvs)
|
||||
try:
|
||||
rc, out, err = b.run("plan", str(planned))
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertEqual(b.calls(), [], "plan must change nothing")
|
||||
return dict(kv.split("=") for kv in out.split()[1:])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_the_ruled_table(self):
|
||||
# disk → pool (GiB), data volume (planned = 32 + mp0), expected internal
|
||||
for pool, mp0, reserve, internal in ((140, 50, 16, 42), (349, 100, 34, 183), (800, 130, 48, 590), (1720, 130, 48, 1510)):
|
||||
got = self.plan(pool, 32 + mp0)
|
||||
self.assertEqual((int(got["reserve"]), int(got["internal"])), (reserve, internal), (pool, got))
|
||||
|
||||
def test_promised_volumes_count_but_vzdump_snapshots_and_other_pools_do_not(self):
|
||||
got = self.plan(349, 0, [thin("vm-9201-disk-0", 32), thin("vm-9201-disk-1", 250),
|
||||
thin("snap_vm-9201-disk-1_vzdump", 250), thin("vm-500-disk-0", 40, pool="other")])
|
||||
self.assertEqual((int(got["promised"]), int(got["internal"])), (282, 33), got)
|
||||
|
||||
|
||||
class Create(unittest.TestCase):
|
||||
def test_a_fresh_box_gets_the_place(self):
|
||||
b = Box(349)
|
||||
try:
|
||||
rc, out, err = b.run("create", "132")
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertRegex(out, r"^created uuid=[0-9a-f-]{36} gib=183$")
|
||||
self.assertEqual(len([c for c in b.calls() if c.startswith("lvcreate")]), 1)
|
||||
self.assertIn("-V 183G -T pve/data -n felhom-belso", b.calls()[0])
|
||||
self.assertIn("-L felhom-belso", b.calls()[1])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_too_little_space_makes_nothing(self):
|
||||
b = Box(53) # demo-hp's 128 GB disk
|
||||
try:
|
||||
rc, out, err = b.run("create", str(32 + 50))
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertTrue(out.startswith("skipped "), out)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_an_existing_place_is_found_never_formatted(self):
|
||||
fs = {"felhom-belso": {"TYPE": "ext4", "LABEL": "felhom-belso", "UUID": "11111111-2222-3333-4444-555555555555"}}
|
||||
b = Box(349, [thin("felhom-belso", 183)], fs)
|
||||
try:
|
||||
rc, out, err = b.run("create", "0")
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertEqual(out, "existing uuid=11111111-2222-3333-4444-555555555555 gib=183")
|
||||
self.assertEqual(b.calls(), [], "an existing place must never be created or formatted again")
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_foreign_filesystem_on_the_name_is_refused(self):
|
||||
# positive control: test_an_existing_place_is_found_never_formatted (same LV, Felhom's label) goes through
|
||||
for fs in ({"TYPE": "xfs", "LABEL": "felhom-belso", "UUID": "x"}, {"TYPE": "ext4", "LABEL": "films", "UUID": "y"}):
|
||||
b = Box(349, [thin("felhom-belso", 183)], {"felhom-belso": fs})
|
||||
try:
|
||||
rc, out, err = b.run("create", "0")
|
||||
self.assertEqual(rc, 1, (fs, out))
|
||||
self.assertIn("REFUSED", err)
|
||||
self.assertEqual(b.calls(), [], fs)
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_crash_between_lvcreate_and_mkfs_is_finished(self):
|
||||
b = Box(349, [thin("felhom-belso", 183)], {})
|
||||
try:
|
||||
rc, out, err = b.run("create", "0")
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertRegex(out, r"^formatted uuid=")
|
||||
self.assertEqual([c.split()[0] for c in b.calls()], ["mkfs.ext4"])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_an_unnamed_signature_is_refused(self):
|
||||
b = Box(349, [thin("felhom-belso", 183)], {"felhom-belso": {"RAW": "something"}})
|
||||
try:
|
||||
rc, out, err = b.run("create", "0")
|
||||
self.assertEqual(rc, 1, out)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_new_volume_that_is_not_empty_is_not_formatted(self):
|
||||
b = Box(349, dirty_new=True)
|
||||
try:
|
||||
rc, out, err = b.run("create", "132")
|
||||
self.assertEqual(rc, 1, out)
|
||||
self.assertNotIn("mkfs", " ".join(b.calls()))
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_bad_arguments_and_no_local_lvm_are_refused(self):
|
||||
b = Box(349, cfg="dir: local\n\tpath /var/lib/vz\n")
|
||||
try:
|
||||
self.assertEqual(b.run("create", "0")[0], 1)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
b = Box(349)
|
||||
try:
|
||||
for args in (("create", "-1"), ("create", "1x"), ("wipe", "0"), ("create",)):
|
||||
self.assertEqual(b.run(*args)[0], 1, args)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
|
||||
class FailsClosed(unittest.TestCase):
|
||||
"""A read that fails must never read as „empty" or „nothing promised". Positive control: Create (same setups)."""
|
||||
|
||||
def test_a_failed_volume_listing_creates_nothing(self):
|
||||
b = Box(349, lvs_fail_listing=True)
|
||||
try:
|
||||
rc, out, err = b.run("create", "132")
|
||||
self.assertEqual(rc, 1, out)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_listing_without_the_pool_creates_nothing(self):
|
||||
b = Box(349, [thin("vm-9201-disk-1", 250)], hide_pool_in_listing=True)
|
||||
try:
|
||||
self.assertEqual(b.run("create", "132")[0], 1)
|
||||
self.assertEqual(b.calls(), [])
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_a_blkid_error_never_reads_as_empty(self):
|
||||
for setup in ({"lvs": [thin("felhom-belso", 183)], "fs": {}}, {}):
|
||||
b = Box(349, **({"lvs": setup["lvs"], "fs": setup["fs"]} if setup else {}), blkid_rc=4)
|
||||
try:
|
||||
rc, out, err = b.run("create", "0" if setup else "132")
|
||||
self.assertEqual(rc, 1, (setup, out))
|
||||
self.assertNotIn("mkfs", " ".join(b.calls()), setup)
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for configs/felhom-mkfs-guarded.sh — the ONLY mkfs path the agent's sudoers permits.
|
||||
|
||||
First tests of this wrapper (agent v0.158.0), written for partitions as places (felhom.eu architecture/16 §6). Every
|
||||
mount of every node of the disk outside /mnt/felhom-drives refuses — INCLUDING a sibling that is Felhom's own drive
|
||||
(its raw /mnt/<name> mount): the agent can make that corroborating mount itself, so the root wrapper must not trust
|
||||
it (security review of 912f7c1). An unlistable disk refuses.
|
||||
|
||||
HOW IT RUNS ANYWHERE: the wrapper is copied with five literal host paths replaced (each replacement is asserted, so a
|
||||
drifted wrapper fails here instead of silently testing nothing): /proc/mounts → a fake mount table, the block-device
|
||||
test → true (no real /dev nodes here), the pvs candidates → a stub, /sys/block → a fake tree, the mkfs binaries → a
|
||||
recorder. `lsblk` is a PATH stub answering the three calls the wrapper makes from one JSON disk description.
|
||||
Every refusal has a positive control (the same disk without the one condition is formatted).
|
||||
Run: python3 configs/test_felhom_mkfs_guarded.py (also run by the Go suite, internal/storage)
|
||||
"""
|
||||
import json, os, pathlib, subprocess, tempfile, unittest
|
||||
|
||||
WRAPPER = pathlib.Path(__file__).resolve().parent / "felhom-mkfs-guarded.sh"
|
||||
|
||||
STUB_LSBLK = r'''#!/usr/bin/env python3
|
||||
import json, os, sys
|
||||
st = json.load(open(os.environ["MG_STATE"]))
|
||||
a = sys.argv[1:]; dev = a[-1]
|
||||
nodes = st["nodes"] # [{"name": "/dev/sdb", "pk": "", "fstype": ""}, ...]
|
||||
if a[0] == "-ndo" and a[1] == "PKNAME":
|
||||
for n in nodes:
|
||||
if n["name"] == dev and n.get("pk"): print(n["pk"])
|
||||
elif a[0] == "-nrpo" and a[1] == "NAME":
|
||||
if not st.get("lsblk_empty"):
|
||||
for n in nodes: print(n["name"])
|
||||
elif a[0] == "-nro" and a[1] == "FSTYPE":
|
||||
for n in nodes: print(n.get("fstype", ""))
|
||||
elif a[0] == "-nro" and a[1] == "MOUNTPOINT": # the pre-v0.158.0 wrapper's call: one mountpoint per node, like lsblk
|
||||
mt = [l.split()[:2] for l in open(os.environ["MG_MOUNTS"]) if l.strip()]
|
||||
for n in nodes:
|
||||
mps = [m for d, m in mt if d == n["name"]]
|
||||
print(mps[0] if mps else "")
|
||||
'''
|
||||
STUB_MKFS = '#!/bin/sh\necho "$0 $*" >> "$MG_CALLS"\n'
|
||||
STUB_PVS = '#!/bin/sh\nprintf "%s\\n" $MG_PVS\n'
|
||||
|
||||
|
||||
class Disk:
|
||||
def __init__(self, nodes, mounts, ro="0", pvs="", lsblk_empty=False):
|
||||
self.t = tempfile.TemporaryDirectory()
|
||||
r = pathlib.Path(self.t.name)
|
||||
self.bin = r / "bin"; self.bin.mkdir()
|
||||
(self.bin / "lsblk").write_text(STUB_LSBLK); (self.bin / "lsblk").chmod(0o755)
|
||||
for n in ("mkfs.ext4", "mkfs.xfs"):
|
||||
(self.bin / n).write_text(STUB_MKFS); (self.bin / n).chmod(0o755)
|
||||
(self.bin / "pvs").write_text(STUB_PVS); (self.bin / "pvs").chmod(0o755)
|
||||
self.state = r / "state.json"
|
||||
self.state.write_text(json.dumps({"nodes": nodes, "lsblk_empty": lsblk_empty}))
|
||||
self.mounts = r / "mounts"
|
||||
self.mounts.write_text("".join(f"{d} {m} ext4 rw 0 0\n" for d, m in mounts))
|
||||
self.sys = r / "sys"
|
||||
for n in nodes:
|
||||
if not n.get("pk"):
|
||||
d = self.sys / os.path.basename(n["name"]); d.mkdir(parents=True); (d / "ro").write_text(ro + "\n")
|
||||
self.calls = r / "calls"
|
||||
self.pvs = pvs
|
||||
src = WRAPPER.read_text()
|
||||
subs = [
|
||||
("/proc/mounts", str(self.mounts)),
|
||||
('[[ -b "$dev" ]]', "true"),
|
||||
("for c in /usr/sbin/pvs /sbin/pvs; do", f"for c in {self.bin}/pvs; do"),
|
||||
('rof="/sys/block/', f'rof="{self.sys}/'),
|
||||
("exec /usr/sbin/mkfs.ext4", f"exec {self.bin}/mkfs.ext4"),
|
||||
("exec /usr/sbin/mkfs.xfs ", f"exec {self.bin}/mkfs.xfs "),
|
||||
]
|
||||
for a, b in subs:
|
||||
assert a in src, f"the wrapper no longer contains {a!r} — update this test's substitutions"
|
||||
src = src.replace(a, b)
|
||||
self.copy = r / "mkfs-guarded"; self.copy.write_text(src)
|
||||
|
||||
def run(self, dev, fstype="ext4"):
|
||||
env = {"PATH": f"{self.bin}:/usr/bin:/bin", "MG_STATE": str(self.state), "MG_CALLS": str(self.calls),
|
||||
"MG_MOUNTS": str(self.mounts),
|
||||
"MG_PVS": self.pvs}
|
||||
p = subprocess.run(["bash", str(self.copy), dev, fstype], env=env, capture_output=True, text=True)
|
||||
made = self.calls.read_text().split("\n")[0] if self.calls.exists() else ""
|
||||
return p.returncode, p.stderr.strip(), made
|
||||
|
||||
def close(self):
|
||||
self.t.cleanup()
|
||||
|
||||
|
||||
def disk_sdb(sdb1_fstype="ext4", sdb2_fstype=""):
|
||||
return [{"name": "/dev/sdb", "pk": "", "fstype": ""},
|
||||
{"name": "/dev/sdb1", "pk": "sdb", "fstype": sdb1_fstype},
|
||||
{"name": "/dev/sdb2", "pk": "sdb", "fstype": sdb2_fstype}]
|
||||
|
||||
|
||||
ROOT = ("/dev/sda2", "/")
|
||||
OWN_SDB1 = [("/dev/sdb1", "/mnt/hdd_1"), ("/dev/sdb1", "/mnt/felhom-drives/hdd_1")]
|
||||
|
||||
|
||||
class Partitions(unittest.TestCase):
|
||||
def check(self, mounts, dev="/dev/sdb2", nodes=None, want_ok=True, **kw):
|
||||
d = Disk(nodes or disk_sdb(), [ROOT] + mounts, **kw)
|
||||
try:
|
||||
rc, err, made = d.run(dev)
|
||||
if want_ok:
|
||||
self.assertEqual(rc, 0, err)
|
||||
self.assertTrue(made.endswith(f"mkfs.ext4 -F {dev}"), made)
|
||||
else:
|
||||
self.assertEqual(rc, 1, (mounts, made))
|
||||
self.assertIn("REFUSED", err)
|
||||
self.assertEqual(made, "", "a refused format must not run mkfs")
|
||||
return err
|
||||
finally:
|
||||
d.close()
|
||||
|
||||
def test_a_free_part_on_an_unmounted_disk_is_formatted(self):
|
||||
self.check([]) # positive control for every refusal below
|
||||
|
||||
def test_the_free_part_next_to_felhoms_own_drive_is_refused(self):
|
||||
# The agent could make the „corroborating" managed mount itself: the root wrapper does not trust it.
|
||||
self.check(OWN_SDB1, want_ok=False)
|
||||
|
||||
def test_a_sibling_mounted_raw_only_is_refused(self):
|
||||
self.check([("/dev/sdb1", "/mnt/hdd_1")], want_ok=False)
|
||||
|
||||
def test_a_sibling_mounted_elsewhere_is_refused_even_with_a_managed_mount(self):
|
||||
for elsewhere in ("/srv/data", "/media/usb", "/mnt/a/b", "/home/x"):
|
||||
self.check([("/dev/sdb1", elsewhere), ("/dev/sdb1", "/mnt/felhom-drives/hdd_1")], want_ok=False)
|
||||
|
||||
def test_the_target_itself_mounted_raw_is_refused(self):
|
||||
self.check([("/dev/sdb2", "/mnt/films"), ("/dev/sdb2", "/mnt/felhom-drives/films")], want_ok=False)
|
||||
|
||||
def test_the_target_mounted_only_under_felhom_drives_is_formatted(self):
|
||||
self.check([("/dev/sdb2", "/mnt/felhom-drives/films")])
|
||||
|
||||
def test_an_unlistable_disk_is_refused(self):
|
||||
self.check([], want_ok=False, lsblk_empty=True)
|
||||
|
||||
def test_the_catastrophic_cases_still_refuse(self):
|
||||
self.check([("/dev/sdb1", "/")], want_ok=False) # system disk
|
||||
self.check([], nodes=disk_sdb(sdb1_fstype="LVM2_member"), want_ok=False) # a member signature
|
||||
self.check([], pvs="/dev/sdb1", want_ok=False) # an LVM PV (pvs)
|
||||
self.check([], ro="1", want_ok=False) # read-only
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -83,7 +83,7 @@ class Fake:
|
||||
return self.clock
|
||||
|
||||
def sleep(self, s):
|
||||
pass
|
||||
self.sleeps = getattr(self, "sleeps", []) + [(len(self.calls), s)] # (calls made before it, seconds)
|
||||
|
||||
def verify_sig(self, signers, key_id, ns, blob, sig):
|
||||
self.verified = (signers, key_id, ns, blob, sig)
|
||||
@@ -196,6 +196,27 @@ class Fake:
|
||||
return 0, self.engine + "\n", ""
|
||||
if cmd == "docker" and a[1:3] == ["ps", "-q"]:
|
||||
return 0, "".join(i + "\n" for i in self.ids), ""
|
||||
# R-528: the memory-kill check's engine (oom_image None = unreadable; oom_state / oom_event the engine's answer)
|
||||
if cmd == "date" and a[1:] == ["+%s"]:
|
||||
# each read is 3 s later than the last, so the order of the reads is visible in the values
|
||||
self.oom_epochs = getattr(self, "oom_epochs", []) + [int(self.clock) + 3 * len(getattr(self, "oom_epochs", []))]
|
||||
return 0, f"{self.oom_epochs[-1]}\n", ""
|
||||
if cmd == "docker" and a[1:4] == ["inspect", "-f", "{{.Config.Image}}"]:
|
||||
img = getattr(self, "oom_image", "gitea.dooplex.hu/admin/felhom-controller:0.300.0")
|
||||
return (0, img + "\n", "") if img is not None else (1, "", "Error: No such object: felhom-controller")
|
||||
if cmd == "docker" and a[1] == "run":
|
||||
self.oom_runs = getattr(self, "oom_runs", []) + [a]
|
||||
return 137, "", ""
|
||||
if cmd == "docker" and a[1:4] == ["inspect", "-f", "{{.State.OOMKilled}} {{.State.ExitCode}}"]:
|
||||
if getattr(self, "oom_inspect_raises", False):
|
||||
raise subprocess.TimeoutExpired(a, 10)
|
||||
return 0, getattr(self, "oom_state", "true 137") + "\n", ""
|
||||
if cmd == "docker" and a[1] == "events":
|
||||
self.oom_events_argv = a
|
||||
return 0, ("oom\n" if getattr(self, "oom_event", True) else ""), ""
|
||||
if cmd == "docker" and a[1:3] == ["rm", "-f"]:
|
||||
self.oom_removed = getattr(self, "oom_removed", []) + a[3:]
|
||||
return 0, a[3] + "\n", ""
|
||||
if cmd == "docker" and a[1] == "inspect":
|
||||
mounts = {"aaa111": "/felhom-controller|/var/run/docker.sock;/app/data;", "bbb222": "/app|/data;"}
|
||||
return 0, mounts.get(a[-1], "/other|;") + "\n", ""
|
||||
@@ -211,6 +232,8 @@ class Fake:
|
||||
return (0, self.daemon_json, "") if self.daemon_json is not None else (1, "", "No such file")
|
||||
if cmd == "cat" and a[1] == "/etc/debian_version":
|
||||
return 0, "13.7\n", ""
|
||||
if cmd == "pveversion":
|
||||
return 0, f"pve-manager/{self.installed.get('pve-manager', '9.2.2')}/abcdef (running kernel: 7.0.14-20-pve)\n", ""
|
||||
if cmd == "uname":
|
||||
return 0, "7.0.14-20-pve\n", ""
|
||||
if cmd == "apt-mark":
|
||||
@@ -263,7 +286,7 @@ class Fake:
|
||||
n, v = x.split("=", 1)
|
||||
if v not in self.avail(n):
|
||||
return 100, "", f"E: Version '{v}' for '{n}' was not found"
|
||||
origin = "Docker CE:trixie" if n in osapply.DOCKER_NAMES else SEC if n == "openssl" else DEB
|
||||
origin = getattr(self, "origins", {}).get(n) or ("Docker CE:trixie" if n in osapply.DOCKER_NAMES else SEC if n == "openssl" else DEB)
|
||||
out += f"Inst {n} [{self.installed[n]}] ({v} {origin} [amd64])\n"
|
||||
out += "".join(l + "\n" for l in self.extra_sim)
|
||||
return 0, out, ""
|
||||
@@ -997,8 +1020,6 @@ class RealSignatureCheck(unittest.TestCase):
|
||||
self.assertNotEqual(r.verify_sig(self.signers, "someone-else", "felhom-op-v1", blob, sig), 0)
|
||||
self.assertNotEqual(r.verify_sig(self.signers, "felhom-op-1", "felhom-op-v1", blob, self.sign(blob, ns="other-ns")), 0)
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
class UnsentReport(unittest.TestCase):
|
||||
@@ -1179,3 +1200,810 @@ class CrashLeftTheJournal(unittest.TestCase):
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertTrue(any("INTERRUPTED" in l for l in f.logs), f.logs)
|
||||
self.assertTrue(any(l.startswith("os-apply: REPAIR ") and l.endswith("forced") for l in f.logs), f.logs)
|
||||
|
||||
|
||||
class OOMCheck(unittest.TestCase):
|
||||
"""R-528 (`09` decision 157): after a Docker engine step the wrapper proves the engine reports a memory kill
|
||||
(OOMKilled=true AND the `oom` event). Reported only; the hub decides. Red-proofs: audits/readback-2026-10-07/F/."""
|
||||
|
||||
def apply(self, **kw):
|
||||
f = docker_fake(signed=signed_job())
|
||||
for k, v in kw.items():
|
||||
setattr(f, k, v)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
return f, rep
|
||||
|
||||
def test_pass_when_oomkilled_and_the_event(self):
|
||||
f, rep = self.apply()
|
||||
oc = rep["oom_check"]
|
||||
self.assertEqual(oc["result"], "pass", oc)
|
||||
self.assertEqual((oc["oom_killed"], oc["oom_event"], oc["exit_code"]), (True, True, 137))
|
||||
self.assertEqual(oc["image"], "gitea.dooplex.hu/admin/felhom-controller:0.300.0")
|
||||
self.assertEqual(sorted(oc), ["detail", "exit_code", "image", "oom_event", "oom_killed", "result"])
|
||||
run_argv = f.oom_runs[0]
|
||||
name = run_argv[run_argv.index("--name") + 1]
|
||||
self.assertTrue(re.match(r"^felhom-oomcheck-[0-9]+-[0-9a-f]{8}$", name), name)
|
||||
for flag, val in (("--pull", "never"), ("--network", "none"), ("--memory", "64m"), ("--memory-swap", "64m"),
|
||||
("--label", "felhom.oomcheck=1"), ("--entrypoint", "sh")):
|
||||
self.assertEqual(run_argv[run_argv.index(flag) + 1], val, flag)
|
||||
self.assertNotIn("-v", run_argv)
|
||||
self.assertNotIn("-p", run_argv)
|
||||
self.assertEqual(run_argv[-3:], ["gitea.dooplex.hu/admin/felhom-controller:0.300.0", "-c", osapply.OOMCHECK_SCRIPT])
|
||||
self.assertIn(f"container={name}", f.oom_events_argv)
|
||||
self.assertIn("event=oom", f.oom_events_argv)
|
||||
self.assertEqual(f.oom_removed, [name], "the check container must be removed")
|
||||
self.assertTrue(rep["health_after"], "health is read before the check")
|
||||
# bounded: every call has a timeout and the clock is read twice — the worst case stays within 90 s
|
||||
self.assertLessEqual(sum(osapply.OOMCHECK_TIMEOUTS.values()) + osapply.OOMCHECK_TIMEOUTS["clock"]
|
||||
+ osapply.OOMCHECK_SETTLE, 90)
|
||||
|
||||
def test_events_window_ends_after_the_settle_wait(self):
|
||||
# Measured (F1/F2): an --until taken right after the run missed the oom event. The window must end after a
|
||||
# wait of >= 2 s that comes AFTER the run, at the guest epoch read after that wait, + 1.
|
||||
f, rep = self.apply()
|
||||
run_i = next(i for i, c in enumerate(f.calls) if c[-1][:2] == ["docker", "run"])
|
||||
date_i = [i for i, c in enumerate(f.calls) if c[-1] == ["date", "+%s"]]
|
||||
waits = [(i, s) for i, s in getattr(f, "sleeps", []) if i > run_i]
|
||||
self.assertTrue(waits and waits[0][1] >= 2, f"no settle wait after the run: {getattr(f, 'sleeps', None)}")
|
||||
self.assertTrue(date_i[-1] >= waits[0][0], "the end epoch must be read after the wait")
|
||||
ev = f.oom_events_argv
|
||||
since, until = int(ev[ev.index("--since") + 1]), int(ev[ev.index("--until") + 1])
|
||||
self.assertEqual(until, f.oom_epochs[-1] + 1, "until = the guest epoch read after the wait, + 1")
|
||||
self.assertGreater(until, f.oom_epochs[0] + 1)
|
||||
self.assertLess(since, f.oom_epochs[0] + 1)
|
||||
|
||||
def test_oomkilled_false_is_fail(self):
|
||||
f, rep = self.apply(oom_state="false 0")
|
||||
oc = rep["oom_check"]
|
||||
self.assertEqual(oc["result"], "fail", oc)
|
||||
self.assertIn("OOMKilled=true", oc["detail"])
|
||||
self.assertTrue(oc["oom_event"])
|
||||
self.assertEqual(len(f.oom_removed), 1)
|
||||
|
||||
def test_no_event_is_fail(self):
|
||||
f, rep = self.apply(oom_event=False)
|
||||
oc = rep["oom_check"]
|
||||
self.assertEqual(oc["result"], "fail", oc)
|
||||
self.assertIn("the oom event", oc["detail"])
|
||||
self.assertTrue(oc["oom_killed"])
|
||||
|
||||
def test_image_unreadable_is_error(self):
|
||||
f, rep = self.apply(oom_image=None)
|
||||
oc = rep["oom_check"]
|
||||
self.assertEqual(oc["result"], "error", oc)
|
||||
self.assertIsNone(oc["image"])
|
||||
self.assertIn("image could not be read", oc["detail"])
|
||||
self.assertFalse(hasattr(f, "oom_runs"), "no container is started without an image")
|
||||
|
||||
def test_container_removed_even_when_inspect_raises(self):
|
||||
f, rep = self.apply(oom_inspect_raises=True)
|
||||
oc = rep["oom_check"]
|
||||
self.assertEqual(oc["result"], "error", oc)
|
||||
self.assertEqual(len(f.oom_removed), 1, "the container must be removed even when inspect raised")
|
||||
self.assertTrue(f.oom_removed[0].startswith(osapply.OOMCHECK_PREFIX))
|
||||
self.assertNotIn("failed", rep, "the check never turns the step into a failure")
|
||||
|
||||
def test_never_on_guest_or_host_or_in_health_mode(self):
|
||||
g = Fake()
|
||||
rc, rep = run(g)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
h = Fake()
|
||||
h.plan["layer"] = "host"
|
||||
h.plan["packages"] = [{"name": "bash", "version": "5.2.37-2+b10", "origin": "Debian"}]
|
||||
h.installed["bash"] = "5.2.37-2+b9"
|
||||
h.live["bash"] = {"5.2.37-2+b10"}
|
||||
rc_h, rep_h = run(h)
|
||||
self.assertEqual(rc_h, 0, rep_h)
|
||||
d = docker_fake(signed=signed_job())
|
||||
d.plan["mode"] = "health"
|
||||
rc_d, rep_d = run(d)
|
||||
self.assertEqual(rc_d, 0, rep_d)
|
||||
for f, r in ((g, rep), (h, rep_h), (d, rep_d)):
|
||||
self.assertNotIn("oom_check", r)
|
||||
self.assertFalse(hasattr(f, "oom_runs"), r.get("layer"))
|
||||
self.assertFalse(any(c[-1][:2] == ["docker", "run"] for c in f.calls))
|
||||
|
||||
def test_mode_oom_check_runs_only_the_check(self):
|
||||
f = docker_fake() # no authority: the check changes nothing, so it needs none
|
||||
f.plan["mode"], f.plan["packages"] = "oom-check", []
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["oom_check"]["result"], "pass", rep)
|
||||
self.assertFalse(any("apt-get" in c[-1] or "dpkg-query" in c[-1] for c in f.calls), f.calls)
|
||||
self.assertEqual(len(f.oom_removed), 1)
|
||||
|
||||
def test_mode_oom_check_keeps_the_refusals(self):
|
||||
f = docker_fake()
|
||||
f.plan["mode"], f.plan["packages"] = "oom-check", []
|
||||
f.files["/etc/pve/lxc/9201.conf"] = "arch: amd64\n"
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R10"), rep)
|
||||
self.assertFalse(hasattr(f, "oom_runs"))
|
||||
g = Fake()
|
||||
g.plan["mode"] = "oom-check"
|
||||
rc, rep = run(g)
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
|
||||
|
||||
|
||||
PVE = "Proxmox Debian Repository:stable"
|
||||
PVE_SET = [{"name": "pve-manager", "version": "9.2.21", "origin": "Proxmox Debian Repository"},
|
||||
{"name": "libpve-common-perl", "version": "9.1.9", "origin": "Proxmox Debian Repository"}]
|
||||
|
||||
|
||||
def pve_fake(signed=None, ring0=False):
|
||||
f = Fake()
|
||||
f.installed.update({"pve-manager": "9.2.2", "libpve-common-perl": "9.1.1", "proxmox-kernel-helper": "9.0.4"})
|
||||
f.live["pve-manager"] = {"9.2.21", "9.2.2"}
|
||||
f.live["libpve-common-perl"] = {"9.1.9", "9.1.1"}
|
||||
f.origins = {"pve-manager": PVE, "libpve-common-perl": PVE, "proxmox-kernel-helper": PVE, "shim-signed": PVE,
|
||||
"proxmox-firewall-data": PVE}
|
||||
f.plan = {"release_id": "os-pve-t1", "layer": "pve", "lane": "slow", "vmid": 9201, "mode": "apply",
|
||||
"packages": [dict(p) for p in PVE_SET]}
|
||||
if ring0:
|
||||
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
|
||||
if signed is not None:
|
||||
f.plan["signed"] = signed
|
||||
return f
|
||||
|
||||
|
||||
class PVELane(unittest.TestCase):
|
||||
"""R-812 option A (`09` §3 decision 163): the host's Proxmox USERSPACE packages, slow lane, no kernel / boot /
|
||||
firmware / microcode (R14), no removal, a new package only from PVE_NEW_ALLOW. Red-proof: audits/day-2026-10-07/B/."""
|
||||
|
||||
def refused(self, f, code):
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 2, rep)
|
||||
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||
self.assertEqual(f.installed["pve-manager"], "9.2.2", "nothing may be installed on a refusal")
|
||||
return rep
|
||||
|
||||
# THE RED TEST (design-R-812 §5): before the pve layer existed this plan was refused R12.
|
||||
def test_pve_manager_plan_is_installed_on_the_host(self):
|
||||
f = pve_fake(signed=signed_job(packages=PVE_SET, op="os_pve_step"))
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.installed["pve-manager"], "9.2.21")
|
||||
self.assertEqual(rep["authority"], "signed")
|
||||
self.assertEqual(rep["pve_manager"], "9.2.21", "pveversion after the step is reported")
|
||||
inst = [c for c in f.calls if c[0] == "host" and "install" in c[1] and "-s" not in c[1] and "-f" not in c[1]
|
||||
and "--print-uris" not in c[1]]
|
||||
self.assertTrue(inst, "the pve layer installs on the HOST")
|
||||
self.assertFalse([c for c in f.calls if c[0] == "guest" and "install" in c[2]], "nothing installed in the guest")
|
||||
self.assertEqual(sorted(rep["health_after"]["host_services"]), sorted(osapply.HOST_SERVICES))
|
||||
|
||||
def test_kernel_in_a_pve_plan_is_refused(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["packages"].append({"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"})
|
||||
self.refused(f, "R14")
|
||||
|
||||
def test_shim_in_a_pve_plan_is_refused(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["packages"].append({"name": "shim-signed", "version": "1.47+pmx1", "origin": "Proxmox Debian Repository"})
|
||||
self.refused(f, "R14")
|
||||
|
||||
def test_kernel_pulled_in_by_the_simulation_is_refused(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.installed["proxmox-kernel-helper"] = "9.0.4"
|
||||
f.extra_sim = ["Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])"]
|
||||
self.refused(f, "R6") # not in the plan — refused before the name check; R14 below when it IS in the plan
|
||||
g = pve_fake(ring0=True)
|
||||
g.live["proxmox-kernel-helper"] = {"9.0.6"}
|
||||
g.plan["packages"] = [dict(PVE_SET[0])]
|
||||
g.extra_sim = ["Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])"]
|
||||
# a kernel-helper the plan did not name is R6; the R14 name check covers a listed one (test above)
|
||||
self.refused(g, "R6")
|
||||
|
||||
def test_debian_package_in_a_pve_plan_is_refused(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["packages"].append({"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"})
|
||||
self.refused(f, "R2")
|
||||
|
||||
def test_debian_origin_in_the_pve_simulation_is_refused(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.origins["libpve-common-perl"] = DEB
|
||||
self.refused(f, "R2")
|
||||
|
||||
def test_docker_package_in_a_pve_plan_is_refused(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["packages"].append({"name": "docker-ce", "version": "5:29.8.2-1~debian.13~trixie", "origin": "Proxmox Debian Repository"})
|
||||
self.refused(f, "R2")
|
||||
|
||||
def test_pve_in_the_fast_lane_is_refused(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["lane"] = "fast"
|
||||
self.refused(f, "R3")
|
||||
|
||||
def test_no_authority_is_refused(self):
|
||||
self.refused(pve_fake(), "R3")
|
||||
|
||||
def test_docker_signed_op_does_not_authorize_a_pve_step(self):
|
||||
self.refused(pve_fake(signed=signed_job(packages=PVE_SET, op="os_docker_step")), "R3")
|
||||
|
||||
def test_pve_on_a_byo_box_is_refused(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||
self.refused(f, "R12")
|
||||
|
||||
def test_unlisted_new_package_is_refused(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.extra_sim = ["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"]
|
||||
self.refused(f, "R6")
|
||||
|
||||
def test_allow_listed_new_package_is_accepted(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.extra_sim = ["Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])"]
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.installed["pve-manager"], "9.2.21")
|
||||
|
||||
def test_allow_new_is_never_an_open_door_in_the_fast_lane(self):
|
||||
f = Fake()
|
||||
f.plan["layer"] = "host"
|
||||
f.extra_sim = ["Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])"]
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R6"), rep)
|
||||
|
||||
def test_undo_is_refused(self):
|
||||
f = pve_fake(signed=signed_job(packages=PVE_SET, op="os_pve_step"))
|
||||
f.plan["undo"] = True
|
||||
self.refused(f, "R5")
|
||||
|
||||
def test_ring0_pending_pve_takes_only_installed_proxmox_userspace(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["select"], f.plan["packages"] = "pending-pve", []
|
||||
f.installed.update({"linux-image-amd64": "6.12.1", "tailscale": "1.102.2"})
|
||||
f.pending_sim = [
|
||||
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst libpve-common-perl [9.1.1] (9.1.9 Proxmox Debian Repository:stable [all])",
|
||||
"Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])",
|
||||
"Inst proxmox-kernel-7.0.14-20-pve-signed (7.0.14-20 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])",
|
||||
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])",
|
||||
"Inst tailscale [1.102.2] (1.102.5 Tailscale:pkgs.tailscale.com [amd64])",
|
||||
]
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["authority"], "ring0")
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["libpve-common-perl", "pve-manager"],
|
||||
"pending-pve: installed, Proxmox-origin, never kernel/boot/firmware, never Debian or other origins")
|
||||
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3")
|
||||
|
||||
# 2026-10-09, demo-hp (Secure Boot on): proxmox-secure-boot-support's upgrade pulls shim-signed-common; in the pve
|
||||
# plan that refused the whole step (R6) and skipped the night's kernel step. It is a boot-chain package: left out.
|
||||
# RED-PROOF: drop proxmox-secure-boot-support from HOST_SLOW_RE -> it is in the plan -> this test fails.
|
||||
def test_ring0_pending_pve_leaves_the_secure_boot_meta_out(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["select"], f.plan["packages"] = "pending-pve", []
|
||||
f.installed.update({"proxmox-secure-boot-support": "9.0.1"})
|
||||
f.pending_sim = [
|
||||
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst proxmox-secure-boot-support [9.0.1] (9.0.2 Proxmox Debian Repository:stable [all])",
|
||||
"Inst shim-signed-common [1.48+pmx1+16.1-1+pmx1] (1.51+pmx1+16.1-2+pmx1 Proxmox Debian Repository:stable [all])",
|
||||
]
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
|
||||
|
||||
def test_select_pending_pve_needs_the_pve_layer(self):
|
||||
f = Fake()
|
||||
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
|
||||
|
||||
|
||||
# ---------- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----------
|
||||
U = "1af1fcc6-639c-416b-a7e5-c4470d41a502"
|
||||
OLD, NEW = "7.0.2-6-pve", "7.0.14-22-pve"
|
||||
KERNEL_SET = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"},
|
||||
{"name": "proxmox-kernel-7.0.14-22-pve-signed", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"}]
|
||||
|
||||
|
||||
class KFake(Fake):
|
||||
"""A Proxmox host with GRUB on UEFI: /boot on the root LV, a vfat ESP, the bundle's two generators. update-grub is
|
||||
emulated like the real 10_linux: WITHOUT the felhom default file the NEWEST kernel becomes the default (measured,
|
||||
R-836 — that is the defect the lane exists for); with it, the kernel it names."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.running = OLD
|
||||
self.boot = {OLD}
|
||||
self.efi, self.esp_fs, self.boot_mount, self.mods, self.snippets, self.pin = True, "vfat", "", True, True, False
|
||||
self.env = None # None = no env block on the ESP; else {"felhom_next": ...}
|
||||
self.tree = {} # files put_file wrote
|
||||
self.reboots = []
|
||||
self.update_grubs = 0
|
||||
self.grub_runs_in_postinst = True
|
||||
self.installed.update({"proxmox-kernel-7.0": "7.0.2-6", "proxmox-default-kernel": "2.1.0",
|
||||
"pve-firmware": "3.18-3", "proxmox-kernel-7.0.2-6-pve-signed": "7.0.2-6",
|
||||
"pve-manager": "9.2.21"})
|
||||
self.live.update({"proxmox-kernel-7.0": {"7.0.14-22", "7.0.2-6"},
|
||||
"proxmox-kernel-7.0.14-22-pve-signed": {"7.0.14-22"},
|
||||
"proxmox-kernel-7.0.14-23-pve-signed": {"7.0.14-23"},
|
||||
"pve-firmware": {"3.18-7", "3.18-3"}})
|
||||
self.origins = {}
|
||||
self.kernel_pending = ["Inst pve-firmware [3.18-3] (3.18-7 Proxmox Debian Repository:stable [all])",
|
||||
"Inst proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst proxmox-kernel-7.0 [7.0.2-6] (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst pve-manager [9.2.21] (9.2.22 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])"]
|
||||
self.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "apply",
|
||||
"select": "pending-kernel", "packages": []}
|
||||
self.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
|
||||
self.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 0})
|
||||
self.cfg = self.render()
|
||||
|
||||
# -- GRUB --
|
||||
def newest(self):
|
||||
best = None
|
||||
for k in self.boot:
|
||||
if best is None or dpkg_cmp(k[:-4], "gt", best[:-4]):
|
||||
best = k
|
||||
return best
|
||||
|
||||
def render(self):
|
||||
d = self.tree.get(osapply.KERNEL_DEFAULT_CFG)
|
||||
if d:
|
||||
dflt = re.search(r'GRUB_DEFAULT="([^"]+)"', d).group(1)
|
||||
else:
|
||||
dflt = f"gnulinux-advanced-{U}>gnulinux-{self.newest()}-advanced-{U}"
|
||||
cfg = ('if [ "${next_entry}" ] ; then\n set default="${next_entry}"\nelse\n'
|
||||
f' set default="{dflt}"\nfi\n'
|
||||
f"submenu 'Advanced options' $menuentry_id_option 'gnulinux-advanced-{U}' {{\n")
|
||||
for k in sorted(self.boot):
|
||||
cfg += f" menuentry 'Proxmox VE, with Linux {k}' $menuentry_id_option 'gnulinux-{k}-advanced-{U}' {{ }}\n"
|
||||
cfg += "}\n"
|
||||
if self.snippets:
|
||||
cfg += "### BEGIN /etc/grub.d/01_felhom_oneshot ###\nload_env felhom_next\n"
|
||||
cfg += "".join(f"menuentry 'Felhom one-shot: {k}' --id felhom-oneshot-{k} {{ }}\n" for k in sorted(self.boot))
|
||||
return cfg
|
||||
|
||||
def lexists(self, p):
|
||||
if p == "/sys/firmware/efi":
|
||||
return self.efi
|
||||
if p.startswith("/usr/lib/grub/x86_64-efi/"):
|
||||
return self.mods
|
||||
if p in (osapply.ONESHOT_SNIPPET, osapply.ONESHOT_ENTRIES):
|
||||
return self.snippets
|
||||
if p == osapply.KERNEL_PIN_FILE:
|
||||
return self.pin
|
||||
m = re.match(r"^/boot/(vmlinuz|initrd\.img)-(.+)$", p)
|
||||
if m:
|
||||
return m.group(2) in self.boot
|
||||
return p in self.tree
|
||||
|
||||
def put_file(self, path, data, mode):
|
||||
self.tree[path] = data.decode()
|
||||
|
||||
def read_file(self, p):
|
||||
if p == osapply.GRUB_CFG:
|
||||
return self.cfg
|
||||
if p in self.tree:
|
||||
return self.tree[p]
|
||||
return super().read_file(p)
|
||||
|
||||
def host(self, argv, timeout=600, stdin=None):
|
||||
if argv[0] == "uname":
|
||||
self.calls.append(("host", argv))
|
||||
return 0, self.running + "\n", ""
|
||||
if argv[0] == "findmnt":
|
||||
self.calls.append(("host", argv))
|
||||
if argv[-1] == osapply.ESP_MOUNT:
|
||||
return (0, self.esp_fs + "\n", "") if self.esp_fs else (1, "", "")
|
||||
return (0, self.boot_mount + "\n", "") if self.boot_mount else (1, "", "")
|
||||
if argv[0] == "grub-editenv":
|
||||
self.calls.append(("host", argv))
|
||||
if argv[2] == "list":
|
||||
return (1, "", "no such file") if self.env is None else \
|
||||
(0, "".join(f"{k}={v}\n" for k, v in self.env.items()), "")
|
||||
if argv[2] == "create":
|
||||
self.env = {}
|
||||
return 0, "", ""
|
||||
if argv[2] == "set":
|
||||
k, v = argv[3].split("=", 1)
|
||||
self.env[k] = v
|
||||
return 0, "", ""
|
||||
if argv[2] == "unset":
|
||||
self.env.pop(argv[3], None)
|
||||
return 0, "", ""
|
||||
if argv[0] == "update-grub":
|
||||
self.calls.append(("host", argv))
|
||||
self.update_grubs += 1
|
||||
self.cfg = self.render()
|
||||
return 0, "", ""
|
||||
if argv[0] == "mkdir":
|
||||
self.calls.append(("host", argv))
|
||||
return 0, "", ""
|
||||
if argv[:2] == ["systemctl", "reboot"]:
|
||||
self.calls.append(("host", argv))
|
||||
self.reboots.append(self.running)
|
||||
return 0, "", ""
|
||||
return super().host(argv, timeout, stdin)
|
||||
|
||||
def emulate(self, argv):
|
||||
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
|
||||
if a[0] == "apt-get" and "install" in a and "-s" not in a and "--print-uris" not in a and "-f" not in a:
|
||||
if self.install_rc:
|
||||
return self.install_rc, "", "E: boom"
|
||||
for x in a:
|
||||
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||
n, v = x.split("=", 1)
|
||||
self.installed[n] = v
|
||||
m = re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)
|
||||
if m:
|
||||
self.installed[f"proxmox-kernel-{v}-pve-signed"] = v
|
||||
if m or osapply.KERNEL_IMAGE_RE.match(n):
|
||||
self.boot.add(v + "-pve")
|
||||
if self.grub_runs_in_postinst:
|
||||
self.cfg = self.render() # the kernel's postinst runs update-grub (zz-update-grub)
|
||||
return 0, "Setting up proxmox-kernel ...\n", ""
|
||||
return super().emulate(argv)
|
||||
|
||||
def sim(self, a):
|
||||
if "--print-uris" in a:
|
||||
return 0, "", ""
|
||||
if "dist-upgrade" in a:
|
||||
return 0, "\n".join(self.kernel_pending) + "\n", ""
|
||||
out, named = "", set()
|
||||
for x in a:
|
||||
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||
named.add(x.split("=", 1)[0])
|
||||
for x in a:
|
||||
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||
n, v = x.split("=", 1)
|
||||
if v not in self.avail(n):
|
||||
return 100, "", f"E: Version '{v}' for '{n}' was not found"
|
||||
o = self.origins.get(n, PVE)
|
||||
out += f"Inst {n} [{self.installed[n]}] ({v} {o} [amd64])\n" if n in self.installed else f"Inst {n} ({v} {o} [amd64])\n"
|
||||
if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n):
|
||||
img = f"proxmox-kernel-{v}-pve-signed"
|
||||
if img not in self.installed and img not in named:
|
||||
out += f"Inst {img} ({v} {PVE} [amd64])\n"
|
||||
out += "".join(l + "\n" for l in self.extra_sim)
|
||||
return 0, out, ""
|
||||
|
||||
|
||||
def kfake(**kw):
|
||||
f = KFake()
|
||||
for k, v in kw.items():
|
||||
setattr(f, k, v)
|
||||
return f
|
||||
|
||||
|
||||
def kmode(f, mode, **extra):
|
||||
f.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": mode, "packages": []}
|
||||
f.plan.update(extra)
|
||||
return run(f)
|
||||
|
||||
|
||||
def staged(f=None):
|
||||
f = f or kfake()
|
||||
rc, rep = run(f)
|
||||
assert rc == 0, rep
|
||||
return f
|
||||
|
||||
|
||||
class KernelLane(unittest.TestCase):
|
||||
"""R-836 / `09` §3 decision 172: stage a kernel once through the ESP flag; the default moves only after a healthy
|
||||
one-shot boot. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
|
||||
|
||||
def refused(self, f, code, mode=None, **extra):
|
||||
rc, rep = kmode(f, mode, **extra) if mode else run(f)
|
||||
self.assertEqual(rc, 2, rep)
|
||||
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||
return rep
|
||||
|
||||
# --- the four red tests the brief names (Part A 3) ---
|
||||
def test_installing_a_kernel_does_not_change_the_grub_default(self):
|
||||
f = kfake()
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertIn(NEW, f.boot, "the new kernel was installed")
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD,
|
||||
"the default must stay the kernel the box runs (R-836: an install made the new one default)")
|
||||
self.assertIn(f"gnulinux-{OLD}-advanced-{U}", f.tree[osapply.KERNEL_DEFAULT_CFG])
|
||||
self.assertEqual(f.env, {"felhom_next": NEW}, "the ONLY way to the new kernel is the one-shot flag")
|
||||
self.assertEqual(f.reboots, [], "staging never reboots")
|
||||
st = json.loads(f.tree[osapply.KERNEL_STATE])
|
||||
self.assertEqual((st["phase"], st["from"], st["to"]), ("staged", OLD, NEW))
|
||||
self.assertEqual(rep["kernel"]["default"], OLD)
|
||||
|
||||
def test_a_box_without_the_esp_flag_is_refused(self):
|
||||
for attr, val, frag in (("esp_fs", "ext4", "vfat"), ("efi", False, "UEFI"), ("snippets", False, "bundle"),
|
||||
("mods", False, "module"), ("boot_mount", "/boot", "separate"), ("pin", True, "pinned")):
|
||||
f = kfake(**{attr: val})
|
||||
rep = self.refused(f, "R20")
|
||||
self.assertIn(frag, rep["refused"]["reason"], attr)
|
||||
self.assertNotIn(NEW, f.boot, f"{attr}: nothing may be installed on a refusal")
|
||||
self.assertIsNone(f.env, f"{attr}: no flag on a refusal")
|
||||
|
||||
def test_a_reboot_without_the_flag_is_refused(self):
|
||||
f = staged()
|
||||
f.env = {"felhom_next": ""}
|
||||
self.refused(f, "R22", mode="kernel-reboot")
|
||||
self.assertEqual(f.reboots, [])
|
||||
|
||||
def test_a_kernel_outside_the_approved_set_is_refused(self):
|
||||
# a signed set that names only the meta-package: the image the sources pull in was never signed for
|
||||
f = kfake()
|
||||
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
f.plan.update(select="listed", packages=[dict(KERNEL_SET[0])],
|
||||
signed=signed_job(packages=[KERNEL_SET[0]], op="os_kernel_step"))
|
||||
self.refused(f, "R23")
|
||||
self.assertNotIn(NEW, f.boot)
|
||||
# a signed set names 7.0.14-22; the sources would ALSO bring 7.0.14-23
|
||||
f2 = kfake()
|
||||
f2.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
f2.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
|
||||
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
|
||||
f2.extra_sim = ["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"]
|
||||
self.refused(f2, "R23")
|
||||
# a name that is not a kernel package at all
|
||||
g = kfake()
|
||||
g.plan.update(select="listed", packages=[{"name": "pve-manager", "version": "9.2.22", "origin": "Proxmox Debian Repository"}])
|
||||
self.refused(g, "R23")
|
||||
# the household was told about another kernel
|
||||
h = kfake()
|
||||
h.plan["expect_kver"] = "7.0.14-23-pve"
|
||||
self.refused(h, "R23")
|
||||
self.assertNotIn(NEW, h.boot)
|
||||
|
||||
def test_the_snippet_clears_the_flag_on_use(self):
|
||||
"""01_felhom_oneshot: the flag is copied, CLEARED and SAVED before any `set default`, all inside the one branch
|
||||
that runs only when the flag is set; a default is set only for an installed kernel's own entry."""
|
||||
text = (HERE / "felhom-grub-oneshot.sh").read_text()
|
||||
body = text[text.index("cat <<EOF"):]
|
||||
i_copy = body.index('set felhom_boot="\\${felhom_next}"')
|
||||
i_clear = body.index("set felhom_next=\n")
|
||||
i_save = body.index("save_env -f (\\$felhom_esp)/EFI/felhom/oneshot.env felhom_next")
|
||||
i_default = body.index('set default="felhom-oneshot-%s"')
|
||||
self.assertLess(i_copy, i_clear)
|
||||
self.assertLess(i_clear, i_save)
|
||||
self.assertLess(i_save, i_default, "the flag must be cleared on disk BEFORE GRUB boots anything")
|
||||
self.assertIn('if [ "${felhom_boot}" = "%s" ]', body, "a default only for a kernel that is installed")
|
||||
self.assertIn(osapply.ONESHOT_ENV[len(osapply.ESP_MOUNT):], text, "the wrapper and GRUB name the same env file")
|
||||
|
||||
# --- the rest of the stage ---
|
||||
def test_option_c_options_are_on_the_one_shot_entry_only(self):
|
||||
text = (HERE / "felhom-grub-oneshot-entries.sh").read_text()
|
||||
self.assertIn(osapply.ONESHOT_ARGS, text)
|
||||
self.assertIn("--id felhom-oneshot-$k", text)
|
||||
self.assertNotIn("set default", text, "the entries file never sets the default")
|
||||
|
||||
def test_both_generators_ride_the_bundle(self):
|
||||
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_SNIPPET][1:4], ("felhom-grub-oneshot.sh", 0o755, "sh"))
|
||||
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_ENTRIES][1:4], ("felhom-grub-oneshot-entries.sh", 0o755, "sh"))
|
||||
|
||||
def test_ring0_pending_kernel_takes_only_the_kernel_set(self):
|
||||
f = kfake()
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["authority"], "ring0")
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["proxmox-kernel-7.0", "pve-firmware"])
|
||||
self.assertEqual(f.installed["pve-manager"], "9.2.21", "a Proxmox userspace package is the pve lane's")
|
||||
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3", "a Debian package is the fast lane's")
|
||||
|
||||
def test_signed_listed_set_is_installed(self):
|
||||
f = kfake()
|
||||
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
f.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
|
||||
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["authority"], "signed")
|
||||
self.assertEqual(f.env, {"felhom_next": NEW})
|
||||
|
||||
def test_no_authority_and_the_wrong_op_are_refused(self):
|
||||
f = kfake()
|
||||
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
self.refused(f, "R3")
|
||||
g = kfake()
|
||||
g.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
g.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
|
||||
signed=signed_job(packages=KERNEL_SET, op="os_pve_step"))
|
||||
self.refused(g, "R3")
|
||||
|
||||
def test_fast_lane_byo_and_wrong_select_are_refused(self):
|
||||
f = kfake()
|
||||
f.plan["lane"] = "fast"
|
||||
self.refused(f, "R3")
|
||||
g = kfake()
|
||||
g.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||
self.refused(g, "R12")
|
||||
h = kfake()
|
||||
h.plan["select"] = "pending-pve"
|
||||
self.refused(h, "R11")
|
||||
k = kfake()
|
||||
k.plan.update(layer="host", lane="fast", mode="kernel-status")
|
||||
self.refused(k, "R11")
|
||||
|
||||
def test_the_crash_guard_must_be_armed_and_quiet(self):
|
||||
f = kfake()
|
||||
f.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": False, "tripped": True, "unclean_boots_in_window": 2})
|
||||
self.refused(f, "R21")
|
||||
g = kfake()
|
||||
g.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 1})
|
||||
self.refused(g, "R21")
|
||||
h = kfake()
|
||||
del h.files[osapply.CRASH_GUARD_STATE]
|
||||
self.refused(h, "R21")
|
||||
self.assertNotIn(NEW, h.boot)
|
||||
|
||||
def test_never_two_steps_in_one_night(self):
|
||||
f = staged()
|
||||
self.refused(f, "R22") # still staged
|
||||
g = staged()
|
||||
st = json.loads(g.tree[osapply.KERNEL_STATE])
|
||||
st["phase"] = "good"
|
||||
g.tree[osapply.KERNEL_STATE] = json.dumps(st)
|
||||
g.clock += 3600
|
||||
self.refused(g, "R22") # done, but within 20 h
|
||||
|
||||
def test_removal_new_non_kernel_two_kernels_and_older_are_refused(self):
|
||||
f = kfake(extra_sim=["Remv pve-firmware [3.18-3]"])
|
||||
self.refused(f, "R4")
|
||||
g = kfake(extra_sim=["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"])
|
||||
self.refused(g, "R6")
|
||||
h = kfake(extra_sim=["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"])
|
||||
self.refused(h, "R23")
|
||||
k = kfake(running="7.0.14-23-pve")
|
||||
k.boot = {"7.0.14-23-pve"}
|
||||
k.cfg = k.render()
|
||||
self.refused(k, "R23")
|
||||
m = kfake()
|
||||
m.origins = {"proxmox-kernel-7.0": DEB}
|
||||
self.refused(m, "R2")
|
||||
|
||||
# R-898: ring 0 stages EXACTLY the told kernel (select listed, the set derived from it) — even when the sources
|
||||
# offer a newer one by night; a told version that can no longer be installed is refused BEFORE any change (R7).
|
||||
def test_ring0_listed_installs_the_told_kernel_not_the_newest(self):
|
||||
f = kfake()
|
||||
f.live["proxmox-kernel-7.0"] = {"7.0.14-20", "7.0.14-22", "7.0.2-6"}
|
||||
f.live["proxmox-kernel-7.0.14-20-pve-signed"] = {"7.0.14-20"}
|
||||
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
|
||||
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
|
||||
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["authority"], "ring0")
|
||||
self.assertEqual(f.env, {"felhom_next": "7.0.14-20-pve"})
|
||||
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.14-20")
|
||||
self.assertNotIn("7.0.14-22-pve", f.boot)
|
||||
|
||||
def test_ring0_told_kernel_gone_is_refused_before_any_change(self):
|
||||
f = kfake()
|
||||
f.live["proxmox-kernel-7.0"] = {"7.0.14-22"} # 7.0.14-20 is no longer in the archive
|
||||
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
|
||||
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
|
||||
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
|
||||
rep = self.refused(f, "R7")
|
||||
self.assertIsNone(f.env)
|
||||
self.assertNotIn(osapply.KERNEL_DEFAULT_CFG, f.tree, "refused before the default was even pinned")
|
||||
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.2-6")
|
||||
|
||||
def test_nothing_pending_changes_nothing(self):
|
||||
f = kfake(kernel_pending=[])
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["upgraded"], [])
|
||||
self.assertIsNone(f.env)
|
||||
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
|
||||
|
||||
def test_a_failed_install_writes_no_flag(self):
|
||||
f = kfake(install_rc=100)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 3, rep)
|
||||
self.assertIsNone(f.env)
|
||||
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
|
||||
|
||||
def test_a_postinst_without_update_grub_is_regenerated_and_proved(self):
|
||||
f = kfake(grub_runs_in_postinst=False)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertIn(f"felhom-oneshot-{NEW}", f.cfg)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
|
||||
|
||||
# --- the reboot, the boot, good / revert ---
|
||||
def test_reboot_of_a_staged_step(self):
|
||||
f = staged()
|
||||
rc, rep = kmode(f, "kernel-reboot")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.reboots, [OLD])
|
||||
st = json.loads(f.tree[osapply.KERNEL_STATE])
|
||||
self.assertEqual(st["phase"], "oneshot")
|
||||
self.assertIn("host_services", st["health_before"])
|
||||
|
||||
def test_reboot_needs_a_staged_step(self):
|
||||
self.refused(kfake(), "R22", mode="kernel-reboot")
|
||||
|
||||
def boot_into(self, f, kver):
|
||||
f.running = kver
|
||||
if f.env and f.env.get("felhom_next"):
|
||||
f.env["felhom_next"] = "" # GRUB cleared it (01_felhom_oneshot)
|
||||
return kmode(f, "kernel-boot")
|
||||
|
||||
def test_healthy_one_shot_becomes_the_default(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
rc, rep = self.boot_into(f, NEW)
|
||||
self.assertEqual((rc, rep["kernel_event"], rep["kernel"]["phase"]), (0, "judging", "judging"), rep)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "judging does not move the default")
|
||||
rc, rep = kmode(f, "kernel-good")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), NEW)
|
||||
self.assertEqual(rep["kernel"]["phase"], "good")
|
||||
|
||||
def test_a_panic_falls_back_and_is_recorded(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
rc, rep = self.boot_into(f, OLD)
|
||||
self.assertEqual((rc, rep["kernel_event"]), (0, "fell_back"), rep)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
|
||||
self.refused(f, "R22", mode="kernel-good")
|
||||
|
||||
def test_one_self_revert_then_never_again(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
self.boot_into(f, NEW)
|
||||
rc, rep = kmode(f, "kernel-revert", reason="the customer guest is not running")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.reboots, [OLD, NEW])
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "the self-revert boots the default, the old kernel")
|
||||
rc, rep = self.boot_into(f, OLD)
|
||||
self.assertEqual(rep["kernel_event"], "self_reverted")
|
||||
# the same step can never revert again
|
||||
st = json.loads(f.tree[osapply.KERNEL_STATE])
|
||||
st["phase"] = "judging"
|
||||
f.tree[osapply.KERNEL_STATE] = json.dumps(st)
|
||||
f.running = NEW
|
||||
self.refused(f, "R22", mode="kernel-revert")
|
||||
self.assertEqual(len(f.reboots), 2)
|
||||
|
||||
def test_a_revert_that_comes_back_new_is_never_retried(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
self.boot_into(f, NEW)
|
||||
kmode(f, "kernel-revert")
|
||||
rc, rep = self.boot_into(f, NEW)
|
||||
self.assertEqual(rep["kernel_event"], "revert_failed")
|
||||
self.refused(f, "R22", mode="kernel-revert")
|
||||
|
||||
def test_revert_refuses_an_unknown_default(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
self.boot_into(f, NEW)
|
||||
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-9.9.9-1-pve-advanced-{U}"\n'
|
||||
f.cfg = f.render()
|
||||
self.refused(f, "R20", mode="kernel-revert")
|
||||
self.assertEqual(len(f.reboots), 1)
|
||||
|
||||
def test_cancel_clears_the_flag(self):
|
||||
f = staged()
|
||||
rc, rep = kmode(f, "kernel-cancel")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertFalse(f.env.get("felhom_next"), "the flag is gone")
|
||||
self.assertEqual(rep["kernel"]["phase"], "cancelled")
|
||||
|
||||
def test_status_is_read_only_and_names_setup_problems(self):
|
||||
f = kfake(esp_fs="ext4")
|
||||
rc, rep = kmode(f, "kernel-status")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertTrue(rep["kernel"]["setup_problems"])
|
||||
self.assertEqual(f.tree, {})
|
||||
self.assertFalse([c for c in f.calls if c[1][0] in ("update-grub", "systemctl", "apt-get")])
|
||||
|
||||
def test_facts_carry_the_kernel_lane(self):
|
||||
f = staged()
|
||||
f.plan = {"release_id": "facts", "layer": "host", "lane": "fast", "vmid": 9201, "mode": "facts", "packages": []}
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
h = rep["facts"]["host"]
|
||||
self.assertEqual(h["kernel_lane"]["phase"], "staged")
|
||||
self.assertEqual(h["kernel_next_boot"], NEW)
|
||||
self.assertIn("one-shot", h["kernel_next_boot_source"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -56,6 +56,18 @@ class FakeHost:
|
||||
def log(self, line):
|
||||
self.logs.append(line)
|
||||
|
||||
# controller-image (R-861 (a) A1): the ref arrives on stdin and is written INSIDE the guest by root.
|
||||
stdin = b""
|
||||
guest_writes = None
|
||||
|
||||
def read_stdin(self, limit):
|
||||
return self.stdin[:limit + 1]
|
||||
|
||||
def write_guest_image(self, vmid, data):
|
||||
if self.guest_writes is None:
|
||||
self.guest_writes = []
|
||||
self.guest_writes.append((vmid, data))
|
||||
|
||||
|
||||
LOCAL_UNIT = """# Managed by felhom-agent — do not edit by hand.
|
||||
[Unit]
|
||||
@@ -316,5 +328,53 @@ class Refuses(unittest.TestCase):
|
||||
self.refused(h, ["wg", "/etc/shadow"], "A1")
|
||||
|
||||
|
||||
class ControllerImage(unittest.TestCase):
|
||||
"""R-861 (a) A1 (`09` §3 decision 165): the agent can no longer `tee` any image ref into the guest. The root verb
|
||||
reads the ref on stdin, requires our registry + our repository + an x.y.z tag, and writes the guest file itself.
|
||||
RED-PROOF: on the pre-A1 wrapper `controller-image` is not a verb (A1 usage, rc 2) — the accepted case fails."""
|
||||
|
||||
def go(self, ref, *argv):
|
||||
h = FakeHost()
|
||||
h.stdin = ref.encode() if isinstance(ref, str) else ref
|
||||
return h, run(h, *(argv or ("controller-image", "9201")))
|
||||
|
||||
def test_our_controller_ref_is_written_in_the_guest(self):
|
||||
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")
|
||||
self.assertEqual(rc, 0, h.logs)
|
||||
self.assertEqual(h.guest_writes, [(9201, b"gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")])
|
||||
|
||||
def test_a_foreign_image_is_refused(self):
|
||||
for ref in ("docker.io/library/alpine:latest\n", "alpine\n",
|
||||
"gitea.dooplex.hu/admin/felhom-controller:latest\n",
|
||||
"gitea.dooplex.hu/admin/other:0.1.0\n",
|
||||
"evil.example/admin/felhom-controller:0.301.0\n",
|
||||
"gitea.dooplex.hu/admin/felhom-controller:0.301.0\nalpine\n",
|
||||
"gitea.dooplex.hu/admin/felhom-controller:0.301.0 x\n",
|
||||
"", "\n"):
|
||||
h, rc = self.go(ref)
|
||||
self.assertEqual(rc, 3, f"{ref!r} was accepted")
|
||||
self.assertFalse(h.guest_writes, f"{ref!r} wrote the guest file")
|
||||
self.assertTrue(any("[I1]" in l for l in h.logs), h.logs)
|
||||
|
||||
def test_oversize_stdin_is_refused(self):
|
||||
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0" + " " * 300)
|
||||
self.assertEqual(rc, 3)
|
||||
self.assertFalse(h.guest_writes)
|
||||
|
||||
def test_vmid_must_be_numeric(self):
|
||||
for argv in (("controller-image", "9201;id"), ("controller-image", "-1"), ("controller-image",),
|
||||
("controller-image", "9201", "9202")):
|
||||
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n", *argv)
|
||||
self.assertIn(rc, (2, 3), argv)
|
||||
self.assertFalse(h.guest_writes, argv)
|
||||
|
||||
def test_self_check_names_the_verb(self):
|
||||
import io, contextlib
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
pa.main(["--self-check"])
|
||||
self.assertIn("controller-image", buf.getvalue())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for configs/felhom-vzdump-partial-clean (R-1006): it removes a DEAD half-written vzdump backup and refuses
|
||||
everything else. Each refusal has a positive control: the same setup without the one condition is removed — so a
|
||||
test cannot pass because the helper refuses everything. A fake storage.cfg and a fake /proc; nothing real is touched.
|
||||
Run: python3 configs/test_felhom_vzdump_partial_clean.py (also run by internal/backup's Go test)
|
||||
"""
|
||||
import os, pathlib, subprocess, tempfile, time, unittest
|
||||
|
||||
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-vzdump-partial-clean"
|
||||
BOOT = 1_791_967_481
|
||||
NAME = "vzdump-lxc-9201-2026_10_14-10_42_25"
|
||||
|
||||
|
||||
class Box:
|
||||
"""A temp tree: a dir storage `local` (path vz), an lvmthin `local-lvm`, a dir storage `other`, and a fake /proc."""
|
||||
|
||||
def __init__(self):
|
||||
self.t = tempfile.TemporaryDirectory()
|
||||
r = pathlib.Path(self.t.name)
|
||||
self.vz, self.other, self.nas, self.proc = r / "vz", r / "other", r / "nas", r / "proc"
|
||||
for d in (self.vz / "dump", self.other / "dump", self.nas / "dump", self.proc):
|
||||
d.mkdir(parents=True)
|
||||
(self.proc / "stat").write_text(f"cpu 1 2 3\nbtime {BOOT}\n")
|
||||
self.cfg = r / "storage.cfg"
|
||||
self.cfg.write_text(f"dir: local\n\tpath {self.vz}\n\tcontent iso,backup\n\n"
|
||||
f"lvmthin: local-lvm\n\tthinpool data\n\tvgname pve\n\n"
|
||||
f"dir: other\n\tpath {self.other}\n\tcontent backup\n\n"
|
||||
f"nfs: nas\n\tpath {self.nas}\n\tserver 192.0.2.1\n\texport /x\n\tcontent backup\n")
|
||||
self.pid = 1000
|
||||
|
||||
def partial(self, name=NAME + ".tar.dat", mtime=BOOT - 100, where=None, size=4096):
|
||||
p = (where or self.vz / "dump") / name
|
||||
if name.endswith(".tmp"):
|
||||
(p / "etc" / "vzdump").mkdir(parents=True)
|
||||
(p / "etc" / "vzdump" / "pct.conf").write_text("x")
|
||||
for q in [p / "etc" / "vzdump" / "pct.conf", p / "etc" / "vzdump", p / "etc", p]:
|
||||
os.utime(q, (mtime, mtime))
|
||||
else:
|
||||
p.write_bytes(b"\0" * size)
|
||||
os.utime(p, (mtime, mtime))
|
||||
return p
|
||||
|
||||
def process(self, cmdline, fds=(), cwd=None):
|
||||
self.pid += 1
|
||||
d = self.proc / str(self.pid)
|
||||
(d / "fd").mkdir(parents=True)
|
||||
(d / "cmdline").write_bytes(b"\0".join(a.encode() for a in cmdline) + b"\0")
|
||||
for i, f in enumerate(fds):
|
||||
os.symlink(str(f), d / "fd" / str(i + 3))
|
||||
os.symlink(str(cwd or "/"), d / "cwd")
|
||||
|
||||
def run(self, storage, vmid, name, quiet=120):
|
||||
script = (f'source "{HELPER}"; STORAGE_CFG="{self.cfg}"; PROC="{self.proc}"; QUIET_S={quiet}; '
|
||||
f'main "$@"')
|
||||
r = subprocess.run(["bash", "-c", script, "x", storage, str(vmid), name], capture_output=True, text=True)
|
||||
return r.returncode, r.stdout + r.stderr
|
||||
|
||||
|
||||
class Removes(unittest.TestCase):
|
||||
def test_dead_tar_dat_before_boot(self):
|
||||
b = Box(); p = b.partial()
|
||||
rc, out = b.run("local", 9201, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("written before this boot", out)
|
||||
|
||||
def test_dead_tmp_dir_before_boot(self):
|
||||
b = Box(); p = b.partial(NAME + ".tmp")
|
||||
rc, out = b.run("local", 9201, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
|
||||
|
||||
def test_quiet_partial_after_boot(self):
|
||||
b = Box(); p = b.partial(mtime=int(time.time()) - 300)
|
||||
rc, out = b.run("local", 9201, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("untouched", out)
|
||||
|
||||
def test_qemu_vma_dat(self):
|
||||
b = Box(); p = b.partial("vzdump-qemu-341-2026_10_14-10_42_25.vma.dat")
|
||||
rc, out = b.run("local", 341, p.name)
|
||||
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
|
||||
|
||||
|
||||
class Keeps(unittest.TestCase):
|
||||
"""Each case: (setup that must be REFUSED, the same setup that must be REMOVED)."""
|
||||
|
||||
def pair(self, refuse, control, quiet=120):
|
||||
b = Box(); target, args = refuse(b)
|
||||
rc, out = b.run(*args, quiet=quiet)
|
||||
self.assertNotEqual(rc, 0, f"must refuse: {out}")
|
||||
self.assertIn("REFUSED", out)
|
||||
self.assertTrue(os.path.lexists(target), "a refused entry must still exist")
|
||||
c = Box(); ctarget, cargs = control(c)
|
||||
crc, cout = c.run(*cargs, quiet=quiet)
|
||||
self.assertEqual(crc, 0, f"positive control must be removed: {cout}")
|
||||
self.assertFalse(os.path.lexists(ctarget))
|
||||
|
||||
def test_complete_archive_of_same_vmid(self):
|
||||
self.pair(lambda b: (b.partial(NAME + ".tar.zst"), ("local", 9201, NAME + ".tar.zst")),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_log_and_notes(self):
|
||||
for suffix in (".log", ".tar.zst.notes"):
|
||||
self.pair(lambda b, s=suffix: (b.partial(NAME + s), ("local", 9201, NAME + s)),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_partial_of_another_vmid(self):
|
||||
other = "vzdump-lxc-9202-2026_10_14-10_42_25.tar.dat"
|
||||
self.pair(lambda b: (b.partial(other), ("local", 9201, other)),
|
||||
lambda b: (b.partial(other), ("local", 9202, other)))
|
||||
|
||||
def test_partial_on_a_non_dir_storage(self):
|
||||
self.pair(lambda b: (b.partial(), ("local-lvm", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_partial_on_a_network_storage_with_a_path(self):
|
||||
# an nfs storage HAS a path: only the „dir only" rule (both its guards) can refuse it
|
||||
self.pair(lambda b: (b.partial(where=b.nas / "dump"), ("nas", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_partial_on_another_storage_than_named(self):
|
||||
self.pair(lambda b: (b.partial(where=b.other / "dump"), ("local", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_unknown_storage(self):
|
||||
self.pair(lambda b: (b.partial(), ("nosuch", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_vzdump_task_in_flight(self):
|
||||
def refuse(b):
|
||||
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzdump:9201:felhom-agent@pve!agent:"]); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzrestore:9202:root@pam:"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_any_vzdump_job_even_without_the_vmid(self):
|
||||
# a job for several guests: `vzdump::` — and a vzdump for ANOTHER guest — both refuse
|
||||
for cmd in (["task UPID:demo-hp:001:002:6ACF4071:vzdump::root@pam:"],
|
||||
["task UPID:demo-hp:001:002:6ACF4071:vzdump:9202:felhom-agent@pve!agent:"]):
|
||||
def refuse(b, c=cmd):
|
||||
p = b.partial(); b.process(c); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzstart:9202:root@pam:"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_vzdump_command_for_the_vmid(self):
|
||||
def refuse(b):
|
||||
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdump", "9201", "--storage", "local"]); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdumpfoo", "9201"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_file_held_open_by_an_orphan_writer(self):
|
||||
def refuse(b):
|
||||
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[p]); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[b.vz / "dump" / "something-else"]); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_tmp_dir_with_a_process_inside(self):
|
||||
def refuse(b):
|
||||
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=p / "etc"); return p, ("local", 9201, p.name)
|
||||
def control(b):
|
||||
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=b.vz); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, control)
|
||||
|
||||
def test_recently_written_after_boot(self):
|
||||
self.pair(lambda b: (b.partial(mtime=int(time.time()) - 10), ("local", 9201, NAME + ".tar.dat")),
|
||||
lambda b: (b.partial(mtime=int(time.time()) - 300), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_symlink(self):
|
||||
def refuse(b):
|
||||
real = b.partial(where=b.vz)
|
||||
link = b.vz / "dump" / (NAME + ".tar.dat")
|
||||
os.symlink(real, link)
|
||||
return link, ("local", 9201, link.name)
|
||||
# quiet=0: the link was made just now, so only the symlink rule may refuse it
|
||||
self.pair(refuse, lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")), quiet=0)
|
||||
|
||||
def test_near_miss_names(self):
|
||||
for bad in (NAME + ".tar.dat.bak", "vzdump-lxc-92011-2026_10_14-10_42_25.tar.dat", NAME + ".tar.da",
|
||||
"x" + NAME + ".tar.dat", NAME + ".TAR.DAT", "vzdump-lxc-9201-2026_10_14-10_42.tar.dat"):
|
||||
self.pair(lambda b, n=bad: (b.partial(n), ("local", 9201, n)),
|
||||
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
|
||||
|
||||
def test_path_traversal_and_bad_args(self):
|
||||
b = Box(); p = b.partial()
|
||||
for args in (("local", 9201, "../dump/" + p.name), ("local", 9201, ""), ("loc al", 9201, p.name),
|
||||
("local", "9201x", p.name), ("local", 12, p.name)):
|
||||
rc, out = b.run(*args)
|
||||
self.assertNotEqual(rc, 0, f"{args}: {out}")
|
||||
self.assertTrue(p.exists())
|
||||
|
||||
def test_wrong_kind(self):
|
||||
def refuse(b): # a FILE named .tmp
|
||||
p = b.vz / "dump" / (NAME + ".tmp"); p.write_bytes(b"x"); os.utime(p, (BOOT - 100, BOOT - 100)); return p, ("local", 9201, p.name)
|
||||
self.pair(refuse, lambda b: (b.partial(NAME + ".tmp"), ("local", 9201, NAME + ".tmp")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -241,3 +241,23 @@ func TestNewestArchiveTime_DistinctPhantomsEachAnnounced(t *testing.T) {
|
||||
t.Errorf("got %d rejection lines for 2 distinct phantoms across 3 polls, want 2:\n%s", n, buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// R-99 (`09` §3 decision 140): the WARN for a PBS phantom ends with the cleanup runbook, so whoever sees it knows the
|
||||
// one sanctioned way to remove it; a tiny archive on a dir storage is not a PBS phantom and gets no pointer.
|
||||
// RED-PROOF: drop the `msg += phantomCleanupPointer` line → "the PBS phantom WARN does not end with the runbook pointer".
|
||||
func TestRejectedArchiveWarnNamesTheCleanupRunbook(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
|
||||
if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const want = "INCOMPLETE archive when computing tier freshness — it is not a successful backup — a phantom leftover; delete it by felhom.eu documentation/runbooks/pbs-phantom-cleanup.md (09 §3 decision 140)"
|
||||
if !strings.Contains(buf.String(), want) {
|
||||
t.Errorf("the PBS phantom WARN does not end with the runbook pointer:\n%s", buf.String())
|
||||
}
|
||||
local := phantomEntry()
|
||||
local.Format, local.VolID = "tar.zst", "local:backup/vzdump-lxc-9201-2026_07_28-05_31_14.tar.zst"
|
||||
if got := rejectedArchiveMessage(local); strings.Contains(got, "pbs-phantom-cleanup") {
|
||||
t.Errorf("a dir-storage archive got the PBS runbook pointer: %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// BackupSuccessState persists the newest SUCCESSFUL whole-guest backup per tier and guest (R-894).
|
||||
//
|
||||
// Why it exists. The due-check (`localapi` handleBackupDue) asks the tier's storage when a backup last
|
||||
// landed (R-84) and falls back to the in-memory record when the storage cannot be read. The in-memory
|
||||
// record is empty after an agent restart (Store, R-348), so "storage unreadable" right after a restart
|
||||
// read as "no record — DUE". Measured 2026-10-05 on demo-hp: the agent restarted at 04:57, the off-site
|
||||
// storage answered "Can't connect" at 06:25, the 7-day tier — last copy 2026-10-01 — read DUE, the
|
||||
// controller asked, and vzdump failed. This file is the last known copy the fallback reads instead.
|
||||
//
|
||||
// It is read ONLY when the storage cannot be read. A storage that answers is the ground truth and wins,
|
||||
// in both directions: an archive found there counts, and an archive absent there is absent even when
|
||||
// this file remembers a success (a pruned or deleted archive must make the tier due — the same reason
|
||||
// R-84 chose the storage over a persisted record). Pinned by
|
||||
// TestBackupDue_R894_SavedCopyIgnoredWhenStorageAnswers.
|
||||
//
|
||||
// Only SUCCESSES are written (the RestoreTestState rule): a failure must stay due and be retried, so a
|
||||
// record of a failure has no reader.
|
||||
type BackupSuccessState struct {
|
||||
path string
|
||||
mu sync.Mutex
|
||||
last map[string]savedSuccess // key(target, vmid) → the newest success
|
||||
}
|
||||
|
||||
type savedSuccess struct {
|
||||
target string
|
||||
vmid int
|
||||
at time.Time
|
||||
}
|
||||
|
||||
// backupSuccessJSON is one entry on disk.
|
||||
type backupSuccessJSON struct {
|
||||
Target string `json:"target"`
|
||||
VMID int `json:"vmid"`
|
||||
StartedAt string `json:"started_at"`
|
||||
}
|
||||
|
||||
func backupStateKey(target string, vmid int) string { return target + "/" + strconv.Itoa(vmid) }
|
||||
|
||||
// NewBackupSuccessState opens (or creates) the state at path. A missing or unreadable file degrades to
|
||||
// "nothing known" — the pre-R-894 behaviour, which is DUE — and never wedges the daemon.
|
||||
func NewBackupSuccessState(path string) *BackupSuccessState {
|
||||
s := &BackupSuccessState{path: path, last: map[string]savedSuccess{}}
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return s
|
||||
}
|
||||
var entries []backupSuccessJSON
|
||||
if json.Unmarshal(data, &entries) != nil {
|
||||
return s
|
||||
}
|
||||
for _, e := range entries {
|
||||
t, perr := time.Parse(time.RFC3339, e.StartedAt)
|
||||
if perr != nil {
|
||||
continue // one unreadable entry must not lose the others
|
||||
}
|
||||
s.last[backupStateKey(e.Target, e.VMID)] = savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// RecordBackupSuccess saves b when it is a success newer than the one on file. target is the tier the
|
||||
// job ran on (the due-check's key); a failure or an unparseable time is ignored.
|
||||
func (s *BackupSuccessState) RecordBackupSuccess(target string, b hub.Backup) error {
|
||||
if s == nil || !b.Success {
|
||||
return nil
|
||||
}
|
||||
t, err := time.Parse(time.RFC3339, b.StartedAt)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
k := backupStateKey(target, b.VMID)
|
||||
if old, ok := s.last[k]; ok && !t.After(old.at) {
|
||||
return nil
|
||||
}
|
||||
s.last[k] = savedSuccess{target: target, vmid: b.VMID, at: t.UTC()}
|
||||
return s.saveLocked()
|
||||
}
|
||||
|
||||
// LastKnownSuccess returns the newest saved success for this tier and guest (ok=false = none on file).
|
||||
func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Time, bool) {
|
||||
if s == nil {
|
||||
return time.Time{}, false
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
e, ok := s.last[backupStateKey(target, vmid)]
|
||||
return e.at, ok
|
||||
}
|
||||
|
||||
// KnownBackupSuccesses returns every saved success as a host-report record (the collector's
|
||||
// KnownBackupReporter, so the hub keeps its evidence across a restart). Only the tier, guest and start
|
||||
// time are known; the archive name is not saved and stays empty. Sorted for a stable report.
|
||||
func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := make([]hub.Backup, 0, len(s.last))
|
||||
for _, e := range s.last {
|
||||
out = append(out, hub.Backup{TargetID: e.target, VMID: e.vmid, Success: true, CrashConsistent: true,
|
||||
StartedAt: e.at.Format(time.RFC3339), UncoveredVolumes: []string{}})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].TargetID != out[j].TargetID {
|
||||
return out[i].TargetID < out[j].TargetID
|
||||
}
|
||||
return out[i].VMID < out[j].VMID
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *BackupSuccessState) saveLocked() error {
|
||||
entries := make([]backupSuccessJSON, 0, len(s.last))
|
||||
for _, e := range s.last {
|
||||
entries = append(entries, backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)})
|
||||
}
|
||||
// Deterministic file content (Go's map order is random).
|
||||
sort.Slice(entries, func(i, j int) bool {
|
||||
if entries[i].Target != entries[j].Target {
|
||||
return entries[i].Target < entries[j].Target
|
||||
}
|
||||
return entries[i].VMID < entries[j].VMID
|
||||
})
|
||||
data, err := json.MarshalIndent(entries, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := s.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, s.path)
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-894: the on-disk newest success per tier survives a restart (a new state from the same file).
|
||||
func TestBackupSuccessState_SurvivesRestart(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
s := NewBackupSuccessState(path)
|
||||
at := time.Date(2026, 10, 1, 20, 15, 0, 0, time.UTC)
|
||||
if err := s.RecordBackupSuccess("felhom-pbs", hub.Backup{VMID: 9201, Success: true, StartedAt: at.Format(time.RFC3339)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, ok := NewBackupSuccessState(path).LastKnownSuccess("felhom-pbs", 9201)
|
||||
if !ok || !got.Equal(at) {
|
||||
t.Fatalf("after a restart the saved copy must read back; got %v ok=%v", got, ok)
|
||||
}
|
||||
if _, ok := NewBackupSuccessState(path).LastKnownSuccess("local", 9201); ok {
|
||||
t.Fatal("another tier must not borrow this tier's copy")
|
||||
}
|
||||
}
|
||||
|
||||
// Only a NEWER success replaces the saved one; failures and unparseable times are ignored.
|
||||
func TestBackupSuccessState_KeepsNewestSuccessOnly(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "s.json")
|
||||
s := NewBackupSuccessState(path)
|
||||
newer := time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
|
||||
older := newer.Add(-48 * time.Hour)
|
||||
for _, b := range []hub.Backup{
|
||||
{VMID: 1, Success: true, StartedAt: newer.Format(time.RFC3339)},
|
||||
{VMID: 1, Success: true, StartedAt: older.Format(time.RFC3339)}, // older: ignored
|
||||
{VMID: 1, Success: false, StartedAt: newer.Add(time.Hour).Format(time.RFC3339)}, // failure: ignored
|
||||
{VMID: 1, Success: true, StartedAt: "not-a-time"}, // unparseable: ignored
|
||||
} {
|
||||
if err := s.RecordBackupSuccess("t", b); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if got, _ := NewBackupSuccessState(path).LastKnownSuccess("t", 1); !got.Equal(newer) {
|
||||
t.Fatalf("want the newest success %v, got %v", newer, got)
|
||||
}
|
||||
}
|
||||
|
||||
// A corrupt file degrades to "nothing known" (the pre-R-894 DUE answer), never a crash.
|
||||
func TestBackupSuccessState_CorruptFileIsEmpty(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "s.json")
|
||||
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, ok := NewBackupSuccessState(path).LastKnownSuccess("t", 1); ok {
|
||||
t.Fatal("a corrupt file must read as nothing known")
|
||||
}
|
||||
}
|
||||
|
||||
// The saved successes come back as host-report records: success, tier, guest, start time (R-894 → the
|
||||
// host report, 2026-10-09). A new state read from the same file gives the same records (the restart case).
|
||||
func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
s := NewBackupSuccessState(path)
|
||||
if err := s.RecordBackupSuccess("felhom-backup", hub.Backup{VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := NewBackupSuccessState(path).KnownBackupSuccesses()
|
||||
if len(got) != 1 || !got[0].Success || got[0].TargetID != "felhom-backup" || got[0].VMID != 9201 || got[0].StartedAt != "2026-10-09T02:40:16Z" {
|
||||
t.Fatalf("got %+v", got)
|
||||
}
|
||||
if got[0].UncoveredVolumes == nil {
|
||||
t.Fatal("uncovered_volumes must marshal as [], not null")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// ForeignKeyLedger (R-366 slice 2, `09` §3 decision 168) holds, per backup tier, the whole-guest archives the
|
||||
// restore-test pick skipped because another key wrote them (R-727 — an earlier install of this box). Since R-727 the
|
||||
// skip was one INFO log line per archive and nothing else, so after a reinstall the operator was never told that the
|
||||
// box's older whole-guest copies are unreadable to it. The host report carries this ledger; the hub raises ONE
|
||||
// operator event when it changes.
|
||||
//
|
||||
// It reports nil until a tier has been evaluated since the agent started, so a restart does not read as "the set
|
||||
// changed to empty" (the hub keeps its last state for an absent field).
|
||||
type ForeignKeyLedger struct {
|
||||
mu sync.Mutex
|
||||
byTarget map[string]hub.ForeignKeyArchives
|
||||
}
|
||||
|
||||
// NewForeignKeyLedger builds an empty ledger.
|
||||
func NewForeignKeyLedger() *ForeignKeyLedger {
|
||||
return &ForeignKeyLedger{}
|
||||
}
|
||||
|
||||
func (l *ForeignKeyLedger) set(target string, n int, oldest, newest int64) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if l.byTarget == nil {
|
||||
l.byTarget = map[string]hub.ForeignKeyArchives{}
|
||||
}
|
||||
e := hub.ForeignKeyArchives{Target: target, Count: n}
|
||||
if n > 0 {
|
||||
e.Oldest = time.Unix(oldest, 0).UTC().Format(time.RFC3339)
|
||||
e.Newest = time.Unix(newest, 0).UTC().Format(time.RFC3339)
|
||||
}
|
||||
l.byTarget[target] = e
|
||||
}
|
||||
|
||||
// ForeignKeyArchives implements hub.ForeignKeyArchiveReporter: nil before any evaluation; otherwise the tiers that
|
||||
// hold such archives (`Tiers` empty, never nil, when none do), sorted by tier.
|
||||
func (l *ForeignKeyLedger) ForeignKeyArchives(context.Context) *hub.ForeignKeyArchivesStanza {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if l.byTarget == nil {
|
||||
return nil
|
||||
}
|
||||
out := []hub.ForeignKeyArchives{}
|
||||
for _, e := range l.byTarget {
|
||||
if e.Count > 0 {
|
||||
out = append(out, e)
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Target < out[j].Target })
|
||||
return &hub.ForeignKeyArchivesStanza{Tiers: out}
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-366 slice 2 (`09` §3 decision 168) — the restore-test's skip of an archive written with another key stops being
|
||||
// silent: the pick records, per tier, how many it skipped and their time range, and the host report carries it.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): remove the `r.foreign.set(...)` call from PickSettledRestoreCandidateOn → this fails
|
||||
// with "after one evaluation the ledger must report felhom-pbs: 2 archives …; got []". Restored.
|
||||
func TestR366_PickRecordsArchivesWrittenWithAnotherKey(t *testing.T) {
|
||||
api := &fakeBackupAPI{
|
||||
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
|
||||
content: []proxmox.StorageContent{
|
||||
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
|
||||
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
|
||||
{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey},
|
||||
},
|
||||
}
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
l := NewForeignKeyLedger()
|
||||
r.SetForeignKeyLedger(l)
|
||||
|
||||
if got := l.ForeignKeyArchives(context.Background()); got != nil {
|
||||
t.Fatalf("before any evaluation the ledger must be nil (the hub keeps its state); got %v", got)
|
||||
}
|
||||
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st := l.ForeignKeyArchives(context.Background())
|
||||
want := hub.ForeignKeyArchives{Target: "felhom-pbs", Count: 2, Oldest: "2026-09-16T17:27:32Z", Newest: "2026-09-16T21:59:54Z"}
|
||||
var got []hub.ForeignKeyArchives
|
||||
if st != nil {
|
||||
got = st.Tiers
|
||||
}
|
||||
if len(got) != 1 || got[0] != want {
|
||||
t.Fatalf("after one evaluation the ledger must report felhom-pbs: 2 archives 2026-09-16T17:27:32Z…21:59:54Z; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Evaluated and none found → the stanza with `tiers: []`; not evaluated → no stanza at all. Never a null on the wire.
|
||||
func TestR366_EvaluatedWithNoneIsAnEmptyList(t *testing.T) {
|
||||
api := &fakeBackupAPI{
|
||||
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
|
||||
content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}},
|
||||
}
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
l := NewForeignKeyLedger()
|
||||
r.SetForeignKeyLedger(l)
|
||||
b, _ := json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
|
||||
if strings.Contains(string(b), "foreign_key_archives") {
|
||||
t.Fatalf("not evaluated must omit the stanza; got %s", b)
|
||||
}
|
||||
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ = json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
|
||||
if !strings.Contains(string(b), `"foreign_key_archives":{"tiers":[]}`) {
|
||||
t.Fatalf("evaluated with none must report tiers: []; got %s", b)
|
||||
}
|
||||
}
|
||||
@@ -81,3 +81,28 @@ func TestR685_FailsOpen(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestR1006_LeftoverCleanupRunsBeforeThePreflight — the night of 2026-10-14: a 12.2 GB half-written archive from a power
|
||||
// cut held the space, so every later backup was skipped. The cleanup step runs BEFORE the preflight reads the free space:
|
||||
// here it „removes" the leftover (free 4 → 16 GiB) and the backup then fits and runs.
|
||||
//
|
||||
// RED-PROOF: drop the r.beforeBackup call from backup() (or move it after spaceFits) → the backup is skipped → fails.
|
||||
func TestR1006_LeftoverCleanupRunsBeforeThePreflight(t *testing.T) {
|
||||
api := spaceAPI(4*gib, 8182759056, "dir")
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
var got []string
|
||||
r.SetBeforeBackup(func(_ context.Context, storage string, vmid int) {
|
||||
got = append(got, storage)
|
||||
if vmid != 9201 {
|
||||
t.Errorf("cleanup asked for vmid %d", vmid)
|
||||
}
|
||||
api.storages[0].Avail = 16 * gib
|
||||
})
|
||||
_, _ = r.Backup(context.Background(), 9201)
|
||||
if len(got) != 1 || got[0] != "local" {
|
||||
t.Fatalf("the cleanup must run once for this target, got %v", got)
|
||||
}
|
||||
if len(api.vzdumps) != 1 {
|
||||
t.Fatalf("after the cleanup freed the space the backup must run, vzdumps=%d", len(api.vzdumps))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -58,16 +58,24 @@ type BackupRunner struct {
|
||||
// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
|
||||
// there is a deliberate act.
|
||||
allowPBSPrune bool
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// beforeBackup runs just before the space preflight (R-1006): it removes a dead half-written vzdump of this guest on
|
||||
// this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing.
|
||||
beforeBackup func(ctx context.Context, storage string, vmid int)
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
|
||||
// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
|
||||
// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
|
||||
// due-check is served from the local-API handler goroutines.
|
||||
rejectedMu sync.Mutex
|
||||
rejected map[string]struct{}
|
||||
// foreign (R-366 slice 2) records, per tier, the archives the pick skipped as another key's. nil = not wired.
|
||||
foreign *ForeignKeyLedger
|
||||
}
|
||||
|
||||
// SetForeignKeyLedger wires the R-366 slice-2 ledger the host report reads.
|
||||
func (r *BackupRunner) SetForeignKeyLedger(l *ForeignKeyLedger) { r.foreign = l }
|
||||
|
||||
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
|
||||
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
|
||||
// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
|
||||
@@ -97,6 +105,11 @@ func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode,
|
||||
now: func() time.Time { return time.Now().UTC() }}
|
||||
}
|
||||
|
||||
// SetBeforeBackup sets the step run just before the space preflight (R-1006: the dead-leftover cleanup).
|
||||
func (r *BackupRunner) SetBeforeBackup(fn func(ctx context.Context, storage string, vmid int)) {
|
||||
r.beforeBackup = fn
|
||||
}
|
||||
|
||||
// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
|
||||
// applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a
|
||||
// separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type
|
||||
@@ -178,6 +191,9 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
|
||||
// R-685 (v0.134.0): will the new archive FIT on a local target? Asked before anything runs, so a
|
||||
// target that cannot hold it is a named SKIP with the numbers, not a nightly "No space left on
|
||||
// device" that only the vzdump log explains (demo-hp, every night from 2026-09-23 — R-684).
|
||||
if r.beforeBackup != nil {
|
||||
r.beforeBackup(ctx, r.target, vmid)
|
||||
}
|
||||
if ok, why := r.spaceFits(ctx, vmid); !ok {
|
||||
rec.Error = BackupSkipNoSpacePrefix + why
|
||||
rec.DurationSeconds = time.Since(start).Seconds()
|
||||
@@ -370,6 +386,8 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
|
||||
var best string
|
||||
var bestCTime int64 = -1
|
||||
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
|
||||
var foreignN int // R-366 slice 2: archives skipped as another key's, and their time range
|
||||
var foreignMin, foreignMax int64
|
||||
for _, e := range contents {
|
||||
if e.Content != "backup" {
|
||||
continue
|
||||
@@ -384,6 +402,13 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
|
||||
}
|
||||
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
|
||||
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
|
||||
foreignN++
|
||||
if foreignMin == 0 || e.CTime < foreignMin {
|
||||
foreignMin = e.CTime
|
||||
}
|
||||
if e.CTime > foreignMax {
|
||||
foreignMax = e.CTime
|
||||
}
|
||||
continue
|
||||
}
|
||||
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
|
||||
@@ -420,6 +445,9 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
|
||||
bestCTime, best = e.CTime, e.VolID
|
||||
}
|
||||
}
|
||||
if r.foreign != nil {
|
||||
r.foreign.set(target, foreignN, foreignMin, foreignMax)
|
||||
}
|
||||
if best == "" {
|
||||
return "", time.Time{}, nil
|
||||
}
|
||||
@@ -518,10 +546,25 @@ func (r *BackupRunner) warnRejectedArchiveOnce(e proxmox.StorageContent, why str
|
||||
if seen {
|
||||
return
|
||||
}
|
||||
r.logger.Warn("backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup",
|
||||
r.logger.Warn(rejectedArchiveMessage(e),
|
||||
"target", r.target, "vmid", e.VMID, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
|
||||
}
|
||||
|
||||
// phantomCleanupPointer names the runbook that removes a PBS phantom (R-99, `09` §3 decision 140: a leftover of an
|
||||
// aborted upload is deleted on the backup server, by a runbook, when one is seen — never automatically).
|
||||
const phantomCleanupPointer = " — a phantom leftover; delete it by felhom.eu documentation/runbooks/pbs-phantom-cleanup.md (09 §3 decision 140)"
|
||||
|
||||
// rejectedArchiveMessage is the WARN text for a rejected archive. Only a PBS entry (format pbs-ct / pbs-vm) gets the
|
||||
// cleanup pointer: the runbook deletes on a PBS datastore, and a tiny archive on a dir storage is not a PBS phantom.
|
||||
// Pinned by TestRejectedArchiveWarnNamesTheCleanupRunbook.
|
||||
func rejectedArchiveMessage(e proxmox.StorageContent) string {
|
||||
msg := "backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup"
|
||||
if strings.HasPrefix(e.Format, "pbs-") {
|
||||
msg += phantomCleanupPointer
|
||||
}
|
||||
return msg
|
||||
}
|
||||
|
||||
// demo-felhom in a single afternoon of deploys (2026-07-26).
|
||||
//
|
||||
// Asking the STORAGE rather than persisting the store is deliberate:
|
||||
|
||||
@@ -30,6 +30,9 @@ import (
|
||||
// 2026-08-20, two consecutive host-reports with `0 backups` while `pvesm list` showed archives on both tiers. What
|
||||
// is unaffected is the hub's VERDICT: it looks back 7 days over stored reports (felhom.eu hub/internal/monitor/
|
||||
// deadline.go backupEvidenceLookback) and the storage stays the ground truth (R-84).
|
||||
// The due-check's fallback for an UNREADABLE storage no longer reads this store alone (R-894): the newest
|
||||
// success per tier is also on disk (BackupSuccessState), so a restart followed by an unreachable storage
|
||||
// reads the last known copy, not "never".
|
||||
type Store struct {
|
||||
mu sync.Mutex
|
||||
byTarget map[string]hub.Backup // latest backup per target id
|
||||
|
||||
@@ -115,6 +115,11 @@ var manifest = []Capability{
|
||||
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
|
||||
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
|
||||
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
|
||||
{"guest-shutdown", "household power-off: guest shutdown (R-998)", "/usr/sbin/pct", []string{"shutdown", "9201", "--timeout", "120"}, false, ""},
|
||||
{"host-poweroff", "household power-off: host power-off (R-998)", "/usr/bin/systemctl", []string{"poweroff"}, false, ""},
|
||||
{"vzdump-partial-clean", "dead half-written vzdump removal (R-1006)", "/usr/local/sbin/felhom-vzdump-partial-clean", []string{"local", "9201", "vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat"}, false, ""},
|
||||
{"internal-place-plan", "internal data place: the size plan (16 §7)", "/usr/local/sbin/felhom-internal-place", []string{"plan", "0"}, false, ""},
|
||||
{"internal-place-create", "internal data place: create on an old box (signed add_internal_place)", "/usr/local/sbin/felhom-internal-place", []string{"create", "0"}, false, ""},
|
||||
|
||||
// ---- LAN split-horizon resolver (dnsmasq) ----
|
||||
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
|
||||
@@ -145,12 +150,17 @@ var manifest = []Capability{
|
||||
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
|
||||
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
|
||||
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
|
||||
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "tee", "/etc/felhom-controller-image"}, true, ""},
|
||||
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
|
||||
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
|
||||
|
||||
// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
|
||||
// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
|
||||
{"stalelock-unlock", "reboot-during-backup stale-lock recovery", "/usr/sbin/pct", []string{"unlock", "9201"}, true, ""},
|
||||
|
||||
// ---- Weekly guest disk trim (FELHOM_FSTRIM, R-444). NON-critical: a missing grant means the thin pool is not
|
||||
// reclaimed this week (the trim job WARNs per guest and the report shows the failure), not a serving outage. ----
|
||||
{"guest-fstrim", "weekly guest disk trim (thin-pool reclaim, R-444)", "/usr/sbin/pct", []string{"fstrim", "9201"}, false, ""},
|
||||
|
||||
// ---- Offsite WG tunnel (FELHOM_WG, S3/v0.64.0; Critical FLIPPED in S4/v0.66.0 — offsite
|
||||
// backups now RIDE the tunnel, so a degraded tunnel capability is operator-alert-worthy: the
|
||||
// conf install, unit enable/restart and the handshake read gate the backup path. apt-install
|
||||
|
||||
@@ -200,7 +200,8 @@ func TestRedProof_DroppedGrantFailsCheck(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0
|
||||
// FELHOM_CONTROLLERSWAP grants: with the `tee /etc/felhom-controller-image` line removed, the
|
||||
// FELHOM_CONTROLLERSWAP grants: with the write grant removed (since R-861 (a) A1 the `felhom-priv-apply controller-image`
|
||||
// line; before it, an agent `tee /etc/felhom-controller-image`), the
|
||||
// controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new
|
||||
// swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update).
|
||||
func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
|
||||
@@ -210,7 +211,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
|
||||
}
|
||||
var kept []string
|
||||
for _, ln := range strings.Split(string(data), "\n") {
|
||||
if strings.Contains(ln, "tee /etc/felhom-controller-image") {
|
||||
if strings.Contains(ln, "felhom-priv-apply ^controller-image") { // R-861 (a) A1: the write's grant
|
||||
continue
|
||||
}
|
||||
kept = append(kept, ln)
|
||||
@@ -229,7 +230,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
|
||||
}
|
||||
cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ")
|
||||
if matchesAny(cmdline, entries) {
|
||||
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping the tee grant")
|
||||
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping its grant")
|
||||
}
|
||||
if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) {
|
||||
t.Errorf("controllerswap-write should be covered by the real sudoers")
|
||||
@@ -310,3 +311,39 @@ func TestWGCapabilityCriticality(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEveryAliasIsGranted closes a gap found while adding FELHOM_POWEROFF (R-998): the coverage check above reads
|
||||
// every Cmnd_Alias body whether or not the alias appears on the grant line, so a manifest command whose alias was
|
||||
// never GRANTED still passed — and failed with „a password is required" on the box. Every defined alias must be
|
||||
// named on the `felhom-agent ALL=(root) NOPASSWD:` line. Red-proof: drop FELHOM_POWEROFF from that line.
|
||||
func TestEveryAliasIsGranted(t *testing.T) {
|
||||
data, err := os.ReadFile(sudoersPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read sudoers: %v", err)
|
||||
}
|
||||
var defined []string
|
||||
granted := map[string]bool{}
|
||||
grantLines := 0
|
||||
for _, ln := range strings.Split(string(data), "\n") {
|
||||
tr := strings.TrimSpace(ln)
|
||||
if strings.HasPrefix(tr, "Cmnd_Alias ") {
|
||||
if f := strings.Fields(tr); len(f) >= 2 {
|
||||
defined = append(defined, f[1])
|
||||
}
|
||||
}
|
||||
if strings.HasPrefix(tr, "felhom-agent ") && strings.Contains(tr, "NOPASSWD:") {
|
||||
grantLines++
|
||||
for _, a := range strings.Split(tr[strings.Index(tr, "NOPASSWD:")+len("NOPASSWD:"):], ",") {
|
||||
granted[strings.TrimSpace(a)] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if grantLines == 0 || len(defined) < 10 {
|
||||
t.Fatalf("parsed %d grant line(s), %d aliases — parser broke", grantLines, len(defined))
|
||||
}
|
||||
for _, a := range defined {
|
||||
if !granted[a] {
|
||||
t.Errorf("Cmnd_Alias %s is defined but never granted to felhom-agent", a)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -49,6 +49,34 @@ var r861Injections = []string{
|
||||
"/usr/local/sbin/felhom-priv-apply unit ../../etc/x.mount",
|
||||
"/usr/local/sbin/felhom-priv-apply dnsmasq /etc/shadow felhom-x.conf",
|
||||
"/usr/local/sbin/felhom-priv-apply wg /etc/shadow",
|
||||
// R-861 (a) A1 (decision 165): the agent wrote ANY image ref into the guest by `tee` — now only the root verb may
|
||||
"/usr/sbin/pct exec 9201 -- tee /etc/felhom-controller-image",
|
||||
"/usr/local/sbin/felhom-priv-apply controller-image 9201 9202",
|
||||
"/usr/local/sbin/felhom-priv-apply controller-image 9201;id",
|
||||
// R-998 (host power-off): the two new grants take exactly their fixed shape — no extra option, no other target
|
||||
"/usr/sbin/pct shutdown 9201 --timeout 120 --forceStop 1",
|
||||
"/usr/sbin/pct shutdown 9201 --forceStop 1 --timeout 120",
|
||||
"/usr/sbin/pct shutdown 9201",
|
||||
"/usr/sbin/pct shutdown 9201 --timeout -1",
|
||||
"/usr/bin/systemctl poweroff --force",
|
||||
"/usr/bin/systemctl poweroff -ff",
|
||||
"/usr/bin/systemctl reboot",
|
||||
"/usr/bin/systemctl halt",
|
||||
"/usr/bin/systemctl kexec",
|
||||
// R-1006: the partial-cleanup helper takes exactly one storage, one vmid, one half-written name.
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.zst",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.log",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 ../../etc/vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat /etc/shadow",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat;id",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local;id 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201",
|
||||
"/usr/local/sbin/felhom-vzdump-partial-clean --help",
|
||||
"/usr/local/sbin/felhom-internal-place create 132",
|
||||
"/usr/local/sbin/felhom-internal-place create 0 extra",
|
||||
"/usr/local/sbin/felhom-internal-place create 0;id",
|
||||
"/usr/local/sbin/felhom-internal-place wipe 0",
|
||||
"/usr/local/sbin/felhom-internal-place create",
|
||||
}
|
||||
|
||||
func TestSudoersRefusesTheR861Injections(t *testing.T) {
|
||||
@@ -63,3 +91,72 @@ func TestSudoersRefusesTheR861Injections(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// R-444: the weekly trim's grant is ONE exact shape — `pct fstrim <vmid>` — and nothing smuggled after it.
|
||||
// The manifest entry (guest-fstrim) proves the real call is still allowed (TestManifestCoveredBySudoers); this
|
||||
// pins the other direction. RED-PROOF: write the rule as the glob `/usr/sbin/pct fstrim [0-9]*` → every decoy
|
||||
// below with a trailing argument matches (the glob's `*` eats spaces).
|
||||
func TestSudoersFstrimRuleIsExact(t *testing.T) {
|
||||
data, err := os.ReadFile(sudoersPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries := parseSudoersEntries(t, string(data))
|
||||
if !matchesAny("/usr/sbin/pct fstrim 9201", entries) {
|
||||
t.Fatal("the sudoers does not allow `pct fstrim 9201` — the weekly trim cannot run")
|
||||
}
|
||||
for _, c := range []string{
|
||||
"/usr/sbin/pct fstrim 9201 --ignore-mountpoints",
|
||||
"/usr/sbin/pct fstrim 9201 --ignore-mountpoints 1",
|
||||
"/usr/sbin/pct fstrim 9201; x",
|
||||
"/usr/sbin/pct fstrim 9201 9202",
|
||||
"/usr/sbin/pct fstrim 92a1",
|
||||
"/usr/sbin/pct fstrim ",
|
||||
"/usr/sbin/pct fstrim -- 9201",
|
||||
"/usr/sbin/pct destroy 9201",
|
||||
"/usr/sbin/pct destroy 9201 --purge",
|
||||
} {
|
||||
if matchesAny(c, entries) {
|
||||
t.Errorf("the sudoers allows a command the trim rule must not: %q", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// R-861 (a) A1: the managed controller update still has its route — the root verb, one numeric vmid.
|
||||
func TestSudoersAllowsTheControllerImageVerb(t *testing.T) {
|
||||
data, err := os.ReadFile(sudoersPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !matchesAny("/usr/local/sbin/felhom-priv-apply controller-image 9201", parseSudoersEntries(t, string(data))) {
|
||||
t.Fatal("the sudoers does not allow `felhom-priv-apply controller-image 9201` — a managed controller update cannot write its image")
|
||||
}
|
||||
}
|
||||
|
||||
// R-861 (b) B2 (decision 165, hygiene): felhom-op's `pct start|stop|unlock` grants are ONE numeric vmid each. The old
|
||||
// glob `[0-9]*` eats spaces, so `pct stop 9201 --skiplock 1` and two vmids matched.
|
||||
// RED-PROOF: on the pre-B2 felhom-op.sudoers (`/usr/sbin/pct stop [0-9]*`) the decoys match.
|
||||
func TestFelhomOpSudoersPctIsExact(t *testing.T) {
|
||||
data, err := os.ReadFile("../../configs/felhom-op.sudoers")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries := parseSudoersEntries(t, string(data))
|
||||
for _, ok := range []string{"/usr/sbin/pct start 9201", "/usr/sbin/pct stop 9201", "/usr/sbin/pct unlock 9201", "/usr/sbin/pct list"} {
|
||||
if !matchesAny(ok, entries) {
|
||||
t.Errorf("felhom-op lost a repair verb: %s", ok)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{
|
||||
"/usr/sbin/pct stop 9201 --skiplock 1",
|
||||
"/usr/sbin/pct start 9201 9202",
|
||||
"/usr/sbin/pct unlock 9201 --whatever",
|
||||
"/usr/sbin/pct start 92a1",
|
||||
"/usr/sbin/pct stop ",
|
||||
"/usr/sbin/pct destroy 9201",
|
||||
} {
|
||||
if matchesAny(bad, entries) {
|
||||
t.Errorf("felhom-op's sudoers allows %q", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -33,6 +33,7 @@ type Config struct {
|
||||
LANResolver LANResolverConfig `json:"lan_resolver"`
|
||||
WGTunnel WGTunnelConfig `json:"wg_tunnel"`
|
||||
GuestNet GuestNetConfig `json:"guest_net"`
|
||||
DiskTrim DiskTrimConfig `json:"disk_trim"`
|
||||
OOB OOBConfig `json:"oob"`
|
||||
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
|
||||
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
|
||||
@@ -139,6 +140,16 @@ func (w WGTunnelConfig) WithDefaults() WGTunnelConfig {
|
||||
return w
|
||||
}
|
||||
|
||||
// DiskTrimConfig configures the R-444 weekly guest disk trim (internal/fstrim). DEFAULT-ON, like GuestNetConfig and for
|
||||
// the same reason: it only acts on guests the agent already owns, and the operator ruled every box trims (`09` §3
|
||||
// decision 139). Opting out is the explicit act: `"disk_trim": {"disable": true}`.
|
||||
type DiskTrimConfig struct {
|
||||
Disable bool `json:"disable"`
|
||||
}
|
||||
|
||||
// Enabled reports whether the weekly trim should run.
|
||||
func (d DiskTrimConfig) Enabled() bool { return !d.Disable }
|
||||
|
||||
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
|
||||
//
|
||||
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
|
||||
|
||||
+50
-10
@@ -59,8 +59,26 @@ var (
|
||||
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
|
||||
// which is the one fact the customer needs to recognise it.
|
||||
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
|
||||
// ErrRetainedUnchecked — the code did NOT open the current package, and NOT every earlier package the hub
|
||||
// holds for this host was tried (R-304, 2026-10-08): the retained list could not be fetched, the hub
|
||||
// withheld rows (over its serve cap, or rows with no key material), a package was malformed, or the
|
||||
// attempt cap stopped the loop. So „the code is wrong" is NOT known — it may be right for a package
|
||||
// nobody tried. Distinct from a mistype for exactly the R-224 reason: never accuse the customer of
|
||||
// something we did not check.
|
||||
ErrRetainedUnchecked = errors.New("escrow: the recovery code did not open the current sealed package, and some earlier packages were NOT checked")
|
||||
)
|
||||
|
||||
// RetainedUncheckedError wraps ErrRetainedUnchecked with how many earlier packages went unchecked
|
||||
// (-1 = unknown: the retained list itself could not be read). No secret.
|
||||
type RetainedUncheckedError struct {
|
||||
Unchecked int
|
||||
}
|
||||
|
||||
func (e *RetainedUncheckedError) Error() string {
|
||||
return fmt.Sprintf("%s (unchecked=%d)", ErrRetainedUnchecked.Error(), e.Unchecked)
|
||||
}
|
||||
func (e *RetainedUncheckedError) Unwrap() error { return ErrRetainedUnchecked }
|
||||
|
||||
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
|
||||
// carries no secret — not the code, not the bundle, not the repository password.
|
||||
type RetainedMatch struct {
|
||||
@@ -102,8 +120,10 @@ type RetainedBlob struct {
|
||||
}
|
||||
|
||||
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
|
||||
// slice is a clean "none". R-311.
|
||||
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error)
|
||||
// slice is a clean "none". R-311. `withheld` counts the earlier packages the hub holds that are NOT in
|
||||
// blobs — rows with no key material, rows over the hub's serve cap, and packages dropped as malformed
|
||||
// (R-304): each is a package the code was never tried against.
|
||||
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, withheld int, err error)
|
||||
|
||||
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
||||
type OffsiteKeyRecoverer struct {
|
||||
@@ -154,9 +174,14 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
|
||||
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
|
||||
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
|
||||
// about our own incuriosity, read by the customer as a statement about their code.
|
||||
if m, ok := r.tryRetained(ctx, recoveryCode); ok {
|
||||
m, ok, unchecked := r.tryRetained(ctx, recoveryCode)
|
||||
if ok {
|
||||
return "", &RetainedOpenedError{Match: m}
|
||||
}
|
||||
// R-304: only when EVERY earlier package the hub holds was tried may this stay a wrong code.
|
||||
if unchecked != 0 {
|
||||
return "", &RetainedUncheckedError{Unchecked: unchecked}
|
||||
}
|
||||
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
||||
}
|
||||
if bundle.ResticRepoPassword == "" {
|
||||
@@ -173,23 +198,38 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
|
||||
// function breaking is the behaviour we had before it existed.
|
||||
//
|
||||
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
|
||||
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) {
|
||||
//
|
||||
// R-304 (2026-10-08): it also returns how many earlier packages were NOT tried — `withheld` from the hub, plus
|
||||
// the ones past the attempt cap — or -1 when the retained list could not be read at all. A nil FetchRetained
|
||||
// (an agent wired without the lookup) reports 0: the pre-R-311 refusal, unchanged.
|
||||
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool, int) {
|
||||
if r.FetchRetained == nil {
|
||||
return RetainedMatch{}, false
|
||||
return RetainedMatch{}, false, 0
|
||||
}
|
||||
blobs, _, err := r.FetchRetained(ctx)
|
||||
if err != nil || len(blobs) == 0 {
|
||||
return RetainedMatch{}, false
|
||||
blobs, withheld, err := r.FetchRetained(ctx)
|
||||
if err != nil {
|
||||
return RetainedMatch{}, false, -1
|
||||
}
|
||||
if withheld < 0 {
|
||||
withheld = 0
|
||||
}
|
||||
if len(blobs) == 0 {
|
||||
return RetainedMatch{}, false, withheld
|
||||
}
|
||||
limit := r.MaxRetainedTried
|
||||
if limit <= 0 {
|
||||
limit = defaultMaxRetainedTried
|
||||
}
|
||||
unchecked := withheld
|
||||
if len(blobs) > limit {
|
||||
unchecked += len(blobs) - limit
|
||||
}
|
||||
for i, rb := range blobs {
|
||||
if i >= limit {
|
||||
break
|
||||
}
|
||||
if len(rb.Blob) == 0 {
|
||||
unchecked++
|
||||
continue
|
||||
}
|
||||
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
|
||||
@@ -204,7 +244,7 @@ func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode strin
|
||||
// correct and must be told so — but the history behind it still cannot be reopened, and
|
||||
// saying otherwise would be a promise this path cannot keep.
|
||||
HasResticPassword: bundle.ResticRepoPassword != "",
|
||||
}, true
|
||||
}, true, 0
|
||||
}
|
||||
return RetainedMatch{}, false
|
||||
return RetainedMatch{}, false, unchecked
|
||||
}
|
||||
|
||||
@@ -115,8 +115,9 @@ func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The
|
||||
// worst outcome of this feature breaking is the behaviour we had before it.
|
||||
// FAIL-SAFE — if the retained lookup itself fails, the lookup's error never reaches the customer and no
|
||||
// retained package is claimed. R-304 (2026-10-08) changed what stands instead: not the wrong-code refusal (the
|
||||
// earlier packages were never tried, so „wrong" is not known) but ErrRetainedUnchecked with Unchecked = -1.
|
||||
//
|
||||
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
|
||||
// gets a new, unexplained failure mode → this FAILS.
|
||||
@@ -140,6 +141,10 @@ func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
|
||||
if containsStr(err.Error(), "hub exploded") {
|
||||
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
|
||||
}
|
||||
var ue *RetainedUncheckedError
|
||||
if !errors.As(err, &ue) || ue.Unchecked != -1 {
|
||||
t.Errorf("err = %v, want RetainedUncheckedError{-1} — the earlier packages were never tried (R-304)", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
|
||||
@@ -228,3 +233,72 @@ func containsStr(hay, needle string) bool {
|
||||
return false
|
||||
})()
|
||||
}
|
||||
|
||||
// ── R-304 (2026-10-08) — „wrong code" only when every earlier package was tried ──────────────────────
|
||||
//
|
||||
// The consequence asserted: is the refusal a WRONG CODE (the only error the screen may answer with „check your
|
||||
// typing")? It may be only when the hub withheld nothing and every served package was tried.
|
||||
//
|
||||
// RED-PROOF: make tryRetained return 0 for `unchecked` → the three „unchecked" cases return the plain refusal
|
||||
// → they FAIL; the all-tried case keeps passing.
|
||||
func TestR304_WrongCodeOnlyWhenEveryEarlierPackageWasTried(t *testing.T) {
|
||||
ensureAge(t)
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "4444567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
other := sealBundle(t, IdentityBundle{ResticRepoPassword: "5555567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
const code = "a code that opens nothing whatsoever in this test"
|
||||
cases := []struct {
|
||||
name string
|
||||
blobs int
|
||||
withheld int
|
||||
limit int
|
||||
unchecked int // 0 = must be a plain wrong code
|
||||
}{
|
||||
{"all tried, nothing withheld", 2, 0, 6, 0},
|
||||
{"the hub withheld rows (no key material / over its cap)", 1, 3, 6, 3},
|
||||
{"only withheld rows, nothing served", 0, 2, 6, 2},
|
||||
{"the attempt cap stopped the loop", 5, 0, 2, 3},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
blobs := make([]RetainedBlob, 0, c.blobs)
|
||||
for i := 0; i < c.blobs; i++ {
|
||||
blobs = append(blobs, RetainedBlob{Blob: other, SupersededAt: "2026-08-01 00:00:00", Index: i})
|
||||
}
|
||||
_, err := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
|
||||
return blobs, c.withheld, nil
|
||||
},
|
||||
MaxRetainedTried: c.limit,
|
||||
}.RecoverOffsiteRepoPassword(context.Background(), code)
|
||||
if err == nil {
|
||||
t.Fatal("a code that opens nothing succeeded")
|
||||
}
|
||||
var ue *RetainedUncheckedError
|
||||
isUnchecked := errors.As(err, &ue)
|
||||
if c.unchecked == 0 {
|
||||
if isUnchecked {
|
||||
t.Fatalf("every earlier package was tried — this IS a wrong code, got %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if !isUnchecked || ue.Unchecked != c.unchecked || !errors.Is(err, ErrRetainedUnchecked) {
|
||||
t.Fatalf("err = %v, want RetainedUncheckedError{%d} — never call the code wrong when packages were not tried", err, c.unchecked)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A malformed (empty) served package counts as not tried.
|
||||
func TestR304_EmptyServedPackageCountsAsUnchecked(t *testing.T) {
|
||||
ensureAge(t)
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "6666567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
_, err := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: nil, SupersededAt: "2026-08-01 00:00:00"}),
|
||||
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
|
||||
var ue *RetainedUncheckedError
|
||||
if !errors.As(err, &ue) || ue.Unchecked != 1 {
|
||||
t.Fatalf("err = %v, want RetainedUncheckedError{1}", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,346 @@
|
||||
// Package fstrim is the weekly guest disk trim (R-444, operator ruling `09` §3 decision 139).
|
||||
//
|
||||
// Why: a thin pool only ever grows from blocks a guest has already FREED — `fstrim` inside the unprivileged container
|
||||
// is refused (FITRIM: Operation not permitted), and nothing else on the box gives the blocks back. A full thin pool
|
||||
// takes every guest on the host read-only, so the pool can reach 100 % from deleted data alone. Measured on demo-hp
|
||||
// 2026-10-06 09:14Z: `pct fstrim 9201` rc 0 in 24.4 s, pool 65.53 % -> 33.40 %, 18/18 app probes 200, max 1.1 s
|
||||
// (audits/ten-answers-2026-10-06/r444-measure.txt).
|
||||
//
|
||||
// The rule, each part pinned by a test in fstrim_test.go:
|
||||
// - Weekly: a guest is DUE from Wednesday 10:00 local until it has been trimmed once since then (a box that was off
|
||||
// on Wednesday catches up at its next eligible hour).
|
||||
// - Daytime only: a trim starts only between 10:00 and 20:59 local — never in the night window (01:00–06:59) where
|
||||
// the backups and the restore-tests run (TestEligibleHourNeverInTheNight).
|
||||
// - Never beside a backup, a restore-test or another heavy operation: the pass holds the host-wide one-heavy-op gate
|
||||
// (backup.InFlight) for its whole run; a busy gate DEFERS the pass to the next hourly tick.
|
||||
// - A failed trim is retried at the next eligible hour, at most MaxAttemptsPerWeek times in one week.
|
||||
// - The last result per guest (time, bytes, ok/fail) is persisted, so a restart neither loses it nor re-trims.
|
||||
//
|
||||
// The command is the ONE exact sudoers shape `pct fstrim <vmid>` (FELHOM_FSTRIM). Only guests from the pool-verified
|
||||
// source (ListLXC ∩ the felhom pool, audit A1) and only RUNNING ones are trimmed.
|
||||
package fstrim
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// Schedule. The weekday/hours are fixed on purpose (one sentence the operator can read on the System page).
|
||||
const (
|
||||
Weekday = time.Wednesday
|
||||
StartHour = 10 // first eligible local hour (inclusive)
|
||||
EndHour = 21 // first NOT-eligible local hour (exclusive): last start is 20:59
|
||||
MaxAttemptsPerWeek = 3
|
||||
// TickInterval is how often the job looks; a deferred or failed pass is therefore retried the next hour.
|
||||
TickInterval = time.Hour
|
||||
// FirstTickDelay lets the agent settle after a start before the first look.
|
||||
FirstTickDelay = 5 * time.Minute
|
||||
// PerGuestTimeout bounds one `pct fstrim` (measured 24.4 s for 84 GiB).
|
||||
PerGuestTimeout = 30 * time.Minute
|
||||
)
|
||||
|
||||
// ScheduleText is the human description carried on the host report.
|
||||
const ScheduleText = "weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
|
||||
// Runner runs a host command (proxmox.ExecRunner in production, through `sudo -n`).
|
||||
type Runner interface {
|
||||
Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
|
||||
}
|
||||
|
||||
// GuestSource yields the guests this agent OWNS (the pool-verified source, never a bare ListLXC).
|
||||
type GuestSource interface {
|
||||
Guests(ctx context.Context) ([]proxmox.Guest, error)
|
||||
}
|
||||
|
||||
// Gate is the host-wide one-heavy-operation gate (*backup.InFlight).
|
||||
type Gate interface {
|
||||
TryAcquire(what string) (release func(), busy string, ok bool)
|
||||
}
|
||||
|
||||
// GateName is what the gate reports as busy while a trim runs.
|
||||
const GateName = "guest-fstrim"
|
||||
|
||||
// Record is one guest's last trim attempt, as persisted.
|
||||
type Record struct {
|
||||
LastAttemptAt time.Time `json:"last_attempt_at"`
|
||||
OK bool `json:"ok"`
|
||||
BytesTrimmed int64 `json:"bytes_trimmed"`
|
||||
Mounts int `json:"mounts"`
|
||||
DurationSeconds float64 `json:"duration_seconds"`
|
||||
LastOKAt time.Time `json:"last_ok_at,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
// Attempts counts the attempts since the current week's due time (reset by the first attempt of a new week).
|
||||
Attempts int `json:"attempts"`
|
||||
}
|
||||
|
||||
// Trimmer is the weekly job.
|
||||
type Trimmer struct {
|
||||
runner Runner
|
||||
guests GuestSource
|
||||
gate Gate
|
||||
statePath string
|
||||
logger *slog.Logger
|
||||
loc *time.Location
|
||||
now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
records map[int]Record
|
||||
}
|
||||
|
||||
// New builds the job and loads the persisted state. A missing state file is an empty state; a corrupt one is logged
|
||||
// and treated as empty (the cost is one extra trim, never a missed one).
|
||||
func New(runner Runner, guests GuestSource, gate Gate, statePath string, logger *slog.Logger) *Trimmer {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
t := &Trimmer{runner: runner, guests: guests, gate: gate, statePath: statePath, logger: logger,
|
||||
loc: time.Local, now: time.Now, records: map[int]Record{}}
|
||||
t.load()
|
||||
return t
|
||||
}
|
||||
|
||||
func (t *Trimmer) load() {
|
||||
data, err := os.ReadFile(t.statePath)
|
||||
if err != nil {
|
||||
if !os.IsNotExist(err) {
|
||||
t.logger.Warn("fstrim: state read failed — starting empty", "path", t.statePath, "err", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
var raw map[string]Record
|
||||
if err := json.Unmarshal(data, &raw); err != nil {
|
||||
t.logger.Warn("fstrim: state file corrupt — starting empty", "path", t.statePath, "err", err)
|
||||
return
|
||||
}
|
||||
for k, r := range raw {
|
||||
if id, err := strconv.Atoi(k); err == nil && id > 0 {
|
||||
t.records[id] = r
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (t *Trimmer) saveLocked() error {
|
||||
raw := make(map[string]Record, len(t.records))
|
||||
for id, r := range t.records {
|
||||
raw[strconv.Itoa(id)] = r
|
||||
}
|
||||
data, err := json.MarshalIndent(raw, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(t.statePath), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := t.statePath + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, t.statePath)
|
||||
}
|
||||
|
||||
// EligibleHour reports whether a trim may START at local time lt.
|
||||
func EligibleHour(lt time.Time) bool {
|
||||
h := lt.Hour()
|
||||
return h >= StartHour && h < EndHour
|
||||
}
|
||||
|
||||
// weekAnchor is the most recent Wednesday StartHour:00 at or before lt (same location as lt).
|
||||
func weekAnchor(lt time.Time) time.Time {
|
||||
daysBack := (int(lt.Weekday()) - int(Weekday) + 7) % 7
|
||||
d := lt.AddDate(0, 0, -daysBack)
|
||||
a := time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
|
||||
if a.After(lt) {
|
||||
d = d.AddDate(0, 0, -7)
|
||||
a = time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
// due reports whether a guest with record r (ok=false: none) is due at local time lt.
|
||||
func due(r Record, has bool, lt time.Time) bool {
|
||||
if !has {
|
||||
return true
|
||||
}
|
||||
anchor := weekAnchor(lt)
|
||||
if r.LastAttemptAt.Before(anchor) {
|
||||
return true // not tried this week
|
||||
}
|
||||
return !r.OK && r.Attempts < MaxAttemptsPerWeek
|
||||
}
|
||||
|
||||
// Run looks every TickInterval until ctx ends. It never returns an error: a failed trim is a reported fact.
|
||||
func (t *Trimmer) Run(ctx context.Context) {
|
||||
t.logger.Info("fstrim: weekly guest disk trim starting", "schedule", ScheduleText)
|
||||
timer := time.NewTimer(FirstTickDelay)
|
||||
defer timer.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-timer.C:
|
||||
t.Pass(ctx)
|
||||
timer.Reset(TickInterval)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Pass is one look: outside the daytime window it does nothing; otherwise it trims every due, running, owned guest
|
||||
// while holding the heavy-op gate.
|
||||
func (t *Trimmer) Pass(ctx context.Context) {
|
||||
lt := t.now().In(t.loc)
|
||||
if !EligibleHour(lt) {
|
||||
t.logger.Debug("fstrim: outside the daytime window — not looking", "local", lt.Format("Mon 15:04"))
|
||||
return
|
||||
}
|
||||
guests, err := t.guests.Guests(ctx)
|
||||
if err != nil {
|
||||
t.logger.Warn("fstrim: owned-guest list unavailable — skipping this pass", "err", err)
|
||||
return
|
||||
}
|
||||
owned := make(map[int]bool, len(guests))
|
||||
var todo []int
|
||||
t.mu.Lock()
|
||||
for _, g := range guests {
|
||||
owned[g.VMID] = true
|
||||
r, has := t.records[g.VMID]
|
||||
if !due(r, has, lt) {
|
||||
continue
|
||||
}
|
||||
if g.Status != "running" {
|
||||
t.logger.Info("fstrim: guest not running — trimmed when it runs", "vmid", g.VMID, "status", g.Status)
|
||||
continue
|
||||
}
|
||||
todo = append(todo, g.VMID)
|
||||
}
|
||||
// A guest the agent no longer owns has no result to report.
|
||||
pruned := false
|
||||
for id := range t.records {
|
||||
if !owned[id] {
|
||||
delete(t.records, id)
|
||||
pruned = true
|
||||
}
|
||||
}
|
||||
if pruned {
|
||||
if err := t.saveLocked(); err != nil {
|
||||
t.logger.Warn("fstrim: state save failed", "err", err)
|
||||
}
|
||||
}
|
||||
t.mu.Unlock()
|
||||
if len(todo) == 0 {
|
||||
return
|
||||
}
|
||||
sort.Ints(todo)
|
||||
release, busy, ok := t.gate.TryAcquire(GateName)
|
||||
if !ok {
|
||||
t.logger.Info("fstrim: deferred — a heavy operation is in flight; retrying next hour", "busy", busy, "due_guests", len(todo))
|
||||
return
|
||||
}
|
||||
defer release()
|
||||
for _, vmid := range todo {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
t.trimOne(ctx, vmid, lt)
|
||||
}
|
||||
}
|
||||
|
||||
var trimmedLine = regexp.MustCompile(`\((\d+) bytes\) trimmed`)
|
||||
|
||||
// ParseTrimmed sums the "(N bytes) trimmed" lines of `pct fstrim` output and counts them (one per mount point), e.g.
|
||||
// `/var/lib/lxc/9201/rootfs/: 30.1 GiB (32277680128 bytes) trimmed`.
|
||||
func ParseTrimmed(out string) (bytes int64, mounts int) {
|
||||
for _, m := range trimmedLine.FindAllStringSubmatch(out, -1) {
|
||||
n, err := strconv.ParseInt(m[1], 10, 64)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
bytes += n
|
||||
mounts++
|
||||
}
|
||||
return bytes, mounts
|
||||
}
|
||||
|
||||
// GiB renders bytes as "30.1 GiB".
|
||||
func GiB(b int64) string { return fmt.Sprintf("%.1f GiB", float64(b)/(1<<30)) }
|
||||
|
||||
func (t *Trimmer) trimOne(ctx context.Context, vmid int, lt time.Time) {
|
||||
start := t.now()
|
||||
cctx, cancel := context.WithTimeout(ctx, PerGuestTimeout)
|
||||
stdout, stderr, err := t.runner.Run(cctx, "pct", "fstrim", strconv.Itoa(vmid))
|
||||
cancel()
|
||||
dur := t.now().Sub(start)
|
||||
bytes, mounts := ParseTrimmed(string(stdout) + "\n" + string(stderr))
|
||||
|
||||
t.mu.Lock()
|
||||
prev, has := t.records[vmid]
|
||||
r := Record{LastAttemptAt: start.UTC(), OK: err == nil, BytesTrimmed: bytes, Mounts: mounts,
|
||||
DurationSeconds: float64(dur.Round(100*time.Millisecond)) / float64(time.Second), LastOKAt: prev.LastOKAt}
|
||||
if has && !prev.LastAttemptAt.Before(weekAnchor(lt)) {
|
||||
r.Attempts = prev.Attempts + 1
|
||||
} else {
|
||||
r.Attempts = 1
|
||||
}
|
||||
if err == nil {
|
||||
r.LastOKAt = start.UTC()
|
||||
} else {
|
||||
msg := strings.TrimSpace(err.Error() + ": " + strings.TrimSpace(string(stderr)))
|
||||
if len(msg) > 300 {
|
||||
msg = msg[:300]
|
||||
}
|
||||
r.Error = msg
|
||||
}
|
||||
t.records[vmid] = r
|
||||
saveErr := t.saveLocked()
|
||||
t.mu.Unlock()
|
||||
|
||||
if err == nil {
|
||||
t.logger.Info(fmt.Sprintf("fstrim: guest %d trimmed %s in %.1fs", vmid, GiB(bytes), r.DurationSeconds),
|
||||
"vmid", vmid, "bytes_trimmed", bytes, "mounts", mounts, "duration_s", r.DurationSeconds)
|
||||
if mounts == 0 {
|
||||
t.logger.Warn("fstrim: pct fstrim succeeded but reported no trimmed mount — output not understood",
|
||||
"vmid", vmid, "stdout", strings.TrimSpace(string(stdout)))
|
||||
}
|
||||
} else {
|
||||
t.logger.Warn(fmt.Sprintf("fstrim: guest %d trim FAILED after %.1fs", vmid, r.DurationSeconds),
|
||||
"vmid", vmid, "attempt", r.Attempts, "max_attempts_per_week", MaxAttemptsPerWeek, "err", r.Error)
|
||||
}
|
||||
if saveErr != nil {
|
||||
t.logger.Warn("fstrim: state save failed — the result will not survive a restart", "path", t.statePath, "err", saveErr)
|
||||
}
|
||||
}
|
||||
|
||||
// GuestDiskTrimStatus implements hub.GuestDiskTrimReporter: a pure read of the persisted results (never runs pct).
|
||||
func (t *Trimmer) GuestDiskTrimStatus(context.Context) *hub.GuestDiskTrimStatus {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
out := &hub.GuestDiskTrimStatus{Schedule: ScheduleText}
|
||||
ids := make([]int, 0, len(t.records))
|
||||
for id := range t.records {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
sort.Ints(ids)
|
||||
for _, id := range ids {
|
||||
r := t.records[id]
|
||||
g := hub.GuestDiskTrim{VMID: id, LastAttemptAt: r.LastAttemptAt.UTC().Format(time.RFC3339), OK: r.OK,
|
||||
BytesTrimmed: r.BytesTrimmed, Mounts: r.Mounts, DurationSeconds: r.DurationSeconds, Error: r.Error}
|
||||
if !r.LastOKAt.IsZero() {
|
||||
g.LastOKAt = r.LastOKAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
out.Guests = append(out.Guests, g)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,267 @@
|
||||
package fstrim
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// The real `pct fstrim 9201` output measured on demo-hp 2026-10-06 (audits/ten-answers-2026-10-06/r444-measure.txt).
|
||||
const measuredOut = "/var/lib/lxc/9201/rootfs/: 30.1 GiB (32277680128 bytes) trimmed\n" +
|
||||
"/var/lib/lxc/9201/rootfs/var/lib/felhom: 53.9 GiB (57865633792 bytes) trimmed\n"
|
||||
|
||||
const measuredBytes = int64(32277680128 + 57865633792)
|
||||
|
||||
type fakeRunner struct {
|
||||
mu sync.Mutex
|
||||
calls [][]string
|
||||
out string
|
||||
err error
|
||||
onRun func()
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
f.mu.Lock()
|
||||
f.calls = append(f.calls, append([]string{name}, args...))
|
||||
f.mu.Unlock()
|
||||
if f.onRun != nil {
|
||||
f.onRun()
|
||||
}
|
||||
if f.err != nil {
|
||||
return nil, []byte("mount busy"), f.err
|
||||
}
|
||||
return []byte(f.out), nil, nil
|
||||
}
|
||||
|
||||
type fakeGuests struct {
|
||||
g []proxmox.Guest
|
||||
err error
|
||||
}
|
||||
|
||||
func (f fakeGuests) Guests(context.Context) ([]proxmox.Guest, error) { return f.g, f.err }
|
||||
|
||||
// A Wednesday 10:30 in a fixed zone (CEST-like), so the tests do not depend on the machine's zone.
|
||||
var zone = time.FixedZone("CEST", 2*3600)
|
||||
|
||||
func at(day, hour, min int) time.Time { return time.Date(2026, 10, day, hour, min, 0, 0, zone) } // 2026-10-07 = Wednesday
|
||||
|
||||
func newT(t *testing.T, r Runner, g GuestSource, gate Gate, now *time.Time) (*Trimmer, *bytes.Buffer, string) {
|
||||
t.Helper()
|
||||
var logs bytes.Buffer
|
||||
path := filepath.Join(t.TempDir(), "guest-disk-trim.json")
|
||||
tr := New(r, g, gate, path, slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelDebug})))
|
||||
tr.loc = zone
|
||||
tr.now = func() time.Time { return *now }
|
||||
return tr, &logs, path
|
||||
}
|
||||
|
||||
func running(ids ...int) fakeGuests {
|
||||
var g []proxmox.Guest
|
||||
for _, id := range ids {
|
||||
g = append(g, proxmox.Guest{VMID: id, Status: "running", Type: "lxc"})
|
||||
}
|
||||
return fakeGuests{g: g}
|
||||
}
|
||||
|
||||
func TestParseTrimmedTheMeasuredOutput(t *testing.T) {
|
||||
b, m := ParseTrimmed(measuredOut)
|
||||
if b != measuredBytes || m != 2 {
|
||||
t.Fatalf("ParseTrimmed = %d bytes over %d mounts, want %d over 2", b, m, measuredBytes)
|
||||
}
|
||||
if b, m := ParseTrimmed("something else\n"); b != 0 || m != 0 {
|
||||
t.Fatalf("unrelated output parsed as %d/%d", b, m)
|
||||
}
|
||||
if got := GiB(measuredBytes); got != "84.0 GiB" {
|
||||
t.Fatalf("GiB = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The night window (01:00–06:59) must never be eligible, and the daytime window is exactly 10:00–20:59.
|
||||
func TestEligibleHourNeverInTheNight(t *testing.T) {
|
||||
for h := 0; h < 24; h++ {
|
||||
lt := time.Date(2026, 10, 7, h, 30, 0, 0, zone)
|
||||
got := EligibleHour(lt)
|
||||
if h >= 1 && h <= 6 && got {
|
||||
t.Errorf("hour %02d is in the night window and must not be eligible", h)
|
||||
}
|
||||
if want := h >= 10 && h <= 20; got != want {
|
||||
t.Errorf("EligibleHour(%02d:30) = %v, want %v", h, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWeekAnchorIsTheLastWednesdayTen(t *testing.T) {
|
||||
cases := map[time.Time]time.Time{
|
||||
at(7, 10, 0): at(7, 10, 0), // Wednesday 10:00 itself
|
||||
at(7, 9, 59): time.Date(2026, 9, 30, 10, 0, 0, 0, zone), // before 10:00 Wednesday → the previous week
|
||||
at(8, 15, 0): at(7, 10, 0), // Thursday
|
||||
at(13, 20, 0): at(7, 10, 0), // next Tuesday
|
||||
at(14, 11, 0): at(14, 10, 0), // next Wednesday
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := weekAnchor(in); !got.Equal(want) {
|
||||
t.Errorf("weekAnchor(%s) = %s, want %s", in.Format("Mon 01-02 15:04"), got.Format("Mon 01-02 15:04"), want.Format("Mon 01-02 15:04"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The consequence: on Wednesday 10:30 a running owned guest is trimmed with the ONE exact argv, the bytes are parsed,
|
||||
// the positive log line is written, the result is persisted, and the host report carries it.
|
||||
func TestPassTrimsADueGuestAndReportsIt(t *testing.T) {
|
||||
now := at(7, 10, 30)
|
||||
r := &fakeRunner{out: measuredOut}
|
||||
tr, logs, path := newT(t, r, running(9201), &backup.InFlight{}, &now)
|
||||
tr.Pass(context.Background())
|
||||
|
||||
if want := [][]string{{"pct", "fstrim", "9201"}}; !reflect.DeepEqual(r.calls, want) {
|
||||
t.Fatalf("runner calls = %q, want %q", r.calls, want)
|
||||
}
|
||||
if !strings.Contains(logs.String(), "fstrim: guest 9201 trimmed 84.0 GiB in ") {
|
||||
t.Fatalf("no positive per-guest log line:\n%s", logs.String())
|
||||
}
|
||||
st := tr.GuestDiskTrimStatus(context.Background())
|
||||
if st == nil || st.Schedule != ScheduleText || len(st.Guests) != 1 {
|
||||
t.Fatalf("report stanza = %+v", st)
|
||||
}
|
||||
g := st.Guests[0]
|
||||
if g.VMID != 9201 || !g.OK || g.BytesTrimmed != measuredBytes || g.Mounts != 2 || g.LastOKAt == "" || g.LastAttemptAt == "" {
|
||||
t.Fatalf("report guest = %+v", g)
|
||||
}
|
||||
// Persisted: a NEW Trimmer over the same file (an agent restart) still has it and does not trim again this week.
|
||||
now = at(8, 11, 0)
|
||||
r2 := &fakeRunner{out: measuredOut}
|
||||
tr2 := New(r2, running(9201), &backup.InFlight{}, path, slog.New(slog.NewTextHandler(&bytes.Buffer{}, nil)))
|
||||
tr2.loc, tr2.now = zone, func() time.Time { return now }
|
||||
if st2 := tr2.GuestDiskTrimStatus(context.Background()); len(st2.Guests) != 1 || st2.Guests[0].BytesTrimmed != measuredBytes {
|
||||
t.Fatalf("result lost over a restart: %+v", st2)
|
||||
}
|
||||
tr2.Pass(context.Background())
|
||||
if len(r2.calls) != 0 {
|
||||
t.Fatalf("trimmed again in the same week after a restart: %q", r2.calls)
|
||||
}
|
||||
// Next week it is due again.
|
||||
now = at(14, 10, 5)
|
||||
tr2.Pass(context.Background())
|
||||
if len(r2.calls) != 1 {
|
||||
t.Fatalf("not trimmed in the next week: %q", r2.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPassNeverRunsInTheNight(t *testing.T) {
|
||||
for _, h := range []int{1, 3, 6, 9, 21, 23} {
|
||||
now := at(7, h, 15)
|
||||
r := &fakeRunner{out: measuredOut}
|
||||
tr, _, _ := newT(t, r, running(9201), &backup.InFlight{}, &now)
|
||||
tr.Pass(context.Background())
|
||||
if len(r.calls) != 0 {
|
||||
t.Errorf("trimmed at %02d:15: %q", h, r.calls)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A backup (or restore-test) holding the heavy-op gate DEFERS the trim; the next hour, gate free, it runs. And while
|
||||
// a trim runs, the gate is held, so a backup cannot start beside it.
|
||||
func TestPassDefersToAHeavyOperationAndRetriesNextHour(t *testing.T) {
|
||||
now := at(7, 10, 30)
|
||||
gate := &backup.InFlight{}
|
||||
release, _, _ := gate.TryAcquire("backup:9201")
|
||||
var busyDuringTrim string
|
||||
r := &fakeRunner{out: measuredOut}
|
||||
r.onRun = func() { busyDuringTrim = gate.Busy() }
|
||||
tr, logs, _ := newT(t, r, running(9201), gate, &now)
|
||||
|
||||
tr.Pass(context.Background())
|
||||
if len(r.calls) != 0 {
|
||||
t.Fatalf("trimmed beside a running backup: %q", r.calls)
|
||||
}
|
||||
if !strings.Contains(logs.String(), "fstrim: deferred") || !strings.Contains(logs.String(), "backup:9201") {
|
||||
t.Fatalf("the deferral is not logged with what holds the gate:\n%s", logs.String())
|
||||
}
|
||||
release()
|
||||
now = now.Add(time.Hour)
|
||||
tr.Pass(context.Background())
|
||||
if len(r.calls) != 1 {
|
||||
t.Fatalf("not retried the next hour: %q", r.calls)
|
||||
}
|
||||
if busyDuringTrim != GateName {
|
||||
t.Fatalf("the heavy-op gate was %q during the trim, want %q", busyDuringTrim, GateName)
|
||||
}
|
||||
if gate.Busy() != "" {
|
||||
t.Fatalf("the gate was not released after the pass: %q", gate.Busy())
|
||||
}
|
||||
}
|
||||
|
||||
func TestFailedTrimWarnsIsRecordedAndRetriedAtMostThreeTimes(t *testing.T) {
|
||||
now := at(7, 10, 30)
|
||||
r := &fakeRunner{err: errors.New("exit status 255")}
|
||||
tr, logs, _ := newT(t, r, running(9201), &backup.InFlight{}, &now)
|
||||
for i := 0; i < 6; i++ {
|
||||
tr.Pass(context.Background())
|
||||
now = now.Add(time.Hour)
|
||||
}
|
||||
if len(r.calls) != MaxAttemptsPerWeek {
|
||||
t.Fatalf("attempts in one week = %d, want %d", len(r.calls), MaxAttemptsPerWeek)
|
||||
}
|
||||
if !strings.Contains(logs.String(), "level=WARN") || !strings.Contains(logs.String(), "fstrim: guest 9201 trim FAILED") {
|
||||
t.Fatalf("no WARN for the failure:\n%s", logs.String())
|
||||
}
|
||||
g := tr.GuestDiskTrimStatus(context.Background()).Guests[0]
|
||||
if g.OK || g.LastOKAt != "" || !strings.Contains(g.Error, "exit status 255") || !strings.Contains(g.Error, "mount busy") {
|
||||
t.Fatalf("failed result not recorded as a failure: %+v", g)
|
||||
}
|
||||
// A success later keeps a clean record.
|
||||
r.err = nil
|
||||
r.out = measuredOut
|
||||
now = at(14, 10, 10)
|
||||
tr.Pass(context.Background())
|
||||
if g := tr.GuestDiskTrimStatus(context.Background()).Guests[0]; !g.OK || g.Error != "" || g.BytesTrimmed != measuredBytes {
|
||||
t.Fatalf("success after failure: %+v", g)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyRunningOwnedGuestsAndAFailedListActsOnNothing(t *testing.T) {
|
||||
now := at(7, 10, 30)
|
||||
r := &fakeRunner{out: measuredOut}
|
||||
g := fakeGuests{g: []proxmox.Guest{{VMID: 9201, Status: "stopped"}, {VMID: 9202, Status: "running"}}}
|
||||
tr, _, _ := newT(t, r, g, &backup.InFlight{}, &now)
|
||||
tr.Pass(context.Background())
|
||||
if want := [][]string{{"pct", "fstrim", "9202"}}; !reflect.DeepEqual(r.calls, want) {
|
||||
t.Fatalf("calls = %q, want only the running guest", r.calls)
|
||||
}
|
||||
r2 := &fakeRunner{out: measuredOut}
|
||||
tr2, logs, _ := newT(t, r2, fakeGuests{err: errors.New("pool read 403")}, &backup.InFlight{}, &now)
|
||||
tr2.Pass(context.Background())
|
||||
if len(r2.calls) != 0 || !strings.Contains(logs.String(), "owned-guest list unavailable") {
|
||||
t.Fatalf("a failed ownership read must act on nothing: calls %q", r2.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReportJSONShape(t *testing.T) {
|
||||
now := at(7, 10, 30)
|
||||
tr, _, _ := newT(t, &fakeRunner{out: measuredOut}, running(9201), &backup.InFlight{}, &now)
|
||||
tr.Pass(context.Background())
|
||||
b, err := json.Marshal(tr.GuestDiskTrimStatus(context.Background()))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, k := range []string{`"schedule":`, `"guests":[{"vmid":9201`, `"last_attempt_at":"2026-10-07T08:30:00Z"`, `"ok":true`,
|
||||
`"bytes_trimmed":90143313920`, `"mounts":2`, `"duration_seconds":`, `"last_ok_at":"2026-10-07T08:30:00Z"`} {
|
||||
if !strings.Contains(string(b), k) {
|
||||
t.Errorf("report JSON lacks %s: %s", k, b)
|
||||
}
|
||||
}
|
||||
if strings.Contains(string(b), `"error"`) {
|
||||
t.Errorf("an ok result must omit error: %s", b)
|
||||
}
|
||||
}
|
||||
+94
-5
@@ -66,6 +66,13 @@ type ProvenRestoreTestReporter interface {
|
||||
ProvenRestoreTests(ctx context.Context) []RestoreTest
|
||||
}
|
||||
|
||||
// KnownBackupReporter is the newest SUCCESSFUL backup per tier and guest kept on disk (R-894's
|
||||
// backup-success-state.json). collectBackups folds it into the report so a restart does not erase the
|
||||
// hub's evidence of a backup that ran minutes before it (the kernel-night shape, 2026-10-09).
|
||||
type KnownBackupReporter interface {
|
||||
KnownBackupSuccesses() []Backup
|
||||
}
|
||||
|
||||
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
|
||||
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
|
||||
type PBSReporter interface {
|
||||
@@ -90,6 +97,12 @@ type GuestNetReporter interface {
|
||||
GuestNetStatus(ctx context.Context) *GuestNetStatus
|
||||
}
|
||||
|
||||
// GuestDiskTrimReporter is the R-444 seam the weekly trim job plugs into (same consumer-side pattern — hub does not
|
||||
// import fstrim). nil (feature not wired) → no guest_disk_trim stanza.
|
||||
type GuestDiskTrimReporter interface {
|
||||
GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus
|
||||
}
|
||||
|
||||
// Collector builds a HostReport from read-only sources. All deps are behind narrow
|
||||
// interfaces for unit testing.
|
||||
type Collector struct {
|
||||
@@ -99,6 +112,7 @@ type Collector struct {
|
||||
backups BackupReporter
|
||||
restoreTests RestoreTestReporter
|
||||
provenTests ProvenRestoreTestReporter
|
||||
knownBackups KnownBackupReporter // R-894 file → host report after a restart (nil → in-memory only)
|
||||
pbs PBSReporter
|
||||
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||
@@ -108,6 +122,9 @@ type Collector struct {
|
||||
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
|
||||
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
||||
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
|
||||
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
|
||||
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
||||
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
|
||||
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
|
||||
@@ -221,6 +238,30 @@ func (c *Collector) SetGuestNetReporter(g GuestNetReporter) *Collector {
|
||||
return c
|
||||
}
|
||||
|
||||
// ForeignKeyArchiveReporter is the R-366 slice-2 seam: the restore-test's ledger of archives written with another
|
||||
// key. nil = not evaluated yet (the stanza is omitted and the hub keeps its state).
|
||||
type ForeignKeyArchiveReporter interface {
|
||||
ForeignKeyArchives(ctx context.Context) *ForeignKeyArchivesStanza
|
||||
}
|
||||
|
||||
// SetForeignKeyArchiveReporter wires the restore-test's foreign-key ledger (R-366 slice 2; nil-safe → omitted).
|
||||
func (c *Collector) SetForeignKeyArchiveReporter(r ForeignKeyArchiveReporter) *Collector {
|
||||
c.foreignKey = r
|
||||
return c
|
||||
}
|
||||
|
||||
// SetGuestDiskTrimReporter wires the R-444 weekly trim job as a report source (nil-safe → stanza omitted).
|
||||
func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector {
|
||||
c.diskTrim = r
|
||||
return c
|
||||
}
|
||||
|
||||
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
|
||||
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
|
||||
c.deliberateOff = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
|
||||
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
|
||||
type SelfUpdateReporter interface {
|
||||
@@ -377,6 +418,14 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
if c.guestNet != nil {
|
||||
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
|
||||
}
|
||||
// R-444: the last weekly trim result per guest (nil reporter = not wired → stanza omitted).
|
||||
if c.diskTrim != nil {
|
||||
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
|
||||
}
|
||||
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
|
||||
if c.foreignKey != nil {
|
||||
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
|
||||
}
|
||||
// D1: agent self-update pending status (nil reporter → pending=false, the steady state).
|
||||
if c.selfUpdate != nil {
|
||||
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()
|
||||
@@ -389,6 +438,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
if c.oob != nil {
|
||||
report.OOB = c.oob.OOBStatus(ctx)
|
||||
}
|
||||
// R-998: a household power-off accepted in this boot rides every report until the box is down.
|
||||
if c.deliberateOff != nil {
|
||||
report.DeliberateOff = c.deliberateOff()
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
@@ -542,14 +595,50 @@ func (c *Collector) collectStorage(ctx context.Context) []StorageTarget {
|
||||
// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state
|
||||
// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil)
|
||||
// list so the collection always marshals as [].
|
||||
//
|
||||
// The saved successes (SetKnownBackupReporter) are added for each tier and guest the in-memory list has
|
||||
// no success for at or after the saved time. Why: the in-memory list is empty after an agent restart,
|
||||
// and the hub's backup-freshness check reads only what the reports carried. Measured 2026-10-09 on
|
||||
// demo-felhom: the night backup landed 02:40 UTC, the kernel step restarted the host at 02:44, no report
|
||||
// fell in between, so two kernel nights in a row left no trace and the hub alarmed „newest backup is 48h
|
||||
// old" at 03:00. Only SUCCESSES are saved, so a failure is never hidden and never invented.
|
||||
// Pinned by TestCollectBackups_SavedSuccessSurvivesARestart.
|
||||
func (c *Collector) collectBackups(ctx context.Context) []Backup {
|
||||
if c.backups == nil {
|
||||
return []Backup{}
|
||||
out := []Backup{}
|
||||
if c.backups != nil {
|
||||
if b := c.backups.Backups(ctx); b != nil {
|
||||
out = append(out, b...)
|
||||
}
|
||||
}
|
||||
if b := c.backups.Backups(ctx); b != nil {
|
||||
return b
|
||||
if c.knownBackups == nil {
|
||||
return out
|
||||
}
|
||||
return []Backup{}
|
||||
for _, k := range c.knownBackups.KnownBackupSuccesses() {
|
||||
kt, err := time.Parse(time.RFC3339, k.StartedAt)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
covered := false
|
||||
for _, b := range out {
|
||||
if !b.Success || b.TargetID != k.TargetID || b.VMID != k.VMID {
|
||||
continue
|
||||
}
|
||||
if bt, err := time.Parse(time.RFC3339, b.StartedAt); err == nil && !bt.Before(kt) {
|
||||
covered = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !covered {
|
||||
out = append(out, k)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SetKnownBackupReporter wires the R-894 saved successes into the host report (nil-safe → in-memory only).
|
||||
func (c *Collector) SetKnownBackupReporter(r KnownBackupReporter) *Collector {
|
||||
c.knownBackups = r
|
||||
return c
|
||||
}
|
||||
|
||||
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-444: the guest_disk_trim stanza must reach a report built through the PRODUCTION collect path, be absent from
|
||||
// the wire when the job is not wired, and carry the keys the hub's System page reads.
|
||||
|
||||
type fakeDiskTrim struct{ st *GuestDiskTrimStatus }
|
||||
|
||||
func (f fakeDiskTrim) GuestDiskTrimStatus(context.Context) *GuestDiskTrimStatus { return f.st }
|
||||
|
||||
func TestCollect_GuestDiskTrim(t *testing.T) {
|
||||
px := &fakePx{node: "n", ns: newTestNodeStatus()}
|
||||
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.150.0", quietLogger())
|
||||
r, err := c.Collect(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("Collect: %v", err)
|
||||
}
|
||||
b, _ := json.Marshal(r)
|
||||
var m map[string]any
|
||||
_ = json.Unmarshal(b, &m)
|
||||
if _, ok := m["guest_disk_trim"]; ok {
|
||||
t.Fatalf("guest_disk_trim on the wire with no reporter wired: %s", b)
|
||||
}
|
||||
|
||||
c.SetGuestDiskTrimReporter(fakeDiskTrim{st: &GuestDiskTrimStatus{Schedule: "weekly", Guests: []GuestDiskTrim{{
|
||||
VMID: 9201, LastAttemptAt: "2026-10-07T08:30:00Z", OK: true, BytesTrimmed: 90143313920, Mounts: 2,
|
||||
DurationSeconds: 24.4, LastOKAt: "2026-10-07T08:30:00Z",
|
||||
}}}})
|
||||
r, err = c.Collect(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("Collect: %v", err)
|
||||
}
|
||||
b, _ = json.Marshal(r)
|
||||
m = nil
|
||||
_ = json.Unmarshal(b, &m)
|
||||
dt, ok := m["guest_disk_trim"].(map[string]any)
|
||||
if !ok || dt["schedule"] != "weekly" {
|
||||
t.Fatalf("guest_disk_trim missing or wrong on the wire: %s", b)
|
||||
}
|
||||
g := dt["guests"].([]any)[0].(map[string]any)
|
||||
for _, k := range []string{"vmid", "last_attempt_at", "ok", "bytes_trimmed", "mounts", "duration_seconds", "last_ok_at"} {
|
||||
if _, ok := g[k]; !ok {
|
||||
t.Fatalf("guest_disk_trim.guests[0] lacks %q: %v", k, g)
|
||||
}
|
||||
}
|
||||
if g["bytes_trimmed"] != float64(90143313920) || g["ok"] != true {
|
||||
t.Fatalf("values did not survive the round trip: %v", g)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type fakeMemBackups []Backup
|
||||
|
||||
func (f fakeMemBackups) Backups(context.Context) []Backup { return f }
|
||||
|
||||
type fakeKnownBackups []Backup
|
||||
|
||||
func (f fakeKnownBackups) KnownBackupSuccesses() []Backup { return f }
|
||||
|
||||
// The consequence, not the mechanism: after a restart (empty in-memory list) the host report still carries
|
||||
// the night's backup, so the hub's freshness check sees it. Measured 2026-10-09 on demo-felhom: without this
|
||||
// the report carried backups: [] and the hub alarmed „newest backup is 48h old" an hour after a good backup.
|
||||
// RED-PROOF: return before the saved-success loop in collectBackups → the first case reports nothing → fails.
|
||||
func TestCollectBackups_SavedSuccessSurvivesARestart(t *testing.T) {
|
||||
saved := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}
|
||||
|
||||
t.Run("restart: memory empty, the saved success is reported", func(t *testing.T) {
|
||||
c := &Collector{backups: fakeMemBackups(nil), knownBackups: fakeKnownBackups{saved}}
|
||||
got := c.collectBackups(context.Background())
|
||||
if len(got) != 1 || got[0].StartedAt != saved.StartedAt || !got[0].Success || got[0].TargetID != "felhom-backup" {
|
||||
t.Fatalf("want the saved success, got %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("memory holds the same or a newer success: no second entry", func(t *testing.T) {
|
||||
mem := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z", Archive: "a"}
|
||||
c := &Collector{backups: fakeMemBackups{mem}, knownBackups: fakeKnownBackups{saved}}
|
||||
if got := c.collectBackups(context.Background()); len(got) != 1 || got[0].Archive != "a" {
|
||||
t.Fatalf("want only the in-memory record, got %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a newer FAILURE in memory never hides the saved success, and is kept", func(t *testing.T) {
|
||||
fail := Backup{TargetID: "felhom-backup", VMID: 9201, Success: false, StartedAt: "2026-10-10T02:40:00Z", Error: "x"}
|
||||
c := &Collector{backups: fakeMemBackups{fail}, knownBackups: fakeKnownBackups{saved}}
|
||||
got := c.collectBackups(context.Background())
|
||||
if len(got) != 2 || got[0].Success || !got[1].Success {
|
||||
t.Fatalf("want the failure and the saved success, got %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("another tier's success does not cover this tier", func(t *testing.T) {
|
||||
other := Backup{TargetID: "felhom-pbs", VMID: 9201, Success: true, StartedAt: "2026-10-10T00:00:00Z"}
|
||||
c := &Collector{backups: fakeMemBackups{other}, knownBackups: fakeKnownBackups{saved}}
|
||||
if got := c.collectBackups(context.Background()); len(got) != 2 {
|
||||
t.Fatalf("want both tiers, got %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("no saved state wired: the in-memory list as before, never nil", func(t *testing.T) {
|
||||
c := &Collector{}
|
||||
if got := c.collectBackups(context.Background()); got == nil || len(got) != 0 {
|
||||
t.Fatalf("want an empty non-nil list, got %#v", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -39,11 +39,14 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
|
||||
State: StorageStateAttached, Reachable: true, ClassHint: "fast",
|
||||
ThinPool: &ThinPoolFill{DataUsedFraction: 0.42},
|
||||
Smart: SmartSummary{Health: SmartUnknown},
|
||||
// v0.157.0 (`16` §3): the identity keys ride on both rows so the key-set guard sees them.
|
||||
Kind: "system", PhysicalDisks: []string{"serial:ata:SanDisk_X600:182195804614"}, Identity: "proven",
|
||||
},
|
||||
{
|
||||
Name: "usb-backup", Type: StorageTypeUSB, DurableID: "uuid:x",
|
||||
State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000,
|
||||
Smart: SmartSummary{Health: SmartUnknown},
|
||||
Kind: "disk", PhysicalDisks: []string{"serial:usb:External:000000000000"}, Identity: "unknown", IdentityReason: "usb_bridge",
|
||||
},
|
||||
// A pbs target so the recipe's pbs coord has a storage.cfg row to resolve its namespace
|
||||
// from (R-106). Without one the fixture produces namespace_state=unknown while the golden
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-998: the power-off path waits on SetReportedHook, so the hook must fire only for a report the hub
|
||||
// ACCEPTED — a failed push must not let the box power off believing the hub knows.
|
||||
func TestLoop_ReportedHookOnlyAfterAcceptedReport(t *testing.T) {
|
||||
var cn, rn int32
|
||||
rep := &HostReport{DeliberateOff: &DeliberateOff{RequestedAt: "2026-10-13T18:00:00Z"}}
|
||||
loop := NewLoop(&fakeCollector{report: rep, n: &cn},
|
||||
&fakeReporter{env: &ControlEnvelope{}, errSeq: []error{errors.New("hub 5xx")}, n: &rn},
|
||||
900*time.Second, quietLogger())
|
||||
var seen []*HostReport
|
||||
loop.SetReportedHook(func(r *HostReport) { seen = append(seen, r) })
|
||||
|
||||
loop.cycle(context.Background(), 900*time.Second) // the push fails
|
||||
if len(seen) != 0 {
|
||||
t.Fatalf("hook fired for a refused report: %d", len(seen))
|
||||
}
|
||||
loop.cycle(context.Background(), 900*time.Second) // the push is accepted
|
||||
if len(seen) != 1 || seen[0].DeliberateOff == nil || seen[0].DeliberateOff.RequestedAt != "2026-10-13T18:00:00Z" {
|
||||
t.Fatalf("hook after an accepted report = %+v", seen)
|
||||
}
|
||||
}
|
||||
@@ -66,8 +66,15 @@ type Loop struct {
|
||||
// retry, no duplicate shipping). Loop state is single-goroutine (cycle only).
|
||||
logTailSource func(maxBytes int) []string
|
||||
logTailPending bool
|
||||
|
||||
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
|
||||
// path waits on it so the hub has the deliberate-off record before the box goes down.
|
||||
reported func(r *HostReport)
|
||||
}
|
||||
|
||||
// SetReportedHook wires the R-998 after-report observer. Optional.
|
||||
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
|
||||
|
||||
// logTailMaxBytes caps the heartbeat log tail (newest lines kept).
|
||||
const logTailMaxBytes = 128 * 1024
|
||||
|
||||
@@ -156,6 +163,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
|
||||
l.logger.Warn("hub: report failed; keeping current interval", "err", err)
|
||||
return current
|
||||
}
|
||||
if l.reported != nil {
|
||||
l.reported(report)
|
||||
}
|
||||
if report.LogTail != nil {
|
||||
// Transparency: the pull is visible in the box's own log (and thus in the ring).
|
||||
l.logger.Info("operator log pull served", "component", "agent", "lines", len(report.LogTail.Lines))
|
||||
|
||||
@@ -124,6 +124,17 @@ type HostReport struct {
|
||||
// on HostReport would have been the only report block named against that convention.
|
||||
GuestNet *GuestNetStatus `json:"guest_net,omitempty"`
|
||||
|
||||
// GuestDiskTrim is the weekly guest disk trim stanza (R-444, `09` §3 decision 139): the schedule and, per owned
|
||||
// guest, the LAST trim result as persisted by the agent (it survives a restart). Present only when the trim job is
|
||||
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
|
||||
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
|
||||
|
||||
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
|
||||
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
|
||||
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
|
||||
// the agent started (the hub keeps its last state); `tiers: []` = evaluated, none found.
|
||||
ForeignKeyArchives *ForeignKeyArchivesStanza `json:"foreign_key_archives,omitempty"`
|
||||
|
||||
// LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent
|
||||
// mirror of the controller's report log_tails channel. Present ONLY on the heartbeat
|
||||
// right after the control envelope requested it (log_tail_requested); consume-once on
|
||||
@@ -133,6 +144,12 @@ type HostReport struct {
|
||||
// in the steady state, so the cross-repo host-report golden stays byte-stable.
|
||||
LogTail *LogTail `json:"log_tail,omitempty"`
|
||||
|
||||
// DeliberateOff (R-998, doc 14): the household asked this box to power off, in THIS boot. Present on
|
||||
// every report from the request until the box goes down; absent again after the next boot — that
|
||||
// absence is how the hub learns the box is back. The hub holds its silence alarms while it is set
|
||||
// (14 days). `omitempty`: absent in the steady state, so the cross-repo host-report golden stays stable.
|
||||
DeliberateOff *DeliberateOff `json:"deliberate_off,omitempty"`
|
||||
|
||||
// OOB is the operator-access health stanza (TASK H1). It answers the operator's question — "can I
|
||||
// get into this box right now, and if not, why" — from the hub: felhom-sshd up + on which port,
|
||||
// locally reachable, the tunnel handshake age (the OOB path rides wg-felhom), whether the operator
|
||||
@@ -208,6 +225,27 @@ type GuestNetGuest struct {
|
||||
Message string `json:"message,omitempty"`
|
||||
}
|
||||
|
||||
// GuestDiskTrimStatus is the R-444 weekly trim stanza. `schedule` is a plain description of when the job runs (local
|
||||
// time of the host); `guests` holds one entry per owned guest that has had at least one trim attempt.
|
||||
type GuestDiskTrimStatus struct {
|
||||
Schedule string `json:"schedule"`
|
||||
Guests []GuestDiskTrim `json:"guests,omitempty"`
|
||||
}
|
||||
|
||||
// GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never
|
||||
// read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is
|
||||
// the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points.
|
||||
type GuestDiskTrim struct {
|
||||
VMID int `json:"vmid"`
|
||||
LastAttemptAt string `json:"last_attempt_at"`
|
||||
OK bool `json:"ok"`
|
||||
BytesTrimmed int64 `json:"bytes_trimmed"`
|
||||
Mounts int `json:"mounts"`
|
||||
DurationSeconds float64 `json:"duration_seconds"`
|
||||
LastOKAt string `json:"last_ok_at,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type PBSDRStatus struct {
|
||||
State string `json:"state"`
|
||||
StorageID string `json:"storage_id,omitempty"`
|
||||
@@ -393,6 +431,15 @@ type StorageTarget struct {
|
||||
// metadata fill is null until Phase B's privileged `lvs` read.
|
||||
ThinPool *ThinPoolFill `json:"thin_pool,omitempty"`
|
||||
|
||||
// Physical-disk identity (felhom.eu architecture/16-storage-physical-identity.md §3, agent v0.157.0):
|
||||
// the place's kind, the keys of the whole physical disks under it, and whether the agent can prove
|
||||
// them (proven | unknown + identity_reason vm|network|usb_bridge|weak_wwn|no_serial). Agent-computed
|
||||
// from /sys + udev only; empty when not computed (an absent device, an older agent). Additive.
|
||||
Kind string `json:"kind,omitempty"`
|
||||
PhysicalDisks []string `json:"physical_disks,omitempty"`
|
||||
Identity string `json:"identity,omitempty"`
|
||||
IdentityReason string `json:"identity_reason,omitempty"`
|
||||
|
||||
// Smart is the disk-health summary, populated in Phase B via smartctl (sudoers). In
|
||||
// Phase A it is {health:"UNKNOWN", <counters null>} — the keys are committed now so the
|
||||
// contract is stable across both phases.
|
||||
@@ -428,6 +475,17 @@ type SmartSummary struct {
|
||||
ReallocatedSectors *int `json:"reallocated_sectors"`
|
||||
PendingSectors *int `json:"pending_sectors"`
|
||||
OfflineUncorrectable *int `json:"offline_uncorrectable"`
|
||||
// R-330 (disk health Phase 2): three more SATA raw counters. omitempty + pointer: absent (an
|
||||
// older agent, an NVMe/USB device, or a drive that does not report the attribute) is OMITTED —
|
||||
// unknown, never a zero (S-39). Wire only: no verdict reads them yet.
|
||||
// 187 Reported_Uncorrect — the failing drive's most telling counter (normalized 1 vs thresh 0,
|
||||
// raw 1001) while SMART still said PASSED.
|
||||
// 188 Command_Timeout — some vendors PACK several counters into the 48-bit raw value, so the
|
||||
// number is carried as reported and must not be compared across vendors.
|
||||
// 199 UDMA_CRC_Error_Count — cabling / link errors, not the medium.
|
||||
ReportedUncorrect *int64 `json:"reported_uncorrect,omitempty"`
|
||||
CommandTimeout *int64 `json:"command_timeout,omitempty"`
|
||||
UDMACRCErrors *int64 `json:"udma_crc_errors,omitempty"`
|
||||
|
||||
// NVMe attributes.
|
||||
CriticalWarning *int `json:"critical_warning"`
|
||||
@@ -593,6 +651,17 @@ type WireOSUpdate struct {
|
||||
// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
|
||||
// approved for the guest is not approved for the host by that fact alone.
|
||||
HostRelease *WireOSRelease `json:"host_release,omitempty"`
|
||||
// Kernel is the kernel lane's instruction for THIS box (R-836, `09` §3 decision 172, `11` §5.11): the kernel the
|
||||
// household was told about, and whether tonight is a told night. Nil (an older hub, or no kernel due) = no kernel
|
||||
// step. The hub sets Tonight only after the household's mail the day before went out — no mail, no step.
|
||||
Kernel *WireKernelStep `json:"kernel,omitempty"`
|
||||
}
|
||||
|
||||
// WireKernelStep is the hub's kernel-lane instruction (hub osupdates.KernelBlock — field-exact, cross-repo).
|
||||
type WireKernelStep struct {
|
||||
Kver string `json:"kver"` // e.g. "7.0.14-22-pve" — the wrapper refuses any other (R23)
|
||||
Tonight bool `json:"tonight"` // the household was mailed the day before: tonight's leg may reboot
|
||||
NotifiedAt string `json:"notified_at,omitempty"` // when that mail went out (RFC 3339), for the log
|
||||
}
|
||||
|
||||
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
|
||||
@@ -689,3 +758,23 @@ type WireRestoreDirective struct {
|
||||
Archive string `json:"archive,omitempty"` // source archive/snapshot to restore from
|
||||
VMID int `json:"vmid,omitempty"`
|
||||
}
|
||||
|
||||
// ForeignKeyArchivesStanza wraps the per-tier list so "evaluated, none" (`tiers: []`) differs from "not evaluated"
|
||||
// (the stanza absent) without a null on the wire.
|
||||
type ForeignKeyArchivesStanza struct {
|
||||
Tiers []ForeignKeyArchives `json:"tiers"`
|
||||
}
|
||||
|
||||
// ForeignKeyArchives is one tier's count of archives written with another key (R-366 slice 2): the count and the
|
||||
// newest/oldest archive time (RFC3339, UTC). No key material — the fingerprints stay on the box.
|
||||
type ForeignKeyArchives struct {
|
||||
Target string `json:"target"`
|
||||
Count int `json:"count"`
|
||||
Oldest string `json:"oldest"`
|
||||
Newest string `json:"newest"`
|
||||
}
|
||||
|
||||
// DeliberateOff is the household-requested power-off record (R-998).
|
||||
type DeliberateOff struct {
|
||||
RequestedAt string `json:"requested_at"` // RFC3339 UTC, the agent's clock
|
||||
}
|
||||
|
||||
@@ -46,6 +46,9 @@
|
||||
"backing_device": "",
|
||||
"class_hint": "fast",
|
||||
"role": "",
|
||||
"kind": "system",
|
||||
"physical_disks": ["serial:ata:SanDisk_X600:182195804614"],
|
||||
"identity": "proven",
|
||||
"thin_pool": { "data_used_fraction": 0.42, "metadata_used_fraction": null },
|
||||
"smart": {
|
||||
"health": "UNKNOWN",
|
||||
@@ -74,6 +77,10 @@
|
||||
"backing_device": "/dev/sdb1",
|
||||
"class_hint": "slow",
|
||||
"role": "",
|
||||
"kind": "disk",
|
||||
"physical_disks": ["serial:usb:External:000000000000"],
|
||||
"identity": "unknown",
|
||||
"identity_reason": "usb_bridge",
|
||||
"smart": {
|
||||
"health": "UNKNOWN",
|
||||
"temperature_c": null,
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
|
||||
// and the gate sub-case fails.
|
||||
func TestAfterPrimaryBackup(t *testing.T) {
|
||||
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
|
||||
run := func(t *testing.T, failErr, path string) (calls []int, gateHeld bool) {
|
||||
gate := &backup.InFlight{}
|
||||
b := &fakeBackups{failErr: failErr}
|
||||
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
|
||||
@@ -34,7 +34,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
||||
done <- struct{}{}
|
||||
})
|
||||
h := srv.Handler()
|
||||
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
|
||||
if do(t, h, "POST", path, "A", "").Code != http.StatusAccepted {
|
||||
t.Fatal("POST /backup not accepted")
|
||||
}
|
||||
select {
|
||||
@@ -47,7 +47,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
||||
return calls, gateHeld
|
||||
}
|
||||
t.Run("success runs the leg under the gate", func(t *testing.T) {
|
||||
calls, held := run(t, "")
|
||||
calls, held := run(t, "", "/backup")
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
|
||||
}
|
||||
@@ -56,8 +56,21 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
||||
}
|
||||
})
|
||||
t.Run("a failed backup runs nothing", func(t *testing.T) {
|
||||
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
|
||||
if calls, _ := run(t, "vzdump exploded", "/backup"); len(calls) != 0 {
|
||||
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
|
||||
}
|
||||
})
|
||||
// R-899: a household press is not the night's backup — no OS leg after it. Same fake, same successful backup as
|
||||
// the first sub-case; only the query differs. Red-proof: make handleBackup ignore `trigger` and this sub-case
|
||||
// fails with the leg run once.
|
||||
t.Run("a manual press runs nothing", func(t *testing.T) {
|
||||
if calls, _ := run(t, "", "/backup?trigger=manual"); len(calls) != 0 {
|
||||
t.Fatalf("the OS leg ran after a manual press: %v", calls)
|
||||
}
|
||||
})
|
||||
t.Run("the scheduled path with the new query still runs the leg", func(t *testing.T) {
|
||||
if calls, _ := run(t, "", "/backup?trigger=night"); len(calls) != 1 {
|
||||
t.Fatalf("the leg ran %d time(s) after a non-manual backup, want 1", len(calls))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -20,10 +19,10 @@ import (
|
||||
// again", and each guard is pinned by the case where acting would be wrong.
|
||||
|
||||
type supExec struct {
|
||||
mu sync.Mutex
|
||||
status map[int]string // docker .State.Status per vmid; "" = container absent
|
||||
inspectErr error // non-nil = pct exec itself failed (unknown)
|
||||
restarts map[int]int
|
||||
mu sync.Mutex
|
||||
status map[int]string // docker .State.Status per vmid; "" = container absent
|
||||
inspectErr error // non-nil = pct exec itself failed (unknown)
|
||||
restarts map[int]int
|
||||
// onRestart, when set, is the status the container reaches after a restart (a crash-looper
|
||||
// stays "exited").
|
||||
onRestart string
|
||||
@@ -54,8 +53,8 @@ func (f *supExec) GuestExec(_ context.Context, vmid int, args ...string) (string
|
||||
}
|
||||
return "", errors.New("supExec: unexpected args")
|
||||
}
|
||||
func (f *supExec) GuestExecStdin(context.Context, int, io.Reader, ...string) (string, error) {
|
||||
return "", errors.New("supExec: no stdin exec expected")
|
||||
func (f *supExec) WriteControllerImage(context.Context, int, string) error {
|
||||
return errors.New("supExec: no image write expected")
|
||||
}
|
||||
func (f *supExec) count(vmid int) int {
|
||||
f.mu.Lock()
|
||||
@@ -250,7 +249,6 @@ func TestControllerSupervisorStanza_WireShape(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ---- R-539 (operator ruling 3 of 2026-09-16): the SLOW crash loop ----------------------------------
|
||||
|
||||
// supKillOnce kills the controller and lets the supervisor restart it (two confirming sweeps), then
|
||||
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
@@ -41,9 +40,10 @@ func ValidControllerImage(ref string) bool { return controllerImageRe.MatchStrin
|
||||
// faked in tests. The single seam the swap composes over (no hand-rolled pct).
|
||||
type GuestExecutor interface {
|
||||
GuestExec(ctx context.Context, vmid int, args ...string) (string, error)
|
||||
// GuestExecStdin is GuestExec with the command's stdin fed from stdin — the swap write pipes the
|
||||
// image ref into an in-guest `tee` (no shell vector).
|
||||
GuestExecStdin(ctx context.Context, vmid int, stdin io.Reader, args ...string) (string, error)
|
||||
// WriteControllerImage writes the image ref into the guest's /etc/felhom-controller-image through the ROOT
|
||||
// verb `felhom-priv-apply controller-image <vmid>` (R-861 (a) A1, `09` §3 decision 165), which re-checks the
|
||||
// ref against our registry + repository + x.y.z. The agent no longer holds a `tee` grant into the guest.
|
||||
WriteControllerImage(ctx context.Context, vmid int, image string) error
|
||||
}
|
||||
|
||||
// ControllerSwapState is the durable record of a swap (crash-safety + status). Written before the swap
|
||||
@@ -141,14 +141,11 @@ func (c *ControllerSwapper) imagePresent(ctx context.Context, vmid int, image st
|
||||
}
|
||||
|
||||
func (c *ControllerSwapper) writeImage(ctx context.Context, vmid int, image string) error {
|
||||
// Non-root path: pipe the image ref into an in-guest `tee` over stdin — no shell, no
|
||||
// interpolation, no `bash -c` (the only swap vector that would have needed an arbitrary-exec
|
||||
// grant). The trailing "\n" makes the on-disk bytes byte-identical to the golden's
|
||||
// `printf '%s\n'`; the bootstrap reads `IMAGE=$(cat …)` so the newline is stripped on read
|
||||
// (spike SPIKE-controllerswap-narrow-grants-2026-06-29). image is strict-validated
|
||||
// (controllerImageRe) upstream in Swap; defence-in-depth, the stdin path can't smuggle anyway.
|
||||
_, err := c.exec.GuestExecStdin(ctx, vmid, strings.NewReader(image+"\n"), "tee", controllerImageFile)
|
||||
return err
|
||||
// R-861 (a) A1: the ROOT verb writes the file (it re-checks the ref — a compromised agent cannot hand the guest's
|
||||
// bootstrap another image). The bytes are `image\n`, byte-identical to the golden's `printf '%s\n'`; the bootstrap
|
||||
// reads `IMAGE=$(cat …)` so the newline is stripped on read. image is also strict-validated (controllerImageRe)
|
||||
// upstream in Swap.
|
||||
return c.exec.WriteControllerImage(ctx, vmid, image)
|
||||
}
|
||||
|
||||
func (c *ControllerSwapper) restartBootstrap(ctx context.Context, vmid int) error {
|
||||
|
||||
@@ -23,7 +23,7 @@ type fakeGuestExec struct {
|
||||
present map[string]bool // images pulled into the guest
|
||||
good map[string]bool // images that report healthy when running
|
||||
containerImg string // image the running container currently has
|
||||
teeStdin []string // raw bytes piped into each `tee` write (the swap's write vector)
|
||||
teeStdin []string // image refs handed to WriteControllerImage (the root verb, R-861 (a) A1)
|
||||
failRestart bool
|
||||
noHealthBlock bool // if set, .State.Health is absent ("none")
|
||||
restartCount int // F1: .RestartCount reported by docker inspect (a crash-looper has >0)
|
||||
@@ -75,20 +75,15 @@ func (f *fakeGuestExec) GuestExec(_ context.Context, _ int, args ...string) (str
|
||||
return "", fmt.Errorf("fake: unexpected exec %v", args)
|
||||
}
|
||||
|
||||
// GuestExecStdin models the swap's write vector: `tee /etc/felhom-controller-image` with the image
|
||||
// piped on stdin. It records the raw stdin bytes and sets the modeled file content (newline-stripped,
|
||||
// as the bootstrap's `IMAGE=$(cat …)` read would see it).
|
||||
func (f *fakeGuestExec) GuestExecStdin(_ context.Context, _ int, stdin io.Reader, args ...string) (string, error) {
|
||||
// WriteControllerImage models the swap's write: the root verb `felhom-priv-apply controller-image <vmid>` (R-861
|
||||
// (a) A1). It records the ref and sets the modeled file content, as the bootstrap's `IMAGE=$(cat …)` would read it.
|
||||
func (f *fakeGuestExec) WriteControllerImage(_ context.Context, _ int, image string) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.calls = append(f.calls, args)
|
||||
b, _ := io.ReadAll(stdin)
|
||||
if len(args) >= 2 && args[0] == "tee" && args[1] == controllerImageFile {
|
||||
f.teeStdin = append(f.teeStdin, string(b))
|
||||
f.imageFile = strings.TrimSpace(string(b))
|
||||
return string(b), nil // tee echoes stdin to stdout
|
||||
}
|
||||
return "", fmt.Errorf("fake: unexpected exec-stdin args=%v stdin=%q", args, string(b))
|
||||
f.calls = append(f.calls, []string{"felhom-priv-apply", "controller-image", image})
|
||||
f.teeStdin = append(f.teeStdin, image+"\n")
|
||||
f.imageFile = image
|
||||
return nil
|
||||
}
|
||||
|
||||
// wrote reports whether the image was written via the stdin `tee` vector with the exact `image\n`
|
||||
@@ -162,7 +157,7 @@ func TestControllerSwap_Happy(t *testing.T) {
|
||||
|
||||
// The write vector must be the stdin `tee` with byte-identical `image\n` and NO shell — the
|
||||
// controllerswap.go writeImage rewrite. This would FAIL on the pre-change `bash -c "printf … >"` impl.
|
||||
func TestControllerSwap_WriteViaStdinTee_NoShell(t *testing.T) {
|
||||
func TestControllerSwap_WriteViaRootVerb_NoShell(t *testing.T) {
|
||||
fe := &fakeGuestExec{
|
||||
imageFile: prevImg,
|
||||
present: map[string]bool{newImg: true},
|
||||
@@ -173,22 +168,15 @@ func TestControllerSwap_WriteViaStdinTee_NoShell(t *testing.T) {
|
||||
t.Fatalf("state = %q, want done", st.State)
|
||||
}
|
||||
if !fe.wrote(newImg) {
|
||||
t.Errorf("expected a tee write of %q+\\n; teeStdin=%q", newImg, fe.teeStdin)
|
||||
t.Errorf("expected the root verb to write %q; writes=%q", newImg, fe.teeStdin)
|
||||
}
|
||||
sawTee := false
|
||||
for _, c := range fe.calls {
|
||||
if len(c) >= 2 && c[0] == "tee" {
|
||||
sawTee = true
|
||||
if c[1] != controllerImageFile {
|
||||
t.Errorf("tee target = %q, want fixed %q", c[1], controllerImageFile)
|
||||
}
|
||||
if len(c) >= 1 && c[0] == "tee" {
|
||||
t.Errorf("the swap still uses an in-guest tee (R-861 (a) A1 removed that grant): %v", c)
|
||||
}
|
||||
}
|
||||
if !sawTee {
|
||||
t.Error("no tee call recorded — writeImage did not use the stdin tee vector")
|
||||
}
|
||||
if fe.usedShell() {
|
||||
t.Errorf("swap used a shell vector (bash/-c/printf) — must be stdin tee only; calls=%v", fe.calls)
|
||||
t.Errorf("swap used a shell vector (bash/-c/printf); calls=%v", fe.calls)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -300,9 +288,7 @@ func (s *inspectScript) GuestExec(_ context.Context, _ int, args ...string) (str
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
func (s *inspectScript) GuestExecStdin(_ context.Context, _ int, _ io.Reader, _ ...string) (string, error) {
|
||||
return "", nil
|
||||
}
|
||||
func (s *inspectScript) WriteControllerImage(context.Context, int, string) error { return nil }
|
||||
|
||||
func fastSwapper(exec GuestExecutor) *ControllerSwapper {
|
||||
s := NewControllerSwapper(exec, "", discardLogger())
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"os"
|
||||
"time"
|
||||
)
|
||||
|
||||
// GET /host/crash-guard (R-856, `09` §3 decision 143): what the host's crash guard
|
||||
// (configs/felhom-crash-guard, `11` §5.9) recorded about the most recent HOST boot. The controller
|
||||
// reads it once after it starts: when the host's last boot followed an UNCLEAN stop, its app mails
|
||||
// wait ~15 minutes instead of the normal 90 s boot grace.
|
||||
//
|
||||
// Read-only and Proxmox-free: the agent reads the guard's state file (root-owned, 0644 — the
|
||||
// non-root agent can read it) and passes four fields through. Host-wide, token-authed (any valid
|
||||
// per-guest token sees the host's view, as GET /host/metrics does).
|
||||
//
|
||||
// NEVER an error page. A missing file (no guard installed, or no boot recorded yet), an unreadable
|
||||
// one, or one that does not parse answers 200 with present:false — the controller reads that as
|
||||
// UNKNOWN and keeps its normal boot grace. Pinned by TestR856_CrashGuard*.
|
||||
|
||||
// defaultCrashGuardStatePath is where configs/felhom-crash-guard writes its state (STATE_DIR there).
|
||||
const defaultCrashGuardStatePath = "/var/lib/felhom-crash-guard/state.json"
|
||||
|
||||
// crashGuardStateMax bounds the read; the real file is well under 4 KiB.
|
||||
const crashGuardStateMax = 1 << 20
|
||||
|
||||
// CrashGuardResponse is the data block of GET /host/crash-guard. Field names are the controller's
|
||||
// agentapi.CrashGuardState (felhom-controller internal/agentapi/crashguard.go) — a wire contract,
|
||||
// pinned by TestR856_CrashGuardWireMatchesControllerClient.
|
||||
type CrashGuardResponse struct {
|
||||
Present bool `json:"present"`
|
||||
LastBootAt string `json:"last_boot_at,omitempty"` // RFC3339 UTC ("2006-01-02T15:04:05Z")
|
||||
LastBootUnclean bool `json:"last_boot_unclean"`
|
||||
Tripped bool `json:"tripped"`
|
||||
}
|
||||
|
||||
// crashGuardFile is the subset of the guard's state.json the route passes through. Every other key
|
||||
// (armed, boot_id, config, unclean_boots, last_trip, ...) is ignored.
|
||||
type crashGuardFile struct {
|
||||
LastBootAt string `json:"last_boot_at"`
|
||||
LastBootUnclean bool `json:"last_boot_unclean"`
|
||||
Tripped bool `json:"tripped"`
|
||||
}
|
||||
|
||||
// readCrashGuardState reads and parses the guard's state file. ok=false on ANY failure (missing,
|
||||
// unreadable, oversized, not a JSON object, a field of the wrong type); reason says which, for the log.
|
||||
func readCrashGuardState(path string) (resp CrashGuardResponse, ok bool, reason string) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return resp, false, "no state file"
|
||||
}
|
||||
return resp, false, "unreadable: " + err.Error()
|
||||
}
|
||||
defer f.Close()
|
||||
raw, err := io.ReadAll(io.LimitReader(f, crashGuardStateMax+1))
|
||||
if err != nil {
|
||||
return resp, false, "read: " + err.Error()
|
||||
}
|
||||
if len(raw) > crashGuardStateMax {
|
||||
return resp, false, "state file too large"
|
||||
}
|
||||
var st crashGuardFile
|
||||
// Unmarshal into a struct fails on a non-object top level (null decodes, so reject it below).
|
||||
if err := json.Unmarshal(raw, &st); err != nil {
|
||||
return resp, false, "unparseable: " + err.Error()
|
||||
}
|
||||
var probe map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &probe); err != nil || probe == nil {
|
||||
return resp, false, "unparseable: not a JSON object"
|
||||
}
|
||||
resp = CrashGuardResponse{Present: true, LastBootUnclean: st.LastBootUnclean, Tripped: st.Tripped}
|
||||
// Normalise to RFC3339 UTC; an unparseable time passes through as-is (the controller reads an
|
||||
// unparseable boot time as "not this start's boot" → its normal grace).
|
||||
if t, perr := time.Parse(time.RFC3339, st.LastBootAt); perr == nil {
|
||||
resp.LastBootAt = t.UTC().Format(time.RFC3339)
|
||||
} else {
|
||||
resp.LastBootAt = st.LastBootAt
|
||||
}
|
||||
return resp, true, ""
|
||||
}
|
||||
|
||||
func (s *Server) handleCrashGuard(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
path := s.crashGuardStatePath
|
||||
if path == "" {
|
||||
path = defaultCrashGuardStatePath
|
||||
}
|
||||
resp, ok, reason := readCrashGuardState(path)
|
||||
if !ok {
|
||||
s.logger.Debug("local-api: /host/crash-guard not present", "vmid", vmid, "reason", reason)
|
||||
writeOK(w, CrashGuardResponse{Present: false})
|
||||
return
|
||||
}
|
||||
s.logger.Debug("local-api: /host/crash-guard served", "vmid", vmid,
|
||||
"last_boot_at", resp.LastBootAt, "last_boot_unclean", resp.LastBootUnclean, "tripped", resp.Tripped)
|
||||
writeOK(w, resp)
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The shape of /var/lib/felhom-crash-guard/state.json as read on demo-hp on 2026-10-06 (values from
|
||||
// that read where they matter; lists/objects kept to the same key set).
|
||||
const crashGuardFixture = `{
|
||||
"armed": true,
|
||||
"boot_id": "3f1c0f1e-6a0b-4d7e-9b7a-0c2d4e6f8a1b",
|
||||
"config": {"LIMIT": 3, "WINDOW_MINUTES": 60, "PANIC_SECONDS": 10},
|
||||
"kernel_panic": 10,
|
||||
"last_boot_at": "2026-10-05T07:56:41Z",
|
||||
"last_boot_unclean": true,
|
||||
"last_trip": {},
|
||||
"rearmed_at": "2026-10-04T14:02:11Z",
|
||||
"rearmed_by": "operator",
|
||||
"tripped": false,
|
||||
"unclean_boots": ["2026-10-05T07:56:41Z"],
|
||||
"unclean_boots_24h": 1,
|
||||
"unclean_boots_in_window": 1,
|
||||
"updated_at": "2026-10-05T07:57:02Z",
|
||||
"version": 1
|
||||
}`
|
||||
|
||||
// controllerCrashGuardState is a COPY of the controller's wire type, felhom-controller
|
||||
// controller/internal/agentapi/crashguard.go `CrashGuardState` (commit 8b13a5e) — same field names,
|
||||
// same tags. If either side renames a key, the contract test below fails.
|
||||
type controllerCrashGuardState struct {
|
||||
Present bool `json:"present"`
|
||||
LastBootAt string `json:"last_boot_at,omitempty"`
|
||||
LastBootUnclean bool `json:"last_boot_unclean"`
|
||||
Tripped bool `json:"tripped"`
|
||||
}
|
||||
|
||||
func newCrashGuardServer(t *testing.T, statePath string) http.Handler {
|
||||
t.Helper()
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0",
|
||||
Guests: &fakeGuests{},
|
||||
Backups: &fakeBackups{},
|
||||
Store: &fakeStore{},
|
||||
Storage: fakeStorage{},
|
||||
Tokens: staticTokens{"A": 8200, "B": 9300},
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("new server: %v", err)
|
||||
}
|
||||
srv.crashGuardStatePath = statePath
|
||||
return srv.Handler()
|
||||
}
|
||||
|
||||
func writeCrashGuardFixture(t *testing.T, body string) string {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), "state.json")
|
||||
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// getCrashGuard calls the route and decodes the envelope with the CONTROLLER's type.
|
||||
func getCrashGuard(t *testing.T, h http.Handler, token string) (int, controllerCrashGuardState, string) {
|
||||
t.Helper()
|
||||
w := do(t, h, "GET", "/host/crash-guard", token, "")
|
||||
var env struct {
|
||||
OK bool `json:"ok"`
|
||||
Data controllerCrashGuardState `json:"data"`
|
||||
}
|
||||
if w.Code == http.StatusOK {
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
|
||||
t.Fatalf("decode %q: %v", w.Body.String(), err)
|
||||
}
|
||||
if !env.OK {
|
||||
t.Fatalf("ok=false: %s", w.Body.String())
|
||||
}
|
||||
}
|
||||
return w.Code, env.Data, w.Body.String()
|
||||
}
|
||||
|
||||
// A present state file (demo-hp's shape) passes the three facts through.
|
||||
func TestR856_CrashGuardPresentFile(t *testing.T) {
|
||||
h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture))
|
||||
code, st, body := getCrashGuard(t, h, "A")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("got %d, want 200 (%s)", code, body)
|
||||
}
|
||||
want := controllerCrashGuardState{Present: true, LastBootAt: "2026-10-05T07:56:41Z", LastBootUnclean: true, Tripped: false}
|
||||
if st != want {
|
||||
t.Fatalf("state = %+v, want %+v", st, want)
|
||||
}
|
||||
|
||||
// A tripped, clean boot reads back as such (both bools are carried, not defaulted).
|
||||
h = newCrashGuardServer(t, writeCrashGuardFixture(t,
|
||||
`{"last_boot_at":"2026-10-05T09:56:41+02:00","last_boot_unclean":false,"tripped":true,"version":1}`))
|
||||
_, st, _ = getCrashGuard(t, h, "B")
|
||||
want = controllerCrashGuardState{Present: true, LastBootAt: "2026-10-05T07:56:41Z", LastBootUnclean: false, Tripped: true}
|
||||
if st != want {
|
||||
t.Fatalf("offset time / tripped: state = %+v, want %+v (time normalised to UTC Z)", st, want)
|
||||
}
|
||||
}
|
||||
|
||||
// No state file (no guard on this host, or no boot recorded yet) → 200 present:false.
|
||||
func TestR856_CrashGuardMissingFile(t *testing.T) {
|
||||
h := newCrashGuardServer(t, filepath.Join(t.TempDir(), "absent", "state.json"))
|
||||
code, st, body := getCrashGuard(t, h, "A")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("missing file: got %d, want 200 (%s)", code, body)
|
||||
}
|
||||
if st.Present || st.LastBootUnclean || st.Tripped || st.LastBootAt != "" {
|
||||
t.Fatalf("missing file: state = %+v, want present:false and nothing else", st)
|
||||
}
|
||||
}
|
||||
|
||||
// A garbled file → 200 present:false, never a 5xx — every shape of garbage.
|
||||
func TestR856_CrashGuardGarbageFile(t *testing.T) {
|
||||
for name, body := range map[string]string{
|
||||
"truncated": crashGuardFixture[:40],
|
||||
"not json": "this is not json\n",
|
||||
"empty": "",
|
||||
"null": "null",
|
||||
"array": `[{"last_boot_unclean":true}]`,
|
||||
"wrong type": `{"last_boot_at":"2026-10-05T07:56:41Z","last_boot_unclean":"yes","tripped":false}`,
|
||||
"lone brace": "{",
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
h := newCrashGuardServer(t, writeCrashGuardFixture(t, body))
|
||||
code, st, raw := getCrashGuard(t, h, "A")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("got %d, want 200 (%s)", code, raw)
|
||||
}
|
||||
if st.Present || st.LastBootUnclean {
|
||||
t.Fatalf("garbage %q read as %+v, want present:false", name, st)
|
||||
}
|
||||
})
|
||||
}
|
||||
// The path is a directory, not a file: unreadable → present:false, 200.
|
||||
h := newCrashGuardServer(t, t.TempDir())
|
||||
if code, st, raw := getCrashGuard(t, h, "A"); code != http.StatusOK || st.Present {
|
||||
t.Fatalf("directory path: got %d %+v (%s), want 200 present:false", code, st, raw)
|
||||
}
|
||||
}
|
||||
|
||||
// No / unknown token → 401, like every sibling route; a cross-guest ?vmid= → 403.
|
||||
func TestR856_CrashGuardRequiresGuestToken(t *testing.T) {
|
||||
h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture))
|
||||
for _, tok := range []string{"", "bogus"} {
|
||||
w := do(t, h, "GET", "/host/crash-guard", tok, "")
|
||||
if w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("token %q: got %d, want 401", tok, w.Code)
|
||||
}
|
||||
if json.Valid(w.Body.Bytes()) {
|
||||
var env struct {
|
||||
Data controllerCrashGuardState `json:"data"`
|
||||
}
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &env)
|
||||
if env.Data.Present || env.Data.LastBootUnclean {
|
||||
t.Fatalf("token %q: the refusal leaked the state: %s", tok, w.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
if w := do(t, h, "GET", "/host/crash-guard?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("cross-guest query: got %d, want 403", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// Wire contract: every key the controller's CrashGuardState decodes is emitted under exactly that
|
||||
// name, and the agent emits no key the controller does not know.
|
||||
func TestR856_CrashGuardWireMatchesControllerClient(t *testing.T) {
|
||||
h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture))
|
||||
w := do(t, h, "GET", "/host/crash-guard", "A", "")
|
||||
var env struct {
|
||||
OK bool `json:"ok"`
|
||||
Data map[string]json.RawMessage `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil || !env.OK {
|
||||
t.Fatalf("envelope: %v %s", err, w.Body.String())
|
||||
}
|
||||
want := []string{"present", "last_boot_at", "last_boot_unclean", "tripped"}
|
||||
for _, k := range want {
|
||||
if _, ok := env.Data[k]; !ok {
|
||||
t.Errorf("agent does not emit %q, which the controller decodes (%s)", k, w.Body.String())
|
||||
}
|
||||
}
|
||||
if len(env.Data) != len(want) {
|
||||
t.Errorf("agent emits %d keys, controller knows %d: %s", len(env.Data), len(want), w.Body.String())
|
||||
}
|
||||
// And the agent's own type agrees with the controller's copy, field for field.
|
||||
var mine CrashGuardResponse
|
||||
var theirs controllerCrashGuardState
|
||||
raw, _ := json.Marshal(env.Data)
|
||||
_ = json.Unmarshal(raw, &mine)
|
||||
_ = json.Unmarshal(raw, &theirs)
|
||||
if (controllerCrashGuardState{mine.Present, mine.LastBootAt, mine.LastBootUnclean, mine.Tripped}) != theirs {
|
||||
t.Errorf("agent %+v vs controller %+v", mine, theirs)
|
||||
}
|
||||
}
|
||||
@@ -188,6 +188,13 @@ type DiskInfo struct {
|
||||
// device that exposes no SMART (USB bridge, unread) is ABSENT, not a misleading zero-value UNKNOWN;
|
||||
// the controller feature-detects presence and renders "Nincs adat" when nil (never alarms).
|
||||
Smart *hub.SmartSummary `json:"smart,omitempty"`
|
||||
// Physical-disk identity (v0.157.0, felhom.eu architecture/16-storage-physical-identity.md §3) — the
|
||||
// same agent-computed verdict the hub report carries. Empty when not computed (an absent drive, no
|
||||
// resolver). Additive; an older controller ignores it.
|
||||
Kind string `json:"kind,omitempty"`
|
||||
PhysicalDisks []string `json:"physical_disks,omitempty"`
|
||||
Identity string `json:"identity,omitempty"`
|
||||
IdentityReason string `json:"identity_reason,omitempty"`
|
||||
}
|
||||
|
||||
// handleDisks lists the host's drives + data-bearing flags (read-only/benign).
|
||||
@@ -227,6 +234,8 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
// really is the target. Reported here so the controller can name the drive in an
|
||||
// absent-target alarm and hide the destructive controls the agent would refuse anyway.
|
||||
BackupTarget: t.Name == primaryTargetID,
|
||||
// `16` §3: copied from the observer, which computed it from /sys + udev.
|
||||
Kind: t.Kind, PhysicalDisks: t.PhysicalDisks, Identity: t.Identity, IdentityReason: t.IdentityReason,
|
||||
}
|
||||
// Intermediary model: the stable in-guest path + whether felhom-data is bound under the parent.
|
||||
// Only user-data /mnt/<name> drives have a guest path (system/backup mounts never cross in).
|
||||
@@ -398,6 +407,10 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
di.Smart = &sm
|
||||
}
|
||||
}
|
||||
// `16` §3: a registry drive's identity, judged on its OWN mount only (never root's).
|
||||
if p, kind, ok := s.ownMountIdentity(d.MountPath); ok {
|
||||
di.Kind, di.PhysicalDisks, di.Identity, di.IdentityReason = kind, p.PhysicalDisks, p.Identity, p.Reason
|
||||
}
|
||||
// R-118: statfs ONLY while the drive's device is present. With the device gone the raw
|
||||
// mountpoint reverts to a bare directory on the ROOT filesystem, and statfs would report
|
||||
// pve-root's size as this drive's (measured: a 4 GB drive advertising 46 GiB). Same trap
|
||||
|
||||
@@ -226,6 +226,9 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
// The reply's fields are read HERE, before the job goroutine may write job.Phase (a data race the race
|
||||
// detector caught, 2026-10-13 — the reply could show the finished phase or a torn value).
|
||||
jobID, phase := job.JobID, job.Phase
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
@@ -274,7 +277,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
|
||||
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
|
||||
}()
|
||||
s.escrowDone = done // tests wait on it; production polls the status endpoint
|
||||
writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": job.JobID, "phase": job.Phase}, "")
|
||||
writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": jobID, "phase": phase}, "")
|
||||
}
|
||||
|
||||
// handleEscrowCeremonyStatus is GET /escrow/ceremony/status: phase + the NON-SECRET summary.
|
||||
|
||||
@@ -127,6 +127,22 @@ func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Req
|
||||
"retained_has_restic_pw": match.HasResticPassword,
|
||||
},
|
||||
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
|
||||
// ── R-304 (2026-10-08) — NOT EVERY EARLIER PACKAGE WAS CHECKED. ─────────────────────────
|
||||
//
|
||||
// The current package refused the code, no retained package opened it — and at least one earlier
|
||||
// package the hub holds was never tried (or the list could not be read). Saying „the code is
|
||||
// wrong" here would claim a check that did not happen. 424 (Failed Dependency): the verdict
|
||||
// depends on packages we could not try. The controller classifies on the status, never on this
|
||||
// sentence; an older controller maps an unknown status to its neutral „we do not know why".
|
||||
case errors.Is(err, escrow.ErrRetainedUnchecked):
|
||||
n := -1
|
||||
var ue *escrow.RetainedUncheckedError
|
||||
if errors.As(err, &ue) {
|
||||
n = ue.Unchecked
|
||||
}
|
||||
s.logger.Warn("local-api: offsite key recovery: the code did not open the current package and earlier packages were NOT all checked — not reported as a wrong code (R-304)", "vmid", vmid, "unchecked", n)
|
||||
writeStatus(w, http.StatusFailedDependency, false, map[string]any{"older_unchecked": n},
|
||||
"the recovery code did not open the current sealed package, and earlier packages the hub holds were not all checked — the code may belong to one of them; nothing was written")
|
||||
case errors.Is(err, escrow.ErrNoResticPassword):
|
||||
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
||||
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
||||
|
||||
@@ -54,6 +54,13 @@ func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
|
||||
wantStatus: 404,
|
||||
mustNotSay: []string{"did not open"},
|
||||
},
|
||||
{
|
||||
// R-304: earlier packages were not all tried — never „did not open the sealed bundle" (the wrong-code words).
|
||||
name: "earlier packages not all checked — not a wrong code",
|
||||
err: &escrow.RetainedUncheckedError{Unchecked: 2},
|
||||
wantStatus: 424,
|
||||
mustNotSay: []string{"did not open the sealed bundle", "could not be fetched"},
|
||||
},
|
||||
{
|
||||
name: "the bundle predates the repository-password field",
|
||||
err: escrow.ErrNoResticPassword,
|
||||
|
||||
@@ -3,11 +3,11 @@ package localapi
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
@@ -105,6 +105,26 @@ func (b *GuestBinder) RebootGuest(ctx context.Context, vmid int) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// ShutdownGuest stops the guest cleanly with a time limit (R-998 host power-off, step 2). `pct shutdown`
|
||||
// blocks until the guest is down or the limit passes; then it fails and the guest is still running.
|
||||
func (b *GuestBinder) ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error {
|
||||
secs := int(timeout / time.Second)
|
||||
if secs < 1 {
|
||||
secs = 1
|
||||
}
|
||||
b.logger.Warn("host-poweroff: shutting the guest down", "vmid", vmid, "timeout_s", secs)
|
||||
if err := b.run(ctx, "pct", "shutdown", strconv.Itoa(vmid), "--timeout", strconv.Itoa(secs)); err != nil {
|
||||
return fmt.Errorf("host-poweroff: pct shutdown %d: %w", vmid, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PowerOffHost powers the Proxmox host off (R-998, step 3). The one fixed vector `/usr/bin/systemctl poweroff`.
|
||||
func (b *GuestBinder) PowerOffHost(ctx context.Context) error {
|
||||
b.logger.Warn("host-poweroff: powering the host off")
|
||||
return b.run(ctx, "/usr/bin/systemctl", "poweroff")
|
||||
}
|
||||
|
||||
func (b *GuestBinder) run(ctx context.Context, name string, args ...string) error {
|
||||
_, stderr, err := b.runner.Run(ctx, name, args...)
|
||||
if err != nil {
|
||||
@@ -126,14 +146,16 @@ func (b *GuestBinder) GuestExec(ctx context.Context, vmid int, args ...string) (
|
||||
return string(out), nil
|
||||
}
|
||||
|
||||
// GuestExecStdin is GuestExec with the in-guest command's stdin fed from stdin. The controller-swap
|
||||
// write uses it to pipe the image ref into an in-guest `tee` (no shell vector, no interpolation),
|
||||
// through the same fenced runner so the `sudo -n` prefix stays in one place.
|
||||
func (b *GuestBinder) GuestExecStdin(ctx context.Context, vmid int, stdin io.Reader, args ...string) (string, error) {
|
||||
pctArgs := append([]string{"exec", strconv.Itoa(vmid), "--"}, args...)
|
||||
out, stderr, err := b.runner.RunStdin(ctx, stdin, "pct", pctArgs...)
|
||||
// privApplyBin is the root content checker (R-861); its `controller-image` verb writes the guest's image file.
|
||||
const privApplyBin = "/usr/local/sbin/felhom-priv-apply"
|
||||
|
||||
// WriteControllerImage pipes `image\n` to `felhom-priv-apply controller-image <vmid>` through the same fenced runner
|
||||
// (the `sudo -n` prefix stays in one place). The verb checks the ref as root and writes the guest file itself
|
||||
// (R-861 (a) A1, `09` §3 decision 165). Pinned by TestR861_WriteControllerImageUsesTheRootVerb.
|
||||
func (b *GuestBinder) WriteControllerImage(ctx context.Context, vmid int, image string) error {
|
||||
_, stderr, err := b.runner.RunStdin(ctx, strings.NewReader(image+"\n"), privApplyBin, "controller-image", strconv.Itoa(vmid))
|
||||
if err != nil {
|
||||
return string(out), fmt.Errorf("pct exec %d %v: %w: %s", vmid, args, err, strings.TrimSpace(string(stderr)))
|
||||
return fmt.Errorf("felhom-priv-apply controller-image %d: %w: %s", vmid, err, strings.TrimSpace(string(stderr)))
|
||||
}
|
||||
return string(out), nil
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
)
|
||||
|
||||
// ── POST /host/poweroff (R-998, documentation/architecture/14-host-power-off.md) ──────────────────────
|
||||
//
|
||||
// The household's „A gép kikapcsolása": shut THIS guest down cleanly, then power the whole host off. The
|
||||
// first local-API route whose effect reaches beyond the caller's own guest — a compromised controller can
|
||||
// now switch the box off, and nobody can switch it on remotely. The mitigations are the limits below:
|
||||
//
|
||||
// - refused in the first 10 minutes after boot (no power-off loop);
|
||||
// - at most ONE accepted request per boot (poweroff.State, persisted, keyed by the boot);
|
||||
// - refused while a backup, restore test, OS update step, controller update, drive format, NAS verify
|
||||
// or escrow ceremony runs, or a vzdump of the guest is in flight — each named in the 409;
|
||||
// - refused while ANY OTHER guest or VM runs on the host (security review 2026-10-13): one guest's
|
||||
// controller must not switch off another household's guest, a scratch guest or a test VM (demo-hp
|
||||
// carries the Tester 1 VM). Read from /proc (`lxc-start -n N`, `kvm -id N`), fail closed;
|
||||
// - the heavy-operation gate is TAKEN on acceptance and never given back (the box is going down), so no
|
||||
// backup, restore test or OS step can start between the checks and the power-off (security review).
|
||||
// Released only if the power-off command itself fails.
|
||||
//
|
||||
// Every check fails CLOSED: an unwired gate, an unreadable task list or /proc refuses.
|
||||
// On acceptance the record is written FIRST, the hub is given a report carrying it (bounded wait), and
|
||||
// only then: guest shutdown (time-limited), host power-off. Pinned by hostpoweroff_test.go.
|
||||
|
||||
// HostPowerer is the privileged half: both run as fixed sudo vectors (configs/felhom-agent.sudoers).
|
||||
type HostPowerer interface {
|
||||
ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error
|
||||
PowerOffHost(ctx context.Context) error
|
||||
}
|
||||
|
||||
// HostPowerOffConfig wires the route. A nil config leaves it answering 503.
|
||||
type HostPowerOffConfig struct {
|
||||
State *poweroff.State
|
||||
Power HostPowerer
|
||||
ReportNow func(ctx context.Context) bool // push a report carrying the record; true = the hub accepted it
|
||||
Uptime func() (time.Duration, bool) // time since host boot; false = unknown (refuse)
|
||||
Running func() ([]int, error) // vmids of every running guest and VM on the host; default /proc
|
||||
MinUptime time.Duration // default 10 min
|
||||
GuestStop time.Duration // pct shutdown --timeout; default 120 s
|
||||
ReportMax time.Duration // longest wait for the hub; default 60 s
|
||||
JobWait time.Duration // longest wait for a job that started after acceptance; default 15 min
|
||||
JobPoll time.Duration // how often that wait looks again; default 10 s
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
// Refusal codes — the controller maps each to a household sentence. Stable wire values.
|
||||
const (
|
||||
PowerOffTooSoon = "too_soon_after_boot"
|
||||
PowerOffAlreadyAsked = "already_requested_this_boot"
|
||||
PowerOffBackup = "backup_running"
|
||||
PowerOffRestoreTest = "restore_test_running"
|
||||
PowerOffOSUpdate = "os_update_running"
|
||||
PowerOffCtrlUpdate = "controller_update_running"
|
||||
PowerOffDrive = "drive_operation_running"
|
||||
PowerOffEscrow = "escrow_ceremony_running"
|
||||
PowerOffBusy = "host_busy"
|
||||
PowerOffOtherGuests = "other_guests_running"
|
||||
PowerOffUnknown = "state_unknown"
|
||||
)
|
||||
|
||||
// SetHostPowerOff enables POST /host/poweroff.
|
||||
func (s *Server) SetHostPowerOff(c HostPowerOffConfig) {
|
||||
if c.MinUptime == 0 {
|
||||
c.MinUptime = 10 * time.Minute
|
||||
}
|
||||
if c.GuestStop == 0 {
|
||||
c.GuestStop = 120 * time.Second
|
||||
}
|
||||
if c.ReportMax == 0 {
|
||||
c.ReportMax = 60 * time.Second
|
||||
}
|
||||
if c.Now == nil {
|
||||
c.Now = time.Now
|
||||
}
|
||||
if c.JobWait == 0 {
|
||||
c.JobWait = 15 * time.Minute
|
||||
}
|
||||
if c.JobPoll == 0 {
|
||||
c.JobPoll = 10 * time.Second
|
||||
}
|
||||
if c.Running == nil {
|
||||
c.Running = func() ([]int, error) { return runningGuestsFromProc("/proc") }
|
||||
}
|
||||
s.powerOff = &c
|
||||
}
|
||||
|
||||
type hostPowerOffRequest struct {
|
||||
VMID int `json:"vmid"`
|
||||
}
|
||||
|
||||
func (s *Server) handleHostPowerOff(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
c := s.powerOff
|
||||
if c == nil || c.State == nil || c.Power == nil {
|
||||
writeErr(w, http.StatusServiceUnavailable, "host power-off not configured on this host")
|
||||
return
|
||||
}
|
||||
if r.ContentLength != 0 {
|
||||
var req hostPowerOffRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||
return
|
||||
}
|
||||
}
|
||||
// One decision at a time: two requests racing must not both pass the guards before either marks.
|
||||
s.powerOffMu.Lock()
|
||||
defer s.powerOffMu.Unlock()
|
||||
if code, detail := s.powerOffBlocker(r.Context(), vmid); code != "" {
|
||||
s.logger.Info("local-api: host power-off refused", "vmid", vmid, "reason", code, "detail", detail)
|
||||
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": detail}, "host power-off refused: "+code)
|
||||
return
|
||||
}
|
||||
// Take the heavy-operation gate for good: nothing heavy may start between these checks and the power-off.
|
||||
release, busy, free := s.inFlight.TryAcquire("host-poweroff")
|
||||
if !free {
|
||||
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": PowerOffBusy, "detail": busy}, "host power-off refused: "+PowerOffBusy)
|
||||
return
|
||||
}
|
||||
rec, err := c.State.Mark(c.Now(), vmid)
|
||||
if err != nil {
|
||||
release()
|
||||
code := PowerOffUnknown
|
||||
if errors.Is(err, poweroff.ErrAlreadyThisBoot) {
|
||||
code = PowerOffAlreadyAsked
|
||||
}
|
||||
s.logger.Warn("local-api: host power-off refused (record)", "vmid", vmid, "reason", code, "err", err)
|
||||
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": err.Error()}, "host power-off refused: "+code)
|
||||
return
|
||||
}
|
||||
s.logger.Warn("local-api: HOST POWER-OFF accepted (household request)", "vmid", vmid,
|
||||
"requested_at", rec.RequestedAt.Format(time.RFC3339))
|
||||
base := s.baseCtx
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
go s.runHostPowerOff(base, vmid, release)
|
||||
writeStatus(w, http.StatusAccepted, true, map[string]any{
|
||||
"vmid": vmid, "powering_off": true, "requested_at": rec.RequestedAt.Format(time.RFC3339),
|
||||
}, "")
|
||||
}
|
||||
|
||||
// runHostPowerOff: hub first (bounded), then the guest (time-limited), then the host. A failed guest
|
||||
// shutdown does not stop the power-off — the household asked for it, and Proxmox's own shutdown stops
|
||||
// any guest still running (pve-guests) before the host goes down.
|
||||
func (s *Server) runHostPowerOff(base context.Context, vmid int, release func()) {
|
||||
c := s.powerOff
|
||||
if c.ReportNow != nil {
|
||||
ctx, cancel := context.WithTimeout(base, c.ReportMax)
|
||||
ok := c.ReportNow(ctx)
|
||||
cancel()
|
||||
s.logger.Warn("host-poweroff: step 1/3 — hub told", "vmid", vmid, "hub_accepted", ok)
|
||||
}
|
||||
if !s.waitJobsClear(base, vmid, "before the guest shutdown") {
|
||||
release()
|
||||
return
|
||||
}
|
||||
start := time.Now()
|
||||
ctx, cancel := context.WithTimeout(base, c.GuestStop+60*time.Second)
|
||||
err := c.Power.ShutdownGuest(ctx, vmid, c.GuestStop)
|
||||
cancel()
|
||||
if err != nil {
|
||||
s.logger.Error("host-poweroff: step 2/3 — guest shutdown failed; powering off anyway", "vmid", vmid,
|
||||
"took", time.Since(start).Round(time.Second), "err", err)
|
||||
} else {
|
||||
s.logger.Warn("host-poweroff: step 2/3 — guest stopped", "vmid", vmid, "took", time.Since(start).Round(time.Second))
|
||||
}
|
||||
if !s.waitJobsClear(base, vmid, "before the host power-off") {
|
||||
release()
|
||||
return
|
||||
}
|
||||
s.logger.Warn("host-poweroff: step 3/3 — powering the host off")
|
||||
ctx, cancel = context.WithTimeout(base, 60*time.Second)
|
||||
defer cancel()
|
||||
if err := c.Power.PowerOffHost(ctx); err != nil {
|
||||
s.logger.Error("host-poweroff: host power-off command failed — heavy-operation gate released", "err", err)
|
||||
release()
|
||||
}
|
||||
}
|
||||
|
||||
// powerOffBlocker returns the first reason to refuse ("" = none) and a detail for the log/operator.
|
||||
func (s *Server) powerOffBlocker(ctx context.Context, vmid int) (string, string) {
|
||||
c := s.powerOff
|
||||
if c.State.Current() != nil {
|
||||
return PowerOffAlreadyAsked, "a power-off was already accepted in this boot"
|
||||
}
|
||||
if c.Uptime == nil {
|
||||
return PowerOffUnknown, "uptime not wired"
|
||||
}
|
||||
up, ok := c.Uptime()
|
||||
if !ok {
|
||||
return PowerOffUnknown, "cannot read the host uptime"
|
||||
}
|
||||
if up < c.MinUptime {
|
||||
return PowerOffTooSoon, "host up " + up.Round(time.Second).String() + ", minimum " + c.MinUptime.String()
|
||||
}
|
||||
if s.inFlight == nil || s.staleLock == nil {
|
||||
return PowerOffUnknown, "the heavy-operation gate or the Proxmox task reader is not wired"
|
||||
}
|
||||
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
|
||||
return code, detail
|
||||
}
|
||||
if busy := s.inFlight.Busy(); busy != "" {
|
||||
switch {
|
||||
case strings.HasPrefix(busy, "backup:"):
|
||||
return PowerOffBackup, busy
|
||||
case busy == "restore-test":
|
||||
return PowerOffRestoreTest, busy
|
||||
case strings.HasPrefix(busy, "os-"):
|
||||
return PowerOffOSUpdate, busy
|
||||
default:
|
||||
return PowerOffBusy, busy
|
||||
}
|
||||
}
|
||||
return s.jobBlocker(ctx, vmid)
|
||||
}
|
||||
|
||||
// jobBlocker is the part of the checks that can change after acceptance — a backup job, a controller update, a drive
|
||||
// or NAS job, an escrow ceremony, a Proxmox vzdump. runHostPowerOff asks it again before each step (security review
|
||||
// 2026-10-13: the checks ran only once). The heavy-operation gate is not here: the power-off holds it.
|
||||
func (s *Server) jobBlocker(ctx context.Context, vmid int) (string, string) {
|
||||
s.jobsMu.Lock()
|
||||
for k, j := range s.jobs {
|
||||
if j != nil && backupInFlight(j.Phase) {
|
||||
s.jobsMu.Unlock()
|
||||
return PowerOffBackup, "backup job " + j.JobID + " on target " + k.target
|
||||
}
|
||||
}
|
||||
s.jobsMu.Unlock()
|
||||
s.swapMu.Lock()
|
||||
for v, on := range s.swapInFlight {
|
||||
if on {
|
||||
s.swapMu.Unlock()
|
||||
return PowerOffCtrlUpdate, "controller swap in flight for guest " + strconv.Itoa(v)
|
||||
}
|
||||
}
|
||||
s.swapMu.Unlock()
|
||||
if s.formatJobs != nil {
|
||||
if j := s.formatJobs.get(); j != nil && j.Phase == formatPhaseRunning {
|
||||
return PowerOffDrive, "drive format running"
|
||||
}
|
||||
}
|
||||
s.netVerifyMu.Lock()
|
||||
nv := s.netVerifyCur != nil && s.netVerifyCur.Phase == netVerifyPhaseRunning
|
||||
s.netVerifyMu.Unlock()
|
||||
if nv {
|
||||
return PowerOffDrive, "network storage verify running"
|
||||
}
|
||||
s.escrowMu.Lock()
|
||||
esc := s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning
|
||||
s.escrowMu.Unlock()
|
||||
if esc {
|
||||
return PowerOffEscrow, "escrow ceremony running"
|
||||
}
|
||||
vz, err := s.staleLock.BackupRunning(ctx, vmid)
|
||||
if err != nil {
|
||||
return PowerOffUnknown, "cannot list running Proxmox tasks: " + err.Error()
|
||||
}
|
||||
if vz {
|
||||
return PowerOffBackup, "vzdump running for guest " + strconv.Itoa(vmid)
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// runningGuestsFromProc lists the vmids of running containers (`lxc-start … -n N`) and VMs (`kvm -id N`) from
|
||||
// the process table — readable by the non-root agent, unlike /run/qemu-server (measured on demo-hp 2026-10-13).
|
||||
// An unreadable /proc is an error (fail closed); an unreadable single process is skipped (it exited).
|
||||
func runningGuestsFromProc(procDir string) ([]int, error) {
|
||||
ents, err := os.ReadDir(procDir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
seen := map[int]bool{}
|
||||
var out []int
|
||||
for _, e := range ents {
|
||||
if _, err := strconv.Atoi(e.Name()); err != nil {
|
||||
continue
|
||||
}
|
||||
raw, err := os.ReadFile(filepath.Join(procDir, e.Name(), "cmdline"))
|
||||
if err != nil || len(raw) == 0 {
|
||||
continue
|
||||
}
|
||||
args := strings.Split(strings.TrimRight(string(raw), "\x00"), "\x00")
|
||||
flag := ""
|
||||
switch filepath.Base(args[0]) {
|
||||
case "lxc-start":
|
||||
flag = "-n"
|
||||
case "kvm", "qemu-system-x86_64":
|
||||
flag = "-id"
|
||||
default:
|
||||
continue
|
||||
}
|
||||
for i := 1; i+1 < len(args); i++ {
|
||||
if args[i] == flag {
|
||||
if v, err := strconv.Atoi(args[i+1]); err == nil && !seen[v] {
|
||||
seen[v] = true
|
||||
out = append(out, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// otherGuestsBlocker refuses while any guest or VM other than the caller's runs (or the list cannot be read).
|
||||
func (s *Server) otherGuestsBlocker(vmid int) (string, string) {
|
||||
running, err := s.powerOff.Running()
|
||||
if err != nil {
|
||||
return PowerOffUnknown, "cannot list the running guests: " + err.Error()
|
||||
}
|
||||
var others []string
|
||||
for _, v := range running {
|
||||
if v != vmid {
|
||||
others = append(others, strconv.Itoa(v))
|
||||
}
|
||||
}
|
||||
if len(others) > 0 {
|
||||
return PowerOffOtherGuests, "other guests or VMs running: " + strings.Join(others, ", ")
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// waitJobsClear waits until jobBlocker finds nothing, at most JobWait. false = still busy (or unreadable) at the end:
|
||||
// the power-off is CANCELLED — the box stays on, the gate is released, the log says why. Better a box that did not
|
||||
// switch off than a backup, update or format cut in the middle.
|
||||
func (s *Server) waitJobsClear(base context.Context, vmid int, when string) bool {
|
||||
c := s.powerOff
|
||||
ctx, cancel := context.WithTimeout(base, c.JobWait)
|
||||
defer cancel()
|
||||
for {
|
||||
// Another guest or VM started after acceptance (security review 2026-10-13): no waiting — cancel at once.
|
||||
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
|
||||
s.logger.Error("host-poweroff: CANCELLED — another guest is running now; the box stays on", "when", when,
|
||||
"reason", code, "detail", detail)
|
||||
return false
|
||||
}
|
||||
code, detail := s.jobBlocker(ctx, vmid)
|
||||
if code == "" {
|
||||
return true
|
||||
}
|
||||
s.logger.Warn("host-poweroff: waiting — a job started after the request", "when", when, "reason", code, "detail", detail)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
s.logger.Error("host-poweroff: CANCELLED — still busy after the wait; the box stays on", "when", when,
|
||||
"reason", code, "detail", detail, "waited", c.JobWait)
|
||||
return false
|
||||
case <-time.After(c.JobPoll):
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,353 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// fakePower records the power-off steps in order.
|
||||
type fakePower struct {
|
||||
mu sync.Mutex
|
||||
steps []string
|
||||
}
|
||||
|
||||
func (f *fakePower) ShutdownGuest(_ context.Context, vmid int, timeout time.Duration) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.steps = append(f.steps, "shutdown:"+itoaT(vmid)+":"+timeout.String())
|
||||
return nil
|
||||
}
|
||||
func (f *fakePower) PowerOffHost(context.Context) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.steps = append(f.steps, "poweroff")
|
||||
return nil
|
||||
}
|
||||
func (f *fakePower) got() []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]string(nil), f.steps...)
|
||||
}
|
||||
|
||||
func itoaT(n int) string { b, _ := json.Marshal(n); return string(b) }
|
||||
|
||||
type fakeVzdump struct{ running bool }
|
||||
|
||||
func (f fakeVzdump) Guests(context.Context) ([]proxmox.Guest, error) { return nil, nil }
|
||||
func (f fakeVzdump) Lock(context.Context, int) (string, bool, error) { return "", true, nil }
|
||||
func (f fakeVzdump) BackupRunning(context.Context, int) (bool, error) { return f.running, nil }
|
||||
func (f fakeVzdump) Unlock(context.Context, int) error { return nil }
|
||||
func (f fakeVzdump) Start(context.Context, int) error { return nil }
|
||||
func (f fakeVzdump) HasVzdumpSnapshot(context.Context, int) (bool, error) { return false, nil }
|
||||
func (f fakeVzdump) DeleteVzdumpSnapshot(context.Context, int) error { return nil }
|
||||
|
||||
type powerRig struct {
|
||||
srv *Server
|
||||
h http.Handler
|
||||
power *fakePower
|
||||
state *poweroff.State
|
||||
report *[]string
|
||||
}
|
||||
|
||||
func newPowerRig(t *testing.T, uptime time.Duration) *powerRig {
|
||||
t.Helper()
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
srv.inFlight = &backup.InFlight{}
|
||||
srv.staleLock = fakeVzdump{}
|
||||
p := &fakePower{}
|
||||
st := poweroff.New(filepath.Join(t.TempDir(), "poweroff.json"), func() string { return "1700000000" })
|
||||
var reports []string
|
||||
srv.SetHostPowerOff(HostPowerOffConfig{
|
||||
State: st,
|
||||
Power: p,
|
||||
ReportNow: func(context.Context) bool {
|
||||
// The record must already exist when the hub is told — that is the point of telling it.
|
||||
if st.Current() == nil {
|
||||
reports = append(reports, "report-without-record")
|
||||
} else if len(p.got()) != 0 {
|
||||
reports = append(reports, "report-after-"+p.got()[0])
|
||||
} else {
|
||||
reports = append(reports, "report")
|
||||
}
|
||||
return true
|
||||
},
|
||||
Uptime: func() (time.Duration, bool) { return uptime, true },
|
||||
Running: func() ([]int, error) { return []int{8200}, nil },
|
||||
Now: func() time.Time { return testNow },
|
||||
})
|
||||
return &powerRig{srv: srv, h: srv.Handler(), power: p, state: st, report: &reports}
|
||||
}
|
||||
|
||||
func (r *powerRig) waitSteps(t *testing.T, n int) []string {
|
||||
t.Helper()
|
||||
for i := 0; i < 200 && len(r.power.got()) < n; i++ {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
return r.power.got()
|
||||
}
|
||||
|
||||
func reasonOf(t *testing.T, body []byte) string {
|
||||
t.Helper()
|
||||
var resp struct {
|
||||
Data struct {
|
||||
Reason string `json:"reason"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
t.Fatalf("body %s: %v", body, err)
|
||||
}
|
||||
return resp.Data.Reason
|
||||
}
|
||||
|
||||
// The whole path: 202, the record written, the hub told FIRST, then guest shutdown (120 s), then host off.
|
||||
func TestHostPowerOff_AcceptedOrder(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusAccepted {
|
||||
t.Fatalf("got %d want 202 (%s)", w.Code, w.Body.String())
|
||||
}
|
||||
steps := r.waitSteps(t, 2)
|
||||
want := []string{"shutdown:8200:2m0s", "poweroff"}
|
||||
if len(steps) != 2 || steps[0] != want[0] || steps[1] != want[1] {
|
||||
t.Fatalf("steps = %v, want %v", steps, want)
|
||||
}
|
||||
// The heavy-operation gate stays taken: no backup can start between the checks and the power-off.
|
||||
if _, busy, ok := r.srv.inFlight.TryAcquire("backup:primary"); ok || busy != "host-poweroff" {
|
||||
t.Fatalf("after acceptance a backup could take the gate (ok=%v busy=%q)", ok, busy)
|
||||
}
|
||||
if len(*r.report) != 1 || (*r.report)[0] != "report" {
|
||||
t.Fatalf("hub report = %v, want one report carrying the record", *r.report)
|
||||
}
|
||||
if rec := r.state.Current(); rec == nil || rec.VMID != 8200 {
|
||||
t.Fatalf("record = %+v", rec)
|
||||
}
|
||||
}
|
||||
|
||||
// Refused during a backup — the shared heavy-operation gate held by a backup tier.
|
||||
func TestHostPowerOff_RefusedDuringBackup(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
release, _, ok := r.srv.inFlight.TryAcquire("backup:primary")
|
||||
if !ok {
|
||||
t.Fatal("could not hold the gate")
|
||||
}
|
||||
defer release()
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 0 {
|
||||
t.Fatalf("power steps ran during a backup: %v", s)
|
||||
}
|
||||
if r.state.Current() != nil {
|
||||
t.Fatal("a refused request must not use up the boot's one request")
|
||||
}
|
||||
// The same guard names a restore test and an OS step.
|
||||
release()
|
||||
for gate, want := range map[string]string{"restore-test": PowerOffRestoreTest, "os-pve-step": PowerOffOSUpdate} {
|
||||
rel, _, _ := r.srv.inFlight.TryAcquire(gate)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
rel()
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != want {
|
||||
t.Fatalf("%s: got %d %s, want 409 %s", gate, w.Code, w.Body.String(), want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A vzdump of the guest in flight (a Proxmox task, not the agent's own gate) refuses too.
|
||||
func TestHostPowerOff_RefusedDuringVzdump(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
r.srv.staleLock = fakeVzdump{running: true}
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
|
||||
}
|
||||
}
|
||||
|
||||
// One accepted request per boot: the second is refused and nothing runs twice.
|
||||
func TestHostPowerOff_SecondRequestSameBootRefused(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("first: %d", w.Code)
|
||||
}
|
||||
r.waitSteps(t, 2)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffAlreadyAsked {
|
||||
t.Fatalf("second: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffAlreadyAsked)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 2 {
|
||||
t.Fatalf("steps = %v, want exactly one run", s)
|
||||
}
|
||||
}
|
||||
|
||||
// Refused in the first 10 minutes after boot; an unknown uptime refuses too (fail closed).
|
||||
func TestHostPowerOff_TooSoonAfterBoot(t *testing.T) {
|
||||
r := newPowerRig(t, 9*time.Minute)
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffTooSoon {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffTooSoon)
|
||||
}
|
||||
r.srv.powerOff.Uptime = func() (time.Duration, bool) { return 0, false }
|
||||
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
|
||||
t.Fatalf("unknown uptime: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
|
||||
}
|
||||
}
|
||||
|
||||
// Not configured → 503; another guest's vmid in the body → 403; no token → 401.
|
||||
func TestHostPowerOff_ScopeAndConfig(t *testing.T) {
|
||||
plain := newTestServer(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
if w := do(t, plain, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("unconfigured: %d", w.Code)
|
||||
}
|
||||
r := newPowerRig(t, time.Hour)
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", `{"vmid":9300}`); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("cross-guest: %d", w.Code)
|
||||
}
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "", ""); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: %d", w.Code)
|
||||
}
|
||||
if r.state.Current() != nil || len(r.power.got()) != 0 {
|
||||
t.Fatal("a refused request changed state")
|
||||
}
|
||||
}
|
||||
|
||||
// Another guest or VM on the host refuses: one controller must not switch off someone else's guest (security
|
||||
// review 2026-10-13; demo-hp runs a scratch guest and the Tester 1 VM). An unreadable process list refuses too.
|
||||
func TestHostPowerOff_RefusedWhileOtherGuestsRun(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
r.srv.powerOff.Running = func() ([]int, error) { return []int{8200, 9202, 341}, nil }
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffOtherGuests {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffOtherGuests)
|
||||
}
|
||||
r.srv.powerOff.Running = func() ([]int, error) { return nil, errors.New("no /proc") }
|
||||
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
|
||||
t.Fatalf("unreadable: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
|
||||
}
|
||||
if len(r.power.got()) != 0 || r.state.Current() != nil {
|
||||
t.Fatal("a refused request changed state")
|
||||
}
|
||||
}
|
||||
|
||||
// Fail closed: an unwired vzdump reader refuses instead of skipping the check.
|
||||
func TestHostPowerOff_FailsClosedWithoutTaskReader(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
r.srv.staleLock = nil
|
||||
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
|
||||
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
|
||||
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
|
||||
}
|
||||
}
|
||||
|
||||
// The /proc reader finds containers and VMs by their real command lines (as measured on demo-hp).
|
||||
func TestRunningGuestsFromProc(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
write := func(pid, cmd string) {
|
||||
_ = os.MkdirAll(filepath.Join(dir, pid), 0o755)
|
||||
_ = os.WriteFile(filepath.Join(dir, pid, "cmdline"), []byte(strings.ReplaceAll(cmd, " ", "\x00")+"\x00"), 0o644)
|
||||
}
|
||||
write("1927", "/usr/bin/lxc-start -F -n 9201")
|
||||
write("1900", "/usr/bin/lxc-start -F -n 9202")
|
||||
write("1823", "/usr/bin/kvm -id 341 -name night1004-tester1 -no-shutdown")
|
||||
write("974", "/usr/libexec/lxc/lxc-monitord --daemon")
|
||||
write("50", "/usr/bin/bash -c echo -n 5")
|
||||
_ = os.MkdirAll(filepath.Join(dir, "self"), 0o755)
|
||||
got, err := runningGuestsFromProc(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sort.Ints(got)
|
||||
if len(got) != 3 || got[0] != 341 || got[1] != 9201 || got[2] != 9202 {
|
||||
t.Fatalf("got %v, want [341 9201 9202]", got)
|
||||
}
|
||||
if _, err := runningGuestsFromProc(filepath.Join(dir, "missing")); err == nil {
|
||||
t.Fatal("an unreadable /proc must be an error")
|
||||
}
|
||||
}
|
||||
|
||||
// switchVzdump is a vzdump reader the test can flip while the power-off runs.
|
||||
type switchVzdump struct {
|
||||
fakeVzdump
|
||||
on *atomic.Bool
|
||||
}
|
||||
|
||||
func (f switchVzdump) BackupRunning(context.Context, int) (bool, error) { return f.on.Load(), nil }
|
||||
|
||||
// A job that starts AFTER acceptance (security review 2026-10-13): the power-off waits; if the job is still running
|
||||
// when the wait ends, the power-off is cancelled — no guest shutdown, no host power-off, the gate given back.
|
||||
func TestHostPowerOff_JobStartedAfterAcceptance(t *testing.T) {
|
||||
// (a) it clears during the wait → the power-off goes ahead after it
|
||||
r := newPowerRig(t, time.Hour)
|
||||
var on atomic.Bool
|
||||
r.srv.staleLock = switchVzdump{on: &on}
|
||||
r.srv.powerOff.JobWait, r.srv.powerOff.JobPoll = 2*time.Second, 2*time.Millisecond
|
||||
r.srv.powerOff.ReportNow = func(context.Context) bool { on.Store(true); return true } // a vzdump starts now
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
time.Sleep(30 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 0 {
|
||||
t.Fatalf("a step ran while the vzdump ran: %v", s)
|
||||
}
|
||||
on.Store(false)
|
||||
if s := r.waitSteps(t, 2); len(s) != 2 {
|
||||
t.Fatalf("after the vzdump ended the steps = %v, want shutdown + poweroff", s)
|
||||
}
|
||||
|
||||
// (b) it never clears → cancelled, nothing switched off, the heavy gate free again
|
||||
r2 := newPowerRig(t, time.Hour)
|
||||
var on2 atomic.Bool
|
||||
r2.srv.staleLock = switchVzdump{on: &on2}
|
||||
r2.srv.powerOff.JobWait, r2.srv.powerOff.JobPoll = 40*time.Millisecond, 2*time.Millisecond
|
||||
r2.srv.powerOff.ReportNow = func(context.Context) bool { on2.Store(true); return true }
|
||||
if w := do(t, r2.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("accept: %d", w.Code)
|
||||
}
|
||||
time.Sleep(150 * time.Millisecond)
|
||||
if s := r2.power.got(); len(s) != 0 {
|
||||
t.Fatalf("cancelled power-off still ran: %v", s)
|
||||
}
|
||||
if busy := r2.srv.inFlight.Busy(); busy != "" {
|
||||
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
|
||||
}
|
||||
}
|
||||
|
||||
// A guest or VM started AFTER acceptance cancels the power-off at once (security review 2026-10-13).
|
||||
func TestHostPowerOff_OtherGuestStartedAfterAcceptance(t *testing.T) {
|
||||
r := newPowerRig(t, time.Hour)
|
||||
var started atomic.Bool
|
||||
r.srv.powerOff.Running = func() ([]int, error) {
|
||||
if started.Load() {
|
||||
return []int{8200, 341}, nil
|
||||
}
|
||||
return []int{8200}, nil
|
||||
}
|
||||
r.srv.powerOff.ReportNow = func(context.Context) bool { started.Store(true); return true } // a VM starts now
|
||||
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
|
||||
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
if s := r.power.got(); len(s) != 0 {
|
||||
t.Fatalf("power-off ran although a VM started: %v", s)
|
||||
}
|
||||
if busy := r.srv.inFlight.Busy(); busy != "" {
|
||||
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// The internal data place („Belső tárhely", agent v0.158.0) — felhom.eu architecture/16-storage-physical-identity.md
|
||||
// §5, §7. On a one-disk box the root helper configs/felhom-internal-place makes a second thin volume in local-lvm
|
||||
// (<vg>/felhom-belso, ext4, label felhom-belso). The installer (a new box) or the signed add_internal_place job (an old
|
||||
// box) then writes InternalPlaceRecord to the agent's state dir. From then on the agent treats the volume exactly like
|
||||
// an enrolled drive: a Felhom mount unit by fs-UUID at /mnt/belso (EnsureMount), the `enrolled` intent, and — when the
|
||||
// controller asks with its own token (POST /disks/guest-attach) — the bind at /mnt/felhom-drives/belso. So it is a
|
||||
// HOST mount under the guest's mp8 parent: not in the guest config, not in vzdump, not in PBS (`16` §5).
|
||||
//
|
||||
// What it is NOT: a drive the household can format, wipe, eject or decommission. Its device is a device-mapper volume
|
||||
// on the system disk, so the role gate (roleForMountPath → system) and the format path (ValidateBlockDevice refuses
|
||||
// dm) refuse all four — pinned by internalplace_test.go. Its identity is the system disk's (build A), so a Tier-2 copy
|
||||
// between it and sys_drive is never a second copy.
|
||||
const (
|
||||
InternalPlaceName = "belso"
|
||||
InternalPlaceWhere = "/mnt/belso"
|
||||
// InternalPlaceFile is the record's name inside the agent's state dir.
|
||||
InternalPlaceFile = "internal-place.json"
|
||||
)
|
||||
|
||||
var reFSUUID = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||
|
||||
// InternalPlaceRecord is the state file the maker writes once the volume exists and is formatted.
|
||||
type InternalPlaceRecord struct {
|
||||
FSUUID string `json:"fs_uuid"`
|
||||
GiB int `json:"gib,omitempty"`
|
||||
MadeBy string `json:"made_by,omitempty"` // "installer" | "signed-job"
|
||||
MadeAt string `json:"made_at,omitempty"`
|
||||
}
|
||||
|
||||
// ReadInternalPlace reads the record; ok=false when there is none (the common case: a box with no internal place).
|
||||
func ReadInternalPlace(file string) (InternalPlaceRecord, bool, error) {
|
||||
var rec InternalPlaceRecord
|
||||
b, err := os.ReadFile(file)
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return rec, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return rec, false, err
|
||||
}
|
||||
if err := json.Unmarshal(b, &rec); err != nil {
|
||||
return rec, false, fmt.Errorf("internal place record %s: %w", file, err)
|
||||
}
|
||||
if !reFSUUID.MatchString(rec.FSUUID) {
|
||||
return rec, false, fmt.Errorf("internal place record %s: fs_uuid %q is not a filesystem UUID", file, rec.FSUUID)
|
||||
}
|
||||
return rec, true, nil
|
||||
}
|
||||
|
||||
// WriteInternalPlace writes the record atomically (0600).
|
||||
func WriteInternalPlace(file string, rec InternalPlaceRecord) error {
|
||||
if !reFSUUID.MatchString(rec.FSUUID) {
|
||||
return fmt.Errorf("internal place: fs_uuid %q is not a filesystem UUID", rec.FSUUID)
|
||||
}
|
||||
b, err := json.MarshalIndent(rec, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := file + ".tmp"
|
||||
if err := os.WriteFile(tmp, append(b, '\n'), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, file)
|
||||
}
|
||||
|
||||
// internalPlaceState memoises what EnsureInternalPlace last saw, so the 20-second tick logs a state CHANGE once and
|
||||
// does no work on the steady state.
|
||||
type internalPlaceState struct {
|
||||
mu sync.Mutex
|
||||
done string // the fs-UUID that is mounted and recorded enrolled
|
||||
lastWhy string // the last reason it could not be ensured (logged once per change)
|
||||
}
|
||||
|
||||
// internalPlaceUUID returns the configured internal place's fs-UUID, or "" when this box has none (or the record is
|
||||
// unreadable — logged by EnsureInternalPlace).
|
||||
func (s *Server) internalPlaceUUID() string {
|
||||
if s.internalPlaceFile == "" {
|
||||
return ""
|
||||
}
|
||||
rec, ok, err := ReadInternalPlace(s.internalPlaceFile)
|
||||
if err != nil || !ok {
|
||||
return ""
|
||||
}
|
||||
return rec.FSUUID
|
||||
}
|
||||
|
||||
// isInternalPlace reports whether a host path is this box's internal place: its raw mount (/mnt/belso) or its bind
|
||||
// under the guests' shared parent (/mnt/felhom-drives/belso — the path POST /disks/same-disk judges).
|
||||
func (s *Server) isInternalPlace(hostPath string) bool {
|
||||
p := path.Clean(hostPath)
|
||||
return (p == InternalPlaceWhere || p == StableParentDir+"/"+InternalPlaceName) && s.internalPlaceUUID() != ""
|
||||
}
|
||||
|
||||
// EnsureInternalPlace mounts the internal place on the host at /mnt/belso by fs-UUID and records it `enrolled`,
|
||||
// when this box has one. Idempotent and cheap on the steady state; safe on the 20-second tick. It never formats,
|
||||
// never unmounts, and never mounts anything at /mnt/belso that is not the recorded UUID: a household drive that
|
||||
// happens to be named „belso" keeps its mount, and the internal place waits (logged once).
|
||||
func (s *Server) EnsureInternalPlace(ctx context.Context) {
|
||||
if s.internalPlaceFile == "" {
|
||||
return
|
||||
}
|
||||
rec, ok, err := ReadInternalPlace(s.internalPlaceFile)
|
||||
if err != nil {
|
||||
s.internalPlaceNote("record unreadable: " + err.Error())
|
||||
return
|
||||
}
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
uuid := rec.FSUUID
|
||||
st := &s.ipState
|
||||
st.mu.Lock()
|
||||
done := st.done == uuid
|
||||
st.mu.Unlock()
|
||||
|
||||
// What is at /mnt/belso now, if anything?
|
||||
mountedUUID, mounted := "", false
|
||||
if mounts, merr := s.hostReader().Mounts(); merr == nil {
|
||||
for _, m := range mounts {
|
||||
if path.Clean(m.MountPoint) == InternalPlaceWhere {
|
||||
mounted = true
|
||||
if u, okU := s.hostReader().ResolveUUID(m.Device); okU {
|
||||
mountedUUID = u
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if mounted && mountedUUID != "" && mountedUUID != uuid {
|
||||
s.internalPlaceNote("another filesystem (uuid " + mountedUUID + ") is mounted at " + InternalPlaceWhere + " — the internal place waits")
|
||||
return
|
||||
}
|
||||
// A Felhom unit for /mnt/belso that names another UUID (a household drive called „belso", unplugged): never
|
||||
// overwrite it — that would silently take the drive's name and mount point.
|
||||
if s.driveTargets != nil {
|
||||
if drives, derr := s.driveTargets.Known(ctx); derr == nil {
|
||||
for _, d := range drives {
|
||||
if path.Clean(d.MountPath) == InternalPlaceWhere && d.UUID != "" && d.UUID != uuid {
|
||||
s.internalPlaceNote("a drive unit for " + InternalPlaceWhere + " names another uuid (" + d.UUID + ") — the internal place waits")
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if mounted && mountedUUID == uuid {
|
||||
if !done {
|
||||
s.recordIntent(ctx, InternalPlaceWhere, "enrolled")
|
||||
st.mu.Lock()
|
||||
st.done, st.lastWhy = uuid, ""
|
||||
st.mu.Unlock()
|
||||
s.logger.Info("internal place: mounted and enrolled", "where", InternalPlaceWhere, "fs_uuid", uuid, "gib", rec.GiB)
|
||||
}
|
||||
return
|
||||
}
|
||||
if s.disks == nil {
|
||||
s.internalPlaceNote("disk management is not configured on this host")
|
||||
return
|
||||
}
|
||||
if s.resolveStorageDevice != nil {
|
||||
if _, rerr := s.resolveStorageDevice("uuid:" + uuid); rerr != nil {
|
||||
s.internalPlaceNote("its volume is not present (uuid " + uuid + ")")
|
||||
return
|
||||
}
|
||||
}
|
||||
start := time.Now()
|
||||
if err := s.disks.EnsureMount(ctx, storage.MountSpec{Name: InternalPlaceName, UUID: uuid, Where: InternalPlaceWhere, FSType: "ext4"}); err != nil {
|
||||
s.internalPlaceNote("mount failed: " + err.Error())
|
||||
return
|
||||
}
|
||||
s.recordIntent(ctx, InternalPlaceWhere, "enrolled")
|
||||
st.mu.Lock()
|
||||
st.done, st.lastWhy = uuid, ""
|
||||
st.mu.Unlock()
|
||||
s.logger.Info("internal place: mounted on the host", "where", InternalPlaceWhere, "fs_uuid", uuid, "gib", rec.GiB,
|
||||
"duration_ms", time.Since(start).Milliseconds())
|
||||
}
|
||||
|
||||
func (s *Server) internalPlaceNote(why string) {
|
||||
st := &s.ipState
|
||||
st.mu.Lock()
|
||||
changed := st.lastWhy != why
|
||||
st.lastWhy, st.done = why, ""
|
||||
st.mu.Unlock()
|
||||
if changed {
|
||||
s.logger.Warn("internal place: not ensured", "why", why)
|
||||
} else {
|
||||
s.logger.Debug("internal place: still not ensured", "why", why)
|
||||
}
|
||||
}
|
||||
|
||||
// InternalPlaceFileIn is the record's path inside a state dir.
|
||||
func InternalPlaceFileIn(stateDir string) string { return filepath.Join(stateDir, InternalPlaceFile) }
|
||||
@@ -0,0 +1,249 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// The internal data place (felhom.eu architecture/16 §5, agent v0.158.0): mounted like an enrolled drive, but never
|
||||
// at the cost of a household drive, never formatted, ejected or decommissioned, and the same disk as sys_drive.
|
||||
|
||||
const ipUUID = "0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c"
|
||||
const ipDev = "/dev/mapper/pve-felhom--belso"
|
||||
|
||||
// ipHost is a HostReader whose mount table the test changes, and whose by-uuid map it controls.
|
||||
type ipHost struct {
|
||||
mu sync.Mutex
|
||||
mounts []storage.Mount
|
||||
uuids map[string]string // device → fs-UUID
|
||||
}
|
||||
|
||||
func (h *ipHost) Mounts() ([]storage.Mount, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
return append([]storage.Mount(nil), h.mounts...), nil
|
||||
}
|
||||
func (h *ipHost) ResolveUUID(dev string) (string, bool) { u, ok := h.uuids[dev]; return u, ok }
|
||||
func (h *ipHost) DeviceExists(string) bool { return true }
|
||||
func (h *ipHost) Rotational(string) (bool, bool) { return false, false }
|
||||
func (h *ipHost) Removable(string) (bool, bool) { return false, false }
|
||||
func (h *ipHost) BlockSlaves(string) ([]string, bool) { return nil, false }
|
||||
|
||||
func newIPHost(extra ...storage.Mount) *ipHost {
|
||||
return &ipHost{
|
||||
// root on a whole-disk partition (as sysOnSDA): the system disk resolves, so a USB drive is user-data.
|
||||
mounts: append([]storage.Mount{{Device: "/dev/sda1", MountPoint: "/", FSType: "ext4"}}, extra...),
|
||||
uuids: map[string]string{ipDev: ipUUID, "/dev/sdb1": "99999999-aaaa-bbbb-cccc-dddddddddddd"},
|
||||
}
|
||||
}
|
||||
|
||||
type ipRig struct {
|
||||
srv *Server
|
||||
disks *fakeDiskOps
|
||||
intent *fakeIntent
|
||||
}
|
||||
|
||||
func newIPRig(t *testing.T, host *ipHost, known []storage.KnownTarget, record bool) ipRig {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, InternalPlaceFile)
|
||||
if record {
|
||||
if err := WriteInternalPlace(file, InternalPlaceRecord{FSUUID: ipUUID, GiB: 183, MadeBy: "installer"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
d, in := &fakeDiskOps{}, newFakeIntent()
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{}, Tokens: staticTokens{"A": 8200}, Disks: d, DiskGate: &fakeGate{},
|
||||
HostReader: host, Intent: in, DriveTargets: fakeKnownTargets{drives: known},
|
||||
InternalPlaceFile: file, Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.resolveStorageDevice = func(id string) (string, error) {
|
||||
if id == "uuid:"+ipUUID {
|
||||
return "/dev/dm-7", nil
|
||||
}
|
||||
return "", errors.New("absent")
|
||||
}
|
||||
return ipRig{srv, d, in}
|
||||
}
|
||||
|
||||
func TestInternalPlace_NoRecord_DoesNothing(t *testing.T) {
|
||||
r := newIPRig(t, newIPHost(), nil, false)
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 0 {
|
||||
t.Fatalf("no record: no mount may be made, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalPlace_Record_MountsByUUIDAtMntBelso(t *testing.T) {
|
||||
r := newIPRig(t, newIPHost(), nil, true)
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 1 {
|
||||
t.Fatalf("want exactly one EnsureMount, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
got := r.disks.mountCalls[0]
|
||||
if got.UUID != ipUUID || got.Where != InternalPlaceWhere || got.FSType != "ext4" || got.Name != InternalPlaceName {
|
||||
t.Fatalf("EnsureMount spec: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalPlace_AlreadyMounted_EnrolledOnceNoRemount(t *testing.T) {
|
||||
host := newIPHost(storage.Mount{Device: ipDev, MountPoint: InternalPlaceWhere, FSType: "ext4"})
|
||||
r := newIPRig(t, host, nil, true)
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 0 {
|
||||
t.Fatalf("already mounted: no EnsureMount, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
if r.intent.Get("uuid:"+ipUUID) != storage.IntentEnrolled {
|
||||
t.Fatalf("the internal place must be recorded enrolled")
|
||||
}
|
||||
}
|
||||
|
||||
// A household drive called „belso" keeps its mount point: the internal place never takes it.
|
||||
// Positive control: TestInternalPlace_Record_MountsByUUIDAtMntBelso (the same record, /mnt/belso free).
|
||||
func TestInternalPlace_NeverTakesAHouseholdDriveNamedBelso(t *testing.T) {
|
||||
mounted := newIPHost(storage.Mount{Device: "/dev/sdb1", MountPoint: InternalPlaceWhere, FSType: "ext4"})
|
||||
r := newIPRig(t, mounted, nil, true)
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 0 || r.intent.Get("uuid:"+ipUUID) != "" {
|
||||
t.Fatalf("a foreign filesystem at /mnt/belso: nothing may be mounted or recorded, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
unplugged := []storage.KnownTarget{{Name: "belso", UUID: "99999999-aaaa-bbbb-cccc-dddddddddddd", MountPath: InternalPlaceWhere, MountBacked: true}}
|
||||
r2 := newIPRig(t, newIPHost(), unplugged, true)
|
||||
r2.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r2.disks.mountCalls) != 0 {
|
||||
t.Fatalf("an unplugged household drive's unit for /mnt/belso must not be overwritten, got %+v", r2.disks.mountCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalPlace_VolumeAbsent_NoMount(t *testing.T) {
|
||||
r := newIPRig(t, newIPHost(), nil, true)
|
||||
r.srv.resolveStorageDevice = func(string) (string, error) { return "", errors.New("absent") }
|
||||
r.srv.EnsureInternalPlace(context.Background())
|
||||
if len(r.disks.mountCalls) != 0 {
|
||||
t.Fatalf("absent volume: no mount, got %+v", r.disks.mountCalls)
|
||||
}
|
||||
}
|
||||
|
||||
// The household cannot eject or decommission it: its device is a device-mapper volume on the system disk.
|
||||
// Positive control: the same calls for a whole-disk USB drive (/dev/sdb1) are not refused by role.
|
||||
func TestInternalPlace_EjectAndDecommissionRefused(t *testing.T) {
|
||||
host := newIPHost(
|
||||
storage.Mount{Device: ipDev, MountPoint: InternalPlaceWhere, FSType: "ext4"},
|
||||
storage.Mount{Device: "/dev/sdb1", MountPoint: "/mnt/hdd_1", FSType: "ext4"},
|
||||
)
|
||||
r := newIPRig(t, host, nil, true)
|
||||
h := r.srv.Handler()
|
||||
for _, route := range []string{"/disks/eject", "/disks/decommission"} {
|
||||
if w := do(t, h, "POST", route, "A", `{"where":"/mnt/belso"}`); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("%s /mnt/belso: want 403, got %d %s", route, w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
if w := do(t, h, "POST", "/disks/eject", "A", `{"where":"/mnt/hdd_1"}`); w.Code == http.StatusForbidden {
|
||||
t.Fatalf("positive control: a USB drive's eject must not be refused by role, got %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
for _, u := range r.disks.unmountCalls {
|
||||
if u == InternalPlaceWhere {
|
||||
t.Fatalf("the internal place must never be unmounted, got %+v", r.disks.unmountCalls)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The consequence `16` §5 needs: a copy between the internal place and sys_drive is the SAME disk (build A then never
|
||||
// counts it), and the place is reported as kind internal.
|
||||
type ipIdentHost struct{ fakeIdentHost }
|
||||
|
||||
func (h *ipIdentHost) KernelName(dev string) (string, bool) {
|
||||
if dev == ipDev {
|
||||
return "dm-7", true
|
||||
}
|
||||
return h.fakeIdentHost.KernelName(dev)
|
||||
}
|
||||
func (h *ipIdentHost) Slaves(n string) []string {
|
||||
if n == "dm-7" {
|
||||
return []string{"dm-3"} // felhom-belso → the thin pool
|
||||
}
|
||||
if n == "dm-3" {
|
||||
return []string{"sda3"} // the pool → the system disk's LVM partition
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestInternalPlace_IsTheSystemDiskAndKindInternal(t *testing.T) {
|
||||
ih := &ipIdentHost{fakeIdentHost{
|
||||
virt: "none",
|
||||
parent: map[string]string{"sda3": "sda"},
|
||||
props: map[string]map[string]string{"sda": {"ID_BUS": "ata", "ID_MODEL": "SanDisk", "ID_SERIAL_SHORT": "1"}},
|
||||
mounts: []storage.Mount{
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"},
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/mnt/felhom-drives", FSType: "ext4"},
|
||||
{Device: ipDev, MountPoint: InternalPlaceWhere, FSType: "ext4"},
|
||||
{Device: ipDev, MountPoint: "/mnt/felhom-drives/belso", FSType: "ext4"},
|
||||
},
|
||||
}}
|
||||
fg := &fakeGuestsCfg{mounts: map[int]map[string]string{8200: {
|
||||
"mp0": "local-lvm:vm-8200-disk-1,mp=/var/lib/felhom,backup=1,size=100G",
|
||||
"mp8": "/mnt/felhom-drives,mp=/mnt/felhom-drives",
|
||||
}}}
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, InternalPlaceFile)
|
||||
if err := WriteInternalPlace(file, InternalPlaceRecord{FSUUID: ipUUID}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: fg, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{provenSSD}}, Tokens: staticTokens{"A": 8200},
|
||||
HostReader: fakeHostReader{mounts: ih.mounts}, Identity: &storage.IdentityResolver{Host: ih},
|
||||
InternalPlaceFile: file, Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
got, code := sameDisk(t, srv.Handler(), "/mnt/felhom-drives/belso/felhom-data/jellyfin", "/mnt/sys_drive")
|
||||
if code != http.StatusOK || got.Different || !got.Proven || got.Reason != "same_disk" {
|
||||
t.Fatalf("internal place vs sys_drive: want same disk, proven; got %d %+v", code, got)
|
||||
}
|
||||
if got.A.Kind != storage.PlaceKindInternal {
|
||||
t.Fatalf("kind: want %q, got %q", storage.PlaceKindInternal, got.A.Kind)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalPlaceRecord_RefusesANonUUID(t *testing.T) {
|
||||
file := filepath.Join(t.TempDir(), InternalPlaceFile)
|
||||
if err := WriteInternalPlace(file, InternalPlaceRecord{FSUUID: "../../etc"}); err == nil {
|
||||
t.Fatal("a non-UUID must not be written")
|
||||
}
|
||||
}
|
||||
|
||||
// The root helper's own tests (configs/test_felhom_internal_place.py — the ruled size table, never formatting what
|
||||
// holds anything) run with the Go suite.
|
||||
func TestInternalPlaceHelperPythonSuite(t *testing.T) {
|
||||
py, err := exec.LookPath("python3")
|
||||
if err != nil {
|
||||
t.Skip("python3 not found")
|
||||
}
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
script := filepath.Join(filepath.Dir(file), "..", "..", "configs", "test_felhom_internal_place.py")
|
||||
out, err := exec.Command(py, script).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("helper suite failed: %v\n%s", err, out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// stdinRecorder is a proxmox.Runner that records each call and the stdin it was handed.
|
||||
type stdinRecorder struct {
|
||||
name string
|
||||
args []string
|
||||
stdin string
|
||||
}
|
||||
|
||||
func (r *stdinRecorder) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.name, r.args = name, args
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
func (r *stdinRecorder) RunStdin(_ context.Context, stdin io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
b, _ := io.ReadAll(stdin)
|
||||
r.name, r.args, r.stdin = name, args, string(b)
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
// R-861 (a) A1 (`09` §3 decision 165): the managed controller update writes the guest's image file through the ROOT
|
||||
// verb, never through an in-guest `tee` the agent could feed any image.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): restore the pre-A1 body (`b.runner.RunStdin(ctx, …, "pct", "exec", vmid, "--",
|
||||
// "tee", controllerImageFile)`) → this fails with "the image write ran pct …, want felhom-priv-apply". Restored.
|
||||
func TestR861_WriteControllerImageUsesTheRootVerb(t *testing.T) {
|
||||
rec := &stdinRecorder{}
|
||||
b := NewGuestBinder(rec, discardLogger())
|
||||
const img = "gitea.dooplex.hu/admin/felhom-controller:0.302.0"
|
||||
if err := b.WriteControllerImage(context.Background(), 9201, img); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if rec.name != privApplyBin {
|
||||
t.Fatalf("the image write ran %s %v, want felhom-priv-apply", rec.name, rec.args)
|
||||
}
|
||||
if len(rec.args) != 2 || rec.args[0] != "controller-image" || rec.args[1] != "9201" {
|
||||
t.Fatalf("argv = %v, want [controller-image 9201] (the sudoers line `^controller-image [0-9]+$`)", rec.args)
|
||||
}
|
||||
if rec.stdin != img+"\n" {
|
||||
t.Fatalf("stdin = %q, want the ref plus one newline", rec.stdin)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-894 — after an agent restart, an UNREADABLE storage must fall back to the last success saved on
|
||||
// disk, not to "never". Measured 2026-10-05 on demo-hp: a restart at 04:57, the off-site storage
|
||||
// unreachable at 06:25, the 7-day tier (last copy 4 days old) read DUE, vzdump failed.
|
||||
//
|
||||
// Every test here builds a NEW server and a NEW BackupSuccessState from the same file — that is the
|
||||
// restart. The in-memory store (fakeStore) is always fresh, as after a real restart.
|
||||
|
||||
// r894Server builds a two-tier server whose off-site tier answers the storage listing with lister.
|
||||
func r894Server(t *testing.T, path string, pbsSvc BackupService) *Server {
|
||||
t.Helper()
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local"}, {Name: "felhom-pbs"}}},
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
BackupTiers: []BackupTier{
|
||||
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
|
||||
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: pbsSvc},
|
||||
},
|
||||
LastKnownBackups: backup.NewBackupSuccessState(path),
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.now = func() time.Time { return testNow }
|
||||
return srv
|
||||
}
|
||||
|
||||
// unreadable is the off-site storage as demo-hp saw it: "Can't connect to 10.77.0.1:8007".
|
||||
func unreadable() archiveLister {
|
||||
return archiveLister{fakeBackups: &fakeBackups{}, err: errStorageRead}
|
||||
}
|
||||
|
||||
// THE R-894 CASE, end to end. Agent 1 takes an off-site backup through POST /backup (the fake
|
||||
// runner's success is 12 h before testNow). The agent restarts. The storage cannot be read. The tier
|
||||
// must read NOT due, from the copy saved on disk.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): delete the `lookup == archiveUnknown && s.lastKnown != nil` block in
|
||||
// handleBackupDue → this fails with "after a restart an unreadable storage must fall back to the saved
|
||||
// copy (12 h old, 7-day tier) — NOT due; got {… Due:true … AgeState:unknown …}". Restored.
|
||||
func TestBackupDue_R894_RestartThenUnreadableStorage_FreshSavedCopyIsNotDue(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
|
||||
// Agent 1: a real backup job through the endpoint the controller calls.
|
||||
first := r894Server(t, path, &fakeBackups{})
|
||||
if rr := do(t, first.Handler(), "POST", "/backup?target=felhom-pbs", "A", ""); rr.Code != http.StatusAccepted {
|
||||
t.Fatalf("POST /backup: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
waitFor(t, func() bool {
|
||||
_, ok := backup.NewBackupSuccessState(path).LastKnownSuccess("felhom-pbs", 8200)
|
||||
return ok
|
||||
})
|
||||
|
||||
// Agent 2: a restart (new server, new state from the same file), and the storage is unreachable.
|
||||
got := dueFor(t, r894Server(t, path, unreadable()).Handler(), "felhom-pbs")
|
||||
if got.Due {
|
||||
t.Fatalf("after a restart an unreadable storage must fall back to the saved copy (12 h old, 7-day tier) — NOT due; got %+v", got)
|
||||
}
|
||||
if got.AgeState != AgeStateKnown || got.AgeSecs == nil || *got.AgeSecs != int64((12*time.Hour).Seconds()) {
|
||||
t.Fatalf("the age must come from the saved copy (12 h, known); got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The deliberate rule stays: an unreadable storage must not suppress a backup that IS due. A saved
|
||||
// copy older than the cadence reads DUE.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make the fallback answer not-due whenever a saved copy exists
|
||||
// (`if fromDisk { …Due:false… }` before the cadence check) → this fails with "a saved copy 9 days old
|
||||
// under a 7-day cadence MUST read due". Restored.
|
||||
func TestBackupDue_R894_RestartThenUnreadableStorage_OldSavedCopyIsDue(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
st := backup.NewBackupSuccessState(path)
|
||||
if err := st.RecordBackupSuccess("felhom-pbs", backupAt("felhom-pbs", 8200, 9*24*time.Hour, true)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := dueFor(t, r894Server(t, path, unreadable()).Handler(), "felhom-pbs")
|
||||
if !got.Due {
|
||||
t.Fatalf("a saved copy 9 days old under a 7-day cadence MUST read due; got %+v", got)
|
||||
}
|
||||
if got.AgeState != AgeStateKnown {
|
||||
t.Fatalf("the age is known (from disk); got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// No saved copy → the pre-R-894 answer, byte for byte: DUE, age UNKNOWN (never ABSENT — the controller
|
||||
// fires its window-gate valve only on absent, R-88).
|
||||
func TestBackupDue_R894_RestartThenUnreadableStorage_NoSavedCopyIsDueUnknown(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
got := dueFor(t, r894Server(t, path, unreadable()).Handler(), "felhom-pbs")
|
||||
if !got.Due || got.AgeState != AgeStateUnknown || got.AgeSecs != nil {
|
||||
t.Fatalf("no saved copy + unreadable storage must stay DUE with age unknown; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A storage that ANSWERS is the ground truth: an archive absent there makes the tier due even when the
|
||||
// file remembers a fresh success (a pruned or deleted copy must be made again).
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): drop `lookup == archiveUnknown &&` from the fallback condition → this
|
||||
// fails with "the storage answered 'no archive' — the saved copy must NOT stand in for it". Restored.
|
||||
func TestBackupDue_R894_SavedCopyIgnoredWhenStorageAnswers(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
st := backup.NewBackupSuccessState(path)
|
||||
if err := st.RecordBackupSuccess("felhom-pbs", backupAt("felhom-pbs", 8200, time.Hour, true)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
absent := archiveLister{fakeBackups: &fakeBackups{}, found: false}
|
||||
got := dueFor(t, r894Server(t, path, absent).Handler(), "felhom-pbs")
|
||||
if !got.Due {
|
||||
t.Fatalf("the storage answered 'no archive' — the saved copy must NOT stand in for it; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A FAILED backup is never saved: it must not make a tier look fresh after a restart.
|
||||
func TestBackupDue_R894_FailedBackupIsNotSaved(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
first := r894Server(t, path, &fakeBackups{failErr: "could not activate storage 'felhom-pbs'"})
|
||||
if rr := do(t, first.Handler(), "POST", "/backup?target=felhom-pbs", "A", ""); rr.Code != http.StatusAccepted {
|
||||
t.Fatalf("POST /backup: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
waitFor(t, func() bool { return len(first.store.Backups(context.Background())) == 1 })
|
||||
if _, ok := backup.NewBackupSuccessState(path).LastKnownSuccess("felhom-pbs", 8200); ok {
|
||||
t.Fatal("a failed backup must never be saved as a success")
|
||||
}
|
||||
got := dueFor(t, r894Server(t, path, unreadable()).Handler(), "felhom-pbs")
|
||||
if !got.Due {
|
||||
t.Fatalf("after a failed backup and a restart the tier must still be due; got %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"path"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// POST /disks/same-disk (v0.157.0) — felhom.eu architecture/16-storage-physical-identity.md §4.
|
||||
//
|
||||
// The controller asks whether two of ITS paths sit on the same physical disk; the agent answers from its
|
||||
// own /sys + udev read, never from the caller's claim (`16` §10). A guest path is mapped only to a place
|
||||
// the agent itself made:
|
||||
// - /mnt/felhom-drives/<name>[/…] → the host mount at exactly /mnt/felhom-drives/<name> (the drive's
|
||||
// felhom-data bind, or a network share's mount). A path whose <name> is not mounted there (an absent
|
||||
// drive: the bare directory on the root disk) is UNMAPPED — never "the system disk".
|
||||
// - /mnt/sys_drive, /var/lib/felhom, /var/lib/docker [/…] → the guest's data volume (mp0, mp=/var/lib/felhom;
|
||||
// or a legacy mp at that exact path) → the PVE storage holding it → that storage's identity.
|
||||
// - anything else → unmapped.
|
||||
// Answer: {different, proven, reason, a, b}. Act on `proven` first: different+proven = a counted second
|
||||
// copy; same_disk+proven = not a second copy; proven=false = unknown (reason vm|network|usb_bridge|
|
||||
// weak_wwn|no_serial|unmapped). Pinned by samedisk_test.go.
|
||||
|
||||
type sameDiskRequest struct {
|
||||
VMID int `json:"vmid,omitempty"`
|
||||
A string `json:"a"`
|
||||
B string `json:"b"`
|
||||
}
|
||||
|
||||
// SamePlace is one side of the answer.
|
||||
type SamePlace struct {
|
||||
Path string `json:"path"`
|
||||
Mapped bool `json:"mapped"`
|
||||
Kind string `json:"kind,omitempty"`
|
||||
PhysicalDisks []string `json:"physical_disks,omitempty"`
|
||||
Identity string `json:"identity,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// SameDiskResponse is the wire answer.
|
||||
type SameDiskResponse struct {
|
||||
Different bool `json:"different"`
|
||||
Proven bool `json:"proven"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
A SamePlace `json:"a"`
|
||||
B SamePlace `json:"b"`
|
||||
}
|
||||
|
||||
// sysDataPaths are the guest paths that live on the guest's one data volume (golden v3: mp0 at
|
||||
// /var/lib/felhom, its docker/ and sys_drive/ folders bound in the guest — 07 §7.5.1).
|
||||
var sysDataPaths = []string{"/var/lib/felhom", "/mnt/sys_drive", "/var/lib/docker"}
|
||||
|
||||
func (s *Server) handleSameDisk(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
if s.ident == nil {
|
||||
writeErr(w, http.StatusServiceUnavailable, "disk identity not configured on this host")
|
||||
return
|
||||
}
|
||||
var req sameDiskRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||
return
|
||||
}
|
||||
if !path.IsAbs(req.A) || !path.IsAbs(req.B) {
|
||||
writeErr(w, http.StatusBadRequest, "a and b must be absolute guest paths")
|
||||
return
|
||||
}
|
||||
a := s.mapGuestPlace(r.Context(), vmid, req.A)
|
||||
b := s.mapGuestPlace(r.Context(), vmid, req.B)
|
||||
resp := SameDiskResponse{A: a.view, B: b.view}
|
||||
resp.Different, resp.Proven, resp.Reason = storage.Relation(a.id, b.id)
|
||||
s.logger.Info("local-api: same-disk", "vmid", vmid, "a", req.A, "b", req.B,
|
||||
"different", resp.Different, "proven", resp.Proven, "reason", resp.Reason)
|
||||
writeOK(w, resp)
|
||||
}
|
||||
|
||||
type mappedPlace struct {
|
||||
view SamePlace
|
||||
id storage.PlaceIdentity
|
||||
}
|
||||
|
||||
func (s *Server) mapGuestPlace(ctx context.Context, vmid int, guestPath string) mappedPlace {
|
||||
p := path.Clean(guestPath)
|
||||
out := mappedPlace{view: SamePlace{Path: p}}
|
||||
set := func(kind string, id storage.PlaceIdentity) mappedPlace {
|
||||
if !id.Computed() {
|
||||
return out
|
||||
}
|
||||
out.id = id
|
||||
out.view.Mapped, out.view.Kind = true, kind
|
||||
out.view.PhysicalDisks, out.view.Identity, out.view.Reason = id.PhysicalDisks, id.Identity, id.Reason
|
||||
return out
|
||||
}
|
||||
if strings.HasPrefix(p, StableParentDir+"/") {
|
||||
name := strings.SplitN(strings.TrimPrefix(p, StableParentDir+"/"), "/", 2)[0]
|
||||
if id, kind, ok := s.ownMountIdentity(StableParentDir + "/" + name); ok {
|
||||
return set(kind, id)
|
||||
}
|
||||
return out
|
||||
}
|
||||
for _, root := range sysDataPaths {
|
||||
if p != root && !strings.HasPrefix(p, root+"/") {
|
||||
continue
|
||||
}
|
||||
id := s.guestDataVolumeIdentity(ctx, vmid, root)
|
||||
return set(storage.PlaceKindSystem, id)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ownMountIdentity judges a HOST path only when it is ITSELF a mountpoint (an enrolled drive's raw mount
|
||||
// or its felhom-data bind, a network share). A path that is not its own mount — an absent drive, whose
|
||||
// directory now sits on the root disk — is not computed, never root's identity (the R-118 trap).
|
||||
func (s *Server) ownMountIdentity(hostPath string) (storage.PlaceIdentity, string, bool) {
|
||||
if s.ident == nil || hostPath == "" {
|
||||
return storage.PlaceIdentity{}, "", false
|
||||
}
|
||||
mounts, err := s.hostReader().Mounts()
|
||||
if err != nil {
|
||||
return storage.PlaceIdentity{}, "", false
|
||||
}
|
||||
m, ok := storage.MountFor(mounts, hostPath)
|
||||
if !ok || path.Clean(m.MountPoint) != path.Clean(hostPath) {
|
||||
return storage.PlaceIdentity{}, "", false
|
||||
}
|
||||
id := s.ident.HostPath(hostPath)
|
||||
if !id.Computed() {
|
||||
return id, "", false
|
||||
}
|
||||
kind := storage.PlaceKindDisk
|
||||
switch {
|
||||
case s.isInternalPlace(hostPath): // v0.158.0 `16` §5: the one-disk box's data place, on the system disk
|
||||
kind = storage.PlaceKindInternal
|
||||
case id.Reason == storage.ReasonNetwork:
|
||||
kind = storage.PlaceKindNetwork
|
||||
case strings.HasPrefix(m.Device, "/dev/") && !strings.HasPrefix(m.Device, "/dev/mapper/") && isPartitionPath(m.Device):
|
||||
kind = storage.PlaceKindPartition
|
||||
}
|
||||
return id, kind, true
|
||||
}
|
||||
|
||||
var rePartitionName = regexp.MustCompile(`^(nvme[0-9]+n[0-9]+p[0-9]+|(sd|vd|hd)[a-z]+[0-9]+)$`)
|
||||
|
||||
func isPartitionPath(dev string) bool { return rePartitionName.MatchString(path.Base(dev)) }
|
||||
|
||||
// guestDataVolumeIdentity finds the guest mountpoint that holds root (mp=/var/lib/felhom on golden v3, or a
|
||||
// legacy mp at that exact path) and returns the identity of the PVE storage that holds its volume.
|
||||
func (s *Server) guestDataVolumeIdentity(ctx context.Context, vmid int, root string) storage.PlaceIdentity {
|
||||
if s.guests == nil || s.storage == nil {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
cfg, err := s.guests.GuestConfig(ctx, vmid)
|
||||
if err != nil {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
storeID, fallback := "", ""
|
||||
for _, spec := range cfg.MountPoints() {
|
||||
st, mp, _ := parseMount(spec)
|
||||
if strings.HasPrefix(st, "/") {
|
||||
continue // a host bind (mp8, mp9), not a volume
|
||||
}
|
||||
switch mp {
|
||||
case root:
|
||||
storeID = st // a legacy mp at exactly this path wins
|
||||
case "/var/lib/felhom":
|
||||
fallback = st // golden v3: the one data volume
|
||||
}
|
||||
}
|
||||
if storeID == "" {
|
||||
storeID = fallback
|
||||
}
|
||||
if storeID == "" {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
targets, err := s.storage.Observe(ctx)
|
||||
if err != nil {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
for _, t := range targets {
|
||||
if t.Name == storeID {
|
||||
return identityOfTarget(t)
|
||||
}
|
||||
}
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
|
||||
func identityOfTarget(t hub.StorageTarget) storage.PlaceIdentity {
|
||||
if t.Identity == "" {
|
||||
return storage.PlaceIdentity{}
|
||||
}
|
||||
disks := t.PhysicalDisks
|
||||
if disks == nil {
|
||||
disks = []string{}
|
||||
}
|
||||
return storage.PlaceIdentity{PhysicalDisks: disks, Identity: t.Identity, Reason: t.IdentityReason}
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// POST /disks/same-disk (`16` §4) — the controller's Tier-2 question, answered from the agent's own read.
|
||||
|
||||
type fakeIdentHost struct {
|
||||
mounts []storage.Mount
|
||||
parent map[string]string
|
||||
props map[string]map[string]string
|
||||
virt string
|
||||
}
|
||||
|
||||
func (f *fakeIdentHost) Mounts() ([]storage.Mount, error) { return f.mounts, nil }
|
||||
func (f *fakeIdentHost) KernelName(dev string) (string, bool) {
|
||||
if len(dev) > 5 && dev[:5] == "/dev/" {
|
||||
return dev[5:], true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
func (f *fakeIdentHost) Parent(n string) (string, bool) { p, ok := f.parent[n]; return p, ok }
|
||||
func (f *fakeIdentHost) Slaves(string) []string { return nil }
|
||||
func (f *fakeIdentHost) LoopBackingFile(string) (string, bool) { return "", false }
|
||||
func (f *fakeIdentHost) UdevProps(n string) map[string]string { return f.props[n] }
|
||||
func (f *fakeIdentHost) Virt() string { return f.virt }
|
||||
|
||||
// sameDiskRig: guest 8200 (token A) with mp0 on local-lvm (the system SSD, proven) + the given mounts.
|
||||
func sameDiskRig(t *testing.T, host *fakeIdentHost, localLVM hub.StorageTarget) http.Handler {
|
||||
t.Helper()
|
||||
fg := &fakeGuestsCfg{mounts: map[int]map[string]string{8200: {
|
||||
"mp0": "local-lvm:vm-8200-disk-1,mp=/var/lib/felhom,backup=1,size=70G",
|
||||
"mp8": "/mnt/felhom-drives,mp=/mnt/felhom-drives",
|
||||
}}}
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: fg, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{localLVM}}, Tokens: staticTokens{"A": 8200},
|
||||
HostReader: fakeHostReader{mounts: host.mounts},
|
||||
Identity: &storage.IdentityResolver{Host: host},
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
return srv.Handler()
|
||||
}
|
||||
|
||||
var provenSSD = hub.StorageTarget{Name: "local-lvm", Type: hub.StorageTypeLVMThin, Kind: "system",
|
||||
PhysicalDisks: []string{"serial:ata:SanDisk:1"}, Identity: "proven"}
|
||||
|
||||
func baseHost() *fakeIdentHost {
|
||||
return &fakeIdentHost{
|
||||
virt: "none",
|
||||
parent: map[string]string{"sdc1": "sdc", "sdc2": "sdc"},
|
||||
props: map[string]map[string]string{
|
||||
"sdb": {"ID_BUS": "ata", "ID_MODEL": "TOSHIBA", "ID_SERIAL_SHORT": "65NOP3HDT"},
|
||||
"sdc": {"ID_BUS": "usb", "ID_MODEL": "Portable", "ID_SERIAL_SHORT": "NAAB7C21"},
|
||||
},
|
||||
mounts: []storage.Mount{
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"},
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/mnt/felhom-drives", FSType: "ext4"},
|
||||
{Device: "/dev/sdb", MountPoint: "/mnt/felhom-drives/hdd_1", FSType: "ext4"},
|
||||
{Device: "/dev/sdc1", MountPoint: "/mnt/felhom-drives/a", FSType: "ext4"},
|
||||
{Device: "/dev/sdc2", MountPoint: "/mnt/felhom-drives/b", FSType: "ext4"},
|
||||
{Device: "//nas/films", MountPoint: "/mnt/felhom-drives/nas", FSType: "cifs"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func sameDisk(t *testing.T, h http.Handler, a, b string) (SameDiskResponse, int) {
|
||||
t.Helper()
|
||||
body, _ := json.Marshal(map[string]string{"a": a, "b": b})
|
||||
w := do(t, h, "POST", "/disks/same-disk", "A", string(body))
|
||||
var env struct {
|
||||
Data SameDiskResponse `json:"data"`
|
||||
}
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &env)
|
||||
return env.Data, w.Code
|
||||
}
|
||||
|
||||
func TestSameDisk_DriveVsSystemData_DifferentProven(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
got, code := sameDisk(t, h, "/mnt/felhom-drives/hdd_1/felhom-data/paperless", "/mnt/sys_drive")
|
||||
if code != http.StatusOK || !got.Different || !got.Proven || got.Reason != "" {
|
||||
t.Fatalf("drive vs sys_drive: want different+proven, got %d %+v", code, got)
|
||||
}
|
||||
if got.B.Kind != "system" || got.A.Kind != "disk" {
|
||||
t.Fatalf("kinds: got a=%q b=%q", got.A.Kind, got.B.Kind)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_TwoPartitionsOfOneDisk_SameProven(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
got, _ := sameDisk(t, h, "/mnt/felhom-drives/a", "/mnt/felhom-drives/b/x")
|
||||
if got.Different || !got.Proven || got.Reason != "same_disk" || got.A.Kind != "partition" {
|
||||
t.Fatalf("two partitions: want same+proven, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_InsideAVM_NotProven(t *testing.T) {
|
||||
host := baseHost()
|
||||
host.virt = "kvm"
|
||||
h := sameDiskRig(t, host, hub.StorageTarget{Name: "local-lvm", Identity: "unknown", IdentityReason: "vm",
|
||||
PhysicalDisks: []string{"serial:scsi:QEMU_HARDDISK:drive-scsi0"}})
|
||||
got, _ := sameDisk(t, h, "/mnt/felhom-drives/hdd_1", "/mnt/sys_drive")
|
||||
if got.Proven || got.Reason != "vm" {
|
||||
t.Fatalf("inside a VM: want not proven (vm), got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_NetworkAndUnmapped(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
if got, _ := sameDisk(t, h, "/mnt/felhom-drives/nas", "/mnt/sys_drive"); got.Proven || got.Reason != "network" {
|
||||
t.Fatalf("a NAS: want not proven (network), got %+v", got)
|
||||
}
|
||||
// An ABSENT drive: its directory sits on the root disk — must be unmapped, never "the system disk".
|
||||
if got, _ := sameDisk(t, h, "/mnt/felhom-drives/gone", "/mnt/sys_drive"); got.Proven || got.Reason != "unmapped" || got.A.Mapped {
|
||||
t.Fatalf("an absent drive: want unmapped, got %+v", got)
|
||||
}
|
||||
if got, _ := sameDisk(t, h, "/opt/elsewhere", "/mnt/sys_drive"); got.Proven || got.Reason != "unmapped" {
|
||||
t.Fatalf("an unknown guest path: want unmapped, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSameDisk_ScopedAndValidated(t *testing.T) {
|
||||
h := sameDiskRig(t, baseHost(), provenSSD)
|
||||
if w := do(t, h, "POST", "/disks/same-disk", "A", `{"vmid":9300,"a":"/mnt/sys_drive","b":"/mnt/sys_drive"}`); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("another guest's vmid: got %d want 403", w.Code)
|
||||
}
|
||||
if w := do(t, h, "POST", "/disks/same-disk", "A", `{"a":"relative","b":"/mnt/sys_drive"}`); w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("a relative path: got %d want 400", w.Code)
|
||||
}
|
||||
if w := do(t, h, "POST", "/disks/same-disk", "", `{"a":"/x","b":"/y"}`); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: got %d want 401", w.Code)
|
||||
}
|
||||
}
|
||||
+131
-60
@@ -85,6 +85,13 @@ type BackupStore interface {
|
||||
RestoreTests(ctx context.Context) []hub.RestoreTest
|
||||
}
|
||||
|
||||
// LastKnownBackupStore (R-894) is the on-disk newest-success-per-tier record. Satisfied by
|
||||
// *backup.BackupSuccessState.
|
||||
type LastKnownBackupStore interface {
|
||||
RecordBackupSuccess(target string, b hub.Backup) error
|
||||
LastKnownSuccess(target string, vmid int) (time.Time, bool)
|
||||
}
|
||||
|
||||
// StorageView yields the host's observed storage targets (for mapping a mount's storage id →
|
||||
// fast/slow class). Satisfied by *storage.Observer.
|
||||
type StorageView interface {
|
||||
@@ -164,6 +171,10 @@ type Options struct {
|
||||
// PRIMARY tier, inside the backup goroutine and BEFORE the host-wide heavy-op gate is released — so the OS leg
|
||||
// that it starts can never overlap another backup or a restore-test (`11` C10). OPTIONAL — nil → nothing runs.
|
||||
AfterPrimaryBackup func(ctx context.Context, vmid int)
|
||||
// LastKnownBackups (R-894) keeps the newest successful backup per tier ON DISK, so the due-check's
|
||||
// fallback for an UNREADABLE storage after an agent restart is the last known copy, not "never".
|
||||
// nil = the pre-R-894 behaviour (in-memory record only).
|
||||
LastKnownBackups LastKnownBackupStore
|
||||
// Privileged runs the fenced root wrappers (E-2a: felhom-backup-target-apply). OPTIONAL — when
|
||||
// nil, POST /backup/target reports "not configured". Satisfied by *proxmox.ExecRunner.
|
||||
Privileged PrivilegedRunner
|
||||
@@ -197,6 +208,9 @@ type Options struct {
|
||||
// Intent records drive enroll/eject intent for the self-heal watchdog (slice 10 P3). OPTIONAL —
|
||||
// when nil, no intent is recorded (self-heal runs ungated).
|
||||
Intent IntentRecorder
|
||||
// InternalPlaceFile (v0.158.0, felhom.eu `16` §5) is the path of the internal data place record
|
||||
// (InternalPlaceRecord). OPTIONAL — "" means this host has no internal place and EnsureInternalPlace is a no-op.
|
||||
InternalPlaceFile string
|
||||
// GuestBinds persists which user-data drives (by durable-id) are enrolled into each guest, so the
|
||||
// startup re-assert (ReassertGuestBinds) can restore a bind that a re-provision dropped (F9).
|
||||
// OPTIONAL — when nil, guest binds are not recorded and the startup re-assert is a no-op.
|
||||
@@ -209,6 +223,10 @@ type Options struct {
|
||||
// ROLE (it backs SystemDisks for the eject role-gate + the /disks role hints). OPTIONAL — when nil
|
||||
// it defaults to the production *storage.ProcHostReader. Injectable so the role-gate is testable.
|
||||
HostReader storage.HostReader
|
||||
// Identity computes physical-disk identity (felhom.eu architecture/16-storage-physical-identity.md §3)
|
||||
// for the /disks rows and POST /disks/same-disk. OPTIONAL — when nil, /disks carries no identity and
|
||||
// same-disk answers 503 "not configured" (the controller then keeps its old st_dev rule).
|
||||
Identity *storage.IdentityResolver
|
||||
// HostMetrics serves GET /host/metrics (slice 9) — host-wide health (cpu%/mem/load/uptime/
|
||||
// cpu-temp) + per-storage capacity, host-wide and token-authed (one-customer-per-host). When
|
||||
// nil the endpoint reports "not configured" (host still reports/reconciles).
|
||||
@@ -229,7 +247,6 @@ type Options struct {
|
||||
// POST /escrow/recover-offsite-password. OPTIONAL — nil → that route reports "not configured"
|
||||
// (503) instead of failing obscurely. Satisfied by escrow.OffsiteKeyRecoverer.
|
||||
EscrowRecovery EscrowRecoverer
|
||||
|
||||
}
|
||||
|
||||
// defaultBackupCadence is the fallback /backup/due window when none is configured.
|
||||
@@ -279,30 +296,37 @@ type Server struct {
|
||||
// is the pre-R-82 shape.
|
||||
tiers []BackupTier
|
||||
// inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
|
||||
inFlight *backup.InFlight
|
||||
inFlight *backup.InFlight
|
||||
afterPrimaryBackup func(ctx context.Context, vmid int) // the OS leg (agent v0.140.0); nil = none
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
lastKnown LastKnownBackupStore // R-894: on-disk newest success per tier; nil = none
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
|
||||
disks DiskOps // slice 8C (optional)
|
||||
diskGate StorageGate // slice 8C (optional)
|
||||
guestList GuestLister // slice 8C (optional)
|
||||
guestAttach GuestAttacher // slice 10 P2 (optional)
|
||||
mem MemoryOps // v0.90.0 R-24 guest RAM resize (optional)
|
||||
memMu sync.Mutex // single-flight around a resize apply (one customer per host)
|
||||
netStorage NetworkStorageOps // Part A1: NAS network mounts (optional)
|
||||
netMountRoot string // the user-data namespace root for the network-mount role gate
|
||||
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
|
||||
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
|
||||
disks DiskOps // slice 8C (optional)
|
||||
diskGate StorageGate // slice 8C (optional)
|
||||
guestList GuestLister // slice 8C (optional)
|
||||
guestAttach GuestAttacher // slice 10 P2 (optional)
|
||||
mem MemoryOps // v0.90.0 R-24 guest RAM resize (optional)
|
||||
memMu sync.Mutex // single-flight around a resize apply (one customer per host)
|
||||
netStorage NetworkStorageOps // Part A1: NAS network mounts (optional)
|
||||
netMountRoot string // the user-data namespace root for the network-mount role gate
|
||||
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
|
||||
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
|
||||
// crashGuardStatePath (R-856) is the host crash guard's state file read by GET /host/crash-guard;
|
||||
// empty = defaultCrashGuardStatePath. A seam: tests point it at a fixture.
|
||||
crashGuardStatePath string
|
||||
// escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed
|
||||
// identity blob from the hub, unseal it with the customer's recovery code, return ONLY the
|
||||
// offsite repository password. OPTIONAL — nil (no hub client configured) makes
|
||||
// POST /escrow/recover-offsite-password answer 503 rather than pretending.
|
||||
escrowRecovery EscrowRecoverer
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||
escrowRecovery EscrowRecoverer
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
internalPlaceFile string // v0.158.0 `16` §5 (optional)
|
||||
ipState internalPlaceState // EnsureInternalPlace's memo
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||
partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional)
|
||||
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
|
||||
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
|
||||
guestPower map[int]guestPowerState
|
||||
@@ -310,7 +334,8 @@ type Server struct {
|
||||
// guestPowerSweeps counts completed guest-power sweeps, for the liveness observable. Touched only
|
||||
// from GuestPowerTick, which the ticker calls serially.
|
||||
guestPowerSweeps int
|
||||
host storage.HostReader // role classification source (optional; defaults to ProcHostReader)
|
||||
host storage.HostReader // role classification source (optional; defaults to ProcHostReader)
|
||||
ident *storage.IdentityResolver // `16` §3 physical-disk identity (optional)
|
||||
|
||||
hostMetrics HostMetricsProvider // slice 9 (optional)
|
||||
hostID string // slice 10B: for the data-bearing-format pending-op hint
|
||||
@@ -380,6 +405,10 @@ type Server struct {
|
||||
// detached pipeline runs through (tests inject; production defaults set in NewServer).
|
||||
netVerifyMu sync.Mutex
|
||||
netVerifyCur *netVerifyJob
|
||||
|
||||
// R-998: POST /host/poweroff (nil = not configured → 503); one guard-and-mark decision at a time.
|
||||
powerOff *HostPowerOffConfig
|
||||
powerOffMu sync.Mutex
|
||||
// netTrigger performs the mount-waking directory read through the automount path (Q1).
|
||||
netTrigger func(where string) error
|
||||
// netMounted judges mount success from /proc/mounts — the §8 truth source (never readability).
|
||||
@@ -431,42 +460,44 @@ func NewServer(o Options) (*Server, error) {
|
||||
cadence = defaultBackupCadence
|
||||
}
|
||||
s := &Server{
|
||||
addr: o.ListenAddr,
|
||||
cert: o.Cert,
|
||||
guests: o.Guests,
|
||||
backups: o.Backups,
|
||||
store: o.Store,
|
||||
storage: o.Storage,
|
||||
driveTargets: o.DriveTargets,
|
||||
smart: o.Smart,
|
||||
tokens: o.Tokens,
|
||||
cadence: cadence,
|
||||
logger: o.Logger,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
disks: o.Disks,
|
||||
diskGate: o.DiskGate,
|
||||
guestList: o.Guests2,
|
||||
guestAttach: o.GuestAttach,
|
||||
mem: o.Memory,
|
||||
netStorage: o.NetStorage,
|
||||
privileged: o.Privileged,
|
||||
configPath: o.ConfigPath,
|
||||
stateDir: o.StateDir,
|
||||
netMountRoot: storage.NetworkMountRoot,
|
||||
smbCredsDir: o.SmbCredsDir,
|
||||
escrowStagePath: o.EscrowStagePath,
|
||||
escrowRecovery: o.EscrowRecovery,
|
||||
intent: o.Intent,
|
||||
guestBinds: o.GuestBinds,
|
||||
formatJobs: o.FormatJobs,
|
||||
staleLock: o.StaleLock,
|
||||
host: o.HostReader,
|
||||
hostMetrics: o.HostMetrics,
|
||||
hostID: o.HostID,
|
||||
agentVersion: o.AgentVersion,
|
||||
logRing: o.LogRing,
|
||||
jobs: map[backupJobKey]*backupJob{},
|
||||
swapInFlight: map[int]bool{},
|
||||
addr: o.ListenAddr,
|
||||
cert: o.Cert,
|
||||
guests: o.Guests,
|
||||
backups: o.Backups,
|
||||
store: o.Store,
|
||||
storage: o.Storage,
|
||||
driveTargets: o.DriveTargets,
|
||||
smart: o.Smart,
|
||||
tokens: o.Tokens,
|
||||
cadence: cadence,
|
||||
logger: o.Logger,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
disks: o.Disks,
|
||||
diskGate: o.DiskGate,
|
||||
guestList: o.Guests2,
|
||||
guestAttach: o.GuestAttach,
|
||||
mem: o.Memory,
|
||||
netStorage: o.NetStorage,
|
||||
privileged: o.Privileged,
|
||||
configPath: o.ConfigPath,
|
||||
stateDir: o.StateDir,
|
||||
netMountRoot: storage.NetworkMountRoot,
|
||||
smbCredsDir: o.SmbCredsDir,
|
||||
escrowStagePath: o.EscrowStagePath,
|
||||
escrowRecovery: o.EscrowRecovery,
|
||||
intent: o.Intent,
|
||||
internalPlaceFile: o.InternalPlaceFile,
|
||||
guestBinds: o.GuestBinds,
|
||||
formatJobs: o.FormatJobs,
|
||||
staleLock: o.StaleLock,
|
||||
host: o.HostReader,
|
||||
ident: o.Identity,
|
||||
hostMetrics: o.HostMetrics,
|
||||
hostID: o.HostID,
|
||||
agentVersion: o.AgentVersion,
|
||||
logRing: o.LogRing,
|
||||
jobs: map[backupJobKey]*backupJob{},
|
||||
swapInFlight: map[int]bool{},
|
||||
}
|
||||
// R-82 tier resolution. Options.BackupTiers is authoritative when supplied; otherwise ONE tier
|
||||
// is synthesized from Backups + BackupCadence — the pre-R-82 shape, so every existing caller
|
||||
@@ -475,6 +506,7 @@ func NewServer(o Options) (*Server, error) {
|
||||
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
|
||||
s.inFlight = o.InFlight
|
||||
s.afterPrimaryBackup = o.AfterPrimaryBackup
|
||||
s.lastKnown = o.LastKnownBackups
|
||||
if s.backups == nil && len(s.tiers) > 0 {
|
||||
s.backups = s.tiers[0].Service
|
||||
}
|
||||
@@ -520,9 +552,13 @@ func (s *Server) Handler() http.Handler {
|
||||
// Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring
|
||||
// view. Host-wide, token-authed, fresh (a live collect, not the 15-min hub snapshot).
|
||||
mux.HandleFunc("GET /host/metrics", s.withGuest(s.handleHostMetrics))
|
||||
// R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller
|
||||
// waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false.
|
||||
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
|
||||
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
||||
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
|
||||
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
|
||||
mux.HandleFunc("POST /disks/same-disk", s.withGuest(s.handleSameDisk)) // `16` §4 (v0.157.0)
|
||||
mux.HandleFunc("POST /disks/assign", s.withGuest(s.handleDiskAssign))
|
||||
mux.HandleFunc("POST /disks/eject", s.withGuest(s.handleDiskEject))
|
||||
mux.HandleFunc("POST /disks/decommission", s.withGuest(s.handleDiskDecommission))
|
||||
@@ -532,6 +568,8 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
|
||||
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
|
||||
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
|
||||
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
|
||||
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
|
||||
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
|
||||
// (live cgroup apply, no reboot). Self-scoped; the agent enforces every bound fresh per request.
|
||||
mux.HandleFunc("GET /guest/memory", s.withGuest(s.handleGuestMemory))
|
||||
@@ -807,6 +845,10 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
return
|
||||
}
|
||||
key := backupJobKey{vmid: vmid, target: tier.TargetID}
|
||||
// R-899 (operator ruling 2026-10-08): a household press („Mentés most") is not the night's backup. A controller
|
||||
// that knows sends `trigger=manual`; then the OS leg does not follow (it belongs to the night, after the night's
|
||||
// own copy). An older controller sends nothing and keeps the old behaviour.
|
||||
manual := r.URL.Query().Get("trigger") == "manual"
|
||||
|
||||
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
|
||||
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
|
||||
@@ -899,9 +941,18 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive)
|
||||
}
|
||||
s.store.RecordBackup(b)
|
||||
if b.Success && s.lastKnown != nil {
|
||||
if err := s.lastKnown.RecordBackupSuccess(tier.TargetID, b); err != nil {
|
||||
// Not fatal: the backup exists. Only the fallback after a restart loses this copy.
|
||||
s.logger.Warn("local-api: could not save the backup on disk for the due-check fallback (R-894)",
|
||||
"vmid", vmid, "target", tier.TargetID, "err", err)
|
||||
}
|
||||
}
|
||||
s.finishJob(key, jobID, b)
|
||||
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
|
||||
if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
|
||||
if b.Success && tier.Primary && s.afterPrimaryBackup != nil && manual {
|
||||
s.logger.Info("local-api: no OS leg after a manual backup — it follows the night's own backup (R-899)", "vmid", vmid, "job", jobID)
|
||||
} else if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
|
||||
s.afterPrimaryBackup(base, vmid)
|
||||
}
|
||||
}()
|
||||
@@ -1061,6 +1112,20 @@ func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid in
|
||||
newest, haveNewest = t, true
|
||||
unparseable = false // ground truth supersedes an unreadable in-memory timestamp
|
||||
}
|
||||
// R-894: the storage could not be read → the last success saved ON DISK stands in for the in-memory
|
||||
// record a restart emptied. ONLY on archiveUnknown: a storage that answers is the ground truth, and an
|
||||
// archive absent there must make the tier due even when the file remembers one (a pruned copy).
|
||||
// A saved copy older than the cadence still reads DUE below — an unreadable storage never suppresses
|
||||
// a backup that is due.
|
||||
fromDisk := false
|
||||
if lookup == archiveUnknown && s.lastKnown != nil {
|
||||
if saved, ok := s.lastKnown.LastKnownSuccess(tier.TargetID, vmid); ok && (!haveNewest || saved.After(newest)) {
|
||||
newest, haveNewest, fromDisk = saved, true, true
|
||||
unparseable = false
|
||||
s.logger.Info("local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)",
|
||||
"vmid", vmid, "target", tier.TargetID, "saved", saved.UTC().Format(time.RFC3339))
|
||||
}
|
||||
}
|
||||
if !haveNewest {
|
||||
// R-88 Part 2: THREE distinct reasons for a nil age, each with its own state. Only ABSENT is a
|
||||
// positive claim of "never backed up"; only that one may license the controller to bypass its
|
||||
@@ -1083,13 +1148,17 @@ func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid in
|
||||
}
|
||||
age := s.now().Sub(newest)
|
||||
ageSecs := int64(age.Seconds())
|
||||
suffix := ""
|
||||
if fromDisk {
|
||||
suffix = " (storage unreadable — age from the last success saved on disk)"
|
||||
}
|
||||
if age >= tier.Cadence {
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeSecs: &ageSecs, AgeState: AgeStateKnown,
|
||||
Reason: "older than cadence", Target: echo})
|
||||
Reason: "older than cadence" + suffix, Target: echo})
|
||||
return
|
||||
}
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: false, AgeSecs: &ageSecs, AgeState: AgeStateKnown,
|
||||
Reason: "within cadence window", Target: echo})
|
||||
Reason: "within cadence window" + suffix, Target: echo})
|
||||
}
|
||||
|
||||
// BackupTiersResponse is GET /backup/tiers (R-82): the tiers this agent serves, primary first.
|
||||
@@ -1477,4 +1546,6 @@ func writeStatus(w http.ResponseWriter, code int, ok bool, data any, errMsg stri
|
||||
}
|
||||
|
||||
// SetAfterPrimaryBackup wires the hook that runs after a successful primary-tier backup (the OS leg, agent v0.140.0).
|
||||
func (s *Server) SetAfterPrimaryBackup(fn func(ctx context.Context, vmid int)) { s.afterPrimaryBackup = fn }
|
||||
func (s *Server) SetAfterPrimaryBackup(fn func(ctx context.Context, vmid int)) {
|
||||
s.afterPrimaryBackup = fn
|
||||
}
|
||||
|
||||
@@ -4,7 +4,11 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
@@ -142,6 +146,10 @@ type staleLockController struct {
|
||||
runner proxmox.Runner
|
||||
pool string // ownership registry: only members of this PVE pool are ever scanned (A1)
|
||||
logger *slog.Logger // the one success-path scan-summary line; nil = silent
|
||||
// procDir is where taskAlive reads /proc/<pid>/stat (R-1008); "" = /proc. Tests point it at a fixture.
|
||||
procDir string
|
||||
deadMu sync.Mutex
|
||||
deadSeen map[string]bool // UPIDs of dead tasks already logged (once per task)
|
||||
}
|
||||
|
||||
// staleLockAPI is the subset of *proxmox.Client the controller uses (kept narrow for clarity/testing).
|
||||
@@ -209,19 +217,100 @@ func (c *staleLockController) Lock(ctx context.Context, vmid int) (string, bool,
|
||||
return cfg.Lock(), cfg.OnBoot(), nil
|
||||
}
|
||||
|
||||
// BackupRunning is THE answer to „is a vzdump in flight for vmid" for all four callers (startup stale-lock
|
||||
// recovery, the guest-power watchdog, the household power-off guard, the controller supervisor): every one of
|
||||
// them reaches this method through the StaleLockController interface. R-1008: Proxmox keeps a task in
|
||||
// `tasks?source=active` after a power cut — nobody is left to end it (measured on demo-hp 2026-10-14: still
|
||||
// listed RUNNING ~30–60 s after the boot) — so a listed task whose worker process is gone is dead, not in flight.
|
||||
func (c *staleLockController) BackupRunning(ctx context.Context, vmid int) (bool, error) {
|
||||
tasks, err := c.px.ListRunningTasks(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
proc := c.procDir
|
||||
if proc == "" {
|
||||
proc = "/proc"
|
||||
}
|
||||
running, dead := vzdumpInFlight(tasks, vmid, proc)
|
||||
for _, t := range dead {
|
||||
c.logDeadOnce(t, vmid)
|
||||
}
|
||||
return running, nil
|
||||
}
|
||||
|
||||
// vzdumpInFlight decides, from the node's active-task list, whether a vzdump for vmid is in flight: a matching
|
||||
// task is in flight while its worker process lives (taskAlive). A task whose worker is gone is dead (returned in dead,
|
||||
// for the log). FAIL SAFE on doubt — the old answer: a UPID that does not parse, or a /proc entry that cannot be read,
|
||||
// counts as in flight. Pure over procDir; pinned by stalelock_inflight_test.go.
|
||||
func vzdumpInFlight(tasks []proxmox.TaskStatus, vmid int, procDir string) (bool, []proxmox.TaskStatus) {
|
||||
id := strconv.Itoa(vmid)
|
||||
var dead []proxmox.TaskStatus
|
||||
for _, t := range tasks {
|
||||
// A single-guest vzdump task carries the vmid in its ID field.
|
||||
if t.Type == "vzdump" && t.ID == id {
|
||||
if t.Type != "vzdump" || t.ID != id {
|
||||
continue
|
||||
}
|
||||
alive, known := taskAlive(procDir, t.UPID)
|
||||
if !known || alive {
|
||||
return true, nil
|
||||
}
|
||||
dead = append(dead, t)
|
||||
}
|
||||
return false, nil
|
||||
return false, dead
|
||||
}
|
||||
|
||||
// taskAlive is Proxmox's own liveness rule for a task (PVE::ProcFSTools::check_process_running): the UPID names the
|
||||
// worker's pid and its start time in clock ticks since boot (pstart); the task lives while /proc/<pid>/stat exists with
|
||||
// that same start time (field 22). No wall clock is involved — a clock step after the boot (an NTP correction after a
|
||||
// power cut) cannot make a live task look dead (the 2026-10-14 security review of the first, btime-based version).
|
||||
// known=false: the UPID does not parse, or /proc/<pid>/stat exists but cannot be read or parsed.
|
||||
func taskAlive(procDir, upid string) (alive, known bool) {
|
||||
u, err := proxmox.ParseUPID(upid)
|
||||
if err != nil || u.PID == 0 {
|
||||
return false, false
|
||||
}
|
||||
raw, err := os.ReadFile(filepath.Join(procDir, strconv.FormatUint(u.PID, 10), "stat"))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
// „no such process" is only an answer if this agent can see other users' processes at all (a /proc mounted
|
||||
// hidepid=, or a unit with ProtectProc=, hides root's workers — every task would then look dead). PID 1 is
|
||||
// root's; not seeing it means not knowing.
|
||||
if _, err1 := os.Stat(filepath.Join(procDir, "1", "stat")); err1 != nil {
|
||||
return false, false
|
||||
}
|
||||
return false, true // no such process in this boot: the worker is gone
|
||||
}
|
||||
return false, false
|
||||
}
|
||||
st := string(raw)
|
||||
i := strings.LastIndexByte(st, ')') // the comm field may hold spaces and parentheses
|
||||
if i < 0 {
|
||||
return false, false
|
||||
}
|
||||
f := strings.Fields(st[i+1:]) // f[0] is field 3 (state), so field 22 (starttime) is f[19]
|
||||
if len(f) < 20 {
|
||||
return false, false
|
||||
}
|
||||
start, err := strconv.ParseUint(f[19], 10, 64)
|
||||
if err != nil {
|
||||
return false, false
|
||||
}
|
||||
// Proxmox's rule exactly: the start tick matches (a pstart of 0 is not checked) and the process is not a zombie.
|
||||
return (u.PStart == 0 || start == u.PStart) && f[0] != "Z", true
|
||||
}
|
||||
|
||||
func (c *staleLockController) logDeadOnce(t proxmox.TaskStatus, vmid int) {
|
||||
c.deadMu.Lock()
|
||||
defer c.deadMu.Unlock()
|
||||
if c.deadSeen == nil {
|
||||
c.deadSeen = map[string]bool{}
|
||||
}
|
||||
if c.deadSeen[t.UPID] || c.logger == nil {
|
||||
return
|
||||
}
|
||||
c.deadSeen[t.UPID] = true
|
||||
c.logger.Warn("backup-in-flight: a vzdump task is listed as active but its worker process is gone — it is dead, not running",
|
||||
"vmid", vmid, "upid", t.UPID)
|
||||
}
|
||||
|
||||
func (c *staleLockController) HasVzdumpSnapshot(ctx context.Context, vmid int) (bool, error) {
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1008: after a power cut Proxmox keeps the dead vzdump task in `tasks?source=active` (measured on demo-hp
|
||||
// 2026-10-14: still RUNNING ~30–60 s after the boot). A task whose worker process is gone is dead, not in flight.
|
||||
// Liveness is Proxmox's own rule (pid + pstart vs /proc/<pid>/stat field 22), never the wall clock: the first version
|
||||
// compared start time with btime and a clock step after the boot could make a live task look dead (security review
|
||||
// 2026-10-14). These tests pin the helper, the production controller, three callers end to end; the fourth (the
|
||||
// controller supervisor) by the structural pin at the bottom.
|
||||
|
||||
// A worker: pid 0x1E6EEC = 1994476, pstart 0x1B4A0D = 1788429 (UPID hex fields).
|
||||
const (
|
||||
testPID = 1994476
|
||||
testPStart = 1788429
|
||||
)
|
||||
|
||||
func vz(vmid string) proxmox.TaskStatus {
|
||||
return proxmox.TaskStatus{UPID: "UPID:demo-hp:001E6EEC:001B4A0D:6ACF4071:vzdump:" + vmid + ":felhom-agent@pve!agent:",
|
||||
Type: "vzdump", ID: vmid, Status: "RUNNING", StartTime: 1791967345}
|
||||
}
|
||||
|
||||
// procFixture writes <dir>/<pid>/stat with the given start ticks (comm with a space and a parenthesis, as real ones can).
|
||||
func procFixture(t *testing.T, pid int, start int) string {
|
||||
return procFixtureState(t, pid, start, "S")
|
||||
}
|
||||
|
||||
func procFixtureState(t *testing.T, pid int, start int, state string) string {
|
||||
t.Helper()
|
||||
d := t.TempDir()
|
||||
// PID 1 is always visible on a real host; procFixtureHidden leaves it out
|
||||
if err := os.MkdirAll(filepath.Join(d, "1"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(d, "1", "stat"), []byte("1 (systemd) S 0 1 1 0 -1 0 0 0 0 0 0 0 0 0 20 0 1 0 2 0 0\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pid > 0 {
|
||||
p := filepath.Join(d, itoaT(pid))
|
||||
if err := os.MkdirAll(p, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f := []string{itoaT(pid), "(task UPID:x (vzdump))", state}
|
||||
for i := 4; i <= 21; i++ {
|
||||
f = append(f, "0")
|
||||
}
|
||||
f = append(f, itoaT(start), "0", "0")
|
||||
if err := os.WriteFile(filepath.Join(p, "stat"), []byte(strings.Join(f, " ")+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// RED-PROOF: make taskAlive return (true, true) for a missing /proc entry → the dead task runs → fails.
|
||||
func TestVzdumpInFlight_WorkerGoneIsDead(t *testing.T) {
|
||||
running, dead := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, 0, 0))
|
||||
if running || len(dead) != 1 {
|
||||
t.Fatalf("a task whose worker is gone must be dead: running=%v dead=%d", running, len(dead))
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the `start == u.PStart` comparison (alive whenever the pid exists) → a reused pid runs → fails.
|
||||
func TestVzdumpInFlight_ReusedPidIsDead(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, testPID, 42)); running {
|
||||
t.Fatal("the pid exists but started at another tick: another process, the task is dead")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the `f[0] != "Z"` test → a zombie worker counts as running → fails.
|
||||
func TestVzdumpInFlight_ZombieIsDead(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixtureState(t, testPID, testPStart, "Z")); running {
|
||||
t.Fatal("a zombie worker is not running (Proxmox's own rule)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVzdumpInFlight_LiveWorkerRuns(t *testing.T) {
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, procFixture(t, testPID, testPStart)); !running {
|
||||
t.Fatal("the worker lives (same pid, same start tick): in flight")
|
||||
}
|
||||
}
|
||||
|
||||
// The reason for the rewrite: the wall clock plays no part. A live task whose list start time lies „before the boot"
|
||||
// (a clock step after the boot) is still running.
|
||||
func TestVzdumpInFlight_ClockStepCannotKillALiveTask(t *testing.T) {
|
||||
tk := vz("9201")
|
||||
tk.StartTime = 1 // absurdly early by the wall clock
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{tk}, 9201, procFixture(t, testPID, testPStart)); !running {
|
||||
t.Fatal("a live worker must count as running whatever its wall-clock start says")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: make an unparsable UPID or unreadable stat „known dead" → fails.
|
||||
func TestVzdumpInFlight_DoubtFailsSafe(t *testing.T) {
|
||||
bad := vz("9201")
|
||||
bad.UPID = "garbage"
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{bad}, 9201, procFixture(t, 0, 0)); !running {
|
||||
t.Fatal("an unparsable UPID must count as running (fail safe)")
|
||||
}
|
||||
d := procFixture(t, testPID, testPStart)
|
||||
if err := os.WriteFile(filepath.Join(d, itoaT(testPID), "stat"), []byte("garbage"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, d); !running {
|
||||
t.Fatal("an unparsable stat must count as running (fail safe)")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the PID-1 visibility check → a hidden /proc makes the live task „gone" → fails.
|
||||
func TestVzdumpInFlight_HiddenProcFailsSafe(t *testing.T) {
|
||||
d := t.TempDir() // nothing visible: not even PID 1 (hidepid / ProtectProc)
|
||||
if running, _ := vzdumpInFlight([]proxmox.TaskStatus{vz("9201")}, 9201, d); !running {
|
||||
t.Fatal("a /proc that hides other users' processes must count as running (fail safe)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVzdumpInFlight_OtherGuestOrTypeIgnored(t *testing.T) {
|
||||
tasks := []proxmox.TaskStatus{vz("9202"), {UPID: vz("9201").UPID, Type: "vzsnapshot", ID: "9201"}}
|
||||
if running, dead := vzdumpInFlight(tasks, 9201, procFixture(t, testPID, testPStart)); running || len(dead) != 0 {
|
||||
t.Fatalf("other guests' and other types' tasks are not 9201's backup: running=%v dead=%d", running, len(dead))
|
||||
}
|
||||
}
|
||||
|
||||
func inflightCtrl(api *fakeStaleLockAPI, runner *recordingRunner, logger *slog.Logger, proc string) *staleLockController {
|
||||
c := NewStaleLockController(api, runner, "felhom", logger).(*staleLockController)
|
||||
c.procDir = proc
|
||||
return c
|
||||
}
|
||||
|
||||
// The production controller answers through the helper, and logs a dead task ONCE.
|
||||
// RED-PROOF: make BackupRunning ignore the helper (any listed task runs) → fails.
|
||||
func TestStaleLockController_BackupRunningIgnoresDeadTask(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")}}
|
||||
c := inflightCtrl(api, &recordingRunner{}, slog.New(slog.NewTextHandler(&buf, nil)), procFixture(t, 0, 0))
|
||||
for i := 0; i < 3; i++ {
|
||||
if running, err := c.BackupRunning(context.Background(), 9201); err != nil || running {
|
||||
t.Fatalf("dead task reported running=%v err=%v", running, err)
|
||||
}
|
||||
}
|
||||
if n := strings.Count(buf.String(), "backup-in-flight: a vzdump task is listed as active but its worker process is gone"); n != 1 {
|
||||
t.Fatalf("the dead task must be logged exactly once, got %d", n)
|
||||
}
|
||||
c.procDir = procFixture(t, testPID, testPStart)
|
||||
if running, _ := c.BackupRunning(context.Background(), 9201); !running {
|
||||
t.Fatal("a live worker must be running")
|
||||
}
|
||||
}
|
||||
|
||||
func fixtureFor(t *testing.T, alive bool) string {
|
||||
if alive {
|
||||
return procFixture(t, testPID, testPStart)
|
||||
}
|
||||
return procFixture(t, 0, 0)
|
||||
}
|
||||
|
||||
func poolOf(vmids ...int) proxmox.PoolInfo {
|
||||
var p proxmox.PoolInfo
|
||||
raw := `{"members":[`
|
||||
for i, v := range vmids {
|
||||
if i > 0 {
|
||||
raw += ","
|
||||
}
|
||||
raw += `{"vmid":` + itoaT(v) + `,"type":"lxc"}`
|
||||
}
|
||||
raw += `]}`
|
||||
_ = json.Unmarshal([]byte(raw), &p)
|
||||
return p
|
||||
}
|
||||
|
||||
// Caller 1 — the startup stale-lock recovery clears the lock a power cut left (the night of 2026-10-14 it said
|
||||
// „genuinely in-flight — leaving the lock").
|
||||
func TestCaller_StartupStaleLockClearsAfterPowerCut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
alive bool
|
||||
cleared bool
|
||||
}{{false, true}, {true, false}} {
|
||||
api := &fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")},
|
||||
cfg: map[int]proxmox.GuestConfig{9201: staleCfg("backup", true)}}
|
||||
runner := &recordingRunner{}
|
||||
s := &Server{staleLock: inflightCtrl(api, runner, nil, fixtureFor(t, tc.alive)), logger: slog.New(slog.NewTextHandler(discardW{}, nil))}
|
||||
s.recoverOneStaleLock(context.Background(), proxmox.Guest{VMID: 9201, Status: "running"})
|
||||
if runner.unlocked("9201") != tc.cleared {
|
||||
t.Fatalf("worker alive=%v: unlocked=%v, want %v", tc.alive, runner.unlocked("9201"), tc.cleared)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Caller 2 — the guest-power watchdog starts a stopped guest despite a dead pre-boot vzdump task.
|
||||
func TestCaller_GuestPowerStartsGuestAfterPowerCut(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
alive bool
|
||||
started bool
|
||||
}{{false, true}, {true, false}} {
|
||||
api := &fakeStaleLockAPI{lxc: []proxmox.Guest{{VMID: 9201, Status: "stopped"}}, pool: poolOf(9201),
|
||||
tasks: []proxmox.TaskStatus{vz("9201")}, cfg: map[int]proxmox.GuestConfig{9201: staleCfg("", true)}}
|
||||
s := gpServer(t, inflightCtrl(api, &recordingRunner{}, nil, fixtureFor(t, tc.alive)), nil)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if (len(api.started) == 1) != tc.started {
|
||||
t.Fatalf("worker alive=%v: started=%v, want %v", tc.alive, api.started, tc.started)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Caller 3 — the household power-off guard does not refuse „backup running" for a dead task.
|
||||
func TestCaller_PowerOffJobBlockerIgnoresDeadTask(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
alive bool
|
||||
want string
|
||||
}{{false, ""}, {true, PowerOffBackup}} {
|
||||
rig := newPowerRig(t, 0)
|
||||
rig.srv.staleLock = inflightCtrl(&fakeStaleLockAPI{tasks: []proxmox.TaskStatus{vz("9201")}}, &recordingRunner{}, nil, fixtureFor(t, tc.alive))
|
||||
if code, _ := rig.srv.jobBlocker(context.Background(), 9201); code != tc.want {
|
||||
t.Fatalf("worker alive=%v: jobBlocker=%q, want %q", tc.alive, code, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Caller 4 (the supervisor), the R-1006 cleanup (VzdumpInFlight) and the rest — structural pin: every non-test BackupRunning call in this package goes through
|
||||
// s.staleLock (whose one production implementation is staleLockController, which uses vzdumpInFlight), and no
|
||||
// other production type implements BackupRunning. If someone adds a second path, this fails.
|
||||
func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
|
||||
files, _ := filepath.Glob("*.go")
|
||||
call := regexp.MustCompile(`([A-Za-z_.]+)\.BackupRunning\(`)
|
||||
impl := regexp.MustCompile(`func \([a-z]+ \*?([A-Za-z]+)\) BackupRunning\(`)
|
||||
callers := 0
|
||||
for _, f := range files {
|
||||
if strings.HasSuffix(f, "_test.go") {
|
||||
continue
|
||||
}
|
||||
b, err := os.ReadFile(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, m := range call.FindAllStringSubmatch(string(b), -1) {
|
||||
if m[1] != "s.staleLock" {
|
||||
t.Errorf("%s: BackupRunning called on %q, not s.staleLock", f, m[1])
|
||||
}
|
||||
callers++
|
||||
}
|
||||
for _, m := range impl.FindAllStringSubmatch(string(b), -1) {
|
||||
if m[1] != "staleLockController" {
|
||||
t.Errorf("%s: a second production BackupRunning on %s", f, m[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
if callers != 5 {
|
||||
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
|
||||
}
|
||||
b, _ := os.ReadFile("stalelock.go")
|
||||
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
|
||||
t.Error("staleLockController.BackupRunning no longer answers through vzdumpInFlight")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package localapi
|
||||
|
||||
import "context"
|
||||
|
||||
// R-1006: the half-written vzdump cleanup's two needs from the Server, and its start-up sweep. The cleaner itself is
|
||||
// internal/vzpartial; the root helper is configs/felhom-vzdump-partial-clean.
|
||||
|
||||
// VzdumpInFlight is the one in-flight answer (staleLockController.BackupRunning, R-1008) for the cleanup. No stale-lock
|
||||
// controller wired → true: fail safe, nothing is removed.
|
||||
func (s *Server) VzdumpInFlight(ctx context.Context, vmid int) (bool, error) {
|
||||
if s.staleLock == nil {
|
||||
return true, nil
|
||||
}
|
||||
return s.staleLock.BackupRunning(ctx, vmid)
|
||||
}
|
||||
|
||||
// SetPartialSweep sets the start-up sweep: called with every owned guest (the felhom pool, ownership proven).
|
||||
func (s *Server) SetPartialSweep(fn func(ctx context.Context, vmid int)) { s.partialSweep = fn }
|
||||
|
||||
// SweepVzdumpPartials runs the sweep once for every owned guest. No-op when unwired.
|
||||
func (s *Server) SweepVzdumpPartials(ctx context.Context) {
|
||||
if s.partialSweep == nil || s.staleLock == nil {
|
||||
return
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
s.logger.Warn("vzdump-partial: guest list unavailable — skipping the start-up sweep", "err", err)
|
||||
return
|
||||
}
|
||||
for _, g := range guests {
|
||||
s.partialSweep(ctx, g.VMID)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,446 @@
|
||||
package osupdate
|
||||
|
||||
// The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). The root half is felhom-os-apply's layer "kernel"
|
||||
// (configs/, its own tests); this file decides WHEN and judges the boot:
|
||||
//
|
||||
// - the night leg (Run → runKernel): after a healthy host step, on a night the hub marks as told (the household was
|
||||
// mailed the day before — no mail, no step): ring 0 STAGES the pending kernel (select pending-kernel, the root-owned
|
||||
// ring-0 mark) and reboots; ring 1 reboots only a step a signed os_kernel_step staged earlier (KernelStepExecutor).
|
||||
// - after a boot (KernelAfterBoot, at daemon start): the wrapper says what became of the step. On the new kernel the
|
||||
// agent JUDGES the boot — the host health rule (`11` §8.2: the Proxmox daemons, the guest running and healthy, the
|
||||
// tunnel) AND the box reaching the hub — for KernelJudgeWait. Healthy → kernel-good (the new kernel becomes the
|
||||
// default). Not healthy by the deadline → ONE self-revert (kernel-revert: a reboot into the old kernel, still the
|
||||
// default). A crash on the new kernel needs nothing from the agent: GRUB already boots the old default.
|
||||
//
|
||||
// The host is rebooted by this file only through the wrapper (kernel-reboot, kernel-revert), and only for a staged step.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"regexp"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
)
|
||||
|
||||
// OpKernelStep is the signed op class that STAGES a kernel set on a ring-1 box (it never reboots: the night leg does,
|
||||
// once the household was told). CC may sign it until the first paying customer (R-530 ruling).
|
||||
const OpKernelStep = "os_kernel_step"
|
||||
|
||||
// DefaultKernelJudgeWait is how long a one-shot boot may take to come back healthy before the agent reverts it ONCE.
|
||||
// Measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`): every container healthy 68 s after a reboot on demo-felhom,
|
||||
// 272 s on demo-hp; 20 minutes stays under the hub's 45-minute host_stale (a box that never comes back alarms after it).
|
||||
const DefaultKernelJudgeWait = 20 * time.Minute
|
||||
|
||||
var kverRE = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$`)
|
||||
|
||||
// KernelView is the wrapper's kernel object (felhom-os-apply Kernel.view).
|
||||
type KernelView struct {
|
||||
Running string `json:"running"`
|
||||
Default string `json:"default"`
|
||||
Flag *string `json:"flag"`
|
||||
Phase string `json:"phase"`
|
||||
From string `json:"from"`
|
||||
To string `json:"to"`
|
||||
SelfRevertUsed bool `json:"self_revert_used"`
|
||||
Reason string `json:"reason"`
|
||||
VMID int `json:"vmid"` // the customer guest the step was staged for (the health rule's guest)
|
||||
}
|
||||
|
||||
// kernelReportRing is the ring an after-boot report carries. In the first second after a boot the agent has not
|
||||
// fetched the hub's block yet, and Block() then answers ring 1 — so a ring-0 box's „judging" report said ring 1
|
||||
// (seen on demo-felhom, 2026-10-08 night, `audits/kernel-night-2026-10-07/readback/`). Order: the fetched block; the
|
||||
// block the daemon saved on disk before the reboot (R-866); ring 1 as before. A label only: the hub's approval reads its
|
||||
// own ring list. Pinned by TestKernelReportRing_BeforeFirstFetch.
|
||||
func (l *Leg) kernelReportRing() int {
|
||||
l.mu.Lock()
|
||||
fetched := l.block
|
||||
l.mu.Unlock()
|
||||
if fetched != nil {
|
||||
return fetched.Ring
|
||||
}
|
||||
if b, _, ok := LoadSavedBlock(l.planDir()); ok && b != nil {
|
||||
return b.Ring
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
func parseKernel(raw json.RawMessage) KernelView {
|
||||
var v KernelView
|
||||
_ = json.Unmarshal(raw, &v)
|
||||
return v
|
||||
}
|
||||
|
||||
// kernelPlan is one kernel-layer wrapper call.
|
||||
func kernelPlan(mode string, vmid int, extra map[string]any) map[string]any {
|
||||
p := map[string]any{"release_id": "kernel", "layer": LayerKernel, "lane": "slow", "vmid": vmid, "mode": mode,
|
||||
"packages": []Package{}}
|
||||
for k, v := range extra {
|
||||
p[k] = v
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// KernelStatus reads the kernel lane's state (read only).
|
||||
func (l *Leg) KernelStatus(ctx context.Context, vmid int) (KernelView, error) {
|
||||
wr, err := l.call(ctx, "kstatus"+l.now().UTC().Format("150405"), kernelPlan("kernel-status", vmid, nil))
|
||||
if err != nil {
|
||||
return KernelView{}, err
|
||||
}
|
||||
if wr.refused() {
|
||||
return KernelView{}, fmt.Errorf("kernel-status refused: %s", wr.Refused)
|
||||
}
|
||||
return parseKernel(wr.Kernel), nil
|
||||
}
|
||||
|
||||
// kernelDue reports whether tonight's leg may take a kernel step, and why not.
|
||||
func kernelDue(blk hub.WireOSUpdate, trigger string) (bool, string) {
|
||||
switch {
|
||||
case trigger != "night":
|
||||
return false, "a kernel step runs only in the night leg (never a debug pass)"
|
||||
case blk.Kernel == nil:
|
||||
return false, "the hub names no kernel step for this box"
|
||||
case !blk.Enabled:
|
||||
return false, "OS updates are switched off for this box"
|
||||
case !kverRE.MatchString(blk.Kernel.Kver):
|
||||
return false, "the hub's kernel " + blk.Kernel.Kver + " is not a kernel version"
|
||||
case !blk.Kernel.Tonight:
|
||||
return false, "the household has not been told about tonight (no mail, no step — `09` §3 decision 172)"
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
// runKernel is the night leg's last step. It returns the stage report (Layer "" when nothing ran). On success the box
|
||||
// is rebooting when it returns.
|
||||
func (l *Leg) runKernel(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate) Report {
|
||||
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", trigger, "ring", blk.Ring)
|
||||
if ok, why := kernelDue(blk, trigger); !ok {
|
||||
lg.Info("osupdate: kernel step skipped — " + why)
|
||||
return Report{}
|
||||
}
|
||||
want := blk.Kernel.Kver
|
||||
st, err := l.KernelStatus(ctx, vmid)
|
||||
if err != nil {
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
|
||||
Mode: "apply", Outcome: "failed", HealthReason: "kernel status unreadable: " + err.Error()})
|
||||
}
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "apply", ReleaseID: want}
|
||||
switch {
|
||||
case st.Phase == "staged" && st.To == want:
|
||||
lg.Info("osupdate: kernel step — a staged kernel waits for tonight", "from", st.From, "to", st.To)
|
||||
rep.Outcome, rep.Healthy = "staged", true
|
||||
case blk.Ring != 0:
|
||||
lg.Info("osupdate: kernel step skipped — ring 1 boots only a kernel a signed os_kernel_step staged", "phase", st.Phase, "staged", st.To, "want", want)
|
||||
return Report{}
|
||||
default:
|
||||
// R-898: EXACTLY the kernel the household was told about — never "whatever is pending tonight" (the sources can
|
||||
// offer a newer one by night; the step then refused, R23, and the night was lost). A version no longer
|
||||
// installable is refused by the wrapper before any change (R7) and the hub tells the household again.
|
||||
wr, cerr := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": "ring0-" + runID,
|
||||
"select": "listed", "packages": KernelSet(want), "expect_kver": want, "run_id": runID, "trigger": trigger,
|
||||
"ring": blk.Ring}))
|
||||
rep.unsent = reportFile(l.planDir(), runID, LayerKernel, "apply")
|
||||
rep.Kernel = rawOrNil(wr.Kernel)
|
||||
switch {
|
||||
case cerr != nil:
|
||||
rep.Outcome, rep.HealthReason = "failed", cerr.Error()
|
||||
return l.finish(ctx, lg, rep)
|
||||
case wr.refused():
|
||||
rep.Outcome, rep.Refused = "refused", wr.Refused
|
||||
return l.finish(ctx, lg, rep)
|
||||
case wr.failed():
|
||||
rep.Outcome, rep.Refused = "failed", wr.Failed
|
||||
return l.finish(ctx, lg, rep)
|
||||
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
|
||||
rep.Outcome, rep.Healthy = "nothing", true
|
||||
return l.finish(ctx, lg, rep)
|
||||
}
|
||||
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
|
||||
rep.RebootNeeded = true
|
||||
}
|
||||
rep = l.finish(ctx, lg, rep) // the hub hears "staged" BEFORE the box goes down
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-reboot", vmid, nil))
|
||||
switch {
|
||||
case err != nil:
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
|
||||
Mode: "kernel-reboot", ReleaseID: want, Outcome: "failed", HealthReason: "kernel-reboot: " + err.Error()})
|
||||
case wr.refused() || wr.failed():
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
|
||||
Mode: "kernel-reboot", ReleaseID: want, Outcome: "refused", Refused: firstRaw(wr.Refused, wr.Failed),
|
||||
Kernel: rawOrNil(wr.Kernel)})
|
||||
}
|
||||
lg.Warn("osupdate: kernel step — the box restarts now for its one-shot boot", "to", want)
|
||||
return rep
|
||||
}
|
||||
|
||||
func firstRaw(a, b json.RawMessage) json.RawMessage {
|
||||
if r := rawOrNil(a); r != nil {
|
||||
return r
|
||||
}
|
||||
return rawOrNil(b)
|
||||
}
|
||||
|
||||
// KernelJudge is what KernelAfterBoot needs besides the leg: the hub reachability probe is the "judging" report itself.
|
||||
type KernelJudge struct {
|
||||
Wait time.Duration // default DefaultKernelJudgeWait
|
||||
Poll time.Duration // default 30 s
|
||||
}
|
||||
|
||||
// KernelAfterBoot runs once at daemon start: what became of a kernel step across the boot. On the new kernel it judges
|
||||
// the boot (blocking up to the wait — run it in a goroutine). vmid 0 = the guest the step recorded (it may not run yet).
|
||||
func (l *Leg) KernelAfterBoot(ctx context.Context, vmid int, j KernelJudge) Report {
|
||||
runID := "boot-" + l.now().UTC().Format("20060102T150405Z")
|
||||
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid)
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-boot", vmid, nil))
|
||||
if err != nil {
|
||||
lg.Warn("osupdate: kernel after-boot check failed", "err", err)
|
||||
return Report{}
|
||||
}
|
||||
if wr.refused() {
|
||||
lg.Info("osupdate: kernel after-boot check refused (an older wrapper, or a BYO host)", "refused", string(wr.Refused))
|
||||
return Report{}
|
||||
}
|
||||
v := parseKernel(wr.Kernel)
|
||||
if vmid <= 0 {
|
||||
vmid = v.VMID // after a boot the guest may not run yet — the step's own record names it
|
||||
}
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-boot",
|
||||
ReleaseID: v.To, Kernel: rawOrNil(wr.Kernel)}
|
||||
switch wr.KernelEvent {
|
||||
case "fell_back":
|
||||
rep.Outcome, rep.HealthReason = "fell_back", v.Reason
|
||||
lg.Warn("osupdate: kernel step FELL BACK — the new kernel did not come up; the box runs the old one", "from", v.From, "to", v.To)
|
||||
return l.finish(ctx, lg, rep)
|
||||
case "self_reverted":
|
||||
rep.Outcome, rep.HealthReason = "self_reverted", v.Reason
|
||||
lg.Warn("osupdate: kernel step SELF-REVERTED — back on the old kernel", "from", v.From, "to", v.To, "reason", v.Reason)
|
||||
return l.finish(ctx, lg, rep)
|
||||
case "revert_failed":
|
||||
rep.Outcome, rep.HealthReason = "revert_failed", v.Reason
|
||||
lg.Error("osupdate: kernel self-revert came back on the NEW kernel — no second revert; the operator decides", "to", v.To)
|
||||
return l.finish(ctx, lg, rep)
|
||||
case "judging":
|
||||
return l.judgeKernel(ctx, runID, vmid, v, wr.HealthBefore, j, lg)
|
||||
}
|
||||
return Report{}
|
||||
}
|
||||
|
||||
// KernelVerdict is THE one-shot boot rule (R-836; pinned by TestKernelVerdict): the host health rule (`11` §8.2 —
|
||||
// the Proxmox daemons and the agent active, the customer guest running and its own rule passing, the tunnel running)
|
||||
// AND the box reached the hub since this boot.
|
||||
func KernelVerdict(before, after *Health, tunnel string, hubReached bool) (bool, string) {
|
||||
if ok, why := HostHealthVerdict(before, after, tunnel); !ok {
|
||||
return false, why
|
||||
}
|
||||
if !hubReached {
|
||||
return false, "the box has not reached the hub since the boot"
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelView, before *Health, j KernelJudge, lg *slog.Logger) Report {
|
||||
wait, poll := j.Wait, j.Poll
|
||||
if wait <= 0 {
|
||||
wait = DefaultKernelJudgeWait
|
||||
}
|
||||
if poll <= 0 {
|
||||
poll = 30 * time.Second
|
||||
}
|
||||
lg.Info("osupdate: kernel step — judging the one-shot boot", "from", v.From, "to", v.To, "wait", wait.String())
|
||||
start := l.now()
|
||||
deadline := start.Add(wait)
|
||||
hubReached := false
|
||||
var why string
|
||||
for {
|
||||
if !hubReached && l.Hub != nil {
|
||||
// the hub's reachability IS this report reaching it (and the operator sees the box is back on the new kernel)
|
||||
body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||
Mode: "kernel-boot", ReleaseID: v.To, Outcome: "judging", Kernel: mustRaw(v)})
|
||||
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
if err := l.Hub.PostOSReport(rctx, body); err == nil {
|
||||
hubReached = true
|
||||
lg.Info("osupdate: kernel step — the box reached the hub on the new kernel", "after", l.now().Sub(start).Round(time.Second).String())
|
||||
}
|
||||
cancel()
|
||||
}
|
||||
var h *Health
|
||||
hr, err := l.call(ctx, runID, kernelPlan("health", vmid, nil))
|
||||
switch {
|
||||
case err != nil:
|
||||
why = "no health reading: " + err.Error()
|
||||
case hr.refused():
|
||||
why = "no health reading: " + string(hr.Refused)
|
||||
default:
|
||||
h = hr.Health
|
||||
}
|
||||
if h != nil {
|
||||
t := hub.TunnelUnknown
|
||||
if l.Tunnel != nil {
|
||||
t, _ = l.Tunnel.Status(ctx)
|
||||
}
|
||||
var ok bool
|
||||
ok, why = KernelVerdict(before, h, t, hubReached)
|
||||
if ok {
|
||||
return l.kernelGood(ctx, runID, vmid, v, start, lg)
|
||||
}
|
||||
}
|
||||
if !l.now().Before(deadline) || ctx.Err() != nil {
|
||||
break
|
||||
}
|
||||
l.sleep(ctx, poll)
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
lg.Warn("osupdate: kernel judging stopped (the agent is stopping) — the next start judges again", "reason", why)
|
||||
return Report{}
|
||||
}
|
||||
// not healthy by the deadline: tell the hub (best effort), then ONE self-revert into the old kernel
|
||||
rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "health_failed", HealthReason: why + " — reverting to " + v.From,
|
||||
Kernel: mustRaw(v)})
|
||||
lg.Error("osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel", "reason", why,
|
||||
"waited", wait.String(), "from", v.From, "to", v.To)
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-revert", vmid, map[string]any{"reason": truncate(why, 280)}))
|
||||
if err != nil || wr.refused() || wr.failed() {
|
||||
lg.Error("osupdate: kernel self-revert did not start — the box stays on the new kernel; the operator decides",
|
||||
"err", err, "refused", string(firstRaw(wr.Refused, wr.Failed)))
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "revert_failed", Refused: firstRaw(wr.Refused, wr.Failed),
|
||||
HealthReason: "the self-revert did not start"})
|
||||
}
|
||||
return rep
|
||||
}
|
||||
|
||||
func (l *Leg) kernelGood(ctx context.Context, runID string, vmid int, v KernelView, start time.Time, lg *slog.Logger) Report {
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-good", vmid, nil))
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-good",
|
||||
ReleaseID: v.To}
|
||||
switch {
|
||||
case err != nil:
|
||||
rep.Outcome, rep.HealthReason = "failed", "kernel-good: "+err.Error()
|
||||
case wr.refused() || wr.failed():
|
||||
rep.Outcome, rep.Refused, rep.HealthReason = "failed", firstRaw(wr.Refused, wr.Failed), "kernel-good did not move the default"
|
||||
default:
|
||||
rep.Outcome, rep.Healthy = "applied", true
|
||||
rep.HealthReason = fmt.Sprintf("healthy %s after the agent started; the new kernel is the default", l.now().Sub(start).Round(time.Second))
|
||||
}
|
||||
rep.Kernel = rawOrNil(wr.Kernel)
|
||||
lg.Info("osupdate: kernel step — "+rep.Outcome, "to", v.To, "reason", rep.HealthReason)
|
||||
return l.finish(ctx, lg, rep)
|
||||
}
|
||||
|
||||
func mustRaw(v any) json.RawMessage {
|
||||
b, _ := json.Marshal(v)
|
||||
return b
|
||||
}
|
||||
|
||||
func truncate(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n]
|
||||
}
|
||||
|
||||
// KernelSet is the package set that installs exactly kver (R-898; the hub's kernelSet, field-exact): the series
|
||||
// meta-package and the signed image, both at the kernel's own version. Proxmox keeps old kernel versions in its archive.
|
||||
// nil for a string that is not a kernel version.
|
||||
func KernelSet(kver string) []Package {
|
||||
m := kverSeriesRE.FindStringSubmatch(kver)
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
v := kver[:len(kver)-len("-pve")]
|
||||
return []Package{{Name: "proxmox-kernel-" + m[1], Version: v, Origin: PVEOrigin},
|
||||
{Name: "proxmox-kernel-" + kver + "-signed", Version: v, Origin: PVEOrigin}}
|
||||
}
|
||||
|
||||
var kverSeriesRE = regexp.MustCompile(`^([0-9]+\.[0-9]+)\.[0-9]+-[0-9]+-pve$`)
|
||||
|
||||
// KernelStepParams are a signed os_kernel_step's params: the exact kernel set (the wrapper compares it with the plan).
|
||||
type KernelStepParams struct {
|
||||
ReleaseID string `json:"release_id"`
|
||||
Packages []Package `json:"packages"`
|
||||
Kver string `json:"kver"`
|
||||
VMID int `json:"vmid,omitempty"`
|
||||
}
|
||||
|
||||
// KernelStepExecutor STAGES a verified os_kernel_step (signedjobs.Executor) under the heavy-op gate. It never reboots:
|
||||
// the night leg reboots a staged kernel on a night the household was told about.
|
||||
type KernelStepExecutor struct {
|
||||
Leg *Leg
|
||||
Guest func(ctx context.Context) (int, error)
|
||||
Gate func(ctx context.Context) (release func(), err error)
|
||||
}
|
||||
|
||||
// Execute implements signedjobs.Executor.
|
||||
func (e KernelStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
|
||||
if op != OpKernelStep {
|
||||
return signedjobs.ErrNoExecutor
|
||||
}
|
||||
so, ok := signedjobs.SignedOpFrom(ctx)
|
||||
if !ok {
|
||||
return fmt.Errorf("os_kernel_step: no signed envelope in the context — the wrapper could not verify it")
|
||||
}
|
||||
var p KernelStepParams
|
||||
if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 || !kverRE.MatchString(p.Kver) {
|
||||
return fmt.Errorf("os_kernel_step: params must name the kernel set and its kver: %v", err)
|
||||
}
|
||||
vmid := p.VMID
|
||||
if vmid == 0 {
|
||||
if e.Guest == nil {
|
||||
return fmt.Errorf("os_kernel_step: no vmid and no guest finder")
|
||||
}
|
||||
v, err := e.Guest(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("os_kernel_step: find the customer guest: %w", err)
|
||||
}
|
||||
vmid = v
|
||||
}
|
||||
if e.Gate != nil {
|
||||
release, err := e.Gate(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("os_kernel_step: heavy-op gate busy (a backup or restore-test runs): %w", err)
|
||||
}
|
||||
defer release()
|
||||
}
|
||||
rep := e.Leg.StageKernelSigned(ctx, vmid, p, so.Blob, string(so.Sig))
|
||||
if rep.Outcome == "staged" || rep.Outcome == "nothing" {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("os_kernel_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused))
|
||||
}
|
||||
|
||||
// StageKernelSigned stages a signed kernel set (ring 1): install + flag, no reboot.
|
||||
func (l *Leg) StageKernelSigned(ctx context.Context, vmid int, p KernelStepParams, blob []byte, sig string) Report {
|
||||
unlock := l.lockPass(true)
|
||||
defer unlock()
|
||||
l.sendUnsentLocked(ctx) // R-868
|
||||
runID := l.now().UTC().Format("20060102T150405Z")
|
||||
rid := p.ReleaseID
|
||||
if rid == "" {
|
||||
rid = "signed-" + runID
|
||||
}
|
||||
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", "signed", "release", rid)
|
||||
wr, err := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": rid, "select": "listed",
|
||||
"packages": p.Packages, "expect_kver": p.Kver, "run_id": runID, "trigger": "signed", "ring": l.Block().Ring,
|
||||
"signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}}))
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "signed", Ring: l.Block().Ring, VMID: vmid, Mode: "apply",
|
||||
ReleaseID: rid, Kernel: rawOrNil(wr.Kernel), unsent: reportFile(l.planDir(), runID, LayerKernel, "apply")}
|
||||
switch {
|
||||
case err != nil:
|
||||
rep.Outcome, rep.HealthReason = "failed", err.Error()
|
||||
case wr.refused():
|
||||
rep.Outcome, rep.Refused = "refused", wr.Refused
|
||||
case wr.failed():
|
||||
rep.Outcome, rep.Refused = "failed", wr.Failed
|
||||
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
|
||||
rep.Outcome, rep.Healthy = "nothing", true
|
||||
default:
|
||||
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
|
||||
rep.RebootNeeded = true
|
||||
}
|
||||
return l.finish(ctx, lg, rep)
|
||||
}
|
||||
@@ -0,0 +1,328 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
)
|
||||
|
||||
// ---- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----
|
||||
|
||||
const kOld, kNew = "7.0.2-6-pve", "7.0.14-22-pve"
|
||||
|
||||
func kview(phase string) json.RawMessage {
|
||||
return mustRaw(KernelView{Running: kOld, Default: kOld, Phase: phase, From: kOld, To: kNew, VMID: 9201})
|
||||
}
|
||||
|
||||
func tonight(ring int) *hub.WireOSUpdate {
|
||||
return &hub.WireOSUpdate{Ring: ring, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}
|
||||
}
|
||||
|
||||
func kernelCalls(w *fakeWrapper) []string {
|
||||
var m []string
|
||||
for _, p := range w.plans {
|
||||
if p["layer"] == LayerKernel {
|
||||
m = append(m, p["mode"].(string))
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the
|
||||
// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night.
|
||||
func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{
|
||||
"kernel-status": {{Kernel: kview("none")}},
|
||||
"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Authority: "ring0", Kernel: kview("staged")}},
|
||||
"kernel-reboot": {{Kernel: kview("oneshot")}},
|
||||
}}
|
||||
l, h := newLeg(t, w, tonight(0))
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply,kernel-reboot" {
|
||||
t.Fatalf("kernel calls = %s", got)
|
||||
}
|
||||
var ap map[string]any
|
||||
for _, x := range w.plans {
|
||||
if x["layer"] == LayerKernel && x["mode"] == "apply" {
|
||||
ap = x
|
||||
}
|
||||
}
|
||||
// R-898: EXACTLY the told kernel — the listed set derived from it, never "pending" (red before the fix: select was
|
||||
// pending-kernel, so a newer kernel in the sources by night was refused R23 and the night was lost)
|
||||
pk, _ := json.Marshal(ap["packages"])
|
||||
if ap["select"] != "listed" || ap["expect_kver"] != kNew || ap["lane"] != "slow" ||
|
||||
string(pk) != `[{"name":"proxmox-kernel-7.0","origin":"Proxmox Debian Repository","version":"7.0.14-22"},{"name":"proxmox-kernel-7.0.14-22-pve-signed","origin":"Proxmox Debian Repository","version":"7.0.14-22"}]` {
|
||||
t.Fatalf("stage plan = %v (packages %s)", ap, pk)
|
||||
}
|
||||
if p.Kernel.Outcome != "staged" || !p.Kernel.Healthy {
|
||||
t.Fatalf("kernel report = %+v", p.Kernel)
|
||||
}
|
||||
last := h.reports[len(h.reports)-1]
|
||||
if last.Layer != LayerKernel || last.Outcome != "staged" {
|
||||
t.Fatalf("the hub must hear 'staged' before the reboot: %+v", h.reports)
|
||||
}
|
||||
// the kernel step is the LAST wrapper call of the night
|
||||
if lp := w.plans[len(w.plans)-1]; lp["layer"] != LayerKernel || lp["mode"] != "kernel-reboot" {
|
||||
t.Fatalf("the reboot must end the night, last call = %v", lp)
|
||||
}
|
||||
}
|
||||
|
||||
// No mail, no step: a kernel the household was NOT told about never runs; nor in a debug pass; nor without a block.
|
||||
// COMPANION RED-PROOF (observed): drop the `!blk.Kernel.Tonight` case in kernelDue → the first sub-case fails.
|
||||
func TestKernel_NoMailNoStep(t *testing.T) {
|
||||
cases := map[string]struct {
|
||||
blk *hub.WireOSUpdate
|
||||
trigger string
|
||||
}{
|
||||
"not told": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: false}}, "night"},
|
||||
"debug pass": {tonight(0), "debug"},
|
||||
"no block": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "night"},
|
||||
"switch off": {&hub.WireOSUpdate{Ring: 0, Enabled: false, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}, "night"},
|
||||
"bad kver": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: "7.0; reboot", Tonight: true}}, "night"},
|
||||
}
|
||||
for name, c := range cases {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, c.blk)
|
||||
p := l.Run(context.Background(), 9201, c.trigger)
|
||||
if len(kernelCalls(w)) != 0 || p.Kernel.Layer != "" {
|
||||
t.Fatalf("%s: a kernel step ran: %v", name, kernelCalls(w))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The kernel step needs a healthy host step on an appliance, and a healthy Proxmox step when one ran.
|
||||
func TestKernel_SkippedWithoutHealthyEarlierSteps(t *testing.T) {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, tonight(0))
|
||||
l.Appliance = false
|
||||
l.Run(context.Background(), 9201, "night")
|
||||
if len(kernelCalls(w)) != 0 {
|
||||
t.Fatalf("a BYO box took a kernel step: %v", kernelCalls(w))
|
||||
}
|
||||
w2 := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}},
|
||||
PVEManager: "9.2.2"}}} // pveversion still old → the pve step is unhealthy
|
||||
l2, _ := newLeg(t, w2, tonight(0))
|
||||
l2.Run(context.Background(), 9201, "night")
|
||||
if len(kernelCalls(w2)) != 0 {
|
||||
t.Fatalf("a kernel step ran after an unhealthy Proxmox step: %v", kernelCalls(w2))
|
||||
}
|
||||
}
|
||||
|
||||
// Ring 1 reboots only a kernel a signed job staged — never stages one itself in the night leg.
|
||||
func TestKernel_Ring1RebootsOnlyASignedStage(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}}}
|
||||
l, _ := newLeg(t, w, tonight(1))
|
||||
l.Run(context.Background(), 9201, "night")
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status" {
|
||||
t.Fatalf("ring 1 without a staged kernel: calls = %s", got)
|
||||
}
|
||||
w2 := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("staged")}},
|
||||
"kernel-reboot": {{Kernel: kview("oneshot")}}}}
|
||||
l2, _ := newLeg(t, w2, tonight(1))
|
||||
p := l2.Run(context.Background(), 9201, "night")
|
||||
if got := strings.Join(kernelCalls(w2), ","); got != "kernel-status,kernel-reboot" || p.Kernel.Outcome != "staged" {
|
||||
t.Fatalf("ring 1 with a staged kernel: calls = %s report = %+v", got, p.Kernel)
|
||||
}
|
||||
}
|
||||
|
||||
// A refused stage never reboots.
|
||||
func TestKernel_RefusedStageNeverReboots(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}},
|
||||
"apply": {{Refused: json.RawMessage(`{"code":"R20","reason":"/boot/efi is not a mounted vfat ESP"}`)}}}}
|
||||
l, h := newLeg(t, w, tonight(0))
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply" || p.Kernel.Outcome != "refused" {
|
||||
t.Fatalf("calls = %s report = %+v", got, p.Kernel)
|
||||
}
|
||||
if last := h.reports[len(h.reports)-1]; last.Layer != LayerKernel || last.Outcome != "refused" {
|
||||
t.Fatalf("the hub must hear the refusal: %+v", last)
|
||||
}
|
||||
}
|
||||
|
||||
// THE one-shot boot rule: the host rule AND the hub reached. COMPANION RED-PROOF (observed): drop the hubReached
|
||||
// check in KernelVerdict → the second case fails.
|
||||
func TestKernelVerdict(t *testing.T) {
|
||||
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, true); !ok {
|
||||
t.Fatalf("a healthy boot read unhealthy: %s", why)
|
||||
}
|
||||
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, false); ok || !strings.Contains(why, "hub") {
|
||||
t.Fatalf("a box that has not reached the hub must not pass: ok=%v %q", ok, why)
|
||||
}
|
||||
down := hostOK()
|
||||
down.GuestRunning = new(bool)
|
||||
if ok, _ := KernelVerdict(hostOK(), down, hub.TunnelRunning, true); ok {
|
||||
t.Fatal("a guest that does not run must fail")
|
||||
}
|
||||
if ok, _ := KernelVerdict(hostOK(), hostOK(), hub.TunnelUnknown, true); ok {
|
||||
t.Fatal("an unknown tunnel must fail (the host rule)")
|
||||
}
|
||||
}
|
||||
|
||||
func judgingLeg(t *testing.T, w *fakeWrapper) (*Leg, *fakeHub) {
|
||||
if w.kernelRep == nil {
|
||||
w.kernelRep = map[string][]WrapperReport{}
|
||||
}
|
||||
if _, ok := w.kernelRep["kernel-boot"]; !ok {
|
||||
w.kernelRep["kernel-boot"] = []WrapperReport{{KernelEvent: "judging", Kernel: mustRaw(KernelView{Running: kNew,
|
||||
Default: kOld, Phase: "judging", From: kOld, To: kNew, VMID: 9201}), HealthBefore: hostOK()}}
|
||||
}
|
||||
return newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
}
|
||||
|
||||
// A healthy one-shot boot: the hub hears "judging", then kernel-good, then "applied".
|
||||
func TestKernelAfterBoot_HealthyBecomesTheDefault(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-good": {{Kernel: kview("good")}}}}
|
||||
l, h := judgingLeg(t, w)
|
||||
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: 30 * time.Second})
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot,health,kernel-good" {
|
||||
t.Fatalf("calls = %s", got)
|
||||
}
|
||||
if r.Outcome != "applied" || !r.Healthy {
|
||||
t.Fatalf("report = %+v", r)
|
||||
}
|
||||
if len(h.reports) != 2 || h.reports[0].Outcome != "judging" || h.reports[1].Outcome != "applied" {
|
||||
t.Fatalf("hub reports = %+v", h.reports)
|
||||
}
|
||||
if w.plans[1]["vmid"] != float64(9201) {
|
||||
t.Fatalf("the health reading must use the step's own guest, got %v", w.plans[1]["vmid"])
|
||||
}
|
||||
}
|
||||
|
||||
// An unhealthy one-shot boot: wait the full judge time, tell the hub, then ONE kernel-revert.
|
||||
// COMPANION RED-PROOF (observed): return before the kernel-revert call in judgeKernel → "calls" fails.
|
||||
func TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait(t *testing.T) {
|
||||
down := hostOK()
|
||||
down.GuestRunning = new(bool)
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"health": {{Health: down}},
|
||||
"kernel-revert": {{Kernel: kview("reverting")}}}}
|
||||
l, h := judgingLeg(t, w)
|
||||
start := l.now()
|
||||
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: time.Minute})
|
||||
calls := kernelCalls(w)
|
||||
if calls[len(calls)-1] != "kernel-revert" || strings.Count(strings.Join(calls, ","), "kernel-revert") != 1 {
|
||||
t.Fatalf("calls = %v", calls)
|
||||
}
|
||||
if waited := l.now().Sub(start); waited < 10*time.Minute {
|
||||
t.Fatalf("reverted after %s — before the judge wait", waited)
|
||||
}
|
||||
if r.Outcome != "health_failed" || !strings.Contains(r.HealthReason, "not running") {
|
||||
t.Fatalf("report = %+v", r)
|
||||
}
|
||||
if last := h.reports[len(h.reports)-1]; last.Outcome != "health_failed" {
|
||||
t.Fatalf("the hub must hear health_failed before the revert reboot: %+v", h.reports)
|
||||
}
|
||||
for _, p := range w.plans {
|
||||
if p["mode"] == "kernel-revert" && !strings.Contains(p["reason"].(string), "not running") {
|
||||
t.Fatalf("the revert must carry the reason: %v", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A box that never reaches the hub is not "healthy" — it reverts too.
|
||||
func TestKernelAfterBoot_NoHubMeansRevert(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-revert": {{Kernel: kview("reverting")}}}}
|
||||
l, _ := judgingLeg(t, w)
|
||||
l.Hub = unreachableHub{}
|
||||
l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 5 * time.Minute, Poll: time.Minute})
|
||||
if c := kernelCalls(w); c[len(c)-1] != "kernel-revert" {
|
||||
t.Fatalf("calls = %v", c)
|
||||
}
|
||||
}
|
||||
|
||||
type unreachableHub struct{}
|
||||
|
||||
func (unreachableHub) PostOSReport(context.Context, []byte) error { return context.DeadlineExceeded }
|
||||
|
||||
// What kernel-boot found becomes the hub's outcome, with no judging and no reboot.
|
||||
func TestKernelAfterBoot_FallBackAndRevertResultsAreReported(t *testing.T) {
|
||||
for ev, want := range map[string]string{"fell_back": "fell_back", "self_reverted": "self_reverted", "revert_failed": "revert_failed"} {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: ev,
|
||||
Kernel: mustRaw(KernelView{Running: kOld, Default: kOld, Phase: ev, From: kOld, To: kNew, Reason: "r", VMID: 9201})}}}}
|
||||
l, h := judgingLeg(t, w)
|
||||
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{})
|
||||
if r.Outcome != want || len(h.reports) != 1 || h.reports[0].Outcome != want {
|
||||
t.Fatalf("%s: report %+v hub %+v", ev, r, h.reports)
|
||||
}
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot" {
|
||||
t.Fatalf("%s: calls = %s", ev, got)
|
||||
}
|
||||
}
|
||||
// nothing to do → nothing reported
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: "none", Kernel: kview("good")}}}}
|
||||
l, h := judgingLeg(t, w)
|
||||
if r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{}); r.Layer != "" || len(h.reports) != 0 {
|
||||
t.Fatalf("an ordinary boot must report nothing: %+v %+v", r, h.reports)
|
||||
}
|
||||
}
|
||||
|
||||
// The signed executor STAGES (listed + the raw envelope + the kver) and never reboots.
|
||||
func TestKernelStepExecutor_StagesNeverReboots(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0",
|
||||
Version: "7.0.14-22"}}, Authority: "signed", Kernel: kview("staged")}}}}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
e := KernelStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
|
||||
params, _ := json.Marshal(KernelStepParams{ReleaseID: "os-kernel-1", Kver: kNew,
|
||||
Packages: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22", Origin: PVEOrigin}}})
|
||||
ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_kernel_step"}`), Sig: []byte("SIG")})
|
||||
if err := e.Execute(ctx, OpKernelStep, params); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pp := w.plans[len(w.plans)-1]
|
||||
sg, _ := pp["signed"].(map[string]any)
|
||||
if pp["mode"] != "apply" || pp["select"] != "listed" || pp["expect_kver"] != kNew || sg == nil ||
|
||||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_kernel_step"}`)) {
|
||||
t.Fatalf("plan = %v", pp)
|
||||
}
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "apply" {
|
||||
t.Fatalf("a signed stage must never reboot: %s", got)
|
||||
}
|
||||
if len(h.reports) != 1 || h.reports[0].Outcome != "staged" {
|
||||
t.Fatalf("hub = %+v", h.reports)
|
||||
}
|
||||
if err := e.Execute(context.Background(), OpKernelStep, params); err == nil {
|
||||
t.Fatal("no envelope must refuse")
|
||||
}
|
||||
bad, _ := json.Marshal(KernelStepParams{Kver: "x", Packages: []Package{{Name: "a"}}})
|
||||
if err := e.Execute(ctx, OpKernelStep, bad); err == nil {
|
||||
t.Fatal("a bad kver must refuse")
|
||||
}
|
||||
if err := e.Execute(ctx, OpPVEStep, params); err != signedjobs.ErrNoExecutor {
|
||||
t.Fatalf("another op must pass through the chain: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// os_kernel_step is never benign.
|
||||
func TestKernelStep_IsDestructiveClass(t *testing.T) {
|
||||
if reconcile.Classify(reconcile.ClassOSKernelStep, reconcile.Provenance{}) != reconcile.Destructive {
|
||||
t.Fatal("os_kernel_step must be destructive-class (signed, operational key)")
|
||||
}
|
||||
}
|
||||
|
||||
// A kept stage report (the agent was killed mid-stage) reaches the hub as "staged" with its kernel view.
|
||||
func TestKernel_KeptStageReportIsStaged(t *testing.T) {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, tonight(0))
|
||||
ring := 0
|
||||
rep := l.reportFromKept(context.Background(), WrapperReport{Layer: LayerKernel, Mode: "apply", Ring: &ring,
|
||||
Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Kernel: kview("staged")}, "/x/report-r-kernel-apply.json")
|
||||
if rep.Outcome != "staged" || !rep.Healthy || len(rep.Kernel) == 0 {
|
||||
t.Fatalf("kept = %+v", rep)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelSet(t *testing.T) {
|
||||
if got := KernelSet("7.0.14-20-pve"); len(got) != 2 || got[0].Name != "proxmox-kernel-7.0" || got[0].Version != "7.0.14-20" ||
|
||||
got[1].Name != "proxmox-kernel-7.0.14-20-pve-signed" {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
if KernelSet("7.0; reboot") != nil || KernelSet("") != nil {
|
||||
t.Fatal("a non-kernel string must give no set")
|
||||
}
|
||||
}
|
||||
+168
-6
@@ -20,6 +20,7 @@ import (
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -27,6 +28,7 @@ import (
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||
)
|
||||
|
||||
// WrapperPath is the pinned sudoers vector (configs/felhom-agent.sudoers FELHOM_OSAPPLY).
|
||||
@@ -40,8 +42,24 @@ const (
|
||||
LayerGuest = "guest"
|
||||
LayerHost = "host"
|
||||
LayerDocker = "docker" // the guest's Docker engine set — slow lane (`11` §5.8)
|
||||
// LayerPVE is the HOST's Proxmox userspace packages — slow lane (R-812 option A, `09` §3 decision 163, `11` §5.10):
|
||||
// ring 0 in the night leg after a healthy host step, ring 1 only inside a signed os_pve_step. Never a kernel.
|
||||
LayerPVE = "pve"
|
||||
// LayerKernel is the HOST's kernel — the kernel lane (R-836, `09` §3 decision 172, `11` §5.11): a one-shot boot of
|
||||
// the new kernel through the ESP flag, the default moved only after a healthy boot (kernel.go).
|
||||
LayerKernel = "kernel"
|
||||
)
|
||||
|
||||
// PVEOrigin is the origin apt prints for download.proxmox.com (the wrapper's PVE_ORIGIN); InstalledPVEOrigin is how
|
||||
// the wrapper's inventory names the same source.
|
||||
const (
|
||||
PVEOrigin = "Proxmox Debian Repository"
|
||||
InstalledPVEOrigin = "Proxmox"
|
||||
)
|
||||
|
||||
// hostSlowRE mirrors the wrapper's HOST_SLOW_RE: kernel, boot, firmware and microcode names never ride the pve lane.
|
||||
var hostSlowRE = regexp.MustCompile(`^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)`)
|
||||
|
||||
// DockerNames are the six packages of the Docker engine set (the wrapper's DOCKER_NAMES).
|
||||
var DockerNames = map[string]bool{"containerd.io": true, "docker-buildx-plugin": true, "docker-ce": true,
|
||||
"docker-ce-cli": true, "docker-ce-rootless-extras": true, "docker-compose-plugin": true}
|
||||
@@ -106,6 +124,16 @@ type WrapperReport struct {
|
||||
LiveRestore json.RawMessage `json:"live_restore"`
|
||||
Facts json.RawMessage `json:"facts"`
|
||||
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
|
||||
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
|
||||
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
|
||||
OOMCheck json.RawMessage `json:"oom_check"`
|
||||
// PVEManager: the pve layer — pveversion's pve-manager version after the step ("unknown" = unreadable).
|
||||
PVEManager string `json:"pve_manager"`
|
||||
// Kernel (R-836): the kernel layer's view {running, default, flag, phase, from, to, …}; KernelEvent what kernel-boot
|
||||
// found after a boot; OutcomeHint "nothing" when no kernel was pending.
|
||||
Kernel json.RawMessage `json:"kernel"`
|
||||
KernelEvent string `json:"kernel_event"`
|
||||
OutcomeHint string `json:"outcome_hint"`
|
||||
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
|
||||
// agent process that started the pass. ReleaseID / VMID were always in the report.
|
||||
RunID string `json:"run_id"`
|
||||
@@ -143,6 +171,15 @@ type Report struct {
|
||||
DockerEngine string `json:"docker_engine,omitempty"` // docker layer: the engine after the step
|
||||
Authority string `json:"authority,omitempty"` // docker layer: ring0 | signed
|
||||
Undo bool `json:"undo,omitempty"` // docker layer: a signed undo (downgrade)
|
||||
// OOMCheck: docker layer — the wrapper's oom_check object, byte-for-byte (R-528; the hub decides approval on it).
|
||||
// Pinned by TestDocker_OOMCheckReachesTheHubUnchanged and TestR868_KeptCopyCarriesTheOOMCheck.
|
||||
OOMCheck json.RawMessage `json:"oom_check,omitempty"`
|
||||
// PVEManager: pve layer — pve-manager's version after the step (the hub's System page; R-812 option A).
|
||||
PVEManager string `json:"pve_manager,omitempty"`
|
||||
// Kernel: kernel layer — the wrapper's kernel view, byte for byte (running, default, flag, phase, from, to). The
|
||||
// outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert
|
||||
// started) | self_reverted | revert_failed | judging | nothing | refused | failed.
|
||||
Kernel json.RawMessage `json:"kernel,omitempty"`
|
||||
|
||||
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
|
||||
}
|
||||
@@ -380,6 +417,36 @@ func EngineOf(pkgVersion string) string {
|
||||
return v
|
||||
}
|
||||
|
||||
// PVEHealthVerdict is THE Proxmox-package-step health rule (R-812 option A; pinned by TestPVEHealthVerdict): the host
|
||||
// rule (every host service active, the guest running and healthy, the tunnel running), plus every container running at
|
||||
// the start still runs as the SAME container (a Proxmox step must not restart the household's apps), plus pveversion
|
||||
// now reports the pve-manager the step installed (wantPVE "" = pve-manager was not in the step).
|
||||
func PVEHealthVerdict(before, after *Health, tunnel, wantPVE, gotPVE string) (bool, string) {
|
||||
if ok, why := HostHealthVerdict(before, after, tunnel); !ok {
|
||||
return false, why
|
||||
}
|
||||
if before != nil && before.Guest != nil && after.Guest != nil {
|
||||
names := make([]string, 0, len(before.Guest.Containers))
|
||||
for n := range before.Guest.Containers {
|
||||
names = append(names, n)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, n := range names {
|
||||
b := before.Guest.Containers[n]
|
||||
if b.State != "running" || b.ID == "" {
|
||||
continue
|
||||
}
|
||||
if a := after.Guest.Containers[n]; a.ID != b.ID {
|
||||
return false, n + " is a new container (id changed) — the Proxmox step restarted the household's app"
|
||||
}
|
||||
}
|
||||
}
|
||||
if wantPVE != "" && gotPVE != wantPVE {
|
||||
return false, "pveversion reads pve-manager " + gotPVE + ", not " + wantPVE
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
// DockerHealthVerdict is THE Docker-step health rule (`11` §5.8; pinned by TestDockerHealthVerdict): the guest rule,
|
||||
// plus every container running at the start still runs as the SAME container (same id — a changed id means the
|
||||
// household's apps restarted, which `live-restore` exists to prevent), plus the engine now reports the version the step
|
||||
@@ -445,7 +512,7 @@ func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (Wrap
|
||||
|
||||
// Pass is one leg's reports; an empty Layer means the step did not run.
|
||||
type Pass struct {
|
||||
Guest, Host, Docker Report
|
||||
Guest, Host, Docker, PVE, Kernel Report
|
||||
}
|
||||
|
||||
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
|
||||
@@ -473,13 +540,54 @@ func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
|
||||
if err := l.EnsureLiveRestore(ctx, g.RunID, vmid); err != nil {
|
||||
p.Docker = l.finish(ctx, lg, Report{RunID: g.RunID, Layer: LayerDocker, Trigger: trigger, Ring: 0, VMID: vmid,
|
||||
Mode: "apply", Outcome: "failed", HealthReason: "live-restore could not be turned on: " + err.Error()})
|
||||
return p
|
||||
} else {
|
||||
p.Docker = l.runLayer(ctx, g.RunID, LayerDocker, vmid, trigger, blk, dockerOpts{})
|
||||
}
|
||||
p.Docker = l.runLayer(ctx, g.RunID, LayerDocker, vmid, trigger, blk, dockerOpts{})
|
||||
}
|
||||
// R-812 option A: the Proxmox package step — ring 0, an appliance, after a HEALTHY host step (it is a host change).
|
||||
// A Docker step's outcome does not gate it (the Docker set lives in the guest). Pinned by TestPVE_*.
|
||||
switch {
|
||||
case blk.Ring != 0 || !blk.Enabled:
|
||||
lg.Info("osupdate: pve step skipped — ring 1 takes a Proxmox set only inside a signed operator job (`11` §5.10)", "ring", blk.Ring, "enabled", blk.Enabled)
|
||||
case !l.Appliance || h.Layer == "" || !okStep(h):
|
||||
lg.Info("osupdate: pve step skipped — no healthy host step this pass (an appliance only)", "appliance", l.Appliance, "host_outcome", h.Outcome)
|
||||
default:
|
||||
p.PVE = l.runPVE(ctx, g.RunID, vmid, trigger, blk, dockerOpts{})
|
||||
}
|
||||
// R-836 (`09` §3 decision 172): the kernel step ENDS the night — an appliance, after a healthy host step (and a
|
||||
// healthy Proxmox step when one ran), only on a night the hub marks as told. It reboots the box. Pinned by TestKernel_*.
|
||||
switch {
|
||||
case !l.Appliance || h.Layer == "" || !okStep(h):
|
||||
lg.Info("osupdate: kernel step skipped — no healthy host step this pass (an appliance only)", "appliance", l.Appliance, "host_outcome", h.Outcome)
|
||||
case p.PVE.Layer != "" && !okStep(p.PVE):
|
||||
lg.Warn("osupdate: kernel step skipped — the Proxmox step did not end healthy", "pve_outcome", p.PVE.Outcome)
|
||||
default:
|
||||
p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// pveDrainWait bounds how long a pve step waits for the agent's own /etc/pve writes in flight (pvegate).
|
||||
var pveDrainWait = 2 * time.Minute
|
||||
|
||||
// runPVE runs the pve layer while holding pvegate: the agent's own /etc/pve writes wait until it ends.
|
||||
func (l *Leg) runPVE(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate, do dockerOpts) Report {
|
||||
lg := l.log().With("run", runID, "layer", LayerPVE, "vmid", vmid, "trigger", trigger)
|
||||
dctx, cancel := context.WithTimeout(ctx, pveDrainWait)
|
||||
end, err := pvegate.Step(dctx)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerPVE, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "apply",
|
||||
ReleaseID: do.releaseID, Outcome: "failed", HealthReason: "an agent write to /etc/pve did not finish in time (pvegate): " + err.Error()})
|
||||
}
|
||||
lg.Info("osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends")
|
||||
defer func() {
|
||||
end()
|
||||
lg.Info("osupdate: pve step released the /etc/pve write gate")
|
||||
}()
|
||||
return l.runLayer(ctx, runID, LayerPVE, vmid, trigger, blk, do)
|
||||
}
|
||||
|
||||
// dockerOpts is a signed Docker step (DockerStepExecutor); the zero value is ring 0's unsigned "pending-docker".
|
||||
type dockerOpts struct {
|
||||
releaseID string
|
||||
@@ -565,7 +673,7 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
||||
wire = blk.HostRelease
|
||||
}
|
||||
lane := "fast"
|
||||
if layer == LayerDocker {
|
||||
if layer == LayerDocker || layer == LayerPVE {
|
||||
lane = "slow"
|
||||
if do.signed != nil {
|
||||
rel = hub.WireOSRelease{ID: do.releaseID}
|
||||
@@ -598,6 +706,13 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
||||
}
|
||||
case layer == LayerDocker:
|
||||
plan["select"] = "pending-docker" // ring 0: the wrapper checks the box's ROOT-OWNED ring-0 mark itself
|
||||
case layer == LayerPVE && do.signed != nil:
|
||||
plan["packages"], plan["signed"] = do.packages, do.signed
|
||||
for _, p := range do.packages {
|
||||
planned[p.Name] = true
|
||||
}
|
||||
case layer == LayerPVE:
|
||||
plan["select"] = "pending-pve" // ring 0: the same root-owned mark; the wrapper picks installed Proxmox userspace
|
||||
case !blk.Enabled:
|
||||
plan["mode"] = "inventory"
|
||||
lg.Info("osupdate: switched OFF for this box — reporting only")
|
||||
@@ -630,6 +745,8 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
||||
}
|
||||
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
|
||||
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
|
||||
rep.OOMCheck = rawOrNil(wr.OOMCheck)
|
||||
rep.PVEManager = wr.PVEManager
|
||||
if rep.Outcome == "" {
|
||||
switch {
|
||||
case rep.Mode == "inventory" && !blk.Enabled:
|
||||
@@ -647,21 +764,27 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
||||
}
|
||||
// Health: compare with the start of the pass; give restarted services time (only after an install).
|
||||
cur := wr.HealthAfter
|
||||
wantEngine := ""
|
||||
wantEngine, wantPVE := "", ""
|
||||
for _, u := range wr.Upgraded {
|
||||
if u.Name == "docker-ce" {
|
||||
wantEngine = EngineOf(u.Version)
|
||||
}
|
||||
if u.Name == "pve-manager" {
|
||||
wantPVE = u.Version
|
||||
}
|
||||
}
|
||||
verdict := func(h *Health) (bool, string) {
|
||||
if layer == LayerDocker {
|
||||
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
|
||||
}
|
||||
if layer == LayerHost {
|
||||
if layer == LayerHost || layer == LayerPVE {
|
||||
t := hub.TunnelUnknown
|
||||
if l.Tunnel != nil {
|
||||
t, _ = l.Tunnel.Status(ctx)
|
||||
}
|
||||
if layer == LayerPVE {
|
||||
return PVEHealthVerdict(wr.HealthBefore, h, t, wantPVE, wr.PVEManager)
|
||||
}
|
||||
return HostHealthVerdict(wr.HealthBefore, h, t)
|
||||
}
|
||||
return HealthVerdict(wr.HealthBefore, h)
|
||||
@@ -701,12 +824,24 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
||||
// the docker report carries the engine set only (the guest report already carries the Debian packages)
|
||||
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
|
||||
rep.NotCovered = nil
|
||||
} else if layer == LayerPVE {
|
||||
// the pve report carries the Proxmox userspace set only — the hub's candidate is built from it
|
||||
rep.Installed, rep.Pending = onlyPVE(wr.Installed), onlyPVEPending(wr.Pending)
|
||||
rep.NotCovered = nil
|
||||
} else {
|
||||
rep.NotCovered = notCovered(wr.Pending, blk.Ring, planned)
|
||||
}
|
||||
return l.finish(ctx, lg, rep)
|
||||
}
|
||||
|
||||
// rawOrNil: a wrapper field that is absent or JSON null stays out of the hub report (omitempty).
|
||||
func rawOrNil(m json.RawMessage) json.RawMessage {
|
||||
if len(m) == 0 || string(m) == "null" {
|
||||
return nil
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func onlyDocker(in []Package) []Package {
|
||||
var out []Package
|
||||
for _, p := range in {
|
||||
@@ -717,6 +852,33 @@ func onlyDocker(in []Package) []Package {
|
||||
return out
|
||||
}
|
||||
|
||||
// onlyPVE keeps the installed Proxmox-origin packages the pve lane may touch (never a kernel / boot / firmware name).
|
||||
func onlyPVE(in []Package) []Package {
|
||||
var out []Package
|
||||
for _, p := range in {
|
||||
if (p.Origin == InstalledPVEOrigin || p.Origin == PVEOrigin) && !hostSlowRE.MatchString(p.Name) && !DockerNames[p.Name] {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func onlyPVEPending(in []Pending) []Pending {
|
||||
var out []Pending
|
||||
for _, p := range in {
|
||||
if p.From == "" || hostSlowRE.MatchString(p.Name) || DockerNames[p.Name] {
|
||||
continue
|
||||
}
|
||||
for _, o := range p.Origin {
|
||||
if o == PVEOrigin {
|
||||
out = append(out, p)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func onlyDockerPending(in []Pending) []Pending {
|
||||
var out []Pending
|
||||
for _, p := range in {
|
||||
|
||||
@@ -13,16 +13,19 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||
)
|
||||
|
||||
// fakeWrapper plays /usr/local/sbin/felhom-os-apply: it reads the plan the leg wrote and answers per layer and mode.
|
||||
type fakeWrapper struct {
|
||||
t *testing.T
|
||||
pending []Pending
|
||||
applyRep map[string]WrapperReport // per layer
|
||||
healthSeq map[string][]*Health // per layer: answers to successive "health" calls
|
||||
plans []map[string]any
|
||||
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
|
||||
t *testing.T
|
||||
pending []Pending
|
||||
applyRep map[string]WrapperReport // per layer
|
||||
healthSeq map[string][]*Health // per layer: answers to successive "health" calls
|
||||
plans []map[string]any
|
||||
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
|
||||
pveGateHeld bool
|
||||
kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats)
|
||||
}
|
||||
|
||||
func yes() *bool { b := true; return &b }
|
||||
@@ -50,9 +53,22 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
|
||||
f.plans = append(f.plans, plan)
|
||||
layer := plan["layer"].(string)
|
||||
ok := guestOK()
|
||||
if layer == LayerHost {
|
||||
if layer == LayerHost || layer == LayerPVE || layer == LayerKernel {
|
||||
ok = hostOK()
|
||||
}
|
||||
if layer == LayerKernel {
|
||||
if seq := f.kernelRep[plan["mode"].(string)]; len(seq) > 0 {
|
||||
rep := seq[0]
|
||||
if len(seq) > 1 {
|
||||
f.kernelRep[plan["mode"].(string)] = seq[1:]
|
||||
}
|
||||
out, _ := json.Marshal(rep)
|
||||
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
|
||||
}
|
||||
}
|
||||
if layer == LayerPVE && plan["mode"] == "apply" {
|
||||
f.pveGateHeld = pvegate.Stepping() // R-812: the /etc/pve write gate must be held while the pve step runs
|
||||
}
|
||||
var rep WrapperReport
|
||||
switch plan["mode"] {
|
||||
case "inventory":
|
||||
@@ -98,9 +114,13 @@ func (f *fakeWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, ar
|
||||
return f.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
type fakeHub struct{ reports []Report }
|
||||
type fakeHub struct {
|
||||
reports []Report
|
||||
bodies [][]byte // the exact bytes posted (R-528: the oom_check object must arrive unchanged)
|
||||
}
|
||||
|
||||
func (h *fakeHub) PostOSReport(_ context.Context, body []byte) error {
|
||||
h.bodies = append(h.bodies, append([]byte(nil), body...))
|
||||
var r Report
|
||||
json.Unmarshal(body, &r)
|
||||
h.reports = append(h.reports, r)
|
||||
@@ -156,8 +176,8 @@ func TestRing0_OneCallPerLayer(t *testing.T) {
|
||||
if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
|
||||
t.Fatalf("guest %+v\nhost %+v", g, ho)
|
||||
}
|
||||
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply" {
|
||||
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore and the ring-0 docker step", calls(w))
|
||||
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply" {
|
||||
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore, the ring-0 docker step and the pve step", calls(w))
|
||||
}
|
||||
for _, p := range w.plans[:2] {
|
||||
if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
|
||||
@@ -167,7 +187,7 @@ func TestRing0_OneCallPerLayer(t *testing.T) {
|
||||
if len(g.NotCovered) != 1 || g.NotCovered[0] != "docker-ce" {
|
||||
t.Fatalf("not covered = %v", g.NotCovered)
|
||||
}
|
||||
if len(h.reports) != 3 || h.reports[0].Layer != LayerGuest || h.reports[1].Layer != LayerHost || h.reports[2].Layer != LayerDocker {
|
||||
if len(h.reports) != 4 || h.reports[0].Layer != LayerGuest || h.reports[1].Layer != LayerHost || h.reports[2].Layer != LayerDocker || h.reports[3].Layer != LayerPVE {
|
||||
t.Fatalf("hub got %+v", h.reports)
|
||||
}
|
||||
}
|
||||
@@ -385,7 +405,7 @@ func TestHostReport_CarriesRebootScanned(t *testing.T) {
|
||||
}}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
run2(l, "night")
|
||||
if len(h.reports) != 3 || !h.reports[1].RebootScanned || !h.reports[1].RebootNeeded || h.reports[0].RebootScanned {
|
||||
if len(h.reports) != 4 || !h.reports[1].RebootScanned || !h.reports[1].RebootNeeded || h.reports[0].RebootScanned {
|
||||
t.Fatalf("hub got %+v", h.reports)
|
||||
}
|
||||
}
|
||||
@@ -432,7 +452,12 @@ func TestDocker_Ring0PlanAndReport(t *testing.T) {
|
||||
DockerEngine: "29.8.2", Authority: "ring0"}}}
|
||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
dp := w.plans[len(w.plans)-1]
|
||||
var dp map[string]any
|
||||
for _, x := range w.plans {
|
||||
if x["layer"] == "docker" {
|
||||
dp = x
|
||||
}
|
||||
}
|
||||
if dp["layer"] != "docker" || dp["lane"] != "slow" || dp["select"] != "pending-docker" {
|
||||
t.Fatalf("docker plan = %v", dp)
|
||||
}
|
||||
@@ -504,3 +529,42 @@ func TestHealthVerdict_ControllerBlindToDockerFails(t *testing.T) {
|
||||
t.Fatal("an older wrapper (no field) must not fail")
|
||||
}
|
||||
}
|
||||
|
||||
// R-528 (`09` decision 157): the wrapper's oom_check object reaches the hub's docker report byte-for-byte; the guest
|
||||
// and host reports carry none. COMPANION RED-PROOF: drop `rep.OOMCheck = rawOrNil(wr.OOMCheck)` in runLayer → "no
|
||||
// oom_check in the docker report".
|
||||
const oomCheckWire = `{"detail":"the engine reported the memory kill: OOMKilled=true and the oom event","exit_code":137,"image":"gitea.dooplex.hu/admin/felhom-controller:0.300.0","oom_event":true,"oom_killed":true,"result":"pass"}`
|
||||
|
||||
func TestDocker_OOMCheckReachesTheHubUnchanged(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerDocker: {
|
||||
Upgraded: []Package{{Name: "docker-ce", Version: "5:29.8.2-1~debian.13~trixie"}},
|
||||
DockerEngine: "29.8.2", Authority: "ring0", OOMCheck: json.RawMessage(oomCheckWire)}}}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
l.Run(context.Background(), 9201, "night")
|
||||
found := false
|
||||
for _, b := range h.bodies {
|
||||
var m map[string]json.RawMessage
|
||||
if err := json.Unmarshal(b, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var layer string
|
||||
json.Unmarshal(m["layer"], &layer)
|
||||
oc, has := m["oom_check"]
|
||||
if layer != LayerDocker {
|
||||
if has {
|
||||
t.Fatalf("the %s report carries an oom_check: %s", layer, oc)
|
||||
}
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
if !has {
|
||||
t.Fatalf("no oom_check in the docker report: %s", b)
|
||||
}
|
||||
if string(oc) != oomCheckWire {
|
||||
t.Fatalf("oom_check changed on the way:\n got %s\nwant %s", oc, oomCheckWire)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("no docker report posted: %s", calls(w))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
)
|
||||
|
||||
// ---- the Proxmox package step (R-812 option A, `09` §3 decision 163, `11` §5.10) ----
|
||||
|
||||
// Ring 0: after a healthy host step the leg runs the pve layer — slow lane, select pending-pve — while holding the
|
||||
// /etc/pve write gate; the report carries only Proxmox userspace packages and pve-manager's version.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): call runLayer instead of runPVE in Run → "the /etc/pve write gate was not held".
|
||||
func TestPVE_Ring0PlanGateAndReport(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {
|
||||
Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}},
|
||||
Installed: []Package{{Name: "pve-manager", Version: "9.2.21", Origin: "Proxmox"},
|
||||
{Name: "proxmox-kernel-helper", Version: "9.0.4", Origin: "Proxmox"}, {Name: "libc6", Version: "u4", Origin: "Debian"}},
|
||||
Pending: []Pending{{Name: "qemu-server", From: "9.0.1", To: "9.0.9", Origin: []string{PVEOrigin}},
|
||||
{Name: "proxmox-kernel-7.0", From: "7.0.2", To: "7.0.14", Origin: []string{PVEOrigin}},
|
||||
{Name: "libc6", From: "u3", To: "u4", Origin: []string{"Debian"}}},
|
||||
PVEManager: "9.2.21", Authority: "ring0"}}}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
var pp map[string]any
|
||||
for _, x := range w.plans {
|
||||
if x["layer"] == LayerPVE {
|
||||
pp = x
|
||||
}
|
||||
}
|
||||
if pp == nil || pp["lane"] != "slow" || pp["select"] != "pending-pve" {
|
||||
t.Fatalf("pve plan = %v (calls %s)", pp, calls(w))
|
||||
}
|
||||
if !w.pveGateHeld {
|
||||
t.Fatal("the /etc/pve write gate was not held while the pve step ran")
|
||||
}
|
||||
if pvegate.Stepping() {
|
||||
t.Fatal("the gate must be released after the step")
|
||||
}
|
||||
r := p.PVE
|
||||
if r.Outcome != "applied" || !r.Healthy || r.PVEManager != "9.2.21" {
|
||||
t.Fatalf("pve report = %+v", r)
|
||||
}
|
||||
if len(r.Installed) != 1 || r.Installed[0].Name != "pve-manager" || len(r.Pending) != 1 || r.Pending[0].Name != "qemu-server" {
|
||||
t.Fatalf("the pve report must carry Proxmox userspace only: installed=%v pending=%v", r.Installed, r.Pending)
|
||||
}
|
||||
if h.reports[len(h.reports)-1].Layer != LayerPVE {
|
||||
t.Fatalf("the hub must get the pve report: %+v", h.reports)
|
||||
}
|
||||
}
|
||||
|
||||
// Ring 1 never takes a Proxmox step in the night leg.
|
||||
func TestPVE_Ring1NightLegNeverSteps(t *testing.T) {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
if p := l.Run(context.Background(), 9201, "night"); p.PVE.Layer != "" || strings.Contains(calls(w), "pve") {
|
||||
t.Fatalf("ring 1 took a pve step: %s", calls(w))
|
||||
}
|
||||
}
|
||||
|
||||
// No healthy host step (a BYO box, or an unhealthy host step) → no pve step.
|
||||
func TestPVE_SkippedWithoutAHealthyHostStep(t *testing.T) {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
l.Appliance = false
|
||||
if p := l.Run(context.Background(), 9201, "night"); p.PVE.Layer != "" || strings.Contains(calls(w), "pve") {
|
||||
t.Fatalf("a BYO box took a pve step: %s", calls(w))
|
||||
}
|
||||
w2 := &fakeWrapper{t: t}
|
||||
l2, _ := newLeg(t, w2, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
l2.Tunnel = fakeTunnel{"stopped"} // the host step reads unhealthy
|
||||
w2.applyRep = map[string]WrapperReport{LayerHost: {Upgraded: []Package{{Name: "libc6", Version: "u4"}}}}
|
||||
if p := l2.Run(context.Background(), 9201, "night"); p.PVE.Layer != "" || strings.Contains(calls(w2), "pve") {
|
||||
t.Fatalf("a pve step ran after an unhealthy host step: %s", calls(w2))
|
||||
}
|
||||
}
|
||||
|
||||
// A write in flight that never finishes makes the pve step give up (failed), never run without the gate.
|
||||
func TestPVE_GivesUpWhenAWriteDoesNotFinish(t *testing.T) {
|
||||
old := pveDrainWait
|
||||
pveDrainWait = 50_000_000 // 50 ms
|
||||
defer func() { pveDrainWait = old }()
|
||||
rel, _, _ := pvegate.Write(context.Background())
|
||||
defer rel()
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
if p.PVE.Outcome != "failed" || strings.Contains(calls(w), "pve") {
|
||||
t.Fatalf("the pve step must fail without a wrapper call: %+v calls=%s", p.PVE, calls(w))
|
||||
}
|
||||
}
|
||||
|
||||
// THE pve health rule. COMPANION RED-PROOF (observed): drop the container-id loop or the pve-manager check in
|
||||
// PVEHealthVerdict → the matching case below fails.
|
||||
func TestPVEHealthVerdict(t *testing.T) {
|
||||
before, after := hostOK(), hostOK()
|
||||
before.Guest.Containers["app"] = Container{State: "running", Health: "healthy", ID: "a1"}
|
||||
after.Guest.Containers["app"] = Container{State: "running", Health: "healthy", ID: "a1"}
|
||||
if ok, why := PVEHealthVerdict(before, after, hub.TunnelRunning, "9.2.21", "9.2.21"); !ok {
|
||||
t.Fatalf("healthy step read unhealthy: %s", why)
|
||||
}
|
||||
if ok, _ := PVEHealthVerdict(before, after, hub.TunnelRunning, "9.2.21", "9.2.2"); ok {
|
||||
t.Fatal("pveversion still on the old pve-manager must fail")
|
||||
}
|
||||
after.Guest.Containers["app"] = Container{State: "running", Health: "healthy", ID: "b2"}
|
||||
if ok, why := PVEHealthVerdict(before, after, hub.TunnelRunning, "", "9.2.2"); ok || !strings.Contains(why, "id changed") {
|
||||
t.Fatalf("an app restarted by the Proxmox step must fail, got ok=%v %q", ok, why)
|
||||
}
|
||||
}
|
||||
|
||||
// The signed executor hands the RAW envelope and the exact list to the wrapper's pve layer.
|
||||
func TestPVEStepExecutor_PassesTheSignedEnvelope(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {
|
||||
Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}}, PVEManager: "9.2.21", Authority: "signed"}}}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
e := PVEStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
|
||||
params, _ := json.Marshal(PVEStepParams{ReleaseID: "os-pve-1", Packages: []Package{{Name: "pve-manager", Version: "9.2.21", Origin: PVEOrigin}}})
|
||||
ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_pve_step"}`), Sig: []byte("SIG")})
|
||||
if err := e.Execute(ctx, OpPVEStep, params); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pp := w.plans[len(w.plans)-1]
|
||||
sg, _ := pp["signed"].(map[string]any)
|
||||
if pp["layer"] != LayerPVE || pp["lane"] != "slow" || pp["release_id"] != "os-pve-1" || sg == nil ||
|
||||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_pve_step"}`)) || sg["sig"] != "SIG" {
|
||||
t.Fatalf("pve plan = %v", pp)
|
||||
}
|
||||
if !w.pveGateHeld || calls(w) != "pve:apply" || len(h.reports) != 1 || h.reports[0].Trigger != "signed" {
|
||||
t.Fatalf("gate=%v calls=%s reports=%+v", w.pveGateHeld, calls(w), h.reports)
|
||||
}
|
||||
if err := e.Execute(context.Background(), OpPVEStep, params); err == nil {
|
||||
t.Fatal("no envelope must refuse")
|
||||
}
|
||||
if err := e.Execute(context.Background(), OpDockerStep, params); err != signedjobs.ErrNoExecutor {
|
||||
t.Fatalf("another op must pass through the chain: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// os_pve_step is never benign.
|
||||
func TestPVEStep_IsDestructiveClass(t *testing.T) {
|
||||
if reconcile.Classify(reconcile.ClassOSPVEStep, reconcile.Provenance{}) != reconcile.Destructive {
|
||||
t.Fatal("os_pve_step must be destructive-class (signed, operational key)")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
)
|
||||
|
||||
// OpPVEStep is the signed op class of a Proxmox package step (R-812 option A, `11` §5.10): a ring-1 box takes an
|
||||
// approved Proxmox set only through it. No undo in this release. CC may sign it until the first paying customer.
|
||||
const OpPVEStep = "os_pve_step"
|
||||
|
||||
// PVEStepParams are the signed params. The wrapper compares Packages with the plan byte-for-byte.
|
||||
type PVEStepParams struct {
|
||||
ReleaseID string `json:"release_id"`
|
||||
Packages []Package `json:"packages"`
|
||||
VMID int `json:"vmid,omitempty"`
|
||||
}
|
||||
|
||||
// PVEStepExecutor runs a verified os_pve_step (signedjobs.Executor) under the host-wide heavy-op gate (Gate) and the
|
||||
// /etc/pve write gate (inside runPVE).
|
||||
type PVEStepExecutor struct {
|
||||
Leg *Leg
|
||||
Guest func(ctx context.Context) (int, error)
|
||||
Gate func(ctx context.Context) (release func(), err error)
|
||||
}
|
||||
|
||||
// Execute implements signedjobs.Executor.
|
||||
func (e PVEStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
|
||||
if op != OpPVEStep {
|
||||
return signedjobs.ErrNoExecutor
|
||||
}
|
||||
so, ok := signedjobs.SignedOpFrom(ctx)
|
||||
if !ok {
|
||||
return fmt.Errorf("os_pve_step: no signed envelope in the context — the wrapper could not verify it")
|
||||
}
|
||||
var p PVEStepParams
|
||||
if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 {
|
||||
return fmt.Errorf("os_pve_step: params must name the Proxmox set: %v", err)
|
||||
}
|
||||
vmid := p.VMID
|
||||
if vmid == 0 {
|
||||
if e.Guest == nil {
|
||||
return fmt.Errorf("os_pve_step: no vmid and no guest finder")
|
||||
}
|
||||
v, err := e.Guest(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("os_pve_step: find the customer guest: %w", err)
|
||||
}
|
||||
vmid = v
|
||||
}
|
||||
if e.Gate != nil {
|
||||
release, err := e.Gate(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("os_pve_step: heavy-op gate busy (a backup or restore-test runs): %w", err)
|
||||
}
|
||||
defer release()
|
||||
}
|
||||
rep := e.Leg.RunPVESigned(ctx, vmid, p, so.Blob, string(so.Sig))
|
||||
switch rep.Outcome {
|
||||
case "applied", "nothing":
|
||||
if rep.Healthy {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("os_pve_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused))
|
||||
}
|
||||
|
||||
// RunPVESigned is one signed Proxmox step (ring 1): the pve layer with the signed envelope, which the wrapper verifies
|
||||
// itself, holding the /etc/pve write gate.
|
||||
func (l *Leg) RunPVESigned(ctx context.Context, vmid int, p PVEStepParams, blob []byte, sig string) Report {
|
||||
unlock := l.lockPass(true)
|
||||
defer unlock()
|
||||
l.sendUnsentLocked(ctx) // R-868
|
||||
runID := l.now().UTC().Format("20060102T150405Z")
|
||||
rid := p.ReleaseID
|
||||
if rid == "" {
|
||||
rid = "signed-" + runID
|
||||
}
|
||||
return l.runPVE(ctx, runID, vmid, "signed", l.Block(), dockerOpts{releaseID: rid, packages: p.Packages,
|
||||
signed: map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}})
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// After a boot the agent has not fetched the hub's block yet; the after-boot kernel report must still carry the
|
||||
// box's real ring (seen 2026-10-08: a ring-0 box's „judging" report said ring 1). Red-proof: return Block().Ring
|
||||
// from kernelReportRing and the saved-block case fails with 1.
|
||||
func TestKernelReportRing_BeforeFirstFetch(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
l := &Leg{PlanDir: dir}
|
||||
if got := l.kernelReportRing(); got != 1 {
|
||||
t.Fatalf("nothing fetched, nothing saved: ring %d, want 1 (the old default)", got)
|
||||
}
|
||||
l.saveBlock(&hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
l2 := &Leg{PlanDir: dir} // a fresh daemon after the reboot: no block fetched yet
|
||||
if got := l2.kernelReportRing(); got != 0 {
|
||||
t.Fatalf("ring-0 block saved before the reboot: ring %d, want 0", got)
|
||||
}
|
||||
l2.SetBlock(&hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
if got := l2.kernelReportRing(); got != 1 {
|
||||
t.Fatalf("a fetched block wins over the saved one: ring %d, want 1", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, SavedBlockFile)); err != nil {
|
||||
t.Fatalf("saved block missing: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -142,23 +142,42 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
|
||||
default:
|
||||
rep.Outcome = "applied"
|
||||
}
|
||||
if wr.Layer == LayerKernel {
|
||||
// a kernel STAGE changes nothing the box runs (the new kernel only boots once, at the night's reboot), so its
|
||||
// kept copy needs no fresh health reading (R-836)
|
||||
rep.Kernel = rawOrNil(wr.Kernel)
|
||||
rep.Upgraded, rep.PassSeconds, rep.Authority = wr.Upgraded, wr.PassSeconds, wr.Authority
|
||||
if rep.Outcome == "applied" {
|
||||
rep.Outcome = "staged"
|
||||
}
|
||||
rep.Healthy, rep.HealthReason = rep.Outcome == "staged" || rep.Outcome == "nothing", prefix
|
||||
return rep
|
||||
}
|
||||
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
|
||||
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
|
||||
wantEngine := ""
|
||||
rep.OOMCheck = rawOrNil(wr.OOMCheck)
|
||||
rep.PVEManager = wr.PVEManager
|
||||
wantEngine, wantPVE := "", ""
|
||||
for _, u := range wr.Upgraded {
|
||||
if u.Name == "docker-ce" {
|
||||
wantEngine = EngineOf(u.Version)
|
||||
}
|
||||
if u.Name == "pve-manager" {
|
||||
wantPVE = u.Version
|
||||
}
|
||||
}
|
||||
verdict := func(h *Health) (bool, string) {
|
||||
switch wr.Layer {
|
||||
case LayerDocker:
|
||||
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
|
||||
case LayerHost:
|
||||
case LayerHost, LayerPVE:
|
||||
t := hub.TunnelUnknown
|
||||
if l.Tunnel != nil {
|
||||
t, _ = l.Tunnel.Status(ctx)
|
||||
}
|
||||
if wr.Layer == LayerPVE {
|
||||
return PVEHealthVerdict(wr.HealthBefore, h, t, wantPVE, wr.PVEManager)
|
||||
}
|
||||
return HostHealthVerdict(wr.HealthBefore, h, t)
|
||||
}
|
||||
return HealthVerdict(wr.HealthBefore, h)
|
||||
@@ -166,7 +185,7 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
|
||||
ok, why := verdict(wr.HealthAfter)
|
||||
if !ok && len(wr.Upgraded) > 0 && wr.VMID > 0 {
|
||||
lane := "fast"
|
||||
if wr.Layer == LayerDocker {
|
||||
if wr.Layer == LayerDocker || wr.Layer == LayerPVE {
|
||||
lane = "slow"
|
||||
}
|
||||
if hr, err := l.call(ctx, "kept-"+runID, map[string]any{"release_id": "kept", "layer": wr.Layer, "lane": lane,
|
||||
@@ -186,6 +205,8 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
|
||||
rep.RebootScanned = wr.RebootScanned
|
||||
if wr.Layer == LayerDocker {
|
||||
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
|
||||
} else if wr.Layer == LayerPVE {
|
||||
rep.Installed, rep.Pending = onlyPVE(wr.Installed), onlyPVEPending(wr.Pending)
|
||||
} else {
|
||||
planned := map[string]bool{}
|
||||
for _, u := range wr.Upgraded {
|
||||
|
||||
@@ -150,3 +150,24 @@ func must(t *testing.T, err error) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// R-528: a kept docker report (the agent was killed) still carries the oom_check object to the hub unchanged.
|
||||
// COMPANION RED-PROOF: drop `rep.OOMCheck = rawOrNil(wr.OOMCheck)` in reportFromKept → "oom_check lost".
|
||||
func TestR868_KeptCopyCarriesTheOOMCheck(t *testing.T) {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
ring := 0
|
||||
kept := WrapperReport{Mode: "apply", Layer: LayerDocker, RunID: "20261007T020000Z", Trigger: "night", Ring: &ring, VMID: 9201,
|
||||
ReleaseID: "ring0-20261007T020000Z", HealthBefore: guestOK(), HealthAfter: guestOK(), DockerEngine: "29.8.2",
|
||||
Upgraded: []Package{{Name: "docker-ce", Version: "5:29.8.2-1~debian.13~trixie"}}, OOMCheck: json.RawMessage(oomCheckWire)}
|
||||
b, _ := json.Marshal(kept)
|
||||
must(t, os.WriteFile(reportFile(l.PlanDir, kept.RunID, LayerDocker, "apply"), b, 0o600))
|
||||
if n := l.SendUnsent(context.Background()); n != 1 || len(h.bodies) != 1 {
|
||||
t.Fatalf("sent %d, bodies %d", n, len(h.bodies))
|
||||
}
|
||||
var m map[string]json.RawMessage
|
||||
must(t, json.Unmarshal(h.bodies[0], &m))
|
||||
if string(m["oom_check"]) != oomCheckWire {
|
||||
t.Fatalf("oom_check lost or changed: %q", m["oom_check"])
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
// Package poweroff keeps the box's record of a household-requested power-off (R-998, doc 14).
|
||||
//
|
||||
// One small JSON file in the agent's state dir. A record belongs to the boot it was made in (the
|
||||
// kernel's btime from /proc/stat), so:
|
||||
// - "at most one accepted request per boot": Mark refuses when this boot already has a record;
|
||||
// - the host report carries the record only while the box is still in that boot (Current), and
|
||||
// the next boot's reports carry nothing — that absence is how the hub learns the box is back.
|
||||
//
|
||||
// Pinned by state_test.go.
|
||||
package poweroff
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrAlreadyThisBoot is Mark's refusal: a power-off was already accepted in the current boot.
|
||||
var ErrAlreadyThisBoot = errors.New("a power-off was already accepted in this boot")
|
||||
|
||||
// ErrBootUnknown is Mark's refusal when the boot cannot be identified — fail closed: without it the
|
||||
// per-boot limit could not hold.
|
||||
var ErrBootUnknown = errors.New("cannot read the host's boot time")
|
||||
|
||||
// Record is one accepted request.
|
||||
type Record struct {
|
||||
RequestedAt time.Time `json:"requested_at"`
|
||||
BootBtime string `json:"boot_btime"` // /proc/stat btime of the boot it was accepted in
|
||||
VMID int `json:"vmid"` // the guest whose controller asked
|
||||
}
|
||||
|
||||
// State is the persisted record.
|
||||
type State struct {
|
||||
path string
|
||||
btime func() string
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// New returns the state at path. btime returns the current boot's /proc/stat btime ("" = unknown).
|
||||
func New(path string, btime func() string) *State {
|
||||
if btime == nil {
|
||||
btime = ProcBtime
|
||||
}
|
||||
return &State{path: path, btime: btime}
|
||||
}
|
||||
|
||||
// Current is the record made in THIS boot, or nil (none, another boot's, unreadable).
|
||||
func (s *State) Current() *Record {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
r := s.load()
|
||||
b := s.btime()
|
||||
if r == nil || b == "" || r.BootBtime != b {
|
||||
return nil
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// Mark records an accepted request for this boot. It refuses a second one in the same boot.
|
||||
func (s *State) Mark(now time.Time, vmid int) (*Record, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
b := s.btime()
|
||||
if b == "" {
|
||||
return nil, ErrBootUnknown
|
||||
}
|
||||
if r := s.load(); r != nil && r.BootBtime == b {
|
||||
return nil, ErrAlreadyThisBoot
|
||||
}
|
||||
r := &Record{RequestedAt: now.UTC().Truncate(time.Second), BootBtime: b, VMID: vmid}
|
||||
data, _ := json.Marshal(r)
|
||||
if err := os.MkdirAll(filepath.Dir(s.path), 0o750); err != nil {
|
||||
return nil, fmt.Errorf("poweroff: state dir: %w", err)
|
||||
}
|
||||
tmp := s.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||
return nil, fmt.Errorf("poweroff: write: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmp, s.path); err != nil {
|
||||
return nil, fmt.Errorf("poweroff: rename: %w", err)
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (s *State) load() *Record {
|
||||
data, err := os.ReadFile(s.path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var r Record
|
||||
if json.Unmarshal(data, &r) != nil || r.BootBtime == "" {
|
||||
return nil
|
||||
}
|
||||
return &r
|
||||
}
|
||||
|
||||
// ProcBtime reads the boot time (seconds since the epoch, as text) from /proc/stat; "" when unreadable.
|
||||
func ProcBtime() string {
|
||||
data, err := os.ReadFile("/proc/stat")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
for _, line := range strings.Split(string(data), "\n") {
|
||||
if strings.HasPrefix(line, "btime ") {
|
||||
return strings.TrimSpace(strings.TrimPrefix(line, "btime "))
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package poweroff
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// One accepted request per boot; the record is "current" only in the boot it was made in.
|
||||
func TestOnePerBootAndCurrentFollowsTheBoot(t *testing.T) {
|
||||
boot := "1000"
|
||||
s := New(filepath.Join(t.TempDir(), "poweroff.json"), func() string { return boot })
|
||||
if s.Current() != nil {
|
||||
t.Fatal("no record yet, Current must be nil")
|
||||
}
|
||||
now := time.Date(2026, 10, 13, 18, 0, 0, 0, time.UTC)
|
||||
if _, err := s.Mark(now, 9201); err != nil {
|
||||
t.Fatalf("first Mark: %v", err)
|
||||
}
|
||||
if r := s.Current(); r == nil || !r.RequestedAt.Equal(now) || r.VMID != 9201 {
|
||||
t.Fatalf("Current after Mark = %+v", r)
|
||||
}
|
||||
if _, err := s.Mark(now.Add(time.Minute), 9201); !errors.Is(err, ErrAlreadyThisBoot) {
|
||||
t.Fatalf("second Mark in the same boot: err = %v, want ErrAlreadyThisBoot", err)
|
||||
}
|
||||
boot = "2000" // the machine was switched on again
|
||||
if s.Current() != nil {
|
||||
t.Fatal("a record from the previous boot must not be current")
|
||||
}
|
||||
if _, err := s.Mark(now.Add(time.Hour), 9201); err != nil {
|
||||
t.Fatalf("Mark in a new boot: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownBootFailsClosed(t *testing.T) {
|
||||
s := New(filepath.Join(t.TempDir(), "p.json"), func() string { return "" })
|
||||
if _, err := s.Mark(time.Now(), 1); !errors.Is(err, ErrBootUnknown) {
|
||||
t.Fatalf("err = %v, want ErrBootUnknown", err)
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
@@ -105,6 +106,15 @@ func (c *Client) do(ctx context.Context, method, path string, body io.Reader, ou
|
||||
// doBody is the single HTTP chokepoint: builds the request, sets auth, executes,
|
||||
// maps non-2xx to APIError, and decodes the data envelope.
|
||||
func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader, contentType string, out any) error {
|
||||
// R-812 option A: every non-GET call may write /etc/pve — it waits while a Proxmox package step restarts pmxcfs
|
||||
// (pvegate). Pinned by TestPVEGate_ClientWriteWaitsGetDoesNot.
|
||||
if method != http.MethodGet {
|
||||
release, _, gerr := pvegate.Write(ctx)
|
||||
if gerr != nil {
|
||||
return fmt.Errorf("proxmox: %s %s held back by a Proxmox package step: %w", method, path, gerr)
|
||||
}
|
||||
defer release()
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, c.base+path, body)
|
||||
if err != nil {
|
||||
return fmt.Errorf("proxmox: building request: %w", err)
|
||||
|
||||
@@ -4,8 +4,10 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||
"io"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
@@ -59,6 +61,15 @@ func (r *ExecRunner) Run(ctx context.Context, name string, args ...string) ([]by
|
||||
// RunStdin is Run with the process stdin fed from stdin (nil = no stdin). The sudo-prefix/mode
|
||||
// handling is identical to Run — kept here so both paths share one place.
|
||||
func (r *ExecRunner) RunStdin(ctx context.Context, stdin io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
// R-812 option A: a root CLI that writes /etc/pve waits while a Proxmox package step runs (pvegate).
|
||||
// Pinned by TestPVEGate_ExecRunnerPctSetWaits / TestWritesEtcPVE.
|
||||
if WritesEtcPVE(name, args) {
|
||||
release, _, gerr := pvegate.Write(ctx)
|
||||
if gerr != nil {
|
||||
return nil, nil, fmt.Errorf("proxmox: %s held back by a Proxmox package step: %w", name, gerr)
|
||||
}
|
||||
defer release()
|
||||
}
|
||||
var cmd *exec.Cmd
|
||||
if r.Mode == RunnerSudo {
|
||||
sudo := r.SudoPath
|
||||
@@ -77,6 +88,28 @@ func (r *ExecRunner) RunStdin(ctx context.Context, stdin io.Reader, name string,
|
||||
return stdout.b, stderr.b, err
|
||||
}
|
||||
|
||||
// WritesEtcPVE reports whether a root command writes /etc/pve: `pct` with a config-changing verb, `pvesm`, `pveum`,
|
||||
// and the PBS storage wrapper's create / reconcile verbs. `pct exec|status|list|config` and every other command do not
|
||||
// (the os-update wrapper itself must never wait on the gate its own step holds). Pinned by TestWritesEtcPVE.
|
||||
func WritesEtcPVE(name string, args []string) bool {
|
||||
base := filepath.Base(name)
|
||||
switch base {
|
||||
case "pvesm", "pveum":
|
||||
return true
|
||||
case "pct":
|
||||
if len(args) == 0 {
|
||||
return false
|
||||
}
|
||||
switch args[0] {
|
||||
case "set", "create", "destroy", "restore", "unlock", "resize", "snapshot", "delsnapshot", "rollback", "move-volume", "start", "stop", "reboot", "shutdown":
|
||||
return true
|
||||
}
|
||||
case "felhom-pbs-apply":
|
||||
return len(args) > 0 && (args[0] == "create" || args[0] == "reconcile")
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Privileged is the root-CLI backend.
|
||||
type Privileged struct {
|
||||
runner Runner
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||
)
|
||||
|
||||
// R-812 option A: while a Proxmox package step holds the gate, a non-GET API call waits and a GET does not.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): delete the pvegate.Write block in doBody → this fails with "a PUT reached the API
|
||||
// while the Proxmox step held the gate". Restored. (audits/day-2026-10-07/B/red-pvegate-chokepoints.txt)
|
||||
func TestPVEGate_ClientWriteWaitsGetDoesNot(t *testing.T) {
|
||||
d := &mockDoer{fn: func(*http.Request) (*http.Response, error) { return jsonResp(200, `{"data":null}`), nil }}
|
||||
c := newTestClient(d)
|
||||
end, err := pvegate.Step(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.get(context.Background(), "/nodes", nil); err != nil {
|
||||
t.Fatalf("a GET must not wait on the gate: %v", err)
|
||||
}
|
||||
if d.calls != 1 {
|
||||
t.Fatalf("the GET must reach the API, calls=%d", d.calls)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
err = c.postForm(ctx, http.MethodPut, "/nodes/x/lxc/9201/config", nil, nil)
|
||||
if d.calls != 1 {
|
||||
end()
|
||||
t.Fatal("a PUT reached the API while the Proxmox step held the gate")
|
||||
}
|
||||
if err == nil {
|
||||
end()
|
||||
t.Fatal("a PUT held back past its deadline must fail")
|
||||
}
|
||||
end()
|
||||
if err := c.postForm(context.Background(), http.MethodPut, "/nodes/x/lxc/9201/config", nil, nil); err != nil || d.calls != 2 {
|
||||
t.Fatalf("after the step the PUT must go through (err=%v calls=%d)", err, d.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// A root `pct set` waits on the gate; `pct exec` does not.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): delete the WritesEtcPVE block in RunStdin → this fails with "pct set ran while the
|
||||
// Proxmox step held the gate". Restored.
|
||||
func TestPVEGate_ExecRunnerPctSetWaits(t *testing.T) {
|
||||
r := &ExecRunner{Mode: RunnerDirect}
|
||||
end, err := pvegate.Step(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer end()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
_, _, err = r.Run(ctx, "/nonexistent/pct", "set", "9201", "-mp8", "/x")
|
||||
if err == nil || time.Since(start) < 90*time.Millisecond {
|
||||
t.Fatalf("pct set ran while the Proxmox step held the gate (err=%v after %s)", err, time.Since(start))
|
||||
}
|
||||
start = time.Now()
|
||||
_, _, _ = r.Run(context.Background(), "/nonexistent/pct", "exec", "9201", "--", "true")
|
||||
if time.Since(start) > 80*time.Millisecond {
|
||||
t.Fatal("pct exec must not wait on the gate")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWritesEtcPVE(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
args []string
|
||||
want bool
|
||||
}{
|
||||
{"pct", []string{"set", "9201", "-mp8", "x"}, true},
|
||||
{"/usr/sbin/pct", []string{"create", "9201"}, true},
|
||||
{"pct", []string{"exec", "9201", "--", "true"}, false},
|
||||
{"pct", []string{"status", "9201"}, false},
|
||||
{"pvesm", []string{"add", "dir", "x"}, true},
|
||||
{"pveum", []string{"acl", "modify"}, true},
|
||||
{"/usr/local/sbin/felhom-pbs-apply", []string{"reconcile"}, true},
|
||||
{"/usr/local/sbin/felhom-pbs-apply", []string{"read"}, false},
|
||||
{"/usr/local/sbin/felhom-os-apply", []string{"--plan", "x"}, false},
|
||||
{"pct", nil, false},
|
||||
} {
|
||||
if got := WritesEtcPVE(c.name, c.args); got != c.want {
|
||||
t.Errorf("WritesEtcPVE(%s %v) = %v, want %v", c.name, c.args, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
// Package pvegate keeps the agent's own writes to /etc/pve out of the way of a Proxmox package step (R-812 option A,
|
||||
// `09` §3 decision 163, `11` §5.10).
|
||||
//
|
||||
// WHY. A `pve` step upgrades pve-cluster / pve-manager / qemu-server / pve-container; their postinst scripts restart
|
||||
// pmxcfs (the FUSE filesystem behind /etc/pve) and the API daemons. A write that lands while pmxcfs restarts fails or,
|
||||
// worse, half-lands (design-R-812 §3 A, "can go wrong"). Backups and restore-tests are already kept out by the
|
||||
// host-wide heavy-op gate; this gate covers everything else the agent writes: every non-GET Proxmox API call
|
||||
// (proxmox.Client.doBody) and every root CLI that writes /etc/pve (proxmox.ExecRunner — `pct set|create|…`, `pvesm`,
|
||||
// `pveum`, `felhom-pbs-apply create|reconcile`).
|
||||
//
|
||||
// THE RULE. Write waits while a step runs (bounded by its own context). Step marks the step and then waits until every
|
||||
// write already in flight has finished; it never waits forever (its context bounds it, and the caller gives up and
|
||||
// does not run the step). One step at a time. Pinned by pvegate_test.go and, at the two chokepoints, by
|
||||
// proxmox TestPVEGate_*.
|
||||
package pvegate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
inFlight int
|
||||
stepping bool
|
||||
stepDone chan struct{}
|
||||
)
|
||||
|
||||
// ErrStepRunning is returned by Step when another step already holds the gate.
|
||||
var ErrStepRunning = errors.New("pvegate: a Proxmox package step is already running")
|
||||
|
||||
// Write marks one /etc/pve write in flight, first waiting while a Proxmox package step runs. The returned release must
|
||||
// be called when the write has finished. waited reports how long the write was held back.
|
||||
func Write(ctx context.Context) (release func(), waited time.Duration, err error) {
|
||||
start := time.Now()
|
||||
for {
|
||||
mu.Lock()
|
||||
if !stepping {
|
||||
inFlight++
|
||||
mu.Unlock()
|
||||
var once sync.Once
|
||||
return func() {
|
||||
once.Do(func() {
|
||||
mu.Lock()
|
||||
inFlight--
|
||||
mu.Unlock()
|
||||
})
|
||||
}, time.Since(start), nil
|
||||
}
|
||||
ch := stepDone
|
||||
mu.Unlock()
|
||||
select {
|
||||
case <-ch:
|
||||
case <-ctx.Done():
|
||||
return nil, time.Since(start), ctx.Err()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Step marks a Proxmox package step and waits until every /etc/pve write already in flight has finished. On error the
|
||||
// gate is released again and the step must not run. end releases the gate and lets the held-back writes go.
|
||||
func Step(ctx context.Context) (end func(), err error) {
|
||||
mu.Lock()
|
||||
if stepping {
|
||||
mu.Unlock()
|
||||
return nil, ErrStepRunning
|
||||
}
|
||||
stepping = true
|
||||
done := make(chan struct{})
|
||||
stepDone = done
|
||||
mu.Unlock()
|
||||
var once sync.Once
|
||||
end = func() {
|
||||
once.Do(func() {
|
||||
mu.Lock()
|
||||
stepping = false
|
||||
close(done)
|
||||
mu.Unlock()
|
||||
})
|
||||
}
|
||||
for {
|
||||
mu.Lock()
|
||||
n := inFlight
|
||||
mu.Unlock()
|
||||
if n == 0 {
|
||||
return end, nil
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
end()
|
||||
return nil, ctx.Err()
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Stepping reports whether a Proxmox package step holds the gate (for logs).
|
||||
func Stepping() bool {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return stepping
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package pvegate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A write that starts while a step runs waits until the step ends.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make Write ignore `stepping` → this fails with "the write went through while the
|
||||
// Proxmox step held the gate". Restored. (audits/day-2026-10-07/B/red-pvegate.txt)
|
||||
func TestWrite_WaitsWhileAStepRuns(t *testing.T) {
|
||||
end, err := Step(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := make(chan time.Time, 1)
|
||||
go func() {
|
||||
rel, _, err := Write(context.Background())
|
||||
if err == nil {
|
||||
rel()
|
||||
}
|
||||
got <- time.Now()
|
||||
}()
|
||||
select {
|
||||
case <-got:
|
||||
end()
|
||||
t.Fatal("the write went through while the Proxmox step held the gate")
|
||||
case <-time.After(150 * time.Millisecond):
|
||||
}
|
||||
ended := time.Now()
|
||||
end()
|
||||
select {
|
||||
case at := <-got:
|
||||
if at.Before(ended) {
|
||||
t.Fatal("the write finished before the step ended")
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("the write never went through after the step ended")
|
||||
}
|
||||
}
|
||||
|
||||
// A step waits for a write already in flight before it starts.
|
||||
func TestStep_WaitsForAWriteInFlight(t *testing.T) {
|
||||
rel, _, err := Write(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
started := make(chan struct{})
|
||||
go func() {
|
||||
end, err := Step(context.Background())
|
||||
if err == nil {
|
||||
close(started)
|
||||
end()
|
||||
}
|
||||
}()
|
||||
select {
|
||||
case <-started:
|
||||
rel()
|
||||
t.Fatal("the step started while a write was in flight")
|
||||
case <-time.After(150 * time.Millisecond):
|
||||
}
|
||||
rel()
|
||||
select {
|
||||
case <-started:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("the step never started after the write finished")
|
||||
}
|
||||
}
|
||||
|
||||
// A step that cannot drain the writes in time gives up and releases the gate (it never waits forever).
|
||||
func TestStep_GivesUpAndReleases(t *testing.T) {
|
||||
rel, _, _ := Write(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
if _, err := Step(ctx); err == nil {
|
||||
t.Fatal("the step must give up while a write is in flight past its deadline")
|
||||
}
|
||||
if Stepping() {
|
||||
t.Fatal("a step that gave up must release the gate")
|
||||
}
|
||||
rel()
|
||||
}
|
||||
|
||||
// A write held back past its own deadline returns the context's error.
|
||||
func TestWrite_HonoursItsContext(t *testing.T) {
|
||||
end, _ := Step(context.Background())
|
||||
defer end()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
if _, _, err := Write(ctx); err == nil {
|
||||
t.Fatal("a write held back past its deadline must fail")
|
||||
}
|
||||
}
|
||||
|
||||
// One step at a time.
|
||||
func TestStep_OneAtATime(t *testing.T) {
|
||||
end, _ := Step(context.Background())
|
||||
defer end()
|
||||
if _, err := Step(context.Background()); err != ErrStepRunning {
|
||||
t.Fatalf("a second step must be refused, got %v", err)
|
||||
}
|
||||
}
|
||||
@@ -52,9 +52,23 @@ const (
|
||||
// (signed, operational key) like agent_update; the root wrapper re-verifies the same signature itself.
|
||||
ClassOSDockerStep OpClass = "os_docker_step"
|
||||
|
||||
// A Proxmox package step on the host (R-812 option A, `11` §5.10) — ring 1. Destructive-class (signed, operational
|
||||
// key) like os_docker_step; the root wrapper re-verifies the same signature itself.
|
||||
ClassOSPVEStep OpClass = "os_pve_step"
|
||||
|
||||
// A kernel step on the host (R-836, `09` §3 decision 172, `11` §5.11) — ring 1: it STAGES a kernel (install + the
|
||||
// one-shot flag; the night leg reboots it). Destructive-class (signed, operational key) like os_pve_step; the root
|
||||
// wrapper re-verifies the same signature itself.
|
||||
ClassOSKernelStep OpClass = "os_kernel_step"
|
||||
|
||||
// The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units).
|
||||
// Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself.
|
||||
ClassAgentConfigUpdate OpClass = "agent_config_update"
|
||||
|
||||
// The internal data place on an OLD one-disk box (agent v0.158.0, felhom.eu `16` §9): thin space on the system
|
||||
// disk, made by the root helper felhom-internal-place. Destructive-class (signed, operational key): it is an
|
||||
// operator act, never a household button, and no agent-internal provenance may start it.
|
||||
ClassAddInternalPlace OpClass = "add_internal_place"
|
||||
)
|
||||
|
||||
// Disposition is the classifier verdict.
|
||||
@@ -123,7 +137,7 @@ func Classify(class OpClass, prov Provenance) Disposition {
|
||||
return Destructive
|
||||
case ClassKeyRotation:
|
||||
return Destructive
|
||||
case ClassAgentUpdate, ClassOSDockerStep, ClassAgentConfigUpdate:
|
||||
case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassOSKernelStep, ClassAgentConfigUpdate, ClassAddInternalPlace:
|
||||
// Never benign — no agent-internal provenance can make replacing the agent binary
|
||||
// unsigned-safe (a compromised process must not be able to self-bless an update).
|
||||
return Destructive
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
package signedjobs
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// opAddInternalPlace is the op class this executor serves (mirrors reconcile.ClassAddInternalPlace).
|
||||
const opAddInternalPlace = "add_internal_place"
|
||||
|
||||
// InternalPlaceHelper is the root helper's path (configs/felhom-internal-place, from the config bundle).
|
||||
const InternalPlaceHelper = "/usr/local/sbin/felhom-internal-place"
|
||||
|
||||
var reHelperResult = regexp.MustCompile(`^(created|existing|formatted) uuid=([0-9a-fA-F-]{36}) gib=([0-9]+)$`)
|
||||
|
||||
// InternalPlaceExecutor adds the internal data place to an OLD one-disk box (felhom.eu architecture/16 §9: „one-disk
|
||||
// boxes get no internal place by themselves; adding one later is a signed operator job, not a household button").
|
||||
// It runs the SAME root helper the installer runs — `felhom-internal-place create 0` (0 GiB planned: the guest's
|
||||
// volumes already exist and are counted) — writes the record the agent mounts from, and asks the local API to mount
|
||||
// it now. Every safety rule (the ruled size, never format what holds anything) lives in the helper, as root.
|
||||
//
|
||||
// The op carries no parameters: the size is the rule's, never the signer's. Destructive-class by default (the
|
||||
// classifier's fail-safe; ClassAddInternalPlace documents it): it takes thin space on the system disk, so only an
|
||||
// operator signature may start it.
|
||||
type InternalPlaceExecutor struct {
|
||||
runner proxmox.Runner
|
||||
recordFile string
|
||||
ensure func(context.Context) // late-bound: the local API server is built after the jobs runner
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
// NewInternalPlaceExecutor wires the executor. ensure may be nil at construction and set later through SetEnsure.
|
||||
func NewInternalPlaceExecutor(runner proxmox.Runner, recordFile string, logger *slog.Logger) *InternalPlaceExecutor {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
return &InternalPlaceExecutor{runner: runner, recordFile: recordFile, logger: logger}
|
||||
}
|
||||
|
||||
// SetEnsure late-binds the mount step (localapi.Server.EnsureInternalPlace).
|
||||
func (e *InternalPlaceExecutor) SetEnsure(f func(context.Context)) { e.ensure = f }
|
||||
|
||||
// Execute implements signedjobs.Executor for add_internal_place.
|
||||
func (e *InternalPlaceExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
|
||||
if op != opAddInternalPlace {
|
||||
return ErrNoExecutor
|
||||
}
|
||||
if p := bytes.TrimSpace(params); len(p) > 0 && !bytes.Equal(p, []byte("{}")) && !bytes.Equal(p, []byte("null")) {
|
||||
return fmt.Errorf("add_internal_place: takes no parameters (the size is the ruled rule's), got %s", p)
|
||||
}
|
||||
if e.runner == nil || e.recordFile == "" {
|
||||
return fmt.Errorf("add_internal_place: not wired on this host (no privileged runner or state dir)")
|
||||
}
|
||||
start := time.Now()
|
||||
e.logger.Warn("add_internal_place: executing operator-signed job (the internal data place, `16` §9)")
|
||||
out, stderr, err := e.runner.Run(ctx, InternalPlaceHelper, "create", "0")
|
||||
line := strings.TrimSpace(string(out))
|
||||
if err != nil {
|
||||
return fmt.Errorf("add_internal_place: helper refused: %v: %s", err, strings.TrimSpace(string(stderr)))
|
||||
}
|
||||
if strings.HasPrefix(line, "skipped ") {
|
||||
// Not an error the operator can retry away: the pool has no room by the ruled rule. Fail the job with the
|
||||
// helper's own numbers so the hub shows why.
|
||||
return fmt.Errorf("add_internal_place: %s", line)
|
||||
}
|
||||
m := reHelperResult.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
return fmt.Errorf("add_internal_place: unexpected helper output %q", line)
|
||||
}
|
||||
gib, _ := strconv.Atoi(m[3])
|
||||
rec := localapi.InternalPlaceRecord{FSUUID: m[2], GiB: gib, MadeBy: "signed-job", MadeAt: time.Now().UTC().Format(time.RFC3339)}
|
||||
if err := localapi.WriteInternalPlace(e.recordFile, rec); err != nil {
|
||||
return fmt.Errorf("add_internal_place: writing the record: %w", err)
|
||||
}
|
||||
if e.ensure != nil {
|
||||
e.ensure(ctx)
|
||||
}
|
||||
e.logger.Warn("add_internal_place: complete", "result", m[1], "fs_uuid", m[2], "gib", gib,
|
||||
"duration_ms", time.Since(start).Milliseconds())
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
package signedjobs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// add_internal_place (felhom.eu architecture/16 §9, agent v0.158.0): the operator's job for an OLD one-disk box.
|
||||
|
||||
type ipRunner struct {
|
||||
out, stderr string
|
||||
err error
|
||||
calls [][]string
|
||||
}
|
||||
|
||||
func (r *ipRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
return []byte(r.out), []byte(r.stderr), r.err
|
||||
}
|
||||
func (r *ipRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
func ipExec(t *testing.T, r *ipRunner) (*InternalPlaceExecutor, string, *int) {
|
||||
t.Helper()
|
||||
file := filepath.Join(t.TempDir(), localapi.InternalPlaceFile)
|
||||
e := NewInternalPlaceExecutor(r, file, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
n := 0
|
||||
e.SetEnsure(func(context.Context) { n++ })
|
||||
return e, file, &n
|
||||
}
|
||||
|
||||
func TestAddInternalPlace_RunsTheHelperWritesTheRecordAndMounts(t *testing.T) {
|
||||
r := &ipRunner{out: "created uuid=0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c gib=33\n"}
|
||||
e, file, ensured := ipExec(t, r)
|
||||
if err := e.Execute(context.Background(), "add_internal_place", []byte(`{}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(r.calls) != 1 || strings.Join(r.calls[0], " ") != InternalPlaceHelper+" create 0" {
|
||||
t.Fatalf("the helper must be called exactly as `create 0`, got %v", r.calls)
|
||||
}
|
||||
rec, ok, err := localapi.ReadInternalPlace(file)
|
||||
if err != nil || !ok || rec.FSUUID != "0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c" || rec.GiB != 33 || rec.MadeBy != "signed-job" {
|
||||
t.Fatalf("record: %+v ok=%v err=%v", rec, ok, err)
|
||||
}
|
||||
if *ensured != 1 {
|
||||
t.Fatalf("the mount must be asked for once, got %d", *ensured)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddInternalPlace_NoRoomOrRefusalOrGarbage_WritesNothing(t *testing.T) {
|
||||
for name, r := range map[string]*ipRunner{
|
||||
"skipped": {out: "skipped only 7 GiB free for it (pool 53, reserve 16, promised 102, planned 0; needs 20)"},
|
||||
"refused": {err: errors.New("exit status 1"), stderr: "felhom-internal-place: REFUSED: holds something"},
|
||||
"garbage": {out: "created uuid=../../etc gib=1"},
|
||||
} {
|
||||
e, file, ensured := ipExec(t, r)
|
||||
if err := e.Execute(context.Background(), "add_internal_place", nil); err == nil {
|
||||
t.Fatalf("%s: want an error", name)
|
||||
}
|
||||
if _, statErr := os.Stat(file); !os.IsNotExist(statErr) || *ensured != 0 {
|
||||
t.Fatalf("%s: no record and no mount may follow, stat=%v ensured=%d", name, statErr, *ensured)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddInternalPlace_TakesNoParametersAndIgnoresOtherOps(t *testing.T) {
|
||||
r := &ipRunner{out: "created uuid=0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c gib=33"}
|
||||
e, _, _ := ipExec(t, r)
|
||||
if err := e.Execute(context.Background(), "add_internal_place", []byte(`{"gib":900}`)); err == nil || len(r.calls) != 0 {
|
||||
t.Fatalf("a size from the signer must be refused before the helper runs, err=%v calls=%v", err, r.calls)
|
||||
}
|
||||
if err := e.Execute(context.Background(), "decommission", []byte(`{}`)); !errors.Is(err, ErrNoExecutor) {
|
||||
t.Fatalf("another op: want ErrNoExecutor, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddInternalPlace_IsSignatureOnly(t *testing.T) {
|
||||
if got := reconcile.Classify(reconcile.ClassAddInternalPlace, reconcile.Provenance{}); got != reconcile.Destructive {
|
||||
t.Fatalf("add_internal_place must need an operator signature, classified %v", got)
|
||||
}
|
||||
if string(reconcile.ClassAddInternalPlace) != opAddInternalPlace {
|
||||
t.Fatalf("op string drift: %q vs %q", reconcile.ClassAddInternalPlace, opAddInternalPlace)
|
||||
}
|
||||
}
|
||||
@@ -13,14 +13,16 @@ import (
|
||||
// (guarded, Impl-1) and/or mounts + binds. Blank disks are "initialize"-only; disks already carrying a
|
||||
// mountable FS are ALSO "attach" candidates.
|
||||
type CandidateDisk struct {
|
||||
Device string `json:"device"` // whole disk, e.g. /dev/sdd
|
||||
SizeBytes int64 `json:"size_bytes"` // 0 when unreadable
|
||||
Device string `json:"device"` // whole disk, e.g. /dev/sdd
|
||||
SizeBytes int64 `json:"size_bytes"` // 0 when unreadable
|
||||
Model string `json:"model,omitempty"`
|
||||
FSType string `json:"fstype,omitempty"` // first filesystem found ("" = blank)
|
||||
DataBearing bool `json:"data_bearing"` // has any FS / partition (→ wipe-confirm downstream)
|
||||
Mountable bool `json:"mountable"` // carries an ext4/xfs FS the agent can mount as-is
|
||||
FSType string `json:"fstype,omitempty"` // first filesystem found ("" = blank)
|
||||
DataBearing bool `json:"data_bearing"` // has any FS / partition (→ wipe-confirm downstream)
|
||||
Mountable bool `json:"mountable"` // carries an ext4/xfs FS the agent can mount as-is
|
||||
MountSource string `json:"mount_source,omitempty"` // the node to mount for attach (e.g. /dev/sdd1)
|
||||
DurableID string `json:"durable_id,omitempty"` // "uuid:<fs-uuid>" if it has an FS; "" for a blank disk
|
||||
// Partitions (v0.158.0, felhom.eu `16` §6): each partition of this disk as its own place — see partitions.go.
|
||||
Partitions []CandidatePartition `json:"partitions,omitempty"`
|
||||
}
|
||||
|
||||
// mountableFSTypes are the filesystems the agent can mount as-is (EnsureMount) → an "attach" candidate.
|
||||
@@ -83,6 +85,8 @@ func (h *SudoHostOps) buildCandidate(dev, name string) CandidateDisk {
|
||||
if len(nodes) > 1 {
|
||||
c.DataBearing = true
|
||||
}
|
||||
c.Partitions = candidatePartitions(lout, name, readSysPartSize, readSysPartNumber, h.host.ResolveUUID,
|
||||
felhomOwnedMounts(dev, nodes, h.mountTable))
|
||||
}
|
||||
}
|
||||
return c
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
@@ -24,8 +25,13 @@ var devDiskRoot = "/dev/disk"
|
||||
const (
|
||||
durableByID = "byid:"
|
||||
durableByUUID = "byuuid:"
|
||||
// partFSUUIDSep joins a partition's by-id link and its fs-UUID in one wipe id (v0.158.0, `16` §6).
|
||||
partFSUUIDSep = "+fsuuid:"
|
||||
)
|
||||
|
||||
// rePartLink matches a by-id link of a PARTITION (udev appends -part<N> to the disk's link).
|
||||
var rePartLink = regexp.MustCompile(`-part[1-9][0-9]{0,2}$`)
|
||||
|
||||
// byIDPriority ranks /dev/disk/by-id link prefixes most-stable-first: wwn (hardware world-wide
|
||||
// name) > nvme-eui (NVMe EUI) > nvme-/ata-/scsi-/usb- (model+serial). dm-/lvm-/md- names are
|
||||
// excluded (they are mapper constructs, not the physical disk we want to bind a wipe to).
|
||||
@@ -41,6 +47,14 @@ func DeviceDurableID(device string) (string, error) {
|
||||
}
|
||||
// 1. best by-id link pointing at this device.
|
||||
if name := bestByIDLink(target); name != "" {
|
||||
// v0.158.0 (felhom.eu `16` §6): a PARTITION's link already carries the disk id + the part number
|
||||
// (wwn-…-part2). When the part holds a filesystem, its fs-UUID is bound too, so a part that was reformatted —
|
||||
// or another disk's part2 that took the link — derives a different id and the wipe refuses.
|
||||
if rePartLink.MatchString(name) {
|
||||
if uuid := byUUIDFor(target); uuid != "" {
|
||||
return durableByID + name + partFSUUIDSep + uuid, nil
|
||||
}
|
||||
}
|
||||
return durableByID + name, nil
|
||||
}
|
||||
// 2. fallback: a filesystem UUID symlink.
|
||||
@@ -56,10 +70,29 @@ func ResolveDurableDevice(durableID string) (string, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(durableID, durableByID):
|
||||
name := strings.TrimPrefix(durableID, durableByID)
|
||||
fsUUID := ""
|
||||
if i := strings.Index(name, partFSUUIDSep); i >= 0 {
|
||||
name, fsUUID = name[:i], name[i+len(partFSUUIDSep):]
|
||||
if !rePartLink.MatchString(name) || !safeLinkName(fsUUID) {
|
||||
return "", fmt.Errorf("storage: unsafe durable id %q", durableID)
|
||||
}
|
||||
}
|
||||
if !safeLinkName(name) {
|
||||
return "", fmt.Errorf("storage: unsafe durable id %q", durableID)
|
||||
}
|
||||
return filepath.EvalSymlinks(filepath.Join(devDiskRoot, "by-id", name))
|
||||
dev, err := filepath.EvalSymlinks(filepath.Join(devDiskRoot, "by-id", name))
|
||||
if err != nil || fsUUID == "" {
|
||||
return dev, err
|
||||
}
|
||||
// Both halves must name the SAME device now (`16` §6): the part by its disk id + number, and its filesystem.
|
||||
byFS, ferr := filepath.EvalSymlinks(filepath.Join(devDiskRoot, "by-uuid", fsUUID))
|
||||
if ferr != nil {
|
||||
return "", fmt.Errorf("storage: durable id %q: its filesystem %s no longer resolves: %w", durableID, fsUUID, ferr)
|
||||
}
|
||||
if byFS != dev {
|
||||
return "", fmt.Errorf("storage: durable id %q: the part (%s) and its filesystem (%s) are different devices now", durableID, dev, byFS)
|
||||
}
|
||||
return dev, nil
|
||||
case strings.HasPrefix(durableID, durableByUUID):
|
||||
uuid := strings.TrimPrefix(durableID, durableByUUID)
|
||||
if !safeLinkName(uuid) {
|
||||
|
||||
@@ -0,0 +1,445 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Physical-disk identity of a storage place — felhom.eu architecture/16-storage-physical-identity.md §3.
|
||||
//
|
||||
// A storage place (a PVE storage, an enrolled drive, the guest's data volume, a network share) is
|
||||
// reduced to the PHYSICAL whole disks under it — partition → parent disk, device-mapper / LVM thin →
|
||||
// sysfs slaves (recursive), loop → its backing file → the filesystem holding that file — and each disk
|
||||
// is named by a key the agent trusts: `eui:` for NVMe, `serial:<bus>:<model>:<serial>` from udev,
|
||||
// `wwn:` only when no serial exists and the WWN carries a vendor (OUI ≠ 000000).
|
||||
//
|
||||
// The verdict is AGENT-COMPUTED evidence (`16` §10): it is read from /sys, /run/udev/data and
|
||||
// /proc/mounts only, never from a caller's claim, so a compromised controller cannot make two places
|
||||
// "different disks". `proven` means every disk under the place has a key the agent trusts; anything
|
||||
// else is `unknown` with ONE reason, chosen by precedence (vm > network > usb_bridge > weak_wwn >
|
||||
// no_serial). Nothing is cached: every call reads the current topology, so a disk swapped in the same
|
||||
// bay is re-judged on the next read (the "right before each Tier-2 run" rule is the caller calling).
|
||||
//
|
||||
// Pinned by identity_test.go (fixture /sys + udev trees, one per §3 case) — the spike that proved the
|
||||
// walk on real devices: felhom.eu documentation/audits/storage-identity-2026-10-15/.
|
||||
|
||||
const (
|
||||
IdentityProven = "proven"
|
||||
IdentityUnknown = "unknown"
|
||||
|
||||
ReasonVM = "vm"
|
||||
ReasonNetwork = "network"
|
||||
ReasonUSBBridge = "usb_bridge"
|
||||
ReasonWeakWWN = "weak_wwn"
|
||||
ReasonNoSerial = "no_serial"
|
||||
|
||||
PlaceKindSystem = "system"
|
||||
PlaceKindInternal = "internal"
|
||||
PlaceKindDisk = "disk"
|
||||
PlaceKindPartition = "partition"
|
||||
PlaceKindNetwork = "network"
|
||||
)
|
||||
|
||||
// reasonRank orders the unknown reasons: the first one a place hits, by this precedence, is reported.
|
||||
var reasonRank = map[string]int{ReasonVM: 0, ReasonNetwork: 1, ReasonUSBBridge: 2, ReasonWeakWWN: 3, ReasonNoSerial: 4}
|
||||
|
||||
// PlaceIdentity is what a storage place reports.
|
||||
type PlaceIdentity struct {
|
||||
PhysicalDisks []string // disk keys, sorted, deduplicated
|
||||
Identity string // proven | unknown | "" (not computed — e.g. the device is absent)
|
||||
Reason string // set only when Identity == unknown
|
||||
}
|
||||
|
||||
// Computed reports whether a verdict was reached at all.
|
||||
func (p PlaceIdentity) Computed() bool { return p.Identity != "" }
|
||||
|
||||
// IdentityHost is the read surface the resolver needs. SysfsIdentityHost is production; tests use the
|
||||
// same struct pointed at fixture trees.
|
||||
type IdentityHost interface {
|
||||
Mounts() ([]Mount, error)
|
||||
// KernelName maps a /dev path (symlinks allowed: /dev/mapper/x, /dev/pve/y) to its kernel block name.
|
||||
KernelName(dev string) (string, bool)
|
||||
// Parent returns the parent disk's kernel name when name is a partition.
|
||||
Parent(name string) (string, bool)
|
||||
// Slaves lists /sys/class/block/<name>/slaves.
|
||||
Slaves(name string) []string
|
||||
// LoopBackingFile returns a loop device's backing file.
|
||||
LoopBackingFile(name string) (string, bool)
|
||||
// UdevProps returns the udev E: properties of a block device.
|
||||
UdevProps(name string) map[string]string
|
||||
// Virt returns the VM technology ("none" on bare metal).
|
||||
Virt() string
|
||||
}
|
||||
|
||||
// SysfsIdentityHost reads the real host. Every path is a field so tests can point it at fixtures.
|
||||
type SysfsIdentityHost struct {
|
||||
SysClass string // default /sys/class/block
|
||||
UdevData string // default /run/udev/data
|
||||
ProcMounts string // default /proc/mounts
|
||||
DevResolve func(string) (string, error) // default filepath.EvalSymlinks
|
||||
VirtFn func() string // default `systemd-detect-virt --vm`, read once
|
||||
|
||||
virtOnce sync.Once
|
||||
virt string
|
||||
}
|
||||
|
||||
// NewSysfsIdentityHost returns the production reader.
|
||||
func NewSysfsIdentityHost() *SysfsIdentityHost { return &SysfsIdentityHost{} }
|
||||
|
||||
func (h *SysfsIdentityHost) sysClass() string {
|
||||
if h.SysClass != "" {
|
||||
return h.SysClass
|
||||
}
|
||||
return "/sys/class/block"
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Mounts() ([]Mount, error) {
|
||||
return (&ProcHostReader{ProcMounts: h.ProcMounts}).Mounts()
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) KernelName(dev string) (string, bool) {
|
||||
if !strings.HasPrefix(dev, "/dev/") {
|
||||
return "", false
|
||||
}
|
||||
resolve := h.DevResolve
|
||||
if resolve == nil {
|
||||
resolve = filepath.EvalSymlinks
|
||||
}
|
||||
real := dev
|
||||
if r, err := resolve(dev); err == nil {
|
||||
real = r
|
||||
}
|
||||
name := filepath.Base(real)
|
||||
if _, err := os.Stat(filepath.Join(h.sysClass(), name)); err != nil {
|
||||
return "", false
|
||||
}
|
||||
return name, true
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Parent(name string) (string, bool) {
|
||||
p := filepath.Join(h.sysClass(), name)
|
||||
if _, err := os.Stat(filepath.Join(p, "partition")); err != nil {
|
||||
return "", false
|
||||
}
|
||||
real, err := filepath.EvalSymlinks(p)
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
return filepath.Base(filepath.Dir(real)), true
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Slaves(name string) []string {
|
||||
entries, err := os.ReadDir(filepath.Join(h.sysClass(), name, "slaves"))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) LoopBackingFile(name string) (string, bool) {
|
||||
if !strings.HasPrefix(name, "loop") {
|
||||
return "", false
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(h.sysClass(), name, "loop", "backing_file"))
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
f := strings.TrimSpace(string(b))
|
||||
return f, f != ""
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) UdevProps(name string) map[string]string {
|
||||
out := map[string]string{}
|
||||
devB, err := os.ReadFile(filepath.Join(h.sysClass(), name, "dev"))
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
dir := h.UdevData
|
||||
if dir == "" {
|
||||
dir = "/run/udev/data"
|
||||
}
|
||||
f, err := os.Open(filepath.Join(dir, "b"+strings.TrimSpace(string(devB))))
|
||||
if err != nil {
|
||||
return out
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if !strings.HasPrefix(line, "E:") {
|
||||
continue
|
||||
}
|
||||
k, v, ok := strings.Cut(line[2:], "=")
|
||||
if ok {
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *SysfsIdentityHost) Virt() string {
|
||||
h.virtOnce.Do(func() {
|
||||
if h.VirtFn != nil {
|
||||
h.virt = h.VirtFn()
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
// --vm: a container is not a reason (the agent runs on the host); exit 1 + "none" on metal.
|
||||
out, _ := exec.CommandContext(ctx, "systemd-detect-virt", "--vm").Output()
|
||||
h.virt = strings.TrimSpace(string(out))
|
||||
if h.virt == "" {
|
||||
h.virt = "unknown"
|
||||
}
|
||||
})
|
||||
return h.virt
|
||||
}
|
||||
|
||||
// IdentityResolver turns devices and host paths into PlaceIdentity.
|
||||
type IdentityResolver struct {
|
||||
Host IdentityHost
|
||||
}
|
||||
|
||||
// NewIdentityResolver returns a resolver over the real host.
|
||||
func NewIdentityResolver() *IdentityResolver {
|
||||
return &IdentityResolver{Host: NewSysfsIdentityHost()}
|
||||
}
|
||||
|
||||
var networkFSTypes = map[string]bool{
|
||||
"nfs": true, "nfs4": true, "cifs": true, "smb3": true, "fuse.sshfs": true, "9p": true, "autofs": true, "ceph": true, "glusterfs": true,
|
||||
}
|
||||
|
||||
// NetworkPlace is the fixed verdict for a network share or a remote store.
|
||||
func NetworkPlace() PlaceIdentity {
|
||||
return PlaceIdentity{PhysicalDisks: []string{}, Identity: IdentityUnknown, Reason: ReasonNetwork}
|
||||
}
|
||||
|
||||
// MountFor returns the mount that holds path (longest mountpoint prefix; the LAST entry wins on an equal
|
||||
// mountpoint, which is the one stacked on top — an automount's real filesystem over its autofs trigger).
|
||||
func MountFor(mounts []Mount, path string) (Mount, bool) {
|
||||
p := filepath.Clean(path)
|
||||
var best Mount
|
||||
found := false
|
||||
for _, m := range mounts {
|
||||
mp := filepath.Clean(m.MountPoint)
|
||||
if p != mp && !strings.HasPrefix(p, strings.TrimSuffix(mp, "/")+"/") {
|
||||
continue
|
||||
}
|
||||
if !found || len(mp) >= len(filepath.Clean(best.MountPoint)) {
|
||||
best, found = m, true
|
||||
}
|
||||
}
|
||||
return best, found
|
||||
}
|
||||
|
||||
// HostPath resolves the filesystem holding a HOST path.
|
||||
func (r *IdentityResolver) HostPath(path string) PlaceIdentity {
|
||||
return r.hostPath(path, 0)
|
||||
}
|
||||
|
||||
func (r *IdentityResolver) hostPath(path string, depth int) PlaceIdentity {
|
||||
if r == nil || r.Host == nil || depth > 4 {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
mounts, err := r.Host.Mounts()
|
||||
if err != nil {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
m, ok := MountFor(mounts, path)
|
||||
if !ok {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
if networkFSTypes[m.FSType] || !strings.HasPrefix(m.Device, "/dev/") {
|
||||
return NetworkPlace()
|
||||
}
|
||||
return r.device(m.Device, depth)
|
||||
}
|
||||
|
||||
// Device resolves a block device (a /dev path, symlinks allowed).
|
||||
func (r *IdentityResolver) Device(dev string) PlaceIdentity {
|
||||
return r.device(dev, 0)
|
||||
}
|
||||
|
||||
func (r *IdentityResolver) device(dev string, depth int) PlaceIdentity {
|
||||
if r == nil || r.Host == nil || dev == "" {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
name, ok := r.Host.KernelName(dev)
|
||||
if !ok {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
keys := map[string]bool{}
|
||||
reason := ""
|
||||
if !r.walk(name, depth, map[string]bool{}, keys, &reason) {
|
||||
return PlaceIdentity{}
|
||||
}
|
||||
out := PlaceIdentity{PhysicalDisks: sortedKeys(keys), Identity: IdentityProven}
|
||||
if reason != "" {
|
||||
out.Identity, out.Reason = IdentityUnknown, reason
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// walk adds the keys of every physical disk under name; ok=false when a branch cannot be grounded.
|
||||
func (r *IdentityResolver) walk(name string, depth int, seen, keys map[string]bool, reason *string) bool {
|
||||
if seen[name] {
|
||||
return true // a diamond (a thin pool's tmeta and tdata on one partition): already counted
|
||||
}
|
||||
if len(seen) > 64 {
|
||||
return false
|
||||
}
|
||||
seen[name] = true
|
||||
if parent, ok := r.Host.Parent(name); ok {
|
||||
return r.walk(parent, depth, seen, keys, reason)
|
||||
}
|
||||
if file, ok := r.Host.LoopBackingFile(name); ok {
|
||||
p := r.hostPath(file, depth+1)
|
||||
if !p.Computed() {
|
||||
return false
|
||||
}
|
||||
for _, k := range p.PhysicalDisks {
|
||||
keys[k] = true
|
||||
}
|
||||
noteReason(reason, p.Reason)
|
||||
return true
|
||||
}
|
||||
if slaves := r.Host.Slaves(name); len(slaves) > 0 {
|
||||
for _, s := range slaves {
|
||||
if !r.walk(s, depth, seen, keys, reason) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
if strings.HasPrefix(name, "dm-") || strings.HasPrefix(name, "md") || strings.HasPrefix(name, "loop") {
|
||||
return false // a virtual device with nothing beneath it cannot be grounded
|
||||
}
|
||||
key, why := r.diskKey(name)
|
||||
keys[key] = true
|
||||
noteReason(reason, why)
|
||||
return true
|
||||
}
|
||||
|
||||
func noteReason(cur *string, why string) {
|
||||
if why == "" {
|
||||
return
|
||||
}
|
||||
if *cur == "" || reasonRank[why] < reasonRank[*cur] {
|
||||
*cur = why
|
||||
}
|
||||
}
|
||||
|
||||
// diskKey names one whole disk and says why it is not trusted ("" when it is).
|
||||
func (r *IdentityResolver) diskKey(name string) (key, reason string) {
|
||||
p := r.Host.UdevProps(name)
|
||||
serial := strings.TrimSpace(p["ID_SERIAL_SHORT"])
|
||||
model := strings.TrimSpace(p["ID_MODEL"])
|
||||
bus := strings.TrimSpace(p["ID_BUS"])
|
||||
wwn := strings.ToLower(strings.TrimSpace(p["ID_WWN"]))
|
||||
if bus == "" && strings.HasPrefix(name, "nvme") {
|
||||
bus = "nvme"
|
||||
}
|
||||
usb := bus == "usb" || strings.Contains(p["DEVPATH"], "/usb")
|
||||
switch {
|
||||
case strings.HasPrefix(name, "nvme") && (strings.HasPrefix(wwn, "eui.") || strings.HasPrefix(wwn, "nvme.")):
|
||||
key = "eui:" + strings.TrimPrefix(strings.TrimPrefix(wwn, "eui."), "nvme.")
|
||||
case serial != "":
|
||||
key = "serial:" + bus + ":" + model + ":" + serial
|
||||
if usb && FakeBridgeSerial(serial) {
|
||||
reason = ReasonUSBBridge
|
||||
}
|
||||
case wwn != "" && WWNHasVendor(wwn):
|
||||
key = "wwn:" + wwn
|
||||
case wwn != "":
|
||||
key, reason = "wwn:"+wwn, ReasonWeakWWN
|
||||
default:
|
||||
key, reason = "dev:"+name, ReasonNoSerial
|
||||
}
|
||||
if v := r.Host.Virt(); v != "none" {
|
||||
reason = ReasonVM // inside a VM a virtual disk says nothing about the host disk (spike cases 5–6)
|
||||
}
|
||||
return key, reason
|
||||
}
|
||||
|
||||
// WWNHasVendor reports whether an NAA-5/6 WWN carries a registered vendor (OUI ≠ 000000). Other NAA
|
||||
// formats are taken as they are. Measured: the N100's AirDisk SSD reports 0x5000000123456f1f.
|
||||
func WWNHasVendor(wwn string) bool {
|
||||
h := strings.TrimPrefix(strings.ToLower(wwn), "0x")
|
||||
if len(h) < 7 {
|
||||
return false
|
||||
}
|
||||
if h[0] == '5' || h[0] == '6' {
|
||||
return h[1:7] != "000000"
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// FakeBridgeSerial reports a serial that a USB bridge reports for any disk: too short, one repeated
|
||||
// character, or a known placeholder. UNPROVEN on a real enclosure (R-1077) — the rule is the design's.
|
||||
func FakeBridgeSerial(s string) bool {
|
||||
s = strings.ToUpper(strings.TrimSpace(s))
|
||||
if len(s) < 6 {
|
||||
return true
|
||||
}
|
||||
if strings.Count(s, s[:1]) == len(s) {
|
||||
return true
|
||||
}
|
||||
switch s {
|
||||
case "123456789", "1234567890", "123456789ABC", "0123456789ABCDEF", "1234567890ABCDEF", "DEADBEEF":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func sortedKeys(m map[string]bool) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// Relation compares two places (`16` §4). A SHARED disk key is a positive fact, so two places on one disk
|
||||
// are "same, proven" even when that disk is unknown (inside a VM, two parts of one virtual disk). Otherwise
|
||||
// proven is true only when both verdicts are proven; different then means "no disk in common". When proven
|
||||
// is false, different only says the known keys do not overlap — callers must act on proven first.
|
||||
// reason: "" (proven, different), "same_disk", "unmapped" (a side has no verdict), or the unknown reason.
|
||||
func Relation(a, b PlaceIdentity) (different, proven bool, reason string) {
|
||||
if !a.Computed() || !b.Computed() {
|
||||
return false, false, "unmapped"
|
||||
}
|
||||
set := map[string]bool{}
|
||||
for _, k := range a.PhysicalDisks {
|
||||
set[k] = true
|
||||
}
|
||||
shared := false
|
||||
for _, k := range b.PhysicalDisks {
|
||||
if set[k] {
|
||||
shared = true
|
||||
}
|
||||
}
|
||||
if shared {
|
||||
return false, true, "same_disk"
|
||||
}
|
||||
if a.Identity == IdentityProven && b.Identity == IdentityProven {
|
||||
return true, true, ""
|
||||
}
|
||||
why := a.Reason
|
||||
if why == "" || (b.Reason != "" && reasonRank[b.Reason] < reasonRank[why]) {
|
||||
why = b.Reason
|
||||
}
|
||||
return true, false, why
|
||||
}
|
||||
@@ -0,0 +1,257 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Fixture /sys/class/block + /run/udev/data + /proc/mounts trees, one per `16` §3 case. The tree mimics the
|
||||
// kernel: /sys/devices/.../<disk>/<part> real dirs (a partition carries a "partition" file), /sys/class/block
|
||||
// symlinks, a "dev" file per node, udev "E:" lines keyed by major:minor. The cases are the ones the spike
|
||||
// measured on real devices (felhom.eu documentation/audits/storage-identity-2026-10-15/).
|
||||
|
||||
type fx struct {
|
||||
t *testing.T
|
||||
root string
|
||||
minor int
|
||||
mounts []string
|
||||
links map[string]string // /dev path -> kernel name
|
||||
virt string
|
||||
}
|
||||
|
||||
func newFx(t *testing.T) *fx {
|
||||
t.Helper()
|
||||
r := t.TempDir()
|
||||
for _, d := range []string{"sys/class/block", "sys/devices", "udev"} {
|
||||
if err := os.MkdirAll(filepath.Join(r, d), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return &fx{t: t, root: r, links: map[string]string{}, virt: "none"}
|
||||
}
|
||||
|
||||
func (f *fx) write(p, s string) {
|
||||
f.t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte(s), 0o644); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// node makes a block node. parent "" → a top-level device. udev lines are "KEY=VAL".
|
||||
func (f *fx) node(name, parent string, udev ...string) {
|
||||
f.t.Helper()
|
||||
real := filepath.Join(f.root, "sys/devices", name)
|
||||
if parent != "" {
|
||||
pr, _ := filepath.EvalSymlinks(filepath.Join(f.root, "sys/class/block", parent))
|
||||
real = filepath.Join(pr, name)
|
||||
f.write(filepath.Join(real, "partition"), "1\n")
|
||||
}
|
||||
f.minor++
|
||||
dev := "8:" + strconv.Itoa(f.minor)
|
||||
f.write(filepath.Join(real, "dev"), dev+"\n")
|
||||
if err := os.Symlink(real, filepath.Join(f.root, "sys/class/block", name)); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
var b strings.Builder
|
||||
for _, u := range udev {
|
||||
b.WriteString("E:" + u + "\n")
|
||||
}
|
||||
f.write(filepath.Join(f.root, "udev", "b"+dev), b.String())
|
||||
f.links["/dev/"+name] = name
|
||||
}
|
||||
|
||||
func (f *fx) slaves(name string, slaves ...string) {
|
||||
f.t.Helper()
|
||||
real, _ := filepath.EvalSymlinks(filepath.Join(f.root, "sys/class/block", name))
|
||||
for _, s := range slaves {
|
||||
if err := os.MkdirAll(filepath.Join(real, "slaves", s), 0o755); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fx) loop(name, file string) {
|
||||
real, _ := filepath.EvalSymlinks(filepath.Join(f.root, "sys/class/block", name))
|
||||
f.write(filepath.Join(real, "loop", "backing_file"), file+"\n")
|
||||
}
|
||||
|
||||
func (f *fx) mount(dev, mp, fstype string) {
|
||||
f.mounts = append(f.mounts, dev+" "+mp+" "+fstype+" rw 0 0")
|
||||
}
|
||||
|
||||
func (f *fx) resolver() *IdentityResolver {
|
||||
f.write(filepath.Join(f.root, "mounts"), strings.Join(f.mounts, "\n")+"\n")
|
||||
return &IdentityResolver{Host: &SysfsIdentityHost{
|
||||
SysClass: filepath.Join(f.root, "sys/class/block"),
|
||||
UdevData: filepath.Join(f.root, "udev"),
|
||||
ProcMounts: filepath.Join(f.root, "mounts"),
|
||||
DevResolve: func(p string) (string, error) {
|
||||
if n, ok := f.links[p]; ok {
|
||||
return "/dev/" + n, nil
|
||||
}
|
||||
return p, nil
|
||||
},
|
||||
VirtFn: func() string { return f.virt },
|
||||
}}
|
||||
}
|
||||
|
||||
var sanDisk = []string{"ID_BUS=ata", "ID_MODEL=SanDisk_X600", "ID_SERIAL_SHORT=182195804614", "ID_WWN=0x5001b448b60a9d6e"}
|
||||
|
||||
// oneDiskLVM builds the PVE default layout on one SATA SSD: sda3 → pve-root, pve-data tmeta/tdata → tpool →
|
||||
// a guest volume (the shape of all three test boxes).
|
||||
func oneDiskLVM(f *fx) {
|
||||
f.node("sda", "", sanDisk...)
|
||||
f.node("sda3", "sda")
|
||||
for _, d := range []string{"dm-1", "dm-2", "dm-3", "dm-4", "dm-6"} {
|
||||
f.node(d, "")
|
||||
}
|
||||
f.slaves("dm-1", "sda3")
|
||||
f.slaves("dm-2", "sda3")
|
||||
f.slaves("dm-3", "sda3")
|
||||
f.slaves("dm-4", "dm-2", "dm-3")
|
||||
f.slaves("dm-6", "dm-4")
|
||||
f.links["/dev/mapper/pve-root"] = "dm-1"
|
||||
f.links["/dev/mapper/pve-data-tpool"] = "dm-4"
|
||||
f.links["/dev/pve/vm-9201-disk-1"] = "dm-6"
|
||||
f.mount("/dev/mapper/pve-root", "/", "ext4")
|
||||
}
|
||||
|
||||
func TestIdentity_OneDiskLVM_AllPlacesSameProvenDisk(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
r := f.resolver()
|
||||
want := []string{"serial:ata:SanDisk_X600:182195804614"}
|
||||
for _, p := range []PlaceIdentity{r.HostPath("/var/lib/vz"), r.Device("/dev/mapper/pve-data-tpool"), r.Device("/dev/pve/vm-9201-disk-1")} {
|
||||
if p.Identity != IdentityProven || !reflect.DeepEqual(p.PhysicalDisks, want) {
|
||||
t.Fatalf("want proven %v, got %+v", want, p)
|
||||
}
|
||||
}
|
||||
diff, proven, why := Relation(r.HostPath("/var/lib/vz"), r.Device("/dev/pve/vm-9201-disk-1"))
|
||||
if diff || !proven || why != "same_disk" {
|
||||
t.Fatalf("root vs guest volume: want same+proven, got diff=%v proven=%v why=%q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_TwoDisks_ProvenDifferent(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.node("sdb", "", "ID_BUS=ata", "ID_MODEL=TOSHIBA_MQ04ABF100", "ID_SERIAL_SHORT=65NOP3HDT", "ID_WWN=0x5000039ddb108568", "DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-3/block/sdb")
|
||||
f.mount("/dev/sdb", "/mnt/hdd_1", "ext4")
|
||||
f.mount("/dev/sdb", "/mnt/felhom-drives/hdd_1", "ext4") // the felhom-data bind shows the source device
|
||||
r := f.resolver()
|
||||
drive := r.HostPath("/mnt/felhom-drives/hdd_1/userdata")
|
||||
if drive.Identity != IdentityProven || drive.PhysicalDisks[0] != "serial:ata:TOSHIBA_MQ04ABF100:65NOP3HDT" {
|
||||
t.Fatalf("USB drive passing its identity: want proven, got %+v", drive)
|
||||
}
|
||||
diff, proven, why := Relation(r.Device("/dev/pve/vm-9201-disk-1"), drive)
|
||||
if !diff || !proven || why != "" {
|
||||
t.Fatalf("system vs drive: want different+proven, got %v %v %q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_TwoPartitionsOneDisk_Same(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("sdc", "", "ID_BUS=usb", "ID_MODEL=Portable", "ID_SERIAL_SHORT=NAAB7C21")
|
||||
f.node("sdc1", "sdc")
|
||||
f.node("sdc2", "sdc")
|
||||
f.mount("/dev/sdc1", "/mnt/a", "ext4")
|
||||
f.mount("/dev/sdc2", "/mnt/b", "ext4")
|
||||
r := f.resolver()
|
||||
diff, proven, why := Relation(r.HostPath("/mnt/a"), r.HostPath("/mnt/b"))
|
||||
if diff || !proven || why != "same_disk" {
|
||||
t.Fatalf("two partitions of one disk: want same+proven, got %v %v %q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_LoopPartitionsOnAFile_ResolveToTheFilesDisk(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("nvme0n1", "", "ID_MODEL=KXG50PNV1T02", "ID_SERIAL_SHORT=58BS11AFT8MQ", "ID_WWN=eui.000000000000001000080d020046c07b")
|
||||
f.mount("/dev/nvme0n1", "/mnt/hdd_1", "ext4")
|
||||
f.node("loop2", "")
|
||||
f.loop("loop2", "/mnt/hdd_1/scratch.img")
|
||||
f.node("loop2p1", "loop2") // the partition step must run BEFORE the loop step (the spike's probe bug)
|
||||
f.mount("/dev/loop2p1", "/tmp/scrA", "ext4")
|
||||
r := f.resolver()
|
||||
p := r.HostPath("/tmp/scrA")
|
||||
if p.Identity != IdentityProven || !reflect.DeepEqual(p.PhysicalDisks, []string{"eui:000000000000001000080d020046c07b"}) {
|
||||
t.Fatalf("loop partition: want the NVMe's eui, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_WWNWithoutVendor(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("sda", "", "ID_BUS=ata", "ID_MODEL=AirDisk_512GB_SSD", "ID_SERIAL_SHORT=QDF922W009654S30EX", "ID_WWN=0x5000000123456f1f")
|
||||
f.node("sdb", "", "ID_BUS=ata", "ID_MODEL=Cheap", "ID_WWN=0x5000000123456f1f")
|
||||
r := f.resolver()
|
||||
if p := r.Device("/dev/sda"); p.Identity != IdentityProven || p.PhysicalDisks[0] != "serial:ata:AirDisk_512GB_SSD:QDF922W009654S30EX" {
|
||||
t.Fatalf("a real serial wins over a vendor-less WWN: got %+v", p)
|
||||
}
|
||||
if p := r.Device("/dev/sdb"); p.Identity != IdentityUnknown || p.Reason != ReasonWeakWWN {
|
||||
t.Fatalf("no serial + vendor-less WWN: want unknown/weak_wwn, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_NoSerial(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("vdb", "", "ID_MODEL=")
|
||||
if p := f.resolver().Device("/dev/vdb"); p.Identity != IdentityUnknown || p.Reason != ReasonNoSerial {
|
||||
t.Fatalf("want unknown/no_serial, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_InsideAVM_EverythingUnknown(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.virt = "kvm"
|
||||
f.node("sdb", "", "ID_BUS=scsi", "ID_MODEL=QEMU_HARDDISK", "ID_SERIAL_SHORT=drive-scsi1")
|
||||
f.mount("/dev/sdb", "/mnt/felhom-drives/adatlemez", "ext4")
|
||||
r := f.resolver()
|
||||
a, b := r.Device("/dev/pve/vm-9201-disk-1"), r.HostPath("/mnt/felhom-drives/adatlemez")
|
||||
for _, p := range []PlaceIdentity{a, b} {
|
||||
if p.Identity != IdentityUnknown || p.Reason != ReasonVM {
|
||||
t.Fatalf("inside a VM: want unknown/vm, got %+v", p)
|
||||
}
|
||||
}
|
||||
diff, proven, why := Relation(a, b)
|
||||
if proven || why != ReasonVM || !diff {
|
||||
t.Fatalf("two virtual disks: want not proven (vm), got %v %v %q", diff, proven, why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_USBBridgeSerial(t *testing.T) {
|
||||
f := newFx(t)
|
||||
f.node("sdd", "", "ID_BUS=usb", "ID_MODEL=External_USB3.0", "ID_SERIAL_SHORT=000000000000")
|
||||
f.node("sde", "", "ID_BUS=usb", "ID_MODEL=External_USB3.0", "ID_SERIAL_SHORT=20231106A1B2")
|
||||
r := f.resolver()
|
||||
if p := r.Device("/dev/sdd"); p.Identity != IdentityUnknown || p.Reason != ReasonUSBBridge {
|
||||
t.Fatalf("a bridge placeholder serial: want unknown/usb_bridge, got %+v", p)
|
||||
}
|
||||
if p := r.Device("/dev/sde"); p.Identity != IdentityProven {
|
||||
t.Fatalf("a USB disk with a plausible serial: want proven, got %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentity_NetworkShareAndUnmapped(t *testing.T) {
|
||||
f := newFx(t)
|
||||
oneDiskLVM(f)
|
||||
f.mount("//192.168.0.123/TesztFilmek", "/mnt/felhom-drives/omv-filmek", "cifs")
|
||||
r := f.resolver()
|
||||
n := r.HostPath("/mnt/felhom-drives/omv-filmek/x")
|
||||
if n.Identity != IdentityUnknown || n.Reason != ReasonNetwork || len(n.PhysicalDisks) != 0 {
|
||||
t.Fatalf("network share: want unknown/network with no disks, got %+v", n)
|
||||
}
|
||||
_, proven, why := Relation(r.HostPath("/"), n)
|
||||
if proven || why != ReasonNetwork {
|
||||
t.Fatalf("system vs network: want not proven (network), got %v %q", proven, why)
|
||||
}
|
||||
if _, proven, why := Relation(PlaceIdentity{}, n); proven || why != "unmapped" {
|
||||
t.Fatalf("an uncomputed side: want unmapped, got %v %q", proven, why)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The root mkfs wrapper's own tests (configs/test_felhom_mkfs_guarded.py, agent v0.158.0 — a sibling partition that
|
||||
// is Felhom's own drive no longer blocks formatting the part next to it; every other mount still refuses) run with
|
||||
// the Go suite.
|
||||
func TestMkfsGuardedPythonSuite(t *testing.T) {
|
||||
py, err := exec.LookPath("python3")
|
||||
if err != nil {
|
||||
t.Skip("python3 not found")
|
||||
}
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
script := filepath.Join(filepath.Dir(file), "..", "..", "configs", "test_felhom_mkfs_guarded.py")
|
||||
out, err := exec.Command(py, script).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("wrapper suite failed: %v\n%s", err, out)
|
||||
}
|
||||
}
|
||||
@@ -40,6 +40,10 @@ type Observer struct {
|
||||
// R-672 (v0.133.0): a thin pool crossing thinPoolAlarmFraction (data OR metadata) requests an
|
||||
// out-of-band host report at once, so the hub's storage-fill alarm sees it in seconds instead of at
|
||||
// the next 15-minute report. Rising edge per pool; re-armed below thinPoolRearmFraction.
|
||||
// ident (agent v0.157.0, `16` §3) computes each target's physical-disk identity. nil → the fields stay
|
||||
// empty (the pre-0.157 shape), which every consumer reads as "not computed".
|
||||
ident *IdentityResolver
|
||||
|
||||
highMu sync.Mutex
|
||||
high map[string]bool
|
||||
onThinHigh func()
|
||||
@@ -50,6 +54,51 @@ const (
|
||||
thinPoolRearmFraction = 0.85
|
||||
)
|
||||
|
||||
// SetIdentity wires the physical-disk identity resolver (main: NewIdentityResolver()).
|
||||
func (o *Observer) SetIdentity(r *IdentityResolver) { o.ident = r }
|
||||
|
||||
// placeIdentity fills a target's identity fields (`16` §3). Network and remote stores are fixed
|
||||
// "unknown/network"; a dir storage is judged only on its OWN mount (the same rule as backing_device —
|
||||
// never the containing filesystem, so an absent drive is "not computed", never the root disk); the
|
||||
// builtin local and an lvmthin pool are the system disk(s) they sit on.
|
||||
func (o *Observer) placeIdentity(tgt *hub.StorageTarget, s proxmox.Storage) {
|
||||
if o.ident == nil {
|
||||
return
|
||||
}
|
||||
var p PlaceIdentity
|
||||
switch tgt.Type {
|
||||
case hub.StorageTypeNFS, hub.StorageTypeCIFS, hub.StorageTypePBS:
|
||||
tgt.Kind = PlaceKindNetwork
|
||||
p = NetworkPlace()
|
||||
case hub.StorageTypeLocal:
|
||||
tgt.Kind = PlaceKindSystem
|
||||
if s.Path != "" {
|
||||
p = o.ident.HostPath(s.Path)
|
||||
}
|
||||
case hub.StorageTypeLVMThin:
|
||||
tgt.Kind = PlaceKindSystem
|
||||
if s.VGName != "" && s.ThinPool != "" {
|
||||
p = o.ident.Device("/dev/mapper/" + dmEscape(s.VGName) + "-" + dmEscape(s.ThinPool) + "-tpool")
|
||||
}
|
||||
default:
|
||||
if tgt.BackingDevice == "" {
|
||||
return // not its own mount (absent, or a dir on root): not computed
|
||||
}
|
||||
tgt.Kind = PlaceKindDisk
|
||||
if _, known := wholeDiskOf(tgt.BackingDevice); known && !reWholeDisk.MatchString(tgt.BackingDevice) {
|
||||
tgt.Kind = PlaceKindPartition
|
||||
}
|
||||
p = o.ident.Device(tgt.BackingDevice)
|
||||
}
|
||||
if !p.Computed() {
|
||||
return
|
||||
}
|
||||
tgt.PhysicalDisks, tgt.Identity, tgt.IdentityReason = p.PhysicalDisks, p.Identity, p.Reason
|
||||
}
|
||||
|
||||
// dmEscape doubles dashes the way device-mapper names an LV (pve + vm-1-disk-0 → pve-vm--1--disk--0).
|
||||
func dmEscape(s string) string { return strings.ReplaceAll(s, "-", "--") }
|
||||
|
||||
// SetThinHighTrigger wires the out-of-band report request (main: the storage trigger channel).
|
||||
func (o *Observer) SetThinHighTrigger(f func()) { o.onThinHigh = f }
|
||||
|
||||
@@ -303,6 +352,9 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
|
||||
o.noteThinFill(s.Storage+"/data", frac)
|
||||
}
|
||||
|
||||
// Physical-disk identity (`16` §3) — after typ/backing are settled; never from a caller's claim.
|
||||
o.placeIdentity(&tgt, s)
|
||||
|
||||
// SMART-only device hint (v0.95.0): a dir-storage that lives INSIDE a shared filesystem (the
|
||||
// builtin `local` on the LVM root) has empty backing by design, yet we can still read the PHYSICAL
|
||||
// disk's SMART by resolving its containing filesystem. Gated to catDir + no own backing + reachable,
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user