admin
0e8fd814ba
The first sweep on felhom-pve logged 'dhclient liveness probe failed: sudo: a password is required' and reported state=unknown — the watchdog fail-safed correctly, but it was blind. TASK-D assumed no sudoers change; the existing grant only covered lanresolver's address read. FELHOM_GUESTNET adds four FIXED vectors (route, interfaces, pgrep, and the incident's heal invocation). Every argument after the numeric vmid is a literal, so nothing the guest or the hub says can widen the grant. Plus four guestnet-* capability rows, non-critical, so a host that has not taken the new sudoers is visibly degraded without paging an operator per box.
196 lines
18 KiB
Go
196 lines
18 KiB
Go
// Package capability is the agent's privileged-capability self-check (slice 1 of agent
|
|
// self-health). It declares the MANIFEST — the (binary, representative-arg-vector) pairs the
|
|
// non-root agent depends on running via `sudo -n` — and a PROBE that lists each against the live
|
|
// sudoers policy (`sudo -n -l`, never executing) + checks the binary exists. The result is a
|
|
// snapshot the agent attaches to its hub report; the hub owns the ok→degraded transition + alert.
|
|
//
|
|
// Why this exists: the 2026-06-28 root→non-root cutover dropped several grants from
|
|
// configs/felhom-agent.sudoers (lxc-info, make-private, restart dnsmasq, …). Each broke a feature
|
|
// silently until a user hit it (the multi-drive flapping incident, audit 2026-06-29). A non-root
|
|
// agent that can't run a command it depends on is DEGRADED and must SAY so — at cutover, not days
|
|
// later. The companion build-time test (manifest_test.go) asserts every manifest vector is covered
|
|
// by a sudoers pattern, catching authoring gaps in CI before they ship.
|
|
package capability
|
|
|
|
import (
|
|
"strings"
|
|
|
|
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
|
|
)
|
|
|
|
// Capability is one privileged command the agent depends on. Name is a stable id; Feature is the
|
|
// human-readable thing that breaks if the grant is missing (used in logs + the operator alert).
|
|
// Binary is the absolute path the runner invokes; ReprArgs is a CONCRETE argument vector that
|
|
// matches the corresponding sudoers glob (e.g. a vmid "9201" matches `[0-9]*`, a device "/dev/sda"
|
|
// matches `/dev/*`). Critical marks the user-facing ones — the hub alerts only when a Critical
|
|
// capability is degraded (non-critical degradations still ride the report snapshot + agent log).
|
|
type Capability struct {
|
|
Name string
|
|
Feature string
|
|
Binary string
|
|
ReprArgs []string
|
|
Critical bool
|
|
// GatedBy names the config gate this capability sits behind ("" = always active). A healthy
|
|
// probe of a gated capability whose gate is OFF reports StatusInactive instead of ok — see
|
|
// Prober.GateActive. Broken plumbing stays degraded regardless of the gate.
|
|
GatedBy string
|
|
}
|
|
|
|
// GatePBSDR is the gate id for the PBS-DR tier: active when this box carries a live (enabled)
|
|
// pbs_dr descriptor or a previously-converged marker (pbsdr.Manager.DRConfigured).
|
|
const GatePBSDR = "pbs_dr"
|
|
|
|
// Manifest is the required set, seeded from the 2026-06-29 sudoers audit (felhom-agent/REPORT.md):
|
|
// the OK + newly-CLOSED rows. The SURFACED/DEFERRED rows are deliberately EXCLUDED — they are not
|
|
// required capabilities: the general `pct exec <vmid> -- *` (controller-swap; arbitrary exec, an
|
|
// open operator decision), `pct create` (golden build, maintenance, no daemon caller), `mount
|
|
// UUID=…` (legacy/unreferenced), and the callerless `sensors -j`. Adding them here would assert
|
|
// grants the agent neither has nor should depend on.
|
|
//
|
|
// Each ReprArgs is a representative instance; the probe LISTS it (`sudo -n -l`) and never runs it,
|
|
// so even mkfs/pct-set entries are side-effect-free to probe.
|
|
//
|
|
// Gating (v0.86.0): every `pbsdr-*` capability is behind GatePBSDR — the whole pbsdr vector IS
|
|
// the DR-tier apply path, so the gate rides the stable name prefix (a new pbsdr-* op is gated
|
|
// automatically; nothing else ever is). Applied here, not in the literals, to keep the manifest
|
|
// entries positional.
|
|
func Manifest() []Capability {
|
|
out := make([]Capability, len(manifest))
|
|
copy(out, manifest)
|
|
for i := range out {
|
|
if strings.HasPrefix(out[i].Name, "pbsdr-") {
|
|
out[i].GatedBy = GatePBSDR
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
// reprFingerprint is a shape-valid all-zero SHA-256 colon fingerprint for list-mode repr vectors
|
|
// (matches the wrapper's fingerprint validation; never executed).
|
|
const reprFingerprint = "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"
|
|
|
|
var manifest = []Capability{
|
|
// ---- Intermediary drive model (the multi-drive path — mostly Critical) ----
|
|
{"guest-init-pid", "drive-gate guest-sees check (multi-drive concurrency)", "/usr/bin/lxc-info", []string{"-n", "9201", "-p", "-H"}, true, ""},
|
|
{"parent-self-bind", "intermediary shared-parent self-bind", "/usr/bin/mount", []string{"--bind", "/mnt/felhom-drives", "/mnt/felhom-drives"}, true, ""},
|
|
{"parent-make-shared", "intermediary shared-parent propagation", "/usr/bin/mount", []string{"--make-shared", "/mnt/felhom-drives"}, true, ""},
|
|
{"parent-make-private", "intermediary shared-parent peer-group isolation", "/usr/bin/mount", []string{"--make-private", "/mnt/felhom-drives"}, true, ""},
|
|
{"drive-bind", "drive attach (felhom-data bind under parent)", "/usr/bin/mount", []string{"--bind", "/mnt/felhom-usb/felhom-data", "/mnt/felhom-drives/felhom-usb"}, true, ""},
|
|
{"drive-umount", "drive detach (fail-closed unmount)", "/usr/bin/umount", []string{"/mnt/felhom-drives/felhom-usb"}, true, ""},
|
|
{"drives-mkdir-parent", "stable parent dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives"}, false, ""},
|
|
{"drives-mkdir-sub", "per-drive stable dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives/felhom-usb"}, false, ""},
|
|
{"drives-mkdir-data", "felhom-data namespace create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-usb/felhom-data"}, false, ""},
|
|
{"drives-chown-data", "felhom-data guest-root chown", "/usr/bin/chown", []string{"100000:100000", "/mnt/felhom-usb/felhom-data"}, false, ""},
|
|
{"parent-script-install", "shared-parent boot script install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-shared-parent-123456789.sh", "/usr/local/sbin/felhom-shared-parent.sh"}, false, ""},
|
|
{"parent-unit-install", "shared-parent boot unit install", "/usr/bin/install", []string{"-m", "0644", "--", "/tmp/felhom-shared-parent-123456789.service", "/etc/systemd/system/felhom-shared-parent.service"}, false, ""},
|
|
{"parent-unit-enable", "shared-parent boot-persistence enable", "/usr/bin/systemctl", []string{"enable", "felhom-shared-parent.service"}, false, ""},
|
|
{"parent-bind-mp8", "parent bind into guest at provision", "/usr/sbin/pct", []string{"set", "9201", "-mp8", "/mnt/felhom-drives"}, false, ""},
|
|
|
|
// ---- Disk inspect / format gate (Critical: the data-bearing classifier + format) ----
|
|
{"disk-blkid", "disk data-bearing classify (format gate)", "/usr/sbin/blkid", []string{"-p", "-o", "export", "/dev/sda"}, true, ""},
|
|
{"disk-lsblk", "disk topology read (format gate)", "/usr/bin/lsblk", []string{"-J", "-o", "NAME,FSTYPE,PTTYPE,MOUNTPOINT", "/dev/sda"}, true, ""},
|
|
{"disk-mkfs-ext4", "guarded format (ext4)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "ext4"}, true, ""},
|
|
{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
|
|
{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
|
|
{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
|
|
|
|
// ---- Storage mount units (watchdog re-mount) ----
|
|
{"mount-unit-install", "fs-UUID mount unit install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0644", "--", "/var/lib/felhom-agent/units/felhom-x.mount", "/etc/systemd/system/felhom-x.mount"}, false, ""},
|
|
{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
|
|
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "felhom-x.mount"}, false, ""},
|
|
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "felhom-x.mount"}, false, ""},
|
|
{"mount-unit-stop", "mount unit stop", "/usr/bin/systemctl", []string{"stop", "--", "felhom-x.mount"}, false, ""},
|
|
|
|
// ---- Network storage re-arm + cleanup (CAMPAIGN-3 F10/F1) ----
|
|
{"netmount-reset-failed", "NAS automount re-arm after start-limit (F10)", "/usr/bin/systemctl", []string{"reset-failed", "--", "mnt-felhom\\x2ddrives-media.automount"}, false, ""},
|
|
{"netmount-rmdir", "removed-share mountpoint cleanup (F1)", "/usr/bin/rmdir", []string{"/mnt/felhom-drives/media"}, false, ""},
|
|
|
|
// ---- Node self-heal (CAMPAIGN-3 F12-class, appliance-gated in code) ----
|
|
{"selfheal-networking-start", "appliance networking recovery at boot (F12 defense in depth)", "/usr/bin/systemctl", []string{"start", "networking.service"}, false, ""},
|
|
|
|
// ---- Provisioning back-half ----
|
|
{"provision-chown", "bootstrap mount guest-root chown", "/usr/bin/chown", []string{"-R", "100000:100000", "/var/lib/felhom-agent/guests/9201"}, false, ""},
|
|
{"provision-config-mount", "bootstrap config bind mount", "/usr/sbin/pct", []string{"set", "9201", "-mp0", "/var/lib/felhom-agent/guests/9201"}, false, ""},
|
|
{"provision-onboot", "customer guest autostart (onboot)", "/usr/sbin/pct", []string{"set", "9201", "-onboot", "1"}, false, ""},
|
|
|
|
// ---- Pre-start self-heal hook + guest lifecycle ----
|
|
{"guesthook-install", "pre-start hook snippet install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-guest-hook-123456789.sh", "/var/lib/vz/snippets/felhom-guest-hook.sh"}, false, ""},
|
|
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
|
|
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
|
|
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
|
|
|
|
// ---- LAN split-horizon resolver (dnsmasq) ----
|
|
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
|
|
{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
|
|
{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
|
|
{"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""},
|
|
{"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""},
|
|
{"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
|
|
{"dnsmasq-guest-ip", "guest LAN IP discovery", "/usr/sbin/pct", []string{"exec", "9201", "--", "ip", "-4", "-o", "addr", "show", "dev", "eth0"}, false, ""},
|
|
{"dnsmasq-guest-domain", "guest domain discovery", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "exec", "felhom-controller", "cat", "/opt/docker/felhom-controller/controller.yaml"}, false, ""},
|
|
|
|
// ---- Guest-network watchdog (FELHOM_GUESTNET, R-54, v0.92.0) ----
|
|
// Deliberately NOT Critical: a host that has not yet taken the new sudoers file should show
|
|
// degraded (visible, diagnosable) without paging an operator for every box on rollout day —
|
|
// the R-50b lesson that an amber fleet is background noise. The address read is covered by
|
|
// dnsmasq-guest-ip above and is not duplicated here (one command, one capability row).
|
|
{"guestnet-route", "guest default-route probe", "/usr/sbin/pct", []string{"exec", "9201", "--", "ip", "route", "show", "default"}, false, ""},
|
|
{"guestnet-ifaces", "guest interface-mode read", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/network/interfaces"}, false, ""},
|
|
{"guestnet-dhclient-probe", "guest DHCP-client liveness probe", "/usr/sbin/pct", []string{"exec", "9201", "--", "pgrep", "-x", "dhclient"}, false, ""},
|
|
{"guestnet-heal", "guest DHCP-client restart (the 2026-07-20 heal)", "/usr/sbin/pct", []string{"exec", "9201", "--", "dhclient", "-pf", "/run/dhclient.eth0.pid", "-lf", "/var/lib/dhcp/dhclient.eth0.leases", "eth0"}, false, ""},
|
|
|
|
// ---- Controller-swap / managed auto-update (FELHOM_CONTROLLERSWAP, v0.45.0; Critical: a
|
|
// silently-broken fleet auto-update is operator-alert-worthy) ----
|
|
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
|
|
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
|
|
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
|
|
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
|
|
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "tee", "/etc/felhom-controller-image"}, true, ""},
|
|
|
|
// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
|
|
// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
|
|
{"stalelock-unlock", "reboot-during-backup stale-lock recovery", "/usr/sbin/pct", []string{"unlock", "9201"}, true, ""},
|
|
|
|
// ---- Offsite WG tunnel (FELHOM_WG, S3/v0.64.0; Critical FLIPPED in S4/v0.66.0 — offsite
|
|
// backups now RIDE the tunnel, so a degraded tunnel capability is operator-alert-worthy: the
|
|
// conf install, unit enable/restart and the handshake read gate the backup path. apt-install
|
|
// (one-time bootstrap) and disable (revocation, a deliberate teardown) stay non-critical. The
|
|
// handshake read is the ONLY wg invocation (never `dump`). ----
|
|
{"wg-tools-install", "wireguard-tools package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "wireguard-tools"}, false, ""},
|
|
{"wg-conf-install", "wg-felhom conf install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0600", "--", "/var/lib/felhom-agent/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"}, true, ""},
|
|
{"wg-enable", "wg-quick@wg-felhom enable", "/usr/bin/systemctl", []string{"enable", "--now", "wg-quick@wg-felhom"}, true, ""},
|
|
{"wg-restart", "wg-quick@wg-felhom restart (conf change)", "/usr/bin/systemctl", []string{"restart", "wg-quick@wg-felhom"}, true, ""},
|
|
{"wg-disable", "wg-quick@wg-felhom disable (revocation)", "/usr/bin/systemctl", []string{"disable", "--now", "wg-quick@wg-felhom"}, false, ""},
|
|
{"wg-handshake-read", "tunnel handshake-age read", "/usr/bin/wg", []string{"show", "wg-felhom", "latest-handshakes"}, true, ""},
|
|
|
|
// ---- PBS DR tier apply (FELHOM_PBSDR, slice 2). NON-critical (the selfupdate rationale):
|
|
// applying the tier is an occasional hub-driven provisioning op, not a steady-state serving
|
|
// path — a degraded grant means "can't provision/reconcile the PBS entry" (the bridge reports
|
|
// loudly anyway), not a serving outage. The steady-state backup path is covered by the wg +
|
|
// storage capabilities. List-mode representations only; never executed. ----
|
|
{"pbsdr-create", "PBS DR storage-entry create (K autogen)", "/usr/local/sbin/felhom-pbs-apply", []string{"create", "felhom-pbs", "10.77.0.1", "felhom-offsite", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
|
|
{"pbsdr-reconcile", "PBS DR storage-entry reconcile (set-only)", "/usr/local/sbin/felhom-pbs-apply", []string{"reconcile", "felhom-pbs", "10.77.0.1", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
|
|
{"pbsdr-grant", "PBS DR storage ACL self-grant", "/usr/local/sbin/felhom-pbs-apply", []string{"grant", "felhom-pbs"}, false, ""},
|
|
// R-39 leg (b), v0.91.0: the credential READ path. Its absence is what made the PBS verify loop
|
|
// permanently blind to an applied-but-401 tier, so a host missing this verb is DEGRADED in a way
|
|
// that matters — it cannot detect the failure this whole tier exists to survive.
|
|
{"pbsdr-read", "PBS DR credential read (verify-loop auth probe)", "/usr/local/sbin/felhom-pbs-apply", []string{"read", "felhom-pbs", "/etc/pve/priv/storage"}, false, ""},
|
|
|
|
// ---- Escrow ceremony (FELHOM_ESCROW, controller-driven, v0.88.0). Critical: the customer
|
|
// wizard's whole run path IS this one grant — a dropped line silently breaks every ceremony.
|
|
// GatedBy is set EXPLICITLY (the name deliberately says "escrow", not "pbsdr-": the feature is
|
|
// the ceremony, but it only exists behind the DR tier — no PBS key, no ceremony). ReprArgs is
|
|
// the SHARED argv constant (internal/escrow/ceremony.go) — the exec runner uses the same one,
|
|
// so runner ↔ manifest can't drift, and TestManifestCoveredBySudoers locks manifest ↔ sudoers.
|
|
// List-mode probe only (`sudo -n -l`), spike-confirmed side-effect-free — never a real ceremony.
|
|
{"escrow-ceremony", "customer recovery-code ceremony (controller-driven)", escrow.CeremonyBinary, escrow.CeremonyArgs(), true, GatePBSDR},
|
|
|
|
// ---- Agent self-update (FELHOM_SELFUPDATE, D1). NON-critical: self-update is an occasional
|
|
// operator-driven op, not a steady-state serving path — a degraded grant means "can't
|
|
// self-update" (fall back to a manual SSH deploy), not a serving outage. The apply repr uses a
|
|
// staging-dir path + a placeholder sha (list-mode never runs it). ----
|
|
{"selfupdate-apply", "agent self-update apply (A/B flip)", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"apply", "/var/lib/felhom-agent/selfupdate/felhom-agent-0.0.0", "0000000000000000000000000000000000000000000000000000000000000000"}, false, ""},
|
|
{"selfupdate-commit", "agent self-update commit", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"commit"}, false, ""},
|
|
{"selfupdate-rollback", "agent self-update rollback", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"rollback"}, false, ""},
|
|
}
|