v0.80.0: PBS DR tier slice 2 — the apply-bridge (pbs_dr consumer, felhom-pbs-apply set-only wrapper, verify-pin-before-consume, adoption-first, loud consumed-failed, escrow seed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -1,3 +1,35 @@
|
||||
## v0.80.0 — PBS DR tier SLICE 2: the apply-bridge (2026-07-10)
|
||||
|
||||
Consumes hub v0.44.0's `pbs_dr` desired-state descriptor (slice 1): hub tick → the box grows the
|
||||
pbs storage entry + K, hands-free. Laws encoded (spike 00afadc + the offsite bridge precedent),
|
||||
each red-proof-verified (REPORT.md):
|
||||
|
||||
- **`internal/pbsdr`** — the bridge (wgtunnel Loop shape; raw-consumer seam; report stanza
|
||||
`pbs_dr`). Flow: **adoption probe first** (existing healthy entry → grant + marker, NO consume;
|
||||
**tenancy identity is entry-owned** — a descriptor naming a different namespace never repoints a
|
||||
live entry: the demo's manual `felhom-offsite` case) → fresh path: **verify-pin-BEFORE-consume**
|
||||
(bare TLS dial pinned to the descriptor fingerprint, `pbs.ProbeFingerprint`) → consume
|
||||
(`POST /api/v1/hosts/{id}/pbs/consume-token`, PLURAL /hosts/ — the slice-1 route; typed
|
||||
`ErrNoPBSSecret`) → wrapper `create` (secret on STDIN; `--encryption-key autogen` → K born; .enc/
|
||||
.pw placed into `backup.pbs_secret_dir` when overridden — the spike §4 escrow-path flag) →
|
||||
`grant` (the Part-0-evidenced dual-grant, exactly) → post-apply active probe → **seed
|
||||
`escrow.pbs_storage_id`** (bare ceremony one-liner; an operator-set different value is never
|
||||
clobbered; unknown config keys preserved) → descriptor-hash marker. **Consumed-but-failed is
|
||||
LOUD**: persistent `consumed_failed` report state, no silent retry — recovery only via the hub
|
||||
Re-issue (a fresh staged secret).
|
||||
- **`configs/felhom-pbs-apply`** (root wrapper, `/usr/local/sbin`, the guarded-mkfs shape) + ONE
|
||||
pinned sudoers alias `FELHOM_PBSDR` (create/reconcile/grant). **THE SET-ONLY LAW**: no deletion
|
||||
verb exists (entry deletion destroys K = un-decryptable backups); re-apply is `pvesm set`-only —
|
||||
grep-gated by `TestSetOnlyLaw` over the shipped file. Secret via wrapper stdin (sudo logs argv).
|
||||
- Wire: `WirePBSDR` on `WireDesiredState` (field-exact with hub `pbsDRDescriptor`, pinned by
|
||||
`TestWireFieldNames`); nil-safe on pre-v0.44.0 hubs. Report: `PBSDRStatus` stanza (adopted/
|
||||
applied/waiting_secret/verify_failed/consumed_failed/disabled) via the `PBSDRReporter` seam.
|
||||
- Capability manifest: `pbsdr-create`/`pbsdr-reconcile`/`pbsdr-grant` (non-critical, the
|
||||
selfupdate rationale). `config.Config.SourcePath` records the loaded file for the escrow seed.
|
||||
- Part 0 (recorded in REPORT.md): status reads ride `FelhomAgentBase` (`Datastore.Audit@/`); the
|
||||
WRITE path 403s without `FelhomAgentStore` on `/storage/<id>` → the grant op = the §4b dual-grant
|
||||
exactly. Demo's purged grants re-asserted; token vzdump to the PBS entry live-proven OK.
|
||||
|
||||
## v0.79.0 — SLICE 3: escrow upload carries sha256 of the sealed restic password (2026-07-09)
|
||||
|
||||
The hub-verified escrow auto-confirm chain, agent third: the escrow-create ceremony now records WHICH
|
||||
|
||||
@@ -135,10 +135,11 @@
|
||||
| `reconcile.OpVerifier` | internal/reconcile/gate.go | `*authz.Verifier` | fake verifier in internal/reconcile gate tests |
|
||||
| `signedjobs.WipeOps` / `Executor` (`ExecutorChain`) | internal/signedjobs/wipe.go + runner.go | `*storage.SudoHostOps`; `WipeExecutor`+`DecommissionExecutor` | internal/signedjobs wipe/runner/decommission tests |
|
||||
| `hub.reporter` / `collectorIface` / `EnvelopeObserver` | internal/hub/loop.go | `*hub.Client`, `*hub.Collector`; `desired.Syncer` + `signedjobs.Runner` | `fakeReporter`/`fakeCollector` internal/hub/loop_test.go |
|
||||
| `pbsdr.StorageReader` / `SecretConsumer` / `Manager.probeFP` (func seam) | internal/pbsdr/manager.go | `*proxmox.Client`; `*hub.Client`; `pbs.ProbeFingerprint` | `fakeStorage`/`fakeConsumer`/`fakeRunner` internal/pbsdr/manager_test.go (argv+stdin recorder) |
|
||||
| `capability.Runner` | internal/capability/probe.go | `*proxmox.ExecRunner` (RunnerDirect) | `fakeRunner` internal/capability/probe_test.go |
|
||||
| Cross-repo: local API ↔ controller | internal/localapi/server.go routes; contract seeded by internal/provision/doc.go (`bootstrap.json`: endpoint + leaf fingerprint + token) | felhom-controller's agentapi client | pin = served leaf cert (memory gotcha) |
|
||||
| Cross-repo: agent ↔ hub | internal/hub/report.go (`HostReport`), `ControlEnvelope`; POST `/api/v1/host-report` | hub mirrors structs field-for-field | new event/report fields need hub-side ingest changes |
|
||||
| Cross-repo: shipped host artifacts | configs/felhom-agent.sudoers, configs/felhom-mkfs-guarded.sh, shared-parent script/unit (inline in internal/localapi/intermediary.go) | deployed WITH the binary | sudoers globs must match `stageTemp` patterns + staging dirs exactly |
|
||||
| Cross-repo: shipped host artifacts | configs/felhom-agent.sudoers, configs/felhom-mkfs-guarded.sh, configs/felhom-pbs-apply, shared-parent script/unit (inline in internal/localapi/intermediary.go) | deployed WITH the binary | sudoers globs must match `stageTemp` patterns + staging dirs exactly |
|
||||
| Operator signing | internal/authz (OpBlob, SSHSIG) | cmd/felhom-opsign (offline CLI) | blob/verify tests in internal/authz |
|
||||
|
||||
## 5. Extension points (where new features plug in)
|
||||
|
||||
@@ -40,6 +40,7 @@ import (
|
||||
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbs"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
@@ -730,6 +731,27 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int {
|
||||
}
|
||||
}
|
||||
|
||||
// PBS DR tier apply-bridge (slice 2): hub-driven — runs whenever the desired-state carries a
|
||||
// pbs_dr descriptor (slice 1). No config gate: an absent/disabled descriptor is a no-op, so a
|
||||
// v0.80.0 rollout changes nothing until the operator enables the tier on the hub. Adoption
|
||||
// first (existing healthy entry → grant + mark, NO consume), verify-pin-before-consume,
|
||||
// set-only re-apply, consumed-but-failed is loud (see internal/pbsdr).
|
||||
var pbsdrLoop *pbsdr.Loop
|
||||
{
|
||||
pdMode := proxmox.RunnerMode(cfg.Privileged.Mode)
|
||||
if pdMode == "" {
|
||||
pdMode = proxmox.RunnerSudo
|
||||
}
|
||||
pdRunner := &proxmox.ExecRunner{Mode: pdMode, SudoPath: cfg.Privileged.SudoPath}
|
||||
secretDir := filepath.Dir(cfg.Backup.PBSSecretPath("x"))
|
||||
pdMgr := pbsdr.NewManager(pdRunner, px, client, cfg.WGTunnel.WithDefaults().StateDir,
|
||||
secretDir, cfg.SourcePath, logger)
|
||||
pbsdrLoop = pbsdr.NewLoop(pdMgr, 60*time.Second, logger)
|
||||
desiredSyncer.AddConsumer(pbsdrLoop) // raw desired-state → the pbs_dr block
|
||||
collector.SetPBSDRReporter(pbsdrLoop)
|
||||
logger.Info("pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)")
|
||||
}
|
||||
|
||||
// Run reconcile, the hub loop, the storage watchdog, the restore-test scheduler, the PBS
|
||||
// verify loop, (optionally) the local-API server, and (optionally) the LAN resolver loop
|
||||
// concurrently; any one returning ends the daemon (ctx cancel tears down the rest).
|
||||
@@ -739,6 +761,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int {
|
||||
go func() { errc <- watchdog.Run(ctx) }()
|
||||
go func() { errc <- scheduler.Run(ctx) }()
|
||||
go func() { errc <- pbsLoop.Run(ctx) }()
|
||||
go func() { errc <- pbsdrLoop.Run(ctx) }()
|
||||
if localSrv != nil {
|
||||
localServers = 1
|
||||
// Host-reboot remount fix: BEFORE binding into the guest, re-assert enrolled drive MOUNTS on the
|
||||
|
||||
@@ -196,6 +196,19 @@ Cmnd_Alias FELHOM_SSHD = \
|
||||
/usr/bin/systemctl reset-failed felhom-sshd, \
|
||||
/usr/bin/wg show wg-felhom latest-handshakes
|
||||
|
||||
# PBS DR tier apply (slice 2, SPIKE-pbs-tier-provisioning-2026-07-10 §2b). Storage-entry
|
||||
# lifecycle is /storage-ROOT-gated in the PVE API (spike Probe 1: create/modify/delete all check
|
||||
# Datastore.Allocate on /storage), so the agent token cannot do it — this wrapper is the pinned
|
||||
# vector. THE SET-ONLY LAW: the wrapper contains NO deletion path (entry deletion destroys the
|
||||
# client encryption key = un-decryptable backups); verbs are create/reconcile/grant only. The
|
||||
# token secret rides the wrapper's STDIN — sudo logs argv, so it must never appear here. The
|
||||
# agent fine-validates every field (charset + descriptor equality) before exec; these globs are
|
||||
# the coarse allowlist.
|
||||
Cmnd_Alias FELHOM_PBSDR = \
|
||||
/usr/local/sbin/felhom-pbs-apply create *, \
|
||||
/usr/local/sbin/felhom-pbs-apply reconcile *, \
|
||||
/usr/local/sbin/felhom-pbs-apply grant *
|
||||
|
||||
# OOB nft belt (TASK H1). The STATIC table `inet felhom_oob` is installed once by host-install; the
|
||||
# agent mutates ONLY its two SETS — @operator_ips (the operator /32) + @ssh_port (the claimed port).
|
||||
# SET ELEMENTS ONLY [trap 4]: NO `nft add rule`, NO `nft -f`, NO `flush ruleset/table` — a rule grant
|
||||
@@ -209,4 +222,4 @@ Cmnd_Alias FELHOM_OOB = \
|
||||
/usr/sbin/nft add element inet felhom_oob operator_ips *, \
|
||||
/usr/sbin/nft add element inet felhom_oob ssh_port *
|
||||
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
#!/bin/bash
|
||||
#===============================================================================
|
||||
# felhom-pbs-apply — the ONLY storage-entry path the felhom-agent sudoers permits for the PBS DR
|
||||
# tier (slice 2; SPIKE-pbs-tier-provisioning-2026-07-10 §2b). The guarded-mkfs shape: the agent's
|
||||
# in-process validation is the PRIMARY gate (descriptor-field charset + equality checks BEFORE
|
||||
# exec); this wrapper is the minimal, auditable second gate as root.
|
||||
#
|
||||
# THE SET-ONLY LAW (spike §4, data-loss class): `pvesm` entry deletion DESTROYS the client
|
||||
# encryption key file (<id>.enc = K) — un-decryptable backups. This wrapper therefore contains
|
||||
# NO deletion path of any kind, and a re-apply is `pvesm set`-only. Grep-assertable; do not add
|
||||
# a "cleanup" verb here, ever. Deprovision is a deliberate future operator op, not this tool.
|
||||
#
|
||||
# SECRET DISCIPLINE (spike §2b): sudo logs its full argv to auth.log → the PBS token secret
|
||||
# arrives on STDIN, never as an argument to this wrapper. Inside, it is passed to pvesm's
|
||||
# --password (root-local, transient ps exposure — the accepted spike posture); it is never
|
||||
# echoed, never written anywhere except by pvesm itself (the 0600 .pw store).
|
||||
#
|
||||
# Ops (non-secret args on argv):
|
||||
# create <id> <server> <datastore> <namespace> <token-id> <fingerprint> <secret-dir>
|
||||
# secret on stdin (required). Creates the pbs entry with --encryption-key autogen
|
||||
# (K born at /etc/pve/priv/storage/<id>.enc), then places .pw/.enc copies in
|
||||
# <secret-dir> when it differs (the §4b WARN-fix dir; escrow-create's PBSEncKeyPath
|
||||
# must find K there — spike §4 flag).
|
||||
# reconcile <id> <server> <namespace> <token-id> <fingerprint> <secret-dir>
|
||||
# secret on stdin (optional; empty = no credential change). `pvesm set` ONLY:
|
||||
# server/fingerprint (+ --password when a secret is fed, e.g. after a hub re-issue).
|
||||
# NOTE datastore is deliberately NOT settable, and namespace/token-id are accepted
|
||||
# for validation parity but NOT applied — tenancy identity is adopt-only (the
|
||||
# demo's live entry must never be repointed at a different namespace).
|
||||
# grant <id>
|
||||
# The Part-0-evidenced dual-grant: FelhomAgentStore on /storage/<id> to the agent
|
||||
# user AND token (privsep intersection). Datastore.Audit reads ride the base role.
|
||||
#===============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
die() { echo "felhom-pbs-apply: REFUSED: $*" >&2; exit 1; }
|
||||
|
||||
op="${1:-}"; id="${2:-}"
|
||||
[[ -n "$op" && -n "$id" ]] || die "usage: felhom-pbs-apply <create|reconcile|grant> <storage-id> ..."
|
||||
|
||||
# Storage id: PVE grammar, conservative. Also the ACL path component — no slashes possible.
|
||||
[[ "$id" =~ ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ ]] || die "bad storage id ($id)"
|
||||
|
||||
STORECFG=/etc/pve/storage.cfg
|
||||
PRIVDIR=/etc/pve/priv/storage
|
||||
|
||||
entry_exists() { grep -Eq "^pbs: ${id}\$" "$STORECFG"; }
|
||||
|
||||
val_server() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9.-]{0,62}$ ]] || die "bad server ($1)"; }
|
||||
val_datastore() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$ ]] || die "bad datastore ($1)"; }
|
||||
val_ns() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$ ]] || die "bad namespace ($1)"; }
|
||||
val_tok() { [[ "$1" =~ ^[A-Za-z0-9_.-]+@[A-Za-z0-9]+![A-Za-z0-9_.-]+$ ]] || die "bad token id ($1)"; }
|
||||
val_fp() { [[ "$1" =~ ^([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}$ ]] || die "bad fingerprint"; }
|
||||
val_sdir() {
|
||||
case "$1" in
|
||||
/etc/pve/priv/storage) : ;;
|
||||
/var/lib/felhom-agent/*) [[ "$1" != *..* ]] || die "secret-dir traversal" ;;
|
||||
*) die "secret-dir must be $PRIVDIR or under /var/lib/felhom-agent ($1)" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
read_secret() { # → SECRET (may be empty when optional)
|
||||
local s
|
||||
s=$(head -c 256 || true)
|
||||
s="${s%$'\n'}"; s="${s%$'\r'}"
|
||||
printf '%s' "$s"
|
||||
}
|
||||
|
||||
place_copies() { # secret-dir — the §4b WARN-fix placement (non-root agent can't read /etc/pve/priv)
|
||||
local sdir="$1"
|
||||
[[ "$sdir" == "$PRIVDIR" ]] && return 0
|
||||
install -d -o felhom-agent -g felhom-agent -m 0700 "$sdir"
|
||||
[[ -f "$PRIVDIR/$id.pw" ]] && install -o felhom-agent -g felhom-agent -m 0600 "$PRIVDIR/$id.pw" "$sdir/$id.pw"
|
||||
# K's copy: escrow-create stats PBSEncKeyPath(<secret-dir>/<id>.enc) — the spike §4 flag.
|
||||
[[ -f "$PRIVDIR/$id.enc" ]] && install -o root -g felhom-agent -m 0640 "$PRIVDIR/$id.enc" "$sdir/$id.enc"
|
||||
return 0
|
||||
}
|
||||
|
||||
case "$op" in
|
||||
create)
|
||||
[[ $# -eq 8 ]] || die "create needs 7 args: <id> <server> <datastore> <namespace> <token-id> <fingerprint> <secret-dir>"
|
||||
server="$3"; datastore="$4"; ns="$5"; tok="$6"; fp="$7"; sdir="$8"
|
||||
val_server "$server"; val_datastore "$datastore"; val_ns "$ns"; val_tok "$tok"; val_fp "$fp"; val_sdir "$sdir"
|
||||
entry_exists && die "entry $id already exists (reconcile is the re-apply path — set-only law)"
|
||||
SECRET=$(read_secret)
|
||||
[[ -n "$SECRET" ]] || die "create requires the token secret on stdin"
|
||||
pvesm add pbs "$id" \
|
||||
--server "$server" --datastore "$datastore" --namespace "$ns" \
|
||||
--username "$tok" --password "$SECRET" --fingerprint "$fp" \
|
||||
--content backup --encryption-key autogen >&2
|
||||
[[ -f "$PRIVDIR/$id.enc" ]] || die "pvesm add succeeded but K ($PRIVDIR/$id.enc) was not born"
|
||||
place_copies "$sdir"
|
||||
echo "felhom-pbs-apply: created $id (K born; encryption-key autogen)" >&2
|
||||
;;
|
||||
reconcile)
|
||||
[[ $# -eq 7 ]] || die "reconcile needs 6 args: <id> <server> <namespace> <token-id> <fingerprint> <secret-dir>"
|
||||
server="$3"; ns="$4"; tok="$5"; fp="$6"; sdir="$7"
|
||||
val_server "$server"; val_ns "$ns"; val_tok "$tok"; val_fp "$fp"; val_sdir "$sdir"
|
||||
entry_exists || die "entry $id does not exist (create is the fresh path)"
|
||||
SECRET=$(read_secret)
|
||||
args=(--server "$server" --fingerprint "$fp")
|
||||
[[ -n "$SECRET" ]] && args+=(--password "$SECRET")
|
||||
pvesm set "$id" "${args[@]}" >&2
|
||||
place_copies "$sdir"
|
||||
echo "felhom-pbs-apply: reconciled $id (set-only; tenancy identity untouched)" >&2
|
||||
;;
|
||||
grant)
|
||||
[[ $# -eq 2 ]] || die "grant takes only <id>"
|
||||
pveum acl modify "/storage/$id" --users felhom-agent@pve --roles FelhomAgentStore >&2
|
||||
pveum acl modify "/storage/$id" --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore >&2
|
||||
echo "felhom-pbs-apply: granted FelhomAgentStore on /storage/$id (user + token)" >&2
|
||||
;;
|
||||
*)
|
||||
die "unknown op ($op)"
|
||||
;;
|
||||
esac
|
||||
@@ -37,6 +37,10 @@ type Capability struct {
|
||||
// so even mkfs/pct-set entries are side-effect-free to probe.
|
||||
func Manifest() []Capability { return manifest }
|
||||
|
||||
// reprFingerprint is a shape-valid all-zero SHA-256 colon fingerprint for list-mode repr vectors
|
||||
// (matches the wrapper's fingerprint validation; never executed).
|
||||
const reprFingerprint = "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"
|
||||
|
||||
var manifest = []Capability{
|
||||
// ---- Intermediary drive model (the multi-drive path — mostly Critical) ----
|
||||
{"guest-init-pid", "drive-gate guest-sees check (multi-drive concurrency)", "/usr/bin/lxc-info", []string{"-n", "9201", "-p", "-H"}, true},
|
||||
@@ -114,6 +118,15 @@ var manifest = []Capability{
|
||||
{"wg-disable", "wg-quick@wg-felhom disable (revocation)", "/usr/bin/systemctl", []string{"disable", "--now", "wg-quick@wg-felhom"}, false},
|
||||
{"wg-handshake-read", "tunnel handshake-age read", "/usr/bin/wg", []string{"show", "wg-felhom", "latest-handshakes"}, true},
|
||||
|
||||
// ---- PBS DR tier apply (FELHOM_PBSDR, slice 2). NON-critical (the selfupdate rationale):
|
||||
// applying the tier is an occasional hub-driven provisioning op, not a steady-state serving
|
||||
// path — a degraded grant means "can't provision/reconcile the PBS entry" (the bridge reports
|
||||
// loudly anyway), not a serving outage. The steady-state backup path is covered by the wg +
|
||||
// storage capabilities. List-mode representations only; never executed. ----
|
||||
{"pbsdr-create", "PBS DR storage-entry create (K autogen)", "/usr/local/sbin/felhom-pbs-apply", []string{"create", "felhom-pbs", "10.77.0.1", "felhom-offsite", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false},
|
||||
{"pbsdr-reconcile", "PBS DR storage-entry reconcile (set-only)", "/usr/local/sbin/felhom-pbs-apply", []string{"reconcile", "felhom-pbs", "10.77.0.1", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false},
|
||||
{"pbsdr-grant", "PBS DR storage ACL self-grant", "/usr/local/sbin/felhom-pbs-apply", []string{"grant", "felhom-pbs"}, false},
|
||||
|
||||
// ---- Agent self-update (FELHOM_SELFUPDATE, D1). NON-critical: self-update is an occasional
|
||||
// operator-driven op, not a steady-state serving path — a degraded grant means "can't
|
||||
// self-update" (fall back to a manual SSH deploy), not a serving outage. The apply repr uses a
|
||||
|
||||
@@ -35,6 +35,10 @@ type Config struct {
|
||||
OOB OOBConfig `json:"oob"`
|
||||
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
|
||||
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
|
||||
|
||||
// SourcePath is the file this config was loaded from ("" = all-env). Set by Load, never
|
||||
// serialized — the pbsdr bridge's escrow.pbs_storage_id seed writes back to it.
|
||||
SourcePath string `json:"-"`
|
||||
}
|
||||
|
||||
// OOBConfig configures the dedicated felhom-sshd OOB access instance + belt (TASK H1). **Enabled
|
||||
@@ -540,6 +544,7 @@ func Load(path string) (Config, error) {
|
||||
return cfg, fmt.Errorf("config: parsing %s: %w", path, err)
|
||||
}
|
||||
}
|
||||
cfg.SourcePath = path // where this config came from (pbsdr's escrow seed writes back here)
|
||||
applyEnv(&cfg)
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
@@ -193,6 +193,50 @@ func (c *Client) RegisterWG(ctx context.Context, pubkey string) (*WGRegisterResp
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
// ErrNoPBSSecret is the typed "404: no unconsumed PBS token secret staged for this host" outcome
|
||||
// (PBS DR slice 2). Absent-or-already-consumed are indistinguishable by design (consume-once).
|
||||
var ErrNoPBSSecret = fmt.Errorf("hub: no unconsumed PBS token secret staged for this host")
|
||||
|
||||
// ConsumePBSToken fetches this host's one-time PBS token secret — EXACTLY ONCE (PBS DR slice 2;
|
||||
// POST /api/v1/hosts/{host_id}/pbs/consume-token, per-host key, self-scoped; NOTE the PLURAL
|
||||
// /hosts/ — the slice-1 route). A 200 burns the secret hub-side: the caller MUST apply it or
|
||||
// surface a loud consumed-but-failed state (never silent-retry). The secret is returned to the
|
||||
// caller only — never logged, never in an error.
|
||||
func (c *Client) ConsumePBSToken(ctx context.Context) (string, error) {
|
||||
if c.hostID == "" {
|
||||
return "", fmt.Errorf("hub: ConsumePBSToken requires a configured host_id")
|
||||
}
|
||||
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/pbs/consume-token"
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+c.apiKey)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
resp, err := c.hc.Do(req)
|
||||
if err != nil {
|
||||
return "", &TransportError{Err: err}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
|
||||
if resp.StatusCode == http.StatusNotFound {
|
||||
return "", ErrNoPBSSecret
|
||||
}
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return "", &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
|
||||
}
|
||||
var out struct {
|
||||
TokenSecret string `json:"token_secret"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &out); err != nil {
|
||||
return "", fmt.Errorf("hub: decoding consume-token response (body withheld — secret channel)")
|
||||
}
|
||||
if out.TokenSecret == "" {
|
||||
return "", fmt.Errorf("hub: consume-token returned an empty secret")
|
||||
}
|
||||
return out.TokenSecret, nil
|
||||
}
|
||||
|
||||
// JobWire is one queued signed-op job as served by GET /hosts/{id}/jobs (slice 10A). The blob is
|
||||
// OPAQUE to the hub — for slice 10B it is a base64 `SignedJobEnvelope` (op-blob + armored SSHSIG)
|
||||
// the agent verifies before executing.
|
||||
|
||||
@@ -55,6 +55,12 @@ type WireguardReporter interface {
|
||||
WireguardStatus(ctx context.Context) *WireguardStatus
|
||||
}
|
||||
|
||||
// PBSDRReporter is the slice-2 seam the pbsdr bridge loop plugs into (same consumer-side
|
||||
// pattern — hub does not import pbsdr). nil (feature not wired) → no pbs_dr stanza.
|
||||
type PBSDRReporter interface {
|
||||
PBSDRStatus(ctx context.Context) *PBSDRStatus
|
||||
}
|
||||
|
||||
// Collector builds a HostReport from read-only sources. All deps are behind narrow
|
||||
// interfaces for unit testing.
|
||||
type Collector struct {
|
||||
@@ -68,6 +74,7 @@ type Collector struct {
|
||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
|
||||
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
|
||||
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
|
||||
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
||||
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
|
||||
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
|
||||
@@ -127,6 +134,13 @@ func (c *Collector) SetWireguardReporter(w WireguardReporter) *Collector {
|
||||
return c
|
||||
}
|
||||
|
||||
// SetPBSDRReporter wires the PBS-DR-tier bridge state source (slice 2; nil-safe → stanza
|
||||
// omitted). Returns the collector for chaining.
|
||||
func (c *Collector) SetPBSDRReporter(p PBSDRReporter) *Collector {
|
||||
c.pbsdr = p
|
||||
return c
|
||||
}
|
||||
|
||||
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
|
||||
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
|
||||
type SelfUpdateReporter interface {
|
||||
@@ -204,6 +218,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
if c.wg != nil {
|
||||
report.Wireguard = c.wg.WireguardStatus(ctx)
|
||||
}
|
||||
// Slice 2: PBS DR tier bridge state (nil reporter = feature not wired → stanza omitted).
|
||||
if c.pbsdr != nil {
|
||||
report.PBSDR = c.pbsdr.PBSDRStatus(ctx)
|
||||
}
|
||||
// D1: agent self-update pending status (nil reporter → pending=false, the steady state).
|
||||
if c.selfUpdate != nil {
|
||||
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()
|
||||
|
||||
@@ -77,6 +77,14 @@ type HostReport struct {
|
||||
// hub-schema change and are absent when the reporter is not wired.
|
||||
MgmtPlane *MgmtPlaneStatus `json:"mgmt_plane,omitempty"`
|
||||
|
||||
// PBSDR is the PBS-DR-tier bridge status stanza (slice 2). Present only when the pbsdr
|
||||
// consumer is wired. `consumed_failed` is the LOUD persistent state: the one-time token
|
||||
// secret was consumed but the apply failed afterwards — the secret is burned, the bridge
|
||||
// will NOT silently retry, the operator must Re-issue on the hub. Stored opaquely hub-side
|
||||
// (the Wireguard precedent) — additive, no hub-schema change; hub rendering joins in slice 3.
|
||||
// Carries NO secret.
|
||||
PBSDR *PBSDRStatus `json:"pbs_dr,omitempty"`
|
||||
|
||||
// OOB is the operator-access health stanza (TASK H1). It answers the operator's question — "can I
|
||||
// get into this box right now, and if not, why" — from the hub: felhom-sshd up + on which port,
|
||||
// locally reachable, the tunnel handshake age (the OOB path rides wg-felhom), whether the operator
|
||||
@@ -86,6 +94,21 @@ type HostReport struct {
|
||||
OOB *OOBStatus `json:"oob,omitempty"`
|
||||
}
|
||||
|
||||
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
|
||||
// "adopted" (existing entry verified + reconciled, no consume), "applied" (fresh entry created,
|
||||
// K born), "waiting_secret" (verified but no unconsumed secret staged — retrying),
|
||||
// "verify_failed" (fingerprint/reachability pre-consume check failing — retrying, NOTHING
|
||||
// consumed), "consumed_failed" (LOUD: secret burned, apply failed, no auto-retry — operator
|
||||
// re-issue required), "disabled" (descriptor enabled:false). Carries no secret.
|
||||
type PBSDRStatus struct {
|
||||
State string `json:"state"`
|
||||
StorageID string `json:"storage_id,omitempty"`
|
||||
Namespace string `json:"namespace,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
ConsumedFailed bool `json:"consumed_failed,omitempty"`
|
||||
AppliedAt string `json:"applied_at,omitempty"` // RFC3339; set on adopted/applied
|
||||
}
|
||||
|
||||
// OOBStatus is the per-heartbeat operator-access health (TASK H1). Carries no secret.
|
||||
type OOBStatus struct {
|
||||
FelhomSshdActive bool `json:"felhom_sshd_active"` // the felhom-sshd unit is active
|
||||
@@ -372,6 +395,21 @@ type WireDesiredState struct {
|
||||
PBSNamespace string `json:"pbs_namespace,omitempty"`
|
||||
RestoreDirective *WireRestoreDirective `json:"restore_directive,omitempty"` // slice 10D (forward-compat)
|
||||
Wireguard *WireWireguard `json:"wireguard,omitempty"` // S3 (doc 06 §3.2; golden-pinned)
|
||||
PBSDR *WirePBSDR `json:"pbs_dr,omitempty"` // PBS DR tier (slice 2 consumer)
|
||||
}
|
||||
|
||||
// WirePBSDR is the hub's PBS-DR-tier descriptor (PBS DR slice 1, hub/internal/web/pbsdr.go
|
||||
// pbsDRDescriptor — field-exact, cross-repo). NON-SECRET by contract: the token secret NEVER
|
||||
// rides the desired-state; the agent fetches it consume-once via ConsumePBSToken. Absent/nil on
|
||||
// pre-v0.44.0 hubs → the pbsdr consumer no-ops (old-hub compat).
|
||||
type WirePBSDR struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
StorageID string `json:"storage_id,omitempty"`
|
||||
PBSTunnelIP string `json:"pbs_tunnel_ip,omitempty"`
|
||||
Datastore string `json:"datastore,omitempty"`
|
||||
Namespace string `json:"namespace,omitempty"`
|
||||
TokenID string `json:"token_id,omitempty"`
|
||||
Fingerprint string `json:"fingerprint,omitempty"`
|
||||
}
|
||||
|
||||
// WireWireguard is the hub-owned offsite-tunnel assignment (S3) — field-exact with the S2 golden
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
package pbs
|
||||
|
||||
// ProbeFingerprint (PBS DR slice 2) — the verify-pin-BEFORE-consume check: a bare TLS dial to
|
||||
// the PBS server pinned to the DESCRIPTOR's fingerprint. Success proves the box at that address
|
||||
// presents exactly the pinned cert (and is reachable over the tunnel); nothing is authenticated
|
||||
// and nothing is consumed. A mismatch or unreachability MUST abort the bridge before the
|
||||
// one-time token secret is touched (the offsite ordering law).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ProbeFingerprint dials server (host or host:port; default port 8007) and verifies the
|
||||
// presented leaf cert against fingerprint (sha256, colon-form ok). Returns nil only when the
|
||||
// pin matches exactly.
|
||||
func ProbeFingerprint(ctx context.Context, server, fingerprint string) error {
|
||||
tlsCfg, err := pinnedTLS(fingerprint)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
addr := server
|
||||
if !strings.Contains(addr, ":") {
|
||||
addr = net.JoinHostPort(addr, "8007")
|
||||
}
|
||||
d := net.Dialer{Timeout: 15 * time.Second}
|
||||
raw, err := d.DialContext(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pbs: fingerprint probe dial %s: %w", addr, err)
|
||||
}
|
||||
defer raw.Close()
|
||||
if dl, ok := ctx.Deadline(); ok {
|
||||
raw.SetDeadline(dl)
|
||||
} else {
|
||||
raw.SetDeadline(time.Now().Add(15 * time.Second))
|
||||
}
|
||||
conn := tls.Client(raw, tlsCfg)
|
||||
if err := conn.HandshakeContext(ctx); err != nil {
|
||||
return fmt.Errorf("pbs: fingerprint probe %s: %w", addr, err)
|
||||
}
|
||||
conn.Close()
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
package pbsdr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// Loop drives the Manager on its own cadence AND consumes fetched desired-state via the
|
||||
// desired.Syncer raw-consumer seam (the wgtunnel Loop shape). fetched=false ("no desired data
|
||||
// seen yet") is never a signal; an absent pbs_dr block on a PRESENT desired-state is a plain
|
||||
// no-op this slice (no teardown — deprovision is a deliberate future op).
|
||||
type Loop struct {
|
||||
mgr *Manager
|
||||
interval time.Duration
|
||||
logger *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
fetched bool
|
||||
block *hub.WirePBSDR
|
||||
|
||||
nudge chan struct{}
|
||||
}
|
||||
|
||||
// NewLoop builds the loop. interval defaults to 60s.
|
||||
func NewLoop(mgr *Manager, interval time.Duration, logger *slog.Logger) *Loop {
|
||||
if interval <= 0 {
|
||||
interval = 60 * time.Second
|
||||
}
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
return &Loop{mgr: mgr, interval: interval, logger: logger, nudge: make(chan struct{}, 1)}
|
||||
}
|
||||
|
||||
// OnDesiredState implements desired.RawConsumer: store the latest pbs_dr block (or its absence)
|
||||
// and nudge the loop. Non-blocking and panic-free by construction.
|
||||
func (l *Loop) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
|
||||
if resp == nil {
|
||||
return
|
||||
}
|
||||
l.mu.Lock()
|
||||
l.fetched = true
|
||||
l.block = resp.DesiredState.PBSDR
|
||||
l.mu.Unlock()
|
||||
select {
|
||||
case l.nudge <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func (l *Loop) snapshot() (bool, *hub.WirePBSDR) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.fetched, l.block
|
||||
}
|
||||
|
||||
// Run applies immediately, then on every tick or desired-state nudge, until ctx is cancelled.
|
||||
func (l *Loop) Run(ctx context.Context) error {
|
||||
fetched, block := l.snapshot()
|
||||
l.mgr.Apply(ctx, fetched, block)
|
||||
t := time.NewTicker(l.interval)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-t.C:
|
||||
case <-l.nudge:
|
||||
}
|
||||
fetched, block = l.snapshot()
|
||||
l.mgr.Apply(ctx, fetched, block)
|
||||
}
|
||||
}
|
||||
|
||||
// PBSDRStatus implements the hub collector's PBSDRReporter seam.
|
||||
func (l *Loop) PBSDRStatus(_ context.Context) *hub.PBSDRStatus {
|
||||
return l.mgr.Status()
|
||||
}
|
||||
@@ -0,0 +1,428 @@
|
||||
// Package pbsdr is the PBS-DR-tier apply-bridge (slice 2) — the host-side twin of the
|
||||
// controller's offsite apply-bridge. It consumes the hub's `pbs_dr` desired-state descriptor
|
||||
// (slice 1) and converges the box: the pbs storage entry exists, K exists, the agent token can
|
||||
// write to it, and the ceremony one-liner finds its storage id.
|
||||
//
|
||||
// The laws this package encodes (spike SPIKE-pbs-tier-provisioning-2026-07-10 + the offsite
|
||||
// bridge precedent — do not "simplify" any of them):
|
||||
//
|
||||
// - SET-ONLY: re-apply never removes the storage entry — entry deletion destroys K
|
||||
// (un-decryptable backups). The wrapper has no deletion verb; this package never asks for one.
|
||||
// - ADOPTION FIRST: an existing healthy entry is adopted non-destructively — verified, granted,
|
||||
// marked — with NO consume. Tenancy identity (namespace/username/datastore) is ENTRY-OWNED on
|
||||
// adoption; a descriptor that names a different namespace does not repoint a live entry (the
|
||||
// demo's manually-built felhom-offsite tenancy is the canonical case).
|
||||
// - VERIFY-PIN-BEFORE-CONSUME: the PBS fingerprint is probed over the tunnel against the
|
||||
// descriptor BEFORE the one-time secret is consumed (the offsite ordering law).
|
||||
// - CONSUMED-BUT-FAILED IS LOUD: after a consume, any failure lands in a persistent alarming
|
||||
// report state. The bridge never silently retries a burned secret; it recovers ONLY when the
|
||||
// operator stages a fresh one (hub Re-issue) — the next consume then succeeds.
|
||||
// - The secret rides the wrapper's STDIN (sudo logs argv) and is never logged.
|
||||
package pbsdr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pbs"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// WrapperPath is the pinned sudoers vector (configs/felhom-pbs-apply).
|
||||
const WrapperPath = "/usr/local/sbin/felhom-pbs-apply"
|
||||
|
||||
// StorageReader is the PVE read seam (satisfied by *proxmox.Client; tests fake it).
|
||||
type StorageReader interface {
|
||||
StorageEntry(ctx context.Context, id string) (*proxmox.StorageEntryConfig, bool, error)
|
||||
StorageActive(ctx context.Context, id string) (bool, error)
|
||||
}
|
||||
|
||||
// SecretConsumer is the hub consume-once seam (satisfied by *hub.Client; tests fake it).
|
||||
type SecretConsumer interface {
|
||||
ConsumePBSToken(ctx context.Context) (string, error)
|
||||
}
|
||||
|
||||
var storageIDRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_.-]{0,27}$`)
|
||||
var nameRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$`)
|
||||
var tokenIDRe = regexp.MustCompile(`^[A-Za-z0-9_.-]+@[A-Za-z0-9]+![A-Za-z0-9_.-]+$`)
|
||||
var fingerprintRe = regexp.MustCompile(`^([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}$`)
|
||||
|
||||
const markerName = "marker.json"
|
||||
const consumedFailedName = "consumed-failed.json"
|
||||
|
||||
// marker is the idempotency record: the descriptor hash last converged + how.
|
||||
type marker struct {
|
||||
Hash string `json:"hash"`
|
||||
State string `json:"state"` // "adopted" | "applied"
|
||||
AppliedAt string `json:"applied_at"`
|
||||
}
|
||||
|
||||
// consumedFailed is the LOUD persistent state: a one-time secret was burned and the apply failed.
|
||||
type consumedFailed struct {
|
||||
Hash string `json:"hash"`
|
||||
Message string `json:"message"`
|
||||
At string `json:"at"`
|
||||
}
|
||||
|
||||
// Manager converges the box toward the pbs_dr descriptor. All deps are seams for tests.
|
||||
type Manager struct {
|
||||
runner proxmox.Runner
|
||||
px StorageReader
|
||||
hub SecretConsumer
|
||||
stateDir string // <agent-state>/pbsdr
|
||||
secretDir string // cfg.Backup pbs secret dir (the wrapper's copy target)
|
||||
configPath string // agent.json — for the escrow.pbs_storage_id seed ("" = no seeding)
|
||||
logger *slog.Logger
|
||||
|
||||
// probeFP is the verify-pin-before-consume seam (default pbs.ProbeFingerprint).
|
||||
probeFP func(ctx context.Context, server, fingerprint string) error
|
||||
now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
status *hub.PBSDRStatus // latest snapshot for the report stanza
|
||||
}
|
||||
|
||||
// NewManager builds the bridge manager.
|
||||
func NewManager(runner proxmox.Runner, px StorageReader, hubc SecretConsumer, stateDir, secretDir, configPath string, logger *slog.Logger) *Manager {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
return &Manager{
|
||||
runner: runner, px: px, hub: hubc,
|
||||
stateDir: filepath.Join(stateDir, "pbsdr"), secretDir: secretDir, configPath: configPath,
|
||||
logger: logger,
|
||||
probeFP: pbs.ProbeFingerprint,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Manager) markerPath() string { return filepath.Join(m.stateDir, markerName) }
|
||||
func (m *Manager) consumedFailedPath() string { return filepath.Join(m.stateDir, consumedFailedName) }
|
||||
|
||||
func (m *Manager) setStatus(s *hub.PBSDRStatus) {
|
||||
m.mu.Lock()
|
||||
m.status = s
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
// Status returns the latest bridge state (the report stanza; nil before the first Apply).
|
||||
func (m *Manager) Status() *hub.PBSDRStatus {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.status
|
||||
}
|
||||
|
||||
// descriptorHash is the idempotency key: sha256 of the canonical (struct-ordered) JSON.
|
||||
func descriptorHash(b *hub.WirePBSDR) string {
|
||||
j, _ := json.Marshal(b)
|
||||
sum := sha256.Sum256(j)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
func (m *Manager) loadMarker() *marker {
|
||||
raw, err := os.ReadFile(m.markerPath())
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var mk marker
|
||||
if json.Unmarshal(raw, &mk) != nil || mk.Hash == "" {
|
||||
return nil
|
||||
}
|
||||
return &mk
|
||||
}
|
||||
|
||||
func (m *Manager) loadConsumedFailed() *consumedFailed {
|
||||
raw, err := os.ReadFile(m.consumedFailedPath())
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var cf consumedFailed
|
||||
if json.Unmarshal(raw, &cf) != nil {
|
||||
return nil
|
||||
}
|
||||
return &cf
|
||||
}
|
||||
|
||||
func (m *Manager) writeState(path string, v any) error {
|
||||
if err := os.MkdirAll(m.stateDir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
raw, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := path + ".tmp"
|
||||
if err := os.WriteFile(tmp, raw, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// validate checks every descriptor field BEFORE any exec (the fine gate under the coarse sudoers).
|
||||
func validate(b *hub.WirePBSDR) error {
|
||||
if !storageIDRe.MatchString(b.StorageID) {
|
||||
return fmt.Errorf("bad storage_id %q", b.StorageID)
|
||||
}
|
||||
if !nameRe.MatchString(b.Datastore) {
|
||||
return fmt.Errorf("bad datastore %q", b.Datastore)
|
||||
}
|
||||
if !nameRe.MatchString(b.Namespace) {
|
||||
return fmt.Errorf("bad namespace %q", b.Namespace)
|
||||
}
|
||||
if !tokenIDRe.MatchString(b.TokenID) {
|
||||
return fmt.Errorf("bad token_id %q", b.TokenID)
|
||||
}
|
||||
if !fingerprintRe.MatchString(b.Fingerprint) {
|
||||
return fmt.Errorf("bad fingerprint (want 32-pair colon sha256)")
|
||||
}
|
||||
if _, err := netip.ParseAddr(b.PBSTunnelIP); err != nil {
|
||||
return fmt.Errorf("bad pbs_tunnel_ip %q", b.PBSTunnelIP)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Apply converges toward the descriptor. fetched=false (no desired data yet) is never a signal.
|
||||
func (m *Manager) Apply(ctx context.Context, fetched bool, block *hub.WirePBSDR) {
|
||||
if !fetched {
|
||||
return
|
||||
}
|
||||
if block == nil {
|
||||
// Absent block: pre-slice-1 hub or tier never enabled — a silent no-op (old-hub compat).
|
||||
return
|
||||
}
|
||||
if !block.Enabled {
|
||||
m.setStatus(&hub.PBSDRStatus{State: "disabled", StorageID: block.StorageID, Namespace: block.Namespace})
|
||||
return // NO teardown in this slice — deprovision is a deliberate future op
|
||||
}
|
||||
if err := validate(block); err != nil {
|
||||
m.logger.Error("pbsdr: descriptor invalid; refusing", "err", err)
|
||||
m.setStatus(&hub.PBSDRStatus{State: "verify_failed", StorageID: block.StorageID, Namespace: block.Namespace,
|
||||
Message: "descriptor invalid: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
h := descriptorHash(block)
|
||||
cf := m.loadConsumedFailed()
|
||||
if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) {
|
||||
m.setStatus(&hub.PBSDRStatus{State: mk.State, StorageID: block.StorageID, Namespace: block.Namespace, AppliedAt: mk.AppliedAt})
|
||||
return // idempotent: this exact descriptor already converged
|
||||
}
|
||||
|
||||
entry, found, err := m.px.StorageEntry(ctx, block.StorageID)
|
||||
if err != nil {
|
||||
m.logger.Warn("pbsdr: storage-entry read failed (transient; retrying next tick)", "err", err)
|
||||
return
|
||||
}
|
||||
if found && entry.Type != "pbs" {
|
||||
msg := fmt.Sprintf("storage id %s exists with type %q (not pbs) — refusing to touch it", block.StorageID, entry.Type)
|
||||
m.logger.Error("pbsdr: " + msg)
|
||||
m.setStatus(&hub.PBSDRStatus{State: "verify_failed", StorageID: block.StorageID, Message: msg})
|
||||
return
|
||||
}
|
||||
|
||||
if found {
|
||||
active, err := m.px.StorageActive(ctx, block.StorageID)
|
||||
if err != nil {
|
||||
m.logger.Warn("pbsdr: storage status probe failed (transient)", "err", err)
|
||||
return
|
||||
}
|
||||
if active {
|
||||
m.adopt(ctx, block, entry, h)
|
||||
return
|
||||
}
|
||||
// Exists but unhealthy → the recovery path: verify → consume → reconcile-with-password.
|
||||
}
|
||||
|
||||
// VERIFY-PIN-BEFORE-CONSUME (the ordering law): a mismatched or unreachable PBS aborts here —
|
||||
// the one-time secret is untouched and the bridge simply retries next tick.
|
||||
if err := m.probeFP(ctx, block.PBSTunnelIP, block.Fingerprint); err != nil {
|
||||
m.logger.Warn("pbsdr: PBS fingerprint verify failed BEFORE consume (nothing consumed; retrying)", "err", err)
|
||||
m.setStatus(&hub.PBSDRStatus{State: "verify_failed", StorageID: block.StorageID, Namespace: block.Namespace,
|
||||
Message: "pre-consume fingerprint verify: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
secret, err := m.hub.ConsumePBSToken(ctx)
|
||||
if errors.Is(err, hub.ErrNoPBSSecret) {
|
||||
if cf != nil && cf.Hash == h {
|
||||
// The burned-secret dead-end: stay LOUD until the operator re-issues (fresh secret).
|
||||
m.setStatus(&hub.PBSDRStatus{State: "consumed_failed", StorageID: block.StorageID, Namespace: block.Namespace,
|
||||
ConsumedFailed: true, Message: cf.Message + " — awaiting operator re-issue (hub: Re-issue PBS credentials)"})
|
||||
return
|
||||
}
|
||||
m.setStatus(&hub.PBSDRStatus{State: "waiting_secret", StorageID: block.StorageID, Namespace: block.Namespace,
|
||||
Message: "verified; no unconsumed token secret staged on the hub"})
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
m.logger.Warn("pbsdr: consume-token failed (transient; nothing consumed hub-side on error)", "err", err)
|
||||
return
|
||||
}
|
||||
m.logger.Info("pbsdr: one-time token secret consumed (single-use; value withheld from logs)",
|
||||
"storage_id", block.StorageID, "secret_len", len(secret))
|
||||
|
||||
// From here the secret is BURNED — every failure below is the loud persistent state.
|
||||
if !found {
|
||||
_, errOut, err := m.runner.RunStdin(ctx, strings.NewReader(secret+"\n"), WrapperPath,
|
||||
"create", block.StorageID, block.PBSTunnelIP, block.Datastore, block.Namespace,
|
||||
block.TokenID, block.Fingerprint, m.secretDir)
|
||||
if err != nil {
|
||||
m.consumedFail(h, fmt.Sprintf("wrapper create failed: %v (stderr: %s)", err, tail(errOut)), block)
|
||||
return
|
||||
}
|
||||
} else {
|
||||
_, errOut, err := m.runner.RunStdin(ctx, strings.NewReader(secret+"\n"), WrapperPath,
|
||||
"reconcile", block.StorageID, block.PBSTunnelIP, block.Namespace,
|
||||
block.TokenID, block.Fingerprint, m.secretDir)
|
||||
if err != nil {
|
||||
m.consumedFail(h, fmt.Sprintf("wrapper reconcile failed: %v (stderr: %s)", err, tail(errOut)), block)
|
||||
return
|
||||
}
|
||||
}
|
||||
if _, errOut, err := m.runner.Run(ctx, WrapperPath, "grant", block.StorageID); err != nil {
|
||||
m.consumedFail(h, fmt.Sprintf("wrapper grant failed: %v (stderr: %s)", err, tail(errOut)), block)
|
||||
return
|
||||
}
|
||||
active, err := m.px.StorageActive(ctx, block.StorageID)
|
||||
if err != nil || !active {
|
||||
m.consumedFail(h, fmt.Sprintf("post-apply status probe failed (active=%v err=%v)", active, err), block)
|
||||
return
|
||||
}
|
||||
|
||||
m.finishConverged(block, h, "applied")
|
||||
}
|
||||
|
||||
// adopt is the non-destructive existing-entry path: NO consume, tenancy identity entry-owned.
|
||||
func (m *Manager) adopt(ctx context.Context, block *hub.WirePBSDR, entry *proxmox.StorageEntryConfig, h string) {
|
||||
if _, errOut, err := m.runner.Run(ctx, WrapperPath, "grant", block.StorageID); err != nil {
|
||||
// No secret involved — a grant failure is transient, retried next tick.
|
||||
m.logger.Warn("pbsdr: adoption grant failed (retrying next tick)", "err", err, "stderr", tail(errOut))
|
||||
return
|
||||
}
|
||||
note := ""
|
||||
if entry.Namespace != block.Namespace {
|
||||
note = fmt.Sprintf("adopted entry keeps its own tenancy (namespace %q; descriptor says %q — entry wins, never repointed)",
|
||||
entry.Namespace, block.Namespace)
|
||||
m.logger.Info("pbsdr: " + note)
|
||||
}
|
||||
m.finishConverged(block, h, "adopted")
|
||||
if note != "" {
|
||||
st := m.Status()
|
||||
st.Message = note
|
||||
st.Namespace = entry.Namespace // report the REAL tenancy
|
||||
m.setStatus(st)
|
||||
}
|
||||
}
|
||||
|
||||
// finishConverged writes the marker, clears any consumed-failed state, seeds the escrow storage
|
||||
// id, and publishes the converged status.
|
||||
func (m *Manager) finishConverged(block *hub.WirePBSDR, h, state string) {
|
||||
at := m.now().Format(time.RFC3339)
|
||||
if err := m.writeState(m.markerPath(), marker{Hash: h, State: state, AppliedAt: at}); err != nil {
|
||||
m.logger.Error("pbsdr: marker write failed (converged, but will re-run next tick)", "err", err)
|
||||
}
|
||||
os.Remove(m.consumedFailedPath())
|
||||
msg := ""
|
||||
if err := m.seedEscrowStorageID(block.StorageID); err != nil {
|
||||
msg = "escrow.pbs_storage_id seed failed: " + err.Error() + " (set it manually before the ceremony)"
|
||||
m.logger.Warn("pbsdr: " + msg)
|
||||
}
|
||||
m.logger.Info("pbsdr: converged", "state", state, "storage_id", block.StorageID)
|
||||
m.setStatus(&hub.PBSDRStatus{State: state, StorageID: block.StorageID, Namespace: block.Namespace,
|
||||
AppliedAt: at, Message: msg})
|
||||
}
|
||||
|
||||
// consumedFail records the LOUD persistent burned-secret state.
|
||||
func (m *Manager) consumedFail(h, msg string, block *hub.WirePBSDR) {
|
||||
m.logger.Error("pbsdr: CONSUMED-BUT-FAILED — the one-time secret is burned; NOT retrying silently. "+
|
||||
"Operator action: Re-issue PBS credentials on the hub.", "detail", msg, "storage_id", block.StorageID)
|
||||
if err := m.writeState(m.consumedFailedPath(), consumedFailed{Hash: h, Message: msg, At: m.now().Format(time.RFC3339)}); err != nil {
|
||||
m.logger.Error("pbsdr: consumed-failed state write failed", "err", err)
|
||||
}
|
||||
m.setStatus(&hub.PBSDRStatus{State: "consumed_failed", StorageID: block.StorageID, Namespace: block.Namespace,
|
||||
ConsumedFailed: true, Message: msg})
|
||||
}
|
||||
|
||||
// seedEscrowStorageID sets escrow.pbs_storage_id in agent.json when empty/absent — the bare
|
||||
// `--selftest=escrow-create` one-liner must find its storage with no flags. A different existing
|
||||
// value is NEVER clobbered (warn-and-keep). Unknown config keys are preserved verbatim
|
||||
// (map[string]RawMessage read-modify-write, atomic rename).
|
||||
func (m *Manager) seedEscrowStorageID(storageID string) error {
|
||||
if m.configPath == "" {
|
||||
return nil
|
||||
}
|
||||
raw, err := os.ReadFile(m.configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var doc map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return fmt.Errorf("parse %s: %w", m.configPath, err)
|
||||
}
|
||||
var esc map[string]json.RawMessage
|
||||
if cur, ok := doc["escrow"]; ok {
|
||||
if err := json.Unmarshal(cur, &esc); err != nil {
|
||||
return fmt.Errorf("parse escrow section: %w", err)
|
||||
}
|
||||
} else {
|
||||
esc = map[string]json.RawMessage{}
|
||||
}
|
||||
if cur, ok := esc["pbs_storage_id"]; ok {
|
||||
var existing string
|
||||
_ = json.Unmarshal(cur, &existing)
|
||||
if existing == storageID {
|
||||
return nil // already seeded
|
||||
}
|
||||
if existing != "" {
|
||||
m.logger.Warn("pbsdr: escrow.pbs_storage_id already set differently — keeping it",
|
||||
"existing", existing, "descriptor", storageID)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
idJSON, _ := json.Marshal(storageID)
|
||||
esc["pbs_storage_id"] = idJSON
|
||||
escJSON, err := json.Marshal(esc)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
doc["escrow"] = escJSON
|
||||
out, err := json.MarshalIndent(doc, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
st, err := os.Stat(m.configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := m.configPath + ".pbsdr-tmp"
|
||||
if err := os.WriteFile(tmp, out, st.Mode().Perm()); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(tmp, m.configPath); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
m.logger.Info("pbsdr: seeded escrow.pbs_storage_id (the ceremony one-liner needs no flags)", "storage_id", storageID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func tail(b []byte) string {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if len(s) > 300 {
|
||||
s = s[len(s)-300:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,402 @@
|
||||
package pbsdr
|
||||
|
||||
// The apply-bridge laws, each pinned by a non-hollow test (fake exec recorder — no docker/pct/
|
||||
// real /dev, the REUSE §4 doctrine): set-only re-apply, secret-on-stdin-never-argv,
|
||||
// verify-pin-BEFORE-consume, non-destructive adoption (no consume, tenancy entry-owned),
|
||||
// consumed-but-failed loud + recover-only-via-fresh-secret, marker idempotency, old-hub compat.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// recordedCall is one exec through the runner seam — argv AND the stdin bytes.
|
||||
type recordedCall struct {
|
||||
Name string
|
||||
Args []string
|
||||
Stdin string
|
||||
}
|
||||
|
||||
type fakeRunner struct {
|
||||
mu sync.Mutex
|
||||
calls []recordedCall
|
||||
// failVerb → error for calls whose first arg matches (e.g. "create").
|
||||
failVerb string
|
||||
failErr error
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Run(ctx context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
return f.RunStdin(ctx, nil, name, args...)
|
||||
}
|
||||
|
||||
func (f *fakeRunner) RunStdin(_ context.Context, stdin io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
var in []byte
|
||||
if stdin != nil {
|
||||
in, _ = io.ReadAll(stdin)
|
||||
}
|
||||
f.mu.Lock()
|
||||
f.calls = append(f.calls, recordedCall{Name: name, Args: args, Stdin: string(in)})
|
||||
f.mu.Unlock()
|
||||
if f.failVerb != "" && len(args) > 0 && args[0] == f.failVerb {
|
||||
return nil, []byte("boom-stderr"), f.failErr
|
||||
}
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
func (f *fakeRunner) recorded() []recordedCall {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]recordedCall(nil), f.calls...)
|
||||
}
|
||||
|
||||
type fakeStorage struct {
|
||||
entry *proxmox.StorageEntryConfig
|
||||
found bool
|
||||
active []bool // consumed per StorageActive call; last value repeats
|
||||
calls int
|
||||
}
|
||||
|
||||
func (f *fakeStorage) StorageEntry(context.Context, string) (*proxmox.StorageEntryConfig, bool, error) {
|
||||
return f.entry, f.found, nil
|
||||
}
|
||||
|
||||
func (f *fakeStorage) StorageActive(context.Context, string) (bool, error) {
|
||||
i := f.calls
|
||||
f.calls++
|
||||
if i >= len(f.active) {
|
||||
if len(f.active) == 0 {
|
||||
return false, nil
|
||||
}
|
||||
return f.active[len(f.active)-1], nil
|
||||
}
|
||||
return f.active[i], nil
|
||||
}
|
||||
|
||||
type fakeConsumer struct {
|
||||
secret string
|
||||
err error
|
||||
calls int
|
||||
}
|
||||
|
||||
func (f *fakeConsumer) ConsumePBSToken(context.Context) (string, error) {
|
||||
f.calls++
|
||||
if f.err != nil {
|
||||
return "", f.err
|
||||
}
|
||||
return f.secret, nil
|
||||
}
|
||||
|
||||
const testFP = "c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd"
|
||||
|
||||
func testBlock() *hub.WirePBSDR {
|
||||
return &hub.WirePBSDR{
|
||||
Enabled: true, StorageID: "felhom-pbs", PBSTunnelIP: "10.77.0.1",
|
||||
Datastore: "felhom-offsite", Namespace: "peti", TokenID: "felhom@pbs!peti",
|
||||
Fingerprint: testFP,
|
||||
}
|
||||
}
|
||||
|
||||
// newTestManager: fakes everywhere; the fingerprint probe defaults to PASS (override per test);
|
||||
// a real temp agent.json so the escrow seed is asserted end-to-end.
|
||||
func newTestManager(t *testing.T, r *fakeRunner, st *fakeStorage, c *fakeConsumer) (*Manager, string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
cfgPath := filepath.Join(dir, "agent.json")
|
||||
if err := os.WriteFile(cfgPath, []byte(`{"log_level":"info","escrow":{"posture":"zero_knowledge"},"custom_unknown":{"keep":1}}`), 0o600); err != nil {
|
||||
t.Fatalf("seed config: %v", err)
|
||||
}
|
||||
m := NewManager(r, st, c, dir, "/etc/pve/priv/storage", cfgPath,
|
||||
slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
m.probeFP = func(context.Context, string, string) error { return nil }
|
||||
return m, cfgPath
|
||||
}
|
||||
|
||||
func TestFreshPath_SecretOnStdinNeverArgv(t *testing.T) {
|
||||
r := &fakeRunner{}
|
||||
st := &fakeStorage{found: false, active: []bool{true}} // post-apply probe active
|
||||
c := &fakeConsumer{secret: "SUPER-SECRET"}
|
||||
m, cfgPath := newTestManager(t, r, st, c)
|
||||
|
||||
m.Apply(context.Background(), true, testBlock())
|
||||
|
||||
calls := r.recorded()
|
||||
if len(calls) != 2 || calls[0].Args[0] != "create" || calls[1].Args[0] != "grant" {
|
||||
t.Fatalf("calls = %+v, want [create, grant]", calls)
|
||||
}
|
||||
// THE STDIN LAW: the secret appears in NO argv, ONLY on the create call's stdin.
|
||||
// (Red-proof: pass it as an argument → this fails with the secret visible in Args.)
|
||||
for _, call := range calls {
|
||||
for _, a := range call.Args {
|
||||
if strings.Contains(a, "SUPER-SECRET") {
|
||||
t.Fatalf("secret leaked into argv: %v", call.Args)
|
||||
}
|
||||
}
|
||||
}
|
||||
if calls[0].Stdin != "SUPER-SECRET\n" {
|
||||
t.Errorf("create stdin = %q, want the secret + newline", calls[0].Stdin)
|
||||
}
|
||||
if calls[1].Stdin != "" {
|
||||
t.Errorf("grant received stdin %q", calls[1].Stdin)
|
||||
}
|
||||
// Non-secret coords ride argv, descriptor-exact.
|
||||
want := []string{"create", "felhom-pbs", "10.77.0.1", "felhom-offsite", "peti", "felhom@pbs!peti", testFP, "/etc/pve/priv/storage"}
|
||||
if fmt.Sprint(calls[0].Args) != fmt.Sprint(want) {
|
||||
t.Errorf("create argv = %v, want %v", calls[0].Args, want)
|
||||
}
|
||||
if c.calls != 1 {
|
||||
t.Errorf("consume calls = %d, want 1", c.calls)
|
||||
}
|
||||
// Converged: marker applied + escrow seeded + unknown config keys preserved.
|
||||
if s := m.Status(); s == nil || s.State != "applied" {
|
||||
t.Fatalf("status = %+v, want applied", s)
|
||||
}
|
||||
raw, _ := os.ReadFile(cfgPath)
|
||||
var doc map[string]json.RawMessage
|
||||
json.Unmarshal(raw, &doc)
|
||||
var esc map[string]string
|
||||
json.Unmarshal(doc["escrow"], &esc)
|
||||
if esc["pbs_storage_id"] != "felhom-pbs" {
|
||||
t.Errorf("escrow not seeded: %s", doc["escrow"])
|
||||
}
|
||||
if esc["posture"] != "zero_knowledge" {
|
||||
t.Errorf("existing escrow fields clobbered: %s", doc["escrow"])
|
||||
}
|
||||
if _, ok := doc["custom_unknown"]; !ok {
|
||||
t.Error("unknown config key dropped by the seed write")
|
||||
}
|
||||
if strings.Contains(string(raw), "SUPER-SECRET") {
|
||||
t.Error("secret leaked into agent.json")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyPinBeforeConsume(t *testing.T) {
|
||||
r := &fakeRunner{}
|
||||
st := &fakeStorage{found: false}
|
||||
c := &fakeConsumer{secret: "S"}
|
||||
m, _ := newTestManager(t, r, st, c)
|
||||
m.probeFP = func(context.Context, string, string) error { return errors.New("pin mismatch") }
|
||||
|
||||
m.Apply(context.Background(), true, testBlock())
|
||||
|
||||
// THE ORDERING LAW: a failing pre-consume verify means NOTHING consumed, NOTHING executed.
|
||||
// (Red-proof: reorder consume before the probe → calls=1 → FAIL.)
|
||||
if c.calls != 0 {
|
||||
t.Fatalf("consume calls = %d, want 0 — the secret was touched before the fingerprint verify", c.calls)
|
||||
}
|
||||
if len(r.recorded()) != 0 {
|
||||
t.Fatalf("runner calls = %+v, want none", r.recorded())
|
||||
}
|
||||
if s := m.Status(); s == nil || s.State != "verify_failed" {
|
||||
t.Fatalf("status = %+v, want verify_failed", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdoption_HealthyEntryNoConsumeTenancyEntryOwned(t *testing.T) {
|
||||
r := &fakeRunner{}
|
||||
st := &fakeStorage{
|
||||
found: true,
|
||||
entry: &proxmox.StorageEntryConfig{Storage: "felhom-offsite", Type: "pbs",
|
||||
Server: "10.77.0.1", Datastore: "felhom-offsite", Namespace: "demo-felhom-01",
|
||||
Username: "felhom@pbs!demo-felhom-01", Fingerprint: testFP},
|
||||
active: []bool{true},
|
||||
}
|
||||
c := &fakeConsumer{secret: "STAGED-BUT-MUST-STAY"}
|
||||
m, _ := newTestManager(t, r, st, c)
|
||||
|
||||
block := testBlock()
|
||||
block.StorageID = "felhom-offsite"
|
||||
block.Namespace = "demo" // the hub-provisioned tenancy differs — the entry must WIN
|
||||
m.Apply(context.Background(), true, block)
|
||||
|
||||
if c.calls != 0 {
|
||||
t.Fatalf("adoption consumed the staged secret (%d calls) — the no-consume law", c.calls)
|
||||
}
|
||||
calls := r.recorded()
|
||||
if len(calls) != 1 || calls[0].Args[0] != "grant" {
|
||||
t.Fatalf("adoption calls = %+v, want exactly [grant]", calls)
|
||||
}
|
||||
s := m.Status()
|
||||
if s == nil || s.State != "adopted" {
|
||||
t.Fatalf("status = %+v, want adopted", s)
|
||||
}
|
||||
if s.Namespace != "demo-felhom-01" {
|
||||
t.Errorf("reported namespace = %q, want the ENTRY's (demo-felhom-01) — tenancy is entry-owned", s.Namespace)
|
||||
}
|
||||
if !strings.Contains(s.Message, "entry wins") {
|
||||
t.Errorf("adoption note missing from message: %q", s.Message)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSetOnlyLaw pins the data-loss-class guard twice over: (1) the re-apply path over an
|
||||
// existing entry uses `reconcile` (pvesm set) — never create, never any deletion verb; (2) the
|
||||
// wrapper script itself contains no deletion path (grep gate over the shipped file).
|
||||
// Red-proof: introduce a remove+re-add path → the recorded verbs change → FAIL.
|
||||
func TestSetOnlyLaw(t *testing.T) {
|
||||
r := &fakeRunner{}
|
||||
st := &fakeStorage{
|
||||
found: true,
|
||||
entry: &proxmox.StorageEntryConfig{Storage: "felhom-pbs", Type: "pbs", Namespace: "peti", Fingerprint: testFP},
|
||||
// unhealthy → recovery path (verify → consume → reconcile) → post-apply healthy
|
||||
active: []bool{false, true},
|
||||
}
|
||||
c := &fakeConsumer{secret: "FRESH"}
|
||||
m, _ := newTestManager(t, r, st, c)
|
||||
|
||||
m.Apply(context.Background(), true, testBlock())
|
||||
|
||||
calls := r.recorded()
|
||||
if len(calls) != 2 || calls[0].Args[0] != "reconcile" || calls[1].Args[0] != "grant" {
|
||||
t.Fatalf("re-apply calls = %+v, want [reconcile, grant] (set-only)", calls)
|
||||
}
|
||||
deletionish := regexp.MustCompile(`remove|delete|destroy`)
|
||||
for _, call := range calls {
|
||||
for _, a := range call.Args {
|
||||
if deletionish.MatchString(a) {
|
||||
t.Fatalf("deletion-class verb reached the wrapper: %v — K destruction path", call.Args)
|
||||
}
|
||||
}
|
||||
}
|
||||
if calls[0].Stdin != "FRESH\n" {
|
||||
t.Errorf("reconcile stdin = %q, want the fresh secret (re-issue recovery)", calls[0].Stdin)
|
||||
}
|
||||
|
||||
// (2) the wrapper file: no deletion path, grep-assertable (the spike's set-only law).
|
||||
wrapper, err := os.ReadFile(filepath.Join("..", "..", "configs", "felhom-pbs-apply"))
|
||||
if err != nil {
|
||||
t.Fatalf("read wrapper: %v", err)
|
||||
}
|
||||
if regexp.MustCompile(`pvesm (remove|delete)`).Match(wrapper) {
|
||||
t.Fatal("configs/felhom-pbs-apply contains a pvesm deletion verb — the set-only law is dead")
|
||||
}
|
||||
if regexp.MustCompile(`rm\s+.*\.enc`).Match(wrapper) {
|
||||
t.Fatal("configs/felhom-pbs-apply deletes a .enc file — K destruction path")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConsumedButFailed_LoudAndRecoversOnlyViaFreshSecret(t *testing.T) {
|
||||
r := &fakeRunner{failVerb: "create", failErr: errors.New("pvesm add exploded")}
|
||||
st := &fakeStorage{found: false}
|
||||
c := &fakeConsumer{secret: "BURNED"}
|
||||
m, _ := newTestManager(t, r, st, c)
|
||||
block := testBlock()
|
||||
|
||||
// 1. consume + create fails → LOUD persistent state.
|
||||
m.Apply(context.Background(), true, block)
|
||||
s := m.Status()
|
||||
if s == nil || s.State != "consumed_failed" || !s.ConsumedFailed {
|
||||
t.Fatalf("status = %+v, want consumed_failed", s)
|
||||
}
|
||||
if _, err := os.Stat(m.consumedFailedPath()); err != nil {
|
||||
t.Fatal("consumed-failed state file not written")
|
||||
}
|
||||
|
||||
// 2. next ticks WITHOUT a fresh secret: no silent recovery, state stays loud.
|
||||
c.err = hub.ErrNoPBSSecret
|
||||
m.Apply(context.Background(), true, block)
|
||||
if s := m.Status(); s == nil || s.State != "consumed_failed" {
|
||||
t.Fatalf("status after no-secret retry = %+v, want consumed_failed (never quiet waiting)", s)
|
||||
}
|
||||
|
||||
// 3. operator re-issue staged a FRESH secret → the bridge recovers on the next tick.
|
||||
c.err = nil
|
||||
c.secret = "FRESH-AFTER-REISSUE"
|
||||
r.failVerb = ""
|
||||
st.active = []bool{true}
|
||||
m.Apply(context.Background(), true, block)
|
||||
if s := m.Status(); s == nil || s.State != "applied" {
|
||||
t.Fatalf("status after re-issue = %+v, want applied", s)
|
||||
}
|
||||
if _, err := os.Stat(m.consumedFailedPath()); !os.IsNotExist(err) {
|
||||
t.Error("consumed-failed state not cleared after recovery")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMarkerIdempotency(t *testing.T) {
|
||||
r := &fakeRunner{}
|
||||
st := &fakeStorage{found: false, active: []bool{true}}
|
||||
c := &fakeConsumer{secret: "S"}
|
||||
m, _ := newTestManager(t, r, st, c)
|
||||
block := testBlock()
|
||||
|
||||
m.Apply(context.Background(), true, block)
|
||||
n := len(r.recorded())
|
||||
m.Apply(context.Background(), true, block) // same descriptor hash → pure no-op
|
||||
if len(r.recorded()) != n || c.calls != 1 {
|
||||
t.Fatalf("re-apply over an unchanged descriptor ran ops (calls %d→%d, consume %d)", n, len(r.recorded()), c.calls)
|
||||
}
|
||||
if s := m.Status(); s == nil || s.State != "applied" || s.AppliedAt == "" {
|
||||
t.Fatalf("status = %+v, want applied with applied_at", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOldHubAndDisabledCompat(t *testing.T) {
|
||||
r := &fakeRunner{}
|
||||
c := &fakeConsumer{secret: "S"}
|
||||
m, _ := newTestManager(t, r, &fakeStorage{}, c)
|
||||
|
||||
m.Apply(context.Background(), false, nil) // no desired data yet
|
||||
m.Apply(context.Background(), true, nil) // pre-slice-1 hub: no pbs_dr key
|
||||
if len(r.recorded()) != 0 || c.calls != 0 || m.Status() != nil {
|
||||
t.Fatalf("nil-block Apply had effects (runner %d, consume %d, status %+v)", len(r.recorded()), c.calls, m.Status())
|
||||
}
|
||||
|
||||
m.Apply(context.Background(), true, &hub.WirePBSDR{Enabled: false, StorageID: "felhom-pbs"})
|
||||
if len(r.recorded()) != 0 || c.calls != 0 {
|
||||
t.Fatal("disabled descriptor ran ops (teardown is not this slice)")
|
||||
}
|
||||
if s := m.Status(); s == nil || s.State != "disabled" {
|
||||
t.Fatalf("status = %+v, want disabled", s)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWireFieldNames pins the cross-repo descriptor contract (hub/internal/web/pbsdr.go
|
||||
// pbsDRDescriptor): the exact JSON the hub writes must land in WirePBSDR field-for-field.
|
||||
func TestWireFieldNames(t *testing.T) {
|
||||
hubJSON := `{"desired_state":{"guests":[],"pbs_dr":{"enabled":true,"storage_id":"felhom-pbs",` +
|
||||
`"pbs_tunnel_ip":"10.77.0.1","datastore":"felhom-offsite","namespace":"peti",` +
|
||||
`"token_id":"felhom@pbs!peti","fingerprint":"aa:bb"}},"generation":7}`
|
||||
var resp hub.DesiredStateResponse
|
||||
if err := json.Unmarshal([]byte(hubJSON), &resp); err != nil {
|
||||
t.Fatalf("unmarshal: %v", err)
|
||||
}
|
||||
b := resp.DesiredState.PBSDR
|
||||
if b == nil || !b.Enabled || b.StorageID != "felhom-pbs" || b.PBSTunnelIP != "10.77.0.1" ||
|
||||
b.Datastore != "felhom-offsite" || b.Namespace != "peti" ||
|
||||
b.TokenID != "felhom@pbs!peti" || b.Fingerprint != "aa:bb" {
|
||||
t.Fatalf("wire mapping wrong: %+v", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEscrowSeed_NeverClobbersDifferentValue(t *testing.T) {
|
||||
r := &fakeRunner{}
|
||||
st := &fakeStorage{found: false, active: []bool{true}}
|
||||
c := &fakeConsumer{secret: "S"}
|
||||
m, cfgPath := newTestManager(t, r, st, c)
|
||||
os.WriteFile(cfgPath, []byte(`{"escrow":{"pbs_storage_id":"operator-set-id"}}`), 0o600)
|
||||
|
||||
m.Apply(context.Background(), true, testBlock())
|
||||
|
||||
raw, _ := os.ReadFile(cfgPath)
|
||||
var doc struct {
|
||||
Escrow struct {
|
||||
PBSStorageID string `json:"pbs_storage_id"`
|
||||
} `json:"escrow"`
|
||||
}
|
||||
json.Unmarshal(raw, &doc)
|
||||
if doc.Escrow.PBSStorageID != "operator-set-id" {
|
||||
t.Fatalf("an operator-set escrow.pbs_storage_id was clobbered: %s", raw)
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Read-only query operations. All API-backed (Datastore.Audit / VM.Audit /
|
||||
@@ -108,6 +109,50 @@ func (c *Client) NodeStorage(ctx context.Context) ([]Storage, error) {
|
||||
return ss, c.get(ctx, "/nodes/"+c.node+"/storage", &ss)
|
||||
}
|
||||
|
||||
// StorageEntryConfig is the storage-entry CONFIG as served by GET /storage/{id} (PBS DR slice 2
|
||||
// adoption probe — the pbs-type fields the bridge compares against the descriptor). All fields
|
||||
// non-secret; the token secret lives in /etc/pve/priv/storage/<id>.pw, never in this response.
|
||||
type StorageEntryConfig struct {
|
||||
Storage string `json:"storage"`
|
||||
Type string `json:"type"`
|
||||
Server string `json:"server,omitempty"`
|
||||
Datastore string `json:"datastore,omitempty"`
|
||||
Namespace string `json:"namespace,omitempty"`
|
||||
Username string `json:"username,omitempty"`
|
||||
Fingerprint string `json:"fingerprint,omitempty"`
|
||||
EncryptionKey string `json:"encryption-key,omitempty"` // K's OWN fingerprint (not the key)
|
||||
}
|
||||
|
||||
// StorageEntry returns GET /storage/{id} — the entry's configuration, or found=false when the id
|
||||
// does not exist (PVE answers HTTP 500 "does not exist" — mapped here so the adoption probe can
|
||||
// branch without string-matching upstream).
|
||||
func (c *Client) StorageEntry(ctx context.Context, id string) (*StorageEntryConfig, bool, error) {
|
||||
var e StorageEntryConfig
|
||||
err := c.get(ctx, "/storage/"+url.PathEscape(id), &e)
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "does not exist") {
|
||||
return nil, false, nil
|
||||
}
|
||||
return nil, false, err
|
||||
}
|
||||
return &e, true, nil
|
||||
}
|
||||
|
||||
// StorageActive returns whether GET /nodes/{node}/storage/{id}/status reports the entry active —
|
||||
// for a pbs entry that means PVE connected to the PBS with the stored credentials (the
|
||||
// post-apply/adoption health probe).
|
||||
func (c *Client) StorageActive(ctx context.Context, id string) (bool, error) {
|
||||
var st struct {
|
||||
Active int `json:"active"`
|
||||
Enabled int `json:"enabled"`
|
||||
}
|
||||
path := fmt.Sprintf("/nodes/%s/storage/%s/status", c.node, url.PathEscape(id))
|
||||
if err := c.get(ctx, path, &st); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return st.Active == 1, nil
|
||||
}
|
||||
|
||||
// StorageContent returns GET /nodes/{node}/storage/{store}/content (e.g. vzdump
|
||||
// archives + CT templates available for a restore).
|
||||
func (c *Client) StorageContent(ctx context.Context, store string) ([]StorageContent, error) {
|
||||
|
||||
Reference in New Issue
Block a user