56 Commits

Author SHA1 Message Date
admin cd6e26785a v0.120.0 — one data volume (R-165, decision D-a, variant V-c)
gates / gates (push) Successful in 5s
build-golden.sh 2.1.0 -> 3.0.0: a layout change is a major. The golden ships
ONE data volume at a NEUTRAL path (/var/lib/felhom); both /var/lib/docker
and /mnt/sys_drive are binds of subdirectories of it. mp1 is gone.

The variant was chosen on measurement. Three candidates were built and
rebooted (SPIKE-r165-phase0-2026-08-03.md); all three boot, reboot 3/3, give
ONE df figure and keep a container's statfs(/) on the merged volume — the
ordering worry that motivated the probe did not materialise. They differ
only in which guarantee they break: volume-at-docker puts customer backups
inside Docker's data-root; volume-at-sys_drive puts Docker's ENTIRE
data-root under /mnt, which the controller container mounts wholesale
(measured: it then sees /mnt/sys_drive/docker). V-c breaks neither.

The four assertions were RETARGETED, never deleted, and each was RUN against
a deliberately wrong shape — a real split guest and a real archive of it:
8 checks, 8 passed. A new 2b asserts both paths are ONE filesystem, which
catches the S2 shape the spike ranked worse than the split. Assertion 5
replaces the old "was mp1 excluded?" guard, whose pattern could no longer
match — a guard that cannot match has silently stopped guarding.

Provisioning: one volume, one grow. SysDataGrowGB is FOLDED IN rather than
dropped, because a census established that felhom-host-install.sh passes
-sysdata-grow and the two do not upgrade in the same instant; dropping it
would silently shrink every appliance by 42 of 250 GiB. The flags stay
accepted for the same reason. The existing test was retargeted to pin the
fold, and it caught the change before I did.
2026-08-03 06:43:38 +02:00
admin 587dbb43fe docs: CHANGELOG + REPORT for the CI workflow (no version bump)
gates / gates (push) Successful in 6s
2026-08-02 16:35:44 +02:00
admin eb99144509 ci: run the gate entry point on every push (R-168)
gates / gates (push) Successful in 6s
Same shape as the other repos. CI clones felhom.eu as a sibling because the shared
reuse_refs_check lives there and this repo's REUSE.md cites hub/internal/store/dr_recipe.go,
which lives in the hub. No uses: step, no version bump, nothing built or deployed.
2026-08-02 16:27:27 +02:00
admin 2c4efed5de REPORT: gate enforcement session (no version bump) 2026-08-02 15:37:04 +02:00
admin 75245a467c docs: CHANGELOG for the gate entry point (no version bump) 2026-08-02 15:28:42 +02:00
admin 054e85a2bf gates: one entry point (scripts/agent_gates.py) + pre-push hook
A census of all thirteen gate scripts across the four felhom repos on 2026-08-02 found that
every check a CLAUDE.md names was passing and two of the four nobody is told to run were
failing. This repo was the extreme case: nothing ran against it at all, and its REUSE.md — 90
cited paths — was checked by no one.

agent_gates.py exists at ONE gate on purpose, so the agent is not the one repo with nowhere to
put a check and so the pre-push hook has the same entry point in all four repos. It grows when
the agent grows a second gate. The shared reuse checker stays in felhom.eu/scripts/ and is
invoked across the workspace — never copied here; an absent sibling clone FAILS the gate and
prints the path tried, which test_agent_gates.py pins by running the entry point from a lone
directory with no sibling.

.githooks/pre-push runs it with --fast and refuses the push. Per-clone and --no-verify-able,
both stated in the hook itself; a manual run WARNS when the clone is unarmed.

Tooling only: no Go change, no build, no deploy, no version bump.
2026-08-02 15:22:58 +02:00
admin 4663df7ff3 REPORT: agent v0.119.0 — host addresses, deployed + published + vouched 2026-07-31 08:54:50 +02:00
admin 14642e3c7b v0.119.0 — the host report carries the box's addresses
A managed box's IP was invisible in every operator surface because nothing
reported one: HostMetrics carried node/cpu/mem/disk/load/uptime/temp/wrapper-sha
and no address of any kind. The hub could not show a host's LAN IP anywhere.

Two things that looked like the answer are traps, both checked before writing
code: lan_resolver.host_ip is an OPTIONAL config value absent unless that feature
is configured, and DeriveHostIP(local_api.listen_addr) returns 169.254.253.1 —
since R-50 the local API binds a link-local address identical on every box. Both
would have produced a confident wrong answer.

New wire field addresses[], one entry per (interface, address). Deliberately
iface+cidr rather than a single lan_ip: a Proxmox host legitimately holds several
(management bridge, tailnet, WG tunnel) and picking one to call "the" LAN IP is a
guess the agent is not entitled to make — silently wrong on a box whose bridge is
not vmbr0. The agent reports what exists; the hub does the labelling.

The filter is one predicate, chosen by MEASURING both demo hosts rather than by
reasoning about interface names. IsGlobalUnicast() alone drops loopback, IPv6
link-local (one per bridge, pure noise) and IPv4 link-local (169.254/16 — exactly
the island address above). It needs no veth/fwbr/tap denylist: that per-guest
plumbing carries no IP at all and self-excludes, verified on both boxes.

No new privilege and no block I/O — net.Interfaces() is a netlink/procfs read, so
the sudoers fence is untouched and the health-check rule is honoured.

The seam DEFAULTS to the real enumerator, inverting the nil-reporter-means-off
convention: this stanza has no config gate, so a forgotten wiring call would have
shipped it silently empty — the inert-seam failure recorded four times here.

Cross-repo: the golden is duplicated byte-identically in felhom.eu and the
contract test fails on top-level key drift, so both goldens moved together and
addresses[0]'s key set is asserted bidirectionally. The field marshals as [],
never null — the repo's own no-nulls invariant caught that on the first run.

Tests +9; three red-proofs (global-unicast filter, down-interface guard, inert
collectAddresses) each run, observed failing, and reverted.
2026-07-31 08:40:58 +02:00
admin 6b5dade4dc R-106 follow-up: mergeConfig dropped the pbs namespace, so v0.118.0's fix was inert (v0.118.1)
Live validation caught what the tests could not. On demo-felhom the recipe read
namespace "root" with namespace_state "resolved" — confident and wrong, a worse
shape than the original defect.

mergeConfig overlays the cluster storage config onto the node entry through a
hand-listed set of fields and Namespace was not among them. NodeStorage does not
return the namespace at all, so PBSNamespace always read "" and latestPBSCoord
correctly treated that as the root namespace.

Every v0.118.0 test built StorageTarget values directly — including the two
through Collector.Collect(), which inject a fakeObserver — so nothing crossed the
merge. Two new tests drive the real Observe path with PVE's actual split returns
and table the merge itself. Red-proof: dropping the added line fails both.

Suite rc=0, 29 packages, 0 FAIL.
2026-07-30 13:23:21 +02:00
admin 1c8a67eece R-106 + R-109: the DR recipe records the resolved namespace and names the backup target (v0.118.0)
Both defects were live on both demo boxes: the recipe said namespace "root" while
storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup
dir storages holds the local archives.

R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once
the list is namespace-scoped, so it was always empty and normalised to "root". It now
resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes
PVE read, so the recipe cannot disagree with the backup.

R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the
function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1
from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not
agent.json on disk — a target move rewrites the file and deliberately does not restart.

Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason,
never a default, an empty string, or a placeholder.

Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target
would otherwise be stored intact and dropped before any operator saw it.

9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
2026-07-30 13:11:08 +02:00
admin 1913e12031 docs(R-117): REPORT.md — v0.117.0 shipped and proven live on demo-hp
Predicate validated on hardware for both dead states (stale-device and
filesystem-aborted), 340-497us per call, no block I/O proven by strace (only
/proc/self/mountinfo, 0 statfs). No regression through the real pipeline: the
live backup-target drive reads bound_under_parent=True via GET /disks with the
controller's own credential.

Records what was NOT covered: the stale-bind repair on hardware (StablePathForRaw
hardcodes the live parent, so it would write into guest 9201's namespace -
R-117h), and sustained-load behaviour, still unmeasured.
2026-07-30 12:42:09 +02:00
admin 966d8f41ff v0.117.0 — R-117: the liveness signal now tests liveness
BoundUnderParent reported a namespace that returned EIO on every read and write
as healthy, and the gate restarted the customer's apps onto it. Both existing
terms parse a mountinfo line and then test only fields[4], the mount POINT.
Field 3 — major:minor — sat in the same parsed slice and was discarded.

Measured on hardware: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb with `shutdown`,
bound_under_parent true, EIO both directions, and the controller taking its
Return branch and emailing backup_target_restored with no alarm on any channel.

BoundUnderParent gains a third term at both /disks construction sites. The new
bindLiveness reads /proc only and asks two questions: the bind must name the
same device as the raw mount, and the filesystem must not have aborted (ext4
`shutdown` or `emergency_ro`).

The second check is not optional. A device that fails WITHOUT disappearing gives
the identical all-signals-healthy state with the devnos EQUAL and the drive never
Disconnected, so the gate produces neither a Stop nor a Return and nothing is
emitted on any channel, indefinitely (R-117a). A devno-only fix would have passed
every payload test.

Three states, never a bool: {Unknown, Live, StaleDevice, Aborted}, read through
Usable(), where Unknown counts as PRESENT — reporting absent stops a working
customer's apps.

No new recovery path; the existing one was unblocked. AttachDrive's normalize leg
already did the repair and three call sites already invoked it, including the
controller's Return branch before it restarts apps. All three died on
`if n == 1 && GuestSeesMount(...)` returning early. Now: StaleDevice ⇒ re-bind
(repairs live, guest never restarts); Aborted ⇒ quiet no-op, because a re-bind
lands on the same dead superblock and this runs every 20s — an infinite silent
retry that masks the state; it surfaces via BoundUnderParent=false instead.

Ordering trap caught by a test: reading the abort flag before comparing devices
classifies the real return state as aborted (its stale bind carries `shutdown`
too) and refuses the repair while still reporting correctly. The abort flag is
read off the RAW mount in the stale case.

Tests 849 → 863, 29/29 packages green. 6 red-proofs, each verified to have
landed. A hollow test was caught during them: the aborted fixture first used a
/dev/mapper device, for which RoleForStorage derives role=system — a system row
has no GuestPath, never runs the conjunction, and reports false by default, so
the assertion passed vacuously and no mutation could fail it. Found because RP1
failed to fail.
2026-07-30 12:26:42 +02:00
admin 6be168d1a0 docs(R-117 Part 1): repeat the no-block-I/O health-check rule where it binds
felhom.eu/CLAUDE.md now carries the standing rule (with the measurement), but
that file does not load in an agent-only session — and health checks are written
in this repo. A standing rule that does not load where it binds is the inert-seam
shape applied to a rule, so the constraint is repeated here as a one-liner with a
pointer to the full text.

No code change, no version bump.
2026-07-30 12:10:24 +02:00
admin d4eb259da2 docs(R-116): record the live proof — four events, two matched pairs, discriminated
v0.116.0 validated on a fresh box: real day-0 from the v1.25.0 ISO on a nested PVE
on demo-hp (per runbooks/target-selection.md), agent installed unaided from the
vouched Day-0 manifest, drives enrolled through the real endpoints, device loss a
real hot-detach.

  07:20:04  backup_target_absent   (error)  Cel meghajto   <- TARGET, specific
  07:22:34  backup_target_restored (info)   Cel meghajto   <- its matching pair
  07:24:04  storage_disconnected   (error)  Adat meghajto  <- NON-target, generic
  07:25:34  storage_reconnected    (info)   Adat meghajto

All four reached the hub; gate fired in 3 s. Discrimination is proven NON-trivially
for the first time -- both prior runs had the target itself emit the generic event,
so their mirror proved nothing. Over-correction passes on a POSITIVE observable: 0
ABSENT lines and 0 drive events over 2m14s with both drives present, while two
RETURNED lines prove the gate was ticking rather than dead.

Caveat recorded, not a regression: the drill's controller was 0.185.1 from the
golden, which predates R-114, so its absent-state banner showed the old false
"backup is on the system disk" copy. The R-114 guard is pinned by unit test and by
the payload, but could not be confirmed on that box. Filed as R-120 -- the golden
is a release behind the deployed controller, which is R-115's class one layer up.

Teardown all three layers, including the hub: VM purged, storage removed with the
space measured back, hub records gate-blocked on ONLINE with the command recorded.

Suite rc=0 read separately from this commit.
2026-07-30 09:33:53 +02:00
admin 21b0164fad R-116 (v0.116.0): give the backup-target flag and the gate's key the same row
The absent-drive alarm was generic while its recovery was specific -- a pair an
operator cannot match. Mechanism now measured, not reasoned (felhom.eu
audits/DIAG-r116-disks-payload-2026-07-30.md): with the device gone /disks returns
4 rows, not 3. The drive appears TWICE and the two facts the controller needs are
on different rows -- the Observe row has backup_target:true but mount_path:"" and
guest_path:"" (so driveTargetByPath registers NO key from it), while the registry
row owns /mnt/felhom-drives/<name>, the key the gate looks up, with BackupTarget
absent from its struct literal => false.

WHY v0.115.0 WAS INERT: its fallback computed StablePathForRaw(t.MountPath), and in
the absent state MountPath is ALSO "" -- emptied by the same exactMount failure
that empties BackingDevice. It assigned nothing. Its test passed because the fixture
supplied a MountPath production never supplies, and the harness left DriveTargets
nil so the union loop never ran. Both corrected here; red-proof 1 replays v0.115.0's
exact code against the real shape and it fails.

THE JOIN, which was the hard part: with the device gone the two records share no
runtime field -- no mount, no backing device, and the Observe row's DurableID has
degraded off the fs-UUID. They share CONFIGURATION: storage.cfg's path on one side,
the .mount unit's Where on the other, both yielding the same stable guest path. New
hub.StorageTarget.ConfigPath (json:"-" -- that struct is a cross-repo contract
pinned by the golden + contract_test key-set comparison, and nothing off-box needs
the value), set from s.Path in observe.go, consulted in disks.go only after MountPath
so the present-state path is byte-identical, plus a guest-path arm on the union dedup
so exactly one row carries the drive.

WHY NEITHER OBVIOUS OPTION WAS TAKEN -- both regress R-114, which shipped yesterday.
backup_target_offer.go:79 reads (BackupTarget && MountPath != "") as "a real
drive with its own mountpoint -- healthy" and returns before its TargetAbsent
branch. Back-filling MountPath onto the Observe row (the smallest change, and the
spec's lean) and teaching the registry row the flag (its MountPath is non-empty, read
from the stale unit file) BOTH manufacture that row while the drive is missing, which
would have told the customer the backup target is fine while its drive is gone.
R-114's correctness rests on the absent-state rows not combining the flag with a
mount path; that coupling was invisible until the payload existed. Pinned by
TestAbsentTargetKeepsR114DegradedSignal.

Role unchanged, BoundUnderParent conjunction not widened, no wire field changed.
Suppressing the registry row in the absent state also removes its false
state:"attached" and its root-filesystem-derived total_bytes -- R-118's symptom
goes incidentally; R-118 is NOT fixed and stays open.

Tests 845 -> 849, suite rc=0 read separately from this commit. Four red-proofs, each
mutation asserted to have landed first.

NOT live-validated at this commit: publish+vouch, C5, discrimination, over-correction.
2026-07-30 08:48:31 +02:00
admin 2f4ccab166 docs: correct two stale claims in CLAUDE.md that misdirect live work
Both found while writing felhom.eu runbooks/target-selection.md.

1. The demo host block said the t740 is the designated drill+build VM host "but no
   drill VM is provisioned there yet". Stale since 2026-07-25 -- VM 300 (drill-r50)
   has been there since. The sentence read as discouragement from the very box the
   operator ruling designates, which is part of why a drill went to DooPlex instead.
   Now says the ruling is realized and to start there, points at the new
   target-selection runbook, and notes drill.qcow2 on DooPlex is a BAKE fixture, not
   a drill target. Agent version dropped (it changes several times a day; the hub
   host list and --version are the authorities) and the t740's PVE node name added.

2. RETRACTED the block's claim that the agent "does not run at all" at the remote
   site because localapi binds the LAN literal 192.168.0.162 and the service has
   "never started" -- with an outstanding config edit needing Viktor GO. That was
   true before R-50 and is false now: since the island migration (2026-07-25)
   localapi binds 169.254.253.1:8443 on vmbr9, which is location-independent by
   design, and proxmox.endpoint is https://127.0.0.1:8006. Verified live 2026-07-30:
   service active, version 0.115.0, and GET /disks answered over the island -- the
   whole R-116 payload capture went through it. A session trusting the old text
   would not have attempted the read that worked. The recorded remote-site address
   was also wrong (.162, not .147), so it now says re-check instead of asserting one.

No code, no version bump, so no CHANGELOG entry (that file is version-keyed) and
REPORT.md is left holding the v0.115.0 record rather than being overwritten by a
docs fix.
2026-07-30 08:25:41 +02:00
admin a58239f6de v0.115.0 — R-116: the backup-target flag reaches the row the controller keys on
Session C measured it live: a drive whose device vanished raised the GENERIC
storage_disconnected while its return raised the SPECIFIC
backup_target_restored -- an alarm and an all-clear an operator cannot pair.
backup_target_absent never fired at all.

The mechanism is not what the Session-C audit first said, and the difference
decides the fix. RoleForStorage returns RoleSystem whenever backingDevice == ""
(internal/storage/role.go:180-181). When the device goes, exactMountDevice
fails, BackingDevice becomes "", the target row's role flips to system and it
loses its guest path -- but keeps its MountPath. The union loop skips any drive
whose MountPath is already seen, so the registry row is DEDUPED AWAY ENTIRELY.
/disks carries no row with that guest path, so isTarget[guestPath] is a MISSING
KEY, not a false. Setting BackupTarget on the union row -- the obvious fix --
could not have worked, because that row is not emitted when the alarm is needed.
The audit is corrected in the same push.

Fix: on the Observe row only, carry the guest path when the row IS the backup
target and its role flipped because the device vanished.

Three gates, verified not assumed:
- t.BackingDevice == "" restricts it to the vanished-device flip; a genuinely
  system-BACKED storage has a real device and is excluded, so a dir storage at
  /mnt/<name> on the root disk cannot acquire a guest path.
- Case B, the common fresh-box shape, is safe twice over: its target is the
  builtin local on /var/lib/vz and StablePathForRaw returns "" for anything not
  exactly /mnt/<name>, so nothing is set even before the gates apply.
- It cannot make the gate read an absent drive as PRESENT. BoundUnderParent is
  assigned at exactly two sites, both inside guest-path blocks a system-role row
  never enters, so it stays false and planDriveGates computes false || false.
  Pinned by TestAbsentTargetRowDoesNotRegisterPresence -- getting this backwards
  would have silenced the alarm the fix exists to raise.

The :213-214 boundary stands: no system or backup mount gains a guest path.

Tests +5, asserting the emitted /disks JSON through a faithful copy of the
controller's driveTargetByPath, because the failure class is "the value is on
the wrong row". Red-proof: removing the block fails with "isTarget[...] is a
MISSING KEY"; reverted byte-identical.

Filed not closed: the two-row shape that produced this survives.
2026-07-29 23:51:03 +02:00
admin b58d7bcf39 v0.114.0 — R-113: drive presence means the DEVICE, not the bind
BoundUnderParent, the one field the controller's drive-absent gate keys on,
reported only "is this path a mount target in the guest's mountinfo". The
drive's raw mount at /mnt/<name> is a systemd mount unit bound to its device and
dies with it, but the agent's own bind of <raw>/felhom-data under the shared
parent is an ordinary bind: nothing ties it to the device, so its mountinfo
entry OUTLIVES the device as a stale shell. Presence read that survivor as true,
planDriveGates never produced a Stop action, and nothing fired on any channel --
not backup_target_absent, not the generic storage_disconnected. Measured live in
E-2d: detached at 10:58:37Z, silent for 4.5 minutes while the agent itself
logged "enrolled drive absent by UUID" every 20s (felhom.eu
audits/E2D-fresh-vm-2026-07-29.md §5.2).

The fix: BoundUnderParent becomes a CONJUNCTION -- bound under the parent AND
the drive's raw host mount still mounted (devicePresent, new deviceCheck seam
defaulting to isHostMountpoint). Applied at BOTH /disks construction sites. The
union path matters more, not less: it hardcodes State:"attached", so the
raw-mount check is the only device truth that row carries, and it is exactly the
shape E-2d detached.

Why a conjunction and not a replacement: half 2 alone would regress boot
ordering, where the raw drive mounts early and the bind lands ~18s later; the
gate depends on that window reading ABSENT. The conjunction leaves that
byte-identical and closes only the case the gate could never see.

Unknown is never absent: devicePresent("") returns TRUE. A false absent stops a
working customer's apps -- the failure mode of this fix, not of the bug.

Controller UNCHANGED, no MinAgent bump. BoundUnderParent has exactly one
functional consumer (planDriveGates, intermediary.go:226); every other mention
in both repos is a comment or a test, and boot convergence deliberately moved
off it to pollLiveBinds/driveBindLive. The alternative -- a new DevicePresent
bool the controller ANDs in -- was rejected as dangerous: a bool absent from an
older agent's JSON decodes to false, so every drive on a pre-0.114.0 agent would
have read ABSENT and stopped its apps.

Tests +6 in internal/localapi (208 -> 214): groups A-D plus a wire-contract test
asserting the ENCODED bound_under_parent, since that is what crosses to the
controller. Four red-proofs run and reverted (drop the conjunction on each path;
invert unknown; drop the bind half); disks.go verified byte-identical after.

NOT LIVE-VALIDATED. No drive was pulled. Leg awaiting Session C: device loss ->
gate Stop -> SetDisconnected -> backup_target_absent on the wire.
2026-07-29 17:20:16 +02:00
admin 58b598b697 v0.113.0 — E-2a: guarded backup-target wrapper + POST /backup/target
The agent cannot create a PVE storage (Datastore.Allocate at /storage) or grant
an ACL (Permissions.Modify) -- it holds neither by design, and widening the role
would trade the whole blast-radius containment model for one feature. The
privileged half therefore lives in a new fenced shim behind a literal
FELHOM_BACKUPTARGET sudoers alias, following the mkfs/pbs-apply pattern.

The wrapper enforces the two laws E-1 paid for on live hardware so no caller can
forget them: F-1 the path must BE the drive's own mountpoint, F-2 is_mountpoint 1
is hardcoded rather than a caller flag. It refuses a root-device target, has NO
storage-removal path of any kind (the pbs-apply no-delete law, grep-assertable),
is idempotent for the same path, and REFUSES to repoint an existing id.

POST /backup/target drives it in a fixed order: create -> grant -> config.
Reversed, a config pointing at an ungranted storage 403s every backup on first
run -- exactly E-1 finding F-3. A failed grant leaves the config untouched.

It deliberately does NOT restart the agent: restarting with a backup in flight
cancels the wait and records a spurious tier failure for a backup that actually
succeeded (E-1 did this to a real felhom-pbs run). It returns restart_required
and the caller restarts behind its own immediate in-flight check.

Config rewrite preserves unknown keys verbatim and writes in place, since
/etc/felhom-agent is root-owned while agent.json is agent-owned 0600.

Green gate: build + vet + test rc=0 (29 packages), run separately from this commit.
2026-07-29 09:05:59 +02:00
admin 958e54f6a6 v0.112.0 — E-2: GET /disks flags the backup-target drive
Additive backup_target field, true for the drive backing the PRIMARY tier.

The controller cannot work this out itself: settings.StoragePath.BackupTarget is
customer INTENT, and on the two boxes migrated by hand in E-1 that intent was
never recorded -- intent is empty while the drive really IS the target. Without
this flag the absent-target alarm could not name the drive on exactly the boxes
that currently have one.

omitempty + false on an older agent, so an old controller degrades to the generic
disconnect alarm rather than a wrong one.

Test asserts the target IS flagged AND the non-target is NOT, as a pair -- a
blanket true would satisfy a naive one-sided check.
2026-07-29 08:20:27 +02:00
admin 38176ada9d v0.111.0 — E-2c: the backup-target drive can no longer be ejected
A regression guard on a configuration that is live right now. E-1 moved each
demo box's whole-guest vzdump target onto its secondary drive at that drive's
own mountpoint -- but RoleForStorage types a local-dir on a non-system device as
user-data, so the existing eject role gate PASSED it. POST /disks/eject on
/mnt/nvme-1tb (demo-hp) or /mnt/hdd_1 (demo-felhom) would have SUCCEEDED
silently, taking the only local whole-guest backup with it, with no alarm and
the box still reporting a configured tier. Found by E-2 Phase 0, not by a
failure.

handleDiskEject and handleDiskDecommission now call refuseIfBackupTarget AFTER
the role gate and refuse with 409, naming the storage and the remedy -- the op
is ordered, not forbidden: reassign the target first.

NOT a role reclassification, which is the obvious fix and the wrong one: making
RoleForStorage return RoleBackup would refuse every legitimate eject of the
customer's own data drive, because on both demo boxes that drive IS the target.
That trades a silent failure for a permanent obstruction.

backupTargetAt resolves through the agent's own storage view, never the caller's
claim, and fails OPEN -- safe because it sits behind the role gate, which fails
SAFE on the same error.

Red-proofed both ways, mutations verified to land first:
  - removing the eject guard  -> "eject of the backup-target drive SUCCEEDED (200)"
  - the over-correction (any backup-content dir storage is the target)
    -> the gate blocks /mnt/spare, failing TestEjectStillAllowedOnANonTargetDrive

Harness note: normalizeBackupTiers DROPS tiers with a nil Service and falls back
to the legacy empty-TargetID tier -- an earlier version of this test exercised
nothing and reported the production bug as if the fix had failed.

Green gate: build + vet + test rc=0 (29 packages), run separately from this commit.
2026-07-29 08:17:14 +02:00
admin d5c769173b REPORT + CONTEXT: F-LEAK closed via the fenced destroy (v0.110.0), all three attempts recorded 2026-07-28 11:34:48 +02:00
admin 50751b8901 F-LEAK third attempt: band-scoped fenced destroy (v0.110.0)
The per-VM ACL is consumed by the destroy it authorises (PVE remove_vm_access,
LXC.pm:906), so it works once per slot. Fourth root-fenced exception, band-enforced in
sudoers literally + in code + at the caller. API destroy still tried first.
2026-07-28 11:28:54 +02:00
admin ff7f68e089 REPORT + CONTEXT: F-REBOOT shipped, F-LEAK's first fix refuted and replaced, v0.109.0 observable 2026-07-28 11:21:42 +02:00
admin 88b3cf03dd gofmt: normalize internal/localapi (whitespace only)
Swept up by gofmt -w on the package while adding the guest-power observable. No
semantic change; 3 of 5 files are tests.
2026-07-28 11:15:45 +02:00
admin f27f7a2659 guest-power: add the liveness observable it shipped without (v0.109.0)
The v0.107.0 watchdog was silent on a healthy box, so its health could only be inferred
from absence — F-OBS's shape, shipped in the same session F-OBS was fixed. INFO summary
every 10th sweep with what it saw; aborted sweeps are not counted. Red-proofs 7 and 8.
2026-07-28 11:14:56 +02:00
admin 8db92947cd F-LEAK: remove the pool-adoption fix — refuted live; the fix is a path-scoped ACL (v0.108.0)
PUT /pools/{pool} ALSO requires VM.Allocate on the VM being added, so Pool.Allocate
cannot bootstrap its own membership. Proven live on demo-hp 2026-07-28. The real fix is
felhom-host-install v1.21.0 granting FelhomAgentGuest at /vms/990000..990009.
2026-07-28 11:05:37 +02:00
admin 367a503a0f F-REBOOT + F-LEAK: the agent's authority over guest lifecycle (v0.107.0)
F-REBOOT — a guest rebooted mid-backup never came back (fault 11: 9m47s of total
appliance outage, no lock, nothing retrying). The existing stale-lock recovery is
correct but missed it two ways: its predicate needs a stale vzdump lock and that
guest was unlocked, and it runs only at agent startup. New periodic guest-power
watchdog acts on 'should be running, is not, is not locked'.

onboot is the should-be-running signal, not invented here: stalelock.go already
uses it for this same decision, it is 0 on scratch/golden, and pve-guests uses it
at host boot. Guards: onboot:0 never touched (Scenario B), a locked guest is left
to the stale-lock path, a guest with a vzdump in flight is left stopped,
unprovable ownership acts on nothing, unconfirmable backup state fails safe.
Bounded retry 3x at 1/2/4m then ERROR (Scenario C) — a healthy start takes ~25s.

F-LEAK — a failed restore-test could not destroy its scratch (403 VM.Allocate).
It is pool membership, not privsep: VM.Allocate is granted at /pool/felhom only,
and a failed restore never completes the --pool association. Fix needs NO new
grant — Pool.Allocate is already held, so the teardown adopts the stranded
scratch into the pool and retries the destroy. Guarded by scratchAdoptAllowed:
scratch provenance AND the numeric band, both required (Scenario E).

Six red-proofs across both fixes, all observed failing.
2026-07-28 10:27:07 +02:00
admin a18b18e5de docs: correct inflight.go's DEFERS claim (F-A1)
The gate's behaviour is correct and unchanged. The comment said 'a caller that
cannot acquire DEFERS to its next cadence' — true of the restore-test caller,
NOT of the backup caller, and it did not say so. The controller recorded the
refusal as a tier failure and emailed the operator; fixed controller-side in
v0.179.0. Comment only, no behaviour change.
2026-07-28 08:50:23 +02:00
admin af1c21abc4 docs: F-CRIT-2 fix — REPORT, CONTEXT (v0.106.0)
Phase 0 discriminator survey, the measured 1 MiB floor and its justification,
four red-proofs with observed failure text, and the live re-test of campaign
fault 2 on demo-hp (both directions). Records that server-side prune does NOT
count phantoms toward keep-last — no retention bug — but never removes them
either (filed as R-99).
2026-07-28 08:07:42 +02:00
admin c9a5cc664a F-CRIT-2: a failed backup must not look like a fresh one (v0.106.0)
NewestArchiveTime counted an aborted PBS upload (1 byte, no manifest, NEWEST)
as a successful backup, so the tier reported fresh, went not-due, and was never
retried. On the real 168h offsite cadence that is 7 days of silence, and neither
the R-88 breaker (defers only DUE tiers) nor the hub deadline monitor (reads the
same freshness) can catch it.

R-84's storage-as-ground-truth was right; the bug is that presence was taken for
validity. Now only plausibly-complete entries count, via a measured size floor
(minPlausibleArchiveBytes = 1 MiB). Undecidable => not counted.

Size is the only tier-agnostic discriminator: verification and encrypted are
absent on EVERY local dir archive (and on a good PBS snapshot until verify-new
catches up), so gating on either would reject 100% of local backups and cause
fleet-wide backup thrash. Floor measured against the fleet: smallest real backup
is 612,397,450 B, so 1 MiB leaves 584x headroom — asserted by a test.

Rejections are announced at WARN once per distinct volid, naming snapshot and
reason; per-poll logging would emit ~288 lines/day and bury the signal.

Four red-proofs, all observed failing.
2026-07-28 07:47:22 +02:00
admin 023655370b seam sweep: compile-time witness for BackupArchiveLister (no version bump)
localapi.BackupArchiveLister is satisfied by a runtime type assertion in
newestArchiveOn; a failed assertion degrades SILENTLY to archiveAbsent, which is
the pre-R-84 in-memory-only behaviour — i.e. the R-84 bug returning with nothing
in any log to say so. There was no compile-time witness anywhere in production
code in either repo.

No defect found: *BackupRunner does satisfy it today, so this is a guard, not a
fix. Verified the guard works — breaking NewestArchiveTime's signature now fails
go build, where before it compiled and vetted clean.

No version bump, no deploy: compile-time only, zero runtime effect.
2026-07-27 18:23:49 +02:00
admin 9842c52853 docs: REPORT for agent v0.105.0 (R-88 Part 2)
Records the wire encoding (string enum, empty = legacy not unknown) and the two
judgement calls: the fail-safe direction is unchanged (unknown is still due), and
a service with no lister stays ABSENT because calling it unknown would starve a
genuinely new box on a pre-R-84 build.
2026-07-27 18:12:04 +02:00
admin 1c2664b0c1 agent v0.105.0 — R-88 Part 2: /backup/due gains age_state
newestArchiveOn's (time.Time, bool) signature could not express the 'unknown'
its own doc comment promised: a read error and a genuine not-found both returned
(zero,false), so /backup/due emitted a POSITIVE 'no successful backup recorded
yet' with a nil age, and the controller fired its window-gate valve on an
unreadable storage.

Three states now: known / absent / unknown, carried as a STRING enum so the zero
value unambiguously means 'legacy agent' rather than masquerading as an answer.
Fail-safe direction unchanged — unknown is still DUE; only the window-gate bypass
narrows to ABSENT.

A service with NO lister deliberately stays ABSENT: calling it unknown would stop
a genuinely new box on a pre-R-84 build from ever backing up outside its window.
An unparseable timestamp becomes unknown — a backup happened, we cannot date it.
2026-07-27 18:00:56 +02:00
Claude Code 5bca7bfc9a R-85 Phase 4: finalise v0.104.0 + register the new helpers in REUSE
Version finalised from v0.104.0-dev — an ldflags version disagreeing with the
CHANGELOG is the reconciliation problem hub 0.73.2 already caused.

REUSE gains backup.InFlight, backup.RestoreTestState and the SpecBuilder/
TierPicker/PickRestoreCandidateOn trio, each with the decision someone could
unknowingly undo: rotation credit only on success; InFlight callers defer and
never cancel; a tier with no archive no-ops rather than failing; SourceTier
comes from the archive, never the configured target.
2026-07-27 07:32:55 +02:00
Claude Code 043c7622bc R-85 Phase 2: tier rotation, persisted state, one heavy op at a time
The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite
tier's archives were never candidates — which is why demo-hp's DR tier reported
'applied' with zero snapshots for five days and nobody noticed.

Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first,
which is where the offsite tier starts. Ties break on target id so ordering is
deterministic rather than following Go's randomised map order. Rotation credit
only on SUCCESS — a permanently failing tier must keep sorting first, not look
freshly proven and stop being retried.

- backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename).
  This genuinely needs persistence unlike R-84: R-84 had ground truth to consult
  (the archive is still on the storage), whereas a restore-test destroys its
  scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'.
- backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup
  path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over
  the same tunnel a backup pushes one, and at ~33 MB/min both drift toward
  timeout, which is how a healthy tier gets recorded as failed. Callers DEFER,
  never cancel.
- PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or
  every fresh box looks broken for its first week.
- An empty tier is skipped and the next tried; it cannot starve, since it is
  still least-recently-proven once it has an archive.
- POST /backup joins the gate (409 naming the holder).

Red-proofs A/E/F observed with the documented text. Full suite green (29
packages, rc=0).
2026-07-26 21:00:42 +02:00
Claude Code 765d8b3168 R-85 Phase 1: the restore-test spec is built PER RUN, not frozen at daemon start
SchedulerOptions.Spec was a VALUE produced by an immediately-invoked function
at daemon start, so storageTier() and restoreTaskTimeout() were evaluated once
and reused for every run for the process lifetime. Nothing tier-varying was
expressible (the offsite tier could never be scheduled), and it was a latent
staleness bug besides: a storage-type or config change did not take effect
until restart.

- backup.SpecBuilder: func(ctx, archive) RestoreTestSpec, called once per run.
  The archive is passed because the tier MUST come from it (v0.100.0 rule) —
  config-derived is what classified a PBS archive as 'local' and killed a
  14.46 GB WAN restore at the 10-minute local bound.
- A nil spec builder SKIPS loudly instead of panicking: a wiring bug must cost a
  restore-test, never the daemon goroutine.

Red-proof observed. Full suite green (29 packages, rc=0).
2026-07-26 20:47:28 +02:00
Claude Code edde8a01ca REPORT: record the PASSED restore round-trip (mount_parity ok, source_tier pbs) 2026-07-26 19:05:05 +02:00
Claude Code a7ef497cc4 REPORT: add v0.103.0 (R-84) + demo-hp's first offsite backup landing (4.25 GB) 2026-07-26 18:24:37 +02:00
Claude Code 5acf1033a2 v0.103.0 — R-84: an agent restart no longer triggers a redundant backup
Observed live: three redundant local backups on demo-felhom in one afternoon of
deploys. The backup Store is in-memory ('lost on restart; the cadence
re-populates'), so after every restart /backup/due said 'no successful backup
recorded yet' and the controller took another one. On the offsite tier that is a
wasted multi-hour WAN upload after every agent deploy.

- BackupRunner.NewestArchiveTime: when a backup last LANDED on this tier's
  storage, read from the storage.
- localapi.BackupArchiveLister (optional BackupService extension): the due-check
  takes whichever is newer, the in-memory record or the storage.

Asking the storage rather than persisting the store is deliberate: it is ground
truth (a pruned archive correctly stops counting, where a persisted record would
keep claiming a backup that no longer exists), needs no new on-disk state, and
answers only 'when did a backup last land' — the richer fields stay with real
records so the host-report never carries invented numbers.

Fail-safes: read error -> fall back to memory (never fake freshness, never
suppress); genuinely empty -> due; old archive -> still due; service without the
lister -> unchanged.

Red-proof observed; full suite green (29 packages).
2026-07-26 18:20:59 +02:00
Claude Code e4f22f4c4f REPORT: R-82 agent arc v0.97.0 -> v0.102.0 (overwrite)
Four defects found by running it rather than reviewing it, the frozen untargeted
contract verified live, the fail-safe directions stated once, and what is NOT
done — including that the scheduled restore-test never selects the offsite tier
and that R-84 is now closer to a prerequisite than a tidy-up.
2026-07-26 17:56:42 +02:00
Claude Code 13ca2d96b2 fix(test): give the tiered-server harness a real storage view (v0.102.0 follow-up)
v0.102.0 defers a tier whose target storage is absent. The Slice A harness used
fakeStorage{} with NO targets, so after that change it deferred every tier and
five Slice A assertions became vacuous failures.

The product behaviour is correct; the harness never modelled a real box, which
has both storages present. Fixed by giving it local + felhom-pbs.

My error, and worth naming: I ran the suite and committed in the same command,
read 'packages ok: 28' and pushed without checking rc=1. That is exactly the
exit-code trap recorded in this arc twice already.

Full agent suite green: rc=0, 29 packages.
2026-07-26 17:41:03 +02:00
Claude Code 005083b558 v0.102.0 — R-82 Slice D: an unprovisioned tier DEFERS instead of failing
Prerequisite for the installer default (host-install 1.20.0). A fresh box now
carries the offsite tier, but felhom-pbs only exists once the hub provisions the
DR tier. Without this the tier would report due in that window and the
controller would quiesce the apps and fire a vzdump at a missing storage every
cadence.

- GET /backup/due?target= defers when the target storage is absent
  (targetStoragePresent): due:false with a reason that says why. The tier goes
  live with NO restart once the storage appears.

Fail-safe: a storage-view ERROR returns present and the tier stays due. 'I could
not check' must never be read as 'not there' — that would silently suppress
backups, the absence-is-not-failure rule relearned three times now (R-80, R-81,
the R-82 wait timeout).

Full suite green.
2026-07-26 17:40:31 +02:00
Claude Code 0fabc15896 v0.101.0 — R-82: a leaked restore-test scratch can no longer auto-start
CORRECTION: I earlier reported that the restore-test would boot a scratch guest
with the live guest's MAC/static island IP/hostname and break the control
plane. That was WRONG — RunRestoreTest step 2 link-downs EVERY interface
(withLinkDown, unit-tested) before the guest is ever started. The design
already handled it.

The real, narrower hazard: a restore that fails BEFORE step 2 (what the v0.100.0
wait bug caused) leaves a scratch holding the SOURCE guest's config verbatim,
including onboot:1. If teardown also fails (403 missing VM.Allocate — PVE
associates the pool only at restore completion), a host reboot would start that
leaked clone alongside the original with NICs up.

- proxmox.RestoreLXCOptions.ConfigOverrides: guest-config params applied AT
  RESTORE TIME.
- The restore-test passes onboot=0 — at restore time, not after, because
  'after' is exactly the path that leaks.

NOT changed: the link-down step (already correct, the primary defence); the
agent's Proxmox privileges (widening VM.Allocate to /vms would remove the
accidental guard that stopped a destructive mid-restore teardown).

restore_test_cadence_seconds was set to -1 on demo-felhom under the mistaken
reading; re-enabled.

Red-proof observed; full suite green (29 packages).
2026-07-26 16:49:40 +02:00
Claude Code a7421b09c7 v0.100.0 — R-82: the restore tier comes from the ARCHIVE, not the configured target
Found by the first real PBS restore round-trip, not by review.

Restoring a felhom-pbs: archive on a box whose primary target is 'local'
failed after exactly 600.76s — the 10-minute LOCAL wait — against a 14.46 GB
WAN restore needing ~2 hours. The selftest derived its tier from
cfg.Backup.BackupTarget() (the configured default), so restoreTaskTimeout
correctly returned the local bound for a PBS archive. The recorded result even
said source_tier=local for a PBS archive.

The tier-aware machinery was already right; it was fed the wrong input. What
broke is an assumption that stopped being true the moment a second tier
existed: 'the configured target' is no longer a proxy for 'the tier this
archive belongs to'.

RestoreTestSpec.RestoreTaskTimeout's doc comment predicts the consequence
exactly, and it happened: teardown fired at a still-restoring guest and was
refused with HTTP 403 missing privilege VM.Allocate (PVE associates the pool
only at restore COMPLETION, and the grant is on /pool/felhom not /vms). That
403 was load-bearing luck — the only reason a destructive teardown did not run
against a half-restored guest. The restore completed unharmed.

- restoreTierForArchive() derives the tier from the archive's own storage
  (archiveStorageID parses the volid prefix), falling back to the configured
  target only when there is no prefix.

Recorded, NOT fixed here: the daemon's scheduled restore-test still only covers
the PRIMARY tier (Pick uses a runner built on BackupTarget(); Spec is built once
at construction, not per tick) — so the offsite tier is never automatically
restore-tested. And the agent still cannot tear down a scratch guest until its
restore completes; widening the token's privileges is deliberately not the fix.

Full suite green (29 packages).
2026-07-26 15:22:08 +02:00
Claude Code 3d955e4edd v0.99.0 — R-82 operator rulings: 2-week offsite retention + one backup at a time
Ruling 1 (2 weeks of weekly offsite backups): localPruneSpec's blanket PBS
refusal is now scoped — an ADDITIONAL tier with an explicit keep_last may
prune its PBS target. The refusal still applies in full to the PRIMARY tier,
because BackupTarget() defaults to felhom-pbs and KeepLast() defaults to 3, so
a box with neither key set would silently prune its offsite DR to 3 restore
points. An additional tier cannot have that accident (keep_last defaults to 0).

Ruling 3 (first backup runs as long as needed; nothing else starts until done):
- additional-tier wait bound 6h -> 12h (measured ~33 MB/min => ~5h for a first
  full 10 GB snapshot; 12h gives margin but stays bounded so a hung task still
  surfaces)
- ONE BACKUP AT A TIME PER GUEST across all tiers: POST /backup returns 409
  when a DIFFERENT tier is in flight, naming the busy tier, with NO data object
  so nothing is parseable as the caller's own job. Same tier still returns that
  job (202, unchanged).
- snapshotted now counts as in-flight, not just running — after the snapshot the
  vzdump is still uploading and holding the lock. The old check left a window
  where a second POST started a real second vzdump. Latent bug, closed.

Full suite green (29 packages); red-proof observed and restored.
2026-07-26 15:05:54 +02:00
Claude Code a667c269c7 v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure)
Found by live validation on demo-felhom, not by review.

The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.

Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.

- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
  (a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
  the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
  NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
  tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
  outer context reproduces the same false failure four hours later.

Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.

Red-proof observed and restored; full suite green.
2026-07-26 14:53:24 +02:00
Claude Code 68bcebe493 REPORT: R-82 Slice A (agent v0.97.0) — per-target tiers built, NOT deployed (no drill target reachable) 2026-07-26 12:36:47 +02:00
Claude Code 739b3c3b58 v0.97.0 — R-82 Slice A: per-target backup tiers (local daily + PBS weekly)
Mechanism only. No box changes behaviour until a backup_targets entry is
added to its config (Slice D); an untouched config resolves to exactly one
tier and behaves byte-identically to v0.96.0.

- config: BackupTargetConfig + ExtraTargets + BackupTiers(); each tier carries
  its OWN cadence and retention (keep-last=3 is three days on a daily tier and
  three weeks on a weekly one). A missing cadence is REJECTED, not defaulted —
  a weekly DR tier silently running daily would fill the 37.2 GB datastore.
  main.go logs every rejection at ERROR.
- /backup/due?target= judges a tier against its OWN newest successful backup.
  Without that filter a fresh local backup satisfies the weekly PBS cadence and
  the DR tier never runs — today's bug, re-created in code.
- GET /backup/tiers advertises the tiers; a 404 is the controller's pre-R-82
  capability probe (Slice B).
- Jobs keyed by (vmid,target): single-flight is per tier, which is what lets
  the weekly night run both backups in ONE quiesce window. Job ids are unique
  per tier by construction, not by clock luck.
- One runner per tier: the runner holds target+retention as immutable state,
  so parameterising one runner would risk pairing tier A's target with tier B's
  retention.

COMPATIBILITY (frozen): untargeted /backup/due, POST /backup and
/backup/status keep the primary tier and the pre-R-82 response BYTES —
Target is omitempty and stays empty. The primary's job-id format is unchanged.

NOT changed: the local tier; PBS is still never pruned by the per-run flag
(keep_last defaults to 0 = never prune — enabling DR pruning is irreversible
and needs an operator ruling).

Tests 748->768. Red-proof #1 observed and restored.
Phase 0: felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md
2026-07-26 12:20:58 +02:00
admin dfd5d731ee v0.96.0 — R-50 island NIC: provision attaches the guest island net1
- LocalAPIConfig.island_bridge + island_guest_addr (+ IslandEnabled, Validate
  all-or-nothing + CIDR guard)
- buildBringUpConfig attaches static net1 (island) on provision + DR when set;
  absent otherwise (pre-R-50 byte-for-byte). Plumbed from cfg.LocalAPI at both
  RunBringUp sites. Endpoint already follows listen_addr (A0: no template change).
- healer stays eth0-only (A3 verify-only) — red-proof test locks the scoping
- example config + firewall example rewritten for the island; REUSE updated
- 3 non-hollow tests; full green. MinAgent unchanged.

Coupling: host-install island config requires agent >= 0.96.0 (vouch first).
2026-07-25 14:16:23 +02:00
admin 36ed6594d4 docs(CLAUDE.md): note demo-hp (t740) as designated drill/build VM host (no drill VM yet) 2026-07-25 09:59:12 +02:00
admin 271aa3d9ed v0.95.0: REPORT (overwrite) — SMART coverage live-verified (system SSD + USB → Rendben + models) 2026-07-25 08:32:09 +02:00
admin ed97232598 v0.95.0: SMART coverage — union-path drives + LVM/dm root + device model
Implements SPIKE-smart-coverage-2026-07-25 fixes B+A (additive; MinAgent unchanged).
Fix B: storage.SmartReader.SMARTForBacking wired into the /disks union path (localapi
Smart seam) so registry/USB drives get a real SMART read (watchdog Known stays
enrich-free). Fix A: smartDeviceFor resolves dm/LVM to the whole disk via
/sys/block/<dm>/slaves (recursive; skips >1-disk); the builtin local dir on the LVM
root gets a SMART-only device from its containing filesystem (never touches
backing/durable_id). SmartSummary.ModelName captured from smartctl. Fix C (-d sat)
stays rejected. Tests + red-proofs (dm multi-disk skip, enrich smartHint, union
routing); Known-path-never-SMARTs asserted.
2026-07-25 08:21:45 +02:00
admin 643899c191 v0.94.0: REPORT — appended (SMART serialized into /disks, live-validated on demo-felhom) 2026-07-24 21:41:01 +02:00
admin 21fee69154 v0.94.0: serialize per-disk SMART into the /disks payload
Additive, backward-compatible (MinAgent floor unchanged). The SMART is already
computed on the request path (storage.Observe -> enrich); this copies the target's
Smart into localapi.DiskInfo (pointer, omitempty) only when Health != "", so an
unread/absent summary stays omitted and the controller renders "Nincs adat".
No new smartctl load, endpoint, or sudoers change.

Test TestDisks_SmartSerialized + red-proof (drop the copy -> fails).
2026-07-24 21:11:13 +02:00
admin c230258542 docs: v0.93.0 publish train executed — built, published, vouched, deployed fleet-wide 2026-07-22 09:03:11 +02:00
74 changed files with 9949 additions and 408 deletions
+104
View File
@@ -0,0 +1,104 @@
# gates — re-run this repo's gate entry point on every push, on a machine that does not care who
# pushed or what they typed.
#
# *** THIS REPORTS. IT CANNOT REFUSE. ***
#
# felhom repos push straight to `main` with no pull request, so there is no merge for a status
# check to stand at. The refusing half is `.githooks/pre-push`, which is local to a clone and which
# `git push --no-verify` skips; this half is what notices when that happened. Neither half is the
# whole thing, and both are named in felhom.eu documentation/backlog/OPEN-ITEMS.md R-168.
#
# NO `uses:` STEP ANYWHERE, deliberately: JavaScript actions need a node runtime in the runner, and
# the runner is a host-mode container with python3 and git and nothing else (see
# homelab-manifests/gitea-system/act-runner.yaml for why it is not privileged). Probe P3 measured
# that a plain `git fetch` of the pushed SHA from the in-cluster Gitea service is enough.
#
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
# layer up. That is the last step, and it runs ONLY on failure.
name: gates
on: [push]
jobs:
gates:
runs-on: felhom-gates
steps:
- name: Fetch the pushed commit and the sibling clone it needs
# This repo's entry point invokes a SHARED checker that lives in the felhom.eu clone next
# door and is deliberately never copied here — so CI has to reproduce the workspace's
# sibling layout or the gate fails closed with "gate is MISSING". The sibling is also
# needed for CONTENT: this repo's REUSE.md cites a path that lives in the hub.
run: |
# Shallow, and pinned to the exact SHA that was pushed — not to the branch tip,
# which can move under us if two pushes race.
mkdir -p ws/felhom-agent
cd ws/felhom-agent
git init -q .
git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom-agent.git
git fetch -q --depth 1 origin "$GITHUB_SHA"
git checkout -q FETCH_HEAD
echo "checked out $(git rev-parse HEAD)"
cd .. && git clone -q --depth 1 http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git felhom.eu
echo "sibling felhom.eu present at $(cd felhom.eu && git rev-parse --short HEAD)"
- name: Run the gate entry point
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy. The
# exit code IS the result: no `|| true`, no pipe that could swallow it.
run: cd ws/felhom-agent && python3 scripts/agent_gates.py --fast
- name: Alarm on failure
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
# is exactly the shape R-29 filed against. So the run sends its own alarm, on the project's
# existing transactional path (Resend, the same one the hub uses), and prints the provider's
# accepted id so "a message left the machine" is an observable, not an assumption.
#
# Pure python3 and urllib, NOT curl: the runner image carries python3 and git and nothing
# else on purpose, and the first version of this step died on `curl: command not found`.
# Reaching for a bigger image to send one HTTP request would have been the wrong trade.
if: failure()
env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
run: |
python3 - <<'PY'
import json, os, sys, urllib.request, urllib.error
key = os.environ.get("RESEND_API_KEY", "")
if not key:
sys.exit("ALARM FAILED: RESEND_API_KEY is empty — the alarm cannot be sent, and a "
"silent alarm is worse than none. Set the user-level Actions secret.")
repo = os.environ.get("GITHUB_REPOSITORY", "?")
sha = os.environ.get("GITHUB_SHA", "?")
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
body = json.dumps({
"from": "Felhom CI <monitoring@felhom.eu>",
"to": ["admin@felhom.eu"],
"subject": "[felhom CI] gates FAILED in %s" % repo,
"text": (
"The gate entry point exited non-zero.\n\n"
"Repository : %s\n"
"Commit : %s\n"
"Run : %s/%s/actions/runs/%s\n\n"
"The failing gate names itself in the run log.\n\n"
"If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
"disagree - that is a finding about the gates themselves, not about CI, and it\n"
"outranks whatever the push was for.\n"
) % (repo, sha, srv, repo, run),
}).encode()
req = urllib.request.Request(
"https://api.resend.com/emails", data=body, method="POST",
headers={"Authorization": "Bearer %s" % key,
"Content-Type": "application/json",
# Cloudflare fronts api.resend.com and BLOCKS the default
# "Python-urllib/3.x" agent with its own 403 (error 1010) — which looks
# exactly like an auth failure and is not one. Measured 2026-08-02.
"User-Agent": "felhom-ci/1.0"})
try:
with urllib.request.urlopen(req, timeout=30) as r:
print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
except urllib.error.HTTPError as e:
sys.exit("ALARM FAILED: Resend returned HTTP %s: %s" % (e.code, e.read().decode()[:300]))
PY
+47
View File
@@ -0,0 +1,47 @@
#!/bin/sh
# pre-push — refuse a push that carries a broken gate. (2026-08-02, R-29 leg (b) first half.)
#
# Runs this repo's ONE gate entry point in --fast mode: only checks that touch no network and no
# container runtime, so a push stays a push and never pulls images or starts containers. The slow
# gates stay deliberate periodic runs; a hook that takes minutes gets bypassed within a week and
# the bypass becomes the habit.
#
# BOTH LINES BELOW ARE DELIBERATE. An absent log line is not evidence a hook ran — a silent pass is
# equally consistent with "gates green" and "hook never fired", so a passing push says so out loud.
#
# HONEST LIMITS, stated so this is not mistaken for enforcement it cannot provide:
# * per-clone — core.hooksPath is local config and a clone does not carry it. Arm a clone once:
# git config core.hooksPath .githooks
# Any manual entry-point run WARNS when the clone is unarmed.
# * skippable — `git push --no-verify` bypasses this entirely. That is on purpose: an escape
# hatch that cannot be reached is one that gets removed the first time it is
# inconvenient. USING IT MUST BE STATED IN THE SESSION REPORT.
# The half that is neither per-clone nor skippable is CI — felhom.eu OPEN-ITEMS.md R-168.
#
# Measured 2026-08-02 (git 2.47.3): a relative core.hooksPath resolves correctly and the hook's cwd
# is the repo root whether `git push` is issued from the root or from any subdirectory. The
# explicit rev-parse below does not depend on that.
set -u
root=$(git rev-parse --show-toplevel 2>/dev/null) || {
echo "pre-push: FAIL - cannot resolve the repo root (git rev-parse --show-toplevel)." >&2
exit 1
}
cd "$root" || exit 1
if ! command -v python3 >/dev/null 2>&1; then
echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
echo " pass by default. Install python3, or push with --no-verify and say so." >&2
exit 1
fi
echo "pre-push [felhom-agent]: running scripts/agent_gates.py --fast ..."
python3 "scripts/agent_gates.py" --fast
rc=$?
if [ "$rc" -ne 0 ]; then
echo "pre-push [felhom-agent]: PUSH REFUSED - gates exited $rc. Fix the finding above, or bypass with" >&2
echo " 'git push --no-verify' and state that you did in the session report." >&2
else
echo "pre-push [felhom-agent]: gates OK - push proceeding."
fi
exit $rc
+1298
View File
File diff suppressed because it is too large Load Diff
+35 -7
View File
@@ -88,7 +88,14 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S
## Demo host (for live tests) ## Demo host (for live tests)
Node **`demo-felhom`**, API `https://192.168.0.162:8006`. SSH alias `felhom-pve` (root@pam) — Node **`demo-felhom`**, API `https://192.168.0.162:8006`. SSH alias `felhom-pve` (root@pam) —
available to CC as plain `ssh felhom-pve`. The agent pins the served leaf cert — verify the available to CC as plain `ssh felhom-pve`. A **second demo node `demo-hp`** (HP t740, node name
`felhom-host`, `ssh demo-hp` — no baked key; break-glass root via hub `host_recovery/demo-hp-bb76ea` +
`sshpass`) is the **designated drill+build VM host** per the 2026-07-25 operator ruling, and that ruling
is **realized** — it hosts drill VM `300` (`drill-r50`), so **start there**, not on DooPlex. (The
historical golden-bake `drill.qcow2` still lives on DooPlex and is a bake fixture, not a drill target.)
**Which box is safe to break, and what may be done to each:
`felhom.eu/documentation/runbooks/target-selection.md`** — read it before any destructive test. Both
nodes + the break-glass recipe: `felhom.eu/documentation/operations/nodes.md`. The agent pins the served leaf cert — verify the
fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the
demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon. demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon.
@@ -97,13 +104,17 @@ demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`;
> felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already > felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already
> points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on > points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on
> return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct** > return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct**
> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP** and currently holds > (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP**; re-check its address
> `192.168.0.147` — so the PVE API is at `https://192.168.0.147:8006` there, and **the agent does > rather than trusting one written here (`ip -br addr show vmbr0` — it read `192.168.0.162/24` on
> not run at all**: `localapi` binds the literal `192.168.0.162` → `bind: cannot assign requested > 2026-07-30, and `felhom-pve-lan` from DooPlex is still `No route to host`). Details + findings:
> address` → the service is `failed` and has never started at the remote site. Fixing it means
> editing `listen_addr` in `/etc/felhom-agent/agent.json` **and** the guest's bootstrap endpoint
> (plus the leaf-cert SAN the controller pins) — Viktor GO required. Details + findings:
> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md` > `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
>
> **The "agent does not run at the remote site" warning this block used to carry is RETRACTED
> (2026-07-30) — it was true before R-50 and is false now.** `localapi` no longer binds a LAN literal:
> since the R-50 island migration (2026-07-25) it binds `169.254.253.1:8443` on `vmbr9`, which is
> location-independent by design, and `proxmox.endpoint` is `https://127.0.0.1:8006`. Verified live:
> `systemctl is-active felhom-agent` → `active`, `felhom-agent --version` → 0.115.0, and the per-guest
> local API answered `GET /disks` over the island. No config edit and no Viktor GO are outstanding.
> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11; `pct` commands over SSH > **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11; `pct` commands over SSH
> needed `export MSYS_NO_PATHCONV=1`, and every remote command used > needed `export MSYS_NO_PATHCONV=1`, and every remote command used
@@ -127,7 +138,24 @@ All shippable work commits **directly to `main`**; `main` equals what is deploye
- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than - Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than
guess** when you'd otherwise invent input/output. guess** when you'd otherwise invent input/output.
- **A health check issues no block I/O** — no `statfs`, no `getdents`, no read, write or `fsync`, **not
even behind a timeout**. Liveness is decided from `/proc` and kernel state. The full rule + the
measurement lives in `felhom.eu/CLAUDE.md` "Code quality rules"; it is repeated here because health
checks are written in THIS repo and that file does not load in an agent-only session. R-117 spike §6.3.
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit). - Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
- **Run `python3 scripts/agent_gates.py` from the repo root after ANY change in this repo.** It is
the ONE entry point for this repo's gates. Today it runs one — `reuse_refs_check` over this
repo's `REUSE.md` — and it exists at one gate on purpose: a census on 2026-08-02 found that every
check a `CLAUDE.md` names was passing and two of the four nobody is told to run were failing, and
this repo was the extreme case, with nothing running against it at all and 90 cited paths checked
by no one. It grows when the agent grows a second check. `--fast` selects the gates that touch no
network and no container runtime; today that is all of them. A missing gate is a FAILURE, never a
skip. **The shared `reuse_refs_check.py` lives in `felhom.eu/scripts/` and is never copied here**
— a copy would recreate the drift it detects; an absent sibling clone FAILS the gate.
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It
is per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run
WARNS when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the
session report when you use it.** Both facts are why CI is still owed (`OPEN-ITEMS.md` R-168).
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill. - Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
- **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring` - **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring`
(remote pulls) — English, keys-never-values, durations on outcomes; full rules in (remote pulls) — English, keys-never-values, durations on outcomes; full rules in
+76
View File
@@ -5,6 +5,82 @@
## Current ## Current
- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
`RecoverStaleLockedGuests` miss campaign fault 11 — that recovery acts only on a **stale vzdump
lock** (fault 11's guest was unlocked) and runs **once at agent startup** (fault 11's guest went
down while the agent was already up). `onboot` is the deliberate-stop discriminator and is *not*
invented here: it is already what `stalelock.go` uses for this decision, it is 0 on scratch/golden
guests, and it is what `pve-guests` consults at host boot — so the agent agrees with the platform
instead of keeping a second private definition of "should be running". Retry bounded at 3
(1m/2m/4m) then escalates **once**; an unbounded silent retry loop is the over-correction here.
Live on demo-hp: **120 s unattended** recovery vs the incident's **587 s** with a human; Scenario B
proven (an `onboot:0` guest left stopped throughout). Detail: `REPORT.md`.
- **2026-07-28 — F-LEAK took THREE attempts; v0.108.0 and v0.110.0 are the corrections.** The cause is
structural: `FelhomAgentGuest` is granted at `/pool/felhom` and a guest joins that pool only when its
restore **completes**, so a *failed* restore-test leaves a pool-less guest out of reach (403).
**(1) v0.107.0 pool adoption — REFUTED LIVE:** `PUT /pools/{pool}` also requires `VM.Allocate` on the
VM being added, so membership cannot bootstrap its own authority; removed in **v0.108.0**.
**(2) host-install v1.21.0 per-path `/vms/990000..990009` ACLs — works, but exactly ONCE per slot:**
PVE's destroy calls `AccessControl::remove_vm_access` (`API2/LXC.pm:906`) which deletes every ACL at
`/vms/<vmid>` (`AccessControl.pm:1898`) — **the grant is consumed by the op it authorises**. Caught by
counting ACL rows after the fix, not by reasoning. **(3) v0.110.0 SHIPPED —
`Privileged.DestroyScratchLXC`, the FOURTH root-fenced exception** (was exactly three: keyctl
`pct create`, USB mount/fstab, SMART/sensors). Band enforced in **sudoers literally**
(`pct destroy 99000[0-9] --purge`) + re-checked in code + journal provenance at the caller; none is
consumed by use. API destroy still tried FIRST; band ACLs stay provisioned so the common case needs no
privileged call. **Ships with a sudoers change — deploy `configs/felhom-agent.sudoers` WITH the
binary.** Live: token 403 on a stranded scratch → fenced path removed the guest and all 3 LVs; sudo
PERMITS the band and REFUSES `9201`/`9100`/`9999`/`990010`/`1`, and refuses `pct start 990000` too.
- **2026-07-28 — v0.109.0: the guest-power watchdog got the observable it shipped without.** A
self-correction: v0.107.0's watchdog logged only at startup and when it *acted*, so on a healthy box
its health could be read only from **absence** — F-OBS's exact shape, shipped in the same session
F-OBS was fixed in the controller. Now an INFO summary every 10th sweep carrying
`sweeps_since_boot`/`guests_evaluated`/`currently_stopped`. An **aborted** sweep (unproven
ownership) does not count, or the heartbeat would claim liveness for a watchdog examining nothing.
- **2026-07-28 — v0.106.0: F-CRIT-2 fixed — a failed backup no longer looks like a fresh one.**
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, NEWEST) as a successful
backup, so the tier reported fresh, went **not due**, and was never retried — 7 days of silence on
the real 168h cadence, invisible to both the R-88 breaker (defers only DUE tiers) and the hub
deadline monitor (reads the same freshness). Now only *plausibly complete* entries count, via a
measured floor `minPlausibleArchiveBytes` = 1 MiB; undecidable ⇒ not counted.
**Size is the only tier-agnostic discriminator**`verification` and `encrypted` are absent on
every local (dir) archive and on a good PBS snapshot until verify-new catches up, so gating on
either would reject 100% of local backups and cause fleet-wide backup THRASH. Floor measured:
smallest real backup on the fleet is 612,397,450 B, so 1 MiB leaves 584x headroom (asserted by a
test). Rejections logged at WARN once per volid. Re-tested live by replaying campaign fault 2 on
demo-hp — both directions, incl. a no-thrash window with 91 scheduler ticks as the positive
observable. Deployed on both boxes. Detail: `REPORT.md`.
**Also established:** server-side prune does NOT count phantoms toward `keep-last` (dry-run kept
2 real + the phantom) ⇒ **no retention/data-loss bug** — but it never removes them either, so they
accumulate. Filed as R-99 (LOW).
- **2026-07-25 — v0.95.0 (additive): SMART coverage fixes (spike B+A) + device model.** Union-path
drives (USB/registry) now get SMART via `storage.SmartReader.SMARTForBacking` wired into the localapi
`/disks` union (localapi `Smart` seam); `smartDeviceFor` resolves dm/LVM to the whole disk via
`/sys/block/<dm>/slaves` (recursive, skips >1-disk); the builtin `local` dir on the LVM root gets a
**SMART-only** device from its containing filesystem (never touches backing/durable_id — the
removable-safety guard in build() stays intact); `SmartSummary.ModelName` captured from smartctl. The
watchdog `Known` path stays enrich-free. Consumed by controller v0.171.0. Source of WHERE:
`felhom.eu/documentation/audits/SPIKE-smart-coverage-2026-07-25.md`.
- **2026-07-24 — v0.94.0 (additive): SMART serialized into /disks.** `localapi.DiskInfo` gains
`Smart *hub.SmartSummary` (omitempty), copied from the target's already-computed Observe-time
enrichment when `Health != ""` — no new smartctl load, no endpoint, no sudoers/MinAgent change. The
controller v0.169.0 renders a "Lemezek állapota" card + 6h degradation alert from it; old controllers
ignore it. **NOTE: at the remote-site vacation window the agent is DOWN (localapi binds .162 → fails),
so live /disks-from-real-agent validation is deferred — the field is unit-proven; publish only.**
- **2026-07-22 — v0.93.0 is the FLEET AGENT.** Built, published (sha `a68b2ff73200622e…`),
Day-0-manifest-vouched (MinAgent also 0.93.0, operator-ruled) and deployed to BOTH boxes
(`demo-felhom-8363b5` + `demo-hp-bb76ea`, the latter over G1 break-glass — still no key baked);
clean-restart 5/5 on both, `.bak-0.92.1` retained. Discharges the onboarding runbook §A5
ceremony gate. Record: `felhom.eu/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`.
**The bullet below ("agent is DOWN … deployed 0.90.0") is SUPERSEDED history** — vmbr0 was made
static .162 on 2026-07-20 (F1 mitigation) and the agent has been up since; kept for the record.
- **2026-07-20 — REMOTE SITE until ~2026-08-02; the agent is DOWN there and cannot self-recover.** - **2026-07-20 — REMOTE SITE until ~2026-08-02; the agent is DOWN there and cannot self-recover.**
felhom-pve moved off the home LAN; `ssh felhom-pve` = tailnet `100.70.170.35` (direct, ~37 ms). The felhom-pve moved off the home LAN; `ssh felhom-pve` = tailnet `100.70.170.35` (direct, ~37 ms). The
host is on DHCP and holds `192.168.0.147`, so `localapi`'s literal `192.168.0.162` bind fails with host is on DHCP and holds `192.168.0.147`, so `localapi`'s literal `192.168.0.162` bind fails with
+30 -68
View File
@@ -1,79 +1,41 @@
# REPORT — escrow recovery-code wordlist fix (TASK-E Part 1), 2026-07-21 # REPORT — CI runs the gate entry point on every push (R-168, 2026-08-02)
**Version: v0.93.0. Baseline `935904f` (v0.92.1), clean tree. NOT built or deployed** — no operator **Overwritten** per the standing rule. The prior contents (session 1's gate entry point, same day) have their durable record in `CHANGELOG.md`.
action needed today; this rides the next agent publish train.
## What was wrong **No version bump, no build, no deploy.** This adds `.gitea/workflows/gates.yml` and a CHANGELOG
entry. Nothing compiled changed.
The EFF large wordlist contains exactly four entries that themselves contain the hyphen the words are ## What changed
joined with: `drop-down` (L2009), `felt-tip` (L2528), `t-shirt` (L6640), `yo-yo` (L7748). Drawing one
produced a recovery code that reads as **11 words instead of 10** — ambiguous to transcribe in the one
situation R exists for, a customer reading a code back during a disaster.
**The "known flake" was this defect, not a flaky test.** Per code the hit rate is `.gitea/workflows/gates.yml` — on every push, a Gitea Actions runner obtains this repo at the
`1 (7772/7776)^10 ≈ 0.51%`; over the test's 50 draws that is ≈ **23%**, which is exactly the "fails **exact pushed SHA** (shallow `git fetch`, no `uses:` step anywhere) and runs
~1/5" that had been documented in the `felhom-testing` skill and re-run past for weeks. `python3 scripts/agent_gates.py --fast` and nothing else. The exit code is the job's result: no `|| true`, no
`felhom.eu/REPORT.md` §6 had it at 3/8 in one session. pipe that could swallow it.
## The fix **It REPORTS, it cannot REFUSE**, and the workflow header says so in the pre-push hook's voice: this
repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand
at. The refusing half is `.githooks/pre-push` (per-clone, `--no-verify`-able); this half notices when
that was skipped. Making CI blocking needs branch protection plus a PR workflow — an operator
decision, tracked as `felhom.eu` `OPEN-ITEMS.md` **R-169**.
`joinSafe` filters those four out of the effective wordlist at init. Generation-only. **A failed run emails the operator.** Probe P5 measured that Gitea itself sends **nothing** on a
failed run — no mail, no notification row, no log line — so the workflow sends its own via Resend and
prints the provider's accepted id, making "a message left the machine" an observable. Demonstrated on
a real red run in `felhom.eu`: `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`.
| | before | after | **CI reproduces the workspace's SIBLING LAYOUT on purpose.** This repo's entry point invokes the
|---|---|---| shared `reuse_refs_check.py` that lives in the `felhom.eu` clone next door and is deliberately never
| draw space | 7776 | **7772** | copied here, and this repo's `REUSE.md` cites `hub/internal/store/dr_recipe.go`, which lives in the hub. The workflow clones
| 10-word code entropy | 129.248 bits | **129.241 bits** | `felhom.eu` as a sibling; without it the gate fails **closed** with `gate is MISSING` — correctly,
| margin over the 128-bit floor | 1.248 | **1.241** | but for the wrong reason.
The cost is **0.007 bits**. Both numbers are asserted in `TestEntropyFloorSurvivesFiltering`, so a ## Verification
future wordlist swap cannot quietly move the floor.
**Every recovery code already issued remains valid**, and this was verified rather than assumed: R is First run: run #1, id=9, conclusion **success**, sha `eb991445`.
consumed as a whole passphrase by the PBS scrypt KDF (`Wrap`/`Unwrap`), and `grep` confirms nothing in
the consume path ever splits it. The joiner, the word count and the KDF path are untouched.
## Tests **CI and the local pre-push hook AGREE**, which is the check that mattered: `90 cited paths — exact 88, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: felhom.eu)` — identical to the local run.
`TestGenerateRecoveryCode_EntropyAndFormat` now counts words by **generation count**, not by splitting The runner is unprivileged host-mode, shared with the other three repos on a single owner-scoped
the joined string — conflating those two is what made it flake. It asserts the segmentation property registration (measured: all four repos' tasks claimed by `felhom-gates-runner`). Full probe detail,
separately, because that is the property `joinSafe` actually buys. the security posture and the teardown: `felhom.eu/documentation/audits/SPIKE-ci-runner-2026-08-02.md`
and `felhom.eu/REPORT.md`.
New in `wordlist_test.go`: the four filtered words are pinned by name (a wordlist swap that changes the
set fails loudly); the entropy numbers above; and a production-wiring test that drives the exported
`GenerateRecoveryCode` 500×, so the fix is proven at the entry point and not only in the helper.
### Red-proof — RUN, both deterministic and probabilistic halves
Pre-fix shape restored (`var wordlist = parseWordlist(wordlistRaw)`), suite re-run:
```
--- FAIL: TestWordlistLoaded
effective wordlist should be 7772 words (7776 EFF - 4 hyphenated), got 7776
--- FAIL: TestGenerateRecoveryCode_EntropyAndFormat
joined code must segment into 10 words, got 11 (a drawn word contained "-")
--- FAIL: TestEntropyFloorSurvivesFiltering
filtered entropy moved: 129.248, expected 129.241
--- FAIL: TestGenerateRecoveryCode_NeverContainsAmbiguousWord
code 193 segmented into 11 parts, want 10
```
Fix restored → green; `git diff` clean.
`TestGeneratedCodeSegments_FilteredVsUnfiltered` is the **deterministic** companion: it drives the
generator against a fixture list where every word is hyphenated, so the pre-fix defect reproduces with
probability 1 rather than ~1/5, and shows the same list through `joinSafe` refuses to generate at all.
## Gates
`go build ./... && go vet ./... && go test ./...`**green**. The ex-flaky package was additionally
run 10× consecutively: 10/10 pass (it previously failed ~1 in 5).
## Observations
- The `felhom-testing` skill's "Known flake … re-run before diagnosing; it is not a regression" line
is now removed. It had been actively harmful: it told every future session to dismiss a true
positive. Replaced with the generalised lesson — "known flake, just re-run it" is a diagnosis and
needs evidence like any other.
- `WordlistSize()` now reports the **effective** (filtered) draw space, 7772. `WordlistFilteredOut()`
is new, for audit. Any external consumer expecting a literal 7776 would need updating; there is none
outside the tests.
+21 -3
View File
@@ -88,7 +88,8 @@
| `pinnedTLS` | internal/pbs/pin.go | `pinnedTLS(fingerprint) (*tls.Config, error)` | PBS leaf pinning | Same model as PVE; 64-hex fingerprint normalized | | `pinnedTLS` | internal/pbs/pin.go | `pinnedTLS(fingerprint) (*tls.Config, error)` | PBS leaf pinning | Same model as PVE; 64-hex fingerprint normalized |
| `hub.Client.Report` | internal/hub/client.go | `Report(ctx, *HostReport) (*ControlEnvelope, error)` | the heartbeat | Typed `TransportError`/`HTTPError`, never contain the bearer token | | `hub.Client.Report` | internal/hub/client.go | `Report(ctx, *HostReport) (*ControlEnvelope, error)` | the heartbeat | Typed `TransportError`/`HTTPError`, never contain the bearer token |
| `hub.Loop` + `MultiObserver` | internal/hub/loop.go | `NewLoop(...)`; `MultiObserver(obs...)` | resilient report loop + envelope fan-out | Errors logged, loop continues; interval clamped 603600 s | | `hub.Loop` + `MultiObserver` | internal/hub/loop.go | `NewLoop(...)`; `MultiObserver(obs...)` | resilient report loop + envelope fan-out | Errors logged, loop continues; interval clamped 603600 s |
| `provision.BackHalf.Provision` | internal/provision/backhalf.go | `Provision(ctx, Input) (Result, error)` | guest bootstrap back-half | mint→render→0600 write→chown 100000:100000→`pct set` ro bind→onboot; token NEVER logged/returned | | `provision.BackHalf.Provision` | internal/provision/backhalf.go | `Provision(ctx, Input) (Result, error)` | guest bootstrap back-half | mint→render→0600 write→chown 100000:100000→`pct set` ro bind→onboot; token NEVER logged/returned. Bootstrap `local_api.endpoint` = the caller's `cfg.LocalAPI.ListenAddr` (main.go) — moving the agent bind to the island moves the guest dial for free (R-50, no template) |
| `buildBringUpConfig` island NIC | internal/reconcile/bringup.go | (pure) `BringUpSpec{IslandBridge,IslandGuestAddr}``params["net1"]` | R-50 island control plane | When BOTH island fields are set (from `cfg.LocalAPI`), attaches a static `net1=name=eth1,bridge=<vmbr9>,ip=<.2/30>` (no hwaddr → fresh MAC), so the controller reaches the agent over a fixed private address immune to LAN/DHCP/site moves. Empty = pre-R-50, no net1. All-or-nothing + CIDR enforced in `LocalAPIConfig.Validate`. The guestnet healer is eth0-only (`parseMode` is dev-scoped) so it never touches the static island NIC |
| `reconcile.Queue.Submit` | internal/reconcile/queue.go | `Submit(vmid, fn) <-chan error` | per-guest serialization of ALL mutations | Same vmid strictly FIFO; lanes parallel across guests | | `reconcile.Queue.Submit` | internal/reconcile/queue.go | `Submit(vmid, fn) <-chan error` | per-guest serialization of ALL mutations | Same vmid strictly FIFO; lanes parallel across guests |
| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled | | `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability | | `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
@@ -109,7 +110,10 @@
| Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only | | Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only |
| Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set | | Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set |
| Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction | | Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction |
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests | | Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
| `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) |
| `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool**`BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs |
| `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice`**re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted`**quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s |
| Detached IN-MEMORY verify job (single slot, deliberately unpersisted) | internal/localapi/netverifyjob.go | claim slot sync (single-flight 409) → detached pipeline off baseCtx → auto-rollback on fail; restart ⇒ slot empty ⇒ the CALLER rolls back (Scenario F) — contrast formatjob (persisted+recovered) | | Detached IN-MEMORY verify job (single slot, deliberately unpersisted) | internal/localapi/netverifyjob.go | claim slot sync (single-flight 409) → detached pipeline off baseCtx → auto-rollback on fail; restart ⇒ slot empty ⇒ the CALLER rolls back (Scenario F) — contrast formatjob (persisted+recovered) |
| Optional dependency degradation | internal/localapi/server.go (`Options`) | nil dep ⇒ endpoint answers "not configured" (503), never a crash | | Optional dependency degradation | internal/localapi/server.go (`Options`) | nil dep ⇒ endpoint answers "not configured" (503), never a crash |
| Version channel (v0.82.0) | internal/localapi/server.go (`Options.AgentVersion`; `Handler()` mux wrap) | sets `X-Felhom-Agent-Version` on EVERY response (all routes/statuses, incl. auth-fail/404) — the controller's capability-comparison source; empty version ⇒ header omitted | | Version channel (v0.82.0) | internal/localapi/server.go (`Options.AgentVersion`; `Handler()` mux wrap) | sets `X-Felhom-Agent-Version` on EVERY response (all routes/statuses, incl. auth-fail/404) — the controller's capability-comparison source; empty version ⇒ header omitted |
@@ -143,11 +147,16 @@
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). | | `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go | | `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go | | `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,t)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test rotation state, persisted (atomic tmp+rename) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target) (string,error)` | The per-run restore-test spec + per-tier candidate lookup (R-85) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", nil)`**`""` is NOT an error**, or every fresh box looks broken for its first week. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go | | `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go | | `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls | | `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment | | `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind | | `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
| `hub.AddressEnumerator` (v0.119.0) | internal/hub/hostaddr.go | **defaults to the REAL `systemInterfaces`** when `Collector.addrEnum` is nil — deliberately inverting the nil-reporter-means-off convention, because this stanza has no config gate and a forgotten wiring call would otherwise ship silently empty (the inert-seam shape, four instances on record) | internal/hub/hostaddr_test.go drives fixtures TRANSCRIBED from `ip -o addr show` on demo-felhom AND demo-hp, including the address-less veth/NIC rows — the "no denylist needed" claim rests on those rows really being empty, so omitting them would prove the claim by assuming it. `filterHostAddresses` keeps GLOBAL UNICAST only: one predicate that drops loopback, `fe80::/10`, and `169.254/16` — the last being the R-50 island literal, identical on every box and actively misleading if surfaced |
| `reconcile.OpVerifier` | internal/reconcile/gate.go | `*authz.Verifier` | fake verifier in internal/reconcile gate tests | | `reconcile.OpVerifier` | internal/reconcile/gate.go | `*authz.Verifier` | fake verifier in internal/reconcile gate tests |
| `signedjobs.WipeOps` / `Executor` (`ExecutorChain`) | internal/signedjobs/wipe.go + runner.go | `*storage.SudoHostOps`; `WipeExecutor`+`DecommissionExecutor` | internal/signedjobs wipe/runner/decommission tests | | `signedjobs.WipeOps` / `Executor` (`ExecutorChain`) | internal/signedjobs/wipe.go + runner.go | `*storage.SudoHostOps`; `WipeExecutor`+`DecommissionExecutor` | internal/signedjobs wipe/runner/decommission tests |
| `hub.reporter` / `collectorIface` / `EnvelopeObserver` | internal/hub/loop.go | `*hub.Client`, `*hub.Collector`; `desired.Syncer` + `signedjobs.Runner` | `fakeReporter`/`fakeCollector` internal/hub/loop_test.go; `recordingReporter` loop_logtail_test.go | | `hub.reporter` / `collectorIface` / `EnvelopeObserver` | internal/hub/loop.go | `*hub.Client`, `*hub.Collector`; `desired.Syncer` + `signedjobs.Runner` | `fakeReporter`/`fakeCollector` internal/hub/loop_test.go; `recordingReporter` loop_logtail_test.go |
@@ -166,6 +175,14 @@
- **New privileged host op**: validate args (internal/storage/validate.go style) → exec via `Runner` → add a `Cmnd_Alias` to configs/felhom-agent.sudoers → add a probe vector to internal/capability/manifest.go (so degradation is visible) → ship sudoers with the binary. - **New privileged host op**: validate args (internal/storage/validate.go style) → exec via `Runner` → add a `Cmnd_Alias` to configs/felhom-agent.sudoers → add a probe vector to internal/capability/manifest.go (so degradation is visible) → ship sudoers with the binary.
- **New reconcile action**: `ActionKind` + `classOfAction` (internal/reconcile/classify.go), plan emission in internal/reconcile/plan.go; destructive ⇒ gate handles it automatically. - **New reconcile action**: `ActionKind` + `classOfAction` (internal/reconcile/classify.go), plan emission in internal/reconcile/plan.go; destructive ⇒ gate handles it automatically.
- **Hub-report field**: extend `hub.HostReport` (internal/hub/report.go) + `Collector` — hub side must mirror + allowlist it (cross-repo). - **Hub-report field**: extend `hub.HostReport` (internal/hub/report.go) + `Collector` — hub side must mirror + allowlist it (cross-repo).
- **DR-recipe section (host-half)**: add the field to `DRRecipeHostHalf` (internal/hub/dr_recipe.go) **AND** to
the hub's `hostHalfShape` + `AssembledRecipe` (felhom.eu `hub/internal/store/dr_recipe.go`). Those two
hub structs are **ALLOW-LISTS**: a section only the agent knows about is stored intact and silently
dropped before any operator sees it — that is R-122, which cost `offsite_restic` its entire existence.
Then update BOTH copies of `testdata/host-report.golden.json` (byte-identical, cross-repo) and extend
`TestAssembleDRRecipe_CarriesEveryEmittedSection`. **A recipe field that cannot be resolved records an
explicit unknown with a reason — never a default, an empty string, or a placeholder** (`DRState*` /
`DRReason*`); a recipe read during a rebuild must not present a guess as a fact.
- **Envelope-driven behavior**: implement `hub.EnvelopeObserver`, add to the `MultiObserver` in cmd/felhom-agent/main.go. - **Envelope-driven behavior**: implement `hub.EnvelopeObserver`, add to the `MultiObserver` in cmd/felhom-agent/main.go.
- **Selftest mode**: `selftestFlag` + `runSelftest*` in cmd/felhom-agent/main.go. - **Selftest mode**: `selftestFlag` + `runSelftest*` in cmd/felhom-agent/main.go.
- **Config**: internal/config/config.go (`Load` + `applyEnv` `FELHOM_AGENT_*` overlay; keep secrets out of `Redacted()` output). - **Config**: internal/config/config.go (`Load` + `applyEnv` `FELHOM_AGENT_*` overlay; keep secrets out of `Redacted()` output).
@@ -174,8 +191,9 @@
- Two lsblk `-J` parsers with near-identical structs: `parseLsblkDevice`/`lsblkDevice` (internal/storage/hostops.go) vs `parseLsblkNodes`/`lsblkDev` (internal/storage/claim.go). - Two lsblk `-J` parsers with near-identical structs: `parseLsblkDevice`/`lsblkDevice` (internal/storage/hostops.go) vs `parseLsblkNodes`/`lsblkDev` (internal/storage/claim.go).
- Two smartctl `-a -j` paths: `SudoHostOps.SMART` (internal/storage/hostops.go, parsed `hub.SmartSummary`) vs `Privileged.SMART` (internal/proxmox/privileged.go, raw map). - Two smartctl `-a -j` paths: `SudoHostOps.SMART` (internal/storage/hostops.go, parsed `hub.SmartSummary`) vs `Privileged.SMART` (internal/proxmox/privileged.go, raw map).
- **SMART device resolution (v0.95.0):** `smartDeviceFor` (internal/storage/observe.go) resolves partition→disk AND dm/LVM→disk (`dmWholeDisk` in internal/storage/smartdev.go, via `/sys/block/<dm>/slaves`, `sysBlockRoot` test seam). `storage.SmartReader.SMARTForBacking` is the shared read the localapi `/disks` union path uses (Fix B) — do NOT re-implement smartctl parsing. The builtin-`local` SMART device comes from `containingMountDevice` (SMART-only; never feeds backing/durable_id).
- Atomic tmp+rename JSON store implemented 3×: `IntentStore.saveLocked` (internal/storage/intent.go), `FormatJobStore.save` (internal/localapi/formatjob.go), `GuestBindStore.saveLocked` (internal/localapi/guestbindstore.go) — comments say "mirrors", no shared helper. - Atomic tmp+rename JSON store implemented 3×: `IntentStore.saveLocked` (internal/storage/intent.go), `FormatJobStore.save` (internal/localapi/formatjob.go), `GuestBindStore.saveLocked` (internal/localapi/guestbindstore.go) — comments say "mirrors", no shared helper.
- `run(ctx, name, args...) error` stderr-wrapping helper duplicated 4×: `SudoHostOps.run`, `Privileged.run`, `BackHalf.run` (internal/provision/backhalf.go), `GuestBinder.run` (internal/localapi/guestbind.go). - `run(ctx, name, args...) error` stderr-wrapping helper duplicated 4×: `SudoHostOps.run`, `Privileged.run`, `BackHalf.run` (internal/provision/backhalf.go), `GuestBinder.run` (internal/localapi/guestbind.go).
- Several independent /proc mount-table readers: `SudoHostOps.mountedSet` (internal/storage/hostops.go), `ProcHostReader.Mounts` (internal/storage/hostread.go), `isHostMountpoint` + `countHostMounts` (internal/localapi/intermediary.go). - Several independent /proc mount-table readers: `SudoHostOps.mountedSet` (internal/storage/hostops.go), `ProcHostReader.Mounts` (internal/storage/hostread.go). **In localapi they were unified in v0.117.0**: `isHostMountpoint` and `countHostMounts` are now one-liners over `hostMountEntries`, the single parser that also yields devno/fstype/super-options for `bindLiveness`.
- Deliberate mirror: `antiRetargetResolveExpect` (internal/localapi/wipe_reresolve.go) duplicates `WipeExecutor.Execute` steps 13 (internal/signedjobs/wipe.go) across packages. - Deliberate mirror: `antiRetargetResolveExpect` (internal/localapi/wipe_reresolve.go) duplicates `WipeExecutor.Execute` steps 13 (internal/signedjobs/wipe.go) across packages.
- `stableParentDir` literal duplicated in internal/provision/backhalf.go to avoid a provision→localapi import edge (commented as intentional); `trim` (internal/storage/hostops.go) vs `trimBody` (internal/proxmox/errors.go) output-truncation twins. - `stableParentDir` literal duplicated in internal/provision/backhalf.go to avoid a provision→localapi import edge (commented as intentional); `trim` (internal/storage/hostops.go) vs `trimBody` (internal/proxmox/errors.go) output-truncation twins.
+23
View File
@@ -0,0 +1,23 @@
package main
import "testing"
// R-82 live regression (2026-07-26): the restore-test derived its tier from the CONFIGURED default
// target instead of the archive's own storage. Restoring a `felhom-pbs:` archive on a box whose
// primary target is "local" was classified "local" → the 10-minute local wait instead of the
// generous PBS one → the wait expired mid-restore at 600s against a 14.46 GB WAN restore, teardown
// fired at a still-restoring guest, and the scratch leaked.
func TestArchiveStorageID(t *testing.T) {
cases := []struct{ in, want string }{
{"felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z", "felhom-pbs"},
{"local:backup/vzdump-lxc-9201-2026_07_26-09_03_19.tar.zst", "local"},
{"", ""},
{"no-prefix", ""},
{":leading-colon", ""}, // i>0 guard: a leading colon is not a storage id
}
for _, c := range cases {
if got := archiveStorageID(c.in); got != c.want {
t.Fatalf("archiveStorageID(%q) = %q, want %q", c.in, got, c.want)
}
}
}
+156 -15
View File
@@ -165,7 +165,7 @@ func main() {
showVersion bool showVersion bool
) )
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)") flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-sysdata-grow/-cores/-memory; keeps the guest)") flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup|bring-up") flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup|bring-up")
flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only") flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only")
flag.StringVar(&archive, "archive", "", "for --selftest=restore-test|bring-up: the backup volid to restore (restore-test: default newest on the local target)") flag.StringVar(&archive, "archive", "", "for --selftest=restore-test|bring-up: the backup volid to restore (restore-test: default newest on the local target)")
@@ -175,8 +175,13 @@ func main() {
flag.IntVar(&rootfsGrow, "rootfs-grow", 0, "for --selftest=bring-up|provision: grow the OS rootfs by this many GiB after restore (0 = keep golden size)") flag.IntVar(&rootfsGrow, "rootfs-grow", 0, "for --selftest=bring-up|provision: grow the OS rootfs by this many GiB after restore (0 = keep golden size)")
flag.IntVar(&dataVolGrow, "datavol-grow", 0, "for --selftest=bring-up|provision: grow the golden's Docker-data volume (mp0) by this many GiB (0 = keep golden size)") flag.IntVar(&dataVolGrow, "datavol-grow", 0, "for --selftest=bring-up|provision: grow the golden's Docker-data volume (mp0) by this many GiB (0 = keep golden size)")
flag.StringVar(&dataVolMount, "datavol-mount", "", "for --selftest=bring-up|provision: the mpN slot of the Docker-data volume to grow (default mp0)") flag.StringVar(&dataVolMount, "datavol-mount", "", "for --selftest=bring-up|provision: the mpN slot of the Docker-data volume to grow (default mp0)")
flag.IntVar(&sysDataGrow, "sysdata-grow", 0, "for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive) by this many GiB (0 = keep golden size)") // R-165: the second volume is gone (build-golden.sh v3.0.0 ships ONE). These two flags are kept
flag.StringVar(&sysDataMount, "sysdata-mount", "", "for --selftest=bring-up|provision: the mpN slot of the user-data volume to grow (default mp1)") // ACCEPTED because felhom-host-install.sh passes -sysdata-grow and an installer and an agent do not
// upgrade in the same instant — removing them would make every install fail on an unknown flag.
// -sysdata-grow is NOT inert: its GiB are folded into the single volume's grow (bringup.go 4b), so
// an old installer still produces the same total capacity. -sysdata-mount selects nothing.
flag.IntVar(&sysDataGrow, "sysdata-grow", 0, "DEPRECATED (R-165): there is one data volume now; this value is ADDED to -datavol-grow rather than growing a second volume. Kept so an older felhom-host-install.sh keeps working")
flag.StringVar(&sysDataMount, "sysdata-mount", "", "DEPRECATED (R-165): ignored — there is no second volume to select")
flag.IntVar(&cores, "cores", 0, "for --selftest=bring-up|provision: cap the guest to N CPU cores (0 = keep golden default). Applied in the pre-start config PUT.") flag.IntVar(&cores, "cores", 0, "for --selftest=bring-up|provision: cap the guest to N CPU cores (0 = keep golden default). Applied in the pre-start config PUT.")
flag.IntVar(&memoryMB, "memory", 0, "for --selftest=bring-up|provision: cap the guest RAM to N MiB (0 = keep golden default). Applied pre-start.") flag.IntVar(&memoryMB, "memory", 0, "for --selftest=bring-up|provision: cap the guest RAM to N MiB (0 = keep golden default). Applied pre-start.")
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)") flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
@@ -462,6 +467,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
pbsTargets := pbsTargetsFromPVE(cfg, px, logger) pbsTargets := pbsTargetsFromPVE(cfg, px, logger)
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger) pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger) collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target
// Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent // Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent
// depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the // depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the
// command), so it uses a DIRECT runner regardless of the agent's privileged mode. Probe once at // command), so it uses a DIRECT runner regardless of the agent's privileged mode. Probe once at
@@ -649,7 +655,11 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// Self-restore-test scheduler (slice 6): the fourth daemon goroutine. Runs the restore- // Self-restore-test scheduler (slice 6): the fourth daemon goroutine. Runs the restore-
// test on the configured cadence (default 24h). Disabled cleanly when the cadence is off // test on the configured cadence (default 24h). Disabled cleanly when the cadence is off
// OR the scratch band / restore storage is misconfigured — the daemon still runs. // OR the scratch band / restore storage is misconfigured — the daemon still runs.
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, logger) // R-85: persisted per-tier restore-test state + the host-wide one-heavy-op gate, both shared
// with the local API so a backup and a restore-test can never run together.
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
heavyOps := &backup.InFlight{}
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
// PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free, // PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free,
// ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot // ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot
@@ -757,7 +767,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
return false return false
}, },
} }
localSrv := buildLocalAPIServer(cfg, px, backupStore, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens) localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localTokens != nil { if localTokens != nil {
defer localTokens.Close() defer localTokens.Close()
} }
@@ -1053,6 +1063,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// the dangling snapshot → start iff onboot). Runs BEFORE the backup loop starts, so a present // the dangling snapshot → start iff onboot). Runs BEFORE the backup loop starts, so a present
// backup lock is stale by definition (guarded by a no-vzdump-running check; fail-safe otherwise). // backup lock is stale by definition (guarded by a no-vzdump-running check; fail-safe otherwise).
localSrv.RecoverStaleLockedGuests(ctx) localSrv.RecoverStaleLockedGuests(ctx)
// F-REBOOT: the startup recovery above only covers a guest left LOCKED by an interrupted
// backup. A guest that simply ends up stopped-and-unlocked (a `pct reboot` whose shutdown
// half completed and whose start half never fired — Campaign 8 fault 11, 9m47s of total
// appliance outage with nothing retrying) needs a PERIODIC check. onboot is the "should be
// running" signal, so a deliberately stopped guest is never touched.
go localSrv.WatchGuestPower(ctx)
go func() { errc <- localSrv.Run(ctx) }() go func() { errc <- localSrv.Run(ctx) }()
} }
if lanLoop != nil { if lanLoop != nil {
@@ -1164,6 +1180,34 @@ func pbsTargetsFromPVE(cfg config.Config, px *proxmox.Client, logger *slog.Logge
// storageTier returns the restore-test source tier for a backup storage id: "pbs" when that // storageTier returns the restore-test source tier for a backup storage id: "pbs" when that
// storage is a PBS datastore, else "local". Best-effort (a lookup failure → "local"). // storage is a PBS datastore, else "local". Best-effort (a lookup failure → "local").
// archiveStorageID returns the storage a volid lives on — "felhom-pbs" from
// "felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z". Empty when there is no storage prefix.
func archiveStorageID(volid string) string {
if i := strings.Index(volid, ":"); i > 0 {
return volid[:i]
}
return ""
}
// restoreTierForArchive derives the restore tier from THE ARCHIVE'S OWN STORAGE, falling back to
// the configured default target only when the volid carries no storage prefix.
//
// R-82 (found live 2026-07-26): this used to read the tier from cfg.Backup.BackupTarget(), i.e. the
// PRIMARY tier's target. Restoring a `felhom-pbs:` archive on a box whose primary is "local" was
// therefore classified "local" and got the 10-MINUTE local wait instead of the generous PBS one —
// the wait expired mid-restore at 600s, teardown fired against a still-restoring guest, and the
// scratch leaked. Exactly the failure RestoreTestSpec.RestoreTaskTimeout's doc comment predicts.
//
// The tier-aware machinery was already correct; it was fed the wrong input. With more than one tier
// configured, "the configured target" is no longer a proxy for "the tier this archive belongs to".
func restoreTierForArchive(ctx context.Context, px *proxmox.Client, archive, fallbackTarget string) string {
id := archiveStorageID(archive)
if id == "" {
id = fallbackTarget
}
return storageTier(ctx, px, id)
}
func storageTier(ctx context.Context, px *proxmox.Client, storageID string) string { func storageTier(ctx context.Context, px *proxmox.Client, storageID string) string {
stores, err := px.ListStorage(ctx) stores, err := px.ListStorage(ctx)
if err != nil { if err != nil {
@@ -1200,11 +1244,35 @@ func readTrimmed(path string) (string, error) {
return s, nil return s, nil
} }
// primaryBackupTargetOf returns the resolver the DR recipe uses to name WHICH storage holds this box's
// local whole-guest archives (R-109).
//
// It reads the PRIMARY tier out of cfg.Backup.BackupTiers() rather than calling BackupTarget() directly.
// Both return the same string today — BackupTiers() builds tier 0 from BackupTarget() — but the tier
// list is the function the scheduler itself consults, so if primary-tier derivation ever changes the
// recipe follows it instead of quietly disagreeing with the backup. One state, one owner.
//
// cfg is captured BY VALUE on purpose: that is the daemon-start snapshot, which is the config actually
// in effect. See SetBackupTargetResolver for why re-reading agent.json here would be wrong.
func primaryBackupTargetOf(cfg config.Config) func() hub.ConfiguredBackupTarget {
return func() hub.ConfiguredBackupTarget {
tiers, _ := cfg.Backup.BackupTiers() // warnings are logged where the tiers are armed
for _, t := range tiers {
if t.Primary {
return hub.ConfiguredBackupTarget{StorageID: t.TargetID, Known: true}
}
}
// Unreachable with today's BackupTiers (tier 0 is always primary), and if that ever stops being
// true the recipe says "I could not tell" rather than picking a tier at random.
return hub.ConfiguredBackupTarget{}
}
}
// buildRestoreTestScheduler constructs the restore-test cadence scheduler from config. It // buildRestoreTestScheduler constructs the restore-test cadence scheduler from config. It
// disables the cadence (returns a scheduler that just waits) when the cadence is off or the // disables the cadence (returns a scheduler that just waits) when the cadence is off or the
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the // scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
// machinery still works on-demand via --selftest=restore-test. // machinery still works on-demand via --selftest=restore-test.
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, logger *slog.Logger) *backup.Scheduler { func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, logger *slog.Logger) *backup.Scheduler {
cadence := cfg.Backup.RestoreTestCadence() cadence := cfg.Backup.RestoreTestCadence()
if cadence > 0 { if cadence > 0 {
if err := cfg.Backup.ValidateForRestoreTest(); err != nil { if err := cfg.Backup.ValidateForRestoreTest(); err != nil {
@@ -1215,12 +1283,29 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
min, max := cfg.Backup.ScratchBand() min, max := cfg.Backup.ScratchBand()
target := cfg.Backup.BackupTarget() target := cfg.Backup.BackupTarget()
runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger) runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger)
// Every configured tier is a rotation candidate, not just the primary.
cfgTiers, _ := cfg.Backup.BackupTiers() // warnings already logged where the tiers are armed
tierIDs := make([]string, 0, len(cfgTiers))
for _, t := range cfgTiers {
tierIDs = append(tierIDs, t.TargetID)
}
return backup.NewScheduler(backup.SchedulerOptions{ return backup.NewScheduler(backup.SchedulerOptions{
Runner: engine, Runner: engine,
Pick: runner.PickRestoreCandidate, Pick: runner.PickRestoreCandidate,
Store: store, Store: store,
Spec: func() reconcile.RestoreTestSpec { // R-85 (1.1): the spec is built PER RUN, from the archive that was picked.
tier := storageTier(context.Background(), px, target) //
// This used to be an immediately-invoked function, so storageTier() and
// restoreTaskTimeout() ran ONCE at daemon start and their result was reused for every run
// forever. That froze the tier — and with it the timeout — making an offsite restore-test
// impossible to schedule, and leaving any storage-type or config change stale until the
// daemon restarted.
//
// The tier comes from the ARCHIVE (restoreTierForArchive, the v0.100.0 rule), never from
// the configured target: config-derived was what classified a PBS archive as "local" and
// killed a 14.46 GB WAN restore at the 10-minute local bound.
Spec: func(ctx context.Context, archive string) reconcile.RestoreTestSpec {
tier := restoreTierForArchive(ctx, px, archive, target)
return reconcile.RestoreTestSpec{ return reconcile.RestoreTestSpec{
RestoreStorage: cfg.Backup.RestoreStorage, RestoreStorage: cfg.Backup.RestoreStorage,
ScratchMin: min, ScratchMin: min,
@@ -1228,9 +1313,17 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
SourceTier: tier, SourceTier: tier,
RestoreTaskTimeout: restoreTaskTimeout(cfg, tier), RestoreTaskTimeout: restoreTaskTimeout(cfg, tier),
} }
}(), },
Cadence: cadence, Cadence: cadence,
Logger: logger, Logger: logger,
// R-85: rotate across EVERY configured tier, oldest-proven first (operator ruling, Option 1).
// Before this the scheduler only ever saw cfg.Backup.BackupTarget(), so the offsite tier's
// archives were never candidates and the DR tier went unproven for its whole existence.
Tiers: tierIDs,
TierPick: runner.PickRestoreCandidateOn,
State: rtState,
InFlight: inFlight,
}) })
} }
@@ -1239,7 +1332,7 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
// leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the // leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the
// daemon — the host still reports/reconciles; only the controller channel is unavailable until // daemon — the host still reports/reconciles; only the controller channel is unavailable until
// fixed. The opened token store is returned via outTokens so the caller can Close it. // fixed. The opened token store is returned via outTokens so the caller can Close it.
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server { func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
if !cfg.LocalAPI.Enabled() { if !cfg.LocalAPI.Enabled() {
return nil return nil
} }
@@ -1269,7 +1362,41 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
} }
// v0.48.0: ride the served leaf fp on every host report so the hub can detect a re-key fleet-wide. // v0.48.0: ride the served leaf fp on every host report so the hub can detect a re-key fleet-wide.
collector.SetLeafFingerprint(fp) collector.SetLeafFingerprint(fp)
runner := backup.NewBackupRunner(px, cfg.Backup.BackupTarget(), "", "felhom local-api", cfg.Backup.PruneBackupsSpec(), logger) // R-82: ONE RUNNER PER TIER. The runner holds its target, mode, notes and retention as
// immutable construction state, and localPruneSpec reads that retention — so parameterising a
// single runner by target would risk a call pairing tier A's target with tier B's retention.
// One runner per tier keeps each tier's policy structurally inseparable from its target.
backupTiers, tierWarnings := cfg.Backup.BackupTiers()
for _, wmsg := range tierWarnings {
// LOUD on purpose: a silently dropped backup tier is an "applied and empty" DR tier, which
// is the exact fault R-82 exists to fix. Never downgrade this to DEBUG.
logger.Error("backup tier REJECTED — this tier will never run", "detail", wmsg)
}
apiTiers := make([]localapi.BackupTier, 0, len(backupTiers))
var runner *backup.BackupRunner
for _, t := range backupTiers {
prune := ""
if t.KeepLast > 0 {
prune = fmt.Sprintf("keep-last=%d", t.KeepLast)
}
// Pruning a PBS target is allowed ONLY for an additional tier with an explicit keep_last
// (the primary's target AND retention both default, so it could prune the DR by accident).
allowPBSPrune := !t.Primary && t.KeepLast > 0
r := backup.NewBackupRunnerFull(px, t.TargetID, "", "felhom local-api", prune, t.WaitTimeout, allowPBSPrune, logger)
if t.Primary {
runner = r
}
apiTiers = append(apiTiers, localapi.BackupTier{
TargetID: t.TargetID,
Cadence: t.Cadence,
WaitTimeout: t.WaitTimeout,
Primary: t.Primary,
Service: r,
})
logger.Info("backup tier armed", "target", t.TargetID, "cadence", t.Cadence.String(),
"keep_last", t.KeepLast, "wait_timeout", t.WaitTimeout.String(),
"prune_pbs_allowed", allowPBSPrune, "primary", t.Primary)
}
// Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown // Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown
// (same fenced ExecRunner the host-storage + provision back-half use). // (same fenced ExecRunner the host-storage + provision back-half use).
gaMode := proxmox.RunnerMode(cfg.Privileged.Mode) gaMode := proxmox.RunnerMode(cfg.Privileged.Mode)
@@ -1283,10 +1410,13 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
Guests: px, Guests: px,
Backups: runner, Backups: runner,
BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary
InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F)
Store: store, Store: store,
Storage: observer, Storage: observer,
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages) DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
Tokens: tokens, Tokens: tokens,
BackupCadence: cfg.Backup.BackupCadence(), BackupCadence: cfg.Backup.BackupCadence(),
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate. // Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
@@ -1296,7 +1426,12 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
GuestAttach: guestBinder, // slice 10 P2: bind enrolled data drives into the guest GuestAttach: guestBinder, // slice 10 P2: bind enrolled data drives into the guest
Memory: px, // v0.90.0 R-24: guest RAM resize (SetConfig live cgroup apply) Memory: px, // v0.90.0 R-24: guest RAM resize (SetConfig live cgroup apply)
// Network storage (NAS) — Part A1: the privileged host network-mount surface (NFS/SMB automount). // Network storage (NAS) — Part A1: the privileged host network-mount surface (NFS/SMB automount).
NetStorage: hostOps, NetStorage: hostOps,
// E-2a: the fenced root shim for the backup-target move. Same runner mode as every other
// privileged call; the sudoers vector is what actually bounds it.
Privileged: &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath},
ConfigPath: cfg.SourcePath,
StateDir: cfg.WGTunnel.WithDefaults().StateDir,
SmbCredsDir: cfg.Privileged.SmbCredsDir, SmbCredsDir: cfg.Privileged.SmbCredsDir,
ControllerSwap: guestBinder, // Phase 1: agentic controller update — in-guest image swap ControllerSwap: guestBinder, // Phase 1: agentic controller update — in-guest image swap
// F2-b: recover a guest left with a stale vzdump lock by a reboot-during-backup. Reads + start // F2-b: recover a guest left with a stale vzdump lock by a reboot-during-backup. Reads + start
@@ -1455,6 +1590,10 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger)
// the selftest reflects exactly what a freshly-restarted daemon's first collect emits. // the selftest reflects exactly what a freshly-restarted daemon's first collect emits.
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargetsFromPVE(cfg, px, logger), pbs.NewSnapshotStore(), pbs.DefaultLiveSnapshotTimeout, logger) pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargetsFromPVE(cfg, px, logger), pbs.NewSnapshotStore(), pbs.DefaultLiveSnapshotTimeout, logger)
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, nil, nil, pbsReporter, cfg.Hub.HostID, version, logger) collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, nil, nil, pbsReporter, cfg.Hub.HostID, version, logger)
// R-109: wire the backup-target resolver here TOO. Without it selftest=hub would print a recipe whose
// backup_target reads unknown/agent_backup_config_unavailable while the daemon's is resolved — and
// this one-shot exists precisely so "the report it would send" can be trusted to match.
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg))
ctx, cancel := context.WithTimeout(ctx, 60*time.Second) ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel() defer cancel()
@@ -1666,7 +1805,7 @@ func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog
} }
min, max := cfg.Backup.ScratchBand() min, max := cfg.Backup.ScratchBand()
fmt.Printf(" restoring %s into scratch band [%d,%d] on %s …\n", archive, min, max, cfg.Backup.RestoreStorage) fmt.Printf(" restoring %s into scratch band [%d,%d] on %s …\n", archive, min, max, cfg.Backup.RestoreStorage)
rtTier := storageTier(ctx, px, target) rtTier := restoreTierForArchive(ctx, px, archive, target)
res := engine.RunRestoreTest(ctx, reconcile.RestoreTestSpec{ res := engine.RunRestoreTest(ctx, reconcile.RestoreTestSpec{
Archive: archive, RestoreStorage: cfg.Backup.RestoreStorage, Archive: archive, RestoreStorage: cfg.Backup.RestoreStorage,
ScratchMin: min, ScratchMax: max, SourceTier: rtTier, ScratchMin: min, ScratchMax: max, SourceTier: rtTier,
@@ -1771,6 +1910,7 @@ func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Log
Cores: sizing.Cores, MemoryMB: sizing.MemoryMB, Cores: sizing.Cores, MemoryMB: sizing.MemoryMB,
RootfsGrowGB: sizing.RootfsGrowGB, DataVolGrowGB: sizing.DataVolGrowGB, DataVolMount: sizing.DataVolMount, RootfsGrowGB: sizing.RootfsGrowGB, DataVolGrowGB: sizing.DataVolGrowGB, DataVolMount: sizing.DataVolMount,
SysDataGrowGB: sizing.SysDataGrowGB, SysDataMount: sizing.SysDataMount, SysDataGrowGB: sizing.SysDataGrowGB, SysDataMount: sizing.SysDataMount,
IslandBridge: cfg.LocalAPI.IslandBridge, IslandGuestAddr: cfg.LocalAPI.IslandGuestAddr, // R-50 island NIC (both empty = pre-R-50)
} }
fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage) fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage)
res := engine.RunBringUp(ctx, spec) res := engine.RunBringUp(ctx, spec)
@@ -1934,6 +2074,7 @@ func runSelftestProvision(ctx context.Context, cfg config.Config, logger *slog.L
Cores: a.sizing.Cores, MemoryMB: a.sizing.MemoryMB, Cores: a.sizing.Cores, MemoryMB: a.sizing.MemoryMB,
RootfsGrowGB: a.sizing.RootfsGrowGB, DataVolGrowGB: a.sizing.DataVolGrowGB, DataVolMount: a.sizing.DataVolMount, RootfsGrowGB: a.sizing.RootfsGrowGB, DataVolGrowGB: a.sizing.DataVolGrowGB, DataVolMount: a.sizing.DataVolMount,
SysDataGrowGB: a.sizing.SysDataGrowGB, SysDataMount: a.sizing.SysDataMount, SysDataGrowGB: a.sizing.SysDataGrowGB, SysDataMount: a.sizing.SysDataMount,
IslandBridge: cfg.LocalAPI.IslandBridge, IslandGuestAddr: cfg.LocalAPI.IslandGuestAddr, // R-50 island NIC (both empty = pre-R-50)
}) })
if res.Err != nil || !res.Pass { if res.Err != nil || !res.Pass {
fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err) fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err)
+28
View File
@@ -0,0 +1,28 @@
package main
import (
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
)
// COMPILE-TIME WITNESSES for OPTIONAL interfaces that are satisfied by a RUNTIME type assertion.
//
// WHY THIS FILE EXISTS. `localapi.BackupArchiveLister` is asserted at server.go's `newestArchiveOn`
// via `tier.Service.(BackupArchiveLister)`. A failed assertion does not error — it degrades to
// `archiveAbsent`, i.e. the pre-R-84 "ask the in-memory record only" behaviour. That degrade is
// SILENT and it is behaviour-relevant: it is exactly the R-84 bug (a cold store after a restart
// reading as "no backup ever") coming back, with nothing in any log to say so.
//
// The precedent is not hypothetical. During R-88 Part 2 the controller's `quiesceBackend` stopped
// satisfying `quiesce.TieredBackend` when a signature changed, and `go build` AND `go vet` both
// passed — because the interface is only ever asserted at runtime. Every box would have degraded to
// the single-tier path, losing R-82's multi-tier backups, with no error anywhere. It was caught by
// accident.
//
// A witness costs one line and converts that class of failure from a silent production degrade into
// a compile error.
//
// THIS DOES NOT MAKE THE INTERFACE REQUIRED. The optionality is deliberate — it is what lets a
// BackupService without a lister still work. The witness pins the IMPLEMENTATION (this concrete type
// really does satisfy it), not the CONTRACT.
var _ localapi.BackupArchiveLister = (*backup.BackupRunner)(nil)
+8 -2
View File
@@ -48,10 +48,16 @@
}, },
"local_api": { "local_api": {
"enable": true, "enable": true,
"listen_addr": "192.168.0.162:8443", "listen_addr": "169.254.253.1:8443",
"cert_file": "/var/lib/felhom-agent/local-api.crt", "cert_file": "/var/lib/felhom-agent/local-api.crt",
"key_file": "/var/lib/felhom-agent/local-api.key", "key_file": "/var/lib/felhom-agent/local-api.key",
"token_store": "/var/lib/felhom-agent/local-tokens.log" "token_store": "/var/lib/felhom-agent/local-tokens.log",
"island_bridge": "vmbr9",
"island_guest_addr": "169.254.253.2/30"
},
"lan_resolver": {
"enable": true,
"host_ip": "192.168.0.162"
}, },
"log_level": "info" "log_level": "info"
} }
+110 -39
View File
@@ -26,20 +26,42 @@
# Build-time registry login for the controller pull (used ONCE inside the build guest, then logged # Build-time registry login for the controller pull (used ONCE inside the build guest, then logged
# out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment. # out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment.
# #
# OS / Docker-data SPLIT (storage-split slice): the golden is built with a SMALL OS rootfs and a # OS / DATA SPLIT, and since v3.0.0 ONE DATA VOLUME (R-165, decision D-a + variant V-c).
# SEPARATE Docker-data volume mounted at /var/lib/docker (mp0, backup=1). The baked controller + #
# infra images land on that volume and travel INSIDE the golden archive — so provisioned guests boot # The golden is built with a SMALL OS rootfs and a SINGLE data volume (mp0, backup=1) mounted at a
# from baked images with no registry pull. The split is for RESILIENCE: an isolated OS rootfs stays # NEUTRAL path, /var/lib/felhom. Both consumer paths are binds of subdirectories of it:
# bootable + agent-recoverable if the Docker volume fills (the controller's prevention layer keeps it #
# from filling). Sizes are env-overridable (OS_SIZE_GB / GOLDEN_DOCKER_GB); provision GROWS the data # /var/lib/felhom/docker --bind--> /var/lib/docker (Docker's data-root)
# volume to the per-customer target (bringup.go DataVolGrowGB). backup=1 is MANDATORY on the data mp: # /var/lib/felhom/sys_drive --bind--> /mnt/sys_drive (the controller's system_data_path)
# without it vzdump EXCLUDES the volume (extra LXC mountpoints default backup=0 — storage-split B3), #
# so the archive would carry NO images and provisioned guests would boot imageless. # WHAT THIS REPLACED, AND WHY. Until v2.1.0 these were TWO volumes (mp0 16 G at /var/lib/docker,
# mp1 8 G at /mnt/sys_drive, grown separately at provision). The second one was a fixed ceiling: an
# app whose local recovery unit outgrew it stopped being backed up even with free space next door.
# D-a removed the wall rather than moving it — one volume, one free-space figure, no ceiling.
#
# WHY A NEUTRAL MOUNT AND NOT SIMPLY NESTING ONE PATH INSIDE THE OTHER. Both simpler shapes were
# built and measured (SPIKE-r165-phase0-2026-08-03.md); both boot and reboot cleanly, and each breaks
# a different documented guarantee:
# * volume at /var/lib/docker -> customer backups live INSIDE Docker's data-root, so `du` there
# stops meaning what it says and the ordinary "clear /var/lib/docker to fix Docker" reflex
# destroys every local recovery unit on the box;
# * volume at /mnt/sys_drive -> Docker's ENTIRE data-root lands under /mnt, which the controller
# container mounts wholesale (`-v /mnt:/mnt:rslave`). Measured: the container then sees
# /mnt/sys_drive/docker. The bootstrap's own claim that /mnt "holds only Felhom's
# felhom-data-namespace mounts" would become false.
# The neutral mount breaks neither, for one extra path and one extra fstab line.
#
# The split from the OS rootfs is still for RESILIENCE: an isolated rootfs stays bootable +
# agent-recoverable if the data volume fills (the controller's prevention layer, and since
# controller v0.192.0 the capture floor, keep it from filling). Size is env-overridable
# (OS_SIZE_GB / GOLDEN_VOLUME_GB); provision GROWS the one volume (bringup.go DataVolGrowGB).
# backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra LXC mountpoints default
# backup=0 — storage-split B3), so the archive would carry no images AND no user data.
set -euo pipefail set -euo pipefail
# Script provenance — logged into every bake transcript next to the baked controller tag, so an # Script provenance — logged into every bake transcript next to the baked controller tag, so an
# archive can always be traced to the script that produced it. Bump on any behavior change. # archive can always be traced to the script that produced it. Bump on any behavior change.
GOLDEN_SCRIPT_VERSION="2.1.0" GOLDEN_SCRIPT_VERSION="3.0.0"
VMID="${1:-9100}" VMID="${1:-9100}"
TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}" TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
@@ -59,25 +81,28 @@ if [ -z "$CONTROLLER_IMAGE" ]; then
exit 1 exit 1
fi fi
REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}" REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
# OS rootfs size (GiB) and the golden's Docker-data volume size (GiB). Keep GOLDEN_DOCKER_GB just # OS rootfs size (GiB) and the golden's SINGLE data volume size (GiB).
# large enough for the baked images + headroom; provision grows it to the per-customer target. #
# ONE VOLUME MEANS ONE NUMBER (v3.0.0). The retired GOLDEN_SYSDATA_GB has no successor: there is
# nothing left to size separately. Keep GOLDEN_VOLUME_GB just large enough for the baked images plus
# headroom for the controller's felhom-data skeleton; provision grows the one volume to the
# per-customer target (bringup.go DataVolGrowGB).
OS_SIZE_GB="${OS_SIZE_GB:-32}" OS_SIZE_GB="${OS_SIZE_GB:-32}"
GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}" # 24 = the retired pair's 16 (docker) + 8 (user-data), so a golden archive carries the same content it
# The golden's SSD user-data volume (GiB) mounted at /mnt/sys_drive (mp1, backup=1) — the controller's # did before the merge. It is deliberately NOT a per-customer size: provision grows it.
# system_data_path. Ships small + near-empty (the controller creates <sys_drive>/felhom-data itself once GOLDEN_VOLUME_GB="${GOLDEN_VOLUME_GB:-24}"
# it's a real mountpoint); provision GROWS it to the per-customer target (bringup.go SysDataGrowGB). Like # The neutral mount path of the single volume. Both consumer paths are binds of subdirectories of it.
# mp0, backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra mountpoints default backup=0 — GOLDEN_VOLUME_MP="/var/lib/felhom"
# storage-split B3) and the user-data area would silently fall out of PBS coverage.
GOLDEN_SYSDATA_GB="${GOLDEN_SYSDATA_GB:-8}"
echo "[golden] build-golden.sh v${GOLDEN_SCRIPT_VERSION} — baking controller ${CONTROLLER_IMAGE}" echo "[golden] build-golden.sh v${GOLDEN_SCRIPT_VERSION} — baking controller ${CONTROLLER_IMAGE}"
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker + user-data ${GOLDEN_SYSDATA_GB}G @ /mnt/sys_drive, both backup=1) …" echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, backup=1) …"
# ONE mpN slot. There is deliberately no mp1: that slot held the retired user-data volume, and the
# whole point of R-165 is that it stops existing rather than being made bigger.
pct create "$VMID" "$TEMPLATE" \ pct create "$VMID" "$TEMPLATE" \
--hostname felhom-golden --unprivileged 1 \ --hostname felhom-golden --unprivileged 1 \
--features nesting=1,keyctl=1 \ --features nesting=1,keyctl=1 \
--rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \ --rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \ --mp0 "${ROOTFS_STORAGE}:${GOLDEN_VOLUME_GB},mp=${GOLDEN_VOLUME_MP},backup=1" \
--mp1 "${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1" \
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0 --net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
echo "[golden] starting + installing Docker (official repo, trixie channel) …" echo "[golden] starting + installing Docker (official repo, trixie channel) …"
@@ -106,8 +131,10 @@ echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshott
# The classic overlay2 driver stores EVERYTHING (images + overlay + volumes) under data-root # The classic overlay2 driver stores EVERYTHING (images + overlay + volumes) under data-root
# (/var/lib/docker) = the data volume, which is exactly what "one data-root = one partition for all # (/var/lib/docker) = the data volume, which is exactly what "one data-root = one partition for all
# images + overlay" requires. It also makes the controller's statfs("/") (its overlay root) report the # images + overlay" requires. It also makes the controller's statfs("/") (its overlay root) report the
# DATA volume, which the prevention layer depends on. /var/lib/docker is the mp0 mount (mounted empty # DATA volume, which the prevention layer depends on — MEASURED to still hold under the v3.0.0 merged
# before docker installs), so data-root needs no override. Log caps kill the most common runaway. # layout (a container's `df /` reports the single volume, phase-0 spike). Since v3.0.0 /var/lib/docker
# is a BIND of <volume>/docker rather than the mp0 mount itself, wired immediately below; data-root
# still needs no override because the path is unchanged. Log caps kill the most common runaway.
pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
{ {
"features": { "containerd-snapshotter": false }, "features": { "containerd-snapshotter": false },
@@ -115,6 +142,32 @@ pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json
"log-opts": { "max-size": "10m", "max-file": "3" } "log-opts": { "max-size": "10m", "max-file": "3" }
} }
JSON' JSON'
echo "[golden] wiring the single data volume (R-165 variant V-c): ${GOLDEN_VOLUME_MP}/{docker,sys_drive} -> binds …"
# docker-ce has already populated /var/lib/docker ON THE ROOTFS by now (it auto-starts on install), so
# the content is MOVED onto the volume before the bind is laid over the top. Doing it the other way
# round would hide those files under the bind and silently ship a golden whose baked images are on the
# rootfs — the exact failure class the assertions below exist to catch.
#
# /etc/fstab, not a hand-run mount: systemd's fstab generator orders both binds under local-fs.target,
# which precedes basic.target and therefore docker.service. MEASURED across 3 reboots per variant in
# the phase-0 spike — the ordering worry that motivated the probe did not materialise.
pct exec "$VMID" -- bash -c "
set -e
systemctl stop docker docker.socket containerd 2>/dev/null || true
mkdir -p '${GOLDEN_VOLUME_MP}/docker' '${GOLDEN_VOLUME_MP}/sys_drive'
if [ -d /var/lib/docker ] && [ -n \"\$(ls -A /var/lib/docker 2>/dev/null)\" ]; then
cp -a /var/lib/docker/. '${GOLDEN_VOLUME_MP}/docker'/
rm -rf /var/lib/docker/*
fi
mkdir -p /var/lib/docker /mnt/sys_drive
printf '%s /var/lib/docker none bind 0 0\n' '${GOLDEN_VOLUME_MP}/docker' >> /etc/fstab
printf '%s /mnt/sys_drive none bind 0 0\n' '${GOLDEN_VOLUME_MP}/sys_drive' >> /etc/fstab
systemctl daemon-reload
mount /var/lib/docker
mount /mnt/sys_drive
systemctl start containerd
"
echo "[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …" echo "[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …"
# RESTART (not start): docker-ce auto-starts on install with the DEFAULT config, so it is already # RESTART (not start): docker-ce auto-starts on install with the DEFAULT config, so it is already
# running by now; only a restart picks up the daemon.json just written (overlay2 + log caps). # running by now; only a restart picks up the daemon.json just written (overlay2 + log caps).
@@ -122,12 +175,20 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would # Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
# mean containerd-snapshotter is still on (the split would leave images on the rootfs). # mean containerd-snapshotter is still on (the split would leave images on the rootfs).
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }' pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
# Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss). # ASSERTION 1 (RETARGETED v3.0.0, not removed). /var/lib/docker must be a real mount — now the V-c
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }' # bind of <volume>/docker rather than the mp0 mount itself. Still fails closed on the same failure:
# Same guard for the SSD user-data volume (mp1): /mnt/sys_drive must be its own mount, not the rootfs # if the bind did not take, Docker's data-root silently sits on the OS rootfs and the golden ships
# device — otherwise the controller's system_data_path lands on the OS drive and it warns (the whole # its baked images there.
# point of this volume is to clear that warning). pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a real mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker | head -1)" || { echo "[golden] FATAL: /var/lib/docker is NOT a mount — the V-c docker bind did not take, so the baked images would land on the OS rootfs"; exit 1; }'
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a separate mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }' # ASSERTION 2 (RETARGETED v3.0.0). /mnt/sys_drive must be a real mount — now the V-c bind of
# <volume>/sys_drive. Otherwise the controller's system_data_path lands on the OS drive and it warns
# (clearing that warning is the whole point of the volume).
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a real mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive | head -1)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a mount — the V-c sys_drive bind did not take, so the controller system_data_path would be the OS rootfs"; exit 1; }'
# ASSERTION 2b (NEW v3.0.0 — the invariant the merge is FOR). Both paths must be backed by the SAME
# device, i.e. ONE filesystem with ONE free-space figure. Two devices here is the S2 shape the R-165
# spike ranked strictly WORSE than the split it replaced: every assertion satisfied, the ceiling still
# there, and a shared pool neither `df` can see coming.
pct exec "$VMID" -- bash -c 'n=$(df --output=source /var/lib/docker /mnt/sys_drive | tail -n +2 | sort -u | wc -l); [ "$n" = "1" ] && echo " both paths are ONE filesystem: $(df --output=source,avail /var/lib/docker | tail -1)" || { echo "[golden] FATAL: /var/lib/docker and /mnt/sys_drive are on $n DIFFERENT filesystems — that is the S2 shape (two ceilings), not the R-165 merge"; exit 1; }'
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …" echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
# docker login is used ONCE here on the trusted build host, then logged out before archiving so # docker login is used ONCE here on the trusted build host, then logged out before archiving so
@@ -307,26 +368,36 @@ pct exec "$VMID" -- bash -c '
echo "[golden] stop + archive …" echo "[golden] stop + archive …"
pct stop "$VMID" pct stop "$VMID"
# --mode stop with mp0 + mp1 backup=1 → BOTH the Docker-data volume (baked images) and the # --mode stop with mp0 backup=1 → the SINGLE data volume (baked images AND the user-data area) is
# /mnt/sys_drive user-data volume are INCLUDED. The log below MUST show "including mount point mp0" # INCLUDED. The log MUST show "including mount point mp0" and must NOT show it being excluded — an
# AND "including mount point mp1" — if either shows "excluding … (disabled)" the backup flag was lost # exclusion means the backup flag was lost and the archive carries neither (storage-split B3 trap).
# and the archive carries no images / no user-data volume (storage-split B3 trap). # Since v3.0.0 there is no mp1; the guard that covered it is retargeted below rather than deleted,
# because a guard whose pattern can no longer match is a guard that has silently stopped guarding.
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting." echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
exit 1 exit 1
fi fi
if grep -q "excluding volume mount point mp1" /tmp/golden-vzdump.log; then # ASSERTION 4 (RETARGETED v3.0.0). The mp1 guard used to catch "the user-data volume fell out of the
echo "[golden] FATAL: mp1 (/mnt/sys_drive) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no user-data volume. Aborting." # archive". After the merge there is no mp1 — so the same failure now looks like the volume being
# mounted at the WRONG PATH, which would carry the images but not the user-data area. Assert the
# inclusion line names the volume's actual mount path.
if ! grep -q "including mount point mp0 ('${GOLDEN_VOLUME_MP}')" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: the archive's mp0 is not ${GOLDEN_VOLUME_MP} — the single data volume is mounted somewhere unexpected, so the archive would not carry both the baked images and the user-data area. Aborting."
grep -iE "mount point" /tmp/golden-vzdump.log || true
exit 1
fi
# ASSERTION 5 (RETARGETED v3.0.0). There must be NO mp1 in the archive at all. A leftover second
# volume means the merge did not take and this golden would ship the very ceiling R-165 removed.
if grep -qE "mount point mp1" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: the archive still carries an mp1 — the R-165 merge did not take and this golden would ship a second, ceilinged volume. Aborting."
exit 1 exit 1
fi fi
grep -q "including mount point mp0" /tmp/golden-vzdump.log \ grep -q "including mount point mp0" /tmp/golden-vzdump.log \
|| echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive." || echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
grep -q "including mount point mp1" /tmp/golden-vzdump.log \
|| echo "[golden] WARN: could not confirm mp1 inclusion in the vzdump log — verify manually before using this archive."
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1) VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G + user-data ${GOLDEN_SYSDATA_GB}G, all in the archive)" echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, all in the archive)"
#------------------------------------------------------------------------------- #-------------------------------------------------------------------------------
# Publish to Gitea (BUNDLE slice) — make this golden fetchable by the host-bootstrap script. # Publish to Gitea (BUNDLE slice) — make this golden fetchable by the host-bootstrap script.
+32 -1
View File
@@ -133,6 +133,28 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
Cmnd_Alias FELHOM_STALELOCK = \ Cmnd_Alias FELHOM_STALELOCK = \
/usr/sbin/pct unlock [0-9]* /usr/sbin/pct unlock [0-9]*
# Restore-test scratch teardown (F-LEAK, Campaign 8, v0.110.0). A restore-test whose restore FAILS
# leaves a scratch guest the API token CANNOT destroy: `FelhomAgentGuest` is granted at /pool/felhom and
# a guest joins that pool only when its restore COMPLETES, so a failed restore leaves a pool-less guest
# out of reach (403 VM.Allocate) holding its disks until a human removes it.
#
# TWO API-SIDE FIXES WERE TRIED AND BOTH REFUTED LIVE on 2026-07-28, which is why this grant exists:
# 1. Adopt the stranded guest into the pool, then retry. `PUT /pools/{pool}` ALSO requires
# VM.Allocate on the VM being added — pool membership cannot bootstrap its own authority.
# 2. Grant FelhomAgentGuest per-path at /vms/990000..990009. Durable for exactly one use per slot:
# PVE's own destroy path calls `AccessControl::remove_vm_access($vmid)` (LXC.pm:906), which DELETES
# every ACL at /vms/<vmid> (AccessControl.pm:1898). The grant is consumed by the operation it
# authorises, so after ten teardowns the band is ungranted and the defect returns.
#
# WHY THIS IS THE TIGHTEST AVAILABLE FENCE, not a widening: sudo matches the vmid LITERALLY, so
# `99000[0-9]` is exactly the ten-slot scratch band the restore-test picks from — nothing else. There is
# no `[0-9]*` coarse allowlist here on purpose: unlike `pct unlock`, this op DESTROYS, so the band must
# be in the policy and not merely validated in the agent. Even a compromised agent asking for
# `pct destroy 9201` is refused by sudo itself. Unlike an ACL, a sudoers rule is not consumed by use.
# The agent re-checks the band in code before exec (defence in depth); this is the outer fence.
Cmnd_Alias FELHOM_SCRATCH_TEARDOWN = \
/usr/sbin/pct destroy 99000[0-9] --purge
# Network storage / NAS (Part A1, SPIKE-nas-storage-2026-06-29). The agent mounts a customer NAS share # Network storage / NAS (Part A1, SPIKE-nas-storage-2026-06-29). The agent mounts a customer NAS share
# HOST-SIDE under /mnt/felhom-drives/<name> via a systemd .automount (+ .mount) pair so it propagates # HOST-SIDE under /mnt/felhom-drives/<name> via a systemd .automount (+ .mount) pair so it propagates
# into the guest through the existing shared bind (an unprivileged LXC cannot mount NFS/CIFS itself). # into the guest through the existing shared bind (an unprivileged LXC cannot mount NFS/CIFS itself).
@@ -223,6 +245,15 @@ Cmnd_Alias FELHOM_SSHD = \
# blind to an `applied`-but-401 tier. It is NOT a general file-read: the wrapper pins the directory # blind to an `applied`-but-401 tier. It is NOT a general file-read: the wrapper pins the directory
# and prefix-asserts the resolved path, and the id grammar admits no slash. The secret goes to # and prefix-asserts the resolved path, and the id grammar admits no slash. The secret goes to
# STDOUT, never argv — sudo logs argv. # STDOUT, never argv — sudo logs argv.
# E-2a: the backup-target storage shim. Creating a PVE storage needs Datastore.Allocate at /storage
# and the grant needs Permissions.Modify -- the agent holds NEITHER by design (blast-radius
# containment; Permissions.Modify would let it rewrite its own authority). Both live behind this
# fixed-vocabulary root shim instead, exactly like the mkfs and pbs-apply wrappers. The wrapper has
# NO storage-removal path, enforces is_mountpoint 1, and refuses a target on the root device.
Cmnd_Alias FELHOM_BACKUPTARGET = \
/usr/local/sbin/felhom-backup-target-apply create *, \
/usr/local/sbin/felhom-backup-target-apply grant *
Cmnd_Alias FELHOM_PBSDR = \ Cmnd_Alias FELHOM_PBSDR = \
/usr/local/sbin/felhom-pbs-apply create *, \ /usr/local/sbin/felhom-pbs-apply create *, \
/usr/local/sbin/felhom-pbs-apply reconcile *, \ /usr/local/sbin/felhom-pbs-apply reconcile *, \
@@ -274,4 +305,4 @@ Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \ /usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
/usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 /usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN
+111
View File
@@ -0,0 +1,111 @@
#!/bin/bash
#===============================================================================
# felhom-backup-target-apply — the ONLY path the felhom-agent sudoers permits for creating the
# whole-guest backup TARGET storage and granting the agent access to it (E-2a).
#
# WHY A WRAPPER AT ALL. Creating a PVE storage needs `Datastore.Allocate` at `/storage`, and the ACL
# grant needs `Permissions.Modify`. The agent holds NEITHER by design — its token is scoped per
# storage path for blast-radius containment, and `Permissions.Modify` would let it rewrite its own
# authority. Widening the PVE role to make the move possible would trade the entire containment model
# for one feature. So the privileged half lives here: a minimal, auditable root shim with a fixed
# vocabulary, exactly like felhom-mkfs-guarded and felhom-pbs-apply.
#
# THE NO-DELETE LAW (inherited from felhom-pbs-apply, same reasoning class). This wrapper contains NO
# storage-removal path of any kind. `pvesm remove` on a dir storage does not delete the archives, but
# it DOES silently orphan a configured backup tier, and a "cleanup" verb here would be reachable by
# any bug in the agent. Retiring a target is a deliberate operator op, not this tool. Grep-assertable;
# do not add one.
#
# THE TWO LAWS E-1 PAID FOR ON LIVE HARDWARE, both enforced here rather than trusted to the caller:
#
# F-1 the storage path must BE the drive's own mountpoint. A subdirectory fails the agent's
# exactMount check, so the target reports `disconnected` FOREVER and its durable id degrades
# off the filesystem UUID. Enforced: `mountpoint -q` must pass on the exact path given.
#
# F-2 --is_mountpoint 1 is not optional. Without it, an unplugged or late-mounting drive leaves a
# bare directory on the ROOT filesystem and vzdump writes the whole-guest backup onto the
# system drive — the exact device the whole change exists to escape — while PVE reports the
# storage `active` and advertises the root filesystem's free space. Proven live: the unguarded
# form had already created dump/ on pve-root. Hardcoded below; not a caller-supplied flag.
#
# Ops (all non-secret; nothing here touches a credential, so nothing arrives on stdin):
# create <id> <mountpoint>
# Create a `dir` storage with content=backup at <mountpoint>, is_mountpoint 1.
# IDEMPOTENT: an existing entry with the SAME path is accepted (re-run safe, and the
# installer re-run path depends on it). An existing entry with a DIFFERENT path is REFUSED
# — silently repointing a live backup target is the failure this whole arc closes.
# grant <id>
# The dual grant: FelhomAgentStore on /storage/<id> to the agent user AND token (privsep
# intersection — a token's rights are the intersection, so granting one is granting neither).
# Without it every backup 403s on first run (E-1 finding F-3, found by the first real backup).
#===============================================================================
set -euo pipefail
die() { echo "felhom-backup-target-apply: REFUSED: $*" >&2; exit 1; }
op="${1:-}"; id="${2:-}"
[[ -n "$op" && -n "$id" ]] || die "usage: felhom-backup-target-apply <create|grant> <storage-id> [mountpoint]"
# Storage id: PVE grammar, conservative. Also the ACL path component — no slashes possible.
[[ "$id" =~ ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ ]] || die "bad storage id ($id)"
STORECFG=/etc/pve/storage.cfg
# current_path_of <id> — the configured `path` of dir storage <id>, or "" when absent/not-a-dir.
current_path_of() {
awk -v want="dir: $1" '
$0 == want { found=1; next }
found && /^[a-z]+: / { exit }
found && $1 == "path" { print $2; exit }
' "$STORECFG" 2>/dev/null || true
}
case "$op" in
create)
[[ $# -eq 3 ]] || die "create takes <id> <mountpoint>"
mp="$3"
# Absolute, normalized, no traversal, no shell metacharacters. The value reaches pvesm and the
# filesystem, so it is validated here rather than assumed well-formed.
[[ "$mp" = /* ]] || die "mountpoint must be absolute ($mp)"
[[ "$mp" != *".."* ]] || die "mountpoint must not contain .. ($mp)"
[[ "$mp" =~ ^[A-Za-z0-9/_.-]+$ ]] || die "mountpoint has unexpected characters ($mp)"
[[ "$mp" != "/" ]] || die "refusing / as a backup target"
# F-1 + F-2, checked as one: the path must BE a mountpoint right now. A bare directory here is
# precisely the silent-retarget shape, and is_mountpoint would make PVE refuse it later anyway —
# better to refuse now, with a reason, than to create a storage that can never activate.
mountpoint -q "$mp" || die "$mp is not a mountpoint — the backup target must be the drive's OWN mountpoint (F-1), and an unmounted path would silently retarget onto the system drive (F-2)"
# Never the system disk: a target on the root filesystem is not drive-loss protection, it is the
# thing we are escaping. The root device and the candidate's device are compared, not their paths.
root_dev="$(findmnt -no SOURCE / 2>/dev/null || true)"
mp_dev="$(findmnt -no SOURCE "$mp" 2>/dev/null || true)"
[[ -n "$mp_dev" ]] || die "could not resolve the backing device of $mp"
[[ "$mp_dev" != "$root_dev" ]] || die "$mp is backed by the ROOT device ($root_dev) — a backup target there protects against corruption only, never drive loss"
existing="$(current_path_of "$id")"
if [[ -n "$existing" ]]; then
if [[ "$existing" == "$mp" ]]; then
echo "felhom-backup-target-apply: storage $id already exists at $mp — nothing to do (idempotent)" >&2
exit 0
fi
die "storage $id already exists at $existing — refusing to repoint it at $mp (a live backup target is never silently moved)"
fi
# is_mountpoint 1 is HARDCODED (F-2). content=backup only: this storage exists for vzdump archives
# and must never become a place guests are allocated on.
pvesm add dir "$id" --path "$mp" --content backup --is_mountpoint 1 >&2
echo "felhom-backup-target-apply: created dir storage $id at $mp (content=backup, is_mountpoint 1)" >&2
;;
grant)
[[ $# -eq 2 ]] || die "grant takes only <id>"
# BOTH, always. A privsep token's rights are the intersection of the user's and the token's ACLs,
# so granting one of the two grants nothing usable.
pveum acl modify "/storage/$id" --users felhom-agent@pve --roles FelhomAgentStore >&2
pveum acl modify "/storage/$id" --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore >&2
echo "felhom-backup-target-apply: granted FelhomAgentStore on /storage/$id (user + token)" >&2
;;
*)
die "unknown op ($op)"
;;
esac
+26 -17
View File
@@ -1,20 +1,30 @@
# felhom-agent local API — host firewall narrowing (doc 03 §6, slice 8A) # felhom-agent local API — host firewall narrowing (doc 03 §6; R-50 island update 2026-07-25)
# #
# Defense-in-depth for the per-guest local API (the controller→agent channel on the host # Defense-in-depth for the per-guest local API (the controller→agent channel). The PER-GUEST BEARER
# bridge). The PER-GUEST BEARER TOKEN is the authorization gate; this firewall rule is an # TOKEN + the served-leaf pin are the authorization gate; a firewall rule is only an ADDITIONAL layer
# ADDITIONAL layer that limits who can even reach the port. The slice-8A spike found no rule # limiting who can even open the port.
# was needed for reachability on the demo (PVE firewall off) — this narrows exposure so that
# only guests on the bridge subnet (not arbitrary LAN hosts) can open a connection.
# #
# The agent already binds the listener to the host BRIDGE IP (local_api.listen_addr), not # === R-50 ISLAND INSTALL (the default on a fresh appliance) =================================
# 0.0.0.0. This file adds the subnet restriction. Apply it at HOST SETUP (it is a host-level # The agent binds local_api.listen_addr on the HOST-INTERNAL island bridge — 169.254.253.1:8443 on
# packet-filter change, intentionally OUTSIDE the agent's 3-exception privileged fence — the # vmbr9, a bridge with NO physical port (bridge-ports none). That bind is the security win:
# agent never mutates the host firewall at runtime). # * Nothing listens on the LAN IP at all, so no LAN host (or off-site attacker on the LAN) can
# reach the local API — the LAN:8443 surface is CLOSED by the bind, not by a rule.
# * vmbr9 has no uplink, so 169.254.253.1:8443 is reachable ONLY from the one guest wired to the
# /30 (169.254.253.2) — the controller. The portless bridge is the isolation.
# So on an island install NO firewall rule is required for exposure; the topology provides it. If you
# want belt-and-suspenders, restrict the port to the island bridge (it changes nothing, since nothing
# off-bridge can route to a portless bridge anyway):
# #
# Replace the bridge IP (192.168.0.162), port (8443), and the guest bridge subnet # nft add rule inet filter input iifname != "vmbr9" ip daddr 169.254.253.1 tcp dport 8443 drop
# (192.168.0.0/24) with this host's values. #
# Verify: from the guest, a TLS connect to 169.254.253.1:8443 succeeds; there is no LAN listener to
# probe (`ss -lnt 'sport = :8443'` shows only the island IP).
#
# === LEGACY LAN BIND (byo, --no-island, or an explicit --bridge-ip) =========================
# When the agent still binds a LAN bridge IP (e.g. 192.168.0.162:8443), the port is exposed to the
# whole LAN and the subnet-narrowing rule below is worth applying. Replace the bridge IP, port, and
# the guest bridge subnet with this host's values.
# #
# ---------------------------------------------------------------------------------------------
# Option A — nftables (recommended on PVE 8/9; inet filter table). Insert ABOVE any accept: # Option A — nftables (recommended on PVE 8/9; inet filter table). Insert ABOVE any accept:
# #
# nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \ # nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
@@ -22,13 +32,11 @@
# nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \ # nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
# ip saddr 192.168.0.0/24 accept # ip saddr 192.168.0.0/24 accept
# #
# ---------------------------------------------------------------------------------------------
# Option B — iptables: # Option B — iptables:
# #
# iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -s 192.168.0.0/24 -j ACCEPT # iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -s 192.168.0.0/24 -j ACCEPT
# iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -j DROP # iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -j DROP
# #
# ---------------------------------------------------------------------------------------------
# Option C — PVE host firewall (/etc/pve/nodes/<node>/host.fw), if the PVE firewall is enabled. # Option C — PVE host firewall (/etc/pve/nodes/<node>/host.fw), if the PVE firewall is enabled.
# Add under [RULES] (and ensure the firewall is enabled in cluster.fw / host.fw): # Add under [RULES] (and ensure the firewall is enabled in cluster.fw / host.fw):
# #
@@ -36,5 +44,6 @@
# IN ACCEPT -source 192.168.0.0/24 -dport 8443 -proto tcp -log nolog # IN ACCEPT -source 192.168.0.0/24 -dport 8443 -proto tcp -log nolog
# IN DROP -dport 8443 -proto tcp -log nolog # IN DROP -dport 8443 -proto tcp -log nolog
# #
# Verify after applying: from a guest ON the bridge, a TLS connect to <bridge-ip>:8443 succeeds; # Apply at HOST SETUP — a host-level packet-filter change, intentionally OUTSIDE the agent's
# from an OFF-bridge host it is refused/dropped. (The token + leaf-pin still gate the request.) # 3-exception privileged fence (the agent never mutates the host firewall at runtime). The token +
# leaf-pin still gate the request regardless of which bind is in force.
@@ -0,0 +1,243 @@
package backup
import (
"bytes"
"context"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// F-CRIT-2 (Campaign 8): a failed backup must not look like a fresh one.
//
// Every fixture below is a VERBATIM shape captured from the live PVE API on 2026-07-28
// (`pvesh get /nodes/<node>/storage/<store>/content`), not a hand-invented struct. That matters:
// the `unparseable` path in this package went untested for months behind a JSON shape that did not
// match production, and the whole point of this fix is that presence != validity.
// phantomEntry is the artefact a PBS daemon killed mid-upload leaves behind: listed as a restorable
// backup, 1 byte, NEWEST, and carrying no `verification`/`encrypted`/`notes` at all because it has
// no manifest (`index.json.blob` is absent on disk).
func phantomEntry() proxmox.StorageContent {
return proxmox.StorageContent{
VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z",
Content: "backup",
Format: "pbs-ct",
Size: 1,
CTime: 1785216674,
VMID: 9201,
}
}
// goodPBSEntry is a real, complete offsite snapshot (demo-hp, 2026-07-28T03:40:42Z).
func goodPBSEntry() proxmox.StorageContent {
return proxmox.StorageContent{
VolID: "felhom-pbs:backup/ct/9201/2026-07-28T03:40:42Z",
Content: "backup",
Format: "pbs-ct",
Size: 4353457559,
CTime: 1785210042,
VMID: 9201,
}
}
// goodLocalEntry is a real, complete LOCAL vzdump (demo-hp). Note it legitimately has no
// `verification` and no `encrypted` on the wire — a dir storage has no such concept — which is
// exactly why those fields must never be used as completeness discriminators.
func goodLocalEntry() proxmox.StorageContent {
return proxmox.StorageContent{
VolID: "local:backup/vzdump-lxc-9201-2026_07_28-07_29_54.tar.zst",
Content: "backup",
Format: "tar.zst",
Size: 1590431865,
CTime: 1785216594,
VMID: 9201,
}
}
func runnerWithContent(t *testing.T, buf *bytes.Buffer, content []proxmox.StorageContent) *BackupRunner {
t.Helper()
lg := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
return NewBackupRunner(&fakeBackupAPI{content: content}, "felhom-pbs", proxmox.ModeSnapshot, "", "", lg)
}
// Group A — the phantom must NOT set tier freshness, even though it is the newest entry.
//
// RED-PROOF: restore the old predicate in NewestArchiveTime
// (`if e.Content == "backup" && e.VMID == vmid && e.CTime > best`) → the phantom's ctime
// (1785216674) wins over the good snapshot's (1785210042) and this test fails with
// "got 1785216674, want 1785210042" — i.e. the exact F-CRIT-2 defect.
func TestNewestArchiveTime_PhantomIsNotCounted(t *testing.T) {
var buf bytes.Buffer
// phantom deliberately listed FIRST and is also the newest by ctime.
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
got, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatalf("NewestArchiveTime: %v", err)
}
if !found {
t.Fatal("found=false — the GOOD snapshot must still be counted; rejecting everything is the thrash path")
}
if got.Unix() != goodPBSEntry().CTime {
t.Errorf("freshness came from the wrong entry: got ctime %d, want %d (the good snapshot)", got.Unix(), goodPBSEntry().CTime)
}
if got.Unix() == phantomEntry().CTime {
t.Error("the 1-byte manifest-less phantom set tier freshness — this is F-CRIT-2")
}
}
// Group A — with ONLY a phantom present the tier must report "no backup", not a fresh one.
// That is what lets the controller see age_state=absent and fire its first-backup valve.
func TestNewestArchiveTime_OnlyPhantomReportsNotFound(t *testing.T) {
var buf bytes.Buffer
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry()})
_, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatalf("NewestArchiveTime: %v", err)
}
if found {
t.Error("found=true with only a phantom present — the tier would report fresh and go silent for a full cadence")
}
}
// Group B — THE SCENARIO-D GUARD. A valid snapshot on EITHER tier must still be counted.
//
// This is what makes Group A safe. A filter that is too aggressive does not merely lose safety
// margin: the tier reports absent on every poll, backs up every cycle, and the R-88 breaker cannot
// save it because those backups SUCCEED. That is a continuous multi-GB write loop across the fleet.
//
// RED-PROOF: make archivePlausiblyComplete return `false, "reject everything"` unconditionally →
// both subtests fail with found=false.
func TestNewestArchiveTime_ValidSnapshotsAreStillCounted(t *testing.T) {
for _, tc := range []struct {
name string
entry proxmox.StorageContent
}{
{"pbs offsite (has verification+encrypted on the wire)", goodPBSEntry()},
{"local dir vzdump (has NEITHER verification NOR encrypted — and must still count)", goodLocalEntry()},
} {
t.Run(tc.name, func(t *testing.T) {
var buf bytes.Buffer
r := runnerWithContent(t, &buf, []proxmox.StorageContent{tc.entry})
got, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatalf("NewestArchiveTime: %v", err)
}
if !found {
t.Fatalf("a REAL %s backup was rejected — this is the backup-thrash path, not extra safety", tc.name)
}
if got.Unix() != tc.entry.CTime {
t.Errorf("got ctime %d, want %d", got.Unix(), tc.entry.CTime)
}
if strings.Contains(buf.String(), "INCOMPLETE archive") {
t.Errorf("a valid archive was announced as incomplete:\n%s", buf.String())
}
})
}
}
// Group B — the smallest REAL backup measured anywhere on the fleet (612,397,450 B, a guest-9100
// vzdump) must clear the floor with room to spare. If someone ever raises
// minPlausibleArchiveBytes past this, that is the fleet-thrash bug and this test is the tripwire.
func TestMinPlausibleArchiveBytes_LeavesHeadroomBelowTheSmallestRealBackup(t *testing.T) {
const smallestObservedRealBackup int64 = 612397450 // fleet survey 2026-07-28
if minPlausibleArchiveBytes >= smallestObservedRealBackup {
t.Fatalf("floor %d B is not below the smallest real backup ever observed (%d B) — this WILL reject real archives",
minPlausibleArchiveBytes, smallestObservedRealBackup)
}
if ratio := smallestObservedRealBackup / minPlausibleArchiveBytes; ratio < 100 {
t.Errorf("floor %d B leaves only %dx headroom below the smallest real backup (%d B) — too tight",
minPlausibleArchiveBytes, ratio, smallestObservedRealBackup)
}
}
// Group C — UNDECIDABLE ⇒ NOT COUNTED (the fail-safe direction).
//
// A zero/absent size is not evidence of a good backup; it is absence of evidence. Erring toward
// "not fresh" costs one extra backup. Erring the other way is F-CRIT-2.
//
// RED-PROOF: flip the comparison in archivePlausiblyComplete to `e.Size > minPlausibleArchiveBytes
// || e.Size == 0` (i.e. treat unknown as complete) → the size-0 case reports ok=true and this fails.
func TestArchivePlausiblyComplete_UndecidableIsNotCounted(t *testing.T) {
for _, tc := range []struct {
name string
size int64
}{
{"the observed phantom", 1},
{"absent size field (unmarshals to 0)", 0},
{"just under the floor", minPlausibleArchiveBytes - 1},
} {
t.Run(tc.name, func(t *testing.T) {
e := phantomEntry()
e.Size = tc.size
ok, why := archivePlausiblyComplete(e)
if ok {
t.Errorf("size %d counted as a complete backup — undecidable must fail safe", tc.size)
}
if why == "" {
t.Error("rejection carried no reason — a silent rejection is a new quiet path")
}
})
}
if ok, why := archivePlausiblyComplete(goodPBSEntry()); !ok {
t.Errorf("a real snapshot was rejected: %s", why)
}
}
// Group D — the rejection is announced ONCE per snapshot, not once per due-check.
//
// The due-check runs every 5 minutes and a phantom persists indefinitely (server-side prune does
// not collect it), so per-poll logging would emit ~288 identical lines a day and bury the signal.
//
// RED-PROOF: delete the `if seen { return }` guard in warnRejectedArchiveOnce → this test reports
// "logged 5 times, want 1".
func TestNewestArchiveTime_RejectionLoggedOncePerSnapshot(t *testing.T) {
var buf bytes.Buffer
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
const polls = 5
for i := 0; i < polls; i++ {
if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
t.Fatalf("poll %d: %v", i, err)
}
}
n := strings.Count(buf.String(), "INCOMPLETE archive")
if n != 1 {
t.Errorf("rejection logged %d times across %d polls, want exactly 1:\n%s", n, polls, buf.String())
}
out := buf.String()
if !strings.Contains(out, phantomEntry().VolID) {
t.Errorf("the log line does not NAME the rejected snapshot:\n%s", out)
}
if !strings.Contains(out, "below the") {
t.Errorf("the log line does not say WHY it was rejected:\n%s", out)
}
if !strings.Contains(out, "level=WARN") {
t.Errorf("rejection was not logged at WARN:\n%s", out)
}
}
// Group D — a SECOND, distinct phantom is announced separately. The dedupe must be per snapshot,
// not a one-shot latch that hides every later phantom.
func TestNewestArchiveTime_DistinctPhantomsEachAnnounced(t *testing.T) {
var buf bytes.Buffer
second := phantomEntry()
second.VolID = "felhom-pbs:backup/ct/9201/2026-07-29T05:31:14Z"
second.CTime = phantomEntry().CTime + 86400
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), second, goodPBSEntry()})
for i := 0; i < 3; i++ {
if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
t.Fatalf("poll %d: %v", i, err)
}
}
if n := strings.Count(buf.String(), "INCOMPLETE archive"); n != 2 {
t.Errorf("got %d rejection lines for 2 distinct phantoms across 3 polls, want 2:\n%s", n, buf.String())
}
}
+6 -3
View File
@@ -168,9 +168,12 @@ func TestScheduler_TickRunsAndRecords(t *testing.T) {
store := NewStore() store := NewStore()
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Pass: true, Verified: "boot+running", Duration: time.Second}} rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Pass: true, Verified: "boot+running", Duration: time.Second}}
s := NewScheduler(SchedulerOptions{ s := NewScheduler(SchedulerOptions{
Runner: rt, Runner: rt,
Pick: func(context.Context) (string, error) { return "vol", nil }, Pick: func(context.Context) (string, error) { return "vol", nil },
Store: store, Store: store,
Spec: func(context.Context, string) reconcile.RestoreTestSpec {
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
},
Cadence: time.Hour, Cadence: time.Hour,
Logger: quiet(), Logger: quiet(),
}) })
+69
View File
@@ -0,0 +1,69 @@
package backup
import "sync"
// InFlight is the host-wide "one heavy guest operation at a time" gate.
//
// R-85 (Scenario F). The operator's R-82 ruling was "one backup at a time per guest"; a restore-test
// must JOIN that single-flight rather than sit outside it. It is not a lock-contention concern —
// a restore-test uses a scratch VMID, so it never touches the live guest's vzdump lock. It is a
// LINK concern: an offsite restore PULLS a multi-GB archive while an offsite backup PUSHES one, over
// the same WireGuard tunnel. On the demo fleet that link runs at ~33 MB/min upstream; running both
// at once makes each slower and pushes both toward their timeouts, which is how a healthy tier ends
// up recorded as failed.
//
// It is deliberately host-wide and coarse rather than per-guest: these boxes carry one customer
// guest, and the resource being protected (the uplink) is shared by everything on the host anyway.
//
// The gate is ADVISORY in one direction only — it never cancels anything already running. A caller
// that cannot acquire DEFERS to its next cadence. Deferring a restore-test costs a few hours of
// coverage; cancelling a running backup costs the backup.
//
// CORRECTED 2026-07-28 (F-A1). That "DEFERS" was true of the restore-test caller and NOT of the
// backup caller, and the comment did not say so. The controller's start path had no 409 branch, so
// a refusal here was recorded as a tier FAILURE: the R-88 breaker armed and the operator was
// emailed "Whole-guest backup FAILED" about a backup that was merely waiting its turn. Campaign 8
// observed it on both demo boxes in the same minute.
//
// Fixed on the CONTROLLER side (v0.179.0), which is where the misreading lived — this gate's
// behaviour was correct throughout and is unchanged. The controller now maps HTTP 409 to a
// contention path: it defers the tier, keeps it DUE, and alarms only if contention outlives the
// agent's own restore-test ceiling. Nothing here needs to change; the claim above is simply now
// true of both callers.
type InFlight struct {
mu sync.Mutex
what string // "" = idle
}
// TryAcquire claims the gate for `what`. ok=false means something else holds it, and `busy` names
// it — the name matters, because "deferred" with no reason is indistinguishable from "broken".
func (g *InFlight) TryAcquire(what string) (release func(), busy string, ok bool) {
if g == nil {
// Not wired (older call sites, tests) → no gating, previous behaviour.
return func() {}, "", true
}
g.mu.Lock()
defer g.mu.Unlock()
if g.what != "" {
return nil, g.what, false
}
g.what = what
var once sync.Once
return func() {
once.Do(func() {
g.mu.Lock()
g.what = ""
g.mu.Unlock()
})
}, "", true
}
// Busy reports what currently holds the gate ("" = idle).
func (g *InFlight) Busy() string {
if g == nil {
return ""
}
g.mu.Lock()
defer g.mu.Unlock()
return g.what
}
+139
View File
@@ -0,0 +1,139 @@
package backup
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"sync"
"time"
)
// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
//
// R-85 (1.4). This one genuinely needs PERSISTENCE, unlike R-84 — and the difference is worth
// stating, because the two look like the same problem and are not:
//
// - R-84 (backup freshness) had a GROUND TRUTH to consult: the archive is still on the storage,
// so the agent could ask "when did a backup last land?" and never persist anything. That is
// strictly better, because a pruned archive correctly stops counting.
// - A restore-test leaves NO artifact — the scratch guest is destroyed as its final act. There is
// nothing to query. "Did we prove this tier restores?" exists only as remembered state, so it
// must be written down or it is lost.
//
// Why it must survive a restart: rotation is oldest-first (the operator ruling), so an in-memory map
// would reset every tier to "never tested" on each restart. Ordering would then depend on map
// iteration order, and one tier could be starved indefinitely while the other is re-tested — with
// agent deploys as routine as they are, that is not a corner case.
//
// Only SUCCESS is recorded. A failed run must not satisfy rotation, or a tier that fails every time
// would look freshly proven and stop being retried — the same "a failure satisfies the cadence"
// trap the backup due-check avoids.
type RestoreTestState struct {
path string
mu sync.Mutex
last map[string]time.Time // target id → last SUCCESSFUL restore-test (UTC)
}
// NewRestoreTestState opens (or creates) the state at path. A missing or unreadable file is NOT an
// error: it degrades to "nothing proven yet", which is the correct starting point and keeps a
// corrupt file from wedging the daemon.
func NewRestoreTestState(path string) *RestoreTestState {
s := &RestoreTestState{path: path, last: map[string]time.Time{}}
data, err := os.ReadFile(path)
if err != nil {
return s
}
var raw map[string]string
if json.Unmarshal(data, &raw) != nil {
return s
}
for target, ts := range raw {
if t, perr := time.Parse(time.RFC3339, ts); perr == nil {
s.last[target] = t.UTC()
}
}
return s
}
// RecordSuccess stamps a tier as proven at t. Only call this for a PASSING restore-test.
func (s *RestoreTestState) RecordSuccess(target string, t time.Time) error {
if target == "" {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
s.last[target] = t.UTC()
return s.saveLocked()
}
// LastSuccess returns when this tier was last proven (ok=false = never).
func (s *RestoreTestState) LastSuccess(target string) (time.Time, bool) {
s.mu.Lock()
defer s.mu.Unlock()
t, ok := s.last[target]
return t, ok
}
// Snapshot returns a copy of the whole map — for the host-report gauge.
func (s *RestoreTestState) Snapshot() map[string]time.Time {
s.mu.Lock()
defer s.mu.Unlock()
out := make(map[string]time.Time, len(s.last))
for k, v := range s.last {
out[k] = v
}
return out
}
// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
//
// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
// naturally prioritises a tier that has never been restore-tested at all — which on this fleet was
// the offsite tier, unproven for its entire existence.
//
// Ties break on target id so the order is deterministic; without that, two tiers proven in the same
// second would rotate by map iteration order, which is randomised in Go and would make the
// behaviour untestable and occasionally starving.
func (s *RestoreTestState) OldestFirst(targets []string) []string {
s.mu.Lock()
defer s.mu.Unlock()
out := append([]string(nil), targets...)
sort.SliceStable(out, func(i, j int) bool {
ti, oki := s.last[out[i]]
tj, okj := s.last[out[j]]
switch {
case !oki && !okj:
return out[i] < out[j] // both never proven → deterministic
case !oki:
return true // never proven wins
case !okj:
return false
case !ti.Equal(tj):
return ti.Before(tj)
default:
return out[i] < out[j]
}
})
return out
}
func (s *RestoreTestState) saveLocked() error {
raw := make(map[string]string, len(s.last))
for target, t := range s.last {
raw[target] = t.UTC().Format(time.RFC3339)
}
data, err := json.MarshalIndent(raw, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
return err
}
tmp := s.path + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, s.path)
}
+335
View File
@@ -0,0 +1,335 @@
package backup
import (
"context"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// R-85 Phase 2 — tier rotation, persisted state, and the one-heavy-operation gate.
//
// The failure this prevents is not hypothetical: demo-hp's DR tier reported `applied` with ZERO
// snapshots for five days and nobody noticed, because the scheduler could only ever see the primary
// tier. Rotation is what makes the offsite tier testable at all.
// rotRunner records which archives it was asked to restore.
type rotRunner struct {
mu sync.Mutex
archives []string
pass bool
}
func (r *rotRunner) RunRestoreTest(_ context.Context, spec reconcile.RestoreTestSpec) reconcile.RestoreTestResult {
r.mu.Lock()
defer r.mu.Unlock()
r.archives = append(r.archives, spec.Archive)
return reconcile.RestoreTestResult{
Archive: spec.Archive, SourceTier: spec.SourceTier,
Pass: r.pass, Verified: "boot+running",
}
}
func (r *rotRunner) seen() []string {
r.mu.Lock()
defer r.mu.Unlock()
return append([]string(nil), r.archives...)
}
// archiveFor is a TierPicker over a fixed map: target → archive ("" = that tier holds none).
func archiveFor(m map[string]string) TierPicker {
return func(_ context.Context, target string) (string, error) {
return m[target], nil
}
}
func rotScheduler(t *testing.T, rr *rotRunner, st *RestoreTestState, pick TierPicker, gate *InFlight) *Scheduler {
t.Helper()
return NewScheduler(SchedulerOptions{
Runner: rr,
Store: NewStore(),
Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
},
Cadence: time.Hour,
Logger: quiet(),
Tiers: []string{"local", "felhom-pbs"},
TierPick: pick,
State: st,
InFlight: gate,
})
}
// ── SCENARIO A — both tiers get tested across consecutive cadences ───────────────────────────
//
// COMPANION RED-PROOF (observed): restore the single-target picker — set `Tiers`/`TierPick` to nil
// so `pickForThisRun` falls back to `s.pick` on the primary runner — and this fails with
// "both tiers must be exercised across 4 cadences; got [local:… local:… local:… local:…]",
// i.e. the offsite tier never appears. That is today's behaviour, and it is why demo-hp's DR tier
// went unproven for its entire existence.
func TestRotation_BothTiersExercisedAcrossCadences(t *testing.T) {
rr := &rotRunner{pass: true}
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
"local": "local:backup/vzdump-lxc-9201-x.tar.zst",
"felhom-pbs": "felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
}), &InFlight{})
s.now = func() time.Time { return time.Now().UTC() }
for i := 0; i < 4; i++ {
s.tick(context.Background())
}
got := rr.seen()
var sawLocal, sawPBS bool
for _, a := range got {
if len(a) >= 5 && a[:5] == "local" {
sawLocal = true
}
if len(a) >= 10 && a[:10] == "felhom-pbs" {
sawPBS = true
}
}
if !sawLocal || !sawPBS {
t.Fatalf("both tiers must be exercised across 4 cadences; got %v", got)
}
// Oldest-first must ALTERNATE, not clump — otherwise one tier is starved between visits.
if len(got) != 4 {
t.Fatalf("want 4 runs, got %d: %v", len(got), got)
}
if got[0] == got[1] {
t.Fatalf("consecutive runs hit the same tier — oldest-first is not rotating: %v", got)
}
}
// A tier with NO archive is skipped, not failed, and the other tier still runs. A brand-new offsite
// tier legitimately has nothing to restore; turning that into a failure would make every fresh box
// look broken for its first week.
func TestRotation_EmptyTierSkippedNotFailed(t *testing.T) {
rr := &rotRunner{pass: true}
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
"local": "local:backup/vzdump-lxc-9201-x.tar.zst",
"felhom-pbs": "", // provisioned but empty
}), &InFlight{})
s.tick(context.Background())
got := rr.seen()
if len(got) != 1 || got[0][:5] != "local" {
t.Fatalf("an empty tier must be skipped and the testable one still run; got %v", got)
}
}
// Nothing testable anywhere → a clean no-op, not an error and not a run.
func TestRotation_NoArchivesAnywhereIsANoOp(t *testing.T) {
rr := &rotRunner{pass: true}
s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
archiveFor(map[string]string{}), &InFlight{})
s.tick(context.Background())
if got := rr.seen(); len(got) != 0 {
t.Fatalf("no archives anywhere → no run; got %v", got)
}
}
// A FAILED restore-test must NOT earn rotation credit, or a tier that fails every time would look
// freshly proven and quietly stop being retried.
func TestRotation_FailureEarnsNoCredit(t *testing.T) {
rr := &rotRunner{pass: false}
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
"local": "local:backup/x.tar.zst",
"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
}), &InFlight{})
s.tick(context.Background())
if _, ok := st.LastSuccess("local"); ok {
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
}
if _, ok := st.LastSuccess("felhom-pbs"); ok {
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
}
}
// ── SCENARIO E — rotation survives a restart ─────────────────────────────────────────────────
//
// COMPANION RED-PROOF (observed): make the state in-memory (construct a fresh
// `NewRestoreTestState` on a DIFFERENT path for the second scheduler, i.e. lose the file) and this
// fails with "after a restart the OTHER tier must be next; got felhom-pbs" — the same tier repeats
// and the other is starved indefinitely, which with agent deploys as routine as they are is not a
// corner case.
func TestRotation_SurvivesRestart(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "rt.json")
picks := archiveFor(map[string]string{
"local": "local:backup/x.tar.zst",
"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
})
// First process: the OFFSITE tier is tested (never-proven sorts first).
rr1 := &rotRunner{pass: true}
st1 := NewRestoreTestState(path)
s1 := rotScheduler(t, rr1, st1, picks, &InFlight{})
s1.tick(context.Background())
first := rr1.seen()
if len(first) != 1 {
t.Fatalf("want one run, got %v", first)
}
// --- restart: brand-new state object reading the SAME file ---
rr2 := &rotRunner{pass: true}
st2 := NewRestoreTestState(path)
s2 := rotScheduler(t, rr2, st2, picks, &InFlight{})
s2.tick(context.Background())
second := rr2.seen()
if len(second) != 1 {
t.Fatalf("want one run after restart, got %v", second)
}
if second[0] == first[0] {
t.Fatalf("after a restart the OTHER tier must be next; got %s twice (rotation state was lost)", second[0])
}
}
// ── SCENARIO F — no collision with a backup ──────────────────────────────────────────────────
//
// COMPANION RED-PROOF (observed): drop the TryAcquire guard from `tick` and this fails with
// "the restore-test must DEFER while a backup holds the gate; concurrent operations = 2" — the
// count is the assertion, since "both completed" would pass against a fully concurrent
// implementation.
func TestRotation_DefersWhileABackupHoldsTheGate(t *testing.T) {
gate := &InFlight{}
release, _, ok := gate.TryAcquire("backup:felhom-pbs")
if !ok {
t.Fatal("precondition: the gate should have been free")
}
defer release()
rr := &rotRunner{pass: true}
s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
s.tick(context.Background())
concurrent := 1 + len(rr.seen()) // the backup holding the gate, plus anything the tick started
if concurrent != 1 {
t.Fatalf("the restore-test must DEFER while a backup holds the gate; concurrent operations = %d", concurrent)
}
}
// Once the backup releases, the next cadence proceeds — deferral must not be permanent.
func TestRotation_ResumesAfterTheGateFrees(t *testing.T) {
gate := &InFlight{}
release, _, _ := gate.TryAcquire("backup:local")
rr := &rotRunner{pass: true}
s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
s.tick(context.Background())
if len(rr.seen()) != 0 {
t.Fatal("should have deferred while held")
}
release()
s.tick(context.Background())
if len(rr.seen()) != 1 {
t.Fatalf("must resume once the gate frees; got %v", rr.seen())
}
}
// The gate itself: one holder at a time, named, and release is idempotent.
func TestInFlight_Semantics(t *testing.T) {
g := &InFlight{}
rel, busy, ok := g.TryAcquire("backup:local")
if !ok || busy != "" {
t.Fatalf("first acquire must succeed; ok=%v busy=%q", ok, busy)
}
if _, busy2, ok2 := g.TryAcquire("restore-test"); ok2 || busy2 != "backup:local" {
t.Fatalf("second acquire must fail and NAME the holder; ok=%v busy=%q", ok2, busy2)
}
rel()
rel() // idempotent — a double release must not free someone else's later claim
if g.Busy() != "" {
t.Fatalf("gate should be idle after release; busy=%q", g.Busy())
}
if _, _, ok3 := g.TryAcquire("restore-test"); !ok3 {
t.Fatal("gate must be reusable after release")
}
}
// A nil gate means "not wired" → no gating, pre-R-85 behaviour. Keeps every existing caller working.
func TestInFlight_NilIsUngated(t *testing.T) {
var g *InFlight
rel, _, ok := g.TryAcquire("x")
if !ok {
t.Fatal("a nil gate must not block")
}
rel()
if g.Busy() != "" {
t.Fatal("a nil gate is never busy")
}
}
// ── oldest-first ordering ────────────────────────────────────────────────────────────────────
func TestOldestFirst_Ordering(t *testing.T) {
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
now := time.Now().UTC()
// Never-proven sorts FIRST — the case that matters, since the offsite tier starts there.
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
// both never proven → deterministic tie-break by id
if got[0] != "felhom-pbs" && got[0] != "local" {
t.Fatalf("unexpected: %v", got)
}
}
_ = st.RecordSuccess("local", now)
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
}
_ = st.RecordSuccess("felhom-pbs", now.Add(time.Hour))
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
t.Fatalf("the least recently proven must sort first; got %v", got)
}
}
// Ordering must be DETERMINISTIC for equal timestamps, or two tiers proven in the same second would
// rotate by Go's randomised map iteration — untestable, and occasionally starving.
func TestOldestFirst_DeterministicOnTies(t *testing.T) {
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
now := time.Now().UTC()
_ = st.RecordSuccess("b-tier", now)
_ = st.RecordSuccess("a-tier", now)
for i := 0; i < 20; i++ {
if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
}
}
}
// The state file round-trips, and a corrupt file degrades to "nothing proven" rather than wedging.
func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "rt.json")
now := time.Now().UTC().Truncate(time.Second)
st := NewRestoreTestState(path)
if err := st.RecordSuccess("felhom-pbs", now); err != nil {
t.Fatal(err)
}
reopened := NewRestoreTestState(path)
got, ok := reopened.LastSuccess("felhom-pbs")
if !ok || !got.Equal(now) {
t.Fatalf("state must round-trip; got %v ok=%v want %v", got, ok, now)
}
bad := filepath.Join(dir, "corrupt.json")
if err := os.WriteFile(bad, []byte("{{{not json"), 0o600); err != nil {
t.Fatal(err)
}
c := NewRestoreTestState(bad)
if _, ok := c.LastSuccess("felhom-pbs"); ok {
t.Fatal("a corrupt state file must degrade to 'nothing proven', not invent a timestamp")
}
}
+175 -7
View File
@@ -6,6 +6,7 @@ import (
"log/slog" "log/slog"
"sort" "sort"
"strings" "strings"
"sync"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub" "gitea.dooplex.hu/admin/felhom-agent/internal/hub"
@@ -38,21 +39,57 @@ type BackupRunner struct {
// each successful backup, so the agent's own backups can't pile up and refill root. Empty → no prune // each successful backup, so the agent's own backups can't pile up and refill root. Empty → no prune
// (the legacy behaviour; restore-test/selftest runners pass ""). NEVER applied to a PBS target. // (the legacy behaviour; restore-test/selftest runners pass ""). NEVER applied to a PBS target.
retention string retention string
logger *slog.Logger // waitTimeout bounds the WaitTask poll on this runner's vzdump. Per-TIER since R-82: 30m is
now func() time.Time // right for a local vzdump and badly wrong for an offsite PBS upload (see the 2026-07-26 live
// failure recorded on config.BackupTargetConfig.WaitTimeoutSeconds). 0 → 30m (legacy).
waitTimeout time.Duration
// allowPBSPrune permits `--prune-backups` on a PBS-type target. OFF by default and ON only for
// an ADDITIONAL tier whose keep_last was set explicitly (operator ruling 2026-07-26: keep two
// weeks of weekly offsite backups).
//
// The blanket PBS refusal it replaces existed for a real reason and still applies to the
// PRIMARY tier: BackupTarget() DEFAULTS to "felhom-pbs" and KeepLast() DEFAULTS to 3, so a box
// with neither key set would silently prune its offsite DR to 3 restore points. An additional
// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
// there is a deliberate act.
allowPBSPrune bool
logger *slog.Logger
now func() time.Time
// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
// due-check is served from the local-API handler goroutines.
rejectedMu sync.Mutex
rejected map[string]struct{}
} }
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and // NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the // for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it. // per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
func NewBackupRunner(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, logger *slog.Logger) *BackupRunner { func NewBackupRunner(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, logger *slog.Logger) *BackupRunner {
return NewBackupRunnerWithWait(api, target, mode, notes, retention, 0, logger)
}
// NewBackupRunnerWithWait is NewBackupRunner plus an explicit vzdump wait bound (0 → 30m).
func NewBackupRunnerWithWait(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, logger *slog.Logger) *BackupRunner {
return NewBackupRunnerFull(api, target, mode, notes, retention, waitTimeout, false, logger)
}
// NewBackupRunnerFull is the full constructor. allowPBSPrune must be true ONLY for an additional
// tier with an explicitly configured keep_last — see BackupRunner.allowPBSPrune.
func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, allowPBSPrune bool, logger *slog.Logger) *BackupRunner {
if mode == "" { if mode == "" {
mode = proxmox.ModeSnapshot mode = proxmox.ModeSnapshot
} }
if logger == nil { if logger == nil {
logger = slog.Default() logger = slog.Default()
} }
return &BackupRunner{api: api, target: target, mode: mode, notes: notes, retention: retention, logger: logger, now: func() time.Time { return time.Now().UTC() }} if waitTimeout <= 0 {
waitTimeout = 30 * time.Minute
}
return &BackupRunner{api: api, target: target, mode: mode, notes: notes, retention: retention,
waitTimeout: waitTimeout, allowPBSPrune: allowPBSPrune, logger: logger,
now: func() time.Time { return time.Now().UTC() }}
} }
// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It // localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
@@ -71,8 +108,11 @@ func (r *BackupRunner) localPruneSpec(ctx context.Context) string {
} }
for _, s := range stores { for _, s := range stores {
if s.Storage == r.target { if s.Storage == r.target {
if s.Type == "pbs" { if s.Type == "pbs" && !r.allowPBSPrune {
return "" // PBS retention is out of scope — never prune the offsite DR // Not opted in → never prune the offsite DR (the pre-R-82 rule, and still the rule
// for the primary tier, whose target+retention both DEFAULT and could prune by
// accident).
return ""
} }
return r.retention return r.retention
} }
@@ -147,7 +187,7 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
defer stopWatch() defer stopWatch()
go r.watchForSnapshot(watchCtx, upid, onSnapshot) go r.watchForSnapshot(watchCtx, upid, onSnapshot)
} }
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: 30 * time.Minute}); err != nil { if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil {
rec.Error = err.Error() rec.Error = err.Error()
rec.DurationSeconds = time.Since(start).Seconds() rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err) return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
@@ -210,7 +250,20 @@ func (r *BackupRunner) watchForSnapshot(ctx context.Context, upid string, onSnap
// PickRestoreCandidate returns the newest backup archive on the target (any guest), or "" // PickRestoreCandidate returns the newest backup archive on the target (any guest), or ""
// when there is none — the restore-test then no-ops cleanly. // when there is none — the restore-test then no-ops cleanly.
func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error) { func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error) {
contents, err := r.api.StorageContent(ctx, r.target) return r.PickRestoreCandidateOn(ctx, r.target)
}
// PickRestoreCandidateOn is PickRestoreCandidate for an ARBITRARY tier's storage (R-85 1.2), so the
// scheduler can rotate across tiers instead of only ever seeing this runner's own target.
//
// Contract preserved: "" + nil error when the storage holds no archive. **A tier with nothing to
// restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and turning
// that into a failure would make every fresh box look broken for its first week.
func (r *BackupRunner) PickRestoreCandidateOn(ctx context.Context, target string) (string, error) {
if target == "" {
return "", nil
}
contents, err := r.api.StorageContent(ctx, target)
if err != nil { if err != nil {
return "", err return "", err
} }
@@ -243,6 +296,121 @@ func (r *BackupRunner) latestArchive(ctx context.Context, vmid int) (string, int
return vol, size, nil return vol, size, nil
} }
// NewestArchiveTime reports when this guest's newest backup archive LANDED ON THIS TARGET, from the
// storage itself. ok=false means the target genuinely holds no archive for this guest.
//
// R-84: this is the cure for the redundant-backup-after-restart problem. The agent's backup Store is
// in-memory ("lost on restart; the cadence re-populates"), so after every restart /backup/due
// reported "no successful backup recorded yet" and the controller dutifully took another one. On the
// local tier that is wasted minutes; on the OFFSITE tier it is a wasted multi-hour WAN upload after
// every agent deploy — and agent deploys are routine. Three redundant local backups were observed on
// minPlausibleArchiveBytes is the floor below which a storage entry cannot be a real whole-guest
// backup and is therefore treated as an INCOMPLETE artefact rather than a successful one.
//
// MEASURED, not chosen by feel — fleet survey 2026-07-28 (Campaign 8, finding F-CRIT-2):
//
// smallest REAL backup anywhere on the fleet ... 612,397,450 B (~584 MiB, a guest-9100 vzdump)
// demo-hp local / PBS ..................... 1.59 GB / 4.35-4.37 GB
// demo-felhom local / PBS ..................... 5.82-5.84 GB / 14.47-14.51 GB
// the phantom left by a PBS daemon killed mid-upload ....... 1 B
//
// 1 MiB sits 584x below the smallest real backup and 1,048,576x above the phantom. The two
// populations are nine orders of magnitude apart, so this floor cannot plausibly clip a real
// archive — which is the property that matters, because a floor set too HIGH does not merely lose
// safety margin, it causes fleet-wide backup THRASH (see archivePlausiblyComplete).
const minPlausibleArchiveBytes int64 = 1 << 20
// archivePlausiblyComplete reports whether a storage entry can be a COMPLETE backup, and if not,
// why. Pure, so the contract is unit-testable without a storage.
//
// WHY SIZE, AND NOTHING ELSE. The richer PBS fields look like better discriminators and are all
// traps, because this runner is TIER-AGNOSTIC — the same predicate runs against a PBS datastore and
// against a plain `dir` storage (verified against the live PVE API, 2026-07-28):
//
// - `verification` is absent on the phantom, but ALSO absent on every local (dir) archive — a dir
// storage has no verification concept — and absent on a good PBS snapshot until verify-new
// catches up. Gating on it would reject 100% of local backups and every freshly-taken offsite
// one: continuous re-backup across the fleet.
// - `encrypted` fails the same way, and for the same reason.
// - `notes` happens to be present on both good tiers today only because the agent sets it; an
// archive written by any other path lacks it. Too fragile to gate freshness on.
//
// Size is the only signal that means the same thing on every tier.
//
// THE FAIL-SAFE DIRECTION, stated explicitly: when completeness cannot be established the entry is
// NOT counted as a successful backup. That errs toward the tier looking LESS fresh, and its worst
// case is one extra backup. Counting an undecidable entry is precisely the F-CRIT-2 defect — a
// failed upload that made its tier look freshly backed up and silenced it for a full cadence.
func archivePlausiblyComplete(e proxmox.StorageContent) (bool, string) {
if e.Size < minPlausibleArchiveBytes {
return false, fmt.Sprintf("size %d B is below the %d B plausibility floor — an aborted/incomplete archive, not a successful backup",
e.Size, minPlausibleArchiveBytes)
}
return true, ""
}
// warnRejectedArchiveOnce announces a rejected archive at WARN exactly once per distinct volid.
//
// A rejected archive must never be silent: a tier that quietly ignores the newest entry on its
// storage is a new quiet path, and quiet paths are what F-CRIT-2 was. But the due-check runs every
// 5 minutes and a phantom persists indefinitely — server-side prune does NOT collect it (verified
// by dry-run 2026-07-28: with keep-last 2 it retained two real snapshots PLUS the phantom) — so
// logging per poll would emit ~288 identical lines a day and bury the one that matters.
func (r *BackupRunner) warnRejectedArchiveOnce(e proxmox.StorageContent, why string) {
r.rejectedMu.Lock()
if r.rejected == nil {
r.rejected = map[string]struct{}{}
}
_, seen := r.rejected[e.VolID]
if !seen {
r.rejected[e.VolID] = struct{}{}
}
r.rejectedMu.Unlock()
if seen {
return
}
r.logger.Warn("backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup",
"target", r.target, "vmid", e.VMID, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
}
// demo-felhom in a single afternoon of deploys (2026-07-26).
//
// Asking the STORAGE rather than persisting the store is deliberate:
// - it is ground truth, not remembered state — if an archive was pruned or deleted it correctly
// stops counting, whereas a persisted record would keep claiming a backup that no longer exists;
// - it needs no new on-disk state and no migration;
// - it is the same source `latestArchive` already trusts to build the post-backup record.
//
// It answers ONLY "when did a plausibly COMPLETE backup last land", which is exactly what the
// due-check needs. Completeness is not optional here: PBS publishes an aborted upload into the same
// listing (manifest-less, 1 byte, and NEWEST), and counting it made the tier report fresh and go
// silent for a whole cadence — F-CRIT-2. Presence is not validity. The
// richer fields (size, duration, uncovered volumes, error) stay with the real in-memory records — a
// synthesized record would put invented numbers into the host-report.
func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Time, bool, error) {
contents, err := r.api.StorageContent(ctx, r.target)
if err != nil {
return time.Time{}, false, err
}
var best int64 = -1
for _, e := range contents {
if e.Content != "backup" || e.VMID != vmid {
continue
}
if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why)
continue
}
if e.CTime > best {
best = e.CTime
}
}
if best < 0 {
return time.Time{}, false, nil
}
return time.Unix(best, 0).UTC(), true, nil
}
// parseBackupMode extracts the actual mode from a vzdump task log line `… backup mode: <x>` // parseBackupMode extracts the actual mode from a vzdump task log line `… backup mode: <x>`
// (e.g. "INFO: backup mode: stop"). Returns "" if not found. // (e.g. "INFO: backup mode: stop"). Returns "" if not found.
func parseBackupMode(lines []string) string { func parseBackupMode(lines []string) string {
+133 -16
View File
@@ -18,6 +18,25 @@ type RestoreTestRunner interface {
// there is none yet (the tick then no-ops). // there is none yet (the tick then no-ops).
type CandidatePicker func(ctx context.Context) (string, error) type CandidatePicker func(ctx context.Context) (string, error)
// SpecBuilder yields the RestoreTestSpec for ONE run, given the archive that was picked.
//
// R-85 (1.1): this REPLACES a frozen spec value. It used to be built by an immediately-invoked
// function at daemon start, so `storageTier()` and `restoreTaskTimeout()` were evaluated ONCE and
// the resulting value reused for every run for the lifetime of the process. Two consequences:
// - nothing tier-varying was expressible at all (the offsite tier could never be scheduled), and
// - it was a latent staleness bug in its own right — a storage-type or config change did not take
// effect until the daemon restarted.
//
// The archive is passed in because the tier MUST be derived from it (the v0.100.0 rule), never from
// the configured target: deriving it from config is what produced the 600 s false failure when a
// PBS archive was classified "local" and got the 10-minute local wait.
type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
// TierPicker resolves the newest archive on a NAMED tier, or "" when that tier holds none.
// (*BackupRunner).PickRestoreCandidateOn satisfies it. "" must NOT be an error — a brand-new
// offsite tier legitimately has nothing to restore yet.
type TierPicker func(ctx context.Context, target string) (string, error)
// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon // Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled // goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
// AND a valid scratch band is configured (validated by the caller before construction). // AND a valid scratch band is configured (validated by the caller before construction).
@@ -25,20 +44,37 @@ type Scheduler struct {
runner RestoreTestRunner runner RestoreTestRunner
pick CandidatePicker pick CandidatePicker
store *Store store *Store
spec reconcile.RestoreTestSpec // archive is filled per-tick spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
cadence time.Duration cadence time.Duration
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
// (single tier via `pick`), which keeps every existing caller and test working untouched.
tiers []string // configured tier target ids, primary first
tierPick TierPicker // newest archive on a named tier
rtState *RestoreTestState // persisted last-successful-per-tier (drives oldest-first)
inFlight *InFlight // shared with the backup path — Scenario F
} }
// SchedulerOptions configures a Scheduler. // SchedulerOptions configures a Scheduler.
type SchedulerOptions struct { type SchedulerOptions struct {
Runner RestoreTestRunner Runner RestoreTestRunner
Pick CandidatePicker Pick CandidatePicker
Store *Store Store *Store
Spec reconcile.RestoreTestSpec // RestoreStorage, ScratchMin/Max, SourceTier, BootTimeout // Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
Cadence time.Duration // 0 → disabled // picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
Spec SpecBuilder
Cadence time.Duration // 0 → disabled
Logger *slog.Logger Logger *slog.Logger
// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
// named tier; State persists last-successful-per-tier; InFlight is the shared one-heavy-op gate.
Tiers []string
TierPick TierPicker
State *RestoreTestState
InFlight *InFlight
} }
// NewScheduler builds a Scheduler. // NewScheduler builds a Scheduler.
@@ -48,13 +84,17 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
logger = slog.Default() logger = slog.Default()
} }
return &Scheduler{ return &Scheduler{
runner: opts.Runner, runner: opts.Runner,
pick: opts.Pick, pick: opts.Pick,
store: opts.Store, store: opts.Store,
spec: opts.Spec, spec: opts.Spec,
cadence: opts.Cadence, cadence: opts.Cadence,
logger: logger, logger: logger,
now: func() time.Time { return time.Now().UTC() }, now: func() time.Time { return time.Now().UTC() },
tiers: append([]string(nil), opts.Tiers...),
tierPick: opts.TierPick,
rtState: opts.State,
inFlight: opts.InFlight,
} }
} }
@@ -63,7 +103,7 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
// is heavy; the first runs one interval in) — on-demand runs use the selftest harness. // is heavy; the first runs one interval in) — on-demand runs use the selftest harness.
// Returns nil on ctx cancellation. // Returns nil on ctx cancellation.
func (s *Scheduler) Run(ctx context.Context) error { func (s *Scheduler) Run(ctx context.Context) error {
if s.cadence <= 0 || s.runner == nil || s.pick == nil { if s.cadence <= 0 || s.runner == nil || s.spec == nil || (s.pick == nil && !s.rotating()) {
s.logger.Info("backup: restore-test cadence disabled") s.logger.Info("backup: restore-test cadence disabled")
<-ctx.Done() <-ctx.Done()
return nil return nil
@@ -85,7 +125,26 @@ func (s *Scheduler) Run(ctx context.Context) error {
// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when // tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when
// no backup exists yet. Deterministic given s.now — tests call it directly. // no backup exists yet. Deterministic given s.now — tests call it directly.
func (s *Scheduler) tick(ctx context.Context) { func (s *Scheduler) tick(ctx context.Context) {
archive, err := s.pick(ctx) if s.spec == nil {
// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
// reachable directly. Skipping loudly beats panicking the daemon goroutine — a missing
// spec must cost a restore-test, never the agent.
s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
return
}
// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
// never cancel what is already running: a deferred restore-test costs hours of coverage, a
// cancelled backup costs the backup.
release, busy, ok := s.inFlight.TryAcquire("restore-test")
if !ok {
s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight", "busy", busy)
return
}
defer release()
archive, target, err := s.pickForThisRun(ctx)
if err != nil { if err != nil {
s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err) s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
return return
@@ -94,7 +153,8 @@ func (s *Scheduler) tick(ctx context.Context) {
s.logger.Info("backup: restore-test skipped; no backup available yet") s.logger.Info("backup: restore-test skipped; no backup available yet")
return return
} }
spec := s.spec // R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
spec := s.spec(ctx, archive)
spec.Archive = archive spec.Archive = archive
res := s.runner.RunRestoreTest(ctx, spec) res := s.runner.RunRestoreTest(ctx, spec)
if res.Skipped { if res.Skipped {
@@ -102,6 +162,13 @@ func (s *Scheduler) tick(ctx context.Context) {
} }
rt := ToHubRestoreTest(res, s.now()) rt := ToHubRestoreTest(res, s.now())
s.store.RecordRestoreTest(rt) s.store.RecordRestoreTest(rt)
// Rotation credit is given ONLY on success. A failing tier must keep sorting first, or a tier
// that fails every time would look freshly proven and quietly stop being retried.
if rt.Pass && s.rtState != nil && target != "" {
if err := s.rtState.RecordSuccess(target, s.now()); err != nil {
s.logger.Warn("backup: could not persist the restore-test rotation state", "target", target, "err", err)
}
}
switch { switch {
case !rt.Pass: case !rt.Pass:
// A failing restore-test is the loudest DR signal there is. // A failing restore-test is the loudest DR signal there is.
@@ -118,3 +185,53 @@ func (s *Scheduler) tick(ctx context.Context) {
"archive", rt.SourceArchive, "duration_s", rt.DurationSeconds, "warnings", res.StartWarnings) "archive", rt.SourceArchive, "duration_s", rt.DurationSeconds, "warnings", res.StartWarnings)
} }
} }
// rotating reports whether multi-tier rotation is wired.
func (s *Scheduler) rotating() bool { return len(s.tiers) > 0 && s.tierPick != nil }
// pickForThisRun chooses the tier and its newest archive.
//
// OLDEST-FIRST (operator ruling 2026-07-26, Option 1): the tier whose last SUCCESSFUL restore-test
// is oldest goes first, never-proven first of all. Self-balancing, no config knob, and it naturally
// prioritises a tier that has never been proven — which on this fleet was the offsite tier, unproven
// for its entire existence while reporting `applied`.
//
// A tier with no archives is SKIPPED, not failed, and the next tier is tried. Skipping to a testable
// tier is strictly better than burning the whole cadence: a brand-new offsite tier has nothing to
// restore yet, and that is normal, not broken. It cannot starve the empty tier either — as soon as
// it has an archive it still sorts first, because it is still the least recently proven.
//
// Returns ("", "", nil) when nothing anywhere is testable.
func (s *Scheduler) pickForThisRun(ctx context.Context) (archive, target string, err error) {
if !s.rotating() {
a, perr := s.pick(ctx)
return a, "", perr // pre-R-85 single-tier path; no rotation credit to record
}
order := s.tiers
if s.rtState != nil {
order = s.rtState.OldestFirst(s.tiers)
}
var firstErr error
for _, t := range order {
a, perr := s.tierPick(ctx, t)
if perr != nil {
// One tier's storage being unreadable must not block the others.
s.logger.Warn("backup: restore-test candidate lookup failed for a tier; trying the next",
"target", t, "err", perr)
if firstErr == nil {
firstErr = perr
}
continue
}
if a == "" {
s.logger.Debug("backup: restore-test tier has no archive yet; trying the next", "target", t)
continue
}
s.logger.Info("backup: restore-test tier selected (oldest-proven first)", "target", t, "archive", a)
return a, t, nil
}
if firstErr != nil {
return "", "", firstErr
}
return "", "", nil
}
+124
View File
@@ -0,0 +1,124 @@
package backup
import (
"context"
"fmt"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// R-85 (1.1) — the spec is built PER RUN, never frozen at construction.
//
// It used to be an immediately-invoked function at daemon start, so storageTier() and
// restoreTaskTimeout() were evaluated ONCE and the value reused for every run for the process
// lifetime. That is what made an offsite restore-test impossible to schedule at all, and it was a
// latent staleness bug besides: a storage-type or config change did not take effect until restart.
type specSpy struct {
mu sync.Mutex
calls int
archives []string
tiers []string // what the builder decided, per call
}
func (sp *specSpy) build(_ context.Context, archive string) reconcile.RestoreTestSpec {
sp.mu.Lock()
defer sp.mu.Unlock()
sp.calls++
sp.archives = append(sp.archives, archive)
// Decide the tier from the ARCHIVE, exactly as main.go does (the v0.100.0 rule).
tier := "local"
if len(archive) > 10 && archive[:10] == "felhom-pbs" {
tier = "pbs"
}
sp.tiers = append(sp.tiers, tier)
return reconcile.RestoreTestSpec{
RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: tier,
}
}
// COMPANION RED-PROOF (observed): change Scheduler.spec back to a frozen
// `reconcile.RestoreTestSpec` value captured at construction → this fails with
// "the spec builder must run ONCE PER RUN, got 1 call(s) across 3 ticks", because a frozen value is
// evaluated exactly once no matter how many ticks fire. Restored.
func TestScheduler_SpecIsBuiltPerRun(t *testing.T) {
sp := &specSpy{}
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
n := 0
s := NewScheduler(SchedulerOptions{
Runner: rt,
Pick: func(context.Context) (string, error) {
n++
return fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", n), nil
},
Store: NewStore(),
Spec: sp.build,
Cadence: time.Hour,
Logger: quiet(),
})
for i := 0; i < 3; i++ {
s.tick(context.Background())
}
sp.mu.Lock()
defer sp.mu.Unlock()
if sp.calls != 3 {
t.Fatalf("the spec builder must run ONCE PER RUN, got %d call(s) across 3 ticks", sp.calls)
}
// And it must see the archive THIS run picked — not a stale one.
for i, a := range sp.archives {
want := fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", i+1)
if a != want {
t.Fatalf("run %d: builder saw archive %q, want %q — the spec is not tracking the picked archive", i+1, a, want)
}
}
}
// The tier must follow the ARCHIVE across runs. A builder that saw only the configured target would
// return the same tier every time — which is exactly the v0.100.0 defect that killed a 14.46 GB WAN
// restore at the 10-minute local bound.
func TestScheduler_SpecTierFollowsTheArchive(t *testing.T) {
sp := &specSpy{}
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
archives := []string{
"local:backup/vzdump-lxc-9201-x.tar.zst",
"felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
}
i := 0
s := NewScheduler(SchedulerOptions{
Runner: rt,
Pick: func(context.Context) (string, error) {
a := archives[i%len(archives)]
i++
return a, nil
},
Store: NewStore(), Spec: sp.build, Cadence: time.Hour, Logger: quiet(),
})
s.tick(context.Background())
s.tick(context.Background())
sp.mu.Lock()
defer sp.mu.Unlock()
if len(sp.tiers) != 2 || sp.tiers[0] != "local" || sp.tiers[1] != "pbs" {
t.Fatalf("the tier must follow the archive per run; got %v", sp.tiers)
}
}
// A nil spec builder must SKIP loudly, not panic — a wiring bug costs a restore-test, never the
// daemon goroutine.
func TestScheduler_NilSpecSkipsInsteadOfPanicking(t *testing.T) {
rt := &fakeRTRunner{}
s := NewScheduler(SchedulerOptions{
Runner: rt,
Pick: func(context.Context) (string, error) { return "vol", nil },
Store: NewStore(), Cadence: time.Hour, Logger: quiet(),
})
s.tick(context.Background()) // must not panic
if rt.runs != 0 {
t.Fatalf("a nil spec must not run a restore-test; got %d run(s)", rt.runs)
}
}
+182
View File
@@ -0,0 +1,182 @@
package config
import (
"encoding/json"
"strings"
"testing"
"time"
)
// R-82 Slice A.1 — per-target cadence + retention resolution.
//
// The load-bearing property is ADDITIVITY: every config that exists on a live box today must
// resolve to exactly one tier that behaves as it does now. The second property is that a
// mis-configured tier is REJECTED LOUDLY rather than defaulted — a weekly DR tier silently running
// daily would fill the datastore, and a silently dropped tier is the "applied and empty" fault
// R-82 exists to fix.
func TestBackupTiers_LegacyConfigIsUnchanged(t *testing.T) {
// Exactly the shape live on demo-felhom today.
var b BackupConfig
raw := `{"local_backup_target":"local","local_backup_retention":3,"backup_cadence_seconds":0}`
if err := json.Unmarshal([]byte(raw), &b); err != nil {
t.Fatal(err)
}
tiers, warnings := b.BackupTiers()
if len(warnings) != 0 {
t.Fatalf("a legacy config must produce NO warnings; got %v", warnings)
}
if len(tiers) != 1 {
t.Fatalf("a config with no backup_targets must resolve to exactly ONE tier; got %+v", tiers)
}
got := tiers[0]
if got.TargetID != "local" || got.Cadence != 24*time.Hour || got.KeepLast != 3 || !got.Primary {
t.Fatalf("legacy tier changed: %+v", got)
}
}
// An empty BackupConfig still resolves — to the felhom-pbs default target, 24h, keep-last 3.
// (Unchanged pre-R-82 behaviour; pinned so the default target can't drift unnoticed.)
func TestBackupTiers_ZeroConfigKeepsDefaults(t *testing.T) {
tiers, warnings := BackupConfig{}.BackupTiers()
if len(warnings) != 0 || len(tiers) != 1 {
t.Fatalf("zero config: tiers=%+v warnings=%v", tiers, warnings)
}
if tiers[0].TargetID != defaultBackupTarget || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
t.Fatalf("zero-config defaults changed: %+v", tiers[0])
}
}
// The whole point: local daily + PBS weekly, each with its OWN retention.
func TestBackupTiers_LocalDailyPlusPBSWeekly(t *testing.T) {
var b BackupConfig
raw := `{
"local_backup_target":"local",
"local_backup_retention":3,
"backup_cadence_seconds":86400,
"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]
}`
if err := json.Unmarshal([]byte(raw), &b); err != nil {
t.Fatal(err)
}
tiers, warnings := b.BackupTiers()
if len(warnings) != 0 {
t.Fatalf("unexpected warnings: %v", warnings)
}
if len(tiers) != 2 {
t.Fatalf("want 2 tiers, got %+v", tiers)
}
if !tiers[0].Primary || tiers[0].TargetID != "local" || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
t.Fatalf("primary tier wrong: %+v", tiers[0])
}
if tiers[1].Primary || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour || tiers[1].KeepLast != 2 {
t.Fatalf("PBS tier wrong: %+v", tiers[1])
}
// THE knob-sharing check: the two retentions are independent values, not one shared number.
if tiers[0].KeepLast == tiers[1].KeepLast {
t.Fatalf("this fixture sets 3 and 2 deliberately — equal values mean the knob is shared: %+v", tiers)
}
}
// A tier with no cadence is REJECTED, not defaulted. Defaulting would turn a weekly DR tier into a
// daily one and fill the 37.2 GB datastore (R-82 Phase 0, P0.3).
func TestBackupTiers_MissingCadenceIsRejectedLoudly(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", KeepLast: 2}},
}
tiers, warnings := b.BackupTiers()
if len(tiers) != 1 {
t.Fatalf("a cadence-less tier must NOT be armed; got %+v", tiers)
}
if len(warnings) != 1 || !strings.Contains(warnings[0], "cadence_seconds must be > 0") {
t.Fatalf("rejection must be reported so the caller can log it loudly; got %v", warnings)
}
if !strings.Contains(warnings[0], "felhom-pbs") {
t.Fatalf("the warning must name the tier it dropped; got %q", warnings[0])
}
}
func TestBackupTiers_RejectsEmptyAndDuplicateTargets(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{
{TargetID: "", CadenceSeconds: 3600},
{TargetID: "local", CadenceSeconds: 3600}, // repeats the primary
{TargetID: "felhom-pbs", CadenceSeconds: 604800}, // good
{TargetID: "felhom-pbs", CadenceSeconds: 99}, // duplicate
},
}
tiers, warnings := b.BackupTiers()
if len(tiers) != 2 || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour {
t.Fatalf("want primary + one PBS tier at the FIRST definition; got %+v", tiers)
}
if len(warnings) != 3 {
t.Fatalf("want 3 rejections (empty, duplicate-of-primary, duplicate); got %v", warnings)
}
}
// keep_last unset means DO NOT PRUNE. That is the fail-safe: a DR tier must never start pruning
// itself because someone forgot a field.
func TestBackupTiers_UnsetKeepLastMeansNoPrune(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 2 {
t.Fatalf("got %+v", tiers)
}
if tiers[1].KeepLast != 0 {
t.Fatalf("an unset keep_last must resolve to 0 = never prune; got %d", tiers[1].KeepLast)
}
// And a negative is clamped to the same fail-safe rather than becoming a prune spec.
b.ExtraTargets[0].KeepLast = -5
tiers, _ = b.BackupTiers()
if tiers[1].KeepLast != 0 {
t.Fatalf("a negative keep_last must clamp to 0 (never prune); got %d", tiers[1].KeepLast)
}
}
// The primary's retention still comes from the legacy knob with its legacy clamp — untouched.
func TestBackupTiers_PrimaryRetentionClampUnchanged(t *testing.T) {
for _, tc := range []struct{ in, want int }{{0, 3}, {-1, 3}, {1, 1}, {7, 7}} {
b := BackupConfig{LocalBackupTarget: "local", LocalBackupRetention: tc.in}
tiers, _ := b.BackupTiers()
if tiers[0].KeepLast != tc.want {
t.Fatalf("LocalBackupRetention=%d → KeepLast=%d, want %d", tc.in, tiers[0].KeepLast, tc.want)
}
}
}
// R-82 live-failure regression (2026-07-26): the runner hard-coded a 30-minute vzdump wait, which
// is right for a local vzdump and wrong for an offsite PBS upload. The first full ~10 GB PBS
// snapshot on demo-felhom ran past 30 min; the agent gave up waiting and recorded success=false
// WHILE THE BACKUP WAS STILL RUNNING — a false failure that leaves the tier permanently "due" and
// makes the next attempt collide with the guest lock vzdump still holds.
func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 2 {
t.Fatalf("got %+v", tiers)
}
if tiers[0].WaitTimeout != 30*time.Minute {
t.Fatalf("the PRIMARY must keep the historical 30m wait (unchanged behaviour); got %s", tiers[0].WaitTimeout)
}
if tiers[1].WaitTimeout != 12*time.Hour {
t.Fatalf("an offsite tier must default to a GENEROUS wait (operator ruling: let the first backup run as long as needed) — a false timeout is worse than a slow pass; got %s", tiers[1].WaitTimeout)
}
// And it must be overridable per tier.
b.ExtraTargets[0].WaitTimeoutSeconds = 3600
tiers, _ = b.BackupTiers()
if tiers[1].WaitTimeout != time.Hour {
t.Fatalf("wait_timeout_seconds must override; got %s", tiers[1].WaitTimeout)
}
// The two tiers must NOT share one bound.
if tiers[0].WaitTimeout == tiers[1].WaitTimeout {
t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
}
}
+153
View File
@@ -235,6 +235,16 @@ type LocalAPIConfig struct {
// TokenStore is the durable, hashed token→guest map (only a HASH of each token is // TokenStore is the durable, hashed token→guest map (only a HASH of each token is
// persisted; the plaintext exists transiently at mint→write-to-mount, then is discarded). // persisted; the plaintext exists transiently at mint→write-to-mount, then is discarded).
TokenStore string `json:"token_store"` // default /var/lib/felhom-agent/local-tokens.log TokenStore string `json:"token_store"` // default /var/lib/felhom-agent/local-tokens.log
// IslandBridge + IslandGuestAddr configure the R-50 host-internal control-plane bridge. When
// BOTH are set, the provisioner attaches each guest a static net1 on IslandBridge with
// IslandGuestAddr, so the controller reaches the agent over a fixed private address that no
// LAN/DHCP/site move can invalidate (the F1 fix — AUDIT-vacation-remote-ops-2026-07-20). Empty
// (the default) = LAN-only, byte-for-byte the pre-R-50 behaviour. On an island install ListenAddr
// is the host side (169.254.253.1:8443); IslandGuestAddr is the guest side (169.254.253.2/30 — a
// /30 is exactly host + one guest). Additive-only: it never removes a NIC, so a guest restored on
// a non-island host (both empty) is unaffected.
IslandBridge string `json:"island_bridge"` // e.g. "vmbr9" (portless host-internal bridge)
IslandGuestAddr string `json:"island_guest_addr"` // guest net1 CIDR, e.g. "169.254.253.2/30"
} }
// Default local-API file locations (under the agent's state dir). // Default local-API file locations (under the agent's state dir).
@@ -249,6 +259,12 @@ func (l LocalAPIConfig) Enabled() bool {
return l.Enable && strings.TrimSpace(l.ListenAddr) != "" return l.Enable && strings.TrimSpace(l.ListenAddr) != ""
} }
// IslandEnabled reports whether the provisioner should attach a guest island NIC (net1). True only
// when BOTH the bridge and the guest CIDR are set (R-50); empty = pre-R-50 LAN-only behaviour.
func (l LocalAPIConfig) IslandEnabled() bool {
return strings.TrimSpace(l.IslandBridge) != "" && strings.TrimSpace(l.IslandGuestAddr) != ""
}
// TokenStorePath returns the configured token-store path (default applied). // TokenStorePath returns the configured token-store path (default applied).
func (l LocalAPIConfig) TokenStorePath() string { func (l LocalAPIConfig) TokenStorePath() string {
if l.TokenStore != "" { if l.TokenStore != "" {
@@ -283,6 +299,17 @@ func (l LocalAPIConfig) Validate() error {
if _, _, err := net.SplitHostPort(l.ListenAddr); err != nil { if _, _, err := net.SplitHostPort(l.ListenAddr); err != nil {
return fmt.Errorf("config: local_api.listen_addr %q is not host:port: %w", l.ListenAddr, err) return fmt.Errorf("config: local_api.listen_addr %q is not host:port: %w", l.ListenAddr, err)
} }
// R-50: island fields are all-or-nothing, and the guest addr must be a CIDR (the net1 ip= value).
// A half-set island (bridge without guest addr, or vice versa) is a provisioning mistake, not a
// silent LAN fallback — fail loudly so a botched install config is caught at load, not at day-0.
if (strings.TrimSpace(l.IslandBridge) != "") != (strings.TrimSpace(l.IslandGuestAddr) != "") {
return fmt.Errorf("config: local_api.island_bridge and local_api.island_guest_addr must be set together (got bridge=%q guest_addr=%q)", l.IslandBridge, l.IslandGuestAddr)
}
if l.IslandEnabled() {
if _, _, err := net.ParseCIDR(strings.TrimSpace(l.IslandGuestAddr)); err != nil {
return fmt.Errorf("config: local_api.island_guest_addr %q is not a CIDR (want e.g. 169.254.253.2/30): %w", l.IslandGuestAddr, err)
}
}
return nil return nil
} }
@@ -339,6 +366,132 @@ type BackupConfig struct {
// default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup. // default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup.
// NEVER applied to a PBS target (offsite retention is a separate lifecycle). // NEVER applied to a PBS target (offsite retention is a separate lifecycle).
LocalBackupRetention int `json:"local_backup_retention"` LocalBackupRetention int `json:"local_backup_retention"`
// ExtraTargets (R-82) are ADDITIONAL backup tiers beyond the primary one above — the shape that
// makes "local daily + PBS weekly" expressible at all. Each carries its OWN cadence and its OWN
// retention, because those are semantically different per tier: keep-last=3 on a daily tier is
// three DAYS of restore points; on a weekly tier it is three WEEKS. Sharing one knob between
// tiers silently means one of them is wrong.
//
// ADDITIVE BY CONSTRUCTION: an existing config with no `backup_targets` key resolves to exactly
// one tier — the primary — and behaves byte-identically to pre-R-82. Nothing here changes the
// local tier.
ExtraTargets []BackupTargetConfig `json:"backup_targets"`
}
// BackupTargetConfig is ONE additional backup tier: a vzdump storage plus its own cadence and
// retention. A tier with no cadence is not a tier — see BackupTiers for why that is rejected loudly
// rather than defaulted.
type BackupTargetConfig struct {
// TargetID is the Proxmox storage id (content=backup), e.g. "felhom-pbs".
TargetID string `json:"target_id"`
// CadenceSeconds is THIS tier's /backup/due window. REQUIRED (>0) — see BackupTiers.
CadenceSeconds int `json:"cadence_seconds"`
// KeepLast is THIS tier's per-run `--prune-backups` keep-last. 0/unset → NEVER prune this tier
// (the fail-safe default, and the current behaviour for every PBS target). A PBS tier is never
// pruned by the per-run flag regardless — see BackupRunner.localPruneSpec.
KeepLast int `json:"keep_last"`
// WaitTimeoutSeconds bounds how long the agent WAITS for this tier's vzdump task. 0/unset →
// defaultExtraTierWaitTimeout.
//
// THIS FIELD EXISTS BECAUSE OF A LIVE FAILURE (2026-07-26, R-82 Slice A validation). The runner
// hard-coded a 30-minute wait, which is right for a local vzdump (minutes) and badly wrong for
// an offsite PBS backup over a home uplink: the first full ~10 GB snapshot ran past 30 min, the
// agent gave up waiting and recorded success=false — WHILE THE BACKUP WAS STILL RUNNING. That
// false failure is worse than a slow pass: the tier stays "due", a retry collides with the
// guest lock vzdump still holds, and the hub sees a DR tier that never succeeds.
//
// Same reasoning as RestoreTestPBSRestoreTimeoutSeconds on the restore side, and the same
// direction: when in doubt wait LONGER. A slow backup is a slow backup; a false timeout is a
// corrupt status plus lock contention.
WaitTimeoutSeconds int `json:"wait_timeout_seconds"`
}
// Per-tier vzdump wait bounds.
//
// The PRIMARY keeps the historical 30 minutes: it is the local tier, a local vzdump takes minutes,
// and one hanging 30 minutes is a genuine fault worth surfacing. Unchanged behaviour.
//
// An ADDITIONAL tier is by construction the offsite/WAN one in this design, where the binding
// constraint is uplink speed, not health. Measured on demo-felhom: ~33 MB/min over the wg link to
// Hetzner, so a first FULL ~10 GB snapshot projects to ~5h. Operator ruling 2026-07-26: "let the
// first backup run as long as needed" — 12h gives that real margin on a slower link while still
// being BOUNDED, so a genuinely hung task eventually surfaces instead of hanging forever.
const (
defaultPrimaryTierWaitTimeout = 30 * time.Minute
defaultExtraTierWaitTimeout = 12 * time.Hour
)
// BackupTier is a RESOLVED backup tier: one target, its own cadence, its own retention. The agent
// builds one runner per tier from these.
type BackupTier struct {
TargetID string
Cadence time.Duration
// WaitTimeout bounds the wait on this tier's vzdump task (see WaitTimeoutSeconds).
WaitTimeout time.Duration
// KeepLast is the per-run prune keep-last; 0 means DO NOT PRUNE this tier.
KeepLast int
// Primary marks the tier that the UNTARGETED local-API endpoints act on — the pre-R-82 tier.
// Exactly one tier is primary, and it is always first.
Primary bool
}
// BackupTiers resolves the effective tier list, primary first, plus any warnings the caller MUST
// log (they describe tiers that were REJECTED, and a silently-dropped backup tier is precisely the
// "applied and empty" fault R-82 exists to fix).
//
// Rules:
// - Tier 0 is always the primary, built from BackupTarget()/BackupCadence()/KeepLast() — so a
// config with no `backup_targets` is byte-identical to pre-R-82.
// - An extra with an empty target_id is rejected.
// - An extra with cadence_seconds <= 0 is REJECTED, not defaulted. Defaulting a PBS tier to the
// 24h local default would quietly turn a weekly tier into a daily one and fill the DR datastore;
// a tier whose cadence you did not state is not a tier.
// - An extra repeating the primary's target is rejected (one policy per target, or the two
// cadences race and neither is the truth).
// - Duplicate extras are rejected after the first.
func (b BackupConfig) BackupTiers() ([]BackupTier, []string) {
primary := BackupTier{
TargetID: b.BackupTarget(),
Cadence: b.BackupCadence(),
KeepLast: b.KeepLast(),
WaitTimeout: defaultPrimaryTierWaitTimeout,
Primary: true,
}
tiers := []BackupTier{primary}
var warnings []string
seen := map[string]bool{primary.TargetID: true}
for i, t := range b.ExtraTargets {
id := strings.TrimSpace(t.TargetID)
switch {
case id == "":
warnings = append(warnings, fmt.Sprintf("backup_targets[%d]: empty target_id — tier ignored", i))
continue
case seen[id]:
warnings = append(warnings, fmt.Sprintf("backup_targets[%d]: target %q already configured — duplicate tier ignored", i, id))
continue
case t.CadenceSeconds <= 0:
warnings = append(warnings, fmt.Sprintf("backup_targets[%d] (%s): cadence_seconds must be > 0 — tier ignored (a cadence is NOT defaulted: a weekly tier silently running daily would fill the DR datastore)", i, id))
continue
}
seen[id] = true
keep := t.KeepLast
if keep < 0 {
keep = 0
}
wait := defaultExtraTierWaitTimeout
if t.WaitTimeoutSeconds > 0 {
wait = time.Duration(t.WaitTimeoutSeconds) * time.Second
}
tiers = append(tiers, BackupTier{
TargetID: id,
Cadence: time.Duration(t.CadenceSeconds) * time.Second,
KeepLast: keep,
WaitTimeout: wait,
})
}
return tiers, warnings
} }
// defaultLocalBackupKeepLast is the local vzdump retention default (newest N restore points kept). // defaultLocalBackupKeepLast is the local vzdump retention default (newest N restore points kept).
+43
View File
@@ -171,3 +171,46 @@ func TestDeploymentModeEnvOverlay(t *testing.T) {
t.Errorf("env overlay did not set deployment_mode: %q", cfg.DeploymentMode) t.Errorf("env overlay did not set deployment_mode: %q", cfg.DeploymentMode)
} }
} }
// R-50: the island NIC fields are all-or-nothing and the guest addr must be a CIDR. A half-set or
// malformed island must fail at config load (a botched install) rather than silently fall back to
// LAN-only, which would leave a guest with an island bind and no island NIC — the exact silent break
// R-50 exists to kill. Covers LocalAPIConfig.Validate + IslandEnabled.
func TestLocalAPIConfig_IslandValidation(t *testing.T) {
base := LocalAPIConfig{Enable: true, ListenAddr: "169.254.253.1:8443"}
// both empty → fine (pre-R-50 default), IslandEnabled false
if err := base.Validate(); err != nil {
t.Errorf("no island config must validate: %v", err)
}
if base.IslandEnabled() {
t.Errorf("IslandEnabled must be false when unset")
}
// both set, valid CIDR → fine, IslandEnabled true
ok := base
ok.IslandBridge, ok.IslandGuestAddr = "vmbr9", "169.254.253.2/30"
if err := ok.Validate(); err != nil {
t.Errorf("valid island config must validate: %v", err)
}
if !ok.IslandEnabled() {
t.Errorf("IslandEnabled must be true when both set")
}
// bridge only → rejected (all-or-nothing)
half := base
half.IslandBridge = "vmbr9"
if err := half.Validate(); err == nil {
t.Errorf("half-set island (bridge only) must be rejected")
}
// guest addr only → rejected
half2 := base
half2.IslandGuestAddr = "169.254.253.2/30"
if err := half2.Validate(); err == nil {
t.Errorf("half-set island (guest addr only) must be rejected")
}
// both set but guest addr is not a CIDR → rejected
bad := base
bad.IslandBridge, bad.IslandGuestAddr = "vmbr9", "169.254.253.2" // missing /30
if err := bad.Validate(); err == nil {
t.Errorf("island guest addr without a CIDR mask must be rejected")
}
}
+18
View File
@@ -548,3 +548,21 @@ func TestClassify_Table(t *testing.T) {
} }
var _ io.Writer = (*bytes.Buffer)(nil) var _ io.Writer = (*bytes.Buffer)(nil)
// A3 (R-50): the guestnet healer is eth0-only and MUST stay blind to the island NIC. A guest on an
// island host presents eth0 DHCP (the LAN leg the healer owns) PLUS eth1 static (the island). Because
// parseMode is interface-scoped, adding eth1 static cannot flip eth0's detected mode — so the healer
// keeps treating eth0 as DHCP and never runs dhclient against the static island NIC (which would
// sabotage it). This is the verify-only guarantee that let R-50 ship the island NIC without a healer
// change. Red-proof: make parseMode scan globally instead of per-dev and the eth0 assertion fails.
func TestParseMode_IslandStaticNICDoesNotConfuseEth0(t *testing.T) {
interfaces := "auto lo\niface lo inet loopback\n\n" +
"auto eth0\niface eth0 inet dhcp\n\n" +
"auto eth1\niface eth1 inet static\n address 169.254.253.2/30\n"
if got := parseMode(interfaces, "eth0"); got != ModeDHCP {
t.Errorf("eth0 must classify DHCP even with an island eth1 static present, got %q", got)
}
if got := parseMode(interfaces, "eth1"); got != ModeStatic {
t.Errorf("eth1 (island) must classify static when asked directly (dev-scoped), got %q", got)
}
}
+28 -1
View File
@@ -83,12 +83,14 @@ type Collector struct {
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp) temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty) capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled) leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted) wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted) pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted) guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false) selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted) mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted) oob OOBReporter // H1: operator-access health (nil → stanza omitted)
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
hostID string hostID string
agentVersion string agentVersion string
logger *slog.Logger logger *slog.Logger
@@ -123,6 +125,30 @@ func (c *Collector) SetTempReader(t TempReader) *Collector {
return c return c
} }
// SetBackupTargetResolver wires the DR recipe to the agent's own backup config (R-109), so the recipe
// can name WHICH storage holds the local whole-guest archives. Returns the collector for chaining.
//
// The resolver MUST report the tier that is IN EFFECT, which is the daemon-start snapshot — NOT the
// current contents of agent.json. A backup-target move rewrites that file and deliberately does not
// restart the agent (the E-1 lesson: restarting mid-backup records a spurious failure for a run that
// succeeded), so between the write and the restart the file names a target no backup is writing to yet.
// Re-reading the file here — the live-reload shape used for escrow.pbs_storage_id — would make the
// recipe point at the new storage while every archive still landed on the old one. One state, one
// owner: the recipe follows what performs the backup.
func (c *Collector) SetBackupTargetResolver(f func() ConfiguredBackupTarget) *Collector {
c.backupTarget = f
return c
}
// configuredBackupTarget consults the resolver. An unwired seam is reported as NOT KNOWN — never as a
// guess — so the recipe records an explicit unknown instead of a target the agent never verified.
func (c *Collector) configuredBackupTarget() ConfiguredBackupTarget {
if c.backupTarget == nil {
return ConfiguredBackupTarget{}
}
return c.backupTarget()
}
// SetCapabilityProber wires the privileged-capability self-check (v0.44.0): each collect runs it // SetCapabilityProber wires the privileged-capability self-check (v0.44.0): each collect runs it
// and attaches the snapshot. nil → the report carries an empty []. Returns the collector for chaining. // and attaches the snapshot. nil → the report carries an empty []. Returns the collector for chaining.
func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capability.Status) *Collector { func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capability.Status) *Collector {
@@ -228,10 +254,11 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
Cloudflared: Cloudflared{Status: c.cloudflaredStatus(ctx)}, Cloudflared: Cloudflared{Status: c.cloudflaredStatus(ctx)},
Capabilities: c.capabilities(ctx), Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP, LeafFingerprint: c.leafFP,
Addresses: c.collectAddresses(),
} }
// DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads). // DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads).
// Secret-free by construction (identifiers/intents/sizes/coordinates only). // Secret-free by construction (identifiers/intents/sizes/coordinates only).
report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots) report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots, c.configuredBackupTarget())
// S3: offsite-tunnel status stanza (nil reporter = feature disabled → omitted; the pubkey in // S3: offsite-tunnel status stanza (nil reporter = feature disabled → omitted; the pubkey in
// it is the operator's revocation-recovery handle). // it is the operator's revocation-recovery handle).
if c.wg != nil { if c.wg != nil {
+17 -1
View File
@@ -45,6 +45,14 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000, State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000,
Smart: SmartSummary{Health: SmartUnknown}, Smart: SmartSummary{Health: SmartUnknown},
}, },
// A pbs target so the recipe's pbs coord has a storage.cfg row to resolve its namespace
// from (R-106). Without one the fixture produces namespace_state=unknown while the golden
// pins a resolved coord — key-set-equal but semantically a fiction.
{
Name: "felhom-pbs", Type: StorageTypePBS, DurableID: "repo+fp", Content: "backup",
State: StorageStateAttached, Reachable: true, PBSNamespace: "felhom-spike",
Smart: SmartSummary{Health: SmartUnknown},
},
}, },
Backups: []Backup{ Backups: []Backup{
{ {
@@ -72,9 +80,13 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
}, },
AuditTail: []AuditEntry{}, AuditTail: []AuditEntry{},
Cloudflared: Cloudflared{Status: "active"}, Cloudflared: Cloudflared{Status: "active"},
// v0.119.0: host addresses. Populated so the bidirectional key-set guard exercises the new
// element keys, not just the presence of the array.
Addresses: []HostAddress{{Iface: "vmbr0", CIDR: "192.168.0.162/24"}},
} }
// dr_recipe host-half: built from the same guest/storage/pbs facts (the production path). // dr_recipe host-half: built from the same guest/storage/pbs facts (the production path).
report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots) report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots,
ConfiguredBackupTarget{StorageID: "usb-backup", Known: true})
b, _ := json.Marshal(report) b, _ := json.Marshal(report)
var got map[string]any var got map[string]any
json.Unmarshal(b, &got) json.Unmarshal(b, &got)
@@ -97,6 +109,9 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
assertSameKeys(t, "restore_tests[0]", firstElem(golden["restore_tests"]), firstElem(got["restore_tests"])) assertSameKeys(t, "restore_tests[0]", firstElem(golden["restore_tests"]), firstElem(got["restore_tests"]))
// slice-6-Phase-B addition — pbs_snapshots[0] key set. // slice-6-Phase-B addition — pbs_snapshots[0] key set.
assertSameKeys(t, "pbs_snapshots[0]", firstElem(golden["pbs_snapshots"]), firstElem(got["pbs_snapshots"])) assertSameKeys(t, "pbs_snapshots[0]", firstElem(golden["pbs_snapshots"]), firstElem(got["pbs_snapshots"]))
// v0.119.0 addition — addresses[0] key set (iface/cidr), the cross-repo wire for the hub's
// Network card.
assertSameKeys(t, "addresses[0]", firstElem(golden["addresses"]), firstElem(got["addresses"]))
// DR-recipe host-half — the agent's secret-free reconstruction-scaffolding section. Assert the // DR-recipe host-half — the agent's secret-free reconstruction-scaffolding section. Assert the
// dr_recipe key set + each sub-array's element key set (the cross-repo wire pinned in the golden). // dr_recipe key set + each sub-array's element key set (the cross-repo wire pinned in the golden).
@@ -106,6 +121,7 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
assertSameKeys(t, "dr_recipe.guests[0]", firstElem(field(grec, "guests")), firstElem(field(srec, "guests"))) assertSameKeys(t, "dr_recipe.guests[0]", firstElem(field(grec, "guests")), firstElem(field(srec, "guests")))
assertSameKeys(t, "dr_recipe.drives[0]", firstElem(field(grec, "drives")), firstElem(field(srec, "drives"))) assertSameKeys(t, "dr_recipe.drives[0]", firstElem(field(grec, "drives")), firstElem(field(srec, "drives")))
assertSameKeys(t, "dr_recipe.pve_storage[0]", firstElem(field(grec, "pve_storage")), firstElem(field(srec, "pve_storage"))) assertSameKeys(t, "dr_recipe.pve_storage[0]", firstElem(field(grec, "pve_storage")), firstElem(field(srec, "pve_storage")))
assertSameKeys(t, "dr_recipe.backup_target", field(grec, "backup_target"), field(srec, "backup_target"))
} }
// field extracts a nested object value from a decoded JSON map (nil if absent/not a map). // field extracts a nested object value from a decoded JSON map (nil if absent/not a map).
+138 -10
View File
@@ -30,6 +30,36 @@ import "sort"
// ignore-unknown (encoding/json default) for forward-compat, mirroring storage_manifest. // ignore-unknown (encoding/json default) for forward-compat, mirroring storage_manifest.
const DRRecipeVersion = 1 const DRRecipeVersion = 1
// Recipe field states (R-106/R-109). A recipe is read at the worst possible moment — by an operator
// rebuilding a machine that is gone — so a field the agent cannot resolve must SAY SO rather than emit
// a default, an empty string, or a plausible-looking placeholder. A guess read as fact costs more than
// an admitted gap: it sends the restore at the wrong archive and nothing contradicts it. This is the
// same cannot-tell-must-not-lie rule R-117 needed a third state for.
const (
DRStateResolved = "resolved"
DRStateUnknown = "unknown"
)
// Reasons a resolved-value field is unknown. Enum-shaped, never free text, so the wire stays pinnable
// and TestDRRecipeHostHalf_NoSecrets has a fixed vocabulary to walk.
const (
// DRReasonNoBackupConfig: the collector was built without a backup-config seam, so the agent could
// not consult the very config its own scheduler reads. Nothing is guessed.
DRReasonNoBackupConfig = "agent_backup_config_unavailable"
// DRReasonNoSuchStorage: the configured target id matches no storage this host observes. The id is
// still recorded (it IS what the config says) and the state says it could not be corroborated.
DRReasonNoSuchStorage = "not_a_known_storage"
// DRReasonNoPBSStorage: snapshots exist but no pbs storage was observed, so there is no storage.cfg
// row to read the namespace from.
DRReasonNoPBSStorage = "no_pbs_storage_observed"
)
// PBSRootNamespace is how the recipe spells PBS's root namespace. The PBS API spells it as the EMPTY
// string (and `pct restore --ns root` would name a namespace that does not exist) — "root" is a display
// convention this wire has always used, kept here so the field's meaning did not change under R-106.
// Only a box with no `namespace` line in its pbs storage.cfg stanza ever emits it.
const PBSRootNamespace = "root"
// DRRecipeHostHalf is the agent-emitted half (guest/drive/storage/PBS scaffolding). Derived entirely // DRRecipeHostHalf is the agent-emitted half (guest/drive/storage/PBS scaffolding). Derived entirely
// from facts the report already collects — no new privileged reads. // from facts the report already collects — no new privileged reads.
type DRRecipeHostHalf struct { type DRRecipeHostHalf struct {
@@ -38,6 +68,45 @@ type DRRecipeHostHalf struct {
PBS *DRPBSCoord `json:"pbs,omitempty"` PBS *DRPBSCoord `json:"pbs,omitempty"`
Drives []DRDrive `json:"drives"` Drives []DRDrive `json:"drives"`
PVEStorage []DRPVEStorage `json:"pve_storage"` PVEStorage []DRPVEStorage `json:"pve_storage"`
// BackupTarget names WHICH storage holds the local whole-guest archives (R-109). Always present —
// its own State field carries "I could not tell", so the section is never simply absent.
BackupTarget *DRBackupTarget `json:"backup_target"`
}
// DRBackupTarget answers the one question pve_storage cannot: of every storage listed there, WHICH one
// does this box's primary backup tier actually write its whole-guest archives to?
//
// Before R-109 the recipe listed each storage's name/type/content and said nothing about the target.
// That was harmless while the target was the well-known `local`; the 2026-07-28 vzdump-target move
// ended that. Every box now carries TWO content=backup dir storages — `felhom-backup` (live) and
// `local` (archives frozen at the move, never refreshed since) — and they are indistinguishable by
// name, type and content alone. A restorer picking the frozen one gets a guest that restores cleanly
// and is silently months out of date, which is the worst shape a backup defect can take.
type DRBackupTarget struct {
// State is DRStateResolved | DRStateUnknown. A reader MUST consult it before trusting StorageID:
// the id is also recorded in one unknown case (see DRReasonNoSuchStorage).
State string `json:"state"`
// StorageID is the PVE storage id of the PRIMARY backup tier. Empty only when the config could not
// be consulted at all.
StorageID string `json:"storage_id,omitempty"`
// MountPath is where that storage's archives land on the host — the disambiguation a restorer
// actually needs, since it is what separates felhom-backup's /mnt/hdd_1 from local's /var/lib/vz.
// "" for a pbs target (no host mount) and when unresolved.
MountPath string `json:"mount_path,omitempty"`
// Reason is why State is unknown (one of the DRReason* constants); "" when resolved.
Reason string `json:"reason,omitempty"`
}
// ConfiguredBackupTarget is what the agent's own backup config says the PRIMARY tier writes to.
//
// Known=false is a REAL state, not a nil-guard: it means the collector was constructed without the
// backup-config seam (the --selftest one-shots did exactly this before v0.118.0), and the recipe then
// records unknown instead of inventing a target. Deliberately a struct rather than a `(string, bool)`
// return — the (value, ok) shape is what made "errors degrade to unknown, never to no-backup"
// unimplementable in newestArchiveOn, and this field has the same three-way reading.
type ConfiguredBackupTarget struct {
StorageID string
Known bool
} }
// DRGuest is the sizing needed to recreate the LXC at the right size (GuestSpec, already on the wire). // DRGuest is the sizing needed to recreate the LXC at the right size (GuestSpec, already on the wire).
@@ -51,8 +120,21 @@ type DRGuest struct {
// DRPBSCoord is WHERE the whole-CT snapshot lives — COORDINATES ONLY. The encryption key is escrow-only; // DRPBSCoord is WHERE the whole-CT snapshot lives — COORDINATES ONLY. The encryption key is escrow-only;
// the access token is identity-escrow-only. Neither is here. // the access token is identity-escrow-only. Neither is here.
type DRPBSCoord struct { type DRPBSCoord struct {
RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token
Namespace string `json:"namespace"` // PBS namespace the restore targets // Namespace is the PBS namespace the restore targets, resolved from the pbs storage's storage.cfg
// stanza — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree
// with the backup that produced the snapshot. PBSRootNamespace when the box has no namespace
// configured; "" when NamespaceState is unknown.
//
// R-106: this used to come from the listed snapshot's own `ns`, which PBS does not echo per item once
// the request is already namespace-scoped via `?ns=` (internal/pbs/client.go). The field was
// therefore always empty, ToHub normalised empty → "root", and every per-customer box reported the
// root namespace while its backups were really in `demo-hp` / `demo-felhom`.
Namespace string `json:"namespace"`
// NamespaceState is DRStateResolved | DRStateUnknown — consult it before trusting Namespace.
NamespaceState string `json:"namespace_state"`
// NamespaceReason is why NamespaceState is unknown; "" when resolved.
NamespaceReason string `json:"namespace_reason,omitempty"`
LatestSnapshotID string `json:"latest_snapshot_id"` // most-recent snapshot's backup_id (a coordinate) LatestSnapshotID string `json:"latest_snapshot_id"` // most-recent snapshot's backup_id (a coordinate)
} }
@@ -83,7 +165,7 @@ const driveIntentEnrolled = "enrolled"
// it is unit-tested directly (no live reads). drives[] = the user-data external drives (usb/local-dir // it is unit-tested directly (no live reads). drives[] = the user-data external drives (usb/local-dir
// with a durable-id); pve_storage[] = every storage target (the storage.cfg scaffolding); pbs = the // with a durable-id); pve_storage[] = every storage target (the storage.cfg scaffolding); pbs = the
// latest PBS snapshot's coordinates; guests[] = each guest's sizing (skip guests with no spec). // latest PBS snapshot's coordinates; guests[] = each guest's sizing (skip guests with no spec).
func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSnapshot) *DRRecipeHostHalf { func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSnapshot, backupTarget ConfiguredBackupTarget) *DRRecipeHostHalf {
h := &DRRecipeHostHalf{ h := &DRRecipeHostHalf{
RecipeVersion: DRRecipeVersion, RecipeVersion: DRRecipeVersion,
Guests: []DRGuest{}, Guests: []DRGuest{},
@@ -103,11 +185,14 @@ func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSna
}) })
} }
var pbsRepoID string var pbsRepoID, pbsNamespace string
var pbsStorageFound bool
for _, t := range targets { for _, t := range targets {
h.PVEStorage = append(h.PVEStorage, DRPVEStorage{Name: t.Name, Type: t.Type, Content: t.Content}) h.PVEStorage = append(h.PVEStorage, DRPVEStorage{Name: t.Name, Type: t.Type, Content: t.Content})
if t.Type == StorageTypePBS && pbsRepoID == "" { if t.Type == StorageTypePBS && !pbsStorageFound {
pbsRepoID = t.Name // the pbs storage id is a coordinate, not the key pbsStorageFound = true
pbsRepoID = t.Name // the pbs storage id is a coordinate, not the key
pbsNamespace = t.PBSNamespace // storage.cfg's namespace — "" here means the ROOT namespace
} }
if isUserDataDrive(t) { if isUserDataDrive(t) {
h.Drives = append(h.Drives, DRDrive{ h.Drives = append(h.Drives, DRDrive{
@@ -119,12 +204,41 @@ func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSna
} }
} }
if c := latestPBSCoord(pbs, pbsRepoID); c != nil { h.BackupTarget = resolveBackupTarget(targets, backupTarget)
if c := latestPBSCoord(pbs, pbsRepoID, pbsNamespace, pbsStorageFound); c != nil {
h.PBS = c h.PBS = c
} }
return h return h
} }
// resolveBackupTarget records WHICH storage the primary backup tier writes to (R-109), or records
// explicitly that it could not tell. Three outcomes, and the two unknowns are deliberately distinct —
// "I could not read my own config" and "my config names a storage that is not here" send an operator
// to different places.
//
// MountPath prefers the live mount and falls back to the CONFIGURED path: during a rebuild the drive is
// frequently absent, and when it is, MountPath empties out while ConfigPath is the only thing left that
// still says which drive the row was about (the R-116 lesson). The storage's absence from the host is a
// separate signal (E-2's backup_target_absent); it does not make the recipe's answer unknown, because
// the question here is which storage.cfg row to restore FROM, and that is still known.
func resolveBackupTarget(targets []StorageTarget, cfg ConfiguredBackupTarget) *DRBackupTarget {
if !cfg.Known || cfg.StorageID == "" {
return &DRBackupTarget{State: DRStateUnknown, Reason: DRReasonNoBackupConfig}
}
for _, t := range targets {
if t.Name != cfg.StorageID {
continue
}
mount := t.MountPath
if mount == "" {
mount = t.ConfigPath
}
return &DRBackupTarget{State: DRStateResolved, StorageID: cfg.StorageID, MountPath: mount}
}
return &DRBackupTarget{State: DRStateUnknown, StorageID: cfg.StorageID, Reason: DRReasonNoSuchStorage}
}
// isUserDataDrive selects the external user-data drives the recipe enumerates (felhom-usb / felhom-flash // isUserDataDrive selects the external user-data drives the recipe enumerates (felhom-usb / felhom-flash
// class): a usb or local-dir storage with a filesystem-UUID durable id and a host mount path. local / // class): a usb or local-dir storage with a filesystem-UUID durable id and a host mount path. local /
// lvmthin / pbs / nfs / cifs are scaffolding (they land in pve_storage) but are not user-data drives. // lvmthin / pbs / nfs / cifs are scaffolding (they land in pve_storage) but are not user-data drives.
@@ -137,16 +251,30 @@ func isUserDataDrive(t StorageTarget) bool {
// latestPBSCoord picks the most-recent snapshot (lexical max of the RFC3339 backup_time) and returns // latestPBSCoord picks the most-recent snapshot (lexical max of the RFC3339 backup_time) and returns
// its coordinates. Returns nil when there is no snapshot to target. // its coordinates. Returns nil when there is no snapshot to target.
func latestPBSCoord(snaps []PBSSnapshot, repoID string) *DRPBSCoord { //
// The namespace comes from the pbs STORAGE (storage.cfg), never from the snapshot — see DRPBSCoord's
// Namespace comment for why the snapshot's own field cannot answer it (R-106). storageFound=false with
// snapshots present is a genuine unknown: something listed snapshots, but there is no storage row to
// read a namespace from, so the recipe says so rather than defaulting to root.
func latestPBSCoord(snaps []PBSSnapshot, repoID, namespace string, storageFound bool) *DRPBSCoord {
if len(snaps) == 0 { if len(snaps) == 0 {
return nil return nil
} }
sorted := append([]PBSSnapshot(nil), snaps...) sorted := append([]PBSSnapshot(nil), snaps...)
sort.Slice(sorted, func(i, j int) bool { return sorted[i].BackupTime > sorted[j].BackupTime }) sort.Slice(sorted, func(i, j int) bool { return sorted[i].BackupTime > sorted[j].BackupTime })
latest := sorted[0] latest := sorted[0]
return &DRPBSCoord{ c := &DRPBSCoord{
RepoID: repoID, RepoID: repoID,
Namespace: latest.Namespace,
LatestSnapshotID: latest.BackupID, LatestSnapshotID: latest.BackupID,
NamespaceState: DRStateUnknown,
NamespaceReason: DRReasonNoPBSStorage,
} }
if storageFound {
c.NamespaceState, c.NamespaceReason = DRStateResolved, ""
// An empty configured namespace is not a missing answer — it IS the root namespace.
if c.Namespace = namespace; c.Namespace == "" {
c.Namespace = PBSRootNamespace
}
}
return c
} }
+295 -2
View File
@@ -1,8 +1,10 @@
package hub package hub
import ( import (
"context"
"encoding/json" "encoding/json"
"regexp" "regexp"
"strings"
"testing" "testing"
) )
@@ -30,7 +32,7 @@ func TestBuildDRRecipeHostHalf(t *testing.T) {
{Namespace: "root", BackupID: "9201", BackupTime: "2026-06-16T08:00:00Z"}, // latest {Namespace: "root", BackupID: "9201", BackupTime: "2026-06-16T08:00:00Z"}, // latest
} }
h := BuildDRRecipeHostHalf(guests, targets, pbs) h := BuildDRRecipeHostHalf(guests, targets, pbs, ConfiguredBackupTarget{StorageID: "felhom-flash", Known: true})
if h.RecipeVersion != 1 { if h.RecipeVersion != 1 {
t.Errorf("recipe_version=%d, want 1", h.RecipeVersion) t.Errorf("recipe_version=%d, want 1", h.RecipeVersion)
@@ -68,7 +70,8 @@ func TestBuildDRRecipeHostHalf(t *testing.T) {
// TestBuildDRRecipeHostHalf_NoPBS: no snapshots → pbs omitted (nil), no panic. // TestBuildDRRecipeHostHalf_NoPBS: no snapshots → pbs omitted (nil), no panic.
func TestBuildDRRecipeHostHalf_NoPBS(t *testing.T) { func TestBuildDRRecipeHostHalf_NoPBS(t *testing.T) {
h := BuildDRRecipeHostHalf(nil, []StorageTarget{{Name: "local", Type: StorageTypeLocal}}, nil) h := BuildDRRecipeHostHalf(nil, []StorageTarget{{Name: "local", Type: StorageTypeLocal}}, nil,
ConfiguredBackupTarget{StorageID: "local", Known: true})
if h.PBS != nil { if h.PBS != nil {
t.Errorf("pbs should be nil with no snapshots, got %+v", h.PBS) t.Errorf("pbs should be nil with no snapshots, got %+v", h.PBS)
} }
@@ -89,6 +92,7 @@ func TestDRRecipeHostHalf_V1DriveShape(t *testing.T) {
MountPath: "/mnt/felhom-usb", TotalBytes: 931 << 30}, MountPath: "/mnt/felhom-usb", TotalBytes: 931 << 30},
}, },
nil, nil,
ConfiguredBackupTarget{StorageID: "felhom-usb", Known: true},
) )
if len(h.Drives) != 1 { if len(h.Drives) != 1 {
t.Fatalf("want 1 drive, got %d", len(h.Drives)) t.Fatalf("want 1 drive, got %d", len(h.Drives))
@@ -124,6 +128,7 @@ func TestDRRecipeHostHalf_NoSecrets(t *testing.T) {
{Name: "felhom-usb", Type: StorageTypeUSB, DurableID: "uuid:da9e7089", Role: "bulk-data", MountPath: "/mnt/felhom-usb", TotalBytes: 1}, {Name: "felhom-usb", Type: StorageTypeUSB, DurableID: "uuid:da9e7089", Role: "bulk-data", MountPath: "/mnt/felhom-usb", TotalBytes: 1},
}, },
[]PBSSnapshot{{Namespace: "root", BackupID: "9201", BackupTime: "2026-06-16T08:00:00Z"}}, []PBSSnapshot{{Namespace: "root", BackupID: "9201", BackupTime: "2026-06-16T08:00:00Z"}},
ConfiguredBackupTarget{StorageID: "felhom-usb", Known: true},
) )
b, err := json.Marshal(h) b, err := json.Marshal(h)
if err != nil { if err != nil {
@@ -132,6 +137,294 @@ func TestDRRecipeHostHalf_NoSecrets(t *testing.T) {
assertNoSecretKeys(t, b) assertNoSecretKeys(t, b)
} }
// ---------------------------------------------------------------------------------------------
// R-106 / R-109 — the recipe records the RESOLVED backup target and the REAL PBS namespace.
// ---------------------------------------------------------------------------------------------
// capturedDemoFelhomTargets is the storage set demo-felhom really had on 2026-07-30, not an invented
// one. PROVENANCE — every field was captured, none composed:
//
// - names/types/contents: the pve_storage block of the box's own PRE-FIX recipe, downloaded from the
// hub at GET /customers/demo-felhom/dr-recipe.json (agent v0.115.0).
// - paths + is_mountpoint + the pbs namespace: `cat /etc/pve/storage.cfg` on felhom-pve, same day —
// `dir: local path /var/lib/vz`, `dir: felhom-backup path /mnt/hdd_1 is_mountpoint 1`,
// `pbs: felhom-pbs ... namespace demo-felhom`.
//
// THE AMBIGUITY THIS PINS IS REAL, and assertBackupCandidateAmbiguity below refuses to let the fixture
// quietly lose it: `local` and `felhom-backup` BOTH carry content=backup, and since the 2026-07-28
// vzdump-target move `local` holds archives frozen at that date. Naming the wrong one restores a guest
// that is silently months stale.
func capturedDemoFelhomTargets() []StorageTarget {
return []StorageTarget{
{Name: "local-lvm", Type: StorageTypeLVMThin, DurableID: "pve/data", Content: "images,rootdir"},
{
Name: "felhom-backup", Type: StorageTypeLocalDir, Content: "backup",
DurableID: "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48",
MountPath: "/mnt/hdd_1", ConfigPath: "/mnt/hdd_1", TotalBytes: 983351140352,
},
{
Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup",
DurableID: "repo+fp", PBSNamespace: "demo-felhom",
},
// The decoy: same content, plausible name, historically THE vzdump target. ConfigPath only —
// `local` lives on the LVM root and is not its own mount, so the observer leaves MountPath empty.
{Name: "local", Type: StorageTypeLocal, Content: "backup,import,vztmpl,iso", ConfigPath: "/var/lib/vz"},
}
}
// capturedDemoFelhomSnapshots mirrors what the box's pre-fix recipe carried: latest_snapshot_id "9201".
// Namespace is deliberately EMPTY on every element — that is exactly what the PBS API returns once the
// list is namespace-scoped via `?ns=`, and it is the input that used to become the bogus "root".
func capturedDemoFelhomSnapshots() []PBSSnapshot {
return []PBSSnapshot{
{Namespace: "", BackupID: "9201", BackupTime: "2026-07-29T22:00:00Z"},
{Namespace: "", BackupID: "9201", BackupTime: "2026-07-30T22:00:00Z"}, // latest
}
}
// assertBackupCandidateAmbiguity fails if the fixture stopped containing TWO plausible content=backup
// storages. Without this the consequence test below could pass on a fixture with only one candidate —
// which is precisely the hollow shape that let two defects ship green earlier in this arc.
func assertBackupCandidateAmbiguity(t *testing.T, h *DRRecipeHostHalf) {
t.Helper()
var candidates []string
for _, s := range h.PVEStorage {
if strings.Contains(s.Content, "backup") && (s.Type == StorageTypeLocalDir || s.Type == StorageTypeLocal) {
candidates = append(candidates, s.Name)
}
}
if len(candidates) < 2 {
t.Fatalf("fixture no longer poses the R-109 problem: want >=2 content=backup dir storages, got %v", candidates)
}
}
// TestDRRecipe_BackupTargetNamesTheLiveStorage is THE consequence assertion for R-109: given a box that
// really carries two content=backup dir storages, the generated recipe names the LIVE one, gives its
// mountpoint, and does not name the frozen one. Not "the function returned a non-empty string".
func TestDRRecipe_BackupTargetNamesTheLiveStorage(t *testing.T) {
h := BuildDRRecipeHostHalf(nil, capturedDemoFelhomTargets(), capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
assertBackupCandidateAmbiguity(t, h)
bt := h.BackupTarget
if bt == nil {
t.Fatal("backup_target is absent — the recipe still cannot say where the local archives are (R-109)")
}
if bt.State != DRStateResolved {
t.Errorf("state=%q want %q (reason=%q)", bt.State, DRStateResolved, bt.Reason)
}
if bt.StorageID != "felhom-backup" {
t.Errorf("storage_id=%q — the recipe must name the LIVE target, not %q", bt.StorageID, "felhom-backup")
}
if bt.MountPath != "/mnt/hdd_1" {
t.Errorf("mount_path=%q want /mnt/hdd_1 — the mountpoint is what separates it from local's /var/lib/vz", bt.MountPath)
}
// Unambiguous: the frozen decoy must not be what the field names.
if bt.StorageID == "local" || bt.MountPath == "/var/lib/vz" {
t.Errorf("recipe names the FROZEN target (%q at %q) — a restore from it is silently stale", bt.StorageID, bt.MountPath)
}
}
// TestDRRecipe_PBSNamespaceIsThePerCustomerOne is the consequence assertion for R-106: the recipe carries
// the namespace the box's backups actually live in, resolved from storage.cfg, and specifically NOT the
// "root" that every box used to report.
func TestDRRecipe_PBSNamespaceIsThePerCustomerOne(t *testing.T) {
h := BuildDRRecipeHostHalf(nil, capturedDemoFelhomTargets(), capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
if h.PBS == nil {
t.Fatal("pbs coord absent with snapshots present")
}
if h.PBS.Namespace == PBSRootNamespace {
t.Errorf("namespace=%q — this is the R-106 symptom: the snapshot's empty ns normalised to root "+
"while the box's backups are in demo-felhom", h.PBS.Namespace)
}
if h.PBS.Namespace != "demo-felhom" {
t.Errorf("namespace=%q want demo-felhom (storage.cfg's `namespace` on the pbs storage)", h.PBS.Namespace)
}
if h.PBS.NamespaceState != DRStateResolved {
t.Errorf("namespace_state=%q want %q (reason=%q)", h.PBS.NamespaceState, DRStateResolved, h.PBS.NamespaceReason)
}
if h.PBS.RepoID != "felhom-pbs" || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("coord drifted: repo=%q snapshot=%q", h.PBS.RepoID, h.PBS.LatestSnapshotID)
}
}
// TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown: a box with a pbs storage and NO namespace line is
// genuinely in the root namespace. That is an answer, not a gap — it must read resolved/"root", so the
// honest root case is never confused with "I could not tell".
func TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup", PBSNamespace: ""}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
if h.PBS.NamespaceState != DRStateResolved {
t.Errorf("namespace_state=%q — an unconfigured namespace IS the root namespace, not an unknown", h.PBS.NamespaceState)
}
if h.PBS.Namespace != PBSRootNamespace {
t.Errorf("namespace=%q want %q", h.PBS.Namespace, PBSRootNamespace)
}
}
// TestDRRecipe_BackupTargetUnknownWhenConfigUnavailable is the WRONG case: the agent could not consult
// its own backup config. The recipe must say so explicitly and emit NO storage_id key at all — an
// absent value must not be representable as a plausible-looking answer.
func TestDRRecipe_BackupTargetUnknownWhenConfigUnavailable(t *testing.T) {
h := BuildDRRecipeHostHalf(nil, capturedDemoFelhomTargets(), nil, ConfiguredBackupTarget{})
bt := h.BackupTarget
if bt == nil {
t.Fatal("backup_target must be PRESENT and say unknown, not vanish")
}
if bt.State != DRStateUnknown || bt.Reason != DRReasonNoBackupConfig {
t.Errorf("state=%q reason=%q want %q/%q", bt.State, bt.Reason, DRStateUnknown, DRReasonNoBackupConfig)
}
// Absence recorded as absence: no id, and no id KEY on the wire.
if bt.StorageID != "" {
t.Errorf("storage_id=%q — an unresolvable target must not be filled in", bt.StorageID)
}
b, err := json.Marshal(bt)
if err != nil {
t.Fatal(err)
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(b, &keys); err != nil {
t.Fatal(err)
}
for _, banned := range []string{"storage_id", "mount_path"} {
if _, ok := keys[banned]; ok {
t.Errorf("unknown backup_target must not carry a %q key; got %s", banned, b)
}
}
// And nothing in it may read as one of the real candidates.
for _, decoy := range []string{"felhom-backup", "local", "/var/lib/vz", "/mnt/hdd_1"} {
if strings.Contains(string(b), decoy) {
t.Errorf("unknown backup_target leaked a plausible value %q: %s", decoy, b)
}
}
}
// TestDRRecipe_BackupTargetUnknownWhenStorageMissing: the config names a storage this host does not
// have. That is unknown for a DIFFERENT reason — and the configured id IS still recorded, because
// "config says felhom-backup, no such storage here" sends an operator somewhere useful while silence
// does not.
func TestDRRecipe_BackupTargetUnknownWhenStorageMissing(t *testing.T) {
targets := []StorageTarget{{Name: "local", Type: StorageTypeLocal, Content: "backup", ConfigPath: "/var/lib/vz"}}
h := BuildDRRecipeHostHalf(nil, targets, nil, ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
bt := h.BackupTarget
if bt.State != DRStateUnknown || bt.Reason != DRReasonNoSuchStorage {
t.Errorf("state=%q reason=%q want %q/%q", bt.State, bt.Reason, DRStateUnknown, DRReasonNoSuchStorage)
}
if bt.StorageID != "felhom-backup" {
t.Errorf("storage_id=%q want the CONFIGURED id recorded even though it matched nothing", bt.StorageID)
}
// It must NOT silently fall back to the only content=backup storage present.
if bt.StorageID == "local" || bt.MountPath == "/var/lib/vz" {
t.Error("resolution fell back to the wrong storage instead of reporting unknown")
}
}
// TestDRRecipe_PBSNamespaceUnknownWithoutPBSStorage: snapshots exist but no pbs storage was observed, so
// there is no storage.cfg row to read a namespace from. The recipe must NOT default to root — that
// default is the entire R-106 defect.
func TestDRRecipe_PBSNamespaceUnknownWithoutPBSStorage(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "local", Type: StorageTypeLocal, Content: "backup"}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "local", Known: true})
if h.PBS == nil {
t.Fatal("pbs coord should still be emitted (the snapshot id is a real coordinate)")
}
if h.PBS.NamespaceState != DRStateUnknown || h.PBS.NamespaceReason != DRReasonNoPBSStorage {
t.Errorf("namespace_state=%q reason=%q want %q/%q",
h.PBS.NamespaceState, h.PBS.NamespaceReason, DRStateUnknown, DRReasonNoPBSStorage)
}
if h.PBS.Namespace != "" {
t.Errorf("namespace=%q — with no storage row to read, the field must be empty, never %q",
h.PBS.Namespace, PBSRootNamespace)
}
}
// TestDRRecipe_BackupTargetUsesConfigPathWhenDeviceGone is the DR-shaped case: the recipe is read while
// the target drive is absent, so MountPath has emptied out. ConfigPath is then the only thing that still
// says where the archives live (the R-116 lesson) — and the target is still RESOLVED, because which
// storage.cfg row to restore from is known regardless of whether its device is currently present.
func TestDRRecipe_BackupTargetUsesConfigPathWhenDeviceGone(t *testing.T) {
targets := []StorageTarget{{
Name: "felhom-backup", Type: StorageTypeLocalDir, Content: "backup",
MountPath: "", ConfigPath: "/mnt/hdd_1", // device gone: observer empties MountPath, keeps ConfigPath
}}
h := BuildDRRecipeHostHalf(nil, targets, nil, ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
bt := h.BackupTarget
if bt.State != DRStateResolved {
t.Errorf("state=%q — an absent device does not make the TARGET unknown", bt.State)
}
if bt.MountPath != "/mnt/hdd_1" {
t.Errorf("mount_path=%q want the configured path /mnt/hdd_1", bt.MountPath)
}
}
// fakePBSReporter is a PBSReporter returning fixed snapshots (the verify loop's seam).
type fakePBSReporter struct{ snaps []PBSSnapshot }
func (f fakePBSReporter) PBSSnapshots(context.Context) []PBSSnapshot { return f.snaps }
// TestCollectDRRecipe_ProductionPath runs the REAL generation path — Collector.Collect(), the method the
// daemon calls every cycle — rather than BuildDRRecipeHostHalf directly. It is here because both defects
// this file fixes were invisible to a direct-call test: the namespace one lived in what the observer put
// on StorageTarget, and the target one lived in whether anything wired the config seam at all. A seam
// that is correct and never wired is the failure mode this repo has hit four times.
func TestCollectDRRecipe_ProductionPath(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
obs := fakeObserver{targets: capturedDemoFelhomTargets()}
pbsRep := fakePBSReporter{snaps: capturedDemoFelhomSnapshots()}
c := NewCollector(px, fakeProber{status: "active"}, obs, nil, nil, pbsRep, "h", "0.118.0", quietLogger())
c.SetBackupTargetResolver(func() ConfiguredBackupTarget {
return ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true}
})
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.DRRecipe == nil {
t.Fatal("collect produced no dr_recipe")
}
if bt := r.DRRecipe.BackupTarget; bt == nil || bt.State != DRStateResolved || bt.StorageID != "felhom-backup" {
t.Errorf("backup_target through Collect = %+v, want resolved/felhom-backup", bt)
}
if p := r.DRRecipe.PBS; p == nil || p.Namespace != "demo-felhom" || p.NamespaceState != DRStateResolved {
t.Errorf("pbs namespace through Collect = %+v, want demo-felhom/resolved", p)
}
}
// TestCollectDRRecipe_UnwiredSeamReportsUnknown: a Collector built WITHOUT the resolver (every
// --selftest one-shot did exactly this before v0.118.0) must produce an explicit unknown. This is the
// test that would have caught shipping the seam without wiring it.
func TestCollectDRRecipe_UnwiredSeamReportsUnknown(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{targets: capturedDemoFelhomTargets()},
nil, nil, nil, "h", "0.118.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
bt := r.DRRecipe.BackupTarget
if bt == nil || bt.State != DRStateUnknown || bt.Reason != DRReasonNoBackupConfig {
t.Fatalf("unwired resolver must yield unknown/%s, got %+v", DRReasonNoBackupConfig, bt)
}
if bt.StorageID != "" {
t.Errorf("unwired resolver invented a target %q", bt.StorageID)
}
}
// assertNoSecretKeys walks decoded JSON and fails on any object key matching secretNameRe. Shared by // assertNoSecretKeys walks decoded JSON and fails on any object key matching secretNameRe. Shared by
// the agent boundary assertions. (durable_id/repo_id/latest_snapshot_id are identifiers/coordinates — // the agent boundary assertions. (durable_id/repo_id/latest_snapshot_id are identifiers/coordinates —
// none match the credential regex.) // none match the credential regex.)
+134
View File
@@ -0,0 +1,134 @@
package hub
import (
"net"
"net/netip"
"sort"
)
// Host addresses (v0.119.0) — "which addresses does this box actually hold?"
//
// The hub could not answer that at all: HostMetrics carried node/cpu/mem/disk/load/uptime/temp and
// no address of any kind, so the LAN IP of a managed host was invisible in every operator surface.
// Two sources looked like answers and are not: `lan_resolver.host_ip` is an OPTIONAL config value
// (absent unless that feature is configured), and DeriveHostIP(local_api.listen_addr) yields the
// R-50 island literal 169.254.253.1 — a link-local address that is the same on every box. Reporting
// either would have produced a confident wrong answer, which is worse than the blank it replaces.
//
// This reads the kernel's own view instead, and it issues NO block I/O (the CLAUDE.md health-check
// rule): net.Interfaces() is a netlink/procfs read, needs no privilege, and touches no filesystem.
// HostAddress is one routable address the host holds, tagged with the interface carrying it.
//
// Deliberately iface+cidr rather than a single `lan_ip`: a Proxmox host legitimately holds several
// (a management bridge, a tailnet, the WG tunnel), and picking one of them to call "the" LAN IP is a
// guess the agent is not entitled to make — on a box whose management bridge is not vmbr0 that guess
// is silently wrong. The agent reports what exists; the hub does the labelling.
type HostAddress struct {
Iface string `json:"iface"` // e.g. "vmbr0", "wg-felhom", "tailscale0"
CIDR string `json:"cidr"` // e.g. "192.168.0.162/24" — prefix length kept, it is operator-relevant
}
// ifaceAddrs is one enumerated interface: the ONLY facts the filter needs. Keeping the seam this
// narrow is what lets the filter be tested against real measured shapes without a network stack.
type ifaceAddrs struct {
Name string
Up bool
Loopback bool
CIDRs []string
}
// AddressEnumerator returns the host's interfaces. Injectable so the filter can be driven with the
// shapes measured on real hardware (see hostaddr_test.go) instead of whatever the test box happens
// to have.
type AddressEnumerator func() ([]ifaceAddrs, error)
// systemInterfaces is the production enumerator: the kernel's interface table.
func systemInterfaces() ([]ifaceAddrs, error) {
ifaces, err := net.Interfaces()
if err != nil {
return nil, err
}
out := make([]ifaceAddrs, 0, len(ifaces))
for _, i := range ifaces {
e := ifaceAddrs{
Name: i.Name,
Up: i.Flags&net.FlagUp != 0,
Loopback: i.Flags&net.FlagLoopback != 0,
}
// A per-interface error is not fatal: one unreadable interface must not cost the report
// every other address (serve-degraded, as everywhere else in the collector).
addrs, aerr := i.Addrs()
if aerr != nil {
out = append(out, e)
continue
}
for _, a := range addrs {
e.CIDRs = append(e.CIDRs, a.String())
}
out = append(out, e)
}
return out, nil
}
// filterHostAddresses keeps every GLOBAL UNICAST address on an up, non-loopback interface.
//
// IsGlobalUnicast() is the whole rule, and it was chosen by measuring both demo hosts rather than by
// listing interface names to exclude. It drops, in one predicate:
// - loopback (127.0.0.1, ::1)
// - IPv6 link-local (fe80::/10) — every bridge carries one, pure noise
// - IPv4 link-local (169.254.0.0/16) — which is exactly the R-50 island address on vmbr9, an
// identical constant on every box and therefore actively misleading if surfaced
//
// It needs NO veth/fwbr/tap denylist: on a Proxmox host that per-guest plumbing carries no IP at
// all, so it self-excludes by having nothing to report. Verified on demo-felhom and demo-hp —
// veth9201i0/i1, fwbr*, and the unused NICs all appear in `ip link` and in no `ip addr` output.
//
// What survives on a real box: vmbr0's LAN address, wg-felhom's tunnel address, and tailscale0's
// tailnet addresses. All three are true and useful; none is labelled here.
func filterHostAddresses(in []ifaceAddrs) []HostAddress {
out := []HostAddress{}
for _, i := range in {
if i.Loopback || !i.Up {
continue
}
for _, c := range i.CIDRs {
p, err := netip.ParsePrefix(c)
if err != nil {
continue // not a CIDR we understand — skip it, never fail the report
}
if !p.Addr().IsGlobalUnicast() {
continue
}
out = append(out, HostAddress{Iface: i.Name, CIDR: p.String()})
}
}
// Deterministic order so a report diff reflects a real change, not interface-table ordering.
sort.Slice(out, func(a, b int) bool {
if out[a].Iface != out[b].Iface {
return out[a].Iface < out[b].Iface
}
return out[a].CIDR < out[b].CIDR
})
return out
}
// collectAddresses is the collector's entry point. It returns a non-nil slice so the field always
// marshals as [] — an absent key and "this box has no routable address" must not look alike to the
// hub, and [] is the honest encoding of the latter.
func (c *Collector) collectAddresses() []HostAddress {
enum := c.addrEnum
if enum == nil {
// Default to the REAL enumerator, deliberately inverting the nil-reporter-means-off
// convention used by the optional stanzas above. Those gate on a config feature; this has
// no dependency and no feature flag, so a forgotten wiring call in main.go would produce a
// silently empty field — the inert-seam failure this repo has shipped four times.
enum = systemInterfaces
}
ifaces, err := enum()
if err != nil {
c.logger.Warn("host addresses: interface enumeration failed", "err", err)
return []HostAddress{}
}
return filterHostAddresses(ifaces)
}
+214
View File
@@ -0,0 +1,214 @@
package hub
import (
"errors"
"log/slog"
"strings"
"testing"
)
// The fixtures below are MEASURED, not invented: `ip -o addr show` on demo-felhom (N100) and
// demo-hp (HP t740) on 2026-07-31, transcribed verbatim including the interfaces that carry no
// address. That matters — the filter's claim that it needs no veth/fwbr denylist rests on those
// interfaces genuinely having nothing to report, and a hand-written fixture that omitted them would
// have proved the claim by assuming it.
// demoFelhomIfaces is demo-felhom's real interface table.
func demoFelhomIfaces() []ifaceAddrs {
return []ifaceAddrs{
{Name: "lo", Up: true, Loopback: true, CIDRs: []string{"127.0.0.1/8", "::1/128"}},
{Name: "enp1s0", Up: false}, // physical NIC, no address
{Name: "wlp2s0", Up: false}, // wifi, no address
{Name: "tailscale0", Up: true, CIDRs: []string{
"100.70.170.35/32", "fd7a:115c:a1e0::5236:aa24/128", "fe80::4197:26fc:ccba:b0d9/64"}},
{Name: "vmbr0", Up: true, CIDRs: []string{"192.168.0.162/24", "fe80::6a1d:efff:fe5d:a664/64"}},
{Name: "veth9201i0", Up: true}, // per-guest plumbing — no address
{Name: "veth9201i1", Up: true}, // per-guest plumbing — no address
{Name: "vmbr9", Up: true, CIDRs: []string{"169.254.253.1/30", "fe80::48d4:f6ff:fe05:2f98/64"}},
{Name: "wg-felhom", Up: true, CIDRs: []string{"10.77.0.2/32"}},
}
}
func hasAddr(got []HostAddress, iface, cidr string) bool {
for _, a := range got {
if a.Iface == iface && a.CIDR == cidr {
return true
}
}
return false
}
func flatten(got []HostAddress) string {
var b strings.Builder
for _, a := range got {
b.WriteString(a.Iface + "=" + a.CIDR + " ")
}
return b.String()
}
// The LAN address is the whole point of the feature — it must survive the filter.
// RED-PROOF 1: drop the `!p.Addr().IsGlobalUnicast()` continue → the vmbr9 + fe80 assertions below
// go red (the LAN one still passes, which is exactly why the negatives are asserted too).
func TestFilterHostAddresses_RealHost(t *testing.T) {
got := filterHostAddresses(demoFelhomIfaces())
// --- what MUST be there ---
if !hasAddr(got, "vmbr0", "192.168.0.162/24") {
t.Fatalf("the LAN address was filtered away — the feature reports nothing: %s", flatten(got))
}
if !hasAddr(got, "wg-felhom", "10.77.0.2/32") {
t.Errorf("the WireGuard address was filtered away: %s", flatten(got))
}
if !hasAddr(got, "tailscale0", "100.70.170.35/32") {
t.Errorf("the tailnet address was filtered away: %s", flatten(got))
}
// --- what MUST NOT be there, each for its own reason ---
for _, bad := range []struct{ iface, cidr, why string }{
{"lo", "127.0.0.1/8", "loopback is not an address of the host on any network"},
{"lo", "::1/128", "IPv6 loopback"},
{"vmbr9", "169.254.253.1/30", "the R-50 island literal — IDENTICAL on every box, so surfacing it is actively misleading"},
{"vmbr0", "fe80::6a1d:efff:fe5d:a664/64", "IPv6 link-local, one per bridge, pure noise"},
{"tailscale0", "fe80::4197:26fc:ccba:b0d9/64", "IPv6 link-local"},
} {
if hasAddr(got, bad.iface, bad.cidr) {
t.Errorf("%s %s must be filtered (%s); got: %s", bad.iface, bad.cidr, bad.why, flatten(got))
}
}
// The no-denylist claim: not one veth/physical interface contributed a row.
for _, a := range got {
if strings.HasPrefix(a.Iface, "veth") || a.Iface == "enp1s0" || a.Iface == "wlp2s0" {
t.Errorf("%s produced a row — the fixture says it has no address, so the filter invented one", a.Iface)
}
}
}
// demo-hp is different hardware (4 unused NICs, different ordering) and must filter identically —
// the rule is about address CLASS, not about one box's interface names.
func TestFilterHostAddresses_SecondHostFiltersIdentically(t *testing.T) {
got := filterHostAddresses([]ifaceAddrs{
{Name: "lo", Up: true, Loopback: true, CIDRs: []string{"127.0.0.1/8", "::1/128"}},
{Name: "enp2s0f0", Up: false}, {Name: "enp1s0f0", Up: false},
{Name: "enp1s0f1", Up: false}, {Name: "enp1s0f2", Up: false},
{Name: "enp1s0f3", Up: false}, {Name: "wlo1", Up: false},
{Name: "tailscale0", Up: true, CIDRs: []string{
"100.76.96.79/32", "fd7a:115c:a1e0::ce36:6051/128", "fe80::e06a:ce64:80e1:7821/64"}},
{Name: "vmbr0", Up: true, CIDRs: []string{"192.168.0.87/24", "fe80::7ed3:aff:fe77:d976/64"}},
{Name: "wg-felhom", Up: true, CIDRs: []string{"10.77.0.3/32"}},
{Name: "vmbr9", Up: true, CIDRs: []string{"169.254.253.1/30", "fe80::2484:92ff:fe7d:52a5/64"}},
{Name: "veth9201i0", Up: true}, {Name: "veth9201i1", Up: true},
})
if !hasAddr(got, "vmbr0", "192.168.0.87/24") {
t.Fatalf("demo-hp's LAN address was filtered away: %s", flatten(got))
}
if hasAddr(got, "vmbr9", "169.254.253.1/30") {
t.Errorf("demo-hp's island address leaked through: %s", flatten(got))
}
// The island address is byte-identical on both boxes — the strongest argument for excluding it.
if strings.Contains(flatten(got), "169.254.") {
t.Errorf("a link-local IPv4 survived: %s", flatten(got))
}
}
// A DOWN interface holding a stale address must not be reported as if the box were reachable there.
// RED-PROOF 2: drop `|| !i.Up` → this goes red.
func TestFilterHostAddresses_DownInterfaceExcluded(t *testing.T) {
got := filterHostAddresses([]ifaceAddrs{
{Name: "vmbr0", Up: true, CIDRs: []string{"192.168.0.162/24"}},
{Name: "vmbr1", Up: false, CIDRs: []string{"10.9.9.9/24"}},
})
if hasAddr(got, "vmbr1", "10.9.9.9/24") {
t.Errorf("a DOWN interface's address was reported: %s", flatten(got))
}
if len(got) != 1 {
t.Errorf("want exactly the one up interface, got: %s", flatten(got))
}
}
// Order must be deterministic, or every report diff shows phantom churn.
func TestFilterHostAddresses_DeterministicOrder(t *testing.T) {
a := filterHostAddresses(demoFelhomIfaces())
// Same facts, opposite enumeration order.
rev := demoFelhomIfaces()
for i, j := 0, len(rev)-1; i < j; i, j = i+1, j-1 {
rev[i], rev[j] = rev[j], rev[i]
}
b := filterHostAddresses(rev)
if flatten(a) != flatten(b) {
t.Errorf("interface-table order changed the report:\n a=%s\n b=%s", flatten(a), flatten(b))
}
}
// A host with nothing routable yields [] and never nil — an absent key and "no addresses" must not
// look alike on the wire.
func TestFilterHostAddresses_EmptyIsNonNil(t *testing.T) {
got := filterHostAddresses([]ifaceAddrs{{Name: "lo", Up: true, Loopback: true, CIDRs: []string{"127.0.0.1/8"}}})
if got == nil {
t.Fatal("filter returned nil — it would marshal as null, not []")
}
if len(got) != 0 {
t.Errorf("want no addresses, got %s", flatten(got))
}
}
// A malformed entry is skipped, never fatal — one bad address must not cost the report the others.
func TestFilterHostAddresses_MalformedSkipped(t *testing.T) {
got := filterHostAddresses([]ifaceAddrs{
{Name: "vmbr0", Up: true, CIDRs: []string{"not-an-address", "192.168.0.162/24"}},
})
if len(got) != 1 || !hasAddr(got, "vmbr0", "192.168.0.162/24") {
t.Errorf("a malformed sibling address broke the good one: %s", flatten(got))
}
}
// --- the WIRING half: the collector must actually call the filter ---
// The seam defaults to the REAL enumerator, so a forgotten wiring call cannot make this inert.
// RED-PROOF 3: replace the collectAddresses body with `return []HostAddress{}` → red.
func TestCollectAddresses_UsesTheInjectedEnumerator(t *testing.T) {
c := &Collector{logger: slog.Default()}
c.addrEnum = func() ([]ifaceAddrs, error) { return demoFelhomIfaces(), nil }
got := c.collectAddresses()
if !hasAddr(got, "vmbr0", "192.168.0.162/24") {
t.Fatalf("the collector did not run the filter over the enumerator's output: %s", flatten(got))
}
}
// An enumeration failure degrades to [] and a WARN — never a failed report.
func TestCollectAddresses_EnumerationErrorDegrades(t *testing.T) {
c := &Collector{logger: slog.Default()}
c.addrEnum = func() ([]ifaceAddrs, error) { return nil, errors.New("netlink is unhappy") }
got := c.collectAddresses()
if got == nil {
t.Fatal("an enumeration error produced nil, which marshals as null")
}
if len(got) != 0 {
t.Errorf("want [] on error, got %s", flatten(got))
}
}
// The production enumerator must return SOMETHING on the machine running the tests, and must not
// panic. This is the only test that touches the real network stack; it asserts the contract
// (non-nil, no error, loopback correctly flagged) rather than any specific address, because the
// test host's addresses are not ours to predict.
func TestSystemInterfaces_ProductionEnumeratorWorks(t *testing.T) {
ifaces, err := systemInterfaces()
if err != nil {
t.Fatalf("systemInterfaces: %v", err)
}
if len(ifaces) == 0 {
t.Fatal("no interfaces at all — even a container has lo")
}
var sawLoopback bool
for _, i := range ifaces {
if i.Loopback {
sawLoopback = true
}
}
if !sawLoopback {
t.Error("no interface reported the loopback flag — the flag mapping is wrong")
}
// And the filter must survive real input without panicking.
_ = filterHostAddresses(ifaces)
}
+35
View File
@@ -43,6 +43,12 @@ type HostReport struct {
// alert. Not a secret (the fp is public; the token is never reported). // alert. Not a secret (the fp is public; the token is never reported).
LeafFingerprint string `json:"leaf_fingerprint"` LeafFingerprint string `json:"leaf_fingerprint"`
// Addresses are the host's routable addresses, one entry per (interface, address) — the LAN
// bridge, the WG tunnel, a tailnet. Added v0.119.0 because the hub could not show a managed
// box's IP anywhere: nothing in this report carried one. Non-nil so it marshals as [];
// see hostaddr.go for why it is iface+cidr rather than a single lan_ip.
Addresses []HostAddress `json:"addresses"`
// DR recipe — the agent (storage/guest/PBS) half of the secret-free reconstruction recipe // DR recipe — the agent (storage/guest/PBS) half of the secret-free reconstruction recipe
// (SPIKE-dr-recipe-2026-06-16). Derived from the facts above; carries ONLY identifiers/intents/ // (SPIKE-dr-recipe-2026-06-16). Derived from the facts above; carries ONLY identifiers/intents/
// sizes/coordinates, never a secret. The hub assembles it with the controller's app half. // sizes/coordinates, never a secret. The hub assembles it with the controller's app half.
@@ -280,6 +286,30 @@ type StorageTarget struct {
MountPath string `json:"mount_path"` // host mountpoint (dir/usb); "" for network/lvm MountPath string `json:"mount_path"` // host mountpoint (dir/usb); "" for network/lvm
BackingDevice string `json:"backing_device"` // resolved block device (e.g. /dev/sdb1); "" for network BackingDevice string `json:"backing_device"` // resolved block device (e.g. /dev/sdb1); "" for network
// ConfigPath is the storage's CONFIGURED path from storage.cfg (proxmox.Storage.Path) — not a
// resolved mount. It is the only identity a dir storage keeps when its device is gone: MountPath
// and BackingDevice both empty out (observe.go's exactMount block) and DurableID degrades off the
// fs-UUID, so the configured path is what still says WHICH drive this row is about (R-116).
//
// `json:"-"` DELIBERATELY. This struct is a cross-repo contract duplicated in felhom.eu/hub and
// pinned by testdata/host-report.golden.json + contract_test.go's key-set comparison; a wire-visible
// field here would need a matching change in the other repo to stay non-drifting. Nothing off-box
// needs this value — its only consumer is the agent's own /disks construction, in-process.
ConfigPath string `json:"-"`
// PBSNamespace is the storage's CONFIGURED PBS namespace from storage.cfg (proxmox.Storage.Namespace)
// — "" for the root namespace, set for S4 per-customer tenancy. Present ONLY on pbs targets.
//
// It exists because storage.cfg is the ONE authority on which namespace this box's backups use:
// `vzdump --storage <pbs>` makes PVE read this exact field, and the agent's own verify client is
// built from it (`cmd/felhom-agent/main.go` → `pbs.Config{Namespace: s.Namespace}`). The DR recipe
// therefore resolves the namespace from HERE and not from a listed snapshot — a namespace-scoped
// PBS list does not echo `ns` per item, so the snapshot's own field is empty and normalising that
// empty to "root" is what made the recipe claim "root" on every per-customer box (R-106).
//
// `json:"-"` for the SAME reason as ConfigPath above: this struct is a cross-repo contract pinned by
// testdata/host-report.golden.json, and nothing off-box reads this value — its only consumer is the
// agent's own dr_recipe construction, in-process.
PBSNamespace string `json:"-"`
// ClassHint is a fast|slow HINT derived from the backing disk's rotational flag — a // ClassHint is a fast|slow HINT derived from the backing disk's rotational flag — a
// hint only; the authoritative class is hub-owned (locked decision). "" when not // hint only; the authoritative class is hub-owned (locked decision). "" when not
// derivable (network targets have no local rotational flag). // derivable (network targets have no local rotational flag).
@@ -316,6 +346,11 @@ type ThinPoolFill struct {
type SmartSummary struct { type SmartSummary struct {
Health string `json:"health"` Health string `json:"health"`
// ModelName is smartctl's own device model (v0.95.0), captured from the JSON already parsed, so
// the UI can show a human label ("TOSHIBA MQ04ABF100") instead of a raw UUID. omitempty +
// pointer: absent on an old agent or a device that reports no model.
ModelName *string `json:"model_name,omitempty"`
TemperatureC *int `json:"temperature_c"` TemperatureC *int `json:"temperature_c"`
PowerOnHours *int `json:"power_on_hours"` PowerOnHours *int `json:"power_on_hours"`
+3 -1
View File
@@ -32,10 +32,11 @@ func TestHostReport_FieldNamesAndEmptyCollections(t *testing.T) {
Cloudflared: Cloudflared{Status: "active"}, Cloudflared: Cloudflared{Status: "active"},
Capabilities: []capability.Status{}, Capabilities: []capability.Status{},
LeafFingerprint: "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245", LeafFingerprint: "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245",
Addresses: []HostAddress{},
} }
// dr_recipe is always set on the real path (Collect); set it here too so the "no null" invariant // dr_recipe is always set on the real path (Collect); set it here too so the "no null" invariant
// covers it (empty pbs is omitempty → omitted, never null). // covers it (empty pbs is omitempty → omitted, never null).
r.DRRecipe = BuildDRRecipeHostHalf(r.Guests, r.StorageTargets, r.PBSSnapshots) r.DRRecipe = BuildDRRecipeHostHalf(r.Guests, r.StorageTargets, r.PBSSnapshots, ConfiguredBackupTarget{})
b, err := json.Marshal(r) b, err := json.Marshal(r)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -51,6 +52,7 @@ func TestHostReport_FieldNamesAndEmptyCollections(t *testing.T) {
// empty collections must be [] not null // empty collections must be [] not null
`"storage_targets":[]`, `"backups":[]`, `"restore_tests":[]`, `"pbs_snapshots":[]`, `"audit_tail":[]`, `"storage_targets":[]`, `"backups":[]`, `"restore_tests":[]`, `"pbs_snapshots":[]`, `"audit_tail":[]`,
`"capabilities":[]`, `"capabilities":[]`,
`"addresses":[]`,
`"leaf_fingerprint":"60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245"`, `"leaf_fingerprint":"60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245"`,
} { } {
if !strings.Contains(got, field) { if !strings.Contains(got, field) {
+13 -3
View File
@@ -134,6 +134,9 @@
"audit_tail": [], "audit_tail": [],
"capabilities": [], "capabilities": [],
"leaf_fingerprint": "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245", "leaf_fingerprint": "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245",
"addresses": [
{ "iface": "vmbr0", "cidr": "192.168.0.162/24" }
],
"dr_recipe": { "dr_recipe": {
"recipe_version": 1, "recipe_version": 1,
"guests": [ "guests": [
@@ -141,7 +144,8 @@
], ],
"pbs": { "pbs": {
"repo_id": "felhom-pbs", "repo_id": "felhom-pbs",
"namespace": "root", "namespace": "felhom-spike",
"namespace_state": "resolved",
"latest_snapshot_id": "9001" "latest_snapshot_id": "9001"
}, },
"drives": [ "drives": [
@@ -154,7 +158,13 @@
], ],
"pve_storage": [ "pve_storage": [
{ "name": "local-lvm", "type": "lvmthin", "content": "rootdir,images" }, { "name": "local-lvm", "type": "lvmthin", "content": "rootdir,images" },
{ "name": "usb-backup", "type": "usb", "content": "backup" } { "name": "usb-backup", "type": "usb", "content": "backup" },
] { "name": "felhom-pbs", "type": "pbs", "content": "backup" }
],
"backup_target": {
"state": "resolved",
"storage_id": "usb-backup",
"mount_path": "/mnt/usb-backup"
}
} }
} }
+160
View File
@@ -0,0 +1,160 @@
package localapi
import (
"context"
"errors"
"io"
"log/slog"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-88 Part 2 — the agent gains a third state.
//
// `newestArchiveOn` promised in its own doc comment that "errors degrade to unknown, never to
// no-backup", while its `(time.Time, bool)` return made that impossible: an error and a genuine
// not-found both produced `(zero, false)`, so `/backup/due` answered a POSITIVE
// "no successful backup recorded yet" with a nil age. The controller read that as "never backed up"
// and fired its window-gate safety valve, quiescing customer app stacks outside the backup window.
//
// These tests assert the WIRE, because the wire is the contract another component reads.
// listerBackups is a fakeBackups that also implements BackupArchiveLister, with a controllable outcome.
type listerBackups struct {
fakeBackups
t time.Time
found bool
err error
}
func (l *listerBackups) NewestArchiveTime(context.Context, int) (time.Time, bool, error) {
return l.t, l.found, l.err
}
// dueWithLister builds a server whose single tier's service is the given lister, and returns the
// /backup/due response.
// NOTE: the server must receive `lb` ITSELF, not its embedded fakeBackups — the tier's Service is
// type-asserted to BackupArchiveLister, and the embedded value does not satisfy it. Passing the
// inner struct silently routes every case to archiveUnknown, which looks like a code bug and is not.
func dueWithLister(t *testing.T, lb *listerBackups, store *fakeStore) BackupDueResponse {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuests{},
Backups: lb,
Store: store,
Storage: fakeStorage{},
Tokens: staticTokens{"A": 8200, "B": 9300},
BackupCadence: 24 * time.Hour,
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatalf("NewServer: %v", err)
}
srv.now = func() time.Time { return testNow }
return dueOf(t, srv.Handler())
}
// ── SCENARIO A (agent half) — an unreadable storage is UNKNOWN, not "never" ──────────────────
//
// COMPANION RED-PROOF (observed): make newestArchiveOn return archiveAbsent on the error path (the
// pre-fix collapse) and this fails with
//
// "an unreadable storage must report age_state=unknown, got \"absent\" — that is a POSITIVE claim
// of 'never backed up' built out of two absences"
//
// Restored.
func TestAgeState_UnreadableStorageIsUnknown(t *testing.T) {
lb := &listerBackups{err: errors.New("proxmox: GET storage content: 500 connection refused")}
got := dueWithLister(t, lb, &fakeStore{}) // empty store = cold in-memory record
if got.AgeState != AgeStateUnknown {
t.Fatalf("an unreadable storage must report age_state=%q, got %q — that is a POSITIVE claim of "+
"'never backed up' built out of two absences", AgeStateUnknown, got.AgeState)
}
if !got.Due {
t.Fatal("FAIL-SAFE DIRECTION: unknown must still be DUE — an unreadable storage must never suppress a backup")
}
if got.AgeSecs != nil {
t.Fatalf("an unknown age must not invent a number; got %d", *got.AgeSecs)
}
}
// ── SCENARIO B (agent half) — a genuine first-ever backup is ABSENT ──────────────────────────
//
// B is what makes A safe: an implementation that reported everything as "unknown" would pass A and
// silently starve a brand-new box, because the controller only fires the first-backup valve on ABSENT.
//
// COMPANION RED-PROOF (observed): make newestArchiveOn return archiveUnknown when !found and this
// fails with
//
// "a genuine never-backed-up tier must report age_state=\"absent\", got \"unknown\" — the
// controller only licenses a first backup outside the window on ABSENT"
//
// Restored.
func TestAgeState_GenuinelyNeverIsAbsent(t *testing.T) {
lb := &listerBackups{found: false} // read SUCCEEDED, nothing there
got := dueWithLister(t, lb, &fakeStore{})
if got.AgeState != AgeStateAbsent {
t.Fatalf("a genuine never-backed-up tier must report age_state=%q, got %q — the controller only "+
"licenses a first backup outside the window on ABSENT", AgeStateAbsent, got.AgeState)
}
if !got.Due {
t.Fatal("a never-backed-up tier must be due")
}
}
// A real archive → known, with a real age.
func TestAgeState_FoundIsKnown(t *testing.T) {
lb := &listerBackups{t: testNow.Add(-2 * time.Hour), found: true}
got := dueWithLister(t, lb, &fakeStore{})
if got.AgeState != AgeStateKnown {
t.Fatalf("a readable archive must report age_state=%q, got %q", AgeStateKnown, got.AgeState)
}
if got.AgeSecs == nil || *got.AgeSecs < 7100 || *got.AgeSecs > 7300 {
t.Fatalf("expected ~7200s age, got %v", got.AgeSecs)
}
if got.Due {
t.Fatal("2h old against a 24h cadence is not due")
}
}
// An unparseable in-memory timestamp is UNKNOWN too — a backup DID happen, we just cannot date it.
// Reporting "absent" there would be the same false-positive claim in a different costume.
func TestAgeState_UnparseableTimestampIsUnknown(t *testing.T) {
st := &fakeStore{backups: []hub.Backup{{VMID: 8200, Success: true, StartedAt: "not-a-timestamp"}}}
lb := &listerBackups{err: errors.New("storage unreadable")}
got := dueWithLister(t, lb, st)
if got.AgeState != AgeStateUnknown {
t.Fatalf("an unparseable backup time means we cannot DATE a backup that exists — want %q, got %q",
AgeStateUnknown, got.AgeState)
}
if !got.Due {
t.Fatal("still due — fail safe toward taking a backup")
}
}
// ── SCENARIO D (agent half) — additive on the wire ───────────────────────────────────────────
//
// An OLD controller decodes into a struct without `age_state` and ignores it. What it MUST still see
// unchanged is every pre-existing field.
func TestAgeState_IsAdditive_PreExistingFieldsUnchanged(t *testing.T) {
lb := &listerBackups{t: testNow.Add(-48 * time.Hour), found: true}
got := dueWithLister(t, lb, &fakeStore{})
if !got.Due || got.Reason != "older than cadence" {
t.Fatalf("pre-existing due/reason semantics changed: due=%v reason=%q", got.Due, got.Reason)
}
if got.AgeSecs == nil {
t.Fatal("age_seconds must still be present for a known age")
}
// And the state rides alongside rather than replacing anything.
if got.AgeState != AgeStateKnown {
t.Fatalf("age_state should be %q, got %q", AgeStateKnown, got.AgeState)
}
}
+212
View File
@@ -0,0 +1,212 @@
package localapi
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"path/filepath"
"strings"
)
// BackupTargetWrapperPath is the pinned sudoers vector (configs/felhom-backup-target-apply). The
// agent cannot create a PVE storage or grant an ACL itself — Datastore.Allocate at /storage and
// Permissions.Modify are deliberately outside its role — so the privileged half runs here.
const BackupTargetWrapperPath = "/usr/local/sbin/felhom-backup-target-apply"
// backupTargetRequest is POST /backup/target: move the PRIMARY whole-guest backup tier onto the
// drive mounted at Where, creating the storage if needed.
type backupTargetRequest struct {
VMID int `json:"vmid"`
Where string `json:"where"` // the drive's OWN host mountpoint (F-1)
ID string `json:"id,omitempty"` // storage id; default backupTargetStorageID
}
// backupTargetStorageID is the conventional id, matching what E-1 created by hand on both demo boxes.
// Keeping the name identical is what makes this endpoint IDEMPOTENT on an already-migrated box: the
// wrapper accepts an existing entry with the same path and changes nothing.
const backupTargetStorageID = "felhom-backup"
// handleSetBackupTarget performs the whole move as one ordered operation: create the storage, grant
// the agent access, repoint the primary tier in agent.json, and hand back what the caller must do to
// make it take effect.
//
// THE ORDER IS THE DESIGN, and each step is a precondition for the next:
//
// create → grant → config
//
// Reversed, a config pointing at a storage that does not exist would make the tier DEFER (harmless
// but silent), and a config pointing at an ungranted storage would make every backup 403 on its
// first run — which is exactly what E-1 hit when the grant was forgotten (finding F-3). Creating and
// granting BEFORE the config means the worst interruption leaves an unused storage, never a broken
// tier.
//
// IT DOES NOT RESTART THE AGENT. That is deliberate and it is the E-1 lesson encoded: the backup
// tiers are built once at daemon start, so the move needs a restart to take effect — but restarting
// while a backup or restore-test is in flight cancels the wait and records a SPURIOUS tier failure
// for a backup that actually succeeded (E-1 did exactly this to a felhom-pbs run). A restart that
// this handler fires itself could never be re-checked against in-flight work by the caller, so the
// response reports `restart_required` and the caller performs it behind its own immediate
// in-flight check.
func (s *Server) handleSetBackupTarget(w http.ResponseWriter, r *http.Request, vmid int) {
if s.privileged == nil {
writeErr(w, http.StatusServiceUnavailable, "privileged runner not configured on this host")
return
}
var req backupTargetRequest
if !decodeBody(w, r, &req) {
return
}
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
return
}
where := strings.TrimSpace(req.Where)
if where == "" {
writeErr(w, http.StatusBadRequest, "where (the drive's own mountpoint) is required")
return
}
id := strings.TrimSpace(req.ID)
if id == "" {
id = backupTargetStorageID
}
// AGENT-SIDE VALIDATION FIRST, from the agent's own storage view — never the caller's claim.
// The wrapper re-checks everything as root (it is the security boundary), but refusing here gives
// the customer a reason instead of a shell error, and keeps a bad request from reaching sudo at all.
if err := s.validateBackupTargetMount(r.Context(), where); err != nil {
s.logger.Warn("local-api: backup-target move refused", "vmid", vmid, "where", where, "err", err)
writeErr(w, http.StatusBadRequest, err.Error())
return
}
if _, errOut, err := s.privileged.Run(r.Context(), BackupTargetWrapperPath, "create", id, where); err != nil {
s.logger.Error("local-api: backup-target create failed", "id", id, "where", where, "err", err, "stderr", string(errOut))
writeErr(w, http.StatusBadGateway, "could not create the backup storage: "+wrapperReason(errOut, err))
return
}
if _, errOut, err := s.privileged.Run(r.Context(), BackupTargetWrapperPath, "grant", id); err != nil {
// The storage exists but the agent cannot write to it. Say so precisely: this is the exact
// state that produced E-1's "403 permission denied at /storage/felhom-backup" on first backup.
s.logger.Error("local-api: backup-target grant failed", "id", id, "err", err, "stderr", string(errOut))
writeErr(w, http.StatusBadGateway, "storage created but the access grant failed — backups would 403: "+wrapperReason(errOut, err))
return
}
if err := s.setConfiguredBackupTarget(id); err != nil {
s.logger.Error("local-api: backup-target config write failed", "id", id, "err", err)
writeErr(w, http.StatusInternalServerError, "storage is ready but the config could not be updated: "+err.Error())
return
}
s.logger.Info("local-api: backup target moved — RESTART REQUIRED for it to take effect",
"vmid", vmid, "target", id, "where", where)
writeOK(w, map[string]any{
"vmid": vmid, "target": id, "where": where,
// The caller must restart the agent BEHIND ITS OWN in-flight check — see the doc comment.
"restart_required": true,
})
}
// validateBackupTargetMount refuses a mount that cannot be a real backup target, from the agent's own
// storage view + mount table. Mirrors the wrapper's laws so the customer gets a reason, not a shell error.
func (s *Server) validateBackupTargetMount(ctx context.Context, where string) error {
if s.storage == nil {
return fmt.Errorf("storage view unavailable")
}
// It must currently BE a mountpoint (F-1/F-2). Resolved from the mount table, which is the same
// source the wrapper's `mountpoint -q` consults.
mounts, err := s.hostReader().Mounts()
if err != nil {
return fmt.Errorf("could not read the mount table")
}
var dev string
for _, m := range mounts {
if m.MountPoint == where {
dev = m.Device
break
}
}
if dev == "" {
return fmt.Errorf("%s is not a mountpoint — the backup target must be the drive's own mountpoint", where)
}
// Never the system disk: a target there protects against corruption only, never drive loss.
for _, m := range mounts {
if m.MountPoint == "/" && m.Device == dev {
return fmt.Errorf("%s is on the system disk — a backup target there cannot survive a drive failure", where)
}
}
return nil
}
// setConfiguredBackupTarget rewrites backup.local_backup_target in agent.json.
//
// Read-modify-write over map[string]json.RawMessage so UNKNOWN KEYS ARE PRESERVED VERBATIM — the
// same discipline as pbsdr.seedEscrowStorageID, and the property that made E-1's hand edit safe to
// begin with. A typed round-trip would silently drop any key this build does not know about.
//
// Written IN PLACE (O_TRUNC), not tmp+rename: /etc/felhom-agent is root-owned while agent.json is
// agent-owned 0600, so the non-root agent cannot rename into that directory. A recovery copy is
// parked first, so a torn write is recoverable.
func (s *Server) setConfiguredBackupTarget(id string) error {
path := s.configPath
if path == "" {
return fmt.Errorf("no config path known to this agent (env-only config)")
}
raw, err := os.ReadFile(path)
if err != nil {
return err
}
var doc map[string]json.RawMessage
if err := json.Unmarshal(raw, &doc); err != nil {
return fmt.Errorf("parse %s: %w", path, err)
}
var bk map[string]json.RawMessage
if cur, ok := doc["backup"]; ok {
if err := json.Unmarshal(cur, &bk); err != nil {
return fmt.Errorf("parse backup section: %w", err)
}
} else {
bk = map[string]json.RawMessage{}
}
idJSON, _ := json.Marshal(id)
bk["local_backup_target"] = idJSON
bkJSON, err := json.Marshal(bk)
if err != nil {
return err
}
doc["backup"] = bkJSON
out, err := json.MarshalIndent(doc, "", " ")
if err != nil {
return err
}
st, err := os.Stat(path)
if err != nil {
return err
}
if s.stateDir != "" {
if err := os.MkdirAll(s.stateDir, 0o700); err == nil {
_ = os.WriteFile(filepath.Join(s.stateDir, "agent.json.pre-backup-target"), raw, 0o600)
}
}
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
if err != nil {
return err
}
if _, err := f.Write(out); err != nil {
f.Close()
return err
}
return f.Close()
}
// wrapperReason surfaces the wrapper's own REFUSED line when it produced one — it explains WHY in
// terms the customer can act on ("not a mountpoint", "already exists at …") — falling back to the
// exec error only when stderr said nothing useful.
func wrapperReason(errOut []byte, err error) string {
for _, line := range strings.Split(string(errOut), "\n") {
if strings.Contains(line, "REFUSED:") {
return strings.TrimSpace(line)
}
}
return err.Error()
}
@@ -0,0 +1,147 @@
package localapi
import (
"context"
"encoding/json"
"io"
"log/slog"
"net/http"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// backupTargetServer builds a server whose PRIMARY tier is `felhom-backup`, mounted at /mnt/nvme-1tb
// on its own non-system device — i.e. the exact live shape E-1 created on demo-hp and demo-felhom:
// the drive is simultaneously the enrolled user-data drive AND the whole-guest vzdump target.
func backupTargetServer(t *testing.T) http.Handler {
t.Helper()
sv := fakeStorage{targets: []hub.StorageTarget{
{
Name: "felhom-backup", Type: hub.StorageTypeLocalDir, State: hub.StorageStateAttached,
Reachable: true, MountPath: "/mnt/nvme-1tb", BackingDevice: "/dev/nvme0n1",
Content: "backup",
},
{
Name: "spare-drive", Type: hub.StorageTypeLocalDir, State: hub.StorageStateAttached,
Reachable: true, MountPath: "/mnt/spare", BackingDevice: "/dev/sdz1",
Content: "backup",
},
}}
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{}, Storage: sv,
Tokens: staticTokens{"A": 8200},
Disks: &fakeDiskOps{}, DiskGate: &fakeGate{}, HostReader: sysOnSDA(),
// Service is load-bearing: normalizeBackupTiers DROPS any tier with a nil Service and falls
// back to the legacy single tier with an empty TargetID — which silently made an earlier
// version of this test exercise nothing.
BackupTiers: []BackupTier{
{TargetID: "felhom-backup", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
{TargetID: "felhom-pbs", Cadence: 168 * time.Hour, Service: &fakeBackups{}},
},
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatalf("new server: %v", err)
}
srv.baseCtx = context.Background()
srv.now = func() time.Time { return testNow }
return srv.Handler()
}
// E-2c — ejecting the drive that holds the only local whole-guest backup must be refused.
//
// This is a REGRESSION GUARD on a live configuration, not a hypothetical. E-1 (2026-07-28) moved the
// vzdump target onto each demo box's secondary drive, and `RoleForStorage` types a local-dir on a
// non-system device as user-data — so the pre-existing role gate PASSES it and the customer could
// self-serve eject the drive holding their backups. It would have succeeded silently.
//
// The assertion is on the CONSEQUENCE (the request is refused) plus the remedy being named, because a
// refusal the customer cannot act on just moves the failure.
func TestEjectRefusedOnTheBackupTargetDrive(t *testing.T) {
h := backupTargetServer(t)
rr := do(t, h, http.MethodPost, "/disks/eject", "A", `{"vmid":8200,"where":"/mnt/nvme-1tb"}`)
if rr.Code == http.StatusOK {
t.Fatalf("eject of the backup-target drive SUCCEEDED (%d) — the box would silently lose its "+
"local drive-loss protection with nothing alarming", rr.Code)
}
body := rr.Body.String()
if !strings.Contains(body, "felhom-backup") {
t.Errorf("refusal must NAME the backup target so the customer knows which role blocks it; got: %s", body)
}
if !strings.Contains(strings.ToLower(body), "reassign") {
t.Errorf("refusal must name the REMEDY (reassign the target first), else it is a dead end; got: %s", body)
}
}
// Decommission strands the target just as thoroughly as eject — it migrates data off and retires the
// drive. Same gate, asserted separately because it is a different handler and a different caller.
func TestDecommissionRefusedOnTheBackupTargetDrive(t *testing.T) {
h := backupTargetServer(t)
rr := do(t, h, http.MethodPost, "/disks/decommission", "A", `{"vmid":8200,"where":"/mnt/nvme-1tb"}`)
if rr.Code == http.StatusOK {
t.Fatalf("decommission of the backup-target drive SUCCEEDED (%d)", rr.Code)
}
if !strings.Contains(rr.Body.String(), "felhom-backup") {
t.Errorf("refusal must name the backup target; got: %s", rr.Body.String())
}
}
// THE OVER-CORRECTION GUARD, and the reason this is a narrow gate instead of a role reclassification.
//
// The tempting fix — make RoleForStorage return RoleBackup for the target — would also refuse every
// OTHER user-data drive op on a box, and on the demo boxes it would refuse the customer's own data
// drive, because that drive IS the target. This pins that a non-target drive stays ejectable: the new
// gate must block exactly one drive, not harden the whole eject path.
//
// It asserts "not blocked BY THIS GATE" rather than "succeeds", because eject has other legitimate
// failure modes in a fake harness; what must never appear is this gate's message.
func TestEjectStillAllowedOnANonTargetDrive(t *testing.T) {
h := backupTargetServer(t)
rr := do(t, h, http.MethodPost, "/disks/eject", "A", `{"vmid":8200,"where":"/mnt/spare"}`)
if strings.Contains(rr.Body.String(), "whole-guest backup target") {
t.Fatalf("the backup-target gate blocked a NON-target drive (/mnt/spare) — over-correction: "+
"it must block exactly the target, not harden the whole eject path; got: %s", rr.Body.String())
}
}
// E-2 — GET /disks must FLAG the backup-target drive, because the controller cannot work it out.
//
// The controller's own StoragePath.BackupTarget is customer INTENT, and on a box migrated by hand
// (E-1, both demo boxes) nobody ever assigned it — intent is empty while the drive really is the
// target. Without this flag the absent-target alarm could not name the drive on exactly the boxes
// that have one, which is the only place it currently matters.
func TestDisksFlagsTheBackupTargetDrive(t *testing.T) {
h := backupTargetServer(t)
rr := do(t, h, http.MethodGet, "/disks", "A", "")
if rr.Code != http.StatusOK {
t.Fatalf("GET /disks = %d, body %s", rr.Code, rr.Body.String())
}
body := rr.Body.String()
// The target must be flagged and the non-target must not be — asserted as a pair, since a
// blanket true would satisfy a naive "is it flagged?" check.
var got struct {
Data struct {
Disks []struct {
Name string `json:"name"`
BackupTarget bool `json:"backup_target"`
} `json:"disks"`
} `json:"data"`
}
if err := json.Unmarshal([]byte(body), &got); err != nil {
t.Fatalf("decode: %v (body %s)", err, body)
}
seen := map[string]bool{}
for _, d := range got.Data.Disks {
seen[d.Name] = d.BackupTarget
}
if !seen["felhom-backup"] {
t.Errorf("felhom-backup is the primary tier's storage but backup_target is false; body: %s", body)
}
if seen["spare-drive"] {
t.Errorf("spare-drive is NOT the target but was flagged — a blanket true is not a signal; body: %s", body)
}
}
@@ -0,0 +1,174 @@
package localapi
import (
"context"
"encoding/json"
"io"
"log/slog"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// wrapperRunner captures the wrapper vector without ever running sudo. The wrapper IS the security
// boundary, so tests substitute it rather than bypassing it — what is asserted here is the ORDER and
// the ARGUMENTS the agent sends, which is the agent's half of the contract.
type wrapperRunner struct {
calls [][]string
failOn string // verb to fail, "" = all succeed
}
func (r *wrapperRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
if len(args) > 0 && args[0] == r.failOn {
return nil, []byte("felhom-backup-target-apply: REFUSED: synthetic " + r.failOn + " failure\n"),
io.ErrUnexpectedEOF
}
return nil, nil, nil
}
func (r *wrapperRunner) verbs() []string {
var out []string
for _, c := range r.calls {
if len(c) > 1 {
out = append(out, c[1])
}
}
return out
}
// moveServer builds a server with /mnt/data mounted on its own device and / on another, plus a
// throwaway agent.json the move can rewrite.
func moveServer(t *testing.T, run *wrapperRunner) (http.Handler, string) {
t.Helper()
dir := t.TempDir()
cfgPath := filepath.Join(dir, "agent.json")
// An UNKNOWN key is deliberately present: the rewrite must preserve it verbatim.
seed := `{"backup":{"local_backup_target":"local","local_backup_retention":3},"some_future_key":{"keep":"me"}}`
if err := os.WriteFile(cfgPath, []byte(seed), 0o600); err != nil {
t.Fatalf("seed config: %v", err)
}
hr := fakeHostReader{mounts: []storage.Mount{
{Device: "/dev/sda1", MountPoint: "/"},
{Device: "/dev/sdb1", MountPoint: "/mnt/data"},
}}
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{}, Storage: fakeStorage{},
Tokens: staticTokens{"A": 8200}, HostReader: hr,
Privileged: run, ConfigPath: cfgPath, StateDir: dir,
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatalf("new server: %v", err)
}
srv.baseCtx = context.Background()
return srv.Handler(), cfgPath
}
// THE ORDER IS THE CONTRACT: create → grant → config. Reversed, a config pointing at an ungranted
// storage makes every backup 403 on first run, which is precisely what E-1 hit (finding F-3).
func TestBackupTargetMoveOrdersCreateThenGrantThenConfig(t *testing.T) {
run := &wrapperRunner{}
h, cfgPath := moveServer(t, run)
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/mnt/data"}`)
if rr.Code != http.StatusOK {
t.Fatalf("move = %d, body %s", rr.Code, rr.Body.String())
}
got := strings.Join(run.verbs(), ",")
if got != "create,grant" {
t.Fatalf("wrapper verbs = %q, want create,grant (in that order)", got)
}
// The config must have been written only AFTER both wrapper calls succeeded.
raw, _ := os.ReadFile(cfgPath)
var doc map[string]json.RawMessage
if err := json.Unmarshal(raw, &doc); err != nil {
t.Fatalf("config unreadable after move: %v", err)
}
var bk map[string]any
_ = json.Unmarshal(doc["backup"], &bk)
if bk["local_backup_target"] != "felhom-backup" {
t.Errorf("local_backup_target = %v, want felhom-backup", bk["local_backup_target"])
}
// Unknown keys preserved verbatim — the property that made E-1's hand edit safe.
if _, ok := doc["some_future_key"]; !ok {
t.Error("the rewrite DROPPED an unknown top-level key — a typed round-trip would do this " +
"and silently discard config this build does not know about")
}
// Sibling keys inside `backup` survive too.
if bk["local_backup_retention"] == nil {
t.Error("the rewrite dropped local_backup_retention from the backup section")
}
}
// A FAILED GRANT MUST NOT LEAVE THE CONFIG POINTING AT THE NEW STORAGE. That state is exactly E-1's
// 403-on-every-backup: the tier looks configured and cannot write.
func TestBackupTargetMoveDoesNotRepointWhenTheGrantFails(t *testing.T) {
run := &wrapperRunner{failOn: "grant"}
h, cfgPath := moveServer(t, run)
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/mnt/data"}`)
if rr.Code == http.StatusOK {
t.Fatalf("move SUCCEEDED despite a failed grant (%d)", rr.Code)
}
if !strings.Contains(rr.Body.String(), "403") {
t.Errorf("the error should name the consequence (backups would 403); got %s", rr.Body.String())
}
raw, _ := os.ReadFile(cfgPath)
if strings.Contains(string(raw), "felhom-backup") {
t.Fatal("the config was repointed at a storage the agent cannot write to — every backup " +
"would 403 while the tier reported as configured")
}
}
// It must NOT restart the agent itself. Restarting with a backup in flight cancels the wait and
// records a spurious tier failure for a backup that actually succeeded — E-1 did exactly that to a
// felhom-pbs run. Only the caller can re-check in-flight work immediately before restarting.
func TestBackupTargetMoveReportsRestartRequiredRatherThanRestarting(t *testing.T) {
run := &wrapperRunner{}
h, _ := moveServer(t, run)
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/mnt/data"}`)
if !strings.Contains(rr.Body.String(), `"restart_required":true`) {
t.Errorf("response must tell the caller a restart is required; got %s", rr.Body.String())
}
for _, c := range run.calls {
joined := strings.Join(c, " ")
if strings.Contains(joined, "systemctl") || strings.Contains(joined, "restart") {
t.Fatalf("the handler restarted the agent itself: %q — the caller must do it behind its "+
"own in-flight check", joined)
}
}
}
// A path that is not a mountpoint is refused BEFORE sudo is reached (F-1): a subdirectory target
// reports disconnected forever, and an unmounted path silently retargets onto the system drive.
func TestBackupTargetMoveRefusesANonMountpoint(t *testing.T) {
run := &wrapperRunner{}
h, _ := moveServer(t, run)
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/mnt/data/sub"}`)
if rr.Code == http.StatusOK {
t.Fatal("a non-mountpoint was accepted as the backup target")
}
if len(run.calls) != 0 {
t.Errorf("a refused request still reached the privileged wrapper: %v", run.calls)
}
}
// The system disk is refused: a target there protects against corruption only, never drive loss —
// which is the entire point of the move.
func TestBackupTargetMoveRefusesTheSystemDisk(t *testing.T) {
run := &wrapperRunner{}
h, _ := moveServer(t, run)
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/"}`)
if rr.Code == http.StatusOK {
t.Fatal("the system disk was accepted as the backup target")
}
if len(run.calls) != 0 {
t.Errorf("a refused request still reached the privileged wrapper: %v", run.calls)
}
}
+59
View File
@@ -0,0 +1,59 @@
package localapi
import "time"
// normalizeBackupTiers resolves the tier list the Server serves.
//
// Contract (R-82), and the reason this is a named function rather than inline setup: the UNTARGETED
// local-API endpoints must keep behaving exactly as they did before multi-tier existed, forever.
// That property lives here.
//
// - tiers == nil → synthesize ONE tier from the legacy (Backups, BackupCadence) pair and mark it
// primary. This is the pre-R-82 shape; every existing caller and test hits this path.
// - tiers supplied → keep order but hoist the primary to the front; if none is marked primary,
// the FIRST becomes primary (a tier list with no primary would leave untargeted requests with
// nothing to act on, which would silently stop backups).
// - tiers with a nil Service are dropped: a tier with no runner cannot back anything up, and
// advertising it would be an "applied and empty" tier — the exact fault R-82 exists to fix.
func normalizeBackupTiers(tiers []BackupTier, legacy BackupService, cadence time.Duration) []BackupTier {
usable := make([]BackupTier, 0, len(tiers))
for _, t := range tiers {
if t.Service == nil || t.TargetID == "" {
continue
}
if t.Cadence <= 0 {
t.Cadence = cadence
}
if t.WaitTimeout <= 0 {
t.WaitTimeout = 2 * time.Hour
}
usable = append(usable, t)
}
if len(usable) == 0 {
if legacy == nil {
return nil
}
return []BackupTier{{TargetID: "", Cadence: cadence, WaitTimeout: 2 * time.Hour, Primary: true, Service: legacy}}
}
primary := -1
for i, t := range usable {
if t.Primary {
primary = i
break
}
}
if primary < 0 {
primary = 0
}
out := make([]BackupTier, 0, len(usable))
usable[primary].Primary = true
out = append(out, usable[primary])
for i, t := range usable {
if i == primary {
continue
}
t.Primary = false
out = append(out, t)
}
return out
}
+671
View File
@@ -0,0 +1,671 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"net/http"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-82 Slice A — per-target cadence, due and runner.
//
// The agent and the controller deploy INDEPENDENTLY. A new agent will serve old controllers for as
// long as it takes the fleet to catch up, so the untargeted contract is frozen, not merely
// "probably fine". These tests pin that freeze; the multi-tier behaviour is additive on top.
// tieredServer builds a two-tier server: primary "local" (24h) + "felhom-pbs" (7d), each with its
// own runner, exactly as main.go wires it.
func tieredServer(t *testing.T, st *fakeStore, localSvc, pbsSvc *fakeBackups) *Server {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuests{},
Backups: localSvc,
Store: st,
// Both tier targets must be PRESENT in the storage view: since v0.102.0 a tier whose target
// storage is absent DEFERS. A real box has both; a fake with no targets would silently
// defer every tier and make these assertions vacuous.
Storage: fakeStorage{targets: []hub.StorageTarget{
{Name: "local", Type: "local"},
{Name: "felhom-pbs", Type: "pbs"},
}},
Tokens: staticTokens{"A": 8200, "B": 9300},
BackupTiers: []BackupTier{
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: localSvc},
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: pbsSvc},
},
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatalf("new server: %v", err)
}
srv.baseCtx = context.Background()
srv.now = func() time.Time { return testNow }
return srv
}
// seen returns the vmids this fake runner was invoked for (mutex-guarded — the backup runs on a
// goroutine).
func (f *fakeBackups) seen() []int {
f.mu.Lock()
defer f.mu.Unlock()
return append([]int(nil), f.vmids...)
}
// waitFor polls cond for up to 2s. POST /backup is fire-and-forget, so the assertion has to wait
// for the goroutine rather than assume it has run.
func waitFor(t *testing.T, cond func() bool) {
t.Helper()
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
if cond() {
return
}
time.Sleep(5 * time.Millisecond)
}
t.Fatalf("condition not met within 2s")
}
func backupAt(target string, vmid int, ago time.Duration, ok bool) hub.Backup {
return hub.Backup{
TargetID: target,
VMID: vmid,
Success: ok,
StartedAt: testNow.Add(-ago).Format(time.RFC3339),
}
}
// ── RED-PROOF 1 — old controller ↔ new agent ────────────────────────────────────────────────
//
// An old controller sends `GET /backup/due` with no query string and parses the pre-R-82 response.
// The response must be BYTE-IDENTICAL — not merely semantically similar. A stray `"target":"local"`
// key is harmless to a tolerant JSON decoder and fatal to a strict one, and we do not get to choose
// which the deployed fleet has.
//
// COMPANION RED-PROOF (observed): drop the `omitempty` from BackupDueResponse.Target and have
// tierFromRequest echo the primary's id for an untargeted request → this test fails with the
// observed body carrying `"target":"local"`. Restored.
func TestBackupDue_Untargeted_ResponseBytesUnchanged(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("local", 8200, 2*time.Hour, true))
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
rr := do(t, h, "GET", "/backup/due", "A", "")
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rr.Code)
}
var got map[string]any
if err := json.Unmarshal(rr.Body.Bytes(), &got); err != nil {
t.Fatalf("unmarshal: %v (body %s)", err, rr.Body.String())
}
data, _ := got["data"].(map[string]any)
if data == nil {
t.Fatalf("no data object in %s", rr.Body.String())
}
if _, present := data["target"]; present {
t.Fatalf("UNTARGETED response MUST NOT carry a target key — an old controller sees a changed contract; body: %s", rr.Body.String())
}
if data["due"] != false {
t.Fatalf("2h-old local backup under a 24h cadence must not be due; body: %s", rr.Body.String())
}
if data["reason"] != "within cadence window" {
t.Fatalf("reason string changed: %v", data["reason"])
}
}
// The untargeted verdict must come from the PRIMARY tier's cadence, not from whichever tier
// happens to be freshest. With a stale local and a fresh PBS backup, untargeted must say DUE.
func TestBackupDue_Untargeted_UsesPrimaryTierOnly(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("local", 8200, 30*time.Hour, true)) // stale for 24h cadence
st.RecordBackup(backupAt("felhom-pbs", 8200, 1*time.Hour, true)) // fresh, different tier
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var resp struct {
Data BackupDueResponse `json:"data"`
}
rr := do(t, h, "GET", "/backup/due", "A", "")
if err := json.Unmarshal(rr.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
if !resp.Data.Due {
t.Fatalf("a fresh backup on ANOTHER tier must not satisfy the primary's cadence; got %+v", resp.Data)
}
}
// ── Per-tier due-ness ────────────────────────────────────────────────────────────────────────
// THE POINT OF THE WHOLE SLICE: a fresh daily local backup must NOT satisfy the weekly PBS tier.
// Without the per-target filter in latestSuccessfulBackupForTarget the DR tier would never run —
// which is exactly today's "applied and empty" state, re-created in code.
func TestBackupDue_PerTier_LocalFreshDoesNotSatisfyPBS(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("local", 8200, 2*time.Hour, true)) // fresh daily
st.RecordBackup(backupAt("felhom-pbs", 8200, 8*24*time.Hour, true)) // 8d — past the 7d weekly
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var local, pbs struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=local", "A", "").Body.Bytes(), &local); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if local.Data.Due {
t.Fatalf("local tier: 2h old under 24h cadence must NOT be due; got %+v", local.Data)
}
if !pbs.Data.Due {
t.Fatalf("PBS tier: 8d old under a 7d cadence MUST be due; got %+v", pbs.Data)
}
if pbs.Data.Target != "felhom-pbs" || local.Data.Target != "local" {
t.Fatalf("a targeted response must echo its tier; got local=%q pbs=%q", local.Data.Target, pbs.Data.Target)
}
}
// A 6-day-old PBS snapshot is INSIDE the weekly window — it must not be due. The mirror of the
// hub-side threshold test in Slice C, asserted here at the source of truth.
func TestBackupDue_PerTier_PBSWithinWeeklyWindow(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("felhom-pbs", 8200, 6*24*time.Hour, true))
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var pbs struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if pbs.Data.Due {
t.Fatalf("6d old under a 7d cadence must NOT be due; got %+v", pbs.Data)
}
}
// A failed backup must not satisfy any tier's cadence (pre-existing rule, re-asserted per-tier).
func TestBackupDue_PerTier_FailedBackupDoesNotCount(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("felhom-pbs", 8200, time.Hour, false))
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var pbs struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if !pbs.Data.Due {
t.Fatalf("a FAILED backup must not satisfy the cadence; got %+v", pbs.Data)
}
}
// The fail-safe-toward-due rule survives per-tier: an unparseable timestamp yields DUE.
// A spurious backup is cheap; a skipped one is not.
func TestBackupDue_PerTier_UnparseableTimestampIsDue(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(hub.Backup{TargetID: "felhom-pbs", VMID: 8200, Success: true, StartedAt: "not-a-time"})
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var pbs struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if !pbs.Data.Due {
t.Fatalf("unparseable timestamp must fail SAFE toward due; got %+v", pbs.Data)
}
}
// An unknown target is a 400 — never a silent fallback to the primary. A controller asking about a
// tier this agent does not serve must find out, not be handed a different tier's freshness and act
// on it.
func TestBackupDue_UnknownTarget_IsAnErrorNotAFallback(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("local", 8200, 2*time.Hour, true))
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
rr := do(t, h, "GET", "/backup/due?target=does-not-exist", "A", "")
if rr.Code != http.StatusBadRequest {
t.Fatalf("unknown target must be 400 (got %d, body %s)", rr.Code, rr.Body.String())
}
}
// ── Tier advertisement ───────────────────────────────────────────────────────────────────────
// GET /backup/tiers is the controller's capability probe. Primary must be first and flagged, so a
// controller can tell which tier the untargeted endpoints act on.
func TestBackupTiers_AdvertisesPrimaryFirst(t *testing.T) {
h := tieredServer(t, &fakeStore{}, &fakeBackups{}, &fakeBackups{}).Handler()
var resp struct {
Data BackupTiersResponse `json:"data"`
}
rr := do(t, h, "GET", "/backup/tiers", "A", "")
if rr.Code != http.StatusOK {
t.Fatalf("status = %d", rr.Code)
}
if err := json.Unmarshal(rr.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
if len(resp.Data.Tiers) != 2 {
t.Fatalf("want 2 tiers, got %+v", resp.Data.Tiers)
}
if !resp.Data.Tiers[0].Primary || resp.Data.Tiers[0].Target != "local" {
t.Fatalf("primary must be first and flagged; got %+v", resp.Data.Tiers)
}
if resp.Data.Tiers[1].Target != "felhom-pbs" || resp.Data.Tiers[1].CadenceSeconds != int64((7*24*time.Hour).Seconds()) {
t.Fatalf("PBS tier mis-advertised: %+v", resp.Data.Tiers[1])
}
}
// ── POST /backup routing + per-tier single-flight ────────────────────────────────────────────
// A targeted POST must run THAT tier's runner. Routing both tiers to one runner would silently
// write every "PBS" backup to local — a DR tier that reports success and stores nothing.
func TestBackupPost_RoutesToTheTargetsOwnRunner(t *testing.T) {
local, pbs := &fakeBackups{}, &fakeBackups{}
srv := tieredServer(t, &fakeStore{}, local, pbs)
h := srv.Handler()
if rr := do(t, h, "POST", "/backup?target=felhom-pbs", "A", ""); rr.Code != http.StatusAccepted {
t.Fatalf("status = %d, body %s", rr.Code, rr.Body.String())
}
waitFor(t, func() bool { return len(pbs.seen()) == 1 })
if got := len(local.seen()); got != 0 {
t.Fatalf("the LOCAL runner must not have run for a PBS-targeted request (ran %d times)", got)
}
}
// Untargeted POST routes to the primary — the old controller's path.
func TestBackupPost_UntargetedRoutesToPrimary(t *testing.T) {
local, pbs := &fakeBackups{}, &fakeBackups{}
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
if rr := do(t, h, "POST", "/backup", "A", ""); rr.Code != http.StatusAccepted {
t.Fatalf("status = %d", rr.Code)
}
waitFor(t, func() bool { return len(local.seen()) == 1 })
if got := len(pbs.seen()); got != 0 {
t.Fatalf("untargeted POST must not touch a non-primary tier (ran %d times)", got)
}
}
// ONE BACKUP AT A TIME PER GUEST (operator ruling 2026-07-26). A second tier's POST while another
// tier is still in flight must be REFUSED — vzdump holds the guest lock, so it could not succeed
// anyway, and attempting it records a spurious failure that leaves the tier permanently due.
//
// Crucially it must NOT be handed the busy tier's job id: that is exactly how a caller comes to
// believe its own backup ran.
func TestBackupPost_SecondTierRefusedWhileAnotherInFlight(t *testing.T) {
localGate := make(chan struct{})
local := &fakeBackups{gate: localGate}
pbs := &fakeBackups{}
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
var first BackupResponse
rr1 := do(t, h, "POST", "/backup", "A", "") // local; blocks on the gate
if err := json.Unmarshal(rr1.Body.Bytes(), &struct {
Data *BackupResponse `json:"data"`
}{Data: &first}); err != nil {
t.Fatal(err)
}
waitFor(t, func() bool { return len(local.seen()) == 1 })
rr2 := do(t, h, "POST", "/backup?target=felhom-pbs", "A", "")
if rr2.Code != http.StatusConflict {
t.Fatalf("a second tier must be REFUSED while another is in flight; got %d body %s", rr2.Code, rr2.Body.String())
}
// The STRUCTURAL requirement: the refusal must not return a job the caller could mistake for
// its own. It is a 409 with ok=false and NO data object, so nothing is parseable as "my job".
// (Naming the busy job in the human-readable message is deliberate and useful for diagnosis —
// what must never happen is handing it back as BackupResponse.JobID on a 202.)
var envelope struct {
OK bool `json:"ok"`
Data *BackupResponse `json:"data"`
Error string `json:"error"`
}
if err := json.Unmarshal(rr2.Body.Bytes(), &envelope); err != nil {
t.Fatal(err)
}
if envelope.OK {
t.Fatalf("a refusal must not be ok=true; body %s", rr2.Body.String())
}
if envelope.Data != nil && envelope.Data.JobID != "" {
t.Fatalf("the refusal must NOT hand back a job id as the caller's own (got %q); body %s",
envelope.Data.JobID, rr2.Body.String())
}
if !strings.Contains(rr2.Body.String(), "local") {
t.Fatalf("the refusal must NAME the busy tier so the caller can diagnose; body %s", rr2.Body.String())
}
if got := len(pbs.seen()); got != 0 {
t.Fatalf("the refused tier must NOT have started a backup (ran %d times)", got)
}
close(localGate)
}
// Once the busy tier finishes, the other tier may start — and gets its OWN tier-scoped job id.
func TestBackupPost_SecondTierAllowedAfterFirstFinishes(t *testing.T) {
local, pbs := &fakeBackups{}, &fakeBackups{}
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
var first, second BackupResponse
if err := json.Unmarshal(do(t, h, "POST", "/backup", "A", "").Body.Bytes(), &struct {
Data *BackupResponse `json:"data"`
}{Data: &first}); err != nil {
t.Fatal(err)
}
waitFor(t, func() bool { return len(local.seen()) == 1 })
// Wait for the local job to leave the in-flight phases.
waitFor(t, func() bool {
rr := do(t, h, "GET", "/backup/status", "A", "")
return strings.Contains(rr.Body.String(), `"phase":"done"`)
})
rr2 := do(t, h, "POST", "/backup?target=felhom-pbs", "A", "")
if rr2.Code != http.StatusAccepted {
t.Fatalf("after the first tier finished the second must be allowed; got %d body %s", rr2.Code, rr2.Body.String())
}
if err := json.Unmarshal(rr2.Body.Bytes(), &struct {
Data *BackupResponse `json:"data"`
}{Data: &second}); err != nil {
t.Fatal(err)
}
if second.JobID == first.JobID {
t.Fatalf("job ids must stay tier-scoped: %q vs %q", first.JobID, second.JobID)
}
waitFor(t, func() bool { return len(pbs.seen()) == 1 })
}
// `snapshotted` still counts as in flight — the vzdump is uploading and still holds the guest lock.
// Checking only `running` (the pre-R-82 code) left a window where a second POST started a real
// second vzdump.
func TestBackupPost_SnapshottedCountsAsInFlight(t *testing.T) {
gate := make(chan struct{})
local := &fakeBackups{gate: gate, fireSnapshot: true} // fires onSnapshot, then blocks
pbs := &fakeBackups{}
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
do(t, h, "POST", "/backup", "A", "")
waitFor(t, func() bool {
rr := do(t, h, "GET", "/backup/status", "A", "")
return strings.Contains(rr.Body.String(), `"phase":"snapshotted"`)
})
rr := do(t, h, "POST", "/backup?target=felhom-pbs", "A", "")
if rr.Code != http.StatusConflict {
t.Fatalf("a SNAPSHOTTED backup still holds the guest — a second tier must be refused; got %d body %s", rr.Code, rr.Body.String())
}
close(gate)
}
// Same tier, still single-flight: a second POST to a running tier returns the SAME job.
func TestBackupPost_SameTierStillSingleFlight(t *testing.T) {
gate := make(chan struct{})
local := &fakeBackups{gate: gate}
h := tieredServer(t, &fakeStore{}, local, &fakeBackups{}).Handler()
var a, b BackupResponse
if err := json.Unmarshal(do(t, h, "POST", "/backup", "A", "").Body.Bytes(), &struct {
Data *BackupResponse `json:"data"`
}{Data: &a}); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(do(t, h, "POST", "/backup", "A", "").Body.Bytes(), &struct {
Data *BackupResponse `json:"data"`
}{Data: &b}); err != nil {
t.Fatal(err)
}
if a.JobID != b.JobID {
t.Fatalf("same tier must single-flight: %q vs %q", a.JobID, b.JobID)
}
close(gate)
}
// ── normalizeBackupTiers — the compatibility core ────────────────────────────────────────────
func TestNormalizeBackupTiers(t *testing.T) {
svc := &fakeBackups{}
t.Run("nil tiers synthesize the legacy single tier", func(t *testing.T) {
got := normalizeBackupTiers(nil, svc, 24*time.Hour)
if len(got) != 1 || !got[0].Primary || got[0].Cadence != 24*time.Hour {
t.Fatalf("legacy synthesis broken: %+v", got)
}
})
t.Run("primary is hoisted to the front", func(t *testing.T) {
got := normalizeBackupTiers([]BackupTier{
{TargetID: "felhom-pbs", Cadence: time.Hour, Service: svc},
{TargetID: "local", Cadence: time.Hour, Primary: true, Service: svc},
}, svc, 24*time.Hour)
if len(got) != 2 || got[0].TargetID != "local" || !got[0].Primary || got[1].Primary {
t.Fatalf("primary not hoisted / uniqueness broken: %+v", got)
}
})
t.Run("no primary marked → first becomes primary", func(t *testing.T) {
got := normalizeBackupTiers([]BackupTier{
{TargetID: "a", Cadence: time.Hour, Service: svc},
{TargetID: "b", Cadence: time.Hour, Service: svc},
}, svc, 24*time.Hour)
if len(got) != 2 || got[0].TargetID != "a" || !got[0].Primary {
t.Fatalf("want first-as-primary, got %+v", got)
}
})
t.Run("a tier with no runner is DROPPED, not advertised", func(t *testing.T) {
got := normalizeBackupTiers([]BackupTier{
{TargetID: "local", Cadence: time.Hour, Primary: true, Service: svc},
{TargetID: "felhom-pbs", Cadence: time.Hour, Service: nil},
}, svc, 24*time.Hour)
if len(got) != 1 || got[0].TargetID != "local" {
t.Fatalf("a serviceless tier must not be advertised (it could never run): %+v", got)
}
})
}
// R-82 Slice D: a tier whose TARGET STORAGE does not exist yet is DEFERRED, not due.
//
// A fresh box carries the offsite tier in its installer defaults, but `felhom-pbs` only appears when
// the hub provisions the DR tier. Reporting "due" in that window would have the controller quiesce
// the apps and fire a vzdump at a non-existent storage every cadence until provisioning happens.
func TestBackupDue_TargetStorageMissing_Defers(t *testing.T) {
st := &fakeStore{}
// Storage view knows only "local" — the PBS tier's target is not provisioned yet.
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: st,
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local", Type: "local"}}},
Tokens: staticTokens{"A": 8200},
BackupTiers: []BackupTier{
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: &fakeBackups{}},
},
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.now = func() time.Time { return testNow }
h := srv.Handler()
var pbs, local struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if pbs.Data.Due {
t.Fatalf("an unprovisioned tier must DEFER, not fire a vzdump at a storage that does not exist; got %+v", pbs.Data)
}
if !strings.Contains(pbs.Data.Reason, "not present") {
t.Fatalf("the deferral must say WHY, or it is indistinguishable from a healthy tier; got %q", pbs.Data.Reason)
}
// The provisioned tier is unaffected — no evidence yet, so due.
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=local", "A", "").Body.Bytes(), &local); err != nil {
t.Fatal(err)
}
if !local.Data.Due {
t.Fatalf("a PROVISIONED tier with no backup yet must still be due; got %+v", local.Data)
}
}
// A storage-view ERROR must NOT defer. "I could not check" is not "not there" — reading it that way
// would silently suppress backups, the absence-is-not-failure rule this project keeps relearning.
func TestBackupDue_StorageViewError_DoesNotSuppress(t *testing.T) {
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
Storage: errStorage{}, // reused from f2_role_fallback_test.go — Observe always fails
Tokens: staticTokens{"A": 8200},
BackupTiers: []BackupTier{
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: &fakeBackups{}},
},
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.now = func() time.Time { return testNow }
var pbs struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, srv.Handler(), "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if !pbs.Data.Due {
t.Fatalf("a storage-view error must not suppress the backup (fail toward due); got %+v", pbs.Data)
}
}
// ── R-84: the cold in-memory store must not cause a redundant backup ─────────────────────────
// archiveLister is a fakeBackups that ALSO knows when a backup last landed on its storage.
type archiveLister struct {
*fakeBackups
at time.Time
found bool
err error
}
func (a archiveLister) NewestArchiveTime(context.Context, int) (time.Time, bool, error) {
return a.at, a.found, a.err
}
func listerServer(t *testing.T, st *fakeStore, pbsSvc BackupService) http.Handler {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: st,
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local"}, {Name: "felhom-pbs"}}},
Tokens: staticTokens{"A": 8200},
BackupTiers: []BackupTier{
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: pbsSvc},
},
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.now = func() time.Time { return testNow }
return srv.Handler()
}
func dueFor(t *testing.T, h http.Handler, target string) BackupDueResponse {
t.Helper()
var out struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target="+target, "A", "").Body.Bytes(), &out); err != nil {
t.Fatal(err)
}
return out.Data
}
// THE R-84 CASE. The in-memory store is EMPTY (the agent just restarted), but the storage holds a
// snapshot from 2 hours ago. The tier must NOT be due — otherwise every agent deploy costs a fresh
// multi-hour offsite upload. Three redundant local backups were observed on demo-felhom in one
// afternoon of deploys before this.
//
// COMPANION RED-PROOF (observed): delete the newestArchiveOn fold-in from handleBackupDue (the
// pre-R-84 shape, in-memory only) → this fails with
// "a restart must NOT make the tier due when the storage holds a 2h-old backup;
//
// got {... Due:true Reason:no successful backup recorded yet ...}". Restored.
func TestBackupDue_ColdStore_UsesStorageGroundTruth(t *testing.T) {
h := listerServer(t, &fakeStore{}, archiveLister{
fakeBackups: &fakeBackups{}, at: testNow.Add(-2 * time.Hour), found: true,
})
got := dueFor(t, h, "felhom-pbs")
if got.Due {
t.Fatalf("a restart must NOT make the tier due when the storage holds a 2h-old backup; got %+v", got)
}
if got.AgeSecs == nil || *got.AgeSecs != int64((2*time.Hour).Seconds()) {
t.Fatalf("the age must come from the storage; got %+v", got)
}
}
// Ground truth that is genuinely OLD still makes the tier due — this must not become a blanket
// suppressor.
func TestBackupDue_ColdStore_OldArchiveIsStillDue(t *testing.T) {
h := listerServer(t, &fakeStore{}, archiveLister{
fakeBackups: &fakeBackups{}, at: testNow.Add(-9 * 24 * time.Hour), found: true,
})
if got := dueFor(t, h, "felhom-pbs"); !got.Due {
t.Fatalf("a 9-day-old archive under a 7-day cadence MUST still be due; got %+v", got)
}
}
// A storage that genuinely holds nothing → due. The fix must not invent a backup.
func TestBackupDue_ColdStore_NoArchiveIsDue(t *testing.T) {
h := listerServer(t, &fakeStore{}, archiveLister{fakeBackups: &fakeBackups{}, found: false})
if got := dueFor(t, h, "felhom-pbs"); !got.Due {
t.Fatalf("no archive anywhere → due; got %+v", got)
}
}
// A storage-read ERROR must fall back to the in-memory record, NOT be read as "a backup exists".
// An unreadable storage must never make a tier look freshly backed up.
func TestBackupDue_StorageReadError_DoesNotFakeFreshness(t *testing.T) {
h := listerServer(t, &fakeStore{}, archiveLister{
fakeBackups: &fakeBackups{}, at: testNow, found: true, err: errStorageRead,
})
if got := dueFor(t, h, "felhom-pbs"); !got.Due {
t.Fatalf("a storage-read error must not fake freshness — the in-memory record is empty, so DUE; got %+v", got)
}
}
var errStorageRead = errors.New("simulated storage read failure")
// The in-memory record WINS when it is newer than the storage listing — a backup that just finished
// this process lifetime is more current than a listing that may lag.
func TestBackupDue_InMemoryRecordWinsWhenNewer(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("felhom-pbs", 8200, time.Hour, true)) // 1h ago, in memory
h := listerServer(t, st, archiveLister{
fakeBackups: &fakeBackups{}, at: testNow.Add(-9 * 24 * time.Hour), found: true, // stale listing
})
got := dueFor(t, h, "felhom-pbs")
if got.Due {
t.Fatalf("the fresher in-memory record must win over a stale listing; got %+v", got)
}
}
// A service WITHOUT the optional lister degrades to the pre-R-84 behaviour, unchanged.
func TestBackupDue_ServiceWithoutLister_UnchangedBehaviour(t *testing.T) {
h := listerServer(t, &fakeStore{}, &fakeBackups{}) // plain BackupService
if got := dueFor(t, h, "felhom-pbs"); !got.Due || got.Reason != "no successful backup recorded yet" {
t.Fatalf("a plain BackupService must behave exactly as before; got %+v", got)
}
}
+270 -10
View File
@@ -8,6 +8,7 @@ import (
"strings" "strings"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage" "gitea.dooplex.hu/admin/felhom-agent/internal/storage"
) )
@@ -78,8 +79,10 @@ type GuestAttacher interface {
DetachDrive(ctx context.Context, where string) error DetachDrive(ctx context.Context, where string) error
// EnsureSharedParent makes the host stable parent shared + installs the boot-persistence unit. // EnsureSharedParent makes the host stable parent shared + installs the boot-persistence unit.
EnsureSharedParent(ctx context.Context) error EnsureSharedParent(ctx context.Context) error
// GuestSeesMount reports whether vmid's guest sees `path` as a mount in its own namespace (the // GuestSeesMount reports whether vmid's guest sees `path` as a mount in its own namespace the
// guest-usable signal distinct from the host having the bind). Backs BoundUnderParent. // guest-VISIBILITY signal, distinct from the host having the bind. One of the three terms behind
// BoundUnderParent. It is a path-presence test and NOT a liveness signal (R-117): a stale bind over a
// dead device is still "seen". Liveness is bindLiveness's job.
GuestSeesMount(ctx context.Context, vmid int, path string) bool GuestSeesMount(ctx context.Context, vmid int, path string) bool
// GuestBootID returns a token that changes on every guest boot (host or guest) but is stable across a // GuestBootID returns a token that changes on every guest boot (host or guest) but is stable across a
// controller-only restart — the deterministic guest-reboot signal the controller recreates apps on. // controller-only restart — the deterministic guest-reboot signal the controller recreates apps on.
@@ -142,15 +145,49 @@ type DiskInfo struct {
// HDD look available when it wasn't bound. Only meaningful for user-data drives. LEGACY (per-drive // HDD look available when it wasn't bound. Only meaningful for user-data drives. LEGACY (per-drive
// `pct set -mpN` model) — the intermediary model uses BoundUnderParent. // `pct set -mpN` model) — the intermediary model uses BoundUnderParent.
GuestAttached bool `json:"guest_attached"` GuestAttached bool `json:"guest_attached"`
// BackupTarget (E-2) reports that this drive backs the PRIMARY whole-guest backup tier. Additive:
// an older controller ignores it. It is the agent's answer, not the controller's intent flag —
// on a hand-migrated box (E-1) intent is unset while the drive really is the target.
BackupTarget bool `json:"backup_target,omitempty"`
// GuestPath is the drive's STABLE in-guest path in the intermediary-mount model // GuestPath is the drive's STABLE in-guest path in the intermediary-mount model
// (/mnt/felhom-drives/<name>). This is what the controller repoints HDD_PATH to and registers as the // (/mnt/felhom-drives/<name>). This is what the controller repoints HDD_PATH to and registers as the
// storage path. Set for /mnt/<name> drives; "" otherwise. Distinct from MountPath (the RAW host PVE // storage path. Set for /mnt/<name> drives; "" otherwise. Distinct from MountPath (the RAW host PVE
// mount the agent ops on). // mount the agent ops on).
GuestPath string `json:"guest_path,omitempty"` GuestPath string `json:"guest_path,omitempty"`
// BoundUnderParent reports whether the drive's felhom-data is currently bound under the shared parent // BoundUnderParent reports whether the drive is live + usable in the guest in the intermediary model.
// at GuestPath (a host mount-table check) — i.e. live + usable in the guest in the intermediary model.
// The controller's drive-absent gate + auto-restart key on this (and State). // The controller's drive-absent gate + auto-restart key on this (and State).
//
// It is a CONJUNCTION of THREE facts, and all three are load-bearing:
// 1. felhom-data is bound under the shared parent at GuestPath (the guest-visible mount check), and
// 2. the drive's RAW host mount is still mounted — i.e. the DEVICE is still there (R-113), and
// 3. the bind actually WORKS: it names the same device as the raw mount, and that filesystem has
// not aborted (R-117, v0.117.0 — see bindLiveness).
// Half 1 alone was the bug R-113 fixed: the raw mount is device-bound and dies with its device, but
// the agent's own bind is not, so half 1 stays true over a stale shell after the device is pulled. The
// controller read that survivor as "present" and the drive-absent alarm could never fire — measured
// live in E-2d (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.2). Half 2 alone would regress boot
// ordering, where the raw mounts early and the bind lands ~18s later; the conjunction keeps that
// window reading absent.
//
// Terms 1 and 2 TOGETHER were still not liveness, which is R-117: both are path-presence tests
// comparing only field 5 of a mountinfo line, so both stay true over a bind that names the drive that
// went away while the raw mount healed onto the returning one via its fs-UUID-keyed unit. Measured
// live: raw on 8:32 /dev/sdc, bind on 8:16 /dev/sdb with `shutdown`, this field TRUE, EIO on every
// read and write, and the gate restarting the customer's apps onto it with no alarm on any channel
// (felhom.eu audits/SPIKE-r117-bind-liveness-2026-07-30.md §5.2).
//
// THE TESTS THAT PIN THIS COMMENT, because for three releases it promised a property nothing tested
// (spike §5.3): disks_bind_liveness_test.go — TestDisks_BindLiveness_StaleBindReadsAbsent (term 3,
// case a), _AbortedFilesystemReadsAbsent (term 3, case b, the steady-state case that emits nothing
// today), _UnknownIsTreatedAsPresent (the cannot-tell rule) and _HealthyReadsPresent (no false
// negative). Each asserts the CONSEQUENCE — what this field reads — not the mechanism.
BoundUnderParent bool `json:"bound_under_parent"` BoundUnderParent bool `json:"bound_under_parent"`
// Smart is the already-computed per-disk SMART health summary (v0.94.0), serialized here so the
// controller can render a disk-health card + degradation alert WITHOUT any new smartctl load — the
// value is copied straight from the target's Observe-time enrichment. omitempty + a pointer so a
// device that exposes no SMART (USB bridge, unread) is ABSENT, not a misleading zero-value UNKNOWN;
// the controller feature-detects presence and renders "Nincs adat" when nil (never alarms).
Smart *hub.SmartSummary `json:"smart,omitempty"`
} }
// handleDisks lists the host's drives + data-bearing flags (read-only/benign). // handleDisks lists the host's drives + data-bearing flags (read-only/benign).
@@ -170,6 +207,9 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
// F9: which host mount paths are actually BOUND into THIS guest's config (guest-usable, not just // F9: which host mount paths are actually BOUND into THIS guest's config (guest-usable, not just
// host-present). A bind's mp= equals the guest path, which is the drive's host mount path (`where`). // host-present). A bind's mp= equals the guest path, which is the drive's host mount path (`where`).
boundPaths := s.guestBoundPaths(r.Context(), vmid) boundPaths := s.guestBoundPaths(r.Context(), vmid)
// E-2: the PRIMARY tier's storage id — the whole-guest vzdump destination. "" when no tier
// carries a target (the legacy single-tier shape), which correctly flags nothing.
primaryTargetID := s.primaryTier().TargetID
out := make([]DiskInfo, 0, len(targets)) out := make([]DiskInfo, 0, len(targets))
for _, t := range targets { for _, t := range targets {
di := DiskInfo{ di := DiskInfo{
@@ -181,13 +221,79 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
UsedBytes: t.UsedBytes, UsedBytes: t.UsedBytes,
UsedFraction: t.UsedFraction, UsedFraction: t.UsedFraction,
GuestAttached: t.MountPath != "" && boundPaths[t.MountPath], GuestAttached: t.MountPath != "" && boundPaths[t.MountPath],
// E-2: is THIS drive the whole-guest backup target? The agent is the only component that
// can answer — the controller's own StoragePath.BackupTarget is customer INTENT, and on a
// box migrated by hand (E-1) nobody ever assigned it, so intent is empty while the drive
// really is the target. Reported here so the controller can name the drive in an
// absent-target alarm and hide the destructive controls the agent would refuse anyway.
BackupTarget: t.Name == primaryTargetID,
} }
// Intermediary model: the stable in-guest path + whether felhom-data is bound under the parent. // Intermediary model: the stable in-guest path + whether felhom-data is bound under the parent.
// Only user-data /mnt/<name> drives have a guest path (system/backup mounts never cross in). // Only user-data /mnt/<name> drives have a guest path (system/backup mounts never cross in).
if di.Role == string(storage.RoleUserData) { if di.Role == string(storage.RoleUserData) {
if gp := StablePathForRaw(t.MountPath); gp != "" { if gp := StablePathForRaw(t.MountPath); gp != "" {
di.GuestPath = gp di.GuestPath = gp
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) // R-113: AND in device presence. A conjunction, deliberately — it leaves the
// boot-ordering behaviour the controller's gate depends on exactly as it was
// (raw mounted early, bind not yet ⇒ still absent) while closing the case the
// gate could never see (bind outlived the device ⇒ now absent).
// R-117: AND in bind LIVENESS. The two terms above are both path-presence tests, so
// both stay true over a bind that names the drive that went away while the raw mount
// healed onto the returning one — EIO on every call, payload healthy.
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) &&
s.devicePresent(t.MountPath) &&
s.bindUsable(gp, t.MountPath)
}
}
// R-116: carry the GUEST PATH on the backup-target row even when its role has flipped to
// system — but ONLY when that flip was caused by the device vanishing.
//
// WHY. The controller keys the drive-absent alarm on the registered StoragePath, which for an
// external drive is the GUEST path. When the device goes, Observe's exactMountDevice fails, so
// t.BackingDevice becomes "" and RoleForStorage returns RoleSystem (role.go:180-181) — the block
// above is skipped and this row loses its guest path. It keeps its MountPath, so the union loop
// below DEDUPES the registry row away (`seen[d.MountPath]`), and /disks ends up carrying NO row
// with that guest path at all. driveTargetByPath then has no entry, isTarget[guestPath] is a
// missing key, and the specific backup_target_absent alarm cannot fire — the generic one goes
// out instead, while the RETURN (rows rejoined) fires the specific recovery. An unmatchable
// pair. Measured live: felhom.eu audits/SESSION-C-2026-07-29.md §5.
//
// THE GATES, each load-bearing:
// di.GuestPath == "" — never touch the user-data path above; this is a fallback, not a rule.
// di.BackupTarget — only the target row. No other system/backup mount gains a guest path,
// so the boundary at :213-214 stands: this is not "system mounts now
// cross into the guest", it is "the drive the alarm is about keeps its
// identity while it is missing".
// t.BackingDevice == "" — ONLY the vanished-device flip. A storage that is RoleSystem because
// it is genuinely system-BACKED has a non-empty BackingDevice and is
// excluded. Without this gate a dir storage at /mnt/<name> living on the
// root disk would acquire a guest path.
//
// Case B (the COMMON fresh-box shape) is safe twice over: the target is the builtin `local` on
// /var/lib/vz, and StablePathForRaw returns "" for anything that is not exactly /mnt/<name>
// (DriveNameFromRaw, intermediary.go:79-88), so nothing is set even before the gates apply.
//
// This cannot make the gate read an absent drive as PRESENT: BoundUnderParent is assigned only
// inside the two guest-path blocks a system-role row never enters, so it stays false, and
// planDriveGates computes present[gp] = present[gp] || d.BoundUnderParent. Inert by construction
// — pinned by TestAbsentTargetRowDoesNotRegisterPresence.
//
// v0.116.0 — WHY v0.115.0 (the MountPath-only form) WAS INERT, measured not reasoned. In the
// absent state t.MountPath is ALSO "" — the same exactMount failure that emptied BackingDevice
// empties it — so StablePathForRaw("") returned "" and this assigned nothing. Captured payload:
// felhom.eu audits/DIAG-r116-disks-payload-2026-07-30.md §6.2.
//
// t.ConfigPath is the fix: the storage's CONFIGURED path from storage.cfg, which is configuration
// and therefore survives the device. MountPath is tried FIRST so the present-state path and
// v0.115.0's tested behaviour are byte-identical; ConfigPath is consulted only when the mount is
// genuinely gone. MountPath is deliberately NOT back-filled from ConfigPath — see the union-dedup
// note below for the consumer that would break, and because a path that is not mounted is not a
// "host mountpoint" (this field's own contract, :152-153).
if di.GuestPath == "" && di.BackupTarget && t.BackingDevice == "" {
if gp := StablePathForRaw(t.MountPath); gp != "" {
di.GuestPath = gp
} else {
di.GuestPath = StablePathForRaw(t.ConfigPath)
} }
} }
// Inspect the backing device for the UI's data-bearing hint (the authoritative check // Inspect the backing device for the UI's data-bearing hint (the authoritative check
@@ -206,6 +312,13 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
di.WipeDurableID = wid di.WipeDurableID = wid
} }
} }
// v0.94.0: surface the already-computed SMART only when it was actually read (Health set).
// A zero-value summary (enrich skipped / no smartctl device) has Health "" → stays omitted, so
// the controller sees "absent" and renders "Nincs adat" rather than a false UNKNOWN.
if t.Smart.Health != "" {
sm := t.Smart
di.Smart = &sm
}
out = append(out, di) out = append(out, di)
} }
// Impl-2a: union in registry+units drives that Observe() does NOT surface (a drive with no PVE // Impl-2a: union in registry+units drives that Observe() does NOT surface (a drive with no PVE
@@ -213,16 +326,43 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
// NOT duplicated, and no Observe row is dropped (so this can never regress the current view). // NOT duplicated, and no Observe row is dropped (so this can never regress the current view).
if s.driveTargets != nil { if s.driveTargets != nil {
seen := make(map[string]bool, len(out)) seen := make(map[string]bool, len(out))
// R-116: dedup ALSO by guest path. `seen` keys on MountPath, the one field the absent state
// empties, so with the device gone /mnt/<name> is absent from `seen` and the registry row was NOT
// skipped — /disks carried the drive TWICE, the Observe row holding BackupTarget with no key and
// the registry row holding both keys with BackupTarget defaulted false. driveTargetByPath
// (controller intermediary.go:602-618) assigns rather than ORs, and the registry row is appended
// LAST, so its false won on both keys. Measured, 4 rows vs 3:
// felhom.eu audits/DIAG-r116-disks-payload-2026-07-30.md §6.2.
//
// THE JOIN, and it is the whole point: with the device gone the two records of one drive share NO
// runtime field — no mount, no backing device, and the Observe row's DurableID has degraded off the
// fs-UUID. What they DO share is CONFIGURATION: the Observe row's storage path (storage.cfg) and the
// registry row's unit `Where` (the .mount unit) are the same path, so both derive the same stable
// guest path. That is the key both sides can still compute, which is why the dedup keys on it.
seenGuest := make(map[string]bool, len(out))
for _, d := range out { for _, d := range out {
if d.MountPath != "" { if d.MountPath != "" {
seen[d.MountPath] = true seen[d.MountPath] = true
} }
if d.GuestPath != "" {
seenGuest[d.GuestPath] = true
}
} }
if drives, derr := s.driveTargets.Known(r.Context()); derr == nil { if drives, derr := s.driveTargets.Known(r.Context()); derr == nil {
for _, d := range drives { for _, d := range drives {
if d.MountPath == "" || seen[d.MountPath] { if d.MountPath == "" || seen[d.MountPath] {
continue continue
} }
// Same drive as an Observe row that already carries this guest path — skip it. Suppressing
// it rather than teaching it BackupTarget is deliberate: the registry row has a non-empty
// MountPath (from the unit file, stale by then), and the controller reads
// `d.BackupTarget && d.MountPath != ""` as "a real drive with its own mountpoint — HEALTHY"
// (backup_target_offer.go:79). Putting the flag on a row with a stale MountPath would have
// silently regressed R-114, telling the customer the backup target is fine while its drive
// is missing. Pinned by TestAbsentTargetKeepsR114DegradedSignal.
if gp := StablePathForRaw(d.MountPath); gp != "" && seenGuest[gp] {
continue
}
di := DiskInfo{ di := DiskInfo{
Name: d.Name, Type: d.Type, State: "attached", Name: d.Name, Type: d.Type, State: "attached",
MountPath: d.MountPath, DurableID: d.DurableID, MountPath: d.MountPath, DurableID: d.DurableID,
@@ -233,7 +373,13 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
// exactly like the Observe path — else the controller reads a registry drive as "Leválasztva". // exactly like the Observe path — else the controller reads a registry drive as "Leválasztva".
if gp := StablePathForRaw(d.MountPath); gp != "" { if gp := StablePathForRaw(d.MountPath); gp != "" {
di.GuestPath = gp di.GuestPath = gp
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) // R-113 + R-117: same three-term conjunction as the Observe path. This path matters
// MORE, not less — a registry drive with no PVE dir-storage is exactly the shape
// E-2d detached, and its State is hardcoded "attached" below, so these checks are
// the only device truth this row carries.
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) &&
s.devicePresent(d.MountPath) &&
s.bindUsable(gp, d.MountPath)
} }
// A registry drive has no PVE `pvesm status` snapshot, so fill backing device + capacity // A registry drive has no PVE `pvesm status` snapshot, so fill backing device + capacity
// from the host directly: resolve the device by fs-UUID, and statfs the mount for size — // from the host directly: resolve the device by fs-UUID, and statfs the mount for size —
@@ -241,10 +387,17 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
if d.UUID != "" { if d.UUID != "" {
// Resolve to the real /dev node (e.g. /dev/sdd), not the by-uuid symlink path, to match // Resolve to the real /dev node (e.g. /dev/sdd), not the by-uuid symlink path, to match
// how Observe-sourced rows display the backing device. // how Observe-sourced rows display the backing device.
if dev, err := storage.ResolveStorageDevice("uuid:" + d.UUID); err == nil { if dev, err := s.resolveStorageDevice("uuid:" + d.UUID); err == nil {
di.BackingDevice = dev di.BackingDevice = dev
} }
} }
// Fix B (v0.95.0): union-path drives skip Observe's enrich, so read SMART here through the
// same seam the dir targets use. Only set when the read actually ran (Health != "").
if di.BackingDevice != "" && s.smart != nil {
if sm := s.smart.SMARTForBacking(r.Context(), di.BackingDevice); sm.Health != "" {
di.Smart = &sm
}
}
if total, used, okc := statfsCapacity(d.MountPath); okc { if total, used, okc := statfsCapacity(d.MountPath); okc {
di.TotalBytes, di.UsedBytes = total, used di.TotalBytes, di.UsedBytes = total, used
di.UsedFraction = float64(used) / float64(total) di.UsedFraction = float64(used) / float64(total)
@@ -357,6 +510,11 @@ func (s *Server) handleDiskEject(w http.ResponseWriter, r *http.Request, vmid in
writeErr(w, http.StatusForbidden, "mount is system/backup-protected — eject refused (role: "+string(role)+")") writeErr(w, http.StatusForbidden, "mount is system/backup-protected — eject refused (role: "+string(role)+")")
return return
} }
// E-2c: the role gate above passes a drive that is BOTH user-data and the vzdump target (E-1 put
// the target on the enrolled drive's own mountpoint). Refuse specifically, naming the remedy.
if s.refuseIfBackupTarget(r.Context(), w, "eject", vmid, req.Where) {
return
}
dependents := s.dependentGuests(r.Context(), req.Where) dependents := s.dependentGuests(r.Context(), req.Where)
// Record the EJECT intent BEFORE unmounting (the target still resolves to its durable-id) so the // Record the EJECT intent BEFORE unmounting (the target still resolves to its durable-id) so the
// self-heal watchdog leaves it alone — an OFFICIAL eject is the only thing that sets this (P3); an // self-heal watchdog leaves it alone — an OFFICIAL eject is the only thing that sets this (P3); an
@@ -410,6 +568,11 @@ func (s *Server) handleDiskDecommission(w http.ResponseWriter, r *http.Request,
writeErr(w, http.StatusForbidden, "mount is system/backup-protected — decommission refused (role: "+string(role)+")") writeErr(w, http.StatusForbidden, "mount is system/backup-protected — decommission refused (role: "+string(role)+")")
return return
} }
// E-2c: same narrow gate as eject — decommission migrates data off and retires the drive, which
// would strand the backup target just as thoroughly.
if s.refuseIfBackupTarget(r.Context(), w, "decommission", vmid, req.Where) {
return
}
dependents := s.dependentGuests(r.Context(), req.Where) dependents := s.dependentGuests(r.Context(), req.Where)
// Resolve the durable-id BEFORE unmounting (it still resolves while mounted) for the bind prune. // Resolve the durable-id BEFORE unmounting (it still resolves while mounted) for the bind prune.
id := s.durableIDForMount(r.Context(), req.Where) id := s.durableIDForMount(r.Context(), req.Where)
@@ -803,9 +966,12 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
} }
// boundUnderParent reports whether a drive's felhom-data is bound at its stable guest path AND visible // boundUnderParent reports whether a drive's felhom-data is bound at its stable guest path AND visible
// inside the guest (the usable-in-guest signal the controller's gate keys on — a guest reboot leaves the // inside the guest — a guest reboot leaves the host bind in place but invisible to the guest until
// host bind in place but invisible to the guest until re-propagated). Injectable via s.boundCheck for // re-propagated. Injectable via s.boundCheck for tests; defaults to the guest-namespace mount check.
// tests; defaults to the guest-namespace mount check. //
// This is VISIBILITY, not liveness (R-117). It compares only the mount point, so it stays true over a
// bind whose device has gone; do not read it as "usable in the guest" — that is the whole three-term
// conjunction at the two /disks construction sites, whose third term is bindUsable.
func (s *Server) boundUnderParent(ctx context.Context, vmid int, stablePath string) bool { func (s *Server) boundUnderParent(ctx context.Context, vmid int, stablePath string) bool {
if s.boundCheck != nil { if s.boundCheck != nil {
return s.boundCheck(stablePath) return s.boundCheck(stablePath)
@@ -816,6 +982,44 @@ func (s *Server) boundUnderParent(ctx context.Context, vmid int, stablePath stri
return s.guestAttach.GuestSeesMount(ctx, vmid, stablePath) return s.guestAttach.GuestSeesMount(ctx, vmid, stablePath)
} }
// devicePresent reports whether the drive's BACKING DEVICE is still there, by asking whether its RAW
// host mount is still a mountpoint (R-113).
//
// WHY THE RAW MOUNT AND NOT THE BIND. The raw mount at /mnt/<name> is a systemd mount unit bound to
// its device: when the device goes, the unit stops and the mountpoint disappears. The agent's own bind
// of <raw>/felhom-data under the shared parent is an ordinary bind — nothing ties it to the device, so
// its mountinfo entry OUTLIVES the device as a stale shell. Measured live in E-2d with the device
// pulled: `/mnt/mentes2` NOT mounted while `/mnt/felhom-drives/mentes2` still read
// `/dev/sdb[/felhom-data]` (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.2). Keying presence on the
// survivor is exactly why the controller's drive-absent gate could never fire.
//
// An empty raw path means we have nothing to ask about — return TRUE (unknown), never false. Absent
// stops a customer's apps, so "cannot tell" must never be reported as "gone".
func (s *Server) devicePresent(rawMountPath string) bool {
if rawMountPath == "" {
return true // cannot tell → never claim absent
}
if s.deviceCheck != nil {
return s.deviceCheck(rawMountPath)
}
return isHostMountpoint(rawMountPath)
}
// bindUsable is the THIRD term of the BoundUnderParent conjunction (R-117): the bind must not only exist
// and be guest-visible, it must actually WORK. The first two terms are path-presence tests and are both
// satisfied by a bind that names the drive that went away — measured live, with EIO on every read and
// write while the payload read healthy and the gate restarted the customer's apps onto it.
//
// UNKNOWN counts as usable, via BindLiveness.Usable — the same "cannot tell → never absent" rule
// devicePresent applies above, and for the same reason: a false absent stops a working customer's apps.
// Injectable via s.livenessCheck; the default reads /proc only and issues NO block I/O (CLAUDE.md).
func (s *Server) bindUsable(stable, rawMountPath string) bool {
if s.livenessCheck != nil {
return s.livenessCheck(stable, rawMountPath).Usable()
}
return bindLiveness(stable, rawMountPath).Usable()
}
// guestBoundPaths returns the set of guest mountpoint paths (the `mp=` of each entry in the guest's // guestBoundPaths returns the set of guest mountpoint paths (the `mp=` of each entry in the guest's
// config) — i.e. the host drives actually BOUND into the guest. F9: this is the guest-attached signal // config) — i.e. the host drives actually BOUND into the guest. F9: this is the guest-attached signal
// (`GuestAttached`) that distinguishes a guest-usable drive from one merely present on the host. A bind // (`GuestAttached`) that distinguishes a guest-usable drive from one merely present on the host. A bind
@@ -1020,6 +1224,62 @@ func (s *Server) hostReader() storage.HostReader {
return storage.NewProcHostReader() return storage.NewProcHostReader()
} }
// backupTargetAt reports the configured backup TIER whose storage is mounted at `where`, or "" when
// none is. E-2c.
//
// WHY THIS IS NOT A ROLE RECLASSIFICATION. The obvious fix is to make RoleForStorage return
// RoleBackup for the target's storage, and it is wrong here: on both demo boxes the drive that now
// holds the whole-guest archives is ALSO the enrolled user-data drive (E-1 put the vzdump target on
// the drive's own mountpoint, beside felhom-data). Reclassifying it would refuse every legitimate
// eject/decommission of the customer's own data drive — an over-correction that trades one silent
// failure for a permanent obstruction. So this is a SEPARATE, narrower gate that names exactly what
// it protects and leaves the role vocabulary alone.
//
// It resolves through the agent's OWN storage view (never the caller's claim) and fails OPEN — an
// unreadable view returns "" so this gate cannot block on a transient error. That is safe because it
// sits BEHIND the role gate, which already fails SAFE on the same error: an unresolvable mount is
// refused there before it ever reaches this check.
func (s *Server) backupTargetAt(ctx context.Context, where string) string {
if where == "" || s.storage == nil {
return ""
}
targets, err := s.storage.Observe(ctx)
if err != nil {
return "" // fail OPEN — the role gate already fails SAFE on this same error
}
for _, t := range s.tiers {
if t.TargetID == "" {
continue
}
for _, tgt := range targets {
if tgt.Name == t.TargetID && tgt.MountPath == where {
return t.TargetID
}
}
}
return ""
}
// refuseIfBackupTarget refuses a destructive drive op when `where` backs a configured backup tier,
// and reports whether it did. The message names the storage AND the remedy: the operation is not
// forbidden forever, it is ordered — reassign the backup target first, then the drive is free.
//
// Ejecting the drive that holds the only local whole-guest backup is exactly the silent-degradation
// class this arc has been closing: it succeeds, nothing alarms, and the box quietly loses its
// drive-loss protection while still reporting a configured tier.
func (s *Server) refuseIfBackupTarget(ctx context.Context, w http.ResponseWriter, op string, vmid int, where string) bool {
target := s.backupTargetAt(ctx, where)
if target == "" {
return false
}
s.logger.Warn("local-api: protected — "+op+" refused: the mount backs a configured backup tier",
"vmid", vmid, "where", where, "target", target)
writeErr(w, http.StatusConflict,
"this drive is the whole-guest backup target ("+target+") — "+op+" refused. "+
"Reassign the backup target to another drive first, or the box loses its local drive-loss protection.")
return true
}
// roleForMountPath resolves the AUTHORITATIVE protection role of the storage mounted at `where`, from // roleForMountPath resolves the AUTHORITATIVE protection role of the storage mounted at `where`, from
// the agent's OWN storage view + host topology (never the caller's claim). It mirrors deviceRole but // the agent's OWN storage view + host topology (never the caller's claim). It mirrors deviceRole but
// keys on the mount path (the eject input). It FAILS SAFE to system (most-protected) on any // keys on the mount path (the eject input). It FAILS SAFE to system (most-protected) on any
@@ -0,0 +1,342 @@
package localapi
import (
"context"
"encoding/json"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// R-116 — the backup-target flag must be reachable from the row the CONTROLLER keys on.
//
// THE DEFECT. The controller resolves the drive-absent alarm by the registered StoragePath, which for
// an external drive is the GUEST path. When the device vanishes, Observe's exactMountDevice fails, so
// BackingDevice becomes "" and RoleForStorage returns RoleSystem (role.go:180-181) — the guest-path
// block is skipped and the flag-bearing row loses its guest path. It keeps its MountPath, so the union
// loop DEDUPES the registry row away, and /disks carries NO row with that guest path at all.
// driveTargetByPath then has no entry, isTarget[guestPath] is a MISSING KEY, and the specific
// backup_target_absent alarm cannot fire — the generic storage_disconnected goes out instead, while
// the RETURN (rows rejoined) fires the specific recovery. An operator gets a pair they cannot match.
// Measured live: felhom.eu audits/SESSION-C-2026-07-29.md §5.
//
// These tests exercise the REAL GET /disks response and assert the emitted JSON, because the failure
// class is "the value is on the wrong row" — a test that hand-builds rows proves nothing about which
// row the handler actually emits.
// targetRowServer builds a /disks server whose primary backup tier is `primaryTarget`, over the given
// Observe targets. boundCheck/deviceCheck are pinned so the R-113 conjunction is not the variable
// under test here.
func targetRowServer(t *testing.T, primaryTarget string, targets []hub.StorageTarget) *Server {
t.Helper()
return targetRowServerWithDrives(t, primaryTarget, targets, nil)
}
// targetRowServerWithDrives additionally wires the REGISTRY union source. v0.115.0's tests left
// DriveTargets nil, so the union loop never ran and the two-row absent shape — the actual defect — was
// invisible to the whole suite. Any test about which row carries what MUST populate this.
func targetRowServerWithDrives(t *testing.T, primaryTarget string, targets []hub.StorageTarget,
drives []storage.KnownTarget) *Server {
t.Helper()
var known storage.KnownTargets
if drives != nil {
known = fakeKnownTargets{drives: drives}
}
srv, err := NewServer(Options{
DriveTargets: known,
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuestsCfg{}, Backups: &fakeBackups{}, Store: &fakeStore{},
Storage: fakeStorage{targets: targets},
// Service is REQUIRED: normalizeBackupTiers (backup_tiers.go:21-22) drops any tier with a nil
// Service, and the legacy fallback then yields TargetID "" — which silently makes every
// BackupTarget false and would make these tests pass for the wrong reason.
BackupTiers: []BackupTier{{TargetID: primaryTarget, Primary: true, Service: &fakeBackups{}}},
Tokens: staticTokens{"A": 8200},
Disks: &fakeDiskOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4"}},
DiskGate: &fakeGate{}, HostReader: sysOnSDA(),
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.boundCheck = func(string) bool { return true }
srv.deviceCheck = func(string) bool { return true }
return srv
}
// wireDisks returns the decoded /disks rows exactly as the controller receives them.
func wireDisks(t *testing.T, srv *Server) []map[string]any {
t.Helper()
body := do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes()
var w struct {
Data struct {
Disks []map[string]any `json:"disks"`
} `json:"data"`
}
if err := json.Unmarshal(body, &w); err != nil {
t.Fatalf("decode /disks: %v (%s)", err, body)
}
return w.Data.Disks
}
// isTargetByPath reproduces the controller's driveTargetByPath EXACTLY (intermediary.go:602-616):
// both keyings, value = backup_target. This is the map whose missing key is the whole defect, so the
// assertion is made against a faithful copy of it rather than against a field in isolation.
func isTargetByPath(disks []map[string]any) map[string]bool {
out := map[string]bool{}
for _, d := range disks {
bt, _ := d["backup_target"].(bool)
if gp, ok := d["guest_path"].(string); ok && gp != "" {
out[gp] = bt
}
if mp, ok := d["mount_path"].(string); ok && mp != "" {
out[mp] = bt
}
}
return out
}
// theAbsentTarget is the absent-target Observe row, CORRECTED in v0.116.0 to the shape the live box
// actually produces.
//
// THIS FIXTURE IS WHY AN INERT FIX SHIPPED GREEN. As written for v0.115.0 it supplied
// `MountPath: "/mnt/mentes"` — a field the real absent state does NOT have. The same exactMount failure
// that empties BackingDevice empties MountPath (observe.go:184-190), so on the live box this row carries
// `mount_path: ""`, and v0.115.0's `StablePathForRaw(t.MountPath)` was therefore
// `StablePathForRaw("")` == "". The fixture handed the code a value production never supplies, the test
// went green, and the fix was inert on real hardware — twice.
//
// Captured payload this now mirrors, field for field:
// felhom.eu audits/DIAG-r116-disks-payload-2026-07-30.md §6.2.
var theAbsentTarget = hub.StorageTarget{
Name: "felhom-backup", Type: hub.StorageTypeLocalDir,
MountPath: "", BackingDevice: "", ConfigPath: "/mnt/mentes",
State: hub.StorageStateDisconnected,
// DurableID degrades off the fs-UUID exactly as the live payload showed (`path:/mnt/cel` there).
DurableID: "path:/mnt/mentes",
}
// theAbsentRegistryRow is the OTHER half of the live absent payload — the registry/union row. Its
// MountPath comes from the systemd .mount unit FILE (registry_known.go:40-75), which never consults the
// mount table, so it survives the device intact. Its presence is what made /disks carry the drive TWICE.
var theAbsentRegistryRow = []storage.KnownTarget{
{Name: "9303-uuid", Type: hub.StorageTypeUSB, MountPath: "/mnt/mentes",
DurableID: "uuid:9303", UUID: "9303"},
}
// ── the observable that must move ───────────────────────────────────────────────────────────────
// RED-PROOF: delete the `di.GuestPath == "" && di.BackupTarget && t.BackingDevice == ""` block and
// this fails with "the guest path the controller keys on is MISSING from /disks entirely".
func TestAbsentBackupTargetIsResolvableByGuestPath(t *testing.T) {
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
[]hub.StorageTarget{theAbsentTarget}, theAbsentRegistryRow))
isTarget := isTargetByPath(disks)
const guestPath = "/mnt/felhom-drives/mentes"
got, present := isTarget[guestPath]
if !present {
t.Fatalf("isTarget[%q] is a MISSING KEY — the guest path the controller keys on is missing from "+
"/disks entirely, so notifyDriveAbsent takes the generic branch and backup_target_absent "+
"can never fire (R-116)", guestPath)
}
if !got {
t.Fatalf("isTarget[%q] = FALSE. Both rows for this drive reached the wire and the registry row — "+
"appended last, BackupTarget defaulted false — overwrote the flag-bearing row's true. This is "+
"the measured live defect, not a hypothetical: rows=%d", guestPath, len(disks))
}
}
// ── V2: the new guest path must NOT make the gate read the drive as PRESENT ─────────────────────
// This is the over-correction guard, in the exact component under test. planDriveGates computes
// present[gp] = present[gp] || d.BoundUnderParent. If the row we now emit carried a true
// BoundUnderParent, this fix would SILENCE the alarm it exists to raise.
func TestAbsentTargetRowDoesNotRegisterPresence(t *testing.T) {
srv := targetRowServer(t, "felhom-backup", []hub.StorageTarget{theAbsentTarget})
// deviceCheck/boundCheck are pinned TRUE — the strongest possible case for a false positive.
// The row must still report bound_under_parent=false, because that field is only ever assigned
// inside the guest-path blocks a system-role row does not enter.
for _, d := range wireDisks(t, srv) {
if d["guest_path"] != "/mnt/felhom-drives/mentes" {
continue
}
if bup, _ := d["bound_under_parent"].(bool); bup {
t.Fatal("the absent backup-target row reports bound_under_parent=true — planDriveGates " +
"would compute present=true, the Stop branch would never run, and this fix would " +
"SUPPRESS the very alarm it exists to raise")
}
return
}
t.Fatal("the absent target row never reached the wire")
}
// ── V1: the gates, each on its own ──────────────────────────────────────────────────────────────
// Case B is the COMMON fresh-box shape, not an edge: the tier target is the builtin `local` on the
// root fs. It must never acquire a guest path.
func TestCaseBLocalTargetGetsNoGuestPath(t *testing.T) {
disks := wireDisks(t, targetRowServer(t, "local", []hub.StorageTarget{
{Name: "local", Type: "local", MountPath: "/var/lib/vz", BackingDevice: "", State: hub.StorageStateAttached},
}))
for _, d := range disks {
if gp, _ := d["guest_path"].(string); gp != "" {
t.Errorf("the Case B target on %v acquired guest path %q — a system-drive backup target "+
"must not cross into the guest", d["mount_path"], gp)
}
}
}
// A storage that is RoleSystem because it is genuinely system-BACKED (non-empty BackingDevice on the
// system disk) must be excluded — this is the case StablePathForRaw would NOT have filtered, since
// /mnt/<name> maps to a real stable path. The BackingDevice gate is what stops it.
func TestSystemBackedTargetUnderMntGetsNoGuestPath(t *testing.T) {
disks := wireDisks(t, targetRowServer(t, "sysbackup", []hub.StorageTarget{
// sysOnSDA() makes /dev/sda the system disk, so this classifies RoleSystem with a REAL device.
{Name: "sysbackup", Type: hub.StorageTypeLocalDir, MountPath: "/mnt/sysbackup",
BackingDevice: "/dev/sda1", State: hub.StorageStateAttached},
}))
for _, d := range disks {
if gp, _ := d["guest_path"].(string); gp != "" {
t.Errorf("a system-BACKED backup target acquired guest path %q — the BackingDevice gate "+
"failed and the :213-214 boundary was widened", gp)
}
}
}
// ── the negative ────────────────────────────────────────────────────────────────────────────────
// A drive that is NOT the target must not acquire the flag on any row, present or absent.
func TestNonTargetDriveNeverCarriesTheFlag(t *testing.T) {
disks := wireDisks(t, targetRowServer(t, "felhom-backup", []hub.StorageTarget{
{Name: "adat", Type: hub.StorageTypeLocalDir, MountPath: "/mnt/adat",
BackingDevice: "", State: hub.StorageStateDisconnected},
}))
for _, d := range disks {
if bt, _ := d["backup_target"].(bool); bt {
t.Errorf("non-target drive %v reports backup_target=true", d["name"])
}
if gp, _ := d["guest_path"].(string); gp != "" {
t.Errorf("an absent NON-target drive acquired guest path %q via the R-116 fallback — the "+
"BackupTarget gate failed", gp)
}
}
}
// ── v0.116.0 — the join, and the regression it must not cause ───────────────────────────────────
// THE JOIN. With the device gone the two records of one drive share no runtime field, so the dedup has
// to key on the one thing both can still derive: the CONFIGURED path (storage.cfg's `path` on the
// Observe side, the .mount unit's `Where` on the registry side), expressed as the stable guest path.
// This pins that exactly one row survives — because driveTargetByPath ASSIGNS rather than ORs, so two
// rows disagreeing on the flag is decided by append order, which is not a contract anyone should rely on.
//
// RED-PROOF: delete the `seenGuest[gp]` skip in the union loop and this fails with rows=2.
func TestAbsentTargetAppearsExactlyOnce(t *testing.T) {
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
[]hub.StorageTarget{theAbsentTarget}, theAbsentRegistryRow))
const guestPath = "/mnt/felhom-drives/mentes"
var rows []map[string]any
for _, d := range disks {
if gp, _ := d["guest_path"].(string); gp == guestPath {
rows = append(rows, d)
}
}
if len(rows) != 1 {
t.Fatalf("the absent drive is carried by %d rows, want exactly 1 — with two rows the flag the "+
"controller reads is decided by append order, not by the fix. rows=%v", len(rows), rows)
}
if bt, _ := rows[0]["backup_target"].(bool); !bt {
t.Error("the surviving row does not carry backup_target=true")
}
}
// THE REGRESSION THIS FIX MUST NOT CAUSE, and the reason neither obvious option was taken.
//
// The controller reads `d.BackupTarget && d.MountPath != ""` as "a real drive with its own mountpoint —
// HEALTHY" and returns immediately (backup_target_offer.go:79). So the two candidate fixes that look
// smallest — back-filling MountPath onto the Observe row, or teaching the registry row the flag (its
// MountPath is non-empty, read from the stale unit file) — BOTH produce a row satisfying that predicate
// while the drive is missing. Either would have silently regressed R-114, which shipped 2026-07-29 and
// tells the customer „A rendszermentés meghajtója nem érhető el" in exactly this state, flipping it back
// to a false healthy.
//
// R-114's correctness currently rests on the absent-state rows NOT combining the flag with a mount path.
// That coupling was invisible until the payload was captured, and it is what this test pins.
//
// RED-PROOF: set `MountPath: "/mnt/mentes"` on theAbsentTarget (v0.115.0's fixture value) and this fails.
func TestAbsentTargetKeepsR114DegradedSignal(t *testing.T) {
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
[]hub.StorageTarget{theAbsentTarget}, theAbsentRegistryRow))
for _, d := range disks {
bt, _ := d["backup_target"].(bool)
mp, _ := d["mount_path"].(string)
if bt && mp != "" {
t.Fatalf("row %v carries backup_target=true AND mount_path=%q while the drive is ABSENT. "+
"resolveBackupTargetState (backup_target_offer.go:79) reads that as \"a real drive with "+
"its own mountpoint — healthy\" and returns before its TargetAbsent branch, so the "+
"customer is told the backup target is fine while its drive is gone. That is R-114, "+
"regressed.", d["name"], mp)
}
}
}
// PRESENT-STATE PARITY. The fix must change nothing when the drive is there. Present state is the
// state every healthy box is in, so a change here reaches the whole fleet; absent state reaches only a
// box with a problem. Both rows are supplied, exactly as on a live present box, and the pre-existing
// MountPath dedup must still collapse them to one COMPLETE row.
func TestPresentTargetPayloadUnchanged(t *testing.T) {
present := hub.StorageTarget{
Name: "felhom-backup", Type: hub.StorageTypeLocalDir,
MountPath: "/mnt/mentes", BackingDevice: "/dev/sdb", ConfigPath: "/mnt/mentes",
State: hub.StorageStateAttached, DurableID: "uuid:9303",
}
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
[]hub.StorageTarget{present}, theAbsentRegistryRow))
var rows []map[string]any
for _, d := range disks {
if d["name"] == "felhom-backup" || d["mount_path"] == "/mnt/mentes" {
rows = append(rows, d)
}
}
if len(rows) != 1 {
t.Fatalf("present state carries the drive on %d rows, want 1 (the MountPath dedup): %v", len(rows), rows)
}
r := rows[0]
for field, want := range map[string]any{
"mount_path": "/mnt/mentes", "guest_path": "/mnt/felhom-drives/mentes",
"backing_device": "/dev/sdb", "role": "user-data", "state": "attached",
"backup_target": true, "bound_under_parent": true, "durable_id": "uuid:9303",
} {
if got := r[field]; got != want {
t.Errorf("present-state %s = %v, want %v — the fix altered the healthy payload", field, got, want)
}
}
}
// The negative, with the union loop actually running: a non-target absent drive gains the flag on no row
// and keeps its own registry row (nothing to dedup against, since no Observe row claims its guest path).
func TestAbsentNonTargetKeepsItsRegistryRowAndNoFlag(t *testing.T) {
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
[]hub.StorageTarget{{Name: "adat", Type: hub.StorageTypeLocalDir, MountPath: "",
BackingDevice: "", ConfigPath: "/mnt/adat", State: hub.StorageStateDisconnected}},
[]storage.KnownTarget{{Name: "adat-uuid", Type: hub.StorageTypeUSB,
MountPath: "/mnt/adat", DurableID: "uuid:1111", UUID: "1111"}}))
isTarget := isTargetByPath(disks)
for k, v := range isTarget {
if v {
t.Errorf("isTarget[%q] = true for a NON-target drive — the BackupTarget gate failed", k)
}
}
if _, ok := isTarget["/mnt/felhom-drives/adat"]; !ok {
t.Error("the non-target drive lost its guest-path key entirely — the union row was over-suppressed")
}
}
@@ -0,0 +1,397 @@
package localapi
import (
"context"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// R-117 — BoundUnderParent must mean THE BIND ACTUALLY WORKS, not "a mount by that name exists".
//
// THE BUG THESE PIN. GuestSeesMount (intermediary.go) and isHostMountpoint both parse a mountinfo line
// and then test only fields[4], the mount POINT. Field 3 — major:minor — sits in the same parsed slice
// and was discarded. So after a drive is detached and returned, the raw host mount heals onto the NEW
// device via its fs-UUID-keyed unit while the bind still names the OLD one, and BOTH existing terms stay
// true. Measured live: raw on 8:32 /dev/sdc, bind on 8:16 /dev/sdb with `shutdown`, BoundUnderParent
// TRUE, EIO on every read and write, and the controller's gate taking its Return branch — restarting the
// customer's apps onto that namespace and emailing backup_target_restored, with no alarm on any channel
// (felhom.eu audits/SPIKE-r117-bind-liveness-2026-07-30.md §3.3, §5.2).
//
// AND THE HALF THAT EMITS NOTHING AT ALL (spike §9, filed R-117a): a device that fails WITHOUT
// disappearing leaves the raw mount active, the devnos EQUAL, and the drive never Disconnected — so the
// gate produces neither a Stop nor a Return action and nothing is emitted, indefinitely. A devno
// comparison alone reads stale-device=false there, which is why term 3 checks the filesystem's own abort
// flags too. TestDisks_BindLiveness_AbortedFilesystemReadsAbsent is that case; a fix that shipped only
// the devno comparison would pass every other test in this file.
//
// WHY THE FIXTURES ARE REAL. Each mountinfo body below is the captured output of the spike run, not a
// hand-written line. R-116's fix shipped green and inert because its fixture supplied a MountPath
// production never supplies. These tests redirect procSelfMountinfo at a fixture file, so the REAL
// parser (hostMountEntries), the REAL predicate (bindLiveness) and the REAL /disks handler all run —
// the data is injected, the logic is not.
//
// RED-PROOFS (each verified to land, see REPORT.md): dropping `&& s.bindUsable(...)` from either /disks
// construction site fails StaleBindReadsAbsent / UnionPath_StaleBindReadsAbsent and
// AbortedFilesystemReadsAbsent; dropping the abort check so only the device comparison remains fails
// AbortedFilesystemReadsAbsent and AbortedWins_WhenDevnosAgree ALONE — that is the P1-only fix, and it is
// the one worth fearing; dropping `emergency_ro` from abortTokensByFS fails only the emergency_ro subtest;
// and returning BindLive instead of BindUnknown for an unreadable table fails UnknownIsTreatedAsPresent.
//
// A THIRD ordering trap, caught by TestBindLiveness_Verdicts during development and worth naming because
// it reports correctly while breaking the repair: reading the abort flag BEFORE comparing devices
// classifies the real return state as BindAborted, since its stale bind carries `shutdown` as well as a
// different device. BoundUnderParent still reads absent — every test in Group A still passes — but
// AttachDrive then refuses the re-bind that actually repairs it, so the self-heal never runs. The verdict
// must answer "would a re-bind help", which means reading the abort flag of the RAW mount (the re-bind's
// target) in the stale case, and of the bind itself only when the devices already agree.
// ── fixtures, from the spike's captures ─────────────────────────────────────────────────────────
//
// Devices, super options, optional-field tags and root paths are verbatim. ONE substitution: the spike
// ran against a SCRATCH shared parent (/mnt/r117-drives) so it could not disturb the live
// /mnt/felhom-drives peer group, whereas the code under test derives the stable path itself, as
// StableParentDir + "/" + DriveNameFromRaw(raw). So /mnt/r117-drives/sd becomes
// /mnt/felhom-drives/r117sd. Substituting anything else would make these fixtures describe a path
// production never produces — which is precisely how R-116's fix shipped green and inert.
// mountinfoHealthy is the HEALTHY state (spike §3.3): raw and bind on the SAME device, no abort token.
const mountinfoHealthy = `748 33 8:16 / /mnt/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512
755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512
`
// mountinfoStaleBind is the R-117 RETURN state (spike §3.3 / §5.2): the drive came back as /dev/sdc
// (8:32) and the raw mount healed onto it, while the bind still names /dev/sdb (8:16) and carries
// `shutdown`. Both pre-R-117 terms read true here.
const mountinfoStaleBind = `755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,shutdown
814 33 8:32 / /mnt/r117sd rw,relatime shared:485 - ext4 /dev/sdc rw,stripe=512
`
// mountinfoAborted is the STEADY-STATE state (spike §9): the device errored in place, so it NEVER LEFT.
// Raw and bind are the SAME device — the device comparison cannot see this — and ext4 has done an
// emergency remount-ro. Today this state emits nothing on any channel.
//
// SECOND substitution, and it is the one that nearly made this test decoration. The spike produced this
// state on a dm device (dm is the only mechanism that can make a device error WITHOUT disappearing), so
// the capture reads 252:11 /dev/mapper/r117cel. Transposed here onto the USB drive shape, because
// RoleForStorage derives role="system" for a /dev/mapper backing device — and a system-role row never
// enters the block that computes BoundUnderParent, so the field stays false by DEFAULT and the assertion
// below passes without term 3 ever running. It did exactly that until RP1 failed to fail (see REPORT.md).
// An in-place abort on a USB drive is the realistic customer case anyway (a link reset that recovers the
// link after ext4 has already given up); only the super options and the matching devnos carry the claim.
const mountinfoAborted = `748 33 8:16 / /mnt/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,emergency_ro
755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,emergency_ro
`
// mountinfoAbortedShutdown is the same in-place shape with the OTHER ext4 abort token, the one a device
// removal sets. Both were measured; a check for only `shutdown` passes mountinfoAborted and a check for
// only `emergency_ro` passes this — which is why abortTokensByFS carries both, and why RP4 exists.
const mountinfoAbortedShutdown = `748 33 8:16 / /mnt/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,shutdown
755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,shutdown
`
// mountinfoUnknownFS is healthy-looking but on a filesystem whose abort vocabulary we have not measured.
// The honest verdict is UNKNOWN — which must be treated as PRESENT, not as live and not as absent.
const mountinfoUnknownFS = `748 33 8:16 / /mnt/r117sd rw,relatime shared:450 - btrfs /dev/sdb rw
755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - btrfs /dev/sdb rw
`
// useMountinfo points the REAL parsers at a fixture for the duration of one test.
func useMountinfo(t *testing.T, body string) {
t.Helper()
p := filepath.Join(t.TempDir(), "mountinfo")
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
prev := procSelfMountinfo
procSelfMountinfo = p
t.Cleanup(func() { procSelfMountinfo = prev })
}
// livenessServer builds a /disks server over one Observe target (or one registry drive) whose raw mount
// is `raw` and stable guest path derives from it. The two PRE-R-117 terms are forced TRUE — that is the
// whole point: they were both true in the measured defect, so term 3 is the only thing that can save us.
func livenessServer(t *testing.T, obs []hub.StorageTarget, known []storage.KnownTarget) *Server {
t.Helper()
opts := Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuestsCfg{},
Backups: &fakeBackups{},
Store: &fakeStore{},
Storage: fakeStorage{targets: obs},
Tokens: staticTokens{"A": 8200},
Disks: &fakeDiskOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4"}},
DiskGate: &fakeGate{},
HostReader: sysOnSDA(),
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
}
if known != nil {
opts.DriveTargets = fakeKnownTargets{drives: known}
}
srv, err := NewServer(opts)
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
// Terms 1 and 2 TRUE — the measured defect's own conditions. livenessCheck is left nil so the real
// bindLiveness runs against the fixture.
srv.boundCheck = func(string) bool { return true }
srv.deviceCheck = func(string) bool { return true }
srv.resolveStorageDevice = func(string) (string, error) { return "/dev/sdb1", nil }
return srv
}
var obsSD = []hub.StorageTarget{
{Name: "sd", Type: hub.StorageTypeUSB, BackingDevice: "/dev/sdb", MountPath: "/mnt/r117sd", State: hub.StorageStateAttached},
}
var knownSD = []storage.KnownTarget{
{Name: "sd", Type: hub.StorageTypeUSB, MountPath: "/mnt/r117sd", DurableID: "uuid:71e1", UUID: "71e1"},
}
// ── Group A — the consequence: a dead namespace reads ABSENT ────────────────────────────────────
// TestDisks_BindLiveness_StaleBindReadsAbsent is R-117 case (a), through the real /disks handler.
// It asserts the CONSEQUENCE — what the controller reads off the wire — not that a comparison happened.
func TestDisks_BindLiveness_StaleBindReadsAbsent(t *testing.T) {
useMountinfo(t, mountinfoStaleBind)
di := diskByMount(t, livenessServer(t, obsSD, nil), "/mnt/r117sd")
if di.BoundUnderParent {
t.Error("BoundUnderParent reports PRESENT over a stale bind (R-117). The bind names 8:16 /dev/sdb " +
"while the raw mount is 8:32 /dev/sdc; every access through it returns EIO. The controller's " +
"gate would take its Return branch (controller intermediary.go:258,:299) and restart the " +
"customer's apps onto a dead namespace, then email backup_target_restored.")
}
}
// TestDisks_BindLiveness_AbortedFilesystemReadsAbsent is R-117a, the steady-state half — and the test a
// devno-only fix would fail. The device NEVER LEFT, so raw and bind agree on 252:11.
func TestDisks_BindLiveness_AbortedFilesystemReadsAbsent(t *testing.T) {
for _, c := range []struct{ name, body string }{
{"emergency_ro (errors=remount-ro fired in place)", mountinfoAborted},
{"shutdown (forced abort)", mountinfoAbortedShutdown},
} {
t.Run(c.name, func(t *testing.T) {
useMountinfo(t, c.body)
// GUARD, earned: assert the row is the shape production emits BEFORE asserting the field.
// A system-role row has no GuestPath, never runs the conjunction, and reports
// BoundUnderParent=false by default — passing this test while proving nothing.
di := diskByMount(t, livenessServer(t, obsSD, nil), "/mnt/r117sd")
if di.Role != "user-data" || di.GuestPath == "" {
t.Fatalf("fixture does not reproduce the production row shape: role=%q guest_path=%q — "+
"the conjunction never runs on such a row, so any assertion below is vacuous",
di.Role, di.GuestPath)
}
if di.BoundUnderParent {
t.Error("BoundUnderParent reports PRESENT over an ABORTED filesystem (R-117a). The devnos " +
"MATCH (the device never disappeared), so the device comparison cannot see this — only " +
"the filesystem's own abort token can. Today this state emits NOTHING on any channel: " +
"the drive is never Disconnected, so the gate produces neither a Stop nor a Return.")
}
})
}
}
// TestDisks_BindLiveness_UnionPath_AbortedReadsAbsent — the union path for the steady-state half. It
// matters more than the Observe one here: this row's Role is hardcoded user-data and its State hardcoded
// attached, so the conjunction is the ONLY thing on the row that can report the abort.
func TestDisks_BindLiveness_UnionPath_AbortedReadsAbsent(t *testing.T) {
useMountinfo(t, mountinfoAborted)
di := diskByMount(t, livenessServer(t, nil, knownSD), "/mnt/r117sd")
if di.BoundUnderParent {
t.Error("union-path drive reports PRESENT over an ABORTED filesystem (R-117a) — and its Role and " +
"State are both hardcoded on this row, so nothing else can contradict it")
}
}
// The union path carries no PVE dir-storage and hardcodes State:"attached", so these terms are the only
// device truth on the row — R-113's reasoning, and it applies to term 3 identically.
func TestDisks_BindLiveness_UnionPath_StaleBindReadsAbsent(t *testing.T) {
useMountinfo(t, mountinfoStaleBind)
di := diskByMount(t, livenessServer(t, nil, knownSD), "/mnt/r117sd")
if di.BoundUnderParent {
t.Error("union-path drive reports PRESENT over a stale bind (R-117) — and its State is hardcoded " +
"attached, so nothing else on the row can contradict it")
}
}
// ── Group B — no false negatives ────────────────────────────────────────────────────────────────
func TestDisks_BindLiveness_HealthyReadsPresent(t *testing.T) {
useMountinfo(t, mountinfoHealthy)
for _, c := range []struct {
name string
obs []hub.StorageTarget
known []storage.KnownTarget
}{
{"observe", obsSD, nil},
{"union", nil, knownSD},
} {
t.Run(c.name, func(t *testing.T) {
di := diskByMount(t, livenessServer(t, c.obs, c.known), "/mnt/r117sd")
if !di.BoundUnderParent {
t.Error("a healthy drive reads ABSENT — a false absent STOPS a working customer's apps, " +
"which is strictly worse than the bug being fixed")
}
})
}
}
// ── Group C — cannot tell must never mean absent ────────────────────────────────────────────────
// TestDisks_BindLiveness_UnknownIsTreatedAsPresent pins the rule in every way it can be reached. The
// workspace's false-invariant table records newestArchiveOn promising exactly this over a signature that
// could not express it; Usable() is the one place it lives, so this is the test that keeps it honest.
func TestDisks_BindLiveness_UnknownIsTreatedAsPresent(t *testing.T) {
t.Run("unreadable mount table", func(t *testing.T) {
prev := procSelfMountinfo
procSelfMountinfo = filepath.Join(t.TempDir(), "does-not-exist")
t.Cleanup(func() { procSelfMountinfo = prev })
if got := bindLiveness("/mnt/felhom-drives/r117sd", "/mnt/r117sd"); got != BindUnknown {
t.Errorf("unreadable /proc gave %v, want BindUnknown", got)
}
di := diskByMount(t, livenessServer(t, obsSD, nil), "/mnt/r117sd")
if !di.BoundUnderParent {
t.Error("an unreadable mount table made the drive read ABSENT — cannot-tell must never stop apps")
}
})
t.Run("no raw mount entry to compare against", func(t *testing.T) {
// Only the bind is in the table. devicePresent is the term that answers device absence; this one
// must abstain rather than double-count it.
useMountinfo(t, `755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw
`)
if got := bindLiveness("/mnt/felhom-drives/r117sd", "/mnt/r117sd"); got != BindUnknown {
t.Errorf("missing raw entry gave %v, want BindUnknown", got)
}
})
t.Run("empty paths", func(t *testing.T) {
if got := bindLiveness("", "/mnt/r117sd"); got != BindUnknown {
t.Errorf("empty stable gave %v, want BindUnknown", got)
}
if got := bindLiveness("/mnt/felhom-drives/r117sd", ""); got != BindUnknown {
t.Errorf("empty raw gave %v, want BindUnknown", got)
}
})
t.Run("filesystem whose abort vocabulary is unmeasured", func(t *testing.T) {
useMountinfo(t, mountinfoUnknownFS)
if got := bindLiveness("/mnt/felhom-drives/r117sd", "/mnt/r117sd"); got != BindUnknown {
t.Errorf("btrfs bind gave %v, want BindUnknown — we cannot read its abort state, so we must "+
"not claim LIVE either", got)
}
di := diskByMount(t, livenessServer(t, obsSD, nil), "/mnt/r117sd")
if !di.BoundUnderParent {
t.Error("an unmeasured filesystem read ABSENT — that would stop apps on every non-ext4 drive")
}
})
t.Run("Usable is the single place the rule lives", func(t *testing.T) {
for _, c := range []struct {
l BindLiveness
want bool
}{
{BindLive, true},
{BindUnknown, true}, // the rule
{BindStaleDevice, false},
{BindAborted, false},
} {
if got := c.l.Usable(); got != c.want {
t.Errorf("%v.Usable() = %v, want %v", c.l, got, c.want)
}
}
})
}
// ── Group D — the verdict itself, including the ordering that matters ───────────────────────────
func TestBindLiveness_Verdicts(t *testing.T) {
for _, c := range []struct {
name, body string
stable, raw string
want BindLiveness
}{
{"healthy", mountinfoHealthy, "/mnt/felhom-drives/r117sd", "/mnt/r117sd", BindLive},
{"stale device (case a)", mountinfoStaleBind, "/mnt/felhom-drives/r117sd", "/mnt/r117sd", BindStaleDevice},
{"aborted in place, emergency_ro (case b)", mountinfoAborted, "/mnt/felhom-drives/r117sd", "/mnt/r117sd", BindAborted},
{"aborted in place, shutdown", mountinfoAbortedShutdown, "/mnt/felhom-drives/r117sd", "/mnt/r117sd", BindAborted},
} {
t.Run(c.name, func(t *testing.T) {
useMountinfo(t, c.body)
if got := bindLiveness(c.stable, c.raw); got != c.want {
t.Errorf("bindLiveness = %v, want %v", got, c.want)
}
})
}
}
// TestBindLiveness_AbortedWins pins the ORDERING, which is load-bearing and not obvious: in the measured
// stale-bind state the filesystem ALSO carries `shutdown`, so both P1 and P2 apply. The verdict must be
// BindAborted-or-BindStaleDevice — never live — but more importantly the in-place state, where ONLY P2
// applies, must not fall through to a devno comparison that reads equal. This test fails if P1 is checked
// before P2 and returns early.
func TestBindLiveness_AbortedWins_WhenDevnosAgree(t *testing.T) {
useMountinfo(t, mountinfoAborted)
got := bindLiveness("/mnt/felhom-drives/r117sd", "/mnt/r117sd")
if got.Usable() {
t.Fatalf("bindLiveness = %v (usable) — the devnos agree because the device never left, so a "+
"P1-first implementation reads this as LIVE and ships R-117's silent half intact", got)
}
if got != BindAborted {
t.Errorf("bindLiveness = %v, want BindAborted (the abort token is the only signal here)", got)
}
}
// ── Group E — the parser, on a real captured table ──────────────────────────────────────────────
// TestHostMountEntries_ParsesDevnoAndSuperOpts pins the field extraction R-117 turned on. The optional
// fields run (shared:NNN master:NNN) is variable-length, so the " - " separator — not a fixed index — is
// what locates fstype and the super options.
func TestHostMountEntries_ParsesDevnoAndSuperOpts(t *testing.T) {
// A guest-side line with BOTH optional-field tags, the longest real shape (spike §5.2).
useMountinfo(t, `759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - ext4 /dev/sdb rw,stripe=512,shutdown
`)
got := hostMountEntries("/mnt/felhom-drives/r117sd")
if len(got) != 1 {
t.Fatalf("got %d entries, want 1", len(got))
}
e := got[0]
if e.Devno != "8:16" {
t.Errorf("Devno = %q, want 8:16 — this is the field R-117 was lost for want of reading", e.Devno)
}
if e.Root != "/felhom-data" {
t.Errorf("Root = %q, want /felhom-data", e.Root)
}
if e.FSType != "ext4" {
t.Errorf("FSType = %q, want ext4 (located via the ' - ' separator, not a fixed index)", e.FSType)
}
if !strings.Contains(e.SuperOpts, "shutdown") {
t.Errorf("SuperOpts = %q, want it to carry `shutdown`", e.SuperOpts)
}
}
// countHostMounts and isHostMountpoint were rewritten onto hostMountEntries; the double-bind convergence
// AttachDrive depends on must survive that (REUSE.md: a boolean could not converge stacked binds).
func TestHostMountEntries_CountsStackedBinds(t *testing.T) {
useMountinfo(t, `755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime - ext4 /dev/sdb rw
756 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime - ext4 /dev/sdb rw
`)
if n := countHostMounts("/mnt/felhom-drives/r117sd"); n != 2 {
t.Errorf("countHostMounts = %d, want 2 — AttachDrive's normalize leg needs the count, not a bool", n)
}
if !isHostMountpoint("/mnt/felhom-drives/r117sd") {
t.Error("isHostMountpoint = false over two stacked binds")
}
if isHostMountpoint("/mnt/nope") {
t.Error("isHostMountpoint = true for a path with no entry")
}
if n := countHostMounts("/mnt/nope"); n != 0 {
t.Errorf("countHostMounts = %d for an absent path, want 0", n)
}
}
+2 -2
View File
@@ -11,9 +11,9 @@ import (
// handleDiskCandidates splits discovery into initialize (all unclaimed) + attach (mountable-FS subset). // handleDiskCandidates splits discovery into initialize (all unclaimed) + attach (mountable-FS subset).
func TestDiskCandidates_Split(t *testing.T) { func TestDiskCandidates_Split(t *testing.T) {
d := &fakeDiskOps{candidates: []storage.CandidateDisk{ d := &fakeDiskOps{candidates: []storage.CandidateDisk{
{Device: "/dev/sdd", SizeBytes: 64 << 30, DataBearing: false}, // blank → initialize only {Device: "/dev/sdd", SizeBytes: 64 << 30, DataBearing: false}, // blank → initialize only
{Device: "/dev/sde", FSType: "ext4", DataBearing: true, Mountable: true, MountSource: "/dev/sde1"}, // FS → init + attach {Device: "/dev/sde", FSType: "ext4", DataBearing: true, Mountable: true, MountSource: "/dev/sde1"}, // FS → init + attach
{Device: "/dev/sdf", FSType: "ntfs", DataBearing: true, Mountable: false}, // ntfs → initialize only {Device: "/dev/sdf", FSType: "ntfs", DataBearing: true, Mountable: false}, // ntfs → initialize only
}} }}
h := newDiskServer(t, d, &fakeGate{}, nil, nil) h := newDiskServer(t, d, &fakeGate{}, nil, nil)
@@ -0,0 +1,186 @@
package localapi
import (
"context"
"encoding/json"
"io"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// R-113 — BoundUnderParent must mean THE DEVICE IS THERE, not "a mount entry with this name exists".
//
// THE BUG THESE PIN. The drive's raw mount at /mnt/<name> is a systemd mount unit bound to its device
// and dies with it. The agent's own bind of <raw>/felhom-data under the shared parent is an ordinary
// bind — nothing ties it to the device — so it OUTLIVES the device as a stale shell. Before v0.114.0
// BoundUnderParent was half 1 only, so a pulled drive kept reporting present, the controller's
// drive-absent gate never produced a Stop action, and NOTHING fired on any channel: not
// backup_target_absent, not the generic storage_disconnected. Measured live in E-2d with the device
// detached — /mnt/mentes2 NOT mounted while /mnt/felhom-drives/mentes2 still read /dev/sdb[/felhom-data]
// (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.2).
//
// RED-PROOF. Drop `&& s.devicePresent(...)` from either construction site in disks.go and
// TestDisks_DevicePresence_ObservePath_DeviceLossReadsAbsent / _UnionPath_... fail with
// "reports present — the bind outlived the device (R-113)".
//
// These drive the REAL production path: NewServer → GET /disks through srv.Handler() → the JSON the
// controller actually parses. The two lowest-level mount reads are injected (a unit test cannot create
// real mounts), but nothing above them is faked, and the wire test below asserts the encoded field.
// presenceServer builds a /disks server over one Observe target and/or one registry drive, with the
// bind and device checks independently controllable — the two conditions whose CONJUNCTION is the fix.
func presenceServer(t *testing.T, obs []hub.StorageTarget, known []storage.KnownTarget, bound, device bool) *Server {
t.Helper()
opts := Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuestsCfg{},
Backups: &fakeBackups{},
Store: &fakeStore{},
Storage: fakeStorage{targets: obs},
Tokens: staticTokens{"A": 8200},
Disks: &fakeDiskOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4"}},
DiskGate: &fakeGate{},
HostReader: sysOnSDA(),
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
}
if known != nil {
opts.DriveTargets = fakeKnownTargets{drives: known}
}
srv, err := NewServer(opts)
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.boundCheck = func(string) bool { return bound }
srv.deviceCheck = func(string) bool { return device }
srv.resolveStorageDevice = func(string) (string, error) { return "/dev/sdb1", nil }
return srv
}
func diskByMount(t *testing.T, srv *Server, mount string) DiskInfo {
t.Helper()
for _, di := range decodeDisks(t, do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes()) {
if di.MountPath == mount {
return di
}
}
t.Fatalf("no disk reported for mount %q", mount)
return DiskInfo{}
}
var obsUSB = []hub.StorageTarget{
{Name: "usb", Type: hub.StorageTypeUSB, BackingDevice: "/dev/sdb1", MountPath: "/mnt/felhom-usb", State: hub.StorageStateAttached},
}
var knownUSB = []storage.KnownTarget{
{Name: "mentes2", Type: hub.StorageTypeUSB, MountPath: "/mnt/mentes2", DurableID: "uuid:9303", UUID: "9303"},
}
// ── Group A — device loss is seen ───────────────────────────────────────────────────────────────
func TestDisks_DevicePresence_ObservePath_DeviceLossReadsAbsent(t *testing.T) {
// The exact E-2d shape: the bind survives (bound=true), the device is gone (device=false).
di := diskByMount(t, presenceServer(t, obsUSB, nil, true, false), "/mnt/felhom-usb")
if di.BoundUnderParent {
t.Error("BoundUnderParent reports present — the bind outlived the device (R-113). " +
"The controller's gate would emit no Stop action, so no alarm can fire.")
}
}
func TestDisks_DevicePresence_UnionPath_DeviceLossReadsAbsent(t *testing.T) {
// The union path matters MORE: a registry drive with no PVE dir-storage hardcodes State:"attached",
// so the raw-mount check is the only device truth the row carries. This is what E-2d detached.
di := diskByMount(t, presenceServer(t, nil, knownUSB, true, false), "/mnt/mentes2")
if di.BoundUnderParent {
t.Error("union-path drive reports present — the bind outlived the device (R-113)")
}
if di.State != hub.StorageStateAttached {
t.Logf("note: union-path State is %q", di.State) // hardcoded; see the OBSERVATION in the report
}
}
// ── Group B — the healthy drive, and the return ─────────────────────────────────────────────────
func TestDisks_DevicePresence_HealthyReadsPresent(t *testing.T) {
for _, c := range []struct {
name string
obs []hub.StorageTarget
known []storage.KnownTarget
mount string
}{
{"observe", obsUSB, nil, "/mnt/felhom-usb"},
{"union", nil, knownUSB, "/mnt/mentes2"},
} {
t.Run(c.name, func(t *testing.T) {
di := diskByMount(t, presenceServer(t, c.obs, c.known, true, true), c.mount)
if !di.BoundUnderParent {
t.Error("a bound drive whose device is present must read PRESENT — " +
"a false absent stops a working customer's apps (Scenario C's failure mode)")
}
})
}
}
// ── Group C — the over-correction guard: boot ordering must not regress ─────────────────────────
func TestDisks_DevicePresence_BootWindowStillReadsAbsent(t *testing.T) {
// Boot ordering: the raw drive mounts EARLY (device=true), the agent binds under the parent ~18s
// LATER (bound=false). Presence must stay FALSE in that window — unchanged from before R-113 — so
// apps stay stopped until the bind is live and the gate's Return branch recreates them.
di := diskByMount(t, presenceServer(t, obsUSB, nil, false, true), "/mnt/felhom-usb")
if di.BoundUnderParent {
t.Error("boot window reports present before the bind landed — this regresses the reboot " +
"convergence the controller's gate comment at intermediary.go:220-224 depends on")
}
}
// ── Group D — unknown must never mean absent ────────────────────────────────────────────────────
func TestDisks_DevicePresence_UnknownIsNotAbsent(t *testing.T) {
// devicePresent has nothing to ask about when there is no raw mount path. It must answer TRUE.
// Absence of a signal is not evidence of absence of a device — and the cost of getting this
// backwards is stopping a healthy customer's apps.
srv := presenceServer(t, nil, nil, true, false)
srv.deviceCheck = nil // exercise the real devicePresent, not the injected fake
if !srv.devicePresent("") {
t.Error("devicePresent(\"\") = false — an unanswerable question was reported as ABSENT")
}
}
// ── The wire contract — what the controller actually parses ─────────────────────────────────────
// TestDisks_DevicePresence_WireFieldIsFalseOnDeviceLoss travels construction → HTTP handler → JSON
// encoding and asserts the ENCODED field, because that is what crosses to the controller. A struct-level
// assertion would not catch the field being dropped from the wire (e.g. an omitempty regression), and
// `bound_under_parent` is the single field the controller's drive-absent gate keys on.
func TestDisks_DevicePresence_WireFieldIsFalseOnDeviceLoss(t *testing.T) {
body := do(t, presenceServer(t, nil, knownUSB, true, false).Handler(), "GET", "/disks", "A", "").Body.Bytes()
if !strings.Contains(string(body), `"bound_under_parent"`) {
t.Fatalf("the wire has no bound_under_parent field at all — the controller's gate reads nothing: %s", body)
}
var wire struct {
Data struct {
Disks []map[string]any `json:"disks"`
} `json:"data"`
}
if err := json.Unmarshal(body, &wire); err != nil {
t.Fatalf("decode /disks: %v", err)
}
var seen bool
for _, d := range wire.Data.Disks {
if d["mount_path"] != "/mnt/mentes2" {
continue
}
seen = true
if v, ok := d["bound_under_parent"].(bool); !ok || v {
t.Errorf("wire bound_under_parent = %v (want false) — the device is gone", d["bound_under_parent"])
}
}
if !seen {
t.Fatalf("the drive never reached the wire: %s", body)
}
}
+139
View File
@@ -0,0 +1,139 @@
package localapi
import (
"context"
"io"
"log/slog"
"net/http"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
func smartIP(v int) *int { return &v }
// TestDisks_SmartSerialized (v0.94.0) — the already-computed SMART summary is copied into the /disks
// payload for a target that has it (Health set), including the SATA counters + temperature; a target
// whose SMART was never read (zero-value summary, Health "") omits the field entirely.
//
// Red-proof: drop the `di.Smart = &sm` copy in handleDisks → the "data" disk's Smart is nil → this fails.
func TestDisks_SmartSerialized(t *testing.T) {
d := &fakeDiskOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4"}}
sv := fakeStorage{targets: []hub.StorageTarget{
{
Name: "data", Type: hub.StorageTypeUSB, BackingDevice: "/dev/sdb1", MountPath: "/mnt/data",
Smart: hub.SmartSummary{
Health: hub.SmartPassed,
TemperatureC: smartIP(34),
ReallocatedSectors: smartIP(3),
PendingSectors: smartIP(0),
},
},
// Zero-value SMART (never read) — Health "" → must be omitted from the payload.
{Name: "nosmart", Type: hub.StorageTypeUSB, BackingDevice: "/dev/sdc1", MountPath: "/mnt/nosmart"},
}}
h := newDiskServer(t, d, &fakeGate{}, sv, nil)
w := do(t, h, "GET", "/disks", "A", "")
if w.Code != http.StatusOK {
t.Fatalf("GET /disks: %d (%s)", w.Code, w.Body.String())
}
byName := map[string]DiskInfo{}
for _, di := range decodeDisks(t, w.Body.Bytes()) {
byName[di.Name] = di
}
ds := byName["data"].Smart
if ds == nil {
t.Fatal("data disk: smart summary was not serialized")
}
if ds.Health != hub.SmartPassed {
t.Errorf("data smart Health = %q, want PASSED", ds.Health)
}
if ds.ReallocatedSectors == nil || *ds.ReallocatedSectors != 3 {
t.Errorf("data ReallocatedSectors = %v, want 3", ds.ReallocatedSectors)
}
if ds.PendingSectors == nil || *ds.PendingSectors != 0 {
t.Errorf("data PendingSectors = %v, want 0 (a real zero, not null)", ds.PendingSectors)
}
if ds.TemperatureC == nil || *ds.TemperatureC != 34 {
t.Errorf("data TemperatureC = %v, want 34", ds.TemperatureC)
}
if byName["nosmart"].Smart != nil {
t.Errorf("nosmart disk: smart must be omitted when Health is empty, got %+v", byName["nosmart"].Smart)
}
}
// ---- Fix B (v0.95.0): the /disks union path reads SMART for registry/USB drives ----
type fakeKnownTargets struct{ drives []storage.KnownTarget }
func (f fakeKnownTargets) Known(context.Context) ([]storage.KnownTarget, error) { return f.drives, nil }
type fakeSmartReader struct {
byDev map[string]hub.SmartSummary
calls []string
}
func (f *fakeSmartReader) SMARTForBacking(_ context.Context, dev string) hub.SmartSummary {
f.calls = append(f.calls, dev)
if s, ok := f.byDev[dev]; ok {
return s
}
return hub.SmartSummary{}
}
func sp(s string) *string { return &s }
// A union-path (registry/USB) drive now gets a real SMART read + model, via the Smart seam — it used
// to ride the enrich-free union path and show "Nincs adat".
// Red-proof: delete the Fix-B block in handleDisks (the s.smart.SMARTForBacking call) → the union
// drive carries no smart and this fails.
func TestDisks_UnionPathReadsSMART(t *testing.T) {
d := &fakeDiskOps{probe: storage.DeviceProbe{Probed: true}}
sm := &fakeSmartReader{byDev: map[string]hub.SmartSummary{
"/dev/sdb1": {Health: hub.SmartPassed, ModelName: sp("TOSHIBA MQ04ABF100")},
}}
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuests{},
Backups: &fakeBackups{},
Store: &fakeStore{},
Storage: fakeStorage{}, // no Observe targets → the union drive is not deduped away
DriveTargets: fakeKnownTargets{drives: []storage.KnownTarget{
{Name: "data-usb", Type: hub.StorageTypeUSB, MountPath: "/mnt/hdd_1", DurableID: "uuid:47a3", UUID: "47a3"},
}},
Smart: sm,
Tokens: staticTokens{"A": 8200},
Disks: d,
HostReader: sysOnSDA(),
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatalf("new server: %v", err)
}
srv.baseCtx = context.Background()
srv.resolveStorageDevice = func(string) (string, error) { return "/dev/sdb1", nil }
disks := decodeDisks(t, do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes())
var usb *DiskInfo
for i := range disks {
if disks[i].Name == "data-usb" {
usb = &disks[i]
}
}
if usb == nil {
t.Fatalf("union drive not in /disks: %+v", disks)
}
if usb.Smart == nil || usb.Smart.Health != hub.SmartPassed {
t.Fatalf("union drive SMART not read: %+v", usb.Smart)
}
if usb.Smart.ModelName == nil || *usb.Smart.ModelName != "TOSHIBA MQ04ABF100" {
t.Errorf("union drive model not carried: %v", usb.Smart)
}
if len(sm.calls) != 1 || sm.calls[0] != "/dev/sdb1" {
t.Errorf("SMART should be read once on /dev/sdb1, got %v", sm.calls)
}
}
+3 -3
View File
@@ -134,9 +134,9 @@ func (s *Server) readMemoryBounds(ctx context.Context, vmid int) (memoryBounds,
return memoryBounds{}, fmt.Errorf("node status: %w", err) return memoryBounds{}, fmt.Errorf("node status: %w", err)
} }
b := memoryBounds{ b := memoryBounds{
allocatedMB: cfg.Memory, // PVE config memory is already MB allocatedMB: cfg.Memory, // PVE config memory is already MB
usageMB: bytesToMBUp(st.Mem), // bytes → MB, rounded up usageMB: bytesToMBUp(st.Mem), // bytes → MB, rounded up
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max) hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
minMB: minGuestMemoryMB, minMB: minGuestMemoryMB,
running: st.Status == "running", running: st.Status == "running",
} }
+270
View File
@@ -0,0 +1,270 @@
package localapi
import (
"context"
"log/slog"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// F-REBOOT (Campaign 8 fault 11) — a guest that should be running and is not.
//
// THE OUTAGE THIS EXISTS TO KILL. A `pct reboot` issued while a vzdump was in flight completed its
// SHUTDOWN half and never issued the start. The guest was found `stopped` with 0 containers, no
// lock, and nothing retrying; it stayed down 9m47s until a human ran `pct start`. The backup itself
// SUCCEEDED — so every alarm the appliance has was silent, because nothing was broken except that
// the customer's entire appliance was off.
//
// WHY THE EXISTING RECOVERY MISSED IT. `RecoverStaleLockedGuests` (stalelock.go) already does
// unlock → delete dangling snapshot → start iff onboot, and it is CORRECT. It missed this by two
// gaps, both narrow:
// - its predicate acts only on a guest holding a stale vzdump lock (`backup`/`snapshot-delete`);
// fault 11's guest was stopped and UNLOCKED, so it returned early;
// - it runs ONCE at agent startup, on the load-bearing invariant that a backup lock present then
// is stale by definition. A guest that goes down while the agent is already up is never
// re-examined.
//
// This watchdog closes exactly those two gaps and nothing more: it is periodic, and it acts on
// "should be running, is not, and is not locked".
//
// ── THE TRAP, WHICH IS THE SAME SHAPE AS F-CRIT-1's ──────────────────────────────────────────
//
// A guest the operator deliberately stopped must NOT be auto-started. Fighting the operator makes
// maintenance impossible and is worse than the outage — the same over-correction that F-CRIT-1's fix
// had to avoid when it stopped whitelisting StateStopped.
//
// The distinction used is `onboot`, and it is deliberately NOT invented here:
// - it is ALREADY the distinction stalelock.go uses for exactly this decision
// (`if onboot && g.Status != "running"`), so the two paths cannot disagree;
// - it is 1 on customer guests and 0 on scratch/golden guests (agent v0.101.0 sets scratch to 0);
// - it is the same flag `pve-guests` itself consults at host boot, so the agent AGREES WITH THE
// PLATFORM rather than maintaining a second, private definition of "should be running".
//
// The hub's desired-state `Run` (internal/desired) is a stronger signal and is wired, but it is
// hub-dependent. `onboot` keeps working on a box that has lost hub contact — which is precisely when
// an unattended appliance most needs to come back up.
const (
// guestPowerInterval is how often the watchdog looks. Matches the guestnet watchdog's cadence so
// the two guest-facing sweeps stay in step, and is far below the 9m47s outage the finding recorded.
guestPowerInterval = 60 * time.Second
// guestPowerMaxAttempts bounds the retry. A guest that will not start must not be started in a
// loop forever (Scenario C) — after this many failures the watchdog stops trying and raises it.
guestPowerMaxAttempts = 3
// guestPowerHeartbeatEvery emits a summary line every Nth sweep. 10 x 60s = 10 minutes, matching
// the controller's deadapp heartbeat.
//
// WHY THIS EXISTS, and it is a correction to this file's OWN first version (v0.107.0): the
// watchdog logged at startup and when it ACTED, and was otherwise silent. A silent watchdog is
// indistinguishable from a dead one — which is F-OBS, the very finding fixed in the same session
// this file shipped in, and it is what standing rule 3 exists to prevent. An operator needs a
// POSITIVE observable that the sweep is running; "no start lines" must not be the only evidence.
guestPowerHeartbeatEvery = 10
)
// guestPowerBackoff is the delay before each retry: 1m, 2m, 4m.
//
// Measured, not picked round: a healthy `pct start` of guest 9201 completed in ~25 s (observed twice
// on 2026-07-28), so even the first 1-minute wait carries 2.4x headroom over a normal start. Three
// attempts bound the disruption at roughly 7 minutes — inside the 9m47s outage this fixes — while
// never becoming an unbounded loop.
var guestPowerBackoff = []time.Duration{time.Minute, 2 * time.Minute, 4 * time.Minute}
// guestPowerState is one guest's recovery attempt record. In-memory on purpose, like the R-88
// breaker: an agent restart re-attempts immediately, which is the cheap direction to fail — a
// forgotten backoff costs one extra start attempt, whereas persisting it could carry a stale
// "this guest won't start" verdict across the restart that fixed it.
type guestPowerState struct {
attempts int
nextAt time.Time
raised bool // the give-up fault has already been raised for this run
}
// WatchGuestPower runs the guest-power sweep every guestPowerInterval until ctx is done. No-op when
// the stale-lock controller is not wired (it supplies the ownership-proven guest list).
func (s *Server) WatchGuestPower(ctx context.Context) {
if s.staleLock == nil {
return
}
s.logger.Info("guest-power: watchdog started", "interval", guestPowerInterval.String(),
"max_attempts", guestPowerMaxAttempts)
t := time.NewTicker(guestPowerInterval)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
s.GuestPowerTick(ctx)
}
}
}
// GuestPowerTick performs one sweep. Exported so a test (and a live check) can drive exactly one
// cycle instead of waiting on the ticker.
func (s *Server) GuestPowerTick(ctx context.Context) {
if s.staleLock == nil {
return
}
guests, err := s.staleLock.Guests(ctx)
if err != nil {
// Unknown ownership ⇒ do nothing. Never fall back to an unfiltered list: starting a
// co-tenant's guest would be worse than leaving ours down.
s.logger.Warn("guest-power: guest list unavailable — skipping sweep (ownership unproven)", "err", err)
return
}
var stopped int
for _, g := range guests {
if ctx.Err() != nil {
return
}
if g.Status != "running" {
stopped++
}
s.recoverOneStoppedGuest(ctx, g)
}
s.guestPowerSweeps++
noteGuestPowerSweep(s.logger, s.guestPowerSweeps, len(guests), stopped)
}
// noteGuestPowerSweep emits the liveness observable every guestPowerHeartbeatEvery sweeps.
//
// It carries WHAT THE SWEEP SAW, not merely that it ran: a line saying "I am alive" cannot
// distinguish "alive, all guests up" from "alive, one guest down and being left alone on purpose",
// and the second is the state an operator needs to see. Pure and separately testable — the mistake
// being corrected here was untestable precisely because it lived inline.
func noteGuestPowerSweep(logger *slog.Logger, sweeps, evaluated, stopped int) {
if logger == nil || sweeps <= 0 || sweeps%guestPowerHeartbeatEvery != 0 {
return
}
logger.Info("guest-power: watchdog alive",
"sweeps_since_boot", sweeps, "guests_evaluated", evaluated, "currently_stopped", stopped)
}
// recoverOneStoppedGuest starts a single guest that should be running and is not.
func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
if g.Status == "running" {
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
return
}
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
if err != nil {
s.logger.Warn("guest-power: read guest config failed — skipping", "vmid", g.VMID, "err", err)
return
}
// SCENARIO B — a deliberately stopped guest is left alone, forever. onboot:0 means the operator
// (or the golden-image provisioning) does not want this guest running.
if !onboot {
return
}
// A locked guest belongs to another operation, mid-flight or stale. The stale-lock recovery owns
// that case and knows how to prove a lock is stale; this watchdog must not race it or start a
// guest whose lock means "a restore is writing my disks right now".
if lock != "" {
s.logger.Info("guest-power: guest is stopped but LOCKED — leaving it to the stale-lock path",
"vmid", g.VMID, "lock", lock)
return
}
// Never start a guest while a vzdump is genuinely in flight for it — a stop-mode backup stops the
// guest ON PURPOSE and starting it underneath would corrupt the backup. Fail safe on doubt.
running, err := s.staleLock.BackupRunning(ctx, g.VMID)
if err != nil {
s.logger.Warn("guest-power: could not confirm no backup is running — NOT starting (fail-safe)",
"vmid", g.VMID, "err", err)
return
}
if running {
s.logger.Info("guest-power: a vzdump is in flight — leaving the guest stopped until it finishes",
"vmid", g.VMID)
return
}
st, due := s.guestPowerDue(g.VMID)
if !due {
return
}
if st.attempts >= guestPowerMaxAttempts {
// SCENARIO C — bounded. Raise it ONCE and stop retrying; an infinite silent retry loop is the
// over-correction here, and a guest that has refused three starts needs a human, not a fourth.
if !st.raised {
s.markGuestPowerRaised(g.VMID)
s.logger.Error("guest-power: GIVING UP — guest should be running (onboot) but failed to start after repeated attempts; it needs operator attention",
"vmid", g.VMID, "attempts", st.attempts, "status", g.Status)
}
return
}
s.logger.Warn("guest-power: guest should be running (onboot) but is stopped and unlocked — starting it",
"vmid", g.VMID, "status", g.Status, "attempt", st.attempts+1, "of", guestPowerMaxAttempts)
if err := s.staleLock.Start(ctx, g.VMID); err != nil {
s.noteGuestPowerFailure(g.VMID)
s.logger.Error("guest-power: start failed", "vmid", g.VMID, "attempt", st.attempts+1, "err", err)
return
}
s.forgetGuestPower(g.VMID)
s.logger.Warn("guest-power: STARTED a guest that should have been running", "vmid", g.VMID)
}
// ---- attempt bookkeeping (guarded by its own mutex; independent of the jobs lock) ----------
var guestPowerMu sync.Mutex
// guestPowerDue reports the guest's attempt state and whether a new attempt is due now.
func (s *Server) guestPowerDue(vmid int) (guestPowerState, bool) {
guestPowerMu.Lock()
defer guestPowerMu.Unlock()
if s.guestPower == nil {
s.guestPower = map[int]guestPowerState{}
}
st := s.guestPower[vmid]
if st.nextAt.IsZero() || !s.now().Before(st.nextAt) {
return st, true
}
return st, false
}
// noteGuestPowerFailure records a failed start and arms the next backoff.
func (s *Server) noteGuestPowerFailure(vmid int) {
guestPowerMu.Lock()
defer guestPowerMu.Unlock()
if s.guestPower == nil {
s.guestPower = map[int]guestPowerState{}
}
st := s.guestPower[vmid]
st.attempts++
i := st.attempts - 1
if i >= len(guestPowerBackoff) {
i = len(guestPowerBackoff) - 1
}
st.nextAt = s.now().Add(guestPowerBackoff[i])
s.guestPower[vmid] = st
}
// markGuestPowerRaised records that the give-up fault has been raised, so it is logged once.
func (s *Server) markGuestPowerRaised(vmid int) {
guestPowerMu.Lock()
defer guestPowerMu.Unlock()
st := s.guestPower[vmid]
st.raised = true
s.guestPower[vmid] = st
}
// forgetGuestPower clears a guest's attempt history — called when it is running again, so a guest
// that recovers does not carry its old failures into the next incident.
func (s *Server) forgetGuestPower(vmid int) {
guestPowerMu.Lock()
defer guestPowerMu.Unlock()
if s.guestPower == nil {
return
}
delete(s.guestPower, vmid)
}
@@ -0,0 +1,120 @@
package localapi
import (
"bytes"
"context"
"errors"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// A CORRECTION TO THIS PACKAGE'S OWN v0.107.0. The guest-power watchdog logged at startup and when it
// ACTED, and was silent otherwise — so on a healthy box the only evidence it was running was the
// absence of start lines, which is equally consistent with the sweep having died. That is F-OBS's
// shape and what standing rule 3 forbids, shipped in the same session F-OBS was fixed.
//
// These tests assert the emitted LINE. Asserting that a function was called would reproduce the
// original mistake, which was invisible precisely because nothing pinned the output.
// RED-PROOF: delete the noteGuestPowerSweep call at the end of GuestPowerTick (or the Info line
// inside it) → this fails with "no liveness observable after 10 sweeps — silence is
// indistinguishable from a dead watchdog".
func TestGuestPowerSweep_EmitsLivenessObservable(t *testing.T) {
var buf bytes.Buffer
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "running"}, {VMID: 9100, Status: "stopped"}},
locks: map[int]string{9201: "", 9100: ""},
onboot: map[int]bool{9201: true, 9100: false}, // 9100 is a deliberate stop
}
s := gpServer(t, ctl, nil)
s.logger = slog.New(slog.NewTextHandler(&buf, nil))
for i := 0; i < guestPowerHeartbeatEvery; i++ {
s.GuestPowerTick(context.Background())
}
out := buf.String()
if !strings.Contains(out, "watchdog alive") {
t.Fatalf("no liveness observable after %d sweeps — silence is indistinguishable from a dead watchdog:\n%s",
guestPowerHeartbeatEvery, out)
}
if !strings.Contains(out, "level=INFO") {
t.Errorf("the observable is not at INFO — a box on the default level would never see it:\n%s", out)
}
// It must carry WHAT IT SAW. "currently_stopped=1" is the operator-relevant fact here: the sweep is
// alive AND is deliberately leaving one guest down, which "I ran" alone cannot express.
for _, want := range []string{"sweeps_since_boot=", "guests_evaluated=2", "currently_stopped=1"} {
if !strings.Contains(out, want) {
t.Errorf("the observable omits %q — it proves the sweep ran but not what it found:\n%s", want, out)
}
}
}
// It must be a summary, not a line per sweep: at 60 s that would be 1440 lines/day, which is the
// pressure that made silence attractive in the first place.
//
// RED-PROOF: change the guard to `sweeps%1 != 0` → this fails with
// "emitted 30 observables across 30 sweeps — that is the flood that made silence attractive".
func TestGuestPowerSweep_IsASummaryNotAFlood(t *testing.T) {
var buf bytes.Buffer
lg := slog.New(slog.NewTextHandler(&buf, nil))
const sweeps = 30
for i := 1; i <= sweeps; i++ {
noteGuestPowerSweep(lg, i, 1, 0)
}
got := strings.Count(buf.String(), "watchdog alive")
want := sweeps / guestPowerHeartbeatEvery
if got == sweeps {
t.Fatalf("emitted %d observables across %d sweeps — that is the flood that made silence attractive", got, sweeps)
}
if got != want {
t.Errorf("emitted %d observables across %d sweeps, want %d", got, sweeps, want)
}
}
// The heartbeat period must stay short enough that a STALLED sweep is obvious well inside the outage
// window this watchdog exists to close (the finding's incident was 9m47s of total appliance
// downtime). If someone widens the cadence to hours the observable stops being a liveness signal.
func TestGuestPowerHeartbeat_StaysUsefulAsALivenessSignal(t *testing.T) {
period := guestPowerHeartbeatEvery * int(guestPowerInterval.Seconds())
if period > 15*60 {
t.Errorf("heartbeat period is %ds (>15min) — too sparse to notice a stalled watchdog", period)
}
if guestPowerHeartbeatEvery < 2 {
t.Errorf("heartbeat every %d sweeps is a per-sweep flood", guestPowerHeartbeatEvery)
}
}
// Off-cadence sweeps stay quiet; a nil logger must not panic (the ticker goroutine has no recovery).
func TestGuestPowerSweep_QuietOffCadenceAndNilSafe(t *testing.T) {
var buf bytes.Buffer
lg := slog.New(slog.NewTextHandler(&buf, nil))
noteGuestPowerSweep(lg, guestPowerHeartbeatEvery-1, 1, 0)
if buf.Len() != 0 {
t.Errorf("emitted off-cadence:\n%s", buf.String())
}
noteGuestPowerSweep(nil, guestPowerHeartbeatEvery, 1, 0) // must not panic
}
// A sweep that ABORTED on unproven ownership must NOT count as a healthy sweep — otherwise the
// heartbeat would report liveness for a watchdog that is examining nothing, which is a worse lie than
// silence.
//
// RED-PROOF: move the s.guestPowerSweeps++ above the Guests() error return → this fails with
// "an aborted sweep was counted as healthy".
func TestGuestPowerSweep_AbortedSweepIsNotCounted(t *testing.T) {
ctl := &fakeGuestPowerCtl{guestsErr: errors.New("pool read failed")}
s := gpServer(t, ctl, nil)
for i := 0; i < guestPowerHeartbeatEvery*2; i++ {
s.GuestPowerTick(context.Background())
}
if s.guestPowerSweeps != 0 {
t.Errorf("an aborted sweep was counted as healthy (sweeps=%d) — the heartbeat would claim liveness for a watchdog examining nothing",
s.guestPowerSweeps)
}
}
+239
View File
@@ -0,0 +1,239 @@
package localapi
import (
"context"
"errors"
"log/slog"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// F-REBOOT (Campaign 8 fault 11): a guest rebooted mid-backup never came back — stopped, unlocked,
// nothing retrying, 9m47s of total appliance outage.
//
// Scenario A (a crashed guest is restarted), B (a deliberately stopped guest is left alone) and
// C (bounded retry, then escalate). B and C are what make A safe.
type fakeGuestPowerCtl struct {
guests []proxmox.Guest
guestsErr error
locks map[int]string // vmid -> lock ("" = unlocked)
onboot map[int]bool
backupRun map[int]bool
backupErr error
started []int
startErr error
}
func (f *fakeGuestPowerCtl) Guests(context.Context) ([]proxmox.Guest, error) {
return f.guests, f.guestsErr
}
func (f *fakeGuestPowerCtl) Lock(_ context.Context, vmid int) (string, bool, error) {
return f.locks[vmid], f.onboot[vmid], nil
}
func (f *fakeGuestPowerCtl) BackupRunning(_ context.Context, vmid int) (bool, error) {
return f.backupRun[vmid], f.backupErr
}
func (f *fakeGuestPowerCtl) HasVzdumpSnapshot(context.Context, int) (bool, error) { return false, nil }
func (f *fakeGuestPowerCtl) Unlock(context.Context, int) error { return nil }
func (f *fakeGuestPowerCtl) DeleteVzdumpSnapshot(context.Context, int) error { return nil }
func (f *fakeGuestPowerCtl) Start(_ context.Context, vmid int) error {
f.started = append(f.started, vmid)
return f.startErr
}
func gpServer(t *testing.T, ctl StaleLockController, now func() time.Time) *Server {
t.Helper()
s := &Server{staleLock: ctl, logger: slog.New(slog.NewTextHandler(discardW{}, nil))}
if now != nil {
s.now = now
} else {
s.now = func() time.Time { return time.Now().UTC() }
}
return s
}
type discardW struct{}
func (discardW) Write(p []byte) (int, error) { return len(p), nil }
// Scenario A — a guest that should be running (onboot) and is stopped-and-unlocked IS started.
//
// RED-PROOF: delete the `s.staleLock.Start(...)` call in recoverOneStoppedGuest (or make the whole
// function return before it) → started is empty and this fails with
// "guest 9201 was NOT started — this is F-REBOOT".
func TestGuestPower_StoppedOnbootGuestIsStarted(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
}
s := gpServer(t, ctl, nil)
s.GuestPowerTick(context.Background())
if len(ctl.started) != 1 || ctl.started[0] != 9201 {
t.Fatalf("guest 9201 was NOT started — this is F-REBOOT (started=%v)", ctl.started)
}
}
// Scenario B — a DELIBERATELY stopped guest (onboot:0) is never started. This is the trap: fighting
// the operator makes maintenance impossible and is worse than the outage being fixed.
//
// RED-PROOF: remove the `if !onboot { return }` guard → the golden/scratch guest is started and this
// fails with "a deliberately stopped guest (onboot:0) was started".
func TestGuestPower_DeliberatelyStoppedGuestIsLeftAlone(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9100, Status: "stopped"}, {VMID: 990000, Status: "stopped"}},
locks: map[int]string{9100: "", 990000: ""},
onboot: map[int]bool{9100: false, 990000: false}, // golden + scratch
}
s := gpServer(t, ctl, nil)
s.GuestPowerTick(context.Background())
if len(ctl.started) != 0 {
t.Errorf("a deliberately stopped guest (onboot:0) was started: %v", ctl.started)
}
}
// A LOCKED stopped guest belongs to the stale-lock path, which knows how to prove a lock is stale.
// This watchdog must not race it.
func TestGuestPower_LockedGuestIsLeftToTheStaleLockPath(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: "snapshot-delete"},
onboot: map[int]bool{9201: true},
}
s := gpServer(t, ctl, nil)
s.GuestPowerTick(context.Background())
if len(ctl.started) != 0 {
t.Errorf("started a LOCKED guest: %v — that races the stale-lock recovery", ctl.started)
}
}
// A guest whose vzdump is genuinely in flight must be left stopped — a stop-mode backup stops the
// guest ON PURPOSE, and starting it underneath would corrupt the backup.
func TestGuestPower_InFlightBackupIsNotDisturbed(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
backupRun: map[int]bool{9201: true},
}
s := gpServer(t, ctl, nil)
s.GuestPowerTick(context.Background())
if len(ctl.started) != 0 {
t.Errorf("started a guest with a vzdump in flight: %v", ctl.started)
}
}
// Fail-safe: if we cannot confirm no backup is running, do NOT start.
func TestGuestPower_UnconfirmableBackupFailsSafe(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
backupErr: errors.New("task list unavailable"),
}
s := gpServer(t, ctl, nil)
s.GuestPowerTick(context.Background())
if len(ctl.started) != 0 {
t.Errorf("started despite being unable to confirm no backup is running: %v", ctl.started)
}
}
// Unknown ownership ⇒ act on nothing. Starting a co-tenant's guest is worse than leaving ours down.
func TestGuestPower_UnprovenOwnershipActsOnNothing(t *testing.T) {
ctl := &fakeGuestPowerCtl{guestsErr: errors.New("pool read failed")}
s := gpServer(t, ctl, nil)
s.GuestPowerTick(context.Background())
if len(ctl.started) != 0 {
t.Errorf("acted with unproven ownership: %v", ctl.started)
}
}
// Scenario C — bounded retry, then escalate. A guest that will not start must NOT be retried forever.
//
// RED-PROOF: remove the `if st.attempts >= guestPowerMaxAttempts` branch → the sweep keeps starting
// on every tick and this fails with "start attempted N times, want at most 3 — infinite retry loop".
func TestGuestPower_RetryIsBoundedThenEscalates(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
startErr: errors.New("cannot start: storage offline"),
}
base := time.Now().UTC()
now := base
s := gpServer(t, ctl, func() time.Time { return now })
// Drive many ticks, advancing well past every backoff each time.
for i := 0; i < 12; i++ {
s.GuestPowerTick(context.Background())
now = now.Add(10 * time.Minute)
}
if len(ctl.started) > guestPowerMaxAttempts {
t.Errorf("start attempted %d times, want at most %d — this is an infinite retry loop",
len(ctl.started), guestPowerMaxAttempts)
}
if len(ctl.started) != guestPowerMaxAttempts {
t.Errorf("start attempted %d times, want exactly %d before giving up", len(ctl.started), guestPowerMaxAttempts)
}
}
// The backoff must actually hold a retry back — otherwise "bounded" is only bounded by luck.
func TestGuestPower_BackoffDefersTheNextAttempt(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
startErr: errors.New("boom"),
}
base := time.Now().UTC()
now := base
s := gpServer(t, ctl, func() time.Time { return now })
s.GuestPowerTick(context.Background()) // attempt 1, arms a 1m backoff
if len(ctl.started) != 1 {
t.Fatalf("precondition: want 1 attempt, got %d", len(ctl.started))
}
now = base.Add(30 * time.Second) // still inside the 1m backoff
s.GuestPowerTick(context.Background())
if len(ctl.started) != 1 {
t.Errorf("retried inside the backoff window (%d attempts) — the bound is not being honoured", len(ctl.started))
}
now = base.Add(90 * time.Second) // past it
s.GuestPowerTick(context.Background())
if len(ctl.started) != 2 {
t.Errorf("did not retry after the backoff lapsed (%d attempts)", len(ctl.started))
}
}
// A guest that comes back healthy must lose its attempt history, so it does not carry old failures
// into the next incident.
func TestGuestPower_RunningGuestClearsAttemptHistory(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
startErr: errors.New("boom"),
}
base := time.Now().UTC()
now := base
s := gpServer(t, ctl, func() time.Time { return now })
s.GuestPowerTick(context.Background())
if _, due := s.guestPowerDue(9201); due {
t.Error("precondition: a backoff should be armed after a failed start")
}
// it comes back up
ctl.guests = []proxmox.Guest{{VMID: 9201, Status: "running"}}
s.GuestPowerTick(context.Background())
if _, due := s.guestPowerDue(9201); !due {
t.Error("attempt history survived the guest coming back healthy")
}
}
+228 -26
View File
@@ -233,7 +233,33 @@ func (b *GuestBinder) AttachDrive(ctx context.Context, vmid int, where string) (
// makes this converge a double-bind to one (the old umount-one+mount-one never did). // makes this converge a double-bind to one (the old umount-one+mount-one never did).
n := countHostMounts(stable) n := countHostMounts(stable)
if n == 1 && b.GuestSeesMount(ctx, vmid, stable) { if n == 1 && b.GuestSeesMount(ctx, vmid, stable) {
return stable, nil // exactly one bind + guest-visible → fully live, no-op // R-117: "one bind + the guest sees it" is NOT liveness. Both of those are path-presence tests, so
// this early return declared a namespace that EIO'd on every call "fully live" and defeated the
// three call sites that already invoke this repair — the 20 s reconcile ticker, agent startup, and
// the controller's Return branch BEFORE it restarts the apps (spike §8.2). The verdict decides:
switch lv := bindLiveness(stable, where); lv {
case BindStaleDevice:
// Case (a). The raw mount has healed onto the returning device; re-binding this stale shell
// onto it REPAIRS the namespace live, with no guest restart (proven, spike §8.1). Fall through
// to the normalize+rebind below. WARN not INFO-per-tick: this fires once, then it is fixed.
b.logger.Warn("guest-attach: bind is STALE — it names a different device than the raw mount; re-binding",
"vmid", vmid, "where", where, "stable", stable, "verdict", lv.String())
case BindAborted:
// Case (b), the Q7 steady-state case. The raw mount is the SAME aborted superblock, so a
// re-bind produces a fresh bind to a still-dead filesystem — and because this runs every 20 s
// it would be an infinite silent retry: exactly the silence Q7 found, with more CPU. Leave the
// mount alone and let the truth travel in BoundUnderParent, which now reads false, so the
// drive gate stops the apps and raises the alarm. Clearing an aborted filesystem needs a
// remount or a fsck — an operator decision, never an automatic one (R-117a).
//
// DEBUG, not WARN: this repeats every tick, and the operator-facing signal is the /disks
// payload plus the customer alarm. Per logging-conventions, INFO is for state changes.
b.logger.Debug("guest-attach: filesystem under the bind has ABORTED — not re-binding (a re-bind cannot clear it); reported not-live instead",
"vmid", vmid, "where", where, "stable", stable, "verdict", lv.String())
return stable, nil
default: // BindLive, BindUnknown — genuinely live, or we cannot tell. Unchanged behaviour.
return stable, nil
}
} }
for i := 0; i < 16 && countHostMounts(stable) > 0; i++ { for i := 0; i < 16 && countHostMounts(stable) > 0; i++ {
if err := b.run(ctx, "umount", stable); err != nil { if err := b.run(ctx, "umount", stable); err != nil {
@@ -250,23 +276,7 @@ func (b *GuestBinder) AttachDrive(ctx context.Context, vmid int, where string) (
// countHostMounts returns how many times `path` appears as a mount target in /proc/self/mountinfo (i.e. // countHostMounts returns how many times `path` appears as a mount target in /proc/self/mountinfo (i.e.
// how many stacked binds are at it). 0 = not mounted; >1 = stacked duplicates. Used to normalize to one. // how many stacked binds are at it). 0 = not mounted; >1 = stacked duplicates. Used to normalize to one.
func countHostMounts(path string) int { func countHostMounts(path string) int { return len(hostMountEntries(path)) }
f, err := os.Open("/proc/self/mountinfo")
if err != nil {
return 0
}
defer f.Close()
n := 0
sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for sc.Scan() {
fields := strings.Fields(sc.Text())
if len(fields) >= 5 && fields[4] == path {
n++
}
}
return n
}
// GuestSeesMount reports whether vmid's guest currently has `path` as a mount target in ITS mount // GuestSeesMount reports whether vmid's guest currently has `path` as a mount target in ITS mount
// namespace (read from /proc/<guest-init-pid>/mountinfo). This is the GUEST-side truth the host-side // namespace (read from /proc/<guest-init-pid>/mountinfo). This is the GUEST-side truth the host-side
@@ -278,7 +288,7 @@ func (b *GuestBinder) GuestSeesMount(ctx context.Context, vmid int, path string)
if pid == "" { if pid == "" {
return false return false
} }
data, err := os.ReadFile("/proc/" + pid + "/mountinfo") data, err := os.ReadFile(procGuestMountinfo(pid))
if err != nil { if err != nil {
return false return false
} }
@@ -391,20 +401,212 @@ func (b *GuestBinder) DetachDrive(ctx context.Context, where string) error {
// isHostMountpoint reports whether path is currently a mount target in the host's mount table // isHostMountpoint reports whether path is currently a mount target in the host's mount table
// (/proc/self/mountinfo). Pure read — used for idempotency (skip re-binding) and the BoundUnderParent // (/proc/self/mountinfo). Pure read — used for idempotency (skip re-binding) and the BoundUnderParent
// report. A read error → false (treat as not-mounted; AttachDrive then (re)binds, which is safe). // report. A read error → false (treat as not-mounted; AttachDrive then (re)binds, which is safe).
func isHostMountpoint(path string) bool { func isHostMountpoint(path string) bool { return len(hostMountEntries(path)) > 0 }
f, err := os.Open("/proc/self/mountinfo")
// procSelfMountinfo is the host mount table every predicate in this file reads. It is a package var
// ONLY so a test can point the REAL parsers at a captured fixture — production never reassigns it, and a
// test that does must restore it (t.Cleanup). Injecting the DATA rather than the verdict is what keeps
// the R-117 tests non-hollow: the parser, the predicate and the /disks handler all run for real.
var procSelfMountinfo = "/proc/self/mountinfo"
// procGuestMountinfo resolves a guest's init PID to its mount-table path. A package var for the same
// single reason as procSelfMountinfo: so a test can point the REAL GuestSeesMount at a captured guest
// mount table. Production never reassigns it.
var procGuestMountinfo = func(pid string) string { return "/proc/" + pid + "/mountinfo" }
// mountEntry is the parsed subset of a mountinfo line the liveness predicate needs. Field numbers are
// the kernel's 1-based numbering (proc(5) "/proc/<pid>/mountinfo"): 3 = major:minor, 4 = root within the
// filesystem, 5 = mount point; after the " - " separator come fstype, source and the per-superblock
// options. Mount points containing spaces are octal-escaped by the kernel, so strings.Fields is safe.
type mountEntry struct {
// Devno is field 3, the backing device as major:minor. THIS is the field R-117 was lost for want of
// reading: it sat in the same parsed slice as the mount point and was discarded.
Devno string
// Root is field 4 — which subtree of the filesystem is mounted (e.g. /felhom-data for our binds).
Root string
// FSType is the filesystem driver, needed to know whether SuperOpts' vocabulary is one we can read.
FSType string
// SuperOpts is the per-superblock option list — where ext4 records that it has stopped serving I/O.
SuperOpts string
}
// hostMountEntries returns every entry in the host mount table whose mount point is `path`. There is
// more than one when binds are stacked (the double-bind case AttachDrive normalizes). A read error
// yields nil — callers treat that as "not mounted"/"cannot tell", never as a positive.
//
// Pure /proc read, NO BLOCK I/O, per CLAUDE.md's health-check rule: a probe that touches a wedged
// device enters uninterruptible sleep and survives SIGKILL (measured, R-117 spike §6.3).
func hostMountEntries(path string) []mountEntry {
f, err := os.Open(procSelfMountinfo)
if err != nil { if err != nil {
return false return nil
} }
defer f.Close() defer f.Close()
var out []mountEntry
sc := bufio.NewScanner(f) sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024) sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for sc.Scan() { for sc.Scan() {
// mountinfo field 5 (0-indexed 4) is the mount point.
fields := strings.Fields(sc.Text()) fields := strings.Fields(sc.Text())
if len(fields) >= 5 && fields[4] == path { if len(fields) < 5 || fields[4] != path {
return true continue
}
e := mountEntry{Devno: fields[2], Root: fields[3]}
// The optional-fields run is variable-length; the " - " separator terminates it.
for i := 5; i < len(fields); i++ {
if fields[i] != "-" {
continue
}
if len(fields) > i+1 {
e.FSType = fields[i+1]
}
if len(fields) > i+3 {
e.SuperOpts = fields[i+3]
}
break
}
out = append(out, e)
}
return out
}
// BindLiveness is the THREE-state answer to "is the bind at the stable path actually usable?".
//
// Three states and not a bool, deliberately. The R-117 fix must be able to say "cannot tell", and the
// cost of getting that wrong is asymmetric: reporting a live drive absent STOPS a customer's apps. The
// workspace's false-invariant table records `newestArchiveOn` promising "errors degrade to unknown,
// never to no-backup" over a (value, bool) signature that made it unrepresentable — the comment was a
// wish. Read every verdict through Usable() and no caller can repeat that.
type BindLiveness int
const (
// BindUnknown — liveness could not be established (unreadable /proc, no raw mount to compare
// against, or a filesystem whose abort vocabulary we have not measured). TREATED AS PRESENT by
// Usable(), the same rule devicePresent applies to an empty path (disks.go).
BindUnknown BindLiveness = iota
// BindLive — the bind names the same device as the raw mount and its filesystem has not aborted.
BindLive
// BindStaleDevice — R-117 case (a), the detach/return case. The bind still references the superblock
// of the drive that went away, while the raw mount has healed onto the returning device via its
// fs-UUID-keyed unit. Every access through the bind fails. RE-BINDING REPAIRS THIS.
BindStaleDevice
// BindAborted — R-117 case (b), the Q7 steady-state case. The filesystem under the bind has given up:
// ext4 sets `shutdown` when its device vanished, `emergency_ro` when errors=remount-ro fired in place.
// The raw mount is the SAME aborted superblock, so RE-BINDING CANNOT REPAIR THIS — it must surface as
// not-live so the drive gate stops the apps and alarms. See AttachDrive's switch.
BindAborted
)
// Usable is the ONLY sanctioned way to turn a verdict into a yes/no, so the unknown-is-present rule
// lives in exactly one place. Pinned by TestBindLiveness_UnknownIsTreatedAsPresent.
func (l BindLiveness) Usable() bool { return l == BindLive || l == BindUnknown }
func (l BindLiveness) String() string {
switch l {
case BindLive:
return "live"
case BindStaleDevice:
return "stale-device"
case BindAborted:
return "filesystem-aborted"
default:
return "unknown"
}
}
// abortTokensByFS maps a filesystem driver to the per-superblock option tokens it sets when it has
// stopped serving I/O. BOTH ext4 tokens are load-bearing and BOTH were measured (R-117 spike §4):
// `shutdown` when the device was removed, `emergency_ro` when errors=remount-ro fired with the device
// still present. A check for only `shutdown` passes the ENTIRE Q7 state, which is the silent half.
//
// ext2/ext3 are served by the ext4 driver on this kernel, so they emit the same tokens. Anything else is
// a customer-supplied filesystem whose vocabulary we have not measured — it yields UNKNOWN, never LIVE
// (the agent itself only ever formats ext4).
var abortTokensByFS = map[string][]string{
"ext4": {"shutdown", "emergency_ro"},
"ext3": {"shutdown", "emergency_ro"},
"ext2": {"shutdown", "emergency_ro"},
}
// fsAborted reports whether the entry's filesystem has aborted, and whether we could tell at all.
// `known` false means the fstype is not in abortTokensByFS — the caller must degrade to BindUnknown
// rather than infer health from the absence of a token it does not know how to look for.
func fsAborted(e mountEntry) (aborted, known bool) {
toks, ok := abortTokensByFS[e.FSType]
if !ok {
return false, false
}
for _, opt := range strings.Split(e.SuperOpts, ",") {
for _, t := range toks {
if opt == t {
return true, true
}
} }
} }
return false return false, true
}
// bindLiveness is the R-117 predicate: does the bind at `stable` actually work? `raw` is the drive's RAW
// host mount (/mnt/<name>). Reads /proc only — NO block I/O, per CLAUDE.md's health-check rule.
//
// WHY THIS EXISTS. GuestSeesMount and isHostMountpoint both compare only field 5 (the mount point) of a
// mountinfo line, so both answer "does a mount by that name exist" and neither can see that the bind and
// the raw mount name DIFFERENT devices. Measured live: raw on 8:32 /dev/sdc while the bind read
// 8:16 /dev/sdb with `shutdown`, BoundUnderParent true, EIO on every read and write, and the gate
// restarting the customer's apps onto it (R-117 spike §5.2).
//
// HOST-SIDE ONLY, deliberately: the host bind and the guest's view of it are the same mount in one
// propagation peer group and carry identical devno and super options (measured, spike §5.2), so this
// needs no lxc-info fork. Guest VISIBILITY is a different question and stays with GuestSeesMount.
func bindLiveness(stable, raw string) BindLiveness {
if stable == "" || raw == "" {
return BindUnknown // nothing to compare — never claim absent
}
binds := hostMountEntries(stable)
if len(binds) == 0 {
return BindUnknown // nothing bound here; that is isHostMountpoint's question, not this one
}
rawEntries := hostMountEntries(raw)
if len(rawEntries) == 0 {
return BindUnknown // the raw mount is gone — devicePresent already reports that as absent
}
// The two cases are distinguished by WHETHER THE DEVICES AGREE, and the abort flag is read off a
// DIFFERENT entry in each. Getting this backwards is a live trap, caught here by
// TestBindLiveness_Verdicts: in the real return state the stale bind carries `shutdown` AND names a
// different device, so an abort-first rule classifies it BindAborted — which reports correctly but
// refuses the re-bind that actually repairs it. The question a verdict must answer for AttachDrive is
// not "is something aborted" but "would a re-bind help".
rawEntry := rawEntries[0]
for _, b := range binds {
if b.Devno != rawEntry.Devno {
// P1 — case (a). The bind references a superblock that is NOT the one the raw mount now has:
// the drive went away and came back, and the raw mount healed onto it via its fs-UUID-keyed
// unit. Sound rather than heuristic — a stale bind pins the dead superblock, which keeps the
// old device index allocated, which FORCES the returning device onto a different number
// (measured both ways, including the control test where releasing the bind let the letter be
// reused, spike §3.4).
//
// Whether a re-bind repairs it depends on the RAW mount, which is what a re-bind would point
// at — not on the stale bind's own abort flag.
if aborted, known := fsAborted(rawEntry); known && aborted {
return BindAborted // re-binding would land on another dead filesystem
}
return BindStaleDevice // re-binding lands on the healthy returning device: repairable
}
}
// Same superblock on both sides, so a re-bind is a no-op by construction. Only the filesystem's own
// abort state can tell us anything — and this is R-117's steady-state half (spike §9), where the
// device NEVER LEFT so the devnos above agree and P1 alone reads healthy.
for _, b := range binds {
if aborted, known := fsAborted(b); known && aborted {
return BindAborted
}
}
// Devices agree and nothing aborted. If we cannot read this filesystem's abort vocabulary we must not
// call it live — say unknown, which Usable() treats as present.
for _, b := range binds {
if _, known := fsAborted(b); !known {
return BindUnknown
}
}
return BindLive
} }
@@ -0,0 +1,187 @@
package localapi
import (
"context"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
)
// R-117 §2.2 — WHAT AttachDrive DOES with each verdict, which is a RULING and not a detail.
//
// The two dead states have DIFFERENT repairs, and the existing self-heal only fits one:
//
// - STALE DEVICE (the detach/return case). The raw mount has healed onto the returning device via its
// fs-UUID-keyed unit, so umount + re-bind lands the namespace on a HEALTHY superblock. Repair is
// correct, and it happens live with no guest restart (proven on hardware, spike §8.1). It MUST run:
// three call sites already invoke it — the 20 s reconcile ticker, agent startup, and the controller's
// Return branch BEFORE it restarts the apps (spike §8.2) — and before v0.117.0 all three were
// short-circuited by `if n == 1 && GuestSeesMount(...)` declaring the dead namespace "fully live".
//
// - ABORTED FILESYSTEM (the steady-state case, R-117a). The raw mount is the SAME aborted superblock,
// so a re-bind produces a fresh bind to a still-dead filesystem. It MUST NOT run: AttachDrive is
// called every 20 s, so re-binding here is an infinite silent retry — the exact silence R-117a
// found, with more CPU — and it would mask the state instead of surfacing it. The truth travels in
// BoundUnderParent (now false), so the drive gate stops the apps and alarms. Clearing an aborted
// filesystem needs a remount or a fsck; that is an operator decision, never an automatic one.
//
// These tests assert the CONSEQUENCE — which privileged commands were issued — not that a verdict was
// computed. RED-PROOF: make the BindAborted arm fall through to the re-bind instead of returning, and
// TestAttachDrive_AbortedFilesystem_DoesNotRebind fails on the recorded umount/mount calls.
// attachRecorder is a proxmox.Runner that records every privileged call and answers `lxc-info -p` with a
// fixed PID, so the REAL GuestSeesMount runs against a captured guest mount table.
type attachRecorder struct {
calls [][]string
pid string
}
func (r *attachRecorder) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
if name == "lxc-info" {
return []byte(r.pid + "\n"), nil, nil
}
return nil, nil, nil
}
func (r *attachRecorder) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
// mountOps returns just the mount-table-mutating calls — the ones that constitute "a repair ran".
func (r *attachRecorder) mountOps() []string {
var out []string
for _, c := range r.calls {
switch c[0] {
case "umount", "mount":
out = append(out, strings.Join(c, " "))
}
}
return out
}
// attachFixture points BOTH mount tables at fixtures: the host one (procSelfMountinfo, which
// countHostMounts and bindLiveness read) and the guest one (procGuestMountinfo, which the REAL
// GuestSeesMount reads). Only the data is injected — every predicate runs for real.
func attachFixture(t *testing.T, hostBody, guestBody string) *attachRecorder {
t.Helper()
dir := t.TempDir()
hp := filepath.Join(dir, "host-mountinfo")
gp := filepath.Join(dir, "guest-mountinfo")
if err := os.WriteFile(hp, []byte(hostBody), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(gp, []byte(guestBody), 0o600); err != nil {
t.Fatal(err)
}
prevHost, prevGuest := procSelfMountinfo, procGuestMountinfo
procSelfMountinfo = hp
procGuestMountinfo = func(string) string { return gp }
t.Cleanup(func() { procSelfMountinfo, procGuestMountinfo = prevHost, prevGuest })
return &attachRecorder{pid: "9301"}
}
// guestSeesStale / guestSeesHealthy / guestSeesAborted are the GUEST-side captures — note `master:450`,
// the slave-of-the-shared-parent tag that proves propagation was wired (spike §3.1). The guest carries the
// same devno and super options as the host bind, because it IS the same mount.
const guestSeesStale = `759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - ext4 /dev/sdb rw,stripe=512,shutdown
`
const guestSeesHealthy = `759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - ext4 /dev/sdb rw,stripe=512
`
const guestSeesAborted = `759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - ext4 /dev/sdb rw,stripe=512,emergency_ro
`
func attachBinder(rec *attachRecorder) *GuestBinder {
return NewGuestBinder(rec, slog.New(slog.NewTextHandler(io.Discard, nil)))
}
// TestAttachDrive_StaleBind_Rebinds is Q6's consequence: the repair that was short-circuited for three
// releases now runs. Exactly the state measured on hardware — bind on 8:16, raw healed onto 8:32.
func TestAttachDrive_StaleBind_Rebinds(t *testing.T) {
rec := attachFixture(t, mountinfoStaleBind, guestSeesStale)
b := attachBinder(rec)
got, err := b.AttachDrive(context.Background(), 9301, "/mnt/r117sd")
if err != nil {
t.Fatalf("AttachDrive: %v", err)
}
if got != "/mnt/felhom-drives/r117sd" {
t.Errorf("stable path = %q", got)
}
ops := rec.mountOps()
if len(ops) == 0 {
t.Fatal("NO repair ran over a stale bind — this is the R-117 short-circuit: `n == 1 && " +
"GuestSeesMount` declared an EIO namespace \"fully live\", so the 20 s ticker, agent startup " +
"and the controller's pre-restart re-attach all did nothing")
}
var sawUmount, sawBind bool
for _, o := range ops {
if strings.HasPrefix(o, "umount /mnt/felhom-drives/r117sd") {
sawUmount = true
}
if o == "mount --bind /mnt/r117sd/felhom-data /mnt/felhom-drives/r117sd" {
sawBind = true
}
}
if !sawUmount || !sawBind {
t.Errorf("repair did not umount-then-rebind; ops=%v", ops)
}
}
// TestAttachDrive_AbortedFilesystem_DoesNotRebind is the §2.2 ruling. A re-bind here cannot repair
// anything (the raw mount is the same aborted superblock) and AttachDrive runs every 20 s, so re-binding
// would be an infinite silent retry that also masks the state.
func TestAttachDrive_AbortedFilesystem_DoesNotRebind(t *testing.T) {
rec := attachFixture(t, mountinfoAborted, guestSeesAborted)
b := attachBinder(rec)
got, err := b.AttachDrive(context.Background(), 9301, "/mnt/r117sd")
if err != nil {
t.Fatalf("AttachDrive returned an error for an aborted filesystem: %v — it must be a quiet no-op; "+
"an error here would log `reconcile: AttachDrive failed` every 20 s", err)
}
if got != "/mnt/felhom-drives/r117sd" {
t.Errorf("stable path = %q", got)
}
if ops := rec.mountOps(); len(ops) != 0 {
t.Errorf("AttachDrive re-bound an ABORTED filesystem: %v\n"+
"A re-bind lands on the SAME dead superblock, and this runs every 20 s — an infinite silent "+
"retry, which is R-117a's silence with more CPU. The state must SURFACE via "+
"BoundUnderParent=false so the gate stops the apps and alarms.", ops)
}
}
// TestAttachDrive_Healthy_IsStillANoOp — the idempotency the reconcile ticker depends on. If this
// regressed, every tick would umount and re-bind a working drive, re-firing propagation into the guest
// 4320 times a day.
func TestAttachDrive_Healthy_IsStillANoOp(t *testing.T) {
rec := attachFixture(t, mountinfoHealthy, guestSeesHealthy)
b := attachBinder(rec)
if _, err := b.AttachDrive(context.Background(), 9301, "/mnt/r117sd"); err != nil {
t.Fatalf("AttachDrive: %v", err)
}
if ops := rec.mountOps(); len(ops) != 0 {
t.Errorf("a healthy bind was disturbed: %v — the 20 s reconcile must stay a no-op", ops)
}
}
// TestAttachDrive_UnknownLiveness_IsANoOp — cannot-tell must not trigger churn either. An unreadable
// mount table making the agent umount and re-bind every 20 s would be a self-inflicted outage.
func TestAttachDrive_UnknownLiveness_IsANoOp(t *testing.T) {
// Host table healthy (so n == 1) but on a filesystem whose abort vocabulary we cannot read.
rec := attachFixture(t, mountinfoUnknownFS,
`759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - btrfs /dev/sdb rw
`)
b := attachBinder(rec)
if _, err := b.AttachDrive(context.Background(), 9301, "/mnt/r117sd"); err != nil {
t.Fatalf("AttachDrive: %v", err)
}
if ops := rec.mountOps(); len(ops) != 0 {
t.Errorf("an UNKNOWN verdict caused a re-bind: %v — cannot-tell must never churn a live mount", ops)
}
}
+5
View File
@@ -73,6 +73,11 @@ func TestDisks_GuestPathAndBoundUnderParent(t *testing.T) {
srv.baseCtx = context.Background() srv.baseCtx = context.Background()
// felhom-usb is bound under the parent; felhom-flash is not. // felhom-usb is bound under the parent; felhom-flash is not.
srv.boundCheck = func(p string) bool { return p == "/mnt/felhom-drives/felhom-usb" } srv.boundCheck = func(p string) bool { return p == "/mnt/felhom-drives/felhom-usb" }
// R-113 (v0.114.0): BoundUnderParent is now `bound && device present`. This test's subject is the
// BIND half, so hold the device half constant at present — otherwise the fixture would be asserting
// a drive that is bound with no raw mount underneath it, which is the absent state, not this test's
// case. Device presence has its own tests (TestDisks_DevicePresence*).
srv.deviceCheck = func(string) bool { return true }
disks := decodeDisks(t, do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes()) disks := decodeDisks(t, do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes())
byMount := map[string]DiskInfo{} byMount := map[string]DiskInfo{}
+6 -6
View File
@@ -41,12 +41,12 @@ type netStorageAddRequest struct {
Name string `json:"name"` Name string `json:"name"`
Protocol string `json:"protocol"` // "nfs" | "smb" Protocol string `json:"protocol"` // "nfs" | "smb"
Server string `json:"server"` Server string `json:"server"`
Export string `json:"export"` // NFS export path | SMB share name Export string `json:"export"` // NFS export path | SMB share name
MappedUID int `json:"mapped_uid"` // container uid (e.g. 1000) MappedUID int `json:"mapped_uid"` // container uid (e.g. 1000)
MappedGID int `json:"mapped_gid"` // container gid MappedGID int `json:"mapped_gid"` // container gid
IdleTimeoutSec int `json:"idle_timeout_sec"` // automount idle-unmount window; 0 → default IdleTimeoutSec int `json:"idle_timeout_sec"` // automount idle-unmount window; 0 → default
Username string `json:"username,omitempty"` // SMB only (secret — written to creds file) Username string `json:"username,omitempty"` // SMB only (secret — written to creds file)
Password string `json:"password,omitempty"` // SMB only (secret — written to creds file) Password string `json:"password,omitempty"` // SMB only (secret — written to creds file)
} }
// handleNetStorageAdd installs a NAS share host-side — verify-before-commit (SPIKE-nas-verify). // handleNetStorageAdd installs a NAS share host-side — verify-before-commit (SPIKE-nas-verify).
+2 -2
View File
@@ -148,8 +148,8 @@ func TestNetVerify_TruthTable(t *testing.T) {
t.Run("ReadDir ok but NOT mounted = FAILED (empty-dir false positive guard)", func(t *testing.T) { t.Run("ReadDir ok but NOT mounted = FAILED (empty-dir false positive guard)", func(t *testing.T) {
n := &fakeNetOps{} n := &fakeNetOps{}
srv := newVerifyServer(t, n, t.TempDir(), verifySeams{ srv := newVerifyServer(t, n, t.TempDir(), verifySeams{
trigger: func(string) error { return nil }, // the read "worked" (empty dir) trigger: func(string) error { return nil }, // the read "worked" (empty dir)
mounted: func(string) bool { return false }, // …but no nfs4/cifs in /proc/mounts mounted: func(string) bool { return false }, // …but no nfs4/cifs in /proc/mounts
journal: func(context.Context, string) (string, error) { return "", nil }, journal: func(context.Context, string) (string, error) { return "", nil },
}) })
h := srv.Handler() h := srv.Handler()
+542 -98
View File
@@ -16,6 +16,7 @@ import (
"sync" "sync"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow" "gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub" "gitea.dooplex.hu/admin/felhom-agent/internal/hub"
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log" applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
@@ -33,6 +34,12 @@ type GuestAPI interface {
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error) WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
} }
// PrivilegedRunner runs a fenced root wrapper. The seam exists so the backup-target move is testable
// without sudo: the wrapper IS the security boundary, so tests substitute it, never bypass it.
type PrivilegedRunner interface {
Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
}
// BackupService enqueues a vzdump/PBS backup of a guest. Satisfied by *backup.BackupRunner. // BackupService enqueues a vzdump/PBS backup of a guest. Satisfied by *backup.BackupRunner.
// BackupWithSnapshotHook (8B.2) invokes onSnapshot once mid-backup when the storage snapshot is // BackupWithSnapshotHook (8B.2) invokes onSnapshot once mid-backup when the storage snapshot is
// taken (snapshot mode only) so the controller can resume its app early; in stop mode it is never // taken (snapshot mode only) so the controller can resume its app early; in stop mode it is never
@@ -41,6 +48,36 @@ type BackupService interface {
BackupWithSnapshotHook(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error) BackupWithSnapshotHook(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error)
} }
// BackupArchiveLister is an OPTIONAL extension to BackupService: "when did a backup last LAND on
// this tier's storage?", answered by the storage rather than by memory. *backup.BackupRunner
// satisfies it.
//
// R-84: the agent's backup Store is in-memory, so after every restart the due-check saw nothing and
// the controller took a redundant backup — a wasted multi-hour WAN upload on the offsite tier after
// every agent deploy. Consulting the storage makes the cold path truthful without persisting
// anything, and it self-corrects: a pruned archive correctly stops counting.
type BackupArchiveLister interface {
NewestArchiveTime(ctx context.Context, vmid int) (time.Time, bool, error)
}
// BackupTier (R-82) binds ONE backup tier's runner to its own policy. The agent builds one per
// resolved config tier; the local API serves each independently so "local daily + PBS weekly" is
// expressible over the wire, not just in config.
//
// COMPATIBILITY CONTRACT (load-bearing — the agent and controller deploy independently):
// the tier whose Primary is true is what EVERY untargeted endpoint acts on. An old controller
// never sends `?target=`, so it sees exactly the pre-R-82 behaviour and response bytes.
type BackupTier struct {
TargetID string
Cadence time.Duration
// WaitTimeout bounds the fire-and-forget backup context. It MUST be >= the runner's own wait
// bound, or the outer context cancels first and the tier reports a false failure while the
// vzdump keeps running (observed live 2026-07-26 with a fixed 2h outer bound).
WaitTimeout time.Duration
Primary bool
Service BackupService
}
// BackupStore records + reads the latest backup/restore-test state. Satisfied by *backup.Store. // BackupStore records + reads the latest backup/restore-test state. Satisfied by *backup.Store.
type BackupStore interface { type BackupStore interface {
RecordBackup(hub.Backup) RecordBackup(hub.Backup)
@@ -54,6 +91,13 @@ type StorageView interface {
Observe(ctx context.Context) ([]hub.StorageTarget, error) Observe(ctx context.Context) ([]hub.StorageTarget, error)
} }
// SmartReader (v0.95.0, Fix B) reads per-disk SMART for the /disks union path so registry/USB drives
// that ride the union (not Observe's enrich) still get a health verdict. A zero-value summary
// (Health "") means "could not read". Satisfied by *storage.SmartReader.
type SmartReader interface {
SMARTForBacking(ctx context.Context, backingDevice string) hub.SmartSummary
}
// TokenAuthority resolves a presented bearer token to its guest VMID. Satisfied by *TokenStore. // TokenAuthority resolves a presented bearer token to its guest VMID. Satisfied by *TokenStore.
type TokenAuthority interface { type TokenAuthority interface {
Lookup(token string) (int, bool) Lookup(token string) (int, bool)
@@ -77,11 +121,22 @@ type Options struct {
// DriveTargets (Impl-2a, optional) yields registry+units-sourced drives for the /disks view, so a // DriveTargets (Impl-2a, optional) yields registry+units-sourced drives for the /disks view, so a
// drive with NO PVE storage still appears. Unioned with Storage.Observe (deduped by mount path). // drive with NO PVE storage still appears. Unioned with Storage.Observe (deduped by mount path).
DriveTargets storage.KnownTargets DriveTargets storage.KnownTargets
Tokens TokenAuthority // Smart (v0.95.0, Fix B) reads per-disk SMART for the /disks UNION path — registry/USB drives ride
// the union (not Observe's enrich), so without this they carry no health verdict. OPTIONAL; nil →
// union rows have no SMART (pre-v0.95.0 behavior). Satisfied by *storage.SmartReader.
Smart SmartReader
Tokens TokenAuthority
// BackupCadence is the per-guest backup interval driving GET /backup/due (slice 8B). A guest // BackupCadence is the per-guest backup interval driving GET /backup/due (slice 8B). A guest
// is "due" when no successful backup is recorded OR the newest one is older than this. 0 → a // is "due" when no successful backup is recorded OR the newest one is older than this. 0 → a
// safe default (24h). The hub-served policy is slice 10; this is the agent-local cadence. // safe default (24h). The hub-served policy is slice 10; this is the agent-local cadence.
// When BackupTiers is supplied this is IGNORED (the primary tier carries its own cadence).
BackupCadence time.Duration BackupCadence time.Duration
// BackupTiers (R-82) is the resolved multi-tier policy, primary first. OPTIONAL: nil → one tier
// synthesized from Backups + BackupCadence, i.e. exactly the pre-R-82 behaviour.
BackupTiers []BackupTier
// InFlight (R-85) is the host-wide one-heavy-operation gate shared with the restore-test
// scheduler. OPTIONAL: nil → no cross-gating (pre-R-85 behaviour). See backup.InFlight.
InFlight *backup.InFlight
// Disk management (slice 8C) — OPTIONAL. When Disks + DiskGate are set, the /disks endpoints // Disk management (slice 8C) — OPTIONAL. When Disks + DiskGate are set, the /disks endpoints
// are served; otherwise they report "not configured". DiskGate authorizes the destructive // are served; otherwise they report "not configured". DiskGate authorizes the destructive
// (data-bearing) format path; Guests lists guests for the eject dependent-warning. // (data-bearing) format path; Guests lists guests for the eject dependent-warning.
@@ -97,6 +152,14 @@ type Options struct {
// NetStorage is the privileged network-mount (NAS) surface (Part A1). OPTIONAL — when nil, the // NetStorage is the privileged network-mount (NAS) surface (Part A1). OPTIONAL — when nil, the
// /netstorage endpoints report "not configured". Satisfied by *storage.SudoHostOps. // /netstorage endpoints report "not configured". Satisfied by *storage.SudoHostOps.
NetStorage NetworkStorageOps NetStorage NetworkStorageOps
// Privileged runs the fenced root wrappers (E-2a: felhom-backup-target-apply). OPTIONAL — when
// nil, POST /backup/target reports "not configured". Satisfied by *proxmox.ExecRunner.
Privileged PrivilegedRunner
// ConfigPath is agent.json, so the backup-target move can repoint the primary tier. "" (env-only
// config) → the move reports it cannot persist rather than pretending it did.
ConfigPath string
// StateDir is where a pre-write recovery copy of agent.json is parked. "" → no copy is parked.
StateDir string
// SmbCredsDir is where the agent writes the 0600 SMB credentials files (out-of-band). "" → // SmbCredsDir is where the agent writes the 0600 SMB credentials files (out-of-band). "" →
// /var/lib/felhom-agent/smb-creds. // /var/lib/felhom-agent/smb-creds.
SmbCredsDir string SmbCredsDir string
@@ -172,36 +235,56 @@ type backupJob struct {
Error string Error string
} }
// backupJobKey identifies one guest's job on ONE tier (R-82).
type backupJobKey struct {
vmid int
target string
}
// Server is the per-guest local API (doc 03 §6). It serves the agent's pinned self-signed leaf // Server is the per-guest local API (doc 03 §6). It serves the agent's pinned self-signed leaf
// and authorizes every request against the token's guest only. // and authorizes every request against the token's guest only.
type Server struct { type Server struct {
addr string addr string
cert tls.Certificate cert tls.Certificate
guests GuestAPI guests GuestAPI
backups BackupService backups BackupService
store BackupStore store BackupStore
storage StorageView storage StorageView
driveTargets storage.KnownTargets // Impl-2a: registry+units drives for /disks (optional) driveTargets storage.KnownTargets // Impl-2a: registry+units drives for /disks (optional)
tokens TokenAuthority smart SmartReader // v0.95.0 Fix B: SMART for the /disks union path (optional)
cadence time.Duration tokens TokenAuthority
logger *slog.Logger cadence time.Duration
now func() time.Time // tiers (R-82) is the resolved backup-tier list, PRIMARY FIRST. Always non-empty: when the
// caller supplies no tiers it holds exactly one, synthesized from Backups+BackupCadence, which
// is the pre-R-82 shape.
tiers []BackupTier
// inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
inFlight *backup.InFlight
logger *slog.Logger
now func() time.Time
disks DiskOps // slice 8C (optional) disks DiskOps // slice 8C (optional)
diskGate StorageGate // slice 8C (optional) diskGate StorageGate // slice 8C (optional)
guestList GuestLister // slice 8C (optional) guestList GuestLister // slice 8C (optional)
guestAttach GuestAttacher // slice 10 P2 (optional) guestAttach GuestAttacher // slice 10 P2 (optional)
mem MemoryOps // v0.90.0 R-24 guest RAM resize (optional) mem MemoryOps // v0.90.0 R-24 guest RAM resize (optional)
memMu sync.Mutex // single-flight around a resize apply (one customer per host) memMu sync.Mutex // single-flight around a resize apply (one customer per host)
netStorage NetworkStorageOps // Part A1: NAS network mounts (optional) netStorage NetworkStorageOps // Part A1: NAS network mounts (optional)
netMountRoot string // the user-data namespace root for the network-mount role gate netMountRoot string // the user-data namespace root for the network-mount role gate
smbCredsDir string // where SMB creds files are written (out-of-band, 0600) smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
intent IntentRecorder // slice 10 P3 (optional) intent IntentRecorder // slice 10 P3 (optional)
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional) guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional) formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
staleLock StaleLockController // F2-b startup stale-lock recovery (optional) staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
host storage.HostReader // role classification source (optional; defaults to ProcHostReader) // guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
guestPower map[int]guestPowerState
// guestPowerSweeps counts completed guest-power sweeps, for the liveness observable. Touched only
// from GuestPowerTick, which the ticker calls serially.
guestPowerSweeps int
host storage.HostReader // role classification source (optional; defaults to ProcHostReader)
hostMetrics HostMetricsProvider // slice 9 (optional) hostMetrics HostMetricsProvider // slice 9 (optional)
hostID string // slice 10B: for the data-bearing-format pending-op hint hostID string // slice 10B: for the data-bearing-format pending-op hint
@@ -212,6 +295,10 @@ type Server struct {
// inline customer-confirmed wipe (durable id → current device, re-derive+match, // inline customer-confirmed wipe (durable id → current device, re-derive+match,
// re-inspect). Defaults to s.reresolveDurableForWipe (real storage funcs); tests // re-inspect). Defaults to s.reresolveDurableForWipe (real storage funcs); tests
// override it to avoid touching real /dev. // override it to avoid touching real /dev.
// resolveStorageDevice maps a durable id (uuid:<fs-uuid>) to its /dev node for the /disks union
// path. Defaults to storage.ResolveStorageDevice (hits /dev/disk/by-*); tests override it.
resolveStorageDevice func(durableID string) (string, error)
reresolveWipe func(ctx context.Context, durableID string) (string, error) reresolveWipe func(ctx context.Context, durableID string) (string, error)
// reresolveBlank is the BLANK-format sibling (audit D3): same anti-retarget // reresolveBlank is the BLANK-format sibling (audit D3): same anti-retarget
@@ -230,8 +317,27 @@ type Server struct {
// Optional — nil defaults to the real host mount-table read (isHostMountpoint); tests inject a fake. // Optional — nil defaults to the real host mount-table read (isHostMountpoint); tests inject a fake.
boundCheck func(string) bool boundCheck func(string) bool
// deviceCheck reports whether a drive's RAW host mount is still mounted — the agent's device-presence
// signal (R-113). Deliberately separate from boundCheck: the raw mount is device-bound (a systemd
// mount unit that dies with its device) while the agent's own bind under the shared parent is NOT,
// so only the raw mount distinguishes "device present" from "the bind outlived the device".
// Optional — nil defaults to isHostMountpoint; tests inject a fake.
deviceCheck func(string) bool
// livenessCheck answers whether the bind at a stable guest path is USABLE, not merely present — the
// third term of the BoundUnderParent conjunction (R-117). Deliberately separate from boundCheck and
// deviceCheck because it is the only one of the three that compares them: boundCheck asks "does the
// guest see a mount by that name", deviceCheck asks "is the raw mount still there", and BOTH are
// satisfied by a bind that names the drive that went away while the raw mount healed onto the
// returning one. Optional — nil defaults to bindLiveness. Prefer redirecting procSelfMountinfo at a
// captured fixture over injecting here: that exercises the real parser and predicate.
livenessCheck func(stable, raw string) BindLiveness
jobsMu sync.Mutex jobsMu sync.Mutex
jobs map[int]*backupJob // per-guest backup job state (slice 8B) // jobs is per-guest-PER-TARGET backup job state (slice 8B; keyed by target too since R-82).
// Keying by vmid alone would let a PBS backup started inside the same quiesce window collide
// with the local one's single-flight and hand the caller the WRONG job id.
jobs map[backupJobKey]*backupJob
// agentic controller update (Phase 1): the swapper + per-guest single-flight gate. // agentic controller update (Phase 1): the swapper + per-guest single-flight gate.
swap *ControllerSwapper swap *ControllerSwapper
@@ -255,15 +361,18 @@ type Server struct {
// holder. R lives ONLY in escrowR (never in the job struct — snapshots must be structurally // holder. R lives ONLY in escrowR (never in the job struct — snapshots must be structurally
// incapable of carrying it) and is zeroed on claim, supersede, or TTL expiry. See // incapable of carrying it) and is zeroed on claim, supersede, or TTL expiry. See
// escrow_ceremony.go for the custody rules. // escrow_ceremony.go for the custody rules.
escrowCeremony *EscrowCeremonyConfig escrowCeremony *EscrowCeremonyConfig
escrowMu sync.Mutex escrowMu sync.Mutex
escrowJob *escrowCeremonyJob escrowJob *escrowCeremonyJob
escrowR []byte escrowR []byte
escrowRClaimed bool escrowRClaimed bool
escrowRExpiry time.Time escrowRExpiry time.Time
escrowDone <-chan struct{} // closes when the detached job finishes (tests wait on it) escrowDone <-chan struct{} // closes when the detached job finishes (tests wait on it)
// ceremonyRun executes the fixed-argv sudo self-invocation (tests inject canned JSON). // ceremonyRun executes the fixed-argv sudo self-invocation (tests inject canned JSON).
ceremonyRun ceremonyRunner ceremonyRun ceremonyRunner
privileged PrivilegedRunner
configPath string
stateDir string
// escrowSudoCheck is the preflight's list-mode grant probe (`sudo -n -l -- <argv>`). // escrowSudoCheck is the preflight's list-mode grant probe (`sudo -n -l -- <argv>`).
escrowSudoCheck func(ctx context.Context) error escrowSudoCheck func(ctx context.Context) error
// escrowLookPath resolves a binary on PATH for preflight (tests inject). // escrowLookPath resolves a binary on PATH for preflight (tests inject).
@@ -290,37 +399,50 @@ func NewServer(o Options) (*Server, error) {
cadence = defaultBackupCadence cadence = defaultBackupCadence
} }
s := &Server{ s := &Server{
addr: o.ListenAddr, addr: o.ListenAddr,
cert: o.Cert, cert: o.Cert,
guests: o.Guests, guests: o.Guests,
backups: o.Backups, backups: o.Backups,
store: o.Store, store: o.Store,
storage: o.Storage, storage: o.Storage,
driveTargets: o.DriveTargets, driveTargets: o.DriveTargets,
tokens: o.Tokens, smart: o.Smart,
cadence: cadence, tokens: o.Tokens,
logger: o.Logger, cadence: cadence,
now: func() time.Time { return time.Now().UTC() }, logger: o.Logger,
disks: o.Disks, now: func() time.Time { return time.Now().UTC() },
diskGate: o.DiskGate, disks: o.Disks,
guestList: o.Guests2, diskGate: o.DiskGate,
guestAttach: o.GuestAttach, guestList: o.Guests2,
mem: o.Memory, guestAttach: o.GuestAttach,
netStorage: o.NetStorage, mem: o.Memory,
netMountRoot: storage.NetworkMountRoot, netStorage: o.NetStorage,
privileged: o.Privileged,
configPath: o.ConfigPath,
stateDir: o.StateDir,
netMountRoot: storage.NetworkMountRoot,
smbCredsDir: o.SmbCredsDir, smbCredsDir: o.SmbCredsDir,
escrowStagePath: o.EscrowStagePath, escrowStagePath: o.EscrowStagePath,
intent: o.Intent, intent: o.Intent,
guestBinds: o.GuestBinds, guestBinds: o.GuestBinds,
formatJobs: o.FormatJobs, formatJobs: o.FormatJobs,
staleLock: o.StaleLock, staleLock: o.StaleLock,
host: o.HostReader, host: o.HostReader,
hostMetrics: o.HostMetrics, hostMetrics: o.HostMetrics,
hostID: o.HostID, hostID: o.HostID,
agentVersion: o.AgentVersion, agentVersion: o.AgentVersion,
logRing: o.LogRing, logRing: o.LogRing,
jobs: map[int]*backupJob{}, jobs: map[backupJobKey]*backupJob{},
swapInFlight: map[int]bool{}, swapInFlight: map[int]bool{},
}
// R-82 tier resolution. Options.BackupTiers is authoritative when supplied; otherwise ONE tier
// is synthesized from Backups + BackupCadence — the pre-R-82 shape, so every existing caller
// (and every existing test) keeps working untouched. Exactly one tier is marked primary, and
// the primary is always first, because that is what the untargeted endpoints act on.
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
s.inFlight = o.InFlight
if s.backups == nil && len(s.tiers) > 0 {
s.backups = s.tiers[0].Service
} }
if s.escrowStagePath == "" { if s.escrowStagePath == "" {
s.escrowStagePath = escrow.StagedResticPasswordPath() s.escrowStagePath = escrow.StagedResticPasswordPath()
@@ -328,6 +450,7 @@ func NewServer(o Options) (*Server, error) {
s.reresolveWipe = s.reresolveDurableForWipe s.reresolveWipe = s.reresolveDurableForWipe
s.reresolveBlank = s.reresolveDurableForBlankFormat s.reresolveBlank = s.reresolveDurableForBlankFormat
s.deviceDurableID = storage.DeviceDurableID s.deviceDurableID = storage.DeviceDurableID
s.resolveStorageDevice = storage.ResolveStorageDevice
s.netTrigger = triggerNetMount s.netTrigger = triggerNetMount
s.netMounted = storage.NetworkMountedAt s.netMounted = storage.NetworkMountedAt
s.netJournal = readUnitJournal s.netJournal = readUnitJournal
@@ -354,6 +477,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /rollback", s.withGuest(s.handleRollback)) mux.HandleFunc("POST /rollback", s.withGuest(s.handleRollback))
mux.HandleFunc("POST /backup", s.withGuest(s.handleBackup)) mux.HandleFunc("POST /backup", s.withGuest(s.handleBackup))
mux.HandleFunc("GET /backup/due", s.withGuest(s.handleBackupDue)) mux.HandleFunc("GET /backup/due", s.withGuest(s.handleBackupDue))
mux.HandleFunc("GET /backup/tiers", s.withGuest(s.handleBackupTiers))
mux.HandleFunc("POST /backup/target", s.withGuest(s.handleSetBackupTarget))
mux.HandleFunc("GET /backup/status", s.withGuest(s.handleBackupStatus)) mux.HandleFunc("GET /backup/status", s.withGuest(s.handleBackupStatus))
mux.HandleFunc("GET /restore-test/status", s.withGuest(s.handleRestoreTestStatus)) mux.HandleFunc("GET /restore-test/status", s.withGuest(s.handleRestoreTestStatus))
// Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring // Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring
@@ -622,6 +747,8 @@ type BackupResponse struct {
VMID int `json:"vmid"` VMID int `json:"vmid"`
JobID string `json:"job_id"` JobID string `json:"job_id"`
Phase string `json:"phase"` Phase string `json:"phase"`
// Target (R-82) echoes the tier; empty + omitted for an untargeted request (pre-R-82 bytes).
Target string `json:"target,omitempty"`
} }
func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int) { func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int) {
@@ -635,17 +762,63 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
return return
} }
} }
// Single-flight per guest: if a backup is already running for this guest, return that job tier, echo, ok := s.tierFromRequest(w, r)
// (don't start a second concurrent vzdump). The controller polls /backup/status on it. if !ok {
s.jobsMu.Lock()
if cur := s.jobs[vmid]; cur != nil && cur.Phase == PhaseRunning {
job := *cur
s.jobsMu.Unlock()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase}, "")
return return
} }
key := backupJobKey{vmid: vmid, target: tier.TargetID}
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
// could not succeed anyway — but without this guard it would be ATTEMPTED, fail on the lock, and
// record a spurious failure that leaves the tier permanently due.
//
// Two distinct cases, deliberately answered differently:
// - SAME tier already in flight → return THAT job (202). Idempotent: the caller re-polls it.
// - DIFFERENT tier in flight → 409. Not a new job, and NOT the other tier's job either —
// handing back a foreign job id is how a caller comes to believe its own backup ran.
//
// "In flight" includes `snapshotted`, not just `running`: after the storage snapshot the vzdump
// is still uploading and still holding the lock. Checking only `running` (the pre-R-82 code)
// left a window where a second POST would start a real second vzdump.
s.jobsMu.Lock()
if cur := s.jobs[key]; cur != nil && backupInFlight(cur.Phase) {
job := *cur
s.jobsMu.Unlock()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase, Target: echo}, "")
return
}
if busyTarget, busyJob, busy := s.otherTierInFlight(vmid, tier.TargetID); busy {
s.jobsMu.Unlock()
s.logger.Info("local-api: backup refused — another tier is still in flight",
"vmid", vmid, "requested_target", tier.TargetID, "busy_target", busyTarget, "busy_job", busyJob)
writeStatus(w, http.StatusConflict, false, nil,
"a backup is already in flight on target "+busyTarget+" (job "+busyJob+") — only one backup runs at a time per guest")
return
}
// Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers
// started inside the same nanosecond (the weekly both-due night, or any injected clock) would
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
// pre-R-82 format byte-for-byte — an old controller stores this string and polls with it — so
// only the additive tiers carry the target segment.
// R-85 Scenario F: a backup and a restore-test must never run together — both move multi-GB over
// the same tunnel. Acquired here (still holding jobsMu is fine: TryAcquire never blocks) and
// released when the fire-and-forget goroutine finishes.
release, busy, free := s.inFlight.TryAcquire("backup:" + tier.TargetID)
if !free {
s.jobsMu.Unlock()
s.logger.Info("local-api: backup refused — a heavy operation is already in flight",
"vmid", vmid, "requested_target", tier.TargetID, "busy", busy)
writeStatus(w, http.StatusConflict, false, nil,
"a heavy operation is already in flight ("+busy+") — only one runs at a time on this host")
return
}
jobID := "backup-" + strconv.Itoa(vmid) + "-" + strconv.FormatInt(s.now().UnixNano(), 10) jobID := "backup-" + strconv.Itoa(vmid) + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
s.jobs[vmid] = &backupJob{JobID: jobID, Phase: PhaseRunning, StartedAt: s.now()} if !tier.Primary && tier.TargetID != "" {
jobID = "backup-" + strconv.Itoa(vmid) + "-" + tier.TargetID + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
}
s.jobs[key] = &backupJob{JobID: jobID, Phase: PhaseRunning, StartedAt: s.now()}
s.jobsMu.Unlock() s.jobsMu.Unlock()
// Enqueue: a vzdump runs for minutes, so we do not block the request. The backup runs on the // Enqueue: a vzdump runs for minutes, so we do not block the request. The backup runs on the
@@ -658,46 +831,78 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
base = context.Background() base = context.Background()
} }
go func() { go func() {
bctx, cancel := context.WithTimeout(base, 2*time.Hour) defer release() // R-85: free the host-wide gate when this backup finishes, however it ends
// Outer bound = the tier's own wait bound + headroom for the pre/post work around WaitTask.
// A fixed 2h here would silently cap a 6h offsite tier.
outer := tier.WaitTimeout
if outer <= 0 {
outer = 2 * time.Hour
}
bctx, cancel := context.WithTimeout(base, outer+15*time.Minute)
defer cancel() defer cancel()
// 8B.2: flip the job to `snapshotted` when the storage snapshot is taken, so the // 8B.2: flip the job to `snapshotted` when the storage snapshot is taken, so the
// controller resumes its app early (snapshot mode only; in stop mode this never fires). // controller resumes its app early (snapshot mode only; in stop mode this never fires).
b, err := s.backups.BackupWithSnapshotHook(bctx, vmid, func() { s.markSnapshotted(vmid, jobID) }) b, err := tier.Service.BackupWithSnapshotHook(bctx, vmid, func() { s.markSnapshotted(key, jobID) })
if err != nil { if err != nil {
b.VMID = vmid b.VMID = vmid
b.Success = false b.Success = false
if b.Error == "" { if b.Error == "" {
b.Error = err.Error() b.Error = err.Error()
} }
s.logger.Error("local-api: backup job failed", "vmid", vmid, "job", jobID, "err", err) // TargetID is what the hub attributes the record to; a failed run must still say which
// tier failed, and the runner may not have set it on the error path.
if b.TargetID == "" {
b.TargetID = tier.TargetID
}
s.logger.Error("local-api: backup job failed", "vmid", vmid, "target", tier.TargetID, "job", jobID, "err", err)
} else { } else {
s.logger.Info("local-api: backup job complete", "vmid", vmid, "job", jobID, "archive", b.Archive) s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive)
} }
s.store.RecordBackup(b) s.store.RecordBackup(b)
s.finishJob(vmid, jobID, b) s.finishJob(key, jobID, b)
}() }()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "") writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
} }
// backupInFlight reports whether a phase means "this backup still holds the guest".
// `snapshotted` counts: the storage snapshot is taken but the vzdump is still uploading.
func backupInFlight(phase string) bool {
return phase == PhaseRunning || phase == PhaseSnapshotted
}
// otherTierInFlight reports whether a DIFFERENT tier has an in-flight backup for this guest.
// Caller must hold s.jobsMu.
func (s *Server) otherTierInFlight(vmid int, target string) (busyTarget, busyJob string, busy bool) {
for k, j := range s.jobs {
if k.vmid != vmid || k.target == target || j == nil {
continue
}
if backupInFlight(j.Phase) {
return k.target, j.JobID, true
}
}
return "", "", false
}
// markSnapshotted flips the guest's running job to the `snapshotted` phase (8B.2) — only if it is // markSnapshotted flips the guest's running job to the `snapshotted` phase (8B.2) — only if it is
// still the current job and still running (don't regress done/failed, and don't touch a newer job). // still the current job and still running (don't regress done/failed, and don't touch a newer job).
func (s *Server) markSnapshotted(vmid int, jobID string) { func (s *Server) markSnapshotted(key backupJobKey, jobID string) {
s.jobsMu.Lock() s.jobsMu.Lock()
defer s.jobsMu.Unlock() defer s.jobsMu.Unlock()
cur := s.jobs[vmid] cur := s.jobs[key]
if cur == nil || cur.JobID != jobID || cur.Phase != PhaseRunning { if cur == nil || cur.JobID != jobID || cur.Phase != PhaseRunning {
return return
} }
cur.Phase = PhaseSnapshotted cur.Phase = PhaseSnapshotted
s.logger.Info("local-api: backup reached snapshotted (app may resume)", "vmid", vmid, "job", jobID) s.logger.Info("local-api: backup reached snapshotted (app may resume)", "vmid", key.vmid, "target", key.target, "job", jobID)
} }
// finishJob transitions the guest's job to done/failed (only if it is still the current job — a // finishJob transitions the guest's job to done/failed (only if it is still the current job — a
// later job started after a single-flight gap must not be overwritten by an older one's result). // later job started after a single-flight gap must not be overwritten by an older one's result).
func (s *Server) finishJob(vmid int, jobID string, b hub.Backup) { func (s *Server) finishJob(key backupJobKey, jobID string, b hub.Backup) {
s.jobsMu.Lock() s.jobsMu.Lock()
defer s.jobsMu.Unlock() defer s.jobsMu.Unlock()
cur := s.jobs[vmid] cur := s.jobs[key]
if cur == nil || cur.JobID != jobID { if cur == nil || cur.JobID != jobID {
return return
} }
@@ -712,10 +917,10 @@ func (s *Server) finishJob(vmid int, jobID string, b hub.Backup) {
} }
// jobSnapshot returns a copy of the guest's current job (ok=false if none). // jobSnapshot returns a copy of the guest's current job (ok=false if none).
func (s *Server) jobSnapshot(vmid int) (backupJob, bool) { func (s *Server) jobSnapshot(key backupJobKey) (backupJob, bool) {
s.jobsMu.Lock() s.jobsMu.Lock()
defer s.jobsMu.Unlock() defer s.jobsMu.Unlock()
if j := s.jobs[vmid]; j != nil { if j := s.jobs[key]; j != nil {
return *j, true return *j, true
} }
return backupJob{}, false return backupJob{}, false
@@ -725,31 +930,185 @@ func (s *Server) jobSnapshot(vmid int) (backupJob, bool) {
// recorded OR the newest successful one is older than the agent-local cadence. A successful // recorded OR the newest successful one is older than the agent-local cadence. A successful
// POST /backup flips this to false for the window, so the controller won't re-quiesce in a loop. // POST /backup flips this to false for the window, so the controller won't re-quiesce in a loop.
// The hub-served policy is slice 10. // The hub-served policy is slice 10.
// BackupAgeState (R-88 Part 2) says WHY AgeSecs is what it is — the distinction the type system
// could not previously express.
//
// Before this, a storage read ERROR and a genuine never-backed-up both produced a nil AgeSecs with
// the same `Reason`, byte-identical on the wire. The controller therefore fired its window-gate
// safety valve ("no backup yet — never withhold the first one") on an unreadable storage, quiescing
// customer app stacks OUTSIDE the backup window. Absence of a signal, read as a specific value —
// the fourth instance of that class in this codebase.
//
// A STRING enum, not a bool: the zero value must mean "legacy agent, no information", and "" says
// that unambiguously where `false` would silently masquerade as a real answer.
type BackupAgeState string
const (
// AgeStateKnown — AgeSecs is set and meaningful.
AgeStateKnown BackupAgeState = "known"
// AgeStateAbsent — a POSITIVE determination that no backup has ever landed for this tier. This is
// the only state that may fire the controller's safety valve.
AgeStateAbsent BackupAgeState = "absent"
// AgeStateUnknown — the agent could not determine the age (storage unreadable, timestamp
// unparseable). Still DUE (an unreadable storage must never suppress a backup), but the window
// gate must NOT be bypassed on it.
AgeStateUnknown BackupAgeState = "unknown"
)
type BackupDueResponse struct { type BackupDueResponse struct {
VMID int `json:"vmid"` VMID int `json:"vmid"`
Due bool `json:"due"` Due bool `json:"due"`
Reason string `json:"reason"` Reason string `json:"reason"`
AgeSecs *int64 `json:"age_seconds,omitempty"` // age of the newest successful backup; null if none AgeSecs *int64 `json:"age_seconds,omitempty"` // age of the newest successful backup; null if none
// Target (R-82) echoes the tier this verdict is about. EMPTY (and omitted) for an untargeted
// request, which is what keeps the pre-R-82 response bytes identical for old controllers.
Target string `json:"target,omitempty"`
// AgeState (R-88 Part 2) disambiguates a nil AgeSecs. Additive: an OLD controller ignores it and
// behaves exactly as before. An EMPTY value on the wire means the agent is pre-v0.105.0 — the
// controller must treat that as "legacy, no information", never as AgeStateUnknown.
AgeState BackupAgeState `json:"age_state,omitempty"`
} }
// archiveLookup is the three-state result of asking a tier's storage when a backup last landed.
// It exists because the old (time.Time, bool) signature could not distinguish "nothing there" from
// "I could not look" — the doc comment on newestArchiveOn promised that distinction for months while
// the type made it impossible.
type archiveLookup int
const (
archiveFound archiveLookup = iota // a backup exists; the time is valid
archiveAbsent // read succeeded, no backup for this guest on this tier
archiveUnknown // could not read (error, or the service has no lister)
)
func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid int) { func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid int) {
latest := s.latestSuccessfulBackupFor(r.Context(), vmid) tier, echo, ok := s.tierFromRequest(w, r)
if latest == nil {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "no successful backup recorded yet"})
return
}
age, ok := backupAge(latest.StartedAt, s.now())
if !ok { if !ok {
// Unparseable timestamp: fail safe toward "due" so a backup still happens.
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "last backup time unparseable — treating as due"})
return return
} }
// A tier whose TARGET STORAGE does not exist yet is DEFERRED, not due (R-82 Slice D).
//
// A fresh box carries the offsite tier in its installer defaults, but `felhom-pbs` only appears
// when the hub provisions the DR tier (`felhom-pbs-apply`). Reporting "due" in that window would
// have the controller quiesce the apps and fire a vzdump at a storage that does not exist —
// every cadence, until provisioning happens. Deferring keeps the tier silent until it is real,
// and it goes live with NO restart the moment the storage appears.
//
// Fail-safe: a storage-view ERROR does not defer. Unknown must never suppress a backup.
if tier.TargetID != "" && !s.targetStoragePresent(r.Context(), tier.TargetID) {
writeOK(w, BackupDueResponse{VMID: vmid, Due: false,
Reason: "target storage not present yet — tier deferred until it is provisioned", Target: echo})
return
}
// Newest backup for THIS tier: the in-memory record if this process took one, otherwise the
// storage itself (R-84 — see BackupArchiveLister). Whichever is newer wins.
var newest time.Time
var haveNewest bool
var unparseable bool
if latest := s.latestSuccessfulBackupForTarget(r.Context(), vmid, tier.TargetID); latest != nil {
if t, ok2 := backupAge2(latest.StartedAt); ok2 {
newest, haveNewest = t, true
} else {
unparseable = true
}
}
t, lookup := s.newestArchiveOn(r.Context(), tier, vmid)
if lookup == archiveFound && (!haveNewest || t.After(newest)) {
newest, haveNewest = t, true
unparseable = false // ground truth supersedes an unreadable in-memory timestamp
}
if !haveNewest {
// R-88 Part 2: THREE distinct reasons for a nil age, each with its own state. Only ABSENT is a
// positive claim of "never backed up"; only that one may license the controller to bypass its
// backup window. All three stay DUE — an agent that cannot tell must never suppress a backup.
switch {
case unparseable:
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeState: AgeStateUnknown,
Reason: "last backup time unparseable — treating as due", Target: echo})
case lookup == archiveUnknown:
// The storage could not be read AND this process holds no record. Previously this emitted
// "no successful backup recorded yet" — a positive claim built out of two absences, which
// is what fired the window-gate valve during the 2026-07-27 PBS outage.
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeState: AgeStateUnknown,
Reason: "backup storage unreadable and no in-memory record — age UNKNOWN, treating as due", Target: echo})
default:
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeState: AgeStateAbsent,
Reason: "no successful backup recorded yet", Target: echo})
}
return
}
age := s.now().Sub(newest)
ageSecs := int64(age.Seconds()) ageSecs := int64(age.Seconds())
if age >= s.cadence { if age >= tier.Cadence {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "older than cadence", AgeSecs: &ageSecs}) writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeSecs: &ageSecs, AgeState: AgeStateKnown,
Reason: "older than cadence", Target: echo})
return return
} }
writeOK(w, BackupDueResponse{VMID: vmid, Due: false, Reason: "within cadence window", AgeSecs: &ageSecs}) writeOK(w, BackupDueResponse{VMID: vmid, Due: false, AgeSecs: &ageSecs, AgeState: AgeStateKnown,
Reason: "within cadence window", Target: echo})
}
// BackupTiersResponse is GET /backup/tiers (R-82): the tiers this agent serves, primary first.
// A controller that gets 404 here is talking to a PRE-R-82 agent and must fall back to the single
// untargeted tier — that 404 is the designed capability probe.
type BackupTiersResponse struct {
VMID int `json:"vmid"`
Tiers []BackupTierInfo `json:"tiers"`
}
// BackupTierInfo is one tier as advertised to the controller.
type BackupTierInfo struct {
Target string `json:"target"`
CadenceSeconds int64 `json:"cadence_seconds"`
Primary bool `json:"primary"`
}
func (s *Server) handleBackupTiers(w http.ResponseWriter, r *http.Request, vmid int) {
resp := BackupTiersResponse{VMID: vmid, Tiers: make([]BackupTierInfo, 0, len(s.tiers))}
for _, t := range s.tiers {
resp.Tiers = append(resp.Tiers, BackupTierInfo{
Target: t.TargetID,
CadenceSeconds: int64(t.Cadence.Seconds()),
Primary: t.Primary,
})
}
writeOK(w, resp)
}
// tierFromRequest resolves the `?target=` query parameter to a tier.
//
// THE COMPATIBILITY RULE (§4): NO target parameter → the PRIMARY tier, and the echoed target is
// EMPTY so the response marshals byte-identically to pre-R-82 (BackupDueResponse.Target is
// omitempty). An old controller cannot tell this agent from the old one.
//
// An UNKNOWN target is a 400, never a silent fallback to the primary: a controller asking about a
// tier this agent does not serve must find out, not be told about a different tier's freshness.
func (s *Server) tierFromRequest(w http.ResponseWriter, r *http.Request) (BackupTier, string, bool) {
want := strings.TrimSpace(r.URL.Query().Get("target"))
if want == "" {
return s.primaryTier(), "", true
}
for _, t := range s.tiers {
if t.TargetID == want {
return t, t.TargetID, true
}
}
writeStatus(w, http.StatusBadRequest, false, nil, "unknown backup target: "+want)
return BackupTier{}, "", false
}
// primaryTier returns the tier every untargeted endpoint acts on. tiers is never empty (New
// synthesizes one), but this stays defensive: a zero tier would silently disable backups.
func (s *Server) primaryTier() BackupTier {
for _, t := range s.tiers {
if t.Primary {
return t
}
}
if len(s.tiers) > 0 {
return s.tiers[0]
}
return BackupTier{TargetID: "", Cadence: defaultBackupCadence, Service: s.backups}
} }
// BackupStatusResponse is GET /backup/status (slice 8B): the current/last job phase + the latest // BackupStatusResponse is GET /backup/status (slice 8B): the current/last job phase + the latest
@@ -760,11 +1119,20 @@ type BackupStatusResponse struct {
JobID string `json:"job_id,omitempty"` JobID string `json:"job_id,omitempty"`
Error string `json:"error,omitempty"` Error string `json:"error,omitempty"`
Backup *hub.Backup `json:"backup,omitempty"` // latest recorded backup for this guest Backup *hub.Backup `json:"backup,omitempty"` // latest recorded backup for this guest
// Target (R-82) echoes the tier; empty + omitted when untargeted (pre-R-82 bytes).
Target string `json:"target,omitempty"`
} }
func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) { func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) {
resp := BackupStatusResponse{VMID: vmid, Phase: PhaseIdle, Backup: s.latestBackupFor(r.Context(), vmid)} tier, echo, ok0 := s.tierFromRequest(w, r)
if job, ok := s.jobSnapshot(vmid); ok { if !ok0 {
return
}
// Untargeted keeps the pre-R-82 meaning EXACTLY: the primary tier's job, and the newest backup
// across ANY target (echo == "" → pickLatestBackup's match-any path).
resp := BackupStatusResponse{VMID: vmid, Phase: PhaseIdle, Target: echo,
Backup: s.pickLatestBackup(r.Context(), vmid, false, echo)}
if job, ok := s.jobSnapshot(backupJobKey{vmid: vmid, target: tier.TargetID}); ok {
resp.Phase = job.Phase resp.Phase = job.Phase
resp.JobID = job.JobID resp.JobID = job.JobID
resp.Error = job.Error resp.Error = job.Error
@@ -787,21 +1155,34 @@ func (s *Server) handleRestoreTestStatus(w http.ResponseWriter, r *http.Request,
// latestBackupFor returns this guest's most recent backup from the store (nil if none). // latestBackupFor returns this guest's most recent backup from the store (nil if none).
func (s *Server) latestBackupFor(ctx context.Context, vmid int) *hub.Backup { func (s *Server) latestBackupFor(ctx context.Context, vmid int) *hub.Backup {
return s.pickLatestBackup(ctx, vmid, false) return s.pickLatestBackup(ctx, vmid, false, "")
} }
// latestSuccessfulBackupFor returns this guest's most recent SUCCESSFUL backup (nil if none) — // latestSuccessfulBackupFor returns this guest's most recent SUCCESSFUL backup (nil if none) —
// the basis for /backup/due (a failed backup must not satisfy the cadence). // the basis for /backup/due (a failed backup must not satisfy the cadence).
func (s *Server) latestSuccessfulBackupFor(ctx context.Context, vmid int) *hub.Backup { func (s *Server) latestSuccessfulBackupFor(ctx context.Context, vmid int) *hub.Backup {
return s.pickLatestBackup(ctx, vmid, true) return s.pickLatestBackup(ctx, vmid, true, "")
} }
func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly bool) *hub.Backup { // latestSuccessfulBackupForTarget is the R-82 per-tier twin: a tier's due-ness must be judged
// against ITS OWN newest successful backup. The store is already keyed by target, so this is a
// filter, not a data-model change — but WITHOUT it a fresh local backup would satisfy the PBS
// tier's cadence and the DR tier would never run.
func (s *Server) latestSuccessfulBackupForTarget(ctx context.Context, vmid int, target string) *hub.Backup {
return s.pickLatestBackup(ctx, vmid, true, target)
}
// pickLatestBackup returns the newest matching record. target "" matches ANY target (the pre-R-82
// behaviour, kept for the untargeted status endpoint).
func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly bool, target string) *hub.Backup {
var latest *hub.Backup var latest *hub.Backup
for _, b := range s.store.Backups(ctx) { for _, b := range s.store.Backups(ctx) {
if b.VMID != vmid || (successOnly && !b.Success) { if b.VMID != vmid || (successOnly && !b.Success) {
continue continue
} }
if target != "" && b.TargetID != target {
continue
}
bb := b bb := b
if latest == nil || bb.StartedAt > latest.StartedAt { // RFC3339 sorts lexically if latest == nil || bb.StartedAt > latest.StartedAt { // RFC3339 sorts lexically
latest = &bb latest = &bb
@@ -810,6 +1191,47 @@ func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly boo
return latest return latest
} }
// newestArchiveOn asks THIS TIER's storage when a backup last landed (R-84). Errors and
// unsupported services degrade to "unknown", never to "no backup" — an unreadable storage must not
// make the tier look freshly backed up, and it must not suppress a backup either: the caller falls
// back to the in-memory record, whose absence means DUE.
func (s *Server) newestArchiveOn(ctx context.Context, tier BackupTier, vmid int) (time.Time, archiveLookup) {
lister, ok := tier.Service.(BackupArchiveLister)
if !ok {
// NO LISTER = the pre-R-84 world, and it must stay ABSENT — not unknown.
//
// "Unknown" is the tempting answer (we cannot consult storage, so we do not know) and it is
// WRONG here, because it would regress Scenario D: the controller fires its first-backup
// safety valve only on ABSENT, so a genuinely new box on a no-lister build would never take
// its first backup outside the window, and nobody would notice for weeks. A loud bug traded
// for a silent one.
//
// The honest reading: on this path the in-memory record is the ONLY registry that exists, so
// its absence means "no backup recorded" in the only terms available — exactly the claim this
// path has always made. UNKNOWN is reserved for a lister that was asked and could not answer.
return time.Time{}, archiveAbsent
}
t, found, err := lister.NewestArchiveTime(ctx, vmid)
if err != nil {
s.logger.Warn("local-api: could not read the backup storage for the due-check — falling back to the in-memory record",
"vmid", vmid, "target", tier.TargetID, "err", err)
return time.Time{}, archiveUnknown
}
if !found {
return time.Time{}, archiveAbsent
}
return t, archiveFound
}
// backupAge2 parses an RFC3339 backup start time, returning it as a time.
func backupAge2(startedAt string) (time.Time, bool) {
t, err := time.Parse(time.RFC3339, startedAt)
if err != nil {
return time.Time{}, false
}
return t.UTC(), true
}
// backupAge parses an RFC3339 backup start time and returns its age relative to now. // backupAge parses an RFC3339 backup start time and returns its age relative to now.
func backupAge(startedAt string, now time.Time) (time.Duration, bool) { func backupAge(startedAt string, now time.Time) (time.Duration, bool) {
t, err := time.Parse(time.RFC3339, startedAt) t, err := time.Parse(time.RFC3339, startedAt)
@@ -819,6 +1241,28 @@ func backupAge(startedAt string, now time.Time) (time.Duration, bool) {
return now.Sub(t), true return now.Sub(t), true
} }
// targetStoragePresent reports whether a backup target exists on this host RIGHT NOW.
//
// Returns TRUE on a storage-view error: "I could not check" must never be read as "not there", or a
// transient probe failure would silently suppress backups — the absence-is-not-failure rule this
// project keeps relearning (R-80, R-81).
func (s *Server) targetStoragePresent(ctx context.Context, target string) bool {
if s.storage == nil {
return true
}
targets, err := s.storage.Observe(ctx)
if err != nil {
s.logger.Warn("local-api: storage view unavailable for the backup-target presence check — assuming present", "target", target, "err", err)
return true
}
for _, t := range targets {
if t.Name == target {
return true
}
}
return false
}
// classByStorage builds a storage-id → class(fast|slow|"") map from the host storage view. A // classByStorage builds a storage-id → class(fast|slow|"") map from the host storage view. A
// view error is logged and yields an empty map (class falls back to "" — a hint, not load-bearing). // view error is logged and yields an empty map (class falls back to "" — a hint, not load-bearing).
func (s *Server) classByStorage(ctx context.Context) map[string]string { func (s *Server) classByStorage(ctx context.Context) map[string]string {
+4 -2
View File
@@ -55,7 +55,8 @@ func TestLiveReporter_CoordPresentWithoutPriorVerify(t *testing.T) {
// The whole point: a recipe built from the live read carries the pbs coord. // The whole point: a recipe built from the live read carries the pbs coord.
h := hub.BuildDRRecipeHostHalf(nil, h := hub.BuildDRRecipeHostHalf(nil,
[]hub.StorageTarget{{Name: "felhom-pbs", Type: hub.StorageTypePBS, Content: "backup"}}, got) []hub.StorageTarget{{Name: "felhom-pbs", Type: hub.StorageTypePBS, Content: "backup"}}, got,
hub.ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
if h.PBS == nil { if h.PBS == nil {
t.Fatal("pbs coord absent despite a reachable PBS — the gap this fixes") t.Fatal("pbs coord absent despite a reachable PBS — the gap this fixes")
} }
@@ -67,7 +68,8 @@ func TestLiveReporter_CoordPresentWithoutPriorVerify(t *testing.T) {
bare := NewSnapshotStore() bare := NewSnapshotStore()
h2 := hub.BuildDRRecipeHostHalf(nil, h2 := hub.BuildDRRecipeHostHalf(nil,
[]hub.StorageTarget{{Name: "felhom-pbs", Type: hub.StorageTypePBS, Content: "backup"}}, []hub.StorageTarget{{Name: "felhom-pbs", Type: hub.StorageTypePBS, Content: "backup"}},
bare.PBSSnapshots(context.Background())) bare.PBSSnapshots(context.Background()),
hub.ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
if h2.PBS != nil { if h2.PBS != nil {
t.Fatal("companion sanity: the bare store should yield NO pbs coord (proves the live read is load-bearing)") t.Fatal("companion sanity: the bare store should yield NO pbs coord (proves the live read is load-bearing)")
} }
+13
View File
@@ -47,6 +47,16 @@ type RestoreLXCOptions struct {
// 'mpN' to bind mount is only possible for root"); replacing it with a throwaway volume needs // 'mpN' to bind mount is only possible for root"); replacing it with a throwaway volume needs
// no root and the boot-verify doesn't need the drive's data. // no root and the boot-verify doesn't need the drive's data.
MountOverrides map[string]string MountOverrides map[string]string
// ConfigOverrides sets arbitrary guest-config params AT RESTORE TIME (they take precedence over
// the archive's own values), for settings that must hold from the instant the guest exists —
// before any post-restore SetConfig could run.
//
// The restore-test uses it for `onboot=0`. A restore that fails BEFORE the post-restore config
// step leaves a scratch guest carrying the SOURCE guest's config verbatim, including
// `onboot: 1` — so a leaked scratch would auto-start on the next host reboot, with the source's
// MAC, static island IP and hostname. Observed live 2026-07-26. The normal path link-downs every
// NIC before boot, so this is defence in depth for the ABNORMAL path, where the leak happens.
ConfigOverrides map[string]string
} }
// RestoreLXC restores an LXC from a vzdump/PBS archive via POST /nodes/{node}/lxc // RestoreLXC restores an LXC from a vzdump/PBS archive via POST /nodes/{node}/lxc
@@ -71,6 +81,9 @@ func (c *Client) RestoreLXC(ctx context.Context, opts RestoreLXCOptions) (string
for k, val := range opts.MountOverrides { for k, val := range opts.MountOverrides {
v.Set(k, val) // e.g. mp0 -> "local-lvm:1,mp=/data,backup=0" (overrides the archive's mp0) v.Set(k, val) // e.g. mp0 -> "local-lvm:1,mp=/data,backup=0" (overrides the archive's mp0)
} }
for k, val := range opts.ConfigOverrides {
v.Set(k, val) // e.g. onboot -> "0" (a leaked scratch must never auto-start)
}
return c.dataString(ctx, http.MethodPost, "/nodes/"+c.node+"/lxc", v) return c.dataString(ctx, http.MethodPost, "/nodes/"+c.node+"/lxc", v)
} }
+33
View File
@@ -136,6 +136,39 @@ func (p *Privileged) CreateGoldenLXC(ctx context.Context, spec GoldenLXCSpec) er
return p.run(ctx, "pct", args...) return p.run(ctx, "pct", args...)
} }
// DestroyScratchLXC destroys a restore-test scratch guest through the fenced root path, refusing any
// vmid outside the caller-supplied scratch band.
//
// WHY THIS CANNOT BE THE API — and this is the fourth fenced exception, so the reasoning is recorded
// in full. A restore-test whose restore FAILS leaves a scratch guest the API token cannot destroy:
// `FelhomAgentGuest` is granted at /pool/felhom, and a guest joins that pool only when its restore
// COMPLETES. A failed restore therefore leaves a guest that exists, is in no pool, and is out of
// reach (403 VM.Allocate) while holding its disks.
//
// TWO API-SIDE FIXES WERE BUILT AND BOTH REFUTED BY LIVE TEST on 2026-07-28:
// - Adopt the stranded guest into the pool, then retry. `PUT /pools/{pool}` ALSO requires
// VM.Allocate on the VM being added, so pool membership cannot bootstrap its own authority.
// - Grant the role per-path at /vms/990000..990009. Durable for exactly ONE use per slot: PVE's own
// destroy calls AccessControl::remove_vm_access (LXC.pm:906), deleting every ACL at /vms/<vmid>
// (AccessControl.pm:1898). The grant is consumed by the operation it authorises.
//
// The band ACLs are still provisioned (host-install v1.21.0) and the API path is still tried FIRST —
// this is the fallback that makes teardown deterministic rather than once-per-slot.
//
// THE FENCE. The band is enforced in THREE places, deliberately: sudoers matches the vmid literally
// (`pct destroy 99000[0-9] --purge` — even a compromised agent asking for 9201 is refused by sudo
// itself), this method re-checks it before exec, and the caller checks its own journal provenance.
// Unlike an ACL, none of these is consumed by use.
func (p *Privileged) DestroyScratchLXC(ctx context.Context, vmid, bandMin, bandMax int) error {
if bandMin <= 0 || bandMax < bandMin {
return fmt.Errorf("proxmox: DestroyScratchLXC needs a configured scratch band, got [%d,%d]", bandMin, bandMax)
}
if vmid < bandMin || vmid > bandMax {
return fmt.Errorf("proxmox: refusing to destroy vmid %d — outside the scratch band [%d,%d]", vmid, bandMin, bandMax)
}
return p.run(ctx, "pct", "destroy", strconv.Itoa(vmid), "--purge")
}
// MountUSBByUUID mounts a filesystem by UUID at target (creating the mountpoint). // MountUSBByUUID mounts a filesystem by UUID at target (creating the mountpoint).
// //
// WHY THIS CANNOT BE THE API: a physical host mount is not a Proxmox API op; it is // WHY THIS CANNOT BE THE API: a physical host mount is not a Proxmox API op; it is
+101
View File
@@ -0,0 +1,101 @@
package proxmox
import (
"context"
"io"
"strings"
"testing"
)
// F-LEAK (Campaign 8): the fourth root-fenced exception. Its whole justification is that the band is
// enforced rather than assumed, so these tests are about the REFUSALS, not the happy path.
//
// The band is checked in three independent places on purpose: sudoers matches the vmid literally
// (`pct destroy 99000[0-9] --purge`), this method re-checks before exec, and the caller checks journal
// provenance. These tests pin the middle one; the sudoers glob is proven live.
type recordingRunner struct {
calls [][]string
err error
}
func (r *recordingRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
return nil, nil, r.err
}
func (r *recordingRunner) RunStdin(_ context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
return nil, nil, r.err
}
// A vmid inside the band is destroyed, with --purge so no config/ACL/firewall residue survives.
//
// RED-PROOF: drop "--purge" from the args → this fails with "destroy is not --purge", and the live
// sudoers rule (which matches the FULL vector including --purge) would refuse the call outright.
func TestDestroyScratchLXC_InBandDestroysWithPurge(t *testing.T) {
r := &recordingRunner{}
if err := NewPrivileged(r, "").DestroyScratchLXC(context.Background(), 990003, 990000, 990009); err != nil {
t.Fatalf("in-band destroy failed: %v", err)
}
if len(r.calls) != 1 {
t.Fatalf("want exactly 1 exec, got %d: %v", len(r.calls), r.calls)
}
got := strings.Join(r.calls[0], " ")
if got != "pct destroy 990003 --purge" {
t.Errorf("exec vector = %q, want %q (it must match the sudoers rule byte for byte)",
got, "pct destroy 990003 --purge")
}
}
// THE ONE THAT MATTERS. A vmid outside the band must be refused WITHOUT EXECUTING ANYTHING — a real
// customer guest, the golden image, a co-tenant's VM.
//
// RED-PROOF: remove the `vmid < bandMin || vmid > bandMax` check → this fails with
// "REFUSAL FAILED: executed [pct destroy 9201 --purge] for out-of-band vmid 9201".
func TestDestroyScratchLXC_RefusesOutOfBandWithoutExecuting(t *testing.T) {
for _, vmid := range []int{
1, // arbitrary
9201, // the LIVE customer guest on both demo boxes
9100, // golden image
9999, // reserved
989999, // one below the band
990010, // one ABOVE the band — the off-by-one
} {
r := &recordingRunner{}
err := NewPrivileged(r, "").DestroyScratchLXC(context.Background(), vmid, 990000, 990009)
if err == nil {
t.Errorf("vmid %d was NOT refused — the fence is open", vmid)
}
if len(r.calls) != 0 {
t.Errorf("REFUSAL FAILED: executed %v for out-of-band vmid %d", r.calls, vmid)
}
if err != nil && !strings.Contains(err.Error(), "outside the scratch band") {
t.Errorf("vmid %d refused with an unhelpful error: %v", vmid, err)
}
}
}
// An unconfigured or inverted band must refuse everything rather than defaulting to something. A zero
// band is what a mis-wired caller looks like, and "destroy vmid 0" must never become reachable.
//
// RED-PROOF: drop the `bandMin <= 0 || bandMax < bandMin` check → the [0,0] case admits vmid 0 and
// this fails with "an unconfigured band admitted vmid 0".
func TestDestroyScratchLXC_RefusesUnconfiguredBand(t *testing.T) {
cases := []struct{ vmid, min, max int }{
{0, 0, 0}, // nothing configured at all
{990000, 0, 0}, // band absent, real scratch vmid
{990000, 0, 990009}, // min unset
{990005, 990009, 990000}, // inverted
{990000, -1, 990009}, // negative
}
for _, c := range cases {
r := &recordingRunner{}
if err := NewPrivileged(r, "").DestroyScratchLXC(context.Background(), c.vmid, c.min, c.max); err == nil {
t.Errorf("band [%d,%d] admitted vmid %d — an unconfigured band must refuse", c.min, c.max, c.vmid)
}
if len(r.calls) != 0 {
t.Errorf("an unconfigured band admitted vmid %d and EXECUTED %v", c.vmid, r.calls)
}
}
}
+48 -30
View File
@@ -51,10 +51,15 @@ const DefaultDataVolMount = "mp0"
// Single source of truth for both restore sites (provision bring-up + restore-test). // Single source of truth for both restore sites (provision bring-up + restore-test).
const DefaultPool = "felhom" const DefaultPool = "felhom"
// DefaultSysDataMount is the mpN slot the golden bakes the SSD user-data volume (/mnt/sys_drive) at. // DefaultSysDataMount is RETIRED (agent v0.120.0, R-165 / decision D-a). The golden no longer bakes a
// This is the controller's system_data_path; provision grows it (SysDataGrowGB) like the Docker-data // second volume: since build-golden.sh v3.0.0 there is ONE data volume at /var/lib/felhom (mp0) and
// volume. mp1 is the natural next bring-up slot (mp8/mp9 are added by the provision back-half). // both /var/lib/docker and /mnt/sys_drive are binds of subdirectories of it, so there is no mp1 to
const DefaultSysDataMount = "mp1" // resize. The constant is kept, and deliberately points at nothing, so that a stale caller fails
// loudly at review rather than silently resizing a slot that does not exist.
//
// SysDataGrowGB itself is NOT removed — see its field comment: the host installer still passes
// `-sysdata-grow`, and its GiB are FOLDED INTO the single volume's grow rather than dropped.
const DefaultSysDataMount = ""
// Structural host-bind mountpoints every provisioned guest carries (GL-5; verdict of // Structural host-bind mountpoints every provisioned guest carries (GL-5; verdict of
// SPIKE-dr-bindmount-source-2026-07-07): the permanent drives parent bind (mp8, // SPIKE-dr-bindmount-source-2026-07-07): the permanent drives parent bind (mp8,
@@ -183,16 +188,28 @@ type BringUpSpec struct {
DataVolGrowGB int DataVolGrowGB int
// DataVolMount is the mpN slot of the golden's Docker-data volume to grow; "" → DefaultDataVolMount ("mp0"). // DataVolMount is the mpN slot of the golden's Docker-data volume to grow; "" → DefaultDataVolMount ("mp0").
DataVolMount string DataVolMount string
// SysDataGrowGB grows the golden-carried SSD user-data volume (SysDataMount, default mp1, mounted at // SysDataGrowGB is a COMPATIBILITY INPUT since agent v0.120.0 (R-165). There is no longer a second
// /mnt/sys_drive = the controller's system_data_path) to the per-customer target. Same online, // volume to grow — but `felhom.eu/scripts/felhom-host-install.sh` computes and passes
// grow-only mechanism as DataVolGrowGB. 0 = skip (keep the golden's small size — the volume is still // `-sysdata-grow` (its step_grows derives both numbers from the thin pool's free space), and an
// a separate mount, so the controller's "not a separate drive" warning clears regardless of grow). // installer and an agent do not upgrade in the same instant.
//
// SO ITS GiB ARE FOLDED INTO THE SINGLE VOLUME'S GROW RATHER THAN DROPPED. Dropping them would
// silently shrink every appliance by the user-data share — on the ≥300 GiB branch that is 42 of
// 250 GiB — which is exactly the "a knob that silently does nothing" outcome R-165 was told to
// avoid. Folding keeps total capacity identical whichever installer version runs.
SysDataGrowGB int SysDataGrowGB int
// SysDataMount is the mpN slot of the golden's user-data volume to grow; "" → DefaultSysDataMount ("mp1"). // SysDataMount is RETIRED and ignored (see DefaultSysDataMount). Kept so an older caller still
// compiles; it selects nothing.
SysDataMount string SysDataMount string
Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test) Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test)
KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live
BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait
// IslandBridge + IslandGuestAddr (R-50): when BOTH are set, the guest gets a static net1 on the
// host-internal island bridge, so the controller reaches the agent over a fixed private address
// that survives any LAN/DHCP/site move (the F1 fix). Empty (default) = no net1, byte-for-byte the
// pre-R-50 config. Set from cfg.LocalAPI (island_bridge/island_guest_addr) at both call sites.
IslandBridge string // e.g. "vmbr9"
IslandGuestAddr string // guest net1 CIDR, e.g. "169.254.253.2/30"
} }
// BringUpResult is the outcome. It reuses the restore-test's WARNINGS surface // BringUpResult is the outcome. It reuses the restore-test's WARNINGS surface
@@ -401,12 +418,20 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
// online (storage-split B4); its OWN call like the rootfs resize. The volume + baked images // online (storage-split B4); its OWN call like the rootfs resize. The volume + baked images
// came in with the restore, so we grow it rather than attach a fresh one that would shadow // came in with the restore, so we grow it rather than attach a fresh one that would shadow
// the baked images. // the baked images.
if spec.DataVolGrowGB > 0 { //
// R-165: ONE volume, therefore ONE grow. `SysDataGrowGB` is FOLDED IN here rather than driving
// a second resize — see its field comment. This is the only arithmetic the merge added.
growGB := spec.DataVolGrowGB + spec.SysDataGrowGB
if growGB > 0 {
mount := spec.DataVolMount mount := spec.DataVolMount
if mount == "" { if mount == "" {
mount = DefaultDataVolMount mount = DefaultDataVolMount
} }
dupid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", spec.DataVolGrowGB)) if spec.SysDataGrowGB > 0 {
e.logger.Info("bring-up: folding the retired sys-data grow into the single data volume (R-165)",
"data_grow_gb", spec.DataVolGrowGB, "sysdata_grow_gb", spec.SysDataGrowGB, "total_gb", growGB, "mount", mount)
}
dupid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", growGB))
if err != nil { if err != nil {
res.Err = fmt.Errorf("reconcile: bring-up data-volume resize (%s): %w", mount, err) res.Err = fmt.Errorf("reconcile: bring-up data-volume resize (%s): %w", mount, err)
return return
@@ -417,25 +442,9 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
} }
} }
// 4c. Grow the golden-carried SSD user-data volume (mp1, /mnt/sys_drive = the controller's // 4c. RETIRED (R-165). There is no second volume: the golden ships ONE, and the sys-data grow is
// system_data_path) to the per-customer target. Same shape as the Docker-data grow: grow-only, // folded into 4b above. Deliberately left as a comment rather than silently vanishing, so a
// online, its OWN call. The volume came in with the restore (separate mount, backup=1), so we // reader of a v0.119.0 archive's provision log can see where the second resize went.
// grow it rather than attach a fresh one.
if spec.SysDataGrowGB > 0 {
mount := spec.SysDataMount
if mount == "" {
mount = DefaultSysDataMount
}
supid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", spec.SysDataGrowGB))
if err != nil {
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize (%s): %w", mount, err)
return
}
if _, err := e.waitTask(ctx, supid, proxmox.WaitOptions{}); err != nil {
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize task (%s): %w", mount, err)
return
}
}
// 4d. DR structural-bind swap (GL-5): replace the two restore-time throwaway volumes (see the // 4d. DR structural-bind swap (GL-5): replace the two restore-time throwaway volumes (see the
// restore call) with the REAL host binds, then delete the displaced volumes so a KEPT DR // restore call) with the REAL host binds, then delete the displaced volumes so a KEPT DR
@@ -606,6 +615,15 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
params["net0"] = withoutHwaddr(net0) // omit hwaddr → PVE generates a fresh MAC (F1) params["net0"] = withoutHwaddr(net0) // omit hwaddr → PVE generates a fresh MAC (F1)
} }
} }
// R-50 island NIC: attach a static net1 on the host-internal bridge so the control plane
// (controller→agent local API) rides a fixed private address, immune to any LAN/DHCP/site move.
// Both modes: a provisioned guest AND a DR-restored guest need to reach the island-bound agent on
// the target host. No hwaddr → PVE mints a fresh per-guest MAC (the /30 is one guest per host, so
// a MAC would not collide either way, but a fresh one keeps net1 symmetric with net0). Additive:
// omitted entirely when the island is not configured, keeping non-island hosts unchanged.
if strings.TrimSpace(spec.IslandBridge) != "" && strings.TrimSpace(spec.IslandGuestAddr) != "" {
params["net1"] = fmt.Sprintf("name=eth1,bridge=%s,ip=%s", spec.IslandBridge, spec.IslandGuestAddr)
}
if spec.Mode == ModeProvision && spec.Hostname != "" { if spec.Mode == ModeProvision && spec.Hostname != "" {
params["hostname"] = spec.Hostname params["hostname"] = spec.Hostname
} }
+57 -18
View File
@@ -165,6 +165,36 @@ func TestBuildBringUpConfig_ResourceCaps(t *testing.T) {
} }
} }
// R-50: with the island configured, bring-up attaches a static net1 on the island bridge; with it
// unset (or half-set), NO net1 is emitted — byte-for-byte the pre-R-50 config on non-island hosts.
// Pure-function check on buildBringUpConfig (the derivation that makes fresh installs F1-immune).
func TestBuildBringUpConfig_IslandNIC(t *testing.T) {
// island set → net1 present, exact shape, no hwaddr (PVE mints a fresh per-guest MAC)
island := buildBringUpConfig(BringUpSpec{
Mode: ModeProvision, IslandBridge: "vmbr9", IslandGuestAddr: "169.254.253.2/30",
}, scratchCfg())
if got, want := island["net1"], "name=eth1,bridge=vmbr9,ip=169.254.253.2/30"; got != want {
t.Errorf("island net1 mismatch:\n got %q\nwant %q", got, want)
}
// DR mode too — a restored customer guest must also reach the island-bound agent on the host.
dr := buildBringUpConfig(BringUpSpec{
Mode: ModeDRGuestLoss, KeepMAC: true, IslandBridge: "vmbr9", IslandGuestAddr: "169.254.253.2/30",
}, scratchCfg())
if _, ok := dr["net1"]; !ok {
t.Errorf("DR bring-up must also attach the island net1, got none")
}
// island unset → NO net1 key (non-island hosts unchanged; the pre-R-50 default)
none := buildBringUpConfig(BringUpSpec{Mode: ModeProvision}, scratchCfg())
if v, ok := none["net1"]; ok {
t.Errorf("net1 must be ABSENT when the island is not configured, got %q", v)
}
// half-configured (bridge only) → still no net1 (all-or-nothing; config.Validate rejects the config too)
half := buildBringUpConfig(BringUpSpec{Mode: ModeProvision, IslandBridge: "vmbr9"}, scratchCfg())
if v, ok := half["net1"]; ok {
t.Errorf("net1 must be ABSENT when only the bridge is set, got %q", v)
}
}
// Both restore sites allocate the guest INTO the felhom pool (SPIKE 3b): the provision bring-up // Both restore sites allocate the guest INTO the felhom pool (SPIKE 3b): the provision bring-up
// threads spec.Pool, and the restore-test hardcodes DefaultPool — else a pool-scoped token 403s on // threads spec.Pool, and the restore-test hardcodes DefaultPool — else a pool-scoped token 403s on
// the created guest's config/start/destroy. Asserts via the fakeAPI's captured RestoreLXCOptions. // the created guest's config/start/destroy. Asserts via the fakeAPI's captured RestoreLXCOptions.
@@ -235,10 +265,16 @@ func TestRunBringUp_StorageSplit_DataVolGrow(t *testing.T) {
} }
} }
// The golden-carried SSD user-data volume (/mnt/sys_drive) is grown via a SEPARATE resize on its // R-165 RETARGETED THIS TEST, and the retarget IS the contract change. There is no longer a second
// mpN slot (mp1), independent of the rootfs and Docker-data grows. With SysDataGrowGB=0 NO mp1 // volume, so `SysDataGrowGB` no longer drives its own resize on mp1 — its GiB are FOLDED INTO the
// resize is issued (the volume stays at the golden size, still a separate mount). // single volume's grow.
func TestRunBringUp_StorageSplit_SysDataGrow(t *testing.T) { //
// FOLDED, NOT DROPPED, and that is the whole point. `felhom-host-install.sh` computes and passes
// `-sysdata-grow` from the thin pool's free space, and an installer and an agent do not upgrade in
// the same instant. Dropping the value would silently shrink every appliance built by an older
// installer by the user-data share — 42 of 250 GiB on the standard branch — which is precisely the
// "a knob that silently does nothing" outcome this work was told to avoid.
func TestRunBringUp_StorageSplit_SysDataGrowIsFoldedIn(t *testing.T) {
const vmid = 8051 const vmid = 8051
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}} api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
e, _, q := newEngine(t, api, EmptyProvider{}) e, _, q := newEngine(t, api, EmptyProvider{})
@@ -247,26 +283,29 @@ func TestRunBringUp_StorageSplit_SysDataGrow(t *testing.T) {
res := e.RunBringUp(context.Background(), BringUpSpec{ res := e.RunBringUp(context.Background(), BringUpSpec{
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid, Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8051", RestoreStorage: "local-lvm", Hostname: "felhom-prov-8051",
DataVolGrowGB: 240, SysDataGrowGB: 42, // grows mp0 AND mp1 (DefaultSysDataMount) DataVolGrowGB: 240, SysDataGrowGB: 42, // ONE volume: 240 + 42 = 282
}) })
if res.Err != nil || !res.Pass { if res.Err != nil || !res.Pass {
t.Fatalf("provision must pass, got %+v", res) t.Fatalf("provision must pass, got %+v", res)
} }
// TWO resizes here: Docker-data mp0 +240G and the user-data volume mp1 +42G (no rootfs grow). // EXACTLY ONE resize. A second one would mean an mp1 the golden no longer ships.
if len(api.resizes) != 2 { if len(api.resizes) != 1 {
t.Fatalf("expected data-volume + sys-data resizes, got %+v", api.resizes) t.Fatalf("expected exactly ONE data-volume resize (there is no mp1 since R-165), got %+v", api.resizes)
} }
var sawData, sawSys bool r := api.resizes[0]
for _, r := range api.resizes { if r.disk != "mp0" {
if r.disk == "mp0" && r.size == "+240G" { t.Fatalf("resized %q, want mp0 — the single data volume", r.disk)
sawData = true
}
if r.disk == "mp1" && r.size == "+42G" {
sawSys = true
}
} }
if !sawData || !sawSys { if r.size != "+282G" {
t.Errorf("want mp0 +240G AND mp1 +42G, got %+v", api.resizes) t.Fatalf("resized %s, want +282G (240 data + 42 folded sys-data). Anything less means the "+
"retired knob's GiB were DROPPED, silently shrinking every appliance an older "+
"felhom-host-install.sh provisions", r.size)
}
for _, rr := range api.resizes {
if rr.disk == "mp1" {
t.Fatalf("an mp1 resize was issued (%+v) — the golden ships no second volume, so this "+
"would fail on a real box", rr)
}
} }
} }
+55 -2
View File
@@ -198,7 +198,7 @@ func (e *Engine) runScratchTest(ctx context.Context, vmid int, spec RestoreTestS
launched := false launched := false
defer func() { defer func() {
if launched { if launched {
e.teardownScratch(ctx, base) e.teardownScratch(ctx, base, spec.ScratchMin, spec.ScratchMax)
return return
} }
e.append(withState(base, OpFailed)) e.append(withState(base, OpFailed))
@@ -231,6 +231,12 @@ func (e *Engine) runScratchTest(ctx context.Context, vmid int, spec RestoreTestS
// Pool=DefaultPool so the scratch guest is created INTO the felhom pool — else a pool-scoped // Pool=DefaultPool so the scratch guest is created INTO the felhom pool — else a pool-scoped
// token 403s on the scratch guest's config/start/destroy (SPIKE residual #2). // token 403s on the scratch guest's config/start/destroy (SPIKE residual #2).
VMID: vmid, Archive: spec.Archive, Storage: spec.RestoreStorage, MountOverrides: mountOverrides, Pool: DefaultPool, VMID: vmid, Archive: spec.Archive, Storage: spec.RestoreStorage, MountOverrides: mountOverrides, Pool: DefaultPool,
// onboot=0 from the instant the guest exists. Step 2 below link-downs every NIC before the
// guest is ever started, so the NORMAL path cannot conflict with the live source. This
// covers the ABNORMAL path: a restore that fails before step 2 (e.g. the wait expiring) can
// leave a scratch carrying the source's `onboot: 1` plus its MAC/static island IP/hostname —
// which a host reboot would then start alongside the original. Observed live 2026-07-26.
ConfigOverrides: map[string]string{"onboot": "0"},
}) })
if err != nil { if err != nil {
if pveAlreadyExists(err) { if pveAlreadyExists(err) {
@@ -440,7 +446,12 @@ func sizeToGB(s string) int {
// teardownScratch destroys the scratch guest (benign, gated) and records the entry terminal. // teardownScratch destroys the scratch guest (benign, gated) and records the entry terminal.
// On any teardown failure it leaves the entry in-flight so Recover reaps the guest later. // On any teardown failure it leaves the entry in-flight so Recover reaps the guest later.
func (e *Engine) teardownScratch(ctx context.Context, base JournalEntry) { //
// F-LEAK (Campaign 8): the API destroy is tried FIRST and is the normal path. It fails on a scratch
// left by a FAILED restore, because such a guest never joined /pool/felhom and the token's
// VM.Allocate lives there — so a band-scoped fallback through the fenced root path follows. See
// proxmox.DestroyScratchLXC for the two API-side fixes that were built and refuted live.
func (e *Engine) teardownScratch(ctx context.Context, base JournalEntry, scratchMin, scratchMax int) {
// Cancel-immune + bounded, so a shutdown mid-test still tears down. // Cancel-immune + bounded, so a shutdown mid-test still tears down.
tctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute) tctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute)
defer cancel() defer cancel()
@@ -453,6 +464,14 @@ func (e *Engine) teardownScratch(ctx context.Context, base JournalEntry) {
} }
upid, err := e.api.DestroyLXC(tctx, base.VMID) upid, err := e.api.DestroyLXC(tctx, base.VMID)
if err != nil { if err != nil {
// A 403 "missing privilege VM.Allocate" here means this scratch is not a felhom-pool member: a
// FAILED restore never completes the `--pool` association, and the token's VM.Allocate is
// granted at /pool/felhom. Fall back to the band-scoped fenced destroy — WITHOUT it the guest
// leaks and holds its disks until a human removes it.
if e.destroyScratchPrivileged(tctx, base.VMID, scratchMin, scratchMax, err) {
e.append(withState(base, OpSucceeded))
return
}
e.logger.Error("restore-test: scratch teardown failed; left for Recover", "vmid", base.VMID, "err", err) e.logger.Error("restore-test: scratch teardown failed; left for Recover", "vmid", base.VMID, "err", err)
return return
} }
@@ -555,3 +574,37 @@ func withUPID(base JournalEntry, upid string, state OpState) JournalEntry {
base.At = time.Now().UTC() base.At = time.Now().UTC()
return base return base
} }
// destroyScratchPrivileged is the F-LEAK fallback: destroy a stranded scratch through the fenced root
// path when the API token cannot. Reports whether the guest is gone.
//
// It refuses unless the guest is BOTH agent-created scratch provenance (this journal entry) and inside
// the configured band. That is the innermost of three checks — sudoers matches the vmid literally and
// proxmox.DestroyScratchLXC re-checks the band — because this op DESTROYS and the band must not rest
// on a single guard.
func (e *Engine) destroyScratchPrivileged(ctx context.Context, vmid, bandMin, bandMax int, apiErr error) bool {
if e.hostRun == nil {
e.logger.Warn("restore-test: no host-root runner wired — cannot reclaim the stranded scratch",
"vmid", vmid, "api_err", apiErr)
return false
}
if bandMin <= 0 || bandMax < bandMin {
e.logger.Error("restore-test: scratch band is not configured — refusing the privileged teardown",
"vmid", vmid, "min", bandMin, "max", bandMax)
return false
}
if vmid < bandMin || vmid > bandMax {
e.logger.Error("restore-test: refusing the privileged teardown — vmid is outside the scratch band",
"vmid", vmid, "min", bandMin, "max", bandMax)
return false
}
e.logger.Warn("restore-test: API teardown failed (stranded scratch is in no pool) — reclaiming via the fenced root path",
"vmid", vmid, "api_err", apiErr)
if err := proxmox.NewPrivileged(e.hostRun, "").DestroyScratchLXC(ctx, vmid, bandMin, bandMax); err != nil {
e.logger.Error("restore-test: privileged scratch teardown ALSO failed; left for Recover",
"vmid", vmid, "err", err)
return false
}
e.logger.Warn("restore-test: stranded scratch guest reclaimed via the fenced root path", "vmid", vmid)
return true
}
+24
View File
@@ -667,3 +667,27 @@ func TestRunRestoreTest_RefusalsPropagate(t *testing.T) {
t.Fatalf("never restore a partial guest to verify it: %+v", apiBind.restores) t.Fatalf("never restore a partial guest to verify it: %+v", apiBind.restores)
} }
} }
// A leaked scratch guest must never AUTO-START. The normal path link-downs every NIC before boot
// (TestRestoreTest… above), so the source can never be conflicted with on the happy path. This
// covers the abnormal one: a restore that fails BEFORE the link-down step leaves a scratch carrying
// the SOURCE guest's config verbatim — including `onboot: 1`, its MAC, its static island IP and its
// hostname. Observed live 2026-07-26, when a wait-timeout left exactly such a guest on demo-felhom.
// onboot=0 is therefore set AT RESTORE TIME, not after: after is too late for the path that leaks.
func TestRestoreTest_RestoreSetsOnbootZero(t *testing.T) {
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
_ = e.RunRestoreTest(context.Background(), RestoreTestSpec{
Archive: "local:backup/x.tar.zst", RestoreStorage: "local-lvm",
ScratchMin: 990000, ScratchMax: 990009, SourceTier: "local",
})
if len(api.restores) != 1 {
t.Fatalf("want one restore, got %+v", api.restores)
}
if got := api.restores[0].ConfigOverrides["onboot"]; got != "0" {
t.Fatalf("the restore MUST set onboot=0 so a leaked scratch cannot auto-start; got %q (%#v)",
got, api.restores[0].ConfigOverrides)
}
}
+67 -13
View File
@@ -58,6 +58,9 @@ type observed struct {
known KnownTarget known KnownTarget
src proxmox.Storage src proxmox.Storage
cat storageCategory cat storageCategory
// smartHint (v0.95.0) is a SMART-ONLY whole-disk device for a dir-storage whose own backing is
// empty (the builtin `local` on the shared LVM root). Never assigned to BackingDevice/durable_id.
smartHint string
} }
// Observe builds the reported []hub.StorageTarget. A non-nil error means the Proxmox read // Observe builds the reported []hub.StorageTarget. A non-nil error means the Proxmox read
@@ -84,13 +87,22 @@ func (o *Observer) enrich(ctx context.Context, ob observed) hub.StorageTarget {
if o.ops == nil { if o.ops == nil {
return t return t
} }
// SMART: only for dir-backed targets with a resolvable whole-disk device. // SMART: for dir-backed targets. The device is the target's own backing (a USB/local-dir exact
if ob.cat == catDir && t.BackingDevice != "" { // mount) OR, for a dir on a shared filesystem whose backing is deliberately empty (the builtin
if dev, ok := smartDeviceFor(t.BackingDevice); ok { // `local` on the LVM root — removable-safety guard in build), the SMART-only hint build() resolved
if sm, err := o.ops.SMART(ctx, dev); err != nil { // from the containing filesystem. smartDeviceFor then resolves dm/LVM/partition to the whole disk.
o.logger.Warn("storage: SMART read failed; health UNKNOWN", "device", dev, "err", err) if ob.cat == catDir {
} else { smartDev := t.BackingDevice
t.Smart = sm if smartDev == "" {
smartDev = ob.smartHint
}
if smartDev != "" {
if dev, ok := smartDeviceFor(smartDev); ok {
if sm, err := o.ops.SMART(ctx, dev); err != nil {
o.logger.Warn("storage: SMART read failed; health UNKNOWN", "device", dev, "err", err)
} else {
t.Smart = sm
}
} }
} }
} }
@@ -223,9 +235,21 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
Content: s.Content, Content: s.Content,
MountPath: mountPath, MountPath: mountPath,
BackingDevice: backingDevice, BackingDevice: backingDevice,
ClassHint: classHint, // R-116: the CONFIGURED path, carried verbatim and never resolved. This is emphatically NOT the
Role: "", // hub-owned; not derivable from a Proxmox def (slice 10) // fallthrough the comment above forbids — that prohibition is about resolving a device or a UUID
Smart: hub.SmartSummary{Health: hub.SmartUnknown}, // from the CONTAINING filesystem when the target is not its own mount, which would hand back
// root's identity and mis-target a DR re-attach. `s.Path` is the storage's own declaration of
// where it lives; it identifies nothing but itself, and it is not used for device or UUID
// resolution anywhere. MountPath stays empty when the mount is gone, which is the truth.
ConfigPath: s.Path,
// R-106: the CONFIGURED PBS namespace, carried verbatim from storage.cfg. Empty for every
// non-pbs storage (Proxmox only emits it on pbs), and empty for a pbs storage in the root
// namespace — the DR recipe distinguishes those two cases by the storage's TYPE, never by
// guessing from this string.
PBSNamespace: s.Namespace,
ClassHint: classHint,
Role: "", // hub-owned; not derivable from a Proxmox def (slice 10)
Smart: hub.SmartSummary{Health: hub.SmartUnknown},
} }
// Thin-pool DATA fill: surfaced prominently for lvmthin (metadata fill is Phase B/lvs). // Thin-pool DATA fill: surfaced prominently for lvmthin (metadata fill is Phase B/lvs).
@@ -238,10 +262,23 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
} }
} }
// SMART-only device hint (v0.95.0): a dir-storage that lives INSIDE a shared filesystem (the
// builtin `local` on the LVM root) has empty backing by design, yet we can still read the PHYSICAL
// disk's SMART by resolving its containing filesystem. Gated to catDir + no own backing + reachable,
// so an UNPLUGGED removable (disconnected) never reads root's SMART, and a mounted removable uses
// its own backing instead.
smartHint := ""
if category == catDir && backingDevice == "" && reachable {
if dev, ok := containingMountDevice(mounts, s.Path); ok {
smartHint = dev
}
}
return observed{ return observed{
target: tgt, target: tgt,
src: s, src: s,
cat: category, cat: category,
smartHint: smartHint,
known: KnownTarget{ known: KnownTarget{
Name: s.Storage, Name: s.Storage,
Type: typ, Type: typ,
@@ -260,6 +297,12 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
// smartctl (which targets the disk, not the partition). Returns ok=false when the result // smartctl (which targets the disk, not the partition). Returns ok=false when the result
// isn't a recognized raw disk (e.g. device-mapper / LVM), so SMART is simply skipped. // isn't a recognized raw disk (e.g. device-mapper / LVM), so SMART is simply skipped.
func smartDeviceFor(device string) (string, bool) { func smartDeviceFor(device string) (string, bool) {
// Device-mapper / LVM (Fix A, v0.95.0): resolve to the single backing whole disk via sysfs
// slaves. This is what finally covers the system SSD under `pve-root`. The sysfs resolution IS
// the existence check, so we do NOT re-run ValidateSMARTDevice on its result.
if strings.HasPrefix(device, "/dev/dm-") || strings.HasPrefix(device, "/dev/mapper/") {
return dmWholeDisk(device)
}
dev := device dev := device
if m := reNVMePart.FindStringSubmatch(device); m != nil { if m := reNVMePart.FindStringSubmatch(device); m != nil {
dev = m[1] // /dev/nvme0n1p2 -> /dev/nvme0n1 dev = m[1] // /dev/nvme0n1p2 -> /dev/nvme0n1
@@ -445,5 +488,16 @@ func mergeConfig(node, cluster proxmox.Storage) proxmox.Storage {
if node.Content == "" { if node.Content == "" {
node.Content = cluster.Content node.Content = cluster.Content
} }
// R-106: the pbs namespace. `NodeStorage` does not return it AT ALL — it is cluster-config only —
// so without this line `Namespace` is always empty on the merged entry and every consumer sees a
// root-namespace box. Found by LIVE VALIDATION, not by the unit tests: the DR-recipe tests supply
// StorageTarget values directly, so they never crossed this merge.
//
// This function is a copy-only-what-is-needed allow-list, which is exactly how the gap arose. If you
// add a consumer of any other type-specific field (`Username` is the remaining unmerged one), add it
// here too and pin it in TestMergeConfig_CarriesPBSNamespace's table.
if node.Namespace == "" {
node.Namespace = cluster.Namespace
}
return node return node
} }
+71
View File
@@ -0,0 +1,71 @@
package storage
import (
"context"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// Fix A (A2): the builtin `local` dir lives on the LVM root, so its backing is empty by design — but
// enrich now resolves the containing filesystem (/ → /dev/mapper/pve-root) and, via the dm sysfs
// slaves, the physical disk (/dev/sda). The system disk stops reading "Nincs adat".
// Red-proof: drop the `smartDev = ob.smartHint` fallback in enrich → local stays UNKNOWN and SMART
// is never called on /dev/sda.
func TestObserve_SystemDirSMARTViaContainingFS(t *testing.T) {
fixtureSysfs(t, map[string][]string{"dm-1": {"sda3"}}, map[string]string{"dm-1": "pve-root"})
api := &fakeStorageAPI{
node: "demo-felhom",
cluster: []proxmox.Storage{{Storage: "local", Type: "dir", Path: "/var/lib/vz"}},
nodeSt: []proxmox.Storage{{Storage: "local", Type: "dir", Path: "/var/lib/vz", Active: 1}},
}
host := &fakeHostReader{
mounts: []Mount{{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"}},
}
ops := &fakeHostOps{smartByDevice: map[string]hub.SmartSummary{"/dev/sda": {Health: hub.SmartPassed, ModelName: strptr("AirDisk 512GB SSD")}}}
got, err := NewObserver(api, host, ops, quietLogger()).Observe(context.Background())
if err != nil {
t.Fatal(err)
}
local := byName(got)["local"]
if local.Smart.Health != hub.SmartPassed {
t.Errorf("system disk SMART not enriched via the containing fs: health=%q", local.Smart.Health)
}
if local.Smart.ModelName == nil || *local.Smart.ModelName != "AirDisk 512GB SSD" {
t.Errorf("model not carried: %v", local.Smart.ModelName)
}
// SMART must have run on the resolved PHYSICAL disk, never the dm/mapper node.
if len(ops.smartDevices) != 1 || ops.smartDevices[0] != "/dev/sda" {
t.Errorf("SMART should target /dev/sda, got %v", ops.smartDevices)
}
// The backing device / durable_id must stay untouched by the SMART-only resolution.
if local.BackingDevice != "" {
t.Errorf("system-dir SMART resolution leaked into BackingDevice: %q", local.BackingDevice)
}
}
// The watchdog Known() path MUST remain enrich-free (its slow root-shelling reads are the reason it
// exists as a separate fast path). Known must never invoke SMART.
// Red-proof: route Known through enrich → smartDevices is non-empty and this fails.
func TestKnown_NeverInvokesSMART(t *testing.T) {
fixtureSysfs(t, map[string][]string{"dm-1": {"sda3"}}, map[string]string{"dm-1": "pve-root"})
api := &fakeStorageAPI{
node: "demo-felhom",
cluster: []proxmox.Storage{{Storage: "local", Type: "dir", Path: "/var/lib/vz"}},
nodeSt: []proxmox.Storage{{Storage: "local", Type: "dir", Path: "/var/lib/vz", Active: 1}},
}
host := &fakeHostReader{mounts: []Mount{{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"}}}
ops := &fakeHostOps{smartByDevice: map[string]hub.SmartSummary{"/dev/sda": {Health: hub.SmartPassed}}}
if _, err := NewObserver(api, host, ops, quietLogger()).Known(context.Background()); err != nil {
t.Fatal(err)
}
if len(ops.smartDevices) != 0 {
t.Errorf("Known() invoked SMART %d time(s) — it must stay enrich-free: %v", len(ops.smartDevices), ops.smartDevices)
}
}
func strptr(s string) *string { return &s }
+63
View File
@@ -292,3 +292,66 @@ func TestObserve_MountReadFailureDegradesNotFatal(t *testing.T) {
t.Errorf("lvmthin still derivable without mounts: %+v", got) t.Errorf("lvmthin still derivable without mounts: %+v", got)
} }
} }
// ---------------------------------------------------------------------------------------------
// R-106 — the pbs namespace must survive the cluster/node merge.
// ---------------------------------------------------------------------------------------------
// TestObserve_CarriesPBSNamespaceThroughMerge pins the gap that shipped in agent v0.118.0 and was caught
// by LIVE VALIDATION rather than by tests: `mergeConfig` copies a hand-listed set of type-specific fields
// from the CLUSTER config onto the NODE entry, and `Namespace` was not on that list. `NodeStorage` does
// not return the namespace at all — it is cluster-config only — so `StorageTarget.PBSNamespace` was
// always empty and the DR recipe reported the root namespace on every per-customer box, exactly the
// R-106 symptom the fix was supposed to remove.
//
// The DR-recipe tests could not catch it: they construct StorageTarget values directly, so nothing
// crossed this merge. This test drives the REAL Observe path with the split PVE returns reproduced —
// namespace present in the cluster list, absent from the node list, which is what PVE actually does.
func TestObserve_CarriesPBSNamespaceThroughMerge(t *testing.T) {
api := &fakeStorageAPI{
node: "demo-felhom",
// Cluster config: carries the type-specific fields, as /storage does.
cluster: []proxmox.Storage{{
Storage: "felhom-pbs", Type: "pbs", Content: "backup",
Server: "10.77.0.1", Datastore: "felhom-offsite", Namespace: "demo-felhom",
}},
// Node entry: live usage + active flag, and NO namespace — the shape that made the bug invisible.
nodeSt: []proxmox.Storage{{
Storage: "felhom-pbs", Type: "pbs", Content: "backup",
Total: 100, Used: 10, Avail: 90, Active: 1, Enabled: 1,
}},
}
o := NewObserver(api, &fakeHostReader{}, nil, quietLogger())
targets, err := o.Observe(context.Background())
if err != nil {
t.Fatalf("Observe: %v", err)
}
if len(targets) != 1 {
t.Fatalf("want 1 target, got %d", len(targets))
}
if got := targets[0].PBSNamespace; got != "demo-felhom" {
t.Errorf("PBSNamespace=%q, want %q — the namespace was lost in mergeConfig, so the DR recipe "+
"reports the root namespace on a per-customer box (R-106)", got, "demo-felhom")
}
}
// TestMergeConfig_CarriesPBSNamespace is the direct table over the merge itself: a node entry that omits
// a field takes the cluster's value, and a node entry that HAS one keeps its own (never clobbered).
func TestMergeConfig_CarriesPBSNamespace(t *testing.T) {
cluster := proxmox.Storage{Storage: "felhom-pbs", Type: "pbs", Namespace: "demo-hp", Datastore: "felhom-offsite"}
// Node omits the namespace (the real PVE shape) → it must be filled from the cluster config.
if got := mergeConfig(proxmox.Storage{Storage: "felhom-pbs"}, cluster).Namespace; got != "demo-hp" {
t.Errorf("namespace absent on the node entry: got %q, want it merged from the cluster config", got)
}
// Node already has one → keep it (the merge is fill-if-empty, never overwrite).
nodeOwn := proxmox.Storage{Storage: "felhom-pbs", Namespace: "node-wins"}
if got := mergeConfig(nodeOwn, cluster).Namespace; got != "node-wins" {
t.Errorf("merge clobbered the node's own namespace: got %q", got)
}
// No cluster row at all → the node entry passes through untouched.
if got := mergeConfig(proxmox.Storage{Storage: "x", Namespace: "keep"}, proxmox.Storage{}).Namespace; got != "keep" {
t.Errorf("empty cluster row altered the node entry: got %q", got)
}
}
+4
View File
@@ -11,6 +11,7 @@ import (
// presence so an absent section (e.g. NVMe fields on a SATA disk, or no SMART at all on a // presence so an absent section (e.g. NVMe fields on a SATA disk, or no SMART at all on a
// USB bridge) decodes cleanly to nil and we degrade to UNKNOWN. // USB bridge) decodes cleanly to nil and we degrade to UNKNOWN.
type smartctlJSON struct { type smartctlJSON struct {
ModelName *string `json:"model_name"`
SmartStatus *struct { SmartStatus *struct {
Passed bool `json:"passed"` Passed bool `json:"passed"`
} `json:"smart_status"` } `json:"smart_status"`
@@ -64,6 +65,9 @@ func parseSMART(raw []byte) hub.SmartSummary {
s.Health = hub.SmartFailing s.Health = hub.SmartFailing
} }
} }
if j.ModelName != nil && *j.ModelName != "" {
s.ModelName = j.ModelName
}
if j.Temperature != nil && j.Temperature.Current != nil { if j.Temperature != nil && j.Temperature.Current != nil {
s.TemperatureC = j.Temperature.Current s.TemperatureC = j.Temperature.Current
} }
+154
View File
@@ -0,0 +1,154 @@
package storage
import (
"context"
"os"
"path/filepath"
"regexp"
"strings"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// sysBlockRoot is the sysfs block directory. A package var so tests can point it at a fixture tree.
var sysBlockRoot = "/sys/block"
// dmWholeDisk resolves a device-mapper / LVM device to its SINGLE backing whole disk, recursing
// through stacked dm layers via /sys/block/<dm>/slaves (Fix A, SPIKE-smart-coverage-2026-07-25).
// Returns ok=false when the device is not dm, sysfs is missing, there are no slaves, or the slaves
// span MORE THAN ONE physical disk — in that last case we deliberately skip rather than guess which
// of two disks to SMART (e.g. a mirrored LV).
func dmWholeDisk(device string) (string, bool) {
name := dmName(device)
if name == "" {
return "", false
}
disks := map[string]bool{}
if !collectSlaveDisks(name, disks, 0) {
return "", false
}
if len(disks) != 1 {
return "", false // no disk, or an ambiguous multi-disk dm — never guess
}
for d := range disks {
return "/dev/" + d, true
}
return "", false
}
// dmName maps a dm device path to its sysfs name (dm-N). Handles /dev/dm-N (and a bare dm-N)
// directly, and /dev/mapper/<name> by matching /sys/block/dm-*/dm/name.
func dmName(device string) string {
base := filepath.Base(device)
if strings.HasPrefix(base, "dm-") {
return base
}
if strings.HasPrefix(device, "/dev/mapper/") {
entries, err := os.ReadDir(sysBlockRoot)
if err != nil {
return ""
}
for _, e := range entries {
if !strings.HasPrefix(e.Name(), "dm-") {
continue
}
b, err := os.ReadFile(filepath.Join(sysBlockRoot, e.Name(), "dm", "name"))
if err == nil && strings.TrimSpace(string(b)) == base {
return e.Name()
}
}
}
return ""
}
// collectSlaveDisks fills `disks` with the whole-disk names backing dm `name`, recursing through
// nested dm. Returns false on missing sysfs, no slaves, or excessive nesting (loop guard).
func collectSlaveDisks(name string, disks map[string]bool, depth int) bool {
if depth > 8 {
return false
}
entries, err := os.ReadDir(filepath.Join(sysBlockRoot, name, "slaves"))
if err != nil {
return false
}
if len(entries) == 0 {
return false
}
for _, e := range entries {
s := e.Name()
if strings.HasPrefix(s, "dm-") {
if !collectSlaveDisks(s, disks, depth+1) {
return false
}
continue
}
disks[wholeDiskName(s)] = true
}
return true
}
// wholeDiskName strips a partition suffix to the whole-disk name (sda3→sda, nvme0n1p3→nvme0n1).
func wholeDiskName(part string) string {
if m := reNVMePartName.FindStringSubmatch(part); m != nil {
return m[1]
}
if m := reSDPartName.FindStringSubmatch(part); m != nil {
return m[1]
}
return part
}
var (
reNVMePartName = regexp.MustCompile(`^(nvme[0-9]+n[0-9]+)p[0-9]+$`)
reSDPartName = regexp.MustCompile(`^((?:sd|hd|vd)[a-z]+)[0-9]+$`)
)
// containingMountDevice returns the device of the mount whose mountpoint is the LONGEST prefix of
// path — the filesystem that actually holds `path`. Used ONLY to pick a whole-disk device for a
// SMART read of a dir-storage that lives inside a shared filesystem (the builtin `local` on the LVM
// root); it never feeds durable_id / backing_device (which stay empty for such targets by design —
// the removable-safety guard in build()).
func containingMountDevice(mounts []Mount, path string) (string, bool) {
clean := cleanMountPath(path)
best, bestLen := "", -1
for _, m := range mounts {
if m.Device == "" {
continue
}
mp := cleanMountPath(m.MountPoint)
if mp == clean || mp == "/" || strings.HasPrefix(clean, strings.TrimRight(mp, "/")+"/") {
if len(mp) > bestLen {
best, bestLen = m.Device, len(mp)
}
}
}
return best, best != ""
}
// SmartReader reads per-disk SMART for a backing device, resolving partition / dm / LVM down to the
// whole disk. It exists so the localapi /disks UNION path (registry/USB drives that skip Observe's
// enrich) gets the SAME SMART read the dir targets get, without duplicating smartDeviceFor (Fix B).
// A zero-value summary (Health "") means "could not read" (nil ops, unresolvable device, or a read
// error) — distinct from a read that returned UNKNOWN — so the caller can omit it exactly like the
// dir path does.
type SmartReader struct{ ops HostOps }
// NewSmartReader wraps a HostOps for the localapi union path.
func NewSmartReader(ops HostOps) *SmartReader { return &SmartReader{ops: ops} }
// SMARTForBacking reads SMART for backingDevice (partition/dm/whole-disk). Never returns an error;
// on any failure the summary's Health is "".
func (r *SmartReader) SMARTForBacking(ctx context.Context, backingDevice string) hub.SmartSummary {
if r == nil || r.ops == nil {
return hub.SmartSummary{}
}
dev, ok := smartDeviceFor(backingDevice)
if !ok {
return hub.SmartSummary{}
}
sm, err := r.ops.SMART(ctx, dev)
if err != nil {
return hub.SmartSummary{}
}
return sm
}
+107
View File
@@ -0,0 +1,107 @@
package storage
import (
"os"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// fixtureSysfs builds a /sys/block-shaped tree and points sysBlockRoot at it. `slaves` maps a dm
// name to its slave entries; `dmNames` maps a dm name to its /dm/name content (for /dev/mapper/*).
func fixtureSysfs(t *testing.T, slaves map[string][]string, dmNames map[string]string) {
t.Helper()
root := t.TempDir()
for dm, sl := range slaves {
for _, s := range sl {
if err := os.MkdirAll(filepath.Join(root, dm, "slaves", s), 0o755); err != nil {
t.Fatal(err)
}
}
}
for dm, name := range dmNames {
dir := filepath.Join(root, dm, "dm")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "name"), []byte(name+"\n"), 0o644); err != nil {
t.Fatal(err)
}
}
old := sysBlockRoot
sysBlockRoot = root
t.Cleanup(func() { sysBlockRoot = old })
}
// Fix A dm/LVM resolution. Red-proof: remove the `len(disks) != 1` all-same-disk guard in
// dmWholeDisk → the "mirror over two disks" case resolves to one of them instead of skipping.
func TestDMWholeDisk(t *testing.T) {
cases := []struct {
name string
slaves map[string][]string
dmNames map[string]string
in string
want string
ok bool
}{
{"single SATA slave", map[string][]string{"dm-1": {"sda3"}}, nil, "/dev/dm-1", "/dev/sda", true},
{"single NVMe slave", map[string][]string{"dm-0": {"nvme0n1p3"}}, nil, "/dev/dm-0", "/dev/nvme0n1", true},
{"stacked dm → one disk", map[string][]string{"dm-2": {"dm-1"}, "dm-1": {"sda3"}}, nil, "/dev/dm-2", "/dev/sda", true},
{"mapper name → dm-1", map[string][]string{"dm-1": {"sda3"}}, map[string]string{"dm-1": "pve-root"}, "/dev/mapper/pve-root", "/dev/sda", true},
{"mirror over two disks → skip", map[string][]string{"dm-1": {"sda3", "sdb3"}}, nil, "/dev/dm-1", "", false},
{"no slaves → skip", map[string][]string{"dm-1": {}}, nil, "/dev/dm-1", "", false},
}
for _, c := range cases {
fixtureSysfs(t, c.slaves, c.dmNames)
got, ok := dmWholeDisk(c.in)
if ok != c.ok || got != c.want {
t.Errorf("%s: dmWholeDisk(%q) = (%q,%v), want (%q,%v)", c.name, c.in, got, ok, c.want, c.ok)
}
}
}
// smartDeviceFor routes dm/mapper devices through the resolver, and whole-disk/partition through the
// regex path unchanged.
func TestSmartDeviceFor_DMBranch(t *testing.T) {
fixtureSysfs(t, map[string][]string{"dm-1": {"sda3"}}, map[string]string{"dm-1": "pve-root"})
if dev, ok := smartDeviceFor("/dev/mapper/pve-root"); !ok || dev != "/dev/sda" {
t.Errorf("smartDeviceFor(/dev/mapper/pve-root) = (%q,%v), want (/dev/sda,true)", dev, ok)
}
// missing sysfs → skip, not a guess
fixtureSysfs(t, map[string][]string{}, nil)
if _, ok := smartDeviceFor("/dev/dm-9"); ok {
t.Error("smartDeviceFor should skip an unresolvable dm device")
}
}
func TestContainingMountDevice(t *testing.T) {
mounts := []Mount{
{Device: "/dev/mapper/pve-root", MountPoint: "/"},
{Device: "/dev/sda2", MountPoint: "/boot/efi"},
{Device: "/dev/sdb1", MountPoint: "/mnt/usb"},
}
// A dir inside root resolves to root's device (longest prefix wins over "/").
if dev, ok := containingMountDevice(mounts, "/var/lib/vz"); !ok || dev != "/dev/mapper/pve-root" {
t.Errorf("containing(/var/lib/vz) = (%q,%v), want /dev/mapper/pve-root", dev, ok)
}
// A path under a more-specific mount picks that mount, not root.
if dev, ok := containingMountDevice(mounts, "/mnt/usb/data"); !ok || dev != "/dev/sdb1" {
t.Errorf("containing(/mnt/usb/data) = (%q,%v), want /dev/sdb1", dev, ok)
}
}
// Model capture (v0.95.0) — smartctl's model_name flows into SmartSummary; absent → nil.
func TestParseSMART_ModelName(t *testing.T) {
withModel := parseSMART([]byte(`{"model_name":"TOSHIBA MQ04ABF100","smart_status":{"passed":true}}`))
if withModel.ModelName == nil || *withModel.ModelName != "TOSHIBA MQ04ABF100" {
t.Errorf("ModelName = %v, want TOSHIBA MQ04ABF100", withModel.ModelName)
}
if withModel.Health != hub.SmartPassed {
t.Errorf("health = %q, want PASSED", withModel.Health)
}
noModel := parseSMART([]byte(`{"smart_status":{"passed":true}}`))
if noModel.ModelName != nil {
t.Errorf("absent model_name should be nil, got %v", noModel.ModelName)
}
}
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""agent_gates.py — THE entry point for this repo's gates. Run from the repo root:
python3 scripts/agent_gates.py # every gate
python3 scripts/agent_gates.py --fast # only gates that touch no network and no container
# runtime (what .githooks/pre-push runs)
Gates (all must pass; **non-zero exit on any failure**):
1. reuse-refs every path cited by this repo's REUSE.md still resolves
WHY THIS FILE EXISTS, WITH ONE GATE (2026-08-02, R-29 leg (b)).
A census of all thirteen gate scripts across the four felhom repos found one clean correlation:
**every check a CLAUDE.md tells a person to run was passing, and two of the four nobody is told
to run were failing** one since 14 July. This repo was the extreme case: nothing at all ran
against it, and its REUSE.md 90 cited paths was checked by no one. This file exists so the
agent is not the one repo with nowhere to put a gate, and so the pre-push hook has the same entry
point in all four repos. It grows when the agent grows a second check.
THE SHARED CHECKER. `reuse_refs_check.py` lives in ONE place `felhom.eu/scripts/` and is
invoked here across the workspace at `<repo-root>/../felhom.eu/scripts/`. It is deliberately NOT
copied into this repo: duplicating it would recreate exactly the drift it exists to detect. If the
sibling clone is absent the gate FAILS and prints the path it tried fail-closed, because a
runner that quietly skips a gate is the inert-seam failure this project has shipped four times.
EXIT CODES. Each gate returns 0 clean / 1 convicted / 2 inconclusive. This runner exits non-zero
if any gate is non-zero, and reports 2 distinctly as INCONCLUSIVE an undetermined result is
never a pass, but it is not a conviction either.
"""
import os
import subprocess
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SHARED_REUSE = os.path.join(os.path.dirname(ROOT), "felhom.eu", "scripts", "reuse_refs_check.py")
# (label, absolute script path, args, fast)
GATES = [
("reuse-refs", SHARED_REUSE, [ROOT], True),
]
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
def hooks_armed_note(root):
"""Print a WARNING (never a failure) when this clone's pre-push hook is not switched on.
core.hooksPath is local config and a clone does not carry it, so an unarmed clone is silent
by construction this is the only place it becomes visible.
"""
try:
val = subprocess.check_output(["git", "config", "--get", "core.hooksPath"],
cwd=root, stderr=subprocess.DEVNULL).decode().strip()
except Exception:
val = ""
norm = val.replace("\\", "/").rstrip("/")
if norm == ".githooks" or norm.endswith("/.githooks"):
return
print("WARNING: this clone is UNARMED — core.hooksPath is %s, so the pre-push hook will not\n"
" run here. Switch it on once with: git config core.hooksPath .githooks"
% (("'" + val + "'") if val else "unset"))
def run_gate(label, path, args):
if not os.path.exists(path):
print("\nFAIL: gate '%s' is MISSING — tried %s" % (label, path))
print(" A missing gate is a failure, never a skip (fail-closed). The reuse-refs")
print(" checker is shared and lives in the felhom.eu sibling clone; it is never copied.")
return 1
print("\n" + "=" * 78)
print("== gate: %s (%s%s)" % (label, os.path.basename(path),
(" " + " ".join(args)) if args else ""))
print("=" * 78, flush=True)
# stream the gate's own output rather than capturing it — its diagnostics are the point.
return subprocess.call([sys.executable, path] + args, cwd=ROOT)
def main(argv):
fast = "--fast" in argv
unknown = [a for a in argv if a != "--fast"]
if unknown:
print("unknown argument(s): %s" % " ".join(unknown))
print("usage: python3 scripts/agent_gates.py [--fast]")
return 2
selected = [g for g in GATES if g[3] or not fast]
skipped = [g[0] for g in GATES if not (g[3] or not fast)]
print("agent_gates — %d gate(s)%s" % (len(selected), " [--fast]" if fast else ""))
if skipped:
print(" --fast SKIPPED (deliberate periodic runs, never in a hook): %s" % ", ".join(skipped))
hooks_armed_note(ROOT)
results = [(label, run_gate(label, path, args)) for label, path, args, _f in selected]
print("\n" + "=" * 78)
print("== summary")
print("=" * 78)
worst = 0
for label, rc in results:
print(" %-18s %-13s (exit %d)" % (label, VERDICT.get(rc, "ERROR"), rc))
if rc != 0:
worst = 1 if rc == 1 or worst == 1 else 2
if worst == 0:
print("\nall agent gates OK")
return 0
convicted = [l for l, rc in results if rc == 1]
undecided = [l for l, rc in results if rc not in (0, 1)]
if convicted:
print("\nCONVICTED: %s" % ", ".join(convicted))
if undecided:
print("UNDETERMINED (never a pass): %s" % ", ".join(undecided))
return worst
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+67
View File
@@ -0,0 +1,67 @@
# -*- coding: utf-8 -*-
"""Seam test for scripts/agent_gates.py.
Run from the repo root: python3 scripts/test_agent_gates.py
WHY THIS EXISTS. An entry point is a seam by definition: a runner that LISTS a gate but never
executes it is inert and fully green, and this project has shipped an inert seam four times. So
the assertion is on the member gate's OWN distinctive stdout — never on the runner's summary
line, which the runner can print without ever calling anything plus the exit code, which is a
runner's actual effect.
The second test is the one that matters here: this repo's only gate lives in a SIBLING clone, so
"the sibling is missing" must be a FAILURE and not a quiet skip.
"""
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ENTRY = os.path.join(ROOT, "scripts", "agent_gates.py")
class AgentGatesTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
p = subprocess.run([sys.executable, ENTRY, "--fast"], cwd=ROOT,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
cls.rc = p.returncode
cls.out = p.stdout.decode("utf-8", "replace")
def test_exit_code_is_zero(self):
self.assertEqual(self.rc, 0, self.out)
def test_member_gate_actually_ran(self):
self.assertIn("cited paths — exact", self.out,
"the reuse-refs gate is listed but its own output never appeared — an inert "
"runner prints the summary without calling anything:\n%s" % self.out)
def test_missing_shared_checker_is_a_failure_not_a_skip(self):
"""Fail-closed. Copy the entry point into a lone directory with no felhom.eu sibling and
confirm it CONVICTS rather than reporting green with nothing run."""
tmp = tempfile.mkdtemp(prefix="agent-gates-")
try:
lone = os.path.join(tmp, "felhom-agent", "scripts")
os.makedirs(lone)
shutil.copy(ENTRY, os.path.join(lone, "agent_gates.py"))
p = subprocess.run([sys.executable, os.path.join(lone, "agent_gates.py"), "--fast"],
cwd=os.path.dirname(lone),
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out = p.stdout.decode("utf-8", "replace")
self.assertNotEqual(p.returncode, 0, out)
self.assertIn("is MISSING — tried", out)
finally:
shutil.rmtree(tmp, ignore_errors=True)
def test_unknown_argument_is_rejected(self):
p = subprocess.run([sys.executable, ENTRY, "--nope"], cwd=ROOT,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
self.assertEqual(p.returncode, 2, p.stdout.decode("utf-8", "replace"))
if __name__ == "__main__":
unittest.main(verbosity=2)