254 Commits

Author SHA1 Message Date
admin 4d82591052 release-agent.sh: the publish leg was unrunnable on its first real use
gates / gates (push) Successful in 7s
R-115's whole point is that publishing cannot be forgotten because it rides the
release script. On the first real release through it (v0.121.0, R-86) it died at
exactly that leg:

  scripts/release-agent.sh: line 101: .../scripts/publish-agent.sh: Permission denied

publish-agent.sh has been mode 0644 since it was created on 2026-06-28 — every
earlier caller ran it as `bash scripts/publish-agent.sh`, so nothing ever noticed,
and release-agent.sh (written the same day it was needed) called it directly.

Two fixes, both small and both wanted: restore the executable bit, and invoke it
through `bash` so the release no longer depends on a file mode — the kind of
thing a checkout, an archive or a copy loses again.

The v0.121.0 tag created by the failed run is withdrawn and recreated on this
commit; nothing was published under it (verified 404 on the package endpoint), so
one version name still means one binary.
2026-08-03 15:04:52 +02:00
admin 4618169036 R-86: restore-test follows the backup, not the clock (v0.121.0)
gates / gates (push) Failing after 7s
The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.

The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.

- state records WHICH archive was proven; legacy files keep their time and yield
  no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
  restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
  meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
  starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
2026-08-03 14:54:57 +02:00
admin 1b14cfd0b4 REPORT: Scenario F measured on real CI (runs 69 vs 70, same commit)
gates / gates (push) Successful in 7s
2026-08-03 12:44:37 +02:00
admin 0db77666c6 REPORT: release path + published-versions gate (no version bump)
gates / gates (push) Failing after 7s
2026-08-03 12:37:09 +02:00
admin dd2d1feb6e release path publishes, and an unreleasable version fails CI (R-115, R-183)
gates / gates (push) Failing after 7s
NO VERSION BUMP and nothing built: no Go code changed. The agent stays v0.120.0.

scripts/release-agent.sh — THE way to release. build -> tag -> publish -> verify
by INDEPENDENT download. Publishing was a separate remembered step and was
forgotten three times in five days (R-111's 17 stranded releases, 0.114.0, and
0.120.0 — deployed to both demo hosts and undownloadable, so a documented-path
reinstall would have silently downgraded them WHILE REPORTING SUCCESS). R-111's
own closing line named this leg and closed SHIPPED without it; it recurred the
same afternoon, which is the evidence that a note is not a mechanism.

It tags because felhom-host-install.sh now fetches the sixteen agent config
files from raw/tag/v<version>/ (R-183): a released version with no tag 404s a
box mid-install, as root, on a virgin machine. It verifies by downloading what
it just published and comparing the sha to what it built — the publish step's
own success is a report on its own write; a fetch returning the right bytes is
a different claim. It refuses a dirty/unpushed tree and refuses to re-release an
existing version. It does NOT vouch: that points machines at a version and stays
the operator's act.

scripts/check-published-versions.py — the gate. Every v<semver> tag must have a
downloadable package AND a tag tree serving the agent's configs. Registered as
NOT --fast (needs network; a push must not fail because Gitea blinked), and the
CI workflow now runs the FULL gate set instead of --fast — otherwise the gate
would have been registered and never run, the built-but-never-wired failure this
project has shipped four times.

The invariant is not the one specified, and the reason was measured, not assumed:
the hub artifact manifest is 401 without a per-customer passphrase and Gitea's
package LISTING api is 401 without a token, while the package DOWNLOAD url and
the git TAGS api are anonymous. So CI cannot ask "what is vouched" without an
operator credential — whose addition is the operator's call. The tag-based
invariant needs none and catches all three recorded instances. What it does not
catch (the hub vouching a version never released at all) is filed as R-184.
2026-08-03 12:34:20 +02:00
admin 9dfd89cb94 docs: agent 0.120.0 published + vouched, proven on two reinstalled boxes (R-178)
gates / gates (push) Successful in 6s
v0.120.0 had been built, committed and deployed to both demo hosts but never
published: the Gitea generic package 404'd and the hub manifest vouched 0.119.0.
Installer step 5 skips only on an exact version match, so a documented-path
reinstall would have downgraded both boxes to the pre-merge agent -- and would
have succeeded, since step_grows passes -sysdata-grow 0 and 0.119.0's mp1 resize
never fires. Published from a clean tree (upload 201, round-trip GET verified,
sha a7763d31b55b5ce7...) and vouched; both reinstalls then fetched and verified
it over the real customer path.

Filed as the third instance of R-115. No version bump, nothing built.
Evidence: felhom.eu/REPORT.md
2026-08-03 09:33:50 +02:00
admin 4bb84fc3ca REPORT: v0.120.0 + golden 3.0.0 — built and proven at the bake, NOT proven on a box
gates / gates (push) Successful in 12s
States the scope reduction first: Phases 6-7 (reinstall both demo boxes and
prove one end to end) were NOT done, nothing was wiped, and the golden is
deliberately left unvouched as a result. Filed as R-178.

Also records the two instrument errors this session: a census truncated by
head -10 that gave the wrong answer about --sysdata-grow (the installer does
pass it), and a wait gated on a marker the bake prints before publishing.
2026-08-03 07:15:29 +02:00
admin cd6e26785a v0.120.0 — one data volume (R-165, decision D-a, variant V-c)
gates / gates (push) Successful in 5s
build-golden.sh 2.1.0 -> 3.0.0: a layout change is a major. The golden ships
ONE data volume at a NEUTRAL path (/var/lib/felhom); both /var/lib/docker
and /mnt/sys_drive are binds of subdirectories of it. mp1 is gone.

The variant was chosen on measurement. Three candidates were built and
rebooted (SPIKE-r165-phase0-2026-08-03.md); all three boot, reboot 3/3, give
ONE df figure and keep a container's statfs(/) on the merged volume — the
ordering worry that motivated the probe did not materialise. They differ
only in which guarantee they break: volume-at-docker puts customer backups
inside Docker's data-root; volume-at-sys_drive puts Docker's ENTIRE
data-root under /mnt, which the controller container mounts wholesale
(measured: it then sees /mnt/sys_drive/docker). V-c breaks neither.

The four assertions were RETARGETED, never deleted, and each was RUN against
a deliberately wrong shape — a real split guest and a real archive of it:
8 checks, 8 passed. A new 2b asserts both paths are ONE filesystem, which
catches the S2 shape the spike ranked worse than the split. Assertion 5
replaces the old "was mp1 excluded?" guard, whose pattern could no longer
match — a guard that cannot match has silently stopped guarding.

Provisioning: one volume, one grow. SysDataGrowGB is FOLDED IN rather than
dropped, because a census established that felhom-host-install.sh passes
-sysdata-grow and the two do not upgrade in the same instant; dropping it
would silently shrink every appliance by 42 of 250 GiB. The flags stay
accepted for the same reason. The existing test was retargeted to pin the
fold, and it caught the change before I did.
2026-08-03 06:43:38 +02:00
admin 587dbb43fe docs: CHANGELOG + REPORT for the CI workflow (no version bump)
gates / gates (push) Successful in 6s
2026-08-02 16:35:44 +02:00
admin eb99144509 ci: run the gate entry point on every push (R-168)
gates / gates (push) Successful in 6s
Same shape as the other repos. CI clones felhom.eu as a sibling because the shared
reuse_refs_check lives there and this repo's REUSE.md cites hub/internal/store/dr_recipe.go,
which lives in the hub. No uses: step, no version bump, nothing built or deployed.
2026-08-02 16:27:27 +02:00
admin 2c4efed5de REPORT: gate enforcement session (no version bump) 2026-08-02 15:37:04 +02:00
admin 75245a467c docs: CHANGELOG for the gate entry point (no version bump) 2026-08-02 15:28:42 +02:00
admin 054e85a2bf gates: one entry point (scripts/agent_gates.py) + pre-push hook
A census of all thirteen gate scripts across the four felhom repos on 2026-08-02 found that
every check a CLAUDE.md names was passing and two of the four nobody is told to run were
failing. This repo was the extreme case: nothing ran against it at all, and its REUSE.md — 90
cited paths — was checked by no one.

agent_gates.py exists at ONE gate on purpose, so the agent is not the one repo with nowhere to
put a check and so the pre-push hook has the same entry point in all four repos. It grows when
the agent grows a second gate. The shared reuse checker stays in felhom.eu/scripts/ and is
invoked across the workspace — never copied here; an absent sibling clone FAILS the gate and
prints the path tried, which test_agent_gates.py pins by running the entry point from a lone
directory with no sibling.

.githooks/pre-push runs it with --fast and refuses the push. Per-clone and --no-verify-able,
both stated in the hook itself; a manual run WARNS when the clone is unarmed.

Tooling only: no Go change, no build, no deploy, no version bump.
2026-08-02 15:22:58 +02:00
admin 4663df7ff3 REPORT: agent v0.119.0 — host addresses, deployed + published + vouched 2026-07-31 08:54:50 +02:00
admin 14642e3c7b v0.119.0 — the host report carries the box's addresses
A managed box's IP was invisible in every operator surface because nothing
reported one: HostMetrics carried node/cpu/mem/disk/load/uptime/temp/wrapper-sha
and no address of any kind. The hub could not show a host's LAN IP anywhere.

Two things that looked like the answer are traps, both checked before writing
code: lan_resolver.host_ip is an OPTIONAL config value absent unless that feature
is configured, and DeriveHostIP(local_api.listen_addr) returns 169.254.253.1 —
since R-50 the local API binds a link-local address identical on every box. Both
would have produced a confident wrong answer.

New wire field addresses[], one entry per (interface, address). Deliberately
iface+cidr rather than a single lan_ip: a Proxmox host legitimately holds several
(management bridge, tailnet, WG tunnel) and picking one to call "the" LAN IP is a
guess the agent is not entitled to make — silently wrong on a box whose bridge is
not vmbr0. The agent reports what exists; the hub does the labelling.

The filter is one predicate, chosen by MEASURING both demo hosts rather than by
reasoning about interface names. IsGlobalUnicast() alone drops loopback, IPv6
link-local (one per bridge, pure noise) and IPv4 link-local (169.254/16 — exactly
the island address above). It needs no veth/fwbr/tap denylist: that per-guest
plumbing carries no IP at all and self-excludes, verified on both boxes.

No new privilege and no block I/O — net.Interfaces() is a netlink/procfs read, so
the sudoers fence is untouched and the health-check rule is honoured.

The seam DEFAULTS to the real enumerator, inverting the nil-reporter-means-off
convention: this stanza has no config gate, so a forgotten wiring call would have
shipped it silently empty — the inert-seam failure recorded four times here.

Cross-repo: the golden is duplicated byte-identically in felhom.eu and the
contract test fails on top-level key drift, so both goldens moved together and
addresses[0]'s key set is asserted bidirectionally. The field marshals as [],
never null — the repo's own no-nulls invariant caught that on the first run.

Tests +9; three red-proofs (global-unicast filter, down-interface guard, inert
collectAddresses) each run, observed failing, and reverted.
2026-07-31 08:40:58 +02:00
admin 6b5dade4dc R-106 follow-up: mergeConfig dropped the pbs namespace, so v0.118.0's fix was inert (v0.118.1)
Live validation caught what the tests could not. On demo-felhom the recipe read
namespace "root" with namespace_state "resolved" — confident and wrong, a worse
shape than the original defect.

mergeConfig overlays the cluster storage config onto the node entry through a
hand-listed set of fields and Namespace was not among them. NodeStorage does not
return the namespace at all, so PBSNamespace always read "" and latestPBSCoord
correctly treated that as the root namespace.

Every v0.118.0 test built StorageTarget values directly — including the two
through Collector.Collect(), which inject a fakeObserver — so nothing crossed the
merge. Two new tests drive the real Observe path with PVE's actual split returns
and table the merge itself. Red-proof: dropping the added line fails both.

Suite rc=0, 29 packages, 0 FAIL.
2026-07-30 13:23:21 +02:00
admin 1c8a67eece R-106 + R-109: the DR recipe records the resolved namespace and names the backup target (v0.118.0)
Both defects were live on both demo boxes: the recipe said namespace "root" while
storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup
dir storages holds the local archives.

R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once
the list is namespace-scoped, so it was always empty and normalised to "root". It now
resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes
PVE read, so the recipe cannot disagree with the backup.

R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the
function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1
from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not
agent.json on disk — a target move rewrites the file and deliberately does not restart.

Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason,
never a default, an empty string, or a placeholder.

Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target
would otherwise be stored intact and dropped before any operator saw it.

9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
2026-07-30 13:11:08 +02:00
admin 1913e12031 docs(R-117): REPORT.md — v0.117.0 shipped and proven live on demo-hp
Predicate validated on hardware for both dead states (stale-device and
filesystem-aborted), 340-497us per call, no block I/O proven by strace (only
/proc/self/mountinfo, 0 statfs). No regression through the real pipeline: the
live backup-target drive reads bound_under_parent=True via GET /disks with the
controller's own credential.

Records what was NOT covered: the stale-bind repair on hardware (StablePathForRaw
hardcodes the live parent, so it would write into guest 9201's namespace -
R-117h), and sustained-load behaviour, still unmeasured.
2026-07-30 12:42:09 +02:00
admin 966d8f41ff v0.117.0 — R-117: the liveness signal now tests liveness
BoundUnderParent reported a namespace that returned EIO on every read and write
as healthy, and the gate restarted the customer's apps onto it. Both existing
terms parse a mountinfo line and then test only fields[4], the mount POINT.
Field 3 — major:minor — sat in the same parsed slice and was discarded.

Measured on hardware: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb with `shutdown`,
bound_under_parent true, EIO both directions, and the controller taking its
Return branch and emailing backup_target_restored with no alarm on any channel.

BoundUnderParent gains a third term at both /disks construction sites. The new
bindLiveness reads /proc only and asks two questions: the bind must name the
same device as the raw mount, and the filesystem must not have aborted (ext4
`shutdown` or `emergency_ro`).

The second check is not optional. A device that fails WITHOUT disappearing gives
the identical all-signals-healthy state with the devnos EQUAL and the drive never
Disconnected, so the gate produces neither a Stop nor a Return and nothing is
emitted on any channel, indefinitely (R-117a). A devno-only fix would have passed
every payload test.

Three states, never a bool: {Unknown, Live, StaleDevice, Aborted}, read through
Usable(), where Unknown counts as PRESENT — reporting absent stops a working
customer's apps.

No new recovery path; the existing one was unblocked. AttachDrive's normalize leg
already did the repair and three call sites already invoked it, including the
controller's Return branch before it restarts apps. All three died on
`if n == 1 && GuestSeesMount(...)` returning early. Now: StaleDevice ⇒ re-bind
(repairs live, guest never restarts); Aborted ⇒ quiet no-op, because a re-bind
lands on the same dead superblock and this runs every 20s — an infinite silent
retry that masks the state; it surfaces via BoundUnderParent=false instead.

Ordering trap caught by a test: reading the abort flag before comparing devices
classifies the real return state as aborted (its stale bind carries `shutdown`
too) and refuses the repair while still reporting correctly. The abort flag is
read off the RAW mount in the stale case.

Tests 849 → 863, 29/29 packages green. 6 red-proofs, each verified to have
landed. A hollow test was caught during them: the aborted fixture first used a
/dev/mapper device, for which RoleForStorage derives role=system — a system row
has no GuestPath, never runs the conjunction, and reports false by default, so
the assertion passed vacuously and no mutation could fail it. Found because RP1
failed to fail.
2026-07-30 12:26:42 +02:00
admin 6be168d1a0 docs(R-117 Part 1): repeat the no-block-I/O health-check rule where it binds
felhom.eu/CLAUDE.md now carries the standing rule (with the measurement), but
that file does not load in an agent-only session — and health checks are written
in this repo. A standing rule that does not load where it binds is the inert-seam
shape applied to a rule, so the constraint is repeated here as a one-liner with a
pointer to the full text.

No code change, no version bump.
2026-07-30 12:10:24 +02:00
admin d4eb259da2 docs(R-116): record the live proof — four events, two matched pairs, discriminated
v0.116.0 validated on a fresh box: real day-0 from the v1.25.0 ISO on a nested PVE
on demo-hp (per runbooks/target-selection.md), agent installed unaided from the
vouched Day-0 manifest, drives enrolled through the real endpoints, device loss a
real hot-detach.

  07:20:04  backup_target_absent   (error)  Cel meghajto   <- TARGET, specific
  07:22:34  backup_target_restored (info)   Cel meghajto   <- its matching pair
  07:24:04  storage_disconnected   (error)  Adat meghajto  <- NON-target, generic
  07:25:34  storage_reconnected    (info)   Adat meghajto

All four reached the hub; gate fired in 3 s. Discrimination is proven NON-trivially
for the first time -- both prior runs had the target itself emit the generic event,
so their mirror proved nothing. Over-correction passes on a POSITIVE observable: 0
ABSENT lines and 0 drive events over 2m14s with both drives present, while two
RETURNED lines prove the gate was ticking rather than dead.

Caveat recorded, not a regression: the drill's controller was 0.185.1 from the
golden, which predates R-114, so its absent-state banner showed the old false
"backup is on the system disk" copy. The R-114 guard is pinned by unit test and by
the payload, but could not be confirmed on that box. Filed as R-120 -- the golden
is a release behind the deployed controller, which is R-115's class one layer up.

Teardown all three layers, including the hub: VM purged, storage removed with the
space measured back, hub records gate-blocked on ONLINE with the command recorded.

Suite rc=0 read separately from this commit.
2026-07-30 09:33:53 +02:00
admin 21b0164fad R-116 (v0.116.0): give the backup-target flag and the gate's key the same row
The absent-drive alarm was generic while its recovery was specific -- a pair an
operator cannot match. Mechanism now measured, not reasoned (felhom.eu
audits/DIAG-r116-disks-payload-2026-07-30.md): with the device gone /disks returns
4 rows, not 3. The drive appears TWICE and the two facts the controller needs are
on different rows -- the Observe row has backup_target:true but mount_path:"" and
guest_path:"" (so driveTargetByPath registers NO key from it), while the registry
row owns /mnt/felhom-drives/<name>, the key the gate looks up, with BackupTarget
absent from its struct literal => false.

WHY v0.115.0 WAS INERT: its fallback computed StablePathForRaw(t.MountPath), and in
the absent state MountPath is ALSO "" -- emptied by the same exactMount failure
that empties BackingDevice. It assigned nothing. Its test passed because the fixture
supplied a MountPath production never supplies, and the harness left DriveTargets
nil so the union loop never ran. Both corrected here; red-proof 1 replays v0.115.0's
exact code against the real shape and it fails.

THE JOIN, which was the hard part: with the device gone the two records share no
runtime field -- no mount, no backing device, and the Observe row's DurableID has
degraded off the fs-UUID. They share CONFIGURATION: storage.cfg's path on one side,
the .mount unit's Where on the other, both yielding the same stable guest path. New
hub.StorageTarget.ConfigPath (json:"-" -- that struct is a cross-repo contract
pinned by the golden + contract_test key-set comparison, and nothing off-box needs
the value), set from s.Path in observe.go, consulted in disks.go only after MountPath
so the present-state path is byte-identical, plus a guest-path arm on the union dedup
so exactly one row carries the drive.

WHY NEITHER OBVIOUS OPTION WAS TAKEN -- both regress R-114, which shipped yesterday.
backup_target_offer.go:79 reads (BackupTarget && MountPath != "") as "a real
drive with its own mountpoint -- healthy" and returns before its TargetAbsent
branch. Back-filling MountPath onto the Observe row (the smallest change, and the
spec's lean) and teaching the registry row the flag (its MountPath is non-empty, read
from the stale unit file) BOTH manufacture that row while the drive is missing, which
would have told the customer the backup target is fine while its drive is gone.
R-114's correctness rests on the absent-state rows not combining the flag with a
mount path; that coupling was invisible until the payload existed. Pinned by
TestAbsentTargetKeepsR114DegradedSignal.

Role unchanged, BoundUnderParent conjunction not widened, no wire field changed.
Suppressing the registry row in the absent state also removes its false
state:"attached" and its root-filesystem-derived total_bytes -- R-118's symptom
goes incidentally; R-118 is NOT fixed and stays open.

Tests 845 -> 849, suite rc=0 read separately from this commit. Four red-proofs, each
mutation asserted to have landed first.

NOT live-validated at this commit: publish+vouch, C5, discrimination, over-correction.
2026-07-30 08:48:31 +02:00
admin 2f4ccab166 docs: correct two stale claims in CLAUDE.md that misdirect live work
Both found while writing felhom.eu runbooks/target-selection.md.

1. The demo host block said the t740 is the designated drill+build VM host "but no
   drill VM is provisioned there yet". Stale since 2026-07-25 -- VM 300 (drill-r50)
   has been there since. The sentence read as discouragement from the very box the
   operator ruling designates, which is part of why a drill went to DooPlex instead.
   Now says the ruling is realized and to start there, points at the new
   target-selection runbook, and notes drill.qcow2 on DooPlex is a BAKE fixture, not
   a drill target. Agent version dropped (it changes several times a day; the hub
   host list and --version are the authorities) and the t740's PVE node name added.

2. RETRACTED the block's claim that the agent "does not run at all" at the remote
   site because localapi binds the LAN literal 192.168.0.162 and the service has
   "never started" -- with an outstanding config edit needing Viktor GO. That was
   true before R-50 and is false now: since the island migration (2026-07-25)
   localapi binds 169.254.253.1:8443 on vmbr9, which is location-independent by
   design, and proxmox.endpoint is https://127.0.0.1:8006. Verified live 2026-07-30:
   service active, version 0.115.0, and GET /disks answered over the island -- the
   whole R-116 payload capture went through it. A session trusting the old text
   would not have attempted the read that worked. The recorded remote-site address
   was also wrong (.162, not .147), so it now says re-check instead of asserting one.

No code, no version bump, so no CHANGELOG entry (that file is version-keyed) and
REPORT.md is left holding the v0.115.0 record rather than being overwritten by a
docs fix.
2026-07-30 08:25:41 +02:00
admin a58239f6de v0.115.0 — R-116: the backup-target flag reaches the row the controller keys on
Session C measured it live: a drive whose device vanished raised the GENERIC
storage_disconnected while its return raised the SPECIFIC
backup_target_restored -- an alarm and an all-clear an operator cannot pair.
backup_target_absent never fired at all.

The mechanism is not what the Session-C audit first said, and the difference
decides the fix. RoleForStorage returns RoleSystem whenever backingDevice == ""
(internal/storage/role.go:180-181). When the device goes, exactMountDevice
fails, BackingDevice becomes "", the target row's role flips to system and it
loses its guest path -- but keeps its MountPath. The union loop skips any drive
whose MountPath is already seen, so the registry row is DEDUPED AWAY ENTIRELY.
/disks carries no row with that guest path, so isTarget[guestPath] is a MISSING
KEY, not a false. Setting BackupTarget on the union row -- the obvious fix --
could not have worked, because that row is not emitted when the alarm is needed.
The audit is corrected in the same push.

Fix: on the Observe row only, carry the guest path when the row IS the backup
target and its role flipped because the device vanished.

Three gates, verified not assumed:
- t.BackingDevice == "" restricts it to the vanished-device flip; a genuinely
  system-BACKED storage has a real device and is excluded, so a dir storage at
  /mnt/<name> on the root disk cannot acquire a guest path.
- Case B, the common fresh-box shape, is safe twice over: its target is the
  builtin local on /var/lib/vz and StablePathForRaw returns "" for anything not
  exactly /mnt/<name>, so nothing is set even before the gates apply.
- It cannot make the gate read an absent drive as PRESENT. BoundUnderParent is
  assigned at exactly two sites, both inside guest-path blocks a system-role row
  never enters, so it stays false and planDriveGates computes false || false.
  Pinned by TestAbsentTargetRowDoesNotRegisterPresence -- getting this backwards
  would have silenced the alarm the fix exists to raise.

The :213-214 boundary stands: no system or backup mount gains a guest path.

Tests +5, asserting the emitted /disks JSON through a faithful copy of the
controller's driveTargetByPath, because the failure class is "the value is on
the wrong row". Red-proof: removing the block fails with "isTarget[...] is a
MISSING KEY"; reverted byte-identical.

Filed not closed: the two-row shape that produced this survives.
2026-07-29 23:51:03 +02:00
admin b58d7bcf39 v0.114.0 — R-113: drive presence means the DEVICE, not the bind
BoundUnderParent, the one field the controller's drive-absent gate keys on,
reported only "is this path a mount target in the guest's mountinfo". The
drive's raw mount at /mnt/<name> is a systemd mount unit bound to its device and
dies with it, but the agent's own bind of <raw>/felhom-data under the shared
parent is an ordinary bind: nothing ties it to the device, so its mountinfo
entry OUTLIVES the device as a stale shell. Presence read that survivor as true,
planDriveGates never produced a Stop action, and nothing fired on any channel --
not backup_target_absent, not the generic storage_disconnected. Measured live in
E-2d: detached at 10:58:37Z, silent for 4.5 minutes while the agent itself
logged "enrolled drive absent by UUID" every 20s (felhom.eu
audits/E2D-fresh-vm-2026-07-29.md §5.2).

The fix: BoundUnderParent becomes a CONJUNCTION -- bound under the parent AND
the drive's raw host mount still mounted (devicePresent, new deviceCheck seam
defaulting to isHostMountpoint). Applied at BOTH /disks construction sites. The
union path matters more, not less: it hardcodes State:"attached", so the
raw-mount check is the only device truth that row carries, and it is exactly the
shape E-2d detached.

Why a conjunction and not a replacement: half 2 alone would regress boot
ordering, where the raw drive mounts early and the bind lands ~18s later; the
gate depends on that window reading ABSENT. The conjunction leaves that
byte-identical and closes only the case the gate could never see.

Unknown is never absent: devicePresent("") returns TRUE. A false absent stops a
working customer's apps -- the failure mode of this fix, not of the bug.

Controller UNCHANGED, no MinAgent bump. BoundUnderParent has exactly one
functional consumer (planDriveGates, intermediary.go:226); every other mention
in both repos is a comment or a test, and boot convergence deliberately moved
off it to pollLiveBinds/driveBindLive. The alternative -- a new DevicePresent
bool the controller ANDs in -- was rejected as dangerous: a bool absent from an
older agent's JSON decodes to false, so every drive on a pre-0.114.0 agent would
have read ABSENT and stopped its apps.

Tests +6 in internal/localapi (208 -> 214): groups A-D plus a wire-contract test
asserting the ENCODED bound_under_parent, since that is what crosses to the
controller. Four red-proofs run and reverted (drop the conjunction on each path;
invert unknown; drop the bind half); disks.go verified byte-identical after.

NOT LIVE-VALIDATED. No drive was pulled. Leg awaiting Session C: device loss ->
gate Stop -> SetDisconnected -> backup_target_absent on the wire.
2026-07-29 17:20:16 +02:00
admin 58b598b697 v0.113.0 — E-2a: guarded backup-target wrapper + POST /backup/target
The agent cannot create a PVE storage (Datastore.Allocate at /storage) or grant
an ACL (Permissions.Modify) -- it holds neither by design, and widening the role
would trade the whole blast-radius containment model for one feature. The
privileged half therefore lives in a new fenced shim behind a literal
FELHOM_BACKUPTARGET sudoers alias, following the mkfs/pbs-apply pattern.

The wrapper enforces the two laws E-1 paid for on live hardware so no caller can
forget them: F-1 the path must BE the drive's own mountpoint, F-2 is_mountpoint 1
is hardcoded rather than a caller flag. It refuses a root-device target, has NO
storage-removal path of any kind (the pbs-apply no-delete law, grep-assertable),
is idempotent for the same path, and REFUSES to repoint an existing id.

POST /backup/target drives it in a fixed order: create -> grant -> config.
Reversed, a config pointing at an ungranted storage 403s every backup on first
run -- exactly E-1 finding F-3. A failed grant leaves the config untouched.

It deliberately does NOT restart the agent: restarting with a backup in flight
cancels the wait and records a spurious tier failure for a backup that actually
succeeded (E-1 did this to a real felhom-pbs run). It returns restart_required
and the caller restarts behind its own immediate in-flight check.

Config rewrite preserves unknown keys verbatim and writes in place, since
/etc/felhom-agent is root-owned while agent.json is agent-owned 0600.

Green gate: build + vet + test rc=0 (29 packages), run separately from this commit.
2026-07-29 09:05:59 +02:00
admin 958e54f6a6 v0.112.0 — E-2: GET /disks flags the backup-target drive
Additive backup_target field, true for the drive backing the PRIMARY tier.

The controller cannot work this out itself: settings.StoragePath.BackupTarget is
customer INTENT, and on the two boxes migrated by hand in E-1 that intent was
never recorded -- intent is empty while the drive really IS the target. Without
this flag the absent-target alarm could not name the drive on exactly the boxes
that currently have one.

omitempty + false on an older agent, so an old controller degrades to the generic
disconnect alarm rather than a wrong one.

Test asserts the target IS flagged AND the non-target is NOT, as a pair -- a
blanket true would satisfy a naive one-sided check.
2026-07-29 08:20:27 +02:00
admin 38176ada9d v0.111.0 — E-2c: the backup-target drive can no longer be ejected
A regression guard on a configuration that is live right now. E-1 moved each
demo box's whole-guest vzdump target onto its secondary drive at that drive's
own mountpoint -- but RoleForStorage types a local-dir on a non-system device as
user-data, so the existing eject role gate PASSED it. POST /disks/eject on
/mnt/nvme-1tb (demo-hp) or /mnt/hdd_1 (demo-felhom) would have SUCCEEDED
silently, taking the only local whole-guest backup with it, with no alarm and
the box still reporting a configured tier. Found by E-2 Phase 0, not by a
failure.

handleDiskEject and handleDiskDecommission now call refuseIfBackupTarget AFTER
the role gate and refuse with 409, naming the storage and the remedy -- the op
is ordered, not forbidden: reassign the target first.

NOT a role reclassification, which is the obvious fix and the wrong one: making
RoleForStorage return RoleBackup would refuse every legitimate eject of the
customer's own data drive, because on both demo boxes that drive IS the target.
That trades a silent failure for a permanent obstruction.

backupTargetAt resolves through the agent's own storage view, never the caller's
claim, and fails OPEN -- safe because it sits behind the role gate, which fails
SAFE on the same error.

Red-proofed both ways, mutations verified to land first:
  - removing the eject guard  -> "eject of the backup-target drive SUCCEEDED (200)"
  - the over-correction (any backup-content dir storage is the target)
    -> the gate blocks /mnt/spare, failing TestEjectStillAllowedOnANonTargetDrive

Harness note: normalizeBackupTiers DROPS tiers with a nil Service and falls back
to the legacy empty-TargetID tier -- an earlier version of this test exercised
nothing and reported the production bug as if the fix had failed.

Green gate: build + vet + test rc=0 (29 packages), run separately from this commit.
2026-07-29 08:17:14 +02:00
admin d5c769173b REPORT + CONTEXT: F-LEAK closed via the fenced destroy (v0.110.0), all three attempts recorded 2026-07-28 11:34:48 +02:00
admin 50751b8901 F-LEAK third attempt: band-scoped fenced destroy (v0.110.0)
The per-VM ACL is consumed by the destroy it authorises (PVE remove_vm_access,
LXC.pm:906), so it works once per slot. Fourth root-fenced exception, band-enforced in
sudoers literally + in code + at the caller. API destroy still tried first.
2026-07-28 11:28:54 +02:00
admin ff7f68e089 REPORT + CONTEXT: F-REBOOT shipped, F-LEAK's first fix refuted and replaced, v0.109.0 observable 2026-07-28 11:21:42 +02:00
admin 88b3cf03dd gofmt: normalize internal/localapi (whitespace only)
Swept up by gofmt -w on the package while adding the guest-power observable. No
semantic change; 3 of 5 files are tests.
2026-07-28 11:15:45 +02:00
admin f27f7a2659 guest-power: add the liveness observable it shipped without (v0.109.0)
The v0.107.0 watchdog was silent on a healthy box, so its health could only be inferred
from absence — F-OBS's shape, shipped in the same session F-OBS was fixed. INFO summary
every 10th sweep with what it saw; aborted sweeps are not counted. Red-proofs 7 and 8.
2026-07-28 11:14:56 +02:00
admin 8db92947cd F-LEAK: remove the pool-adoption fix — refuted live; the fix is a path-scoped ACL (v0.108.0)
PUT /pools/{pool} ALSO requires VM.Allocate on the VM being added, so Pool.Allocate
cannot bootstrap its own membership. Proven live on demo-hp 2026-07-28. The real fix is
felhom-host-install v1.21.0 granting FelhomAgentGuest at /vms/990000..990009.
2026-07-28 11:05:37 +02:00
admin 367a503a0f F-REBOOT + F-LEAK: the agent's authority over guest lifecycle (v0.107.0)
F-REBOOT — a guest rebooted mid-backup never came back (fault 11: 9m47s of total
appliance outage, no lock, nothing retrying). The existing stale-lock recovery is
correct but missed it two ways: its predicate needs a stale vzdump lock and that
guest was unlocked, and it runs only at agent startup. New periodic guest-power
watchdog acts on 'should be running, is not, is not locked'.

onboot is the should-be-running signal, not invented here: stalelock.go already
uses it for this same decision, it is 0 on scratch/golden, and pve-guests uses it
at host boot. Guards: onboot:0 never touched (Scenario B), a locked guest is left
to the stale-lock path, a guest with a vzdump in flight is left stopped,
unprovable ownership acts on nothing, unconfirmable backup state fails safe.
Bounded retry 3x at 1/2/4m then ERROR (Scenario C) — a healthy start takes ~25s.

F-LEAK — a failed restore-test could not destroy its scratch (403 VM.Allocate).
It is pool membership, not privsep: VM.Allocate is granted at /pool/felhom only,
and a failed restore never completes the --pool association. Fix needs NO new
grant — Pool.Allocate is already held, so the teardown adopts the stranded
scratch into the pool and retries the destroy. Guarded by scratchAdoptAllowed:
scratch provenance AND the numeric band, both required (Scenario E).

Six red-proofs across both fixes, all observed failing.
2026-07-28 10:27:07 +02:00
admin a18b18e5de docs: correct inflight.go's DEFERS claim (F-A1)
The gate's behaviour is correct and unchanged. The comment said 'a caller that
cannot acquire DEFERS to its next cadence' — true of the restore-test caller,
NOT of the backup caller, and it did not say so. The controller recorded the
refusal as a tier failure and emailed the operator; fixed controller-side in
v0.179.0. Comment only, no behaviour change.
2026-07-28 08:50:23 +02:00
admin af1c21abc4 docs: F-CRIT-2 fix — REPORT, CONTEXT (v0.106.0)
Phase 0 discriminator survey, the measured 1 MiB floor and its justification,
four red-proofs with observed failure text, and the live re-test of campaign
fault 2 on demo-hp (both directions). Records that server-side prune does NOT
count phantoms toward keep-last — no retention bug — but never removes them
either (filed as R-99).
2026-07-28 08:07:42 +02:00
admin c9a5cc664a F-CRIT-2: a failed backup must not look like a fresh one (v0.106.0)
NewestArchiveTime counted an aborted PBS upload (1 byte, no manifest, NEWEST)
as a successful backup, so the tier reported fresh, went not-due, and was never
retried. On the real 168h offsite cadence that is 7 days of silence, and neither
the R-88 breaker (defers only DUE tiers) nor the hub deadline monitor (reads the
same freshness) can catch it.

R-84's storage-as-ground-truth was right; the bug is that presence was taken for
validity. Now only plausibly-complete entries count, via a measured size floor
(minPlausibleArchiveBytes = 1 MiB). Undecidable => not counted.

Size is the only tier-agnostic discriminator: verification and encrypted are
absent on EVERY local dir archive (and on a good PBS snapshot until verify-new
catches up), so gating on either would reject 100% of local backups and cause
fleet-wide backup thrash. Floor measured against the fleet: smallest real backup
is 612,397,450 B, so 1 MiB leaves 584x headroom — asserted by a test.

Rejections are announced at WARN once per distinct volid, naming snapshot and
reason; per-poll logging would emit ~288 lines/day and bury the signal.

Four red-proofs, all observed failing.
2026-07-28 07:47:22 +02:00
admin 023655370b seam sweep: compile-time witness for BackupArchiveLister (no version bump)
localapi.BackupArchiveLister is satisfied by a runtime type assertion in
newestArchiveOn; a failed assertion degrades SILENTLY to archiveAbsent, which is
the pre-R-84 in-memory-only behaviour — i.e. the R-84 bug returning with nothing
in any log to say so. There was no compile-time witness anywhere in production
code in either repo.

No defect found: *BackupRunner does satisfy it today, so this is a guard, not a
fix. Verified the guard works — breaking NewestArchiveTime's signature now fails
go build, where before it compiled and vetted clean.

No version bump, no deploy: compile-time only, zero runtime effect.
2026-07-27 18:23:49 +02:00
admin 9842c52853 docs: REPORT for agent v0.105.0 (R-88 Part 2)
Records the wire encoding (string enum, empty = legacy not unknown) and the two
judgement calls: the fail-safe direction is unchanged (unknown is still due), and
a service with no lister stays ABSENT because calling it unknown would starve a
genuinely new box on a pre-R-84 build.
2026-07-27 18:12:04 +02:00
admin 1c2664b0c1 agent v0.105.0 — R-88 Part 2: /backup/due gains age_state
newestArchiveOn's (time.Time, bool) signature could not express the 'unknown'
its own doc comment promised: a read error and a genuine not-found both returned
(zero,false), so /backup/due emitted a POSITIVE 'no successful backup recorded
yet' with a nil age, and the controller fired its window-gate valve on an
unreadable storage.

Three states now: known / absent / unknown, carried as a STRING enum so the zero
value unambiguously means 'legacy agent' rather than masquerading as an answer.
Fail-safe direction unchanged — unknown is still DUE; only the window-gate bypass
narrows to ABSENT.

A service with NO lister deliberately stays ABSENT: calling it unknown would stop
a genuinely new box on a pre-R-84 build from ever backing up outside its window.
An unparseable timestamp becomes unknown — a backup happened, we cannot date it.
2026-07-27 18:00:56 +02:00
Claude Code 5bca7bfc9a R-85 Phase 4: finalise v0.104.0 + register the new helpers in REUSE
Version finalised from v0.104.0-dev — an ldflags version disagreeing with the
CHANGELOG is the reconciliation problem hub 0.73.2 already caused.

REUSE gains backup.InFlight, backup.RestoreTestState and the SpecBuilder/
TierPicker/PickRestoreCandidateOn trio, each with the decision someone could
unknowingly undo: rotation credit only on success; InFlight callers defer and
never cancel; a tier with no archive no-ops rather than failing; SourceTier
comes from the archive, never the configured target.
2026-07-27 07:32:55 +02:00
Claude Code 043c7622bc R-85 Phase 2: tier rotation, persisted state, one heavy op at a time
The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite
tier's archives were never candidates — which is why demo-hp's DR tier reported
'applied' with zero snapshots for five days and nobody noticed.

Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first,
which is where the offsite tier starts. Ties break on target id so ordering is
deterministic rather than following Go's randomised map order. Rotation credit
only on SUCCESS — a permanently failing tier must keep sorting first, not look
freshly proven and stop being retried.

- backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename).
  This genuinely needs persistence unlike R-84: R-84 had ground truth to consult
  (the archive is still on the storage), whereas a restore-test destroys its
  scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'.
- backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup
  path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over
  the same tunnel a backup pushes one, and at ~33 MB/min both drift toward
  timeout, which is how a healthy tier gets recorded as failed. Callers DEFER,
  never cancel.
- PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or
  every fresh box looks broken for its first week.
- An empty tier is skipped and the next tried; it cannot starve, since it is
  still least-recently-proven once it has an archive.
- POST /backup joins the gate (409 naming the holder).

Red-proofs A/E/F observed with the documented text. Full suite green (29
packages, rc=0).
2026-07-26 21:00:42 +02:00
Claude Code 765d8b3168 R-85 Phase 1: the restore-test spec is built PER RUN, not frozen at daemon start
SchedulerOptions.Spec was a VALUE produced by an immediately-invoked function
at daemon start, so storageTier() and restoreTaskTimeout() were evaluated once
and reused for every run for the process lifetime. Nothing tier-varying was
expressible (the offsite tier could never be scheduled), and it was a latent
staleness bug besides: a storage-type or config change did not take effect
until restart.

- backup.SpecBuilder: func(ctx, archive) RestoreTestSpec, called once per run.
  The archive is passed because the tier MUST come from it (v0.100.0 rule) —
  config-derived is what classified a PBS archive as 'local' and killed a
  14.46 GB WAN restore at the 10-minute local bound.
- A nil spec builder SKIPS loudly instead of panicking: a wiring bug must cost a
  restore-test, never the daemon goroutine.

Red-proof observed. Full suite green (29 packages, rc=0).
2026-07-26 20:47:28 +02:00
Claude Code edde8a01ca REPORT: record the PASSED restore round-trip (mount_parity ok, source_tier pbs) 2026-07-26 19:05:05 +02:00
Claude Code a7ef497cc4 REPORT: add v0.103.0 (R-84) + demo-hp's first offsite backup landing (4.25 GB) 2026-07-26 18:24:37 +02:00
Claude Code 5acf1033a2 v0.103.0 — R-84: an agent restart no longer triggers a redundant backup
Observed live: three redundant local backups on demo-felhom in one afternoon of
deploys. The backup Store is in-memory ('lost on restart; the cadence
re-populates'), so after every restart /backup/due said 'no successful backup
recorded yet' and the controller took another one. On the offsite tier that is a
wasted multi-hour WAN upload after every agent deploy.

- BackupRunner.NewestArchiveTime: when a backup last LANDED on this tier's
  storage, read from the storage.
- localapi.BackupArchiveLister (optional BackupService extension): the due-check
  takes whichever is newer, the in-memory record or the storage.

Asking the storage rather than persisting the store is deliberate: it is ground
truth (a pruned archive correctly stops counting, where a persisted record would
keep claiming a backup that no longer exists), needs no new on-disk state, and
answers only 'when did a backup last land' — the richer fields stay with real
records so the host-report never carries invented numbers.

Fail-safes: read error -> fall back to memory (never fake freshness, never
suppress); genuinely empty -> due; old archive -> still due; service without the
lister -> unchanged.

Red-proof observed; full suite green (29 packages).
2026-07-26 18:20:59 +02:00
Claude Code e4f22f4c4f REPORT: R-82 agent arc v0.97.0 -> v0.102.0 (overwrite)
Four defects found by running it rather than reviewing it, the frozen untargeted
contract verified live, the fail-safe directions stated once, and what is NOT
done — including that the scheduled restore-test never selects the offsite tier
and that R-84 is now closer to a prerequisite than a tidy-up.
2026-07-26 17:56:42 +02:00
Claude Code 13ca2d96b2 fix(test): give the tiered-server harness a real storage view (v0.102.0 follow-up)
v0.102.0 defers a tier whose target storage is absent. The Slice A harness used
fakeStorage{} with NO targets, so after that change it deferred every tier and
five Slice A assertions became vacuous failures.

The product behaviour is correct; the harness never modelled a real box, which
has both storages present. Fixed by giving it local + felhom-pbs.

My error, and worth naming: I ran the suite and committed in the same command,
read 'packages ok: 28' and pushed without checking rc=1. That is exactly the
exit-code trap recorded in this arc twice already.

Full agent suite green: rc=0, 29 packages.
2026-07-26 17:41:03 +02:00
Claude Code 005083b558 v0.102.0 — R-82 Slice D: an unprovisioned tier DEFERS instead of failing
Prerequisite for the installer default (host-install 1.20.0). A fresh box now
carries the offsite tier, but felhom-pbs only exists once the hub provisions the
DR tier. Without this the tier would report due in that window and the
controller would quiesce the apps and fire a vzdump at a missing storage every
cadence.

- GET /backup/due?target= defers when the target storage is absent
  (targetStoragePresent): due:false with a reason that says why. The tier goes
  live with NO restart once the storage appears.

Fail-safe: a storage-view ERROR returns present and the tier stays due. 'I could
not check' must never be read as 'not there' — that would silently suppress
backups, the absence-is-not-failure rule relearned three times now (R-80, R-81,
the R-82 wait timeout).

Full suite green.
2026-07-26 17:40:31 +02:00
Claude Code 0fabc15896 v0.101.0 — R-82: a leaked restore-test scratch can no longer auto-start
CORRECTION: I earlier reported that the restore-test would boot a scratch guest
with the live guest's MAC/static island IP/hostname and break the control
plane. That was WRONG — RunRestoreTest step 2 link-downs EVERY interface
(withLinkDown, unit-tested) before the guest is ever started. The design
already handled it.

The real, narrower hazard: a restore that fails BEFORE step 2 (what the v0.100.0
wait bug caused) leaves a scratch holding the SOURCE guest's config verbatim,
including onboot:1. If teardown also fails (403 missing VM.Allocate — PVE
associates the pool only at restore completion), a host reboot would start that
leaked clone alongside the original with NICs up.

- proxmox.RestoreLXCOptions.ConfigOverrides: guest-config params applied AT
  RESTORE TIME.
- The restore-test passes onboot=0 — at restore time, not after, because
  'after' is exactly the path that leaks.

NOT changed: the link-down step (already correct, the primary defence); the
agent's Proxmox privileges (widening VM.Allocate to /vms would remove the
accidental guard that stopped a destructive mid-restore teardown).

restore_test_cadence_seconds was set to -1 on demo-felhom under the mistaken
reading; re-enabled.

Red-proof observed; full suite green (29 packages).
2026-07-26 16:49:40 +02:00
Claude Code a7421b09c7 v0.100.0 — R-82: the restore tier comes from the ARCHIVE, not the configured target
Found by the first real PBS restore round-trip, not by review.

Restoring a felhom-pbs: archive on a box whose primary target is 'local'
failed after exactly 600.76s — the 10-minute LOCAL wait — against a 14.46 GB
WAN restore needing ~2 hours. The selftest derived its tier from
cfg.Backup.BackupTarget() (the configured default), so restoreTaskTimeout
correctly returned the local bound for a PBS archive. The recorded result even
said source_tier=local for a PBS archive.

The tier-aware machinery was already right; it was fed the wrong input. What
broke is an assumption that stopped being true the moment a second tier
existed: 'the configured target' is no longer a proxy for 'the tier this
archive belongs to'.

RestoreTestSpec.RestoreTaskTimeout's doc comment predicts the consequence
exactly, and it happened: teardown fired at a still-restoring guest and was
refused with HTTP 403 missing privilege VM.Allocate (PVE associates the pool
only at restore COMPLETION, and the grant is on /pool/felhom not /vms). That
403 was load-bearing luck — the only reason a destructive teardown did not run
against a half-restored guest. The restore completed unharmed.

- restoreTierForArchive() derives the tier from the archive's own storage
  (archiveStorageID parses the volid prefix), falling back to the configured
  target only when there is no prefix.

Recorded, NOT fixed here: the daemon's scheduled restore-test still only covers
the PRIMARY tier (Pick uses a runner built on BackupTarget(); Spec is built once
at construction, not per tick) — so the offsite tier is never automatically
restore-tested. And the agent still cannot tear down a scratch guest until its
restore completes; widening the token's privileges is deliberately not the fix.

Full suite green (29 packages).
2026-07-26 15:22:08 +02:00
Claude Code 3d955e4edd v0.99.0 — R-82 operator rulings: 2-week offsite retention + one backup at a time
Ruling 1 (2 weeks of weekly offsite backups): localPruneSpec's blanket PBS
refusal is now scoped — an ADDITIONAL tier with an explicit keep_last may
prune its PBS target. The refusal still applies in full to the PRIMARY tier,
because BackupTarget() defaults to felhom-pbs and KeepLast() defaults to 3, so
a box with neither key set would silently prune its offsite DR to 3 restore
points. An additional tier cannot have that accident (keep_last defaults to 0).

Ruling 3 (first backup runs as long as needed; nothing else starts until done):
- additional-tier wait bound 6h -> 12h (measured ~33 MB/min => ~5h for a first
  full 10 GB snapshot; 12h gives margin but stays bounded so a hung task still
  surfaces)
- ONE BACKUP AT A TIME PER GUEST across all tiers: POST /backup returns 409
  when a DIFFERENT tier is in flight, naming the busy tier, with NO data object
  so nothing is parseable as the caller's own job. Same tier still returns that
  job (202, unchanged).
- snapshotted now counts as in-flight, not just running — after the snapshot the
  vzdump is still uploading and holding the lock. The old check left a window
  where a second POST started a real second vzdump. Latent bug, closed.

Full suite green (29 packages); red-proof observed and restored.
2026-07-26 15:05:54 +02:00
Claude Code a667c269c7 v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure)
Found by live validation on demo-felhom, not by review.

The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.

Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.

- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
  (a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
  the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
  NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
  tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
  outer context reproduces the same false failure four hours later.

Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.

Red-proof observed and restored; full suite green.
2026-07-26 14:53:24 +02:00
Claude Code 68bcebe493 REPORT: R-82 Slice A (agent v0.97.0) — per-target tiers built, NOT deployed (no drill target reachable) 2026-07-26 12:36:47 +02:00
Claude Code 739b3c3b58 v0.97.0 — R-82 Slice A: per-target backup tiers (local daily + PBS weekly)
Mechanism only. No box changes behaviour until a backup_targets entry is
added to its config (Slice D); an untouched config resolves to exactly one
tier and behaves byte-identically to v0.96.0.

- config: BackupTargetConfig + ExtraTargets + BackupTiers(); each tier carries
  its OWN cadence and retention (keep-last=3 is three days on a daily tier and
  three weeks on a weekly one). A missing cadence is REJECTED, not defaulted —
  a weekly DR tier silently running daily would fill the 37.2 GB datastore.
  main.go logs every rejection at ERROR.
- /backup/due?target= judges a tier against its OWN newest successful backup.
  Without that filter a fresh local backup satisfies the weekly PBS cadence and
  the DR tier never runs — today's bug, re-created in code.
- GET /backup/tiers advertises the tiers; a 404 is the controller's pre-R-82
  capability probe (Slice B).
- Jobs keyed by (vmid,target): single-flight is per tier, which is what lets
  the weekly night run both backups in ONE quiesce window. Job ids are unique
  per tier by construction, not by clock luck.
- One runner per tier: the runner holds target+retention as immutable state,
  so parameterising one runner would risk pairing tier A's target with tier B's
  retention.

COMPATIBILITY (frozen): untargeted /backup/due, POST /backup and
/backup/status keep the primary tier and the pre-R-82 response BYTES —
Target is omitempty and stays empty. The primary's job-id format is unchanged.

NOT changed: the local tier; PBS is still never pruned by the per-run flag
(keep_last defaults to 0 = never prune — enabling DR pruning is irreversible
and needs an operator ruling).

Tests 748->768. Red-proof #1 observed and restored.
Phase 0: felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md
2026-07-26 12:20:58 +02:00
admin dfd5d731ee v0.96.0 — R-50 island NIC: provision attaches the guest island net1
- LocalAPIConfig.island_bridge + island_guest_addr (+ IslandEnabled, Validate
  all-or-nothing + CIDR guard)
- buildBringUpConfig attaches static net1 (island) on provision + DR when set;
  absent otherwise (pre-R-50 byte-for-byte). Plumbed from cfg.LocalAPI at both
  RunBringUp sites. Endpoint already follows listen_addr (A0: no template change).
- healer stays eth0-only (A3 verify-only) — red-proof test locks the scoping
- example config + firewall example rewritten for the island; REUSE updated
- 3 non-hollow tests; full green. MinAgent unchanged.

Coupling: host-install island config requires agent >= 0.96.0 (vouch first).
2026-07-25 14:16:23 +02:00
admin 36ed6594d4 docs(CLAUDE.md): note demo-hp (t740) as designated drill/build VM host (no drill VM yet) 2026-07-25 09:59:12 +02:00
admin 271aa3d9ed v0.95.0: REPORT (overwrite) — SMART coverage live-verified (system SSD + USB → Rendben + models) 2026-07-25 08:32:09 +02:00
admin ed97232598 v0.95.0: SMART coverage — union-path drives + LVM/dm root + device model
Implements SPIKE-smart-coverage-2026-07-25 fixes B+A (additive; MinAgent unchanged).
Fix B: storage.SmartReader.SMARTForBacking wired into the /disks union path (localapi
Smart seam) so registry/USB drives get a real SMART read (watchdog Known stays
enrich-free). Fix A: smartDeviceFor resolves dm/LVM to the whole disk via
/sys/block/<dm>/slaves (recursive; skips >1-disk); the builtin local dir on the LVM
root gets a SMART-only device from its containing filesystem (never touches
backing/durable_id). SmartSummary.ModelName captured from smartctl. Fix C (-d sat)
stays rejected. Tests + red-proofs (dm multi-disk skip, enrich smartHint, union
routing); Known-path-never-SMARTs asserted.
2026-07-25 08:21:45 +02:00
admin 643899c191 v0.94.0: REPORT — appended (SMART serialized into /disks, live-validated on demo-felhom) 2026-07-24 21:41:01 +02:00
admin 21fee69154 v0.94.0: serialize per-disk SMART into the /disks payload
Additive, backward-compatible (MinAgent floor unchanged). The SMART is already
computed on the request path (storage.Observe -> enrich); this copies the target's
Smart into localapi.DiskInfo (pointer, omitempty) only when Health != "", so an
unread/absent summary stays omitted and the controller renders "Nincs adat".
No new smartctl load, endpoint, or sudoers change.

Test TestDisks_SmartSerialized + red-proof (drop the copy -> fails).
2026-07-24 21:11:13 +02:00
admin c230258542 docs: v0.93.0 publish train executed — built, published, vouched, deployed fleet-wide 2026-07-22 09:03:11 +02:00
admin eba040d0be docs(report): the recovery-code wordlist fix and its red-proof 2026-07-21 15:33:30 +02:00
admin a452dc3314 escrow: a recovery code can no longer contain a hyphenated word (v0.93.0)
The EFF large list has exactly 4 entries containing the join separator
(drop-down, felt-tip, t-shirt, yo-yo). Drawing one made a code read as 11
words instead of 10 - ambiguous to transcribe in precisely the situation R
exists for. Filter them at init; the draw space goes 7776 -> 7772 and the
10-word code goes 129.248 -> 129.241 bits, still well over the 128 floor.

Generation-only: already-issued codes stay valid, R is verified as a whole
passphrase and never re-split.

Also fixes the ~1/5 flake this same defect caused: the test counted words by
splitting the joined string. It now counts what the generator drew and
asserts segmentation separately, plus a deterministic red-proof fixture.
2026-07-21 14:46:51 +02:00
admin 935904fa4e docs(report): guest_net confirmed on the wire hub-side; STOP-3 done; damping state is in-memory only 2026-07-21 13:39:33 +02:00
admin 332235bfd1 docs(report): quiet window closed — 30 healthy cycles, 0 heals; log_level reverted 2026-07-21 13:17:35 +02:00
admin 3f58121b98 docs(report): STOP-2 evidence — the incident replayed and PREVENTED (tunnel never dropped) 2026-07-21 12:58:12 +02:00
admin 71e1777369 docs(report): TASK-D Part 3 — R-54 shipped, healthy cycle proven live; STOP-2 pending 2026-07-21 12:39:12 +02:00
admin 98adb72204 chore(v0.92.1): supersede v0.92.0 — the published artifact predates the sudoers grant
v0.92.0 was published before live verification showed three of the four guestnet probes had
no sudoers grant, so that artifact lacks FELHOM_GUESTNET and the guestnet-* capability rows.
Superseded, not overwritten: a published version stays immutable (the v0.91.0 -> v0.91.1
precedent). Watchdog code is unchanged; the functional fix is the sudoers file, which must
ship with the binary.
2026-07-21 12:33:05 +02:00
admin 0e8fd814ba fix(v0.92.0): the guestnet probes need their own sudoers grant (found live)
The first sweep on felhom-pve logged 'dhclient liveness probe failed: sudo: a password is
required' and reported state=unknown — the watchdog fail-safed correctly, but it was blind.
TASK-D assumed no sudoers change; the existing grant only covered lanresolver's address read.

FELHOM_GUESTNET adds four FIXED vectors (route, interfaces, pgrep, and the incident's heal
invocation). Every argument after the numeric vmid is a literal, so nothing the guest or the
hub says can widen the grant. Plus four guestnet-* capability rows, non-critical, so a host
that has not taken the new sudoers is visibly degraded without paging an operator per box.
2026-07-21 12:32:27 +02:00
admin c0966d753d feat(v0.92.0): guest-network watchdog (R-54) — supervise the guest's DHCP client
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).

So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.

internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.

guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.

Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.

Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.

Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
2026-07-21 12:29:37 +02:00
admin 08b55a1015 docs(report): STOP-2 DONE + PROVEN — the R-39 chain closed in 13 seconds 2026-07-21 10:53:16 +02:00
admin 042a401e71 docs(report): STOP-3 DONE; STOP-2 still pending (offsite re-issue was pressed, not PBS-DR) 2026-07-21 10:36:53 +02:00
admin edac02ec9f docs(report): TASK-B agent half — v0.91.2 shipped + deployed; STOP-2/3 need the operator 2026-07-21 10:23:05 +02:00
admin d4a7a5bad3 v0.91.2 — log a healthy credential probe at Debug
The probe logged only on failure, so a healthy one was silent: "no auth_failed" was
indistinguishable from "never probed", and the leg could not be demonstrated as running.
That is exactly how v0.91.0 shipped it inert unnoticed.
2026-07-21 10:20:10 +02:00
admin dcf3a5530b v0.91.1 — wire the credential probe (v0.91.0 shipped the seam inert)
v0.91.0 built the AuthSink seam and the NoteAuthResult consumer, and main.go never
called SetAuthSink. The reporter skips probing when no sink is attached, so the entire
auth-honesty leg was silently inert — no probe, no auth_failed, no self-heal — and
nothing failed, because every unit test injected the sink directly.

Caught during STOP-1 live verification by checking the wiring instead of trusting it.
Same class as the controller v0.154.0 defect the day before: a table test over a seam
proves the seam, not the caller.

The published 0.91.0 artifact is superseded, not overwritten — a published version stays
immutable. TestLiveReporter_NoSinkMeansNoProbe pins the no-sink-no-probe contract so the
inert case is documented behaviour rather than an accident; only live evidence can prove
the wiring itself.
2026-07-21 10:18:41 +02:00
admin ab8f682f0b test: assert the agent upgrade alone is hash-neutral (no spurious re-apply at STOP-1) 2026-07-21 10:15:24 +02:00
admin b2ca63ee9f v0.91.0 — the DR tier can no longer be applied and dead at the same time (R-39 + R-50b(a))
Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.

Three compounding defects let a box report `applied` while every PBS request 401'd:

1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
   token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
   content hash never moved — the converged agent short-circuited and never consumed the
   fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
   hash now, because descriptorHash marshals this struct.

2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
   through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
   read verb — so the target resolver got "permission denied" every cycle, warned, and
   skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
   narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
   secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
   refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.

3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
   sentinel) runs on the 15-minute collect path and its verdict becomes a loud
   `auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
   or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
   unauthorized, since re-keying a too-narrow token would mint forever without fixing
   anything. A transport error is UNKNOWN, never a rejection — otherwise every network
   blip burns a credential. Recovery self-clears.

R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.

Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
2026-07-21 10:12:31 +02:00
admin 8c55ac7fda docs: TEMPORARY remote-site access for felhom-pve (tailnet) — see AUDIT-vacation-remote-ops-2026-07-20
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nhk3eBHT8Mg5L8c2aj57aU
2026-07-20 08:07:43 +02:00
admin d81fddc7b1 docs: migrate workflow to DooPlex-local execution 2026-07-19 12:16:13 +02:00
admin 8fc44d8d9e docs(report): golden bakes every infra image (build tooling v2.1.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-19 10:01:11 +02:00
admin 25c30ba625 build-golden v2.1.0 — bake EVERY infra image, asked from the controller
Enabling Megosztás on a fresh box pulled felhom-samba from the registry with
zero feedback — minutes of silent nothing. Observed live, twice.

Cause: this script carried its own hand-maintained array of three image tags,
with a comment telling the reader to keep it in sync with the controller's
internal/infra constants. It drifted the moment a fourth stack was added.
felhom-samba was never added here, so the golden baked 3 of 4.

The fix is structural rather than a fourth copy of the list: it now comes from
the controller image the bake just pulled (--print-infra-images, backed by
infra.Images(), which derives from the pins themselves). The golden bakes
exactly what THAT controller version will request, so the two cannot disagree.

Ordering fix this exposed: docker logout ran immediately after the controller
pull, but felhom-samba is on the same private registry, so the infra loop would
have 401'd. Logout moved after the loop, plus a hard assertion that no
credential remains in the guest before it is archived.

Pre-0.147.0 controllers have no such flag; the bake falls back to the historical
3-image list and says so loudly — the fallback IS the drift-prone thing being
removed, so it must never pass silently.

No agent version bump: build tooling only. Effective at the next golden build;
the current golden is not rebuilt for this.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-19 09:31:12 +02:00
admin f22f70ce58 docs(report): R-39 — the demo box is healed and proven
Supersedes the "not yet healed" hand-off section. Viktor clicked "Re-issue PBS
credentials" and the chain closed in 9 seconds: hub re-issued 20:28:44 -> agent
consumed 20:28:51 -> `pbsdr: converged state=applied` 20:28:53, with the PATCHED
wrapper. consumed-failed.json cleared automatically, marker.json rewritten. The
agent picked it up unaided because marker.json was absent, so the L235 hash
short-circuit did not apply.

Proof the tier is alive rather than merely reporting alive:
  pvesm status        401/inactive -> ACTIVE
  token /version      401          -> 200
  real backup         none         -> felhom-pbs:backup/ct/9201/
                                      2026-07-18T18:31:06Z, 9744319312 B, 13m36s

Encrypted under fingerprint 7e:a6:af:f7:ea:6d:3e:d9 — the ESCROWED key, the one
customer zero holds the recovery code for. First real DR-tier backup on the
reborn box.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-18 21:00:09 +02:00
admin 9596d5a8d0 v0.90.1 — R-39 hotfix: PBS reconcile must not pass --server to pvesm set
Config-only (wrapper + red-proof); the Go binary is unchanged, so this ships
with the next agent deploy as a config artifact.

The reconcile verb built `args=(--server "$server" --fingerprint "$fp")`. PVE
treats a PBS storage's `server` as a CREATE-ONLY parameter and rejects the
ENTIRE `pvesm set` call — "can't change value of fixed parameter 'server'" —
even when the value passed is byte-identical to the stored one. So reconcile
could never succeed against an existing entry; it exited 255 every time.

That is severe rather than cosmetic because the agent consumes the hub's
ONE-TIME PBS token secret BEFORE invoking the wrapper. Each hub "Re-issue PBS
credentials" therefore minted a secret, the agent burned it, the wrapper
rejected the apply, and the entry stayed pinned to the revoked credential —
a PBS DR tier authenticating 401 indefinitely while the agent reported
`pbsdr: converged state=applied`.

Live-diagnosed on the N100 during the rehearsal wrap (felhom.eu
tests/VALIDATION-n100-rehearsal-2026-07-18.md F2, ROADMAP R-39). Proven on the
live entry before writing code: `pvesm set <id> --server <same> --fingerprint
<same>` -> rejected; the same call without --server -> rc 0. K (<id>.enc) and
the .pw store verified byte-untouched after the rejected call — PVE rejects
atomically, so the set-only law held.

Fix: drop --server. The server address is immutable by construction (relocating
a PBS endpoint needs a fresh create), so there was never anything to reconcile
there. --fingerprint (+ --password when a secret is fed) remain.

Red-proof TestReconcileNeverPassesServerToPvesmSet: isolates the reconcile)
block from the shipped wrapper, asserts no --server reaches `pvesm set` and
that --fingerprint is still pushed. Verified RED on the unfixed wrapper, GREEN
after. Handles two vacuous-pass traps that both fired while authoring it: the
pattern is line-ending tolerant (\r?\n — this repo is cloned on Windows, and an
\n-only pattern matches nothing and passes silently), and comment lines are
stripped before matching (the WHY note quotes the very flag under test).

NOT fixed here, both still open and riding the spec'd R-39 agent train:
 1. R-39's primary half — the agent re-applies on a change of the DESCRIPTOR
    HASH (manager.go ~L235), but a credential re-issue leaves the descriptor
    byte-identical (same token_id/fingerprint; only the side-table secret
    rotates) and bumps only the generation, so a converged agent still ignores
    a fresh secret. This makes the apply succeed once it re-applies; it does
    not make it re-apply.
 2. The verify loop reads /etc/pve/priv/storage/<id>.pw directly as non-root —
    a path it can only ever WRITE through the root wrapper (0700 root:www-data;
    sudoers exposes create|reconcile|grant, no read verb), so it is permanently
    blind to the failure it exists to catch.

Demo box: wrapper hotfixed in place (.bak-20260718-preR39 kept). NOT yet healed
— diagnosis consumed the pending secret against the unfixed wrapper; the agent
parked correctly in consumed-failed (no burn loop). Healing needs Viktor to
click "Re-issue PBS credentials"; the agent will then pick it up unaided
(marker.json absent, so the L235 short-circuit does not apply).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-18 20:27:14 +02:00
admin f222a7b366 docs: repoint build path to /mnt/5_hdd/felhom.eu (DooPlex build root moved off SSD 2026-07-18) 2026-07-18 09:55:47 +02:00
admin 8208a80cc7 docs: v0.90.0 REPORT + CONTEXT + REUSE (guest RAM resize R-24 + fast-tick R-28; Phase-0 proof; deploy verified both hosts; floor = operator GO) 2026-07-17 19:25:54 +02:00
admin ac112c956e v0.90.0 — guest RAM resize (R-24) + fast-tick-until-convergence (R-28)
MinAgent coupling: felhom-controller v0.143.0 gates its guest-memory-resize UI on
this agent (FeatureGuestMemoryResize, MinAgent 0.90.0).

R-24 guest RAM resize (internal/localapi/guestmemory.go): self-scoped GET/POST
/guest/memory. Agent enforces every bound FRESH per request (min 2048, max
host_total-2048, shrink floor max(2048, usage+512)); applies via PVE SetConfig —
live cgroup apply, no reboot (Phase-0 proven on the nested demo box). Verify-after-apply
re-reads maxmem before claiming success. New narrow MemoryOps seam (GuestAPI untouched);
Options.Memory nil -> 503. Memory only.

R-28 fast-tick (internal/fasttick): while any desired-state item is unapplied -
including the pre-tunnel window a hub poke can't reach - pulse the shared out-of-band
trigger every 30s, self-disarm on convergence. Four cached sources (desired-gen==0,
reconcile Planned-Pending>0, pbsdr waiting_secret only, wgtunnel desired-not-operational);
LOUD pbsdr states + pending_signature excluded. Seams: reconcile.Engine.LastResult() +
wgtunnel.Manager.TunnelConvergence() (cached, no per-tick exec).

Guests-0/0: hypothesis REFUTED live (9201 IS a pool member; 0/0 was the pre-provision
window; PoolAddVMID re-assert already covers restore-over-existing). No code change; the
fast-tick mitigates the window.

Tests + red-proofs (i floor guard, ii max guard, iii always-pulse) all restored green.
2026-07-17 19:09:40 +02:00
admin 9127f547f9 docs: v0.89.0 live legs PROVEN — Scenario 4 A/B train, Scenario 1 R-22 self-heal (~3s), Scenario 3 poke→tick (~31ms) 2026-07-17 08:32:07 +02:00
admin 9effc85a47 docs: v0.89.0 REPORT + CONTEXT (self-grant R-22, escrow live-reload, poke listener; live legs operator-gated) 2026-07-16 22:59:14 +02:00
admin a659e5dc09 v0.89.0: pbsdr self-grant (R-22) + escrow config live-reload + agent-plane poke listener (Direction-2a)
- pbsdr: on a 403 pre-check (non-default storage id, no ACL yet) self-grant via the root wrapper then re-read, instead of aborting before the grant — closes F4/R-22. Red-proof TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant.
- escrow preflight: late-bound CurrentPBSStorageID re-reads agent.json so a pbsdr-seeded pbs_storage_id flips the row green in-process (no restart). Red-proof TestEscrowPreflight_PBSStorageIDLiveReload.
- internal/poke: contentless UDP poke listener bound exclusively to the box WG /32 (port 51822), leading-edge debounced, fires the hub-loop out-of-band trigger for an immediate desired-state cycle. First slice of R-13. Red-proofs TestBindConfinement + TestDebounceCoalescesBurst.
2026-07-16 22:47:22 +02:00
admin c040c180e9 docs: v0.88.0 REPORT (deploy + endpoint-exact live validation: 63/63 caps both hosts, ceremony ~4s, one-shot claim + 410, hub hash covering; red-proofs 1/2/3/6 run-fail-revert) + CONTEXT 2026-07-13 19:24:17 +02:00
admin 1c3a3ef9ad v0.88.0: controller-driven escrow ceremony — --output=json machine mode (escrowCeremony extraction, text mode byte-identical), the ONE fixed argv (escrow.CeremonyArgs, shared by exec+manifest+FELHOM_ESCROW sudoers, pin-tested), localapi ceremony job (single-flight, 60s) + one-shot in-memory R claim (10min TTL, unclaimed_void) + preflight; escrow-ceremony capability (Critical, pbs_dr-gated) 2026-07-13 19:01:11 +02:00
admin adf7882f7d docs: §13.2 wizard leg COMPLETE (offered/enrolled/formatted via operator UI + GO; pipeline-level teardown, boxes as found) + Day-0 manifest vouched 0.87.0; cross-ref the two controller bugs the leg surfaced (0.126.3/0.126.4) 2026-07-13 14:15:02 +02:00
admin 4e6171442f docs: v0.87.0 REPORT — deploy transcript (drill first, then felhom-pve 62/62 caps), 13.3 byte-identical regression, the live IA replay (hot-added /dev/sdb now offered; sda never), Gitea publish sha; supervised UI leg + manifest vouch remain 2026-07-13 13:22:24 +02:00
admin 3c174bc6f2 agent v0.87.0: SystemDisks device-mapper walk — legacy-boot hosts get a working drive wizard (IA finding 2, MEDIUM)
Operator ruling 2026-07-13: walk the root's backing device through /sys/block/<dev>/slaves
recursively down to physical disks (dm AND md; topology, never VG names); those + any mounted-ESP
holder are system; the all-system fail-safe returns to being the WALK-FAILURE error case only.
SAFETY DIRECTION: a root-backing disk classified candidate is made impossible — per-branch
conservatism (any unresolvable slave fails the WHOLE walk -> ok=false -> the unchanged all-system
path).

- physicalDisksOf/walkSlaves in role.go (symlink canon -> wholeDiskOf fast path -> recursive
  slaves walk; cycle/depth guard; non-/dev sources unwalkable)
- HostReader.BlockSlaves(name) — the ONE new seam method; ProcHostReader reads
  /sys/block/<name>/slaves; all four test fakes mirror it
- role_walk_test.go: signature table (root-backing disk ALWAYS system across legacy-LVM /
  md-raid / EFI+raw / EFI+LVM / nested dm-on-md — NEVER weaken) + dead-wizard-lives +
  dangling-slave fail-safe (real sysKnown=false path) + cycle + empty-slaves; red-proofs A/B/D
  run->fail->revert (recorded in REPORT)
- §3 spike transcripts (drill legacy: dm-1->sda3->sda; felhom-pve: ESP+walk agree on sda ->
  byte-identical regression); caller audit: none relied on all-system as a feature
- format/mkfs paths, data-bearing guards, wizard UI untouched
2026-07-13 13:15:03 +02:00
admin c20814e6c2 agent v0.86.0: DR-tier-by-default — capability inactive state (GatedBy/GateActive, pbsdr gate via DRConfigured) + F-3 root-run provision parent ownership
Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 20:06:27 +02:00
admin bcb8dad2aa agent v0.85.0: CHANGELOG + REPORT — boot/recovery plane shipped + live matrix (F12 dead, self-heal proven)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 08:26:14 +02:00
admin dbd6d4c57a test: localapi invisible-share test asserts re-arm + re-verify
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 07:59:20 +02:00
admin b6300250aa F11 matrix-correction: re-arm on guest-blind (active mounts not inherited by rebooted guest — live finding)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 07:58:39 +02:00
admin 47eb0bf967 agent v0.85.0: boot/recovery plane (F12/F11/F10/F9/F2/F1) + appliance self-heal — CHANGELOG
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 07:52:11 +02:00
admin 0b55a11667 test: update RemoveNetworkMount_Commands for F1 rmdir step
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 07:50:48 +02:00
admin a527b29ac3 test: fix unused var in reassert test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 07:49:58 +02:00
admin bc4eda926b agent v0.85.0 WIP: F12/F11/F10/F9/F2/F1 boot-recovery plane + appliance self-heal (pre-build)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
2026-07-12 07:48:49 +02:00
admin bec4bac076 docs: REPORT — agent 0.84.0 PUBLISHED to Gitea (anon-verified sha 2e31b9a8…); operator: manifest vouch pending
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 21:48:27 +02:00
admin a8b773f892 docs: REPORT for v0.84.0 (deployed felhom-pve; Q1c reboot GREEN; equivalence proven, no sudoers widening)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 21:25:10 +02:00
admin 474b858c0b v0.84.0: ReassertNetworkMounts — NAS automount survives guest reboots (RCA fix 1)
Storage §8 decision table (stop + enable --now on idle triggers; active mounts untouched),
daemon leg at startup with per-running-guest visibility verify, guest-hook post-start leg
(root, direct systemctl, non-fatal). Red-proofs: always-rearm table FAIL; unwired hook FAIL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 20:46:59 +02:00
admin 0df72ea643 docs: REPORT + CONTEXT + CLAUDE logging rule for v0.83.0 (deployed felhom-pve)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 17:18:55 +02:00
admin cb692f8788 v0.83.0: observability pass — always-DEBUG capture ring + GET /debug/logs + heartbeat log-pull + gap-fill sweep
Capture layer: applog.New returns (logger, Ring) — slog fan-out, stderr at the
configured level, ~1000-entry ring fixed at LevelDebug (remote diagnostics
without a config flip). GET /debug/logs (token-authed, ?raw=1) + request-level
DEBUG middleware. Heartbeat log-pull mirrors the report logtail pattern:
envelope log_tail_requested -> next heartbeat carries log_tail (128KB cap,
consume-once, failed-push retry proven). Gap-fill sweep over netverify/
netstorage/netmount/signedjobs/selfupdate/disks/controller-swap/desired/loop.
Red-proofs: ring-at-emit-level FAILs capture test; drain removed FAILs
consume-once; dropped phase line FAILs the S7 log-sequence smoke.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 16:24:07 +02:00
admin 461eaf42c1 docs: REPORT v0.82.0 (version channel deployed) + REUSE version-channel seam
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 15:45:04 +02:00
admin fa9c7fe198 v0.82.0: X-Felhom-Agent-Version response header — the controller capability channel
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 14:51:48 +02:00
admin 1e60e88eb2 docs: REPORT for v0.81.0 (deployed; task-wide record in felhom-controller/REPORT.md)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 10:32:19 +02:00
admin added9d226 v0.81.0: NAS verify-before-commit — retry=0, journal classifier, detached verify job + auto-rollback
Agent half of the verify-before-commit task (SPIKE-nas-verify-2026-07-11, b57f6c1):
retry=0 in the production NFS options (Q4-vi); ClassifyNetVerifyFailure on the live
Q4 strings (nfs_export merges not-found/not-permitted); add = sync fast-fail (2s TCP
pre-probe, nothing installed) + detached in-memory verify job judging /proc/mounts
only, auto-rollback on failure; GET /netstorage/verify-status (phase none = the
controller's Scenario-F rollback signal); unprivileged journalctl (systemd-journal
group, NO new sudoers grants).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 09:44:06 +02:00
admin 300f06722b REPORT: v0.80.0 slice-2 deployed + adoption-proven + published; Peti phase pending
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 07:26:35 +02:00
admin 3d1323cc09 pbsdr: escrow seed writes in place (config dir is root-owned; tmp+rename impossible for the non-root agent — demo live finding); recovery copy parked in the state dir
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-10 21:59:51 +02:00
admin e5e8f3920a v0.80.0: PBS DR tier slice 2 — the apply-bridge (pbs_dr consumer, felhom-pbs-apply set-only wrapper, verify-pin-before-consume, adoption-first, loud consumed-failed, escrow seed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-10 21:50:36 +02:00
admin a6e8bcb475 docs: v0.79.0 REPORT — deployed felhom-pve (56/56)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 23:25:10 +02:00
admin 301c84d9b5 v0.79.0: escrow upload carries restic_pw_sha256 (SLICE 3 auto-confirm, agent third)
HashResticPassword = sha256 hex over the trimmed password (pinned
cross-repo vector). escrowUploadRequest gains restic_pw_sha256,omitempty
— set only when a staged password was sealed into the blob. Contract test
updated; hub mirrors next.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 23:05:30 +02:00
admin bd9e777f41 docs: v0.78.0 REPORT — DELETE stage-secret deployed felhom-pve (56/56), live-validated
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 22:47:22 +02:00
admin 98ea1ed49c v0.78.0: DELETE /escrow/stage-secret — idempotent staged-secret wipe
The staged offsite repo password was wiped only by the escrow-create
ceremony; a confirm without a fresh ceremony (password already escrowed)
left the 0600 staged file behind. The controller calls this on every
EscrowState flip to escrowed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 22:19:10 +02:00
admin 4449118783 docs: v0.77.0 REPORT — fork-4 offsite password escrow; deployed felhom-pve (56/56 caps)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 15:19:31 +02:00
admin 0c22b9bbf3 v0.77.0: fork-4 — escrow the offsite restic repo password under R
IdentityBundle gains ResticRepoPassword (rides existing age-under-R
WrapIdentityBundle; custody spike febdc56 proved a recovered value opens the
real repo). POST /escrow/stage-secret (withGuest, scopedFromBody) transiently
stages the controller-pushed password (0600, atomic, NEVER logged), which the
escrow-create ceremony auto-injects then wipes. Adds AttachResticPassword +
StagedResticPasswordPath + WipeStagedResticPassword; EscrowStagePath injectable
for tests. Tests: bundle carries pw byte-exact + not-in-blob + wrong-R fails
closed; stage 0600 + non-secret ack + cross-guest 403 + value-not-in-log.
Additive; PBS-K escrow untouched. NOT yet live-validated (supervised ceremony).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-09 14:56:09 +02:00
admin 7f07393623 docs: v0.76.0 CHANGELOG + REPORT + CONTEXT (GL-5b shipped, published)
AGENT_VERSION=0.76.0
AGENT_SHA256=9828c5f75e719fb2e1fc3523f9c322b595a963ec1dcdf37ca42a96bedab7f50b

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 09:55:19 +02:00
admin b1697874ec GL-5b: restore-test full-fidelity verification (v0.76.0)
The restore-test had GL-5 finding #2's mirror image: its live-source-config
bind-override path tripped PVE's drop-unlisted-mountpoints rule, so scratch
guests boot-verified WITHOUT their storage mpN - weaker verification than
claimed. Params now derive from the ARCHIVE's own embedded config via
ExtractArchiveConfig + drRestoreOverrides (the object under test; full
layout, content genuinely extracted - the added runtime IS the
verification); unreadable/unknown-topology archives refuse up front. NEW
mount-parity assert (2b, pre-start): restored mpN set vs the archive's -
missing/mispathed/undersized/extra mpN fail the test naming the delta, so
constraint (b) can never regress into a green light. MountParity +
MountInventory ride the result + hub wire record (additive). Dead
bindMountOverrides/archiveVMID path deleted with its tests (no reachable
lookalike). DR bring-up untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 09:46:30 +02:00
admin 5a72a4b59c docs: v0.75.0 CHANGELOG + REPORT + CONTEXT (GL-5 shipped, live-validated)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 09:24:14 +02:00
admin b3446213df GL-5: DR restore passes the FULL archive layout (live finding #2)
The live scratch DR exposed the second half of PVE's all-or-nothing
explicit-params restore: mountpoints NOT named in the params are silently
DROPPED - the DR guest came up without its mp0/mp1 data volumes (boot
passed; the customer's world did not ride along). drRestoreOverrides now
derives the COMPLETE param set from the archive's extracted config:
explicit rootfs, every storage-backed mpN passed through (size + in-guest
path + backup flag preserved so vzrestore extracts its content), the two
structural binds replaced by 4d-swapped throwaways; unknown bind mpN or
unparseable size refuses loudly. Snapshot sections never shadow the
current config.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 09:10:54 +02:00
admin 3bf0110697 GL-5: explicit rootfs override for DR restore (live-discovered PVE constraint)
The live validation hit PVE's all-or-nothing restore rule: mpN params
without an explicit rootfs -> HTTP 500 "mount points configured, but
'rootfs' not set" (the same constraint restoretest.go:211 documents for the
live-config path; the spike never ran an override restore). The lost guest
has no live config, so the rootfs SIZE now comes from the archive's own
embedded config via NEW Client.ExtractArchiveConfig (GET vzdump/
extractconfig - verified live: answers 200 under the scoped agent token;
PBS keys stay server-side, the spike's candidate-1 rejection holds; used
for the SIZE ONLY - the bind layout stays the platform constants).
Unparseable/unreadable archive config -> clean refusal before any restore.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 09:02:36 +02:00
admin c12b512316 GL-5: DR bring-up structural bind overrides + 4d real-bind swap
ModeDRGuestLoss now passes restore-time MountOverrides for the two
platform-constant structural binds (mp8 parent, mp9 bootstrap) via the
shared throwaway-volume format helper - without them a customer-archive
restore under the privsep token fails outright ("restoring 'mp8' to bind
mount is only possible for root"). New post-restore step 4d swaps the real
binds in via the host runner (root pct set, one slot per call), deletes the
displaced unusedN volumes (API config PUT; a scoped-token refusal logs the
residue loudly instead of widening privileges), and respects the
committed/launched rollback envelope. Provision passes nil overrides -
byte-identical behavior (regression contract test).

Engine grows an optional HostRunner + StateDir seam (DR refuses up front
without a runner); selftest bring-up wires the ExecRunner + cleans the
scratch mp9 host dir on teardown; proxmox.GuestConfig.Unused() added.
6 new tests incl. C2 mid-swap rollback + C3 older-archive + 403-warn paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 08:48:11 +02:00
admin 4c40846769 docs: CONTEXT — v0.74.0 Gitea-published (RUNBOOK GL-1); vouch = operator step
Closes the standing publish-0.6x + Day-0-vouch OPEN items (0.64-0.73 never
published; 0.74.0 is the vouch target, sha 1ec3f588..76af05).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-07 21:07:17 +02:00
admin ced60ddf76 docs: v0.74.0 REPORT + CONTEXT — pool re-assertion; R1 was a symptom, restore-test works live
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-07 19:07:26 +02:00
admin ca0b169a4e feat(reconcile): re-assert pool membership after restore-over-existing (campaign-2 R2, v0.74.0)
Pool membership is what lets the pool-scoped token reach a guest; pct restore
--pool sets it only at CREATE, so a restore over an existing VMID drops the guest
from the felhom pool and 403s the next restore-test/DR on VM.Audit. This empty-pool
state is the true root cause of the campaign's "R1" (bind-mount restore failing was
a symptom — restore-test's existing bind neutralization never ran without config-read).

Add Client.PoolAddVMID (PUT /pools, additive+idempotent, Pool.Allocate) and call it
in bring-up after liveness when spec.Pool!="" — warn-not-fail on a hiccup (liveness
wins). B3 scratch-teardown 403 diagnosed as a cascade (restoretest already passes
Pool). Role/ACL untouched. Tests + red-proof.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-07 18:42:19 +02:00
admin e04b75e1f8 docs: v0.73.0 REPORT + CONTEXT — F2 mount-role fallback closed, live-validated
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-06 20:41:58 +02:00
admin 7545af8a2c fix(localapi): F2 mount-role fallback — enrolled user-data drives ejectable again (v0.73.0)
roleForMountPath resolved role only from the PVE storage view; a bind-mounted
RAW enrolled user-data drive is not a PVE storage, so it fail-safe'd to system
and the eject/decommission gates 403'd EVERY user-data drive in the standard
topology (campaign F2, where=/mnt/teszt_enroll role=system). Add a mount-table
fallback mirroring durableIDForMount Impl-2b: device-keyed classification with a
whole-disk containment pass (new storage.SameWholeDisk) and the Observe-error
early return kept BEFORE the fallback (else a blind view -> permissive). Only
roleForMountPath touched. Tests A1/B1/B2/C1-C3 + 3 red-proofs; existing RoleGated
tests green unmodified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-06 20:07:45 +02:00
admin 26664d6d51 docs(H1): v0.72.0 CHANGELOG + REPORT + CONTEXT (OOB operator access, live-validated)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 23:03:32 +02:00
admin 223c235f01 fix(felhomsshd): operator_peer_configured from belt @operator_ips (agent-readable)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:58:58 +02:00
admin 72d6132907 fix(felhomsshd): reachable=listener-present (belt blocks a dial); operator/key configured from persistent state
reachable now checks a bound listener via ss (the belt correctly drops even
localhost→felhom-sshd, so a dial always failed). operator_peer_configured derives
from the persistent wg-felhom.conf (2nd AllowedIPs /32) + operator_key from the
installed authorized_keys file — accurate immediately after an agent restart,
before the next desired-state fetch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:57:35 +02:00
admin 2bf9354c0e fix(felhomsshd): don't empty the belt/authkeys on an unfetched block (operator lockout)
Mirrors wgtunnel fetched=false-is-never-a-teardown: until the desired-state is
delivered, @operator_ips + felhom-op authorized_keys are left untouched (a nil
block on agent restart would otherwise empty @operator_ips and lock the operator
out until the next fetch). Belt.Sync split into SyncPort (always) + SyncOperator
(fetched only).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:51:30 +02:00
admin b99d02b7a9 fix(felhomsshd): keep the persisted port unconditionally (no self-listen flip-flop)
On a continuous reconcile felhom-sshd itself listens on the claimed port, so
re-probing isFree(persisted) found it 'busy' by our own daemon and thrashed to
another candidate every tick. A persisted port is ours — keep it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:43:22 +02:00
admin d880289b06 fix(felhomsshd): persist claimed port in agent StateDir (non-root can't write /etc)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:41:14 +02:00
admin a34aac64d0 feat(configs): OOB static belt table + loader unit + felhom-op sudoers (H1 Part 5)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:33:47 +02:00
admin 9d9e4a5ea7 test(hub): collector OOB stanza wiring (H1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:30:01 +02:00
admin c983a25609 feat(felhomsshd): dedicated OOB sshd instance + port-adaptive belt (H1 Parts 2-4 agent)
internal/felhomsshd: agent-managed felhom-sshd (claim port [8822,2222,8022,62222]
loud-fail-on-exhaustion; render config→sshd -t→reload never-restart-on-change
[SF-2]; operator authorized_keys from the hub block outside ~/.ssh [SF-3]); the
static-table nft belt mutating ONLY @operator_ips + @ssh_port [trap 4]; health/heal
(reset-failed-then-restart with 10min cooldown, NEVER restart onto an invalid
config) + the oob heartbeat stanza. configs/felhom-sshd.service (SAFE, no
RuntimeDirectory [SF-1]). FELHOM_SSHD + FELHOM_OOB sudoers (set-elements only).
oob.enabled config DEFAULT FALSE. Wired into main like wgtunnel.

Non-hollow tests: claim clean/contention/idempotent/exhaustion; config
safe+byte-stable+refuses-:22; belt mutate-then-idempotent + never-touches-rules;
heal no-restart-on-invalid-config + cooldown; status reflects block.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:27:02 +02:00
admin effff53f99 feat(wgtunnel): render operator OOB /32 into wg-felhom AllowedIPs (H1 Part 1, [OF-1])
renderConf appends a validated, deterministically-SORTED oob_peer_ip /32 so the
operator peer survives self-heal/restart (a runtime wg set was wiped by self-heal).
WireWireguard gains oob_peer_ip (omitempty — absent = byte-identical pre-H1 conf).
Non-hollow tests: sorted+byte-stable, lower-IP-sorts-first, absent-identical,
invalid/v6 rejected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 22:08:04 +02:00
admin 4b7c5bf128 docs(context): v0.71.0 break-glass watchdog + mgmt_plane shipped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 19:24:12 +02:00
admin 8675bff3da docs(G1): REPORT.md — break-glass agent half (v0.71.0) live-validated
Auto-heal drill with the agent STOPPED healed /run/sshd in 30.0s (measured);
mgmt_plane stanza + hub mgmt_plane_healed warning proven end-to-end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 19:23:05 +02:00
admin fd4e177216 feat(mgmtplane): break-glass privsep-dir watchdog + mgmt_plane health (TASK G1) — v0.71.0
Prerequisite for felhom-sshd (H1). Closes the SPIKE-felhom-sshd §8 lockout: a
second sshd's RuntimeDirectory=sshd removed the SHARED /run/sshd privsep dir and
took stock sshd on :22 down (sessions reset after KEXINIT).

Host artifacts (configs/, installed by felhom-host-install):
- felhom-privsep.tmpfiles: layer 1, boot-persistent /run/sshd owned by no unit
- felhom-mgmt-watchdog.sh/.service/.timer: layer 2, AGENT-INDEPENDENT ~60s heal
  (stat-first recreate + reset-failed sshd only if failed + heal-marker); never
  RuntimeDirectory=, never restarts stock sshd, never touches a healthy dir.

Go (internal/mgmtplane): read-only Reporter → additive omitempty mgmt_plane
heartbeat stanza (privsep_dir_ok/sshd_reachable/healed_recently/privsep_healed_at),
wired via Collector.SetMgmtPlaneReporter. Non-hollow tests + red-proofs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 18:49:27 +02:00
admin 1c75a45a42 docs(context): D1 live-validated — 4 drills PASS, box on canonical v0.70.0
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 16:04:38 +02:00
admin 084fc691f5 docs(agent): D1 REPORT — v0.70.0 shipped + live-validated (happy path, ~2s crash-rollback, no-pending guard, gate refusal)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 16:04:11 +02:00
admin 915642aaaa docs(agent): D1 — README self-update section, REUSE, CHANGELOG v0.70.0, CONTEXT
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 15:36:13 +02:00
admin 8033a522cd feat: D1 Part 2 — agent self-update Go plumbing (op class, opsign, executor, commit, report)
- reconcile: ClassAgentUpdate op class; always Destructive (no provenance
  blesses replacing the root-adjacent binary). classify test + companion
  (TestClassify_AgentUpdateAlwaysDestructive).
- opsign: `-op agent_update` with -agent-version + -sha256 (isHex64-validated);
  params {version,sha256}. isHex64 test (Group D).
- config: SelfUpdateConfig{URLTemplate,Username,Token,StateDir,DwellSeconds}
  + WithDefaults + Token redaction.
- internal/selfupdate: Executor (download → verify vs the SIGNED sha → sudo -n
  wrapper `apply`; sha is the only integrity root — mismatch refuses + removes,
  agent untouched); Manager (startup dwell → `commit`; version-mismatch → no
  commit + loud WARN + marker left for report visibility; shutdown-before-dwell
  leaves pending). WrapperRunner seam → tests never shell out.
- hub report: additive selfupdate_pending(+version) via SetSelfUpdateReporter
  seam; both omitempty (Wireguard precedent) so the cross-repo golden contract
  stays byte-stable — no hub change.
- capability manifest: 3 non-critical FELHOM_SELFUPDATE probes.
- main.go: updateExec appended to the executor chain; commit-manager wired to
  the report seam + MaybeCommit goroutine after core init.

Tests: Group A (executor happy/sha-mismatch+companion/bad-params/wrapper-fail),
B (agent_update rides the real gate: pinned-key executes, non-pinned +
retarget rejected), C (commit/version-mismatch/no-pending/shutdown), D (opsign).
C2 companion red-proof verified (neutered Go verify → bad binary reaches apply
→ test fails), reverted. Full go test ./... green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 15:32:15 +02:00
admin b7cbded429 feat(configs): D1 Part 1 — self-update host artifacts (guarded wrapper, rollback unit, limits drop-in, sudoers alias)
Design provenance SPIKE-agent-selfupdate-2026-07-05 (SF findings binding):
- felhom-selfupdate-guarded: apply <staged> <sha256> / commit / rollback.
  Ordering [SF-7]: temp-sweep → path confinement (staging dir only, no '..')
  → 64-hex + sha256 RE-verify as root (BEFORE .prev) → same-fs assert →
  .prev snapshot → root-owned staging copy → atomic mv → pending marker →
  reset-failed [SF-4/5] → detached systemd-run restart, verbatim [SF-6].
  rollback is pending-guarded (no pending → exit 0 no-op, [SF-1]) and clears
  pending BEFORE its restart so per-crash OnFailure re-fires no-op. commit
  idempotent, .prev retained (S3d). No env-overridable paths (path-fixedness
  is the security property). shellcheck clean.
- felhom-agent-rollback.service: Type=oneshot OnFailure target; comment block
  documents the systemd-257 per-crash firing reality [SF-1].
- felhom-agent-limits.conf: [Unit]-ONLY drop-in [SF-3] with the spike's tuned
  values verbatim [SF-2]: StartLimitIntervalSec=120, StartLimitBurst=4,
  OnFailure=felhom-agent-rollback.service.
- sudoers: FELHOM_SELFUPDATE alias (coarse apply glob per S4b — the wrapper
  re-verify is the real gate) appended to the grant line. visudo -cf OK.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-05 15:20:37 +02:00
admin 72d7f05bf1 docs: REPORT + CONTEXT for S5 safe halves (WG-key install + directive→plan)
Parts 1-2 shipped + deployed (v0.69.0); Part 3 not needed (sqlite3 export);
Part 4-A re-attach safety already unit-proven; Part 4-B destructive 9201 drill
prepared + operator-gated (pre-flight green), not executed by CC.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 21:12:19 +02:00
admin bd4bced771 dr: recovered WG-key install + host_loss directive→restore-PLAN (S5 safe halves)
wgtunnel.InstallRecoveredKey: write an escrow-recovered WG private key (create-
only, refuse-overwrite) so the tunnel re-establishes with the same identity/pubkey
(same /32), no keygen. Wired into identity-consume -install-wg-key (opt-in;
pre-S3 blob → logged fresh-keygen fallback). Value never logged.

internal/dr (new): consume the host_loss restore_directive (was logged-ignored)
into an inspectable RestorePlan via the AddConsumer raw seam — per guest
{vmid,archive,target,sizing} + per drive {durable_id→mount} + offsite PBS coord.
DERIVE-AND-SURFACE only; the Consumer has no restore/destroy dependency (execute-
nothing is structural). guest_loss/absent → no plan.

Tests + red-proofs (WG create-only overwrite; plan mode-gate). No secrets on
argv/stdout/logs. The destructive in-place restore is a separate operator-present
STOP-gated drill.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 21:06:31 +02:00
admin 567cf9f401 docs: REPORT + CONTEXT for S4.1 (tier-aware deadline + teardown phantom confirmed)
Live diagnosis on the agent-token path: offsite restore-test pass:true,
verified:boot+running, teardown succeeded (no 403), scratch band clean. The
VM.Allocate follow-up was a PHANTOM — the earlier 403 was the 10m-timeout firing
teardown against a not-yet-pool-associated guest; with the timeout fixed the
guest is in /pool/felhom and teardown is authorized. No ACL/host-install change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 19:54:46 +02:00
admin dc70e15d28 reconcile: tier-aware restore-task deadline (S4.1 unattended offsite restore-test)
A WAN (pbs-tier) restore of a large guest exceeds the restore-task wait's 10m
default → the wait expired mid-restore, teardown fired against a still-restoring
(not-yet-pool-associated) scratch guest → leak + a phantom VM.Allocate 403.

- RestoreTestSpec.RestoreTaskTimeout (0→10m default); the restore WaitTask passes
  it. Local tier unchanged (10m).
- config RestoreTestPBSRestoreTimeoutSeconds + accessor (default 120m).
- main restoreTaskTimeout(cfg,tier): configured PBS timeout only when tier==pbs,
  else 0. Both scheduler + selftest spec builds.
- Tests + WaitOptions red-proof + accessor contract.

The "grant scratch-band VM.Allocate" follow-up is diagnosed not blind-applied:
the scratch is restored INTO /pool/felhom (ACL already grants VM.Allocate), so
the earlier 403 was a consequence of the timeout. No ACL/host-install change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 19:45:26 +02:00
admin fee1fcfccd docs: REPORT + CONTEXT for S4 (PBS over the tunnel, live-validated)
Scenario-D green: real vzdump of 9201 → ciphertext in ns over the tunnel;
ns-scoped verify=ok under the box's own DatastoreBackup token; WARN gone;
restore round-tripped (decrypt with box-born key → boot → teardown).

Findings recorded: retarget field is local_backup_target (reverted to local —
single-target offsite is near-continuous given the ~30-min controller cadence;
needs Tier-1/Tier-2 split); restore-test scheduler needs a WAN deadline +
scratch VM.Allocate; escrow left as an operator-present step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 17:25:49 +02:00
admin 027948bf3f pbs: namespace-aware client for per-customer offsite tenancy (S4)
Phase-1 live probe (felhom-hetzner) proved backup/restore/list/isolation over
the tunnel with a per-customer DatastoreBackup token, but the agent's PBS client
was namespace-unaware: Snapshots hit the datastore root (403 for a scoped token)
and Verify was whole-datastore (needs Datastore.Verify ~ admin). Operator-
approved fix.

- pbs.Config.Namespace + Client.namespace; Snapshots appends ?ns=; Verify sends
  ns= (ns-scoped verify works with DatastoreBackup on the own ns — no admin
  widening, Phase-1 confirmed). Root-ns clients unchanged (whole-datastore).
- proxmox.Storage.Namespace (parsed from /storage `namespace`).
- pbsTargetsFromPVE threads s.Namespace into the client.

Confirmed tenant ACL: DatastoreBackup on /datastore/felhom-offsite/<ns> (NOT
/ns/<ns>) to BOTH felhom@pbs (user) AND felhom@pbs!<ns> (token) — PBS privsep =
intersection; isolation holds (cross-ns 403 proven). TestClient_NamespaceScoping
red-proofed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 16:12:15 +02:00
admin 734f45c422 wgtunnel: v4-pin + re-resolve watchdog; FELHOM_WG Critical flips (S4 agent half)
v4-pin (doc 06 §4.2): renderConf takes a pre-resolved IPv4 literal and writes
Endpoint=<ip>:<port> — never the DNS name, never AAAA. Resolver seam (A records
only, LookupNetIP "ip4"); multiple A → lowest (deterministic fleet-wide);
renderConf stays pure. Resolved IP cached: steady-state Apply = zero DNS + zero
execs. DNS failure keeps the last conf (never a teardown).

Watchdog (loop-only, so Apply's zero-exec steady state is untouched): handshake
age > wg_tunnel.stale_after_seconds (default 180) → re-resolve; IP changed →
re-render + restart (endpoint re-IP recovery); IP same → no churn (throttled
warn). Staleness read reuses wg show latest-handshakes (never dump).

Capability: wg-conf-install/enable/restart/handshake-read flipped Critical=true
(backups ride the tunnel from S4); apt-install + disable stay non-critical.
TestWGCapabilityCriticality pins the set.

Tests + red-proofs a/b/d all fire. No new sudoers grant; no wire/JSON change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 15:47:17 +02:00
admin c618fc69f7 docs(REPORT): publish + Day-0 vouch of 0.65.0 completed (operator-assisted)
Gitea publish sha256 ae282669… round-trip verified; hub manifest now vouches
agent 0.65.0 + golden 0.98.3 (flash + reload-persist verified); /offsite +
Hosts pages eyeballed (0.65.0 ONLINE).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 14:15:10 +02:00
admin ef18454e65 docs: REPORT + CONTEXT for S3.1 MTU 1280 (v0.65.0 live on felhom-pve)
Deploy + wired smoke recorded: agent re-rendered conf on restart, live iface
MTU 1280, PBS page loads (no regression). Red-proof outcome recorded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 14:00:20 +02:00
admin 312fd5ee29 wgtunnel: client MTU 1420 → 1280 (S3.1, resolve §4.3 open decision)
The 2026-07-04 CGNAT smoke test found MTU 1420 silently black-holes bulk TCP
on sub-~1480 paths (mobile ~1400, DS-Lite ~1452): handshake+ping stay healthy,
PBS TLS page (and at S4 the backup itself) drops. Set a fleet-wide, permanent,
family-agnostic client MTU of 1280 (RFC 8200 IPv6-minimum floor; outer 1340 v4
/ 1360 v6 fits every realistic path).

Client-only by construction — interface MTU caps box→PBS, advertised MSS caps
PBS→box; the endpoint's wg0 is untouched (zero live-endpoint risk). New const
clientMTU=1280 as the single home; golden pins exact "MTU = 1280" (red-proofed
against a 1420 flip). Stale report.go comment updated. No wire/JSON change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 13:57:20 +02:00
admin 03edb40fc2 docs: v0.64.0 shipped — CHANGELOG + CONTEXT + REPORT (S3 live-validated on felhom-pve)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 08:57:56 +02:00
admin e2b6c63ea2 wgtunnel: S3 Part 3 — FELHOM_WG sudoers + capabilities + config (DEFAULT OFF) + main wiring + escrow join
Sudoers: fixed-path conf install, enable/restart/disable, latest-handshakes-only
wg read (dump FORBIDDEN — the S1 incident). 6 capability-manifest entries
(Critical=false until S4 makes the tunnel load-bearing). WGTunnelConfig with
enabled=false DEFAULT (the safety gate: a v0.64.0 rollout without explicit
config is a no-op). Daemon wiring mirrors lanresolver + AddConsumer +
SetWireguardReporter; --selftest=wgtunnel single-shot. IdentityBundle
+wg_private_key (omitempty; pre-S3 blobs cannot be retrofitted — documented)
with escrow-create auto-inject (field name only in logs). Red-proof (e) run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 07:14:33 +02:00
admin fb248961c6 wgtunnel: S3 Part 2 — manager state machine + loop + desired raw-consumer seam
Manager: one-shot registration (marker gate; backoff cap 15m), adopt-lost-marker,
re-key-on-mismatch, REVOKED-STAYS-REVOKED teardown (marker kept, zero execs on
later ticks), no-teardown-on-absent-data, hash-gated apply (zero execs steady
state), restart-not-reload on conf change, self-heal enable. Status stanza with
latest-handshakes-ONLY wg read. Collector WireguardReporter seam. desired.Syncer
AddConsumer fan-out with panic containment. Red-proofs a/b/d run + reverted;
no-key-material-in-logs asserted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 07:09:01 +02:00
admin 0daae92c4f wgtunnel: S3 Part 1 — pure-Go keygen + hub wire (WireWireguard, report stanza, RegisterWG)
key.go: create-once 0600/0700, corrupt-refusal (never overwrite — may be escrowed
identity), clamp for CANONICAL STORED form (x/crypto X25519 clamps derivation
internally — discovered during red-proof (c); the stored-clamped test is the
real anchor). Fixed vectors generated with real wg pubkey (provenance in test).
hub: WireDesiredState.Wireguard + WireguardStatus report stanza + RegisterWG
client (typed errors, token-free). S2 golden copied BYTE-IDENTICAL + field-exact
decode test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 07:00:19 +02:00
admin 4ba1b144d6 docs: REPORT/CONTEXT — golden 0.98.3 rebuild validated + vouched (B5+B1 closed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 18:38:20 +02:00
admin ceca355493 configs: build-golden.sh v2.0.0 — mandatory controller tag (B5) + bootstrap .path unit (B1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 18:21:21 +02:00
admin c9f963d9f4 docs: REPORT/CONTEXT — v0.63.0 shipped + live-verified (B3 reload-on-miss, B2 snippets dir)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 17:26:29 +02:00
admin f31a76f788 v0.63.0: B3+B2 fresh-install fixes — TokenStore reload-on-miss + guesthook snippets dir
B3: Lookup re-reads the append-only store once on a miss (cross-process
coherence with the one-shot provisioner; size short-circuit bounds the cost;
behind the TokenAuthority seam). B2: fenced mkdir -p /var/lib/vz/snippets
before the snippet install + the one narrow sudoers grant. Both red-proofed;
drill findings DRILL-day0-cleanroom-2026-07-03 B3/B2.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 15:48:24 +02:00
admin 84f3f7ddb1 docs: REPORT — v0.62.0 A1 fix shipped + live-verified (rescope → deploy → 47/47 caps, pool scan pool=felhom listed=1 scanned=1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 13:36:08 +02:00
admin 3f37c5fc23 v0.62.0 — A1: pool-membership ownership check for the stale-lock reaper
staleLockController.Guests() = ListLXC ∩ GET /pools/felhom members (ownership
PROVEN via the pool registry, never assumed from enumeration scope); pool-read
failure fail-safes the whole recovery through the existing guest-list guard.
New Client.Pool read (needs Pool.Audit — host-install v1.9.0; Pool.Allocate
does NOT satisfy it, spike T2). Composed pve:pool-read capability (non-critical)
+ --selftest pool-read line. Red-proofed negative tests drive the REAL
controller over a broad-token-shaped fake.

Per SPIKE-a1-pool-membership-read-2026-07-03.md; audit A1
(AUDIT-blast-radius-hostroot-localapi-2026-07-02).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 13:25:05 +02:00
admin 3e760a559a docs: CLAUDE.md refresh — version narrative removed, layout completed (17 pkgs + opsign), skill pointers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 11:25:56 +02:00
admin 2a024b6039 docs: REUSE.md introduced — cross-repo reuse map (exec/format-safety guards, durable-id seams, traps; docs-only, no version bump)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 09:39:32 +02:00
admin 0db09bf34f docs: REPORT — v0.61.0 audit-fix batch shipped, live-verified on felhom-pve, Gitea-published
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 07:31:32 +02:00
admin 3f382bf762 v0.61.0: audit fixes B1 (random temp staging) + D1 (mkfs wrapper member/RO re-checks) + D2 (empty-lsblk fail-safe) + D3 (blank-format anti-retarget)
From AUDIT-blast-radius-hostroot-localapi-2026-07-02.md. Each fix ships with a
non-hollow test + a companion red-proof (shown failing on the pre-fix impl).
Sudoers install-source grants became globs — deploy the sudoers drop-in with
the binary. A1 (stale-lock pool-membership) deliberately excluded (spike).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-03 07:25:50 +02:00
admin cc93dae792 docs: REPORT — v0.60.0 proof-of-launch gating shipped, live-proven, published
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 10:28:18 +02:00
admin b9356d60ab v0.60.0: proof-of-launch destroy gating (F1a/b/c) + restore-test band-advance (F2)
Campaign pool-effects F1 (HIGH): the bring-up compensating rollback and the
restore-test teardown destroyed the target vmid even when RestoreLXC failed
synchronously without creating anything — destroying a guest the transaction
never made (only the pool ACL 403 contained it). A RestoreLXC UPID is now the
sole destroy authorization in all three destroy paths (in-process bring-up
defer, in-process restore-test teardown, Recover). F2: the restore-test
advances past an 'already exists' band vmid (invisible squatter) instead of
failing + false-alerting; a fully-occupied band Skips.

Red-proof verified: with the gates reverted, the four new tests fail with the
innocent-guest destroy. go build/vet/test clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 10:18:51 +02:00
admin 55ade9e254 docs: REPORT — add agent v0.59.0 (registry-row device+capacity)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 21:37:21 +02:00
admin 39ec76dc5a v0.59.0: resolve registry-row BackingDevice to the real /dev node
Use ResolveStorageDevice (EvalSymlinks) so a registry drive shows /dev/sdd, not
the /dev/disk/by-uuid/<uuid> symlink path — matching Observe-sourced rows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 21:35:48 +02:00
admin 5ff5f8e0ab v0.59.0: report backing device + capacity for a registry-sourced /disks row
Agent-view showed "—" device + no size for a raw (no-PVE-storage) drive because
the registry union row never set backing_device/total_bytes/used_bytes (Observe
drives get those from pvesm status). Resolve BackingDevice via ByUUIDDevicePath +
read capacity via statfsCapacity (build-tagged syscall.Statfs; no-op off-Linux).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 21:34:11 +02:00
admin b52680ab47 docs: REPORT for raw-drive first-class support (agent v0.56-0.58; live-validated)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 21:27:22 +02:00
admin 3825664aed v0.58.0: report GuestPath/BoundUnderParent for a registry-sourced /disks row
The Impl-2a registry union row omitted GuestPath + BoundUnderParent, so the
controller read a registry-only (raw) drive as "Leválasztva" despite being mounted
+ bound live. Populate them like the Observe path (StablePathForRaw +
boundUnderParent). Last piece of first-class raw-drive support.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 21:21:59 +02:00
admin 6448b80361 v0.57.0 fixup: wire HostReader into the local-API server (was nil in prod)
buildLocalAPIServer never passed Options.HostReader, so s.host was nil — the
v0.56.0 durableIDForMount raw-mount fallback + the role gate's host classification
silently no-op'd. Wire storage.NewProcHostReader(). This makes the raw-drive
intent/guest-bind resolution actually fire live.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 21:18:21 +02:00
admin 67e2144f11 v0.57.0: re-assert a raw drive's guest-bind (ReassertGuestBinds mount-table fallback)
ReassertGuestBinds mapped durable-id->mount from Observe() only, so a raw enrolled
drive was never found and its in-guest bind wasn't re-asserted after reboot/re-mount
(drive showed "Leválasztva"). Augment the map from the mount table (raw /mnt/<name>
-> device fs-UUID via HostReader), skipping the /mnt/felhom-drives bind; Observe
failure no longer aborts. Completes v0.56.0's raw-drive durability.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:37:31 +02:00
admin e593fb277a v0.56.0: resolve a raw enrolled drive's durable-id from the mount table
durableIDForMount was Observe-only, so a raw (non-PVE-storage) drive's enroll
intent + guest-bind went unrecorded ("durable-id unresolved") — it mounted+bound
but wasn't intent-tracked (RegistryKnownTargets skips intent==new). Fall back to
resolving the mount's device fs-UUID (HostReader.Mounts + ResolveUUID) →
uuid:<fs-uuid>, same scheme as Observe. Test + red-proof. Residual:
ReassertGuestBinds still Observe-based (raw guest-bind not re-asserted post-reboot).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 18:29:22 +02:00
admin 4f8d2fcb57 docs: REPORT for agent v0.55.0 (Impl-2a discovery + registry tracking; validated)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 17:38:52 +02:00
admin 91f6a26490 v0.55.0: raw-device discovery + registry-sourced drive tracking (Impl-2a)
GET /disks/candidates enumerates host disks the Impl-1 unclaimed filter proves
free (init/attach split). RegistryKnownTargets sources the watchdog's known-drive
set from the intent registry + Felhom .mount units (not Observe/PVE storages) —
decouples drive health from PVE storage (closes the registry-only false-detach
class); Observe kept for real PVE storages + a deduped /disks union. Idempotent
existing-drive migration at start. Tests + red-proof (Observe misses a
registry-only drive; registry provider tracks it). go build/vet/test clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 17:33:49 +02:00
admin 066e3bf153 docs: REPORT for agent v0.54.0 (Impl-1 format-safety; live-validated)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 17:01:08 +02:00
admin 52098302ab v0.54.0: format-safety foundation — unclaimed-disk guard + guarded-mkfs wrapper
Impl-1. Format now runs a mandatory unclaimed-disk guard (internal/storage/claim.go:
SystemDisks + lsblk member-FSTYPE + foreign-mount + RO + pvs/zpool; fail-safe →
CLAIMED) before any mkfs — refuses the OS disk / LVM PV / ZFS-mdraid member /
foreign-mounted device even when non-data-bearing (guard sits in Format, not the
handler). Below the agent, mkfs goes ONLY through configs/felhom-mkfs-guarded.sh
(sudoers no longer allowlists raw mkfs.*), which re-checks the catastrophic cases
as root. Read-only pvs/zpool added to FELHOM_DISK. Tests + red-proof; capability
manifest updated. go build/vet/test clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 16:52:31 +02:00
admin 05f22a9ab4 docs: REPORT for agent v0.53.0 (pool-scoped restore; supervised swap validated)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 13:01:29 +02:00
admin 7639ab5c4b v0.53.0: restore guests INTO the felhom pool (pool-scoped-ACL enabler)
RestoreLXCOptions.Pool → pct restore --pool (omit-when-empty). New
reconcile.DefaultPool="felhom"; BringUpSpec.Pool threaded to the bring-up
restore; BOTH restore sites pool the guest (provision/DR via spec.Pool set to
DefaultPool by the CLI; restore-test scratch via DefaultPool = SPIKE residual
#2). No agent ACL change (ships in host-install v1.6.0); the pool param is inert
until the token has Pool.Allocate + the pool exists, so publishing is safe ahead
of the coordinated swap. Tests + red-proofs; go build/vet/test clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 12:32:41 +02:00
admin 33e3443545 docs: REPORT for agent v0.52.0 (CPU/RAM cap; T-A scratch proof; manifest follow-up)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 08:19:14 +02:00
admin 0573ec95c1 v0.52.0: operator-opt-in CPU/RAM cap for the provisioned guest (-cores/-memory)
New -cores N / -memory M (MiB) flags for --selftest=bring-up|provision (0 = keep
golden default), flowed through bringUpSizing into reconcile.BringUpSpec so the
existing buildBringUpConfig emits cores/memory into the pre-start config PUT (the
cap lands before first boot). No engine change. New pure-function test
TestBuildBringUpConfig_ResourceCaps asserts the set + omit-when-zero cases
(red-proof run + reverted). go build/vet/test clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 08:03:57 +02:00
admin 1502ca644a docs: REPORT for agent v0.51.0 (vzdump retention, live-validated keep-last=3)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
2026-06-30 20:03:42 +02:00
admin 06e0bc9c25 agent v0.51.0: local vzdump retention default (--prune-backups keep-last=3)
The preventive counterpart to host_disk + storage_fill detectors: the periodic local
whole-guest vzdump now prunes its own old archives (keep-last=3, clamped >=1) so a box
can't refill its own root via its own backups. Local target only — PBS never pruned
(resolved via ListStorage; fail-safe skip on unknown). Seeded in host-install.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
2026-06-30 19:44:07 +02:00
admin 79eb0a8486 docs: REPORT for v0.50.0 (NAS Part A1, sim-validated A-D)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
2026-06-30 11:40:08 +02:00
admin 63aa63d0d6 agent v0.50.0: NAS network storage Part A1 (NFS/SMB automount foundation)
Host-side NFS/SMB automount of a bulk-media NAS share under /mnt/felhom-drives/<name>
(propagates into the guest via the existing shared bind), the +100000 uid recipe,
per-share liveness, and add/list/remove local-API endpoints. A NAS is a distinct class
that bypasses the drive enroll/eject/decommission/SMART/watchdog machinery.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
2026-06-30 11:12:27 +02:00
admin 7aeb7caefe docs: §D supervised reboot — F2-a + F2-b live-validated (both findings CLOSED)
F2-a red-proof PASS: parent came up own peer group (shared:94, not
shared:1), no doubling, guest sees both drives. F2-b PASS: reboot with the
exact snapshot-delete stuck state → agent auto-recovered (unlock → removed
real dangling vzdump snapshot → started CT). Zero spurious pages. Two
live-caught fixes noted (?source=active, AllowWarnings).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
2026-06-30 10:04:58 +02:00
admin d077c8e4dd agent: stale-lock start allows the benign LXC nesting WARNINGS advisory
Live on 9999: the recovery's start succeeded (guest running) but WaitTask
false-failed on "WARNINGS: 1" (the systemd-nesting notice). Mirror the
restore-test's start step (AllowWarnings) so a booted guest isn't logged
as a start error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
2026-06-30 09:07:50 +02:00
admin 1c078b943d agent: stale-lock guard uses tasks?source=active (PVE 9.x rejects ?running=1)
Live test on felhom-pve (PVE 9.2.2) caught GET /nodes/{node}/tasks?running=1
returning HTTP 400 "property not defined in schema". The fail-safe correctly
left the lock, but the BackupRunning guard always errored. source=active is
the supported filter and returns the RUNNING tasks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
2026-06-30 09:06:01 +02:00
admin 6e38e2f921 agent v0.49.0: reboot-during-backup stale-lock recovery (F2-b) + shared-parent script redeploy fix (F2-a)
F2-b: at startup, recover a guest left with a stale vzdump lock by a
reboot-during-backup — pct unlock -> delete dangling vzdump snapshot ->
start iff onboot, guarded by a no-vzdump-running invariant (fail-safe).
New internal/localapi/stalelock.go; proxmox GuestConfig.Lock()/OnBoot(),
ListSnapshots, ListRunningTasks, Snapshot type. New narrow sudoers grant
FELHOM_STALELOCK (pct unlock) + Critical capability stalelock-unlock.

F2-a: EnsureSharedParent only redeployed the boot script when the UNIT
differed, so the v0.36.6 make-private fix never reached hosts whose unit
was current -> /mnt/felhom-drives stayed in root's shared:1 and doubled
every drive bind. New sharedParentInstallStale compares BOTH script and
unit. Boot-time-only; never churns the live mount.

Both root causes confirmed live on felhom-pve before fixing. Green gate
(build/vet/test) all pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
2026-06-30 09:00:32 +02:00
admin 81954c3d2a docs: REPORT for v0.48.0 (agent reports served leaf fp, Part A)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
2026-06-29 23:32:11 +02:00
admin bf8e3be3f4 agent: report served local-API leaf fingerprint (hub re-key detection, Part A) v0.48.0
HostReport.LeafFingerprint rides the served fp (from EnsureLeaf) on every report; empty when local
API disabled. Collector.SetLeafFingerprint threads it like Capabilities. Hub watches it for a re-key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
2026-06-29 23:14:52 +02:00
admin 61f9b4dcc3 docs: REPORT + README for v0.47.0 F1 verify hardening (live rollback re-test PASS)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
2026-06-29 22:50:16 +02:00
admin 3844df7c23 controllerswap: F1 verify hardening — reject crash-looping no-healthcheck image v0.47.0
controllerHealthy reads RestartCount (running&&rc>0 -> not ok) + signals needsDwell for no-healthcheck;
verify requires verifyDwell(=3) consecutive ok polls for a no-healthcheck image (real healthcheck
trusted immediately). Closes the F1 hole (alpine crash-loop passed the point-in-time check). Red-proof
+ dwell + real-image tests. No sudoers/orchestration change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
2026-06-29 22:45:41 +02:00
admin bb548e3c5a docs: REPORT for v0.46.0 leaf-regenerate loud-log (live-validated)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
2026-06-29 22:01:40 +02:00
admin 9b0d6c2c82 agent: EnsureLeaf signals + loud-WARNs a regenerated leaf (prevention B.1) v0.46.0
EnsureLeaf returns generated bool; call-site logs INFO 'leaf LOADED' vs WARN 'leaf REGENERATED —
previously issued bootstrap pins now INVALID'. Catches the 2026-06-28 silent-regen incident class.
Test: first=generated, second=loaded+same fp. No new sudo surface.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
2026-06-29 21:45:25 +02:00
admin de6b72651b docs: REPORT + README for v0.45.0 controller-swap (stdin tee + FELHOM_CONTROLLERSWAP)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
2026-06-29 19:45:17 +02:00
admin 8a4ccab3e6 controllerswap: stdin tee write + narrow FELHOM_CONTROLLERSWAP grants (non-root, v0.45.0)
writeImage drops bash -c/printf for GuestExecStdin(img+\n -> tee /etc/felhom-controller-image);
new Runner.RunStdin/GuestExecStdin route stdin through the fenced sudo -n runner. 5 narrow,
auditable sudoers grants (no general pct exec, no bash -c) + capability manifest entries (Critical)
so the self-probe watches them and the build-test asserts coverage (companion red-proof). No
controller change; swap orchestration/rollback/state unchanged. Spike GO.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
2026-06-29 19:42:30 +02:00
admin 61c89a7efa docs: REPORT + README for v0.44.0 capability self-probe
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
2026-06-29 18:57:22 +02:00
admin 25024d9dda capability: agent privileged-capability self-probe (manifest + build-test + runtime snapshot) v0.44.0
New internal/capability: Manifest of required sudo -n grants + Prober that LISTS each
via 'sudo -n -l' (never executes) + binary-exists check → ok/degraded snapshot on the hub
report. Build-time test asserts manifest⊆sudoers (red-proof: dropping lxc-info FAILs the
gate). Startup logs N/N ok + ERROR per degraded. Serve-degraded; no allowlist change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
2026-06-29 18:43:49 +02:00
admin 19582046ba REPORT: sudoers completeness audit — multi-drive restored; controller-swap surfaced
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
2026-06-29 16:06:40 +02:00
admin 55888e5686 sudoers: close non-root allowlist gaps (lxc-info, make-private, restart dnsmasq, pct reboot/onboot) — restores multi-drive
Root cause of multi-drive flapping: lxc-info absent from the allowlist → empty guest
PID → bound_under_parent=false for every drive → controller gate stops apps. Audit-driven
completeness pass; surfaces GuestExec/pct-exec (controller-swap) for operator decision.
No binary change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
2026-06-29 15:58:41 +02:00
admin c3020ee3a8 REPORT: v0.43.0 BUNDLE slice (canonical unit + Gitea publish + live install)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 09:03:23 +02:00
admin a8d14fc384 sudoers: escape commas in lvs/lsblk -o arg lists (visudo -cf rejected the file)
Bare commas are command separators in sudoers; the lvs/lsblk -o option lists need
escaped commas. The file had never been visudo-validated live (the demo host ran the
agent root+direct). Surfaced by the BUNDLE host-install visudo -cf gate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 08:56:16 +02:00
admin 29aeaa6bb4 v0.43.0: canonical systemd unit + publish agent binary + golden to Gitea (BUNDLE slice)
- configs/felhom-agent.service: canonical non-root unit (User=felhom-agent, sudo model);
  deliberately NO NoNewPrivileges (breaks sudo) and NO mount-namespacing hardening (breaks
  the intermediary-mount drive propagation into guests) — documented inline.
- scripts/publish-agent.sh: build (optional) + PUT binary to Gitea generic + sha256 +
  GET round-trip. Pinned version, idempotent (delete-then-PUT).
- configs/build-golden.sh: after vzdump, compute sha256 + PUT golden.tar.zst to Gitea
  generic (version = baked controller version). Opt-in; local auto-discovery stays fallback.
- cmd/felhom-agent/main.go: version 0.42.0 -> 0.43.0.
- README: process model now canonical (non-root + publish/install).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 08:38:01 +02:00
admin aaa276a7b9 build-golden.sh: default controller image → current (0.85.1); golden rebuilt
The CONTROLLER_IMAGE default (arg 6) was a stale :0.43.0, so an argument-less
golden build baked an ancient controller (fresh Day-0 boxes started at 0.77).
Bumped the default to the current :0.85.1; always pass it explicitly per rebuild.
Golden rebuilt at 0.85.1 on felhom-pve (volid vzdump-lxc-9100-2026_06_27-11_42_51);
baked-image verify confirmed :0.85.1 in the build guest. No agent binary change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
2026-06-27 11:56:40 +02:00
admin 2dcee2d7e8 docs: v0.42.0 controller-swap — README endpoint + REPORT (live-validated)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
2026-06-26 21:43:22 +02:00
admin b17d07bf2d fix: register controller-swap in --selftest allowlist
selftestFlag.Set rejected -selftest=controller-swap (missing case), so the
primitive could not be exercised. Found in live testing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
2026-06-26 21:29:17 +02:00
admin 6f14b66191 v0.42.0: agentic controller update — in-guest image swap + rollback (Phase 1)
New local-API POST /controller/swap (+ GET /controller/swap/status), withGuest-
scoped: the agent records the previous image, confirms the target is present,
rewrites /etc/felhom-controller-image, restarts felhom-controller-bootstrap.service,
verifies the new controller is healthy (docker inspect, <=90s), and ROLLS BACK to
the previous image if not. Single-flight per guest; strict image-ref validation;
crash-safety state file. GuestBinder.GuestExec is the pct-exec seam.
--selftest=controller-swap exercises it directly.

Tests: happy/rollback-on-unhealthy(+red-proof)/image-absent/no-healthcheck/
bad-image-400/single-flight-409.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
2026-06-26 21:26:12 +02:00
admin 4725396c81 REPORT/CHANGELOG: F3 capstone live-validated (re-provision onboot:1 + host-reboot auto-start)
Destroyed + re-provisioned 9201 through the real provision chain with v0.41.0: fresh pct config
showed onboot:1 with no manual set. Host reboot of felhom-pve -> 9201 auto-started (running, no
manual pct start); onboot:0 scratch guests stayed stopped; drives re-bound at stable, sys_drive
separate, controller + infra healthy. Demo controller redeployed to v0.82.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FpBYrZCt9sFDqLgbG5GRGD
2026-06-24 12:48:04 +02:00
admin 3b752b48f2 REPORT: agent v0.41.0 deployed to felhom-pve; 9201 remediated onboot:1
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FpBYrZCt9sFDqLgbG5GRGD
2026-06-24 11:22:43 +02:00
admin 166a1c8bcb v0.41.0: provision back-half sets onboot:1 so customer guests auto-start after host reboot (F3)
The golden bakes --onboot 0 (template safety) and the back-half never overrode it, so every
provisioned customer guest was onboot:0 -> after a host reboot/power-cut the customer's whole
home-server stayed stopped until a manual pct start. Add a fatal 'pct set <vmid> -onboot 1' step
to BackHalf.Provision (right after the config-mount attach), mirroring the existing pct set ops.
No startup/boot-order: the v0.75 mountpoint-gate covers the drive-bind race at boot.

Golden build-golden.sh unchanged (templates must not auto-start). Unit-tested
(TestProvision_SetsOnbootOne + red-proof). RUNBOOK note added.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FpBYrZCt9sFDqLgbG5GRGD
2026-06-24 11:20:45 +02:00
admin db95d5106e REPORT: v0.40.0 live acceptance DONE — golden rebuilt (mp0+mp1 included), 9201 re-provisioned, /mnt/sys_drive separate+warning-cleared+PBS-covered
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
2026-06-23 16:48:39 +02:00
admin 64b7436015 REPORT: v0.40.0 third CT volume — code shipped + agent v0.40.0 deployed live; golden/re-provision pending operator secrets+CHECKPOINT
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
2026-06-23 16:30:11 +02:00
admin 459dad954b v0.40.0: third CT volume — SSD user-data (/mnt/sys_drive, mp1) baked + -sysdata-grow
Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).

- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
  findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
  new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).

Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
2026-06-23 16:25:14 +02:00
admin b908b9a8e5 REPORT: v0.39.0 DR-recipe completion (live PBS coord + v1 drive-shape) — live-validated
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 20:34:31 +02:00
admin fbe113011d v0.39.0 — DR-recipe completion: live PBS coord + drop role/restic_repo_coord from v1 drive shape
Live PBS coord: new internal/pbs/live_reporter.go (LiveSnapshotReporter implements
hub.PBSReporter via the cheap Client.Snapshots() list with last-known-good fallback,
bounded by an 8s timeout, list-only — never triggers a verify). Closes the gap where
the recipe's pbs block was omitted whenever the verify-loop SnapshotStore was empty
(one-shot collect + the first ~6h after a daemon restart). SnapshotStore.Get added
(per-datastore LKG). Wired into the collector in both runDaemon and runSelftestHub;
the verify loop keeps Recording into the SAME shared store via one hoisted pbsTargets.

v1 host-half drive shape: dropped drives[].role (hub/operator-owned manifest concept,
not host-derivable) and drives[].restic_repo_coord (named a backup tier that doesn't
exist). Drive shape is now {durable_id, mount_path, intent, fs_type?, total_bytes}.
Hub reads drives as json.RawMessage → no hub struct change; goldens re-pinned
byte-identical (agent + hub copies).

Tests: live_reporter_test.go (T1 load-bearing coord-without-verify + T2..T6),
TestDRRecipeHostHalf_V1DriveShape; each companion demonstrated to fail pre-fix then
reverted. go build/vet/test green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 20:25:09 +02:00
admin 8abc1b8852 agent v0.38.0: DR recipe — emit secret-free storage/guest/PBS half in host-report
DR recipe slice (agent half), grounded in SPIKE-dr-recipe-2026-06-16. Additive
`dr_recipe` host-report section = the non-secret reconstruction scaffolding the
operator must rebuild before PBS bytes can land. Built by pure
BuildDRRecipeHostHalf from facts the report already collects (no new reads):
guests[] sizing, drives[] (user-data by durable_id/role/mount/intent),
pve_storage[] (storage.cfg), pbs coordinates.

BOUNDARY (Phase-1 lesson): every field is an identifier/intent/size/coordinate —
never a key/password/token/hash/ENC:. PBS key stays in escrow; restic password
stays in escrow; the recipe names only the coordinates the restore targets.

Tests: BuildDRRecipeHostHalf selection, NoPBS, NoSecrets (boundary mirror),
dr_recipe key-set in the cross-repo golden contract test. recipe_version=1,
ignore-unknown on read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 18:32:15 +02:00
admin c8d7ec6bb5 docs(REPORT): live host-reboot x2 acceptance — sdb<->sdc swap tolerated by UUID
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 18:08:29 +02:00
admin 3e39dbb4f8 agent v0.37.0: re-assert mounted-but-DISABLED units (live felhom-usb fix)
The skip-if-mounted optimization defeated the actual root cause: felhom-usb is
mounted now but its unit is `disabled`, so a host reboot would not auto-mount
it. ReassertEnrolledMounts now skips ONLY the durable steady state (mounted AND
enabled) via the pure shouldReassertMount; a mounted-but-disabled unit is
re-asserted so enable --now re-creates the wants-symlink. Enabled-state read by
privilege-free Lstat of the multi-user.target.wants symlink (unitEnabled) — no
systemctl is-enabled subprocess, no new sudoers entry.

Tests: TestShouldReassertMount (4 combos), TestUnitEnabled (wants-symlink).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 17:55:48 +02:00
admin a621f4c5a0 agent v0.37.0: host-reboot remount re-resolves enrolled drives by fs-UUID
TASK A — close out the reboot story (agent half). Root cause (pinned live on
felhom-pve): an enrolled .mount unit left `disabled` by a prior detach never
auto-mounts at boot, and kernel re-enumeration can move a drive's node
(/dev/sdb->sdc). Fix re-asserts every enrolled mount by filesystem UUID at
startup + on the periodic tick.

- ResolveStorageDevice: resolve uuid:<fs-uuid> -> current /dev node via
  /dev/disk/by-uuid (never a cached node); errors if absent.
- parseFelhomMountUnit: pure inverse of renderMountUnit (marker-gated).
- (*SudoHostOps).ReassertEnrolledMounts: re-run EnsureMount (enable --now) for
  any enrolled unit not in /proc/mounts; idempotent, skips mounted/absent.
- main.go: runs before ReassertGuestBinds at startup + on the 20s tick.
- tests (Linux, seam=device resolution): letter-move tolerated (sdb->sdc) +
  red-proof companion, absent/scheme rejection, render->parse round-trip.

TASK A2 verdict: enrolling a NEW drive does NOT need an LXC restart — the path
lands on the live AttachDrive (shared parent, named live slots), not RebootGuest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 17:49:00 +02:00
admin 437f096d9d docs: REPORT — storage lifecycle + acceptance sweep (agent half)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:42:04 +02:00
admin 1e205840dc agent v0.36.7: isolate shared parent only on create (no peer-group churn)
make-private+make-shared only when first creating the self-bind; re-running it
churns the peer group and orphans the guest's slave (propagation dies).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:31:38 +02:00
admin 2b17419e60 agent v0.36.6: shared parent make-private before make-shared (ROOT CAUSE of double-bind)
The self-bind inherited /'s shared peer group, so binds under /mnt/felhom-drives
propagated back and doubled. make-private (own group) before make-shared fixes it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:28:57 +02:00
admin 2da4c38773 agent v0.36.5: AttachDrive normalizes to exactly one bind (converges stacked binds)
countHostMounts + normalize: no-op only when exactly one bind is guest-visible;
else strip all binds and lay one fresh. Converges a stacked double-bind to one
(the old umount-one+mount-one never did).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:24:42 +02:00
admin a356d6def4 agent v0.36.4: serialize AttachDrive/DetachDrive (no double-bind TOCTOU race)
A GuestBinder mutex prevents a concurrent reconnect + periodic reconcile from
both passing isHostMountpoint and double-binding a stable path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:19:07 +02:00
admin cff9284453 agent v0.36.3: DetachDrive loop-umounts stacked binds (full detach)
Detach now removes ALL stacked binds at a stable path, not just one layer, so an
eject fully detaches even with >1 bind (keeping fail-close intact).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:16:47 +02:00
admin 038f0ecd5d agent v0.36.2: eject keeps raw mounted (reconnectable, like decommission)
Eject now DetachDrive's the bind under the parent but leaves the raw /mnt/<name>
mounted, so disconnect->reconnect re-binds on a non-removable drive. Tests updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:12:44 +02:00
admin 281c7b7e3f agent v0.36.1: decommission keeps raw mounted (re-enrollable)
The decommission unmounted the raw /mnt/<name>, orphaning a non-removable drive
so re-enroll bound an empty dir. Now DetachDrive only (bind under parent); raw
stays mounted so re-enroll re-binds. Test: raw NOT unmounted + DetachDrive called.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:06:23 +02:00
admin 05be509e6e agent v0.36.0: guest boot-id on /disks (deterministic guest-reboot recreate)
GET /disks emits guest_boot_id = <host-btime>-<guest-init-starttime>: changes on
every guest/host boot, stable across controller-only restarts. The controller
persists it + deterministically recreates drive-backed apps on change (replaces
the timed state-sample). Non-hollow parser test + companion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 19:13:15 +02:00
admin 9b4ae3df28 docs: REPORT — intermediary-mount re-architecture (agent half)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 18:01:49 +02:00
admin a2374ca679 agent v0.35.1: shared-parent unit runs before pve-guests on host boot
The shared-parent oneshot never ran before pve-guests (WantedBy=multi-user
wasn't pulled into the boot transaction), so on a host reboot the guest bound a
not-yet-shared parent -> private -> propagation broken. Now WantedBy=pve-guests
(pve-guests pulls it in + Before= orders it first); EnsureSharedParent reinstalls
the unit when content differs so the fix deploys.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 17:58:52 +02:00
admin 26c6d1e4d1 agent v0.35.0: intermediary mount — guest-reboot re-propagation (load-bearing)
A guest's parent bind is non-recursive, so a guest reboot leaves enrolled drives
bound on the HOST but invisible in the fresh guest ns (propagation only delivers
new events). AttachDrive(vmid) now checks GuestSeesMount (/proc/<pid>/mountinfo)
and force re-binds (umount+mount) to re-propagate; a 20s periodic reconcile
self-heals guest reboots without an agent restart; BoundUnderParent reflects guest
visibility (the controller gate's signal). Caught + fixed in the live migration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 17:26:32 +02:00
admin 3a9be73875 agent v0.34.0: intermediary mount model — shared parent + host-side attach/detach + reconcile
Replaces the per-drive 'pct set -mpN' bind with ONE permanent parent bind
/mnt/felhom-drives plus host-side felhom-data swaps underneath it (propagates
into the running guest live, no pct, no reboot; C1-immune; confined; fail-closed
when absent). EnsureSharedParent installs a boot unit ordered Before=pve-guests.
ReassertGuestBinds is now a pure host-side reconcile. /disks reports GuestPath +
BoundUnderParent for the controller repoint+gate. Non-hollow tests + companions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 16:29:32 +02:00
admin 44cdf82631 agent v0.33.0: C1 net — pre-start self-heal hook + decommission mp-delete
Pre-start PVE hookscript (internal/guesthook) creates host-root placeholders for
absent bind-mount sources so the guest always boots (fail-closed); decommission
now pct set --delete's the dead mp (GuestBinder.DetachBind) so a missing source
can't brick the next reboot (B3 C1 bug). Non-hollow tests + companions. Installed
+ registered per-guest by the provision back-half. Transitional ahead of the
intermediary-mount re-architecture which makes C1 structural.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 16:11:10 +02:00
admin 2a4affc3a8 docs(claude): refine A4 live-test rule (pipeline-bypass is the line, not API-vs-UI)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 10:07:55 +02:00
admin 76f9cb5977 docs: REPORT for v0.32.0 self-serve decommission + intent-aware re-assert
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 20:18:01 +02:00
admin f43697c881 v0.32.0: self-serve decommission endpoint + intent-aware re-assert (B2a)
POST /disks/decommission mirrors eject (withGuest, user-data role gate) — no
operator signature, non-destructive (never formats): sets IntentDecommissioned,
prunes the GuestBindStore entry, unmounts. ReassertGuestBinds is now intent-aware
(skip non-enrolled) so a decommissioned-but-present drive never auto-rebinds on
agent restart — the load-bearing F9-reconnect fix. GuestBindStore.Remove added.
Operator-signed DecommissionExecutor + classify untouched. Non-hollow tests incl.
the intent-aware reassert companion (mutation-proven to fail on intent-blind code).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 19:55:17 +02:00
admin 8e6d00a57f docs(claude): require real UI/user-flow live validation (A4)
Per the F9 storage episode — live validation of a user-facing feature
must exercise the real flow end-to-end, not shortcut via direct
API/agent/CLI calls. Low-level mechanism tests are exempt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 17:33:33 +02:00
admin 20e03902c8 docs: v0.31.0 CHANGELOG + REPORT + CLAUDE banner (F9/F20-BUG2/BUG3 live-validated) 2026-06-14 15:41:59 +02:00
admin 4777f8a221 F20-BUG3: run mkfs detached (survives request deadline + agent restart); v0.31.0
The format ran mkfs under the HTTP request context, so the controller's 15s client
timeout cancelled it → SIGKILL mid-write → corrupt disk. Now mkfs runs DETACHED off
s.baseCtx (a dropped request can't kill it) via a persisted formatJob record; the handler
still waits to return the synchronous result (backward-compatible with the v0.62.0
controller) but abandoning the wait on client-disconnect leaves the mkfs running to
completion. New GET /disks/format/status surfaces the job (additive). RecoverFormatJob
runs on agent startup: a record left 'running' (agent died mid-format) is re-resolved by
durable-id (anti-retarget — absent/swapped disk NOT re-formatted) and the mkfs re-run; a
blank/path-bound interrupted format is marked failed (retry), never auto-re-run.

Tests: detached run persists running→done + binds durable-id; status endpoint; recovery
re-runs an interrupted durable-id-bound format; skips blank; skips unresolvable durable-id.
Version 0.30.0 → 0.31.0.
2026-06-14 15:16:01 +02:00
admin 4cd1d024e9 F9: auto-re-assert enrolled guest data-drive binds on agent startup
The in-guest bind (pct set -mpN) is config state that a destroy+re-provision drops, and
nothing restored it — so a re-provisioned guest came up with its enrolled HDD unattached
(the live-drive F9 finding). New GuestBindStore persists, per guest, the durable-ids of
enrolled drives (recorded at guest-attach); ReassertGuestBinds runs on agent startup (the
host's bring-up/reconcile trigger) and re-adds any bind a guest is MISSING — but ONLY when
the durable-id still resolves to a present, mounted drive (a swapped/absent drive is never
auto-bound) and the guest lacks it (idempotent). The re-added bind activates on the guest's
next reboot, like the enroll flow. Wired in main.go (store opened beside drive-intents.json;
ReassertGuestBinds called before the local API serves).

Tests: restores a missing bind with no manual call (the operator's real-trigger proof);
skips absent/swapped durable-id; no-op when already bound; store survives reopen (restart).
2026-06-14 15:07:37 +02:00
admin a2a76e7624 F20-BUG2 + F9-reporting: /disks surfaces wipe_durable_id (gate scheme) + guest_attached
F20-BUG2: the /disks list only carried DurableID in the uuid: scheme (for /disks/assign),
but the wipe gate resolves devices in the byid:/byuuid: scheme — so a customer confirming a
wipe with the advertised id was refused (binding_mismatch). Added a shared s.deviceDurableID
seam used by BOTH handleDisks (new DiskInfo.WipeDurableID) and the format gate, so the id the
customer copies from the list is exactly the id the gate accepts. DurableID (uuid:) is unchanged
(still feeds assign).

F9 (reporting half): added DiskInfo.GuestAttached — whether the drive's namespace is actually
bound into THIS guest's config (guestBoundPaths), distinct from mere host presence (State). This
is the signal whose absence made the HDD look available when it wasn't attached, and resolves the
F2 hdd_configured-vs-/disks disagreement.

Tests: wipe_durable_id is the gate scheme + distinct from uuid:; the list's wipe id matches the
gate's device-id binding (no mismatch); guest_attached true iff bound into the guest.
2026-06-14 15:00:56 +02:00
261 changed files with 43124 additions and 913 deletions
+116
View File
@@ -0,0 +1,116 @@
# gates — re-run this repo's gate entry point on every push, on a machine that does not care who
# pushed or what they typed.
#
# *** THIS REPORTS. IT CANNOT REFUSE. ***
#
# felhom repos push straight to `main` with no pull request, so there is no merge for a status
# check to stand at. The refusing half is `.githooks/pre-push`, which is local to a clone and which
# `git push --no-verify` skips; this half is what notices when that happened. Neither half is the
# whole thing, and both are named in felhom.eu documentation/backlog/OPEN-ITEMS.md R-168.
#
# NO `uses:` STEP ANYWHERE, deliberately: JavaScript actions need a node runtime in the runner, and
# the runner is a host-mode container with python3 and git and nothing else (see
# homelab-manifests/gitea-system/act-runner.yaml for why it is not privileged). Probe P3 measured
# that a plain `git fetch` of the pushed SHA from the in-cluster Gitea service is enough.
#
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
# layer up. That is the last step, and it runs ONLY on failure.
name: gates
on: [push]
jobs:
gates:
runs-on: felhom-gates
steps:
- name: Fetch the pushed commit and the sibling clone it needs
# This repo's entry point invokes a SHARED checker that lives in the felhom.eu clone next
# door and is deliberately never copied here — so CI has to reproduce the workspace's
# sibling layout or the gate fails closed with "gate is MISSING". The sibling is also
# needed for CONTENT: this repo's REUSE.md cites a path that lives in the hub.
run: |
# Shallow, and pinned to the exact SHA that was pushed — not to the branch tip,
# which can move under us if two pushes race.
mkdir -p ws/felhom-agent
cd ws/felhom-agent
git init -q .
git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom-agent.git
git fetch -q --depth 1 origin "$GITHUB_SHA"
git checkout -q FETCH_HEAD
echo "checked out $(git rev-parse HEAD)"
cd .. && git clone -q --depth 1 http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git felhom.eu
echo "sibling felhom.eu present at $(cd felhom.eu && git rev-parse --short HEAD)"
- name: Run the gate entry point
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy. The
# exit code IS the result: no `|| true`, no pipe that could swallow it.
#
# THE FULL SET, NOT `--fast` (R-115, 2026-08-03). `--fast` means "no network and no
# container runtime" and exists for `.githooks/pre-push`, where a push must not fail
# because Gitea blinked or because someone is on a train. CI is the opposite machine: it
# has the network, it is not in anyone's way, and it is the half that emails. The
# published-versions gate — the R-115 mechanism, which asks Gitea whether a released
# version can actually be downloaded — is network-bound and therefore runs ONLY here.
# Leaving `--fast` in place would have registered that gate and never run it, which is the
# built-but-never-wired failure this project has shipped four times.
env:
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
- name: Alarm on failure
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
# is exactly the shape R-29 filed against. So the run sends its own alarm, on the project's
# existing transactional path (Resend, the same one the hub uses), and prints the provider's
# accepted id so "a message left the machine" is an observable, not an assumption.
#
# Pure python3 and urllib, NOT curl: the runner image carries python3 and git and nothing
# else on purpose, and the first version of this step died on `curl: command not found`.
# Reaching for a bigger image to send one HTTP request would have been the wrong trade.
if: failure()
env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
run: |
python3 - <<'PY'
import json, os, sys, urllib.request, urllib.error
key = os.environ.get("RESEND_API_KEY", "")
if not key:
sys.exit("ALARM FAILED: RESEND_API_KEY is empty — the alarm cannot be sent, and a "
"silent alarm is worse than none. Set the user-level Actions secret.")
repo = os.environ.get("GITHUB_REPOSITORY", "?")
sha = os.environ.get("GITHUB_SHA", "?")
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
body = json.dumps({
"from": "Felhom CI <monitoring@felhom.eu>",
"to": ["admin@felhom.eu"],
"subject": "[felhom CI] gates FAILED in %s" % repo,
"text": (
"The gate entry point exited non-zero.\n\n"
"Repository : %s\n"
"Commit : %s\n"
"Run : %s/%s/actions/runs/%s\n\n"
"The failing gate names itself in the run log.\n\n"
"If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
"disagree - that is a finding about the gates themselves, not about CI, and it\n"
"outranks whatever the push was for.\n"
) % (repo, sha, srv, repo, run),
}).encode()
req = urllib.request.Request(
"https://api.resend.com/emails", data=body, method="POST",
headers={"Authorization": "Bearer %s" % key,
"Content-Type": "application/json",
# Cloudflare fronts api.resend.com and BLOCKS the default
# "Python-urllib/3.x" agent with its own 403 (error 1010) — which looks
# exactly like an auth failure and is not one. Measured 2026-08-02.
"User-Agent": "felhom-ci/1.0"})
try:
with urllib.request.urlopen(req, timeout=30) as r:
print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
except urllib.error.HTTPError as e:
sys.exit("ALARM FAILED: Resend returned HTTP %s: %s" % (e.code, e.read().decode()[:300]))
PY
+47
View File
@@ -0,0 +1,47 @@
#!/bin/sh
# pre-push — refuse a push that carries a broken gate. (2026-08-02, R-29 leg (b) first half.)
#
# Runs this repo's ONE gate entry point in --fast mode: only checks that touch no network and no
# container runtime, so a push stays a push and never pulls images or starts containers. The slow
# gates stay deliberate periodic runs; a hook that takes minutes gets bypassed within a week and
# the bypass becomes the habit.
#
# BOTH LINES BELOW ARE DELIBERATE. An absent log line is not evidence a hook ran — a silent pass is
# equally consistent with "gates green" and "hook never fired", so a passing push says so out loud.
#
# HONEST LIMITS, stated so this is not mistaken for enforcement it cannot provide:
# * per-clone — core.hooksPath is local config and a clone does not carry it. Arm a clone once:
# git config core.hooksPath .githooks
# Any manual entry-point run WARNS when the clone is unarmed.
# * skippable — `git push --no-verify` bypasses this entirely. That is on purpose: an escape
# hatch that cannot be reached is one that gets removed the first time it is
# inconvenient. USING IT MUST BE STATED IN THE SESSION REPORT.
# The half that is neither per-clone nor skippable is CI — felhom.eu OPEN-ITEMS.md R-168.
#
# Measured 2026-08-02 (git 2.47.3): a relative core.hooksPath resolves correctly and the hook's cwd
# is the repo root whether `git push` is issued from the root or from any subdirectory. The
# explicit rev-parse below does not depend on that.
set -u
root=$(git rev-parse --show-toplevel 2>/dev/null) || {
echo "pre-push: FAIL - cannot resolve the repo root (git rev-parse --show-toplevel)." >&2
exit 1
}
cd "$root" || exit 1
if ! command -v python3 >/dev/null 2>&1; then
echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
echo " pass by default. Install python3, or push with --no-verify and say so." >&2
exit 1
fi
echo "pre-push [felhom-agent]: running scripts/agent_gates.py --fast ..."
python3 "scripts/agent_gates.py" --fast
rc=$?
if [ "$rc" -ne 0 ]; then
echo "pre-push [felhom-agent]: PUSH REFUSED - gates exited $rc. Fix the finding above, or bypass with" >&2
echo " 'git push --no-verify' and state that you did in the session report." >&2
else
echo "pre-push [felhom-agent]: gates OK - push proceeding."
fi
exit $rc
+3558 -1
View File
File diff suppressed because it is too large Load Diff
+157 -69
View File
@@ -1,101 +1,189 @@
# CLAUDE.md — `felhom-agent`
> Place at the repo root (`felhom-agent/CLAUDE.md`). Loads when Claude Code touches this repo. Keep under ~200 lines. The cross-repo orientation lives in the workspace-root `e:\git\CLAUDE.md`; this file is `felhom-agent`-specific.
> Loads when Claude Code touches this repo. Stable orientation only — **current state lives in
> `CONTEXT.md` and the top of `CHANGELOG.md`**, never here. Cross-repo orientation: workspace-root
> `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`.
## What this repo is
`felhom-agent` is the operator-tier **host agent** that runs on each Proxmox host and owns **all** Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub control loop, and a narrow per-guest local API. It is the **most privilege-sensitive** component.
`felhom-agent` is the operator-tier **host agent** that runs on each Proxmox host and owns **all**
Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub
control loop, and a narrow per-guest local API. It is the **most privilege-sensitive** component.
- It is the renamed former `proxmox-controller` repo.
- **Distinct from `felhom-controller`** — that is the *in-guest* controller (Docker-only, no Proxmox creds). Do not confuse them.
- Renamed former `proxmox-controller` repo.
- **Distinct from `felhom-controller`** — that is the *in-guest* controller (Docker-only, no Proxmox
creds). Do not confuse them.
- Control plane, not data plane: if the agent dies, apps keep serving; only management degrades.
## Read before writing code
- **`REUSE.md`** — canonical helpers, format-safety guards, traps, seams. Check it first; update it
in the same commit that changes a shared helper or pattern.
- `CONTEXT.md` (current state + open threads) and the top `CHANGELOG.md` entry (authoritative history).
- Design doc: `felhom.eu/documentation/architecture/03-host-agent.md` (locked). Platform facts:
`felhom.eu/documentation/proxmox-platform.md` + `tests/phase{0,1-2,3,4}-findings.md`.
## Layout (verified against the tree)
```
cmd/felhom-agent/ main + flags + --selftest modes + the daemon entry
cmd/felhom-opsign/ offline operator signing CLI (SSHSIG)
internal/authz/ operator signed-op verifier (SSHSIG) + durable FileNonceStore
internal/backup/ vzdump backup runner + restore-test scheduler + report store
internal/capability/ live sudo-policy capability probe (degradation visibility)
internal/config/ JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
internal/desired/ hub desired-state syncer (envelope observer)
internal/escrow/ PBS-key escrow (zero-knowledge recovery code)
internal/guesthook/ pre-start self-heal hookscript install
internal/hub/ daemon: HostReport collector + Bearer client + resilient Loop
internal/lanresolver/ split-horizon DNS on guest IP change (dnsmasq RESTART, not reload)
internal/localapi/ per-guest local API: token store, disks/format, guest binds, controller swap,
stale-lock recovery, pinned self-signed leaf
internal/log/ slog setup
internal/pbs/ PBS-API client (fingerprint-pinned) + verify maintenance loop
internal/provision/ guest bootstrap back-half (token mint → bootstrap.json → pct bind)
internal/proxmox/ API-first Client + fenced root-CLI Privileged + UPID WaitTask
internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery
internal/signedjobs/ operator-signed destructive executors (wipe, decommission)
internal/storage/ storage observer + durable ids + role/claim classifiers + SudoHostOps + watchdog
```
## Build / run
- Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`).
- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks.
- `go.mod` directive **go 1.25.0**; dep `golang.org/x/crypto v0.52.0` (declares go 1.25, will NOT build on Go 1.24). The **build server (192.168.0.180) runs go1.26.0** (upstream Go on PATH, backward-compatible). Build/run the agent there for live tests (same LAN as the demo host).
- Version: `version` var in `cmd/felhom-agent/main.go`, overridable via `-ldflags "-X main.version=<v>"`; `--version` flag. Bump on meaningful changes + add a CHANGELOG entry. **See `CHANGELOG.md` (top only) for the authoritative current state.**
- **Current: v0.30.0** (2026-06-13; deployed on demo host `felhom-pve`). Recent state (summarized from the CHANGELOG top — verify there, not here):
- **v0.30.0** — **AGENT-001** security fix: the inline customer-confirmed wipe (`localapi handleDiskFormat`) re-resolves the confirmed durable id → current device, re-derives+matches, re-inspects, and formats the **re-resolved** device (never the mutable `req.Device`), closing a classify→mkfs TOCTOU. At parity with `signedjobs.WipeExecutor` (`internal/localapi/wipe_reresolve.go`).
- **v0.29.x** — OS / Docker-data storage **split** (golden bakes split rootfs + Docker-data volume; provision) + `lanresolver` split-horizon DNS fix (RESTART, not reload, dnsmasq on a guest IP change).
- **v0.28.0** — backup re-target to offsite **`felhom-pbs`** (DR) + operator-signed **decommission**.
- **v0.230.27** — device-**ROLE** classification + tiered storage-wipe gate (system/backup operator-only, user-data customer-confirmable); eject role-gate; user-data drive **enroll/bind** into the guest (slice 10 P2/P3); self-heal watchdog + 4-state intent model.
- **Foundations (slice 810):** per-guest **local-API server** (`internal/localapi`, self-scoped endpoints, hashed per-guest token store, pinned self-signed leaf); the `/disks` data-bearing classifier + signed-job destructive gate (`POST /disks/format` inspects the device itself; data-bearing → gate → `pending_signature` refused, caller's claim ignored; blank → benign `mkfs`); the provisioning back-half (`internal/provision`: mint token → `bootstrap.json` 0600 → `chown 100000:100000``pct set` bind, no registry cred in guest); host metrics; hub desired-state; operator-signed completion + offline signing CLI; PBS escrow + identity-restore (slice 10D). Runtime dep: `proxmox-backup-client`.
- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks. `go.mod` directive go 1.25.0;
DooPlex (192.168.0.180, where CC runs) has the Go toolchain and is on the same LAN as the demo
host — build and run live tests locally.
- Version via `-ldflags "-X main.version=<v>"`; `--version` flag. Bump on meaningful changes + CHANGELOG entry.
- **Full build/deploy/publish runbook: use the `felhom-build-deploy` skill.** Summary:
## Layout
> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).
```
cmd/felhom-agent/ main + flag handling + --selftest modes + the daemon entry
internal/config/ JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
internal/log/ slog setup
internal/proxmox/ API-first Client + fenced root-CLI Privileged + UPID WaitTask
internal/authz/ operator signed-op verifier (SSHSIG); durable FileNonceStore
internal/hub/ daemon: HostReport collector + Bearer client + resilient Loop
internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery + restore-test
internal/storage/ storage-target observer + durable_id + fast-poll watchdog (slice 5)
internal/backup/ vzdump backup runner + restore-test scheduler + report store (slice 6)
internal/pbs/ PBS-API client (fingerprint-pinned) + verify maintenance loop (slice 6 Phase B)
```
> **RELEASING IS ONE COMMAND, AND IT PUBLISHES (R-115).** There used to be a raw `go build` line
> here and a *separate* "Publish" row, so publishing was a step someone had to remember — and it was
> **forgotten three times in five days**, the last leaving agent v0.120.0 deployed on both demo hosts
> and undownloadable, where a documented-path reinstall would have silently downgraded them while
> reporting success. Do not hand-roll the build: the script also creates the `v<version>` git TAG
> that `felhom-host-install.sh` fetches this version's sixteen config files from (R-183), and it
> verifies by an **independent download** rather than trusting the publish step's own output.
> `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather
> than reimplementing it.
| Step | Where | One-liner |
|---|---|---|
| **Release** (build + tag + publish + verify) | DooPlex (local) | `GITEA_USER=admin GITEA_TOKEN=<tok> scripts/release-agent.sh <ver>` — refuses a dirty/unpushed tree and refuses to re-release an existing version |
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
| Deploy | felhom-pve | backup `.bak-<old>``install -m0755``systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
## Proxmox model (the load-bearing rules)
- **API-first** via a scoped `FelhomAgent` token (16 privileges). Raw root-CLI is **fenced to exactly 3 exceptions**: keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors. `Client` never shells out; `Privileged` never makes HTTP calls (asserted by tests). Keep that fence.
- **Every mutating op is async** → returns a UPID → `WaitTask` asserts `exitstatus == "OK"`. A 200 on the POST is **not** success; authorization can fail at task execution, not the POST.
- **TLS:** SHA-256 leaf-cert pinning (the host serves a self-signed cert). No insecure default.
- **Privsep token gotcha:** a `--privsep 1` token's rights = intersection of the backing user's perms AND the token's ACLs — so the role must be granted on **both** user and token, or every call 403s. (Token provisioning is out-of-band / human-run; the agent only consumes the token.)
## Design + platform facts (read before designing)
- Design doc: `felhom.eu/documentation/architecture/03-host-agent.md` (locked).
- Platform facts: `felhom.eu/documentation/proxmox-platform.md` + `tests/phase{0,1-2,3,4}-findings.md`.
## Current state
Built in slices, all on `main`:
- **v0.1.0** slice 1 — scaffold + `internal/proxmox` + `internal/config`/`log` + `--selftest`.
- **v0.2.0** slice 2 — `internal/authz` signed-op verifier.
- **v0.3.0** slice 3 — `internal/hub`: the first **daemon loop** (no-`--selftest` mode) posting a read-only `HostReport` to the hub (= the heartbeat). Report's storage/backup/restore/pbs/audit fields are **defined-but-empty** (slices 5/6); the envelope's desired-state/signed-ops fields are **parsed-but-ignored** (slice 4).
- **v0.3.1** — slice-3 validation follow-ups.
- **v0.3.2** — slice-4 pre-check: reversible `SetConfig` step added to `--selftest=task`; passed live on guest 9999. Findings: LXC `description` write is **synchronous** (empty UPID — dual-mode modeling confirmed); PVE appends a trailing `\n` to `description` on read (reconcile must normalize). First live `VM.Config.*` exercise.
- **v0.4.0-rc1** — slice-4 **Phase A** (structural): `internal/reconcile` — engine, per-guest serializer (§10), desired-state model + `DesiredProvider` seam, normalization layer (`NormDescription` promoted out of main.go), plan/diff engine (benign Start/Stop/SetConfig set), durable op journal + idempotency store. Wired into `runDaemon` sharing the queue. Runs **live but unfed** (EmptyProvider → zero mutations until slice 10).
- **v0.4.0** — slice-4 **Phase B** (security core): the benign/destructive **classifier** (provenance + data-bearing, not by verb; scratch/same-txn provenance is agent-internal, never hub-sourced), the **reversibility gate** (destructive → `pending_signature` unless a verified, role-scoped, action-bound operator signature), the **signed-op consuming layer** over `internal/authz` (role-scoping per doc 04 §4, op-to-action binding, idempotency-by-nonce, audit), and the **crash-recovery consumer** (`Recover` over `InFlight()`, resume-or-rollback). The gate fronts the queue's executor (every mutation passes it). **Inert this slice** — no destructive deltas served until slice 10; the destructive path is classified, gated, and adversarially tested but not wired to live execution. `authz` surface untouched.
- **v0.5.0-rc1** — slice-5 **Phase A** (read-only, live): `internal/storage` — the `StorageTarget` wire contract (filled the slice-3 stub), `durable_id` derivation per type, the `Observer`, and the **storage watchdog** (third daemon goroutine; fast-poll → debounced out-of-band report on a known target's attach/disconnect). Hub ingest accepts/persists `storage_targets`; cross-repo golden byte-identical.
- **v0.5.0** — slice-5 **Phase B** (the host-root surface): the `HostOps` seam + `SudoHostOps` (systemd `.mount` units by fs-UUID, detach, SMART, lvs) behind a **strict argument validator** (the adversarial matrix is the headline security test — hostile UUID/path/device refused with zero exec); SMART (SATA+NVMe) + thin-pool metadata enrichment; the watchdog's benign **re-mount response** (off the poll path); the **disk-grow executor** (`pct resize`, grow-only, benign) and **destructive storage ops** through the slice-4 gate (target-scoped; built + tested, inert live); `--selftest=storage [-watch]`; `configs/felhom-agent.sudoers`.
- **v0.6.0-rc1** — slice-6 **Phase A** (backup + self-restore-test, local target): proxmox `DestroyLXC`/`Vzdump`-notes/`LatestBackupVolID`; `Engine.RunRestoreTest` (journaled scratch lifecycle: restore-to-new → net link-down → boot → verify running → defer teardown, all benign); `Recover` extended to reap a leaked scratch guest (Scratch journal flag, special-cased before the UPID path); `internal/backup` (runner + bulk-gap + cadence scheduler + report store); hub `Backup`/`RestoreTest` filled (cross-repo golden + hub logs a failed restore-test); `--selftest=backup`/`--selftest=restore-test`. Live-validated on demo-felhom.
- **v0.6.0** — slice-6 **Phase B** (PBS offsite tier): `internal/pbs` — a fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/TaskStatus, node-from-UPID); the verify maintenance loop (own cadence, NOT gated/journaled — like the watchdog); `PBSSnapshot` reporting filled (cross-repo golden + hub failed-verify WARN); truthful vzdump mode from the task log; `--selftest=pbs-verify`. Backup/restore-to-PBS reuse Phase A unchanged. Live-validated against the spike's DooPlex PBS.
- **Next: slice 7 (provisioning + identity-reset + golden base, §9)** — the unified bring-up primitive; restore-overwrite + decommission executors the gate already guards; escrow + host-loss DR.
- **API-first** via a scoped `FelhomAgent` token. Raw root-CLI is **fenced to exactly 3 exceptions**:
keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors. `Client` never shells out;
`Privileged` never makes HTTP calls (asserted by `routing_test.go`). Keep that fence.
- **Every mutating op is async** → returns a UPID → `WaitTask` asserts `exitstatus == "OK"`. A 200 on
the POST is **not** success; authorization can fail at task execution.
- **TLS:** SHA-256 leaf-cert pinning (self-signed host cert). No insecure default.
- **Privsep token gotcha:** a `--privsep 1` token's rights = intersection of the backing user's perms
AND the token's ACLs — the role must be granted on **both**, or every call 403s.
- Destructive ops go through the reconcile gate / signed-jobs path — never call `Client.DestroyLXC`/
`Vzdump`/`SetConfig` ad-hoc (REUSE.md §3).
## Demo host (for live tests)
Node **`demo-felhom`**, API `https://192.168.0.162:8006`, PVE 9.2.2; leaf-cert SHA-256 fingerprint starts `BA:7C:99:7D:45:D0…` (verify it still matches before a live run — the agent pins it). `pveum`/`pct` ops need `root@pam` on the PVE (SSH alias `felhom-pve`) - available to Claude Code
Node **`demo-felhom`**, API `https://192.168.0.162:8006`. SSH alias `felhom-pve` (root@pam) —
available to CC as plain `ssh felhom-pve`. A **second demo node `demo-hp`** (HP t740, node name
`felhom-host`, `ssh demo-hp` — no baked key; break-glass root via hub `host_recovery/demo-hp-bb76ea` +
`sshpass`) is the **designated drill+build VM host** per the 2026-07-25 operator ruling, and that ruling
is **realized** — it hosts drill VM `300` (`drill-r50`), so **start there**, not on DooPlex. (The
historical golden-bake `drill.qcow2` still lives on DooPlex and is a bake fixture, not a drill target.)
**Which box is safe to break, and what may be done to each:
`felhom.eu/documentation/runbooks/target-selection.md`** — read it before any destructive test. Both
nodes + the break-glass recipe: `felhom.eu/documentation/operations/nodes.md`. The agent pins the served leaf cert — verify the
fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the
demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon.
Selftest modes (run from the build server, pointed at the demo API):
- `--selftest` / `--selftest=read` — read-only health checks.
- `--selftest=task -vmid N` — reversible snapshot→rollback→delete on guest N (gated; never under bare `--selftest`).
- `--selftest=hub` — one collect + report round-trip to the hub.
- No flag → the **daemon** (poll loop); requires `hub` config.
> **TEMPORARY — felhom-pve is at a remote site (until ~2026-08-02).** The home-LAN literal
> `192.168.0.162` is NOT reachable from DooPlex for the duration. Access via Tailscale:
> felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already
> points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on
> return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct**
> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP**; re-check its address
> rather than trusting one written here (`ip -br addr show vmbr0` — it read `192.168.0.162/24` on
> 2026-07-30, and `felhom-pve-lan` from DooPlex is still `No route to host`). Details + findings:
> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
>
> **The "agent does not run at the remote site" warning this block used to carry is RETRACTED
> (2026-07-30) — it was true before R-50 and is false now.** `localapi` no longer binds a LAN literal:
> since the R-50 island migration (2026-07-25) it binds `169.254.253.1:8443` on `vmbr9`, which is
> location-independent by design, and `proxmox.endpoint` is `https://127.0.0.1:8006`. Verified live:
> `systemctl is-active felhom-agent` → `active`, `felhom-agent --version` → 0.115.0, and the per-guest
> local API answered `GET /disks` over the island. No config edit and no Viktor GO are outstanding.
> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11; `pct` commands over SSH
> needed `export MSYS_NO_PATHCONV=1`, and every remote command used
> `SSH=/c/Windows/System32/OpenSSH/ssh.exe`. Agent deploy was a two-hop copy via the Windows box
> (`cygpath -w` for the local scp path; CRLF hazard on config files).
## Conventions
### Trunk-based — no branches
All shippable work commits **directly to `main`**; `main` is always equal to what is deployed. **Do NOT create feature/fix branches.**
- **Report-only artifacts** (audits, findings, fixspecs, reconciliations) → `felhom.eu/documentation/` (`audits/`, `backlog/`), committed to `main`. Never a branch, never left loose at the repo root.
- **Risky/supervised fixes** (agent / golden / provisioning / destructive) are spec'd, then implemented **during the supervised session itself, directly on `main`** — not prepared ahead on a branch. (This is the common case for *this* repo.)
- **Unattended escape hatch:** if a fix can't be cleanly verified/shipped, **revert it and report it** for a supervised redo (or paste the diff into the spec doc in `documentation/`) — do not park it on a branch.
- This supersedes any older "prepared on branch `fix/…`, pending review" pattern.
All shippable work commits **directly to `main`**; `main` equals what is deployed.
- Report-only artifacts (audits, findings, fixspecs) → `felhom.eu/documentation/` (`audits/`, `backlog/`).
- Risky/supervised fixes are spec'd, then implemented **during the supervised session, on `main`**.
- Unattended escape hatch: if a fix can't be cleanly verified/shipped, revert + report — never park on a branch.
> **In every repository where you make a change, update both files in that repo:**
> - **`CHANGELOG.md`** — a cumulative log of **all** changes; newest entry on top.
> - **`REPORT.md`** — **overwrite** with a summary of the **most recent** implementation (or significant validation/operational run) only; not cumulative.
> - **`CHANGELOG.md`** — cumulative log, newest on top.
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
>
> **Never write secrets** — tokens, passwords, private keys, API keys — into `CHANGELOG.md`, `REPORT.md`, or any committed file. Reference them as "stored out-of-band" instead.
> **Never write secrets** into any committed file — reference them as "stored out-of-band".
- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than guess** when you'd otherwise invent input/output.
- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than
guess** when you'd otherwise invent input/output.
- **A health check issues no block I/O** — no `statfs`, no `getdents`, no read, write or `fsync`, **not
even behind a timeout**. Liveness is decided from `/proc` and kernel state. The full rule + the
measurement lives in `felhom.eu/CLAUDE.md` "Code quality rules"; it is repeated here because health
checks are written in THIS repo and that file does not load in an agent-only session. R-117 spike §6.3.
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
- **Run `python3 scripts/agent_gates.py` from the repo root after ANY change in this repo.** It is
the ONE entry point for this repo's gates. Today it runs one — `reuse_refs_check` over this
repo's `REUSE.md` — and it exists at one gate on purpose: a census on 2026-08-02 found that every
check a `CLAUDE.md` names was passing and two of the four nobody is told to run were failing, and
this repo was the extreme case, with nothing running against it at all and 90 cited paths checked
by no one. It grows when the agent grows a second check. `--fast` selects the gates that touch no
network and no container runtime; today that is all of them. A missing gate is a FAILURE, never a
skip. **The shared `reuse_refs_check.py` lives in `felhom.eu/scripts/` and is never copied here**
— a copy would recreate the drift it detects; an absent sibling clone FAILS the gate.
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It
is per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run
WARNS when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the
session report when you use it.** Both facts are why CI is still owed (`OPEN-ITEMS.md` R-168).
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
- **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring`
(remote pulls) — English, keys-never-values, durations on outcomes; full rules in
`felhom.eu/documentation/runbooks/logging-conventions.md`.
### Live validation
Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end. The forbidden shortcut is
BYPASSING it (the F9 episode: raw guest-attach + hand-set state). Invoking the exact endpoint the UI
invokes is an acceptable proxy when a browser isn't available — say which method was used. Low-level
mechanism tests where the direct call IS the mechanism are exempt.
## Workflow & artifacts
- Implement **`TASK.md` / `TASK-*.md`** specs (when placed as `TASK.md` or told to implement one), then push + CHANGELOG + REPORT.md.
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability for, including live validation on the demo nodes and the demo Proxmox host (CC has root@felhom-pve SSH + the felhom-agent token). A step is human-only only when it genuinely needs physical presence, a real-world decision, or credentials CC truly lacks — mark those steps HUMAN. Do not decline a whole procedure because it touches a live host or a privileged token. (Judgment still applies: confirm before irreversible ops on real customer data — but demo scratch guests are fair game.)
- Implement **`TASK.md` / `TASK-*.md`** specs (when placed as `TASK.md` or told to), then push +
CHANGELOG + REPORT.md.
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability
for, including live validation on the demo Proxmox host (CC has root@felhom-pve SSH + the
felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a real-world
decision, or credentials CC truly lacks. Judgment still applies: confirm before irreversible ops on
real customer data — demo scratch guests are fair game.
+409
View File
@@ -0,0 +1,409 @@
# CONTEXT — felhom-agent working state
> Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
> entry = current); the end-of-task detail lives in `REPORT.md`.
## Current
- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
`settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
weekly tier → weekly on its own; newborn → UNKNOWN. **The trap, so it is not reintroduced:** the
literal reading of R-86 — *"due when the newest archive is ≥ 24 h old"* — is NEVER true on a daily
tier (a new archive resets the age before it reaches the lag), so it switches restore-testing off
where it matters most. Red-proved at 0 runs over 5 simulated days.
- **The state now records WHICH archive was proven**, not just when a tier passed. A pre-R-86 file
keeps its time (ordering survives) and yields no proven archive → each tier is due once after the
upgrade, deliberately.
- **The old cadence key:** `restore_test_cadence_seconds` is DEPRECATED. Negative still DISABLES
(verbatim); a positive value now seeds the **settle lag** and the daemon WARNs once at start-up
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
(default 24 h). It is NOT carried into the evaluation interval.
- **6 h is bounded from both ends:** measured evaluation cost (local 18 ms, PBS-over-WAN 392 ms,
both 430 ms) says cost is irrelevant; the ceiling is that a FAILING tier stays due, so the
evaluation interval is also its retry interval for a multi-GB restore.
- The due-check now runs **before** the heavy-operation gate is taken (a frequent poll must not be
able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
- New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
- **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
`/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
host tier has never been restore-testable there, and the due-check cannot distinguish that from
a newborn tier.
- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
`RecoverStaleLockedGuests` miss campaign fault 11 — that recovery acts only on a **stale vzdump
lock** (fault 11's guest was unlocked) and runs **once at agent startup** (fault 11's guest went
down while the agent was already up). `onboot` is the deliberate-stop discriminator and is *not*
invented here: it is already what `stalelock.go` uses for this decision, it is 0 on scratch/golden
guests, and it is what `pve-guests` consults at host boot — so the agent agrees with the platform
instead of keeping a second private definition of "should be running". Retry bounded at 3
(1m/2m/4m) then escalates **once**; an unbounded silent retry loop is the over-correction here.
Live on demo-hp: **120 s unattended** recovery vs the incident's **587 s** with a human; Scenario B
proven (an `onboot:0` guest left stopped throughout). Detail: `REPORT.md`.
- **2026-07-28 — F-LEAK took THREE attempts; v0.108.0 and v0.110.0 are the corrections.** The cause is
structural: `FelhomAgentGuest` is granted at `/pool/felhom` and a guest joins that pool only when its
restore **completes**, so a *failed* restore-test leaves a pool-less guest out of reach (403).
**(1) v0.107.0 pool adoption — REFUTED LIVE:** `PUT /pools/{pool}` also requires `VM.Allocate` on the
VM being added, so membership cannot bootstrap its own authority; removed in **v0.108.0**.
**(2) host-install v1.21.0 per-path `/vms/990000..990009` ACLs — works, but exactly ONCE per slot:**
PVE's destroy calls `AccessControl::remove_vm_access` (`API2/LXC.pm:906`) which deletes every ACL at
`/vms/<vmid>` (`AccessControl.pm:1898`) — **the grant is consumed by the op it authorises**. Caught by
counting ACL rows after the fix, not by reasoning. **(3) v0.110.0 SHIPPED —
`Privileged.DestroyScratchLXC`, the FOURTH root-fenced exception** (was exactly three: keyctl
`pct create`, USB mount/fstab, SMART/sensors). Band enforced in **sudoers literally**
(`pct destroy 99000[0-9] --purge`) + re-checked in code + journal provenance at the caller; none is
consumed by use. API destroy still tried FIRST; band ACLs stay provisioned so the common case needs no
privileged call. **Ships with a sudoers change — deploy `configs/felhom-agent.sudoers` WITH the
binary.** Live: token 403 on a stranded scratch → fenced path removed the guest and all 3 LVs; sudo
PERMITS the band and REFUSES `9201`/`9100`/`9999`/`990010`/`1`, and refuses `pct start 990000` too.
- **2026-07-28 — v0.109.0: the guest-power watchdog got the observable it shipped without.** A
self-correction: v0.107.0's watchdog logged only at startup and when it *acted*, so on a healthy box
its health could be read only from **absence** — F-OBS's exact shape, shipped in the same session
F-OBS was fixed in the controller. Now an INFO summary every 10th sweep carrying
`sweeps_since_boot`/`guests_evaluated`/`currently_stopped`. An **aborted** sweep (unproven
ownership) does not count, or the heartbeat would claim liveness for a watchdog examining nothing.
- **2026-07-28 — v0.106.0: F-CRIT-2 fixed — a failed backup no longer looks like a fresh one.**
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, NEWEST) as a successful
backup, so the tier reported fresh, went **not due**, and was never retried — 7 days of silence on
the real 168h cadence, invisible to both the R-88 breaker (defers only DUE tiers) and the hub
deadline monitor (reads the same freshness). Now only *plausibly complete* entries count, via a
measured floor `minPlausibleArchiveBytes` = 1 MiB; undecidable ⇒ not counted.
**Size is the only tier-agnostic discriminator**`verification` and `encrypted` are absent on
every local (dir) archive and on a good PBS snapshot until verify-new catches up, so gating on
either would reject 100% of local backups and cause fleet-wide backup THRASH. Floor measured:
smallest real backup on the fleet is 612,397,450 B, so 1 MiB leaves 584x headroom (asserted by a
test). Rejections logged at WARN once per volid. Re-tested live by replaying campaign fault 2 on
demo-hp — both directions, incl. a no-thrash window with 91 scheduler ticks as the positive
observable. Deployed on both boxes. Detail: `REPORT.md`.
**Also established:** server-side prune does NOT count phantoms toward `keep-last` (dry-run kept
2 real + the phantom) ⇒ **no retention/data-loss bug** — but it never removes them either, so they
accumulate. Filed as R-99 (LOW).
- **2026-07-25 — v0.95.0 (additive): SMART coverage fixes (spike B+A) + device model.** Union-path
drives (USB/registry) now get SMART via `storage.SmartReader.SMARTForBacking` wired into the localapi
`/disks` union (localapi `Smart` seam); `smartDeviceFor` resolves dm/LVM to the whole disk via
`/sys/block/<dm>/slaves` (recursive, skips >1-disk); the builtin `local` dir on the LVM root gets a
**SMART-only** device from its containing filesystem (never touches backing/durable_id — the
removable-safety guard in build() stays intact); `SmartSummary.ModelName` captured from smartctl. The
watchdog `Known` path stays enrich-free. Consumed by controller v0.171.0. Source of WHERE:
`felhom.eu/documentation/audits/SPIKE-smart-coverage-2026-07-25.md`.
- **2026-07-24 — v0.94.0 (additive): SMART serialized into /disks.** `localapi.DiskInfo` gains
`Smart *hub.SmartSummary` (omitempty), copied from the target's already-computed Observe-time
enrichment when `Health != ""` — no new smartctl load, no endpoint, no sudoers/MinAgent change. The
controller v0.169.0 renders a "Lemezek állapota" card + 6h degradation alert from it; old controllers
ignore it. **NOTE: at the remote-site vacation window the agent is DOWN (localapi binds .162 → fails),
so live /disks-from-real-agent validation is deferred — the field is unit-proven; publish only.**
- **2026-07-22 — v0.93.0 is the FLEET AGENT.** Built, published (sha `a68b2ff73200622e…`),
Day-0-manifest-vouched (MinAgent also 0.93.0, operator-ruled) and deployed to BOTH boxes
(`demo-felhom-8363b5` + `demo-hp-bb76ea`, the latter over G1 break-glass — still no key baked);
clean-restart 5/5 on both, `.bak-0.92.1` retained. Discharges the onboarding runbook §A5
ceremony gate. Record: `felhom.eu/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`.
**The bullet below ("agent is DOWN … deployed 0.90.0") is SUPERSEDED history** — vmbr0 was made
static .162 on 2026-07-20 (F1 mitigation) and the agent has been up since; kept for the record.
- **2026-07-20 — REMOTE SITE until ~2026-08-02; the agent is DOWN there and cannot self-recover.**
felhom-pve moved off the home LAN; `ssh felhom-pve` = tailnet `100.70.170.35` (direct, ~37 ms). The
host is on DHCP and holds `192.168.0.147`, so `localapi`'s literal `192.168.0.162` bind fails with
`bind: cannot assign requested address` — the daemon exits ~1.1 s after start, systemd gave up after
4 retries, and a manual restart reproduces it exactly. Deployed binary is **0.90.0**. Fix needs
`listen_addr` in `/etc/felhom-agent/agent.json` **and** the guest bootstrap endpoint (plus the
pinned leaf's SAN) → **Viktor GO**; re-pinning to another literal just re-breaks on the next lease.
Also re-observed each start: `pbs: cannot read token secret … /etc/pve/priv/storage/felhom-pbs.pw:
permission denied` (R-39-adjacent). Evidence + ranked findings:
`felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
- **v0.90.0** (2026-07-17) — **agent train: guest RAM resize (R-24) + fast-tick (R-28); LIVE on BOTH
demo hosts (felhom-pve + nested demo-vm-felhom-4846bc).** MinAgent coupling: felhom-controller
v0.143.0 gates its resize UI on this agent. (1) **R-24 guest RAM resize (controller-direct)**
self-scoped `GET`/`POST /guest/memory` (`internal/localapi/guestmemory.go`); the AGENT enforces
every bound fresh per request (min 2048 / max host_total2048 / shrink floor max(2048, usage+512)) and
applies via PVE `SetConfig`**live cgroup apply, no reboot** (Phase-0 PROVEN on the nested box; the
break-glass access path + the proof are in `~/.claude/.../nested-vm-access-breakglass.md`). Verify-
after-apply re-reads maxmem before claiming success. New narrow `MemoryOps` seam (GuestAPI untouched);
memory only. (2) **R-28 fast-tick** (`internal/fasttick/`) — while any desired-state item is unapplied
(esp. the pre-tunnel WG-registration window a hub poke can't reach) pulse the shared out-of-band
trigger every 30 s, self-disarm on convergence; four cached sources (desired-gen==0, reconcile
PlannedPending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational). Seams:
`reconcile.Engine.LastResult()` + `wgtunnel.Manager.TunnelConvergence()` (cached — no per-tick exec).
(3) **Guests-0/0** REFUTED live: the 0/0 was the pre-provision window (guest not yet created), not a
pool-membership bug; the fast-tick shortens that window. **OPEN (operator GO):** publish 0.90.0 +
hub Day-0 manifest vouch + MinAgent-floor raise to 0.90.0 (password-gated UI; the safety gate — both
agents on 0.90.0 — is satisfied and the coupling is proven live via the version header). See REPORT.md.
- **v0.89.0** (2026-07-16) — **agent train: three bundled agent-plane items; built + published to
Gitea (sha256 `3969fd91…`); paired with hub 0.59.0 (LIVE).** (1) **pbsdr self-grant (R-22)**
closes the F4 self-deadlock: a 403 on the token-auth `StorageEntry` pre-check now self-grants via
the root wrapper + re-reads instead of aborting before the grant (the demo's `felhom-offsite`
case). (2) **escrow config live-reload**`/escrow/preflight`'s `pbs_storage_id` row now reads
the live agent.json (late-bound `CurrentPBSStorageID`) so a pbsdr-seeded id flips green with no
restart. (3) **agent-plane poke listener (Direction-2a)**`internal/poke`: contentless UDP poke
bound to the box WG /32 (port **51822**), leading-edge debounced, fires the hub-loop out-of-band
trigger for an immediate desired-state cycle; enabled with `wg_tunnel.enabled`; first slice of
R-13. Red-proofs for all three (run-fail-revert). **ALL THREE LIVE LEGS PROVEN on the demo
(2026-07-17), demo now LIVE on 0.89.0:** Scenario 4 floor-driven A/B train 0.88→0.89 (operator
signed+enqueued the `agent_update` op — the vouch+floor alone does NOT trigger it; committed, no
rollback); Scenario 1 R-22 self-heal (marker aside + ACLs revoked → `pre-check 403 … self-granting
(R-22)``converged state=adopted` in ~3 s, ACLs restored, offsite active); Scenario 3 poke→tick
~31 ms ep0→box + immediate report cycle (save→tick ≈ ~0.45 s). Details: REPORT.md.
- **v0.88.0** (2026-07-13 eve) — **controller-driven escrow ceremony (agent half), LIVE on demo
host + drill VM (63/63 capabilities both).** `--output=json` machine mode (text mode
byte-identical; extraction into `escrowCeremony()`); the ONE fixed argv
(`escrow.CeremonyArgs()` — shared by the localapi exec + the `escrow-ceremony` capability
(Critical, pbs_dr-gated EXPLICIT) + the new `FELHOM_ESCROW` sudoers alias, three-way
pin-tested); localapi job endpoints (`POST /escrow/ceremony` single-flight 60 s,
status, ONE-SHOT claim → 410, 10-min TTL → `unclaimed_void`, `GET /escrow/preflight`).
R in-memory ONLY (never the job struct — snapshot-hygiene-tested; restart loses it safely).
Live-proven on drill endpoint-exact: stage → preflight all-green (live FELHOM_ESCROW
list-probe) → job ~4 s → hub blob `restic_pw_sha256` covering (repaired the spike's hash-less
blob) → claim 200 once → 410. Coupled: controller v0.127.0 (MinAgent 0.88.0 for the wizard).
**OPEN: publish 0.88.0 + Day-0 manifest vouch (operator) at the next train; deployed hosts got
direct deploys.** Details: REPORT.md + felhom.eu RUNBOOK-escrow-ceremony.md (F1 threat model).
- **v0.87.0** (2026-07-13) — **SystemDisks device-mapper walk (IA finding 2, MEDIUM):
legacy-boot hosts get a working drive wizard.** Operator ruling (approved 2026-07-13,
verbatim): *resolve device-mapper/raid parents — for the root filesystem's backing block
device, walk `/sys/block/<dev>/slaves` recursively down to physical disks; those, plus any
ESP holder when present, are system. Disks outside that set become wizard candidates (still
subject to the existing data-bearing guards). The all-system fail-safe remains ONLY for walk
failure — it returns to being the error case, not the legacy-boot common case.* Implemented as
`physicalDisksOf`/`walkSlaves` + `HostReader.BlockSlaves` (one seam method); per-branch
conservatism (any unresolvable slave → ok=false → unchanged all-system path); signature test
`TestSystemDisks_WalkTopologies` (root-backing disk ALWAYS system — never weaken). §3 spike
transcripts: drill (legacy) dm-1→sda3→sda; felhom-pve (EFI+LVM) ESP+walk agree on sda →
byte-identical regression. §13.2 wizard leg COMPLETE (offered → enrolled → formatted →
torn down, boxes as found) + Day-0 manifest vouched to 0.87.0 (operator). The leg also
surfaced two CONTROLLER bugs (fixed same-day: v0.126.3 claimed-box wizard CSRF, v0.126.4
502-through-CF + native-alert ban).
- **v0.83.0** (2026-07-11, LIVE on felhom-pve; NOT published — Peti stays 0.81.0) — **observability
pass** (pairs with controller v0.116.1 + hub v0.46.0). `applog.New``(logger, *Ring)`: slog
fan-out, journald at the configured level, ~1000-entry ring FIXED at DEBUG. `GET /debug/logs`
(local API, token-authed; the controller Debug page's Ügynök tab) + request-level DEBUG
middleware. Heartbeat log-pull: envelope `log_tail_requested` → next heartbeat ships `log_tail`
(128 KB, consume-once; failed push re-armed by the next envelope; `operator log pull served` INFO
on fulfillment). Gap-fill sweep: netverify phase/verdict lines (job start, trigger outcome,
/proc/mounts verdict, journal bytes, classification code, rollback outcome, durations), netmount
unit steps, signedjobs op-received (class/host/expiry — never signatures) + fetch duration,
selfupdate invariants + download sha/duration, disks outcome INFOs, controller-swap pre-pull +
health verdicts, desired/loop per-exchange DEBUG. Logging conventions:
`felhom.eu/documentation/runbooks/logging-conventions.md`. OPEN: the hub-side live pull awaits
the operator's button click (hub UI password-gated); pre-existing lanresolver
permission-denied WARN on /var/lib/felhom-agent/guests noted in REPORT.
- **v0.77.0** (2026-07-09) — **fork-4: escrow the offsite restic repo password under R.** `IdentityBundle`
gains `ResticRepoPassword` (rides the existing age-under-R `WrapIdentityBundle` path — validated by the
custody spike `febdc56`). New `POST /escrow/stage-secret` (`withGuest`) transiently stages the
controller-pushed password (0600, never logged), which the `--selftest=escrow-create` ceremony
auto-injects into the bundle and then wipes. `AttachResticPassword`/`StagedResticPasswordPath`/
`WipeStagedResticPassword` added. Pairs with controller v0.105.0 (push + atomicity gate + DR inject +
`DRResticCoord`). **NOT yet live-validated** — the supervised escrow ceremony is operator-run.
- **v0.76.0** (2026-07-08, LIVE on felhom-pve + **PUBLISHED sha `9828c5f7…f50b`** — THE Day-0
manifest bump target; **0.75.0 superseded unpublished**) — **GL-5b / G12: restore-test
full-fidelity**. Params derive from the ARCHIVE's embedded config (`drRestoreOverrides`, same as
DR — the old live-source-config path verified the wrong object AND dropped storage mpN per PVE's
all-or-nothing rule; deleted with `bindMountOverrides`/`archiveVMID`). NEW mount-parity assert
(restored mpN vs archive; miss/mispath/undersize/extra = FAIL naming the delta) + `MountParity`/
`MountInventory` on the wire record (additive). Live-proven: scratch 990000 ← 6.5GB 9201 archive,
parity ok, inventory mp0 200G+mp1 50G+2 throwaways, **3m4s local tier** (cheaper than feared);
rotated-out archive volid → clean up-front refusal (nice failure mode). bringup.go untouched.
- **v0.75.0** (2026-07-08, LIVE on felhom-pve) — **GL-5 / go-live G8: guest-loss DR bring-up
actually restores** (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the
COMPLETE explicit restore param set derived from the archive's embedded config (NEW
`Client.ExtractArchiveConfig`, 200 under the scoped token) — **two live-discovered PVE rules: mpN
params need an explicit rootfs, AND unlisted mountpoints are silently DROPPED** (first run booted
without mp0/mp1!) — storage mpN passed through, structural mp8/mp9 → throwaways, then step 4d
swaps the REAL binds in via the host runner (root pct; new `EngineOptions.HostRunner`+`StateDir`
seam) and deletes the unusedN residue. Scratch-DR live-proven end-to-end (9310 from a real 9201
archive: mp0 200G + mp1 50G + real binds + no residue + clean teardown). Provision = nil
overrides (regression-tested). NOTE: published/vouch-pending agent is 0.74.0 — publish 0.75.0
before/with the manifest bump. OBSERVATION: the DR selftest hardcodes KeepMAC=true — a scratch DR
while the SOURCE guest is live briefly duplicates its MAC on the bridge (pre-existing; fine for
supervised runs, worth a -keep-mac flag someday). Full customer-data DR drill = GL-6/S5 family.
- **2026-07-07 — v0.74.0 Gitea-PUBLISHED (RUNBOOK GL-1)** — the LIVE felhom-pve binary's exact
bytes, sha256 `1ec3f58842edce1e…76af05`, anon-fetch-verified. This supersedes/closes every
standing "publish 0.6x + Day-0 vouch" OPEN item below (0.64→0.73 were never published; 0.74.0
is the vouch target). Golden 0.103.0 published in the same run (felhom.eu execution record
`documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`). **Day-0 manifest vouch = operator
step** (agent 0.74.0 / golden 0.103.0).
- **v0.74.0** (2026-07-07) — **campaign-2 R2 CLOSED; the mislabelled "R1" was a symptom** (LIVE on
felhom-pve). Pool membership is what lets the pool-scoped token reach a guest; `pct restore --pool`
sets it only at CREATE, so a restore-over-existing dropped 9201 from the `felhom` pool → no
`VM.Audit` → restore-test's *existing* `bindMountOverrides` never ran → "mp8 … only possible for
root". Fix: `Client.PoolAddVMID` + bring-up re-asserts membership post-restore (warn-not-fail).
Role/ACL + `bindMountOverrides` untouched (both correct). **Live restore-test PASSED for the first
time** once the pool was healed (Part A one-liner): read config → neutralize 2 binds → restore →
boot+running → clean teardown, 4m35s. B3 (scratch-teardown 403) confirmed a cascade — no code.
OPEN: DR `bring-up -mode dr` bind-override gap (spike `SPIKE-dr-bindmount-source-2026-07-07.md`:
small known-constant override reusing `bindMountOverrides`; mp8/mp9 are structural constants).
- **v0.73.0** (2026-07-06) — **F2 mount-role fallback CLOSED** (LIVE on felhom-pve). `roleForMountPath`
gained a mount-table fallback (Impl-2b style): a bind-mounted RAW enrolled user-data drive is not a PVE
storage, so it fail-safe'd to `system` and the eject/decommission gates 403'd EVERY user-data drive
(campaign F2, `where=/mnt/teszt_enroll role=system`). Device-keyed classification + whole-disk containment
(`storage.SameWholeDisk`); Observe-error keeps the fail-safe BEFORE the fallback. Only `roleForMountPath`
touched. Live-proven full lifecycle on teszt_enroll (eject/decommission 200, no-rebind across restart,
end==pre). OPEN follow-up: the `deviceRole`/`roleForMountPath` unification refactor (deferred).
- **v0.72.0** (2026-07-05) — **OOB operator access (merged E1+H1)** — TASK H1, provenance both
`SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`. Operator `/32` RENDERED into wg-felhom
AllowedIPs (survives self-heal, [OF-1]); dedicated `internal/felhomsshd` (port claim + config
render→sshd -t→reload + operator authorized_keys + heal + oob heartbeat stanza); static
`inet felhom_oob` belt (agent mutates SET ELEMENTS ONLY); `configs/felhom-sshd.service` (NO
RuntimeDirectory [SF-1]) + `felhom-oob.nft` + `felhom-op.sudoers`; `FELHOM_SSHD`+`FELHOM_OOB`
grants; `oob.enabled` DEFAULT FALSE. Live on felhom-pve (8822, belt filled, operator SSH as
felhom-op with scoped sudo); hub v0.35.0. Rollback `.bak-0.71.0`. 5 live-found bugs fixed (port
path, self-listen flip-flop, nil-block lockout, reachable-via-dial, operator-configured source).
- **v0.71.0** (2026-07-05) — **management-plane break-glass: privsep-dir watchdog + mgmt_plane
health** — TASK G1 (prereq for felhom-sshd/H1), provenance `SPIKE-felhom-sshd-2026-07-05` §8.
Host artifacts (`configs/felhom-privsep.tmpfiles` + `felhom-mgmt-watchdog.{sh,service,timer}`) make
`/run/sshd` boot-persistent AND auto-heal it every ~60s **agent-independently** (heals with the
agent stopped — proven live: `/run/sshd` removed → restored in 30.0s, `:22` back, no login).
`internal/mgmtplane` reports the additive `mgmt_plane` heartbeat stanza; hub v0.34.1 raises
`mgmt_plane_healed`. **NO unit declares `RuntimeDirectory=`** (the incident cause). H1 may now
assume `/run/sshd` is guaranteed present. Live on felhom-pve; rollback `.bak-0.70.0`.
- **v0.70.0** (2026-07-05) — **agent self-update (operator-signed A/B slots + crash-loop
auto-rollback)** — TASK D1, provenance `SPIKE-agent-selfupdate-2026-07-05`. An operator-signed
`agent_update` op (version+sha256, sha is the only integrity root) rides the signed-jobs gate;
`internal/selfupdate.Executor` downloads+verifies+hands to `felhom-selfupdate-guarded apply` (root
re-verify → A/B atomic flip → pending marker → detached restart); the new binary commits after a
60s dwell; a crash-looping binary is auto-reverted by `OnFailure=felhom-agent-rollback.service`
(first-crash trigger [SF-1]) with the tuned `[Unit]` start-limit (120s/4) as backstop. Host
artifacts + sudoers `FELHOM_SELFUPDATE` + `felhom-host-install.sh` day-0 install + report field
`selfupdate_pending`. Green tests + companions. **LIVE-VALIDATED on felhom-pve (2026-07-05): all 4
drills PASS** — happy path (0.70.0→0.70.1 signed op → download+verify+flip+commit), crash-rollback
(0.70.2-crash → OnFailure → **~2s crash-to-recovered**, byte-identical revert, no loop), no-pending
guard, gate refusal (non-pinned key). Full agent-side pipeline ran real (envelope injected into the
hub `signed_jobs` queue — CC lacks the hub global operator key; hub enqueue-auth is hub-unit-tested).
Box restored to canonical **v0.70.0** (host artifacts KEPT installed; scratch operator key REMOVED —
self-update dormant until an operator pins a real key, a Day-0-vouch-style follow-up). Rollback
`felhom-agent.bak-0.69.0`. OPEN (v1 scope-outs): no hub-floor auto-update, no failed-update
auto-retry, no pending-timeout auto-rollback; per-crash OnFailure can double-fire (idempotent —
future: serialize the rollback oneshot). Detail: REPORT.md.
- **v0.69.0** (2026-07-04, live on felhom-pve) — **S5: host-loss DR — safe halves shipped**.
**Part 1** `wgtunnel.InstallRecoveredKey` — writes an escrow-recovered WG privkey (create-only,
refuse-overwrite) so the tunnel re-establishes with the SAME identity/pubkey (same /32), no keygen;
wired into `--selftest=identity-consume -install-wg-key` (opt-in; pre-S3 blob → logged fresh-keygen
fallback). **Part 2** new `internal/dr` — consumes the host_loss `restore_directive` (was
logged-ignored) into an inspectable RestorePlan via AddConsumer: per-guest {vmid,archive,target,
sizing} + per-drive {durable_id→mount} + offsite PBS coord; DERIVE-AND-SURFACE only (Consumer has
no restore/destroy dep — execute-nothing is structural). Tests + red-proofs (WG create-only; plan
mode-gate). **Part 3** hub escrow-GET NOT needed (operator exports the blob via `sqlite3 writefile`
on a cp'd hub.db). **Part 4-A** re-attach wrong-disk safety already unit-proven
(`ResolveStorageDevice`: match resolves, absent/mismatch ERRORS, non-uuid scheme refused — never a
near disk). **Part 4-B (destructive in-place 9201 restore) PREPARED + OPERATOR-GATED, NOT executed**
— pre-flight green (offsite ct/9201 restorable per S4.1); the operator runs the R-consume steps +
confirms the destroy (§9-4a: CC never runs a consume/R command — see [[operator-present-one-time-secrets]]).
OPEN: the operator-run 4-B drill; guest_loss DR; hub-driven full-auto DR. Rollback
felhom-agent.bak-0.68.0. Detail: REPORT.md + doc-06 §3.5/S5.
- **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**.
**Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from
`config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when
`SourceTier=="pbs"` (`main.restoreTaskTimeout`); local tier UNCHANGED. Fixes the WAN restore being
killed at 10m → mid-restore teardown → leaked scratch. **Teardown "VM.Allocate" follow-up =
PHANTOM (diagnosed, not blind-fixed):** ran the restore-test on the AGENT-TOKEN path sourcing the
offsite (pbs) backup → `pass:true verified:boot+running`, teardown succeeded (`torn down
vmid=990000`, no 403), scratch band clean. The earlier 403 was the 10m-timeout consequence (guest
not yet pool-associated); the scratch is restored INTO `/pool/felhom` (ACL already grants
VM.Allocate) so teardown is authorized once the restore completes. **No ACL/host-install change.**
OPEN: publish 0.68.0 + Day-0 vouch; Tier-1/Tier-2 split for offsite-as-default; S5 DR consume.
Rollback `felhom-agent.bak-0.67.0`. Detail: REPORT.md.
- **v0.66.0 + v0.67.0** (2026-07-04, live on felhom-pve) — **S4: PBS over the tunnel**. **v0.66.0**:
wgtunnel **v4-pin** (renderConf writes the resolved A LITERAL, never DNS/AAAA; `Resolver` seam,
lowest addr; cached → steady-state zero-DNS/zero-exec) + **re-resolve watchdog** (`Manager.Watchdog`,
loop-only; handshake stale > `stale_after_seconds`=180 → re-resolve → IP-changed re-render+restart)
+ FELHOM_WG **Critical** flips (conf-install/enable/restart/handshake-read). **v0.67.0**:
**namespace-aware PBS client** (Config.Namespace → `Snapshots ?ns=`, `Verify ns=`; root-ns
unchanged) — the operator-approved fix after Phase-1 showed the ns-unaware datastore-root 403s a
per-tenant token. **Live Scenario-D (all green):** real vzdump of 9201 → **ciphertext** in ns
`demo-felhom-01` over the tunnel; ns-scoped verify=ok under the box's own `felhom@pbs!demo-felhom-01`
token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown).
**Confirmed tenant ACL (felhom-hetzner):** `DatastoreBackup` on `/datastore/felhom-offsite/<ns>`
(NOT `/ns/<ns>`) to BOTH user `felhom@pbs` AND token (privsep=intersection; cross-ns 403);
DatastoreBackup can't prune (safety). **FINDINGS:** retarget field is `local_backup_target` (not
`backup_target`); retarget REVERTED to `local` (controller backs up ~every 30 min → single-target
offsite = near-continuous 20-min uploads; needs Tier-1/Tier-2 split); restore-test scheduler needs
a WAN restore deadline + scratch-band `VM.Allocate` before it runs offsite unattended. **OPEN:**
escrow-create (OPERATOR-PRESENT, new R); publish 0.66/0.67 + Day-0 vouch; S5 DR consume. Rollback:
`felhom-agent.bak-0.65.0`/`.bak-0.66.0`. Detail: REPORT.md + doc-06 §3.4/§4.2 + runbook §4a/§4b.
- **v0.65.0** (2026-07-04, live on felhom-pve) — **S3.1 offsite-tunnel client MTU 1420 → 1280**:
resolves `06 §4.3`'s OPEN DECISION left by the CGNAT smoke test. 1420 **silently black-holed bulk
TCP** on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page
(and at S4 the backup itself) drops. New `const clientMTU = 1280` (RFC 8200 IPv6-minimum floor;
outer 1340 v4 / 1360 v6 fits every realistic path), **permanent + fleet-wide + family-agnostic**.
**Client-only by construction** — interface MTU caps box→PBS, advertised MSS caps PBS→box, so the
endpoint's `wg0` is untouched (zero live-endpoint risk). Golden pins exact `MTU = 1280`
(red-proofed vs a 1420 flip); no wire/JSON change. Live: agent re-rendered on restart (hash-gated
apply), conf + live iface both 1280, PBS page loads at 1280 (no regression on wired). OPEN:
true-CGNAT-SIM retest (low risk); publish 0.65.0 + Day-0 vouch (operator); S4 PBS-over-tunnel.
Rollback: `felhom-agent.bak-0.64.0` on the box. The v4-pin (§4.2 determinism) is a separate,
optional future note — NOT needed for MTU correctness.
- **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel`
(keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from
the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam,
revoked-stays-revoked teardown, report stanza) + `FELHOM_WG` sudoers/capabilities +
`IdentityBundle.WGPrivateKey` escrow auto-inject. **`wg_tunnel.enabled` DEFAULTS FALSE** (safety
gate — rollout to Peti's box is a no-op until the production endpoint exists; enabled explicitly
on felhom-pve only). Live: tunnel to ep0.felhom.eu:443 up 3 s after enable (PBS page through
10.77.0.1:8007), reboot-persistent, revocation drill clean, 30-min keepalive soak. GOTCHAS:
hub envelope poll_interval_seconds (hub-side const 900 s) silently overrides agent poll_seconds
on cycle 1; `wg show <if> dump` leaks the PRIVATE key (forbidden everywhere — sudoers only
grants `latest-handshakes`). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix);
publish 0.64.0 to Gitea + Day-0 vouch (operator); S4 points PBS at the tunnel.
- **configs: build-golden.sh v2.0.0** (2026-07-03, @ `ceca355`; no agent version change) — **drill
findings B5 + B1 FIXED** (`DRILL-golden-098-2026-07-03.md`): the controller tag is a MANDATORY
argument (the default rotted twice — a fresh install booted a pre-floor controller, forcing the
guide's manual D.1b update) and the golden now bakes a `felhom-controller-bootstrap.path` unit
(controller deploys the moment the back-half hot-plugs the bootstrap mount — no reboot; installer
v1.9.1's reboot is a redundant belt, kept). **Golden 0.98.3** baked on the drill VM, clean-room
validated (bake integrity → isolated hot-plug proof → local-golden Day-0 → published-artifact
Day-0), published (sha256 b9a02ef1…fd01) + operator-vouched — Day-0 manifest now vouches
**agent 0.63.0 + golden 0.98.3** (the v0.63.0 vouch follow-up below is DONE). Fresh installs land
current and self-manage. NEW operator follow-up (SECURITY): the customer-config `git.token` has
Gitea package-WRITE rights — scope down + rotate (evidence-doc observation O1).
- **v0.63.0** (2026-07-03, live on felhom-pve + Gitea-published sha256 b4a89c81…) — **drill findings
B3 + B2 FIXED** (`DRILL-day0-cleanroom-2026-07-03.md`): `TokenStore.Lookup` reloads the append-only
store once on a miss (cross-process coherence with the one-shot provisioner — no more fresh-install
`/controller/swap` 401 / manual restart; size short-circuit bounds the cost; behind the
`TokenAuthority` seam) + `guesthook.InstallSnippet` issues a fenced `mkdir -p /var/lib/vz/snippets`
first (fresh boxes lacked the dir → the self-heal hook silently never installed). Sudoers gained
exactly that one grant — **ship sudoers WITH the binary** (done on felhom-pve). Red-proofed both;
Scenario-E method: compiled test suite run ON felhom-pve + live channel-health hit-path.
**OPERATOR FOLLOW-UP: bump the hub Day-0 manifest to agent 0.63.0** — until then fresh installs get
0.62.0 and the guide's D.1b restart-first step still applies (narrowed to "< v0.63.0" in the guide).
- **v0.62.0** (2026-07-03) — **audit A1 RESOLVED**: the stale-lock reaper's scan is now
pool-intersected (`staleLockController.Guests()` = `ListLXC``Client.Pool("felhom")` members),
fail-safe skip on pool-read failure; `pve:pool-read` capability (non-critical) + `--selftest`
"pool read" line. Companion host-install **v1.9.0** adds `Pool.Audit` to `FelhomAgentGuest`
**deploy order on any box: rescope ACL first, then this agent.** Per
`SPIKE-a1-pool-membership-read-2026-07-03.md`; red-proofed tests in stalelock_pool_test.go.
- **2026-07-03 — CLAUDE.md refreshed**: version narrative removed (state lives HERE + CHANGELOG top), layout completed (all 17 internal packages + cmd/felhom-opsign); deploy runbook now in the `felhom-build-deploy` skill (`felhom.eu/skills/`).
- **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / format-safety guards / traps / seams, code-verified); maintenance rule active: update it in the same commit that changes a shared helper.
- **v0.61.0** (2026-07-03) — blast-radius audit fixes **B1 + D1 + D2 + D3** from
`felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md`: random temp
staging for root-installed scripts (+ sudoers/manifest glob updates), mkfs-wrapper member/RO
re-checks (validated by `scripts/mkfs-guarded-harness.sh`), classifyClaim empty-lsblk fail-safe,
and the blank-format anti-retarget (durable-id-bound, AGENT-001's benign-branch twin).
- Deployed on demo host `felhom-pve` (node `demo-felhom`), non-root `felhom-agent` service user,
pool-scoped token (`felhom` pool).
## Open threads
- Deferred audit items (housekeeping/design, all INFO): C1 (controller-swap version floor), C2 (NAS
server allowlist), A2 (gate journal cross-check), B2B5, E1/E2.
- Drive-enrollment leftovers: (a) `runStorageInit` slow-device detached-format polling; (b) Impl-3
shared-box operator format gate.
- BUNDLE leftover: non-root agent can't read the PBS key; migration must preserve cert/key/tokens.
- Not run (needs a supervised session): the destructive D1/D3 live proofs (real mkfs on a crafted
member; a live /dev re-enumeration race during a real format).
+168 -4
View File
@@ -23,9 +23,35 @@ internal/authz/ # operator signed-op verifier (SSHSIG); durable nonce sto
internal/hub/ # daemon: host-report collector + Bearer client + resilient poll loop
internal/reconcile/ # reconcile engine + reversibility gate + op journal + crash recovery
internal/storage/ # storage-target observer + durable_id + fast-poll watchdog (slice 5)
internal/capability/ # privileged-capability self-probe (manifest + sudo -n -l check, v0.44.0)
configs/agent.example.json
```
## The `capability` package — privileged self-check (v0.44.0)
The non-root agent depends on a fixed set of `sudo -n` grants (`configs/felhom-agent.sudoers`). When
the 2026-06-28 root→non-root cutover silently dropped some (lxc-info, make-private, …), features broke
unnoticed until a user hit them. `internal/capability` makes that loud:
- **`Manifest()` is the source of truth** for the required `(binary, representative-arg)` vectors,
seeded from the sudoers audit (the OK + CLOSED rows; the surfaced/deferred rows are excluded). When
you add a privileged `runner.Run` call that needs a NEW grant, add it to BOTH the sudoers file AND
the manifest — `manifest_test.go` asserts **manifest ⊆ sudoers** at build time (with a red-proof
that dropping the lxc-info grant fails the gate), so a missing grant is caught in CI, not in prod.
- **`Prober.Probe`** lists each vector with `sudo -n -l` (a policy LIST — never executes, safe for
mkfs/pct) + an `os.Stat` existence check → an `ok`/`degraded` snapshot. It runs at startup (logged
loud) and on every collect, riding the hub report as `HostReport.Capabilities`; the hub
(`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
— the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin`
`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.
## The `storage` package — observe + watchdog (slice 5)
Read-only this slice (no hub desired-state until slice 10):
@@ -47,6 +73,18 @@ The reported `StorageTarget` shape is a cross-repo contract duplicated in `felho
`internal/hub/testdata/host-report.golden.json` is byte-identical with the hub's copy and a
bidirectional key-set test guards drift.
### DR recipe — the storage/guest/PBS half (v0.38.0)
The host-report carries an additive `dr_recipe` section (`internal/hub/dr_recipe.go`) — the agent half
of the secret-free reconstruction recipe (`SPIKE-dr-recipe-2026-06-16.md`). It is the non-secret
**re-provision scaffolding** that complements escrow (keys) + PBS/restic (bytes): `guests[]` sizing,
`drives[]` (user-data drives by `durable_id` → role → mount → intent), `pve_storage[]` (the
`storage.cfg` defs), and `pbs` coordinates. Built by the pure `BuildDRRecipeHostHalf` from facts the
report already collects — **no new reads**. **Boundary:** every field is an identifier/intent/size/
coordinate — never a key/password/token/hash/`ENC:`; the PBS key + restic password stay in escrow.
`recipe_version=1`, ignore-unknown on read, pinned in the cross-repo golden. The hub assembles it with
the controller's app half into one customer recipe.
### The privileged `HostOps` surface (slice 5 Phase B)
The write side — the one place the agent steps outside its Proxmox API token into OS-root —
@@ -58,6 +96,13 @@ and no shell**; tests use a fake (no real root in the suite).
(`What=/dev/disk/by-uuid/<UUID>`, enabled so they survive reboot) — not raw fstab or a
transient `mount`. Benign re-mount is idempotent; **detach** (stop+disable) is destructive
and routes through the gate.
- **Host-reboot remount re-resolution (v0.37.0):** at startup (before binding drives into the
guest) and on the periodic tick, `ReassertEnrolledMounts` re-asserts every enrolled `.mount`
unit that isn't currently mounted: it re-resolves the drive's `uuid:<fs-uuid>` durable id to
its **current** `/dev` node by re-scanning `/dev/disk/by-uuid` (`ResolveStorageDevice`, never a
cached node) and re-runs the idempotent `enable --now`. This re-enables a unit a prior detach
left `disabled` AND tolerates kernel re-enumeration moving a drive's letter (`/dev/sdb``sdc`)
— the reshuffle is a no-op. Already-mounted drives and genuinely-absent UUIDs are skipped.
- **Every argument is validated before any command is constructed** (`validate.go`): UUIDs
against a strict hex regex, mount paths confined + traversal-checked, SMART devices
whitelisted to raw disks, LVM names charset-checked. The adversarial matrix in
@@ -122,6 +167,78 @@ See the table in [`internal/proxmox/doc.go`](internal/proxmox/doc.go). Summary:
lifecycle **including restore** is API-token-covered; OS-root is confined to golden-image
`keyctl` create, host mounts, and SMART/sensors (phase3 §B3).
### Controller swap (agentic controller update, Phase 1 — v0.42.0)
The local API (`internal/localapi/`) owns the in-guest controller image **swap** — the new-architecture
replacement for the controller's dead in-container `docker compose` self-update. The in-guest controller
pre-pulls the target image then calls the agent:
- `POST /controller/swap {image}` (`withGuest`-scoped) → **202**, then async: record previous
(`/var/lib/felhom-agent/controller-swap-<vmid>.json`) → confirm the target is present in the guest →
write `/etc/felhom-controller-image``systemctl restart felhom-controller-bootstrap.service` → poll
the new controller to healthy (`docker inspect`, ≤90s) → **roll back** to the previous image if not (the
guest is never left without a controller). Strict image-ref gate; single-flight per guest (409).
- **(v0.47.0) Verify hardening (F1):** the health poll reads `{{.RestartCount}}` (a 4th inspect
field) — `running && RestartCount>0` is **not** healthy (a process that already crash-restarted
isn't stably up), and a **no-healthcheck** image must report ok on 3 **consecutive** polls (a
stability dwell) before acceptance; a real `healthy` result is trusted immediately. This closes the
hole where a no-HEALTHCHECK image that crash-loops could land one "Running" instant and false-pass
the point-in-time check → no rollback. Verify predicate only — the rollback orchestration + the
`docker inspect -f *` grant (the `*` spans the extended template) are unchanged.
- `GET /controller/swap/status``{state, current, previous, target, error}`.
The agent is **external** to the controller container, so it survives the controller being killed
mid-swap (which the controller cannot do to itself). `GuestBinder.GuestExec` is the single `pct exec`
seam. Exercise directly with `--selftest=controller-swap -vmid <id> -image <ref>`.
## Agent self-update (operator-signed, A/B slots, crash-loop auto-rollback — v0.70.0, TASK D1)
The agent updates ITSELF the same way it swaps the controller: **the thing that performs rollback is
never the thing being updated.** For the agent that means systemd + an ~80-line root shell wrapper
(`configs/felhom-selfupdate-guarded`) that changes almost never; the Go binary is what flips.
**Trust model.** An update is an **operator-signed `agent_update` op** delivered through the existing
signed-jobs pipeline (same LOCKED authz gate as `storage_wipe`/`decommission`). The signed params pin
the exact **version + sha256**, so the pinned sha is the *only* integrity root — **neither a
compromised hub (dumb transport) nor a compromised Gitea (dumb storage) can substitute a binary.**
The operator signs offline with `felhom-opsign -op agent_update -agent-version <v> -sha256 <hex>`.
**The flow** (`internal/selfupdate/` = the Go half; the wrapper = the root half):
1. The control loop sees a pending signed op → the gate verifies it (pinned-key SSHSIG → namespace →
allow-list → crypto → host → time → **durable nonce-burn**) → the `agent_update` executor runs.
2. Executor downloads the binary for the signed version from the config'd artifact host
(`selfupdate.url_template`, `{version}` interpolated) to `/var/lib/felhom-agent/selfupdate/`,
verifies its sha256 against the **signed** value (mismatch → refuse, remove, agent untouched),
and hands it to `sudo -n felhom-selfupdate-guarded apply <staged> <sha>`. The job is completed on
the hub **after verify+download, before apply** (the nonce is already burned — a queued job would
only re-fetch and no-op on the spent nonce; a failed/rolled-back update is visible via the report).
3. The wrapper (as root) **re-verifies** the sha, confines the staged path to the staging dir, asserts
same-filesystem (the atomic-rename guarantee), snapshots the current binary to `.prev`, atomically
`mv`s the new binary into place, writes a `pending.json` marker, `reset-failed`s, and schedules a
**detached** restart (`systemd-run --on-active=2s … systemctl restart felhom-agent`, so the caller
survives to log the handoff).
4. The **new** binary boots; after it has run cleanly for a dwell (`selfupdate.dwell_seconds`, default
60) *and* core init is done, `internal/selfupdate.Manager` calls the wrapper's `commit` (clears the
marker; `.prev` retained as a manual net). A pending marker naming a *different* version than the
running binary is **not** committed — loud WARN, marker left so the report shows why (a human
decides).
5. **Crash-loop auto-rollback (the safety property).** If the new binary crashes, systemd's
`OnFailure=felhom-agent-rollback.service` (the `felhom-agent-limits.conf` drop-in) runs the
wrapper's `rollback`: pending marker present → restore `.prev` byte-identical → clear marker →
restart → the old binary is back **within seconds of the first crash**. On systemd 257 `OnFailure=`
fires on *every* crash, so rollback triggers at the first one; the marker-guard makes every later
fire (and any crash with no update in flight) a harmless no-op. The tuned start-limit
(`[Unit] StartLimitIntervalSec=120 + StartLimitBurst=4`) is the terminal **backstop** (e.g. an
environmental crash loop of the known-good binary → terminal `failed` ≈20s → the hub's
`host_staleness` dead-man's-switch alerts the operator).
**Design provenance:** every systemd behaviour above is empirically validated in
`felhom.eu/documentation/audits/SPIKE-agent-selfupdate-2026-07-05.md` (the SF-findings). The host
report carries `selfupdate_pending` (+ version) so a runs-but-never-commits binary is visible even
though it never crashes. v1 scope: no hub-floor auto-update, no auto-retry of a failed update, no
pending-timeout auto-rollback (a stuck-but-alive binary is caught by `host_staleness`).
### TLS trust
The host serves a self-signed cert. Verification is **not** blanket-disabled. Pick one in
@@ -169,11 +286,58 @@ so cleanly if the token/endpoint isn't configured.
`--selftest=task --vmid N` (explicitly gated) exercises `WaitTask` on a **reversible** op
(snapshot → rollback → delete-snapshot) against guest `N`. Default `--selftest` never mutates.
## Process model (proposed, not finalized — see 03 §3/§12)
`--selftest=bring-up|provision` accept an optional operator **CPU/RAM cap**: `-cores N` and
`-memory M` (MiB). Both default to `0` = keep the golden's baked sizes. When set, the cap is written
into the SAME pre-start config PUT as the identity reset (via `BringUpSpec.Cores/MemoryMB`), so the
guest never boots uncapped — useful when the appliance shares a host with other guests.
Native Go binary, systemd service, **non-root** service user holding the scoped token, with a
**narrow sudoers allowlist** for the three fenced ops. `privileged.mode: "sudo"` matches this;
`"direct"` is for dev/CI where the agent is already root.
**Pool-scoped restore (v0.53.0):** every restore (provision bring-up, DR, and restore-test) allocates
the guest INTO the `felhom` PVE pool (`reconcile.DefaultPool`; `RestoreLXCOptions.Pool``pct restore
--pool`). This is what lets the agent token be scoped to `/pool/felhom` + `/storage/<targets>` instead
of `/` (blast-radius containment on a shared host) — the restore is how a fresh vmid is allocated under
that scoped token (`VM.Allocate`+`Pool.Allocate` at `/pool/felhom`). Layout + validation:
`felhom.eu/documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md`.
**Drive discovery + tracking (v0.55.0, Impl-2a):** `GET /disks/candidates` lists host disks the Impl-1
filter proves are free to enroll (split initialize/attach). The watchdog's known-drive set is sourced
from the intent registry + Felhom `.mount` units (`RegistryKnownTargets`), NOT PVE storages — so a drive
enrolled with no PVE dir-storage is still health-tracked (`Observe()` stays for real PVE storages). The
controller wizard consuming this is Impl-2b.
**Format safety (v0.54.0, Impl-1; hardened v0.61.0, audit D1/D2/D3):** `Format` (mkfs) is gated by a
mandatory **unclaimed-disk guard** (`internal/storage/claim.go`) — it refuses any device not provably
free for Felhom (OS disk, LVM PV, ZFS/mdraid member, foreign mount, read-only; fail-safe on any read
error, and on an empty/target-absent lsblk topology — D2), independent of `DataBearing`. Below the
agent, mkfs runs ONLY through `configs/felhom-mkfs-guarded.sh` (the sole mkfs the sudoers permits),
which re-checks the catastrophic cases as root: system disk, LVM PV (absolute-path pvs), foreign
mount, read-only device, and any LVM/ZFS/mdraid/LUKS/swap member signature (D1 — validated by
`scripts/mkfs-guarded-harness.sh`, a loop-device + recorder harness). The blank-format local-API path
binds to the device's durable id and anti-retarget re-resolves before mkfs, same as the confirmed
wipe (D3, AGENT-001's benign-branch twin). The pool-scoped token does NOT touch mkfs (sudo op) — the
filter + wrapper are the guard. See `SPIKE-drive-enrollment-2026-07-01.md` and
`felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md`.
## Process model
Native Go binary, systemd service, **non-root** `felhom-agent` service user holding the scoped token,
with a **narrow sudoers allowlist** for the fenced host-root ops. `privileged.mode: "sudo"` matches
this; `"direct"` is for dev/CI where the agent is already root.
The canonical artifacts (BUNDLE slice):
- **`configs/felhom-agent.service`** — the canonical unit (`User=felhom-agent`,
`ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`). It deliberately sets
**no** `NoNewPrivileges` and **no** mount-namespacing hardening (`ProtectHome`/`PrivateTmp`/…): the
first would block the `sudo` the agent needs, the second would put the agent in a private mount
namespace so its `mount --bind` drive enrollments wouldn't propagate into the running guest. The
security boundary is the sudoers allowlist, not systemd sandboxing.
- **`configs/felhom-agent.sudoers`** → `/etc/sudoers.d/felhom-agent` (0440, `visudo -cf`-validated).
- **`scripts/publish-agent.sh`** publishes the binary to Gitea as a generic package
(`/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent`), printing the sha256 the operator
records in the hub artifact manifest.
**Install is automated.** The host-bootstrap script (`felhom.eu/scripts/felhom-host-install.sh`) fetches
the binary from Gitea, verifies its sha256 against the hub-vouched manifest, then installs the user +
binary + sudoers + unit + config — no manual agent install step.
## Test
+88 -33
View File
@@ -1,45 +1,100 @@
# REPORT — AGENT-001 anti-retarget wipe fix: merge + deploy (v0.30.0)
# REPORT — releasing publishes, and an unreleasable version fails CI (R-115, R-183)
**Date:** 2026-06-13 (supervised deploy session)
**Outcome:** SUCCESS — merged to `main`, built v0.30.0, deployed to the demo Proxmox host, verified non-destructively. No rollback.
**Date:** 2026-08-03 · **Repo:** `felhom-agent` · **NO VERSION BUMP** — the agent stays **v0.120.0**,
no Go code changed, nothing was built or deployed.
## What shipped
## What changed
`v0.30.0` — AGENT-001 fix (from the 2026-06-13 deep-sweep audit). The inline customer-confirmed wipe in
`internal/localapi/disks.go` `handleDiskFormat` previously ran `mkfs` on the caller-supplied mutable
`/dev` path (`req.Device`); a USB re-enumeration between inspection and `mkfs` could wipe the wrong
disk (classify→mkfs TOCTOU). Now it re-resolves the confirmed durable id → current device, requires a
re-derived exact match, re-inspects data-bearing, and formats the **re-resolved** device — at parity
with `signedjobs.WipeExecutor`. New `internal/localapi/wipe_reresolve.go` (`antiRetargetResolve`,
injectable `reresolveWipe` seam); refusals → `409`, no `mkfs`. Merged from
`fix/agent-001-wipe-durable-reresolve`.
| File | |
|---|---|
| `scripts/release-agent.sh` | **new** — THE release path: build → tag → publish → verify by independent download |
| `scripts/check-published-versions.py` | **new** — the R-115 gate |
| `scripts/agent_gates.py` | registers the gate as **not `--fast`** (it needs network) |
| `.gitea/workflows/gates.yml` | CI now runs the **full** gate set, not `--fast` |
| `CLAUDE.md` | the raw `go build` line is replaced by the release script; a **Vouch** row replaces the old Publish row |
## Deploy
## Why
- Build server (192.168.0.180, go1.26.0): green gate on the branch (`build`/`vet`/`test` clean; `TestAntiRetargetResolve` incl. `retarget-mismatch-refused` PASS). Built `v0.30.0` (ldflags `-X main.version=0.30.0`).
- Demo Proxmox host (`felhom-pve`): binary verified by sha256 at each hop; backed up the running binary to `/usr/local/bin/felhom-agent.bak-0.29.1`; installed the new binary; `systemctl restart felhom-agent.service`.
- **Rollback target was v0.29.1** (not needed).
- Pin check before the live run: served PVE leaf-cert SHA-256 prefix matches the pinned value (`BA:7C:99:7D:45:D0…`).
- Scope: **agent-only**. No golden rebake, no re-provision, no controller change. Guest 9201 controller stayed online through a brief local-API blip.
Publishing was a step someone had to remember and was **forgotten three times in five days**
R-111's seventeen stranded releases, 0.114.0, and 0.120.0, which sat deployed on both demo hosts and
undownloadable, so a documented-path reinstall would have silently downgraded them to the pre-merge
agent **while reporting success**. R-111's own closing line named this leg and closed SHIPPED without
it; it recurred the same afternoon. A note is not a mechanism.
## Verification (non-destructive)
The script also **tags**, because `felhom-host-install.sh` now fetches the agent's sixteen config
files from `raw/tag/v<version>/` (R-183). A released version with no tag 404s a box mid-install, as
root, on a virgin machine. Tag and package are two halves of one release.
It **verifies by downloading what it just published** and comparing the sha to what it built. The
publish step's own success is a report on its own write; a fetch returning the right bytes is a
different claim, and it is the one that matters.
It **does not vouch** — that points machines at a version and stays the operator's act.
## The gate's invariant — not the one specified, and the reason was measured
The task's §8.4 asked for *"the version the hub tells machines to install must be downloadable"*.
**CI cannot see that**, measured rather than assumed (P-C):
| Endpoint | Anonymous |
|---|---|
| Gitea package **download** | **200** (and **404** for a fake version — it discriminates) |
| Gitea **tags** api | **200** |
| Gitea package **listing** api | **401** — token required |
| Hub `/api/v1/artifacts/<customer>` | **401** — per-customer passphrase required |
So a credential-free gate can ask *"is this version installable"* but not *"which version is
vouched"*. Adding an operator credential to CI to close that is the operator's call, not a gate
author's. The implemented invariant — **every `v<semver>` tag must have a downloadable package and a
tag tree that serves the agent's configs** — needs no credential and **catches all three recorded
instances**, because the release script creates the tag and publishes in one act.
**What it does not catch, stated rather than assumed away:** the hub vouching a version that was
never released at all. Nothing here can see that; it belongs at vouch time in the hub. → **R-184**.
## Proof
| Check | Result |
|---|---|
| Service active + version | `felhom-agent.service` active/running; `--version` = **0.30.0**; clean startup logs (local-api on :8443, hub desired gen=2, PBS verify ok) |
| Local API reachable (full chain) | controller→agent `/disks` proxy returns the disk list (vmid 9201); guest 9201 controller healthy on the hub |
| Regression A — data-bearing device, NO confirmation | **REFUSED** (HTTP 409, `needs_confirmation:true`, `formatted:false`); no `mkfs` |
| Regression B — confirmed + WRONG durable_id | **REFUSED** (HTTP 409, `formatted:false`); a non-matching confirmation did not authorize a wipe; no `mkfs` |
| Data safety | `felhom-usb` (`/dev/sdb1`, RomM data) still mounted, `used_bytes` unchanged — untouched |
| Blank-device benign-format | SKIPPED — no disposable blank device available on the host |
| Happy-path destructive wipe | SKIPPED — HUMAN-only (never wipe real data to verify); covered by the unit test |
| `go build ./... && go vet ./...` | OK |
| `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
| `agent_gates.py --fast` | `published` correctly **SKIPPED** — the pre-push hook must not fail because Gitea blinked |
| `agent_gates.py` (full) | `reuse-refs` OK, `published` OK |
| release script: re-release guard | `ERROR: tag v0.120.0 already exists — releasing over it would make one version name two binaries`, rc=1 |
| release script: clean-tree guard | `ERROR: working tree is dirty — commit and push first`, rc=1 |
Note: `reresolveWipe`'s post-gate refusal runs only when the gate ALLOWS (a correctly-matching
confirmation), which on a real device leads to the destructive success path — so it is not safely
live-triggerable and is verified by the unit test `retarget-mismatch-refused`. Live, both the
no-confirmation (A) and wrong-confirmation (B) gate refusals confirm no un-gated destructive path.
### Red-proof F — both directions
## Branch / commits
- **A tagged-but-unpublished version** (`v9.9.9` created for the purpose): gate **rc=1**,
`binary NOT downloadable (HTTP 404 …)`. This is the R-115 shape exactly.
- **The gate deregistered from the entry point**, same bad state: `agent_gates.py` → **rc=0, "all
agent gates OK"**. Restored → **rc=1, CONVICTED: published**. The guard is what catches it, not
something else.
- `main` merge + version bump pushed. Branch `fix/agent-001-wipe-durable-reresolve` left in place (per the runbook) until the operator confirms.
- felhom.eu audit record (`documentation/audits/deep-sweep-2026-06-13.md`) updated: AGENT-001 → MERGED + DEPLOYED (agent v0.30.0).
### Scenario F measured on REAL CI, not inferred
Runs **69** and **70** are on the **same commit** `0db7766`:
| run | state of the repo | CI |
|---|---|---|
| 69 | no `v9.9.9` | **success** |
| 70 | `v9.9.9` tagged, not published | **failure** |
Same code, same workflow, one variable — so the gate demonstrably RUNS in CI and fails for exactly
the R-115 condition. This also retrospectively explains runs 67/68, which were red in the window when
`v9.9.9` first existed. **One deliberate CI failure e-mail reached the operator — that was this
proof, not an incident.**
I could not read CI's own step log to attribute those runs directly: the Gitea jobs endpoint requires
an API token, and the only credential available on this host (`~/.docker/config.json`) is a registry
password, which the API rejects. The controlled before/after above replaced that log rather than an
assumption standing in for it.
`v9.9.9` was deleted afterwards; `git ls-remote --tags` shows only `v0.120.0`.
## Tag convention
`v<semver>`, at the commit the binary was built from. `v0.120.0` was created retroactively at
`cd6e267` — the commit that produced the published binary (sha `a7763d31b55b5ce7…`). `configs/` is
byte-identical between that commit and `main`, so nothing about the sixteen fetched files depends on
the choice; `cd6e267` is tagged because it is the honest one.
+199
View File
@@ -0,0 +1,199 @@
# REUSE.md — felhom-agent
> Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid.
> Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper.
> Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.
## 1. Canonical helpers (MUST reuse — do not reinvent)
### Allowlisted exec / privileged surface (sudoers)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `Runner` / `ExecRunner.Run`, `RunStdin` | internal/proxmox/privileged.go | `Run(ctx, name, args...) (stdout, stderr []byte, err)` | ALL host command exec (direct or `sudo -n` prefix) | Arg vectors only, never a shell string; `capBuf` caps output at 1 MiB |
| `Privileged` (CreateGoldenLXC/MountUSBByUUID/SMART/Sensors) | internal/proxmox/privileged.go | methods on `*Privileged` | the 3 fenced root-CLI exceptions ONLY | Do NOT add methods — fence is structural (`routing_test.go` asserts it) |
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
| `stageTemp` | internal/localapi/intermediary.go | `stageTemp(pattern, content) (path, err)` | random-named temp before a root `install` (audit B1) | Fixed /tmp names are a TOCTOU — sudoers globs expect `/tmp/felhom-*-*.ext` |
| `guesthook.InstallSnippet` / `Register` | internal/guesthook/install.go | `InstallSnippet(ctx, runner) error` | pre-start self-heal hook install (C1 net) | Same random-temp+install pattern; snippet delegates to the agent binary (no shell logic). Issues `mkdir -p /var/lib/vz/snippets` FIRST (v0.63.0, B2 — fresh boxes lack the dir; sudoers grants exactly that argv) |
### Disk / format safety (role gates, durable IDs, format guards)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `SudoHostOps.Format` | internal/storage/hostops.go | `Format(ctx, device, fstype) error` | THE only mkfs path | Guards, in order: `ValidateBlockDevice` + `ValidateFSType` → mandatory `deviceUnclaimed` (claim.go) → exec `felhom-mkfs-guarded` (sudoers allowlists ONLY the wrapper, not raw mkfs) |
| `SudoHostOps.InspectDevice` + `DeviceProbe.DataBearing` | internal/storage/hostops.go | `InspectDevice(ctx, device) (DeviceProbe, error)` | data-bearing verdict from the AGENT's own read | Fail-safe: `Probed=false` ⇒ DataBearing=true; blkid output is evidence, lsblk is read-success authority |
| `classifyClaim` / `SudoHostOps.deviceUnclaimed` | internal/storage/claim.go | `classifyClaim(claimFacts) (unclaimed bool, reason string)` | "is this disk provably free to format" | Pure function of `gatherClaimFacts`; ANY read error/ambiguity/empty-lsblk ⇒ CLAIMED (audit D2); Felhom's own `/mnt/felhom-drives` mounts are not a foreign claim |
| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
| `storage.ResolveStorageDevice` | internal/storage/durable_device.go | `ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)` | re-resolve enrolled STORAGE drives (remount) | `uuid:` scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node |
| `deriveDurableID` | internal/storage/durableid.go | `deriveDurableID(typ, s, backingDevice, uuid) string` | storage-target durable id (DR re-attach key) | Deterministic per type; `uuid:` for usb/local-dir; PBS id includes `#<fingerprint>` |
| `SystemDisks` / `isSystemBacked` / `RoleForStorage` / `RoleForRawDevice` | internal/storage/role.go | `RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole` | protection-tier classification | Fails safe to `system` (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied |
| `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process |
| `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` |
### Mount lifecycle (host + guest binds)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `SudoHostOps.EnsureMount` | internal/storage/hostops.go | `EnsureMount(ctx, MountSpec) error` | persistent by-UUID systemd .mount | Validate→render→stage→`install``enable --now`; idempotent |
| `SudoHostOps.Unmount` | internal/storage/hostops.go | `Unmount(ctx, where) error` | detach a mount unit | DESTRUCTIVE — caller MUST have gated it; does not self-authorize |
| `SudoHostOps.ReassertEnrolledMounts` | internal/storage/hostops.go | `ReassertEnrolledMounts(ctx)` | reboot remount (re-resolve by UUID) | Re-asserts unless mounted AND enabled (`shouldReassertMount`); skips absent UUIDs |
| `GuestBinder.AttachDrive` / `DetachDrive` | internal/localapi/intermediary.go | `AttachDrive(ctx, vmid, where) (guestPath, err)` | live drive hot-swap under `/mnt/felhom-drives` | Normalizes to EXACTLY ONE bind via `countHostMounts` (converges double-binds); force re-bind when guest can't see it |
| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
| `GuestBinder.GuestSeesMount` / `GuestBootID` | internal/localapi/intermediary.go | `GuestSeesMount(ctx, vmid, path) bool` | guest-visible (usable) signal; reboot detection | Host bind present ≠ guest sees it (non-recursive parent bind) |
| `SudoHostOps.EnsureNetworkMount` / `RemoveNetworkMount` / `ListNetworkMounts` | internal/storage/netmount.go | `EnsureNetworkMount(ctx, spec) error` | NAS automount pair | rm glob confined to `mnt-felhom*` units; NAS ≠ drive (no durable-id/SMART/wipe); RemoveNetworkMount doubles as the verify-fail rollback (idempotent) |
| `NetworkMountedAt` / `NetworkEndpointReachable` | internal/storage/netmount.go | `NetworkMountedAt(where) bool` | verify mount-truth + the 2 s add pre-probe | /proc/mounts is the ONLY mount-success judge (autofs trigger ≠ mounted; readability ≠ mounted — SPIKE-nas-verify §8) |
| `SudoHostOps.ReassertNetworkAutomounts` + `Server.ReassertNetworkMounts` + `guesthook.PostStartNetworkReassert` | internal/storage/netreassert.go, internal/localapi/netreassert.go, internal/guesthook/netreassert.go | `ReassertNetworkAutomounts(ctx) []NetReassertResult` | NAS guest-reboot heal (RCA fix 1): re-arm idle automount triggers (stop + enable --now) so the fresh mount event propagates into running guests | NEVER call from periodic health paths (an idle trigger is HEALTHY); active real mounts are never touched; hook leg runs as root (direct systemctl), daemon leg via sudo |
| `ClassifyNetVerifyFailure` | internal/storage/netverify.go | `ClassifyNetVerifyFailure(journalTail, tcpReachable) (code, hint)` | NAS verify failure categories | String-based BY DESIGN (every mount failure is rc=32); substrings verbatim from SPIKE-nas-verify Q4; `nfs_export` merges not-found/not-permitted (NFSv4 identical) |
### Durable stores (atomic state)
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `IntentStore` (`Get/SetEnrolled/SetEjected/SetDecommissioned/OnAbsent`) | internal/storage/intent.go | `OpenIntentStore(path)` | drive intent (4-state self-heal) | Keyed by durable-id only; `OnAbsent` is the ONLY ejected→enrolled path; refuses empty ids |
| `GuestBindStore` (`Record/Remove/Guests`) | internal/localapi/guestbindstore.go | `OpenGuestBindStore(path)` | per-guest enrolled binds (F9 re-assert) | Same tmp+rename 0600 pattern as IntentStore |
| `FormatJobStore` + `startFormatDetached` + `RecoverFormatJob` | internal/localapi/formatjob.go | `startFormatDetached(device, durableID, fstype, blank) <-chan error` | detached, restart-surviving mkfs (F20-BUG3) | Runs off `s.baseCtx` (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank |
| `TokenStore.Mint` / `Lookup` | internal/localapi/tokenstore.go | `Mint(vmid) (plaintext, error)` | per-guest local-API tokens | Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup RELOADS the file once on a miss (v0.63.0, B3): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence without a restart; append-only size check bounds the re-read |
| `FileNonceStore.SeenOrRecord` | internal/authz/noncestore.go | `SeenOrRecord(nonce, exp) bool` | durable anti-replay | fsync'd before returning false; prune only after exp |
| `Journal` (`Append/Latest/InFlight/AlreadyApplied`) | internal/reconcile/journal.go | `OpenJournal(path)` | op journal + idempotency + crash recovery | `Recover` consumes `InFlight()`; scratch entries special-cased |
### Local-API plumbing
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `Server.withGuest` | internal/localapi/server.go | `withGuest(fn(w, r, vmid)) http.HandlerFunc` | bearer auth + self-scoping for EVERY route | Token→vmid is the authority; explicit `?vmid=` only a consistency check (403 on mismatch) |
| `Server.scopedFromBody` | internal/localapi/server.go | `scopedFromBody(w, bodyVMID, tokenVMID, path) bool` | POST-body vmid self-scope check | Call right after decode; false = already 403'd |
| `decodeBody` / `writeOK` / `writeErr` / `writeStatus` | internal/localapi/server.go | `writeStatus(w, code, ok, data, errMsg)` | ALL local-API JSON I/O | Do not hand-roll response envelopes |
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC``Client.Pool`), fail-safe skip on pool-read failure |
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
### Proxmox client / hub / PBS / provisioning
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
| `Client.PoolAddVMID` | internal/proxmox/mutate.go | `PoolAddVMID(ctx, pool, vmid) error` | re-assert pool membership after a restore-over-existing (campaign-2 R2) | SYNC (no UPID, don't WaitTask); PVE `PUT /pools` is additive (merge, not replace) — `delete=1` removes; idempotent (already-member swallowed); needs `Pool.Allocate` at `/pool/<pool>`. `pct restore --pool` sets membership only at CREATE — a restore over an existing vmid drops it, so bring-up re-asserts post-restore |
| `TLSConfig.build` / `normalizeFingerprint` | internal/proxmox/tls.go | `build() (*tls.Config, error)` | PVE leaf-cert SHA-256 pinning | No insecure default |
| `pinnedTLS` | internal/pbs/pin.go | `pinnedTLS(fingerprint) (*tls.Config, error)` | PBS leaf pinning | Same model as PVE; 64-hex fingerprint normalized |
| `hub.Client.Report` | internal/hub/client.go | `Report(ctx, *HostReport) (*ControlEnvelope, error)` | the heartbeat | Typed `TransportError`/`HTTPError`, never contain the bearer token |
| `hub.Loop` + `MultiObserver` | internal/hub/loop.go | `NewLoop(...)`; `MultiObserver(obs...)` | resilient report loop + envelope fan-out | Errors logged, loop continues; interval clamped 603600 s |
| `provision.BackHalf.Provision` | internal/provision/backhalf.go | `Provision(ctx, Input) (Result, error)` | guest bootstrap back-half | mint→render→0600 write→chown 100000:100000→`pct set` ro bind→onboot; token NEVER logged/returned. Bootstrap `local_api.endpoint` = the caller's `cfg.LocalAPI.ListenAddr` (main.go) — moving the agent bind to the island moves the guest dial for free (R-50, no template) |
| `buildBringUpConfig` island NIC | internal/reconcile/bringup.go | (pure) `BringUpSpec{IslandBridge,IslandGuestAddr}``params["net1"]` | R-50 island control plane | When BOTH island fields are set (from `cfg.LocalAPI`), attaches a static `net1=name=eth1,bridge=<vmbr9>,ip=<.2/30>` (no hwaddr → fresh MAC), so the controller reaches the agent over a fixed private address immune to LAN/DHCP/site moves. Empty = pre-R-50, no net1. All-or-nothing + CIDR enforced in `LocalAPIConfig.Validate`. The guestnet healer is eth0-only (`parseMode` is dev-scoped) so it never touches the static island NIC |
| `reconcile.Queue.Submit` | internal/reconcile/queue.go | `Submit(vmid, fn) <-chan error` | per-guest serialization of ALL mutations | Same vmid strictly FIFO; lanes parallel across guests |
| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
| `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew |
| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v1) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--``-` (spike §2.2) |
| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
| `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) |
| `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (PlannedPending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker |
| `wgtunnel.Manager.TunnelConvergence` / `reconcile.Engine.LastResult` | internal/wgtunnel/manager.go, internal/reconcile/engine.go | `TunnelConvergence() (desired, operational bool)`; `LastResult() (Result, bool)` | cached convergence snapshots for the fast-tick | TunnelConvergence is refreshed at the END of every `Apply` (its own cadence) so the fast-tick never execs `wg`/`systemctl`; LastResult is mutex-recorded per `reconcileOnce`, ok=false until the first pass |
## 2. Canonical patterns (copy structure from THE named file)
| Pattern | Canonical file | Key traits |
|---|---|---|
| Validate-then-exec privileged op | internal/storage/hostops.go (`EnsureMount`) | validate EVERY arg (validate.go) → render → stage in agent dir → root `install``systemctl`; refuse before any command exists |
| Fail-safe pure classifier over gathered facts | internal/storage/claim.go (`classifyClaim` over `claimFacts`) | pure function ⇒ fixture-testable; any error/ambiguity refuses; gather separated from verdict |
| Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only |
| Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set |
| Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction |
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
| `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) |
| `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool**`BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs |
| `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice`**re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted`**quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s |
| Detached IN-MEMORY verify job (single slot, deliberately unpersisted) | internal/localapi/netverifyjob.go | claim slot sync (single-flight 409) → detached pipeline off baseCtx → auto-rollback on fail; restart ⇒ slot empty ⇒ the CALLER rolls back (Scenario F) — contrast formatjob (persisted+recovered) |
| Optional dependency degradation | internal/localapi/server.go (`Options`) | nil dep ⇒ endpoint answers "not configured" (503), never a crash |
| Version channel (v0.82.0) | internal/localapi/server.go (`Options.AgentVersion`; `Handler()` mux wrap) | sets `X-Felhom-Agent-Version` on EVERY response (all routes/statuses, incl. auth-fail/404) — the controller's capability-comparison source; empty version ⇒ header omitted |
| Root-file install via random temp | internal/localapi/intermediary.go (`installSharedParentUnit`) | `stageTemp` (os.CreateTemp) → sudoers-globbed `install -m` → pinned destination |
| Detached destructive job + restart recovery | internal/localapi/formatjob.go | persist `running` → run off baseCtx → record outcome; recovery re-resolves durable id, never a path |
| Signed-op verify pipeline | internal/authz/verifier.go (`Verify`) | armor→namespace→key-material allowlist→crypto over RAW bytes→blob→target→window→nonce LAST |
| Resilient daemon loop | internal/hub/loop.go (`Loop.Run`) | ctx-cancel returns nil; errors logged and loop continues |
| Guarded-below-the-agent wrapper | configs/felhom-mkfs-guarded.sh | root re-checks catastrophic cases (system disk/LVM PV/foreign mount/RO/member FS) even against an agent bug |
## 3. Dangerous lookalikes — do NOT reuse
| Trap | Why it bites | Use instead |
|---|---|---|
| Acting on the caller's `req.Device` (or any remembered /dev path) after inspection | classify→mkfs TOCTOU (AGENT-001): /dev re-enumeration retargets the node to a different physical disk | `Server.reresolveWipe`/`reresolveBlank` → format the RE-RESOLVED device |
| Exec'ing raw `mkfs.*` (incl. `Binaries.MkfsExt4/MkfsXfs`) | sudoers no longer allowlists raw mkfs; bypasses the claim filter + wrapper re-checks | `SudoHostOps.Format` (→ `felhom-mkfs-guarded`) |
| `DiskInfo.DurableID` (`uuid:`) as a wipe-confirmation id | wipe gate resolves `byid:`/`byuuid:` — a `uuid:` id is a `binding_mismatch` (F20-BUG2) | `DiskInfo.WipeDurableID` / `storage.DeviceDurableID` |
| `ResolveDurableDevice` for enrolled-storage remounts (or vice versa) | schemes differ: wipe = `byid:`/`byuuid:`, storage = `uuid:` — each refuses the other | `ResolveStorageDevice` for mounts; `ResolveDurableDevice` for wipes |
| `authz.MemoryNonceStore` on a real host | replay protection dies on restart | `authz.FileNonceStore` |
| Adding methods to `proxmox.Privileged` | breaks the 3-exception root-CLI fence (`routing_test.go`) | `proxmox.Runner` + a new sudoers Cmnd_Alias + validate.go-style checks |
| Calling `Client.DestroyLXC`/`Vzdump`/`SetConfig` outside the gate/queue/journal | skips classification, signature, per-guest serialization, crash recovery | `reconcile.Engine` paths / `RunSignedJob`; queue via `Queue.Submit` |
| `GuestBinder.AttachBind`/`DetachBind` (per-drive `pct set -mpN`) | legacy model; a missing bind source can brick guest boot (C1) | `AttachDrive`/`DetachDrive` (intermediary model) |
| `isHostMountpoint` to reconcile bind state | boolean can't converge stacked double-binds (the /mnt doubling bug) | `countHostMounts` normalization inside `AttachDrive` |
| Acting on a raw `ListLXC` list as if it were "guests the agent owns" | audit A1 (pre-v0.62.0 the stale-lock reaper did exactly this — contained only by the pool-scoped token) | ownership must be PROVEN: intersect with `Client.Pool` membership like `staleLockController.Guests()` (v0.62.0), fail-safe on read failure |
## 4. Seams & interfaces (testing + cross-repo)
| Interface | Defined in | Implemented by | Fakes/tests at |
|---|---|---|---|
| `proxmox.Runner` | internal/proxmox/privileged.go | `*ExecRunner` (direct/sudo) | `mockRunner` internal/proxmox/mock_test.go; runner fakes in storage tests |
| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,t)` / `ProvenArchive(target)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)`**`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
| `hub.AddressEnumerator` (v0.119.0) | internal/hub/hostaddr.go | **defaults to the REAL `systemInterfaces`** when `Collector.addrEnum` is nil — deliberately inverting the nil-reporter-means-off convention, because this stanza has no config gate and a forgotten wiring call would otherwise ship silently empty (the inert-seam shape, four instances on record) | internal/hub/hostaddr_test.go drives fixtures TRANSCRIBED from `ip -o addr show` on demo-felhom AND demo-hp, including the address-less veth/NIC rows — the "no denylist needed" claim rests on those rows really being empty, so omitting them would prove the claim by assuming it. `filterHostAddresses` keeps GLOBAL UNICAST only: one predicate that drops loopback, `fe80::/10`, and `169.254/16` — the last being the R-50 island literal, identical on every box and actively misleading if surfaced |
| `reconcile.OpVerifier` | internal/reconcile/gate.go | `*authz.Verifier` | fake verifier in internal/reconcile gate tests |
| `signedjobs.WipeOps` / `Executor` (`ExecutorChain`) | internal/signedjobs/wipe.go + runner.go | `*storage.SudoHostOps`; `WipeExecutor`+`DecommissionExecutor` | internal/signedjobs wipe/runner/decommission tests |
| `hub.reporter` / `collectorIface` / `EnvelopeObserver` | internal/hub/loop.go | `*hub.Client`, `*hub.Collector`; `desired.Syncer` + `signedjobs.Runner` | `fakeReporter`/`fakeCollector` internal/hub/loop_test.go; `recordingReporter` loop_logtail_test.go |
| `applog.Ring` (always-DEBUG capture ring) + fan-out `applog.New → (logger, ring)` | internal/log/log.go | wired in cmd/felhom-agent/main.go → `localapi.Options.LogRing` + `Loop.SetLogTailSource(ring.Lines)` | internal/log/log_test.go; localapi/debuglogs_test.go — v0.83.0; the byte-capped `Lines` is the heartbeat tail source |
| `pbsdr.StorageReader` / `SecretConsumer` / `Manager.probeFP` (func seam) | internal/pbsdr/manager.go | `*proxmox.Client`; `*hub.Client`; `pbs.ProbeFingerprint` | `fakeStorage`/`fakeConsumer`/`fakeRunner` internal/pbsdr/manager_test.go (argv+stdin recorder) |
| `capability.Runner` | internal/capability/probe.go | `*proxmox.ExecRunner` (RunnerDirect) | `fakeRunner` internal/capability/probe_test.go |
| Cross-repo: local API ↔ controller | internal/localapi/server.go routes; contract seeded by internal/provision/doc.go (`bootstrap.json`: endpoint + leaf fingerprint + token) | felhom-controller's agentapi client | pin = served leaf cert (memory gotcha) |
| Cross-repo: agent ↔ hub | internal/hub/report.go (`HostReport`), `ControlEnvelope`; POST `/api/v1/host-report` | hub mirrors structs field-for-field | new event/report fields need hub-side ingest changes |
| Cross-repo: shipped host artifacts | configs/felhom-agent.sudoers, configs/felhom-mkfs-guarded.sh, configs/felhom-pbs-apply, shared-parent script/unit (inline in internal/localapi/intermediary.go) | deployed WITH the binary | sudoers globs must match `stageTemp` patterns + staging dirs exactly |
| Operator signing | internal/authz (OpBlob, SSHSIG) | cmd/felhom-opsign (offline CLI) | blob/verify tests in internal/authz |
## 5. Extension points (where new features plug in)
- **Local-API route**: add to `Server.Handler` (internal/localapi/server.go) wrapped in `s.withGuest`; new deps go into `Options` as OPTIONAL fields degrading to "not configured".
- **New signed-job verb**: implement `signedjobs.Executor` (return `ErrNoExecutor` for foreign ops) and append to the `signedjobs.ExecutorChain` in cmd/felhom-agent/main.go; add the op class + role scoping in internal/reconcile (classify.go, gate.go `roleAuthorizes`).
- **New privileged host op**: validate args (internal/storage/validate.go style) → exec via `Runner` → add a `Cmnd_Alias` to configs/felhom-agent.sudoers → add a probe vector to internal/capability/manifest.go (so degradation is visible) → ship sudoers with the binary.
- **New reconcile action**: `ActionKind` + `classOfAction` (internal/reconcile/classify.go), plan emission in internal/reconcile/plan.go; destructive ⇒ gate handles it automatically.
- **Hub-report field**: extend `hub.HostReport` (internal/hub/report.go) + `Collector` — hub side must mirror + allowlist it (cross-repo).
- **DR-recipe section (host-half)**: add the field to `DRRecipeHostHalf` (internal/hub/dr_recipe.go) **AND** to
the hub's `hostHalfShape` + `AssembledRecipe` (felhom.eu `hub/internal/store/dr_recipe.go`). Those two
hub structs are **ALLOW-LISTS**: a section only the agent knows about is stored intact and silently
dropped before any operator sees it — that is R-122, which cost `offsite_restic` its entire existence.
Then update BOTH copies of `testdata/host-report.golden.json` (byte-identical, cross-repo) and extend
`TestAssembleDRRecipe_CarriesEveryEmittedSection`. **A recipe field that cannot be resolved records an
explicit unknown with a reason — never a default, an empty string, or a placeholder** (`DRState*` /
`DRReason*`); a recipe read during a rebuild must not present a guess as a fact.
- **Envelope-driven behavior**: implement `hub.EnvelopeObserver`, add to the `MultiObserver` in cmd/felhom-agent/main.go.
- **Selftest mode**: `selftestFlag` + `runSelftest*` in cmd/felhom-agent/main.go.
- **Config**: internal/config/config.go (`Load` + `applyEnv` `FELHOM_AGENT_*` overlay; keep secrets out of `Redacted()` output).
## 6. Known duplication (observed — NOT fixed)
- Two lsblk `-J` parsers with near-identical structs: `parseLsblkDevice`/`lsblkDevice` (internal/storage/hostops.go) vs `parseLsblkNodes`/`lsblkDev` (internal/storage/claim.go).
- Two smartctl `-a -j` paths: `SudoHostOps.SMART` (internal/storage/hostops.go, parsed `hub.SmartSummary`) vs `Privileged.SMART` (internal/proxmox/privileged.go, raw map).
- **SMART device resolution (v0.95.0):** `smartDeviceFor` (internal/storage/observe.go) resolves partition→disk AND dm/LVM→disk (`dmWholeDisk` in internal/storage/smartdev.go, via `/sys/block/<dm>/slaves`, `sysBlockRoot` test seam). `storage.SmartReader.SMARTForBacking` is the shared read the localapi `/disks` union path uses (Fix B) — do NOT re-implement smartctl parsing. The builtin-`local` SMART device comes from `containingMountDevice` (SMART-only; never feeds backing/durable_id).
- Atomic tmp+rename JSON store implemented 3×: `IntentStore.saveLocked` (internal/storage/intent.go), `FormatJobStore.save` (internal/localapi/formatjob.go), `GuestBindStore.saveLocked` (internal/localapi/guestbindstore.go) — comments say "mirrors", no shared helper.
- `run(ctx, name, args...) error` stderr-wrapping helper duplicated 4×: `SudoHostOps.run`, `Privileged.run`, `BackHalf.run` (internal/provision/backhalf.go), `GuestBinder.run` (internal/localapi/guestbind.go).
- Several independent /proc mount-table readers: `SudoHostOps.mountedSet` (internal/storage/hostops.go), `ProcHostReader.Mounts` (internal/storage/hostread.go). **In localapi they were unified in v0.117.0**: `isHostMountpoint` and `countHostMounts` are now one-liners over `hostMountEntries`, the single parser that also yields devno/fstype/super-options for `bindLiveness`.
- Deliberate mirror: `antiRetargetResolveExpect` (internal/localapi/wipe_reresolve.go) duplicates `WipeExecutor.Execute` steps 13 (internal/signedjobs/wipe.go) across packages.
- `stableParentDir` literal duplicated in internal/provision/backhalf.go to avoid a provision→localapi import edge (commented as intentional); `trim` (internal/storage/hostops.go) vs `trimBody` (internal/proxmox/errors.go) output-truncation twins.
+59 -26
View File
@@ -1,23 +1,38 @@
# RUNBOOK — provisioning the OS / Docker-data storage split
# RUNBOOK — provisioning the OS / Docker-data / user-data storage split
How the controller guest's **OS rootfs** and **Docker data** are carved onto separate `local-lvm`
volumes, and how to size them per node/customer. Grounded by the storage-split spike
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 256 GB
Docker-data**, rest of the thin pool free for other guests.
How the controller guest's **OS rootfs**, **Docker data**, and **SSD user-data** are carved onto three
separate `local-lvm` volumes, and how to size them per node/customer. Grounded by the storage-split spike
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data
+ 50 GB user-data**, rest of the thin pool free for other guests.
## The design (locked)
Three thin volumes are baked into the golden and grown at provision:
| mp slot | Mount | Holds | Golden default | Provision grow knob |
|---------|-------|-------|----------------|---------------------|
| rootfs | `/` | OS only | `OS_SIZE_GB=32` | `-rootfs-grow` |
| `mp0` | `/var/lib/docker` | ALL Docker images + overlay + named volumes | `GOLDEN_DOCKER_GB=16` | `-datavol-grow` |
| `mp1` | `/mnt/sys_drive` | SSD user-data (the controller's `system_data_path`; `<sys_drive>/felhom-data` = SSD-only app data + DBs) | `GOLDEN_SYSDATA_GB=8` | `-sysdata-grow` |
- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
if the Docker volume fills. Both volumes ride into PBS regardless (so the backup size is unchanged) —
the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is
unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
(controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
baked log rotation), not by placement.
- **`backup=1` on the Docker-data volume is non-negotiable.** Extra LXC mountpoints default to
`backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs fall out of the
snapshot AND the volume (with baked images) is excluded from the golden archive.
- **`/mnt/sys_drive` is a static CT volume, NOT an enrolled drive.** It never enrolls, ejects, or
decommissions — it is part of the golden's baked layout. The controller auto-discovers
`<sys_drive>/felhom-data` and, via `system.IsMountPoint`, **warns when `/mnt/sys_drive` is not its own
mount** (`selftest.go`). Making `mp1` a real mount clears that warning with **zero controller change**;
the controller is untouched by this slice. The `mp` mounted under the guest's `/mnt` propagates into
the controller container through the existing `-v /mnt:/mnt:rslave` bind (the same path enrolled drives
use).
- **`backup=1` on the Docker-data AND user-data volumes is non-negotiable.** Extra LXC mountpoints
default to `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app
data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.
## Golden build (per controller-image baseline)
@@ -25,15 +40,20 @@ Docker-data**, rest of the thin pool free for other guests.
```
# defaults shown; override via env
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
```
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`.
- Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it.
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}`
+ `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
+ `--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1`.
- Keep `GOLDEN_DOCKER_GB` / `GOLDEN_SYSDATA_GB` small (the golden ships them near-empty) — provision
grows both. The controller creates `<sys_drive>/felhom-data` itself once `/mnt/sys_drive` is a real
mount; the golden does NOT pre-create it.
- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
is a separate mount + the driver is overlay2, and **aborts if vzdump excludes mp0** (the B3 trap).
AND `/mnt/sys_drive` are separate mounts + the driver is overlay2, and **aborts if vzdump excludes mp0
or mp1** (the B3 trap).
- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
@@ -43,33 +63,46 @@ OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
## Provision (per customer guest)
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data volume + baked images),
then sizes per customer. The Docker-data volume is **grown** (not re-attached) so the baked images are
never shadowed:
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes +
baked images), then sizes per customer. Each volume is **grown** (not re-attached) so the baked images
are never shadowed:
```
felhom-agent --selftest=provision \
-archive <golden-volid> -vmid <VMID> \
-customer-id <id> -hub-password <retrieval-passphrase> \
-rootfs-grow <N> # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 240 → 256 total)
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
-datavol-mount mp0 # the golden's Docker-data slot (default mp0)
-sysdata-grow <N> # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
-sysdata-mount mp1 # the golden's user-data slot (default mp1)
```
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount}` — both grows are online
(`pct resize`, spike B4), each its own task. `GuestMount.Backup` (→ `,backup=1`) is set on any
data-bearing additive mount (`bringup.go` `buildConfigParams`). The bootstrap mount is `mp9`; USB/data
drive enroll picks the lowest free slot (`freeMountSlot`), so it never collides with the docker mp0.
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount}`
— all grows are online (`pct resize`, spike B4), each its own task. **Separateness, not size, clears the
controller warning** — `-sysdata-grow 0` still yields a separate, warning-free mount at the golden size;
the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on any data-bearing additive
mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`;
USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1.
**Auto-start on host reboot (`onboot:1`).** The provision back-half sets the customer guest to
**`onboot:1`** (`provision/backhalf.go`, v0.41.0) so the whole home-server (controller + apps) comes back
on its own after a host reboot/power-cut. The **golden template stays `onboot:0`** (`build-golden.sh`) —
a template must not auto-start. No `startup`/boot-order is used: the v0.75 mountpoint-gate covers the
drive-bind race at boot, so the controller won't write app data onto the rootfs while drives re-bind.
(Guests provisioned before v0.41.0 need a one-time `pct set <vmid> -onboot 1`.)
**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
### Default sizing guidance
- ~512 GB SSD: 32 GB rootfs + 256 GB Docker-data (`-rootfs-grow 0 -datavol-grow 240`), leaving the rest
of the thin pool free for other guests.
- Other SSDs: size the data volume to "most of the free thin pool minus a reserve for other guests."
`local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's written.
- ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data
(`-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`), leaving the rest of the thin pool free for
other guests.
- Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve
for other guests." `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's
written.
## Fresh PVE host install — thin-pool carving (host-level, do this at install)
+23
View File
@@ -0,0 +1,23 @@
package main
import "testing"
// R-82 live regression (2026-07-26): the restore-test derived its tier from the CONFIGURED default
// target instead of the archive's own storage. Restoring a `felhom-pbs:` archive on a box whose
// primary target is "local" was classified "local" → the 10-minute local wait instead of the
// generous PBS one → the wait expired mid-restore at 600s against a 14.46 GB WAN restore, teardown
// fired at a still-restoring guest, and the scratch leaked.
func TestArchiveStorageID(t *testing.T) {
cases := []struct{ in, want string }{
{"felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z", "felhom-pbs"},
{"local:backup/vzdump-lxc-9201-2026_07_26-09_03_19.tar.zst", "local"},
{"", ""},
{"no-prefix", ""},
{":leading-colon", ""}, // i>0 guard: a leading colon is not a storage id
}
for _, c := range cases {
if got := archiveStorageID(c.in); got != c.want {
t.Fatalf("archiveStorageID(%q) = %q, want %q", c.in, got, c.want)
}
}
}
+10 -2
View File
@@ -11,7 +11,7 @@ import (
// struct (felhom-hub api.escrowUploadRequest). Cross-repo, no shared module — this is the agent
// half of the contract guard; the hub has the mirror in its own test.
func TestEscrowUploadContract(t *testing.T) {
b, _ := json.Marshal(escrowUploadRequest{BlobB64: "x", KeyFingerprint: "y", Posture: "z", CreatedAt: "t"})
b, _ := json.Marshal(escrowUploadRequest{BlobB64: "x", KeyFingerprint: "y", Posture: "z", CreatedAt: "t", ResticPwSHA256: "h"})
var m map[string]any
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
@@ -21,8 +21,16 @@ func TestEscrowUploadContract(t *testing.T) {
got = append(got, k)
}
sort.Strings(got)
want := []string{"blob_b64", "created_at", "key_fingerprint", "posture"}
want := []string{"blob_b64", "created_at", "key_fingerprint", "posture", "restic_pw_sha256"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("escrow wire contract drift: got %v want %v (must match the hub ingest struct)", got, want)
}
// SLICE 3: no staged password folded in → the hash field is OMITTED on the wire (the hub stores NULL →
// the controller never matches → stays pending; correct — the blob doesn't cover the key).
b2, _ := json.Marshal(escrowUploadRequest{BlobB64: "x", KeyFingerprint: "y", Posture: "z", CreatedAt: "t"})
var m2 map[string]any
_ = json.Unmarshal(b2, &m2)
if _, present := m2["restic_pw_sha256"]; present {
t.Fatal("restic_pw_sha256 must be omitted when no staged password was sealed")
}
}
@@ -0,0 +1,60 @@
package main
import (
"context"
"testing"
"time"
)
// F10/rc255 (CAMPAIGN-3): a guest-hook phase body that PANICS must never crash the process — the hook
// must return cleanly so the guest start proceeds (a nonzero exit blocks the start). runHookPhase
// recovers the panic and returns.
func TestRunHookPhase_PanicRecovered(t *testing.T) {
done := make(chan struct{})
go func() {
defer close(done)
runHookPhase("9201", "pre-start", 5*time.Second, func(context.Context) {
panic("simulated heal panic (e.g. a future Heal bug)")
})
}()
select {
case <-done:
// returned cleanly — the guest start would proceed
case <-time.After(3 * time.Second):
t.Fatal("runHookPhase did not return after a panicking body (would have crashed the hook)")
}
}
// A phase body that overruns the timeout must be abandoned — the hook returns rather than hanging the
// PVE start task. (The body's context is cancelled; the hook does not wait for the body to notice.)
func TestRunHookPhase_TimeoutReturns(t *testing.T) {
bodyCtxCancelled := make(chan struct{}, 1)
done := make(chan struct{})
go func() {
defer close(done)
runHookPhase("9201", "post-start", 20*time.Millisecond, func(ctx context.Context) {
<-ctx.Done() // simulate a body that respects cancellation eventually
bodyCtxCancelled <- struct{}{}
})
}()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("runHookPhase did not return after the timeout (would hang the guest start)")
}
select {
case <-bodyCtxCancelled:
// the body's context was cancelled at the deadline — the intended signal
case <-time.After(time.Second):
t.Fatal("the phase body's context was not cancelled at the timeout")
}
}
// A body that errors (returns normally, no panic) is fine — the hook returns cleanly.
func TestRunHookPhase_NormalBodyReturns(t *testing.T) {
ran := false
runHookPhase("9201", "pre-start", time.Second, func(context.Context) { ran = true })
if !ran {
t.Fatal("the phase body must run")
}
}
+33
View File
@@ -0,0 +1,33 @@
package main
import (
"context"
"testing"
)
// The hook WIRING red-proof target: `guest-hook <vmid> post-start` must invoke the network
// reassert with the vmid; pre-start and unknown phases must NOT. (Companion red-proof: remove the
// PhasePostStart case from runGuestHook → the invoked assertion fails.)
func TestRunGuestHook_PostStartInvokesNetworkReassert(t *testing.T) {
orig := postStartNetworkReassertFn
t.Cleanup(func() { postStartNetworkReassertFn = orig })
var gotVMIDs []string
postStartNetworkReassertFn = func(_ context.Context, vmid string) {
gotVMIDs = append(gotVMIDs, vmid)
}
runGuestHook([]string{"9201", "post-start"})
if len(gotVMIDs) != 1 || gotVMIDs[0] != "9201" {
t.Fatalf("post-start must invoke the network reassert with vmid 9201, got %v", gotVMIDs)
}
// pre-start must not touch the network reassert (it is the placeholder-heal phase; the heal
// no-ops on a nonexistent config path and never blocks).
runGuestHook([]string{"9201", "pre-start"})
// unknown phases are ignored entirely.
runGuestHook([]string{"9201", "pre-stop"})
if len(gotVMIDs) != 1 {
t.Fatalf("only post-start may invoke the network reassert, got %v", gotVMIDs)
}
}
+100
View File
@@ -0,0 +1,100 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-54 §9 rule 6 — the seam-discipline test, and the one this repo has the most reason to write:
// v0.91.0 shipped the PBS auth-probe seam with `main.go` never calling `SetAuthSink`, every unit
// test green because they all injected the seam directly. The guestnet watchdog has the identical
// shape (a component + a reporter seam + a goroutine), so its wiring is asserted here rather than
// trusted.
//
// This walks the AST rather than grepping the source: a commented-out call still satisfies a
// substring match (found while red-proofing the controller's twin of this test), and a comment is
// not a caller.
func TestMainWiresGuestNetWatchdog(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var constructed, reporterWired, started bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
switch fn := node.Fun.(type) {
case *ast.SelectorExpr:
switch fn.Sel.Name {
case "New":
// guestnet.New(...)
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "guestnet" {
constructed = true
}
case "SetGuestNetReporter":
reporterWired = true
}
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Watch" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "gnWatchdog" {
started = true
}
}
}
return true
})
if !constructed {
t.Error("main.go never calls guestnet.New — the watchdog does not exist at runtime")
}
if !reporterWired {
t.Error("main.go never calls collector.SetGuestNetReporter — the guest_net stanza would " +
"never reach the hub (the exact v0.91.0 inert-seam defect)")
}
if !started {
t.Error("main.go never starts the watchdog with `go gnWatchdog.Watch(ctx)` — it would be " +
"constructed, reported on, and never probe anything")
}
}
// The watchdog must NOT join the errc fan-out: a guest-network watchdog that can terminate the
// agent turns a customer's DHCP problem into an operator-plane outage. If it is ever changed to
// `errc <- ...`, the drain bound at the bottom of main() also has to change — this catches the
// first half of that mistake.
func TestGuestNetWatchdogIsNotInTheErrcFanout(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
bad := false
ast.Inspect(f, func(n ast.Node) bool {
send, ok := n.(*ast.SendStmt)
if !ok {
return true
}
if ch, ok := send.Chan.(*ast.Ident); !ok || ch.Name != "errc" {
return true
}
call, ok := send.Value.(*ast.CallExpr)
if !ok {
return true
}
if sel, ok := call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Watch" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "gnWatchdog" {
bad = true
}
}
return true
})
if bad {
t.Fatal("the guestnet watchdog was added to the errc fan-out — a watchdog over customer " +
"guests must never be able to bring the agent down (and the drain bound in main() " +
"would now be off by one, hanging shutdown)")
}
}
+1317 -109
View File
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,113 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-86 Scenario I — the seam-discipline test for the due-check.
//
// A due-check is worth nothing if the daemon still wires the OLD picker: every unit test in
// internal/backup would stay green (they inject the seam directly), the scheduler would ask for the
// newest archive with no settle cutoff, and the per-archive rule would run against a candidate that
// changes every time a backup lands. That is the same shape as the v0.91.0 inert seam — built,
// tested, never called — and this repo has shipped it four times.
//
// It walks main.go's AST rather than grepping: a commented-out call still satisfies a substring
// match, and a comment is not a caller.
func TestMainWiresTheSettleAwareTierPicker(t *testing.T) {
f := parseMainForWiring(t)
var settlePicker, oldPicker, settleWired, evalInterval bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.SelectorExpr:
// runner.PickSettledRestoreCandidateOn passed as a value (not called).
switch node.Sel.Name {
case "PickSettledRestoreCandidateOn":
settlePicker = true
case "PickRestoreCandidateOn":
oldPicker = true
}
case *ast.KeyValueExpr:
key, ok := node.Key.(*ast.Ident)
if !ok {
return true
}
if key.Name == "Settle" {
settleWired = true
}
case *ast.CallExpr:
if sel, ok := node.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "RestoreTestEvalInterval" {
evalInterval = true
}
}
return true
})
if !settlePicker {
t.Error("main.go never passes runner.PickSettledRestoreCandidateOn as the scheduler's TierPick — " +
"the due-check would run without a settle cutoff, i.e. against an archive that may still be being written")
}
if oldPicker {
t.Error("main.go still wires the pre-R-86 PickRestoreCandidateOn as a tier picker — " +
"two pickers means the one under test is not the one running")
}
if !settleWired {
t.Error("main.go never sets SchedulerOptions.Settle — the settle lag would default to 0 in the daemon " +
"and every freshly-landed archive would be an immediate candidate")
}
if !evalInterval {
t.Error("main.go never calls cfg.Backup.RestoreTestEvalInterval() — the scheduler would be driven by " +
"the retired cadence knob")
}
}
// The two R-85 guarantees the due-check must not have quietly dropped: the spec is still built PER
// RUN, and the shared heavy-operation gate is still handed to the scheduler.
func TestMainStillWiresTheHeavyOperationGateAndPerRunSpec(t *testing.T) {
f := parseMainForWiring(t)
var inFlightWired, specIsAFunc bool
ast.Inspect(f, func(n ast.Node) bool {
kv, ok := n.(*ast.KeyValueExpr)
if !ok {
return true
}
key, ok := kv.Key.(*ast.Ident)
if !ok {
return true
}
switch key.Name {
case "InFlight":
inFlightWired = true
case "Spec":
// A FuncLit means it is evaluated per run; anything else is a frozen value.
if _, isFunc := kv.Value.(*ast.FuncLit); isFunc {
specIsAFunc = true
}
}
return true
})
if !inFlightWired {
t.Error("main.go no longer hands the scheduler the shared InFlight gate — a restore-test could pull a " +
"multi-GB archive over the same tunnel an offsite backup is pushing one over (Scenario F)")
}
if !specIsAFunc {
t.Error("SchedulerOptions.Spec is no longer a function literal — a frozen spec is the R-85 defect " +
"(the tier and its timeout evaluated once at daemon start, forever)")
}
}
func parseMainForWiring(t *testing.T) *ast.File {
t.Helper()
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
return f
}
+28
View File
@@ -0,0 +1,28 @@
package main
import (
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
)
// COMPILE-TIME WITNESSES for OPTIONAL interfaces that are satisfied by a RUNTIME type assertion.
//
// WHY THIS FILE EXISTS. `localapi.BackupArchiveLister` is asserted at server.go's `newestArchiveOn`
// via `tier.Service.(BackupArchiveLister)`. A failed assertion does not error — it degrades to
// `archiveAbsent`, i.e. the pre-R-84 "ask the in-memory record only" behaviour. That degrade is
// SILENT and it is behaviour-relevant: it is exactly the R-84 bug (a cold store after a restart
// reading as "no backup ever") coming back, with nothing in any log to say so.
//
// The precedent is not hypothetical. During R-88 Part 2 the controller's `quiesceBackend` stopped
// satisfying `quiesce.TieredBackend` when a signature changed, and `go build` AND `go vet` both
// passed — because the interface is only ever asserted at runtime. Every box would have degraded to
// the single-tier path, losing R-82's multi-tier backups, with no error anywhere. It was caught by
// accident.
//
// A witness costs one line and converts that class of failure from a silent production degrade into
// a compile error.
//
// THIS DOES NOT MAKE THE INTERFACE REQUIRED. The optionality is deliberate — it is what lets a
// BackupService without a lister still work. The witness pins the IMPLEMENTATION (this concrete type
// really does satisfy it), not the CONTRACT.
var _ localapi.BackupArchiveLister = (*backup.BackupRunner)(nil)
+27 -1
View File
@@ -43,13 +43,15 @@ func main() {
func run() error {
var (
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission")
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update")
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
paramsRaw = flag.String("params", "", "op params as JSON (overrides -durable-id/-fstype)")
durableID = flag.String("durable-id", "", "storage_wipe: the DURABLE device id (byid:…|byuuid:…); decommission: the drive's STORAGE durable-id (e.g. uuid:<fs-uuid>)")
fstype = flag.String("fstype", "ext4", "for storage_wipe: the filesystem to mkfs after wipe")
agentVer = flag.String("agent-version", "", "for agent_update: the target agent version (e.g. 0.70.1)")
sha256Hex = flag.String("sha256", "", "for agent_update: the pinned lowercase-hex sha256 of the target binary")
keyFile = flag.String("key", "", "operator signing key (ssh private key / sk- key handle) for ssh-keygen -Y sign")
ttl = flag.Duration("ttl", 30*time.Minute, "validity window from now (issued_at..expires_at)")
nonce = flag.String("nonce", "", "explicit nonce (default: a fresh 128-bit random nonce)")
@@ -82,6 +84,17 @@ func run() error {
}
pj, _ := json.Marshal(map[string]string{"durable_id": *durableID})
params = string(pj)
case "agent_update":
// The agent downloads the binary for -agent-version and verifies it against -sha256.
// The sha is the ONLY integrity root, so both are mandatory and the sha is strict-validated.
if *agentVer == "" || *sha256Hex == "" {
return fmt.Errorf("agent_update needs -agent-version and -sha256 (the pinned binary hash)")
}
if !isHex64(*sha256Hex) {
return fmt.Errorf("agent_update -sha256 must be 64 lowercase hex chars (got %d)", len(*sha256Hex))
}
pj, _ := json.Marshal(map[string]string{"version": *agentVer, "sha256": *sha256Hex})
params = string(pj)
default:
params = "{}"
}
@@ -141,6 +154,19 @@ func run() error {
return nil
}
// isHex64 reports whether s is exactly 64 lowercase hex chars (a sha256 hex digest).
func isHex64(s string) bool {
if len(s) != 64 {
return false
}
for _, c := range s {
if (c < '0' || c > '9') && (c < 'a' || c > 'f') {
return false
}
}
return true
}
// signWithSSHKeygen signs `message` with `ssh-keygen -Y sign -n <namespace>`, the hardware-ready
// path (sk-/YubiKey keys work unchanged). It writes the message to a temp file, runs ssh-keygen,
// and reads the armored SSHSIG it produces. The namespace is the agent's FIXED domain separator.
+24
View File
@@ -0,0 +1,24 @@
package main
import "testing"
// TASK D1 Group D — agent_update opsign param validation. isHex64 is the sha gate the CLI applies
// before it will build an agent_update envelope (the agent re-validates too, but a bad sha should
// never even be signed).
func TestIsHex64(t *testing.T) {
good := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" // 64 'a'
if !isHex64(good) {
t.Errorf("isHex64(%q) = false, want true", good)
}
for name, bad := range map[string]string{
"too short": "abcdef",
"too long": good + "a",
"uppercase hex": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
"non-hex char": "g" + good[1:],
"empty": "",
} {
if isHex64(bad) {
t.Errorf("%s: isHex64(%q) = true, want false", name, bad)
}
}
}
+8 -2
View File
@@ -48,10 +48,16 @@
},
"local_api": {
"enable": true,
"listen_addr": "192.168.0.162:8443",
"listen_addr": "169.254.253.1:8443",
"cert_file": "/var/lib/felhom-agent/local-api.crt",
"key_file": "/var/lib/felhom-agent/local-api.key",
"token_store": "/var/lib/felhom-agent/local-tokens.log"
"token_store": "/var/lib/felhom-agent/local-tokens.log",
"island_bridge": "vmbr9",
"island_guest_addr": "169.254.253.2/30"
},
"lan_resolver": {
"enable": true,
"host_ip": "192.168.0.162"
},
"log_level": "info"
}
+233 -37
View File
@@ -20,39 +20,89 @@
# config mount (/etc/felhom-bootstrap/bootstrap.json) — no docker login/pull at deploy. Refreshing
# the golden bumps the controller baseline; controller self-update covers in-between drift.
#
# Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] [CONTROLLER_IMAGE]
# Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE
# CONTROLLER_IMAGE is REQUIRED (no default) — pass the released controller tag explicitly,
# e.g. gitea.dooplex.hu/admin/felhom-controller:0.98.3.
# Build-time registry login for the controller pull (used ONCE inside the build guest, then logged
# out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment.
#
# OS / Docker-data SPLIT (storage-split slice): the golden is built with a SMALL OS rootfs and a
# SEPARATE Docker-data volume mounted at /var/lib/docker (mp0, backup=1). The baked controller +
# infra images land on that volume and travel INSIDE the golden archive — so provisioned guests boot
# from baked images with no registry pull. The split is for RESILIENCE: an isolated OS rootfs stays
# bootable + agent-recoverable if the Docker volume fills (the controller's prevention layer keeps it
# from filling). Sizes are env-overridable (OS_SIZE_GB / GOLDEN_DOCKER_GB); provision GROWS the data
# volume to the per-customer target (bringup.go DataVolGrowGB). backup=1 is MANDATORY on the data mp:
# without it vzdump EXCLUDES the volume (extra LXC mountpoints default backup=0 — storage-split B3),
# so the archive would carry NO images and provisioned guests would boot imageless.
# OS / DATA SPLIT, and since v3.0.0 ONE DATA VOLUME (R-165, decision D-a + variant V-c).
#
# The golden is built with a SMALL OS rootfs and a SINGLE data volume (mp0, backup=1) mounted at a
# NEUTRAL path, /var/lib/felhom. Both consumer paths are binds of subdirectories of it:
#
# /var/lib/felhom/docker --bind--> /var/lib/docker (Docker's data-root)
# /var/lib/felhom/sys_drive --bind--> /mnt/sys_drive (the controller's system_data_path)
#
# WHAT THIS REPLACED, AND WHY. Until v2.1.0 these were TWO volumes (mp0 16 G at /var/lib/docker,
# mp1 8 G at /mnt/sys_drive, grown separately at provision). The second one was a fixed ceiling: an
# app whose local recovery unit outgrew it stopped being backed up even with free space next door.
# D-a removed the wall rather than moving it — one volume, one free-space figure, no ceiling.
#
# WHY A NEUTRAL MOUNT AND NOT SIMPLY NESTING ONE PATH INSIDE THE OTHER. Both simpler shapes were
# built and measured (SPIKE-r165-phase0-2026-08-03.md); both boot and reboot cleanly, and each breaks
# a different documented guarantee:
# * volume at /var/lib/docker -> customer backups live INSIDE Docker's data-root, so `du` there
# stops meaning what it says and the ordinary "clear /var/lib/docker to fix Docker" reflex
# destroys every local recovery unit on the box;
# * volume at /mnt/sys_drive -> Docker's ENTIRE data-root lands under /mnt, which the controller
# container mounts wholesale (`-v /mnt:/mnt:rslave`). Measured: the container then sees
# /mnt/sys_drive/docker. The bootstrap's own claim that /mnt "holds only Felhom's
# felhom-data-namespace mounts" would become false.
# The neutral mount breaks neither, for one extra path and one extra fstab line.
#
# The split from the OS rootfs is still for RESILIENCE: an isolated rootfs stays bootable +
# agent-recoverable if the data volume fills (the controller's prevention layer, and since
# controller v0.192.0 the capture floor, keep it from filling). Size is env-overridable
# (OS_SIZE_GB / GOLDEN_VOLUME_GB); provision GROWS the one volume (bringup.go DataVolGrowGB).
# backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra LXC mountpoints default
# backup=0 — storage-split B3), so the archive would carry no images AND no user data.
set -euo pipefail
# Script provenance — logged into every bake transcript next to the baked controller tag, so an
# archive can always be traced to the script that produced it. Bump on any behavior change.
GOLDEN_SCRIPT_VERSION="3.0.0"
VMID="${1:-9100}"
TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
ROOTFS_STORAGE="${3:-local-lvm}"
ARCHIVE_STORAGE="${4:-local}"
BRIDGE="${5:-vmbr0}"
CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.43.0}"
# CONTROLLER_IMAGE is MANDATORY — no default. The hand-bumped default rotted twice (0.43.0 →
# 0.85.1 → stale again): each time, a fresh provision would have booted a pre-floor controller
# needing a manual install-day update (drill finding B5). A required argument cannot rot, and
# auto-resolving "latest" could bake a tag the hub manifest never vouched — so the caller states
# the released controller tag explicitly at every rebuild.
CONTROLLER_IMAGE="${6:-}"
if [ -z "$CONTROLLER_IMAGE" ]; then
echo "[golden] FATAL: CONTROLLER_IMAGE (argument 6) is required — pass the released controller tag explicitly." >&2
echo "Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE" >&2
echo " e.g.: build-golden.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.98.3" >&2
exit 1
fi
REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
# OS rootfs size (GiB) and the golden's Docker-data volume size (GiB). Keep GOLDEN_DOCKER_GB just
# large enough for the baked images + headroom; provision grows it to the per-customer target.
# OS rootfs size (GiB) and the golden's SINGLE data volume size (GiB).
#
# ONE VOLUME MEANS ONE NUMBER (v3.0.0). The retired GOLDEN_SYSDATA_GB has no successor: there is
# nothing left to size separately. Keep GOLDEN_VOLUME_GB just large enough for the baked images plus
# headroom for the controller's felhom-data skeleton; provision grows the one volume to the
# per-customer target (bringup.go DataVolGrowGB).
OS_SIZE_GB="${OS_SIZE_GB:-32}"
GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}"
# 24 = the retired pair's 16 (docker) + 8 (user-data), so a golden archive carries the same content it
# did before the merge. It is deliberately NOT a per-customer size: provision grows it.
GOLDEN_VOLUME_GB="${GOLDEN_VOLUME_GB:-24}"
# The neutral mount path of the single volume. Both consumer paths are binds of subdirectories of it.
GOLDEN_VOLUME_MP="/var/lib/felhom"
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker, backup=1) …"
echo "[golden] build-golden.sh v${GOLDEN_SCRIPT_VERSION} — baking controller ${CONTROLLER_IMAGE}"
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, backup=1) …"
# ONE mpN slot. There is deliberately no mp1: that slot held the retired user-data volume, and the
# whole point of R-165 is that it stops existing rather than being made bigger.
pct create "$VMID" "$TEMPLATE" \
--hostname felhom-golden --unprivileged 1 \
--features nesting=1,keyctl=1 \
--rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_VOLUME_GB},mp=${GOLDEN_VOLUME_MP},backup=1" \
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
echo "[golden] starting + installing Docker (official repo, trixie channel) …"
@@ -81,8 +131,10 @@ echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshott
# The classic overlay2 driver stores EVERYTHING (images + overlay + volumes) under data-root
# (/var/lib/docker) = the data volume, which is exactly what "one data-root = one partition for all
# images + overlay" requires. It also makes the controller's statfs("/") (its overlay root) report the
# DATA volume, which the prevention layer depends on. /var/lib/docker is the mp0 mount (mounted empty
# before docker installs), so data-root needs no override. Log caps kill the most common runaway.
# DATA volume, which the prevention layer depends on — MEASURED to still hold under the v3.0.0 merged
# layout (a container's `df /` reports the single volume, phase-0 spike). Since v3.0.0 /var/lib/docker
# is a BIND of <volume>/docker rather than the mp0 mount itself, wired immediately below; data-root
# still needs no override because the path is unchanged. Log caps kill the most common runaway.
pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
{
"features": { "containerd-snapshotter": false },
@@ -90,6 +142,32 @@ pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json
"log-opts": { "max-size": "10m", "max-file": "3" }
}
JSON'
echo "[golden] wiring the single data volume (R-165 variant V-c): ${GOLDEN_VOLUME_MP}/{docker,sys_drive} -> binds …"
# docker-ce has already populated /var/lib/docker ON THE ROOTFS by now (it auto-starts on install), so
# the content is MOVED onto the volume before the bind is laid over the top. Doing it the other way
# round would hide those files under the bind and silently ship a golden whose baked images are on the
# rootfs — the exact failure class the assertions below exist to catch.
#
# /etc/fstab, not a hand-run mount: systemd's fstab generator orders both binds under local-fs.target,
# which precedes basic.target and therefore docker.service. MEASURED across 3 reboots per variant in
# the phase-0 spike — the ordering worry that motivated the probe did not materialise.
pct exec "$VMID" -- bash -c "
set -e
systemctl stop docker docker.socket containerd 2>/dev/null || true
mkdir -p '${GOLDEN_VOLUME_MP}/docker' '${GOLDEN_VOLUME_MP}/sys_drive'
if [ -d /var/lib/docker ] && [ -n \"\$(ls -A /var/lib/docker 2>/dev/null)\" ]; then
cp -a /var/lib/docker/. '${GOLDEN_VOLUME_MP}/docker'/
rm -rf /var/lib/docker/*
fi
mkdir -p /var/lib/docker /mnt/sys_drive
printf '%s /var/lib/docker none bind 0 0\n' '${GOLDEN_VOLUME_MP}/docker' >> /etc/fstab
printf '%s /mnt/sys_drive none bind 0 0\n' '${GOLDEN_VOLUME_MP}/sys_drive' >> /etc/fstab
systemctl daemon-reload
mount /var/lib/docker
mount /mnt/sys_drive
systemctl start containerd
"
echo "[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …"
# RESTART (not start): docker-ce auto-starts on install with the DEFAULT config, so it is already
# running by now; only a restart picks up the daemon.json just written (overlay2 + log caps).
@@ -97,8 +175,20 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
# mean containerd-snapshotter is still on (the split would leave images on the rootfs).
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
# Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss).
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'
# ASSERTION 1 (RETARGETED v3.0.0, not removed). /var/lib/docker must be a real mount — now the V-c
# bind of <volume>/docker rather than the mp0 mount itself. Still fails closed on the same failure:
# if the bind did not take, Docker's data-root silently sits on the OS rootfs and the golden ships
# its baked images there.
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a real mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker | head -1)" || { echo "[golden] FATAL: /var/lib/docker is NOT a mount — the V-c docker bind did not take, so the baked images would land on the OS rootfs"; exit 1; }'
# ASSERTION 2 (RETARGETED v3.0.0). /mnt/sys_drive must be a real mount — now the V-c bind of
# <volume>/sys_drive. Otherwise the controller's system_data_path lands on the OS drive and it warns
# (clearing that warning is the whole point of the volume).
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a real mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive | head -1)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a mount — the V-c sys_drive bind did not take, so the controller system_data_path would be the OS rootfs"; exit 1; }'
# ASSERTION 2b (NEW v3.0.0 — the invariant the merge is FOR). Both paths must be backed by the SAME
# device, i.e. ONE filesystem with ONE free-space figure. Two devices here is the S2 shape the R-165
# spike ranked strictly WORSE than the split it replaced: every assertion satisfied, the ceiling still
# there, and a shared pool neither `df` can see coming.
pct exec "$VMID" -- bash -c 'n=$(df --output=source /var/lib/docker /mnt/sys_drive | tail -n +2 | sort -u | wc -l); [ "$n" = "1" ] && echo " both paths are ONE filesystem: $(df --output=source,avail /var/lib/docker | tail -1)" || { echo "[golden] FATAL: /var/lib/docker and /mnt/sys_drive are on $n DIFFERENT filesystems — that is the S2 shape (two ceilings), not the R-165 merge"; exit 1; }'
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
# docker login is used ONCE here on the trusted build host, then logged out before archiving so
@@ -107,29 +197,52 @@ if [ -n "${REGISTRY_USER:-}" ] && [ -n "${REGISTRY_TOKEN:-}" ]; then
pct exec "$VMID" -- bash -c "systemctl start docker; sleep 1; echo '$REGISTRY_TOKEN' | docker login '$REGISTRY_HOST' -u '$REGISTRY_USER' --password-stdin >/dev/null"
fi
pct exec "$VMID" -- bash -c "docker pull '$CONTROLLER_IMAGE'"
pct exec "$VMID" -- bash -c "docker logout '$REGISTRY_HOST' >/dev/null 2>&1 || true; rm -f /root/.docker/config.json"
# Record the baked image ref for the bootstrap unit (so the unit needs no login/pull).
pct exec "$VMID" -- bash -c "printf '%s\n' '$CONTROLLER_IMAGE' > /etc/felhom-controller-image"
# Bake the base-infrastructure images (traefik, cloudflared, filebrowser) so the controller's
# first-boot bring-up (EnsureBaseStack) is OFFLINE-capable — no registry pull at deploy. These are
# PUBLIC Docker Hub images (no cred needed). The PINNED tags MUST match the controller's
# internal/infra constants (TraefikImage / CloudflaredImage / FileBrowserImage); a drift means the
# golden bakes one image and the controller requests another (→ a pull at deploy, defeating the goal).
INFRA_IMAGES=(
"traefik:v3.6.7"
"cloudflare/cloudflared:2026.6.0"
"gtstef/filebrowser:1.3.3-stable"
)
echo "[golden] baking base-infra images: ${INFRA_IMAGES[*]}"
# Bake EVERY controller-managed infra image so enabling an infra stack on a fresh box is near-instant
# instead of a multi-minute silent registry pull.
#
# THE LIST COMES FROM THE CONTROLLER WE JUST PULLED, not from a copy here. This script used to carry
# a hand-maintained array of three tags with a comment telling the reader to keep it in sync with the
# controller's internal/infra constants. It drifted the moment a fourth stack was added: felhom-samba
# was never added here, so the golden baked 3 of 4 and turning on Megosztás pulled from the registry
# with zero UI feedback — observed live, twice. Asking the binary (`--print-infra-images`, backed by
# infra.Images()) makes golden-vs-controller drift structurally impossible: the golden bakes exactly
# what THIS controller version will request.
echo "[golden] asking the controller which infra images it manages …"
INFRA_LIST="$(pct exec "$VMID" -- bash -c "docker run --rm --entrypoint /usr/local/bin/felhom-controller '$CONTROLLER_IMAGE' --print-infra-images 2>/dev/null" || true)"
mapfile -t INFRA_IMAGES < <(printf '%s\n' "$INFRA_LIST" | grep -E '^[a-z0-9._/-]+:[A-Za-z0-9._-]+$' || true)
if [ "${#INFRA_IMAGES[@]}" -eq 0 ]; then
# Controllers older than v0.147.0 have no --print-infra-images. Fall back to the historical list so
# an old-controller bake still works — but say so loudly, because the fallback is exactly the
# drift-prone thing this change removed, and it CANNOT know about stacks added after it was written.
echo "[golden] WARN: '$CONTROLLER_IMAGE' does not support --print-infra-images (pre-0.147.0)."
echo "[golden] WARN: falling back to the historical 3-image list. felhom-samba will NOT be baked,"
echo "[golden] WARN: so enabling Megosztás on this golden will pull at runtime. Bake >=0.147.0."
INFRA_IMAGES=(
"traefik:v3.6.7"
"cloudflare/cloudflared:2026.6.0"
"gtstef/filebrowser:1.3.3-stable"
)
fi
echo "[golden] baking infra images (${#INFRA_IMAGES[@]}): ${INFRA_IMAGES[*]}"
for img in "${INFRA_IMAGES[@]}"; do
# Hard gate: fail loudly BEFORE pulling if a pinned tag doesn't resolve (a bad pin otherwise fails
# mid-bake with a confusing error).
pct exec "$VMID" -- bash -c "docker manifest inspect '$img' >/dev/null 2>&1" \
|| { echo "[golden] FATAL: pinned base-infra image does not resolve: $img"; exit 1; }
|| { echo "[golden] FATAL: pinned infra image does not resolve: $img"; exit 1; }
pct exec "$VMID" -- bash -c "docker pull '$img'"
done
# Log out LAST — not right after the controller pull. felhom-samba lives on the same private registry
# as the controller, so the infra loop above needs the credential; logging out first made that pull
# 401. The credential is still never archived: the logout + config.json removal happen before the
# guest is stopped and templated.
pct exec "$VMID" -- bash -c "docker logout '$REGISTRY_HOST' >/dev/null 2>&1 || true; rm -f /root/.docker/config.json"
pct exec "$VMID" -- bash -c '[ ! -s /root/.docker/config.json ]' \
|| { echo "[golden] FATAL: registry credential still present in the guest — refusing to archive"; exit 1; }
echo "[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …"
pct push "$VMID" /dev/stdin /usr/local/sbin/felhom-controller-bootstrap.sh --perms 700 <<'BOOTSH'
#!/bin/bash
@@ -203,6 +316,26 @@ WantedBy=multi-user.target
UNIT
systemctl enable felhom-controller-bootstrap.service'
# B1 (DRILL-day0-cleanroom-2026-07-03 R6/B1): the service's ConditionPathExists is evaluated only
# when the service is STARTED — i.e. at boot via multi-user.target — but the agent back-half
# hot-plugs the bootstrap mount into the ALREADY-RUNNING guest, so on a provision the boot-time
# start races the mount and loses on slow hardware. This path unit watches for bootstrap.json and
# starts the service when it APPEARS — covering the provision hot-plug without a reboot. The boot
# case is still served by the enabled service itself; RemainAfterExit=yes on the service means the
# path unit cannot re-trigger it in a loop once it has run.
echo "[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …"
pct exec "$VMID" -- bash -c 'cat > /etc/systemd/system/felhom-controller-bootstrap.path <<UNIT
[Unit]
Description=Watch for the agent-populated bootstrap config; start the controller bootstrap when it appears
[Path]
PathExists=/etc/felhom-bootstrap/bootstrap.json
[Install]
WantedBy=multi-user.target
UNIT
systemctl enable felhom-controller-bootstrap.path'
echo "[golden] baking the first-boot SSH host-key regeneration unit (F3) …"
pct exec "$VMID" -- bash -c 'cat > /etc/systemd/system/felhom-regen-hostkeys.service <<UNIT
[Unit]
@@ -235,17 +368,80 @@ pct exec "$VMID" -- bash -c '
echo "[golden] stop + archive …"
pct stop "$VMID"
# --mode stop with mp0 backup=1 → the Docker-data volume (with baked images) is INCLUDED. The log
# below MUST show "including mount point mp0" — if it shows "excluding … (disabled)" the backup flag
# was lost and the archive carries no images (storage-split B3 trap).
# --mode stop with mp0 backup=1 → the SINGLE data volume (baked images AND the user-data area) is
# INCLUDED. The log MUST show "including mount point mp0" and must NOT show it being excluded — an
# exclusion means the backup flag was lost and the archive carries neither (storage-split B3 trap).
# Since v3.0.0 there is no mp1; the guard that covered it is retargeted below rather than deleted,
# because a guard whose pattern can no longer match is a guard that has silently stopped guarding.
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
exit 1
fi
# ASSERTION 4 (RETARGETED v3.0.0). The mp1 guard used to catch "the user-data volume fell out of the
# archive". After the merge there is no mp1 — so the same failure now looks like the volume being
# mounted at the WRONG PATH, which would carry the images but not the user-data area. Assert the
# inclusion line names the volume's actual mount path.
if ! grep -q "including mount point mp0 ('${GOLDEN_VOLUME_MP}')" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: the archive's mp0 is not ${GOLDEN_VOLUME_MP} — the single data volume is mounted somewhere unexpected, so the archive would not carry both the baked images and the user-data area. Aborting."
grep -iE "mount point" /tmp/golden-vzdump.log || true
exit 1
fi
# ASSERTION 5 (RETARGETED v3.0.0). There must be NO mp1 in the archive at all. A leftover second
# volume means the merge did not take and this golden would ship the very ceiling R-165 removed.
if grep -qE "mount point mp1" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: the archive still carries an mp1 — the R-165 merge did not take and this golden would ship a second, ceilinged volume. Aborting."
exit 1
fi
grep -q "including mount point mp0" /tmp/golden-vzdump.log \
|| echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G, both in the archive)"
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, all in the archive)"
#-------------------------------------------------------------------------------
# Publish to Gitea (BUNDLE slice) — make this golden fetchable by the host-bootstrap script.
#-------------------------------------------------------------------------------
# The host-install script fetches the golden from Gitea
# (/api/packages/admin/generic/felhom-golden/<golden-version>/golden.tar.zst) and verifies its sha256
# against the hub-vouched artifact manifest before importing it. <golden-version> = the BAKED
# controller version (so the golden's published version tracks what it ships). Publishing is OPT-IN:
# only runs when GITEA_USER + GITEA_TOKEN (or REGISTRY_USER/REGISTRY_TOKEN) are set. The local-golden
# auto-discovery in the host-install script stays as a fallback either way.
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
GITEA_OWNER="${GITEA_OWNER:-admin}"
PUB_USER="${GITEA_USER:-${REGISTRY_USER:-}}"
PUB_TOKEN="${GITEA_TOKEN:-${REGISTRY_TOKEN:-}}"
# Golden version = the baked controller tag's version (strip the image path + any leading 'v').
GOLDEN_VERSION="${GOLDEN_VERSION:-${CONTROLLER_IMAGE##*:}}"; GOLDEN_VERSION="${GOLDEN_VERSION#v}"
if [ -z "$VOLID" ]; then
echo "[golden] WARN: could not resolve the archive volid — skipping Gitea publish."
elif [ -z "$PUB_USER" ] || [ -z "$PUB_TOKEN" ]; then
echo "[golden] Gitea publish SKIPPED (set GITEA_USER+GITEA_TOKEN or REGISTRY_USER+REGISTRY_TOKEN to enable)."
echo "[golden] would publish version=$GOLDEN_VERSION from volid $VOLID"
else
# Resolve the archive's on-disk path (pvesm path turns a volid into a filesystem path).
ARCHIVE_PATH="$(pvesm path "$VOLID" 2>/dev/null || true)"
if [ -z "$ARCHIVE_PATH" ] || [ ! -f "$ARCHIVE_PATH" ]; then
echo "[golden] WARN: cannot resolve archive path for $VOLID — skipping publish."
else
GOLDEN_SHA256="$(sha256sum "$ARCHIVE_PATH" | awk '{print $1}')"
PUB_URL="${GITEA_BASE}/api/packages/${GITEA_OWNER}/generic/felhom-golden/${GOLDEN_VERSION}/golden.tar.zst"
echo "[golden] publishing golden ($(wc -c < "$ARCHIVE_PATH") bytes, sha256 ${GOLDEN_SHA256:0:16}…) → $PUB_URL"
# Delete-then-PUT so re-publishing the same version overwrites cleanly (idempotent).
dcode="$(curl -fsS -o /dev/null -w '%{http_code}' -u "${PUB_USER}:${PUB_TOKEN}" -X DELETE "$PUB_URL" 2>/dev/null || true)"
echo "[golden] pre-delete existing: HTTP ${dcode} (404/204 expected)"
ucode="$(curl -sS -o /dev/null -w '%{http_code}' -u "${PUB_USER}:${PUB_TOKEN}" -X PUT --upload-file "$ARCHIVE_PATH" "$PUB_URL")"
if [ "$ucode" = "201" ] || [ "$ucode" = "200" ]; then
echo "[golden] upload OK (HTTP $ucode)"
echo "GOLDEN_VERSION=${GOLDEN_VERSION}"
echo "GOLDEN_SHA256=${GOLDEN_SHA256}"
echo "[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden ${GOLDEN_VERSION} / ${GOLDEN_SHA256}"
else
echo "[golden] WARN: golden upload failed (HTTP $ucode) — the local archive is still usable via auto-discovery."
fi
fi
fi
echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)"
+19
View File
@@ -0,0 +1,19 @@
# felhom-agent-limits.conf — start-limit + rollback-trigger drop-in for felhom-agent.service
# (TASK D1). Install as /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf
# and `systemctl daemon-reload`.
#
# Values are the SPIKE-agent-selfupdate-2026-07-05 tuned set, verbatim [SF-2]: with the unit's
# Restart=on-failure + RestartSec=5s and systemd 257's compiled defaults (10s/5), a crash-looping
# binary NEVER trips the start limit and loops forever (35 starts/180s measured). 120s/4 gives a
# terminal `failed` ≈20s after the first crash.
#
# PLACEMENT TRAP [SF-3]: these keys MUST be in [Unit]. On systemd 257 a [Service] placement is
# HALF-APPLIED — StartLimitBurst is accepted while StartLimitIntervalSec is silently ignored
# (journal warning only). Never split them; never put them in [Service].
#
# OnFailure fires on EVERY crash on systemd 257 [SF-1] — see the comment block in
# felhom-agent-rollback.service for why that is safe (pending-marker guard).
[Unit]
StartLimitIntervalSec=120
StartLimitBurst=4
OnFailure=felhom-agent-rollback.service
+21
View File
@@ -0,0 +1,21 @@
# felhom-agent-rollback.service — the OnFailure= target that auto-reverts a crash-looping agent
# self-update (TASK D1; SPIKE-agent-selfupdate-2026-07-05).
#
# Install as /etc/systemd/system/felhom-agent-rollback.service. It is referenced by the
# felhom-agent-limits.conf drop-in's OnFailure= line.
#
# THE PER-CRASH-FIRING REALITY [SF-1]: on systemd 257, OnFailure= fires on EVERY crash of the main
# unit — even while it is merely `activating (auto-restart)`, long before (and regardless of) the
# terminal start-limit `failed` state. During one crash incident this unit therefore runs MANY
# times. That is safe BY DESIGN: the wrapper's rollback verb is pending-marker-guarded — the first
# fire after a bad update reverts the binary and clears the marker; every later fire (and any fire
# with no update in flight at all) is a logged no-op that touches nothing. Consequence: a bad
# update is rolled back at the FIRST crash (~seconds), not after the start-limit burst — the tuned
# start-limit in the drop-in is the terminal BACKSTOP (e.g. an environmental crash loop of the
# known-good binary), not the rollback trigger.
[Unit]
Description=Felhom agent self-update auto-rollback (pending-guarded; fires per crash, no-ops without a pending update)
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/felhom-selfupdate-guarded rollback
+50
View File
@@ -0,0 +1,50 @@
# felhom-agent.service — canonical systemd unit for the Felhom host agent.
#
# Install as /etc/systemd/system/felhom-agent.service (the host-bootstrap script installs this from
# the Gitea-published binary; previously this unit was hand-made on each host). The agent runs as the
# NON-root `felhom-agent` service user (the documented production model — README "Process model";
# `privileged.mode: "sudo"`) and shells the few host-root ops out via `sudo -n` against the fixed-arg
# allowlist in /etc/sudoers.d/felhom-agent (configs/felhom-agent.sudoers). The User= here and the
# sudoers `felhom-agent ALL=(root) NOPASSWD: …` MUST name the SAME user.
#
# Paths MUST match what the sudoers / host-install script expect:
# binary /usr/local/bin/felhom-agent
# config /etc/felhom-agent/agent.json (0600 felhom-agent:felhom-agent — secrets live here)
# state /var/lib/felhom-agent (nonces, local-api cert/key/tokens, staged units, guests)
#
# === DELIBERATELY NO SANDBOXING — read before adding any hardening directive ===
#
# 1. NoNewPrivileges is NOT set. It is INCOMPATIBLE with the agent's privilege model: it blocks the
# setuid `sudo` the agent relies on for EVERY host-root op (mount, format, pct, dnsmasq …), so the
# agent would silently lose all privileged capability. The narrow surface comes from the sudoers
# fixed-arg allowlist + the agent's in-process fine validation (internal/storage/validate.go), NOT
# from NoNewPrivileges. Do not add it.
#
# 2. NO mount-namespacing hardening (ProtectHome, ProtectSystem, PrivateTmp, ReadOnlyPaths,
# ProtectControlGroups, …). Any of these give the unit a PRIVATE mount namespace — and the agent's
# intermediary-mount drive model does `mount --make-shared /mnt/felhom-drives` + `mount --bind` and
# relies on those propagating into the RUNNING customer guest. In a private namespace the binds
# would be invisible to the host/guest and every external-drive enrollment would silently break.
# The agent MUST share the host mount namespace. The security boundary is the sudoers allowlist.
[Unit]
Description=Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)
Documentation=https://gitea.dooplex.hu/admin/felhom-agent
After=network-online.target pve-cluster.service pveproxy.service
Wants=network-online.target
[Service]
Type=simple
User=felhom-agent
Group=felhom-agent
ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json
Restart=on-failure
RestartSec=5s
# State dir: created 0750 felhom-agent:felhom-agent on start if absent (local-api cert/key/tokens,
# nonces, staged .mount units, per-guest bootstrap dirs all live here).
StateDirectory=felhom-agent
StateDirectoryMode=0750
[Install]
WantedBy=multi-user.target
+255 -11
View File
@@ -24,7 +24,9 @@ Cmnd_Alias FELHOM_DISK = \
/usr/sbin/smartctl -a -j /dev/nvme[0-9]*n[0-9]*, \
/usr/sbin/smartctl -a -j /dev/vd[a-z]*, \
/usr/sbin/smartctl -a -j /dev/hd[a-z]*, \
/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- *
/usr/sbin/lvs --reportformat json --units b -o lv_name\,data_percent\,metadata_percent -- *, \
/usr/sbin/pvs --reportformat json --noheadings -o pv_name, \
/usr/sbin/zpool status -P
# Provisioning back-half (slice 8A, doc 03 §6): populate a guest's bootstrap config mount
# host-side (internal/provision). These are host-root ops the API token cannot do — a bind mount
@@ -34,18 +36,18 @@ Cmnd_Alias FELHOM_DISK = \
# writes there is the only thing these touch. ':' is escaped per sudoers grammar.
Cmnd_Alias FELHOM_PROVISION = \
/usr/bin/chown -R 100000\:100000 /var/lib/felhom-agent/guests/*, \
/usr/sbin/pct set [0-9]* -mp[0-9]* /var/lib/felhom-agent/guests/*
/usr/sbin/pct set [0-9]* -mp[0-9]* /var/lib/felhom-agent/guests/*, \
/usr/sbin/pct set [0-9]* -onboot 1
# Disk inspection + format (slice 8C). blkid/lsblk read the device's data-bearing evidence (the
# agent decides data-bearing-ness from THIS, never the caller's claim); mkfs.* formats a device the
# agent already classified blank (a data-bearing format is refused pending an operator signature).
# The agent fine-validates the device path (ValidateBlockDevice: raw disk / partition under /dev
# only) + fstype before any exec — the wildcard is the coarse allowlist, the agent is the fine gate.
# Disk inspection + format (slice 8C + Impl-1). blkid/lsblk read the device's data-bearing evidence
# (the agent decides data-bearing-ness from THIS, never the caller's claim). Format goes ONLY through
# felhom-mkfs-guarded (Impl-1 Part B): raw mkfs.* is NO LONGER allowlisted, so even a bad agent cannot
# mkfs the OS disk — the wrapper re-checks the catastrophic cases (system disk / LVM PV / foreign mount)
# as root and refuses, and the agent's unclaimed-disk filter (claim.go) is the primary guard above it.
Cmnd_Alias FELHOM_FORMAT = \
/usr/sbin/blkid -p -o export /dev/*, \
/usr/bin/lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT /dev/*, \
/usr/sbin/mkfs.ext4 -F /dev/*, \
/usr/sbin/mkfs.xfs -f /dev/*
/usr/bin/lsblk -J -o NAME\,FSTYPE\,PTTYPE\,MOUNTPOINT /dev/*, \
/usr/local/sbin/felhom-mkfs-guarded /dev/* *
# LAN split-horizon resolver (internal/lanresolver): the agent manages a host-side dnsmasq that
# answers *.<customer-domain> with each guest's live LAN IP. install only ever writes felhom-*.conf
@@ -57,8 +59,250 @@ Cmnd_Alias FELHOM_DNSMASQ = \
/usr/bin/install -m 0644 /tmp/felhom-resolver-*.conf /etc/dnsmasq.d/felhom-*.conf, \
/usr/bin/systemctl enable --now dnsmasq, \
/usr/bin/systemctl reload dnsmasq, \
/usr/bin/systemctl restart dnsmasq, \
/usr/bin/rm -f /etc/dnsmasq.d/felhom-*.conf, \
/usr/sbin/pct exec [0-9]* -- ip -4 -o addr show dev eth0, \
/usr/sbin/pct exec [0-9]* -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller.yaml
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ
# Guest mountpoint lifecycle (intermediary-mount re-architecture + C1 net). The pre-start self-heal hook
# wrapper is installed once into the PVE snippets dir (from an agent-written /tmp file) and registered
# per-guest; decommission/eject DELETE the dead mountpoint slot so a missing bind source can't brick the
# guest at next boot (the B3 C1 fix). The agent fine-validates the vmid (numeric) + slot (mp[0-9]+) and
# the snippet path is fixed — the wildcards are the coarse allowlist. The install SOURCE is a
# random-named agent temp (os.CreateTemp, audit B1 — a fixed /tmp name was a local TOCTOU), hence the
# glob; the DESTINATION stays pinned. The `mkdir -p` creates the snippets dir on a FRESH box —
# `install` won't create parents, so without it the hook install failed silently on Day-0 boxes
# (B2, DRILL-day0-cleanroom-2026-07-03; fixed agent v0.63.0).
Cmnd_Alias FELHOM_GUESTHOOK = \
/usr/bin/mkdir -p /var/lib/vz/snippets, \
/usr/bin/install -m 0755 -- /tmp/felhom-guest-hook-*.sh /var/lib/vz/snippets/felhom-guest-hook.sh, \
/usr/sbin/pct set [0-9]* --hookscript local\:snippets/felhom-guest-hook.sh, \
/usr/sbin/pct set [0-9]* --delete mp[0-9]*, \
/usr/sbin/pct reboot [0-9]*
# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
# /mnt/felhom-drives (self-bind + make-shared + a boot-persistence systemd unit) and binds/unbinds each
# drive's felhom-data namespace UNDERNEATH it so the change propagates into the running guest live (no
# pct, no reboot). The agent fine-validates the drive name + confines paths before any exec; the trailing
# `*` (matching the comma-laden mp spec) mirrors the existing FELHOM_PROVISION pattern.
# `lxc-info -n <vmid> -p -H` resolves the guest init PID for the GuestSeesMount / bound_under_parent check
# (a READ — the drive-gate's "is the drive live in the guest?" signal); WITHOUT it the non-root agent gets
# an empty PID and reports every drive absent (multi-drive flapping, audit 2026-06-29). `make-private`
# isolates the parent's peer group on FIRST setup only (EnsureSharedParent guards on mountpoint, so it
# never re-churns a live parent); without it the parent stays in root's group and submounts double.
Cmnd_Alias FELHOM_INTERMEDIARY = \
/usr/bin/mkdir -p /mnt/felhom-drives, \
/usr/bin/mkdir -p /mnt/felhom-drives/*, \
/usr/bin/mkdir -p /mnt/*/felhom-data, \
/usr/bin/chown 100000\:100000 /mnt/*/felhom-data, \
/usr/bin/mount --bind /mnt/felhom-drives /mnt/felhom-drives, \
/usr/bin/mount --make-shared /mnt/felhom-drives, \
/usr/bin/mount --make-private /mnt/felhom-drives, \
/usr/bin/mount --bind /mnt/*/felhom-data /mnt/felhom-drives/*, \
/usr/bin/umount /mnt/felhom-drives/*, \
/usr/bin/install -m 0755 -- /tmp/felhom-shared-parent-*.sh /usr/local/sbin/felhom-shared-parent.sh, \
/usr/bin/install -m 0644 -- /tmp/felhom-shared-parent-*.service /etc/systemd/system/felhom-shared-parent.service, \
/usr/bin/systemctl enable felhom-shared-parent.service, \
/usr/bin/lxc-info -n [0-9]* -p -H, \
/usr/sbin/pct set [0-9]* -mp8 /mnt/felhom-drives*
# Controller-swap / managed auto-update (Option A, non-root). The agent owns the in-guest controller
# image SWAP (it survives the controller being killed mid-swap): read the baked image ref, check the
# pre-pulled target is present, rewrite /etc/felhom-controller-image, restart the bootstrap unit,
# health-check, roll back on failure. Each grant is bounded — NO general `pct exec` and NO `bash -c`:
# cat <fixed file> — read the current image ref (read-only)
# docker image inspect * — is the pre-pulled target present? (read-only)
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
# + container across spaces, spike-confirmed)
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
# tee <FIXED image file> — WRITE the ref; content is fed on STDIN (no shell, no interpolation),
# the agent strict-validates the ref (controllerImageRe) before the write.
# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct exec [0-9]* -- cat /etc/felhom-controller-image, \
/usr/sbin/pct exec [0-9]* -- docker image inspect *, \
/usr/sbin/pct exec [0-9]* -- docker inspect -f *, \
/usr/sbin/pct exec [0-9]* -- systemctl restart felhom-controller-bootstrap.service, \
/usr/sbin/pct exec [0-9]* -- tee /etc/felhom-controller-image
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
# agent clears the STALE lock at startup (only when no vzdump is in-flight). `pct unlock` is the one op
# with no API equivalent (snapshot-delete + start go through the API token); the agent fine-validates the
# vmid (numeric) before exec — the `[0-9]*` is the coarse allowlist.
Cmnd_Alias FELHOM_STALELOCK = \
/usr/sbin/pct unlock [0-9]*
# Restore-test scratch teardown (F-LEAK, Campaign 8, v0.110.0). A restore-test whose restore FAILS
# leaves a scratch guest the API token CANNOT destroy: `FelhomAgentGuest` is granted at /pool/felhom and
# a guest joins that pool only when its restore COMPLETES, so a failed restore leaves a pool-less guest
# out of reach (403 VM.Allocate) holding its disks until a human removes it.
#
# TWO API-SIDE FIXES WERE TRIED AND BOTH REFUTED LIVE on 2026-07-28, which is why this grant exists:
# 1. Adopt the stranded guest into the pool, then retry. `PUT /pools/{pool}` ALSO requires
# VM.Allocate on the VM being added — pool membership cannot bootstrap its own authority.
# 2. Grant FelhomAgentGuest per-path at /vms/990000..990009. Durable for exactly one use per slot:
# PVE's own destroy path calls `AccessControl::remove_vm_access($vmid)` (LXC.pm:906), which DELETES
# every ACL at /vms/<vmid> (AccessControl.pm:1898). The grant is consumed by the operation it
# authorises, so after ten teardowns the band is ungranted and the defect returns.
#
# WHY THIS IS THE TIGHTEST AVAILABLE FENCE, not a widening: sudo matches the vmid LITERALLY, so
# `99000[0-9]` is exactly the ten-slot scratch band the restore-test picks from — nothing else. There is
# no `[0-9]*` coarse allowlist here on purpose: unlike `pct unlock`, this op DESTROYS, so the band must
# be in the policy and not merely validated in the agent. Even a compromised agent asking for
# `pct destroy 9201` is refused by sudo itself. Unlike an ACL, a sudoers rule is not consumed by use.
# The agent re-checks the band in code before exec (defence in depth); this is the outer fence.
Cmnd_Alias FELHOM_SCRATCH_TEARDOWN = \
/usr/sbin/pct destroy 99000[0-9] --purge
# Network storage / NAS (Part A1, SPIKE-nas-storage-2026-06-29). The agent mounts a customer NAS share
# HOST-SIDE under /mnt/felhom-drives/<name> via a systemd .automount (+ .mount) pair so it propagates
# into the guest through the existing shared bind (an unprivileged LXC cannot mount NFS/CIFS itself).
# A NAS is NOT a drive — no durable-id, no SMART, no wipe; these grants only install/enable/remove the
# unit pair. The agent fine-validates every value (share name, server, export, uid/gid, creds path) before
# any unit is rendered (internal/storage/netmount.go ValidateNetworkMountSpec); the trailing globs are the
# COARSE allowlist. The `.mount` install/enable/disable/stop reuse FELHOM_MOUNT; this alias adds the
# `.automount` variants + the unit-file removal. The unit FILE name is the systemd-escaped mountpoint,
# which always begins `mnt-felhom` (the mountpoint is /mnt/felhom-drives/<name>), so the rm glob is scoped
# to felhom mount units only. mkdir of the mountpoint reuses FELHOM_INTERMEDIARY's /mnt/felhom-drives/*.
# CAMPAIGN-3 additions (loud, per the no-widening rule):
# - `systemctl reset-failed -- mnt-felhom*`: F10 (CRITICAL) — a NAS automount that hit
# mount-start-limit-hit during an outage was re-armable by NO platform path; the reassert now
# reset-failed's the stuck unit before `enable --now` (which the start-limit otherwise refuses),
# and RemoveNetworkMount clears failed-state residue (F2). Scoped to felhom mount units (the unit
# name is the systemd-escaped mountpoint, always beginning `mnt-felhom`). reset-failed only clears
# a unit's failed latch — it cannot start/stop/alter anything.
# - `rmdir /mnt/felhom-drives/*`: F1 — remove the now-empty mountpoint dir a removed share leaves
# behind (the campaign accumulated 10 stub-shaped leftovers). rmdir ONLY (never rm -rf): it refuses
# a non-empty dir, so unexpected data is preserved, not destroyed — a fail-safe grant.
Cmnd_Alias FELHOM_NETMOUNT = \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/* /etc/systemd/system/*.automount, \
/usr/bin/systemctl enable --now -- *.automount, \
/usr/bin/systemctl disable -- *.automount, \
/usr/bin/systemctl stop -- *.automount, \
/usr/bin/systemctl reset-failed -- mnt-felhom*, \
/usr/bin/rmdir /mnt/felhom-drives/*, \
/usr/bin/rm -f /etc/systemd/system/mnt-felhom*
# Offsite WG tunnel (S3, doc 06 §3.3). The agent manages wg-quick@wg-felhom as an agent-managed
# host service (the dnsmasq/lanresolver shape): conf staged in the agent-owned StateDir (never
# /tmp), installed 0600 to the FIXED destination, unit enable/restart/disable. The ONLY wg read
# is `latest-handshakes` — `wg show <if> dump` is FORBIDDEN everywhere (its interface line
# carries the PRIVATE KEY; the S1 session-log incident). Both install paths are FIXED (no glob):
# the agent has exactly one tunnel conf to manage.
Cmnd_Alias FELHOM_WG = \
/usr/bin/apt-get install -y -q wireguard-tools, \
/usr/bin/install -o root -g root -m 0600 -- /var/lib/felhom-agent/wg/wg-felhom.conf /etc/wireguard/wg-felhom.conf, \
/usr/bin/systemctl enable --now wg-quick@wg-felhom, \
/usr/bin/systemctl restart wg-quick@wg-felhom, \
/usr/bin/systemctl disable --now wg-quick@wg-felhom, \
/usr/bin/wg show wg-felhom latest-handshakes
# Agent self-update (TASK D1, SPIKE-agent-selfupdate-2026-07-05). The agent downloads the
# operator-SIGNED binary (sha256 pinned in the signed op — neither hub nor Gitea compromise can
# substitute it), verifies the sha in-process, then hands off to the guarded wrapper, which
# RE-verifies the sha as root, confines the staged path to /var/lib/felhom-agent/selfupdate/,
# performs the A/B flip (atomic same-fs rename, .prev retained) and schedules a detached restart.
# The apply args are a COARSE glob (spike S4b: sudoers fnmatch makes a [a-f0-9]* sha pattern
# first-char-only anyway) — the wrapper's own sha re-verify + path confinement is the real gate.
# `rollback` is normally run by felhom-agent-rollback.service (root, OnFailure=), not via sudo;
# granting it here keeps the verb probe-able (capability self-check) and operator-invokable.
Cmnd_Alias FELHOM_SELFUPDATE = \
/usr/local/sbin/felhom-selfupdate-guarded apply /var/lib/felhom-agent/selfupdate/* *, \
/usr/local/sbin/felhom-selfupdate-guarded commit, \
/usr/local/sbin/felhom-selfupdate-guarded rollback
# Dedicated OOB sshd (TASK H1). The agent manages felhom-sshd like wg-felhom/dnsmasq: it RENDERS the
# config (Port from its claim) + the operator's authorized_keys, validates with `sshd -t`, and reloads
# (never restart-on-change [SF-2]). Both install SOURCES are the agent-owned staged files under
# StateDir; both DESTINATIONS are FIXED. `sshd -t/-T` are the validate/discover reads. The
# systemctl verbs are SCOPED to felhom-sshd only. reset-failed precedes a deliberate restart [SF-5].
# NOTHING here can touch the stock sshd, :22, or /etc/ssh.
Cmnd_Alias FELHOM_SSHD = \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/sshd_config /etc/felhom-sshd/sshd_config, \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op /etc/felhom-sshd/authorized_keys/felhom-op, \
/usr/sbin/sshd -t -f /var/lib/felhom-agent/felhom-sshd/sshd_config, \
/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config, \
/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config, \
/usr/bin/systemctl enable --now felhom-sshd, \
/usr/bin/systemctl reload felhom-sshd, \
/usr/bin/systemctl restart felhom-sshd, \
/usr/bin/systemctl reset-failed felhom-sshd, \
/usr/bin/wg show wg-felhom latest-handshakes
# PBS DR tier apply (slice 2, SPIKE-pbs-tier-provisioning-2026-07-10 §2b). Storage-entry
# lifecycle is /storage-ROOT-gated in the PVE API (spike Probe 1: create/modify/delete all check
# Datastore.Allocate on /storage), so the agent token cannot do it — this wrapper is the pinned
# vector. THE SET-ONLY LAW: the wrapper contains NO deletion path (entry deletion destroys the
# client encryption key = un-decryptable backups); verbs are create/reconcile/grant only. The
# token secret rides the wrapper's STDIN — sudo logs argv, so it must never appear here. The
# agent fine-validates every field (charset + descriptor equality) before exec; these globs are
# the coarse allowlist.
#
# `read` (R-39 leg b, agent v0.91.0) is the ONE added verb. It prints a token secret to stdout and
# performs no mutation. It exists because the agent writes that file through this wrapper but could
# never read it back (/etc/pve/priv is 0700 root:www-data), leaving its PBS verify loop permanently
# blind to an `applied`-but-401 tier. It is NOT a general file-read: the wrapper pins the directory
# and prefix-asserts the resolved path, and the id grammar admits no slash. The secret goes to
# STDOUT, never argv — sudo logs argv.
# E-2a: the backup-target storage shim. Creating a PVE storage needs Datastore.Allocate at /storage
# and the grant needs Permissions.Modify -- the agent holds NEITHER by design (blast-radius
# containment; Permissions.Modify would let it rewrite its own authority). Both live behind this
# fixed-vocabulary root shim instead, exactly like the mkfs and pbs-apply wrappers. The wrapper has
# NO storage-removal path, enforces is_mountpoint 1, and refuses a target on the root device.
Cmnd_Alias FELHOM_BACKUPTARGET = \
/usr/local/sbin/felhom-backup-target-apply create *, \
/usr/local/sbin/felhom-backup-target-apply grant *
Cmnd_Alias FELHOM_PBSDR = \
/usr/local/sbin/felhom-pbs-apply create *, \
/usr/local/sbin/felhom-pbs-apply reconcile *, \
/usr/local/sbin/felhom-pbs-apply grant *, \
/usr/local/sbin/felhom-pbs-apply read *
# OOB nft belt (TASK H1). The STATIC table `inet felhom_oob` is installed once by host-install; the
# agent mutates ONLY its two SETS — @operator_ips (the operator /32) + @ssh_port (the claimed port).
# SET ELEMENTS ONLY [trap 4]: NO `nft add rule`, NO `nft -f`, NO `flush ruleset/table` — a rule grant
# would let the agent firewall anything. The agent fine-validates every element (netip / int range)
# before exec; the trailing wildcards are the coarse allowlist (values only).
Cmnd_Alias FELHOM_OOB = \
/usr/sbin/nft list set inet felhom_oob operator_ips, \
/usr/sbin/nft list set inet felhom_oob ssh_port, \
/usr/sbin/nft flush set inet felhom_oob operator_ips, \
/usr/sbin/nft flush set inet felhom_oob ssh_port, \
/usr/sbin/nft add element inet felhom_oob operator_ips *, \
/usr/sbin/nft add element inet felhom_oob ssh_port *
# Escrow ceremony (controller-driven, TASK 2026-07-13; mechanics validated by
# SPIKE-controller-escrow-2026-07-13). ONE fixed argv — sudoers matches the argument vector
# byte-for-byte (spike §2.2): any alteration (value, extra flag, order, config path) is refused.
# --config pinned: env_reset strips FELHOM_AGENT_CONFIG and the pin closes alternate-config
# injection. R rides the subprocess stdout pipe only; sudo logs argv = secrets-free. The argv
# MUST stay byte-identical to escrow.CeremonyArgs() (internal/escrow/ceremony.go) — the
# capability manifest entry + TestManifestCoveredBySudoers lock the three copies together.
Cmnd_Alias FELHOM_ESCROW = \
/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload --output=json
# Node self-heal (CAMPAIGN-3 Part 6, F12-class defense in depth). The ONE fixed unit the appliance
# watchdog may (re)start when a boot leaves networking down — the exact command the morning recovery
# ran by hand after the F12 host loss. FIXED unit, no glob: this grant alone cannot harm — starting
# networking.service is precisely what the boot should have done. The remedy is ALSO code-gated on
# deployment_mode="appliance" (the Manager refuses to invoke it on a byo host); the sudoers grant is
# the coarse floor, the mode gate is the fine one.
Cmnd_Alias FELHOM_SELFHEAL = \
/usr/bin/systemctl start networking.service
# Guest-network watchdog (internal/guestnet, R-54). The guest's DHCP client is unsupervised — when it
# died on 2026-07-20 the box lost its address ~80 minutes later and went off the internet for 1h15m
# (INCIDENT-guest-dhclient-killed-2026-07-20). Four FIXED read vectors plus ONE fixed heal vector; the
# heal is the incident's own restored invocation, byte for byte. This is NOT a general `pct exec`: every
# argument after the numeric vmid is a literal, so the grant cannot be widened by anything the guest or
# the hub says. The address read is deliberately NOT duplicated here — it is already FELHOM_DNSMASQ's,
# and the same command must not be granted twice under two names.
Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct exec [0-9]* -- ip route show default, \
/usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \
/usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
/usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN
+111
View File
@@ -0,0 +1,111 @@
#!/bin/bash
#===============================================================================
# felhom-backup-target-apply — the ONLY path the felhom-agent sudoers permits for creating the
# whole-guest backup TARGET storage and granting the agent access to it (E-2a).
#
# WHY A WRAPPER AT ALL. Creating a PVE storage needs `Datastore.Allocate` at `/storage`, and the ACL
# grant needs `Permissions.Modify`. The agent holds NEITHER by design — its token is scoped per
# storage path for blast-radius containment, and `Permissions.Modify` would let it rewrite its own
# authority. Widening the PVE role to make the move possible would trade the entire containment model
# for one feature. So the privileged half lives here: a minimal, auditable root shim with a fixed
# vocabulary, exactly like felhom-mkfs-guarded and felhom-pbs-apply.
#
# THE NO-DELETE LAW (inherited from felhom-pbs-apply, same reasoning class). This wrapper contains NO
# storage-removal path of any kind. `pvesm remove` on a dir storage does not delete the archives, but
# it DOES silently orphan a configured backup tier, and a "cleanup" verb here would be reachable by
# any bug in the agent. Retiring a target is a deliberate operator op, not this tool. Grep-assertable;
# do not add one.
#
# THE TWO LAWS E-1 PAID FOR ON LIVE HARDWARE, both enforced here rather than trusted to the caller:
#
# F-1 the storage path must BE the drive's own mountpoint. A subdirectory fails the agent's
# exactMount check, so the target reports `disconnected` FOREVER and its durable id degrades
# off the filesystem UUID. Enforced: `mountpoint -q` must pass on the exact path given.
#
# F-2 --is_mountpoint 1 is not optional. Without it, an unplugged or late-mounting drive leaves a
# bare directory on the ROOT filesystem and vzdump writes the whole-guest backup onto the
# system drive — the exact device the whole change exists to escape — while PVE reports the
# storage `active` and advertises the root filesystem's free space. Proven live: the unguarded
# form had already created dump/ on pve-root. Hardcoded below; not a caller-supplied flag.
#
# Ops (all non-secret; nothing here touches a credential, so nothing arrives on stdin):
# create <id> <mountpoint>
# Create a `dir` storage with content=backup at <mountpoint>, is_mountpoint 1.
# IDEMPOTENT: an existing entry with the SAME path is accepted (re-run safe, and the
# installer re-run path depends on it). An existing entry with a DIFFERENT path is REFUSED
# — silently repointing a live backup target is the failure this whole arc closes.
# grant <id>
# The dual grant: FelhomAgentStore on /storage/<id> to the agent user AND token (privsep
# intersection — a token's rights are the intersection, so granting one is granting neither).
# Without it every backup 403s on first run (E-1 finding F-3, found by the first real backup).
#===============================================================================
set -euo pipefail
die() { echo "felhom-backup-target-apply: REFUSED: $*" >&2; exit 1; }
op="${1:-}"; id="${2:-}"
[[ -n "$op" && -n "$id" ]] || die "usage: felhom-backup-target-apply <create|grant> <storage-id> [mountpoint]"
# Storage id: PVE grammar, conservative. Also the ACL path component — no slashes possible.
[[ "$id" =~ ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ ]] || die "bad storage id ($id)"
STORECFG=/etc/pve/storage.cfg
# current_path_of <id> — the configured `path` of dir storage <id>, or "" when absent/not-a-dir.
current_path_of() {
awk -v want="dir: $1" '
$0 == want { found=1; next }
found && /^[a-z]+: / { exit }
found && $1 == "path" { print $2; exit }
' "$STORECFG" 2>/dev/null || true
}
case "$op" in
create)
[[ $# -eq 3 ]] || die "create takes <id> <mountpoint>"
mp="$3"
# Absolute, normalized, no traversal, no shell metacharacters. The value reaches pvesm and the
# filesystem, so it is validated here rather than assumed well-formed.
[[ "$mp" = /* ]] || die "mountpoint must be absolute ($mp)"
[[ "$mp" != *".."* ]] || die "mountpoint must not contain .. ($mp)"
[[ "$mp" =~ ^[A-Za-z0-9/_.-]+$ ]] || die "mountpoint has unexpected characters ($mp)"
[[ "$mp" != "/" ]] || die "refusing / as a backup target"
# F-1 + F-2, checked as one: the path must BE a mountpoint right now. A bare directory here is
# precisely the silent-retarget shape, and is_mountpoint would make PVE refuse it later anyway —
# better to refuse now, with a reason, than to create a storage that can never activate.
mountpoint -q "$mp" || die "$mp is not a mountpoint — the backup target must be the drive's OWN mountpoint (F-1), and an unmounted path would silently retarget onto the system drive (F-2)"
# Never the system disk: a target on the root filesystem is not drive-loss protection, it is the
# thing we are escaping. The root device and the candidate's device are compared, not their paths.
root_dev="$(findmnt -no SOURCE / 2>/dev/null || true)"
mp_dev="$(findmnt -no SOURCE "$mp" 2>/dev/null || true)"
[[ -n "$mp_dev" ]] || die "could not resolve the backing device of $mp"
[[ "$mp_dev" != "$root_dev" ]] || die "$mp is backed by the ROOT device ($root_dev) — a backup target there protects against corruption only, never drive loss"
existing="$(current_path_of "$id")"
if [[ -n "$existing" ]]; then
if [[ "$existing" == "$mp" ]]; then
echo "felhom-backup-target-apply: storage $id already exists at $mp — nothing to do (idempotent)" >&2
exit 0
fi
die "storage $id already exists at $existing — refusing to repoint it at $mp (a live backup target is never silently moved)"
fi
# is_mountpoint 1 is HARDCODED (F-2). content=backup only: this storage exists for vzdump archives
# and must never become a place guests are allocated on.
pvesm add dir "$id" --path "$mp" --content backup --is_mountpoint 1 >&2
echo "felhom-backup-target-apply: created dir storage $id at $mp (content=backup, is_mountpoint 1)" >&2
;;
grant)
[[ $# -eq 2 ]] || die "grant takes only <id>"
# BOTH, always. A privsep token's rights are the intersection of the user's and the token's ACLs,
# so granting one of the two grants nothing usable.
pveum acl modify "/storage/$id" --users felhom-agent@pve --roles FelhomAgentStore >&2
pveum acl modify "/storage/$id" --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore >&2
echo "felhom-backup-target-apply: granted FelhomAgentStore on /storage/$id (user + token)" >&2
;;
*)
die "unknown op ($op)"
;;
esac
+26 -17
View File
@@ -1,20 +1,30 @@
# felhom-agent local API — host firewall narrowing (doc 03 §6, slice 8A)
# felhom-agent local API — host firewall narrowing (doc 03 §6; R-50 island update 2026-07-25)
#
# Defense-in-depth for the per-guest local API (the controller→agent channel on the host
# bridge). The PER-GUEST BEARER TOKEN is the authorization gate; this firewall rule is an
# ADDITIONAL layer that limits who can even reach the port. The slice-8A spike found no rule
# was needed for reachability on the demo (PVE firewall off) — this narrows exposure so that
# only guests on the bridge subnet (not arbitrary LAN hosts) can open a connection.
# Defense-in-depth for the per-guest local API (the controller→agent channel). The PER-GUEST BEARER
# TOKEN + the served-leaf pin are the authorization gate; a firewall rule is only an ADDITIONAL layer
# limiting who can even open the port.
#
# The agent already binds the listener to the host BRIDGE IP (local_api.listen_addr), not
# 0.0.0.0. This file adds the subnet restriction. Apply it at HOST SETUP (it is a host-level
# packet-filter change, intentionally OUTSIDE the agent's 3-exception privileged fence — the
# agent never mutates the host firewall at runtime).
# === R-50 ISLAND INSTALL (the default on a fresh appliance) =================================
# The agent binds local_api.listen_addr on the HOST-INTERNAL island bridge — 169.254.253.1:8443 on
# vmbr9, a bridge with NO physical port (bridge-ports none). That bind is the security win:
# * Nothing listens on the LAN IP at all, so no LAN host (or off-site attacker on the LAN) can
# reach the local API — the LAN:8443 surface is CLOSED by the bind, not by a rule.
# * vmbr9 has no uplink, so 169.254.253.1:8443 is reachable ONLY from the one guest wired to the
# /30 (169.254.253.2) — the controller. The portless bridge is the isolation.
# So on an island install NO firewall rule is required for exposure; the topology provides it. If you
# want belt-and-suspenders, restrict the port to the island bridge (it changes nothing, since nothing
# off-bridge can route to a portless bridge anyway):
#
# Replace the bridge IP (192.168.0.162), port (8443), and the guest bridge subnet
# (192.168.0.0/24) with this host's values.
# nft add rule inet filter input iifname != "vmbr9" ip daddr 169.254.253.1 tcp dport 8443 drop
#
# Verify: from the guest, a TLS connect to 169.254.253.1:8443 succeeds; there is no LAN listener to
# probe (`ss -lnt 'sport = :8443'` shows only the island IP).
#
# === LEGACY LAN BIND (byo, --no-island, or an explicit --bridge-ip) =========================
# When the agent still binds a LAN bridge IP (e.g. 192.168.0.162:8443), the port is exposed to the
# whole LAN and the subnet-narrowing rule below is worth applying. Replace the bridge IP, port, and
# the guest bridge subnet with this host's values.
#
# ---------------------------------------------------------------------------------------------
# Option A — nftables (recommended on PVE 8/9; inet filter table). Insert ABOVE any accept:
#
# nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
@@ -22,13 +32,11 @@
# nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
# ip saddr 192.168.0.0/24 accept
#
# ---------------------------------------------------------------------------------------------
# Option B — iptables:
#
# iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -s 192.168.0.0/24 -j ACCEPT
# iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -j DROP
#
# ---------------------------------------------------------------------------------------------
# Option C — PVE host firewall (/etc/pve/nodes/<node>/host.fw), if the PVE firewall is enabled.
# Add under [RULES] (and ensure the firewall is enabled in cluster.fw / host.fw):
#
@@ -36,5 +44,6 @@
# IN ACCEPT -source 192.168.0.0/24 -dport 8443 -proto tcp -log nolog
# IN DROP -dport 8443 -proto tcp -log nolog
#
# Verify after applying: from a guest ON the bridge, a TLS connect to <bridge-ip>:8443 succeeds;
# from an OFF-bridge host it is refused/dropped. (The token + leaf-pin still gate the request.)
# Apply at HOST SETUP — a host-level packet-filter change, intentionally OUTSIDE the agent's
# 3-exception privileged fence (the agent never mutates the host firewall at runtime). The token +
# leaf-pin still gate the request regardless of which bind is in force.
+19
View File
@@ -0,0 +1,19 @@
# felhom-mgmt-watchdog.service (TASK G1) — install as /etc/systemd/system/felhom-mgmt-watchdog.service.
#
# LAYER 2 (auto-heal without login): a DUMB oneshot that runs /usr/local/sbin/felhom-mgmt-watchdog,
# triggered every ~60s by felhom-mgmt-watchdog.timer. It recreates a missing /run/sshd and clears a
# start-limited stock sshd — so a management-plane lockout self-corrects in ≤1 tick with NOBODY logged
# in and WITHOUT the felhom-agent binary running (the agent only OBSERVES/reports; the heal must not
# depend on it — SPIKE-felhom-sshd-2026-07-05 finding #9 / TASK G1 trap 1).
#
# CRITICAL: this unit MUST NOT declare `RuntimeDirectory=` — that directive (with value `sshd`) is the
# incident it exists to fix. It writes only to /run (dir + heal marker) as root; no state dir needed.
[Unit]
Description=Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)
# Ordering only — the timer drives cadence; no hard dependency so it runs even in degraded boots.
After=systemd-tmpfiles-setup.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/felhom-mgmt-watchdog
# Deliberately NO RuntimeDirectory=, NO PrivateTmp, NO sandboxing that would hide /run/sshd from it.
+68
View File
@@ -0,0 +1,68 @@
#!/bin/sh
# felhom-mgmt-watchdog — the DUMB, agent-INDEPENDENT management-plane healer (TASK G1).
#
# WHY THIS EXISTS: /run/sshd is OpenSSH's compiled-in privilege-separation directory, SHARED by
# every sshd on the host. If it goes missing, sshd's listener stays up but every new session RESETS
# right after SSH2_MSG_KEXINIT — a management-plane lockout on an otherwise-healthy box
# (SPIKE-felhom-sshd-2026-07-05 §8, caused live by a second unit's `RuntimeDirectory=sshd` cleanup).
# The tmpfiles.d entry (felhom-privsep.tmpfiles) PREVENTS the known cause; this script AUTO-HEALS any
# novel cause with ZERO login and ZERO dependency on the felhom-agent binary (so it self-corrects even
# when the agent is down — the whole point). It is run by felhom-mgmt-watchdog.timer every ~60s.
#
# CONTRACT (all four are load-bearing):
# 1. STAT-FIRST / IDEMPOTENT — a HEALTHY /run/sshd is NEVER touched (no mkdir/chmod/chown call), so
# steady state is zero mutation and the dir's mtime is preserved. Only a MISSING or
# wrong-mode/owner dir is corrected.
# 2. reset-failed the STOCK sshd ONLY when it is in the `failed` state (a start-limit lockout) —
# never otherwise, and NEVER `restart` it (mkdir alone restores it; PID stays, spike-proven).
# 3. HEAL MARKER — writes an RFC3339 UTC timestamp to $MARKER only on a real heal, so the agent can
# REPORT the condition to the hub (a recurring clobber must surface BEFORE it becomes a lockout).
# 4. NO `RuntimeDirectory=` anywhere in this feature (that IS the incident cause) — the unit that
# runs this script must not declare one either.
#
# Touches host /run + the stock sshd unit ONLY — no guests (pool-scoping is moot here). Fail-safe:
# any single step failing is logged and does not abort the others (`|| true`); the next tick retries.
set -u
PRIVSEP=/run/sshd
MARKER=/run/felhom-mgmt-watchdog.healed
SSHD_UNIT=ssh.service # Debian/PVE stock sshd unit name (sshd.service is an alias)
healed=0
log() { logger -t felhom-mgmt-watchdog "$*" 2>/dev/null || true; }
# 1. Privsep dir — stat-first: create only when missing; correct mode/owner only when wrong.
if [ ! -d "$PRIVSEP" ]; then
if mkdir -p "$PRIVSEP" && chown root:root "$PRIVSEP" && chmod 0755 "$PRIVSEP"; then
healed=1
log "recreated missing privsep dir $PRIVSEP (0755 root:root)"
else
log "ERROR: failed to recreate $PRIVSEP"
fi
else
mode=$(stat -c %a "$PRIVSEP" 2>/dev/null || echo "")
owner=$(stat -c %U:%G "$PRIVSEP" 2>/dev/null || echo "")
if [ "$mode" != "755" ]; then
chmod 0755 "$PRIVSEP" && healed=1 && log "corrected $PRIVSEP mode ($mode -> 755)"
fi
if [ "$owner" != "root:root" ]; then
chown root:root "$PRIVSEP" && healed=1 && log "corrected $PRIVSEP owner ($owner -> root:root)"
fi
fi
# 2. Stock sshd — clear a start-limit lockout ONLY when the unit is genuinely `failed`. Never restart
# it (unnecessary — a recreated privsep dir is picked up by the next fork; the listener never died).
if systemctl is-failed --quiet "$SSHD_UNIT" 2>/dev/null; then
if systemctl reset-failed "$SSHD_UNIT" 2>/dev/null; then
healed=1
log "reset-failed $SSHD_UNIT (was in the failed state)"
fi
fi
# 3. Heal marker — only on a real heal, so the agent reports the condition (hub raises a warning).
if [ "$healed" = "1" ]; then
date -u +%Y-%m-%dT%H:%M:%SZ > "$MARKER" 2>/dev/null || true
fi
exit 0
+18
View File
@@ -0,0 +1,18 @@
# felhom-mgmt-watchdog.timer (TASK G1) — install as /etc/systemd/system/felhom-mgmt-watchdog.timer.
#
# Drives felhom-mgmt-watchdog.service on a ~60s cadence (the auto-heal tick). OnBootSec fires shortly
# after boot (belt-and-suspenders with the tmpfiles layer); OnUnitActiveSec=60s gives the ≤1-tick
# heal budget the G1 acceptance drill measures. Persistent=true runs a missed tick immediately after a
# resume/late boot. The oneshot is idempotent, so a fast cadence never churns a healthy host.
[Unit]
Description=Felhom management-plane watchdog timer (~60s privsep-dir auto-heal tick)
[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=5s
Persistent=true
Unit=felhom-mgmt-watchdog.service
[Install]
WantedBy=timers.target
+94
View File
@@ -0,0 +1,94 @@
#!/bin/bash
#===============================================================================
# felhom-mkfs-guarded — the ONLY mkfs path the felhom-agent sudoers permits (Impl-1 Part B,
# SPIKE-drive-enrollment-2026-07-01 §SQ3). Defense-in-depth BELOW the agent: even a buggy or
# compromised agent cannot mkfs a catastrophic target through this — it re-checks, as root, the
# cheap catastrophic cases (OS/system disk, LVM physical volume, a foreign mount, a read-only
# device, and any LVM/ZFS/mdraid/LUKS/swap member signature — audit D1) and refuses.
#
# The agent's full unclaimed-disk filter (internal/storage/claim.go) is the PRIMARY guard; this
# wrapper is a deliberately minimal, auditable second gate. It is NOT the place for the full filter.
#
# Usage: felhom-mkfs-guarded <device> <fstype:ext4|xfs>
#===============================================================================
set -euo pipefail
die() { echo "felhom-mkfs-guarded: REFUSED: $*" >&2; exit 1; }
dev="${1:-}"; fstype="${2:-}"
[[ -n "$dev" && -n "$fstype" ]] || die "usage: felhom-mkfs-guarded <device> <fstype>"
# The device must be a REAL block-device node under /dev — no symlink (e.g. /dev/disk/by-*), no traversal.
[[ "$dev" == /dev/* ]] || die "device must be under /dev ($dev)"
[[ "$dev" != *..* ]] || die "path traversal ($dev)"
[[ -b "$dev" ]] || die "not a block device ($dev)"
[[ ! -L "$dev" ]] || die "device must be a real node, not a symlink ($dev)"
# Whole-disk of the target (a partition's parent, else the disk itself).
pk="$(lsblk -ndo PKNAME "$dev" 2>/dev/null || true)"
whole="$dev"; [[ -n "$pk" ]] && whole="/dev/$pk"
# 1) OS/system disk — does the target's whole-disk back /, /boot or /boot/efi?
while read -r src mnt _rest; do
case "$mnt" in
/|/boot|/boot/efi)
spk="$(lsblk -ndo PKNAME "$src" 2>/dev/null || true)"
swhole="$src"; [[ -n "$spk" ]] && swhole="/dev/$spk"
[[ "$swhole" == "$whole" || "$src" == "$dev" || "$src" == "$whole" ]] && die "system/OS disk ($dev backs $mnt)"
;;
esac
done < /proc/mounts
# 2) LVM physical volume anywhere on the target disk or its partitions. pvs is resolved by ABSOLUTE
# path (audit D1: `command -v pvs` silently skipped this check when pvs wasn't on the caller's
# PATH); if neither candidate exists, check 5's LVM2_member FSTYPE loop still catches a PV
# independently — pvs-absent never silently drops LVM detection.
pvsbin=""
for c in /usr/sbin/pvs /sbin/pvs; do
[[ -x "$c" ]] && { pvsbin="$c"; break; }
done
if [[ -n "$pvsbin" ]]; then
while read -r pv; do
pv="${pv//[[:space:]]/}"; [[ -z "$pv" ]] && continue
pvpk="$(lsblk -ndo PKNAME "$pv" 2>/dev/null || true)"
pvwhole="$pv"; [[ -n "$pvpk" ]] && pvwhole="/dev/$pvpk"
[[ "$pvwhole" == "$whole" ]] && die "device holds an LVM physical volume ($pv)"
done < <("$pvsbin" --noheadings -o pv_name 2>/dev/null || true)
fi
# 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under
# /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed.
while read -r mp; do
[[ -z "$mp" ]] && continue
case "$mp" in
/mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
*) die "device (or a partition) is mounted at $mp ($dev)" ;;
esac
done < <(lsblk -nro MOUNTPOINT "$whole" 2>/dev/null || true)
# 4) read-only device (audit D1): a device the kernel marks RO is never a formattable data disk.
wbase="${whole#/dev/}"
rof="/sys/block/$wbase/ro"
if [[ -r "$rof" ]]; then
ro="$(cat "$rof" 2>/dev/null || true)"
[[ "$ro" == "1" ]] && die "read-only device ($whole)"
fi
# 5) member/active FSTYPEs anywhere on the target disk or its partitions (audit D1). Mirrors
# claim.go memberFSTypes exactly: a member of LVM/ZFS/mdraid/LUKS or active-swap signature is
# always a claim, never a plain formattable data disk. This also independently catches an LVM PV
# when pvs is not installed (check 2's belt-and-suspenders).
while read -r fst; do
[[ -z "$fst" ]] && continue
case "$fst" in
LVM2_member|zfs_member|linux_raid_member|crypto_LUKS|swap)
die "device holds a $fst signature ($whole)" ;;
esac
done < <(lsblk -nro FSTYPE "$whole" 2>/dev/null || true)
# Passed the catastrophic checks → format. exec so the mkfs exit status is the wrapper's.
case "$fstype" in
ext4) exec /usr/sbin/mkfs.ext4 -F "$dev" ;;
xfs) exec /usr/sbin/mkfs.xfs -f "$dev" ;;
*) die "unsupported fstype ($fstype)" ;;
esac
+19
View File
@@ -0,0 +1,19 @@
# felhom-oob-nft.service (TASK H1) — install as /etc/systemd/system/felhom-oob-nft.service.
#
# Loads the STATIC felhom_oob belt table on boot (the agent then fills its sets each tick). Oneshot +
# RemainAfterExit so `systemctl status` reflects "loaded". Independent of pve-firewall/nftables.service
# (the table is a self-contained `inet felhom_oob`, additive — it never flushes the ruleset).
[Unit]
Description=Felhom OOB belt — load the static felhom_oob nft table (H1)
After=network-pre.target
Before=network.target felhom-agent.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/nft -f /etc/felhom-oob.nft
# On stop, remove the table (best-effort) so an uninstall/disable leaves nft clean.
ExecStop=-/usr/sbin/nft delete table inet felhom_oob
[Install]
WantedBy=multi-user.target
+37
View File
@@ -0,0 +1,37 @@
#!/usr/sbin/nft -f
# felhom OOB belt — STATIC table (TASK H1). Installed by host-install; loaded on boot by
# felhom-oob-nft.service. The agent mutates ONLY the two SETS (@operator_ips, @ssh_port) via the
# narrow FELHOM_OOB sudoers — it NEVER touches the rules [trap 4], so the enforcement semantics are
# fixed here and cannot be changed by the agent.
#
# Enforcement (scoped to iifname "wg-felhom" — the offsite tunnel — so :22 and every other interface
# are UNTOUCHED [SF-3/OF-5]):
# - ESTABLISHED,RELATED replies always pass (covers the PBS path + reply traffic; PMTU-safe).
# - felhom-sshd's port is reachable from the operator /32 (in @operator_ips) over the tunnel ONLY.
# - any other tunnel source to that port is DROPPED at the host (defense-in-depth vs the endpoint).
# - the port is unreachable OFF the tunnel entirely.
# Empty sets (before the agent's first sync) = the port rules match nothing = no enforcement yet
# (felhom-sshd is not started until the agent renders its config either). priority -5 sits just above
# the default; policy accept so this table only ever ADDS drops for the felhom-sshd port.
#
# Idempotent load: create-then-delete-then-define (a re-run/boot yields a clean table with EMPTY sets;
# the agent refills them within one tick).
table inet felhom_oob
delete table inet felhom_oob
table inet felhom_oob {
set operator_ips {
type ipv4_addr
}
set ssh_port {
type inet_service
}
chain input {
type filter hook input priority -5; policy accept;
ct state established,related accept
iifname "wg-felhom" tcp dport @ssh_port ip saddr @operator_ips accept
iifname "wg-felhom" tcp dport @ssh_port drop
tcp dport @ssh_port iifname != "wg-felhom" drop
}
}
+23
View File
@@ -0,0 +1,23 @@
# felhom-op operator sudoers (TASK H1) — install as /etc/sudoers.d/felhom-op (0440 root:root).
#
# The DEFAULT operator identity (login via felhom-sshd only; key in /etc/felhom-sshd/authorized_keys/
# felhom-op, outside ~/.ssh, so the customer's sshd never honours it [SF-3]). felhom-op is a normal
# login user with NO ambient privilege — it escalates ONLY through these explicit repair verbs, so a
# leaked operator key's blast radius is this verb set, not the box (locked decision §6). Deep repairs
# that genuinely need root (pct, deeper systemctl) remain available via `root` as a tunnel-gated
# break-glass account (PermitRootLogin prohibit-password on felhom-sshd), but the day-to-day identity
# is felhom-op.
Cmnd_Alias FELHOM_OP_REPAIR = \
/usr/bin/mkdir -p /run/sshd, \
/usr/bin/systemctl reset-failed wg-quick@wg-felhom, \
/usr/bin/systemctl restart wg-quick@wg-felhom, \
/usr/bin/systemctl start felhom-agent, \
/usr/bin/systemctl restart felhom-agent, \
/usr/bin/systemctl reset-failed felhom-sshd, \
/usr/bin/systemctl restart felhom-sshd, \
/usr/sbin/pct list, \
/usr/sbin/pct start [0-9]*, \
/usr/sbin/pct stop [0-9]*, \
/usr/sbin/pct unlock [0-9]*
felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR
+159
View File
@@ -0,0 +1,159 @@
#!/bin/bash
#===============================================================================
# felhom-pbs-apply — the ONLY storage-entry path the felhom-agent sudoers permits for the PBS DR
# tier (slice 2; SPIKE-pbs-tier-provisioning-2026-07-10 §2b). The guarded-mkfs shape: the agent's
# in-process validation is the PRIMARY gate (descriptor-field charset + equality checks BEFORE
# exec); this wrapper is the minimal, auditable second gate as root.
#
# THE SET-ONLY LAW (spike §4, data-loss class): `pvesm` entry deletion DESTROYS the client
# encryption key file (<id>.enc = K) — un-decryptable backups. This wrapper therefore contains
# NO deletion path of any kind, and a re-apply is `pvesm set`-only. Grep-assertable; do not add
# a "cleanup" verb here, ever. Deprovision is a deliberate future operator op, not this tool.
#
# SECRET DISCIPLINE (spike §2b): sudo logs its full argv to auth.log → the PBS token secret
# arrives on STDIN, never as an argument to this wrapper. Inside, it is passed to pvesm's
# --password (root-local, transient ps exposure — the accepted spike posture); it is never
# echoed, never written anywhere except by pvesm itself (the 0600 .pw store).
#
# Ops (non-secret args on argv):
# create <id> <server> <datastore> <namespace> <token-id> <fingerprint> <secret-dir>
# secret on stdin (required). Creates the pbs entry with --encryption-key autogen
# (K born at /etc/pve/priv/storage/<id>.enc), then places .pw/.enc copies in
# <secret-dir> when it differs (the §4b WARN-fix dir; escrow-create's PBSEncKeyPath
# must find K there — spike §4 flag).
# reconcile <id> <server> <namespace> <token-id> <fingerprint> <secret-dir>
# secret on stdin (optional; empty = no credential change). `pvesm set` ONLY:
# server/fingerprint (+ --password when a secret is fed, e.g. after a hub re-issue).
# NOTE datastore is deliberately NOT settable, and namespace/token-id are accepted
# for validation parity but NOT applied — tenancy identity is adopt-only (the
# demo's live entry must never be repointed at a different namespace).
# read <id> <secret-dir>
# R-39 leg (b): print the storage's token secret to STDOUT and nothing else.
# The non-root agent WRITES this file through this wrapper but could never read it
# back (/etc/pve/priv is 0700 root:www-data and there is no read verb), so its
# 15-minute PBS verify loop was permanently blind to the one failure it exists to
# catch — an `applied` tier authenticating 401. This verb is that missing read.
# It is deliberately the narrowest thing that works: no network, no mutation, no
# logging of the value, one file, prefix-asserted under the given secret dir.
#
# grant <id>
# The Part-0-evidenced dual-grant: FelhomAgentStore on /storage/<id> to the agent
# user AND token (privsep intersection). Datastore.Audit reads ride the base role.
#===============================================================================
set -euo pipefail
die() { echo "felhom-pbs-apply: REFUSED: $*" >&2; exit 1; }
op="${1:-}"; id="${2:-}"
[[ -n "$op" && -n "$id" ]] || die "usage: felhom-pbs-apply <create|reconcile|grant|read> <storage-id> ..."
# Storage id: PVE grammar, conservative. Also the ACL path component — no slashes possible.
[[ "$id" =~ ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ ]] || die "bad storage id ($id)"
STORECFG=/etc/pve/storage.cfg
PRIVDIR=/etc/pve/priv/storage
entry_exists() { grep -Eq "^pbs: ${id}\$" "$STORECFG"; }
val_server() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9.-]{0,62}$ ]] || die "bad server ($1)"; }
val_datastore() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$ ]] || die "bad datastore ($1)"; }
val_ns() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$ ]] || die "bad namespace ($1)"; }
val_tok() { [[ "$1" =~ ^[A-Za-z0-9_.-]+@[A-Za-z0-9]+![A-Za-z0-9_.-]+$ ]] || die "bad token id ($1)"; }
val_fp() { [[ "$1" =~ ^([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}$ ]] || die "bad fingerprint"; }
val_sdir() {
case "$1" in
/etc/pve/priv/storage) : ;;
/var/lib/felhom-agent/*) [[ "$1" != *..* ]] || die "secret-dir traversal" ;;
*) die "secret-dir must be $PRIVDIR or under /var/lib/felhom-agent ($1)" ;;
esac
}
read_secret() { # → SECRET (may be empty when optional)
local s
s=$(head -c 256 || true)
s="${s%$'\n'}"; s="${s%$'\r'}"
printf '%s' "$s"
}
place_copies() { # secret-dir — the §4b WARN-fix placement (non-root agent can't read /etc/pve/priv)
local sdir="$1"
[[ "$sdir" == "$PRIVDIR" ]] && return 0
install -d -o felhom-agent -g felhom-agent -m 0700 "$sdir"
[[ -f "$PRIVDIR/$id.pw" ]] && install -o felhom-agent -g felhom-agent -m 0600 "$PRIVDIR/$id.pw" "$sdir/$id.pw"
# K's copy: escrow-create stats PBSEncKeyPath(<secret-dir>/<id>.enc) — the spike §4 flag.
[[ -f "$PRIVDIR/$id.enc" ]] && install -o root -g felhom-agent -m 0640 "$PRIVDIR/$id.enc" "$sdir/$id.enc"
return 0
}
case "$op" in
create)
[[ $# -eq 8 ]] || die "create needs 7 args: <id> <server> <datastore> <namespace> <token-id> <fingerprint> <secret-dir>"
server="$3"; datastore="$4"; ns="$5"; tok="$6"; fp="$7"; sdir="$8"
val_server "$server"; val_datastore "$datastore"; val_ns "$ns"; val_tok "$tok"; val_fp "$fp"; val_sdir "$sdir"
entry_exists && die "entry $id already exists (reconcile is the re-apply path — set-only law)"
SECRET=$(read_secret)
[[ -n "$SECRET" ]] || die "create requires the token secret on stdin"
pvesm add pbs "$id" \
--server "$server" --datastore "$datastore" --namespace "$ns" \
--username "$tok" --password "$SECRET" --fingerprint "$fp" \
--content backup --encryption-key autogen >&2
[[ -f "$PRIVDIR/$id.enc" ]] || die "pvesm add succeeded but K ($PRIVDIR/$id.enc) was not born"
place_copies "$sdir"
echo "felhom-pbs-apply: created $id (K born; encryption-key autogen)" >&2
;;
reconcile)
[[ $# -eq 7 ]] || die "reconcile needs 6 args: <id> <server> <namespace> <token-id> <fingerprint> <secret-dir>"
server="$3"; ns="$4"; tok="$5"; fp="$6"; sdir="$7"
val_server "$server"; val_ns "$ns"; val_tok "$tok"; val_fp "$fp"; val_sdir "$sdir"
entry_exists || die "entry $id does not exist (create is the fresh path)"
SECRET=$(read_secret)
# R-39 (2026-07-18): NEVER pass --server to `pvesm set`. PVE treats `server` as a CREATE-ONLY
# parameter and rejects the ENTIRE call — "can't change value of fixed parameter 'server'" —
# even when the value is byte-identical to the stored one. That made every reconcile exit 255,
# so each hub-re-issued one-time secret was consumed-then-burned and the tier stayed pinned to
# a revoked credential (401 forever). Proven live on the N100 demo host: `pvesm set <id>
# --server <same> --fingerprint <same>` -> rejected; the same call without --server -> rc 0.
# The server address is immutable by construction (relocating a PBS endpoint needs a fresh
# create), so there is nothing here to reconcile. Guarded by
# TestReconcileNeverPassesServerToPvesmSet.
args=(--fingerprint "$fp")
[[ -n "$SECRET" ]] && args+=(--password "$SECRET")
pvesm set "$id" "${args[@]}" >&2
place_copies "$sdir"
echo "felhom-pbs-apply: reconciled $id (set-only; tenancy identity untouched)" >&2
;;
read)
# R-39(b): the missing read path. Prints the secret to STDOUT, nothing else — no stderr note (it
# would be the only verb whose success line could be confused with the value), no mutation.
#
# Traversal is refused three times over, because this is the one verb that EXFILTRATES a file and
# its argv is attacker-shaped if the agent is ever compromised:
# 1. `id` already matched ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ at the top — it cannot start with a dot
# and cannot contain a slash, so "../../etc/shadow" never reaches here;
# 2. val_sdir pins the directory to PRIVDIR or under /var/lib/felhom-agent, rejecting "..";
# 3. the RESOLVED path is prefix-asserted under that directory below, so even a future change to
# either grammar cannot walk out.
[[ $# -eq 3 ]] || die "read needs 2 args: <id> <secret-dir>"
sdir="$3"
val_sdir "$sdir"
target="$sdir/$id.pw"
# Belt: resolve and re-check the prefix (guards a symlinked <id>.pw pointing outside the dir).
resolved=$(readlink -f -- "$target" 2>/dev/null || true)
[[ -n "$resolved" ]] || die "secret file not found ($target)"
case "$resolved" in
"$sdir"/*) : ;;
*) die "resolved secret path escapes the secret dir" ;;
esac
[[ -f "$resolved" ]] || die "secret file not found ($target)"
cat -- "$resolved"
;;
grant)
[[ $# -eq 2 ]] || die "grant takes only <id>"
pveum acl modify "/storage/$id" --users felhom-agent@pve --roles FelhomAgentStore >&2
pveum acl modify "/storage/$id" --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore >&2
echo "felhom-pbs-apply: granted FelhomAgentStore on /storage/$id (user + token)" >&2
;;
*)
die "unknown op ($op)"
;;
esac
+15
View File
@@ -0,0 +1,15 @@
# felhom privsep-dir persistence (TASK G1) — install as /etc/tmpfiles.d/felhom-privsep.conf.
#
# LAYER 1 (prevent): make OpenSSH's shared privilege-separation directory /run/sshd boot-persistent
# and owned by NO systemd unit's lifecycle. This closes the exact incident cause from
# SPIKE-felhom-sshd-2026-07-05 §8: a second sshd unit declaring `RuntimeDirectory=sshd` had systemd
# REMOVE the shared /run/sshd on that unit's stop/failure, taking the stock sshd on :22 down with it
# (sessions reset right after SSH2_MSG_KEXINIT). A tmpfiles.d entry recreates the dir at every boot
# independently of any unit, so no unit's RuntimeDirectory cleanup can be the sole owner.
#
# systemd-tmpfiles is idempotent: `systemd-tmpfiles --create` re-run is a no-op on an existing,
# correct dir (it only creates/fixes, never churns). Complemented at runtime by the
# felhom-mgmt-watchdog timer (layer 2), which re-heals a dir removed AFTER boot.
#
# Type d = create the directory if absent (leaves an existing one, only fixing mode/owner).
d /run/sshd 0755 root root -
+139
View File
@@ -0,0 +1,139 @@
#!/bin/sh
# felhom-selfupdate-guarded — the ROOT half of the agent's A/B self-update (TASK D1).
#
# Install as /usr/local/sbin/felhom-selfupdate-guarded (0755 root:root). The non-root agent invokes
# `apply`/`commit` via `sudo -n` (FELHOM_SELFUPDATE alias); `rollback` is ALSO the ExecStart of
# felhom-agent-rollback.service — the OnFailure= target that auto-reverts a crash-looping update.
#
# Design provenance: SPIKE-agent-selfupdate-2026-07-05 (its SF-findings are cited inline). The core
# principle: the thing that performs rollback is never the thing being updated — this wrapper +
# systemd change almost never; the agent binary is what flips.
#
# Trust model: the agent verifies the download against the OPERATOR-SIGNED sha before staging; this
# wrapper RE-verifies the same sha as root (defense in depth — the sudoers arg glob is coarse, the
# sha check here is the real gate). Path confinement: apply only ever reads from the agent's own
# staging dir and only ever writes the fixed live path + its siblings. NO env-overridable paths —
# path-fixedness IS the security property (a test-mode override would be a root escalation hole).
#
# Verbs:
# apply <staged> <sha256> stage-verify → .prev → atomic flip → pending marker → detached restart
# commit clear the pending marker (idempotent; .prev retained as a manual net)
# rollback pending-guarded revert to .prev + restart (no pending → exit 0 no-op)
set -u
BIN=/usr/local/bin/felhom-agent
PREV=$BIN.prev
STAGING=/var/lib/felhom-agent/selfupdate
PENDING=$STAGING/pending.json
UNIT=felhom-agent.service
# Every refusal/decision goes to stderr AND the journal (strict rule 10).
log() { echo "felhom-selfupdate-guarded: $*" >&2; logger -t felhom-selfupdate-guarded -- "$*" 2>/dev/null || true; }
case "${1:-}" in
apply)
staged=${2:-}; want=${3:-}
# [SF-7] entry sweep: a kill between staging-copy and mv leaves an orphaned temp — harmless,
# but sweep it so temps can never accumulate.
rm -f "$BIN".new.*
if [ -z "$staged" ] || [ -z "$want" ]; then
log "refusing apply: usage: apply <staged> <sha256>"
exit 2
fi
# Root-side path confinement: the staged binary MUST live in the agent's staging dir.
case "$staged" in
"$STAGING"/*) ;;
*) log "refusing apply: staged path outside $STAGING: $staged"; exit 1 ;;
esac
case "$staged" in
*..*) log "refusing apply: staged path contains '..'"; exit 1 ;;
esac
[ -f "$staged" ] || { log "refusing apply: staged file missing: $staged"; exit 1; }
# The sha must be 64 lowercase hex chars — anything else is refused before any hashing.
case "$want" in
*[!0-9a-f]*) log "refusing apply: sha256 is not lowercase hex"; exit 1 ;;
esac
[ "${#want}" -eq 64 ] || { log "refusing apply: sha256 must be 64 hex chars (got ${#want})"; exit 1; }
# [SF-7] sha-verify FIRST — before .prev, before any mutation (spike S3a companion ordering).
got=$(sha256sum "$staged" | awk '{print $1}')
if [ "$got" != "$want" ]; then
log "refusing apply: sha mismatch (got=$got want=$want)"
exit 1
fi
# Same-fs assert (§8): the atomic-rename guarantee only holds within one filesystem.
if [ "$(stat -c %d "$staged")" != "$(stat -c %d /usr/local/bin)" ]; then
log "refusing apply: staging and /usr/local/bin are on different filesystems — atomic rename impossible"
exit 1
fi
old_ver=$("$BIN" --version 2>/dev/null | awk '{print $2}')
[ -n "$old_ver" ] || old_ver=unknown
# The staged filename is felhom-agent-<version> (executor contract) — version without executing.
new_ver=$(basename "$staged"); new_ver=${new_ver#felhom-agent-}
cp -p "$BIN" "$PREV" || { log "apply failed: cannot snapshot current binary to .prev"; exit 1; }
# Stage a root-owned 0755 copy next to the live path, then ATOMIC same-fs rename.
if ! cp "$staged" "$BIN.new.$$" || ! chmod 0755 "$BIN.new.$$" || ! chown root:root "$BIN.new.$$"; then
rm -f "$BIN.new.$$"; log "apply failed: staging copy"; exit 1
fi
mv "$BIN.new.$$" "$BIN" || { rm -f "$BIN.new.$$"; log "apply failed: atomic rename"; exit 1; }
# Pending marker: written AFTER the flip — its existence means "an update is awaiting commit",
# which is exactly the rollback unit's trigger condition.
printf '{"old_version":"%s","new_version":"%s","sha256":"%s","applied_at":"%s"}\n' \
"$old_ver" "$new_ver" "$want" "$(date -Is)" > "$PENDING" \
|| { log "apply failed: cannot write pending marker"; exit 1; }
# [SF-4/5] deliberate restarts consume start-limit budget — clear the counter first.
systemctl reset-failed "$UNIT" 2>/dev/null || true
# [SF-6] the spike's S2b winner, verbatim: detached transient timer OUTSIDE the agent's cgroup,
# so the sudo/agent caller survives to log the handoff and the restart cannot be torn down
# by its own requester dying.
systemd-run --on-active=2s --timer-property=AccuracySec=100ms systemctl restart "$UNIT" \
|| { log "apply: flip done but detached restart scheduling FAILED — restart $UNIT manually"; exit 1; }
log "applied $new_ver (prev $old_ver, sha $want); detached restart scheduled"
;;
commit)
if [ ! -f "$PENDING" ]; then
log "commit: no pending — no-op"
exit 0
fi
# .prev is deliberately RETAINED (spike S3d) — a manual safety net until the next apply.
rm -f "$PENDING" || { log "commit failed: cannot remove pending marker"; exit 1; }
log "committed (pending cleared, .prev retained)"
;;
rollback)
# [SF-1] On systemd 257 OnFailure= fires on EVERY crash, so this verb runs MANY times per
# incident — the pending-guard makes every fire after the first a harmless no-op, and makes a
# crash with NO update in flight touch nothing at all (spike S1d/S3e).
if [ ! -f "$PENDING" ]; then
log "rollback: no pending update — no-op"
exit 0
fi
[ -f "$PREV" ] || { log "rollback FAILED: pending exists but no .prev binary"; exit 1; }
rm -f "$BIN".new.*
if ! cp "$PREV" "$BIN.new.$$" || ! chmod 0755 "$BIN.new.$$" || ! chown root:root "$BIN.new.$$"; then
rm -f "$BIN.new.$$"; log "rollback FAILED: staging copy"; exit 1
fi
mv "$BIN.new.$$" "$BIN" || { rm -f "$BIN.new.$$"; log "rollback FAILED: atomic rename"; exit 1; }
# Clear pending BEFORE the restart: once the binary is reverted, later OnFailure fires must
# no-op (the guard above) instead of re-copying .prev forever.
rm -f "$PENDING"
# [SF-4/5] the crash burst has been eating the start-limit budget — reset before starting.
systemctl reset-failed "$UNIT" 2>/dev/null || true
# Direct restart is correct HERE: this caller is the rollback oneshot, OUTSIDE the agent cgroup.
systemctl restart "$UNIT" || { log "rollback: binary reverted but restart FAILED"; exit 1; }
log "rolled back to previous binary and restarted $UNIT"
;;
*)
log "usage: felhom-selfupdate-guarded apply <staged> <sha256> | commit | rollback"
exit 2
;;
esac
+28
View File
@@ -0,0 +1,28 @@
# felhom-sshd.service (TASK H1) — install as /etc/systemd/system/felhom-sshd.service.
#
# The dedicated OOB sshd instance: a SECOND sshd on a claimed non-22 port, COEXISTING with the
# customer's/stock sshd on :22 (never touched). Config is AGENT-RENDERED at /etc/felhom-sshd/sshd_config
# (Port from the agent's claim); the agent reloads on change (never restart-on-change [SF-2]).
#
# CRITICAL [SF-1]: this unit MUST NOT declare `RuntimeDirectory=` — that directive (value `sshd`)
# removed the SHARED /run/sshd and broke the stock sshd live (SPIKE-felhom-sshd §8). /run/sshd is
# guaranteed by G1 (tmpfiles + watchdog); the ExecStartPre mkdir is a harmless extra guarantee. The
# host-install RuntimeDirectory guard will REJECT this unit if the directive is ever added.
[Unit]
Description=Felhom OOB sshd (dedicated instance, agent-managed; H1)
After=network.target
[Service]
Type=notify
ExecStartPre=-/usr/bin/mkdir -p /run/sshd
ExecStartPre=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config
ExecStart=/usr/sbin/sshd -D -f /etc/felhom-sshd/sshd_config
# Config-change path: validate then HUP — a bad reload is REFUSED and the running daemon SURVIVES
# [SF-2]. The agent always writes→sshd -t→reload; it never `restart`s on a config change.
ExecReload=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,243 @@
package backup
import (
"bytes"
"context"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// F-CRIT-2 (Campaign 8): a failed backup must not look like a fresh one.
//
// Every fixture below is a VERBATIM shape captured from the live PVE API on 2026-07-28
// (`pvesh get /nodes/<node>/storage/<store>/content`), not a hand-invented struct. That matters:
// the `unparseable` path in this package went untested for months behind a JSON shape that did not
// match production, and the whole point of this fix is that presence != validity.
// phantomEntry is the artefact a PBS daemon killed mid-upload leaves behind: listed as a restorable
// backup, 1 byte, NEWEST, and carrying no `verification`/`encrypted`/`notes` at all because it has
// no manifest (`index.json.blob` is absent on disk).
func phantomEntry() proxmox.StorageContent {
return proxmox.StorageContent{
VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z",
Content: "backup",
Format: "pbs-ct",
Size: 1,
CTime: 1785216674,
VMID: 9201,
}
}
// goodPBSEntry is a real, complete offsite snapshot (demo-hp, 2026-07-28T03:40:42Z).
func goodPBSEntry() proxmox.StorageContent {
return proxmox.StorageContent{
VolID: "felhom-pbs:backup/ct/9201/2026-07-28T03:40:42Z",
Content: "backup",
Format: "pbs-ct",
Size: 4353457559,
CTime: 1785210042,
VMID: 9201,
}
}
// goodLocalEntry is a real, complete LOCAL vzdump (demo-hp). Note it legitimately has no
// `verification` and no `encrypted` on the wire — a dir storage has no such concept — which is
// exactly why those fields must never be used as completeness discriminators.
func goodLocalEntry() proxmox.StorageContent {
return proxmox.StorageContent{
VolID: "local:backup/vzdump-lxc-9201-2026_07_28-07_29_54.tar.zst",
Content: "backup",
Format: "tar.zst",
Size: 1590431865,
CTime: 1785216594,
VMID: 9201,
}
}
func runnerWithContent(t *testing.T, buf *bytes.Buffer, content []proxmox.StorageContent) *BackupRunner {
t.Helper()
lg := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
return NewBackupRunner(&fakeBackupAPI{content: content}, "felhom-pbs", proxmox.ModeSnapshot, "", "", lg)
}
// Group A — the phantom must NOT set tier freshness, even though it is the newest entry.
//
// RED-PROOF: restore the old predicate in NewestArchiveTime
// (`if e.Content == "backup" && e.VMID == vmid && e.CTime > best`) → the phantom's ctime
// (1785216674) wins over the good snapshot's (1785210042) and this test fails with
// "got 1785216674, want 1785210042" — i.e. the exact F-CRIT-2 defect.
func TestNewestArchiveTime_PhantomIsNotCounted(t *testing.T) {
var buf bytes.Buffer
// phantom deliberately listed FIRST and is also the newest by ctime.
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
got, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatalf("NewestArchiveTime: %v", err)
}
if !found {
t.Fatal("found=false — the GOOD snapshot must still be counted; rejecting everything is the thrash path")
}
if got.Unix() != goodPBSEntry().CTime {
t.Errorf("freshness came from the wrong entry: got ctime %d, want %d (the good snapshot)", got.Unix(), goodPBSEntry().CTime)
}
if got.Unix() == phantomEntry().CTime {
t.Error("the 1-byte manifest-less phantom set tier freshness — this is F-CRIT-2")
}
}
// Group A — with ONLY a phantom present the tier must report "no backup", not a fresh one.
// That is what lets the controller see age_state=absent and fire its first-backup valve.
func TestNewestArchiveTime_OnlyPhantomReportsNotFound(t *testing.T) {
var buf bytes.Buffer
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry()})
_, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatalf("NewestArchiveTime: %v", err)
}
if found {
t.Error("found=true with only a phantom present — the tier would report fresh and go silent for a full cadence")
}
}
// Group B — THE SCENARIO-D GUARD. A valid snapshot on EITHER tier must still be counted.
//
// This is what makes Group A safe. A filter that is too aggressive does not merely lose safety
// margin: the tier reports absent on every poll, backs up every cycle, and the R-88 breaker cannot
// save it because those backups SUCCEED. That is a continuous multi-GB write loop across the fleet.
//
// RED-PROOF: make archivePlausiblyComplete return `false, "reject everything"` unconditionally →
// both subtests fail with found=false.
func TestNewestArchiveTime_ValidSnapshotsAreStillCounted(t *testing.T) {
for _, tc := range []struct {
name string
entry proxmox.StorageContent
}{
{"pbs offsite (has verification+encrypted on the wire)", goodPBSEntry()},
{"local dir vzdump (has NEITHER verification NOR encrypted — and must still count)", goodLocalEntry()},
} {
t.Run(tc.name, func(t *testing.T) {
var buf bytes.Buffer
r := runnerWithContent(t, &buf, []proxmox.StorageContent{tc.entry})
got, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatalf("NewestArchiveTime: %v", err)
}
if !found {
t.Fatalf("a REAL %s backup was rejected — this is the backup-thrash path, not extra safety", tc.name)
}
if got.Unix() != tc.entry.CTime {
t.Errorf("got ctime %d, want %d", got.Unix(), tc.entry.CTime)
}
if strings.Contains(buf.String(), "INCOMPLETE archive") {
t.Errorf("a valid archive was announced as incomplete:\n%s", buf.String())
}
})
}
}
// Group B — the smallest REAL backup measured anywhere on the fleet (612,397,450 B, a guest-9100
// vzdump) must clear the floor with room to spare. If someone ever raises
// minPlausibleArchiveBytes past this, that is the fleet-thrash bug and this test is the tripwire.
func TestMinPlausibleArchiveBytes_LeavesHeadroomBelowTheSmallestRealBackup(t *testing.T) {
const smallestObservedRealBackup int64 = 612397450 // fleet survey 2026-07-28
if minPlausibleArchiveBytes >= smallestObservedRealBackup {
t.Fatalf("floor %d B is not below the smallest real backup ever observed (%d B) — this WILL reject real archives",
minPlausibleArchiveBytes, smallestObservedRealBackup)
}
if ratio := smallestObservedRealBackup / minPlausibleArchiveBytes; ratio < 100 {
t.Errorf("floor %d B leaves only %dx headroom below the smallest real backup (%d B) — too tight",
minPlausibleArchiveBytes, ratio, smallestObservedRealBackup)
}
}
// Group C — UNDECIDABLE ⇒ NOT COUNTED (the fail-safe direction).
//
// A zero/absent size is not evidence of a good backup; it is absence of evidence. Erring toward
// "not fresh" costs one extra backup. Erring the other way is F-CRIT-2.
//
// RED-PROOF: flip the comparison in archivePlausiblyComplete to `e.Size > minPlausibleArchiveBytes
// || e.Size == 0` (i.e. treat unknown as complete) → the size-0 case reports ok=true and this fails.
func TestArchivePlausiblyComplete_UndecidableIsNotCounted(t *testing.T) {
for _, tc := range []struct {
name string
size int64
}{
{"the observed phantom", 1},
{"absent size field (unmarshals to 0)", 0},
{"just under the floor", minPlausibleArchiveBytes - 1},
} {
t.Run(tc.name, func(t *testing.T) {
e := phantomEntry()
e.Size = tc.size
ok, why := archivePlausiblyComplete(e)
if ok {
t.Errorf("size %d counted as a complete backup — undecidable must fail safe", tc.size)
}
if why == "" {
t.Error("rejection carried no reason — a silent rejection is a new quiet path")
}
})
}
if ok, why := archivePlausiblyComplete(goodPBSEntry()); !ok {
t.Errorf("a real snapshot was rejected: %s", why)
}
}
// Group D — the rejection is announced ONCE per snapshot, not once per due-check.
//
// The due-check runs every 5 minutes and a phantom persists indefinitely (server-side prune does
// not collect it), so per-poll logging would emit ~288 identical lines a day and bury the signal.
//
// RED-PROOF: delete the `if seen { return }` guard in warnRejectedArchiveOnce → this test reports
// "logged 5 times, want 1".
func TestNewestArchiveTime_RejectionLoggedOncePerSnapshot(t *testing.T) {
var buf bytes.Buffer
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
const polls = 5
for i := 0; i < polls; i++ {
if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
t.Fatalf("poll %d: %v", i, err)
}
}
n := strings.Count(buf.String(), "INCOMPLETE archive")
if n != 1 {
t.Errorf("rejection logged %d times across %d polls, want exactly 1:\n%s", n, polls, buf.String())
}
out := buf.String()
if !strings.Contains(out, phantomEntry().VolID) {
t.Errorf("the log line does not NAME the rejected snapshot:\n%s", out)
}
if !strings.Contains(out, "below the") {
t.Errorf("the log line does not say WHY it was rejected:\n%s", out)
}
if !strings.Contains(out, "level=WARN") {
t.Errorf("rejection was not logged at WARN:\n%s", out)
}
}
// Group D — a SECOND, distinct phantom is announced separately. The dedupe must be per snapshot,
// not a one-shot latch that hides every later phantom.
func TestNewestArchiveTime_DistinctPhantomsEachAnnounced(t *testing.T) {
var buf bytes.Buffer
second := phantomEntry()
second.VolID = "felhom-pbs:backup/ct/9201/2026-07-29T05:31:14Z"
second.CTime = phantomEntry().CTime + 86400
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), second, goodPBSEntry()})
for i := 0; i < 3; i++ {
if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
t.Fatalf("poll %d: %v", i, err)
}
}
if n := strings.Count(buf.String(), "INCOMPLETE archive"); n != 2 {
t.Errorf("got %d rejection lines for 2 distinct phantoms across 3 polls, want 2:\n%s", n, buf.String())
}
}
+39 -10
View File
@@ -24,6 +24,8 @@ type fakeBackupAPI struct {
cfgErr error
content []proxmox.StorageContent
contentErr error
storages []proxmox.Storage // returned by ListStorage (the local-prune scope gate)
storageErr error
vzdumps []proxmox.VzdumpOptions
logLines []string // returned by TaskLogTail (e.g. "INFO: backup mode: stop")
waitGate chan struct{} // if non-nil, WaitTask blocks until closed (8B.2 watcher timing)
@@ -45,6 +47,9 @@ func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConf
func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
return f.content, f.contentErr
}
func (f *fakeBackupAPI) ListStorage(_ context.Context) ([]proxmox.Storage, error) {
return f.storages, f.storageErr
}
func (f *fakeBackupAPI) TaskLogTail(_ context.Context, _ string, _ int) ([]string, error) {
return f.logLines, nil
}
@@ -73,7 +78,7 @@ func TestBackup_SuccessResolvesArchiveAndBulkGap(t *testing.T) {
{VolID: "local:backup/other-9002.tar.zst", Content: "backup", VMID: 9002, Size: 7, CTime: 999},
},
}
r := NewBackupRunner(api, "local", "", "felhom test", quiet())
r := NewBackupRunner(api, "local", "", "felhom test", "", quiet())
rec, err := r.Backup(context.Background(), 9001)
if err != nil {
t.Fatalf("Backup: %v", err)
@@ -107,7 +112,7 @@ func TestBackup_ReportsActualModeFromTaskLog(t *testing.T) {
content: []proxmox.StorageContent{{VolID: "v", Content: "backup", VMID: 9001, Size: 10, CTime: 1}},
logLines: []string{"INFO: CT Name: spike", "INFO: backup mode: stop", "INFO: Finished"},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", quiet())
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "", quiet())
rec, err := r.Backup(context.Background(), 9001)
if err != nil {
t.Fatal(err)
@@ -119,7 +124,7 @@ func TestBackup_ReportsActualModeFromTaskLog(t *testing.T) {
func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
api := &fakeBackupAPI{vzdumpErr: errors.New("vzdump boom")}
r := NewBackupRunner(api, "local", "", "", quiet())
r := NewBackupRunner(api, "local", "", "", "", quiet())
rec, err := r.Backup(context.Background(), 9001)
if err == nil {
t.Fatal("expected error")
@@ -130,12 +135,13 @@ func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
}
func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
const big = 4 << 30 // a plausible whole-guest archive
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "a", Content: "backup", CTime: 10},
{VolID: "b", Content: "backup", CTime: 99},
{VolID: "iso", Content: "iso", CTime: 999}, // not a backup → ignored
{VolID: "a", Content: "backup", CTime: 10, Size: big},
{VolID: "b", Content: "backup", CTime: 99, Size: big},
{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
}}
r := NewBackupRunner(api, "local", "", "", quiet())
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
if err != nil || vol != "b" {
t.Fatalf("pick = %q,%v want newest 'b'", vol, err)
@@ -147,6 +153,26 @@ func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
}
}
// R-86: the NEWEST entry is not a candidate if it cannot be a complete archive. An incomplete
// artefact (F-CRIT-2's 1-byte phantom, which server-side prune does not collect) would otherwise be
// picked forever, fail its restore forever, never earn proof, and so leave the tier due at every
// evaluation — turning the evaluation interval into the retry rate for a multi-GB restore.
//
// COMPANION RED-PROOF (observed): drop the `archivePlausiblyComplete` guard from
// PickSettledRestoreCandidateOn and this fails with
// `pick = "phantom" want the newest COMPLETE archive 'real'`.
func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
}}
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
if err != nil || vol != "real" {
t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
}
}
// --- scheduler ---
type fakeRTRunner struct {
@@ -163,9 +189,12 @@ func TestScheduler_TickRunsAndRecords(t *testing.T) {
store := NewStore()
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Pass: true, Verified: "boot+running", Duration: time.Second}}
s := NewScheduler(SchedulerOptions{
Runner: rt,
Pick: func(context.Context) (string, error) { return "vol", nil },
Store: store,
Runner: rt,
Pick: func(context.Context) (string, error) { return "vol", nil },
Store: store,
Spec: func(context.Context, string) reconcile.RestoreTestSpec {
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
},
Cadence: time.Hour,
Logger: quiet(),
})
+69
View File
@@ -0,0 +1,69 @@
package backup
import "sync"
// InFlight is the host-wide "one heavy guest operation at a time" gate.
//
// R-85 (Scenario F). The operator's R-82 ruling was "one backup at a time per guest"; a restore-test
// must JOIN that single-flight rather than sit outside it. It is not a lock-contention concern —
// a restore-test uses a scratch VMID, so it never touches the live guest's vzdump lock. It is a
// LINK concern: an offsite restore PULLS a multi-GB archive while an offsite backup PUSHES one, over
// the same WireGuard tunnel. On the demo fleet that link runs at ~33 MB/min upstream; running both
// at once makes each slower and pushes both toward their timeouts, which is how a healthy tier ends
// up recorded as failed.
//
// It is deliberately host-wide and coarse rather than per-guest: these boxes carry one customer
// guest, and the resource being protected (the uplink) is shared by everything on the host anyway.
//
// The gate is ADVISORY in one direction only — it never cancels anything already running. A caller
// that cannot acquire DEFERS to its next cadence. Deferring a restore-test costs a few hours of
// coverage; cancelling a running backup costs the backup.
//
// CORRECTED 2026-07-28 (F-A1). That "DEFERS" was true of the restore-test caller and NOT of the
// backup caller, and the comment did not say so. The controller's start path had no 409 branch, so
// a refusal here was recorded as a tier FAILURE: the R-88 breaker armed and the operator was
// emailed "Whole-guest backup FAILED" about a backup that was merely waiting its turn. Campaign 8
// observed it on both demo boxes in the same minute.
//
// Fixed on the CONTROLLER side (v0.179.0), which is where the misreading lived — this gate's
// behaviour was correct throughout and is unchanged. The controller now maps HTTP 409 to a
// contention path: it defers the tier, keeps it DUE, and alarms only if contention outlives the
// agent's own restore-test ceiling. Nothing here needs to change; the claim above is simply now
// true of both callers.
type InFlight struct {
mu sync.Mutex
what string // "" = idle
}
// TryAcquire claims the gate for `what`. ok=false means something else holds it, and `busy` names
// it — the name matters, because "deferred" with no reason is indistinguishable from "broken".
func (g *InFlight) TryAcquire(what string) (release func(), busy string, ok bool) {
if g == nil {
// Not wired (older call sites, tests) → no gating, previous behaviour.
return func() {}, "", true
}
g.mu.Lock()
defer g.mu.Unlock()
if g.what != "" {
return nil, g.what, false
}
g.what = what
var once sync.Once
return func() {
once.Do(func() {
g.mu.Lock()
g.what = ""
g.mu.Unlock()
})
}, "", true
}
// Busy reports what currently holds the gate ("" = idle).
func (g *InFlight) Busy() string {
if g == nil {
return ""
}
g.mu.Lock()
defer g.mu.Unlock()
return g.what
}
+99
View File
@@ -0,0 +1,99 @@
package backup
import (
"context"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// localTarget is a non-PBS dir storage; pbsTarget is a PBS storage — for the scope gate.
var (
localTargetStores = []proxmox.Storage{{Storage: "local", Type: "dir", Content: "backup"}}
pbsTargetStores = []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", Content: "backup"}}
)
func okAPI(stores []proxmox.Storage) *fakeBackupAPI {
return &fakeBackupAPI{vzdumpUPID: "UPID:vzdump:1", storages: stores}
}
// TestPrune_LocalCarriesKeepLast: a LOCAL-target backup with retention carries `--prune-backups
// keep-last=3` on the vzdump. Companion: the SAME runner built with no retention ("") emits NO prune
// option — proving the flag only rides when retention is set (a no-prune build accumulates).
func TestPrune_LocalCarriesKeepLast(t *testing.T) {
api := okAPI(localTargetStores)
r := NewBackupRunner(api, "local", proxmox.ModeStop, "", "keep-last=3", quiet())
_, _ = r.Backup(context.Background(), 9201) // archive-resolution may fail in the fake; we assert the captured vzdump opts
if len(api.vzdumps) != 1 || api.vzdumps[0].PruneBackups != "keep-last=3" {
t.Fatalf("local backup must carry prune-backups keep-last=3, got %q", api.vzdumps[0].PruneBackups)
}
// COMPANION: no retention → no prune option (dumps would accumulate).
api2 := okAPI(localTargetStores)
r2 := NewBackupRunner(api2, "local", proxmox.ModeStop, "", "", quiet())
_, _ = r2.Backup(context.Background(), 9201)
if api2.vzdumps[0].PruneBackups != "" {
t.Fatalf("a no-retention runner must NOT prune, got %q", api2.vzdumps[0].PruneBackups)
}
}
// TestPrune_NeverPrunesPBS is the scope rule (§9): retention is NOT applied when the target is a PBS
// storage (offsite retention is a separate lifecycle). Companion: the identical retention on a LOCAL
// target IS applied — proving the gate keys on storage type, not luck.
func TestPrune_NeverPrunesPBS(t *testing.T) {
api := okAPI(pbsTargetStores)
r := NewBackupRunner(api, "felhom-pbs", proxmox.ModeStop, "", "keep-last=3", quiet())
_, _ = r.Backup(context.Background(), 9201)
if api.vzdumps[0].PruneBackups != "" {
t.Fatalf("a PBS target must NEVER be pruned by the per-run flag, got %q", api.vzdumps[0].PruneBackups)
}
// COMPANION: same retention, local target → applied.
api2 := okAPI(localTargetStores)
r2 := NewBackupRunner(api2, "local", proxmox.ModeStop, "", "keep-last=3", quiet())
_, _ = r2.Backup(context.Background(), 9201)
if api2.vzdumps[0].PruneBackups != "keep-last=3" {
t.Fatalf("control: a local target with the same retention MUST be pruned, got %q", api2.vzdumps[0].PruneBackups)
}
}
// TestPrune_FailSafeOnUnknownTarget: if the target's type can't be confirmed (lookup error / not in the
// list), the run SKIPS pruning rather than risk pruning a PBS/unknown storage.
func TestPrune_FailSafeOnUnknownTarget(t *testing.T) {
// target not present in the list → skip.
api := &fakeBackupAPI{vzdumpUPID: "UPID:vzdump:1", storages: localTargetStores}
r := NewBackupRunner(api, "some-other-store", proxmox.ModeStop, "", "keep-last=3", quiet())
_, _ = r.Backup(context.Background(), 9201)
if api.vzdumps[0].PruneBackups != "" {
t.Fatalf("an unknown target must skip pruning (fail-safe), got %q", api.vzdumps[0].PruneBackups)
}
}
// TestPrune_KeepLastClamp (§7-B): a 0/negative/unset LocalBackupRetention clamps to ≥1 (default 3) so the
// vzdump NEVER prunes the archive it just made. Companion: a no-clamp impl that returns 0 would emit
// keep-last=0 → PVE prunes everything → FAILS the "≥1" assertion.
func TestPrune_KeepLastClamp(t *testing.T) {
cases := []struct {
set int
want int
}{
{0, 3}, // unset → default
{-5, 3}, // negative → default
{1, 1}, // honored
{3, 3}, // honored
{10, 10}, // honored
}
for _, c := range cases {
b := config.BackupConfig{LocalBackupRetention: c.set}
if got := b.KeepLast(); got != c.want {
t.Errorf("KeepLast(%d) = %d, want %d", c.set, got, c.want)
}
if b.KeepLast() < 1 {
t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set)
}
}
if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-last=3" {
t.Fatalf("default PruneBackupsSpec = %q, want keep-last=3", spec)
}
}
+142
View File
@@ -0,0 +1,142 @@
package backup
import (
"context"
"fmt"
"time"
)
// R-86 — a restore-test follows the BACKUP, not the clock.
//
// ── WHAT WAS WRONG ───────────────────────────────────────────────────────────────────────────
//
// The trigger was `time.NewTicker(cadence)` started at daemon start, and the tier was chosen by
// oldest-proven rotation. Its phase was therefore the PROCESS'S UPTIME: agent deploys are routine,
// so the test drifted to an arbitrary time of day every week; a fresh archive could sit unproven
// while an older one was re-tested; and a weekly tier was tested on the same rhythm as a daily one,
// sometimes twice on the same archive.
//
// ── THE RULE, AND THE TRAP IN ITS OBVIOUS FORM ───────────────────────────────────────────────
//
// R-86's ask reads "test a tier ~24 h after its own newest archive". Implemented literally —
// *"due when the newest archive is at least `settle` old"* — a DAILY tier is NEVER due: a new
// archive lands every day, so the newest archive's age resets to zero long before it reaches 24 h.
// The naive rule silently switches restore-testing off for the tier that matters most, and it is
// the version a reasonable person would write. It has a red-proof of its own
// (TestDue_NaiveNewestArchiveAgeRuleNeverFiresOnADailyTier).
//
// The rule implemented here:
//
// Let A = the newest archive on this tier that is at least `settle` old.
// The tier is DUE when A exists and A HAS NOT ALREADY BEEN PROVEN.
//
// daily tier → A is yesterday's archive; a new one settles each day → proved once per day
// weekly tier → A is last week's until the next settles → proved once per week
// newborn tier → A does not exist → UNKNOWN, never a fault
//
// Per-archive due-ness IS the pacing: one test per archive generation and no more. There is
// deliberately no second rate limiter on top of it (§8.4) — two independent pacing mechanisms
// produce a cadence nobody can predict from either.
//
// ── WHAT DID NOT CHANGE ──────────────────────────────────────────────────────────────────────
//
// The one-heavy-operation gate, the success-only proof credit, the oldest-proven ordering (now the
// tie-break between two DUE tiers), the restore-test itself, its journal and its scratch band. Only
// the trigger changed.
// DueVerdict is one tier's due-ness, and the evidence for it. Every field is logged: a due-check
// that cannot say WHY is a quiet path, and quiet paths are what this monitor family keeps shipping.
type DueVerdict struct {
Target string // the tier's storage target id
// Due is true only when Archive is set and has not been proven.
Due bool
// Archive is the settled candidate A ("" when the tier holds none).
Archive string
// Landed is when A landed on the tier (zero when Archive is "").
Landed time.Time
// ProvenArchive is what the state says was last proven on this tier ("" = nothing/legacy).
ProvenArchive string
// Err is a candidate-lookup failure. A tier whose archives cannot be listed is UNKNOWN — it is
// NEVER reported as "not due", which would silently retire a tier the moment its storage
// stopped answering. Due stays false (we have no archive to test) and the error travels.
Err error
// Reason is the one-line human account of this verdict.
Reason string
}
// String renders a verdict for the operator log / selftest output.
func (v DueVerdict) String() string {
return fmt.Sprintf("tier=%s due=%v archive=%q reason=%s", v.Target, v.Due, v.Archive, v.Reason)
}
// EvaluateDue returns the due verdict for every configured tier, ordered oldest-proven first.
//
// Ordering is the R-85 rotation, demoted to a TIE-BREAK: it no longer decides whether a test
// happens (due-ness does), only which of several due tiers goes first. Keeping it means a tier can
// still never be starved — a tier that has waited longest is served first — and keeping it as the
// order rather than as the trigger is the whole of this change.
func (s *Scheduler) EvaluateDue(ctx context.Context) []DueVerdict {
if !s.rotating() {
return nil
}
order := s.tiers
if s.rtState != nil {
order = s.rtState.OldestFirst(s.tiers)
}
cutoff := s.settleCutoff()
out := make([]DueVerdict, 0, len(order))
for _, target := range order {
out = append(out, s.evaluateTier(ctx, target, cutoff))
}
return out
}
// settleCutoff is the newest landing time an archive may have and still count as settled.
func (s *Scheduler) settleCutoff() time.Time {
if s.settle <= 0 {
return time.Time{} // no settle requirement configured → any archive is a candidate
}
return s.now().Add(-s.settle)
}
// evaluateTier is the per-tier due-check. PURE given the picker and the state, so the rule is
// unit-tested directly rather than inferred from whether a fake runner happened to be called.
func (s *Scheduler) evaluateTier(ctx context.Context, target string, cutoff time.Time) DueVerdict {
v := DueVerdict{Target: target}
archive, landed, err := s.tierPick(ctx, target, cutoff)
if err != nil {
// UNKNOWN, never "not due", and never silent.
v.Err = err
v.Reason = fmt.Sprintf("candidate lookup FAILED (%v) — tier is unknown this evaluation, not proven and not dismissed", err)
return v
}
v.Archive, v.Landed = archive, landed
if archive == "" {
v.Reason = "no settled archive yet — nothing to prove (newborn or still settling)"
return v
}
proven, ok := "", false
if s.rtState != nil {
proven, ok = s.rtState.ProvenArchive(target)
}
v.ProvenArchive = proven
if ok && proven == archive {
v.Reason = fmt.Sprintf("newest settled archive (landed %s) is already proven", landed.Format(time.RFC3339))
return v
}
v.Due = true
switch {
case !ok && proven == "":
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven; nothing proven on this tier yet", landed.Format(time.RFC3339))
default:
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven (last proven archive was a different one)", landed.Format(time.RFC3339))
}
return v
}
// EvaluateDueTier is EvaluateDue for ONE named tier — the selftest's per-tier cost probe, so the
// WAN leg of an offsite lookup is attributable rather than buried in an aggregate.
func (s *Scheduler) EvaluateDueTier(ctx context.Context, target string) DueVerdict {
return s.evaluateTier(ctx, target, s.settleCutoff())
}
+458
View File
@@ -0,0 +1,458 @@
package backup
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// R-86 — the restore-test follows the BACKUP, not the clock.
//
// Every test here DRIVES time (`s.now` is injected and stepped) rather than waiting for it. A test
// that slept could not say anything about a 24-hour rule in under 24 hours, and one that only
// asserted "no error" would pass against a scheduler that never ran anything at all — which is
// precisely the failure mode §8.1's trap produces. So the assertions are: did a test run, on WHICH
// archive, and did a second evaluation correctly run NOTHING.
// ── the fake tier storage ────────────────────────────────────────────────────────────────────
// archiveStub is one archive on a tier: its volid and when it landed.
type archiveStub struct {
volid string
landed time.Time
}
// tierStorage is a TierPicker over per-tier archive lists. It implements the SAME contract as the
// production picker (*BackupRunner).PickSettledRestoreCandidateOn — newest archive that landed at
// or before the cutoff — which is itself covered against a fake PVE API in backup_test.go, and
// end-to-end by the live run. Naming the seam explicitly: everything below is true up to this
// picker; that the real picker obeys the same rule is asserted there, not here.
type tierStorage struct {
archives map[string][]archiveStub
err map[string]error // target → lookup failure
}
func (ts *tierStorage) pick(_ context.Context, target string, notAfter time.Time) (string, time.Time, error) {
if e, ok := ts.err[target]; ok && e != nil {
return "", time.Time{}, e
}
var best archiveStub
for _, a := range ts.archives[target] {
if !notAfter.IsZero() && a.landed.After(notAfter) {
continue // not settled yet
}
if best.volid == "" || a.landed.After(best.landed) {
best = a
}
}
return best.volid, best.landed, nil
}
// dueHarness is a scheduler with a driven clock over a fake tier storage.
type dueHarness struct {
s *Scheduler
rr *rotRunner
st *RestoreTestState
ts *tierStorage
clock time.Time
path string
}
func newDueHarness(t *testing.T, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
t.Helper()
return newDueHarnessAt(t, filepath.Join(t.TempDir(), "rt.json"), start, settle, pass, tiers, ts)
}
func newDueHarnessAt(t *testing.T, statePath string, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
t.Helper()
h := &dueHarness{rr: &rotRunner{pass: pass}, ts: ts, clock: start, path: statePath}
h.st = NewRestoreTestState(statePath)
h.s = NewScheduler(SchedulerOptions{
Runner: h.rr,
Store: NewStore(),
Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
},
Cadence: time.Hour,
Settle: settle,
Logger: quiet(),
Tiers: tiers,
TierPick: ts.pick,
State: h.st,
InFlight: &InFlight{},
})
h.s.now = func() time.Time { return h.clock }
return h
}
// advance steps the clock by step, evaluating once at every step — the scheduler's real shape.
func (h *dueHarness) advance(step, total time.Duration) {
for elapsed := time.Duration(0); elapsed < total; elapsed += step {
h.clock = h.clock.Add(step)
h.s.tick(context.Background())
}
}
var day0 = time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)
// dailyArchives lands one archive a day at 02:00 for n days, starting at day0.
func dailyArchives(tier string, n int) []archiveStub {
out := make([]archiveStub, 0, n)
for d := 0; d < n; d++ {
out = append(out, archiveStub{
volid: fmt.Sprintf("%s:backup/vzdump-lxc-9201-day%d.tar.zst", tier, d),
landed: day0.AddDate(0, 0, d),
})
}
return out
}
// ── SCENARIO A — a daily tier is proved daily, on its own archive ────────────────────────────
//
// THE TRAP THIS PINS (§8.1). R-86 reads "trigger a tier ~24 h after its own newest archive", and
// the literal implementation of that — *due when the newest archive is at least `settle` old* — is
// NEVER true on a daily tier: a new archive lands every day, so the newest archive's age resets to
// zero long before it reaches 24 h. The literal reading silently switches restore-testing OFF for
// the tier that matters most.
//
// COMPANION RED-PROOF (observed 2026-08-03). In Scheduler.evaluateTier, the per-archive comparison
// was replaced by the naive age rule:
//
// - if ok && proven == archive { … not due … }
// + if s.now().Sub(landed) < s.settle { … not due … } // and the proven-archive check deleted
//
// and the picker cutoff was removed (`cutoff := time.Time{}`), i.e. exactly "is the newest archive
// old enough". Result:
//
// --- FAIL: TestDue_DailyTierIsProvedDailyOnItsOwnArchive
// restoretest_due_test.go: a daily tier must be proved once per day; got 0 run(s) over 5 days
//
// Zero runs — restore-testing off. Restored immediately afterwards.
func TestDue_DailyTierIsProvedDailyOnItsOwnArchive(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 6)}}
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
// Five days, evaluated hourly.
h.advance(time.Hour, 5*24*time.Hour)
got := h.rr.seen()
if len(got) != 5 {
t.Fatalf("a daily tier must be proved once per day; got %d run(s) over 5 days: %v", len(got), got)
}
// And each run must be on the archive that settled that day — day0's on day 1, and so on.
for i, a := range got {
want := fmt.Sprintf("local:backup/vzdump-lxc-9201-day%d.tar.zst", i)
if a != want {
t.Fatalf("run %d tested %q, want %q — the test is not following the archive", i+1, a, want)
}
}
// The newest archive is NEVER the one tested: it has not settled.
if last := got[len(got)-1]; last == "local:backup/vzdump-lxc-9201-day5.tar.zst" {
t.Fatal("the still-settling archive was tested — the settle cutoff is not being applied")
}
}
// ── SCENARIO B — a weekly tier is proved weekly, not every other day ─────────────────────────
func TestDue_WeeklyTierIsProvedOncePerArchive(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {
{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0},
{volid: "felhom-pbs:backup/ct/9201/w1", landed: day0.AddDate(0, 0, 7)},
{volid: "felhom-pbs:backup/ct/9201/w2", landed: day0.AddDate(0, 0, 14)},
}}}
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
// Three weeks, evaluated every 6 hours — 84 evaluations.
h.advance(6*time.Hour, 21*24*time.Hour)
got := h.rr.seen()
want := []string{
"felhom-pbs:backup/ct/9201/w0",
"felhom-pbs:backup/ct/9201/w1",
"felhom-pbs:backup/ct/9201/w2",
}
if len(got) != len(want) {
t.Fatalf("a weekly tier must be proved ONCE PER ARCHIVE (3 archives over 3 weeks); got %d run(s): %v", len(got), got)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("run %d tested %q, want %q", i+1, got[i], want[i])
}
}
}
// ── SCENARIO C — an agent restart does not change the schedule ───────────────────────────────
//
// This is the defect a person actually notices: today every deploy restarts the ticker, so a
// restore-test runs one interval after each deploy regardless of what has already been proven.
//
// COMPANION RED-PROOF (observed 2026-08-03): revert the state to per-tier TIME by making
// ProvenArchive ignore the stored archive —
//
// - if !ok || p.Archive == "" { return "", false }
// + return "", false // per-tier time only, the pre-R-86 state
//
// → --- FAIL: TestDue_RestartRunsNothing
// restoretest_due_test.go:226: an agent restart must not trigger a restore-test; 2 restart(s)
// produced 4 run(s)
//
// Four: the same already-proven archive re-tested on EVERY evaluation after EVERY restart, which is
// today's behaviour with the ticker's phase reset by the deploy. Restored.
func TestDue_RestartRunsNothing(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "rt.json")
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
start := day0.AddDate(0, 0, 1).Add(time.Hour) // day 1, 03:00 — day0's archive has settled
h := newDueHarnessAt(t, path, start, 24*time.Hour, true, []string{"local"}, ts)
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("precondition: the settled archive should have been proved once; got %d run(s)", n)
}
// --- two restarts: brand-new scheduler + brand-new state object over the SAME file ---
total := 0
for i := 0; i < 2; i++ {
h2 := newDueHarnessAt(t, path, start.Add(time.Duration(i+1)*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
h2.s.tick(context.Background())
h2.s.tick(context.Background())
total += len(h2.rr.seen())
}
if total != 0 {
t.Fatalf("an agent restart must not trigger a restore-test; 2 restart(s) produced %d run(s)", total)
}
}
// ── SCENARIO D — a new archive makes a tier due even if it was tested yesterday ──────────────
func TestDue_NewSettledArchiveMakesAProvedTierDueAgain(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
h.s.tick(context.Background()) // proves day0's archive
h.s.tick(context.Background()) // nothing new has settled → nothing
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("want exactly 1 run before the new archive settles, got %d: %v", n, h.rr.seen())
}
// Day 2, 03:00 — day1's archive has now settled.
h.clock = day0.AddDate(0, 0, 2).Add(time.Hour)
h.s.tick(context.Background())
got := h.rr.seen()
if len(got) != 2 {
t.Fatalf("a newly settled archive must make the tier due again; got %v", got)
}
if got[1] != "local:backup/vzdump-lxc-9201-day1.tar.zst" {
t.Fatalf("the NEW archive must be the one tested; got %q", got[1])
}
}
// ── SCENARIO E — a failing tier keeps being retried, and earns no proof ──────────────────────
//
// COMPANION RED-PROOF (observed 2026-08-03): give credit on failure in Scheduler.tick —
//
// - if rt.Pass && s.rtState != nil && target != "" {
// + if s.rtState != nil && target != "" {
//
// → --- FAIL: TestDue_FailingTierIsRetriedAndNeverProven
// restoretest_due_test.go: a failing tier must keep being retried; got 1 run(s) over 3
// evaluations
//
// A single failure would have retired the archive as proven — a permanently broken DR tier looking
// freshly verified, which is the loudest signal this system produces going silent. Restored.
func TestDue_FailingTierIsRetriedAndNeverProven(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 1)}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, false, []string{"local"}, ts)
for i := 0; i < 3; i++ {
h.s.tick(context.Background())
}
got := h.rr.seen()
if len(got) != 3 {
t.Fatalf("a failing tier must keep being retried; got %d run(s) over 3 evaluations: %v", len(got), got)
}
if _, ok := h.st.ProvenArchive("local"); ok {
t.Fatal("a FAILED restore-test must not record the archive as proven")
}
if _, ok := h.st.LastSuccess("local"); ok {
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
}
}
// ── SCENARIO F — two tiers due at once do not run at once ────────────────────────────────────
func TestDue_TwoDueTiersRunOneAtATime(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{
"local": {{volid: "local:backup/a.tar.zst", landed: day0}},
"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/a", landed: day0}},
}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
// Both tiers are due at this instant.
due := h.s.EvaluateDue(context.Background())
if len(due) != 2 || !due[0].Due || !due[1].Due {
t.Fatalf("precondition: both tiers should be due; got %v", due)
}
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("ONE evaluation must start ONE restore-test, never two multi-GB restores over one link; got %d: %v", n, h.rr.seen())
}
// The other tier was DEFERRED, not cancelled: it is still due and runs on the next evaluation.
h.s.tick(context.Background())
got := h.rr.seen()
if len(got) != 2 || got[0] == got[1] {
t.Fatalf("the deferred tier must run on the NEXT evaluation, on its own archive; got %v", got)
}
}
// The heavy-operation gate still holds, and a tier deferred behind a backup stays DUE.
func TestDue_DeferredBehindABackupStaysDue(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
gate := &InFlight{}
h.s.inFlight = gate
release, _, _ := gate.TryAcquire("backup:felhom-pbs")
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 0 {
t.Fatalf("the restore-test must DEFER while a backup holds the gate; got %d run(s)", n)
}
if due := h.s.EvaluateDue(context.Background()); !due[0].Due {
t.Fatal("a deferred tier must remain DUE — deferral is not dismissal")
}
release()
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("must resume once the gate frees; got %d run(s)", n)
}
}
// ── SCENARIO H — a newborn box is UNKNOWN, not stale and not a fault ─────────────────────────
func TestDue_NewbornTierIsNotDueAndNotAnError(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": nil}}
h := newDueHarness(t, day0, 24*time.Hour, true, []string{"felhom-pbs"}, ts)
due := h.s.EvaluateDue(context.Background())
if len(due) != 1 {
t.Fatalf("want one verdict, got %v", due)
}
v := due[0]
if v.Due || v.Err != nil || v.Archive != "" {
t.Fatalf("a tier with no archive is UNKNOWN — not due, not an error; got %+v", v)
}
if v.Reason == "" {
t.Fatal("every verdict must carry a reason — a due-check that cannot say why is a quiet path")
}
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 0 {
t.Fatalf("a newborn tier must not be restore-tested; got %d run(s)", n)
}
}
// An archive that exists but has NOT settled yet is not a candidate — and that is not an error.
func TestDue_UnsettledArchiveIsNotACandidate(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/fresh.tar.zst", landed: day0}}}}
h := newDueHarness(t, day0.Add(2*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
if v := h.s.EvaluateDue(context.Background())[0]; v.Due || v.Archive != "" {
t.Fatalf("an archive 2h old must not be a candidate under a 24h settle lag; got %+v", v)
}
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 0 {
t.Fatalf("nothing settled → no run; got %d", n)
}
}
// A tier whose archives cannot be LISTED is UNKNOWN — never silently "not due", and never silent.
// Treating a lookup failure as "not due" would retire a tier the moment its storage stopped
// answering, which is the same absence-is-not-evidence error this monitor family keeps making.
func TestDue_LookupFailureIsUnknownNotNotDue(t *testing.T) {
boom := errors.New("storage unreachable")
ts := &tierStorage{
archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}},
err: map[string]error{"felhom-pbs": boom},
}
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
var pbs DueVerdict
for _, v := range h.s.EvaluateDue(context.Background()) {
if v.Target == "felhom-pbs" {
pbs = v
}
}
if pbs.Err == nil {
t.Fatal("a lookup failure must travel in the verdict, not be swallowed")
}
if pbs.Due {
t.Fatal("a tier we could not list must not be reported DUE — we have no archive to test")
}
if pbs.Reason == "" {
t.Fatal("the failure must be explained, not merely flagged")
}
// And the OTHER tier still runs: one tier's storage being unreadable must not cost the other
// tier its proof.
h.s.tick(context.Background())
if got := h.rr.seen(); len(got) != 1 || got[0] != "local:backup/a.tar.zst" {
t.Fatalf("the readable tier must still be proved; got %v", got)
}
}
// ── the state's migration (§8.2) ─────────────────────────────────────────────────────────────
// A pre-R-86 state file carries a TIME and no archive. It must keep its time (rotation ordering
// survives the upgrade) and yield NO proven archive, so each tier is due exactly once. Reading a
// legacy time as proof of the CURRENT archive would mark an unproven archive proven — a guarantee
// invented by a migration.
func TestRestoreTestState_LegacyFileMigratesToNothingProven(t *testing.T) {
path := filepath.Join(t.TempDir(), "rt.json")
legacy := `{"local":"2026-08-01T02:00:00Z","felhom-pbs":"2026-07-30T02:00:00Z"}`
if err := writeFileForTest(path, legacy); err != nil {
t.Fatal(err)
}
st := NewRestoreTestState(path)
if _, ok := st.ProvenArchive("local"); ok {
t.Fatal("a legacy record names no archive — it must NOT be read as proof of the current one")
}
at, ok := st.LastSuccess("local")
if !ok || !at.Equal(time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)) {
t.Fatalf("the legacy TIME must survive (rotation ordering depends on it); got %v ok=%v", at, ok)
}
// Ordering still works off the legacy times.
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
t.Fatalf("oldest-first must still order legacy records; got %v", got)
}
}
// The new shape round-trips, archive and all.
func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
path := filepath.Join(t.TempDir(), "rt.json")
now := time.Now().UTC().Truncate(time.Second)
st := NewRestoreTestState(path)
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil {
t.Fatal(err)
}
re := NewRestoreTestState(path)
got, ok := re.ProvenArchive("felhom-pbs")
if !ok || got != "felhom-pbs:backup/ct/9201/x" {
t.Fatalf("the proven ARCHIVE must survive a restart; got %q ok=%v", got, ok)
}
at, ok := re.LastSuccess("felhom-pbs")
if !ok || !at.Equal(now) {
t.Fatalf("the proven TIME must survive too; got %v ok=%v", at, ok)
}
}
// writeFileForTest is a tiny helper so the legacy-migration fixture reads clearly above.
func writeFileForTest(path, content string) error {
return os.WriteFile(path, []byte(content), 0o600)
}
+203
View File
@@ -0,0 +1,203 @@
package backup
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"sync"
"time"
)
// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
//
// R-85 (1.4). This one genuinely needs PERSISTENCE, unlike R-84 — and the difference is worth
// stating, because the two look like the same problem and are not:
//
// - R-84 (backup freshness) had a GROUND TRUTH to consult: the archive is still on the storage,
// so the agent could ask "when did a backup last land?" and never persist anything. That is
// strictly better, because a pruned archive correctly stops counting.
// - A restore-test leaves NO artifact — the scratch guest is destroyed as its final act. There is
// nothing to query. "Did we prove this tier restores?" exists only as remembered state, so it
// must be written down or it is lost.
//
// Why it must survive a restart: rotation is oldest-first (the operator ruling), so an in-memory map
// would reset every tier to "never tested" on each restart. Ordering would then depend on map
// iteration order, and one tier could be starved indefinitely while the other is re-tested — with
// agent deploys as routine as they are, that is not a corner case.
//
// Only SUCCESS is recorded. A failed run must not satisfy rotation, or a tier that fails every time
// would look freshly proven and stop being retried — the same "a failure satisfies the cadence"
// trap the backup due-check avoids. R-86 keeps that property unchanged and gives it a second job:
// the due-check reads this state, so a failure that recorded proof would ALSO stop the tier from
// ever becoming due again. The rule earns its keep twice now.
//
// R-86 (1.2) — WHICH ARCHIVE, not just when.
//
// A timestamp alone cannot answer the question the due-check asks. "This tier passed at 04:00" is
// consistent both with "yesterday's archive is proven" and with "an archive from a week ago is
// proven and nothing since has been looked at". Restore-testing is now per ARCHIVE GENERATION —
// a tier is due once it holds a settled archive that has not been proven — so the identity of the
// proven archive is the state, and the time is metadata (rotation ordering, operator reporting).
//
// This is the same class as the workspace rule "a timestamp records an ATTEMPT, not a RESULT":
// here it records a result, but not WHICH result, and that is just as unable to answer the question
// being asked of it.
type RestoreTestState struct {
path string
mu sync.Mutex
last map[string]provenTier // target id → what was last PROVEN on that tier
}
// provenTier is one tier's proof: the archive that passed, and when it passed.
type provenTier struct {
Archive string // volid of the archive that PASSED; "" = a legacy record with no archive
At time.Time // when that run passed (UTC)
}
// provenTierJSON is the on-disk shape (R-86). The legacy shape was a bare RFC3339 STRING per
// target; both are read, only this one is written — see NewRestoreTestState.
type provenTierJSON struct {
Archive string `json:"archive"`
ProvenAt string `json:"proven_at"`
}
// NewRestoreTestState opens (or creates) the state at path. A missing or unreadable file is NOT an
// error: it degrades to "nothing proven yet", which is the correct starting point and keeps a
// corrupt file from wedging the daemon.
//
// MIGRATION (R-86). The pre-R-86 file is `{"<target>": "<RFC3339>"}` — a time and no archive. A
// legacy record keeps its TIME (rotation ordering survives a deploy, which is why the file exists
// at all) but yields NO proven archive, so every tier is due exactly once on first evaluation after
// the upgrade. One extra restore-test per tier, once, is the safe direction: the alternative is to
// read a legacy time as proof of whatever archive happens to be current, which would mark an
// unproven archive proven — inventing a guarantee out of a migration.
func NewRestoreTestState(path string) *RestoreTestState {
s := &RestoreTestState{path: path, last: map[string]provenTier{}}
data, err := os.ReadFile(path)
if err != nil {
return s
}
var raw map[string]json.RawMessage
if json.Unmarshal(data, &raw) != nil {
return s
}
for target, msg := range raw {
// Legacy shape: a bare RFC3339 string.
var legacy string
if json.Unmarshal(msg, &legacy) == nil {
if t, perr := time.Parse(time.RFC3339, legacy); perr == nil {
s.last[target] = provenTier{At: t.UTC()} // no archive → due once, deliberately
}
continue
}
var cur provenTierJSON
if json.Unmarshal(msg, &cur) != nil {
continue // one unreadable entry must not lose the others
}
t, perr := time.Parse(time.RFC3339, cur.ProvenAt)
if perr != nil {
continue
}
s.last[target] = provenTier{Archive: cur.Archive, At: t.UTC()}
}
return s
}
// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed. Only call this for a
// PASSING restore-test — the archive is what makes the tier not-due, so recording one for a failed
// run would retire the archive unproven.
func (s *RestoreTestState) RecordSuccess(target, archive string, t time.Time) error {
if target == "" {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
s.last[target] = provenTier{Archive: archive, At: t.UTC()}
return s.saveLocked()
}
// LastSuccess returns when this tier was last proven (ok=false = never).
func (s *RestoreTestState) LastSuccess(target string) (time.Time, bool) {
s.mu.Lock()
defer s.mu.Unlock()
p, ok := s.last[target]
return p.At, ok
}
// ProvenArchive returns the archive last PROVEN on this tier (ok=false = none — either never tested,
// or a legacy record carrying only a time). It is the due-check's whole question: an archive that is
// not this one has not been proven.
func (s *RestoreTestState) ProvenArchive(target string) (string, bool) {
s.mu.Lock()
defer s.mu.Unlock()
p, ok := s.last[target]
if !ok || p.Archive == "" {
return "", false
}
return p.Archive, true
}
// Snapshot returns a copy of the last-proven TIMES — for the host-report gauge.
func (s *RestoreTestState) Snapshot() map[string]time.Time {
s.mu.Lock()
defer s.mu.Unlock()
out := make(map[string]time.Time, len(s.last))
for k, v := range s.last {
out[k] = v.At
}
return out
}
// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
//
// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
// naturally prioritises a tier that has never been restore-tested at all — which on this fleet was
// the offsite tier, unproven for its entire existence.
//
// Ties break on target id so the order is deterministic; without that, two tiers proven in the same
// second would rotate by map iteration order, which is randomised in Go and would make the
// behaviour untestable and occasionally starving.
func (s *RestoreTestState) OldestFirst(targets []string) []string {
s.mu.Lock()
defer s.mu.Unlock()
out := append([]string(nil), targets...)
sort.SliceStable(out, func(i, j int) bool {
pi, oki := s.last[out[i]]
pj, okj := s.last[out[j]]
ti, tj := pi.At, pj.At
switch {
case !oki && !okj:
return out[i] < out[j] // both never proven → deterministic
case !oki:
return true // never proven wins
case !okj:
return false
case !ti.Equal(tj):
return ti.Before(tj)
default:
return out[i] < out[j]
}
})
return out
}
func (s *RestoreTestState) saveLocked() error {
raw := make(map[string]provenTierJSON, len(s.last))
for target, p := range s.last {
raw[target] = provenTierJSON{Archive: p.Archive, ProvenAt: p.At.UTC().Format(time.RFC3339)}
}
data, err := json.MarshalIndent(raw, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
return err
}
tmp := s.path + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, s.path)
}
+354
View File
@@ -0,0 +1,354 @@
package backup
import (
"context"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// R-85 Phase 2 — tier rotation, persisted state, and the one-heavy-operation gate.
//
// The failure this prevents is not hypothetical: demo-hp's DR tier reported `applied` with ZERO
// snapshots for five days and nobody noticed, because the scheduler could only ever see the primary
// tier. Rotation is what makes the offsite tier testable at all.
// rotRunner records which archives it was asked to restore.
type rotRunner struct {
mu sync.Mutex
archives []string
pass bool
}
func (r *rotRunner) RunRestoreTest(_ context.Context, spec reconcile.RestoreTestSpec) reconcile.RestoreTestResult {
r.mu.Lock()
defer r.mu.Unlock()
r.archives = append(r.archives, spec.Archive)
return reconcile.RestoreTestResult{
Archive: spec.Archive, SourceTier: spec.SourceTier,
Pass: r.pass, Verified: "boot+running",
}
}
func (r *rotRunner) seen() []string {
r.mu.Lock()
defer r.mu.Unlock()
return append([]string(nil), r.archives...)
}
// testLanded is a landing time old enough to be settled under any cutoff these tests use. R-86
// widened the TierPicker seam with the archive's landing time; the rotation tests below are about
// tier ORDER and the heavy-operation gate, not about settling, so they hold it constant.
var testLanded = time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC)
// archiveFor is a TierPicker over a fixed map: target → archive ("" = that tier holds none).
func archiveFor(m map[string]string) TierPicker {
return func(_ context.Context, target string, _ time.Time) (string, time.Time, error) {
a := m[target]
if a == "" {
return "", time.Time{}, nil
}
return a, testLanded, nil
}
}
func rotScheduler(t *testing.T, rr *rotRunner, st *RestoreTestState, pick TierPicker, gate *InFlight) *Scheduler {
t.Helper()
return NewScheduler(SchedulerOptions{
Runner: rr,
Store: NewStore(),
Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
},
Cadence: time.Hour,
Logger: quiet(),
Tiers: []string{"local", "felhom-pbs"},
TierPick: pick,
State: st,
InFlight: gate,
})
}
// ── SCENARIO A — both tiers get tested, each ONCE per archive ────────────────────────────────
//
// R-86 CHANGED THIS TEST'S CONTRACT, deliberately, and the old assertion is worth recording because
// it was a faithful statement of the defect. It read:
//
// 4 ticks → 4 runs, and consecutive runs must hit different tiers
//
// i.e. every tick produced a heavy restore-test, because the ticker WAS the trigger. Under R-86 a
// tick is an EVALUATION: both tiers are still exercised (rotation is intact), but a tier whose
// newest settled archive is already proven is not re-tested just because time passed. So the
// assertion is now 2 runs across 4 evaluations — one per tier, one per archive — which is a
// STRICTLY STRONGER statement: it pins both the coverage R-85 won and the pacing R-86 adds.
//
// COMPANION RED-PROOF (observed): restore the single-target picker — set `Tiers`/`TierPick` to nil
// so `pickForThisRun` falls back to `s.pick` on the primary runner — and this fails with
// "both tiers must be exercised; got [local:…]", i.e. the offsite tier never appears. That is
// pre-R-85 behaviour, and it is why demo-hp's DR tier went unproven for its entire existence.
func TestRotation_BothTiersExercisedOncePerArchive(t *testing.T) {
rr := &rotRunner{pass: true}
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
"local": "local:backup/vzdump-lxc-9201-x.tar.zst",
"felhom-pbs": "felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
}), &InFlight{})
s.now = func() time.Time { return time.Now().UTC() }
for i := 0; i < 4; i++ {
s.tick(context.Background())
}
got := rr.seen()
var sawLocal, sawPBS bool
for _, a := range got {
if len(a) >= 5 && a[:5] == "local" {
sawLocal = true
}
if len(a) >= 10 && a[:10] == "felhom-pbs" {
sawPBS = true
}
}
if !sawLocal || !sawPBS {
t.Fatalf("both tiers must be exercised; got %v", got)
}
// Exactly one run per tier: the archives never changed, so nothing became due a second time.
if len(got) != 2 {
t.Fatalf("want 2 runs across 4 evaluations (one per archive generation), got %d: %v", len(got), got)
}
if got[0] == got[1] {
t.Fatalf("the two runs must be different tiers — oldest-first is not ordering due tiers: %v", got)
}
}
// A tier with NO archive is skipped, not failed, and the other tier still runs. A brand-new offsite
// tier legitimately has nothing to restore; turning that into a failure would make every fresh box
// look broken for its first week.
func TestRotation_EmptyTierSkippedNotFailed(t *testing.T) {
rr := &rotRunner{pass: true}
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
"local": "local:backup/vzdump-lxc-9201-x.tar.zst",
"felhom-pbs": "", // provisioned but empty
}), &InFlight{})
s.tick(context.Background())
got := rr.seen()
if len(got) != 1 || got[0][:5] != "local" {
t.Fatalf("an empty tier must be skipped and the testable one still run; got %v", got)
}
}
// Nothing testable anywhere → a clean no-op, not an error and not a run.
func TestRotation_NoArchivesAnywhereIsANoOp(t *testing.T) {
rr := &rotRunner{pass: true}
s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
archiveFor(map[string]string{}), &InFlight{})
s.tick(context.Background())
if got := rr.seen(); len(got) != 0 {
t.Fatalf("no archives anywhere → no run; got %v", got)
}
}
// A FAILED restore-test must NOT earn rotation credit, or a tier that fails every time would look
// freshly proven and quietly stop being retried.
func TestRotation_FailureEarnsNoCredit(t *testing.T) {
rr := &rotRunner{pass: false}
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
"local": "local:backup/x.tar.zst",
"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
}), &InFlight{})
s.tick(context.Background())
if _, ok := st.LastSuccess("local"); ok {
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
}
if _, ok := st.LastSuccess("felhom-pbs"); ok {
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
}
}
// ── SCENARIO E — rotation survives a restart ─────────────────────────────────────────────────
//
// COMPANION RED-PROOF (observed): make the state in-memory (construct a fresh
// `NewRestoreTestState` on a DIFFERENT path for the second scheduler, i.e. lose the file) and this
// fails with "after a restart the OTHER tier must be next; got felhom-pbs" — the same tier repeats
// and the other is starved indefinitely, which with agent deploys as routine as they are is not a
// corner case.
func TestRotation_SurvivesRestart(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "rt.json")
picks := archiveFor(map[string]string{
"local": "local:backup/x.tar.zst",
"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
})
// First process: the OFFSITE tier is tested (never-proven sorts first).
rr1 := &rotRunner{pass: true}
st1 := NewRestoreTestState(path)
s1 := rotScheduler(t, rr1, st1, picks, &InFlight{})
s1.tick(context.Background())
first := rr1.seen()
if len(first) != 1 {
t.Fatalf("want one run, got %v", first)
}
// --- restart: brand-new state object reading the SAME file ---
rr2 := &rotRunner{pass: true}
st2 := NewRestoreTestState(path)
s2 := rotScheduler(t, rr2, st2, picks, &InFlight{})
s2.tick(context.Background())
second := rr2.seen()
if len(second) != 1 {
t.Fatalf("want one run after restart, got %v", second)
}
if second[0] == first[0] {
t.Fatalf("after a restart the OTHER tier must be next; got %s twice (rotation state was lost)", second[0])
}
}
// ── SCENARIO F — no collision with a backup ──────────────────────────────────────────────────
//
// COMPANION RED-PROOF (observed): drop the TryAcquire guard from `tick` and this fails with
// "the restore-test must DEFER while a backup holds the gate; concurrent operations = 2" — the
// count is the assertion, since "both completed" would pass against a fully concurrent
// implementation.
func TestRotation_DefersWhileABackupHoldsTheGate(t *testing.T) {
gate := &InFlight{}
release, _, ok := gate.TryAcquire("backup:felhom-pbs")
if !ok {
t.Fatal("precondition: the gate should have been free")
}
defer release()
rr := &rotRunner{pass: true}
s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
s.tick(context.Background())
concurrent := 1 + len(rr.seen()) // the backup holding the gate, plus anything the tick started
if concurrent != 1 {
t.Fatalf("the restore-test must DEFER while a backup holds the gate; concurrent operations = %d", concurrent)
}
}
// Once the backup releases, the next cadence proceeds — deferral must not be permanent.
func TestRotation_ResumesAfterTheGateFrees(t *testing.T) {
gate := &InFlight{}
release, _, _ := gate.TryAcquire("backup:local")
rr := &rotRunner{pass: true}
s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
s.tick(context.Background())
if len(rr.seen()) != 0 {
t.Fatal("should have deferred while held")
}
release()
s.tick(context.Background())
if len(rr.seen()) != 1 {
t.Fatalf("must resume once the gate frees; got %v", rr.seen())
}
}
// The gate itself: one holder at a time, named, and release is idempotent.
func TestInFlight_Semantics(t *testing.T) {
g := &InFlight{}
rel, busy, ok := g.TryAcquire("backup:local")
if !ok || busy != "" {
t.Fatalf("first acquire must succeed; ok=%v busy=%q", ok, busy)
}
if _, busy2, ok2 := g.TryAcquire("restore-test"); ok2 || busy2 != "backup:local" {
t.Fatalf("second acquire must fail and NAME the holder; ok=%v busy=%q", ok2, busy2)
}
rel()
rel() // idempotent — a double release must not free someone else's later claim
if g.Busy() != "" {
t.Fatalf("gate should be idle after release; busy=%q", g.Busy())
}
if _, _, ok3 := g.TryAcquire("restore-test"); !ok3 {
t.Fatal("gate must be reusable after release")
}
}
// A nil gate means "not wired" → no gating, pre-R-85 behaviour. Keeps every existing caller working.
func TestInFlight_NilIsUngated(t *testing.T) {
var g *InFlight
rel, _, ok := g.TryAcquire("x")
if !ok {
t.Fatal("a nil gate must not block")
}
rel()
if g.Busy() != "" {
t.Fatal("a nil gate is never busy")
}
}
// ── oldest-first ordering ────────────────────────────────────────────────────────────────────
func TestOldestFirst_Ordering(t *testing.T) {
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
now := time.Now().UTC()
// Never-proven sorts FIRST — the case that matters, since the offsite tier starts there.
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
// both never proven → deterministic tie-break by id
if got[0] != "felhom-pbs" && got[0] != "local" {
t.Fatalf("unexpected: %v", got)
}
}
_ = st.RecordSuccess("local", "local:backup/a.tar.zst", now)
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
}
_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", now.Add(time.Hour))
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
t.Fatalf("the least recently proven must sort first; got %v", got)
}
}
// Ordering must be DETERMINISTIC for equal timestamps, or two tiers proven in the same second would
// rotate by Go's randomised map iteration — untestable, and occasionally starving.
func TestOldestFirst_DeterministicOnTies(t *testing.T) {
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
now := time.Now().UTC()
_ = st.RecordSuccess("b-tier", "b:archive", now)
_ = st.RecordSuccess("a-tier", "a:archive", now)
for i := 0; i < 20; i++ {
if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
}
}
}
// The state file round-trips, and a corrupt file degrades to "nothing proven" rather than wedging.
func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "rt.json")
now := time.Now().UTC().Truncate(time.Second)
st := NewRestoreTestState(path)
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil {
t.Fatal(err)
}
reopened := NewRestoreTestState(path)
got, ok := reopened.LastSuccess("felhom-pbs")
if !ok || !got.Equal(now) {
t.Fatalf("state must round-trip; got %v ok=%v want %v", got, ok, now)
}
bad := filepath.Join(dir, "corrupt.json")
if err := os.WriteFile(bad, []byte("{{{not json"), 0o600); err != nil {
t.Fatal(err)
}
c := NewRestoreTestState(bad)
if _, ok := c.LastSuccess("felhom-pbs"); ok {
t.Fatal("a corrupt state file must degrade to 'nothing proven', not invent a timestamp")
}
}
+252 -10
View File
@@ -6,6 +6,7 @@ import (
"log/slog"
"sort"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
@@ -19,6 +20,8 @@ type BackupAPI interface {
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
StorageContent(ctx context.Context, store string) ([]proxmox.StorageContent, error)
// ListStorage enumerates storages (name+type) — used to scope local-only retention (never prune PBS).
ListStorage(ctx context.Context) ([]proxmox.Storage, error)
// TaskLogTail reads trailing task-log lines — used to read the ACTUAL vzdump mode
// (PVE may downgrade a requested snapshot to stop for a stopped guest — spike B1).
TaskLogTail(ctx context.Context, upid string, limit int) ([]string, error)
@@ -32,20 +35,90 @@ type BackupRunner struct {
target string // backup storage (content=backup)
mode proxmox.BackupMode // default ModeSnapshot
notes string // optional notes-template
logger *slog.Logger
now func() time.Time
// retention is the per-run `--prune-backups` spec (e.g. "keep-last=3") applied to a LOCAL target after
// each successful backup, so the agent's own backups can't pile up and refill root. Empty → no prune
// (the legacy behaviour; restore-test/selftest runners pass ""). NEVER applied to a PBS target.
retention string
// waitTimeout bounds the WaitTask poll on this runner's vzdump. Per-TIER since R-82: 30m is
// right for a local vzdump and badly wrong for an offsite PBS upload (see the 2026-07-26 live
// failure recorded on config.BackupTargetConfig.WaitTimeoutSeconds). 0 → 30m (legacy).
waitTimeout time.Duration
// allowPBSPrune permits `--prune-backups` on a PBS-type target. OFF by default and ON only for
// an ADDITIONAL tier whose keep_last was set explicitly (operator ruling 2026-07-26: keep two
// weeks of weekly offsite backups).
//
// The blanket PBS refusal it replaces existed for a real reason and still applies to the
// PRIMARY tier: BackupTarget() DEFAULTS to "felhom-pbs" and KeepLast() DEFAULTS to 3, so a box
// with neither key set would silently prune its offsite DR to 3 restore points. An additional
// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
// there is a deliberate act.
allowPBSPrune bool
logger *slog.Logger
now func() time.Time
// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
// due-check is served from the local-API handler goroutines.
rejectedMu sync.Mutex
rejected map[string]struct{}
}
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support.
func NewBackupRunner(api BackupAPI, target string, mode proxmox.BackupMode, notes string, logger *slog.Logger) *BackupRunner {
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
func NewBackupRunner(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, logger *slog.Logger) *BackupRunner {
return NewBackupRunnerWithWait(api, target, mode, notes, retention, 0, logger)
}
// NewBackupRunnerWithWait is NewBackupRunner plus an explicit vzdump wait bound (0 → 30m).
func NewBackupRunnerWithWait(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, logger *slog.Logger) *BackupRunner {
return NewBackupRunnerFull(api, target, mode, notes, retention, waitTimeout, false, logger)
}
// NewBackupRunnerFull is the full constructor. allowPBSPrune must be true ONLY for an additional
// tier with an explicitly configured keep_last — see BackupRunner.allowPBSPrune.
func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, allowPBSPrune bool, logger *slog.Logger) *BackupRunner {
if mode == "" {
mode = proxmox.ModeSnapshot
}
if logger == nil {
logger = slog.Default()
}
return &BackupRunner{api: api, target: target, mode: mode, notes: notes, logger: logger, now: func() time.Time { return time.Now().UTC() }}
if waitTimeout <= 0 {
waitTimeout = 30 * time.Minute
}
return &BackupRunner{api: api, target: target, mode: mode, notes: notes, retention: retention,
waitTimeout: waitTimeout, allowPBSPrune: allowPBSPrune, logger: logger,
now: func() time.Time { return time.Now().UTC() }}
}
// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
// applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a
// separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type
// can't be confirmed (lookup error / not found), it SKIPS pruning rather than risk pruning PBS — the
// host_disk + storage_fill detectors remain the safety net.
func (r *BackupRunner) localPruneSpec(ctx context.Context) string {
if r.retention == "" {
return ""
}
stores, err := r.api.ListStorage(ctx)
if err != nil {
r.logger.Warn("backup: could not resolve target storage type — skipping local prune this run", "target", r.target, "err", err)
return ""
}
for _, s := range stores {
if s.Storage == r.target {
if s.Type == "pbs" && !r.allowPBSPrune {
// Not opted in → never prune the offsite DR (the pre-R-82 rule, and still the rule
// for the primary tier, whose target+retention both DEFAULT and could prune by
// accident).
return ""
}
return r.retention
}
}
r.logger.Warn("backup: target storage not found in list — skipping local prune this run (fail-safe)", "target", r.target)
return ""
}
// snapshotMarker is the vzdump task-log line that signals the storage snapshot has been created
@@ -99,6 +172,7 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
})
if err != nil {
rec.Error = err.Error()
@@ -113,7 +187,7 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
defer stopWatch()
go r.watchForSnapshot(watchCtx, upid, onSnapshot)
}
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: 30 * time.Minute}); err != nil {
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil {
rec.Error = err.Error()
rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
@@ -176,18 +250,69 @@ func (r *BackupRunner) watchForSnapshot(ctx context.Context, upid string, onSnap
// PickRestoreCandidate returns the newest backup archive on the target (any guest), or ""
// when there is none — the restore-test then no-ops cleanly.
func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error) {
contents, err := r.api.StorageContent(ctx, r.target)
return r.PickRestoreCandidateOn(ctx, r.target)
}
// PickRestoreCandidateOn is PickRestoreCandidate for an ARBITRARY tier's storage (R-85 1.2), so the
// scheduler can rotate across tiers instead of only ever seeing this runner's own target.
//
// Contract preserved: "" + nil error when the storage holds no archive. **A tier with nothing to
// restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and turning
// that into a failure would make every fresh box look broken for its first week.
func (r *BackupRunner) PickRestoreCandidateOn(ctx context.Context, target string) (string, error) {
archive, _, err := r.PickSettledRestoreCandidateOn(ctx, target, time.Time{})
return archive, err
}
// PickSettledRestoreCandidateOn is the R-86 due-check's picker: the newest archive on target that
// landed AT OR BEFORE notAfter (the settle cutoff), with the time it landed. A zero notAfter means
// "no cutoff" — that is the pre-R-86 behaviour, which is why PickRestoreCandidateOn is now a
// one-line call into this and its contract is untouched (one scan, one owner).
//
// WHY A CUTOFF AT ALL. An archive that landed minutes ago may still be settling — R-71a's
// settle-gate exists because the offsite tier's day-0 consume raced its own floor update — and
// restore-testing the archive a backup is still writing proves nothing about the backup that
// finished. The due-check therefore asks about the newest SETTLED archive, and §8.1's rule is built
// on that: the tier is due when a settled archive exists that has not been proven.
//
// The plausibility floor is applied here and not in the old path on purpose. Under R-86 the picked
// archive becomes the tier's due-ness: an incomplete 1-byte phantom (F-CRIT-2's artefact — server
// prune does NOT collect it) would be selected forever, fail its restore forever, never earn proof,
// and so make the tier due at EVERY evaluation. Skipping it is what keeps the retry rate bounded by
// the archive generation rather than by the evaluation interval.
//
// Contract preserved: ("", zero, nil) when the storage holds no eligible archive. **A tier with
// nothing to restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and
// turning that into a failure would make every fresh box look broken for its first week.
func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target string, notAfter time.Time) (string, time.Time, error) {
if target == "" {
return "", time.Time{}, nil
}
contents, err := r.api.StorageContent(ctx, target)
if err != nil {
return "", err
return "", time.Time{}, err
}
var best string
var bestCTime int64 = -1
for _, e := range contents {
if e.Content == "backup" && e.CTime > bestCTime {
if e.Content != "backup" {
continue
}
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
continue // not settled yet — a newer archive is not a reason to re-prove an older one
}
if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why)
continue
}
if e.CTime > bestCTime {
bestCTime, best = e.CTime, e.VolID
}
}
return best, nil
if best == "" {
return "", time.Time{}, nil
}
return best, time.Unix(bestCTime, 0).UTC(), nil
}
// latestArchive finds the newest backup archive volid + size for vmid on the target.
@@ -209,6 +334,121 @@ func (r *BackupRunner) latestArchive(ctx context.Context, vmid int) (string, int
return vol, size, nil
}
// NewestArchiveTime reports when this guest's newest backup archive LANDED ON THIS TARGET, from the
// storage itself. ok=false means the target genuinely holds no archive for this guest.
//
// R-84: this is the cure for the redundant-backup-after-restart problem. The agent's backup Store is
// in-memory ("lost on restart; the cadence re-populates"), so after every restart /backup/due
// reported "no successful backup recorded yet" and the controller dutifully took another one. On the
// local tier that is wasted minutes; on the OFFSITE tier it is a wasted multi-hour WAN upload after
// every agent deploy — and agent deploys are routine. Three redundant local backups were observed on
// minPlausibleArchiveBytes is the floor below which a storage entry cannot be a real whole-guest
// backup and is therefore treated as an INCOMPLETE artefact rather than a successful one.
//
// MEASURED, not chosen by feel — fleet survey 2026-07-28 (Campaign 8, finding F-CRIT-2):
//
// smallest REAL backup anywhere on the fleet ... 612,397,450 B (~584 MiB, a guest-9100 vzdump)
// demo-hp local / PBS ..................... 1.59 GB / 4.35-4.37 GB
// demo-felhom local / PBS ..................... 5.82-5.84 GB / 14.47-14.51 GB
// the phantom left by a PBS daemon killed mid-upload ....... 1 B
//
// 1 MiB sits 584x below the smallest real backup and 1,048,576x above the phantom. The two
// populations are nine orders of magnitude apart, so this floor cannot plausibly clip a real
// archive — which is the property that matters, because a floor set too HIGH does not merely lose
// safety margin, it causes fleet-wide backup THRASH (see archivePlausiblyComplete).
const minPlausibleArchiveBytes int64 = 1 << 20
// archivePlausiblyComplete reports whether a storage entry can be a COMPLETE backup, and if not,
// why. Pure, so the contract is unit-testable without a storage.
//
// WHY SIZE, AND NOTHING ELSE. The richer PBS fields look like better discriminators and are all
// traps, because this runner is TIER-AGNOSTIC — the same predicate runs against a PBS datastore and
// against a plain `dir` storage (verified against the live PVE API, 2026-07-28):
//
// - `verification` is absent on the phantom, but ALSO absent on every local (dir) archive — a dir
// storage has no verification concept — and absent on a good PBS snapshot until verify-new
// catches up. Gating on it would reject 100% of local backups and every freshly-taken offsite
// one: continuous re-backup across the fleet.
// - `encrypted` fails the same way, and for the same reason.
// - `notes` happens to be present on both good tiers today only because the agent sets it; an
// archive written by any other path lacks it. Too fragile to gate freshness on.
//
// Size is the only signal that means the same thing on every tier.
//
// THE FAIL-SAFE DIRECTION, stated explicitly: when completeness cannot be established the entry is
// NOT counted as a successful backup. That errs toward the tier looking LESS fresh, and its worst
// case is one extra backup. Counting an undecidable entry is precisely the F-CRIT-2 defect — a
// failed upload that made its tier look freshly backed up and silenced it for a full cadence.
func archivePlausiblyComplete(e proxmox.StorageContent) (bool, string) {
if e.Size < minPlausibleArchiveBytes {
return false, fmt.Sprintf("size %d B is below the %d B plausibility floor — an aborted/incomplete archive, not a successful backup",
e.Size, minPlausibleArchiveBytes)
}
return true, ""
}
// warnRejectedArchiveOnce announces a rejected archive at WARN exactly once per distinct volid.
//
// A rejected archive must never be silent: a tier that quietly ignores the newest entry on its
// storage is a new quiet path, and quiet paths are what F-CRIT-2 was. But the due-check runs every
// 5 minutes and a phantom persists indefinitely — server-side prune does NOT collect it (verified
// by dry-run 2026-07-28: with keep-last 2 it retained two real snapshots PLUS the phantom) — so
// logging per poll would emit ~288 identical lines a day and bury the one that matters.
func (r *BackupRunner) warnRejectedArchiveOnce(e proxmox.StorageContent, why string) {
r.rejectedMu.Lock()
if r.rejected == nil {
r.rejected = map[string]struct{}{}
}
_, seen := r.rejected[e.VolID]
if !seen {
r.rejected[e.VolID] = struct{}{}
}
r.rejectedMu.Unlock()
if seen {
return
}
r.logger.Warn("backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup",
"target", r.target, "vmid", e.VMID, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
}
// demo-felhom in a single afternoon of deploys (2026-07-26).
//
// Asking the STORAGE rather than persisting the store is deliberate:
// - it is ground truth, not remembered state — if an archive was pruned or deleted it correctly
// stops counting, whereas a persisted record would keep claiming a backup that no longer exists;
// - it needs no new on-disk state and no migration;
// - it is the same source `latestArchive` already trusts to build the post-backup record.
//
// It answers ONLY "when did a plausibly COMPLETE backup last land", which is exactly what the
// due-check needs. Completeness is not optional here: PBS publishes an aborted upload into the same
// listing (manifest-less, 1 byte, and NEWEST), and counting it made the tier report fresh and go
// silent for a whole cadence — F-CRIT-2. Presence is not validity. The
// richer fields (size, duration, uncovered volumes, error) stay with the real in-memory records — a
// synthesized record would put invented numbers into the host-report.
func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Time, bool, error) {
contents, err := r.api.StorageContent(ctx, r.target)
if err != nil {
return time.Time{}, false, err
}
var best int64 = -1
for _, e := range contents {
if e.Content != "backup" || e.VMID != vmid {
continue
}
if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why)
continue
}
if e.CTime > best {
best = e.CTime
}
}
if best < 0 {
return time.Time{}, false, nil
}
return time.Unix(best, 0).UTC(), true, nil
}
// parseBackupMode extracts the actual mode from a vzdump task log line `… backup mode: <x>`
// (e.g. "INFO: backup mode: stop"). Returns "" if not found.
func parseBackupMode(lines []string) string {
@@ -274,6 +514,8 @@ func ToHubRestoreTest(res reconcile.RestoreTestResult, testedAt time.Time) hub.R
DurationSeconds: res.Duration.Seconds(),
Warnings: res.StartWarnings,
WarningsRecognized: res.WarningsRecognized,
MountParity: res.MountParity,
MountInventory: res.MountInventory,
}
if res.Err != nil {
rt.Error = res.Err.Error()
+2 -2
View File
@@ -22,7 +22,7 @@ func TestBackupWithSnapshotHook_FiresOnMarker(t *testing.T) {
logLines: []string{"INFO: backup mode: snapshot", "INFO: create storage snapshot 'vzdump'"},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9001-x", Content: "backup", VMID: 9001, Size: 100, CTime: 1}},
}
r := NewBackupRunner(api, "local", "", "", quiet())
r := NewBackupRunner(api, "local", "", "", "", quiet())
var fired int32
done := make(chan struct{})
@@ -55,7 +55,7 @@ func TestBackupWithSnapshotHook_StopMode_NeverFires(t *testing.T) {
logLines: []string{"INFO: backup mode: stop"}, // downgraded; no snapshot marker
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9001-x", Content: "backup", VMID: 9001, Size: 100, CTime: 1}},
}
r := NewBackupRunner(api, "local", "", "", quiet())
r := NewBackupRunner(api, "local", "", "", "", quiet())
var fired int32
done := make(chan struct{})
+194 -29
View File
@@ -18,27 +18,77 @@ type RestoreTestRunner interface {
// there is none yet (the tick then no-ops).
type CandidatePicker func(ctx context.Context) (string, error)
// SpecBuilder yields the RestoreTestSpec for ONE run, given the archive that was picked.
//
// R-85 (1.1): this REPLACES a frozen spec value. It used to be built by an immediately-invoked
// function at daemon start, so `storageTier()` and `restoreTaskTimeout()` were evaluated ONCE and
// the resulting value reused for every run for the lifetime of the process. Two consequences:
// - nothing tier-varying was expressible at all (the offsite tier could never be scheduled), and
// - it was a latent staleness bug in its own right — a storage-type or config change did not take
// effect until the daemon restarted.
//
// The archive is passed in because the tier MUST be derived from it (the v0.100.0 rule), never from
// the configured target: deriving it from config is what produced the 600 s false failure when a
// PBS archive was classified "local" and got the 10-minute local wait.
type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
// TierPicker resolves the newest archive on a NAMED tier that landed AT OR BEFORE notAfter (the
// settle cutoff), together with when it landed. (*BackupRunner).PickSettledRestoreCandidateOn
// satisfies it. A zero notAfter means "no settle requirement".
//
// R-86 widened this seam from (target) → archive. The landing time is what makes the due-check's
// verdict explainable — "archive X, which landed at T, has not been proven" — and the cutoff is
// what makes the rule per-ARCHIVE-GENERATION instead of per-interval. "" must NOT be an error: a
// brand-new offsite tier legitimately has nothing to restore yet.
type TierPicker func(ctx context.Context, target string, notAfter time.Time) (archive string, landed time.Time, err error)
// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
// AND a valid scratch band is configured (validated by the caller before construction).
type Scheduler struct {
runner RestoreTestRunner
pick CandidatePicker
store *Store
spec reconcile.RestoreTestSpec // archive is filled per-tick
runner RestoreTestRunner
pick CandidatePicker
store *Store
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
// cadence is the EVALUATION interval (R-86) — how often "is anything due?" is asked. It is no
// longer the thing that decides a test happens; see restoretest_due.go.
cadence time.Duration
logger *slog.Logger
now func() time.Time
// settle is how long an archive must have sat before it is a candidate (R-86).
settle time.Duration
logger *slog.Logger
now func() time.Time
// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
// (single tier via `pick`), which keeps every existing caller and test working untouched.
tiers []string // configured tier target ids, primary first
tierPick TierPicker // newest archive on a named tier
rtState *RestoreTestState // persisted last-successful-per-tier (drives oldest-first)
inFlight *InFlight // shared with the backup path — Scenario F
}
// SchedulerOptions configures a Scheduler.
type SchedulerOptions struct {
Runner RestoreTestRunner
Pick CandidatePicker
Store *Store
Spec reconcile.RestoreTestSpec // RestoreStorage, ScratchMin/Max, SourceTier, BootTimeout
Cadence time.Duration // 0 → disabled
Logger *slog.Logger
Runner RestoreTestRunner
Pick CandidatePicker
Store *Store
// Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
// picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
Spec SpecBuilder
// Cadence is the EVALUATION interval — how often due-ness is asked, NOT how often a test runs
// (R-86). 0 → disabled.
Cadence time.Duration
// Settle is how long an archive must have sat before it is a restore-test candidate (R-86).
// 0 → no settle requirement (any archive is a candidate).
Settle time.Duration
Logger *slog.Logger
// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
// named tier; State persists last-successful-per-tier; InFlight is the shared one-heavy-op gate.
Tiers []string
TierPick TierPicker
State *RestoreTestState
InFlight *InFlight
}
// NewScheduler builds a Scheduler.
@@ -48,27 +98,43 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
logger = slog.Default()
}
return &Scheduler{
runner: opts.Runner,
pick: opts.Pick,
store: opts.Store,
spec: opts.Spec,
cadence: opts.Cadence,
logger: logger,
now: func() time.Time { return time.Now().UTC() },
runner: opts.Runner,
pick: opts.Pick,
store: opts.Store,
spec: opts.Spec,
cadence: opts.Cadence,
settle: opts.Settle,
logger: logger,
now: func() time.Time { return time.Now().UTC() },
tiers: append([]string(nil), opts.Tiers...),
tierPick: opts.TierPick,
rtState: opts.State,
inFlight: opts.InFlight,
}
}
// Run fires a restore-test on the cadence until ctx is cancelled. A 0 cadence disables it
// (the goroutine just waits for shutdown). It does NOT fire immediately on start (a restore
// is heavy; the first runs one interval in) — on-demand runs use the selftest harness.
// Run EVALUATES due-ness on the interval until ctx is cancelled, and runs a restore-test only when
// a tier is actually due (R-86). A 0 interval disables it (the goroutine just waits for shutdown).
//
// The ticker survives as the evaluation interval and nothing else. It is emphatically NOT the
// trigger any more: its phase is the process's uptime, and agent deploys reset it, which is exactly
// the defect R-86 removes. What decides that a test happens is `EvaluateDue`.
//
// It still does NOT evaluate immediately on start — the first evaluation is one interval in. That
// is an EARNED restraint, kept deliberately: a restore is heavy, agent restarts are routine, and a
// crash-loop that evaluated at start would hammer a permanently-failing tier as fast as it could
// restart. Due-ness does not expire while we wait, so the only cost is up to one interval of
// latency on a tier that just became due. On-demand runs use `--selftest=restore-test`.
//
// Returns nil on ctx cancellation.
func (s *Scheduler) Run(ctx context.Context) error {
if s.cadence <= 0 || s.runner == nil || s.pick == nil {
if s.cadence <= 0 || s.runner == nil || s.spec == nil || (s.pick == nil && !s.rotating()) {
s.logger.Info("backup: restore-test cadence disabled")
<-ctx.Done()
return nil
}
s.logger.Info("backup: restore-test scheduler starting", "cadence", s.cadence)
s.logger.Info("backup: restore-test scheduler starting (per-archive due-check)",
"eval_interval", s.cadence, "settle", s.settle)
t := time.NewTicker(s.cadence)
defer t.Stop()
for {
@@ -82,19 +148,56 @@ func (s *Scheduler) Run(ctx context.Context) error {
}
}
// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when
// no backup exists yet. Deterministic given s.now — tests call it directly.
// tick is ONE EVALUATION: gate → due-check → run the first due tier → record which archive was
// proven. No-ops cleanly when nothing is due, when no backup exists yet, or when a heavy operation
// is already in flight. Deterministic given s.now — tests call it directly.
//
// One run per evaluation, by construction (Scenario F): a second due tier is left DUE and picked up
// by the next evaluation. Deferred, never cancelled, and never two multi-GB restores over one link.
func (s *Scheduler) tick(ctx context.Context) {
archive, err := s.pick(ctx)
if s.spec == nil {
// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
// reachable directly. Skipping loudly beats panicking the daemon goroutine — a missing
// spec must cost a restore-test, never the agent.
s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
return
}
// The due-check runs BEFORE the gate is taken, and that ORDER is load-bearing under R-86.
//
// It used to be the other way round, and correctly so: the gate was held for one heavy run a
// day, and the candidate lookup rode along inside it. Evaluations are now frequent, and the
// lookup is a storage listing that for the offsite tier crosses the WAN. Holding the
// one-heavy-operation gate for a read that answers "nothing to do" would open a small window at
// EVERY evaluation in which a starting backup cannot acquire — and a backup that cannot acquire
// does not merely wait, it records a failure and pages the operator (F-A1). A cheap poll must
// not be able to manufacture that.
//
// Nothing is lost by checking first: due-ness does not expire, and the gate is still taken
// before anything heavy begins.
archive, target, err := s.pickForThisRun(ctx)
if err != nil {
s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
return
}
if archive == "" {
s.logger.Info("backup: restore-test skipped; no backup available yet")
s.logger.Debug("backup: restore-test not due this evaluation")
return
}
spec := s.spec
// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
// never cancel what is already running: a deferred restore-test costs hours of coverage, a
// cancelled backup costs the backup. A deferred tier stays DUE, so the next evaluation retries it.
release, busy, ok := s.inFlight.TryAcquire("restore-test")
if !ok {
s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight",
"busy", busy, "target", target, "archive", archive)
return
}
defer release()
// R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
spec := s.spec(ctx, archive)
spec.Archive = archive
res := s.runner.RunRestoreTest(ctx, spec)
if res.Skipped {
@@ -102,6 +205,15 @@ func (s *Scheduler) tick(ctx context.Context) {
}
rt := ToHubRestoreTest(res, s.now())
s.store.RecordRestoreTest(rt)
// Rotation credit is given ONLY on success. A failing tier must keep sorting first, or a tier
// that fails every time would look freshly proven and quietly stop being retried.
if rt.Pass && s.rtState != nil && target != "" {
// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
// not-due until a NEWER archive settles, and what makes a proof survive a restart.
if err := s.rtState.RecordSuccess(target, archive, s.now()); err != nil {
s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
}
}
switch {
case !rt.Pass:
// A failing restore-test is the loudest DR signal there is.
@@ -118,3 +230,56 @@ func (s *Scheduler) tick(ctx context.Context) {
"archive", rt.SourceArchive, "duration_s", rt.DurationSeconds, "warnings", res.StartWarnings)
}
}
// rotating reports whether multi-tier rotation is wired.
func (s *Scheduler) rotating() bool { return len(s.tiers) > 0 && s.tierPick != nil }
// pickForThisRun chooses the tier to test THIS evaluation: the first DUE tier, in oldest-proven
// order.
//
// R-86 changed what this answers. It used to answer "whose turn is it?", and the answer was always
// somebody's — the ticker had fired, so a test was going to happen. It now answers "is anything
// due?", and "nothing" is a normal, frequent and correct answer.
//
// OLDEST-FIRST (operator ruling 2026-07-26, Option 1) survives as the ORDER among due tiers: the
// tier whose last successful restore-test is oldest goes first, never-proven first of all. It is
// self-balancing, needs no config knob, and it still cannot starve a tier — but it no longer decides
// that a test happens at all.
//
// A tier with no settled archive is SKIPPED, not failed — a brand-new offsite tier has nothing to
// restore yet, and that is normal, not broken. A tier whose archives cannot be LISTED is likewise
// skipped, loudly, and its error is returned only when no other tier was testable: one tier's
// storage being unreadable must not cost the other tier its proof, and must not be silent either.
//
// Returns ("", "", nil) when nothing anywhere is due.
func (s *Scheduler) pickForThisRun(ctx context.Context) (archive, target string, err error) {
if !s.rotating() {
// Pre-R-85 single-tier path (tests and any caller that wires only `Pick`): there is no tier
// identity and no persisted proof here, so there is nothing to compare an archive against
// and no due-check is possible. It runs on every evaluation, exactly as it always did.
a, perr := s.pick(ctx)
return a, "", perr
}
var firstErr error
for _, v := range s.EvaluateDue(ctx) {
if v.Err != nil {
s.logger.Warn("backup: restore-test candidate lookup failed for a tier; trying the next",
"target", v.Target, "err", v.Err)
if firstErr == nil {
firstErr = v.Err
}
continue
}
if !v.Due {
s.logger.Debug("backup: restore-test tier is not due", "target", v.Target, "reason", v.Reason)
continue
}
s.logger.Info("backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)",
"target", v.Target, "archive", v.Archive, "landed", v.Landed.Format(time.RFC3339), "reason", v.Reason)
return v.Archive, v.Target, nil
}
if firstErr != nil {
return "", "", firstErr
}
return "", "", nil
}
+124
View File
@@ -0,0 +1,124 @@
package backup
import (
"context"
"fmt"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// R-85 (1.1) — the spec is built PER RUN, never frozen at construction.
//
// It used to be an immediately-invoked function at daemon start, so storageTier() and
// restoreTaskTimeout() were evaluated ONCE and the value reused for every run for the process
// lifetime. That is what made an offsite restore-test impossible to schedule at all, and it was a
// latent staleness bug besides: a storage-type or config change did not take effect until restart.
type specSpy struct {
mu sync.Mutex
calls int
archives []string
tiers []string // what the builder decided, per call
}
func (sp *specSpy) build(_ context.Context, archive string) reconcile.RestoreTestSpec {
sp.mu.Lock()
defer sp.mu.Unlock()
sp.calls++
sp.archives = append(sp.archives, archive)
// Decide the tier from the ARCHIVE, exactly as main.go does (the v0.100.0 rule).
tier := "local"
if len(archive) > 10 && archive[:10] == "felhom-pbs" {
tier = "pbs"
}
sp.tiers = append(sp.tiers, tier)
return reconcile.RestoreTestSpec{
RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: tier,
}
}
// COMPANION RED-PROOF (observed): change Scheduler.spec back to a frozen
// `reconcile.RestoreTestSpec` value captured at construction → this fails with
// "the spec builder must run ONCE PER RUN, got 1 call(s) across 3 ticks", because a frozen value is
// evaluated exactly once no matter how many ticks fire. Restored.
func TestScheduler_SpecIsBuiltPerRun(t *testing.T) {
sp := &specSpy{}
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
n := 0
s := NewScheduler(SchedulerOptions{
Runner: rt,
Pick: func(context.Context) (string, error) {
n++
return fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", n), nil
},
Store: NewStore(),
Spec: sp.build,
Cadence: time.Hour,
Logger: quiet(),
})
for i := 0; i < 3; i++ {
s.tick(context.Background())
}
sp.mu.Lock()
defer sp.mu.Unlock()
if sp.calls != 3 {
t.Fatalf("the spec builder must run ONCE PER RUN, got %d call(s) across 3 ticks", sp.calls)
}
// And it must see the archive THIS run picked — not a stale one.
for i, a := range sp.archives {
want := fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", i+1)
if a != want {
t.Fatalf("run %d: builder saw archive %q, want %q — the spec is not tracking the picked archive", i+1, a, want)
}
}
}
// The tier must follow the ARCHIVE across runs. A builder that saw only the configured target would
// return the same tier every time — which is exactly the v0.100.0 defect that killed a 14.46 GB WAN
// restore at the 10-minute local bound.
func TestScheduler_SpecTierFollowsTheArchive(t *testing.T) {
sp := &specSpy{}
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
archives := []string{
"local:backup/vzdump-lxc-9201-x.tar.zst",
"felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
}
i := 0
s := NewScheduler(SchedulerOptions{
Runner: rt,
Pick: func(context.Context) (string, error) {
a := archives[i%len(archives)]
i++
return a, nil
},
Store: NewStore(), Spec: sp.build, Cadence: time.Hour, Logger: quiet(),
})
s.tick(context.Background())
s.tick(context.Background())
sp.mu.Lock()
defer sp.mu.Unlock()
if len(sp.tiers) != 2 || sp.tiers[0] != "local" || sp.tiers[1] != "pbs" {
t.Fatalf("the tier must follow the archive per run; got %v", sp.tiers)
}
}
// A nil spec builder must SKIP loudly, not panic — a wiring bug costs a restore-test, never the
// daemon goroutine.
func TestScheduler_NilSpecSkipsInsteadOfPanicking(t *testing.T) {
rt := &fakeRTRunner{}
s := NewScheduler(SchedulerOptions{
Runner: rt,
Pick: func(context.Context) (string, error) { return "vol", nil },
Store: NewStore(), Cadence: time.Hour, Logger: quiet(),
})
s.tick(context.Background()) // must not panic
if rt.runs != 0 {
t.Fatalf("a nil spec must not run a restore-test; got %d run(s)", rt.runs)
}
}
+195
View File
@@ -0,0 +1,195 @@
// Package capability is the agent's privileged-capability self-check (slice 1 of agent
// self-health). It declares the MANIFEST — the (binary, representative-arg-vector) pairs the
// non-root agent depends on running via `sudo -n` — and a PROBE that lists each against the live
// sudoers policy (`sudo -n -l`, never executing) + checks the binary exists. The result is a
// snapshot the agent attaches to its hub report; the hub owns the ok→degraded transition + alert.
//
// Why this exists: the 2026-06-28 root→non-root cutover dropped several grants from
// configs/felhom-agent.sudoers (lxc-info, make-private, restart dnsmasq, …). Each broke a feature
// silently until a user hit it (the multi-drive flapping incident, audit 2026-06-29). A non-root
// agent that can't run a command it depends on is DEGRADED and must SAY so — at cutover, not days
// later. The companion build-time test (manifest_test.go) asserts every manifest vector is covered
// by a sudoers pattern, catching authoring gaps in CI before they ship.
package capability
import (
"strings"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// Capability is one privileged command the agent depends on. Name is a stable id; Feature is the
// human-readable thing that breaks if the grant is missing (used in logs + the operator alert).
// Binary is the absolute path the runner invokes; ReprArgs is a CONCRETE argument vector that
// matches the corresponding sudoers glob (e.g. a vmid "9201" matches `[0-9]*`, a device "/dev/sda"
// matches `/dev/*`). Critical marks the user-facing ones — the hub alerts only when a Critical
// capability is degraded (non-critical degradations still ride the report snapshot + agent log).
type Capability struct {
Name string
Feature string
Binary string
ReprArgs []string
Critical bool
// GatedBy names the config gate this capability sits behind ("" = always active). A healthy
// probe of a gated capability whose gate is OFF reports StatusInactive instead of ok — see
// Prober.GateActive. Broken plumbing stays degraded regardless of the gate.
GatedBy string
}
// GatePBSDR is the gate id for the PBS-DR tier: active when this box carries a live (enabled)
// pbs_dr descriptor or a previously-converged marker (pbsdr.Manager.DRConfigured).
const GatePBSDR = "pbs_dr"
// Manifest is the required set, seeded from the 2026-06-29 sudoers audit (felhom-agent/REPORT.md):
// the OK + newly-CLOSED rows. The SURFACED/DEFERRED rows are deliberately EXCLUDED — they are not
// required capabilities: the general `pct exec <vmid> -- *` (controller-swap; arbitrary exec, an
// open operator decision), `pct create` (golden build, maintenance, no daemon caller), `mount
// UUID=…` (legacy/unreferenced), and the callerless `sensors -j`. Adding them here would assert
// grants the agent neither has nor should depend on.
//
// Each ReprArgs is a representative instance; the probe LISTS it (`sudo -n -l`) and never runs it,
// so even mkfs/pct-set entries are side-effect-free to probe.
//
// Gating (v0.86.0): every `pbsdr-*` capability is behind GatePBSDR — the whole pbsdr vector IS
// the DR-tier apply path, so the gate rides the stable name prefix (a new pbsdr-* op is gated
// automatically; nothing else ever is). Applied here, not in the literals, to keep the manifest
// entries positional.
func Manifest() []Capability {
out := make([]Capability, len(manifest))
copy(out, manifest)
for i := range out {
if strings.HasPrefix(out[i].Name, "pbsdr-") {
out[i].GatedBy = GatePBSDR
}
}
return out
}
// reprFingerprint is a shape-valid all-zero SHA-256 colon fingerprint for list-mode repr vectors
// (matches the wrapper's fingerprint validation; never executed).
const reprFingerprint = "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"
var manifest = []Capability{
// ---- Intermediary drive model (the multi-drive path — mostly Critical) ----
{"guest-init-pid", "drive-gate guest-sees check (multi-drive concurrency)", "/usr/bin/lxc-info", []string{"-n", "9201", "-p", "-H"}, true, ""},
{"parent-self-bind", "intermediary shared-parent self-bind", "/usr/bin/mount", []string{"--bind", "/mnt/felhom-drives", "/mnt/felhom-drives"}, true, ""},
{"parent-make-shared", "intermediary shared-parent propagation", "/usr/bin/mount", []string{"--make-shared", "/mnt/felhom-drives"}, true, ""},
{"parent-make-private", "intermediary shared-parent peer-group isolation", "/usr/bin/mount", []string{"--make-private", "/mnt/felhom-drives"}, true, ""},
{"drive-bind", "drive attach (felhom-data bind under parent)", "/usr/bin/mount", []string{"--bind", "/mnt/felhom-usb/felhom-data", "/mnt/felhom-drives/felhom-usb"}, true, ""},
{"drive-umount", "drive detach (fail-closed unmount)", "/usr/bin/umount", []string{"/mnt/felhom-drives/felhom-usb"}, true, ""},
{"drives-mkdir-parent", "stable parent dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives"}, false, ""},
{"drives-mkdir-sub", "per-drive stable dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives/felhom-usb"}, false, ""},
{"drives-mkdir-data", "felhom-data namespace create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-usb/felhom-data"}, false, ""},
{"drives-chown-data", "felhom-data guest-root chown", "/usr/bin/chown", []string{"100000:100000", "/mnt/felhom-usb/felhom-data"}, false, ""},
{"parent-script-install", "shared-parent boot script install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-shared-parent-123456789.sh", "/usr/local/sbin/felhom-shared-parent.sh"}, false, ""},
{"parent-unit-install", "shared-parent boot unit install", "/usr/bin/install", []string{"-m", "0644", "--", "/tmp/felhom-shared-parent-123456789.service", "/etc/systemd/system/felhom-shared-parent.service"}, false, ""},
{"parent-unit-enable", "shared-parent boot-persistence enable", "/usr/bin/systemctl", []string{"enable", "felhom-shared-parent.service"}, false, ""},
{"parent-bind-mp8", "parent bind into guest at provision", "/usr/sbin/pct", []string{"set", "9201", "-mp8", "/mnt/felhom-drives"}, false, ""},
// ---- Disk inspect / format gate (Critical: the data-bearing classifier + format) ----
{"disk-blkid", "disk data-bearing classify (format gate)", "/usr/sbin/blkid", []string{"-p", "-o", "export", "/dev/sda"}, true, ""},
{"disk-lsblk", "disk topology read (format gate)", "/usr/bin/lsblk", []string{"-J", "-o", "NAME,FSTYPE,PTTYPE,MOUNTPOINT", "/dev/sda"}, true, ""},
{"disk-mkfs-ext4", "guarded format (ext4)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "ext4"}, true, ""},
{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
// ---- Storage mount units (watchdog re-mount) ----
{"mount-unit-install", "fs-UUID mount unit install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0644", "--", "/var/lib/felhom-agent/units/felhom-x.mount", "/etc/systemd/system/felhom-x.mount"}, false, ""},
{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "felhom-x.mount"}, false, ""},
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "felhom-x.mount"}, false, ""},
{"mount-unit-stop", "mount unit stop", "/usr/bin/systemctl", []string{"stop", "--", "felhom-x.mount"}, false, ""},
// ---- Network storage re-arm + cleanup (CAMPAIGN-3 F10/F1) ----
{"netmount-reset-failed", "NAS automount re-arm after start-limit (F10)", "/usr/bin/systemctl", []string{"reset-failed", "--", "mnt-felhom\\x2ddrives-media.automount"}, false, ""},
{"netmount-rmdir", "removed-share mountpoint cleanup (F1)", "/usr/bin/rmdir", []string{"/mnt/felhom-drives/media"}, false, ""},
// ---- Node self-heal (CAMPAIGN-3 F12-class, appliance-gated in code) ----
{"selfheal-networking-start", "appliance networking recovery at boot (F12 defense in depth)", "/usr/bin/systemctl", []string{"start", "networking.service"}, false, ""},
// ---- Provisioning back-half ----
{"provision-chown", "bootstrap mount guest-root chown", "/usr/bin/chown", []string{"-R", "100000:100000", "/var/lib/felhom-agent/guests/9201"}, false, ""},
{"provision-config-mount", "bootstrap config bind mount", "/usr/sbin/pct", []string{"set", "9201", "-mp0", "/var/lib/felhom-agent/guests/9201"}, false, ""},
{"provision-onboot", "customer guest autostart (onboot)", "/usr/sbin/pct", []string{"set", "9201", "-onboot", "1"}, false, ""},
// ---- Pre-start self-heal hook + guest lifecycle ----
{"guesthook-install", "pre-start hook snippet install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-guest-hook-123456789.sh", "/var/lib/vz/snippets/felhom-guest-hook.sh"}, false, ""},
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
// ---- LAN split-horizon resolver (dnsmasq) ----
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
{"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""},
{"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""},
{"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
{"dnsmasq-guest-ip", "guest LAN IP discovery", "/usr/sbin/pct", []string{"exec", "9201", "--", "ip", "-4", "-o", "addr", "show", "dev", "eth0"}, false, ""},
{"dnsmasq-guest-domain", "guest domain discovery", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "exec", "felhom-controller", "cat", "/opt/docker/felhom-controller/controller.yaml"}, false, ""},
// ---- Guest-network watchdog (FELHOM_GUESTNET, R-54, v0.92.0) ----
// Deliberately NOT Critical: a host that has not yet taken the new sudoers file should show
// degraded (visible, diagnosable) without paging an operator for every box on rollout day —
// the R-50b lesson that an amber fleet is background noise. The address read is covered by
// dnsmasq-guest-ip above and is not duplicated here (one command, one capability row).
{"guestnet-route", "guest default-route probe", "/usr/sbin/pct", []string{"exec", "9201", "--", "ip", "route", "show", "default"}, false, ""},
{"guestnet-ifaces", "guest interface-mode read", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/network/interfaces"}, false, ""},
{"guestnet-dhclient-probe", "guest DHCP-client liveness probe", "/usr/sbin/pct", []string{"exec", "9201", "--", "pgrep", "-x", "dhclient"}, false, ""},
{"guestnet-heal", "guest DHCP-client restart (the 2026-07-20 heal)", "/usr/sbin/pct", []string{"exec", "9201", "--", "dhclient", "-pf", "/run/dhclient.eth0.pid", "-lf", "/var/lib/dhcp/dhclient.eth0.leases", "eth0"}, false, ""},
// ---- Controller-swap / managed auto-update (FELHOM_CONTROLLERSWAP, v0.45.0; Critical: a
// silently-broken fleet auto-update is operator-alert-worthy) ----
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "tee", "/etc/felhom-controller-image"}, true, ""},
// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
{"stalelock-unlock", "reboot-during-backup stale-lock recovery", "/usr/sbin/pct", []string{"unlock", "9201"}, true, ""},
// ---- Offsite WG tunnel (FELHOM_WG, S3/v0.64.0; Critical FLIPPED in S4/v0.66.0 — offsite
// backups now RIDE the tunnel, so a degraded tunnel capability is operator-alert-worthy: the
// conf install, unit enable/restart and the handshake read gate the backup path. apt-install
// (one-time bootstrap) and disable (revocation, a deliberate teardown) stay non-critical. The
// handshake read is the ONLY wg invocation (never `dump`). ----
{"wg-tools-install", "wireguard-tools package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "wireguard-tools"}, false, ""},
{"wg-conf-install", "wg-felhom conf install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0600", "--", "/var/lib/felhom-agent/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"}, true, ""},
{"wg-enable", "wg-quick@wg-felhom enable", "/usr/bin/systemctl", []string{"enable", "--now", "wg-quick@wg-felhom"}, true, ""},
{"wg-restart", "wg-quick@wg-felhom restart (conf change)", "/usr/bin/systemctl", []string{"restart", "wg-quick@wg-felhom"}, true, ""},
{"wg-disable", "wg-quick@wg-felhom disable (revocation)", "/usr/bin/systemctl", []string{"disable", "--now", "wg-quick@wg-felhom"}, false, ""},
{"wg-handshake-read", "tunnel handshake-age read", "/usr/bin/wg", []string{"show", "wg-felhom", "latest-handshakes"}, true, ""},
// ---- PBS DR tier apply (FELHOM_PBSDR, slice 2). NON-critical (the selfupdate rationale):
// applying the tier is an occasional hub-driven provisioning op, not a steady-state serving
// path — a degraded grant means "can't provision/reconcile the PBS entry" (the bridge reports
// loudly anyway), not a serving outage. The steady-state backup path is covered by the wg +
// storage capabilities. List-mode representations only; never executed. ----
{"pbsdr-create", "PBS DR storage-entry create (K autogen)", "/usr/local/sbin/felhom-pbs-apply", []string{"create", "felhom-pbs", "10.77.0.1", "felhom-offsite", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
{"pbsdr-reconcile", "PBS DR storage-entry reconcile (set-only)", "/usr/local/sbin/felhom-pbs-apply", []string{"reconcile", "felhom-pbs", "10.77.0.1", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
{"pbsdr-grant", "PBS DR storage ACL self-grant", "/usr/local/sbin/felhom-pbs-apply", []string{"grant", "felhom-pbs"}, false, ""},
// R-39 leg (b), v0.91.0: the credential READ path. Its absence is what made the PBS verify loop
// permanently blind to an applied-but-401 tier, so a host missing this verb is DEGRADED in a way
// that matters — it cannot detect the failure this whole tier exists to survive.
{"pbsdr-read", "PBS DR credential read (verify-loop auth probe)", "/usr/local/sbin/felhom-pbs-apply", []string{"read", "felhom-pbs", "/etc/pve/priv/storage"}, false, ""},
// ---- Escrow ceremony (FELHOM_ESCROW, controller-driven, v0.88.0). Critical: the customer
// wizard's whole run path IS this one grant — a dropped line silently breaks every ceremony.
// GatedBy is set EXPLICITLY (the name deliberately says "escrow", not "pbsdr-": the feature is
// the ceremony, but it only exists behind the DR tier — no PBS key, no ceremony). ReprArgs is
// the SHARED argv constant (internal/escrow/ceremony.go) — the exec runner uses the same one,
// so runner ↔ manifest can't drift, and TestManifestCoveredBySudoers locks manifest ↔ sudoers.
// List-mode probe only (`sudo -n -l`), spike-confirmed side-effect-free — never a real ceremony.
{"escrow-ceremony", "customer recovery-code ceremony (controller-driven)", escrow.CeremonyBinary, escrow.CeremonyArgs(), true, GatePBSDR},
// ---- Agent self-update (FELHOM_SELFUPDATE, D1). NON-critical: self-update is an occasional
// operator-driven op, not a steady-state serving path — a degraded grant means "can't
// self-update" (fall back to a manual SSH deploy), not a serving outage. The apply repr uses a
// staging-dir path + a placeholder sha (list-mode never runs it). ----
{"selfupdate-apply", "agent self-update apply (A/B flip)", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"apply", "/var/lib/felhom-agent/selfupdate/felhom-agent-0.0.0", "0000000000000000000000000000000000000000000000000000000000000000"}, false, ""},
{"selfupdate-commit", "agent self-update commit", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"commit"}, false, ""},
{"selfupdate-rollback", "agent self-update rollback", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"rollback"}, false, ""},
}
+296
View File
@@ -0,0 +1,296 @@
package capability
import (
"os"
"reflect"
"regexp"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// sudoersPath is the in-repo allowlist, relative to this test file (internal/capability/).
const sudoersPath = "../../configs/felhom-agent.sudoers"
// parseSudoersEntries returns every command pattern from the Cmnd_Alias blocks, with the sudoers
// escapes (`\,` `\:`) unescaped. It joins continuation lines and splits the alias RHS on commas
// that are NOT backslash-escaped (escaped commas are literal arg chars, e.g. the lvs `-o` list).
func parseSudoersEntries(t *testing.T, text string) []string {
t.Helper()
// 1. Collapse line continuations, keeping only Cmnd_Alias RHS text.
var rhs strings.Builder
lines := strings.Split(text, "\n")
inAlias := false
for _, ln := range lines {
trimmed := strings.TrimSpace(ln)
if strings.HasPrefix(trimmed, "#") {
continue
}
if strings.HasPrefix(trimmed, "Cmnd_Alias ") {
inAlias = true
if eq := strings.IndexByte(trimmed, '='); eq >= 0 {
trimmed = trimmed[eq+1:]
}
} else if !inAlias {
continue
}
// The final NOPASSWD line ("felhom-agent ALL=...") ends the alias region.
if strings.Contains(trimmed, "ALL=(") {
inAlias = false
continue
}
cont := strings.HasSuffix(trimmed, "\\")
rhs.WriteString(strings.TrimSuffix(trimmed, "\\"))
rhs.WriteString(" ")
if !cont {
// A non-continued line is the last entry of this alias. Emit a comma so it does not
// merge with the next alias's first entry when all RHS text is concatenated.
rhs.WriteString(", ")
inAlias = false
}
}
// 2. Split on unescaped commas → individual command entries.
raw := rhs.String()
var entries []string
var cur strings.Builder
for i := 0; i < len(raw); i++ {
c := raw[i]
if c == '\\' && i+1 < len(raw) {
cur.WriteByte(raw[i+1]) // unescape: keep the next char literally (\, → , ; \: → :)
i++
continue
}
if c == ',' {
entries = appendTrimmed(entries, cur.String())
cur.Reset()
continue
}
cur.WriteByte(c)
}
entries = appendTrimmed(entries, cur.String())
return entries
}
func appendTrimmed(entries []string, s string) []string {
if t := strings.Join(strings.Fields(s), " "); t != "" {
return append(entries, t)
}
return entries
}
// globToRegex translates a sudoers fnmatch pattern to an anchored regex. It is NOT a perfect sudo
// emulator — it only needs to catch a removed/renamed grant (the real failure mode). `*` → `.*`,
// `[...]` char classes pass through (valid regex), regex metachars are escaped.
func globToRegex(pat string) *regexp.Regexp {
var b strings.Builder
b.WriteString("^")
for i := 0; i < len(pat); i++ {
c := pat[i]
switch {
case c == '*':
b.WriteString(".*")
case c == '[': // copy the char class verbatim (valid in regex too)
if j := strings.IndexByte(pat[i:], ']'); j > 0 {
b.WriteString(pat[i : i+j+1])
i += j
continue
}
b.WriteString("\\[")
case strings.IndexByte(`.+()|{}^$\?`, c) >= 0:
b.WriteByte('\\')
b.WriteByte(c)
default:
b.WriteByte(c)
}
}
b.WriteString("$")
return regexp.MustCompile(b.String())
}
// matchesAny reports whether cmdline matches at least one sudoers entry pattern.
func matchesAny(cmdline string, entries []string) bool {
for _, e := range entries {
if globToRegex(e).MatchString(cmdline) {
return true
}
}
return false
}
// TestManifestCoveredBySudoers is the headline build-time gate: EVERY manifest capability's
// representative command line must be permitted by at least one sudoers pattern. This is exactly
// the check that would have caught the lxc-info / make-private grants being dropped at the
// 2026-06-28 cutover — in CI, before shipping.
func TestManifestCoveredBySudoers(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatalf("read sudoers %s: %v", sudoersPath, err)
}
entries := parseSudoersEntries(t, string(data))
if len(entries) < 20 {
t.Fatalf("parsed only %d sudoers entries — parser likely broke", len(entries))
}
for _, c := range Manifest() {
cmdline := strings.TrimSpace(c.Binary + " " + strings.Join(c.ReprArgs, " "))
if !matchesAny(cmdline, entries) {
t.Errorf("capability %q (%s) NOT covered by any sudoers grant:\n %s",
c.Name, c.Feature, cmdline)
}
}
}
// TestRedProof_DroppedGrantFailsCheck is the companion red-proof: with the lxc-info line removed
// from an in-memory copy of the sudoers, the coverage check for guest-init-pid MUST fail. Proves
// the build gate actually catches the regression (a green test that can never go red is hollow).
func TestRedProof_DroppedGrantFailsCheck(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatalf("read sudoers: %v", err)
}
// Drop the lxc-info grant line.
var kept []string
for _, ln := range strings.Split(string(data), "\n") {
if strings.Contains(ln, "lxc-info") {
continue
}
kept = append(kept, ln)
}
mutated := strings.Join(kept, "\n")
if strings.Contains(mutated, "lxc-info") {
t.Fatal("setup: lxc-info line not removed")
}
entries := parseSudoersEntries(t, mutated)
var guestInit Capability
for _, c := range Manifest() {
if c.Name == "guest-init-pid" {
guestInit = c
}
}
if guestInit.Name == "" {
t.Fatal("manifest missing guest-init-pid")
}
cmdline := guestInit.Binary + " " + strings.Join(guestInit.ReprArgs, " ")
if matchesAny(cmdline, entries) {
t.Errorf("red-proof FAILED: guest-init-pid still matches after dropping the lxc-info grant — the build gate would NOT catch the regression")
}
// Sanity: the UNMUTATED file MUST cover it (so the failure above is specific to the drop).
full := parseSudoersEntries(t, string(data))
if !matchesAny(cmdline, full) {
t.Errorf("guest-init-pid should be covered by the real sudoers")
}
}
// TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0
// FELHOM_CONTROLLERSWAP grants: with the `tee /etc/felhom-controller-image` line removed, the
// controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new
// swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update).
func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatalf("read sudoers: %v", err)
}
var kept []string
for _, ln := range strings.Split(string(data), "\n") {
if strings.Contains(ln, "tee /etc/felhom-controller-image") {
continue
}
kept = append(kept, ln)
}
mutated := strings.Join(kept, "\n")
entries := parseSudoersEntries(t, mutated)
var write Capability
for _, c := range Manifest() {
if c.Name == "controllerswap-write" {
write = c
}
}
if write.Name == "" {
t.Fatal("manifest missing controllerswap-write")
}
cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ")
if matchesAny(cmdline, entries) {
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping the tee grant")
}
if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) {
t.Errorf("controllerswap-write should be covered by the real sudoers")
}
}
// TestEscrowCeremonyArgvPinned locks the ceremony argv copies together (Scenario G, v0.88.0).
// The exec runner and the manifest entry both consume escrow.CeremonyArgs() (one shared source),
// and TestManifestCoveredBySudoers proves manifest ⊆ sudoers — so pinning the shared source to
// the EXPECTED literal here transitively locks all three: runner == manifest == sudoers.
// Red-proof: mutate one element of the argv in internal/escrow/ceremony.go and THIS test fails
// (and so does the sudoers coverage); a sudoers-side mutation is caught by the existing
// TestRedProof_* machinery.
func TestEscrowCeremonyArgvPinned(t *testing.T) {
wantBinary := "/usr/local/bin/felhom-agent"
wantArgs := []string{"--config", "/etc/felhom-agent/agent.json", "--selftest=escrow-create", "--upload", "--output=json"}
if escrow.CeremonyBinary != wantBinary {
t.Errorf("escrow.CeremonyBinary = %q, want %q", escrow.CeremonyBinary, wantBinary)
}
if got := escrow.CeremonyArgs(); !reflect.DeepEqual(got, wantArgs) {
t.Errorf("escrow.CeremonyArgs() = %q, want %q (the sudoers line + manifest entry must stay byte-identical)", got, wantArgs)
}
var entry Capability
for _, c := range Manifest() {
if c.Name == "escrow-ceremony" {
entry = c
}
}
if entry.Name == "" {
t.Fatal("manifest missing escrow-ceremony")
}
if entry.Binary != escrow.CeremonyBinary || !reflect.DeepEqual(entry.ReprArgs, escrow.CeremonyArgs()) {
t.Errorf("manifest escrow-ceremony argv diverged from the shared constant: %s %q", entry.Binary, entry.ReprArgs)
}
if !entry.Critical {
t.Error("escrow-ceremony must be Critical (the wizard's whole run path is this one grant)")
}
if entry.GatedBy != GatePBSDR {
t.Errorf("escrow-ceremony GatedBy = %q, want %q (no PBS key → no ceremony; inactive, never red, on a DR-off box)", entry.GatedBy, GatePBSDR)
}
// CeremonyArgs must return a COPY — a caller mutating its slice must not poison the source.
mutated := escrow.CeremonyArgs()
mutated[0] = "--poisoned"
if got := escrow.CeremonyArgs(); !reflect.DeepEqual(got, wantArgs) {
t.Error("escrow.CeremonyArgs() shares its backing array — callers can mutate the source")
}
}
// TestWGCapabilityCriticality pins the exact S4 (v0.66.0) Critical set for the FELHOM_WG entries:
// the backup path (conf install, unit enable/restart, handshake read) is operator-alert-worthy now
// that offsite backups ride the tunnel; the one-time apt install and the deliberate disable
// (revocation) are NOT. Red-proof: flip any one entry's Critical in manifest.go and this fails.
func TestWGCapabilityCriticality(t *testing.T) {
wantCritical := map[string]bool{
"wg-tools-install": false,
"wg-conf-install": true,
"wg-enable": true,
"wg-restart": true,
"wg-disable": false,
"wg-handshake-read": true,
}
seen := map[string]bool{}
for _, c := range Manifest() {
want, ok := wantCritical[c.Name]
if !ok {
continue
}
seen[c.Name] = true
if c.Critical != want {
t.Errorf("%s: Critical = %v, want %v", c.Name, c.Critical, want)
}
}
for name := range wantCritical {
if !seen[name] {
t.Errorf("manifest missing wg capability %q", name)
}
}
}
+122
View File
@@ -0,0 +1,122 @@
package capability
import (
"context"
"os"
)
// Status is one capability's live result — the wire shape the agent attaches to its hub report
// (HostReport.Capabilities). The hub mirrors this struct field-for-field and keys its alert on
// Critical+degraded. Reason is empty when ok.
//
// "inactive" (v0.86.0, DR-tier-by-default): a config-GATED capability whose plumbing is HEALTHY
// (binary present, sudo granted) but whose gating feature is disabled by configuration. Distinct
// from degraded on purpose — disabled ≠ broken; the hub renders it as a neutral chip, never red.
// Broken plumbing (binary missing / grant denied) stays DEGRADED even when the gate is off: an
// un-migrated box must never look deliberately disabled.
type Status struct {
Name string `json:"name"`
Feature string `json:"feature"`
Critical bool `json:"critical"`
Status string `json:"status"` // "ok" | "degraded" | "inactive"
Reason string `json:"reason,omitempty"`
}
const (
StatusOK = "ok"
StatusDegraded = "degraded"
StatusInactive = "inactive"
)
// ReasonInactive is the fixed reason string for the inactive state (the hub + operator docs
// reference it verbatim).
const ReasonInactive = "disabled by configuration"
// Runner is the minimal exec seam the probe needs (satisfied by proxmox.ExecRunner). The probe
// runs `sudo -n -l -- <binary> <args…>` LITERALLY — a sudo POLICY LIST that never executes the
// command — so the Runner MUST be a DIRECT runner (RunnerDirect), not the sudo-prepending one
// (else it would double-sudo). exit 0 ⇔ the command is permitted under the NOPASSWD allowlist.
type Runner interface {
Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
}
// Prober checks the manifest against the live host. Exists defaults to an os.Stat check on the
// absolute binary path (what `command -v` would resolve for an absolute path) when nil.
// GateActive answers "is the feature behind this gate id configured on?" for GATED capabilities
// (Capability.GatedBy). nil, or a gate it answers true for, keeps the historical behavior; false
// downgrades a HEALTHY probe to StatusInactive (broken plumbing stays degraded regardless).
type Prober struct {
Runner Runner
Exists func(path string) bool // nil → os.Stat
GateActive func(gate string) bool // nil → every gate treated active
}
// Probe lists every manifest capability against the sudo policy and checks its binary exists,
// mapping to ok/degraded (§8 of the spec). It NEVER executes a probed command and NEVER returns a
// fatal error (serve-degraded): a probe failure is reported, not raised. If sudo itself is
// unavailable for the agent (the drop-in is missing / the user has no sudo at all), it collapses
// to ONE aggregate degraded signal instead of N identical ones.
func (p Prober) Probe(ctx context.Context) []Status {
exists := p.Exists
if exists == nil {
exists = func(path string) bool { _, err := os.Stat(path); return err == nil }
}
caps := Manifest()
// Preflight: a bare `sudo -n -l` lists the user's allowed commands. For our NOPASSWD service
// user it exits 0; if it fails, the drop-in isn't installed (or sudo is gone) and EVERY vector
// would individually fail — collapse to one aggregate signal so the operator gets one alert.
if p.Runner != nil {
if _, _, err := p.Runner.Run(ctx, "sudo", "-n", "-l"); err != nil {
return []Status{{
Name: "sudo",
Feature: "the entire privileged surface (mount/format/pct/dnsmasq/lxc-info)",
Critical: true,
Status: StatusDegraded,
Reason: "sudoers drop-in not installed / sudo unavailable",
}}
}
}
out := make([]Status, 0, len(caps))
for _, c := range caps {
s := Status{Name: c.Name, Feature: c.Feature, Critical: c.Critical, Status: StatusOK}
switch {
case !exists(c.Binary):
s.Status, s.Reason = StatusDegraded, "binary not found"
case p.Runner != nil && !p.granted(ctx, c):
s.Status, s.Reason = StatusDegraded, "sudo policy denied"
}
// Config gate (v0.86.0): only a HEALTHY probe is downgraded to inactive — a degraded one
// stays degraded (missing binary/grant = un-migrated or mis-installed box, never "off").
if s.Status == StatusOK && c.GatedBy != "" && p.GateActive != nil && !p.GateActive(c.GatedBy) {
s.Status, s.Reason = StatusInactive, ReasonInactive
}
out = append(out, s)
}
return out
}
// granted reports whether `sudo -n -l -- <binary> <reprArgs…>` is permitted (exit 0). List-mode is
// side-effect-free — the command is matched against the policy, never run.
func (p Prober) granted(ctx context.Context, c Capability) bool {
args := append([]string{"-n", "-l", "--", c.Binary}, c.ReprArgs...)
_, _, err := p.Runner.Run(ctx, "sudo", args...)
return err == nil
}
// Summarize returns (okCount, total, degraded) for logging. degraded lists DEGRADED statuses
// only — inactive is a deliberate, healthy state and must not land in the error log (it is
// counted via len(statuses)-ok-len(degraded) by callers that want it).
func Summarize(statuses []Status) (ok, total int, degraded []Status) {
total = len(statuses)
for _, s := range statuses {
switch s.Status {
case StatusOK:
ok++
case StatusDegraded:
degraded = append(degraded, s)
}
}
return ok, total, degraded
}
+202
View File
@@ -0,0 +1,202 @@
package capability
import (
"context"
"errors"
"strings"
"testing"
)
// fakeRunner returns a canned error per (command line) and records calls. deny holds binaries (or
// the bare "sudo -n -l" preflight) that should fail; everything else exits 0.
type fakeRunner struct {
preflightErr error
denyBinary map[string]bool // binary path → policy-denied
calls int
executedReal bool // set if a probed command was ever run WITHOUT -l (must never happen)
}
func (f *fakeRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
f.calls++
// Preflight is `sudo -n -l` (exactly 2 args, no `--`).
if name == "sudo" && len(args) == 2 && args[0] == "-n" && args[1] == "-l" {
return nil, nil, f.preflightErr
}
// Every real probe must be a LIST: `sudo -n -l -- <binary> …`.
if name != "sudo" || len(args) < 4 || args[0] != "-n" || args[1] != "-l" || args[2] != "--" {
f.executedReal = true
return nil, nil, nil
}
binary := args[3]
if f.denyBinary[binary] {
return nil, nil, errors.New("sudo: a password is required")
}
return nil, nil, nil
}
func find(statuses []Status, name string) Status {
for _, s := range statuses {
if s.Name == name {
return s
}
}
return Status{}
}
// §7-A: all grants present + binaries exist → every capability ok.
func TestProbe_AllOK(t *testing.T) {
r := &fakeRunner{denyBinary: map[string]bool{}}
p := Prober{Runner: r, Exists: func(string) bool { return true }}
statuses := p.Probe(context.Background())
ok, total, degraded := Summarize(statuses)
if total != len(Manifest()) {
t.Fatalf("total=%d want %d", total, len(Manifest()))
}
if ok != total || len(degraded) != 0 {
t.Fatalf("expected all ok, got %d/%d (degraded: %+v)", ok, total, degraded)
}
if r.executedReal {
t.Fatal("probe executed a command without -l (must be list-only)")
}
}
// §7-B: one grant denied → that capability degraded "sudo policy denied", others ok. Serve-degraded.
func TestProbe_OneDenied(t *testing.T) {
r := &fakeRunner{denyBinary: map[string]bool{"/usr/bin/lxc-info": true}}
p := Prober{Runner: r, Exists: func(string) bool { return true }}
statuses := p.Probe(context.Background())
gi := find(statuses, "guest-init-pid")
if gi.Status != StatusDegraded || gi.Reason != "sudo policy denied" {
t.Fatalf("guest-init-pid = %+v, want degraded/sudo policy denied", gi)
}
if !gi.Critical {
t.Fatal("guest-init-pid should be Critical")
}
// A sibling stays ok.
if s := find(statuses, "drive-bind"); s.Status != StatusOK {
t.Fatalf("drive-bind = %+v, want ok", s)
}
ok, total, _ := Summarize(statuses)
if ok != total-1 {
t.Fatalf("expected exactly one degraded, got ok=%d total=%d", ok, total)
}
}
// §7-D: binary missing but policy granted → degraded "binary not found".
func TestProbe_BinaryMissing(t *testing.T) {
r := &fakeRunner{denyBinary: map[string]bool{}}
p := Prober{Runner: r, Exists: func(path string) bool { return path != "/usr/bin/lxc-info" }}
statuses := p.Probe(context.Background())
gi := find(statuses, "guest-init-pid")
if gi.Status != StatusDegraded || gi.Reason != "binary not found" {
t.Fatalf("guest-init-pid = %+v, want degraded/binary not found", gi)
}
}
// §8 aggregate: sudo itself unavailable for the user → ONE aggregate degraded, not N.
func TestProbe_SudoUnavailableAggregates(t *testing.T) {
r := &fakeRunner{preflightErr: errors.New("Sorry, user felhom-agent may not run sudo"), denyBinary: map[string]bool{}}
p := Prober{Runner: r, Exists: func(string) bool { return true }}
statuses := p.Probe(context.Background())
if len(statuses) != 1 {
t.Fatalf("expected 1 aggregate status, got %d", len(statuses))
}
s := statuses[0]
if s.Name != "sudo" || s.Status != StatusDegraded || !s.Critical || !strings.Contains(s.Reason, "drop-in not installed") {
t.Fatalf("aggregate = %+v, want critical degraded sudo-unavailable", s)
}
}
// Probe must never raise — even with a nil runner (e.g. a dev path) it returns statuses.
func TestProbe_NilRunnerNoPanic(t *testing.T) {
p := Prober{Runner: nil, Exists: func(string) bool { return true }}
if got := len(p.Probe(context.Background())); got != len(Manifest()) {
t.Fatalf("nil-runner probe returned %d statuses, want %d", got, len(Manifest()))
}
}
// ── DR-tier gate (v0.86.0) ─────────────────────────────────────────────────────────────────────
// Gate OFF + healthy plumbing → the gated pbsdr-* capabilities report INACTIVE (the neutral
// "disabled by configuration" state), NOT ok and NOT degraded — Scenario B of the DR-by-default
// spec. Ungated siblings are untouched. Red-proof partner: collapse inactive into ok (drop the
// gate branch in Probe) → this fails while TestProbe_GateOffBinaryMissingStaysDegraded passes.
func TestProbe_GateOffHealthyIsInactive(t *testing.T) {
r := &fakeRunner{denyBinary: map[string]bool{}}
p := Prober{
Runner: r,
Exists: func(string) bool { return true },
GateActive: func(gate string) bool { return gate != GatePBSDR }, // DR tier OFF
}
statuses := p.Probe(context.Background())
// v0.88.0: escrow-ceremony joins the gate EXPLICITLY (non-pbsdr name, GatedBy literal) —
// the ceremony only exists behind the DR tier (no PBS key, no ceremony).
// v0.91.0: pbsdr-read (the R-39 credential-read verb) rides the same `pbsdr-` prefix gate — a new
// pbsdr-* op is gated by construction, which is exactly the property this list is here to hold.
for _, name := range []string{"pbsdr-create", "pbsdr-reconcile", "pbsdr-grant", "pbsdr-read", "escrow-ceremony"} {
s := find(statuses, name)
if s.Status != StatusInactive || s.Reason != ReasonInactive {
t.Fatalf("%s = %+v, want inactive/%q", name, s, ReasonInactive)
}
}
// An ungated sibling stays plain ok.
if s := find(statuses, "drive-bind"); s.Status != StatusOK {
t.Fatalf("drive-bind = %+v, want ok (ungated)", s)
}
// Summarize must NOT count inactive as degraded (it is not error-log-worthy).
ok, total, degraded := Summarize(statuses)
if len(degraded) != 0 {
t.Fatalf("inactive leaked into degraded: %+v", degraded)
}
if ok != total-5 {
t.Fatalf("ok=%d total=%d, want exactly the 5 gated ones non-ok", ok, total)
}
}
// Gate OFF + BROKEN plumbing (binary missing) → DEGRADED stays degraded. An un-migrated
// pre-v1.15.0 box must never masquerade as deliberately disabled ("never silently pretend").
func TestProbe_GateOffBinaryMissingStaysDegraded(t *testing.T) {
r := &fakeRunner{denyBinary: map[string]bool{}}
p := Prober{
Runner: r,
Exists: func(path string) bool { return path != "/usr/local/sbin/felhom-pbs-apply" },
GateActive: func(gate string) bool { return gate != GatePBSDR }, // DR tier OFF
}
statuses := p.Probe(context.Background())
for _, name := range []string{"pbsdr-create", "pbsdr-reconcile", "pbsdr-grant"} {
s := find(statuses, name)
if s.Status != StatusDegraded || s.Reason != "binary not found" {
t.Fatalf("%s = %+v, want degraded/binary not found even with the gate off", name, s)
}
}
}
// Gate ON (DR configured) + healthy plumbing → plain ok, exactly the pre-v0.86.0 behavior.
// A nil GateActive must behave the same (fails ACTIVE).
func TestProbe_GateOnOrNilIsOK(t *testing.T) {
for _, gate := range []func(string) bool{nil, func(string) bool { return true }} {
r := &fakeRunner{denyBinary: map[string]bool{}}
p := Prober{Runner: r, Exists: func(string) bool { return true }, GateActive: gate}
statuses := p.Probe(context.Background())
if s := find(statuses, "pbsdr-create"); s.Status != StatusOK {
t.Fatalf("pbsdr-create = %+v, want ok (gate active/nil)", s)
}
}
}
// The gate covers exactly the pbsdr-* entries (name-prefix mechanism) PLUS escrow-ceremony (an
// explicit GatedBy literal — v0.88.0: the ceremony only exists behind the DR tier, but its name
// says what the feature is). Nothing else may be gated (a regression here would silently un-gate
// the tier or gate an unrelated capability).
func TestManifest_ExactlyPBSDRGated(t *testing.T) {
for _, c := range Manifest() {
wantGated := strings.HasPrefix(c.Name, "pbsdr-") || c.Name == "escrow-ceremony"
if gated := c.GatedBy == GatePBSDR; gated != wantGated {
t.Fatalf("%s: GatedBy=%q, want gated=%v", c.Name, c.GatedBy, wantGated)
}
if c.GatedBy != "" && c.GatedBy != GatePBSDR {
t.Fatalf("%s: unknown gate id %q", c.Name, c.GatedBy)
}
}
}
+182
View File
@@ -0,0 +1,182 @@
package config
import (
"encoding/json"
"strings"
"testing"
"time"
)
// R-82 Slice A.1 — per-target cadence + retention resolution.
//
// The load-bearing property is ADDITIVITY: every config that exists on a live box today must
// resolve to exactly one tier that behaves as it does now. The second property is that a
// mis-configured tier is REJECTED LOUDLY rather than defaulted — a weekly DR tier silently running
// daily would fill the datastore, and a silently dropped tier is the "applied and empty" fault
// R-82 exists to fix.
func TestBackupTiers_LegacyConfigIsUnchanged(t *testing.T) {
// Exactly the shape live on demo-felhom today.
var b BackupConfig
raw := `{"local_backup_target":"local","local_backup_retention":3,"backup_cadence_seconds":0}`
if err := json.Unmarshal([]byte(raw), &b); err != nil {
t.Fatal(err)
}
tiers, warnings := b.BackupTiers()
if len(warnings) != 0 {
t.Fatalf("a legacy config must produce NO warnings; got %v", warnings)
}
if len(tiers) != 1 {
t.Fatalf("a config with no backup_targets must resolve to exactly ONE tier; got %+v", tiers)
}
got := tiers[0]
if got.TargetID != "local" || got.Cadence != 24*time.Hour || got.KeepLast != 3 || !got.Primary {
t.Fatalf("legacy tier changed: %+v", got)
}
}
// An empty BackupConfig still resolves — to the felhom-pbs default target, 24h, keep-last 3.
// (Unchanged pre-R-82 behaviour; pinned so the default target can't drift unnoticed.)
func TestBackupTiers_ZeroConfigKeepsDefaults(t *testing.T) {
tiers, warnings := BackupConfig{}.BackupTiers()
if len(warnings) != 0 || len(tiers) != 1 {
t.Fatalf("zero config: tiers=%+v warnings=%v", tiers, warnings)
}
if tiers[0].TargetID != defaultBackupTarget || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
t.Fatalf("zero-config defaults changed: %+v", tiers[0])
}
}
// The whole point: local daily + PBS weekly, each with its OWN retention.
func TestBackupTiers_LocalDailyPlusPBSWeekly(t *testing.T) {
var b BackupConfig
raw := `{
"local_backup_target":"local",
"local_backup_retention":3,
"backup_cadence_seconds":86400,
"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]
}`
if err := json.Unmarshal([]byte(raw), &b); err != nil {
t.Fatal(err)
}
tiers, warnings := b.BackupTiers()
if len(warnings) != 0 {
t.Fatalf("unexpected warnings: %v", warnings)
}
if len(tiers) != 2 {
t.Fatalf("want 2 tiers, got %+v", tiers)
}
if !tiers[0].Primary || tiers[0].TargetID != "local" || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
t.Fatalf("primary tier wrong: %+v", tiers[0])
}
if tiers[1].Primary || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour || tiers[1].KeepLast != 2 {
t.Fatalf("PBS tier wrong: %+v", tiers[1])
}
// THE knob-sharing check: the two retentions are independent values, not one shared number.
if tiers[0].KeepLast == tiers[1].KeepLast {
t.Fatalf("this fixture sets 3 and 2 deliberately — equal values mean the knob is shared: %+v", tiers)
}
}
// A tier with no cadence is REJECTED, not defaulted. Defaulting would turn a weekly DR tier into a
// daily one and fill the 37.2 GB datastore (R-82 Phase 0, P0.3).
func TestBackupTiers_MissingCadenceIsRejectedLoudly(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", KeepLast: 2}},
}
tiers, warnings := b.BackupTiers()
if len(tiers) != 1 {
t.Fatalf("a cadence-less tier must NOT be armed; got %+v", tiers)
}
if len(warnings) != 1 || !strings.Contains(warnings[0], "cadence_seconds must be > 0") {
t.Fatalf("rejection must be reported so the caller can log it loudly; got %v", warnings)
}
if !strings.Contains(warnings[0], "felhom-pbs") {
t.Fatalf("the warning must name the tier it dropped; got %q", warnings[0])
}
}
func TestBackupTiers_RejectsEmptyAndDuplicateTargets(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{
{TargetID: "", CadenceSeconds: 3600},
{TargetID: "local", CadenceSeconds: 3600}, // repeats the primary
{TargetID: "felhom-pbs", CadenceSeconds: 604800}, // good
{TargetID: "felhom-pbs", CadenceSeconds: 99}, // duplicate
},
}
tiers, warnings := b.BackupTiers()
if len(tiers) != 2 || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour {
t.Fatalf("want primary + one PBS tier at the FIRST definition; got %+v", tiers)
}
if len(warnings) != 3 {
t.Fatalf("want 3 rejections (empty, duplicate-of-primary, duplicate); got %v", warnings)
}
}
// keep_last unset means DO NOT PRUNE. That is the fail-safe: a DR tier must never start pruning
// itself because someone forgot a field.
func TestBackupTiers_UnsetKeepLastMeansNoPrune(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 2 {
t.Fatalf("got %+v", tiers)
}
if tiers[1].KeepLast != 0 {
t.Fatalf("an unset keep_last must resolve to 0 = never prune; got %d", tiers[1].KeepLast)
}
// And a negative is clamped to the same fail-safe rather than becoming a prune spec.
b.ExtraTargets[0].KeepLast = -5
tiers, _ = b.BackupTiers()
if tiers[1].KeepLast != 0 {
t.Fatalf("a negative keep_last must clamp to 0 (never prune); got %d", tiers[1].KeepLast)
}
}
// The primary's retention still comes from the legacy knob with its legacy clamp — untouched.
func TestBackupTiers_PrimaryRetentionClampUnchanged(t *testing.T) {
for _, tc := range []struct{ in, want int }{{0, 3}, {-1, 3}, {1, 1}, {7, 7}} {
b := BackupConfig{LocalBackupTarget: "local", LocalBackupRetention: tc.in}
tiers, _ := b.BackupTiers()
if tiers[0].KeepLast != tc.want {
t.Fatalf("LocalBackupRetention=%d → KeepLast=%d, want %d", tc.in, tiers[0].KeepLast, tc.want)
}
}
}
// R-82 live-failure regression (2026-07-26): the runner hard-coded a 30-minute vzdump wait, which
// is right for a local vzdump and wrong for an offsite PBS upload. The first full ~10 GB PBS
// snapshot on demo-felhom ran past 30 min; the agent gave up waiting and recorded success=false
// WHILE THE BACKUP WAS STILL RUNNING — a false failure that leaves the tier permanently "due" and
// makes the next attempt collide with the guest lock vzdump still holds.
func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 2 {
t.Fatalf("got %+v", tiers)
}
if tiers[0].WaitTimeout != 30*time.Minute {
t.Fatalf("the PRIMARY must keep the historical 30m wait (unchanged behaviour); got %s", tiers[0].WaitTimeout)
}
if tiers[1].WaitTimeout != 12*time.Hour {
t.Fatalf("an offsite tier must default to a GENEROUS wait (operator ruling: let the first backup run as long as needed) — a false timeout is worse than a slow pass; got %s", tiers[1].WaitTimeout)
}
// And it must be overridable per tier.
b.ExtraTargets[0].WaitTimeoutSeconds = 3600
tiers, _ = b.BackupTiers()
if tiers[1].WaitTimeout != time.Hour {
t.Fatalf("wait_timeout_seconds must override; got %s", tiers[1].WaitTimeout)
}
// The two tiers must NOT share one bound.
if tiers[0].WaitTimeout == tiers[1].WaitTimeout {
t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
}
}
+449 -19
View File
@@ -22,16 +22,161 @@ import (
// Config is the agent configuration.
type Config struct {
Proxmox ProxmoxConfig `json:"proxmox"`
Privileged PrivilegedConfig `json:"privileged"`
Authz AuthzConfig `json:"authz"`
Hub HubConfig `json:"hub"`
Storage StorageConfig `json:"storage"`
Backup BackupConfig `json:"backup"`
Proxmox ProxmoxConfig `json:"proxmox"`
Privileged PrivilegedConfig `json:"privileged"`
Authz AuthzConfig `json:"authz"`
Hub HubConfig `json:"hub"`
Storage StorageConfig `json:"storage"`
Backup BackupConfig `json:"backup"`
Escrow EscrowConfig `json:"escrow"`
LocalAPI LocalAPIConfig `json:"local_api"`
LANResolver LANResolverConfig `json:"lan_resolver"`
WGTunnel WGTunnelConfig `json:"wg_tunnel"`
GuestNet GuestNetConfig `json:"guest_net"`
OOB OOBConfig `json:"oob"`
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
// DeploymentMode gates host-service self-heal (CAMPAIGN-3 Part 6). "appliance" = a Felhom-managed
// node the agent may remediate (e.g. start networking at boot — F12-class defense in depth). Any
// other value, including absent/unknown, is treated as "byo" (a customer's own host): the self-heal
// CHECK still runs and WARNs, but the REMEDY is structurally unreachable. Fail-safe to byo — never
// touch a host we do not own. Distinct from Privileged.Mode (sudo vs direct exec) — do NOT overload.
DeploymentMode string `json:"deployment_mode,omitempty"`
// SourcePath is the file this config was loaded from ("" = all-env). Set by Load, never
// serialized — the pbsdr bridge's escrow.pbs_storage_id seed writes back to it.
SourcePath string `json:"-"`
}
// DeploymentModeAppliance is the ONLY value that unlocks host-service self-heal. Everything else,
// including "" and any typo, is byo (fail-safe — a host we do not own is never remediated).
const DeploymentModeAppliance = "appliance"
// IsAppliance reports whether this node is a Felhom-managed appliance (self-heal remedies allowed).
// Fail-safe: absent/unknown → false (byo).
func (c *Config) IsAppliance() bool { return c.DeploymentMode == DeploymentModeAppliance }
// OOBConfig configures the dedicated felhom-sshd OOB access instance + belt (TASK H1). **Enabled
// DEFAULTS TO FALSE** — a rollout to a box without explicit oob.enabled=true is a no-op (no port
// claim, no config render, no belt mutation, no oob report stanza) until the operator endpoint +
// static belt table exist.
type OOBConfig struct {
Enabled bool `json:"enabled"`
IntervalSeconds int `json:"interval_seconds"` // reconcile cadence; default 60
StateDir string `json:"state_dir"` // staged config/authkeys under <StateDir>/felhom-sshd/; default /var/lib/felhom-agent
}
// WithDefaults fills the OOB reconcile cadence + state dir.
func (o OOBConfig) WithDefaults() OOBConfig {
if o.IntervalSeconds == 0 {
o.IntervalSeconds = 60
}
if o.StateDir == "" {
o.StateDir = "/var/lib/felhom-agent"
}
return o
}
// SelfUpdateConfig configures the operator-signed agent self-update (TASK D1). The artifact HOST
// is operator-controlled config; the artifact INTEGRITY comes only from the sha256 pinned inside
// the operator-signed op — the hub's Day-0 manifest plays no role here, and a compromised Gitea
// can serve garbage but never a binary that passes the signed sha.
type SelfUpdateConfig struct {
// URLTemplate is the download URL with a literal "{version}" placeholder. Default mirrors the
// day-0 host-install scheme (Gitea generic package).
URLTemplate string `json:"url_template"`
// Username/Token are optional HTTP basic-auth credentials for the artifact host (the same git
// read token day-0 uses). Token is a secret — redacted in Config.Redacted.
Username string `json:"username,omitempty"`
Token string `json:"token,omitempty"`
// StateDir holds the staging subdir (<StateDir>/selfupdate/); default /var/lib/felhom-agent.
StateDir string `json:"state_dir,omitempty"`
// DwellSeconds is how long the NEW binary must run cleanly (after core init) before it commits
// the update; default 60.
DwellSeconds int `json:"dwell_seconds,omitempty"`
}
// WithDefaults fills the artifact URL template, state dir and dwell.
func (s SelfUpdateConfig) WithDefaults() SelfUpdateConfig {
if s.URLTemplate == "" {
s.URLTemplate = "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/{version}/felhom-agent"
}
if s.StateDir == "" {
s.StateDir = "/var/lib/felhom-agent"
}
if s.DwellSeconds == 0 {
s.DwellSeconds = 60
}
return s
}
// WGTunnelConfig configures the offsite WireGuard tunnel (S3, doc 06). **Enabled DEFAULTS TO
// FALSE — the safety gate:** agent releases roll to near-production boxes, and auto-registering
// one into the DEV endpoint on update would be wrong. Enable explicitly per box; the default
// flips only when the production endpoint exists (a later, deliberate decision).
type WGTunnelConfig struct {
Enabled bool `json:"enabled"`
IntervalSeconds int `json:"interval_seconds"` // reconcile cadence; default 60
StateDir string `json:"state_dir"` // key/marker/staged-conf under <StateDir>/wg/; default /var/lib/felhom-agent (the FELHOM_WG sudoers install entry hard-codes this default)
// StaleAfterSeconds is the handshake-age threshold (S4, doc 06 §4.2) beyond which the manager
// re-resolves the endpoint's A record and re-applies on an IP change (endpoint re-IP recovery).
// Default 180 (≈ 3× the 25s keepalive → a healthy tunnel never trips it).
StaleAfterSeconds int `json:"stale_after_seconds"`
}
// WithDefaults fills interval + state dir + staleness threshold.
func (w WGTunnelConfig) WithDefaults() WGTunnelConfig {
if w.IntervalSeconds == 0 {
w.IntervalSeconds = 60
}
if w.StateDir == "" {
w.StateDir = "/var/lib/felhom-agent"
}
if w.StaleAfterSeconds == 0 {
w.StaleAfterSeconds = 180
}
return w
}
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
//
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
// gate here is `Enabled bool` defaulting to false, because those features reach outward (an offsite
// endpoint, an OOB tunnel) and enrolling a box into one by an update would be wrong. This one only
// looks INWARD at guests the agent already owns, and the failure it prevents — an unsupervised DHCP
// client dying and taking the box off the internet 1-2 hours later, invisibly
// (INCIDENT-guest-dhclient-killed-2026-07-20) — is one every box has today. A watchdog that must be
// remembered per box is a watchdog that is missing on the box that needed it. Opting out is
// therefore the explicit act: `"guest_net": {"disable": true}`.
type GuestNetConfig struct {
Disable bool `json:"disable"` // explicit opt-OUT; default is enabled
IntervalSeconds int `json:"interval_seconds"` // probe cadence; default 60
MinHealIntervalSeconds int `json:"min_heal_interval_seconds"` // per-guest cool-off; default 600
MaxHealsPerHour int `json:"max_heals_per_hour"` // per-guest hourly cap; default 3
SettleSeconds int `json:"settle_seconds"` // boot-race guard (guest AND agent uptime); default 180
}
// Enabled reports whether the guest-network watchdog should run.
func (g GuestNetConfig) Enabled() bool { return !g.Disable }
// WithDefaults fills the cadence and the three dampers. A NEGATIVE value is honoured as-is by the
// watchdog constructor's own guards, so an operator can set 0 to mean "package default" without
// having to know the number.
func (g GuestNetConfig) WithDefaults() GuestNetConfig {
if g.IntervalSeconds == 0 {
g.IntervalSeconds = 60
}
if g.MinHealIntervalSeconds == 0 {
g.MinHealIntervalSeconds = 600
}
if g.MaxHealsPerHour == 0 {
g.MaxHealsPerHour = 3
}
if g.SettleSeconds == 0 {
g.SettleSeconds = 180
}
return g
}
// LANResolverConfig configures the host-level split-horizon DNS resolver (internal/lanresolver): a
@@ -90,6 +235,16 @@ type LocalAPIConfig struct {
// TokenStore is the durable, hashed token→guest map (only a HASH of each token is
// persisted; the plaintext exists transiently at mint→write-to-mount, then is discarded).
TokenStore string `json:"token_store"` // default /var/lib/felhom-agent/local-tokens.log
// IslandBridge + IslandGuestAddr configure the R-50 host-internal control-plane bridge. When
// BOTH are set, the provisioner attaches each guest a static net1 on IslandBridge with
// IslandGuestAddr, so the controller reaches the agent over a fixed private address that no
// LAN/DHCP/site move can invalidate (the F1 fix — AUDIT-vacation-remote-ops-2026-07-20). Empty
// (the default) = LAN-only, byte-for-byte the pre-R-50 behaviour. On an island install ListenAddr
// is the host side (169.254.253.1:8443); IslandGuestAddr is the guest side (169.254.253.2/30 — a
// /30 is exactly host + one guest). Additive-only: it never removes a NIC, so a guest restored on
// a non-island host (both empty) is unaffected.
IslandBridge string `json:"island_bridge"` // e.g. "vmbr9" (portless host-internal bridge)
IslandGuestAddr string `json:"island_guest_addr"` // guest net1 CIDR, e.g. "169.254.253.2/30"
}
// Default local-API file locations (under the agent's state dir).
@@ -104,6 +259,12 @@ func (l LocalAPIConfig) Enabled() bool {
return l.Enable && strings.TrimSpace(l.ListenAddr) != ""
}
// IslandEnabled reports whether the provisioner should attach a guest island NIC (net1). True only
// when BOTH the bridge and the guest CIDR are set (R-50); empty = pre-R-50 LAN-only behaviour.
func (l LocalAPIConfig) IslandEnabled() bool {
return strings.TrimSpace(l.IslandBridge) != "" && strings.TrimSpace(l.IslandGuestAddr) != ""
}
// TokenStorePath returns the configured token-store path (default applied).
func (l LocalAPIConfig) TokenStorePath() string {
if l.TokenStore != "" {
@@ -138,6 +299,17 @@ func (l LocalAPIConfig) Validate() error {
if _, _, err := net.SplitHostPort(l.ListenAddr); err != nil {
return fmt.Errorf("config: local_api.listen_addr %q is not host:port: %w", l.ListenAddr, err)
}
// R-50: island fields are all-or-nothing, and the guest addr must be a CIDR (the net1 ip= value).
// A half-set island (bridge without guest addr, or vice versa) is a provisioning mistake, not a
// silent LAN fallback — fail loudly so a botched install config is caught at load, not at day-0.
if (strings.TrimSpace(l.IslandBridge) != "") != (strings.TrimSpace(l.IslandGuestAddr) != "") {
return fmt.Errorf("config: local_api.island_bridge and local_api.island_guest_addr must be set together (got bridge=%q guest_addr=%q)", l.IslandBridge, l.IslandGuestAddr)
}
if l.IslandEnabled() {
if _, _, err := net.ParseCIDR(strings.TrimSpace(l.IslandGuestAddr)); err != nil {
return fmt.Errorf("config: local_api.island_guest_addr %q is not a CIDR (want e.g. 169.254.253.2/30): %w", l.IslandGuestAddr, err)
}
}
return nil
}
@@ -161,14 +333,34 @@ type BackupConfig struct {
LocalBackupTarget string `json:"local_backup_target"`
// RestoreStorage is where a restore-test's restored rootfs lands, e.g. "local-lvm".
RestoreStorage string `json:"restore_storage"`
// RestoreTestCadenceSeconds is the self-restore-test interval; 0 → default (24h).
// Set negative to DISABLE the automatic cadence (on-demand selftest still works).
// RestoreTestCadenceSeconds is the LEGACY restore-test knob, retained for one meaning only:
// NEGATIVE still DISABLES the automatic restore-test entirely (on-demand selftest still works),
// and 0 still means "use the default". It no longer sets how often a test runs — R-86 replaced
// the interval trigger with a per-archive due-check — so a positive value now seeds
// RestoreTestSettleSeconds instead (see RestoreTestSettle). Prefer the two explicit keys below.
RestoreTestCadenceSeconds int `json:"restore_test_cadence_seconds"`
// RestoreTestEvalIntervalSeconds is how often the scheduler ASKS whether any tier is due
// (R-86); 0 → default. It is not how often a test runs: a tier is tested once per archive
// generation no matter how often it is asked. This interval sets two things — the latency
// between an archive settling and its proof, and the retry rate of a tier whose restore-test
// keeps failing. See defaultRestoreTestEvalInterval for the measurement it was chosen from.
RestoreTestEvalIntervalSeconds int `json:"restore_test_eval_interval_seconds"`
// RestoreTestSettleSeconds is how long an archive must have sat on its tier before it is a
// restore-test candidate (R-86); 0 → default (24h), negative → 0 (no settle requirement).
// Restore-testing an archive a backup is still writing proves nothing about the backup that
// finished — this is the same settle discipline R-71a's gate applies to the offsite consume.
RestoreTestSettleSeconds int `json:"restore_test_settle_seconds"`
// ScratchVMIDMin/Max bound the throwaway restore-test scratch-guest VMID band. The
// restore-test refuses to run unless this is a valid band (min>0, max>=min); 9999 is
// always excluded. Defaults to 990000990009.
ScratchVMIDMin int `json:"scratch_vmid_min"`
ScratchVMIDMax int `json:"scratch_vmid_max"`
// RestoreTestPBSRestoreTimeoutSeconds bounds the wait on a PBS-tier (offsite/WAN) restore-test
// restore task; 0 → default 120m. A large guest restored over a slow home uplink runs long, and
// for an UNATTENDED nightly test a false timeout (→ mid-restore teardown → leaked scratch) is
// worse than a slow pass. Very large guests may need a higher value. LOCAL-tier restores keep
// the 10m WaitOptions default (a local restore hanging 10m is a genuine fault).
RestoreTestPBSRestoreTimeoutSeconds int `json:"restore_test_pbs_restore_timeout_seconds"`
// PBS (slice 6 Phase B). The verify maintenance loop runs on its own cadence (cheaper +
// more frequent than the full restore-test); 0 → default (6h), negative → disabled.
@@ -181,6 +373,157 @@ type BackupConfig struct {
// its newest successful backup is older than this (or none exists). 0 → default (24h). The
// hub-served per-guest policy is slice 10; this is the agent-local cadence.
BackupCadenceSeconds int `json:"backup_cadence_seconds"`
// LocalBackupRetention is keep-last=N for the per-run `--prune-backups` on a LOCAL vzdump target —
// so the agent's own local whole-guest backups can't pile up and refill root (the felhom-pve incident;
// the host_disk + storage_fill checkers are the detectors, this is the preventive default). 0/unset →
// default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup.
// NEVER applied to a PBS target (offsite retention is a separate lifecycle).
LocalBackupRetention int `json:"local_backup_retention"`
// ExtraTargets (R-82) are ADDITIONAL backup tiers beyond the primary one above — the shape that
// makes "local daily + PBS weekly" expressible at all. Each carries its OWN cadence and its OWN
// retention, because those are semantically different per tier: keep-last=3 on a daily tier is
// three DAYS of restore points; on a weekly tier it is three WEEKS. Sharing one knob between
// tiers silently means one of them is wrong.
//
// ADDITIVE BY CONSTRUCTION: an existing config with no `backup_targets` key resolves to exactly
// one tier — the primary — and behaves byte-identically to pre-R-82. Nothing here changes the
// local tier.
ExtraTargets []BackupTargetConfig `json:"backup_targets"`
}
// BackupTargetConfig is ONE additional backup tier: a vzdump storage plus its own cadence and
// retention. A tier with no cadence is not a tier — see BackupTiers for why that is rejected loudly
// rather than defaulted.
type BackupTargetConfig struct {
// TargetID is the Proxmox storage id (content=backup), e.g. "felhom-pbs".
TargetID string `json:"target_id"`
// CadenceSeconds is THIS tier's /backup/due window. REQUIRED (>0) — see BackupTiers.
CadenceSeconds int `json:"cadence_seconds"`
// KeepLast is THIS tier's per-run `--prune-backups` keep-last. 0/unset → NEVER prune this tier
// (the fail-safe default, and the current behaviour for every PBS target). A PBS tier is never
// pruned by the per-run flag regardless — see BackupRunner.localPruneSpec.
KeepLast int `json:"keep_last"`
// WaitTimeoutSeconds bounds how long the agent WAITS for this tier's vzdump task. 0/unset →
// defaultExtraTierWaitTimeout.
//
// THIS FIELD EXISTS BECAUSE OF A LIVE FAILURE (2026-07-26, R-82 Slice A validation). The runner
// hard-coded a 30-minute wait, which is right for a local vzdump (minutes) and badly wrong for
// an offsite PBS backup over a home uplink: the first full ~10 GB snapshot ran past 30 min, the
// agent gave up waiting and recorded success=false — WHILE THE BACKUP WAS STILL RUNNING. That
// false failure is worse than a slow pass: the tier stays "due", a retry collides with the
// guest lock vzdump still holds, and the hub sees a DR tier that never succeeds.
//
// Same reasoning as RestoreTestPBSRestoreTimeoutSeconds on the restore side, and the same
// direction: when in doubt wait LONGER. A slow backup is a slow backup; a false timeout is a
// corrupt status plus lock contention.
WaitTimeoutSeconds int `json:"wait_timeout_seconds"`
}
// Per-tier vzdump wait bounds.
//
// The PRIMARY keeps the historical 30 minutes: it is the local tier, a local vzdump takes minutes,
// and one hanging 30 minutes is a genuine fault worth surfacing. Unchanged behaviour.
//
// An ADDITIONAL tier is by construction the offsite/WAN one in this design, where the binding
// constraint is uplink speed, not health. Measured on demo-felhom: ~33 MB/min over the wg link to
// Hetzner, so a first FULL ~10 GB snapshot projects to ~5h. Operator ruling 2026-07-26: "let the
// first backup run as long as needed" — 12h gives that real margin on a slower link while still
// being BOUNDED, so a genuinely hung task eventually surfaces instead of hanging forever.
const (
defaultPrimaryTierWaitTimeout = 30 * time.Minute
defaultExtraTierWaitTimeout = 12 * time.Hour
)
// BackupTier is a RESOLVED backup tier: one target, its own cadence, its own retention. The agent
// builds one runner per tier from these.
type BackupTier struct {
TargetID string
Cadence time.Duration
// WaitTimeout bounds the wait on this tier's vzdump task (see WaitTimeoutSeconds).
WaitTimeout time.Duration
// KeepLast is the per-run prune keep-last; 0 means DO NOT PRUNE this tier.
KeepLast int
// Primary marks the tier that the UNTARGETED local-API endpoints act on — the pre-R-82 tier.
// Exactly one tier is primary, and it is always first.
Primary bool
}
// BackupTiers resolves the effective tier list, primary first, plus any warnings the caller MUST
// log (they describe tiers that were REJECTED, and a silently-dropped backup tier is precisely the
// "applied and empty" fault R-82 exists to fix).
//
// Rules:
// - Tier 0 is always the primary, built from BackupTarget()/BackupCadence()/KeepLast() — so a
// config with no `backup_targets` is byte-identical to pre-R-82.
// - An extra with an empty target_id is rejected.
// - An extra with cadence_seconds <= 0 is REJECTED, not defaulted. Defaulting a PBS tier to the
// 24h local default would quietly turn a weekly tier into a daily one and fill the DR datastore;
// a tier whose cadence you did not state is not a tier.
// - An extra repeating the primary's target is rejected (one policy per target, or the two
// cadences race and neither is the truth).
// - Duplicate extras are rejected after the first.
func (b BackupConfig) BackupTiers() ([]BackupTier, []string) {
primary := BackupTier{
TargetID: b.BackupTarget(),
Cadence: b.BackupCadence(),
KeepLast: b.KeepLast(),
WaitTimeout: defaultPrimaryTierWaitTimeout,
Primary: true,
}
tiers := []BackupTier{primary}
var warnings []string
seen := map[string]bool{primary.TargetID: true}
for i, t := range b.ExtraTargets {
id := strings.TrimSpace(t.TargetID)
switch {
case id == "":
warnings = append(warnings, fmt.Sprintf("backup_targets[%d]: empty target_id — tier ignored", i))
continue
case seen[id]:
warnings = append(warnings, fmt.Sprintf("backup_targets[%d]: target %q already configured — duplicate tier ignored", i, id))
continue
case t.CadenceSeconds <= 0:
warnings = append(warnings, fmt.Sprintf("backup_targets[%d] (%s): cadence_seconds must be > 0 — tier ignored (a cadence is NOT defaulted: a weekly tier silently running daily would fill the DR datastore)", i, id))
continue
}
seen[id] = true
keep := t.KeepLast
if keep < 0 {
keep = 0
}
wait := defaultExtraTierWaitTimeout
if t.WaitTimeoutSeconds > 0 {
wait = time.Duration(t.WaitTimeoutSeconds) * time.Second
}
tiers = append(tiers, BackupTier{
TargetID: id,
Cadence: time.Duration(t.CadenceSeconds) * time.Second,
KeepLast: keep,
WaitTimeout: wait,
})
}
return tiers, warnings
}
// defaultLocalBackupKeepLast is the local vzdump retention default (newest N restore points kept).
const defaultLocalBackupKeepLast = 3
// KeepLast returns the effective local-backup keep-last, clamped to ≥1 (0/unset → default 3, negative →
// default). The clamp is load-bearing: keep-last=0 would tell PVE to prune EVERY archive, including the
// one just made — a mis-config must never self-destruct the fresh backup.
func (b BackupConfig) KeepLast() int {
if b.LocalBackupRetention < 1 {
return defaultLocalBackupKeepLast
}
return b.LocalBackupRetention
}
// PruneBackupsSpec returns the PVE `--prune-backups` value for the local vzdump (e.g. "keep-last=3").
func (b BackupConfig) PruneBackupsSpec() string {
return fmt.Sprintf("keep-last=%d", b.KeepLast())
}
// BackupCadence returns the per-guest /backup/due window: positive as-is, else 24h default.
@@ -191,6 +534,14 @@ func (b BackupConfig) BackupCadence() time.Duration {
return 24 * time.Hour
}
// RestoreTestPBSRestoreTimeout returns the PBS-tier restore-task wait: positive as-is, else 120m.
func (b BackupConfig) RestoreTestPBSRestoreTimeout() time.Duration {
if b.RestoreTestPBSRestoreTimeoutSeconds > 0 {
return time.Duration(b.RestoreTestPBSRestoreTimeoutSeconds) * time.Second
}
return 120 * time.Minute
}
// defaultBackupTarget is the offsite PBS storage whole-guest backups land on by default. It is
// SEPARATE HARDWARE from the guest's own disk (a PBS datastore on the DooPlex box), so a host
// disk/hardware failure doesn't take the backups with it — that's what makes it real DR. Proven
@@ -208,26 +559,92 @@ func (b BackupConfig) BackupTarget() string {
return defaultBackupTarget
}
// Default scratch VMID band + restore-test cadence.
// Default scratch VMID band + the two R-86 restore-test knobs.
const (
defaultScratchVMIDMin = 990000
defaultScratchVMIDMax = 990009
defaultRestoreTestCadence = 24 * time.Hour
defaultScratchVMIDMin = 990000
defaultScratchVMIDMax = 990009
// defaultRestoreTestEvalInterval is how often due-ness is ASKED. It is bounded from BOTH sides,
// and neither bound alone would have picked it:
//
// FLOOR — what one evaluation costs. MEASURED on demo-felhom, 2026-08-03 (R-86 Part 1.4), via
// --selftest=restore-test-due and by timing the underlying API call directly. One evaluation
// is one storage-content listing per tier:
//
// local dir storage (3 archives) ....... 18 ms (18.7 / 18.3 / 18.5)
// PBS tier, WAN to ep0 (2 snapshots) ... 392 ms (375 / 378 / 424)
// both tiers together .................. 430 ms
//
// So cost does NOT set this: even at one evaluation a minute the offsite leg would be ~0.7 %
// of a WAN link's time and ~9 minutes of ep0's day. Worth writing down anyway, because the
// number that would have forbidden a frequent poll is the one nobody measures.
//
// CEILING — the retry rate of a FAILING tier. Under a per-archive due-check a tier whose
// restore-test keeps failing stays due, so the evaluation interval IS its retry interval, and
// a retry is a multi-GB restore. Every few minutes would be an incident of its own; the old
// timer retried a broken tier once a day.
//
// 6h sits between them: four heavy retries a day at the very worst, latency from settle to
// proof of at most 6h against a 24h settle lag (so a daily tier is still proved daily), and no
// second rate limiter anywhere — the pacing remains one test per archive generation.
defaultRestoreTestEvalInterval = 6 * time.Hour
// defaultRestoreTestSettle is how long an archive must sit before it may be restore-tested.
// 24h is R-86's own figure ("~24 h after its own newest archive") and it is what makes the
// candidate on a daily tier YESTERDAY's archive rather than the one still being written.
defaultRestoreTestSettle = 24 * time.Hour
)
// RestoreTestCadence returns the configured restore-test interval: a positive value as-is,
// 0 → 24h default, negative → 0 (disabled).
func (b BackupConfig) RestoreTestCadence() time.Duration {
// RestoreTestEvalInterval returns how often the scheduler evaluates due-ness (R-86): a positive
// value as-is, 0 → the measured default, negative → 0 (disabled).
//
// The LEGACY `restore_test_cadence_seconds` keeps exactly one power here, the one a box may be
// relying on: a NEGATIVE value still disables the automatic restore-test outright. It no longer
// sets the interval, because the interval no longer decides that a test happens.
func (b BackupConfig) RestoreTestEvalInterval() time.Duration {
if b.RestoreTestCadenceSeconds < 0 {
return 0 // legacy DISABLE — preserved verbatim
}
switch {
case b.RestoreTestCadenceSeconds > 0:
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second
case b.RestoreTestCadenceSeconds < 0:
case b.RestoreTestEvalIntervalSeconds > 0:
return time.Duration(b.RestoreTestEvalIntervalSeconds) * time.Second
case b.RestoreTestEvalIntervalSeconds < 0:
return 0 // disabled
default:
return defaultRestoreTestCadence
return defaultRestoreTestEvalInterval
}
}
// RestoreTestSettle returns how long an archive must have sat before it is a restore-test
// candidate (R-86): a positive value as-is, negative → 0 (no settle requirement), 0 → the default.
//
// WHAT HAPPENED TO THE OLD KEY. A box that set `restore_test_cadence_seconds` to a positive value
// was expressing "how long may pass between a backup and the confidence that it restores". That
// quantity survives R-86 as the SETTLE LAG, so a positive legacy value seeds this rather than being
// dropped or silently repurposed as the evaluation interval — and the daemon says so at start-up
// (see RestoreTestLegacyCadenceInUse). It is deliberately not carried into the evaluation interval:
// a box that set 72h to spare a weak endpoint would otherwise get a 72h-latency due-check, whereas
// what it actually wanted — fewer heavy restores — is what per-archive due-ness already gives it.
func (b BackupConfig) RestoreTestSettle() time.Duration {
switch {
case b.RestoreTestSettleSeconds > 0:
return time.Duration(b.RestoreTestSettleSeconds) * time.Second
case b.RestoreTestSettleSeconds < 0:
return 0 // explicitly no settle requirement
case b.RestoreTestCadenceSeconds > 0:
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second // legacy seeding
default:
return defaultRestoreTestSettle
}
}
// RestoreTestLegacyCadenceInUse reports whether the deprecated key is what is deciding the settle
// lag, so the daemon can name both replacements ONCE at start-up. A config key that changed meaning
// without saying so is exactly the silent repurposing §8.3 forbids.
func (b BackupConfig) RestoreTestLegacyCadenceInUse() bool {
return b.RestoreTestCadenceSeconds > 0 && b.RestoreTestSettleSeconds == 0
}
// PBSVerifyCadence returns the verify-loop interval: positive as-is, 0 → 6h default,
// negative → 0 (disabled).
func (b BackupConfig) PBSVerifyCadence() time.Duration {
@@ -386,6 +803,10 @@ type PrivilegedConfig struct {
Install string `json:"install"` // default /usr/bin/install
Smartctl string `json:"smartctl"` // default /usr/sbin/smartctl
Lvs string `json:"lvs"` // default /usr/sbin/lvs
// SmbCredsDir is where the agent writes 0600 SMB credentials files for network storage (Part A1).
// Out-of-band: never committed, never logged. Default /var/lib/felhom-agent/smb-creds (agent-owned).
SmbCredsDir string `json:"smb_creds_dir"`
}
// Default returns a Config pre-populated with sane defaults.
@@ -412,6 +833,7 @@ func Load(path string) (Config, error) {
return cfg, fmt.Errorf("config: parsing %s: %w", path, err)
}
}
cfg.SourcePath = path // where this config came from (pbsdr's escrow seed writes back here)
applyEnv(&cfg)
return cfg, nil
}
@@ -442,6 +864,9 @@ func applyEnv(cfg *Config) {
if v := os.Getenv("FELHOM_AGENT_LOG_LEVEL"); v != "" {
cfg.LogLevel = v
}
if v := os.Getenv("FELHOM_AGENT_DEPLOYMENT_MODE"); v != "" {
cfg.DeploymentMode = v
}
// hub
if v := os.Getenv("FELHOM_AGENT_HUB_URL"); v != "" {
cfg.Hub.URL = v
@@ -465,6 +890,8 @@ func applyEnv(cfg *Config) {
cfg.Backup.RestoreStorage = v
}
cfg.Backup.RestoreTestCadenceSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_CADENCE_SECONDS", cfg.Backup.RestoreTestCadenceSeconds)
cfg.Backup.RestoreTestEvalIntervalSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_EVAL_INTERVAL_SECONDS", cfg.Backup.RestoreTestEvalIntervalSeconds)
cfg.Backup.RestoreTestSettleSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_SETTLE_SECONDS", cfg.Backup.RestoreTestSettleSeconds)
}
// envInt overlays an int env var, keeping cur (with a stderr warning) on parse
@@ -507,6 +934,9 @@ func (c Config) Redacted() Config {
if c.Hub.APIKey != "" {
c.Hub.APIKey = "********"
}
if c.SelfUpdate.Token != "" {
c.SelfUpdate.Token = "********"
}
return c
}
+102
View File
@@ -5,8 +5,29 @@ import (
"path/filepath"
"strings"
"testing"
"time"
)
// TestRestoreTestPBSRestoreTimeout mirrors the BackupCadence accessor contract: positive as-is,
// 0 → default (120m), negative → default.
func TestRestoreTestPBSRestoreTimeout(t *testing.T) {
cases := []struct {
secs int
want time.Duration
}{
{0, 120 * time.Minute},
{-5, 120 * time.Minute},
{1800, 30 * time.Minute},
{7200, 120 * time.Minute},
}
for _, c := range cases {
got := BackupConfig{RestoreTestPBSRestoreTimeoutSeconds: c.secs}.RestoreTestPBSRestoreTimeout()
if got != c.want {
t.Errorf("RestoreTestPBSRestoreTimeout(secs=%d) = %v, want %v", c.secs, got, c.want)
}
}
}
func TestRedactedMasksSecret(t *testing.T) {
c := Default()
c.Proxmox.Token = "felhom-agent@pve!agent=b6547d9d-08ec-4f22-beb8-a551dc2cd69d"
@@ -112,3 +133,84 @@ func TestLoadFileThenEnvOverride(t *testing.T) {
t.Errorf("default endpoint lost: %q", cfg.Proxmox.Endpoint)
}
}
// CAMPAIGN-3 Part 6: deployment_mode gates node self-heal, and it is FAIL-SAFE to byo — absent or any
// unknown value is byo, ONLY the exact "appliance" unlocks the remedy.
func TestIsAppliance_FailSafeToByo(t *testing.T) {
cases := []struct {
mode string
want bool
}{
{"appliance", true},
{"byo", false},
{"", false}, // absent field → byo (fail-safe)
{"Appliance", false}, // case-sensitive — a typo must not unlock the remedy
{"garbage", false},
}
for _, c := range cases {
cfg := &Config{DeploymentMode: c.mode}
if got := cfg.IsAppliance(); got != c.want {
t.Errorf("IsAppliance(mode=%q) = %t, want %t", c.mode, got, c.want)
}
}
}
// The env overlay can set deployment_mode (FELHOM_AGENT_DEPLOYMENT_MODE).
func TestDeploymentModeEnvOverlay(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "agent.json")
if err := os.WriteFile(path, []byte(`{"proxmox":{"node":"n","token":"u@pve!t=s"},"deployment_mode":"byo"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("FELHOM_AGENT_DEPLOYMENT_MODE", "appliance")
cfg, err := Load(path)
if err != nil {
t.Fatalf("Load: %v", err)
}
if !cfg.IsAppliance() {
t.Errorf("env overlay did not set deployment_mode: %q", cfg.DeploymentMode)
}
}
// R-50: the island NIC fields are all-or-nothing and the guest addr must be a CIDR. A half-set or
// malformed island must fail at config load (a botched install) rather than silently fall back to
// LAN-only, which would leave a guest with an island bind and no island NIC — the exact silent break
// R-50 exists to kill. Covers LocalAPIConfig.Validate + IslandEnabled.
func TestLocalAPIConfig_IslandValidation(t *testing.T) {
base := LocalAPIConfig{Enable: true, ListenAddr: "169.254.253.1:8443"}
// both empty → fine (pre-R-50 default), IslandEnabled false
if err := base.Validate(); err != nil {
t.Errorf("no island config must validate: %v", err)
}
if base.IslandEnabled() {
t.Errorf("IslandEnabled must be false when unset")
}
// both set, valid CIDR → fine, IslandEnabled true
ok := base
ok.IslandBridge, ok.IslandGuestAddr = "vmbr9", "169.254.253.2/30"
if err := ok.Validate(); err != nil {
t.Errorf("valid island config must validate: %v", err)
}
if !ok.IslandEnabled() {
t.Errorf("IslandEnabled must be true when both set")
}
// bridge only → rejected (all-or-nothing)
half := base
half.IslandBridge = "vmbr9"
if err := half.Validate(); err == nil {
t.Errorf("half-set island (bridge only) must be rejected")
}
// guest addr only → rejected
half2 := base
half2.IslandGuestAddr = "169.254.253.2/30"
if err := half2.Validate(); err == nil {
t.Errorf("half-set island (guest addr only) must be rejected")
}
// both set but guest addr is not a CIDR → rejected
bad := base
bad.IslandBridge, bad.IslandGuestAddr = "vmbr9", "169.254.253.2" // missing /30
if err := bad.Validate(); err == nil {
t.Errorf("island guest addr without a CIDR mask must be rejected")
}
}
+90
View File
@@ -0,0 +1,90 @@
package desired
// S3 Group C — the raw-consumer fan-out seam: called on generation advance, NOT on no-advance,
// and a panicking consumer is contained (the guest reconcile path must never break).
import (
"context"
"io"
"log/slog"
"sync"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
type recordingConsumer struct {
mu sync.Mutex
calls []*hub.DesiredStateResponse
panic bool
}
func (r *recordingConsumer) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
r.mu.Lock()
r.calls = append(r.calls, resp)
r.mu.Unlock()
if r.panic {
panic("consumer exploded")
}
}
func (r *recordingConsumer) count() int { r.mu.Lock(); defer r.mu.Unlock(); return len(r.calls) }
type stubFetcher struct{ resp *hub.DesiredStateResponse }
func (s *stubFetcher) FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error) {
return s.resp, nil
}
func testResp(gen int64) *hub.DesiredStateResponse {
return &hub.DesiredStateResponse{
Generation: gen,
DesiredState: hub.WireDesiredState{
Guests: []hub.WireDesiredGuest{},
Wireguard: &hub.WireWireguard{Pubkey: "PK", AssignedIP: "10.77.0.2/32"},
},
}
}
func TestSyncer_ConsumerCalledOnAdvanceOnly(t *testing.T) {
provider := reconcile.NewCachingProvider()
f := &stubFetcher{resp: testResp(2)}
s := NewSyncer(f, provider, slog.New(slog.NewTextHandler(io.Discard, nil)))
c := &recordingConsumer{}
s.AddConsumer(c)
// Advance → fetch → consumer called with the raw doc (wireguard block intact).
s.OnEnvelope(context.Background(), &hub.ControlEnvelope{DesiredGeneration: 2})
if c.count() != 1 {
t.Fatalf("consumer calls = %d, want 1", c.count())
}
if c.calls[0].DesiredState.Wireguard == nil || c.calls[0].DesiredState.Wireguard.Pubkey != "PK" {
t.Fatalf("consumer got %+v — the raw wireguard block must ride through", c.calls[0].DesiredState.Wireguard)
}
// No advance → no fetch → no consumer call (the negative).
s.OnEnvelope(context.Background(), &hub.ControlEnvelope{DesiredGeneration: 2})
if c.count() != 1 {
t.Errorf("consumer called without a generation advance: %d", c.count())
}
}
func TestSyncer_PanickingConsumerContained(t *testing.T) {
provider := reconcile.NewCachingProvider()
f := &stubFetcher{resp: testResp(1)}
s := NewSyncer(f, provider, slog.New(slog.NewTextHandler(io.Discard, nil)))
bomb := &recordingConsumer{panic: true}
after := &recordingConsumer{}
s.AddConsumer(bomb)
s.AddConsumer(after)
// Must not panic out; the second consumer still runs; the provider still updated.
s.OnEnvelope(context.Background(), &hub.ControlEnvelope{DesiredGeneration: 1})
if after.count() != 1 {
t.Errorf("consumer after the panicking one not called: %d", after.count())
}
if provider.Generation() != 1 {
t.Errorf("provider generation = %d, want 1 (guest path unaffected)", provider.Generation())
}
}
+44 -4
View File
@@ -12,6 +12,7 @@ package desired
import (
"context"
"log/slog"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
@@ -22,11 +23,28 @@ type Fetcher interface {
FetchDesiredState(ctx context.Context) (*hub.DesiredStateResponse, error)
}
// RawConsumer receives the FULL fetched desired-state document after each successful
// generation-advance fetch (S3 seam — internal/wgtunnel consumes its wireguard block this way
// without the reconcile engine learning about tunnels). Implementations must not block: do the
// cheap store-and-nudge, never network/exec inline.
type RawConsumer interface {
OnDesiredState(ctx context.Context, resp *hub.DesiredStateResponse)
}
// Syncer keeps the engine's CachingProvider in step with the hub's authoritative desired-state.
type Syncer struct {
fetcher Fetcher
provider *reconcile.CachingProvider
logger *slog.Logger
fetcher Fetcher
provider *reconcile.CachingProvider
consumers []RawConsumer
logger *slog.Logger
}
// AddConsumer registers a raw desired-state consumer (nil-safe no-op). Not concurrency-safe —
// call during wiring, before the hub loop starts.
func (s *Syncer) AddConsumer(c RawConsumer) {
if c != nil {
s.consumers = append(s.consumers, c)
}
}
// NewSyncer builds a Syncer over the hub fetcher and the engine's provider.
@@ -49,24 +67,44 @@ func (s *Syncer) OnEnvelope(ctx context.Context, env *hub.ControlEnvelope) {
if env.DesiredGeneration <= have {
return // cached: the heavy desired-state moves only on a generation advance
}
s.logger.Debug("desired: generation advanced — fetching desired-state",
"have_generation", have, "envelope_generation", env.DesiredGeneration)
start := time.Now()
resp, err := s.fetcher.FetchDesiredState(ctx)
if err != nil {
s.logger.Warn("desired: fetch failed; keeping cached desired-state",
"have_generation", have, "envelope_generation", env.DesiredGeneration, "err", err)
return
}
s.logger.Debug("desired: fetched", "generation", resp.Generation,
"duration_ms", time.Since(start).Milliseconds())
state := mapWire(resp.DesiredState, s.logger)
// Cache against the FETCHED generation (not the envelope's) — robust to a generation that
// advanced again between the heartbeat and this fetch (we won't re-fetch the same state).
s.provider.Update(resp.Generation, state)
s.logger.Info("desired: updated from hub",
"generation", resp.Generation, "guests", len(state.Guests))
// S3: fan the raw document out to registered consumers (wgtunnel etc). A panicking consumer
// is contained — the guest reconcile path must never break over a tunnel add-on.
for _, c := range s.consumers {
s.notifyConsumer(ctx, c, resp)
}
if env.HasSignedOps {
// 10A only notes the flag; fetching + verifying + executing signed ops is slice 10B.
s.logger.Info("desired: hub reports pending signed ops (fetch/execute is slice 10B)")
}
}
// notifyConsumer delivers one raw document with panic containment.
func (s *Syncer) notifyConsumer(ctx context.Context, c RawConsumer, resp *hub.DesiredStateResponse) {
defer func() {
if r := recover(); r != nil {
s.logger.Error("desired: raw consumer panicked (contained)", "panic", r)
}
}()
c.OnDesiredState(ctx, resp)
}
// mapWire maps the hub wire desired-state to the reconcile domain. 10A acts only on guests; the
// forward-compat fields (restore_directive — 10D — etc.) are carried on the wire and logged, but
// not translated into actions here.
@@ -90,7 +128,9 @@ func mapWire(w hub.WireDesiredState, logger *slog.Logger) reconcile.DesiredState
guests[g.VMID] = dg
}
if w.RestoreDirective != nil {
logger.Info("desired: restore_directive present (consumed in slice 10D — ignored in 10A)",
// The reconcile mapping does NOT act on the directive; the DR consumer (raw-consumer seam,
// S5 internal/dr) surfaces it as an inspectable restore PLAN — no restore is executed here.
logger.Info("desired: restore_directive present (surfaced as a restore PLAN by the DR consumer; not acted on in the reconcile mapping)",
"mode", w.RestoreDirective.Mode)
}
return reconcile.DesiredState{Guests: guests}
+134
View File
@@ -0,0 +1,134 @@
// Package dr consumes the host-loss restore_directive (slice 10D / S5) into an inspectable restore
// PLAN. It is DERIVE-AND-SURFACE only: the plan is logged (and exposed for the report), never
// executed — the destructive restore is a separate, operator-present, STOP-gated step. The Consumer
// has NO restore/destroy API by construction, so "execute nothing" is a structural guarantee.
package dr
import (
"context"
"log/slog"
"sync"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// RestorePlan is the derived-but-not-executed host-loss plan: per guest → restore coords + sizing;
// per drive → durable_id → expected mount. No secrets (coordinates/identifiers/sizes only).
type RestorePlan struct {
Mode string `json:"mode"`
Guests []PlannedGuest `json:"guests"`
Drives []PlannedDrive `json:"drives"`
PBS *hub.DRPBSCoord `json:"pbs,omitempty"` // WHERE the offsite backups live (repo/ns/latest snapshot)
}
// PlannedGuest is one guest to restore in place, from the offsite datastore, at its original sizing.
type PlannedGuest struct {
VMID int `json:"vmid"`
Archive string `json:"archive,omitempty"` // explicit archive from the directive; "" → resolve latest at restore time
TargetStorage string `json:"target_storage"` // where the restored volumes land (e.g. local-lvm)
Cores int `json:"cores"`
MemoryBytes int64 `json:"memory_bytes"`
DiskBytes int64 `json:"disk_bytes"`
}
// PlannedDrive is one data drive to re-attach BY DURABLE_ID (the wrong-disk guard: a match attaches,
// a non-match is refused — the matcher, exercised in the Part-4 spike, never resolves to a near disk).
type PlannedDrive struct {
DurableID string `json:"durable_id"`
ExpectedMount string `json:"expected_mount"`
Intent string `json:"intent"`
}
// BuildRestorePlan derives the plan from a host_loss directive + the live DR recipe. Returns
// (nil,false) for a guest_loss/absent directive or a nil recipe (nothing to plan). PURE: reads
// nothing, executes nothing — the whole point of this slice's safe half.
func BuildRestorePlan(directive *hub.WireRestoreDirective, recipe *hub.DRRecipeHostHalf, restoreStorage string) (*RestorePlan, bool) {
if directive == nil || directive.Mode != "host_loss" || recipe == nil {
return nil, false
}
plan := &RestorePlan{Mode: directive.Mode, PBS: recipe.PBS}
for _, g := range recipe.Guests {
pg := PlannedGuest{
VMID: g.VMID,
TargetStorage: restoreStorage,
Cores: g.Cores,
MemoryBytes: g.MemoryBytes,
DiskBytes: g.DiskBytes,
}
// The directive may name an explicit archive for a specific guest (else the restore step
// resolves the latest snapshot from the PBS coord at execution time).
if directive.Archive != "" && (directive.VMID == 0 || directive.VMID == g.VMID) {
pg.Archive = directive.Archive
}
plan.Guests = append(plan.Guests, pg)
}
for _, d := range recipe.Drives {
plan.Drives = append(plan.Drives, PlannedDrive{
DurableID: d.DurableID,
ExpectedMount: d.MountPath,
Intent: d.Intent,
})
}
return plan, true
}
// RecipeFunc yields the current DR recipe (the agent-derived scaffolding). It is called ONLY when a
// host_loss directive is present (a rare DR event), so an on-demand Collect is acceptable.
type RecipeFunc func(ctx context.Context) *hub.DRRecipeHostHalf
// Consumer implements desired.RawConsumer: on a host_loss restore_directive it builds + SURFACES the
// plan (structured log + LastPlan for the report/inspection) and executes NOTHING. A guest_loss or
// absent directive clears the plan. It holds no restore/destroy dependency — surfacing is all it can do.
type Consumer struct {
recipe RecipeFunc
restoreStorage string
logger *slog.Logger
mu sync.Mutex
lastPlan *RestorePlan
}
// NewConsumer builds the DR plan consumer. recipe may be nil (then no plan can be built — logged).
func NewConsumer(recipe RecipeFunc, restoreStorage string, logger *slog.Logger) *Consumer {
if logger == nil {
logger = slog.Default()
}
return &Consumer{recipe: recipe, restoreStorage: restoreStorage, logger: logger}
}
// OnDesiredState implements desired.RawConsumer. Non-host_loss → clear + no-op.
func (c *Consumer) OnDesiredState(ctx context.Context, resp *hub.DesiredStateResponse) {
if resp == nil {
return
}
dir := resp.DesiredState.RestoreDirective
if dir == nil || dir.Mode != "host_loss" {
c.mu.Lock()
c.lastPlan = nil
c.mu.Unlock()
return
}
var recipe *hub.DRRecipeHostHalf
if c.recipe != nil {
recipe = c.recipe(ctx)
}
plan, ok := BuildRestorePlan(dir, recipe, c.restoreStorage)
if !ok {
c.logger.Warn("dr: host_loss restore_directive present but no DR recipe available yet — cannot build a plan",
"directive_vmid", dir.VMID)
return
}
c.mu.Lock()
c.lastPlan = plan
c.mu.Unlock()
// SURFACE only — the destructive restore is a separate, operator-present step.
c.logger.Warn("dr: host_loss RESTORE PLAN derived (NOT executed — supervised in-place restore is a separate, gated step)",
"mode", plan.Mode, "guests", len(plan.Guests), "drives", len(plan.Drives), "plan", plan)
}
// LastPlan returns the most recently derived plan (nil if none / cleared). For the report + tests.
func (c *Consumer) LastPlan() *RestorePlan {
c.mu.Lock()
defer c.mu.Unlock()
return c.lastPlan
}
+92
View File
@@ -0,0 +1,92 @@
package dr
import (
"context"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
func sampleRecipe() *hub.DRRecipeHostHalf {
return &hub.DRRecipeHostHalf{
RecipeVersion: 1,
Guests: []hub.DRGuest{{VMID: 9201, Cores: 2, MemoryBytes: 12 << 30, DiskBytes: 32 << 30}},
PBS: &hub.DRPBSCoord{RepoID: "felhom-offsite", Namespace: "demo-felhom-01", LatestSnapshotID: "9201"},
Drives: []hub.DRDrive{{DurableID: "uuid:abc", MountPath: "/mnt/felhom-drives/photos", Intent: "enrolled", TotalBytes: 500 << 30}},
}
}
// TestBuildRestorePlan_HostLoss: a host_loss directive + recipe yields per-guest {vmid, archive,
// target, sizing} + per-drive {durable_id → mount} + the offsite PBS coord.
func TestBuildRestorePlan_HostLoss(t *testing.T) {
dir := &hub.WireRestoreDirective{Mode: "host_loss", VMID: 9201, Archive: "felhom-offsite:backup/ct/9201/2026-07-04T14:55:44Z"}
plan, ok := BuildRestorePlan(dir, sampleRecipe(), "local-lvm")
if !ok || plan == nil {
t.Fatal("host_loss must yield a plan")
}
if plan.Mode != "host_loss" || len(plan.Guests) != 1 || len(plan.Drives) != 1 {
t.Fatalf("plan shape = %+v", plan)
}
g := plan.Guests[0]
if g.VMID != 9201 || g.TargetStorage != "local-lvm" || g.Cores != 2 || g.DiskBytes != 32<<30 {
t.Errorf("planned guest = %+v", g)
}
if g.Archive != dir.Archive {
t.Errorf("planned guest archive = %q, want the directive's %q", g.Archive, dir.Archive)
}
d := plan.Drives[0]
if d.DurableID != "uuid:abc" || d.ExpectedMount != "/mnt/felhom-drives/photos" {
t.Errorf("planned drive (durable_id→mount) = %+v", d)
}
if plan.PBS == nil || plan.PBS.RepoID != "felhom-offsite" {
t.Errorf("plan must carry the offsite PBS coord, got %+v", plan.PBS)
}
}
// TestBuildRestorePlan_NoPlanCases is the red-proof anchor: guest_loss / absent / nil-recipe yield
// NO plan (execute-nothing on the wrong mode). Relaxing the mode gate → the guest_loss case fails.
func TestBuildRestorePlan_NoPlanCases(t *testing.T) {
if _, ok := BuildRestorePlan(&hub.WireRestoreDirective{Mode: "guest_loss", VMID: 9201}, sampleRecipe(), "local-lvm"); ok {
t.Error("guest_loss must NOT yield a host-loss plan")
}
if _, ok := BuildRestorePlan(nil, sampleRecipe(), "local-lvm"); ok {
t.Error("absent directive must NOT yield a plan")
}
if _, ok := BuildRestorePlan(&hub.WireRestoreDirective{Mode: "host_loss"}, nil, "local-lvm"); ok {
t.Error("nil recipe must NOT yield a plan")
}
}
// TestConsumer_SurfacesPlanNeverExecutes: the consumer surfaces the plan on host_loss, consults the
// recipe only then, and clears it otherwise. It has NO restore/destroy dependency (execute-nothing
// is structural — the type literally cannot call a restore).
func TestConsumer_SurfacesPlanNeverExecutes(t *testing.T) {
recipeCalls := 0
c := NewConsumer(func(context.Context) *hub.DRRecipeHostHalf { recipeCalls++; return sampleRecipe() }, "local-lvm", nil)
ds := func(d *hub.WireRestoreDirective) *hub.DesiredStateResponse {
return &hub.DesiredStateResponse{DesiredState: hub.WireDesiredState{RestoreDirective: d}}
}
// non-host_loss → no plan, recipe NOT consulted.
c.OnDesiredState(context.Background(), ds(&hub.WireRestoreDirective{Mode: "guest_loss"}))
if c.LastPlan() != nil {
t.Error("guest_loss set a plan")
}
if recipeCalls != 0 {
t.Errorf("recipe consulted on a non-host_loss directive (%d calls)", recipeCalls)
}
// host_loss → plan surfaced, recipe consulted once.
c.OnDesiredState(context.Background(), ds(&hub.WireRestoreDirective{Mode: "host_loss", VMID: 9201}))
p := c.LastPlan()
if p == nil || len(p.Guests) != 1 || p.Guests[0].VMID != 9201 {
t.Fatalf("host_loss plan = %+v", p)
}
if recipeCalls != 1 {
t.Errorf("recipe calls = %d, want 1", recipeCalls)
}
// absent directive clears the plan.
c.OnDesiredState(context.Background(), ds(nil))
if c.LastPlan() != nil {
t.Error("absent directive did not clear the plan")
}
}
+49
View File
@@ -0,0 +1,49 @@
package escrow
// Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by
// felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API
// re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers
// matches that vector byte-for-byte (spike §2.2: any alteration — value, extra flag, order,
// config path — is refused), so the argv below is the SINGLE SOURCE OF TRUTH shared by the
// exec (localapi), the capability manifest entry, and (byte-identically) the FELHOM_ESCROW
// sudoers line. Never build it with flag helpers and never normalize `--` to `-` — Go's flag
// package would accept either spelling, sudoers only the literal one.
// CeremonyBinary is the installed agent binary path the sudoers line pins.
const CeremonyBinary = "/usr/local/bin/felhom-agent"
// ceremonyArgv is the fixed vector. --config is pinned explicitly: `sudo -n` env_reset strips
// FELHOM_AGENT_CONFIG, and the pin closes env-injection of an alternate config (spike probe (e)).
var ceremonyArgv = []string{
"--config", "/etc/felhom-agent/agent.json",
"--selftest=escrow-create",
"--upload",
"--output=json",
}
// CeremonyArgs returns a fresh copy of the fixed argv (callers must not be able to mutate the
// shared source).
func CeremonyArgs() []string {
out := make([]string, len(ceremonyArgv))
copy(out, ceremonyArgv)
return out
}
// CeremonyOutput is the --output=json wire contract (version 1): the ONE JSON object json mode
// emits on stdout — nothing else lands there; every human/info line goes to stderr. RecoveryCode
// is the only secret field: the consumer must extract it, hand it to the one-shot claim holder,
// and zero both the parsed struct and the raw stdout buffer. (Best-effort — Go's GC may hold
// stale copies; the discipline still shrinks the exposure window.)
type CeremonyOutput struct {
Version int `json:"version"`
RecoveryCode string `json:"recovery_code"`
KeyFingerprint string `json:"key_fingerprint"`
EntropyBits float64 `json:"entropy_bits"`
BlobBytes int `json:"blob_bytes"`
IdentityBlobBytes int `json:"identity_blob_bytes"`
ResticPwSealed bool `json:"restic_pw_sealed"`
Uploaded bool `json:"uploaded"`
}
// CeremonyOutputVersion is the current CeremonyOutput.Version value.
const CeremonyOutputVersion = 1
+22 -6
View File
@@ -11,8 +11,16 @@ import (
)
func TestWordlistLoaded(t *testing.T) {
if WordlistSize() != 7776 {
t.Fatalf("EFF large wordlist should be 7776 words, got %d", WordlistSize())
// The EFF large list is 7776 entries; joinSafe removes the 4 that contain RecoveryCodeSep
// (drop-down, felt-tip, t-shirt, yo-yo), leaving 7772 as the effective draw space.
if got := WordlistSize(); got != 7772 {
t.Fatalf("effective wordlist should be 7772 words (7776 EFF - 4 hyphenated), got %d", got)
}
if got := WordlistFilteredOut(); got != 4 {
t.Fatalf("joinSafe should have removed exactly 4 hyphenated entries, removed %d", got)
}
if got := WordlistSize() + WordlistFilteredOut(); got != 7776 {
t.Fatalf("filtered + removed should reconstitute the 7776-word EFF list, got %d", got)
}
}
@@ -25,19 +33,27 @@ func TestGenerateRecoveryCode_EntropyAndFormat(t *testing.T) {
inList[w] = true
}
for i := 0; i < 50; i++ {
r, err := GenerateRecoveryCode()
// Count words by GENERATION count, not by re-splitting the joined string: the two agree
// only because joinSafe holds, and conflating them is what made this test flake ~1/5.
words, err := generateWords(wordlist)
if err != nil {
t.Fatalf("GenerateRecoveryCode: %v", err)
t.Fatalf("generateWords: %v", err)
}
words := strings.Split(r, "-")
if len(words) != RecoveryCodeWords {
t.Fatalf("recovery code must be %d words, got %d (%q)", RecoveryCodeWords, len(words), r)
t.Fatalf("generator must draw %d words, drew %d", RecoveryCodeWords, len(words))
}
for _, w := range words {
if !inList[w] {
t.Errorf("recovery-code word %q is not from the EFF wordlist", w)
}
}
// Separately assert the property joinSafe buys: the joined code segments back to the same
// count. Never print r — it is a live-shaped secret.
r := strings.Join(words, RecoveryCodeSep)
if got := len(strings.Split(r, RecoveryCodeSep)); got != RecoveryCodeWords {
t.Fatalf("joined code must segment into %d words, got %d (a drawn word contained %q)",
RecoveryCodeWords, got, RecoveryCodeSep)
}
}
}
+82
View File
@@ -2,10 +2,14 @@ package escrow
import (
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
)
// Slice 10D.1 — IDENTITY escrow. The K-escrow (above) wraps the PBS *encryption key* via the
@@ -24,6 +28,84 @@ var ageBinary = "/usr/bin/age"
type IdentityBundle struct {
TunnelToken string `json:"tunnel_token"` // the Cloudflare tunnel connector token
PBSToken string `json:"pbs_token"` // the PBS access token (steady-state; rotated on re-establish)
// WGPrivateKey is the offsite WG tunnel private key (S3; base64, 32 bytes). OPTIONAL: escrow
// blobs created before S3 lack it and CANNOT be retro-fitted (R is never retained) — S5 DR
// falls back to fresh-key re-registration, which keeps the box's /32 (hub S2 re-key-in-place).
WGPrivateKey string `json:"wg_private_key,omitempty"`
// ResticRepoPassword is the offsite restic repo password (fork-4). OPTIONAL: escrow blobs created
// before fork-4 lack it and CANNOT be retro-fitted (R is never retained). It is the DATA key for the
// offsite tier — irreplaceable (unlike the SFTP access key, which is regenerable at DR). The
// controller's atomicity gate ensures no offsite ciphertext exists until this is escrowed.
ResticRepoPassword string `json:"restic_repo_password,omitempty"`
}
// StagedResticPasswordPath is the well-known 0600 file where the controller-pushed restic repo password
// is transiently staged (by the local API) for the escrow-create ceremony to pick up, then wiped. A fixed
// path so the local-API writer and the CLI ceremony reader agree without threading config through.
func StagedResticPasswordPath() string {
return filepath.Join("/var/lib/felhom-agent", "escrow-stage", "restic_repo_password")
}
// WipeStagedResticPassword removes the staged restic password (called by the ceremony after a successful
// escrow-create — the secret now lives only inside the R-wrapped blob). A missing file is a clean no-op.
func WipeStagedResticPassword() error {
if err := os.Remove(StagedResticPasswordPath()); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("escrow: wipe staged restic password: %w", err)
}
return nil
}
// HashResticPassword is the CANONICAL hasher for the offsite restic repo password (SLICE 3 hub-verified
// escrow auto-confirm): sha256 hex of the TRIMMED password string — exactly the value AttachResticPassword
// seals into the blob and the value the controller uses (both sides TrimSpace their file reads, so the
// trimmed string is the drift-free convention; pinned by the SAME test vector in felhom-agent and
// felhom-controller). The hash of a 256-bit random secret is non-reversible and non-brute-forceable —
// safe to store on the hub and serve in report ACKs; the PASSWORD itself is never logged or served.
func HashResticPassword(pw string) string {
sum := sha256.Sum256([]byte(strings.TrimSpace(pw)))
return hex.EncodeToString(sum[:])
}
// AttachResticPassword injects the offsite restic repo password from the staged 0600 file into the bundle
// when it exists (fork-4 escrow-create auto-inject). Returns whether it attached. The VALUE is validated
// (non-empty) but NEVER logged by callers — log the field NAME only (mirrors AttachWGKey). A missing file
// is a clean no-attach (pre-fork-4 behavior, byte-compatible bundle).
func AttachResticPassword(b *IdentityBundle, stagePath string) (bool, error) {
raw, err := os.ReadFile(stagePath)
if err != nil {
if os.IsNotExist(err) {
return false, nil
}
return false, fmt.Errorf("escrow: reading staged restic password: %w", err)
}
pw := strings.TrimSpace(string(raw))
if pw == "" {
return false, fmt.Errorf("escrow: staged restic password file %s is empty", stagePath)
}
b.ResticRepoPassword = pw
return true, nil
}
// AttachWGKey injects the offsite WG private key into the bundle when the key file exists (S3
// escrow-create auto-inject). Returns whether it attached. The VALUE is validated (base64, 32
// bytes) but never logged by callers — log the field NAME only. A missing key file is a clean
// no-attach (pre-S3 behavior, byte-compatible bundle); a corrupt one is an error (the operator
// should know their escrow would silently lack a live identity).
func AttachWGKey(b *IdentityBundle, keyPath string) (bool, error) {
raw, err := os.ReadFile(keyPath)
if err != nil {
if os.IsNotExist(err) {
return false, nil
}
return false, fmt.Errorf("escrow: reading wg key file: %w", err)
}
s := strings.TrimSpace(string(raw))
dec, err := base64.StdEncoding.DecodeString(s)
if err != nil || len(dec) != 32 {
return false, fmt.Errorf("escrow: wg key file %s is corrupt (not 32-byte base64)", keyPath)
}
b.WGPrivateKey = s
return true, nil
}
// WrapIdentity wraps arbitrary bundle bytes under `R` via `age -p` (scrypt + ChaCha20-Poly1305) and
+72
View File
@@ -3,7 +3,9 @@ package escrow
import (
"bytes"
"context"
"os"
"os/exec"
"path/filepath"
"runtime"
"testing"
)
@@ -65,6 +67,76 @@ func TestIdentity_RoundTrip(t *testing.T) {
}
}
// fork-4: the IdentityBundle carries the offsite restic repo password under R, byte-exact and encrypted;
// a wrong R fails closed. (The spike proved a recovered value opens the real repo; this guards the field.)
func TestIdentity_RoundTrip_CarriesResticPassword(t *testing.T) {
ensureAge(t)
ctx := context.Background()
const R = "throwaway-correct-horse-battery-staple-fork4"
const pw = "deadbeefcafef00d0123456789abcdef0123456789abcdef0123456789abcdef" // 64 hex, synthetic
bundle := IdentityBundle{TunnelToken: "tt", PBSToken: "pt", ResticRepoPassword: pw}
blob, err := WrapIdentityBundle(ctx, bundle, R)
if err != nil {
t.Fatalf("WrapIdentityBundle: %v", err)
}
if bytes.Contains(blob, []byte(pw)) {
t.Fatal("the blob leaks the restic password plaintext — not encrypted")
}
got, err := UnwrapIdentityBundle(ctx, blob, R)
if err != nil {
t.Fatalf("UnwrapIdentityBundle: %v", err)
}
if got.ResticRepoPassword != pw {
t.Fatalf("recovered restic password not byte-exact: got %q", got.ResticRepoPassword)
}
if got != bundle {
t.Fatalf("recovered bundle = %+v, want %+v", got, bundle)
}
if _, err := UnwrapIdentityBundle(ctx, blob, R+"-WRONG"); err == nil {
t.Fatal("a wrong recovery code must fail closed (no bundle, no restic password)")
}
}
// PINNED CROSS-REPO TEST VECTOR (SLICE 3): the same vector is asserted in felhom-controller — if either
// side drifts (trailing newline, encoding, trim behavior), its half of this test fails and auto-confirm
// can never silently mismatch. Convention: sha256 hex over the TRIMMED password string.
func TestHashResticPassword_PinnedVector(t *testing.T) {
const vector = "cafef00ddeadbeef0123456789abcdef0123456789abcdef0123456789abcdef"
const want = "dbfc02f987e1ac0c91911d5761267089b1144628745a3343e4d96194e43c08e4"
if got := HashResticPassword(vector); got != want {
t.Fatalf("pinned vector drift: got %s want %s", got, want)
}
// trim convention: surrounding whitespace/newlines do not change the hash (both sides trim)
if got := HashResticPassword(" " + vector + "\n"); got != want {
t.Fatalf("whitespace must not change the hash (trim convention), got %s", got)
}
}
// AttachResticPassword: missing file → clean no-attach; staged file → trimmed value attached; empty → error.
func TestAttachResticPassword(t *testing.T) {
b := &IdentityBundle{}
if ok, err := AttachResticPassword(b, filepath.Join(t.TempDir(), "absent")); ok || err != nil {
t.Fatalf("missing staged file must be a clean no-attach, got ok=%v err=%v", ok, err)
}
f := filepath.Join(t.TempDir(), "pw")
if err := os.WriteFile(f, []byte(" abc123def \n"), 0o600); err != nil {
t.Fatal(err)
}
ok, err := AttachResticPassword(b, f)
if err != nil || !ok {
t.Fatalf("attach from staged file: ok=%v err=%v", ok, err)
}
if b.ResticRepoPassword != "abc123def" {
t.Fatalf("want trimmed value, got %q", b.ResticRepoPassword)
}
if err := os.WriteFile(f, []byte(" \n"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := AttachResticPassword(&IdentityBundle{}, f); err == nil {
t.Fatal("an empty staged file must error (an operator would want to know)")
}
}
// Wrong R fails CLOSED — no bundle emitted.
func TestIdentity_WrongRFailsClosed(t *testing.T) {
ensureAge(t)
+51
View File
@@ -0,0 +1,51 @@
package escrow
// S3 Group D — the WG-key escrow join. Red-proof (e): remove the auto-inject call and the
// bundle-contains-key assertion fails.
import (
"encoding/base64"
"encoding/json"
"os"
"path/filepath"
"testing"
)
func TestAttachWGKey(t *testing.T) {
dir := t.TempDir()
keyPath := filepath.Join(dir, "private.key")
// Missing key file → clean no-attach (pre-S3 bundles stay byte-compatible).
b := &IdentityBundle{TunnelToken: "tt", PBSToken: "pt"}
attached, err := AttachWGKey(b, keyPath)
if err != nil || attached {
t.Fatalf("missing file: attached=%v err=%v", attached, err)
}
raw, _ := json.Marshal(b)
if string(raw) != `{"tunnel_token":"tt","pbs_token":"pt"}` {
t.Fatalf("bundle without key marshals with extra fields: %s", raw)
}
// Present key file → attached, field carried.
key := base64.StdEncoding.EncodeToString(make([]byte, 32))
os.WriteFile(keyPath, []byte(key+"\n"), 0o600)
attached, err = AttachWGKey(b, keyPath)
if err != nil || !attached {
t.Fatalf("present file: attached=%v err=%v", attached, err)
}
if b.WGPrivateKey != key {
t.Fatalf("bundle key = %q", b.WGPrivateKey)
}
raw, _ = json.Marshal(b)
var back IdentityBundle
json.Unmarshal(raw, &back)
if back.WGPrivateKey != key {
t.Fatal("wg_private_key does not survive the bundle round-trip")
}
// Corrupt key file → error (the operator must know their escrow would lack the identity).
os.WriteFile(keyPath, []byte("garbage"), 0o600)
if _, err := AttachWGKey(&IdentityBundle{}, keyPath); err == nil {
t.Fatal("corrupt key file attached silently")
}
}
+58 -17
View File
@@ -29,9 +29,20 @@ import (
//go:embed eff_large_wordlist.txt
var wordlistRaw []byte
// wordlist is the EFF large wordlist (7776 words, 12.92 bits/word) — the diceware standard for
// human-transcribed passphrases. Parsed once at init.
var wordlist = parseWordlist(wordlistRaw)
// RecoveryCodeSep joins the words of a recovery code R. It is ALSO the reason for the
// joinSafe filter below: a word that itself contains the separator makes the joined code
// ambiguous to segment by eye, which is unaffordable in the one situation R exists for — a
// customer transcribing it during a disaster. Do not change it: R is consumed as a whole
// passphrase (see Wrap/Unwrap), so the separator is a transcription aid, not a parsed delimiter.
const RecoveryCodeSep = "-"
// wordlist is the EFF large wordlist (the diceware standard for human-transcribed passphrases),
// minus the handful of entries that contain RecoveryCodeSep. Parsed and filtered once at init.
// Sizes are asserted in wordlist_test.go so a wordlist swap cannot silently move the entropy floor.
var wordlist = joinSafe(parseWordlist(wordlistRaw))
// wordlistRawSize is the unfiltered parse length, kept for audit (see WordlistFilteredOut).
var wordlistRawSize = len(parseWordlist(wordlistRaw))
func parseWordlist(raw []byte) []string {
var w []string
@@ -44,28 +55,55 @@ func parseWordlist(raw []byte) []string {
return w
}
// RecoveryCodeWords is the number of words in a recovery code R. 10 words from the 7776-word EFF
// list ≈ 129.2 bits (≥128) — the chosen entropy floor (slice7-escrow-spike-findings.md §4).
// joinSafe drops every word containing RecoveryCodeSep, so that a generated code always segments
// back into exactly RecoveryCodeWords words. In the EFF large list this removes exactly 4 entries
// (drop-down, felt-tip, t-shirt, yo-yo) of 7776, costing ~0.0007 bits/word — the floor still holds
// (asserted in the tests). Generation-time only: codes already issued remain valid, because R is
// verified as a whole passphrase and is never re-split.
func joinSafe(words []string) []string {
out := make([]string, 0, len(words))
for _, w := range words {
if strings.Contains(w, RecoveryCodeSep) {
continue
}
out = append(out, w)
}
return out
}
// RecoveryCodeWords is the number of words in a recovery code R. 10 words from the filtered EFF
// list (7772 words) ≈ 129.2 bits (≥128) — the chosen entropy floor (slice7-escrow-spike-findings.md §4).
const RecoveryCodeWords = 10
// GenerateRecoveryCode returns a fresh recovery code R: RecoveryCodeWords words chosen uniformly
// (crypto/rand via big.Int — no modulo bias) from the EFF large wordlist, hyphen-joined.
//
// SECRET: the returned string is R. Surface it to the customer exactly once; never log or persist it.
func GenerateRecoveryCode() (string, error) {
if len(wordlist) < 2 {
return "", fmt.Errorf("escrow: wordlist not loaded (%d words)", len(wordlist))
// generateWords draws RecoveryCodeWords words uniformly (crypto/rand via big.Int — no modulo bias)
// from list. Split out from GenerateRecoveryCode so tests can drive an unfiltered list and prove
// the filter is what keeps a code segmentable.
func generateWords(list []string) ([]string, error) {
if len(list) < 2 {
return nil, fmt.Errorf("escrow: wordlist not loaded (%d words)", len(list))
}
n := big.NewInt(int64(len(wordlist)))
n := big.NewInt(int64(len(list)))
words := make([]string, RecoveryCodeWords)
for i := range words {
idx, err := rand.Int(rand.Reader, n)
if err != nil {
return "", fmt.Errorf("escrow: recovery-code rng: %w", err)
return nil, fmt.Errorf("escrow: recovery-code rng: %w", err)
}
words[i] = wordlist[idx.Int64()]
words[i] = list[idx.Int64()]
}
return strings.Join(words, "-"), nil
return words, nil
}
// GenerateRecoveryCode returns a fresh recovery code R: RecoveryCodeWords words chosen uniformly
// from the filtered EFF large wordlist, joined with RecoveryCodeSep.
//
// SECRET: the returned string is R. Surface it to the customer exactly once; never log or persist it.
func GenerateRecoveryCode() (string, error) {
words, err := generateWords(wordlist)
if err != nil {
return "", err
}
return strings.Join(words, RecoveryCodeSep), nil
}
// RecoveryCodeEntropyBits is the approximate entropy of a generated code, for display/audit only
@@ -77,5 +115,8 @@ func RecoveryCodeEntropyBits() float64 {
return float64(RecoveryCodeWords) * math.Log2(float64(len(wordlist)))
}
// WordlistSize is the loaded wordlist length (for audit/tests).
// WordlistSize is the effective (filtered) wordlist length — the draw space. For audit/tests.
func WordlistSize() int { return len(wordlist) }
// WordlistFilteredOut is how many parsed entries joinSafe removed. For audit/tests.
func WordlistFilteredOut() int { return wordlistRawSize - len(wordlist) }
+126
View File
@@ -0,0 +1,126 @@
package escrow
import (
"math"
"strings"
"testing"
)
// The four EFF large-list entries that contain RecoveryCodeSep. Named here so a wordlist swap that
// changes the set fails loudly rather than silently re-opening the ambiguity.
var hyphenatedEFFWords = []string{"drop-down", "felt-tip", "t-shirt", "yo-yo"}
func TestJoinSafe_RemovesExactlyTheHyphenatedEFFWords(t *testing.T) {
raw := parseWordlist(wordlistRaw)
rawSet := make(map[string]bool, len(raw))
for _, w := range raw {
rawSet[w] = true
}
for _, w := range hyphenatedEFFWords {
if !rawSet[w] {
t.Fatalf("fixture drift: %q is no longer in the embedded EFF list", w)
}
}
filtered := joinSafe(raw)
if len(raw)-len(filtered) != len(hyphenatedEFFWords) {
t.Fatalf("joinSafe removed %d entries, expected exactly %d",
len(raw)-len(filtered), len(hyphenatedEFFWords))
}
got := make(map[string]bool, len(filtered))
for _, w := range filtered {
if strings.Contains(w, RecoveryCodeSep) {
t.Errorf("filtered wordlist still contains a separator-bearing word %q", w)
}
got[w] = true
}
for _, w := range hyphenatedEFFWords {
if got[w] {
t.Errorf("joinSafe kept %q, which contains %q", w, RecoveryCodeSep)
}
}
}
// TestEntropyFloorSurvivesFiltering states the numbers explicitly: dropping 4 of 7776 words costs
// ~0.0007 bits/word, so the 10-word code stays above the 128-bit floor with room to spare.
func TestEntropyFloorSurvivesFiltering(t *testing.T) {
const floorBits = 128.0
before := float64(RecoveryCodeWords) * math.Log2(7776)
after := RecoveryCodeEntropyBits()
if after < floorBits {
t.Fatalf("filtered entropy %.3f bits is below the %.0f-bit floor", after, floorBits)
}
if want := float64(RecoveryCodeWords) * math.Log2(float64(WordlistSize())); math.Abs(after-want) > 1e-9 {
t.Fatalf("RecoveryCodeEntropyBits() = %.6f, want %.6f (10 * log2(%d))", after, want, WordlistSize())
}
// Concrete expectations, so a wordlist change that quietly erodes the margin is visible:
// 10*log2(7776) = 129.248 bits before, 10*log2(7772) = 129.241 bits after — a 0.007-bit cost.
if math.Abs(before-129.248) > 0.001 {
t.Fatalf("unfiltered entropy baseline moved: %.3f, expected 129.248", before)
}
if math.Abs(after-129.241) > 0.001 {
t.Fatalf("filtered entropy moved: %.3f, expected 129.241", after)
}
if cost := before - after; cost > 0.01 {
t.Fatalf("filtering cost %.4f bits, expected well under 0.01", cost)
}
}
// TestGeneratedCodeSegments_FilteredVsUnfiltered is the deterministic red-proof companion.
//
// Against a list where EVERY word contains the separator, a 10-word draw MUST segment into more
// than 10 parts — that is the pre-fix behaviour, reproduced with probability 1 instead of the ~1/5
// flake the real list produced. Against the same list run through joinSafe, generation must refuse
// (nothing is left to draw from), proving joinSafe — not luck — is what makes a code segmentable.
func TestGeneratedCodeSegments_FilteredVsUnfiltered(t *testing.T) {
unfiltered := hyphenatedEFFWords
words, err := generateWords(unfiltered)
if err != nil {
t.Fatalf("generateWords(unfiltered): %v", err)
}
if len(words) != RecoveryCodeWords {
t.Fatalf("generator drew %d words, want %d", len(words), RecoveryCodeWords)
}
joined := strings.Join(words, RecoveryCodeSep)
segs := len(strings.Split(joined, RecoveryCodeSep))
if segs <= RecoveryCodeWords {
t.Fatalf("unfiltered draw segmented into %d parts; the pre-fix defect should yield more than %d",
segs, RecoveryCodeWords)
}
if segs != 2*RecoveryCodeWords {
t.Fatalf("every fixture word has exactly one separator, so 10 words must segment into 20 parts, got %d", segs)
}
// Same fixture, filtered: the draw space is empty, so generation must error rather than
// silently fall back to something ambiguous.
if _, err := generateWords(joinSafe(unfiltered)); err == nil {
t.Fatal("generateWords on a fully-filtered list must fail, not return a code")
}
}
// TestGenerateRecoveryCode_NeverContainsAmbiguousWord is the production-wiring test: it asserts the
// exported entry point (not just the helper) draws from the filtered list.
func TestGenerateRecoveryCode_NeverContainsAmbiguousWord(t *testing.T) {
inFiltered := make(map[string]bool, len(wordlist))
for _, w := range wordlist {
inFiltered[w] = true
}
for i := 0; i < 500; i++ {
r, err := GenerateRecoveryCode()
if err != nil {
t.Fatalf("GenerateRecoveryCode: %v", err)
}
parts := strings.Split(r, RecoveryCodeSep)
if len(parts) != RecoveryCodeWords {
// Do not print r: it is a live-shaped secret.
t.Fatalf("code %d segmented into %d parts, want %d", i, len(parts), RecoveryCodeWords)
}
for _, p := range parts {
if !inFiltered[p] {
t.Fatalf("segment %q is not a filtered-wordlist word", p)
}
}
}
}
+105
View File
@@ -0,0 +1,105 @@
// Package fasttick is the agent-plane immediacy SECONDARY (v0.90.0, R-28). While ANY desired-state
// item is still unapplied — most importantly the pre-tunnel WG-registration window where a hub poke
// is undeliverable by construction — it pulses the hub control loop's out-of-band report trigger on
// a fast (30 s) cadence, and self-disarms EMERGENTLY the instant everything converges. It is the
// state-based complement to the poke: the poke handles hub→box changes once the tunnel exists; the
// fast-tick handles the window before that (and any lingering unapplied drift) from the box side.
//
// By ruling it is STATE-BASED, not a fixed burst and not a timer: there is nothing to journal
// (stateless across restarts) and nothing to leak. A perma-unconverged box fast-ticks at ~2 small
// reports/min, bounded and visible; the LOUD pbsdr states (consumed_failed/verify_failed) are
// deliberately EXCLUDED from the sources so a stuck-loud box does not hammer (§8).
//
// It pulses the SAME cap-1 channel the storage watchdog and the poke listener use, so a pulse
// coalesces with a poke/watchdog nudge for free — no extra debounce here.
package fasttick
import (
"context"
"log/slog"
"time"
)
// DefaultInterval is the ruled fast cadence while unconverged.
const DefaultInterval = 30 * time.Second
// Source reports whether one subsystem still has unapplied desired-state. Implementations MUST be a
// cheap, CACHED read — no exec, no network per call (the fast-tick calls every source each tick).
type Source interface {
Unconverged() (unconverged bool, reason string)
}
// SourceFunc adapts a plain func to a Source (main.go closes over each subsystem).
type SourceFunc func() (bool, string)
// Unconverged implements Source.
func (f SourceFunc) Unconverged() (bool, string) { return f() }
// Loop evaluates the sources on a ticker and pulses the out-of-band channel while any is unconverged.
type Loop struct {
sources []Source
out chan<- struct{}
interval time.Duration
logger *slog.Logger
armed bool // for armed↔disarmed transition logging (avoids 30 s reason spam)
}
// New builds a fast-tick loop. out is the hub loop's out-of-band trigger channel (cap-1). A
// non-positive interval falls back to DefaultInterval.
func New(out chan<- struct{}, interval time.Duration, logger *slog.Logger, sources ...Source) *Loop {
if interval <= 0 {
interval = DefaultInterval
}
if logger == nil {
logger = slog.Default()
}
return &Loop{sources: sources, out: out, interval: interval, logger: logger}
}
// Run evaluates the sources every interval until ctx is cancelled. Stateless — nothing to recover.
func (l *Loop) Run(ctx context.Context) error {
l.logger.Info("fast-tick armed: "+l.interval.String()+" out-of-band cadence while desired-state is unapplied", "interval", l.interval)
ticker := time.NewTicker(l.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
l.logger.Info("fast-tick: shutting down", "reason", ctx.Err())
return nil
case <-ticker.C:
l.step()
}
}
}
// step evaluates the sources once. If any is unconverged it pulses the channel (non-blocking: a full
// channel means an out-of-band report is already pending, so the pulse coalesces harmlessly) and, on
// the disarmed→armed edge, logs the reason. When all converge it logs the armed→disarmed edge once.
// Returns whether this tick found the box unconverged (test hook). No per-tick logging when steady.
func (l *Loop) step() bool {
unconverged, reason := l.evaluate()
if unconverged {
select {
case l.out <- struct{}{}:
default: // an out-of-band report is already queued — coalesce, never block
}
if !l.armed {
l.logger.Info("fast-tick: desired-state unapplied — pulsing out-of-band reports", "reason", reason, "cadence", l.interval)
l.armed = true
}
} else if l.armed {
l.logger.Info("fast-tick: desired-state converged — back to the normal cadence")
l.armed = false
}
return unconverged
}
// evaluate returns the first unconverged source's reason (order = priority for the log line).
func (l *Loop) evaluate() (bool, string) {
for _, s := range l.sources {
if u, reason := s.Unconverged(); u {
return true, reason
}
}
return false, ""
}
+121
View File
@@ -0,0 +1,121 @@
package fasttick
import (
"io"
"log/slog"
"testing"
"time"
)
func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
// flagSource is a fake Source whose convergence the test flips at will.
type flagSource struct {
unconverged bool
reason string
}
func (f *flagSource) Unconverged() (bool, string) { return f.unconverged, f.reason }
// drain reports how many pulses are queued (channel is cap-1 in production; tests may use larger).
func drain(ch chan struct{}) int {
n := 0
for {
select {
case <-ch:
n++
default:
return n
}
}
}
// D1 — any source unconverged → one pulse per tick (drained between ticks).
func TestFastTick_UnconvergedPulses(t *testing.T) {
out := make(chan struct{}, 1)
src := &flagSource{unconverged: true, reason: "test drift"}
l := New(out, time.Hour, quiet(), src)
for i := 0; i < 3; i++ {
if !l.step() {
t.Fatalf("tick %d: step reported converged, want unconverged", i)
}
if got := drain(out); got != 1 {
t.Fatalf("tick %d: pulses = %d, want 1", i, got)
}
}
}
// D2 — ALL sources converged → zero pulses across N ticks (the emergent disarm).
func TestFastTick_ConvergedSilent(t *testing.T) {
out := make(chan struct{}, 4)
l := New(out, time.Hour, quiet(), &flagSource{unconverged: false}, &flagSource{unconverged: false})
for i := 0; i < 5; i++ {
if l.step() {
t.Fatalf("tick %d: step reported unconverged with all sources converged", i)
}
}
if got := drain(out); got != 0 {
t.Fatalf("pulses on a fully-converged box = %d, want 0", got)
}
}
// D3 (the ruled red-proof) — flip unconverged→converged mid-run → pulses STOP from the next tick.
func TestFastTick_ConvergenceDisarms(t *testing.T) {
out := make(chan struct{}, 8)
src := &flagSource{unconverged: true, reason: "drift"}
l := New(out, time.Hour, quiet(), src)
// Two unconverged ticks pulse.
l.step()
l.step()
if got := drain(out); got != 2 {
t.Fatalf("pre-convergence pulses = %d, want 2", got)
}
// Converge.
src.unconverged = false
// Every subsequent tick is silent — the cadence returns to normal.
for i := 0; i < 4; i++ {
if l.step() {
t.Fatalf("post-convergence tick %d still unconverged", i)
}
}
if got := drain(out); got != 0 {
t.Fatalf("pulses fired after convergence = %d, want 0 (disarm failed)", got)
}
}
// D4 — the out channel is already full (a poke just landed): the non-blocking send drops, no block,
// no goroutine leak, no queue growth.
func TestFastTick_ChannelFullDrops(t *testing.T) {
out := make(chan struct{}, 1)
out <- struct{}{} // pre-fill: an out-of-band report is already pending
l := New(out, time.Hour, quiet(), &flagSource{unconverged: true, reason: "drift"})
done := make(chan bool, 1)
go func() {
l.step() // must NOT block on the full channel
l.step()
done <- true
}()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("step blocked on a full channel (non-blocking send violated)")
}
if len(out) != 1 {
t.Fatalf("channel depth = %d, want 1 (coalesced, no queue growth)", len(out))
}
}
// Priority ordering: the first unconverged source supplies the reason.
func TestFastTick_FirstReasonWins(t *testing.T) {
out := make(chan struct{}, 1)
l := New(out, time.Hour, quiet(),
&flagSource{unconverged: false},
&flagSource{unconverged: true, reason: "second"},
&flagSource{unconverged: true, reason: "third"})
if u, r := l.evaluate(); !u || r != "second" {
t.Fatalf("evaluate = (%v, %q), want (true, second)", u, r)
}
}
+133
View File
@@ -0,0 +1,133 @@
package felhomsshd
import (
"context"
"log/slog"
"net/netip"
"regexp"
"sort"
"strconv"
"strings"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// Belt keeps the STATIC nft table `inet felhom_oob` (installed by host-install) converged on the
// current OOB state by mutating only its SETS — @operator_ips (the operator /32 allowed to reach
// felhom-sshd over wg-felhom) and @ssh_port (felhom-sshd's claimed port). The agent NEVER adds/removes
// RULES [trap 4]: set-element mutation can't change rule semantics, so the narrow sudoers grant stays
// safe. Idempotent: reads the current elements and mutates only on a difference (no churn).
type Belt struct {
runner proxmox.Runner
logger *slog.Logger
table string // "felhom_oob"
}
// NewBelt builds a Belt over the narrow runner. table defaults to "felhom_oob".
func NewBelt(runner proxmox.Runner, logger *slog.Logger) *Belt {
if logger == nil {
logger = slog.Default()
}
return &Belt{runner: runner, logger: logger, table: "felhom_oob"}
}
// SyncPort converges @ssh_port on the claimed port (0 empties it). Always safe to call — the port
// comes from the agent's own claim, not the hub desired-state.
func (b *Belt) SyncPort(ctx context.Context, port int) {
wantPorts := []string{}
if port > 0 && port <= 65535 {
wantPorts = []string{strconv.Itoa(port)}
}
b.syncSet(ctx, "ssh_port", wantPorts)
}
// SyncOperator converges @operator_ips on the operator /32 (operatorIP "" = OOB explicitly off →
// empty the set). Call ONLY when the desired-state has actually been FETCHED — a nil/unfetched block
// must NOT empty the set (that would lock the operator out until the next fetch, the wgtunnel
// fetched=false-is-never-a-teardown rule). Every value is netip-validated before it reaches nft.
func (b *Belt) SyncOperator(ctx context.Context, operatorIP string) {
wantOps := []string{}
if operatorIP != "" {
ip, err := netip.ParseAddr(operatorIP)
if err != nil || !ip.Is4() {
b.logger.Error("felhomsshd belt: operator ip invalid — refusing", "ip", operatorIP)
return
}
wantOps = []string{ip.String()}
}
b.syncSet(ctx, "operator_ips", wantOps)
}
var nftElemRe = regexp.MustCompile(`elements\s*=\s*\{([^}]*)\}`)
// syncSet converges one named set on `want` (sorted, deduped). Reads current elements; if they match,
// ZERO nft mutations (the idempotency the scenario asserts). Otherwise flush + add the desired
// elements. A read failure (table/set absent) → one warning, no mutation.
func (b *Belt) syncSet(ctx context.Context, setName string, want []string) {
want = sortedUnique(want)
cur, ok := b.readSet(ctx, setName)
if !ok {
b.logger.Warn("felhomsshd belt: set unreadable (table not installed?) — skipping", "set", setName)
return
}
if equalStringSlices(cur, want) {
return // steady state: no mutation
}
if _, errOut, err := b.runner.Run(ctx, "nft", "flush", "set", "inet", b.table, setName); err != nil {
b.logger.Error("felhomsshd belt: flush failed", "set", setName, "err", err, "stderr", strings.TrimSpace(string(errOut)))
return
}
for _, e := range want {
if _, errOut, err := b.runner.Run(ctx, "nft", "add", "element", "inet", b.table, setName, "{ "+e+" }"); err != nil {
b.logger.Error("felhomsshd belt: add element failed", "set", setName, "elem", e, "err", err, "stderr", strings.TrimSpace(string(errOut)))
return
}
}
b.logger.Info("felhomsshd belt: set synced", "set", setName, "elements", strings.Join(want, ","))
}
// readSet returns the current elements of a named set (sorted), or ok=false if the set can't be read.
func (b *Belt) readSet(ctx context.Context, setName string) ([]string, bool) {
out, _, err := b.runner.Run(ctx, "nft", "list", "set", "inet", b.table, setName)
if err != nil {
return nil, false
}
m := nftElemRe.FindSubmatch(out)
if m == nil {
return []string{}, true // set exists but empty (no "elements = {}" block)
}
var elems []string
for _, f := range strings.Split(string(m[1]), ",") {
f = strings.TrimSpace(f)
if f != "" {
elems = append(elems, f)
}
}
return sortedUnique(elems), true
}
func sortedUnique(in []string) []string {
seen := map[string]bool{}
out := make([]string, 0, len(in))
for _, s := range in {
if !seen[s] {
seen[s] = true
out = append(out, s)
}
}
sort.Strings(out)
return out
}
func equalStringSlices(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
+179
View File
@@ -0,0 +1,179 @@
package felhomsshd
import (
"context"
"io"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// scriptRunner records every exec and returns scripted stdout per "name+first-arg" key; scripted
// errors per the same key. Everything else returns empty success.
type scriptRunner struct {
mu sync.Mutex
calls [][]string
stdout map[string]string
errs map[string]error
}
func newScriptRunner() *scriptRunner {
return &scriptRunner{stdout: map[string]string{}, errs: map[string]error{}}
}
func key(name string, args ...string) string {
// `nft list set inet felhom_oob <setName>` → key by the set name (last arg), so the two set reads
// are distinguishable. Everything else keys by first arg.
if name == "nft" && len(args) > 0 && args[0] == "list" {
return "nft list " + args[len(args)-1]
}
if len(args) > 0 {
return name + " " + args[0]
}
return name
}
func (r *scriptRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
r.calls = append(r.calls, append([]string{name}, args...))
k := key(name, args...)
out := r.stdout[k]
err := r.errs[k]
r.mu.Unlock()
if err != nil {
return nil, []byte("scripted failure"), err
}
return []byte(out), nil, nil
}
func (r *scriptRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func (r *scriptRunner) count(name, firstArg string) int {
r.mu.Lock()
defer r.mu.Unlock()
n := 0
for _, c := range r.calls {
if c[0] == name && len(c) > 1 && c[1] == firstArg {
n++
}
}
return n
}
func (r *scriptRunner) sawRuleMutation() bool {
r.mu.Lock()
defer r.mu.Unlock()
for _, c := range r.calls {
// trap 4: the agent must NEVER run `nft add rule` / `nft -f` / `nft flush ruleset|table`.
if c[0] == "nft" && len(c) > 1 {
if c[1] == "-f" || (len(c) > 2 && c[1] == "add" && c[2] == "rule") {
return true
}
if c[1] == "flush" && len(c) > 2 && (c[2] == "ruleset" || c[2] == "table") {
return true
}
}
}
return false
}
func TestBelt_SyncMutatesThenIdempotent(t *testing.T) {
r := newScriptRunner()
// first read: both sets empty (no "elements" block)
r.stdout["nft list operator_ips"] = "set operator_ips {\n\ttype ipv4_addr\n}"
r.stdout["nft list ssh_port"] = "set ssh_port {\n\ttype inet_service\n}"
b := NewBelt(r, nil)
b.SyncPort(context.Background(), 8822)
b.SyncOperator(context.Background(), "10.77.0.250")
// mutations happened: flush + add for BOTH sets
if r.count("nft", "flush") < 2 || r.count("nft", "add") < 2 {
t.Fatalf("first sync must flush+add both sets; flushes=%d adds=%d", r.count("nft", "flush"), r.count("nft", "add"))
}
if r.sawRuleMutation() {
t.Fatal("belt must NEVER mutate rules — only set elements (trap 4)")
}
// second sync: the reads now return the desired elements → ZERO mutations (idempotent)
r2 := newScriptRunner()
r2.stdout["nft list operator_ips"] = "elements = { 10.77.0.250 }"
r2.stdout["nft list ssh_port"] = "elements = { 8822 }"
b2 := NewBelt(r2, nil)
b2.SyncPort(context.Background(), 8822)
b2.SyncOperator(context.Background(), "10.77.0.250")
if r2.count("nft", "flush") != 0 || r2.count("nft", "add") != 0 {
t.Fatalf("idempotent sync must do ZERO mutations; flushes=%d adds=%d", r2.count("nft", "flush"), r2.count("nft", "add"))
}
}
func TestBelt_OperatorEmptyEmptiesSet(t *testing.T) {
r := newScriptRunner()
r.stdout["nft list operator_ips"] = "elements = { 10.77.0.250 }" // currently has an operator IP
r.stdout["nft list ssh_port"] = "elements = { 8822 }"
b := NewBelt(r, nil)
b.SyncOperator(context.Background(), "") // OOB explicitly off → operator set must be emptied
if r.count("nft", "flush") < 1 {
t.Fatal("emptying the operator set must flush it")
}
}
// health: inactive + INVALID config → NO restart (the red-proof); inactive + valid → restart.
func TestHeal_NoRestartOnInvalidConfig(t *testing.T) {
r := newScriptRunner()
r.errs["sshd -t"] = context.DeadlineExceeded // sshd -t FAILS (config invalid)
m := newTestManager(r, false) // inactive
m.HealAndCheck(context.Background(), 8822)
if r.count("systemctl", "restart") != 0 {
t.Fatal("a broken config must NOT trigger a restart (never degraded→dead)")
}
}
func TestHeal_RestartsWhenDownWithValidConfigThenCooldown(t *testing.T) {
r := newScriptRunner() // sshd -t passes (no error)
m := newTestManager(r, false)
m.HealAndCheck(context.Background(), 8822)
if r.count("systemctl", "restart") != 1 {
t.Fatalf("down + valid config → exactly one restart, got %d", r.count("systemctl", "restart"))
}
// cooldown: a second call inside the window → still one restart
m.HealAndCheck(context.Background(), 8822)
if r.count("systemctl", "restart") != 1 {
t.Fatalf("restart cooldown violated: %d restarts", r.count("systemctl", "restart"))
}
}
func TestStatus_ReflectsBlockAndPort(t *testing.T) {
r := newScriptRunner()
r.stdout["sshd -T"] = "port 8822\nsomethingelse yes\n"
m := newTestManager(r, true) // active
m.port = 8822
st := m.Status(context.Background(), &hub.WireWireguard{OOBPeerIP: "10.77.0.250", OOBOperatorSSHKey: "ssh-ed25519 AAAA op"})
if !st.FelhomSshdActive || st.FelhomSshdPort != 8822 {
t.Fatalf("status: %+v", st)
}
if !st.OperatorPeerConfigured || !st.OperatorKeyConfigured {
t.Fatalf("status must reflect operator peer + key configured: %+v", st)
}
}
// newTestManager builds a Manager with injected active-state + a now clock; sshd -T/-t go through the
// runner. isFree unused here.
func newTestManager(r *scriptRunner, active bool) *Manager {
m := NewManager(r, "/tmp/felhomsshd-test", nil)
m.isActive = func(context.Context) bool { return active }
m.isFailed = func(context.Context) bool { return false }
m.now = func() time.Time { return time.Unix(1783270000, 0) }
return m
}
func TestConfig_NoRuntimeDirectoryString(t *testing.T) {
c, _ := renderConfig(2222)
if strings.Contains(c, "RuntimeDirectory") {
t.Fatal("SF-1: config must never contain RuntimeDirectory")
}
}
+42
View File
@@ -0,0 +1,42 @@
package felhomsshd
import "fmt"
// Candidates is the ordered OOB-port candidate list (spike §2). First free wins; NEVER :22 or a
// random port. Package-var (not const) so tests can shrink it.
var Candidates = []int{8822, 2222, 8022, 62222}
// ErrPortsExhausted is returned when every candidate is busy — a LOUD failure [SF-4/trap 6], never a
// silent fallback to :22 or a random high port.
var ErrPortsExhausted = fmt.Errorf("felhomsshd: all candidate OOB ports are busy — refusing to fall back to :22 or a random port")
// portProbe reports whether a TCP port is free (nothing listening AND a real bind succeeds). Injected
// for tests; production impl = probeFree (ss + net.Listen).
type portProbe func(port int) bool
// claimPort returns the OOB port:
// - if a port is already PERSISTED (and != 22) → keep it unconditionally. It is OUR port; on a
// continuous reconcile felhom-sshd is itself LISTENING on it, so re-probing with isFree would
// (wrongly) find it "busy" by our own daemon and thrash to another candidate every tick. Once
// claimed, the port is stable (the belt @ssh_port and the operator's known port depend on it).
// - else the FIRST free candidate → persist + return (isFree = ss-empty AND a real bind succeeds).
// - else ErrPortsExhausted (LOUD — never :22 or a random port).
//
// persist writes the port file; readPersisted reads it. isFree is the probe. All injected for tests.
func claimPort(candidates []int, isFree portProbe, readPersisted func() (int, bool), persist func(int) error) (int, error) {
if cur, ok := readPersisted(); ok && cur != 22 {
return cur, nil // persisted = ours; keep it (no thrash — felhom-sshd holds it)
}
for _, p := range candidates {
if p == 22 {
continue // defensive: never :22
}
if isFree(p) {
if err := persist(p); err != nil {
return 0, fmt.Errorf("felhomsshd: persist claimed port %d: %w", p, err)
}
return p, nil
}
}
return 0, ErrPortsExhausted
}
+65
View File
@@ -0,0 +1,65 @@
// Package felhomsshd manages the dedicated OOB sshd instance (TASK H1). It is a SECOND sshd —
// separate port, config, host keys, AuthorizedKeysFile, and systemd unit — that COEXISTS with the
// customer's/stock sshd on :22 (never touched). Design + safety earned by
// SPIKE-felhom-sshd-2026-07-05 (§2 claim, §3 SAFE unit, §5 reload-not-restart, §7 AuthorizedKeysFile
// isolation) and SPIKE-oob-wg-operator-peer-2026-07-05 (the tunnel-only belt).
//
// The agent RENDERS the config (Port from the claim) and reloads on change — the wg-felhom pattern.
// It NEVER declares RuntimeDirectory= (G1 [SF-1]) and NEVER restarts on a config change [SF-2].
package felhomsshd
import (
"fmt"
"strings"
)
const (
// ConfDir is the dedicated config tree (host-install creates it; the agent renders the config).
ConfDir = "/etc/felhom-sshd"
// ConfPath is the rendered sshd config (referenced by the static unit's ExecStart/ExecReload).
ConfPath = ConfDir + "/sshd_config"
// HostKeyPath is the dedicated ed25519 host key (host-install generates it; stable across reloads).
HostKeyPath = ConfDir + "/ssh_host_ed25519_key"
// AuthKeysDir holds per-user authorized_keys OUTSIDE ~/.ssh, so the customer's sshd (which reads
// ~/.ssh/authorized_keys) structurally cannot honour the operator key [SF-3/§7].
AuthKeysDir = ConfDir + "/authorized_keys"
// PortFile persists the claimed port (idempotent re-pick).
PortFile = ConfDir + "/port"
// PidFile is the instance pidfile (NOT a RuntimeDirectory — that is the G1 incident cause).
PidFile = "/run/felhom-sshd.pid"
// Unit is the systemd unit name.
Unit = "felhom-sshd"
// OperatorUser is the default operator login (scoped sudo; key in AuthKeysDir only).
OperatorUser = "felhom-op"
)
// renderConfig builds the felhom-sshd config for a claimed port. Pure + deterministic (byte-stable
// for a given port → a stable conf-hash, no reload churn). The security posture is the SAFE template
// from the spike §3: key-only, dedicated host key + AuthorizedKeysFile, AllowUsers scoped to
// root+felhom-op, binds 0.0.0.0 (+ ::) so it never waits on a late interface, no RuntimeDirectory.
func renderConfig(port int) (string, error) {
if port < 1 || port > 65535 {
return "", fmt.Errorf("felhomsshd: port %d out of range", port)
}
if port == 22 {
// The whole point is coexistence — the dedicated instance must NEVER claim :22 [SF-4/trap 5].
return "", fmt.Errorf("felhomsshd: refusing to render on :22 (the stock/customer sshd port)")
}
var b strings.Builder
b.WriteString("# felhom OOB sshd — agent-managed (H1); DO NOT EDIT\n")
fmt.Fprintf(&b, "Port %d\n", port)
b.WriteString("ListenAddress 0.0.0.0\n")
b.WriteString("ListenAddress ::\n")
fmt.Fprintf(&b, "HostKey %s\n", HostKeyPath)
fmt.Fprintf(&b, "PidFile %s\n", PidFile)
fmt.Fprintf(&b, "AuthorizedKeysFile %s/%%u\n", AuthKeysDir)
b.WriteString("PasswordAuthentication no\n")
b.WriteString("PermitRootLogin prohibit-password\n")
b.WriteString("PubkeyAuthentication yes\n")
b.WriteString("KbdInteractiveAuthentication no\n")
b.WriteString("UsePAM yes\n")
fmt.Fprintf(&b, "AllowUsers root %s\n", OperatorUser)
b.WriteString("X11Forwarding no\n")
b.WriteString("Subsystem sftp internal-sftp\n")
return b.String(), nil
}
+101
View File
@@ -0,0 +1,101 @@
package felhomsshd
import (
"strings"
"testing"
)
func TestRenderConfig_SafeTemplateAndByteStable(t *testing.T) {
c, err := renderConfig(8822)
if err != nil {
t.Fatal(err)
}
for _, must := range []string{
"Port 8822\n",
"AuthorizedKeysFile /etc/felhom-sshd/authorized_keys/%u\n",
"HostKey /etc/felhom-sshd/ssh_host_ed25519_key\n",
"PasswordAuthentication no\n",
"PermitRootLogin prohibit-password\n",
"AllowUsers root felhom-op\n",
"PidFile /run/felhom-sshd.pid\n",
} {
if !strings.Contains(c, must) {
t.Errorf("config missing %q:\n%s", must, c)
}
}
// [SF-1] the incident cause must NEVER appear.
if strings.Contains(c, "RuntimeDirectory") {
t.Fatal("config/unit must never mention RuntimeDirectory")
}
// byte-stable for a given port (conf-hash stability → no reload churn).
c2, _ := renderConfig(8822)
if c != c2 {
t.Fatal("renderConfig not deterministic")
}
}
func TestRenderConfig_RefusesPort22AndOutOfRange(t *testing.T) {
if _, err := renderConfig(22); err == nil {
t.Fatal("renderConfig(22) must be refused — never claim the stock/customer port")
}
if _, err := renderConfig(0); err == nil {
t.Fatal("port 0 accepted")
}
if _, err := renderConfig(70000); err == nil {
t.Fatal("out-of-range port accepted")
}
}
// claim harness: a fake free-set + an in-memory persisted port.
func claimHarness(free map[int]bool, persisted int) (portProbe, func() (int, bool), func(int) error, *int) {
stored := persisted
isFree := func(p int) bool { return free[p] }
read := func() (int, bool) {
if stored == 0 {
return 0, false
}
return stored, true
}
write := func(p int) error { stored = p; return nil }
return isFree, read, write, &stored
}
func TestClaimPort_CleanContentionIdempotentExhaustion(t *testing.T) {
cands := []int{8822, 2222, 8022, 62222}
// clean → first candidate
isFree, read, write, stored := claimHarness(map[int]bool{8822: true, 2222: true, 8022: true, 62222: true}, 0)
if p, err := claimPort(cands, isFree, read, write); err != nil || p != 8822 {
t.Fatalf("clean claim = %d / %v, want 8822", p, err)
}
if *stored != 8822 {
t.Fatalf("clean claim not persisted, stored=%d", *stored)
}
// contention: 8822 busy → 2222
isFree, read, write, _ = claimHarness(map[int]bool{8822: false, 2222: true, 8022: true, 62222: true}, 0)
if p, err := claimPort(cands, isFree, read, write); err != nil || p != 2222 {
t.Fatalf("contention claim = %d / %v, want 2222", p, err)
}
// idempotent: a persisted port is KEPT unconditionally (even when isFree(cur) reports busy —
// that's felhom-sshd itself holding it on a continuous reconcile; re-probing would flip-flop).
isFree, read, write, _ = claimHarness(map[int]bool{8822: true, 2222: false, 8022: true, 62222: true}, 2222)
if p, err := claimPort(cands, isFree, read, write); err != nil || p != 2222 {
t.Fatalf("persisted port must be kept even when isFree says busy (own daemon), got %d / %v", p, err)
}
// exhaustion: all busy → LOUD error, no fallback
isFree, read, write, stored = claimHarness(map[int]bool{8822: false, 2222: false, 8022: false, 62222: false}, 0)
p, err := claimPort(cands, isFree, read, write)
if err != ErrPortsExhausted {
t.Fatalf("exhaustion must return ErrPortsExhausted, got %d / %v", p, err)
}
if p == 22 || p != 0 {
t.Fatalf("exhaustion must NOT yield a port (esp. :22), got %d", p)
}
if *stored != 0 {
t.Fatalf("exhaustion must persist nothing, stored=%d", *stored)
}
}
+157
View File
@@ -0,0 +1,157 @@
package felhomsshd
import (
"context"
"os"
"os/exec"
"strconv"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// HealMarkerPath records the last felhom-sshd auto-heal (RFC3339), so the heartbeat surfaces a
// recurring failure to the operator (the mgmt_plane pattern, scoped to the OOB daemon).
const HealMarkerPath = "/run/felhom-sshd.healed"
// HealAndCheck restores felhom-sshd if it is down AND its config is valid — ONE restart per cooldown
// [SF-5 / spike §6], so a persistently-broken instance is reported (Status), not restart-stormed. It
// NEVER restarts onto a broken config [trap 8]: if `sshd -t` fails, it hands off (Status reports
// config_invalid; the hub warns). A healthy/active instance is untouched.
func (m *Manager) HealAndCheck(ctx context.Context, port int) {
if m.isActive(ctx) {
return // healthy — nothing to heal
}
// Down. Only restart if the config is VALID — never convert degraded into dead.
if _, errOut, err := m.runner.Run(ctx, "sshd", "-t", "-f", ConfPath); err != nil {
m.logger.Warn("felhomsshd: down AND config invalid — NOT restarting (report-only)",
"stderr", strings.TrimSpace(string(errOut)))
return
}
// Cooldown: at most one deliberate restart per window.
now := m.now()
if !m.lastRestartAt.IsZero() && now.Sub(m.lastRestartAt) < restartCooldown {
return
}
m.lastRestartAt = now
// reset-failed BEFORE restart [SF-5]: a start-limit lockout otherwise refuses the restart.
if m.isFailed(ctx) {
_, _, _ = m.runner.Run(ctx, "systemctl", "reset-failed", Unit)
}
if err := m.systemctl(ctx, "restart"); err != nil {
return
}
// record the heal (best-effort; a failed marker write never fails the heal)
_ = os.WriteFile(HealMarkerPath, []byte(now.UTC().Format(time.RFC3339)+"\n"), 0o644)
m.logger.Warn("felhomsshd: was down with a valid config — restarted (heal)", "port", port)
}
// Status builds the OOB heartbeat stanza (Part 4). Read-only. Discovers the effective port(s) via
// `sshd -T` (authoritative — catches a non-default/multi-Port config), dials the port locally to
// prove reachability, reads the wg-felhom handshake age, and reflects operator-peer/key config from
// the desired-state block. NEVER `wg show dump` (the S1 ban) — `latest-handshakes` only.
func (m *Manager) Status(ctx context.Context, block *hub.WireWireguard) *hub.OOBStatus {
st := &hub.OOBStatus{
FelhomSshdActive: m.isActive(ctx),
FelhomSshdPort: m.port,
}
// Authoritative port(s) from `sshd -T` (may differ from m.port if the config was hand-edited).
if ports := m.sshdEffectivePorts(ctx); len(ports) > 0 {
st.FelhomSshdPort = ports[0]
}
// config validity
if _, _, err := m.runner.Run(ctx, "sshd", "-t", "-f", ConfPath); err != nil {
st.ConfigInvalid = true
}
// Reachability = a LISTENER is bound on the OOB port (catches "active but crashed post-fork"). We
// do NOT dial: the belt (correctly) drops even localhost→felhom-sshd (tunnel-only), so a local
// dial always fails and would misreport a healthy daemon as unreachable.
if st.FelhomSshdPort > 0 {
st.Reachable = listenerPresent(st.FelhomSshdPort)
}
// wg-felhom handshake age (the OOB path rides the tunnel)
if age, ok := m.wgHandshakeAge(ctx); ok {
st.WGHandshakeAgeS = &age
}
// Operator-peer configured: from the in-memory block when fetched, OR (robust across an agent
// restart, before the next desired-state fetch) from the PERSISTENT belt @operator_ips set — the
// agent can read it via the sudo nft-list grant (unlike the 0600 root-owned wg-felhom.conf). This
// keeps the report + the oob_degraded alert gate accurate immediately after a restart.
st.OperatorPeerConfigured = m.beltOperatorConfigured(ctx)
if block != nil {
if block.OOBPeerIP != "" {
st.OperatorPeerConfigured = true
}
st.OperatorKeyConfigured = strings.TrimSpace(block.OOBOperatorSSHKey) != ""
}
// Operator key: robust across restart via the installed authorized_keys file.
if fi, err := os.Stat(AuthKeysUserPath); err == nil && fi.Size() > 0 {
st.OperatorKeyConfigured = true
}
// last auto-heal
if raw, err := os.ReadFile(HealMarkerPath); err == nil {
st.HealedAt = strings.TrimSpace(string(raw))
}
return st
}
// sshdEffectivePorts parses `sshd -T -f <conf>` for the effective Port line(s) (handles multi-Port).
func (m *Manager) sshdEffectivePorts(ctx context.Context) []int {
out, _, err := m.runner.Run(ctx, "sshd", "-T", "-f", ConfPath)
if err != nil {
return nil
}
var ports []int
for _, line := range strings.Split(string(out), "\n") {
f := strings.Fields(strings.ToLower(line))
if len(f) == 2 && f[0] == "port" {
if p, err := strconv.Atoi(f[1]); err == nil && p > 0 && p <= 65535 {
ports = append(ports, p)
}
}
}
return ports
}
// wgHandshakeAge reads wg-felhom's latest-handshake age in seconds (latest-handshakes ONLY — the
// dump ban). ok=false when the tunnel is down or unreadable.
func (m *Manager) wgHandshakeAge(ctx context.Context) (int64, bool) {
out, _, err := m.runner.Run(ctx, "wg", "show", "wg-felhom", "latest-handshakes")
if err != nil {
return 0, false
}
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Fields(line)
if len(f) != 2 {
continue
}
epoch, err := strconv.ParseInt(f[1], 10, 64)
if err != nil || epoch <= 0 {
continue
}
age := m.now().Unix() - epoch
if age < 0 {
age = 0
}
return age, true
}
return 0, false
}
// listenerPresent reports whether something is LISTENing on the port (via ss — reads kernel state,
// so the belt never blocks it, unlike a real dial).
func listenerPresent(port int) bool {
out, err := exec.Command("ss", "-Htln", "sport = :"+strconv.Itoa(port)).Output()
return err == nil && strings.TrimSpace(string(out)) != ""
}
// beltOperatorConfigured reports whether the belt's @operator_ips set is non-empty (an operator /32
// is allowed to reach felhom-sshd). Read via the sudo nft-list grant — persistent + agent-readable.
func (m *Manager) beltOperatorConfigured(ctx context.Context) bool {
out, _, err := m.runner.Run(ctx, "nft", "list", "set", "inet", "felhom_oob", "operator_ips")
if err != nil {
return false
}
return strings.Contains(string(out), "elements")
}
+113
View File
@@ -0,0 +1,113 @@
package felhomsshd
import (
"context"
"log/slog"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// Loop drives the felhom-sshd Manager on its own cadence (the wgtunnel/lanresolver shape) and
// consumes the hub desired-state's wireguard block via the desired.Syncer raw-consumer seam (for
// oob_peer_ip → the belt, and oob_operator_ssh_key → authorized_keys). Each tick: claim/render/reload
// the instance, sync the belt sets, and run the health/heal check.
type Loop struct {
mgr *Manager
belt *Belt // Part 3 (nil-safe: no belt sync when unset)
interval time.Duration
logger *slog.Logger
mu sync.Mutex
fetched bool // a desired-state document has been delivered (false = never a teardown signal)
block *hub.WireWireguard
nudge chan struct{}
}
// NewLoop builds the loop. interval defaults to 60s. belt may be nil (belt sync skipped).
func NewLoop(mgr *Manager, belt *Belt, interval time.Duration, logger *slog.Logger) *Loop {
if interval <= 0 {
interval = 60 * time.Second
}
if logger == nil {
logger = slog.Default()
}
return &Loop{mgr: mgr, belt: belt, interval: interval, logger: logger, nudge: make(chan struct{}, 1)}
}
// OnDesiredState implements desired.RawConsumer: store the latest wireguard block and nudge.
func (l *Loop) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
if resp == nil {
return
}
l.mu.Lock()
l.fetched = true
l.block = resp.DesiredState.Wireguard
l.mu.Unlock()
select {
case l.nudge <- struct{}{}:
default:
}
}
func (l *Loop) snapshot() (bool, *hub.WireWireguard) {
l.mu.Lock()
defer l.mu.Unlock()
return l.fetched, l.block
}
// Run reconciles immediately, then on every tick or desired-state nudge, until ctx is cancelled.
func (l *Loop) Run(ctx context.Context) error {
l.reconcile(ctx)
t := time.NewTicker(l.interval)
defer t.Stop()
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-t.C:
case <-l.nudge:
}
l.reconcile(ctx)
}
}
// reconcile runs one full pass: instance apply → belt sync → health/heal. The `fetched` gate mirrors
// wgtunnel: until a desired-state document arrives, the operator inputs (authorized_keys, @operator_ips)
// are LEFT UNTOUCHED — an unfetched block must never empty the belt (operator lockout) or wipe the key.
func (l *Loop) reconcile(ctx context.Context) {
fetched, block := l.snapshot()
// Pass the block to Apply only when fetched, so authorized_keys is applied only from real desired
// state (a nil block on Apply skips the authorized_keys write, leaving the existing file).
applyBlock := block
if !fetched {
applyBlock = nil
}
port, err := l.mgr.Apply(ctx, applyBlock)
if err != nil {
return // Apply logged; a claim/exhaustion or install error — retry next tick
}
if l.belt != nil {
l.belt.SyncPort(ctx, port) // always — the port is the agent's own claim
if fetched {
l.belt.SyncOperator(ctx, oobPeerIP(block)) // only from real desired state
}
}
l.mgr.HealAndCheck(ctx, port)
}
// oobPeerIP extracts the operator /32 source (bare IP) from the block, or "" when OOB is off.
func oobPeerIP(block *hub.WireWireguard) string {
if block == nil {
return ""
}
return block.OOBPeerIP
}
// OOBStatus implements the hub collector's reporter seam (Part 4).
func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus {
_, block := l.snapshot()
return l.mgr.Status(ctx, block)
}
+225
View File
@@ -0,0 +1,225 @@
package felhomsshd
import (
"context"
"crypto/sha256"
"encoding/hex"
"log/slog"
"net"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// AuthKeysUserPath is felhom-op's authorized_keys (the default operator identity — key OUTSIDE
// ~/.ssh so the customer's sshd never honours it [SF-3]). FIXED (sudoers install target).
const AuthKeysUserPath = AuthKeysDir + "/" + OperatorUser
const stagedConfName = "sshd_config"
// Manager renders + applies the felhom-sshd config and drives the unit through the narrow-sudoers
// runner. Config changes go write→`sshd -t`→`reload` (NEVER restart-on-change [SF-2]); a deliberate
// restart is `reset-failed`-then-`restart` [SF-5], rate-limited by a cooldown (no flap).
type Manager struct {
runner proxmox.Runner
stateDir string
logger *slog.Logger
// injectable seams (tests)
isActive func(ctx context.Context) bool
isFailed func(ctx context.Context) bool
isFree portProbe
now func() time.Time
port int // the claimed port (0 until Apply claims it)
hash string // sha256 of the last-applied config (steady-state zero-exec gate)
akHash string // sha256 of the last-applied felhom-op authorized_keys
lastRestartAt time.Time // heal cooldown (Part 4)
}
// restartCooldown bounds deliberate felhom-sshd restarts (heal path) — one attempt per window, so a
// persistently-broken instance is reported, not restart-stormed (spike §6 / notification-cooldown shape).
const restartCooldown = 10 * time.Minute
// NewManager builds a Manager. stateDir is the agent state dir (staged config lives under it).
func NewManager(runner proxmox.Runner, stateDir string, logger *slog.Logger) *Manager {
if logger == nil {
logger = slog.Default()
}
return &Manager{
runner: runner,
stateDir: stateDir,
logger: logger,
isActive: func(ctx context.Context) bool {
out, _ := exec.CommandContext(ctx, "systemctl", "is-active", Unit).Output()
return strings.TrimSpace(string(out)) == "active"
},
isFailed: func(ctx context.Context) bool {
out, _ := exec.CommandContext(ctx, "systemctl", "is-failed", Unit).Output()
return strings.TrimSpace(string(out)) == "failed"
},
isFree: probeFree,
now: time.Now,
}
}
// Port returns the claimed port (0 before the first successful Apply).
func (m *Manager) Port() int { return m.port }
func (m *Manager) sshdDir() string { return filepath.Join(m.stateDir, "felhom-sshd") }
func (m *Manager) stagedConfPath() string { return filepath.Join(m.sshdDir(), stagedConfName) }
func (m *Manager) portFilePath() string { return filepath.Join(m.sshdDir(), "port") }
// readPort/writePort persist the claimed port in the AGENT-OWNED state dir (the agent is non-root and
// cannot write the root-owned /etc/felhom-sshd). A port is not a secret.
func (m *Manager) readPort() (int, bool) {
raw, err := os.ReadFile(m.portFilePath())
if err != nil {
return 0, false
}
p, err := strconv.Atoi(strings.TrimSpace(string(raw)))
if err != nil || p < 1 || p > 65535 {
return 0, false
}
return p, true
}
func (m *Manager) writePort(port int) error {
if err := os.MkdirAll(m.sshdDir(), 0o700); err != nil {
return err
}
return os.WriteFile(m.portFilePath(), []byte(strconv.Itoa(port)+"\n"), 0o600)
}
// Apply claims the port, renders the config, reconciles the running unit, and (when the desired-state
// block carries it) installs the operator's authorized_keys. Idempotent: no change → at most an
// is-active check. block may be nil (no desired-state yet) — the instance still runs; only the
// operator login/belt inputs are skipped. Returns the claimed port (0 on a claim/exhaustion error) —
// the caller (belt sync) needs it.
func (m *Manager) Apply(ctx context.Context, block *hub.WireWireguard) (int, error) {
port, err := claimPort(Candidates, m.isFree, m.readPort, m.writePort)
if err != nil {
m.logger.Error("felhomsshd: port claim failed", "err", err)
return 0, err
}
m.port = port
// Operator authorized_keys (from the hub-driven block) — written to felhom-op's dedicated file,
// outside ~/.ssh [SF-3]. Independent of the config-reload path (a key change never reloads sshd).
if block != nil {
m.applyAuthorizedKeys(ctx, block.OOBOperatorSSHKey)
}
conf, err := renderConfig(port)
if err != nil {
m.logger.Error("felhomsshd: refusing to apply invalid config", "err", err)
return port, err
}
sum := sha256.Sum256([]byte(conf))
hash := hex.EncodeToString(sum[:])
active := m.isActive(ctx)
if m.hash == hash && active {
return port, nil // steady state: zero execs
}
// Stage → validate → install → reload/enable. Never restart on a config change [SF-2].
if err := os.MkdirAll(m.sshdDir(), 0o700); err != nil {
m.logger.Error("felhomsshd: state dir", "err", err)
return port, err
}
if err := os.WriteFile(m.stagedConfPath(), []byte(conf), 0o600); err != nil {
m.logger.Error("felhomsshd: staging config", "err", err)
return port, err
}
// Validate the STAGED config before it is installed — a bad render never reaches the live path.
if _, errOut, err := m.runner.Run(ctx, "sshd", "-t", "-f", m.stagedConfPath()); err != nil {
m.logger.Error("felhomsshd: staged config failed sshd -t — NOT installing", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return port, err
}
if _, errOut, err := m.runner.Run(ctx, "install", "-o", "root", "-g", "root", "-m", "0644", "--", m.stagedConfPath(), ConfPath); err != nil {
m.logger.Error("felhomsshd: config install failed", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return port, err
}
verb := "reload"
if !active {
verb = "enable" // first apply / down → enable --now brings it up
}
if err := m.systemctl(ctx, verb); err != nil {
return port, err
}
m.hash = hash
m.logger.Info("felhomsshd: config applied", "port", port, "action", verb)
return port, nil
}
// applyAuthorizedKeys installs (or clears) felhom-op's authorized_keys from the hub-delivered
// operator SSH key. Hash-gated (no churn); the key is public, never a secret. A write failure is
// logged, not fatal — the instance keeps running. NO sshd reload needed (sshd reads the file per-auth).
func (m *Manager) applyAuthorizedKeys(ctx context.Context, sshKey string) {
content := ""
if k := strings.TrimSpace(sshKey); k != "" {
content = k + "\n"
}
sum := sha256.Sum256([]byte(content))
h := hex.EncodeToString(sum[:])
if h == m.akHash {
return // unchanged
}
staged := filepath.Join(m.sshdDir(), "authorized_keys."+OperatorUser)
if err := os.MkdirAll(m.sshdDir(), 0o700); err != nil {
m.logger.Error("felhomsshd: state dir for authorized_keys", "err", err)
return
}
if err := os.WriteFile(staged, []byte(content), 0o600); err != nil {
m.logger.Error("felhomsshd: staging authorized_keys", "err", err)
return
}
if _, errOut, err := m.runner.Run(ctx, "install", "-o", "root", "-g", "root", "-m", "0644", "--", staged, AuthKeysUserPath); err != nil {
m.logger.Error("felhomsshd: authorized_keys install failed", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return
}
m.akHash = h
m.logger.Info("felhomsshd: operator authorized_keys updated", "user", OperatorUser, "present", content != "")
}
// systemctl runs the reconcile verbs through the narrow runner. `enable` = `enable --now`; `reload`
// HUPs (config change, running instance survives a bad reload via the unit's ExecReload sshd -t gate).
func (m *Manager) systemctl(ctx context.Context, verb string) error {
var args []string
switch verb {
case "enable":
args = []string{"enable", "--now", Unit}
case "reload", "restart":
args = []string{verb, Unit}
default:
return nil
}
if _, errOut, err := m.runner.Run(ctx, "systemctl", args...); err != nil {
m.logger.Error("felhomsshd: systemctl "+verb+" failed", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return err
}
return nil
}
// probeFree is the production port probe: nothing LISTENing (ss) AND a real bind succeeds (a bind
// that succeeds-then-closes proves the port is actually claimable, not just ss-silent).
func probeFree(port int) bool {
p := strconv.Itoa(port)
out, err := exec.Command("ss", "-Htln", "sport = :"+p).Output()
if err == nil && strings.TrimSpace(string(out)) != "" {
return false // something is listening
}
ln, err := net.Listen("tcp", "0.0.0.0:"+p)
if err != nil {
return false
}
_ = ln.Close()
return true
}
+139
View File
@@ -0,0 +1,139 @@
// Package guesthook is the LXC guest pre-start self-heal (C1 net, transitional).
//
// THE BUG IT FIXES (C1, B3 audit): in the per-drive bind model an external data drive is bound into the
// guest as `pct set <vmid> -mpN <drive>/felhom-data,mp=/mnt/<name>`. When that drive is ABSENT at guest
// boot, the bind SOURCE `<drive>/felhom-data` does not exist, `pct start` fails the mount, and the guest
// BRICKS (pre-start exit 255 — ALL apps down). Today nothing recovers it.
//
// THE FIX: a PVE `pre-start` hookscript runs this code; for every BIND mountpoint whose source path is
// missing it CREATES an empty, host-root-owned placeholder directory so the mount succeeds and the guest
// boots. It is fail-closed: the placeholder is owned by host root (uid 0), which is UNMAPPED in the
// unprivileged-LXC user namespace, so the in-guest controller/apps (even as guest-root) cannot write to
// it — and a returning drive simply shadows it (the agent mounts over it).
//
// WHY CREATE, NOT DELETE: removing the dead mp would need `pct set <vmid> --delete mpN`, which takes the
// per-guest config lock the start task ALREADY holds → it dead-times-out (~10s) and the guest still
// bricks. So in pre-start we NEUTRALISE (placeholder) rather than mutate config. Proper mp removal runs
// OUTSIDE the start lock — at decommission (handleDiskDecommission → DetachBind) and the startup
// reconcile. The intermediary-mount re-architecture later makes C1 STRUCTURAL (the only bind source is
// the permanent, always-present /mnt/felhom-drives parent), after which this hook is pure defense-in-depth.
package guesthook
import (
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// PhasePreStart is the PVE hook phase at which we self-heal (before the container mounts are set up).
const PhasePreStart = "pre-start"
// PhasePostStart is the PVE hook phase after the container started — the NAS automount reassert
// point (the fresh guest namespace has no idle autofs triggers; see netreassert.go).
const PhasePostStart = "post-start"
// placeholderMode is the mode for a created bind-source placeholder. Host-root-owned + this mode =
// fail-closed against the unprivileged guest (host uid 0 is unmapped in the guest userns).
const placeholderMode = 0o755
// ParseConfMounts parses an LXC config file body (/etc/pve/lxc/<vmid>.conf) and returns each mount key
// (`mp0`..`mp255`, plus `rootfs`) mapped to its SOURCE — the first comma-field of the value, before any
// `mp=`/`size=`/`backup=` options. A BIND mount has an absolute-path source (`/mnt/...`); a storage
// volume has a `<storage>:<volid>` source (no leading slash). Lines that aren't a mountpoint/rootfs key
// are ignored.
func ParseConfMounts(conf string) map[string]string {
out := map[string]string{}
for _, line := range strings.Split(conf, "\n") {
line = strings.TrimSpace(line)
colon := strings.IndexByte(line, ':')
if colon <= 0 {
continue
}
key := line[:colon]
if key != "rootfs" && !(strings.HasPrefix(key, "mp") && isAllDigits(strings.TrimPrefix(key, "mp"))) {
continue
}
val := strings.TrimSpace(line[colon+1:])
if val == "" {
continue
}
src := val
if c := strings.IndexByte(val, ','); c >= 0 {
src = val[:c]
}
out[key] = strings.TrimSpace(src)
}
return out
}
// isBindSource reports whether an mp source is a host-path BIND (an absolute path) rather than a PVE
// storage volume (`<storage>:<volid>`, never absolute). On the Linux host a bind source is `/mnt/...`
// (leading slash); the filepath.IsAbs arm additionally recognises an OS-absolute path so the real-IO
// tests pass under a Windows temp dir too — on Linux both arms agree and a storage volid matches neither.
func isBindSource(src string) bool {
return strings.HasPrefix(src, "/") || filepath.IsAbs(src)
}
func isAllDigits(s string) bool {
if s == "" {
return false
}
for _, c := range s {
if c < '0' || c > '9' {
return false
}
}
return true
}
// MissingBindSources returns the BIND-mount source paths (absolute host paths) that do NOT exist, sorted
// and de-duplicated. Storage-volume sources (`<storage>:<volid>`, no leading '/') are NEVER returned —
// only a real host-path bind can have a vanished source we must heal; a storage volume that's missing is
// PVE's own concern, not ours to mkdir. `exists` reports whether a path is present (injected for tests).
func MissingBindSources(mounts map[string]string, exists func(string) bool) []string {
seen := map[string]bool{}
var miss []string
for _, src := range mounts {
if !isBindSource(src) { // storage volume (<storage>:<volid>), not a host-path bind — never touch
continue
}
if seen[src] || exists(src) {
continue
}
seen[src] = true
miss = append(miss, src)
}
sort.Strings(miss)
return miss
}
// Heal reads the LXC config at confPath and creates a placeholder directory for every bind-mount source
// that is missing, returning the list of paths it created. It never returns a fatal error for an
// unreadable/empty config (a guest with no config simply has nothing to heal) — the hook must NEVER block
// a start. A mkdir failure on one path is collected into err but the others still proceed.
func Heal(confPath string) (created []string, err error) {
data, readErr := os.ReadFile(confPath)
if readErr != nil {
// No config = nothing to heal. Never block the start over a read error.
return nil, nil
}
mounts := ParseConfMounts(string(data))
miss := MissingBindSources(mounts, func(p string) bool {
_, statErr := os.Stat(p)
return statErr == nil
})
var errs []string
for _, p := range miss {
if mkErr := os.MkdirAll(p, placeholderMode); mkErr != nil {
errs = append(errs, fmt.Sprintf("%s: %v", p, mkErr))
continue
}
created = append(created, p)
}
if len(errs) > 0 {
return created, fmt.Errorf("guesthook: placeholder creation failed for: %s", strings.Join(errs, "; "))
}
return created, nil
}
+118
View File
@@ -0,0 +1,118 @@
package guesthook
import (
"os"
"path/filepath"
"reflect"
"testing"
)
const sampleConf = `arch: amd64
cores: 2
hostname: demo-felhom
memory: 12288
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=256G
mp1: /mnt/felhom-usb/felhom-data,mp=/mnt/felhom-usb
mp2: /mnt/felhom-flash/felhom-data,mp=/mnt/felhom-flash
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
net0: name=eth0,bridge=vmbr0
rootfs: local-lvm:vm-9201-disk-0,size=32G
swap: 4096
unprivileged: 1
`
func TestParseConfMounts(t *testing.T) {
got := ParseConfMounts(sampleConf)
want := map[string]string{
"mp0": "local-lvm:vm-9201-disk-1",
"mp1": "/mnt/felhom-usb/felhom-data",
"mp2": "/mnt/felhom-flash/felhom-data",
"mp9": "/var/lib/felhom-agent/guests/9201/bootstrap",
"rootfs": "local-lvm:vm-9201-disk-0",
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("ParseConfMounts mismatch:\n got=%v\nwant=%v", got, want)
}
// net0/arch/etc. (non-mount keys) must NOT leak in.
if _, bad := got["net0"]; bad {
t.Fatalf("net0 was parsed as a mount source")
}
}
// TestMissingBindSources is the load-bearing selector test. The flash drive is absent (its felhom-data
// source missing); usb is present; mp0/rootfs are STORAGE volumes (must never be selected); mp9's
// bootstrap source is present. Only the flash source may be returned.
//
// COMPANION GUARD — this test FAILS on the two trivial impls the spec warns about:
// - "return nothing" (the pre-fix no-op hook) → flash not selected → guest still bricks → FAIL.
// - "return every source" (mkdir everything) → would include the present usb bind AND the
// local-lvm storage volumes (creating bogus dirs that shadow real data) → FAIL.
func TestMissingBindSources(t *testing.T) {
mounts := ParseConfMounts(sampleConf)
present := map[string]bool{
"/mnt/felhom-usb/felhom-data": true, // usb attached
"/var/lib/felhom-agent/guests/9201/bootstrap": true, // bootstrap always present
// "/mnt/felhom-flash/felhom-data" is ABSENT (drive unplugged)
}
got := MissingBindSources(mounts, func(p string) bool { return present[p] })
want := []string{"/mnt/felhom-flash/felhom-data"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("MissingBindSources mismatch:\n got=%v\nwant=%v", got, want)
}
// Explicit companion assertions (pin both failure directions independently of want):
for _, p := range got {
if p == "/mnt/felhom-usb/felhom-data" {
t.Fatalf("selected a PRESENT bind source — over-eager (would shadow live data)")
}
if p == "local-lvm:vm-9201-disk-1" || p == "local-lvm:vm-9201-disk-0" {
t.Fatalf("selected a STORAGE VOLUME source — must only heal host-path binds")
}
}
if len(got) == 0 {
t.Fatalf("selected nothing — the absent flash bind would brick the guest (pre-fix no-op)")
}
}
// TestHealCreatesOnlyMissingBind drives the real filesystem path in a temp dir: a present bind source is
// left untouched, an absent one is created (so the guest boots), a storage volume is never created.
func TestHealCreatesOnlyMissingBind(t *testing.T) {
root := t.TempDir()
presentSrc := filepath.Join(root, "usb", "felhom-data")
absentSrc := filepath.Join(root, "flash", "felhom-data")
if err := os.MkdirAll(presentSrc, 0o755); err != nil {
t.Fatal(err)
}
conf := "" +
"mp0: local-lvm:vm-9-disk-0,mp=/var/lib/docker,backup=1\n" +
"mp1: " + presentSrc + ",mp=/mnt/usb\n" +
"mp2: " + absentSrc + ",mp=/mnt/flash\n" +
"rootfs: local-lvm:vm-9-disk-1,size=32G\n"
confPath := filepath.Join(root, "9.conf")
if err := os.WriteFile(confPath, []byte(conf), 0o644); err != nil {
t.Fatal(err)
}
created, err := Heal(confPath)
if err != nil {
t.Fatalf("Heal: %v", err)
}
if !reflect.DeepEqual(created, []string{absentSrc}) {
t.Fatalf("Heal created %v, want [%s]", created, absentSrc)
}
if _, err := os.Stat(absentSrc); err != nil {
t.Fatalf("absent bind source not created — guest would still brick: %v", err)
}
// A storage-volume mp must never produce a bogus host directory.
if _, err := os.Stat(filepath.Join(root, "local-lvm:vm-9-disk-0")); err == nil {
t.Fatalf("a storage volume source was materialised as a directory")
}
}
// TestHealMissingConfNeverErrors — a hook must never block a start, even for an unreadable config.
func TestHealMissingConfNeverErrors(t *testing.T) {
created, err := Heal(filepath.Join(t.TempDir(), "does-not-exist.conf"))
if err != nil || created != nil {
t.Fatalf("Heal on missing conf: created=%v err=%v (want nil,nil)", created, err)
}
}
+82
View File
@@ -0,0 +1,82 @@
package guesthook
import (
"context"
"fmt"
"os"
"path/filepath"
"strconv"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// Install/registration of the pre-start hook. The wrapper lives in a PVE `snippets`-enabled storage dir
// (the `local` storage maps to /var/lib/vz/snippets) and is referenced per-guest by its volid.
const (
// SnippetDir is the local-storage snippets directory PVE serves hookscripts from.
SnippetDir = "/var/lib/vz/snippets"
// SnippetName is the wrapper filename.
SnippetName = "felhom-guest-hook.sh"
// HookVolID is the volid form `pct set --hookscript` expects.
HookVolID = "local:snippets/" + SnippetName
// AgentBin is the installed agent binary the wrapper delegates to.
AgentBin = "/usr/local/bin/felhom-agent"
)
// SnippetPath is the absolute path of the installed wrapper.
var SnippetPath = filepath.Join(SnippetDir, SnippetName)
// snippetBody is the tiny wrapper PVE execs as `<script> <vmid> <phase>`. It delegates to the agent
// binary so the heal LOGIC is the unit-tested Go, never duplicated (divergence-proof) shell. Executable.
// The wrapper NEVER exec's and ALWAYS exits 0 (CAMPAIGN-3 F10/rc255 belt): a hook that exits nonzero
// aborts the guest start. `exec` would surface the binary's exit code to PVE; instead we run it as a
// child, swallow any nonzero (missing/crashed binary, OOM-kill), and `exit 0` unconditionally. The Go
// side has its own recover + per-phase timeout — this is the second belt at the shell layer.
const snippetBody = `#!/bin/sh
# felhom-agent guest pre-start self-heal hook (C1 net). PVE calls: <script> <vmid> <phase>.
` + AgentBin + ` guest-hook "$1" "$2" || true
exit 0
`
// InstallSnippet writes the pre-start hook wrapper into the PVE snippets dir (idempotent, root-owned,
// executable). The agent runs as a non-root service user, so it writes an agent-writable temp file then
// `install`s it host-root (same pattern as the bootstrap mount + dnsmasq drop-ins). Safe to call repeatedly.
// The temp file is a RANDOM-named os.CreateTemp (audit B1): a fixed, predictable /tmp name could be
// pre-created by another local user and rewritten between our write and root's install (TOCTOU into a
// root-executed hookscript). The final mode comes from `install -m`, so the 0600 temp is fine.
func InstallSnippet(ctx context.Context, runner proxmox.Runner) error {
f, err := os.CreateTemp("", "felhom-guest-hook-*.sh")
if err != nil {
return fmt.Errorf("guesthook: create temp snippet: %w", err)
}
tmp := f.Name()
defer os.Remove(tmp)
if _, err := f.WriteString(snippetBody); err != nil {
f.Close()
return fmt.Errorf("guesthook: write temp snippet: %w", err)
}
if err := f.Close(); err != nil {
return fmt.Errorf("guesthook: close temp snippet: %w", err)
}
// Ensure the snippets dir exists FIRST (B2, DRILL-day0-cleanroom-2026-07-03): a fresh PVE has
// no /var/lib/vz/snippets, and `install` (without -D) won't create the parent — the whole
// hook install silently failed on a freshly-bootstrapped box. Fenced root op like the install
// itself; idempotent.
if _, stderr, err := runner.Run(ctx, "mkdir", "-p", SnippetDir); err != nil {
return fmt.Errorf("guesthook: ensure snippets dir %s: %w: %s", SnippetDir, err, string(stderr))
}
if _, stderr, err := runner.Run(ctx, "install", "-m", "0755", "--", tmp, SnippetPath); err != nil {
return fmt.Errorf("guesthook: install snippet to %s: %w: %s", SnippetPath, err, string(stderr))
}
return nil
}
// Register points a guest at the pre-start hook (`pct set <vmid> --hookscript <volid>`). Idempotent —
// re-setting the same hookscript is a no-op. Safe on a running guest (a config edit, not a start, so no
// start-lock contention).
func Register(ctx context.Context, runner proxmox.Runner, vmid int) error {
if _, stderr, err := runner.Run(ctx, "pct", "set", strconv.Itoa(vmid), "--hookscript", HookVolID); err != nil {
return fmt.Errorf("guesthook: register hookscript on %d: %w: %s", vmid, err, string(stderr))
}
return nil
}
+128
View File
@@ -0,0 +1,128 @@
package guesthook
import (
"context"
"io"
"os"
"regexp"
"testing"
)
// recordingRunner is a fake proxmox.Runner that records every call and snapshots the content of the
// install SOURCE file at call time (the deferred os.Remove would erase it before the test can look).
type recordingRunner struct {
calls [][]string
srcContent []string
}
func (r *recordingRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
if name == "install" && len(args) > 0 {
src := args[len(args)-2]
b, _ := os.ReadFile(src)
r.srcContent = append(r.srcContent, string(b))
}
return nil, nil, nil
}
func (r *recordingRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
// TestInstallSnippet_RandomTempName is the audit-B1 negative test: the staged install SOURCE must be a
// RANDOM os.CreateTemp name (felhom-guest-hook-<random>.sh), never the fixed, pre-creatable
// /tmp/felhom-guest-hook.sh (a local TOCTOU into a root-executed hookscript), and two consecutive
// installs must stage through DIFFERENT paths.
func TestInstallSnippet_RandomTempName(t *testing.T) {
r := &recordingRunner{}
if err := InstallSnippet(context.Background(), r); err != nil {
t.Fatalf("InstallSnippet #1: %v", err)
}
if err := InstallSnippet(context.Background(), r); err != nil {
t.Fatalf("InstallSnippet #2: %v", err)
}
var installs [][]string
for _, call := range r.calls {
if call[0] == "install" {
installs = append(installs, call)
}
}
if len(installs) != 2 {
t.Fatalf("expected 2 install calls, got %d: %v", len(installs), r.calls)
}
randomName := regexp.MustCompile(`felhom-guest-hook-[^/\\]+\.sh$`)
fixedName := regexp.MustCompile(`felhom-guest-hook\.sh$`)
var srcs []string
for i, call := range installs {
// install -m 0755 -- <src> <dest>
if len(call) != 6 {
t.Fatalf("call %d: unexpected vector %v", i, call)
}
src, dest := call[4], call[5]
if dest != SnippetPath {
t.Errorf("call %d: dest = %q, want %q", i, dest, SnippetPath)
}
if !randomName.MatchString(src) {
t.Errorf("call %d: source %q does not match the random felhom-guest-hook-*.sh pattern", i, src)
}
if fixedName.MatchString(src) {
t.Errorf("call %d: source %q is the FIXED predictable temp name (B1 TOCTOU)", i, src)
}
srcs = append(srcs, src)
}
if srcs[0] == srcs[1] {
t.Errorf("two consecutive installs staged through the SAME source path %q — must be random per call", srcs[0])
}
// Non-hollow: the staged file must actually carry the snippet body at install time.
for i, c := range r.srcContent {
if c != snippetBody {
t.Errorf("call %d: staged content is not the snippet body (got %d bytes)", i, len(c))
}
}
// And the temp is cleaned up after.
for _, src := range srcs {
if _, err := os.Stat(src); err == nil {
t.Errorf("staged temp %q left behind (defer os.Remove missing)", src)
}
}
}
// B2 Scenario D (DRILL-day0-cleanroom-2026-07-03): on a fresh PVE, /var/lib/vz/snippets does not
// exist and `install` (no -D) cannot create it — the drill saw
// `install: cannot create regular file … No such file or directory` and the guest silently got no
// pre-start self-heal hook. InstallSnippet must therefore issue a `mkdir -p <SnippetDir>` fenced op
// BEFORE the `install` op. Pre-fix wrong outcome: no mkdir call at all — only the doomed install.
func TestInstallSnippet_EnsuresSnippetsDirFirst(t *testing.T) {
r := &recordingRunner{}
if err := InstallSnippet(context.Background(), r); err != nil {
t.Fatalf("InstallSnippet: %v", err)
}
mkdirIdx, installIdx := -1, -1
for i, call := range r.calls {
switch call[0] {
case "mkdir":
if mkdirIdx == -1 {
mkdirIdx = i
want := []string{"mkdir", "-p", SnippetDir}
if len(call) != 3 || call[1] != want[1] || call[2] != want[2] {
t.Errorf("mkdir vector = %v, want %v (the sudoers fence matches exactly this argv)", call, want)
}
}
case "install":
if installIdx == -1 {
installIdx = i
}
}
}
if mkdirIdx == -1 {
t.Fatalf("no `mkdir -p %s` op issued — on a fresh box the snippet install fails ENOENT (B2); calls: %v", SnippetDir, r.calls)
}
if installIdx == -1 {
t.Fatalf("no install op issued; calls: %v", r.calls)
}
if mkdirIdx > installIdx {
t.Fatalf("mkdir (call %d) must PRECEDE install (call %d) — order: %v", mkdirIdx, installIdx, r.calls)
}
}
+94
View File
@@ -0,0 +1,94 @@
package guesthook
import (
"context"
"fmt"
"log/slog"
"os"
"strings"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// post-start network-storage reassert (RCA AUDIT-nas-cwa-rca-2026-07-11 fix 1, hook leg).
//
// A freshly started guest's namespace does NOT inherit an idle NAS autofs trigger (only real
// mounts), so its /mnt/felhom-drives/<share> path is a silent local stub until the trigger is
// re-created host-side. PVE runs the hookscript as root in the start task, so this leg calls
// systemctl DIRECTLY (no sudo) — the daemon leg (localapi.ReassertNetworkMounts) is the sudo path.
// Like the pre-start heal, this must NEVER fail the hook: all errors go to stderr (the PVE task
// log) and the guest start proceeds regardless.
// netReasserter is the reassert capability (satisfied by *storage.SudoHostOps; faked in tests).
type netReasserter interface {
ReassertNetworkAutomounts(ctx context.Context) []storage.NetReassertResult
RearmNetworkAutomount(ctx context.Context, where string) error
}
// PostStartNetworkReassert re-arms idle NAS automount triggers after vmid started, then verifies
// the (now running) guest actually sees each share path. Best-effort throughout.
func PostStartNetworkReassert(ctx context.Context, vmid string) {
runner := &proxmox.ExecRunner{Mode: proxmox.RunnerDirect}
ops := storage.NewSudoHostOps(storage.SudoHostOpsConfig{
Runner: runner,
Logger: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo})),
})
postStartNetworkReassert(ctx, vmid, ops, func(ctx context.Context, vmid, path string) bool {
return GuestSeesPath(ctx, runner, vmid, path)
})
}
// postStartNetworkReassert is the seam-injected core (unit-tested; the wrapper above binds the
// real host surface).
func postStartNetworkReassert(ctx context.Context, vmid string, ops netReasserter, sees func(ctx context.Context, vmid, path string) bool) {
for _, res := range ops.ReassertNetworkAutomounts(ctx) {
if !res.Remediates() {
continue // foreign/errored rows expect nothing in the guest (already logged by the ops layer)
}
if sees(ctx, vmid, res.Where) {
fmt.Fprintf(os.Stderr, "felhom-agent guest-hook: vmid %s post-start — network share %s visible in guest (%s)\n",
vmid, res.Name, res.Action)
continue
}
// F11 matrix-correction: the guest is blind even though the pass classified this share
// skip-active/rearmed. A freshly-started namespace does NOT inherit an ACTIVE host mount — only
// a FRESH trigger event propagates in. Re-arm this specific share and re-verify once.
fmt.Fprintf(os.Stderr, "felhom-agent guest-hook: vmid %s post-start — network share %s not visible after reassert (%s) — re-arming\n",
vmid, res.Name, res.Action)
if err := ops.RearmNetworkAutomount(ctx, res.Where); err != nil {
fmt.Fprintf(os.Stderr, "felhom-agent guest-hook: vmid %s post-start — WARNING: re-arm of %s failed: %v\n", vmid, res.Name, err)
continue
}
if sees(ctx, vmid, res.Where) {
fmt.Fprintf(os.Stderr, "felhom-agent guest-hook: vmid %s post-start — network share %s visible in guest after re-arm (healed)\n", vmid, res.Name)
} else {
fmt.Fprintf(os.Stderr, "felhom-agent guest-hook: vmid %s post-start — WARNING: network share %s STILL NOT visible after re-arm\n", vmid, res.Name)
}
}
}
// GuestSeesPath reports whether vmid's guest has `path` as a mount target in its own namespace —
// the hook-process mirror of localapi's GuestBinder.GuestSeesMount (which is method-bound to the
// daemon's binder and unavailable here). Resolution/read errors → false.
func GuestSeesPath(ctx context.Context, runner proxmox.Runner, vmid, path string) bool {
out, _, err := runner.Run(ctx, "lxc-info", "-n", vmid, "-p", "-H")
if err != nil {
return false
}
pid := strings.TrimSpace(string(out))
if pid == "" {
return false
}
data, err := os.ReadFile("/proc/" + pid + "/mountinfo")
if err != nil {
return false
}
for _, line := range strings.Split(string(data), "\n") {
f := strings.Fields(line)
if len(f) >= 5 && f[4] == path {
return true
}
}
return false
}
+88
View File
@@ -0,0 +1,88 @@
package guesthook
import (
"context"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
type fakeReasserter struct {
invoked int
rearmed []string
rearmErr error
results []storage.NetReassertResult
}
func (f *fakeReasserter) ReassertNetworkAutomounts(context.Context) []storage.NetReassertResult {
f.invoked++
return f.results
}
func (f *fakeReasserter) RearmNetworkAutomount(_ context.Context, where string) error {
f.rearmed = append(f.rearmed, where)
return f.rearmErr
}
// The post-start core must run the reassert pass and verify guest visibility for every share the
// pass acted on (or found actively mounted) — and never for foreign/errored rows.
func TestPostStartNetworkReassert_Core(t *testing.T) {
ops := &fakeReasserter{results: []storage.NetReassertResult{
{Name: "media", Where: "/mnt/felhom-drives/media", Action: storage.NetReassertRearmed},
{Name: "active", Where: "/mnt/felhom-drives/active", Action: storage.NetReassertSkipActive},
{Name: "foreign", Where: "/mnt/felhom-drives/foreign", Action: storage.NetReassertSkipForeign},
}}
var verified []string
postStartNetworkReassert(context.Background(), "9201", ops, func(_ context.Context, vmid, path string) bool {
if vmid != "9201" {
t.Errorf("verify called with vmid %q, want 9201", vmid)
}
verified = append(verified, path)
return true
})
if ops.invoked != 1 {
t.Fatalf("reassert pass invoked %d times, want 1", ops.invoked)
}
if len(verified) != 2 || verified[0] != "/mnt/felhom-drives/media" || verified[1] != "/mnt/felhom-drives/active" {
t.Fatalf("verify must cover rearmed + skip-active only, got %v", verified)
}
}
// F11 matrix-correction: a guest that does NOT see a share (even a skip-active one) must trigger a
// re-arm of that specific share, then a re-verify. Encodes the live 2026-07-12 finding that a rebooted
// guest inherits neither an idle trigger nor an active mount.
func TestPostStartNetworkReassert_RearmsWhenGuestBlind(t *testing.T) {
ops := &fakeReasserter{results: []storage.NetReassertResult{
{Name: "media", Where: "/mnt/felhom-drives/media", Action: storage.NetReassertSkipActive},
}}
// sees: false the first time (blind), true the second (post-rearm) — the heal path.
calls := 0
postStartNetworkReassert(context.Background(), "9201", ops, func(context.Context, string, string) bool {
calls++
return calls >= 2
})
if len(ops.rearmed) != 1 || ops.rearmed[0] != "/mnt/felhom-drives/media" {
t.Fatalf("a blind guest must trigger exactly one re-arm of the share, got %v", ops.rearmed)
}
}
// A failed verify must be non-fatal even if the re-arm also fails: the core returns normally (hook
// exits 0 regardless).
func TestPostStartNetworkReassert_VerifyFailureNonFatal(t *testing.T) {
ops := &fakeReasserter{
rearmErr: context.DeadlineExceeded,
results: []storage.NetReassertResult{
{Name: "media", Where: "/mnt/felhom-drives/media", Action: storage.NetReassertRearmed},
},
}
// Must not panic or abort; the WARNING goes to stderr (PVE task log). Guest never sees it, re-arm errors.
postStartNetworkReassert(context.Background(), "9201", ops, func(context.Context, string, string) bool {
return false
})
if ops.invoked != 1 {
t.Fatalf("reassert pass invoked %d times, want 1", ops.invoked)
}
if len(ops.rearmed) != 1 {
t.Fatalf("a blind guest must attempt one re-arm, got %d", len(ops.rearmed))
}
}
+239
View File
@@ -0,0 +1,239 @@
// Package guestnet implements R-54: the host-tier watchdog for each customer guest's own network.
//
// Origin — INCIDENT-guest-dhclient-killed-2026-07-20 §5 "OPEN RISK". The guest's DHCP client is
// started once by ifupdown at boot and NOTHING supervises it. When it was killed on 2026-07-20 the
// guest kept working for another ~80 minutes on its unexpired lease; only when the lease expired did
// the address and default route vanish, taking the Cloudflare tunnel, the hub reports, the catalog
// sync and the controller→agent channel with them. Total outage ~1h15m, and for the first 80 minutes
// every observable signal said healthy.
//
// The design consequence is the whole point of this package: **liveness of the DHCP client process
// is itself a probe**, not a detail. Waiting for the IP to disappear is waiting out the exact silent
// window the incident proved exists. See TestProbe_DeadDHClientWithLiveLeaseIsUnhealthy.
//
// The agent is the right tier for this: it lives on the host, keeps its own line to the hub, and can
// still see and repair a guest that has gone completely mute. The controller cannot fix its own
// missing default route.
package guestnet
import (
"context"
"strings"
)
// Runner is the privileged-exec seam (satisfied by *proxmox.ExecRunner). Declared consumer-side so
// tests inject a scripted runner and no unit test goes near pct.
type Runner interface {
Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
}
// Mode is how the guest is configured to get its address.
type Mode string
const (
ModeDHCP Mode = "dhcp"
ModeStatic Mode = "static"
ModeUnknown Mode = "unknown"
)
// State is a guest's verdict for one cycle.
type State string
const (
StateHealthy State = "healthy"
// StateUnhealthy: DHCP-configured and something is wrong that dhclient can fix.
StateUnhealthy State = "unhealthy"
// StateStaticFault: a static guest missing its address/route. Reported loudly, NEVER healed —
// re-running dhclient on a statically-configured guest would fight its own configuration, and
// the durable answer is R-50 (island bridge), not a point fix here.
StateStaticFault State = "static_fault"
// StateUnknown: the guest could not be probed at all (pct exec failed, an interface file we
// cannot read, a probe tool missing). Never healed — acting blind is how the incident happened.
StateUnknown State = "unknown"
)
// Probe is one guest's observed network facts.
type Probe struct {
VMID int
Mode Mode
IP string // empty when absent
HasRoute bool
DHCPAlive bool
Reachable bool // pct exec worked at all
Detail string // human-readable reason, operator-tier English
}
// eth0 is the guest interface every felhom guest uses (the LXC veth peer inside the guest).
const eth0 = "eth0"
// probe runs the four fixed-shape reads. Every argv is a constant plus the vmid — no guest-supplied
// data is ever interpolated into a command, and there is no shell anywhere in this path.
func (w *Watchdog) probe(ctx context.Context, vmid int) Probe {
p := Probe{VMID: vmid, Mode: ModeUnknown}
id := itoa(vmid)
// 1. Address. This also settles reachability: if pct exec cannot run here, nothing else is
// worth attempting.
out, errOut, err := w.runner.Run(ctx, "pct", "exec", id, "--", "ip", "-4", "-o", "addr", "show", "dev", eth0)
if err != nil {
p.Detail = "address probe failed: " + firstLine(string(errOut))
return p // Reachable stays false → StateUnknown
}
p.Reachable = true
p.IP = parseInet(string(out))
// 2. Default route.
out, _, err = w.runner.Run(ctx, "pct", "exec", id, "--", "ip", "route", "show", "default")
if err == nil {
p.HasRoute = hasDefaultRoute(string(out))
}
// 3. Configured mode. An unreadable interfaces file leaves ModeUnknown, which never heals.
out, _, err = w.runner.Run(ctx, "pct", "exec", id, "--", "cat", "/etc/network/interfaces")
if err == nil {
p.Mode = parseMode(string(out), eth0)
}
// 4. DHCP client liveness — the probe the incident was invisible to.
// pgrep exits 1 with EMPTY stderr when there is no match; anything on stderr means the probe
// itself failed (pgrep absent, guest wedged), which must read as unknown rather than as a
// dead client, or a missing tool would trigger heals forever.
out, errOut, err = w.runner.Run(ctx, "pct", "exec", id, "--", "pgrep", "-x", "dhclient")
switch {
case err == nil && strings.TrimSpace(string(out)) != "":
p.DHCPAlive = true
case err != nil && strings.TrimSpace(string(errOut)) != "":
p.Reachable = false
p.Detail = "dhclient liveness probe failed: " + firstLine(string(errOut))
default:
p.DHCPAlive = false
}
return p
}
// classify turns observed facts into the verdict. Pure — table-tested.
func classify(p Probe) (State, string) {
if !p.Reachable {
d := p.Detail
if d == "" {
d = "guest not reachable via pct exec"
}
return StateUnknown, d
}
switch p.Mode {
case ModeStatic:
if p.IP != "" && p.HasRoute {
return StateHealthy, "static address and default route present"
}
return StateStaticFault, "statically configured guest is missing its address or default route — reported only; dhclient must never be run against a static configuration (R-50 owns the durable fix)"
case ModeDHCP:
switch {
case p.IP == "":
return StateUnhealthy, "no IPv4 address on " + eth0
case !p.HasRoute:
return StateUnhealthy, "no default route"
case !p.DHCPAlive:
// THE incident state: address and route still present on an unexpired lease, with
// nothing left to renew them. Damage is ~1-2 h in the future and invisible today.
return StateUnhealthy, "dhclient is not running — the lease will not be renewed (the 2026-07-20 failure mode; address still present, renewal already dead)"
default:
return StateHealthy, "address, default route and dhclient all present"
}
default:
return StateUnknown, "interface configuration mode could not be determined — not healing"
}
}
// --- parsing (fixtures captured live from guest 9201 on 2026-07-21, probe P3) -------------------
// parseInet extracts the address from `ip -4 -o addr show dev eth0` output, e.g.
//
// 2: eth0 inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0\ valid_lft ...
//
// Returns "" when there is no inet line at all (the post-lease-expiry state: the command succeeds
// and prints NOTHING).
func parseInet(out string) string {
for _, line := range strings.Split(out, "\n") {
fields := strings.Fields(line)
for i, f := range fields {
if f == "inet" && i+1 < len(fields) {
addr := fields[i+1]
if idx := strings.IndexByte(addr, '/'); idx > 0 {
addr = addr[:idx]
}
return addr
}
}
}
return ""
}
// hasDefaultRoute parses `ip route show default`, e.g. "default via 192.168.0.1 dev eth0 ".
// Empty output = no default route (the incident state).
func hasDefaultRoute(out string) bool {
for _, line := range strings.Split(out, "\n") {
if strings.HasPrefix(strings.TrimSpace(line), "default ") {
return true
}
}
return false
}
// parseMode reads the iface stanza for dev out of /etc/network/interfaces:
//
// auto eth0
// iface eth0 inet dhcp
//
// Anything else (no stanza, a manual/loopback mode, a file we could not read) is ModeUnknown, and
// unknown never heals.
func parseMode(out, dev string) Mode {
for _, line := range strings.Split(out, "\n") {
f := strings.Fields(strings.TrimSpace(line))
// iface <dev> inet <mode>
if len(f) >= 4 && f[0] == "iface" && f[1] == dev && f[2] == "inet" {
switch f[3] {
case "dhcp":
return ModeDHCP
case "static":
return ModeStatic
default:
return ModeUnknown
}
}
}
return ModeUnknown
}
func firstLine(s string) string {
s = strings.TrimSpace(s)
if i := strings.IndexByte(s, '\n'); i >= 0 {
s = s[:i]
}
if len(s) > 200 {
s = s[:200]
}
return s
}
// itoa avoids pulling strconv into every call site's readability.
func itoa(i int) string {
if i == 0 {
return "0"
}
neg := i < 0
if neg {
i = -i
}
var b [20]byte
pos := len(b)
for i > 0 {
pos--
b[pos] = byte('0' + i%10)
i /= 10
}
if neg {
pos--
b[pos] = '-'
}
return string(b[pos:])
}
+38
View File
@@ -0,0 +1,38 @@
package guestnet
import (
"context"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// GuestNetStatus implements hub.GuestNetReporter: the heartbeat stanza built from the last sweep.
// Pure read of already-collected state — it never probes, so a hub report can never trigger a pct
// exec storm.
//
// It returns a stanza even when no guest has been probed yet (empty guests + a checked_at), because
// "the watchdog is running and has nothing to say" must be distinguishable on the hub from "the
// watchdog is not wired", which is the shape the v0.91.0 inert seam hid behind.
func (w *Watchdog) GuestNetStatus(context.Context) *hub.GuestNetStatus {
snap := w.Snapshot()
out := &hub.GuestNetStatus{CheckedAt: w.now().UTC().Format(time.RFC3339)}
for _, g := range snap {
out.Guests = append(out.Guests, hub.GuestNetGuest{
VMID: g.VMID,
State: g.State,
Mode: g.Mode,
IP: g.IP,
HasRoute: g.HasRoute,
DHClientAlive: g.DHClientAlive,
CheckedAt: g.CheckedAt,
Healed: g.Healed,
HealSucceeded: g.HealSucceeded,
LastHealAt: g.LastHealAt,
HealsLastHour: g.HealsLastHour,
Damped: g.Damped,
Message: g.Message,
})
}
return out
}
+377
View File
@@ -0,0 +1,377 @@
package guestnet
import (
"context"
"log/slog"
"sort"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// GuestSource yields the guests this agent OWNS. Production passes the pool-verified source
// (ListLXC ∩ pool membership, audit A1) — never a bare ListLXC, which under a broad token would let
// the watchdog run dhclient inside a co-tenant's container.
type GuestSource interface {
Guests(ctx context.Context) ([]proxmox.Guest, error)
}
// Defaults. Every one of these is a damper: this watchdog runs a privileged command inside a
// customer's container, so it is designed to under-act.
const (
DefaultInterval = 60 * time.Second
// DefaultMinHealInterval is the per-guest cool-off between heals.
DefaultMinHealInterval = 10 * time.Minute
// DefaultMaxHealsPerHour caps a guest's heals; beyond it the watchdog only reports, because a
// guest needing a fourth heal in an hour has a problem dhclient cannot fix.
DefaultMaxHealsPerHour = 3
// DefaultSettle is the boot-race guard, applied to BOTH the guest's uptime and the agent's own.
// A guest that booted 40 s ago legitimately has no lease yet.
DefaultSettle = 3 * time.Minute
// requiredBadProbes: two CONSECUTIVE bad cycles before any heal. One blip is not a diagnosis.
requiredBadProbes = 2
)
// Watchdog probes each owned, running guest's network every interval and heals a DHCP guest whose
// client has died. It never returns an error: a guest-network fault is a reported fact, not an agent
// failure.
type Watchdog struct {
runner Runner
guests GuestSource
logger *slog.Logger
interval time.Duration
minHealInterval time.Duration
maxHealsPerHour int
settle time.Duration
now func() time.Time
startedAt time.Time
mu sync.Mutex
state map[int]*guestState
}
type guestState struct {
badProbes int
heals []time.Time // heal timestamps, pruned to the last hour
lastHealAt time.Time
lastState State
report GuestReport
}
// New builds a Watchdog with the shipped dampers. interval <= 0 uses DefaultInterval.
func New(runner Runner, guests GuestSource, interval time.Duration, logger *slog.Logger) *Watchdog {
if interval <= 0 {
interval = DefaultInterval
}
if logger == nil {
logger = slog.Default()
}
w := &Watchdog{
runner: runner,
guests: guests,
logger: logger,
interval: interval,
minHealInterval: DefaultMinHealInterval,
maxHealsPerHour: DefaultMaxHealsPerHour,
settle: DefaultSettle,
now: time.Now,
state: map[int]*guestState{},
}
w.startedAt = w.now()
return w
}
// SetDampers overrides the three rate limits from config. Non-positive values keep the default,
// the same "0 = package default" convention the storage watchdog and wg loop use.
func (w *Watchdog) SetDampers(minHealInterval time.Duration, maxHealsPerHour int, settle time.Duration) {
w.mu.Lock()
defer w.mu.Unlock()
if minHealInterval > 0 {
w.minHealInterval = minHealInterval
}
if maxHealsPerHour > 0 {
w.maxHealsPerHour = maxHealsPerHour
}
if settle > 0 {
w.settle = settle
}
}
// Watch runs until ctx is cancelled. Started with `go wd.Watch(ctx)` — deliberately not part of the
// errc fan-out, because a guest-network watchdog must never be able to bring the agent down.
func (w *Watchdog) Watch(ctx context.Context) {
w.logger.Info("guestnet: watchdog starting",
"interval", w.interval, "min_heal_interval", w.minHealInterval,
"max_heals_per_hour", w.maxHealsPerHour, "settle", w.settle)
t := time.NewTicker(w.interval)
defer t.Stop()
w.Tick(ctx) // immediate baseline
for {
select {
case <-ctx.Done():
w.logger.Info("guestnet: watchdog shutting down", "reason", ctx.Err())
return
case <-t.C:
w.Tick(ctx)
}
}
}
// Tick performs one full sweep. Exported so the wiring test and the live STOP leg can drive exactly
// one cycle instead of waiting on a ticker.
func (w *Watchdog) Tick(ctx context.Context) {
guests, err := w.guests.Guests(ctx)
if err != nil {
// Unknown ownership ⇒ do nothing. Never fall back to an unfiltered guest list.
w.logger.Warn("guestnet: guest list unavailable — skipping sweep (ownership unproven)", "err", err)
return
}
for _, g := range guests {
if ctx.Err() != nil {
return
}
if g.Status != "running" {
w.forget(g.VMID)
continue
}
w.checkGuest(ctx, g)
}
}
// forget drops state for a guest that is no longer running, so a stopped-and-restarted guest starts
// from a clean slate rather than inheriting a stale bad-probe count.
func (w *Watchdog) forget(vmid int) {
w.mu.Lock()
delete(w.state, vmid)
w.mu.Unlock()
}
func (w *Watchdog) checkGuest(ctx context.Context, g proxmox.Guest) {
now := w.now()
p := w.probe(ctx, g.VMID)
state, detail := classify(p)
w.mu.Lock()
st := w.state[g.VMID]
if st == nil {
st = &guestState{}
w.state[g.VMID] = st
}
prev := st.lastState
st.lastState = state
st.pruneHeals(now)
rep := GuestReport{
VMID: g.VMID, Mode: string(p.Mode), IP: p.IP, HasRoute: p.HasRoute,
DHClientAlive: p.DHCPAlive, State: string(state), Message: detail,
HealsLastHour: len(st.heals), CheckedAt: now.UTC().Format(time.RFC3339),
}
if !st.lastHealAt.IsZero() {
rep.LastHealAt = st.lastHealAt.UTC().Format(time.RFC3339)
}
switch state {
case StateHealthy:
st.badProbes = 0
w.mu.Unlock()
// The healthy path MUST be observable. v0.91.2's lesson, learned the hard way one day
// earlier: if a healthy cycle logs nothing, "no alarms" and "never probed" are the same
// line of evidence, and an inert watchdog is indistinguishable from a working one.
w.logger.Debug("guestnet: guest network healthy", "vmid", g.VMID, "mode", string(p.Mode),
"has_route", p.HasRoute, "dhclient_alive", p.DHCPAlive)
if prev != "" && prev != StateHealthy {
w.logger.Info("guestnet: guest network recovered", "vmid", g.VMID, "previous_state", string(prev))
}
w.record(g.VMID, rep)
return
case StateUnknown, StateStaticFault:
st.badProbes = 0 // neither is a dhclient fault; don't accumulate toward a heal
w.mu.Unlock()
if prev != state { // loud once per transition, not once per minute
w.logger.Warn("guestnet: guest network not actionable — reporting only",
"vmid", g.VMID, "state", string(state), "mode", string(p.Mode),
"has_ip", p.IP != "", "has_route", p.HasRoute, "detail", detail)
}
w.record(g.VMID, rep)
return
}
// --- StateUnhealthy: a DHCP guest with something dhclient can fix ---------------------------
st.badProbes++
bad := st.badProbes
lastHeal := st.lastHealAt
healsInHour := len(st.heals)
w.mu.Unlock()
if reason, ok := w.observeOnly(g, now); !ok {
rep.Damped = true
rep.Message = detail + " — observing only: " + reason
w.logger.Info("guestnet: guest network unhealthy but not acting", "vmid", g.VMID,
"reason", reason, "detail", detail)
w.record(g.VMID, rep)
return
}
if bad < requiredBadProbes {
rep.Message = detail + " — awaiting a second consecutive bad probe before healing"
w.logger.Info("guestnet: guest network unhealthy (first bad probe — not acting yet)",
"vmid", g.VMID, "detail", detail, "bad_probes", bad, "required", requiredBadProbes)
w.record(g.VMID, rep)
return
}
if damped, reason := w.damped(lastHeal, healsInHour, now); damped {
rep.Damped = true
rep.Message = detail + " — heal damped: " + reason
w.logger.Warn("guestnet: guest network unhealthy but healing is DAMPED — reporting only",
"vmid", g.VMID, "reason", reason, "heals_last_hour", healsInHour, "detail", detail)
w.record(g.VMID, rep)
return
}
// --- heal ----------------------------------------------------------------------------------
w.logger.Warn("guestnet: guest network unhealthy — healing",
"vmid", g.VMID, "detail", detail, "bad_probes", bad)
healed, healErr := w.heal(ctx, g.VMID)
w.mu.Lock()
st = w.state[g.VMID]
if st != nil {
st.heals = append(st.heals, now)
st.lastHealAt = now
st.badProbes = 0 // the post-heal probe below is the new evidence
healsInHour = len(st.heals)
}
w.mu.Unlock()
after := w.probe(ctx, g.VMID)
afterState, afterDetail := classify(after)
rep = GuestReport{
VMID: g.VMID, Mode: string(after.Mode), IP: after.IP, HasRoute: after.HasRoute,
DHClientAlive: after.DHCPAlive, State: string(afterState), Message: afterDetail,
Healed: true, HealSucceeded: afterState == StateHealthy,
LastHealAt: now.UTC().Format(time.RFC3339), HealsLastHour: healsInHour,
CheckedAt: w.now().UTC().Format(time.RFC3339),
}
if healErr != nil {
rep.Message = "heal command failed: " + healErr.Error() + "; " + afterDetail
}
if afterState == StateHealthy {
w.logger.Info("guestnet: guest network healed", "vmid", g.VMID, "ip", after.IP,
"has_route", after.HasRoute, "dhclient_alive", after.DHCPAlive, "heals_last_hour", healsInHour)
} else {
w.logger.Error("guestnet: heal did not restore the guest network", "vmid", g.VMID,
"state", string(afterState), "detail", afterDetail, "heal_ran", healed, "err", healErr)
}
w.mu.Lock()
if st = w.state[g.VMID]; st != nil {
st.lastState = afterState
}
w.mu.Unlock()
w.record(g.VMID, rep)
}
// heal runs the incident's restored invocation, VERBATIM (INCIDENT-guest-dhclient-killed-2026-07-20
// §5) — the same argv that brought guest 9201 back at 10:04:3x UTC. Fixed shape, no shell, no guest
// data interpolated. Logged at INFO before it runs so the operator sees the exact command.
func (w *Watchdog) heal(ctx context.Context, vmid int) (bool, error) {
args := []string{"exec", itoa(vmid), "--", "dhclient",
"-pf", "/run/dhclient." + eth0 + ".pid",
"-lf", "/var/lib/dhcp/dhclient." + eth0 + ".leases", eth0}
w.logger.Info("guestnet: running heal command", "vmid", vmid, "cmd", "pct "+joinArgs(args))
_, errOut, err := w.runner.Run(ctx, "pct", args...)
if err != nil {
w.logger.Error("guestnet: heal command failed", "vmid", vmid, "stderr", firstLine(string(errOut)), "err", err)
return false, err
}
return true, nil
}
// observeOnly reports whether a boot race means this cycle must look and not touch.
func (w *Watchdog) observeOnly(g proxmox.Guest, now time.Time) (string, bool) {
if now.Sub(w.startedAt) < w.settle {
return "agent started less than " + w.settle.String() + " ago", false
}
if g.Uptime > 0 && time.Duration(g.Uptime)*time.Second < w.settle {
return "guest has been up for less than " + w.settle.String(), false
}
return "", true
}
// damped applies the two rate limits.
func (w *Watchdog) damped(lastHeal time.Time, healsInHour int, now time.Time) (bool, string) {
if !lastHeal.IsZero() && now.Sub(lastHeal) < w.minHealInterval {
return true, "last heal was less than " + w.minHealInterval.String() + " ago"
}
if healsInHour >= w.maxHealsPerHour {
return true, "heal budget for the hour is spent (a guest needing more than this has a fault dhclient cannot fix)"
}
return false, ""
}
func (s *guestState) pruneHeals(now time.Time) {
kept := s.heals[:0]
for _, t := range s.heals {
if now.Sub(t) < time.Hour {
kept = append(kept, t)
}
}
s.heals = kept
}
func (w *Watchdog) record(vmid int, rep GuestReport) {
w.mu.Lock()
defer w.mu.Unlock()
if st := w.state[vmid]; st != nil {
st.report = rep
}
}
func joinArgs(args []string) string {
out := ""
for i, a := range args {
if i > 0 {
out += " "
}
out += a
}
return out
}
// --- the hub report block ------------------------------------------------------------------------
// GuestReport is one guest's last observed network state, mirrored to the hub.
type GuestReport struct {
VMID int `json:"vmid"`
State string `json:"state"` // healthy | unhealthy | static_fault | unknown
Mode string `json:"mode"` // dhcp | static | unknown
IP string `json:"ip,omitempty"`
HasRoute bool `json:"has_route"`
DHClientAlive bool `json:"dhclient_alive"`
CheckedAt string `json:"checked_at,omitempty"`
Healed bool `json:"healed,omitempty"` // a heal ran on THIS cycle
HealSucceeded bool `json:"heal_succeeded,omitempty"` // and the re-probe came back healthy
LastHealAt string `json:"last_heal_at,omitempty"`
HealsLastHour int `json:"heals_last_hour,omitempty"`
Damped bool `json:"damped,omitempty"`
Message string `json:"message,omitempty"`
}
// Snapshot returns the per-guest blocks for the heartbeat, VMID-sorted for a stable wire shape.
func (w *Watchdog) Snapshot() []GuestReport {
w.mu.Lock()
defer w.mu.Unlock()
out := make([]GuestReport, 0, len(w.state))
for _, st := range w.state {
if st.report.VMID != 0 {
out = append(out, st.report)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].VMID < out[j].VMID })
return out
}
+568
View File
@@ -0,0 +1,568 @@
package guestnet
import (
"bytes"
"context"
"errors"
"io"
"log/slog"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// --- fixtures captured LIVE from guest 9201 on 2026-07-21 (probe P3, via `ssh felhom-pve`) -------
//
// These are the byte shapes the parser must survive; note the literal backslash `ip -o` emits and
// the trailing space on the route line.
const (
fxAddr = "2: eth0 inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0\\ valid_lft 4916sec preferred_lft 4916sec\n"
fxRoute = "default via 192.168.0.1 dev eth0 \n"
fxPgrep = "235839\n"
fxIfacesDHCP = "auto lo\niface lo inet loopback\n\nauto eth0\niface eth0 inet dhcp\n"
fxIfacesStat = "auto lo\niface lo inet loopback\n\nauto eth0\niface eth0 inet static\n\taddress 192.168.0.162/24\n\tgateway 192.168.0.1\n"
// pct exec against a guest that does not exist / is not running (rc=2, message on stderr).
fxNoGuestErr = "Configuration file 'nodes/demo-felhom/lxc/9999.conf' does not exist\n"
)
// scriptedRunner answers per probe kind and records EVERY argv. The counts are the assertions that
// matter: a watchdog that heals when it must not is worse than one that never heals.
type scriptedRunner struct {
mu sync.Mutex
out map[string]string // kind → stdout
fail map[string]error // kind → error
errs map[string]string // kind → stderr
call [][]string
// healFixes models what a successful dhclient actually does: the client is running again and
// the lease is renewed. Set false to model a guest whose network is broken beyond dhclient.
healFixes bool
}
func newRunner() *scriptedRunner {
return &scriptedRunner{
out: map[string]string{
"addr": fxAddr, "route": fxRoute, "iface": fxIfacesDHCP, "pgrep": fxPgrep, "dhclient": "",
},
fail: map[string]error{},
errs: map[string]string{},
healFixes: true,
}
}
// kind classifies a `pct exec <vmid> -- <cmd> ...` argv.
func kind(args []string) string {
if len(args) < 4 || args[0] != "exec" {
return "other"
}
rest := args[3:]
switch rest[0] {
case "ip":
if len(rest) > 1 && rest[1] == "route" {
return "route"
}
return "addr"
case "cat":
return "iface"
case "pgrep":
return "pgrep"
case "dhclient":
return "dhclient"
}
return "other"
}
func (r *scriptedRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
r.call = append(r.call, append([]string{name}, args...))
k := kind(args)
if k == "dhclient" && r.fail[k] == nil && r.healFixes {
// a real dhclient re-acquires the lease and stays resident
r.out["pgrep"], r.out["addr"], r.out["route"] = fxPgrep, fxAddr, fxRoute
delete(r.fail, "pgrep")
}
return []byte(r.out[k]), []byte(r.errs[k]), r.fail[k]
}
func (r *scriptedRunner) countOf(k string) int {
r.mu.Lock()
defer r.mu.Unlock()
n := 0
for _, c := range r.call {
if len(c) > 1 && kind(c[1:]) == k {
n++
}
}
return n
}
func (r *scriptedRunner) lastOf(k string) []string {
r.mu.Lock()
defer r.mu.Unlock()
for i := len(r.call) - 1; i >= 0; i-- {
if len(r.call[i]) > 1 && kind(r.call[i][1:]) == k {
return r.call[i]
}
}
return nil
}
// killDHClient models the exact 2026-07-20 state: lease still valid (address AND route present),
// dhclient gone. This is the fixture the whole feature exists for.
func (r *scriptedRunner) killDHClient() {
r.mu.Lock()
defer r.mu.Unlock()
r.out["pgrep"] = ""
r.fail["pgrep"] = errors.New("exit status 1") // pgrep: no match, EMPTY stderr
}
func (r *scriptedRunner) reviveDHClient() {
r.mu.Lock()
defer r.mu.Unlock()
r.out["pgrep"] = fxPgrep
delete(r.fail, "pgrep")
}
// expireLease models the state 80 minutes later: address and route gone too.
func (r *scriptedRunner) expireLease() {
r.mu.Lock()
defer r.mu.Unlock()
r.out["addr"] = ""
r.out["route"] = ""
}
type fakeGuests struct {
guests []proxmox.Guest
err error
}
func (f *fakeGuests) Guests(context.Context) ([]proxmox.Guest, error) { return f.guests, f.err }
func running9201() *fakeGuests {
return &fakeGuests{guests: []proxmox.Guest{
{VMID: 9201, Name: "felhom-demo", Status: "running", Type: "lxc", Uptime: 7200},
}}
}
// newTestWatchdog wires a watchdog with a manual clock the test advances, and captures the log so
// the "healthy cycles are observable" contract can be asserted rather than assumed.
func newTestWatchdog(r Runner, g GuestSource) (*Watchdog, *time.Time, *bytes.Buffer) {
clock := time.Unix(1_784_000_000, 0).UTC()
buf := &bytes.Buffer{}
logger := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
w := New(r, g, time.Minute, logger)
w.now = func() time.Time { return clock }
// The agent's own settle window is measured from startedAt, which New stamped with the REAL
// clock; restamp it against the fake one, well in the past.
w.startedAt = clock.Add(-time.Hour)
return w, &clock, buf
}
// --- Scenario E: the TIMED failure, detected instantly ------------------------------------------
//
// RED-PROOF (recorded in REPORT.md): reverting classify()'s dhcp arm to IP-presence-only —
//
// case ModeDHCP:
// if p.IP == "" { return StateUnhealthy, ... }
// return StateHealthy, ...
//
// makes TestProbe_DeadDHClientWithLiveLeaseIsUnhealthy report "healthy" for the July-20 fixture, and
// TestWatchdog_HealsTheIncidentState records ZERO heals. That is the 80-minute silent window, exactly
// as it happened.
func TestProbe_DeadDHClientWithLiveLeaseIsUnhealthy(t *testing.T) {
r := newRunner()
r.killDHClient()
w, _, _ := newTestWatchdog(r, running9201())
p := w.probe(context.Background(), 9201)
if !p.Reachable {
t.Fatalf("guest must read as reachable: %+v", p)
}
if p.IP != "192.168.0.104" || !p.HasRoute {
t.Fatalf("the lease is still live in this fixture — IP/route must be present: %+v", p)
}
if p.DHCPAlive {
t.Fatalf("dhclient must read as dead: %+v", p)
}
state, detail := classify(p)
if state != StateUnhealthy {
t.Fatalf("classify = %q, want %q — waiting for the IP to vanish is the 80-minute silent "+
"window the incident proved (detail: %s)", state, StateUnhealthy, detail)
}
if !strings.Contains(detail, "dhclient") {
t.Fatalf("the reason must name the dead client, got %q", detail)
}
}
func TestWatchdog_HealsTheIncidentState(t *testing.T) {
r := newRunner()
r.killDHClient()
w, clock, logBuf := newTestWatchdog(r, running9201())
ctx := context.Background()
// Cycle 1: unhealthy, but one bad probe is not a diagnosis.
w.Tick(ctx)
if n := r.countOf("dhclient"); n != 0 {
t.Fatalf("healed after ONE bad probe (%d heals) — a single blip must never trigger a heal", n)
}
// Cycle 2: second consecutive bad probe → heal. The heal makes the client live again.
*clock = clock.Add(time.Minute)
w.Tick(ctx)
if n := r.countOf("dhclient"); n != 1 {
t.Fatalf("heal ran %d times, want exactly 1", n)
}
// The invocation must be the incident's, verbatim.
want := []string{"pct", "exec", "9201", "--", "dhclient",
"-pf", "/run/dhclient.eth0.pid", "-lf", "/var/lib/dhcp/dhclient.eth0.leases", "eth0"}
got := r.lastOf("dhclient")
if len(got) != len(want) {
t.Fatalf("heal argv = %v, want %v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("heal argv[%d] = %q, want %q (full: %v)", i, got[i], want[i], got)
}
}
// The report must show the heal AND the verified-healthy re-probe.
snap := w.Snapshot()
if len(snap) != 1 {
t.Fatalf("snapshot = %+v, want one guest", snap)
}
g := snap[0]
if !g.Healed || !g.HealSucceeded {
t.Fatalf("report must record a successful heal: %+v", g)
}
if g.State != string(StateHealthy) || g.LastHealAt == "" || g.HealsLastHour != 1 {
t.Fatalf("post-heal report is wrong: %+v", g)
}
if !strings.Contains(logBuf.String(), "guestnet: guest network healed") {
t.Fatalf("the heal was not logged: %s", logBuf.String())
}
}
// A healthy box must be silent about alarms but NOT silent about having looked.
func TestWatchdog_HealthyCycleProbesAndNeverHeals(t *testing.T) {
r := newRunner()
w, _, logBuf := newTestWatchdog(r, running9201())
w.Tick(context.Background())
if n := r.countOf("dhclient"); n != 0 {
t.Fatalf("a healthy guest was healed %d times, want 0", n)
}
if r.countOf("pgrep") != 1 || r.countOf("addr") != 1 {
t.Fatalf("the healthy path must still probe: %v", r.call)
}
if !strings.Contains(logBuf.String(), "guest network healthy") {
t.Fatalf("a healthy cycle must be observable — otherwise 'no alarms' and 'never probed' "+
"are the same evidence (v0.91.2's lesson). Log: %s", logBuf.String())
}
snap := w.Snapshot()
if len(snap) != 1 || snap[0].State != string(StateHealthy) || !snap[0].DHClientAlive {
t.Fatalf("healthy snapshot wrong: %+v", snap)
}
}
// --- Scenario F: configuration and damping guards ------------------------------------------------
func TestWatchdog_StaticGuestIsNeverHealedWithDHClient(t *testing.T) {
r := newRunner()
r.out["iface"] = fxIfacesStat
r.killDHClient() // on a static guest this is NORMAL
w, clock, _ := newTestWatchdog(r, running9201())
for i := 0; i < 5; i++ {
w.Tick(context.Background())
*clock = clock.Add(time.Minute)
}
if n := r.countOf("dhclient"); n != 0 {
t.Fatalf("a static guest was healed with dhclient %d times, want 0", n)
}
if s := w.Snapshot(); len(s) != 1 || s[0].State != string(StateHealthy) {
t.Fatalf("a static guest with address+route is healthy, got %+v", s)
}
}
func TestWatchdog_StaticGuestMissingAddressReportsButNeverHeals(t *testing.T) {
r := newRunner()
r.out["iface"] = fxIfacesStat
r.expireLease() // no address, no route on a STATIC guest → R-50 territory, not ours
w, clock, logBuf := newTestWatchdog(r, running9201())
for i := 0; i < 5; i++ {
w.Tick(context.Background())
*clock = clock.Add(time.Minute)
}
if n := r.countOf("dhclient"); n != 0 {
t.Fatalf("healed a static guest %d times, want 0 — dhclient must never fight a static config", n)
}
s := w.Snapshot()
if len(s) != 1 || s[0].State != string(StateStaticFault) {
t.Fatalf("state = %+v, want static_fault", s)
}
if !strings.Contains(logBuf.String(), "not actionable") {
t.Fatalf("a static fault must be reported loudly: %s", logBuf.String())
}
// Loud ONCE per transition, not once per cycle.
if n := strings.Count(logBuf.String(), "not actionable"); n != 1 {
t.Fatalf("static fault logged %d times over 5 cycles, want 1 (per transition)", n)
}
}
func TestWatchdog_UnreachableGuestIsUnknownAndNeverHealed(t *testing.T) {
r := newRunner()
r.fail["addr"] = errors.New("exit status 2")
r.errs["addr"] = fxNoGuestErr
w, clock, _ := newTestWatchdog(r, running9201())
for i := 0; i < 4; i++ {
w.Tick(context.Background())
*clock = clock.Add(time.Minute)
}
if n := r.countOf("dhclient"); n != 0 {
t.Fatalf("healed a guest we could not probe %d times, want 0 — acting blind is how the "+
"incident happened", n)
}
if s := w.Snapshot(); len(s) != 1 || s[0].State != string(StateUnknown) {
t.Fatalf("state = %+v, want unknown", s)
}
}
// A missing pgrep (or any probe tool) must read as unknown, never as a dead client — otherwise a
// broken probe would heal forever.
func TestWatchdog_ProbeToolFailureIsUnknownNotDead(t *testing.T) {
r := newRunner()
r.fail["pgrep"] = errors.New("exit status 127")
r.errs["pgrep"] = "pgrep: command not found\n"
w, clock, _ := newTestWatchdog(r, running9201())
for i := 0; i < 4; i++ {
w.Tick(context.Background())
*clock = clock.Add(time.Minute)
}
if n := r.countOf("dhclient"); n != 0 {
t.Fatalf("a failed liveness probe caused %d heals, want 0", n)
}
if s := w.Snapshot(); len(s) != 1 || s[0].State != string(StateUnknown) {
t.Fatalf("state = %+v, want unknown", s)
}
}
func TestWatchdog_DampingCeilings(t *testing.T) {
r := newRunner()
r.killDHClient()
r.healFixes = false // the heal "works" but the client dies again immediately
w, clock, _ := newTestWatchdog(r, running9201())
ctx := context.Background()
// 10 hours of one-minute cycles against a permanently broken guest.
for i := 0; i < 600; i++ {
w.Tick(ctx)
*clock = clock.Add(time.Minute)
}
heals := r.countOf("dhclient")
// Ceiling: 3 per hour AND ≥10 min apart ⇒ at most 3 in any rolling hour. Over 10 h the
// min-interval rule dominates: 6 slots/hour capped to 3/hour ⇒ ≤ 30.
if heals > 30 {
t.Fatalf("heals = %d over 10 h, want ≤ 30 (≤3/hour) — the damper is not holding", heals)
}
if heals == 0 {
t.Fatalf("heals = 0 — the damper has become a mute")
}
}
func TestWatchdog_MinimumIntervalBetweenHeals(t *testing.T) {
r := newRunner()
r.killDHClient()
r.healFixes = false
w, clock, _ := newTestWatchdog(r, running9201())
ctx := context.Background()
w.Tick(ctx) // bad probe 1
*clock = clock.Add(time.Minute)
w.Tick(ctx) // bad probe 2 → heal #1
if r.countOf("dhclient") != 1 {
t.Fatalf("expected exactly one heal by now, got %d", r.countOf("dhclient"))
}
// Nine more minutes of failure: still inside the 10-minute cool-off.
for i := 0; i < 9; i++ {
*clock = clock.Add(time.Minute)
w.Tick(ctx)
}
if n := r.countOf("dhclient"); n != 1 {
t.Fatalf("heals = %d within the 10-minute cool-off, want 1", n)
}
if s := w.Snapshot(); len(s) != 1 || !s[0].Damped {
t.Fatalf("a damped cycle must say so in the report: %+v", s)
}
// Past the cool-off, one more heal is allowed.
*clock = clock.Add(2 * time.Minute)
w.Tick(ctx)
if n := r.countOf("dhclient"); n != 2 {
t.Fatalf("heals = %d after the cool-off expired, want 2", n)
}
}
func TestWatchdog_BootRacesObserveOnly(t *testing.T) {
t.Run("young guest", func(t *testing.T) {
r := newRunner()
r.killDHClient()
g := running9201()
g.guests[0].Uptime = 40 // seconds
w, clock, _ := newTestWatchdog(r, g)
for i := 0; i < 4; i++ {
w.Tick(context.Background())
*clock = clock.Add(time.Minute)
}
if n := r.countOf("dhclient"); n != 0 {
t.Fatalf("healed a guest that booted 40 s ago %d times, want 0 — it has no lease YET", n)
}
})
t.Run("young agent", func(t *testing.T) {
r := newRunner()
r.killDHClient()
w, clock, _ := newTestWatchdog(r, running9201())
w.startedAt = *clock // the agent just started
for i := 0; i < 2; i++ {
w.Tick(context.Background())
*clock = clock.Add(time.Minute)
}
if n := r.countOf("dhclient"); n != 0 {
t.Fatalf("healed %d times within the agent's own settle window, want 0", n)
}
})
}
func TestWatchdog_StoppedGuestIsNotProbed(t *testing.T) {
r := newRunner()
g := running9201()
g.guests[0].Status = "stopped"
w, _, _ := newTestWatchdog(r, g)
w.Tick(context.Background())
if len(r.call) != 0 {
t.Fatalf("a stopped guest was probed: %v", r.call)
}
}
func TestWatchdog_GuestListFailureSkipsTheSweep(t *testing.T) {
r := newRunner()
w, _, logBuf := newTestWatchdog(r, &fakeGuests{err: errors.New("pool membership read: 403")})
w.Tick(context.Background())
if len(r.call) != 0 {
t.Fatalf("acted with unproven ownership: %v", r.call)
}
if !strings.Contains(logBuf.String(), "ownership unproven") {
t.Fatalf("the skip must be logged: %s", logBuf.String())
}
}
// A transient bad probe followed by recovery must never heal, and must log the recovery.
func TestWatchdog_SingleBlipNeverHeals(t *testing.T) {
r := newRunner()
w, clock, logBuf := newTestWatchdog(r, running9201())
ctx := context.Background()
w.Tick(ctx) // healthy
r.killDHClient()
*clock = clock.Add(time.Minute)
w.Tick(ctx) // bad probe 1
r.reviveDHClient()
*clock = clock.Add(time.Minute)
w.Tick(ctx) // healthy again
if n := r.countOf("dhclient"); n != 0 {
t.Fatalf("a single blip caused %d heals, want 0", n)
}
if !strings.Contains(logBuf.String(), "guest network recovered") {
t.Fatalf("the recovery must be visible: %s", logBuf.String())
}
}
// --- parsers over the live P3 fixtures ------------------------------------------------------------
func TestParsers_OverLiveFixtures(t *testing.T) {
if got := parseInet(fxAddr); got != "192.168.0.104" {
t.Fatalf("parseInet = %q, want 192.168.0.104", got)
}
if got := parseInet(""); got != "" {
t.Fatalf("parseInet(empty) = %q, want empty (the post-expiry state prints nothing)", got)
}
if !hasDefaultRoute(fxRoute) {
t.Fatalf("hasDefaultRoute(%q) = false", fxRoute)
}
if hasDefaultRoute("") || hasDefaultRoute("172.17.0.0/16 dev docker0 proto kernel scope link\n") {
t.Fatal("docker bridge routes must not read as a default route (the incident's exact leftovers)")
}
if got := parseMode(fxIfacesDHCP, "eth0"); got != ModeDHCP {
t.Fatalf("parseMode(dhcp) = %q", got)
}
if got := parseMode(fxIfacesStat, "eth0"); got != ModeStatic {
t.Fatalf("parseMode(static) = %q", got)
}
if got := parseMode("auto lo\niface lo inet loopback\n", "eth0"); got != ModeUnknown {
t.Fatalf("parseMode(no eth0 stanza) = %q, want unknown", got)
}
if got := parseMode("iface eth0 inet manual\n", "eth0"); got != ModeUnknown {
t.Fatalf("parseMode(manual) = %q, want unknown", got)
}
}
func TestClassify_Table(t *testing.T) {
cases := []struct {
name string
p Probe
want State
}{
{"dhcp all green", Probe{Reachable: true, Mode: ModeDHCP, IP: "1.2.3.4", HasRoute: true, DHCPAlive: true}, StateHealthy},
{"dhcp dead client, live lease", Probe{Reachable: true, Mode: ModeDHCP, IP: "1.2.3.4", HasRoute: true}, StateUnhealthy},
{"dhcp no address", Probe{Reachable: true, Mode: ModeDHCP, DHCPAlive: true}, StateUnhealthy},
{"dhcp no route", Probe{Reachable: true, Mode: ModeDHCP, IP: "1.2.3.4", DHCPAlive: true}, StateUnhealthy},
{"static green", Probe{Reachable: true, Mode: ModeStatic, IP: "1.2.3.4", HasRoute: true}, StateHealthy},
{"static no client is normal", Probe{Reachable: true, Mode: ModeStatic, IP: "1.2.3.4", HasRoute: true}, StateHealthy},
{"static broken", Probe{Reachable: true, Mode: ModeStatic}, StateStaticFault},
{"unreachable", Probe{Mode: ModeDHCP}, StateUnknown},
{"unknown mode", Probe{Reachable: true, Mode: ModeUnknown, IP: "1.2.3.4", HasRoute: true}, StateUnknown},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got, _ := classify(tc.p); got != tc.want {
t.Fatalf("classify = %q, want %q", got, tc.want)
}
})
}
}
var _ io.Writer = (*bytes.Buffer)(nil)
// A3 (R-50): the guestnet healer is eth0-only and MUST stay blind to the island NIC. A guest on an
// island host presents eth0 DHCP (the LAN leg the healer owns) PLUS eth1 static (the island). Because
// parseMode is interface-scoped, adding eth1 static cannot flip eth0's detected mode — so the healer
// keeps treating eth0 as DHCP and never runs dhclient against the static island NIC (which would
// sabotage it). This is the verify-only guarantee that let R-50 ship the island NIC without a healer
// change. Red-proof: make parseMode scan globally instead of per-dev and the eth0 assertion fails.
func TestParseMode_IslandStaticNICDoesNotConfuseEth0(t *testing.T) {
interfaces := "auto lo\niface lo inet loopback\n\n" +
"auto eth0\niface eth0 inet dhcp\n\n" +
"auto eth1\niface eth1 inet static\n address 169.254.253.2/30\n"
if got := parseMode(interfaces, "eth0"); got != ModeDHCP {
t.Errorf("eth0 must classify DHCP even with an island eth1 static present, got %q", got)
}
if got := parseMode(interfaces, "eth1"); got != ModeStatic {
t.Errorf("eth1 (island) must classify static when asked directly (dev-scoped), got %q", got)
}
}
+93
View File
@@ -144,6 +144,99 @@ func (c *Client) FetchDesiredState(ctx context.Context) (*DesiredStateResponse,
return &out, nil
}
// WGRegisterResponse is the hub's answer to a WG pubkey registration (S3; hub S2
// handleRegisterHostWG). Existed=true = idempotent re-register (nothing moved hub-side).
type WGRegisterResponse struct {
Pubkey string `json:"pubkey"`
AssignedIP string `json:"assigned_ip"` // "10.77.0.2/32"
Existed bool `json:"existed"`
Generation int64 `json:"generation"`
Sync string `json:"sync"` // hub→endpoint push status: ok | deferred:… | disabled | unchanged
}
// RegisterWG registers this host's WG public key with the hub (S3 — doc 06 §3.3 step 2; POST
// /hosts/{host_id}/wg, per-host key, self-scoped server-side). The hub allocates/keeps the /32,
// bumps the desired generation on real change, and pushes the peer to the endpoint. Errors are
// typed (transport vs HTTP: 403 auth, 404 unknown host, 409 conflict/endpoint-unset) and never
// include the bearer token. Only the PUBLIC key ever travels.
func (c *Client) RegisterWG(ctx context.Context, pubkey string) (*WGRegisterResponse, error) {
if c.hostID == "" {
return nil, fmt.Errorf("hub: RegisterWG requires a configured host_id")
}
body, err := json.Marshal(map[string]string{"pubkey": pubkey})
if err != nil {
return nil, fmt.Errorf("hub: marshaling wg registration: %w", err)
}
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/wg"
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("hub: building wg-register request: %w", err)
}
req.Header.Set("Authorization", "Bearer "+c.apiKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
resp, err := c.hc.Do(req)
if err != nil {
return nil, &TransportError{Err: err}
}
defer resp.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
}
var out WGRegisterResponse
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("hub: decoding wg-register response: %w", err)
}
return &out, nil
}
// ErrNoPBSSecret is the typed "404: no unconsumed PBS token secret staged for this host" outcome
// (PBS DR slice 2). Absent-or-already-consumed are indistinguishable by design (consume-once).
var ErrNoPBSSecret = fmt.Errorf("hub: no unconsumed PBS token secret staged for this host")
// ConsumePBSToken fetches this host's one-time PBS token secret — EXACTLY ONCE (PBS DR slice 2;
// POST /api/v1/hosts/{host_id}/pbs/consume-token, per-host key, self-scoped; NOTE the PLURAL
// /hosts/ — the slice-1 route). A 200 burns the secret hub-side: the caller MUST apply it or
// surface a loud consumed-but-failed state (never silent-retry). The secret is returned to the
// caller only — never logged, never in an error.
func (c *Client) ConsumePBSToken(ctx context.Context) (string, error) {
if c.hostID == "" {
return "", fmt.Errorf("hub: ConsumePBSToken requires a configured host_id")
}
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/pbs/consume-token"
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, nil)
if err != nil {
return "", err
}
req.Header.Set("Authorization", "Bearer "+c.apiKey)
req.Header.Set("Accept", "application/json")
resp, err := c.hc.Do(req)
if err != nil {
return "", &TransportError{Err: err}
}
defer resp.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
if resp.StatusCode == http.StatusNotFound {
return "", ErrNoPBSSecret
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
}
var out struct {
TokenSecret string `json:"token_secret"`
}
if err := json.Unmarshal(raw, &out); err != nil {
return "", fmt.Errorf("hub: decoding consume-token response (body withheld — secret channel)")
}
if out.TokenSecret == "" {
return "", fmt.Errorf("hub: consume-token returned an empty secret")
}
return out.TokenSecret, nil
}
// JobWire is one queued signed-op job as served by GET /hosts/{id}/jobs (slice 10A). The blob is
// OPAQUE to the hub — for slice 10B it is a base64 `SignedJobEnvelope` (op-blob + armored SSHSIG)
// the agent verifies before executing.
+201 -4
View File
@@ -2,10 +2,15 @@ package hub
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"log/slog"
"os"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
@@ -48,6 +53,24 @@ type PBSReporter interface {
PBSSnapshots(ctx context.Context) []PBSSnapshot
}
// WireguardReporter is the S3 seam the wgtunnel loop plugs into (same consumer-side pattern —
// hub does not import wgtunnel). nil (feature disabled) → no wireguard stanza on the report.
type WireguardReporter interface {
WireguardStatus(ctx context.Context) *WireguardStatus
}
// PBSDRReporter is the slice-2 seam the pbsdr bridge loop plugs into (same consumer-side
// pattern — hub does not import pbsdr). nil (feature not wired) → no pbs_dr stanza.
type PBSDRReporter interface {
PBSDRStatus(ctx context.Context) *PBSDRStatus
}
// GuestNetReporter is the R-54 seam the guestnet watchdog plugs into (same consumer-side pattern —
// hub does not import guestnet). nil (feature not wired) → no guest_net stanza.
type GuestNetReporter interface {
GuestNetStatus(ctx context.Context) *GuestNetStatus
}
// Collector builds a HostReport from read-only sources. All deps are behind narrow
// interfaces for unit testing.
type Collector struct {
@@ -57,7 +80,17 @@ type Collector struct {
backups BackupReporter
restoreTests RestoreTestReporter
pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
hostID string
agentVersion string
logger *slog.Logger
@@ -92,6 +125,105 @@ func (c *Collector) SetTempReader(t TempReader) *Collector {
return c
}
// SetBackupTargetResolver wires the DR recipe to the agent's own backup config (R-109), so the recipe
// can name WHICH storage holds the local whole-guest archives. Returns the collector for chaining.
//
// The resolver MUST report the tier that is IN EFFECT, which is the daemon-start snapshot — NOT the
// current contents of agent.json. A backup-target move rewrites that file and deliberately does not
// restart the agent (the E-1 lesson: restarting mid-backup records a spurious failure for a run that
// succeeded), so between the write and the restart the file names a target no backup is writing to yet.
// Re-reading the file here — the live-reload shape used for escrow.pbs_storage_id — would make the
// recipe point at the new storage while every archive still landed on the old one. One state, one
// owner: the recipe follows what performs the backup.
func (c *Collector) SetBackupTargetResolver(f func() ConfiguredBackupTarget) *Collector {
c.backupTarget = f
return c
}
// configuredBackupTarget consults the resolver. An unwired seam is reported as NOT KNOWN — never as a
// guess — so the recipe records an explicit unknown instead of a target the agent never verified.
func (c *Collector) configuredBackupTarget() ConfiguredBackupTarget {
if c.backupTarget == nil {
return ConfiguredBackupTarget{}
}
return c.backupTarget()
}
// SetCapabilityProber wires the privileged-capability self-check (v0.44.0): each collect runs it
// and attaches the snapshot. nil → the report carries an empty []. Returns the collector for chaining.
func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capability.Status) *Collector {
c.capProbe = probe
return c
}
// SetLeafFingerprint records the served local-API leaf fp (v0.48.0) to ride every host report (the hub
// watches it for a re-key). Static per process — set once at startup. "" when the local API is
// disabled. Returns the collector for chaining.
func (c *Collector) SetLeafFingerprint(fp string) *Collector {
c.leafFP = fp
return c
}
// SetWireguardReporter wires the offsite-tunnel status source (S3; nil-safe → stanza omitted).
// Returns the collector for chaining.
func (c *Collector) SetWireguardReporter(w WireguardReporter) *Collector {
c.wg = w
return c
}
// SetPBSDRReporter wires the PBS-DR-tier bridge state source (slice 2; nil-safe → stanza
// omitted). Returns the collector for chaining.
func (c *Collector) SetPBSDRReporter(p PBSDRReporter) *Collector {
c.pbsdr = p
return c
}
// SetGuestNetReporter wires the R-54 guest-network watchdog as a report source (nil-safe → stanza
// omitted). Returns the collector for chaining.
func (c *Collector) SetGuestNetReporter(g GuestNetReporter) *Collector {
c.guestNet = g
return c
}
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
type SelfUpdateReporter interface {
// SelfUpdatePending reports whether a signed update has flipped the binary but not yet
// committed, and the awaited version.
SelfUpdatePending() (pending bool, version string)
}
// SetSelfUpdateReporter wires the agent self-update pending-status source (D1; nil-safe → false).
// Returns the collector for chaining.
func (c *Collector) SetSelfUpdateReporter(s SelfUpdateReporter) *Collector {
c.selfUpdate = s
return c
}
// MgmtPlaneReporter is the G1 seam the mgmtplane observer plugs into (same consumer-side pattern —
// hub does not import mgmtplane). nil (feature not wired) → no mgmt_plane stanza on the report.
type MgmtPlaneReporter interface {
MgmtPlaneStatus(ctx context.Context) *MgmtPlaneStatus
}
// SetMgmtPlaneReporter wires the management-plane health source (G1; nil-safe → stanza omitted).
// Returns the collector for chaining.
func (c *Collector) SetMgmtPlaneReporter(m MgmtPlaneReporter) *Collector {
c.mgmtPlane = m
return c
}
// OOBReporter is the H1 seam the felhom-sshd loop plugs into (nil → no oob stanza).
type OOBReporter interface {
OOBStatus(ctx context.Context) *OOBStatus
}
// SetOOBReporter wires the operator-access health source (H1; nil-safe → stanza omitted).
func (c *Collector) SetOOBReporter(o OOBReporter) *Collector {
c.oob = o
return c
}
// Collect builds the report. Best-effort liveness: a failed NodeStatus is a hard
// error (no useful report — the cycle skips the POST); a failed per-guest
// GuestConfig degrades that guest to status="unknown" without spec but still sends;
@@ -103,7 +235,8 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
}
host := hostMetrics(c.px.Node(), ns)
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable
report := &HostReport{
HostID: c.hostID,
ReportedAt: c.now().Format(time.RFC3339),
@@ -117,12 +250,55 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
RestoreTests: c.collectRestoreTests(ctx),
PBSSnapshots: c.collectPBSSnapshots(ctx),
AuditTail: []AuditEntry{},
Cloudflared: Cloudflared{Status: c.cloudflaredStatus(ctx)},
AuditTail: []AuditEntry{},
Cloudflared: Cloudflared{Status: c.cloudflaredStatus(ctx)},
Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP,
Addresses: c.collectAddresses(),
}
// DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads).
// Secret-free by construction (identifiers/intents/sizes/coordinates only).
report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots, c.configuredBackupTarget())
// S3: offsite-tunnel status stanza (nil reporter = feature disabled → omitted; the pubkey in
// it is the operator's revocation-recovery handle).
if c.wg != nil {
report.Wireguard = c.wg.WireguardStatus(ctx)
}
// Slice 2: PBS DR tier bridge state (nil reporter = feature not wired → stanza omitted).
if c.pbsdr != nil {
report.PBSDR = c.pbsdr.PBSDRStatus(ctx)
}
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
if c.guestNet != nil {
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
}
// D1: agent self-update pending status (nil reporter → pending=false, the steady state).
if c.selfUpdate != nil {
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()
}
// G1: management-plane health (nil reporter = feature not wired → stanza omitted).
if c.mgmtPlane != nil {
report.MgmtPlane = c.mgmtPlane.MgmtPlaneStatus(ctx)
}
// H1: operator-access (OOB) health (nil reporter = feature not wired → stanza omitted).
if c.oob != nil {
report.OOB = c.oob.OOBStatus(ctx)
}
return report, nil
}
// capabilities runs the privileged-capability self-check for this report (v0.44.0), or returns an
// empty (non-nil) slice when no prober is wired (dev/test). Never fatal — serve-degraded.
func (c *Collector) capabilities(ctx context.Context) []capability.Status {
if c.capProbe == nil {
return []capability.Status{}
}
if s := c.capProbe(ctx); s != nil {
return s
}
return []capability.Status{}
}
// HostMetricsNow does a FRESH NodeStatus + CPU-temp read and returns just the host block (no
// guests/storage). It is the source for the local API's GET /host/metrics (slice 9) — current
// cpu%/temp, not the 15-min hub-report snapshot. Storage targets come from the observer
@@ -146,6 +322,27 @@ func (c *Collector) cpuTempC(ctx context.Context) *int {
return c.temp.CPUTempC(ctx)
}
// pbsWrapperPath is the installed PBS-DR apply wrapper. Duplicated from internal/pbsdr.WrapperPath
// rather than imported, to keep the report collector free of a dependency on the DR bridge.
const pbsWrapperPath = "/usr/local/sbin/felhom-pbs-apply"
// pbsWrapperSHA256 hashes the installed wrapper for the report (R-50b(a)). Best-effort: a missing or
// unreadable file yields "", which the hub reads as UNKNOWN rather than as drift — a host that
// legitimately has no DR wrapper must not light up amber. The file is 0755, so no privilege is
// needed to read it.
func pbsWrapperSHA256() string {
f, err := os.Open(pbsWrapperPath)
if err != nil {
return ""
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
return ""
}
return hex.EncodeToString(h.Sum(nil))
}
func hostMetrics(node string, ns proxmox.NodeStatus) HostMetrics {
h := HostMetrics{
Node: node,
+80
View File
@@ -0,0 +1,80 @@
package hub
import (
"context"
"encoding/json"
"testing"
)
// R-54 §9 rule 6: the guest_net stanza must appear in a report built through the PRODUCTION collect
// path, not only in a struct a test constructed. The v0.91.0 defect was exactly this gap — a seam
// with green tests and no caller.
type fakeGuestNet struct{ st *GuestNetStatus }
func (f fakeGuestNet) GuestNetStatus(context.Context) *GuestNetStatus { return f.st }
func TestCollect_GuestNetOmittedWhenReporterNil(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.GuestNet != nil {
t.Fatalf("no reporter wired → guest_net must be omitted, got %+v", r.GuestNet)
}
// And it must be absent from the WIRE, not merely nil in Go — an always-present empty stanza
// would make "watchdog not wired" indistinguishable from "watchdog found nothing".
b, _ := json.Marshal(r)
var m map[string]any
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
if _, ok := m["guest_net"]; ok {
t.Fatalf("guest_net key present on the wire with no reporter wired: %s", b)
}
}
func TestCollect_GuestNetPopulatedWhenWired(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
c.SetGuestNetReporter(fakeGuestNet{st: &GuestNetStatus{
CheckedAt: "2026-07-21T10:00:00Z",
Guests: []GuestNetGuest{{
VMID: 9201, State: "healthy", Mode: "dhcp", IP: "192.168.0.104",
HasRoute: true, DHClientAlive: true, CheckedAt: "2026-07-21T10:00:00Z",
}},
}})
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.GuestNet == nil || len(r.GuestNet.Guests) != 1 {
t.Fatalf("guest_net stanza missing from a collected report: %+v", r.GuestNet)
}
// The wire keys are the contract the hub will read; pin the ones an operator diagnoses with.
b, _ := json.Marshal(r)
var m map[string]any
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
gn, ok := m["guest_net"].(map[string]any)
if !ok {
t.Fatalf("guest_net missing or wrong shape on the wire: %s", b)
}
guests, ok := gn["guests"].([]any)
if !ok || len(guests) != 1 {
t.Fatalf("guest_net.guests wrong on the wire: %v", gn)
}
g := guests[0].(map[string]any)
for _, key := range []string{"vmid", "state", "mode", "ip", "has_route", "dhclient_alive"} {
if _, ok := g[key]; !ok {
t.Fatalf("guest_net.guests[0] is missing the %q key: %v", key, g)
}
}
if g["dhclient_alive"] != true {
t.Fatalf("dhclient_alive must survive the round trip: %v", g)
}
}
+65
View File
@@ -0,0 +1,65 @@
package hub
import (
"context"
"testing"
)
// fakeMgmtPlane is a MgmtPlaneReporter returning a fixed stanza (or nil).
type fakeMgmtPlane struct{ st *MgmtPlaneStatus }
func (f fakeMgmtPlane) MgmtPlaneStatus(context.Context) *MgmtPlaneStatus { return f.st }
func TestCollect_MgmtPlaneOmittedWhenReporterNil(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.MgmtPlane != nil {
t.Fatalf("no reporter wired → mgmt_plane must be omitted (nil), got %+v", r.MgmtPlane)
}
}
func TestCollect_MgmtPlanePopulatedWhenWired(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
c.SetMgmtPlaneReporter(fakeMgmtPlane{st: &MgmtPlaneStatus{
PrivsepDirOK: true, SshdReachable: true, HealedRecently: true, PrivsepHealedAt: "2026-07-05T16:42:17Z",
}})
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.MgmtPlane == nil {
t.Fatal("wired reporter → mgmt_plane must be present")
}
if !r.MgmtPlane.HealedRecently || r.MgmtPlane.PrivsepHealedAt != "2026-07-05T16:42:17Z" {
t.Fatalf("mgmt_plane not carried through: %+v", r.MgmtPlane)
}
}
// fakeOOB is an OOBReporter returning a fixed stanza (or nil).
type fakeOOB struct{ st *OOBStatus }
func (f fakeOOB) OOBStatus(context.Context) *OOBStatus { return f.st }
func TestCollect_OOBOmittedWhenNil(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
r, _ := c.Collect(context.Background())
if r.OOB != nil {
t.Fatalf("no reporter → oob omitted, got %+v", r.OOB)
}
}
func TestCollect_OOBPopulatedWhenWired(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
c.SetOOBReporter(fakeOOB{st: &OOBStatus{FelhomSshdActive: true, FelhomSshdPort: 8822, Reachable: true}})
r, _ := c.Collect(context.Background())
if r.OOB == nil || r.OOB.FelhomSshdPort != 8822 || !r.OOB.Reachable {
t.Fatalf("oob not carried through: %+v", r.OOB)
}
}
+24
View File
@@ -146,3 +146,27 @@ func TestCollect_CloudflaredProbeErrorIsUnknown(t *testing.T) {
t.Error("empty collections must be non-nil")
}
}
// Part A: the served leaf fp rides the report when set (v0.48.0); empty when the local API is disabled
// (no SetLeafFingerprint). Companion: the unset case proves the threading is what populates it.
func TestCollect_LeafFingerprint(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
const fp = "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245"
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
c.SetLeafFingerprint(fp)
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.LeafFingerprint != fp {
t.Fatalf("leaf_fingerprint = %q, want %q", r.LeafFingerprint, fp)
}
// Companion: no SetLeafFingerprint (local API disabled) → empty, never a fabricated value.
c2 := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
r2, _ := c2.Collect(context.Background())
if r2.LeafFingerprint != "" {
t.Fatalf("unset leaf_fingerprint = %q, want empty", r2.LeafFingerprint)
}
}
+28 -1
View File
@@ -42,7 +42,15 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
},
{
Name: "usb-backup", Type: StorageTypeUSB, DurableID: "uuid:x",
State: StorageStateAttached, Reachable: true,
State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000,
Smart: SmartSummary{Health: SmartUnknown},
},
// A pbs target so the recipe's pbs coord has a storage.cfg row to resolve its namespace
// from (R-106). Without one the fixture produces namespace_state=unknown while the golden
// pins a resolved coord — key-set-equal but semantically a fiction.
{
Name: "felhom-pbs", Type: StorageTypePBS, DurableID: "repo+fp", Content: "backup",
State: StorageStateAttached, Reachable: true, PBSNamespace: "felhom-spike",
Smart: SmartSummary{Health: SmartUnknown},
},
},
@@ -72,7 +80,13 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
},
AuditTail: []AuditEntry{},
Cloudflared: Cloudflared{Status: "active"},
// v0.119.0: host addresses. Populated so the bidirectional key-set guard exercises the new
// element keys, not just the presence of the array.
Addresses: []HostAddress{{Iface: "vmbr0", CIDR: "192.168.0.162/24"}},
}
// dr_recipe host-half: built from the same guest/storage/pbs facts (the production path).
report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots,
ConfiguredBackupTarget{StorageID: "usb-backup", Known: true})
b, _ := json.Marshal(report)
var got map[string]any
json.Unmarshal(b, &got)
@@ -95,6 +109,19 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
assertSameKeys(t, "restore_tests[0]", firstElem(golden["restore_tests"]), firstElem(got["restore_tests"]))
// slice-6-Phase-B addition — pbs_snapshots[0] key set.
assertSameKeys(t, "pbs_snapshots[0]", firstElem(golden["pbs_snapshots"]), firstElem(got["pbs_snapshots"]))
// v0.119.0 addition — addresses[0] key set (iface/cidr), the cross-repo wire for the hub's
// Network card.
assertSameKeys(t, "addresses[0]", firstElem(golden["addresses"]), firstElem(got["addresses"]))
// DR-recipe host-half — the agent's secret-free reconstruction-scaffolding section. Assert the
// dr_recipe key set + each sub-array's element key set (the cross-repo wire pinned in the golden).
grec, srec := golden["dr_recipe"], got["dr_recipe"]
assertSameKeys(t, "dr_recipe", grec, srec)
assertSameKeys(t, "dr_recipe.pbs", field(grec, "pbs"), field(srec, "pbs"))
assertSameKeys(t, "dr_recipe.guests[0]", firstElem(field(grec, "guests")), firstElem(field(srec, "guests")))
assertSameKeys(t, "dr_recipe.drives[0]", firstElem(field(grec, "drives")), firstElem(field(srec, "drives")))
assertSameKeys(t, "dr_recipe.pve_storage[0]", firstElem(field(grec, "pve_storage")), firstElem(field(srec, "pve_storage")))
assertSameKeys(t, "dr_recipe.backup_target", field(grec, "backup_target"), field(srec, "backup_target"))
}
// field extracts a nested object value from a decoded JSON map (nil if absent/not a map).
+280
View File
@@ -0,0 +1,280 @@
package hub
import "sort"
// DR recipe — the agent (storage/guest/PBS) HALF of the secret-free reconstruction recipe
// (SPIKE-dr-recipe-2026-06-16). The recipe complements escrow (keys) + PBS (bytes): it is
// the non-secret SCAFFOLDING an operator must rebuild before the PBS bytes can land — guest sizing,
// drive inventory (durable-id → mount → intent → size), PVE storage defs, and PBS coordinates.
//
// BOUNDARY (non-negotiable, the Phase-1 lesson): every field here is an identifier, intent, size, or
// coordinate — NEVER a key, password, token, hash, or ENC: value. Secrets live in the PBS whole-CT
// snapshot + escrow blobs, recovered with R, never regenerated, never here. TestDRRecipeHostHalf_NoSecrets
// asserts no field name matches the secret regex. The hub assembles this half with the controller's
// app half into one customer recipe.
//
// v1 host-half drive shape (v0.39.0) = identifiers/intent/size ONLY: {durable_id, mount_path, intent,
// fs_type?, total_bytes}. Two fields were deliberately DROPPED from v1:
// - role — a drive's purpose (primary/bulk-data/…) is a hub/operator-owned manifest concept, not
// cleanly derivable host-side (both demo externals are content=backup, yet one is the primary
// data drive and the other holds no apps). Deferred until the hub/operator stamps it.
// - restic_repo_coord — RESERVED for a future offsite bulk-volume backup tier. None exists today:
// external-drive data has no offsite/second-failure-domain copy (cross-drive backup is rsync to
// the SAME internal SSD), so the field named nothing real. Re-add when that tier ships.
//
// The pbs coord, by contrast, is resolved LIVE each collect (LiveSnapshotReporter) so the restore
// SOURCE is present whenever PBS is reachable — not gated on the 6 h verify cadence.
//
// recipe_version=1. The wire shape is byte-pinned in the cross-repo golden (host-report.golden.json
// here + the hub's copy) — see the manual checksum-diff discipline in CHANGELOG. Read is
// ignore-unknown (encoding/json default) for forward-compat, mirroring storage_manifest.
const DRRecipeVersion = 1
// Recipe field states (R-106/R-109). A recipe is read at the worst possible moment — by an operator
// rebuilding a machine that is gone — so a field the agent cannot resolve must SAY SO rather than emit
// a default, an empty string, or a plausible-looking placeholder. A guess read as fact costs more than
// an admitted gap: it sends the restore at the wrong archive and nothing contradicts it. This is the
// same cannot-tell-must-not-lie rule R-117 needed a third state for.
const (
DRStateResolved = "resolved"
DRStateUnknown = "unknown"
)
// Reasons a resolved-value field is unknown. Enum-shaped, never free text, so the wire stays pinnable
// and TestDRRecipeHostHalf_NoSecrets has a fixed vocabulary to walk.
const (
// DRReasonNoBackupConfig: the collector was built without a backup-config seam, so the agent could
// not consult the very config its own scheduler reads. Nothing is guessed.
DRReasonNoBackupConfig = "agent_backup_config_unavailable"
// DRReasonNoSuchStorage: the configured target id matches no storage this host observes. The id is
// still recorded (it IS what the config says) and the state says it could not be corroborated.
DRReasonNoSuchStorage = "not_a_known_storage"
// DRReasonNoPBSStorage: snapshots exist but no pbs storage was observed, so there is no storage.cfg
// row to read the namespace from.
DRReasonNoPBSStorage = "no_pbs_storage_observed"
)
// PBSRootNamespace is how the recipe spells PBS's root namespace. The PBS API spells it as the EMPTY
// string (and `pct restore --ns root` would name a namespace that does not exist) — "root" is a display
// convention this wire has always used, kept here so the field's meaning did not change under R-106.
// Only a box with no `namespace` line in its pbs storage.cfg stanza ever emits it.
const PBSRootNamespace = "root"
// DRRecipeHostHalf is the agent-emitted half (guest/drive/storage/PBS scaffolding). Derived entirely
// from facts the report already collects — no new privileged reads.
type DRRecipeHostHalf struct {
RecipeVersion int `json:"recipe_version"`
Guests []DRGuest `json:"guests"`
PBS *DRPBSCoord `json:"pbs,omitempty"`
Drives []DRDrive `json:"drives"`
PVEStorage []DRPVEStorage `json:"pve_storage"`
// BackupTarget names WHICH storage holds the local whole-guest archives (R-109). Always present —
// its own State field carries "I could not tell", so the section is never simply absent.
BackupTarget *DRBackupTarget `json:"backup_target"`
}
// DRBackupTarget answers the one question pve_storage cannot: of every storage listed there, WHICH one
// does this box's primary backup tier actually write its whole-guest archives to?
//
// Before R-109 the recipe listed each storage's name/type/content and said nothing about the target.
// That was harmless while the target was the well-known `local`; the 2026-07-28 vzdump-target move
// ended that. Every box now carries TWO content=backup dir storages — `felhom-backup` (live) and
// `local` (archives frozen at the move, never refreshed since) — and they are indistinguishable by
// name, type and content alone. A restorer picking the frozen one gets a guest that restores cleanly
// and is silently months out of date, which is the worst shape a backup defect can take.
type DRBackupTarget struct {
// State is DRStateResolved | DRStateUnknown. A reader MUST consult it before trusting StorageID:
// the id is also recorded in one unknown case (see DRReasonNoSuchStorage).
State string `json:"state"`
// StorageID is the PVE storage id of the PRIMARY backup tier. Empty only when the config could not
// be consulted at all.
StorageID string `json:"storage_id,omitempty"`
// MountPath is where that storage's archives land on the host — the disambiguation a restorer
// actually needs, since it is what separates felhom-backup's /mnt/hdd_1 from local's /var/lib/vz.
// "" for a pbs target (no host mount) and when unresolved.
MountPath string `json:"mount_path,omitempty"`
// Reason is why State is unknown (one of the DRReason* constants); "" when resolved.
Reason string `json:"reason,omitempty"`
}
// ConfiguredBackupTarget is what the agent's own backup config says the PRIMARY tier writes to.
//
// Known=false is a REAL state, not a nil-guard: it means the collector was constructed without the
// backup-config seam (the --selftest one-shots did exactly this before v0.118.0), and the recipe then
// records unknown instead of inventing a target. Deliberately a struct rather than a `(string, bool)`
// return — the (value, ok) shape is what made "errors degrade to unknown, never to no-backup"
// unimplementable in newestArchiveOn, and this field has the same three-way reading.
type ConfiguredBackupTarget struct {
StorageID string
Known bool
}
// DRGuest is the sizing needed to recreate the LXC at the right size (GuestSpec, already on the wire).
type DRGuest struct {
VMID int `json:"vmid"`
Cores int `json:"cores"`
MemoryBytes int64 `json:"memory_bytes"`
DiskBytes int64 `json:"disk_bytes"`
}
// DRPBSCoord is WHERE the whole-CT snapshot lives — COORDINATES ONLY. The encryption key is escrow-only;
// the access token is identity-escrow-only. Neither is here.
type DRPBSCoord struct {
RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token
// Namespace is the PBS namespace the restore targets, resolved from the pbs storage's storage.cfg
// stanza — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree
// with the backup that produced the snapshot. PBSRootNamespace when the box has no namespace
// configured; "" when NamespaceState is unknown.
//
// R-106: this used to come from the listed snapshot's own `ns`, which PBS does not echo per item once
// the request is already namespace-scoped via `?ns=` (internal/pbs/client.go). The field was
// therefore always empty, ToHub normalised empty → "root", and every per-customer box reported the
// root namespace while its backups were really in `demo-hp` / `demo-felhom`.
Namespace string `json:"namespace"`
// NamespaceState is DRStateResolved | DRStateUnknown — consult it before trusting Namespace.
NamespaceState string `json:"namespace_state"`
// NamespaceReason is why NamespaceState is unknown; "" when resolved.
NamespaceReason string `json:"namespace_reason,omitempty"`
LatestSnapshotID string `json:"latest_snapshot_id"` // most-recent snapshot's backup_id (a coordinate)
}
// DRDrive is one user-data drive: identifiers + intent + size. v1 carries ONLY these fields (role +
// restic_repo_coord were dropped — see the file header for why). Every field is an identifier, intent,
// or size; none is a credential.
type DRDrive struct {
DurableID string `json:"durable_id"` // uuid:<fs-uuid> — a hardware identifier, not a credential
MountPath string `json:"mount_path"`
Intent string `json:"intent"` // enrolled | ejected | decommissioned
FSType string `json:"fs_type,omitempty"`
TotalBytes int64 `json:"total_bytes"`
}
// DRPVEStorage is a PVE storage definition (to rebuild /etc/pve/storage.cfg scaffolding) — no auth.
type DRPVEStorage struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
}
// driveIntentEnrolled is the v1 intent for an emitted user-data drive. The agent's authoritative
// per-drive intent (enrolled/ejected/decommissioned) lives in the GuestBindStore; v1 emits the
// reachable user-data drives it observes as enrolled, with the field present for forward refinement.
const driveIntentEnrolled = "enrolled"
// BuildDRRecipeHostHalf assembles the agent half from the already-collected report facts — pure, so
// it is unit-tested directly (no live reads). drives[] = the user-data external drives (usb/local-dir
// with a durable-id); pve_storage[] = every storage target (the storage.cfg scaffolding); pbs = the
// latest PBS snapshot's coordinates; guests[] = each guest's sizing (skip guests with no spec).
func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSnapshot, backupTarget ConfiguredBackupTarget) *DRRecipeHostHalf {
h := &DRRecipeHostHalf{
RecipeVersion: DRRecipeVersion,
Guests: []DRGuest{},
Drives: []DRDrive{},
PVEStorage: []DRPVEStorage{},
}
for _, g := range guests {
if g.Spec == nil { // status unknown — no sizing to recreate from
continue
}
h.Guests = append(h.Guests, DRGuest{
VMID: g.VMID,
Cores: g.Spec.Cores,
MemoryBytes: g.Spec.MemoryBytes,
DiskBytes: g.Spec.DiskBytes,
})
}
var pbsRepoID, pbsNamespace string
var pbsStorageFound bool
for _, t := range targets {
h.PVEStorage = append(h.PVEStorage, DRPVEStorage{Name: t.Name, Type: t.Type, Content: t.Content})
if t.Type == StorageTypePBS && !pbsStorageFound {
pbsStorageFound = true
pbsRepoID = t.Name // the pbs storage id is a coordinate, not the key
pbsNamespace = t.PBSNamespace // storage.cfg's namespace — "" here means the ROOT namespace
}
if isUserDataDrive(t) {
h.Drives = append(h.Drives, DRDrive{
DurableID: t.DurableID,
MountPath: t.MountPath,
Intent: driveIntentEnrolled,
TotalBytes: t.TotalBytes,
})
}
}
h.BackupTarget = resolveBackupTarget(targets, backupTarget)
if c := latestPBSCoord(pbs, pbsRepoID, pbsNamespace, pbsStorageFound); c != nil {
h.PBS = c
}
return h
}
// resolveBackupTarget records WHICH storage the primary backup tier writes to (R-109), or records
// explicitly that it could not tell. Three outcomes, and the two unknowns are deliberately distinct —
// "I could not read my own config" and "my config names a storage that is not here" send an operator
// to different places.
//
// MountPath prefers the live mount and falls back to the CONFIGURED path: during a rebuild the drive is
// frequently absent, and when it is, MountPath empties out while ConfigPath is the only thing left that
// still says which drive the row was about (the R-116 lesson). The storage's absence from the host is a
// separate signal (E-2's backup_target_absent); it does not make the recipe's answer unknown, because
// the question here is which storage.cfg row to restore FROM, and that is still known.
func resolveBackupTarget(targets []StorageTarget, cfg ConfiguredBackupTarget) *DRBackupTarget {
if !cfg.Known || cfg.StorageID == "" {
return &DRBackupTarget{State: DRStateUnknown, Reason: DRReasonNoBackupConfig}
}
for _, t := range targets {
if t.Name != cfg.StorageID {
continue
}
mount := t.MountPath
if mount == "" {
mount = t.ConfigPath
}
return &DRBackupTarget{State: DRStateResolved, StorageID: cfg.StorageID, MountPath: mount}
}
return &DRBackupTarget{State: DRStateUnknown, StorageID: cfg.StorageID, Reason: DRReasonNoSuchStorage}
}
// isUserDataDrive selects the external user-data drives the recipe enumerates (felhom-usb / felhom-flash
// class): a usb or local-dir storage with a filesystem-UUID durable id and a host mount path. local /
// lvmthin / pbs / nfs / cifs are scaffolding (they land in pve_storage) but are not user-data drives.
func isUserDataDrive(t StorageTarget) bool {
if t.Type != StorageTypeUSB && t.Type != StorageTypeLocalDir {
return false
}
return t.DurableID != "" && t.MountPath != ""
}
// latestPBSCoord picks the most-recent snapshot (lexical max of the RFC3339 backup_time) and returns
// its coordinates. Returns nil when there is no snapshot to target.
//
// The namespace comes from the pbs STORAGE (storage.cfg), never from the snapshot — see DRPBSCoord's
// Namespace comment for why the snapshot's own field cannot answer it (R-106). storageFound=false with
// snapshots present is a genuine unknown: something listed snapshots, but there is no storage row to
// read a namespace from, so the recipe says so rather than defaulting to root.
func latestPBSCoord(snaps []PBSSnapshot, repoID, namespace string, storageFound bool) *DRPBSCoord {
if len(snaps) == 0 {
return nil
}
sorted := append([]PBSSnapshot(nil), snaps...)
sort.Slice(sorted, func(i, j int) bool { return sorted[i].BackupTime > sorted[j].BackupTime })
latest := sorted[0]
c := &DRPBSCoord{
RepoID: repoID,
LatestSnapshotID: latest.BackupID,
NamespaceState: DRStateUnknown,
NamespaceReason: DRReasonNoPBSStorage,
}
if storageFound {
c.NamespaceState, c.NamespaceReason = DRStateResolved, ""
// An empty configured namespace is not a missing answer — it IS the root namespace.
if c.Namespace = namespace; c.Namespace == "" {
c.Namespace = PBSRootNamespace
}
}
return c
}
+455
View File
@@ -0,0 +1,455 @@
package hub
import (
"context"
"encoding/json"
"regexp"
"strings"
"testing"
)
// secretNameRe matches any JSON key that smells like a credential. The DR recipe must contain NONE
// (the Phase-1 lesson: the retired infra-backup shipped encryption_key_b64/restic_password/cf_api_token).
// Mirrored on the controller app-half emitter (the heavier boundary test lives there).
var secretNameRe = regexp.MustCompile(`(?i)(password|secret|token|hash|passphrase|api[_-]?key|\bkey\b|enc:)`)
func TestBuildDRRecipeHostHalf(t *testing.T) {
guests := []Guest{
{VMID: 9201, Name: "cust", Status: "running", Spec: &GuestSpec{Cores: 4, MemoryBytes: 12 << 30, DiskBytes: 32 << 30}},
{VMID: 9202, Name: "unknown", Status: "unknown"}, // nil Spec → skipped (no sizing)
}
targets := []StorageTarget{
{Name: "local", Type: StorageTypeLocal, Content: "vztmpl,iso"},
{Name: "local-lvm", Type: StorageTypeLVMThin, DurableID: "pve/data", Content: "rootdir,images"},
{Name: "felhom-pbs", Type: StorageTypePBS, DurableID: "repo+fp", Content: "backup"},
{Name: "felhom-usb", Type: StorageTypeUSB, DurableID: "uuid:da9e7089", Role: "bulk-data",
MountPath: "/mnt/felhom-usb", TotalBytes: 931 << 30},
{Name: "felhom-flash", Type: StorageTypeLocalDir, DurableID: "uuid:81a26531", Role: "primary",
MountPath: "/mnt/felhom-flash", TotalBytes: 119 << 30},
}
pbs := []PBSSnapshot{
{Namespace: "root", BackupID: "9201", BackupTime: "2026-06-10T00:00:00Z"},
{Namespace: "root", BackupID: "9201", BackupTime: "2026-06-16T08:00:00Z"}, // latest
}
h := BuildDRRecipeHostHalf(guests, targets, pbs, ConfiguredBackupTarget{StorageID: "felhom-flash", Known: true})
if h.RecipeVersion != 1 {
t.Errorf("recipe_version=%d, want 1", h.RecipeVersion)
}
// guests: only the spec'd one.
if len(h.Guests) != 1 || h.Guests[0].VMID != 9201 || h.Guests[0].Cores != 4 || h.Guests[0].MemoryBytes != 12<<30 {
t.Errorf("guests = %+v, want only vmid 9201 with its sizing", h.Guests)
}
// pve_storage: ALL five targets (the storage.cfg scaffolding).
if len(h.PVEStorage) != 5 {
t.Errorf("pve_storage len=%d, want 5 (every target)", len(h.PVEStorage))
}
// drives: ONLY the two user-data drives (usb + local-dir with uuid + mount). NOT local/lvm/pbs.
if len(h.Drives) != 2 {
t.Fatalf("drives len=%d, want 2 user-data drives, got %+v", len(h.Drives), h.Drives)
}
byDur := map[string]DRDrive{}
for _, d := range h.Drives {
byDur[d.DurableID] = d
if d.Intent != "enrolled" {
t.Errorf("drive %s intent=%q, want enrolled", d.DurableID, d.Intent)
}
}
if d, ok := byDur["uuid:da9e7089"]; !ok || d.MountPath != "/mnt/felhom-usb" || d.TotalBytes != 931<<30 {
t.Errorf("felhom-usb drive wrong: %+v", d)
}
if _, ok := byDur["uuid:81a26531"]; !ok {
t.Error("felhom-flash (local-dir user-data drive) missing from drives")
}
// pbs: latest snapshot's coords + the pbs storage id as repo_id.
if h.PBS == nil || h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != "root" || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want repo felhom-pbs/root/9201", h.PBS)
}
}
// TestBuildDRRecipeHostHalf_NoPBS: no snapshots → pbs omitted (nil), no panic.
func TestBuildDRRecipeHostHalf_NoPBS(t *testing.T) {
h := BuildDRRecipeHostHalf(nil, []StorageTarget{{Name: "local", Type: StorageTypeLocal}}, nil,
ConfiguredBackupTarget{StorageID: "local", Known: true})
if h.PBS != nil {
t.Errorf("pbs should be nil with no snapshots, got %+v", h.PBS)
}
if h.Guests == nil || h.Drives == nil || h.PVEStorage == nil {
t.Error("slices must be non-nil (marshal as [], not null)")
}
}
// TestDRRecipeHostHalf_V1DriveShape pins the v1 host-half drive shape: a drive object carries ONLY
// {durable_id, mount_path, intent, total_bytes} (fs_type is omitempty) — and specifically NEITHER the
// dropped "role" NOR "restic_repo_coord" keys. Re-adding either field to DRDrive makes this fail
// (the companion: `Role string \`json:"role"\“ reintroduces the "role" key → caught here).
func TestDRRecipeHostHalf_V1DriveShape(t *testing.T) {
h := BuildDRRecipeHostHalf(
nil,
[]StorageTarget{
{Name: "felhom-usb", Type: StorageTypeUSB, DurableID: "uuid:da9e7089", Role: "bulk-data",
MountPath: "/mnt/felhom-usb", TotalBytes: 931 << 30},
},
nil,
ConfiguredBackupTarget{StorageID: "felhom-usb", Known: true},
)
if len(h.Drives) != 1 {
t.Fatalf("want 1 drive, got %d", len(h.Drives))
}
b, err := json.Marshal(h.Drives[0])
if err != nil {
t.Fatal(err)
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(b, &keys); err != nil {
t.Fatal(err)
}
for _, banned := range []string{"role", "restic_repo_coord"} {
if _, ok := keys[banned]; ok {
t.Errorf("v1 drive must NOT carry %q key (it was dropped); got %s", banned, b)
}
}
for _, want := range []string{"durable_id", "mount_path", "intent", "total_bytes"} {
if _, ok := keys[want]; !ok {
t.Errorf("v1 drive missing required key %q; got %s", want, b)
}
}
}
// TestDRRecipeHostHalf_NoSecrets is the agent-side boundary assertion (the lighter mirror of the
// controller's load-bearing boundary test): a fully-populated host-half must carry NO field whose
// name smells like a credential. If a future field leaks a key/token/hash in, this fails.
func TestDRRecipeHostHalf_NoSecrets(t *testing.T) {
h := BuildDRRecipeHostHalf(
[]Guest{{VMID: 9201, Spec: &GuestSpec{Cores: 4, MemoryBytes: 1, DiskBytes: 1}}},
[]StorageTarget{
{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup"},
{Name: "felhom-usb", Type: StorageTypeUSB, DurableID: "uuid:da9e7089", Role: "bulk-data", MountPath: "/mnt/felhom-usb", TotalBytes: 1},
},
[]PBSSnapshot{{Namespace: "root", BackupID: "9201", BackupTime: "2026-06-16T08:00:00Z"}},
ConfiguredBackupTarget{StorageID: "felhom-usb", Known: true},
)
b, err := json.Marshal(h)
if err != nil {
t.Fatal(err)
}
assertNoSecretKeys(t, b)
}
// ---------------------------------------------------------------------------------------------
// R-106 / R-109 — the recipe records the RESOLVED backup target and the REAL PBS namespace.
// ---------------------------------------------------------------------------------------------
// capturedDemoFelhomTargets is the storage set demo-felhom really had on 2026-07-30, not an invented
// one. PROVENANCE — every field was captured, none composed:
//
// - names/types/contents: the pve_storage block of the box's own PRE-FIX recipe, downloaded from the
// hub at GET /customers/demo-felhom/dr-recipe.json (agent v0.115.0).
// - paths + is_mountpoint + the pbs namespace: `cat /etc/pve/storage.cfg` on felhom-pve, same day —
// `dir: local path /var/lib/vz`, `dir: felhom-backup path /mnt/hdd_1 is_mountpoint 1`,
// `pbs: felhom-pbs ... namespace demo-felhom`.
//
// THE AMBIGUITY THIS PINS IS REAL, and assertBackupCandidateAmbiguity below refuses to let the fixture
// quietly lose it: `local` and `felhom-backup` BOTH carry content=backup, and since the 2026-07-28
// vzdump-target move `local` holds archives frozen at that date. Naming the wrong one restores a guest
// that is silently months stale.
func capturedDemoFelhomTargets() []StorageTarget {
return []StorageTarget{
{Name: "local-lvm", Type: StorageTypeLVMThin, DurableID: "pve/data", Content: "images,rootdir"},
{
Name: "felhom-backup", Type: StorageTypeLocalDir, Content: "backup",
DurableID: "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48",
MountPath: "/mnt/hdd_1", ConfigPath: "/mnt/hdd_1", TotalBytes: 983351140352,
},
{
Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup",
DurableID: "repo+fp", PBSNamespace: "demo-felhom",
},
// The decoy: same content, plausible name, historically THE vzdump target. ConfigPath only —
// `local` lives on the LVM root and is not its own mount, so the observer leaves MountPath empty.
{Name: "local", Type: StorageTypeLocal, Content: "backup,import,vztmpl,iso", ConfigPath: "/var/lib/vz"},
}
}
// capturedDemoFelhomSnapshots mirrors what the box's pre-fix recipe carried: latest_snapshot_id "9201".
// Namespace is deliberately EMPTY on every element — that is exactly what the PBS API returns once the
// list is namespace-scoped via `?ns=`, and it is the input that used to become the bogus "root".
func capturedDemoFelhomSnapshots() []PBSSnapshot {
return []PBSSnapshot{
{Namespace: "", BackupID: "9201", BackupTime: "2026-07-29T22:00:00Z"},
{Namespace: "", BackupID: "9201", BackupTime: "2026-07-30T22:00:00Z"}, // latest
}
}
// assertBackupCandidateAmbiguity fails if the fixture stopped containing TWO plausible content=backup
// storages. Without this the consequence test below could pass on a fixture with only one candidate —
// which is precisely the hollow shape that let two defects ship green earlier in this arc.
func assertBackupCandidateAmbiguity(t *testing.T, h *DRRecipeHostHalf) {
t.Helper()
var candidates []string
for _, s := range h.PVEStorage {
if strings.Contains(s.Content, "backup") && (s.Type == StorageTypeLocalDir || s.Type == StorageTypeLocal) {
candidates = append(candidates, s.Name)
}
}
if len(candidates) < 2 {
t.Fatalf("fixture no longer poses the R-109 problem: want >=2 content=backup dir storages, got %v", candidates)
}
}
// TestDRRecipe_BackupTargetNamesTheLiveStorage is THE consequence assertion for R-109: given a box that
// really carries two content=backup dir storages, the generated recipe names the LIVE one, gives its
// mountpoint, and does not name the frozen one. Not "the function returned a non-empty string".
func TestDRRecipe_BackupTargetNamesTheLiveStorage(t *testing.T) {
h := BuildDRRecipeHostHalf(nil, capturedDemoFelhomTargets(), capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
assertBackupCandidateAmbiguity(t, h)
bt := h.BackupTarget
if bt == nil {
t.Fatal("backup_target is absent — the recipe still cannot say where the local archives are (R-109)")
}
if bt.State != DRStateResolved {
t.Errorf("state=%q want %q (reason=%q)", bt.State, DRStateResolved, bt.Reason)
}
if bt.StorageID != "felhom-backup" {
t.Errorf("storage_id=%q — the recipe must name the LIVE target, not %q", bt.StorageID, "felhom-backup")
}
if bt.MountPath != "/mnt/hdd_1" {
t.Errorf("mount_path=%q want /mnt/hdd_1 — the mountpoint is what separates it from local's /var/lib/vz", bt.MountPath)
}
// Unambiguous: the frozen decoy must not be what the field names.
if bt.StorageID == "local" || bt.MountPath == "/var/lib/vz" {
t.Errorf("recipe names the FROZEN target (%q at %q) — a restore from it is silently stale", bt.StorageID, bt.MountPath)
}
}
// TestDRRecipe_PBSNamespaceIsThePerCustomerOne is the consequence assertion for R-106: the recipe carries
// the namespace the box's backups actually live in, resolved from storage.cfg, and specifically NOT the
// "root" that every box used to report.
func TestDRRecipe_PBSNamespaceIsThePerCustomerOne(t *testing.T) {
h := BuildDRRecipeHostHalf(nil, capturedDemoFelhomTargets(), capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
if h.PBS == nil {
t.Fatal("pbs coord absent with snapshots present")
}
if h.PBS.Namespace == PBSRootNamespace {
t.Errorf("namespace=%q — this is the R-106 symptom: the snapshot's empty ns normalised to root "+
"while the box's backups are in demo-felhom", h.PBS.Namespace)
}
if h.PBS.Namespace != "demo-felhom" {
t.Errorf("namespace=%q want demo-felhom (storage.cfg's `namespace` on the pbs storage)", h.PBS.Namespace)
}
if h.PBS.NamespaceState != DRStateResolved {
t.Errorf("namespace_state=%q want %q (reason=%q)", h.PBS.NamespaceState, DRStateResolved, h.PBS.NamespaceReason)
}
if h.PBS.RepoID != "felhom-pbs" || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("coord drifted: repo=%q snapshot=%q", h.PBS.RepoID, h.PBS.LatestSnapshotID)
}
}
// TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown: a box with a pbs storage and NO namespace line is
// genuinely in the root namespace. That is an answer, not a gap — it must read resolved/"root", so the
// honest root case is never confused with "I could not tell".
func TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup", PBSNamespace: ""}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
if h.PBS.NamespaceState != DRStateResolved {
t.Errorf("namespace_state=%q — an unconfigured namespace IS the root namespace, not an unknown", h.PBS.NamespaceState)
}
if h.PBS.Namespace != PBSRootNamespace {
t.Errorf("namespace=%q want %q", h.PBS.Namespace, PBSRootNamespace)
}
}
// TestDRRecipe_BackupTargetUnknownWhenConfigUnavailable is the WRONG case: the agent could not consult
// its own backup config. The recipe must say so explicitly and emit NO storage_id key at all — an
// absent value must not be representable as a plausible-looking answer.
func TestDRRecipe_BackupTargetUnknownWhenConfigUnavailable(t *testing.T) {
h := BuildDRRecipeHostHalf(nil, capturedDemoFelhomTargets(), nil, ConfiguredBackupTarget{})
bt := h.BackupTarget
if bt == nil {
t.Fatal("backup_target must be PRESENT and say unknown, not vanish")
}
if bt.State != DRStateUnknown || bt.Reason != DRReasonNoBackupConfig {
t.Errorf("state=%q reason=%q want %q/%q", bt.State, bt.Reason, DRStateUnknown, DRReasonNoBackupConfig)
}
// Absence recorded as absence: no id, and no id KEY on the wire.
if bt.StorageID != "" {
t.Errorf("storage_id=%q — an unresolvable target must not be filled in", bt.StorageID)
}
b, err := json.Marshal(bt)
if err != nil {
t.Fatal(err)
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(b, &keys); err != nil {
t.Fatal(err)
}
for _, banned := range []string{"storage_id", "mount_path"} {
if _, ok := keys[banned]; ok {
t.Errorf("unknown backup_target must not carry a %q key; got %s", banned, b)
}
}
// And nothing in it may read as one of the real candidates.
for _, decoy := range []string{"felhom-backup", "local", "/var/lib/vz", "/mnt/hdd_1"} {
if strings.Contains(string(b), decoy) {
t.Errorf("unknown backup_target leaked a plausible value %q: %s", decoy, b)
}
}
}
// TestDRRecipe_BackupTargetUnknownWhenStorageMissing: the config names a storage this host does not
// have. That is unknown for a DIFFERENT reason — and the configured id IS still recorded, because
// "config says felhom-backup, no such storage here" sends an operator somewhere useful while silence
// does not.
func TestDRRecipe_BackupTargetUnknownWhenStorageMissing(t *testing.T) {
targets := []StorageTarget{{Name: "local", Type: StorageTypeLocal, Content: "backup", ConfigPath: "/var/lib/vz"}}
h := BuildDRRecipeHostHalf(nil, targets, nil, ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
bt := h.BackupTarget
if bt.State != DRStateUnknown || bt.Reason != DRReasonNoSuchStorage {
t.Errorf("state=%q reason=%q want %q/%q", bt.State, bt.Reason, DRStateUnknown, DRReasonNoSuchStorage)
}
if bt.StorageID != "felhom-backup" {
t.Errorf("storage_id=%q want the CONFIGURED id recorded even though it matched nothing", bt.StorageID)
}
// It must NOT silently fall back to the only content=backup storage present.
if bt.StorageID == "local" || bt.MountPath == "/var/lib/vz" {
t.Error("resolution fell back to the wrong storage instead of reporting unknown")
}
}
// TestDRRecipe_PBSNamespaceUnknownWithoutPBSStorage: snapshots exist but no pbs storage was observed, so
// there is no storage.cfg row to read a namespace from. The recipe must NOT default to root — that
// default is the entire R-106 defect.
func TestDRRecipe_PBSNamespaceUnknownWithoutPBSStorage(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "local", Type: StorageTypeLocal, Content: "backup"}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "local", Known: true})
if h.PBS == nil {
t.Fatal("pbs coord should still be emitted (the snapshot id is a real coordinate)")
}
if h.PBS.NamespaceState != DRStateUnknown || h.PBS.NamespaceReason != DRReasonNoPBSStorage {
t.Errorf("namespace_state=%q reason=%q want %q/%q",
h.PBS.NamespaceState, h.PBS.NamespaceReason, DRStateUnknown, DRReasonNoPBSStorage)
}
if h.PBS.Namespace != "" {
t.Errorf("namespace=%q — with no storage row to read, the field must be empty, never %q",
h.PBS.Namespace, PBSRootNamespace)
}
}
// TestDRRecipe_BackupTargetUsesConfigPathWhenDeviceGone is the DR-shaped case: the recipe is read while
// the target drive is absent, so MountPath has emptied out. ConfigPath is then the only thing that still
// says where the archives live (the R-116 lesson) — and the target is still RESOLVED, because which
// storage.cfg row to restore from is known regardless of whether its device is currently present.
func TestDRRecipe_BackupTargetUsesConfigPathWhenDeviceGone(t *testing.T) {
targets := []StorageTarget{{
Name: "felhom-backup", Type: StorageTypeLocalDir, Content: "backup",
MountPath: "", ConfigPath: "/mnt/hdd_1", // device gone: observer empties MountPath, keeps ConfigPath
}}
h := BuildDRRecipeHostHalf(nil, targets, nil, ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
bt := h.BackupTarget
if bt.State != DRStateResolved {
t.Errorf("state=%q — an absent device does not make the TARGET unknown", bt.State)
}
if bt.MountPath != "/mnt/hdd_1" {
t.Errorf("mount_path=%q want the configured path /mnt/hdd_1", bt.MountPath)
}
}
// fakePBSReporter is a PBSReporter returning fixed snapshots (the verify loop's seam).
type fakePBSReporter struct{ snaps []PBSSnapshot }
func (f fakePBSReporter) PBSSnapshots(context.Context) []PBSSnapshot { return f.snaps }
// TestCollectDRRecipe_ProductionPath runs the REAL generation path — Collector.Collect(), the method the
// daemon calls every cycle — rather than BuildDRRecipeHostHalf directly. It is here because both defects
// this file fixes were invisible to a direct-call test: the namespace one lived in what the observer put
// on StorageTarget, and the target one lived in whether anything wired the config seam at all. A seam
// that is correct and never wired is the failure mode this repo has hit four times.
func TestCollectDRRecipe_ProductionPath(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
obs := fakeObserver{targets: capturedDemoFelhomTargets()}
pbsRep := fakePBSReporter{snaps: capturedDemoFelhomSnapshots()}
c := NewCollector(px, fakeProber{status: "active"}, obs, nil, nil, pbsRep, "h", "0.118.0", quietLogger())
c.SetBackupTargetResolver(func() ConfiguredBackupTarget {
return ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true}
})
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.DRRecipe == nil {
t.Fatal("collect produced no dr_recipe")
}
if bt := r.DRRecipe.BackupTarget; bt == nil || bt.State != DRStateResolved || bt.StorageID != "felhom-backup" {
t.Errorf("backup_target through Collect = %+v, want resolved/felhom-backup", bt)
}
if p := r.DRRecipe.PBS; p == nil || p.Namespace != "demo-felhom" || p.NamespaceState != DRStateResolved {
t.Errorf("pbs namespace through Collect = %+v, want demo-felhom/resolved", p)
}
}
// TestCollectDRRecipe_UnwiredSeamReportsUnknown: a Collector built WITHOUT the resolver (every
// --selftest one-shot did exactly this before v0.118.0) must produce an explicit unknown. This is the
// test that would have caught shipping the seam without wiring it.
func TestCollectDRRecipe_UnwiredSeamReportsUnknown(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{targets: capturedDemoFelhomTargets()},
nil, nil, nil, "h", "0.118.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
bt := r.DRRecipe.BackupTarget
if bt == nil || bt.State != DRStateUnknown || bt.Reason != DRReasonNoBackupConfig {
t.Fatalf("unwired resolver must yield unknown/%s, got %+v", DRReasonNoBackupConfig, bt)
}
if bt.StorageID != "" {
t.Errorf("unwired resolver invented a target %q", bt.StorageID)
}
}
// assertNoSecretKeys walks decoded JSON and fails on any object key matching secretNameRe. Shared by
// the agent boundary assertions. (durable_id/repo_id/latest_snapshot_id are identifiers/coordinates —
// none match the credential regex.)
func assertNoSecretKeys(t *testing.T, jsonBytes []byte) {
t.Helper()
var v any
if err := json.Unmarshal(jsonBytes, &v); err != nil {
t.Fatal(err)
}
var walk func(prefix string, node any)
walk = func(prefix string, node any) {
switch n := node.(type) {
case map[string]any:
for k, child := range n {
if secretNameRe.MatchString(k) {
t.Errorf("secret-shaped key %q at %s — the recipe must carry no credential field", k, prefix)
}
walk(prefix+"."+k, child)
}
case []any:
for i, child := range n {
walk(prefix, child)
_ = i
}
}
}
walk("<root>", v)
}
+134
View File
@@ -0,0 +1,134 @@
package hub
import (
"net"
"net/netip"
"sort"
)
// Host addresses (v0.119.0) — "which addresses does this box actually hold?"
//
// The hub could not answer that at all: HostMetrics carried node/cpu/mem/disk/load/uptime/temp and
// no address of any kind, so the LAN IP of a managed host was invisible in every operator surface.
// Two sources looked like answers and are not: `lan_resolver.host_ip` is an OPTIONAL config value
// (absent unless that feature is configured), and DeriveHostIP(local_api.listen_addr) yields the
// R-50 island literal 169.254.253.1 — a link-local address that is the same on every box. Reporting
// either would have produced a confident wrong answer, which is worse than the blank it replaces.
//
// This reads the kernel's own view instead, and it issues NO block I/O (the CLAUDE.md health-check
// rule): net.Interfaces() is a netlink/procfs read, needs no privilege, and touches no filesystem.
// HostAddress is one routable address the host holds, tagged with the interface carrying it.
//
// Deliberately iface+cidr rather than a single `lan_ip`: a Proxmox host legitimately holds several
// (a management bridge, a tailnet, the WG tunnel), and picking one of them to call "the" LAN IP is a
// guess the agent is not entitled to make — on a box whose management bridge is not vmbr0 that guess
// is silently wrong. The agent reports what exists; the hub does the labelling.
type HostAddress struct {
Iface string `json:"iface"` // e.g. "vmbr0", "wg-felhom", "tailscale0"
CIDR string `json:"cidr"` // e.g. "192.168.0.162/24" — prefix length kept, it is operator-relevant
}
// ifaceAddrs is one enumerated interface: the ONLY facts the filter needs. Keeping the seam this
// narrow is what lets the filter be tested against real measured shapes without a network stack.
type ifaceAddrs struct {
Name string
Up bool
Loopback bool
CIDRs []string
}
// AddressEnumerator returns the host's interfaces. Injectable so the filter can be driven with the
// shapes measured on real hardware (see hostaddr_test.go) instead of whatever the test box happens
// to have.
type AddressEnumerator func() ([]ifaceAddrs, error)
// systemInterfaces is the production enumerator: the kernel's interface table.
func systemInterfaces() ([]ifaceAddrs, error) {
ifaces, err := net.Interfaces()
if err != nil {
return nil, err
}
out := make([]ifaceAddrs, 0, len(ifaces))
for _, i := range ifaces {
e := ifaceAddrs{
Name: i.Name,
Up: i.Flags&net.FlagUp != 0,
Loopback: i.Flags&net.FlagLoopback != 0,
}
// A per-interface error is not fatal: one unreadable interface must not cost the report
// every other address (serve-degraded, as everywhere else in the collector).
addrs, aerr := i.Addrs()
if aerr != nil {
out = append(out, e)
continue
}
for _, a := range addrs {
e.CIDRs = append(e.CIDRs, a.String())
}
out = append(out, e)
}
return out, nil
}
// filterHostAddresses keeps every GLOBAL UNICAST address on an up, non-loopback interface.
//
// IsGlobalUnicast() is the whole rule, and it was chosen by measuring both demo hosts rather than by
// listing interface names to exclude. It drops, in one predicate:
// - loopback (127.0.0.1, ::1)
// - IPv6 link-local (fe80::/10) — every bridge carries one, pure noise
// - IPv4 link-local (169.254.0.0/16) — which is exactly the R-50 island address on vmbr9, an
// identical constant on every box and therefore actively misleading if surfaced
//
// It needs NO veth/fwbr/tap denylist: on a Proxmox host that per-guest plumbing carries no IP at
// all, so it self-excludes by having nothing to report. Verified on demo-felhom and demo-hp —
// veth9201i0/i1, fwbr*, and the unused NICs all appear in `ip link` and in no `ip addr` output.
//
// What survives on a real box: vmbr0's LAN address, wg-felhom's tunnel address, and tailscale0's
// tailnet addresses. All three are true and useful; none is labelled here.
func filterHostAddresses(in []ifaceAddrs) []HostAddress {
out := []HostAddress{}
for _, i := range in {
if i.Loopback || !i.Up {
continue
}
for _, c := range i.CIDRs {
p, err := netip.ParsePrefix(c)
if err != nil {
continue // not a CIDR we understand — skip it, never fail the report
}
if !p.Addr().IsGlobalUnicast() {
continue
}
out = append(out, HostAddress{Iface: i.Name, CIDR: p.String()})
}
}
// Deterministic order so a report diff reflects a real change, not interface-table ordering.
sort.Slice(out, func(a, b int) bool {
if out[a].Iface != out[b].Iface {
return out[a].Iface < out[b].Iface
}
return out[a].CIDR < out[b].CIDR
})
return out
}
// collectAddresses is the collector's entry point. It returns a non-nil slice so the field always
// marshals as [] — an absent key and "this box has no routable address" must not look alike to the
// hub, and [] is the honest encoding of the latter.
func (c *Collector) collectAddresses() []HostAddress {
enum := c.addrEnum
if enum == nil {
// Default to the REAL enumerator, deliberately inverting the nil-reporter-means-off
// convention used by the optional stanzas above. Those gate on a config feature; this has
// no dependency and no feature flag, so a forgotten wiring call in main.go would produce a
// silently empty field — the inert-seam failure this repo has shipped four times.
enum = systemInterfaces
}
ifaces, err := enum()
if err != nil {
c.logger.Warn("host addresses: interface enumeration failed", "err", err)
return []HostAddress{}
}
return filterHostAddresses(ifaces)
}
+214
View File
@@ -0,0 +1,214 @@
package hub
import (
"errors"
"log/slog"
"strings"
"testing"
)
// The fixtures below are MEASURED, not invented: `ip -o addr show` on demo-felhom (N100) and
// demo-hp (HP t740) on 2026-07-31, transcribed verbatim including the interfaces that carry no
// address. That matters — the filter's claim that it needs no veth/fwbr denylist rests on those
// interfaces genuinely having nothing to report, and a hand-written fixture that omitted them would
// have proved the claim by assuming it.
// demoFelhomIfaces is demo-felhom's real interface table.
func demoFelhomIfaces() []ifaceAddrs {
return []ifaceAddrs{
{Name: "lo", Up: true, Loopback: true, CIDRs: []string{"127.0.0.1/8", "::1/128"}},
{Name: "enp1s0", Up: false}, // physical NIC, no address
{Name: "wlp2s0", Up: false}, // wifi, no address
{Name: "tailscale0", Up: true, CIDRs: []string{
"100.70.170.35/32", "fd7a:115c:a1e0::5236:aa24/128", "fe80::4197:26fc:ccba:b0d9/64"}},
{Name: "vmbr0", Up: true, CIDRs: []string{"192.168.0.162/24", "fe80::6a1d:efff:fe5d:a664/64"}},
{Name: "veth9201i0", Up: true}, // per-guest plumbing — no address
{Name: "veth9201i1", Up: true}, // per-guest plumbing — no address
{Name: "vmbr9", Up: true, CIDRs: []string{"169.254.253.1/30", "fe80::48d4:f6ff:fe05:2f98/64"}},
{Name: "wg-felhom", Up: true, CIDRs: []string{"10.77.0.2/32"}},
}
}
func hasAddr(got []HostAddress, iface, cidr string) bool {
for _, a := range got {
if a.Iface == iface && a.CIDR == cidr {
return true
}
}
return false
}
func flatten(got []HostAddress) string {
var b strings.Builder
for _, a := range got {
b.WriteString(a.Iface + "=" + a.CIDR + " ")
}
return b.String()
}
// The LAN address is the whole point of the feature — it must survive the filter.
// RED-PROOF 1: drop the `!p.Addr().IsGlobalUnicast()` continue → the vmbr9 + fe80 assertions below
// go red (the LAN one still passes, which is exactly why the negatives are asserted too).
func TestFilterHostAddresses_RealHost(t *testing.T) {
got := filterHostAddresses(demoFelhomIfaces())
// --- what MUST be there ---
if !hasAddr(got, "vmbr0", "192.168.0.162/24") {
t.Fatalf("the LAN address was filtered away — the feature reports nothing: %s", flatten(got))
}
if !hasAddr(got, "wg-felhom", "10.77.0.2/32") {
t.Errorf("the WireGuard address was filtered away: %s", flatten(got))
}
if !hasAddr(got, "tailscale0", "100.70.170.35/32") {
t.Errorf("the tailnet address was filtered away: %s", flatten(got))
}
// --- what MUST NOT be there, each for its own reason ---
for _, bad := range []struct{ iface, cidr, why string }{
{"lo", "127.0.0.1/8", "loopback is not an address of the host on any network"},
{"lo", "::1/128", "IPv6 loopback"},
{"vmbr9", "169.254.253.1/30", "the R-50 island literal — IDENTICAL on every box, so surfacing it is actively misleading"},
{"vmbr0", "fe80::6a1d:efff:fe5d:a664/64", "IPv6 link-local, one per bridge, pure noise"},
{"tailscale0", "fe80::4197:26fc:ccba:b0d9/64", "IPv6 link-local"},
} {
if hasAddr(got, bad.iface, bad.cidr) {
t.Errorf("%s %s must be filtered (%s); got: %s", bad.iface, bad.cidr, bad.why, flatten(got))
}
}
// The no-denylist claim: not one veth/physical interface contributed a row.
for _, a := range got {
if strings.HasPrefix(a.Iface, "veth") || a.Iface == "enp1s0" || a.Iface == "wlp2s0" {
t.Errorf("%s produced a row — the fixture says it has no address, so the filter invented one", a.Iface)
}
}
}
// demo-hp is different hardware (4 unused NICs, different ordering) and must filter identically —
// the rule is about address CLASS, not about one box's interface names.
func TestFilterHostAddresses_SecondHostFiltersIdentically(t *testing.T) {
got := filterHostAddresses([]ifaceAddrs{
{Name: "lo", Up: true, Loopback: true, CIDRs: []string{"127.0.0.1/8", "::1/128"}},
{Name: "enp2s0f0", Up: false}, {Name: "enp1s0f0", Up: false},
{Name: "enp1s0f1", Up: false}, {Name: "enp1s0f2", Up: false},
{Name: "enp1s0f3", Up: false}, {Name: "wlo1", Up: false},
{Name: "tailscale0", Up: true, CIDRs: []string{
"100.76.96.79/32", "fd7a:115c:a1e0::ce36:6051/128", "fe80::e06a:ce64:80e1:7821/64"}},
{Name: "vmbr0", Up: true, CIDRs: []string{"192.168.0.87/24", "fe80::7ed3:aff:fe77:d976/64"}},
{Name: "wg-felhom", Up: true, CIDRs: []string{"10.77.0.3/32"}},
{Name: "vmbr9", Up: true, CIDRs: []string{"169.254.253.1/30", "fe80::2484:92ff:fe7d:52a5/64"}},
{Name: "veth9201i0", Up: true}, {Name: "veth9201i1", Up: true},
})
if !hasAddr(got, "vmbr0", "192.168.0.87/24") {
t.Fatalf("demo-hp's LAN address was filtered away: %s", flatten(got))
}
if hasAddr(got, "vmbr9", "169.254.253.1/30") {
t.Errorf("demo-hp's island address leaked through: %s", flatten(got))
}
// The island address is byte-identical on both boxes — the strongest argument for excluding it.
if strings.Contains(flatten(got), "169.254.") {
t.Errorf("a link-local IPv4 survived: %s", flatten(got))
}
}
// A DOWN interface holding a stale address must not be reported as if the box were reachable there.
// RED-PROOF 2: drop `|| !i.Up` → this goes red.
func TestFilterHostAddresses_DownInterfaceExcluded(t *testing.T) {
got := filterHostAddresses([]ifaceAddrs{
{Name: "vmbr0", Up: true, CIDRs: []string{"192.168.0.162/24"}},
{Name: "vmbr1", Up: false, CIDRs: []string{"10.9.9.9/24"}},
})
if hasAddr(got, "vmbr1", "10.9.9.9/24") {
t.Errorf("a DOWN interface's address was reported: %s", flatten(got))
}
if len(got) != 1 {
t.Errorf("want exactly the one up interface, got: %s", flatten(got))
}
}
// Order must be deterministic, or every report diff shows phantom churn.
func TestFilterHostAddresses_DeterministicOrder(t *testing.T) {
a := filterHostAddresses(demoFelhomIfaces())
// Same facts, opposite enumeration order.
rev := demoFelhomIfaces()
for i, j := 0, len(rev)-1; i < j; i, j = i+1, j-1 {
rev[i], rev[j] = rev[j], rev[i]
}
b := filterHostAddresses(rev)
if flatten(a) != flatten(b) {
t.Errorf("interface-table order changed the report:\n a=%s\n b=%s", flatten(a), flatten(b))
}
}
// A host with nothing routable yields [] and never nil — an absent key and "no addresses" must not
// look alike on the wire.
func TestFilterHostAddresses_EmptyIsNonNil(t *testing.T) {
got := filterHostAddresses([]ifaceAddrs{{Name: "lo", Up: true, Loopback: true, CIDRs: []string{"127.0.0.1/8"}}})
if got == nil {
t.Fatal("filter returned nil — it would marshal as null, not []")
}
if len(got) != 0 {
t.Errorf("want no addresses, got %s", flatten(got))
}
}
// A malformed entry is skipped, never fatal — one bad address must not cost the report the others.
func TestFilterHostAddresses_MalformedSkipped(t *testing.T) {
got := filterHostAddresses([]ifaceAddrs{
{Name: "vmbr0", Up: true, CIDRs: []string{"not-an-address", "192.168.0.162/24"}},
})
if len(got) != 1 || !hasAddr(got, "vmbr0", "192.168.0.162/24") {
t.Errorf("a malformed sibling address broke the good one: %s", flatten(got))
}
}
// --- the WIRING half: the collector must actually call the filter ---
// The seam defaults to the REAL enumerator, so a forgotten wiring call cannot make this inert.
// RED-PROOF 3: replace the collectAddresses body with `return []HostAddress{}` → red.
func TestCollectAddresses_UsesTheInjectedEnumerator(t *testing.T) {
c := &Collector{logger: slog.Default()}
c.addrEnum = func() ([]ifaceAddrs, error) { return demoFelhomIfaces(), nil }
got := c.collectAddresses()
if !hasAddr(got, "vmbr0", "192.168.0.162/24") {
t.Fatalf("the collector did not run the filter over the enumerator's output: %s", flatten(got))
}
}
// An enumeration failure degrades to [] and a WARN — never a failed report.
func TestCollectAddresses_EnumerationErrorDegrades(t *testing.T) {
c := &Collector{logger: slog.Default()}
c.addrEnum = func() ([]ifaceAddrs, error) { return nil, errors.New("netlink is unhappy") }
got := c.collectAddresses()
if got == nil {
t.Fatal("an enumeration error produced nil, which marshals as null")
}
if len(got) != 0 {
t.Errorf("want [] on error, got %s", flatten(got))
}
}
// The production enumerator must return SOMETHING on the machine running the tests, and must not
// panic. This is the only test that touches the real network stack; it asserts the contract
// (non-nil, no error, loopback correctly flagged) rather than any specific address, because the
// test host's addresses are not ours to predict.
func TestSystemInterfaces_ProductionEnumeratorWorks(t *testing.T) {
ifaces, err := systemInterfaces()
if err != nil {
t.Fatalf("systemInterfaces: %v", err)
}
if len(ifaces) == 0 {
t.Fatal("no interfaces at all — even a container has lo")
}
var sawLoopback bool
for _, i := range ifaces {
if i.Loopback {
sawLoopback = true
}
}
if !sawLoopback {
t.Error("no interface reported the loopback flag — the flag mapping is wrong")
}
// And the filter must survive real input without panicking.
_ = filterHostAddresses(ifaces)
}
+45 -7
View File
@@ -57,8 +57,20 @@ type Loop struct {
logger *slog.Logger
trigger <-chan struct{} // optional: an out-of-band report request (storage watchdog)
observer EnvelopeObserver // optional: the slice-10A desired-state sync hook
// Heartbeat log-pull (v0.83.0): logTailSource yields the debug ring's formatted
// lines newest-kept within a byte budget (applog.Ring.Lines). logTailPending is
// armed by an envelope's log_tail_requested and drained onto the NEXT report —
// the report-channel logtail.go consume-once shape: a failed push leaves the
// hub's request pending, so the next successful envelope re-arms it (fail-safe
// retry, no duplicate shipping). Loop state is single-goroutine (cycle only).
logTailSource func(maxBytes int) []string
logTailPending bool
}
// logTailMaxBytes caps the heartbeat log tail (newest lines kept).
const logTailMaxBytes = 128 * 1024
// NewLoop builds the loop. interval is the starting cadence (the hub may override it
// per-cycle via the control envelope).
func NewLoop(collector collectorIface, client reporter, interval time.Duration, logger *slog.Logger) *Loop {
@@ -69,9 +81,10 @@ func NewLoop(collector collectorIface, client reporter, interval time.Duration,
}
// SetTrigger wires an out-of-band report channel. A receive on it runs one extra
// collect→report cycle immediately WITHOUT disturbing the regular ticker cadence — used by
// the storage watchdog to push a disconnect to the hub in seconds. The watchdog debounces,
// so this fires at most once per debounce window.
// collect→report cycle immediately WITHOUT disturbing the regular ticker cadence. Two producers
// fan into this one channel: the storage watchdog (push a disconnect in seconds) and the
// agent-plane poke listener (v0.89.0 immediate-sync). Both debounce, and the channel is cap-1
// non-blocking, so a burst coalesces to at most one pending extra cycle.
func (l *Loop) SetTrigger(ch <-chan struct{}) { l.trigger = ch }
// SetEnvelopeObserver wires the slice-10A desired-state sync hook. It is called once per cycle
@@ -79,6 +92,10 @@ func (l *Loop) SetTrigger(ch <-chan struct{}) { l.trigger = ch }
// desired-state when the generation advances. Optional — unset is a clean no-op.
func (l *Loop) SetEnvelopeObserver(o EnvelopeObserver) { l.observer = o }
// SetLogTailSource wires the debug ring for the heartbeat log-pull (v0.83.0).
// Optional — unset means an envelope's log_tail_requested is ignored.
func (l *Loop) SetLogTailSource(src func(maxBytes int) []string) { l.logTailSource = src }
// Run reports immediately, then on each tick, until ctx is cancelled (then nil).
func (l *Loop) Run(ctx context.Context) error {
interval := l.interval
@@ -99,10 +116,10 @@ func (l *Loop) Run(ctx context.Context) error {
ticker.Reset(interval)
}
case <-l.trigger:
// Out-of-band report (storage watchdog). Run a cycle now; keep the regular
// cadence (do not reset the ticker). The envelope's interval is still adopted
// Out-of-band report (storage watchdog OR agent-plane poke). Run a cycle now; keep the
// regular cadence (do not reset the ticker). The envelope's interval is still adopted
// if it changed, mirroring the normal path.
l.logger.Info("hub: out-of-band report triggered (storage watchdog)")
l.logger.Info("hub: out-of-band report triggered (watchdog/poke)")
next := l.cycle(ctx, interval)
if next != interval {
l.logger.Info("hub: poll interval changed", "from", interval, "to", next)
@@ -116,19 +133,40 @@ func (l *Loop) Run(ctx context.Context) error {
// cycle runs one collect→report→adopt. It never returns an error: failures are
// logged and the current interval is kept, so the loop keeps running.
func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
start := time.Now()
report, err := l.collector.Collect(ctx)
if err != nil {
l.logger.Warn("hub: collect failed; skipping this cycle's report", "err", err)
return current
}
// Fulfill a pending log-pull: attach the ring tail to THIS report and clear the
// local pending flag (consume-once). On a failed push the hub's request is still
// pending and the next envelope re-arms it — logtail.go's fail-safe retry shape.
// The explicit nil first makes this robust to a collector reusing its report struct.
report.LogTail = nil
if l.logTailPending && l.logTailSource != nil {
report.LogTail = &LogTail{
CollectedAt: time.Now().UTC().Format(time.RFC3339),
Lines: l.logTailSource(logTailMaxBytes),
}
}
l.logTailPending = false
env, err := l.client.Report(ctx, report)
if err != nil {
l.logger.Warn("hub: report failed; keeping current interval", "err", err)
return current
}
if report.LogTail != nil {
// Transparency: the pull is visible in the box's own log (and thus in the ring).
l.logger.Info("operator log pull served", "component", "agent", "lines", len(report.LogTail.Lines))
}
l.logger.Debug("hub: report sent",
"guests", len(report.Guests),
"guests", len(report.Guests), "duration_ms", time.Since(start).Milliseconds(),
"blocked", env.Blocked, "desired_generation", env.DesiredGeneration, "has_signed_ops", env.HasSignedOps)
if env.LogTailRequested {
l.logger.Debug("hub: log tail requested — shipping on the next heartbeat")
l.logTailPending = true
}
// Slice 10A: hand the envelope to the desired-state sync hook (fetch desired-state on a
// generation advance). Done off the report's critical path semantics — a sync/fetch failure
+121
View File
@@ -0,0 +1,121 @@
package hub
import (
"context"
"errors"
"testing"
"time"
)
// recordingReporter records each pushed report and serves a scripted per-call
// (envelope, error) sequence — the S2 heartbeat log-pull harness.
type recordingReporter struct {
reports []*HostReport
script []struct {
env *ControlEnvelope
err error
}
}
func (r *recordingReporter) Report(_ context.Context, rep *HostReport) (*ControlEnvelope, error) {
// Copy the LogTail pointer state at push time (the loop reuses collector reports).
cp := *rep
r.reports = append(r.reports, &cp)
i := len(r.reports) - 1
if i < len(r.script) {
return r.script[i].env, r.script[i].err
}
return &ControlEnvelope{}, nil
}
func tailLoop(rep *recordingReporter) *Loop {
var cn int32
l := NewLoop(&fakeCollector{report: &HostReport{}, n: &cn}, rep, time.Hour, quietLogger())
l.SetLogTailSource(func(maxBytes int) []string { return []string{"line-a", "line-b"} })
return l
}
// S2 (agent half): an envelope's log_tail_requested arms the pull; the NEXT report
// carries log_tail; the one after (request cleared hub-side) carries nothing —
// consume-once. Companion red-proof: drop the `l.logTailPending = false` drain →
// report 3 also carries a tail → the last assertion fails.
func TestLoop_LogTailRequestedShipsOnNextReportOnce(t *testing.T) {
rep := &recordingReporter{script: []struct {
env *ControlEnvelope
err error
}{
{env: &ControlEnvelope{LogTailRequested: true}},
{env: &ControlEnvelope{}}, // the tail arrived — hub cleared the request
{env: &ControlEnvelope{}},
}}
l := tailLoop(rep)
ctx := context.Background()
l.cycle(ctx, time.Hour)
l.cycle(ctx, time.Hour)
l.cycle(ctx, time.Hour)
if len(rep.reports) != 3 {
t.Fatalf("reports = %d, want 3", len(rep.reports))
}
if rep.reports[0].LogTail != nil {
t.Errorf("report 1 must not carry a tail (the request only arrived in its envelope)")
}
got := rep.reports[1].LogTail
if got == nil || len(got.Lines) != 2 || got.Lines[0] != "line-a" || got.CollectedAt == "" {
t.Fatalf("report 2 log_tail = %+v, want the 2 ring lines + collected_at", got)
}
if rep.reports[2].LogTail != nil {
t.Errorf("report 3 carries a tail again — consume-once broken: %+v", rep.reports[2].LogTail)
}
}
// S2 companion (fail-safe retry): the push CARRYING the tail fails → the local pending
// is spent, but the hub's request is still pending, so the next envelope re-arms it and
// the following report fulfills. Asserts the retry ships the tail exactly once more.
func TestLoop_FailedTailPushIsReArmedByNextEnvelope(t *testing.T) {
rep := &recordingReporter{script: []struct {
env *ControlEnvelope
err error
}{
{env: &ControlEnvelope{LogTailRequested: true}}, // arm
{err: errors.New("hub 5xx")}, // the carrying push FAILS
{env: &ControlEnvelope{LogTailRequested: true}}, // hub still pending → re-arm
{env: &ControlEnvelope{}}, // fulfilled
}}
l := tailLoop(rep)
ctx := context.Background()
for i := 0; i < 4; i++ {
l.cycle(ctx, time.Hour)
}
if len(rep.reports) != 4 {
t.Fatalf("reports = %d, want 4", len(rep.reports))
}
if rep.reports[1].LogTail == nil {
t.Errorf("report 2 (the failed push) should have carried the tail")
}
if rep.reports[2].LogTail != nil {
t.Errorf("report 3 must not carry a tail (pending was spent; envelope re-arms only after it)")
}
if rep.reports[3].LogTail == nil {
t.Errorf("report 4 must fulfill the re-armed request — retry lost")
}
}
// No source wired → the request is ignored (clean no-op, no panic).
func TestLoop_LogTailRequestIgnoredWithoutSource(t *testing.T) {
rep := &recordingReporter{script: []struct {
env *ControlEnvelope
err error
}{
{env: &ControlEnvelope{LogTailRequested: true}},
{env: &ControlEnvelope{}},
}}
var cn int32
l := NewLoop(&fakeCollector{report: &HostReport{}, n: &cn}, rep, time.Hour, quietLogger())
ctx := context.Background()
l.cycle(ctx, time.Hour)
l.cycle(ctx, time.Hour)
if rep.reports[1].LogTail != nil {
t.Errorf("tail shipped with no source wired: %+v", rep.reports[1].LogTail)
}
}

Some files were not shown because too many files have changed in this diff Show More