Compare commits

..

75 Commits

Author SHA1 Message Date
admin a165d53c88 REPORT: the second burn-down night
gates / gates (push) Successful in 49s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 00:14:50 +02:00
admin adaf86ad57 R-330: the agent sends SMART 187/188/199 (raw; omitted when unknown)
gates / gates (push) Successful in 53s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 22:13:04 +02:00
admin 74b5eae5b0 R-894: after a restart the agent remembers the last backup per tier
gates / gates (push) Successful in 35s
An unreadable storage right after an agent restart read the off-site tier
DUE (the in-memory record was empty). The newest success per tier is now
kept on disk and read ONLY when the storage cannot be read: fresh -> not
due, older than the cadence -> due, none -> due (unknown) as before. A
storage that answers stays the ground truth.

Ships with v0.150.0 after the 2026-10-07 read-back; nothing delivered.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 20:33:27 +02:00
admin 7e82f325b8 CHANGELOG: the memory-kill check (unreleased, to ship as v0.150.0 after the night read-back)
gates / gates (push) Successful in 50s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 19:36:09 +02:00
admin acccb66bd3 R-528 (09 decision 157): after a Docker engine step the wrapper proves the engine reports a memory kill
felhom-os-apply: a docker-layer apply runs oom_check() after health_after and reports
"oom_check": {result pass|fail|error, oom_killed, oom_event, exit_code, image, detail}.
One throwaway container (the controller's image, --pull never, --network none, 64m cap,
label felhom.oomcheck=1) runs dd bs=200M; pass only with OOMKilled=true AND the oom event
(read after a 2 s settle, --until = guest epoch + 1: measured on demo-hp, an --until taken
right after the run missed the event). docker rm -f always runs in a finally; every call
is bounded (<= 90 s). It never changes the step's outcome or health. New wrapper-only mode
"oom-check" (docker layer) runs the check alone; check_guest etc. still apply.

Agent: WrapperReport/Report gain OOMCheck (json:"oom_check"), copied unchanged in runLayer
and in the kept-copy path.

Tests: 9 wrapper tests + 2 Go tests, each red-proofed (audits/readback-2026-10-07/F/red-*.txt).
Also: test_felhom_os_apply.py's `if __name__` sat mid-file, so 11 tests (UnsentReport,
SaveReportOnDisk, AgentDiesMidPass, CrashLeftTheJournal) never ran as a script or from
TestWrapperSuite; moved to the end (they pass).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 19:34:55 +02:00
admin de812bc027 The shared rule file (09 §3 decision 152), identical to the other copies; no code change
gates / gates (push) Successful in 44s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 16:06:46 +02:00
admin 3e8ebeb96c Instruction files kept true (09 §3 decision 150): stale gate lists, paths and facts corrected; no code change
gates / gates (push) Successful in 46s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 13:43:57 +02:00
admin cefdc731a4 REPORT: the operator's ten answers (2026-10-06)
gates / gates (push) Successful in 24s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 12:28:58 +02:00
admin e56dcb8a4c CHANGELOG/v0.149.0 released (shas)
gates / gates (push) Successful in 37s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:50:21 +02:00
admin f277e619e2 CHANGELOG: unreleased — R-856 GET /host/crash-guard
gates / gates (push) Successful in 38s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:48:40 +02:00
admin 386f51edc6 R-856: GET /host/crash-guard — the host crash guard's last-boot record for the controller (09 decision 143)
The controller waits ~15 minutes with app mails after a crash boot of the host; it learns of the
crash boot from this route. Reads /var/lib/felhom-crash-guard/state.json (read-only, no Proxmox call)
and passes present/last_boot_at/last_boot_unclean/tripped through; a missing, unreadable or garbled
file answers 200 present:false. Guest-token authed like every sibling route.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:48:01 +02:00
admin 130e3ed882 CHANGELOG: unreleased — R-444 weekly guest disk trim, R-99 runbook pointer
gates / gates (push) Successful in 41s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:25:22 +02:00
admin be398f92e8 R-99: the PBS phantom WARN names the cleanup runbook (09 §3 decision 140)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:24:45 +02:00
admin ee71abd1d4 R-444: weekly guest disk trim (pct fstrim) outside the night, under the heavy-op gate
Operator ruling 09 §3 decision 139. One exact sudoers rule FELHOM_FSTRIM
(`/usr/sbin/pct ^fstrim [0-9]+$`) + manifest entry guest-fstrim; new
internal/fstrim job: due Wednesday from 10:00 host-local, starts only
10:00-20:59, holds backup.InFlight (busy -> deferred to the next hourly
tick), failed trim retried at most 3x per week, bytes parsed from the
"(N bytes) trimmed" lines, last result per guest persisted in
<state_dir>/guest-disk-trim.json and reported as guest_disk_trim.
Config opt-out: "disk_trim": {"disable": true}.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:24:45 +02:00
admin 37e98f452b REPORT: the burn-down night (2026-10-06)
gates / gates (push) Successful in 45s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 02:13:10 +02:00
admin b2b82ae828 bundle test: the ISO first-boot files the installer names under KEPT are not installer-written (go test red since felhom.eu 85de3f9b); CHANGELOG unreleased (R-426 decoys)
gates / gates (push) Successful in 42s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 01:42:25 +02:00
admin b78a0ff3ac R-426: decoys for the shared reuse-refs, instructions and observations gates
COVERS "reuse-refs", "instructions", "observations": the three shared
felhom.eu scripts run against a scratch clone of THIS repo (in a scratch
workspace symlinking the sibling clones they reach across to), so the
plant is in the agent's own REUSE.md / CLAUDE.md / REPORT.md. Convicted:
a missing cited .go and .md path, a version literal in CLAUDE.md's
effective text, R-419's prose-only Observations note. Passed: the real
files, the version inside an HTML comment, both genuine markers.
DECOY_SHARED_DIR lets a red-proof judge a mutated copy of the shared
scripts without editing the felhom.eu clone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 01:42:03 +02:00
admin 96047453cb R-426: decoys for the release-complete gate
COVERS "release-complete": the working-tree gate runs in a scratch clone
whose origin is a scratch bare repo, against the fake Gitea. Convicted:
the newest `## vX.Y.Z` with no tag anywhere, a tag parked on an unrelated
commit, a tag with no package, and no-tag wins over a registry 500.
Inconclusive: a registry 500. Passed: the genuine release, an
`## Unreleased` heading above it, a newer version named only in prose or
under `###` (the withdrawn sweep decoy, now asserted the right way round),
a tag only origin has (the shallow-CI shape), and a LOCAL-only tag BY
DESIGN (CI's fresh clone and the published gate's converse probe see it).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 01:42:03 +02:00
admin 4bf5db6875 R-426: decoy suite for the published gate, against a fake Gitea
scripts/test_gate_decoys.py (new; COVERS "published"): an http.server on
127.0.0.1 stands in for Gitea through the gate's existing GITEA_BASE
seam, proxies stripped, so no case reaches the real registry. Facts
convicted: a tag whose package 404s, a tag tree without the configs, a
package one patch past the newest tag (never tagged), a patch-gap orphan,
a missing package that lexical sorting would drop out of the retention
window. Inconclusive, never a pass: tags api 500, a non-JSON 200, Gitea
unreachable. Passed: a clean registry, a non-semver tag, a version older
than the retention window (BY DESIGN).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 01:42:03 +02:00
admin 87977ff40a CHANGELOG/v0.148.0 released (shas)
gates / gates (push) Successful in 42s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 00:37:07 +02:00
admin 861d32a4b4 CHANGELOG: unreleased — R-349 agent_sha256, R-25 agent half (burn-down night)
gates / gates (push) Successful in 43s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 21:16:51 +02:00
admin 769c4c3cf2 R-25 (agent half): the format answer carries the NEW filesystem's UUID, bound to the durable id
After mkfs the agent re-resolves the bound durable id, requires it to name the
device it just formatted, reads the superblock back (blkid -p, requested fstype)
and returns fs_uuid in POST /disks/format and GET /disks/format/status. Anything
unverified returns "" — never a path-resolved guess. The controller half
(mount fs_uuid instead of re-resolving the UUID from the /dev path) is owed
in felhom-controller.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 21:16:11 +02:00
admin 64f704d0f7 R-349: the host report carries the sha256 of the RUNNING agent binary
A hand-built proof binary and the published artifact share a version string
but not their bytes, so no version check could see the divergence. The agent
now reports agent_sha256 (hash of /proc/self/exe, once per process; empty =
unknown) beside agent_version, the same mechanism as host.wrapper_sha256.
The hub half (compare against the vouched agent_sha256, surface drift) is
owed in the hub repo.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 21:16:11 +02:00
admin 208fac8027 CHANGELOG/REPORT: v0.147.0 released (shas)
gates / gates (push) Successful in 56s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 18:58:37 +02:00
admin f1b9b41214 R-124 recipe root namespace as PBS spells it; R-118 no root size for an absent drive; R-269 rotated-out token rejected at once; R-317 dnsmasq install probed by its unit (burn-down round 2)
gates / gates (push) Successful in 47s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 18:56:13 +02:00
admin d83316326e R-291 retention record source, R-348 restart comment (no binary change; burn-down)
gates / gates (push) Successful in 22s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 16:56:18 +02:00
admin e06ed97fa8 agent v0.146.1 REPORT
gates / gates (push) Successful in 22s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 13:00:22 +02:00
admin e4b5cf9693 R-880: build-step-bundle.py — the transition bundle for a release whose bundle adds paths (an installed felhom-os-apply refuses unknown paths, R16)
gates / gates (push) Successful in 21s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:23:55 +02:00
admin faa3cad92e agent v0.146.1 CHANGELOG (released)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:14:07 +02:00
admin fdd87178d2 R-861 review fixes: the signed update hands the A/B wrapper a root-owned copy of the hashed bytes; mount units accept no Wants/Requires/Before and no continuation lines; the escrow read walks the path without following any symlink
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:13:42 +02:00
admin 0342c7bb57 agent v0.146.0 CHANGELOG (released)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:01:25 +02:00
admin 6ab1e7c56c R-861: narrow the agent's root grants — exact sudo patterns, felhom-priv-apply content checker, fixed hook/parent files in the bundle, signed self-update verified as root, escrow root reads pinned
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:00:43 +02:00
admin 61345790ed REPORT: the 2026-10-05 catch-up session
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 10:29:12 +02:00
admin 7c4b8e599f v0.145.0: CHANGELOG (released 894da35c…, bundle 78c00adc…)
gates / gates (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 09:23:48 +02:00
admin 56ef1d6655 v0.145.0 code: the OS wrapper repairs dpkg's update journal by itself after a power cut (R-876); restore-test first check 30 min after start (R-874); neutral "sent late" text (R-875)
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 09:23:23 +02:00
admin 78c890e4bb REPORT: the 2026-10-05 night-fixes session
gates / gates (push) Successful in 21s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 08:25:49 +02:00
admin d48f1bbb23 v0.144.1: CHANGELOG (released 6ccd521d…, bundle e89a9ddf…)
gates / gates (push) Successful in 21s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 07:49:29 +02:00
admin 8401a30917 v0.144.1 code: the wrapper survives a dead reader (BrokenPipe) so a killed pass still keeps its report; the agent looks for kept copies every 5 min (R-868, measured live)
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 07:49:04 +02:00
admin d1b6004458 v0.144.0: CHANGELOG (released f18093c3…, bundle 6acf42fe…)
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 07:15:07 +02:00
admin ca78c17b29 v0.144.0 code: R8 measures the real download (R-865); an OS pass's report survives a killed agent (R-868); the debug pass runs from the saved block when the hub is away (R-866)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 07:14:32 +02:00
admin c8d12f1f2a REPORT + CONTEXT: 2026-10-04 night (R-840 / R-860)
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 20:39:47 +02:00
admin dc9164c5af CHANGELOG v0.143.0 (R-840); build-golden.sh 3.2.0: GOLDEN_GUEST_PKGS — the approved guest release at bake time, first-night count
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 20:08:16 +02:00
admin c9fa2e717b R-840: the config bundle — a signed agent_config_update brings a box's root-owned files (sudoers, wrappers, units)
gates / gates (push) Successful in 18s
felhom-os-apply gains mode 'bundle' (signed, verified by the wrapper itself against the root-owned
signers file — or, when that file is missing, only the installer's pinned key, which it then creates)
and --install-bundle (the installer's root entry). BUNDLE_FILES is the one table of paths; every check
(visudo, sh/bash -n, python, unit sections, RuntimeDirectory guard, nft -c, the route itself) runs
before the first write; a failed write or self-check puts every previous copy back. The trust root is
never a bundle path (R17). scripts/build-config-bundle.py builds it reproducibly; release-agent.sh
publishes it beside the binary. The agent reports the bundle record in system.config_bundle.
felhom-opsign signs agent_config_update. 43 wrapper tests (22 mutants red), Go executor tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 19:36:58 +02:00
admin 0af1187e03 CHANGELOG + REPORT: v0.142.1 released (R-858)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 18:21:46 +02:00
admin 495003051b R-858: after a Docker engine step the wrapper restarts ONLY the containers that mount the docker socket (controller, traefik); the health rule fails when the controller cannot reach Docker from inside its container (ruling 95)
gates / gates (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 18:21:24 +02:00
admin 42af3ab9bc build-golden.sh 3.1.0: live-restore on in the golden (fail-closed assertion), GOLDEN_DOCKER_PKGS pins the approved Docker engine set
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 17:30:19 +02:00
admin f24dce5b95 CHANGELOG + REPORT: v0.142.0 released
gates / gates (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 16:15:15 +02:00
admin b1746c25af Docker engine slow lane (live-restore once by reload; ring-0 pending-docker under a root-owned ring-0 mark; ring 1 and undo only by a signed os_docker_step the wrapper re-verifies against a root-owned signers file; same-container-id health), the version report (facts mode -> host report system stanza, R-852), guest restart scan every pass (R-849), the crash guard (kernel.panic=10, the 3rd unclean stop in 60 min stays off, 24 h re-arm)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 16:14:36 +02:00
admin 2e2e8f56b8 CHANGELOG + REPORT: v0.141.1 released (host reboot-needed fixes)
gates / gates (push) Successful in 17s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 13:39:35 +02:00
admin a6bc3f1197 os-apply: the host restart scan no longer hides lxc-start (skip ':/lxc/' not 'lxc'), and the host scans on every pass so a reboot clears 'reboot needed' (reboot_scanned reaches the hub) — both found live on demo-felhom
gates / gates (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 13:39:14 +02:00
admin 3bf77c3423 CHANGELOG + REPORT: v0.141.0 released (host fast lane, true tunnel status, fast leg)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 13:19:02 +02:00
admin cfba0d022a contract: the desired-state golden gains host_release (byte-identical with hub v0.131.0); TestOSUpdateGolden_Decodes checks it
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 13:18:43 +02:00
admin c3d08b4821 OS updates host fast lane + true tunnel status + fast leg: wrapper host layer (R12 appliance proof from the root-owned install record, R14 kernel/boot/firmware refused), select pending-fast, one call per layer, host-side version checks, restart scan only after an install, reboot-needed for PID 1/lxc-start; the leg runs the host step after a healthy guest step; GuestTunnelProber reads the cloudflared container + its readiness check (R-841)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 12:54:02 +02:00
admin a55eedcf2c CHANGELOG + REPORT: v0.140.0 released (OS updates, guest fast lane)
gates / gates (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 11:30:14 +02:00
admin 9cac3462bb osupdate: the health baseline is the start of the leg (inventory reading merged with the apply's own) — an app that stops during the run fails it (found live on demo-hp)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 11:22:03 +02:00
admin 1bb8608e88 os-apply: tell an UPDATED conffile from a KEPT one (dpkg's two shapes, measured live on demo-hp)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 11:08:43 +02:00
admin b84e0dd1bd selftest flag accepts os-update and wgtunnel (both dispatched, both refused); a test pins every dispatched mode
gates / gates (push) Successful in 19s
Found live 2026-10-04: the OS leg's debug action could not run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 11:04:03 +02:00
admin 23a8ef3de4 OS updates, guest fast lane (11 §8 step 2): felhom-os-apply wrapper (R1-R13 refusals, repair first, snapshot.debian.org fallback), FELHOM_OSAPPLY sudoers, the OS leg after the primary backup, hub os_update block + os-report, --selftest=os-update
gates / gates (push) Successful in 18s
No automatic undo: a customer guest cannot be snapshotted (R-837, measured).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 10:44:29 +02:00
admin 596238cc2e CHANGELOG + REPORT: v0.139.0 released (R-834)
gates / gates (push) Successful in 17s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 08:52:57 +02:00
admin 475bdce7e4 DR bring-up refuses beside a live original (R-834): source guest present, drives bind, or unreadable config
gates / gates (push) Successful in 18s
The DR route keeps onboot 1, binds the real drives and starts the guest: right on a replaced
host, a second box on the same drives beside a live original. The restore-test's no-host-bind
half is now pinned too (measured safe live on demo-hp 2026-10-04).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 08:52:35 +02:00
admin d766666ff8 CHANGELOG: v0.138.0 vouched with golden 0.283.1
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 11:40:20 +02:00
admin a4c09a7c11 REPORT: v0.138.0 (R-727)
gates / gates (push) Successful in 16s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 11:00:45 +02:00
admin 904dc20466 CHANGELOG: v0.138.0 released (R-727)
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 10:34:25 +02:00
admin e1b8269be0 restore test takes only this box's archives (R-727): an archive encrypted with another key is another box's
gates / gates (push) Successful in 16s
Red-proof RP39. Released as v0.138.0 by release-agent.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 10:33:52 +02:00
admin 5c68c869b6 docs: v0.137.0 vouched with golden 0.276.0 (2026-09-28)
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-28 09:45:08 +02:00
admin 728d12b1a0 CHANGELOG: v0.137.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:01:26 +02:00
admin dd81866b16 REPORT: v0.136.0 and v0.137.0
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:01:02 +02:00
admin 3ef095fb71 agent: a guest outside the agent's ACL is not a known guest (R-689, v0.136.0 regression)
gates / gates (push) Successful in 14s
PVE answers 403 permission denied, not "does not exist", for a vmid outside the felhom pool;
v0.136.0 turned that into a lookup failure and the local tier read UNKNOWN every evaluation
(measured on demo-hp). Such an archive is skipped. Red-proofed; verified read-only on demo-hp
with the pre-release binary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:00:42 +02:00
admin 7c986915ca CHANGELOG: v0.136.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:46:22 +02:00
admin 16dbc83221 agent: the restore test takes only archives of a guest that still exists (R-689, second half)
gates / gates (push) Successful in 14s
Measured on demo-hp right after v0.135.0: with the golden skipped the pick fell to a leftover
archive of guest 9100, deleted in August. "does not exist" skips it; any other lookup error
makes the tier unknown. Red-proofed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:45:56 +02:00
admin 9555a7f93b REPORT: v0.135.0 released and signed-delivered to both demo hosts
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:02:10 +02:00
admin 9ff937d8fb CHANGELOG: v0.135.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 12:16:17 +02:00
admin d4be12ca95 agent: the restore test proves only backups of a guest (R-689)
gates / gates (push) Successful in 13s
The golden template in local:backup/ was picked as the newest settled archive on demo-hp and
failed every 6 h. Candidates are now vzdump-<type>-<vmid> files or PBS ct|vm/<vmid> snapshots
with a reported vmid. Red-proofed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 12:15:42 +02:00
admin 7403c2a838 REPORT + CONTEXT: v0.133.0 and v0.134.0 delivered to both demo boxes; restore test back on
gates / gates (push) Successful in 13s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-25 04:55:26 +02:00
admin 309e368731 CHANGELOG: v0.134.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-24 22:49:19 +02:00
124 changed files with 13036 additions and 644 deletions
+3 -2
View File
@@ -21,6 +21,7 @@ fast, and wrong.
This rule used to be duplicated verbatim in felhom-agent/CLAUDE.md with a note explaining that
felhom.eu/CLAUDE.md "does not load in an agent-only session". That reasoning was correct before
path-scoped rules existed. The single source is now felhom.eu/CLAUDE.md "Code quality rules"; this
file is the scoped copy that loads exactly where health checks are written. (2026-08-06)
path-scoped rules existed. Deliberate scoped copies now live in felhom.eu/.claude/rules/hub.md and
felhom-controller/.claude/rules/gates.md (hub.md's comment names them); none is the single source. This
file is the copy that loads exactly where agent health checks are written. (2026-08-06; corrected 2026-10-06)
-->
+69
View File
@@ -0,0 +1,69 @@
---
unconditional: true
---
# Unprompted work — rules for any session without a task file
> Goal sessions, nightly sessions, "work the register" sessions. **A session that starts from
> `/goal` or a standing brief inherits these rules exactly as it inherits the gates.** They are the
> part of `PROMPT-TEMPLATE.md` that a task file used to carry and a goal does not. Same wording lives
> in `felhom.eu`, `felhom-controller`, `felhom-agent` and `app-catalog-felhom.eu` `.claude/rules/`, and in the workspace
> root's unversioned `.claude/rules/`; change all five or none.
## 1. What you may pick up on your own
- A register row **you or another CC session filed**, with owner CC, at P3 or a bounded P2, that
needs **no operator decision**, touches **no customer data by design**, and introduces **no
mechanism nobody has measured**. Smallest first.
- A defect you find while exercising the product, filed as a row **before** you fix it — **unless it is small**:
a small finding is fixed in the session and never filed (the size rule, `OPEN-ITEMS.md` „How a row is filed").
- Hygiene: register compression, stale citations, rows with no owner, documents that contradict
live source.
**Not yours, ever, without a task file or an operator word:** money; anything that changes risk to
customer data; anything that changes a promise the product makes to a customer; anything that
reverses a documented design decision (`documentation/architecture/` — a design decision is not a
defect, R-370); anything on DooPlex or ep0; baking or vouching a golden; promoting a
catalog version; a new external dependency.
## 2. When you may decide instead of ask (operator grant, 2026-09-14)
You may take a decision yourself when **all** of these hold: the architecture folder and the register
give a clear direction; your choice follows that direction; it is reversible without customer-data
risk; and you can write it in the `09-update-architecture.md` §3 shape — one answerable sentence, the
options, what each costs, why this one. **Then record it** as a dated decision in `CONTEXT.md` and
the owning architecture document, tagged *decided by CC unattended — operator may reverse*, and put
it **first** in the morning note. A decision you cannot write in that shape is one you do not take.
## 3. The discipline a task file used to carry
1. **Baselines first.** Read each repo's `main` hash and version from live source before touching it.
2. **Read the architecture document for the area, and name it** in the report, before any claim.
3. **Red-proof every correctness fix.** A test never seen failing has not been shown to test anything.
4. **Live-validate on a Tier-0 box** through the endpoints the UI invokes. `demo-hp` is `ssh hp`.
Throwaway apps only; the standing apps and `bentopdf` stay.
5. **Evidence off the machine at the end of each phase**, before any revert (R-320).
6. **One release per repo per session**, with a CHANGELOG entry (controller: with its `MinAgent`
line), REPORT overwritten, floor raised to deliver it. **No golden unless a drill or fresh install
needs one** (the waiver, R-468). **No `--no-verify`.**
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
Prose is not a record.
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
9. **Never leave a half-state.** If time runs out, revert to clean and say what was reverted.
10. **Teardown, three layers, stated** — machine, host, hub — or "provisioned nothing".
## 4. The morning note
One screen, plain language, in this order: **decisions you took** (§2) first; what you exercised;
what broke and whether you fixed it; rows opened and closed with the register size before and after;
what needs the operator, each with what happens if they do nothing. No file paths, no function
names, no row numbers as the subject of a sentence.
## 5. Instruction files
**Instruction files (`CLAUDE.md`, `.claude/rules/*`) are kept true by the session that finds them wrong**
(operator ruling 2026-10-06, `09` §3 decision 150). A session MAY, without asking: correct a stale fact (a command, a
count, a version, a path, a description of what a gate does), add a fact it proved, and remove a reference to something
that no longer exists. Each edit is named in the report (file, line, before, after, why). A session MAY NOT, without the
operator's word: loosen a safety rule, a fence, a „never", a protected machine, a secret rule, or a review step; or
remove a rule. When in doubt, it is a rule change, and it goes to the operator. If Claude Code's own permission check
asks before such an edit, wait for the operator's click; if it refuses, record that and file the exact line.
+3
View File
@@ -9,3 +9,6 @@
# go
/vendor/
# Python bytecode written by configs/test_felhom_os_apply.py
configs/__pycache__/
+507
View File
@@ -1,3 +1,510 @@
## Unreleased (2026-10-06 night, later) — after a restart the agent remembers the last backup per tier (R-894); three more SMART counters on the wire (R-330)
Ships with the memory-kill check below as v0.150.0, AFTER the 2026-10-07 night read-back. Nothing delivered tonight.
- **The defect (measured 2026-10-05 on demo-hp):** the agent restarted at 04:57; at 06:25 the off-site storage answered *Can't connect*; the per-tier backup record is in memory only, so the due-check fell back to an EMPTY record and the 7-day tier (last copy 4 days old) read DUE; the controller asked and vzdump failed.
- New `internal/backup/backup_state.go` `BackupSuccessState`: the newest SUCCESSFUL backup per tier and guest, on disk (`<oob state dir>/backup-success-state.json`, atomic tmp+rename, 0600). Only successes are written; a corrupt file reads as nothing known.
- `internal/localapi` `handleBackupDue`: when the tier's storage CANNOT be read, the saved copy stands in for the in-memory record. A fresh copy → not due („… (storage unreadable — age from the last success saved on disk)"); a copy older than the cadence → DUE; no copy → the old answer (DUE, age unknown). A storage that answers stays the ground truth: an archive absent there is due even when the file remembers one.
- Wired in `buildLocalAPIServer` (`LastKnownBackups`); the local API's backup job saves each success.
- Tests: `TestBackupDue_R894_*` (restart = a new server and a new state from the same file; fresh / old / none / storage answers / failed backup not saved), `TestBackupSuccessState_*`, `TestR894_LastKnownBackupsIsWiredIntoTheDaemon` (AST). Four red-proofs observed (`felhom.eu/documentation/audits/night-burndown-2026-10-06/s4/`).
- **R-330 (disk health Phase 2, the wire only):** the SMART summary carries three more SATA raw counters — `reported_uncorrect` (187), `command_timeout` (188, carried as the vendor reports it; some pack several counters), `udma_crc_errors` (199). Pointer + omitempty: an attribute the drive does not report is OMITTED (unknown), never 0. No verdict reads them yet. Tests `TestParseSMART_R330_*` (two red-proofs, `felhom.eu/documentation/audits/night-burndown-2026-10-06/r330/`).
## Unreleased (2026-10-06 night) — the Docker step proves the engine reports a memory kill (`09` §3 decision 157, R-528)
To be released as v0.150.0 with its config bundle AFTER the 2026-10-07 night read-back (the night of 2026-10-06 runs v0.149.0 on purpose).
- `configs/felhom-os-apply`: after a docker-layer APPLY (after `health_after`) the wrapper runs `oom_check()`: a throwaway container from the image the running controller uses (`--pull never`, `--network none`, no volume, label `felhom.oomcheck=1`, 64 MB cap) asks for one 200 MB block; „pass" only when `OOMKilled=true` AND the `oom` event; it waits 2 s and reads the events window to the guest's epoch + 1 (measured: a window closed in the same second missed the event); the container is always removed. Reported as `oom_check`; it never changes the step's outcome or health. A wrapper-only mode `oom-check` runs the check alone (no apt, no engine change), by hand as root.
- `internal/osupdate`: `WrapperReport` and `Report` carry `oom_check` verbatim, on the normal pass and on the kept-copy path (R-868).
- `configs/test_felhom_os_apply.py`: its `unittest.main()` sat in the middle of the file, so 11 tests (UnsentReport, SaveReportOnDisk, AgentDiesMidPass, CrashLeftTheJournal) never ran — moved to the end; all pass.
- Tests: the OOMCheck class (pass, OOMKilled=false, no event, unreadable image, removal on an inspect error, not on other layers or in health mode, the events window after the settle wait, mode oom-check alone and its refusals); TestDocker_OOMCheckReachesTheHubUnchanged, TestR868_KeptCopyCarriesTheOOMCheck. 12 red-proofs in `felhom.eu/documentation/audits/readback-2026-10-07/F/`.
## Unreleased (2026-10-06 evening) — the shared rule file (`09` §3 decision 152); no code change
- `.claude/rules/unprompted-work.md` added, byte-identical to the copies in felhom.eu, felhom-controller, app-catalog-felhom.eu and the workspace root (checked with `diff` against the controller's copy and one md5 across all five). Its copies line names five copies.
## Unreleased (2026-10-06 afternoon) — instruction files kept true (`09` §3 decision 150); no code change
- `CLAUDE.md` „Gates — ONE entry point": the runner runs every gate in its `GATES` table (five: three shared, `published`, `release-complete`); `--fast` skips `published` (network). It said two gates and „all of them".
- `CLAUDE.md`: the decoy gate and its audit are named with their `felhom.eu/` prefix (they do not exist in this repo).
- `.claude/rules/health-checks.md` (comment): the health-check rule's copies live in felhom.eu `hub.md` and the controller's `gates.md`; it named felhom.eu `CLAUDE.md` „Code quality rules", which holds no such rule.
## v0.149.0 — a weekly disk trim of each customer guest, the crash-boot fact for the controller, the phantom WARN names its runbook (R-444, R-856, R-99; operator rulings `09` §3 139, 143, 140) (2026-10-06)
Released by `scripts/release-agent.sh`: binary sha256 `6bcae9c2eb5d97e8285316583870059835793893299e291891a53a4ce505585f`
config bundle sha256 `e182c82dcf4a67faa3bcb74dbe4ffa7b06e0b27dc8451cb7574d6339ce91ad66` (tag `v0.149.0` = `f277e61`).
**The bundle carries the new sudoers rule for the trim (`FELHOM_FSTRIM`) — deliver it with the binary:** signed
`agent_update`, then signed `agent_config_update`.
- R-856 (`09` §3 decision 143): new local-API route `GET /host/crash-guard` — passes the host crash guard's last-boot record (present, last_boot_at, last_boot_unclean, tripped) from /var/lib/felhom-crash-guard/state.json to the controller, which waits ~15 min with app mails after a crash boot. Read-only, no Proxmox call, guest-token authed; a missing/unreadable/garbled file answers 200 present:false (never an error page). An older agent answers 404, which the controller reads as unknown (normal 90 s grace) — no controller MinAgent raise needed.
- R-444 (`09` §3 decision 139): weekly guest disk trim. New sudoers alias FELHOM_FSTRIM with ONE exact rule `/usr/sbin/pct ^fstrim [0-9]+$` (rides the signed config bundle; decoys pinned by TestSudoersFstrimRuleIsExact) and capability guest-fstrim (non-critical). New internal/fstrim job: each owned RUNNING guest gets `pct fstrim <vmid>` once a week - due Wednesday from 10:00 host-local, starts only 10:00-20:59 (never the 01:00-06:59 night), holds the one-heavy-op gate so it never runs beside a backup or restore-test (busy -> deferred to the next hourly tick; a box that was off catches up at its next daytime hour); a failed trim WARNs and is retried at most 3 times that week; bytes parsed from `pct fstrim`'s "(N bytes) trimmed" lines; positive log `fstrim: guest N trimmed X GiB in Ys`; last result per guest persisted in <state_dir>/guest-disk-trim.json and reported as the new omitempty host-report stanza `guest_disk_trim`. Opt-out: agent.json "disk_trim": {"disable": true}.
- R-99 (`09` §3 decision 140): the agent's WARN for a PBS archive below the 1 MiB plausibility floor now ends with the pointer to the sanctioned cleanup (`documentation/runbooks/pbs-phantom-cleanup.md`); detection only — nothing is deleted automatically. Dir-storage archives keep the old text.
## unreleased
- R-426: scripts/test_gate_decoys.py (new) — the published gate judged against a fake Gitea (127.0.0.1, via GITEA_BASE; never the real registry): 11 cases; COVERS published — `felhom-agent/published` leaves the decoy-coverage EXEMPT list.
- R-426: release-complete gate — 10 decoy cases (scratch clone + scratch bare origin + fake Gitea); COVERS release-complete — `felhom-agent/release-complete` leaves the decoy-coverage EXEMPT list.
- R-426: the shared reuse-refs/instructions/observations gates get agent-side decoys (9 cases on a scratch clone of this repo); COVERS reuse-refs, instructions, observations — three `felhom-agent/*` entries leave the decoy-coverage EXEMPT list.
- **Fixed without a row:** `configs/test_felhom_config_bundle.py` read the two ISO first-boot files that installer 1.32.0 now NAMES under KEPT (R-275) as files the installer writes — `go test ./internal/osupdate` was red on DooPlex from 21:25 to 01:55 (felhom.eu `85de3f9b`); they are listed with why. Test only; agent v0.148.0's code is unaffected.
## v0.148.0 — the host report names the running binary's sha; the format answer carries the new filesystem's UUID (burn-down night: R-349, R-25 agent halves) (2026-10-06)
Released by `scripts/release-agent.sh`: binary sha256 `3e68a0870e0e2ce262cb4819294edddeb0a73e8c558a31611a20329a6d9ee283`
config bundle sha256 `a6fa4f589d184b58c9911303bd087e300be1e75b3647e4302c9594df6989c4de` (tag `v0.148.0` = `861d32a`).
Delivery order as for v0.147.0: signed `agent_update`, then signed `agent_config_update`.
- **R-349:** the host report carries `agent_sha256`, the sha256 of the running agent binary (read once from
`/proc/self/exe`; empty = unknown), so a hand-built binary under the vouched version name becomes visible. The hub
comparison is a separate hub change. Test `TestCollect_AgentSHA256IsTheRunningBinary`; red-proved.
- **R-25 (agent half):** `POST /disks/format` and `GET /disks/format/status` return `fs_uuid`, the new filesystem's UUID
read back after mkfs only when the bound durable id still resolves to the formatted device and the superblock is the
requested type (empty = not verified); `DeviceProbe` gains `FSUUID` from blkid. Tests `TestFormat_*FSUUID*`; three
red-proofs. The controller half (mount that UUID) is a controller change.
## v0.147.0 — the recovery recipe spells the root namespace the way PBS does; a removed drive no longer shows the root disk's size; a rotated-out token stops at once; the dnsmasq check looks at the right package (burn-down round 2: R-124, R-118, R-269, R-317) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `642c4d196c48671c14ff653118303c5903af1670b7abb550edeaf73e701cd5b8`,
config bundle sha256 `326527d0993c9a62df2f790c7700ca645cedbf0673dcfb6dc1768d8610b8007d` (tag `v0.147.0` = `f1b9b41`).
Delivery order as for v0.146.1: signed `agent_update`, then signed `agent_config_update`.
MinAgent impact: none (the controller needs nothing new from this agent). Config bundle content unchanged from v0.146.1.
- **R-124 (operator ruling 2026-10-05: fix it):** the DR recipe's `pbs.namespace` for a box in PBS's ROOT namespace is
now `""` — PBS's own spelling — beside `namespace_state: resolved`; it used to be the word `root`, which no namespace
is named, so `--ns root` failed in a recovery. `hub.PBSRootNamespace`; `TestR124_RootNamespaceOnTheWireIsPBSSpelling`
(red-proof: back to "root" → FAIL). Runbook: `felhom.eu runbooks/ep0-datastore-copy.md` step 2 says how to read it
(and to treat a recorded `root` from older agents as empty). The hub stores the recipe raw; its fixture follows.
- **R-118:** the local API's drive list reads a drive's capacity only while its DEVICE is present — with the device gone
the bare mountpoint is a directory on the root filesystem, whose size was reported as the drive's.
`TestDisks_UnionPath_AbsentDeviceReportsNoRootCapacity` (red-proof convicts).
- **R-269:** the token store re-reads its shared file whenever it has grown, BEFORE answering — so a token rotated out by
another process stops authorizing on its next use (it used to keep working until an unrelated miss). One `stat` per
call. `TestTokenStore_RotatedOutTokenRejectedFirst` (red-proof convicts).
- **R-317:** the LAN resolver decides whether to install `dnsmasq` by its service UNIT, not by `/usr/sbin/dnsmasq` (which
the `dnsmasq-base` package also ships). Same `apt-get install` command; no sudoers change. `TestEnsureDnsmasq_*`
(red-proof convicts). Red-proofs: `felhom.eu/documentation/audits/burndown2-2026-10-05/agent-red-proofs.txt`,
`r124-red-proof.txt`.
Also in this release (no binary effect; from burn-down round 1):
- **R-291:** `scripts/retention-policy.json` names where its 10 comes from — the R-267 newest-10 prune of generic
packages, established 2026-08-10 (R-287) — instead of „observed, no located ruling"; the non-existent
`registry-retention.md` reader is dropped. `check-published-versions.py` still reads 10 (checked).
- **R-348:** `internal/backup/store.go` no longer says backups are „unaffected" by a restart: the reported backup list
reads 0 until the next backup runs; only the hub's verdict (7-day look-back) is unaffected.
## v0.146.1 — R-861 review fixes: the signed update flips a root-owned copy; no Wants=/continuations in mount units; the escrow read follows no symlink anywhere (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `badd6c9a2e40c8bfe856d2d1a203443b21b7eb92ecc35d6090ab44518d4d082a`,
config bundle sha256 `42333e969028867ad8142335e6c1bc4040eec231de0d8d330c2d4b2cf7bc3442`. **Supersedes v0.146.0, which
was released but never vouched or delivered to any box.** The same order applies: signed `agent_update` first, then the
signed `agent_config_update`.
**Delivery needs a STEP bundle (R-880, found while delivering).** An installed `felhom-os-apply` checks an incoming
bundle's paths against its OWN table (R16), so every box on the v0.145.0 bundle REFUSES the v0.146.1 bundle (it adds 4
paths). `scripts/build-step-bundle.py` builds the transition: the box's current bundle with ONLY `felhom-os-apply`
replaced (same paths — the old wrapper accepts it), published as bundle version `0.146.1-step1`; then the release's own
bundle. Order on a box: `agent_update` 0.146.1 → `agent_config_update` 0.146.1-step1 → `agent_config_update` 0.146.1.
Tests `StepBundle` (the R16 refusal reproduced; the step accepted; exactly one file changed). Tooling only — not in the
binary or the bundle.
A background security review of the v0.146.0 commit found three holes in the new code; each is fixed and red-proved
(`felhom.eu/documentation/audits/hub-safety-2026-10-05/partF/red-proof.txt`, S1–S3):
- **S1 — a race in the signed update.** `felhom-os-apply` hashed the agent's staged file and then let the A/B wrapper copy
it BY PATH; the agent owns that directory and could swap the file in between. Now the root step reads the file ONCE
(`read_staged_once`: O_NOFOLLOW, fstat, owner, size), hashes those bytes, writes them to a root-owned directory
(`/var/lib/felhom-os-apply/agent-update/`) and hands ONLY that copy to `felhom-selfupdate-guarded apply`, which now
refuses any other directory, a symlink, or a file not owned by root. Tests: `AgentUpdate` (+1),
`SelfupdateWrapperConfinement`.
- **S2 — an allowlist escape in `felhom-priv-apply`.** `[Unit]` accepted `Wants=`/`Requires=`/`Before=` naming any unit, so
a mount unit could start e.g. `reboot.target`. `[Unit]` now holds only `Description` and `After=local-fs-pre.target`
(what the renderers write), and any line ending in a backslash (a systemd continuation this parser would read
differently) is refused. Tests `test_U2_wants_starts_another_unit`, `test_U2_continuation_line`.
- **S3 — a path traversal in the escrow read.** `O_NOFOLLOW` guards only the last component; a symlinked DIRECTORY in the
agent's own state dir still redirected the root read. `readStagedNoFollow` now walks the path from `/` with
`openat(O_NOFOLLOW)` per component. Test `TestAttach_RefusesASymlinkedDirectory`.
## v0.146.0 — the agent's root grants narrowed: exact sudo patterns, a root content checker, fixed files from the bundle, the signed update checked as root (R-861) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `b860af465076041e07f35fed1b12d64ae2b2985d8995f0ce167418d39c2b00d5`,
config bundle sha256 `161c737e523aa7910cf32ce41b83f989569bee55b8c5938e7211c92aef68548e`. **Order on a box: the signed
`agent_update` FIRST (the old bundle still grants the old flip), then the signed `agent_config_update`.** Between the
two (minutes) the new agent's checker calls are refused and retried; nothing is lost. After the bundle, an agent BELOW
0.146.0 cannot update itself on that box any more (the unsigned flip grant is gone) — deliver both together.
Design: `felhom.eu/documentation/architecture/03-host-agent.md` §3.1 (new). Measured before the change (real sudo
1.9.16, a throwaway container): the v0.145.0 sudoers let **23 of 29** attack command lines through; v0.146.0 lets
**0** through and still allows all **64** commands the agent's capability check uses.
- **Exact patterns.** A sudoers `*` in the arguments also matches spaces: `pct set [0-9]* -onboot 1` matched
`pct set 100 --dev0 /dev/sda -onboot 1` (a raw host disk for a guest), `mount --bind /mnt/*/felhom-data
/mnt/felhom-drives/*` matched a `..` path onto `/etc/sudoers.d`, `nft add element … *` took a chained `; flush
ruleset`. Every varying argument list is now a sudo regex (`^…$`): one value per slot, a fixed character set, no
`..`, no extra argument. `TestSudoersRefusesTheR861Injections` (29 attacks) + `TestManifestCoveredBySudoers`
(regex-aware now).
- **`felhom-priv-apply`** (new root wrapper, in the bundle). A systemd mount/automount unit, a dnsmasq drop-in, the
WireGuard config and the OOB sshd config + felhom-op key reach their root-read places only through it: fixed source,
fixed destination, CONTENT checked against what the agent's renderers write (no `[Service]`, `Where=` only
`/mnt/<name>` or `/mnt/felhom-drives/<name>` and equal to the unit name, no `bind`/`suid`; a network share must carry
`nosuid,nodev`; no `dhcp-script=`; no `PostUp=`; the sshd config only the one template with its Port). Its 30 tests
(`configs/test_felhom_priv_apply.py`) + Go contract tests feeding each renderer's real output
(`internal/privapplytest`). Pre-flight: every live file on both demo boxes reads OK.
- **NFS/SMB options gain `nosuid,nodev`** (a set-uid file on a server outside the box never acts on the host).
- **Fixed files from the bundle.** The guest pre-start hook (`/var/lib/vz/snippets/felhom-guest-hook.sh`, run as root at
every guest start) and the shared drive parent script + unit are bundle files now (byte-identical to the agent's
constants, pinned). The agent no longer installs them from `/tmp`; it checks them (`guesthook.SnippetReady`,
`ensureSharedParentBoot`) and only registers / enables.
- **The signed update is checked as root.** `felhom-os-apply` mode `agent_update` verifies the operator signature
(root-owned signers, this host, the window, the nonce), re-hashes the staged binary against the SIGNED sha, then runs
the A/B flip; `felhom-selfupdate-guarded apply` is no longer in the agent's sudoers. 7 tests (`AgentUpdate`).
- **The root escrow run reads no path from the agent's config.** As root it pins the PVE secret dir and the WireGuard
state dir to their defaults, refuses a storage id that is a path, and reads its two staged files without following a
symlink (`readStagedNoFollow`) — before, a symlink in the agent's own directory sealed any root file into the blob.
- **Not narrowed here (named in `03` §3.1):** `FELHOM_CONTROLLERSWAP` stays guest-scoped (a compromised agent can run a
chosen controller image in the guest — the household's data, not host root); `FELHOM_ESCROW` still hands the agent R
by design (the agent relays the ceremony); the mkfs / pbs-apply / backup-target wrappers keep a coarse argument and
their own checks.
- Red-proofs F1–F9: `felhom.eu/documentation/audits/hub-safety-2026-10-05/partF/red-proof.txt` (F1's first run did NOT
convict — the name rule masked it — and the test now uses the pair only the Where rule stops).
## v0.145.0 — the OS update repairs itself after a power cut; a short-session box gets restore-tested; "sent late" (R-876, R-874, R-875) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `894da35c7b9e1ac78885690b78352b634e6831e7d99b321573c75d878db8886e`,
config bundle sha256 `78c00adce662d2d966b2ac50ebde46cde1ae225f0107c6a7c02b70ec8ce80c4f`. The wrapper changed: a box
needs the signed `agent_update` AND the signed `agent_config_update`.
- **R-876.** After a crash during an install, `dpkg --audit` can read clean while dpkg's update journal
(`/var/lib/dpkg/updates/`) is not — and apt refuses every install until `dpkg --configure -a` (measured on demo-hp
2026-10-05: every later pass failed until a person typed it). The wrapper now reads `--audit` and the journal in ONE
`sh -c` call (`DPKG_STATE_SCRIPT`) — a clean pass still costs one call (R-845's speed, pinned) — and repairs when
either shows something; as a belt, when apt itself says "dpkg was interrupted", it repairs and retries the install
ONCE. `REPAIR` now logs `journal=N`; a journal still not empty after the repair refuses (R13). Tests
`CrashLeftTheJournal` (the measured shape, the speed, the belt); 3 red-proofs.
- **R-874.** The restore-test's first due-check runs 30 minutes after the agent starts (`DefaultFirstEval`), then every
interval; a box whose power-on sessions are shorter than the 6 h interval never evaluated. A crash-looping agent
restarting faster than 30 minutes still never evaluates (the earned restraint, pinned).
- **R-875.** A kept report's reason is neutral — "sent late — kept on the box until the hub could take it" — the copy
cannot tell a killed agent from an absent hub.
- Red-proofs: `felhom.eu/documentation/audits/catchup-2026-10-05/part{C,D}/`.
## v0.144.1 — a killed pass really keeps its report: the wrapper survives a dead reader; the agent looks again every 5 minutes (R-868, measured live) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `6ccd521d47e64999017e8eb5bc613d724543cfdc5ef9b13bcae9e3ea8c53b8f3`,
config bundle sha256 `e89a9ddfb767e177f8874d56f3dcd3bfd47409d157ff830d362653333bf815e8`. The wrapper changed again:
a box needs the signed `agent_update` AND the signed `agent_config_update`.
- **Found live on demo-hp 2026-10-05 05:45 UTC with v0.144.0** (the night's A5 shape: kill -9 of the pass and the
daemon while apt-get ran): apt finished all 13 packages, but the wrapper's next log line went to a stderr pipe no
process read any more → `BrokenPipeError` → the wrapper died before it saved its report copy (journal: `PLAN
upgrade=13`, then nothing; no copy; the hub got nothing). v0.144.0's mechanism was right and never reached.
`Runner.log` and the final `OSAPPLY-REPORT` line now survive a dead reader (the journal still gets every line).
Test `AgentDiesMidPass` drives the REAL `log()` into a pipe that breaks while apt-get runs.
- **Also found live:** the restarted daemon looked for kept copies ~7 s before the orphaned wrapper wrote one. The
daemon now looks at start and every 5 minutes (`Leg.SendUnsentLoop`); `TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop`.
- Red-proofs: `felhom.eu/documentation/audits/night-fixes-2026-10-05/partD/r868-brokenpipe-red-proof.txt`.
- A second agent release in one session, against "one release per repo": recorded as `09` decision 108 (operator may
reverse) — the alternative was to ship a fix proven not to work.
## v0.144.0 — R8 measures the real download; an OS pass reports even when its agent was killed; the debug pass runs with the hub away (R-865, R-868, R-866) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `f18093c3466749ec4cd47f83f97a401160a24bad1e704f1183051adad14db928`,
config bundle `felhom-config-bundle.json` sha256 `6acf42fe46df5223384d767801cb2bf73238ba4dab82debb7811ca2f790591d8`.
The wrapper `felhom-os-apply` changed, so a box needs BOTH the signed `agent_update` and the signed `agent_config_update`.
- **R-865.** `download_bytes` runs `apt-get --print-uris` WITHOUT `-s`: with `-s` apt prints the simulation and no URI
list, so R8 summed 0 B and only its 500 MB floor ever applied. `--print-uris` alone downloads nothing (measured on
9202: the archive cache and the versions unchanged). The test fake now answers like real apt (with `-s`: no URIs),
and `test_R8_counts_the_real_download` / `test_download_bytes_never_simulates` pin it.
- **R-868.** The wrapper writes every apply pass's report to `<plan dir>/report-<run>-<layer>-apply.json` before it
prints it (root writes into the agent's dir: the dir opened O_NOFOLLOW and checked to be the agent's own, the file
created O_EXCL|O_NOFOLLOW, 0600, handed to the agent). The plan now carries `run_id`, `trigger`, `ring`, echoed in
the report. The agent deletes the copy once the hub has the report; a copy left on disk (the agent was killed, or
the hub was away) is sent at the agent's start and before every pass (`Leg.SendUnsent`), then deleted. A pass lock
(flock on `pass.lock`, across the daemon and a selftest) keeps the sender off a pass that is still running.
- **R-866.** The daemon saves the hub's newest os_update block (`os-update-block.json`); `--selftest=os-update` uses it
when the hub cannot be reached and says so in its header (`block=SAVED(<time>; hub unreachable: …)`); with no hub
and nothing saved it does not run.
- Tests: `configs/test_felhom_os_apply.py` (UnsentReport, SaveReportOnDisk — real files, symlink cases), Go
`TestR868_*`, `TestR866_*`. Red-proofs: `felhom.eu/documentation/audits/night-fixes-2026-10-05/part{C,D}/`.
## v0.143.0 — the config bundle: a signed route for a box's root-owned files (R-840, decision 96) (2026-10-04)
Released by `scripts/release-agent.sh`: binary sha256 `41c0d3060013dfda795262147454248149bee0888c0935170fdb85de6e7a35da`,
config bundle `felhom-config-bundle.json` sha256 `8d7273cf5313ef62b867cb6f831c631923a436452d6f90b8ff7f0771170396ba`.
- **The bundle.** Every root-owned file the installer's step 5 writes (sudoers ×2, the five wrappers, the crash guard
and its units, the agent and rollback units, the start-limit drop-in, the mgmt watchdog, the OOB belt's files) as ONE
reproducible JSON file, built by `scripts/build-config-bundle.py` from `BUNDLE_FILES` in `configs/felhom-os-apply`
(one table) and published beside the binary. The installer (1.31.0) installs the same file.
- **The route.** A signed `agent_config_update` {agent_version, bundle_sha256} (`felhom-opsign -op agent_config_update
-bundle-sha256 …`). The agent is the courier (downloads, checks the sha, hands over); `felhom-os-apply` mode `bundle`
verifies it ITSELF: the operator signature against the root-owned `/etc/felhom/operator-signers` (or, when that file
is missing, ONLY the installer's pinned key, after which it creates the file with exactly that key), the host binding,
the window, its own nonce; the bundle sha; every path in `BUNDLE_FILES` (R16) and never a trust file (R17); every
content check before the first write (visudo, sh/bash -n, python, unit sections, the RuntimeDirectory guard, User=,
nft -c, and that the route itself survives). Atomic per file, previous copies kept under
`/var/lib/felhom-os-apply/bundle-prev/`; a self-check after (visudo -c, `sudo -l` lists the route, the new wrapper's
`--self-check`, the self-update wrapper's usage, the crash guard's status = kernel.panic); any failure puts every
previous copy back. A newly installed crash guard is started (`enable --now`: kernel.panic for this boot, no reboot).
Record `/etc/felhom/config-bundle.json`; the agent reports it as `system.config_bundle`, the facts mode adds drift.
- **Bootstrap.** A box whose `felhom-os-apply` predates 0.143.0 cannot take the first bundle by the route (nothing on it
can write a root file from a signed job): `felhom.eu/scripts/felhom-bundle-bootstrap.sh` is the one by-hand step.
- **`build-golden.sh` 3.2.0** (not part of the binary): `GOLDEN_GUEST_PKGS` brings the template to exactly the
approved guest release (only installed packages, never newer, never a removal or a new package) and prints the
first-night count.
- Tests: `configs/test_felhom_config_bundle.py` (43; 22 of 22 mutants red), `internal/osupdate/bundle_test.go`,
`internal/hub/bundle_record_test.go`. Live: `felhom.eu/documentation/audits/r840-config-bundle-2026-10-04/partB/`.
## v0.142.1 — a Docker step no longer leaves the controller and traefik blind (R-858, `09` decision 95)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.142.1` (`4950030`), sha256
> `003f882a59abfc10021a1f97a4744ab78ab1e916b1392dcef132e7067f3c62bd`, verified by download. Not vouched at release time.
**MinAgent impact:** none. A same-day patch release, by the operator's ruling 95 after a live incident.
- **The incident.** v0.142.0's Docker step on demo-felhom (14:13 UTC) restarted dockerd, which recreates
`/run/docker.sock`. live-restore kept every container running — including `felhom-controller` and `traefik`, which
bind-mount the socket FILE and so kept the deleted inode. The controller could not reach Docker for 1 h 44 min (hub:
DOWN, "docker not reachable"); its own health check stayed "healthy", so the step's health rule PASSED.
- **The fix.** After a Docker step that installed something, the wrapper restarts ONLY the containers that mount
`/var/run/docker.sock` or `/run/docker.sock` (`os-apply: SOCKET-USERS restarted=…`; the apps and the engine untouched).
The guest health reading gains `controller_docker_ok` (`docker exec felhom-controller docker version`), and the health
rule fails when it is false — the consequence, not the mechanism.
- Proven live on demo-hp BEFORE the release (signed undo to 29.7.2): the two restarted, guest / controller / traefik on
the same socket inode, healthy. Red-proofs: `felhom.eu/documentation/audits/os-docker-crash-2026-10-04/partE-incident/`.
## v0.142.0 — the Docker engine slow lane, the version report, the crash guard (`11` §5.8, §5.9; `09` decisions 87–89)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.142.0` (`b1746c2`), sha256
> `7beb32224d6495e9561acfd3ad8a48393a799520f196080011cb27fceb6d1de6`, verified by download. Not vouched at release time.
**MinAgent impact:** none. **Needs hub v0.132.0** for the System page, the Docker approval and the crash events; an older
hub stores the new `system` stanza unread. **Needs the new root files** on an installed box (R-840): the wrapper,
`/etc/felhom/os-trust.json`, `/etc/felhom/operator-signers` and the crash guard — the installer 1.30.0 writes them; the
demo boxes got them by hand.
- **Docker `live-restore` ON** (decision 87). Wrapper mode `live-restore-on`: merge `"live-restore": true` into the
guest's `/etc/docker/daemon.json` and `systemctl reload docker` — never a restart (R-835). An invalid daemon.json is
left alone (R16); a reload that does not enable it puts the old file back. The leg runs it once before a Docker step;
`--selftest=live-restore -vmid N` runs it by hand. Measured: demo-hp 24 containers, demo-felhom 5 — the same ids after.
- **The Docker engine slow lane** (`11` §5.8). Wrapper layer `docker`, lane `slow` only, the six Docker packages only,
origin `Docker CE` only (R2; a Docker package in a fast-lane plan is refused). **R3 — the wrapper checks the authority
itself**, never the agent's config: a ring-1 step or ANY undo needs a signed `os_docker_step` it verifies with
`ssh-keygen -Y verify` against the ROOT-owned `/etc/felhom/operator-signers` (namespace `felhom-op-v1`, the blob's
`key_id`), bound to `/etc/felhom/os-trust.json` `host_id`, inside its time window, never replayed (a root-owned nonce
file), with exactly the signed packages and undo flag; an unsigned ring-0 step needs that file's
`"ring0_slow_lane": true` (the demo boxes only, set by hand). **R15:** live-restore must be on. An undo may downgrade
(`--allow-downgrades`) only inside a signed job. The leg: ring 0 runs the Docker step at night after a healthy guest
and host step (`select pending-docker`); ring 1 never does — only `DockerStepExecutor` (signed job, heavy-op gate).
**Health:** the guest rule + every container running at the start has the SAME id after + the engine reports the
installed version; a changed id is `health_failed`. Measured ring 0: 29.7.x → 29.8.2 on both demo boxes, every id kept.
- **The version report** (R-852, decision 89). Wrapper mode `facts` (read-only): host Debian, running and next-boot
kernel (`next_entry` > saved default > newest installed, by dpkg order), held packages (R-848), kernel taint (oops,
warn), `kernel.panic`, the crash guard state; guest Debian, Docker engine, containerd, live-restore. The host report
gains `system {pve_version, kernel_version, vmid, facts, facts_error}`, read at most every 10 min (~2 s);
`--selftest=os-facts -vmid N`. A value nobody could read is `unknown`.
- **R-849:** the guest is scanned for "restart needed" on every pass too, so a guest restart clears it.
- **The crash guard** (decision 88, R-851): `configs/felhom-crash-guard` + `felhom-crash-guard.service` (early boot;
its ExecStop writes a clean-stop marker) + an hourly re-arm timer + `/etc/felhom/crash-guard.conf`. A boot without
the marker followed an unclean stop (a crash, a power cut or a hard reset — pstore saved nothing for a real panic on
demo-hp, so they cannot be told apart). Armed: `kernel.panic = 10`. After the 2nd unclean boot within 60 min it
TRIPS (`kernel.panic = 0`), so the 3rd crash within the hour leaves the box off; it re-arms after 24 h of normal
running or `felhom-crash-guard rearm`. State in `/var/lib/felhom-crash-guard/state.json` (0644; read by facts).
- **After the tag (main only, not shipped to boxes): `configs/build-golden.sh` 3.1.0** — the golden's `daemon.json`
carries `"live-restore": true` with a fail-closed assertion, and `GOLDEN_DOCKER_PKGS` pins the approved Docker engine
set (all six `name=version`); without it the bake log warns that the set is the newest, not an approved one.
- Tests: wrapper 76 (DockerLane, LiveRestore, Facts, RealSignatureCheck with a throwaway key), crash guard 9, Go leg +
executor; red-proofs `felhom.eu/documentation/audits/os-docker-crash-2026-10-04/partB/agent-redproofs.txt` (17 caught).
## v0.141.1 — "reboot needed" is true on the host (found live on demo-felhom, 2026-10-04)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.141.1` (`a6bc3f1`), sha256
> `b712f577099fd2d374f648df1e825874821302fe1b46e93c7b4b70044fbe84b5`, verified by download. Not vouched at release time.
**MinAgent impact:** none. **Pairs with hub v0.131.1** (reads `reboot_scanned`); an older hub ignores the field.
A patch release in the same session as v0.141.0 — a deliberate exception to "one release per repo": v0.141.0's host
"reboot needed" was wrong in two ways, and it feeds an operator alarm.
- **The scan hid `lxc-start`.** The host scan skipped every process whose cgroup line contains `lxc`, to leave out the
guests' own processes. `lxc-start` lives in `0::/lxc.monitor/<vmid>`, so it was skipped too. Measured: after a
108-package host pass (libc6 included) `lxc-start` mapped 20 deleted files and the report said `reboot_needed:
false`. The pattern is now `:/lxc/` (`RESTART_SKIP_CGROUP`), pinned by a test that runs `grep` against the measured
cgroup lines.
- **A reboot never cleared it.** v0.141.0 scanned only after an install (R-845). The host now scans on EVERY pass (it
is local, no `pct exec`); the guest still scans only after an install. The report carries `reboot_scanned`.
- Red-proofs: `felhom.eu/documentation/audits/os-host-lane-2026-10-04/partB/live-defects-redproofs.txt`.
## v0.141.0 — OS updates: the host fast lane (`11` §8 step 3); the tunnel status is true (R-841); the leg is fast (R-845)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.141.0` (`cfba0d0`), sha256
> `6eaad9809613c4fca64aefc30cc16415528499bf48efe3a1daaa08af8a611aff`, verified by download. Not vouched at release time.
**MinAgent impact:** none required by any controller. **Needs hub v0.131.0** (`host_release`, the tunnel's three
states, layer-tagged OS reports). With an older hub the host step finds no host release (ring 1 → nothing) and the
tunnel's `detail` is ignored. Reads the cloudflared health check controller v0.292.0 adds; with an older controller
the tunnel is judged on the container state alone and `detail` says so.
- **R-841 — the tunnel.** The agent used to run `systemctl is-active cloudflared` on the HOST — a unit that does not
exist (cloudflared is a container in the customer guest), so every box reported `inactive`. `GuestTunnelProber`
now reads the guest's `cloudflared` container through the EXISTING sudoers line (`pct exec N -- docker inspect -f
*`): state, exit code, and the Docker health status. Three states: `running` (healthy), `not_running` (stopped,
absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting). `detail` says why.
- **The host step** (`11` §8 step 3). After a healthy guest step, under the same heavy-op gate, the leg runs the same
wrapper with `layer: host`. Debian origin only; never kernel, boot or firmware packages (new refusal **R14**);
**only on an appliance** (**R12** lifted for the host fast lane: proof is the ROOT-owned install record
`/var/lib/felhom-install/state.json` `mode: appliance` — the agent-writable `agent.json` is not trusted for this).
A failed or unhealthy guest step skips it. **Host health rule:** the agent, pveproxy, pvedaemon, pvestatd and
pve-cluster are active; the customer guest runs; the guest health rule passes; the tunnel is `running` (an
`unknown` tunnel does not fail the rule; `not_running` does). Never reboots: "reboot needed" is reported when PID 1
or `lxc-start` runs a replaced library. No automatic undo — the by-hand runbook is `os-updates-host-undo.md`.
- **R-845 — the leg is fast.** The wrapper asked about each package in its own `pct exec` (about 0.9 s each). It now
makes one call per layer for the version checks (`apt-cache madison` for all names at once, `dpkg --compare-versions`
on the host), scans for restart-needed only after an install, repairs only when `dpkg --audit` reports something,
and reports its own `pass_seconds`.
- **Wrapper tests:** `configs/test_felhom_os_apply.py` 46 tests (host layer, R12, R14, lxc-start, the speed rules).
Red-proofs: `felhom.eu/documentation/audits/os-host-lane-2026-10-04/partA/`, `partB/`, `partC/agent-golden-redproof.txt`.
- **Contract:** the desired-state golden gains `host_release` (byte-identical with the hub's); `TestOSUpdateGolden_Decodes`
checks it.
## v0.140.0 — OS updates, guest fast lane (`11-os-updates.md` §8 step 2; `09` §3 decisions 76, 79, 80)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.140.0` (`9cac346`), sha256
> `ae2d60b794869c51d6b063c8e31e2da98ecdbd36c6b75febd64e2fb4266c1250`, verified by download. Not vouched at release time.
**MinAgent impact:** none required by any controller. **Needs hub v0.130.0** (`os-report`, the `os_update` block); an
older hub serves no block and the leg then reports and installs nothing (ring 1, no release).
- **`configs/felhom-os-apply`** — the root wrapper (Python 3, stdlib). One sudoers entry, `FELHOM_OSAPPLY`:
`felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json`. Modes `inventory` / `apply` / `health`. Refuses (exit
2, nothing changed) on R1–R13: plan path/owner/JSON, a non-Debian origin, the slow lane, any removal, any downgrade,
a new or unlisted package, a version not downloadable even from the snapshot, low space, a lock (apt or a guest lock
such as a backup), a vmid that is not the box's own customer guest (it must bind `/mnt/felhom-drives`), malformed
names/versions, the host layer, dpkg still broken after the repair. Repairs first (`dpkg --configure -a`,
`apt-get -f install`). A version Debian already replaced comes from `snapshot.debian.org` at the approval time
(decision 79). Reports the full installed set with origins, pending, restart-needed (outside containers), health.
`configs/test_felhom_os_apply.py`: 35 tests; every refusal red-proved.
- **`internal/osupdate`** — the leg: after a SUCCESSFUL primary whole-guest backup, still holding the heavy-op gate
(never beside another backup or a restore-test), once per night, 90 s after the backup. Ring 0 installs every pending
Debian / Debian-Security fix; ring 1 exactly the hub's newest approved release; switched OFF → reports only. The
health rule: docker answers, the network resolves, the controller is healthy, every container running at the START
of the leg runs (and is healthy if it was) — a 5-minute wait. **No automatic undo:** a customer guest cannot be
snapshotted (R-837). A failure is `health_failed` → the hub mails the operator.
- **`--selftest=os-update -vmid N`** — the debug action (trigger `debug`, never throttled, not a night run).
- **The `--selftest` flag also accepts `wgtunnel`** — it was dispatched but refused since S3 (found by the new
`TestSelftestFlag_AcceptsEveryDispatchedMode`, which also caught `os-update` live).
- Proven live 2026-10-04 on both demo boxes (ring 0: 53 packages each; ring 1: exactly 3 approved versions; a failed
health check → `health_failed`, operator mailed): `felhom.eu/documentation/audits/os-guest-lane-2026-10-04/`.
## v0.139.0 — a DR restore never lands beside a live original (2026-10-04, R-834)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.139.0` (`475bdce`), sha256
> `8534a9be368a6d24d8065db77436e86900443c5c6554c71f6fe91c2bdb9d0b9c`, verified by download. **Not vouched.**
**MinAgent impact:** none required by any controller.
- The DR bring-up (`--selftest=bring-up -mode dr`) keeps the archive's `onboot: 1`, binds the host's REAL drives
(`mp8 /mnt/felhom-drives`) and STARTS the guest — right on a replaced host, wrong beside a live original (a second
controller for the same household on the same drives). It now REFUSES, before any restore, when the archive's
source guest still exists on the host, when any guest binds the drives parent, or when a guest's config cannot be
read (fail closed). On a replaced host it proceeds and keeps its binds, unchanged.
- The restore-test was MEASURED safe live on demo-hp (onboot 0 and throwaway stand-ins for mp8/mp9 from the first
config read to teardown); a test now pins its "no host path" half beside the existing onboot test.
- No sudoers change: the restore-test sets onboot 0 through the API create call, and DR refuses rather than degrade,
so no `-onboot 0` line is needed.
- Tests: `TestRunBringUp_DRRefusesBesideALiveOriginal` (source guest present / drives bind on another guest / an
unreadable config refuse; a replaced host proceeds and keeps the drives bind), `TestRunBringUp_ProvisionNotBlockedByADrivesBind`,
`TestArchiveSourceVMID`, `TestRestoreTest_NoHostPathBindBesideTheOriginal`. Red-proofs: the DR check returning ""
→ three refusal cases restore and START; the restore-test's mp8 override set to the host path → fails.
## v0.138.0 — the restore test takes only THIS box's archives (2026-09-30, R-727, `09` §3 decision 51)
> **RELEASED 2026-09-30** by `scripts/release-agent.sh` — tag `v0.138.0` (`e1b8269`), sha256
> `55916026001790a79ebf97d32c032610cfde8e09d02979b9b9d8c2cbc5d88195`, verified by download. **Not vouched** (the golden
> keeps 0.137.0 until the next bake); delivered to the demo boxes by signed `agent_update` jobs.
>
> **VOUCHED 2026-09-30** with golden 0.283.1 (`min_agent` 0.131.0), on the operator's word: the hub logged
> `Artifact manifest set: agent=0.138.0 golden=0.283.1 min_agent="0.131.0"`. Evidence:
> `felhom.eu/documentation/audits/evidence-golden-0283-2026-09-30/`.
**MinAgent impact:** none required by any controller.
- A returning customer's PBS namespace can hold archives of EARLIER boxes: same guest id (9201), same token, written
with a different key. Measured 2026-09-30: the newest SETTLED archive was an earlier box's, and the test failed
`wrong key … manifest's key 6b:ca:5f:3f… does not match provided key de:51:7a:18…` every evaluation. The archive
carries no host id; it carries its key fingerprint (PVE content `encrypted`), and the storage carries its own
(`GET /storage` → `encryption-key`). `PickSettledRestoreCandidateOn` now skips — and logs by name, once — an archive
whose fingerprint is not the storage's own; an unencrypted storage is not filtered; a failed storage read is an
error (tier UNKNOWN), never "nothing to prove".
- Tests: `TestR727_TheRestoreTestTakesOnlyThisBoxsArchives` (the 2026-09-30 shape: nothing picked while this box's
archive settles, then exactly it), `TestR727_UnencryptedStorageIsNotFiltered`, `TestR727_KeyLookupFailureIsUnknown`.
Red-proof RP39: the skip removed → the earlier box's `2026-09-16T21:59:54Z` is picked.
## v0.137.0 — a guest outside the agent's ACL is not a known guest (2026-09-27, R-689, v0.136.0 regression)
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.137.0` (`3ef095f`), sha256 `766c9166916a1bd3674b0dc69081f8a7619e770f1402d8ad7705b395937e7627`, verified by download. **NOT vouched** (the operator's act).
>
> **VOUCHED 2026-09-28** with golden 0.276.0 (`min_agent` 0.131.0), on the operator's word of 2026-09-27: the hub logged
> `Artifact manifest set: agent=0.137.0 golden=0.276.0 min_agent="0.131.0"`, and a Day-0 test install fetched this binary
> through the manifest and sha-verified it. Evidence: `felhom.eu/documentation/audits/evidence-golden-0276-2026-09-28/`.
**MinAgent impact:** none required by any controller.
- v0.136.0 asked `GuestConfig` whether an archive's guest exists and treated anything but "does not exist" as a lookup
failure. PVE answers **403 "permission denied at /vms/<id>"** for a vmid outside the token's pool — so on demo-hp the
deleted guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot read is not one it
manages; its archive is skipped. `TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest`, red-proofed. Verified read-only
on demo-hp with the pre-release binary before the release.
## v0.136.0 — … of a guest that still EXISTS (2026-09-27, R-689 second half)
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.136.0` (`16dbc83`), sha256 `2eb0b5ebe253defd68b322312bbac12418c051b0a7a0d2d1831310d97fa6d755`, verified by download. **NOT vouched** (the operator's act).
**MinAgent impact:** none required by any controller.
- **R-689, second half.** Read on demo-hp right after v0.135.0 with the read-only `-selftest=restore-test-due`: the
golden was skipped, and the pick fell to `vzdump-lxc-9100-2026_08_21…` — a leftover of a guest deleted in August. A
candidate's guest must now exist on this node (`GuestConfig`): "does not exist" skips the archive (INFO once per
volid); any other lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove". Tests
`TestR689_AnArchiveOfADeletedGuestIsNeverPicked` (red-proofed: without the check it picks the 9100 leftover),
`TestR689_AGuestLookupFailureIsUnknownNotEmpty`. A second agent release in one session — the first half was found
incomplete on the box.
## v0.135.0 — the restore test proves only backups OF A GUEST (2026-09-27, R-689)
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.135.0` (`d4be12c`), sha256 `ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (the operator's act).
**MinAgent impact:** none required by any controller.
- **R-689** (`backup/runner.go` `guestBackupArchive`). demo-hp keeps its golden template in `local:backup/` — content
"backup", 654 MB, plausibly complete, the newest settled entry — and the scheduled restore test picked it every 6 h and
failed `extractconfig` with a 403, while the guest's real archive went untested. A restore-test candidate is now a
`vzdump-{lxc,qemu}-<vmid>-…` file or a PBS `backup/{ct,vm}/<vmid>/…` snapshot whose vmid the storage reports; anything
else is skipped with one INFO line per volid (not the "INCOMPLETE archive" WARN). Tests
`TestR689_TheRestoreTestNeverPicksTheGolden` (red-proof: without the check it picks the golden) and
`TestR689_GuestBackupArchiveShapes`; two older picker tests' fixtures moved to real archive names.
Evidence: `felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`.
## v0.134.0 — a whole-box backup that cannot fit is skipped with a reason, before anything starts (2026-09-25 night, R-685)
> **RELEASED 2026-09-24 night** by `scripts/release-agent.sh` — tag `v0.134.0` (`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download. **NOT vouched** (Day-0 stays on the previous version).
**MinAgent impact:** none required by any controller.
- **R-685 — the backup space preflight** (`backup/runner.go` `spaceFits`). Before a vzdump to a LOCAL (non-PBS)
target, the newest archive of that guest on that target × 1.25 + 1 GiB must be free — PVE prunes old archives
only AFTER a successful backup, so the kept ones still stand during the run. A shortfall is a skip: nothing
starts, the record's Error reads `skipped: not enough space: <target> has X GiB free; the last archive of guest N
was Y GiB, so a new one needs about Z GiB …` (stable prefix `BackupSkipNoSpacePrefix`), and it reaches the
controller's tier view and, through the quiesce loop's tier notifier, the operator's `whole_guest_backup_failed`.
It FAILS OPEN on what is not known (PBS target, first backup, unreadable usage).
- **Free space is read from `GET /nodes/<node>/storage`** (`NodeStorage`), never `GET /storage` — the latter is the
cluster DEFINITIONS and carries no usage. **Found live, before release:** the first build read `/storage`,
failed open, and a real vzdump of demo-hp 9201 started from the live test; it was aborted after 5 min 16 s, no
archive left (`felhom.eu/documentation/audits/night-2026-09-25/F/`). The test fake's `ListStorage` now strips
usage like production. Red-proofs: two (`…/F/redproof-r685-*.txt`).
- Live proof (demo-hp, safe builds with a hard stop before vzdump): ×10 margin → refused, "local has 14.9 GiB
free … needs about 77.2 GiB"; release margin → "space preflight passed" need 11.3 GB, avail 16.0 GB.
## v0.133.0 — a restore-test can never fill a box's disk; leftovers retried on a timer (2026-09-24, R-672, R-673)
> **RELEASED 2026-09-24** by `scripts/release-agent.sh` — tag `v0.133.0` (`9bdb4da`), sha256 `3aa303452b8c6be58573d00af01a0ab4a0d97e4f885ffecd0144a18ac24e69b6`, verified by an independent anonymous download. **NOT delivered** (needs an operator-signed `agent_update` job per box, R-530) and **NOT vouched**.
+7 -7
View File
@@ -52,11 +52,11 @@ This is in the core because breaching it is how this component stops being audit
## Gates — ONE entry point
**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs this
repo's gates — `reuse_refs_check` and `instructions_gate`, both the **shared** copies in
`felhom.eu/scripts/`, never copied into this repo (a copy recreates the drift they detect; an absent
sibling clone FAILS). `--fast` selects the gates touching no network and no container runtime; today
that is all of them. **A missing gate is a FAILURE, never a skip.**
**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs every
gate in its `GATES` table (that table is the list); the shared ones — `reuse_refs_check`,
`instructions_gate`, `observations_gate` — are the copies in `felhom.eu/scripts/`, never copied into
this repo (a copy recreates the drift they detect; an absent sibling clone FAILS). `--fast` selects the
gates touching no network and no container runtime, and skips `published` (network), naming it. **A missing gate is a FAILURE, never a skip.**
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
**per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
@@ -104,8 +104,8 @@ the mechanism are exempt.
**A gate ships with a decoy test that has been seen to fail (R-421).** A decoy is the LABEL without
the FACT — a directory with the right name and no bake log, a note whose prose mentions the marker it
lacks. `scripts/decoy_coverage_gate.py` refuses a new gate that has neither a decoy nor a named
lacks. `felhom.eu/scripts/decoy_coverage_gate.py` (run by felhom.eu's `repo_gates.py`, for all four repos) refuses a new gate that has neither a decoy nor a named
exemption carrying its row. The four shapes, the 2026-09-01 sweep that fooled 16 of 29 gates, and the
decoys withdrawn as illegitimate: `documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and
decoys withdrawn as illegitimate: `felhom.eu/documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and
`felhom-controller/.claude/rules/gates.md`. **Scope is a fact too** — prefer `os.walk` over
`os.listdir`, and a glob over a hand-maintained list.
+16
View File
@@ -1,5 +1,21 @@
# CONTEXT — felhom-agent working state
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
> `release-agent.sh` publishes it. A box whose `felhom-os-apply` predates 0.143.0 needs ONE by-hand bootstrap
> (`felhom.eu/scripts/felhom-bundle-bootstrap.sh`) — done on both demo boxes; Tester 2 waits for the operator (R-862).
> Both demo boxes: agent 0.143.0, bundle 0.143.0 (record `/etc/felhom/config-bundle.json`). R-861 found: the sudoers is
> root-equivalent. `build-golden.sh` 3.2.0 (`GOLDEN_GUEST_PKGS`). Runbook `felhom.eu/documentation/runbooks/config-bundle.md`.
> **2026-09-25 night — v0.133.0 AND v0.134.0 DELIVERED to both demo boxes (CC-signed `agent_update`, ruling 1);
> restore test back ON (the `-1` config kept as `agent.json.night-0925-off`). v0.134.0 = R-685:** `backup/runner.go`
> `spaceFits` — a vzdump to a LOCAL target needs free ≥ newest archive of that guest × 1.25 + 1 GiB (PVE prunes
> only after success); a shortfall is a named skip (`BackupSkipNoSpacePrefix`), fail-open on PBS / first backup /
> unknown usage. **Free space comes from `NodeStorage` (`GET /nodes/<n>/storage`) — `ListStorage` (`GET /storage`)
> has NO usage**; the first build read it and a real vzdump started in its own live test (aborted, no archive).
> demo-hp: `local_backup_retention` 1 (operator option A, saved `agent.json.pre-a4-retention`). Peti's box: nothing.
> **2026-09-24 — v0.133.0 RELEASED, NOT DELIVERED (R-672, R-673).** Restore-test space preflight
> (`reconcile/restoretest_space.go`, `internal/restorespace`): uncompressed size from the vzdump log / PBS size,
+12 -27
View File
@@ -1,29 +1,14 @@
# REPORT — agent v0.133.0: a restore-test can never fill a box's disk (2026-09-24, R-672, R-673)
# REPORT — the second burn-down night (2026-10-06 → 07): R-894 and the R-330 wire half, on main, unreleased
Full record: `felhom.eu/documentation/audits/r672-2026-09-24/README.md`.
Nothing was released or delivered (brief fence). Both changes ship with the memory-kill check as **v0.150.0**, after the
2026-10-07 night read-back. Session report: `felhom.eu/REPORT-night-burndown-2026-10-06.md`; night log
`felhom.eu/documentation/audits/night-burndown-2026-10-06/NIGHT-LOG.md`.
## Shipped (released, not delivered)
Tag `v0.133.0` (`9bdb4da`), package sha256 `3aa30345…e69b6`, verified by anonymous download. Delivery needs the
operator's signed `agent_update` job per box.
- **Space preflight** before anything is created: free ≥ restored × 1.2 + 5 GiB, `restored` UNCOMPRESSED (vzdump
log "Total bytes written" / PBS snapshot size), thin metadata with room, off the tested guest's pool when
another eligible storage fits, unknown refuses, reported as a non-pass (`skipped`).
- **Scratch teardown retried every 10 min**; operator told once after 3 failed tries.
- **Thin pool ≥ 90 %** → immediate host report (hub v0.124.0 alarms `storage_fill_critical`, per pool per 6 h).
- **R-673:** the stale-lock sweep on the same timer, under the one-heavy-op gate.
## Red-proofs (each seen failing; files in `felhom.eu/documentation/audits/r672-2026-09-24/redproofs/`)
1. preflight removed → the 2026-09-24 restore issued again; 2. archive FILE size used → "the compressed file size
was used"; 3. timer pass a no-op → "the leaked scratch was not destroyed by the timer"; 4. sweep without the gate →
"the sweep ran while a backup held the gate"; 5. no 90 % edge → "0 report requests, want 1"; 6. every skip dropped →
"a space refusal never reached the host report". Full suite `go test ./...` green; `agent_gates.py` OK.
## Live (demo-hp, the on-demand `--selftest=restore-test`, cadence OFF)
(a) factor 10 → refused: needs 215.5 GiB, has 22.1 GiB. (b) normal margin → refused: restoring 21.1 GiB needs 30.3 GiB,
has 22.1 GiB — correct: no full restore-test fits demo-hp under 80 % pool use. (c) a forced teardown failure could not
run live (it needs a scratch guest, which (b) shows cannot be created within the 80 % rule) — unit tests only. Pool
58.99 % before and after every run; nothing created.
## Observations
1. demo-hp has no second eligible storage for a restore-test: `nvme-scratch` takes `rootdir`, but the agent holds no grant there. FILED: R-672
- **R-894** (`74b5eae`): the newest successful backup per tier is kept on disk (`backup-success-state.json` in the OOB
state dir) and read by `/backup/due` **only when the tier's storage cannot be read**: fresh → not due, older than the
cadence → due, none → due with age unknown (as before). A storage that answers stays the ground truth. Tests
`TestBackupDue_R894_*`, `TestBackupSuccessState_*`, `TestR894_LastKnownBackupsIsWiredIntoTheDaemon`; four red-proofs
`felhom.eu/documentation/audits/night-burndown-2026-10-06/s4/`. Architecture: `07` §6.1 updated.
- **R-330, wire only** (`adaf86a`): the host report's SMART summary carries 187 `reported_uncorrect`, 188
`command_timeout`, 199 `udma_crc_errors` (pointers, omitted when unknown). No verdict reads them.
- Suite `go test ./...` rc 0; `scripts/agent_gates.py` (full) all OK; CI green on each push (runs 1451, 1467).
+15 -5
View File
@@ -14,8 +14,15 @@
| `Privileged` (CreateGoldenLXC/MountUSBByUUID/SMART/Sensors) | internal/proxmox/privileged.go | methods on `*Privileged` | the 3 fenced root-CLI exceptions ONLY | Do NOT add methods — fence is structural (`routing_test.go` asserts it) |
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
| `stageTemp` | internal/localapi/intermediary.go | `stageTemp(pattern, content) (path, err)` | random-named temp before a root `install` (audit B1) | Fixed /tmp names are a TOCTOU — sudoers globs expect `/tmp/felhom-*-*.ext` |
| `guesthook.InstallSnippet` / `Register` | internal/guesthook/install.go | `InstallSnippet(ctx, runner) error` | pre-start self-heal hook install (C1 net) | Same random-temp+install pattern; snippet delegates to the agent binary (no shell logic). Issues `mkdir -p /var/lib/vz/snippets` FIRST (v0.63.0, B2 — fresh boxes lack the dir; sudoers grants exactly that argv) |
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key` | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
| `osupdate.Leg.SendUnsent` / `lockPass` (v0.144.0, R-868) | internal/osupdate/unsent.go | `(ctx) int` | an OS-pass report the agent never sent (killed mid-pass): the wrapper keeps `report-<run>-<layer>-apply.json` beside the plan; the agent deletes it once the hub has it | any new caller that runs an apply pass must hold `lockPass` (flock, across processes) — the sender must never take a running pass's copy |
| `osupdate.LoadSavedBlock` (v0.144.0, R-866) | internal/osupdate/leg.go | `(planDir) (block, savedAt, ok)` | the hub's newest os_update block as the daemon last received it (`os-update-block.json`) | the debug pass uses it ONLY when the hub cannot be reached, and says so in its header; no saved block → no pass |
| `dpkg_state()` / `DPKG_STATE_SCRIPT` (v0.145.0, R-876) | configs/felhom-os-apply | `audit, journal = self.dpkg_state()` | dpkg's state in ONE call: `--audit` AND the update journal | never gate a repair on `--audit` alone — a crash leaves only the journal (measured); keep it one call (R-845) |
| `guesthook.SnippetReady` / `Register` (v0.146.0, R-861) | internal/guesthook/install.go | `SnippetReady(path) error` | is the pre-start hook (a FIXED file from the bundle) in place — register only then | The agent never installs the hook (Proxmox runs it as root); a missing hookscript stops a guest start, so never `Register` without `SnippetReady` |
### Disk / format safety (role gates, durable IDs, format guards)
@@ -59,8 +66,8 @@
|---|---|---|---|---|
| `IntentStore` (`Get/SetEnrolled/SetEjected/SetDecommissioned/OnAbsent`) | internal/storage/intent.go | `OpenIntentStore(path)` | drive intent (4-state self-heal) | Keyed by durable-id only; `OnAbsent` is the ONLY ejected→enrolled path; refuses empty ids |
| `GuestBindStore` (`Record/Remove/Guests`) | internal/localapi/guestbindstore.go | `OpenGuestBindStore(path)` | per-guest enrolled binds (F9 re-assert) | Same tmp+rename 0600 pattern as IntentStore |
| `FormatJobStore` + `startFormatDetached` + `RecoverFormatJob` | internal/localapi/formatjob.go | `startFormatDetached(device, durableID, fstype, blank) <-chan error` | detached, restart-surviving mkfs (F20-BUG3) | Runs off `s.baseCtx` (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank |
| `TokenStore.Mint` / `Lookup` | internal/localapi/tokenstore.go | `Mint(vmid) (plaintext, error)` | per-guest local-API tokens | Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup RELOADS the file once on a miss (v0.63.0, B3): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence without a restart; append-only size check bounds the re-read |
| `FormatJobStore` + `startFormatDetached` + `RecoverFormatJob` | internal/localapi/formatjob.go | `startFormatDetached(device, durableID, fstype, blank) (*formatJob, <-chan error)` | detached, restart-surviving mkfs (F20-BUG3) | Runs off `s.baseCtx` (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank; on success `job.FSUUID` = the new filesystem's UUID, read back only when the durable id still resolves to the formatted device (R-25) — read it only after `done` delivers |
| `TokenStore.Mint` / `Lookup` | internal/localapi/tokenstore.go | `Mint(vmid) (plaintext, error)` | per-guest local-API tokens | Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup stats the file on EVERY call and reloads BEFORE answering when the append-only log grew (R-269; was reload-on-miss only, v0.63.0 B3, which let a token rotated out by another process keep authorizing as a map hit): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence both ways without a restart; unchanged size = no re-read. Pinned by `TestTokenStore_RotatedOutTokenRejectedFirst` |
| `FileNonceStore.SeenOrRecord` | internal/authz/noncestore.go | `SeenOrRecord(nonce, exp) bool` | durable anti-replay | fsync'd before returning false; prune only after exp |
| `Journal` (`Append/Latest/InFlight/AlreadyApplied`) | internal/reconcile/journal.go | `OpenJournal(path)` | op journal + idempotency + crash recovery | `Recover` consumes `InFlight()`; scratch entries special-cased |
@@ -114,7 +121,7 @@
| Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only |
| Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set |
| Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction |
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`; R-856 `crashGuardStatePath` — GET /host/crash-guard's state file, internal/localapi/crashguard.go) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
| `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) |
| `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool** — `BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs |
| `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted` ⇒ **quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s |
@@ -150,10 +157,13 @@
| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `lanresolver.hostRoot` + `dnsmasqUnitPaths` (data seam, R-317) | internal/lanresolver/lanresolver.go | prod `hostRoot = "/"`; probe = the `dnsmasq` package's systemd UNIT, never `/usr/sbin/dnsmasq` (owned by `dnsmasq-base`) | internal/lanresolver/ensure_dnsmasq_test.go — fixture root tree + recording `proxmox.Runner`; the REAL `os.Stat` probe and `EnsureDnsmasq` run. `TestEnsureDnsmasq_ProductionProbeIsTheUnit` pins the production wiring |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct |
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
+53
View File
@@ -0,0 +1,53 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-444: the weekly trim has the guestnet shape (component + reporter seam + goroutine), so its wiring is asserted
// from the AST like TestMainWiresGuestNetWatchdog — a unit-green trim job that main.go never starts is the inert-seam
// defect. It must also share the ONE heavy-op gate (heavyOps), or it could run beside a backup.
func TestMainWiresGuestDiskTrim(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var constructedWithGate, reporterWired, started bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
if fn, ok := node.Fun.(*ast.SelectorExpr); ok {
switch fn.Sel.Name {
case "New":
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "fstrim" && len(node.Args) >= 3 {
if id, ok := node.Args[2].(*ast.Ident); ok && id.Name == "heavyOps" {
constructedWithGate = true
}
}
case "SetGuestDiskTrimReporter":
reporterWired = true
}
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "diskTrim" {
started = true
}
}
}
return true
})
if !constructedWithGate {
t.Error("main.go never calls fstrim.New(..., heavyOps, ...) — no trim job, or one outside the heavy-op gate")
}
if !reporterWired {
t.Error("main.go never calls collector.SetGuestDiskTrimReporter — the guest_disk_trim stanza never reaches the hub")
}
if !started {
t.Error("main.go never starts the trim job with `go diskTrim.Run(ctx)`")
}
}
+326 -11
View File
@@ -22,6 +22,7 @@ import (
"os/exec"
"os/signal"
"path/filepath"
"regexp"
"strconv"
"strings"
"sync"
@@ -37,6 +38,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
"gitea.dooplex.hu/admin/felhom-agent/internal/fasttick"
"gitea.dooplex.hu/admin/felhom-agent/internal/felhomsshd"
"gitea.dooplex.hu/admin/felhom-agent/internal/fstrim"
"gitea.dooplex.hu/admin/felhom-agent/internal/guesthook"
"gitea.dooplex.hu/admin/felhom-agent/internal/guestnet"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
@@ -44,6 +46,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
"gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/pbs"
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
"gitea.dooplex.hu/admin/felhom-agent/internal/poke"
@@ -242,6 +245,12 @@ func main() {
os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive))
case "restore-test-due":
os.Exit(runSelftestRestoreTestDue(context.Background(), cfg, logger))
case "os-update":
os.Exit(runSelftestOSUpdate(context.Background(), cfg, logger, vmid))
case "os-facts":
os.Exit(runSelftestFacts(context.Background(), cfg, logger, vmid))
case "live-restore":
os.Exit(runSelftestLiveRestore(context.Background(), cfg, logger, vmid))
case "pbs-verify":
os.Exit(runSelftestPBSVerify(context.Background(), cfg, logger))
case "lanresolver":
@@ -782,7 +791,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
pbsStore := pbs.NewSnapshotStore()
pbsTargets := pbsTargetsFromPVE(cfg, px, logger)
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target
// Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent
// depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the
@@ -839,6 +848,11 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// The "Down" channel sync hook: on each heartbeat, fetch desired-state when the generation
// advances. The loop calls it via the EnvelopeObserver seam (hub does not import desired).
desiredSyncer := desired.NewSyncer(client, desiredProvider, logger)
// OS updates, guest fast lane (agent v0.140.0, `11-os-updates.md` §8 step 2): the leg consumes the hub's
// os_update block and runs after each successful primary whole-guest backup (wired on the local API below).
osLeg := newOSLeg(cfg, client, px, logger)
collector.SetSystemReporter(&factsReporter{leg: osLeg, guest: firstGuest(px)}) // R-852: the versions
desiredSyncer.AddConsumer(osLeg)
// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
@@ -856,6 +870,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
logger.Info("felhom-agent daemon starting",
"version", version, "host_id", cfg.Hub.HostID, "hub_url", cfg.Hub.URL,
"interval_s", hcfg.PollSeconds) // hub key intentionally not logged
// R-868 (v0.144.0): an OS pass whose agent was killed kept its report on disk — send it now (a pass that starts
// first sends it itself; the pass lock keeps the two apart).
// v0.144.1: and again every 5 minutes — the wrapper of a killed pass can finish AFTER the restart (measured).
go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) {
logger.Info("osupdate: sent kept report(s)", "count", n)
})
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
@@ -1080,7 +1100,27 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// recurring clobber before it locks the box out. Port 22 for G1 (H1 passes the felhom-sshd port).
collector.SetMgmtPlaneReporter(mgmtplane.NewReporter(mgmtplane.DefaultPrivsepDir, mgmtplane.DefaultHealMarker, mgmtplane.DefaultSshdPort))
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec}, cfg.Hub.HostID, logger)
// Agent v0.142.0: a signed Docker engine step (`11` §5.8) — ring 1 and every undo; under the heavy-op gate.
dockerExec := osupdate.DockerStepExecutor{Leg: osLeg, Guest: firstGuest(px),
Gate: func(ctx context.Context) (func(), error) {
release, busy, ok := heavyOps.TryAcquire("os-docker-step")
if !ok {
return nil, fmt.Errorf("busy: %s", busy)
}
return release, nil
}}
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
AfterInstall: func(ctx context.Context) {
ok, total, degraded := capability.Summarize(probeAll(ctx))
names := make([]string, 0, len(degraded))
for _, d := range degraded {
names = append(names, d.Name)
}
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
}}
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, bundleExec}, cfg.Hub.HostID, logger)
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
@@ -1112,6 +1152,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
},
}
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil {
localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) {
// Let the controller finish bringing its apps back after the backup, then run (still under the gate).
select {
case <-ctx.Done():
return
case <-time.After(90 * time.Second):
}
_ = osLeg.Run(ctx, vmid, "night")
})
}
if localTokens != nil {
defer localTokens.Close()
}
@@ -1429,6 +1480,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
go runJanitor(ctx, jd)
}
// R-444 (`09` §3 decision 139): the weekly guest disk trim — `pct fstrim <vmid>` of every owned, running guest,
// Wednesday from 10:00 local, daytime only, under the one-heavy-op gate. Not part of the errc fan-out: a trim job
// must never be able to bring the agent down.
if cfg.DiskTrim.Enabled() {
dtMode := proxmox.RunnerMode(cfg.Privileged.Mode)
if dtMode == "" {
dtMode = proxmox.RunnerSudo
}
dtRunner := &proxmox.ExecRunner{Mode: dtMode, SudoPath: cfg.Privileged.SudoPath}
dtGuests := localapi.NewStaleLockController(px, dtRunner, reconcile.DefaultPool, logger)
if dtGuests != nil {
diskTrim := fstrim.New(dtRunner, dtGuests, heavyOps,
filepath.Join(cfg.OOB.WithDefaults().StateDir, "guest-disk-trim.json"), logger)
collector.SetGuestDiskTrimReporter(diskTrim)
go diskTrim.Run(ctx)
}
} else {
logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)")
}
if lanLoop != nil {
lanServers = 1
go func() { errc <- lanLoop.Run(ctx) }()
@@ -1831,12 +1901,15 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary
InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F)
Store: store,
Storage: observer,
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
Tokens: tokens,
BackupCadence: cfg.Backup.BackupCadence(),
// R-894: the newest success per tier on disk — the due-check's fallback when the storage cannot be
// read right after a restart. Same state dir as restore-test-state.json.
LastKnownBackups: backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json")),
Storage: observer,
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
Tokens: tokens,
BackupCadence: cfg.Backup.BackupCadence(),
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
Disks: hostOps,
DiskGate: storageGateAdapter{gate: gate, hostID: cfg.Hub.HostID},
@@ -1851,7 +1924,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
ConfigPath: cfg.SourcePath,
StateDir: cfg.WGTunnel.WithDefaults().StateDir,
SmbCredsDir: cfg.Privileged.SmbCredsDir,
ControllerSwap: guestBinder, // Phase 1: agentic controller update — in-guest image swap
ControllerSwap: guestBinder, // Phase 1: agentic controller update — in-guest image swap
GuestsStateDir: "/var/lib/felhom-agent/guests", // R-523: <vmid>/bootstrap + controller-parked marker
// F2-b: recover a guest left with a stale vzdump lock by a reboot-during-backup. Reads + start
// go through the API client; the `pct unlock` is the one fenced root-CLI op (no API equivalent).
@@ -2008,7 +2081,7 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger)
// pbs coord. The live reporter lists snapshots directly (fresh store, last-known-good fallback) so
// the selftest reflects exactly what a freshly-restarted daemon's first collect emits.
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargetsFromPVE(cfg, px, logger), pbs.NewSnapshotStore(), pbs.DefaultLiveSnapshotTimeout, logger)
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, nil, nil, pbsReporter, cfg.Hub.HostID, version, logger)
collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, nil, nil, pbsReporter, cfg.Hub.HostID, version, logger)
// R-109: wire the backup-target resolver here TOO. Without it selftest=hub would print a recipe whose
// backup_target reads unknown/agent_backup_config_unavailable while the daemon's is resolved — and
// this one-shot exists precisely so "the report it would send" can be trusted to match.
@@ -2648,6 +2721,9 @@ func (e *escrowCeremonyErr) Error() string { return e.err.Error() }
// restic password auto-attach), Create (R + self-verified blob), wipe the staged secret, upload
// when asked. It PRINTS NOTHING — the output-mode shells own every byte of stdout/stderr. R is
// returned for the caller to surface exactly once; escrow.Create never logs it and neither do we.
// pbsStorageIDRe is a PVE storage id (letters, digits, '-', '_', '.'; starts with a letter) — never a path (R-861).
var pbsStorageIDRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_.-]{0,63}$`)
func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger, opts escrowCeremonyOpts) (escrowCeremonyOutcome, *escrowCeremonyErr) {
var out escrowCeremonyOutcome
storage := opts.storage
@@ -2657,6 +2733,16 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
if storage == "" {
return out, &escrowCeremonyErr{kind: "usage", err: fmt.Errorf("selftest=escrow-create requires -storage <pbs-storage-id> (or escrow.pbs_storage_id)")}
}
// R-861 (v0.146.0): this runs as ROOT through FELHOM_ESCROW, but agent.json is owned by the agent user. So the
// paths a root run reads never come from it: the PVE secret dir and the WireGuard state dir are the fixed defaults,
// and the storage id is a plain PVE id (no slash, no dot-dot) — a crafted id or dir would read another root file.
if os.Geteuid() == 0 {
cfg.Backup.PBSSecretDir = ""
cfg.WGTunnel.StateDir = ""
}
if !pbsStorageIDRe.MatchString(storage) {
return out, &escrowCeremonyErr{kind: "usage", err: fmt.Errorf("selftest=escrow-create: storage id %q is not a plain PVE storage id", storage)}
}
out.Storage = storage
keyPath := cfg.Backup.PBSEncKeyPath(storage)
if _, err := os.Stat(keyPath); err != nil {
@@ -3478,8 +3564,237 @@ func (f *selftestFlag) Set(v string) error {
f.mode = "identity-consume"
case "controller-swap":
f.mode = "controller-swap"
case "os-update":
f.mode = "os-update"
case "os-facts", "live-restore": // agent v0.142.0
f.mode = v
case "wgtunnel": // dispatched since S3 but refused here until 2026-10-04 (TestSelftestFlag_AcceptsEveryDispatchedMode)
f.mode = "wgtunnel"
default:
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v)
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update|os-facts|live-restore)", v)
}
return nil
}
// newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and
// the once-per-night marker live in the agent's own os/ dir.
func newOSLeg(cfg config.Config, client *hub.Client, px *proxmox.Client, logger *slog.Logger) *osupdate.Leg {
mode := proxmox.RunnerMode(cfg.Privileged.Mode)
if mode == "" {
mode = proxmox.RunnerSudo
}
l := &osupdate.Leg{
Runner: &proxmox.ExecRunner{Mode: mode, SudoPath: cfg.Privileged.SudoPath},
Logger: logger,
PlanDir: osupdate.DefaultPlanDir,
StatePath: filepath.Join(osupdate.DefaultPlanDir, "last-night-run"),
// The host step (agent v0.141.0) runs only on an appliance install; the wrapper re-checks the ROOT-owned record.
Appliance: cfg.IsAppliance(),
Tunnel: newTunnelProber(cfg, px),
}
if client != nil {
l.Hub = client
}
return l
}
// runSelftestOSUpdate is the OS leg's DEBUG ACTION (agent v0.140.0): one pass for -vmid, now, exactly as the night
// runs it after a backup — the hub's os_update block (fetched fresh), the wrapper via sudo, the health wait, the
// report to the hub — with trigger "debug" (never throttled, and it does NOT count as a night run for approval).
// Run it as the agent user: sudo -u felhom-agent felhom-agent --config … --selftest=os-update -vmid 9201
func runSelftestOSUpdate(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
if vmid <= 0 {
fmt.Fprintln(os.Stderr, "selftest=os-update: -vmid is required")
return 2
}
client, err := hub.NewClient(cfg.Hub, logger)
if err != nil {
fmt.Fprintln(os.Stderr, "selftest=os-update: hub client:", err)
return 1
}
px, perr := newProxmoxClient(cfg)
if perr != nil {
fmt.Fprintln(os.Stderr, "selftest=os-update: proxmox client:", perr)
return 1
}
leg := newOSLeg(cfg, client, px, logger)
blk, source, ok := selftestOSBlock(ctx, client, leg.PlanDir)
if !ok {
fmt.Fprintln(os.Stderr, "selftest=os-update:", source)
return 1
}
leg.SetBlock(blk)
b := leg.Block()
fmt.Printf("=== felhom-agent %s selftest=os-update vmid=%d ring=%d enabled=%v guest-release=%v host-release=%v appliance=%v block=%s ===\n",
version, vmid, b.Ring, b.Enabled, b.Release != nil, b.HostRelease != nil, leg.Appliance, source)
start := time.Now()
pass := leg.Run(ctx, vmid, "debug")
worst := pass.Guest
for i, rep := range []osupdate.Report{pass.Guest, pass.Host, pass.Docker} {
if rep.Layer == "" {
fmt.Printf(" %s step: skipped (see the log line above)\n", []string{"guest", "host", "docker"}[i])
continue
}
printJSON("os-update report ("+rep.Layer+")", map[string]any{"run_id": rep.RunID, "ring": rep.Ring, "release_id": rep.ReleaseID,
"mode": rep.Mode, "outcome": rep.Outcome, "healthy": rep.Healthy, "health_reason": rep.HealthReason,
"upgraded": rep.Upgraded, "pending": len(rep.Pending), "not_covered": rep.NotCovered,
"restart_needed": rep.RestartNeeded, "reboot_needed": rep.RebootNeeded, "wrapper_seconds": rep.PassSeconds,
"refused": rep.Refused, "docker_engine": rep.DockerEngine, "authority": rep.Authority})
if !(rep.Outcome == "applied" || rep.Outcome == "nothing" || rep.Outcome == "inventory" || rep.Outcome == "skipped") {
worst = rep
}
}
fmt.Printf(" pass took %s\n", time.Since(start).Round(100*time.Millisecond))
switch worst.Outcome {
case "applied", "nothing", "inventory", "skipped":
return 0
}
return 1
}
// selftestOSBlock is the debug pass's os_update block (R-866, v0.144.0): the hub's, fetched fresh; when the hub cannot
// be reached, the block the daemon last saved — named in the selftest's first line, so a pass with the hub away can
// be exercised by hand (the daemon's own leg already ran from its last block; the selftest stopped). No saved block
// and no hub → not run (ok=false), never a guessed block.
func selftestOSBlock(ctx context.Context, f interface {
FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error)
}, planDir string) (*hub.WireOSUpdate, string, bool) {
resp, err := f.FetchDesiredState(ctx)
if err == nil {
return resp.DesiredState.OSUpdate, "hub", true
}
saved, at, ok := osupdate.LoadSavedBlock(planDir)
if !ok {
return nil, fmt.Sprintf("desired state: %v — and no saved block (the daemon saves one when the hub sends it)", err), false
}
return saved, fmt.Sprintf("SAVED(%s; hub unreachable: %v)", at.UTC().Format(time.RFC3339), err), true
}
// runSelftestFacts prints the versions the host report carries (R-852, agent v0.142.0) — read-only.
//
// sudo -u felhom-agent felhom-agent --config … --selftest=os-facts -vmid 9201
func runSelftestFacts(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
if vmid <= 0 {
fmt.Fprintln(os.Stderr, "selftest=os-facts: -vmid is required")
return 2
}
px, _ := newProxmoxClient(cfg)
leg := newOSLeg(cfg, nil, px, logger)
start := time.Now()
f, err := leg.Facts(ctx, vmid)
if err != nil {
fmt.Fprintln(os.Stderr, "selftest=os-facts:", err)
return 1
}
var v any
_ = json.Unmarshal(f, &v)
printJSON(fmt.Sprintf("facts (vmid %d, %s)", vmid, time.Since(start).Round(100*time.Millisecond)), v)
return 0
}
// runSelftestLiveRestore is the ONE-TIME live-restore step (`09` decision 87) as a debug action — the night leg does
// the same before a ring-0 Docker step. It prints the container ids before and after (they must not change).
//
// sudo -u felhom-agent felhom-agent --config … --selftest=live-restore -vmid 9202
func runSelftestLiveRestore(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
if vmid <= 0 {
fmt.Fprintln(os.Stderr, "selftest=live-restore: -vmid is required")
return 2
}
px, _ := newProxmoxClient(cfg)
leg := newOSLeg(cfg, nil, px, logger)
if err := leg.EnsureLiveRestore(ctx, time.Now().UTC().Format("20060102T150405Z"), vmid); err != nil {
fmt.Fprintln(os.Stderr, "selftest=live-restore:", err)
return 1
}
fmt.Println("live-restore: on (see the wrapper's LIVE-RESTORE line above for the container ids)")
return 0
}
// newTunnelProber reads the box's REAL tunnel (R-841, agent v0.141.0): the cloudflared container in each running
// customer guest — a guest that binds /mnt/felhom-drives, the same rule the OS wrapper's R10 uses — through the
// existing `pct exec [0-9]* -- docker inspect -f *` sudoers line.
func newTunnelProber(cfg config.Config, px *proxmox.Client) hub.CloudflaredProber {
mode := proxmox.RunnerMode(cfg.Privileged.Mode)
if mode == "" {
mode = proxmox.RunnerSudo
}
return hub.GuestTunnelProber{
Runner: &proxmox.ExecRunner{Mode: mode, SudoPath: cfg.Privileged.SudoPath},
Guests: customerGuests(px),
}
}
// customerGuests lists the running guests that bind /mnt/felhom-drives — the box's customer guest(s), the same rule the
// OS wrapper's R10 uses. Shared by the tunnel probe, the facts read and the signed Docker step.
func customerGuests(px *proxmox.Client) func(ctx context.Context) ([]int, error) {
return func(ctx context.Context) ([]int, error) {
if px == nil {
return nil, fmt.Errorf("no proxmox client")
}
gs, err := px.ListLXC(ctx)
if err != nil {
return nil, err
}
var out []int
for _, g := range gs {
if g.Status != "running" {
continue
}
gc, err := px.GuestConfig(ctx, g.VMID)
if err != nil {
continue
}
for _, v := range gc.MountPoints() {
if src, _, _ := strings.Cut(v, ","); src == "/mnt/felhom-drives" {
out = append(out, g.VMID)
break
}
}
}
return out, nil
}
}
// firstGuest is the single customer guest (an error when there is none).
func firstGuest(px *proxmox.Client) func(ctx context.Context) (int, error) {
f := customerGuests(px)
return func(ctx context.Context) (int, error) {
v, err := f(ctx)
if err != nil {
return 0, err
}
if len(v) == 0 {
return 0, fmt.Errorf("no running customer guest")
}
return v[0], nil
}
}
// factsReporter feeds the host report's `system` stanza (R-852, agent v0.142.0) from the wrapper's read-only facts
// mode, at most every 10 minutes (each read is ~2 s of pct exec; the host reports every 15 min).
type factsReporter struct {
leg *osupdate.Leg
guest func(ctx context.Context) (int, error)
mu sync.Mutex
at time.Time
vmid int
facts json.RawMessage
err error
}
func (f *factsReporter) SystemFacts(ctx context.Context) (int, json.RawMessage, error) {
f.mu.Lock()
defer f.mu.Unlock()
if !f.at.IsZero() && time.Since(f.at) < 10*time.Minute {
return f.vmid, f.facts, f.err
}
f.at = time.Now()
f.vmid, f.err = f.guest(ctx)
if f.err != nil {
f.facts = nil
return 0, nil, f.err
}
f.facts, f.err = f.leg.Facts(ctx, f.vmid)
return f.vmid, f.facts, f.err
}
@@ -0,0 +1,31 @@
package main
import (
"context"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// R-861 (agent v0.146.0): the root escrow ceremony builds a file path from the storage id; an id that is a path is
// refused before anything is read. RED-PROOF: drop the pbsStorageIDRe check → the "../" ids reach the key stat and
// come back as a "setup" error instead of "usage".
func TestEscrowCeremony_StorageIDIsNeverAPath(t *testing.T) {
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
for _, id := range []string{"../../../etc/shadow", "a/b", "/etc/pve/priv/x", ".hidden", ""} {
cfg := config.Default()
cfg.Escrow.PBSStorageID = "" // the flag decides here
_, e := escrowCeremony(context.Background(), cfg, lg, escrowCeremonyOpts{storage: id})
if e == nil || e.kind != "usage" {
t.Errorf("storage id %q was not refused as usage (got %+v)", id, e)
}
}
cfg := config.Default()
cfg.Backup.PBSSecretDir = t.TempDir()
_, e := escrowCeremony(context.Background(), cfg, lg, escrowCeremonyOpts{storage: "felhom-pbs"})
if e == nil || e.kind != "setup" {
t.Fatalf("control: a plain id must pass the check and fail later on the missing key (setup), got %+v", e)
}
}
@@ -0,0 +1,55 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
)
type r866Fetcher struct{ err error }
func (f r866Fetcher) FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error) {
if f.err != nil {
return nil, f.err
}
r := &hub.DesiredStateResponse{}
r.DesiredState.OSUpdate = &hub.WireOSUpdate{Ring: 1, Enabled: true}
return r, nil
}
// R-866 (v0.144.0). THE NIGHT'S SHAPE (A3, Tester 1 box, hub blocked): `selftest=os-update: desired state: hub:
// transport error … connect: invalid argument` — the debug pass could not run at all. Now it runs from the block the
// daemon saved, and its header says so.
// COMPANION RED-PROOF: return at once on a fetch error in selftestOSBlock → "the pass did not run from the saved block".
func TestR866_DebugPassUsesTheSavedBlockWhenTheHubIsAway(t *testing.T) {
dir := t.TempDir()
leg := &osupdate.Leg{PlanDir: dir}
r := &hub.DesiredStateResponse{}
r.DesiredState.OSUpdate = &hub.WireOSUpdate{Ring: 0, Enabled: true}
leg.OnDesiredState(context.Background(), r) // the daemon received a block and saved it
away := r866Fetcher{err: errors.New("hub: transport error: connect: invalid argument")}
b, src, ok := selftestOSBlock(context.Background(), away, dir)
if !ok || b == nil || b.Ring != 0 {
t.Fatalf("the pass did not run from the saved block: ok=%v block=%+v src=%q", ok, b, src)
}
if !strings.HasPrefix(src, "SAVED(") || !strings.Contains(src, "hub unreachable") {
t.Fatalf("the header must say the block is the saved one: %q", src)
}
// the hub reachable: its block wins, and the header says "hub"
b, src, ok = selftestOSBlock(context.Background(), r866Fetcher{}, dir)
if !ok || b.Ring != 1 || src != "hub" {
t.Fatalf("hub block not used: %+v %q", b, src)
}
}
// No hub and nothing saved: the pass does not run on a guessed block.
func TestR866_NoHubNoSavedBlockDoesNotRun(t *testing.T) {
_, why, ok := selftestOSBlock(context.Background(), r866Fetcher{err: errors.New("down")}, t.TempDir())
if ok || !strings.Contains(why, "no saved block") {
t.Fatalf("ok=%v why=%q", ok, why)
}
}
+74
View File
@@ -0,0 +1,74 @@
package main
import (
"go/ast"
"testing"
)
// R-894 — the on-disk backup record is WIRED on the daemon path (the built-but-never-wired class).
// main → runDaemon → buildLocalAPIServer, and inside it the localapi.Options literal carries
// LastKnownBackups built by backup.NewBackupSuccessState. An AST walk, not a string match, for the
// reasons in escrow_recover_wiring_test.go.
//
// COMPANION RED-PROOF (observed): delete the `LastKnownBackups:` line from buildLocalAPIServer → this
// fails with "localapi.Options in buildLocalAPIServer has no LastKnownBackups field". Restored.
func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
if !callsWithin(f, "main")["runDaemon"] || !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
t.Fatal("main → runDaemon → buildLocalAPIServer is broken — the path this test asserts is not the live one")
}
var field, built bool
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
cl, ok := n.(*ast.CompositeLit)
if !ok {
return true
}
sel, ok := cl.Type.(*ast.SelectorExpr)
if !ok {
return true
}
if pkg, _ := sel.X.(*ast.Ident); pkg == nil || pkg.Name+"."+sel.Sel.Name != "localapi.Options" {
return true
}
for _, el := range cl.Elts {
kv, ok := el.(*ast.KeyValueExpr)
if !ok {
continue
}
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "LastKnownBackups" {
field = true
if callsIn(kv.Value)["backup.NewBackupSuccessState"] {
built = true
}
}
}
return true
})
}
if !field {
t.Fatal("localapi.Options in buildLocalAPIServer has no LastKnownBackups field")
}
if !built {
t.Fatal("LastKnownBackups is not built by backup.NewBackupSuccessState")
}
}
func callsIn(n ast.Node) map[string]bool {
out := map[string]bool{}
ast.Inspect(n, func(n ast.Node) bool {
if ce, ok := n.(*ast.CallExpr); ok {
if fn, ok := ce.Fun.(*ast.SelectorExpr); ok {
if x, ok := fn.X.(*ast.Ident); ok {
out[x.Name+"."+fn.Sel.Name] = true
}
}
}
return true
})
return out
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"os"
"regexp"
"strings"
"testing"
)
// Every mode the dispatcher (`switch selftest.mode`) runs must be ACCEPTED by the --selftest flag. Found live
// 2026-10-04: --selftest=os-update had a dispatch case and a function but the flag's allow-list refused it, so the
// debug action could not run. Red-proof: drop the "os-update" case from selftestFlag.Set and this fails.
func TestSelftestFlag_AcceptsEveryDispatchedMode(t *testing.T) {
src, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
s := string(src)
i := strings.Index(s, "switch selftest.mode {")
if i < 0 {
t.Fatal("dispatch switch not found")
}
block := s[i:]
block = block[:strings.Index(block, "\n\t}\n")]
modes := regexp.MustCompile(`(?m)^\tcase "([a-z-]+)":`).FindAllStringSubmatch(block, -1)
if len(modes) < 5 {
t.Fatalf("parsed only %d dispatch cases — the parser is wrong", len(modes))
}
var bad []string
for _, m := range modes {
var f selftestFlag
if err := f.Set(m[1]); err != nil {
bad = append(bad, m[1])
}
}
if len(bad) > 0 {
t.Fatalf("dispatched but refused by --selftest: %v", bad)
}
}
+13 -1
View File
@@ -43,7 +43,7 @@ func main() {
func run() error {
var (
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update")
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | agent_config_update")
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
@@ -52,6 +52,7 @@ func run() error {
fstype = flag.String("fstype", "ext4", "for storage_wipe: the filesystem to mkfs after wipe")
agentVer = flag.String("agent-version", "", "for agent_update: the target agent version (e.g. 0.70.1)")
sha256Hex = flag.String("sha256", "", "for agent_update: the pinned lowercase-hex sha256 of the target binary")
bundleSHA = flag.String("bundle-sha256", "", "for agent_config_update: the pinned sha256 of felhom-config-bundle.json (R-840)")
keyFile = flag.String("key", "", "operator signing key (ssh private key / sk- key handle) for ssh-keygen -Y sign")
ttl = flag.Duration("ttl", 30*time.Minute, "validity window from now (issued_at..expires_at)")
nonce = flag.String("nonce", "", "explicit nonce (default: a fresh 128-bit random nonce)")
@@ -95,6 +96,17 @@ func run() error {
}
pj, _ := json.Marshal(map[string]string{"version": *agentVer, "sha256": *sha256Hex})
params = string(pj)
case "agent_config_update":
// R-840: the box's root-owned files. The ROOT wrapper verifies this signature itself and refuses a bundle
// whose sha256 is not exactly this one.
if *agentVer == "" || *bundleSHA == "" {
return fmt.Errorf("agent_config_update needs -agent-version and -bundle-sha256 (the pinned bundle hash)")
}
if !isHex64(*bundleSHA) {
return fmt.Errorf("agent_config_update -bundle-sha256 must be 64 lowercase hex chars (got %d)", len(*bundleSHA))
}
pj, _ := json.Marshal(map[string]string{"agent_version": *agentVer, "bundle_sha256": *bundleSHA})
params = string(pj)
default:
params = "{}"
}
+63 -3
View File
@@ -61,7 +61,7 @@ set -euo pipefail
# Script provenance — logged into every bake transcript next to the baked controller tag, so an
# archive can always be traced to the script that produced it. Bump on any behavior change.
GOLDEN_SCRIPT_VERSION="3.0.0"
GOLDEN_SCRIPT_VERSION="3.2.0"
VMID="${1:-9100}"
TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
@@ -112,7 +112,15 @@ for i in $(seq 1 30); do
if pct exec "$VMID" -- getent hosts download.docker.com >/dev/null 2>&1; then break; fi
sleep 1
done
pct exec "$VMID" -- bash -c '
# v3.1.0 (`11` §5.8): GOLDEN_DOCKER_PKGS pins the APPROVED Docker engine set (the hub's newest Docker release, all six
# "name=version"); without it the newest stable set is installed and the bake log says so.
GOLDEN_DOCKER_PKGS="${GOLDEN_DOCKER_PKGS:-}"
if [[ -n "$GOLDEN_DOCKER_PKGS" ]]; then
echo "[golden] Docker engine set PINNED to the approved release: $GOLDEN_DOCKER_PKGS"
else
echo "[golden] WARNING: GOLDEN_DOCKER_PKGS not set — installing the newest stable Docker set, not an approved one"
fi
pct exec "$VMID" -- env GOLDEN_DOCKER_PKGS="$GOLDEN_DOCKER_PKGS" bash -c '
set -e
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
@@ -122,7 +130,54 @@ pct exec "$VMID" -- bash -c '
echo "deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update -qq
apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
if [ -n "$GOLDEN_DOCKER_PKGS" ]; then
apt-get install -y -qq $GOLDEN_DOCKER_PKGS >/dev/null
else
apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
fi
dpkg-query -W containerd.io docker-buildx-plugin docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin 2>/dev/null | sed "s/^/ installed: /"
'
# v3.2.0 (Part F of the R-840 brief, `11` §5.3): GOLDEN_GUEST_PKGS = the newest APPROVED guest release, as
# "name=version …" (the hub's os_releases row for layer guest, IN FORCE — never a cancelled test approval). The bake
# brings every package the template HAS to exactly that version, under felhom-os-apply's rules: never a package the
# template lacks (--only-upgrade), never newer than approved, never a removal or a new package (a simulation is checked
# first and the bake FAILS on either). Empty = no approved guest release in force: the template's versions stay, and
# the box's first night installs whatever release is approved then. Either way the bake PRINTS the first-night count:
# how many installed packages are older than the approved version (target 0).
GOLDEN_GUEST_PKGS="${GOLDEN_GUEST_PKGS:-}"
pct exec "$VMID" -- env GOLDEN_GUEST_PKGS="$GOLDEN_GUEST_PKGS" bash -c '
set -e
export DEBIAN_FRONTEND=noninteractive
if [ -z "$GOLDEN_GUEST_PKGS" ]; then
echo "[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a"
echo "[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): $(apt list --upgradable 2>/dev/null | grep -c /)"
exit 0
fi
want=""
for nv in $GOLDEN_GUEST_PKGS; do
n=${nv%%=*}; v=${nv#*=}
cur=$(dpkg-query -W -f="\${Version}" "$n" 2>/dev/null) || continue # not in the template: never added
dpkg --compare-versions "$cur" lt "$v" && want="$want $n=$v"
done
if [ -n "$want" ]; then
sim=$(apt-get -s install --only-upgrade -o Dpkg::Options::=--force-confold $want)
if echo "$sim" | grep -q "^Remv "; then echo "[golden] FATAL: the approved guest set would REMOVE a package"; echo "$sim" | grep "^Remv "; exit 1; fi
for p in $(echo "$sim" | awk "/^Inst /{print \$2}"); do
dpkg-query -W "$p" >/dev/null 2>&1 || { echo "[golden] FATAL: the approved guest set would ADD $p - not in the template"; exit 1; }
done
apt-get install -y -qq --only-upgrade -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef $want >/dev/null
echo "[golden] approved guest release installed: $(echo $want | wc -w) package(s) brought to the approved version"
else
echo "[golden] approved guest release: the template already runs every approved version"
fi
left=0
for nv in $GOLDEN_GUEST_PKGS; do
n=${nv%%=*}; v=${nv#*=}
cur=$(dpkg-query -W -f="\${Version}" "$n" 2>/dev/null) || continue
dpkg --compare-versions "$cur" lt "$v" && { left=$((left+1)); echo " still older: $n $cur < $v"; }
done
echo "[golden] first-night count vs the approved guest release: $left (target 0)"
[ "$left" -eq 0 ] || { echo "[golden] FATAL: $left package(s) stayed older than the approved release"; exit 1; }
'
echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …"
# containerd-snapshotter (Docker 28+/29 default) keeps the IMAGE content store under
@@ -135,9 +190,12 @@ echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshott
# layout (a container's `df /` reports the single volume, phase-0 spike). Since v3.0.0 /var/lib/docker
# is a BIND of <volume>/docker rather than the mp0 mount itself, wired immediately below; data-root
# still needs no override because the path is unchanged. Log caps kill the most common runaway.
# v3.1.0: "live-restore": true (`09` decision 87) — a Docker engine update then restarts no app (`11` C5). A box made
# from this golden never needs the agent's one-time live-restore-on step. NEVER removed by a plain restart (R-835).
pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
{
"features": { "containerd-snapshotter": false },
"live-restore": true,
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
@@ -175,6 +233,8 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
# mean containerd-snapshotter is still on (the split would leave images on the rootfs).
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
# v3.1.0 ASSERTION: live-restore is ON in the running daemon (decision 87), or the bake fails closed.
pct exec "$VMID" -- bash -c 'lr=$(docker info --format "{{.LiveRestoreEnabled}}" 2>/dev/null); [ "$lr" = "true" ] && echo " live-restore: on" || { echo "[golden] FATAL: live-restore is $lr, expected true (decision 87)"; exit 1; }'
# ASSERTION 1 (RETARGETED v3.0.0, not removed). /var/lib/docker must be a real mount — now the V-c
# bind of <volume>/docker rather than the mp0 mount itself. Still fails closed on the same failure:
# if the bind did not take, Docker's data-root silently sits on the OS rootfs and the golden ships
+8
View File
@@ -0,0 +1,8 @@
# /etc/felhom/crash-guard.conf — read by /usr/local/sbin/felhom-crash-guard (`11` §5.9).
# The LIMIT-th unclean stop within WINDOW_MINUTES leaves the box off. Decided by CC unattended — operator may reverse.
LIMIT=3
WINDOW_MINUTES=60
# kernel.panic while armed: seconds after a crash before the kernel restarts the box.
PANIC_SECONDS=10
# A tripped guard re-arms after this many hours of normal running (or `felhom-crash-guard rearm`).
REARM_HOURS=24
+112 -105
View File
@@ -1,30 +1,38 @@
# felhom-agent sudoers allowlist — the NARROW host-root surface (slice 5 Phase B, doc 03 §3/§7).
# felhom-agent sudoers allowlist — the NARROW host-root surface (slice 5 Phase B, doc 03 §3/§7; narrowed R-861).
#
# Install as a drop-in: /etc/sudoers.d/felhom-agent (mode 0440, root:root), validated with
# `visudo -cf`. The agent runs as the non-root `felhom-agent` service user and shells out via
# `sudo -n` with FIXED argument vectors (no shell). The fine-grained validation is done IN
# the agent BEFORE exec (internal/storage/validate.go): UUIDs against a strict hex regex,
# mount paths confined+traversal-checked, SMART devices whitelisted to raw disks, LVM names
# charset-checked. These sudoers wildcards are the COARSE allowlist; the agent is the fine
# gate, so a wildcard can never be abused by a value the agent didn't already validate.
# Install as a drop-in: /etc/sudoers.d/felhom-agent (mode 0440, root:root), validated with `visudo -cf`. It rides the
# signed config bundle (R-840). The agent runs as the non-root `felhom-agent` user and shells out via `sudo -n` with
# FIXED argument vectors (no shell).
#
# Binary paths MUST match the agent config (privileged.systemctl/install/smartctl/lvs). Adjust
# for your distro (Debian/PVE shown). A missing/declined entry degrades the agent with a
# warning (SMART→UNKNOWN, mount→logged error), it does not crash.
# R-861 (agent v0.146.0) — EXACT PATTERNS, NOT GLOBS. A sudoers `*` in the ARGUMENTS also matches spaces, so
# `pct set [0-9]* -onboot 1` matched `pct set 100 --dev0 /dev/sda -onboot 1` (a raw host disk for the guest), and
# `mount --bind /mnt/*/felhom-data /mnt/felhom-drives/*` matched a `..` path onto /etc. Every argument list that varies
# is now a sudo regular expression (`^...$`, sudo >= 1.9.10; Debian 13 / PVE 9 ship 1.9.16): one value per slot, a
# fixed character set, no `..`, no extra argument. Lines with no variable part stay literal. The patterns are pinned
# by the capability manifest (every real call must match: TestManifestCoveredBySudoers) and by injection cases that
# must NOT match (configs/test_sudoers_patterns.py, and live with `sudo -l -U felhom-agent` on the demo boxes).
#
# R-861 — NO FILE THE AGENT WROTE IS INSTALLED WHERE ROOT READS IT. The `install` lines are gone: a mount unit, a
# dnsmasq drop-in, the WireGuard config and the OOB sshd config + key go through `felhom-priv-apply`, a root wrapper
# from the bundle that checks the CONTENT against the agent's own renderers; the guest pre-start hook and the shared
# drive parent are FIXED files that come with the bundle itself; the agent binary is replaced only by an
# operator-signed agent_update that `felhom-os-apply` verifies as root (`felhom-selfupdate-guarded apply` is no longer
# here). The two remaining root runs of agent code (FELHOM_ESCROW, the guest hook) therefore run only a signed binary.
#
# Binary paths MUST match the agent config (privileged.systemctl/install/smartctl/lvs). A missing/declined entry
# degrades the agent with a warning (SMART→UNKNOWN, mount→logged error), it does not crash; the capability probe
# reports it to the hub.
Cmnd_Alias FELHOM_MOUNT = \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/* /etc/systemd/system/*.mount, \
/usr/local/sbin/felhom-priv-apply ^unit mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$, \
/usr/bin/systemctl daemon-reload, \
/usr/bin/systemctl enable --now -- *.mount, \
/usr/bin/systemctl disable -- *.mount, \
/usr/bin/systemctl stop -- *.mount
/usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl ^stop -- mnt-[A-Za-z0-9_.\\-]+\.mount$
Cmnd_Alias FELHOM_DISK = \
/usr/sbin/smartctl -a -j /dev/sd[a-z]*, \
/usr/sbin/smartctl -a -j /dev/nvme[0-9]*n[0-9]*, \
/usr/sbin/smartctl -a -j /dev/vd[a-z]*, \
/usr/sbin/smartctl -a -j /dev/hd[a-z]*, \
/usr/sbin/lvs --reportformat json --units b -o lv_name\,data_percent\,metadata_percent -- *, \
/usr/sbin/smartctl ^-a -j /dev/(sd[a-z]+|nvme[0-9]+n[0-9]+|vd[a-z]+|hd[a-z]+)$, \
/usr/sbin/lvs ^--reportformat json --units b -o lv_name\,data_percent\,metadata_percent -- [A-Za-z0-9_.+-]+(/[A-Za-z0-9_.+-]+)?$, \
/usr/sbin/pvs --reportformat json --noheadings -o pv_name, \
/usr/sbin/zpool status -P
@@ -35,9 +43,9 @@ Cmnd_Alias FELHOM_DISK = \
# (the wildcard only ever names a path the agent itself created), and the bootstrap file the agent
# writes there is the only thing these touch. ':' is escaped per sudoers grammar.
Cmnd_Alias FELHOM_PROVISION = \
/usr/bin/chown -R 100000\:100000 /var/lib/felhom-agent/guests/*, \
/usr/sbin/pct set [0-9]* -mp[0-9]* /var/lib/felhom-agent/guests/*, \
/usr/sbin/pct set [0-9]* -onboot 1
/usr/bin/chown ^-R 100000\:100000 /var/lib/felhom-agent/guests/[0-9]+(/bootstrap)?$, \
/usr/sbin/pct ^set [0-9]+ -mp[0-9]+ /var/lib/felhom-agent/guests/[0-9]+/bootstrap\,mp\=/[A-Za-z0-9/_.-]+(\,ro\=1)?$, \
/usr/sbin/pct ^set [0-9]+ -onboot 1$
# Disk inspection + format (slice 8C + Impl-1). blkid/lsblk read the device's data-bearing evidence
# (the agent decides data-bearing-ness from THIS, never the caller's claim). Format goes ONLY through
@@ -45,66 +53,54 @@ Cmnd_Alias FELHOM_PROVISION = \
# mkfs the OS disk — the wrapper re-checks the catastrophic cases (system disk / LVM PV / foreign mount)
# as root and refuses, and the agent's unclaimed-disk filter (claim.go) is the primary guard above it.
Cmnd_Alias FELHOM_FORMAT = \
/usr/sbin/blkid -p -o export /dev/*, \
/usr/bin/lsblk -J -o NAME\,FSTYPE\,PTTYPE\,MOUNTPOINT /dev/*, \
/usr/local/sbin/felhom-mkfs-guarded /dev/* *
/usr/sbin/blkid ^-p -o export /dev/[^ ]+$, \
/usr/bin/lsblk ^-J -o NAME\,FSTYPE\,PTTYPE\,MOUNTPOINT /dev/[^ ]+$, \
/usr/local/sbin/felhom-mkfs-guarded ^/dev/[^ ]+ (ext4|xfs)$
# LAN split-horizon resolver (internal/lanresolver): the agent manages a host-side dnsmasq that
# answers *.<customer-domain> with each guest's live LAN IP. install only ever writes felhom-*.conf
# drop-ins (from agent-written /tmp temp files); the two `pct exec` reads are FIXED command vectors
# answers *.<customer-domain> with each guest's live LAN IP. A felhom-*.conf drop-in reaches /etc/dnsmasq.d
# only through felhom-priv-apply, which allows exactly the lines the resolver renders (R-861: a `dhcp-script=` would
# run as root); the two `pct exec` reads are FIXED command vectors
# (the guest's eth0 IPv4 + the controller's pulled controller.yaml for the domain) — NOT a general
# `pct exec`. systemctl is scoped to the dnsmasq unit only. The agent never edits /etc/resolv.conf.
Cmnd_Alias FELHOM_DNSMASQ = \
/usr/bin/apt-get install -y -q dnsmasq, \
/usr/bin/install -m 0644 /tmp/felhom-resolver-*.conf /etc/dnsmasq.d/felhom-*.conf, \
/usr/local/sbin/felhom-priv-apply ^dnsmasq /tmp/felhom-resolver-[0-9]+\.conf felhom-[a-z0-9][a-z0-9._-]*\.conf$, \
/usr/bin/systemctl enable --now dnsmasq, \
/usr/bin/systemctl reload dnsmasq, \
/usr/bin/systemctl restart dnsmasq, \
/usr/bin/rm -f /etc/dnsmasq.d/felhom-*.conf, \
/usr/sbin/pct exec [0-9]* -- ip -4 -o addr show dev eth0, \
/usr/sbin/pct exec [0-9]* -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller.yaml
/usr/bin/rm ^-f /etc/dnsmasq\.d/felhom-[a-z0-9][a-z0-9._-]*\.conf$, \
/usr/sbin/pct ^exec [0-9]+ -- ip -4 -o addr show dev eth0$, \
/usr/sbin/pct ^exec [0-9]+ -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller\.yaml$
# Guest mountpoint lifecycle (intermediary-mount re-architecture + C1 net). The pre-start self-heal hook
# wrapper is installed once into the PVE snippets dir (from an agent-written /tmp file) and registered
# per-guest; decommission/eject DELETE the dead mountpoint slot so a missing bind source can't brick the
# guest at next boot (the B3 C1 fix). The agent fine-validates the vmid (numeric) + slot (mp[0-9]+) and
# the snippet path is fixed — the wildcards are the coarse allowlist. The install SOURCE is a
# random-named agent temp (os.CreateTemp, audit B1 — a fixed /tmp name was a local TOCTOU), hence the
# glob; the DESTINATION stays pinned. The `mkdir -p` creates the snippets dir on a FRESH box —
# `install` won't create parents, so without it the hook install failed silently on Day-0 boxes
# (B2, DRILL-day0-cleanroom-2026-07-03; fixed agent v0.63.0).
# Guest mountpoint lifecycle (intermediary-mount re-architecture + C1 net). The pre-start self-heal hook is a FIXED file
# from the config bundle (/var/lib/vz/snippets/felhom-guest-hook.sh — R-861: the agent no longer installs it from /tmp;
# Proxmox runs it as root at every guest start). The agent only registers it per guest, deletes a dead mountpoint slot
# (the B3 C1 fix) and reboots a guest to activate binds — each with an exact vmid / slot.
Cmnd_Alias FELHOM_GUESTHOOK = \
/usr/bin/mkdir -p /var/lib/vz/snippets, \
/usr/bin/install -m 0755 -- /tmp/felhom-guest-hook-*.sh /var/lib/vz/snippets/felhom-guest-hook.sh, \
/usr/sbin/pct set [0-9]* --hookscript local\:snippets/felhom-guest-hook.sh, \
/usr/sbin/pct set [0-9]* --delete mp[0-9]*, \
/usr/sbin/pct reboot [0-9]*
/usr/sbin/pct ^set [0-9]+ --hookscript local\:snippets/felhom-guest-hook\.sh$, \
/usr/sbin/pct ^set [0-9]+ --delete mp[0-9]+$, \
/usr/sbin/pct ^reboot [0-9]+$
# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
# /mnt/felhom-drives (self-bind + make-shared + a boot-persistence systemd unit) and binds/unbinds each
# drive's felhom-data namespace UNDERNEATH it so the change propagates into the running guest live (no
# pct, no reboot). The agent fine-validates the drive name + confines paths before any exec; the trailing
# `*` (matching the comma-laden mp spec) mirrors the existing FELHOM_PROVISION pattern.
# `lxc-info -n <vmid> -p -H` resolves the guest init PID for the GuestSeesMount / bound_under_parent check
# (a READ — the drive-gate's "is the drive live in the guest?" signal); WITHOUT it the non-root agent gets
# an empty PID and reports every drive absent (multi-drive flapping, audit 2026-06-29). `make-private`
# isolates the parent's peer group on FIRST setup only (EnsureSharedParent guards on mountpoint, so it
# never re-churns a live parent); without it the parent stays in root's group and submounts double.
# /mnt/felhom-drives (self-bind + make-shared) and binds/unbinds each drive's felhom-data namespace UNDERNEATH it so the
# change propagates into the running guest live. The boot-persistence script + unit are FIXED files from the config
# bundle (R-861: the agent no longer installs them from /tmp); the agent only enables the unit. A drive name is one
# path segment that cannot start with a dot (no `..`); `lxc-info -n <vmid> -p -H` resolves the guest init PID for the
# GuestSeesMount check; `make-private` isolates the parent's peer group on FIRST setup only.
Cmnd_Alias FELHOM_INTERMEDIARY = \
/usr/bin/mkdir -p /mnt/felhom-drives, \
/usr/bin/mkdir -p /mnt/felhom-drives/*, \
/usr/bin/mkdir -p /mnt/*/felhom-data, \
/usr/bin/chown 100000\:100000 /mnt/*/felhom-data, \
/usr/bin/mkdir ^-p /mnt/felhom-drives/[A-Za-z0-9_-][A-Za-z0-9_.-]*$, \
/usr/bin/mkdir ^-p /mnt/[A-Za-z0-9_-][A-Za-z0-9_.-]*/felhom-data$, \
/usr/bin/chown ^100000\:100000 /mnt/[A-Za-z0-9_-][A-Za-z0-9_.-]*/felhom-data$, \
/usr/bin/mount --bind /mnt/felhom-drives /mnt/felhom-drives, \
/usr/bin/mount --make-shared /mnt/felhom-drives, \
/usr/bin/mount --make-private /mnt/felhom-drives, \
/usr/bin/mount --bind /mnt/*/felhom-data /mnt/felhom-drives/*, \
/usr/bin/umount /mnt/felhom-drives/*, \
/usr/bin/install -m 0755 -- /tmp/felhom-shared-parent-*.sh /usr/local/sbin/felhom-shared-parent.sh, \
/usr/bin/install -m 0644 -- /tmp/felhom-shared-parent-*.service /etc/systemd/system/felhom-shared-parent.service, \
/usr/bin/mount ^--bind /mnt/[A-Za-z0-9_-][A-Za-z0-9_.-]*/felhom-data /mnt/felhom-drives/[A-Za-z0-9_-][A-Za-z0-9_.-]*$, \
/usr/bin/umount ^/mnt/felhom-drives/[A-Za-z0-9_-][A-Za-z0-9_.-]*$, \
/usr/bin/systemctl enable felhom-shared-parent.service, \
/usr/bin/lxc-info -n [0-9]* -p -H, \
/usr/sbin/pct set [0-9]* -mp8 /mnt/felhom-drives*
/usr/bin/lxc-info ^-n [0-9]+ -p -H$, \
/usr/sbin/pct ^set [0-9]+ -mp8 /mnt/felhom-drives\,mp\=/mnt/felhom-drives$
# Controller-swap / managed auto-update (Option A, non-root). The agent owns the in-guest controller
# image SWAP (it survives the controller being killed mid-swap): read the baked image ref, check the
@@ -119,11 +115,11 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# the agent strict-validates the ref (controllerImageRe) before the write.
# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct exec [0-9]* -- cat /etc/felhom-controller-image, \
/usr/sbin/pct exec [0-9]* -- docker image inspect *, \
/usr/sbin/pct exec [0-9]* -- docker inspect -f *, \
/usr/sbin/pct exec [0-9]* -- systemctl restart felhom-controller-bootstrap.service, \
/usr/sbin/pct exec [0-9]* -- tee /etc/felhom-controller-image
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/felhom-controller-image$, \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
@@ -131,7 +127,15 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
# with no API equivalent (snapshot-delete + start go through the API token); the agent fine-validates the
# vmid (numeric) before exec — the `[0-9]*` is the coarse allowlist.
Cmnd_Alias FELHOM_STALELOCK = \
/usr/sbin/pct unlock [0-9]*
/usr/sbin/pct ^unlock [0-9]+$
# Weekly guest disk trim (R-444, operator ruling `09` §3 decision 139). A thin pool only ever grows from blocks the
# guest has FREED: `fstrim` inside the unprivileged container is refused (FITRIM: Operation not permitted), so the host
# trims the guest's mounts. Measured on demo-hp 2026-10-06: `pct fstrim 9201` rc 0 in 24.4 s, pool 65.53 % -> 33.40 %,
# apps kept answering. ONE exact pattern: a vmid and nothing else — no `--ignore-mountpoints`, no second argument
# (pinned: TestSudoersFstrimRuleIsExact). The agent runs it on a weekly daytime timer under the heavy-op gate.
Cmnd_Alias FELHOM_FSTRIM = \
/usr/sbin/pct ^fstrim [0-9]+$
# Restore-test scratch teardown (F-LEAK, Campaign 8, v0.110.0). A restore-test whose restore FAILS
# leaves a scratch guest the API token CANNOT destroy: `FelhomAgentGuest` is granted at /pool/felhom and
@@ -160,8 +164,9 @@ Cmnd_Alias FELHOM_SCRATCH_TEARDOWN = \
# into the guest through the existing shared bind (an unprivileged LXC cannot mount NFS/CIFS itself).
# A NAS is NOT a drive — no durable-id, no SMART, no wipe; these grants only install/enable/remove the
# unit pair. The agent fine-validates every value (share name, server, export, uid/gid, creds path) before
# any unit is rendered (internal/storage/netmount.go ValidateNetworkMountSpec); the trailing globs are the
# COARSE allowlist. The `.mount` install/enable/disable/stop reuse FELHOM_MOUNT; this alias adds the
# any unit is rendered (internal/storage/netmount.go ValidateNetworkMountSpec); the unit FILE reaches
# /etc/systemd/system only through `felhom-priv-apply unit` (FELHOM_MOUNT), which requires nosuid,nodev on a network
# share (R-861). The `.mount` enable/disable/stop reuse FELHOM_MOUNT; this alias adds the
# `.automount` variants + the unit-file removal. The unit FILE name is the systemd-escaped mountpoint,
# which always begins `mnt-felhom` (the mountpoint is /mnt/felhom-drives/<name>), so the rm glob is scoped
# to felhom mount units only. mkdir of the mountpoint reuses FELHOM_INTERMEDIARY's /mnt/felhom-drives/*.
@@ -176,51 +181,46 @@ Cmnd_Alias FELHOM_SCRATCH_TEARDOWN = \
# behind (the campaign accumulated 10 stub-shaped leftovers). rmdir ONLY (never rm -rf): it refuses
# a non-empty dir, so unexpected data is preserved, not destroyed — a fail-safe grant.
Cmnd_Alias FELHOM_NETMOUNT = \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/* /etc/systemd/system/*.automount, \
/usr/bin/systemctl enable --now -- *.automount, \
/usr/bin/systemctl disable -- *.automount, \
/usr/bin/systemctl stop -- *.automount, \
/usr/bin/systemctl reset-failed -- mnt-felhom*, \
/usr/bin/rmdir /mnt/felhom-drives/*, \
/usr/bin/rm -f /etc/systemd/system/mnt-felhom*
/usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.automount$, \
/usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.automount$, \
/usr/bin/systemctl ^stop -- mnt-[A-Za-z0-9_.\\-]+\.automount$, \
/usr/bin/systemctl ^reset-failed -- mnt-felhom[A-Za-z0-9_.\\-]*\.(mount|automount)$, \
/usr/bin/rmdir ^/mnt/felhom-drives/[A-Za-z0-9_-][A-Za-z0-9_.-]*$, \
/usr/bin/rm ^-f /etc/systemd/system/mnt-felhom[A-Za-z0-9_.\\-]*\.(mount|automount)$
# Offsite WG tunnel (S3, doc 06 §3.3). The agent manages wg-quick@wg-felhom as an agent-managed
# host service (the dnsmasq/lanresolver shape): conf staged in the agent-owned StateDir (never
# /tmp), installed 0600 to the FIXED destination, unit enable/restart/disable. The ONLY wg read
# /tmp), installed 0600 to the FIXED destination by `felhom-priv-apply wg`, which refuses any key renderConf never
# writes (R-861: PostUp/PreUp run as root under wg-quick), unit enable/restart/disable. The ONLY wg read
# is `latest-handshakes` — `wg show <if> dump` is FORBIDDEN everywhere (its interface line
# carries the PRIVATE KEY; the S1 session-log incident). Both install paths are FIXED (no glob):
# the agent has exactly one tunnel conf to manage.
# carries the PRIVATE KEY; the S1 session-log incident). Source and destination are fixed in the wrapper.
Cmnd_Alias FELHOM_WG = \
/usr/bin/apt-get install -y -q wireguard-tools, \
/usr/bin/install -o root -g root -m 0600 -- /var/lib/felhom-agent/wg/wg-felhom.conf /etc/wireguard/wg-felhom.conf, \
/usr/local/sbin/felhom-priv-apply wg, \
/usr/bin/systemctl enable --now wg-quick@wg-felhom, \
/usr/bin/systemctl restart wg-quick@wg-felhom, \
/usr/bin/systemctl disable --now wg-quick@wg-felhom, \
/usr/bin/wg show wg-felhom latest-handshakes
# Agent self-update (TASK D1, SPIKE-agent-selfupdate-2026-07-05). The agent downloads the
# operator-SIGNED binary (sha256 pinned in the signed op — neither hub nor Gitea compromise can
# substitute it), verifies the sha in-process, then hands off to the guarded wrapper, which
# RE-verifies the sha as root, confines the staged path to /var/lib/felhom-agent/selfupdate/,
# performs the A/B flip (atomic same-fs rename, .prev retained) and schedules a detached restart.
# The apply args are a COARSE glob (spike S4b: sudoers fnmatch makes a [a-f0-9]* sha pattern
# first-char-only anyway) — the wrapper's own sha re-verify + path confinement is the real gate.
# `rollback` is normally run by felhom-agent-rollback.service (root, OnFailure=), not via sudo;
# granting it here keeps the verb probe-able (capability self-check) and operator-invokable.
# Agent self-update (TASK D1; R-861). The A/B flip (`felhom-selfupdate-guarded apply`) is NO LONGER the agent's: the
# agent hands the operator-SIGNED agent_update to felhom-os-apply (FELHOM_OSAPPLY, mode agent_update), which verifies
# the signature as root and only then runs the flip. Until v0.146.0 the agent passed the sha itself, so a compromised
# agent could install any binary — the binary FELHOM_ESCROW and the guest hook run as root. `commit` (clear the pending
# marker) and `rollback` (pending-guarded revert, normally run by felhom-agent-rollback.service) stay.
Cmnd_Alias FELHOM_SELFUPDATE = \
/usr/local/sbin/felhom-selfupdate-guarded apply /var/lib/felhom-agent/selfupdate/* *, \
/usr/local/sbin/felhom-selfupdate-guarded commit, \
/usr/local/sbin/felhom-selfupdate-guarded rollback
# Dedicated OOB sshd (TASK H1). The agent manages felhom-sshd like wg-felhom/dnsmasq: it RENDERS the
# config (Port from its claim) + the operator's authorized_keys, validates with `sshd -t`, and reloads
# (never restart-on-change [SF-2]). Both install SOURCES are the agent-owned staged files under
# StateDir; both DESTINATIONS are FIXED. `sshd -t/-T` are the validate/discover reads. The
# (never restart-on-change [SF-2]). Both files reach /etc/felhom-sshd only through felhom-priv-apply (R-861): the config
# must be the ONE template with only the Port varying (an AuthorizedKeysFile the agent owns + `StrictModes no` would be
# a root login), the key file one plain public key without options. `sshd -t/-T` are the validate/discover reads. The
# systemctl verbs are SCOPED to felhom-sshd only. reset-failed precedes a deliberate restart [SF-5].
# NOTHING here can touch the stock sshd, :22, or /etc/ssh.
Cmnd_Alias FELHOM_SSHD = \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/sshd_config /etc/felhom-sshd/sshd_config, \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op /etc/felhom-sshd/authorized_keys/felhom-op, \
/usr/local/sbin/felhom-priv-apply sshd-config, \
/usr/local/sbin/felhom-priv-apply sshd-key, \
/usr/sbin/sshd -t -f /var/lib/felhom-agent/felhom-sshd/sshd_config, \
/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config, \
/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config, \
@@ -270,8 +270,8 @@ Cmnd_Alias FELHOM_OOB = \
/usr/sbin/nft list set inet felhom_oob ssh_port, \
/usr/sbin/nft flush set inet felhom_oob operator_ips, \
/usr/sbin/nft flush set inet felhom_oob ssh_port, \
/usr/sbin/nft add element inet felhom_oob operator_ips *, \
/usr/sbin/nft add element inet felhom_oob ssh_port *
/usr/sbin/nft ^add element inet felhom_oob operator_ips \{ [0-9.]+(/[0-9]+)? \}$, \
/usr/sbin/nft ^add element inet felhom_oob ssh_port \{ [0-9]+ \}$
# Escrow ceremony (controller-driven, TASK 2026-07-13; mechanics validated by
# SPIKE-controller-escrow-2026-07-13). ONE fixed argv — sudoers matches the argument vector
@@ -299,10 +299,17 @@ Cmnd_Alias FELHOM_SELFHEAL = \
# argument after the numeric vmid is a literal, so the grant cannot be widened by anything the guest or
# the hub says. The address read is deliberately NOT duplicated here — it is already FELHOM_DNSMASQ's,
# and the same command must not be granted twice under two names.
Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct exec [0-9]* -- ip route show default, \
/usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \
/usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
/usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
# OS updates, guest fast lane (`11-os-updates.md` §5.4.1, agent v0.140.0). The ONLY entry: the root wrapper with
# one plan file in the agent's own os/ dir. Every safety rule (no removal, no downgrade, no new or unlisted package,
# Debian origin only, the box's own customer guest only) lives in the wrapper, red-proved per rule
# (configs/test_felhom_os_apply.py). The agent gets NO apt grant of its own.
Cmnd_Alias FELHOM_OSAPPLY = \
/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN
Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct ^exec [0-9]+ -- ip route show default$, \
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/network/interfaces$, \
/usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \
/usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY
+235
View File
@@ -0,0 +1,235 @@
#!/usr/bin/python3
# felhom-crash-guard — a crashed host restarts by itself, but not forever (`09` decision 88, R-851, `11` §5.9).
#
# Install as /usr/local/sbin/felhom-crash-guard (0755 root:root), with felhom-crash-guard.service (boot / clean-stop)
# and felhom-crash-guard-check.timer (hourly re-arm check). Python 3, standard library only.
# Tests: configs/test_felhom_crash_guard.py (temp dirs; nothing real is touched).
#
# WHAT IT DOES
# boot early at every boot. Was the previous boot ended CLEANLY? (the clean-stop marker exists). If not, this
# boot follows an UNCLEAN stop — a kernel crash, a power cut or a hard reset (they cannot be told apart
# on these boxes: measured 2026-10-04 on demo-hp, efi_pstore is on yet saved NOTHING for a real panic;
# the journal and `last` show only "no shutdown"). It records the unclean boot, counts those in the last
# WINDOW_MINUTES, and sets kernel.panic:
# - fewer than LIMIT-1 recent unclean boots → kernel.panic = PANIC_SECONDS (a crash restarts the box);
# - LIMIT-1 or more → the guard TRIPS: kernel.panic = 0, so the LIMIT-th crash within the window
# leaves the box OFF (operator's own words: "if it crashes 3 times within one hour, it stays off").
# A tripped guard stays tripped across further boots until it re-arms.
# clean-stop ExecStop of the service: writes the clean-stop marker during an orderly shutdown or reboot.
# check hourly: a tripped guard re-arms after REARM_HOURS of normal running (since the trip AND since boot).
# rearm the operator re-arms by hand (`felhom-crash-guard rearm`).
# status prints the state.
# The state is /var/lib/felhom-crash-guard/state.json (0644: the non-root agent reads it into its host report).
# Before the service runs (very early boot) the kernel default kernel.panic = 0 applies, so a crash THAT early leaves
# the box off — the safe side: a box that cannot reach userspace must not loop.
import json
import os
import sys
import time
CONF = "/etc/felhom/crash-guard.conf"
STATE_DIR = "/var/lib/felhom-crash-guard"
DEFAULTS = {"LIMIT": 3, "WINDOW_MINUTES": 60, "PANIC_SECONDS": 10, "REARM_HOURS": 24}
class Env:
"""Paths and clock; tests replace them."""
def __init__(self, conf=CONF, state_dir=STATE_DIR, panic_path="/proc/sys/kernel/panic",
uptime_path="/proc/uptime", boot_id_path="/proc/sys/kernel/random/boot_id"):
self.conf, self.state_dir = conf, state_dir
self.panic_path, self.uptime_path, self.boot_id_path = panic_path, uptime_path, boot_id_path
def now(self):
return time.time()
def log(self, line):
print(line, file=sys.stderr, flush=True)
try:
import subprocess
subprocess.run(["logger", "-t", "felhom-crash-guard", line], timeout=10)
except Exception:
pass
def iso(t):
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(t))
def parse_iso(s):
import calendar
return calendar.timegm(time.strptime(s, "%Y-%m-%dT%H:%M:%SZ"))
def load_conf(env):
c = dict(DEFAULTS)
try:
for line in open(env.conf):
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
k, v = (x.strip() for x in line.split("=", 1))
if k in c and v.isdigit() and int(v) >= (1 if k != "PANIC_SECONDS" else 1):
c[k] = int(v)
except OSError:
pass
return c
def state_path(env):
return os.path.join(env.state_dir, "state.json")
def marker_path(env):
return os.path.join(env.state_dir, "clean-stop")
def load_state(env):
try:
with open(state_path(env)) as f:
s = json.load(f)
return s if isinstance(s, dict) else None
except (OSError, ValueError):
return None
def save_state(env, s):
os.makedirs(env.state_dir, mode=0o755, exist_ok=True)
tmp = state_path(env) + ".tmp"
with open(tmp, "w") as f:
json.dump(s, f, indent=2, sort_keys=True)
f.write("\n")
os.chmod(tmp, 0o644)
os.replace(tmp, state_path(env))
def set_panic(env, seconds):
with open(env.panic_path, "w") as f:
f.write(f"{seconds}\n")
def read(path, default=""):
try:
with open(path) as f:
return f.read().strip()
except OSError:
return default
def summarize(s, c, now):
window = c["WINDOW_MINUTES"] * 60
times = [parse_iso(t) for t in s.get("unclean_boots", [])]
after = parse_iso(s["rearmed_at"]) if s.get("rearmed_at") else 0
# a re-arm starts a fresh window (or the next unclean boot would trip again at once); the history stays
s["unclean_boots_in_window"] = sum(1 for t in times if now - t <= window and t > after)
s["unclean_boots_24h"] = sum(1 for t in times if now - t <= 86400)
s["config"] = c
s["updated_at"] = iso(now)
def boot(env):
c = load_conf(env)
now = env.now()
try:
up = float(read(env.uptime_path, "0").split()[0])
except (ValueError, IndexError):
up = 0.0
boot_at = now - up
prev = load_state(env)
first = prev is None
s = prev or {"version": 1, "unclean_boots": [], "tripped": False}
clean = os.path.exists(marker_path(env))
unclean = (not first) and (not clean)
try:
os.remove(marker_path(env))
except OSError:
pass
# keep 7 days of history (the 24 h figure and the operator's view), drop older
s["unclean_boots"] = [t for t in s.get("unclean_boots", []) if now - parse_iso(t) <= 7 * 86400]
if unclean:
s["unclean_boots"].append(iso(boot_at))
s["last_boot_at"] = iso(boot_at)
s["last_boot_unclean"] = unclean
s["boot_id"] = read(env.boot_id_path, "unknown")
summarize(s, c, now)
if not s.get("tripped") and s["unclean_boots_in_window"] >= c["LIMIT"] - 1:
s["tripped"], s["tripped_at"] = True, iso(now)
s["tripped_reason"] = (f"{s['unclean_boots_in_window']} unclean boots within {c['WINDOW_MINUTES']} minutes — "
f"the next crash leaves the box off (limit {c['LIMIT']})")
env.log(f"crash-guard: TRIPPED: {s['tripped_reason']}")
panic = 0 if s.get("tripped") else c["PANIC_SECONDS"]
set_panic(env, panic)
s["kernel_panic"] = panic
s["armed"] = not s.get("tripped")
save_state(env, s)
env.log(f"crash-guard: boot first={first} unclean={unclean} in-window={s['unclean_boots_in_window']} "
f"tripped={s.get('tripped')} kernel.panic={panic}")
return 0
def clean_stop(env):
os.makedirs(env.state_dir, mode=0o755, exist_ok=True)
with open(marker_path(env), "w") as f:
f.write(iso(env.now()) + "\n")
env.log("crash-guard: clean stop recorded")
return 0
def rearm(env, by):
c = load_conf(env)
now = env.now()
s = load_state(env) or {"version": 1, "unclean_boots": []}
was = bool(s.get("tripped"))
s["tripped"] = False
s["armed"] = True
s["rearmed_at"], s["rearmed_by"] = iso(now), by
if was:
s["last_trip"] = {"at": s.get("tripped_at"), "reason": s.get("tripped_reason")}
s.pop("tripped_at", None)
s.pop("tripped_reason", None)
summarize(s, c, now)
set_panic(env, c["PANIC_SECONDS"])
s["kernel_panic"] = c["PANIC_SECONDS"]
save_state(env, s)
env.log(f"crash-guard: RE-ARMED by {by} (was tripped: {was}); kernel.panic={c['PANIC_SECONDS']}")
return 0
def check(env):
c = load_conf(env)
now = env.now()
s = load_state(env)
if not s:
return 0
if s.get("tripped"):
since = max(parse_iso(s["tripped_at"]), parse_iso(s.get("last_boot_at", s["tripped_at"])))
if now - since >= c["REARM_HOURS"] * 3600:
return rearm(env, f"timer ({c['REARM_HOURS']} h of normal running)")
summarize(s, c, now)
save_state(env, s)
return 0
def main(argv, env=None):
env = env or Env()
cmd = argv[1] if len(argv) == 2 else ""
if cmd == "boot":
return boot(env)
if cmd == "clean-stop":
return clean_stop(env)
if cmd == "check":
return check(env)
if cmd == "rearm":
return rearm(env, "operator")
if cmd == "status":
print(json.dumps(load_state(env), indent=2, sort_keys=True))
return 0
print("usage: felhom-crash-guard boot|clean-stop|check|rearm|status", file=sys.stderr)
return 2
if __name__ == "__main__":
if os.geteuid() != 0:
print("felhom-crash-guard: must run as root", file=sys.stderr)
sys.exit(2)
sys.exit(main(sys.argv))
+7
View File
@@ -0,0 +1,7 @@
# Hourly: a tripped crash guard re-arms after REARM_HOURS of normal running (felhom-crash-guard check).
[Unit]
Description=Felhom crash guard re-arm check
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/felhom-crash-guard check
+9
View File
@@ -0,0 +1,9 @@
[Unit]
Description=Felhom crash guard re-arm check (hourly)
[Timer]
OnBootSec=15min
OnUnitActiveSec=1h
[Install]
WantedBy=timers.target
+19
View File
@@ -0,0 +1,19 @@
# felhom-crash-guard — a crashed host restarts by itself, with a limit (`09` decision 88, R-851, `11` §5.9).
# Starts early at boot (sets kernel.panic for THIS boot); its ExecStop writes the clean-stop marker during an orderly
# shutdown or reboot. A boot that finds no marker followed a crash, a power cut or a hard reset.
[Unit]
Description=Felhom crash guard (restart after a kernel crash, with a limit)
DefaultDependencies=no
After=local-fs.target
Before=sysinit.target shutdown.target
Conflicts=shutdown.target
RequiresMountsFor=/var/lib
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/felhom-crash-guard boot
ExecStop=/usr/local/sbin/felhom-crash-guard clean-stop
[Install]
WantedBy=sysinit.target
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
# felhom-agent guest pre-start self-heal hook (C1 net). PVE calls: <script> <vmid> <phase>.
/usr/local/bin/felhom-agent guest-hook "$1" "$2" || true
exit 0
+1693
View File
File diff suppressed because it is too large Load Diff
+419
View File
@@ -0,0 +1,419 @@
#!/usr/bin/python3
"""felhom-priv-apply — the ROOT half of every file the agent writes into a root-read place (R-861, `03` §3.1).
Install as /usr/local/sbin/felhom-priv-apply (0755 root:root) — it rides the signed config bundle (R-840).
WHY IT EXISTS. Until agent v0.146.0 the agent's sudoers let it `install` a file it had written itself into a place a
root program reads: a systemd .mount unit (a bind mount of an agent-owned directory over /etc/sudoers.d is a root
shell), a dnsmasq drop-in (`dhcp-script=` runs as root), the WireGuard config (`PostUp=` runs as root) and the OOB
sshd config (`AuthorizedKeysFile` + `StrictModes no`). A compromised agent PROCESS was therefore root on its host.
Now the agent stages the file and this wrapper — root-owned, delivered only by an operator-signed bundle — checks
the CONTENT against the exact grammar the agent's own renderers produce, and refuses anything else. The agent can no
longer name the destination: each verb has a fixed source and a fixed (or strictly named) destination.
Verbs (each one sudoers line, exact-match pattern):
unit <name> /var/lib/felhom-agent/units/<name> -> /etc/systemd/system/<name> (.mount | .automount)
dnsmasq <tmp> <name> /tmp/felhom-resolver-<digits>.conf -> /etc/dnsmasq.d/felhom-<...>.conf
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
sshd-key /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op -> /etc/felhom-sshd/authorized_keys/felhom-op
--self-check prints "felhom-priv-apply ok verbs=..." (the bundle's self-check)
Exit codes: 0 installed (or already identical), 2 usage, 3 refused (content or source), 4 install failed.
Every refusal is logged to the journal (tag felhom-priv-apply) with its rule; file CONTENT is never logged.
Pinned by configs/test_felhom_priv_apply.py (one test per rule, red-proofs in the R-861 audit).
"""
import ipaddress
import os
import re
import stat
import subprocess
import sys
AGENT_USER = "felhom-agent"
STATE = "/var/lib/felhom-agent"
UNITS_SRC = STATE + "/units"
UNIT_DIR = "/etc/systemd/system"
DNSMASQ_DIR = "/etc/dnsmasq.d"
WG_SRC, WG_DEST = STATE + "/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
MAX_BYTES = 64 * 1024
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key")
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
DNSMASQ_NAME_RE = re.compile(r"^felhom-[a-z0-9][a-z0-9._-]*\.conf$")
SEG = r"[A-Za-z0-9_-][A-Za-z0-9_.-]*"
WHERE_RE = re.compile(r"^/mnt/(felhom-drives/)?" + SEG + r"$")
UUID_RE = re.compile(r"^[A-Fa-f0-9]{4,}(-[A-Fa-f0-9]+){0,4}$")
HOST_RE = re.compile(r"^[A-Za-z0-9._:-]{1,255}$")
NET_PATH_RE = re.compile(r"^[A-Za-z0-9._/@+-]{1,512}$")
OPT_RE = re.compile(r"^[A-Za-z0-9_.:/@+-]+(=[A-Za-z0-9_.:/@+-]+)?$")
LOCAL_TYPES = {"ext4", "xfs", "btrfs", "exfat", "vfat", "ntfs3", "ntfs"}
NET_TYPES = {"nfs", "nfs4", "cifs"}
# Options that turn a device mount into something else, or let set-uid/device files act on the host.
FORBIDDEN_OPTS = {"bind", "rbind", "move", "rmove", "remount", "suid", "dev", "user", "users", "owner", "group",
"x-mount.mkdir", "helper"}
DESC_RE = re.compile(r"^[^\x00-\x1f\x7f]{0,200}$")
WG_KEY_RE = re.compile(r"^[A-Za-z0-9+/]{42}[AEIMQUYcgkosw480]=$")
KEY_LINE_RE = re.compile(r"^(ssh-ed25519|ssh-rsa|ecdsa-sha2-nistp(256|384|521)|sk-ssh-ed25519@openssh\.com) "
r"[A-Za-z0-9+/]+={0,3}( [ -~]{0,200})?$")
class Refused(Exception):
def __init__(self, rule, reason):
super().__init__(reason)
self.rule, self.reason = rule, reason
class Host:
"""Every filesystem / process effect, so the tests can play the box in memory."""
def agent_uid(self):
import pwd
return pwd.getpwnam(AGENT_USER).pw_uid
def read_source(self, path):
"""The staged file: a REGULAR file owned by the agent, never a symlink, at most MAX_BYTES."""
try:
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC)
except OSError as e:
raise Refused("P1", f"cannot open the staged file {path}: {e.strerror}")
try:
st = os.fstat(fd)
if not stat.S_ISREG(st.st_mode):
raise Refused("P1", f"{path} is not a regular file")
if st.st_uid != self.agent_uid():
raise Refused("P1", f"{path} is not owned by {AGENT_USER}")
if st.st_size > MAX_BYTES:
raise Refused("P1", f"{path} is larger than {MAX_BYTES} bytes")
with os.fdopen(fd, "rb") as f:
fd = -1
return f.read(MAX_BYTES + 1)
finally:
if fd >= 0:
os.close(fd)
def read_dest(self, path):
try:
with open(path, "rb") as f:
return f.read()
except OSError:
return None
def install(self, dest, data, mode):
"""Atomic, root-owned: a temp file beside the destination, fsync, rename."""
d = os.path.dirname(dest)
os.makedirs(d, mode=0o755, exist_ok=True)
tmp = os.path.join(d, f".{os.path.basename(dest)}.felhom-new.{os.getpid()}")
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
try:
with os.fdopen(fd, "wb") as f:
f.write(data)
f.flush()
os.fchown(f.fileno(), 0, 0)
os.fchmod(f.fileno(), mode)
os.fsync(f.fileno())
os.replace(tmp, dest)
except BaseException:
try:
os.remove(tmp)
except OSError:
pass
raise
def log(self, line):
print(line, file=sys.stderr)
try:
subprocess.run(["logger", "-t", "felhom-priv-apply", "--", line], timeout=10, check=False)
except (OSError, subprocess.SubprocessError):
pass
def systemd_escape_path(path):
"""`systemd-escape --path`: strip the slashes at both ends, `/` -> `-`, every byte outside [A-Za-z0-9:_.] (and a
leading `.`) -> `\\xNN`."""
p = path.strip("/")
out = []
for i, ch in enumerate(p):
if ch == "/":
out.append("-")
elif (ch.isascii() and (ch.isalnum() or ch in ":_.")) and not (i == 0 and ch == "."):
out.append(ch)
else:
out.extend("\\x%02x" % b for b in ch.encode())
return "".join(out)
def text_of(data, what):
if len(data) > MAX_BYTES:
raise Refused("P1", f"{what} is too large")
try:
text = data.decode("utf-8")
except UnicodeDecodeError:
raise Refused("P2", f"{what} is not UTF-8 text")
if "\x00" in text or "\r" in text:
raise Refused("P2", f"{what} carries a NUL or CR byte")
return text
def parse_ini(text, what):
"""[Section] / Key=Value / comments / blank lines. A key outside a section or a repeated key is refused."""
sections, cur = {}, None
for n, raw in enumerate(text.split("\n"), 1):
line = raw.strip()
if line.endswith("\\"):
# systemd joins a line ending in a backslash with the next one; this parser does not. Refused, so the two
# can never read the same bytes differently (review 2026-10-05).
raise Refused("U2", f"{what}: line {n} ends with a backslash (a continuation)")
if not line or line.startswith("#") or line.startswith(";"):
continue
m = re.match(r"^\[([A-Za-z]+)\]$", line)
if m:
cur = m.group(1)
if cur in sections:
raise Refused("U2", f"{what}: section [{cur}] twice")
sections[cur] = {}
continue
if cur is None or "=" not in line:
raise Refused("U2", f"{what}: line {n} is not Key=Value inside a section")
k, v = line.split("=", 1)
k, v = k.strip(), v.strip()
if k in sections[cur]:
raise Refused("U2", f"{what}: {cur}.{k} given twice")
sections[cur][k] = v
return sections
# ---------- the unit verb ----------
def check_unit(name, text):
if not UNIT_NAME_RE.match(name) or "/" in name:
raise Refused("U1", f"unit name {name!r} is not mnt-<escaped path>.mount|.automount")
kind = "automount" if name.endswith(".automount") else "mount"
s = parse_ini(text, name)
body = "Automount" if kind == "automount" else "Mount"
# [Unit] holds ONLY what the renderers write: Description, and After=local-fs-pre.target on a local mount. A
# Wants=/Requires=/Before= naming any unit would start it with the mount (Wants=reboot.target — found by review
# 2026-10-05), so none of them is accepted.
allowed = {"Unit": {"Description", "After"},
body: {"Where", "TimeoutIdleSec"} if kind == "automount" else {"What", "Where", "Type", "Options"},
"Install": {"WantedBy"}}
for sec, keys in s.items():
if sec not in allowed:
raise Refused("U2", f"{name}: section [{sec}] is not allowed")
bad = set(keys) - allowed[sec]
if bad:
raise Refused("U2", f"{name}: [{sec}] key(s) {sorted(bad)} not allowed")
u = s.get("Unit", {})
if not DESC_RE.match(u.get("Description", "")):
raise Refused("U2", f"{name}: Description has control characters")
if "After" in u and u["After"] != "local-fs-pre.target":
raise Refused("U2", f"{name}: After= may only be local-fs-pre.target")
inst = s.get("Install", {})
if inst and inst.get("WantedBy") != "multi-user.target":
raise Refused("U2", f"{name}: WantedBy must be multi-user.target")
m = s.get(body)
if not m or "Where" not in m:
raise Refused("U3", f"{name}: no [{body}] Where=")
where = m["Where"]
if not WHERE_RE.match(where):
raise Refused("U3", f"{name}: Where={where} is not /mnt/<name> or /mnt/felhom-drives/<name>")
if systemd_escape_path(where) + "." + kind != name:
raise Refused("U3", f"{name}: the unit name does not match Where={where}")
if kind == "automount":
t = m.get("TimeoutIdleSec", "")
if t and not re.match(r"^[0-9]{1,6}$", t):
raise Refused("U2", f"{name}: TimeoutIdleSec must be seconds")
return
what, typ = m.get("What", ""), m.get("Type", "")
opts = [o for o in m.get("Options", "").split(",") if o]
net = False
mu = re.match(r"^/dev/disk/by-uuid/(.+)$", what)
if mu:
if not UUID_RE.match(mu.group(1)):
raise Refused("U4", f"{name}: What= is not a filesystem UUID")
if typ and typ not in LOCAL_TYPES:
raise Refused("U4", f"{name}: Type={typ} is not a local filesystem")
else:
net = True
if typ not in NET_TYPES:
raise Refused("U4", f"{name}: What= is neither /dev/disk/by-uuid/<uuid> nor a network source with Type=nfs/nfs4/cifs")
if typ == "cifs":
mm = re.match(r"^//([^/]+)/(.+)$", what)
else:
mm = re.match(r"^([^/:][^:]*):(/.*)$", what)
if not mm or not HOST_RE.match(mm.group(1)) or not NET_PATH_RE.match(mm.group(2)) or ".." in mm.group(2).split("/"):
raise Refused("U4", f"{name}: What= is not a clean {typ} source")
if not where.startswith("/mnt/felhom-drives/"):
raise Refused("U3", f"{name}: a network share mounts only under /mnt/felhom-drives/")
for o in opts:
if not OPT_RE.match(o):
raise Refused("U5", f"{name}: mount option {o!r} has characters a mount option never needs")
if o.split("=", 1)[0].lower() in FORBIDDEN_OPTS or o.lower().startswith("x-mount."):
raise Refused("U5", f"{name}: mount option {o.split('=', 1)[0]!r} is not allowed")
if net and not {"nosuid", "nodev"} <= set(opts):
# A network server is outside the box: a set-uid file on it must never run as root here.
raise Refused("U5", f"{name}: a network share must carry nosuid,nodev")
# ---------- dnsmasq ----------
def _ip(v, v6=True):
try:
a = ipaddress.ip_address(v)
except ValueError:
return False
return v6 or a.version == 4
DOMAIN_RE = re.compile(r"^[A-Za-z0-9]([A-Za-z0-9-]{0,62})(\.[A-Za-z0-9]([A-Za-z0-9-]{0,62}))*$")
def check_dnsmasq(text):
for n, raw in enumerate(text.split("\n"), 1):
line = raw.strip()
if not line or line.startswith("#"):
continue
if line in ("bind-interfaces", "no-resolv"):
continue
k, _, v = line.partition("=")
if k == "listen-address" and _ip(v, v6=False):
continue
if k == "server" and (_ip(v) or (v.count("#") == 1 and _ip(v.split("#")[0]) and v.split("#")[1].isdigit())):
continue
m = re.match(r"^/([^/]+)/$", v)
if k == "local" and m and DOMAIN_RE.match(m.group(1)):
continue
m = re.match(r"^/([^/]+)/([^/]+)$", v)
if k == "address" and m and DOMAIN_RE.match(m.group(1)) and _ip(m.group(2), v6=False):
continue
raise Refused("D1", f"dnsmasq line {n} ({k or line[:20]!r}) is not one the resolver writes")
# ---------- WireGuard ----------
def check_wg(text):
s = parse_ini(text, "wg-felhom.conf")
if set(s) != {"Interface", "Peer"}:
raise Refused("W1", "wg-felhom.conf must hold exactly [Interface] and [Peer]")
i, p = s["Interface"], s["Peer"]
if set(i) - {"PrivateKey", "Address", "MTU"} or set(p) - {"PublicKey", "Endpoint", "AllowedIPs", "PersistentKeepalive"}:
raise Refused("W1", "wg-felhom.conf carries a key the agent never writes (PostUp/PreUp/... run as root)")
if not WG_KEY_RE.match(i.get("PrivateKey", "")) or not WG_KEY_RE.match(p.get("PublicKey", "")):
raise Refused("W2", "a WireGuard key is not 32 bytes of base64")
try:
a = ipaddress.ip_network(i.get("Address", ""), strict=False)
if a.version != 4 or a.prefixlen != 32:
raise ValueError
if not (1280 <= int(i.get("MTU", "1280")) <= 1500):
raise ValueError
host, _, port = p.get("Endpoint", "").rpartition(":")
if ipaddress.ip_address(host).version != 4 or not (1 <= int(port) <= 65535):
raise ValueError
for n in p.get("AllowedIPs", "").split(","):
if ipaddress.ip_network(n.strip(), strict=True).prefixlen != 32:
raise ValueError
if not (0 <= int(p.get("PersistentKeepalive", "25")) <= 3600):
raise ValueError
except ValueError:
raise Refused("W2", "an Address/MTU/Endpoint/AllowedIPs/PersistentKeepalive value is not what the agent renders")
# ---------- OOB sshd ----------
def render_sshd(port):
"""Byte-identical to felhomsshd.renderConfig (internal/felhomsshd/config.go) — pinned by a Go test."""
return ("# felhom OOB sshd — agent-managed (H1); DO NOT EDIT\n"
f"Port {port}\n"
"ListenAddress 0.0.0.0\n"
"ListenAddress ::\n"
"HostKey /etc/felhom-sshd/ssh_host_ed25519_key\n"
"PidFile /run/felhom-sshd.pid\n"
"AuthorizedKeysFile /etc/felhom-sshd/authorized_keys/%u\n"
"PasswordAuthentication no\n"
"PermitRootLogin prohibit-password\n"
"PubkeyAuthentication yes\n"
"KbdInteractiveAuthentication no\n"
"UsePAM yes\n"
"AllowUsers root felhom-op\n"
"X11Forwarding no\n"
"Subsystem sftp internal-sftp\n")
def check_sshd(text):
m = re.search(r"^Port ([0-9]{1,5})$", text, re.M)
if not m or not (1 <= int(m.group(1)) <= 65535) or int(m.group(1)) == 22:
raise Refused("S1", "sshd_config has no Port (or claims :22, the household's sshd)")
if text != render_sshd(int(m.group(1))):
raise Refused("S1", "sshd_config differs from the one fixed template (only the Port may vary)")
def check_key(text):
lines = [l for l in text.split("\n") if l.strip()]
if len(lines) > 1:
raise Refused("S2", "felhom-op's authorized_keys holds more than one key")
if lines and not KEY_LINE_RE.match(lines[0]):
raise Refused("S2", "the key line is not a plain public key (no options such as command= or from=)")
# ---------- main ----------
def plan(argv):
"""(verb, source, dest, mode, checker) for an argv, or Refused("A1")."""
if not argv or argv[0] not in VERBS:
raise Refused("A1", "usage: felhom-priv-apply unit <name> | dnsmasq <tmp> <name> | wg | sshd-config | sshd-key")
v, rest = argv[0], argv[1:]
if v == "unit" and len(rest) == 1:
if not UNIT_NAME_RE.match(rest[0]):
raise Refused("U1", f"unit name {rest[0]!r} is not mnt-<escaped path>.mount|.automount")
return v, os.path.join(UNITS_SRC, rest[0]), os.path.join(UNIT_DIR, rest[0]), 0o644, lambda t: check_unit(rest[0], t)
if v == "dnsmasq" and len(rest) == 2:
if not DNSMASQ_TMP_RE.match(rest[0]) or not DNSMASQ_NAME_RE.match(rest[1]):
raise Refused("D2", "dnsmasq wants /tmp/felhom-resolver-<digits>.conf and felhom-<name>.conf")
return v, rest[0], os.path.join(DNSMASQ_DIR, rest[1]), 0o644, check_dnsmasq
if v == "wg" and not rest:
return v, WG_SRC, WG_DEST, 0o600, check_wg
if v == "sshd-config" and not rest:
return v, SSHD_SRC, SSHD_DEST, 0o644, check_sshd
if v == "sshd-key" and not rest:
return v, KEY_SRC, KEY_DEST, 0o644, check_key
raise Refused("A1", f"wrong arguments for {v}")
def main(argv, host=None):
host = host or Host()
if argv == ["--self-check"]:
print("felhom-priv-apply ok verbs=" + ",".join(VERBS))
return 0
if len(argv) >= 3 and argv[0] == "--check":
# CHECK ONLY (tests and the Go contract tests): `--check <verb> [<name>] <file>` validates <file> as that
# verb would and installs nothing. Not in sudoers. Prints OK or the rule; never the content.
verb, file = argv[1], argv[-1]
try:
_, _, _, _, checker = plan([verb] + argv[2:-1] if verb != "dnsmasq" else
[verb, "/tmp/felhom-resolver-1.conf", argv[2]])
with open(file, "rb") as f:
checker(text_of(f.read(), file))
except Refused as e:
print(f"REFUSED [{e.rule}] {e.reason}")
return 3
print("OK")
return 0
try:
verb, src, dest, mode, checker = plan(argv)
data = host.read_source(src)
checker(text_of(data, src))
except Refused as e:
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] {' '.join(argv)[:160]}: {e.reason}")
return 2 if e.rule == "A1" else 3
if host.read_dest(dest) == data:
host.log(f"felhom-priv-apply: SAME {verb} {dest}")
return 0
try:
host.install(dest, data, mode)
except OSError as e:
host.log(f"felhom-priv-apply: FAILED {verb} {dest}: {e}")
return 4
host.log(f"felhom-priv-apply: INSTALLED {verb} {dest} ({len(data)} bytes)")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+11 -3
View File
@@ -24,6 +24,11 @@ set -u
BIN=/usr/local/bin/felhom-agent
PREV=$BIN.prev
STAGING=/var/lib/felhom-agent/selfupdate
# R-861 (agent v0.146.1): `apply` takes ONLY the root-owned copy felhom-os-apply writes after it has verified the
# operator's signature and hashed exactly those bytes (mode agent_update). The agent cannot call `apply` any more (it
# left the sudoers), and the agent's own staging dir is no longer accepted: a file in a directory the agent owns can be
# swapped between this script's sha check and its copy.
ROOT_STAGING=/var/lib/felhom-os-apply/agent-update
PENDING=$STAGING/pending.json
UNIT=felhom-agent.service
@@ -42,11 +47,14 @@ apply)
log "refusing apply: usage: apply <staged> <sha256>"
exit 2
fi
# Root-side path confinement: the staged binary MUST live in the agent's staging dir.
# Root-side path confinement: the staged binary MUST be felhom-os-apply's root-owned copy (R-861).
case "$staged" in
"$STAGING"/*) ;;
*) log "refusing apply: staged path outside $STAGING: $staged"; exit 1 ;;
"$ROOT_STAGING"/*) ;;
*) log "refusing apply: staged path outside $ROOT_STAGING: $staged"; exit 1 ;;
esac
if [ -L "$staged" ] || [ "$(stat -c %u "$staged" 2>/dev/null)" != "0" ]; then
log "refusing apply: $staged is a symlink or not root-owned"; exit 1
fi
case "$staged" in
*..*) log "refusing apply: staged path contains '..'"; exit 1 ;;
esac
+13
View File
@@ -0,0 +1,13 @@
[Unit]
Description=Felhom stable drive parent (shared bind for live drive hot-swap)
After=local-fs.target
Before=pve-guests.service
ConditionPathExists=/usr/local/sbin/felhom-shared-parent.sh
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/felhom-shared-parent.sh
[Install]
WantedBy=pve-guests.service multi-user.target
+15
View File
@@ -0,0 +1,15 @@
#!/bin/sh
# felhom stable drive parent: a SHARED bind so the agent can swap backing drives underneath it and the
# guest sees the change live (no restart). MUST run before pve-guests so the guest's parent bind inherits
# the shared peer group (slave). Installed + enabled by felhom-agent. Idempotent.
set -e
mkdir -p /mnt/felhom-drives
# Isolate + share ONLY when first creating the self-bind (a fresh boot). The self-bind inherits the root
# mount's shared peer group, so make-private detaches it (else binds under it DOUBLE via the root peer),
# then make-shared gives it its own group whose only slave is the guest's parent bind. Re-running this on
# an existing parent would churn the peer-group id and orphan the guest's slave — so guard on mountpoint.
if ! mountpoint -q /mnt/felhom-drives; then
mount --bind /mnt/felhom-drives /mnt/felhom-drives
mount --make-private /mnt/felhom-drives
mount --make-shared /mnt/felhom-drives
fi
+736
View File
@@ -0,0 +1,736 @@
#!/usr/bin/env python3
"""Tests for the config bundle (R-840, `11` §5.4.2): felhom-os-apply's `bundle` mode and `--install-bundle`, and
scripts/build-config-bundle.py. An in-memory host plays the files; nothing real is written or run. Each refusal has a
test; each test names the rule it pins. Red-proof: `audits/r840-config-bundle-2026-10-04/partB/redproof.txt`.
Run: python3 configs/test_felhom_config_bundle.py (also run by internal/osupdate's Go test)
"""
import sys
sys.dont_write_bytecode = True # importing the builder must not leave scripts/__pycache__ behind
import base64
import contextlib
import hashlib
import importlib.machinery
import importlib.util
import io
import json
import os
import pathlib
import re
import stat as statmod
import unittest
HERE = pathlib.Path(__file__).resolve().parent
REPO = HERE.parent
_loader = importlib.machinery.SourceFileLoader("osapply", os.environ.get("OSAPPLY_UNDER_TEST", str(HERE / "felhom-os-apply")))
_spec = importlib.util.spec_from_loader("osapply", _loader)
osapply = importlib.util.module_from_spec(_spec)
_loader.exec_module(osapply)
_bl = importlib.machinery.SourceFileLoader("bundlebuild", str(REPO / "scripts" / "build-config-bundle.py"))
_bs = importlib.util.spec_from_loader("bundlebuild", _bl)
builder = importlib.util.module_from_spec(_bs)
_bl.exec_module(builder)
PLAN = "/var/lib/felhom-agent/os/plan-b1.json"
BUNDLE = "/var/lib/felhom-agent/os/bundle-0.143.0.json"
HOST = "demo-hp-bb76ea"
INSTALLER = REPO.parent / "felhom.eu" / "scripts" / "felhom-host-install.sh"
class St:
def __init__(self, mode, uid):
self.st_mode, self.st_uid, self.st_size = mode, uid, 100
class Box:
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
def __init__(self, bundle_bytes, signed, oob=False, signers=True):
self.files, self.modes, self.uids = {}, {}, {}
self.dirs = {osapply.OOB_DIR} if oob else set()
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
if signers:
self.put(osapply.TRUST_SIGNERS, b'felhom-op-1 namespaces="felhom-op-v1" ssh-ed25519 AAAA felhom-op-1\n', 0o644)
self.put("/proc/sys/kernel/panic", b"0\n", 0o644)
self.plan = {"release_id": "bundle-0.143.0", "layer": "host", "mode": "bundle", "bundle": BUNDLE, "signed": signed}
self.put(PLAN, json.dumps(self.plan).encode(), 0o600, uid=999)
self.put(BUNDLE, bundle_bytes, 0o600, uid=999)
self.sig_rc, self.nonces, self.clock = 0, {}, 1791115200.0 # 2026-10-04T12:00:00Z
self.calls, self.logs, self.writes = [], [], []
self.visudo_fail = False # the WHOLE sudoers (`visudo -c`) after install
self.sudo_l = " (root) NOPASSWD: /usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json\n"
self.guard = {"armed": True, "kernel_panic": 10}
def put(self, p, data, mode, uid=0):
self.files[p], self.modes[p], self.uids[p] = data, mode, uid
# Runner interface
def now(self):
return self.clock
def log(self, line):
self.logs.append(line)
def agent_uid(self):
return 999
def verify_sig(self, signers, key_id, ns, blob, sig):
self.verified = (signers, key_id, ns)
return self.sig_rc
def read_nonces(self):
return dict(self.nonces)
def write_nonces(self, d):
self.nonces = dict(d)
def read_file(self, p):
if p not in self.files:
raise OSError("no such file")
return self.files[p].decode()
def read_bytes(self, p):
if p not in self.files:
raise OSError("no such file")
return self.files[p]
def read_staged_once(self, p, owner_uid, limit):
if p not in self.files:
raise OSError("no such file")
if self.uids[p] != owner_uid:
raise osapply.Refused("R19", f"{p} is not a regular file owned by felhom-agent")
self.staged_reads = getattr(self, "staged_reads", 0) + 1
return self.files[p]
def stat(self, p):
if p not in self.files:
raise OSError("no such file")
return St(statmod.S_IFREG | self.modes[p], self.uids[p])
def lexists(self, p):
return p in self.files
def isdir(self, p):
return p in self.dirs
def put_file(self, p, data, mode):
self.writes.append(p)
self.put(p, data, mode)
def remove(self, p):
self.writes.append("rm " + p)
del self.files[p]
def list_dir(self, p):
return sorted({k[len(p) + 1:].split("/")[0] for k in self.files if k.startswith(p + "/")})
def rmtree(self, p):
for k in [k for k in self.files if k.startswith(p + "/")]:
del self.files[k]
def check_content(self, kind, data):
return (1, f"{kind}: syntax error") if b"BROKEN-SYNTAX" in data else (0, "")
def host(self, argv, timeout=600, stdin=None):
self.calls.append(argv)
if argv[:2] == ["visudo", "-c"]:
return (1, "", "parse error") if self.visudo_fail else (0, "ok", "")
if argv[:2] == ["sudo", "-n"]:
return 0, self.sudo_l, ""
if argv[-2:] == ["/usr/local/sbin/felhom-priv-apply", "--self-check"]:
body = self.files.get("/usr/local/sbin/felhom-priv-apply", b"")
return (0, "felhom-priv-apply ok verbs=unit\n", "") if b"VERBS" in body else (1, "", "boom")
if argv[-1] == "--self-check":
body = self.files.get("/usr/local/sbin/felhom-os-apply", b"")
return (0, "felhom-os-apply ok bundle-format=1 files=22\n", "") if b"BUNDLE_OP" in body else (1, "", "boom")
if argv[-1] == "/usr/local/sbin/felhom-selfupdate-guarded":
return 2, "", "felhom-selfupdate-guarded: usage: ...\n"
if argv[-1] == "status" and argv[0].endswith("felhom-crash-guard"):
return 0, json.dumps(self.guard), ""
if argv[:3] == ["systemctl", "enable", "--now"] and "felhom-crash-guard.service" in argv:
self.put("/proc/sys/kernel/panic", f"{self.guard['kernel_panic']}\n".encode(), 0o644)
return 0, "", ""
def signed_job(sha, version="0.143.0", host=HOST, op="agent_config_update", nonce="b1",
issued="2026-10-04T11:50:00Z", expires="2026-10-04T12:30:00Z"):
blob = json.dumps({"expires_at": expires, "issued_at": issued, "key_id": "felhom-op-1", "nonce": nonce, "op": op,
"params": {"agent_version": version, "bundle_sha256": sha},
"target": {"guest_id": "", "host_id": host}}, sort_keys=True).encode()
return {"blob_b64": base64.b64encode(blob).decode(), "sig": "-----BEGIN SSH SIGNATURE-----\nx\n-----END SSH SIGNATURE-----\n"}
def real_bundle(version="0.143.0"):
data = builder.build(version)
return data, hashlib.sha256(data).hexdigest()
def edited_bundle(edit):
"""The real bundle, with edit(files_list) applied and every sha recomputed — a SIGNED bundle with bad content."""
b = json.loads(builder.build("0.143.0"))
edit(b["files"])
for e in b["files"]:
e["sha256"] = hashlib.sha256(base64.b64decode(e["content_b64"])).hexdigest()
data = (json.dumps(b, indent=1, sort_keys=True) + "\n").encode()
return data, hashlib.sha256(data).hexdigest()
def replace_content(files, path, fn):
for e in files:
if e["path"] == path:
e["content_b64"] = base64.b64encode(fn(base64.b64decode(e["content_b64"]))).decode()
def run(box, argv=None, environ=None):
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
rc = osapply.main(argv or ["felhom-os-apply", "--plan", PLAN], runner=box, environ=environ or {})
line = [l for l in buf.getvalue().splitlines() if l.startswith("OSAPPLY-REPORT ")][-1]
return rc, json.loads(line[len("OSAPPLY-REPORT "):])
def box_files(box):
return {p: box.files[p] for p in box.files if p in osapply.BUNDLE_DESTS}
class Install(unittest.TestCase):
def test_fresh_box_gets_every_file_and_a_record(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
b = rep["bundle"]
# every path but the four OOB ones (no belt on this box)
self.assertEqual(len(b["written"]), len(osapply.BUNDLE_FILES) - 4, b)
self.assertEqual(len(b["skipped"]), 4)
self.assertEqual(box.files["/etc/sudoers.d/felhom-agent"], (REPO / "configs" / "felhom-agent.sudoers").read_bytes())
self.assertEqual(box.modes["/etc/sudoers.d/felhom-agent"], 0o440)
rec = json.loads(box.files[osapply.BUNDLE_RECORD])
self.assertEqual((rec["agent_version"], rec["bundle_sha256"], rec["authority"]), ("0.143.0", sha, "signed"))
self.assertIn("b1", box.nonces, "the job is consumed")
self.assertEqual(b["self_check"]["crash_guard"], {"armed": True, "kernel_panic": 10})
self.assertIn(["systemctl", "daemon-reload"], box.calls)
def test_sudoers_is_written_after_every_wrapper(self):
"""Order: a referenced wrapper is in place before the sudoers line that allows it."""
data, sha = edited_bundle(lambda f: f.reverse()) # the bundle lists the sudoers FIRST; the wrapper must reorder
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
w = [p for p in box.writes if p in osapply.BUNDLE_DESTS]
self.assertEqual(w[-1], "/etc/sudoers.d/felhom-agent")
self.assertLess(w.index("/usr/local/sbin/felhom-os-apply"), w.index("/etc/sudoers.d/felhom-agent"))
def test_identical_box_writes_nothing(self):
"""The demo boxes' case: hand-copied files equal to the release → 0 written, all 'same'."""
data, sha = real_bundle()
box = Box(data, signed_job(sha))
for dest, src, mode, _, policy in osapply.BUNDLE_FILES:
if policy != "oob":
box.put(dest, (REPO / "configs" / src).read_bytes(), mode)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["bundle"]["written"], [])
self.assertEqual(rep["bundle"]["same"], len(osapply.BUNDLE_FILES) - 5) # 4 oob skipped + crash-guard.conf kept
self.assertEqual(rep["bundle"]["kept"], ["/etc/felhom/crash-guard.conf"])
def test_a_wrong_mode_is_rewritten(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put("/etc/sudoers.d/felhom-agent", (REPO / "configs" / "felhom-agent.sudoers").read_bytes(), 0o644)
rc, rep = run(box)
self.assertIn("/etc/sudoers.d/felhom-agent", rep["bundle"]["written"])
self.assertEqual(box.modes["/etc/sudoers.d/felhom-agent"], 0o440)
def test_tuned_crash_guard_conf_is_kept(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put("/etc/felhom/crash-guard.conf", b"LIMIT=5\n", 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files["/etc/felhom/crash-guard.conf"], b"LIMIT=5\n")
def test_oob_files_only_on_a_box_with_the_belt(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), oob=True)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertIn("/etc/sudoers.d/felhom-op", rep["bundle"]["written"])
self.assertEqual(rep["bundle"]["skipped"], [])
def test_previous_copies_are_kept(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put("/usr/local/sbin/felhom-pbs-apply", b"#!/bin/bash\necho old\n", 0o755)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files[rep["bundle"]["prev_dir"] + "/usr/local/sbin/felhom-pbs-apply"], b"#!/bin/bash\necho old\n")
class Refusals(unittest.TestCase):
"""Each: refused, and NOTHING on the box changed."""
def refused(self, box, code):
before = dict(box_files(box))
rc, rep = run(box)
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
self.assertEqual(box_files(box), before, "a refusal changed a file")
self.assertNotIn(osapply.BUNDLE_RECORD, box.files)
return rep
def test_wrong_sha_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job("0" * 64))
self.refused(box, "R18")
self.assertEqual(box.nonces, {}, "a wrong sha must not burn the job")
def test_bad_signature_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.sig_rc = 255
self.refused(box, "R3")
def test_other_op_is_refused(self):
data, sha = real_bundle()
self.refused(Box(data, signed_job(sha, op="agent_update")), "R3")
def test_other_host_is_refused(self):
data, sha = real_bundle()
self.refused(Box(data, signed_job(sha, host="demo-felhom-8363b5")), "R3")
def test_expired_job_is_refused(self):
data, sha = real_bundle()
self.refused(Box(data, signed_job(sha, expires="2026-10-04T11:55:00Z")), "R3")
def test_replayed_job_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.nonces = {"b1": box.clock + 600}
self.refused(box, "R3")
def test_version_mismatch_is_refused(self):
data, sha = real_bundle()
self.refused(Box(data, signed_job(sha, version="0.142.1")), "R18")
def test_sudoers_failing_visudo_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/sudoers.d/felhom-agent", lambda c: c + b"BROKEN-SYNTAX\n"))
self.refused(Box(data, signed_job(sha)), "R18")
def test_sudoers_dropping_the_route_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/sudoers.d/felhom-agent",
lambda c: c.replace(b"/usr/local/sbin/felhom-os-apply --plan", b"/bin/true --plan")))
self.refused(Box(data, signed_job(sha)), "R18")
def test_wrapper_without_bundle_mode_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/usr/local/sbin/felhom-os-apply",
lambda c: c.replace(b'BUNDLE_OP = "agent_config_update"', b'BUNDLE_OPX = 1')))
self.refused(Box(data, signed_job(sha)), "R18")
def test_python_syntax_error_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/usr/local/sbin/felhom-crash-guard", lambda c: c + b"\ndef (\n"))
self.refused(Box(data, signed_job(sha)), "R18")
def test_unit_with_runtime_directory_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/systemd/system/felhom-mgmt-watchdog.service",
lambda c: c + b"RuntimeDirectory=sshd\n"))
self.refused(Box(data, signed_job(sha)), "R18")
def test_agent_unit_not_as_the_agent_user_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/systemd/system/felhom-agent.service",
lambda c: c.replace(b"User=felhom-agent", b"User=root")))
self.refused(Box(data, signed_job(sha)), "R18")
def test_a_bundle_that_changes_a_signer_is_refused(self):
"""R17: the trust root is not a bundle's to change — not even a signed one."""
def add(f):
f.append({"path": osapply.TRUST_SIGNERS, "content_b64": base64.b64encode(b"evil-key\n").decode()})
data, sha = edited_bundle(add)
box = Box(data, signed_job(sha))
self.refused(box, "R17")
self.assertIn(b"felhom-op-1", box.files[osapply.TRUST_SIGNERS])
def test_a_path_outside_the_table_is_refused(self):
def add(f):
f.append({"path": "/etc/shadow", "content_b64": base64.b64encode(b"root::0:0\n").decode()})
data, sha = edited_bundle(add)
self.refused(Box(data, signed_job(sha)), "R16")
def test_content_not_matching_its_sha_is_refused(self):
b = json.loads(builder.build("0.143.0"))
b["files"][0]["content_b64"] = base64.b64encode(b"#!/bin/bash\nexit 0\n").decode()
data = json.dumps(b).encode()
self.refused(Box(data, signed_job(hashlib.sha256(data).hexdigest())), "R18")
def test_bundle_outside_the_plan_dir_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.plan["bundle"] = "/tmp/bundle-0.143.0.json"
box.files[PLAN] = json.dumps(box.plan).encode()
self.refused(box, "R1")
def test_bundle_not_owned_by_the_agent_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.uids[BUNDLE] = 0
self.refused(box, "R1")
def test_no_trust_file_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
del box.files[osapply.TRUST_FILE]
self.refused(box, "R3")
class SelfCheckUndo(unittest.TestCase):
def assert_restored(self, box, before, rep):
self.assertTrue(rep["bundle"]["rolled_back"], rep)
self.assertEqual(box_files(box), before, "the previous files must be back, byte for byte")
self.assertNotIn(osapply.BUNDLE_RECORD, box.files)
def with_old_files(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put("/usr/local/sbin/felhom-pbs-apply", b"#!/bin/bash\necho old\n", 0o755)
box.put("/etc/sudoers.d/felhom-agent", b"# old sudoers\n", 0o440)
return box
def test_route_missing_after_install_puts_everything_back(self):
box = self.with_old_files()
before = dict(box_files(box))
box.sudo_l = " (root) NOPASSWD: /bin/true\n"
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assert_restored(box, before, rep)
self.assertEqual(box.calls[-1], ["visudo", "-c"], "the undo re-checks the whole sudoers")
def test_visudo_failing_after_install_puts_everything_back(self):
box = self.with_old_files()
before = dict(box_files(box))
box.visudo_fail = True
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assert_restored(box, before, rep)
def test_crash_guard_disagreeing_with_kernel_panic_puts_everything_back(self):
box = self.with_old_files()
before = dict(box_files(box))
box.guard = {"armed": True, "kernel_panic": 10}
box.host_orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:3] == ["systemctl", "enable", "--now"]:
box.calls.append(argv)
return 0, "", "" # the unit "started" but kernel.panic stayed 0
return box.host_orig(argv, timeout, stdin)
box.host = host
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assert_restored(box, before, rep)
class TrustBootstrap(unittest.TestCase):
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), signers=False)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.verified[0], osapply.PINNED_SIGNERS, "verified against the pinned key, nothing else")
self.assertTrue(rep["bundle"]["signers_created"])
self.assertEqual(box.files[osapply.TRUST_SIGNERS], osapply.pinned_signers_line().encode())
self.assertEqual(box.modes[osapply.TRUST_SIGNERS], 0o644)
def test_missing_signers_and_a_job_the_pinned_key_did_not_sign(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), signers=False)
box.sig_rc = 255
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"))
self.assertNotIn(osapply.TRUST_SIGNERS, box.files)
def test_present_signers_are_never_touched(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put(osapply.TRUST_SIGNERS, b'felhom-op-2 namespaces="felhom-op-v1" ssh-ed25519 BBBB felhom-op-2\n', 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.verified[0], osapply.TRUST_SIGNERS)
self.assertFalse(rep["bundle"]["signers_created"])
self.assertIn(b"felhom-op-2", box.files[osapply.TRUST_SIGNERS])
def test_agent_writable_signers_are_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.uids[osapply.TRUST_SIGNERS] = 999
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"))
def test_pinned_operator_key_equals_the_installers(self):
"""The bootstrap key is exactly the one felhom-host-install.sh pins (OPERATOR_KEY_OPERATIONAL_*)."""
text = INSTALLER.read_text()
kid = re.search(r'^OPERATOR_KEY_OPERATIONAL_ID="([^"]+)"', text, re.M).group(1)
line = re.search(r'^OPERATOR_KEY_OPERATIONAL_LINE="([^"]+)"', text, re.M).group(1)
self.assertEqual((osapply.PINNED_OPERATOR_KEY_ID, osapply.PINNED_OPERATOR_KEY_LINE), (kid, line))
# and the file format is the installer's printf, byte for byte
self.assertIn("printf '%s namespaces=\"felhom-op-v1\" %s\\n'", text)
class InstallerEntry(unittest.TestCase):
def test_installer_installs_without_a_signature(self):
data, sha = real_bundle()
box = Box(data, None)
box.put("/root/bundle.json", data, 0o600)
rc, rep = run(box, ["felhom-os-apply", "--install-bundle", "/root/bundle.json", "--sha256", sha])
self.assertEqual(rc, 0, rep)
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["authority"], "installer")
def test_installer_entry_checks_the_sha(self):
data, sha = real_bundle()
box = Box(data, None)
box.put("/root/bundle.json", data, 0o600)
rc, rep = run(box, ["felhom-os-apply", "--install-bundle", "/root/bundle.json", "--sha256", "1" * 64])
self.assertEqual((rc, rep["refused"]["code"]), (2, "R18"))
def test_installer_entry_is_refused_through_sudo(self):
data, sha = real_bundle()
box = Box(data, None)
box.put("/root/bundle.json", data, 0o600)
rc, rep = run(box, ["felhom-os-apply", "--install-bundle", "/root/bundle.json", "--sha256", sha], {"SUDO_UID": "999"})
self.assertEqual((rc, rep["refused"]["code"]), (2, "R1"))
self.assertNotIn(osapply.BUNDLE_RECORD, box.files)
def test_self_check_answers(self):
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
rc = osapply.main(["felhom-os-apply", "--self-check"], runner=Box(b"", None))
self.assertEqual(rc, 0)
self.assertIn("bundle-format=1", buf.getvalue())
class Facts(unittest.TestCase):
def test_state_reports_drift_against_the_record(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
run(box)
box.put("/usr/local/sbin/felhom-pbs-apply", b"#!/bin/bash\necho by hand\n", 0o755)
a = osapply.Apply(box, PLAN)
st = osapply.Bundle(a).state()
self.assertEqual(st["version"], "0.143.0")
self.assertEqual(st["drift"], ["/usr/local/sbin/felhom-pbs-apply"])
self.assertTrue(st["signers_present"])
def test_state_without_a_record_says_none(self):
box = Box(b"", None)
st = osapply.Bundle(osapply.Apply(box, PLAN)).state()
self.assertEqual(st["version"], "none")
self.assertNotIn("drift", st)
self.assertEqual(st["live"]["/etc/sudoers.d/felhom-agent"], "absent")
class Builder(unittest.TestCase):
def test_reproducible(self):
self.assertEqual(builder.build("0.143.0"), builder.build("0.143.0"))
def test_every_source_exists_and_every_dest_is_unique(self):
dests = [e[0] for e in osapply.BUNDLE_FILES]
self.assertEqual(len(dests), len(set(dests)))
for _, src, *_ in osapply.BUNDLE_FILES:
self.assertTrue((REPO / "configs" / src).is_file(), src)
def test_every_root_file_the_installer_writes_is_in_the_bundle(self):
"""One source of truth: a felhom root-owned path the installer names must be a bundle path, a trust file, or a
path the AGENT itself writes at run time (named here, with why). Scope is a regex over the WHOLE installer."""
text = INSTALLER.read_text()
found = set(re.findall(r"(/usr/local/sbin/felhom-[a-z-]+|/etc/systemd/system/felhom-[a-z.-]+|"
r"/etc/sudoers\.d/felhom-[a-z-]+|/etc/felhom-oob\.nft|/etc/tmpfiles\.d/felhom-[a-z.-]+|"
r"/etc/felhom/[a-z.-]+)", text))
agent_writes = {"/usr/local/sbin/felhom-shared-parent", "/etc/systemd/system/felhom-shared-parent.service"}
trust = {osapply.TRUST_FILE, osapply.TRUST_SIGNERS, osapply.TRUST_SIGNERS + ".tmp", osapply.BUNDLE_RECORD}
# Written by the appliance ISO's first boot (felhom.eu scripts/iso/felhom-bootstrap.sh), never by the installer:
# since installer 1.32.0 (R-275) the uninstall only NAMES them under KEPT.
iso_writes = {"/etc/felhom/.bootstrap-done", "/etc/felhom/appliance-pairing-code"}
missing = sorted(p for p in found if p not in osapply.BUNDLE_DESTS and p not in agent_writes | trust | iso_writes)
self.assertEqual(missing, [], "the installer writes these root files, but the bundle does not carry them")
# the limits drop-in is named through $AGENT_UNIT in the installer
self.assertIn("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", osapply.BUNDLE_DESTS)
# ---------- R-861 (agent v0.146.0): the agent binary only by an operator-signed agent_update, checked as root ----------
STAGED = "/var/lib/felhom-agent/selfupdate/felhom-agent-0.146.0"
NEW_BIN = b"\x7fELF the new agent"
def update_job(sha, version="0.146.0", op="agent_update", nonce="u1", host=HOST):
blob = json.dumps({"expires_at": "2026-10-04T12:30:00Z", "issued_at": "2026-10-04T11:50:00Z", "key_id": "felhom-op-1",
"nonce": nonce, "op": op, "params": {"sha256": sha, "version": version},
"target": {"guest_id": "", "host_id": host}}, sort_keys=True).encode()
return {"blob_b64": base64.b64encode(blob).decode(), "sig": "-----BEGIN SSH SIGNATURE-----\nx\n-----END SSH SIGNATURE-----\n"}
def update_box(job, staged=STAGED, content=NEW_BIN):
box = Box(b"{}", None)
box.put(PLAN, json.dumps({"release_id": "agent-0.146.0", "layer": "host", "mode": "agent_update",
"signed": job, "staged": staged}).encode(), 0o600, uid=999)
box.put(STAGED, content, 0o755, uid=999)
return box
def wrapper_calls(box):
return [c for c in box.calls if c and c[0] == osapply.SELFUPDATE_WRAPPER and c[1:2] == ["apply"]]
class AgentUpdate(unittest.TestCase):
"""RED-PROOF: make agent_update skip verify_signed → test_a_bad_signature_never_reaches_the_wrapper fails."""
def test_signed_update_flips_and_burns_the_nonce(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
root_copy = osapply.SELFUPDATE_ROOT_DIR + "/felhom-agent-0.146.0"
# the wrapper gets the ROOT-OWNED copy of the bytes that were hashed — never the agent's path (review 2026-10-05)
self.assertEqual(wrapper_calls(box), [[osapply.SELFUPDATE_WRAPPER, "apply", root_copy, sha]])
self.assertIn(root_copy, box.writes)
self.assertNotIn(root_copy, box.files, "the root copy is removed after the flip")
self.assertEqual(box.staged_reads, 1, "the agent's file is read exactly once")
self.assertIn("u1", box.nonces)
self.assertEqual(rep["agent_update"]["version"], "0.146.0")
def test_a_staged_file_the_agent_does_not_own_is_refused(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha))
box.uids[STAGED] = 0
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R19"), rep)
self.assertEqual(wrapper_calls(box), [])
def test_a_bad_signature_never_reaches_the_wrapper(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha))
box.sig_rc = 1
rc, rep = run(box)
self.assertTrue(rep.get("refused"), f"a job whose signature does not verify was NOT refused: {rep}")
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"), rep)
self.assertEqual(wrapper_calls(box), [])
def test_a_staged_binary_the_signature_does_not_pin_is_refused(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha), content=b"\x7fELF something the agent put there")
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R19"), rep)
self.assertEqual(wrapper_calls(box), [])
self.assertNotIn("u1", box.nonces, "a refused job keeps its nonce (the operator fixes the file, not the key)")
def test_another_staging_path_is_refused(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha), staged="/tmp/felhom-agent-0.146.0")
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R19"), rep)
self.assertEqual(wrapper_calls(box), [])
def test_a_bundle_job_is_not_an_agent_update(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha, op="agent_config_update"))
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"), rep)
def test_another_hosts_job_and_a_replay_are_refused(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha, host="tester-1-d70be4"))
self.assertEqual(run(box)[1]["refused"]["code"], "R3")
box2 = update_box(update_job(sha))
box2.nonces["u1"] = 1999999999
self.assertEqual(run(box2)[1]["refused"]["code"], "R3")
self.assertEqual(wrapper_calls(box) + wrapper_calls(box2), [])
def test_a_failed_flip_keeps_the_nonce(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha))
orig = box.host
box.host = lambda argv, timeout=600, stdin=None: (1, "", "refusing apply: sha mismatch") if argv[:1] == [osapply.SELFUPDATE_WRAPPER] else orig(argv, timeout, stdin)
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assertNotIn("u1", box.nonces)
class SelfupdateWrapperConfinement(unittest.TestCase):
"""R-861 (v0.146.1): the A/B wrapper takes only felhom-os-apply's root-owned copy, never the agent's staging dir
(a file there can be swapped between the wrapper's sha check and its copy). The path check runs before anything
is touched, so the real script can be run here unprivileged.
RED-PROOF: point ROOT_STAGING back at /var/lib/felhom-agent/selfupdate → this fails (the path is accepted and the
script goes on to `staged file missing`)."""
def test_the_agents_staging_dir_is_refused(self):
import subprocess
sha = "0" * 64
p = subprocess.run(["sh", str(HERE / "felhom-selfupdate-guarded"), "apply",
"/var/lib/felhom-agent/selfupdate/felhom-agent-0.146.1", sha], capture_output=True, text=True)
self.assertEqual(p.returncode, 1, p.stderr)
self.assertIn("outside /var/lib/felhom-os-apply/agent-update", p.stderr)
_sb = importlib.machinery.SourceFileLoader("stepbuild", str(REPO / "scripts" / "build-step-bundle.py"))
_ss = importlib.util.spec_from_loader("stepbuild", _sb)
stepbuild = importlib.util.module_from_spec(_ss)
_sb.exec_module(stepbuild)
NEW_IN_0146 = {"/usr/local/sbin/felhom-priv-apply", "/var/lib/vz/snippets/felhom-guest-hook.sh",
"/usr/local/sbin/felhom-shared-parent.sh", "/etc/systemd/system/felhom-shared-parent.service"}
class StepBundle(unittest.TestCase):
"""R-880 (agent v0.146.1): an INSTALLED wrapper checks an incoming bundle's paths against its OWN table (R16), so a
release that adds paths needs a step bundle: the boxes' current bundle with only felhom-os-apply replaced.
RED-PROOF: deliver the full bundle to the old table → R16 (test_the_full_bundle_is_refused_by_an_old_table)."""
def old_world(self):
"""The base bundle an older wrapper (no R-861 paths) installed, and that wrapper's table."""
full = json.loads(builder.build("0.145.0"))
full["files"] = [e for e in full["files"] if e["path"] not in NEW_IN_0146]
old_wrapper = b'# the v0.145.0 wrapper stands in here\nBUNDLE_OP = "agent_config_update"\n'
for e in full["files"]:
if e["path"] == "/usr/local/sbin/felhom-os-apply":
e["content_b64"], e["sha256"] = base64.b64encode(old_wrapper).decode(), hashlib.sha256(old_wrapper).hexdigest()
base = (json.dumps(full, indent=1, sort_keys=True) + "\n").encode()
old_dests = {k: v for k, v in osapply.BUNDLE_DESTS.items() if k not in NEW_IN_0146}
return base, old_dests
def parse_with_table(self, data, dests, version):
saved = osapply.BUNDLE_DESTS
osapply.BUNDLE_DESTS = dests
try:
return osapply.Bundle(osapply.Apply(Box(b"{}", None), "")).parse(data, hashlib.sha256(data).hexdigest(), version)
finally:
osapply.BUNDLE_DESTS = saved
def test_the_full_bundle_is_refused_by_an_old_table(self):
_, old_dests = self.old_world()
full = builder.build("0.146.1")
with self.assertRaises(osapply.Refused) as cm:
self.parse_with_table(full, old_dests, "0.146.1")
self.assertEqual(cm.exception.code, "R16")
def test_the_step_bundle_is_accepted_by_the_old_table_and_changes_only_the_wrapper(self):
base, old_dests = self.old_world()
new_wrapper = (HERE / "felhom-os-apply").read_bytes()
step = stepbuild.build_step(base, "0.146.1-step1", new_wrapper)
ver, files = self.parse_with_table(step, old_dests, "0.146.1-step1")
self.assertEqual(ver, "0.146.1-step1")
b, s_ = json.loads(base), json.loads(step)
self.assertEqual(sorted(e["path"] for e in b["files"]), sorted(e["path"] for e in s_["files"]), "the paths must not change")
changed = [e["path"] for e, f in zip(sorted(b["files"], key=lambda x: x["path"]), sorted(s_["files"], key=lambda x: x["path"]))
if e != f]
self.assertEqual(changed, ["/usr/local/sbin/felhom-os-apply"], "exactly the wrapper changes")
installed = dict((d, c) for d, c, *_ in files)
self.assertEqual(installed["/usr/local/sbin/felhom-os-apply"], new_wrapper)
# and the NEW wrapper (now installed) knows every path the release's full bundle names
self.assertTrue({e["path"] for e in json.loads(builder.build("0.146.1"))["files"]} <= set(osapply.BUNDLE_DESTS))
def test_a_step_version_must_carry_a_suffix(self):
base, _ = self.old_world()
with self.assertRaises(SystemExit):
stepbuild.build_step(base, "0.146.1", b"x")
if __name__ == "__main__":
unittest.main()
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/python3
"""Tests for felhom-crash-guard (`11` §5.9). Temp dirs only; nothing real is touched. Red-proof seam: CRASHGUARD_UNDER_TEST."""
import importlib.machinery
import importlib.util
import json
import os
import pathlib
import tempfile
import unittest
HERE = pathlib.Path(__file__).resolve().parent
_loader = importlib.machinery.SourceFileLoader("crashguard", os.environ.get("CRASHGUARD_UNDER_TEST", str(HERE / "felhom-crash-guard")))
_spec = importlib.util.spec_from_loader("crashguard", _loader)
cg = importlib.util.module_from_spec(_spec)
_loader.exec_module(cg)
T0 = 1791115200.0 # 2026-10-04T12:00:00Z
class FakeEnv(cg.Env):
def __init__(self, d):
super().__init__(conf=os.path.join(d, "conf"), state_dir=os.path.join(d, "state"),
panic_path=os.path.join(d, "panic"), uptime_path=os.path.join(d, "uptime"),
boot_id_path=os.path.join(d, "bootid"))
self.t = T0
self.logs = []
open(self.panic_path, "w").write("0\n")
open(self.uptime_path, "w").write("20.00 10.00\n")
def now(self):
return self.t
def log(self, line):
self.logs.append(line)
def panic(self):
return int(open(self.panic_path).read())
def state(self):
return json.load(open(os.path.join(self.state_dir, "state.json")))
class Guard(unittest.TestCase):
def setUp(self):
self.d = tempfile.TemporaryDirectory()
self.e = FakeEnv(self.d.name)
def tearDown(self):
self.d.cleanup()
def crash_boot(self, minutes_later):
self.e.t += minutes_later * 60
cg.main(["x", "boot"], self.e) # no clean-stop before it: an unclean stop
def clean_reboot(self, minutes_later):
cg.main(["x", "clean-stop"], self.e)
self.e.t += minutes_later * 60
cg.main(["x", "boot"], self.e)
def test_first_boot_is_not_a_crash_and_arms(self):
cg.main(["x", "boot"], self.e)
s = self.e.state()
self.assertFalse(s["last_boot_unclean"])
self.assertEqual(self.e.panic(), 10)
self.assertTrue(s["armed"])
def test_clean_reboots_never_count(self):
cg.main(["x", "boot"], self.e)
for _ in range(5):
self.clean_reboot(1)
s = self.e.state()
self.assertEqual(s["unclean_boots_in_window"], 0)
self.assertEqual(self.e.panic(), 10)
def test_third_crash_in_an_hour_leaves_the_box_off(self):
# operator's words: "if it crashes 3 times within one hour, it stays off" — after crash 2 the guard trips,
# so crash 3 (kernel.panic = 0) does not restart the box.
cg.main(["x", "boot"], self.e)
self.crash_boot(5)
self.assertEqual(self.e.panic(), 10, "one crash: still restarts")
self.crash_boot(5)
s = self.e.state()
self.assertTrue(s["tripped"], s)
self.assertEqual(self.e.panic(), 0, "after the 2nd crash boot the 3rd crash must leave the box off")
self.assertIn("2 unclean boots within 60 minutes", s["tripped_reason"])
def test_crashes_spread_over_more_than_the_window_do_not_trip(self):
cg.main(["x", "boot"], self.e)
self.crash_boot(5)
self.crash_boot(61)
self.assertFalse(self.e.state()["tripped"])
self.assertEqual(self.e.panic(), 10)
def test_tripped_stays_tripped_across_boots(self):
cg.main(["x", "boot"], self.e)
self.crash_boot(5)
self.crash_boot(5)
self.clean_reboot(30) # the operator switched it on; even a clean boot keeps the trip
self.assertTrue(self.e.state()["tripped"])
self.assertEqual(self.e.panic(), 0)
def test_rearms_after_24h_of_normal_running(self):
cg.main(["x", "boot"], self.e)
self.crash_boot(5)
self.crash_boot(5)
self.e.t += 23 * 3600
cg.main(["x", "check"], self.e)
self.assertTrue(self.e.state()["tripped"], "not before 24 h")
self.e.t += 3600
cg.main(["x", "check"], self.e)
s = self.e.state()
self.assertFalse(s["tripped"])
self.assertEqual(self.e.panic(), 10)
self.assertIn("timer", s["rearmed_by"])
def test_operator_rearm_starts_a_fresh_window(self):
cg.main(["x", "boot"], self.e)
self.crash_boot(5)
self.crash_boot(5)
self.e.t += 60
cg.main(["x", "rearm"], self.e)
s = self.e.state()
self.assertFalse(s["tripped"])
self.assertEqual(s["rearmed_by"], "operator")
self.assertEqual(s["unclean_boots_24h"], 2, "the history stays")
self.crash_boot(5)
self.assertFalse(self.e.state()["tripped"], "one crash after a re-arm must not trip at once")
def test_config_numbers_are_read(self):
open(self.e.conf, "w").write("LIMIT=2\nPANIC_SECONDS=30\n")
cg.main(["x", "boot"], self.e)
self.assertEqual(self.e.panic(), 30)
self.crash_boot(1)
self.assertTrue(self.e.state()["tripped"], "LIMIT=2: the first crash boot trips")
def test_state_is_world_readable_for_the_agent(self):
cg.main(["x", "boot"], self.e)
mode = os.stat(os.path.join(self.e.state_dir, "state.json")).st_mode & 0o777
self.assertEqual(mode, 0o644)
if __name__ == "__main__":
unittest.main()
File diff suppressed because it is too large Load Diff
+320
View File
@@ -0,0 +1,320 @@
#!/usr/bin/env python3
"""Tests for felhom-priv-apply (R-861, `03` §3.1). An in-memory host plays the files; nothing real is written or run.
Each refusal rule has a test that feeds it the ATTACK it exists for; each accepted shape is a file the agent really
renders (the Go contract tests feed the live renderers too). Red-proof: audits/hub-safety-2026-10-05/partF/.
Run: python3 configs/test_felhom_priv_apply.py (also run by internal/privapply's Go test)
"""
import sys
sys.dont_write_bytecode = True
import importlib.machinery
import importlib.util
import os
import pathlib
import unittest
HERE = pathlib.Path(__file__).resolve().parent
_loader = importlib.machinery.SourceFileLoader("privapply", os.environ.get("PRIVAPPLY_UNDER_TEST", str(HERE / "felhom-priv-apply")))
_spec = importlib.util.spec_from_loader("privapply", _loader)
pa = importlib.util.module_from_spec(_spec)
_loader.exec_module(pa)
AGENT_UID = 999
class FakeHost:
def __init__(self):
self.src, self.dest, self.logs, self.writes = {}, {}, [], []
self.owner, self.kind = {}, {}
def stage(self, path, text, uid=AGENT_UID, kind="file"):
self.src[path] = text.encode() if isinstance(text, str) else text
self.owner[path], self.kind[path] = uid, kind
def agent_uid(self):
return AGENT_UID
def read_source(self, path):
# mirrors Host.read_source's refusals: absent, not a regular file (a symlink is refused by O_NOFOLLOW), owner, size
if path not in self.src:
raise pa.Refused("P1", f"cannot open the staged file {path}")
if self.kind[path] != "file":
raise pa.Refused("P1", f"{path} is not a regular file")
if self.owner[path] != AGENT_UID:
raise pa.Refused("P1", f"{path} is not owned by felhom-agent")
if len(self.src[path]) > pa.MAX_BYTES:
raise pa.Refused("P1", f"{path} is larger than {pa.MAX_BYTES} bytes")
return self.src[path]
def read_dest(self, path):
return self.dest.get(path)
def install(self, dest, data, mode):
self.writes.append((dest, mode))
self.dest[dest] = data
def log(self, line):
self.logs.append(line)
LOCAL_UNIT = """# Managed by felhom-agent — do not edit by hand.
[Unit]
Description=Felhom storage mount 91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
After=local-fs-pre.target
[Mount]
What=/dev/disk/by-uuid/91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
Where=/mnt/hdd_1
Type=ext4
[Install]
WantedBy=multi-user.target
"""
NET_MOUNT = """# felhom network storage — do not edit by hand.
[Unit]
Description=Felhom network storage media (nfs)
[Mount]
What=nas.lan:/volume1/media
Where=/mnt/felhom-drives/media
Type=nfs4
Options=vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev,retry=0,nosuid,nodev
"""
NET_AUTOMOUNT = """# felhom network storage — do not edit by hand.
[Unit]
Description=Felhom network storage automount media (nfs)
[Automount]
Where=/mnt/felhom-drives/media
TimeoutIdleSec=600
[Install]
WantedBy=multi-user.target
"""
NET_NAME = "mnt-felhom\\x2ddrives-media.mount"
DNS_BASE = """# felhom split-horizon resolver — host base config (agent-managed; DO NOT EDIT)
bind-interfaces
listen-address=192.168.0.104
listen-address=127.0.0.1
no-resolv
server=1.1.1.1
server=9.9.9.9
"""
DNS_GUEST = """# felhom split-horizon DNS — customer demo-hp (agent-managed; DO NOT EDIT)
local=/enkisfelhom.hu/
address=/enkisfelhom.hu/192.168.0.138
"""
K = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=" # base64 of 32 bytes
WG = f"""# felhom offsite tunnel — agent-managed (S3); DO NOT EDIT
[Interface]
PrivateKey = {K}
Address = 10.77.0.3/32
MTU = 1280
[Peer]
PublicKey = {K}
Endpoint = 49.12.1.2:51820
AllowedIPs = 10.77.0.1/32, 10.77.0.250/32
PersistentKeepalive = 25
"""
KEYLINE = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL8z0qCNgA3x2xxAB0Qj5ro8waFjGZ8Ta/sWB63tlLw+ felhom-op-1\n"
def run(host, *argv):
return pa.main(list(argv), host=host)
class Accepts(unittest.TestCase):
"""What the agent really writes is accepted and installed, root-owned, at the fixed destination."""
def test_local_unit_installed(self):
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/mnt-hdd_1.mount", LOCAL_UNIT)
self.assertEqual(run(h, "unit", "mnt-hdd_1.mount"), 0)
self.assertEqual(h.writes, [("/etc/systemd/system/mnt-hdd_1.mount", 0o644)])
def test_local_unit_without_type(self): # the N100's unit has no Type= (autodetect)
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/mnt-hdd_1.mount", LOCAL_UNIT.replace("Type=ext4\n", ""))
self.assertEqual(run(h, "unit", "mnt-hdd_1.mount"), 0)
def test_network_pair(self):
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/" + NET_NAME, NET_MOUNT)
h.stage("/var/lib/felhom-agent/units/mnt-felhom\\x2ddrives-media.automount", NET_AUTOMOUNT)
self.assertEqual(run(h, "unit", NET_NAME), 0)
self.assertEqual(run(h, "unit", "mnt-felhom\\x2ddrives-media.automount"), 0)
def test_identical_is_not_rewritten(self):
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/mnt-hdd_1.mount", LOCAL_UNIT)
h.dest["/etc/systemd/system/mnt-hdd_1.mount"] = LOCAL_UNIT.encode()
self.assertEqual(run(h, "unit", "mnt-hdd_1.mount"), 0)
self.assertEqual(h.writes, [])
def test_dnsmasq_both_dropins(self):
h = FakeHost()
h.stage("/tmp/felhom-resolver-123.conf", DNS_BASE)
self.assertEqual(run(h, "dnsmasq", "/tmp/felhom-resolver-123.conf", "felhom-resolver-base.conf"), 0)
h.stage("/tmp/felhom-resolver-124.conf", DNS_GUEST)
self.assertEqual(run(h, "dnsmasq", "/tmp/felhom-resolver-124.conf", "felhom-demo-hp.conf"), 0)
self.assertIn(("/etc/dnsmasq.d/felhom-demo-hp.conf", 0o644), h.writes)
def test_wg_installed_0600(self):
h = FakeHost()
h.stage(pa.WG_SRC, WG)
self.assertEqual(run(h, "wg"), 0)
self.assertEqual(h.writes, [(pa.WG_DEST, 0o600)])
def test_sshd_template_and_key(self):
h = FakeHost()
h.stage(pa.SSHD_SRC, pa.render_sshd(8822))
h.stage(pa.KEY_SRC, KEYLINE)
self.assertEqual(run(h, "sshd-config"), 0)
self.assertEqual(run(h, "sshd-key"), 0)
h.stage(pa.KEY_SRC, "") # clearing the operator login is allowed
self.assertEqual(run(h, "sshd-key"), 0)
def test_escape_matches_systemd(self):
self.assertEqual(pa.systemd_escape_path("/mnt/felhom-drives/media"), "mnt-felhom\\x2ddrives-media")
self.assertEqual(pa.systemd_escape_path("/mnt/hdd_1"), "mnt-hdd_1")
self.assertEqual(pa.systemd_escape_path("/mnt/.x"), "mnt-.x") # a dot is escaped only at the very start
class Refuses(unittest.TestCase):
"""Each rule, with the attack it exists for. Nothing is written on a refusal."""
def refused(self, h, argv, rule):
rc = run(h, *argv)
self.assertIn(rc, (2, 3), f"{argv} was accepted")
self.assertEqual(h.writes, [], f"{argv} wrote something")
self.assertTrue(any(f"[{rule}]" in l for l in h.logs), f"{argv}: rule {rule} not logged: {h.logs}")
def unit(self, text, name="mnt-hdd_1.mount"):
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/" + name, text)
return h, ["unit", name]
def test_U1_name_outside_mnt(self):
h = FakeHost()
self.refused(h, ["unit", "etc-sudoers.d.mount"], "U1")
self.refused(h, ["unit", "../../etc/x.mount"], "U1")
self.refused(h, ["unit", "mnt-x.service"], "U1")
def test_U2_service_section(self):
self.refused(*self.unit(LOCAL_UNIT + "\n[Service]\nExecStart=/bin/sh -c id\n"), "U2")
def test_U2_wants_starts_another_unit(self): # review 2026-10-05: Wants=reboot.target would reboot the host
for extra in ("Wants=reboot.target", "Requires=felhom-agent-rollback.service", "Before=pve-guests.service"):
self.refused(*self.unit(LOCAL_UNIT.replace("After=local-fs-pre.target", "After=local-fs-pre.target\n" + extra)), "U2")
self.refused(*self.unit(LOCAL_UNIT.replace("After=local-fs-pre.target", "After=poweroff.target")), "U2")
def test_U2_continuation_line(self):
t = LOCAL_UNIT.replace("Description=Felhom storage mount 91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae",
"Description=Felhom storage mount \\")
self.refused(*self.unit(t), "U2")
self.refused(*self.unit(LOCAL_UNIT.replace("# Managed by felhom-agent", "# comment \\\n# Managed by felhom-agent")), "U2")
def test_U2_unknown_key(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Type=ext4", "Type=ext4\nDirectoryMode=0777")), "U2")
def test_U3_bind_over_sudoers_dir(self):
# the R-861 attack: mount an agent-owned directory over /etc/sudoers.d
t = LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/etc/sudoers.d")
self.refused(*self.unit(t, "mnt-hdd_1.mount"), "U3")
def test_U3_name_must_match_where(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/mnt/other")), "U3")
def test_U3_traversal_in_where(self):
# the name passes U1 and equals the escaped Where — ONLY the Where rule stops a mount at /mnt/../etc = /etc
self.assertEqual(pa.systemd_escape_path("/mnt/../etc") + ".mount", "mnt-..-etc.mount")
self.refused(*self.unit(LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/mnt/../etc"), "mnt-..-etc.mount"), "U3")
def test_U4_what_is_an_agent_directory(self):
t = LOCAL_UNIT.replace("What=/dev/disk/by-uuid/91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae", "What=/var/lib/felhom-agent/evil")
self.refused(*self.unit(t), "U4")
def test_U4_tmpfs(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Type=ext4", "Type=tmpfs")), "U4")
def test_U5_bind_option(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Type=ext4", "Type=ext4\nOptions=bind")), "U5")
def test_U5_network_without_nosuid(self):
t = NET_MOUNT.replace(",nosuid,nodev", "")
self.refused(*self.unit(t, NET_NAME), "U5")
def test_U5_suid_option(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Type=ext4", "Type=ext4\nOptions=suid,dev")), "U5")
def test_U3_network_outside_drives(self):
t = NET_MOUNT.replace("/mnt/felhom-drives/media", "/mnt/media")
self.refused(*self.unit(t, "mnt-media.mount"), "U3")
def test_D1_dhcp_script(self): # runs as root
h = FakeHost()
h.stage("/tmp/felhom-resolver-1.conf", DNS_BASE + "dhcp-script=/var/lib/felhom-agent/x.sh\n")
self.refused(h, ["dnsmasq", "/tmp/felhom-resolver-1.conf", "felhom-x.conf"], "D1")
def test_D1_conf_dir_and_log_file(self):
for extra in ("conf-dir=/var/lib/felhom-agent\n", "log-facility=/etc/sudoers.d/x\n", "user=root\n"):
h = FakeHost()
h.stage("/tmp/felhom-resolver-1.conf", DNS_GUEST + extra)
self.refused(h, ["dnsmasq", "/tmp/felhom-resolver-1.conf", "felhom-x.conf"], "D1")
def test_D2_paths(self):
h = FakeHost()
self.refused(h, ["dnsmasq", "/etc/shadow", "felhom-x.conf"], "D2")
self.refused(h, ["dnsmasq", "/tmp/felhom-resolver-1.conf", "../sudoers.d/x.conf"], "D2")
def test_W1_postup(self): # wg-quick runs PostUp as root
h = FakeHost()
h.stage(pa.WG_SRC, WG.replace("MTU = 1280", "MTU = 1280\nPostUp = /bin/sh -c id"))
self.refused(h, ["wg"], "W1")
def test_W2_values(self):
h = FakeHost()
h.stage(pa.WG_SRC, WG.replace("AllowedIPs = 10.77.0.1/32, 10.77.0.250/32", "AllowedIPs = 0.0.0.0/0"))
self.refused(h, ["wg"], "W2")
def test_S1_sshd_strictmodes(self): # an AuthorizedKeysFile the agent owns + StrictModes no = root login
h = FakeHost()
h.stage(pa.SSHD_SRC, pa.render_sshd(8822) + "StrictModes no\n")
self.refused(h, ["sshd-config"], "S1")
h2 = FakeHost()
h2.stage(pa.SSHD_SRC, pa.render_sshd(8822).replace("/etc/felhom-sshd/authorized_keys/%u", "/var/lib/felhom-agent/k"))
self.refused(h2, ["sshd-config"], "S1")
def test_S1_port_22(self):
h = FakeHost()
h.stage(pa.SSHD_SRC, pa.render_sshd(22))
self.refused(h, ["sshd-config"], "S1")
def test_S2_key_options_and_two_keys(self):
h = FakeHost()
h.stage(pa.KEY_SRC, 'command="/bin/sh" ' + KEYLINE)
self.refused(h, ["sshd-key"], "S2")
h2 = FakeHost()
h2.stage(pa.KEY_SRC, KEYLINE + KEYLINE)
self.refused(h2, ["sshd-key"], "S2")
def test_P1_symlink_owner_size(self):
h = FakeHost()
h.stage(pa.WG_SRC, WG, kind="symlink")
self.refused(h, ["wg"], "P1")
h2 = FakeHost()
h2.stage(pa.WG_SRC, WG, uid=0)
self.refused(h2, ["wg"], "P1")
h3 = FakeHost()
h3.stage(pa.WG_SRC, "#" * (pa.MAX_BYTES + 1))
self.refused(h3, ["wg"], "P1")
def test_A1_usage(self):
h = FakeHost()
self.refused(h, ["install", "/etc/shadow"], "A1")
self.refused(h, ["wg", "/etc/shadow"], "A1")
if __name__ == "__main__":
unittest.main(verbosity=2)
@@ -241,3 +241,23 @@ func TestNewestArchiveTime_DistinctPhantomsEachAnnounced(t *testing.T) {
t.Errorf("got %d rejection lines for 2 distinct phantoms across 3 polls, want 2:\n%s", n, buf.String())
}
}
// R-99 (`09` §3 decision 140): the WARN for a PBS phantom ends with the cleanup runbook, so whoever sees it knows the
// one sanctioned way to remove it; a tiny archive on a dir storage is not a PBS phantom and gets no pointer.
// RED-PROOF: drop the `msg += phantomCleanupPointer` line → "the PBS phantom WARN does not end with the runbook pointer".
func TestRejectedArchiveWarnNamesTheCleanupRunbook(t *testing.T) {
var buf bytes.Buffer
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
t.Fatal(err)
}
const want = "INCOMPLETE archive when computing tier freshness — it is not a successful backup — a phantom leftover; delete it by felhom.eu documentation/runbooks/pbs-phantom-cleanup.md (09 §3 decision 140)"
if !strings.Contains(buf.String(), want) {
t.Errorf("the PBS phantom WARN does not end with the runbook pointer:\n%s", buf.String())
}
local := phantomEntry()
local.Format, local.VolID = "tar.zst", "local:backup/vzdump-lxc-9201-2026_07_28-05_31_14.tar.zst"
if got := rejectedArchiveMessage(local); strings.Contains(got, "pbs-phantom-cleanup") {
t.Errorf("a dir-storage archive got the PBS runbook pointer: %s", got)
}
}
+131
View File
@@ -0,0 +1,131 @@
package backup
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"strconv"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// BackupSuccessState persists the newest SUCCESSFUL whole-guest backup per tier and guest (R-894).
//
// Why it exists. The due-check (`localapi` handleBackupDue) asks the tier's storage when a backup last
// landed (R-84) and falls back to the in-memory record when the storage cannot be read. The in-memory
// record is empty after an agent restart (Store, R-348), so "storage unreadable" right after a restart
// read as "no record — DUE". Measured 2026-10-05 on demo-hp: the agent restarted at 04:57, the off-site
// storage answered "Can't connect" at 06:25, the 7-day tier — last copy 2026-10-01 — read DUE, the
// controller asked, and vzdump failed. This file is the last known copy the fallback reads instead.
//
// It is read ONLY when the storage cannot be read. A storage that answers is the ground truth and wins,
// in both directions: an archive found there counts, and an archive absent there is absent even when
// this file remembers a success (a pruned or deleted archive must make the tier due — the same reason
// R-84 chose the storage over a persisted record). Pinned by
// TestBackupDue_R894_SavedCopyIgnoredWhenStorageAnswers.
//
// Only SUCCESSES are written (the RestoreTestState rule): a failure must stay due and be retried, so a
// record of a failure has no reader.
type BackupSuccessState struct {
path string
mu sync.Mutex
last map[string]savedSuccess // key(target, vmid) → the newest success
}
type savedSuccess struct {
target string
vmid int
at time.Time
}
// backupSuccessJSON is one entry on disk.
type backupSuccessJSON struct {
Target string `json:"target"`
VMID int `json:"vmid"`
StartedAt string `json:"started_at"`
}
func backupStateKey(target string, vmid int) string { return target + "/" + strconv.Itoa(vmid) }
// NewBackupSuccessState opens (or creates) the state at path. A missing or unreadable file degrades to
// "nothing known" — the pre-R-894 behaviour, which is DUE — and never wedges the daemon.
func NewBackupSuccessState(path string) *BackupSuccessState {
s := &BackupSuccessState{path: path, last: map[string]savedSuccess{}}
data, err := os.ReadFile(path)
if err != nil {
return s
}
var entries []backupSuccessJSON
if json.Unmarshal(data, &entries) != nil {
return s
}
for _, e := range entries {
t, perr := time.Parse(time.RFC3339, e.StartedAt)
if perr != nil {
continue // one unreadable entry must not lose the others
}
s.last[backupStateKey(e.Target, e.VMID)] = savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()}
}
return s
}
// RecordBackupSuccess saves b when it is a success newer than the one on file. target is the tier the
// job ran on (the due-check's key); a failure or an unparseable time is ignored.
func (s *BackupSuccessState) RecordBackupSuccess(target string, b hub.Backup) error {
if s == nil || !b.Success {
return nil
}
t, err := time.Parse(time.RFC3339, b.StartedAt)
if err != nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
k := backupStateKey(target, b.VMID)
if old, ok := s.last[k]; ok && !t.After(old.at) {
return nil
}
s.last[k] = savedSuccess{target: target, vmid: b.VMID, at: t.UTC()}
return s.saveLocked()
}
// LastKnownSuccess returns the newest saved success for this tier and guest (ok=false = none on file).
func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Time, bool) {
if s == nil {
return time.Time{}, false
}
s.mu.Lock()
defer s.mu.Unlock()
e, ok := s.last[backupStateKey(target, vmid)]
return e.at, ok
}
func (s *BackupSuccessState) saveLocked() error {
entries := make([]backupSuccessJSON, 0, len(s.last))
for _, e := range s.last {
entries = append(entries, backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)})
}
// Deterministic file content (Go's map order is random).
sort.Slice(entries, func(i, j int) bool {
if entries[i].Target != entries[j].Target {
return entries[i].Target < entries[j].Target
}
return entries[i].VMID < entries[j].VMID
})
data, err := json.MarshalIndent(entries, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
return err
}
tmp := s.path + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, s.path)
}
+59
View File
@@ -0,0 +1,59 @@
package backup
import (
"os"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-894: the on-disk newest success per tier survives a restart (a new state from the same file).
func TestBackupSuccessState_SurvivesRestart(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
at := time.Date(2026, 10, 1, 20, 15, 0, 0, time.UTC)
if err := s.RecordBackupSuccess("felhom-pbs", hub.Backup{VMID: 9201, Success: true, StartedAt: at.Format(time.RFC3339)}); err != nil {
t.Fatal(err)
}
got, ok := NewBackupSuccessState(path).LastKnownSuccess("felhom-pbs", 9201)
if !ok || !got.Equal(at) {
t.Fatalf("after a restart the saved copy must read back; got %v ok=%v", got, ok)
}
if _, ok := NewBackupSuccessState(path).LastKnownSuccess("local", 9201); ok {
t.Fatal("another tier must not borrow this tier's copy")
}
}
// Only a NEWER success replaces the saved one; failures and unparseable times are ignored.
func TestBackupSuccessState_KeepsNewestSuccessOnly(t *testing.T) {
path := filepath.Join(t.TempDir(), "s.json")
s := NewBackupSuccessState(path)
newer := time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
older := newer.Add(-48 * time.Hour)
for _, b := range []hub.Backup{
{VMID: 1, Success: true, StartedAt: newer.Format(time.RFC3339)},
{VMID: 1, Success: true, StartedAt: older.Format(time.RFC3339)}, // older: ignored
{VMID: 1, Success: false, StartedAt: newer.Add(time.Hour).Format(time.RFC3339)}, // failure: ignored
{VMID: 1, Success: true, StartedAt: "not-a-time"}, // unparseable: ignored
} {
if err := s.RecordBackupSuccess("t", b); err != nil {
t.Fatal(err)
}
}
if got, _ := NewBackupSuccessState(path).LastKnownSuccess("t", 1); !got.Equal(newer) {
t.Fatalf("want the newest success %v, got %v", newer, got)
}
}
// A corrupt file degrades to "nothing known" (the pre-R-894 DUE answer), never a crash.
func TestBackupSuccessState_CorruptFileIsEmpty(t *testing.T) {
path := filepath.Join(t.TempDir(), "s.json")
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
t.Fatal(err)
}
if _, ok := NewBackupSuccessState(path).LastKnownSuccess("t", 1); ok {
t.Fatal("a corrupt file must read as nothing known")
}
}
+17 -7
View File
@@ -2,6 +2,7 @@ package backup
import (
"context"
"fmt"
"encoding/json"
"errors"
"io"
@@ -21,6 +22,8 @@ type fakeBackupAPI struct {
vzdumpErr error
waitErr error
cfg proxmox.GuestConfig
goneGuests map[int]bool // R-689: vmids whose config lookup answers "does not exist"
aclGuests map[int]bool // R-689: vmids outside the token's ACL — PVE answers 403 "permission denied"
cfgErr error
content []proxmox.StorageContent
contentErr error
@@ -43,7 +46,13 @@ func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOpti
}
return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
}
func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
func (f *fakeBackupAPI) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
if f.aclGuests[vmid] {
return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 403: permission denied at /vms/%d (missing privilege VM.Audit)", vmid, vmid)
}
if f.goneGuests[vmid] { // R-689: PVE's answer for a deleted guest
return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 500: Configuration file 'nodes/n/lxc/%d.conf' does not exist", vmid, vmid)
}
return f.cfg, f.cfgErr
}
func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
@@ -150,13 +159,14 @@ func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
const big = 4 << 30 // a plausible whole-guest archive
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "a", Content: "backup", CTime: 10, Size: big},
{VolID: "b", Content: "backup", CTime: 99, Size: big},
// R-689: real vzdump names with their vmid — only a backup OF A GUEST is a candidate.
{VolID: "local:backup/vzdump-lxc-9001-a.tar.zst", VMID: 9001, Content: "backup", CTime: 10, Size: big},
{VolID: "local:backup/vzdump-lxc-9001-b.tar.zst", VMID: 9001, Content: "backup", CTime: 99, Size: big},
{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
}}
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
if err != nil || vol != "b" {
if err != nil || vol != "local:backup/vzdump-lxc-9001-b.tar.zst" {
t.Fatalf("pick = %q,%v want newest 'b'", vol, err)
}
// no backups → "".
@@ -176,12 +186,12 @@ func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
// `pick = "phantom" want the newest COMPLETE archive 'real'`.
func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
{VolID: "felhom-pbs:backup/ct/9001/real", VMID: 9001, Content: "backup", CTime: 10, Size: 4 << 30},
{VolID: "felhom-pbs:backup/ct/9001/phantom", VMID: 9001, Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
}}
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
if err != nil || vol != "real" {
if err != nil || vol != "felhom-pbs:backup/ct/9001/real" {
t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
}
}
+112
View File
@@ -0,0 +1,112 @@
package backup
import (
"context"
"fmt"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-689 (v0.135.0) — demo-hp keeps its golden template in `local:backup/`. It is content "backup",
// 654 MB and so "plausibly complete", and it was the newest SETTLED entry: the restore test picked it
// every 6 h and failed extractconfig with a 403 (measured 2026-09-24 10:36, 09-25 04:57 and 10:57),
// while the guest's own archive — younger than the 24 h settle — went untested and nothing was proven.
//
// COMPANION RED-PROOF (REPORT.md): drop the guestBackupArchive call from PickSettledRestoreCandidateOn —
// this test then picks `local:backup/felhom-golden-0.236.0.tar.zst`.
func TestR689_TheRestoreTestNeverPicksTheGolden(t *testing.T) {
const day = int64(86400)
now := int64(1790370000) // 2026-09-25 ~19:00Z
api := &fakeBackupAPI{content: []proxmox.StorageContent{
// the guest's real archive, settled (older than the cutoff below)
{VolID: "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 3*day},
// the golden: newer, settled, big, and NOT a backup of a guest
{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 2*day},
// a hand-copied tarball that PVE happens to attribute to a vmid — the name is not a vzdump's
{VolID: "local:backup/copy-of-9201.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 2*day},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
if err != nil {
t.Fatal(err)
}
if got != "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst" {
t.Fatalf("picked %q — the restore test must prove a backup OF A GUEST", got)
}
}
func TestR689_GuestBackupArchiveShapes(t *testing.T) {
for _, c := range []struct {
e proxmox.StorageContent
ok bool
}{
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst", VMID: 9201}, true},
{proxmox.StorageContent{VolID: "local:backup/vzdump-qemu-300-2026_09_24-21_59_25.vma.zst", VMID: 300}, true},
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z", VMID: 9201}, true},
{proxmox.StorageContent{VolID: "felhom-pbs:backup/vm/300/2026-07-28T05:31:14Z", VMID: 300}, true},
{proxmox.StorageContent{VolID: "local:backup/felhom-golden-0.236.0.tar.zst"}, false},
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", VMID: 9202}, false}, // vmid disagrees with the name
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
} {
if ok, why := guestBackupArchive(c.e); ok != c.ok {
t.Errorf("%s vmid=%d: ok=%v (%s), want %v", c.e.VolID, c.e.VMID, ok, why, c.ok)
}
}
}
// R-689 (v0.136.0) — the measured demo-hp shape right after v0.135.0: the golden (skipped), a leftover archive
// of guest 9100 deleted in August (settled), and today's archive of 9201 (not settled yet). The pick must be
// NOTHING — never the deleted guest's archive. With 9201's archive settled, that one.
//
// COMPANION RED-PROOF (REPORT.md): drop the known-guest check — the pick is the 9100 leftover.
func TestR689_AnArchiveOfADeletedGuestIsNeverPicked(t *testing.T) {
const day = int64(86400)
now := int64(1790476000)
api := &fakeBackupAPI{goneGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 14*day},
{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
if err != nil || got != "" {
t.Fatalf("picked %q err=%v — a deleted guest's archive proves nothing about this box", got, err)
}
got, _, _ = r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now, 0).UTC())
if got != "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst" {
t.Fatalf("with 9201's archive settled the pick is %q", got)
}
}
// Any OTHER lookup failure is not "the guest is gone": the tier must read UNKNOWN (an error), never
// "nothing to prove".
func TestR689_AGuestLookupFailureIsUnknownNotEmpty(t *testing.T) {
api := &fakeBackupAPI{cfgErr: fmt.Errorf("proxmox: connection refused"), content: []proxmox.StorageContent{
{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: 10},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err == nil {
t.Fatal("a failed guest lookup read as a clean answer")
}
}
// v0.137.0 — THE MEASURED ANSWER: the agent's token sees only its pool, so for the deleted guest PVE says 403
// "permission denied at /vms/9100", not "does not exist" (demo-hp, right after v0.136.0 — the local tier read
// UNKNOWN). Such a guest is not one this agent manages: its archive is skipped, the tier is not an error.
//
// COMPANION RED-PROOF (REPORT.md): drop the "permission denied" case — the pick errors.
func TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest(t *testing.T) {
const day = int64(86400)
now := int64(1790476000)
api := &fakeBackupAPI{aclGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
if err != nil || got != "" {
t.Fatalf("picked %q err=%v — want nothing and no error (the only settled archive is not ours)", got, err)
}
}
+74
View File
@@ -0,0 +1,74 @@
package backup
import (
"context"
"errors"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
const (
thisBoxKey = "de:51:7a:18:cb:39:22:30:2c:84:f5:8b:d1:91:4b:7e:81:bb:69:b8:89:0f:57:ac:d3:59:e1:1a:62:25:11:2c"
earlierBox1 = "6b:ca:5f:3f:ca:0f:e2:3f:fb:24:62:89:bf:e7:64:59:9a:41:c5:e6:e3:9f:3f:5f:e1:71:7b:a1:9d:24:67:82"
earlierBox2 = "fe:3d:db:95:d4:df:ab:e1:7d:4a:89:fa:2b:07:53:6a:e4:d2:85:95:d1:90:27:4b:d9:c6:92:20:95:04:e5:d4"
)
// R-727 (v0.138.0) — the 2026-09-30 shape, measured on a fresh box for a returning customer: the PBS
// namespace held two archives of earlier boxes (same guest 9201, same token) and this box's own, which was not
// settled yet. The old picker chose the earlier box's newest settled archive and failed `wrong key`.
// The CONSEQUENCE asserted: no archive of another box is ever picked; with this box's archive settled it is picked.
// COMPANION RED-PROOF: remove the `ownKey != "" && !EqualFold(...)` skip → the first case picks 2026-09-16T21:59:54Z.
func TestR727_TheRestoreTestTakesOnlyThisBoxsArchives(t *testing.T) {
day := int64(86400)
now := int64(1790740000) // 2026-09-30 ~04:00Z
own := proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
own,
},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
// 1. The night of 2026-09-30: this box's own archive is ~6 h old, not settled (cutoff 24 h) — nothing to prove.
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now-day, 0).UTC())
if err != nil {
t.Fatal(err)
}
if got != "" {
t.Fatalf("picked %q — an archive of ANOTHER box is never this box's proof (R-727)", got)
}
// 2. A day later this box's own archive is settled — it is the one picked.
got, _, err = r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now+day, 0).UTC())
if err != nil {
t.Fatal(err)
}
if got != own.VolID {
t.Fatalf("picked %q, want this box's own %q", got, own.VolID)
}
}
// An unencrypted storage (a local dir) holds only this box's vzdumps — no key filter applies.
func TestR727_UnencryptedStorageIsNotFiltered(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_09_29-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1790710025}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err != nil || got == "" {
t.Fatalf("got %q err %v", got, err)
}
}
// A storage-list failure makes the tier UNKNOWN (an error), never "nothing to prove".
func TestR727_KeyLookupFailureIsUnknown(t *testing.T) {
api := &fakeBackupAPI{storageErr: errors.New("proxmox: GET /storage -> HTTP 500"), content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/x", Content: "backup", VMID: 9201}}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err == nil {
t.Fatal("a failed key lookup must surface as an error (tier UNKNOWN)")
}
}
+63
View File
@@ -0,0 +1,63 @@
package backup
import (
"context"
"fmt"
"sync/atomic"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// R-874 (v0.145.0). THE MEASURED SHAPE (Part F spike, Tester 2): power-on sessions of ~1.5 h and ~5 min against a
// 6 h evaluation ticker that restarts at every start — no restore-test ever evaluated. Now the first evaluation runs
// FirstEval after start.
// COMPANION RED-PROOF: drop the first-evaluation timer in Run (back to the bare ticker) → "no evaluation within".
func TestR874_FirstEvaluationAfterStart(t *testing.T) {
var picks int32
s := NewScheduler(SchedulerOptions{
Runner: &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}},
Pick: func(context.Context) (string, error) {
atomic.AddInt32(&picks, 1)
return fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", atomic.LoadInt32(&picks)), nil
},
Store: NewStore(), Spec: (&specSpy{}).build,
Cadence: 6 * time.Hour, FirstEval: 30 * time.Millisecond, Logger: quiet(),
})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { _ = s.Run(ctx); close(done) }()
deadline := time.Now().Add(3 * time.Second)
for atomic.LoadInt32(&picks) == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
cancel()
<-done
if atomic.LoadInt32(&picks) == 0 {
t.Fatal("no evaluation within 3 s of start (FirstEval 30 ms) — a box with short sessions never gets a restore-test")
}
}
// The earned restraint stays: an agent that restarts before FirstEval never evaluates (a crash loop does not
// hammer a failing tier).
func TestR874_CrashLoopNeverEvaluates(t *testing.T) {
var picks int32
for i := 0; i < 5; i++ { // five quick "restarts"
s := NewScheduler(SchedulerOptions{
Runner: &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true}},
Pick: func(context.Context) (string, error) { atomic.AddInt32(&picks, 1); return "x", nil },
Store: NewStore(), Spec: (&specSpy{}).build,
Cadence: 6 * time.Hour, FirstEval: 200 * time.Millisecond, Logger: quiet(),
})
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
_ = s.Run(ctx)
cancel()
}
if n := atomic.LoadInt32(&picks); n != 0 {
t.Fatalf("a restart before FirstEval evaluated %d time(s)", n)
}
if DefaultFirstEval != 30*time.Minute {
t.Fatalf("DefaultFirstEval = %s, the documented 30 min", DefaultFirstEval)
}
}
+132 -1
View File
@@ -5,6 +5,7 @@ import (
"fmt"
"log/slog"
"sort"
"strconv"
"strings"
"sync"
"time"
@@ -356,12 +357,58 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
if err != nil {
return "", time.Time{}, err
}
// R-727 (v0.138.0): on an ENCRYPTED storage, only archives written with THIS storage's key are this box's.
// Measured 2026-09-30 on a fresh box for a returning customer: the PBS namespace still held two archives
// of earlier boxes (same guest id 9201, same token), the newest settled one was an earlier box's, and the
// test failed `wrong key` every evaluation. The archive carries no host id; its key fingerprint is the
// discriminator (PVE's content `encrypted`, the storage's `encryption-key`). A lookup failure returns
// the error — the tier reads UNKNOWN, never "nothing to prove".
ownKey, err := r.storageKeyFingerprint(ctx, target)
if err != nil {
return "", time.Time{}, fmt.Errorf("reading the key fingerprint of storage %s: %w", target, err)
}
var best string
var bestCTime int64 = -1
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
for _, e := range contents {
if e.Content != "backup" {
continue
}
// R-689 (v0.135.0): only a backup OF A GUEST is a restore-test candidate. demo-hp keeps its golden
// template in `local:backup/` — content "backup", 654 MB, plausibly complete — and it was picked as
// the newest settled archive every 6 h and failed extractconfig (403) each time, while the guest's
// real archive went untested.
if ok, why := guestBackupArchive(e); !ok {
r.noteNotAGuestBackupOnce(e, why)
continue
}
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
continue
}
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
// v0.135.0: with the golden skipped, the pick fell to `vzdump-lxc-9100-2026_08_21…`, a leftover of a
// guest deleted in August — proving nothing about any guest this box runs. "Does not exist" skips the
// archive; any OTHER lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove".
if _, seen := known[e.VMID]; !seen {
_, err := r.api.GuestConfig(ctx, e.VMID)
switch {
case err == nil:
known[e.VMID] = true
case strings.Contains(err.Error(), "does not exist"), strings.Contains(err.Error(), "permission denied"):
// v0.137.0: PVE answers 403 "permission denied at /vms/<id>" — not "does not exist" — for a guest
// outside the agent's ACL (the `felhom` pool). Measured on demo-hp after v0.136.0: the deleted
// guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot
// read is not one it manages; its archive is not a candidate.
known[e.VMID] = false
default:
return "", time.Time{}, fmt.Errorf("checking whether guest %d still exists: %w", e.VMID, err)
}
}
if !known[e.VMID] {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("guest %d does not exist on this node or is not one this agent manages", e.VMID))
continue
}
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
continue // not settled yet — a newer archive is not a reason to re-prove an older one
}
@@ -471,10 +518,25 @@ func (r *BackupRunner) warnRejectedArchiveOnce(e proxmox.StorageContent, why str
if seen {
return
}
r.logger.Warn("backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup",
r.logger.Warn(rejectedArchiveMessage(e),
"target", r.target, "vmid", e.VMID, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
}
// phantomCleanupPointer names the runbook that removes a PBS phantom (R-99, `09` §3 decision 140: a leftover of an
// aborted upload is deleted on the backup server, by a runbook, when one is seen — never automatically).
const phantomCleanupPointer = " — a phantom leftover; delete it by felhom.eu documentation/runbooks/pbs-phantom-cleanup.md (09 §3 decision 140)"
// rejectedArchiveMessage is the WARN text for a rejected archive. Only a PBS entry (format pbs-ct / pbs-vm) gets the
// cleanup pointer: the runbook deletes on a PBS datastore, and a tiny archive on a dir storage is not a PBS phantom.
// Pinned by TestRejectedArchiveWarnNamesTheCleanupRunbook.
func rejectedArchiveMessage(e proxmox.StorageContent) string {
msg := "backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup"
if strings.HasPrefix(e.Format, "pbs-") {
msg += phantomCleanupPointer
}
return msg
}
// demo-felhom in a single afternoon of deploys (2026-07-26).
//
// Asking the STORAGE rather than persisting the store is deliberate:
@@ -591,3 +653,72 @@ func ToHubRestoreTest(res reconcile.RestoreTestResult, testedAt time.Time) hub.R
}
return rt
}
// guestBackupArchive reports whether a storage entry is a whole-guest backup of a known guest — a
// `vzdump-<type>-<vmid>-…` file on a dir storage, or a `backup/{ct,vm}/<vmid>/<time>` snapshot on a PBS
// datastore — whose vmid the storage itself reports. Anything else in a backup content type (a golden
// template, a hand-copied tarball) is not a backup of a guest and is never restore-tested (R-689).
// Pure, so the rule is unit-tested without a storage.
func guestBackupArchive(e proxmox.StorageContent) (bool, string) {
if e.VMID <= 0 {
return false, "not a backup of a guest (the storage reports no vmid)"
}
vol := e.VolID
if i := strings.Index(vol, ":"); i >= 0 {
vol = vol[i+1:]
}
vol = strings.TrimPrefix(vol, "backup/")
vmid := strconv.Itoa(e.VMID)
switch {
case strings.HasPrefix(vol, "vzdump-lxc-"+vmid+"-"), strings.HasPrefix(vol, "vzdump-qemu-"+vmid+"-"):
return true, ""
case strings.HasPrefix(vol, "ct/"+vmid+"/"), strings.HasPrefix(vol, "vm/"+vmid+"/"):
return true, ""
}
return false, "not a vzdump archive or a PBS snapshot of guest " + vmid
}
// noteNotAGuestBackupOnce logs, once per volid, that a backup-content entry is not a restore-test
// candidate because it is not a backup of a guest (R-689). INFO, not WARN: a golden template kept in
// the backup directory is the operator's, and not a fault.
func (r *BackupRunner) noteNotAGuestBackupOnce(e proxmox.StorageContent, why string) {
r.rejectedMu.Lock()
if r.rejected == nil {
r.rejected = map[string]struct{}{}
}
_, seen := r.rejected[e.VolID]
if !seen {
r.rejected[e.VolID] = struct{}{}
}
r.rejectedMu.Unlock()
if !seen {
r.logger.Info("backup: restore-test skips an entry that is not a backup of a guest",
"target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
}
}
// storageKeyFingerprint returns the named storage's client-side encryption key fingerprint ("" when the
// storage is not encrypted — a local dir holds only this box's own vzdumps).
func (r *BackupRunner) storageKeyFingerprint(ctx context.Context, target string) (string, error) {
sts, err := r.api.ListStorage(ctx)
if err != nil {
return "", err
}
for _, st := range sts {
if st.Storage == target {
return strings.TrimSpace(st.EncryptionKey), nil
}
}
return "", nil
}
// shortFP is the first 8 bytes of a key fingerprint, for a log line.
func shortFP(fp string) string {
if fp == "" {
return "none"
}
if len(fp) > 23 {
return fp[:23] + "…"
}
return fp
}
+34 -6
View File
@@ -60,12 +60,21 @@ type Scheduler struct {
// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
// (single tier via `pick`), which keeps every existing caller and test working untouched.
tiers []string // configured tier target ids, primary first
tierPick TierPicker // newest archive on a named tier
rtState *RestoreTestState // persisted last-successful-per-tier (drives oldest-first)
inFlight *InFlight // shared with the backup path — Scenario F
tiers []string // configured tier target ids, primary first
tierPick TierPicker // newest archive on a named tier
rtState *RestoreTestState // persisted last-successful-per-tier (drives oldest-first)
inFlight *InFlight // shared with the backup path — Scenario F
firstEval time.Duration // R-874: the first evaluation after start
}
// DefaultFirstEval (R-874): the first due-ness evaluation runs 30 minutes after the agent starts, then every
// cadence. MEASURED need (2026-10-05 Part F spike): a box whose power-on sessions are all shorter than the 6 h
// interval (Tester 2: ~1.5 h and ~5 min) NEVER evaluated, because the ticker restarts at each start. 30 minutes
// keeps the earned restraint below — a crash-looping agent restarts far more often than that and still never
// evaluates — while a box that stays on for half an hour gets its due test. Pinned by
// TestR874_FirstEvaluationAfterStart and TestR874_CrashLoopNeverEvaluates.
const DefaultFirstEval = 30 * time.Minute
// SchedulerOptions configures a Scheduler.
type SchedulerOptions struct {
Runner RestoreTestRunner
@@ -81,6 +90,8 @@ type SchedulerOptions struct {
// 0 → no settle requirement (any archive is a candidate).
Settle time.Duration
Logger *slog.Logger
// FirstEval (R-874, v0.145.0) is when the FIRST evaluation runs after start; 0 → DefaultFirstEval.
FirstEval time.Duration
// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
@@ -110,6 +121,12 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
tierPick: opts.TierPick,
rtState: opts.State,
inFlight: opts.InFlight,
firstEval: func() time.Duration {
if opts.FirstEval > 0 {
return opts.FirstEval
}
return DefaultFirstEval
}(),
}
}
@@ -120,8 +137,9 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
// trigger any more: its phase is the process's uptime, and agent deploys reset it, which is exactly
// the defect R-86 removes. What decides that a test happens is `EvaluateDue`.
//
// It still does NOT evaluate immediately on start — the first evaluation is one interval in. That
// is an EARNED restraint, kept deliberately: a restore is heavy, agent restarts are routine, and a
// It still does NOT evaluate immediately on start. v0.145.0 (R-874): the first evaluation is
// firstEval (30 min) in, then every interval — it was one full interval in, which a box with short
// power-on sessions never reached. The restraint itself is EARNED and kept: a restore is heavy, agent restarts are routine, and a
// crash-loop that evaluated at start would hammer a permanently-failing tier as fast as it could
// restart. Due-ness does not expire while we wait, so the only cost is up to one interval of
// latency on a tier that just became due. On-demand runs use `--selftest=restore-test`.
@@ -135,6 +153,16 @@ func (s *Scheduler) Run(ctx context.Context) error {
}
s.logger.Info("backup: restore-test scheduler starting (per-archive due-check)",
"eval_interval", s.cadence, "settle", s.settle)
first := time.NewTimer(s.firstEval)
defer first.Stop()
select {
case <-ctx.Done():
s.logger.Info("backup: restore-test scheduler shutting down", "reason", ctx.Err())
return nil
case <-first.C:
s.logger.Info("backup: restore-test first evaluation after start (R-874)", "after", s.firstEval)
s.tick(ctx)
}
t := time.NewTicker(s.cadence)
defer t.Stop()
for {
+8 -1
View File
@@ -25,7 +25,14 @@ import (
// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
// Backups are NOT unaffected (corrected 2026-10-05, R-348): byTarget is in memory too, so after a restart the
// reported backup LIST reads 0 until the next backup of each tier runs (daily local, weekly offsite) — measured
// 2026-08-20, two consecutive host-reports with `0 backups` while `pvesm list` showed archives on both tiers. What
// is unaffected is the hub's VERDICT: it looks back 7 days over stored reports (felhom.eu hub/internal/monitor/
// deadline.go backupEvidenceLookback) and the storage stays the ground truth (R-84).
// The due-check's fallback for an UNREADABLE storage no longer reads this store alone (R-894): the newest
// success per tier is also on disk (BackupSuccessState), so a restart followed by an unreachable storage
// reads the last known copy, not "never".
type Store struct {
mu sync.Mutex
byTarget map[string]hub.Backup // latest backup per target id
+15 -12
View File
@@ -81,10 +81,8 @@ var manifest = []Capability{
{"drives-mkdir-sub", "per-drive stable dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives/felhom-usb"}, false, ""},
{"drives-mkdir-data", "felhom-data namespace create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-usb/felhom-data"}, false, ""},
{"drives-chown-data", "felhom-data guest-root chown", "/usr/bin/chown", []string{"100000:100000", "/mnt/felhom-usb/felhom-data"}, false, ""},
{"parent-script-install", "shared-parent boot script install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-shared-parent-123456789.sh", "/usr/local/sbin/felhom-shared-parent.sh"}, false, ""},
{"parent-unit-install", "shared-parent boot unit install", "/usr/bin/install", []string{"-m", "0644", "--", "/tmp/felhom-shared-parent-123456789.service", "/etc/systemd/system/felhom-shared-parent.service"}, false, ""},
{"parent-unit-enable", "shared-parent boot-persistence enable", "/usr/bin/systemctl", []string{"enable", "felhom-shared-parent.service"}, false, ""},
{"parent-bind-mp8", "parent bind into guest at provision", "/usr/sbin/pct", []string{"set", "9201", "-mp8", "/mnt/felhom-drives"}, false, ""},
{"parent-bind-mp8", "parent bind into guest at provision", "/usr/sbin/pct", []string{"set", "9201", "-mp8", "/mnt/felhom-drives,mp=/mnt/felhom-drives"}, false, ""},
// ---- Disk inspect / format gate (Critical: the data-bearing classifier + format) ----
{"disk-blkid", "disk data-bearing classify (format gate)", "/usr/sbin/blkid", []string{"-p", "-o", "export", "/dev/sda"}, true, ""},
@@ -95,11 +93,11 @@ var manifest = []Capability{
{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
// ---- Storage mount units (watchdog re-mount) ----
{"mount-unit-install", "fs-UUID mount unit install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0644", "--", "/var/lib/felhom-agent/units/felhom-x.mount", "/etc/systemd/system/felhom-x.mount"}, false, ""},
{"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "felhom-x.mount"}, false, ""},
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "felhom-x.mount"}, false, ""},
{"mount-unit-stop", "mount unit stop", "/usr/bin/systemctl", []string{"stop", "--", "felhom-x.mount"}, false, ""},
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-stop", "mount unit stop", "/usr/bin/systemctl", []string{"stop", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
// ---- Network storage re-arm + cleanup (CAMPAIGN-3 F10/F1) ----
{"netmount-reset-failed", "NAS automount re-arm after start-limit (F10)", "/usr/bin/systemctl", []string{"reset-failed", "--", "mnt-felhom\\x2ddrives-media.automount"}, false, ""},
@@ -110,19 +108,21 @@ var manifest = []Capability{
// ---- Provisioning back-half ----
{"provision-chown", "bootstrap mount guest-root chown", "/usr/bin/chown", []string{"-R", "100000:100000", "/var/lib/felhom-agent/guests/9201"}, false, ""},
{"provision-config-mount", "bootstrap config bind mount", "/usr/sbin/pct", []string{"set", "9201", "-mp0", "/var/lib/felhom-agent/guests/9201"}, false, ""},
{"provision-config-mount", "bootstrap config bind mount", "/usr/sbin/pct", []string{"set", "9201", "-mp0", "/var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1"}, false, ""},
{"provision-onboot", "customer guest autostart (onboot)", "/usr/sbin/pct", []string{"set", "9201", "-onboot", "1"}, false, ""},
// ---- Pre-start self-heal hook + guest lifecycle ----
{"guesthook-install", "pre-start hook snippet install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-guest-hook-123456789.sh", "/var/lib/vz/snippets/felhom-guest-hook.sh"}, false, ""},
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
// ---- LAN split-horizon resolver (dnsmasq) ----
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
{"dnsmasq-write", "dnsmasq drop-in write (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"dnsmasq", "/tmp/felhom-resolver-123456789.conf", "felhom-x.conf"}, false, ""},
{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
// ---- OS updates, guest fast lane (`11` §5.4.1; the wrapper holds every rule) ----
{"osapply-run", "OS update wrapper (guest fast lane)", "/usr/local/sbin/felhom-os-apply", []string{"--plan", "/var/lib/felhom-agent/os/plan-x.json"}, false, ""},
{"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""},
{"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""},
{"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
@@ -151,13 +151,17 @@ var manifest = []Capability{
// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
{"stalelock-unlock", "reboot-during-backup stale-lock recovery", "/usr/sbin/pct", []string{"unlock", "9201"}, true, ""},
// ---- Weekly guest disk trim (FELHOM_FSTRIM, R-444). NON-critical: a missing grant means the thin pool is not
// reclaimed this week (the trim job WARNs per guest and the report shows the failure), not a serving outage. ----
{"guest-fstrim", "weekly guest disk trim (thin-pool reclaim, R-444)", "/usr/sbin/pct", []string{"fstrim", "9201"}, false, ""},
// ---- Offsite WG tunnel (FELHOM_WG, S3/v0.64.0; Critical FLIPPED in S4/v0.66.0 — offsite
// backups now RIDE the tunnel, so a degraded tunnel capability is operator-alert-worthy: the
// conf install, unit enable/restart and the handshake read gate the backup path. apt-install
// (one-time bootstrap) and disable (revocation, a deliberate teardown) stay non-critical. The
// handshake read is the ONLY wg invocation (never `dump`). ----
{"wg-tools-install", "wireguard-tools package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "wireguard-tools"}, false, ""},
{"wg-conf-install", "wg-felhom conf install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0600", "--", "/var/lib/felhom-agent/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"}, true, ""},
{"wg-conf-install", "wg-felhom conf install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"wg"}, true, ""},
{"wg-enable", "wg-quick@wg-felhom enable", "/usr/bin/systemctl", []string{"enable", "--now", "wg-quick@wg-felhom"}, true, ""},
{"wg-restart", "wg-quick@wg-felhom restart (conf change)", "/usr/bin/systemctl", []string{"restart", "wg-quick@wg-felhom"}, true, ""},
{"wg-disable", "wg-quick@wg-felhom disable (revocation)", "/usr/bin/systemctl", []string{"disable", "--now", "wg-quick@wg-felhom"}, false, ""},
@@ -189,7 +193,6 @@ var manifest = []Capability{
// operator-driven op, not a steady-state serving path — a degraded grant means "can't
// self-update" (fall back to a manual SSH deploy), not a serving outage. The apply repr uses a
// staging-dir path + a placeholder sha (list-mode never runs it). ----
{"selfupdate-apply", "agent self-update apply (A/B flip)", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"apply", "/var/lib/felhom-agent/selfupdate/felhom-agent-0.0.0", "0000000000000000000000000000000000000000000000000000000000000000"}, false, ""},
{"selfupdate-commit", "agent self-update commit", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"commit"}, false, ""},
{"selfupdate-rollback", "agent self-update rollback", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"rollback"}, false, ""},
}
+19 -3
View File
@@ -56,8 +56,10 @@ func parseSudoersEntries(t *testing.T, text string) []string {
var cur strings.Builder
for i := 0; i < len(raw); i++ {
c := raw[i]
if c == '\\' && i+1 < len(raw) {
cur.WriteByte(raw[i+1]) // unescape: keep the next char literally (\, → , ; \: → :)
if c == '\\' && i+1 < len(raw) && strings.IndexByte(",:=", raw[i+1]) >= 0 {
// unescape the sudoers grammar escapes only (\, → , ; \: → : ; \= → =). Every other backslash is kept: in
// a regex entry (R-861) `\.` `\{` `\\` mean exactly what sudo's regex engine reads.
cur.WriteByte(raw[i+1])
i++
continue
}
@@ -108,10 +110,24 @@ func globToRegex(pat string) *regexp.Regexp {
return regexp.MustCompile(b.String())
}
// entryRegex compiles one sudoers entry. R-861 (v0.146.0): an entry whose ARGUMENTS are a sudo regular expression
// (`^...$`, sudo >= 1.9.10) is matched as one — the binary literally, then a space, then the arguments joined by spaces
// (sudo's own rule). Every other entry is an fnmatch glob (globToRegex). The parser has already undone the sudoers
// escapes (`\,` `\:` `\=` `\\`), which leaves a valid RE2 pattern.
func entryRegex(e string) *regexp.Regexp {
if i := strings.IndexByte(e, ' '); i > 0 {
bin, args := e[:i], e[i+1:]
if strings.HasPrefix(args, "^") && strings.HasSuffix(args, "$") {
return regexp.MustCompile("^" + regexp.QuoteMeta(bin) + " " + args[1:])
}
}
return globToRegex(e)
}
// matchesAny reports whether cmdline matches at least one sudoers entry pattern.
func matchesAny(cmdline string, entries []string) bool {
for _, e := range entries {
if globToRegex(e).MatchString(cmdline) {
if entryRegex(e).MatchString(cmdline) {
return true
}
}
@@ -0,0 +1,95 @@
package capability
import (
"os"
"testing"
)
// R-861 (agent v0.146.0): the attacks the old globs let through, each as the exact argv a compromised agent would
// send. NONE may match any entry of the new sudoers. The same list ran against the REAL sudo 1.9.16 (a throwaway
// container, and `sudo -l -U felhom-agent` on both demo boxes after the bundle): audits/hub-safety-2026-10-05/partF/.
//
// RED-PROOF: run this list against the v0.145.0 sudoers (git show v0.145.0:configs/felhom-agent.sudoers) → most of
// these MATCH (recorded in the same evidence folder).
var r861Injections = []string{
// a raw host disk for a guest (pct options smuggled through a vmid glob)
"/usr/sbin/pct set 9201 --dev0 /dev/sda -onboot 1",
"/usr/sbin/pct set 9201 --dev0 /dev/sda -mp8 /mnt/felhom-drives",
"/usr/sbin/pct set 9201 --delete mp0 --dev0 /dev/sda",
"/usr/sbin/pct set 9201 -mp0 /var/lib/felhom-agent/guests/9201/bootstrap,mp=/x --dev0 /dev/sda",
// a bind mount over /etc through traversal
"/usr/bin/mount --bind /mnt/../var/lib/felhom-agent/x/felhom-data /mnt/felhom-drives/x",
"/usr/bin/mount --bind /mnt/a/felhom-data /mnt/felhom-drives/../../etc/sudoers.d",
"/usr/bin/umount /mnt/felhom-drives/x /",
"/usr/bin/chown 100000:100000 /mnt/a/felhom-data /etc/shadow",
"/usr/bin/mkdir -p /mnt/felhom-drives/x /etc/systemd/system/evil.mount",
// root-read files the agent writes: gone as `install` lines
"/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/x.mount /etc/systemd/system/etc-sudoers.d.mount",
"/usr/bin/install -m 0755 -- /tmp/felhom-guest-hook-1.sh /var/lib/vz/snippets/felhom-guest-hook.sh",
"/usr/bin/install -m 0755 -- /tmp/felhom-shared-parent-1.sh /usr/local/sbin/felhom-shared-parent.sh",
"/usr/bin/install -m 0644 /tmp/felhom-resolver-1.conf /etc/dnsmasq.d/felhom-x.conf",
"/usr/bin/install -o root -g root -m 0600 -- /var/lib/felhom-agent/wg/wg-felhom.conf /etc/wireguard/wg-felhom.conf",
"/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/sshd_config /etc/felhom-sshd/sshd_config",
// the unsigned binary flip
"/usr/local/sbin/felhom-selfupdate-guarded apply /var/lib/felhom-agent/selfupdate/felhom-agent-9.9.9 0000000000000000000000000000000000000000000000000000000000000000",
// enabling or removing anything that is not ours
"/usr/bin/systemctl enable --now -- mnt-hdd_1.mount evil.service",
"/usr/bin/systemctl enable --now -- etc-sudoers.d.mount",
"/usr/bin/rm -f /etc/systemd/system/mnt-felhomx /etc/passwd",
"/usr/bin/rm -f /etc/dnsmasq.d/felhom-x.conf /etc/shadow",
"/usr/bin/rmdir /mnt/felhom-drives/x /etc",
// nftables commands chained after a set element
"/usr/sbin/nft add element inet felhom_oob operator_ips { 10.77.0.250 } ; flush ruleset",
// extra options to read-only tools
"/usr/sbin/smartctl -a -j /dev/sda -s off",
"/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data --config x",
"/usr/sbin/pct exec 9201 --keep-env -- docker inspect -f x felhom-controller",
"/usr/sbin/pct unlock 9201 --whatever",
// the checker with a path it must never take
"/usr/local/sbin/felhom-priv-apply unit ../../etc/x.mount",
"/usr/local/sbin/felhom-priv-apply dnsmasq /etc/shadow felhom-x.conf",
"/usr/local/sbin/felhom-priv-apply wg /etc/shadow",
}
func TestSudoersRefusesTheR861Injections(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatal(err)
}
entries := parseSudoersEntries(t, string(data))
for _, c := range r861Injections {
if matchesAny(c, entries) {
t.Errorf("the sudoers still allows: %s", c)
}
}
}
// R-444: the weekly trim's grant is ONE exact shape — `pct fstrim <vmid>` — and nothing smuggled after it.
// The manifest entry (guest-fstrim) proves the real call is still allowed (TestManifestCoveredBySudoers); this
// pins the other direction. RED-PROOF: write the rule as the glob `/usr/sbin/pct fstrim [0-9]*` → every decoy
// below with a trailing argument matches (the glob's `*` eats spaces).
func TestSudoersFstrimRuleIsExact(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatal(err)
}
entries := parseSudoersEntries(t, string(data))
if !matchesAny("/usr/sbin/pct fstrim 9201", entries) {
t.Fatal("the sudoers does not allow `pct fstrim 9201` — the weekly trim cannot run")
}
for _, c := range []string{
"/usr/sbin/pct fstrim 9201 --ignore-mountpoints",
"/usr/sbin/pct fstrim 9201 --ignore-mountpoints 1",
"/usr/sbin/pct fstrim 9201; x",
"/usr/sbin/pct fstrim 9201 9202",
"/usr/sbin/pct fstrim 92a1",
"/usr/sbin/pct fstrim ",
"/usr/sbin/pct fstrim -- 9201",
"/usr/sbin/pct destroy 9201",
"/usr/sbin/pct destroy 9201 --purge",
} {
if matchesAny(c, entries) {
t.Errorf("the sudoers allows a command the trim rule must not: %q", c)
}
}
}
+11
View File
@@ -33,6 +33,7 @@ type Config struct {
LANResolver LANResolverConfig `json:"lan_resolver"`
WGTunnel WGTunnelConfig `json:"wg_tunnel"`
GuestNet GuestNetConfig `json:"guest_net"`
DiskTrim DiskTrimConfig `json:"disk_trim"`
OOB OOBConfig `json:"oob"`
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
@@ -139,6 +140,16 @@ func (w WGTunnelConfig) WithDefaults() WGTunnelConfig {
return w
}
// DiskTrimConfig configures the R-444 weekly guest disk trim (internal/fstrim). DEFAULT-ON, like GuestNetConfig and for
// the same reason: it only acts on guests the agent already owns, and the operator ruled every box trims (`09` §3
// decision 139). Opting out is the explicit act: `"disk_trim": {"disable": true}`.
type DiskTrimConfig struct {
Disable bool `json:"disable"`
}
// Enabled reports whether the weekly trim should run.
func (d DiskTrimConfig) Enabled() bool { return !d.Disable }
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
//
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
+48 -2
View File
@@ -7,9 +7,11 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"syscall"
)
// Slice 10D.1 — IDENTITY escrow. The K-escrow (above) wraps the PBS *encryption key* via the
@@ -71,7 +73,7 @@ func HashResticPassword(pw string) string {
// (non-empty) but NEVER logged by callers — log the field NAME only (mirrors AttachWGKey). A missing file
// is a clean no-attach (pre-fork-4 behavior, byte-compatible bundle).
func AttachResticPassword(b *IdentityBundle, stagePath string) (bool, error) {
raw, err := os.ReadFile(stagePath)
raw, err := readStagedNoFollow(stagePath)
if err != nil {
if os.IsNotExist(err) {
return false, nil
@@ -92,7 +94,7 @@ func AttachResticPassword(b *IdentityBundle, stagePath string) (bool, error) {
// no-attach (pre-S3 behavior, byte-compatible bundle); a corrupt one is an error (the operator
// should know their escrow would silently lack a live identity).
func AttachWGKey(b *IdentityBundle, keyPath string) (bool, error) {
raw, err := os.ReadFile(keyPath)
raw, err := readStagedNoFollow(keyPath)
if err != nil {
if os.IsNotExist(err) {
return false, nil
@@ -180,3 +182,47 @@ func UnwrapIdentityBundle(ctx context.Context, blob []byte, recoveryCode string)
}
return b, nil
}
// readStagedNoFollow reads a file the AGENT staged, for the escrow ceremony that runs as ROOT (FELHOM_ESCROW). R-861
// (agent v0.146.0/0.146.1): both files live in the agent's own directory, so a compromised agent could put a SYMLINK
// there — at the file OR at any directory on the way (review 2026-10-05) — to a root-only file, and the root ceremony
// would seal that file into the blob and hand the agent R. So the path is walked from "/" one component at a time with
// openat(O_NOFOLLOW): no symlink anywhere, the last a regular file of at most 4 KiB. Once a directory is open, renaming
// it does not redirect the walk. A missing file keeps its os.IsNotExist meaning. Pinned by TestAttach_RefusesASymlink*.
func readStagedNoFollow(path string) ([]byte, error) {
if !filepath.IsAbs(path) {
return nil, fmt.Errorf("%s is not an absolute path", path)
}
clean := filepath.Clean(path)
parts := strings.Split(strings.TrimPrefix(clean, "/"), "/")
dirfd, err := syscall.Open("/", syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_CLOEXEC, 0)
if err != nil {
return nil, err
}
for i, part := range parts {
last := i == len(parts)-1
flags := syscall.O_RDONLY | syscall.O_NOFOLLOW | syscall.O_CLOEXEC
if !last {
flags |= syscall.O_DIRECTORY
}
fd, err := syscall.Openat(dirfd, part, flags, 0)
syscall.Close(dirfd)
if err != nil {
return nil, &os.PathError{Op: "open", Path: clean, Err: err}
}
dirfd = fd
}
f := os.NewFile(uintptr(dirfd), clean)
defer f.Close()
fi, err := f.Stat()
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("%s is not a regular file", path)
}
if fi.Size() > 4096 {
return nil, fmt.Errorf("%s is larger than 4 KiB", path)
}
return io.ReadAll(io.LimitReader(f, 4097))
}
+63
View File
@@ -0,0 +1,63 @@
package escrow
import (
"os"
"path/filepath"
"testing"
)
// R-861 (agent v0.146.0): the root escrow ceremony reads two files from the AGENT's directory. A symlink there to a
// root-only file must never be read (it would be sealed under R and handed to the agent).
// RED-PROOF (audits/hub-safety-2026-10-05/partF/red-proof.txt): use os.ReadFile in readStagedNoFollow → this fails.
func TestAttach_RefusesASymlink(t *testing.T) {
d := t.TempDir()
secret := filepath.Join(d, "root-only")
if err := os.WriteFile(secret, []byte("ROOT-ONLY-CANARY"), 0o600); err != nil {
t.Fatal(err)
}
link := filepath.Join(d, "restic_repo_password")
if err := os.Symlink(secret, link); err != nil {
t.Fatal(err)
}
var b IdentityBundle
if ok, err := AttachResticPassword(&b, link); err == nil || ok || b.ResticRepoPassword != "" {
t.Fatalf("a symlinked staged file was read: ok=%v err=%v value-set=%v", ok, err, b.ResticRepoPassword != "")
}
if ok, err := AttachWGKey(&b, link); err == nil || ok {
t.Fatalf("a symlinked wg key was read: ok=%v err=%v", ok, err)
}
// control: the real staged file is still read; a missing one is still a clean no-attach
real := filepath.Join(d, "real")
_ = os.WriteFile(real, []byte("pw\n"), 0o600)
if ok, err := AttachResticPassword(&b, real); err != nil || !ok || b.ResticRepoPassword != "pw" {
t.Fatalf("control: the plain staged file was not read: %v %v", ok, err)
}
if ok, err := AttachResticPassword(&b, filepath.Join(d, "absent")); err != nil || ok {
t.Fatalf("control: a missing file must stay a clean no-attach: %v %v", ok, err)
}
}
// Review 2026-10-05: a symlinked DIRECTORY on the way must stop the read too (O_NOFOLLOW alone guards only the last
// component). RED-PROOF: open the full path with O_NOFOLLOW only → this fails.
func TestAttach_RefusesASymlinkedDirectory(t *testing.T) {
d := t.TempDir()
secretDir := filepath.Join(d, "root-only-dir")
_ = os.Mkdir(secretDir, 0o700)
_ = os.WriteFile(filepath.Join(secretDir, "private.key"), []byte("AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=\n"), 0o600)
agentDir := filepath.Join(d, "agent")
_ = os.Mkdir(agentDir, 0o700)
if err := os.Symlink(secretDir, filepath.Join(agentDir, "wg")); err != nil {
t.Fatal(err)
}
var b IdentityBundle
if ok, err := AttachWGKey(&b, filepath.Join(agentDir, "wg", "private.key")); err == nil || ok || b.WGPrivateKey != "" {
t.Fatalf("a key behind a symlinked directory was read: ok=%v err=%v", ok, err)
}
// control: the same key under a REAL directory is read
_ = os.Remove(filepath.Join(agentDir, "wg"))
_ = os.Mkdir(filepath.Join(agentDir, "wg"), 0o700)
_ = os.WriteFile(filepath.Join(agentDir, "wg", "private.key"), []byte("AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=\n"), 0o600)
if ok, err := AttachWGKey(&b, filepath.Join(agentDir, "wg", "private.key")); err != nil || !ok {
t.Fatalf("control: a key under a real directory was not read: %v %v", ok, err)
}
}
+2
View File
@@ -27,6 +27,8 @@ const (
PortFile = ConfDir + "/port"
// PidFile is the instance pidfile (NOT a RuntimeDirectory — that is the G1 incident cause).
PidFile = "/run/felhom-sshd.pid"
// PrivApply is the root content checker that installs the config and felhom-op's key (R-861, agent v0.146.0).
PrivApply = "/usr/local/sbin/felhom-priv-apply"
// Unit is the systemd unit name.
Unit = "felhom-sshd"
// OperatorUser is the default operator login (scoped sudo; key in AuthKeysDir only).
+5 -2
View File
@@ -143,7 +143,8 @@ func (m *Manager) Apply(ctx context.Context, block *hub.WireWireguard) (int, err
m.logger.Error("felhomsshd: staged config failed sshd -t — NOT installing", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return port, err
}
if _, errOut, err := m.runner.Run(ctx, "install", "-o", "root", "-g", "root", "-m", "0644", "--", m.stagedConfPath(), ConfPath); err != nil {
// R-861 (v0.146.0): the root checker installs it, and only if it is renderConfig's template for some port.
if _, errOut, err := m.runner.Run(ctx, PrivApply, "sshd-config"); err != nil {
m.logger.Error("felhomsshd: config install failed", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return port, err
}
@@ -181,7 +182,9 @@ func (m *Manager) applyAuthorizedKeys(ctx context.Context, sshKey string) {
m.logger.Error("felhomsshd: staging authorized_keys", "err", err)
return
}
if _, errOut, err := m.runner.Run(ctx, "install", "-o", "root", "-g", "root", "-m", "0644", "--", staged, AuthKeysUserPath); err != nil {
// R-861: the root checker installs it — one plain public key, no options (command=, from=, …), or empty.
_ = staged
if _, errOut, err := m.runner.Run(ctx, PrivApply, "sshd-key"); err != nil {
m.logger.Error("felhomsshd: authorized_keys install failed", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return
}
@@ -0,0 +1,26 @@
package felhomsshd
import (
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/privapplytest"
)
// R-861: renderConfig is byte-identical to the checker's template (only the Port varies); a changed line is refused.
// RED-PROOF: change one directive in renderConfig → this fails (the two templates drifted).
func TestPrivApply_AcceptsTheRenderedConfig(t *testing.T) {
for _, port := range []int{2222, 8822, 60022} {
conf, err := renderConfig(port)
if err != nil {
t.Fatal(err)
}
if got := privapplytest.Check(t, "sshd-config", "", conf); got != "OK" {
t.Errorf("port %d: %s", port, got)
}
}
conf, _ := renderConfig(8822)
if got := privapplytest.Check(t, "sshd-config", "", conf+"StrictModes no\n"); !strings.HasPrefix(got, "REFUSED") {
t.Fatalf("control: an extra directive was not refused: %s", got)
}
}
+346
View File
@@ -0,0 +1,346 @@
// Package fstrim is the weekly guest disk trim (R-444, operator ruling `09` §3 decision 139).
//
// Why: a thin pool only ever grows from blocks a guest has already FREED — `fstrim` inside the unprivileged container
// is refused (FITRIM: Operation not permitted), and nothing else on the box gives the blocks back. A full thin pool
// takes every guest on the host read-only, so the pool can reach 100 % from deleted data alone. Measured on demo-hp
// 2026-10-06 09:14Z: `pct fstrim 9201` rc 0 in 24.4 s, pool 65.53 % -> 33.40 %, 18/18 app probes 200, max 1.1 s
// (audits/ten-answers-2026-10-06/r444-measure.txt).
//
// The rule, each part pinned by a test in fstrim_test.go:
// - Weekly: a guest is DUE from Wednesday 10:00 local until it has been trimmed once since then (a box that was off
// on Wednesday catches up at its next eligible hour).
// - Daytime only: a trim starts only between 10:00 and 20:59 local — never in the night window (01:00–06:59) where
// the backups and the restore-tests run (TestEligibleHourNeverInTheNight).
// - Never beside a backup, a restore-test or another heavy operation: the pass holds the host-wide one-heavy-op gate
// (backup.InFlight) for its whole run; a busy gate DEFERS the pass to the next hourly tick.
// - A failed trim is retried at the next eligible hour, at most MaxAttemptsPerWeek times in one week.
// - The last result per guest (time, bytes, ok/fail) is persisted, so a restart neither loses it nor re-trims.
//
// The command is the ONE exact sudoers shape `pct fstrim <vmid>` (FELHOM_FSTRIM). Only guests from the pool-verified
// source (ListLXC ∩ the felhom pool, audit A1) and only RUNNING ones are trimmed.
package fstrim
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// Schedule. The weekday/hours are fixed on purpose (one sentence the operator can read on the System page).
const (
Weekday = time.Wednesday
StartHour = 10 // first eligible local hour (inclusive)
EndHour = 21 // first NOT-eligible local hour (exclusive): last start is 20:59
MaxAttemptsPerWeek = 3
// TickInterval is how often the job looks; a deferred or failed pass is therefore retried the next hour.
TickInterval = time.Hour
// FirstTickDelay lets the agent settle after a start before the first look.
FirstTickDelay = 5 * time.Minute
// PerGuestTimeout bounds one `pct fstrim` (measured 24.4 s for 84 GiB).
PerGuestTimeout = 30 * time.Minute
)
// ScheduleText is the human description carried on the host report.
const ScheduleText = "weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
// Runner runs a host command (proxmox.ExecRunner in production, through `sudo -n`).
type Runner interface {
Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
}
// GuestSource yields the guests this agent OWNS (the pool-verified source, never a bare ListLXC).
type GuestSource interface {
Guests(ctx context.Context) ([]proxmox.Guest, error)
}
// Gate is the host-wide one-heavy-operation gate (*backup.InFlight).
type Gate interface {
TryAcquire(what string) (release func(), busy string, ok bool)
}
// GateName is what the gate reports as busy while a trim runs.
const GateName = "guest-fstrim"
// Record is one guest's last trim attempt, as persisted.
type Record struct {
LastAttemptAt time.Time `json:"last_attempt_at"`
OK bool `json:"ok"`
BytesTrimmed int64 `json:"bytes_trimmed"`
Mounts int `json:"mounts"`
DurationSeconds float64 `json:"duration_seconds"`
LastOKAt time.Time `json:"last_ok_at,omitempty"`
Error string `json:"error,omitempty"`
// Attempts counts the attempts since the current week's due time (reset by the first attempt of a new week).
Attempts int `json:"attempts"`
}
// Trimmer is the weekly job.
type Trimmer struct {
runner Runner
guests GuestSource
gate Gate
statePath string
logger *slog.Logger
loc *time.Location
now func() time.Time
mu sync.Mutex
records map[int]Record
}
// New builds the job and loads the persisted state. A missing state file is an empty state; a corrupt one is logged
// and treated as empty (the cost is one extra trim, never a missed one).
func New(runner Runner, guests GuestSource, gate Gate, statePath string, logger *slog.Logger) *Trimmer {
if logger == nil {
logger = slog.Default()
}
t := &Trimmer{runner: runner, guests: guests, gate: gate, statePath: statePath, logger: logger,
loc: time.Local, now: time.Now, records: map[int]Record{}}
t.load()
return t
}
func (t *Trimmer) load() {
data, err := os.ReadFile(t.statePath)
if err != nil {
if !os.IsNotExist(err) {
t.logger.Warn("fstrim: state read failed — starting empty", "path", t.statePath, "err", err)
}
return
}
var raw map[string]Record
if err := json.Unmarshal(data, &raw); err != nil {
t.logger.Warn("fstrim: state file corrupt — starting empty", "path", t.statePath, "err", err)
return
}
for k, r := range raw {
if id, err := strconv.Atoi(k); err == nil && id > 0 {
t.records[id] = r
}
}
}
func (t *Trimmer) saveLocked() error {
raw := make(map[string]Record, len(t.records))
for id, r := range t.records {
raw[strconv.Itoa(id)] = r
}
data, err := json.MarshalIndent(raw, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(t.statePath), 0o755); err != nil {
return err
}
tmp := t.statePath + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, t.statePath)
}
// EligibleHour reports whether a trim may START at local time lt.
func EligibleHour(lt time.Time) bool {
h := lt.Hour()
return h >= StartHour && h < EndHour
}
// weekAnchor is the most recent Wednesday StartHour:00 at or before lt (same location as lt).
func weekAnchor(lt time.Time) time.Time {
daysBack := (int(lt.Weekday()) - int(Weekday) + 7) % 7
d := lt.AddDate(0, 0, -daysBack)
a := time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
if a.After(lt) {
d = d.AddDate(0, 0, -7)
a = time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
}
return a
}
// due reports whether a guest with record r (ok=false: none) is due at local time lt.
func due(r Record, has bool, lt time.Time) bool {
if !has {
return true
}
anchor := weekAnchor(lt)
if r.LastAttemptAt.Before(anchor) {
return true // not tried this week
}
return !r.OK && r.Attempts < MaxAttemptsPerWeek
}
// Run looks every TickInterval until ctx ends. It never returns an error: a failed trim is a reported fact.
func (t *Trimmer) Run(ctx context.Context) {
t.logger.Info("fstrim: weekly guest disk trim starting", "schedule", ScheduleText)
timer := time.NewTimer(FirstTickDelay)
defer timer.Stop()
for {
select {
case <-ctx.Done():
return
case <-timer.C:
t.Pass(ctx)
timer.Reset(TickInterval)
}
}
}
// Pass is one look: outside the daytime window it does nothing; otherwise it trims every due, running, owned guest
// while holding the heavy-op gate.
func (t *Trimmer) Pass(ctx context.Context) {
lt := t.now().In(t.loc)
if !EligibleHour(lt) {
t.logger.Debug("fstrim: outside the daytime window — not looking", "local", lt.Format("Mon 15:04"))
return
}
guests, err := t.guests.Guests(ctx)
if err != nil {
t.logger.Warn("fstrim: owned-guest list unavailable — skipping this pass", "err", err)
return
}
owned := make(map[int]bool, len(guests))
var todo []int
t.mu.Lock()
for _, g := range guests {
owned[g.VMID] = true
r, has := t.records[g.VMID]
if !due(r, has, lt) {
continue
}
if g.Status != "running" {
t.logger.Info("fstrim: guest not running — trimmed when it runs", "vmid", g.VMID, "status", g.Status)
continue
}
todo = append(todo, g.VMID)
}
// A guest the agent no longer owns has no result to report.
pruned := false
for id := range t.records {
if !owned[id] {
delete(t.records, id)
pruned = true
}
}
if pruned {
if err := t.saveLocked(); err != nil {
t.logger.Warn("fstrim: state save failed", "err", err)
}
}
t.mu.Unlock()
if len(todo) == 0 {
return
}
sort.Ints(todo)
release, busy, ok := t.gate.TryAcquire(GateName)
if !ok {
t.logger.Info("fstrim: deferred — a heavy operation is in flight; retrying next hour", "busy", busy, "due_guests", len(todo))
return
}
defer release()
for _, vmid := range todo {
if ctx.Err() != nil {
return
}
t.trimOne(ctx, vmid, lt)
}
}
var trimmedLine = regexp.MustCompile(`\((\d+) bytes\) trimmed`)
// ParseTrimmed sums the "(N bytes) trimmed" lines of `pct fstrim` output and counts them (one per mount point), e.g.
// `/var/lib/lxc/9201/rootfs/: 30.1 GiB (32277680128 bytes) trimmed`.
func ParseTrimmed(out string) (bytes int64, mounts int) {
for _, m := range trimmedLine.FindAllStringSubmatch(out, -1) {
n, err := strconv.ParseInt(m[1], 10, 64)
if err != nil {
continue
}
bytes += n
mounts++
}
return bytes, mounts
}
// GiB renders bytes as "30.1 GiB".
func GiB(b int64) string { return fmt.Sprintf("%.1f GiB", float64(b)/(1<<30)) }
func (t *Trimmer) trimOne(ctx context.Context, vmid int, lt time.Time) {
start := t.now()
cctx, cancel := context.WithTimeout(ctx, PerGuestTimeout)
stdout, stderr, err := t.runner.Run(cctx, "pct", "fstrim", strconv.Itoa(vmid))
cancel()
dur := t.now().Sub(start)
bytes, mounts := ParseTrimmed(string(stdout) + "\n" + string(stderr))
t.mu.Lock()
prev, has := t.records[vmid]
r := Record{LastAttemptAt: start.UTC(), OK: err == nil, BytesTrimmed: bytes, Mounts: mounts,
DurationSeconds: float64(dur.Round(100*time.Millisecond)) / float64(time.Second), LastOKAt: prev.LastOKAt}
if has && !prev.LastAttemptAt.Before(weekAnchor(lt)) {
r.Attempts = prev.Attempts + 1
} else {
r.Attempts = 1
}
if err == nil {
r.LastOKAt = start.UTC()
} else {
msg := strings.TrimSpace(err.Error() + ": " + strings.TrimSpace(string(stderr)))
if len(msg) > 300 {
msg = msg[:300]
}
r.Error = msg
}
t.records[vmid] = r
saveErr := t.saveLocked()
t.mu.Unlock()
if err == nil {
t.logger.Info(fmt.Sprintf("fstrim: guest %d trimmed %s in %.1fs", vmid, GiB(bytes), r.DurationSeconds),
"vmid", vmid, "bytes_trimmed", bytes, "mounts", mounts, "duration_s", r.DurationSeconds)
if mounts == 0 {
t.logger.Warn("fstrim: pct fstrim succeeded but reported no trimmed mount — output not understood",
"vmid", vmid, "stdout", strings.TrimSpace(string(stdout)))
}
} else {
t.logger.Warn(fmt.Sprintf("fstrim: guest %d trim FAILED after %.1fs", vmid, r.DurationSeconds),
"vmid", vmid, "attempt", r.Attempts, "max_attempts_per_week", MaxAttemptsPerWeek, "err", r.Error)
}
if saveErr != nil {
t.logger.Warn("fstrim: state save failed — the result will not survive a restart", "path", t.statePath, "err", saveErr)
}
}
// GuestDiskTrimStatus implements hub.GuestDiskTrimReporter: a pure read of the persisted results (never runs pct).
func (t *Trimmer) GuestDiskTrimStatus(context.Context) *hub.GuestDiskTrimStatus {
t.mu.Lock()
defer t.mu.Unlock()
out := &hub.GuestDiskTrimStatus{Schedule: ScheduleText}
ids := make([]int, 0, len(t.records))
for id := range t.records {
ids = append(ids, id)
}
sort.Ints(ids)
for _, id := range ids {
r := t.records[id]
g := hub.GuestDiskTrim{VMID: id, LastAttemptAt: r.LastAttemptAt.UTC().Format(time.RFC3339), OK: r.OK,
BytesTrimmed: r.BytesTrimmed, Mounts: r.Mounts, DurationSeconds: r.DurationSeconds, Error: r.Error}
if !r.LastOKAt.IsZero() {
g.LastOKAt = r.LastOKAt.UTC().Format(time.RFC3339)
}
out.Guests = append(out.Guests, g)
}
return out
}
+267
View File
@@ -0,0 +1,267 @@
package fstrim
import (
"bytes"
"context"
"encoding/json"
"errors"
"log/slog"
"path/filepath"
"reflect"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// The real `pct fstrim 9201` output measured on demo-hp 2026-10-06 (audits/ten-answers-2026-10-06/r444-measure.txt).
const measuredOut = "/var/lib/lxc/9201/rootfs/: 30.1 GiB (32277680128 bytes) trimmed\n" +
"/var/lib/lxc/9201/rootfs/var/lib/felhom: 53.9 GiB (57865633792 bytes) trimmed\n"
const measuredBytes = int64(32277680128 + 57865633792)
type fakeRunner struct {
mu sync.Mutex
calls [][]string
out string
err error
onRun func()
}
func (f *fakeRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
f.mu.Lock()
f.calls = append(f.calls, append([]string{name}, args...))
f.mu.Unlock()
if f.onRun != nil {
f.onRun()
}
if f.err != nil {
return nil, []byte("mount busy"), f.err
}
return []byte(f.out), nil, nil
}
type fakeGuests struct {
g []proxmox.Guest
err error
}
func (f fakeGuests) Guests(context.Context) ([]proxmox.Guest, error) { return f.g, f.err }
// A Wednesday 10:30 in a fixed zone (CEST-like), so the tests do not depend on the machine's zone.
var zone = time.FixedZone("CEST", 2*3600)
func at(day, hour, min int) time.Time { return time.Date(2026, 10, day, hour, min, 0, 0, zone) } // 2026-10-07 = Wednesday
func newT(t *testing.T, r Runner, g GuestSource, gate Gate, now *time.Time) (*Trimmer, *bytes.Buffer, string) {
t.Helper()
var logs bytes.Buffer
path := filepath.Join(t.TempDir(), "guest-disk-trim.json")
tr := New(r, g, gate, path, slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelDebug})))
tr.loc = zone
tr.now = func() time.Time { return *now }
return tr, &logs, path
}
func running(ids ...int) fakeGuests {
var g []proxmox.Guest
for _, id := range ids {
g = append(g, proxmox.Guest{VMID: id, Status: "running", Type: "lxc"})
}
return fakeGuests{g: g}
}
func TestParseTrimmedTheMeasuredOutput(t *testing.T) {
b, m := ParseTrimmed(measuredOut)
if b != measuredBytes || m != 2 {
t.Fatalf("ParseTrimmed = %d bytes over %d mounts, want %d over 2", b, m, measuredBytes)
}
if b, m := ParseTrimmed("something else\n"); b != 0 || m != 0 {
t.Fatalf("unrelated output parsed as %d/%d", b, m)
}
if got := GiB(measuredBytes); got != "84.0 GiB" {
t.Fatalf("GiB = %q", got)
}
}
// The night window (01:00–06:59) must never be eligible, and the daytime window is exactly 10:00–20:59.
func TestEligibleHourNeverInTheNight(t *testing.T) {
for h := 0; h < 24; h++ {
lt := time.Date(2026, 10, 7, h, 30, 0, 0, zone)
got := EligibleHour(lt)
if h >= 1 && h <= 6 && got {
t.Errorf("hour %02d is in the night window and must not be eligible", h)
}
if want := h >= 10 && h <= 20; got != want {
t.Errorf("EligibleHour(%02d:30) = %v, want %v", h, got, want)
}
}
}
func TestWeekAnchorIsTheLastWednesdayTen(t *testing.T) {
cases := map[time.Time]time.Time{
at(7, 10, 0): at(7, 10, 0), // Wednesday 10:00 itself
at(7, 9, 59): time.Date(2026, 9, 30, 10, 0, 0, 0, zone), // before 10:00 Wednesday → the previous week
at(8, 15, 0): at(7, 10, 0), // Thursday
at(13, 20, 0): at(7, 10, 0), // next Tuesday
at(14, 11, 0): at(14, 10, 0), // next Wednesday
}
for in, want := range cases {
if got := weekAnchor(in); !got.Equal(want) {
t.Errorf("weekAnchor(%s) = %s, want %s", in.Format("Mon 01-02 15:04"), got.Format("Mon 01-02 15:04"), want.Format("Mon 01-02 15:04"))
}
}
}
// The consequence: on Wednesday 10:30 a running owned guest is trimmed with the ONE exact argv, the bytes are parsed,
// the positive log line is written, the result is persisted, and the host report carries it.
func TestPassTrimsADueGuestAndReportsIt(t *testing.T) {
now := at(7, 10, 30)
r := &fakeRunner{out: measuredOut}
tr, logs, path := newT(t, r, running(9201), &backup.InFlight{}, &now)
tr.Pass(context.Background())
if want := [][]string{{"pct", "fstrim", "9201"}}; !reflect.DeepEqual(r.calls, want) {
t.Fatalf("runner calls = %q, want %q", r.calls, want)
}
if !strings.Contains(logs.String(), "fstrim: guest 9201 trimmed 84.0 GiB in ") {
t.Fatalf("no positive per-guest log line:\n%s", logs.String())
}
st := tr.GuestDiskTrimStatus(context.Background())
if st == nil || st.Schedule != ScheduleText || len(st.Guests) != 1 {
t.Fatalf("report stanza = %+v", st)
}
g := st.Guests[0]
if g.VMID != 9201 || !g.OK || g.BytesTrimmed != measuredBytes || g.Mounts != 2 || g.LastOKAt == "" || g.LastAttemptAt == "" {
t.Fatalf("report guest = %+v", g)
}
// Persisted: a NEW Trimmer over the same file (an agent restart) still has it and does not trim again this week.
now = at(8, 11, 0)
r2 := &fakeRunner{out: measuredOut}
tr2 := New(r2, running(9201), &backup.InFlight{}, path, slog.New(slog.NewTextHandler(&bytes.Buffer{}, nil)))
tr2.loc, tr2.now = zone, func() time.Time { return now }
if st2 := tr2.GuestDiskTrimStatus(context.Background()); len(st2.Guests) != 1 || st2.Guests[0].BytesTrimmed != measuredBytes {
t.Fatalf("result lost over a restart: %+v", st2)
}
tr2.Pass(context.Background())
if len(r2.calls) != 0 {
t.Fatalf("trimmed again in the same week after a restart: %q", r2.calls)
}
// Next week it is due again.
now = at(14, 10, 5)
tr2.Pass(context.Background())
if len(r2.calls) != 1 {
t.Fatalf("not trimmed in the next week: %q", r2.calls)
}
}
func TestPassNeverRunsInTheNight(t *testing.T) {
for _, h := range []int{1, 3, 6, 9, 21, 23} {
now := at(7, h, 15)
r := &fakeRunner{out: measuredOut}
tr, _, _ := newT(t, r, running(9201), &backup.InFlight{}, &now)
tr.Pass(context.Background())
if len(r.calls) != 0 {
t.Errorf("trimmed at %02d:15: %q", h, r.calls)
}
}
}
// A backup (or restore-test) holding the heavy-op gate DEFERS the trim; the next hour, gate free, it runs. And while
// a trim runs, the gate is held, so a backup cannot start beside it.
func TestPassDefersToAHeavyOperationAndRetriesNextHour(t *testing.T) {
now := at(7, 10, 30)
gate := &backup.InFlight{}
release, _, _ := gate.TryAcquire("backup:9201")
var busyDuringTrim string
r := &fakeRunner{out: measuredOut}
r.onRun = func() { busyDuringTrim = gate.Busy() }
tr, logs, _ := newT(t, r, running(9201), gate, &now)
tr.Pass(context.Background())
if len(r.calls) != 0 {
t.Fatalf("trimmed beside a running backup: %q", r.calls)
}
if !strings.Contains(logs.String(), "fstrim: deferred") || !strings.Contains(logs.String(), "backup:9201") {
t.Fatalf("the deferral is not logged with what holds the gate:\n%s", logs.String())
}
release()
now = now.Add(time.Hour)
tr.Pass(context.Background())
if len(r.calls) != 1 {
t.Fatalf("not retried the next hour: %q", r.calls)
}
if busyDuringTrim != GateName {
t.Fatalf("the heavy-op gate was %q during the trim, want %q", busyDuringTrim, GateName)
}
if gate.Busy() != "" {
t.Fatalf("the gate was not released after the pass: %q", gate.Busy())
}
}
func TestFailedTrimWarnsIsRecordedAndRetriedAtMostThreeTimes(t *testing.T) {
now := at(7, 10, 30)
r := &fakeRunner{err: errors.New("exit status 255")}
tr, logs, _ := newT(t, r, running(9201), &backup.InFlight{}, &now)
for i := 0; i < 6; i++ {
tr.Pass(context.Background())
now = now.Add(time.Hour)
}
if len(r.calls) != MaxAttemptsPerWeek {
t.Fatalf("attempts in one week = %d, want %d", len(r.calls), MaxAttemptsPerWeek)
}
if !strings.Contains(logs.String(), "level=WARN") || !strings.Contains(logs.String(), "fstrim: guest 9201 trim FAILED") {
t.Fatalf("no WARN for the failure:\n%s", logs.String())
}
g := tr.GuestDiskTrimStatus(context.Background()).Guests[0]
if g.OK || g.LastOKAt != "" || !strings.Contains(g.Error, "exit status 255") || !strings.Contains(g.Error, "mount busy") {
t.Fatalf("failed result not recorded as a failure: %+v", g)
}
// A success later keeps a clean record.
r.err = nil
r.out = measuredOut
now = at(14, 10, 10)
tr.Pass(context.Background())
if g := tr.GuestDiskTrimStatus(context.Background()).Guests[0]; !g.OK || g.Error != "" || g.BytesTrimmed != measuredBytes {
t.Fatalf("success after failure: %+v", g)
}
}
func TestOnlyRunningOwnedGuestsAndAFailedListActsOnNothing(t *testing.T) {
now := at(7, 10, 30)
r := &fakeRunner{out: measuredOut}
g := fakeGuests{g: []proxmox.Guest{{VMID: 9201, Status: "stopped"}, {VMID: 9202, Status: "running"}}}
tr, _, _ := newT(t, r, g, &backup.InFlight{}, &now)
tr.Pass(context.Background())
if want := [][]string{{"pct", "fstrim", "9202"}}; !reflect.DeepEqual(r.calls, want) {
t.Fatalf("calls = %q, want only the running guest", r.calls)
}
r2 := &fakeRunner{out: measuredOut}
tr2, logs, _ := newT(t, r2, fakeGuests{err: errors.New("pool read 403")}, &backup.InFlight{}, &now)
tr2.Pass(context.Background())
if len(r2.calls) != 0 || !strings.Contains(logs.String(), "owned-guest list unavailable") {
t.Fatalf("a failed ownership read must act on nothing: calls %q", r2.calls)
}
}
func TestReportJSONShape(t *testing.T) {
now := at(7, 10, 30)
tr, _, _ := newT(t, &fakeRunner{out: measuredOut}, running(9201), &backup.InFlight{}, &now)
tr.Pass(context.Background())
b, err := json.Marshal(tr.GuestDiskTrimStatus(context.Background()))
if err != nil {
t.Fatal(err)
}
for _, k := range []string{`"schedule":`, `"guests":[{"vmid":9201`, `"last_attempt_at":"2026-10-07T08:30:00Z"`, `"ok":true`,
`"bytes_trimmed":90143313920`, `"mounts":2`, `"duration_seconds":`, `"last_ok_at":"2026-10-07T08:30:00Z"`} {
if !strings.Contains(string(b), k) {
t.Errorf("report JSON lacks %s: %s", k, b)
}
}
if strings.Contains(string(b), `"error"`) {
t.Errorf("an ok result must omit error: %s", b)
}
}
+16 -25
View File
@@ -38,35 +38,26 @@ const snippetBody = `#!/bin/sh
exit 0
`
// InstallSnippet writes the pre-start hook wrapper into the PVE snippets dir (idempotent, root-owned,
// executable). The agent runs as a non-root service user, so it writes an agent-writable temp file then
// `install`s it host-root (same pattern as the bootstrap mount + dnsmasq drop-ins). Safe to call repeatedly.
// The temp file is a RANDOM-named os.CreateTemp (audit B1): a fixed, predictable /tmp name could be
// pre-created by another local user and rewritten between our write and root's install (TOCTOU into a
// root-executed hookscript). The final mode comes from `install -m`, so the 0600 temp is fine.
func InstallSnippet(ctx context.Context, runner proxmox.Runner) error {
f, err := os.CreateTemp("", "felhom-guest-hook-*.sh")
// SnippetReady (R-861, agent v0.146.0) reports whether the pre-start hook is in place: a regular file at path whose
// content is exactly snippetBody. The hook is a FIXED, ROOT-OWNED file that arrives with the signed config bundle
// (configs/felhom-guest-hook.sh, pinned byte-identical by TestSnippetEqualsTheBundle). The agent no longer installs it:
// until v0.146.0 it `install`ed it from /tmp, and Proxmox runs a hookscript as root at every guest start — so the
// install grant was a root shell for a compromised agent. A missing or different hook is an error the caller logs; it
// then does NOT register the hook (a guest whose hookscript is missing does not start).
func SnippetReady(path string) error {
fi, err := os.Lstat(path)
if err != nil {
return fmt.Errorf("guesthook: create temp snippet: %w", err)
return fmt.Errorf("guesthook: %s is missing — it arrives with the signed config bundle (agent_config_update): %w", path, err)
}
tmp := f.Name()
defer os.Remove(tmp)
if _, err := f.WriteString(snippetBody); err != nil {
f.Close()
return fmt.Errorf("guesthook: write temp snippet: %w", err)
if !fi.Mode().IsRegular() {
return fmt.Errorf("guesthook: %s is not a regular file", path)
}
if err := f.Close(); err != nil {
return fmt.Errorf("guesthook: close temp snippet: %w", err)
b, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("guesthook: read %s: %w", path, err)
}
// Ensure the snippets dir exists FIRST (B2, DRILL-day0-cleanroom-2026-07-03): a fresh PVE has
// no /var/lib/vz/snippets, and `install` (without -D) won't create the parent — the whole
// hook install silently failed on a freshly-bootstrapped box. Fenced root op like the install
// itself; idempotent.
if _, stderr, err := runner.Run(ctx, "mkdir", "-p", SnippetDir); err != nil {
return fmt.Errorf("guesthook: ensure snippets dir %s: %w: %s", SnippetDir, err, string(stderr))
}
if _, stderr, err := runner.Run(ctx, "install", "-m", "0755", "--", tmp, SnippetPath); err != nil {
return fmt.Errorf("guesthook: install snippet to %s: %w: %s", SnippetPath, err, string(stderr))
if string(b) != snippetBody {
return fmt.Errorf("guesthook: %s differs from this agent's hook — the next config bundle replaces it", path)
}
return nil
}
+24 -90
View File
@@ -4,7 +4,7 @@ import (
"context"
"io"
"os"
"regexp"
"path/filepath"
"testing"
)
@@ -29,100 +29,34 @@ func (r *recordingRunner) RunStdin(ctx context.Context, _ io.Reader, name string
return r.Run(ctx, name, args...)
}
// TestInstallSnippet_RandomTempName is the audit-B1 negative test: the staged install SOURCE must be a
// RANDOM os.CreateTemp name (felhom-guest-hook-<random>.sh), never the fixed, pre-creatable
// /tmp/felhom-guest-hook.sh (a local TOCTOU into a root-executed hookscript), and two consecutive
// installs must stage through DIFFERENT paths.
func TestInstallSnippet_RandomTempName(t *testing.T) {
r := &recordingRunner{}
if err := InstallSnippet(context.Background(), r); err != nil {
t.Fatalf("InstallSnippet #1: %v", err)
// R-861 (agent v0.146.0): the hook file comes with the signed config bundle; the agent never installs it, only checks.
// RED-PROOF (audits/hub-safety-2026-10-05/partF/red-proof.txt): make SnippetReady accept any content → the
// "differs" case fails.
func TestSnippetReady(t *testing.T) {
d := t.TempDir()
p := filepath.Join(d, "felhom-guest-hook.sh")
if err := SnippetReady(p); err == nil {
t.Fatal("a missing hook read as ready — the guest would get a hookscript that does not exist")
}
if err := InstallSnippet(context.Background(), r); err != nil {
t.Fatalf("InstallSnippet #2: %v", err)
_ = os.WriteFile(p, []byte("#!/bin/sh\nid > /tmp/x\n"), 0o755)
if err := SnippetReady(p); err == nil {
t.Fatal("a hook with other content read as ready")
}
var installs [][]string
for _, call := range r.calls {
if call[0] == "install" {
installs = append(installs, call)
}
_ = os.WriteFile(p, []byte(snippetBody), 0o755)
if err := SnippetReady(p); err != nil {
t.Fatalf("the bundle's hook was not accepted: %v", err)
}
if len(installs) != 2 {
t.Fatalf("expected 2 install calls, got %d: %v", len(installs), r.calls)
}
randomName := regexp.MustCompile(`felhom-guest-hook-[^/\\]+\.sh$`)
fixedName := regexp.MustCompile(`felhom-guest-hook\.sh$`)
var srcs []string
for i, call := range installs {
// install -m 0755 -- <src> <dest>
if len(call) != 6 {
t.Fatalf("call %d: unexpected vector %v", i, call)
}
src, dest := call[4], call[5]
if dest != SnippetPath {
t.Errorf("call %d: dest = %q, want %q", i, dest, SnippetPath)
}
if !randomName.MatchString(src) {
t.Errorf("call %d: source %q does not match the random felhom-guest-hook-*.sh pattern", i, src)
}
if fixedName.MatchString(src) {
t.Errorf("call %d: source %q is the FIXED predictable temp name (B1 TOCTOU)", i, src)
}
srcs = append(srcs, src)
}
if srcs[0] == srcs[1] {
t.Errorf("two consecutive installs staged through the SAME source path %q — must be random per call", srcs[0])
}
// Non-hollow: the staged file must actually carry the snippet body at install time.
for i, c := range r.srcContent {
if c != snippetBody {
t.Errorf("call %d: staged content is not the snippet body (got %d bytes)", i, len(c))
}
}
// And the temp is cleaned up after.
for _, src := range srcs {
if _, err := os.Stat(src); err == nil {
t.Errorf("staged temp %q left behind (defer os.Remove missing)", src)
}
l := filepath.Join(d, "link.sh")
_ = os.Symlink(p, l)
if err := SnippetReady(l); err == nil {
t.Fatal("a symlink read as the hook")
}
}
// B2 Scenario D (DRILL-day0-cleanroom-2026-07-03): on a fresh PVE, /var/lib/vz/snippets does not
// exist and `install` (no -D) cannot create it — the drill saw
// `install: cannot create regular file … No such file or directory` and the guest silently got no
// pre-start self-heal hook. InstallSnippet must therefore issue a `mkdir -p <SnippetDir>` fenced op
// BEFORE the `install` op. Pre-fix wrong outcome: no mkdir call at all — only the doomed install.
func TestInstallSnippet_EnsuresSnippetsDirFirst(t *testing.T) {
r := &recordingRunner{}
if err := InstallSnippet(context.Background(), r); err != nil {
t.Fatalf("InstallSnippet: %v", err)
}
mkdirIdx, installIdx := -1, -1
for i, call := range r.calls {
switch call[0] {
case "mkdir":
if mkdirIdx == -1 {
mkdirIdx = i
want := []string{"mkdir", "-p", SnippetDir}
if len(call) != 3 || call[1] != want[1] || call[2] != want[2] {
t.Errorf("mkdir vector = %v, want %v (the sudoers fence matches exactly this argv)", call, want)
}
}
case "install":
if installIdx == -1 {
installIdx = i
}
}
}
if mkdirIdx == -1 {
t.Fatalf("no `mkdir -p %s` op issued — on a fresh box the snippet install fails ENOENT (B2); calls: %v", SnippetDir, r.calls)
}
if installIdx == -1 {
t.Fatalf("no install op issued; calls: %v", r.calls)
}
if mkdirIdx > installIdx {
t.Fatalf("mkdir (call %d) must PRECEDE install (call %d) — order: %v", mkdirIdx, installIdx, r.calls)
// The bundle's copy is byte-identical to the body the agent checks against.
func TestSnippetEqualsTheBundle(t *testing.T) {
got, err := os.ReadFile(filepath.Join("..", "..", "configs", "felhom-guest-hook.sh"))
if err != nil || string(got) != snippetBody {
t.Fatalf("configs/felhom-guest-hook.sh differs from snippetBody (err %v)", err)
}
}
+25
View File
@@ -0,0 +1,25 @@
package hub
import (
"os"
"path/filepath"
"testing"
)
// R-840: the agent reports the bundle record itself — "none" when no bundle ever reached the box (so an old wrapper
// cannot hide that), "unknown" when it cannot be read, else the version and sha the root wrapper recorded.
func TestReadBundleRecord(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "config-bundle.json")
if got := string(readBundleRecord(p)); got != `{"version":"none"}` {
t.Fatalf("absent: %s", got)
}
os.WriteFile(p, []byte("{broken"), 0o644)
if got := string(readBundleRecord(p)); got != `{"version":"unknown"}` {
t.Fatalf("broken: %s", got)
}
os.WriteFile(p, []byte(`{"format":1,"agent_version":"0.143.0","bundle_sha256":"abc","installed_at":"2026-10-04T20:00:00Z","authority":"signed","files":{"/x":"y"}}`), 0o644)
if got := string(readBundleRecord(p)); got != `{"authority":"signed","bundle_sha256":"abc","installed_at":"2026-10-04T20:00:00Z","version":"0.143.0"}` {
t.Fatalf("record: %s", got)
}
}
+24
View File
@@ -425,3 +425,27 @@ func (c *Client) FetchRetainedIdentityEscrow(ctx context.Context) (*RetainedEscr
}
return &out, nil
}
// PostOSReport sends the OS-update leg's report after every run (hub v0.130.0): POST /api/v1/hosts/{id}/os-report.
// Per-host key, self-scoped on the hub. Errors are typed like RegisterWG's and never include the bearer.
func (c *Client) PostOSReport(ctx context.Context, body []byte) error {
if c.hostID == "" {
return fmt.Errorf("hub: PostOSReport requires a configured host_id")
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/api/v1/hosts/"+c.hostID+"/os-report", bytes.NewReader(body))
if err != nil {
return fmt.Errorf("hub: building os-report request: %w", err)
}
req.Header.Set("Authorization", "Bearer "+c.apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := c.hc.Do(req)
if err != nil {
return &TransportError{Err: err}
}
defer resp.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
}
return nil
}
+97 -31
View File
@@ -2,45 +2,111 @@ package hub
import (
"context"
"os/exec"
"fmt"
"strings"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// CloudflaredProber reports the cloudflared tunnel service health. It is a
// READ-ONLY probe: the agent does NOT manage or restart cloudflared in this slice
// (that is the tunnel-management slice — this is the seam for it). Injectable so
// tests use a fake and never exec.
// Tunnel states the agent reports (R-841, agent v0.141.0). THREE, never two: a probe that could not ask is
// `unknown`, which the hub never shows as up or down and never alarms on (R-96 rule 3).
const (
TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED
TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which)
TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting)
)
// CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec.
type CloudflaredProber interface {
// Status returns one of: "active" | "inactive" | "failed" | "unknown".
Status(ctx context.Context) (string, error)
// Status returns one of the Tunnel* states and a short detail (why not_running / why unknown).
Status(ctx context.Context) (status, detail string)
}
// SystemctlProber runs `systemctl is-active cloudflared`. This is NOT a Privileged
// (root-CLI) op — `is-active` is non-root readable and is not one of the three
// proven root exceptions, so it does not go through internal/proxmox.Privileged.
type SystemctlProber struct {
Unit string // defaults to "cloudflared"
// GuestTunnelProber reads the REAL tunnel: the `cloudflared` container in the box's own customer guest.
//
// Before v0.141.0 the agent ran `systemctl is-active cloudflared` on the HOST — a unit that does not exist (cloudflared
// is a guest container, `11-os-updates.md` C8), so every box reported `inactive` (R-841).
//
// It uses ONLY the existing sudoers line `pct exec [0-9]* -- docker inspect -f *` (03 §3): the container's state, exit
// code and Docker health status. The health status comes from the compose health check controller v0.292.0 adds
// (`cloudflared tunnel --metrics localhost:20241 ready` → /ready: 200 only with ≥ 1 connection). Measured 2026-10-04:
// with a wrong token the container stays "running" while /ready answers 503 — so the container state alone would lie.
// A container with no health check (an older controller) is judged on its state alone, and Detail says so.
type GuestTunnelProber struct {
Runner proxmox.Runner
// Guests returns the box's customer guest vmids (running pool guests that bind /mnt/felhom-drives).
Guests func(ctx context.Context) ([]int, error)
}
// Status maps `systemctl is-active` output to the report vocabulary. systemctl
// exits non-zero for inactive/failed, so the output string is authoritative over
// the exit code; any exec error (binary missing, etc.) maps to "unknown".
func (p SystemctlProber) Status(ctx context.Context) (string, error) {
unit := p.Unit
if unit == "" {
unit = "cloudflared"
const tunnelInspect = `{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}`
// Status probes every customer guest and reports the worst state (normally there is exactly one guest).
func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
if p.Runner == nil || p.Guests == nil {
return TunnelUnknown, "no probe wired"
}
out, _ := exec.CommandContext(ctx, "systemctl", "is-active", unit).Output()
switch strings.TrimSpace(string(out)) {
case "active":
return "active", nil
case "failed":
return "failed", nil
case "inactive", "deactivating", "activating":
return "inactive", nil
case "":
return "unknown", nil // no output → systemctl/exec problem
default:
return "unknown", nil
vmids, err := p.Guests(ctx)
if err != nil {
return TunnelUnknown, "could not list the customer guest: " + err.Error()
}
if len(vmids) == 0 {
return TunnelUnknown, "no running customer guest"
}
worst, wdetail := "", ""
rank := map[string]int{TunnelRunning: 0, TunnelUnknown: 1, TunnelNotRunning: 2}
for _, v := range vmids {
out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared")
st, d := ClassifyTunnel(string(out), string(errOut), err)
if len(vmids) > 1 {
d = fmt.Sprintf("guest %d: %s", v, d)
}
if worst == "" || rank[st] > rank[worst] {
worst, wdetail = st, d
}
}
return worst, wdetail
}
// ClassifyTunnel maps one `docker inspect` answer to a state. Pure; pinned by TestClassifyTunnel.
func ClassifyTunnel(stdout, stderr string, err error) (string, string) {
out := strings.TrimSpace(stdout)
if err != nil || out == "" {
if strings.Contains(stderr, "No such object") || strings.Contains(stderr, "No such container") {
return TunnelNotRunning, "no cloudflared container in the guest"
}
return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err)
}
parts := strings.Split(out, "|")
if len(parts) != 3 {
return TunnelUnknown, "unreadable docker answer: " + out
}
state, code, health := parts[0], parts[1], parts[2]
if state != "running" {
return TunnelNotRunning, fmt.Sprintf("container %s, exit code %s", state, code)
}
switch health {
case "healthy":
return TunnelRunning, "connected"
case "unhealthy":
return TunnelNotRunning, "container running but the tunnel is NOT connected (cloudflared /ready fails)"
case "starting":
return TunnelUnknown, "container running, readiness check still starting"
case "none":
return TunnelRunning, "container running (no readiness check on this controller — connection not checked)"
}
return TunnelUnknown, "unknown health state " + health
}
func firstLine(stderr string, err error) string {
s := strings.TrimSpace(stderr)
if i := strings.IndexByte(s, '\n'); i >= 0 {
s = s[:i]
}
if s == "" && err != nil {
s = err.Error()
}
if len(s) > 160 {
s = s[:160]
}
return s
}
+67
View File
@@ -0,0 +1,67 @@
package hub
import (
"context"
"errors"
"io"
"strings"
"testing"
)
// R-841: the three states from one `docker inspect` answer. Red-proof: map "unhealthy" to running (the container
// state alone — what a plain "is it running" probe would say) and the "running but not connected" case fails.
func TestClassifyTunnel(t *testing.T) {
cases := []struct {
name, out, errOut string
err error
want string
detail string
}{
{"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"},
{"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"},
{"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelNotRunning, "no cloudflared container"},
{"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"},
{"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"},
{"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"},
{"sudo refused", "", "sudo: a password is required", errors.New("exit status 1"), TunnelUnknown, "could not ask"},
}
for _, c := range cases {
st, d := ClassifyTunnel(c.out, c.errOut, c.err)
if st != c.want || !strings.Contains(d, c.detail) {
t.Errorf("%s: got %q (%s), want %q (…%s…)", c.name, st, d, c.want, c.detail)
}
}
}
type tunnelRunner struct {
calls []string
out map[string]string
}
func (r *tunnelRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
line := name + " " + strings.Join(args, " ")
r.calls = append(r.calls, line)
return []byte(r.out[args[1]]), nil, nil
}
func (r *tunnelRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
// The probe uses EXACTLY the existing sudoers shape `pct exec <vmid> -- docker inspect -f <tmpl> cloudflared`, and
// with no customer guest it is unknown, never down.
func TestGuestTunnelProber(t *testing.T) {
r := &tunnelRunner{out: map[string]string{"9201": "running|0|healthy"}}
p := GuestTunnelProber{Runner: r, Guests: func(context.Context) ([]int, error) { return []int{9201}, nil }}
if st, d := p.Status(context.Background()); st != TunnelRunning || d != "connected" {
t.Fatalf("got %q %q", st, d)
}
want := "/usr/sbin/pct exec 9201 -- docker inspect -f " + tunnelInspect + " cloudflared"
if len(r.calls) != 1 || r.calls[0] != want {
t.Fatalf("command = %q, want %q", r.calls, want)
}
none := GuestTunnelProber{Runner: r, Guests: func(context.Context) ([]int, error) { return nil, nil }}
if st, _ := none.Status(context.Background()); st != TunnelUnknown {
t.Fatalf("no guest → %q, want unknown", st)
}
}
+144 -33
View File
@@ -4,10 +4,13 @@ import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"log/slog"
"os"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
@@ -87,32 +90,42 @@ type GuestNetReporter interface {
GuestNetStatus(ctx context.Context) *GuestNetStatus
}
// GuestDiskTrimReporter is the R-444 seam the weekly trim job plugs into (same consumer-side pattern — hub does not
// import fstrim). nil (feature not wired) → no guest_disk_trim stanza.
type GuestDiskTrimReporter interface {
GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus
}
// Collector builds a HostReport from read-only sources. All deps are behind narrow
// interfaces for unit testing.
type Collector struct {
px proxmoxReader
cf CloudflaredProber
storage StorageObserver
backups BackupReporter
restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter
pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
hostID string
agentVersion string
logger *slog.Logger
now func() time.Time
px proxmoxReader
cf CloudflaredProber
storage StorageObserver
backups BackupReporter
restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter
pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
system SystemReporter // R-852: the box versions (nil → API fields only)
bundleRecordPath string // R-840: test seam; "" = BundleRecordPath
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
hostID string
agentVersion string
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
logger *slog.Logger
now func() time.Time
}
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
@@ -131,6 +144,7 @@ func NewCollector(px proxmoxReader, cf CloudflaredProber, storage StorageObserve
temp: SysfsTempReader{}, // slice 9: real sysfs reader by default; tests inject a fake
hostID: hostID,
agentVersion: agentVersion,
selfSHA: runningBinarySHA256,
logger: logger,
now: func() time.Time { return time.Now().UTC() },
}
@@ -214,6 +228,12 @@ func (c *Collector) SetGuestNetReporter(g GuestNetReporter) *Collector {
return c
}
// SetGuestDiskTrimReporter wires the R-444 weekly trim job as a report source (nil-safe → stanza omitted).
func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector {
c.diskTrim = r
return c
}
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
type SelfUpdateReporter interface {
@@ -248,6 +268,69 @@ type OOBReporter interface {
}
// SetOOBReporter wires the operator-access health source (H1; nil-safe → stanza omitted).
// SystemReporter reads the box's versions (R-852): the customer guest's vmid and the wrapper's raw facts.
type SystemReporter interface {
SystemFacts(ctx context.Context) (vmid int, facts json.RawMessage, err error)
}
// SetSystemReporter wires the facts read (agent v0.142.0). Without it the stanza carries the Proxmox API fields only.
func (c *Collector) SetSystemReporter(r SystemReporter) *Collector {
c.system = r
return c
}
// BundleRecordPath is the root-owned record felhom-os-apply writes after a config bundle installs (R-840).
const BundleRecordPath = "/etc/felhom/config-bundle.json"
// readBundleRecord returns the record's summary: version/sha/installed_at, "none" when the file is absent, "unknown"
// when it cannot be read or parsed (never a guess).
func readBundleRecord(path string) json.RawMessage {
if path == "" {
path = BundleRecordPath
}
b, err := os.ReadFile(path)
if os.IsNotExist(err) {
return json.RawMessage(`{"version":"none"}`)
}
var rec struct {
AgentVersion string `json:"agent_version"`
BundleSHA256 string `json:"bundle_sha256"`
InstalledAt string `json:"installed_at"`
Authority string `json:"authority"`
}
if err != nil || json.Unmarshal(b, &rec) != nil || rec.AgentVersion == "" {
return json.RawMessage(`{"version":"unknown"}`)
}
out, _ := json.Marshal(map[string]string{"version": rec.AgentVersion, "bundle_sha256": rec.BundleSHA256,
"installed_at": rec.InstalledAt, "authority": rec.Authority})
return out
}
func unknownIfEmpty(s string) string {
if strings.TrimSpace(s) == "" {
return "unknown"
}
return s
}
// systemInfo builds the `system` stanza. Never fatal: a failed facts read is FactsError, the API fields stay.
func (c *Collector) systemInfo(ctx context.Context, ns proxmox.NodeStatus) *SystemInfo {
si := &SystemInfo{PVEVersion: unknownIfEmpty(ns.PVEVersion), KernelVersion: unknownIfEmpty(ns.KVersion),
ReadAt: c.now().Format(time.RFC3339)}
si.ConfigBundle = readBundleRecord(c.bundleRecordPath)
if c.system == nil {
si.FactsError = "no facts reader wired"
return si
}
vmid, f, err := c.system.SystemFacts(ctx)
si.VMID, si.Facts = vmid, f
if err != nil {
si.FactsError = err.Error()
c.logger.Debug("hub: system facts unavailable", "err", err)
}
return si
}
func (c *Collector) SetOOBReporter(o OOBReporter) *Collector {
c.oob = o
return c
@@ -270,6 +353,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
HostID: c.hostID,
ReportedAt: c.now().Format(time.RFC3339),
AgentVersion: c.agentVersion,
AgentSHA256: c.agentSHA256(),
Host: host,
Guests: c.collectGuests(ctx),
// storage_targets populated this slice (slice 5) via the observer; the rest stay
@@ -280,10 +364,11 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
PBSSnapshots: c.collectPBSSnapshots(ctx),
AuditTail: []AuditEntry{},
Cloudflared: Cloudflared{Status: c.cloudflaredStatus(ctx)},
Cloudflared: c.cloudflared(ctx),
Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP,
Addresses: c.collectAddresses(),
System: c.systemInfo(ctx, ns),
}
// DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads).
// Secret-free by construction (identifiers/intents/sizes/coordinates only).
@@ -305,6 +390,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.guestNet != nil {
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
}
// R-444: the last weekly trim result per guest (nil reporter = not wired → stanza omitted).
if c.diskTrim != nil {
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
}
// D1: agent self-update pending status (nil reporter → pending=false, the steady state).
if c.selfUpdate != nil {
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()
@@ -363,8 +452,28 @@ const pbsWrapperPath = "/usr/local/sbin/felhom-pbs-apply"
// unreadable file yields "", which the hub reads as UNKNOWN rather than as drift — a host that
// legitimately has no DR wrapper must not light up amber. The file is 0755, so no privilege is
// needed to read it.
func pbsWrapperSHA256() string {
f, err := os.Open(pbsWrapperPath)
func pbsWrapperSHA256() string { return fileSHA256(pbsWrapperPath) }
// selfExePath is the running binary as the kernel holds it. /proc/self/exe, not the installed path:
// after an A/B flip the file at /usr/local/bin/felhom-agent may already be the NEXT binary while this
// process still runs the old one, and the report must describe what runs (R-349). Test seam.
var selfExePath = "/proc/self/exe"
// runningBinarySHA256 hashes the running binary ONCE per process — the bytes cannot change under a
// running process, and re-hashing ~20 MB every report cycle buys nothing. A failed read is cached as
// "" (UNKNOWN); it never fails the report.
var runningBinarySHA256 = sync.OnceValue(func() string { return fileSHA256(selfExePath) })
func (c *Collector) agentSHA256() string {
if c.selfSHA == nil {
return ""
}
return c.selfSHA()
}
// fileSHA256 is the hex sha256 of a file's bytes, or "" when it cannot be read.
func fileSHA256(path string) string {
f, err := os.Open(path)
if err != nil {
return ""
}
@@ -555,16 +664,18 @@ func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot {
return []PBSSnapshot{}
}
func (c *Collector) cloudflaredStatus(ctx context.Context) string {
func (c *Collector) cloudflared(ctx context.Context) Cloudflared {
if c.cf == nil {
return "unknown"
return Cloudflared{Status: TunnelUnknown, Detail: "no probe wired"}
}
st, err := c.cf.Status(ctx)
if err != nil || st == "" {
c.logger.Warn("hub: cloudflared probe failed", "err", err)
return "unknown"
st, d := c.cf.Status(ctx)
if st == "" {
st = TunnelUnknown
}
return st
if st == TunnelUnknown {
c.logger.Debug("hub: tunnel probe could not decide", "detail", d)
}
return Cloudflared{Status: st, Detail: d}
}
func percent(used, total int64) float64 {
+64
View File
@@ -0,0 +1,64 @@
package hub
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// R-349: the report carries the sha256 of the binary that is RUNNING, so the hub can tell a
// hand-built proof binary from the vouched artifact of the same version string. The consequence
// asserted: the wire field equals the hash of this very test binary's bytes (read independently via
// os.Executable, a different channel from /proc/self/exe), and it is on the wire as agent_sha256.
func TestCollect_AgentSHA256IsTheRunningBinary(t *testing.T) {
exe, err := os.Executable()
if err != nil {
t.Skipf("os.Executable: %v", err)
}
raw, err := os.ReadFile(exe)
if err != nil {
t.Fatalf("read own binary: %v", err)
}
sum := sha256.Sum256(raw)
want := hex.EncodeToString(sum[:])
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "running"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.AgentSHA256 != want {
t.Fatalf("agent_sha256 = %q, want the running binary's %q", r.AgentSHA256, want)
}
b, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(b), `"agent_sha256":"`+want+`"`) {
t.Fatalf("agent_sha256 not on the wire: %s", b)
}
}
// An unreadable binary is UNKNOWN (empty, omitted) — never a made-up hash, never a failed report.
func TestFileSHA256_UnreadableIsEmpty(t *testing.T) {
if got := fileSHA256(filepath.Join(t.TempDir(), "absent")); got != "" {
t.Fatalf("absent file hashed to %q, want empty", got)
}
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "running"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
c.selfSHA = func() string { return "" }
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect must not fail on an unreadable binary: %v", err)
}
b, _ := json.Marshal(r)
if strings.Contains(string(b), "agent_sha256") {
t.Fatalf("empty agent_sha256 must be omitted: %s", b)
}
}
+54
View File
@@ -0,0 +1,54 @@
package hub
import (
"context"
"encoding/json"
"testing"
)
// R-444: the guest_disk_trim stanza must reach a report built through the PRODUCTION collect path, be absent from
// the wire when the job is not wired, and carry the keys the hub's System page reads.
type fakeDiskTrim struct{ st *GuestDiskTrimStatus }
func (f fakeDiskTrim) GuestDiskTrimStatus(context.Context) *GuestDiskTrimStatus { return f.st }
func TestCollect_GuestDiskTrim(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.150.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
b, _ := json.Marshal(r)
var m map[string]any
_ = json.Unmarshal(b, &m)
if _, ok := m["guest_disk_trim"]; ok {
t.Fatalf("guest_disk_trim on the wire with no reporter wired: %s", b)
}
c.SetGuestDiskTrimReporter(fakeDiskTrim{st: &GuestDiskTrimStatus{Schedule: "weekly", Guests: []GuestDiskTrim{{
VMID: 9201, LastAttemptAt: "2026-10-07T08:30:00Z", OK: true, BytesTrimmed: 90143313920, Mounts: 2,
DurationSeconds: 24.4, LastOKAt: "2026-10-07T08:30:00Z",
}}}})
r, err = c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
b, _ = json.Marshal(r)
m = nil
_ = json.Unmarshal(b, &m)
dt, ok := m["guest_disk_trim"].(map[string]any)
if !ok || dt["schedule"] != "weekly" {
t.Fatalf("guest_disk_trim missing or wrong on the wire: %s", b)
}
g := dt["guests"].([]any)[0].(map[string]any)
for _, k := range []string{"vmid", "last_attempt_at", "ok", "bytes_trimmed", "mounts", "duration_seconds", "last_ok_at"} {
if _, ok := g[k]; !ok {
t.Fatalf("guest_disk_trim.guests[0] lacks %q: %v", k, g)
}
}
if g["bytes_trimmed"] != float64(90143313920) || g["ok"] != true {
t.Fatalf("values did not survive the round trip: %v", g)
}
}
+2 -2
View File
@@ -16,7 +16,7 @@ func (f fakeGuestNet) GuestNetStatus(context.Context) *GuestNetStatus { return f
func TestCollect_GuestNetOmittedWhenReporterNil(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
@@ -38,7 +38,7 @@ func TestCollect_GuestNetOmittedWhenReporterNil(t *testing.T) {
func TestCollect_GuestNetPopulatedWhenWired(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
c.SetGuestNetReporter(fakeGuestNet{st: &GuestNetStatus{
CheckedAt: "2026-07-21T10:00:00Z",
Guests: []GuestNetGuest{{
+4 -4
View File
@@ -12,7 +12,7 @@ func (f fakeMgmtPlane) MgmtPlaneStatus(context.Context) *MgmtPlaneStatus { retur
func TestCollect_MgmtPlaneOmittedWhenReporterNil(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
@@ -24,7 +24,7 @@ func TestCollect_MgmtPlaneOmittedWhenReporterNil(t *testing.T) {
func TestCollect_MgmtPlanePopulatedWhenWired(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
c.SetMgmtPlaneReporter(fakeMgmtPlane{st: &MgmtPlaneStatus{
PrivsepDirOK: true, SshdReachable: true, HealedRecently: true, PrivsepHealedAt: "2026-07-05T16:42:17Z",
}})
@@ -47,7 +47,7 @@ func (f fakeOOB) OOBStatus(context.Context) *OOBStatus { return f.st }
func TestCollect_OOBOmittedWhenNil(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
r, _ := c.Collect(context.Background())
if r.OOB != nil {
t.Fatalf("no reporter → oob omitted, got %+v", r.OOB)
@@ -56,7 +56,7 @@ func TestCollect_OOBOmittedWhenNil(t *testing.T) {
func TestCollect_OOBPopulatedWhenWired(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
c.SetOOBReporter(fakeOOB{st: &OOBStatus{FelhomSshdActive: true, FelhomSshdPort: 8822, Reachable: true}})
r, _ := c.Collect(context.Background())
if r.OOB == nil || r.OOB.FelhomSshdPort != 8822 || !r.OOB.Reachable {
+9 -9
View File
@@ -33,7 +33,7 @@ func TestCollect_StorageTargetsFromObserver(t *testing.T) {
obs := fakeObserver{targets: []StorageTarget{
{Name: "local-lvm", Type: StorageTypeLVMThin, State: StorageStateAttached, Reachable: true},
}}
c := NewCollector(px, fakeProber{status: "active"}, obs, nil, nil, nil, "h", "0.5.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, obs, nil, nil, nil, "h", "0.5.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
@@ -45,7 +45,7 @@ func TestCollect_StorageTargetsFromObserver(t *testing.T) {
func TestCollect_StorageObserverErrorDegradesToEmpty(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{err: errors.New("proxmox down")}, nil, nil, nil, "h", "0.5.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{err: errors.New("proxmox down")}, nil, nil, nil, "h", "0.5.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("a storage observe error must not sink the heartbeat: %v", err)
@@ -64,7 +64,7 @@ func TestCollect_HostAndGuests(t *testing.T) {
},
cfg: map[int]proxmox.GuestConfig{100: {Cores: 2, Memory: 2048}},
}
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "demo-host-01", "0.3.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "demo-host-01", "0.3.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
@@ -88,7 +88,7 @@ func TestCollect_HostAndGuests(t *testing.T) {
if g.Spec.Cores != 2 || g.Spec.MemoryBytes != 2147483648 || g.Spec.DiskBytes != 21474836480 {
t.Errorf("spec = %+v", g.Spec)
}
if r.Cloudflared.Status != "active" {
if r.Cloudflared.Status != "running" || r.Cloudflared.Detail != "connected" {
t.Errorf("cloudflared = %q", r.Cloudflared.Status)
}
}
@@ -104,7 +104,7 @@ func TestCollect_GuestConfigFailureKeepsStatusOmitsSpec(t *testing.T) {
cfg: map[int]proxmox.GuestConfig{100: {Cores: 2}},
cfgErr: map[int]error{200: errors.New("config read failed")},
}
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.3.1", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.3.1", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("a per-guest failure must NOT fail the whole report: %v", err)
@@ -125,7 +125,7 @@ func TestCollect_GuestConfigFailureKeepsStatusOmitsSpec(t *testing.T) {
func TestCollect_NodeStatusFailureIsHardError(t *testing.T) {
px := &fakePx{node: "n", nsErr: errors.New("proxmox down")}
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
if _, err := c.Collect(context.Background()); err == nil {
t.Fatal("NodeStatus failure must be a hard error (no useful report)")
}
@@ -133,7 +133,7 @@ func TestCollect_NodeStatusFailureIsHardError(t *testing.T) {
func TestCollect_CloudflaredProbeErrorIsUnknown(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{err: errors.New("no systemctl")}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
c := NewCollector(px, fakeProber{status: "", detail: "could not ask"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("cloudflared failure must not be fatal: %v", err)
@@ -153,7 +153,7 @@ func TestCollect_LeafFingerprint(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
const fp = "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245"
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
c.SetLeafFingerprint(fp)
r, err := c.Collect(context.Background())
if err != nil {
@@ -164,7 +164,7 @@ func TestCollect_LeafFingerprint(t *testing.T) {
}
// Companion: no SetLeafFingerprint (local API disabled) → empty, never a fabricated value.
c2 := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
c2 := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
r2, _ := c2.Collect(context.Background())
if r2.LeafFingerprint != "" {
t.Fatalf("unset leaf_fingerprint = %q, want empty", r2.LeafFingerprint)
+9 -7
View File
@@ -54,11 +54,13 @@ const (
DRReasonNoPBSStorage = "no_pbs_storage_observed"
)
// PBSRootNamespace is how the recipe spells PBS's root namespace. The PBS API spells it as the EMPTY
// string (and `pct restore --ns root` would name a namespace that does not exist) — "root" is a display
// convention this wire has always used, kept here so the field's meaning did not change under R-106.
// Only a box with no `namespace` line in its pbs storage.cfg stanza ever emits it.
const PBSRootNamespace = "root"
// PBSRootNamespace is how the recipe spells PBS's root namespace: the EMPTY string, PBS's own spelling (R-124,
// agent v0.147.0). It used to be the display word "root", which no PBS namespace is named — an operator pasting it
// into `proxmox-backup-client … --ns root` during a real recovery got a failure. An empty namespace is ambiguous on
// its own, so READ IT WITH namespace_state: resolved + "" = the root namespace (pass no --ns, or --ns ""); unknown +
// "" = the agent could not tell. Only a box with no `namespace` line in its pbs storage.cfg stanza emits it.
// Pinned by TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown and TestR124_RootNamespaceOnTheWireIsPBSSpelling.
const PBSRootNamespace = ""
// DRRecipeHostHalf is the agent-emitted half (guest/drive/storage/PBS scaffolding). Derived entirely
// from facts the report already collects — no new privileged reads.
@@ -123,8 +125,8 @@ type DRPBSCoord struct {
RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token
// Namespace is the PBS namespace the restore targets, resolved from the pbs storage's storage.cfg
// stanza — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree
// with the backup that produced the snapshot. PBSRootNamespace when the box has no namespace
// configured; "" when NamespaceState is unknown.
// with the backup that produced the snapshot. PBSRootNamespace ("", PBS's spelling, R-124) when the box has no
// namespace configured; also "" when NamespaceState is unknown — consult NamespaceState.
//
// R-106: this used to come from the listed snapshot's own `ns`, which PBS does not echo per item once
// the request is already namespace-scoped via `?ns=` (internal/pbs/client.go). The field was
+41 -5
View File
@@ -63,8 +63,8 @@ func TestBuildDRRecipeHostHalf(t *testing.T) {
t.Error("felhom-flash (local-dir user-data drive) missing from drives")
}
// pbs: latest snapshot's coords + the pbs storage id as repo_id.
if h.PBS == nil || h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != "root" || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want repo felhom-pbs/root/9201", h.PBS)
if h.PBS == nil || h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != PBSRootNamespace || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want repo felhom-pbs, the root namespace (\"\", R-124), snapshot 9201", h.PBS)
}
}
@@ -252,7 +252,7 @@ func TestDRRecipe_PBSNamespaceIsThePerCustomerOne(t *testing.T) {
}
// TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown: a box with a pbs storage and NO namespace line is
// genuinely in the root namespace. That is an answer, not a gap — it must read resolved/"root", so the
// genuinely in the root namespace. That is an answer, not a gap — it must read resolved/"" (PBS's spelling, R-124), so the
// honest root case is never confused with "I could not tell".
func TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
@@ -384,7 +384,7 @@ func TestCollectDRRecipe_ProductionPath(t *testing.T) {
obs := fakeObserver{targets: capturedDemoFelhomTargets()}
pbsRep := fakePBSReporter{snaps: capturedDemoFelhomSnapshots()}
c := NewCollector(px, fakeProber{status: "active"}, obs, nil, nil, pbsRep, "h", "0.118.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, obs, nil, nil, pbsRep, "h", "0.118.0", quietLogger())
c.SetBackupTargetResolver(func() ConfiguredBackupTarget {
return ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true}
})
@@ -409,7 +409,7 @@ func TestCollectDRRecipe_ProductionPath(t *testing.T) {
// test that would have caught shipping the seam without wiring it.
func TestCollectDRRecipe_UnwiredSeamReportsUnknown(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{targets: capturedDemoFelhomTargets()},
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{targets: capturedDemoFelhomTargets()},
nil, nil, nil, "h", "0.118.0", quietLogger())
r, err := c.Collect(context.Background())
@@ -453,3 +453,39 @@ func assertNoSecretKeys(t *testing.T, jsonBytes []byte) {
}
walk("<root>", v)
}
// R-124: on the WIRE the root namespace is PBS's own spelling — an empty string, present (not omitted), beside
// namespace_state "resolved". The display word "root" names no PBS namespace, and `--ns root` fails in a recovery.
// RED-PROOF: set PBSRootNamespace back to "root" → this test fails.
func TestR124_RootNamespaceOnTheWireIsPBSSpelling(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup", PBSNamespace: ""}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
b, err := json.Marshal(h.PBS)
if err != nil {
t.Fatal(err)
}
var m map[string]any
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
ns, present := m["namespace"]
if !present {
t.Fatalf("namespace key missing from %s — an omitted key reads as 'unknown', not 'root'", b)
}
if ns != "" {
t.Fatalf("root namespace on the wire = %q, want \"\" (PBS's spelling; no namespace is named %q)", ns, ns)
}
if m["namespace_state"] != DRStateResolved {
t.Fatalf("namespace_state = %v, want %q beside the empty root namespace", m["namespace_state"], DRStateResolved)
}
// A configured namespace still passes through unchanged.
h2 := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup", PBSNamespace: "demo-felhom"}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
if h2.PBS.Namespace != "demo-felhom" {
t.Fatalf("configured namespace = %q, want demo-felhom", h2.PBS.Namespace)
}
}
+4 -4
View File
@@ -16,7 +16,7 @@ func intp(v int) *int { return &v }
// HostMetricsNow returns a fresh host block with cpu% from NodeStatus and the temp from the reader.
func TestHostMetricsNow_PopulatesTemp(t *testing.T) {
px := &fakePx{node: "demo-felhom", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
SetTempReader(fakeTemp{c: intp(46)})
h, err := c.HostMetricsNow(context.Background())
if err != nil {
@@ -36,7 +36,7 @@ func TestHostMetricsNow_PopulatesTemp(t *testing.T) {
// A missing temp sensor gracefully nulls cpu_temp_c without failing the host read.
func TestHostMetricsNow_GracefulNullTemp(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
SetTempReader(fakeTemp{c: nil})
h, err := c.HostMetricsNow(context.Background())
if err != nil {
@@ -50,7 +50,7 @@ func TestHostMetricsNow_GracefulNullTemp(t *testing.T) {
// A NodeStatus failure is a hard error (no useful host view).
func TestHostMetricsNow_NodeStatusErrorIsHard(t *testing.T) {
px := &fakePx{node: "n", nsErr: errors.New("proxmox down")}
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger())
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger())
if _, err := c.HostMetricsNow(context.Background()); err == nil {
t.Fatal("NodeStatus failure must be a hard error")
}
@@ -59,7 +59,7 @@ func TestHostMetricsNow_NodeStatusErrorIsHard(t *testing.T) {
// Collect() (the hub report) also carries the temp now — the operator freebie.
func TestCollect_HostReportCarriesTemp(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
SetTempReader(fakeTemp{c: intp(51)})
r, err := c.Collect(context.Background())
if err != nil {
+2 -2
View File
@@ -56,7 +56,7 @@ func (f *fakePx) GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig
// fakeProber is a fake CloudflaredProber.
type fakeProber struct {
status string
err error
detail string
}
func (p fakeProber) Status(ctx context.Context) (string, error) { return p.status, p.err }
func (p fakeProber) Status(ctx context.Context) (string, string) { return p.status, p.detail }
+36
View File
@@ -0,0 +1,36 @@
package hub
import (
"encoding/json"
"os"
"testing"
)
// The os_update block is a cross-repo contract: testdata/desired-state-osupdate.golden.json is byte-identical with
// felhom.eu/hub/internal/api/testdata (the hub's TestOSUpdate_DesiredBlockMatchesTheGolden proves the hub SERVES
// it). Here: the agent DECODES every field. A renamed json tag on either side fails one of the two tests.
func TestOSUpdateGolden_Decodes(t *testing.T) {
raw, err := os.ReadFile("testdata/desired-state-osupdate.golden.json")
if err != nil {
t.Fatal(err)
}
var resp DesiredStateResponse
if err := json.Unmarshal(raw, &resp); err != nil {
t.Fatal(err)
}
o := resp.DesiredState.OSUpdate
if o == nil || o.Ring != 1 || !o.Enabled || o.Release == nil {
t.Fatalf("os_update = %+v", o)
}
r := o.Release
if r.ID != "os-guest-20261004-120000" || r.Snapshot != "20261004T120000Z" || len(r.Packages) != 2 ||
r.Packages[1].Name != "openssl" || r.Packages[1].Version != "3.5.7-1~deb13u3" || r.Packages[1].Origin != "Debian-Security" {
t.Fatalf("release = %+v", r)
}
// v0.141.0: the host layer's own approved set (`11` §8 step 3).
h := o.HostRelease
if h == nil || h.ID != "os-host-20261004-120000" || h.Snapshot != "20261004T120000Z" || len(h.Packages) != 1 ||
h.Packages[0].Name != "libssl3t64" || h.Packages[0].Origin != "Debian-Security" {
t.Fatalf("host_release = %+v", h)
}
}
+99 -2
View File
@@ -18,6 +18,16 @@ type HostReport struct {
HostID string `json:"host_id"` // echoes config.Hub.HostID
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
AgentVersion string `json:"agent_version"`
// AgentSHA256 is the sha256 of the binary this process is RUNNING (read through /proc/self/exe,
// once per process), R-349. The version string cannot tell a hand-built proof binary from the
// published, vouched artifact of the same version — same source, different bytes (`-trimpath
// -buildvcs=false` in release-agent.sh) — so self-update sees "already installed" and never
// corrects it. Reporting the bytes lets the hub compare against the vouched agent_sha256, the
// same mechanism host.wrapper_sha256 is for the PBS wrapper (R-50b(a)).
//
// Empty = unreadable, which the hub must treat as UNKNOWN, never as drift. Pinned by
// TestCollect_AgentSHA256IsTheRunningBinary.
AgentSHA256 string `json:"agent_sha256,omitempty"`
Host HostMetrics `json:"host"`
Guests []Guest `json:"guests"`
@@ -89,6 +99,12 @@ type HostReport struct {
// hub-schema change and are absent when the reporter is not wired.
MgmtPlane *MgmtPlaneStatus `json:"mgmt_plane,omitempty"`
// System is the box's versions for the hub's System page (agent v0.142.0, R-852, `09` decision 89): Proxmox and the
// running kernel from the Proxmox API, and the wrapper's read-only facts (host Debian, next-boot kernel, held
// packages, taint, the crash guard; guest Debian, Docker engine, containerd, live-restore). A value nobody could
// read is "unknown", never empty and never guessed. The hub v0.132.0 consumes it (hosts + System pages).
System *SystemInfo `json:"system,omitempty"`
// PBSDR is the PBS-DR-tier bridge status stanza (slice 2). Present only when the pbsdr
// consumer is wired. `consumed_failed` is the LOUD persistent state: the one-time token
// secret was consumed but the apply failed afterwards — the secret is burned, the bridge
@@ -108,6 +124,11 @@ type HostReport struct {
// on HostReport would have been the only report block named against that convention.
GuestNet *GuestNetStatus `json:"guest_net,omitempty"`
// GuestDiskTrim is the weekly guest disk trim stanza (R-444, `09` §3 decision 139): the schedule and, per owned
// guest, the LAST trim result as persisted by the agent (it survives a restart). Present only when the trim job is
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
// LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent
// mirror of the controller's report log_tails channel. Present ONLY on the heartbeat
// right after the control envelope requested it (log_tail_requested); consume-once on
@@ -192,6 +213,27 @@ type GuestNetGuest struct {
Message string `json:"message,omitempty"`
}
// GuestDiskTrimStatus is the R-444 weekly trim stanza. `schedule` is a plain description of when the job runs (local
// time of the host); `guests` holds one entry per owned guest that has had at least one trim attempt.
type GuestDiskTrimStatus struct {
Schedule string `json:"schedule"`
Guests []GuestDiskTrim `json:"guests,omitempty"`
}
// GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never
// read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is
// the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points.
type GuestDiskTrim struct {
VMID int `json:"vmid"`
LastAttemptAt string `json:"last_attempt_at"`
OK bool `json:"ok"`
BytesTrimmed int64 `json:"bytes_trimmed"`
Mounts int `json:"mounts"`
DurationSeconds float64 `json:"duration_seconds"`
LastOKAt string `json:"last_ok_at,omitempty"`
Error string `json:"error,omitempty"`
}
type PBSDRStatus struct {
State string `json:"state"`
StorageID string `json:"storage_id,omitempty"`
@@ -241,6 +283,20 @@ type WireguardStatus struct {
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
}
// SystemInfo is the `system` stanza (see HostReport.System).
type SystemInfo struct {
PVEVersion string `json:"pve_version"` // GET /nodes/{node}/status pveversion
KernelVersion string `json:"kernel_version"` // GET /nodes/{node}/status kversion
VMID int `json:"vmid,omitempty"` // the customer guest the facts read
Facts json.RawMessage `json:"facts,omitempty"`
FactsError string `json:"facts_error,omitempty"`
ReadAt string `json:"read_at"`
// ConfigBundle is the box's root-owned config bundle record (R-840, agent v0.143.0), read by the agent itself from
// /etc/felhom/config-bundle.json (0644): {"version":"none"} on a box no bundle reached, so an OLD wrapper cannot
// hide it. The wrapper's facts carry the same record plus the drift (files changed by hand).
ConfigBundle json.RawMessage `json:"config_bundle,omitempty"`
}
// HostMetrics is the host block, sourced from proxmox NodeStatus.
type HostMetrics struct {
Node string `json:"node"`
@@ -289,9 +345,10 @@ type GuestSpec struct {
DiskBytes int64 `json:"disk_bytes"`
}
// Cloudflared is the tunnel service health (read-only probe this slice).
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
type Cloudflared struct {
Status string `json:"status"` // active | inactive | failed | unknown
Status string `json:"status"` // running | not_running | unknown (TunnelRunning …)
Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected"
}
// The following element types are declared now so the empty collections above are
@@ -397,6 +454,17 @@ type SmartSummary struct {
ReallocatedSectors *int `json:"reallocated_sectors"`
PendingSectors *int `json:"pending_sectors"`
OfflineUncorrectable *int `json:"offline_uncorrectable"`
// R-330 (disk health Phase 2): three more SATA raw counters. omitempty + pointer: absent (an
// older agent, an NVMe/USB device, or a drive that does not report the attribute) is OMITTED —
// unknown, never a zero (S-39). Wire only: no verdict reads them yet.
// 187 Reported_Uncorrect — the failing drive's most telling counter (normalized 1 vs thresh 0,
// raw 1001) while SMART still said PASSED.
// 188 Command_Timeout — some vendors PACK several counters into the 48-bit raw value, so the
// number is carried as reported and must not be compared across vendors.
// 199 UDMA_CRC_Error_Count — cabling / link errors, not the medium.
ReportedUncorrect *int64 `json:"reported_uncorrect,omitempty"`
CommandTimeout *int64 `json:"command_timeout,omitempty"`
UDMACRCErrors *int64 `json:"udma_crc_errors,omitempty"`
// NVMe attributes.
CriticalWarning *int `json:"critical_warning"`
@@ -548,6 +616,35 @@ type WireDesiredState struct {
RestoreDirective *WireRestoreDirective `json:"restore_directive,omitempty"` // slice 10D (forward-compat)
Wireguard *WireWireguard `json:"wireguard,omitempty"` // S3 (doc 06 §3.2; golden-pinned)
PBSDR *WirePBSDR `json:"pbs_dr,omitempty"` // PBS DR tier (slice 2 consumer)
OSUpdate *WireOSUpdate `json:"os_update,omitempty"` // OS updates, guest fast lane (agent v0.140.0)
}
// WireOSUpdate is the hub-OWNED OS-update block (hub v0.130.0, `11-os-updates.md` §5.3), merged into the served
// document at read time. Ring 0 installs every pending Debian / Debian-Security fix; ring 1 installs exactly the
// newest approved release. Absent (older hub) → the agent treats the box as ring 1, ON, no release: it reports
// and installs nothing. Golden: testdata/desired-state-osupdate.golden.json (byte-identical with the hub's).
type WireOSUpdate struct {
Ring int `json:"ring"`
Enabled bool `json:"enabled"`
Release *WireOSRelease `json:"release,omitempty"`
// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
// approved for the guest is not approved for the host by that fact alone.
HostRelease *WireOSRelease `json:"host_release,omitempty"`
}
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
// for snapshot.debian.org when Debian has already replaced a version (decision 79).
type WireOSRelease struct {
ID string `json:"id"`
Snapshot string `json:"snapshot"`
Packages []WireOSPackage `json:"packages"`
}
// WireOSPackage is one approved name=version and its origin ("Debian" | "Debian-Security").
type WireOSPackage struct {
Name string `json:"name"`
Version string `json:"version"`
Origin string `json:"origin"`
}
// WirePBSDR is the hub's PBS-DR-tier descriptor (PBS DR slice 1, hub/internal/web/pbsdr.go
@@ -0,0 +1,24 @@
{
"generation": 1,
"desired_state": {
"os_update": {
"ring": 1,
"enabled": true,
"release": {
"id": "os-guest-20261004-120000",
"snapshot": "20261004T120000Z",
"packages": [
{"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"},
{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}
]
},
"host_release": {
"id": "os-host-20261004-120000",
"snapshot": "20261004T120000Z",
"packages": [
{"name": "libssl3t64", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}
]
}
}
}
}
+122
View File
@@ -0,0 +1,122 @@
package lanresolver
import (
"context"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"testing"
)
// recRunner records every privileged command EnsureDnsmasq would run and succeeds — nothing reaches
// apt, systemctl or the root checker.
type recRunner struct {
mu sync.Mutex
calls []string
}
func (r *recRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
r.calls = append(r.calls, strings.Join(append([]string{name}, args...), " "))
return nil, nil, nil
}
func (r *recRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func (r *recRunner) installed() bool {
for _, c := range r.calls {
if strings.HasPrefix(c, "apt-get install") && strings.HasSuffix(c, " dnsmasq") {
return true
}
}
return false
}
// fixtureRoot builds a fake host root holding exactly the given relative files and points the REAL
// probe at it for the test's duration.
func fixtureRoot(t *testing.T, files ...string) {
t.Helper()
root := t.TempDir()
for _, f := range files {
p := filepath.Join(root, f)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, nil, 0o644); err != nil {
t.Fatal(err)
}
}
prev := hostRoot
hostRoot = root
t.Cleanup(func() { hostRoot = prev })
}
func ensure(t *testing.T) *recRunner {
t.Helper()
r := &recRunner{}
m := NewManager(r, "192.0.2.10", []string{"1.1.1.1"}, slog.New(slog.NewTextHandler(io.Discard, nil)))
if err := m.EnsureDnsmasq(context.Background()); err != nil {
t.Fatalf("EnsureDnsmasq: %v", err)
}
return r
}
// R-317: a host with `dnsmasq-base` (the /usr/sbin/dnsmasq binary) but WITHOUT the `dnsmasq` package
// (the service unit) must get the package installed — else the following `systemctl enable --now
// dnsmasq` hits a unit that does not exist and LAN name resolution silently never comes up.
//
// RED-PROOF: probe "usr/sbin/dnsmasq" instead of the unit paths in dnsmasqUnitInstalled → this fails
// with "install was skipped".
func TestEnsureDnsmasq_BinaryWithoutUnitInstalls(t *testing.T) {
fixtureRoot(t, "usr/sbin/dnsmasq")
r := ensure(t)
if !r.installed() {
t.Fatalf("install was skipped on a dnsmasq-base-only host (binary present, unit absent) — "+
"the enable that follows targets a missing unit (R-317). calls: %q", r.calls)
}
}
func TestEnsureDnsmasq_UnitPresentSkipsInstall(t *testing.T) {
for _, unit := range []string{"usr/lib/systemd/system/dnsmasq.service", "lib/systemd/system/dnsmasq.service"} {
t.Run(unit, func(t *testing.T) {
fixtureRoot(t, "usr/sbin/dnsmasq", unit)
if r := ensure(t); r.installed() {
t.Fatalf("apt-get install ran although the dnsmasq unit is present at %s: %q", unit, r.calls)
}
})
}
}
func TestEnsureDnsmasq_NothingPresentInstalls(t *testing.T) {
fixtureRoot(t)
if r := ensure(t); !r.installed() {
t.Fatalf("install skipped on a host with no dnsmasq at all: %q", r.calls)
}
}
// Production wiring for the hostRoot seam: the shipped probe resolves against the real root and asks
// about the unit the `dnsmasq` package owns — never the dnsmasq-base binary.
func TestEnsureDnsmasq_ProductionProbeIsTheUnit(t *testing.T) {
if hostRoot != "/" {
t.Fatalf("hostRoot default = %q, want \"/\" — the production probe would look in the wrong tree", hostRoot)
}
var sawUsrLib bool
for _, p := range dnsmasqUnitPaths {
full := filepath.Join(hostRoot, p)
if strings.HasSuffix(full, "/sbin/dnsmasq") || strings.HasSuffix(full, "/bin/dnsmasq") {
t.Errorf("probe path %s is the dnsmasq-base binary, not the dnsmasq unit (R-317)", full)
}
if full == "/usr/lib/systemd/system/dnsmasq.service" {
sawUsrLib = true
}
}
if !sawUsrLib {
t.Errorf("probe paths %q miss /usr/lib/systemd/system/dnsmasq.service (dpkg -S: owned by dnsmasq)", dnsmasqUnitPaths)
}
}
+35 -3
View File
@@ -33,6 +33,8 @@ const (
DropinDir = "/etc/dnsmasq.d"
// BaseDropin holds the host-wide listen/upstream config (one per host).
BaseDropin = "felhom-resolver-base.conf"
// PrivApply is the root content checker that installs a drop-in (R-861).
PrivApply = "/usr/local/sbin/felhom-priv-apply"
)
// RenderBase returns the host-wide dnsmasq drop-in: bind to the host LAN IP (+ loopback), no-resolv,
@@ -99,11 +101,35 @@ func NewManager(runner proxmox.Runner, hostIP string, upstreams []string, logger
}
}
// hostRoot is the filesystem root the install probe resolves against: "/" in production; a test
// points it at a fixture tree so the REAL probe runs against files it controls.
var hostRoot = "/"
// dnsmasqUnitPaths are where the `dnsmasq` package ships its systemd unit (Debian; /lib is the
// pre-usrmerge spelling). R-317: probe the UNIT, never /usr/sbin/dnsmasq — that binary belongs to
// `dnsmasq-base`, so a host carrying dnsmasq-base without dnsmasq used to skip the install and then
// `systemctl enable --now dnsmasq` failed against a unit that is not there (resolver never up).
// Pinned by TestEnsureDnsmasq_BinaryWithoutUnitInstalls.
var dnsmasqUnitPaths = []string{
"usr/lib/systemd/system/dnsmasq.service",
"lib/systemd/system/dnsmasq.service",
}
// dnsmasqUnitInstalled reports whether the dnsmasq service unit (the `dnsmasq` package) is present.
func dnsmasqUnitInstalled() bool {
for _, p := range dnsmasqUnitPaths {
if _, err := os.Stat(filepath.Join(hostRoot, p)); err == nil {
return true
}
}
return false
}
// EnsureDnsmasq makes dnsmasq present + enabled and writes the host base config. Idempotent: it
// installs the package only when absent, and writes the base drop-in only when its content changes.
func (m *Manager) EnsureDnsmasq(ctx context.Context) error {
if _, err := os.Stat("/usr/sbin/dnsmasq"); err != nil { // metadata read, no privilege needed
m.logger.Info("lanresolver: dnsmasq absent — installing")
if !dnsmasqUnitInstalled() { // metadata read, no privilege needed
m.logger.Info("lanresolver: dnsmasq service unit absent — installing")
if out, errOut, ierr := m.runner.Run(ctx, "apt-get", "install", "-y", "-q", "dnsmasq"); ierr != nil {
return fmt.Errorf("install dnsmasq: %s: %w", strings.TrimSpace(string(errOut))+string(out), ierr)
}
@@ -263,7 +289,13 @@ func (m *Manager) writeFileIfChanged(ctx context.Context, path, content, mode st
return false, fmt.Errorf("write temp: %w", err)
}
tmp.Close()
if _, errOut, err := m.runner.Run(ctx, "install", "-m", mode, tmpName, path); err != nil {
// R-861 (v0.146.0): a dnsmasq drop-in reaches /etc/dnsmasq.d only through the root checker, which allows exactly
// the lines RenderBase/RenderGuestDropin write (a `dhcp-script=` would run as root). Mode is fixed (0644) there.
_ = mode
if filepath.Dir(path) != DropinDir {
return false, fmt.Errorf("drop-in %s is not in %s", path, DropinDir)
}
if _, errOut, err := m.runner.Run(ctx, PrivApply, "dnsmasq", tmpName, filepath.Base(path)); err != nil {
return false, fmt.Errorf("install %s: %s: %w", path, strings.TrimSpace(string(errOut)), err)
}
return true, nil
@@ -0,0 +1,22 @@
package lanresolver
import (
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/privapplytest"
)
// R-861: both drop-ins the resolver renders are accepted by the root checker; a dhcp-script line is not.
func TestPrivApply_AcceptsTheRenderedDropins(t *testing.T) {
if got := privapplytest.Check(t, "dnsmasq", BaseDropin, RenderBase("192.168.0.104", []string{"1.1.1.1", "8.8.8.8"})); got != "OK" {
t.Errorf("base drop-in: %s", got)
}
if got := privapplytest.Check(t, "dnsmasq", DropinName("demo-hp"), RenderGuestDropin("demo-hp", "enkisfelhom.hu", "192.168.0.138")); got != "OK" {
t.Errorf("guest drop-in: %s", got)
}
evil := RenderGuestDropin("demo-hp", "enkisfelhom.hu", "192.168.0.138") + "dhcp-script=/var/lib/felhom-agent/x\n"
if got := privapplytest.Check(t, "dnsmasq", "felhom-x.conf", evil); !strings.HasPrefix(got, "REFUSED") {
t.Fatalf("control: dhcp-script was not refused: %s", got)
}
}
+63
View File
@@ -0,0 +1,63 @@
package localapi
import (
"context"
"net/http"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
)
// The OS leg (agent v0.140.0) runs after a SUCCESSFUL primary backup, and only then; and it runs BEFORE the
// host-wide heavy-op gate is released, so a restore-test cannot start in the middle of it (`11` C10).
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
// and the gate sub-case fails.
func TestAfterPrimaryBackup(t *testing.T) {
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
gate := &backup.InFlight{}
b := &fakeBackups{failErr: failErr}
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
srv.inFlight = gate
var mu sync.Mutex
done := make(chan struct{}, 1)
srv.SetAfterPrimaryBackup(func(_ context.Context, vmid int) {
rel, _, ok := gate.TryAcquire("probe")
mu.Lock()
calls = append(calls, vmid)
gateHeld = !ok
mu.Unlock()
if ok {
rel()
}
done <- struct{}{}
})
h := srv.Handler()
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
t.Fatal("POST /backup not accepted")
}
select {
case <-done:
case <-time.After(500 * time.Millisecond):
}
time.Sleep(20 * time.Millisecond)
mu.Lock()
defer mu.Unlock()
return calls, gateHeld
}
t.Run("success runs the leg under the gate", func(t *testing.T) {
calls, held := run(t, "")
if len(calls) != 1 {
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
}
if !held {
t.Fatal("the heavy-op gate was free while the leg ran — a restore-test could overlap it")
}
})
t.Run("a failed backup runs nothing", func(t *testing.T) {
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
}
})
}
+102
View File
@@ -0,0 +1,102 @@
package localapi
import (
"encoding/json"
"errors"
"io"
"io/fs"
"net/http"
"os"
"time"
)
// GET /host/crash-guard (R-856, `09` §3 decision 143): what the host's crash guard
// (configs/felhom-crash-guard, `11` §5.9) recorded about the most recent HOST boot. The controller
// reads it once after it starts: when the host's last boot followed an UNCLEAN stop, its app mails
// wait ~15 minutes instead of the normal 90 s boot grace.
//
// Read-only and Proxmox-free: the agent reads the guard's state file (root-owned, 0644 — the
// non-root agent can read it) and passes four fields through. Host-wide, token-authed (any valid
// per-guest token sees the host's view, as GET /host/metrics does).
//
// NEVER an error page. A missing file (no guard installed, or no boot recorded yet), an unreadable
// one, or one that does not parse answers 200 with present:false — the controller reads that as
// UNKNOWN and keeps its normal boot grace. Pinned by TestR856_CrashGuard*.
// defaultCrashGuardStatePath is where configs/felhom-crash-guard writes its state (STATE_DIR there).
const defaultCrashGuardStatePath = "/var/lib/felhom-crash-guard/state.json"
// crashGuardStateMax bounds the read; the real file is well under 4 KiB.
const crashGuardStateMax = 1 << 20
// CrashGuardResponse is the data block of GET /host/crash-guard. Field names are the controller's
// agentapi.CrashGuardState (felhom-controller internal/agentapi/crashguard.go) — a wire contract,
// pinned by TestR856_CrashGuardWireMatchesControllerClient.
type CrashGuardResponse struct {
Present bool `json:"present"`
LastBootAt string `json:"last_boot_at,omitempty"` // RFC3339 UTC ("2006-01-02T15:04:05Z")
LastBootUnclean bool `json:"last_boot_unclean"`
Tripped bool `json:"tripped"`
}
// crashGuardFile is the subset of the guard's state.json the route passes through. Every other key
// (armed, boot_id, config, unclean_boots, last_trip, ...) is ignored.
type crashGuardFile struct {
LastBootAt string `json:"last_boot_at"`
LastBootUnclean bool `json:"last_boot_unclean"`
Tripped bool `json:"tripped"`
}
// readCrashGuardState reads and parses the guard's state file. ok=false on ANY failure (missing,
// unreadable, oversized, not a JSON object, a field of the wrong type); reason says which, for the log.
func readCrashGuardState(path string) (resp CrashGuardResponse, ok bool, reason string) {
f, err := os.Open(path)
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return resp, false, "no state file"
}
return resp, false, "unreadable: " + err.Error()
}
defer f.Close()
raw, err := io.ReadAll(io.LimitReader(f, crashGuardStateMax+1))
if err != nil {
return resp, false, "read: " + err.Error()
}
if len(raw) > crashGuardStateMax {
return resp, false, "state file too large"
}
var st crashGuardFile
// Unmarshal into a struct fails on a non-object top level (null decodes, so reject it below).
if err := json.Unmarshal(raw, &st); err != nil {
return resp, false, "unparseable: " + err.Error()
}
var probe map[string]json.RawMessage
if err := json.Unmarshal(raw, &probe); err != nil || probe == nil {
return resp, false, "unparseable: not a JSON object"
}
resp = CrashGuardResponse{Present: true, LastBootUnclean: st.LastBootUnclean, Tripped: st.Tripped}
// Normalise to RFC3339 UTC; an unparseable time passes through as-is (the controller reads an
// unparseable boot time as "not this start's boot" → its normal grace).
if t, perr := time.Parse(time.RFC3339, st.LastBootAt); perr == nil {
resp.LastBootAt = t.UTC().Format(time.RFC3339)
} else {
resp.LastBootAt = st.LastBootAt
}
return resp, true, ""
}
func (s *Server) handleCrashGuard(w http.ResponseWriter, r *http.Request, vmid int) {
path := s.crashGuardStatePath
if path == "" {
path = defaultCrashGuardStatePath
}
resp, ok, reason := readCrashGuardState(path)
if !ok {
s.logger.Debug("local-api: /host/crash-guard not present", "vmid", vmid, "reason", reason)
writeOK(w, CrashGuardResponse{Present: false})
return
}
s.logger.Debug("local-api: /host/crash-guard served", "vmid", vmid,
"last_boot_at", resp.LastBootAt, "last_boot_unclean", resp.LastBootUnclean, "tripped", resp.Tripped)
writeOK(w, resp)
}
+205
View File
@@ -0,0 +1,205 @@
package localapi
import (
"encoding/json"
"io"
"log/slog"
"net/http"
"os"
"path/filepath"
"testing"
)
// The shape of /var/lib/felhom-crash-guard/state.json as read on demo-hp on 2026-10-06 (values from
// that read where they matter; lists/objects kept to the same key set).
const crashGuardFixture = `{
"armed": true,
"boot_id": "3f1c0f1e-6a0b-4d7e-9b7a-0c2d4e6f8a1b",
"config": {"LIMIT": 3, "WINDOW_MINUTES": 60, "PANIC_SECONDS": 10},
"kernel_panic": 10,
"last_boot_at": "2026-10-05T07:56:41Z",
"last_boot_unclean": true,
"last_trip": {},
"rearmed_at": "2026-10-04T14:02:11Z",
"rearmed_by": "operator",
"tripped": false,
"unclean_boots": ["2026-10-05T07:56:41Z"],
"unclean_boots_24h": 1,
"unclean_boots_in_window": 1,
"updated_at": "2026-10-05T07:57:02Z",
"version": 1
}`
// controllerCrashGuardState is a COPY of the controller's wire type, felhom-controller
// controller/internal/agentapi/crashguard.go `CrashGuardState` (commit 8b13a5e) — same field names,
// same tags. If either side renames a key, the contract test below fails.
type controllerCrashGuardState struct {
Present bool `json:"present"`
LastBootAt string `json:"last_boot_at,omitempty"`
LastBootUnclean bool `json:"last_boot_unclean"`
Tripped bool `json:"tripped"`
}
func newCrashGuardServer(t *testing.T, statePath string) http.Handler {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuests{},
Backups: &fakeBackups{},
Store: &fakeStore{},
Storage: fakeStorage{},
Tokens: staticTokens{"A": 8200, "B": 9300},
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatalf("new server: %v", err)
}
srv.crashGuardStatePath = statePath
return srv.Handler()
}
func writeCrashGuardFixture(t *testing.T, body string) string {
t.Helper()
p := filepath.Join(t.TempDir(), "state.json")
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
return p
}
// getCrashGuard calls the route and decodes the envelope with the CONTROLLER's type.
func getCrashGuard(t *testing.T, h http.Handler, token string) (int, controllerCrashGuardState, string) {
t.Helper()
w := do(t, h, "GET", "/host/crash-guard", token, "")
var env struct {
OK bool `json:"ok"`
Data controllerCrashGuardState `json:"data"`
}
if w.Code == http.StatusOK {
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
t.Fatalf("decode %q: %v", w.Body.String(), err)
}
if !env.OK {
t.Fatalf("ok=false: %s", w.Body.String())
}
}
return w.Code, env.Data, w.Body.String()
}
// A present state file (demo-hp's shape) passes the three facts through.
func TestR856_CrashGuardPresentFile(t *testing.T) {
h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture))
code, st, body := getCrashGuard(t, h, "A")
if code != http.StatusOK {
t.Fatalf("got %d, want 200 (%s)", code, body)
}
want := controllerCrashGuardState{Present: true, LastBootAt: "2026-10-05T07:56:41Z", LastBootUnclean: true, Tripped: false}
if st != want {
t.Fatalf("state = %+v, want %+v", st, want)
}
// A tripped, clean boot reads back as such (both bools are carried, not defaulted).
h = newCrashGuardServer(t, writeCrashGuardFixture(t,
`{"last_boot_at":"2026-10-05T09:56:41+02:00","last_boot_unclean":false,"tripped":true,"version":1}`))
_, st, _ = getCrashGuard(t, h, "B")
want = controllerCrashGuardState{Present: true, LastBootAt: "2026-10-05T07:56:41Z", LastBootUnclean: false, Tripped: true}
if st != want {
t.Fatalf("offset time / tripped: state = %+v, want %+v (time normalised to UTC Z)", st, want)
}
}
// No state file (no guard on this host, or no boot recorded yet) → 200 present:false.
func TestR856_CrashGuardMissingFile(t *testing.T) {
h := newCrashGuardServer(t, filepath.Join(t.TempDir(), "absent", "state.json"))
code, st, body := getCrashGuard(t, h, "A")
if code != http.StatusOK {
t.Fatalf("missing file: got %d, want 200 (%s)", code, body)
}
if st.Present || st.LastBootUnclean || st.Tripped || st.LastBootAt != "" {
t.Fatalf("missing file: state = %+v, want present:false and nothing else", st)
}
}
// A garbled file → 200 present:false, never a 5xx — every shape of garbage.
func TestR856_CrashGuardGarbageFile(t *testing.T) {
for name, body := range map[string]string{
"truncated": crashGuardFixture[:40],
"not json": "this is not json\n",
"empty": "",
"null": "null",
"array": `[{"last_boot_unclean":true}]`,
"wrong type": `{"last_boot_at":"2026-10-05T07:56:41Z","last_boot_unclean":"yes","tripped":false}`,
"lone brace": "{",
} {
t.Run(name, func(t *testing.T) {
h := newCrashGuardServer(t, writeCrashGuardFixture(t, body))
code, st, raw := getCrashGuard(t, h, "A")
if code != http.StatusOK {
t.Fatalf("got %d, want 200 (%s)", code, raw)
}
if st.Present || st.LastBootUnclean {
t.Fatalf("garbage %q read as %+v, want present:false", name, st)
}
})
}
// The path is a directory, not a file: unreadable → present:false, 200.
h := newCrashGuardServer(t, t.TempDir())
if code, st, raw := getCrashGuard(t, h, "A"); code != http.StatusOK || st.Present {
t.Fatalf("directory path: got %d %+v (%s), want 200 present:false", code, st, raw)
}
}
// No / unknown token → 401, like every sibling route; a cross-guest ?vmid= → 403.
func TestR856_CrashGuardRequiresGuestToken(t *testing.T) {
h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture))
for _, tok := range []string{"", "bogus"} {
w := do(t, h, "GET", "/host/crash-guard", tok, "")
if w.Code != http.StatusUnauthorized {
t.Fatalf("token %q: got %d, want 401", tok, w.Code)
}
if json.Valid(w.Body.Bytes()) {
var env struct {
Data controllerCrashGuardState `json:"data"`
}
_ = json.Unmarshal(w.Body.Bytes(), &env)
if env.Data.Present || env.Data.LastBootUnclean {
t.Fatalf("token %q: the refusal leaked the state: %s", tok, w.Body.String())
}
}
}
if w := do(t, h, "GET", "/host/crash-guard?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
t.Fatalf("cross-guest query: got %d, want 403", w.Code)
}
}
// Wire contract: every key the controller's CrashGuardState decodes is emitted under exactly that
// name, and the agent emits no key the controller does not know.
func TestR856_CrashGuardWireMatchesControllerClient(t *testing.T) {
h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture))
w := do(t, h, "GET", "/host/crash-guard", "A", "")
var env struct {
OK bool `json:"ok"`
Data map[string]json.RawMessage `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil || !env.OK {
t.Fatalf("envelope: %v %s", err, w.Body.String())
}
want := []string{"present", "last_boot_at", "last_boot_unclean", "tripped"}
for _, k := range want {
if _, ok := env.Data[k]; !ok {
t.Errorf("agent does not emit %q, which the controller decodes (%s)", k, w.Body.String())
}
}
if len(env.Data) != len(want) {
t.Errorf("agent emits %d keys, controller knows %d: %s", len(env.Data), len(want), w.Body.String())
}
// And the agent's own type agrees with the controller's copy, field for field.
var mine CrashGuardResponse
var theirs controllerCrashGuardState
raw, _ := json.Marshal(env.Data)
_ = json.Unmarshal(raw, &mine)
_ = json.Unmarshal(raw, &theirs)
if (controllerCrashGuardState{mine.Present, mine.LastBootAt, mine.LastBootUnclean, mine.Tripped}) != theirs {
t.Errorf("agent %+v vs controller %+v", mine, theirs)
}
}
+20 -8
View File
@@ -398,9 +398,16 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
di.Smart = &sm
}
}
if total, used, okc := statfsCapacity(d.MountPath); okc {
di.TotalBytes, di.UsedBytes = total, used
di.UsedFraction = float64(used) / float64(total)
// R-118: statfs ONLY while the drive's device is present. With the device gone the raw
// mountpoint reverts to a bare directory on the ROOT filesystem, and statfs would report
// pve-root's size as this drive's (measured: a 4 GB drive advertising 46 GiB). Same trap
// observe.go guards on the Observe path. Absent → capacity left zero (unknown), never root's.
// Pinned by TestDisks_UnionPath_AbsentDeviceReportsNoRootCapacity.
if s.devicePresent(d.MountPath) {
if total, used, okc := statfsCapacity(d.MountPath); okc {
di.TotalBytes, di.UsedBytes = total, used
di.UsedFraction = float64(used) / float64(total)
}
}
out = append(out, di)
}
@@ -761,6 +768,11 @@ type FormatResponse struct {
// signature — the customer authorizes the wipe of their own data drive.
NeedsConfirmation bool `json:"needs_confirmation,omitempty"`
DurableID string `json:"durable_id,omitempty"` // the durable id to confirm against (user-data)
// FSUUID (on Formatted) is the UUID of the filesystem the agent just made, verified against the
// bound durable id after mkfs (R-25). The caller mounts THIS — re-resolving a UUID from the /dev
// path later can name another disk if /dev re-enumerated. "" = not verified: the caller must not
// substitute a path-resolved guess silently.
FSUUID string `json:"fs_uuid,omitempty"`
// PendingOp is set on a SYSTEM/BACKUP data-bearing refusal — the exact op the operator must sign.
PendingOp *PendingOp `json:"pending_op,omitempty"`
}
@@ -808,7 +820,7 @@ func (s *Server) handleDiskFormatStatus(w http.ResponseWriter, r *http.Request,
writeOK(w, map[string]any{
"vmid": vmid, "phase": job.Phase, "device": job.Device, "fstype": job.FSType,
"durable_id": job.DurableID, "error": job.Error, "started_at": job.StartedAt, "updated_at": job.UpdatedAt,
"job_id": job.JobID,
"job_id": job.JobID, "fs_uuid": job.FSUUID, // R-25: "" until done + verified
})
}
@@ -871,7 +883,7 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
"format refused (device may have changed since inspection): "+rerr.Error())
return
}
done := s.startFormatDetached(device, blankDurable, req.FSType, true)
job, done := s.startFormatDetached(device, blankDurable, req.FSType, true)
if err := s.awaitFormat(r.Context(), done, vmid, device); err != nil {
if err == errFormatClientGone {
return // client gone; mkfs continues detached + the job record records the outcome
@@ -880,7 +892,7 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
writeErr(w, http.StatusBadGateway, "format failed: "+err.Error())
return
}
writeOK(w, FormatResponse{VMID: vmid, Device: device, Formatted: true, DataBearing: false, DurableID: blankDurable, Reason: "blank device formatted " + req.FSType})
writeOK(w, FormatResponse{VMID: vmid, Device: device, Formatted: true, DataBearing: false, DurableID: blankDurable, FSUUID: job.FSUUID, Reason: "blank device formatted " + req.FSType})
return
}
@@ -917,7 +929,7 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
// F20-BUG3: run the destructive mkfs DETACHED off s.baseCtx (bound durable id recorded for
// restart-recovery), so a request/client deadline can never SIGKILL it mid-write and corrupt the
// disk. We still wait to return the synchronous result (backward-compatible with the controller).
done := s.startFormatDetached(device, deviceDurable, req.FSType, false)
job, done := s.startFormatDetached(device, deviceDurable, req.FSType, false)
if err := s.awaitFormat(r.Context(), done, vmid, device); err != nil {
if err == errFormatClientGone {
return // client gone; the wipe continues detached + survives a restart via the job record
@@ -929,7 +941,7 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
s.logger.Warn("local-api: USER-DATA data-bearing format — CUSTOMER CONFIRMED (no operator signature)",
"vmid", vmid, "device", device, "durable_id", deviceDurable, "fstype", req.FSType, "why", probe.Reason())
writeOK(w, FormatResponse{VMID: vmid, Device: device, Formatted: true, DataBearing: true,
Role: string(role), DurableID: deviceDurable, Reason: "customer-confirmed wipe (" + probe.Reason() + ")"})
Role: string(role), DurableID: deviceDurable, FSUUID: job.FSUUID, Reason: "customer-confirmed wipe (" + probe.Reason() + ")"})
return
}
@@ -5,6 +5,7 @@ import (
"encoding/json"
"io"
"log/slog"
"runtime"
"strings"
"testing"
@@ -184,3 +185,39 @@ func TestDisks_DevicePresence_WireFieldIsFalseOnDeviceLoss(t *testing.T) {
t.Fatalf("the drive never reached the wire: %s", body)
}
}
// ── R-118 — an absent drive must not advertise the ROOT filesystem's capacity ───────────────────
// TestDisks_UnionPath_AbsentDeviceReportsNoRootCapacity drives the REAL statfsCapacity (no capacity
// seam): the registry drive's mount path is a real, bare temp directory — exactly what /mnt/<name>
// becomes once its device is gone (a plain directory on the host's filesystem). With the device absent
// the row must carry NO capacity; before R-118 the union path statfs'd that bare directory and reported
// the host filesystem's size and usage as the drive's (46 GiB at 9.2 % for a 4 GB drive, measured).
// The present half proves the test is not hollow: the same directory DOES yield capacity when the
// device is there, so a zero on the absent half is the guard's doing, not a statfs failure.
//
// RED-PROOF: drop the `if s.devicePresent(d.MountPath)` guard around statfsCapacity in disks.go → the
// absent subtest fails with "advertises ... bytes".
func TestDisks_UnionPath_AbsentDeviceReportsNoRootCapacity(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("statfsCapacity is linux-only; production target is linux")
}
bare := t.TempDir()
known := []storage.KnownTarget{
{Name: "cel", Type: hub.StorageTypeUSB, MountPath: bare, DurableID: "uuid:4242", UUID: "4242"},
}
t.Run("absent", func(t *testing.T) {
di := diskByMount(t, presenceServer(t, nil, known, true, false), bare)
if di.TotalBytes != 0 || di.UsedBytes != 0 || di.UsedFraction != 0 {
t.Errorf("absent drive advertises total=%d used=%d frac=%.3f — that is the filesystem UNDER "+
"the bare mountpoint, not the drive (R-118)", di.TotalBytes, di.UsedBytes, di.UsedFraction)
}
})
t.Run("present", func(t *testing.T) {
di := diskByMount(t, presenceServer(t, nil, known, true, true), bare)
if di.TotalBytes <= 0 {
t.Errorf("present drive reports no capacity (total=%d) — the guard over-corrected and the "+
"size bar is gone for every healthy registry drive", di.TotalBytes)
}
})
}
+6
View File
@@ -28,6 +28,7 @@ type fakeDiskOps struct {
unmountCalls []string
candidates []storage.CandidateDisk // returned by ListCandidateDisks
candErr error
afterFormat *storage.DeviceProbe // R-25: when set, InspectDevice returns it once a format ran
}
func (f *fakeDiskOps) ListCandidateDisks(_ context.Context) ([]storage.CandidateDisk, error) {
@@ -35,7 +36,12 @@ func (f *fakeDiskOps) ListCandidateDisks(_ context.Context) ([]storage.Candidate
}
func (f *fakeDiskOps) InspectDevice(_ context.Context, device string) (storage.DeviceProbe, error) {
f.mu.Lock()
p := f.probe
if f.afterFormat != nil && len(f.formatCalls) > 0 {
p = *f.afterFormat
}
f.mu.Unlock()
p.Device = device
return p, f.inspectErr
}
+96
View File
@@ -0,0 +1,96 @@
package localapi
import (
"context"
"encoding/json"
"net/http"
"sync"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// R-25: the format answer carries the UUID of the filesystem the agent JUST made, verified against the
// bound durable id after mkfs, so the controller mounts that filesystem rather than whatever the /dev
// path resolves to a few requests later. The consequence asserted: the UUID on the wire (and in the
// polled job record) is the new superblock's — and is EMPTY whenever the binding cannot be re-proved.
const newFSUUID = "0fc63daf-8483-4772-8e79-3d69d8477de4"
func confirmedFormat(t *testing.T, d *fakeDiskOps, srv *Server, fj *FormatJobStore) (string, *formatJob) {
t.Helper()
w := do(t, srv.Handler(), "POST", "/disks/format", "A", `{"device":"/dev/sdb1","fstype":"ext4","confirmed":true,"durable_id":"byid:wwn-/dev/sdb1"}`)
if w.Code != http.StatusOK {
t.Fatalf("confirmed format: %d (%s)", w.Code, w.Body.String())
}
var resp struct {
Data FormatResponse `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("decode: %v (%s)", err, w.Body.String())
}
if !resp.Data.Formatted {
t.Fatalf("not formatted: %s", w.Body.String())
}
return resp.Data.FSUUID, waitFormatPhase(t, fj, formatPhaseDone)
}
func confirmedGate() *fakeGate {
return &fakeGate{decision: WipeDecision{Allowed: true, Tier: "customer_confirmable", Reason: "customer_confirmed"}}
}
func TestFormat_ReportsNewFilesystemUUID(t *testing.T) {
d := &fakeDiskOps{probe: deviceProbeDataBearing(),
afterFormat: &storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4", FSUUID: newFSUUID}}
fj := tempFormatStore(t)
srv := formatServer(t, d, confirmedGate(), fj)
got, job := confirmedFormat(t, d, srv, fj)
if got != newFSUUID {
t.Fatalf("response fs_uuid = %q, want the new filesystem's %q", got, newFSUUID)
}
if job.FSUUID != newFSUUID {
t.Fatalf("job record fs_uuid = %q, want %q (the polled status path)", job.FSUUID, newFSUUID)
}
}
// The node moved between mkfs and the read-back: the bound durable id now resolves elsewhere. The
// UUID must NOT be reported — reading it would name the other disk's filesystem.
func TestFormat_FSUUIDWithheldWhenDurableIDMoved(t *testing.T) {
d := &fakeDiskOps{probe: deviceProbeDataBearing(),
afterFormat: &storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4", FSUUID: newFSUUID}}
fj := tempFormatStore(t)
srv := formatServer(t, d, confirmedGate(), fj)
var mu sync.Mutex
calls := 0
srv.reresolveWipe = func(_ context.Context, _ string) (string, error) {
mu.Lock()
defer mu.Unlock()
calls++
if calls == 1 {
return "/dev/sdb", nil // the pre-mkfs anti-retarget re-resolve
}
return "/dev/sdc", nil // after mkfs: the durable id now names another node
}
got, job := confirmedFormat(t, d, srv, fj)
if got != "" || job.FSUUID != "" {
t.Fatalf("fs_uuid reported after the durable id moved (response %q, job %q) — must be empty", got, job.FSUUID)
}
if calls < 2 {
t.Fatalf("the post-mkfs re-resolve never ran (calls=%d)", calls)
}
}
// The superblock did not read back as the requested filesystem → not verified → empty.
func TestFormat_FSUUIDWithheldOnFSTypeMismatch(t *testing.T) {
d := &fakeDiskOps{probe: deviceProbeDataBearing(),
afterFormat: &storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "xfs", FSUUID: newFSUUID}}
fj := tempFormatStore(t)
srv := formatServer(t, d, confirmedGate(), fj)
got, job := confirmedFormat(t, d, srv, fj)
if got != "" || job.FSUUID != "" {
t.Fatalf("fs_uuid reported for a superblock of the wrong type (response %q, job %q)", got, job.FSUUID)
}
}
+39 -3
View File
@@ -22,6 +22,10 @@ type formatJob struct {
Blank bool `json:"blank,omitempty"` // audit D3: blank (benign) format — recovery re-checks STILL-blank, not data-bearing
Phase string `json:"phase"` // running | done | failed
Error string `json:"error,omitempty"`
// FSUUID is the filesystem UUID of the NEW filesystem, read by the agent right after mkfs on the
// device the bound durable id still resolves to (R-25). "" = not verified (the controller must not
// read that as a UUID). Set only on phase done.
FSUUID string `json:"fs_uuid,omitempty"`
StartedAt string `json:"started_at"`
UpdatedAt string `json:"updated_at"`
}
@@ -96,7 +100,10 @@ func (s *FormatJobStore) save(j *formatJob) error {
// runs to completion and records the outcome. device is the ALREADY anti-retarget-resolved device; the
// record carries durableID so a restart can re-resolve + re-run. blank marks a benign (blank-device)
// format, so restart recovery re-checks STILL-blank rather than data-bearing (audit D3).
func (s *Server) startFormatDetached(device, durableID, fstype string, blank bool) <-chan error {
//
// The returned job may be read (FSUUID) only AFTER a value arrives on done — the goroutine writes it
// before the send, which is the happens-before edge.
func (s *Server) startFormatDetached(device, durableID, fstype string, blank bool) (*formatJob, <-chan error) {
base := s.baseCtx
if base == nil {
base = context.Background()
@@ -116,10 +123,39 @@ func (s *Server) startFormatDetached(device, durableID, fstype string, blank boo
ctx, cancel := context.WithTimeout(base, 60*time.Minute)
defer cancel()
err := s.disks.Format(ctx, device, fstype)
if err == nil {
job.FSUUID = s.formattedFSUUID(ctx, device, durableID, fstype)
}
s.finishFormatJob(job, err)
done <- err
}()
return done
return job, done
}
// formattedFSUUID reads the UUID of the filesystem the agent has JUST made (R-25). The caller used to
// re-resolve the UUID from the mutable /dev path afterwards, over separate requests — a re-enumeration
// in that window could hand it ANOTHER disk's filesystem to mount. Here the bound durable id must still
// resolve to the very device that was formatted (and re-derive to the same id), and the superblock must
// carry the fstype that was asked for; anything else returns "" (not verified), never a guess.
func (s *Server) formattedFSUUID(ctx context.Context, device, durableID, fstype string) string {
if durableID == "" || s.reresolveWipe == nil {
return ""
}
// The device now holds a filesystem, so the data-bearing anti-retarget re-resolve is the right one.
now, err := s.reresolveWipe(ctx, durableID)
if err != nil || now != device {
s.logger.Warn("format: new filesystem UUID NOT reported — bound durable id no longer resolves to the formatted device",
"device", device, "durable_id", durableID, "resolves_to", now, "err", err)
return ""
}
probe, err := s.disks.InspectDevice(ctx, device)
if err != nil || !probe.Probed || probe.FSType != fstype || probe.FSUUID == "" {
s.logger.Warn("format: new filesystem UUID NOT reported — superblock did not read back as the requested filesystem",
"device", device, "want_fstype", fstype, "got_fstype", probe.FSType, "has_uuid", probe.FSUUID != "", "err", err)
return ""
}
s.logger.Info("format: new filesystem bound to its durable id", "device", device, "durable_id", durableID, "fs_uuid", probe.FSUUID)
return probe.FSUUID
}
// finishFormatJob updates the persisted record to done/failed.
@@ -172,7 +208,7 @@ func (s *Server) RecoverFormatJob(ctx context.Context) {
return
}
s.logger.Warn("format-job recover: re-running interrupted format detached", "durable_id", job.DurableID, "device", device, "fstype", job.FSType, "blank", job.Blank)
_ = s.startFormatDetached(device, job.DurableID, job.FSType, job.Blank) // detached; updates the record on completion
_, _ = s.startFormatDetached(device, job.DurableID, job.FSType, job.Blank) // detached; updates the record on completion
}
// nowFn returns the server clock (testable), defaulting to time.Now.
+20 -50
View File
@@ -121,32 +121,33 @@ func (b *GuestBinder) EnsureSharedParent(ctx context.Context) error {
// only the script (the unit was unchanged), so the earlier unit-only gate never redeployed it —
// leaving hosts running the pre-fix script (no make-private), whose self-bind stays in root's shared
// peer group and DOUBLES every drive bind. Comparing both files closes that deploy gap.
if sharedParentInstallStale(sharedParentUnitPath, sharedParentScriptPath) {
if ierr := b.installSharedParentUnit(ctx); ierr != nil {
b.logger.Warn("shared-parent: boot-persistence (re)install failed (live setup OK; survives until host reboot)", "err", ierr)
}
if err := b.ensureSharedParentBoot(ctx, sharedParentUnitPath, sharedParentScriptPath, sharedParentWantsLink); err != nil {
b.logger.Warn("shared-parent: boot persistence not enabled (live setup OK; survives until host reboot)", "err", err)
}
return nil
}
// stageTemp writes content to a fresh random-named temp file (os.CreateTemp pattern — `*` is replaced
// by a random string) and returns its path. Caller removes it after the privileged `install`.
func stageTemp(pattern, content string) (string, error) {
f, err := os.CreateTemp("", pattern)
if err != nil {
return "", err
// sharedParentWantsLink exists once `systemctl enable felhom-shared-parent.service` ran (WantedBy=pve-guests.service).
const sharedParentWantsLink = "/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service"
// ensureSharedParentBoot (R-861, agent v0.146.0) — the boot script and its unit are ROOT-OWNED FIXED files that arrive
// with the signed config bundle (configs/felhom-shared-parent.{sh,service}, byte-identical to the constants above,
// pinned by TestSharedParentFilesEqualTheBundle). The agent NEVER installs them any more: until v0.146.0 it installed
// them from /tmp, and a script a root unit runs at boot was a root shell for a compromised agent. Here it only checks
// them, and enables the unit (a fixed sudoers line) when the bundle has put it in place but nothing has enabled it — a
// fresh install. Missing or different files: one warning, nothing run (the next bundle brings them).
func (b *GuestBinder) ensureSharedParentBoot(ctx context.Context, unitPath, scriptPath, wantsLink string) error {
if sharedParentInstallStale(unitPath, scriptPath) {
return fmt.Errorf("%s or %s is missing or differs from this agent's — it arrives with the signed config bundle (agent_config_update); the agent does not install it (R-861)", scriptPath, unitPath)
}
name := f.Name()
if _, err := f.WriteString(content); err != nil {
f.Close()
os.Remove(name)
return "", err
if _, err := os.Lstat(wantsLink); err == nil {
return nil
}
if err := f.Close(); err != nil {
os.Remove(name)
return "", err
if err := b.run(ctx, "systemctl", "enable", "felhom-shared-parent.service"); err != nil {
return fmt.Errorf("enable unit: %w", err)
}
return name, nil
b.logger.Info("shared-parent: boot-persistence unit enabled (files from the config bundle)")
return nil
}
// sharedParentInstallStale reports whether the on-disk boot script OR unit is missing or differs from
@@ -162,37 +163,6 @@ func sharedParentInstallStale(unitPath, scriptPath string) bool {
return false
}
// installSharedParentUnit writes the script + unit (from agent-written temps) and enables the unit so the
// shared parent is re-established on every host boot before pve-guests. Idempotent. The temps are
// RANDOM-named os.CreateTemp files (audit B1): a fixed, predictable /tmp name could be pre-created by
// another local user and rewritten between our write and root's install (TOCTOU into a root-executed
// boot script). The final modes come from `install -m`, so the 0600 temps are fine.
func (b *GuestBinder) installSharedParentUnit(ctx context.Context) error {
tmpScript, err := stageTemp("felhom-shared-parent-*.sh", sharedParentScript)
if err != nil {
return fmt.Errorf("write temp script: %w", err)
}
defer os.Remove(tmpScript)
if err := b.run(ctx, "install", "-m", "0755", "--", tmpScript, sharedParentScriptPath); err != nil {
return fmt.Errorf("install script: %w", err)
}
tmpUnit, err := stageTemp("felhom-shared-parent-*.service", sharedParentUnit)
if err != nil {
return fmt.Errorf("write temp unit: %w", err)
}
defer os.Remove(tmpUnit)
if err := b.run(ctx, "install", "-m", "0644", "--", tmpUnit, sharedParentUnitPath); err != nil {
return fmt.Errorf("install unit: %w", err)
}
if err := b.run(ctx, "systemctl", "daemon-reload"); err != nil {
return fmt.Errorf("daemon-reload: %w", err)
}
if err := b.run(ctx, "systemctl", "enable", "felhom-shared-parent.service"); err != nil {
return fmt.Errorf("enable unit: %w", err)
}
return nil
}
// AttachDrive binds a drive's felhom-data namespace under the stable parent so it appears live in the
// guest at the returned stable path (via propagation — no pct, no reboot). `where` is the drive's RAW
// host PVE mount (/mnt/<name>); only `<where>/felhom-data` crosses into the guest (confinement). The
+58 -70
View File
@@ -4,94 +4,82 @@ import (
"context"
"io"
"os"
"regexp"
"path/filepath"
"testing"
)
// stagingRecorderRunner is a fake proxmox.Runner recording every call vector and snapshotting each
// install SOURCE file's content at call time (the deferred os.Remove erases it afterwards).
type stagingRecorderRunner struct {
calls [][]string
srcContent map[string]string // install dest → staged source content
}
// R-861 (agent v0.146.0): the shared-parent boot script and unit arrive with the signed config bundle; the agent never
// installs them. It checks them and enables the unit when nothing has.
//
// RED-PROOF (audits/hub-safety-2026-10-05/partF/red-proof.txt): put the old installSharedParentUnit call back (an
// `install` of a /tmp file) → TestSharedParentBoot_NeverInstalls fails.
func (r *stagingRecorderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
type callRecorder struct{ calls [][]string }
func (r *callRecorder) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
if name == "install" && len(args) >= 2 {
src, dest := args[len(args)-2], args[len(args)-1]
if r.srcContent == nil {
r.srcContent = map[string]string{}
}
b, _ := os.ReadFile(src)
r.srcContent[dest] = string(b)
}
return nil, nil, nil
}
func (r *stagingRecorderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
func (r *callRecorder) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
// installSources returns the install-call source paths keyed by destination.
func (r *stagingRecorderRunner) installSources() map[string][]string {
out := map[string][]string{}
for _, c := range r.calls {
if c[0] == "install" && len(c) >= 3 {
src, dest := c[len(c)-2], c[len(c)-1]
out[dest] = append(out[dest], src)
}
func bootFiles(t *testing.T, script, unit string) (string, string, string) {
t.Helper()
d := t.TempDir()
sp, up := filepath.Join(d, "felhom-shared-parent.sh"), filepath.Join(d, "felhom-shared-parent.service")
if script != "" {
_ = os.WriteFile(sp, []byte(script), 0o755)
}
return out
if unit != "" {
_ = os.WriteFile(up, []byte(unit), 0o644)
}
return sp, up, filepath.Join(d, "wants-link")
}
// TestInstallSharedParent_RandomTempName is the audit-B1 negative test for the shared-parent boot
// persistence install: both staged install SOURCES (script + unit) must be RANDOM os.CreateTemp names
// (felhom-shared-parent-<random>.sh / .service), never the fixed, pre-creatable /tmp names (a local
// TOCTOU into a root-executed boot script), and two consecutive installs must use DIFFERENT paths.
func TestInstallSharedParent_RandomTempName(t *testing.T) {
r := &stagingRecorderRunner{}
func TestSharedParentBoot_NeverInstalls(t *testing.T) {
for _, c := range []struct{ name, script, unit string }{
{"both missing", "", ""},
{"script differs", "#!/bin/sh\necho old\n", sharedParentUnit},
{"unit missing", sharedParentScript, ""},
} {
r := &callRecorder{}
b := NewGuestBinder(r, nil)
sp, up, link := bootFiles(t, c.script, c.unit)
if err := b.ensureSharedParentBoot(context.Background(), up, sp, link); err == nil {
t.Errorf("%s: no error — the operator would not learn the bundle is missing", c.name)
}
if len(r.calls) != 0 {
t.Errorf("%s: the agent ran %v — it must install nothing (R-861)", c.name, r.calls)
}
}
}
func TestSharedParentBoot_EnablesOnceTheBundleBroughtTheFiles(t *testing.T) {
r := &callRecorder{}
b := NewGuestBinder(r, nil)
if err := b.installSharedParentUnit(context.Background()); err != nil {
t.Fatalf("installSharedParentUnit #1: %v", err)
sp, up, link := bootFiles(t, sharedParentScript, sharedParentUnit)
if err := b.ensureSharedParentBoot(context.Background(), up, sp, link); err != nil {
t.Fatal(err)
}
if err := b.installSharedParentUnit(context.Background()); err != nil {
t.Fatalf("installSharedParentUnit #2: %v", err)
if len(r.calls) != 1 || len(r.calls[0]) != 3 || r.calls[0][0] != "systemctl" || r.calls[0][1] != "enable" ||
r.calls[0][2] != "felhom-shared-parent.service" {
t.Fatalf("want exactly `systemctl enable felhom-shared-parent.service`, got %v", r.calls)
}
_ = os.Symlink(up, link)
r.calls = nil
if err := b.ensureSharedParentBoot(context.Background(), up, sp, link); err != nil || len(r.calls) != 0 {
t.Fatalf("already enabled: want no calls, got %v (%v)", r.calls, err)
}
}
srcs := r.installSources()
cases := []struct {
dest string
random *regexp.Regexp
fixed *regexp.Regexp
content string
}{
{sharedParentScriptPath, regexp.MustCompile(`felhom-shared-parent-[^/\\]+\.sh$`),
regexp.MustCompile(`felhom-shared-parent\.sh$`), sharedParentScript},
{sharedParentUnitPath, regexp.MustCompile(`felhom-shared-parent-[^/\\]+\.service$`),
regexp.MustCompile(`felhom-shared-parent\.service$`), sharedParentUnit},
}
for _, tc := range cases {
got := srcs[tc.dest]
if len(got) != 2 {
t.Fatalf("dest %s: expected 2 install calls, got %d (%v)", tc.dest, len(got), got)
}
for i, src := range got {
if !tc.random.MatchString(src) {
t.Errorf("dest %s call %d: source %q does not match the random temp pattern", tc.dest, i, src)
}
if tc.fixed.MatchString(src) {
t.Errorf("dest %s call %d: source %q is the FIXED predictable temp name (B1 TOCTOU)", tc.dest, i, src)
}
if _, err := os.Stat(src); err == nil {
t.Errorf("dest %s: staged temp %q left behind (defer os.Remove missing)", tc.dest, src)
}
}
if got[0] == got[1] {
t.Errorf("dest %s: two consecutive installs staged through the SAME source %q — must be random per call", tc.dest, got[0])
}
// Non-hollow: the staged file carried the real content at install time.
if r.srcContent[tc.dest] != tc.content {
t.Errorf("dest %s: staged content mismatch (got %d bytes, want %d)", tc.dest, len(r.srcContent[tc.dest]), len(tc.content))
// The bundle's copies are byte-identical to the constants the agent compares against.
func TestSharedParentFilesEqualTheBundle(t *testing.T) {
for file, want := range map[string]string{"felhom-shared-parent.sh": sharedParentScript, "felhom-shared-parent.service": sharedParentUnit} {
got, err := os.ReadFile(filepath.Join("..", "..", "configs", file))
if err != nil || string(got) != want {
t.Errorf("configs/%s differs from the agent's constant (err %v)", file, err)
}
}
}
+144
View File
@@ -0,0 +1,144 @@
package localapi
import (
"context"
"io"
"log/slog"
"net/http"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-894 — after an agent restart, an UNREADABLE storage must fall back to the last success saved on
// disk, not to "never". Measured 2026-10-05 on demo-hp: a restart at 04:57, the off-site storage
// unreachable at 06:25, the 7-day tier (last copy 4 days old) read DUE, vzdump failed.
//
// Every test here builds a NEW server and a NEW BackupSuccessState from the same file — that is the
// restart. The in-memory store (fakeStore) is always fresh, as after a real restart.
// r894Server builds a two-tier server whose off-site tier answers the storage listing with lister.
func r894Server(t *testing.T, path string, pbsSvc BackupService) *Server {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local"}, {Name: "felhom-pbs"}}},
Tokens: staticTokens{"A": 8200},
BackupTiers: []BackupTier{
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: pbsSvc},
},
LastKnownBackups: backup.NewBackupSuccessState(path),
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.now = func() time.Time { return testNow }
return srv
}
// unreadable is the off-site storage as demo-hp saw it: "Can't connect to 10.77.0.1:8007".
func unreadable() archiveLister {
return archiveLister{fakeBackups: &fakeBackups{}, err: errStorageRead}
}
// THE R-894 CASE, end to end. Agent 1 takes an off-site backup through POST /backup (the fake
// runner's success is 12 h before testNow). The agent restarts. The storage cannot be read. The tier
// must read NOT due, from the copy saved on disk.
//
// COMPANION RED-PROOF (observed): delete the `lookup == archiveUnknown && s.lastKnown != nil` block in
// handleBackupDue → this fails with "after a restart an unreadable storage must fall back to the saved
// copy (12 h old, 7-day tier) — NOT due; got {… Due:true … AgeState:unknown …}". Restored.
func TestBackupDue_R894_RestartThenUnreadableStorage_FreshSavedCopyIsNotDue(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
// Agent 1: a real backup job through the endpoint the controller calls.
first := r894Server(t, path, &fakeBackups{})
if rr := do(t, first.Handler(), "POST", "/backup?target=felhom-pbs", "A", ""); rr.Code != http.StatusAccepted {
t.Fatalf("POST /backup: %d %s", rr.Code, rr.Body.String())
}
waitFor(t, func() bool {
_, ok := backup.NewBackupSuccessState(path).LastKnownSuccess("felhom-pbs", 8200)
return ok
})
// Agent 2: a restart (new server, new state from the same file), and the storage is unreachable.
got := dueFor(t, r894Server(t, path, unreadable()).Handler(), "felhom-pbs")
if got.Due {
t.Fatalf("after a restart an unreadable storage must fall back to the saved copy (12 h old, 7-day tier) — NOT due; got %+v", got)
}
if got.AgeState != AgeStateKnown || got.AgeSecs == nil || *got.AgeSecs != int64((12*time.Hour).Seconds()) {
t.Fatalf("the age must come from the saved copy (12 h, known); got %+v", got)
}
}
// The deliberate rule stays: an unreadable storage must not suppress a backup that IS due. A saved
// copy older than the cadence reads DUE.
//
// COMPANION RED-PROOF (observed): make the fallback answer not-due whenever a saved copy exists
// (`if fromDisk { …Due:false… }` before the cadence check) → this fails with "a saved copy 9 days old
// under a 7-day cadence MUST read due". Restored.
func TestBackupDue_R894_RestartThenUnreadableStorage_OldSavedCopyIsDue(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
st := backup.NewBackupSuccessState(path)
if err := st.RecordBackupSuccess("felhom-pbs", backupAt("felhom-pbs", 8200, 9*24*time.Hour, true)); err != nil {
t.Fatal(err)
}
got := dueFor(t, r894Server(t, path, unreadable()).Handler(), "felhom-pbs")
if !got.Due {
t.Fatalf("a saved copy 9 days old under a 7-day cadence MUST read due; got %+v", got)
}
if got.AgeState != AgeStateKnown {
t.Fatalf("the age is known (from disk); got %+v", got)
}
}
// No saved copy → the pre-R-894 answer, byte for byte: DUE, age UNKNOWN (never ABSENT — the controller
// fires its window-gate valve only on absent, R-88).
func TestBackupDue_R894_RestartThenUnreadableStorage_NoSavedCopyIsDueUnknown(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
got := dueFor(t, r894Server(t, path, unreadable()).Handler(), "felhom-pbs")
if !got.Due || got.AgeState != AgeStateUnknown || got.AgeSecs != nil {
t.Fatalf("no saved copy + unreadable storage must stay DUE with age unknown; got %+v", got)
}
}
// A storage that ANSWERS is the ground truth: an archive absent there makes the tier due even when the
// file remembers a fresh success (a pruned or deleted copy must be made again).
//
// COMPANION RED-PROOF (observed): drop `lookup == archiveUnknown &&` from the fallback condition → this
// fails with "the storage answered 'no archive' — the saved copy must NOT stand in for it". Restored.
func TestBackupDue_R894_SavedCopyIgnoredWhenStorageAnswers(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
st := backup.NewBackupSuccessState(path)
if err := st.RecordBackupSuccess("felhom-pbs", backupAt("felhom-pbs", 8200, time.Hour, true)); err != nil {
t.Fatal(err)
}
absent := archiveLister{fakeBackups: &fakeBackups{}, found: false}
got := dueFor(t, r894Server(t, path, absent).Handler(), "felhom-pbs")
if !got.Due {
t.Fatalf("the storage answered 'no archive' — the saved copy must NOT stand in for it; got %+v", got)
}
}
// A FAILED backup is never saved: it must not make a tier look fresh after a restart.
func TestBackupDue_R894_FailedBackupIsNotSaved(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
first := r894Server(t, path, &fakeBackups{failErr: "could not activate storage 'felhom-pbs'"})
if rr := do(t, first.Handler(), "POST", "/backup?target=felhom-pbs", "A", ""); rr.Code != http.StatusAccepted {
t.Fatalf("POST /backup: %d %s", rr.Code, rr.Body.String())
}
waitFor(t, func() bool { return len(first.store.Backups(context.Background())) == 1 })
if _, ok := backup.NewBackupSuccessState(path).LastKnownSuccess("felhom-pbs", 8200); ok {
t.Fatal("a failed backup must never be saved as a success")
}
got := dueFor(t, r894Server(t, path, unreadable()).Handler(), "felhom-pbs")
if !got.Due {
t.Fatalf("after a failed backup and a restart the tier must still be due; got %+v", got)
}
}
+79 -21
View File
@@ -85,6 +85,13 @@ type BackupStore interface {
RestoreTests(ctx context.Context) []hub.RestoreTest
}
// LastKnownBackupStore (R-894) is the on-disk newest-success-per-tier record. Satisfied by
// *backup.BackupSuccessState.
type LastKnownBackupStore interface {
RecordBackupSuccess(target string, b hub.Backup) error
LastKnownSuccess(target string, vmid int) (time.Time, bool)
}
// StorageView yields the host's observed storage targets (for mapping a mount's storage id →
// fast/slow class). Satisfied by *storage.Observer.
type StorageView interface {
@@ -160,6 +167,14 @@ type Options struct {
// NetStorage is the privileged network-mount (NAS) surface (Part A1). OPTIONAL — when nil, the
// /netstorage endpoints report "not configured". Satisfied by *storage.SudoHostOps.
NetStorage NetworkStorageOps
// AfterPrimaryBackup (agent v0.140.0, `11-os-updates.md` §8 step 2) runs right after a SUCCESSFUL backup on the
// PRIMARY tier, inside the backup goroutine and BEFORE the host-wide heavy-op gate is released — so the OS leg
// that it starts can never overlap another backup or a restore-test (`11` C10). OPTIONAL — nil → nothing runs.
AfterPrimaryBackup func(ctx context.Context, vmid int)
// LastKnownBackups (R-894) keeps the newest successful backup per tier ON DISK, so the due-check's
// fallback for an UNREADABLE storage after an agent restart is the last known copy, not "never".
// nil = the pre-R-894 behaviour (in-memory record only).
LastKnownBackups LastKnownBackupStore
// Privileged runs the fenced root wrappers (E-2a: felhom-backup-target-apply). OPTIONAL — when
// nil, POST /backup/target reports "not configured". Satisfied by *proxmox.ExecRunner.
Privileged PrivilegedRunner
@@ -225,7 +240,6 @@ type Options struct {
// POST /escrow/recover-offsite-password. OPTIONAL — nil → that route reports "not configured"
// (503) instead of failing obscurely. Satisfied by escrow.OffsiteKeyRecoverer.
EscrowRecovery EscrowRecoverer
}
// defaultBackupCadence is the fallback /backup/due window when none is configured.
@@ -275,29 +289,34 @@ type Server struct {
// is the pre-R-82 shape.
tiers []BackupTier
// inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
inFlight *backup.InFlight
logger *slog.Logger
now func() time.Time
inFlight *backup.InFlight
afterPrimaryBackup func(ctx context.Context, vmid int) // the OS leg (agent v0.140.0); nil = none
lastKnown LastKnownBackupStore // R-894: on-disk newest success per tier; nil = none
logger *slog.Logger
now func() time.Time
disks DiskOps // slice 8C (optional)
diskGate StorageGate // slice 8C (optional)
guestList GuestLister // slice 8C (optional)
guestAttach GuestAttacher // slice 10 P2 (optional)
mem MemoryOps // v0.90.0 R-24 guest RAM resize (optional)
memMu sync.Mutex // single-flight around a resize apply (one customer per host)
netStorage NetworkStorageOps // Part A1: NAS network mounts (optional)
netMountRoot string // the user-data namespace root for the network-mount role gate
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
disks DiskOps // slice 8C (optional)
diskGate StorageGate // slice 8C (optional)
guestList GuestLister // slice 8C (optional)
guestAttach GuestAttacher // slice 10 P2 (optional)
mem MemoryOps // v0.90.0 R-24 guest RAM resize (optional)
memMu sync.Mutex // single-flight around a resize apply (one customer per host)
netStorage NetworkStorageOps // Part A1: NAS network mounts (optional)
netMountRoot string // the user-data namespace root for the network-mount role gate
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
// crashGuardStatePath (R-856) is the host crash guard's state file read by GET /host/crash-guard;
// empty = defaultCrashGuardStatePath. A seam: tests point it at a fixture.
crashGuardStatePath string
// escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed
// identity blob from the hub, unseal it with the customer's recovery code, return ONLY the
// offsite repository password. OPTIONAL — nil (no hub client configured) makes
// POST /escrow/recover-offsite-password answer 503 rather than pretending.
escrowRecovery EscrowRecoverer
intent IntentRecorder // slice 10 P3 (optional)
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
escrowRecovery EscrowRecoverer
intent IntentRecorder // slice 10 P3 (optional)
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
guestPower map[int]guestPowerState
@@ -469,6 +488,8 @@ func NewServer(o Options) (*Server, error) {
// the primary is always first, because that is what the untargeted endpoints act on.
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
s.inFlight = o.InFlight
s.afterPrimaryBackup = o.AfterPrimaryBackup
s.lastKnown = o.LastKnownBackups
if s.backups == nil && len(s.tiers) > 0 {
s.backups = s.tiers[0].Service
}
@@ -514,6 +535,9 @@ func (s *Server) Handler() http.Handler {
// Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring
// view. Host-wide, token-authed, fresh (a live collect, not the 15-min hub snapshot).
mux.HandleFunc("GET /host/metrics", s.withGuest(s.handleHostMetrics))
// R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller
// waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false.
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
@@ -893,7 +917,18 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive)
}
s.store.RecordBackup(b)
if b.Success && s.lastKnown != nil {
if err := s.lastKnown.RecordBackupSuccess(tier.TargetID, b); err != nil {
// Not fatal: the backup exists. Only the fallback after a restart loses this copy.
s.logger.Warn("local-api: could not save the backup on disk for the due-check fallback (R-894)",
"vmid", vmid, "target", tier.TargetID, "err", err)
}
}
s.finishJob(key, jobID, b)
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
s.afterPrimaryBackup(base, vmid)
}
}()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
}
@@ -1051,6 +1086,20 @@ func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid in
newest, haveNewest = t, true
unparseable = false // ground truth supersedes an unreadable in-memory timestamp
}
// R-894: the storage could not be read → the last success saved ON DISK stands in for the in-memory
// record a restart emptied. ONLY on archiveUnknown: a storage that answers is the ground truth, and an
// archive absent there must make the tier due even when the file remembers one (a pruned copy).
// A saved copy older than the cadence still reads DUE below — an unreadable storage never suppresses
// a backup that is due.
fromDisk := false
if lookup == archiveUnknown && s.lastKnown != nil {
if saved, ok := s.lastKnown.LastKnownSuccess(tier.TargetID, vmid); ok && (!haveNewest || saved.After(newest)) {
newest, haveNewest, fromDisk = saved, true, true
unparseable = false
s.logger.Info("local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)",
"vmid", vmid, "target", tier.TargetID, "saved", saved.UTC().Format(time.RFC3339))
}
}
if !haveNewest {
// R-88 Part 2: THREE distinct reasons for a nil age, each with its own state. Only ABSENT is a
// positive claim of "never backed up"; only that one may license the controller to bypass its
@@ -1073,13 +1122,17 @@ func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid in
}
age := s.now().Sub(newest)
ageSecs := int64(age.Seconds())
suffix := ""
if fromDisk {
suffix = " (storage unreadable — age from the last success saved on disk)"
}
if age >= tier.Cadence {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeSecs: &ageSecs, AgeState: AgeStateKnown,
Reason: "older than cadence", Target: echo})
Reason: "older than cadence" + suffix, Target: echo})
return
}
writeOK(w, BackupDueResponse{VMID: vmid, Due: false, AgeSecs: &ageSecs, AgeState: AgeStateKnown,
Reason: "within cadence window", Target: echo})
Reason: "within cadence window" + suffix, Target: echo})
}
// BackupTiersResponse is GET /backup/tiers (R-82): the tiers this agent serves, primary first.
@@ -1465,3 +1518,8 @@ func writeStatus(w http.ResponseWriter, code int, ok bool, data any, errMsg stri
w.WriteHeader(code)
_ = json.NewEncoder(w).Encode(apiResponse{OK: ok, Data: data, Error: errMsg})
}
// SetAfterPrimaryBackup wires the hook that runs after a successful primary-tier backup (the OS leg, agent v0.140.0).
func (s *Server) SetAfterPrimaryBackup(fn func(ctx context.Context, vmid int)) {
s.afterPrimaryBackup = fn
}
+31 -17
View File
@@ -154,12 +154,15 @@ func (s *TokenStore) Mint(vmid int) (string, error) {
// looks it up; the per-candidate comparison is constant-time to avoid a timing oracle on the
// stored hash. ok is false for an unknown/empty token.
//
// Reload-on-miss (B3): the store FILE is shared across processes — the one-shot provisioner
// (`--selftest=provision`) Mints into it while the long-lived daemon serves Lookup from an index
// built at open. On a miss, re-read the file ONCE and re-check, so a token minted after this
// process started authorizes without a daemon restart (the drill's fresh-install 401). The
// append-only log makes an unchanged file size proof of no new records, so a genuinely unknown
// token costs at most one stat once the index is current — never a reload loop.
// Reload-on-change (B3, R-269): the store FILE is shared across processes — the one-shot
// provisioner (`--selftest=provision`) Mints into it while the long-lived daemon serves Lookup from
// an index built at open. Every Lookup stats the file first and re-reads it when the append-only
// log has grown, BEFORE answering — so a token minted elsewhere authorizes without a restart AND a
// token rotated out elsewhere stops authorizing on its very next presentation. (Before R-269 the
// re-read ran only on a MISS, so a superseded token was a direct map hit and kept authorizing until
// some unrelated miss forced the reload.) An unchanged size is proof of no new records, so the
// steady state costs one stat per call and never a reload loop. Pinned by
// TestTokenStore_RotatedOutTokenRejectedFirst.
func (s *TokenStore) Lookup(token string) (int, bool) {
if token == "" {
return 0, false
@@ -167,23 +170,34 @@ func (s *TokenStore) Lookup(token string) (int, bool) {
want := hashToken(token)
s.mu.Lock()
defer s.mu.Unlock()
// Direct map hit is the common path; the constant-time compare guards against a timing
// side-channel by re-checking the matched key (map lookup itself is not the secret-bearing
// comparison — the hash of a random 256-bit token is not feasibly guessable regardless).
if vmid, ok := s.byHash[want]; ok {
if subtle.ConstantTimeCompare([]byte(want), []byte(s.byVMID[vmid])) == 1 {
return vmid, true
st, statErr := os.Stat(s.path)
if statErr == nil && st.Size() != s.loadedSize {
// The log changed under us (another process minted/rotated): converge first, then answer.
s.reloads++
if err := s.reloadLocked(); err != nil {
return 0, false // unreadable store: fail closed, never crash the auth path
}
return s.matchLocked(want)
}
// Miss: skip the re-read when the append-only log has not grown (nothing new to see).
// A stat error falls through to the reload, which handles a missing file as empty.
if st, err := os.Stat(s.path); err == nil && st.Size() == s.loadedSize {
return 0, false
if vmid, ok := s.matchLocked(want); ok {
return vmid, true
}
if statErr == nil {
return 0, false // file unchanged since the last (re)load: genuinely unknown
}
// Stat failed (e.g. the file vanished): reload, which treats a missing file as empty.
s.reloads++
if err := s.reloadLocked(); err != nil {
return 0, false // unreadable store: fail closed, never crash the auth path
return 0, false
}
return s.matchLocked(want)
}
// matchLocked answers from the in-memory index. Direct map hit is the common path; the
// constant-time compare re-checks the matched key against the guest's CURRENT hash (map lookup
// itself is not the secret-bearing comparison — the hash of a random 256-bit token is not
// feasibly guessable regardless). Caller holds the mutex.
func (s *TokenStore) matchLocked(want string) (int, bool) {
if vmid, ok := s.byHash[want]; ok {
if subtle.ConstantTimeCompare([]byte(want), []byte(s.byVMID[vmid])) == 1 {
return vmid, true
+45
View File
@@ -210,6 +210,51 @@ func TestTokenStore_ReloadOnMiss_RemintCoherence(t *testing.T) {
}
}
// R-269: a token rotated out by ANOTHER process must stop authorizing on its very next
// presentation — with NO intervening lookup of the new token. This is the order the operator hits
// after rotating a leaked token: the leaked one is presented first. RemintCoherence above looks the
// NEW token up first, and that miss is what used to evict the old hash, so it passed while the leaked
// token kept returning HTTP 200 on hardware (2026-08-09) until something unrelated forced a reload.
//
// RED-PROOF: restore the reload-on-MISS-only Lookup (answer a map hit before stat-ing the file) and
// this fails with "rotated-out token still authorizes".
func TestTokenStore_RotatedOutTokenRejectedFirst(t *testing.T) {
path := filepath.Join(t.TempDir(), "tokens.log")
daemon, err := OpenTokenStore(path)
if err != nil {
t.Fatalf("open daemon store: %v", err)
}
defer daemon.Close()
minter, err := OpenTokenStore(path)
if err != nil {
t.Fatalf("open minter store: %v", err)
}
defer minter.Close()
old, err := minter.Mint(130)
if err != nil {
t.Fatalf("mint old: %v", err)
}
if vmid, ok := daemon.Lookup(old); !ok || vmid != 130 { // the daemon has learned the old token
t.Fatalf("old token before rotation: (%d,%v), want (130,true)", vmid, ok)
}
fresh, err := minter.Mint(130) // rotation, written by another process
if err != nil {
t.Fatalf("mint fresh: %v", err)
}
if vmid, ok := daemon.Lookup(old); ok { // the leaked token FIRST
t.Fatalf("rotated-out token still authorizes vmid %d on its first presentation after rotation — "+
"Mint's 'any previous token for this guest is revoked' is false across processes (R-269)", vmid)
}
if vmid, ok := daemon.Lookup(fresh); !ok || vmid != 130 {
t.Fatalf("fresh token after rotation: (%d,%v), want (130,true)", vmid, ok)
}
if vmid, ok := daemon.Lookup(old); ok {
t.Fatalf("rotated-out token authorizes vmid %d after the fresh one was seen", vmid)
}
}
// §8 edge: the store file deleted between open and a miss — reload treats it as empty; Lookup
// fails closed, no crash.
func TestTokenStore_ReloadOnMiss_MissingFile(t *testing.T) {
+157
View File
@@ -0,0 +1,157 @@
package osupdate
import (
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"regexp"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// OpConfigUpdate is the signed op that brings a box's ROOT-OWNED files (sudoers, wrappers, units) to a release's config
// bundle (R-840, `11` §5.4.2). The params pin the agent version and the bundle's sha256; the root wrapper re-verifies
// the signature, the host binding, the nonce and the sha ITSELF — this executor is only the courier.
const OpConfigUpdate = "agent_config_update"
// BundleFileName is the bundle's name beside the binary in the Gitea generic package felhom-agent/<version>/.
const BundleFileName = "felhom-config-bundle.json"
var (
bundleVersionRe = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$`)
bundleSHARe = regexp.MustCompile(`^[0-9a-f]{64}$`)
)
// ConfigUpdateParams are the signed params (the wrapper reads the same two names out of the signed blob).
type ConfigUpdateParams struct {
AgentVersion string `json:"agent_version"`
BundleSHA256 string `json:"bundle_sha256"`
}
// BundleURL derives the bundle's URL from the agent binary's URL template (".../felhom-agent/{version}/felhom-agent").
func BundleURL(binaryTemplate, version string) (string, error) {
if !strings.HasSuffix(binaryTemplate, "/felhom-agent") || !strings.Contains(binaryTemplate, "{version}") {
return "", fmt.Errorf("cannot derive the bundle URL from %q (want …/{version}/felhom-agent)", binaryTemplate)
}
t := strings.TrimSuffix(binaryTemplate, "felhom-agent") + BundleFileName
return strings.ReplaceAll(t, "{version}", version), nil
}
// ConfigUpdateExecutor runs a verified agent_config_update (signedjobs.Executor).
type ConfigUpdateExecutor struct {
Leg *Leg
URLTemplate string // the agent binary's template (config.SelfUpdate.URLTemplate)
Username string
Token string
HTTPClient *http.Client
// AfterInstall runs after a bundle installed (the capability re-probe; nil = none).
AfterInstall func(ctx context.Context)
}
// Execute implements signedjobs.Executor.
func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
if op != OpConfigUpdate {
return signedjobs.ErrNoExecutor
}
so, ok := signedjobs.SignedOpFrom(ctx)
if !ok {
return fmt.Errorf("agent_config_update: no signed envelope in the context — the wrapper could not verify it")
}
var p ConfigUpdateParams
if err := json.Unmarshal(params, &p); err != nil {
return fmt.Errorf("agent_config_update: bad params: %w", err)
}
if !bundleVersionRe.MatchString(p.AgentVersion) || !bundleSHARe.MatchString(p.BundleSHA256) {
return fmt.Errorf("agent_config_update: params must pin agent_version (semver) and bundle_sha256 (64 hex)")
}
lg := e.Leg.log().With("op", OpConfigUpdate, "agent_version", p.AgentVersion)
url, err := BundleURL(e.URLTemplate, p.AgentVersion)
if err != nil {
return fmt.Errorf("agent_config_update: %w", err)
}
dir := e.Leg.PlanDir
if dir == "" {
dir = DefaultPlanDir
}
if err := os.MkdirAll(dir, 0o700); err != nil {
return fmt.Errorf("agent_config_update: plan dir: %w", err)
}
path := filepath.Join(dir, "bundle-"+p.AgentVersion+".json")
start := time.Now()
got, err := e.download(ctx, url, path)
if err != nil {
_ = os.Remove(path)
return fmt.Errorf("agent_config_update: download %s: %w", url, err)
}
defer os.Remove(path)
// The agent's own check is a courtesy (an early, clear error); the wrapper's is the gate.
if got != p.BundleSHA256 {
return fmt.Errorf("agent_config_update: the downloaded bundle's sha256 is %s, the signed job pins %s — nothing installed", got, p.BundleSHA256)
}
lg.Info("osupdate: config bundle downloaded; handing it to the root wrapper", "sha256", got[:16], "duration_ms", time.Since(start).Milliseconds())
runID := "bundle-" + e.Leg.now().UTC().Format("20060102T150405Z")
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-" + p.AgentVersion, "layer": LayerHost,
"mode": "bundle", "bundle": path,
"signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(so.Blob), "sig": string(so.Sig)}})
if err != nil {
return fmt.Errorf("agent_config_update: %w", err)
}
if wr.refused() {
lg.Warn("osupdate: config bundle REFUSED by the wrapper — nothing changed", "refused", string(wr.Refused))
return fmt.Errorf("agent_config_update: refused: %s", wr.Refused)
}
if wr.failed() {
lg.Error("osupdate: config bundle FAILED — the wrapper put the previous files back", "failed", string(wr.Failed), "bundle", string(wr.Bundle))
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
}
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
if e.AfterInstall != nil {
e.AfterInstall(ctx)
}
return nil
}
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
hc := e.HTTPClient
if hc == nil {
hc = &http.Client{Timeout: 2 * time.Minute}
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return "", err
}
if e.Username != "" || e.Token != "" {
req.SetBasicAuth(e.Username, e.Token)
}
resp, err := hc.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", fmt.Errorf("HTTP %d", resp.StatusCode)
}
f, err := os.OpenFile(dest, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
if err != nil {
return "", err
}
h := sha256.New()
// 4 MB is the wrapper's own limit; read one byte more so an oversized bundle fails there, visibly.
if _, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 4*1024*1024+1)); err != nil {
f.Close()
return "", err
}
if err := f.Close(); err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
+155
View File
@@ -0,0 +1,155 @@
package osupdate
import (
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"io"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// bundleWrapper plays felhom-os-apply for mode "bundle": it records the plan and the bundle file's bytes AT CALL TIME
// (the executor deletes the file afterwards), and answers with rep.
type bundleWrapper struct {
t *testing.T
rep string
plans []map[string]any
bodies [][]byte
}
func (b *bundleWrapper) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
if name != WrapperPath || len(args) != 2 || args[0] != "--plan" {
b.t.Fatalf("unexpected command %s %v", name, args)
}
raw, err := os.ReadFile(args[1])
if err != nil {
b.t.Fatal(err)
}
var plan map[string]any
_ = json.Unmarshal(raw, &plan)
b.plans = append(b.plans, plan)
body, _ := os.ReadFile(plan["bundle"].(string))
b.bodies = append(b.bodies, body)
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
}
func serveBundle(t *testing.T, body []byte) (*httptest.Server, string) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/felhom-agent/0.143.0/"+BundleFileName {
http.NotFound(w, r)
return
}
_, _ = w.Write(body)
}))
t.Cleanup(srv.Close)
sum := sha256.Sum256(body)
return srv, hex.EncodeToString(sum[:])
}
func bundleExec(t *testing.T, w *bundleWrapper, srvURL string) (ConfigUpdateExecutor, *bool) {
l, _ := newLeg(t, &fakeWrapper{t: t}, nil)
l.Runner = w
called := false
return ConfigUpdateExecutor{Leg: l, URLTemplate: srvURL + "/felhom-agent/{version}/felhom-agent",
AfterInstall: func(context.Context) { called = true }}, &called
}
func signedCtx() context.Context {
return signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"agent_config_update"}`), Sig: []byte("SIG")})
}
func params(v, sha string) json.RawMessage {
p, _ := json.Marshal(ConfigUpdateParams{AgentVersion: v, BundleSHA256: sha})
return p
}
// The courier hands the wrapper the bundle bytes it downloaded and the RAW signed envelope (the wrapper verifies both
// itself), then runs the capability probe. Red-proof: drop "signed" from the plan → the plan check below fails.
func TestConfigUpdate_PassesBundleAndEnvelopeToTheWrapper(t *testing.T) {
body := []byte(`{"format":1,"agent_version":"0.143.0","files":[]}`)
srv, sha := serveBundle(t, body)
w := &bundleWrapper{t: t, rep: `{"mode":"bundle","bundle":{"agent_version":"0.143.0","written":["/etc/sudoers.d/felhom-agent"]}}`}
e, called := bundleExec(t, w, srv.URL)
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err != nil {
t.Fatal(err)
}
p := w.plans[0]
sg, _ := p["signed"].(map[string]any)
if p["mode"] != "bundle" || p["layer"] != "host" || sg == nil || sg["sig"] != "SIG" ||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"agent_config_update"}`)) {
t.Fatalf("plan = %v", p)
}
if string(w.bodies[0]) != string(body) || !strings.HasSuffix(p["bundle"].(string), "/bundle-0.143.0.json") {
t.Fatalf("the wrapper got %q at %v", w.bodies[0], p["bundle"])
}
if !*called {
t.Fatal("the capability probe must run after an install")
}
if _, err := os.Stat(p["bundle"].(string)); !os.IsNotExist(err) {
t.Fatal("the downloaded bundle must be removed after the call")
}
}
func TestConfigUpdate_WrongShaNeverReachesTheWrapper(t *testing.T) {
srv, _ := serveBundle(t, []byte("tampered"))
w := &bundleWrapper{t: t}
e, called := bundleExec(t, w, srv.URL)
err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", strings.Repeat("a", 64)))
if err == nil || len(w.plans) != 0 || *called {
t.Fatalf("err=%v plans=%d", err, len(w.plans))
}
}
func TestConfigUpdate_RefusedAndFailedAreErrors(t *testing.T) {
for _, rep := range []string{`{"refused":{"code":"R17","reason":"trust root"}}`, `{"failed":{"rc":3},"bundle":{"rolled_back":["/x"]}}`} {
srv, sha := serveBundle(t, []byte("{}"))
w := &bundleWrapper{t: t, rep: rep}
e, called := bundleExec(t, w, srv.URL)
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err == nil || *called {
t.Fatalf("%s: err=%v called=%v", rep, err, *called)
}
}
}
func TestConfigUpdate_GuardsBeforeAnyDownload(t *testing.T) {
w := &bundleWrapper{t: t}
e, _ := bundleExec(t, w, "http://127.0.0.1:1")
if err := e.Execute(signedCtx(), "agent_update", nil); !errors.Is(err, signedjobs.ErrNoExecutor) {
t.Fatalf("another op must pass through the chain: %v", err)
}
if err := e.Execute(context.Background(), OpConfigUpdate, params("0.143.0", strings.Repeat("a", 64))); err == nil {
t.Fatal("no envelope must refuse")
}
for _, p := range []json.RawMessage{params("0.143", strings.Repeat("a", 64)), params("0.143.0", "ABC"), json.RawMessage(`nope`)} {
if err := e.Execute(signedCtx(), OpConfigUpdate, p); err == nil {
t.Fatalf("bad params %s must refuse", p)
}
}
if len(w.plans) != 0 {
t.Fatal("no wrapper call on a refusal")
}
}
func TestBundleURL(t *testing.T) {
u, err := BundleURL("https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/{version}/felhom-agent", "0.143.0")
if err != nil || u != "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.143.0/felhom-config-bundle.json" {
t.Fatalf("%s %v", u, err)
}
if _, err := BundleURL("https://example/felhom-agent-{version}.bin", "0.143.0"); err == nil {
t.Fatal("an underivable template must refuse")
}
}
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return b.Run(ctx, name, args...)
}
+91
View File
@@ -0,0 +1,91 @@
package osupdate
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// OpDockerStep is the signed op class of a Docker engine step (`11` §5.8): a ring-1 box takes an approved engine set,
// and every box takes an UNDO, only through it. CC may sign it until the first paying customer (R-530 ruling).
const OpDockerStep = "os_docker_step"
// DockerStepParams are the signed params. The wrapper compares Packages and Undo with the plan byte-for-byte.
type DockerStepParams struct {
ReleaseID string `json:"release_id"`
Packages []Package `json:"packages"`
Undo bool `json:"undo"`
VMID int `json:"vmid,omitempty"`
}
// DockerStepExecutor runs a verified os_docker_step (signedjobs.Executor). Guest finds the box's customer guest when
// the params name none; Gate (optional) takes the host-wide heavy-op gate so a step never runs beside a backup.
type DockerStepExecutor struct {
Leg *Leg
Guest func(ctx context.Context) (int, error)
Gate func(ctx context.Context) (release func(), err error)
}
// Execute implements signedjobs.Executor.
func (e DockerStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
if op != OpDockerStep {
return signedjobs.ErrNoExecutor
}
so, ok := signedjobs.SignedOpFrom(ctx)
if !ok {
return fmt.Errorf("os_docker_step: no signed envelope in the context — the wrapper could not verify it")
}
var p DockerStepParams
if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 {
return fmt.Errorf("os_docker_step: params must name the engine set: %v", err)
}
vmid := p.VMID
if vmid == 0 {
if e.Guest == nil {
return fmt.Errorf("os_docker_step: no vmid and no guest finder")
}
v, err := e.Guest(ctx)
if err != nil {
return fmt.Errorf("os_docker_step: find the customer guest: %w", err)
}
vmid = v
}
if e.Gate != nil {
release, err := e.Gate(ctx)
if err != nil {
return fmt.Errorf("os_docker_step: heavy-op gate busy (a backup or restore-test runs): %w", err)
}
defer release()
}
rep := e.Leg.RunDockerSigned(ctx, vmid, p, so.Blob, string(so.Sig))
switch rep.Outcome {
case "applied", "nothing":
if rep.Healthy {
return nil
}
}
return fmt.Errorf("os_docker_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused))
}
// RunDockerSigned is one signed Docker step (ring 1 or an undo): live-restore first (decision 87, a no-op when on),
// then the docker layer with the signed envelope, which the wrapper verifies itself.
func (l *Leg) RunDockerSigned(ctx context.Context, vmid int, p DockerStepParams, blob []byte, sig string) Report {
unlock := l.lockPass(true)
defer unlock()
l.sendUnsentLocked(ctx) // R-868
runID := l.now().UTC().Format("20060102T150405Z")
lg := l.log().With("run", runID, "vmid", vmid, "trigger", "signed", "release", p.ReleaseID, "undo", p.Undo)
if err := l.EnsureLiveRestore(ctx, runID, vmid); err != nil {
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerDocker, Trigger: "signed", Ring: l.Block().Ring, VMID: vmid,
Mode: "apply", ReleaseID: p.ReleaseID, Outcome: "failed", HealthReason: "live-restore could not be turned on: " + err.Error()})
}
rid := p.ReleaseID
if rid == "" {
rid = "signed-" + runID
}
return l.runLayer(ctx, runID, LayerDocker, vmid, "signed", l.Block(), dockerOpts{releaseID: rid, packages: p.Packages,
undo: p.Undo, signed: map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}})
}
+49
View File
@@ -0,0 +1,49 @@
package osupdate
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// The executor hands the RAW signed bytes to the wrapper (which verifies them itself) and the exact signed package
// list. Red-proof: drop the `signed` field from the docker plan in runLayer and the plan check fails.
func TestDockerStepExecutor_PassesTheSignedEnvelope(t *testing.T) {
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerDocker: {
Upgraded: []Package{{Name: "docker-ce", Version: "5:29.8.2-1~debian.13~trixie"}}, DockerEngine: "29.8.2", Authority: "signed"}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
e := DockerStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
params, _ := json.Marshal(DockerStepParams{ReleaseID: "os-docker-1", Packages: []Package{{Name: "docker-ce", Version: "5:29.8.2-1~debian.13~trixie", Origin: "Docker CE"}}})
ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_docker_step"}`), Sig: []byte("SIG")})
if err := e.Execute(ctx, OpDockerStep, params); err != nil {
t.Fatal(err)
}
dp := w.plans[len(w.plans)-1]
sg, _ := dp["signed"].(map[string]any)
if dp["layer"] != "docker" || dp["lane"] != "slow" || dp["release_id"] != "os-docker-1" || sg == nil ||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_docker_step"}`)) || sg["sig"] != "SIG" {
t.Fatalf("docker plan = %v", dp)
}
if calls(w) != "guest:live-restore-on,docker:apply" || len(h.reports) != 1 || h.reports[0].Trigger != "signed" {
t.Fatalf("calls=%s reports=%+v", calls(w), h.reports)
}
}
func TestDockerStepExecutor_RefusesWithoutEnvelopeAndPassesOtherOps(t *testing.T) {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
e := DockerStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
if err := e.Execute(context.Background(), "agent_update", nil); !errors.Is(err, signedjobs.ErrNoExecutor) {
t.Fatalf("another op must pass through the chain: %v", err)
}
params, _ := json.Marshal(DockerStepParams{Packages: []Package{{Name: "docker-ce", Version: "1"}}})
if err := e.Execute(context.Background(), OpDockerStep, params); err == nil || len(w.plans) != 0 {
t.Fatalf("no envelope must refuse before any wrapper call: err=%v calls=%s", err, calls(w))
}
}
+774
View File
@@ -0,0 +1,774 @@
// Package osupdate is the agent's OS-update leg (`11-os-updates.md` §8 steps 2–3, §5.8): the customer GUEST's Debian
// fast lane (agent v0.140.0), after it in the same pass the HOST's (agent v0.141.0), and then — ring 0 only — the
// guest's DOCKER engine set, the slow lane (agent v0.142.0; a ring-1 box takes a Docker step only inside a signed
// operator job, DockerStepExecutor). It also reads the box's versions for the hub's System page (Facts, R-852).
//
// It runs right after the night's successful whole-guest backup, while the backup goroutine still holds the host-wide
// heavy-op gate (so it never overlaps a backup or a restore-test, `11` C10), at most once per night. All root work is
// the wrapper `felhom-os-apply` (configs/, its own tests); this package only builds plans, calls the wrapper through
// sudo, judges health and reports to the hub — one report per layer.
//
// NO AUTOMATIC UNDO (R-837 measured; `09` §3 decision 81): a failed health check stops, reports `health_failed` and the
// hub mails the operator; the whole-guest backup taken minutes earlier is the guest's undo, by hand; a host package is
// put back by hand from the previous release's snapshot (runbook). The host is NEVER rebooted by this package.
package osupdate
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// WrapperPath is the pinned sudoers vector (configs/felhom-agent.sudoers FELHOM_OSAPPLY).
const WrapperPath = "/usr/local/sbin/felhom-os-apply"
// DefaultPlanDir is where plans are written (the sudoers glob names it).
const DefaultPlanDir = "/var/lib/felhom-agent/os"
// Layers.
const (
LayerGuest = "guest"
LayerHost = "host"
LayerDocker = "docker" // the guest's Docker engine set — slow lane (`11` §5.8)
)
// DockerNames are the six packages of the Docker engine set (the wrapper's DOCKER_NAMES).
var DockerNames = map[string]bool{"containerd.io": true, "docker-buildx-plugin": true, "docker-ce": true,
"docker-ce-cli": true, "docker-ce-rootless-extras": true, "docker-compose-plugin": true}
// Package is one name=version with its origin.
type Package struct {
Name string `json:"name"`
Version string `json:"version"`
Origin string `json:"origin"`
}
// Pending is one update the sources offer (origin as apt names it, possibly several).
type Pending struct {
Name string `json:"name"`
From string `json:"from"`
To string `json:"to"`
Origin []string `json:"origin"`
}
// Container is one container's state as the wrapper saw it.
type Container struct {
State string `json:"state"`
Health string `json:"health"` // healthy | unhealthy | starting | none
ID string `json:"id,omitempty"`
}
// Health is one health reading. Guest layer: DockerOK..Containers. Host layer: HostServices, GuestRunning and the
// guest's own reading in Guest.
type Health struct {
DockerOK bool `json:"docker_ok"`
NetworkOK bool `json:"network_ok"`
Controller string `json:"controller"`
Containers map[string]Container `json:"containers"`
// ControllerDockerOK: the controller reaches the engine from INSIDE its container (R-858, wrapper ≥ v0.142.1;
// nil from an older wrapper = not checked). Its own health check stayed "healthy" while it was blind.
ControllerDockerOK *bool `json:"controller_docker_ok,omitempty"`
HostServices map[string]string `json:"host_services,omitempty"`
GuestRunning *bool `json:"guest_running,omitempty"`
Guest *Health `json:"guest,omitempty"`
}
// WrapperReport is the wrapper's OSAPPLY-REPORT object.
type WrapperReport struct {
Mode string `json:"mode"`
Layer string `json:"layer"`
Refused json.RawMessage `json:"refused"`
Failed json.RawMessage `json:"failed"`
Upgraded []Package `json:"upgraded"`
Installed []Package `json:"installed"`
Pending []Pending `json:"pending"`
RestartNeeded []string `json:"restart_needed"`
DockerRestartNeeded bool `json:"docker_restart_needed"`
RebootNeeded bool `json:"reboot_needed"`
RebootScanned bool `json:"reboot_scanned"`
HealthBefore *Health `json:"health_before"`
HealthAfter *Health `json:"health_after"`
Health *Health `json:"health"`
PassSeconds float64 `json:"pass_seconds"`
DockerEngine string `json:"docker_engine"`
Authority string `json:"authority"`
Undo bool `json:"undo"`
LiveRestore json.RawMessage `json:"live_restore"`
Facts json.RawMessage `json:"facts"`
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
OOMCheck json.RawMessage `json:"oom_check"`
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
// agent process that started the pass. ReleaseID / VMID were always in the report.
RunID string `json:"run_id"`
Trigger string `json:"trigger"`
Ring *int `json:"ring"`
ReleaseID string `json:"release_id"`
VMID int `json:"vmid"`
}
func (w WrapperReport) refused() bool { return len(w.Refused) > 0 && string(w.Refused) != "null" }
func (w WrapperReport) failed() bool { return len(w.Failed) > 0 && string(w.Failed) != "null" }
// Report is what the hub receives per layer (hub osupdates.Report — field-exact).
type Report struct {
RunID string `json:"run_id"`
Layer string `json:"layer"`
Trigger string `json:"trigger"`
Mode string `json:"mode"`
Ring int `json:"ring"`
ReleaseID string `json:"release_id"`
Outcome string `json:"outcome"`
Healthy bool `json:"healthy"`
HealthReason string `json:"health_reason,omitempty"`
VMID int `json:"vmid"`
Upgraded []Package `json:"upgraded,omitempty"`
Installed []Package `json:"installed,omitempty"`
Pending []Pending `json:"pending,omitempty"`
NotCovered []string `json:"not_covered,omitempty"`
RestartNeeded []string `json:"restart_needed,omitempty"`
DockerRestartNeeded bool `json:"docker_restart_needed,omitempty"`
RebootNeeded bool `json:"reboot_needed,omitempty"`
RebootScanned bool `json:"reboot_scanned,omitempty"` // the pass looked (host: every pass) — a false RebootNeeded then means "not needed"
Refused json.RawMessage `json:"refused,omitempty"`
PassSeconds float64 `json:"pass_seconds,omitempty"`
DockerEngine string `json:"docker_engine,omitempty"` // docker layer: the engine after the step
Authority string `json:"authority,omitempty"` // docker layer: ring0 | signed
Undo bool `json:"undo,omitempty"` // docker layer: a signed undo (downgrade)
// OOMCheck: docker layer — the wrapper's oom_check object, byte-for-byte (R-528; the hub decides approval on it).
// Pinned by TestDocker_OOMCheckReachesTheHubUnchanged and TestR868_KeptCopyCarriesTheOOMCheck.
OOMCheck json.RawMessage `json:"oom_check,omitempty"`
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
}
// Reporter posts a report to the hub (*hub.Client).
type Reporter interface {
PostOSReport(ctx context.Context, body []byte) error
}
// Leg runs one OS-update pass for the customer guest and then the host.
type Leg struct {
Runner proxmox.Runner
Hub Reporter
Tunnel hub.CloudflaredProber // the host health rule needs the tunnel `running` (R-841)
Appliance bool // agent.json deployment_mode; the wrapper re-checks the ROOT-owned record (R12)
Logger *slog.Logger
PlanDir string
StatePath string // last night run (once per night)
HealthWait time.Duration // how long health may take to come back (default 5 min)
HealthPoll time.Duration // default 15 s
MinGap time.Duration // between night runs (default 20 h)
Now func() time.Time
Sleep func(context.Context, time.Duration)
mu sync.Mutex
block *hub.WireOSUpdate
}
// planFile / reportFile: the plan the agent writes and the copy of the report the wrapper keeps beside it (R-868).
func planFile(dir, runID, layer, mode string) string {
return filepath.Join(dir, fmt.Sprintf("plan-%s-%s-%s.json", runID, layer, mode))
}
func reportFile(dir, runID, layer, mode string) string {
return filepath.Join(dir, fmt.Sprintf("report-%s-%s-%s.json", runID, layer, mode))
}
func (l *Leg) planDir() string {
if l.PlanDir == "" {
return DefaultPlanDir
}
return l.PlanDir
}
// OnDesiredState stores the hub's os_update block (desired.RawConsumer — store only, never block).
func (l *Leg) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
if resp == nil {
return
}
l.mu.Lock()
l.block = resp.DesiredState.OSUpdate
l.mu.Unlock()
l.saveBlock(resp.DesiredState.OSUpdate)
}
// SavedBlockFile is the hub's newest os_update block as the daemon last received it (R-866, v0.144.0): the debug
// pass falls back to it when the hub cannot be reached, and says so.
const SavedBlockFile = "os-update-block.json"
type savedBlock struct {
SavedAt time.Time `json:"saved_at"`
Block *hub.WireOSUpdate `json:"block"`
}
func (l *Leg) saveBlock(b *hub.WireOSUpdate) {
dir := l.planDir()
if err := os.MkdirAll(dir, 0o700); err != nil {
return
}
body, _ := json.Marshal(savedBlock{SavedAt: l.now().UTC(), Block: b})
tmp := filepath.Join(dir, SavedBlockFile+".tmp")
if err := os.WriteFile(tmp, body, 0o600); err == nil {
_ = os.Rename(tmp, filepath.Join(dir, SavedBlockFile))
}
}
// LoadSavedBlock reads the block the daemon saved (R-866). ok=false: none saved yet.
func LoadSavedBlock(dir string) (b *hub.WireOSUpdate, savedAt time.Time, ok bool) {
raw, err := os.ReadFile(filepath.Join(dir, SavedBlockFile))
if err != nil {
return nil, time.Time{}, false
}
var s savedBlock
if json.Unmarshal(raw, &s) != nil {
return nil, time.Time{}, false
}
return s.Block, s.SavedAt, true
}
// Block returns the newest os_update block. No block (an older hub, or nothing fetched yet) = ring 1, ON, no
// release: the box reports and installs nothing.
func (l *Leg) Block() hub.WireOSUpdate {
l.mu.Lock()
defer l.mu.Unlock()
if l.block == nil {
return hub.WireOSUpdate{Ring: 1, Enabled: true}
}
return *l.block
}
// SetBlock sets the block directly (the selftest fetches the desired state itself).
func (l *Leg) SetBlock(b *hub.WireOSUpdate) {
l.mu.Lock()
defer l.mu.Unlock()
l.block = b
}
func (l *Leg) now() time.Time {
if l.Now != nil {
return l.Now()
}
return time.Now()
}
func (l *Leg) log() *slog.Logger {
if l.Logger != nil {
return l.Logger
}
return slog.Default()
}
func (l *Leg) sleep(ctx context.Context, d time.Duration) {
if l.Sleep != nil {
l.Sleep(ctx, d)
return
}
select {
case <-ctx.Done():
case <-time.After(d):
}
}
// IsFast reports whether every origin apt names is Debian / Debian-Security (the fast lane, `11` C3).
func IsFast(origins []string) bool {
if len(origins) == 0 {
return false
}
for _, o := range origins {
if o != "Debian" && o != "Debian-Security" {
return false
}
}
return true
}
// HealthVerdict is THE guest health rule (`11` §8.1; pinned by TestHealthVerdict*): docker answers, the guest's
// network resolves, the controller's own health check is `healthy`, and every container that was running at the
// start of the pass runs again — and healthy again if it was. "starting" is not yet healthy.
func HealthVerdict(before, after *Health) (bool, string) {
if after == nil {
return false, "no health reading"
}
if !after.DockerOK {
return false, "docker does not answer"
}
if !after.NetworkOK {
return false, "the guest cannot resolve deb.debian.org"
}
if after.Controller != "healthy" {
return false, "the controller is " + after.Controller
}
if after.ControllerDockerOK != nil && !*after.ControllerDockerOK {
return false, "the controller cannot reach Docker (it holds an old socket — R-858)"
}
if before == nil {
return true, ""
}
names := make([]string, 0, len(before.Containers))
for n := range before.Containers {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
b := before.Containers[n]
if b.State != "running" {
continue
}
a, ok := after.Containers[n]
if !ok || a.State != "running" {
return false, n + " was running and is not"
}
if b.Health == "healthy" && a.Health != "healthy" {
return false, n + " was healthy and is " + a.Health
}
}
return true, ""
}
// HostHealthVerdict is THE host health rule (`11` §8.2; pinned by TestHostHealthVerdict): the Proxmox daemons and the
// agent are active, the customer guest still runs, the guest's own rule still passes against the start of the pass,
// and the tunnel is `running` (R-841).
func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
if after == nil {
return false, "no health reading"
}
svcs := make([]string, 0, len(after.HostServices))
for s := range after.HostServices {
svcs = append(svcs, s)
}
sort.Strings(svcs)
if len(svcs) == 0 {
return false, "no host service reading"
}
for _, s := range svcs {
if after.HostServices[s] != "active" {
return false, s + " is " + after.HostServices[s]
}
}
if after.GuestRunning == nil || !*after.GuestRunning {
return false, "the customer guest is not running"
}
var gb *Health
if before != nil {
gb = before.Guest
}
if ok, why := HealthVerdict(gb, after.Guest); !ok {
return false, "guest: " + why
}
if tunnel != hub.TunnelRunning {
return false, "the tunnel is " + tunnel
}
return true, ""
}
// EngineOf is the engine version `docker version` prints for a docker-ce package version: "5:29.8.2-1~debian.13~trixie"
// → "29.8.2" (no epoch, no Debian revision).
func EngineOf(pkgVersion string) string {
v := pkgVersion
if i := strings.Index(v, ":"); i >= 0 {
v = v[i+1:]
}
if i := strings.Index(v, "-"); i >= 0 {
v = v[:i]
}
return v
}
// DockerHealthVerdict is THE Docker-step health rule (`11` §5.8; pinned by TestDockerHealthVerdict): the guest rule,
// plus every container running at the start still runs as the SAME container (same id — a changed id means the
// household's apps restarted, which `live-restore` exists to prevent), plus the engine now reports the version the step
// installed (wantEngine "" = no engine change expected).
func DockerHealthVerdict(before, after *Health, wantEngine, gotEngine string) (bool, string) {
if ok, why := HealthVerdict(before, after); !ok {
return false, why
}
if before != nil {
names := make([]string, 0, len(before.Containers))
for n := range before.Containers {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
b := before.Containers[n]
if b.State != "running" || b.ID == "" {
continue
}
if a := after.Containers[n]; a.ID != b.ID {
return false, n + " is a new container (id changed) — the engine step restarted it"
}
}
}
if wantEngine != "" && gotEngine != wantEngine {
return false, "the engine is " + gotEngine + ", not " + wantEngine
}
return true, ""
}
// call writes the plan and runs the wrapper once.
func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (WrapperReport, error) {
dir := l.planDir()
if err := os.MkdirAll(dir, 0o700); err != nil {
return WrapperReport{}, fmt.Errorf("osupdate: plan dir: %w", err)
}
b, _ := json.Marshal(plan)
path := planFile(dir, runID, fmt.Sprint(plan["layer"]), fmt.Sprint(plan["mode"]))
if err := os.WriteFile(path, b, 0o600); err != nil {
return WrapperReport{}, fmt.Errorf("osupdate: write plan: %w", err)
}
defer os.Remove(path)
stdout, stderr, err := l.Runner.Run(ctx, WrapperPath, "--plan", path)
for _, line := range strings.Split(strings.TrimSpace(string(stderr)), "\n") {
if strings.HasPrefix(line, "os-apply: ") {
l.log().Info("osupdate: wrapper", "line", line)
}
}
var rep WrapperReport
found := false
for _, line := range strings.Split(string(stdout), "\n") {
if strings.HasPrefix(line, "OSAPPLY-REPORT ") {
if jerr := json.Unmarshal([]byte(strings.TrimPrefix(line, "OSAPPLY-REPORT ")), &rep); jerr == nil {
found = true
}
}
}
if !found {
return rep, fmt.Errorf("osupdate: wrapper gave no report (err %v): %s", err, strings.TrimSpace(string(stderr)))
}
return rep, nil // a refusal / failure is IN the report (exit 2 / 3), not an error here
}
// Pass is one leg's reports; an empty Layer means the step did not run.
type Pass struct {
Guest, Host, Docker Report
}
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
// only, after good earlier steps) the Docker engine set. trigger is "night" or "debug".
func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
unlock := l.lockPass(true)
defer unlock()
l.sendUnsentLocked(ctx) // R-868: a report a killed agent never sent goes first
g, h := l.runFast(ctx, vmid, trigger)
p := Pass{Guest: g, Host: h}
if g.Outcome == "skipped" {
return p
}
blk := l.Block()
okStep := func(r Report) bool {
return (r.Outcome == "applied" || r.Outcome == "nothing" || r.Outcome == "inventory") && r.Healthy
}
lg := l.log().With("run", g.RunID, "vmid", vmid, "trigger", trigger)
switch {
case blk.Ring != 0 || !blk.Enabled:
lg.Info("osupdate: docker step skipped — ring 1 takes an engine set only inside a signed operator job (`11` §5.8)", "ring", blk.Ring, "enabled", blk.Enabled)
case !okStep(g) || (h.Layer != "" && !okStep(h)):
lg.Warn("osupdate: docker step skipped — an earlier step did not end healthy")
default:
if err := l.EnsureLiveRestore(ctx, g.RunID, vmid); err != nil {
p.Docker = l.finish(ctx, lg, Report{RunID: g.RunID, Layer: LayerDocker, Trigger: trigger, Ring: 0, VMID: vmid,
Mode: "apply", Outcome: "failed", HealthReason: "live-restore could not be turned on: " + err.Error()})
return p
}
p.Docker = l.runLayer(ctx, g.RunID, LayerDocker, vmid, trigger, blk, dockerOpts{})
}
return p
}
// dockerOpts is a signed Docker step (DockerStepExecutor); the zero value is ring 0's unsigned "pending-docker".
type dockerOpts struct {
releaseID string
packages []Package
undo bool
signed map[string]string // blob_b64, sig — the wrapper verifies them ITSELF
}
// EnsureLiveRestore is the ONE-TIME step of `09` decision 87: the wrapper merges `"live-restore": true` into the guest's
// daemon.json and RELOADS docker (never a restart, R-835). A no-op when it is already on.
func (l *Leg) EnsureLiveRestore(ctx context.Context, runID string, vmid int) error {
wr, err := l.call(ctx, runID, map[string]any{"release_id": "live-restore", "layer": LayerGuest, "lane": "fast",
"vmid": vmid, "mode": "live-restore-on", "packages": []Package{}})
if err != nil {
return err
}
if wr.refused() {
return fmt.Errorf("refused: %s", wr.Refused)
}
if wr.failed() {
return fmt.Errorf("failed: %s", wr.Failed)
}
l.log().Info("osupdate: live-restore", "vmid", vmid, "result", string(wr.LiveRestore))
return nil
}
// Facts reads the box's versions through the wrapper's read-only facts mode (R-852): host Debian, kernels, held
// packages, taint, the crash guard; guest Debian, Docker engine, containerd, live-restore. Raw JSON, the wrapper's shape.
func (l *Leg) Facts(ctx context.Context, vmid int) (json.RawMessage, error) {
wr, err := l.call(ctx, "facts"+l.now().UTC().Format("150405"), map[string]any{"release_id": "facts", "layer": LayerHost,
"lane": "fast", "vmid": vmid, "mode": "facts", "packages": []Package{}})
if err != nil {
return nil, err
}
if wr.refused() {
return nil, fmt.Errorf("facts refused: %s", wr.Refused)
}
if len(wr.Facts) == 0 {
return nil, fmt.Errorf("facts: the wrapper returned none (an older wrapper?)")
}
return wr.Facts, nil
}
// runFast is the guest + host fast lane (agent v0.141.x behaviour).
func (l *Leg) runFast(ctx context.Context, vmid int, trigger string) (guest Report, host Report) {
runID := l.now().UTC().Format("20060102T150405Z")
lg := l.log().With("run", runID, "vmid", vmid, "trigger", trigger)
if trigger == "night" && l.StatePath != "" {
gap := l.MinGap
if gap == 0 {
gap = 20 * time.Hour
}
if b, err := os.ReadFile(l.StatePath); err == nil {
if last, perr := time.Parse(time.RFC3339, strings.TrimSpace(string(b))); perr == nil && l.now().Sub(last) < gap {
lg.Info("osupdate: skipped — already ran tonight", "last", last.UTC().Format(time.RFC3339))
return Report{RunID: runID, Layer: LayerGuest, Outcome: "skipped"}, Report{}
}
}
}
blk := l.Block()
guest = l.runLayer(ctx, runID, LayerGuest, vmid, trigger, blk, dockerOpts{})
if trigger == "night" && l.StatePath != "" {
_ = os.WriteFile(l.StatePath, []byte(l.now().UTC().Format(time.RFC3339)), 0o600)
}
switch {
case !l.Appliance:
lg.Info("osupdate: host step skipped — not an appliance install (a BYO host belongs to its owner, `11` §1)")
case !(guest.Outcome == "applied" || guest.Outcome == "nothing" || guest.Outcome == "inventory") || !guest.Healthy:
lg.Warn("osupdate: host step skipped — the guest step did not end healthy", "guest_outcome", guest.Outcome, "reason", guest.HealthReason)
default:
host = l.runLayer(ctx, runID, LayerHost, vmid, trigger, blk, dockerOpts{})
}
return guest, host
}
func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigger string, blk hub.WireOSUpdate, do dockerOpts) Report {
rel := hub.WireOSRelease{ID: "ring0-" + runID}
var wire *hub.WireOSRelease
switch layer {
case LayerGuest:
wire = blk.Release
case LayerHost:
wire = blk.HostRelease
}
lane := "fast"
if layer == LayerDocker {
lane = "slow"
if do.signed != nil {
rel = hub.WireOSRelease{ID: do.releaseID}
}
} else if blk.Ring == 1 {
rel = hub.WireOSRelease{}
if wire != nil {
rel = *wire
}
}
rep := Report{RunID: runID, Layer: layer, Trigger: trigger, Ring: blk.Ring, VMID: vmid, ReleaseID: rel.ID}
lg := l.log().With("run", runID, "layer", layer, "vmid", vmid, "ring", blk.Ring, "trigger", trigger)
lg.Info("osupdate: START", "enabled", blk.Enabled, "release", rel.ID)
plan := map[string]any{"release_id": rel.ID, "layer": layer, "lane": lane, "vmid": vmid, "snapshot": rel.Snapshot,
"packages": []Package{}, "mode": "apply", "select": "listed",
"run_id": runID, "trigger": trigger, "ring": blk.Ring} // R-868: echoed into the wrapper's kept copy
if rel.ID == "" {
plan["release_id"] = "none"
}
planned := map[string]bool{}
switch {
case layer == LayerDocker && do.signed != nil:
plan["packages"], plan["signed"] = do.packages, do.signed
if do.undo {
plan["undo"] = true
}
for _, p := range do.packages {
planned[p.Name] = true
}
case layer == LayerDocker:
plan["select"] = "pending-docker" // ring 0: the wrapper checks the box's ROOT-OWNED ring-0 mark itself
case !blk.Enabled:
plan["mode"] = "inventory"
lg.Info("osupdate: switched OFF for this box — reporting only")
case blk.Ring == 0:
plan["select"] = "pending-fast" // the wrapper picks every pending Debian / Debian-Security upgrade
case len(rel.Packages) == 0:
plan["mode"] = "inventory" // ring 1 with no approved release for this layer: nothing to install
default:
var pk []Package
for _, p := range rel.Packages {
pk = append(pk, Package{Name: p.Name, Version: p.Version, Origin: p.Origin})
planned[p.Name] = true
}
plan["packages"] = pk
}
rep.Mode = plan["mode"].(string)
wr, err := l.call(ctx, runID, plan)
if rep.Mode == "apply" {
rep.unsent = reportFile(l.planDir(), runID, layer, rep.Mode) // the wrapper kept a copy (R-868)
}
switch {
case err != nil:
rep.Outcome, rep.HealthReason = "failed", err.Error()
return l.finish(ctx, lg, rep)
case wr.refused():
rep.Outcome, rep.Refused = "refused", wr.Refused
return l.finish(ctx, lg, rep)
case wr.failed():
rep.Outcome, rep.Refused = "failed", wr.Failed
}
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
rep.OOMCheck = rawOrNil(wr.OOMCheck)
if rep.Outcome == "" {
switch {
case rep.Mode == "inventory" && !blk.Enabled:
rep.Outcome = "inventory"
case len(wr.Upgraded) == 0:
rep.Outcome = "nothing"
default:
rep.Outcome = "applied"
}
}
if blk.Ring == 0 {
for _, u := range wr.Upgraded {
planned[u.Name] = true
}
}
// Health: compare with the start of the pass; give restarted services time (only after an install).
cur := wr.HealthAfter
wantEngine := ""
for _, u := range wr.Upgraded {
if u.Name == "docker-ce" {
wantEngine = EngineOf(u.Version)
}
}
verdict := func(h *Health) (bool, string) {
if layer == LayerDocker {
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
}
if layer == LayerHost {
t := hub.TunnelUnknown
if l.Tunnel != nil {
t, _ = l.Tunnel.Status(ctx)
}
return HostHealthVerdict(wr.HealthBefore, h, t)
}
return HealthVerdict(wr.HealthBefore, h)
}
if len(wr.Upgraded) > 0 {
wait, poll := l.HealthWait, l.HealthPoll
if wait == 0 {
wait = 5 * time.Minute
}
if poll == 0 {
poll = 15 * time.Second
}
deadline := l.now().Add(wait)
for {
ok, why := verdict(cur)
rep.Healthy, rep.HealthReason = ok, why
if ok || !l.now().Before(deadline) || ctx.Err() != nil {
break
}
l.sleep(ctx, poll)
hp := map[string]any{"release_id": plan["release_id"], "layer": layer, "lane": lane, "vmid": vmid, "mode": "health", "packages": []Package{}}
hr, herr := l.call(ctx, runID, hp)
if herr == nil && hr.Health != nil {
cur = hr.Health
}
}
if !rep.Healthy && rep.Outcome == "applied" {
rep.Outcome = "health_failed"
}
} else {
rep.Healthy, rep.HealthReason = verdict(cur)
}
rep.Installed, rep.Pending = wr.Installed, wr.Pending
rep.RestartNeeded, rep.DockerRestartNeeded, rep.RebootNeeded = wr.RestartNeeded, wr.DockerRestartNeeded, wr.RebootNeeded
rep.RebootScanned = wr.RebootScanned
if layer == LayerDocker {
// the docker report carries the engine set only (the guest report already carries the Debian packages)
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
rep.NotCovered = nil
} else {
rep.NotCovered = notCovered(wr.Pending, blk.Ring, planned)
}
return l.finish(ctx, lg, rep)
}
// rawOrNil: a wrapper field that is absent or JSON null stays out of the hub report (omitempty).
func rawOrNil(m json.RawMessage) json.RawMessage {
if len(m) == 0 || string(m) == "null" {
return nil
}
return m
}
func onlyDocker(in []Package) []Package {
var out []Package
for _, p := range in {
if DockerNames[p.Name] {
out = append(out, p)
}
}
return out
}
func onlyDockerPending(in []Pending) []Pending {
var out []Pending
for _, p := range in {
if DockerNames[p.Name] {
out = append(out, p)
}
}
return out
}
// notCovered lists pending updates no approved release covers: in ring 0 everything outside the fast lane; in ring 1
// also every fast-lane update the release did not name.
func notCovered(pending []Pending, ring int, planned map[string]bool) []string {
var out []string
for _, p := range pending {
if !IsFast(p.Origin) || (ring == 1 && !planned[p.Name]) {
out = append(out, p.Name)
}
}
return out
}
func (l *Leg) finish(ctx context.Context, lg *slog.Logger, rep Report) Report {
lg.Info("osupdate: DONE", "outcome", rep.Outcome, "healthy", rep.Healthy, "reason", rep.HealthReason,
"upgraded", len(rep.Upgraded), "pending", len(rep.Pending), "not_covered", len(rep.NotCovered),
"restart_needed", len(rep.RestartNeeded), "reboot_needed", rep.RebootNeeded, "wrapper_seconds", rep.PassSeconds)
if l.Hub != nil {
body, _ := json.Marshal(rep)
rctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
defer cancel()
if err := l.Hub.PostOSReport(rctx, body); err != nil {
lg.Warn("osupdate: reporting to the hub failed (the run itself is done; the kept copy is sent at the next start or pass)", "err", err)
} else if rep.unsent != "" {
if rerr := os.Remove(rep.unsent); rerr != nil && !os.IsNotExist(rerr) {
lg.Warn("osupdate: could not delete the sent report's kept copy (it may be sent twice)", "path", rep.unsent, "err", rerr)
}
}
}
return rep
}
+549
View File
@@ -0,0 +1,549 @@
package osupdate
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// fakeWrapper plays /usr/local/sbin/felhom-os-apply: it reads the plan the leg wrote and answers per layer and mode.
type fakeWrapper struct {
t *testing.T
pending []Pending
applyRep map[string]WrapperReport // per layer
healthSeq map[string][]*Health // per layer: answers to successive "health" calls
plans []map[string]any
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
}
func yes() *bool { b := true; return &b }
func guestOK() *Health {
return &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "running", Health: "healthy"}}}
}
func hostOK() *Health {
return &Health{HostServices: map[string]string{"pveproxy": "active", "pvedaemon": "active", "pvestatd": "active",
"pve-cluster": "active", "felhom-agent": "active"}, GuestRunning: yes(), Guest: guestOK()}
}
func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
if name != WrapperPath || len(args) != 2 || args[0] != "--plan" {
f.t.Fatalf("unexpected command %s %v", name, args)
}
b, err := os.ReadFile(args[1])
if err != nil {
f.t.Fatal(err)
}
var plan map[string]any
json.Unmarshal(b, &plan)
f.plans = append(f.plans, plan)
layer := plan["layer"].(string)
ok := guestOK()
if layer == LayerHost {
ok = hostOK()
}
var rep WrapperReport
switch plan["mode"] {
case "inventory":
rep = WrapperReport{Mode: "inventory", Pending: f.pending, HealthBefore: ok, HealthAfter: ok,
Installed: []Package{{Name: "libc6", Version: "u3", Origin: "Debian"}}}
case "apply":
rep = f.applyRep[layer]
rep.Mode = "apply"
if rep.HealthBefore == nil {
rep.HealthBefore = ok
}
if rep.HealthAfter == nil {
rep.HealthAfter = ok
}
case "health":
if seq := f.healthSeq[layer]; len(seq) > 0 {
rep.Health, f.healthSeq[layer] = seq[0], seq[1:]
} else {
rep.Health = ok
}
}
if f.keep && plan["mode"] == "apply" {
kept := rep
kept.Layer, kept.RunID, _ = layer, fmt.Sprint(plan["run_id"]), 0
if tr, ok := plan["trigger"].(string); ok {
kept.Trigger = tr
}
if r, ok := plan["ring"].(float64); ok {
ri := int(r)
kept.Ring = &ri
}
kb, _ := json.Marshal(kept)
dst := filepath.Join(filepath.Dir(args[1]), "report-"+strings.TrimPrefix(filepath.Base(args[1]), "plan-"))
if err := os.WriteFile(dst, kb, 0o600); err != nil {
f.t.Fatal(err)
}
}
out, _ := json.Marshal(rep)
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
}
func (f *fakeWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return f.Run(ctx, name, args...)
}
type fakeHub struct {
reports []Report
bodies [][]byte // the exact bytes posted (R-528: the oom_check object must arrive unchanged)
}
func (h *fakeHub) PostOSReport(_ context.Context, body []byte) error {
h.bodies = append(h.bodies, append([]byte(nil), body...))
var r Report
json.Unmarshal(body, &r)
h.reports = append(h.reports, r)
return nil
}
type fakeTunnel struct{ st string }
func (t fakeTunnel) Status(context.Context) (string, string) { return t.st, "" }
func newLeg(t *testing.T, w *fakeWrapper, blk *hub.WireOSUpdate) (*Leg, *fakeHub) {
h := &fakeHub{}
now := time.Date(2026, 10, 4, 4, 0, 0, 0, time.UTC)
if w.applyRep == nil {
w.applyRep = map[string]WrapperReport{}
}
if w.healthSeq == nil {
w.healthSeq = map[string][]*Health{}
}
l := &Leg{Runner: w, Hub: h, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last"),
HealthWait: time.Minute, HealthPoll: 10 * time.Second, Appliance: true, Tunnel: fakeTunnel{hub.TunnelRunning},
Now: func() time.Time { return now },
Sleep: func(_ context.Context, d time.Duration) { now = now.Add(d) }}
if blk != nil {
l.SetBlock(blk)
}
return l, h
}
var pend = []Pending{
{Name: "libc6", From: "u3", To: "u4", Origin: []string{"Debian"}},
{Name: "openssl", From: "u1", To: "u3", Origin: []string{"Debian-Security", "Debian"}},
{Name: "docker-ce", From: "29.7", To: "29.8", Origin: []string{"Docker CE"}},
}
func calls(w *fakeWrapper) string {
var m []string
for _, p := range w.plans {
m = append(m, p["layer"].(string)+":"+p["mode"].(string))
}
return strings.Join(m, ",")
}
// Ring 0: ONE wrapper call per layer (R-845), select pending-fast (the wrapper picks every Debian / Debian-Security
// upgrade), from live sources; the guest step first, then the host step.
func TestRing0_OneCallPerLayer(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{
LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "u4"}, {Name: "openssl", Version: "u3"}}, Pending: pend[2:]},
LayerHost: {Upgraded: []Package{{Name: "openssl", Version: "u3"}}},
}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
g, ho := run2(l, "night")
if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
t.Fatalf("guest %+v\nhost %+v", g, ho)
}
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply" {
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore and the ring-0 docker step", calls(w))
}
for _, p := range w.plans[:2] {
if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
t.Fatalf("ring-0 plan = %v", p)
}
}
if len(g.NotCovered) != 1 || g.NotCovered[0] != "docker-ce" {
t.Fatalf("not covered = %v", g.NotCovered)
}
if len(h.reports) != 3 || h.reports[0].Layer != LayerGuest || h.reports[1].Layer != LayerHost || h.reports[2].Layer != LayerDocker {
t.Fatalf("hub got %+v", h.reports)
}
}
// Ring 1 installs EXACTLY each layer's own approved release (a guest release is not a host release).
func TestRing1_EachLayerItsOwnRelease(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{
LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "g-u4"}}, Pending: pend[1:]},
LayerHost: {Upgraded: []Package{{Name: "openssl", Version: "h-u3"}}},
}}
gr := &hub.WireOSRelease{ID: "os-g", Snapshot: "20261004T080000Z", Packages: []hub.WireOSPackage{{Name: "libc6", Version: "g-u4", Origin: "Debian"}}}
hr := &hub.WireOSRelease{ID: "os-h", Snapshot: "20261004T090000Z", Packages: []hub.WireOSPackage{{Name: "openssl", Version: "h-u3", Origin: "Debian-Security"}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true, Release: gr, HostRelease: hr})
g, ho := run2(l, "night")
if g.ReleaseID != "os-g" || ho.ReleaseID != "os-h" {
t.Fatalf("release ids %q %q", g.ReleaseID, ho.ReleaseID)
}
gp, hp := w.plans[0], w.plans[1]
if gp["snapshot"] != "20261004T080000Z" || gp["packages"].([]any)[0].(map[string]any)["version"] != "g-u4" {
t.Fatalf("guest plan %v", gp)
}
if hp["layer"] != LayerHost || hp["snapshot"] != "20261004T090000Z" || hp["packages"].([]any)[0].(map[string]any)["name"] != "openssl" {
t.Fatalf("host plan %v", hp)
}
if strings.Join(g.NotCovered, ",") != "openssl,docker-ce" {
t.Fatalf("guest not covered = %v", g.NotCovered)
}
}
func TestRing1_NoReleaseIsInventory(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
g, ho := run2(l, "night")
if g.Outcome != "nothing" || ho.Outcome != "nothing" || calls(w) != "guest:inventory,host:inventory" {
t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
}
}
// No block from the hub (an older hub): ring 1, ON, no release → reports, installs nothing.
func TestNoBlock_IsRing1Nothing(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend}
l, _ := newLeg(t, w, nil)
if g, _ := run2(l, "night"); g.Outcome != "nothing" || g.Ring != 1 {
t.Fatalf("g=%+v calls=%s", g, calls(w))
}
}
// Switched OFF: the box reports but installs nothing, on both layers.
func TestSwitchOff_ReportsOnly(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: false})
g, ho := run2(l, "night")
if g.Outcome != "inventory" || ho.Outcome != "inventory" || calls(w) != "guest:inventory,host:inventory" || len(h.reports) != 2 {
t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
}
}
// Not an appliance (BYO, `11` §1): the host step never runs — no host plan at all.
func TestBYO_NoHostPlan(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6"}}}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l.Appliance = false
_, ho := run2(l, "night")
// the guest (and so its Docker engine) is ours on a BYO box too: only the HOST is the owner's
if ho.Outcome != "" || calls(w) != "guest:apply,guest:live-restore-on,docker:apply" || len(h.reports) != 2 {
t.Fatalf("a BYO box got a host step: host=%+v calls=%s", ho, calls(w))
}
}
// A failed guest step skips the host step that night.
func TestGuestFailure_SkipsTheHost(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{
LayerGuest: {Refused: json.RawMessage(`{"code":"R6","reason":"x"}`)}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
g, ho := run2(l, "night")
if g.Outcome != "refused" || ho.Outcome != "" || calls(w) != "guest:apply" {
t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
}
}
// Unhealthy after the run, and still unhealthy at the end of the wait → health_failed; the host step is skipped.
func TestHealth_FailsAfterTheWait(t *testing.T) {
bad := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "exited"}}}
w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6"}}, HealthAfter: bad}},
healthSeq: map[string][]*Health{LayerGuest: {bad, bad, bad, bad, bad, bad, bad, bad}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
g, ho := run2(l, "night")
if g.Outcome != "health_failed" || g.Healthy || !strings.Contains(g.HealthReason, "app was running") || ho.Outcome != "" {
t.Fatalf("g=%+v h=%+v", g, ho)
}
if h.reports[0].Outcome != "health_failed" {
t.Fatal("the hub was not told")
}
}
// A service that takes a moment to come back is not a failure: the poll sees it recover inside the wait.
func TestHealth_RecoversInsideTheWait(t *testing.T) {
starting := &Health{DockerOK: true, NetworkOK: true, Controller: "starting"}
w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6"}}, HealthAfter: starting}},
healthSeq: map[string][]*Health{LayerGuest: {starting}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
if g, _ := run2(l, "night"); g.Outcome != "applied" || !g.Healthy {
t.Fatalf("g = %+v", g)
}
}
// The host step judged unhealthy when the tunnel is down after it.
func TestHost_TunnelDownFailsTheHostStep(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerHost: {Upgraded: []Package{{Name: "openssl"}}}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l.Tunnel = fakeTunnel{hub.TunnelNotRunning}
_, ho := run2(l, "night")
if ho.Outcome != "health_failed" || !strings.Contains(ho.HealthReason, "tunnel") {
t.Fatalf("host = %+v", ho)
}
}
func TestHealthVerdict(t *testing.T) {
ok := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"a": {State: "running", Health: "healthy"}}}
cases := []struct {
name string
before *Health
after *Health
want bool
}{
{"all good", ok, ok, true},
{"no reading", ok, nil, false},
{"docker down", ok, &Health{NetworkOK: true, Controller: "healthy"}, false},
{"no network", ok, &Health{DockerOK: true, Controller: "healthy"}, false},
{"controller starting", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "starting"}, false},
{"only the controller differs", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "unhealthy", Containers: map[string]Container{"a": {State: "running", Health: "healthy"}}}, false},
{"app gone", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{}}, false},
{"app unhealthy", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"a": {State: "running", Health: "unhealthy"}}}, false},
{"stopped before stays stopped", &Health{Containers: map[string]Container{"x": {State: "exited"}}}, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy"}, true},
}
for _, c := range cases {
if got, why := HealthVerdict(c.before, c.after); got != c.want {
t.Errorf("%s: got %v (%s), want %v", c.name, got, why, c.want)
}
}
}
// The host rule: every listed daemon active, the guest running and passing its own rule, the tunnel running.
// Red-proofs: drop any one check and its case fails.
func TestHostHealthVerdict(t *testing.T) {
no := false
svcDown := hostOK()
svcDown.HostServices["pveproxy"] = "failed"
guestDown := hostOK()
guestDown.GuestRunning = &no
guestApp := hostOK()
guestApp.Guest = &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"felhom-controller": {State: "running", Health: "healthy"}}}
cases := []struct {
name string
after *Health
tunnel string
want bool
why string
}{
{"all good", hostOK(), hub.TunnelRunning, true, ""},
{"a daemon down", svcDown, hub.TunnelRunning, false, "pveproxy"},
{"the guest stopped", guestDown, hub.TunnelRunning, false, "guest is not running"},
{"an app in the guest gone", guestApp, hub.TunnelRunning, false, "app was running"},
{"the tunnel down", hostOK(), hub.TunnelNotRunning, false, "tunnel"},
{"the tunnel unknown", hostOK(), hub.TunnelUnknown, false, "tunnel"},
{"no services read", &Health{GuestRunning: yes(), Guest: guestOK()}, hub.TunnelRunning, false, "no host service"},
}
for _, c := range cases {
got, why := HostHealthVerdict(hostOK(), c.after, c.tunnel)
if got != c.want || !strings.Contains(why, c.why) {
t.Errorf("%s: got %v (%s), want %v (…%s…)", c.name, got, why, c.want, c.why)
}
}
}
func TestOncePerNight(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6"}}}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
run2(l, "night")
n := len(w.plans)
if g, _ := run2(l, "night"); g.Outcome != "skipped" || len(w.plans) != n {
t.Fatalf("a second night run in the same night ran: %+v", g)
}
if g, _ := run2(l, "debug"); g.Outcome == "skipped" {
t.Fatal("the debug action must not be throttled")
}
}
// The root wrapper's own suite (configs/test_felhom_os_apply.py) runs with `go test ./...` so CI covers it.
func TestWrapperSuite(t *testing.T) {
py, err := exec.LookPath("python3")
if err != nil {
t.Skip("python3 not available")
}
// the OS wrapper and (agent v0.142.0) the crash guard — both root programs in configs/ with their own suites
for _, suite := range []string{"../../configs/test_felhom_os_apply.py", "../../configs/test_felhom_crash_guard.py", "../../configs/test_felhom_config_bundle.py", "../../configs/test_felhom_priv_apply.py"} {
cmd := exec.Command(py, "-B", suite)
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("%s failed: %v\n%s", suite, err, out)
}
if !strings.Contains(string(out), "OK") {
t.Fatalf("%s did not report OK:\n%s", suite, out)
}
}
}
// The host's restart scan result reaches the hub with reboot_scanned, so the hub can tell "looked: not needed" (a
// reboot cleared it) from "did not look". Red-proof: drop the RebootScanned copy in runLayer and this fails.
func TestHostReport_CarriesRebootScanned(t *testing.T) {
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{
LayerGuest: {},
LayerHost: {RebootScanned: true, RebootNeeded: true, RestartNeeded: []string{"lxc-start"}},
}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
run2(l, "night")
if len(h.reports) != 3 || !h.reports[1].RebootScanned || !h.reports[1].RebootNeeded || h.reports[0].RebootScanned {
t.Fatalf("hub got %+v", h.reports)
}
}
// run2 is the guest + host reports of one pass (the tests written before the docker step).
func run2(l *Leg, trigger string) (Report, Report) {
p := l.Run(context.Background(), 9201, trigger)
return p.Guest, p.Host
}
// ---- the Docker step (`11` §5.8, agent v0.142.0) ----
// Ring 1 never takes an engine step in the night leg — only inside a signed operator job. Red-proof: drop the
// `blk.Ring != 0` case in Run and the ring-1 pass makes a docker call.
func TestDocker_Ring1NightLegNeverSteps(t *testing.T) {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
if p.Docker.Layer != "" || strings.Contains(calls(w), "docker") || strings.Contains(calls(w), "live-restore") {
t.Fatalf("ring 1 took a docker step: %s", calls(w))
}
}
// An unhealthy earlier step skips the docker step.
func TestDocker_SkippedAfterAnUnhealthyStep(t *testing.T) {
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6"}}}},
healthSeq: map[string][]*Health{}}
bad := guestOK()
bad.Controller = "unhealthy"
w.applyRep[LayerGuest] = WrapperReport{Upgraded: []Package{{Name: "libc6"}}, HealthAfter: bad}
w.healthSeq[LayerGuest] = []*Health{bad, bad, bad, bad, bad, bad, bad, bad}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
if p.Docker.Layer != "" || strings.Contains(calls(w), "docker") {
t.Fatalf("docker step ran after an unhealthy guest step: %s", calls(w))
}
}
// The docker plan is the slow lane, pending-docker for ring 0; the report carries only the engine set.
func TestDocker_Ring0PlanAndReport(t *testing.T) {
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerDocker: {
Upgraded: []Package{{Name: "docker-ce", Version: "5:29.8.2-1~debian.13~trixie"}},
Installed: []Package{{Name: "docker-ce", Version: "5:29.8.2-1~debian.13~trixie", Origin: "Docker"}, {Name: "libc6", Version: "u4", Origin: "Debian"}},
DockerEngine: "29.8.2", Authority: "ring0"}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
dp := w.plans[len(w.plans)-1]
if dp["layer"] != "docker" || dp["lane"] != "slow" || dp["select"] != "pending-docker" {
t.Fatalf("docker plan = %v", dp)
}
d := p.Docker
if d.Outcome != "applied" || !d.Healthy || d.DockerEngine != "29.8.2" || len(d.Installed) != 1 || d.Installed[0].Name != "docker-ce" {
t.Fatalf("docker report = %+v", d)
}
}
// THE docker health rule. Red-proof: drop the id comparison (or the engine check) in DockerHealthVerdict and a case fails.
func TestDockerHealthVerdict(t *testing.T) {
before := guestOK()
before.Containers = map[string]Container{"felhom-controller": {State: "running", Health: "healthy", ID: "a"},
"app": {State: "running", Health: "healthy", ID: "b"}}
same := guestOK()
same.Containers = map[string]Container{"felhom-controller": {State: "running", Health: "healthy", ID: "a"},
"app": {State: "running", Health: "healthy", ID: "b"}}
moved := guestOK()
moved.Containers = map[string]Container{"felhom-controller": {State: "running", Health: "healthy", ID: "a"},
"app": {State: "running", Health: "healthy", ID: "c"}}
if ok, why := DockerHealthVerdict(before, same, "29.8.2", "29.8.2"); !ok {
t.Fatalf("same ids, right engine: %s", why)
}
if ok, _ := DockerHealthVerdict(before, moved, "29.8.2", "29.8.2"); ok {
t.Fatal("a changed container id passed — live-restore failed and the apps restarted")
}
if ok, _ := DockerHealthVerdict(before, same, "29.8.2", "29.7.2"); ok {
t.Fatal("the engine did not move and the step passed")
}
if EngineOf("5:29.8.2-1~debian.13~trixie") != "29.8.2" {
t.Fatalf("EngineOf = %q", EngineOf("5:29.8.2-1~debian.13~trixie"))
}
}
// A changed id after the step → health_failed (the consequence, not only the verdict).
func TestDocker_ChangedIDIsHealthFailed(t *testing.T) {
before := guestOK()
before.Containers = map[string]Container{"felhom-controller": {State: "running", Health: "healthy", ID: "a"}}
after := guestOK()
after.Containers = map[string]Container{"felhom-controller": {State: "running", Health: "healthy", ID: "z"}}
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerDocker: {
Upgraded: []Package{{Name: "docker-ce", Version: "5:29.8.2-1"}}, DockerEngine: "29.8.2",
HealthBefore: before, HealthAfter: after}}, healthSeq: map[string][]*Health{}}
w.healthSeq[LayerDocker] = []*Health{after, after, after, after, after, after, after, after}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
if p.Docker.Outcome != "health_failed" || !strings.Contains(p.Docker.HealthReason, "id changed") {
t.Fatalf("docker = %+v", p.Docker)
}
}
// R-858: a controller that cannot reach Docker fails the health rule even though its own check says healthy.
// Red-proof: drop the ControllerDockerOK check in HealthVerdict and this fails.
func TestHealthVerdict_ControllerBlindToDockerFails(t *testing.T) {
no, yes2 := false, true
after := guestOK()
after.ControllerDockerOK = &no
if ok, why := HealthVerdict(guestOK(), after); ok || !strings.Contains(why, "R-858") {
t.Fatalf("a blind controller passed: %v %q", ok, why)
}
if ok, _ := DockerHealthVerdict(guestOK(), after, "", ""); ok {
t.Fatal("the Docker rule passed a blind controller")
}
after.ControllerDockerOK = &yes2
if ok, why := HealthVerdict(guestOK(), after); !ok {
t.Fatalf("a seeing controller failed: %s", why)
}
if ok, _ := HealthVerdict(guestOK(), guestOK()); !ok {
t.Fatal("an older wrapper (no field) must not fail")
}
}
// R-528 (`09` decision 157): the wrapper's oom_check object reaches the hub's docker report byte-for-byte; the guest
// and host reports carry none. COMPANION RED-PROOF: drop `rep.OOMCheck = rawOrNil(wr.OOMCheck)` in runLayer → "no
// oom_check in the docker report".
const oomCheckWire = `{"detail":"the engine reported the memory kill: OOMKilled=true and the oom event","exit_code":137,"image":"gitea.dooplex.hu/admin/felhom-controller:0.300.0","oom_event":true,"oom_killed":true,"result":"pass"}`
func TestDocker_OOMCheckReachesTheHubUnchanged(t *testing.T) {
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerDocker: {
Upgraded: []Package{{Name: "docker-ce", Version: "5:29.8.2-1~debian.13~trixie"}},
DockerEngine: "29.8.2", Authority: "ring0", OOMCheck: json.RawMessage(oomCheckWire)}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l.Run(context.Background(), 9201, "night")
found := false
for _, b := range h.bodies {
var m map[string]json.RawMessage
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
var layer string
json.Unmarshal(m["layer"], &layer)
oc, has := m["oom_check"]
if layer != LayerDocker {
if has {
t.Fatalf("the %s report carries an oom_check: %s", layer, oc)
}
continue
}
found = true
if !has {
t.Fatalf("no oom_check in the docker report: %s", b)
}
if string(oc) != oomCheckWire {
t.Fatalf("oom_check changed on the way:\n got %s\nwant %s", oc, oomCheckWire)
}
}
if !found {
t.Fatalf("no docker report posted: %s", calls(w))
}
}
+198
View File
@@ -0,0 +1,198 @@
package osupdate
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"syscall"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// ── R-868 (v0.144.0): a pass whose agent was killed still reports ─────────────────────────────────────
//
// MEASURED 2026-10-05 02:57 UTC on demo-hp (night drill A5): the debug pass and the agent daemon were kill -9-ed while
// apt-get ran. The root wrapper (its own process under sudo) finished all six packages, but the agent that would have
// read its stdout and posted the report was gone; the hub never learned what the pass installed.
//
// THE MECHANISM: the wrapper writes its apply report to <plan dir>/report-<run>-<layer>-apply.json BEFORE printing
// it (configs/felhom-os-apply save_report). The agent deletes that copy once the hub has the report (finish). A copy
// still on disk is a report nobody sent: SendUnsent posts it — at the agent's start, and before every pass — and then
// deletes it. A pass lock (flock on <plan dir>/pass.lock, released by the kernel when a process dies) keeps the
// sender from picking up the copy of a pass that is still running, also across the daemon and a selftest process.
// Pinned by TestR868_* (unsent_test.go).
// lockPass takes the pass lock. block=false returns ok=false at once when another pass holds it. A lock that cannot
// be opened at all (no plan dir yet) does not stop a pass: the unlock is then a no-op.
func (l *Leg) lockPass(block bool) (unlock func()) {
u, _ := l.tryLockPass(block)
return u
}
func (l *Leg) tryLockPass(block bool) (unlock func(), ok bool) {
dir := l.planDir()
_ = os.MkdirAll(dir, 0o700)
f, err := os.OpenFile(filepath.Join(dir, "pass.lock"), os.O_CREATE|os.O_RDWR, 0o600)
if err != nil {
l.log().Warn("osupdate: pass lock unavailable — continuing without it", "err", err)
return func() {}, true
}
how := syscall.LOCK_EX
if !block {
how |= syscall.LOCK_NB
}
if err := syscall.Flock(int(f.Fd()), how); err != nil {
f.Close()
return func() {}, false
}
return func() { _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN); f.Close() }, true
}
// SendUnsent posts every report a pass kept on disk and nobody sent (R-868). It skips when a pass runs now (that
// pass sends them first). Called at the agent's start.
func (l *Leg) SendUnsent(ctx context.Context) int {
unlock, ok := l.tryLockPass(false)
if !ok {
l.log().Info("osupdate: a pass is running — its start sends any kept report")
return 0
}
defer unlock()
return l.sendUnsentLocked(ctx)
}
// SendUnsentLoop runs SendUnsent now and then every `every` until ctx ends (v0.144.1). MEASURED live on demo-hp
// 2026-10-05: after a kill -9 the daemon restarted in ~5 s, while the orphaned root wrapper was still installing — its
// copy appeared ~7 s AFTER the start-time sender had looked. One look at start is therefore not enough. A glob of the
// plan dir every few minutes costs nothing; the pass lock keeps it off a running pass. Pinned by
// TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop.
func (l *Leg) SendUnsentLoop(ctx context.Context, every time.Duration, onSent func(int)) {
for {
if n := l.SendUnsent(ctx); n > 0 && onSent != nil {
onSent(n)
}
select {
case <-ctx.Done():
return
case <-time.After(every):
}
}
}
func (l *Leg) sendUnsentLocked(ctx context.Context) int {
if l.Hub == nil {
return 0 // nobody to send to: keep the copies for a process that has the hub
}
files, _ := filepath.Glob(filepath.Join(l.planDir(), "report-*.json"))
sent := 0
for _, f := range files {
b, err := os.ReadFile(f)
if err != nil {
l.log().Warn("osupdate: a kept report cannot be read", "path", f, "err", err)
continue
}
var wr WrapperReport
if err := json.Unmarshal(b, &wr); err != nil || wr.Layer == "" {
l.log().Warn("osupdate: a kept report is not a report — moved aside", "path", f, "err", err)
_ = os.Rename(f, f+".bad")
continue
}
rep := l.reportFromKept(ctx, wr, f)
lg := l.log().With("run", rep.RunID, "layer", rep.Layer, "vmid", rep.VMID, "ring", rep.Ring, "trigger", rep.Trigger)
lg.Info("osupdate: sending a kept report late (the agent was stopped mid-pass or the hub was away, R-868)", "path", f)
before := rep.unsent
_ = l.finish(ctx, lg, rep)
if _, err := os.Stat(before); os.IsNotExist(err) {
sent++
// the pass's plan file is left behind too when the agent was killed inside call()
_ = os.Remove(filepath.Join(filepath.Dir(f), "plan-"+strings.TrimPrefix(filepath.Base(f), "report-")))
}
}
return sent
}
// reportFromKept builds the hub report from a kept wrapper report, as runLayer would have. Health: the copy's own
// before/after reading; when that is not healthy after an install, one fresh reading (services restart after an
// install, and the pass that would have waited for them is gone).
func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string) Report {
ring := 1
if wr.Ring != nil {
ring = *wr.Ring
}
runID, trigger := wr.RunID, wr.Trigger
if runID == "" {
runID = strings.TrimSuffix(strings.TrimPrefix(filepath.Base(path), "report-"), ".json")
}
if trigger == "" {
trigger = "unknown"
}
rep := Report{RunID: runID, Layer: wr.Layer, Trigger: trigger, Mode: wr.Mode, Ring: ring, ReleaseID: wr.ReleaseID,
VMID: wr.VMID, unsent: path}
prefix := "sent late — kept on the box until the hub could take it (R-868, R-875)"
switch {
case wr.refused():
rep.Outcome, rep.Refused, rep.HealthReason = "refused", wr.Refused, prefix
return rep
case wr.failed():
rep.Outcome, rep.Refused = "failed", wr.Failed
case len(wr.Upgraded) == 0:
rep.Outcome = "nothing"
default:
rep.Outcome = "applied"
}
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
rep.OOMCheck = rawOrNil(wr.OOMCheck)
wantEngine := ""
for _, u := range wr.Upgraded {
if u.Name == "docker-ce" {
wantEngine = EngineOf(u.Version)
}
}
verdict := func(h *Health) (bool, string) {
switch wr.Layer {
case LayerDocker:
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
case LayerHost:
t := hub.TunnelUnknown
if l.Tunnel != nil {
t, _ = l.Tunnel.Status(ctx)
}
return HostHealthVerdict(wr.HealthBefore, h, t)
}
return HealthVerdict(wr.HealthBefore, h)
}
ok, why := verdict(wr.HealthAfter)
if !ok && len(wr.Upgraded) > 0 && wr.VMID > 0 {
lane := "fast"
if wr.Layer == LayerDocker {
lane = "slow"
}
if hr, err := l.call(ctx, "kept-"+runID, map[string]any{"release_id": "kept", "layer": wr.Layer, "lane": lane,
"vmid": wr.VMID, "mode": "health", "packages": []Package{}}); err == nil && hr.Health != nil {
ok, why = verdict(hr.Health)
}
}
rep.Healthy, rep.HealthReason = ok, prefix
if why != "" {
rep.HealthReason = prefix + ": " + why
}
if !ok && rep.Outcome == "applied" {
rep.Outcome = "health_failed"
}
rep.Installed, rep.Pending = wr.Installed, wr.Pending
rep.RestartNeeded, rep.DockerRestartNeeded, rep.RebootNeeded = wr.RestartNeeded, wr.DockerRestartNeeded, wr.RebootNeeded
rep.RebootScanned = wr.RebootScanned
if wr.Layer == LayerDocker {
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
} else {
planned := map[string]bool{}
for _, u := range wr.Upgraded {
planned[u.Name] = true
}
rep.NotCovered = notCovered(wr.Pending, ring, planned)
}
return rep
}
+173
View File
@@ -0,0 +1,173 @@
package osupdate
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-868 (v0.144.0). THE NIGHT'S SHAPE (A5, demo-hp 2026-10-05 02:57 UTC): the wrapper finished six packages, the agent
// was kill -9-ed before it read the report; the hub never got it. Here: the wrapper's kept copy and the plan file are
// on disk, a NEW agent process starts — it must send exactly one `applied` report and delete both files.
// COMPANION RED-PROOF: drop the SendUnsent body (return 0) → "the hub got no report".
func TestR868_KilledPassIsReportedAtStart(t *testing.T) {
w := &fakeWrapper{t: t}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
ring := 0
kept := WrapperReport{Mode: "apply", Layer: LayerGuest, RunID: "20261005T025700Z", Trigger: "debug", Ring: &ring, VMID: 9201,
ReleaseID: "ring0-20261005T025700Z", HealthBefore: guestOK(), HealthAfter: guestOK(),
Upgraded: []Package{{Name: "libc6", Version: "u4"}, {Name: "openssl", Version: "u3"}}}
b, _ := json.Marshal(kept)
rp := reportFile(l.PlanDir, kept.RunID, LayerGuest, "apply")
pp := planFile(l.PlanDir, kept.RunID, LayerGuest, "apply")
must(t, os.WriteFile(rp, b, 0o600))
must(t, os.WriteFile(pp, []byte("{}"), 0o600))
if n := l.SendUnsent(context.Background()); n != 1 {
t.Fatalf("sent %d, want 1", n)
}
if len(h.reports) != 1 {
t.Fatalf("the hub got no report (or several): %+v", h.reports)
}
r := h.reports[0]
if r.Outcome != "applied" || !r.Healthy || r.Trigger != "debug" || r.RunID != kept.RunID || r.Ring != 0 || len(r.Upgraded) != 2 || r.VMID != 9201 {
t.Fatalf("report = %+v", r)
}
// R-875 (v0.145.0): neutral — the copy cannot tell a killed agent from an absent hub.
if !strings.HasPrefix(r.HealthReason, "sent late") || strings.Contains(r.HealthReason, "stopped mid-pass") {
t.Fatalf("health_reason = %q, want the neutral \"sent late …\"", r.HealthReason)
}
for _, p := range []string{rp, pp} {
if _, err := os.Stat(p); !os.IsNotExist(err) {
t.Fatalf("%s still on disk after the hub got it", filepath.Base(p))
}
}
// a second start sends nothing again — no duplicate report
if n := l.SendUnsent(context.Background()); n != 0 || len(h.reports) != 1 {
t.Fatalf("sent again: %d, reports %d", n, len(h.reports))
}
}
// A normal pass: the hub gets ONE report per layer and the kept copies are gone after it (nothing resent later).
// COMPANION RED-PROOF: drop the os.Remove(rep.unsent) in finish → the next pass resends → "2 guest reports".
func TestR868_NormalPassLeavesNoCopyAndNoDuplicate(t *testing.T) {
w := &fakeWrapper{t: t, keep: true, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "u4"}}}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l.Run(context.Background(), 9201, "debug")
left, _ := filepath.Glob(filepath.Join(l.PlanDir, "report-*.json"))
if len(left) != 0 {
t.Fatalf("kept copies left after the hub got them: %v", left)
}
l.Run(context.Background(), 9201, "debug") // the next pass sends kept copies first
guest := 0
for _, r := range h.reports {
if r.Layer == LayerGuest && r.Outcome == "applied" {
guest++
}
}
if guest != 2 {
t.Fatalf("%d guest reports for 2 passes (a duplicate or a loss)", guest)
}
}
type failingHub struct{ n int }
func (h *failingHub) PostOSReport(context.Context, []byte) error {
h.n++
return errors.New("hub away")
}
// The hub away: the copy stays, and goes at the next chance.
func TestR868_HubAwayKeepsTheCopy(t *testing.T) {
w := &fakeWrapper{t: t, keep: true, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "u4"}}}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l.Appliance = false
l.Hub = &failingHub{}
l.Run(context.Background(), 9201, "night")
left, _ := filepath.Glob(filepath.Join(l.PlanDir, "report-*.json"))
if len(left) != 2 { // ring 0: the guest step and the Docker step each kept one
t.Fatalf("the copies must stay while the hub is away: %v", left)
}
h := &fakeHub{}
l.Hub = h
if n := l.SendUnsent(context.Background()); n != 2 {
t.Fatalf("sent %d: %+v", n, h.reports)
}
for _, r := range h.reports {
if r.Trigger != "night" || r.Ring != 0 {
t.Fatalf("the kept report lost its ids: %+v", r)
}
}
}
// A pass in progress holds the lock: the sender at start must not take that pass's copy (it would be sent twice).
func TestR868_RunningPassKeepsTheSenderOff(t *testing.T) {
w := &fakeWrapper{t: t}
l, h := newLeg(t, w, nil)
must(t, os.WriteFile(reportFile(l.PlanDir, "r1", LayerGuest, "apply"), []byte(`{"mode":"apply","layer":"guest"}`), 0o600))
unlock := l.lockPass(true)
if n := l.SendUnsent(context.Background()); n != 0 || len(h.reports) != 0 {
t.Fatalf("sent while a pass ran: %d", n)
}
unlock()
if n := l.SendUnsent(context.Background()); n != 1 {
t.Fatalf("not sent after the pass: %d", n)
}
}
// v0.144.1 — THE LIVE SHAPE (demo-hp 2026-10-05 05:45 UTC): the restarted daemon looked at 05:45:09, the orphaned
// wrapper wrote its copy at ~05:45:16. The loop must still send it.
// COMPANION RED-PROOF: make SendUnsentLoop return after the first look → "the late copy was never sent".
func TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop(t *testing.T) {
w := &fakeWrapper{t: t}
l, h := newLeg(t, w, nil)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
sent := make(chan int, 4)
go l.SendUnsentLoop(ctx, 20*time.Millisecond, func(n int) { sent <- n })
time.Sleep(50 * time.Millisecond) // the start-time look found nothing
must(t, os.WriteFile(reportFile(l.PlanDir, "late", LayerGuest, "apply"),
[]byte(`{"mode":"apply","layer":"guest","run_id":"late","trigger":"debug","ring":0,"vmid":9201,"upgraded":[{"name":"openssl","version":"u3"}]}`), 0o600))
select {
case n := <-sent:
if n != 1 || len(h.reports) != 1 || h.reports[0].RunID != "late" || h.reports[0].Outcome == "" {
t.Fatalf("sent %d: %+v", n, h.reports)
}
case <-time.After(3 * time.Second):
t.Fatal("the late copy was never sent")
}
}
func must(t *testing.T, err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
// R-528: a kept docker report (the agent was killed) still carries the oom_check object to the hub unchanged.
// COMPANION RED-PROOF: drop `rep.OOMCheck = rawOrNil(wr.OOMCheck)` in reportFromKept → "oom_check lost".
func TestR868_KeptCopyCarriesTheOOMCheck(t *testing.T) {
w := &fakeWrapper{t: t}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
ring := 0
kept := WrapperReport{Mode: "apply", Layer: LayerDocker, RunID: "20261007T020000Z", Trigger: "night", Ring: &ring, VMID: 9201,
ReleaseID: "ring0-20261007T020000Z", HealthBefore: guestOK(), HealthAfter: guestOK(), DockerEngine: "29.8.2",
Upgraded: []Package{{Name: "docker-ce", Version: "5:29.8.2-1~debian.13~trixie"}}, OOMCheck: json.RawMessage(oomCheckWire)}
b, _ := json.Marshal(kept)
must(t, os.WriteFile(reportFile(l.PlanDir, kept.RunID, LayerDocker, "apply"), b, 0o600))
if n := l.SendUnsent(context.Background()); n != 1 || len(h.bodies) != 1 {
t.Fatalf("sent %d, bodies %d", n, len(h.bodies))
}
var m map[string]json.RawMessage
must(t, json.Unmarshal(h.bodies[0], &m))
if string(m["oom_check"]) != oomCheckWire {
t.Fatalf("oom_check lost or changed: %q", m["oom_check"])
}
}
+2 -2
View File
@@ -60,8 +60,8 @@ func TestLiveReporter_CoordPresentWithoutPriorVerify(t *testing.T) {
if h.PBS == nil {
t.Fatal("pbs coord absent despite a reachable PBS — the gap this fixes")
}
if h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != "root" || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want felhom-pbs/root/9201", h.PBS)
if h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != hub.PBSRootNamespace || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want felhom-pbs, the root namespace (\"\", R-124), 9201", h.PBS)
}
// COMPANION (pre-fix): the bare SnapshotStore (no live read) with an empty store omits pbs.
+34
View File
@@ -0,0 +1,34 @@
// Package privapplytest runs configs/felhom-priv-apply in CHECK-ONLY mode from Go tests (R-861): each renderer's
// real output must be accepted by the root checker, so the two can never drift apart unnoticed. Test-only helper.
package privapplytest
import (
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"testing"
)
// Check writes content to a temp file and runs `felhom-priv-apply --check <verb> [name] <file>`. It returns the
// checker's verdict line ("OK" or "REFUSED [rule] …"). Skips when python3 is absent.
func Check(t *testing.T, verb, name, content string) string {
t.Helper()
py, err := exec.LookPath("python3")
if err != nil {
t.Skip("python3 not available")
}
_, here, _, _ := runtime.Caller(0)
wrapper := filepath.Join(filepath.Dir(here), "..", "..", "configs", "felhom-priv-apply")
f := filepath.Join(t.TempDir(), "staged")
if err := os.WriteFile(f, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
args := []string{"-B", wrapper, "--check", verb}
if name != "" {
args = append(args, name)
}
out, _ := exec.Command(py, append(args, f)...).CombinedOutput()
return strings.TrimSpace(string(out))
}
+3 -2
View File
@@ -186,8 +186,9 @@ func (b *BackHalf) Provision(ctx context.Context, in Input) (Result, error) {
// 6. Install + register the pre-start self-heal hook (C1 net): if a data drive is absent at a future
// boot, the hook creates a placeholder for its missing bind source so the guest still starts.
// Best-effort + non-fatal — it's defense-in-depth; a provision must not fail over the hook.
if err := guesthook.InstallSnippet(ctx, b.runner); err != nil {
b.logger.Warn("provision: pre-start hook snippet install failed (non-fatal)", "vmid", in.VMID, "err", err)
// R-861 (v0.146.0): the hook FILE comes with the signed config bundle; the agent only checks it and registers it.
if err := guesthook.SnippetReady(guesthook.SnippetPath); err != nil {
b.logger.Warn("provision: pre-start hook not in place — not registering it (non-fatal)", "vmid", in.VMID, "err", err)
} else if err := guesthook.Register(ctx, b.runner, in.VMID); err != nil {
b.logger.Warn("provision: pre-start hook registration failed (non-fatal)", "vmid", in.VMID, "err", err)
}

Some files were not shown because too many files have changed in this diff Show More