Compare commits

..
177 Commits
Author SHA1 Message Date
admin db01ae0f2a agent (unreleased): a cut write of agent.json cannot leave a blind box — a journal of the old bytes, a repair at start (R-1380)
gates / gates (push) Successful in 42s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-15 08:20:49 +01:00
admin 54ac2c2ac9 agent (unreleased): the CA copy keeps its previous root only for the changeover, and is compared exactly (the commit's security review)
gates / gates (push) Successful in 47s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 19:20:07 +01:00
admin ec7e699fd7 agent (unreleased): fixes after an independent review — one lock for the writers of agent.json, the trust run serialised and bounded, the CA copy keeps its previous root, the profile stanza from the wrapper's answer, a stale give-up ends, both boot tokens from one read
gates / gates (push) Successful in 48s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 19:09:21 +01:00
admin 61feb95c06 agent (unreleased): the hub is told what the kernel clean-up did and what the profile refresh did (R-1383)
gates / gates (push) Successful in 58s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 18:06:12 +01:00
admin c0bf97de80 agent: gofmt (formatting only)
gates / gates (push) Successful in 40s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:47:42 +01:00
admin d5ce5c364a agent (unreleased): a boot token that a clock step cannot move (R-1395)
/disks answers guest_boot_key (the kernel's boot_id and the guest init's start) beside the unchanged guest_boot_id.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:43:33 +01:00
admin a1135018c7 agent (unreleased): a refused signed job and a watchdog give-up reach the hub (R-1383, D78)
gates / gates (push) Successful in 42s
The job runner clears a job with its outcome, a closed reason code and the op's name. The guest-power watchdog's
give-up rides the host report until the guest runs.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:29:53 +01:00
admin ede5d0e6c7 agent (unreleased): the report carries the box's clock at the moment of sending (R-1384, D79)
gates / gates (push) Successful in 46s
sent_at is stamped by Client.Report at every send, never by the collector. The hub reads it to tell a wrong box clock.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:09:12 +01:00
admin 386a0c4cf4 2b (unreleased): the agent moves from a pinned Proxmox certificate to the box's root CA copy — copy, proof, then the config; the cause of a trust failure is named (R-1380)
gates / gates (push) Successful in 38s
2026-11-14 16:35:22 +01:00
admin a9ab47ec71 2b (unreleased): the root step that keeps a copy of Proxmox's root CA (wrapper mode pve-ca-copy, R-1380)
gates / gates (push) Successful in 44s
2026-11-14 16:28:04 +01:00
admin 2706e35198 v0.174.0 RELEASED and VOUCHED: CHANGELOG, the day-0 pin, CONTEXT, REPORT
gates / gates (push) Successful in 40s
2026-11-14 15:23:16 +01:00
admin cc63dd5204 Old kernels removed by the box (R-1368, wrapper mode kernel-cleanup); security profiles refreshed after a kernel change (R-1388, mode profile-refresh); Go 1.27.4 (R-1391)
gates / gates (push) Successful in 40s
2026-11-14 12:11:06 +01:00
admin 91a2924ad0 docs: the tag push takes its token from the secret store; a repository address holds no credential (R-925)
gates / gates (push) Successful in 40s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-11 11:12:25 +01:00
admin 9edaa8c082 v0.173.0 vouched and delivered to the three test boxes; pinned.day0_agent = 0.173.0 (D50)
gates / gates (push) Successful in 43s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 16:30:01 +01:00
admin 5cebb766e3 v0.173.0 released (not vouched): CHANGELOG, REPORT, CONTEXT — the folder clean-up measured on three boxes, the rollback proven on Tester 1
gates / gates (push) Successful in 42s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 16:16:42 +01:00
admin 776025c824 selfupdate: the staging folder keeps what it needs and no more (R-1368 part 2) — not released
gates / gates (push) Successful in 35s
Each good update left its staged binary in <state dir>/selfupdate for ever
(49 files / 740 MB on demo-hp). After the wrapper's commit returned success,
MaybeCommit now removes every staged binary but the running version's.

Nothing is removed while a pending marker exists or cannot be read, at a
plain start, or after a rollback. Only a regular file with the exact name
Execute writes is removed. The rollback reads pending.json and
felhom-agent.prev, no staged binary: pinned by a test on the wrapper's text
and by a test that runs the wrapper's real commit and rollback verbs in a
sandbox. Red seen three times.

No version bump, no build, no vouch. /usr/local/bin/felhom-agent.bak-* is
not reached (root-owned; R-1368 part 4).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 14:10:00 +01:00
adminandClaude Opus 5.5 41d5eb8051 v0.172.1 vouched and delivered to the three test boxes (agent + bundle, 77/77); pinned.day0_agent = 0.172.1 (D50)
gates / gates (push) Successful in 44s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-07 23:29:24 +01:00
adminandClaude Opus 5.5 8bd137787e v0.172.1 released (not vouched): CHANGELOG, REPORT with the operator's PowerShell delivery, CONTEXT
gates / gates (push) Successful in 33s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-07 22:43:34 +01:00
adminandClaude Opus 5.5 39db03065a os-apply: proxmox-grub joins the boot-chain lane (pve step refused R6 nightly on demo-felhom)
gates / gates (push) Successful in 36s
proxmox-grub 2.12-9+pmx4 depends on grub-efi-amd64 of the same version. HOST_SLOW_RE
anchors `grub` at the start of a name, so proxmox-grub landed in the ring-0 pve plan,
the simulation pulled grub-efi-amd64, and the whole pve step was refused R6 every
night since 2026-11-05. Same class as the 2026-10-09 proxmox-secure-boot-support fix.

Test test_ring0_pending_pve_leaves_proxmox_grub_out: red before the fix, green after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-07 22:41:05 +01:00
admin ba409de4a2 v0.172.0 vouched: pinned.day0_agent = 0.172.0 (D50); CHANGELOG heading; REPORT + CONTEXT
gates / gates (push) Successful in 34s
2026-11-03 10:29:51 +01:00
admin a4f1280399 v0.172.0 released, not yet vouched: CHANGELOG heading (R-1302; agent 2c0222cd, bundle cc34b0bc)
gates / gates (push) Successful in 36s
2026-11-03 09:08:07 +01:00
admin ecbb0cc479 REPORT: 2026-11-02 night, R-1302, commit only
gates / gates (push) Successful in 36s
2026-11-03 04:13:52 +01:00
admin 50eeccc986 R-1302: the first grant of a tier logs INFO granted; a grant that existed and vanished keeps the R-190 ERROR. Commit only.
gates / gates (push) Successful in 36s
2026-11-02 23:16:10 +01:00
admin 808cdb36ad REPORT + CONTEXT: v0.171.0 released and vouched; R-1344 restart proven live
gates / gates (push) Successful in 33s
2026-10-30 11:09:48 +01:00
admin 84726c9f27 v0.171.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.171.0 (D50)
gates / gates (push) Successful in 33s
2026-10-30 10:49:46 +01:00
admin bd18347c1c R-1344 decided from the night's measurement: a controller that runs but does not answer is restarted at 10 unhealthy sweeps (alarm at 6) — busy file + the brake refuse; stanza carries unresponsive_restart_after_s / unresponsive_restart_at
gates / gates (push) Successful in 33s
2026-10-30 09:20:37 +01:00
admin 98bd1b71e9 REPORT + CONTEXT: v0.170.0 released and vouched; reboot_guest proven on the release builds
gates / gates (push) Successful in 39s
2026-10-29 20:24:29 +01:00
admin 6578a8b91e v0.170.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.170.0 (D50)
gates / gates (push) Successful in 36s
2026-10-29 18:59:05 +01:00
admin ca15a31690 D65 = A: reboot_guest joins the agent's closed operator-action list — guards (busy controller file, heavy gate, jobs, lock, parked, 1/hour, the brake), result with both times, GET /guest/operator-reboot for the household line; one read-only sudo vector
gates / gates (push) Successful in 33s
2026-10-29 16:04:19 +01:00
admin bdea4694a4 REPORT + CONTEXT: v0.169.0 released and vouched; restart_controller proven on the release builds
gates / gates (push) Successful in 35s
2026-10-29 15:11:11 +01:00
admin db25eb0e71 v0.169.0 released and vouched (14:06:09 CET): CHANGELOG; D50 pinned.day0_agent = 0.169.0
gates / gates (push) Successful in 42s
2026-10-29 14:09:04 +01:00
admin 637c92c762 Operator actions for the agent: restart_controller from the hub (closed list, the supervisor's guards, the brake counts operator restarts); R-1344: a running-but-unhealthy controller raises unresponsive — alarm only; tests red first
gates / gates (push) Successful in 36s
2026-10-29 12:30:36 +01:00
admin 63e63ee8bb v0.168.0 released and vouched: CONTEXT, REPORT
gates / gates (push) Successful in 37s
2026-10-29 10:48:37 +01:00
admin 6b0b093199 D50: pinned.day0_agent = 0.168.0 (vouched by the operator 2026-10-29 10:21:56 CET); CHANGELOG notes the vouch
gates / gates (push) Successful in 36s
2026-10-29 10:33:58 +01:00
admin f4e63a55da v0.168.0: CHANGELOG — released (agent 1e70196c…, bundle 3415c034…), committed on demo-felhom, Tester 1, demo-hp; not vouched yet
gates / gates (push) Successful in 37s
2026-10-29 10:15:55 +01:00
admin 91bd96c755 REPORT + CONTEXT: R-528 and R-1254 on main, not released (night 2026-10-28)
gates / gates (push) Successful in 35s
2026-10-29 02:51:00 +01:00
admin 6f933d6ee3 R-528: GET /guest/oom-kills — the kernel's per-container OOM counter of the token's guest, read on the host (no sudo)
gates / gates (push) Successful in 34s
Tests TestR528_* (fake cgroup tree), red-proved by mutation. Commit only; the agent release needs the vouch.
2026-10-28 23:22:40 +01:00
admin 5ddd124cfc R-1254 (ceremony half, HUNT-3 H2): after a failed escrow upload, ask the hub which package it holds before wording the result
gates / gates (push) Successful in 44s
The hub holds the new package -> success; another package -> 502 'previous code still works' (now true);
cannot be asked -> 504, no promise. Tests TestR1254_* red before the fix. Commit only; the agent release needs the vouch.
2026-10-28 22:51:08 +01:00
admin 3055bc54c1 D50 pin: day0_agent 0.167.0 (the operator's vouch 2026-10-27, read from the hub)
gates / gates (push) Successful in 1m4s
2026-10-27 20:45:25 +01:00
admin 007199342f v0.167.0 RELEASED (not vouched yet): CHANGELOG header, REPORT
gates / gates (push) Successful in 1m20s
2026-10-27 15:49:30 +01:00
admin 54d3a1057b CHANGELOG: the unreleased header names R-972 (Go 1.27.3)
gates / gates (push) Successful in 59s
2026-10-26 22:21:47 +01:00
admin fbbb168d1f build(R-972): Go 1.27.3 (GO-2026-6809, crypto/x509) — govulncheck 1 → 0 called
gates / gates (push) Successful in 1m22s
2026-10-26 20:34:25 +01:00
admin 3911cc6f52 fix(R-1232): the tier due-check counts only this box's archives (own key) — a reinstalled box's first whole-guest copy is not a week late
gates / gates (push) Successful in 1m20s
Red-proved (the old box's archive was counted). Commit only.
2026-10-26 20:15:39 +01:00
admin ad80ee749a docs: CONTEXT — R-1285 state and Tuesday's line test
gates / gates (push) Successful in 1m15s
2026-10-26 18:11:24 +01:00
admin 0683cc2dab REPORT: the 2026-10-26 afternoon run. Docs only.
gates / gates (push) Successful in 1m48s
2026-10-26 14:38:15 +01:00
admin b1bc63dd32 R-1285: the new agent runs the bundle policy pass once after its self-update commits — in the normal two-job delivery the old process installs the bundle. Test in the real order (commit, then one bundle-policy plan); red-proved. Commit only, not released.
gates / gates (push) Successful in 2m15s
2026-10-26 12:44:25 +01:00
admin e66a48f2b7 REPORT: the 2026-10-26 attended release run. Docs only.
gates / gates (push) Successful in 1m41s
2026-10-26 10:38:03 +01:00
admin b890b57540 v0.166.0 released and vouched by the operator (10:04:55 CET): CHANGELOG header (R-1280 live, R-1285 back to OPEN — the bundle job runs in the old process); D50 day0_agent pin → 0.166.0. Docs only.
gates / gates (push) Successful in 1m52s
2026-10-26 10:07:15 +01:00
admin aa69114d18 REPORT: the 2026-10-25 night — committed fixes, CI by hash, teardown. Docs only.
gates / gates (push) Successful in 1m43s
2026-10-25 23:54:07 +01:00
admin abcc1e6833 R-1285: after every bundle install the agent asks the wrapper now on disk for the appliance-only policy pass, so a BYO host upgraded from an older wrapper loses our SSH drop-in in one update. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:46:38 +01:00
admin d912307bb7 R-1280: a signed job's validity window is at most 24 h — the verifier, the root wrapper and felhom-opsign. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:42:18 +01:00
admin feaaa0e8ff D50 = A (R-1290): the published gate asserts the newest 5 released versions plus the pinned MinAgent (0.158.1) and day-0 agent (0.165.0); a pin with no tag fails; decoys red-proved. No binary change.
gates / gates (push) Successful in 1m41s
2026-10-25 20:20:30 +01:00
admin 6b6ba0c17f v0.165.0 vouched by the operator (18:38:49 CET); REPORT: release, delivery to four hosts, the BYO proof (removal one bundle late, felhom.eu R-1285). Docs only.
gates / gates (push) Failing after 1m49s
2026-10-25 19:02:48 +01:00
admin 4697f8ac9e v0.165.0: CHANGELOG names the release (R-1257 D43 = A, R-1250); released from a clean clone, sha256 e9d04a47…, bundle 8dea1f34…; not vouched
gates / gates (push) Successful in 1m38s
2026-10-25 17:00:02 +01:00
admin 4a1a84941c D43 = A run close-out: REPORT and CONTEXT. Docs only.
gates / gates (push) Successful in 1m40s
2026-10-24 21:04:19 +02:00
admin c8d3c17b13 D43 = A (R-1257): the root-key-only sshd drop-in only on an appliance; on BYO nothing is written and our copy is removed; the in-force check reads the LAN addresses (X3); a failing sshd -t undoes only the SSH step (X4). Commit only, not released.
gates / gates (push) Successful in 1m32s
2026-10-24 20:47:17 +02:00
admin f5c3d77a61 Review round 2: .claude/rules/unprompted-work.md aligned with its four siblings (the missing 'A version number is pushed once' fact paragraph); no code change
gates / gates (push) Successful in 1m32s
2026-10-24 15:50:59 +02:00
admin 5e05627c4e REPORT: R-1250 test on request, live proof on demo-hp
gates / gates (push) Successful in 1m41s
2026-10-24 13:48:07 +02:00
admin 1e087bb158 linetest: waiting_for_quiet needs the full watched window (R-1250)
gates / gates (push) Successful in 1m35s
The live proof found a fresh agent answering waiting before it had watched one minute; a quiet
box's torrent app would be held for nothing.
2026-10-24 13:24:41 +02:00
admin 29bcf4403e linetest: a test on request for a box that seeds all day (R-1250)
gates / gates (push) Successful in 1m37s
GET /host/line-test reports waiting_for_quiet (due, in the window, tunnel up, only the quiet
missing); POST /host/line-test/run runs one measurement with a 5 s settle and a 10 s quiet
check instead of 10 quiet minutes, runs 5 s apart (~55 s). Every other weekly rule holds.
Commit only; the release rides the next train.
2026-10-24 13:18:59 +02:00
admin 08ca67861f CHANGELOG: v0.164.0 released (the weekly upload line test, R-1247), delivered to the three test boxes, vouched by the operator; REPORT
gates / gates (push) Successful in 1m32s
2026-10-24 09:46:00 +02:00
admin f7e1dcbb27 R-1247: the weekly upload line test — the agent's own iperf3 client to ep0's tunnel-only server, Tuesday 10:00-15:59 after 10 quiet minutes, under the heavy-op gate, best of 3; line_test stanza + GET /host/line-test; 15 tests red-proved; unreleased (next v0.164.0)
gates / gates (push) Successful in 1m35s
2026-10-24 08:59:42 +02:00
admin 2d4b1930df REPORT: day 2026-10-23 — agent 0.163.0 released, delivered (step bundle), vouched by the operator
gates / gates (push) Successful in 1m48s
2026-10-23 11:08:14 +02:00
admin b4721eee11 CHANGELOG: v0.163.0 released (D30, D4 root SSH by key only, R-1204)
gates / gates (push) Successful in 2m5s
2026-10-23 08:44:19 +02:00
admin 1109092def D4 (R-982): root SSH from the LAN by key only — the bundle's sshd drop-in, try-reload-or-restart, an sshd -T read-back reported when not in force, a reload in undo (the reviewed v2 patch); unreleased
gates / gates (push) Successful in 2m9s
2026-10-23 08:22:17 +02:00
admin b45c47be6c REPORT: night 2026-10-22/23 (D30 live, hardened; D4 patches v2)
gates / gates (push) Successful in 2m1s
2026-10-23 00:06:13 +02:00
admin b9b831fb90 D30: a wait stamp dated after now (a clock that jumped back) is dropped, never a block until that date — the commit's security review; red-proved; unreleased
gates / gates (push) Successful in 2m3s
2026-10-22 23:08:25 +02:00
admin 39fb9cbf6c D30 rotation hardened after review: the key file is replaced by an fsynced rename over it (never missing or empty), the first old key stays private.key.prev, later ones dated (3 kept), the once-an-hour bound survives a restart; red-proved; CONTEXT; unreleased
gates / gates (push) Successful in 2m24s
2026-10-22 23:00:05 +02:00
admin ae69ad4512 D30 (R-1204): a handshake stamped in the future for 15 min gives a new WireGuard key, registered on the existing peer path (the hub re-keys in place; its full-list push swaps the key on ep0 — nothing new on ep0); old key kept as private.key.prev; red-proved; unreleased
gates / gates (push) Successful in 2m0s
2026-10-22 20:52:52 +02:00
admin e803ad2b91 night 2026-10-22: REPORT; CHANGELOG corrected — a push of the installer to main publishes nothing (R-110, tag installer-v1.35.0); D4 stays a patch because both halves need a release
gates / gates (push) Successful in 2m24s
2026-10-22 18:13:35 +02:00
admin 7651461be2 D4 (R-982) moved out of main into a ready patch — its installer half would publish the installer (forbidden tonight); R-1204 stays; unreleased
gates / gates (push) Successful in 2m45s
2026-10-22 17:10:46 +02:00
admin a0c88e5183 R-982 (D4): root SSH from the LAN by key only — a bundle drop-in, checked and reloaded by felhom-os-apply; R-1204: the heartbeat reports a future-stamped tunnel handshake; unreleased
gates / gates (push) Successful in 2m23s
2026-10-22 16:58:45 +02:00
admin 2d5840ba11 v0.162.0: released (operator yes 2026-10-22), delivered to the 3 test boxes; CHANGELOG + REPORT
gates / gates (push) Successful in 2m10s
2026-10-22 10:39:06 +02:00
admin b310773dbf REPORT: the night of 2026-10-21 (host swap report, R-1194), unreleased
gates / gates (push) Successful in 2m7s
2026-10-22 00:06:21 +02:00
admin 0da792e670 R-1192 follow-up: report host swap vs the guests' swap allowance (log once, hub alarm, local API); R-1194: no OOB stanza before the first felhom-sshd pass; unreleased
gates / gates (push) Successful in 1m56s
2026-10-21 21:29:07 +02:00
admin 286a7e3ecf R-1192: every guest may swap as much as its RAM (bring-up, RAM resize, once at start) — the measured 8 GB freeze; unreleased
gates / gates (push) Successful in 1m39s
2026-10-21 15:43:18 +02:00
admin 4e1621c650 CHANGELOG + CONTEXT + REPORT: v0.161.0 released (operator yes) and delivered to demo-hp, Tester 1, demo-felhom; not vouched; R-1083 left out
gates / gates (push) Successful in 1m27s
2026-10-20 09:07:30 +02:00
admin 51521230c9 REPORT: the night's agent fixes (R-921, R-1070, R-1010), not released
gates / gates (push) Successful in 1m25s
2026-10-19 21:47:40 +02:00
admin 84ef5818d5 security-scan gate (shared felhom.eu/scripts/security_scan_gate.py) in the fast set and CI, warn-first, with the accepted-hits file and decoys (R-986)
gates / gates (push) Successful in 1m30s
2026-10-19 21:11:46 +02:00
admin 56aabc729c Unreleased: host_busy on GET /backup/status (R-921), a missing tunnel container is 'absent' and not-applicable after an OS update (R-1070), the janitor forces a half-removed vzdump snapshot once (R-1010)
gates / gates (push) Successful in 58s
2026-10-19 19:52:37 +02:00
admin 0cbd500154 gofmt gate in agent_gates.py with its decoys (R-1160); REPORT for the day
gates / gates (push) Successful in 49s
2026-10-19 10:26:27 +02:00
admin 1edfd8e331 gofmt -w: 14 files, formatting only, no behaviour change (R-1160) 2026-10-19 10:24:32 +02:00
admin 0bc0bc5da0 CONTEXT + REPORT: v0.160.0 released, on the three test hosts, not vouched
gates / gates (push) Successful in 48s
2026-10-19 05:41:34 +02:00
admin 45458196d1 Two texts made true after 0.160.0: the ceremony log line names the confirm order; the R-973 comments name whole_guest_backup_refused (no behaviour change)
gates / gates (push) Successful in 52s
2026-10-19 05:25:58 +02:00
admin 124164db73 CHANGELOG v0.160.0 (released: R-978, R-973, R-1093, R-1149, R-1151)
gates / gates (push) Successful in 46s
2026-10-19 03:22:00 +02:00
admin c11e3e2a87 The new escrow package reaches the hub only after the household typed two words of the code back: POST /escrow/ceremony/confirm; an old controller keeps the claim order (R-1151, unreleased)
gates / gates (push) Successful in 48s
2026-10-19 03:03:37 +02:00
admin 1b6e31675b The local whole-guest tier keeps one restore point per day for 3 days: keep-daily instead of keep-last (R-1149, operator D17, unreleased)
gates / gates (push) Successful in 47s
2026-10-19 02:59:34 +02:00
admin 1693da4d1a REPORT: the three unreleased fixes (R-1093, R-978, R-973) with the R-1093 loop-device check on demo-hp
gates / gates (push) Successful in 47s
2026-10-18 21:48:03 +02:00
admin 79ba884859 CI workflow: the failure mail links the public run page, by run id (R-1135)
gates / gates (push) Successful in 49s
The alarm step built its link from GITHUB_SERVER_URL, which on this runner
is the in-cluster Gitea address, and from the run NUMBER, which Gitea 1.26
reads as a run id first (felhom.eu /actions/runs/1116 opened run id 1116 =
run number 613). Now https://gitea.dooplex.hu + GITHUB_RUN_ID, and the link
is printed into the job log as ALARM-LINK. Same change in all four repos.
2026-10-18 20:34:59 +02:00
admin f17e1979bd A cadence cap on the local-API whole-guest backup route (R-973, unreleased)
gates / gates (push) Successful in 49s
Each successful run prunes its tier, so a compromised controller calling POST /backup in a loop
pushed out every pre-compromise restore point. A tier with a cadence of at most 24 h now allows 3
successful runs per rolling 24 h (night, a press, the next night); a longer tier 1 per
max(cadence - 24 h, cadence/2), 6 days for the weekly off-site tier. Only successes count. A
refusal is a 429, a WARN, saved, and for 24 h a critical host-report status that the hub turns
into its existing operator-only agent_capability_degraded event. The success history and the
refusal are kept in the R-894 state file, so a restart does not reset the cap. No sudoers change.
Tests TestBackupCap_* and the state, status and wiring tests, red-proved.
2026-10-18 19:53:15 +02:00
admin 0963bae3c3 The partition probe fails closed on every path (R-1093 review, unreleased)
PART_ENTRY_* comes from the parent's table, so it proves nothing about a partition's contents
unless blkid also completed: lsblk's PTTYPE is now set aside only on a blkid exit 0 (exit 2, an
I/O error or a missing binary keep "device has a partition table"). An lsblk that exits 0 with
output that does not parse into a device, or lists none, leaves the device unprobed instead of
blank (a hole that predates R-1093). Six more TestInspect_R1093_PartitionVerdicts rows, each
red-proved.
2026-10-18 19:47:15 +02:00
admin 8561dc8a5c The new escrow package reaches the hub only when its code is claimed (R-978, unreleased)
The root ceremony uploaded the new package before the new code was claimed, and the hub
overwrites the current package when the sealed secrets are unchanged: a customer who started the
ceremony again and walked away kept a written-down code that opened nothing. Json mode (the
FELHOM_ESCROW argv, unchanged) now returns the package (CeremonyOutput v2, PendingUpload); the
daemon uploads it at POST /escrow/ceremony/claim before handing out the code. A failed upload is a
502 with the code still claimable; TTL and a new start wait for an in-flight claim upload; no
uploader is a 503. Text mode still uploads at once. No sudoers change.
Tests TestEscrowCeremony_R978_* (localapi) and the cmd-side defer/delivery/wiring tests, red-proved.
2026-10-18 19:42:02 +02:00
admin f7b7308c4d An empty partition is blank: lsblk's PTTYPE on a partition is the parent's (R-1093, unreleased)
lsblk prints the disk's table type on every partition, so InspectDevice read an empty partition as
"device has a partition table" and the format path asked the typed erase confirmation. lsblk's
PTTYPE is now set aside only when blkid -p on the partition itself proved it read the device and
that it is a partition of a table of that exact type (PART_ENTRY_NUMBER + matching
PART_ENTRY_SCHEME). A filesystem or a nested table in the partition, a silent blkid, a whole disk
or a different scheme keep the data-bearing reading. Same allow-listed argv, no sudoers change.
Test TestInspect_R1093_PartitionVerdicts, red-proved against the old predicate.
2026-10-18 19:35:03 +02:00
admin 27bbdaaa22 REPORT/CHANGELOG: agent 0.159.2 delivered and proven on demo-hp
gates / gates (push) Successful in 51s
2026-10-18 14:45:47 +02:00
admin e25454ccc3 CHANGELOG: v0.159.2 released (the after-failure orphan check)
gates / gates (push) Successful in 59s
2026-10-18 12:54:28 +02:00
admin 65c1994d3b CHANGELOG (Unreleased): the live proof the next release owes for the after-failure orphan check
gates / gates (push) Successful in 47s
2026-10-17 23:54:14 +02:00
admin 6545cccff7 The orphan thin check runs only after a failed snapshot backup (R-1012 follow-up, unreleased)
gates / gates (push) Successful in 48s
v0.159.0/v0.159.1 ran the check before every snapshot-mode whole-guest backup, which takes a
metadata snapshot of the live pool. Now explainFailure (vzdump start or task failure) is its only
caller; the finding is still appended to the failed backup's error. Tests: failure = 1 call,
success = 0 calls; red-proved against the v0.159.1 runner.go (2 calls / calls=1).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 22:43:03 +02:00
admin eb38d78945 REPORT/CONTEXT: agent 0.159.1, the catalog allow-list (controller v0.328.0), its catalog gate (2026-10-17)
gates / gates (push) Successful in 58s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 17:29:50 +02:00
admin 4a1c07b288 CHANGELOG: v0.159.0 (superseded) and v0.159.1 — released, not vouched
gates / gates (push) Successful in 48s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 16:42:00 +02:00
admin 5c3b218822 The signed decommission prunes the drive from every guest's bind record (found live on demo-hp, R-1124 proof)
gates / gates (push) Successful in 47s
The self-serve decommission already dropped the drive from the guest-bind record; the operator-signed one did
not, so a forgotten drive left a record the 20-second re-assert warned about forever ("skipping non-enrolled
drive"). DecommissionExecutor.SetGuestBinds; red-proved. Code only; the CHANGELOG follows the release (0.159.1).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 16:41:12 +02:00
admin e15d1d30ea R-1124 + R-1012: forget a decommissioned drive that is gone; name an orphan thin device in a backup failure
gates / gates (push) Successful in 47s
R-1124: ForgetRetiredDrives (start-up + the 20 s tick) disables and removes the host mount unit of a
DECOMMISSIONED drive whose disk is absent and unmounted; the agent then no longer lists it. A present
decommissioned drive keeps its unit (one-click re-enroll). The file goes only through the new
felhom-priv-apply verb `unit-forget`, which re-checks marker, disabled, unmounted and absent as root.

R-1012: the read-only root helper felhom-thin-check <vg> <pool> (bundle) lists the pool's thin ids
(metadata snapshot reserved and released) next to LVM's before and after. BackupRunner.SetThinCheck runs
it before a snapshot-mode backup (log) and again after a failure, appending the finding in plain words to
the error the controller mails to the operator. Two reads, intersected; fails open; never removes.

Red-proofs: felhom.eu audits/agent-allowlist-rotation-2026-10-17/P1/. Code only; the CHANGELOG entry
follows the release.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 15:20:34 +02:00
admin 344c4f9030 CI workflow: a manual proof that the failure mail can be sent (R-1129)
gates / gates (push) Successful in 59s
workflow_dispatch with input force_alarm (default false). A dispatch-only
step prints the RESEND_API_KEY length (never the key) and, when forced,
fails so the alarm step sends its real mail, subject marked FORCED TEST.
Push runs are unchanged. No agent release.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 12:49:37 +02:00
admin c52d591851 The decoy suite is a gate of the full run (R-1116); scripts only, no release
gates / gates (push) Successful in 49s
scripts/test_gate_decoys.py is now agent_gates.py's `gate-decoys` row: the full run (which CI runs on every
push), not --fast (~20 s in the workspace, 30 cases). Declared in its own COVERS; test_agent_gates.py pins it.
Red-proved: a reuse-refs FACT case flipped to expect rc 0 -> CONVICTED: gate-decoys; restored, green.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 11:01:25 +02:00
admin 37b99d73b3 Go 1.27.2 and x/crypto v0.58.0 (R-972) — built and tested, not released
gates / gates (push) Successful in 31s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-16 18:53:33 +02:00
admin dd8b2ace02 release-agent.sh: only a clean 404 proves a version is new (R-1098)
gates / gates (push) Successful in 31s
An unreachable registry (000) or a server error passed the existing-version check as new. Test:
scripts/test_release_push_once.sh, red-proved. No agent release.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-16 17:23:46 +02:00
admin 383522450a REPORT + CONTEXT: v0.158.1 build B, vouched; R-1093 filed
gates / gates (push) Successful in 32s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 10:10:21 +02:00
admin 1a8bed11e7 CHANGELOG: v0.158.1 (security review of 0.158.0; 0.158.0 superseded, do not vouch)
gates / gates (push) Successful in 25s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 04:02:17 +02:00
admin 98f3cdd76a Security review of 912f7c1: the mkfs wrapper trusts no sibling claim; the internal-place helper fails closed
gates / gates (push) Successful in 25s
- felhom-mkfs-guarded: the 0.158.0 exemption for a sibling part mounted at /mnt/<name> + under /mnt/felhom-drives is
  removed — the agent can make that corroborating mount itself. Every mount of every node outside /mnt/felhom-drives
  refuses (kernel mount table); an unlistable disk refuses. CandidatePartition.Formattable is false for every part of a
  disk with a mounted node, so it is never offered for formatting (attach still is).
- felhom-internal-place: a failed lvs listing, a listing without the pool, or a blkid exit other than 0/2 refuses
  (they read as "nothing promised" / "no signature" before).
Every new test was seen failing against the 0.158.0 versions.

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 04:02:00 +02:00
admin 888a98571b CHANGELOG: v0.158.0 — the internal data place, partitions, add_internal_place
gates / gates (push) Successful in 28s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 03:56:29 +02:00
admin 912f7c127e The internal data place, partitions as places, add_internal_place (felhom.eu architecture/16 §5-§7, §9; build B)
gates / gates (push) Successful in 27s
- internal place: the record /var/lib/felhom-agent/internal-place.json -> mounted at /mnt/belso by fs-UUID like an
  enrolled drive (startup + 20 s tick), kind "internal" in GET /disks; never takes a household drive's /mnt/belso;
  eject/decommission/format refused (dm on the system disk).
- configs/felhom-internal-place: the ruled size rule and the only lvcreate/mkfs of a system-disk volume (installer +
  the new signed op add_internal_place, operator key, no parameters).
- partitions listed per free disk (empty / filesystem / in_use); a data-bearing part's wipe id binds disk id + part
  number + fs-UUID.
- felhom-mkfs-guarded: Felhom's own sibling drive no longer blocks formatting the free part; an unlistable disk refuses.
Tests red-proved (script mutations, Go mutations, the old wrapper measured against the new tests).

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 03:55:47 +02:00
admin e569ec6c5f REPORT + CONTEXT: v0.157.0 live proof on the three test boxes; not vouched
gates / gates (push) Successful in 30s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-15 21:54:58 +02:00
admin 52ed52c131 CHANGELOG: v0.157.0 — physical-disk identity + POST /disks/same-disk
gates / gates (push) Successful in 27s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-15 21:09:54 +02:00
admin c5daa9480d Physical-disk identity of every storage place + POST /disks/same-disk (felhom.eu architecture/16 §3/§4)
gates / gates (push) Successful in 31s
The agent names the physical disk(s) under each storage place from /sys, udev and /proc/mounts only
(partition -> disk, dm/LVM thin -> slaves, loop -> backing file), keys them (eui / serial:<bus>:<model>
:<serial> / vendor WWN) and says proven or unknown (vm, network, usb_bridge, weak_wwn, no_serial).
GET /disks and the hub report's storage_targets gain kind, physical_disks, identity,
identity_reason (golden updated; the hub copy changes in the same session). POST /disks/same-disk
maps two guest paths to places the agent itself made and answers {different, proven, reason};
an absent drive is unmapped, never the root disk. Tests on fixture trees for every spike case,
red-proofed. CHANGELOG entry follows the release (release-complete gate).

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-15 21:09:18 +02:00
admin d2053e7d47 Rule file: a release needs no golden; the floor carries it after the three test boxes; golden weekly (09 §3 decision 250, operator 2026-10-15)
gates / gates (push) Successful in 27s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-15 11:43:27 +02:00
admin 7d4e564758 Rule file: bake first — bake → vouch → floor, never a floor with no golden (09 §3 decision 247, operator 2026-10-15)
gates / gates (push) Successful in 25s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-15 08:18:13 +02:00
admin 9b749ee7ad v0.156.1: CHANGELOG release entry (sha, bundle, step bundle, tag); 0.156.0 superseded
gates / gates (push) Successful in 52s
2026-10-14 11:36:39 +02:00
admin 7ae3081a76 R-1008: in flight = the worker process lives (Proxmox's pid+pstart rule), not a wall-clock start vs btime (security review: a clock step could make a live backup look dead)
gates / gates (push) Successful in 46s
2026-10-14 11:36:04 +02:00
admin 7ecd63cd61 v0.156.0: CHANGELOG release entry (sha, bundle, step bundle, tag)
gates / gates (push) Successful in 32s
2026-10-14 11:30:46 +02:00
admin a2f3aeeeff After a power cut: remove the dead half-written backup (R-1006); a backup task from before the boot is not in flight (R-1008)
gates / gates (push) Successful in 35s
2026-10-14 11:29:41 +02:00
admin e7d4ecd607 REPORT: the machine came back by its power button; the hub ended the hold (R-998 live proof complete)
gates / gates (push) Successful in 36s
2026-10-14 00:05:55 +02:00
admin 10ca63aa32 REPORT: power-off (R-998) released and proven on the N100
gates / gates (push) Successful in 34s
2026-10-13 21:41:30 +02:00
admin 68a2130681 v0.155.0: CHANGELOG release entry (sha, bundle, tag)
gates / gates (push) Successful in 31s
2026-10-13 20:18:21 +02:00
admin bdf25f019c Host power-off: a guest started after acceptance cancels it; escrow reply data race fixed
gates / gates (push) Successful in 30s
Security review of c6d9435: the other-guests check ran only at acceptance. waitJobsClear now
looks again before each step and CANCELS at once when another guest or VM runs (gate released).
Red-proofed. Also: go test -race found a data race in the escrow ceremony start reply (reads
job.Phase while the job goroutine writes it) — the reply fields are copied before the goroutine
starts; -race was red before, green 3/3 after.
2026-10-13 19:38:23 +02:00
admin c6d9435316 Host power-off: re-check jobs before each step; cancel if one is still running (R-998)
gates / gates (push) Successful in 29s
Security review of f38b03c: a controller update, drive/NAS job, escrow ceremony or Proxmox
vzdump could start between acceptance and the power-off (the heavy gate does not cover them).
runHostPowerOff now asks jobBlocker again before the guest shutdown and before the host
power-off, waits up to 15 min, and CANCELS (gate released, box stays on) if still busy.
Red-proofed: no wait, and a gate kept after a cancel, each convicted.
2026-10-13 19:36:01 +02:00
admin f38b03cdb8 Host power-off: refuse while another guest or VM runs; hold the heavy gate; fail closed (R-998)
gates / gates (push) Successful in 33s
Security review of 3403c87: (1) any guest's controller could switch off a shared host — demo-hp
runs a scratch guest and the Tester 1 VM. Now refused (other_guests_running) when /proc shows any
other lxc-start -n N or kvm -id N; an unreadable /proc refuses. (2) the checks ran once, so a
backup could start before the power-off: the heavy-operation gate is now taken on acceptance and
kept (released only if the power-off command fails). (3) an unwired gate or task reader refused
nothing; now refuses. Each red-proofed.
2026-10-13 19:33:09 +02:00
admin 3403c87c80 Host power-off for the household (R-998): POST /host/poweroff
gates / gates (push) Successful in 34s
The guest's controller asks; the agent refuses in the first 10 min after boot, a second time in
one boot (persisted, keyed by /proc/stat btime), and while a backup, restore test, OS step,
controller update, drive format, NAS verify, escrow ceremony or vzdump runs (each a named 409
reason). On acceptance: the record first, a host report carrying deliberate_off accepted by the
hub (bounded 60 s), then pct shutdown --timeout 120, then systemctl poweroff.

Sudoers: two exact vectors (pct ^shutdown N --timeout N$, systemctl poweroff) in FELHOM_POWEROFF;
nine injection cases. New TestEveryAliasIsGranted: a defined alias missing from the grant line
passed the coverage check before. Every new test red-proofed.
2026-10-13 19:29:52 +02:00
admin 86adbcaf8a rules: "The system poster stays true" (section 6, all five copies identical)
gates / gates (push) Successful in 57s
The shared rule file carries the same wording in felhom.eu, felhom-controller,
felhom-agent, app-catalog-felhom.eu and the workspace root on DooPlex --
"change all five or none" -- so this is this repo's copy of a rule added in
felhom.eu a08bd3cbd5.

A session that changes a fact listed in
architecture/felhom-system-poster.facts.md (a machine, a role, a traffic path,
a backup tier, a time, a retention, a key, a known gap) updates that file in
the same commit; fixes the poster's text if the change is text only; or adds
"System poster needs a refresh: <what changed>" to STATUS. The report names
which of the three it did.

The poster is drawn in Claude Design and nothing in the repo renders it, so
scripts/poster_facts_gate.py only WARNS when the facts outrun the drawing --
it never fails a push, because a refresh needs the operator and another tool.

Verified identical to the other four copies by diff, before and after.
2026-10-09 18:10:19 +02:00
adminandClaude Opus 5.5 24ea960229 REPORT: v0.154.0 released and delivered (2026-10-09)
gates / gates (push) Successful in 1m7s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 08:36:51 +02:00
adminandClaude Opus 5.5 cff6fb5a45 v0.154.0: CHANGELOG release entry (sha, bundle, tag)
gates / gates (push) Successful in 1m11s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 06:58:00 +02:00
adminandClaude Opus 5.5 db25b469ba Host report keeps the last backup across a restart; Secure Boot meta-package leaves the Proxmox lane
gates / gates (push) Successful in 1m7s
Found in the 2026-10-09 kernel-night read-back:
- demo-felhom: the kernel step restarted the host 4 minutes after the night
  backup; the in-memory backup list was empty after the restart, so the hub
  alarmed "host tier: newest backup is 48h old". The report now adds R-894's
  saved newest success per tier (one shared instance).
- demo-hp: proxmox-secure-boot-support pulled shim-signed-common into the
  ring-0 Proxmox plan; the step was refused R6 and the kernel step skipped.
  It joins HOST_SLOW_RE with shim and GRUB.

Red-proved: TestCollectBackups_SavedSuccessSurvivesARestart,
TestR894_LastKnownBackupsIsWiredIntoTheDaemon,
test_ring0_pending_pve_leaves_the_secure_boot_meta_out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 06:47:58 +02:00
adminandClaude Opus 5.5 b228b44f0e REPORT/CONTEXT: 2026-10-08 day (R-899, R-304 on main, unreleased)
gates / gates (push) Successful in 52s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 08:08:53 +02:00
adminandClaude Opus 5.5 91b9405c81 R-304: no 'wrong code' when earlier sealed packages were not all checked (424 older_unchecked)
gates / gates (push) Successful in 55s
Unreleased; ships with tomorrow's release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 07:53:01 +02:00
adminandClaude Opus 5.5 4c69c25b48 R-899: no OS leg after a household press (trigger=manual); after-boot kernel reports carry the saved ring
gates / gates (push) Successful in 50s
Unreleased; ships with tomorrow's release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 07:34:19 +02:00
adminandClaude Opus 5.5 c9013bb47d CHANGELOG/REPORT: v0.153.0 released and delivered
gates / gates (push) Successful in 1m33s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 19:06:04 +02:00
adminandClaude Opus 5.5 2d1e5d0774 R-898: ring 0 stages exactly the told kernel (select listed, KernelSet(kver))
gates / gates (push) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 18:45:28 +02:00
adminandClaude Opus 5.5 f09c53efc1 REPORT/CONTEXT: v0.152.0 the kernel lane
gates / gates (push) Successful in 1m9s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 18:20:11 +02:00
adminandClaude Opus 5.5 92d647a6f6 CHANGELOG: v0.152.0 released (shas, step bundle); host_stale is 45 min (comment fix)
gates / gates (push) Successful in 1m0s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 15:42:26 +02:00
adminandClaude Opus 5.5 d03ab7f1f5 R-836: the kernel lane — one-shot boot through the ESP flag, boot good / one self-revert, night step on a told night
gates / gates (push) Successful in 44s
Wrapper layer kernel (stage / reboot / boot / good / revert / cancel / status;
R20-R23), the two GRUB generators in the bundle (option C on the one-shot
entry), the agent's night step and after-boot judge (host health rule + hub
reached, 20 min measured), the signed os_kernel_step (stage only).
Red-proofs: felhom.eu audits/kernel-lane-2026-10-07/A/redproof.txt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 15:18:24 +02:00
adminandClaude Opus 5.5 b17d1c597d REPORT/CHANGELOG: 2026-10-07 day
gates / gates (push) Successful in 59s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 14:21:01 +02:00
adminandClaude Opus 5.5 0ae01dfb52 CHANGELOG: v0.151.0 released (R-861, R-812 A, R-366, R-105)
gates / gates (push) Successful in 58s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 13:14:47 +02:00
adminandClaude Opus 5.5 dd7cdc09e7 R-366 slice 2 (decision 168): the host report carries the archives the restore-test skipped as another key's
gates / gates (push) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:38 +02:00
adminandClaude Opus 5.5 7b0a8b234b R-105 (decision 169): remove the escrow-create -directive flag and the upload's directive field
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:38 +02:00
adminandClaude Opus 5.5 ce1a4b4758 R-812 option A: the Proxmox package lane (layer pve) + the /etc/pve write gate
The wrapper gains layer "pve" (slow lane): the host's Proxmox userspace
packages only — origin "Proxmox Debian Repository", never a kernel / boot /
firmware / microcode name (R14), no removal, no undo, a new package only from
an allow-list; authority = a signed os_pve_step or the root-owned ring-0 mark.
The night leg runs it in ring 0 after a healthy host step; ring 1 only by a
signed job (PVEStepExecutor). While it runs, the agent's own /etc/pve writes
(every non-GET API call, pct config verbs, pvesm, pveum, felhom-pbs-apply)
wait on internal/pvegate. Health = the host rule + unchanged container ids +
pveversion reads the installed pve-manager.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:38 +02:00
adminandClaude Opus 5.5 ac90169a5d R-861 (a) A1 + (b) B2: the image ref goes to a root verb that checks it; the agent's in-guest tee grant is gone; felhom-op's pct lines are exact (09 §3 decision 165)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:28 +02:00
adminandClaude Opus 5.5 154d6dcaa9 Shared rule file: no hub image build or deploy in a session the operator does not attend (09 §3 decision 162)
gates / gates (push) Successful in 56s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 09:55:02 +02:00
adminandClaude Opus 5.5 2f7072050f REPORT: released and delivered (2026-10-07)
gates / gates (push) Successful in 1m4s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 09:31:58 +02:00
adminandClaude Opus 5.5 85e799f360 CHANGELOG: v0.150.0 released (R-528, R-894, R-330)
gates / gates (push) Successful in 46s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 08:59:08 +02:00
adminandClaude Opus 5.5 3a72a4811b Shared rule file: rule 11 — every helper prompt carries the brief's fences in full (09 §3 decision 160)
gates / gates (push) Successful in 54s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 08:52:36 +02:00
adminandClaude Opus 5.5 a165d53c88 REPORT: the second burn-down night
gates / gates (push) Successful in 49s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 00:14:50 +02:00
adminandClaude Opus 5.5 adaf86ad57 R-330: the agent sends SMART 187/188/199 (raw; omitted when unknown)
gates / gates (push) Successful in 53s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 22:13:04 +02:00
adminandClaude Opus 5.5 74b5eae5b0 R-894: after a restart the agent remembers the last backup per tier
gates / gates (push) Successful in 35s
An unreadable storage right after an agent restart read the off-site tier
DUE (the in-memory record was empty). The newest success per tier is now
kept on disk and read ONLY when the storage cannot be read: fresh -> not
due, older than the cadence -> due, none -> due (unknown) as before. A
storage that answers stays the ground truth.

Ships with v0.150.0 after the 2026-10-07 read-back; nothing delivered.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 20:33:27 +02:00
adminandClaude Opus 5.5 7e82f325b8 CHANGELOG: the memory-kill check (unreleased, to ship as v0.150.0 after the night read-back)
gates / gates (push) Successful in 50s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 19:36:09 +02:00
adminandClaude Opus 5.5 acccb66bd3 R-528 (09 decision 157): after a Docker engine step the wrapper proves the engine reports a memory kill
felhom-os-apply: a docker-layer apply runs oom_check() after health_after and reports
"oom_check": {result pass|fail|error, oom_killed, oom_event, exit_code, image, detail}.
One throwaway container (the controller's image, --pull never, --network none, 64m cap,
label felhom.oomcheck=1) runs dd bs=200M; pass only with OOMKilled=true AND the oom event
(read after a 2 s settle, --until = guest epoch + 1: measured on demo-hp, an --until taken
right after the run missed the event). docker rm -f always runs in a finally; every call
is bounded (<= 90 s). It never changes the step's outcome or health. New wrapper-only mode
"oom-check" (docker layer) runs the check alone; check_guest etc. still apply.

Agent: WrapperReport/Report gain OOMCheck (json:"oom_check"), copied unchanged in runLayer
and in the kept-copy path.

Tests: 9 wrapper tests + 2 Go tests, each red-proofed (audits/readback-2026-10-07/F/red-*.txt).
Also: test_felhom_os_apply.py's `if __name__` sat mid-file, so 11 tests (UnsentReport,
SaveReportOnDisk, AgentDiesMidPass, CrashLeftTheJournal) never ran as a script or from
TestWrapperSuite; moved to the end (they pass).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 19:34:55 +02:00
adminandClaude Opus 5.5 de812bc027 The shared rule file (09 §3 decision 152), identical to the other copies; no code change
gates / gates (push) Successful in 44s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 16:06:46 +02:00
adminandClaude Opus 5.5 3e8ebeb96c Instruction files kept true (09 §3 decision 150): stale gate lists, paths and facts corrected; no code change
gates / gates (push) Successful in 46s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 13:43:57 +02:00
adminandClaude Opus 5.5 cefdc731a4 REPORT: the operator's ten answers (2026-10-06)
gates / gates (push) Successful in 24s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 12:28:58 +02:00
adminandClaude Opus 5.5 e56dcb8a4c CHANGELOG/v0.149.0 released (shas)
gates / gates (push) Successful in 37s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:50:21 +02:00
adminandClaude Opus 5.5 f277e619e2 CHANGELOG: unreleased — R-856 GET /host/crash-guard
gates / gates (push) Successful in 38s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:48:40 +02:00
adminandClaude Opus 5.5 386f51edc6 R-856: GET /host/crash-guard — the host crash guard's last-boot record for the controller (09 decision 143)
The controller waits ~15 minutes with app mails after a crash boot of the host; it learns of the
crash boot from this route. Reads /var/lib/felhom-crash-guard/state.json (read-only, no Proxmox call)
and passes present/last_boot_at/last_boot_unclean/tripped through; a missing, unreadable or garbled
file answers 200 present:false. Guest-token authed like every sibling route.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:48:01 +02:00
adminandClaude Opus 5.5 130e3ed882 CHANGELOG: unreleased — R-444 weekly guest disk trim, R-99 runbook pointer
gates / gates (push) Successful in 41s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:25:22 +02:00
adminandClaude Opus 5.5 be398f92e8 R-99: the PBS phantom WARN names the cleanup runbook (09 §3 decision 140)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:24:45 +02:00
adminandClaude Opus 5.5 ee71abd1d4 R-444: weekly guest disk trim (pct fstrim) outside the night, under the heavy-op gate
Operator ruling 09 §3 decision 139. One exact sudoers rule FELHOM_FSTRIM
(`/usr/sbin/pct ^fstrim [0-9]+$`) + manifest entry guest-fstrim; new
internal/fstrim job: due Wednesday from 10:00 host-local, starts only
10:00-20:59, holds backup.InFlight (busy -> deferred to the next hourly
tick), failed trim retried at most 3x per week, bytes parsed from the
"(N bytes) trimmed" lines, last result per guest persisted in
<state_dir>/guest-disk-trim.json and reported as guest_disk_trim.
Config opt-out: "disk_trim": {"disable": true}.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:24:45 +02:00
adminandClaude Opus 5.5 37e98f452b REPORT: the burn-down night (2026-10-06)
gates / gates (push) Successful in 45s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 02:13:10 +02:00
adminandClaude Opus 5.5 b2b82ae828 bundle test: the ISO first-boot files the installer names under KEPT are not installer-written (go test red since felhom.eu 85de3f9b); CHANGELOG unreleased (R-426 decoys)
gates / gates (push) Successful in 42s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 01:42:25 +02:00
adminandClaude Opus 5.5 b78a0ff3ac R-426: decoys for the shared reuse-refs, instructions and observations gates
COVERS "reuse-refs", "instructions", "observations": the three shared
felhom.eu scripts run against a scratch clone of THIS repo (in a scratch
workspace symlinking the sibling clones they reach across to), so the
plant is in the agent's own REUSE.md / CLAUDE.md / REPORT.md. Convicted:
a missing cited .go and .md path, a version literal in CLAUDE.md's
effective text, R-419's prose-only Observations note. Passed: the real
files, the version inside an HTML comment, both genuine markers.
DECOY_SHARED_DIR lets a red-proof judge a mutated copy of the shared
scripts without editing the felhom.eu clone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 01:42:03 +02:00
adminandClaude Opus 5.5 96047453cb R-426: decoys for the release-complete gate
COVERS "release-complete": the working-tree gate runs in a scratch clone
whose origin is a scratch bare repo, against the fake Gitea. Convicted:
the newest `## vX.Y.Z` with no tag anywhere, a tag parked on an unrelated
commit, a tag with no package, and no-tag wins over a registry 500.
Inconclusive: a registry 500. Passed: the genuine release, an
`## Unreleased` heading above it, a newer version named only in prose or
under `###` (the withdrawn sweep decoy, now asserted the right way round),
a tag only origin has (the shallow-CI shape), and a LOCAL-only tag BY
DESIGN (CI's fresh clone and the published gate's converse probe see it).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 01:42:03 +02:00
adminandClaude Opus 5.5 4bf5db6875 R-426: decoy suite for the published gate, against a fake Gitea
scripts/test_gate_decoys.py (new; COVERS "published"): an http.server on
127.0.0.1 stands in for Gitea through the gate's existing GITEA_BASE
seam, proxies stripped, so no case reaches the real registry. Facts
convicted: a tag whose package 404s, a tag tree without the configs, a
package one patch past the newest tag (never tagged), a patch-gap orphan,
a missing package that lexical sorting would drop out of the retention
window. Inconclusive, never a pass: tags api 500, a non-JSON 200, Gitea
unreachable. Passed: a clean registry, a non-semver tag, a version older
than the retention window (BY DESIGN).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 01:42:03 +02:00
adminandClaude Opus 5.5 87977ff40a CHANGELOG/v0.148.0 released (shas)
gates / gates (push) Successful in 42s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 00:37:07 +02:00
adminandClaude Opus 5.5 861d32a4b4 CHANGELOG: unreleased — R-349 agent_sha256, R-25 agent half (burn-down night)
gates / gates (push) Successful in 43s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 21:16:51 +02:00
adminandClaude Opus 5.5 769c4c3cf2 R-25 (agent half): the format answer carries the NEW filesystem's UUID, bound to the durable id
After mkfs the agent re-resolves the bound durable id, requires it to name the
device it just formatted, reads the superblock back (blkid -p, requested fstype)
and returns fs_uuid in POST /disks/format and GET /disks/format/status. Anything
unverified returns "" — never a path-resolved guess. The controller half
(mount fs_uuid instead of re-resolving the UUID from the /dev path) is owed
in felhom-controller.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 21:16:11 +02:00
adminandClaude Opus 5.5 64f704d0f7 R-349: the host report carries the sha256 of the RUNNING agent binary
A hand-built proof binary and the published artifact share a version string
but not their bytes, so no version check could see the divergence. The agent
now reports agent_sha256 (hash of /proc/self/exe, once per process; empty =
unknown) beside agent_version, the same mechanism as host.wrapper_sha256.
The hub half (compare against the vouched agent_sha256, surface drift) is
owed in the hub repo.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 21:16:11 +02:00
adminandClaude Opus 5.5 208fac8027 CHANGELOG/REPORT: v0.147.0 released (shas)
gates / gates (push) Successful in 56s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 18:58:37 +02:00
adminandClaude Opus 5.5 f1b9b41214 R-124 recipe root namespace as PBS spells it; R-118 no root size for an absent drive; R-269 rotated-out token rejected at once; R-317 dnsmasq install probed by its unit (burn-down round 2)
gates / gates (push) Successful in 47s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 18:56:13 +02:00
adminandClaude Opus 5.5 d83316326e R-291 retention record source, R-348 restart comment (no binary change; burn-down)
gates / gates (push) Successful in 22s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 16:56:18 +02:00
adminandClaude Opus 5.5 e06ed97fa8 agent v0.146.1 REPORT
gates / gates (push) Successful in 22s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 13:00:22 +02:00
adminandClaude Opus 5.5 e4b5cf9693 R-880: build-step-bundle.py — the transition bundle for a release whose bundle adds paths (an installed felhom-os-apply refuses unknown paths, R16)
gates / gates (push) Successful in 21s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:23:55 +02:00
adminandClaude Opus 5.5 faa3cad92e agent v0.146.1 CHANGELOG (released)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:14:07 +02:00
243 changed files with 28325 additions and 621 deletions

No files matched your search

+3 -2
View File
@@ -21,6 +21,7 @@ fast, and wrong.
This rule used to be duplicated verbatim in felhom-agent/CLAUDE.md with a note explaining that
felhom.eu/CLAUDE.md "does not load in an agent-only session". That reasoning was correct before
path-scoped rules existed. The single source is now felhom.eu/CLAUDE.md "Code quality rules"; this
file is the scoped copy that loads exactly where health checks are written. (2026-08-06)
path-scoped rules existed. Deliberate scoped copies now live in felhom.eu/.claude/rules/hub.md and
felhom-controller/.claude/rules/gates.md (hub.md's comment names them); none is the single source. This
file is the copy that loads exactly where agent health checks are written. (2026-08-06; corrected 2026-10-06)
-->
+101
View File
@@ -0,0 +1,101 @@
---
unconditional: true
---
# Unprompted work — rules for any session without a task file
> Goal sessions, nightly sessions, "work the register" sessions. **A session that starts from
> `/goal` or a standing brief inherits these rules exactly as it inherits the gates.** They are the
> part of `PROMPT-TEMPLATE.md` that a task file used to carry and a goal does not. Same wording lives
> in `felhom.eu`, `felhom-controller`, `felhom-agent` and `app-catalog-felhom.eu` `.claude/rules/`, and in the workspace
> root's unversioned `.claude/rules/`; change all five or none.
## 1. What you may pick up on your own
- A register row **you or another CC session filed**, with owner CC, at P3 or a bounded P2, that
needs **no operator decision**, touches **no customer data by design**, and introduces **no
mechanism nobody has measured**. Smallest first.
- A defect you find while exercising the product, filed as a row **before** you fix it — **unless it is small**:
a small finding is fixed in the session and never filed (the size rule, `OPEN-ITEMS.md` „How a row is filed").
- Hygiene: register compression, stale citations, rows with no owner, documents that contradict
live source.
**Not yours, ever, without a task file or an operator word:** money; anything that changes risk to
customer data; anything that changes a promise the product makes to a customer; anything that
reverses a documented design decision (`documentation/architecture/` — a design decision is not a
defect, R-370); anything on DooPlex or ep0; baking or vouching a golden; promoting a
catalog version; a new external dependency; **a hub image build or hub deploy in a session the operator does not
attend** (operator ruling 2026-10-07, `09` §3 decision 162).
## 2. When you may decide instead of ask (operator grant, 2026-09-14)
You may take a decision yourself when **all** of these hold: the architecture folder and the register
give a clear direction; your choice follows that direction; it is reversible without customer-data
risk; and you can write it in the `09-update-architecture.md` §3 shape — one answerable sentence, the
options, what each costs, why this one. **Then record it** as a dated decision in `CONTEXT.md` and
the owning architecture document, tagged *decided by CC unattended — operator may reverse*, and put
it **first** in the morning note. A decision you cannot write in that shape is one you do not take.
## 3. The discipline a task file used to carry
1. **Baselines first.** Read each repo's `main` hash and version from live source before touching it.
2. **Read the architecture document for the area, and name it** in the report, before any claim.
3. **Red-proof every correctness fix.** A test never seen failing has not been shown to test anything.
4. **Live-validate on a Tier-0 box** through the endpoints the UI invokes. `demo-hp` is `ssh hp`.
Throwaway apps only; the standing apps and `bentopdf` stay.
5. **Evidence off the machine at the end of each phase**, before any revert (R-320).
6. **One release per repo per session**, with a CHANGELOG entry (controller: with its `MinAgent`
line) and REPORT overwritten. **A controller release needs no golden** (operator ruling 2026-10-15
11:26, `09` §3 decision 250; decision 7 stands): push the image only when it is ready for EVERY box (the
household's update button offers the highest tag); raise the floor, MinAgent declared, only after it ran
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
on all three — not at night unless the night brief says so. **Agent releases still stop for the
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
**A version number is pushed once** — `felhom-controller/controller/build.sh` now refuses a second
push of a tag that exists, and a registry it cannot reach is a refusal too. This is a FACT about
what the build does, not a new rule: a fix for something already pushed is the next patch version,
and a box on a bad image is moved by the normal update path, never by re-tagging. Earned on
2026-10-15, when 0.315.0 was pushed three times from three different commits and a cache kept
serving the first one (`09-update-architecture.md` §0).
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
Prose is not a record.
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
9. **Never leave a half-state.** If time runs out, revert to clean and say what was reverted.
10. **Teardown, three layers, stated** — machine, host, hub — or "provisioned nothing".
11. **Every helper prompt carries the brief's fences in full** (operator ruling 2026-10-07). A helper session (a
subagent, a fork, a workflow agent) gets the brief's fence list word for word — every protected machine, every
„no", every delivery and Docker limit — not a summary and not „the usual fences". Earned on 2026-10-06 night: two
helpers whose prompts carried only part of the fences ran `docker volume prune` on the bench and a Docker-using
gate on DooPlex.
## 4. The morning note
One screen, plain language, in this order: **decisions you took** (§2) first; what you exercised;
what broke and whether you fixed it; rows opened and closed with the register size before and after;
what needs the operator, each with what happens if they do nothing. No file paths, no function
names, no row numbers as the subject of a sentence.
## 5. Instruction files
**Instruction files (`CLAUDE.md`, `.claude/rules/*`) are kept true by the session that finds them wrong**
(operator ruling 2026-10-06, `09` §3 decision 150). A session MAY, without asking: correct a stale fact (a command, a
count, a version, a path, a description of what a gate does), add a fact it proved, and remove a reference to something
that no longer exists. Each edit is named in the report (file, line, before, after, why). A session MAY NOT, without the
operator's word: loosen a safety rule, a fence, a „never", a protected machine, a secret rule, or a review step; or
remove a rule. When in doubt, it is a rule change, and it goes to the operator. If Claude Code's own permission check
asks before such an edit, wait for the operator's click; if it refuses, record that and file the exact line.
## 6. The system poster stays true
**A session that changes a fact listed in `architecture/felhom-system-poster.facts.md` (a machine, a
role, a traffic path, a backup tier, a time, a retention, a key, a known gap) updates that file in
the same commit.** If the change is text only, it also edits the matching text in
`felhom-system-poster.html`. If it needs a new drawing, it adds the line
**"System poster needs a refresh: <what changed>"** to `STATUS.md`'s "waiting on the operator" list.
**The session report names which of the three it did.**
Why the three-way split: the poster is a drawing made in Claude Design, and **nothing in this
repository renders it** — unlike `where-felhom-stands.html`, which has `render_stands.py`. So the
facts file is the source of truth and the drawing trails it. `scripts/poster_facts_gate.py` warns
when the facts file has a newer commit than the poster; it **never fails a push**, because a refresh
needs the operator and another tool, and a gate nobody can clear is a gate people learn to route
around.
+44 -4
View File
@@ -16,7 +16,16 @@
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
# layer up. That is the last step, and it runs ONLY on failure.
name: gates
on: [push]
# R-1129: `workflow_dispatch` is the manual PROOF that the failure mail can be sent. A push run is
# unchanged by it: the proof step below is skipped on a push, and `force_alarm` is then empty.
on:
push:
workflow_dispatch:
inputs:
force_alarm:
description: "Fail this run on purpose so the alarm step sends its real mail (a proof, not a gate result)"
type: boolean
default: false
jobs:
gates:
@@ -55,8 +64,27 @@ jobs:
env:
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
FELHOM_SCAN_FETCH: "1"
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
# Runs ONLY on a manual dispatch, never on a push. It prints the key's LENGTH, never the key,
# so "CI can see the secret" is an observable rather than an assumption. With
# force_alarm=true it then fails on purpose so the alarm step below sends its real mail;
# the commit itself stays green, because its push run skips this step.
if: github.event_name == 'workflow_dispatch'
env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
FORCE_ALARM: ${{ github.event.inputs.force_alarm }}
run: |
python3 -c 'import os;print("RESEND_API_KEY length: %d" % len(os.environ.get("RESEND_API_KEY","")))'
if [ "$FORCE_ALARM" = "true" ]; then
echo "force_alarm=true - failing on purpose so the alarm step runs (R-1129 proof)"
exit 1
fi
- name: Alarm on failure
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
@@ -70,6 +98,8 @@ jobs:
if: failure()
env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
# Empty on a push, so a push run's mail is exactly what it was before R-1129.
FORCE_ALARM: ${{ github.event.inputs.force_alarm }}
run: |
python3 - <<'PY'
import json, os, sys, urllib.request, urllib.error
@@ -81,13 +111,23 @@ jobs:
repo = os.environ.get("GITHUB_REPOSITORY", "?")
sha = os.environ.get("GITHUB_SHA", "?")
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
# R-1135: the link is for a PERSON, so it is the public address and the run ID.
# NOT GITHUB_SERVER_URL: the runner registers at the in-cluster address, so that variable
# reads http://gitea.gitea-system.svc.cluster.local:3000, which nobody outside the cluster
# can open (forced-test mail, job 1997). NOT GITHUB_RUN_NUMBER: Gitea 1.26 reads
# /actions/runs/<n> as a run id first, so a run number that is also an older run's id
# opens THAT run (measured 2026-10-18: felhom.eu /actions/runs/1116 opened the run whose
# id is 1116 = run number 613). The link is printed too, so the job log carries it.
run = os.environ.get("GITHUB_RUN_ID", "?")
srv = "https://gitea.dooplex.hu"
print("ALARM-LINK %s/%s/actions/runs/%s" % (srv, repo, run))
body = json.dumps({
"from": "Felhom CI <monitoring@felhom.eu>",
"to": ["admin@felhom.eu"],
"subject": "[felhom CI] gates FAILED in %s" % repo,
"subject": ("[felhom CI] FORCED TEST, not a real failure - gates FAILED in %s"
if os.environ.get("FORCE_ALARM") == "true"
else "[felhom CI] gates FAILED in %s") % repo,
"text": (
"The gate entry point exited non-zero.\n\n"
"Repository : %s\n"
+18
View File
@@ -0,0 +1,18 @@
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
gitleaks generic-api-key@1ad7cc8eb57e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46ce5a3eccc4 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46fe126846b0 internal/escrow/identity_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@56474f1754f5 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@631c53997f7e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@6e790ef41ef3 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@78334b854242 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@7b88d323a44f internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b0eeee93cf24 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b1cb2659967a internal/localapi/escrow_stage_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b66bec43cc8a internal/escrow/recover_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e0525160fe03 internal/pbsdr/wrapper_read_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e442e455c360 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f23f93f461b2 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f471b78e0f6e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
+1295
View File
File diff suppressed because it is too large. Load diff
+8 -7
View File
@@ -52,11 +52,12 @@ This is in the core because breaching it is how this component stops being audit
## Gates — ONE entry point
**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs this
repo's gates — `reuse_refs_check` and `instructions_gate`, both the **shared** copies in
`felhom.eu/scripts/`, never copied into this repo (a copy recreates the drift they detect; an absent
sibling clone FAILS). `--fast` selects the gates touching no network and no container runtime; today
that is all of them. **A missing gate is a FAILURE, never a skip.**
**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs every
gate in its `GATES` table (that table is the list); the shared ones — `reuse_refs_check`,
`instructions_gate`, `observations_gate` — are the copies in `felhom.eu/scripts/`, never copied into
this repo (a copy recreates the drift they detect; an absent sibling clone FAILS). `--fast` selects the
gates touching no network and no container runtime, and skips `published` (network) and `gate-decoys` (the decoy
suite, ~20 s; R-1116), naming them — CI runs the full set. **A missing gate is a FAILURE, never a skip.**
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
**per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
@@ -104,8 +105,8 @@ the mechanism are exempt.
**A gate ships with a decoy test that has been seen to fail (R-421).** A decoy is the LABEL without
the FACT — a directory with the right name and no bake log, a note whose prose mentions the marker it
lacks. `scripts/decoy_coverage_gate.py` refuses a new gate that has neither a decoy nor a named
lacks. `felhom.eu/scripts/decoy_coverage_gate.py` (run by felhom.eu's `repo_gates.py`, for all four repos) refuses a new gate that has neither a decoy nor a named
exemption carrying its row. The four shapes, the 2026-09-01 sweep that fooled 16 of 29 gates, and the
decoys withdrawn as illegitimate: `documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and
decoys withdrawn as illegitimate: `felhom.eu/documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and
`felhom-controller/.claude/rules/gates.md`. **Scope is a fact too** — prefer `os.walk` over
`os.listdir`, and a glob over a hand-maintained list.
+118
View File
@@ -1,5 +1,123 @@
# CONTEXT — felhom-agent working state
> **2026-11-14 (day, attended) — v0.174.0 RELEASED and VOUCHED (15:21:45 CET); on the three test boxes (agent + bundle, 77/77).** sha `6efbb978…`, bundle `7d87d058…`. Two new wrapper modes: `kernel-cleanup` (layer kernel; old kernel images by the hand-proven rule; ring-0 mark only; the plan names nothing; the night leg's last call, never with a kernel step) and `profile-refresh` (layer host; AppArmor profiles from a fresh compile once per kernel; one restart for a failed service whose cached profile was stale). Go 1.27.4. The N100's time service was repaired by the product path (+32.7 s step). A real kernel removal ran on Tester 1 by hand with a temporary ring-0 mark (operator's yes). **Next:** read the N100's and demo-hp's night (the clean-up by the agent at night is not proven yet); a ruling for ring-1 boxes; release 2b (trust in Proxmox by the root CA copy, D80). Report: felhom.eu `REPORT-release-2a-2026-11-14.md`.
> **2026-11-10 (afternoon, attended) — v0.173.0 RELEASED and VOUCHED (16:29:16 CET); on the three test boxes.** sha `752b2727…`, bundle `51ebc1f4…` (`configs/` unchanged since 0.172.1, so no bundle job). Measured: a box cleans its self-update folder at the commit of the update that BRINGS the code (49 → 1, 48 → 1, 33 → 1 entries; about 16 MB stays). The line below („at the first good update AFTER that release") was wrong. A rollback after the clean-up was proven on Tester 1 with a signed update to 0.172.1 and one crash in the dwell. `pinned.day0_agent` = 0.173.0. Open: the `.bak-*` files (R-1368 part 4).
> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: the self-update folder cleans itself (R-1368 part 2).** After a healthy `commit` the agent removes every staged binary but the running version's (`internal/selfupdate/prune.go`; tests `TestPrune_*`, the wrapper's real `rollback` verb run in a sandbox). No version bump, no build, no vouch: it rides the next agent release. A box cleans its folder at the first good update AFTER that release. Not reached: `/usr/local/bin/felhom-agent.bak-*` (root-owned, R-1368 part 4). The old-kernel clean-up is a design only (felhom.eu `11-os-updates.md` §5.11.1, proven by hand on demo-hp); nothing of it is in this repo.
> **2026-11-07 (attended) — v0.172.1 RELEASED and VOUCHED; on the three test boxes (agent + bundle, 77/77).** `proxmox-grub` joins `HOST_SLOW_RE` (the N100's ring-0 pve step was refused R6 three nights: proxmox-grub pmx4 pulls grub-efi-amd64). sha `505a0594…`, bundle `3e7acaee…` (only `felhom-os-apply` changed vs 0.170.0, no step bundle). Done: vouch, both signed jobs on the three boxes, `pinned.day0_agent` = 0.172.1. Next: the N100's pve step on its next night (positive line: `os-apply: DONE` for `layer=pve:9201`, no R6).
> **2026-11-03 (day, attended) — v0.172.0 RELEASED and VOUCHED (10:18:10 CET), R-1302.** `2c0222cd…`, bundle `cc34b0bc…`, `pinned.day0_agent` = 0.172.0; on the three test boxes by signed `agent_update`. The appliance path pre-positions the grant, so a plain first start logs neither line; staged live both ways (felhom.eu `audits/release-2026-11-03/part3/`).
>
> **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
> but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
> alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
> refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
> (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
> carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
> `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
> updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
> `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
> **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
> `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
> sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
> (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
> `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
> (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
> **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
> `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
> counts operator restarts, then `opBrakeUntil` refuses presses 30 min). R-1344: 6 unhealthy sweeps → `unresponsive_since`
> → the hub's `controller_unresponsive` — ALARM ONLY until the night's health log is read (to 2026-10-30 07:00 UTC).
> Proven live on Tester 1: stopped controller restarted, 4th press refused, one press on the release builds „done" in 16 s.
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
> R-528 proven on the release (Tester 1: one `app_oom` „seen by docker,cgroup", host counter 1). R-1254: the happy path ran
> live (a ceremony on 0.168.0 + controller 0.347.0); the lost-answer branch has tests only.
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
> on demo-hp (felhom.eu R-1250, read, not forced).
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
> put back. Release + the operator's vouch: Monday's list (felhom.eu STATUS).
> **2026-10-24 (attended release run) — v0.164.0: the weekly upload line test (R-1247; operator ruling 07:26).**
> `internal/linetest`: the agent's own iperf3 client (operator's pick — nothing installed on hosts) to ep0's tunnel-only
> server `10.77.0.1:5201`, Tuesday 10:00-15:59, after 10 quiet minutes, under the heavy-op gate, best of 3; `line_test`
> stanza + `GET /host/line-test` (the controller's qBittorrent limit, D41 = 50 %). demo-hp measured 90.7 Mbit/s. Release
> needs the operator's vouch.
> **2026-10-22/23 night — D30 on `main` (`ae69ad4`), unreleased; the D4 patches v2 ready.** Operator ruling D30
> (2026-10-22): a handshake stamped in the future for 15 min → a new WG key pair (`wgtunnel/rekey.go`; old key kept as
> `wg/private.key.prev`), registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the
> key on ep0). Live on a test box (felhom.eu `audits/night-2026-10-23/P2-D30/`): with ep0 free to start a handshake the
> tunnel heals by itself (no re-key, correct); with ep0's starts dropped it stays dead and the agent re-keyed at 15 min.
> The interface switches at the next desired-state fetch. Gap: a reboot forgets the future stamp (R-1223). The D4 bundle
> patch v2 (`P3-review/felhom-agent-bundle-D4-v2.patch`): `try-reload-or-restart`, an `sshd -T` read-back reported as
> `sshd_dropin_not_in_force`, a reload in undo. Next agent release = D30 + D4 v2 + R-1204's field, the operator vouches.
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
> `ff4e6dc5…99d3b5`, bundle `485f36c9…12c47`. R-1083 left out on purpose: an agent key for 9202 would make its
> controller book whole-guest backups through the agent, one tier writing to ep0 — the R-1083 row says what is needed.
> **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978,
> R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with
> `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn
> positions; only those two words are held, 30-min window; a claim without it keeps the R-978 order). Vouch: version
> 0.160.0, sha256 `7ebb547f…dfca1`, bundle `6b060901…e202f`. After it, unreleased: `4545819` (two texts). felhom.eu
> `REPORT-day-2026-10-19.md`.
> **2026-10-18 (night) — UNRELEASED on main: R-1093, R-978, R-973** (release + vouch: the operator's). R-1093 an empty
> partition is blank (lsblk's PTTYPE on a partition is the parent's; set aside only on a clean blkid exit 0 with a
> matching `PART_ENTRY_SCHEME`; an unparsable lsblk is unprobed). R-978 the escrow package reaches the hub only at the
> claim (ceremony output v2, `PendingUpload`; `EscrowCeremonyConfig.Upload`). R-973 the `POST /backup` cadence cap.
> **Decision (decided by CC unattended — operator may reverse): the cap's numbers.** Q: how many whole-guest runs may
> the guest ask for? Options: (a) 2 per 24 h on the daily tier — keeps one local point older than a day, but refuses
> the NIGHT after a press (R-899's night can fall inside the same 24 h); (b) 3 per 24 h + 1 per 6 days on the weekly
> tier — the schedule never refused, a second press in a day refused, the local tier still replaceable in a day;
> (c) none — status quo. Chose (b): the off-site tier is the one `07` §8 row 10 relies on and it is fully protected
> for ~6 days; the local half's real fix is a time-bucketed prune (keep-daily), which is a retention change for the
> operator. No sudoers or bundle change in any of the three.
> **2026-10-17 — v0.159.1 RELEASED (0.159.0 superseded, do not vouch).** Go 1.27.2; R-1124 a decommissioned drive that is
> gone is forgotten (`ForgetRetiredDrives`, root verb `unit-forget`); R-1012 the read-only `felhom-thin-check` names an
> orphan thin device in a failed backup's text; the signed decommission prunes the guest-bind record. Bundle adds a path →
> step bundle `0.159.0-step1`. Proven on demo-hp; delivered by signed jobs on the operator's yes. Vouch: the operator's.
> **2026-10-16 — v0.158.1 RELEASED and VOUCHED (storage identity build B; 0.158.0 superseded, do not vouch).** The internal
> data place (root helper `felhom-internal-place`, the record in the state dir, mounted at /mnt/belso like a drive), the
> signed `add_internal_place`, partitions in `/disks/candidates` (`formattable` only while nothing on the disk is mounted),
> the partition wipe id `byid:<…-partN>+fsuuid:<uuid>`, the strict mkfs wrapper. Delivered through step bundle
> `0.158.1-step1` (a bundle that adds a path, R16). Next: R-1093 (an empty part's probe reads the parent's partition table).
> **2026-10-15 (night) — v0.157.0 RELEASED, NOT VOUCHED (storage identity build A, felhom.eu `16` §13).** Every storage
> place carries `kind`, `physical_disks`, `identity` (+ reason) in `GET /disks` and the hub report; `POST /disks/same-disk`
> answers the controller (v0.317.0, MinAgent 0.157.0). Computed on every read, never cached. Delivered by signed jobs to
> Tester 1, demo-hp, N100 only. Next: the operator vouches 0.157.0 → the global floor 0.317.0 / MinAgent 0.157.0. Build B
> (partition candidates, the internal place) waits for a task file (R-1079).
> **2026-10-08 (day) — UNRELEASED on main, ships with tomorrow's release (decision 178).** R-899: `POST /backup?trigger=manual`
> runs no OS leg (`localapi/server.go`); after-boot kernel reports carry the saved block's ring (`kernelReportRing`). R-304:
> `escrow.ErrRetainedUnchecked` → HTTP 424 `older_unchecked` when not every earlier package was tried (withheld, caps,
> malformed, unreadable list). Binary-only delivery; no wrapper or root file changed.
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
+8 -7
View File
@@ -44,13 +44,14 @@ unnoticed until a user hit them. `internal/capability` makes that loud:
(`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
— the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin` →
`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.
**Controller-swap under non-root (v0.45.0; the write since R-861 (a) A1).** The agent-owned controller image swap
(`internal/localapi/controllerswap.go`) no longer shells out. The write goes on **stdin** to the ROOT verb
`felhom-priv-apply controller-image <vmid>` (`GuestBinder.WriteControllerImage` → `Runner.RunStdin`, the same fenced
`sudo -n` runner), which re-checks the ref against our registry + repository + an x.y.z tag and writes
`/etc/felhom-controller-image` inside the guest itself; the agent has no in-guest `tee` grant any more (before, a
compromised agent could feed any image — sudo cannot see stdin). Its grants live in the `FELHOM_CONTROLLERSWAP` sudoers
alias (read-only or fixed-target) and in the capability manifest (Critical), so a dropped grant is a build failure + a
live degraded signal. No general `pct exec` is granted.
## The `storage` package — observe + watchdog (slice 5)
+28 -11
View File
@@ -1,14 +1,31 @@
# REPORT — agent v0.145.0 (2026-10-05, afternoon): the OS update repairs itself after a power cut
# REPORT — v0.174.0 released, vouched and delivered to the three test boxes (2026-11-14, attended)
Brief: the 2026-10-05 catch-up brief (operator), Parts C (R-874, R-875) and D (R-876). Full session report:
`felhom.eu/REPORT-catchup-2026-10-05.md`. Architecture: `11-os-updates.md` §5.4.1, §8.4–8.5; `09` decisions 117–118.
**Plain words first.** A box now removes its old kernels by itself, and it reloads its security profiles after it
starts on another kernel. Version 0.174.0 is released and runs on Tester 1, demo-hp and the N100. The operator vouched
it at 15:21 CET. The N100's time service runs again; the product repaired it, with no hand step.
| Row | Fix | Proof |
|---|---|---|
| R-876 | the wrapper reads `dpkg --audit` AND the update journal in ONE call, repairs on either; belt: repair + retry once when apt says "interrupted" | 3 tests + 3 red-proofs; **live (operator's go): crash mid-unpack on demo-hp → the next pass `REPAIR … journal=1` → `DONE rc=0 upgraded=12`, nobody touched the box** |
| R-874 | the restore-test's first due-check 30 min after start | 2 tests + red-proof; live on demo-felhom: passed restore-test at start + 30 min |
| R-875 | a kept report's reason is neutral ("sent late …") | test + red-proof |
The full report of the run is in `felhom.eu`: `REPORT-release-2a-2026-11-14.md`.
Released by `scripts/release-agent.sh`: v0.145.0 (`894da35c…`, bundle `78c00adc…`), verified by download; signed
`agent_update` + `agent_config_update` to demo-hp, demo-felhom, tester-1 (71/71 after each bundle); vouched with
golden 0.295.0, min_agent 0.131.0. `go test ./...` rc 0, Python suites OK, `agent_gates.py` OK.
## What a reader must know
1. **The clean-up runs on a box with the root-owned ring-0 mark only.** Tester 1 has no mark, so the program answers
„not run" there. For the proof the operator allowed the mark on Tester 1 for some minutes; it is back as before.
2. **The real removal on Tester 1 was started by hand**, with the agent's own command line. The night leg cannot be
started by day: it follows only the night's own backup (R-899). So „the step by the agent at night" is **not proven
yet**; the N100's and demo-hp's night of 2026-11-15 is the proof.
3. **The profile refresh was spiked first** on demo-hp (felhom.eu `audits/profile-refresh-spike-2026-11-14/`). Three
things it found are built in: a stale copy is told by the cached file changing, not by the kernel's hash; one
profile never compiles the same twice; a refresh hides other kernel audit lines for a few seconds.
4. **A clock step makes the controller read a new boot** (felhom.eu R-1395, a controller row): on the N100 it
recreated three apps once.
| | |
|---|---|
| Version | 0.174.0, sha256 `6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92`, bundle `7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f`, Go 1.27.4 |
| Code commit, CI | `cc63dd5`, run id 2641 success |
| Tests | wrapper 191 (42 new), agent `internal/osupdate` green; 30 mutations, each seen red (felhom.eu `audits/release-2a-2026-11-14/A`, `B`) |
| Tester 1 | 3 kernels → 2, 1.12 GB gained, 8.5 s; second run: nothing; one restart, same kernel, healthy, clean stop |
| demo-hp | nothing to remove; refusals P1, P2, P3 by dry run |
| The N100 | time service repaired (offset −32.55 s → +0.18 s); tonight it removes 6 of 8 kernels |
| Vouch | the operator, 2026-11-14 15:21:45 CET; `pinned.day0_agent` = 0.174.0 |
| Not proven | the step at night by the agent; a fell-back box and a self-revert into the kept kernel on a box; K1 on a box |
+30 -9
View File
@@ -15,13 +15,14 @@
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key` | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| unit-forget <name> (v0.159.0, R-1124) \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key \| controller-image <vmid>` (the last reads the ref on stdin, R-861 (a) A1) | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
| `osupdate.Leg.SendUnsent` / `lockPass` (v0.144.0, R-868) | internal/osupdate/unsent.go | `(ctx) int` | an OS-pass report the agent never sent (killed mid-pass): the wrapper keeps `report-<run>-<layer>-apply.json` beside the plan; the agent deletes it once the hub has it | any new caller that runs an apply pass must hold `lockPass` (flock, across processes) — the sender must never take a running pass's copy |
| `osupdate.LoadSavedBlock` (v0.144.0, R-866) | internal/osupdate/leg.go | `(planDir) (block, savedAt, ok)` | the hub's newest os_update block as the daemon last received it (`os-update-block.json`) | the debug pass uses it ONLY when the hub cannot be reached, and says so in its header; no saved block → no pass |
| `dpkg_state()` / `DPKG_STATE_SCRIPT` (v0.145.0, R-876) | configs/felhom-os-apply | `audit, journal = self.dpkg_state()` | dpkg's state in ONE call: `--audit` AND the update journal | never gate a repair on `--audit` alone — a crash leaves only the journal (measured); keep it one call (R-845) |
| `thincheck.Checker.Check` + `Describe` (v0.159.0, R-1012) | internal/thincheck/thincheck.go | `Check(ctx, vmid) []Finding` | name an ORPHAN thin device (pool holds it, LVM does not) in a whole-guest backup failure — `BackupRunner.SetThinCheck` | READ-ONLY by design: the root helper `felhom-thin-check <vg> <pool>` sends only reserve/release_metadata_snap; an orphan needs two reads; fails open (never stops a backup). The repair stays human |
| `guesthook.SnippetReady` / `Register` (v0.146.0, R-861) | internal/guesthook/install.go | `SnippetReady(path) error` | is the pre-start hook (a FIXED file from the bundle) in place — register only then | The agent never installs the hook (Proxmox runs it as root); a missing hookscript stops a guest start, so never `Register` without `SnippetReady` |
### Disk / format safety (role gates, durable IDs, format guards)
@@ -34,13 +35,18 @@
| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()`. **The staging folder cleans itself** (`prune.go`, R-1368): after a healthy `commit` only the running version's staged binary stays; a new file kept in `<state dir>/selfupdate/` must NOT be named `felhom-agent-<version>` or the prune removes it |
| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
| `storage.ResolveStorageDevice` | internal/storage/durable_device.go | `ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)` | re-resolve enrolled STORAGE drives (remount) | `uuid:` scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node |
| `deriveDurableID` | internal/storage/durableid.go | `deriveDurableID(typ, s, backingDevice, uuid) string` | storage-target durable id (DR re-attach key) | Deterministic per type; `uuid:` for usb/local-dir; PBS id includes `#<fingerprint>` |
| `SystemDisks` / `isSystemBacked` / `RoleForStorage` / `RoleForRawDevice` | internal/storage/role.go | `RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole` | protection-tier classification | Fails safe to `system` (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied |
| `storage.IdentityResolver` / `Relation` / `MountFor` | internal/storage/identity.go | `HostPath(path) PlaceIdentity`, `Device(dev) PlaceIdentity`, `Relation(a, b) (different, proven, reason)` | physical-disk identity of a storage place (felhom.eu `16-storage-physical-identity.md` §3, v0.157.0) | Reads /sys + /run/udev/data + /proc/mounts only (no block I/O). Judge a drive ONLY on its own mount (`localapi.ownMountIdentity`) — never the containing fs, or an absent drive reads as the root disk. A shared disk key is "same, proven" even when unknown. Pinned by identity_test.go (fixture trees) + samedisk_test.go |
| `localapi.Server.EnsureInternalPlace` / `InternalPlaceRecord` | internal/localapi/internalplace.go | `EnsureInternalPlace(ctx)` (startup + 20 s tick); `ReadInternalPlace` / `WriteInternalPlace` | the one-disk box's internal data place (felhom.eu `16` §5, v0.158.0): mounts it at /mnt/belso by fs-UUID like an enrolled drive, records it `enrolled`; the controller binds + registers it | Never takes /mnt/belso from a household drive (a foreign mount, or a unit naming another UUID, waits). Never formats/ejects (dm on the system disk → role system). The ONLY maker is the root helper `configs/felhom-internal-place` (installer + signed `add_internal_place`). Pinned by internalplace_test.go |
| `configs/felhom-internal-place` | configs/ (bundle → /usr/local/sbin) | `plan\|create <planned GiB>` | the ruled size rule (reserve max(16, 10 %) ≤ 48 GiB; ≥ 20 GiB or nothing) + the only lvcreate/mkfs of a system-disk volume | Formats only a felhom-belso with NO signature; refuses a foreign one; never resizes/removes. Sudoers: `^(plan\|create) 0$` only. Pinned by configs/test_felhom_internal_place.py |
| `storage.candidatePartitions` / `CandidatePartition` | internal/storage/partitions.go | `CandidateDisk.Partitions` | partitions as places (`16` §6): empty / filesystem / in_use per part of a free disk; `Formattable` only while no node of the disk is mounted | A part with children or a member type is never listed; a raw /mnt mount counts as Felhom's only when corroborated (felhomOwnedMounts) — for LISTING only: the root mkfs wrapper trusts no such claim (v0.158.1). Pinned by partitions_test.go |
| partition wipe id `byid:<…-partN>+fsuuid:<uuid>` | internal/storage/durable_device.go | `DeviceDurableID` / `ResolveDurableDevice` | a data-bearing PART binds disk id + part number + filesystem (`16` §6) | Both halves must resolve to the SAME device or the wipe refuses. Whole disks keep the plain id. Pinned by partition_durable_test.go |
| `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process |
| `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` |
@@ -51,10 +57,12 @@
| `SudoHostOps.EnsureMount` | internal/storage/hostops.go | `EnsureMount(ctx, MountSpec) error` | persistent by-UUID systemd .mount | Validate→render→stage→`install`→`enable --now`; idempotent |
| `SudoHostOps.Unmount` | internal/storage/hostops.go | `Unmount(ctx, where) error` | detach a mount unit | DESTRUCTIVE — caller MUST have gated it; does not self-authorize |
| `SudoHostOps.ReassertEnrolledMounts` | internal/storage/hostops.go | `ReassertEnrolledMounts(ctx)` | reboot remount (re-resolve by UUID) | Re-asserts unless mounted AND enabled (`shouldReassertMount`); skips absent UUIDs |
| `SudoHostOps.ForgetRetiredDrives` (v0.159.0, R-1124) | internal/storage/forget_retired.go | `ForgetRetiredDrives(ctx, IntentReader) int` | drop the host mount unit of a DECOMMISSIONED drive that is GONE (start-up + the 20 s tick) | Only decommissioned + absent + unmounted; a present decommissioned drive keeps its unit (H3 re-enroll). The file goes only through `felhom-priv-apply unit-forget`, which re-checks all three as root |
| `GuestBinder.AttachDrive` / `DetachDrive` | internal/localapi/intermediary.go | `AttachDrive(ctx, vmid, where) (guestPath, err)` | live drive hot-swap under `/mnt/felhom-drives` | Normalizes to EXACTLY ONE bind via `countHostMounts` (converges double-binds); force re-bind when guest can't see it |
| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
| `GuestBinder.GuestSeesMount` / `GuestBootID` | internal/localapi/intermediary.go | `GuestSeesMount(ctx, vmid, path) bool` | guest-visible (usable) signal; reboot detection | Host bind present ≠ guest sees it (non-recursive parent bind) |
| `GuestBinder.GuestBootKey` (R-1395) | internal/localapi/intermediary.go | `GuestBootKey(ctx, vmid) string` | "is this a new boot of the guest?" — `<kernel boot_id>-<guest init start>` | **`GuestBootID` carries `/proc/stat` `btime`, which the kernel computes from the wall clock: a clock STEP moves it.** Use the key for any „same boot?” question; `""` = unreadable. Seams `hostBtimeFn` / `hostBootIDFn` / `procStarttimeFn` |
| `SudoHostOps.EnsureNetworkMount` / `RemoveNetworkMount` / `ListNetworkMounts` | internal/storage/netmount.go | `EnsureNetworkMount(ctx, spec) error` | NAS automount pair | rm glob confined to `mnt-felhom*` units; NAS ≠ drive (no durable-id/SMART/wipe); RemoveNetworkMount doubles as the verify-fail rollback (idempotent) |
| `NetworkMountedAt` / `NetworkEndpointReachable` | internal/storage/netmount.go | `NetworkMountedAt(where) bool` | verify mount-truth + the 2 s add pre-probe | /proc/mounts is the ONLY mount-success judge (autofs trigger ≠ mounted; readability ≠ mounted — SPIKE-nas-verify §8) |
| `SudoHostOps.ReassertNetworkAutomounts` + `Server.ReassertNetworkMounts` + `guesthook.PostStartNetworkReassert` | internal/storage/netreassert.go, internal/localapi/netreassert.go, internal/guesthook/netreassert.go | `ReassertNetworkAutomounts(ctx) []NetReassertResult` | NAS guest-reboot heal (RCA fix 1): re-arm idle automount triggers (stop + enable --now) so the fresh mount event propagates into running guests | NEVER call from periodic health paths (an idle trigger is HEALTHY); active real mounts are never touched; hook leg runs as root (direct systemctl), daemon leg via sudo |
@@ -66,8 +74,8 @@
|---|---|---|---|---|
| `IntentStore` (`Get/SetEnrolled/SetEjected/SetDecommissioned/OnAbsent`) | internal/storage/intent.go | `OpenIntentStore(path)` | drive intent (4-state self-heal) | Keyed by durable-id only; `OnAbsent` is the ONLY ejected→enrolled path; refuses empty ids |
| `GuestBindStore` (`Record/Remove/Guests`) | internal/localapi/guestbindstore.go | `OpenGuestBindStore(path)` | per-guest enrolled binds (F9 re-assert) | Same tmp+rename 0600 pattern as IntentStore |
| `FormatJobStore` + `startFormatDetached` + `RecoverFormatJob` | internal/localapi/formatjob.go | `startFormatDetached(device, durableID, fstype, blank) <-chan error` | detached, restart-surviving mkfs (F20-BUG3) | Runs off `s.baseCtx` (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank |
| `TokenStore.Mint` / `Lookup` | internal/localapi/tokenstore.go | `Mint(vmid) (plaintext, error)` | per-guest local-API tokens | Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup RELOADS the file once on a miss (v0.63.0, B3): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence without a restart; append-only size check bounds the re-read |
| `FormatJobStore` + `startFormatDetached` + `RecoverFormatJob` | internal/localapi/formatjob.go | `startFormatDetached(device, durableID, fstype, blank) (*formatJob, <-chan error)` | detached, restart-surviving mkfs (F20-BUG3) | Runs off `s.baseCtx` (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank; on success `job.FSUUID` = the new filesystem's UUID, read back only when the durable id still resolves to the formatted device (R-25) — read it only after `done` delivers |
| `TokenStore.Mint` / `Lookup` | internal/localapi/tokenstore.go | `Mint(vmid) (plaintext, error)` | per-guest local-API tokens | Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup stats the file on EVERY call and reloads BEFORE answering when the append-only log grew (R-269; was reload-on-miss only, v0.63.0 B3, which let a token rotated out by another process keep authorizing as a map hit): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence both ways without a restart; unchanged size = no re-read. Pinned by `TestTokenStore_RotatedOutTokenRejectedFirst` |
| `FileNonceStore.SeenOrRecord` | internal/authz/noncestore.go | `SeenOrRecord(nonce, exp) bool` | durable anti-replay | fsync'd before returning false; prune only after exp |
| `Journal` (`Append/Latest/InFlight/AlreadyApplied`) | internal/reconcile/journal.go | `OpenJournal(path)` | op journal + idempotency + crash recovery | `Recover` consumes `InFlight()`; scratch entries special-cased |
@@ -81,13 +89,20 @@
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause (`supervisorBrakeAllows` — every supervisor restart passes it; `recordSupervisorRestart` puts one on the record + R-539's slow counter); R-1344 (2026-10-30): a RUNNING controller whose Docker health reads „unhealthy" is alarmed at 6 sweeps and restarted at 10 (`restartUnresponsive`: the controller's busy file, then the brake); record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller`, `reboot_guest` since D65 — `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
### Proxmox client / hub / PBS / provisioning
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
| `Leg.runKernelCleanup` / `kernelCleanupSkip` / `Leg.KernelCleanupDryRun` + the wrapper's `Kernel.cleanup` | internal/osupdate/kernel.go; configs/felhom-os-apply (mode `kernel-cleanup`) | `kernelCleanupSkip(appliance, trigger, host, pve, kernel, blk, okStep) string` ("" = call it) | R-1368 (`11` §5.11.1): old kernel images are removed by the WRAPPER's rule; the agent decides only WHEN (the night leg, after a healthy host step, never in a night with a kernel step) | The plan names NO package (C1). Never `apt-get autoremove`. Whether a box may do it is the root-owned ring-0 mark, never the hub's ring. The hub report carries NO kernel view and no kernel version (the hub would read it as a kernel step). `--selftest=kernel-cleanup-dry-run` changes nothing. |
| `Leg.ProfileRefresh` + the wrapper's `Apply.profile_refresh` | internal/osupdate/profilerefresh.go; configs/felhom-os-apply (mode `profile-refresh`) | `ProfileRefresh(ctx, why, gate, tries, wait) string` (the outcome) | R-1388 (`11` §5.12): load every AppArmor profile from a fresh compile once per kernel the box starts on; restart only a FAILED service whose own program's cached profile was stale | The plan names nothing (F1). A stale copy is told by the CACHED FILE changing at the rewrite — never by the kernel's loaded hash (it differs between a cache load and a fresh load for multi-profile files; `lxc-containers` never compiles the same twice). Called at agent start and after a bundle install, under the heavy-op gate. |
| `pvetrust.Manager` (`Ensure`, `OnCallError`, `Status`) + `proxmox.Client.SetTLS` / `proxmox.TrustProblem` + `Leg.PVECACopy` + the wrapper's `pve_ca_copy` | internal/pvetrust/pvetrust.go; internal/proxmox/{client,trust}.go; internal/osupdate/pvecacopy.go; configs/felhom-os-apply (mode `pve-ca-copy`) | `Ensure(ctx, why) (changed bool)`; `TrustProblem(err) string` | R-1380 (felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA; the move from a pinned certificate; the cause of a trust failure | THE ORDER: the copy, a real call with the copy ALONE, then ONE config write that clears the fingerprint LAST (a set fingerprint wins over `ca_file` in `TLSConfig.build`). Never clear the fingerprint anywhere else. The wrapper's plan names nothing (T1). `agent.json` is written by the agent in place, with a parked recovery copy — the same way as the backup-target move. |
| `config.ReplaceInPlace` / `config.RecoverCutWrite` (+ `config.FileMu`) | internal/config/cutwrite.go; internal/config/filelock.go | `ReplaceInPlace(path, prev, next) error`; `RecoverCutWrite(path) (note string, err error)` | R-1380 (felhom.eu `03` §12a item 7): the only way to write `agent.json` — the folder is root-owned, so no rename; the old bytes go to a journal in the state folder first, and the daemon repairs a cut write at start, before it reads the config | A new writer takes `FileMu`, calls `ReplaceInPlace`, opens no file by hand, and is added to the two source pins in `cmd/felhom-agent` (`…TakeTheSharedLock`, `…UseTheJournalledWrite`). A package whose tests write the config sets `config.CutWriteDir` in `TestMain` |
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
@@ -105,8 +120,8 @@
| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
| `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew |
| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v1) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) |
| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v2, R-978: json mode returns the package in `PendingUpload` instead of uploading) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) |
| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); **R-978: the new package reaches the hub only at the claim, before R is handed out (`EscrowCeremonyConfig.Upload`); TTL and a new start wait while that upload is in flight — an abandoned ceremony leaves the previous package current**; **R-1151: a claim with `{"defer_upload":true}` hands out R + two word positions and keeps the package until `POST /escrow/ceremony/confirm` brings those two words back (30-min window, only the two words kept, never R); a claim without it keeps the R-978 order (old controllers)**; subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
| `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) |
| `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker |
@@ -121,7 +136,7 @@
| Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only |
| Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set |
| Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction |
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`; R-856 `crashGuardStatePath` — GET /host/crash-guard's state file, internal/localapi/crashguard.go) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
| `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) |
| `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool** — `BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs |
| `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted` ⇒ **quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s |
@@ -156,16 +171,22 @@
| `proxmox.Runner` | internal/proxmox/privileged.go | `*ExecRunner` (direct/sudo) | `mockRunner` internal/proxmox/mock_test.go; runner fakes in storage tests |
| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
| `storage.IdentityHost` | internal/storage/identity.go | `*SysfsIdentityHost` | fixture /sys + udev trees in identity_test.go (`newFx`); `fakeIdentHost` in internal/localapi/samedisk_test.go. A SEPARATE interface from HostReader on purpose — adding methods to HostReader would break every existing fake. |
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `lanresolver.hostRoot` + `dnsmasqUnitPaths` (data seam, R-317) | internal/lanresolver/lanresolver.go | prod `hostRoot = "/"`; probe = the `dnsmasq` package's systemd UNIT, never `/usr/sbin/dnsmasq` (owned by `dnsmasq-base`) | internal/lanresolver/ensure_dnsmasq_test.go — fixture root tree + recording `proxmox.Runner`; the REAL `os.Stat` probe and `EnsureDnsmasq` run. `TestEnsureDnsmasq_ProductionProbeIsTheUnit` pins the production wiring |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct |
| `linetest.Tester` + `linetest.Client` (R-1247) | internal/linetest/linetest.go · internal/linetest/iperf.go | `New(runner, target, gate, uplink, hostID, statePath, logger)` / `Tick(ctx)` / `Value()` / `LineTestStatus(ctx)`; `Client{Streams, Time}.Run(ctx, addr)` | the weekly upload line test: a 10-second iperf3 upload (the agent speaks the iperf3 protocol — nothing installed on the host) to ep0's tunnel-only server (`wgtunnel.Manager.LineTestTarget`), Tuesday 10:00-15:59 local from a per-host slot, after 10 quiet minutes of the PHYSICAL uplink (`SysUplinkTx`: the bridge's ports with a `device` link — the bridge counter misses bridged guest traffic), under `backup.InFlight`; best of 3; reported as `line_test`, served as GET /host/line-test | **The value is the SERVER's received bytes**, never the client's written ones (the send buffer). `Run` refuses a context without a deadline. iperf3 numbers streams 1, 3, 4, 5 — the server matches results by those ids. „server busy" (ACCESS_DENIED) stops the measurement and is retried at the next tick, at most `MaxTriesPerDay` |
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
| localapi backup cadence cap (R-973) | internal/localapi/backup_cap.go | `backupCadenceCap(cadence)`, `Server.capRefuses`, `Server.BackupCapRefusal()` | POST /backup refuses (429) a run the `07` §6 schedule would never ask for: ≤24 h tier → 3 successes per rolling 24 h; longer tier → 1 per max(cadence−24 h, cadence/2) | Counts SUCCESSES only, from this process AND `BackupSuccessState.RecentBackupSuccesses` (so a restart does not reset it); refusals saved (`RecordBackupRefusal`) and raised for 24 h as the CRITICAL host-report status `localapi:backup-cadence-cap` (`backupCapStatus` in main) → the hub's existing `agent_capability_degraded`. Tests `TestBackupCap_*`. |
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec; the image write via `felhom-priv-apply controller-image`) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
+67
View File
@@ -0,0 +1,67 @@
package main
import (
"go/ast"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
)
// R-973: a cadence-cap refusal becomes a CRITICAL degraded status — what the hub's existing
// agent_capability_degraded event (operator-only) keys on — and clears when the refusal ages out.
func TestBackupCapStatus_RefusalIsCriticalDegraded(t *testing.T) {
at := time.Date(2026, 10, 18, 3, 0, 0, 0, time.UTC)
s := backupCapStatus(func() (string, int, time.Time, bool) { return "felhom-pbs", 9201, at, true })
if !s.Critical || s.Status != capability.StatusDegraded || !strings.Contains(s.Reason, "9201") || !strings.Contains(s.Reason, "felhom-pbs") {
t.Fatalf("a refusal must raise a critical degraded status naming the guest and the tier: %+v", s)
}
if s := backupCapStatus(func() (string, int, time.Time, bool) { return "", 0, time.Time{}, false }); s.Status != capability.StatusOK || !s.Critical {
t.Fatalf("no refusal → ok (and still critical, so a later refusal alerts): %+v", s)
}
if s := backupCapStatus(nil); s.Status != capability.StatusOK {
t.Fatalf("no local API → ok: %+v", s)
}
}
// Wiring (the built-but-never-wired class): runDaemon's capability probe appends backupCapStatus, and the
// late-bound source is set from the local API server.
//
// COMPANION RED-PROOF (observed 2026-10-18): delete `backupCapRefusal = localSrv.BackupCapRefusal` →
// "runDaemon never binds backupCapRefusal to the local API server". Restored.
func TestR973_CapRefusalIsWiredIntoTheReport(t *testing.T) {
_, f := parseMain(t)
var appended, bound bool
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "runDaemon" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.CallExpr:
if id, ok := x.Fun.(*ast.Ident); ok && id.Name == "backupCapStatus" && len(x.Args) == 1 {
if a, ok := x.Args[0].(*ast.Ident); ok && a.Name == "backupCapRefusal" {
appended = true
}
}
case *ast.AssignStmt:
if len(x.Lhs) == 1 && len(x.Rhs) == 1 {
if l, ok := x.Lhs[0].(*ast.Ident); ok && l.Name == "backupCapRefusal" {
if sel, ok := x.Rhs[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "BackupCapRefusal" {
bound = true
}
}
}
}
return true
})
}
if !appended {
t.Fatal("runDaemon's capability probe does not call backupCapStatus(backupCapRefusal)")
}
if !bound {
t.Fatal("runDaemon never binds backupCapRefusal to the local API server")
}
}
+141
View File
@@ -0,0 +1,141 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// A box whose agent.json was cut in the middle of a write must start (R-1380): main repairs the file from the
// journal BEFORE it reads it. The staging is the state a power cut leaves: a durable journal of the old bytes and a
// half-written config.
func TestMainRecoversACutConfigWrite(t *testing.T) {
dir := t.TempDir()
prev := config.CutWriteDir
config.CutWriteDir = filepath.Join(dir, "state")
t.Cleanup(func() { config.CutWriteDir = prev })
if err := os.MkdirAll(config.CutWriteDir, 0o700); err != nil {
t.Fatal(err)
}
cfgPath := filepath.Join(dir, "agent.json")
old := `{"proxmox":{"node":"pve","tls":{"fingerprint":"AA:BB"}}}`
if err := os.WriteFile(filepath.Join(config.CutWriteDir, "agent.json.write-in-flight"), []byte(old), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(cfgPath, []byte(`{"proxmox":{"node":"pv`), 0o600); err != nil {
t.Fatal(err)
}
if _, err := config.Load(cfgPath); err == nil {
t.Fatal("the staging is wrong: the cut config loads")
}
note, err := recoverCutConfigWrite(cfgPath)
if err != nil || note == "" {
t.Fatalf("note=%q err=%v", note, err)
}
cfg, err := config.Load(cfgPath)
if err != nil {
t.Fatalf("the agent does not start after the repair: %v", err)
}
if cfg.Proxmox.TLS.Fingerprint != "AA:BB" {
t.Fatalf("the repaired config lost the fingerprint: %+v", cfg.Proxmox.TLS)
}
}
// main must call the repair before config.Load, or the repair never runs on the box that needs it (a seam that is
// built and not wired). Read from the source: the call's position in main's body.
func TestMainCallsTheConfigRepairBeforeItLoadsTheConfig(t *testing.T) {
src, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
f, err := parser.ParseFile(token.NewFileSet(), "main.go", src, 0)
if err != nil {
t.Fatal(err)
}
var repair, load token.Pos
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Name.Name != "main" || fn.Body == nil {
continue
}
ast.Inspect(fn.Body, func(n ast.Node) bool {
c, ok := n.(*ast.CallExpr)
if !ok {
return true
}
if id, ok := c.Fun.(*ast.Ident); ok && id.Name == "recoverCutConfigWrite" && repair == 0 {
repair = c.Pos()
}
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Load" {
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" && load == 0 {
load = c.Pos()
}
}
return true
})
}
if repair == 0 || load == 0 {
t.Fatalf("main: repair call found=%v, config.Load found=%v", repair != 0, load != 0)
}
if repair > load {
t.Fatal("main reads the config BEFORE it repairs a cut write — a cut config stops the agent at start")
}
}
// Every writer of agent.json goes through config.ReplaceInPlace and none truncates the file itself: a writer that
// opens the config with O_TRUNC by hand is the unprotected write again.
func TestAgentJSONWritersUseTheJournalledWrite(t *testing.T) {
for _, w := range []struct{ file, fn string }{
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
} {
src, err := os.ReadFile(w.file)
if err != nil {
t.Fatal(err)
}
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, w.file, src, 0)
if err != nil {
t.Fatal(err)
}
found := false
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
continue
}
found = true
body := string(src[fset.Position(fn.Body.Pos()).Offset:fset.Position(fn.Body.End()).Offset])
calls := 0
ast.Inspect(fn.Body, func(n ast.Node) bool {
if c, ok := n.(*ast.CallExpr); ok {
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "ReplaceInPlace" {
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" {
calls++
}
}
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "OpenFile" {
t.Errorf("%s %s opens a file by hand — the config is written only by config.ReplaceInPlace", w.file, w.fn)
}
}
return true
})
if calls != 1 {
t.Errorf("%s %s: %d call(s) to config.ReplaceInPlace, want 1", w.file, w.fn, calls)
}
if strings.Contains(body, "O_TRUNC") {
t.Errorf("%s %s truncates a file itself", w.file, w.fn)
}
}
if !found {
t.Errorf("%s: function %s not found — the pin reads nothing", w.file, w.fn)
}
}
}
@@ -0,0 +1,62 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// Three functions in this process read agent.json whole, change one section and write it whole: the trust move, the
// PBS seed and the backup-target move. Without one shared lock two of them interleave and the second puts back the
// document it read before the first one's change (found by review, 2026-11-14). Each must take config.FileMu as its
// FIRST statement and release it by defer — asserted from the source, because the race itself is not a thing a unit
// test can stage reliably.
func TestAgentJSONWritersTakeTheSharedLock(t *testing.T) {
for _, w := range []struct{ file, fn string }{
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
} {
f, err := parser.ParseFile(token.NewFileSet(), w.file, nil, 0)
if err != nil {
t.Fatalf("%s: %v", w.file, err)
}
found := false
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
continue
}
found = true
isLockCall := func(e ast.Expr, name string) bool {
c, ok := e.(*ast.CallExpr)
if !ok {
return false
}
sel, ok := c.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != name {
return false
}
inner, ok := sel.X.(*ast.SelectorExpr)
if !ok || inner.Sel.Name != "FileMu" {
return false
}
pkg, ok := inner.X.(*ast.Ident)
return ok && pkg.Name == "config"
}
if len(fn.Body.List) < 2 {
t.Errorf("%s %s: too short to hold the lock", w.file, w.fn)
continue
}
first, ok1 := fn.Body.List[0].(*ast.ExprStmt)
second, ok2 := fn.Body.List[1].(*ast.DeferStmt)
if !ok1 || !isLockCall(first.X, "Lock") || !ok2 || !isLockCall(second.Call, "Unlock") {
t.Errorf("%s %s: must start with config.FileMu.Lock() and defer config.FileMu.Unlock()", w.file, w.fn)
}
}
if !found {
t.Errorf("%s: function %s not found — a writer of agent.json was renamed; keep it under the shared lock and update this test", w.file, w.fn)
}
}
}
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
// passes as held → FAILS.
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
body := func(id string) []byte {
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
return b
}
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
if err != nil {
return nil, err
}
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
}
}
ctx := context.Background()
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
}
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
}
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
t.Fatal("a package without an identity blob cannot be compared — an error")
}
}
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"context"
"encoding/json"
"go/ast"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// R-978, the root half: the controller-driven ceremony (json mode, the FELHOM_ESCROW argv with
// `--upload`) must hand its package back instead of sending it — the daemon sends it at the claim.
//
// COMPANION RED-PROOF (observed 2026-10-18): set `deferUpload: false` in escrowOptsFromCLI → this fails
// with "json mode must defer the upload (R-978): got deferUpload=false", and the delivery test's json leg
// then sends the package at create time ("put called 1 times"). Restored.
func TestEscrowOptsFromCLI_JSONModeDefersUpload(t *testing.T) {
// The FELHOM_ESCROW argv: --selftest=escrow-create --upload --output=json.
if o := escrowOptsFromCLI("", false, false, true, "", true); !o.deferUpload || !o.upload {
t.Fatalf("json mode must defer the upload (R-978): got deferUpload=%v upload=%v", o.deferUpload, o.upload)
}
// Text mode (an operator at a terminal) keeps the immediate upload; no --upload means no upload at all.
if o := escrowOptsFromCLI("", false, false, true, "", false); o.deferUpload {
t.Fatal("text mode must upload at once")
}
if o := escrowOptsFromCLI("", false, false, false, "", true); o.deferUpload {
t.Fatal("without --upload there is nothing to defer")
}
}
func TestDeliverEscrowPackage_JSONModeSendsNothing(t *testing.T) {
cfg := config.Config{}
cfg.Hub.URL, cfg.Hub.HostID, cfg.Hub.APIKey = "https://hub.invalid", "host-1", "k"
res := escrow.CreateResult{Blob: []byte("ciphertext"), KeyFingerprint: "f2:87", Posture: escrow.PostureZeroKnowledge, IdentityBlob: []byte("id")}
for _, jsonMode := range []bool{true, false} {
calls := 0
var sent []byte
put := func(_ context.Context, _ config.Config, body []byte) error { calls++; sent = body; return nil }
o := escrowOptsFromCLI("", false, false, true, "", jsonMode)
pending, uploaded, err := deliverEscrowPackage(context.Background(), cfg, res, "abc", o.deferUpload, put)
if err != nil {
t.Fatal(err)
}
if jsonMode {
if calls != 0 || uploaded {
t.Fatalf("json mode: put called %d times (uploaded=%v), want 0 — the package must wait for the claim", calls, uploaded)
}
var m map[string]any
if json.Unmarshal(pending, &m) != nil || m["blob_b64"] == "" || m["identity_blob_b64"] == "" || m["restic_pw_sha256"] != "abc" {
t.Fatalf("json mode: the pending package is not the hub upload body: %s", pending)
}
} else if calls != 1 || !uploaded || pending != nil || len(sent) == 0 {
t.Fatalf("text mode: put called %d times, uploaded=%v, pending=%q — want one immediate upload", calls, uploaded, pending)
}
}
// A hub that is not configured fails in both modes (as the pre-R-978 upload did).
if _, _, err := deliverEscrowPackage(context.Background(), config.Config{}, res, "", true, nil); err == nil {
t.Fatal("an unconfigured hub must fail the ceremony even when the upload is deferred")
}
}
// R-978 wiring (the built-but-never-wired class): runDaemon's localapi.EscrowCeremonyConfig carries an
// Upload that calls putEscrowBody. Without it every claim of a new package answers 503 and no customer
// can finish a ceremony.
//
// COMPANION RED-PROOF (observed 2026-10-18): delete the `Upload:` line in runDaemon → this fails with
// "runDaemon's EscrowCeremonyConfig has no Upload calling putEscrowBody". Restored.
func TestR978_CeremonyUploadIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
if !callsWithin(f, "main")["runDaemon"] {
t.Fatal("main → runDaemon is broken — the path this test asserts is not the live one")
}
wired := false
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "runDaemon" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
cl, ok := n.(*ast.CompositeLit)
if !ok {
return true
}
sel, ok := cl.Type.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "EscrowCeremonyConfig" {
return true
}
for _, el := range cl.Elts {
if kv, ok := el.(*ast.KeyValueExpr); ok {
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "Upload" {
ast.Inspect(kv.Value, func(n ast.Node) bool {
if ce, ok := n.(*ast.CallExpr); ok {
if id, ok := ce.Fun.(*ast.Ident); ok && id.Name == "putEscrowBody" {
wired = true
}
}
return true
})
}
}
}
return true
})
}
if !wired {
t.Fatal("runDaemon's EscrowCeremonyConfig has no Upload calling putEscrowBody")
}
}
+53
View File
@@ -0,0 +1,53 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-444: the weekly trim has the guestnet shape (component + reporter seam + goroutine), so its wiring is asserted
// from the AST like TestMainWiresGuestNetWatchdog — a unit-green trim job that main.go never starts is the inert-seam
// defect. It must also share the ONE heavy-op gate (heavyOps), or it could run beside a backup.
func TestMainWiresGuestDiskTrim(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var constructedWithGate, reporterWired, started bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
if fn, ok := node.Fun.(*ast.SelectorExpr); ok {
switch fn.Sel.Name {
case "New":
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "fstrim" && len(node.Args) >= 3 {
if id, ok := node.Args[2].(*ast.Ident); ok && id.Name == "heavyOps" {
constructedWithGate = true
}
}
case "SetGuestDiskTrimReporter":
reporterWired = true
}
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "diskTrim" {
started = true
}
}
}
return true
})
if !constructedWithGate {
t.Error("main.go never calls fstrim.New(..., heavyOps, ...) — no trim job, or one outside the heavy-op gate")
}
if !reporterWired {
t.Error("main.go never calls collector.SetGuestDiskTrimReporter — the guest_disk_trim stanza never reaches the hub")
}
if !started {
t.Error("main.go never starts the trim job with `go diskTrim.Run(ctx)`")
}
}
+72
View File
@@ -0,0 +1,72 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
)
// R-1250: SetLineTest takes a LineTestReader and finds the request half by a type assertion, so a Tester that lost
// WaitingForQuiet or RunNow would compile and silently answer 503 to every seeding box. This line makes it a build
// failure instead.
var _ localapi.LineTestRequester = (*linetest.Tester)(nil)
// R-1247: the weekly line test has the fstrim shape (component + reporter seam + goroutine) plus a local-API reader,
// so its wiring is asserted from the AST. It must share the ONE heavy-op gate (heavyOps) — or it could run beside a
// backup — and take its server from the tunnel manager, never a literal address.
func TestMainWiresTheLineTest(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var withGate, withTunnelTarget, reporterWired, localAPIWired, started bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
if fn, ok := node.Fun.(*ast.SelectorExpr); ok {
switch fn.Sel.Name {
case "New":
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "linetest" && len(node.Args) >= 3 {
if id, ok := node.Args[2].(*ast.Ident); ok && id.Name == "heavyOps" {
withGate = true
}
if sel, ok := node.Args[1].(*ast.SelectorExpr); ok && sel.Sel.Name == "LineTestTarget" {
withTunnelTarget = true
}
}
case "SetLineTestReporter":
reporterWired = true
case "SetLineTest":
localAPIWired = true
}
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "lineTest" {
started = true
}
}
}
return true
})
if !withGate {
t.Error("main.go never calls linetest.New(..., heavyOps, ...) — no line test, or one outside the heavy-op gate")
}
if !withTunnelTarget {
t.Error("linetest.New does not take its server from wgMgr.LineTestTarget")
}
if !reporterWired {
t.Error("main.go never calls collector.SetLineTestReporter — the line_test stanza never reaches the hub")
}
if !localAPIWired {
t.Error("main.go never calls localSrv.SetLineTest — the controller's GET /host/line-test always reads present:false")
}
if !started {
t.Error("main.go never starts the line test with `go lineTest.Run(ctx)`")
}
}
+588 -93
View File
File diff suppressed because it is too large. Load diff
+51
View File
@@ -0,0 +1,51 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// 2026-10-29 (operator brief, Part 1): the agent's operator actions are unit-green in localapi; main.go must also put
// the server in the report loop's observer chain, give the collector its results and give the server the report
// trigger — an inert seam otherwise (the „seam built but never wired" class). Asserted from the AST.
// RED-PROOF: seen failing before the wiring existed (all three false).
func TestMainWiresTheAgentOperatorActions(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var observed, results, trigger bool
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
fn, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
switch fn.Sel.Name {
case "MultiObserver":
for _, a := range call.Args {
if id, ok := a.(*ast.Ident); ok && id.Name == "localSrv" {
observed = true
}
}
case "SetOperatorActionResults":
if len(call.Args) == 1 {
if sel, ok := call.Args[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "OperatorActionResults" {
results = true
}
}
case "SetOperatorActionTrigger":
trigger = true
}
return true
})
if !observed || !results || !trigger {
t.Fatalf("operator actions wiring: observer=%v results=%v trigger=%v — all three must be wired in main.go", observed, results, trigger)
}
}
+71
View File
@@ -0,0 +1,71 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-1380: the trust move is a component with seams (the root step, the proof, the running client, the report stanza,
// the failed-call hook). A unit-green pvetrust that main.go never starts is the inert-seam defect, so the wiring is
// asserted from the AST: built once, run at start and after a bundle install, on a timer, fed by a failed collect, and
// reported. The same for the profile refresh (R-1388), which had no such test.
func TestMainWiresPVETrust(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
seen := map[string]int{}
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
sel, ok := node.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
x, _ := sel.X.(*ast.Ident)
switch {
case x != nil && x.Name == "pveTrust" && sel.Sel.Name == "Ensure":
seen["ensure"]++
case sel.Sel.Name == "SetProxmoxTrustSource":
seen["stanza"]++
case sel.Sel.Name == "SetProfileRefreshSource" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "ProfileRefreshStanza" {
seen["profile refresh → report"]++ // R-1383
}
case sel.Sel.Name == "SetGuestPowerGaveUpSource" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "GuestPowerGaveUpStanza" {
seen["guest-power give-up → report"]++ // R-1383
}
case sel.Sel.Name == "SetCollectErrorHook" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "OnCallError" {
seen["hook"]++
}
case x != nil && x.Name == "osLeg" && sel.Sel.Name == "ProfileRefresh":
seen["profile-refresh"]++
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "pveTrust" {
seen["timer"]++
}
}
case *ast.KeyValueExpr:
if k, ok := node.Key.(*ast.Ident); ok && k.Name == "Apply" {
if v, ok := node.Value.(*ast.SelectorExpr); ok && v.Sel.Name == "SetTLS" {
seen["apply=SetTLS"]++
}
}
}
return true
})
want := map[string]int{"ensure": 2, "stanza": 1, "hook": 1, "timer": 1, "apply=SetTLS": 1, "profile-refresh": 2,
"guest-power give-up → report": 1, "profile refresh → report": 1}
for k, n := range want {
if seen[k] < n {
t.Errorf("main.go wires %q %d time(s), want at least %d (the two-call ones: at start and after a bundle install)", k, seen[k], n)
}
}
}
@@ -0,0 +1,91 @@
package main
import (
"context"
"encoding/json"
"io"
"log/slog"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
)
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
type orderRunner struct {
mu sync.Mutex
order []string
}
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
var p map[string]any
b, _ := os.ReadFile(args[1])
_ = json.Unmarshal(b, &p)
mode, _ := p["mode"].(string)
r.order = append(r.order, "os-apply:"+mode)
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
}
if len(args) >= 1 {
r.order = append(r.order, filepath.Base(name)+":"+args[0])
}
return []byte("ok"), nil, nil
}
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
t.Helper()
stateDir := t.TempDir()
if marker != nil {
dir := filepath.Join(stateDir, "selfupdate")
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
b, _ := json.Marshal(marker)
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
t.Fatal(err)
}
}
r := &orderRunner{}
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
mgr.MaybeCommit(context.Background())
r.mu.Lock()
defer r.mu.Unlock()
return append([]string(nil), r.order...)
}
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
}
}
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
}
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 0 {
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
}
}
+105
View File
@@ -0,0 +1,105 @@
package main
import (
"go/ast"
"testing"
)
// R-894 — the on-disk backup record is WIRED on the daemon path (the built-but-never-wired class).
// main → runDaemon → buildLocalAPIServer, and inside it the localapi.Options literal carries
// LastKnownBackups built by backup.NewBackupSuccessState. An AST walk, not a string match, for the
// reasons in escrow_recover_wiring_test.go.
//
// COMPANION RED-PROOF (observed): delete the `LastKnownBackups:` line from buildLocalAPIServer → this
// fails with "localapi.Options in buildLocalAPIServer has no LastKnownBackups field". Restored.
//
// 2026-10-09: the SAME instance also feeds the host report (collector.SetKnownBackupReporter), so a
// restart right after a backup no longer erases the hub's evidence of it. The field may name a local
// variable; the variable must be built by backup.NewBackupSuccessState and be the one handed to the
// collector. RED-PROOF (observed): drop the SetKnownBackupReporter call → "not handed to the collector".
func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
if !callsWithin(f, "main")["runDaemon"] || !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
t.Fatal("main → runDaemon → buildLocalAPIServer is broken — the path this test asserts is not the live one")
}
var field, built bool
var fieldVar, handed string
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
cl, ok := n.(*ast.CompositeLit)
if !ok {
return true
}
sel, ok := cl.Type.(*ast.SelectorExpr)
if !ok {
return true
}
if pkg, _ := sel.X.(*ast.Ident); pkg == nil || pkg.Name+"."+sel.Sel.Name != "localapi.Options" {
return true
}
for _, el := range cl.Elts {
kv, ok := el.(*ast.KeyValueExpr)
if !ok {
continue
}
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "LastKnownBackups" {
field = true
if callsIn(kv.Value)["backup.NewBackupSuccessState"] {
built = true
}
if id, ok := kv.Value.(*ast.Ident); ok {
fieldVar = id.Name
}
}
}
return true
})
// a local variable: built by NewBackupSuccessState, and handed to the collector
ast.Inspect(fd.Body, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.AssignStmt:
for i, l := range x.Lhs {
if id, ok := l.(*ast.Ident); ok && fieldVar != "" && id.Name == fieldVar && i < len(x.Rhs) &&
callsIn(x.Rhs[i])["backup.NewBackupSuccessState"] {
built = true
}
}
case *ast.CallExpr:
if fn, ok := x.Fun.(*ast.SelectorExpr); ok && fn.Sel.Name == "SetKnownBackupReporter" && len(x.Args) == 1 {
if id, ok := x.Args[0].(*ast.Ident); ok {
handed = id.Name
}
}
}
return true
})
}
if !field {
t.Fatal("localapi.Options in buildLocalAPIServer has no LastKnownBackups field")
}
if !built {
t.Fatal("LastKnownBackups is not built by backup.NewBackupSuccessState")
}
if handed == "" || handed != fieldVar {
t.Fatalf("the saved backups (%q) are not handed to the collector (SetKnownBackupReporter got %q)", fieldVar, handed)
}
}
func callsIn(n ast.Node) map[string]bool {
out := map[string]bool{}
ast.Inspect(n, func(n ast.Node) bool {
if ce, ok := n.(*ast.CallExpr); ok {
if fn, ok := ce.Fun.(*ast.SelectorExpr); ok {
if x, ok := fn.X.(*ast.Ident); ok {
out[x.Name+"."+fn.Sel.Name] = true
}
}
}
return true
})
return out
}
+101
View File
@@ -1,11 +1,14 @@
package main
import (
"bytes"
"context"
"errors"
"go/ast"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -415,3 +418,101 @@ func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
"be missed entirely", storeGrantRepairReportWindow)
}
}
// ── R-1302 — the first grant of a tier is no loss ──────────────────────────────────────
//
// A fresh install arms the whole-guest tier before its storage exists, so its first probe found no grant, repaired it and
// logged „GRANT WAS MISSING … investigate the loss" with a DEGRADED capability — on every new box (VM 9375, 2026-10-26).
//
// RED-PROOF (2026-11-02): seen failing before storeGrantAfterRepair existed (build), and with its first-grant branch removed
// TestR1302_TheFirstGrantIsNoLoss fails on „a first grant must not report a loss".
func r1302Repairer(t *testing.T, path string) (*storeGrantRepairer, *bytes.Buffer) {
t.Helper()
var buf bytes.Buffer
return &storeGrantRepairer{run: (&fakeRepairRunner{}).Run, seenPath: path,
log: slog.New(slog.NewTextHandler(&buf, nil))}, &buf
}
func TestR1302_TheFirstGrantIsNoLoss(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
r, buf := r1302Repairer(t, path)
healthy := capability.Status{Name: "pve:store-grant:felhom-pbs", Status: capability.StatusDegraded, Reason: "missing", Critical: true}
s := storeGrantAfterRepair("felhom-pbs", true, r, healthy)
if s.Status != capability.StatusOK || s.Reason != "" {
t.Fatalf("a first grant must not report a loss: status %q reason %q", s.Status, s.Reason)
}
got := buf.String()
if strings.Contains(got, "investigate the loss") || strings.Contains(got, "level=ERROR") {
t.Fatalf("a first grant must not log the loss ERROR: %s", got)
}
if !strings.Contains(got, "level=INFO") || !strings.Contains(got, "granted") || !strings.Contains(got, "felhom-pbs") {
t.Fatalf("a first grant says so at INFO: %s", got)
}
if r.recentlyRepaired("felhom-pbs", time.Now()) {
t.Fatal("a first grant must not arm the loss report's latch")
}
}
func TestR1302_AGrantThatExistedAndVanishedKeepsTheError(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
first, _ := r1302Repairer(t, path)
first.noteHeld("felhom-pbs") // a healthy probe, in an earlier run of the agent
r, buf := r1302Repairer(t, path) // the agent restarted: the memory is the file, not the process
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
if s.Status != capability.StatusDegraded || !strings.Contains(s.Feature, "R-190") {
t.Fatalf("a grant that existed and vanished must keep its degraded cycle; got %q / %q", s.Status, s.Feature)
}
if got := buf.String(); !strings.Contains(got, "level=ERROR") || !strings.Contains(got, "investigate the loss") {
t.Fatalf("a grant that existed and vanished must keep its ERROR: %s", got)
}
if !r.recentlyRepaired("felhom-pbs", time.Now()) {
t.Fatal("the loss must arm the report's latch, or the hub may never hear it")
}
}
// After a first grant the tier counts as held: the NEXT disappearance is a loss.
func TestR1302_AfterTheFirstGrantALossIsALoss(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
r, _ := r1302Repairer(t, path)
storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
if s.Status != capability.StatusDegraded {
t.Fatalf("the second disappearance is a loss; got %q", s.Status)
}
}
// The direction of a doubt: a broken record, a repairer with no record, or no repairer at all keep the loud way.
func TestR1302_ADoubtKeepsTheError(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
t.Fatal(err)
}
r, _ := r1302Repairer(t, path)
if s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{}); s.Status != capability.StatusDegraded {
t.Fatalf("a corrupt record must not quieten a loss; got %q", s.Status)
}
noMemory, _ := r1302Repairer(t, "")
if s := storeGrantAfterRepair("felhom-pbs", true, noMemory, capability.Status{}); s.Status != capability.StatusDegraded {
t.Fatalf("no record must not quieten a loss; got %q", s.Status)
}
if s := storeGrantAfterRepair("felhom-pbs", true, nil, capability.Status{}); s.Status != capability.StatusDegraded {
t.Fatalf("no repairer must not quieten a loss; got %q", s.Status)
}
}
// The wiring, read from the source: production names the file, and a healthy probe records the tier.
func TestR1302_TheMemoryIsWired(t *testing.T) {
b, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
src := string(b)
if !strings.Contains(src, `seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json")`) {
t.Fatal("main.go must give the repairer its file, or every first grant reads as a loss")
}
start := strings.Index(src, "func storeGrantStatus(")
if start < 0 || !strings.Contains(src[start:start+3000], "repair.noteHeld(targetID)") {
t.Fatal("a healthy probe must record the tier, or a real loss would read as a first grant")
}
}
+5 -1
View File
@@ -43,7 +43,7 @@ func main() {
func run() error {
var (
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | agent_config_update")
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update | add_internal_place")
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
@@ -123,6 +123,10 @@ func run() error {
now := time.Now().UTC()
issued := now
if *ttl <= 0 || *ttl > authz.MaxWindow {
fmt.Fprintf(os.Stderr, "felhom-opsign: -ttl %s refused: a box accepts a window of at most %s (R-1280)\n", *ttl, authz.MaxWindow)
os.Exit(2)
}
expires := now.Add(*ttl)
// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
+46 -5
View File
@@ -25,6 +25,7 @@
Cmnd_Alias FELHOM_MOUNT = \
/usr/local/sbin/felhom-priv-apply ^unit mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$, \
/usr/local/sbin/felhom-priv-apply ^unit-forget mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl daemon-reload, \
/usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
@@ -80,7 +81,34 @@ Cmnd_Alias FELHOM_DNSMASQ = \
Cmnd_Alias FELHOM_GUESTHOOK = \
/usr/sbin/pct ^set [0-9]+ --hookscript local\:snippets/felhom-guest-hook\.sh$, \
/usr/sbin/pct ^set [0-9]+ --delete mp[0-9]+$, \
/usr/sbin/pct ^reboot [0-9]+$
/usr/sbin/pct ^reboot [0-9]+$, \
/usr/sbin/pct ^shutdown [0-9]+ --timeout [0-9]+$
# Host power-off (R-998, doc 14): the household's „A gép kikapcsolása" — guest shutdown above, then this one fixed
# vector. Guarded agent-side (10 min after boot, one per boot, never during a backup/restore/update/drive job).
Cmnd_Alias FELHOM_POWEROFF = \
/usr/bin/systemctl poweroff
# Dead half-written vzdump removal (R-1006, power-cut-fixes-2026-10-14): a power cut mid-backup leaves
# vzdump-lxc-<vmid>-<date>.tar.dat + .tmp/ on the dir storage; nothing removed them and every later backup was skipped
# „not enough space". ONE root helper (configs/felhom-vzdump-partial-clean, from the config bundle) re-checks everything
# as root — a dir storage from storage.cfg, the exact name of THAT vmid, not in use, dead — and refuses otherwise. The
# arguments here are its exact shape (storage id, vmid, one half-written name); the vmid-in-name match is the helper's.
Cmnd_Alias FELHOM_VZPARTIAL = \
/usr/local/sbin/felhom-vzdump-partial-clean ^[A-Za-z][A-Za-z0-9_-]{0\,63} [1-9][0-9]{2\,8} vzdump-(lxc|qemu)-[1-9][0-9]{2\,8}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$
# Internal data place („Belső tárhely", felhom.eu architecture/16 §5, §7; agent v0.158.0). ONE root helper
# (configs/felhom-internal-place, from the config bundle) creates <vg>/felhom-belso in the local-lvm thin pool by the
# ruled size rule and formats ONLY that volume, ONLY when it carries no signature. The agent calls it from the signed
# `add_internal_place` job (an old box), always with 0 GiB planned: its guest's volumes already exist. `plan` changes
# nothing. The installer runs the helper as root itself (a new box).
# R-1012 (agent v0.159.0): the READ-ONLY orphan-thin check before and after a whole-guest backup — lists the pool's thin
# device ids (through a metadata snapshot it reserves and releases) next to LVM's. It sends no other pool message; the
# repair stays a human step. Pinned by configs/test_felhom_thin_check.py.
Cmnd_Alias FELHOM_THINCHECK = \
/usr/local/sbin/felhom-thin-check ^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0\,63} [A-Za-z0-9_+][A-Za-z0-9_.+-]{0\,63}$
Cmnd_Alias FELHOM_INTERNALPLACE = \
/usr/local/sbin/felhom-internal-place ^(plan|create) 0$
# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
# /mnt/felhom-drives (self-bind + make-shared) and binds/unbinds each drive's felhom-data namespace UNDERNEATH it so the
@@ -111,15 +139,20 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
# + container across spaces, spike-confirmed)
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
# tee <FIXED image file> — WRITE the ref; content is fed on STDIN (no shell, no interpolation),
# the agent strict-validates the ref (controllerImageRe) before the write.
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
# compromised agent could hand the guest's bootstrap ANY image (sudo cannot see stdin).
# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/felhom-controller-image$, \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
@@ -129,6 +162,14 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
Cmnd_Alias FELHOM_STALELOCK = \
/usr/sbin/pct ^unlock [0-9]+$
# Weekly guest disk trim (R-444, operator ruling `09` §3 decision 139). A thin pool only ever grows from blocks the
# guest has FREED: `fstrim` inside the unprivileged container is refused (FITRIM: Operation not permitted), so the host
# trims the guest's mounts. Measured on demo-hp 2026-10-06: `pct fstrim 9201` rc 0 in 24.4 s, pool 65.53 % -> 33.40 %,
# apps kept answering. ONE exact pattern: a vmid and nothing else — no `--ignore-mountpoints`, no second argument
# (pinned: TestSudoersFstrimRuleIsExact). The agent runs it on a weekly daytime timer under the heavy-op gate.
Cmnd_Alias FELHOM_FSTRIM = \
/usr/sbin/pct ^fstrim [0-9]+$
# Restore-test scratch teardown (F-LEAK, Campaign 8, v0.110.0). A restore-test whose restore FAILS
# leaves a scratch guest the API token CANNOT destroy: `FelhomAgentGuest` is granted at /pool/felhom and
# a guest joins that pool only when its restore COMPLETES, so a failed restore leaves a pool-less guest
@@ -304,4 +345,4 @@ Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \
/usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY, FELHOM_POWEROFF, FELHOM_VZPARTIAL, FELHOM_INTERNALPLACE, FELHOM_THINCHECK
+39
View File
@@ -0,0 +1,39 @@
#!/bin/sh
# /etc/grub.d/42_felhom_oneshot — the kernel lane's one-shot ENTRIES (R-836, `09` §3 decision 172, `11` §5.11).
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it.
#
# One menu entry per installed Proxmox kernel, id `felhom-oneshot-<version>`, booted ONLY when 01_felhom_oneshot found
# the flag naming it. It is the normal entry plus option C (decision 172): softlockup_panic=1 hardlockup_panic=1
# hung_task_panic=1 panic=10 — a lockup the kernel can detect becomes a panic, and a panic restarts the box in 10 s into
# the default (the old kernel). A true dead freeze still needs a person (spike candidate 3 failed on all three boxes).
# It sorts AFTER 10_linux, so it is never entry 0 and never the default.
#
# No vfat ESP at /boot/efi → prints nothing (no flag can name these entries).
set -e
prefix="/usr"
exec_prefix="/usr"
datarootdir="/usr/share"
. "$datarootdir/grub/grub-mkconfig_lib"
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
[ -n "$esp_uuid" ] || exit 0
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
[ -n "$kernels" ] || exit 0
case "${GRUB_DEVICE}" in
/dev/mapper/*|/dev/dm-*|"") root_arg="root=${GRUB_DEVICE}" ;;
*) if [ -n "${GRUB_DEVICE_UUID}" ]; then root_arg="root=UUID=${GRUB_DEVICE_UUID}"; else root_arg="root=${GRUB_DEVICE}"; fi ;;
esac
[ -n "${GRUB_DEVICE}" ] || root_arg="root=$(findmnt -n -o SOURCE /)"
rel=$(make_system_path_relative_to_its_root /boot)
prep=$(prepare_grub_to_access_device "$(${grub_probe:-grub-probe} --target=device /boot)" | sed 's/^/ /')
for k in $kernels; do
[ -f "/boot/initrd.img-$k" ] || continue
cat <<EOF
menuentry 'Felhom one-shot: $k' --class proxmox --id felhom-oneshot-$k {
insmod gzio
$prep
echo 'Loading Linux $k (felhom one-shot) ...'
linux $rel/vmlinuz-$k $root_arg ro ${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
initrd $rel/initrd.img-$k
}
EOF
done
+36
View File
@@ -0,0 +1,36 @@
#!/bin/sh
# /etc/grub.d/01_felhom_oneshot — the kernel lane's ONE-SHOT boot (R-836, `09` §3 decisions 164 + 172, `11` §5.11).
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it; it prints GRUB script.
#
# At boot, GRUB reads `felhom_next` from an environment block on the ESP (vfat — GRUB can rewrite a file there; it
# cannot on the LVM /boot, R-836), CLEARS it, and — only if it names an installed kernel — boots that kernel's one-shot
# entry (42_felhom_oneshot) instead of the default. The next boot uses the default again whatever happens: a new kernel
# that panics comes back on the old one by itself. felhom-os-apply writes the flag (mode apply) and never the default
# for a new kernel. Measured on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
# `audits/kernel-spike-2026-10-07/` (candidate 2).
#
# No vfat ESP at /boot/efi, or no Proxmox kernel → prints nothing (the box boots exactly as before).
set -e
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
[ -n "$esp_uuid" ] || exit 0
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
[ -n "$kernels" ] || exit 0
cat <<EOF
# felhom kernel lane: a one-shot kernel named on the ESP, read and cleared before the menu
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=felhom_esp $esp_uuid
if [ -f (\$felhom_esp)/EFI/felhom/oneshot.env ]; then
load_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
if [ "\${felhom_next}" ]; then
set felhom_boot="\${felhom_next}"
set felhom_next=
save_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
EOF
for k in $kernels; do
printf ' if [ "${felhom_boot}" = "%s" ]; then set default="felhom-oneshot-%s"; fi\n' "$k" "$k"
done
cat <<EOF
fi
fi
EOF
+118
View File
@@ -0,0 +1,118 @@
#!/bin/bash
# felhom-internal-place — make the one-disk box's internal data place („Belső tárhely"), felhom.eu
# architecture/16-storage-physical-identity.md §5, §7 (sizes ruled by the operator 2026-10-16). Installed as
# /usr/local/sbin/felhom-internal-place (config bundle). Called by felhom-host-install.sh (a new box) and by the agent's
# signed `add_internal_place` job (an old box) — ONE implementation of the size rule and of the only mkfs it does.
#
# felhom-internal-place plan <planned GiB> → prints the numbers, changes nothing
# felhom-internal-place create <planned GiB> → makes the place (or finds it), prints the result
#
# <planned GiB> is thin space still to be promised to the guest (rootfs + data volume) that does not exist yet:
# the installer passes 32 + the data volume; an old box passes 0 (its guest's volumes are already counted).
#
# The rule (16 §7, ruled): reserve = max(16 GiB, 10 % of the pool), at most 48 GiB; internal = pool − reserve −
# every thin volume already promised in the pool − <planned>, made only when ≥ 20 GiB. So nothing in the pool is ever
# promised beyond pool − reserve: a pool at 100 % stops every volume on the box.
#
# It trusts nothing it is told. Every rule is checked here, as root, and any doubt REFUSES (exit 1):
# - the pool is the `lvmthin: local-lvm` section of /etc/pve/storage.cfg (its vgname/thinpool), nothing else;
# - the only volume it creates, formats or reads is <vg>/felhom-belso;
# - it formats ONLY a felhom-belso that carries no signature at all (blkid -p finds nothing) — the one it just made,
# or one a crash left between lvcreate and mkfs; any other content REFUSES; it never resizes or removes anything.
# Output, one line: „created uuid=<fs-uuid> gib=<n>", „formatted uuid=… gib=…" (a crash left it empty), „existing uuid=<fs-uuid> gib=<n>", „skipped <why>" (exit 0),
# or „plan pool=… reserve=… promised=… planned=… internal=…". Pinned by configs/test_felhom_internal_place.py.
set -u
STORAGE_CFG=${FELHOM_STORAGE_CFG:-/etc/pve/storage.cfg}
LV_NAME=felhom-belso
LABEL=felhom-belso
MIN_GIB=20
die() { echo "felhom-internal-place: REFUSED: $*" >&2; exit 1; }
pool_of_local_lvm() { # prints "<vg> <thinpool>" of the lvmthin storage local-lvm, or nothing
awk '
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (type == "lvmthin" && id == "local-lvm"); next }
in_s && /^[ \t]+vgname[ \t]/ { vg = $2 }
in_s && /^[ \t]+thinpool[ \t]/ { tp = $2 }
END { if (vg != "" && tp != "") print vg, tp }
' "$STORAGE_CFG"
}
gib_floor() { echo $(( $1 / 1073741824 )); }
[[ $# -eq 2 ]] || die "usage: felhom-internal-place plan|create <planned GiB>"
MODE=$1; PLANNED=$2
[[ "$MODE" == plan || "$MODE" == create ]] || die "mode must be plan or create"
[[ "$PLANNED" =~ ^(0|[1-9][0-9]{0,5})$ ]] || die "planned GiB must be a whole number"
read -r VG POOL < <(pool_of_local_lvm) || true
[[ -n "${VG:-}" && -n "${POOL:-}" ]] || die "no lvmthin storage local-lvm in $STORAGE_CFG"
[[ "$VG" =~ ^[A-Za-z0-9_.+-]+$ && "$POOL" =~ ^[A-Za-z0-9_.+-]+$ ]] || die "unexpected vg/pool name"
pool_b=$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$POOL" 2>/dev/null | tr -d ' ')
[[ "$pool_b" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$POOL"
pool_gib=$(gib_floor "$pool_b")
# Every thin volume promised in the pool (its virtual size) — except the place itself and vzdump's short-lived
# snapshots (snap_*_vzdump), which exist only during a whole-box copy and are what the reserve is for.
# FAIL CLOSED: a failed or empty listing must never read as „nothing promised" (that would size the place too big).
listing=$(lvs --noheadings --nosuffix --units b -o lv_name,pool_lv,lv_size "$VG" 2>/dev/null) || die "cannot list the volumes of $VG"
pool_seen=false; promised_b=0; have_lv=false
while read -r name pl size; do
[[ -n "${name:-}" ]] || continue
[[ "$name" == "$POOL" ]] && pool_seen=true
[[ "$pl" == "$POOL" ]] || continue
if [[ "$name" == "$LV_NAME" ]]; then have_lv=true; continue; fi
[[ "$name" =~ ^snap_.*_vzdump$ ]] && continue
[[ "$size" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$name"
promised_b=$(( promised_b + size ))
done <<< "$listing"
$pool_seen || die "the listing of $VG does not contain the pool $POOL — refusing to guess what is promised"
promised_gib=$(( (promised_b + 1073741823) / 1073741824 )) # rounded UP: never promise a partial GiB twice
reserve=$(( pool_gib / 10 )); (( reserve < 16 )) && reserve=16; (( reserve > 48 )) && reserve=48
internal=$(( pool_gib - reserve - promised_gib - PLANNED ))
if [[ "$MODE" == plan ]]; then
echo "plan pool=$pool_gib reserve=$reserve promised=$promised_gib planned=$PLANNED internal=$internal exists=$have_lv"
exit 0
fi
DEV=/dev/$VG/$LV_NAME
is_block() { [[ "$(stat -L -c %F "$1" 2>/dev/null)" == "block special file" ]]; }
fs_uuid() { blkid -p -s UUID -o value "$DEV" 2>/dev/null; }
fs_type() { blkid -p -s TYPE -o value "$DEV" 2>/dev/null; }
fs_label() { blkid -p -s LABEL -o value "$DEV" 2>/dev/null; }
# has_signature: FAIL CLOSED. blkid -p exits 0 when it found something and 2 when it found NOTHING; any other result (an
# error, an ambivalent probe, blkid missing) must never read as „empty" — that is the only gate in front of mkfs.
has_signature() {
local rc=0
blkid -p -o value "$DEV" >/dev/null 2>&1 || rc=$?
case $rc in
0) return 0 ;;
2) return 1 ;;
*) die "cannot probe $DEV for a signature (blkid exit $rc) — refusing" ;;
esac
}
lv_gib() { gib_floor "$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$LV_NAME" 2>/dev/null | tr -d ' ')"; }
if $have_lv; then
is_block "$DEV" || die "$DEV is not an active block device"
t=$(fs_type)
if [[ -n "$t" ]]; then
[[ "$t" == ext4 && "$(fs_label)" == "$LABEL" ]] || die "$DEV holds something that is not Felhom's ($t) — left alone"
echo "existing uuid=$(fs_uuid) gib=$(lv_gib)"; exit 0
fi
# A felhom-belso with NO signature: a crash between lvcreate and mkfs. Nothing can be on it — format it.
has_signature && die "$DEV carries a signature blkid cannot name — left alone"
else
(( internal >= MIN_GIB )) || { echo "skipped only $internal GiB free for it (pool $pool_gib, reserve $reserve, promised $promised_gib, planned $PLANNED; needs $MIN_GIB)"; exit 0; }
lvcreate -q -y -V "${internal}G" -T "$VG/$POOL" -n "$LV_NAME" >/dev/null || die "lvcreate failed"
is_block "$DEV" || die "$DEV did not appear after lvcreate"
has_signature && die "the new $DEV already carries a signature — left alone"
made=created
fi
mkfs.ext4 -q -L "$LABEL" "$DEV" >/dev/null || die "mkfs.ext4 failed on $DEV"
u=$(fs_uuid)
[[ "$u" =~ ^[0-9a-fA-F-]{36}$ ]] || die "no filesystem UUID after mkfs on $DEV"
echo "${made:-formatted} uuid=$u gib=$(lv_gib)"
+20 -8
View File
@@ -58,14 +58,26 @@ fi
# 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under
# /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed.
while read -r mp; do
[[ -z "$mp" ]] && continue
case "$mp" in
/mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
*) die "device (or a partition) is mounted at $mp ($dev)" ;;
esac
done < <(lsblk -nro MOUNTPOINT "$whole" 2>/dev/null || true)
# Agent v0.158.0: read from the kernel's mount table for EVERY node of the disk (lsblk's MOUNTPOINT shows one mount
# per node; a bind elsewhere hid behind it), and a disk whose nodes cannot be listed REFUSES (it passed before).
# Deliberately NO exemption for a sibling that is Felhom's own enrolled drive (raw /mnt/<name> + /mnt/felhom-drives):
# the only corroboration available here is a mount the agent itself can make, so trusting it would let a compromised
# agent unlock a host owner's disk. Consequence (felhom.eu `16` §13): the free part next to an in-use Felhom drive
# cannot be formatted; the dashboard does not offer it.
PROC_MOUNTS=/proc/mounts
node_mounts() { awk -v d="$1" '$1 == d { print $2 }' "$PROC_MOUNTS"; }
disk_nodes="$(lsblk -nrpo NAME "$whole" 2>/dev/null || true)"
[[ -n "$disk_nodes" ]] || die "cannot list the nodes of $whole — refusing"
while read -r node; do
[[ -z "$node" ]] && continue
while read -r mp; do
[[ -z "$mp" ]] && continue
case "$mp" in
/mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
*) die "device (or a partition) is mounted at $mp ($node)" ;;
esac
done < <(node_mounts "$node")
done <<< "$disk_nodes"
# 4) read-only device (audit D1): a device the kernel marks RO is never a formattable data disk.
wbase="${whole#/dev/}"
rof="/sys/block/$wbase/ro"
+5 -3
View File
@@ -16,8 +16,10 @@ Cmnd_Alias FELHOM_OP_REPAIR = \
/usr/bin/systemctl reset-failed felhom-sshd, \
/usr/bin/systemctl restart felhom-sshd, \
/usr/sbin/pct list, \
/usr/sbin/pct start [0-9]*, \
/usr/sbin/pct stop [0-9]*, \
/usr/sbin/pct unlock [0-9]*
/usr/sbin/pct ^start [0-9]+$, \
/usr/sbin/pct ^stop [0-9]+$, \
/usr/sbin/pct ^unlock [0-9]+$
# R-861 (b) B2 (`09` §3 decision 165, hygiene): one numeric vmid per pct verb, anchored — the old glob `[0-9]*` also
# matched spaces, so `pct stop 9201 --skiplock 1` passed. Pinned by TestFelhomOpSudoersPctIsExact.
felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR
+1324 -28
View File
File diff suppressed because it is too large. Load diff
+128 -1
View File
@@ -13,10 +13,14 @@ longer name the destination: each verb has a fixed source and a fixed (or strict
Verbs (each one sudoers line, exact-match pattern):
unit <name> /var/lib/felhom-agent/units/<name> -> /etc/systemd/system/<name> (.mount | .automount)
unit-forget <name> REMOVES /etc/systemd/system/<name> (R-1124): only a felhom by-UUID drive .mount unit that is
disabled, not mounted, and whose drive is absent (its /dev/disk/by-uuid link is gone)
dnsmasq <tmp> <name> /tmp/felhom-resolver-<digits>.conf -> /etc/dnsmasq.d/felhom-<...>.conf
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
sshd-key /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op -> /etc/felhom-sshd/authorized_keys/felhom-op
controller-image <vmid> the ref on STDIN -> /etc/felhom-controller-image INSIDE guest <vmid> (R-861 (a) A1): only
our registry + our repository + an x.y.z tag; the agent no longer has a `tee` grant
--self-check prints "felhom-priv-apply ok verbs=..." (the bundle's self-check)
Exit codes: 0 installed (or already identical), 2 usage, 3 refused (content or source), 4 install failed.
@@ -39,9 +43,21 @@ WG_SRC, WG_DEST = STATE + "/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
MAX_BYTES = 64 * 1024
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key")
VERBS = ("unit", "unit-forget", "dnsmasq", "wg", "sshd-config", "sshd-key", "controller-image")
# R-861 (a) A1 (`09` §3 decision 165): the SAME pattern as the agent's controllerImageRe (internal/localapi/
# controllerswap.go) — a compromised agent cannot hand the guest's bootstrap any other image. Pinned by
# configs/test_felhom_priv_apply.py ControllerImage.
CONTROLLER_IMAGE_RE = re.compile(r"^gitea\.dooplex\.hu/admin/felhom-controller:[0-9]+\.[0-9]+\.[0-9]+$")
CONTROLLER_IMAGE_FILE = "/etc/felhom-controller-image"
CONTROLLER_IMAGE_MAX = 256
VMID_RE = re.compile(r"^[0-9]{1,9}$")
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
DRIVE_UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.mount$")
DRIVE_UNIT_MARKER = "Managed by felhom-agent"
NET_UNIT_MARKER = "felhom network storage"
WANTS_DIR = UNIT_DIR + "/multi-user.target.wants"
BY_UUID_DIR = "/dev/disk/by-uuid"
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
DNSMASQ_NAME_RE = re.compile(r"^felhom-[a-z0-9][a-z0-9._-]*\.conf$")
SEG = r"[A-Za-z0-9_-][A-Za-z0-9_.-]*"
@@ -123,6 +139,46 @@ class Host:
pass
raise
def read_stdin(self, limit):
return sys.stdin.buffer.read(limit + 1)
# ---- unit-forget (R-1124) ----
def read_installed(self, path):
"""The installed unit: None when absent; Refused when it is not a regular file (a symlink is never followed)."""
try:
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC)
except FileNotFoundError:
return None
except OSError as e:
raise Refused("F1", f"cannot open {path}: {e.strerror}")
with os.fdopen(fd, "rb") as f:
if not stat.S_ISREG(os.fstat(f.fileno()).st_mode):
raise Refused("F1", f"{path} is not a regular file")
return f.read(MAX_BYTES + 1)
def lexists(self, path):
return os.path.lexists(path)
def mountpoints(self):
"""Every mount point in /proc/self/mountinfo (field 5, octal escapes decoded)."""
out = set()
with open("/proc/self/mountinfo", "r", encoding="utf-8", errors="replace") as f:
for line in f:
parts = line.split(" ")
if len(parts) > 4:
out.add(re.sub(r"\\([0-7]{3})", lambda m: chr(int(m.group(1), 8)), parts[4]))
return out
def remove(self, path):
os.remove(path)
def write_guest_image(self, vmid, data):
"""As root: `pct exec <vmid> -- tee <the fixed file>` with the checked ref on stdin (no shell)."""
r = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", CONTROLLER_IMAGE_FILE],
input=data, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=60)
if r.returncode != 0:
raise OSError(f"pct exec {vmid} tee exited {r.returncode}: {r.stderr.decode(errors='replace').strip()[:200]}")
def log(self, line):
print(line, file=sys.stderr)
try:
@@ -377,6 +433,73 @@ def plan(argv):
raise Refused("A1", f"wrong arguments for {v}")
def controller_image(rest, host):
"""R-861 (a) A1: read the ref on stdin, check it, write it INSIDE the guest as root."""
try:
if len(rest) != 1 or not VMID_RE.match(rest[0]):
raise Refused("A1", "usage: felhom-priv-apply controller-image <vmid> (the ref on stdin)")
raw = host.read_stdin(CONTROLLER_IMAGE_MAX)
if len(raw) > CONTROLLER_IMAGE_MAX:
raise Refused("I1", f"the image ref is longer than {CONTROLLER_IMAGE_MAX} bytes")
try:
text = raw.decode("ascii")
except UnicodeDecodeError:
raise Refused("I1", "the image ref is not ASCII")
ref = text[:-1] if text.endswith("\n") else text
if not CONTROLLER_IMAGE_RE.match(ref) or "\n" in ref:
raise Refused("I1", "the image ref is not gitea.dooplex.hu/admin/felhom-controller:<x.y.z>")
except Refused as e:
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] controller-image {' '.join(rest)[:40]}: {e.reason}")
return 2 if e.rule == "A1" else 3
vmid = int(rest[0])
try:
host.write_guest_image(vmid, (ref + "\n").encode())
except (OSError, subprocess.SubprocessError) as e:
host.log(f"felhom-priv-apply: FAILED controller-image {vmid}: {e}")
return 4
host.log(f"felhom-priv-apply: WROTE controller-image {vmid} {ref}")
return 0
def unit_forget(rest, host):
"""R-1124: remove ONE felhom drive mount unit that the agent has already disabled, whose drive is gone. A drive that
is still present or still mounted keeps its unit — this verb can only forget, never unmount or take a drive away."""
try:
if len(rest) != 1 or not DRIVE_UNIT_NAME_RE.match(rest[0]) or "/" in rest[0]:
raise Refused("A1", "usage: felhom-priv-apply unit-forget mnt-<escaped path>.mount")
name = rest[0]
path = os.path.join(UNIT_DIR, name)
data = host.read_installed(path)
if data is None:
host.log(f"felhom-priv-apply: ABSENT unit-forget {path} (nothing to remove)")
return 0
text = text_of(data, path)
if DRIVE_UNIT_MARKER not in text or NET_UNIT_MARKER in text:
raise Refused("F2", f"{name} is not a felhom drive unit")
check_unit(name, text)
s = parse_ini(text, name)
what, where = s["Mount"].get("What", ""), s["Mount"]["Where"]
mu = re.match(r"^/dev/disk/by-uuid/(.+)$", what)
if not mu:
raise Refused("F2", f"{name} is not a by-UUID drive unit")
if host.lexists(os.path.join(WANTS_DIR, name)):
raise Refused("F3", f"{name} is still enabled (disable it first)")
if where in host.mountpoints():
raise Refused("F4", f"{where} is mounted")
if host.lexists(os.path.join(BY_UUID_DIR, mu.group(1))):
raise Refused("F5", f"the drive of {name} is present (filesystem UUID {mu.group(1)})")
except Refused as e:
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] unit-forget {' '.join(rest)[:120]}: {e.reason}")
return 2 if e.rule == "A1" else 3
try:
host.remove(path)
except OSError as e:
host.log(f"felhom-priv-apply: FAILED unit-forget {path}: {e}")
return 4
host.log(f"felhom-priv-apply: REMOVED unit-forget {path} (where={where})")
return 0
def main(argv, host=None):
host = host or Host()
if argv == ["--self-check"]:
@@ -396,6 +519,10 @@ def main(argv, host=None):
return 3
print("OK")
return 0
if argv and argv[0] == "controller-image":
return controller_image(argv[1:], host)
if argv and argv[0] == "unit-forget":
return unit_forget(argv[1:], host)
try:
verb, src, dest, mode, checker = plan(argv)
data = host.read_source(src)
+7
View File
@@ -0,0 +1,7 @@
# MANAGED BY FELHOM (the agent's config bundle) — R-982, operator ruling D4 (2026-10-22).
# The Proxmox host's OWN sshd — the listener the household network reaches on port 22: root logs in with a KEY only.
# The root password still works on the console (the break-glass password: felhom.eu runbooks/break-glass.md). The
# operator's door (felhom-sshd, port 8822, /etc/felhom-sshd/sshd_config) is a separate daemon whose config includes
# nothing from here, so it is not changed. Debian's sshd_config reads this folder BEFORE its own settings, and for
# sshd the first value wins; the bundle checks the effective value with `sshd -T` after the reload.
PermitRootLogin prohibit-password
+74
View File
@@ -0,0 +1,74 @@
#!/bin/bash
# felhom-thin-check — list the thin device ids a thin pool holds next to the ones LVM knows (R-1012). READ-ONLY: it
# names an orphan, it never removes one. Installed as /usr/local/sbin/felhom-thin-check (config bundle).
#
# felhom-thin-check <vg> <pool>
#
# A full host root while vzdump takes its snapshot can leave a thin snapshot inside the pool that LVM no longer knows
# (demo-hp 2026-10-14: `thin_ls` device 4, LVM only 1 and 2); from then on every snapshot backup of that guest fails
# („Creation of new snapshot 4 of device 2 failed") and nothing said why. The repair (`dmsetup message … delete <id>`)
# stays a HUMAN step, decided by the operator — this helper only reads.
#
# Reading the live pool's device list needs a metadata snapshot: the kernel's documented way to read metadata while
# the pool is in use (`reserve_metadata_snap`, `thin_ls --metadata-snap`, `release_metadata_snap`). It changes no data
# and no mapping; it is held for the length of one `thin_ls -o DEV` (well under a second on these pools). Rules:
# 1. the arguments are an LVM name pair; <vg>/<pool> is a thin pool (lv_attr starts with `t`);
# 2. only one run at a time (a lock file); a metadata snapshot that someone ELSE holds is never touched — the run
# refuses (exit 3) and does not release it;
# 3. LVM's list is read before AND after the pool's, so a volume created in between is not reported as an orphan
# (the caller treats an id as an orphan only when it is in the pool and in neither LVM list).
# Output, one record per line: lvm-before <id> <lv> | pool <id> | lvm-after <id> <lv> | ok
# Exit: 0 read, 1 refused (arguments / not a thin pool), 3 a metadata snapshot is already held or cannot be reserved,
# 4 the pool could not be read. Pinned by configs/test_felhom_thin_check.py.
set -u
LVS=/usr/sbin/lvs
DMSETUP=/usr/sbin/dmsetup
THIN_LS=/usr/sbin/thin_ls
LOCK=/run/lock/felhom-thin-check.lock
NAME_RE='^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0,63}$'
die() { echo "felhom-thin-check: REFUSED: $1" >&2; exit "${2:-1}"; } # $1 the reason, $2 the exit code
lvm_ids() { # $1 tag → "<tag> <id> <lv>" for every thin volume of this pool
"$LVS" --noheadings --separator , -o lv_name,thin_id --select "pool_lv=$POOL" -- "$VG" 2>/dev/null |
awk -F, -v tag="$1" '{ gsub(/ /, ""); if ($2 ~ /^[0-9]+$/) print tag, $2, $1 }'
}
release() { "$DMSETUP" message "$TPOOL" 0 release_metadata_snap >/dev/null 2>&1; }
main() {
[[ $# -eq 2 ]] || die "usage: felhom-thin-check <vg> <pool>"
VG=$1 POOL=$2
local attr held ids
[[ "$VG" =~ $NAME_RE && "$POOL" =~ $NAME_RE ]] || die "not an LVM name pair"
attr=$("$LVS" --noheadings -o lv_attr -- "$VG/$POOL" 2>/dev/null | tr -d ' ')
[[ "$attr" == t* ]] || die "$VG/$POOL is not a thin pool"
DM="${VG//-/--}-${POOL//-/--}"
TPOOL="$DM-tpool"
TMETA="/dev/mapper/${DM}_tmeta"
exec 9>"$LOCK" || die "cannot open the lock file" 4
flock -w 30 9 || die "another check holds the lock" 3
# Field 7 of the thin-pool status is the held metadata root: "-" when no metadata snapshot exists.
held=$("$DMSETUP" status "$TPOOL" 2>/dev/null | awk '$3 == "thin-pool" { print $7 }')
[[ -n "$held" ]] || die "cannot read the status of $TPOOL" 4
[[ "$held" == "-" ]] || die "a metadata snapshot of $TPOOL is already held (by someone else) — not touched" 3
lvm_ids lvm-before
"$DMSETUP" message "$TPOOL" 0 reserve_metadata_snap >/dev/null 2>&1 || die "could not reserve a metadata snapshot of $TPOOL" 3
trap release EXIT
trap 'release; trap - EXIT; exit 4' INT TERM HUP
if ! ids=$("$THIN_LS" --metadata-snap --no-headers -o DEV "$TMETA" 2>/dev/null); then
die "thin_ls could not read $TMETA" 4
fi
release
trap - EXIT INT TERM HUP
awk '$1 ~ /^[0-9]+$/ { print "pool", $1 }' <<<"$ids"
lvm_ids lvm-after
echo ok
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then main "$@"; fi
+99
View File
@@ -0,0 +1,99 @@
#!/bin/bash
# felhom-vzdump-partial-clean — remove ONE dead, half-written vzdump backup (R-1006). The only delete the agent's
# sudoers allows for backup files. Installed as /usr/local/sbin/felhom-vzdump-partial-clean (config bundle).
#
# felhom-vzdump-partial-clean <storage> <vmid> <name>
#
# A power cut during the night backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` (+ its `.tmp/` directory) on the
# storage; nothing removes it, every later backup is then skipped „not enough space", and a full host root breaks
# LVM's own metadata writes (power-cut-fixes-2026-10-14 C4). This helper removes such a leftover — and NOTHING else.
# It trusts nothing the agent says; every rule is checked here, as root, and any doubt REFUSES (exit 1):
# 1. the arguments: a storage id, a vmid, and a name that is exactly
# vzdump-(lxc|qemu)-<THAT vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp) — never a finished archive
# (.tar.zst, .vma.zst, …), never a .log/.notes, never another vmid;
# 2. the storage: its section in /etc/pve/storage.cfg is `dir: <storage>` with a `path`; the target is
# <path>/dump/<name>, not a symlink, and its real directory IS <path>/dump;
# 3. not in use: no vzdump runs on the host at all (a task `…:vzdump:…` or a `vzdump` command — a job for several
# guests carries no vmid in its task name), and no
# process holds the target (or, for the .tmp directory, anything under it) open or as its working directory
# (C2: after a killed worker, tar and zstd kept writing into the .tar.dat for 4 min 40 s, none named „vzdump");
# 4. dead: last modified before this boot (`btime` in /proc/stat), or untouched for at least QUIET_S seconds.
# Prints one line on success: „removed <name> <bytes> B (<why>)". Pinned by configs/test_felhom_vzdump_partial_clean.py.
set -u
STORAGE_CFG=/etc/pve/storage.cfg
PROC=/proc
QUIET_S=120
die() { echo "felhom-vzdump-partial-clean: REFUSED: $*" >&2; exit 1; }
storage_path() { # $1 storage id → prints the dir storage's path, or nothing
awk -v want="$1" '
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (id == want); next }
in_s && /^[ \t]+path[ \t]/ { if (type == "dir") { sub(/^[ \t]+path[ \t]+/, ""); sub(/[ \t]+$/, ""); print; exit } }
' "$STORAGE_CFG"
}
storage_type() {
awk -v want="$1" '/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); id = h[2]; sub(/[ \t]+$/, "", id); if (id == want) { print h[1]; exit } }' "$STORAGE_CFG"
}
boot_time() { awk '$1 == "btime" { print $2 }' "$PROC/stat"; }
newest_mtime() { # a file's mtime, or the newest mtime of anything in a directory (the dir itself included)
if [[ -d "$1" ]]; then find "$1" -xdev -printf '%T@\n' 2>/dev/null | sort -n | tail -1 | cut -d. -f1
else stat -c %Y "$1"; fi
}
busy_reason() { # $1 vmid, $2 target → prints why it is in use, or nothing
local vmid=$1 target=$2 p link cmd
for p in "$PROC"/[0-9]*; do
[[ -r "$p/cmdline" ]] || continue
cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)
# ANY vzdump on the host refuses: a job for several guests runs as `task UPID:…:vzdump::…` (no vmid in it), and
# its tar may be between this guest's files. Waiting costs one backup run; deleting a live one costs the backup.
if [[ "$cmd" == *":vzdump:"* ]] || [[ "$cmd" =~ (^|/)vzdump([[:space:]]|$) ]]; then
echo "a vzdump runs on this host (pid ${p##*/})"; return
fi
for link in "$p/cwd" "$p"/fd/*; do
local to
to=$(readlink "$link" 2>/dev/null) || continue
if [[ "$to" == "$target" || "$to" == "$target/"* ]]; then
echo "pid ${p##*/} has it open"; return
fi
done
done
}
main() {
local storage=${1:-} vmid=${2:-} name=${3:-}
[[ $# -eq 3 ]] || die "usage: felhom-vzdump-partial-clean <storage> <vmid> <name>"
[[ "$storage" =~ ^[A-Za-z][A-Za-z0-9_-]{0,63}$ ]] || die "bad storage id"
[[ "$vmid" =~ ^[1-9][0-9]{2,8}$ ]] || die "bad vmid"
[[ "$name" =~ ^vzdump-(lxc|qemu)-${vmid}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$ ]] \
|| die "name is not a half-written vzdump of guest $vmid: $name"
[[ "$(storage_type "$storage")" == "dir" ]] || die "storage $storage is not a dir storage"
local path; path=$(storage_path "$storage")
[[ "$path" == /* ]] || die "storage $storage has no path"
local dump="$path/dump" target="$path/dump/$name"
[[ -e "$target" ]] || die "no such entry: $target"
[[ ! -L "$target" ]] || die "a symlink: $target"
[[ "$(realpath -e "$dump")" == "$(realpath -e "$(dirname "$target")")" ]] || die "not directly in $dump"
if [[ "$name" == *.tmp ]]; then [[ -d "$target" ]] || die ".tmp is not a directory"
else [[ -f "$target" ]] || die "not a regular file"; fi
local busy; busy=$(busy_reason "$vmid" "$target")
[[ -z "$busy" ]] || die "in use: $busy"
local bt mt now why
bt=$(boot_time); mt=$(newest_mtime "$target"); now=$(date +%s)
[[ "$bt" =~ ^[0-9]+$ && "$mt" =~ ^[0-9]+$ ]] || die "cannot read the boot time or the modification time"
if (( mt < bt )); then why="written before this boot"
elif (( now - mt >= QUIET_S )); then why="untouched for $(( now - mt )) s, no vzdump for $vmid, nothing holds it"
else die "modified $(( now - mt )) s ago — may still be written"; fi
local size; size=$(du -sb "$target" 2>/dev/null | cut -f1)
if [[ -d "$target" ]]; then rm -rf --one-file-system -- "$target"; else rm -f -- "$target"; fi
[[ ! -e "$target" ]] || die "remove failed: $target"
echo "removed $name ${size:-?} B ($why)"
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
main "$@"
fi
+304 -2
View File
@@ -45,8 +45,10 @@ class St:
class Box:
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
def __init__(self, bundle_bytes, signed, oob=False, signers=True):
def __init__(self, bundle_bytes, signed, oob=False, signers=True, install_mode="appliance"):
self.files, self.modes, self.uids = {}, {}, {}
if install_mode is not None: # the installer's ROOT-OWNED record (D43 = A reads it for the sshd drop-in)
self.put(osapply.INSTALL_STATE, json.dumps({"completed": [], "mode": install_mode}).encode(), 0o644)
self.dirs = {osapply.OOB_DIR} if oob else set()
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
if signers:
@@ -429,6 +431,241 @@ class SelfCheckUndo(unittest.TestCase):
self.assert_restored(box, before, rep)
class RootKeyOnly(unittest.TestCase):
"""R-982, operator ruling D4 (2026-10-22): the bundle gives the host's own sshd a root-key-only drop-in, tests the
whole config and reloads sshd — only when the file was written; a drop-in that says anything but
`prohibit-password` is refused before anything is written.
RED-PROOF: with the after_install block removed, test_fresh_box_takes_the_dropin_and_reloads fails (no reload);
with the content rule removed, test_a_dropin_that_says_yes_or_no_is_refused fails — seen failing 2026-10-22."""
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
def test_fresh_box_takes_the_dropin_and_reloads(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertIn(b"PermitRootLogin prohibit-password", box.files[self.DEST])
self.assertEqual(box.modes[self.DEST], 0o644)
self.assertIn(["/usr/sbin/sshd", "-t"], box.calls)
self.assertIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
self.assertFalse(any("felhom-sshd" in " ".join(c) for c in box.calls), "the operator's door must not be touched")
def test_an_unchanged_dropin_is_not_reloaded(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put(self.DEST, (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes(), 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_dropin_that_says_yes_or_no_is_refused(self):
for bad in (b"PermitRootLogin yes\n", b"PermitRootLogin no\n", b"# nothing\n",
b"PermitRootLogin prohibit-password\nPermitRootLogin yes\n"):
def edit(files, bad=bad):
for e in files:
if e["path"] == self.DEST:
e["content_b64"] = base64.b64encode(bad).decode()
data, sha = edited_bundle(edit)
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertNotEqual(rc, 0, bad)
self.assertNotIn(self.DEST, box.files, bad)
# Review 2026-10-22 night. A2: the effective value is read back with `sshd -T`; a drop-in that is not in force is
# said in the log and the report (not refused — a refusal would freeze the bundle route). A3: a bundle undone
# after the reload reloads sshd again with the old state. RED-PROOF: with the sshd -T check removed,
# test_a_dropin_not_in_force_is_reported fails; with the undo reload removed, test_an_undone_bundle_reloads_sshd
# fails — seen failing 2026-10-22 night.
def _sshd_T(self, box, value):
orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
box.calls.append(argv)
return 0, f"port 22\npermitrootlogin {value}\npasswordauthentication yes\n", ""
return orig(argv, timeout, stdin)
box.host = host
def test_the_effective_value_is_checked(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "without-password")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(any(c[:2] == ["/usr/sbin/sshd", "-T"] for c in box.calls))
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
def test_a_dropin_not_in_force_is_reported(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "yes") # e.g. no Include of sshd_config.d
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
self.assertTrue(any("NOT in force" in l for l in box.logs), box.logs)
def test_an_undone_bundle_reloads_sshd(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.visudo_fail = True # the self-check fails AFTER the install → undo
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assertNotIn(self.DEST, box.files)
reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
class ByoKeepsItsSSH(unittest.TestCase):
"""D43 = A (operator 2026-10-24, R-1257): the root-key-only drop-in goes ONLY to a box whose root-owned install record
says appliance; on a BYO host nothing is written and a copy WE wrote is removed; a box with no record is left as it
is. X3: the in-force check asks sshd about each of the host's own LAN addresses. X4: a failing `sshd -t` puts back
only the SSH step. RED-PROOF (2026-10-24, felhom-agent f5c3d77 + these tests): with the drop-in's policy back to
`replace`, test_a_byo_host_gets_no_dropin and test_our_copy_comes_off_a_byo_host fail; with the probe back to
192.0.2.1 only, test_a_match_block_for_the_lan_is_not_in_force fails; with `sshd -t` back to `must`,
test_a_failing_sshd_t_undoes_only_the_ssh_step fails (rc 3, the whole bundle undone)."""
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
OURS = (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes()
def test_a_byo_host_gets_no_dropin(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertNotIn(["/usr/sbin/sshd", "-t"], box.calls)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
self.assertIn(self.DEST, rep["bundle"]["skipped"])
rec = json.loads(box.files[osapply.BUNDLE_RECORD])
self.assertEqual(rec["files"][self.DEST], "skipped")
def test_our_copy_comes_off_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
box.put(self.DEST, self.OURS, 0o644) # an earlier bundle wrote it (before D43)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertEqual(rep["bundle"]["removed"], [self.DEST])
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files),
"the previous copy must be kept for the undo")
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], "removed")
def test_the_owners_own_file_stays_on_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
box.put(self.DEST, b"# my own\nPermitRootLogin yes\n", 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files[self.DEST], b"# my own\nPermitRootLogin yes\n")
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_box_with_no_record_is_left_as_it_is(self):
for setup in ("none", "agent-writable"):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode=None if setup == "none" else "byo")
if setup == "agent-writable":
box.modes[osapply.INSTALL_STATE] = 0o666 # proves nothing
box.put(self.DEST, self.OURS, 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), self.OURS, setup)
self.assertIn(self.DEST, rep["bundle"]["skipped"], setup)
def _fake(self, box, addrs=("192.168.0.50",), match=None, sshd_t_rc=0):
orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:3] == ["ip", "-4", "-o"]:
box.calls.append(argv)
return 0, "".join(f"2: vmbr0 inet {a}/24 brd x scope global vmbr0\\ valid_lft forever\n" for a in addrs), ""
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
box.calls.append(argv)
val = "yes" if match and f"addr={match}" in argv[-1] else "without-password"
return 0, f"port 22\npermitrootlogin {val}\n", ""
if argv == ["/usr/sbin/sshd", "-t"]:
box.calls.append(argv)
return (sshd_t_rc, "", "/etc/ssh/sshd_config line 9: Bad configuration option") if sshd_t_rc else (0, "", "")
return orig(argv, timeout, stdin)
box.host = host
# R-1285: the policy pass the agent runs right after every bundle install, with the wrapper NOW on disk. No
# signature (it only removes a file we wrote). RED-PROOF: make policy_pass return before the loop → our copy stays.
POLICY_PLAN = {"release_id": "bundle-policy-0.166.0", "layer": "host", "mode": "bundle-policy"}
def test_the_policy_pass_removes_our_copy_from_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644) # the old wrapper's bundle run left it
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertEqual(rep["bundle_policy"], {"install_mode": "byo", "removed": [self.DEST]}, rep)
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files))
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
def test_the_policy_pass_leaves_an_appliance_and_the_owners_file(self):
for mode, content in (("appliance", None), ("byo", b"# my own\nPermitRootLogin yes\n")):
data, sha = real_bundle()
box = Box(data, None, install_mode=mode)
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
body = content if content is not None else self.OURS
box.put(self.DEST, body, 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), body, mode)
self.assertEqual(rep["bundle_policy"]["removed"], [], mode)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, mode)
def test_the_policy_pass_puts_our_copy_back_when_sshd_t_fails(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644)
self._fake(box, sshd_t_rc=1)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), self.OURS)
self.assertEqual(rep["bundle_policy"]["removed"], [])
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_match_block_for_the_lan_is_not_in_force(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"), match="192.168.0.50")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
probed = [c[-1] for c in box.calls if c[:2] == ["/usr/sbin/sshd", "-T"]]
self.assertTrue(any("addr=192.168.0.50" in p for p in probed) and any("addr=10.77.0.4" in p for p in probed), probed)
self.assertFalse(any("addr=192.0.2.1" in p for p in probed), probed)
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
self.assertTrue(any("NOT in force for 192.168.0.50" in l for l in box.logs), box.logs)
def test_the_lan_check_passes_when_every_address_is_key_only(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
def test_a_failing_sshd_t_undoes_only_the_ssh_step(self):
for before in (None, b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982 old\nPermitRootLogin prohibit-password\n"):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
if before is not None:
box.put(self.DEST, before, 0o644)
self._fake(box, sshd_t_rc=255)
rc, rep = run(box)
self.assertEqual(rc, 0, rep) # the bundle stays
self.assertEqual(box.files.get(self.DEST), before, "only the SSH step is put back")
self.assertIn("/usr/local/sbin/felhom-os-apply", box.files, "the rest of the bundle stays")
self.assertIn("sshd -t failed", rep["bundle"].get("sshd_step_undone", ""), rep)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, "sshd never ran the refused state")
want = "absent" if before is None else hashlib.sha256(before).hexdigest()
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], want)
class TrustBootstrap(unittest.TestCase):
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
data, sha = real_bundle()
@@ -546,7 +783,10 @@ class Builder(unittest.TestCase):
r"/etc/felhom/[a-z.-]+)", text))
agent_writes = {"/usr/local/sbin/felhom-shared-parent", "/etc/systemd/system/felhom-shared-parent.service"}
trust = {osapply.TRUST_FILE, osapply.TRUST_SIGNERS, osapply.TRUST_SIGNERS + ".tmp", osapply.BUNDLE_RECORD}
missing = sorted(p for p in found if p not in osapply.BUNDLE_DESTS and p not in agent_writes | trust)
# Written by the appliance ISO's first boot (felhom.eu scripts/iso/felhom-bootstrap.sh), never by the installer:
# since installer 1.32.0 (R-275) the uninstall only NAMES them under KEPT.
iso_writes = {"/etc/felhom/.bootstrap-done", "/etc/felhom/appliance-pairing-code"}
missing = sorted(p for p in found if p not in osapply.BUNDLE_DESTS and p not in agent_writes | trust | iso_writes)
self.assertEqual(missing, [], "the installer writes these root files, but the bundle does not carry them")
# the limits drop-in is named through $AGENT_UNIT in the installer
self.assertIn("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", osapply.BUNDLE_DESTS)
@@ -667,5 +907,67 @@ class SelfupdateWrapperConfinement(unittest.TestCase):
self.assertEqual(p.returncode, 1, p.stderr)
self.assertIn("outside /var/lib/felhom-os-apply/agent-update", p.stderr)
_sb = importlib.machinery.SourceFileLoader("stepbuild", str(REPO / "scripts" / "build-step-bundle.py"))
_ss = importlib.util.spec_from_loader("stepbuild", _sb)
stepbuild = importlib.util.module_from_spec(_ss)
_sb.exec_module(stepbuild)
NEW_IN_0146 = {"/usr/local/sbin/felhom-priv-apply", "/var/lib/vz/snippets/felhom-guest-hook.sh",
"/usr/local/sbin/felhom-shared-parent.sh", "/etc/systemd/system/felhom-shared-parent.service"}
class StepBundle(unittest.TestCase):
"""R-880 (agent v0.146.1): an INSTALLED wrapper checks an incoming bundle's paths against its OWN table (R16), so a
release that adds paths needs a step bundle: the boxes' current bundle with only felhom-os-apply replaced.
RED-PROOF: deliver the full bundle to the old table → R16 (test_the_full_bundle_is_refused_by_an_old_table)."""
def old_world(self):
"""The base bundle an older wrapper (no R-861 paths) installed, and that wrapper's table."""
full = json.loads(builder.build("0.145.0"))
full["files"] = [e for e in full["files"] if e["path"] not in NEW_IN_0146]
old_wrapper = b'# the v0.145.0 wrapper stands in here\nBUNDLE_OP = "agent_config_update"\n'
for e in full["files"]:
if e["path"] == "/usr/local/sbin/felhom-os-apply":
e["content_b64"], e["sha256"] = base64.b64encode(old_wrapper).decode(), hashlib.sha256(old_wrapper).hexdigest()
base = (json.dumps(full, indent=1, sort_keys=True) + "\n").encode()
old_dests = {k: v for k, v in osapply.BUNDLE_DESTS.items() if k not in NEW_IN_0146}
return base, old_dests
def parse_with_table(self, data, dests, version):
saved = osapply.BUNDLE_DESTS
osapply.BUNDLE_DESTS = dests
try:
return osapply.Bundle(osapply.Apply(Box(b"{}", None), "")).parse(data, hashlib.sha256(data).hexdigest(), version)
finally:
osapply.BUNDLE_DESTS = saved
def test_the_full_bundle_is_refused_by_an_old_table(self):
_, old_dests = self.old_world()
full = builder.build("0.146.1")
with self.assertRaises(osapply.Refused) as cm:
self.parse_with_table(full, old_dests, "0.146.1")
self.assertEqual(cm.exception.code, "R16")
def test_the_step_bundle_is_accepted_by_the_old_table_and_changes_only_the_wrapper(self):
base, old_dests = self.old_world()
new_wrapper = (HERE / "felhom-os-apply").read_bytes()
step = stepbuild.build_step(base, "0.146.1-step1", new_wrapper)
ver, files = self.parse_with_table(step, old_dests, "0.146.1-step1")
self.assertEqual(ver, "0.146.1-step1")
b, s_ = json.loads(base), json.loads(step)
self.assertEqual(sorted(e["path"] for e in b["files"]), sorted(e["path"] for e in s_["files"]), "the paths must not change")
changed = [e["path"] for e, f in zip(sorted(b["files"], key=lambda x: x["path"]), sorted(s_["files"], key=lambda x: x["path"]))
if e != f]
self.assertEqual(changed, ["/usr/local/sbin/felhom-os-apply"], "exactly the wrapper changes")
installed = dict((d, c) for d, c, *_ in files)
self.assertEqual(installed["/usr/local/sbin/felhom-os-apply"], new_wrapper)
# and the NEW wrapper (now installed) knows every path the release's full bundle names
self.assertTrue({e["path"] for e in json.loads(builder.build("0.146.1"))["files"]} <= set(osapply.BUNDLE_DESTS))
def test_a_step_version_must_carry_a_suffix(self):
base, _ = self.old_world()
with self.assertRaises(SystemExit):
stepbuild.build_step(base, "0.146.1", b"x")
if __name__ == "__main__":
unittest.main()
+57
View File
@@ -139,5 +139,62 @@ class Guard(unittest.TestCase):
self.assertEqual(mode, 0o644)
class KernelStepCannotLeaveTheBoxOff(unittest.TestCase):
"""R-836 / `11` §5.11 Part B 4: a kernel step's planned reboot, one crash and one self-revert cannot add up to the
box staying off. The wrapper starts a step only when the guard is armed with NO unclean boot in its window
(felhom-os-apply Kernel.check_guard, R21); the planned reboot and the self-revert are orderly (`systemctl reboot` —
the clean-stop marker); so the step adds at most ONE unclean boot, and the box stays off only after the LIMIT-th
(3rd) within the hour. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
def setUp(self):
self.d = tempfile.TemporaryDirectory()
self.e = FakeEnv(self.d.name)
cg.main(["x", "boot"], self.e)
s = self.e.state()
self.assertTrue(s["armed"])
self.assertEqual(s["unclean_boots_in_window"], 0, "the wrapper's precondition (R21)")
def tearDown(self):
self.d.cleanup()
def planned(self, minutes):
cg.main(["x", "clean-stop"], self.e)
self.e.t += minutes * 60
cg.main(["x", "boot"], self.e)
def crash(self, minutes):
self.e.t += minutes * 60
cg.main(["x", "boot"], self.e)
def test_planned_reboot_one_crash_one_self_revert(self):
self.planned(2) # the step's one-shot reboot (orderly)
self.crash(3) # the new kernel crashes after the guard ran; the box restarts (panic=10)
self.planned(2) # the self-revert (orderly)
s = self.e.state()
self.assertFalse(s["tripped"], s)
self.assertTrue(s["armed"])
self.assertEqual(self.e.panic(), 10, "the box still restarts after a crash")
self.assertEqual(s["unclean_boots_in_window"], 1, "the step added exactly one unclean boot")
def test_a_panic_before_userspace_is_not_even_counted(self):
# the one-shot kernel panics before the guard's unit runs (measured: rdinit= and init= missing): the planned
# reboot's clean-stop marker is still there when the old kernel boots, so this boot counts as clean.
cg.main(["x", "clean-stop"], self.e)
self.e.t += 120 # the panicking boot: no userspace, the guard never ran
cg.main(["x", "boot"], self.e)
s = self.e.state()
self.assertEqual(s["unclean_boots_in_window"], 0, s)
self.assertEqual(self.e.panic(), 10)
def test_the_box_stays_off_only_after_two_more_crashes_than_the_step_makes(self):
self.planned(2)
self.crash(3) # the step's one crash
self.planned(2) # the self-revert
self.crash(5) # an UNRELATED crash within the hour: the guard trips (the 3rd would leave it off)
s = self.e.state()
self.assertTrue(s["tripped"])
self.assertEqual(s["unclean_boots_in_window"], 2, "two unclean boots: one from the step, one not")
if __name__ == "__main__":
unittest.main()
+255
View File
@@ -0,0 +1,255 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-internal-place (felhom.eu architecture/16 §5, §7 — sizes ruled 2026-10-16).
The helper is run for real under bash; `lvs`, `lvcreate`, `blkid`, `mkfs.ext4` and `stat` are PATH stubs that read and
write one JSON state file, so nothing on this machine is touched. Every refusal has a positive control: the same setup
without the one condition goes through — so a test cannot pass because the helper refuses everything.
Run: python3 configs/test_felhom_internal_place.py (also run by the Go suite, internal/internalplace)
"""
import json, os, pathlib, stat, subprocess, tempfile, unittest
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-internal-place"
GIB = 1 << 30
STUB_LVS = r'''#!/usr/bin/env python3
import json, os, sys
st = json.load(open(os.environ["IP_STATE"]))
if st.get("lvs_fail_listing") and "lv_name,pool_lv,lv_size" in sys.argv: sys.exit(5)
args = [a for a in sys.argv[1:] if not a.startswith("-") and a not in ("b",)]
fields = sys.argv[sys.argv.index("-o") + 1].split(",")
target = args[-1]
listing = target == st["vg"]
for lv in st["lvs"]:
full = st["vg"] + "/" + lv["name"]
if listing and st.get("hide_pool_in_listing") and lv.get("pool", "") == "":
continue # a listing that silently lost the pool row (the pool's own size stays readable)
if listing or target == full:
row = {"lv_name": lv["name"], "pool_lv": lv.get("pool", ""), "lv_size": str(lv["size"])}
print(" " + " ".join(row[f] for f in fields))
'''
STUB_LVCREATE = r'''#!/usr/bin/env python3
import json, os, sys
p = os.environ["IP_STATE"]; st = json.load(open(p))
a = sys.argv[1:]
size = int(a[a.index("-V") + 1].rstrip("G")) * (1 << 30)
vg, pool = a[a.index("-T") + 1].split("/")
st["lvs"].append({"name": a[a.index("-n") + 1], "pool": pool, "size": size, "active": True})
st.setdefault("calls", []).append("lvcreate " + " ".join(a))
if st.get("dirty_new"): st["fs"]["felhom-belso"] = {"TYPE": "", "RAW": "junk"}
json.dump(st, open(p, "w"))
'''
STUB_BLKID = r'''#!/usr/bin/env python3
# like blkid -p: exit 0 when something is found, 2 when nothing is; st["blkid_rc"] forces an error exit
import json, os, sys
st = json.load(open(os.environ["IP_STATE"]))
if st.get("blkid_rc"): sys.exit(st["blkid_rc"])
fs = st["fs"].get(os.path.basename(sys.argv[-1]), {})
if "-s" in sys.argv:
v = fs.get(sys.argv[sys.argv.index("-s") + 1], "")
if v: print(v)
sys.exit(0 if v else 2)
vals = [fs.get(k, "") for k in ("TYPE", "LABEL", "UUID", "RAW") if fs.get(k)]
if vals: print("\n".join(vals))
sys.exit(0 if vals else 2)
'''
STUB_MKFS = r'''#!/usr/bin/env python3
import json, os, sys
p = os.environ["IP_STATE"]; st = json.load(open(p))
dev = os.path.basename(sys.argv[-1])
st["fs"][dev] = {"TYPE": "ext4", "LABEL": sys.argv[sys.argv.index("-L") + 1], "UUID": "0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c"}
st.setdefault("calls", []).append("mkfs.ext4 " + " ".join(sys.argv[1:]))
json.dump(st, open(p, "w"))
'''
STUB_STAT = r'''#!/usr/bin/env python3
import json, os, sys
st = json.load(open(os.environ["IP_STATE"]))
name = os.path.basename(sys.argv[-1])
if any(lv["name"] == name and lv.get("active", True) for lv in st["lvs"]): print("block special file")
else: sys.exit(1)
'''
CFG = "dir: local\n\tpath /var/lib/vz\n\tcontent iso,backup\n\nlvmthin: local-lvm\n\tthinpool data\n\tvgname pve\n\tcontent rootdir,images\n"
class Box:
def __init__(self, pool_gib, lvs=(), fs=None, cfg=CFG, **extra):
self.t = tempfile.TemporaryDirectory()
r = pathlib.Path(self.t.name)
self.bin = r / "bin"; self.bin.mkdir()
for name, body in (("lvs", STUB_LVS), ("lvcreate", STUB_LVCREATE), ("blkid", STUB_BLKID),
("mkfs.ext4", STUB_MKFS), ("stat", STUB_STAT)):
f = self.bin / name; f.write_text(body); f.chmod(0o755)
self.cfg = r / "storage.cfg"; self.cfg.write_text(cfg)
self.state = r / "state.json"
st = {"vg": "pve", "lvs": [{"name": "data", "pool": "", "size": pool_gib * GIB}] + [dict(l) for l in lvs],
"fs": fs or {}}
st.update(extra)
self.state.write_text(json.dumps(st))
def run(self, *args):
env = {"PATH": f"{self.bin}:/usr/bin:/bin", "IP_STATE": str(self.state), "FELHOM_STORAGE_CFG": str(self.cfg)}
p = subprocess.run(["bash", str(HELPER), *args], env=env, capture_output=True, text=True)
return p.returncode, p.stdout.strip(), p.stderr.strip()
def st(self):
return json.loads(self.state.read_text())
def calls(self):
return self.st().get("calls", [])
def close(self):
self.t.cleanup()
def thin(name, gib, pool="data"):
return {"name": name, "pool": pool, "size": gib * GIB}
class Sizes(unittest.TestCase):
"""The operator's table (SPACE-CHECK §4, ruled 2026-10-16), as the helper computes it on a fresh box."""
def plan(self, pool, planned, lvs=()):
b = Box(pool, lvs)
try:
rc, out, err = b.run("plan", str(planned))
self.assertEqual(rc, 0, err)
self.assertEqual(b.calls(), [], "plan must change nothing")
return dict(kv.split("=") for kv in out.split()[1:])
finally:
b.close()
def test_the_ruled_table(self):
# disk → pool (GiB), data volume (planned = 32 + mp0), expected internal
for pool, mp0, reserve, internal in ((140, 50, 16, 42), (349, 100, 34, 183), (800, 130, 48, 590), (1720, 130, 48, 1510)):
got = self.plan(pool, 32 + mp0)
self.assertEqual((int(got["reserve"]), int(got["internal"])), (reserve, internal), (pool, got))
def test_promised_volumes_count_but_vzdump_snapshots_and_other_pools_do_not(self):
got = self.plan(349, 0, [thin("vm-9201-disk-0", 32), thin("vm-9201-disk-1", 250),
thin("snap_vm-9201-disk-1_vzdump", 250), thin("vm-500-disk-0", 40, pool="other")])
self.assertEqual((int(got["promised"]), int(got["internal"])), (282, 33), got)
class Create(unittest.TestCase):
def test_a_fresh_box_gets_the_place(self):
b = Box(349)
try:
rc, out, err = b.run("create", "132")
self.assertEqual(rc, 0, err)
self.assertRegex(out, r"^created uuid=[0-9a-f-]{36} gib=183$")
self.assertEqual(len([c for c in b.calls() if c.startswith("lvcreate")]), 1)
self.assertIn("-V 183G -T pve/data -n felhom-belso", b.calls()[0])
self.assertIn("-L felhom-belso", b.calls()[1])
finally:
b.close()
def test_too_little_space_makes_nothing(self):
b = Box(53) # demo-hp's 128 GB disk
try:
rc, out, err = b.run("create", str(32 + 50))
self.assertEqual(rc, 0, err)
self.assertTrue(out.startswith("skipped "), out)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_an_existing_place_is_found_never_formatted(self):
fs = {"felhom-belso": {"TYPE": "ext4", "LABEL": "felhom-belso", "UUID": "11111111-2222-3333-4444-555555555555"}}
b = Box(349, [thin("felhom-belso", 183)], fs)
try:
rc, out, err = b.run("create", "0")
self.assertEqual(rc, 0, err)
self.assertEqual(out, "existing uuid=11111111-2222-3333-4444-555555555555 gib=183")
self.assertEqual(b.calls(), [], "an existing place must never be created or formatted again")
finally:
b.close()
def test_a_foreign_filesystem_on_the_name_is_refused(self):
# positive control: test_an_existing_place_is_found_never_formatted (same LV, Felhom's label) goes through
for fs in ({"TYPE": "xfs", "LABEL": "felhom-belso", "UUID": "x"}, {"TYPE": "ext4", "LABEL": "films", "UUID": "y"}):
b = Box(349, [thin("felhom-belso", 183)], {"felhom-belso": fs})
try:
rc, out, err = b.run("create", "0")
self.assertEqual(rc, 1, (fs, out))
self.assertIn("REFUSED", err)
self.assertEqual(b.calls(), [], fs)
finally:
b.close()
def test_a_crash_between_lvcreate_and_mkfs_is_finished(self):
b = Box(349, [thin("felhom-belso", 183)], {})
try:
rc, out, err = b.run("create", "0")
self.assertEqual(rc, 0, err)
self.assertRegex(out, r"^formatted uuid=")
self.assertEqual([c.split()[0] for c in b.calls()], ["mkfs.ext4"])
finally:
b.close()
def test_an_unnamed_signature_is_refused(self):
b = Box(349, [thin("felhom-belso", 183)], {"felhom-belso": {"RAW": "something"}})
try:
rc, out, err = b.run("create", "0")
self.assertEqual(rc, 1, out)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_a_new_volume_that_is_not_empty_is_not_formatted(self):
b = Box(349, dirty_new=True)
try:
rc, out, err = b.run("create", "132")
self.assertEqual(rc, 1, out)
self.assertNotIn("mkfs", " ".join(b.calls()))
finally:
b.close()
def test_bad_arguments_and_no_local_lvm_are_refused(self):
b = Box(349, cfg="dir: local\n\tpath /var/lib/vz\n")
try:
self.assertEqual(b.run("create", "0")[0], 1)
self.assertEqual(b.calls(), [])
finally:
b.close()
b = Box(349)
try:
for args in (("create", "-1"), ("create", "1x"), ("wipe", "0"), ("create",)):
self.assertEqual(b.run(*args)[0], 1, args)
self.assertEqual(b.calls(), [])
finally:
b.close()
class FailsClosed(unittest.TestCase):
"""A read that fails must never read as „empty" or „nothing promised". Positive control: Create (same setups)."""
def test_a_failed_volume_listing_creates_nothing(self):
b = Box(349, lvs_fail_listing=True)
try:
rc, out, err = b.run("create", "132")
self.assertEqual(rc, 1, out)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_a_listing_without_the_pool_creates_nothing(self):
b = Box(349, [thin("vm-9201-disk-1", 250)], hide_pool_in_listing=True)
try:
self.assertEqual(b.run("create", "132")[0], 1)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_a_blkid_error_never_reads_as_empty(self):
for setup in ({"lvs": [thin("felhom-belso", 183)], "fs": {}}, {}):
b = Box(349, **({"lvs": setup["lvs"], "fs": setup["fs"]} if setup else {}), blkid_rc=4)
try:
rc, out, err = b.run("create", "0" if setup else "132")
self.assertEqual(rc, 1, (setup, out))
self.assertNotIn("mkfs", " ".join(b.calls()), setup)
finally:
b.close()
if __name__ == "__main__":
unittest.main()
+145
View File
@@ -0,0 +1,145 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-mkfs-guarded.sh — the ONLY mkfs path the agent's sudoers permits.
First tests of this wrapper (agent v0.158.0), written for partitions as places (felhom.eu architecture/16 §6). Every
mount of every node of the disk outside /mnt/felhom-drives refuses — INCLUDING a sibling that is Felhom's own drive
(its raw /mnt/<name> mount): the agent can make that corroborating mount itself, so the root wrapper must not trust
it (security review of 912f7c1). An unlistable disk refuses.
HOW IT RUNS ANYWHERE: the wrapper is copied with five literal host paths replaced (each replacement is asserted, so a
drifted wrapper fails here instead of silently testing nothing): /proc/mounts → a fake mount table, the block-device
test → true (no real /dev nodes here), the pvs candidates → a stub, /sys/block → a fake tree, the mkfs binaries → a
recorder. `lsblk` is a PATH stub answering the three calls the wrapper makes from one JSON disk description.
Every refusal has a positive control (the same disk without the one condition is formatted).
Run: python3 configs/test_felhom_mkfs_guarded.py (also run by the Go suite, internal/storage)
"""
import json, os, pathlib, subprocess, tempfile, unittest
WRAPPER = pathlib.Path(__file__).resolve().parent / "felhom-mkfs-guarded.sh"
STUB_LSBLK = r'''#!/usr/bin/env python3
import json, os, sys
st = json.load(open(os.environ["MG_STATE"]))
a = sys.argv[1:]; dev = a[-1]
nodes = st["nodes"] # [{"name": "/dev/sdb", "pk": "", "fstype": ""}, ...]
if a[0] == "-ndo" and a[1] == "PKNAME":
for n in nodes:
if n["name"] == dev and n.get("pk"): print(n["pk"])
elif a[0] == "-nrpo" and a[1] == "NAME":
if not st.get("lsblk_empty"):
for n in nodes: print(n["name"])
elif a[0] == "-nro" and a[1] == "FSTYPE":
for n in nodes: print(n.get("fstype", ""))
elif a[0] == "-nro" and a[1] == "MOUNTPOINT": # the pre-v0.158.0 wrapper's call: one mountpoint per node, like lsblk
mt = [l.split()[:2] for l in open(os.environ["MG_MOUNTS"]) if l.strip()]
for n in nodes:
mps = [m for d, m in mt if d == n["name"]]
print(mps[0] if mps else "")
'''
STUB_MKFS = '#!/bin/sh\necho "$0 $*" >> "$MG_CALLS"\n'
STUB_PVS = '#!/bin/sh\nprintf "%s\\n" $MG_PVS\n'
class Disk:
def __init__(self, nodes, mounts, ro="0", pvs="", lsblk_empty=False):
self.t = tempfile.TemporaryDirectory()
r = pathlib.Path(self.t.name)
self.bin = r / "bin"; self.bin.mkdir()
(self.bin / "lsblk").write_text(STUB_LSBLK); (self.bin / "lsblk").chmod(0o755)
for n in ("mkfs.ext4", "mkfs.xfs"):
(self.bin / n).write_text(STUB_MKFS); (self.bin / n).chmod(0o755)
(self.bin / "pvs").write_text(STUB_PVS); (self.bin / "pvs").chmod(0o755)
self.state = r / "state.json"
self.state.write_text(json.dumps({"nodes": nodes, "lsblk_empty": lsblk_empty}))
self.mounts = r / "mounts"
self.mounts.write_text("".join(f"{d} {m} ext4 rw 0 0\n" for d, m in mounts))
self.sys = r / "sys"
for n in nodes:
if not n.get("pk"):
d = self.sys / os.path.basename(n["name"]); d.mkdir(parents=True); (d / "ro").write_text(ro + "\n")
self.calls = r / "calls"
self.pvs = pvs
src = WRAPPER.read_text()
subs = [
("/proc/mounts", str(self.mounts)),
('[[ -b "$dev" ]]', "true"),
("for c in /usr/sbin/pvs /sbin/pvs; do", f"for c in {self.bin}/pvs; do"),
('rof="/sys/block/', f'rof="{self.sys}/'),
("exec /usr/sbin/mkfs.ext4", f"exec {self.bin}/mkfs.ext4"),
("exec /usr/sbin/mkfs.xfs ", f"exec {self.bin}/mkfs.xfs "),
]
for a, b in subs:
assert a in src, f"the wrapper no longer contains {a!r} — update this test's substitutions"
src = src.replace(a, b)
self.copy = r / "mkfs-guarded"; self.copy.write_text(src)
def run(self, dev, fstype="ext4"):
env = {"PATH": f"{self.bin}:/usr/bin:/bin", "MG_STATE": str(self.state), "MG_CALLS": str(self.calls),
"MG_MOUNTS": str(self.mounts),
"MG_PVS": self.pvs}
p = subprocess.run(["bash", str(self.copy), dev, fstype], env=env, capture_output=True, text=True)
made = self.calls.read_text().split("\n")[0] if self.calls.exists() else ""
return p.returncode, p.stderr.strip(), made
def close(self):
self.t.cleanup()
def disk_sdb(sdb1_fstype="ext4", sdb2_fstype=""):
return [{"name": "/dev/sdb", "pk": "", "fstype": ""},
{"name": "/dev/sdb1", "pk": "sdb", "fstype": sdb1_fstype},
{"name": "/dev/sdb2", "pk": "sdb", "fstype": sdb2_fstype}]
ROOT = ("/dev/sda2", "/")
OWN_SDB1 = [("/dev/sdb1", "/mnt/hdd_1"), ("/dev/sdb1", "/mnt/felhom-drives/hdd_1")]
class Partitions(unittest.TestCase):
def check(self, mounts, dev="/dev/sdb2", nodes=None, want_ok=True, **kw):
d = Disk(nodes or disk_sdb(), [ROOT] + mounts, **kw)
try:
rc, err, made = d.run(dev)
if want_ok:
self.assertEqual(rc, 0, err)
self.assertTrue(made.endswith(f"mkfs.ext4 -F {dev}"), made)
else:
self.assertEqual(rc, 1, (mounts, made))
self.assertIn("REFUSED", err)
self.assertEqual(made, "", "a refused format must not run mkfs")
return err
finally:
d.close()
def test_a_free_part_on_an_unmounted_disk_is_formatted(self):
self.check([]) # positive control for every refusal below
def test_the_free_part_next_to_felhoms_own_drive_is_refused(self):
# The agent could make the „corroborating" managed mount itself: the root wrapper does not trust it.
self.check(OWN_SDB1, want_ok=False)
def test_a_sibling_mounted_raw_only_is_refused(self):
self.check([("/dev/sdb1", "/mnt/hdd_1")], want_ok=False)
def test_a_sibling_mounted_elsewhere_is_refused_even_with_a_managed_mount(self):
for elsewhere in ("/srv/data", "/media/usb", "/mnt/a/b", "/home/x"):
self.check([("/dev/sdb1", elsewhere), ("/dev/sdb1", "/mnt/felhom-drives/hdd_1")], want_ok=False)
def test_the_target_itself_mounted_raw_is_refused(self):
self.check([("/dev/sdb2", "/mnt/films"), ("/dev/sdb2", "/mnt/felhom-drives/films")], want_ok=False)
def test_the_target_mounted_only_under_felhom_drives_is_formatted(self):
self.check([("/dev/sdb2", "/mnt/felhom-drives/films")])
def test_an_unlistable_disk_is_refused(self):
self.check([], want_ok=False, lsblk_empty=True)
def test_the_catastrophic_cases_still_refuse(self):
self.check([("/dev/sdb1", "/")], want_ok=False) # system disk
self.check([], nodes=disk_sdb(sdb1_fstype="LVM2_member"), want_ok=False) # a member signature
self.check([], pvs="/dev/sdb1", want_ok=False) # an LVM PV (pvs)
self.check([], ro="1", want_ok=False) # read-only
if __name__ == "__main__":
unittest.main()
File diff suppressed because it is too large. Load diff
+122
View File
@@ -56,6 +56,18 @@ class FakeHost:
def log(self, line):
self.logs.append(line)
# controller-image (R-861 (a) A1): the ref arrives on stdin and is written INSIDE the guest by root.
stdin = b""
guest_writes = None
def read_stdin(self, limit):
return self.stdin[:limit + 1]
def write_guest_image(self, vmid, data):
if self.guest_writes is None:
self.guest_writes = []
self.guest_writes.append((vmid, data))
LOCAL_UNIT = """# Managed by felhom-agent — do not edit by hand.
[Unit]
@@ -316,5 +328,115 @@ class Refuses(unittest.TestCase):
self.refused(h, ["wg", "/etc/shadow"], "A1")
class ControllerImage(unittest.TestCase):
"""R-861 (a) A1 (`09` §3 decision 165): the agent can no longer `tee` any image ref into the guest. The root verb
reads the ref on stdin, requires our registry + our repository + an x.y.z tag, and writes the guest file itself.
RED-PROOF: on the pre-A1 wrapper `controller-image` is not a verb (A1 usage, rc 2) — the accepted case fails."""
def go(self, ref, *argv):
h = FakeHost()
h.stdin = ref.encode() if isinstance(ref, str) else ref
return h, run(h, *(argv or ("controller-image", "9201")))
def test_our_controller_ref_is_written_in_the_guest(self):
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")
self.assertEqual(rc, 0, h.logs)
self.assertEqual(h.guest_writes, [(9201, b"gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")])
def test_a_foreign_image_is_refused(self):
for ref in ("docker.io/library/alpine:latest\n", "alpine\n",
"gitea.dooplex.hu/admin/felhom-controller:latest\n",
"gitea.dooplex.hu/admin/other:0.1.0\n",
"evil.example/admin/felhom-controller:0.301.0\n",
"gitea.dooplex.hu/admin/felhom-controller:0.301.0\nalpine\n",
"gitea.dooplex.hu/admin/felhom-controller:0.301.0 x\n",
"", "\n"):
h, rc = self.go(ref)
self.assertEqual(rc, 3, f"{ref!r} was accepted")
self.assertFalse(h.guest_writes, f"{ref!r} wrote the guest file")
self.assertTrue(any("[I1]" in l for l in h.logs), h.logs)
def test_oversize_stdin_is_refused(self):
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0" + " " * 300)
self.assertEqual(rc, 3)
self.assertFalse(h.guest_writes)
def test_vmid_must_be_numeric(self):
for argv in (("controller-image", "9201;id"), ("controller-image", "-1"), ("controller-image",),
("controller-image", "9201", "9202")):
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n", *argv)
self.assertIn(rc, (2, 3), argv)
self.assertFalse(h.guest_writes, argv)
def test_self_check_names_the_verb(self):
import io, contextlib
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
pa.main(["--self-check"])
self.assertIn("controller-image", buf.getvalue())
class UnitForget(unittest.TestCase):
"""R-1124: the agent forgets a decommissioned drive whose disk is gone — the root half removes its unit file, and
ONLY a felhom by-UUID drive unit that is disabled, not mounted, and whose /dev/disk/by-uuid link is absent.
RED-PROOF: on the pre-R-1124 wrapper `unit-forget` is not a verb (A1 usage, rc 2) — the removal case fails."""
NAME = "mnt-hdd_1.mount"
PATH = "/etc/systemd/system/mnt-hdd_1.mount"
UUID_LINK = "/dev/disk/by-uuid/91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae"
def host(self, text=LOCAL_UNIT, kind="file", links=(), mounted=()):
h = FakeHost()
h.installed = {} if text is None else {self.PATH: (kind, text.encode())}
h.links, h.mounted, h.removed = set(links), set(mounted), []
h.read_installed = lambda p: (None if p not in h.installed else
(_ for _ in ()).throw(pa.Refused("F1", f"{p} is not a regular file"))
if h.installed[p][0] != "file" else h.installed[p][1])
h.lexists = lambda p: p in h.links
h.mountpoints = lambda: set(h.mounted)
h.remove = lambda p: h.removed.append(p)
return h
def test_an_absent_disabled_unmounted_drive_unit_is_removed(self):
h = self.host()
self.assertEqual(run(h, "unit-forget", self.NAME), 0, h.logs)
self.assertEqual(h.removed, [self.PATH])
def test_already_gone_is_fine(self):
h = self.host(text=None)
self.assertEqual(run(h, "unit-forget", self.NAME), 0, h.logs)
self.assertEqual(h.removed, [])
def refused(self, h, rule, *argv):
rc = run(h, *(argv or ("unit-forget", self.NAME)))
self.assertIn(rc, (2, 3), h.logs)
self.assertEqual(h.removed, [], h.logs)
self.assertTrue(any(f"[{rule}]" in l for l in h.logs), h.logs)
def test_F5_a_present_drive_keeps_its_unit(self):
self.refused(self.host(links=[self.UUID_LINK]), "F5")
def test_F4_a_mounted_drive_keeps_its_unit(self):
self.refused(self.host(mounted=["/mnt/hdd_1"]), "F4")
def test_F3_an_enabled_unit_is_not_removed(self):
self.refused(self.host(links=["/etc/systemd/system/multi-user.target.wants/" + self.NAME]), "F3")
def test_F2_a_network_unit_or_a_foreign_unit_is_not_removed(self):
self.refused(self.host(text=LOCAL_UNIT.replace("Managed by felhom-agent", "Written by someone")), "F2")
net = LOCAL_UNIT.replace("# Managed by felhom-agent", "# Managed by felhom-agent felhom network storage")
self.refused(self.host(text=net), "F2")
def test_F1_a_symlinked_unit_is_not_followed(self):
self.refused(self.host(kind="symlink"), "F1")
def test_A1_names(self):
for argv in (("unit-forget", "mnt-hdd_1.automount"), ("unit-forget", "sshd.service"),
("unit-forget", "../mnt-hdd_1.mount"), ("unit-forget",), ("unit-forget", self.NAME, "x")):
self.refused(self.host(), "A1", *argv)
def test_U3_a_unit_whose_name_lies_about_where_is_not_removed(self):
self.refused(self.host(text=LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/mnt/other")), "U3")
if __name__ == "__main__":
unittest.main(verbosity=2)
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-thin-check (R-1012): it lists the thin pool's device ids next to LVM's and NEVER removes
anything. Fake `lvs`, `dmsetup` and `thin_ls` record every call; nothing real is touched. Each refusal has a positive
control (the healthy read), so a test cannot pass because the helper refuses everything.
Run: python3 configs/test_felhom_thin_check.py (also run by internal/thincheck's Go test)
"""
import os, pathlib, re, subprocess, tempfile, unittest
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-thin-check"
class Box:
def __init__(self, pool_ids=(1, 2), lvm=(("vm-9201-disk-0", 1), ("vm-9201-disk-1", 2)), attr="twi-aotz--",
held="-", reserve_rc=0, thin_ls_rc=0):
self.t = tempfile.TemporaryDirectory()
r = pathlib.Path(self.t.name)
self.log = r / "calls"
self.log.write_text("")
lvm_lines = "\\n".join(f" {n},{i}" for n, i in lvm)
pool_lines = "\\n".join(str(i) for i in pool_ids)
self.bins = {}
for name, body in {
"lvs": f'''case "$*" in *lv_attr*) echo " {attr}";; *thin_id*) printf "{lvm_lines}\\n";; esac''',
"dmsetup": f'''case "$1" in
status) echo "0 112975872 thin-pool 585 6860/262144 517080/882624 {held} rw discard_passdown queue_if_no_space - 1024";;
message) [ "$4" = reserve_metadata_snap ] && exit {reserve_rc}; exit 0;;
esac''',
"thin_ls": f'''[ {thin_ls_rc} -ne 0 ] && exit {thin_ls_rc}; printf "{pool_lines}\\n"''',
}.items():
p = r / name
p.write_text(f'#!/bin/bash\necho "{name} $*" >> "{self.log}"\n{body}\n')
p.chmod(0o755)
self.bins[name] = p
self.lock = r / "lock"
def run(self, *args):
script = (f'source "{HELPER}"; LVS="{self.bins["lvs"]}"; DMSETUP="{self.bins["dmsetup"]}"; '
f'THIN_LS="{self.bins["thin_ls"]}"; LOCK="{self.lock}"; main "$@"')
r = subprocess.run(["bash", "-c", script, "x", *args], capture_output=True, text=True)
return r.returncode, r.stdout, r.stderr
def calls(self):
return self.log.read_text().splitlines()
def messages(calls):
return [c.split()[-1] for c in calls if c.startswith("dmsetup message")]
class Reads(unittest.TestCase):
def test_healthy_pool_lists_both_sides_and_releases(self):
b = Box()
rc, out, err = b.run("pve", "data")
self.assertEqual(rc, 0, err)
self.assertEqual(out.splitlines(), ["lvm-before 1 vm-9201-disk-0", "lvm-before 2 vm-9201-disk-1",
"pool 1", "pool 2",
"lvm-after 1 vm-9201-disk-0", "lvm-after 2 vm-9201-disk-1", "ok"])
self.assertEqual(messages(b.calls()), ["reserve_metadata_snap", "release_metadata_snap"])
self.assertIn("thin_ls --metadata-snap --no-headers -o DEV /dev/mapper/pve-data_tmeta", b.calls())
def test_the_2026_10_14_orphan_is_listed(self):
rc, out, _ = Box(pool_ids=(1, 2, 4)).run("pve", "data")
self.assertEqual(rc, 0)
self.assertIn("pool 4", out.splitlines())
def test_dashed_names_map_to_the_device_mapper_name(self):
b = Box()
rc, _, err = b.run("felhom-scratch", "thin-pool")
self.assertEqual(rc, 0, err)
self.assertIn("dmsetup status felhom--scratch-thin--pool-tpool", b.calls())
class Refuses(unittest.TestCase):
def refused(self, box, rc_want, *args):
rc, out, err = box.run(*args)
self.assertEqual(rc, rc_want, (out, err))
self.assertNotIn("ok", out.splitlines())
return box.calls()
def test_arguments(self):
for args in (("pve",), ("pve", "data", "x"), ("pve;id", "data"), ("-pve", "data"), ("pve", "../data"),
("pve", "da ta")):
calls = self.refused(Box(), 1, *args)
self.assertEqual(calls, [], args)
def test_not_a_thin_pool(self):
calls = self.refused(Box(attr="Vwi-aotz--"), 1, "pve", "vm-9201-disk-1")
self.assertEqual(messages(calls), [])
def test_a_snapshot_someone_else_holds_is_never_touched(self):
b = Box(held="1234")
rc, _, err = b.run("pve", "data")
self.assertEqual(rc, 3)
self.assertEqual(messages(b.calls()), []) # neither reserve nor release
# the reason line ends with the reason, not the exit code (found live on demo-hp 2026-10-17)
self.assertTrue(err.strip().endswith("not touched"), err)
def test_reserve_failure_does_not_release(self):
calls = self.refused(Box(reserve_rc=1), 3, "pve", "data")
self.assertEqual(messages(calls), ["reserve_metadata_snap"])
def test_thin_ls_failure_still_releases(self):
calls = self.refused(Box(thin_ls_rc=1), 4, "pve", "data")
self.assertEqual(messages(calls), ["reserve_metadata_snap", "release_metadata_snap"])
class NeverDeletes(unittest.TestCase):
def test_the_only_pool_messages_are_reserve_and_release(self):
src = HELPER.read_text()
sent = re.findall(r'"\$DMSETUP" message "\$TPOOL" 0 (\S+)', src)
self.assertEqual(sorted(set(sent)), ["release_metadata_snap", "reserve_metadata_snap"])
for word in ("delete", "lvremove", "thin_repair", "create_thin", "create_snap", " rm "):
code = "\n".join(l for l in src.splitlines() if not l.lstrip().startswith("#"))
self.assertNotIn(word, code, word)
if __name__ == "__main__":
unittest.main(verbosity=2)
+198
View File
@@ -0,0 +1,198 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-vzdump-partial-clean (R-1006): it removes a DEAD half-written vzdump backup and refuses
everything else. Each refusal has a positive control: the same setup without the one condition is removed — so a
test cannot pass because the helper refuses everything. A fake storage.cfg and a fake /proc; nothing real is touched.
Run: python3 configs/test_felhom_vzdump_partial_clean.py (also run by internal/backup's Go test)
"""
import os, pathlib, subprocess, tempfile, time, unittest
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-vzdump-partial-clean"
BOOT = 1_791_967_481
NAME = "vzdump-lxc-9201-2026_10_14-10_42_25"
class Box:
"""A temp tree: a dir storage `local` (path vz), an lvmthin `local-lvm`, a dir storage `other`, and a fake /proc."""
def __init__(self):
self.t = tempfile.TemporaryDirectory()
r = pathlib.Path(self.t.name)
self.vz, self.other, self.nas, self.proc = r / "vz", r / "other", r / "nas", r / "proc"
for d in (self.vz / "dump", self.other / "dump", self.nas / "dump", self.proc):
d.mkdir(parents=True)
(self.proc / "stat").write_text(f"cpu 1 2 3\nbtime {BOOT}\n")
self.cfg = r / "storage.cfg"
self.cfg.write_text(f"dir: local\n\tpath {self.vz}\n\tcontent iso,backup\n\n"
f"lvmthin: local-lvm\n\tthinpool data\n\tvgname pve\n\n"
f"dir: other\n\tpath {self.other}\n\tcontent backup\n\n"
f"nfs: nas\n\tpath {self.nas}\n\tserver 192.0.2.1\n\texport /x\n\tcontent backup\n")
self.pid = 1000
def partial(self, name=NAME + ".tar.dat", mtime=BOOT - 100, where=None, size=4096):
p = (where or self.vz / "dump") / name
if name.endswith(".tmp"):
(p / "etc" / "vzdump").mkdir(parents=True)
(p / "etc" / "vzdump" / "pct.conf").write_text("x")
for q in [p / "etc" / "vzdump" / "pct.conf", p / "etc" / "vzdump", p / "etc", p]:
os.utime(q, (mtime, mtime))
else:
p.write_bytes(b"\0" * size)
os.utime(p, (mtime, mtime))
return p
def process(self, cmdline, fds=(), cwd=None):
self.pid += 1
d = self.proc / str(self.pid)
(d / "fd").mkdir(parents=True)
(d / "cmdline").write_bytes(b"\0".join(a.encode() for a in cmdline) + b"\0")
for i, f in enumerate(fds):
os.symlink(str(f), d / "fd" / str(i + 3))
os.symlink(str(cwd or "/"), d / "cwd")
def run(self, storage, vmid, name, quiet=120):
script = (f'source "{HELPER}"; STORAGE_CFG="{self.cfg}"; PROC="{self.proc}"; QUIET_S={quiet}; '
f'main "$@"')
r = subprocess.run(["bash", "-c", script, "x", storage, str(vmid), name], capture_output=True, text=True)
return r.returncode, r.stdout + r.stderr
class Removes(unittest.TestCase):
def test_dead_tar_dat_before_boot(self):
b = Box(); p = b.partial()
rc, out = b.run("local", 9201, p.name)
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("written before this boot", out)
def test_dead_tmp_dir_before_boot(self):
b = Box(); p = b.partial(NAME + ".tmp")
rc, out = b.run("local", 9201, p.name)
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
def test_quiet_partial_after_boot(self):
b = Box(); p = b.partial(mtime=int(time.time()) - 300)
rc, out = b.run("local", 9201, p.name)
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("untouched", out)
def test_qemu_vma_dat(self):
b = Box(); p = b.partial("vzdump-qemu-341-2026_10_14-10_42_25.vma.dat")
rc, out = b.run("local", 341, p.name)
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
class Keeps(unittest.TestCase):
"""Each case: (setup that must be REFUSED, the same setup that must be REMOVED)."""
def pair(self, refuse, control, quiet=120):
b = Box(); target, args = refuse(b)
rc, out = b.run(*args, quiet=quiet)
self.assertNotEqual(rc, 0, f"must refuse: {out}")
self.assertIn("REFUSED", out)
self.assertTrue(os.path.lexists(target), "a refused entry must still exist")
c = Box(); ctarget, cargs = control(c)
crc, cout = c.run(*cargs, quiet=quiet)
self.assertEqual(crc, 0, f"positive control must be removed: {cout}")
self.assertFalse(os.path.lexists(ctarget))
def test_complete_archive_of_same_vmid(self):
self.pair(lambda b: (b.partial(NAME + ".tar.zst"), ("local", 9201, NAME + ".tar.zst")),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_log_and_notes(self):
for suffix in (".log", ".tar.zst.notes"):
self.pair(lambda b, s=suffix: (b.partial(NAME + s), ("local", 9201, NAME + s)),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_partial_of_another_vmid(self):
other = "vzdump-lxc-9202-2026_10_14-10_42_25.tar.dat"
self.pair(lambda b: (b.partial(other), ("local", 9201, other)),
lambda b: (b.partial(other), ("local", 9202, other)))
def test_partial_on_a_non_dir_storage(self):
self.pair(lambda b: (b.partial(), ("local-lvm", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_partial_on_a_network_storage_with_a_path(self):
# an nfs storage HAS a path: only the „dir only" rule (both its guards) can refuse it
self.pair(lambda b: (b.partial(where=b.nas / "dump"), ("nas", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
def test_partial_on_another_storage_than_named(self):
self.pair(lambda b: (b.partial(where=b.other / "dump"), ("local", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
def test_unknown_storage(self):
self.pair(lambda b: (b.partial(), ("nosuch", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_vzdump_task_in_flight(self):
def refuse(b):
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzdump:9201:felhom-agent@pve!agent:"]); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzrestore:9202:root@pam:"]); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_any_vzdump_job_even_without_the_vmid(self):
# a job for several guests: `vzdump::` — and a vzdump for ANOTHER guest — both refuse
for cmd in (["task UPID:demo-hp:001:002:6ACF4071:vzdump::root@pam:"],
["task UPID:demo-hp:001:002:6ACF4071:vzdump:9202:felhom-agent@pve!agent:"]):
def refuse(b, c=cmd):
p = b.partial(); b.process(c); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzstart:9202:root@pam:"]); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_vzdump_command_for_the_vmid(self):
def refuse(b):
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdump", "9201", "--storage", "local"]); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdumpfoo", "9201"]); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_file_held_open_by_an_orphan_writer(self):
def refuse(b):
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[p]); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[b.vz / "dump" / "something-else"]); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_tmp_dir_with_a_process_inside(self):
def refuse(b):
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=p / "etc"); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=b.vz); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_recently_written_after_boot(self):
self.pair(lambda b: (b.partial(mtime=int(time.time()) - 10), ("local", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(mtime=int(time.time()) - 300), ("local", 9201, NAME + ".tar.dat")))
def test_symlink(self):
def refuse(b):
real = b.partial(where=b.vz)
link = b.vz / "dump" / (NAME + ".tar.dat")
os.symlink(real, link)
return link, ("local", 9201, link.name)
# quiet=0: the link was made just now, so only the symlink rule may refuse it
self.pair(refuse, lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")), quiet=0)
def test_near_miss_names(self):
for bad in (NAME + ".tar.dat.bak", "vzdump-lxc-92011-2026_10_14-10_42_25.tar.dat", NAME + ".tar.da",
"x" + NAME + ".tar.dat", NAME + ".TAR.DAT", "vzdump-lxc-9201-2026_10_14-10_42.tar.dat"):
self.pair(lambda b, n=bad: (b.partial(n), ("local", 9201, n)),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_path_traversal_and_bad_args(self):
b = Box(); p = b.partial()
for args in (("local", 9201, "../dump/" + p.name), ("local", 9201, ""), ("loc al", 9201, p.name),
("local", "9201x", p.name), ("local", 12, p.name)):
rc, out = b.run(*args)
self.assertNotEqual(rc, 0, f"{args}: {out}")
self.assertTrue(p.exists())
def test_wrong_kind(self):
def refuse(b): # a FILE named .tmp
p = b.vz / "dump" / (NAME + ".tmp"); p.write_bytes(b"x"); os.utime(p, (BOOT - 100, BOOT - 100)); return p, ("local", 9201, p.name)
self.pair(refuse, lambda b: (b.partial(NAME + ".tmp"), ("local", 9201, NAME + ".tmp")))
if __name__ == "__main__":
unittest.main()
+5 -4
View File
@@ -1,7 +1,8 @@
module gitea.dooplex.hu/admin/felhom-agent
go 1.25.0
go 1.27.4
require golang.org/x/crypto v0.52.0
require golang.org/x/sys v0.45.0 // indirect
require (
golang.org/x/crypto v0.58.0
golang.org/x/sys v0.49.0
)
+6 -6
View File
@@ -1,6 +1,6 @@
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
+2
View File
@@ -22,6 +22,8 @@ var (
ErrExpired = errors.New("authz: op expired")
// ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
ErrNotYetValid = errors.New("authz: op not yet valid")
// ErrWindowTooLong: expires_at - issued_at > MaxWindow (R-1280).
ErrWindowTooLong = errors.New("authz: op validity window too long")
// ErrReplay: the nonce was already recorded in the window.
ErrReplay = errors.New("authz: replay (nonce already seen)")
)
+8
View File
@@ -20,6 +20,11 @@ const Namespace = "felhom-op-v1"
// the validity window stays an honest upper bound.
const DefaultClockSkew = 2 * time.Minute
// MaxWindow is the longest validity window a signed op may carry (R-1280, D44 #22, operator 2026-10-25): a job signed
// with a long window by mistake would stay usable that long, and the nonce store would keep its record as long. The
// root wrapper (configs/felhom-os-apply, MAX_WINDOW_S) and felhom-opsign (-ttl) hold the same 24 h.
const MaxWindow = 24 * time.Hour
// KeyRole tags a pinned operator key (doc 04 §3 two-key model).
type KeyRole string
@@ -150,6 +155,9 @@ func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
}
if w := op.ExpiresAt.Sub(op.IssuedAt); w > MaxWindow {
return nil, fmt.Errorf("%w: issued_at=%s expires_at=%s (%s, the most is %s)", ErrWindowTooLong, op.IssuedAt, op.ExpiresAt, w, MaxWindow)
}
if now.After(op.ExpiresAt) {
return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
}
+20
View File
@@ -117,6 +117,26 @@ func TestVerify_RejectsPerStage(t *testing.T) {
rejects(t, err, ErrTarget)
})
// R-1280: a window longer than 24 h is refused even while it is current; exactly 24 h is accepted.
// RED-PROOF: drop the MaxWindow check → the 24 h + 1 s case verifies.
t.Run("window longer than 24 h", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-1", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow+time.Second))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
_, err := v.Verify(blob, sig)
rejects(t, err, ErrWindowTooLong)
})
t.Run("window of exactly 24 h is accepted", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-2", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
if _, err := v.Verify(blob, sig); err != nil {
t.Fatalf("a 24 h window must verify: %v", err)
}
})
t.Run("expired", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
@@ -241,3 +241,23 @@ func TestNewestArchiveTime_DistinctPhantomsEachAnnounced(t *testing.T) {
t.Errorf("got %d rejection lines for 2 distinct phantoms across 3 polls, want 2:\n%s", n, buf.String())
}
}
// R-99 (`09` §3 decision 140): the WARN for a PBS phantom ends with the cleanup runbook, so whoever sees it knows the
// one sanctioned way to remove it; a tiny archive on a dir storage is not a PBS phantom and gets no pointer.
// RED-PROOF: drop the `msg += phantomCleanupPointer` line → "the PBS phantom WARN does not end with the runbook pointer".
func TestRejectedArchiveWarnNamesTheCleanupRunbook(t *testing.T) {
var buf bytes.Buffer
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
t.Fatal(err)
}
const want = "INCOMPLETE archive when computing tier freshness — it is not a successful backup — a phantom leftover; delete it by felhom.eu documentation/runbooks/pbs-phantom-cleanup.md (09 §3 decision 140)"
if !strings.Contains(buf.String(), want) {
t.Errorf("the PBS phantom WARN does not end with the runbook pointer:\n%s", buf.String())
}
local := phantomEntry()
local.Format, local.VolID = "tar.zst", "local:backup/vzdump-lxc-9201-2026_07_28-05_31_14.tar.zst"
if got := rejectedArchiveMessage(local); strings.Contains(got, "pbs-phantom-cleanup") {
t.Errorf("a dir-storage archive got the PBS runbook pointer: %s", got)
}
}
+244
View File
@@ -0,0 +1,244 @@
package backup
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"strconv"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// BackupSuccessState persists the newest SUCCESSFUL whole-guest backup per tier and guest (R-894).
//
// Why it exists. The due-check (`localapi` handleBackupDue) asks the tier's storage when a backup last
// landed (R-84) and falls back to the in-memory record when the storage cannot be read. The in-memory
// record is empty after an agent restart (Store, R-348), so "storage unreadable" right after a restart
// read as "no record — DUE". Measured 2026-10-05 on demo-hp: the agent restarted at 04:57, the off-site
// storage answered "Can't connect" at 06:25, the 7-day tier — last copy 2026-10-01 — read DUE, the
// controller asked, and vzdump failed. This file is the last known copy the fallback reads instead.
//
// It is read ONLY when the storage cannot be read. A storage that answers is the ground truth and wins,
// in both directions: an archive found there counts, and an archive absent there is absent even when
// this file remembers a success (a pruned or deleted archive must make the tier due — the same reason
// R-84 chose the storage over a persisted record). Pinned by
// TestBackupDue_R894_SavedCopyIgnoredWhenStorageAnswers.
//
// Only SUCCESSES are written (the RestoreTestState rule): a failure must stay due and be retried, so a
// record of a failure has no reader.
type BackupSuccessState struct {
path string
mu sync.Mutex
last map[string]savedSuccess // key(target, vmid) → the newest success
}
type savedSuccess struct {
target string
vmid int
at time.Time
// recent (R-973) holds the start times of the newest successes (at most maxRecentSuccesses, newest
// last) — what the local-API backup cadence cap counts, so an agent restart does not reset the cap.
recent []time.Time
// refusedAt (R-973) is the newest time the cap refused a backup of this guest on this tier.
refusedAt time.Time
}
// maxRecentSuccesses bounds the per-tier success history kept for the cadence cap (R-973). The cap
// allows at most 3 runs in a window, so 8 is ample.
const maxRecentSuccesses = 8
// backupSuccessJSON is one entry on disk. Recent and LastRefusedAt are additive (R-973): a file written
// before them reads as "no history", and an older agent ignores them.
type backupSuccessJSON struct {
Target string `json:"target"`
VMID int `json:"vmid"`
StartedAt string `json:"started_at"`
Recent []string `json:"recent,omitempty"`
LastRefusedAt string `json:"last_refused_at,omitempty"`
}
func backupStateKey(target string, vmid int) string { return target + "/" + strconv.Itoa(vmid) }
// NewBackupSuccessState opens (or creates) the state at path. A missing or unreadable file degrades to
// "nothing known" — the pre-R-894 behaviour, which is DUE — and never wedges the daemon.
func NewBackupSuccessState(path string) *BackupSuccessState {
s := &BackupSuccessState{path: path, last: map[string]savedSuccess{}}
data, err := os.ReadFile(path)
if err != nil {
return s
}
var entries []backupSuccessJSON
if json.Unmarshal(data, &entries) != nil {
return s
}
for _, e := range entries {
t, perr := time.Parse(time.RFC3339, e.StartedAt)
if perr != nil {
continue // one unreadable entry must not lose the others
}
ss := savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()}
for _, r := range e.Recent {
if rt, rerr := time.Parse(time.RFC3339, r); rerr == nil {
ss.recent = append(ss.recent, rt.UTC())
}
}
if len(ss.recent) == 0 {
ss.recent = []time.Time{ss.at} // a pre-R-973 file: its one known success counts
}
if rt, rerr := time.Parse(time.RFC3339, e.LastRefusedAt); rerr == nil {
ss.refusedAt = rt.UTC()
}
s.last[backupStateKey(e.Target, e.VMID)] = ss
}
return s
}
// RecordBackupSuccess saves b when it is a success newer than the one on file. target is the tier the
// job ran on (the due-check's key); a failure or an unparseable time is ignored.
func (s *BackupSuccessState) RecordBackupSuccess(target string, b hub.Backup) error {
if s == nil || !b.Success {
return nil
}
t, err := time.Parse(time.RFC3339, b.StartedAt)
if err != nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
k := backupStateKey(target, b.VMID)
old, had := s.last[k]
if had && !t.After(old.at) {
return nil
}
ss := savedSuccess{target: target, vmid: b.VMID, at: t.UTC(), refusedAt: old.refusedAt}
ss.recent = append(append([]time.Time(nil), old.recent...), t.UTC())
if len(ss.recent) > maxRecentSuccesses {
ss.recent = ss.recent[len(ss.recent)-maxRecentSuccesses:]
}
s.last[k] = ss
return s.saveLocked()
}
// RecentBackupSuccesses returns the saved success start times for this tier and guest that are not
// older than since (R-973 — the cadence cap's restart-proof count).
func (s *BackupSuccessState) RecentBackupSuccesses(target string, vmid int, since time.Time) []time.Time {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
var out []time.Time
for _, t := range s.last[backupStateKey(target, vmid)].recent {
if !t.Before(since) {
out = append(out, t)
}
}
return out
}
// RecordBackupRefusal saves that the cadence cap refused a backup of this guest on this tier at at
// (R-973), so the operator signal it raises survives an agent restart.
func (s *BackupSuccessState) RecordBackupRefusal(target string, vmid int, at time.Time) error {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
k := backupStateKey(target, vmid)
ss, ok := s.last[k]
if !ok {
return nil // nothing to cap against without a success; the cap never refuses then
}
if !at.After(ss.refusedAt) {
return nil
}
ss.refusedAt = at.UTC()
s.last[k] = ss
return s.saveLocked()
}
// LastBackupRefusal returns the newest saved cadence-cap refusal across every tier and guest.
func (s *BackupSuccessState) LastBackupRefusal() (target string, vmid int, at time.Time, ok bool) {
if s == nil {
return "", 0, time.Time{}, false
}
s.mu.Lock()
defer s.mu.Unlock()
for _, e := range s.last {
if !e.refusedAt.IsZero() && e.refusedAt.After(at) {
target, vmid, at, ok = e.target, e.vmid, e.refusedAt, true
}
}
return target, vmid, at, ok
}
// LastKnownSuccess returns the newest saved success for this tier and guest (ok=false = none on file).
func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Time, bool) {
if s == nil {
return time.Time{}, false
}
s.mu.Lock()
defer s.mu.Unlock()
e, ok := s.last[backupStateKey(target, vmid)]
return e.at, ok
}
// KnownBackupSuccesses returns every saved success as a host-report record (the collector's
// KnownBackupReporter, so the hub keeps its evidence across a restart). Only the tier, guest and start
// time are known; the archive name is not saved and stays empty. Sorted for a stable report.
func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
out := make([]hub.Backup, 0, len(s.last))
for _, e := range s.last {
out = append(out, hub.Backup{TargetID: e.target, VMID: e.vmid, Success: true, CrashConsistent: true,
StartedAt: e.at.Format(time.RFC3339), UncoveredVolumes: []string{}})
}
sort.Slice(out, func(i, j int) bool {
if out[i].TargetID != out[j].TargetID {
return out[i].TargetID < out[j].TargetID
}
return out[i].VMID < out[j].VMID
})
return out
}
func (s *BackupSuccessState) saveLocked() error {
entries := make([]backupSuccessJSON, 0, len(s.last))
for _, e := range s.last {
j := backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)}
for _, r := range e.recent {
j.Recent = append(j.Recent, r.Format(time.RFC3339))
}
if !e.refusedAt.IsZero() {
j.LastRefusedAt = e.refusedAt.Format(time.RFC3339)
}
entries = append(entries, j)
}
// Deterministic file content (Go's map order is random).
sort.Slice(entries, func(i, j int) bool {
if entries[i].Target != entries[j].Target {
return entries[i].Target < entries[j].Target
}
return entries[i].VMID < entries[j].VMID
})
data, err := json.MarshalIndent(entries, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
return err
}
tmp := s.path + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, s.path)
}
+113
View File
@@ -0,0 +1,113 @@
package backup
import (
"os"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-894: the on-disk newest success per tier survives a restart (a new state from the same file).
func TestBackupSuccessState_SurvivesRestart(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
at := time.Date(2026, 10, 1, 20, 15, 0, 0, time.UTC)
if err := s.RecordBackupSuccess("felhom-pbs", hub.Backup{VMID: 9201, Success: true, StartedAt: at.Format(time.RFC3339)}); err != nil {
t.Fatal(err)
}
got, ok := NewBackupSuccessState(path).LastKnownSuccess("felhom-pbs", 9201)
if !ok || !got.Equal(at) {
t.Fatalf("after a restart the saved copy must read back; got %v ok=%v", got, ok)
}
if _, ok := NewBackupSuccessState(path).LastKnownSuccess("local", 9201); ok {
t.Fatal("another tier must not borrow this tier's copy")
}
}
// Only a NEWER success replaces the saved one; failures and unparseable times are ignored.
func TestBackupSuccessState_KeepsNewestSuccessOnly(t *testing.T) {
path := filepath.Join(t.TempDir(), "s.json")
s := NewBackupSuccessState(path)
newer := time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
older := newer.Add(-48 * time.Hour)
for _, b := range []hub.Backup{
{VMID: 1, Success: true, StartedAt: newer.Format(time.RFC3339)},
{VMID: 1, Success: true, StartedAt: older.Format(time.RFC3339)}, // older: ignored
{VMID: 1, Success: false, StartedAt: newer.Add(time.Hour).Format(time.RFC3339)}, // failure: ignored
{VMID: 1, Success: true, StartedAt: "not-a-time"}, // unparseable: ignored
} {
if err := s.RecordBackupSuccess("t", b); err != nil {
t.Fatal(err)
}
}
if got, _ := NewBackupSuccessState(path).LastKnownSuccess("t", 1); !got.Equal(newer) {
t.Fatalf("want the newest success %v, got %v", newer, got)
}
}
// A corrupt file degrades to "nothing known" (the pre-R-894 DUE answer), never a crash.
func TestBackupSuccessState_CorruptFileIsEmpty(t *testing.T) {
path := filepath.Join(t.TempDir(), "s.json")
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
t.Fatal(err)
}
if _, ok := NewBackupSuccessState(path).LastKnownSuccess("t", 1); ok {
t.Fatal("a corrupt file must read as nothing known")
}
}
// The saved successes come back as host-report records: success, tier, guest, start time (R-894 → the
// host report, 2026-10-09). A new state read from the same file gives the same records (the restart case).
func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
if err := s.RecordBackupSuccess("felhom-backup", hub.Backup{VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}); err != nil {
t.Fatal(err)
}
got := NewBackupSuccessState(path).KnownBackupSuccesses()
if len(got) != 1 || !got[0].Success || got[0].TargetID != "felhom-backup" || got[0].VMID != 9201 || got[0].StartedAt != "2026-10-09T02:40:16Z" {
t.Fatalf("got %+v", got)
}
if got[0].UncoveredVolumes == nil {
t.Fatal("uncovered_volumes must marshal as [], not null")
}
}
// R-973: the cadence cap's history and refusals survive a reopen; the history is bounded; a file
// written before R-973 counts its one known success (never "no history" for a tier that has a copy).
func TestBackupSuccessState_R973_RecentHistoryAndRefusalSurviveAReopen(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
base := time.Date(2026, 10, 18, 2, 30, 0, 0, time.UTC)
for i := 0; i < 10; i++ {
at := base.Add(time.Duration(i) * time.Hour)
if err := s.RecordBackupSuccess("local", hub.Backup{VMID: 9201, Success: true, StartedAt: at.Format(time.RFC3339)}); err != nil {
t.Fatal(err)
}
}
refused := base.Add(11 * time.Hour)
if err := s.RecordBackupRefusal("local", 9201, refused); err != nil {
t.Fatal(err)
}
r := NewBackupSuccessState(path)
all := r.RecentBackupSuccesses("local", 9201, time.Time{})
if len(all) != maxRecentSuccesses || !all[len(all)-1].Equal(base.Add(9*time.Hour)) {
t.Fatalf("recent history after a reopen: %v (want the newest %d, ending at +9h)", all, maxRecentSuccesses)
}
if got := r.RecentBackupSuccesses("local", 9201, base.Add(7*time.Hour)); len(got) != 3 {
t.Fatalf("successes since +7h: %d, want 3", len(got))
}
if tg, vm, at, ok := r.LastBackupRefusal(); !ok || tg != "local" || vm != 9201 || !at.Equal(refused) {
t.Fatalf("refusal after a reopen: %q %d %v %v", tg, vm, at, ok)
}
old := filepath.Join(t.TempDir(), "old.json")
if err := os.WriteFile(old, []byte(`[{"target":"felhom-pbs","vmid":9201,"started_at":"2026-10-17T03:00:00Z"}]`), 0o600); err != nil {
t.Fatal(err)
}
if got := NewBackupSuccessState(old).RecentBackupSuccesses("felhom-pbs", 9201, time.Time{}); len(got) != 1 {
t.Fatalf("a pre-R-973 file must count its one success: %v", got)
}
}
+1 -1
View File
@@ -2,9 +2,9 @@ package backup
import (
"context"
"fmt"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"testing"
+60
View File
@@ -0,0 +1,60 @@
package backup
import (
"context"
"sort"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// ForeignKeyLedger (R-366 slice 2, `09` §3 decision 168) holds, per backup tier, the whole-guest archives the
// restore-test pick skipped because another key wrote them (R-727 — an earlier install of this box). Since R-727 the
// skip was one INFO log line per archive and nothing else, so after a reinstall the operator was never told that the
// box's older whole-guest copies are unreadable to it. The host report carries this ledger; the hub raises ONE
// operator event when it changes.
//
// It reports nil until a tier has been evaluated since the agent started, so a restart does not read as "the set
// changed to empty" (the hub keeps its last state for an absent field).
type ForeignKeyLedger struct {
mu sync.Mutex
byTarget map[string]hub.ForeignKeyArchives
}
// NewForeignKeyLedger builds an empty ledger.
func NewForeignKeyLedger() *ForeignKeyLedger {
return &ForeignKeyLedger{}
}
func (l *ForeignKeyLedger) set(target string, n int, oldest, newest int64) {
l.mu.Lock()
defer l.mu.Unlock()
if l.byTarget == nil {
l.byTarget = map[string]hub.ForeignKeyArchives{}
}
e := hub.ForeignKeyArchives{Target: target, Count: n}
if n > 0 {
e.Oldest = time.Unix(oldest, 0).UTC().Format(time.RFC3339)
e.Newest = time.Unix(newest, 0).UTC().Format(time.RFC3339)
}
l.byTarget[target] = e
}
// ForeignKeyArchives implements hub.ForeignKeyArchiveReporter: nil before any evaluation; otherwise the tiers that
// hold such archives (`Tiers` empty, never nil, when none do), sorted by tier.
func (l *ForeignKeyLedger) ForeignKeyArchives(context.Context) *hub.ForeignKeyArchivesStanza {
l.mu.Lock()
defer l.mu.Unlock()
if l.byTarget == nil {
return nil
}
out := []hub.ForeignKeyArchives{}
for _, e := range l.byTarget {
if e.Count > 0 {
out = append(out, e)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Target < out[j].Target })
return &hub.ForeignKeyArchivesStanza{Tiers: out}
}
+2 -2
View File
@@ -93,7 +93,7 @@ func TestPrune_KeepLastClamp(t *testing.T) {
t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set)
}
}
if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-last=3" {
t.Fatalf("default PruneBackupsSpec = %q, want keep-last=3", spec)
if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-daily=3" {
t.Fatalf("default PruneBackupsSpec = %q, want keep-daily=3 (R-1149)", spec)
}
}
+67
View File
@@ -0,0 +1,67 @@
package backup
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1012: a snapshot backup that fails carries the orphan finding in its error — the text the controller mails to
// the operator (`whole_guest_backup_failed`). The check runs ONCE, after the failure — never before the run.
// RED-PROOF (P1/redproof-thincheck.txt): with explainFailure's append removed this test fails. RED-PROOF (2026-10-17
// night, felhom-agent REPORT.md): with the v0.159.0 pre-run check restored, calls = 2 and this test fails.
func TestBackup_SnapshotFailure_NamesTheOrphan(t *testing.T) {
const why = "probable cause: thin pool pve/data holds device 4 that LVM does not know"
for name, api := range map[string]*fakeBackupAPI{
"task fails": {vzdumpUPID: "UPID:x", waitErr: errors.New("task failed: job errors")},
"start fails": {vzdumpErr: errors.New("vzdump: 500")},
} {
t.Run(name, func(t *testing.T) {
r := NewBackupRunner(api, "local", "", "", "", quiet())
calls := 0
r.SetThinCheck(func(context.Context, int) string { calls++; return why })
rec, err := r.Backup(context.Background(), 9201)
if err == nil || rec.Success {
t.Fatalf("want a failure, got %+v", rec)
}
if !strings.Contains(rec.Error, why) {
t.Fatalf("the failure does not name the orphan: %q", rec.Error)
}
if calls != 1 {
t.Fatalf("thin check ran %d times, want 1 (after the failure only)", calls)
}
})
}
}
// A successful backup never runs the check (it takes a metadata snapshot of the live pool, and nobody needs the answer)
// and keeps a clean record; a stop-mode backup takes no snapshot, so the check never runs; an unwired runner behaves
// exactly as before. RED-PROOF (2026-10-17 night): with the v0.159.0 pre-run check restored, the success leg fails
// with calls=1.
func TestBackup_ThinCheck_SilentWhenNotApplicable(t *testing.T) {
ok := &fakeBackupAPI{vzdumpUPID: "UPID:x",
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-x", Content: "backup", VMID: 9201, Size: 100, CTime: 1}}}
r := NewBackupRunner(ok, "local", "", "", "", quiet())
calls := 0
r.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
rec, err := r.Backup(context.Background(), 9201)
if err != nil || !rec.Success || rec.Error != "" || calls != 0 {
t.Fatalf("success: err=%v rec=%+v calls=%d (want no check on a successful backup, clean record)", err, rec, calls)
}
stop := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", proxmox.ModeStop, "", "", quiet())
calls = 0
stop.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
rec, _ = stop.Backup(context.Background(), 9201)
if calls != 0 || strings.Contains(rec.Error, "probable cause") {
t.Fatalf("stop mode: calls=%d error=%q", calls, rec.Error)
}
plain := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", "", "", "", quiet())
if rec, _ := plain.Backup(context.Background(), 9201); rec.Error != "boom" {
t.Fatalf("unwired: error = %q, want the task error unchanged", rec.Error)
}
}
@@ -0,0 +1,55 @@
package backup
import (
"context"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1232 — the 2026-10-23 test night: a reinstalled box (new install, new PBS key, same customer namespace, same guest
// number 9201) found the OLD box's archive (2.6 days old) when the due-check asked „when did this tier last land?", so
// the whole-guest tier read „not due" for up to its 168-h cadence and the ep0 copy stayed the OLD box's. The due-check
// now counts only archives written with THIS storage's key (the R-727 discriminator; the archive carries no host id).
// The CONSEQUENCE asserted: with only a foreign archive the answer is „none found" (the tier is due); with the box's
// own archive, that archive's time. COMPANION RED-PROOF: drop the key filter in NewestArchiveTime → case 1 returns the
// old box's time.
func TestR1232_TheDueCheckCountsOnlyThisBoxsArchives(t *testing.T) {
oldBox := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-20T21:10:00Z", Content: "backup", VMID: 9201,
Size: 3490689830, CTime: time.Date(2026, 10, 20, 21, 10, 0, 0, time.UTC).Unix(), Encrypted: earlierBox1}
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbsdr", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{oldBox},
}
r := NewBackupRunner(api, "felhom-pbsdr", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
// 1. The fresh box has written nothing yet: the old box's archive must not count.
when, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatal(err)
}
if found {
t.Fatalf("the due-check counted ANOTHER box's archive (%s) as this tier's last backup — the tier would wait a week", when)
}
// 2. After the box's own first copy, that copy is the answer.
own := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-24T02:30:00Z", Content: "backup", VMID: 9201,
Size: 3600000000, CTime: time.Date(2026, 10, 24, 2, 30, 0, 0, time.UTC).Unix(), Encrypted: thisBoxKey}
api.content = append(api.content, own)
when, found, err = r.NewestArchiveTime(context.Background(), 9201)
if err != nil || !found || when.Unix() != own.CTime {
t.Fatalf("want this box's own archive time %d, got %v found=%v err=%v", own.CTime, when, found, err)
}
}
// An unencrypted storage (a local dir) holds only this box's vzdumps: no key filter, the newest counts.
func TestR1232_UnencryptedStorageCountsEveryArchive(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_10_25-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1793050025}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, found, err := r.NewestArchiveTime(context.Background(), 9201); err != nil || !found {
t.Fatalf("an unencrypted local archive was not counted: found=%v err=%v", found, err)
}
}
@@ -0,0 +1,69 @@
package backup
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-366 slice 2 (`09` §3 decision 168) — the restore-test's skip of an archive written with another key stops being
// silent: the pick records, per tier, how many it skipped and their time range, and the host report carries it.
//
// COMPANION RED-PROOF (observed): remove the `r.foreign.set(...)` call from PickSettledRestoreCandidateOn → this fails
// with "after one evaluation the ledger must report felhom-pbs: 2 archives …; got []". Restored.
func TestR366_PickRecordsArchivesWrittenWithAnotherKey(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey},
},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
l := NewForeignKeyLedger()
r.SetForeignKeyLedger(l)
if got := l.ForeignKeyArchives(context.Background()); got != nil {
t.Fatalf("before any evaluation the ledger must be nil (the hub keeps its state); got %v", got)
}
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
t.Fatal(err)
}
st := l.ForeignKeyArchives(context.Background())
want := hub.ForeignKeyArchives{Target: "felhom-pbs", Count: 2, Oldest: "2026-09-16T17:27:32Z", Newest: "2026-09-16T21:59:54Z"}
var got []hub.ForeignKeyArchives
if st != nil {
got = st.Tiers
}
if len(got) != 1 || got[0] != want {
t.Fatalf("after one evaluation the ledger must report felhom-pbs: 2 archives 2026-09-16T17:27:32Z…21:59:54Z; got %v", got)
}
}
// Evaluated and none found → the stanza with `tiers: []`; not evaluated → no stanza at all. Never a null on the wire.
func TestR366_EvaluatedWithNoneIsAnEmptyList(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
l := NewForeignKeyLedger()
r.SetForeignKeyLedger(l)
b, _ := json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
if strings.Contains(string(b), "foreign_key_archives") {
t.Fatalf("not evaluated must omit the stanza; got %s", b)
}
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
t.Fatal(err)
}
b, _ = json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
if !strings.Contains(string(b), `"foreign_key_archives":{"tiers":[]}`) {
t.Fatalf("evaluated with none must report tiers: []; got %s", b)
}
}
+25
View File
@@ -81,3 +81,28 @@ func TestR685_FailsOpen(t *testing.T) {
}
}
}
// TestR1006_LeftoverCleanupRunsBeforeThePreflight — the night of 2026-10-14: a 12.2 GB half-written archive from a power
// cut held the space, so every later backup was skipped. The cleanup step runs BEFORE the preflight reads the free space:
// here it „removes" the leftover (free 4 → 16 GiB) and the backup then fits and runs.
//
// RED-PROOF: drop the r.beforeBackup call from backup() (or move it after spaceFits) → the backup is skipped → fails.
func TestR1006_LeftoverCleanupRunsBeforeThePreflight(t *testing.T) {
api := spaceAPI(4*gib, 8182759056, "dir")
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
var got []string
r.SetBeforeBackup(func(_ context.Context, storage string, vmid int) {
got = append(got, storage)
if vmid != 9201 {
t.Errorf("cleanup asked for vmid %d", vmid)
}
api.storages[0].Avail = 16 * gib
})
_, _ = r.Backup(context.Background(), 9201)
if len(got) != 1 || got[0] != "local" {
t.Fatalf("the cleanup must run once for this target, got %v", got)
}
if len(api.vzdumps) != 1 {
t.Fatalf("after the cleanup freed the space the backup must run, vzdumps=%d", len(api.vzdumps))
}
}
+1 -1
View File
@@ -48,7 +48,7 @@ func TestR689_GuestBackupArchiveShapes(t *testing.T) {
{proxmox.StorageContent{VolID: "felhom-pbs:backup/vm/300/2026-07-28T05:31:14Z", VMID: 300}, true},
{proxmox.StorageContent{VolID: "local:backup/felhom-golden-0.236.0.tar.zst"}, false},
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", VMID: 9202}, false}, // vmid disagrees with the name
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
} {
if ok, why := guestBackupArchive(c.e); ok != c.ok {
t.Errorf("%s vmid=%d: ok=%v (%s), want %v", c.e.VolID, c.e.VMID, ok, why, c.ok)
+83 -3
View File
@@ -58,16 +58,29 @@ type BackupRunner struct {
// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
// there is a deliberate act.
allowPBSPrune bool
logger *slog.Logger
now func() time.Time
// beforeBackup runs just before the space preflight (R-1006): it removes a dead half-written vzdump of this guest on
// this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing.
beforeBackup func(ctx context.Context, storage string, vmid int)
// thinCheck (R-1012) names an orphan thin device in the guest's pools in plain words ("" = none / not checked). It
// runs ONLY when a snapshot-mode vzdump has failed (start or task), so the failure the controller mails to the
// operator says why; a successful backup never runs it (it takes a metadata snapshot of the live pool). It never
// removes anything. nil = not wired.
thinCheck func(ctx context.Context, vmid int) string
logger *slog.Logger
now func() time.Time
// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
// due-check is served from the local-API handler goroutines.
rejectedMu sync.Mutex
rejected map[string]struct{}
// foreign (R-366 slice 2) records, per tier, the archives the pick skipped as another key's. nil = not wired.
foreign *ForeignKeyLedger
}
// SetForeignKeyLedger wires the R-366 slice-2 ledger the host report reads.
func (r *BackupRunner) SetForeignKeyLedger(l *ForeignKeyLedger) { r.foreign = l }
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
@@ -97,6 +110,27 @@ func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode,
now: func() time.Time { return time.Now().UTC() }}
}
// SetBeforeBackup sets the step run just before the space preflight (R-1006: the dead-leftover cleanup).
func (r *BackupRunner) SetBeforeBackup(fn func(ctx context.Context, storage string, vmid int)) {
r.beforeBackup = fn
}
// SetThinCheck wires the R-1012 orphan check (see BackupRunner.thinCheck).
func (r *BackupRunner) SetThinCheck(fn func(ctx context.Context, vmid int) string) { r.thinCheck = fn }
// explainFailure appends the orphan finding to a failed snapshot-mode backup's error — the only place the check runs, so
// it sees the pool as the failure left it. Bounded, so a stuck helper cannot hold the job.
func (r *BackupRunner) explainFailure(ctx context.Context, vmid int, rec *hub.Backup) {
if r.thinCheck == nil || r.mode != proxmox.ModeSnapshot {
return
}
cctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute)
defer cancel()
if why := r.thinCheck(cctx, vmid); why != "" {
rec.Error = rec.Error + " — " + why
}
}
// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
// applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a
// separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type
@@ -178,6 +212,9 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
// R-685 (v0.134.0): will the new archive FIT on a local target? Asked before anything runs, so a
// target that cannot hold it is a named SKIP with the numbers, not a nightly "No space left on
// device" that only the vzdump log explains (demo-hp, every night from 2026-09-23 — R-684).
if r.beforeBackup != nil {
r.beforeBackup(ctx, r.target, vmid)
}
if ok, why := r.spaceFits(ctx, vmid); !ok {
rec.Error = BackupSkipNoSpacePrefix + why
rec.DurationSeconds = time.Since(start).Seconds()
@@ -185,12 +222,15 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
return rec, fmt.Errorf("backup: %s", rec.Error)
}
// R-1012: no orphan check here. It takes a metadata snapshot of the live pool, and its answer is only needed when the
// snapshot step has failed — explainFailure runs it then (pinned by TestBackup_ThinCheck_SilentWhenNotApplicable).
upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
})
if err != nil {
rec.Error = err.Error()
r.explainFailure(ctx, vmid, &rec)
rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err)
}
@@ -204,6 +244,7 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
}
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil {
rec.Error = err.Error()
r.explainFailure(ctx, vmid, &rec)
rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
}
@@ -370,6 +411,8 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
var best string
var bestCTime int64 = -1
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
var foreignN int // R-366 slice 2: archives skipped as another key's, and their time range
var foreignMin, foreignMax int64
for _, e := range contents {
if e.Content != "backup" {
continue
@@ -384,6 +427,13 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
}
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
foreignN++
if foreignMin == 0 || e.CTime < foreignMin {
foreignMin = e.CTime
}
if e.CTime > foreignMax {
foreignMax = e.CTime
}
continue
}
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
@@ -420,6 +470,9 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
bestCTime, best = e.CTime, e.VolID
}
}
if r.foreign != nil {
r.foreign.set(target, foreignN, foreignMin, foreignMax)
}
if best == "" {
return "", time.Time{}, nil
}
@@ -518,10 +571,25 @@ func (r *BackupRunner) warnRejectedArchiveOnce(e proxmox.StorageContent, why str
if seen {
return
}
r.logger.Warn("backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup",
r.logger.Warn(rejectedArchiveMessage(e),
"target", r.target, "vmid", e.VMID, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
}
// phantomCleanupPointer names the runbook that removes a PBS phantom (R-99, `09` §3 decision 140: a leftover of an
// aborted upload is deleted on the backup server, by a runbook, when one is seen — never automatically).
const phantomCleanupPointer = " — a phantom leftover; delete it by felhom.eu documentation/runbooks/pbs-phantom-cleanup.md (09 §3 decision 140)"
// rejectedArchiveMessage is the WARN text for a rejected archive. Only a PBS entry (format pbs-ct / pbs-vm) gets the
// cleanup pointer: the runbook deletes on a PBS datastore, and a tiny archive on a dir storage is not a PBS phantom.
// Pinned by TestRejectedArchiveWarnNamesTheCleanupRunbook.
func rejectedArchiveMessage(e proxmox.StorageContent) string {
msg := "backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup"
if strings.HasPrefix(e.Format, "pbs-") {
msg += phantomCleanupPointer
}
return msg
}
// demo-felhom in a single afternoon of deploys (2026-07-26).
//
// Asking the STORAGE rather than persisting the store is deliberate:
@@ -541,11 +609,23 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
if err != nil {
return time.Time{}, false, err
}
// R-1232: on an ENCRYPTED storage only archives written with THIS storage's key are this box's (the R-727
// discriminator — the archive carries no host id). A reinstalled box shares the customer's namespace and guest
// number with the old box, and the old box's archive made the tier read „not due" for a whole cadence (168 h).
// A key lookup failure is returned: the due-check reads UNKNOWN, never a foreign archive's age.
ownKey, err := r.storageKeyFingerprint(ctx, r.target)
if err != nil {
return time.Time{}, false, fmt.Errorf("reading the key fingerprint of storage %s: %w", r.target, err)
}
var best int64 = -1
for _, e := range contents {
if e.Content != "backup" || e.VMID != vmid {
continue
}
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this tier's last backup (R-1232)", shortFP(e.Encrypted), shortFP(ownKey)))
continue
}
if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why)
continue
+8 -1
View File
@@ -25,7 +25,14 @@ import (
// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
// Backups are NOT unaffected (corrected 2026-10-05, R-348): byTarget is in memory too, so after a restart the
// reported backup LIST reads 0 until the next backup of each tier runs (daily local, weekly offsite) — measured
// 2026-08-20, two consecutive host-reports with `0 backups` while `pvesm list` showed archives on both tiers. What
// is unaffected is the hub's VERDICT: it looks back 7 days over stored reports (felhom.eu hub/internal/monitor/
// deadline.go backupEvidenceLookback) and the storage stays the ground truth (R-84).
// The due-check's fallback for an UNREADABLE storage no longer reads this store alone (R-894): the newest
// success per tier is also on disk (BackupSuccessState), so a restart followed by an unreachable storage
// reads the last known copy, not "never".
type Store struct {
mu sync.Mutex
byTarget map[string]hub.Backup // latest backup per target id
+14 -1
View File
@@ -91,9 +91,11 @@ var manifest = []Capability{
{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
{"thin-orphan-check", "orphan thin device check after a failed whole-guest backup (read-only, R-1012)", "/usr/local/sbin/felhom-thin-check", []string{"pve", "data"}, false, ""},
// ---- Storage mount units (watchdog re-mount) ----
{"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-forget", "decommissioned-drive unit removal (root check, R-1124)", "/usr/local/sbin/felhom-priv-apply", []string{"unit-forget", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
@@ -115,6 +117,11 @@ var manifest = []Capability{
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
{"guest-shutdown", "household power-off: guest shutdown (R-998)", "/usr/sbin/pct", []string{"shutdown", "9201", "--timeout", "120"}, false, ""},
{"host-poweroff", "household power-off: host power-off (R-998)", "/usr/bin/systemctl", []string{"poweroff"}, false, ""},
{"vzdump-partial-clean", "dead half-written vzdump removal (R-1006)", "/usr/local/sbin/felhom-vzdump-partial-clean", []string{"local", "9201", "vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat"}, false, ""},
{"internal-place-plan", "internal data place: the size plan (16 §7)", "/usr/local/sbin/felhom-internal-place", []string{"plan", "0"}, false, ""},
{"internal-place-create", "internal data place: create on an old box (signed add_internal_place)", "/usr/local/sbin/felhom-internal-place", []string{"create", "0"}, false, ""},
// ---- LAN split-horizon resolver (dnsmasq) ----
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
@@ -144,13 +151,19 @@ var manifest = []Capability{
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "tee", "/etc/felhom-controller-image"}, true, ""},
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
{"stalelock-unlock", "reboot-during-backup stale-lock recovery", "/usr/sbin/pct", []string{"unlock", "9201"}, true, ""},
// ---- Weekly guest disk trim (FELHOM_FSTRIM, R-444). NON-critical: a missing grant means the thin pool is not
// reclaimed this week (the trim job WARNs per guest and the report shows the failure), not a serving outage. ----
{"guest-fstrim", "weekly guest disk trim (thin-pool reclaim, R-444)", "/usr/sbin/pct", []string{"fstrim", "9201"}, false, ""},
// ---- Offsite WG tunnel (FELHOM_WG, S3/v0.64.0; Critical FLIPPED in S4/v0.66.0 — offsite
// backups now RIDE the tunnel, so a degraded tunnel capability is operator-alert-worthy: the
// conf install, unit enable/restart and the handshake read gate the backup path. apt-install
+40 -3
View File
@@ -200,7 +200,8 @@ func TestRedProof_DroppedGrantFailsCheck(t *testing.T) {
}
// TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0
// FELHOM_CONTROLLERSWAP grants: with the `tee /etc/felhom-controller-image` line removed, the
// FELHOM_CONTROLLERSWAP grants: with the write grant removed (since R-861 (a) A1 the `felhom-priv-apply controller-image`
// line; before it, an agent `tee /etc/felhom-controller-image`), the
// controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new
// swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update).
func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
@@ -210,7 +211,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
}
var kept []string
for _, ln := range strings.Split(string(data), "\n") {
if strings.Contains(ln, "tee /etc/felhom-controller-image") {
if strings.Contains(ln, "felhom-priv-apply ^controller-image") { // R-861 (a) A1: the write's grant
continue
}
kept = append(kept, ln)
@@ -229,7 +230,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
}
cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ")
if matchesAny(cmdline, entries) {
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping the tee grant")
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping its grant")
}
if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) {
t.Errorf("controllerswap-write should be covered by the real sudoers")
@@ -310,3 +311,39 @@ func TestWGCapabilityCriticality(t *testing.T) {
}
}
}
// TestEveryAliasIsGranted closes a gap found while adding FELHOM_POWEROFF (R-998): the coverage check above reads
// every Cmnd_Alias body whether or not the alias appears on the grant line, so a manifest command whose alias was
// never GRANTED still passed — and failed with „a password is required" on the box. Every defined alias must be
// named on the `felhom-agent ALL=(root) NOPASSWD:` line. Red-proof: drop FELHOM_POWEROFF from that line.
func TestEveryAliasIsGranted(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatalf("read sudoers: %v", err)
}
var defined []string
granted := map[string]bool{}
grantLines := 0
for _, ln := range strings.Split(string(data), "\n") {
tr := strings.TrimSpace(ln)
if strings.HasPrefix(tr, "Cmnd_Alias ") {
if f := strings.Fields(tr); len(f) >= 2 {
defined = append(defined, f[1])
}
}
if strings.HasPrefix(tr, "felhom-agent ") && strings.Contains(tr, "NOPASSWD:") {
grantLines++
for _, a := range strings.Split(tr[strings.Index(tr, "NOPASSWD:")+len("NOPASSWD:"):], ",") {
granted[strings.TrimSpace(a)] = true
}
}
}
if grantLines == 0 || len(defined) < 10 {
t.Fatalf("parsed %d grant line(s), %d aliases — parser broke", grantLines, len(defined))
}
for _, a := range defined {
if !granted[a] {
t.Errorf("Cmnd_Alias %s is defined but never granted to felhom-agent", a)
}
}
}
+1 -1
View File
@@ -47,7 +47,7 @@ type Runner interface {
// downgrades a HEALTHY probe to StatusInactive (broken plumbing stays degraded regardless).
type Prober struct {
Runner Runner
Exists func(path string) bool // nil → os.Stat
Exists func(path string) bool // nil → os.Stat
GateActive func(gate string) bool // nil → every gate treated active
}
@@ -49,6 +49,34 @@ var r861Injections = []string{
"/usr/local/sbin/felhom-priv-apply unit ../../etc/x.mount",
"/usr/local/sbin/felhom-priv-apply dnsmasq /etc/shadow felhom-x.conf",
"/usr/local/sbin/felhom-priv-apply wg /etc/shadow",
// R-861 (a) A1 (decision 165): the agent wrote ANY image ref into the guest by `tee` — now only the root verb may
"/usr/sbin/pct exec 9201 -- tee /etc/felhom-controller-image",
"/usr/local/sbin/felhom-priv-apply controller-image 9201 9202",
"/usr/local/sbin/felhom-priv-apply controller-image 9201;id",
// R-998 (host power-off): the two new grants take exactly their fixed shape — no extra option, no other target
"/usr/sbin/pct shutdown 9201 --timeout 120 --forceStop 1",
"/usr/sbin/pct shutdown 9201 --forceStop 1 --timeout 120",
"/usr/sbin/pct shutdown 9201",
"/usr/sbin/pct shutdown 9201 --timeout -1",
"/usr/bin/systemctl poweroff --force",
"/usr/bin/systemctl poweroff -ff",
"/usr/bin/systemctl reboot",
"/usr/bin/systemctl halt",
"/usr/bin/systemctl kexec",
// R-1006: the partial-cleanup helper takes exactly one storage, one vmid, one half-written name.
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.zst",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.log",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 ../../etc/vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat /etc/shadow",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat;id",
"/usr/local/sbin/felhom-vzdump-partial-clean local;id 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201",
"/usr/local/sbin/felhom-vzdump-partial-clean --help",
"/usr/local/sbin/felhom-internal-place create 132",
"/usr/local/sbin/felhom-internal-place create 0 extra",
"/usr/local/sbin/felhom-internal-place create 0;id",
"/usr/local/sbin/felhom-internal-place wipe 0",
"/usr/local/sbin/felhom-internal-place create",
}
func TestSudoersRefusesTheR861Injections(t *testing.T) {
@@ -63,3 +91,72 @@ func TestSudoersRefusesTheR861Injections(t *testing.T) {
}
}
}
// R-444: the weekly trim's grant is ONE exact shape — `pct fstrim <vmid>` — and nothing smuggled after it.
// The manifest entry (guest-fstrim) proves the real call is still allowed (TestManifestCoveredBySudoers); this
// pins the other direction. RED-PROOF: write the rule as the glob `/usr/sbin/pct fstrim [0-9]*` → every decoy
// below with a trailing argument matches (the glob's `*` eats spaces).
func TestSudoersFstrimRuleIsExact(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatal(err)
}
entries := parseSudoersEntries(t, string(data))
if !matchesAny("/usr/sbin/pct fstrim 9201", entries) {
t.Fatal("the sudoers does not allow `pct fstrim 9201` — the weekly trim cannot run")
}
for _, c := range []string{
"/usr/sbin/pct fstrim 9201 --ignore-mountpoints",
"/usr/sbin/pct fstrim 9201 --ignore-mountpoints 1",
"/usr/sbin/pct fstrim 9201; x",
"/usr/sbin/pct fstrim 9201 9202",
"/usr/sbin/pct fstrim 92a1",
"/usr/sbin/pct fstrim ",
"/usr/sbin/pct fstrim -- 9201",
"/usr/sbin/pct destroy 9201",
"/usr/sbin/pct destroy 9201 --purge",
} {
if matchesAny(c, entries) {
t.Errorf("the sudoers allows a command the trim rule must not: %q", c)
}
}
}
// R-861 (a) A1: the managed controller update still has its route — the root verb, one numeric vmid.
func TestSudoersAllowsTheControllerImageVerb(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatal(err)
}
if !matchesAny("/usr/local/sbin/felhom-priv-apply controller-image 9201", parseSudoersEntries(t, string(data))) {
t.Fatal("the sudoers does not allow `felhom-priv-apply controller-image 9201` — a managed controller update cannot write its image")
}
}
// R-861 (b) B2 (decision 165, hygiene): felhom-op's `pct start|stop|unlock` grants are ONE numeric vmid each. The old
// glob `[0-9]*` eats spaces, so `pct stop 9201 --skiplock 1` and two vmids matched.
// RED-PROOF: on the pre-B2 felhom-op.sudoers (`/usr/sbin/pct stop [0-9]*`) the decoys match.
func TestFelhomOpSudoersPctIsExact(t *testing.T) {
data, err := os.ReadFile("../../configs/felhom-op.sudoers")
if err != nil {
t.Fatal(err)
}
entries := parseSudoersEntries(t, string(data))
for _, ok := range []string{"/usr/sbin/pct start 9201", "/usr/sbin/pct stop 9201", "/usr/sbin/pct unlock 9201", "/usr/sbin/pct list"} {
if !matchesAny(ok, entries) {
t.Errorf("felhom-op lost a repair verb: %s", ok)
}
}
for _, bad := range []string{
"/usr/sbin/pct stop 9201 --skiplock 1",
"/usr/sbin/pct start 9201 9202",
"/usr/sbin/pct unlock 9201 --whatever",
"/usr/sbin/pct start 92a1",
"/usr/sbin/pct stop ",
"/usr/sbin/pct destroy 9201",
} {
if matchesAny(bad, entries) {
t.Errorf("felhom-op's sudoers allows %q", bad)
}
}
}
+32
View File
@@ -180,3 +180,35 @@ func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
}
}
// R-1149 (operator decision D17, 2026-10-19): the local whole-guest tier keeps one restore point per day for N days
// (`--prune-backups keep-daily=N`), not the newest N; an additional tier keeps its explicit count.
//
// COMPANION RED-PROOF (observed 2026-10-19): make PruneSpec return keep-last for the primary → this fails with
// `primary tier prune = "keep-last=3", want "keep-daily=3"`. Restored.
func TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
LocalBackupRetention: 3,
ExtraTargets: []BackupTargetConfig{
{TargetID: "felhom-pbs", CadenceSeconds: 604800, KeepLast: 2},
{TargetID: "nas", CadenceSeconds: 86400},
},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 3 {
t.Fatalf("want 3 tiers, got %d", len(tiers))
}
for _, c := range []struct {
i int
want string
}{{0, "keep-daily=3"}, {1, "keep-last=2"}, {2, ""}} {
if got := tiers[c.i].PruneSpec(); got != c.want {
t.Fatalf("%s tier prune = %q, want %q", map[bool]string{true: "primary", false: tiers[c.i].TargetID}[tiers[c.i].Primary], got, c.want)
}
}
// The unset retention defaults to 3 days, never to „prune everything".
if got := (BackupConfig{}).PruneBackupsSpec(); got != "keep-daily=3" {
t.Fatalf("default local prune = %q, want keep-daily=3", got)
}
}
+43 -3
View File
@@ -33,6 +33,8 @@ type Config struct {
LANResolver LANResolverConfig `json:"lan_resolver"`
WGTunnel WGTunnelConfig `json:"wg_tunnel"`
GuestNet GuestNetConfig `json:"guest_net"`
DiskTrim DiskTrimConfig `json:"disk_trim"`
LineTest LineTestConfig `json:"line_test"`
OOB OOBConfig `json:"oob"`
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
@@ -139,6 +141,26 @@ func (w WGTunnelConfig) WithDefaults() WGTunnelConfig {
return w
}
// DiskTrimConfig configures the R-444 weekly guest disk trim (internal/fstrim). DEFAULT-ON, like GuestNetConfig and for
// the same reason: it only acts on guests the agent already owns, and the operator ruled every box trims (`09` §3
// decision 139). Opting out is the explicit act: `"disk_trim": {"disable": true}`.
type DiskTrimConfig struct {
Disable bool `json:"disable"`
}
// Enabled reports whether the weekly trim should run.
func (d DiskTrimConfig) Enabled() bool { return !d.Disable }
// LineTestConfig configures the R-1247 weekly upload line test (internal/linetest). DEFAULT-ON: it reaches only ep0's
// tunnel-only iperf3 server, so it runs only on a box that already has an operational offsite tunnel (operator
// ruling 2026-10-24 07:26: every box with a tunnel measures its line weekly).
type LineTestConfig struct {
Disable bool `json:"disable"`
}
// Enabled reports whether the weekly line test should run.
func (l LineTestConfig) Enabled() bool { return !l.Disable }
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
//
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
@@ -380,7 +402,8 @@ type BackupConfig struct {
// hub-served per-guest policy is slice 10; this is the agent-local cadence.
BackupCadenceSeconds int `json:"backup_cadence_seconds"`
// LocalBackupRetention is keep-last=N for the per-run `--prune-backups` on a LOCAL vzdump target —
// LocalBackupRetention is N for the per-run `--prune-backups keep-daily=N` on a LOCAL vzdump target (keep-last=N
// before R-1149) —
// so the agent's own local whole-guest backups can't pile up and refill root (the felhom-pve incident;
// the host_disk + storage_fill checkers are the detectors, this is the preventive default). 0/unset →
// default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup.
@@ -527,9 +550,26 @@ func (b BackupConfig) KeepLast() int {
return b.LocalBackupRetention
}
// PruneBackupsSpec returns the PVE `--prune-backups` value for the local vzdump (e.g. "keep-last=3").
// PruneBackupsSpec returns the PVE `--prune-backups` value for the local vzdump: "keep-daily=N" (R-1149, operator
// decision D17 2026-10-19 — one restore point per day for N days; `07` §6.1). keep-last=N had the same disk cost but
// let N extra runs on one day push out every older point; keep-daily keeps the newest copy of each of the N newest
// days that have one, so a flood inside one day replaces only that day's point. Pinned by
// TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay and, as the consequence, TestBackupCap_R1149_LocalFloodKeepsOlderDays.
func (b BackupConfig) PruneBackupsSpec() string {
return fmt.Sprintf("keep-last=%d", b.KeepLast())
return fmt.Sprintf("keep-daily=%d", b.KeepLast())
}
// PruneSpec is this tier's per-run `--prune-backups` value, or "" for never prune. The PRIMARY (local) tier keeps one
// point per day (keep-daily, R-1149, from KeepLast() — always ≥1); an additional tier keeps its explicit keep_last
// count (a weekly off-site tier's „last 2" is two weeks, and PBS prunes stay opt-in per tier, R-82).
func (t BackupTier) PruneSpec() string {
if t.KeepLast <= 0 {
return ""
}
if t.Primary {
return fmt.Sprintf("keep-daily=%d", t.KeepLast)
}
return fmt.Sprintf("keep-last=%d", t.KeepLast)
}
// BackupCadence returns the per-guest /backup/due window: positive as-is, else 24h default.
+188
View File
@@ -0,0 +1,188 @@
package config
import (
"bytes"
"encoding/json"
"fmt"
"os"
"path/filepath"
)
// A cut write of agent.json must never leave a blind box (R-1380, 2026-11-15).
//
// agent.json is the agent's own file inside a ROOT-owned folder, so the agent cannot write a new file and rename it
// over the old one: every writer truncates the file and writes it again. A power cut between the truncate and the
// sync leaves an empty or half file, the agent does not start, and nothing can repair a box whose agent does not
// start from the hub. So the three writers (the trust move, the PBS seed, the backup-target move) go through
// ReplaceInPlace: the OLD bytes are first made durable in a journal file in the agent's own state folder (a new file
// and a rename, which the agent CAN do there), then the config is written, synced and read back, then the journal is
// removed. RecoverCutWrite runs at every start, before the config is read: a journal beside a config that does not
// parse means the write was cut, and the old bytes go back. A journal beside a config that parses means the cut fell
// before the truncate or after the sync — the file is whole, old or new, and only the journal is removed.
//
// Pinned by TestReplaceInPlace_* and TestRecoverCutWrite_* (this package) and by TestMainRecoversACutConfigWrite
// (cmd/felhom-agent).
// CutWriteDir is where the journal lives. It is NOT read from the config: the recovery runs before the config can be
// read. /var/lib/felhom-agent is the unit's StateDirectory. A test points it at a temp folder.
var CutWriteDir = "/var/lib/felhom-agent"
// cutWriteJournal is the journal's file name: the bytes agent.json held before the write that is in flight.
const cutWriteJournal = "agent.json.write-in-flight"
// cutPoint lets a test stop a write where a power cut would (the names are the steps of ReplaceInPlace).
var cutPoint = func(step string) error { return nil }
// ReplaceInPlace writes next over the config at path, whose present content is prev. The caller holds FileMu.
// With no journal folder there is no write: an unprotected write is the defect this exists to remove.
func ReplaceInPlace(path string, prev, next []byte) error {
if !json.Valid(prev) || !json.Valid(next) {
return fmt.Errorf("config: refusing to write %s: the old or the new content is not JSON", path)
}
st, err := os.Stat(path)
if err != nil {
return err
}
if err := os.MkdirAll(CutWriteDir, 0o700); err != nil {
return fmt.Errorf("config: the journal folder: %w", err)
}
journal := filepath.Join(CutWriteDir, cutWriteJournal)
if err := writeDurable(journal, prev); err != nil {
return fmt.Errorf("config: the journal of the old content: %w", err)
}
if err := cutPoint("journal-written"); err != nil {
return err
}
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
if err != nil {
_ = removeDurable(journal)
return err
}
if err := cutPoint("truncated"); err != nil {
f.Close()
return err
}
half := len(next) / 2
_, werr := f.Write(next[:half])
if werr == nil {
if err := cutPoint("half-written"); err != nil {
f.Close()
return err
}
_, werr = f.Write(next[half:])
}
if werr == nil {
werr = f.Sync()
}
if cerr := f.Close(); werr == nil {
werr = cerr
}
if werr == nil {
var back []byte
if back, werr = os.ReadFile(path); werr == nil && !bytes.Equal(back, next) {
werr = fmt.Errorf("the file did not read back as written")
}
}
if werr != nil {
// The write failed with the process alive: put the old bytes back now. If that fails too, the journal stays
// and the next start repairs the file.
if rerr := restoreInPlace(path, prev); rerr != nil {
return fmt.Errorf("config: writing %s failed (%v) and the old content could not be put back (%v) — the next start repairs it from the journal", path, werr, rerr)
}
_ = removeDurable(journal)
return fmt.Errorf("config: writing %s failed, the old content was put back: %w", path, werr)
}
if err := cutPoint("synced"); err != nil {
return err
}
return removeDurable(journal)
}
// RecoverCutWrite repairs a config whose last write was cut. It returns what it did, for the start's log line:
// "" (no journal), "journal removed — the config is whole", or "the old config was put back".
func RecoverCutWrite(path string) (string, error) {
journal := filepath.Join(CutWriteDir, cutWriteJournal)
prev, err := os.ReadFile(journal)
if os.IsNotExist(err) {
return "", nil
}
if err != nil {
return "", fmt.Errorf("config: reading the write journal: %w", err)
}
cur, cerr := os.ReadFile(path)
if cerr == nil && isJSONObject(cur) {
return "journal removed — the config is whole", removeDurable(journal)
}
if !isJSONObject(prev) {
// A journal that is itself broken cannot have been followed by a truncate (it is made durable first), so
// this is not a cut write. Leave both for a person.
return "", fmt.Errorf("config: %s does not parse and the write journal does not parse either", path)
}
if err := restoreInPlace(path, prev); err != nil {
return "", fmt.Errorf("config: putting the old content back into %s: %w", path, err)
}
return "the old config was put back", removeDurable(journal)
}
func isJSONObject(b []byte) bool {
var doc map[string]json.RawMessage
return json.Unmarshal(b, &doc) == nil && doc != nil
}
// restoreInPlace writes content into the EXISTING file (no create: the agent cannot make a file in the config's
// folder), so the file keeps its owner and mode.
func restoreInPlace(path string, content []byte) error {
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, 0)
if err != nil {
return err
}
if _, err := f.Write(content); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
return f.Close()
}
// writeDurable makes path hold content, whole, across a power cut: a new file, synced, renamed, the folder synced.
func writeDurable(path string, content []byte) error {
tmp := path + ".tmp"
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
return err
}
if _, err := f.Write(content); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
if err := f.Close(); err != nil {
return err
}
if err := os.Rename(tmp, path); err != nil {
return err
}
return syncDir(filepath.Dir(path))
}
func removeDurable(path string) error {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return err
}
return syncDir(filepath.Dir(path))
}
func syncDir(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
defer d.Close()
return d.Sync()
}
+140
View File
@@ -0,0 +1,140 @@
package config
import (
"errors"
"os"
"path/filepath"
"testing"
)
const (
oldCfg = `{"proxmox":{"tls":{"fingerprint":"AA:BB"}},"keep":"me"}`
newCfg = `{"proxmox":{"tls":{"ca_file":"/etc/felhom-agent/pve-root-ca.pem","fingerprint":""}},"keep":"me"}`
)
// cutBox is a config file and a journal folder in a temp dir, with the write cut at one named step.
func cutBox(t *testing.T) (cfg string) {
t.Helper()
dir := t.TempDir()
cfg = filepath.Join(dir, "etc", "agent.json")
if err := os.MkdirAll(filepath.Dir(cfg), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(cfg, []byte(oldCfg), 0o600); err != nil {
t.Fatal(err)
}
prevDir, prevCut := CutWriteDir, cutPoint
CutWriteDir = filepath.Join(dir, "state")
t.Cleanup(func() { CutWriteDir, cutPoint = prevDir, prevCut })
return cfg
}
func cutAt(step string) {
cutPoint = func(s string) error {
if s == step {
return errors.New("power cut at " + s)
}
return nil
}
}
func read(t *testing.T, p string) string {
t.Helper()
b, err := os.ReadFile(p)
if err != nil {
t.Fatal(err)
}
return string(b)
}
// The consequence that matters: after a power cut at ANY step of the write and the repair at the next start, the
// config is the whole old file or the whole new file — never a broken one — and Load reads it.
func TestRecoverCutWrite_AfterACutAtEveryStepTheConfigIsWholeAndLoads(t *testing.T) {
want := map[string]string{
"journal-written": oldCfg, // nothing was touched yet
"truncated": oldCfg, // the file is EMPTY at the cut
"half-written": oldCfg, // the file is HALF at the cut
"synced": newCfg, // the new file is whole; only the journal is left
}
for step, wantAfter := range want {
t.Run(step, func(t *testing.T) {
cfg := cutBox(t)
cutAt(step)
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
t.Fatal("the staged cut did not stop the write")
}
atCut := read(t, cfg)
if (step == "truncated" || step == "half-written") && isJSONObject([]byte(atCut)) {
t.Fatalf("the staging is wrong: at %s the file must be broken, it is %q", step, atCut)
}
cutPoint = func(string) error { return nil } // the next start: no cut
note, err := RecoverCutWrite(cfg)
if err != nil {
t.Fatalf("repair: %v", err)
}
if note == "" {
t.Fatal("the repair saw no journal — a cut write would stay broken")
}
if got := read(t, cfg); got != wantAfter {
t.Fatalf("after a cut at %s and the repair the config is %q, want %q", step, got, wantAfter)
}
if _, err := Load(cfg); err != nil {
t.Fatalf("the agent does not start with the repaired config: %v", err)
}
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
t.Fatalf("the journal stayed after the repair (%v) — an old journal would undo a later change", err)
}
})
}
}
// A write that is not cut leaves the new file and NO journal: a journal that stayed would put an old config back
// at some later start.
func TestReplaceInPlace_AWholeWriteLeavesTheNewFileAndNoJournal(t *testing.T) {
cfg := cutBox(t)
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err != nil {
t.Fatal(err)
}
if got := read(t, cfg); got != newCfg {
t.Fatalf("config = %q", got)
}
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
t.Fatalf("the journal stayed after a whole write: %v", err)
}
if note, err := RecoverCutWrite(cfg); err != nil || note != "" {
t.Fatalf("a start after a whole write must do nothing: note=%q err=%v", note, err)
}
if got := read(t, cfg); got != newCfg {
t.Fatalf("the start changed a whole config to %q", got)
}
}
// No journal folder, no write: the old file stays as it was.
func TestReplaceInPlace_WithoutAJournalNothingIsWritten(t *testing.T) {
cfg := cutBox(t)
// the journal cannot be made: a folder sits where its new file must be written
if err := os.MkdirAll(filepath.Join(CutWriteDir, cutWriteJournal+".tmp"), 0o700); err != nil {
t.Fatal(err)
}
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
t.Fatal("the write went on with no journal")
}
if got := read(t, cfg); got != oldCfg {
t.Fatalf("the config changed with no journal: %q", got)
}
}
// A config that is broken with NO journal is not a cut write: the repair does not touch it, and says nothing.
func TestRecoverCutWrite_ABrokenConfigWithNoJournalIsLeftAlone(t *testing.T) {
cfg := cutBox(t)
if err := os.WriteFile(cfg, []byte(`{"half":`), 0o600); err != nil {
t.Fatal(err)
}
note, err := RecoverCutWrite(cfg)
if err != nil || note != "" {
t.Fatalf("note=%q err=%v", note, err)
}
if got := read(t, cfg); got != `{"half":` {
t.Fatalf("the repair wrote a config it had no journal for: %q", got)
}
}
+12
View File
@@ -0,0 +1,12 @@
package config
import "sync"
// FileMu is held by EVERY read-modify-write of agent.json inside this process: the trust move (pvetrust), the PBS
// seed (pbsdr) and the backup-target move (localapi). Each of them reads the whole file, changes one section and
// writes the whole file back in place — without one lock two of them could interleave, and the second writer would
// put back the document it read BEFORE the first one's change (a silently reverted `escrow.pbs_storage_id`,
// `backup.local_backup_target` or trust section). Found by review on 2026-11-14, the night the third writer was
// added. Hold it for the whole read … write, never across a call to Proxmox or the root wrapper.
// Pinned by TestAgentJSONWritersTakeTheSharedLock (cmd/felhom-agent).
var FileMu sync.Mutex
+12 -2
View File
@@ -1,5 +1,7 @@
package escrow
import "encoding/json"
// Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by
// felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API
// re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers
@@ -43,7 +45,15 @@ type CeremonyOutput struct {
IdentityBlobBytes int `json:"identity_blob_bytes"`
ResticPwSealed bool `json:"restic_pw_sealed"`
Uploaded bool `json:"uploaded"`
// PendingUpload (version 2, R-978) is the hub upload body — the R-wrapped blobs (ciphertext), the key
// fingerprint and the sealed password's hash — that json mode hands back INSTEAD of uploading. The agent
// daemon uploads it only when the customer claims the new recovery code, so a ceremony that is started
// and abandoned never replaces the package the customer's previous code opens. Not a secret without R,
// but never logged. Uploaded is false whenever this is set.
PendingUpload json.RawMessage `json:"pending_upload,omitempty"`
}
// CeremonyOutputVersion is the current CeremonyOutput.Version value.
const CeremonyOutputVersion = 1
// CeremonyOutputVersion is the current CeremonyOutput.Version value. 2 (R-978): json mode no longer
// uploads; it returns PendingUpload. The daemon refuses any other version (a binary/daemon mismatch
// fails the ceremony instead of losing a package).
const CeremonyOutputVersion = 2
+2 -2
View File
@@ -21,10 +21,10 @@ import (
// The four inputs (the consumption contract — spike findings):
// - blob: the opaque R-wrapped escrow blob (hub-served restore directive in 10D).
// - recoveryCode (R): BY HAND from the customer — NEVER hub-sourced (zero-knowledge holds: a
// hub compromise alone cannot decrypt). A secret: never logged/persisted.
// hub compromise alone cannot decrypt). A secret: never logged/persisted.
// - expectedFingerprint: hub-served — the gate target (F-C4).
// - keyDest: where the PBS restore reads the key (`--keyfile` or the default
// `$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json`).
// `$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json`).
//
// Order (spike F-C2/F-C3/F-C4/F-C6): Unwrap → fingerprint-gate → install. On ANY failure there is
// NO partial install (nothing at keyDest) and no key material leaks: the recovered key lives only
+50 -10
View File
@@ -59,8 +59,26 @@ var (
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
// which is the one fact the customer needs to recognise it.
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
// ErrRetainedUnchecked — the code did NOT open the current package, and NOT every earlier package the hub
// holds for this host was tried (R-304, 2026-10-08): the retained list could not be fetched, the hub
// withheld rows (over its serve cap, or rows with no key material), a package was malformed, or the
// attempt cap stopped the loop. So „the code is wrong" is NOT known — it may be right for a package
// nobody tried. Distinct from a mistype for exactly the R-224 reason: never accuse the customer of
// something we did not check.
ErrRetainedUnchecked = errors.New("escrow: the recovery code did not open the current sealed package, and some earlier packages were NOT checked")
)
// RetainedUncheckedError wraps ErrRetainedUnchecked with how many earlier packages went unchecked
// (-1 = unknown: the retained list itself could not be read). No secret.
type RetainedUncheckedError struct {
Unchecked int
}
func (e *RetainedUncheckedError) Error() string {
return fmt.Sprintf("%s (unchecked=%d)", ErrRetainedUnchecked.Error(), e.Unchecked)
}
func (e *RetainedUncheckedError) Unwrap() error { return ErrRetainedUnchecked }
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
// carries no secret — not the code, not the bundle, not the repository password.
type RetainedMatch struct {
@@ -102,8 +120,10 @@ type RetainedBlob struct {
}
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
// slice is a clean "none". R-311.
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error)
// slice is a clean "none". R-311. `withheld` counts the earlier packages the hub holds that are NOT in
// blobs — rows with no key material, rows over the hub's serve cap, and packages dropped as malformed
// (R-304): each is a package the code was never tried against.
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, withheld int, err error)
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
type OffsiteKeyRecoverer struct {
@@ -154,9 +174,14 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
// about our own incuriosity, read by the customer as a statement about their code.
if m, ok := r.tryRetained(ctx, recoveryCode); ok {
m, ok, unchecked := r.tryRetained(ctx, recoveryCode)
if ok {
return "", &RetainedOpenedError{Match: m}
}
// R-304: only when EVERY earlier package the hub holds was tried may this stay a wrong code.
if unchecked != 0 {
return "", &RetainedUncheckedError{Unchecked: unchecked}
}
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
}
if bundle.ResticRepoPassword == "" {
@@ -173,23 +198,38 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
// function breaking is the behaviour we had before it existed.
//
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) {
//
// R-304 (2026-10-08): it also returns how many earlier packages were NOT tried — `withheld` from the hub, plus
// the ones past the attempt cap — or -1 when the retained list could not be read at all. A nil FetchRetained
// (an agent wired without the lookup) reports 0: the pre-R-311 refusal, unchanged.
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool, int) {
if r.FetchRetained == nil {
return RetainedMatch{}, false
return RetainedMatch{}, false, 0
}
blobs, _, err := r.FetchRetained(ctx)
if err != nil || len(blobs) == 0 {
return RetainedMatch{}, false
blobs, withheld, err := r.FetchRetained(ctx)
if err != nil {
return RetainedMatch{}, false, -1
}
if withheld < 0 {
withheld = 0
}
if len(blobs) == 0 {
return RetainedMatch{}, false, withheld
}
limit := r.MaxRetainedTried
if limit <= 0 {
limit = defaultMaxRetainedTried
}
unchecked := withheld
if len(blobs) > limit {
unchecked += len(blobs) - limit
}
for i, rb := range blobs {
if i >= limit {
break
}
if len(rb.Blob) == 0 {
unchecked++
continue
}
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
@@ -204,7 +244,7 @@ func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode strin
// correct and must be told so — but the history behind it still cannot be reopened, and
// saying otherwise would be a promise this path cannot keep.
HasResticPassword: bundle.ResticRepoPassword != "",
}, true
}, true, 0
}
return RetainedMatch{}, false
return RetainedMatch{}, false, unchecked
}
+76 -2
View File
@@ -115,8 +115,9 @@ func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
}
}
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The
// worst outcome of this feature breaking is the behaviour we had before it.
// FAIL-SAFE — if the retained lookup itself fails, the lookup's error never reaches the customer and no
// retained package is claimed. R-304 (2026-10-08) changed what stands instead: not the wrong-code refusal (the
// earlier packages were never tried, so „wrong" is not known) but ErrRetainedUnchecked with Unchecked = -1.
//
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
// gets a new, unexplained failure mode → this FAILS.
@@ -140,6 +141,10 @@ func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
if containsStr(err.Error(), "hub exploded") {
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
}
var ue *RetainedUncheckedError
if !errors.As(err, &ue) || ue.Unchecked != -1 {
t.Errorf("err = %v, want RetainedUncheckedError{-1} — the earlier packages were never tried (R-304)", err)
}
}
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
@@ -228,3 +233,72 @@ func containsStr(hay, needle string) bool {
return false
})()
}
// ── R-304 (2026-10-08) — „wrong code" only when every earlier package was tried ──────────────────────
//
// The consequence asserted: is the refusal a WRONG CODE (the only error the screen may answer with „check your
// typing")? It may be only when the hub withheld nothing and every served package was tried.
//
// RED-PROOF: make tryRetained return 0 for `unchecked` → the three „unchecked" cases return the plain refusal
// → they FAIL; the all-tried case keeps passing.
func TestR304_WrongCodeOnlyWhenEveryEarlierPackageWasTried(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "4444567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
other := sealBundle(t, IdentityBundle{ResticRepoPassword: "5555567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
const code = "a code that opens nothing whatsoever in this test"
cases := []struct {
name string
blobs int
withheld int
limit int
unchecked int // 0 = must be a plain wrong code
}{
{"all tried, nothing withheld", 2, 0, 6, 0},
{"the hub withheld rows (no key material / over its cap)", 1, 3, 6, 3},
{"only withheld rows, nothing served", 0, 2, 6, 2},
{"the attempt cap stopped the loop", 5, 0, 2, 3},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
blobs := make([]RetainedBlob, 0, c.blobs)
for i := 0; i < c.blobs; i++ {
blobs = append(blobs, RetainedBlob{Blob: other, SupersededAt: "2026-08-01 00:00:00", Index: i})
}
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
return blobs, c.withheld, nil
},
MaxRetainedTried: c.limit,
}.RecoverOffsiteRepoPassword(context.Background(), code)
if err == nil {
t.Fatal("a code that opens nothing succeeded")
}
var ue *RetainedUncheckedError
isUnchecked := errors.As(err, &ue)
if c.unchecked == 0 {
if isUnchecked {
t.Fatalf("every earlier package was tried — this IS a wrong code, got %v", err)
}
return
}
if !isUnchecked || ue.Unchecked != c.unchecked || !errors.Is(err, ErrRetainedUnchecked) {
t.Fatalf("err = %v, want RetainedUncheckedError{%d} — never call the code wrong when packages were not tried", err, c.unchecked)
}
})
}
}
// A malformed (empty) served package counts as not tried.
func TestR304_EmptyServedPackageCountsAsUnchecked(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "6666567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: nil, SupersededAt: "2026-08-01 00:00:00"}),
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
var ue *RetainedUncheckedError
if !errors.As(err, &ue) || ue.Unchecked != 1 {
t.Fatalf("err = %v, want RetainedUncheckedError{1}", err)
}
}
+22 -1
View File
@@ -22,6 +22,11 @@ type Loop struct {
mu sync.Mutex
fetched bool // a desired-state document has been delivered (false = never a teardown signal)
block *hub.WireWireguard
// passed (R-1194) is set once the first reconcile pass has run. Until then the instance has never been rendered,
// so its health says nothing — and the first host report, sent before that pass, used to read „config invalid,
// port 0" and raise a false oob_degraded on EVERY fresh box (rehearsal 2026-10-20; reproduced 2026-10-21: the
// report at 20:34:00 read an absent config, the apply landed 0.4 s later).
passed bool
nudge chan struct{}
}
@@ -85,6 +90,7 @@ func (l *Loop) reconcile(ctx context.Context) {
if !fetched {
applyBlock = nil
}
defer l.markPassed()
port, err := l.mgr.Apply(ctx, applyBlock)
if err != nil {
return // Apply logged; a claim/exhaustion or install error — retry next tick
@@ -106,8 +112,23 @@ func oobPeerIP(block *hub.WireWireguard) string {
return block.OOBPeerIP
}
// OOBStatus implements the hub collector's reporter seam (Part 4).
// markPassed records that a reconcile pass has run (R-1194) — after a failed Apply too, so a real, lasting failure
// is still reported from the next report on.
func (l *Loop) markPassed() {
l.mu.Lock()
l.passed = true
l.mu.Unlock()
}
// OOBStatus implements the hub collector's reporter seam (Part 4). nil — no stanza, which the hub does not evaluate —
// until the first reconcile pass has run (R-1194).
func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus {
l.mu.Lock()
passed := l.passed
l.mu.Unlock()
if !passed {
return nil
}
_, block := l.snapshot()
return l.mgr.Status(ctx, block)
}
@@ -0,0 +1,29 @@
package felhomsshd
import (
"context"
"testing"
"time"
)
// R-1194: on every fresh box the first host report — sent before the OOB loop's first pass — read „config invalid,
// port 0" (the config was not rendered yet) and the hub mailed oob_degraded. Reproduced 2026-10-21 on a fresh box
// (felhom.eu audits/night-2026-10-21/P5-checks/): the report at 20:34:00, the apply 0.4 s later, the daemon healthy.
// The loop now reports NO stanza (the hub does not evaluate a missing one) until a pass has run — a failed pass
// counts, so a real lasting failure is still reported from the next report on.
// RED-PROOF: with OOBStatus not gated on `passed`, the first assertion fails (a stanza with config_invalid).
func TestR1194_NoOOBStanzaBeforeTheFirstPass(t *testing.T) {
r := newScriptRunner()
m := NewManager(r, t.TempDir(), nil)
m.isActive = func(context.Context) bool { return false }
m.isFailed = func(context.Context) bool { return false }
m.now = func() time.Time { return time.Unix(1783270000, 0) }
l := NewLoop(m, nil, time.Minute, nil)
if st := l.OOBStatus(context.Background()); st != nil {
t.Fatalf("a stanza before the first pass reads a config that was never rendered: %+v", st)
}
l.reconcile(context.Background()) // fails or succeeds — either way a pass has run
if st := l.OOBStatus(context.Background()); st == nil {
t.Fatal("after the first pass the health must be reported (a real failure must reach the hub)")
}
}
+346
View File
@@ -0,0 +1,346 @@
// Package fstrim is the weekly guest disk trim (R-444, operator ruling `09` §3 decision 139).
//
// Why: a thin pool only ever grows from blocks a guest has already FREED — `fstrim` inside the unprivileged container
// is refused (FITRIM: Operation not permitted), and nothing else on the box gives the blocks back. A full thin pool
// takes every guest on the host read-only, so the pool can reach 100 % from deleted data alone. Measured on demo-hp
// 2026-10-06 09:14Z: `pct fstrim 9201` rc 0 in 24.4 s, pool 65.53 % -> 33.40 %, 18/18 app probes 200, max 1.1 s
// (audits/ten-answers-2026-10-06/r444-measure.txt).
//
// The rule, each part pinned by a test in fstrim_test.go:
// - Weekly: a guest is DUE from Wednesday 10:00 local until it has been trimmed once since then (a box that was off
// on Wednesday catches up at its next eligible hour).
// - Daytime only: a trim starts only between 10:00 and 20:59 local — never in the night window (01:00–06:59) where
// the backups and the restore-tests run (TestEligibleHourNeverInTheNight).
// - Never beside a backup, a restore-test or another heavy operation: the pass holds the host-wide one-heavy-op gate
// (backup.InFlight) for its whole run; a busy gate DEFERS the pass to the next hourly tick.
// - A failed trim is retried at the next eligible hour, at most MaxAttemptsPerWeek times in one week.
// - The last result per guest (time, bytes, ok/fail) is persisted, so a restart neither loses it nor re-trims.
//
// The command is the ONE exact sudoers shape `pct fstrim <vmid>` (FELHOM_FSTRIM). Only guests from the pool-verified
// source (ListLXC ∩ the felhom pool, audit A1) and only RUNNING ones are trimmed.
package fstrim
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// Schedule. The weekday/hours are fixed on purpose (one sentence the operator can read on the System page).
const (
Weekday = time.Wednesday
StartHour = 10 // first eligible local hour (inclusive)
EndHour = 21 // first NOT-eligible local hour (exclusive): last start is 20:59
MaxAttemptsPerWeek = 3
// TickInterval is how often the job looks; a deferred or failed pass is therefore retried the next hour.
TickInterval = time.Hour
// FirstTickDelay lets the agent settle after a start before the first look.
FirstTickDelay = 5 * time.Minute
// PerGuestTimeout bounds one `pct fstrim` (measured 24.4 s for 84 GiB).
PerGuestTimeout = 30 * time.Minute
)
// ScheduleText is the human description carried on the host report.
const ScheduleText = "weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
// Runner runs a host command (proxmox.ExecRunner in production, through `sudo -n`).
type Runner interface {
Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
}
// GuestSource yields the guests this agent OWNS (the pool-verified source, never a bare ListLXC).
type GuestSource interface {
Guests(ctx context.Context) ([]proxmox.Guest, error)
}
// Gate is the host-wide one-heavy-operation gate (*backup.InFlight).
type Gate interface {
TryAcquire(what string) (release func(), busy string, ok bool)
}
// GateName is what the gate reports as busy while a trim runs.
const GateName = "guest-fstrim"
// Record is one guest's last trim attempt, as persisted.
type Record struct {
LastAttemptAt time.Time `json:"last_attempt_at"`
OK bool `json:"ok"`
BytesTrimmed int64 `json:"bytes_trimmed"`
Mounts int `json:"mounts"`
DurationSeconds float64 `json:"duration_seconds"`
LastOKAt time.Time `json:"last_ok_at,omitempty"`
Error string `json:"error,omitempty"`
// Attempts counts the attempts since the current week's due time (reset by the first attempt of a new week).
Attempts int `json:"attempts"`
}
// Trimmer is the weekly job.
type Trimmer struct {
runner Runner
guests GuestSource
gate Gate
statePath string
logger *slog.Logger
loc *time.Location
now func() time.Time
mu sync.Mutex
records map[int]Record
}
// New builds the job and loads the persisted state. A missing state file is an empty state; a corrupt one is logged
// and treated as empty (the cost is one extra trim, never a missed one).
func New(runner Runner, guests GuestSource, gate Gate, statePath string, logger *slog.Logger) *Trimmer {
if logger == nil {
logger = slog.Default()
}
t := &Trimmer{runner: runner, guests: guests, gate: gate, statePath: statePath, logger: logger,
loc: time.Local, now: time.Now, records: map[int]Record{}}
t.load()
return t
}
func (t *Trimmer) load() {
data, err := os.ReadFile(t.statePath)
if err != nil {
if !os.IsNotExist(err) {
t.logger.Warn("fstrim: state read failed — starting empty", "path", t.statePath, "err", err)
}
return
}
var raw map[string]Record
if err := json.Unmarshal(data, &raw); err != nil {
t.logger.Warn("fstrim: state file corrupt — starting empty", "path", t.statePath, "err", err)
return
}
for k, r := range raw {
if id, err := strconv.Atoi(k); err == nil && id > 0 {
t.records[id] = r
}
}
}
func (t *Trimmer) saveLocked() error {
raw := make(map[string]Record, len(t.records))
for id, r := range t.records {
raw[strconv.Itoa(id)] = r
}
data, err := json.MarshalIndent(raw, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(t.statePath), 0o755); err != nil {
return err
}
tmp := t.statePath + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, t.statePath)
}
// EligibleHour reports whether a trim may START at local time lt.
func EligibleHour(lt time.Time) bool {
h := lt.Hour()
return h >= StartHour && h < EndHour
}
// weekAnchor is the most recent Wednesday StartHour:00 at or before lt (same location as lt).
func weekAnchor(lt time.Time) time.Time {
daysBack := (int(lt.Weekday()) - int(Weekday) + 7) % 7
d := lt.AddDate(0, 0, -daysBack)
a := time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
if a.After(lt) {
d = d.AddDate(0, 0, -7)
a = time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
}
return a
}
// due reports whether a guest with record r (ok=false: none) is due at local time lt.
func due(r Record, has bool, lt time.Time) bool {
if !has {
return true
}
anchor := weekAnchor(lt)
if r.LastAttemptAt.Before(anchor) {
return true // not tried this week
}
return !r.OK && r.Attempts < MaxAttemptsPerWeek
}
// Run looks every TickInterval until ctx ends. It never returns an error: a failed trim is a reported fact.
func (t *Trimmer) Run(ctx context.Context) {
t.logger.Info("fstrim: weekly guest disk trim starting", "schedule", ScheduleText)
timer := time.NewTimer(FirstTickDelay)
defer timer.Stop()
for {
select {
case <-ctx.Done():
return
case <-timer.C:
t.Pass(ctx)
timer.Reset(TickInterval)
}
}
}
// Pass is one look: outside the daytime window it does nothing; otherwise it trims every due, running, owned guest
// while holding the heavy-op gate.
func (t *Trimmer) Pass(ctx context.Context) {
lt := t.now().In(t.loc)
if !EligibleHour(lt) {
t.logger.Debug("fstrim: outside the daytime window — not looking", "local", lt.Format("Mon 15:04"))
return
}
guests, err := t.guests.Guests(ctx)
if err != nil {
t.logger.Warn("fstrim: owned-guest list unavailable — skipping this pass", "err", err)
return
}
owned := make(map[int]bool, len(guests))
var todo []int
t.mu.Lock()
for _, g := range guests {
owned[g.VMID] = true
r, has := t.records[g.VMID]
if !due(r, has, lt) {
continue
}
if g.Status != "running" {
t.logger.Info("fstrim: guest not running — trimmed when it runs", "vmid", g.VMID, "status", g.Status)
continue
}
todo = append(todo, g.VMID)
}
// A guest the agent no longer owns has no result to report.
pruned := false
for id := range t.records {
if !owned[id] {
delete(t.records, id)
pruned = true
}
}
if pruned {
if err := t.saveLocked(); err != nil {
t.logger.Warn("fstrim: state save failed", "err", err)
}
}
t.mu.Unlock()
if len(todo) == 0 {
return
}
sort.Ints(todo)
release, busy, ok := t.gate.TryAcquire(GateName)
if !ok {
t.logger.Info("fstrim: deferred — a heavy operation is in flight; retrying next hour", "busy", busy, "due_guests", len(todo))
return
}
defer release()
for _, vmid := range todo {
if ctx.Err() != nil {
return
}
t.trimOne(ctx, vmid, lt)
}
}
var trimmedLine = regexp.MustCompile(`\((\d+) bytes\) trimmed`)
// ParseTrimmed sums the "(N bytes) trimmed" lines of `pct fstrim` output and counts them (one per mount point), e.g.
// `/var/lib/lxc/9201/rootfs/: 30.1 GiB (32277680128 bytes) trimmed`.
func ParseTrimmed(out string) (bytes int64, mounts int) {
for _, m := range trimmedLine.FindAllStringSubmatch(out, -1) {
n, err := strconv.ParseInt(m[1], 10, 64)
if err != nil {
continue
}
bytes += n
mounts++
}
return bytes, mounts
}
// GiB renders bytes as "30.1 GiB".
func GiB(b int64) string { return fmt.Sprintf("%.1f GiB", float64(b)/(1<<30)) }
func (t *Trimmer) trimOne(ctx context.Context, vmid int, lt time.Time) {
start := t.now()
cctx, cancel := context.WithTimeout(ctx, PerGuestTimeout)
stdout, stderr, err := t.runner.Run(cctx, "pct", "fstrim", strconv.Itoa(vmid))
cancel()
dur := t.now().Sub(start)
bytes, mounts := ParseTrimmed(string(stdout) + "\n" + string(stderr))
t.mu.Lock()
prev, has := t.records[vmid]
r := Record{LastAttemptAt: start.UTC(), OK: err == nil, BytesTrimmed: bytes, Mounts: mounts,
DurationSeconds: float64(dur.Round(100*time.Millisecond)) / float64(time.Second), LastOKAt: prev.LastOKAt}
if has && !prev.LastAttemptAt.Before(weekAnchor(lt)) {
r.Attempts = prev.Attempts + 1
} else {
r.Attempts = 1
}
if err == nil {
r.LastOKAt = start.UTC()
} else {
msg := strings.TrimSpace(err.Error() + ": " + strings.TrimSpace(string(stderr)))
if len(msg) > 300 {
msg = msg[:300]
}
r.Error = msg
}
t.records[vmid] = r
saveErr := t.saveLocked()
t.mu.Unlock()
if err == nil {
t.logger.Info(fmt.Sprintf("fstrim: guest %d trimmed %s in %.1fs", vmid, GiB(bytes), r.DurationSeconds),
"vmid", vmid, "bytes_trimmed", bytes, "mounts", mounts, "duration_s", r.DurationSeconds)
if mounts == 0 {
t.logger.Warn("fstrim: pct fstrim succeeded but reported no trimmed mount — output not understood",
"vmid", vmid, "stdout", strings.TrimSpace(string(stdout)))
}
} else {
t.logger.Warn(fmt.Sprintf("fstrim: guest %d trim FAILED after %.1fs", vmid, r.DurationSeconds),
"vmid", vmid, "attempt", r.Attempts, "max_attempts_per_week", MaxAttemptsPerWeek, "err", r.Error)
}
if saveErr != nil {
t.logger.Warn("fstrim: state save failed — the result will not survive a restart", "path", t.statePath, "err", saveErr)
}
}
// GuestDiskTrimStatus implements hub.GuestDiskTrimReporter: a pure read of the persisted results (never runs pct).
func (t *Trimmer) GuestDiskTrimStatus(context.Context) *hub.GuestDiskTrimStatus {
t.mu.Lock()
defer t.mu.Unlock()
out := &hub.GuestDiskTrimStatus{Schedule: ScheduleText}
ids := make([]int, 0, len(t.records))
for id := range t.records {
ids = append(ids, id)
}
sort.Ints(ids)
for _, id := range ids {
r := t.records[id]
g := hub.GuestDiskTrim{VMID: id, LastAttemptAt: r.LastAttemptAt.UTC().Format(time.RFC3339), OK: r.OK,
BytesTrimmed: r.BytesTrimmed, Mounts: r.Mounts, DurationSeconds: r.DurationSeconds, Error: r.Error}
if !r.LastOKAt.IsZero() {
g.LastOKAt = r.LastOKAt.UTC().Format(time.RFC3339)
}
out.Guests = append(out.Guests, g)
}
return out
}
+267
View File
@@ -0,0 +1,267 @@
package fstrim
import (
"bytes"
"context"
"encoding/json"
"errors"
"log/slog"
"path/filepath"
"reflect"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// The real `pct fstrim 9201` output measured on demo-hp 2026-10-06 (audits/ten-answers-2026-10-06/r444-measure.txt).
const measuredOut = "/var/lib/lxc/9201/rootfs/: 30.1 GiB (32277680128 bytes) trimmed\n" +
"/var/lib/lxc/9201/rootfs/var/lib/felhom: 53.9 GiB (57865633792 bytes) trimmed\n"
const measuredBytes = int64(32277680128 + 57865633792)
type fakeRunner struct {
mu sync.Mutex
calls [][]string
out string
err error
onRun func()
}
func (f *fakeRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
f.mu.Lock()
f.calls = append(f.calls, append([]string{name}, args...))
f.mu.Unlock()
if f.onRun != nil {
f.onRun()
}
if f.err != nil {
return nil, []byte("mount busy"), f.err
}
return []byte(f.out), nil, nil
}
type fakeGuests struct {
g []proxmox.Guest
err error
}
func (f fakeGuests) Guests(context.Context) ([]proxmox.Guest, error) { return f.g, f.err }
// A Wednesday 10:30 in a fixed zone (CEST-like), so the tests do not depend on the machine's zone.
var zone = time.FixedZone("CEST", 2*3600)
func at(day, hour, min int) time.Time { return time.Date(2026, 10, day, hour, min, 0, 0, zone) } // 2026-10-07 = Wednesday
func newT(t *testing.T, r Runner, g GuestSource, gate Gate, now *time.Time) (*Trimmer, *bytes.Buffer, string) {
t.Helper()
var logs bytes.Buffer
path := filepath.Join(t.TempDir(), "guest-disk-trim.json")
tr := New(r, g, gate, path, slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelDebug})))
tr.loc = zone
tr.now = func() time.Time { return *now }
return tr, &logs, path
}
func running(ids ...int) fakeGuests {
var g []proxmox.Guest
for _, id := range ids {
g = append(g, proxmox.Guest{VMID: id, Status: "running", Type: "lxc"})
}
return fakeGuests{g: g}
}
func TestParseTrimmedTheMeasuredOutput(t *testing.T) {
b, m := ParseTrimmed(measuredOut)
if b != measuredBytes || m != 2 {
t.Fatalf("ParseTrimmed = %d bytes over %d mounts, want %d over 2", b, m, measuredBytes)
}
if b, m := ParseTrimmed("something else\n"); b != 0 || m != 0 {
t.Fatalf("unrelated output parsed as %d/%d", b, m)
}
if got := GiB(measuredBytes); got != "84.0 GiB" {
t.Fatalf("GiB = %q", got)
}
}
// The night window (01:00–06:59) must never be eligible, and the daytime window is exactly 10:00–20:59.
func TestEligibleHourNeverInTheNight(t *testing.T) {
for h := 0; h < 24; h++ {
lt := time.Date(2026, 10, 7, h, 30, 0, 0, zone)
got := EligibleHour(lt)
if h >= 1 && h <= 6 && got {
t.Errorf("hour %02d is in the night window and must not be eligible", h)
}
if want := h >= 10 && h <= 20; got != want {
t.Errorf("EligibleHour(%02d:30) = %v, want %v", h, got, want)
}
}
}
func TestWeekAnchorIsTheLastWednesdayTen(t *testing.T) {
cases := map[time.Time]time.Time{
at(7, 10, 0): at(7, 10, 0), // Wednesday 10:00 itself
at(7, 9, 59): time.Date(2026, 9, 30, 10, 0, 0, 0, zone), // before 10:00 Wednesday → the previous week
at(8, 15, 0): at(7, 10, 0), // Thursday
at(13, 20, 0): at(7, 10, 0), // next Tuesday
at(14, 11, 0): at(14, 10, 0), // next Wednesday
}
for in, want := range cases {
if got := weekAnchor(in); !got.Equal(want) {
t.Errorf("weekAnchor(%s) = %s, want %s", in.Format("Mon 01-02 15:04"), got.Format("Mon 01-02 15:04"), want.Format("Mon 01-02 15:04"))
}
}
}
// The consequence: on Wednesday 10:30 a running owned guest is trimmed with the ONE exact argv, the bytes are parsed,
// the positive log line is written, the result is persisted, and the host report carries it.
func TestPassTrimsADueGuestAndReportsIt(t *testing.T) {
now := at(7, 10, 30)
r := &fakeRunner{out: measuredOut}
tr, logs, path := newT(t, r, running(9201), &backup.InFlight{}, &now)
tr.Pass(context.Background())
if want := [][]string{{"pct", "fstrim", "9201"}}; !reflect.DeepEqual(r.calls, want) {
t.Fatalf("runner calls = %q, want %q", r.calls, want)
}
if !strings.Contains(logs.String(), "fstrim: guest 9201 trimmed 84.0 GiB in ") {
t.Fatalf("no positive per-guest log line:\n%s", logs.String())
}
st := tr.GuestDiskTrimStatus(context.Background())
if st == nil || st.Schedule != ScheduleText || len(st.Guests) != 1 {
t.Fatalf("report stanza = %+v", st)
}
g := st.Guests[0]
if g.VMID != 9201 || !g.OK || g.BytesTrimmed != measuredBytes || g.Mounts != 2 || g.LastOKAt == "" || g.LastAttemptAt == "" {
t.Fatalf("report guest = %+v", g)
}
// Persisted: a NEW Trimmer over the same file (an agent restart) still has it and does not trim again this week.
now = at(8, 11, 0)
r2 := &fakeRunner{out: measuredOut}
tr2 := New(r2, running(9201), &backup.InFlight{}, path, slog.New(slog.NewTextHandler(&bytes.Buffer{}, nil)))
tr2.loc, tr2.now = zone, func() time.Time { return now }
if st2 := tr2.GuestDiskTrimStatus(context.Background()); len(st2.Guests) != 1 || st2.Guests[0].BytesTrimmed != measuredBytes {
t.Fatalf("result lost over a restart: %+v", st2)
}
tr2.Pass(context.Background())
if len(r2.calls) != 0 {
t.Fatalf("trimmed again in the same week after a restart: %q", r2.calls)
}
// Next week it is due again.
now = at(14, 10, 5)
tr2.Pass(context.Background())
if len(r2.calls) != 1 {
t.Fatalf("not trimmed in the next week: %q", r2.calls)
}
}
func TestPassNeverRunsInTheNight(t *testing.T) {
for _, h := range []int{1, 3, 6, 9, 21, 23} {
now := at(7, h, 15)
r := &fakeRunner{out: measuredOut}
tr, _, _ := newT(t, r, running(9201), &backup.InFlight{}, &now)
tr.Pass(context.Background())
if len(r.calls) != 0 {
t.Errorf("trimmed at %02d:15: %q", h, r.calls)
}
}
}
// A backup (or restore-test) holding the heavy-op gate DEFERS the trim; the next hour, gate free, it runs. And while
// a trim runs, the gate is held, so a backup cannot start beside it.
func TestPassDefersToAHeavyOperationAndRetriesNextHour(t *testing.T) {
now := at(7, 10, 30)
gate := &backup.InFlight{}
release, _, _ := gate.TryAcquire("backup:9201")
var busyDuringTrim string
r := &fakeRunner{out: measuredOut}
r.onRun = func() { busyDuringTrim = gate.Busy() }
tr, logs, _ := newT(t, r, running(9201), gate, &now)
tr.Pass(context.Background())
if len(r.calls) != 0 {
t.Fatalf("trimmed beside a running backup: %q", r.calls)
}
if !strings.Contains(logs.String(), "fstrim: deferred") || !strings.Contains(logs.String(), "backup:9201") {
t.Fatalf("the deferral is not logged with what holds the gate:\n%s", logs.String())
}
release()
now = now.Add(time.Hour)
tr.Pass(context.Background())
if len(r.calls) != 1 {
t.Fatalf("not retried the next hour: %q", r.calls)
}
if busyDuringTrim != GateName {
t.Fatalf("the heavy-op gate was %q during the trim, want %q", busyDuringTrim, GateName)
}
if gate.Busy() != "" {
t.Fatalf("the gate was not released after the pass: %q", gate.Busy())
}
}
func TestFailedTrimWarnsIsRecordedAndRetriedAtMostThreeTimes(t *testing.T) {
now := at(7, 10, 30)
r := &fakeRunner{err: errors.New("exit status 255")}
tr, logs, _ := newT(t, r, running(9201), &backup.InFlight{}, &now)
for i := 0; i < 6; i++ {
tr.Pass(context.Background())
now = now.Add(time.Hour)
}
if len(r.calls) != MaxAttemptsPerWeek {
t.Fatalf("attempts in one week = %d, want %d", len(r.calls), MaxAttemptsPerWeek)
}
if !strings.Contains(logs.String(), "level=WARN") || !strings.Contains(logs.String(), "fstrim: guest 9201 trim FAILED") {
t.Fatalf("no WARN for the failure:\n%s", logs.String())
}
g := tr.GuestDiskTrimStatus(context.Background()).Guests[0]
if g.OK || g.LastOKAt != "" || !strings.Contains(g.Error, "exit status 255") || !strings.Contains(g.Error, "mount busy") {
t.Fatalf("failed result not recorded as a failure: %+v", g)
}
// A success later keeps a clean record.
r.err = nil
r.out = measuredOut
now = at(14, 10, 10)
tr.Pass(context.Background())
if g := tr.GuestDiskTrimStatus(context.Background()).Guests[0]; !g.OK || g.Error != "" || g.BytesTrimmed != measuredBytes {
t.Fatalf("success after failure: %+v", g)
}
}
func TestOnlyRunningOwnedGuestsAndAFailedListActsOnNothing(t *testing.T) {
now := at(7, 10, 30)
r := &fakeRunner{out: measuredOut}
g := fakeGuests{g: []proxmox.Guest{{VMID: 9201, Status: "stopped"}, {VMID: 9202, Status: "running"}}}
tr, _, _ := newT(t, r, g, &backup.InFlight{}, &now)
tr.Pass(context.Background())
if want := [][]string{{"pct", "fstrim", "9202"}}; !reflect.DeepEqual(r.calls, want) {
t.Fatalf("calls = %q, want only the running guest", r.calls)
}
r2 := &fakeRunner{out: measuredOut}
tr2, logs, _ := newT(t, r2, fakeGuests{err: errors.New("pool read 403")}, &backup.InFlight{}, &now)
tr2.Pass(context.Background())
if len(r2.calls) != 0 || !strings.Contains(logs.String(), "owned-guest list unavailable") {
t.Fatalf("a failed ownership read must act on nothing: calls %q", r2.calls)
}
}
func TestReportJSONShape(t *testing.T) {
now := at(7, 10, 30)
tr, _, _ := newT(t, &fakeRunner{out: measuredOut}, running(9201), &backup.InFlight{}, &now)
tr.Pass(context.Background())
b, err := json.Marshal(tr.GuestDiskTrimStatus(context.Background()))
if err != nil {
t.Fatal(err)
}
for _, k := range []string{`"schedule":`, `"guests":[{"vmid":9201`, `"last_attempt_at":"2026-10-07T08:30:00Z"`, `"ok":true`,
`"bytes_trimmed":90143313920`, `"mounts":2`, `"duration_seconds":`, `"last_ok_at":"2026-10-07T08:30:00Z"`} {
if !strings.Contains(string(b), k) {
t.Errorf("report JSON lacks %s: %s", k, b)
}
}
if strings.Contains(string(b), `"error"`) {
t.Errorf("an ok result must omit error: %s", b)
}
}
+2 -2
View File
@@ -50,8 +50,8 @@ func TestParseConfMounts(t *testing.T) {
func TestMissingBindSources(t *testing.T) {
mounts := ParseConfMounts(sampleConf)
present := map[string]bool{
"/mnt/felhom-usb/felhom-data": true, // usb attached
"/var/lib/felhom-agent/guests/9201/bootstrap": true, // bootstrap always present
"/mnt/felhom-usb/felhom-data": true, // usb attached
"/var/lib/felhom-agent/guests/9201/bootstrap": true, // bootstrap always present
// "/mnt/felhom-flash/felhom-data" is ABSENT (drive unplugged)
}
got := MissingBindSources(mounts, func(p string) bool { return present[p] })
+30 -1
View File
@@ -10,6 +10,7 @@ import (
"io"
"log/slog"
"net/http"
"net/url"
"os"
"strings"
"time"
@@ -86,6 +87,7 @@ func (e *HTTPError) Error() string {
// IS the heartbeat (locked decision 1). Errors are typed (transport vs HTTP) and
// never include the bearer token.
func (c *Client) Report(ctx context.Context, r *HostReport) (*ControlEnvelope, error) {
r.SentAt = time.Now().UTC().Format(time.RFC3339Nano) // the box clock at SENDING (R-1384); see HostReport.SentAt
body, err := json.Marshal(r)
if err != nil {
return nil, fmt.Errorf("hub: marshaling report: %w", err)
@@ -285,10 +287,37 @@ func (c *Client) Jobs(ctx context.Context) ([]JobWire, error) {
// /hosts/{id}/jobs/{job_id}, self-scoped. Called after a job is executed OR permanently rejected
// (the nonce is already durably burned on a passing verify, so re-processing is replay-safe).
func (c *Client) CompleteJob(ctx context.Context, jobID string) error {
return c.CompleteJobWithOutcome(ctx, jobID, JobOutcome{})
}
// JobOutcome says HOW a signed job ended when the agent clears it (R-1383, D78): Outcome is done | rejected | failed
// | malformed; Reason is one code of signedjobs.ReasonCodes (empty for done); Op is the job's op class. It rides the
// DELETE's query, so a hub that does not know it ignores it. Until this, „ran” and „refused” looked the same to the
// hub, and a refused agent update left no trace outside the box's journal. No secret: no signature, no nonce, no
// parameter of the job is in it.
type JobOutcome struct {
Outcome string
Reason string
Op string
}
// CompleteJobWithOutcome clears the job and tells the hub how it ended. The zero JobOutcome sends no query.
func (c *Client) CompleteJobWithOutcome(ctx context.Context, jobID string, o JobOutcome) error {
if c.hostID == "" || jobID == "" {
return fmt.Errorf("hub: CompleteJob requires host_id + job_id")
}
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, c.baseURL+"/api/v1/hosts/"+c.hostID+"/jobs/"+jobID, nil)
u := c.baseURL + "/api/v1/hosts/" + c.hostID + "/jobs/" + jobID
if o.Outcome != "" {
q := url.Values{"outcome": {o.Outcome}}
if o.Reason != "" {
q.Set("reason", o.Reason)
}
if o.Op != "" {
q.Set("op", o.Op)
}
u += "?" + q.Encode()
}
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, u, nil)
if err != nil {
return err
}
+76
View File
@@ -0,0 +1,76 @@
package hub
import (
"context"
"encoding/json"
"io"
"net/http"
"testing"
"time"
)
// R-1384 (D79): the hub tells a wrong box clock by the time stamped at the moment of SENDING. A report whose collect
// started long ago (an old reported_at) must still leave with a fresh sent_at, and a second send of the SAME report
// struct must carry a new one — or a report that waited on the box reads as a clock that is behind.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-sent-at.txt).
func TestReport_StampsSentAtAtSending(t *testing.T) {
var bodies []map[string]any
c := testClient(func(r *http.Request) (*http.Response, error) {
raw, _ := io.ReadAll(r.Body)
var m map[string]any
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatalf("body: %v", err)
}
bodies = append(bodies, m)
return httpResp(200, `{"status":"ok"}`), nil
})
old := time.Now().Add(-40 * time.Minute).UTC()
rep := &HostReport{HostID: "h", ReportedAt: old.Format(time.RFC3339), SentAt: old.Format(time.RFC3339Nano)}
for i := 0; i < 2; i++ {
if _, err := c.Report(context.Background(), rep); err != nil {
t.Fatalf("Report: %v", err)
}
time.Sleep(5 * time.Millisecond)
}
var sent []time.Time
for i, m := range bodies {
s, _ := m["sent_at"].(string)
at, err := time.Parse(time.RFC3339Nano, s)
if err != nil {
t.Fatalf("send %d: sent_at = %q, want an RFC3339 time", i, s)
}
if d := time.Since(at); d < 0 || d > 5*time.Second {
t.Errorf("send %d: sent_at is %s old, want the moment of sending (reported_at was 40 minutes old)", i, d)
}
if m["reported_at"] != old.Format(time.RFC3339) {
t.Errorf("send %d: reported_at changed to %v — the collect time is not the send time's to touch", i, m["reported_at"])
}
sent = append(sent, at)
}
if len(sent) == 2 && !sent[1].After(sent[0]) {
t.Errorf("the second send carries %s, not after the first %s — the stamp is not taken per send", sent[1], sent[0])
}
}
// R-1383 (D78): the clearing call says how the job ended, in the DELETE's query; a plain CompleteJob sends none (an
// older hub, and the „nothing to say” case, see the same request as before).
func TestCompleteJob_OutcomeRidesTheQuery(t *testing.T) {
var urls []string
c := testClient(func(r *http.Request) (*http.Response, error) {
if r.Method != http.MethodDelete {
t.Errorf("method = %s", r.Method)
}
urls = append(urls, r.URL.Path+"?"+r.URL.RawQuery)
return httpResp(200, `{"status":"ok"}`), nil
})
if err := c.CompleteJob(context.Background(), "j1"); err != nil {
t.Fatal(err)
}
if err := c.CompleteJobWithOutcome(context.Background(), "j2", JobOutcome{Outcome: "rejected", Reason: "expired", Op: "agent_update"}); err != nil {
t.Fatal(err)
}
want := []string{"/api/v1/hosts/demo-host-01/jobs/j1?", "/api/v1/hosts/demo-host-01/jobs/j2?op=agent_update&outcome=rejected&reason=expired"}
if len(urls) != 2 || urls[0] != want[0] || urls[1] != want[1] {
t.Fatalf("requests = %q, want %q", urls, want)
}
}
+10 -3
View File
@@ -14,6 +14,13 @@ const (
TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED
TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which)
TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting)
// TunnelAbsent (R-1070, agent Unreleased 2026-10-19): the guest answered and has NO cloudflared container — the
// controller makes none when no tunnel token is configured (a LAN-only or test box). Not a failure by itself: the
// hub never alarms on it, and the OS-update host rule reads it as „not applicable" only when the container was
// also absent before the update (HostHealthVerdict). Docker 29 prints „error: no such object" in lower case, so
// before this state existed the case-sensitive match fell through to `unknown` — which failed the host rule on
// every box without a tunnel (Tester4, 2026-10-15).
TunnelAbsent = "absent"
)
// CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec.
@@ -53,7 +60,7 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
return TunnelUnknown, "no running customer guest"
}
worst, wdetail := "", ""
rank := map[string]int{TunnelRunning: 0, TunnelUnknown: 1, TunnelNotRunning: 2}
rank := map[string]int{TunnelRunning: 0, TunnelAbsent: 1, TunnelUnknown: 2, TunnelNotRunning: 3}
for _, v := range vmids {
out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared")
st, d := ClassifyTunnel(string(out), string(errOut), err)
@@ -71,8 +78,8 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
func ClassifyTunnel(stdout, stderr string, err error) (string, string) {
out := strings.TrimSpace(stdout)
if err != nil || out == "" {
if strings.Contains(stderr, "No such object") || strings.Contains(stderr, "No such container") {
return TunnelNotRunning, "no cloudflared container in the guest"
if e := strings.ToLower(stderr); strings.Contains(e, "no such object") || strings.Contains(e, "no such container") {
return TunnelAbsent, "no cloudflared container in the guest (no tunnel set up on this box)"
}
return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err)
}
+4 -1
View File
@@ -20,7 +20,10 @@ func TestClassifyTunnel(t *testing.T) {
{"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"},
{"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"},
{"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelNotRunning, "no cloudflared container"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
// R-1070: Docker 29's wording, measured on demo-hp 2026-10-19 (`docker inspect` of a missing name).
{"absent, docker 29", "", "error: no such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no tunnel set up"},
{"absent, no such container", "", "Error response from daemon: No such container: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
{"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"},
{"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"},
{"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"},
+205 -14
View File
@@ -10,6 +10,8 @@ import (
"log/slog"
"os"
"strings"
"sync"
"sync/atomic"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
@@ -65,6 +67,13 @@ type ProvenRestoreTestReporter interface {
ProvenRestoreTests(ctx context.Context) []RestoreTest
}
// KnownBackupReporter is the newest SUCCESSFUL backup per tier and guest kept on disk (R-894's
// backup-success-state.json). collectBackups folds it into the report so a restart does not erase the
// hub's evidence of a backup that ran minutes before it (the kernel-night shape, 2026-10-09).
type KnownBackupReporter interface {
KnownBackupSuccesses() []Backup
}
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
type PBSReporter interface {
@@ -89,6 +98,18 @@ type GuestNetReporter interface {
GuestNetStatus(ctx context.Context) *GuestNetStatus
}
// GuestDiskTrimReporter is the R-444 seam the weekly trim job plugs into (same consumer-side pattern — hub does not
// import fstrim). nil (feature not wired) → no guest_disk_trim stanza.
type GuestDiskTrimReporter interface {
GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus
}
// LineTestReporter is the R-1247 seam the weekly line test plugs into (hub does not import linetest). nil → no
// line_test stanza.
type LineTestReporter interface {
LineTestStatus(ctx context.Context) *LineTestStatus
}
// Collector builds a HostReport from read-only sources. All deps are behind narrow
// interfaces for unit testing.
type Collector struct {
@@ -98,15 +119,24 @@ type Collector struct {
backups BackupReporter
restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter
knownBackups KnownBackupReporter // R-894 file → host report after a restart (nil → in-memory only)
pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
pveTrust func() any // R-1380: the proxmox_trust stanza (nil = omitted)
guestPowerGaveUp func() any // R-1383: the guest_power_gave_up stanza (nil = omitted)
profileRefresh func() any // R-1383: the profile_refresh stanza (nil = omitted)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
opResults func() []OperatorActionResult // 2026-10-29: the operator actions' answers (nil → none)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
@@ -115,8 +145,11 @@ type Collector struct {
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
hostID string
agentVersion string
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
logger *slog.Logger
now func() time.Time
// swapShortWarned (R-1192): the host-swap-short WARN is logged once per state change, not every report.
swapShortWarned atomic.Bool
}
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
@@ -135,6 +168,7 @@ func NewCollector(px proxmoxReader, cf CloudflaredProber, storage StorageObserve
temp: SysfsTempReader{}, // slice 9: real sysfs reader by default; tests inject a fake
hostID: hostID,
agentVersion: agentVersion,
selfSHA: runningBinarySHA256,
logger: logger,
now: func() time.Time { return time.Now().UTC() },
}
@@ -178,6 +212,24 @@ func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capabi
return c
}
// SetProxmoxTrustSource wires the `proxmox_trust` stanza (R-1380). Optional; nil = the stanza is omitted.
func (c *Collector) SetProxmoxTrustSource(f func() any) *Collector {
c.pveTrust = f
return c
}
// SetProfileRefreshSource wires the `profile_refresh` stanza (R-1383). Optional; nil, or a nil result, omits it.
func (c *Collector) SetProfileRefreshSource(f func() any) *Collector {
c.profileRefresh = f
return c
}
// SetGuestPowerGaveUpSource wires the `guest_power_gave_up` stanza (R-1383). Optional; nil, or a nil result, omits it.
func (c *Collector) SetGuestPowerGaveUpSource(f func() any) *Collector {
c.guestPowerGaveUp = f
return c
}
// SetLeafFingerprint records the served local-API leaf fp (v0.48.0) to ride every host report (the hub
// watches it for a re-key). Static per process — set once at startup. "" when the local API is
// disabled. Returns the collector for chaining.
@@ -205,6 +257,12 @@ type ControllerSupervisorReporter interface {
ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus
}
// SetOperatorActionResults wires the source of the operator actions' answers (2026-10-29; nil → none sent).
func (c *Collector) SetOperatorActionResults(f func() []OperatorActionResult) *Collector {
c.opResults = f
return c
}
// SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe).
func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector {
c.ctrlSup = r
@@ -218,6 +276,36 @@ func (c *Collector) SetGuestNetReporter(g GuestNetReporter) *Collector {
return c
}
// ForeignKeyArchiveReporter is the R-366 slice-2 seam: the restore-test's ledger of archives written with another
// key. nil = not evaluated yet (the stanza is omitted and the hub keeps its state).
type ForeignKeyArchiveReporter interface {
ForeignKeyArchives(ctx context.Context) *ForeignKeyArchivesStanza
}
// SetForeignKeyArchiveReporter wires the restore-test's foreign-key ledger (R-366 slice 2; nil-safe → omitted).
func (c *Collector) SetForeignKeyArchiveReporter(r ForeignKeyArchiveReporter) *Collector {
c.foreignKey = r
return c
}
// SetGuestDiskTrimReporter wires the R-444 weekly trim job as a report source (nil-safe → stanza omitted).
func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector {
c.diskTrim = r
return c
}
// SetLineTestReporter wires the R-1247 weekly line test as a report source (nil-safe → stanza omitted).
func (c *Collector) SetLineTestReporter(r LineTestReporter) *Collector {
c.lineTest = r
return c
}
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
c.deliberateOff = f
return c
}
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
type SelfUpdateReporter interface {
@@ -324,6 +412,27 @@ func (c *Collector) SetOOBReporter(o OOBReporter) *Collector {
// error (no useful report — the cycle skips the POST); a failed per-guest
// GuestConfig degrades that guest to status="unknown" without spec but still sends;
// a cloudflared probe failure yields status="unknown" and is never fatal.
func (c *Collector) profileRefreshStanza() any {
if c.profileRefresh == nil {
return nil
}
return c.profileRefresh()
}
func (c *Collector) guestPowerGaveUpStanza() any {
if c.guestPowerGaveUp == nil {
return nil
}
return c.guestPowerGaveUp()
}
func (c *Collector) proxmoxTrust() any {
if c.pveTrust == nil {
return nil
}
return c.pveTrust()
}
func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
ns, err := c.px.NodeStatus(ctx)
if err != nil {
@@ -331,14 +440,17 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
}
host := hostMetrics(c.px.Node(), ns)
guests := c.collectGuests(ctx)
c.applySwap(&host, ns, guests) // R-1192: host swap vs the guests' swap allowance
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable
report := &HostReport{
HostID: c.hostID,
ReportedAt: c.now().Format(time.RFC3339),
AgentVersion: c.agentVersion,
AgentSHA256: c.agentSHA256(),
Host: host,
Guests: c.collectGuests(ctx),
Guests: guests,
// storage_targets populated this slice (slice 5) via the observer; the rest stay
// defined-but-empty (slice 6). Non-nil so they marshal as [].
StorageTargets: c.collectStorage(ctx),
@@ -346,12 +458,15 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
RestoreTests: c.collectRestoreTests(ctx),
PBSSnapshots: c.collectPBSSnapshots(ctx),
AuditTail: []AuditEntry{},
Cloudflared: c.cloudflared(ctx),
Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP,
Addresses: c.collectAddresses(),
System: c.systemInfo(ctx, ns),
AuditTail: []AuditEntry{},
Cloudflared: c.cloudflared(ctx),
Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP,
ProxmoxTrust: c.proxmoxTrust(),
GuestPowerGaveUp: c.guestPowerGaveUpStanza(),
ProfileRefresh: c.profileRefreshStanza(),
Addresses: c.collectAddresses(),
System: c.systemInfo(ctx, ns),
}
// DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads).
// Secret-free by construction (identifiers/intents/sizes/coordinates only).
@@ -369,10 +484,25 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.ctrlSup != nil {
report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx)
}
if c.opResults != nil {
report.OperatorActionResults = c.opResults()
}
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
if c.guestNet != nil {
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
}
// R-444: the last weekly trim result per guest (nil reporter = not wired → stanza omitted).
if c.diskTrim != nil {
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
}
// R-1247: the last weekly line test (nil reporter = not wired → stanza omitted).
if c.lineTest != nil {
report.LineTest = c.lineTest.LineTestStatus(ctx)
}
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
if c.foreignKey != nil {
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
}
// D1: agent self-update pending status (nil reporter → pending=false, the steady state).
if c.selfUpdate != nil {
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()
@@ -385,6 +515,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.oob != nil {
report.OOB = c.oob.OOBStatus(ctx)
}
// R-998: a household power-off accepted in this boot rides every report until the box is down.
if c.deliberateOff != nil {
report.DeliberateOff = c.deliberateOff()
}
return report, nil
}
@@ -431,8 +565,28 @@ const pbsWrapperPath = "/usr/local/sbin/felhom-pbs-apply"
// unreadable file yields "", which the hub reads as UNKNOWN rather than as drift — a host that
// legitimately has no DR wrapper must not light up amber. The file is 0755, so no privilege is
// needed to read it.
func pbsWrapperSHA256() string {
f, err := os.Open(pbsWrapperPath)
func pbsWrapperSHA256() string { return fileSHA256(pbsWrapperPath) }
// selfExePath is the running binary as the kernel holds it. /proc/self/exe, not the installed path:
// after an A/B flip the file at /usr/local/bin/felhom-agent may already be the NEXT binary while this
// process still runs the old one, and the report must describe what runs (R-349). Test seam.
var selfExePath = "/proc/self/exe"
// runningBinarySHA256 hashes the running binary ONCE per process — the bytes cannot change under a
// running process, and re-hashing ~20 MB every report cycle buys nothing. A failed read is cached as
// "" (UNKNOWN); it never fails the report.
var runningBinarySHA256 = sync.OnceValue(func() string { return fileSHA256(selfExePath) })
func (c *Collector) agentSHA256() string {
if c.selfSHA == nil {
return ""
}
return c.selfSHA()
}
// fileSHA256 is the hex sha256 of a file's bytes, or "" when it cannot be read.
func fileSHA256(path string) string {
f, err := os.Open(path)
if err != nil {
return ""
}
@@ -490,6 +644,7 @@ func (c *Collector) collectGuests(ctx context.Context) []Guest {
Cores: cfg.Cores,
MemoryBytes: g.MaxMem,
DiskBytes: g.MaxDisk,
SwapBytes: cfg.Swap * (1 << 20),
}
}
guests = append(guests, entry)
@@ -518,14 +673,50 @@ func (c *Collector) collectStorage(ctx context.Context) []StorageTarget {
// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state
// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil)
// list so the collection always marshals as [].
//
// The saved successes (SetKnownBackupReporter) are added for each tier and guest the in-memory list has
// no success for at or after the saved time. Why: the in-memory list is empty after an agent restart,
// and the hub's backup-freshness check reads only what the reports carried. Measured 2026-10-09 on
// demo-felhom: the night backup landed 02:40 UTC, the kernel step restarted the host at 02:44, no report
// fell in between, so two kernel nights in a row left no trace and the hub alarmed „newest backup is 48h
// old" at 03:00. Only SUCCESSES are saved, so a failure is never hidden and never invented.
// Pinned by TestCollectBackups_SavedSuccessSurvivesARestart.
func (c *Collector) collectBackups(ctx context.Context) []Backup {
if c.backups == nil {
return []Backup{}
out := []Backup{}
if c.backups != nil {
if b := c.backups.Backups(ctx); b != nil {
out = append(out, b...)
}
}
if b := c.backups.Backups(ctx); b != nil {
return b
if c.knownBackups == nil {
return out
}
return []Backup{}
for _, k := range c.knownBackups.KnownBackupSuccesses() {
kt, err := time.Parse(time.RFC3339, k.StartedAt)
if err != nil {
continue
}
covered := false
for _, b := range out {
if !b.Success || b.TargetID != k.TargetID || b.VMID != k.VMID {
continue
}
if bt, err := time.Parse(time.RFC3339, b.StartedAt); err == nil && !bt.Before(kt) {
covered = true
break
}
}
if !covered {
out = append(out, k)
}
}
return out
}
// SetKnownBackupReporter wires the R-894 saved successes into the host report (nil-safe → in-memory only).
func (c *Collector) SetKnownBackupReporter(r KnownBackupReporter) *Collector {
c.knownBackups = r
return c
}
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
+64
View File
@@ -0,0 +1,64 @@
package hub
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// R-349: the report carries the sha256 of the binary that is RUNNING, so the hub can tell a
// hand-built proof binary from the vouched artifact of the same version string. The consequence
// asserted: the wire field equals the hash of this very test binary's bytes (read independently via
// os.Executable, a different channel from /proc/self/exe), and it is on the wire as agent_sha256.
func TestCollect_AgentSHA256IsTheRunningBinary(t *testing.T) {
exe, err := os.Executable()
if err != nil {
t.Skipf("os.Executable: %v", err)
}
raw, err := os.ReadFile(exe)
if err != nil {
t.Fatalf("read own binary: %v", err)
}
sum := sha256.Sum256(raw)
want := hex.EncodeToString(sum[:])
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "running"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.AgentSHA256 != want {
t.Fatalf("agent_sha256 = %q, want the running binary's %q", r.AgentSHA256, want)
}
b, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(b), `"agent_sha256":"`+want+`"`) {
t.Fatalf("agent_sha256 not on the wire: %s", b)
}
}
// An unreadable binary is UNKNOWN (empty, omitted) — never a made-up hash, never a failed report.
func TestFileSHA256_UnreadableIsEmpty(t *testing.T) {
if got := fileSHA256(filepath.Join(t.TempDir(), "absent")); got != "" {
t.Fatalf("absent file hashed to %q, want empty", got)
}
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "running"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
c.selfSHA = func() string { return "" }
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect must not fail on an unreadable binary: %v", err)
}
b, _ := json.Marshal(r)
if strings.Contains(string(b), "agent_sha256") {
t.Fatalf("empty agent_sha256 must be omitted: %s", b)
}
}
+54
View File
@@ -0,0 +1,54 @@
package hub
import (
"context"
"encoding/json"
"testing"
)
// R-444: the guest_disk_trim stanza must reach a report built through the PRODUCTION collect path, be absent from
// the wire when the job is not wired, and carry the keys the hub's System page reads.
type fakeDiskTrim struct{ st *GuestDiskTrimStatus }
func (f fakeDiskTrim) GuestDiskTrimStatus(context.Context) *GuestDiskTrimStatus { return f.st }
func TestCollect_GuestDiskTrim(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.150.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
b, _ := json.Marshal(r)
var m map[string]any
_ = json.Unmarshal(b, &m)
if _, ok := m["guest_disk_trim"]; ok {
t.Fatalf("guest_disk_trim on the wire with no reporter wired: %s", b)
}
c.SetGuestDiskTrimReporter(fakeDiskTrim{st: &GuestDiskTrimStatus{Schedule: "weekly", Guests: []GuestDiskTrim{{
VMID: 9201, LastAttemptAt: "2026-10-07T08:30:00Z", OK: true, BytesTrimmed: 90143313920, Mounts: 2,
DurationSeconds: 24.4, LastOKAt: "2026-10-07T08:30:00Z",
}}}})
r, err = c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
b, _ = json.Marshal(r)
m = nil
_ = json.Unmarshal(b, &m)
dt, ok := m["guest_disk_trim"].(map[string]any)
if !ok || dt["schedule"] != "weekly" {
t.Fatalf("guest_disk_trim missing or wrong on the wire: %s", b)
}
g := dt["guests"].([]any)[0].(map[string]any)
for _, k := range []string{"vmid", "last_attempt_at", "ok", "bytes_trimmed", "mounts", "duration_seconds", "last_ok_at"} {
if _, ok := g[k]; !ok {
t.Fatalf("guest_disk_trim.guests[0] lacks %q: %v", k, g)
}
}
if g["bytes_trimmed"] != float64(90143313920) || g["ok"] != true {
t.Fatalf("values did not survive the round trip: %v", g)
}
}
+62
View File
@@ -0,0 +1,62 @@
package hub
import (
"context"
"testing"
)
type fakeMemBackups []Backup
func (f fakeMemBackups) Backups(context.Context) []Backup { return f }
type fakeKnownBackups []Backup
func (f fakeKnownBackups) KnownBackupSuccesses() []Backup { return f }
// The consequence, not the mechanism: after a restart (empty in-memory list) the host report still carries
// the night's backup, so the hub's freshness check sees it. Measured 2026-10-09 on demo-felhom: without this
// the report carried backups: [] and the hub alarmed „newest backup is 48h old" an hour after a good backup.
// RED-PROOF: return before the saved-success loop in collectBackups → the first case reports nothing → fails.
func TestCollectBackups_SavedSuccessSurvivesARestart(t *testing.T) {
saved := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}
t.Run("restart: memory empty, the saved success is reported", func(t *testing.T) {
c := &Collector{backups: fakeMemBackups(nil), knownBackups: fakeKnownBackups{saved}}
got := c.collectBackups(context.Background())
if len(got) != 1 || got[0].StartedAt != saved.StartedAt || !got[0].Success || got[0].TargetID != "felhom-backup" {
t.Fatalf("want the saved success, got %+v", got)
}
})
t.Run("memory holds the same or a newer success: no second entry", func(t *testing.T) {
mem := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z", Archive: "a"}
c := &Collector{backups: fakeMemBackups{mem}, knownBackups: fakeKnownBackups{saved}}
if got := c.collectBackups(context.Background()); len(got) != 1 || got[0].Archive != "a" {
t.Fatalf("want only the in-memory record, got %+v", got)
}
})
t.Run("a newer FAILURE in memory never hides the saved success, and is kept", func(t *testing.T) {
fail := Backup{TargetID: "felhom-backup", VMID: 9201, Success: false, StartedAt: "2026-10-10T02:40:00Z", Error: "x"}
c := &Collector{backups: fakeMemBackups{fail}, knownBackups: fakeKnownBackups{saved}}
got := c.collectBackups(context.Background())
if len(got) != 2 || got[0].Success || !got[1].Success {
t.Fatalf("want the failure and the saved success, got %+v", got)
}
})
t.Run("another tier's success does not cover this tier", func(t *testing.T) {
other := Backup{TargetID: "felhom-pbs", VMID: 9201, Success: true, StartedAt: "2026-10-10T00:00:00Z"}
c := &Collector{backups: fakeMemBackups{other}, knownBackups: fakeKnownBackups{saved}}
if got := c.collectBackups(context.Background()); len(got) != 2 {
t.Fatalf("want both tiers, got %+v", got)
}
})
t.Run("no saved state wired: the in-memory list as before, never nil", func(t *testing.T) {
c := &Collector{}
if got := c.collectBackups(context.Background()); got == nil || len(got) != 0 {
t.Fatalf("want an empty non-nil list, got %#v", got)
}
})
}
+3
View File
@@ -39,11 +39,14 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
State: StorageStateAttached, Reachable: true, ClassHint: "fast",
ThinPool: &ThinPoolFill{DataUsedFraction: 0.42},
Smart: SmartSummary{Health: SmartUnknown},
// v0.157.0 (`16` §3): the identity keys ride on both rows so the key-set guard sees them.
Kind: "system", PhysicalDisks: []string{"serial:ata:SanDisk_X600:182195804614"}, Identity: "proven",
},
{
Name: "usb-backup", Type: StorageTypeUSB, DurableID: "uuid:x",
State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000,
Smart: SmartSummary{Health: SmartUnknown},
Kind: "disk", PhysicalDisks: []string{"serial:usb:External:000000000000"}, Identity: "unknown", IdentityReason: "usb_bridge",
},
// A pbs target so the recipe's pbs coord has a storage.cfg row to resolve its namespace
// from (R-106). Without one the fixture produces namespace_state=unknown while the golden
+29
View File
@@ -0,0 +1,29 @@
package hub
import (
"context"
"errors"
"testing"
"time"
)
// R-998: the power-off path waits on SetReportedHook, so the hook must fire only for a report the hub
// ACCEPTED — a failed push must not let the box power off believing the hub knows.
func TestLoop_ReportedHookOnlyAfterAcceptedReport(t *testing.T) {
var cn, rn int32
rep := &HostReport{DeliberateOff: &DeliberateOff{RequestedAt: "2026-10-13T18:00:00Z"}}
loop := NewLoop(&fakeCollector{report: rep, n: &cn},
&fakeReporter{env: &ControlEnvelope{}, errSeq: []error{errors.New("hub 5xx")}, n: &rn},
900*time.Second, quietLogger())
var seen []*HostReport
loop.SetReportedHook(func(r *HostReport) { seen = append(seen, r) })
loop.cycle(context.Background(), 900*time.Second) // the push fails
if len(seen) != 0 {
t.Fatalf("hook fired for a refused report: %d", len(seen))
}
loop.cycle(context.Background(), 900*time.Second) // the push is accepted
if len(seen) != 1 || seen[0].DeliberateOff == nil || seen[0].DeliberateOff.RequestedAt != "2026-10-13T18:00:00Z" {
t.Fatalf("hook after an accepted report = %+v", seen)
}
}
+9 -7
View File
@@ -54,11 +54,13 @@ const (
DRReasonNoPBSStorage = "no_pbs_storage_observed"
)
// PBSRootNamespace is how the recipe spells PBS's root namespace. The PBS API spells it as the EMPTY
// string (and `pct restore --ns root` would name a namespace that does not exist) — "root" is a display
// convention this wire has always used, kept here so the field's meaning did not change under R-106.
// Only a box with no `namespace` line in its pbs storage.cfg stanza ever emits it.
const PBSRootNamespace = "root"
// PBSRootNamespace is how the recipe spells PBS's root namespace: the EMPTY string, PBS's own spelling (R-124,
// agent v0.147.0). It used to be the display word "root", which no PBS namespace is named — an operator pasting it
// into `proxmox-backup-client … --ns root` during a real recovery got a failure. An empty namespace is ambiguous on
// its own, so READ IT WITH namespace_state: resolved + "" = the root namespace (pass no --ns, or --ns ""); unknown +
// "" = the agent could not tell. Only a box with no `namespace` line in its pbs storage.cfg stanza emits it.
// Pinned by TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown and TestR124_RootNamespaceOnTheWireIsPBSSpelling.
const PBSRootNamespace = ""
// DRRecipeHostHalf is the agent-emitted half (guest/drive/storage/PBS scaffolding). Derived entirely
// from facts the report already collects — no new privileged reads.
@@ -123,8 +125,8 @@ type DRPBSCoord struct {
RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token
// Namespace is the PBS namespace the restore targets, resolved from the pbs storage's storage.cfg
// stanza — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree
// with the backup that produced the snapshot. PBSRootNamespace when the box has no namespace
// configured; "" when NamespaceState is unknown.
// with the backup that produced the snapshot. PBSRootNamespace ("", PBS's spelling, R-124) when the box has no
// namespace configured; also "" when NamespaceState is unknown — consult NamespaceState.
//
// R-106: this used to come from the listed snapshot's own `ns`, which PBS does not echo per item once
// the request is already namespace-scoped via `?ns=` (internal/pbs/client.go). The field was
+39 -3
View File
@@ -63,8 +63,8 @@ func TestBuildDRRecipeHostHalf(t *testing.T) {
t.Error("felhom-flash (local-dir user-data drive) missing from drives")
}
// pbs: latest snapshot's coords + the pbs storage id as repo_id.
if h.PBS == nil || h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != "root" || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want repo felhom-pbs/root/9201", h.PBS)
if h.PBS == nil || h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != PBSRootNamespace || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want repo felhom-pbs, the root namespace (\"\", R-124), snapshot 9201", h.PBS)
}
}
@@ -252,7 +252,7 @@ func TestDRRecipe_PBSNamespaceIsThePerCustomerOne(t *testing.T) {
}
// TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown: a box with a pbs storage and NO namespace line is
// genuinely in the root namespace. That is an answer, not a gap — it must read resolved/"root", so the
// genuinely in the root namespace. That is an answer, not a gap — it must read resolved/"" (PBS's spelling, R-124), so the
// honest root case is never confused with "I could not tell".
func TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
@@ -453,3 +453,39 @@ func assertNoSecretKeys(t *testing.T, jsonBytes []byte) {
}
walk("<root>", v)
}
// R-124: on the WIRE the root namespace is PBS's own spelling — an empty string, present (not omitted), beside
// namespace_state "resolved". The display word "root" names no PBS namespace, and `--ns root` fails in a recovery.
// RED-PROOF: set PBSRootNamespace back to "root" → this test fails.
func TestR124_RootNamespaceOnTheWireIsPBSSpelling(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup", PBSNamespace: ""}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
b, err := json.Marshal(h.PBS)
if err != nil {
t.Fatal(err)
}
var m map[string]any
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
ns, present := m["namespace"]
if !present {
t.Fatalf("namespace key missing from %s — an omitted key reads as 'unknown', not 'root'", b)
}
if ns != "" {
t.Fatalf("root namespace on the wire = %q, want \"\" (PBS's spelling; no namespace is named %q)", ns, ns)
}
if m["namespace_state"] != DRStateResolved {
t.Fatalf("namespace_state = %v, want %q beside the empty root namespace", m["namespace_state"], DRStateResolved)
}
// A configured namespace still passes through unchanged.
h2 := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup", PBSNamespace: "demo-felhom"}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
if h2.PBS.Namespace != "demo-felhom" {
t.Fatalf("configured namespace = %q, want demo-felhom", h2.PBS.Namespace)
}
}
+39
View File
@@ -0,0 +1,39 @@
package hub
import (
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the guest swap fix (every guest may swap as much as its RAM,
// localapi/guestswap.go) moves a memory burst onto the HOST's swap. Measured on demo-hp the same day: our installer
// gives an 8 GB / 256 GB machine 7.75 GB of host swap, so a 4096 MB guest's 4096 MB swap fits. A host with no swap
// (a BYO install, or swap turned off) gets nothing from the fix. So the report carries both numbers, the agent says it
// once in its log, and the hub raises host_swap_short. The agent never creates host swap itself: changing a host its
// owner set up is a decision (felhom.eu register).
// HostSwapShort reports whether the host's swap is below the guests' swap allowance. A host whose guests have no
// swap allowance (0) is never short.
func HostSwapShort(hostMB, guestMB int64) bool { return guestMB > 0 && hostMB < guestMB }
// applySwap fills the host block's swap numbers from the node status and the guests' specs, and logs the short
// state once per change.
func (c *Collector) applySwap(h *HostMetrics, ns proxmox.NodeStatus, guests []Guest) {
hostMB := ns.Swap.Total / (1 << 20)
var guestMB int64
for _, g := range guests {
if g.Spec != nil {
guestMB += g.Spec.SwapBytes / (1 << 20)
}
}
h.HostSwapMB, h.GuestSwapMB = &hostMB, &guestMB
if HostSwapShort(hostMB, guestMB) {
if c.swapShortWarned.CompareAndSwap(false, true) {
c.logger.Warn("host-swap: the host's swap is below its guests' swap allowance — a memory burst can freeze the box (R-1192); the hub is told (host_swap_short)",
"host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
return
}
if c.swapShortWarned.CompareAndSwap(true, false) {
c.logger.Info("host-swap: the host's swap covers its guests' swap allowance again", "host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
}
+19
View File
@@ -66,8 +66,21 @@ type Loop struct {
// retry, no duplicate shipping). Loop state is single-goroutine (cycle only).
logTailSource func(maxBytes int) []string
logTailPending bool
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
// path waits on it so the hub has the deliberate-off record before the box goes down.
reported func(r *HostReport)
// collectErr (R-1380) sees every failed collect.
collectErr func(ctx context.Context, err error)
}
// SetCollectErrorHook wires an observer of a failed collect (R-1380: a call to Proxmox that fails on TRUST starts the
// move to the CA copy, or a reload of a refreshed copy). Optional; it must not block for long.
func (l *Loop) SetCollectErrorHook(f func(ctx context.Context, err error)) { l.collectErr = f }
// SetReportedHook wires the R-998 after-report observer. Optional.
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
// logTailMaxBytes caps the heartbeat log tail (newest lines kept).
const logTailMaxBytes = 128 * 1024
@@ -137,6 +150,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
report, err := l.collector.Collect(ctx)
if err != nil {
l.logger.Warn("hub: collect failed; skipping this cycle's report", "err", err)
if l.collectErr != nil {
l.collectErr(ctx, err)
}
return current
}
// Fulfill a pending log-pull: attach the ring tail to THIS report and clear the
@@ -156,6 +172,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
l.logger.Warn("hub: report failed; keeping current interval", "err", err)
return current
}
if l.reported != nil {
l.reported(report)
}
if report.LogTail != nil {
// Transparency: the pull is visible in the box's own log (and thus in the ring).
l.logger.Info("operator log pull served", "component", "agent", "lines", len(report.LogTail.Lines))
+4 -4
View File
@@ -77,10 +77,10 @@ func TestLoop_FailedTailPushIsReArmedByNextEnvelope(t *testing.T) {
env *ControlEnvelope
err error
}{
{env: &ControlEnvelope{LogTailRequested: true}}, // arm
{err: errors.New("hub 5xx")}, // the carrying push FAILS
{env: &ControlEnvelope{LogTailRequested: true}}, // hub still pending → re-arm
{env: &ControlEnvelope{}}, // fulfilled
{env: &ControlEnvelope{LogTailRequested: true}}, // arm
{err: errors.New("hub 5xx")}, // the carrying push FAILS
{env: &ControlEnvelope{LogTailRequested: true}}, // hub still pending → re-arm
{env: &ControlEnvelope{}}, // fulfilled
}}
l := tailLoop(rep)
ctx := context.Background()
+42
View File
@@ -0,0 +1,42 @@
package hub
import (
"bytes"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the host report carries the host's swap and the guests' swap allowance, and the
// agent says ONCE in its log when the host's swap is below it (and once when it is covered again).
// RED-PROOF: with applySwap not called the numbers are nil; with the CompareAndSwap removed the WARN repeats.
func TestR1192_HostReportCarriesSwapAndWarnsOnce(t *testing.T) {
var buf bytes.Buffer
c := &Collector{logger: slog.New(slog.NewTextHandler(&buf, nil))}
guests := []Guest{{VMID: 9201, Spec: &GuestSpec{SwapBytes: 4096 << 20}}, {VMID: 9202, Spec: nil}}
var ns proxmox.NodeStatus // a host with no swap
for i := 0; i < 3; i++ {
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if h.HostSwapMB == nil || h.GuestSwapMB == nil || *h.HostSwapMB != 0 || *h.GuestSwapMB != 4096 {
t.Fatalf("report numbers: host %v guest %v", h.HostSwapMB, h.GuestSwapMB)
}
}
if n := strings.Count(buf.String(), "host-swap: the host's swap is below"); n != 1 {
t.Fatalf("the short WARN must be logged once, got %d:\n%s", n, buf.String())
}
ns.Swap.Total = 7935 << 20 // the installer's 8 GB host
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if *h.HostSwapMB != 7935 || HostSwapShort(*h.HostSwapMB, *h.GuestSwapMB) {
t.Fatalf("a covered host must not be short: %v / %v", *h.HostSwapMB, *h.GuestSwapMB)
}
if !strings.Contains(buf.String(), "covers its guests' swap allowance again") {
t.Fatalf("the recovery must be said once:\n%s", buf.String())
}
if HostSwapShort(0, 0) {
t.Fatal("a host whose guests have no swap allowance is never short")
}
}
+181 -4
View File
@@ -15,9 +15,24 @@ import (
// server-side received_at is the hub's dead-man's-switch liveness signal. There is
// no separate heartbeat endpoint.
type HostReport struct {
HostID string `json:"host_id"` // echoes config.Hub.HostID
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
HostID string `json:"host_id"` // echoes config.Hub.HostID
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
// SentAt (R-1384, D79) is this box's clock at the moment the report is SENT — stamped by Client.Report, never by
// the collector. The hub compares it with its own clock to tell a wrong box clock. ReportedAt cannot serve: it is
// stamped when the collect STARTS, and a collect that waits on Proxmox makes it old on a box whose clock is right.
// Pinned by TestReport_StampsSentAtAtSending.
SentAt string `json:"sent_at,omitempty"`
AgentVersion string `json:"agent_version"`
// AgentSHA256 is the sha256 of the binary this process is RUNNING (read through /proc/self/exe,
// once per process), R-349. The version string cannot tell a hand-built proof binary from the
// published, vouched artifact of the same version — same source, different bytes (`-trimpath
// -buildvcs=false` in release-agent.sh) — so self-update sees "already installed" and never
// corrects it. Reporting the bytes lets the hub compare against the vouched agent_sha256, the
// same mechanism host.wrapper_sha256 is for the PBS wrapper (R-50b(a)).
//
// Empty = unreadable, which the hub must treat as UNKNOWN, never as drift. Pinned by
// TestCollect_AgentSHA256IsTheRunningBinary.
AgentSHA256 string `json:"agent_sha256,omitempty"`
Host HostMetrics `json:"host"`
Guests []Guest `json:"guests"`
@@ -43,6 +58,11 @@ type HostReport struct {
// alert. Not a secret (the fp is public; the token is never reported).
LeafFingerprint string `json:"leaf_fingerprint"`
// ProxmoxTrust (R-1380, felhom.eu `03` §12a): how the agent trusts Proxmox's certificate (`mode`: ca | fingerprint |
// …), and the CAUSE when something about it is wrong (`problem`, `detail`): the proof with the CA copy failed, the
// copy ends soon, the root step gave no copy. Omitted when not wired. No secret: a mode, a cause, a date.
ProxmoxTrust any `json:"proxmox_trust,omitempty"`
// Addresses are the host's routable addresses, one entry per (interface, address) — the LAN
// bridge, the WG tunnel, a tailnet. Added v0.119.0 because the hub could not show a managed
// box's IP anywhere: nothing in this report carried one. Non-nil so it marshals as [];
@@ -114,6 +134,22 @@ type HostReport struct {
// on HostReport would have been the only report block named against that convention.
GuestNet *GuestNetStatus `json:"guest_net,omitempty"`
// GuestDiskTrim is the weekly guest disk trim stanza (R-444, `09` §3 decision 139): the schedule and, per owned
// guest, the LAST trim result as persisted by the agent (it survives a restart). Present only when the trim job is
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
// LineTest is the weekly upload line test stanza (R-1247; operator ruling 2026-10-24): the schedule, the last
// measured upload line (best of the runs, Mbit/s, as ep0's iperf3 server counted it) and the last attempt. Present
// only when the job is wired. No secret.
LineTest *LineTestStatus `json:"line_test,omitempty"`
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
// the agent started (the hub keeps its last state); `tiers: []` = evaluated, none found.
ForeignKeyArchives *ForeignKeyArchivesStanza `json:"foreign_key_archives,omitempty"`
// LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent
// mirror of the controller's report log_tails channel. Present ONLY on the heartbeat
// right after the control envelope requested it (log_tail_requested); consume-once on
@@ -123,6 +159,12 @@ type HostReport struct {
// in the steady state, so the cross-repo host-report golden stays byte-stable.
LogTail *LogTail `json:"log_tail,omitempty"`
// DeliberateOff (R-998, doc 14): the household asked this box to power off, in THIS boot. Present on
// every report from the request until the box goes down; absent again after the next boot — that
// absence is how the hub learns the box is back. The hub holds its silence alarms while it is set
// (14 days). `omitempty`: absent in the steady state, so the cross-repo host-report golden stays stable.
DeliberateOff *DeliberateOff `json:"deliberate_off,omitempty"`
// OOB is the operator-access health stanza (TASK H1). It answers the operator's question — "can I
// get into this box right now, and if not, why" — from the hub: felhom-sshd up + on which port,
// locally reachable, the tunnel handshake age (the OOB path rides wg-felhom), whether the operator
@@ -140,6 +182,32 @@ type HostReport struct {
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
// ProfileRefresh (R-1383, release 3): the box's last security-profile refresh and how the agent's last ask ended
// (osupdate.ProfileRefreshStatus, field-exact; hub sysfacts.ProfileRefresh). `any` + omitempty: absent when there
// is nothing to say or the source is not wired.
ProfileRefresh any `json:"profile_refresh,omitempty"`
// GuestPowerGaveUp (R-1383, D78): the guests the guest-power watchdog stopped trying to start (localapi
// GuestPowerGiveUp, field-exact: vmid, since, attempts). The hub raises one operator alarm per guest and clears it
// when the guest runs. `any` + omitempty: absent when there is none or the source is not wired.
GuestPowerGaveUp any `json:"guest_power_gave_up,omitempty"`
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
// when there is nothing to answer, so the golden stays byte-stable.
OperatorActionResults []OperatorActionResult `json:"operator_action_results,omitempty"`
}
// OperatorAction is ONE entry of the reply's operator_actions (hub store.OperatorActionDirective, field-exact).
type OperatorAction struct {
ID int64 `json:"id"`
Action string `json:"action"`
Arg string `json:"arg,omitempty"`
}
// OperatorActionResult is one answer (hub api.hostOperatorActionResult, field-exact): outcome done | refused | failed.
type OperatorActionResult struct {
ID int64 `json:"id"`
Outcome string `json:"outcome"`
Message string `json:"message"`
}
// ControllerSupervisorStatus is the R-523 stanza. Carries no secret.
@@ -163,6 +231,17 @@ type ControllerSupervisorGuest struct {
Restarts24h int `json:"restarts_24h"`
SlowCrashloop bool `json:"slow_crashloop"`
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
// R-1344 (2026-10-29) — the controller RUNS but Docker's health check has read „unhealthy" on several consecutive
// sweeps. ALARM ONLY: the agent does not restart it for this (the measurement must first show a busy night never
// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
Unresponsive bool `json:"unresponsive"`
UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
// R-1344 restart (decided 2026-10-30 from the night's measurement): UnresponsiveRestartAfterS is how long after the
// alarm the agent restarts a controller that still does not answer (0 / absent = an older agent: alarm only — the
// hub's message says which). UnresponsiveRestartAt is the last such restart; the hub keys on it MOVING
// (controller_unresponsive_restart) and it equals LastRestartAt for that act.
UnresponsiveRestartAfterS int `json:"unresponsive_restart_after_s,omitempty"`
UnresponsiveRestartAt string `json:"unresponsive_restart_at,omitempty"` // RFC3339
}
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
@@ -198,6 +277,40 @@ type GuestNetGuest struct {
Message string `json:"message,omitempty"`
}
// GuestDiskTrimStatus is the R-444 weekly trim stanza. `schedule` is a plain description of when the job runs (local
// time of the host); `guests` holds one entry per owned guest that has had at least one trim attempt.
type GuestDiskTrimStatus struct {
Schedule string `json:"schedule"`
Guests []GuestDiskTrim `json:"guests,omitempty"`
}
// LineTestStatus is the R-1247 weekly upload line test stanza. `last_ok_at` + `mbps` are the last SUCCESSFUL
// measurement ("" / 0 = never); `last_attempt_at` + `last_error` the last attempt, whatever its verdict — never read
// the attempt time as a result.
type LineTestStatus struct {
Schedule string `json:"schedule"`
LastOKAt string `json:"last_ok_at,omitempty"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Server string `json:"server,omitempty"`
LastAttemptAt string `json:"last_attempt_at,omitempty"`
LastError string `json:"last_error,omitempty"`
}
// GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never
// read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is
// the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points.
type GuestDiskTrim struct {
VMID int `json:"vmid"`
LastAttemptAt string `json:"last_attempt_at"`
OK bool `json:"ok"`
BytesTrimmed int64 `json:"bytes_trimmed"`
Mounts int `json:"mounts"`
DurationSeconds float64 `json:"duration_seconds"`
LastOKAt string `json:"last_ok_at,omitempty"`
Error string `json:"error,omitempty"`
}
type PBSDRStatus struct {
State string `json:"state"`
StorageID string `json:"storage_id,omitempty"`
@@ -244,7 +357,11 @@ type WireguardStatus struct {
Registered bool `json:"registered"` // the registration marker exists
Active bool `json:"active"` // wg-quick@wg-felhom is-active
LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"`
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
// HandshakeFutureS (R-1204): seconds the latest handshake lies in the FUTURE by this box's clock — set (> 0) only
// then. A box that ran once with its clock ahead keeps a handshake stamped in the future; WireGuard on ep0 then
// refuses every newer handshake, and LastHandshakeAgeS (clamped at 0) reads „just now" for ever. The hub alarms on it.
HandshakeFutureS *int64 `json:"handshake_future_s,omitempty"`
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
}
// SystemInfo is the `system` stanza (see HostReport.System).
@@ -290,6 +407,11 @@ type HostMetrics struct {
//
// Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift.
WrapperSHA256 string `json:"wrapper_sha256,omitempty"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up, night 2026-10-21): the host's swap (Proxmox node status — the
// kernel's SwapTotal) and the sum of its guests' swap allowance. The guest swap fix (swap = RAM) needs swap ON THE
// HOST behind it; the hub raises host_swap_short when the first is below the second. nil = not read (older agent).
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
}
// Guest is one LXC. The agent reports vmid; the hub derives the guest PK
@@ -307,11 +429,12 @@ type GuestSpec struct {
Cores int `json:"cores"`
MemoryBytes int64 `json:"memory_bytes"`
DiskBytes int64 `json:"disk_bytes"`
SwapBytes int64 `json:"swap_bytes,omitempty"` // R-1192: the guest's swap allowance (PVE config `swap`, MB → bytes)
}
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
type Cloudflared struct {
Status string `json:"status"` // running | not_running | unknown (TunnelRunning …)
Status string `json:"status"` // running | not_running | unknown | absent (TunnelRunning …; absent since R-1070)
Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected"
}
@@ -383,6 +506,15 @@ type StorageTarget struct {
// metadata fill is null until Phase B's privileged `lvs` read.
ThinPool *ThinPoolFill `json:"thin_pool,omitempty"`
// Physical-disk identity (felhom.eu architecture/16-storage-physical-identity.md §3, agent v0.157.0):
// the place's kind, the keys of the whole physical disks under it, and whether the agent can prove
// them (proven | unknown + identity_reason vm|network|usb_bridge|weak_wwn|no_serial). Agent-computed
// from /sys + udev only; empty when not computed (an absent device, an older agent). Additive.
Kind string `json:"kind,omitempty"`
PhysicalDisks []string `json:"physical_disks,omitempty"`
Identity string `json:"identity,omitempty"`
IdentityReason string `json:"identity_reason,omitempty"`
// Smart is the disk-health summary, populated in Phase B via smartctl (sudoers). In
// Phase A it is {health:"UNKNOWN", <counters null>} — the keys are committed now so the
// contract is stable across both phases.
@@ -418,6 +550,17 @@ type SmartSummary struct {
ReallocatedSectors *int `json:"reallocated_sectors"`
PendingSectors *int `json:"pending_sectors"`
OfflineUncorrectable *int `json:"offline_uncorrectable"`
// R-330 (disk health Phase 2): three more SATA raw counters. omitempty + pointer: absent (an
// older agent, an NVMe/USB device, or a drive that does not report the attribute) is OMITTED —
// unknown, never a zero (S-39). Wire only: no verdict reads them yet.
// 187 Reported_Uncorrect — the failing drive's most telling counter (normalized 1 vs thresh 0,
// raw 1001) while SMART still said PASSED.
// 188 Command_Timeout — some vendors PACK several counters into the 48-bit raw value, so the
// number is carried as reported and must not be compared across vendors.
// 199 UDMA_CRC_Error_Count — cabling / link errors, not the medium.
ReportedUncorrect *int64 `json:"reported_uncorrect,omitempty"`
CommandTimeout *int64 `json:"command_timeout,omitempty"`
UDMACRCErrors *int64 `json:"udma_crc_errors,omitempty"`
// NVMe attributes.
CriticalWarning *int `json:"critical_warning"`
@@ -535,6 +678,9 @@ type ControlEnvelope struct {
// NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror).
// Absent/false on an old hub → nothing happens.
LogTailRequested bool `json:"log_tail_requested"`
// OperatorActions (2026-10-29): the operator's pending actions for THIS host's agent — a CLOSED list on both sides
// (`04` §6.1). Absent on an older hub.
OperatorActions []OperatorAction `json:"operator_actions,omitempty"`
}
// LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail).
@@ -583,6 +729,17 @@ type WireOSUpdate struct {
// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
// approved for the guest is not approved for the host by that fact alone.
HostRelease *WireOSRelease `json:"host_release,omitempty"`
// Kernel is the kernel lane's instruction for THIS box (R-836, `09` §3 decision 172, `11` §5.11): the kernel the
// household was told about, and whether tonight is a told night. Nil (an older hub, or no kernel due) = no kernel
// step. The hub sets Tonight only after the household's mail the day before went out — no mail, no step.
Kernel *WireKernelStep `json:"kernel,omitempty"`
}
// WireKernelStep is the hub's kernel-lane instruction (hub osupdates.KernelBlock — field-exact, cross-repo).
type WireKernelStep struct {
Kver string `json:"kver"` // e.g. "7.0.14-22-pve" — the wrapper refuses any other (R23)
Tonight bool `json:"tonight"` // the household was mailed the day before: tonight's leg may reboot
NotifiedAt string `json:"notified_at,omitempty"` // when that mail went out (RFC 3339), for the log
}
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
@@ -679,3 +836,23 @@ type WireRestoreDirective struct {
Archive string `json:"archive,omitempty"` // source archive/snapshot to restore from
VMID int `json:"vmid,omitempty"`
}
// ForeignKeyArchivesStanza wraps the per-tier list so "evaluated, none" (`tiers: []`) differs from "not evaluated"
// (the stanza absent) without a null on the wire.
type ForeignKeyArchivesStanza struct {
Tiers []ForeignKeyArchives `json:"tiers"`
}
// ForeignKeyArchives is one tier's count of archives written with another key (R-366 slice 2): the count and the
// newest/oldest archive time (RFC3339, UTC). No key material — the fingerprints stay on the box.
type ForeignKeyArchives struct {
Target string `json:"target"`
Count int `json:"count"`
Oldest string `json:"oldest"`
Newest string `json:"newest"`
}
// DeliberateOff is the household-requested power-off record (R-998).
type DeliberateOff struct {
RequestedAt string `json:"requested_at"` // RFC3339 UTC, the agent's clock
}
+7
View File
@@ -46,6 +46,9 @@
"backing_device": "",
"class_hint": "fast",
"role": "",
"kind": "system",
"physical_disks": ["serial:ata:SanDisk_X600:182195804614"],
"identity": "proven",
"thin_pool": { "data_used_fraction": 0.42, "metadata_used_fraction": null },
"smart": {
"health": "UNKNOWN",
@@ -74,6 +77,10 @@
"backing_device": "/dev/sdb1",
"class_hint": "slow",
"role": "",
"kind": "disk",
"physical_disks": ["serial:usb:External:000000000000"],
"identity": "unknown",
"identity_reason": "usb_bridge",
"smart": {
"health": "UNKNOWN",
"temperature_c": null,
+122
View File
@@ -0,0 +1,122 @@
package lanresolver
import (
"context"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"testing"
)
// recRunner records every privileged command EnsureDnsmasq would run and succeeds — nothing reaches
// apt, systemctl or the root checker.
type recRunner struct {
mu sync.Mutex
calls []string
}
func (r *recRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
r.calls = append(r.calls, strings.Join(append([]string{name}, args...), " "))
return nil, nil, nil
}
func (r *recRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func (r *recRunner) installed() bool {
for _, c := range r.calls {
if strings.HasPrefix(c, "apt-get install") && strings.HasSuffix(c, " dnsmasq") {
return true
}
}
return false
}
// fixtureRoot builds a fake host root holding exactly the given relative files and points the REAL
// probe at it for the test's duration.
func fixtureRoot(t *testing.T, files ...string) {
t.Helper()
root := t.TempDir()
for _, f := range files {
p := filepath.Join(root, f)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, nil, 0o644); err != nil {
t.Fatal(err)
}
}
prev := hostRoot
hostRoot = root
t.Cleanup(func() { hostRoot = prev })
}
func ensure(t *testing.T) *recRunner {
t.Helper()
r := &recRunner{}
m := NewManager(r, "192.0.2.10", []string{"1.1.1.1"}, slog.New(slog.NewTextHandler(io.Discard, nil)))
if err := m.EnsureDnsmasq(context.Background()); err != nil {
t.Fatalf("EnsureDnsmasq: %v", err)
}
return r
}
// R-317: a host with `dnsmasq-base` (the /usr/sbin/dnsmasq binary) but WITHOUT the `dnsmasq` package
// (the service unit) must get the package installed — else the following `systemctl enable --now
// dnsmasq` hits a unit that does not exist and LAN name resolution silently never comes up.
//
// RED-PROOF: probe "usr/sbin/dnsmasq" instead of the unit paths in dnsmasqUnitInstalled → this fails
// with "install was skipped".
func TestEnsureDnsmasq_BinaryWithoutUnitInstalls(t *testing.T) {
fixtureRoot(t, "usr/sbin/dnsmasq")
r := ensure(t)
if !r.installed() {
t.Fatalf("install was skipped on a dnsmasq-base-only host (binary present, unit absent) — "+
"the enable that follows targets a missing unit (R-317). calls: %q", r.calls)
}
}
func TestEnsureDnsmasq_UnitPresentSkipsInstall(t *testing.T) {
for _, unit := range []string{"usr/lib/systemd/system/dnsmasq.service", "lib/systemd/system/dnsmasq.service"} {
t.Run(unit, func(t *testing.T) {
fixtureRoot(t, "usr/sbin/dnsmasq", unit)
if r := ensure(t); r.installed() {
t.Fatalf("apt-get install ran although the dnsmasq unit is present at %s: %q", unit, r.calls)
}
})
}
}
func TestEnsureDnsmasq_NothingPresentInstalls(t *testing.T) {
fixtureRoot(t)
if r := ensure(t); !r.installed() {
t.Fatalf("install skipped on a host with no dnsmasq at all: %q", r.calls)
}
}
// Production wiring for the hostRoot seam: the shipped probe resolves against the real root and asks
// about the unit the `dnsmasq` package owns — never the dnsmasq-base binary.
func TestEnsureDnsmasq_ProductionProbeIsTheUnit(t *testing.T) {
if hostRoot != "/" {
t.Fatalf("hostRoot default = %q, want \"/\" — the production probe would look in the wrong tree", hostRoot)
}
var sawUsrLib bool
for _, p := range dnsmasqUnitPaths {
full := filepath.Join(hostRoot, p)
if strings.HasSuffix(full, "/sbin/dnsmasq") || strings.HasSuffix(full, "/bin/dnsmasq") {
t.Errorf("probe path %s is the dnsmasq-base binary, not the dnsmasq unit (R-317)", full)
}
if full == "/usr/lib/systemd/system/dnsmasq.service" {
sawUsrLib = true
}
}
if !sawUsrLib {
t.Errorf("probe paths %q miss /usr/lib/systemd/system/dnsmasq.service (dpkg -S: owned by dnsmasq)", dnsmasqUnitPaths)
}
}
+26 -2
View File
@@ -101,11 +101,35 @@ func NewManager(runner proxmox.Runner, hostIP string, upstreams []string, logger
}
}
// hostRoot is the filesystem root the install probe resolves against: "/" in production; a test
// points it at a fixture tree so the REAL probe runs against files it controls.
var hostRoot = "/"
// dnsmasqUnitPaths are where the `dnsmasq` package ships its systemd unit (Debian; /lib is the
// pre-usrmerge spelling). R-317: probe the UNIT, never /usr/sbin/dnsmasq — that binary belongs to
// `dnsmasq-base`, so a host carrying dnsmasq-base without dnsmasq used to skip the install and then
// `systemctl enable --now dnsmasq` failed against a unit that is not there (resolver never up).
// Pinned by TestEnsureDnsmasq_BinaryWithoutUnitInstalls.
var dnsmasqUnitPaths = []string{
"usr/lib/systemd/system/dnsmasq.service",
"lib/systemd/system/dnsmasq.service",
}
// dnsmasqUnitInstalled reports whether the dnsmasq service unit (the `dnsmasq` package) is present.
func dnsmasqUnitInstalled() bool {
for _, p := range dnsmasqUnitPaths {
if _, err := os.Stat(filepath.Join(hostRoot, p)); err == nil {
return true
}
}
return false
}
// EnsureDnsmasq makes dnsmasq present + enabled and writes the host base config. Idempotent: it
// installs the package only when absent, and writes the base drop-in only when its content changes.
func (m *Manager) EnsureDnsmasq(ctx context.Context) error {
if _, err := os.Stat("/usr/sbin/dnsmasq"); err != nil { // metadata read, no privilege needed
m.logger.Info("lanresolver: dnsmasq absent — installing")
if !dnsmasqUnitInstalled() { // metadata read, no privilege needed
m.logger.Info("lanresolver: dnsmasq service unit absent — installing")
if out, errOut, ierr := m.runner.Run(ctx, "apt-get", "install", "-y", "-q", "dnsmasq"); ierr != nil {
return fmt.Errorf("install dnsmasq: %s: %w", strings.TrimSpace(string(errOut))+string(out), ierr)
}
+294
View File
@@ -0,0 +1,294 @@
package linetest
import (
"context"
"crypto/rand"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"strconv"
"sync"
"sync/atomic"
"time"
)
// A minimal iperf3 CLIENT for one TCP upload test (client → server), speaking the iperf3 control protocol (3.x) to the
// stock `iperf3 --server` on ep0 (operator ruling 2026-10-24: the agent speaks iperf3 itself, nothing is installed on
// the host). The number that counts is the SERVER's: the bytes it received, from its results JSON — bytes the client
// handed to its socket include what still sat in the send buffer when the timer ended.
//
// The protocol, as the server drives it (one state byte at a time on the control connection):
//
// client → 37-byte cookie
// server → PARAM_EXCHANGE(9) client → params JSON (4-byte big-endian length + bytes)
// server → CREATE_STREAMS(10) client opens N data connections, each starts with the same cookie
// server → TEST_START(1), TEST_RUNNING(2) client sends for the test time, then → TEST_END(4)
// server → EXCHANGE_RESULTS(13) client → its results JSON; server → its results JSON
// server → DISPLAY_RESULTS(14) client → IPERF_DONE(16)
//
// A server already running a test answers ACCESS_DENIED(-1) and closes: that is ErrBusy (another box is testing; try
// again later the same day). SERVER_ERROR(-3) / SERVER_TERMINATE(-2) end the test with an error.
const (
stParamExchange = 9
stCreateStreams = 10
stTestStart = 1
stTestRunning = 2
stTestEnd = 4
stExchangeResults = 13
stDisplayResults = 14
stIperfDone = 16
stAccessDenied = -1
stServerTerminate = -2
stServerError = -3
cookieSize = 37 // 36 characters + NUL
blockSize = 128 * 1024
maxJSONBytes = 1 << 20
)
// ErrBusy: the server is running another client's test.
var ErrBusy = errors.New("iperf3 server busy")
// Result is one upload test.
type Result struct {
Mbps float64 // the server's received bits per second / 1e6
ServerBytes int64 // bytes the server received, all streams
ClientBytes int64 // bytes the client wrote (diagnostics only)
ServerSeconds float64 // the server's measured duration
}
// Dialer opens a TCP connection (net.Dialer in production; a seam for tests).
type Dialer interface {
DialContext(ctx context.Context, network, address string) (net.Conn, error)
}
// Client runs one test. The whole run is bounded by ctx; Run sets every connection's deadline from it.
type Client struct {
Dialer Dialer
Streams int // parallel streams (-P)
Time time.Duration // test time (-t)
}
func (c Client) dialer() Dialer {
if c.Dialer != nil {
return c.Dialer
}
return &net.Dialer{Timeout: 5 * time.Second}
}
func newCookie() ([]byte, error) {
const alphabet = "abcdefghijklmnopqrstuvwxyz234567"
b := make([]byte, cookieSize)
if _, err := rand.Read(b[:cookieSize-1]); err != nil {
return nil, err
}
for i := 0; i < cookieSize-1; i++ {
b[i] = alphabet[int(b[i])%len(alphabet)]
}
b[cookieSize-1] = 0
return b, nil
}
func writeJSON(w io.Writer, v any) error {
data, err := json.Marshal(v)
if err != nil {
return err
}
var hdr [4]byte
binary.BigEndian.PutUint32(hdr[:], uint32(len(data)))
if _, err := w.Write(append(hdr[:], data...)); err != nil {
return err
}
return nil
}
func readJSON(r io.Reader, v any) error {
var hdr [4]byte
if _, err := io.ReadFull(r, hdr[:]); err != nil {
return err
}
n := binary.BigEndian.Uint32(hdr[:])
if n == 0 || n > maxJSONBytes {
return fmt.Errorf("iperf3: results JSON of %d bytes refused", n)
}
buf := make([]byte, n)
if _, err := io.ReadFull(r, buf); err != nil {
return err
}
return json.Unmarshal(buf, v)
}
type streamResult struct {
ID int `json:"id"`
Bytes int64 `json:"bytes"`
Retransmits int64 `json:"retransmits"`
Jitter float64 `json:"jitter"`
Errors int64 `json:"errors"`
OmittedErrors int64 `json:"omitted_errors"`
Packets int64 `json:"packets"`
OmittedPackets int64 `json:"omitted_packets"`
StartTime float64 `json:"start_time"`
EndTime float64 `json:"end_time"`
}
type results struct {
CPUTotal float64 `json:"cpu_util_total"`
CPUUser float64 `json:"cpu_util_user"`
CPUSystem float64 `json:"cpu_util_system"`
SenderHasRetransmits int `json:"sender_has_retransmits"`
CongestionUsed string `json:"congestion_used,omitempty"`
Streams []streamResult `json:"streams"`
}
// streamIDs are the ids iperf3 gives its streams: the first is 1, every later one is (streams so far) + 2 — so 1, 3,
// 4, 5 … (iperf_add_stream). The server matches our results by these ids.
func streamIDs(n int) []int {
ids := make([]int, n)
for i := range ids {
if i == 0 {
ids[i] = 1
} else {
ids[i] = i + 2
}
}
return ids
}
// Run performs one upload test to addr ("10.77.0.1:5201").
func (c Client) Run(ctx context.Context, addr string) (Result, error) {
var res Result
streams, dur := c.Streams, c.Time
if streams < 1 {
streams = 1
}
if dur <= 0 {
dur = 10 * time.Second
}
deadline, ok := ctx.Deadline()
if !ok {
return res, errors.New("iperf3: Run needs a context with a deadline")
}
cookie, err := newCookie()
if err != nil {
return res, err
}
ctrl, err := c.dialer().DialContext(ctx, "tcp", addr)
if err != nil {
return res, fmt.Errorf("iperf3: connect: %w", err)
}
defer ctrl.Close()
ctrl.SetDeadline(deadline)
if _, err := ctrl.Write(cookie); err != nil {
return res, fmt.Errorf("iperf3: cookie: %w", err)
}
var data []net.Conn
defer func() {
for _, d := range data {
d.Close()
}
}()
var sent atomic.Int64
var server results
var wg sync.WaitGroup
for {
var st [1]byte
if _, err := io.ReadFull(ctrl, st[:]); err != nil {
return res, fmt.Errorf("iperf3: control read: %w", err)
}
switch int8(st[0]) {
case stParamExchange:
params := map[string]any{"tcp": true, "omit": 0, "time": int(dur / time.Second), "num": 0, "blockcount": 0,
"parallel": streams, "len": blockSize, "pacing_timer": 1000, "client_version": "3.18"}
if err := writeJSON(ctrl, params); err != nil {
return res, fmt.Errorf("iperf3: params: %w", err)
}
case stCreateStreams:
for i := 0; i < streams; i++ {
d, err := c.dialer().DialContext(ctx, "tcp", addr)
if err != nil {
return res, fmt.Errorf("iperf3: data stream %d: %w", i+1, err)
}
d.SetDeadline(deadline)
data = append(data, d)
if _, err := d.Write(cookie); err != nil {
return res, fmt.Errorf("iperf3: data stream %d cookie: %w", i+1, err)
}
}
case stTestStart:
case stTestRunning:
stopAt := time.Now().Add(dur)
if stopAt.After(deadline) {
return res, errors.New("iperf3: no time left for the test inside the deadline")
}
buf := make([]byte, blockSize)
for _, d := range data {
d.SetWriteDeadline(stopAt)
wg.Add(1)
go func(d net.Conn) {
defer wg.Done()
for {
n, err := d.Write(buf)
sent.Add(int64(n))
if err != nil {
return
}
}
}(d)
}
wg.Wait()
if _, err := ctrl.Write([]byte{stTestEnd}); err != nil {
return res, fmt.Errorf("iperf3: test end: %w", err)
}
case stExchangeResults:
ids := streamIDs(len(data))
mine := results{SenderHasRetransmits: -1, CongestionUsed: "cubic"}
per := sent.Load() / int64(max(len(data), 1))
for _, id := range ids {
mine.Streams = append(mine.Streams, streamResult{ID: id, Bytes: per, Retransmits: -1, EndTime: dur.Seconds()})
}
if err := writeJSON(ctrl, mine); err != nil {
return res, fmt.Errorf("iperf3: send results: %w", err)
}
if err := readJSON(ctrl, &server); err != nil {
return res, fmt.Errorf("iperf3: read server results: %w", err)
}
case stDisplayResults:
ctrl.Write([]byte{stIperfDone})
return summarize(server, sent.Load(), dur)
case stAccessDenied:
return res, ErrBusy
case stServerTerminate, stServerError:
var codes [8]byte
io.ReadFull(ctrl, codes[:])
return res, fmt.Errorf("iperf3: server error (state %d, iperf errno %d, errno %d)", int8(st[0]),
int32(binary.BigEndian.Uint32(codes[:4])), int32(binary.BigEndian.Uint32(codes[4:])))
default:
return res, fmt.Errorf("iperf3: unexpected state %s", strconv.Itoa(int(int8(st[0]))))
}
}
}
// summarize turns the server's results into a Result. No stream bytes is an error, never a zero line.
func summarize(server results, clientBytes int64, dur time.Duration) (Result, error) {
res := Result{ClientBytes: clientBytes}
for _, s := range server.Streams {
res.ServerBytes += s.Bytes
if s.EndTime > res.ServerSeconds {
res.ServerSeconds = s.EndTime
}
}
if res.ServerBytes <= 0 {
return res, errors.New("iperf3: the server reported no received bytes")
}
secs := res.ServerSeconds
if secs <= 0 {
secs = dur.Seconds()
}
res.Mbps = float64(res.ServerBytes) * 8 / secs / 1e6
return res, nil
}
Loaded 100 of 243 files, more files were not shown because too many files have changed in this diff. Show more