Compare commits

...
117 Commits
Author SHA1 Message Date
admin 54d3a1057b CHANGELOG: the unreleased header names R-972 (Go 1.27.3)
gates / gates (push) Successful in 59s
2026-10-26 22:21:47 +01:00
admin fbbb168d1f build(R-972): Go 1.27.3 (GO-2026-6809, crypto/x509) — govulncheck 1 → 0 called
gates / gates (push) Successful in 1m22s
2026-10-26 20:34:25 +01:00
admin 3911cc6f52 fix(R-1232): the tier due-check counts only this box's archives (own key) — a reinstalled box's first whole-guest copy is not a week late
gates / gates (push) Successful in 1m20s
Red-proved (the old box's archive was counted). Commit only.
2026-10-26 20:15:39 +01:00
admin ad80ee749a docs: CONTEXT — R-1285 state and Tuesday's line test
gates / gates (push) Successful in 1m15s
2026-10-26 18:11:24 +01:00
admin 0683cc2dab REPORT: the 2026-10-26 afternoon run. Docs only.
gates / gates (push) Successful in 1m48s
2026-10-26 14:38:15 +01:00
admin b1bc63dd32 R-1285: the new agent runs the bundle policy pass once after its self-update commits — in the normal two-job delivery the old process installs the bundle. Test in the real order (commit, then one bundle-policy plan); red-proved. Commit only, not released.
gates / gates (push) Successful in 2m15s
2026-10-26 12:44:25 +01:00
admin e66a48f2b7 REPORT: the 2026-10-26 attended release run. Docs only.
gates / gates (push) Successful in 1m41s
2026-10-26 10:38:03 +01:00
admin b890b57540 v0.166.0 released and vouched by the operator (10:04:55 CET): CHANGELOG header (R-1280 live, R-1285 back to OPEN — the bundle job runs in the old process); D50 day0_agent pin → 0.166.0. Docs only.
gates / gates (push) Successful in 1m52s
2026-10-26 10:07:15 +01:00
admin aa69114d18 REPORT: the 2026-10-25 night — committed fixes, CI by hash, teardown. Docs only.
gates / gates (push) Successful in 1m43s
2026-10-25 23:54:07 +01:00
admin abcc1e6833 R-1285: after every bundle install the agent asks the wrapper now on disk for the appliance-only policy pass, so a BYO host upgraded from an older wrapper loses our SSH drop-in in one update. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:46:38 +01:00
admin d912307bb7 R-1280: a signed job's validity window is at most 24 h — the verifier, the root wrapper and felhom-opsign. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:42:18 +01:00
admin feaaa0e8ff D50 = A (R-1290): the published gate asserts the newest 5 released versions plus the pinned MinAgent (0.158.1) and day-0 agent (0.165.0); a pin with no tag fails; decoys red-proved. No binary change.
gates / gates (push) Successful in 1m41s
2026-10-25 20:20:30 +01:00
admin 6b6ba0c17f v0.165.0 vouched by the operator (18:38:49 CET); REPORT: release, delivery to four hosts, the BYO proof (removal one bundle late, felhom.eu R-1285). Docs only.
gates / gates (push) Failing after 1m49s
2026-10-25 19:02:48 +01:00
admin 4697f8ac9e v0.165.0: CHANGELOG names the release (R-1257 D43 = A, R-1250); released from a clean clone, sha256 e9d04a47…, bundle 8dea1f34…; not vouched
gates / gates (push) Successful in 1m38s
2026-10-25 17:00:02 +01:00
admin 4a1a84941c D43 = A run close-out: REPORT and CONTEXT. Docs only.
gates / gates (push) Successful in 1m40s
2026-10-24 21:04:19 +02:00
admin c8d3c17b13 D43 = A (R-1257): the root-key-only sshd drop-in only on an appliance; on BYO nothing is written and our copy is removed; the in-force check reads the LAN addresses (X3); a failing sshd -t undoes only the SSH step (X4). Commit only, not released.
gates / gates (push) Successful in 1m32s
2026-10-24 20:47:17 +02:00
admin f5c3d77a61 Review round 2: .claude/rules/unprompted-work.md aligned with its four siblings (the missing 'A version number is pushed once' fact paragraph); no code change
gates / gates (push) Successful in 1m32s
2026-10-24 15:50:59 +02:00
admin 5e05627c4e REPORT: R-1250 test on request, live proof on demo-hp
gates / gates (push) Successful in 1m41s
2026-10-24 13:48:07 +02:00
admin 1e087bb158 linetest: waiting_for_quiet needs the full watched window (R-1250)
gates / gates (push) Successful in 1m35s
The live proof found a fresh agent answering waiting before it had watched one minute; a quiet
box's torrent app would be held for nothing.
2026-10-24 13:24:41 +02:00
admin 29bcf4403e linetest: a test on request for a box that seeds all day (R-1250)
gates / gates (push) Successful in 1m37s
GET /host/line-test reports waiting_for_quiet (due, in the window, tunnel up, only the quiet
missing); POST /host/line-test/run runs one measurement with a 5 s settle and a 10 s quiet
check instead of 10 quiet minutes, runs 5 s apart (~55 s). Every other weekly rule holds.
Commit only; the release rides the next train.
2026-10-24 13:18:59 +02:00
admin 08ca67861f CHANGELOG: v0.164.0 released (the weekly upload line test, R-1247), delivered to the three test boxes, vouched by the operator; REPORT
gates / gates (push) Successful in 1m32s
2026-10-24 09:46:00 +02:00
admin f7e1dcbb27 R-1247: the weekly upload line test — the agent's own iperf3 client to ep0's tunnel-only server, Tuesday 10:00-15:59 after 10 quiet minutes, under the heavy-op gate, best of 3; line_test stanza + GET /host/line-test; 15 tests red-proved; unreleased (next v0.164.0)
gates / gates (push) Successful in 1m35s
2026-10-24 08:59:42 +02:00
admin 2d4b1930df REPORT: day 2026-10-23 — agent 0.163.0 released, delivered (step bundle), vouched by the operator
gates / gates (push) Successful in 1m48s
2026-10-23 11:08:14 +02:00
admin b4721eee11 CHANGELOG: v0.163.0 released (D30, D4 root SSH by key only, R-1204)
gates / gates (push) Successful in 2m5s
2026-10-23 08:44:19 +02:00
admin 1109092def D4 (R-982): root SSH from the LAN by key only — the bundle's sshd drop-in, try-reload-or-restart, an sshd -T read-back reported when not in force, a reload in undo (the reviewed v2 patch); unreleased
gates / gates (push) Successful in 2m9s
2026-10-23 08:22:17 +02:00
admin b45c47be6c REPORT: night 2026-10-22/23 (D30 live, hardened; D4 patches v2)
gates / gates (push) Successful in 2m1s
2026-10-23 00:06:13 +02:00
admin b9b831fb90 D30: a wait stamp dated after now (a clock that jumped back) is dropped, never a block until that date — the commit's security review; red-proved; unreleased
gates / gates (push) Successful in 2m3s
2026-10-22 23:08:25 +02:00
admin 39fb9cbf6c D30 rotation hardened after review: the key file is replaced by an fsynced rename over it (never missing or empty), the first old key stays private.key.prev, later ones dated (3 kept), the once-an-hour bound survives a restart; red-proved; CONTEXT; unreleased
gates / gates (push) Successful in 2m24s
2026-10-22 23:00:05 +02:00
admin ae69ad4512 D30 (R-1204): a handshake stamped in the future for 15 min gives a new WireGuard key, registered on the existing peer path (the hub re-keys in place; its full-list push swaps the key on ep0 — nothing new on ep0); old key kept as private.key.prev; red-proved; unreleased
gates / gates (push) Successful in 2m0s
2026-10-22 20:52:52 +02:00
admin e803ad2b91 night 2026-10-22: REPORT; CHANGELOG corrected — a push of the installer to main publishes nothing (R-110, tag installer-v1.35.0); D4 stays a patch because both halves need a release
gates / gates (push) Successful in 2m24s
2026-10-22 18:13:35 +02:00
admin 7651461be2 D4 (R-982) moved out of main into a ready patch — its installer half would publish the installer (forbidden tonight); R-1204 stays; unreleased
gates / gates (push) Successful in 2m45s
2026-10-22 17:10:46 +02:00
admin a0c88e5183 R-982 (D4): root SSH from the LAN by key only — a bundle drop-in, checked and reloaded by felhom-os-apply; R-1204: the heartbeat reports a future-stamped tunnel handshake; unreleased
gates / gates (push) Successful in 2m23s
2026-10-22 16:58:45 +02:00
admin 2d5840ba11 v0.162.0: released (operator yes 2026-10-22), delivered to the 3 test boxes; CHANGELOG + REPORT
gates / gates (push) Successful in 2m10s
2026-10-22 10:39:06 +02:00
admin b310773dbf REPORT: the night of 2026-10-21 (host swap report, R-1194), unreleased
gates / gates (push) Successful in 2m7s
2026-10-22 00:06:21 +02:00
admin 0da792e670 R-1192 follow-up: report host swap vs the guests' swap allowance (log once, hub alarm, local API); R-1194: no OOB stanza before the first felhom-sshd pass; unreleased
gates / gates (push) Successful in 1m56s
2026-10-21 21:29:07 +02:00
admin 286a7e3ecf R-1192: every guest may swap as much as its RAM (bring-up, RAM resize, once at start) — the measured 8 GB freeze; unreleased
gates / gates (push) Successful in 1m39s
2026-10-21 15:43:18 +02:00
admin 4e1621c650 CHANGELOG + CONTEXT + REPORT: v0.161.0 released (operator yes) and delivered to demo-hp, Tester 1, demo-felhom; not vouched; R-1083 left out
gates / gates (push) Successful in 1m27s
2026-10-20 09:07:30 +02:00
admin 51521230c9 REPORT: the night's agent fixes (R-921, R-1070, R-1010), not released
gates / gates (push) Successful in 1m25s
2026-10-19 21:47:40 +02:00
admin 84ef5818d5 security-scan gate (shared felhom.eu/scripts/security_scan_gate.py) in the fast set and CI, warn-first, with the accepted-hits file and decoys (R-986)
gates / gates (push) Successful in 1m30s
2026-10-19 21:11:46 +02:00
admin 56aabc729c Unreleased: host_busy on GET /backup/status (R-921), a missing tunnel container is 'absent' and not-applicable after an OS update (R-1070), the janitor forces a half-removed vzdump snapshot once (R-1010)
gates / gates (push) Successful in 58s
2026-10-19 19:52:37 +02:00
admin 0cbd500154 gofmt gate in agent_gates.py with its decoys (R-1160); REPORT for the day
gates / gates (push) Successful in 49s
2026-10-19 10:26:27 +02:00
admin 1edfd8e331 gofmt -w: 14 files, formatting only, no behaviour change (R-1160) 2026-10-19 10:24:32 +02:00
admin 0bc0bc5da0 CONTEXT + REPORT: v0.160.0 released, on the three test hosts, not vouched
gates / gates (push) Successful in 48s
2026-10-19 05:41:34 +02:00
admin 45458196d1 Two texts made true after 0.160.0: the ceremony log line names the confirm order; the R-973 comments name whole_guest_backup_refused (no behaviour change)
gates / gates (push) Successful in 52s
2026-10-19 05:25:58 +02:00
admin 124164db73 CHANGELOG v0.160.0 (released: R-978, R-973, R-1093, R-1149, R-1151)
gates / gates (push) Successful in 46s
2026-10-19 03:22:00 +02:00
admin c11e3e2a87 The new escrow package reaches the hub only after the household typed two words of the code back: POST /escrow/ceremony/confirm; an old controller keeps the claim order (R-1151, unreleased)
gates / gates (push) Successful in 48s
2026-10-19 03:03:37 +02:00
admin 1b6e31675b The local whole-guest tier keeps one restore point per day for 3 days: keep-daily instead of keep-last (R-1149, operator D17, unreleased)
gates / gates (push) Successful in 47s
2026-10-19 02:59:34 +02:00
admin 1693da4d1a REPORT: the three unreleased fixes (R-1093, R-978, R-973) with the R-1093 loop-device check on demo-hp
gates / gates (push) Successful in 47s
2026-10-18 21:48:03 +02:00
admin 79ba884859 CI workflow: the failure mail links the public run page, by run id (R-1135)
gates / gates (push) Successful in 49s
The alarm step built its link from GITHUB_SERVER_URL, which on this runner
is the in-cluster Gitea address, and from the run NUMBER, which Gitea 1.26
reads as a run id first (felhom.eu /actions/runs/1116 opened run id 1116 =
run number 613). Now https://gitea.dooplex.hu + GITHUB_RUN_ID, and the link
is printed into the job log as ALARM-LINK. Same change in all four repos.
2026-10-18 20:34:59 +02:00
admin f17e1979bd A cadence cap on the local-API whole-guest backup route (R-973, unreleased)
gates / gates (push) Successful in 49s
Each successful run prunes its tier, so a compromised controller calling POST /backup in a loop
pushed out every pre-compromise restore point. A tier with a cadence of at most 24 h now allows 3
successful runs per rolling 24 h (night, a press, the next night); a longer tier 1 per
max(cadence - 24 h, cadence/2), 6 days for the weekly off-site tier. Only successes count. A
refusal is a 429, a WARN, saved, and for 24 h a critical host-report status that the hub turns
into its existing operator-only agent_capability_degraded event. The success history and the
refusal are kept in the R-894 state file, so a restart does not reset the cap. No sudoers change.
Tests TestBackupCap_* and the state, status and wiring tests, red-proved.
2026-10-18 19:53:15 +02:00
admin 0963bae3c3 The partition probe fails closed on every path (R-1093 review, unreleased)
PART_ENTRY_* comes from the parent's table, so it proves nothing about a partition's contents
unless blkid also completed: lsblk's PTTYPE is now set aside only on a blkid exit 0 (exit 2, an
I/O error or a missing binary keep "device has a partition table"). An lsblk that exits 0 with
output that does not parse into a device, or lists none, leaves the device unprobed instead of
blank (a hole that predates R-1093). Six more TestInspect_R1093_PartitionVerdicts rows, each
red-proved.
2026-10-18 19:47:15 +02:00
admin 8561dc8a5c The new escrow package reaches the hub only when its code is claimed (R-978, unreleased)
The root ceremony uploaded the new package before the new code was claimed, and the hub
overwrites the current package when the sealed secrets are unchanged: a customer who started the
ceremony again and walked away kept a written-down code that opened nothing. Json mode (the
FELHOM_ESCROW argv, unchanged) now returns the package (CeremonyOutput v2, PendingUpload); the
daemon uploads it at POST /escrow/ceremony/claim before handing out the code. A failed upload is a
502 with the code still claimable; TTL and a new start wait for an in-flight claim upload; no
uploader is a 503. Text mode still uploads at once. No sudoers change.
Tests TestEscrowCeremony_R978_* (localapi) and the cmd-side defer/delivery/wiring tests, red-proved.
2026-10-18 19:42:02 +02:00
admin f7b7308c4d An empty partition is blank: lsblk's PTTYPE on a partition is the parent's (R-1093, unreleased)
lsblk prints the disk's table type on every partition, so InspectDevice read an empty partition as
"device has a partition table" and the format path asked the typed erase confirmation. lsblk's
PTTYPE is now set aside only when blkid -p on the partition itself proved it read the device and
that it is a partition of a table of that exact type (PART_ENTRY_NUMBER + matching
PART_ENTRY_SCHEME). A filesystem or a nested table in the partition, a silent blkid, a whole disk
or a different scheme keep the data-bearing reading. Same allow-listed argv, no sudoers change.
Test TestInspect_R1093_PartitionVerdicts, red-proved against the old predicate.
2026-10-18 19:35:03 +02:00
admin 27bbdaaa22 REPORT/CHANGELOG: agent 0.159.2 delivered and proven on demo-hp
gates / gates (push) Successful in 51s
2026-10-18 14:45:47 +02:00
admin e25454ccc3 CHANGELOG: v0.159.2 released (the after-failure orphan check)
gates / gates (push) Successful in 59s
2026-10-18 12:54:28 +02:00
admin 65c1994d3b CHANGELOG (Unreleased): the live proof the next release owes for the after-failure orphan check
gates / gates (push) Successful in 47s
2026-10-17 23:54:14 +02:00
admin 6545cccff7 The orphan thin check runs only after a failed snapshot backup (R-1012 follow-up, unreleased)
gates / gates (push) Successful in 48s
v0.159.0/v0.159.1 ran the check before every snapshot-mode whole-guest backup, which takes a
metadata snapshot of the live pool. Now explainFailure (vzdump start or task failure) is its only
caller; the finding is still appended to the failed backup's error. Tests: failure = 1 call,
success = 0 calls; red-proved against the v0.159.1 runner.go (2 calls / calls=1).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 22:43:03 +02:00
admin eb38d78945 REPORT/CONTEXT: agent 0.159.1, the catalog allow-list (controller v0.328.0), its catalog gate (2026-10-17)
gates / gates (push) Successful in 58s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 17:29:50 +02:00
admin 4a1c07b288 CHANGELOG: v0.159.0 (superseded) and v0.159.1 — released, not vouched
gates / gates (push) Successful in 48s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 16:42:00 +02:00
admin 5c3b218822 The signed decommission prunes the drive from every guest's bind record (found live on demo-hp, R-1124 proof)
gates / gates (push) Successful in 47s
The self-serve decommission already dropped the drive from the guest-bind record; the operator-signed one did
not, so a forgotten drive left a record the 20-second re-assert warned about forever ("skipping non-enrolled
drive"). DecommissionExecutor.SetGuestBinds; red-proved. Code only; the CHANGELOG follows the release (0.159.1).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 16:41:12 +02:00
admin e15d1d30ea R-1124 + R-1012: forget a decommissioned drive that is gone; name an orphan thin device in a backup failure
gates / gates (push) Successful in 47s
R-1124: ForgetRetiredDrives (start-up + the 20 s tick) disables and removes the host mount unit of a
DECOMMISSIONED drive whose disk is absent and unmounted; the agent then no longer lists it. A present
decommissioned drive keeps its unit (one-click re-enroll). The file goes only through the new
felhom-priv-apply verb `unit-forget`, which re-checks marker, disabled, unmounted and absent as root.

R-1012: the read-only root helper felhom-thin-check <vg> <pool> (bundle) lists the pool's thin ids
(metadata snapshot reserved and released) next to LVM's before and after. BackupRunner.SetThinCheck runs
it before a snapshot-mode backup (log) and again after a failure, appending the finding in plain words to
the error the controller mails to the operator. Two reads, intersected; fails open; never removes.

Red-proofs: felhom.eu audits/agent-allowlist-rotation-2026-10-17/P1/. Code only; the CHANGELOG entry
follows the release.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 15:20:34 +02:00
admin 344c4f9030 CI workflow: a manual proof that the failure mail can be sent (R-1129)
gates / gates (push) Successful in 59s
workflow_dispatch with input force_alarm (default false). A dispatch-only
step prints the RESEND_API_KEY length (never the key) and, when forced,
fails so the alarm step sends its real mail, subject marked FORCED TEST.
Push runs are unchanged. No agent release.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 12:49:37 +02:00
admin c52d591851 The decoy suite is a gate of the full run (R-1116); scripts only, no release
gates / gates (push) Successful in 49s
scripts/test_gate_decoys.py is now agent_gates.py's `gate-decoys` row: the full run (which CI runs on every
push), not --fast (~20 s in the workspace, 30 cases). Declared in its own COVERS; test_agent_gates.py pins it.
Red-proved: a reuse-refs FACT case flipped to expect rc 0 -> CONVICTED: gate-decoys; restored, green.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 11:01:25 +02:00
admin 37b99d73b3 Go 1.27.2 and x/crypto v0.58.0 (R-972) — built and tested, not released
gates / gates (push) Successful in 31s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-16 18:53:33 +02:00
admin dd8b2ace02 release-agent.sh: only a clean 404 proves a version is new (R-1098)
gates / gates (push) Successful in 31s
An unreachable registry (000) or a server error passed the existing-version check as new. Test:
scripts/test_release_push_once.sh, red-proved. No agent release.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-16 17:23:46 +02:00
admin 383522450a REPORT + CONTEXT: v0.158.1 build B, vouched; R-1093 filed
gates / gates (push) Successful in 32s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 10:10:21 +02:00
admin 1a8bed11e7 CHANGELOG: v0.158.1 (security review of 0.158.0; 0.158.0 superseded, do not vouch)
gates / gates (push) Successful in 25s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 04:02:17 +02:00
admin 98f3cdd76a Security review of 912f7c1: the mkfs wrapper trusts no sibling claim; the internal-place helper fails closed
gates / gates (push) Successful in 25s
- felhom-mkfs-guarded: the 0.158.0 exemption for a sibling part mounted at /mnt/<name> + under /mnt/felhom-drives is
  removed — the agent can make that corroborating mount itself. Every mount of every node outside /mnt/felhom-drives
  refuses (kernel mount table); an unlistable disk refuses. CandidatePartition.Formattable is false for every part of a
  disk with a mounted node, so it is never offered for formatting (attach still is).
- felhom-internal-place: a failed lvs listing, a listing without the pool, or a blkid exit other than 0/2 refuses
  (they read as "nothing promised" / "no signature" before).
Every new test was seen failing against the 0.158.0 versions.

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 04:02:00 +02:00
admin 888a98571b CHANGELOG: v0.158.0 — the internal data place, partitions, add_internal_place
gates / gates (push) Successful in 28s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 03:56:29 +02:00
admin 912f7c127e The internal data place, partitions as places, add_internal_place (felhom.eu architecture/16 §5-§7, §9; build B)
gates / gates (push) Successful in 27s
- internal place: the record /var/lib/felhom-agent/internal-place.json -> mounted at /mnt/belso by fs-UUID like an
  enrolled drive (startup + 20 s tick), kind "internal" in GET /disks; never takes a household drive's /mnt/belso;
  eject/decommission/format refused (dm on the system disk).
- configs/felhom-internal-place: the ruled size rule and the only lvcreate/mkfs of a system-disk volume (installer +
  the new signed op add_internal_place, operator key, no parameters).
- partitions listed per free disk (empty / filesystem / in_use); a data-bearing part's wipe id binds disk id + part
  number + fs-UUID.
- felhom-mkfs-guarded: Felhom's own sibling drive no longer blocks formatting the free part; an unlistable disk refuses.
Tests red-proved (script mutations, Go mutations, the old wrapper measured against the new tests).

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 03:55:47 +02:00
admin e569ec6c5f REPORT + CONTEXT: v0.157.0 live proof on the three test boxes; not vouched
gates / gates (push) Successful in 30s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-15 21:54:58 +02:00
admin 52ed52c131 CHANGELOG: v0.157.0 — physical-disk identity + POST /disks/same-disk
gates / gates (push) Successful in 27s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-15 21:09:54 +02:00
admin c5daa9480d Physical-disk identity of every storage place + POST /disks/same-disk (felhom.eu architecture/16 §3/§4)
gates / gates (push) Successful in 31s
The agent names the physical disk(s) under each storage place from /sys, udev and /proc/mounts only
(partition -> disk, dm/LVM thin -> slaves, loop -> backing file), keys them (eui / serial:<bus>:<model>
:<serial> / vendor WWN) and says proven or unknown (vm, network, usb_bridge, weak_wwn, no_serial).
GET /disks and the hub report's storage_targets gain kind, physical_disks, identity,
identity_reason (golden updated; the hub copy changes in the same session). POST /disks/same-disk
maps two guest paths to places the agent itself made and answers {different, proven, reason};
an absent drive is unmapped, never the root disk. Tests on fixture trees for every spike case,
red-proofed. CHANGELOG entry follows the release (release-complete gate).

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-15 21:09:18 +02:00
admin d2053e7d47 Rule file: a release needs no golden; the floor carries it after the three test boxes; golden weekly (09 §3 decision 250, operator 2026-10-15)
gates / gates (push) Successful in 27s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-15 11:43:27 +02:00
admin 7d4e564758 Rule file: bake first — bake → vouch → floor, never a floor with no golden (09 §3 decision 247, operator 2026-10-15)
gates / gates (push) Successful in 25s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-15 08:18:13 +02:00
admin 9b749ee7ad v0.156.1: CHANGELOG release entry (sha, bundle, step bundle, tag); 0.156.0 superseded
gates / gates (push) Successful in 52s
2026-10-14 11:36:39 +02:00
admin 7ae3081a76 R-1008: in flight = the worker process lives (Proxmox's pid+pstart rule), not a wall-clock start vs btime (security review: a clock step could make a live backup look dead)
gates / gates (push) Successful in 46s
2026-10-14 11:36:04 +02:00
admin 7ecd63cd61 v0.156.0: CHANGELOG release entry (sha, bundle, step bundle, tag)
gates / gates (push) Successful in 32s
2026-10-14 11:30:46 +02:00
admin a2f3aeeeff After a power cut: remove the dead half-written backup (R-1006); a backup task from before the boot is not in flight (R-1008)
gates / gates (push) Successful in 35s
2026-10-14 11:29:41 +02:00
admin e7d4ecd607 REPORT: the machine came back by its power button; the hub ended the hold (R-998 live proof complete)
gates / gates (push) Successful in 36s
2026-10-14 00:05:55 +02:00
admin 10ca63aa32 REPORT: power-off (R-998) released and proven on the N100
gates / gates (push) Successful in 34s
2026-10-13 21:41:30 +02:00
admin 68a2130681 v0.155.0: CHANGELOG release entry (sha, bundle, tag)
gates / gates (push) Successful in 31s
2026-10-13 20:18:21 +02:00
admin bdf25f019c Host power-off: a guest started after acceptance cancels it; escrow reply data race fixed
gates / gates (push) Successful in 30s
Security review of c6d9435: the other-guests check ran only at acceptance. waitJobsClear now
looks again before each step and CANCELS at once when another guest or VM runs (gate released).
Red-proofed. Also: go test -race found a data race in the escrow ceremony start reply (reads
job.Phase while the job goroutine writes it) — the reply fields are copied before the goroutine
starts; -race was red before, green 3/3 after.
2026-10-13 19:38:23 +02:00
admin c6d9435316 Host power-off: re-check jobs before each step; cancel if one is still running (R-998)
gates / gates (push) Successful in 29s
Security review of f38b03c: a controller update, drive/NAS job, escrow ceremony or Proxmox
vzdump could start between acceptance and the power-off (the heavy gate does not cover them).
runHostPowerOff now asks jobBlocker again before the guest shutdown and before the host
power-off, waits up to 15 min, and CANCELS (gate released, box stays on) if still busy.
Red-proofed: no wait, and a gate kept after a cancel, each convicted.
2026-10-13 19:36:01 +02:00
admin f38b03cdb8 Host power-off: refuse while another guest or VM runs; hold the heavy gate; fail closed (R-998)
gates / gates (push) Successful in 33s
Security review of 3403c87: (1) any guest's controller could switch off a shared host — demo-hp
runs a scratch guest and the Tester 1 VM. Now refused (other_guests_running) when /proc shows any
other lxc-start -n N or kvm -id N; an unreadable /proc refuses. (2) the checks ran once, so a
backup could start before the power-off: the heavy-operation gate is now taken on acceptance and
kept (released only if the power-off command fails). (3) an unwired gate or task reader refused
nothing; now refuses. Each red-proofed.
2026-10-13 19:33:09 +02:00
admin 3403c87c80 Host power-off for the household (R-998): POST /host/poweroff
gates / gates (push) Successful in 34s
The guest's controller asks; the agent refuses in the first 10 min after boot, a second time in
one boot (persisted, keyed by /proc/stat btime), and while a backup, restore test, OS step,
controller update, drive format, NAS verify, escrow ceremony or vzdump runs (each a named 409
reason). On acceptance: the record first, a host report carrying deliberate_off accepted by the
hub (bounded 60 s), then pct shutdown --timeout 120, then systemctl poweroff.

Sudoers: two exact vectors (pct ^shutdown N --timeout N$, systemctl poweroff) in FELHOM_POWEROFF;
nine injection cases. New TestEveryAliasIsGranted: a defined alias missing from the grant line
passed the coverage check before. Every new test red-proofed.
2026-10-13 19:29:52 +02:00
admin 86adbcaf8a rules: "The system poster stays true" (section 6, all five copies identical)
gates / gates (push) Successful in 57s
The shared rule file carries the same wording in felhom.eu, felhom-controller,
felhom-agent, app-catalog-felhom.eu and the workspace root on DooPlex --
"change all five or none" -- so this is this repo's copy of a rule added in
felhom.eu a08bd3cbd5.

A session that changes a fact listed in
architecture/felhom-system-poster.facts.md (a machine, a role, a traffic path,
a backup tier, a time, a retention, a key, a known gap) updates that file in
the same commit; fixes the poster's text if the change is text only; or adds
"System poster needs a refresh: <what changed>" to STATUS. The report names
which of the three it did.

The poster is drawn in Claude Design and nothing in the repo renders it, so
scripts/poster_facts_gate.py only WARNS when the facts outrun the drawing --
it never fails a push, because a refresh needs the operator and another tool.

Verified identical to the other four copies by diff, before and after.
2026-10-09 18:10:19 +02:00
adminandClaude Opus 5.5 24ea960229 REPORT: v0.154.0 released and delivered (2026-10-09)
gates / gates (push) Successful in 1m7s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 08:36:51 +02:00
adminandClaude Opus 5.5 cff6fb5a45 v0.154.0: CHANGELOG release entry (sha, bundle, tag)
gates / gates (push) Successful in 1m11s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 06:58:00 +02:00
adminandClaude Opus 5.5 db25b469ba Host report keeps the last backup across a restart; Secure Boot meta-package leaves the Proxmox lane
gates / gates (push) Successful in 1m7s
Found in the 2026-10-09 kernel-night read-back:
- demo-felhom: the kernel step restarted the host 4 minutes after the night
  backup; the in-memory backup list was empty after the restart, so the hub
  alarmed "host tier: newest backup is 48h old". The report now adds R-894's
  saved newest success per tier (one shared instance).
- demo-hp: proxmox-secure-boot-support pulled shim-signed-common into the
  ring-0 Proxmox plan; the step was refused R6 and the kernel step skipped.
  It joins HOST_SLOW_RE with shim and GRUB.

Red-proved: TestCollectBackups_SavedSuccessSurvivesARestart,
TestR894_LastKnownBackupsIsWiredIntoTheDaemon,
test_ring0_pending_pve_leaves_the_secure_boot_meta_out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 06:47:58 +02:00
adminandClaude Opus 5.5 b228b44f0e REPORT/CONTEXT: 2026-10-08 day (R-899, R-304 on main, unreleased)
gates / gates (push) Successful in 52s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 08:08:53 +02:00
adminandClaude Opus 5.5 91b9405c81 R-304: no 'wrong code' when earlier sealed packages were not all checked (424 older_unchecked)
gates / gates (push) Successful in 55s
Unreleased; ships with tomorrow's release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 07:53:01 +02:00
adminandClaude Opus 5.5 4c69c25b48 R-899: no OS leg after a household press (trigger=manual); after-boot kernel reports carry the saved ring
gates / gates (push) Successful in 50s
Unreleased; ships with tomorrow's release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 07:34:19 +02:00
adminandClaude Opus 5.5 c9013bb47d CHANGELOG/REPORT: v0.153.0 released and delivered
gates / gates (push) Successful in 1m33s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 19:06:04 +02:00
adminandClaude Opus 5.5 2d1e5d0774 R-898: ring 0 stages exactly the told kernel (select listed, KernelSet(kver))
gates / gates (push) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 18:45:28 +02:00
adminandClaude Opus 5.5 f09c53efc1 REPORT/CONTEXT: v0.152.0 the kernel lane
gates / gates (push) Successful in 1m9s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 18:20:11 +02:00
adminandClaude Opus 5.5 92d647a6f6 CHANGELOG: v0.152.0 released (shas, step bundle); host_stale is 45 min (comment fix)
gates / gates (push) Successful in 1m0s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 15:42:26 +02:00
adminandClaude Opus 5.5 d03ab7f1f5 R-836: the kernel lane — one-shot boot through the ESP flag, boot good / one self-revert, night step on a told night
gates / gates (push) Successful in 44s
Wrapper layer kernel (stage / reboot / boot / good / revert / cancel / status;
R20-R23), the two GRUB generators in the bundle (option C on the one-shot
entry), the agent's night step and after-boot judge (host health rule + hub
reached, 20 min measured), the signed os_kernel_step (stage only).
Red-proofs: felhom.eu audits/kernel-lane-2026-10-07/A/redproof.txt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 15:18:24 +02:00
adminandClaude Opus 5.5 b17d1c597d REPORT/CHANGELOG: 2026-10-07 day
gates / gates (push) Successful in 59s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 14:21:01 +02:00
adminandClaude Opus 5.5 0ae01dfb52 CHANGELOG: v0.151.0 released (R-861, R-812 A, R-366, R-105)
gates / gates (push) Successful in 58s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 13:14:47 +02:00
adminandClaude Opus 5.5 dd7cdc09e7 R-366 slice 2 (decision 168): the host report carries the archives the restore-test skipped as another key's
gates / gates (push) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:38 +02:00
adminandClaude Opus 5.5 7b0a8b234b R-105 (decision 169): remove the escrow-create -directive flag and the upload's directive field
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:38 +02:00
adminandClaude Opus 5.5 ce1a4b4758 R-812 option A: the Proxmox package lane (layer pve) + the /etc/pve write gate
The wrapper gains layer "pve" (slow lane): the host's Proxmox userspace
packages only — origin "Proxmox Debian Repository", never a kernel / boot /
firmware / microcode name (R14), no removal, no undo, a new package only from
an allow-list; authority = a signed os_pve_step or the root-owned ring-0 mark.
The night leg runs it in ring 0 after a healthy host step; ring 1 only by a
signed job (PVEStepExecutor). While it runs, the agent's own /etc/pve writes
(every non-GET API call, pct config verbs, pvesm, pveum, felhom-pbs-apply)
wait on internal/pvegate. Health = the host rule + unchanged container ids +
pveversion reads the installed pve-manager.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:38 +02:00
adminandClaude Opus 5.5 ac90169a5d R-861 (a) A1 + (b) B2: the image ref goes to a root verb that checks it; the agent's in-guest tee grant is gone; felhom-op's pct lines are exact (09 §3 decision 165)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:28 +02:00
adminandClaude Opus 5.5 154d6dcaa9 Shared rule file: no hub image build or deploy in a session the operator does not attend (09 §3 decision 162)
gates / gates (push) Successful in 56s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 09:55:02 +02:00
adminandClaude Opus 5.5 2f7072050f REPORT: released and delivered (2026-10-07)
gates / gates (push) Successful in 1m4s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 09:31:58 +02:00
adminandClaude Opus 5.5 85e799f360 CHANGELOG: v0.150.0 released (R-528, R-894, R-330)
gates / gates (push) Successful in 46s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 08:59:08 +02:00
adminandClaude Opus 5.5 3a72a4811b Shared rule file: rule 11 — every helper prompt carries the brief's fences in full (09 §3 decision 160)
gates / gates (push) Successful in 54s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 08:52:36 +02:00
adminandClaude Opus 5.5 a165d53c88 REPORT: the second burn-down night
gates / gates (push) Successful in 49s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 00:14:50 +02:00
adminandClaude Opus 5.5 adaf86ad57 R-330: the agent sends SMART 187/188/199 (raw; omitted when unknown)
gates / gates (push) Successful in 53s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 22:13:04 +02:00
adminandClaude Opus 5.5 74b5eae5b0 R-894: after a restart the agent remembers the last backup per tier
gates / gates (push) Successful in 35s
An unreadable storage right after an agent restart read the off-site tier
DUE (the in-memory record was empty). The newest success per tier is now
kept on disk and read ONLY when the storage cannot be read: fresh -> not
due, older than the cadence -> due, none -> due (unknown) as before. A
storage that answers stays the ground truth.

Ships with v0.150.0 after the 2026-10-07 read-back; nothing delivered.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 20:33:27 +02:00
adminandClaude Opus 5.5 7e82f325b8 CHANGELOG: the memory-kill check (unreleased, to ship as v0.150.0 after the night read-back)
gates / gates (push) Successful in 50s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 19:36:09 +02:00
adminandClaude Opus 5.5 acccb66bd3 R-528 (09 decision 157): after a Docker engine step the wrapper proves the engine reports a memory kill
felhom-os-apply: a docker-layer apply runs oom_check() after health_after and reports
"oom_check": {result pass|fail|error, oom_killed, oom_event, exit_code, image, detail}.
One throwaway container (the controller's image, --pull never, --network none, 64m cap,
label felhom.oomcheck=1) runs dd bs=200M; pass only with OOMKilled=true AND the oom event
(read after a 2 s settle, --until = guest epoch + 1: measured on demo-hp, an --until taken
right after the run missed the event). docker rm -f always runs in a finally; every call
is bounded (<= 90 s). It never changes the step's outcome or health. New wrapper-only mode
"oom-check" (docker layer) runs the check alone; check_guest etc. still apply.

Agent: WrapperReport/Report gain OOMCheck (json:"oom_check"), copied unchanged in runLayer
and in the kept-copy path.

Tests: 9 wrapper tests + 2 Go tests, each red-proofed (audits/readback-2026-10-07/F/red-*.txt).
Also: test_felhom_os_apply.py's `if __name__` sat mid-file, so 11 tests (UnsentReport,
SaveReportOnDisk, AgentDiesMidPass, CrashLeftTheJournal) never ran as a script or from
TestWrapperSuite; moved to the end (they pass).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 19:34:55 +02:00
adminandClaude Opus 5.5 de812bc027 The shared rule file (09 §3 decision 152), identical to the other copies; no code change
gates / gates (push) Successful in 44s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 16:06:46 +02:00
adminandClaude Opus 5.5 3e8ebeb96c Instruction files kept true (09 §3 decision 150): stale gate lists, paths and facts corrected; no code change
gates / gates (push) Successful in 46s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 13:43:57 +02:00
adminandClaude Opus 5.5 cefdc731a4 REPORT: the operator's ten answers (2026-10-06)
gates / gates (push) Successful in 24s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 12:28:58 +02:00
adminandClaude Opus 5.5 e56dcb8a4c CHANGELOG/v0.149.0 released (shas)
gates / gates (push) Successful in 37s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-06 11:50:21 +02:00
173 changed files with 18076 additions and 328 deletions

No files matched your search

+3 -2
View File
@@ -21,6 +21,7 @@ fast, and wrong.
This rule used to be duplicated verbatim in felhom-agent/CLAUDE.md with a note explaining that This rule used to be duplicated verbatim in felhom-agent/CLAUDE.md with a note explaining that
felhom.eu/CLAUDE.md "does not load in an agent-only session". That reasoning was correct before felhom.eu/CLAUDE.md "does not load in an agent-only session". That reasoning was correct before
path-scoped rules existed. The single source is now felhom.eu/CLAUDE.md "Code quality rules"; this path-scoped rules existed. Deliberate scoped copies now live in felhom.eu/.claude/rules/hub.md and
file is the scoped copy that loads exactly where health checks are written. (2026-08-06) felhom-controller/.claude/rules/gates.md (hub.md's comment names them); none is the single source. This
file is the copy that loads exactly where agent health checks are written. (2026-08-06; corrected 2026-10-06)
--> -->
+101
View File
@@ -0,0 +1,101 @@
---
unconditional: true
---
# Unprompted work — rules for any session without a task file
> Goal sessions, nightly sessions, "work the register" sessions. **A session that starts from
> `/goal` or a standing brief inherits these rules exactly as it inherits the gates.** They are the
> part of `PROMPT-TEMPLATE.md` that a task file used to carry and a goal does not. Same wording lives
> in `felhom.eu`, `felhom-controller`, `felhom-agent` and `app-catalog-felhom.eu` `.claude/rules/`, and in the workspace
> root's unversioned `.claude/rules/`; change all five or none.
## 1. What you may pick up on your own
- A register row **you or another CC session filed**, with owner CC, at P3 or a bounded P2, that
needs **no operator decision**, touches **no customer data by design**, and introduces **no
mechanism nobody has measured**. Smallest first.
- A defect you find while exercising the product, filed as a row **before** you fix it — **unless it is small**:
a small finding is fixed in the session and never filed (the size rule, `OPEN-ITEMS.md` „How a row is filed").
- Hygiene: register compression, stale citations, rows with no owner, documents that contradict
live source.
**Not yours, ever, without a task file or an operator word:** money; anything that changes risk to
customer data; anything that changes a promise the product makes to a customer; anything that
reverses a documented design decision (`documentation/architecture/` — a design decision is not a
defect, R-370); anything on DooPlex or ep0; baking or vouching a golden; promoting a
catalog version; a new external dependency; **a hub image build or hub deploy in a session the operator does not
attend** (operator ruling 2026-10-07, `09` §3 decision 162).
## 2. When you may decide instead of ask (operator grant, 2026-09-14)
You may take a decision yourself when **all** of these hold: the architecture folder and the register
give a clear direction; your choice follows that direction; it is reversible without customer-data
risk; and you can write it in the `09-update-architecture.md` §3 shape — one answerable sentence, the
options, what each costs, why this one. **Then record it** as a dated decision in `CONTEXT.md` and
the owning architecture document, tagged *decided by CC unattended — operator may reverse*, and put
it **first** in the morning note. A decision you cannot write in that shape is one you do not take.
## 3. The discipline a task file used to carry
1. **Baselines first.** Read each repo's `main` hash and version from live source before touching it.
2. **Read the architecture document for the area, and name it** in the report, before any claim.
3. **Red-proof every correctness fix.** A test never seen failing has not been shown to test anything.
4. **Live-validate on a Tier-0 box** through the endpoints the UI invokes. `demo-hp` is `ssh hp`.
Throwaway apps only; the standing apps and `bentopdf` stay.
5. **Evidence off the machine at the end of each phase**, before any revert (R-320).
6. **One release per repo per session**, with a CHANGELOG entry (controller: with its `MinAgent`
line) and REPORT overwritten. **A controller release needs no golden** (operator ruling 2026-10-15
11:26, `09` §3 decision 250; decision 7 stands): push the image only when it is ready for EVERY box (the
household's update button offers the highest tag); raise the floor, MinAgent declared, only after it ran
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
on all three — not at night unless the night brief says so. **Agent releases still stop for the
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
**A version number is pushed once** — `felhom-controller/controller/build.sh` now refuses a second
push of a tag that exists, and a registry it cannot reach is a refusal too. This is a FACT about
what the build does, not a new rule: a fix for something already pushed is the next patch version,
and a box on a bad image is moved by the normal update path, never by re-tagging. Earned on
2026-10-15, when 0.315.0 was pushed three times from three different commits and a cache kept
serving the first one (`09-update-architecture.md` §0).
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
Prose is not a record.
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
9. **Never leave a half-state.** If time runs out, revert to clean and say what was reverted.
10. **Teardown, three layers, stated** — machine, host, hub — or "provisioned nothing".
11. **Every helper prompt carries the brief's fences in full** (operator ruling 2026-10-07). A helper session (a
subagent, a fork, a workflow agent) gets the brief's fence list word for word — every protected machine, every
„no", every delivery and Docker limit — not a summary and not „the usual fences". Earned on 2026-10-06 night: two
helpers whose prompts carried only part of the fences ran `docker volume prune` on the bench and a Docker-using
gate on DooPlex.
## 4. The morning note
One screen, plain language, in this order: **decisions you took** (§2) first; what you exercised;
what broke and whether you fixed it; rows opened and closed with the register size before and after;
what needs the operator, each with what happens if they do nothing. No file paths, no function
names, no row numbers as the subject of a sentence.
## 5. Instruction files
**Instruction files (`CLAUDE.md`, `.claude/rules/*`) are kept true by the session that finds them wrong**
(operator ruling 2026-10-06, `09` §3 decision 150). A session MAY, without asking: correct a stale fact (a command, a
count, a version, a path, a description of what a gate does), add a fact it proved, and remove a reference to something
that no longer exists. Each edit is named in the report (file, line, before, after, why). A session MAY NOT, without the
operator's word: loosen a safety rule, a fence, a „never", a protected machine, a secret rule, or a review step; or
remove a rule. When in doubt, it is a rule change, and it goes to the operator. If Claude Code's own permission check
asks before such an edit, wait for the operator's click; if it refuses, record that and file the exact line.
## 6. The system poster stays true
**A session that changes a fact listed in `architecture/felhom-system-poster.facts.md` (a machine, a
role, a traffic path, a backup tier, a time, a retention, a key, a known gap) updates that file in
the same commit.** If the change is text only, it also edits the matching text in
`felhom-system-poster.html`. If it needs a new drawing, it adds the line
**"System poster needs a refresh: <what changed>"** to `STATUS.md`'s "waiting on the operator" list.
**The session report names which of the three it did.**
Why the three-way split: the poster is a drawing made in Claude Design, and **nothing in this
repository renders it** — unlike `where-felhom-stands.html`, which has `render_stands.py`. So the
facts file is the source of truth and the drawing trails it. `scripts/poster_facts_gate.py` warns
when the facts file has a newer commit than the poster; it **never fails a push**, because a refresh
needs the operator and another tool, and a gate nobody can clear is a gate people learn to route
around.
+44 -4
View File
@@ -16,7 +16,16 @@
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one # A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
# layer up. That is the last step, and it runs ONLY on failure. # layer up. That is the last step, and it runs ONLY on failure.
name: gates name: gates
on: [push] # R-1129: `workflow_dispatch` is the manual PROOF that the failure mail can be sent. A push run is
# unchanged by it: the proof step below is skipped on a push, and `force_alarm` is then empty.
on:
push:
workflow_dispatch:
inputs:
force_alarm:
description: "Fail this run on purpose so the alarm step sends its real mail (a proof, not a gate result)"
type: boolean
default: false
jobs: jobs:
gates: gates:
@@ -55,8 +64,27 @@ jobs:
env: env:
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain. # In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000 GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
FELHOM_SCAN_FETCH: "1"
run: cd ws/felhom-agent && python3 scripts/agent_gates.py run: cd ws/felhom-agent && python3 scripts/agent_gates.py
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
# Runs ONLY on a manual dispatch, never on a push. It prints the key's LENGTH, never the key,
# so "CI can see the secret" is an observable rather than an assumption. With
# force_alarm=true it then fails on purpose so the alarm step below sends its real mail;
# the commit itself stays green, because its push run skips this step.
if: github.event_name == 'workflow_dispatch'
env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
FORCE_ALARM: ${{ github.event.inputs.force_alarm }}
run: |
python3 -c 'import os;print("RESEND_API_KEY length: %d" % len(os.environ.get("RESEND_API_KEY","")))'
if [ "$FORCE_ALARM" = "true" ]; then
echo "force_alarm=true - failing on purpose so the alarm step runs (R-1129 proof)"
exit 1
fi
- name: Alarm on failure - name: Alarm on failure
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO # THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches # notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
@@ -70,6 +98,8 @@ jobs:
if: failure() if: failure()
env: env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }} RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
# Empty on a push, so a push run's mail is exactly what it was before R-1129.
FORCE_ALARM: ${{ github.event.inputs.force_alarm }}
run: | run: |
python3 - <<'PY' python3 - <<'PY'
import json, os, sys, urllib.request, urllib.error import json, os, sys, urllib.request, urllib.error
@@ -81,13 +111,23 @@ jobs:
repo = os.environ.get("GITHUB_REPOSITORY", "?") repo = os.environ.get("GITHUB_REPOSITORY", "?")
sha = os.environ.get("GITHUB_SHA", "?") sha = os.environ.get("GITHUB_SHA", "?")
run = os.environ.get("GITHUB_RUN_NUMBER", "?") # R-1135: the link is for a PERSON, so it is the public address and the run ID.
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu") # NOT GITHUB_SERVER_URL: the runner registers at the in-cluster address, so that variable
# reads http://gitea.gitea-system.svc.cluster.local:3000, which nobody outside the cluster
# can open (forced-test mail, job 1997). NOT GITHUB_RUN_NUMBER: Gitea 1.26 reads
# /actions/runs/<n> as a run id first, so a run number that is also an older run's id
# opens THAT run (measured 2026-10-18: felhom.eu /actions/runs/1116 opened the run whose
# id is 1116 = run number 613). The link is printed too, so the job log carries it.
run = os.environ.get("GITHUB_RUN_ID", "?")
srv = "https://gitea.dooplex.hu"
print("ALARM-LINK %s/%s/actions/runs/%s" % (srv, repo, run))
body = json.dumps({ body = json.dumps({
"from": "Felhom CI <monitoring@felhom.eu>", "from": "Felhom CI <monitoring@felhom.eu>",
"to": ["admin@felhom.eu"], "to": ["admin@felhom.eu"],
"subject": "[felhom CI] gates FAILED in %s" % repo, "subject": ("[felhom CI] FORCED TEST, not a real failure - gates FAILED in %s"
if os.environ.get("FORCE_ALARM") == "true"
else "[felhom CI] gates FAILED in %s") % repo,
"text": ( "text": (
"The gate entry point exited non-zero.\n\n" "The gate entry point exited non-zero.\n\n"
"Repository : %s\n" "Repository : %s\n"
+18
View File
@@ -0,0 +1,18 @@
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
gitleaks generic-api-key@1ad7cc8eb57e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46ce5a3eccc4 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46fe126846b0 internal/escrow/identity_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@56474f1754f5 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@631c53997f7e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@6e790ef41ef3 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@78334b854242 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@7b88d323a44f internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b0eeee93cf24 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b1cb2659967a internal/localapi/escrow_stage_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b66bec43cc8a internal/escrow/recover_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e0525160fe03 internal/pbsdr/wrapper_read_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e442e455c360 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f23f93f461b2 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f471b78e0f6e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
+862 -1
View File
@@ -1,4 +1,865 @@
## unreleased ## Unreleased (next: v0.167.0) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — committed, not released
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
newest. Full `go build/vet/test` green. The controller and hub move the same night.
- **R-1232 (P2):** `backup.BackupRunner.NewestArchiveTime` — the tier due-check — counts only archives written with THIS
storage's key on an encrypted storage (the R-727 discriminator; the archive carries no host id). Before: a reinstalled
box shares the customer's PBS namespace and guest number with the old box, the old box's archive read as „the last
backup", and the whole-guest tier waited up to its 168-h cadence (the 2026-10-23 test night). A key lookup failure is
an error (the due-check reads UNKNOWN), never a foreign age. Tests `TestR1232_*` (foreign only → none; own → its time;
an unencrypted dir → unfiltered), red-proved (case 1 returned the old box's time).
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
second later — so 0.166.0's pass never ran on the bundle that delivered it.
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
agent-only — holding the bundle job until the commit would need new hub queue logic.
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
removed → commit only, no pass — the demo-hp symptom).
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
0.165.0 box passes even without this fix.
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
„BUNDLE POLICY DONE install_mode=appliance removed=0".
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
drop-in until its next bundle. A change inside the wrapper alone cannot fix the run that delivers it.
- **The fix — the new AGENT asks the new wrapper:** `internal/osupdate/bundle.go` runs a policy pass right after every
bundle install (`mode: "bundle-policy"`, host layer, no envelope); the wrapper on disk is the bundle's own by then.
`configs/felhom-os-apply` `Bundle.policy_pass`: on a root-owned BYO record it removes the appliance-only files WE
wrote (our header), keeps the previous copy beside the bundle's, `sshd -t` then reloads ssh — a failing `sshd -t` puts
the file back. It writes nothing, so it needs no signature; an appliance and the owner's own file are left alone. Never
fatal for the agent: an older wrapper refuses the unknown mode and the next bundle does it. Report key `bundle_policy`.
- Tests: `TestR1285_AnInstalledBundleIsFollowedByThePolicyPass` (a refused bundle runs no pass);
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
`documentation/audits/night-2026-10-25/redproof-r1285.txt`.
### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
usable that long, and the nonce store no longer keeps its record that long.
- `configs/felhom-os-apply`: the root wrapper holds the same 24 h (`MAX_WINDOW_S`, refusal R3 „the signed job's window
is longer than 24 h").
- `cmd/felhom-opsign`: `-ttl` above 24 h (or ≤ 0) is refused before anything is signed.
- Tests: `TestVerify_RejectsPerStage/window_longer_than_24_h` and `…/window_of_exactly_24_h_is_accepted`;
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
(`hub_min_agent` 0.158.1, `day0_agent` 0.165.0 — copies of hub state, because CI holds no hub secret; the session that
records a vouch or a new MinAgent updates them in the same commit). `check-published-versions.py` asserts the window
plus every pin; a pin with no tag FAILS. CI was red since the operator's hand prune of 0.159.0–0.160.0 (job 2334).
- **Decoys** (`test_gate_decoys.py`, via `FELHOM_RETENTION_POLICY`, test only): a pinned version older than the window is
asserted; its 404 convicts; a pin with no tag convicts. Seen failing on the old gate (two LIVE HOLES): felhom.eu
`documentation/audits/night-2026-10-25/redproof-d50.txt`.
## v0.165.0 (2026-10-25) — D43 = A: BYO hosts keep their own SSH settings (R-1257); the line test for a box that seeds all day (R-1250) — released in the attended release run; **VOUCHED by the operator 2026-10-25 18:38:49 CET** (binary `e9d04a47…`, bundle `8dea1f34…`)
**Found at the release (felhom.eu R-1285, P4):** on a BYO host upgraded from 0.164.0 the drop-in is removed only on the bundle run AFTER the one that installs this wrapper — the first run is planned by the old wrapper. Proven on a BYO test host (VM 9371 on demo-hp): the second `agent_config_update` logged `BUNDLE REMOVED … (ours, on a BYO host — D43)`.
### BYO hosts keep their own SSH settings (R-1257, D43 = A) — 2026-10-24 night
- **The root-key-only sshd drop-in (`/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`, R-982 / D4) goes only to an
appliance** — operator ruling D43 = A (2026-10-24): a BYO host belongs to its owner (`11` §1). The table entry's policy
is now `appliance`: written only when the ROOT-OWNED install record (`/var/lib/felhom-install/state.json`) says
`appliance`. On a `byo` record nothing is written, and a copy WE wrote (this exact path, beginning with our header
line) is REMOVED, its previous copy kept for the undo, then `sshd -t` and a reload. A box whose record is absent or
agent-writable proves nothing and is left exactly as it is. The bundle report lists `removed`; the record says
`removed` / `skipped`. The installer records the mode before it installs the bundle, so a new appliance still gets it.
- **X3: the „in force" check asks sshd about each of the host's own LAN addresses** (`ip -4 -o addr show scope
global`), not only 192.0.2.1 — a `Match Address <LAN>` block that re-allows root passwords no longer reads as in force.
- **X4: a failing `sshd -t` puts back ONLY the SSH step** (`undo_sshd_step`: the previous copy, or none; sshd is not
reloaded, so it never ran the refused state) and the rest of the bundle stays; it used to refuse and undo the whole
bundle. The report carries `sshd_step_undone`; the record names what is on disk.
- Tests (`configs/test_felhom_config_bundle.py`, class `ByoKeepsItsSSH`, 7 tests; each rule seen failing with it taken
out): a BYO host gets no drop-in; our copy comes off a BYO host; the owner's own file stays; no record → left as it
is; a LAN `Match` block is not in force; every address key-only passes; a failing `sshd -t` undoes only the SSH step.
The fake box now carries an appliance record by default (every earlier test unchanged).
### The line test for a box that seeds all day (R-1250) — 2026-10-24
Attended brief 2026-10-24, Part 4 (commit only; the release rides the next normal train). The weekly test (v0.164.0)
starts only after 10 quiet minutes; a torrent app that seeds all day never gives them, so that box never got a tested
line and qBittorrent never got its 50 % limit. Now:
- **`linetest.Tester.WaitingForQuiet`** — due, inside the window, tries left, tunnel up, and only the quiet missing.
`GET /host/line-test` carries it as `waiting_for_quiet`; a controller with a seeding torrent app acts only on it.
It needs the full watched window (11 minute samples): a freshly started agent is not known to be busy, so it never
invites a hold (found in the live proof, 2026-10-24: the first build said `true` before one minute was watched;
`TestFreshTesterIsNotWaiting`).
- **`linetest.Tester.RunNow`** and **`POST /host/line-test/run`** (synchronous, about 55 s) — a test on request while the
controller holds its app's upload low: every weekly rule holds (due, window, `MaxTriesPerDay`, tunnel, the one heavy-op
gate) except the 10 quiet minutes, which become a 5 s settle and a 10 s quiet check (below 2 Mbit/s, so a request
never measures the family's own upload); the three runs are 5 s apart instead of 30. A refusal answers its code
(`not-due`, `outside-window`, `tries-used`, `no-tunnel`, `heavy-op`, `not-quiet`, `server-busy`); a failed request
spends one of the day's tries. `Tick` and `RunNow` now share one `ready()` check.
- Tests (each seen failing): `TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt` (the sleeps, the 60 s bound),
`TestRunNowRefusesAnUplinkThatIsStillBusy`, `TestRunNowKeepsTheWeeklyRules`, `TestRunNowFailuresSpendTheDaysTries`,
`TestQuietBoxIsNotWaiting`, `TestLineTestRunRoute`; a compile-time assertion that the Tester serves the request half.
**No sudoers or config-bundle change. MinController: none** — an older controller never calls the route.
## 2026-10-24 (review round 2, unattended) — `.claude/rules/unprompted-work.md` aligned with its four siblings
- The copy lacked item 6's paragraph „A version number is pushed once …" that the felhom.eu and felhom-controller copies
have carried since 2026-10-15; the file's header says „change all five or none". The paragraph states a fact
(`felhom-controller/controller/build.sh` refuses an existing tag and an unreachable registry). Now byte-identical to
the other four (md5 b1891bd4). No code change. Record: felhom.eu `documentation/audits/review-round-2-2026-10-24/EDITS.md`.
## v0.164.0 — the weekly upload line test (R-1247) (2026-10-24) — RELEASED on the operator's ruling (attended release run); vouched by the operator 09:44:58 CEST
Binary sha256 `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`; config bundle sha256
`1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341` (configs unchanged — no bundle job sent). Built from
`f7e1dcb` (CI 2277 green) in a fresh clone by `scripts/release-agent.sh`. Delivered by one signed `agent_update` per box
(07:25Z): Tester 1, demo-hp and demo-felhom — COMPLETED, self-check 76/76, committed after the 1-minute dwell, the line
test started (felhom.eu `audits/line-test-2026-10-24/agent-0164-boxes.txt`).
Operator ruling 2026-10-24 07:26: once a week the box's HOST measures its upload line with a 10-second iperf3 upload to an
iperf3 server on ep0 that answers **only inside the WireGuard tunnel** (felhom.eu `scripts/ep0-iperf3/`; installed the same
day). The agent speaks the iperf3 protocol itself (operator's pick, 2026-10-24: nothing is installed on any host).
**No sudoers or config-bundle change.** **MinController: none** — the controller (≥ 0.342.0) reads the new route and
treats a 404 as „not measured".
- **`internal/linetest`** — `Client` (the iperf3 3.x client protocol: cookie, params, N data streams, TEST_END, results
exchange; the value is the SERVER's received bytes) and `Tester` (the schedule): due weekly from Tuesday 10:00 local,
starts 10:00–15:59 only (never in the night chain), from a per-host slot spread over 10:00–13:59; only after 10 quiet
minutes (every one-minute transmit rate of the host's physical uplink below 2 Mbit/s); under the one-heavy-op gate
(`line-test`); 3 runs of 10 s with 4 streams, 30 s apart, the highest counts; each run bounded by 25 s; „server busy" →
tried again at the next tick the same day, at most 6 tries a day; no operational tunnel → no test, no value. The last
result is persisted (`<oob state dir>/line-test.json`).
- **Report:** a `line_test` stanza (`schedule`, `last_ok_at`, `mbps`, `runs_mbps`, `server`, `last_attempt_at`,
`last_error`). **Local API:** `GET /host/line-test` → `{present, mbps, measured_at, method:"iperf3-tunnel"}`.
- `wgtunnel.Manager.LineTestTarget()` — the desired block's `pbs_tunnel_ip`, only while the tunnel is operational
(cached, no exec). Config `line_test.disable` (default on).
- **Proven live** (felhom.eu `audits/line-test-2026-10-24/`): a probe built from this package on demo-hp's host read
**90.7 Mbit/s** in 10.4 s (the push floor said „at least 10"); a second test 2 s later → „server busy" in 0.1 s while the
first went on undisturbed (90.6).
- Tests, each red-proved by a mutation (15): `TestClientSpeaksTheProtocolAndCountsTheServersBytes` (against a fake
iperf3 server), `TestClientBusyServerIsErrBusy`, `TestClientHonoursTheDeadlineAgainstAStalledServer`,
`TestEligibleWindowNeverInTheNightChain`, `TestNeverTestsInTheNightOrBeforeTuesday`, `TestBusyServerIsRetriedLaterTheSameDay`,
`TestTriesPerDayAreBounded`, `TestNoTunnelNoTest`, `TestBusyUplinkOrHeavyOpWaits`, `TestTickMeasuresTheHighestOfThreeRuns`,
`TestStateSurvivesARestartAndReportsIt`, `TestUplinkCountsTheBridgesPhysicalPortsOnly`, `TestLineTestRoute`,
`TestLineTestTargetOnlyWhileOperational`, `TestMainWiresTheLineTest`.
## v0.163.0 — the tunnel heals itself after a clock that ran ahead (D30, R-1204); root SSH from the LAN by key only (D4, R-982) (2026-10-23) — RELEASED on the operator's yes (day brief); the vouch is the operator's
Binary sha256 `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c`; config bundle sha256 `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8`. Built from `1109092` (CI 2246 green). Both sections below are in it.
### The tunnel re-key (D30) and the D4 bundle drop-in
- **D30 (operator 2026-10-22: the agent makes a new key and registers it on the existing peer path).** When the latest
handshake stays stamped in the future for 15 minutes (the R-1204 sign: ep0 keeps the future timestamp and refuses
every newer handshake from that key), the agent makes a new WireGuard key pair, keeps the old private key as
`wg/private.key.prev` (0600 — the escrow may hold it), and registers the new public key through `POST /hosts/<id>/wg`.
The hub re-keys the peer IN PLACE (same /32, generation bump) and its full-list push to ep0 drops the old key and adds
the new one — nothing new runs on ep0. A new key is a new peer for ep0, with no stored timestamp, so the next
handshake is accepted. At most once an hour; not while the tunnel unit is down; never while a re-register is already
in flight. A failed registration is retried by the existing re-key-in-place path. Logs carry public keys only.
Tests `TestD30_*` (4), red-proved. A host-loss DR that installs the escrowed (older) key registers it again, and the
hub re-keys in place back to it — the same path.
- **Review of the night's own code (felhom.eu `REPORT-night-2026-10-23.md`, N6, N7).** The rotation never leaves the box
without a readable key: the new key is written and fsynced to a temp file, the current key gets its kept name by a
hard link, then the temp file is renamed OVER `private.key` and the directory fsynced (a power cut or a kill between
two renames used to leave no key, or an empty one). `private.key.prev` keeps the FIRST old key (the escrow may hold it);
later ones are `private.key.prev.<unix>`, the newest three kept. The once-an-hour bound is saved in `wg/last-rekey`, so
an agent restart does not reset it; a start or a last re-key dated AFTER now (a clock that jumped back) is dropped,
never a block until that date (the commit's own security review). Tests `TestD30_TwoRotationsKeepTheFirstOldKey`,
`TestD30_ALastReKeyFromTheFutureDoesNotBlock`,
`TestD30_TheHourBoundSurvivesARestart`, red-proved. **Live (felhom.eu `audits/night-2026-10-23/P2-D30/`):** the D30
re-key ran on a test box (before these two changes): clock corrected → 15 min → new key registered → the box switched
at its next desired-state fetch → a handshake with the new key.
- **D4 (operator ruling 2026-10-22, R-982) — root SSH from the LAN by key only, on `main` now (2026-10-23 day brief).** The
config bundle carries `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`;
content check: exactly that line, `sshd -t -f`). After an install that wrote it: `sshd -t` (the whole config), then
`systemctl try-reload-or-restart ssh` (a stopped or socket-activated sshd no longer refuses the bundle), then `sshd -T`
is read back and a drop-in that is NOT in force (no Include, an earlier drop-in, a Match block) is said loudly and
reported as `sshd_dropin_not_in_force` — never a refusal; an undone bundle reloads sshd again. The console and the
web page keep the root password; felhom-sshd (8822) is not touched. Tests `RootKeyOnly` (6), red-proved; the
installer's uninstall removes the file (felhom.eu installer 1.36.0). Reviewed 2026-10-23 night (A1–A7).
- **Known gap, unchanged (filed):** after a reboot the kernel forgets the future stamp (latest handshake 0), so neither
this re-key nor the hub's alarm sees a box whose clock was corrected and which then restarted (R-1223).
### The future-stamped handshake is reported (R-1204) — first written on the 2026-10-22 night
- **R-982 / D4 — a ready patch on that night; on `main` since `1109092` and released in v0.163.0 (see above).** Built and proven the same night (felhom.eu
`audits/night-2026-10-22/P2-D4/`): a bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
(`PermitRootLogin prohibit-password`), a `sshd` content check (exactly `prohibit-password`; `sshd -t -f`), and after
an install that wrote it `sshd -t` + `systemctl reload ssh`; tests `RootKeyOnly` (3), red-proved. Live on a fresh
nested box from ISO 1.30.0: root password over SSH refused, root key works, the console and the web page still take
the password. It was committed (`a0c88e5`) and moved back out (`7651461`): the installer's uninstall must remove every bundle file (felhom.eu `scripts/test_hostinstall.py`), the bundle half reaches a box only with an agent release, and the installer half only with a new `installer-v…` tag (R-110 — a push to `main` alone publishes nothing) — both forbidden tonight. *(Corrected the same night: the first reason written here, „a push of `felhom-host-install.sh` publishes the installer", was wrong — the website serves `/scripts/` from the tag `installer-v1.35.0`, read live in `manifests/webpage.yaml` and the running Deployment.)* Both halves are patches: `felhom-agent-bundle-D4.patch` and `felhom.eu-installer-D4.patch` in that folder; apply them together with the agent release and the next installer tag.
- **R-1204 — `wireguard.handshake_future_s`.** A box that ran once with its clock ahead keeps a handshake stamped in the
future; ep0 refuses every newer one, and `last_handshake_age_s` (clamped at 0) read „just now" for ever. The heartbeat
now carries how many seconds the handshake lies ahead (> 60 s only); the age keeps its clamp. The hub alarms on it
(felhom.eu hub `wg_tunnel_stale`). Test `TestR1204_AFutureHandshakeIsReported`, red-proved.
## v0.162.0 — every guest may swap as much as its RAM (R-1192, the 8 GB freeze); the host's swap is reported and checked; no false „operator access degraded" on a fresh box (R-1194) (2026-10-22) — RELEASED on the operator's yes; the vouch is the operator's
Released 2026-10-22 07:40 UTC (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at
`b310773`: binary sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (the same bytes as the
night's candidate), config bundle sha256 `ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5`, tag
`v0.162.0`; no `configs/` change since v0.161.0. Delivered by one signed `agent_update` job per box to demo-hp, Tester 1
and demo-felhom (07:49–07:58 UTC): each `signed op COMPLETED`, `capabilities self-check ok=76 total=76`, `update
committed`, then `guest-swap: raised` for its 9201 (512 MB → its RAM). Evidence felhom.eu
`documentation/audits/morning-2026-10-22/release/04-agent-delivery.txt`. **Found at once:** demo-hp (host swap 8191 MB,
guest swap 25898 MB) and demo-felhom (8191 MB vs 12288 MB) logged the host-swap WARN and the hub mailed
`host_swap_short` — the check compares the host's swap with the guests' whole swap allowance, which a guest larger
than the installer's 8 GB host swap always exceeds (felhom.eu register R-1216). Soak (night of 2026-10-21, 8 GB box):
11 h, no freeze, the host's swap peaked at 2.7 GB used.
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
never set it. On the 8 GB machine (guest 4096 MB) the first uploads made Immich load its analysis models; the guest hit
its limit with the swap full and could evict only program code — 150 000–600 000 refaults and ~1 GB of disk reads per
10 s, memory „full" pressure 92–95 %, load 155, the dashboard dead for 6½ minutes, no app killed. Raising the swap cap
to 4096 MB live ended it within a minute while the uploads ran on.
- `reconcile.buildBringUpConfig`: a new guest gets `swap` = its memory.
- `POST /guest/memory` (RAM resize): `swap` moves with `memory` (`localapi.GuestSwapMB`).
- At agent start (one minute in): `localapi.EnsureGuestSwap` raises every LXC in the `felhom` pool whose swap is below
its RAM to it — a live cgroup change, no restart; never lowers a higher swap; one guest's failure is logged and the
others are still done.
- Tests `TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM`, `TestR1192_BringUpSetsSwapToTheRAM`, the resize test's
new swap assertion — each red-proved (the write removed → `raised 0, want 2`; `swap=""` on both other paths).
- Privilege: `swap` is under `VM.Config.Memory`, which the RAM resize already uses; no role, sudoers or bundle change.
Added the night of 2026-10-21 (still unreleased, the same release):
- **The host's swap is reported and checked (R-1192 follow-up).** The guest fix moves a memory burst onto the HOST's
swap; a host with none gets nothing from it. The host report carries `host.host_swap_mb` (Proxmox node status — the
kernel's SwapTotal, the same number as `/proc/meminfo`) and `host.guest_swap_mb` (the sum of the guests' `swap`; each
guest spec also carries `swap_bytes`). When the first is below the second the agent logs one WARN (and one INFO when
it is covered again); the hub raises `host_swap_short`. `GET /guest/memory` carries `host_swap_mb` and
`guest_swap_mb` for the controller's install page. The agent never creates host swap (a decision — felhom.eu
register). Measured the same night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap.
- **R-1194 — no false „operator access degraded" on a fresh box.** The first host report was sent before the
felhom-sshd loop's first pass, read a config that was never rendered („config invalid, port 0") and the hub mailed
`oob_degraded` — reproduced on a fresh install 2026-10-21 (the report at 20:34:00, the apply 0.4 s later, the daemon
healthy; the check runs `sshd -t` through sudo, so the non-root theory was false). The loop now reports no OOB stanza
until its first pass has run (a failed pass counts, so a real lasting failure still reaches the hub).
- Tests `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
`TestR1194_NoOOBStanzaBeforeTheFirstPass` — each red-proved.
## v0.161.0 — three fixes (R-921, R-1070, R-1010); two texts made true after 0.160.0 (2026-10-20) — RELEASED on the operator's yes, NOT VOUCHED
Released 2026-10-20 (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at `5152123`:
binary sha256 `ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5`, config bundle sha256
`485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47`, tag `v0.161.0`. Delivered by one signed
`agent_update` job per box to demo-hp (06:38 UTC), Tester 1 (06:54) and demo-felhom (06:55): each `signed op COMPLETED`,
`capabilities self-check ok=76 total=76`, `update committed`. **The vouch is the operator's.** Live on demo-hp: `host_busy`
answers `{"busy": true, "what": "backup:local"}` during a „Mentés most" whole-guest backup and `{"busy": false}` before
and after (read from inside 9201 with its own token; felhom.eu `audits/night2-2026-10-20/P1/`). R-1083 (a key for an
existing guest) is NOT in this release: see the R-1083 row. No sudoers, wrapper or config-bundle change.
- **R-921 — `GET /backup/status` says when the host is busy.** New field `host_busy` (`{busy, what}`): whether the
host-wide heavy-operation gate is held right now (another guest's backup, the restore test, a night OS step, the
stale-lock sweep, a power-off) — the state that makes a `POST /backup` answer 409. Served on every request; absent
when the gate is not wired. Controller ≥ 0.335.0 reads it before it stops the household's apps for a tier, so a
refused tier costs no stop (the measured case of 2026-10-08: the night OS step, all 15 apps stopped for a refusal).
A read, never a reservation. No new route. Test `TestBackupStatus_R921_HostBusy`, red-proved (assignment removed →
`free gate: host_busy = <nil>`).
- **R-1070 — a box with no tunnel set up no longer fails its host OS update.** Docker 29 prints „error: no such
object" in lower case, so the tunnel probe's case-sensitive match fell through to `unknown` („could not ask the
guest"), and the host health rule failed every box without a cloudflared container (Tester4, 2026-10-15,
`os_update_health_failed`). Now the match ignores case and a missing container is a new state, **`absent`** („no
tunnel set up on this box"); the host rule reads it as not applicable only when the reading from before the update
(Docker answered) also had no cloudflared container. A tunnel that is down, gone or unreadable still fails. The hub
alarms only on `not_running`, so `absent` never mails. Tests `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows) and
three `TestClassifyTunnel` rows (Docker 29's wording measured on demo-hp), red-proved both ways.
- **R-1010 — the stale-lock janitor clears a half-removed `vzdump` snapshot.** When the delete fails ONLY with
„Failed to find logical volume …_vzdump" (one disk's snapshot already gone — demo-hp 2026-10-14 after a backup ran
out of space), it retries once with Proxmox's own `force=1` on the same API call (`DeleteSnapshotForce`; the same
token privilege, no sudoers line, no root command — `pvesh usage … -command delete` lists `--force`). Any other error
is returned unchanged; a failed forced delete is returned, never a second force. Tests
`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone` (6 rows) and `TestDeleteSnapshotForce_R1010`, red-proved
(single plain delete → `calls [plain], want [plain force]`).
- **The security-scan gate (R-986), tooling only:** `agent_gates.py` registers the SHARED
`felhom.eu/scripts/security_scan_gate.py` (FAST set: the hook and CI) — govulncheck (called vulnerabilities) and gitleaks
on the pushed commits, **warn-first**. `.security-accepted` holds the first scan's 15 accepted hits (escrow and
local-API test constants), each with a reason. CI fetches the pinned gitleaks (`FELHOM_SCAN_FETCH=1`). Decoys: the
shared 8 plus the gate on this repo, in `test_gate_decoys.py`.
### Earlier on 2026-10-19 (texts only)
- The ceremony-complete log line now says the package uploads at the claim **or at the typed-back confirm** (R-1151);
it read „package uploads at the claim" for both orders (seen in demo-hp's journal, 2026-10-19 01:47 UTC).
- The R-973 comments in `cmd/felhom-agent/main.go` and `internal/localapi/backup_cap.go` named the hub event
`agent_capability_degraded`; since hub v0.159.0 (R-1150) a refused backup raises `whole_guest_backup_refused`.
Found by the hub helper of the day brief 2026-10-19.
- **gofmt, R-1160:** `1edfd8e` formats the 14 Go files that were not gofmt-clean (formatting only, `go test ./...`
green). New gate `scripts/gofmt_gate.py` (the controller's R-454 shape) in `agent_gates.py`, fast set; its decoys in
`scripts/test_gate_decoys.py` plant an unformatted file in a SCRATCH copy (convicted by name), the formatted twin
(passes), and the no-Go modes (CI passes out loud, a dev machine is INCONCLUSIVE). Red-proved: with the conviction
switched off the FACT case reports a LIVE HOLE. The next release carries it; no release for it.
## v0.160.0 — five fixes: an abandoned or un-typed-back escrow ceremony keeps the old code (R-978, R-1151); a whole-guest backup flood is capped (R-973) and the local tier keeps one point per day (R-1149); an empty partition is not „data" (R-1093) (2026-10-19)
Released 2026-10-19 on the operator's yes (02:48 answers, D17 „one per day, 3 days"). sha256
`7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1`, config bundle
`6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f`. The vouch is the operator's.
- **R-1093 — an EMPTY partition no longer asks the typed „data will be erased" confirmation.** `lsblk` prints the
parent disk's table type (PTTYPE) on every partition, so `SudoHostOps.InspectDevice` read a blank partition as
„device has a partition table". Now lsblk's PTTYPE is set aside only when `blkid -p` on the partition itself proved
it read the device and that it is a partition of a table of exactly that type (`PART_ENTRY_NUMBER` +
`PART_ENTRY_SCHEME` equal to lsblk's PTTYPE). The partition's own signatures still come from `blkid -p`: a
filesystem or a nested table inside it stays data-bearing; a blkid that printed nothing, a whole disk, or a scheme
that differs keeps the old, safe reading. **No new command, no sudoers or config-bundle change** (the two probes are
the same allow-listed argv). Test `TestInspect_R1093_PartitionVerdicts` (8 rows), red-proved: with the old
`if pttype != ""` both empty-partition rows fail `DataBearing=true reason="device has a partition table"`.
**Hardened after the same night's review (fail-closed on every path):** `PART_ENTRY_*` is read from the PARENT's
table, so lsblk's PTTYPE is set aside only when blkid also **exited 0** — exit 2, an I/O error or a missing binary
keep „device has a partition table"; and an `lsblk` that exits 0 with output that does not parse into a device (or
lists none) now leaves the device UNPROBED („could not be reliably inspected") instead of „blank" — that hole
predates R-1093. Six more rows in the same test, each red-proved (`blkidErr` check removed → „device is blank";
`&& parsed` removed → `Probed=true … „device is blank"`).
- **R-978 — an abandoned second escrow ceremony no longer makes the customer's recovery code useless.** The root
ceremony (the FELHOM_ESCROW argv, unchanged: `--upload --output=json`) used to upload the new package before the new
code was claimed, and the hub overwrites the current package when the sealed secrets are unchanged — so a customer
who started the ceremony again and walked away (or an agent restart in the 10-minute window) was left with a
written-down code that opened nothing. Now json mode hands the package back (`CeremonyOutput.PendingUpload`, output
**version 2**; text mode still uploads at once) and the daemon uploads it at `POST /escrow/ceremony/claim`, **before**
the code is handed out (`EscrowCeremonyConfig.Upload` → `putEscrowBody`, the same PUT and per-host key). A failed
upload answers 502 and keeps the code claimable; the TTL and a new start wait while the claim's upload is in flight;
no uploader → 503; an output that uploaded by itself or carries no package fails the ceremony. Status `uploaded`
now reads false until the claim. The controller is unchanged (it never read `uploaded`; its claim already assumed
the package is on the hub when the claim returns). **No sudoers or config-bundle change.** Tests (localapi)
`TestEscrowCeremony_R978_*` (abandon by TTL / by a new start / by a restart → the hub's package still opens with
code 1; claim stores before handing out; upload failure; in-flight TTL; no uploader; no package) and (cmd)
`TestEscrowOptsFromCLI_JSONModeDefersUpload`, `TestDeliverEscrowPackage_JSONModeSendsNothing`,
`TestR978_CeremonyUploadIsWiredIntoTheDaemon`. Red-proved: upload-at-create restored → „code 1 no longer opens it
(uploads=2)"; the in-flight TTL guard removed → „the TTL voided the code while its package was uploading";
`deferUpload: false` → „json mode must defer the upload"; the `Upload:` line removed → „has no Upload". **Residual:**
the customer's typed-back check after the claim is browser-only, so „confirmed" here means „claimed" — a customer
who opens the code and closes the page without writing it down has replaced the package.
- **R-1151 — the new recovery code replaces the hub's package only after the household typed it back.** R-978 moved
the upload to the claim; but the wizard's typed-back check was the browser's alone, so a household that opened the
new code and closed the page without writing it down had already replaced the package their old code opens. Now a
claim with `{"defer_upload": true}` (controller ≥ 0.332.0) hands out the code with two word positions
(`verify_positions`, drawn by crypto/rand) and keeps the package; the new `POST /escrow/ceremony/confirm
{"words": [w1, w2]}` checks the two words (constant time, case and spaces aside) and only then uploads it. Right → 200;
wrong → 422, still waiting; one word → 400; nothing waiting → 404; the 30-minute window ran out → 410 and the job
reads `unconfirmed_void` (the package dropped, the previous code still works); hub failure → 502, still waiting. Only
the two words are kept for the check, never R; a new start, the window and a restart drop them with the package.
Status gains `awaiting_confirm` and `confirm_expires_in_sec`. **An old controller (≤ 0.331.0) claims with `{}` and
never confirms: it keeps the R-978 order (upload at the claim)**, so its household's ceremony still completes; a new
controller on an old agent sees no `confirm_required` and needs no confirm. No sudoers or bundle change. Tests
`TestEscrowCeremony_R1151_*` (shown but not typed back — by the window, a new start, a restart — code 1 still opens
the hub's package; typing; the old controller's order; hub failure at the confirm) and `TestPickConfirmPositions`.
Red-proved: the defer flag ignored → „a code that was shown but never typed back replaced the package: code 1 no
longer opens it (uploads=2)" in all three; the word check skipped → „one wrong word: got 200 (uploads=1)".
- **R-1149 — the local whole-guest tier keeps one restore point per day for 3 days** (operator decision D17,
2026-10-19). The primary (local) tier's per-run prune is now `--prune-backups keep-daily=N` (N =
`local_backup_retention`, default 3) instead of `keep-last=N` — the same at most 3 archives, but a run of extra
backups inside one day replaces only that day's point, so with the R-973 cap a flood cannot push out the earlier
days' copies. Additional tiers keep their explicit `keep_last` (the weekly off-site „last 2" is two weeks).
`config.BackupTier.PruneSpec()` is the one place the spec is built (runDaemon used to format keep-last itself); the
„backup tier armed" log line names it. Tests `TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay` and the
consequence `TestBackupCap_R1149_LocalFloodKeepsOlderDays` (the agent's real spec, a Proxmox-shaped prune model,
three nights then a day-long flood → the two earlier nights survive). Red-proved: keep-last for the primary →
„primary tier prune = "keep-last=3", want "keep-daily=3"" and „holds 0 copies made before the compromise". No
config or sudoers change.
- **R-973 — the local-API whole-guest backup route has a cadence cap.** Every successful run prunes its tier, so a
compromised controller calling `POST /backup` in a loop pushed out every pre-compromise restore point (local
keep-last 3; off-site via ep0's keep-last 2). Now (`internal/localapi/backup_cap.go`): a tier whose cadence is ≤ 24 h
allows **3 successful runs per rolling 24 h** (the `07` §6 day: the night run, a „Mentés most" press, and the next
night, which since R-899 still runs and can land inside the same 24 h — so a press never costs the night); a longer
tier allows **1 per max(cadence − 24 h, cadence/2)** — 6 days for the 7-day off-site tier, which the controller only
runs when due and the button never targets. Only successes count (a failed run prunes nothing). A refusal answers
**429** with the next allowed time, is logged at WARN, is saved, and for 24 h raises the CRITICAL host-report status
`localapi:backup-cadence-cap` — the hub's existing operator-only `agent_capability_degraded` event, **no hub change**.
**The cap survives an agent restart:** `BackupSuccessState` (the R-894 file) now also keeps the newest 8 success
times and the last refusal per tier and guest (additive JSON fields; a pre-R-973 file counts its one known
success). Tests `TestBackupCap_*` (an off-site flood keeps a pre-compromise copy for 5+ days; a local flood runs 3
a day; the scheduled day is never refused; restart; failures; the operator signal) + `TestBackupCadenceCap`,
`TestBackupSuccessState_R973_*`, `TestBackupCapStatus_*`, `TestR973_CapRefusalIsWiredIntoTheReport`. Red-proved:
the cap disabled → „the off-site tier lost every pre-compromise copy: 10 runs"; the disk history ignored → „after a
restart the 4th run in a day was allowed"; the wiring line removed → „never binds backupCapRefusal". **Limits, stated:**
with keep-last 3 the local tier's three points can still be replaced within one day (3 per day is the schedule's own
pace plus one press); the hub's event text reads „agent privileged capability degraded" (the type is generic). No
sudoers or bundle change.
- **CI workflow (R-1135, no code change).** `.gitea/workflows/gates.yml`, the alarm step: the mail's run link is now `https://gitea.dooplex.hu/<repo>/actions/runs/<run id>`.
It used `GITHUB_SERVER_URL`, which on this runner is the in-cluster address (`http://gitea.gitea-system.svc.cluster.local:3000`,
the address the runner registers at), so nobody outside the cluster could open it. It also used the run NUMBER, and
Gitea 1.26 reads `/actions/runs/<n>` as a run id first: a number that is also an older run's id opened that other run
(measured 2026-10-18: `felhom.eu/actions/runs/1116` opened run id 1116, run number 613). It now uses `GITHUB_RUN_ID` and
prints the link (`ALARM-LINK …`) into the job log. Same change in all four repos' workflows.
## v0.159.2 — the orphan thin check runs only after a failed backup (R-1012 follow-up) (2026-10-18)
**Released 2026-10-18 (committed 2026-10-17): sha256 `4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9`, bundle `477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9`; the vouch is the operator's.** No `configs/` change since v0.159.1. The independent review of 2026-10-18 found nothing in it. v0.159.0/v0.159.1 ran the R-1012 orphan check
before EVERY snapshot-mode whole-guest backup, and that check takes a short metadata snapshot of the live thin pool.
Now `BackupRunner` runs it only when the snapshot step has failed (the vzdump start or its task) — the moment the
answer is needed. The finding is still appended in plain words to the failed backup's error, exactly as before; a
successful backup never runs the check and keeps a clean record; stop mode and an unwired runner are unchanged. The
`thin-orphan-check` capability text says „after a failed whole-guest backup". Tests
`TestBackup_SnapshotFailure_NamesTheOrphan` (now: exactly 1 call, after the failure) and
`TestBackup_ThinCheck_SilentWhenNotApplicable` (now: 0 calls on a successful backup), red-proved against the v0.159.1
`runner.go`: failure legs „thin check ran 2 times, want 1", success leg `calls=1`. No root file, sudoers or bundle
change. (Installer side, felhom.eu, same night: `felhom-host-install.sh --rotate-recovery-only`, R-1141.)
**Proven live 2026-10-18 (demo-hp):** a green snapshot backup with 0 `felhom-thin-check` calls (the 0.159.1 night backup
had 1). The failed leg stays covered by the fixture test.
## v0.159.1 — the signed decommission prunes the guest-bind record (found by v0.159.0's own live proof) (2026-10-17)
**v0.159.0 is superseded — do not vouch it.** Everything of v0.159.0 below, plus: `signedjobs.DecommissionExecutor`
drops the drive from every guest's bind record (`SetGuestBinds`, wired to the agent's `GuestBindStore`), as the
self-serve decommission always did. Without it a drive decommissioned by an operator-signed job and then forgotten
(R-1124) left a record the 20-second guest-bind re-assert warned about every tick („reconcile: skipping non-enrolled
drive") — seen on demo-hp minutes after the R-1124 proof. Test `TestDecommissionExecutor_PrunesTheGuestBindRecord`,
red-proved. Released: tag `v0.159.1`, sha256 `de8f6ad9…aff9`, bundle `2af3d27d…7cd6` (the same root files as v0.159.0).
**Not vouched.** Delivery to a box on the v0.158.1 bundle: `agent_update 0.159.1` → `agent_config_update 0.159.0-step1`
(sha `1d65edb3…d6cf`, published) → `agent_config_update 0.159.1` (the bundle adds `felhom-thin-check`, R16).
## v0.159.0 — forget a decommissioned drive that is gone (R-1124); name an orphan thin device in a backup failure (R-1012); Go 1.27.2 (R-972) (2026-10-17)
**Superseded by v0.159.1 — do not vouch.** Task: „agent release, the catalog allow-list, break-glass rotation, off-site
history check" (felhom.eu `REPORT-agent-allowlist-rotation-2026-10-17.md`).
- **Go 1.27.2 + x/crypto v0.58.0** (R-972, `37b99d7`): the first release that carries them (v0.158.1 was built with
go1.26.0).
- **R-1124 — a decommissioned drive that is gone is forgotten.** `SudoHostOps.ForgetRetiredDrives` (agent start and the
20-second storage tick): for a Felhom drive unit whose intent is `decommissioned`, whose `/dev/disk/by-uuid` link is
gone and which is not mounted, `systemctl disable` + the new root verb `felhom-priv-apply unit-forget <unit>` +
`daemon-reload`. The unit is gone, so a later boot no longer waits for the drive and the agent stops listing it (no
„absent by UUID" line every 20 s). A decommissioned drive still plugged in KEEPS its unit and raw mount, so the
one-click re-enroll (H3) still works. `unit-forget` re-checks as root: a felhom by-UUID drive unit (not a network
unit), disabled, its `Where` not mounted, its by-UUID link absent — otherwise refused. Sudoers `FELHOM_MOUNT` gains the
one verb. Tests `TestForgetRetiredDrives_*` and `configs/test_felhom_priv_apply.py` `UnitForget`, red-proved.
**Live on demo-hp:** a loop-backed test drive enrolled through the local API, decommissioned by a signed job (kept
its unit while present), unplugged → 8 s later the unit and its wants link were removed and `GET /disks` dropped it.
- **R-1012 — an orphan thin device is NAMED in the backup failure, never removed.** New READ-ONLY root helper
`felhom-thin-check <vg> <pool>` (config bundle): LVM's thin ids before and after, the pool's ids through a metadata
snapshot it reserves and releases (refuses — exit 3 — when someone else holds one; releases on every exit; sends no
other pool message). `internal/thincheck`: an orphan = in the pool and in neither LVM list, on two reads 3 s apart;
fails open. `BackupRunner.SetThinCheck`: before a snapshot-mode backup (a WARN when found) and again after a failure —
the finding is appended in plain words to the error the controller mails to the operator, pointing at the new
runbook `whole-guest-backup-orphan-thin.md`. Sudoers `FELHOM_THINCHECK`, capability `thin-orphan-check`. Tests
`internal/thincheck`, `TestBackup_SnapshotFailure_NamesTheOrphan`, `configs/test_felhom_thin_check.py`, red-proved.
**Live on demo-hp:** on a scratch thin pool (a loop file on `/mnt/hdd_1`, never `pve/data`) the helper listed a made
orphan (device 7) and refused a snapshot held by someone else; on `pve/data` it read 1, 2 = LVM's and released; a
normal whole-guest backup of 9201 ran the check first and finished green.
- Released: tag `v0.159.0`, sha256 `3cfd0a25…edfed5`, bundle `40663c4f…6d681`; step bundle `0.159.0-step1`.
## CI only — a manual proof that the failure mail can be sent (R-1129, 2026-10-17)
No agent release; no binary changes.
`.gitea/workflows/gates.yml` gains a manual `workflow_dispatch` trigger with one input, `force_alarm` (default false), and
one step that runs ONLY on a dispatch: it prints the length of `RESEND_API_KEY` (never the key) and, with
`force_alarm=true`, fails on purpose so the existing alarm step sends its real mail; the mail's subject then starts
„FORCED TEST, not a real failure". A push run is unchanged: the new step is skipped and the subject is the old one. Same
change in all four repos' workflows. Purpose: prove the failure mail now goes out after the operator set the key
(every failed run until today ended „ALARM FAILED: RESEND_API_KEY is empty").
## scripts only — the decoy suite is a gate of the full run (R-1116, 2026-10-17)
No agent release; no binary changes. `scripts/test_gate_decoys.py` is now the `gate-decoys` row of `agent_gates.py`:
the full run (which CI runs on every push), not `--fast` — it takes ~20 s in the workspace (measured: 30 cases; ~5 s in a
CI-shaped shallow clone, also green). It was in no runner, the shape that let the catalog's R-1115 fail unseen for a
night. It plants only in scratch clones and asks a fake Gitea on 127.0.0.1; it does not call the runner (no
recursion). Declared in its own `COVERS`; `test_agent_gates.py` pins it (registered once, full run only, named by
`--fast`'s SKIPPED line). Red-proved: a reuse-refs FACT case flipped to expect rc 0 turned the full run red
(`CONVICTED: gate-decoys`); restored, green. CLAUDE.md's `--fast` sentence names it.
## unreleased — Go 1.27.2 and golang.org/x/crypto v0.58.0 (R-972, 2026-10-16)
**Not released, not deployed** — agent releases wait for the operator's vouch; the next vouched release carries this.
`go.mod`: `go 1.27.2` (was 1.25.0; the shipped 0.158.1 binary was built with go1.26.0), x/crypto v0.58.0 (was v0.52.0),
x/sys v0.49.0. `go build`, `go vet`, the full `go test ./...` and `agent_gates.py --fast` green on Go 1.27.2.
`release-agent.sh` builds with the toolchain `go.mod` names (GOTOOLCHAIN=auto), so no script change.
## scripts only — `release-agent.sh` refuses when the registry cannot say the version is new (R-1098, 2026-10-16)
No agent release; no binary changes. `scripts/release-agent.sh` already refused a version whose tag exists or whose
package answers 200. It let everything else through — a registry it could not reach (curl prints `000`) or a 500 read
as „new". Now only a clean `404` proves the version is new; anything else refuses with the code it got. New
`scripts/test_release_push_once.sh` (the real script, a fake `curl` and `go` on PATH; published / absent / unreachable /
server error / tag exists), red-proved: with the 404 rule removed, the unreachable and server-error cases fail. Same rule
as the controller `build.sh` and, since today, felhom.eu `scripts/build-hub.sh`.
## v0.158.1 — security review of v0.158.0: the mkfs wrapper trusts no sibling claim; the internal-place helper fails closed (2026-10-16)
**v0.158.0 is superseded — do not vouch it.** A security review of `912f7c1` found two defects; both fixed here, each test
seen failing against the 0.158.0 script:
- **`felhom-mkfs-guarded`**: 0.158.0 let a format through when a SIBLING part was mounted at /mnt/<name> AND under
/mnt/felhom-drives („Felhom's own drive"). That corroboration is a mount the agent can make itself, so a compromised agent
could unlock a host owner's disk — the exact case this root wrapper exists for. Removed: any mount of any node outside
/mnt/felhom-drives refuses again (now read from the kernel's mount table for every node; an unlistable disk refuses).
Consequence: the free part next to an in-use Felhom drive cannot be formatted — `CandidatePartition.Formattable` is false
for every part of a disk with a mounted node, and the controller does not offer it (a part with a filesystem can still be
USED). Pinned by configs/test_felhom_mkfs_guarded.py + partitions_test.go.
- **`felhom-internal-place`**: a failed `lvs` listing read as „nothing promised" (a too-large place) and a `blkid` error read
as „no signature" (the only gate before mkfs). Now: a failed listing, a listing without the pool, or any blkid exit other
than 0/2 REFUSES. Three new tests, all failing against 0.158.0.
- Released: tag `v0.158.1` (code `98f3cdd`), sha256 `20264df6…84b76a1`, bundle `c23c3d35…cb1c41b`. **Not vouched.**
## v0.158.0 — the internal data place; partitions as places; add_internal_place (2026-10-16)
Design: felhom.eu `architecture/16-storage-physical-identity.md` §5, §6, §7, §9 (operator rulings D3–D5; the D3 sizes
ruled 2026-10-16 after the space check, felhom.eu `audits/storage-identity-buildB-2026-10-15/SPACE-CHECK.md`).
- **The internal data place („Belső tárhely")** (`internal/localapi/internalplace.go`): when the record
`/var/lib/felhom-agent/internal-place.json` names a filesystem, the agent mounts it at `/mnt/belso` by fs-UUID like an
enrolled drive (startup + every 20 s tick) and records it `enrolled`; `GET /disks` reports it with kind `internal`.
It never takes `/mnt/belso` from a household drive, and eject/decommission/format of it are refused (device-mapper on
the system disk). Its identity is the system disk's, so build A never counts a copy to `sys_drive` as a second copy.
- **Root helper `configs/felhom-internal-place`** (new in the bundle): `plan|create <planned GiB>` — the ruled size rule
(reserve max(16 GiB, 10 %) ≤ 48 GiB, made only when ≥ 20 GiB, nothing promised beyond pool − reserve), the only
lvcreate/mkfs of a system-disk volume; formats only a `felhom-belso` with no signature. Run by the installer (1.33.0)
and by the new signed op.
- **Signed op `add_internal_place`** (old one-disk boxes, `16` §9): no parameters (the size is the rule's), destructive
class (operator key). Runs the helper with `create 0`, writes the record, mounts.
- **Partitions as places** (`internal/storage/partitions.go`): `GET /disks/candidates` lists each partition of a free
disk as empty / filesystem / in_use. A data-bearing part's wipe id binds disk id + part number + fs-UUID
(`byid:<…-partN>+fsuuid:<uuid>`); both must name the same device at format time.
- **`felhom-mkfs-guarded`** (REVERTED in v0.158.1): a SIBLING part that is Felhom's own drive no longer blocked formatting
the free part next to it. First tests of the wrapper: `configs/test_felhom_mkfs_guarded.py`.
- `configs/` changed (sudoers `FELHOM_INTERNALPLACE`, the helper, the wrapper, the bundle list) — **a config-bundle job
is needed** with the agent update. **Not vouched** (the operator's act).
- Released: tag `v0.158.0` (code `912f7c1`), sha256 `a3589d1d…2caba`, bundle `17066273…fb5c76`. **Superseded by v0.158.1 —
do not vouch** (its mkfs wrapper trusted a sibling claim the agent can fake; its helper failed open).
## v0.157.0 — every storage place knows its physical disk(s); POST /disks/same-disk (2026-10-15)
Design: felhom.eu `architecture/16-storage-physical-identity.md` §3, §4, §10 (operator rulings D1–D6, 2026-10-15). Spike:
felhom.eu `audits/storage-identity-2026-10-15/`.
- **Identity of a storage place** (`internal/storage/identity.go`): the physical disk(s) under it, from `/sys`, `/run/udev/data`
and `/proc/mounts` only — partition → disk, device-mapper / LVM thin → slaves, loop → backing file → its disk. Keys:
`eui:` (NVMe), `serial:<bus>:<model>:<serial>`, `wwn:` only without a serial and only with a vendor (OUI ≠ 000000).
`proven`, or `unknown` with one reason: `vm` (systemd-detect-virt), `network`, `usb_bridge` (a placeholder serial — the rule
is the design's, UNPROVEN on a real enclosure, R-1077), `weak_wwn`, `no_serial`. No block I/O.
- **Reported:** `GET /disks` rows and the hub report's `storage_targets` gain `kind`, `physical_disks`, `identity`,
`identity_reason` (additive; the golden and the hub's copy change together). A dir storage is judged only on its OWN mount — an
absent drive is "not computed", never the root disk.
- **`POST /disks/same-disk`** `{a, b}` (guest paths) → `{different, proven, reason, a, b}`. Maps `/mnt/felhom-drives/<name>` to
the host mount at exactly that path, and `/mnt/sys_drive`, `/var/lib/felhom`, `/var/lib/docker` to the guest's data volume's
storage. Anything else, or an absent drive → `unmapped`. A shared disk key is "same, proven" even when the disk is unknown.
Scoped like every `/disks` call.
- Tests: fixture `/sys` + udev trees for every spike case (one disk LVM thin, two disks, two partitions, a loop on a file, NVMe eui,
vendor-less WWN, a VM, a USB placeholder serial, a network share) and the same-disk call (equal, different, VM, NAS, absent
drive, scope); red-proofed (the LVM "diamond" bug was seen failing first; the VM, WWN, own-mount and sys_drive rules broken on purpose).
- `configs/` unchanged — no config-bundle job needed. **Not vouched** (the operator's act).
## Unreleased (2026-10-15, afternoon) — the shared rule file; no code change
- `.claude/rules/unprompted-work.md` §3 item 6: **a controller release needs no golden** — the floor carries it after the
three test boxes; agent releases still wait for the operator's vouch; the golden is weekly (operator ruling 2026-10-15
11:26, `09` §3 decision 250; the same morning's „bake first", decision 247, reversed). Same text in all five copies.
## Unreleased (2026-10-15) — the shared rule file; no code change
- `.claude/rules/unprompted-work.md` §3 item 6: **bake first** — a controller release is released only when its golden is
baked; bake → the operator vouches → the operator raises the floor; never ask for a floor for a version with no golden
(operator ruling 2026-10-15, `09` §3 decision 247, superseding decision 7). Same text in all five copies.
## v0.156.1 — R-1008 by process liveness, not the clock; 0.156.0 superseded — do not vouch it (2026-10-14)
Released by `scripts/release-agent.sh`: binary sha256 `83c7da27d457b63ba8a4cde5a8717b3e5f9db9f83a46e63f38902dc7471bffb9`, bundle
`5a678def792c70ec1cbbd198c14542161fac91d11a69bd0ce23dd86469e96ec3` (tag `v0.156.1` = `7ae3081`). Step bundle `0.156.1-step1`
sha256 `ac83ea41507616d4c1056cccfd35de1410ff01e0296f4e64ff925207a43a8490` (the 0.155.0 bundle with only `felhom-os-apply`
replaced), published as package version `0.156.1-step1`. **Not vouched** (the operator's act). Delivery as 0.156.0: per box
`agent_update 0.156.1` → `agent_config_update 0.156.1-step1` → `agent_config_update 0.156.1`.
- **0.156.0 is superseded — never vouch it.** A background security review of the pushed commit flagged „fail-open state
drift": 0.156.0 called a listed vzdump task dead when its wall-clock start was before the kernel's `btime`. `btime` is
derived from the wall clock and moves when the clock is corrected after the boot (an NTP step after a power cut), so a
backup started between the boot and the correction could look „from before the boot" — and the stale-lock pass would
clear a LIVE backup's lock.
- **Now:** `taskAlive` is Proxmox's own rule (`PVE::ProcFSTools::check_process_running`, read on demo-hp): the UPID's pid and
pstart against `/proc/<pid>/stat` — same start tick (field 22) and not a zombie. No clock is involved. Fails safe (running)
when the UPID does not parse, the stat cannot be read, or this agent cannot see root's processes at all (PID 1 not visible:
a `hidepid` /proc or `ProtectProc=` would hide every worker — demo-hp: `/proc` plain, `ProtectProc=default`).
- Tests: worker gone, pid reused, zombie, live worker, a „clock step" (wall-clock start absurdly early, worker alive → still
running), doubt, hidden /proc — each red-proofed; 19 guards in all (`power-cut-fixes-2026-10-14/redproof.txt`).
- R-1006 (the cleanup) is unchanged; its „before this boot" rule still reads `btime`, but a drifted `btime` cannot remove a
live partial: the Go side asks this liveness rule, and the root helper refuses while any vzdump runs or anything holds
the file open.
## v0.156.0 — after a power cut: the dead half-written backup is removed (R-1006); a backup task from before the boot is not „in flight" (R-1008) (2026-10-14)
Released by `scripts/release-agent.sh`: binary sha256 `8db2278838dcb733d5201fba84b75a829b9adebdf899c89dd7de0939ec10d51a`, bundle
`a21605217966ebc953d55a35157a9eb0243ca542c50dbd94fc9fa67257f27369` (tag `v0.156.0` = `a2f3aee`). Step bundle `0.156.0-step1` sha256
`8be2bb8f3d54bff8e7140cf99714664df31adecf13aa721597323fb2b9d43db5` (the 0.155.0 bundle `795f2ab4…` with only `felhom-os-apply`
replaced; `scripts/build-step-bundle.py`), published as package version `0.156.0-step1`. **Not vouched** (the operator's act).
**Delivery: the agent binary AND the config bundle, and the bundle ADDS a path** (`/usr/local/sbin/felhom-vzdump-partial-clean`)
— so per box: signed `agent_update 0.156.0` → `agent_config_update 0.156.0-step1` (the step bundle) → `agent_config_update
0.156.0`. Evidence: felhom.eu `documentation/audits/power-cut-fixes-2026-10-14/` (the spike C1–C4, the red-proofs).
- **R-1008 — one in-flight answer.** `staleLockController.BackupRunning` (the ONE implementation all four callers reach:
the start-up stale-lock pass, the guest-power watchdog, the household power-off guard, the controller supervisor) now
answers through `vzdumpInFlight`: a vzdump task for the vmid that started at or after the host's boot. Proxmox keeps a
dead task in `tasks?source=active` for ~30–60 s after a power cut (measured, C3) — the agent starts inside that window
and called it „genuinely in-flight". A task from before the boot is dead; logged once (vmid, UPID, start, boot). The
start time comes from the list or, if absent, the UPID; an unreadable boot time or start time FAILS SAFE (running), as
before. The boot time is `poweroff.ProcBtime` (no third /proc/stat parser).
- **R-1006 — the half-written backup is removed.** New `internal/vzpartial`: on a `dir` backup target it finds exactly
`vzdump-(lxc|qemu)-<vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp)` of THAT guest (never a finished archive, a log,
another guest, a symlink, a wrong kind), and — only when no vzdump of the guest is in flight (R-1008) and the entry is
dead (modified before this boot, or untouched for 120 s) — asks the root helper to remove it. Runs at agent start (after
the stale-lock pass, every owned guest × every backup tier) and before every backup's space preflight
(`BackupRunner.SetBeforeBackup`). Any doubt leaves the entry and logs why.
- **The root helper `configs/felhom-vzdump-partial-clean`** (config bundle; sudoers `FELHOM_VZPARTIAL`, one exact line: a
storage id, a vmid, one half-written name) re-checks everything as root: a `dir` storage from `/etc/pve/storage.cfg`; the
name of that vmid; not a symlink; directly in `<path>/dump`; **no vzdump on the host at all** (a job for several guests
carries no vmid in its task name); **no process holds the entry open** (C2: after a killed worker, `tar` and `zstd` kept
writing into the `.tar.dat` for 4 min 40 s, none named „vzdump"); dead (before this boot, or quiet 120 s).
- **Not changed, by evidence (C4):** the space preflight's sizing (R-1006 part 2) — on „No space left" vzdump removes its
own partial on both its paths (rsync and tar); only a power cut leaves one.
- **Tests:** `stalelock_inflight_test.go` (the helper, the production controller, three callers end to end, a structural
pin that every `BackupRunning` call goes through `s.staleLock` and there is one implementation), `vzpartial_test.go`
(removes; keeps in flight / unknown / non-dir / other names / recent / wrong kind / symlink / a helper refusal — each keep
with its positive control), `r685_space_test.go` (the cleanup runs BEFORE the preflight), the helper's own 21 Python
tests (`configs/test_felhom_vzdump_partial_clean.py`, run by the Go suite), 8 sudoers injection cases. **17 guards
red-proofed** by mutation (`power-cut-fixes-2026-10-14/redproof.txt`).
- **Fixed without a row:** `internal/localapi/guestbind.go` and `controllersupervisor_test.go` were not gofmt-clean on
`main` (an import order and a struct alignment) — formatted.
## v0.155.0 — the household can turn the machine off: POST /host/poweroff (R-998); the escrow reply race fixed (2026-10-13)
Released by `scripts/release-agent.sh`: binary sha256 `bb339e944eff4aa6d10ae09158e0943de2e5c3777f52bf993c796d57e46800c3`, bundle
`795f2ab41661e993e3fc4ce65df4f11ce94280d94871ab2e5cf1993ae3fbff7d` (tag `v0.155.0`). **Not vouched** (the operator's act).
**The bundle changes `felhom-agent.sudoers`** (no new path — no step bundle): delivery is a signed `agent_update` +
`agent_config_update`. Design: felhom.eu `documentation/architecture/14-host-power-off.md`.
- **`POST /host/poweroff`** (local API, `withGuest`-scoped like `/guest/reboot`) — the controller's „A gép kikapcsolása"
(controller v0.310.0). Refused (409, a stable `reason`) in the first 10 min after boot; a second time in one boot
(`internal/poweroff`, a file keyed by `/proc/stat` btime); while ANY other guest or VM runs on the host (read from
`/proc`: `lxc-start -n N`, `kvm -id N` — demo-hp's scratch guest and Tester 1 VM block it); while the heavy-operation
gate is held (backup, restore test, OS step, sweep, trim), a backup job, a controller swap, a drive format, a NAS verify
or an escrow ceremony runs, or Proxmox runs a vzdump of the guest. Every check fails closed.
- On acceptance: the record first; the heavy gate TAKEN and kept; a host report carrying `deliberate_off.requested_at`
accepted by the hub (bounded 60 s, `Loop.SetReportedHook`); then before each step the jobs and other guests are
checked again — a job waits (≤ 15 min), another guest cancels at once, a job still running after the wait cancels
(gate released, box stays on); `pct shutdown <vmid> --timeout 120`; `systemctl poweroff`.
- **Sudoers:** `FELHOM_POWEROFF` = `/usr/bin/systemctl poweroff`; `pct ^shutdown [0-9]+ --timeout [0-9]+$` joins
`FELHOM_GUESTHOOK`; nine injection cases (`--forceStop`, `poweroff --force`, `reboot`, `halt`, `kexec`…).
**New `TestEveryAliasIsGranted`:** an alias missing from the grant line passed the coverage check before.
- **Host report:** `deliberate_off` (omitempty; the cross-repo golden unchanged). Hub v0.149.0 holds its silence alarms
for 14 days on it.
- **Fixed without a row:** `go test -race` found a data race in the escrow ceremony start reply (it read `job.Phase`
while the job goroutine wrote it); the reply fields are copied first. Red before, green 3/3 after.
- Security review (three passes the same evening): other guests, a gate not held, fail-open unwired checks, jobs and
guests starting after acceptance — each fixed and red-proofed (`hostpoweroff_test.go`, 11 tests).
## v0.154.0 — no OS leg after a household press; the after-boot kernel report carries the real ring (R-899); no „wrong code" when older packages were not checked (R-304); the last backup survives a restart in the host report; the Secure Boot meta-package leaves the Proxmox lane (2026-10-09)
Released by `scripts/release-agent.sh`: binary sha256 `36a54ad20e896cd61d7b9944e2a521f4209c2b76940d8f1827af05b645463226`, bundle
`93487989f50d4a3039a05cec0f91ec478d62489a414705e49403e9b1340d20e0` (tag `v0.154.0` = `db25b46`).
**Delivery: the agent binary AND the config bundle** — the wrapper `felhom-os-apply` changed (2026-10-09).
- **The hub's backup evidence survives a restart (2026-10-09, found in the kernel-night read-back).** The host report's
`backups` list came only from memory, which is empty after an agent restart. On demo-felhom the night backup landed
02:40 UTC and the kernel step restarted the host at 02:44 — no report fell in between, two nights running — so the hub
alarmed „host tier: newest backup is 48h old" at 03:00 after a good backup. The report now adds R-894's saved newest
success per tier and guest (`backup-success-state.json`, ONE instance shared with the local API) where memory holds no
success at or after it. Only successes are saved, so a failure is never hidden. `hub.KnownBackupReporter`,
`BackupSuccessState.KnownBackupSuccesses`; tests `TestCollectBackups_SavedSuccessSurvivesARestart` (red-proved: the
restart case reported nothing), `TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen`, and
`TestR894_LastKnownBackupsIsWiredIntoTheDaemon` now also asserts the collector gets the same instance (red-proved).
- **`proxmox-secure-boot-support` is a boot-chain package (2026-10-09).** On demo-hp (Secure Boot on) its upgrade pulled
`shim-signed-common`; in the ring-0 Proxmox plan that refused the whole step (R6) and skipped the night's kernel step,
after the household had been told „tonight". It joins `HOST_SLOW_RE` with shim and GRUB (they stay out of every lane,
as before). `test_ring0_pending_pve_leaves_the_secure_boot_meta_out` (red-proved).
- **R-899 (operator ruling 2026-10-08, option A):** `POST /backup?trigger=manual` (a „Mentés most" press, sent by the
controller from its next release) runs no OS leg after it: the leg belongs to the night, after the night's own copy.
Before, a press ran the leg at once (in the day; on 2026-10-07 08:51 demo-hp skipped it only by the 20-hour rule). A
request without the parameter (an older controller, the scheduled path) behaves as before.
`internal/localapi/server.go`; `TestAfterPrimaryBackup` gained two sub-cases (red-proved: ignoring `trigger`, the leg
ran once after a press).
- **R-304 — „wrong code" only when every earlier package was tried.** `POST /escrow/recover-offsite-password` answered
400 („the recovery code did not open the sealed bundle") whenever the current package and the TRIED earlier packages
refused the code — even when the hub withheld earlier packages (rows with no key material, rows over its serve cap), a
served package was malformed, the 6-attempt cap stopped the loop, or the retained list could not be read at all. Now
those cases answer **424** with `older_unchecked` (the count, -1 = unknown) and a sentence that does not call the code
wrong. `escrow.ErrRetainedUnchecked` / `RetainedUncheckedError`; the retained fetcher's second value is now every
withheld package (unopenable + truncated + malformed). Tests `TestR304_*` (real age crypto; red-proved: without the
check, four cases returned the wrong-code error) and a 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus`.
An older controller maps the unknown 424 to its neutral „we do not know why" sentence.
- **The „ring 1" label after a boot:** in the first second after a reboot the agent has not fetched the hub's block, and
the after-boot kernel reports (`judging`, `good`, `revert`, `fell_back`…) said ring 1 on a ring-0 box (demo-felhom,
2026-10-08 night). Now they read the fetched block, else the block the daemon saved on disk before the reboot (R-866),
else ring 1 as before. A label only — the hub's approval reads its own ring list. `kernelReportRing`,
`TestKernelReportRing_BeforeFirstFetch` (red-proved: ring 1, want 0).
## v0.153.0 — ring 0 stages exactly the told kernel (R-898; `09` §3 decision 176) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `b204ebe6d65944f43b70dcfa4ae0dfb90da38c92e2ba6a38a1826e488eeb6630`, bundle
`6db216275eb0dd39188d93481a2045998a69e8cb7838ad908a58d65d9a6a9b57` (tag `v0.153.0` = `2d1e5d0`). Delivered (binary only — the
bundle files are unchanged from 0.152.0) to demo-hp, demo-felhom, Tester 1 on 2026-10-07 19:03.
**Delivery: the agent binary only** — no root file changed (the wrapper is unchanged; its tests gained two cases).
- `internal/osupdate/kernel.go`: ring 0's night kernel step stages EXACTLY the kernel the household was told about
(select `listed`, `KernelSet(kver)` = the series meta-package and the signed image at the kernel's own version) instead
of "whatever is pending tonight". Seen 2026-10-07: demo-felhom was told about 7.0.14-20 while its sources offered
7.0.14-22 by night — the old code staged `pending-kernel` and the wrapper refused it (R23), losing the night. A told
version that is no longer installable is refused by the wrapper before any change (R7) and the hub tells the household
again for the newer kernel (hub v0.143.1). Tests `TestKernel_Ring0ToldNightStagesThenReboots` (red-proved against the
old select), `TestKernelSet`; wrapper `test_ring0_listed_installs_the_told_kernel_not_the_newest`,
`test_ring0_told_kernel_gone_is_refused_before_any_change`.
## v0.152.0 — the kernel lane (R-836; `09` §3 decisions 164, 172; `11` §5.11) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `95ff42208e36ba49b6e2b09a97e81a6fa11562ecc8042f1ed18d378b8b1f88b1`,
config bundle sha256 `f0c2cec374b711b3c131c955012b33fd0ad495337d049b7a743c3eef9e85c20b` (tag `v0.152.0` = `d03ab7f`).
Step bundle `0.152.0-step1` sha256 `0b71d32b054cf3b7ade0234ffcbb0df159901f542cde540adaee411db466f48e` (the 0.151.0 bundle
with only `felhom-os-apply` replaced; `scripts/build-step-bundle.py`), published as package version `0.152.0-step1`.
**Delivery: agent binary, then the STEP bundle `0.152.0-step1`, then the bundle `0.152.0`** — the bundle ADDS two paths
(the GRUB generators), and an installed `felhom-os-apply` refuses a path its own table lacks (R16, R-880).
A new kernel boots ONCE; if it crashes the box comes back on the old kernel by itself; it becomes the default only after
a healthy boot; a booted-but-unhealthy kernel is reverted ONCE by the agent with no person. Built on the spike's
candidate 2 (`audits/kernel-spike-2026-10-07/`), with option C on the one-shot entry.
- `configs/felhom-grub-oneshot.sh` → `/etc/grub.d/01_felhom_oneshot` (bundle): reads `felhom_next` from a GRUB env block
on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and sets the default to that kernel's
one-shot entry only when the name is an installed kernel. No vfat ESP → prints nothing.
- `configs/felhom-grub-oneshot-entries.sh` → `/etc/grub.d/42_felhom_oneshot` (bundle): one entry per installed kernel,
id `felhom-oneshot-<ver>`, the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`
(option C). Sorted after `10_linux`: never entry 0, never the default.
- `configs/felhom-os-apply`: layer `kernel` (lane slow; an appliance; authority = a signed `os_kernel_step` or the
root-owned ring-0 mark). Modes: `apply` STAGES (select `pending-kernel` or a signed `listed` set; `expect_kver` = the
kernel the household was told about): pins the GRUB default to the RUNNING kernel in
`/etc/default/grub.d/zz-felhom-kernel-default.cfg` and proves it from grub.cfg, installs, proves the default did not
move and the one-shot entry exists, writes the flag; never reboots. `kernel-reboot` (a staged step only),
`kernel-boot` (judging | fell_back | self_reverted | revert_failed), `kernel-good` (the new kernel becomes the
default, proved), `kernel-revert` (ONE per step; refused when the default is not the old kernel), `kernel-cancel`,
`kernel-status`. New refusals: R20 (the box cannot do a one-shot: not UEFI, no vfat ESP, a separate /boot, GRUB
without fat/loadenv, the generators missing, a hand pin), R21 (the crash guard tripped or an unclean boot in its
window), R22 (the phase does not allow the mode; never two steps within 20 h), R23 (not exactly one newer kernel, or
not the one signed / told). State `/var/lib/felhom-kernel/state.json`. Facts carry `kernel_lane`; the next-boot
kernel reads the flag and the grub.cfg default. Tests: `KernelLane` (27), red-proof
`felhom.eu/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`.
- `configs/felhom-crash-guard` unchanged; `KernelStepCannotLeaveTheBoxOff` (3 tests) pins that a step's planned reboot,
one crash and one self-revert add ONE unclean boot (a panic before userspace adds none), so the box cannot stay off.
- `internal/osupdate/kernel.go`: the night leg ends with the kernel step — after a healthy host step (and a healthy
Proxmox step when one ran), trigger `night` only, on a night the hub's `os_update.kernel` block marks `tonight` (the
household was mailed the day before — no mail, no step). Ring 0 stages + reboots; ring 1 reboots only a kernel a
signed `os_kernel_step` staged (`KernelStepExecutor`: stage only, under the heavy-op gate). The hub hears `staged`
BEFORE the reboot. At every start `KernelAfterBoot`: on the new kernel it JUDGES the boot — `KernelVerdict` = the
host health rule (`11` §8.2) AND the box reached the hub (the `judging` report itself) — for 20 minutes (measured:
everything healthy 68 s after the reboot on demo-felhom, 272 s on demo-hp; under the hub's 45-minute `host_stale` (`alerting.stale_threshold`)).
Healthy → `kernel-good`, outcome `applied`; not healthy → outcome `health_failed`, then ONE `kernel-revert`.
Tests: `TestKernel*` (13); red-proofs in the same file.
- `internal/hub`: `WireOSUpdate.Kernel` {kver, tonight, notified_at}. `internal/reconcile`: `os_kernel_step` is
destructive-class. `cmd/felhom-opsign`: the op is listed.
## v0.151.0 — the agent can no longer hand the guest any image; the Proxmox package lane; the other-key archives reported; the DR directive retired (R-861, R-812 A, R-366, R-105; `09` §3 163, 165, 168, 169) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `0464354f2cdf452a7c5d2a74d9191fe91415fcfa244154480d26d5b30e10b194`
config bundle sha256 `bacd1d175a9392bc1755341d01a106abbd72aba48ab575e7a32dd819d8f5da4c` (tag `v0.151.0` = `dd7cdc0`).
**Deliver the binary FIRST, then the bundle:** the bundle's sudoers removes the `tee` grant the 0.150.0 binary still uses.
No path added (26 → 26), so no step bundle.
### Part of v0.151.0 — the agent can no longer hand the guest any image; felhom-op's pct lines are exact (R-861 (a) A1, (b) B2; `09` §3 decision 165)
**Delivery order: agent binary FIRST, then the config bundle.** The new sudoers drops the agent's in-guest `tee`
grant; an older binary still calls `tee`, so a bundle that lands before the binary would stop managed controller
updates (and the old binary's capability probe would read `controllerswap-write` degraded). No bundle path is added
(`felhom-priv-apply` and `/etc/sudoers.d/felhom-op` are already bundle files), so no step bundle.
- `configs/felhom-priv-apply`: new verb `controller-image <vmid>` — reads the ref on stdin (≤ 256 bytes, ASCII, one
optional trailing newline), requires `^gitea\.dooplex\.hu/admin/felhom-controller:[0-9]+\.[0-9]+\.[0-9]+$` (the agent's
own `controllerImageRe`), then runs `pct exec <vmid> -- tee /etc/felhom-controller-image` AS ROOT; refusal rule `I1`
(rc 3), a bad vmid `A1` (rc 2); listed in `--self-check`.
- `configs/felhom-agent.sudoers` `FELHOM_CONTROLLERSWAP`: `pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$`
REMOVED; `/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$` added.
- `internal/localapi`: `GuestExecutor.GuestExecStdin` replaced by `WriteControllerImage`; `GuestBinder.WriteControllerImage`
pipes `ref\n` to the verb through the fenced runner; the swap's `writeImage` calls it. Capability `controllerswap-write`
now probes the verb.
- `configs/felhom-op.sudoers` (B2, hygiene): `pct start|stop|unlock [0-9]*` → `pct ^start [0-9]+$` etc. (the glob's `*`
matched spaces: `pct stop 9201 --skiplock 1` passed).
- Tests: `ControllerImage` (5, `configs/test_felhom_priv_apply.py`), `TestSudoersRefusesTheR861Injections` (+3 lines),
`TestSudoersAllowsTheControllerImageVerb`, `TestFelhomOpSudoersPctIsExact`, `TestR861_WriteControllerImageUsesTheRootVerb`,
`TestControllerSwap_WriteViaRootVerb_NoShell`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/C/`.
- `README.md`: the controller-swap paragraph described the removed `tee` path — corrected.
### Part of v0.151.0 — the Proxmox package lane (R-812 option A, `09` §3 decision 163)
**MinAgent impact: none** (a new layer; an older hub ignores the pve report). **The bundle carries the new
`felhom-os-apply` — deliver it with the binary** (signed `agent_update`, then signed `agent_config_update`).
- `configs/felhom-os-apply`: new layer `pve`, lane `slow` only — the host's Proxmox USERSPACE packages: origin `Proxmox Debian Repository` only (R2), never a kernel / boot / firmware / microcode name (R14, `HOST_SLOW_RE` — the kernel is R-836's lane), no removal (R4), no undo (R5), a new package only from `PVE_NEW_ALLOW` (`proxmox-firewall-data`, measured on demo-felhom; R6 otherwise), an appliance only (R12), authority = a signed `os_pve_step` or the root-owned ring-0 mark (R3). Select `pending-pve` (ring 0): installed Proxmox-origin packages with a pending upgrade. The report carries `pve_manager` (pveversion after the step).
- `internal/pvegate` (new): the agent's own writes to /etc/pve wait while a pve step runs (pmxcfs restarts); the step waits for writes in flight (bounded, 2 min — then it fails and does not run). Wired at `proxmox.Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile).
- `internal/osupdate`: `LayerPVE`; the night leg runs the pve step in ring 0 after a healthy host step (an appliance; ring 1 never in the night leg); `PVEHealthVerdict` = the host rule + every running container keeps its id + pveversion reads the installed pve-manager; the pve report carries Proxmox userspace only (the hub's candidate set). `PVEStepExecutor` (signed `os_pve_step`, ring 1, under the heavy-op gate and the /etc/pve gate); `reconcile.ClassOSPVEStep` (destructive-class); `felhom-opsign -op os_pve_step` (params by `-params`).
- Tests: wrapper `PVELane` (17; red first — the `pve-manager` plan was refused R12 on the old code), `pvegate` (5), `TestPVEGate_*` + `TestWritesEtcPVE`, `TestPVE_*`, `TestPVEHealthVerdict`, `TestPVEStepExecutor_*`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/B/`.
### Part of v0.151.0
- R-366 slice 2 (`09` §3 decision 168): the restore-test pick records, per tier, the archives it skipped as written with another key (count, oldest, newest — no key material) in a `ForeignKeyLedger`; the host report carries it as `foreign_key_archives.tiers` (the stanza absent until a tier was evaluated since start, `tiers: []` when none — no null on the wire, the report contract forbids it). The hub turns a change into one operator line. Tests `TestR366_PickRecordsArchivesWrittenWithAnotherKey`, `TestR366_EvaluatedWithNoneIsAnEmptyList` (red-proved, `felhom.eu/documentation/audits/day-2026-10-07/E/`).
- R-105 option A (`09` §3 decision 169): the `--selftest=escrow-create -directive <file>` flag and the escrow upload's `directive` field are removed — nothing read the directive; the DR path reads the recipe, tenantsync and the escrow blob. The hub ignores a `directive` from an older agent.
## v0.150.0 — the Docker step proves the engine reports a memory kill; after a restart the agent remembers the last backup per tier; three more SMART counters on the wire (R-528, R-894, R-330; `09` §3 decisions 157, 161) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `a23d1c9085bc7fd4fc48fe0327f6504aa83e6331510fb4a3d23e042dddb26f9c`
config bundle sha256 `88456b386d9b1027bd22861cac8c23df004bf9fd9f67644d6595bfca8c94498e` (tag `v0.150.0` = `3a72a48`).
**The bundle carries the new `felhom-os-apply` (the memory-kill check) — deliver it with the binary:** signed
`agent_update`, then signed `agent_config_update`. No path added (26 → 26), so no step bundle.
### Part of v0.150.0 (2026-10-06 night, later) — after a restart the agent remembers the last backup per tier (R-894); three more SMART counters on the wire (R-330)
Ships with the memory-kill check below as v0.150.0, AFTER the 2026-10-07 night read-back. Nothing delivered tonight.
- **The defect (measured 2026-10-05 on demo-hp):** the agent restarted at 04:57; at 06:25 the off-site storage answered *Can't connect*; the per-tier backup record is in memory only, so the due-check fell back to an EMPTY record and the 7-day tier (last copy 4 days old) read DUE; the controller asked and vzdump failed.
- New `internal/backup/backup_state.go` `BackupSuccessState`: the newest SUCCESSFUL backup per tier and guest, on disk (`<oob state dir>/backup-success-state.json`, atomic tmp+rename, 0600). Only successes are written; a corrupt file reads as nothing known.
- `internal/localapi` `handleBackupDue`: when the tier's storage CANNOT be read, the saved copy stands in for the in-memory record. A fresh copy → not due („… (storage unreadable — age from the last success saved on disk)"); a copy older than the cadence → DUE; no copy → the old answer (DUE, age unknown). A storage that answers stays the ground truth: an archive absent there is due even when the file remembers one.
- Wired in `buildLocalAPIServer` (`LastKnownBackups`); the local API's backup job saves each success.
- Tests: `TestBackupDue_R894_*` (restart = a new server and a new state from the same file; fresh / old / none / storage answers / failed backup not saved), `TestBackupSuccessState_*`, `TestR894_LastKnownBackupsIsWiredIntoTheDaemon` (AST). Four red-proofs observed (`felhom.eu/documentation/audits/night-burndown-2026-10-06/s4/`).
- **R-330 (disk health Phase 2, the wire only):** the SMART summary carries three more SATA raw counters — `reported_uncorrect` (187), `command_timeout` (188, carried as the vendor reports it; some pack several counters), `udma_crc_errors` (199). Pointer + omitempty: an attribute the drive does not report is OMITTED (unknown), never 0. No verdict reads them yet. Tests `TestParseSMART_R330_*` (two red-proofs, `felhom.eu/documentation/audits/night-burndown-2026-10-06/r330/`).
### Part of v0.150.0 (2026-10-06 night) — the Docker step proves the engine reports a memory kill (`09` §3 decision 157, R-528)
To be released as v0.150.0 with its config bundle AFTER the 2026-10-07 night read-back (the night of 2026-10-06 runs v0.149.0 on purpose).
- `configs/felhom-os-apply`: after a docker-layer APPLY (after `health_after`) the wrapper runs `oom_check()`: a throwaway container from the image the running controller uses (`--pull never`, `--network none`, no volume, label `felhom.oomcheck=1`, 64 MB cap) asks for one 200 MB block; „pass" only when `OOMKilled=true` AND the `oom` event; it waits 2 s and reads the events window to the guest's epoch + 1 (measured: a window closed in the same second missed the event); the container is always removed. Reported as `oom_check`; it never changes the step's outcome or health. A wrapper-only mode `oom-check` runs the check alone (no apt, no engine change), by hand as root.
- `internal/osupdate`: `WrapperReport` and `Report` carry `oom_check` verbatim, on the normal pass and on the kept-copy path (R-868).
- `configs/test_felhom_os_apply.py`: its `unittest.main()` sat in the middle of the file, so 11 tests (UnsentReport, SaveReportOnDisk, AgentDiesMidPass, CrashLeftTheJournal) never ran — moved to the end; all pass.
- Tests: the OOMCheck class (pass, OOMKilled=false, no event, unreadable image, removal on an inspect error, not on other layers or in health mode, the events window after the settle wait, mode oom-check alone and its refusals); TestDocker_OOMCheckReachesTheHubUnchanged, TestR868_KeptCopyCarriesTheOOMCheck. 12 red-proofs in `felhom.eu/documentation/audits/readback-2026-10-07/F/`.
### Part of v0.150.0 (2026-10-06 evening) — the shared rule file (`09` §3 decision 152); no code change
- `.claude/rules/unprompted-work.md` added, byte-identical to the copies in felhom.eu, felhom-controller, app-catalog-felhom.eu and the workspace root (checked with `diff` against the controller's copy and one md5 across all five). Its copies line names five copies.
### Part of v0.150.0 (2026-10-06 afternoon) — instruction files kept true (`09` §3 decision 150); no code change
- `CLAUDE.md` „Gates — ONE entry point": the runner runs every gate in its `GATES` table (five: three shared, `published`, `release-complete`); `--fast` skips `published` (network). It said two gates and „all of them".
- `CLAUDE.md`: the decoy gate and its audit are named with their `felhom.eu/` prefix (they do not exist in this repo).
- `.claude/rules/health-checks.md` (comment): the health-check rule's copies live in felhom.eu `hub.md` and the controller's `gates.md`; it named felhom.eu `CLAUDE.md` „Code quality rules", which holds no such rule.
## v0.149.0 — a weekly disk trim of each customer guest, the crash-boot fact for the controller, the phantom WARN names its runbook (R-444, R-856, R-99; operator rulings `09` §3 139, 143, 140) (2026-10-06)
Released by `scripts/release-agent.sh`: binary sha256 `6bcae9c2eb5d97e8285316583870059835793893299e291891a53a4ce505585f`
config bundle sha256 `e182c82dcf4a67faa3bcb74dbe4ffa7b06e0b27dc8451cb7574d6339ce91ad66` (tag `v0.149.0` = `f277e61`).
**The bundle carries the new sudoers rule for the trim (`FELHOM_FSTRIM`) — deliver it with the binary:** signed
`agent_update`, then signed `agent_config_update`.
- R-856 (`09` §3 decision 143): new local-API route `GET /host/crash-guard` — passes the host crash guard's last-boot record (present, last_boot_at, last_boot_unclean, tripped) from /var/lib/felhom-crash-guard/state.json to the controller, which waits ~15 min with app mails after a crash boot. Read-only, no Proxmox call, guest-token authed; a missing/unreadable/garbled file answers 200 present:false (never an error page). An older agent answers 404, which the controller reads as unknown (normal 90 s grace) — no controller MinAgent raise needed. - R-856 (`09` §3 decision 143): new local-API route `GET /host/crash-guard` — passes the host crash guard's last-boot record (present, last_boot_at, last_boot_unclean, tripped) from /var/lib/felhom-crash-guard/state.json to the controller, which waits ~15 min with app mails after a crash boot. Read-only, no Proxmox call, guest-token authed; a missing/unreadable/garbled file answers 200 present:false (never an error page). An older agent answers 404, which the controller reads as unknown (normal 90 s grace) — no controller MinAgent raise needed.
- R-444 (`09` §3 decision 139): weekly guest disk trim. New sudoers alias FELHOM_FSTRIM with ONE exact rule `/usr/sbin/pct ^fstrim [0-9]+$` (rides the signed config bundle; decoys pinned by TestSudoersFstrimRuleIsExact) and capability guest-fstrim (non-critical). New internal/fstrim job: each owned RUNNING guest gets `pct fstrim <vmid>` once a week - due Wednesday from 10:00 host-local, starts only 10:00-20:59 (never the 01:00-06:59 night), holds the one-heavy-op gate so it never runs beside a backup or restore-test (busy -> deferred to the next hourly tick; a box that was off catches up at its next daytime hour); a failed trim WARNs and is retried at most 3 times that week; bytes parsed from `pct fstrim`'s "(N bytes) trimmed" lines; positive log `fstrim: guest N trimmed X GiB in Ys`; last result per guest persisted in <state_dir>/guest-disk-trim.json and reported as the new omitempty host-report stanza `guest_disk_trim`. Opt-out: agent.json "disk_trim": {"disable": true}. - R-444 (`09` §3 decision 139): weekly guest disk trim. New sudoers alias FELHOM_FSTRIM with ONE exact rule `/usr/sbin/pct ^fstrim [0-9]+$` (rides the signed config bundle; decoys pinned by TestSudoersFstrimRuleIsExact) and capability guest-fstrim (non-critical). New internal/fstrim job: each owned RUNNING guest gets `pct fstrim <vmid>` once a week - due Wednesday from 10:00 host-local, starts only 10:00-20:59 (never the 01:00-06:59 night), holds the one-heavy-op gate so it never runs beside a backup or restore-test (busy -> deferred to the next hourly tick; a box that was off catches up at its next daytime hour); a failed trim WARNs and is retried at most 3 times that week; bytes parsed from `pct fstrim`'s "(N bytes) trimmed" lines; positive log `fstrim: guest N trimmed X GiB in Ys`; last result per guest persisted in <state_dir>/guest-disk-trim.json and reported as the new omitempty host-report stanza `guest_disk_trim`. Opt-out: agent.json "disk_trim": {"disable": true}.
+8 -7
View File
@@ -52,11 +52,12 @@ This is in the core because breaching it is how this component stops being audit
## Gates — ONE entry point ## Gates — ONE entry point
**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs this **Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs every
repo's gates — `reuse_refs_check` and `instructions_gate`, both the **shared** copies in gate in its `GATES` table (that table is the list); the shared ones — `reuse_refs_check`,
`felhom.eu/scripts/`, never copied into this repo (a copy recreates the drift they detect; an absent `instructions_gate`, `observations_gate` — are the copies in `felhom.eu/scripts/`, never copied into
sibling clone FAILS). `--fast` selects the gates touching no network and no container runtime; today this repo (a copy recreates the drift they detect; an absent sibling clone FAILS). `--fast` selects the
that is all of them. **A missing gate is a FAILURE, never a skip.** gates touching no network and no container runtime, and skips `published` (network) and `gate-decoys` (the decoy
suite, ~20 s; R-1116), naming them — CI runs the full set. **A missing gate is a FAILURE, never a skip.**
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is **The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
**per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS **per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
@@ -104,8 +105,8 @@ the mechanism are exempt.
**A gate ships with a decoy test that has been seen to fail (R-421).** A decoy is the LABEL without **A gate ships with a decoy test that has been seen to fail (R-421).** A decoy is the LABEL without
the FACT — a directory with the right name and no bake log, a note whose prose mentions the marker it the FACT — a directory with the right name and no bake log, a note whose prose mentions the marker it
lacks. `scripts/decoy_coverage_gate.py` refuses a new gate that has neither a decoy nor a named lacks. `felhom.eu/scripts/decoy_coverage_gate.py` (run by felhom.eu's `repo_gates.py`, for all four repos) refuses a new gate that has neither a decoy nor a named
exemption carrying its row. The four shapes, the 2026-09-01 sweep that fooled 16 of 29 gates, and the exemption carrying its row. The four shapes, the 2026-09-01 sweep that fooled 16 of 29 gates, and the
decoys withdrawn as illegitimate: `documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and decoys withdrawn as illegitimate: `felhom.eu/documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and
`felhom-controller/.claude/rules/gates.md`. **Scope is a fact too** — prefer `os.walk` over `felhom-controller/.claude/rules/gates.md`. **Scope is a fact too** — prefer `os.walk` over
`os.listdir`, and a glob over a hand-maintained list. `os.listdir`, and a glob over a hand-maintained list.
+71
View File
@@ -1,5 +1,76 @@
# CONTEXT — felhom-agent working state # CONTEXT — felhom-agent working state
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
> on demo-hp (felhom.eu R-1250, read, not forced).
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
> put back. Release + the operator's vouch: Monday's list (felhom.eu STATUS).
> **2026-10-24 (attended release run) — v0.164.0: the weekly upload line test (R-1247; operator ruling 07:26).**
> `internal/linetest`: the agent's own iperf3 client (operator's pick — nothing installed on hosts) to ep0's tunnel-only
> server `10.77.0.1:5201`, Tuesday 10:00-15:59, after 10 quiet minutes, under the heavy-op gate, best of 3; `line_test`
> stanza + `GET /host/line-test` (the controller's qBittorrent limit, D41 = 50 %). demo-hp measured 90.7 Mbit/s. Release
> needs the operator's vouch.
> **2026-10-22/23 night — D30 on `main` (`ae69ad4`), unreleased; the D4 patches v2 ready.** Operator ruling D30
> (2026-10-22): a handshake stamped in the future for 15 min → a new WG key pair (`wgtunnel/rekey.go`; old key kept as
> `wg/private.key.prev`), registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the
> key on ep0). Live on a test box (felhom.eu `audits/night-2026-10-23/P2-D30/`): with ep0 free to start a handshake the
> tunnel heals by itself (no re-key, correct); with ep0's starts dropped it stays dead and the agent re-keyed at 15 min.
> The interface switches at the next desired-state fetch. Gap: a reboot forgets the future stamp (R-1223). The D4 bundle
> patch v2 (`P3-review/felhom-agent-bundle-D4-v2.patch`): `try-reload-or-restart`, an `sshd -T` read-back reported as
> `sshd_dropin_not_in_force`, a reload in undo. Next agent release = D30 + D4 v2 + R-1204's field, the operator vouches.
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
> `ff4e6dc5…99d3b5`, bundle `485f36c9…12c47`. R-1083 left out on purpose: an agent key for 9202 would make its
> controller book whole-guest backups through the agent, one tier writing to ep0 — the R-1083 row says what is needed.
> **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978,
> R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with
> `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn
> positions; only those two words are held, 30-min window; a claim without it keeps the R-978 order). Vouch: version
> 0.160.0, sha256 `7ebb547f…dfca1`, bundle `6b060901…e202f`. After it, unreleased: `4545819` (two texts). felhom.eu
> `REPORT-day-2026-10-19.md`.
> **2026-10-18 (night) — UNRELEASED on main: R-1093, R-978, R-973** (release + vouch: the operator's). R-1093 an empty
> partition is blank (lsblk's PTTYPE on a partition is the parent's; set aside only on a clean blkid exit 0 with a
> matching `PART_ENTRY_SCHEME`; an unparsable lsblk is unprobed). R-978 the escrow package reaches the hub only at the
> claim (ceremony output v2, `PendingUpload`; `EscrowCeremonyConfig.Upload`). R-973 the `POST /backup` cadence cap.
> **Decision (decided by CC unattended — operator may reverse): the cap's numbers.** Q: how many whole-guest runs may
> the guest ask for? Options: (a) 2 per 24 h on the daily tier — keeps one local point older than a day, but refuses
> the NIGHT after a press (R-899's night can fall inside the same 24 h); (b) 3 per 24 h + 1 per 6 days on the weekly
> tier — the schedule never refused, a second press in a day refused, the local tier still replaceable in a day;
> (c) none — status quo. Chose (b): the off-site tier is the one `07` §8 row 10 relies on and it is fully protected
> for ~6 days; the local half's real fix is a time-bucketed prune (keep-daily), which is a retention change for the
> operator. No sudoers or bundle change in any of the three.
> **2026-10-17 — v0.159.1 RELEASED (0.159.0 superseded, do not vouch).** Go 1.27.2; R-1124 a decommissioned drive that is
> gone is forgotten (`ForgetRetiredDrives`, root verb `unit-forget`); R-1012 the read-only `felhom-thin-check` names an
> orphan thin device in a failed backup's text; the signed decommission prunes the guest-bind record. Bundle adds a path →
> step bundle `0.159.0-step1`. Proven on demo-hp; delivered by signed jobs on the operator's yes. Vouch: the operator's.
> **2026-10-16 — v0.158.1 RELEASED and VOUCHED (storage identity build B; 0.158.0 superseded, do not vouch).** The internal
> data place (root helper `felhom-internal-place`, the record in the state dir, mounted at /mnt/belso like a drive), the
> signed `add_internal_place`, partitions in `/disks/candidates` (`formattable` only while nothing on the disk is mounted),
> the partition wipe id `byid:<…-partN>+fsuuid:<uuid>`, the strict mkfs wrapper. Delivered through step bundle
> `0.158.1-step1` (a bundle that adds a path, R16). Next: R-1093 (an empty part's probe reads the parent's partition table).
> **2026-10-15 (night) — v0.157.0 RELEASED, NOT VOUCHED (storage identity build A, felhom.eu `16` §13).** Every storage
> place carries `kind`, `physical_disks`, `identity` (+ reason) in `GET /disks` and the hub report; `POST /disks/same-disk`
> answers the controller (v0.317.0, MinAgent 0.157.0). Computed on every read, never cached. Delivered by signed jobs to
> Tester 1, demo-hp, N100 only. Next: the operator vouches 0.157.0 → the global floor 0.317.0 / MinAgent 0.157.0. Build B
> (partition candidates, the internal place) waits for a task file (R-1079).
> **2026-10-08 (day) — UNRELEASED on main, ships with tomorrow's release (decision 178).** R-899: `POST /backup?trigger=manual`
> runs no OS leg (`localapi/server.go`); after-boot kernel reports carry the saved block's ring (`kernelReportRing`). R-304:
> `escrow.ErrRetainedUnchecked` → HTTP 424 `older_unchecked` when not every earlier package was tried (withheld, caps,
> malformed, unreadable list). Binary-only delivery; no wrapper or root file changed.
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode > **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) + > `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`; > `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
+8 -7
View File
@@ -44,13 +44,14 @@ unnoticed until a user hit them. `internal/capability` makes that loud:
(`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded (`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
— the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.) — the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap **Controller-swap under non-root (v0.45.0; the write since R-861 (a) A1).** The agent-owned controller image swap
(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on (`internal/localapi/controllerswap.go`) no longer shells out. The write goes on **stdin** to the ROOT verb
**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin` → `felhom-priv-apply controller-image <vmid>` (`GuestBinder.WriteControllerImage` → `Runner.RunStdin`, the same fenced
`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow `sudo -n` runner), which re-checks the ref against our registry + repository + an x.y.z tag and writes
grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee` `/etc/felhom-controller-image` inside the guest itself; the agent has no in-guest `tee` grant any more (before, a
target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a compromised agent could feed any image — sudo cannot see stdin). Its grants live in the `FELHOM_CONTROLLERSWAP` sudoers
dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted. alias (read-only or fixed-target) and in the capability manifest (Critical), so a dropped grant is a build failure + a
live degraded signal. No general `pct exec` is granted.
## The `storage` package — observe + watchdog (slice 5) ## The `storage` package — observe + watchdog (slice 5)
+17 -12
View File
@@ -1,16 +1,21 @@
# REPORT — agent v0.148.0 (2026-10-06, the burn-down night) # REPORT — R-1285: the BYO SSH clean-up in the same update — committed, NOT released (2026-10-26 afternoon)
Full session report: `felhom.eu/REPORT-burndown3-2026-10-06.md`. Baseline `208fac8` (v0.147.0). **Plain words first.** A BYO box coming from 0.164.0 kept our SSH file one release too long: in the normal update the
OLD agent installs the config bundle, so the new agent's clean-up pass never ran on it. Now the new agent runs that pass
once, right after its self-update commits. Not released today (operator: „not today"); its live proof needs a BYO box on
0.164.0, which rides the next agent release.
**Released:** v0.148.0 (tag = `861d32a`; binary sha256 `3e68a087…`, bundle `a6fa4f58…`, verified by download). R-349 ## The fix (`b1bc63d`, CI 2363 ✓)
(the host report carries `agent_sha256` — the hub's host pages read „matches vouched” for all three boxes) and R-25's
agent half (the format answer carries the verified `fs_uuid`). **Delivered** by signed `agent_update` (all three on
0.148.0 by 22:50Z) and `agent_config_update` (root files 0.148.0 by 22:58Z) to demo-hp, demo-felhom, Tester 1. Tester 2:
nothing sent (off).
**On main, unreleased:** R-426 decoys (new `scripts/test_gate_decoys.py`: published against a fake Gitea, - `internal/selfupdate/commit.go`: `Manager.SetOnCommitted` — a hook run once after „update committed".
release-complete, the shared reuse-refs/instructions/observations). - `internal/osupdate/bundle.go`: `ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's `bundle-policy` mode, host
layer, no envelope); `cmd/felhom-agent/main.go` `wireBundlePolicyAfterCommit` hooks it on.
- The smaller of the two fixes: agent-only — holding the bundle job until the commit would need new hub queue logic.
**Said plainly:** `go test ./internal/osupdate` was red on DooPlex from 21:25 to 01:55 — `configs/test_felhom_config_bundle.py` ## Proof
read the installer 1.32.0 KEPT names as installer-written files. The v0.148.0 binary was released inside that window;
its code is unaffected (a test-only sibling coupling; CI has no Go). Fixed in `b2b82ae`. - `cmd/felhom-agent/r1285_policy_after_commit_test.go`, in the real order: a pending marker for the running version →
`commit`, then exactly one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
removed → commit only, no pass — the demo-hp symptom of 2026-10-26).
- **Not live:** only a box coming FROM 0.164.0 can show it (0.165.0's wrapper already applies D43 at plan time), which
needs a signed downgrade chain on a BYO box.
+17 -4
View File
@@ -15,13 +15,14 @@
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called | | `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) | | `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle | | ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key` | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line | | `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| unit-forget <name> (v0.159.0, R-1124) \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key \| controller-image <vmid>` (the last reads the ref on stdin, R-861 (a) A1) | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control | | `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) | | `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself | | `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
| `osupdate.Leg.SendUnsent` / `lockPass` (v0.144.0, R-868) | internal/osupdate/unsent.go | `(ctx) int` | an OS-pass report the agent never sent (killed mid-pass): the wrapper keeps `report-<run>-<layer>-apply.json` beside the plan; the agent deletes it once the hub has it | any new caller that runs an apply pass must hold `lockPass` (flock, across processes) — the sender must never take a running pass's copy | | `osupdate.Leg.SendUnsent` / `lockPass` (v0.144.0, R-868) | internal/osupdate/unsent.go | `(ctx) int` | an OS-pass report the agent never sent (killed mid-pass): the wrapper keeps `report-<run>-<layer>-apply.json` beside the plan; the agent deletes it once the hub has it | any new caller that runs an apply pass must hold `lockPass` (flock, across processes) — the sender must never take a running pass's copy |
| `osupdate.LoadSavedBlock` (v0.144.0, R-866) | internal/osupdate/leg.go | `(planDir) (block, savedAt, ok)` | the hub's newest os_update block as the daemon last received it (`os-update-block.json`) | the debug pass uses it ONLY when the hub cannot be reached, and says so in its header; no saved block → no pass | | `osupdate.LoadSavedBlock` (v0.144.0, R-866) | internal/osupdate/leg.go | `(planDir) (block, savedAt, ok)` | the hub's newest os_update block as the daemon last received it (`os-update-block.json`) | the debug pass uses it ONLY when the hub cannot be reached, and says so in its header; no saved block → no pass |
| `dpkg_state()` / `DPKG_STATE_SCRIPT` (v0.145.0, R-876) | configs/felhom-os-apply | `audit, journal = self.dpkg_state()` | dpkg's state in ONE call: `--audit` AND the update journal | never gate a repair on `--audit` alone — a crash leaves only the journal (measured); keep it one call (R-845) | | `dpkg_state()` / `DPKG_STATE_SCRIPT` (v0.145.0, R-876) | configs/felhom-os-apply | `audit, journal = self.dpkg_state()` | dpkg's state in ONE call: `--audit` AND the update journal | never gate a repair on `--audit` alone — a crash leaves only the journal (measured); keep it one call (R-845) |
| `thincheck.Checker.Check` + `Describe` (v0.159.0, R-1012) | internal/thincheck/thincheck.go | `Check(ctx, vmid) []Finding` | name an ORPHAN thin device (pool holds it, LVM does not) in a whole-guest backup failure — `BackupRunner.SetThinCheck` | READ-ONLY by design: the root helper `felhom-thin-check <vg> <pool>` sends only reserve/release_metadata_snap; an orphan needs two reads; fails open (never stops a backup). The repair stays human |
| `guesthook.SnippetReady` / `Register` (v0.146.0, R-861) | internal/guesthook/install.go | `SnippetReady(path) error` | is the pre-start hook (a FIXED file from the bundle) in place — register only then | The agent never installs the hook (Proxmox runs it as root); a missing hookscript stops a guest start, so never `Register` without `SnippetReady` | | `guesthook.SnippetReady` / `Register` (v0.146.0, R-861) | internal/guesthook/install.go | `SnippetReady(path) error` | is the pre-start hook (a FIXED file from the bundle) in place — register only then | The agent never installs the hook (Proxmox runs it as root); a missing hookscript stops a guest start, so never `Register` without `SnippetReady` |
### Disk / format safety (role gates, durable IDs, format guards) ### Disk / format safety (role gates, durable IDs, format guards)
@@ -41,6 +42,11 @@
| `storage.ResolveStorageDevice` | internal/storage/durable_device.go | `ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)` | re-resolve enrolled STORAGE drives (remount) | `uuid:` scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node | | `storage.ResolveStorageDevice` | internal/storage/durable_device.go | `ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)` | re-resolve enrolled STORAGE drives (remount) | `uuid:` scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node |
| `deriveDurableID` | internal/storage/durableid.go | `deriveDurableID(typ, s, backingDevice, uuid) string` | storage-target durable id (DR re-attach key) | Deterministic per type; `uuid:` for usb/local-dir; PBS id includes `#<fingerprint>` | | `deriveDurableID` | internal/storage/durableid.go | `deriveDurableID(typ, s, backingDevice, uuid) string` | storage-target durable id (DR re-attach key) | Deterministic per type; `uuid:` for usb/local-dir; PBS id includes `#<fingerprint>` |
| `SystemDisks` / `isSystemBacked` / `RoleForStorage` / `RoleForRawDevice` | internal/storage/role.go | `RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole` | protection-tier classification | Fails safe to `system` (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied | | `SystemDisks` / `isSystemBacked` / `RoleForStorage` / `RoleForRawDevice` | internal/storage/role.go | `RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole` | protection-tier classification | Fails safe to `system` (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied |
| `storage.IdentityResolver` / `Relation` / `MountFor` | internal/storage/identity.go | `HostPath(path) PlaceIdentity`, `Device(dev) PlaceIdentity`, `Relation(a, b) (different, proven, reason)` | physical-disk identity of a storage place (felhom.eu `16-storage-physical-identity.md` §3, v0.157.0) | Reads /sys + /run/udev/data + /proc/mounts only (no block I/O). Judge a drive ONLY on its own mount (`localapi.ownMountIdentity`) — never the containing fs, or an absent drive reads as the root disk. A shared disk key is "same, proven" even when unknown. Pinned by identity_test.go (fixture trees) + samedisk_test.go |
| `localapi.Server.EnsureInternalPlace` / `InternalPlaceRecord` | internal/localapi/internalplace.go | `EnsureInternalPlace(ctx)` (startup + 20 s tick); `ReadInternalPlace` / `WriteInternalPlace` | the one-disk box's internal data place (felhom.eu `16` §5, v0.158.0): mounts it at /mnt/belso by fs-UUID like an enrolled drive, records it `enrolled`; the controller binds + registers it | Never takes /mnt/belso from a household drive (a foreign mount, or a unit naming another UUID, waits). Never formats/ejects (dm on the system disk → role system). The ONLY maker is the root helper `configs/felhom-internal-place` (installer + signed `add_internal_place`). Pinned by internalplace_test.go |
| `configs/felhom-internal-place` | configs/ (bundle → /usr/local/sbin) | `plan\|create <planned GiB>` | the ruled size rule (reserve max(16, 10 %) ≤ 48 GiB; ≥ 20 GiB or nothing) + the only lvcreate/mkfs of a system-disk volume | Formats only a felhom-belso with NO signature; refuses a foreign one; never resizes/removes. Sudoers: `^(plan\|create) 0$` only. Pinned by configs/test_felhom_internal_place.py |
| `storage.candidatePartitions` / `CandidatePartition` | internal/storage/partitions.go | `CandidateDisk.Partitions` | partitions as places (`16` §6): empty / filesystem / in_use per part of a free disk; `Formattable` only while no node of the disk is mounted | A part with children or a member type is never listed; a raw /mnt mount counts as Felhom's only when corroborated (felhomOwnedMounts) — for LISTING only: the root mkfs wrapper trusts no such claim (v0.158.1). Pinned by partitions_test.go |
| partition wipe id `byid:<…-partN>+fsuuid:<uuid>` | internal/storage/durable_device.go | `DeviceDurableID` / `ResolveDurableDevice` | a data-bearing PART binds disk id + part number + filesystem (`16` §6) | Both halves must resolve to the SAME device or the wipe refuses. Whole disks keep the plain id. Pinned by partition_durable_test.go |
| `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process | | `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process |
| `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` | | `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` |
@@ -51,6 +57,7 @@
| `SudoHostOps.EnsureMount` | internal/storage/hostops.go | `EnsureMount(ctx, MountSpec) error` | persistent by-UUID systemd .mount | Validate→render→stage→`install`→`enable --now`; idempotent | | `SudoHostOps.EnsureMount` | internal/storage/hostops.go | `EnsureMount(ctx, MountSpec) error` | persistent by-UUID systemd .mount | Validate→render→stage→`install`→`enable --now`; idempotent |
| `SudoHostOps.Unmount` | internal/storage/hostops.go | `Unmount(ctx, where) error` | detach a mount unit | DESTRUCTIVE — caller MUST have gated it; does not self-authorize | | `SudoHostOps.Unmount` | internal/storage/hostops.go | `Unmount(ctx, where) error` | detach a mount unit | DESTRUCTIVE — caller MUST have gated it; does not self-authorize |
| `SudoHostOps.ReassertEnrolledMounts` | internal/storage/hostops.go | `ReassertEnrolledMounts(ctx)` | reboot remount (re-resolve by UUID) | Re-asserts unless mounted AND enabled (`shouldReassertMount`); skips absent UUIDs | | `SudoHostOps.ReassertEnrolledMounts` | internal/storage/hostops.go | `ReassertEnrolledMounts(ctx)` | reboot remount (re-resolve by UUID) | Re-asserts unless mounted AND enabled (`shouldReassertMount`); skips absent UUIDs |
| `SudoHostOps.ForgetRetiredDrives` (v0.159.0, R-1124) | internal/storage/forget_retired.go | `ForgetRetiredDrives(ctx, IntentReader) int` | drop the host mount unit of a DECOMMISSIONED drive that is GONE (start-up + the 20 s tick) | Only decommissioned + absent + unmounted; a present decommissioned drive keeps its unit (H3 re-enroll). The file goes only through `felhom-priv-apply unit-forget`, which re-checks all three as root |
| `GuestBinder.AttachDrive` / `DetachDrive` | internal/localapi/intermediary.go | `AttachDrive(ctx, vmid, where) (guestPath, err)` | live drive hot-swap under `/mnt/felhom-drives` | Normalizes to EXACTLY ONE bind via `countHostMounts` (converges double-binds); force re-bind when guest can't see it | | `GuestBinder.AttachDrive` / `DetachDrive` | internal/localapi/intermediary.go | `AttachDrive(ctx, vmid, where) (guestPath, err)` | live drive hot-swap under `/mnt/felhom-drives` | Normalizes to EXACTLY ONE bind via `countHostMounts` (converges double-binds); force re-bind when guest can't see it |
| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness | | `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from | | `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
@@ -88,6 +95,8 @@
| Symbol | File | Short signature | Use for | Gotchas | | Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---| |---|---|---|---|---|
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in | | `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them | | `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc | | `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
@@ -105,8 +114,8 @@
| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled | | `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability | | `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
| `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew | | `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew |
| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v1) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) | | `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v2, R-978: json mode returns the package in `PendingUpload` instead of uploading) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) |
| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged | | localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); **R-978: the new package reaches the hub only at the claim, before R is handed out (`EscrowCeremonyConfig.Upload`); TTL and a new start wait while that upload is in flight — an abandoned ceremony leaves the previous package current**; **R-1151: a claim with `{"defer_upload":true}` hands out R + two word positions and keeps the package until `POST /escrow/ceremony/confirm` brings those two words back (30-min window, only the two words kept, never R); a claim without it keeps the R-978 order (old controllers)**; subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` | | `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
| `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) | | `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) |
| `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker | | `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker |
@@ -156,18 +165,22 @@
| `proxmox.Runner` | internal/proxmox/privileged.go | `*ExecRunner` (direct/sudo) | `mockRunner` internal/proxmox/mock_test.go; runner fakes in storage tests | | `proxmox.Runner` | internal/proxmox/privileged.go | `*ExecRunner` (direct/sudo) | `mockRunner` internal/proxmox/mock_test.go; runner fakes in storage tests |
| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go | | `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). | | `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
| `storage.IdentityHost` | internal/storage/identity.go | `*SysfsIdentityHost` | fixture /sys + udev trees in identity_test.go (`newFx`); `fakeIdentHost` in internal/localapi/samedisk_test.go. A SEPARATE interface from HostReader on purpose — adding methods to HostReader would break every existing fake. |
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go | | `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `lanresolver.hostRoot` + `dnsmasqUnitPaths` (data seam, R-317) | internal/lanresolver/lanresolver.go | prod `hostRoot = "/"`; probe = the `dnsmasq` package's systemd UNIT, never `/usr/sbin/dnsmasq` (owned by `dnsmasq-base`) | internal/lanresolver/ensure_dnsmasq_test.go — fixture root tree + recording `proxmox.Runner`; the REAL `os.Stat` probe and `EnsureDnsmasq` run. `TestEnsureDnsmasq_ProductionProbeIsTheUnit` pins the production wiring | | `lanresolver.hostRoot` + `dnsmasqUnitPaths` (data seam, R-317) | internal/lanresolver/lanresolver.go | prod `hostRoot = "/"`; probe = the `dnsmasq` package's systemd UNIT, never `/usr/sbin/dnsmasq` (owned by `dnsmasq-base`) | internal/lanresolver/ensure_dnsmasq_test.go — fixture root tree + recording `proxmox.Runner`; the REAL `os.Stat` probe and `EnsureDnsmasq` run. `TestEnsureDnsmasq_ProductionProbeIsTheUnit` pins the production wiring |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go | | `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). | | `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct | | `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct |
| `linetest.Tester` + `linetest.Client` (R-1247) | internal/linetest/linetest.go · internal/linetest/iperf.go | `New(runner, target, gate, uplink, hostID, statePath, logger)` / `Tick(ctx)` / `Value()` / `LineTestStatus(ctx)`; `Client{Streams, Time}.Run(ctx, addr)` | the weekly upload line test: a 10-second iperf3 upload (the agent speaks the iperf3 protocol — nothing installed on the host) to ep0's tunnel-only server (`wgtunnel.Manager.LineTestTarget`), Tuesday 10:00-15:59 local from a per-host slot, after 10 quiet minutes of the PHYSICAL uplink (`SysUplinkTx`: the bridge's ports with a `device` link — the bridge counter misses bridged guest traffic), under `backup.InFlight`; best of 3; reported as `line_test`, served as GET /host/line-test | **The value is the SERVER's received bytes**, never the client's written ones (the send buffer). `Run` refuses a context without a deadline. iperf3 numbers streams 1, 3, 4, 5 — the server matches results by those ids. „server busy" (ACCESS_DENIED) stops the measurement and is retried at the next tick, at most `MaxTriesPerDay` |
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. | | `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. | | `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
| localapi backup cadence cap (R-973) | internal/localapi/backup_cap.go | `backupCadenceCap(cadence)`, `Server.capRefuses`, `Server.BackupCapRefusal()` | POST /backup refuses (429) a run the `07` §6 schedule would never ask for: ≤24 h tier → 3 successes per rolling 24 h; longer tier → 1 per max(cadence−24 h, cadence/2) | Counts SUCCESSES only, from this process AND `BackupSuccessState.RecentBackupSuccesses` (so a restart does not reset it); refusals saved (`RecordBackupRefusal`) and raised for 24 h as the CRITICAL host-report status `localapi:backup-cadence-cap` (`backupCapStatus` in main) → the hub's existing `agent_capability_degraded`. Tests `TestBackupCap_*`. |
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. | | `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. | | `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). | | `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go | | `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go | | `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec; the image write via `felhom-priv-apply controller-image`) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls | | `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment | | `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind | | `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
+67
View File
@@ -0,0 +1,67 @@
package main
import (
"go/ast"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
)
// R-973: a cadence-cap refusal becomes a CRITICAL degraded status — what the hub's existing
// agent_capability_degraded event (operator-only) keys on — and clears when the refusal ages out.
func TestBackupCapStatus_RefusalIsCriticalDegraded(t *testing.T) {
at := time.Date(2026, 10, 18, 3, 0, 0, 0, time.UTC)
s := backupCapStatus(func() (string, int, time.Time, bool) { return "felhom-pbs", 9201, at, true })
if !s.Critical || s.Status != capability.StatusDegraded || !strings.Contains(s.Reason, "9201") || !strings.Contains(s.Reason, "felhom-pbs") {
t.Fatalf("a refusal must raise a critical degraded status naming the guest and the tier: %+v", s)
}
if s := backupCapStatus(func() (string, int, time.Time, bool) { return "", 0, time.Time{}, false }); s.Status != capability.StatusOK || !s.Critical {
t.Fatalf("no refusal → ok (and still critical, so a later refusal alerts): %+v", s)
}
if s := backupCapStatus(nil); s.Status != capability.StatusOK {
t.Fatalf("no local API → ok: %+v", s)
}
}
// Wiring (the built-but-never-wired class): runDaemon's capability probe appends backupCapStatus, and the
// late-bound source is set from the local API server.
//
// COMPANION RED-PROOF (observed 2026-10-18): delete `backupCapRefusal = localSrv.BackupCapRefusal` →
// "runDaemon never binds backupCapRefusal to the local API server". Restored.
func TestR973_CapRefusalIsWiredIntoTheReport(t *testing.T) {
_, f := parseMain(t)
var appended, bound bool
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "runDaemon" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.CallExpr:
if id, ok := x.Fun.(*ast.Ident); ok && id.Name == "backupCapStatus" && len(x.Args) == 1 {
if a, ok := x.Args[0].(*ast.Ident); ok && a.Name == "backupCapRefusal" {
appended = true
}
}
case *ast.AssignStmt:
if len(x.Lhs) == 1 && len(x.Rhs) == 1 {
if l, ok := x.Lhs[0].(*ast.Ident); ok && l.Name == "backupCapRefusal" {
if sel, ok := x.Rhs[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "BackupCapRefusal" {
bound = true
}
}
}
}
return true
})
}
if !appended {
t.Fatal("runDaemon's capability probe does not call backupCapStatus(backupCapRefusal)")
}
if !bound {
t.Fatal("runDaemon never binds backupCapRefusal to the local API server")
}
}
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"context"
"encoding/json"
"go/ast"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// R-978, the root half: the controller-driven ceremony (json mode, the FELHOM_ESCROW argv with
// `--upload`) must hand its package back instead of sending it — the daemon sends it at the claim.
//
// COMPANION RED-PROOF (observed 2026-10-18): set `deferUpload: false` in escrowOptsFromCLI → this fails
// with "json mode must defer the upload (R-978): got deferUpload=false", and the delivery test's json leg
// then sends the package at create time ("put called 1 times"). Restored.
func TestEscrowOptsFromCLI_JSONModeDefersUpload(t *testing.T) {
// The FELHOM_ESCROW argv: --selftest=escrow-create --upload --output=json.
if o := escrowOptsFromCLI("", false, false, true, "", true); !o.deferUpload || !o.upload {
t.Fatalf("json mode must defer the upload (R-978): got deferUpload=%v upload=%v", o.deferUpload, o.upload)
}
// Text mode (an operator at a terminal) keeps the immediate upload; no --upload means no upload at all.
if o := escrowOptsFromCLI("", false, false, true, "", false); o.deferUpload {
t.Fatal("text mode must upload at once")
}
if o := escrowOptsFromCLI("", false, false, false, "", true); o.deferUpload {
t.Fatal("without --upload there is nothing to defer")
}
}
func TestDeliverEscrowPackage_JSONModeSendsNothing(t *testing.T) {
cfg := config.Config{}
cfg.Hub.URL, cfg.Hub.HostID, cfg.Hub.APIKey = "https://hub.invalid", "host-1", "k"
res := escrow.CreateResult{Blob: []byte("ciphertext"), KeyFingerprint: "f2:87", Posture: escrow.PostureZeroKnowledge, IdentityBlob: []byte("id")}
for _, jsonMode := range []bool{true, false} {
calls := 0
var sent []byte
put := func(_ context.Context, _ config.Config, body []byte) error { calls++; sent = body; return nil }
o := escrowOptsFromCLI("", false, false, true, "", jsonMode)
pending, uploaded, err := deliverEscrowPackage(context.Background(), cfg, res, "abc", o.deferUpload, put)
if err != nil {
t.Fatal(err)
}
if jsonMode {
if calls != 0 || uploaded {
t.Fatalf("json mode: put called %d times (uploaded=%v), want 0 — the package must wait for the claim", calls, uploaded)
}
var m map[string]any
if json.Unmarshal(pending, &m) != nil || m["blob_b64"] == "" || m["identity_blob_b64"] == "" || m["restic_pw_sha256"] != "abc" {
t.Fatalf("json mode: the pending package is not the hub upload body: %s", pending)
}
} else if calls != 1 || !uploaded || pending != nil || len(sent) == 0 {
t.Fatalf("text mode: put called %d times, uploaded=%v, pending=%q — want one immediate upload", calls, uploaded, pending)
}
}
// A hub that is not configured fails in both modes (as the pre-R-978 upload did).
if _, _, err := deliverEscrowPackage(context.Background(), config.Config{}, res, "", true, nil); err == nil {
t.Fatal("an unconfigured hub must fail the ceremony even when the upload is deferred")
}
}
// R-978 wiring (the built-but-never-wired class): runDaemon's localapi.EscrowCeremonyConfig carries an
// Upload that calls putEscrowBody. Without it every claim of a new package answers 503 and no customer
// can finish a ceremony.
//
// COMPANION RED-PROOF (observed 2026-10-18): delete the `Upload:` line in runDaemon → this fails with
// "runDaemon's EscrowCeremonyConfig has no Upload calling putEscrowBody". Restored.
func TestR978_CeremonyUploadIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
if !callsWithin(f, "main")["runDaemon"] {
t.Fatal("main → runDaemon is broken — the path this test asserts is not the live one")
}
wired := false
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "runDaemon" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
cl, ok := n.(*ast.CompositeLit)
if !ok {
return true
}
sel, ok := cl.Type.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "EscrowCeremonyConfig" {
return true
}
for _, el := range cl.Elts {
if kv, ok := el.(*ast.KeyValueExpr); ok {
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "Upload" {
ast.Inspect(kv.Value, func(n ast.Node) bool {
if ce, ok := n.(*ast.CallExpr); ok {
if id, ok := ce.Fun.(*ast.Ident); ok && id.Name == "putEscrowBody" {
wired = true
}
}
return true
})
}
}
}
return true
})
}
if !wired {
t.Fatal("runDaemon's EscrowCeremonyConfig has no Upload calling putEscrowBody")
}
}
+72
View File
@@ -0,0 +1,72 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
)
// R-1250: SetLineTest takes a LineTestReader and finds the request half by a type assertion, so a Tester that lost
// WaitingForQuiet or RunNow would compile and silently answer 503 to every seeding box. This line makes it a build
// failure instead.
var _ localapi.LineTestRequester = (*linetest.Tester)(nil)
// R-1247: the weekly line test has the fstrim shape (component + reporter seam + goroutine) plus a local-API reader,
// so its wiring is asserted from the AST. It must share the ONE heavy-op gate (heavyOps) — or it could run beside a
// backup — and take its server from the tunnel manager, never a literal address.
func TestMainWiresTheLineTest(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var withGate, withTunnelTarget, reporterWired, localAPIWired, started bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
if fn, ok := node.Fun.(*ast.SelectorExpr); ok {
switch fn.Sel.Name {
case "New":
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "linetest" && len(node.Args) >= 3 {
if id, ok := node.Args[2].(*ast.Ident); ok && id.Name == "heavyOps" {
withGate = true
}
if sel, ok := node.Args[1].(*ast.SelectorExpr); ok && sel.Sel.Name == "LineTestTarget" {
withTunnelTarget = true
}
}
case "SetLineTestReporter":
reporterWired = true
case "SetLineTest":
localAPIWired = true
}
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "lineTest" {
started = true
}
}
}
return true
})
if !withGate {
t.Error("main.go never calls linetest.New(..., heavyOps, ...) — no line test, or one outside the heavy-op gate")
}
if !withTunnelTarget {
t.Error("linetest.New does not take its server from wgMgr.LineTestTarget")
}
if !reporterWired {
t.Error("main.go never calls collector.SetLineTestReporter — the line_test stanza never reaches the hub")
}
if !localAPIWired {
t.Error("main.go never calls localSrv.SetLineTest — the controller's GET /host/line-test always reads present:false")
}
if !started {
t.Error("main.go never starts the line test with `go lineTest.Run(ctx)`")
}
}
+299 -45
View File
@@ -43,6 +43,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/guestnet" "gitea.dooplex.hu/admin/felhom-agent/internal/guestnet"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub" "gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver" "gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi" "gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log" applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
"gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane" "gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane"
@@ -50,6 +51,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/pbs" "gitea.dooplex.hu/admin/felhom-agent/internal/pbs"
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr" "gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
"gitea.dooplex.hu/admin/felhom-agent/internal/poke" "gitea.dooplex.hu/admin/felhom-agent/internal/poke"
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
"gitea.dooplex.hu/admin/felhom-agent/internal/provision" "gitea.dooplex.hu/admin/felhom-agent/internal/provision"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
@@ -58,6 +60,8 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate" "gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs" "gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage" "gitea.dooplex.hu/admin/felhom-agent/internal/storage"
"gitea.dooplex.hu/admin/felhom-agent/internal/thincheck"
"gitea.dooplex.hu/admin/felhom-agent/internal/vzpartial"
"gitea.dooplex.hu/admin/felhom-agent/internal/wgtunnel" "gitea.dooplex.hu/admin/felhom-agent/internal/wgtunnel"
) )
@@ -164,7 +168,6 @@ func main() {
keyDest string keyDest string
installWGKey bool installWGKey bool
idBundlePath string idBundlePath string
directivePath string
swapImage string swapImage string
outputMode string outputMode string
showVersion bool showVersion bool
@@ -192,13 +195,12 @@ func main() {
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)") flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)") flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)")
flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))") flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))")
flag.BoolVar(&upload, "upload", false, "for --selftest=escrow-create: upload the opaque blob to the hub") flag.BoolVar(&upload, "upload", false, "for --selftest=escrow-create: upload the opaque blob to the hub (with --output=json: hand the package back in pending_upload — the agent daemon uploads it when the code is claimed, R-978)")
flag.StringVar(&blobPath, "blob", "", "for --selftest=escrow-consume: path to the R-wrapped escrow blob file") flag.StringVar(&blobPath, "blob", "", "for --selftest=escrow-consume: path to the R-wrapped escrow blob file")
flag.StringVar(&expectedFP, "fingerprint", "", "for --selftest=escrow-consume: the EXPECTED key fingerprint (the gate target)") flag.StringVar(&expectedFP, "fingerprint", "", "for --selftest=escrow-consume: the EXPECTED key fingerprint (the gate target)")
flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)") flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)")
flag.BoolVar(&installWGKey, "install-wg-key", false, "for --selftest=identity-consume: ALSO install the recovered wg_private_key into wgtunnel's key file (S5 DR; create-only, refuses to overwrite)") flag.BoolVar(&installWGKey, "install-wg-key", false, "for --selftest=identity-consume: ALSO install the recovered wg_private_key into wgtunnel's key file (S5 DR; create-only, refuses to overwrite)")
flag.StringVar(&idBundlePath, "identity-bundle", "", "for --selftest=escrow-create: a 0600 JSON file {tunnel_token,pbs_token} to ALSO escrow under R (10D)") flag.StringVar(&idBundlePath, "identity-bundle", "", "for --selftest=escrow-create: a 0600 JSON file {tunnel_token,pbs_token} to ALSO escrow under R (10D)")
flag.StringVar(&directivePath, "directive", "", "for --selftest=escrow-create: a JSON file with the non-secret DR directive (pbs repo/ns, expected fingerprint, tunnel id)")
flag.StringVar(&custID, "customer-id", "", "for --selftest=provision: the customer id — the hub config-pull target, baked into the guest's bootstrap") flag.StringVar(&custID, "customer-id", "", "for --selftest=provision: the customer id — the hub config-pull target, baked into the guest's bootstrap")
flag.StringVar(&hubPassword, "hub-password", "", "for --selftest=provision: the customer's hub RETRIEVAL PASSPHRASE (SECRET) — baked into bootstrap.json so the controller pulls its config (and the customer-scoped hub key) from the hub. The customer must already exist in the hub.") flag.StringVar(&hubPassword, "hub-password", "", "for --selftest=provision: the customer's hub RETRIEVAL PASSPHRASE (SECRET) — baked into bootstrap.json so the controller pulls its config (and the customer-scoped hub key) from the hub. The customer must already exist in the hub.")
flag.StringVar(&swapImage, "image", "", "for --selftest=controller-swap: the target controller image ref (gitea.dooplex.hu/admin/felhom-controller:<semver>) — must already be pulled in the guest") flag.StringVar(&swapImage, "image", "", "for --selftest=controller-swap: the target controller image ref (gitea.dooplex.hu/admin/felhom-controller:<semver>) — must already be pulled in the guest")
@@ -269,7 +271,7 @@ func main() {
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount, Cores: cores, MemoryMB: memoryMB}, SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount, Cores: cores, MemoryMB: memoryMB},
})) }))
case "escrow-create": case "escrow-create":
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath, outputMode)) os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, outputMode))
case "escrow-consume": case "escrow-consume":
os.Exit(runSelftestEscrowConsume(context.Background(), logger, blobPath, expectedFP, keyDest)) os.Exit(runSelftestEscrowConsume(context.Background(), logger, blobPath, expectedFP, keyDest))
case "identity-consume": case "identity-consume":
@@ -403,6 +405,31 @@ func (r *gateRemounter) Remount(ctx context.Context, t storage.KnownTarget) {
// poolReadStatus probes the felhom-pool membership read (GET /pools/felhom) the stale-lock reaper's // poolReadStatus probes the felhom-pool membership read (GET /pools/felhom) the stale-lock reaper's
// ownership scoping depends on (v0.62.0, audit A1). Needs `Pool.Audit` at `/pool/felhom` — granted // ownership scoping depends on (v0.62.0, audit A1). Needs `Pool.Audit` at `/pool/felhom` — granted
// by host-install v1.9.0; on an older ACL this reports degraded and the reaper fail-safes (skips). // by host-install v1.9.0; on an older ACL this reports degraded and the reaper fail-safes (skips).
// backupCapStatusName is the host-report status the R-973 cadence cap raises.
const backupCapStatusName = "localapi:backup-cadence-cap"
// backupCapStatus turns the local-API backup cadence cap's newest refusal (R-973) into a CRITICAL
// capability status: degraded for 24 h after a refusal, so the hub raises its operator-only
// whole_guest_backup_refused event (hub v0.159.0, R-1150; agent_capability_degraded before). The Reason names the guest, the tier and the time, and
// says what to do. nil source (local API off) → ok.
func backupCapStatus(refusal func() (string, int, time.Time, bool)) capability.Status {
s := capability.Status{
Name: backupCapStatusName,
Feature: "whole-guest restore points: a guest asked for more whole-guest backups than its schedule allows, and the extra ones were refused (R-973)",
Critical: true,
Status: capability.StatusOK,
}
if refusal == nil {
return s
}
if target, vmid, at, ok := refusal(); ok {
s.Status = capability.StatusDegraded
s.Reason = fmt.Sprintf("refused a whole-guest backup of guest %d on target %q at %s — more than the schedule allows; the guest's controller may be misbehaving or compromised: check the guest",
vmid, target, at.UTC().Format(time.RFC3339))
}
return s
}
func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status { func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status {
s := capability.Status{ s := capability.Status{
Name: "pve:pool-read", Name: "pve:pool-read",
@@ -777,6 +804,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
hostReader := storage.NewProcHostReader() hostReader := storage.NewProcHostReader()
hostOps := newHostOps(cfg, logger) hostOps := newHostOps(cfg, logger)
observer := storage.NewObserver(px, hostReader, hostOps, logger) observer := storage.NewObserver(px, hostReader, hostOps, logger)
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
// Backup + restore-test state store (slice 6): holds the latest backup-per-target + // Backup + restore-test state store (slice 6): holds the latest backup-per-target +
// latest restore-test result; the collector reads it via the BackupReporter / // latest restore-test result; the collector reads it via the BackupReporter /
// RestoreTestReporter seams; the cadence scheduler writes it. // RestoreTestReporter seams; the cadence scheduler writes it.
@@ -792,6 +820,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
pbsTargets := pbsTargetsFromPVE(cfg, px, logger) pbsTargets := pbsTargetsFromPVE(cfg, px, logger)
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger) pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)
collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger) collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
// R-366 slice 2: the restore-test's ledger of archives written with another key → the host report.
foreignKeys := backup.NewForeignKeyLedger()
collector.SetForeignKeyArchiveReporter(foreignKeys)
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target
// Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent // Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent
// depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the // depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the
@@ -825,14 +856,37 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run, run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
log: logger, log: logger,
} }
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
// drConfigured: the local API server is built further down, before the loop starts.
var backupCapRefusal func() (string, int, time.Time, bool)
probeAll := func(ctx context.Context) []capability.Status { probeAll := func(ctx context.Context) []capability.Status {
out := append(capProber.Probe(ctx), poolReadStatus(ctx, px)) out := append(capProber.Probe(ctx), poolReadStatus(ctx, px))
out = append(out, backupCapStatus(backupCapRefusal))
return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...) return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...)
} }
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot // (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
// already carries the gated view — v0.86.0.) // already carries the gated view — v0.86.0.)
collector.SetCapabilityProber(probeAll) collector.SetCapabilityProber(probeAll)
// R-998 (doc 14): the household power-off record of THIS boot rides every report until the box is down.
powerOffState := poweroff.New(filepath.Join(cfg.OOB.WithDefaults().StateDir, "poweroff.json"), poweroff.ProcBtime)
collector.SetDeliberateOffSource(func() *hub.DeliberateOff {
if r := powerOffState.Current(); r != nil {
return &hub.DeliberateOff{RequestedAt: r.RequestedAt.UTC().Format(time.RFC3339)}
}
return nil
})
loop := hub.NewLoop(collector, client, time.Duration(hcfg.PollSeconds)*time.Second, logger) loop := hub.NewLoop(collector, client, time.Duration(hcfg.PollSeconds)*time.Second, logger)
// R-998: the power-off path waits until the hub ACCEPTED a report carrying the record.
powerOffAck := make(chan struct{}, 1)
loop.SetReportedHook(func(r *hub.HostReport) {
if r != nil && r.DeliberateOff != nil {
select {
case powerOffAck <- struct{}{}:
default:
}
}
})
interval := time.Duration(hcfg.PollSeconds) * time.Second interval := time.Duration(hcfg.PollSeconds) * time.Second
// Heartbeat log-pull (v0.83.0): when the control envelope requests it, the NEXT // Heartbeat log-pull (v0.83.0): when the control envelope requests it, the NEXT
// heartbeat carries the debug ring's tail (newest-kept, byte-capped in the loop). // heartbeat carries the debug ring's tail (newest-kept, byte-capped in the loop).
@@ -876,6 +930,29 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) { go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) {
logger.Info("osupdate: sent kept report(s)", "count", n) logger.Info("osupdate: sent kept report(s)", "count", n)
}) })
// R-836 (`09` §3 decision 172): what became of a kernel step across this boot; on a one-shot boot of a new kernel,
// judge it (the host health rule + the hub reached) for KernelJudgeWait, then make it the default or revert ONCE.
// The wait: measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`) — every container healthy 68 s after the
// reboot on demo-felhom and 272 s on demo-hp (the hub reached at 63 s / 189 s); 20 minutes leaves room for a slow
// network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
// after start so the guest has settled; a failure is logged and the next start retries.
go func() {
select {
case <-ctx.Done():
return
case <-time.After(time.Minute):
}
sctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
if n, err := localapi.EnsureGuestSwap(sctx, px, reconcile.DefaultPool, logger); err != nil {
logger.Warn("guest-swap: check skipped", "err", err)
} else {
logger.Info("guest-swap: checked", "raised", n)
}
}()
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue // Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving // (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
@@ -930,10 +1007,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// state. Gates the watchdog's self-heal re-mount to ENROLLED drives, and the local API records // state. Gates the watchdog's self-heal re-mount to ENROLLED drives, and the local API records
// enroll/eject into it. Open failure → ungated legacy remount (logged) so the daemon still runs. // enroll/eject into it. Open failure → ungated legacy remount (logged) so the daemon still runs.
var intentReader storage.IntentReader var intentReader storage.IntentReader
intentStateDir := "/var/lib/felhom-agent" intentStateDir := agentStateDir(cfg)
if cfg.Authz.NonceStorePath != "" {
intentStateDir = filepath.Dir(cfg.Authz.NonceStorePath) // share the agent's durable state dir
}
intentStore, ierr := storage.OpenIntentStore(filepath.Join(intentStateDir, "drive-intents.json")) intentStore, ierr := storage.OpenIntentStore(filepath.Join(intentStateDir, "drive-intents.json"))
if ierr != nil { if ierr != nil {
logger.Warn("storage: intent store unavailable — self-heal runs UNGATED (legacy remount-any)", "err", ierr) logger.Warn("storage: intent store unavailable — self-heal runs UNGATED (legacy remount-any)", "err", ierr)
@@ -1017,7 +1091,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// with the local API so a backup and a restore-test can never run together. // with the local API so a backup and a restore-test can never run together.
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json")) rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
heavyOps := &backup.InFlight{} heavyOps := &backup.InFlight{}
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger) scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, foreignKeys, logger)
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store // R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
// holds only this process's latest run, and under per-archive due-ness the agent will not // holds only this process's latest run, and under per-archive due-ness the agent will not
// re-test an archive it has already proven — so without this the hub can report a tier unproven // re-test an archive it has already proven — so without this the hub can report a tier unproven
@@ -1065,6 +1139,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
decommIntent = intentStore decommIntent = intentStore
} }
decommExec := signedjobs.NewDecommissionExecutor(decommIntent, logger) decommExec := signedjobs.NewDecommissionExecutor(decommIntent, logger)
if guestBindStore != nil { // typed-nil guard, as above
decommExec.SetGuestBinds(guestBindStore) // v0.159.1: prune the bind record like the self-serve decommission
}
// Agent self-update (TASK D1): the agent_update executor downloads the operator-signed binary, // Agent self-update (TASK D1): the agent_update executor downloads the operator-signed binary,
// verifies it against the SIGNED sha256, and hands it to the root guarded wrapper (A/B flip + // verifies it against the SIGNED sha256, and hands it to the root guarded wrapper (A/B flip +
@@ -1109,6 +1186,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} }
return release, nil return release, nil
}} }}
// R-812 option A: a signed Proxmox package step (`11` §5.10) — ring 1; under the heavy-op gate and the /etc/pve gate.
pveExec := osupdate.PVEStepExecutor{Leg: osLeg, Guest: firstGuest(px),
Gate: func(ctx context.Context) (func(), error) {
release, busy, ok := heavyOps.TryAcquire("os-pve-step")
if !ok {
return nil, fmt.Errorf("busy: %s", busy)
}
return release, nil
}}
// R-836 (`09` §3 decision 172): a signed kernel step STAGES a kernel on a ring-1 box (install + the one-shot flag,
// never a reboot — the night leg reboots it on a night the household was told about); under the heavy-op gate.
kernelExec := osupdate.KernelStepExecutor{Leg: osLeg, Guest: firstGuest(px),
Gate: func(ctx context.Context) (func(), error) {
release, busy, ok := heavyOps.TryAcquire("os-kernel-step")
if !ok {
return nil, fmt.Errorf("busy: %s", busy)
}
return release, nil
}}
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it. // Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token, bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants. // The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
@@ -1120,7 +1216,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} }
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ",")) logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
}} }}
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, bundleExec}, cfg.Hub.HostID, logger) // R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
// to the local API server built below).
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
localapi.InternalPlaceFileIn(intentStateDir), logger)
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, kernelExec, bundleExec, internalPlaceExec}, cfg.Hub.HostID, logger)
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner)) loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate — // Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
@@ -1144,6 +1247,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
return cfg.Escrow.PBSStorageID return cfg.Escrow.PBSStorageID
}, },
HubConfigured: cfg.Hub.URL != "" && cfg.Hub.HostID != "" && cfg.Hub.APIKey != "", HubConfigured: cfg.Hub.URL != "" && cfg.Hub.HostID != "" && cfg.Hub.APIKey != "",
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
// subprocess used, under the same per-host key, before the code is handed out.
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
DRConfigured: func() bool { DRConfigured: func() bool {
if drConfigured != nil { if drConfigured != nil {
return drConfigured() return drConfigured()
@@ -1153,6 +1259,36 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} }
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens) localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil { if localSrv != nil {
backupCapRefusal = localSrv.BackupCapRefusal // R-973
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
pwMode := proxmox.RunnerMode(cfg.Privileged.Mode)
if pwMode == "" {
pwMode = proxmox.RunnerSudo
}
localSrv.SetHostPowerOff(localapi.HostPowerOffConfig{
State: powerOffState,
Power: localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: pwMode, SudoPath: cfg.Privileged.SudoPath}, logger),
ReportNow: func(ctx context.Context) bool {
select {
case storageTrigger <- struct{}{}:
default: // a report is already queued; it runs after the record was written
}
select {
case <-powerOffAck:
return true
case <-ctx.Done():
return false
}
},
Uptime: func() (time.Duration, bool) {
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
if err != nil || b <= 0 {
return 0, false
}
return time.Since(time.Unix(b, 0)), true
},
})
localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) { localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) {
// Let the controller finish bringing its apps back after the backup, then run (still under the gate). // Let the controller finish bringing its apps back after the backup, then run (still under the gate).
select { select {
@@ -1343,7 +1479,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// filesystem UUID and re-mounts (idempotent `enable --now`) so a letter reshuffle is a no-op. // filesystem UUID and re-mounts (idempotent `enable --now`) so a letter reshuffle is a no-op.
// Type-asserted (the concrete op exposes it; the interface stays lean). // Type-asserted (the concrete op exposes it; the interface stays lean).
mountReasserter := hostOps // concrete *storage.SudoHostOps — exposes ReassertEnrolledMounts mountReasserter := hostOps // concrete *storage.SudoHostOps — exposes ReassertEnrolledMounts
// v0.158.0 (felhom.eu `16` §5): the internal data place is mounted like an enrolled drive — first, so the
// re-assert below sees its unit.
localSrv.EnsureInternalPlace(ctx)
if mountReasserter != nil { if mountReasserter != nil {
// R-1124: a decommissioned drive that is gone loses its host mount unit first (start-up and every tick), so
// no later boot waits for it and the re-assert below no longer lists it.
mountReasserter.ForgetRetiredDrives(ctx, intentReader)
mountReasserter.ReassertEnrolledMounts(ctx) mountReasserter.ReassertEnrolledMounts(ctx)
} }
// F9: on startup (the host's bring-up/reconcile trigger), re-assert any enrolled guest data-drive // F9: on startup (the host's bring-up/reconcile trigger), re-assert any enrolled guest data-drive
@@ -1380,7 +1522,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
case <-t.C: case <-t.C:
// Re-assert mounts first (handles a USB that enumerated late after a host reboot, or a // Re-assert mounts first (handles a USB that enumerated late after a host reboot, or a
// unit re-disabled at runtime), then re-assert the guest binds against the now-live mounts. // unit re-disabled at runtime), then re-assert the guest binds against the now-live mounts.
localSrv.EnsureInternalPlace(ctx) // `16` §5: a record written since the last tick (the signed job)
if mountReasserter != nil { if mountReasserter != nil {
mountReasserter.ForgetRetiredDrives(ctx, intentReader) // R-1124: decommissioned + gone → unit removed
mountReasserter.ReassertEnrolledMounts(ctx) mountReasserter.ReassertEnrolledMounts(ctx)
} }
localSrv.ReassertGuestBinds(ctx) localSrv.ReassertGuestBinds(ctx)
@@ -1458,6 +1602,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// the dangling snapshot → start iff onboot). Runs BEFORE the backup loop starts, so a present // the dangling snapshot → start iff onboot). Runs BEFORE the backup loop starts, so a present
// backup lock is stale by definition (guarded by a no-vzdump-running check; fail-safe otherwise). // backup lock is stale by definition (guarded by a no-vzdump-running check; fail-safe otherwise).
localSrv.RecoverStaleLockedGuests(ctx) localSrv.RecoverStaleLockedGuests(ctx)
// R-1006: then remove a dead half-written backup a power cut left (the lock recovery first: same order as a human).
localSrv.SweepVzdumpPartials(ctx)
// F-REBOOT: the startup recovery above only covers a guest left LOCKED by an interrupted // F-REBOOT: the startup recovery above only covers a guest left LOCKED by an interrupted
// backup. A guest that simply ends up stopped-and-unlocked (a `pct reboot` whose shutdown // backup. A guest that simply ends up stopped-and-unlocked (a `pct reboot` whose shutdown
// half completed and whose start half never fired — Campaign 8 fault 11, 9m47s of total // half completed and whose start half never fired — Campaign 8 fault 11, 9m47s of total
@@ -1499,6 +1645,24 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} else { } else {
logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)") logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)")
} }
// R-1247 (operator ruling 2026-10-24 07:26): the weekly upload line test — a 10-second iperf3 upload from this host
// to ep0's tunnel-only server, Tuesday 10:00-15:59 local, after 10 quiet minutes, under the one-heavy-op gate. Only
// with the offsite tunnel (the server is reachable nowhere else). Not part of the errc fan-out.
switch {
case !cfg.LineTest.Enabled():
logger.Info("linetest: weekly upload line test disabled by config (line_test.disable)")
case wgMgr == nil:
logger.Info("linetest: weekly upload line test off — no offsite tunnel on this host")
default:
lineTest := linetest.New(linetest.Client{Streams: linetest.Streams, Time: linetest.TestTime}, wgMgr.LineTestTarget,
heavyOps, linetest.SysUplinkTx("/"), cfg.Hub.HostID,
filepath.Join(cfg.OOB.WithDefaults().StateDir, "line-test.json"), logger)
collector.SetLineTestReporter(lineTest)
if localSrv != nil {
localSrv.SetLineTest(lineTest)
}
go lineTest.Run(ctx)
}
if lanLoop != nil { if lanLoop != nil {
lanServers = 1 lanServers = 1
go func() { errc <- lanLoop.Run(ctx) }() go func() { errc <- lanLoop.Run(ctx) }()
@@ -1700,7 +1864,7 @@ func primaryBackupTargetOf(cfg config.Config) func() hub.ConfiguredBackupTarget
// disables the cadence (returns a scheduler that just waits) when the cadence is off or the // disables the cadence (returns a scheduler that just waits) when the cadence is off or the
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the // scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
// machinery still works on-demand via --selftest=restore-test. // machinery still works on-demand via --selftest=restore-test.
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, logger *slog.Logger) *backup.Scheduler { func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, foreign *backup.ForeignKeyLedger, logger *slog.Logger) *backup.Scheduler {
// R-86: this is the EVALUATION interval, not the trigger. What decides a test happens is the // R-86: this is the EVALUATION interval, not the trigger. What decides a test happens is the
// per-archive due-check in internal/backup/restoretest_due.go. // per-archive due-check in internal/backup/restoretest_due.go.
cadence := cfg.Backup.RestoreTestEvalInterval() cadence := cfg.Backup.RestoreTestEvalInterval()
@@ -1719,6 +1883,9 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
min, max := cfg.Backup.ScratchBand() min, max := cfg.Backup.ScratchBand()
target := cfg.Backup.BackupTarget() target := cfg.Backup.BackupTarget()
runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger) runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger)
if foreign != nil {
runner.SetForeignKeyLedger(foreign) // R-366 slice 2: the pick records archives written with another key
}
// Every configured tier is a rotation candidate, not just the primary. // Every configured tier is a rotation candidate, not just the primary.
cfgTiers, _ := cfg.Backup.BackupTiers() // warnings already logged where the tiers are armed cfgTiers, _ := cfg.Backup.BackupTiers() // warnings already logged where the tiers are armed
tierIDs := make([]string, 0, len(cfgTiers)) tierIDs := make([]string, 0, len(cfgTiers))
@@ -1815,11 +1982,10 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
} }
apiTiers := make([]localapi.BackupTier, 0, len(backupTiers)) apiTiers := make([]localapi.BackupTier, 0, len(backupTiers))
var runner *backup.BackupRunner var runner *backup.BackupRunner
var tierRunners []*backup.BackupRunner
var tierTargets []string
for _, t := range backupTiers { for _, t := range backupTiers {
prune := "" prune := t.PruneSpec() // primary: keep-daily=N (R-1149); an extra tier: keep-last=N; 0 → no prune
if t.KeepLast > 0 {
prune = fmt.Sprintf("keep-last=%d", t.KeepLast)
}
// Pruning a PBS target is allowed ONLY for an additional tier with an explicit keep_last // Pruning a PBS target is allowed ONLY for an additional tier with an explicit keep_last
// (the primary's target AND retention both default, so it could prune the DR by accident). // (the primary's target AND retention both default, so it could prune the DR by accident).
allowPBSPrune := !t.Primary && t.KeepLast > 0 allowPBSPrune := !t.Primary && t.KeepLast > 0
@@ -1827,6 +1993,8 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
if t.Primary { if t.Primary {
runner = r runner = r
} }
tierRunners = append(tierRunners, r)
tierTargets = append(tierTargets, t.TargetID)
apiTiers = append(apiTiers, localapi.BackupTier{ apiTiers = append(apiTiers, localapi.BackupTier{
TargetID: t.TargetID, TargetID: t.TargetID,
Cadence: t.Cadence, Cadence: t.Cadence,
@@ -1835,7 +2003,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
Service: r, Service: r,
}) })
logger.Info("backup tier armed", "target", t.TargetID, "cadence", t.Cadence.String(), logger.Info("backup tier armed", "target", t.TargetID, "cadence", t.Cadence.String(),
"keep_last", t.KeepLast, "wait_timeout", t.WaitTimeout.String(), "keep_last", t.KeepLast, "prune", prune, "wait_timeout", t.WaitTimeout.String(),
"prune_pbs_allowed", allowPBSPrune, "primary", t.Primary) "prune_pbs_allowed", allowPBSPrune, "primary", t.Primary)
} }
// Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown // Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown
@@ -1874,11 +2042,15 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
return nil, 0, ferr return nil, 0, ferr
} }
out := make([]escrow.RetainedBlob, 0, len(resp.Packages)) out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
// R-304: every package the hub holds and the code will NOT be tried against — no key material,
// over the hub's cap, or malformed here. A refusal may call the code wrong only when this is 0.
withheld := resp.UnopenableCount + resp.TruncatedCount
for _, p := range resp.Packages { for _, p := range resp.Packages {
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64) blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
if derr != nil || len(blob) == 0 { if derr != nil || len(blob) == 0 {
// One malformed package must not sink the rest — the customer's code may open a // One malformed package must not sink the rest — the customer's code may open a
// later one, and a skipped entry is strictly better than a refusal we cannot justify. // later one, and a skipped entry is strictly better than a refusal we cannot justify.
withheld++
continue continue
} }
out = append(out, escrow.RetainedBlob{ out = append(out, escrow.RetainedBlob{
@@ -1888,9 +2060,13 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
Index: p.Index, Index: p.Index,
}) })
} }
return out, resp.UnopenableCount, nil return out, withheld, nil
}, },
} }
// R-894: ONE instance — the local API writes it, the host report reads it (2026-10-09: a restart right
// after the night backup erased the hub's evidence of it).
lastKnownBackups := backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json"))
collector.SetKnownBackupReporter(lastKnownBackups)
srv, err := localapi.NewServer(localapi.Options{ srv, err := localapi.NewServer(localapi.Options{
EscrowRecovery: escrowRecoverer, EscrowRecovery: escrowRecoverer,
ListenAddr: cfg.LocalAPI.ListenAddr, ListenAddr: cfg.LocalAPI.ListenAddr,
@@ -1901,10 +2077,14 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary
InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F) InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F)
Store: store, Store: store,
// R-894: the newest success per tier on disk — the due-check's fallback when the storage cannot be
// read right after a restart. Same state dir as restore-test-state.json.
LastKnownBackups: lastKnownBackups,
Storage: observer, Storage: observer,
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages) DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
Identity: storage.NewIdentityResolver(), // v0.157.0: `16` §3 identity on /disks + POST /disks/same-disk
Tokens: tokens, Tokens: tokens,
BackupCadence: cfg.Backup.BackupCadence(), BackupCadence: cfg.Backup.BackupCadence(),
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate. // Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
@@ -1928,6 +2108,8 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
// A1 (v0.62.0): the scan is restricted to felhom-pool members (ownership proven, not assumed). // A1 (v0.62.0): the scan is restricted to felhom-pool members (ownership proven, not assumed).
StaleLock: localapi.NewStaleLockController(px, &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, reconcile.DefaultPool, logger), StaleLock: localapi.NewStaleLockController(px, &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, reconcile.DefaultPool, logger),
Intent: intent, // slice 10 P3: record enroll/eject intent for self-heal Intent: intent, // slice 10 P3: record enroll/eject intent for self-heal
// v0.158.0 (felhom.eu `16` §5): the internal data place record, beside the intent store.
InternalPlaceFile: localapi.InternalPlaceFileIn(agentStateDir(cfg)),
GuestBinds: guestBinds, // F9: per-guest bind record for the startup re-assert GuestBinds: guestBinds, // F9: per-guest bind record for the startup re-assert
FormatJobs: formatJobs, // F20-BUG3: detached-format job record + restart recovery FormatJobs: formatJobs, // F20-BUG3: detached-format job record + restart recovery
@@ -1944,6 +2126,30 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
logger.Warn("daemon: local-api disabled (server build)", "err", err) logger.Warn("daemon: local-api disabled (server build)", "err", err)
return nil return nil
} }
// R-1006: a dead half-written vzdump (a power cut mid-backup) is removed at start-up and before every backup of the
// guest on that target, through the root helper; the in-flight answer is R-1008's (a task from before the boot is dead).
cleaner := &vzpartial.Cleaner{
API: px,
Runner: &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath},
InFlight: srv.VzdumpInFlight,
Boot: func() (int64, bool) {
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
return b, err == nil && b > 0
},
Logger: logger,
}
// R-1012: an orphan thin device (a full host root during a snapshot) is NAMED in the backup failure the controller
// mails to the operator, through the read-only root helper. Never removed — the repair stays a human step.
thinChecker := &thincheck.Checker{API: px, Runner: &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, Logger: logger}
for _, r := range tierRunners {
r.SetBeforeBackup(func(ctx context.Context, storage string, vmid int) { cleaner.Clean(ctx, storage, vmid) })
r.SetThinCheck(func(ctx context.Context, vmid int) string { return thincheck.Describe(thinChecker.Check(ctx, vmid)) })
}
srv.SetPartialSweep(func(ctx context.Context, vmid int) {
for _, t := range tierTargets {
cleaner.Clean(ctx, t, vmid)
}
})
return srv return srv
} }
@@ -2073,6 +2279,7 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger)
return 1 return 1
} }
observer := storage.NewObserver(px, storage.NewProcHostReader(), newHostOps(cfg, logger), logger) observer := storage.NewObserver(px, storage.NewProcHostReader(), newHostOps(cfg, logger), logger)
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
// Wire the LIVE PBS reporter here too (v0.39.0): selftest=hub is a separate one-shot process — no // Wire the LIVE PBS reporter here too (v0.39.0): selftest=hub is a separate one-shot process — no
// verify loop runs — so a nil reporter previously yielded pbs_snapshots:[] and an absent dr_recipe // verify loop runs — so a nil reporter previously yielded pbs_snapshots:[] and an absent dr_recipe
// pbs coord. The live reporter lists snapshots directly (fresh store, last-known-good fallback) so // pbs coord. The live reporter lists snapshots directly (fresh store, last-known-good fallback) so
@@ -2129,6 +2336,7 @@ func runSelftestStorage(ctx context.Context, cfg config.Config, logger *slog.Log
hostReader := storage.NewProcHostReader() hostReader := storage.NewProcHostReader()
hostOps := newHostOps(cfg, logger) hostOps := newHostOps(cfg, logger)
observer := storage.NewObserver(px, hostReader, hostOps, logger) observer := storage.NewObserver(px, hostReader, hostOps, logger)
observer.SetIdentity(storage.NewIdentityResolver()) // `16` §3 physical-disk identity
octx, cancel := context.WithTimeout(ctx, 60*time.Second) octx, cancel := context.WithTimeout(ctx, 60*time.Second)
fmt.Printf("=== felhom-agent %s selftest=storage ===\n", version) fmt.Printf("=== felhom-agent %s selftest=storage ===\n", version)
@@ -2261,7 +2469,7 @@ func runSelftestRestoreTestDue(ctx context.Context, cfg config.Config, logger *s
return 1 return 1
} }
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json")) rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, logger) sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, nil, logger)
fmt.Printf("eval_interval=%s settle=%s\n", cfg.Backup.RestoreTestEvalInterval(), cfg.Backup.RestoreTestSettle()) fmt.Printf("eval_interval=%s settle=%s\n", cfg.Backup.RestoreTestEvalInterval(), cfg.Backup.RestoreTestSettle())
start := time.Now() start := time.Now()
@@ -2687,7 +2895,20 @@ type escrowCeremonyOpts struct {
offline bool offline bool
upload bool upload bool
identityBundlePath string identityBundlePath string
directivePath string // deferUpload (R-978): with upload, build the hub upload body and hand it back in
// CeremonyOutput.PendingUpload instead of sending it. Set for json mode — the controller-driven
// ceremony, whose daemon uploads the package only when the customer claims the new code.
deferUpload bool
}
// escrowOptsFromCLI maps the escrow-create flags onto the ceremony options. json mode defers the upload
// (R-978, pinned by TestEscrowOptsFromCLI_JSONModeDefersUpload); text mode — an operator at a terminal,
// who sees R at once — keeps the immediate upload.
func escrowOptsFromCLI(storage string, paperkey, offline, upload bool, identityBundlePath string, jsonMode bool) escrowCeremonyOpts {
return escrowCeremonyOpts{
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
identityBundlePath: identityBundlePath, deferUpload: upload && jsonMode,
}
} }
// escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the // escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the
@@ -2746,11 +2967,10 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("PBS key for %q not found (%s): %v", storage, keyPath, err)} return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("PBS key for %q not found (%s): %v", storage, keyPath, err)}
} }
// Slice 10D.1: optionally ALSO wrap the identity bundle under the same R, and carry the non-secret // Slice 10D.1: optionally ALSO wrap the identity bundle under the same R. The bundle file is a 0600 secret
// directive for the hub. The bundle file is a 0600 secret (tunnel/pbs tokens); the directive is // (tunnel/pbs tokens). The non-secret "directive" that used to ride along is retired (R-105, `09` §3 decision 169:
// non-secret (pbs repo/ns, expected fingerprint, tunnel id). // nothing read it; the DR path reads the recipe, tenantsync and the escrow blob).
var identity *escrow.IdentityBundle var identity *escrow.IdentityBundle
var directive json.RawMessage
if opts.identityBundlePath != "" { if opts.identityBundlePath != "" {
raw, err := os.ReadFile(opts.identityBundlePath) raw, err := os.ReadFile(opts.identityBundlePath)
if err != nil { if err != nil {
@@ -2761,11 +2981,6 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("identity bundle is not valid JSON {tunnel_token,pbs_token}: %v", err)} return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("identity bundle is not valid JSON {tunnel_token,pbs_token}: %v", err)}
} }
identity = &b identity = &b
if opts.directivePath != "" {
if d, err := os.ReadFile(opts.directivePath); err == nil && json.Valid(d) {
directive = d
}
}
} }
// S3: auto-inject the offsite WG private key into the escrowed identity when the key file // S3: auto-inject the offsite WG private key into the escrowed identity when the key file
// exists — a NEW escrow run should always capture the live tunnel identity. Field NAME only // exists — a NEW escrow run should always capture the live tunnel identity. Field NAME only
@@ -2844,16 +3059,36 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
ResticPwSealed: resticStaged, ResticPwSealed: resticStaged,
} }
if opts.upload { if opts.upload {
if err := uploadEscrowBlob(ctx, cfg, res, directive, resticPwSHA256); err != nil { pending, uploaded, err := deliverEscrowPackage(ctx, cfg, res, resticPwSHA256, opts.deferUpload, putEscrowBody)
if err != nil {
// R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets // R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets
// the text shell keep the pre-extraction order (R surfaced, THEN the failure). // the text shell keep the pre-extraction order (R surfaced, THEN the failure).
return out, &escrowCeremonyErr{kind: "upload", err: err} return out, &escrowCeremonyErr{kind: "upload", err: err}
} }
out.Sum.Uploaded = true out.Sum.PendingUpload, out.Sum.Uploaded = pending, uploaded
} }
return out, nil return out, nil
} }
// deliverEscrowPackage builds the hub upload body and either sends it now (text mode) or returns it for
// the caller to send later (deferUpload — json mode, R-978: the daemon sends it when the customer claims
// the code). A hub that is not configured fails in both modes, as before: a package that can never be
// delivered is not a ceremony. put is the sender seam (putEscrowBody in production).
func deliverEscrowPackage(ctx context.Context, cfg config.Config, res escrow.CreateResult, resticPwSHA256 string, deferUpload bool,
put func(ctx context.Context, cfg config.Config, body []byte) error) (pending []byte, uploaded bool, err error) {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return nil, false, fmt.Errorf("hub not configured (url/host_id/api_key)")
}
body := escrowUploadBody(res, resticPwSHA256)
if deferUpload {
return body, false, nil
}
if err := put(ctx, cfg, body); err != nil {
return nil, false, err
}
return nil, true, nil
}
// printEscrowTextBanner prints the text-mode header line (shared by the success and the // printEscrowTextBanner prints the text-mode header line (shared by the success and the
// post-banner failure paths so the pre-extraction output stays byte-identical — including the // post-banner failure paths so the pre-extraction output stays byte-identical — including the
// posture: the historical banner always printed the DEFAULT posture, not the result's). // posture: the historical banner always printed the DEFAULT posture, not the result's).
@@ -2901,7 +3136,7 @@ func printEscrowTextRBlock(out *escrowCeremonyOutcome) {
// NOTHING else there; every human/info line goes to stderr; failures exit non-zero with no // NOTHING else there; every human/info line goes to stderr; failures exit non-zero with no
// partial JSON. This is the controller-driven ceremony's parse surface (spike §2.3: the text // partial JSON. This is the controller-driven ceremony's parse surface (spike §2.3: the text
// banner is positionally brittle). // banner is positionally brittle).
func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slog.Logger, storage string, paperkey, offline, upload bool, identityBundlePath, directivePath, outputMode string) int { func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slog.Logger, storage string, paperkey, offline, upload bool, identityBundlePath, outputMode string) int {
switch outputMode { switch outputMode {
case "", "text", "json": case "", "text", "json":
default: default:
@@ -2917,10 +3152,7 @@ func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slo
return 2 return 2
} }
out, cerr := escrowCeremony(ctx, cfg, logger, escrowCeremonyOpts{ out, cerr := escrowCeremony(ctx, cfg, logger, escrowOptsFromCLI(storage, paperkey, offline, upload, identityBundlePath, jsonMode))
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
identityBundlePath: identityBundlePath, directivePath: directivePath,
})
if cerr != nil { if cerr != nil {
switch cerr.kind { switch cerr.kind {
case "usage": case "usage":
@@ -2958,6 +3190,9 @@ func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slo
if out.Sum.Uploaded { if out.Sum.Uploaded {
fmt.Fprintln(os.Stderr, " uploaded the opaque blob(s) to the hub (host record); the hub cannot open them") fmt.Fprintln(os.Stderr, " uploaded the opaque blob(s) to the hub (host record); the hub cannot open them")
} }
if len(out.Sum.PendingUpload) > 0 {
fmt.Fprintln(os.Stderr, " package prepared, NOT uploaded: the agent uploads it when the new code is claimed (R-978)")
}
wire := out.Sum wire := out.Sum
wire.RecoveryCode = out.R wire.RecoveryCode = out.R
if err := json.NewEncoder(os.Stdout).Encode(wire); err != nil { if err := json.NewEncoder(os.Stdout).Encode(wire); err != nil {
@@ -3102,9 +3337,8 @@ type escrowUploadRequest struct {
BlobB64 string `json:"blob_b64"` // base64 of the opaque R-wrapped blob (ciphertext) BlobB64 string `json:"blob_b64"` // base64 of the opaque R-wrapped blob (ciphertext)
KeyFingerprint string `json:"key_fingerprint"` // for operator display only KeyFingerprint string `json:"key_fingerprint"` // for operator display only
Posture string `json:"posture"` // e.g. "zero_knowledge" Posture string `json:"posture"` // e.g. "zero_knowledge"
// Slice 10D.1 — optional DR bundle (identity escrow + non-secret directive). Omitted in slice-7. // Slice 10D.1 — optional identity escrow. Omitted in slice-7. (The `directive` is retired — R-105.)
IdentityBlobB64 string `json:"identity_blob_b64,omitempty"` IdentityBlobB64 string `json:"identity_blob_b64,omitempty"`
DirectiveJSON json.RawMessage `json:"directive,omitempty"`
CreatedAt string `json:"created_at"` // RFC3339 CreatedAt string `json:"created_at"` // RFC3339
// SLICE 3 — sha256 hex of the offsite restic repo password sealed in the identity blob (present only // SLICE 3 — sha256 hex of the offsite restic repo password sealed in the identity blob (present only
// when a staged password was folded in). Non-reversible hash of a 256-bit random secret — safe to // when a staged password was folded in). Non-reversible hash of a 256-bit random secret — safe to
@@ -3112,13 +3346,10 @@ type escrowUploadRequest struct {
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"` ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
} }
// uploadEscrowBlob PUTs the opaque blob (and, for 10D, the identity blob + non-secret directive) to // escrowUploadBody is the hub upload body for one ceremony's result: the opaque blob (and, for 10D, the
// the hub, authed with the per-host key. The hub stores ciphertext + non-secret fields; no usable // identity blob) plus non-secret fields. Ciphertext only — useless without R; no usable secret leaves
// secret leaves the agent. // the agent. putEscrowBody sends it, authed with the per-host key.
func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateResult, directive json.RawMessage, resticPwSHA256 string) error { func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)")
}
upReq := escrowUploadRequest{ upReq := escrowUploadRequest{
BlobB64: base64.StdEncoding.EncodeToString(res.Blob), BlobB64: base64.StdEncoding.EncodeToString(res.Blob),
KeyFingerprint: res.KeyFingerprint, KeyFingerprint: res.KeyFingerprint,
@@ -3128,9 +3359,17 @@ func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateR
} }
if len(res.IdentityBlob) > 0 { if len(res.IdentityBlob) > 0 {
upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob) upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob)
upReq.DirectiveJSON = directive
} }
body, _ := json.Marshal(upReq) body, _ := json.Marshal(upReq)
return body
}
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)")
}
url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow" url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow"
req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewReader(body)) req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewReader(body))
if err != nil { if err != nil {
@@ -3795,3 +4034,18 @@ func (f *factsReporter) SystemFacts(ctx context.Context) (int, json.RawMessage,
f.facts, f.err = f.leg.Facts(ctx, f.vmid) f.facts, f.err = f.leg.Facts(ctx, f.vmid)
return f.vmid, f.facts, f.err return f.vmid, f.facts, f.err
} }
// agentStateDir is the agent's durable state dir: beside the nonce store when one is configured, else the default.
// The intent store, the guest-bind record and the internal place record (v0.158.0) all live here.
func agentStateDir(cfg config.Config) string {
if cfg.Authz.NonceStorePath != "" {
return filepath.Dir(cfg.Authz.NonceStorePath)
}
return "/var/lib/felhom-agent"
}
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
}
@@ -0,0 +1,91 @@
package main
import (
"context"
"encoding/json"
"io"
"log/slog"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
)
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
type orderRunner struct {
mu sync.Mutex
order []string
}
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
var p map[string]any
b, _ := os.ReadFile(args[1])
_ = json.Unmarshal(b, &p)
mode, _ := p["mode"].(string)
r.order = append(r.order, "os-apply:"+mode)
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
}
if len(args) >= 1 {
r.order = append(r.order, filepath.Base(name)+":"+args[0])
}
return []byte("ok"), nil, nil
}
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
t.Helper()
stateDir := t.TempDir()
if marker != nil {
dir := filepath.Join(stateDir, "selfupdate")
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
b, _ := json.Marshal(marker)
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
t.Fatal(err)
}
}
r := &orderRunner{}
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
mgr.MaybeCommit(context.Background())
r.mu.Lock()
defer r.mu.Unlock()
return append([]string(nil), r.order...)
}
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
}
}
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
}
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 0 {
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
}
}
+105
View File
@@ -0,0 +1,105 @@
package main
import (
"go/ast"
"testing"
)
// R-894 — the on-disk backup record is WIRED on the daemon path (the built-but-never-wired class).
// main → runDaemon → buildLocalAPIServer, and inside it the localapi.Options literal carries
// LastKnownBackups built by backup.NewBackupSuccessState. An AST walk, not a string match, for the
// reasons in escrow_recover_wiring_test.go.
//
// COMPANION RED-PROOF (observed): delete the `LastKnownBackups:` line from buildLocalAPIServer → this
// fails with "localapi.Options in buildLocalAPIServer has no LastKnownBackups field". Restored.
//
// 2026-10-09: the SAME instance also feeds the host report (collector.SetKnownBackupReporter), so a
// restart right after a backup no longer erases the hub's evidence of it. The field may name a local
// variable; the variable must be built by backup.NewBackupSuccessState and be the one handed to the
// collector. RED-PROOF (observed): drop the SetKnownBackupReporter call → "not handed to the collector".
func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
if !callsWithin(f, "main")["runDaemon"] || !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
t.Fatal("main → runDaemon → buildLocalAPIServer is broken — the path this test asserts is not the live one")
}
var field, built bool
var fieldVar, handed string
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
cl, ok := n.(*ast.CompositeLit)
if !ok {
return true
}
sel, ok := cl.Type.(*ast.SelectorExpr)
if !ok {
return true
}
if pkg, _ := sel.X.(*ast.Ident); pkg == nil || pkg.Name+"."+sel.Sel.Name != "localapi.Options" {
return true
}
for _, el := range cl.Elts {
kv, ok := el.(*ast.KeyValueExpr)
if !ok {
continue
}
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "LastKnownBackups" {
field = true
if callsIn(kv.Value)["backup.NewBackupSuccessState"] {
built = true
}
if id, ok := kv.Value.(*ast.Ident); ok {
fieldVar = id.Name
}
}
}
return true
})
// a local variable: built by NewBackupSuccessState, and handed to the collector
ast.Inspect(fd.Body, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.AssignStmt:
for i, l := range x.Lhs {
if id, ok := l.(*ast.Ident); ok && fieldVar != "" && id.Name == fieldVar && i < len(x.Rhs) &&
callsIn(x.Rhs[i])["backup.NewBackupSuccessState"] {
built = true
}
}
case *ast.CallExpr:
if fn, ok := x.Fun.(*ast.SelectorExpr); ok && fn.Sel.Name == "SetKnownBackupReporter" && len(x.Args) == 1 {
if id, ok := x.Args[0].(*ast.Ident); ok {
handed = id.Name
}
}
}
return true
})
}
if !field {
t.Fatal("localapi.Options in buildLocalAPIServer has no LastKnownBackups field")
}
if !built {
t.Fatal("LastKnownBackups is not built by backup.NewBackupSuccessState")
}
if handed == "" || handed != fieldVar {
t.Fatalf("the saved backups (%q) are not handed to the collector (SetKnownBackupReporter got %q)", fieldVar, handed)
}
}
func callsIn(n ast.Node) map[string]bool {
out := map[string]bool{}
ast.Inspect(n, func(n ast.Node) bool {
if ce, ok := n.(*ast.CallExpr); ok {
if fn, ok := ce.Fun.(*ast.SelectorExpr); ok {
if x, ok := fn.X.(*ast.Ident); ok {
out[x.Name+"."+fn.Sel.Name] = true
}
}
}
return true
})
return out
}
+5 -1
View File
@@ -43,7 +43,7 @@ func main() {
func run() error { func run() error {
var ( var (
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | agent_config_update") op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update | add_internal_place")
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)") host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)") guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)") keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
@@ -123,6 +123,10 @@ func run() error {
now := time.Now().UTC() now := time.Now().UTC()
issued := now issued := now
if *ttl <= 0 || *ttl > authz.MaxWindow {
fmt.Fprintf(os.Stderr, "felhom-opsign: -ttl %s refused: a box accepts a window of at most %s (R-1280)\n", *ttl, authz.MaxWindow)
os.Exit(2)
}
expires := now.Add(*ttl) expires := now.Add(*ttl)
// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over. // Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
+35 -5
View File
@@ -25,6 +25,7 @@
Cmnd_Alias FELHOM_MOUNT = \ Cmnd_Alias FELHOM_MOUNT = \
/usr/local/sbin/felhom-priv-apply ^unit mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$, \ /usr/local/sbin/felhom-priv-apply ^unit mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$, \
/usr/local/sbin/felhom-priv-apply ^unit-forget mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl daemon-reload, \ /usr/bin/systemctl daemon-reload, \
/usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \ /usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \ /usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
@@ -80,7 +81,34 @@ Cmnd_Alias FELHOM_DNSMASQ = \
Cmnd_Alias FELHOM_GUESTHOOK = \ Cmnd_Alias FELHOM_GUESTHOOK = \
/usr/sbin/pct ^set [0-9]+ --hookscript local\:snippets/felhom-guest-hook\.sh$, \ /usr/sbin/pct ^set [0-9]+ --hookscript local\:snippets/felhom-guest-hook\.sh$, \
/usr/sbin/pct ^set [0-9]+ --delete mp[0-9]+$, \ /usr/sbin/pct ^set [0-9]+ --delete mp[0-9]+$, \
/usr/sbin/pct ^reboot [0-9]+$ /usr/sbin/pct ^reboot [0-9]+$, \
/usr/sbin/pct ^shutdown [0-9]+ --timeout [0-9]+$
# Host power-off (R-998, doc 14): the household's „A gép kikapcsolása" — guest shutdown above, then this one fixed
# vector. Guarded agent-side (10 min after boot, one per boot, never during a backup/restore/update/drive job).
Cmnd_Alias FELHOM_POWEROFF = \
/usr/bin/systemctl poweroff
# Dead half-written vzdump removal (R-1006, power-cut-fixes-2026-10-14): a power cut mid-backup leaves
# vzdump-lxc-<vmid>-<date>.tar.dat + .tmp/ on the dir storage; nothing removed them and every later backup was skipped
# „not enough space". ONE root helper (configs/felhom-vzdump-partial-clean, from the config bundle) re-checks everything
# as root — a dir storage from storage.cfg, the exact name of THAT vmid, not in use, dead — and refuses otherwise. The
# arguments here are its exact shape (storage id, vmid, one half-written name); the vmid-in-name match is the helper's.
Cmnd_Alias FELHOM_VZPARTIAL = \
/usr/local/sbin/felhom-vzdump-partial-clean ^[A-Za-z][A-Za-z0-9_-]{0\,63} [1-9][0-9]{2\,8} vzdump-(lxc|qemu)-[1-9][0-9]{2\,8}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$
# Internal data place („Belső tárhely", felhom.eu architecture/16 §5, §7; agent v0.158.0). ONE root helper
# (configs/felhom-internal-place, from the config bundle) creates <vg>/felhom-belso in the local-lvm thin pool by the
# ruled size rule and formats ONLY that volume, ONLY when it carries no signature. The agent calls it from the signed
# `add_internal_place` job (an old box), always with 0 GiB planned: its guest's volumes already exist. `plan` changes
# nothing. The installer runs the helper as root itself (a new box).
# R-1012 (agent v0.159.0): the READ-ONLY orphan-thin check before and after a whole-guest backup — lists the pool's thin
# device ids (through a metadata snapshot it reserves and releases) next to LVM's. It sends no other pool message; the
# repair stays a human step. Pinned by configs/test_felhom_thin_check.py.
Cmnd_Alias FELHOM_THINCHECK = \
/usr/local/sbin/felhom-thin-check ^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0\,63} [A-Za-z0-9_+][A-Za-z0-9_.+-]{0\,63}$
Cmnd_Alias FELHOM_INTERNALPLACE = \
/usr/local/sbin/felhom-internal-place ^(plan|create) 0$
# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent # Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
# /mnt/felhom-drives (self-bind + make-shared) and binds/unbinds each drive's felhom-data namespace UNDERNEATH it so the # /mnt/felhom-drives (self-bind + make-shared) and binds/unbinds each drive's felhom-data namespace UNDERNEATH it so the
@@ -111,15 +139,17 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template # docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
# + container across spaces, spike-confirmed) # + container across spaces, spike-confirmed)
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change) # systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
# tee <FIXED image file> — WRITE the ref; content is fed on STDIN (no shell, no interpolation), # felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
# the agent strict-validates the ref (controllerImageRe) before the write. # STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
# compromised agent could hand the guest's bootstrap ANY image (sudo cannot see stdin).
# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md. # Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
Cmnd_Alias FELHOM_CONTROLLERSWAP = \ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/felhom-controller-image$, \ /usr/sbin/pct ^exec [0-9]+ -- cat /etc/felhom-controller-image$, \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \ /usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \ /usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \ /usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$ /usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a # Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The # `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
@@ -312,4 +342,4 @@ Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \ /usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \
/usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$ /usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY, FELHOM_POWEROFF, FELHOM_VZPARTIAL, FELHOM_INTERNALPLACE, FELHOM_THINCHECK
+39
View File
@@ -0,0 +1,39 @@
#!/bin/sh
# /etc/grub.d/42_felhom_oneshot — the kernel lane's one-shot ENTRIES (R-836, `09` §3 decision 172, `11` §5.11).
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it.
#
# One menu entry per installed Proxmox kernel, id `felhom-oneshot-<version>`, booted ONLY when 01_felhom_oneshot found
# the flag naming it. It is the normal entry plus option C (decision 172): softlockup_panic=1 hardlockup_panic=1
# hung_task_panic=1 panic=10 — a lockup the kernel can detect becomes a panic, and a panic restarts the box in 10 s into
# the default (the old kernel). A true dead freeze still needs a person (spike candidate 3 failed on all three boxes).
# It sorts AFTER 10_linux, so it is never entry 0 and never the default.
#
# No vfat ESP at /boot/efi → prints nothing (no flag can name these entries).
set -e
prefix="/usr"
exec_prefix="/usr"
datarootdir="/usr/share"
. "$datarootdir/grub/grub-mkconfig_lib"
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
[ -n "$esp_uuid" ] || exit 0
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
[ -n "$kernels" ] || exit 0
case "${GRUB_DEVICE}" in
/dev/mapper/*|/dev/dm-*|"") root_arg="root=${GRUB_DEVICE}" ;;
*) if [ -n "${GRUB_DEVICE_UUID}" ]; then root_arg="root=UUID=${GRUB_DEVICE_UUID}"; else root_arg="root=${GRUB_DEVICE}"; fi ;;
esac
[ -n "${GRUB_DEVICE}" ] || root_arg="root=$(findmnt -n -o SOURCE /)"
rel=$(make_system_path_relative_to_its_root /boot)
prep=$(prepare_grub_to_access_device "$(${grub_probe:-grub-probe} --target=device /boot)" | sed 's/^/ /')
for k in $kernels; do
[ -f "/boot/initrd.img-$k" ] || continue
cat <<EOF
menuentry 'Felhom one-shot: $k' --class proxmox --id felhom-oneshot-$k {
insmod gzio
$prep
echo 'Loading Linux $k (felhom one-shot) ...'
linux $rel/vmlinuz-$k $root_arg ro ${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
initrd $rel/initrd.img-$k
}
EOF
done
+36
View File
@@ -0,0 +1,36 @@
#!/bin/sh
# /etc/grub.d/01_felhom_oneshot — the kernel lane's ONE-SHOT boot (R-836, `09` §3 decisions 164 + 172, `11` §5.11).
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it; it prints GRUB script.
#
# At boot, GRUB reads `felhom_next` from an environment block on the ESP (vfat — GRUB can rewrite a file there; it
# cannot on the LVM /boot, R-836), CLEARS it, and — only if it names an installed kernel — boots that kernel's one-shot
# entry (42_felhom_oneshot) instead of the default. The next boot uses the default again whatever happens: a new kernel
# that panics comes back on the old one by itself. felhom-os-apply writes the flag (mode apply) and never the default
# for a new kernel. Measured on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
# `audits/kernel-spike-2026-10-07/` (candidate 2).
#
# No vfat ESP at /boot/efi, or no Proxmox kernel → prints nothing (the box boots exactly as before).
set -e
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
[ -n "$esp_uuid" ] || exit 0
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
[ -n "$kernels" ] || exit 0
cat <<EOF
# felhom kernel lane: a one-shot kernel named on the ESP, read and cleared before the menu
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=felhom_esp $esp_uuid
if [ -f (\$felhom_esp)/EFI/felhom/oneshot.env ]; then
load_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
if [ "\${felhom_next}" ]; then
set felhom_boot="\${felhom_next}"
set felhom_next=
save_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
EOF
for k in $kernels; do
printf ' if [ "${felhom_boot}" = "%s" ]; then set default="felhom-oneshot-%s"; fi\n' "$k" "$k"
done
cat <<EOF
fi
fi
EOF
+118
View File
@@ -0,0 +1,118 @@
#!/bin/bash
# felhom-internal-place — make the one-disk box's internal data place („Belső tárhely"), felhom.eu
# architecture/16-storage-physical-identity.md §5, §7 (sizes ruled by the operator 2026-10-16). Installed as
# /usr/local/sbin/felhom-internal-place (config bundle). Called by felhom-host-install.sh (a new box) and by the agent's
# signed `add_internal_place` job (an old box) — ONE implementation of the size rule and of the only mkfs it does.
#
# felhom-internal-place plan <planned GiB> → prints the numbers, changes nothing
# felhom-internal-place create <planned GiB> → makes the place (or finds it), prints the result
#
# <planned GiB> is thin space still to be promised to the guest (rootfs + data volume) that does not exist yet:
# the installer passes 32 + the data volume; an old box passes 0 (its guest's volumes are already counted).
#
# The rule (16 §7, ruled): reserve = max(16 GiB, 10 % of the pool), at most 48 GiB; internal = pool − reserve −
# every thin volume already promised in the pool − <planned>, made only when ≥ 20 GiB. So nothing in the pool is ever
# promised beyond pool − reserve: a pool at 100 % stops every volume on the box.
#
# It trusts nothing it is told. Every rule is checked here, as root, and any doubt REFUSES (exit 1):
# - the pool is the `lvmthin: local-lvm` section of /etc/pve/storage.cfg (its vgname/thinpool), nothing else;
# - the only volume it creates, formats or reads is <vg>/felhom-belso;
# - it formats ONLY a felhom-belso that carries no signature at all (blkid -p finds nothing) — the one it just made,
# or one a crash left between lvcreate and mkfs; any other content REFUSES; it never resizes or removes anything.
# Output, one line: „created uuid=<fs-uuid> gib=<n>", „formatted uuid=… gib=…" (a crash left it empty), „existing uuid=<fs-uuid> gib=<n>", „skipped <why>" (exit 0),
# or „plan pool=… reserve=… promised=… planned=… internal=…". Pinned by configs/test_felhom_internal_place.py.
set -u
STORAGE_CFG=${FELHOM_STORAGE_CFG:-/etc/pve/storage.cfg}
LV_NAME=felhom-belso
LABEL=felhom-belso
MIN_GIB=20
die() { echo "felhom-internal-place: REFUSED: $*" >&2; exit 1; }
pool_of_local_lvm() { # prints "<vg> <thinpool>" of the lvmthin storage local-lvm, or nothing
awk '
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (type == "lvmthin" && id == "local-lvm"); next }
in_s && /^[ \t]+vgname[ \t]/ { vg = $2 }
in_s && /^[ \t]+thinpool[ \t]/ { tp = $2 }
END { if (vg != "" && tp != "") print vg, tp }
' "$STORAGE_CFG"
}
gib_floor() { echo $(( $1 / 1073741824 )); }
[[ $# -eq 2 ]] || die "usage: felhom-internal-place plan|create <planned GiB>"
MODE=$1; PLANNED=$2
[[ "$MODE" == plan || "$MODE" == create ]] || die "mode must be plan or create"
[[ "$PLANNED" =~ ^(0|[1-9][0-9]{0,5})$ ]] || die "planned GiB must be a whole number"
read -r VG POOL < <(pool_of_local_lvm) || true
[[ -n "${VG:-}" && -n "${POOL:-}" ]] || die "no lvmthin storage local-lvm in $STORAGE_CFG"
[[ "$VG" =~ ^[A-Za-z0-9_.+-]+$ && "$POOL" =~ ^[A-Za-z0-9_.+-]+$ ]] || die "unexpected vg/pool name"
pool_b=$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$POOL" 2>/dev/null | tr -d ' ')
[[ "$pool_b" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$POOL"
pool_gib=$(gib_floor "$pool_b")
# Every thin volume promised in the pool (its virtual size) — except the place itself and vzdump's short-lived
# snapshots (snap_*_vzdump), which exist only during a whole-box copy and are what the reserve is for.
# FAIL CLOSED: a failed or empty listing must never read as „nothing promised" (that would size the place too big).
listing=$(lvs --noheadings --nosuffix --units b -o lv_name,pool_lv,lv_size "$VG" 2>/dev/null) || die "cannot list the volumes of $VG"
pool_seen=false; promised_b=0; have_lv=false
while read -r name pl size; do
[[ -n "${name:-}" ]] || continue
[[ "$name" == "$POOL" ]] && pool_seen=true
[[ "$pl" == "$POOL" ]] || continue
if [[ "$name" == "$LV_NAME" ]]; then have_lv=true; continue; fi
[[ "$name" =~ ^snap_.*_vzdump$ ]] && continue
[[ "$size" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$name"
promised_b=$(( promised_b + size ))
done <<< "$listing"
$pool_seen || die "the listing of $VG does not contain the pool $POOL — refusing to guess what is promised"
promised_gib=$(( (promised_b + 1073741823) / 1073741824 )) # rounded UP: never promise a partial GiB twice
reserve=$(( pool_gib / 10 )); (( reserve < 16 )) && reserve=16; (( reserve > 48 )) && reserve=48
internal=$(( pool_gib - reserve - promised_gib - PLANNED ))
if [[ "$MODE" == plan ]]; then
echo "plan pool=$pool_gib reserve=$reserve promised=$promised_gib planned=$PLANNED internal=$internal exists=$have_lv"
exit 0
fi
DEV=/dev/$VG/$LV_NAME
is_block() { [[ "$(stat -L -c %F "$1" 2>/dev/null)" == "block special file" ]]; }
fs_uuid() { blkid -p -s UUID -o value "$DEV" 2>/dev/null; }
fs_type() { blkid -p -s TYPE -o value "$DEV" 2>/dev/null; }
fs_label() { blkid -p -s LABEL -o value "$DEV" 2>/dev/null; }
# has_signature: FAIL CLOSED. blkid -p exits 0 when it found something and 2 when it found NOTHING; any other result (an
# error, an ambivalent probe, blkid missing) must never read as „empty" — that is the only gate in front of mkfs.
has_signature() {
local rc=0
blkid -p -o value "$DEV" >/dev/null 2>&1 || rc=$?
case $rc in
0) return 0 ;;
2) return 1 ;;
*) die "cannot probe $DEV for a signature (blkid exit $rc) — refusing" ;;
esac
}
lv_gib() { gib_floor "$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$LV_NAME" 2>/dev/null | tr -d ' ')"; }
if $have_lv; then
is_block "$DEV" || die "$DEV is not an active block device"
t=$(fs_type)
if [[ -n "$t" ]]; then
[[ "$t" == ext4 && "$(fs_label)" == "$LABEL" ]] || die "$DEV holds something that is not Felhom's ($t) — left alone"
echo "existing uuid=$(fs_uuid) gib=$(lv_gib)"; exit 0
fi
# A felhom-belso with NO signature: a crash between lvcreate and mkfs. Nothing can be on it — format it.
has_signature && die "$DEV carries a signature blkid cannot name — left alone"
else
(( internal >= MIN_GIB )) || { echo "skipped only $internal GiB free for it (pool $pool_gib, reserve $reserve, promised $promised_gib, planned $PLANNED; needs $MIN_GIB)"; exit 0; }
lvcreate -q -y -V "${internal}G" -T "$VG/$POOL" -n "$LV_NAME" >/dev/null || die "lvcreate failed"
is_block "$DEV" || die "$DEV did not appear after lvcreate"
has_signature && die "the new $DEV already carries a signature — left alone"
made=created
fi
mkfs.ext4 -q -L "$LABEL" "$DEV" >/dev/null || die "mkfs.ext4 failed on $DEV"
u=$(fs_uuid)
[[ "$u" =~ ^[0-9a-fA-F-]{36}$ ]] || die "no filesystem UUID after mkfs on $DEV"
echo "${made:-formatted} uuid=$u gib=$(lv_gib)"
+15 -3
View File
@@ -58,14 +58,26 @@ fi
# 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under # 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under
# /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed. # /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed.
# Agent v0.158.0: read from the kernel's mount table for EVERY node of the disk (lsblk's MOUNTPOINT shows one mount
# per node; a bind elsewhere hid behind it), and a disk whose nodes cannot be listed REFUSES (it passed before).
# Deliberately NO exemption for a sibling that is Felhom's own enrolled drive (raw /mnt/<name> + /mnt/felhom-drives):
# the only corroboration available here is a mount the agent itself can make, so trusting it would let a compromised
# agent unlock a host owner's disk. Consequence (felhom.eu `16` §13): the free part next to an in-use Felhom drive
# cannot be formatted; the dashboard does not offer it.
PROC_MOUNTS=/proc/mounts
node_mounts() { awk -v d="$1" '$1 == d { print $2 }' "$PROC_MOUNTS"; }
disk_nodes="$(lsblk -nrpo NAME "$whole" 2>/dev/null || true)"
[[ -n "$disk_nodes" ]] || die "cannot list the nodes of $whole — refusing"
while read -r node; do
[[ -z "$node" ]] && continue
while read -r mp; do while read -r mp; do
[[ -z "$mp" ]] && continue [[ -z "$mp" ]] && continue
case "$mp" in case "$mp" in
/mnt/felhom-drives|/mnt/felhom-drives/*) : ;; /mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
*) die "device (or a partition) is mounted at $mp ($dev)" ;; *) die "device (or a partition) is mounted at $mp ($node)" ;;
esac esac
done < <(lsblk -nro MOUNTPOINT "$whole" 2>/dev/null || true) done < <(node_mounts "$node")
done <<< "$disk_nodes"
# 4) read-only device (audit D1): a device the kernel marks RO is never a formattable data disk. # 4) read-only device (audit D1): a device the kernel marks RO is never a formattable data disk.
wbase="${whole#/dev/}" wbase="${whole#/dev/}"
rof="/sys/block/$wbase/ro" rof="/sys/block/$wbase/ro"
+5 -3
View File
@@ -16,8 +16,10 @@ Cmnd_Alias FELHOM_OP_REPAIR = \
/usr/bin/systemctl reset-failed felhom-sshd, \ /usr/bin/systemctl reset-failed felhom-sshd, \
/usr/bin/systemctl restart felhom-sshd, \ /usr/bin/systemctl restart felhom-sshd, \
/usr/sbin/pct list, \ /usr/sbin/pct list, \
/usr/sbin/pct start [0-9]*, \ /usr/sbin/pct ^start [0-9]+$, \
/usr/sbin/pct stop [0-9]*, \ /usr/sbin/pct ^stop [0-9]+$, \
/usr/sbin/pct unlock [0-9]* /usr/sbin/pct ^unlock [0-9]+$
# R-861 (b) B2 (`09` §3 decision 165, hygiene): one numeric vmid per pct verb, anchored — the old glob `[0-9]*` also
# matched spaces, so `pct stop 9201 --skiplock 1` passed. Pinned by TestFelhomOpSudoersPctIsExact.
felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR
+844 -28
View File
File diff suppressed because it is too large. Load diff
+128 -1
View File
@@ -13,10 +13,14 @@ longer name the destination: each verb has a fixed source and a fixed (or strict
Verbs (each one sudoers line, exact-match pattern): Verbs (each one sudoers line, exact-match pattern):
unit <name> /var/lib/felhom-agent/units/<name> -> /etc/systemd/system/<name> (.mount | .automount) unit <name> /var/lib/felhom-agent/units/<name> -> /etc/systemd/system/<name> (.mount | .automount)
unit-forget <name> REMOVES /etc/systemd/system/<name> (R-1124): only a felhom by-UUID drive .mount unit that is
disabled, not mounted, and whose drive is absent (its /dev/disk/by-uuid link is gone)
dnsmasq <tmp> <name> /tmp/felhom-resolver-<digits>.conf -> /etc/dnsmasq.d/felhom-<...>.conf dnsmasq <tmp> <name> /tmp/felhom-resolver-<digits>.conf -> /etc/dnsmasq.d/felhom-<...>.conf
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600) wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
sshd-key /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op -> /etc/felhom-sshd/authorized_keys/felhom-op sshd-key /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op -> /etc/felhom-sshd/authorized_keys/felhom-op
controller-image <vmid> the ref on STDIN -> /etc/felhom-controller-image INSIDE guest <vmid> (R-861 (a) A1): only
our registry + our repository + an x.y.z tag; the agent no longer has a `tee` grant
--self-check prints "felhom-priv-apply ok verbs=..." (the bundle's self-check) --self-check prints "felhom-priv-apply ok verbs=..." (the bundle's self-check)
Exit codes: 0 installed (or already identical), 2 usage, 3 refused (content or source), 4 install failed. Exit codes: 0 installed (or already identical), 2 usage, 3 refused (content or source), 4 install failed.
@@ -39,9 +43,21 @@ WG_SRC, WG_DEST = STATE + "/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config" SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op" KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
MAX_BYTES = 64 * 1024 MAX_BYTES = 64 * 1024
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key") VERBS = ("unit", "unit-forget", "dnsmasq", "wg", "sshd-config", "sshd-key", "controller-image")
# R-861 (a) A1 (`09` §3 decision 165): the SAME pattern as the agent's controllerImageRe (internal/localapi/
# controllerswap.go) — a compromised agent cannot hand the guest's bootstrap any other image. Pinned by
# configs/test_felhom_priv_apply.py ControllerImage.
CONTROLLER_IMAGE_RE = re.compile(r"^gitea\.dooplex\.hu/admin/felhom-controller:[0-9]+\.[0-9]+\.[0-9]+$")
CONTROLLER_IMAGE_FILE = "/etc/felhom-controller-image"
CONTROLLER_IMAGE_MAX = 256
VMID_RE = re.compile(r"^[0-9]{1,9}$")
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$") UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
DRIVE_UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.mount$")
DRIVE_UNIT_MARKER = "Managed by felhom-agent"
NET_UNIT_MARKER = "felhom network storage"
WANTS_DIR = UNIT_DIR + "/multi-user.target.wants"
BY_UUID_DIR = "/dev/disk/by-uuid"
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$") DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
DNSMASQ_NAME_RE = re.compile(r"^felhom-[a-z0-9][a-z0-9._-]*\.conf$") DNSMASQ_NAME_RE = re.compile(r"^felhom-[a-z0-9][a-z0-9._-]*\.conf$")
SEG = r"[A-Za-z0-9_-][A-Za-z0-9_.-]*" SEG = r"[A-Za-z0-9_-][A-Za-z0-9_.-]*"
@@ -123,6 +139,46 @@ class Host:
pass pass
raise raise
def read_stdin(self, limit):
return sys.stdin.buffer.read(limit + 1)
# ---- unit-forget (R-1124) ----
def read_installed(self, path):
"""The installed unit: None when absent; Refused when it is not a regular file (a symlink is never followed)."""
try:
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC)
except FileNotFoundError:
return None
except OSError as e:
raise Refused("F1", f"cannot open {path}: {e.strerror}")
with os.fdopen(fd, "rb") as f:
if not stat.S_ISREG(os.fstat(f.fileno()).st_mode):
raise Refused("F1", f"{path} is not a regular file")
return f.read(MAX_BYTES + 1)
def lexists(self, path):
return os.path.lexists(path)
def mountpoints(self):
"""Every mount point in /proc/self/mountinfo (field 5, octal escapes decoded)."""
out = set()
with open("/proc/self/mountinfo", "r", encoding="utf-8", errors="replace") as f:
for line in f:
parts = line.split(" ")
if len(parts) > 4:
out.add(re.sub(r"\\([0-7]{3})", lambda m: chr(int(m.group(1), 8)), parts[4]))
return out
def remove(self, path):
os.remove(path)
def write_guest_image(self, vmid, data):
"""As root: `pct exec <vmid> -- tee <the fixed file>` with the checked ref on stdin (no shell)."""
r = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", CONTROLLER_IMAGE_FILE],
input=data, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=60)
if r.returncode != 0:
raise OSError(f"pct exec {vmid} tee exited {r.returncode}: {r.stderr.decode(errors='replace').strip()[:200]}")
def log(self, line): def log(self, line):
print(line, file=sys.stderr) print(line, file=sys.stderr)
try: try:
@@ -377,6 +433,73 @@ def plan(argv):
raise Refused("A1", f"wrong arguments for {v}") raise Refused("A1", f"wrong arguments for {v}")
def controller_image(rest, host):
"""R-861 (a) A1: read the ref on stdin, check it, write it INSIDE the guest as root."""
try:
if len(rest) != 1 or not VMID_RE.match(rest[0]):
raise Refused("A1", "usage: felhom-priv-apply controller-image <vmid> (the ref on stdin)")
raw = host.read_stdin(CONTROLLER_IMAGE_MAX)
if len(raw) > CONTROLLER_IMAGE_MAX:
raise Refused("I1", f"the image ref is longer than {CONTROLLER_IMAGE_MAX} bytes")
try:
text = raw.decode("ascii")
except UnicodeDecodeError:
raise Refused("I1", "the image ref is not ASCII")
ref = text[:-1] if text.endswith("\n") else text
if not CONTROLLER_IMAGE_RE.match(ref) or "\n" in ref:
raise Refused("I1", "the image ref is not gitea.dooplex.hu/admin/felhom-controller:<x.y.z>")
except Refused as e:
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] controller-image {' '.join(rest)[:40]}: {e.reason}")
return 2 if e.rule == "A1" else 3
vmid = int(rest[0])
try:
host.write_guest_image(vmid, (ref + "\n").encode())
except (OSError, subprocess.SubprocessError) as e:
host.log(f"felhom-priv-apply: FAILED controller-image {vmid}: {e}")
return 4
host.log(f"felhom-priv-apply: WROTE controller-image {vmid} {ref}")
return 0
def unit_forget(rest, host):
"""R-1124: remove ONE felhom drive mount unit that the agent has already disabled, whose drive is gone. A drive that
is still present or still mounted keeps its unit — this verb can only forget, never unmount or take a drive away."""
try:
if len(rest) != 1 or not DRIVE_UNIT_NAME_RE.match(rest[0]) or "/" in rest[0]:
raise Refused("A1", "usage: felhom-priv-apply unit-forget mnt-<escaped path>.mount")
name = rest[0]
path = os.path.join(UNIT_DIR, name)
data = host.read_installed(path)
if data is None:
host.log(f"felhom-priv-apply: ABSENT unit-forget {path} (nothing to remove)")
return 0
text = text_of(data, path)
if DRIVE_UNIT_MARKER not in text or NET_UNIT_MARKER in text:
raise Refused("F2", f"{name} is not a felhom drive unit")
check_unit(name, text)
s = parse_ini(text, name)
what, where = s["Mount"].get("What", ""), s["Mount"]["Where"]
mu = re.match(r"^/dev/disk/by-uuid/(.+)$", what)
if not mu:
raise Refused("F2", f"{name} is not a by-UUID drive unit")
if host.lexists(os.path.join(WANTS_DIR, name)):
raise Refused("F3", f"{name} is still enabled (disable it first)")
if where in host.mountpoints():
raise Refused("F4", f"{where} is mounted")
if host.lexists(os.path.join(BY_UUID_DIR, mu.group(1))):
raise Refused("F5", f"the drive of {name} is present (filesystem UUID {mu.group(1)})")
except Refused as e:
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] unit-forget {' '.join(rest)[:120]}: {e.reason}")
return 2 if e.rule == "A1" else 3
try:
host.remove(path)
except OSError as e:
host.log(f"felhom-priv-apply: FAILED unit-forget {path}: {e}")
return 4
host.log(f"felhom-priv-apply: REMOVED unit-forget {path} (where={where})")
return 0
def main(argv, host=None): def main(argv, host=None):
host = host or Host() host = host or Host()
if argv == ["--self-check"]: if argv == ["--self-check"]:
@@ -396,6 +519,10 @@ def main(argv, host=None):
return 3 return 3
print("OK") print("OK")
return 0 return 0
if argv and argv[0] == "controller-image":
return controller_image(argv[1:], host)
if argv and argv[0] == "unit-forget":
return unit_forget(argv[1:], host)
try: try:
verb, src, dest, mode, checker = plan(argv) verb, src, dest, mode, checker = plan(argv)
data = host.read_source(src) data = host.read_source(src)
+7
View File
@@ -0,0 +1,7 @@
# MANAGED BY FELHOM (the agent's config bundle) — R-982, operator ruling D4 (2026-10-22).
# The Proxmox host's OWN sshd — the listener the household network reaches on port 22: root logs in with a KEY only.
# The root password still works on the console (the break-glass password: felhom.eu runbooks/break-glass.md). The
# operator's door (felhom-sshd, port 8822, /etc/felhom-sshd/sshd_config) is a separate daemon whose config includes
# nothing from here, so it is not changed. Debian's sshd_config reads this folder BEFORE its own settings, and for
# sshd the first value wins; the bundle checks the effective value with `sshd -T` after the reload.
PermitRootLogin prohibit-password
+74
View File
@@ -0,0 +1,74 @@
#!/bin/bash
# felhom-thin-check — list the thin device ids a thin pool holds next to the ones LVM knows (R-1012). READ-ONLY: it
# names an orphan, it never removes one. Installed as /usr/local/sbin/felhom-thin-check (config bundle).
#
# felhom-thin-check <vg> <pool>
#
# A full host root while vzdump takes its snapshot can leave a thin snapshot inside the pool that LVM no longer knows
# (demo-hp 2026-10-14: `thin_ls` device 4, LVM only 1 and 2); from then on every snapshot backup of that guest fails
# („Creation of new snapshot 4 of device 2 failed") and nothing said why. The repair (`dmsetup message … delete <id>`)
# stays a HUMAN step, decided by the operator — this helper only reads.
#
# Reading the live pool's device list needs a metadata snapshot: the kernel's documented way to read metadata while
# the pool is in use (`reserve_metadata_snap`, `thin_ls --metadata-snap`, `release_metadata_snap`). It changes no data
# and no mapping; it is held for the length of one `thin_ls -o DEV` (well under a second on these pools). Rules:
# 1. the arguments are an LVM name pair; <vg>/<pool> is a thin pool (lv_attr starts with `t`);
# 2. only one run at a time (a lock file); a metadata snapshot that someone ELSE holds is never touched — the run
# refuses (exit 3) and does not release it;
# 3. LVM's list is read before AND after the pool's, so a volume created in between is not reported as an orphan
# (the caller treats an id as an orphan only when it is in the pool and in neither LVM list).
# Output, one record per line: lvm-before <id> <lv> | pool <id> | lvm-after <id> <lv> | ok
# Exit: 0 read, 1 refused (arguments / not a thin pool), 3 a metadata snapshot is already held or cannot be reserved,
# 4 the pool could not be read. Pinned by configs/test_felhom_thin_check.py.
set -u
LVS=/usr/sbin/lvs
DMSETUP=/usr/sbin/dmsetup
THIN_LS=/usr/sbin/thin_ls
LOCK=/run/lock/felhom-thin-check.lock
NAME_RE='^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0,63}$'
die() { echo "felhom-thin-check: REFUSED: $1" >&2; exit "${2:-1}"; } # $1 the reason, $2 the exit code
lvm_ids() { # $1 tag → "<tag> <id> <lv>" for every thin volume of this pool
"$LVS" --noheadings --separator , -o lv_name,thin_id --select "pool_lv=$POOL" -- "$VG" 2>/dev/null |
awk -F, -v tag="$1" '{ gsub(/ /, ""); if ($2 ~ /^[0-9]+$/) print tag, $2, $1 }'
}
release() { "$DMSETUP" message "$TPOOL" 0 release_metadata_snap >/dev/null 2>&1; }
main() {
[[ $# -eq 2 ]] || die "usage: felhom-thin-check <vg> <pool>"
VG=$1 POOL=$2
local attr held ids
[[ "$VG" =~ $NAME_RE && "$POOL" =~ $NAME_RE ]] || die "not an LVM name pair"
attr=$("$LVS" --noheadings -o lv_attr -- "$VG/$POOL" 2>/dev/null | tr -d ' ')
[[ "$attr" == t* ]] || die "$VG/$POOL is not a thin pool"
DM="${VG//-/--}-${POOL//-/--}"
TPOOL="$DM-tpool"
TMETA="/dev/mapper/${DM}_tmeta"
exec 9>"$LOCK" || die "cannot open the lock file" 4
flock -w 30 9 || die "another check holds the lock" 3
# Field 7 of the thin-pool status is the held metadata root: "-" when no metadata snapshot exists.
held=$("$DMSETUP" status "$TPOOL" 2>/dev/null | awk '$3 == "thin-pool" { print $7 }')
[[ -n "$held" ]] || die "cannot read the status of $TPOOL" 4
[[ "$held" == "-" ]] || die "a metadata snapshot of $TPOOL is already held (by someone else) — not touched" 3
lvm_ids lvm-before
"$DMSETUP" message "$TPOOL" 0 reserve_metadata_snap >/dev/null 2>&1 || die "could not reserve a metadata snapshot of $TPOOL" 3
trap release EXIT
trap 'release; trap - EXIT; exit 4' INT TERM HUP
if ! ids=$("$THIN_LS" --metadata-snap --no-headers -o DEV "$TMETA" 2>/dev/null); then
die "thin_ls could not read $TMETA" 4
fi
release
trap - EXIT INT TERM HUP
awk '$1 ~ /^[0-9]+$/ { print "pool", $1 }' <<<"$ids"
lvm_ids lvm-after
echo ok
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then main "$@"; fi
+99
View File
@@ -0,0 +1,99 @@
#!/bin/bash
# felhom-vzdump-partial-clean — remove ONE dead, half-written vzdump backup (R-1006). The only delete the agent's
# sudoers allows for backup files. Installed as /usr/local/sbin/felhom-vzdump-partial-clean (config bundle).
#
# felhom-vzdump-partial-clean <storage> <vmid> <name>
#
# A power cut during the night backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` (+ its `.tmp/` directory) on the
# storage; nothing removes it, every later backup is then skipped „not enough space", and a full host root breaks
# LVM's own metadata writes (power-cut-fixes-2026-10-14 C4). This helper removes such a leftover — and NOTHING else.
# It trusts nothing the agent says; every rule is checked here, as root, and any doubt REFUSES (exit 1):
# 1. the arguments: a storage id, a vmid, and a name that is exactly
# vzdump-(lxc|qemu)-<THAT vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp) — never a finished archive
# (.tar.zst, .vma.zst, …), never a .log/.notes, never another vmid;
# 2. the storage: its section in /etc/pve/storage.cfg is `dir: <storage>` with a `path`; the target is
# <path>/dump/<name>, not a symlink, and its real directory IS <path>/dump;
# 3. not in use: no vzdump runs on the host at all (a task `…:vzdump:…` or a `vzdump` command — a job for several
# guests carries no vmid in its task name), and no
# process holds the target (or, for the .tmp directory, anything under it) open or as its working directory
# (C2: after a killed worker, tar and zstd kept writing into the .tar.dat for 4 min 40 s, none named „vzdump");
# 4. dead: last modified before this boot (`btime` in /proc/stat), or untouched for at least QUIET_S seconds.
# Prints one line on success: „removed <name> <bytes> B (<why>)". Pinned by configs/test_felhom_vzdump_partial_clean.py.
set -u
STORAGE_CFG=/etc/pve/storage.cfg
PROC=/proc
QUIET_S=120
die() { echo "felhom-vzdump-partial-clean: REFUSED: $*" >&2; exit 1; }
storage_path() { # $1 storage id → prints the dir storage's path, or nothing
awk -v want="$1" '
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (id == want); next }
in_s && /^[ \t]+path[ \t]/ { if (type == "dir") { sub(/^[ \t]+path[ \t]+/, ""); sub(/[ \t]+$/, ""); print; exit } }
' "$STORAGE_CFG"
}
storage_type() {
awk -v want="$1" '/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); id = h[2]; sub(/[ \t]+$/, "", id); if (id == want) { print h[1]; exit } }' "$STORAGE_CFG"
}
boot_time() { awk '$1 == "btime" { print $2 }' "$PROC/stat"; }
newest_mtime() { # a file's mtime, or the newest mtime of anything in a directory (the dir itself included)
if [[ -d "$1" ]]; then find "$1" -xdev -printf '%T@\n' 2>/dev/null | sort -n | tail -1 | cut -d. -f1
else stat -c %Y "$1"; fi
}
busy_reason() { # $1 vmid, $2 target → prints why it is in use, or nothing
local vmid=$1 target=$2 p link cmd
for p in "$PROC"/[0-9]*; do
[[ -r "$p/cmdline" ]] || continue
cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)
# ANY vzdump on the host refuses: a job for several guests runs as `task UPID:…:vzdump::…` (no vmid in it), and
# its tar may be between this guest's files. Waiting costs one backup run; deleting a live one costs the backup.
if [[ "$cmd" == *":vzdump:"* ]] || [[ "$cmd" =~ (^|/)vzdump([[:space:]]|$) ]]; then
echo "a vzdump runs on this host (pid ${p##*/})"; return
fi
for link in "$p/cwd" "$p"/fd/*; do
local to
to=$(readlink "$link" 2>/dev/null) || continue
if [[ "$to" == "$target" || "$to" == "$target/"* ]]; then
echo "pid ${p##*/} has it open"; return
fi
done
done
}
main() {
local storage=${1:-} vmid=${2:-} name=${3:-}
[[ $# -eq 3 ]] || die "usage: felhom-vzdump-partial-clean <storage> <vmid> <name>"
[[ "$storage" =~ ^[A-Za-z][A-Za-z0-9_-]{0,63}$ ]] || die "bad storage id"
[[ "$vmid" =~ ^[1-9][0-9]{2,8}$ ]] || die "bad vmid"
[[ "$name" =~ ^vzdump-(lxc|qemu)-${vmid}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$ ]] \
|| die "name is not a half-written vzdump of guest $vmid: $name"
[[ "$(storage_type "$storage")" == "dir" ]] || die "storage $storage is not a dir storage"
local path; path=$(storage_path "$storage")
[[ "$path" == /* ]] || die "storage $storage has no path"
local dump="$path/dump" target="$path/dump/$name"
[[ -e "$target" ]] || die "no such entry: $target"
[[ ! -L "$target" ]] || die "a symlink: $target"
[[ "$(realpath -e "$dump")" == "$(realpath -e "$(dirname "$target")")" ]] || die "not directly in $dump"
if [[ "$name" == *.tmp ]]; then [[ -d "$target" ]] || die ".tmp is not a directory"
else [[ -f "$target" ]] || die "not a regular file"; fi
local busy; busy=$(busy_reason "$vmid" "$target")
[[ -z "$busy" ]] || die "in use: $busy"
local bt mt now why
bt=$(boot_time); mt=$(newest_mtime "$target"); now=$(date +%s)
[[ "$bt" =~ ^[0-9]+$ && "$mt" =~ ^[0-9]+$ ]] || die "cannot read the boot time or the modification time"
if (( mt < bt )); then why="written before this boot"
elif (( now - mt >= QUIET_S )); then why="untouched for $(( now - mt )) s, no vzdump for $vmid, nothing holds it"
else die "modified $(( now - mt )) s ago — may still be written"; fi
local size; size=$(du -sb "$target" 2>/dev/null | cut -f1)
if [[ -d "$target" ]]; then rm -rf --one-file-system -- "$target"; else rm -f -- "$target"; fi
[[ ! -e "$target" ]] || die "remove failed: $target"
echo "removed $name ${size:-?} B ($why)"
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
main "$@"
fi
+238 -1
View File
@@ -45,8 +45,10 @@ class St:
class Box: class Box:
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set.""" """An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
def __init__(self, bundle_bytes, signed, oob=False, signers=True): def __init__(self, bundle_bytes, signed, oob=False, signers=True, install_mode="appliance"):
self.files, self.modes, self.uids = {}, {}, {} self.files, self.modes, self.uids = {}, {}, {}
if install_mode is not None: # the installer's ROOT-OWNED record (D43 = A reads it for the sshd drop-in)
self.put(osapply.INSTALL_STATE, json.dumps({"completed": [], "mode": install_mode}).encode(), 0o644)
self.dirs = {osapply.OOB_DIR} if oob else set() self.dirs = {osapply.OOB_DIR} if oob else set()
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644) self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
if signers: if signers:
@@ -429,6 +431,241 @@ class SelfCheckUndo(unittest.TestCase):
self.assert_restored(box, before, rep) self.assert_restored(box, before, rep)
class RootKeyOnly(unittest.TestCase):
"""R-982, operator ruling D4 (2026-10-22): the bundle gives the host's own sshd a root-key-only drop-in, tests the
whole config and reloads sshd — only when the file was written; a drop-in that says anything but
`prohibit-password` is refused before anything is written.
RED-PROOF: with the after_install block removed, test_fresh_box_takes_the_dropin_and_reloads fails (no reload);
with the content rule removed, test_a_dropin_that_says_yes_or_no_is_refused fails — seen failing 2026-10-22."""
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
def test_fresh_box_takes_the_dropin_and_reloads(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertIn(b"PermitRootLogin prohibit-password", box.files[self.DEST])
self.assertEqual(box.modes[self.DEST], 0o644)
self.assertIn(["/usr/sbin/sshd", "-t"], box.calls)
self.assertIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
self.assertFalse(any("felhom-sshd" in " ".join(c) for c in box.calls), "the operator's door must not be touched")
def test_an_unchanged_dropin_is_not_reloaded(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put(self.DEST, (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes(), 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_dropin_that_says_yes_or_no_is_refused(self):
for bad in (b"PermitRootLogin yes\n", b"PermitRootLogin no\n", b"# nothing\n",
b"PermitRootLogin prohibit-password\nPermitRootLogin yes\n"):
def edit(files, bad=bad):
for e in files:
if e["path"] == self.DEST:
e["content_b64"] = base64.b64encode(bad).decode()
data, sha = edited_bundle(edit)
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertNotEqual(rc, 0, bad)
self.assertNotIn(self.DEST, box.files, bad)
# Review 2026-10-22 night. A2: the effective value is read back with `sshd -T`; a drop-in that is not in force is
# said in the log and the report (not refused — a refusal would freeze the bundle route). A3: a bundle undone
# after the reload reloads sshd again with the old state. RED-PROOF: with the sshd -T check removed,
# test_a_dropin_not_in_force_is_reported fails; with the undo reload removed, test_an_undone_bundle_reloads_sshd
# fails — seen failing 2026-10-22 night.
def _sshd_T(self, box, value):
orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
box.calls.append(argv)
return 0, f"port 22\npermitrootlogin {value}\npasswordauthentication yes\n", ""
return orig(argv, timeout, stdin)
box.host = host
def test_the_effective_value_is_checked(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "without-password")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(any(c[:2] == ["/usr/sbin/sshd", "-T"] for c in box.calls))
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
def test_a_dropin_not_in_force_is_reported(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "yes") # e.g. no Include of sshd_config.d
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
self.assertTrue(any("NOT in force" in l for l in box.logs), box.logs)
def test_an_undone_bundle_reloads_sshd(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.visudo_fail = True # the self-check fails AFTER the install → undo
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assertNotIn(self.DEST, box.files)
reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
class ByoKeepsItsSSH(unittest.TestCase):
"""D43 = A (operator 2026-10-24, R-1257): the root-key-only drop-in goes ONLY to a box whose root-owned install record
says appliance; on a BYO host nothing is written and a copy WE wrote is removed; a box with no record is left as it
is. X3: the in-force check asks sshd about each of the host's own LAN addresses. X4: a failing `sshd -t` puts back
only the SSH step. RED-PROOF (2026-10-24, felhom-agent f5c3d77 + these tests): with the drop-in's policy back to
`replace`, test_a_byo_host_gets_no_dropin and test_our_copy_comes_off_a_byo_host fail; with the probe back to
192.0.2.1 only, test_a_match_block_for_the_lan_is_not_in_force fails; with `sshd -t` back to `must`,
test_a_failing_sshd_t_undoes_only_the_ssh_step fails (rc 3, the whole bundle undone)."""
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
OURS = (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes()
def test_a_byo_host_gets_no_dropin(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertNotIn(["/usr/sbin/sshd", "-t"], box.calls)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
self.assertIn(self.DEST, rep["bundle"]["skipped"])
rec = json.loads(box.files[osapply.BUNDLE_RECORD])
self.assertEqual(rec["files"][self.DEST], "skipped")
def test_our_copy_comes_off_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
box.put(self.DEST, self.OURS, 0o644) # an earlier bundle wrote it (before D43)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertEqual(rep["bundle"]["removed"], [self.DEST])
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files),
"the previous copy must be kept for the undo")
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], "removed")
def test_the_owners_own_file_stays_on_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
box.put(self.DEST, b"# my own\nPermitRootLogin yes\n", 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files[self.DEST], b"# my own\nPermitRootLogin yes\n")
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_box_with_no_record_is_left_as_it_is(self):
for setup in ("none", "agent-writable"):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode=None if setup == "none" else "byo")
if setup == "agent-writable":
box.modes[osapply.INSTALL_STATE] = 0o666 # proves nothing
box.put(self.DEST, self.OURS, 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), self.OURS, setup)
self.assertIn(self.DEST, rep["bundle"]["skipped"], setup)
def _fake(self, box, addrs=("192.168.0.50",), match=None, sshd_t_rc=0):
orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:3] == ["ip", "-4", "-o"]:
box.calls.append(argv)
return 0, "".join(f"2: vmbr0 inet {a}/24 brd x scope global vmbr0\\ valid_lft forever\n" for a in addrs), ""
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
box.calls.append(argv)
val = "yes" if match and f"addr={match}" in argv[-1] else "without-password"
return 0, f"port 22\npermitrootlogin {val}\n", ""
if argv == ["/usr/sbin/sshd", "-t"]:
box.calls.append(argv)
return (sshd_t_rc, "", "/etc/ssh/sshd_config line 9: Bad configuration option") if sshd_t_rc else (0, "", "")
return orig(argv, timeout, stdin)
box.host = host
# R-1285: the policy pass the agent runs right after every bundle install, with the wrapper NOW on disk. No
# signature (it only removes a file we wrote). RED-PROOF: make policy_pass return before the loop → our copy stays.
POLICY_PLAN = {"release_id": "bundle-policy-0.166.0", "layer": "host", "mode": "bundle-policy"}
def test_the_policy_pass_removes_our_copy_from_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644) # the old wrapper's bundle run left it
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertEqual(rep["bundle_policy"], {"install_mode": "byo", "removed": [self.DEST]}, rep)
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files))
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
def test_the_policy_pass_leaves_an_appliance_and_the_owners_file(self):
for mode, content in (("appliance", None), ("byo", b"# my own\nPermitRootLogin yes\n")):
data, sha = real_bundle()
box = Box(data, None, install_mode=mode)
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
body = content if content is not None else self.OURS
box.put(self.DEST, body, 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), body, mode)
self.assertEqual(rep["bundle_policy"]["removed"], [], mode)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, mode)
def test_the_policy_pass_puts_our_copy_back_when_sshd_t_fails(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644)
self._fake(box, sshd_t_rc=1)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), self.OURS)
self.assertEqual(rep["bundle_policy"]["removed"], [])
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_match_block_for_the_lan_is_not_in_force(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"), match="192.168.0.50")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
probed = [c[-1] for c in box.calls if c[:2] == ["/usr/sbin/sshd", "-T"]]
self.assertTrue(any("addr=192.168.0.50" in p for p in probed) and any("addr=10.77.0.4" in p for p in probed), probed)
self.assertFalse(any("addr=192.0.2.1" in p for p in probed), probed)
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
self.assertTrue(any("NOT in force for 192.168.0.50" in l for l in box.logs), box.logs)
def test_the_lan_check_passes_when_every_address_is_key_only(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
def test_a_failing_sshd_t_undoes_only_the_ssh_step(self):
for before in (None, b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982 old\nPermitRootLogin prohibit-password\n"):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
if before is not None:
box.put(self.DEST, before, 0o644)
self._fake(box, sshd_t_rc=255)
rc, rep = run(box)
self.assertEqual(rc, 0, rep) # the bundle stays
self.assertEqual(box.files.get(self.DEST), before, "only the SSH step is put back")
self.assertIn("/usr/local/sbin/felhom-os-apply", box.files, "the rest of the bundle stays")
self.assertIn("sshd -t failed", rep["bundle"].get("sshd_step_undone", ""), rep)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, "sshd never ran the refused state")
want = "absent" if before is None else hashlib.sha256(before).hexdigest()
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], want)
class TrustBootstrap(unittest.TestCase): class TrustBootstrap(unittest.TestCase):
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self): def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
data, sha = real_bundle() data, sha = real_bundle()
+57
View File
@@ -139,5 +139,62 @@ class Guard(unittest.TestCase):
self.assertEqual(mode, 0o644) self.assertEqual(mode, 0o644)
class KernelStepCannotLeaveTheBoxOff(unittest.TestCase):
"""R-836 / `11` §5.11 Part B 4: a kernel step's planned reboot, one crash and one self-revert cannot add up to the
box staying off. The wrapper starts a step only when the guard is armed with NO unclean boot in its window
(felhom-os-apply Kernel.check_guard, R21); the planned reboot and the self-revert are orderly (`systemctl reboot` —
the clean-stop marker); so the step adds at most ONE unclean boot, and the box stays off only after the LIMIT-th
(3rd) within the hour. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
def setUp(self):
self.d = tempfile.TemporaryDirectory()
self.e = FakeEnv(self.d.name)
cg.main(["x", "boot"], self.e)
s = self.e.state()
self.assertTrue(s["armed"])
self.assertEqual(s["unclean_boots_in_window"], 0, "the wrapper's precondition (R21)")
def tearDown(self):
self.d.cleanup()
def planned(self, minutes):
cg.main(["x", "clean-stop"], self.e)
self.e.t += minutes * 60
cg.main(["x", "boot"], self.e)
def crash(self, minutes):
self.e.t += minutes * 60
cg.main(["x", "boot"], self.e)
def test_planned_reboot_one_crash_one_self_revert(self):
self.planned(2) # the step's one-shot reboot (orderly)
self.crash(3) # the new kernel crashes after the guard ran; the box restarts (panic=10)
self.planned(2) # the self-revert (orderly)
s = self.e.state()
self.assertFalse(s["tripped"], s)
self.assertTrue(s["armed"])
self.assertEqual(self.e.panic(), 10, "the box still restarts after a crash")
self.assertEqual(s["unclean_boots_in_window"], 1, "the step added exactly one unclean boot")
def test_a_panic_before_userspace_is_not_even_counted(self):
# the one-shot kernel panics before the guard's unit runs (measured: rdinit= and init= missing): the planned
# reboot's clean-stop marker is still there when the old kernel boots, so this boot counts as clean.
cg.main(["x", "clean-stop"], self.e)
self.e.t += 120 # the panicking boot: no userspace, the guard never ran
cg.main(["x", "boot"], self.e)
s = self.e.state()
self.assertEqual(s["unclean_boots_in_window"], 0, s)
self.assertEqual(self.e.panic(), 10)
def test_the_box_stays_off_only_after_two_more_crashes_than_the_step_makes(self):
self.planned(2)
self.crash(3) # the step's one crash
self.planned(2) # the self-revert
self.crash(5) # an UNRELATED crash within the hour: the guard trips (the 3rd would leave it off)
s = self.e.state()
self.assertTrue(s["tripped"])
self.assertEqual(s["unclean_boots_in_window"], 2, "two unclean boots: one from the step, one not")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+255
View File
@@ -0,0 +1,255 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-internal-place (felhom.eu architecture/16 §5, §7 — sizes ruled 2026-10-16).
The helper is run for real under bash; `lvs`, `lvcreate`, `blkid`, `mkfs.ext4` and `stat` are PATH stubs that read and
write one JSON state file, so nothing on this machine is touched. Every refusal has a positive control: the same setup
without the one condition goes through — so a test cannot pass because the helper refuses everything.
Run: python3 configs/test_felhom_internal_place.py (also run by the Go suite, internal/internalplace)
"""
import json, os, pathlib, stat, subprocess, tempfile, unittest
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-internal-place"
GIB = 1 << 30
STUB_LVS = r'''#!/usr/bin/env python3
import json, os, sys
st = json.load(open(os.environ["IP_STATE"]))
if st.get("lvs_fail_listing") and "lv_name,pool_lv,lv_size" in sys.argv: sys.exit(5)
args = [a for a in sys.argv[1:] if not a.startswith("-") and a not in ("b",)]
fields = sys.argv[sys.argv.index("-o") + 1].split(",")
target = args[-1]
listing = target == st["vg"]
for lv in st["lvs"]:
full = st["vg"] + "/" + lv["name"]
if listing and st.get("hide_pool_in_listing") and lv.get("pool", "") == "":
continue # a listing that silently lost the pool row (the pool's own size stays readable)
if listing or target == full:
row = {"lv_name": lv["name"], "pool_lv": lv.get("pool", ""), "lv_size": str(lv["size"])}
print(" " + " ".join(row[f] for f in fields))
'''
STUB_LVCREATE = r'''#!/usr/bin/env python3
import json, os, sys
p = os.environ["IP_STATE"]; st = json.load(open(p))
a = sys.argv[1:]
size = int(a[a.index("-V") + 1].rstrip("G")) * (1 << 30)
vg, pool = a[a.index("-T") + 1].split("/")
st["lvs"].append({"name": a[a.index("-n") + 1], "pool": pool, "size": size, "active": True})
st.setdefault("calls", []).append("lvcreate " + " ".join(a))
if st.get("dirty_new"): st["fs"]["felhom-belso"] = {"TYPE": "", "RAW": "junk"}
json.dump(st, open(p, "w"))
'''
STUB_BLKID = r'''#!/usr/bin/env python3
# like blkid -p: exit 0 when something is found, 2 when nothing is; st["blkid_rc"] forces an error exit
import json, os, sys
st = json.load(open(os.environ["IP_STATE"]))
if st.get("blkid_rc"): sys.exit(st["blkid_rc"])
fs = st["fs"].get(os.path.basename(sys.argv[-1]), {})
if "-s" in sys.argv:
v = fs.get(sys.argv[sys.argv.index("-s") + 1], "")
if v: print(v)
sys.exit(0 if v else 2)
vals = [fs.get(k, "") for k in ("TYPE", "LABEL", "UUID", "RAW") if fs.get(k)]
if vals: print("\n".join(vals))
sys.exit(0 if vals else 2)
'''
STUB_MKFS = r'''#!/usr/bin/env python3
import json, os, sys
p = os.environ["IP_STATE"]; st = json.load(open(p))
dev = os.path.basename(sys.argv[-1])
st["fs"][dev] = {"TYPE": "ext4", "LABEL": sys.argv[sys.argv.index("-L") + 1], "UUID": "0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c"}
st.setdefault("calls", []).append("mkfs.ext4 " + " ".join(sys.argv[1:]))
json.dump(st, open(p, "w"))
'''
STUB_STAT = r'''#!/usr/bin/env python3
import json, os, sys
st = json.load(open(os.environ["IP_STATE"]))
name = os.path.basename(sys.argv[-1])
if any(lv["name"] == name and lv.get("active", True) for lv in st["lvs"]): print("block special file")
else: sys.exit(1)
'''
CFG = "dir: local\n\tpath /var/lib/vz\n\tcontent iso,backup\n\nlvmthin: local-lvm\n\tthinpool data\n\tvgname pve\n\tcontent rootdir,images\n"
class Box:
def __init__(self, pool_gib, lvs=(), fs=None, cfg=CFG, **extra):
self.t = tempfile.TemporaryDirectory()
r = pathlib.Path(self.t.name)
self.bin = r / "bin"; self.bin.mkdir()
for name, body in (("lvs", STUB_LVS), ("lvcreate", STUB_LVCREATE), ("blkid", STUB_BLKID),
("mkfs.ext4", STUB_MKFS), ("stat", STUB_STAT)):
f = self.bin / name; f.write_text(body); f.chmod(0o755)
self.cfg = r / "storage.cfg"; self.cfg.write_text(cfg)
self.state = r / "state.json"
st = {"vg": "pve", "lvs": [{"name": "data", "pool": "", "size": pool_gib * GIB}] + [dict(l) for l in lvs],
"fs": fs or {}}
st.update(extra)
self.state.write_text(json.dumps(st))
def run(self, *args):
env = {"PATH": f"{self.bin}:/usr/bin:/bin", "IP_STATE": str(self.state), "FELHOM_STORAGE_CFG": str(self.cfg)}
p = subprocess.run(["bash", str(HELPER), *args], env=env, capture_output=True, text=True)
return p.returncode, p.stdout.strip(), p.stderr.strip()
def st(self):
return json.loads(self.state.read_text())
def calls(self):
return self.st().get("calls", [])
def close(self):
self.t.cleanup()
def thin(name, gib, pool="data"):
return {"name": name, "pool": pool, "size": gib * GIB}
class Sizes(unittest.TestCase):
"""The operator's table (SPACE-CHECK §4, ruled 2026-10-16), as the helper computes it on a fresh box."""
def plan(self, pool, planned, lvs=()):
b = Box(pool, lvs)
try:
rc, out, err = b.run("plan", str(planned))
self.assertEqual(rc, 0, err)
self.assertEqual(b.calls(), [], "plan must change nothing")
return dict(kv.split("=") for kv in out.split()[1:])
finally:
b.close()
def test_the_ruled_table(self):
# disk → pool (GiB), data volume (planned = 32 + mp0), expected internal
for pool, mp0, reserve, internal in ((140, 50, 16, 42), (349, 100, 34, 183), (800, 130, 48, 590), (1720, 130, 48, 1510)):
got = self.plan(pool, 32 + mp0)
self.assertEqual((int(got["reserve"]), int(got["internal"])), (reserve, internal), (pool, got))
def test_promised_volumes_count_but_vzdump_snapshots_and_other_pools_do_not(self):
got = self.plan(349, 0, [thin("vm-9201-disk-0", 32), thin("vm-9201-disk-1", 250),
thin("snap_vm-9201-disk-1_vzdump", 250), thin("vm-500-disk-0", 40, pool="other")])
self.assertEqual((int(got["promised"]), int(got["internal"])), (282, 33), got)
class Create(unittest.TestCase):
def test_a_fresh_box_gets_the_place(self):
b = Box(349)
try:
rc, out, err = b.run("create", "132")
self.assertEqual(rc, 0, err)
self.assertRegex(out, r"^created uuid=[0-9a-f-]{36} gib=183$")
self.assertEqual(len([c for c in b.calls() if c.startswith("lvcreate")]), 1)
self.assertIn("-V 183G -T pve/data -n felhom-belso", b.calls()[0])
self.assertIn("-L felhom-belso", b.calls()[1])
finally:
b.close()
def test_too_little_space_makes_nothing(self):
b = Box(53) # demo-hp's 128 GB disk
try:
rc, out, err = b.run("create", str(32 + 50))
self.assertEqual(rc, 0, err)
self.assertTrue(out.startswith("skipped "), out)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_an_existing_place_is_found_never_formatted(self):
fs = {"felhom-belso": {"TYPE": "ext4", "LABEL": "felhom-belso", "UUID": "11111111-2222-3333-4444-555555555555"}}
b = Box(349, [thin("felhom-belso", 183)], fs)
try:
rc, out, err = b.run("create", "0")
self.assertEqual(rc, 0, err)
self.assertEqual(out, "existing uuid=11111111-2222-3333-4444-555555555555 gib=183")
self.assertEqual(b.calls(), [], "an existing place must never be created or formatted again")
finally:
b.close()
def test_a_foreign_filesystem_on_the_name_is_refused(self):
# positive control: test_an_existing_place_is_found_never_formatted (same LV, Felhom's label) goes through
for fs in ({"TYPE": "xfs", "LABEL": "felhom-belso", "UUID": "x"}, {"TYPE": "ext4", "LABEL": "films", "UUID": "y"}):
b = Box(349, [thin("felhom-belso", 183)], {"felhom-belso": fs})
try:
rc, out, err = b.run("create", "0")
self.assertEqual(rc, 1, (fs, out))
self.assertIn("REFUSED", err)
self.assertEqual(b.calls(), [], fs)
finally:
b.close()
def test_a_crash_between_lvcreate_and_mkfs_is_finished(self):
b = Box(349, [thin("felhom-belso", 183)], {})
try:
rc, out, err = b.run("create", "0")
self.assertEqual(rc, 0, err)
self.assertRegex(out, r"^formatted uuid=")
self.assertEqual([c.split()[0] for c in b.calls()], ["mkfs.ext4"])
finally:
b.close()
def test_an_unnamed_signature_is_refused(self):
b = Box(349, [thin("felhom-belso", 183)], {"felhom-belso": {"RAW": "something"}})
try:
rc, out, err = b.run("create", "0")
self.assertEqual(rc, 1, out)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_a_new_volume_that_is_not_empty_is_not_formatted(self):
b = Box(349, dirty_new=True)
try:
rc, out, err = b.run("create", "132")
self.assertEqual(rc, 1, out)
self.assertNotIn("mkfs", " ".join(b.calls()))
finally:
b.close()
def test_bad_arguments_and_no_local_lvm_are_refused(self):
b = Box(349, cfg="dir: local\n\tpath /var/lib/vz\n")
try:
self.assertEqual(b.run("create", "0")[0], 1)
self.assertEqual(b.calls(), [])
finally:
b.close()
b = Box(349)
try:
for args in (("create", "-1"), ("create", "1x"), ("wipe", "0"), ("create",)):
self.assertEqual(b.run(*args)[0], 1, args)
self.assertEqual(b.calls(), [])
finally:
b.close()
class FailsClosed(unittest.TestCase):
"""A read that fails must never read as „empty" or „nothing promised". Positive control: Create (same setups)."""
def test_a_failed_volume_listing_creates_nothing(self):
b = Box(349, lvs_fail_listing=True)
try:
rc, out, err = b.run("create", "132")
self.assertEqual(rc, 1, out)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_a_listing_without_the_pool_creates_nothing(self):
b = Box(349, [thin("vm-9201-disk-1", 250)], hide_pool_in_listing=True)
try:
self.assertEqual(b.run("create", "132")[0], 1)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_a_blkid_error_never_reads_as_empty(self):
for setup in ({"lvs": [thin("felhom-belso", 183)], "fs": {}}, {}):
b = Box(349, **({"lvs": setup["lvs"], "fs": setup["fs"]} if setup else {}), blkid_rc=4)
try:
rc, out, err = b.run("create", "0" if setup else "132")
self.assertEqual(rc, 1, (setup, out))
self.assertNotIn("mkfs", " ".join(b.calls()), setup)
finally:
b.close()
if __name__ == "__main__":
unittest.main()
+145
View File
@@ -0,0 +1,145 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-mkfs-guarded.sh — the ONLY mkfs path the agent's sudoers permits.
First tests of this wrapper (agent v0.158.0), written for partitions as places (felhom.eu architecture/16 §6). Every
mount of every node of the disk outside /mnt/felhom-drives refuses — INCLUDING a sibling that is Felhom's own drive
(its raw /mnt/<name> mount): the agent can make that corroborating mount itself, so the root wrapper must not trust
it (security review of 912f7c1). An unlistable disk refuses.
HOW IT RUNS ANYWHERE: the wrapper is copied with five literal host paths replaced (each replacement is asserted, so a
drifted wrapper fails here instead of silently testing nothing): /proc/mounts → a fake mount table, the block-device
test → true (no real /dev nodes here), the pvs candidates → a stub, /sys/block → a fake tree, the mkfs binaries → a
recorder. `lsblk` is a PATH stub answering the three calls the wrapper makes from one JSON disk description.
Every refusal has a positive control (the same disk without the one condition is formatted).
Run: python3 configs/test_felhom_mkfs_guarded.py (also run by the Go suite, internal/storage)
"""
import json, os, pathlib, subprocess, tempfile, unittest
WRAPPER = pathlib.Path(__file__).resolve().parent / "felhom-mkfs-guarded.sh"
STUB_LSBLK = r'''#!/usr/bin/env python3
import json, os, sys
st = json.load(open(os.environ["MG_STATE"]))
a = sys.argv[1:]; dev = a[-1]
nodes = st["nodes"] # [{"name": "/dev/sdb", "pk": "", "fstype": ""}, ...]
if a[0] == "-ndo" and a[1] == "PKNAME":
for n in nodes:
if n["name"] == dev and n.get("pk"): print(n["pk"])
elif a[0] == "-nrpo" and a[1] == "NAME":
if not st.get("lsblk_empty"):
for n in nodes: print(n["name"])
elif a[0] == "-nro" and a[1] == "FSTYPE":
for n in nodes: print(n.get("fstype", ""))
elif a[0] == "-nro" and a[1] == "MOUNTPOINT": # the pre-v0.158.0 wrapper's call: one mountpoint per node, like lsblk
mt = [l.split()[:2] for l in open(os.environ["MG_MOUNTS"]) if l.strip()]
for n in nodes:
mps = [m for d, m in mt if d == n["name"]]
print(mps[0] if mps else "")
'''
STUB_MKFS = '#!/bin/sh\necho "$0 $*" >> "$MG_CALLS"\n'
STUB_PVS = '#!/bin/sh\nprintf "%s\\n" $MG_PVS\n'
class Disk:
def __init__(self, nodes, mounts, ro="0", pvs="", lsblk_empty=False):
self.t = tempfile.TemporaryDirectory()
r = pathlib.Path(self.t.name)
self.bin = r / "bin"; self.bin.mkdir()
(self.bin / "lsblk").write_text(STUB_LSBLK); (self.bin / "lsblk").chmod(0o755)
for n in ("mkfs.ext4", "mkfs.xfs"):
(self.bin / n).write_text(STUB_MKFS); (self.bin / n).chmod(0o755)
(self.bin / "pvs").write_text(STUB_PVS); (self.bin / "pvs").chmod(0o755)
self.state = r / "state.json"
self.state.write_text(json.dumps({"nodes": nodes, "lsblk_empty": lsblk_empty}))
self.mounts = r / "mounts"
self.mounts.write_text("".join(f"{d} {m} ext4 rw 0 0\n" for d, m in mounts))
self.sys = r / "sys"
for n in nodes:
if not n.get("pk"):
d = self.sys / os.path.basename(n["name"]); d.mkdir(parents=True); (d / "ro").write_text(ro + "\n")
self.calls = r / "calls"
self.pvs = pvs
src = WRAPPER.read_text()
subs = [
("/proc/mounts", str(self.mounts)),
('[[ -b "$dev" ]]', "true"),
("for c in /usr/sbin/pvs /sbin/pvs; do", f"for c in {self.bin}/pvs; do"),
('rof="/sys/block/', f'rof="{self.sys}/'),
("exec /usr/sbin/mkfs.ext4", f"exec {self.bin}/mkfs.ext4"),
("exec /usr/sbin/mkfs.xfs ", f"exec {self.bin}/mkfs.xfs "),
]
for a, b in subs:
assert a in src, f"the wrapper no longer contains {a!r} — update this test's substitutions"
src = src.replace(a, b)
self.copy = r / "mkfs-guarded"; self.copy.write_text(src)
def run(self, dev, fstype="ext4"):
env = {"PATH": f"{self.bin}:/usr/bin:/bin", "MG_STATE": str(self.state), "MG_CALLS": str(self.calls),
"MG_MOUNTS": str(self.mounts),
"MG_PVS": self.pvs}
p = subprocess.run(["bash", str(self.copy), dev, fstype], env=env, capture_output=True, text=True)
made = self.calls.read_text().split("\n")[0] if self.calls.exists() else ""
return p.returncode, p.stderr.strip(), made
def close(self):
self.t.cleanup()
def disk_sdb(sdb1_fstype="ext4", sdb2_fstype=""):
return [{"name": "/dev/sdb", "pk": "", "fstype": ""},
{"name": "/dev/sdb1", "pk": "sdb", "fstype": sdb1_fstype},
{"name": "/dev/sdb2", "pk": "sdb", "fstype": sdb2_fstype}]
ROOT = ("/dev/sda2", "/")
OWN_SDB1 = [("/dev/sdb1", "/mnt/hdd_1"), ("/dev/sdb1", "/mnt/felhom-drives/hdd_1")]
class Partitions(unittest.TestCase):
def check(self, mounts, dev="/dev/sdb2", nodes=None, want_ok=True, **kw):
d = Disk(nodes or disk_sdb(), [ROOT] + mounts, **kw)
try:
rc, err, made = d.run(dev)
if want_ok:
self.assertEqual(rc, 0, err)
self.assertTrue(made.endswith(f"mkfs.ext4 -F {dev}"), made)
else:
self.assertEqual(rc, 1, (mounts, made))
self.assertIn("REFUSED", err)
self.assertEqual(made, "", "a refused format must not run mkfs")
return err
finally:
d.close()
def test_a_free_part_on_an_unmounted_disk_is_formatted(self):
self.check([]) # positive control for every refusal below
def test_the_free_part_next_to_felhoms_own_drive_is_refused(self):
# The agent could make the „corroborating" managed mount itself: the root wrapper does not trust it.
self.check(OWN_SDB1, want_ok=False)
def test_a_sibling_mounted_raw_only_is_refused(self):
self.check([("/dev/sdb1", "/mnt/hdd_1")], want_ok=False)
def test_a_sibling_mounted_elsewhere_is_refused_even_with_a_managed_mount(self):
for elsewhere in ("/srv/data", "/media/usb", "/mnt/a/b", "/home/x"):
self.check([("/dev/sdb1", elsewhere), ("/dev/sdb1", "/mnt/felhom-drives/hdd_1")], want_ok=False)
def test_the_target_itself_mounted_raw_is_refused(self):
self.check([("/dev/sdb2", "/mnt/films"), ("/dev/sdb2", "/mnt/felhom-drives/films")], want_ok=False)
def test_the_target_mounted_only_under_felhom_drives_is_formatted(self):
self.check([("/dev/sdb2", "/mnt/felhom-drives/films")])
def test_an_unlistable_disk_is_refused(self):
self.check([], want_ok=False, lsblk_empty=True)
def test_the_catastrophic_cases_still_refuse(self):
self.check([("/dev/sdb1", "/")], want_ok=False) # system disk
self.check([], nodes=disk_sdb(sdb1_fstype="LVM2_member"), want_ok=False) # a member signature
self.check([], pvs="/dev/sdb1", want_ok=False) # an LVM PV (pvs)
self.check([], ro="1", want_ok=False) # read-only
if __name__ == "__main__":
unittest.main()
+840 -4
View File
@@ -83,7 +83,7 @@ class Fake:
return self.clock return self.clock
def sleep(self, s): def sleep(self, s):
pass self.sleeps = getattr(self, "sleeps", []) + [(len(self.calls), s)] # (calls made before it, seconds)
def verify_sig(self, signers, key_id, ns, blob, sig): def verify_sig(self, signers, key_id, ns, blob, sig):
self.verified = (signers, key_id, ns, blob, sig) self.verified = (signers, key_id, ns, blob, sig)
@@ -196,6 +196,27 @@ class Fake:
return 0, self.engine + "\n", "" return 0, self.engine + "\n", ""
if cmd == "docker" and a[1:3] == ["ps", "-q"]: if cmd == "docker" and a[1:3] == ["ps", "-q"]:
return 0, "".join(i + "\n" for i in self.ids), "" return 0, "".join(i + "\n" for i in self.ids), ""
# R-528: the memory-kill check's engine (oom_image None = unreadable; oom_state / oom_event the engine's answer)
if cmd == "date" and a[1:] == ["+%s"]:
# each read is 3 s later than the last, so the order of the reads is visible in the values
self.oom_epochs = getattr(self, "oom_epochs", []) + [int(self.clock) + 3 * len(getattr(self, "oom_epochs", []))]
return 0, f"{self.oom_epochs[-1]}\n", ""
if cmd == "docker" and a[1:4] == ["inspect", "-f", "{{.Config.Image}}"]:
img = getattr(self, "oom_image", "gitea.dooplex.hu/admin/felhom-controller:0.300.0")
return (0, img + "\n", "") if img is not None else (1, "", "Error: No such object: felhom-controller")
if cmd == "docker" and a[1] == "run":
self.oom_runs = getattr(self, "oom_runs", []) + [a]
return 137, "", ""
if cmd == "docker" and a[1:4] == ["inspect", "-f", "{{.State.OOMKilled}} {{.State.ExitCode}}"]:
if getattr(self, "oom_inspect_raises", False):
raise subprocess.TimeoutExpired(a, 10)
return 0, getattr(self, "oom_state", "true 137") + "\n", ""
if cmd == "docker" and a[1] == "events":
self.oom_events_argv = a
return 0, ("oom\n" if getattr(self, "oom_event", True) else ""), ""
if cmd == "docker" and a[1:3] == ["rm", "-f"]:
self.oom_removed = getattr(self, "oom_removed", []) + a[3:]
return 0, a[3] + "\n", ""
if cmd == "docker" and a[1] == "inspect": if cmd == "docker" and a[1] == "inspect":
mounts = {"aaa111": "/felhom-controller|/var/run/docker.sock;/app/data;", "bbb222": "/app|/data;"} mounts = {"aaa111": "/felhom-controller|/var/run/docker.sock;/app/data;", "bbb222": "/app|/data;"}
return 0, mounts.get(a[-1], "/other|;") + "\n", "" return 0, mounts.get(a[-1], "/other|;") + "\n", ""
@@ -211,6 +232,8 @@ class Fake:
return (0, self.daemon_json, "") if self.daemon_json is not None else (1, "", "No such file") return (0, self.daemon_json, "") if self.daemon_json is not None else (1, "", "No such file")
if cmd == "cat" and a[1] == "/etc/debian_version": if cmd == "cat" and a[1] == "/etc/debian_version":
return 0, "13.7\n", "" return 0, "13.7\n", ""
if cmd == "pveversion":
return 0, f"pve-manager/{self.installed.get('pve-manager', '9.2.2')}/abcdef (running kernel: 7.0.14-20-pve)\n", ""
if cmd == "uname": if cmd == "uname":
return 0, "7.0.14-20-pve\n", "" return 0, "7.0.14-20-pve\n", ""
if cmd == "apt-mark": if cmd == "apt-mark":
@@ -263,7 +286,7 @@ class Fake:
n, v = x.split("=", 1) n, v = x.split("=", 1)
if v not in self.avail(n): if v not in self.avail(n):
return 100, "", f"E: Version '{v}' for '{n}' was not found" return 100, "", f"E: Version '{v}' for '{n}' was not found"
origin = "Docker CE:trixie" if n in osapply.DOCKER_NAMES else SEC if n == "openssl" else DEB origin = getattr(self, "origins", {}).get(n) or ("Docker CE:trixie" if n in osapply.DOCKER_NAMES else SEC if n == "openssl" else DEB)
out += f"Inst {n} [{self.installed[n]}] ({v} {origin} [amd64])\n" out += f"Inst {n} [{self.installed[n]}] ({v} {origin} [amd64])\n"
out += "".join(l + "\n" for l in self.extra_sim) out += "".join(l + "\n" for l in self.extra_sim)
return 0, out, "" return 0, out, ""
@@ -810,6 +833,14 @@ class DockerLane(unittest.TestCase):
def test_expired_signed_job_is_refused(self): def test_expired_signed_job_is_refused(self):
self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3") self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3")
def test_a_window_longer_than_24h_is_refused(self):
# R-1280. RED-PROOF: drop the MAX_WINDOW_S check → the job (current, signed, 24 h + 1 s) runs.
self.refused(docker_fake(signed=signed_job(issued="2026-10-03T11:59:59Z", expires="2026-10-04T12:00:00Z")), "R3")
def test_a_window_of_24h_is_accepted(self):
rc, rep = run(docker_fake(signed=signed_job(issued="2026-10-03T12:30:00Z", expires="2026-10-04T12:30:00Z")))
self.assertNotEqual((rep.get("refused") or {}).get("code"), "R3", rep)
def test_replayed_signed_job_is_refused(self): def test_replayed_signed_job_is_refused(self):
f = docker_fake(signed=signed_job()) f = docker_fake(signed=signed_job())
f.nonces = {"n1": f.clock + 600} f.nonces = {"n1": f.clock + 600}
@@ -997,8 +1028,6 @@ class RealSignatureCheck(unittest.TestCase):
self.assertNotEqual(r.verify_sig(self.signers, "someone-else", "felhom-op-v1", blob, sig), 0) self.assertNotEqual(r.verify_sig(self.signers, "someone-else", "felhom-op-v1", blob, sig), 0)
self.assertNotEqual(r.verify_sig(self.signers, "felhom-op-1", "felhom-op-v1", blob, self.sign(blob, ns="other-ns")), 0) self.assertNotEqual(r.verify_sig(self.signers, "felhom-op-1", "felhom-op-v1", blob, self.sign(blob, ns="other-ns")), 0)
if __name__ == "__main__":
unittest.main()
class UnsentReport(unittest.TestCase): class UnsentReport(unittest.TestCase):
@@ -1179,3 +1208,810 @@ class CrashLeftTheJournal(unittest.TestCase):
self.assertEqual(rc, 0, rep) self.assertEqual(rc, 0, rep)
self.assertTrue(any("INTERRUPTED" in l for l in f.logs), f.logs) self.assertTrue(any("INTERRUPTED" in l for l in f.logs), f.logs)
self.assertTrue(any(l.startswith("os-apply: REPAIR ") and l.endswith("forced") for l in f.logs), f.logs) self.assertTrue(any(l.startswith("os-apply: REPAIR ") and l.endswith("forced") for l in f.logs), f.logs)
class OOMCheck(unittest.TestCase):
"""R-528 (`09` decision 157): after a Docker engine step the wrapper proves the engine reports a memory kill
(OOMKilled=true AND the `oom` event). Reported only; the hub decides. Red-proofs: audits/readback-2026-10-07/F/."""
def apply(self, **kw):
f = docker_fake(signed=signed_job())
for k, v in kw.items():
setattr(f, k, v)
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
return f, rep
def test_pass_when_oomkilled_and_the_event(self):
f, rep = self.apply()
oc = rep["oom_check"]
self.assertEqual(oc["result"], "pass", oc)
self.assertEqual((oc["oom_killed"], oc["oom_event"], oc["exit_code"]), (True, True, 137))
self.assertEqual(oc["image"], "gitea.dooplex.hu/admin/felhom-controller:0.300.0")
self.assertEqual(sorted(oc), ["detail", "exit_code", "image", "oom_event", "oom_killed", "result"])
run_argv = f.oom_runs[0]
name = run_argv[run_argv.index("--name") + 1]
self.assertTrue(re.match(r"^felhom-oomcheck-[0-9]+-[0-9a-f]{8}$", name), name)
for flag, val in (("--pull", "never"), ("--network", "none"), ("--memory", "64m"), ("--memory-swap", "64m"),
("--label", "felhom.oomcheck=1"), ("--entrypoint", "sh")):
self.assertEqual(run_argv[run_argv.index(flag) + 1], val, flag)
self.assertNotIn("-v", run_argv)
self.assertNotIn("-p", run_argv)
self.assertEqual(run_argv[-3:], ["gitea.dooplex.hu/admin/felhom-controller:0.300.0", "-c", osapply.OOMCHECK_SCRIPT])
self.assertIn(f"container={name}", f.oom_events_argv)
self.assertIn("event=oom", f.oom_events_argv)
self.assertEqual(f.oom_removed, [name], "the check container must be removed")
self.assertTrue(rep["health_after"], "health is read before the check")
# bounded: every call has a timeout and the clock is read twice — the worst case stays within 90 s
self.assertLessEqual(sum(osapply.OOMCHECK_TIMEOUTS.values()) + osapply.OOMCHECK_TIMEOUTS["clock"]
+ osapply.OOMCHECK_SETTLE, 90)
def test_events_window_ends_after_the_settle_wait(self):
# Measured (F1/F2): an --until taken right after the run missed the oom event. The window must end after a
# wait of >= 2 s that comes AFTER the run, at the guest epoch read after that wait, + 1.
f, rep = self.apply()
run_i = next(i for i, c in enumerate(f.calls) if c[-1][:2] == ["docker", "run"])
date_i = [i for i, c in enumerate(f.calls) if c[-1] == ["date", "+%s"]]
waits = [(i, s) for i, s in getattr(f, "sleeps", []) if i > run_i]
self.assertTrue(waits and waits[0][1] >= 2, f"no settle wait after the run: {getattr(f, 'sleeps', None)}")
self.assertTrue(date_i[-1] >= waits[0][0], "the end epoch must be read after the wait")
ev = f.oom_events_argv
since, until = int(ev[ev.index("--since") + 1]), int(ev[ev.index("--until") + 1])
self.assertEqual(until, f.oom_epochs[-1] + 1, "until = the guest epoch read after the wait, + 1")
self.assertGreater(until, f.oom_epochs[0] + 1)
self.assertLess(since, f.oom_epochs[0] + 1)
def test_oomkilled_false_is_fail(self):
f, rep = self.apply(oom_state="false 0")
oc = rep["oom_check"]
self.assertEqual(oc["result"], "fail", oc)
self.assertIn("OOMKilled=true", oc["detail"])
self.assertTrue(oc["oom_event"])
self.assertEqual(len(f.oom_removed), 1)
def test_no_event_is_fail(self):
f, rep = self.apply(oom_event=False)
oc = rep["oom_check"]
self.assertEqual(oc["result"], "fail", oc)
self.assertIn("the oom event", oc["detail"])
self.assertTrue(oc["oom_killed"])
def test_image_unreadable_is_error(self):
f, rep = self.apply(oom_image=None)
oc = rep["oom_check"]
self.assertEqual(oc["result"], "error", oc)
self.assertIsNone(oc["image"])
self.assertIn("image could not be read", oc["detail"])
self.assertFalse(hasattr(f, "oom_runs"), "no container is started without an image")
def test_container_removed_even_when_inspect_raises(self):
f, rep = self.apply(oom_inspect_raises=True)
oc = rep["oom_check"]
self.assertEqual(oc["result"], "error", oc)
self.assertEqual(len(f.oom_removed), 1, "the container must be removed even when inspect raised")
self.assertTrue(f.oom_removed[0].startswith(osapply.OOMCHECK_PREFIX))
self.assertNotIn("failed", rep, "the check never turns the step into a failure")
def test_never_on_guest_or_host_or_in_health_mode(self):
g = Fake()
rc, rep = run(g)
self.assertEqual(rc, 0, rep)
h = Fake()
h.plan["layer"] = "host"
h.plan["packages"] = [{"name": "bash", "version": "5.2.37-2+b10", "origin": "Debian"}]
h.installed["bash"] = "5.2.37-2+b9"
h.live["bash"] = {"5.2.37-2+b10"}
rc_h, rep_h = run(h)
self.assertEqual(rc_h, 0, rep_h)
d = docker_fake(signed=signed_job())
d.plan["mode"] = "health"
rc_d, rep_d = run(d)
self.assertEqual(rc_d, 0, rep_d)
for f, r in ((g, rep), (h, rep_h), (d, rep_d)):
self.assertNotIn("oom_check", r)
self.assertFalse(hasattr(f, "oom_runs"), r.get("layer"))
self.assertFalse(any(c[-1][:2] == ["docker", "run"] for c in f.calls))
def test_mode_oom_check_runs_only_the_check(self):
f = docker_fake() # no authority: the check changes nothing, so it needs none
f.plan["mode"], f.plan["packages"] = "oom-check", []
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["oom_check"]["result"], "pass", rep)
self.assertFalse(any("apt-get" in c[-1] or "dpkg-query" in c[-1] for c in f.calls), f.calls)
self.assertEqual(len(f.oom_removed), 1)
def test_mode_oom_check_keeps_the_refusals(self):
f = docker_fake()
f.plan["mode"], f.plan["packages"] = "oom-check", []
f.files["/etc/pve/lxc/9201.conf"] = "arch: amd64\n"
rc, rep = run(f)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R10"), rep)
self.assertFalse(hasattr(f, "oom_runs"))
g = Fake()
g.plan["mode"] = "oom-check"
rc, rep = run(g)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
PVE = "Proxmox Debian Repository:stable"
PVE_SET = [{"name": "pve-manager", "version": "9.2.21", "origin": "Proxmox Debian Repository"},
{"name": "libpve-common-perl", "version": "9.1.9", "origin": "Proxmox Debian Repository"}]
def pve_fake(signed=None, ring0=False):
f = Fake()
f.installed.update({"pve-manager": "9.2.2", "libpve-common-perl": "9.1.1", "proxmox-kernel-helper": "9.0.4"})
f.live["pve-manager"] = {"9.2.21", "9.2.2"}
f.live["libpve-common-perl"] = {"9.1.9", "9.1.1"}
f.origins = {"pve-manager": PVE, "libpve-common-perl": PVE, "proxmox-kernel-helper": PVE, "shim-signed": PVE,
"proxmox-firewall-data": PVE}
f.plan = {"release_id": "os-pve-t1", "layer": "pve", "lane": "slow", "vmid": 9201, "mode": "apply",
"packages": [dict(p) for p in PVE_SET]}
if ring0:
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
if signed is not None:
f.plan["signed"] = signed
return f
class PVELane(unittest.TestCase):
"""R-812 option A (`09` §3 decision 163): the host's Proxmox USERSPACE packages, slow lane, no kernel / boot /
firmware / microcode (R14), no removal, a new package only from PVE_NEW_ALLOW. Red-proof: audits/day-2026-10-07/B/."""
def refused(self, f, code):
rc, rep = run(f)
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
self.assertEqual(f.installed["pve-manager"], "9.2.2", "nothing may be installed on a refusal")
return rep
# THE RED TEST (design-R-812 §5): before the pve layer existed this plan was refused R12.
def test_pve_manager_plan_is_installed_on_the_host(self):
f = pve_fake(signed=signed_job(packages=PVE_SET, op="os_pve_step"))
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.installed["pve-manager"], "9.2.21")
self.assertEqual(rep["authority"], "signed")
self.assertEqual(rep["pve_manager"], "9.2.21", "pveversion after the step is reported")
inst = [c for c in f.calls if c[0] == "host" and "install" in c[1] and "-s" not in c[1] and "-f" not in c[1]
and "--print-uris" not in c[1]]
self.assertTrue(inst, "the pve layer installs on the HOST")
self.assertFalse([c for c in f.calls if c[0] == "guest" and "install" in c[2]], "nothing installed in the guest")
self.assertEqual(sorted(rep["health_after"]["host_services"]), sorted(osapply.HOST_SERVICES))
def test_kernel_in_a_pve_plan_is_refused(self):
f = pve_fake(ring0=True)
f.plan["packages"].append({"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"})
self.refused(f, "R14")
def test_shim_in_a_pve_plan_is_refused(self):
f = pve_fake(ring0=True)
f.plan["packages"].append({"name": "shim-signed", "version": "1.47+pmx1", "origin": "Proxmox Debian Repository"})
self.refused(f, "R14")
def test_kernel_pulled_in_by_the_simulation_is_refused(self):
f = pve_fake(ring0=True)
f.installed["proxmox-kernel-helper"] = "9.0.4"
f.extra_sim = ["Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])"]
self.refused(f, "R6") # not in the plan — refused before the name check; R14 below when it IS in the plan
g = pve_fake(ring0=True)
g.live["proxmox-kernel-helper"] = {"9.0.6"}
g.plan["packages"] = [dict(PVE_SET[0])]
g.extra_sim = ["Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])"]
# a kernel-helper the plan did not name is R6; the R14 name check covers a listed one (test above)
self.refused(g, "R6")
def test_debian_package_in_a_pve_plan_is_refused(self):
f = pve_fake(ring0=True)
f.plan["packages"].append({"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"})
self.refused(f, "R2")
def test_debian_origin_in_the_pve_simulation_is_refused(self):
f = pve_fake(ring0=True)
f.origins["libpve-common-perl"] = DEB
self.refused(f, "R2")
def test_docker_package_in_a_pve_plan_is_refused(self):
f = pve_fake(ring0=True)
f.plan["packages"].append({"name": "docker-ce", "version": "5:29.8.2-1~debian.13~trixie", "origin": "Proxmox Debian Repository"})
self.refused(f, "R2")
def test_pve_in_the_fast_lane_is_refused(self):
f = pve_fake(ring0=True)
f.plan["lane"] = "fast"
self.refused(f, "R3")
def test_no_authority_is_refused(self):
self.refused(pve_fake(), "R3")
def test_docker_signed_op_does_not_authorize_a_pve_step(self):
self.refused(pve_fake(signed=signed_job(packages=PVE_SET, op="os_docker_step")), "R3")
def test_pve_on_a_byo_box_is_refused(self):
f = pve_fake(ring0=True)
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
self.refused(f, "R12")
def test_unlisted_new_package_is_refused(self):
f = pve_fake(ring0=True)
f.extra_sim = ["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"]
self.refused(f, "R6")
def test_allow_listed_new_package_is_accepted(self):
f = pve_fake(ring0=True)
f.extra_sim = ["Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])"]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.installed["pve-manager"], "9.2.21")
def test_allow_new_is_never_an_open_door_in_the_fast_lane(self):
f = Fake()
f.plan["layer"] = "host"
f.extra_sim = ["Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])"]
rc, rep = run(f)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R6"), rep)
def test_undo_is_refused(self):
f = pve_fake(signed=signed_job(packages=PVE_SET, op="os_pve_step"))
f.plan["undo"] = True
self.refused(f, "R5")
def test_ring0_pending_pve_takes_only_installed_proxmox_userspace(self):
f = pve_fake(ring0=True)
f.plan["select"], f.plan["packages"] = "pending-pve", []
f.installed.update({"linux-image-amd64": "6.12.1", "tailscale": "1.102.2"})
f.pending_sim = [
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
"Inst libpve-common-perl [9.1.1] (9.1.9 Proxmox Debian Repository:stable [all])",
"Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])",
"Inst proxmox-kernel-7.0.14-20-pve-signed (7.0.14-20 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])",
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])",
"Inst tailscale [1.102.2] (1.102.5 Tailscale:pkgs.tailscale.com [amd64])",
]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "ring0")
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["libpve-common-perl", "pve-manager"],
"pending-pve: installed, Proxmox-origin, never kernel/boot/firmware, never Debian or other origins")
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3")
# 2026-10-09, demo-hp (Secure Boot on): proxmox-secure-boot-support's upgrade pulls shim-signed-common; in the pve
# plan that refused the whole step (R6) and skipped the night's kernel step. It is a boot-chain package: left out.
# RED-PROOF: drop proxmox-secure-boot-support from HOST_SLOW_RE -> it is in the plan -> this test fails.
def test_ring0_pending_pve_leaves_the_secure_boot_meta_out(self):
f = pve_fake(ring0=True)
f.plan["select"], f.plan["packages"] = "pending-pve", []
f.installed.update({"proxmox-secure-boot-support": "9.0.1"})
f.pending_sim = [
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-secure-boot-support [9.0.1] (9.0.2 Proxmox Debian Repository:stable [all])",
"Inst shim-signed-common [1.48+pmx1+16.1-1+pmx1] (1.51+pmx1+16.1-2+pmx1 Proxmox Debian Repository:stable [all])",
]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
def test_select_pending_pve_needs_the_pve_layer(self):
f = Fake()
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
rc, rep = run(f)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
# ---------- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----------
U = "1af1fcc6-639c-416b-a7e5-c4470d41a502"
OLD, NEW = "7.0.2-6-pve", "7.0.14-22-pve"
KERNEL_SET = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"},
{"name": "proxmox-kernel-7.0.14-22-pve-signed", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"}]
class KFake(Fake):
"""A Proxmox host with GRUB on UEFI: /boot on the root LV, a vfat ESP, the bundle's two generators. update-grub is
emulated like the real 10_linux: WITHOUT the felhom default file the NEWEST kernel becomes the default (measured,
R-836 — that is the defect the lane exists for); with it, the kernel it names."""
def __init__(self):
super().__init__()
self.running = OLD
self.boot = {OLD}
self.efi, self.esp_fs, self.boot_mount, self.mods, self.snippets, self.pin = True, "vfat", "", True, True, False
self.env = None # None = no env block on the ESP; else {"felhom_next": ...}
self.tree = {} # files put_file wrote
self.reboots = []
self.update_grubs = 0
self.grub_runs_in_postinst = True
self.installed.update({"proxmox-kernel-7.0": "7.0.2-6", "proxmox-default-kernel": "2.1.0",
"pve-firmware": "3.18-3", "proxmox-kernel-7.0.2-6-pve-signed": "7.0.2-6",
"pve-manager": "9.2.21"})
self.live.update({"proxmox-kernel-7.0": {"7.0.14-22", "7.0.2-6"},
"proxmox-kernel-7.0.14-22-pve-signed": {"7.0.14-22"},
"proxmox-kernel-7.0.14-23-pve-signed": {"7.0.14-23"},
"pve-firmware": {"3.18-7", "3.18-3"}})
self.origins = {}
self.kernel_pending = ["Inst pve-firmware [3.18-3] (3.18-7 Proxmox Debian Repository:stable [all])",
"Inst proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-kernel-7.0 [7.0.2-6] (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
"Inst pve-manager [9.2.21] (9.2.22 Proxmox Debian Repository:stable [amd64])",
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])"]
self.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "apply",
"select": "pending-kernel", "packages": []}
self.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
self.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 0})
self.cfg = self.render()
# -- GRUB --
def newest(self):
best = None
for k in self.boot:
if best is None or dpkg_cmp(k[:-4], "gt", best[:-4]):
best = k
return best
def render(self):
d = self.tree.get(osapply.KERNEL_DEFAULT_CFG)
if d:
dflt = re.search(r'GRUB_DEFAULT="([^"]+)"', d).group(1)
else:
dflt = f"gnulinux-advanced-{U}>gnulinux-{self.newest()}-advanced-{U}"
cfg = ('if [ "${next_entry}" ] ; then\n set default="${next_entry}"\nelse\n'
f' set default="{dflt}"\nfi\n'
f"submenu 'Advanced options' $menuentry_id_option 'gnulinux-advanced-{U}' {{\n")
for k in sorted(self.boot):
cfg += f" menuentry 'Proxmox VE, with Linux {k}' $menuentry_id_option 'gnulinux-{k}-advanced-{U}' {{ }}\n"
cfg += "}\n"
if self.snippets:
cfg += "### BEGIN /etc/grub.d/01_felhom_oneshot ###\nload_env felhom_next\n"
cfg += "".join(f"menuentry 'Felhom one-shot: {k}' --id felhom-oneshot-{k} {{ }}\n" for k in sorted(self.boot))
return cfg
def lexists(self, p):
if p == "/sys/firmware/efi":
return self.efi
if p.startswith("/usr/lib/grub/x86_64-efi/"):
return self.mods
if p in (osapply.ONESHOT_SNIPPET, osapply.ONESHOT_ENTRIES):
return self.snippets
if p == osapply.KERNEL_PIN_FILE:
return self.pin
m = re.match(r"^/boot/(vmlinuz|initrd\.img)-(.+)$", p)
if m:
return m.group(2) in self.boot
return p in self.tree
def put_file(self, path, data, mode):
self.tree[path] = data.decode()
def read_file(self, p):
if p == osapply.GRUB_CFG:
return self.cfg
if p in self.tree:
return self.tree[p]
return super().read_file(p)
def host(self, argv, timeout=600, stdin=None):
if argv[0] == "uname":
self.calls.append(("host", argv))
return 0, self.running + "\n", ""
if argv[0] == "findmnt":
self.calls.append(("host", argv))
if argv[-1] == osapply.ESP_MOUNT:
return (0, self.esp_fs + "\n", "") if self.esp_fs else (1, "", "")
return (0, self.boot_mount + "\n", "") if self.boot_mount else (1, "", "")
if argv[0] == "grub-editenv":
self.calls.append(("host", argv))
if argv[2] == "list":
return (1, "", "no such file") if self.env is None else \
(0, "".join(f"{k}={v}\n" for k, v in self.env.items()), "")
if argv[2] == "create":
self.env = {}
return 0, "", ""
if argv[2] == "set":
k, v = argv[3].split("=", 1)
self.env[k] = v
return 0, "", ""
if argv[2] == "unset":
self.env.pop(argv[3], None)
return 0, "", ""
if argv[0] == "update-grub":
self.calls.append(("host", argv))
self.update_grubs += 1
self.cfg = self.render()
return 0, "", ""
if argv[0] == "mkdir":
self.calls.append(("host", argv))
return 0, "", ""
if argv[:2] == ["systemctl", "reboot"]:
self.calls.append(("host", argv))
self.reboots.append(self.running)
return 0, "", ""
return super().host(argv, timeout, stdin)
def emulate(self, argv):
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
if a[0] == "apt-get" and "install" in a and "-s" not in a and "--print-uris" not in a and "-f" not in a:
if self.install_rc:
return self.install_rc, "", "E: boom"
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
n, v = x.split("=", 1)
self.installed[n] = v
m = re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)
if m:
self.installed[f"proxmox-kernel-{v}-pve-signed"] = v
if m or osapply.KERNEL_IMAGE_RE.match(n):
self.boot.add(v + "-pve")
if self.grub_runs_in_postinst:
self.cfg = self.render() # the kernel's postinst runs update-grub (zz-update-grub)
return 0, "Setting up proxmox-kernel ...\n", ""
return super().emulate(argv)
def sim(self, a):
if "--print-uris" in a:
return 0, "", ""
if "dist-upgrade" in a:
return 0, "\n".join(self.kernel_pending) + "\n", ""
out, named = "", set()
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
named.add(x.split("=", 1)[0])
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
n, v = x.split("=", 1)
if v not in self.avail(n):
return 100, "", f"E: Version '{v}' for '{n}' was not found"
o = self.origins.get(n, PVE)
out += f"Inst {n} [{self.installed[n]}] ({v} {o} [amd64])\n" if n in self.installed else f"Inst {n} ({v} {o} [amd64])\n"
if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n):
img = f"proxmox-kernel-{v}-pve-signed"
if img not in self.installed and img not in named:
out += f"Inst {img} ({v} {PVE} [amd64])\n"
out += "".join(l + "\n" for l in self.extra_sim)
return 0, out, ""
def kfake(**kw):
f = KFake()
for k, v in kw.items():
setattr(f, k, v)
return f
def kmode(f, mode, **extra):
f.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": mode, "packages": []}
f.plan.update(extra)
return run(f)
def staged(f=None):
f = f or kfake()
rc, rep = run(f)
assert rc == 0, rep
return f
class KernelLane(unittest.TestCase):
"""R-836 / `09` §3 decision 172: stage a kernel once through the ESP flag; the default moves only after a healthy
one-shot boot. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
def refused(self, f, code, mode=None, **extra):
rc, rep = kmode(f, mode, **extra) if mode else run(f)
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
return rep
# --- the four red tests the brief names (Part A 3) ---
def test_installing_a_kernel_does_not_change_the_grub_default(self):
f = kfake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertIn(NEW, f.boot, "the new kernel was installed")
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD,
"the default must stay the kernel the box runs (R-836: an install made the new one default)")
self.assertIn(f"gnulinux-{OLD}-advanced-{U}", f.tree[osapply.KERNEL_DEFAULT_CFG])
self.assertEqual(f.env, {"felhom_next": NEW}, "the ONLY way to the new kernel is the one-shot flag")
self.assertEqual(f.reboots, [], "staging never reboots")
st = json.loads(f.tree[osapply.KERNEL_STATE])
self.assertEqual((st["phase"], st["from"], st["to"]), ("staged", OLD, NEW))
self.assertEqual(rep["kernel"]["default"], OLD)
def test_a_box_without_the_esp_flag_is_refused(self):
for attr, val, frag in (("esp_fs", "ext4", "vfat"), ("efi", False, "UEFI"), ("snippets", False, "bundle"),
("mods", False, "module"), ("boot_mount", "/boot", "separate"), ("pin", True, "pinned")):
f = kfake(**{attr: val})
rep = self.refused(f, "R20")
self.assertIn(frag, rep["refused"]["reason"], attr)
self.assertNotIn(NEW, f.boot, f"{attr}: nothing may be installed on a refusal")
self.assertIsNone(f.env, f"{attr}: no flag on a refusal")
def test_a_reboot_without_the_flag_is_refused(self):
f = staged()
f.env = {"felhom_next": ""}
self.refused(f, "R22", mode="kernel-reboot")
self.assertEqual(f.reboots, [])
def test_a_kernel_outside_the_approved_set_is_refused(self):
# a signed set that names only the meta-package: the image the sources pull in was never signed for
f = kfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
f.plan.update(select="listed", packages=[dict(KERNEL_SET[0])],
signed=signed_job(packages=[KERNEL_SET[0]], op="os_kernel_step"))
self.refused(f, "R23")
self.assertNotIn(NEW, f.boot)
# a signed set names 7.0.14-22; the sources would ALSO bring 7.0.14-23
f2 = kfake()
f2.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
f2.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
f2.extra_sim = ["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"]
self.refused(f2, "R23")
# a name that is not a kernel package at all
g = kfake()
g.plan.update(select="listed", packages=[{"name": "pve-manager", "version": "9.2.22", "origin": "Proxmox Debian Repository"}])
self.refused(g, "R23")
# the household was told about another kernel
h = kfake()
h.plan["expect_kver"] = "7.0.14-23-pve"
self.refused(h, "R23")
self.assertNotIn(NEW, h.boot)
def test_the_snippet_clears_the_flag_on_use(self):
"""01_felhom_oneshot: the flag is copied, CLEARED and SAVED before any `set default`, all inside the one branch
that runs only when the flag is set; a default is set only for an installed kernel's own entry."""
text = (HERE / "felhom-grub-oneshot.sh").read_text()
body = text[text.index("cat <<EOF"):]
i_copy = body.index('set felhom_boot="\\${felhom_next}"')
i_clear = body.index("set felhom_next=\n")
i_save = body.index("save_env -f (\\$felhom_esp)/EFI/felhom/oneshot.env felhom_next")
i_default = body.index('set default="felhom-oneshot-%s"')
self.assertLess(i_copy, i_clear)
self.assertLess(i_clear, i_save)
self.assertLess(i_save, i_default, "the flag must be cleared on disk BEFORE GRUB boots anything")
self.assertIn('if [ "${felhom_boot}" = "%s" ]', body, "a default only for a kernel that is installed")
self.assertIn(osapply.ONESHOT_ENV[len(osapply.ESP_MOUNT):], text, "the wrapper and GRUB name the same env file")
# --- the rest of the stage ---
def test_option_c_options_are_on_the_one_shot_entry_only(self):
text = (HERE / "felhom-grub-oneshot-entries.sh").read_text()
self.assertIn(osapply.ONESHOT_ARGS, text)
self.assertIn("--id felhom-oneshot-$k", text)
self.assertNotIn("set default", text, "the entries file never sets the default")
def test_both_generators_ride_the_bundle(self):
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_SNIPPET][1:4], ("felhom-grub-oneshot.sh", 0o755, "sh"))
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_ENTRIES][1:4], ("felhom-grub-oneshot-entries.sh", 0o755, "sh"))
def test_ring0_pending_kernel_takes_only_the_kernel_set(self):
f = kfake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "ring0")
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["proxmox-kernel-7.0", "pve-firmware"])
self.assertEqual(f.installed["pve-manager"], "9.2.21", "a Proxmox userspace package is the pve lane's")
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3", "a Debian package is the fast lane's")
def test_signed_listed_set_is_installed(self):
f = kfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
f.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "signed")
self.assertEqual(f.env, {"felhom_next": NEW})
def test_no_authority_and_the_wrong_op_are_refused(self):
f = kfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
self.refused(f, "R3")
g = kfake()
g.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
g.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
signed=signed_job(packages=KERNEL_SET, op="os_pve_step"))
self.refused(g, "R3")
def test_fast_lane_byo_and_wrong_select_are_refused(self):
f = kfake()
f.plan["lane"] = "fast"
self.refused(f, "R3")
g = kfake()
g.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
self.refused(g, "R12")
h = kfake()
h.plan["select"] = "pending-pve"
self.refused(h, "R11")
k = kfake()
k.plan.update(layer="host", lane="fast", mode="kernel-status")
self.refused(k, "R11")
def test_the_crash_guard_must_be_armed_and_quiet(self):
f = kfake()
f.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": False, "tripped": True, "unclean_boots_in_window": 2})
self.refused(f, "R21")
g = kfake()
g.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 1})
self.refused(g, "R21")
h = kfake()
del h.files[osapply.CRASH_GUARD_STATE]
self.refused(h, "R21")
self.assertNotIn(NEW, h.boot)
def test_never_two_steps_in_one_night(self):
f = staged()
self.refused(f, "R22") # still staged
g = staged()
st = json.loads(g.tree[osapply.KERNEL_STATE])
st["phase"] = "good"
g.tree[osapply.KERNEL_STATE] = json.dumps(st)
g.clock += 3600
self.refused(g, "R22") # done, but within 20 h
def test_removal_new_non_kernel_two_kernels_and_older_are_refused(self):
f = kfake(extra_sim=["Remv pve-firmware [3.18-3]"])
self.refused(f, "R4")
g = kfake(extra_sim=["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"])
self.refused(g, "R6")
h = kfake(extra_sim=["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"])
self.refused(h, "R23")
k = kfake(running="7.0.14-23-pve")
k.boot = {"7.0.14-23-pve"}
k.cfg = k.render()
self.refused(k, "R23")
m = kfake()
m.origins = {"proxmox-kernel-7.0": DEB}
self.refused(m, "R2")
# R-898: ring 0 stages EXACTLY the told kernel (select listed, the set derived from it) — even when the sources
# offer a newer one by night; a told version that can no longer be installed is refused BEFORE any change (R7).
def test_ring0_listed_installs_the_told_kernel_not_the_newest(self):
f = kfake()
f.live["proxmox-kernel-7.0"] = {"7.0.14-20", "7.0.14-22", "7.0.2-6"}
f.live["proxmox-kernel-7.0.14-20-pve-signed"] = {"7.0.14-20"}
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "ring0")
self.assertEqual(f.env, {"felhom_next": "7.0.14-20-pve"})
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.14-20")
self.assertNotIn("7.0.14-22-pve", f.boot)
def test_ring0_told_kernel_gone_is_refused_before_any_change(self):
f = kfake()
f.live["proxmox-kernel-7.0"] = {"7.0.14-22"} # 7.0.14-20 is no longer in the archive
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
rep = self.refused(f, "R7")
self.assertIsNone(f.env)
self.assertNotIn(osapply.KERNEL_DEFAULT_CFG, f.tree, "refused before the default was even pinned")
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.2-6")
def test_nothing_pending_changes_nothing(self):
f = kfake(kernel_pending=[])
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["upgraded"], [])
self.assertIsNone(f.env)
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
def test_a_failed_install_writes_no_flag(self):
f = kfake(install_rc=100)
rc, rep = run(f)
self.assertEqual(rc, 3, rep)
self.assertIsNone(f.env)
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
def test_a_postinst_without_update_grub_is_regenerated_and_proved(self):
f = kfake(grub_runs_in_postinst=False)
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertIn(f"felhom-oneshot-{NEW}", f.cfg)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
# --- the reboot, the boot, good / revert ---
def test_reboot_of_a_staged_step(self):
f = staged()
rc, rep = kmode(f, "kernel-reboot")
self.assertEqual(rc, 0, rep)
self.assertEqual(f.reboots, [OLD])
st = json.loads(f.tree[osapply.KERNEL_STATE])
self.assertEqual(st["phase"], "oneshot")
self.assertIn("host_services", st["health_before"])
def test_reboot_needs_a_staged_step(self):
self.refused(kfake(), "R22", mode="kernel-reboot")
def boot_into(self, f, kver):
f.running = kver
if f.env and f.env.get("felhom_next"):
f.env["felhom_next"] = "" # GRUB cleared it (01_felhom_oneshot)
return kmode(f, "kernel-boot")
def test_healthy_one_shot_becomes_the_default(self):
f = staged()
kmode(f, "kernel-reboot")
rc, rep = self.boot_into(f, NEW)
self.assertEqual((rc, rep["kernel_event"], rep["kernel"]["phase"]), (0, "judging", "judging"), rep)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "judging does not move the default")
rc, rep = kmode(f, "kernel-good")
self.assertEqual(rc, 0, rep)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), NEW)
self.assertEqual(rep["kernel"]["phase"], "good")
def test_a_panic_falls_back_and_is_recorded(self):
f = staged()
kmode(f, "kernel-reboot")
rc, rep = self.boot_into(f, OLD)
self.assertEqual((rc, rep["kernel_event"]), (0, "fell_back"), rep)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
self.refused(f, "R22", mode="kernel-good")
def test_one_self_revert_then_never_again(self):
f = staged()
kmode(f, "kernel-reboot")
self.boot_into(f, NEW)
rc, rep = kmode(f, "kernel-revert", reason="the customer guest is not running")
self.assertEqual(rc, 0, rep)
self.assertEqual(f.reboots, [OLD, NEW])
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "the self-revert boots the default, the old kernel")
rc, rep = self.boot_into(f, OLD)
self.assertEqual(rep["kernel_event"], "self_reverted")
# the same step can never revert again
st = json.loads(f.tree[osapply.KERNEL_STATE])
st["phase"] = "judging"
f.tree[osapply.KERNEL_STATE] = json.dumps(st)
f.running = NEW
self.refused(f, "R22", mode="kernel-revert")
self.assertEqual(len(f.reboots), 2)
def test_a_revert_that_comes_back_new_is_never_retried(self):
f = staged()
kmode(f, "kernel-reboot")
self.boot_into(f, NEW)
kmode(f, "kernel-revert")
rc, rep = self.boot_into(f, NEW)
self.assertEqual(rep["kernel_event"], "revert_failed")
self.refused(f, "R22", mode="kernel-revert")
def test_revert_refuses_an_unknown_default(self):
f = staged()
kmode(f, "kernel-reboot")
self.boot_into(f, NEW)
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-9.9.9-1-pve-advanced-{U}"\n'
f.cfg = f.render()
self.refused(f, "R20", mode="kernel-revert")
self.assertEqual(len(f.reboots), 1)
def test_cancel_clears_the_flag(self):
f = staged()
rc, rep = kmode(f, "kernel-cancel")
self.assertEqual(rc, 0, rep)
self.assertFalse(f.env.get("felhom_next"), "the flag is gone")
self.assertEqual(rep["kernel"]["phase"], "cancelled")
def test_status_is_read_only_and_names_setup_problems(self):
f = kfake(esp_fs="ext4")
rc, rep = kmode(f, "kernel-status")
self.assertEqual(rc, 0, rep)
self.assertTrue(rep["kernel"]["setup_problems"])
self.assertEqual(f.tree, {})
self.assertFalse([c for c in f.calls if c[1][0] in ("update-grub", "systemctl", "apt-get")])
def test_facts_carry_the_kernel_lane(self):
f = staged()
f.plan = {"release_id": "facts", "layer": "host", "lane": "fast", "vmid": 9201, "mode": "facts", "packages": []}
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
h = rep["facts"]["host"]
self.assertEqual(h["kernel_lane"]["phase"], "staged")
self.assertEqual(h["kernel_next_boot"], NEW)
self.assertIn("one-shot", h["kernel_next_boot_source"])
if __name__ == "__main__":
unittest.main()
+122
View File
@@ -56,6 +56,18 @@ class FakeHost:
def log(self, line): def log(self, line):
self.logs.append(line) self.logs.append(line)
# controller-image (R-861 (a) A1): the ref arrives on stdin and is written INSIDE the guest by root.
stdin = b""
guest_writes = None
def read_stdin(self, limit):
return self.stdin[:limit + 1]
def write_guest_image(self, vmid, data):
if self.guest_writes is None:
self.guest_writes = []
self.guest_writes.append((vmid, data))
LOCAL_UNIT = """# Managed by felhom-agent — do not edit by hand. LOCAL_UNIT = """# Managed by felhom-agent — do not edit by hand.
[Unit] [Unit]
@@ -316,5 +328,115 @@ class Refuses(unittest.TestCase):
self.refused(h, ["wg", "/etc/shadow"], "A1") self.refused(h, ["wg", "/etc/shadow"], "A1")
class ControllerImage(unittest.TestCase):
"""R-861 (a) A1 (`09` §3 decision 165): the agent can no longer `tee` any image ref into the guest. The root verb
reads the ref on stdin, requires our registry + our repository + an x.y.z tag, and writes the guest file itself.
RED-PROOF: on the pre-A1 wrapper `controller-image` is not a verb (A1 usage, rc 2) — the accepted case fails."""
def go(self, ref, *argv):
h = FakeHost()
h.stdin = ref.encode() if isinstance(ref, str) else ref
return h, run(h, *(argv or ("controller-image", "9201")))
def test_our_controller_ref_is_written_in_the_guest(self):
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")
self.assertEqual(rc, 0, h.logs)
self.assertEqual(h.guest_writes, [(9201, b"gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")])
def test_a_foreign_image_is_refused(self):
for ref in ("docker.io/library/alpine:latest\n", "alpine\n",
"gitea.dooplex.hu/admin/felhom-controller:latest\n",
"gitea.dooplex.hu/admin/other:0.1.0\n",
"evil.example/admin/felhom-controller:0.301.0\n",
"gitea.dooplex.hu/admin/felhom-controller:0.301.0\nalpine\n",
"gitea.dooplex.hu/admin/felhom-controller:0.301.0 x\n",
"", "\n"):
h, rc = self.go(ref)
self.assertEqual(rc, 3, f"{ref!r} was accepted")
self.assertFalse(h.guest_writes, f"{ref!r} wrote the guest file")
self.assertTrue(any("[I1]" in l for l in h.logs), h.logs)
def test_oversize_stdin_is_refused(self):
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0" + " " * 300)
self.assertEqual(rc, 3)
self.assertFalse(h.guest_writes)
def test_vmid_must_be_numeric(self):
for argv in (("controller-image", "9201;id"), ("controller-image", "-1"), ("controller-image",),
("controller-image", "9201", "9202")):
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n", *argv)
self.assertIn(rc, (2, 3), argv)
self.assertFalse(h.guest_writes, argv)
def test_self_check_names_the_verb(self):
import io, contextlib
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
pa.main(["--self-check"])
self.assertIn("controller-image", buf.getvalue())
class UnitForget(unittest.TestCase):
"""R-1124: the agent forgets a decommissioned drive whose disk is gone — the root half removes its unit file, and
ONLY a felhom by-UUID drive unit that is disabled, not mounted, and whose /dev/disk/by-uuid link is absent.
RED-PROOF: on the pre-R-1124 wrapper `unit-forget` is not a verb (A1 usage, rc 2) — the removal case fails."""
NAME = "mnt-hdd_1.mount"
PATH = "/etc/systemd/system/mnt-hdd_1.mount"
UUID_LINK = "/dev/disk/by-uuid/91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae"
def host(self, text=LOCAL_UNIT, kind="file", links=(), mounted=()):
h = FakeHost()
h.installed = {} if text is None else {self.PATH: (kind, text.encode())}
h.links, h.mounted, h.removed = set(links), set(mounted), []
h.read_installed = lambda p: (None if p not in h.installed else
(_ for _ in ()).throw(pa.Refused("F1", f"{p} is not a regular file"))
if h.installed[p][0] != "file" else h.installed[p][1])
h.lexists = lambda p: p in h.links
h.mountpoints = lambda: set(h.mounted)
h.remove = lambda p: h.removed.append(p)
return h
def test_an_absent_disabled_unmounted_drive_unit_is_removed(self):
h = self.host()
self.assertEqual(run(h, "unit-forget", self.NAME), 0, h.logs)
self.assertEqual(h.removed, [self.PATH])
def test_already_gone_is_fine(self):
h = self.host(text=None)
self.assertEqual(run(h, "unit-forget", self.NAME), 0, h.logs)
self.assertEqual(h.removed, [])
def refused(self, h, rule, *argv):
rc = run(h, *(argv or ("unit-forget", self.NAME)))
self.assertIn(rc, (2, 3), h.logs)
self.assertEqual(h.removed, [], h.logs)
self.assertTrue(any(f"[{rule}]" in l for l in h.logs), h.logs)
def test_F5_a_present_drive_keeps_its_unit(self):
self.refused(self.host(links=[self.UUID_LINK]), "F5")
def test_F4_a_mounted_drive_keeps_its_unit(self):
self.refused(self.host(mounted=["/mnt/hdd_1"]), "F4")
def test_F3_an_enabled_unit_is_not_removed(self):
self.refused(self.host(links=["/etc/systemd/system/multi-user.target.wants/" + self.NAME]), "F3")
def test_F2_a_network_unit_or_a_foreign_unit_is_not_removed(self):
self.refused(self.host(text=LOCAL_UNIT.replace("Managed by felhom-agent", "Written by someone")), "F2")
net = LOCAL_UNIT.replace("# Managed by felhom-agent", "# Managed by felhom-agent felhom network storage")
self.refused(self.host(text=net), "F2")
def test_F1_a_symlinked_unit_is_not_followed(self):
self.refused(self.host(kind="symlink"), "F1")
def test_A1_names(self):
for argv in (("unit-forget", "mnt-hdd_1.automount"), ("unit-forget", "sshd.service"),
("unit-forget", "../mnt-hdd_1.mount"), ("unit-forget",), ("unit-forget", self.NAME, "x")):
self.refused(self.host(), "A1", *argv)
def test_U3_a_unit_whose_name_lies_about_where_is_not_removed(self):
self.refused(self.host(text=LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/mnt/other")), "U3")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main(verbosity=2) unittest.main(verbosity=2)
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-thin-check (R-1012): it lists the thin pool's device ids next to LVM's and NEVER removes
anything. Fake `lvs`, `dmsetup` and `thin_ls` record every call; nothing real is touched. Each refusal has a positive
control (the healthy read), so a test cannot pass because the helper refuses everything.
Run: python3 configs/test_felhom_thin_check.py (also run by internal/thincheck's Go test)
"""
import os, pathlib, re, subprocess, tempfile, unittest
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-thin-check"
class Box:
def __init__(self, pool_ids=(1, 2), lvm=(("vm-9201-disk-0", 1), ("vm-9201-disk-1", 2)), attr="twi-aotz--",
held="-", reserve_rc=0, thin_ls_rc=0):
self.t = tempfile.TemporaryDirectory()
r = pathlib.Path(self.t.name)
self.log = r / "calls"
self.log.write_text("")
lvm_lines = "\\n".join(f" {n},{i}" for n, i in lvm)
pool_lines = "\\n".join(str(i) for i in pool_ids)
self.bins = {}
for name, body in {
"lvs": f'''case "$*" in *lv_attr*) echo " {attr}";; *thin_id*) printf "{lvm_lines}\\n";; esac''',
"dmsetup": f'''case "$1" in
status) echo "0 112975872 thin-pool 585 6860/262144 517080/882624 {held} rw discard_passdown queue_if_no_space - 1024";;
message) [ "$4" = reserve_metadata_snap ] && exit {reserve_rc}; exit 0;;
esac''',
"thin_ls": f'''[ {thin_ls_rc} -ne 0 ] && exit {thin_ls_rc}; printf "{pool_lines}\\n"''',
}.items():
p = r / name
p.write_text(f'#!/bin/bash\necho "{name} $*" >> "{self.log}"\n{body}\n')
p.chmod(0o755)
self.bins[name] = p
self.lock = r / "lock"
def run(self, *args):
script = (f'source "{HELPER}"; LVS="{self.bins["lvs"]}"; DMSETUP="{self.bins["dmsetup"]}"; '
f'THIN_LS="{self.bins["thin_ls"]}"; LOCK="{self.lock}"; main "$@"')
r = subprocess.run(["bash", "-c", script, "x", *args], capture_output=True, text=True)
return r.returncode, r.stdout, r.stderr
def calls(self):
return self.log.read_text().splitlines()
def messages(calls):
return [c.split()[-1] for c in calls if c.startswith("dmsetup message")]
class Reads(unittest.TestCase):
def test_healthy_pool_lists_both_sides_and_releases(self):
b = Box()
rc, out, err = b.run("pve", "data")
self.assertEqual(rc, 0, err)
self.assertEqual(out.splitlines(), ["lvm-before 1 vm-9201-disk-0", "lvm-before 2 vm-9201-disk-1",
"pool 1", "pool 2",
"lvm-after 1 vm-9201-disk-0", "lvm-after 2 vm-9201-disk-1", "ok"])
self.assertEqual(messages(b.calls()), ["reserve_metadata_snap", "release_metadata_snap"])
self.assertIn("thin_ls --metadata-snap --no-headers -o DEV /dev/mapper/pve-data_tmeta", b.calls())
def test_the_2026_10_14_orphan_is_listed(self):
rc, out, _ = Box(pool_ids=(1, 2, 4)).run("pve", "data")
self.assertEqual(rc, 0)
self.assertIn("pool 4", out.splitlines())
def test_dashed_names_map_to_the_device_mapper_name(self):
b = Box()
rc, _, err = b.run("felhom-scratch", "thin-pool")
self.assertEqual(rc, 0, err)
self.assertIn("dmsetup status felhom--scratch-thin--pool-tpool", b.calls())
class Refuses(unittest.TestCase):
def refused(self, box, rc_want, *args):
rc, out, err = box.run(*args)
self.assertEqual(rc, rc_want, (out, err))
self.assertNotIn("ok", out.splitlines())
return box.calls()
def test_arguments(self):
for args in (("pve",), ("pve", "data", "x"), ("pve;id", "data"), ("-pve", "data"), ("pve", "../data"),
("pve", "da ta")):
calls = self.refused(Box(), 1, *args)
self.assertEqual(calls, [], args)
def test_not_a_thin_pool(self):
calls = self.refused(Box(attr="Vwi-aotz--"), 1, "pve", "vm-9201-disk-1")
self.assertEqual(messages(calls), [])
def test_a_snapshot_someone_else_holds_is_never_touched(self):
b = Box(held="1234")
rc, _, err = b.run("pve", "data")
self.assertEqual(rc, 3)
self.assertEqual(messages(b.calls()), []) # neither reserve nor release
# the reason line ends with the reason, not the exit code (found live on demo-hp 2026-10-17)
self.assertTrue(err.strip().endswith("not touched"), err)
def test_reserve_failure_does_not_release(self):
calls = self.refused(Box(reserve_rc=1), 3, "pve", "data")
self.assertEqual(messages(calls), ["reserve_metadata_snap"])
def test_thin_ls_failure_still_releases(self):
calls = self.refused(Box(thin_ls_rc=1), 4, "pve", "data")
self.assertEqual(messages(calls), ["reserve_metadata_snap", "release_metadata_snap"])
class NeverDeletes(unittest.TestCase):
def test_the_only_pool_messages_are_reserve_and_release(self):
src = HELPER.read_text()
sent = re.findall(r'"\$DMSETUP" message "\$TPOOL" 0 (\S+)', src)
self.assertEqual(sorted(set(sent)), ["release_metadata_snap", "reserve_metadata_snap"])
for word in ("delete", "lvremove", "thin_repair", "create_thin", "create_snap", " rm "):
code = "\n".join(l for l in src.splitlines() if not l.lstrip().startswith("#"))
self.assertNotIn(word, code, word)
if __name__ == "__main__":
unittest.main(verbosity=2)
+198
View File
@@ -0,0 +1,198 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-vzdump-partial-clean (R-1006): it removes a DEAD half-written vzdump backup and refuses
everything else. Each refusal has a positive control: the same setup without the one condition is removed — so a
test cannot pass because the helper refuses everything. A fake storage.cfg and a fake /proc; nothing real is touched.
Run: python3 configs/test_felhom_vzdump_partial_clean.py (also run by internal/backup's Go test)
"""
import os, pathlib, subprocess, tempfile, time, unittest
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-vzdump-partial-clean"
BOOT = 1_791_967_481
NAME = "vzdump-lxc-9201-2026_10_14-10_42_25"
class Box:
"""A temp tree: a dir storage `local` (path vz), an lvmthin `local-lvm`, a dir storage `other`, and a fake /proc."""
def __init__(self):
self.t = tempfile.TemporaryDirectory()
r = pathlib.Path(self.t.name)
self.vz, self.other, self.nas, self.proc = r / "vz", r / "other", r / "nas", r / "proc"
for d in (self.vz / "dump", self.other / "dump", self.nas / "dump", self.proc):
d.mkdir(parents=True)
(self.proc / "stat").write_text(f"cpu 1 2 3\nbtime {BOOT}\n")
self.cfg = r / "storage.cfg"
self.cfg.write_text(f"dir: local\n\tpath {self.vz}\n\tcontent iso,backup\n\n"
f"lvmthin: local-lvm\n\tthinpool data\n\tvgname pve\n\n"
f"dir: other\n\tpath {self.other}\n\tcontent backup\n\n"
f"nfs: nas\n\tpath {self.nas}\n\tserver 192.0.2.1\n\texport /x\n\tcontent backup\n")
self.pid = 1000
def partial(self, name=NAME + ".tar.dat", mtime=BOOT - 100, where=None, size=4096):
p = (where or self.vz / "dump") / name
if name.endswith(".tmp"):
(p / "etc" / "vzdump").mkdir(parents=True)
(p / "etc" / "vzdump" / "pct.conf").write_text("x")
for q in [p / "etc" / "vzdump" / "pct.conf", p / "etc" / "vzdump", p / "etc", p]:
os.utime(q, (mtime, mtime))
else:
p.write_bytes(b"\0" * size)
os.utime(p, (mtime, mtime))
return p
def process(self, cmdline, fds=(), cwd=None):
self.pid += 1
d = self.proc / str(self.pid)
(d / "fd").mkdir(parents=True)
(d / "cmdline").write_bytes(b"\0".join(a.encode() for a in cmdline) + b"\0")
for i, f in enumerate(fds):
os.symlink(str(f), d / "fd" / str(i + 3))
os.symlink(str(cwd or "/"), d / "cwd")
def run(self, storage, vmid, name, quiet=120):
script = (f'source "{HELPER}"; STORAGE_CFG="{self.cfg}"; PROC="{self.proc}"; QUIET_S={quiet}; '
f'main "$@"')
r = subprocess.run(["bash", "-c", script, "x", storage, str(vmid), name], capture_output=True, text=True)
return r.returncode, r.stdout + r.stderr
class Removes(unittest.TestCase):
def test_dead_tar_dat_before_boot(self):
b = Box(); p = b.partial()
rc, out = b.run("local", 9201, p.name)
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("written before this boot", out)
def test_dead_tmp_dir_before_boot(self):
b = Box(); p = b.partial(NAME + ".tmp")
rc, out = b.run("local", 9201, p.name)
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
def test_quiet_partial_after_boot(self):
b = Box(); p = b.partial(mtime=int(time.time()) - 300)
rc, out = b.run("local", 9201, p.name)
self.assertEqual(rc, 0, out); self.assertFalse(p.exists()); self.assertIn("untouched", out)
def test_qemu_vma_dat(self):
b = Box(); p = b.partial("vzdump-qemu-341-2026_10_14-10_42_25.vma.dat")
rc, out = b.run("local", 341, p.name)
self.assertEqual(rc, 0, out); self.assertFalse(p.exists())
class Keeps(unittest.TestCase):
"""Each case: (setup that must be REFUSED, the same setup that must be REMOVED)."""
def pair(self, refuse, control, quiet=120):
b = Box(); target, args = refuse(b)
rc, out = b.run(*args, quiet=quiet)
self.assertNotEqual(rc, 0, f"must refuse: {out}")
self.assertIn("REFUSED", out)
self.assertTrue(os.path.lexists(target), "a refused entry must still exist")
c = Box(); ctarget, cargs = control(c)
crc, cout = c.run(*cargs, quiet=quiet)
self.assertEqual(crc, 0, f"positive control must be removed: {cout}")
self.assertFalse(os.path.lexists(ctarget))
def test_complete_archive_of_same_vmid(self):
self.pair(lambda b: (b.partial(NAME + ".tar.zst"), ("local", 9201, NAME + ".tar.zst")),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_log_and_notes(self):
for suffix in (".log", ".tar.zst.notes"):
self.pair(lambda b, s=suffix: (b.partial(NAME + s), ("local", 9201, NAME + s)),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_partial_of_another_vmid(self):
other = "vzdump-lxc-9202-2026_10_14-10_42_25.tar.dat"
self.pair(lambda b: (b.partial(other), ("local", 9201, other)),
lambda b: (b.partial(other), ("local", 9202, other)))
def test_partial_on_a_non_dir_storage(self):
self.pair(lambda b: (b.partial(), ("local-lvm", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_partial_on_a_network_storage_with_a_path(self):
# an nfs storage HAS a path: only the „dir only" rule (both its guards) can refuse it
self.pair(lambda b: (b.partial(where=b.nas / "dump"), ("nas", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
def test_partial_on_another_storage_than_named(self):
self.pair(lambda b: (b.partial(where=b.other / "dump"), ("local", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(where=b.other / "dump"), ("other", 9201, NAME + ".tar.dat")))
def test_unknown_storage(self):
self.pair(lambda b: (b.partial(), ("nosuch", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_vzdump_task_in_flight(self):
def refuse(b):
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzdump:9201:felhom-agent@pve!agent:"]); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzrestore:9202:root@pam:"]); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_any_vzdump_job_even_without_the_vmid(self):
# a job for several guests: `vzdump::` — and a vzdump for ANOTHER guest — both refuse
for cmd in (["task UPID:demo-hp:001:002:6ACF4071:vzdump::root@pam:"],
["task UPID:demo-hp:001:002:6ACF4071:vzdump:9202:felhom-agent@pve!agent:"]):
def refuse(b, c=cmd):
p = b.partial(); b.process(c); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(); b.process(["task UPID:demo-hp:001:002:6ACF4071:vzstart:9202:root@pam:"]); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_vzdump_command_for_the_vmid(self):
def refuse(b):
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdump", "9201", "--storage", "local"]); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(); b.process(["/usr/bin/perl", "/usr/bin/vzdumpfoo", "9201"]); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_file_held_open_by_an_orphan_writer(self):
def refuse(b):
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[p]); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(); b.process(["zstd", "--threads=1"], fds=[b.vz / "dump" / "something-else"]); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_tmp_dir_with_a_process_inside(self):
def refuse(b):
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=p / "etc"); return p, ("local", 9201, p.name)
def control(b):
p = b.partial(NAME + ".tmp"); b.process(["tar", "cpf", "-"], cwd=b.vz); return p, ("local", 9201, p.name)
self.pair(refuse, control)
def test_recently_written_after_boot(self):
self.pair(lambda b: (b.partial(mtime=int(time.time()) - 10), ("local", 9201, NAME + ".tar.dat")),
lambda b: (b.partial(mtime=int(time.time()) - 300), ("local", 9201, NAME + ".tar.dat")))
def test_symlink(self):
def refuse(b):
real = b.partial(where=b.vz)
link = b.vz / "dump" / (NAME + ".tar.dat")
os.symlink(real, link)
return link, ("local", 9201, link.name)
# quiet=0: the link was made just now, so only the symlink rule may refuse it
self.pair(refuse, lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")), quiet=0)
def test_near_miss_names(self):
for bad in (NAME + ".tar.dat.bak", "vzdump-lxc-92011-2026_10_14-10_42_25.tar.dat", NAME + ".tar.da",
"x" + NAME + ".tar.dat", NAME + ".TAR.DAT", "vzdump-lxc-9201-2026_10_14-10_42.tar.dat"):
self.pair(lambda b, n=bad: (b.partial(n), ("local", 9201, n)),
lambda b: (b.partial(), ("local", 9201, NAME + ".tar.dat")))
def test_path_traversal_and_bad_args(self):
b = Box(); p = b.partial()
for args in (("local", 9201, "../dump/" + p.name), ("local", 9201, ""), ("loc al", 9201, p.name),
("local", "9201x", p.name), ("local", 12, p.name)):
rc, out = b.run(*args)
self.assertNotEqual(rc, 0, f"{args}: {out}")
self.assertTrue(p.exists())
def test_wrong_kind(self):
def refuse(b): # a FILE named .tmp
p = b.vz / "dump" / (NAME + ".tmp"); p.write_bytes(b"x"); os.utime(p, (BOOT - 100, BOOT - 100)); return p, ("local", 9201, p.name)
self.pair(refuse, lambda b: (b.partial(NAME + ".tmp"), ("local", 9201, NAME + ".tmp")))
if __name__ == "__main__":
unittest.main()
+5 -4
View File
@@ -1,7 +1,8 @@
module gitea.dooplex.hu/admin/felhom-agent module gitea.dooplex.hu/admin/felhom-agent
go 1.25.0 go 1.27.3
require golang.org/x/crypto v0.52.0 require (
golang.org/x/crypto v0.58.0
require golang.org/x/sys v0.45.0 // indirect golang.org/x/sys v0.49.0
)
+6 -6
View File
@@ -1,6 +1,6 @@
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988= golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc= golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4= golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk= golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
+2
View File
@@ -22,6 +22,8 @@ var (
ErrExpired = errors.New("authz: op expired") ErrExpired = errors.New("authz: op expired")
// ErrNotYetValid: now < issued_at (minus clock-skew tolerance). // ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
ErrNotYetValid = errors.New("authz: op not yet valid") ErrNotYetValid = errors.New("authz: op not yet valid")
// ErrWindowTooLong: expires_at - issued_at > MaxWindow (R-1280).
ErrWindowTooLong = errors.New("authz: op validity window too long")
// ErrReplay: the nonce was already recorded in the window. // ErrReplay: the nonce was already recorded in the window.
ErrReplay = errors.New("authz: replay (nonce already seen)") ErrReplay = errors.New("authz: replay (nonce already seen)")
) )
+8
View File
@@ -20,6 +20,11 @@ const Namespace = "felhom-op-v1"
// the validity window stays an honest upper bound. // the validity window stays an honest upper bound.
const DefaultClockSkew = 2 * time.Minute const DefaultClockSkew = 2 * time.Minute
// MaxWindow is the longest validity window a signed op may carry (R-1280, D44 #22, operator 2026-10-25): a job signed
// with a long window by mistake would stay usable that long, and the nonce store would keep its record as long. The
// root wrapper (configs/felhom-os-apply, MAX_WINDOW_S) and felhom-opsign (-ttl) hold the same 24 h.
const MaxWindow = 24 * time.Hour
// KeyRole tags a pinned operator key (doc 04 §3 two-key model). // KeyRole tags a pinned operator key (doc 04 §3 two-key model).
type KeyRole string type KeyRole string
@@ -150,6 +155,9 @@ func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
if now.Before(op.IssuedAt.Add(-v.ClockSkew)) { if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now) return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
} }
if w := op.ExpiresAt.Sub(op.IssuedAt); w > MaxWindow {
return nil, fmt.Errorf("%w: issued_at=%s expires_at=%s (%s, the most is %s)", ErrWindowTooLong, op.IssuedAt, op.ExpiresAt, w, MaxWindow)
}
if now.After(op.ExpiresAt) { if now.After(op.ExpiresAt) {
return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now) return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
} }
+20
View File
@@ -117,6 +117,26 @@ func TestVerify_RejectsPerStage(t *testing.T) {
rejects(t, err, ErrTarget) rejects(t, err, ErrTarget)
}) })
// R-1280: a window longer than 24 h is refused even while it is current; exactly 24 h is accepted.
// RED-PROOF: drop the MaxWindow check → the 24 h + 1 s case verifies.
t.Run("window longer than 24 h", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-1", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow+time.Second))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
_, err := v.Verify(blob, sig)
rejects(t, err, ErrWindowTooLong)
})
t.Run("window of exactly 24 h is accepted", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-2", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
if _, err := v.Verify(blob, sig); err != nil {
t.Fatalf("a 24 h window must verify: %v", err)
}
})
t.Run("expired", func(t *testing.T) { t.Run("expired", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}", blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
refNow.Add(-2*time.Hour), refNow.Add(-time.Hour)) refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
+244
View File
@@ -0,0 +1,244 @@
package backup
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"strconv"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// BackupSuccessState persists the newest SUCCESSFUL whole-guest backup per tier and guest (R-894).
//
// Why it exists. The due-check (`localapi` handleBackupDue) asks the tier's storage when a backup last
// landed (R-84) and falls back to the in-memory record when the storage cannot be read. The in-memory
// record is empty after an agent restart (Store, R-348), so "storage unreadable" right after a restart
// read as "no record — DUE". Measured 2026-10-05 on demo-hp: the agent restarted at 04:57, the off-site
// storage answered "Can't connect" at 06:25, the 7-day tier — last copy 2026-10-01 — read DUE, the
// controller asked, and vzdump failed. This file is the last known copy the fallback reads instead.
//
// It is read ONLY when the storage cannot be read. A storage that answers is the ground truth and wins,
// in both directions: an archive found there counts, and an archive absent there is absent even when
// this file remembers a success (a pruned or deleted archive must make the tier due — the same reason
// R-84 chose the storage over a persisted record). Pinned by
// TestBackupDue_R894_SavedCopyIgnoredWhenStorageAnswers.
//
// Only SUCCESSES are written (the RestoreTestState rule): a failure must stay due and be retried, so a
// record of a failure has no reader.
type BackupSuccessState struct {
path string
mu sync.Mutex
last map[string]savedSuccess // key(target, vmid) → the newest success
}
type savedSuccess struct {
target string
vmid int
at time.Time
// recent (R-973) holds the start times of the newest successes (at most maxRecentSuccesses, newest
// last) — what the local-API backup cadence cap counts, so an agent restart does not reset the cap.
recent []time.Time
// refusedAt (R-973) is the newest time the cap refused a backup of this guest on this tier.
refusedAt time.Time
}
// maxRecentSuccesses bounds the per-tier success history kept for the cadence cap (R-973). The cap
// allows at most 3 runs in a window, so 8 is ample.
const maxRecentSuccesses = 8
// backupSuccessJSON is one entry on disk. Recent and LastRefusedAt are additive (R-973): a file written
// before them reads as "no history", and an older agent ignores them.
type backupSuccessJSON struct {
Target string `json:"target"`
VMID int `json:"vmid"`
StartedAt string `json:"started_at"`
Recent []string `json:"recent,omitempty"`
LastRefusedAt string `json:"last_refused_at,omitempty"`
}
func backupStateKey(target string, vmid int) string { return target + "/" + strconv.Itoa(vmid) }
// NewBackupSuccessState opens (or creates) the state at path. A missing or unreadable file degrades to
// "nothing known" — the pre-R-894 behaviour, which is DUE — and never wedges the daemon.
func NewBackupSuccessState(path string) *BackupSuccessState {
s := &BackupSuccessState{path: path, last: map[string]savedSuccess{}}
data, err := os.ReadFile(path)
if err != nil {
return s
}
var entries []backupSuccessJSON
if json.Unmarshal(data, &entries) != nil {
return s
}
for _, e := range entries {
t, perr := time.Parse(time.RFC3339, e.StartedAt)
if perr != nil {
continue // one unreadable entry must not lose the others
}
ss := savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()}
for _, r := range e.Recent {
if rt, rerr := time.Parse(time.RFC3339, r); rerr == nil {
ss.recent = append(ss.recent, rt.UTC())
}
}
if len(ss.recent) == 0 {
ss.recent = []time.Time{ss.at} // a pre-R-973 file: its one known success counts
}
if rt, rerr := time.Parse(time.RFC3339, e.LastRefusedAt); rerr == nil {
ss.refusedAt = rt.UTC()
}
s.last[backupStateKey(e.Target, e.VMID)] = ss
}
return s
}
// RecordBackupSuccess saves b when it is a success newer than the one on file. target is the tier the
// job ran on (the due-check's key); a failure or an unparseable time is ignored.
func (s *BackupSuccessState) RecordBackupSuccess(target string, b hub.Backup) error {
if s == nil || !b.Success {
return nil
}
t, err := time.Parse(time.RFC3339, b.StartedAt)
if err != nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
k := backupStateKey(target, b.VMID)
old, had := s.last[k]
if had && !t.After(old.at) {
return nil
}
ss := savedSuccess{target: target, vmid: b.VMID, at: t.UTC(), refusedAt: old.refusedAt}
ss.recent = append(append([]time.Time(nil), old.recent...), t.UTC())
if len(ss.recent) > maxRecentSuccesses {
ss.recent = ss.recent[len(ss.recent)-maxRecentSuccesses:]
}
s.last[k] = ss
return s.saveLocked()
}
// RecentBackupSuccesses returns the saved success start times for this tier and guest that are not
// older than since (R-973 — the cadence cap's restart-proof count).
func (s *BackupSuccessState) RecentBackupSuccesses(target string, vmid int, since time.Time) []time.Time {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
var out []time.Time
for _, t := range s.last[backupStateKey(target, vmid)].recent {
if !t.Before(since) {
out = append(out, t)
}
}
return out
}
// RecordBackupRefusal saves that the cadence cap refused a backup of this guest on this tier at at
// (R-973), so the operator signal it raises survives an agent restart.
func (s *BackupSuccessState) RecordBackupRefusal(target string, vmid int, at time.Time) error {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
k := backupStateKey(target, vmid)
ss, ok := s.last[k]
if !ok {
return nil // nothing to cap against without a success; the cap never refuses then
}
if !at.After(ss.refusedAt) {
return nil
}
ss.refusedAt = at.UTC()
s.last[k] = ss
return s.saveLocked()
}
// LastBackupRefusal returns the newest saved cadence-cap refusal across every tier and guest.
func (s *BackupSuccessState) LastBackupRefusal() (target string, vmid int, at time.Time, ok bool) {
if s == nil {
return "", 0, time.Time{}, false
}
s.mu.Lock()
defer s.mu.Unlock()
for _, e := range s.last {
if !e.refusedAt.IsZero() && e.refusedAt.After(at) {
target, vmid, at, ok = e.target, e.vmid, e.refusedAt, true
}
}
return target, vmid, at, ok
}
// LastKnownSuccess returns the newest saved success for this tier and guest (ok=false = none on file).
func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Time, bool) {
if s == nil {
return time.Time{}, false
}
s.mu.Lock()
defer s.mu.Unlock()
e, ok := s.last[backupStateKey(target, vmid)]
return e.at, ok
}
// KnownBackupSuccesses returns every saved success as a host-report record (the collector's
// KnownBackupReporter, so the hub keeps its evidence across a restart). Only the tier, guest and start
// time are known; the archive name is not saved and stays empty. Sorted for a stable report.
func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
out := make([]hub.Backup, 0, len(s.last))
for _, e := range s.last {
out = append(out, hub.Backup{TargetID: e.target, VMID: e.vmid, Success: true, CrashConsistent: true,
StartedAt: e.at.Format(time.RFC3339), UncoveredVolumes: []string{}})
}
sort.Slice(out, func(i, j int) bool {
if out[i].TargetID != out[j].TargetID {
return out[i].TargetID < out[j].TargetID
}
return out[i].VMID < out[j].VMID
})
return out
}
func (s *BackupSuccessState) saveLocked() error {
entries := make([]backupSuccessJSON, 0, len(s.last))
for _, e := range s.last {
j := backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)}
for _, r := range e.recent {
j.Recent = append(j.Recent, r.Format(time.RFC3339))
}
if !e.refusedAt.IsZero() {
j.LastRefusedAt = e.refusedAt.Format(time.RFC3339)
}
entries = append(entries, j)
}
// Deterministic file content (Go's map order is random).
sort.Slice(entries, func(i, j int) bool {
if entries[i].Target != entries[j].Target {
return entries[i].Target < entries[j].Target
}
return entries[i].VMID < entries[j].VMID
})
data, err := json.MarshalIndent(entries, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
return err
}
tmp := s.path + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, s.path)
}
+113
View File
@@ -0,0 +1,113 @@
package backup
import (
"os"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-894: the on-disk newest success per tier survives a restart (a new state from the same file).
func TestBackupSuccessState_SurvivesRestart(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
at := time.Date(2026, 10, 1, 20, 15, 0, 0, time.UTC)
if err := s.RecordBackupSuccess("felhom-pbs", hub.Backup{VMID: 9201, Success: true, StartedAt: at.Format(time.RFC3339)}); err != nil {
t.Fatal(err)
}
got, ok := NewBackupSuccessState(path).LastKnownSuccess("felhom-pbs", 9201)
if !ok || !got.Equal(at) {
t.Fatalf("after a restart the saved copy must read back; got %v ok=%v", got, ok)
}
if _, ok := NewBackupSuccessState(path).LastKnownSuccess("local", 9201); ok {
t.Fatal("another tier must not borrow this tier's copy")
}
}
// Only a NEWER success replaces the saved one; failures and unparseable times are ignored.
func TestBackupSuccessState_KeepsNewestSuccessOnly(t *testing.T) {
path := filepath.Join(t.TempDir(), "s.json")
s := NewBackupSuccessState(path)
newer := time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC)
older := newer.Add(-48 * time.Hour)
for _, b := range []hub.Backup{
{VMID: 1, Success: true, StartedAt: newer.Format(time.RFC3339)},
{VMID: 1, Success: true, StartedAt: older.Format(time.RFC3339)}, // older: ignored
{VMID: 1, Success: false, StartedAt: newer.Add(time.Hour).Format(time.RFC3339)}, // failure: ignored
{VMID: 1, Success: true, StartedAt: "not-a-time"}, // unparseable: ignored
} {
if err := s.RecordBackupSuccess("t", b); err != nil {
t.Fatal(err)
}
}
if got, _ := NewBackupSuccessState(path).LastKnownSuccess("t", 1); !got.Equal(newer) {
t.Fatalf("want the newest success %v, got %v", newer, got)
}
}
// A corrupt file degrades to "nothing known" (the pre-R-894 DUE answer), never a crash.
func TestBackupSuccessState_CorruptFileIsEmpty(t *testing.T) {
path := filepath.Join(t.TempDir(), "s.json")
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
t.Fatal(err)
}
if _, ok := NewBackupSuccessState(path).LastKnownSuccess("t", 1); ok {
t.Fatal("a corrupt file must read as nothing known")
}
}
// The saved successes come back as host-report records: success, tier, guest, start time (R-894 → the
// host report, 2026-10-09). A new state read from the same file gives the same records (the restart case).
func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
if err := s.RecordBackupSuccess("felhom-backup", hub.Backup{VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}); err != nil {
t.Fatal(err)
}
got := NewBackupSuccessState(path).KnownBackupSuccesses()
if len(got) != 1 || !got[0].Success || got[0].TargetID != "felhom-backup" || got[0].VMID != 9201 || got[0].StartedAt != "2026-10-09T02:40:16Z" {
t.Fatalf("got %+v", got)
}
if got[0].UncoveredVolumes == nil {
t.Fatal("uncovered_volumes must marshal as [], not null")
}
}
// R-973: the cadence cap's history and refusals survive a reopen; the history is bounded; a file
// written before R-973 counts its one known success (never "no history" for a tier that has a copy).
func TestBackupSuccessState_R973_RecentHistoryAndRefusalSurviveAReopen(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
base := time.Date(2026, 10, 18, 2, 30, 0, 0, time.UTC)
for i := 0; i < 10; i++ {
at := base.Add(time.Duration(i) * time.Hour)
if err := s.RecordBackupSuccess("local", hub.Backup{VMID: 9201, Success: true, StartedAt: at.Format(time.RFC3339)}); err != nil {
t.Fatal(err)
}
}
refused := base.Add(11 * time.Hour)
if err := s.RecordBackupRefusal("local", 9201, refused); err != nil {
t.Fatal(err)
}
r := NewBackupSuccessState(path)
all := r.RecentBackupSuccesses("local", 9201, time.Time{})
if len(all) != maxRecentSuccesses || !all[len(all)-1].Equal(base.Add(9*time.Hour)) {
t.Fatalf("recent history after a reopen: %v (want the newest %d, ending at +9h)", all, maxRecentSuccesses)
}
if got := r.RecentBackupSuccesses("local", 9201, base.Add(7*time.Hour)); len(got) != 3 {
t.Fatalf("successes since +7h: %d, want 3", len(got))
}
if tg, vm, at, ok := r.LastBackupRefusal(); !ok || tg != "local" || vm != 9201 || !at.Equal(refused) {
t.Fatalf("refusal after a reopen: %q %d %v %v", tg, vm, at, ok)
}
old := filepath.Join(t.TempDir(), "old.json")
if err := os.WriteFile(old, []byte(`[{"target":"felhom-pbs","vmid":9201,"started_at":"2026-10-17T03:00:00Z"}]`), 0o600); err != nil {
t.Fatal(err)
}
if got := NewBackupSuccessState(old).RecentBackupSuccesses("felhom-pbs", 9201, time.Time{}); len(got) != 1 {
t.Fatalf("a pre-R-973 file must count its one success: %v", got)
}
}
+1 -1
View File
@@ -2,9 +2,9 @@ package backup
import ( import (
"context" "context"
"fmt"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"io" "io"
"log/slog" "log/slog"
"testing" "testing"
+60
View File
@@ -0,0 +1,60 @@
package backup
import (
"context"
"sort"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// ForeignKeyLedger (R-366 slice 2, `09` §3 decision 168) holds, per backup tier, the whole-guest archives the
// restore-test pick skipped because another key wrote them (R-727 — an earlier install of this box). Since R-727 the
// skip was one INFO log line per archive and nothing else, so after a reinstall the operator was never told that the
// box's older whole-guest copies are unreadable to it. The host report carries this ledger; the hub raises ONE
// operator event when it changes.
//
// It reports nil until a tier has been evaluated since the agent started, so a restart does not read as "the set
// changed to empty" (the hub keeps its last state for an absent field).
type ForeignKeyLedger struct {
mu sync.Mutex
byTarget map[string]hub.ForeignKeyArchives
}
// NewForeignKeyLedger builds an empty ledger.
func NewForeignKeyLedger() *ForeignKeyLedger {
return &ForeignKeyLedger{}
}
func (l *ForeignKeyLedger) set(target string, n int, oldest, newest int64) {
l.mu.Lock()
defer l.mu.Unlock()
if l.byTarget == nil {
l.byTarget = map[string]hub.ForeignKeyArchives{}
}
e := hub.ForeignKeyArchives{Target: target, Count: n}
if n > 0 {
e.Oldest = time.Unix(oldest, 0).UTC().Format(time.RFC3339)
e.Newest = time.Unix(newest, 0).UTC().Format(time.RFC3339)
}
l.byTarget[target] = e
}
// ForeignKeyArchives implements hub.ForeignKeyArchiveReporter: nil before any evaluation; otherwise the tiers that
// hold such archives (`Tiers` empty, never nil, when none do), sorted by tier.
func (l *ForeignKeyLedger) ForeignKeyArchives(context.Context) *hub.ForeignKeyArchivesStanza {
l.mu.Lock()
defer l.mu.Unlock()
if l.byTarget == nil {
return nil
}
out := []hub.ForeignKeyArchives{}
for _, e := range l.byTarget {
if e.Count > 0 {
out = append(out, e)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Target < out[j].Target })
return &hub.ForeignKeyArchivesStanza{Tiers: out}
}
+2 -2
View File
@@ -93,7 +93,7 @@ func TestPrune_KeepLastClamp(t *testing.T) {
t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set) t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set)
} }
} }
if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-last=3" { if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-daily=3" {
t.Fatalf("default PruneBackupsSpec = %q, want keep-last=3", spec) t.Fatalf("default PruneBackupsSpec = %q, want keep-daily=3 (R-1149)", spec)
} }
} }
+67
View File
@@ -0,0 +1,67 @@
package backup
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1012: a snapshot backup that fails carries the orphan finding in its error — the text the controller mails to
// the operator (`whole_guest_backup_failed`). The check runs ONCE, after the failure — never before the run.
// RED-PROOF (P1/redproof-thincheck.txt): with explainFailure's append removed this test fails. RED-PROOF (2026-10-17
// night, felhom-agent REPORT.md): with the v0.159.0 pre-run check restored, calls = 2 and this test fails.
func TestBackup_SnapshotFailure_NamesTheOrphan(t *testing.T) {
const why = "probable cause: thin pool pve/data holds device 4 that LVM does not know"
for name, api := range map[string]*fakeBackupAPI{
"task fails": {vzdumpUPID: "UPID:x", waitErr: errors.New("task failed: job errors")},
"start fails": {vzdumpErr: errors.New("vzdump: 500")},
} {
t.Run(name, func(t *testing.T) {
r := NewBackupRunner(api, "local", "", "", "", quiet())
calls := 0
r.SetThinCheck(func(context.Context, int) string { calls++; return why })
rec, err := r.Backup(context.Background(), 9201)
if err == nil || rec.Success {
t.Fatalf("want a failure, got %+v", rec)
}
if !strings.Contains(rec.Error, why) {
t.Fatalf("the failure does not name the orphan: %q", rec.Error)
}
if calls != 1 {
t.Fatalf("thin check ran %d times, want 1 (after the failure only)", calls)
}
})
}
}
// A successful backup never runs the check (it takes a metadata snapshot of the live pool, and nobody needs the answer)
// and keeps a clean record; a stop-mode backup takes no snapshot, so the check never runs; an unwired runner behaves
// exactly as before. RED-PROOF (2026-10-17 night): with the v0.159.0 pre-run check restored, the success leg fails
// with calls=1.
func TestBackup_ThinCheck_SilentWhenNotApplicable(t *testing.T) {
ok := &fakeBackupAPI{vzdumpUPID: "UPID:x",
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-x", Content: "backup", VMID: 9201, Size: 100, CTime: 1}}}
r := NewBackupRunner(ok, "local", "", "", "", quiet())
calls := 0
r.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
rec, err := r.Backup(context.Background(), 9201)
if err != nil || !rec.Success || rec.Error != "" || calls != 0 {
t.Fatalf("success: err=%v rec=%+v calls=%d (want no check on a successful backup, clean record)", err, rec, calls)
}
stop := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", proxmox.ModeStop, "", "", quiet())
calls = 0
stop.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
rec, _ = stop.Backup(context.Background(), 9201)
if calls != 0 || strings.Contains(rec.Error, "probable cause") {
t.Fatalf("stop mode: calls=%d error=%q", calls, rec.Error)
}
plain := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", "", "", "", quiet())
if rec, _ := plain.Backup(context.Background(), 9201); rec.Error != "boom" {
t.Fatalf("unwired: error = %q, want the task error unchanged", rec.Error)
}
}
@@ -0,0 +1,55 @@
package backup
import (
"context"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1232 — the 2026-10-23 test night: a reinstalled box (new install, new PBS key, same customer namespace, same guest
// number 9201) found the OLD box's archive (2.6 days old) when the due-check asked „when did this tier last land?", so
// the whole-guest tier read „not due" for up to its 168-h cadence and the ep0 copy stayed the OLD box's. The due-check
// now counts only archives written with THIS storage's key (the R-727 discriminator; the archive carries no host id).
// The CONSEQUENCE asserted: with only a foreign archive the answer is „none found" (the tier is due); with the box's
// own archive, that archive's time. COMPANION RED-PROOF: drop the key filter in NewestArchiveTime → case 1 returns the
// old box's time.
func TestR1232_TheDueCheckCountsOnlyThisBoxsArchives(t *testing.T) {
oldBox := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-20T21:10:00Z", Content: "backup", VMID: 9201,
Size: 3490689830, CTime: time.Date(2026, 10, 20, 21, 10, 0, 0, time.UTC).Unix(), Encrypted: earlierBox1}
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbsdr", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{oldBox},
}
r := NewBackupRunner(api, "felhom-pbsdr", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
// 1. The fresh box has written nothing yet: the old box's archive must not count.
when, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatal(err)
}
if found {
t.Fatalf("the due-check counted ANOTHER box's archive (%s) as this tier's last backup — the tier would wait a week", when)
}
// 2. After the box's own first copy, that copy is the answer.
own := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-24T02:30:00Z", Content: "backup", VMID: 9201,
Size: 3600000000, CTime: time.Date(2026, 10, 24, 2, 30, 0, 0, time.UTC).Unix(), Encrypted: thisBoxKey}
api.content = append(api.content, own)
when, found, err = r.NewestArchiveTime(context.Background(), 9201)
if err != nil || !found || when.Unix() != own.CTime {
t.Fatalf("want this box's own archive time %d, got %v found=%v err=%v", own.CTime, when, found, err)
}
}
// An unencrypted storage (a local dir) holds only this box's vzdumps: no key filter, the newest counts.
func TestR1232_UnencryptedStorageCountsEveryArchive(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_10_25-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1793050025}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, found, err := r.NewestArchiveTime(context.Background(), 9201); err != nil || !found {
t.Fatalf("an unencrypted local archive was not counted: found=%v err=%v", found, err)
}
}
@@ -0,0 +1,69 @@
package backup
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-366 slice 2 (`09` §3 decision 168) — the restore-test's skip of an archive written with another key stops being
// silent: the pick records, per tier, how many it skipped and their time range, and the host report carries it.
//
// COMPANION RED-PROOF (observed): remove the `r.foreign.set(...)` call from PickSettledRestoreCandidateOn → this fails
// with "after one evaluation the ledger must report felhom-pbs: 2 archives …; got []". Restored.
func TestR366_PickRecordsArchivesWrittenWithAnotherKey(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey},
},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
l := NewForeignKeyLedger()
r.SetForeignKeyLedger(l)
if got := l.ForeignKeyArchives(context.Background()); got != nil {
t.Fatalf("before any evaluation the ledger must be nil (the hub keeps its state); got %v", got)
}
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
t.Fatal(err)
}
st := l.ForeignKeyArchives(context.Background())
want := hub.ForeignKeyArchives{Target: "felhom-pbs", Count: 2, Oldest: "2026-09-16T17:27:32Z", Newest: "2026-09-16T21:59:54Z"}
var got []hub.ForeignKeyArchives
if st != nil {
got = st.Tiers
}
if len(got) != 1 || got[0] != want {
t.Fatalf("after one evaluation the ledger must report felhom-pbs: 2 archives 2026-09-16T17:27:32Z…21:59:54Z; got %v", got)
}
}
// Evaluated and none found → the stanza with `tiers: []`; not evaluated → no stanza at all. Never a null on the wire.
func TestR366_EvaluatedWithNoneIsAnEmptyList(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
l := NewForeignKeyLedger()
r.SetForeignKeyLedger(l)
b, _ := json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
if strings.Contains(string(b), "foreign_key_archives") {
t.Fatalf("not evaluated must omit the stanza; got %s", b)
}
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
t.Fatal(err)
}
b, _ = json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
if !strings.Contains(string(b), `"foreign_key_archives":{"tiers":[]}`) {
t.Fatalf("evaluated with none must report tiers: []; got %s", b)
}
}
+25
View File
@@ -81,3 +81,28 @@ func TestR685_FailsOpen(t *testing.T) {
} }
} }
} }
// TestR1006_LeftoverCleanupRunsBeforeThePreflight — the night of 2026-10-14: a 12.2 GB half-written archive from a power
// cut held the space, so every later backup was skipped. The cleanup step runs BEFORE the preflight reads the free space:
// here it „removes" the leftover (free 4 → 16 GiB) and the backup then fits and runs.
//
// RED-PROOF: drop the r.beforeBackup call from backup() (or move it after spaceFits) → the backup is skipped → fails.
func TestR1006_LeftoverCleanupRunsBeforeThePreflight(t *testing.T) {
api := spaceAPI(4*gib, 8182759056, "dir")
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
var got []string
r.SetBeforeBackup(func(_ context.Context, storage string, vmid int) {
got = append(got, storage)
if vmid != 9201 {
t.Errorf("cleanup asked for vmid %d", vmid)
}
api.storages[0].Avail = 16 * gib
})
_, _ = r.Backup(context.Background(), 9201)
if len(got) != 1 || got[0] != "local" {
t.Fatalf("the cleanup must run once for this target, got %v", got)
}
if len(api.vzdumps) != 1 {
t.Fatalf("after the cleanup freed the space the backup must run, vzdumps=%d", len(api.vzdumps))
}
}
+65
View File
@@ -58,6 +58,14 @@ type BackupRunner struct {
// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value // tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
// there is a deliberate act. // there is a deliberate act.
allowPBSPrune bool allowPBSPrune bool
// beforeBackup runs just before the space preflight (R-1006): it removes a dead half-written vzdump of this guest on
// this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing.
beforeBackup func(ctx context.Context, storage string, vmid int)
// thinCheck (R-1012) names an orphan thin device in the guest's pools in plain words ("" = none / not checked). It
// runs ONLY when a snapshot-mode vzdump has failed (start or task), so the failure the controller mails to the
// operator says why; a successful backup never runs it (it takes a metadata snapshot of the live pool). It never
// removes anything. nil = not wired.
thinCheck func(ctx context.Context, vmid int) string
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete // rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
@@ -66,8 +74,13 @@ type BackupRunner struct {
// due-check is served from the local-API handler goroutines. // due-check is served from the local-API handler goroutines.
rejectedMu sync.Mutex rejectedMu sync.Mutex
rejected map[string]struct{} rejected map[string]struct{}
// foreign (R-366 slice 2) records, per tier, the archives the pick skipped as another key's. nil = not wired.
foreign *ForeignKeyLedger
} }
// SetForeignKeyLedger wires the R-366 slice-2 ledger the host report reads.
func (r *BackupRunner) SetForeignKeyLedger(l *ForeignKeyLedger) { r.foreign = l }
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and // NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the // for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it. // per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
@@ -97,6 +110,27 @@ func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode,
now: func() time.Time { return time.Now().UTC() }} now: func() time.Time { return time.Now().UTC() }}
} }
// SetBeforeBackup sets the step run just before the space preflight (R-1006: the dead-leftover cleanup).
func (r *BackupRunner) SetBeforeBackup(fn func(ctx context.Context, storage string, vmid int)) {
r.beforeBackup = fn
}
// SetThinCheck wires the R-1012 orphan check (see BackupRunner.thinCheck).
func (r *BackupRunner) SetThinCheck(fn func(ctx context.Context, vmid int) string) { r.thinCheck = fn }
// explainFailure appends the orphan finding to a failed snapshot-mode backup's error — the only place the check runs, so
// it sees the pool as the failure left it. Bounded, so a stuck helper cannot hold the job.
func (r *BackupRunner) explainFailure(ctx context.Context, vmid int, rec *hub.Backup) {
if r.thinCheck == nil || r.mode != proxmox.ModeSnapshot {
return
}
cctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute)
defer cancel()
if why := r.thinCheck(cctx, vmid); why != "" {
rec.Error = rec.Error + " — " + why
}
}
// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It // localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
// applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a // applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a
// separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type // separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type
@@ -178,6 +212,9 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
// R-685 (v0.134.0): will the new archive FIT on a local target? Asked before anything runs, so a // R-685 (v0.134.0): will the new archive FIT on a local target? Asked before anything runs, so a
// target that cannot hold it is a named SKIP with the numbers, not a nightly "No space left on // target that cannot hold it is a named SKIP with the numbers, not a nightly "No space left on
// device" that only the vzdump log explains (demo-hp, every night from 2026-09-23 — R-684). // device" that only the vzdump log explains (demo-hp, every night from 2026-09-23 — R-684).
if r.beforeBackup != nil {
r.beforeBackup(ctx, r.target, vmid)
}
if ok, why := r.spaceFits(ctx, vmid); !ok { if ok, why := r.spaceFits(ctx, vmid); !ok {
rec.Error = BackupSkipNoSpacePrefix + why rec.Error = BackupSkipNoSpacePrefix + why
rec.DurationSeconds = time.Since(start).Seconds() rec.DurationSeconds = time.Since(start).Seconds()
@@ -185,12 +222,15 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
return rec, fmt.Errorf("backup: %s", rec.Error) return rec, fmt.Errorf("backup: %s", rec.Error)
} }
// R-1012: no orphan check here. It takes a metadata snapshot of the live pool, and its answer is only needed when the
// snapshot step has failed — explainFailure runs it then (pinned by TestBackup_ThinCheck_SilentWhenNotApplicable).
upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{ upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes, VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune) PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
}) })
if err != nil { if err != nil {
rec.Error = err.Error() rec.Error = err.Error()
r.explainFailure(ctx, vmid, &rec)
rec.DurationSeconds = time.Since(start).Seconds() rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err) return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err)
} }
@@ -204,6 +244,7 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
} }
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil { if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil {
rec.Error = err.Error() rec.Error = err.Error()
r.explainFailure(ctx, vmid, &rec)
rec.DurationSeconds = time.Since(start).Seconds() rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err) return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
} }
@@ -370,6 +411,8 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
var best string var best string
var bestCTime int64 = -1 var bestCTime int64 = -1
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick) known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
var foreignN int // R-366 slice 2: archives skipped as another key's, and their time range
var foreignMin, foreignMax int64
for _, e := range contents { for _, e := range contents {
if e.Content != "backup" { if e.Content != "backup" {
continue continue
@@ -384,6 +427,13 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
} }
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) { if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey))) r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
foreignN++
if foreignMin == 0 || e.CTime < foreignMin {
foreignMin = e.CTime
}
if e.CTime > foreignMax {
foreignMax = e.CTime
}
continue continue
} }
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after // R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
@@ -420,6 +470,9 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
bestCTime, best = e.CTime, e.VolID bestCTime, best = e.CTime, e.VolID
} }
} }
if r.foreign != nil {
r.foreign.set(target, foreignN, foreignMin, foreignMax)
}
if best == "" { if best == "" {
return "", time.Time{}, nil return "", time.Time{}, nil
} }
@@ -556,11 +609,23 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
if err != nil { if err != nil {
return time.Time{}, false, err return time.Time{}, false, err
} }
// R-1232: on an ENCRYPTED storage only archives written with THIS storage's key are this box's (the R-727
// discriminator — the archive carries no host id). A reinstalled box shares the customer's namespace and guest
// number with the old box, and the old box's archive made the tier read „not due" for a whole cadence (168 h).
// A key lookup failure is returned: the due-check reads UNKNOWN, never a foreign archive's age.
ownKey, err := r.storageKeyFingerprint(ctx, r.target)
if err != nil {
return time.Time{}, false, fmt.Errorf("reading the key fingerprint of storage %s: %w", r.target, err)
}
var best int64 = -1 var best int64 = -1
for _, e := range contents { for _, e := range contents {
if e.Content != "backup" || e.VMID != vmid { if e.Content != "backup" || e.VMID != vmid {
continue continue
} }
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this tier's last backup (R-1232)", shortFP(e.Encrypted), shortFP(ownKey)))
continue
}
if ok, why := archivePlausiblyComplete(e); !ok { if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why) r.warnRejectedArchiveOnce(e, why)
continue continue
+3
View File
@@ -30,6 +30,9 @@ import (
// 2026-08-20, two consecutive host-reports with `0 backups` while `pvesm list` showed archives on both tiers. What // 2026-08-20, two consecutive host-reports with `0 backups` while `pvesm list` showed archives on both tiers. What
// is unaffected is the hub's VERDICT: it looks back 7 days over stored reports (felhom.eu hub/internal/monitor/ // is unaffected is the hub's VERDICT: it looks back 7 days over stored reports (felhom.eu hub/internal/monitor/
// deadline.go backupEvidenceLookback) and the storage stays the ground truth (R-84). // deadline.go backupEvidenceLookback) and the storage stays the ground truth (R-84).
// The due-check's fallback for an UNREADABLE storage no longer reads this store alone (R-894): the newest
// success per tier is also on disk (BackupSuccessState), so a restart followed by an unreachable storage
// reads the last known copy, not "never".
type Store struct { type Store struct {
mu sync.Mutex mu sync.Mutex
byTarget map[string]hub.Backup // latest backup per target id byTarget map[string]hub.Backup // latest backup per target id
+9 -1
View File
@@ -91,9 +91,11 @@ var manifest = []Capability{
{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""}, {"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""}, {"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""}, {"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
{"thin-orphan-check", "orphan thin device check after a failed whole-guest backup (read-only, R-1012)", "/usr/local/sbin/felhom-thin-check", []string{"pve", "data"}, false, ""},
// ---- Storage mount units (watchdog re-mount) ---- // ---- Storage mount units (watchdog re-mount) ----
{"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""}, {"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-forget", "decommissioned-drive unit removal (root check, R-1124)", "/usr/local/sbin/felhom-priv-apply", []string{"unit-forget", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""}, {"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "mnt-felhom\\x2dx.mount"}, false, ""}, {"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "mnt-felhom\\x2dx.mount"}, false, ""}, {"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
@@ -115,6 +117,11 @@ var manifest = []Capability{
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""}, {"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""}, {"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""}, {"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
{"guest-shutdown", "household power-off: guest shutdown (R-998)", "/usr/sbin/pct", []string{"shutdown", "9201", "--timeout", "120"}, false, ""},
{"host-poweroff", "household power-off: host power-off (R-998)", "/usr/bin/systemctl", []string{"poweroff"}, false, ""},
{"vzdump-partial-clean", "dead half-written vzdump removal (R-1006)", "/usr/local/sbin/felhom-vzdump-partial-clean", []string{"local", "9201", "vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat"}, false, ""},
{"internal-place-plan", "internal data place: the size plan (16 §7)", "/usr/local/sbin/felhom-internal-place", []string{"plan", "0"}, false, ""},
{"internal-place-create", "internal data place: create on an old box (signed add_internal_place)", "/usr/local/sbin/felhom-internal-place", []string{"create", "0"}, false, ""},
// ---- LAN split-horizon resolver (dnsmasq) ---- // ---- LAN split-horizon resolver (dnsmasq) ----
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""}, {"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
@@ -145,7 +152,8 @@ var manifest = []Capability{
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""}, {"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""}, {"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""}, {"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "tee", "/etc/felhom-controller-image"}, true, ""}, // R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale // ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ---- // reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
+40 -3
View File
@@ -200,7 +200,8 @@ func TestRedProof_DroppedGrantFailsCheck(t *testing.T) {
} }
// TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0 // TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0
// FELHOM_CONTROLLERSWAP grants: with the `tee /etc/felhom-controller-image` line removed, the // FELHOM_CONTROLLERSWAP grants: with the write grant removed (since R-861 (a) A1 the `felhom-priv-apply controller-image`
// line; before it, an agent `tee /etc/felhom-controller-image`), the
// controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new // controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new
// swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update). // swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update).
func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) { func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
@@ -210,7 +211,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
} }
var kept []string var kept []string
for _, ln := range strings.Split(string(data), "\n") { for _, ln := range strings.Split(string(data), "\n") {
if strings.Contains(ln, "tee /etc/felhom-controller-image") { if strings.Contains(ln, "felhom-priv-apply ^controller-image") { // R-861 (a) A1: the write's grant
continue continue
} }
kept = append(kept, ln) kept = append(kept, ln)
@@ -229,7 +230,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
} }
cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ") cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ")
if matchesAny(cmdline, entries) { if matchesAny(cmdline, entries) {
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping the tee grant") t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping its grant")
} }
if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) { if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) {
t.Errorf("controllerswap-write should be covered by the real sudoers") t.Errorf("controllerswap-write should be covered by the real sudoers")
@@ -310,3 +311,39 @@ func TestWGCapabilityCriticality(t *testing.T) {
} }
} }
} }
// TestEveryAliasIsGranted closes a gap found while adding FELHOM_POWEROFF (R-998): the coverage check above reads
// every Cmnd_Alias body whether or not the alias appears on the grant line, so a manifest command whose alias was
// never GRANTED still passed — and failed with „a password is required" on the box. Every defined alias must be
// named on the `felhom-agent ALL=(root) NOPASSWD:` line. Red-proof: drop FELHOM_POWEROFF from that line.
func TestEveryAliasIsGranted(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatalf("read sudoers: %v", err)
}
var defined []string
granted := map[string]bool{}
grantLines := 0
for _, ln := range strings.Split(string(data), "\n") {
tr := strings.TrimSpace(ln)
if strings.HasPrefix(tr, "Cmnd_Alias ") {
if f := strings.Fields(tr); len(f) >= 2 {
defined = append(defined, f[1])
}
}
if strings.HasPrefix(tr, "felhom-agent ") && strings.Contains(tr, "NOPASSWD:") {
grantLines++
for _, a := range strings.Split(tr[strings.Index(tr, "NOPASSWD:")+len("NOPASSWD:"):], ",") {
granted[strings.TrimSpace(a)] = true
}
}
}
if grantLines == 0 || len(defined) < 10 {
t.Fatalf("parsed %d grant line(s), %d aliases — parser broke", grantLines, len(defined))
}
for _, a := range defined {
if !granted[a] {
t.Errorf("Cmnd_Alias %s is defined but never granted to felhom-agent", a)
}
}
}
@@ -49,6 +49,34 @@ var r861Injections = []string{
"/usr/local/sbin/felhom-priv-apply unit ../../etc/x.mount", "/usr/local/sbin/felhom-priv-apply unit ../../etc/x.mount",
"/usr/local/sbin/felhom-priv-apply dnsmasq /etc/shadow felhom-x.conf", "/usr/local/sbin/felhom-priv-apply dnsmasq /etc/shadow felhom-x.conf",
"/usr/local/sbin/felhom-priv-apply wg /etc/shadow", "/usr/local/sbin/felhom-priv-apply wg /etc/shadow",
// R-861 (a) A1 (decision 165): the agent wrote ANY image ref into the guest by `tee` — now only the root verb may
"/usr/sbin/pct exec 9201 -- tee /etc/felhom-controller-image",
"/usr/local/sbin/felhom-priv-apply controller-image 9201 9202",
"/usr/local/sbin/felhom-priv-apply controller-image 9201;id",
// R-998 (host power-off): the two new grants take exactly their fixed shape — no extra option, no other target
"/usr/sbin/pct shutdown 9201 --timeout 120 --forceStop 1",
"/usr/sbin/pct shutdown 9201 --forceStop 1 --timeout 120",
"/usr/sbin/pct shutdown 9201",
"/usr/sbin/pct shutdown 9201 --timeout -1",
"/usr/bin/systemctl poweroff --force",
"/usr/bin/systemctl poweroff -ff",
"/usr/bin/systemctl reboot",
"/usr/bin/systemctl halt",
"/usr/bin/systemctl kexec",
// R-1006: the partial-cleanup helper takes exactly one storage, one vmid, one half-written name.
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.zst",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.log",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 ../../etc/vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat /etc/shadow",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat;id",
"/usr/local/sbin/felhom-vzdump-partial-clean local;id 9201 vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat",
"/usr/local/sbin/felhom-vzdump-partial-clean local 9201",
"/usr/local/sbin/felhom-vzdump-partial-clean --help",
"/usr/local/sbin/felhom-internal-place create 132",
"/usr/local/sbin/felhom-internal-place create 0 extra",
"/usr/local/sbin/felhom-internal-place create 0;id",
"/usr/local/sbin/felhom-internal-place wipe 0",
"/usr/local/sbin/felhom-internal-place create",
} }
func TestSudoersRefusesTheR861Injections(t *testing.T) { func TestSudoersRefusesTheR861Injections(t *testing.T) {
@@ -93,3 +121,42 @@ func TestSudoersFstrimRuleIsExact(t *testing.T) {
} }
} }
} }
// R-861 (a) A1: the managed controller update still has its route — the root verb, one numeric vmid.
func TestSudoersAllowsTheControllerImageVerb(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatal(err)
}
if !matchesAny("/usr/local/sbin/felhom-priv-apply controller-image 9201", parseSudoersEntries(t, string(data))) {
t.Fatal("the sudoers does not allow `felhom-priv-apply controller-image 9201` — a managed controller update cannot write its image")
}
}
// R-861 (b) B2 (decision 165, hygiene): felhom-op's `pct start|stop|unlock` grants are ONE numeric vmid each. The old
// glob `[0-9]*` eats spaces, so `pct stop 9201 --skiplock 1` and two vmids matched.
// RED-PROOF: on the pre-B2 felhom-op.sudoers (`/usr/sbin/pct stop [0-9]*`) the decoys match.
func TestFelhomOpSudoersPctIsExact(t *testing.T) {
data, err := os.ReadFile("../../configs/felhom-op.sudoers")
if err != nil {
t.Fatal(err)
}
entries := parseSudoersEntries(t, string(data))
for _, ok := range []string{"/usr/sbin/pct start 9201", "/usr/sbin/pct stop 9201", "/usr/sbin/pct unlock 9201", "/usr/sbin/pct list"} {
if !matchesAny(ok, entries) {
t.Errorf("felhom-op lost a repair verb: %s", ok)
}
}
for _, bad := range []string{
"/usr/sbin/pct stop 9201 --skiplock 1",
"/usr/sbin/pct start 9201 9202",
"/usr/sbin/pct unlock 9201 --whatever",
"/usr/sbin/pct start 92a1",
"/usr/sbin/pct stop ",
"/usr/sbin/pct destroy 9201",
} {
if matchesAny(bad, entries) {
t.Errorf("felhom-op's sudoers allows %q", bad)
}
}
}
+32
View File
@@ -180,3 +180,35 @@ func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers) t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
} }
} }
// R-1149 (operator decision D17, 2026-10-19): the local whole-guest tier keeps one restore point per day for N days
// (`--prune-backups keep-daily=N`), not the newest N; an additional tier keeps its explicit count.
//
// COMPANION RED-PROOF (observed 2026-10-19): make PruneSpec return keep-last for the primary → this fails with
// `primary tier prune = "keep-last=3", want "keep-daily=3"`. Restored.
func TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
LocalBackupRetention: 3,
ExtraTargets: []BackupTargetConfig{
{TargetID: "felhom-pbs", CadenceSeconds: 604800, KeepLast: 2},
{TargetID: "nas", CadenceSeconds: 86400},
},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 3 {
t.Fatalf("want 3 tiers, got %d", len(tiers))
}
for _, c := range []struct {
i int
want string
}{{0, "keep-daily=3"}, {1, "keep-last=2"}, {2, ""}} {
if got := tiers[c.i].PruneSpec(); got != c.want {
t.Fatalf("%s tier prune = %q, want %q", map[bool]string{true: "primary", false: tiers[c.i].TargetID}[tiers[c.i].Primary], got, c.want)
}
}
// The unset retention defaults to 3 days, never to „prune everything".
if got := (BackupConfig{}).PruneBackupsSpec(); got != "keep-daily=3" {
t.Fatalf("default local prune = %q, want keep-daily=3", got)
}
}
+32 -3
View File
@@ -34,6 +34,7 @@ type Config struct {
WGTunnel WGTunnelConfig `json:"wg_tunnel"` WGTunnel WGTunnelConfig `json:"wg_tunnel"`
GuestNet GuestNetConfig `json:"guest_net"` GuestNet GuestNetConfig `json:"guest_net"`
DiskTrim DiskTrimConfig `json:"disk_trim"` DiskTrim DiskTrimConfig `json:"disk_trim"`
LineTest LineTestConfig `json:"line_test"`
OOB OOBConfig `json:"oob"` OOB OOBConfig `json:"oob"`
SelfUpdate SelfUpdateConfig `json:"selfupdate"` SelfUpdate SelfUpdateConfig `json:"selfupdate"`
LogLevel string `json:"log_level"` // debug|info|warn|error (default info) LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
@@ -150,6 +151,16 @@ type DiskTrimConfig struct {
// Enabled reports whether the weekly trim should run. // Enabled reports whether the weekly trim should run.
func (d DiskTrimConfig) Enabled() bool { return !d.Disable } func (d DiskTrimConfig) Enabled() bool { return !d.Disable }
// LineTestConfig configures the R-1247 weekly upload line test (internal/linetest). DEFAULT-ON: it reaches only ep0's
// tunnel-only iperf3 server, so it runs only on a box that already has an operational offsite tunnel (operator
// ruling 2026-10-24 07:26: every box with a tunnel measures its line weekly).
type LineTestConfig struct {
Disable bool `json:"disable"`
}
// Enabled reports whether the weekly line test should run.
func (l LineTestConfig) Enabled() bool { return !l.Disable }
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet). // GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
// //
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other // **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
@@ -391,7 +402,8 @@ type BackupConfig struct {
// hub-served per-guest policy is slice 10; this is the agent-local cadence. // hub-served per-guest policy is slice 10; this is the agent-local cadence.
BackupCadenceSeconds int `json:"backup_cadence_seconds"` BackupCadenceSeconds int `json:"backup_cadence_seconds"`
// LocalBackupRetention is keep-last=N for the per-run `--prune-backups` on a LOCAL vzdump target — // LocalBackupRetention is N for the per-run `--prune-backups keep-daily=N` on a LOCAL vzdump target (keep-last=N
// before R-1149) —
// so the agent's own local whole-guest backups can't pile up and refill root (the felhom-pve incident; // so the agent's own local whole-guest backups can't pile up and refill root (the felhom-pve incident;
// the host_disk + storage_fill checkers are the detectors, this is the preventive default). 0/unset → // the host_disk + storage_fill checkers are the detectors, this is the preventive default). 0/unset →
// default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup. // default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup.
@@ -538,9 +550,26 @@ func (b BackupConfig) KeepLast() int {
return b.LocalBackupRetention return b.LocalBackupRetention
} }
// PruneBackupsSpec returns the PVE `--prune-backups` value for the local vzdump (e.g. "keep-last=3"). // PruneBackupsSpec returns the PVE `--prune-backups` value for the local vzdump: "keep-daily=N" (R-1149, operator
// decision D17 2026-10-19 — one restore point per day for N days; `07` §6.1). keep-last=N had the same disk cost but
// let N extra runs on one day push out every older point; keep-daily keeps the newest copy of each of the N newest
// days that have one, so a flood inside one day replaces only that day's point. Pinned by
// TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay and, as the consequence, TestBackupCap_R1149_LocalFloodKeepsOlderDays.
func (b BackupConfig) PruneBackupsSpec() string { func (b BackupConfig) PruneBackupsSpec() string {
return fmt.Sprintf("keep-last=%d", b.KeepLast()) return fmt.Sprintf("keep-daily=%d", b.KeepLast())
}
// PruneSpec is this tier's per-run `--prune-backups` value, or "" for never prune. The PRIMARY (local) tier keeps one
// point per day (keep-daily, R-1149, from KeepLast() — always ≥1); an additional tier keeps its explicit keep_last
// count (a weekly off-site tier's „last 2" is two weeks, and PBS prunes stay opt-in per tier, R-82).
func (t BackupTier) PruneSpec() string {
if t.KeepLast <= 0 {
return ""
}
if t.Primary {
return fmt.Sprintf("keep-daily=%d", t.KeepLast)
}
return fmt.Sprintf("keep-last=%d", t.KeepLast)
} }
// BackupCadence returns the per-guest /backup/due window: positive as-is, else 24h default. // BackupCadence returns the per-guest /backup/due window: positive as-is, else 24h default.
+12 -2
View File
@@ -1,5 +1,7 @@
package escrow package escrow
import "encoding/json"
// Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by // Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by
// felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API // felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API
// re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers // re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers
@@ -43,7 +45,15 @@ type CeremonyOutput struct {
IdentityBlobBytes int `json:"identity_blob_bytes"` IdentityBlobBytes int `json:"identity_blob_bytes"`
ResticPwSealed bool `json:"restic_pw_sealed"` ResticPwSealed bool `json:"restic_pw_sealed"`
Uploaded bool `json:"uploaded"` Uploaded bool `json:"uploaded"`
// PendingUpload (version 2, R-978) is the hub upload body — the R-wrapped blobs (ciphertext), the key
// fingerprint and the sealed password's hash — that json mode hands back INSTEAD of uploading. The agent
// daemon uploads it only when the customer claims the new recovery code, so a ceremony that is started
// and abandoned never replaces the package the customer's previous code opens. Not a secret without R,
// but never logged. Uploaded is false whenever this is set.
PendingUpload json.RawMessage `json:"pending_upload,omitempty"`
} }
// CeremonyOutputVersion is the current CeremonyOutput.Version value. // CeremonyOutputVersion is the current CeremonyOutput.Version value. 2 (R-978): json mode no longer
const CeremonyOutputVersion = 1 // uploads; it returns PendingUpload. The daemon refuses any other version (a binary/daemon mismatch
// fails the ceremony instead of losing a package).
const CeremonyOutputVersion = 2
+50 -10
View File
@@ -59,8 +59,26 @@ var (
// It carries no material and no code: only WHICH earlier package opened, by its supersession date, // It carries no material and no code: only WHICH earlier package opened, by its supersession date,
// which is the one fact the customer needs to recognise it. // which is the one fact the customer needs to recognise it.
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one") ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
// ErrRetainedUnchecked — the code did NOT open the current package, and NOT every earlier package the hub
// holds for this host was tried (R-304, 2026-10-08): the retained list could not be fetched, the hub
// withheld rows (over its serve cap, or rows with no key material), a package was malformed, or the
// attempt cap stopped the loop. So „the code is wrong" is NOT known — it may be right for a package
// nobody tried. Distinct from a mistype for exactly the R-224 reason: never accuse the customer of
// something we did not check.
ErrRetainedUnchecked = errors.New("escrow: the recovery code did not open the current sealed package, and some earlier packages were NOT checked")
) )
// RetainedUncheckedError wraps ErrRetainedUnchecked with how many earlier packages went unchecked
// (-1 = unknown: the retained list itself could not be read). No secret.
type RetainedUncheckedError struct {
Unchecked int
}
func (e *RetainedUncheckedError) Error() string {
return fmt.Sprintf("%s (unchecked=%d)", ErrRetainedUnchecked.Error(), e.Unchecked)
}
func (e *RetainedUncheckedError) Unwrap() error { return ErrRetainedUnchecked }
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it // RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
// carries no secret — not the code, not the bundle, not the repository password. // carries no secret — not the code, not the bundle, not the repository password.
type RetainedMatch struct { type RetainedMatch struct {
@@ -102,8 +120,10 @@ type RetainedBlob struct {
} }
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty // RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
// slice is a clean "none". R-311. // slice is a clean "none". R-311. `withheld` counts the earlier packages the hub holds that are NOT in
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error) // blobs — rows with no key material, rows over the hub's serve cap, and packages dropped as malformed
// (R-304): each is a package the code was never tried against.
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, withheld int, err error)
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R. // OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
type OffsiteKeyRecoverer struct { type OffsiteKeyRecoverer struct {
@@ -154,9 +174,14 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and // from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence // the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
// about our own incuriosity, read by the customer as a statement about their code. // about our own incuriosity, read by the customer as a statement about their code.
if m, ok := r.tryRetained(ctx, recoveryCode); ok { m, ok, unchecked := r.tryRetained(ctx, recoveryCode)
if ok {
return "", &RetainedOpenedError{Match: m} return "", &RetainedOpenedError{Match: m}
} }
// R-304: only when EVERY earlier package the hub holds was tried may this stay a wrong code.
if unchecked != 0 {
return "", &RetainedUncheckedError{Unchecked: unchecked}
}
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
} }
if bundle.ResticRepoPassword == "" { if bundle.ResticRepoPassword == "" {
@@ -173,23 +198,38 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
// function breaking is the behaviour we had before it existed. // function breaking is the behaviour we had before it existed.
// //
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password. // NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) { //
// R-304 (2026-10-08): it also returns how many earlier packages were NOT tried — `withheld` from the hub, plus
// the ones past the attempt cap — or -1 when the retained list could not be read at all. A nil FetchRetained
// (an agent wired without the lookup) reports 0: the pre-R-311 refusal, unchanged.
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool, int) {
if r.FetchRetained == nil { if r.FetchRetained == nil {
return RetainedMatch{}, false return RetainedMatch{}, false, 0
} }
blobs, _, err := r.FetchRetained(ctx) blobs, withheld, err := r.FetchRetained(ctx)
if err != nil || len(blobs) == 0 { if err != nil {
return RetainedMatch{}, false return RetainedMatch{}, false, -1
}
if withheld < 0 {
withheld = 0
}
if len(blobs) == 0 {
return RetainedMatch{}, false, withheld
} }
limit := r.MaxRetainedTried limit := r.MaxRetainedTried
if limit <= 0 { if limit <= 0 {
limit = defaultMaxRetainedTried limit = defaultMaxRetainedTried
} }
unchecked := withheld
if len(blobs) > limit {
unchecked += len(blobs) - limit
}
for i, rb := range blobs { for i, rb := range blobs {
if i >= limit { if i >= limit {
break break
} }
if len(rb.Blob) == 0 { if len(rb.Blob) == 0 {
unchecked++
continue continue
} }
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode) bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
@@ -204,7 +244,7 @@ func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode strin
// correct and must be told so — but the history behind it still cannot be reopened, and // correct and must be told so — but the history behind it still cannot be reopened, and
// saying otherwise would be a promise this path cannot keep. // saying otherwise would be a promise this path cannot keep.
HasResticPassword: bundle.ResticRepoPassword != "", HasResticPassword: bundle.ResticRepoPassword != "",
}, true }, true, 0
} }
return RetainedMatch{}, false return RetainedMatch{}, false, unchecked
} }
+76 -2
View File
@@ -115,8 +115,9 @@ func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
} }
} }
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The // FAIL-SAFE — if the retained lookup itself fails, the lookup's error never reaches the customer and no
// worst outcome of this feature breaking is the behaviour we had before it. // retained package is claimed. R-304 (2026-10-08) changed what stands instead: not the wrong-code refusal (the
// earlier packages were never tried, so „wrong" is not known) but ErrRetainedUnchecked with Unchecked = -1.
// //
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer // RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
// gets a new, unexplained failure mode → this FAILS. // gets a new, unexplained failure mode → this FAILS.
@@ -140,6 +141,10 @@ func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
if containsStr(err.Error(), "hub exploded") { if containsStr(err.Error(), "hub exploded") {
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent") t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
} }
var ue *RetainedUncheckedError
if !errors.As(err, &ue) || ue.Unchecked != -1 {
t.Errorf("err = %v, want RetainedUncheckedError{-1} — the earlier packages were never tried (R-304)", err)
}
} }
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be // A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
@@ -228,3 +233,72 @@ func containsStr(hay, needle string) bool {
return false return false
})() })()
} }
// ── R-304 (2026-10-08) — „wrong code" only when every earlier package was tried ──────────────────────
//
// The consequence asserted: is the refusal a WRONG CODE (the only error the screen may answer with „check your
// typing")? It may be only when the hub withheld nothing and every served package was tried.
//
// RED-PROOF: make tryRetained return 0 for `unchecked` → the three „unchecked" cases return the plain refusal
// → they FAIL; the all-tried case keeps passing.
func TestR304_WrongCodeOnlyWhenEveryEarlierPackageWasTried(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "4444567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
other := sealBundle(t, IdentityBundle{ResticRepoPassword: "5555567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
const code = "a code that opens nothing whatsoever in this test"
cases := []struct {
name string
blobs int
withheld int
limit int
unchecked int // 0 = must be a plain wrong code
}{
{"all tried, nothing withheld", 2, 0, 6, 0},
{"the hub withheld rows (no key material / over its cap)", 1, 3, 6, 3},
{"only withheld rows, nothing served", 0, 2, 6, 2},
{"the attempt cap stopped the loop", 5, 0, 2, 3},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
blobs := make([]RetainedBlob, 0, c.blobs)
for i := 0; i < c.blobs; i++ {
blobs = append(blobs, RetainedBlob{Blob: other, SupersededAt: "2026-08-01 00:00:00", Index: i})
}
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
return blobs, c.withheld, nil
},
MaxRetainedTried: c.limit,
}.RecoverOffsiteRepoPassword(context.Background(), code)
if err == nil {
t.Fatal("a code that opens nothing succeeded")
}
var ue *RetainedUncheckedError
isUnchecked := errors.As(err, &ue)
if c.unchecked == 0 {
if isUnchecked {
t.Fatalf("every earlier package was tried — this IS a wrong code, got %v", err)
}
return
}
if !isUnchecked || ue.Unchecked != c.unchecked || !errors.Is(err, ErrRetainedUnchecked) {
t.Fatalf("err = %v, want RetainedUncheckedError{%d} — never call the code wrong when packages were not tried", err, c.unchecked)
}
})
}
}
// A malformed (empty) served package counts as not tried.
func TestR304_EmptyServedPackageCountsAsUnchecked(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "6666567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: nil, SupersededAt: "2026-08-01 00:00:00"}),
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
var ue *RetainedUncheckedError
if !errors.As(err, &ue) || ue.Unchecked != 1 {
t.Fatalf("err = %v, want RetainedUncheckedError{1}", err)
}
}
+22 -1
View File
@@ -22,6 +22,11 @@ type Loop struct {
mu sync.Mutex mu sync.Mutex
fetched bool // a desired-state document has been delivered (false = never a teardown signal) fetched bool // a desired-state document has been delivered (false = never a teardown signal)
block *hub.WireWireguard block *hub.WireWireguard
// passed (R-1194) is set once the first reconcile pass has run. Until then the instance has never been rendered,
// so its health says nothing — and the first host report, sent before that pass, used to read „config invalid,
// port 0" and raise a false oob_degraded on EVERY fresh box (rehearsal 2026-10-20; reproduced 2026-10-21: the
// report at 20:34:00 read an absent config, the apply landed 0.4 s later).
passed bool
nudge chan struct{} nudge chan struct{}
} }
@@ -85,6 +90,7 @@ func (l *Loop) reconcile(ctx context.Context) {
if !fetched { if !fetched {
applyBlock = nil applyBlock = nil
} }
defer l.markPassed()
port, err := l.mgr.Apply(ctx, applyBlock) port, err := l.mgr.Apply(ctx, applyBlock)
if err != nil { if err != nil {
return // Apply logged; a claim/exhaustion or install error — retry next tick return // Apply logged; a claim/exhaustion or install error — retry next tick
@@ -106,8 +112,23 @@ func oobPeerIP(block *hub.WireWireguard) string {
return block.OOBPeerIP return block.OOBPeerIP
} }
// OOBStatus implements the hub collector's reporter seam (Part 4). // markPassed records that a reconcile pass has run (R-1194) — after a failed Apply too, so a real, lasting failure
// is still reported from the next report on.
func (l *Loop) markPassed() {
l.mu.Lock()
l.passed = true
l.mu.Unlock()
}
// OOBStatus implements the hub collector's reporter seam (Part 4). nil — no stanza, which the hub does not evaluate —
// until the first reconcile pass has run (R-1194).
func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus { func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus {
l.mu.Lock()
passed := l.passed
l.mu.Unlock()
if !passed {
return nil
}
_, block := l.snapshot() _, block := l.snapshot()
return l.mgr.Status(ctx, block) return l.mgr.Status(ctx, block)
} }
@@ -0,0 +1,29 @@
package felhomsshd
import (
"context"
"testing"
"time"
)
// R-1194: on every fresh box the first host report — sent before the OOB loop's first pass — read „config invalid,
// port 0" (the config was not rendered yet) and the hub mailed oob_degraded. Reproduced 2026-10-21 on a fresh box
// (felhom.eu audits/night-2026-10-21/P5-checks/): the report at 20:34:00, the apply 0.4 s later, the daemon healthy.
// The loop now reports NO stanza (the hub does not evaluate a missing one) until a pass has run — a failed pass
// counts, so a real lasting failure is still reported from the next report on.
// RED-PROOF: with OOBStatus not gated on `passed`, the first assertion fails (a stanza with config_invalid).
func TestR1194_NoOOBStanzaBeforeTheFirstPass(t *testing.T) {
r := newScriptRunner()
m := NewManager(r, t.TempDir(), nil)
m.isActive = func(context.Context) bool { return false }
m.isFailed = func(context.Context) bool { return false }
m.now = func() time.Time { return time.Unix(1783270000, 0) }
l := NewLoop(m, nil, time.Minute, nil)
if st := l.OOBStatus(context.Background()); st != nil {
t.Fatalf("a stanza before the first pass reads a config that was never rendered: %+v", st)
}
l.reconcile(context.Background()) // fails or succeeds — either way a pass has run
if st := l.OOBStatus(context.Background()); st == nil {
t.Fatal("after the first pass the health must be reported (a real failure must reach the hub)")
}
}
+10 -3
View File
@@ -14,6 +14,13 @@ const (
TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED
TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which) TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which)
TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting) TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting)
// TunnelAbsent (R-1070, agent Unreleased 2026-10-19): the guest answered and has NO cloudflared container — the
// controller makes none when no tunnel token is configured (a LAN-only or test box). Not a failure by itself: the
// hub never alarms on it, and the OS-update host rule reads it as „not applicable" only when the container was
// also absent before the update (HostHealthVerdict). Docker 29 prints „error: no such object" in lower case, so
// before this state existed the case-sensitive match fell through to `unknown` — which failed the host rule on
// every box without a tunnel (Tester4, 2026-10-15).
TunnelAbsent = "absent"
) )
// CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec. // CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec.
@@ -53,7 +60,7 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
return TunnelUnknown, "no running customer guest" return TunnelUnknown, "no running customer guest"
} }
worst, wdetail := "", "" worst, wdetail := "", ""
rank := map[string]int{TunnelRunning: 0, TunnelUnknown: 1, TunnelNotRunning: 2} rank := map[string]int{TunnelRunning: 0, TunnelAbsent: 1, TunnelUnknown: 2, TunnelNotRunning: 3}
for _, v := range vmids { for _, v := range vmids {
out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared") out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared")
st, d := ClassifyTunnel(string(out), string(errOut), err) st, d := ClassifyTunnel(string(out), string(errOut), err)
@@ -71,8 +78,8 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
func ClassifyTunnel(stdout, stderr string, err error) (string, string) { func ClassifyTunnel(stdout, stderr string, err error) (string, string) {
out := strings.TrimSpace(stdout) out := strings.TrimSpace(stdout)
if err != nil || out == "" { if err != nil || out == "" {
if strings.Contains(stderr, "No such object") || strings.Contains(stderr, "No such container") { if e := strings.ToLower(stderr); strings.Contains(e, "no such object") || strings.Contains(e, "no such container") {
return TunnelNotRunning, "no cloudflared container in the guest" return TunnelAbsent, "no cloudflared container in the guest (no tunnel set up on this box)"
} }
return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err) return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err)
} }
+4 -1
View File
@@ -20,7 +20,10 @@ func TestClassifyTunnel(t *testing.T) {
{"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"}, {"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"},
{"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"}, {"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"},
{"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"}, {"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelNotRunning, "no cloudflared container"}, {"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
// R-1070: Docker 29's wording, measured on demo-hp 2026-10-19 (`docker inspect` of a missing name).
{"absent, docker 29", "", "error: no such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no tunnel set up"},
{"absent, no such container", "", "Error response from daemon: No such container: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
{"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"}, {"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"},
{"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"}, {"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"},
{"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"}, {"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"},
+101 -6
View File
@@ -11,6 +11,7 @@ import (
"os" "os"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability" "gitea.dooplex.hu/admin/felhom-agent/internal/capability"
@@ -66,6 +67,13 @@ type ProvenRestoreTestReporter interface {
ProvenRestoreTests(ctx context.Context) []RestoreTest ProvenRestoreTests(ctx context.Context) []RestoreTest
} }
// KnownBackupReporter is the newest SUCCESSFUL backup per tier and guest kept on disk (R-894's
// backup-success-state.json). collectBackups folds it into the report so a restart does not erase the
// hub's evidence of a backup that ran minutes before it (the kernel-night shape, 2026-10-09).
type KnownBackupReporter interface {
KnownBackupSuccesses() []Backup
}
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern). // PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty. // Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
type PBSReporter interface { type PBSReporter interface {
@@ -96,6 +104,12 @@ type GuestDiskTrimReporter interface {
GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus
} }
// LineTestReporter is the R-1247 seam the weekly line test plugs into (hub does not import linetest). nil → no
// line_test stanza.
type LineTestReporter interface {
LineTestStatus(ctx context.Context) *LineTestStatus
}
// Collector builds a HostReport from read-only sources. All deps are behind narrow // Collector builds a HostReport from read-only sources. All deps are behind narrow
// interfaces for unit testing. // interfaces for unit testing.
type Collector struct { type Collector struct {
@@ -105,6 +119,7 @@ type Collector struct {
backups BackupReporter backups BackupReporter
restoreTests RestoreTestReporter restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter provenTests ProvenRestoreTestReporter
knownBackups KnownBackupReporter // R-894 file → host report after a restart (nil → in-memory only)
pbs PBSReporter pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp) temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty) capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
@@ -115,6 +130,9 @@ type Collector struct {
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted) ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted) guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted) diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false) selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted) mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted) oob OOBReporter // H1: operator-access health (nil → stanza omitted)
@@ -126,6 +144,8 @@ type Collector struct {
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256 selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
// swapShortWarned (R-1192): the host-swap-short WARN is logged once per state change, not every report.
swapShortWarned atomic.Bool
} }
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is // NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
@@ -228,12 +248,36 @@ func (c *Collector) SetGuestNetReporter(g GuestNetReporter) *Collector {
return c return c
} }
// ForeignKeyArchiveReporter is the R-366 slice-2 seam: the restore-test's ledger of archives written with another
// key. nil = not evaluated yet (the stanza is omitted and the hub keeps its state).
type ForeignKeyArchiveReporter interface {
ForeignKeyArchives(ctx context.Context) *ForeignKeyArchivesStanza
}
// SetForeignKeyArchiveReporter wires the restore-test's foreign-key ledger (R-366 slice 2; nil-safe → omitted).
func (c *Collector) SetForeignKeyArchiveReporter(r ForeignKeyArchiveReporter) *Collector {
c.foreignKey = r
return c
}
// SetGuestDiskTrimReporter wires the R-444 weekly trim job as a report source (nil-safe → stanza omitted). // SetGuestDiskTrimReporter wires the R-444 weekly trim job as a report source (nil-safe → stanza omitted).
func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector { func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector {
c.diskTrim = r c.diskTrim = r
return c return c
} }
// SetLineTestReporter wires the R-1247 weekly line test as a report source (nil-safe → stanza omitted).
func (c *Collector) SetLineTestReporter(r LineTestReporter) *Collector {
c.lineTest = r
return c
}
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
c.deliberateOff = f
return c
}
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side // SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report. // pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
type SelfUpdateReporter interface { type SelfUpdateReporter interface {
@@ -347,6 +391,8 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
} }
host := hostMetrics(c.px.Node(), ns) host := hostMetrics(c.px.Node(), ns)
guests := c.collectGuests(ctx)
c.applySwap(&host, ns, guests) // R-1192: host swap vs the guests' swap allowance
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable
report := &HostReport{ report := &HostReport{
@@ -355,7 +401,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
AgentVersion: c.agentVersion, AgentVersion: c.agentVersion,
AgentSHA256: c.agentSHA256(), AgentSHA256: c.agentSHA256(),
Host: host, Host: host,
Guests: c.collectGuests(ctx), Guests: guests,
// storage_targets populated this slice (slice 5) via the observer; the rest stay // storage_targets populated this slice (slice 5) via the observer; the rest stay
// defined-but-empty (slice 6). Non-nil so they marshal as []. // defined-but-empty (slice 6). Non-nil so they marshal as [].
StorageTargets: c.collectStorage(ctx), StorageTargets: c.collectStorage(ctx),
@@ -394,6 +440,14 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.diskTrim != nil { if c.diskTrim != nil {
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx) report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
} }
// R-1247: the last weekly line test (nil reporter = not wired → stanza omitted).
if c.lineTest != nil {
report.LineTest = c.lineTest.LineTestStatus(ctx)
}
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
if c.foreignKey != nil {
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
}
// D1: agent self-update pending status (nil reporter → pending=false, the steady state). // D1: agent self-update pending status (nil reporter → pending=false, the steady state).
if c.selfUpdate != nil { if c.selfUpdate != nil {
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending() report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()
@@ -406,6 +460,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.oob != nil { if c.oob != nil {
report.OOB = c.oob.OOBStatus(ctx) report.OOB = c.oob.OOBStatus(ctx)
} }
// R-998: a household power-off accepted in this boot rides every report until the box is down.
if c.deliberateOff != nil {
report.DeliberateOff = c.deliberateOff()
}
return report, nil return report, nil
} }
@@ -531,6 +589,7 @@ func (c *Collector) collectGuests(ctx context.Context) []Guest {
Cores: cfg.Cores, Cores: cfg.Cores,
MemoryBytes: g.MaxMem, MemoryBytes: g.MaxMem,
DiskBytes: g.MaxDisk, DiskBytes: g.MaxDisk,
SwapBytes: cfg.Swap * (1 << 20),
} }
} }
guests = append(guests, entry) guests = append(guests, entry)
@@ -559,14 +618,50 @@ func (c *Collector) collectStorage(ctx context.Context) []StorageTarget {
// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state // collectBackups / collectRestoreTests read the agent's latest backup + restore-test state
// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil) // via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil)
// list so the collection always marshals as []. // list so the collection always marshals as [].
//
// The saved successes (SetKnownBackupReporter) are added for each tier and guest the in-memory list has
// no success for at or after the saved time. Why: the in-memory list is empty after an agent restart,
// and the hub's backup-freshness check reads only what the reports carried. Measured 2026-10-09 on
// demo-felhom: the night backup landed 02:40 UTC, the kernel step restarted the host at 02:44, no report
// fell in between, so two kernel nights in a row left no trace and the hub alarmed „newest backup is 48h
// old" at 03:00. Only SUCCESSES are saved, so a failure is never hidden and never invented.
// Pinned by TestCollectBackups_SavedSuccessSurvivesARestart.
func (c *Collector) collectBackups(ctx context.Context) []Backup { func (c *Collector) collectBackups(ctx context.Context) []Backup {
if c.backups == nil { out := []Backup{}
return []Backup{} if c.backups != nil {
}
if b := c.backups.Backups(ctx); b != nil { if b := c.backups.Backups(ctx); b != nil {
return b out = append(out, b...)
} }
return []Backup{} }
if c.knownBackups == nil {
return out
}
for _, k := range c.knownBackups.KnownBackupSuccesses() {
kt, err := time.Parse(time.RFC3339, k.StartedAt)
if err != nil {
continue
}
covered := false
for _, b := range out {
if !b.Success || b.TargetID != k.TargetID || b.VMID != k.VMID {
continue
}
if bt, err := time.Parse(time.RFC3339, b.StartedAt); err == nil && !bt.Before(kt) {
covered = true
break
}
}
if !covered {
out = append(out, k)
}
}
return out
}
// SetKnownBackupReporter wires the R-894 saved successes into the host report (nil-safe → in-memory only).
func (c *Collector) SetKnownBackupReporter(r KnownBackupReporter) *Collector {
c.knownBackups = r
return c
} }
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189). // collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
+62
View File
@@ -0,0 +1,62 @@
package hub
import (
"context"
"testing"
)
type fakeMemBackups []Backup
func (f fakeMemBackups) Backups(context.Context) []Backup { return f }
type fakeKnownBackups []Backup
func (f fakeKnownBackups) KnownBackupSuccesses() []Backup { return f }
// The consequence, not the mechanism: after a restart (empty in-memory list) the host report still carries
// the night's backup, so the hub's freshness check sees it. Measured 2026-10-09 on demo-felhom: without this
// the report carried backups: [] and the hub alarmed „newest backup is 48h old" an hour after a good backup.
// RED-PROOF: return before the saved-success loop in collectBackups → the first case reports nothing → fails.
func TestCollectBackups_SavedSuccessSurvivesARestart(t *testing.T) {
saved := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}
t.Run("restart: memory empty, the saved success is reported", func(t *testing.T) {
c := &Collector{backups: fakeMemBackups(nil), knownBackups: fakeKnownBackups{saved}}
got := c.collectBackups(context.Background())
if len(got) != 1 || got[0].StartedAt != saved.StartedAt || !got[0].Success || got[0].TargetID != "felhom-backup" {
t.Fatalf("want the saved success, got %+v", got)
}
})
t.Run("memory holds the same or a newer success: no second entry", func(t *testing.T) {
mem := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z", Archive: "a"}
c := &Collector{backups: fakeMemBackups{mem}, knownBackups: fakeKnownBackups{saved}}
if got := c.collectBackups(context.Background()); len(got) != 1 || got[0].Archive != "a" {
t.Fatalf("want only the in-memory record, got %+v", got)
}
})
t.Run("a newer FAILURE in memory never hides the saved success, and is kept", func(t *testing.T) {
fail := Backup{TargetID: "felhom-backup", VMID: 9201, Success: false, StartedAt: "2026-10-10T02:40:00Z", Error: "x"}
c := &Collector{backups: fakeMemBackups{fail}, knownBackups: fakeKnownBackups{saved}}
got := c.collectBackups(context.Background())
if len(got) != 2 || got[0].Success || !got[1].Success {
t.Fatalf("want the failure and the saved success, got %+v", got)
}
})
t.Run("another tier's success does not cover this tier", func(t *testing.T) {
other := Backup{TargetID: "felhom-pbs", VMID: 9201, Success: true, StartedAt: "2026-10-10T00:00:00Z"}
c := &Collector{backups: fakeMemBackups{other}, knownBackups: fakeKnownBackups{saved}}
if got := c.collectBackups(context.Background()); len(got) != 2 {
t.Fatalf("want both tiers, got %+v", got)
}
})
t.Run("no saved state wired: the in-memory list as before, never nil", func(t *testing.T) {
c := &Collector{}
if got := c.collectBackups(context.Background()); got == nil || len(got) != 0 {
t.Fatalf("want an empty non-nil list, got %#v", got)
}
})
}
+3
View File
@@ -39,11 +39,14 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
State: StorageStateAttached, Reachable: true, ClassHint: "fast", State: StorageStateAttached, Reachable: true, ClassHint: "fast",
ThinPool: &ThinPoolFill{DataUsedFraction: 0.42}, ThinPool: &ThinPoolFill{DataUsedFraction: 0.42},
Smart: SmartSummary{Health: SmartUnknown}, Smart: SmartSummary{Health: SmartUnknown},
// v0.157.0 (`16` §3): the identity keys ride on both rows so the key-set guard sees them.
Kind: "system", PhysicalDisks: []string{"serial:ata:SanDisk_X600:182195804614"}, Identity: "proven",
}, },
{ {
Name: "usb-backup", Type: StorageTypeUSB, DurableID: "uuid:x", Name: "usb-backup", Type: StorageTypeUSB, DurableID: "uuid:x",
State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000, State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000,
Smart: SmartSummary{Health: SmartUnknown}, Smart: SmartSummary{Health: SmartUnknown},
Kind: "disk", PhysicalDisks: []string{"serial:usb:External:000000000000"}, Identity: "unknown", IdentityReason: "usb_bridge",
}, },
// A pbs target so the recipe's pbs coord has a storage.cfg row to resolve its namespace // A pbs target so the recipe's pbs coord has a storage.cfg row to resolve its namespace
// from (R-106). Without one the fixture produces namespace_state=unknown while the golden // from (R-106). Without one the fixture produces namespace_state=unknown while the golden
+29
View File
@@ -0,0 +1,29 @@
package hub
import (
"context"
"errors"
"testing"
"time"
)
// R-998: the power-off path waits on SetReportedHook, so the hook must fire only for a report the hub
// ACCEPTED — a failed push must not let the box power off believing the hub knows.
func TestLoop_ReportedHookOnlyAfterAcceptedReport(t *testing.T) {
var cn, rn int32
rep := &HostReport{DeliberateOff: &DeliberateOff{RequestedAt: "2026-10-13T18:00:00Z"}}
loop := NewLoop(&fakeCollector{report: rep, n: &cn},
&fakeReporter{env: &ControlEnvelope{}, errSeq: []error{errors.New("hub 5xx")}, n: &rn},
900*time.Second, quietLogger())
var seen []*HostReport
loop.SetReportedHook(func(r *HostReport) { seen = append(seen, r) })
loop.cycle(context.Background(), 900*time.Second) // the push fails
if len(seen) != 0 {
t.Fatalf("hook fired for a refused report: %d", len(seen))
}
loop.cycle(context.Background(), 900*time.Second) // the push is accepted
if len(seen) != 1 || seen[0].DeliberateOff == nil || seen[0].DeliberateOff.RequestedAt != "2026-10-13T18:00:00Z" {
t.Fatalf("hook after an accepted report = %+v", seen)
}
}
+39
View File
@@ -0,0 +1,39 @@
package hub
import (
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the guest swap fix (every guest may swap as much as its RAM,
// localapi/guestswap.go) moves a memory burst onto the HOST's swap. Measured on demo-hp the same day: our installer
// gives an 8 GB / 256 GB machine 7.75 GB of host swap, so a 4096 MB guest's 4096 MB swap fits. A host with no swap
// (a BYO install, or swap turned off) gets nothing from the fix. So the report carries both numbers, the agent says it
// once in its log, and the hub raises host_swap_short. The agent never creates host swap itself: changing a host its
// owner set up is a decision (felhom.eu register).
// HostSwapShort reports whether the host's swap is below the guests' swap allowance. A host whose guests have no
// swap allowance (0) is never short.
func HostSwapShort(hostMB, guestMB int64) bool { return guestMB > 0 && hostMB < guestMB }
// applySwap fills the host block's swap numbers from the node status and the guests' specs, and logs the short
// state once per change.
func (c *Collector) applySwap(h *HostMetrics, ns proxmox.NodeStatus, guests []Guest) {
hostMB := ns.Swap.Total / (1 << 20)
var guestMB int64
for _, g := range guests {
if g.Spec != nil {
guestMB += g.Spec.SwapBytes / (1 << 20)
}
}
h.HostSwapMB, h.GuestSwapMB = &hostMB, &guestMB
if HostSwapShort(hostMB, guestMB) {
if c.swapShortWarned.CompareAndSwap(false, true) {
c.logger.Warn("host-swap: the host's swap is below its guests' swap allowance — a memory burst can freeze the box (R-1192); the hub is told (host_swap_short)",
"host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
return
}
if c.swapShortWarned.CompareAndSwap(true, false) {
c.logger.Info("host-swap: the host's swap covers its guests' swap allowance again", "host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
}
+10
View File
@@ -66,8 +66,15 @@ type Loop struct {
// retry, no duplicate shipping). Loop state is single-goroutine (cycle only). // retry, no duplicate shipping). Loop state is single-goroutine (cycle only).
logTailSource func(maxBytes int) []string logTailSource func(maxBytes int) []string
logTailPending bool logTailPending bool
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
// path waits on it so the hub has the deliberate-off record before the box goes down.
reported func(r *HostReport)
} }
// SetReportedHook wires the R-998 after-report observer. Optional.
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
// logTailMaxBytes caps the heartbeat log tail (newest lines kept). // logTailMaxBytes caps the heartbeat log tail (newest lines kept).
const logTailMaxBytes = 128 * 1024 const logTailMaxBytes = 128 * 1024
@@ -156,6 +163,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
l.logger.Warn("hub: report failed; keeping current interval", "err", err) l.logger.Warn("hub: report failed; keeping current interval", "err", err)
return current return current
} }
if l.reported != nil {
l.reported(report)
}
if report.LogTail != nil { if report.LogTail != nil {
// Transparency: the pull is visible in the box's own log (and thus in the ring). // Transparency: the pull is visible in the box's own log (and thus in the ring).
l.logger.Info("operator log pull served", "component", "agent", "lines", len(report.LogTail.Lines)) l.logger.Info("operator log pull served", "component", "agent", "lines", len(report.LogTail.Lines))
+42
View File
@@ -0,0 +1,42 @@
package hub
import (
"bytes"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the host report carries the host's swap and the guests' swap allowance, and the
// agent says ONCE in its log when the host's swap is below it (and once when it is covered again).
// RED-PROOF: with applySwap not called the numbers are nil; with the CompareAndSwap removed the WARN repeats.
func TestR1192_HostReportCarriesSwapAndWarnsOnce(t *testing.T) {
var buf bytes.Buffer
c := &Collector{logger: slog.New(slog.NewTextHandler(&buf, nil))}
guests := []Guest{{VMID: 9201, Spec: &GuestSpec{SwapBytes: 4096 << 20}}, {VMID: 9202, Spec: nil}}
var ns proxmox.NodeStatus // a host with no swap
for i := 0; i < 3; i++ {
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if h.HostSwapMB == nil || h.GuestSwapMB == nil || *h.HostSwapMB != 0 || *h.GuestSwapMB != 4096 {
t.Fatalf("report numbers: host %v guest %v", h.HostSwapMB, h.GuestSwapMB)
}
}
if n := strings.Count(buf.String(), "host-swap: the host's swap is below"); n != 1 {
t.Fatalf("the short WARN must be logged once, got %d:\n%s", n, buf.String())
}
ns.Swap.Total = 7935 << 20 // the installer's 8 GB host
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if *h.HostSwapMB != 7935 || HostSwapShort(*h.HostSwapMB, *h.GuestSwapMB) {
t.Fatalf("a covered host must not be short: %v / %v", *h.HostSwapMB, *h.GuestSwapMB)
}
if !strings.Contains(buf.String(), "covers its guests' swap allowance again") {
t.Fatalf("the recovery must be said once:\n%s", buf.String())
}
if HostSwapShort(0, 0) {
t.Fatal("a host whose guests have no swap allowance is never short")
}
}
+92 -1
View File
@@ -129,6 +129,17 @@ type HostReport struct {
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret. // wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"` GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
// LineTest is the weekly upload line test stanza (R-1247; operator ruling 2026-10-24): the schedule, the last
// measured upload line (best of the runs, Mbit/s, as ep0's iperf3 server counted it) and the last attempt. Present
// only when the job is wired. No secret.
LineTest *LineTestStatus `json:"line_test,omitempty"`
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
// the agent started (the hub keeps its last state); `tiers: []` = evaluated, none found.
ForeignKeyArchives *ForeignKeyArchivesStanza `json:"foreign_key_archives,omitempty"`
// LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent // LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent
// mirror of the controller's report log_tails channel. Present ONLY on the heartbeat // mirror of the controller's report log_tails channel. Present ONLY on the heartbeat
// right after the control envelope requested it (log_tail_requested); consume-once on // right after the control envelope requested it (log_tail_requested); consume-once on
@@ -138,6 +149,12 @@ type HostReport struct {
// in the steady state, so the cross-repo host-report golden stays byte-stable. // in the steady state, so the cross-repo host-report golden stays byte-stable.
LogTail *LogTail `json:"log_tail,omitempty"` LogTail *LogTail `json:"log_tail,omitempty"`
// DeliberateOff (R-998, doc 14): the household asked this box to power off, in THIS boot. Present on
// every report from the request until the box goes down; absent again after the next boot — that
// absence is how the hub learns the box is back. The hub holds its silence alarms while it is set
// (14 days). `omitempty`: absent in the steady state, so the cross-repo host-report golden stays stable.
DeliberateOff *DeliberateOff `json:"deliberate_off,omitempty"`
// OOB is the operator-access health stanza (TASK H1). It answers the operator's question — "can I // OOB is the operator-access health stanza (TASK H1). It answers the operator's question — "can I
// get into this box right now, and if not, why" — from the hub: felhom-sshd up + on which port, // get into this box right now, and if not, why" — from the hub: felhom-sshd up + on which port,
// locally reachable, the tunnel handshake age (the OOB path rides wg-felhom), whether the operator // locally reachable, the tunnel handshake age (the OOB path rides wg-felhom), whether the operator
@@ -220,6 +237,19 @@ type GuestDiskTrimStatus struct {
Guests []GuestDiskTrim `json:"guests,omitempty"` Guests []GuestDiskTrim `json:"guests,omitempty"`
} }
// LineTestStatus is the R-1247 weekly upload line test stanza. `last_ok_at` + `mbps` are the last SUCCESSFUL
// measurement ("" / 0 = never); `last_attempt_at` + `last_error` the last attempt, whatever its verdict — never read
// the attempt time as a result.
type LineTestStatus struct {
Schedule string `json:"schedule"`
LastOKAt string `json:"last_ok_at,omitempty"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Server string `json:"server,omitempty"`
LastAttemptAt string `json:"last_attempt_at,omitempty"`
LastError string `json:"last_error,omitempty"`
}
// GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never // GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never
// read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is // read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is
// the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points. // the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points.
@@ -280,6 +310,10 @@ type WireguardStatus struct {
Registered bool `json:"registered"` // the registration marker exists Registered bool `json:"registered"` // the registration marker exists
Active bool `json:"active"` // wg-quick@wg-felhom is-active Active bool `json:"active"` // wg-quick@wg-felhom is-active
LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"` LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"`
// HandshakeFutureS (R-1204): seconds the latest handshake lies in the FUTURE by this box's clock — set (> 0) only
// then. A box that ran once with its clock ahead keeps a handshake stamped in the future; WireGuard on ep0 then
// refuses every newer handshake, and LastHandshakeAgeS (clamped at 0) reads „just now" for ever. The hub alarms on it.
HandshakeFutureS *int64 `json:"handshake_future_s,omitempty"`
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32" AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
} }
@@ -326,6 +360,11 @@ type HostMetrics struct {
// //
// Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift. // Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift.
WrapperSHA256 string `json:"wrapper_sha256,omitempty"` WrapperSHA256 string `json:"wrapper_sha256,omitempty"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up, night 2026-10-21): the host's swap (Proxmox node status — the
// kernel's SwapTotal) and the sum of its guests' swap allowance. The guest swap fix (swap = RAM) needs swap ON THE
// HOST behind it; the hub raises host_swap_short when the first is below the second. nil = not read (older agent).
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
} }
// Guest is one LXC. The agent reports vmid; the hub derives the guest PK // Guest is one LXC. The agent reports vmid; the hub derives the guest PK
@@ -343,11 +382,12 @@ type GuestSpec struct {
Cores int `json:"cores"` Cores int `json:"cores"`
MemoryBytes int64 `json:"memory_bytes"` MemoryBytes int64 `json:"memory_bytes"`
DiskBytes int64 `json:"disk_bytes"` DiskBytes int64 `json:"disk_bytes"`
SwapBytes int64 `json:"swap_bytes,omitempty"` // R-1192: the guest's swap allowance (PVE config `swap`, MB → bytes)
} }
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest. // Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
type Cloudflared struct { type Cloudflared struct {
Status string `json:"status"` // running | not_running | unknown (TunnelRunning …) Status string `json:"status"` // running | not_running | unknown | absent (TunnelRunning …; absent since R-1070)
Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected" Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected"
} }
@@ -419,6 +459,15 @@ type StorageTarget struct {
// metadata fill is null until Phase B's privileged `lvs` read. // metadata fill is null until Phase B's privileged `lvs` read.
ThinPool *ThinPoolFill `json:"thin_pool,omitempty"` ThinPool *ThinPoolFill `json:"thin_pool,omitempty"`
// Physical-disk identity (felhom.eu architecture/16-storage-physical-identity.md §3, agent v0.157.0):
// the place's kind, the keys of the whole physical disks under it, and whether the agent can prove
// them (proven | unknown + identity_reason vm|network|usb_bridge|weak_wwn|no_serial). Agent-computed
// from /sys + udev only; empty when not computed (an absent device, an older agent). Additive.
Kind string `json:"kind,omitempty"`
PhysicalDisks []string `json:"physical_disks,omitempty"`
Identity string `json:"identity,omitempty"`
IdentityReason string `json:"identity_reason,omitempty"`
// Smart is the disk-health summary, populated in Phase B via smartctl (sudoers). In // Smart is the disk-health summary, populated in Phase B via smartctl (sudoers). In
// Phase A it is {health:"UNKNOWN", <counters null>} — the keys are committed now so the // Phase A it is {health:"UNKNOWN", <counters null>} — the keys are committed now so the
// contract is stable across both phases. // contract is stable across both phases.
@@ -454,6 +503,17 @@ type SmartSummary struct {
ReallocatedSectors *int `json:"reallocated_sectors"` ReallocatedSectors *int `json:"reallocated_sectors"`
PendingSectors *int `json:"pending_sectors"` PendingSectors *int `json:"pending_sectors"`
OfflineUncorrectable *int `json:"offline_uncorrectable"` OfflineUncorrectable *int `json:"offline_uncorrectable"`
// R-330 (disk health Phase 2): three more SATA raw counters. omitempty + pointer: absent (an
// older agent, an NVMe/USB device, or a drive that does not report the attribute) is OMITTED —
// unknown, never a zero (S-39). Wire only: no verdict reads them yet.
// 187 Reported_Uncorrect — the failing drive's most telling counter (normalized 1 vs thresh 0,
// raw 1001) while SMART still said PASSED.
// 188 Command_Timeout — some vendors PACK several counters into the 48-bit raw value, so the
// number is carried as reported and must not be compared across vendors.
// 199 UDMA_CRC_Error_Count — cabling / link errors, not the medium.
ReportedUncorrect *int64 `json:"reported_uncorrect,omitempty"`
CommandTimeout *int64 `json:"command_timeout,omitempty"`
UDMACRCErrors *int64 `json:"udma_crc_errors,omitempty"`
// NVMe attributes. // NVMe attributes.
CriticalWarning *int `json:"critical_warning"` CriticalWarning *int `json:"critical_warning"`
@@ -619,6 +679,17 @@ type WireOSUpdate struct {
// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version // HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
// approved for the guest is not approved for the host by that fact alone. // approved for the guest is not approved for the host by that fact alone.
HostRelease *WireOSRelease `json:"host_release,omitempty"` HostRelease *WireOSRelease `json:"host_release,omitempty"`
// Kernel is the kernel lane's instruction for THIS box (R-836, `09` §3 decision 172, `11` §5.11): the kernel the
// household was told about, and whether tonight is a told night. Nil (an older hub, or no kernel due) = no kernel
// step. The hub sets Tonight only after the household's mail the day before went out — no mail, no step.
Kernel *WireKernelStep `json:"kernel,omitempty"`
}
// WireKernelStep is the hub's kernel-lane instruction (hub osupdates.KernelBlock — field-exact, cross-repo).
type WireKernelStep struct {
Kver string `json:"kver"` // e.g. "7.0.14-22-pve" — the wrapper refuses any other (R23)
Tonight bool `json:"tonight"` // the household was mailed the day before: tonight's leg may reboot
NotifiedAt string `json:"notified_at,omitempty"` // when that mail went out (RFC 3339), for the log
} }
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses // WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
@@ -715,3 +786,23 @@ type WireRestoreDirective struct {
Archive string `json:"archive,omitempty"` // source archive/snapshot to restore from Archive string `json:"archive,omitempty"` // source archive/snapshot to restore from
VMID int `json:"vmid,omitempty"` VMID int `json:"vmid,omitempty"`
} }
// ForeignKeyArchivesStanza wraps the per-tier list so "evaluated, none" (`tiers: []`) differs from "not evaluated"
// (the stanza absent) without a null on the wire.
type ForeignKeyArchivesStanza struct {
Tiers []ForeignKeyArchives `json:"tiers"`
}
// ForeignKeyArchives is one tier's count of archives written with another key (R-366 slice 2): the count and the
// newest/oldest archive time (RFC3339, UTC). No key material — the fingerprints stay on the box.
type ForeignKeyArchives struct {
Target string `json:"target"`
Count int `json:"count"`
Oldest string `json:"oldest"`
Newest string `json:"newest"`
}
// DeliberateOff is the household-requested power-off record (R-998).
type DeliberateOff struct {
RequestedAt string `json:"requested_at"` // RFC3339 UTC, the agent's clock
}
+7
View File
@@ -46,6 +46,9 @@
"backing_device": "", "backing_device": "",
"class_hint": "fast", "class_hint": "fast",
"role": "", "role": "",
"kind": "system",
"physical_disks": ["serial:ata:SanDisk_X600:182195804614"],
"identity": "proven",
"thin_pool": { "data_used_fraction": 0.42, "metadata_used_fraction": null }, "thin_pool": { "data_used_fraction": 0.42, "metadata_used_fraction": null },
"smart": { "smart": {
"health": "UNKNOWN", "health": "UNKNOWN",
@@ -74,6 +77,10 @@
"backing_device": "/dev/sdb1", "backing_device": "/dev/sdb1",
"class_hint": "slow", "class_hint": "slow",
"role": "", "role": "",
"kind": "disk",
"physical_disks": ["serial:usb:External:000000000000"],
"identity": "unknown",
"identity_reason": "usb_bridge",
"smart": { "smart": {
"health": "UNKNOWN", "health": "UNKNOWN",
"temperature_c": null, "temperature_c": null,
+294
View File
@@ -0,0 +1,294 @@
package linetest
import (
"context"
"crypto/rand"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"strconv"
"sync"
"sync/atomic"
"time"
)
// A minimal iperf3 CLIENT for one TCP upload test (client → server), speaking the iperf3 control protocol (3.x) to the
// stock `iperf3 --server` on ep0 (operator ruling 2026-10-24: the agent speaks iperf3 itself, nothing is installed on
// the host). The number that counts is the SERVER's: the bytes it received, from its results JSON — bytes the client
// handed to its socket include what still sat in the send buffer when the timer ended.
//
// The protocol, as the server drives it (one state byte at a time on the control connection):
//
// client → 37-byte cookie
// server → PARAM_EXCHANGE(9) client → params JSON (4-byte big-endian length + bytes)
// server → CREATE_STREAMS(10) client opens N data connections, each starts with the same cookie
// server → TEST_START(1), TEST_RUNNING(2) client sends for the test time, then → TEST_END(4)
// server → EXCHANGE_RESULTS(13) client → its results JSON; server → its results JSON
// server → DISPLAY_RESULTS(14) client → IPERF_DONE(16)
//
// A server already running a test answers ACCESS_DENIED(-1) and closes: that is ErrBusy (another box is testing; try
// again later the same day). SERVER_ERROR(-3) / SERVER_TERMINATE(-2) end the test with an error.
const (
stParamExchange = 9
stCreateStreams = 10
stTestStart = 1
stTestRunning = 2
stTestEnd = 4
stExchangeResults = 13
stDisplayResults = 14
stIperfDone = 16
stAccessDenied = -1
stServerTerminate = -2
stServerError = -3
cookieSize = 37 // 36 characters + NUL
blockSize = 128 * 1024
maxJSONBytes = 1 << 20
)
// ErrBusy: the server is running another client's test.
var ErrBusy = errors.New("iperf3 server busy")
// Result is one upload test.
type Result struct {
Mbps float64 // the server's received bits per second / 1e6
ServerBytes int64 // bytes the server received, all streams
ClientBytes int64 // bytes the client wrote (diagnostics only)
ServerSeconds float64 // the server's measured duration
}
// Dialer opens a TCP connection (net.Dialer in production; a seam for tests).
type Dialer interface {
DialContext(ctx context.Context, network, address string) (net.Conn, error)
}
// Client runs one test. The whole run is bounded by ctx; Run sets every connection's deadline from it.
type Client struct {
Dialer Dialer
Streams int // parallel streams (-P)
Time time.Duration // test time (-t)
}
func (c Client) dialer() Dialer {
if c.Dialer != nil {
return c.Dialer
}
return &net.Dialer{Timeout: 5 * time.Second}
}
func newCookie() ([]byte, error) {
const alphabet = "abcdefghijklmnopqrstuvwxyz234567"
b := make([]byte, cookieSize)
if _, err := rand.Read(b[:cookieSize-1]); err != nil {
return nil, err
}
for i := 0; i < cookieSize-1; i++ {
b[i] = alphabet[int(b[i])%len(alphabet)]
}
b[cookieSize-1] = 0
return b, nil
}
func writeJSON(w io.Writer, v any) error {
data, err := json.Marshal(v)
if err != nil {
return err
}
var hdr [4]byte
binary.BigEndian.PutUint32(hdr[:], uint32(len(data)))
if _, err := w.Write(append(hdr[:], data...)); err != nil {
return err
}
return nil
}
func readJSON(r io.Reader, v any) error {
var hdr [4]byte
if _, err := io.ReadFull(r, hdr[:]); err != nil {
return err
}
n := binary.BigEndian.Uint32(hdr[:])
if n == 0 || n > maxJSONBytes {
return fmt.Errorf("iperf3: results JSON of %d bytes refused", n)
}
buf := make([]byte, n)
if _, err := io.ReadFull(r, buf); err != nil {
return err
}
return json.Unmarshal(buf, v)
}
type streamResult struct {
ID int `json:"id"`
Bytes int64 `json:"bytes"`
Retransmits int64 `json:"retransmits"`
Jitter float64 `json:"jitter"`
Errors int64 `json:"errors"`
OmittedErrors int64 `json:"omitted_errors"`
Packets int64 `json:"packets"`
OmittedPackets int64 `json:"omitted_packets"`
StartTime float64 `json:"start_time"`
EndTime float64 `json:"end_time"`
}
type results struct {
CPUTotal float64 `json:"cpu_util_total"`
CPUUser float64 `json:"cpu_util_user"`
CPUSystem float64 `json:"cpu_util_system"`
SenderHasRetransmits int `json:"sender_has_retransmits"`
CongestionUsed string `json:"congestion_used,omitempty"`
Streams []streamResult `json:"streams"`
}
// streamIDs are the ids iperf3 gives its streams: the first is 1, every later one is (streams so far) + 2 — so 1, 3,
// 4, 5 … (iperf_add_stream). The server matches our results by these ids.
func streamIDs(n int) []int {
ids := make([]int, n)
for i := range ids {
if i == 0 {
ids[i] = 1
} else {
ids[i] = i + 2
}
}
return ids
}
// Run performs one upload test to addr ("10.77.0.1:5201").
func (c Client) Run(ctx context.Context, addr string) (Result, error) {
var res Result
streams, dur := c.Streams, c.Time
if streams < 1 {
streams = 1
}
if dur <= 0 {
dur = 10 * time.Second
}
deadline, ok := ctx.Deadline()
if !ok {
return res, errors.New("iperf3: Run needs a context with a deadline")
}
cookie, err := newCookie()
if err != nil {
return res, err
}
ctrl, err := c.dialer().DialContext(ctx, "tcp", addr)
if err != nil {
return res, fmt.Errorf("iperf3: connect: %w", err)
}
defer ctrl.Close()
ctrl.SetDeadline(deadline)
if _, err := ctrl.Write(cookie); err != nil {
return res, fmt.Errorf("iperf3: cookie: %w", err)
}
var data []net.Conn
defer func() {
for _, d := range data {
d.Close()
}
}()
var sent atomic.Int64
var server results
var wg sync.WaitGroup
for {
var st [1]byte
if _, err := io.ReadFull(ctrl, st[:]); err != nil {
return res, fmt.Errorf("iperf3: control read: %w", err)
}
switch int8(st[0]) {
case stParamExchange:
params := map[string]any{"tcp": true, "omit": 0, "time": int(dur / time.Second), "num": 0, "blockcount": 0,
"parallel": streams, "len": blockSize, "pacing_timer": 1000, "client_version": "3.18"}
if err := writeJSON(ctrl, params); err != nil {
return res, fmt.Errorf("iperf3: params: %w", err)
}
case stCreateStreams:
for i := 0; i < streams; i++ {
d, err := c.dialer().DialContext(ctx, "tcp", addr)
if err != nil {
return res, fmt.Errorf("iperf3: data stream %d: %w", i+1, err)
}
d.SetDeadline(deadline)
data = append(data, d)
if _, err := d.Write(cookie); err != nil {
return res, fmt.Errorf("iperf3: data stream %d cookie: %w", i+1, err)
}
}
case stTestStart:
case stTestRunning:
stopAt := time.Now().Add(dur)
if stopAt.After(deadline) {
return res, errors.New("iperf3: no time left for the test inside the deadline")
}
buf := make([]byte, blockSize)
for _, d := range data {
d.SetWriteDeadline(stopAt)
wg.Add(1)
go func(d net.Conn) {
defer wg.Done()
for {
n, err := d.Write(buf)
sent.Add(int64(n))
if err != nil {
return
}
}
}(d)
}
wg.Wait()
if _, err := ctrl.Write([]byte{stTestEnd}); err != nil {
return res, fmt.Errorf("iperf3: test end: %w", err)
}
case stExchangeResults:
ids := streamIDs(len(data))
mine := results{SenderHasRetransmits: -1, CongestionUsed: "cubic"}
per := sent.Load() / int64(max(len(data), 1))
for _, id := range ids {
mine.Streams = append(mine.Streams, streamResult{ID: id, Bytes: per, Retransmits: -1, EndTime: dur.Seconds()})
}
if err := writeJSON(ctrl, mine); err != nil {
return res, fmt.Errorf("iperf3: send results: %w", err)
}
if err := readJSON(ctrl, &server); err != nil {
return res, fmt.Errorf("iperf3: read server results: %w", err)
}
case stDisplayResults:
ctrl.Write([]byte{stIperfDone})
return summarize(server, sent.Load(), dur)
case stAccessDenied:
return res, ErrBusy
case stServerTerminate, stServerError:
var codes [8]byte
io.ReadFull(ctrl, codes[:])
return res, fmt.Errorf("iperf3: server error (state %d, iperf errno %d, errno %d)", int8(st[0]),
int32(binary.BigEndian.Uint32(codes[:4])), int32(binary.BigEndian.Uint32(codes[4:])))
default:
return res, fmt.Errorf("iperf3: unexpected state %s", strconv.Itoa(int(int8(st[0]))))
}
}
}
// summarize turns the server's results into a Result. No stream bytes is an error, never a zero line.
func summarize(server results, clientBytes int64, dur time.Duration) (Result, error) {
res := Result{ClientBytes: clientBytes}
for _, s := range server.Streams {
res.ServerBytes += s.Bytes
if s.EndTime > res.ServerSeconds {
res.ServerSeconds = s.EndTime
}
}
if res.ServerBytes <= 0 {
return res, errors.New("iperf3: the server reported no received bytes")
}
secs := res.ServerSeconds
if secs <= 0 {
secs = dur.Seconds()
}
res.Mbps = float64(res.ServerBytes) * 8 / secs / 1e6
return res, nil
}
+541
View File
@@ -0,0 +1,541 @@
// Package linetest is the weekly upload line-speed test (R-1247; operator ruling 2026-10-24 07:26).
//
// Why: the controller's upload watch and qBittorrent's default limit (50 % of the line, D41) need the household's
// upload line. The passive reading — the nightly off-site push — is only a low floor (2026-10-24 night: demo-hp „at
// least 10 Mbit/s"; this test read 90.7 on the same line). So once a week the HOST (the tunnel lives on the host)
// runs a 10-second iperf3 upload to ep0's tunnel-only iperf3 server (felhom.eu scripts/ep0-iperf3). No file is left
// anywhere and nothing counts against the household's storage.
//
// The rule, each part pinned by a test in linetest_test.go:
// - Weekly: due from Tuesday 10:00 local until one measurement succeeded since then (a box that was off catches up
// on a later day, inside the window).
// - Daytime only: a test starts only 10:00–15:59 local — never in the night chain's window (01:00–06:59), and each
// host starts from its own minute (SlotMinute, spread over 10:00–13:59) so the fleet does not test at once.
// - Never beside a backup, a restore-test, an OS update or a trim: the test holds the host-wide one-heavy-op gate.
// - Only after QuietMinutes of quiet uplink: every one-minute transmit rate of the host's physical uplink below
// QuietMbps — so neither the family's upload nor a restore is measured against itself.
// - Up to RunsPerMeasurement runs of TestTime each; the highest counts. A run is bounded by RunTimeout.
// - „Server busy" (another box is testing) → tried again at the next tick the same day; at most MaxTriesPerDay.
// - No tunnel (no operational wg-felhom with a server address) → no test and no value; the controller then keeps its
// passive floor and sets no qBittorrent limit.
// - The last result is persisted, so a restart neither loses it nor re-tests.
// - A box whose torrent app seeds all day is never quiet for 10 minutes (R-1250). For it the controller holds the
// app's upload limit very low and asks for a test on request (RunNow): every rule above except the 10 quiet
// minutes still holds, the quiet is checked over RequestQuietWindow instead, and the runs are RequestRunGap
// apart, so the whole request ends in about 55 s.
package linetest
import (
"context"
"encoding/json"
"errors"
"fmt"
"hash/fnv"
"log/slog"
"net/netip"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
const (
Weekday = time.Tuesday
StartHour = 10 // first eligible local hour (inclusive)
EndHour = 16 // first NOT-eligible local hour: the last start is 15:59
SlotSpreadMinutes = 240
QuietMinutes = 10
QuietMbps = 2.0
RunsPerMeasurement = 3
TestTime = 10 * time.Second
Streams = 4
RunTimeout = 25 * time.Second
RunGap = 30 * time.Second
MaxTriesPerDay = 6
ServerPort = 5201
TickInterval = time.Minute
FirstTickDelay = 5 * time.Minute
GateName = "line-test"
// A test on request (R-1250): the controller holds the app's limit while it waits, so it is short.
RequestSettle = 5 * time.Second // the held app's last packets leave before the quiet is read
RequestQuietWindow = 10 * time.Second // the uplink must send below QuietMbps over this window
RequestRunGap = 5 * time.Second
)
// ScheduleText is carried on the host report.
const ScheduleText = "weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update"
// Gate is the host-wide one-heavy-operation gate (*backup.InFlight).
type Gate interface {
TryAcquire(what string) (release func(), busy string, ok bool)
}
// Target answers the iperf3 server's tunnel address, ok=false when the tunnel is not operational.
type Target func() (netip.Addr, bool)
// Runner runs one upload test (Client in production).
type Runner interface {
Run(ctx context.Context, addr string) (Result, error)
}
// UplinkTx reads the host's physical-uplink transmit byte counter.
type UplinkTx func() (uint64, error)
// State is what is persisted (line-test.json).
type State struct {
LastOKAt time.Time `json:"last_ok_at,omitempty"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Server string `json:"server,omitempty"`
LastAttemptAt time.Time `json:"last_attempt_at,omitempty"`
LastError string `json:"last_error,omitempty"`
TriesDay string `json:"tries_day,omitempty"` // local date of TriesToday
TriesToday int `json:"tries_today,omitempty"`
}
type txSample struct {
at time.Time
tx uint64
}
// Tester is the weekly job.
type Tester struct {
runner Runner
target Target
gate Gate
uplink UplinkTx
statePath string
slot int // minutes after StartHour:00 this host may first start
logger *slog.Logger
loc *time.Location
now func() time.Time
sleep func(context.Context, time.Duration) bool
mu sync.Mutex
st State
samples []txSample
}
// New builds the job and loads the persisted state (missing = empty; corrupt = logged, empty — the cost is one extra
// test, never a lost value elsewhere).
func New(runner Runner, target Target, gate Gate, uplink UplinkTx, hostID, statePath string, logger *slog.Logger) *Tester {
if logger == nil {
logger = slog.Default()
}
t := &Tester{runner: runner, target: target, gate: gate, uplink: uplink, statePath: statePath,
slot: SlotMinute(hostID), logger: logger, loc: time.Local, now: time.Now, sleep: sleepCtx}
t.load()
return t
}
func sleepCtx(ctx context.Context, d time.Duration) bool {
tm := time.NewTimer(d)
defer tm.Stop()
select {
case <-ctx.Done():
return false
case <-tm.C:
return true
}
}
// SlotMinute spreads hosts over the first SlotSpreadMinutes of the window, stable per host id.
func SlotMinute(hostID string) int {
h := fnv.New32a()
h.Write([]byte(hostID))
return int(h.Sum32() % SlotSpreadMinutes)
}
func (t *Tester) load() {
data, err := os.ReadFile(t.statePath)
if err != nil {
if !os.IsNotExist(err) {
t.logger.Warn("linetest: state read failed — starting empty", "path", t.statePath, "err", err)
}
return
}
if err := json.Unmarshal(data, &t.st); err != nil {
t.logger.Warn("linetest: state file corrupt — starting empty", "path", t.statePath, "err", err)
t.st = State{}
}
}
func (t *Tester) saveLocked() error {
data, err := json.MarshalIndent(t.st, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(t.statePath), 0o755); err != nil {
return err
}
tmp := t.statePath + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, t.statePath)
}
// EligibleAt reports whether a test may START at local time lt for a host whose slot is slot minutes after
// StartHour:00 on the due day. On a catch-up day (anchorDay=false) the slot is not waited for again.
func EligibleAt(lt time.Time, slot int, anchorDay bool) bool {
h := lt.Hour()
if h < StartHour || h >= EndHour {
return false
}
if anchorDay {
mins := (h-StartHour)*60 + lt.Minute()
return mins >= slot
}
return true
}
// weekAnchor is the most recent Tuesday StartHour:00 at or before lt.
func weekAnchor(lt time.Time) time.Time {
daysBack := (int(lt.Weekday()) - int(Weekday) + 7) % 7
d := lt.AddDate(0, 0, -daysBack)
a := time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
if a.After(lt) {
d = d.AddDate(0, 0, -7)
a = time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
}
return a
}
// Due reports whether a measurement is due at lt given the state.
func Due(st State, lt time.Time) bool {
return st.LastOKAt.IsZero() || st.LastOKAt.In(lt.Location()).Before(weekAnchor(lt))
}
// sample records the uplink counter (once a minute) and keeps QuietMinutes+1 samples.
func (t *Tester) sample() {
if t.uplink == nil {
return
}
tx, err := t.uplink()
if err != nil {
t.logger.Debug("linetest: uplink counter unreadable", "err", err)
t.mu.Lock()
t.samples = nil // a gap is not quiet
t.mu.Unlock()
return
}
t.mu.Lock()
t.samples = append(t.samples, txSample{at: t.now(), tx: tx})
if len(t.samples) > QuietMinutes+1 {
t.samples = t.samples[len(t.samples)-(QuietMinutes+1):]
}
t.mu.Unlock()
}
// quiet reports whether the last QuietMinutes minutes each sent below QuietMbps, and the highest minute seen.
func quiet(samples []txSample) (bool, float64) {
if len(samples) < QuietMinutes+1 {
return false, 0
}
worst := 0.0
for i := 1; i < len(samples); i++ {
dt := samples[i].at.Sub(samples[i-1].at).Seconds()
if dt <= 0 || dt > 3*TickInterval.Seconds() || samples[i].tx < samples[i-1].tx {
return false, worst // a gap or a counter reset is not proof of quiet
}
mbps := float64(samples[i].tx-samples[i-1].tx) * 8 / dt / 1e6
if mbps > worst {
worst = mbps
}
}
return worst < QuietMbps, worst
}
// Run ticks every TickInterval until ctx ends. It never returns an error: a failed test is a reported fact.
func (t *Tester) Run(ctx context.Context) {
t.logger.Info("linetest: weekly upload line test starting", "schedule", ScheduleText, "slot_minute", t.slot)
if !t.sleep(ctx, FirstTickDelay) {
return
}
for {
t.sample()
t.Tick(ctx)
if !t.sleep(ctx, TickInterval) {
return
}
}
}
// Refusal is why a test on request did not run (RunNow). Code is stable for the controller.
type Refusal struct {
Code string // not-due | outside-window | tries-used | no-tunnel | heavy-op | not-quiet
Msg string
}
func (r *Refusal) Error() string { return "linetest: " + r.Code + ": " + r.Msg }
// ready applies every rule except the quiet one: due this week, inside the window (the slot on the due day), tries
// left today, the tunnel up. It answers the server address and the local day.
func (t *Tester) ready() (addr, day string, refusal *Refusal) {
lt := t.now().In(t.loc)
t.mu.Lock()
st := t.st
t.mu.Unlock()
if !Due(st, lt) {
return "", "", &Refusal{"not-due", "measured this week"}
}
anchor := weekAnchor(lt)
anchorDay := lt.YearDay() == anchor.YearDay() && lt.Year() == anchor.Year()
if !EligibleAt(lt, t.slot, anchorDay) {
return "", "", &Refusal{"outside-window", "a test starts 10:00-15:59 local, from this host's slot on the due day"}
}
day = lt.Format("2006-01-02")
if st.TriesDay == day && st.TriesToday >= MaxTriesPerDay {
return "", "", &Refusal{"tries-used", fmt.Sprintf("%d tries today", st.TriesToday)}
}
ip, ok := t.target()
if !ok {
return "", "", &Refusal{"no-tunnel", "no operational tunnel"}
}
return netip.AddrPortFrom(ip, ServerPort).String(), day, nil
}
// Tick is one look.
func (t *Tester) Tick(ctx context.Context) {
addr, day, refusal := t.ready()
if refusal != nil {
if refusal.Code == "no-tunnel" {
t.logger.Debug("linetest: no operational tunnel — no test")
}
return
}
t.mu.Lock()
samples := append([]txSample(nil), t.samples...)
t.mu.Unlock()
if q, worst := quiet(samples); !q {
t.logger.Debug("linetest: uplink not quiet — waiting", "worst_minute_mbps", fmt.Sprintf("%.1f", worst), "samples", len(samples))
return
}
release, busy, ok := t.gate.TryAcquire(GateName)
if !ok {
t.logger.Info("linetest: deferred — a heavy operation is in flight", "busy", busy)
return
}
defer release()
t.measure(ctx, addr, day, RunGap)
}
// WaitingForQuiet reports that a measurement is due and could start now but the uplink has not been quiet for
// QuietMinutes — the case a seeding torrent app makes permanent (R-1250). The controller reads it from
// GET /host/line-test and only then holds its app's limit and calls RunNow.
func (t *Tester) WaitingForQuiet() bool {
if _, _, refusal := t.ready(); refusal != nil {
return false
}
t.mu.Lock()
samples := append([]txSample(nil), t.samples...)
t.mu.Unlock()
if len(samples) < QuietMinutes+1 {
return false // not yet watched for 10 minutes (a fresh start): not known to be busy, so no hold
}
q, _ := quiet(samples)
return !q
}
// RunNow is a test on request (POST /host/line-test/run, R-1250): every rule of the weekly test except the 10 quiet
// minutes, which the caller replaces by holding its own upload down. The uplink must still be quiet over
// RequestQuietWindow (after RequestSettle), so a request never measures the family's upload against itself. A
// *Refusal says why nothing ran; any other error is a failed measurement (the same as a failed weekly attempt).
func (t *Tester) RunNow(ctx context.Context) (mbps float64, runs []float64, err error) {
addr, day, refusal := t.ready()
if refusal != nil {
return 0, nil, refusal
}
release, busy, ok := t.gate.TryAcquire(GateName)
if !ok {
return 0, nil, &Refusal{"heavy-op", busy}
}
defer release()
if t.uplink == nil {
return 0, nil, &Refusal{"not-quiet", "no uplink counter"}
}
if !t.sleep(ctx, RequestSettle) {
return 0, nil, ctx.Err()
}
tx0, err0 := t.uplink()
at0 := t.now()
if !t.sleep(ctx, RequestQuietWindow) {
return 0, nil, ctx.Err()
}
tx1, err1 := t.uplink()
if err0 != nil || err1 != nil || tx1 < tx0 {
return 0, nil, &Refusal{"not-quiet", "uplink counter unreadable or reset"}
}
secs := t.now().Sub(at0).Seconds()
if secs <= 0 {
secs = RequestQuietWindow.Seconds()
}
if m := float64(tx1-tx0) * 8 / secs / 1e6; m >= QuietMbps {
t.logger.Info("linetest: test on request refused — the uplink is not quiet", "mbps", fmt.Sprintf("%.1f", m))
return 0, nil, &Refusal{"not-quiet", fmt.Sprintf("the uplink sent %.1f Mbit/s while held", m)}
}
t.logger.Info("linetest: test on request (a held torrent app, R-1250)", "server", addr)
if err := t.measure(ctx, addr, day, RequestRunGap); err != nil {
return 0, nil, err
}
t.mu.Lock()
defer t.mu.Unlock()
return t.st.Mbps, append([]float64(nil), t.st.RunsMbps...), nil
}
// measure runs the runs, gap apart, and stores the result. It returns nil only when a value was stored.
func (t *Tester) measure(ctx context.Context, addr, day string, gap time.Duration) error {
start := t.now()
var runs []float64
var lastErr error
busy := false
for i := 0; i < RunsPerMeasurement; i++ {
if i > 0 && !t.sleep(ctx, gap) {
break
}
rctx, cancel := context.WithTimeout(ctx, RunTimeout)
r, err := t.runner.Run(rctx, addr)
cancel()
if errors.Is(err, ErrBusy) {
busy, lastErr = true, err
break
}
if err != nil {
lastErr = err
t.logger.Warn("linetest: run failed", "run", i+1, "server", addr, "err", err)
continue
}
runs = append(runs, r.Mbps)
t.logger.Info(fmt.Sprintf("linetest: run %d: %.1f Mbit/s", i+1, r.Mbps), "server", addr,
"server_bytes", r.ServerBytes, "server_seconds", r.ServerSeconds)
}
t.mu.Lock()
defer t.mu.Unlock()
if t.st.TriesDay != day {
t.st.TriesDay, t.st.TriesToday = day, 0
}
t.st.TriesToday++
t.st.LastAttemptAt = start.UTC()
if len(runs) > 0 {
best := runs[0]
for _, r := range runs {
if r > best {
best = r
}
}
t.st.LastOKAt, t.st.Mbps, t.st.RunsMbps, t.st.Server, t.st.LastError = start.UTC(), best, runs, addr, ""
t.logger.Info(fmt.Sprintf("linetest: upload line measured: %.1f Mbit/s (best of %d)", best, len(runs)),
"server", addr, "runs", len(runs), "duration_s", t.now().Sub(start).Seconds())
} else {
msg := "no run succeeded"
if lastErr != nil {
msg = lastErr.Error()
}
if len(msg) > 300 {
msg = msg[:300]
}
t.st.LastError = msg
if busy {
t.logger.Info("linetest: the test server is busy — trying again later today", "server", addr, "tries_today", t.st.TriesToday)
} else {
t.logger.Warn("linetest: measurement failed", "server", addr, "tries_today", t.st.TriesToday, "err", msg)
}
}
if err := t.saveLocked(); err != nil {
t.logger.Warn("linetest: state save failed — the result will not survive a restart", "path", t.statePath, "err", err)
}
if len(runs) > 0 {
return nil
}
if lastErr == nil {
lastErr = errors.New("linetest: no run succeeded")
}
return lastErr
}
// Value is the measured line (ok=false: never measured).
func (t *Tester) Value() (mbps float64, at time.Time, ok bool) {
t.mu.Lock()
defer t.mu.Unlock()
if t.st.LastOKAt.IsZero() || t.st.Mbps <= 0 {
return 0, time.Time{}, false
}
return t.st.Mbps, t.st.LastOKAt, true
}
// LineTestStatus implements hub.LineTestReporter: a pure read of the persisted state.
func (t *Tester) LineTestStatus(context.Context) *hub.LineTestStatus {
t.mu.Lock()
defer t.mu.Unlock()
out := &hub.LineTestStatus{Schedule: ScheduleText, Mbps: t.st.Mbps, RunsMbps: t.st.RunsMbps, Server: t.st.Server,
LastError: t.st.LastError}
if !t.st.LastOKAt.IsZero() {
out.LastOKAt = t.st.LastOKAt.UTC().Format(time.RFC3339)
}
if !t.st.LastAttemptAt.IsZero() {
out.LastAttemptAt = t.st.LastAttemptAt.UTC().Format(time.RFC3339)
}
return out
}
// ── the host's physical uplink counter ──────────────────────────────────────────────────────────────────────────
// SysUplinkTx reads the transmit bytes of the host's physical uplink under root (normally "/"): the interface of the
// default route in /proc/net/route, and — when it is a bridge (vmbr0), as on every Proxmox host — the sum of its
// PHYSICAL member ports (those with a `device` link). The bridge's own counter does not see the guests' bridged
// traffic, which is the traffic this check exists for.
func SysUplinkTx(root string) UplinkTx {
return func() (uint64, error) {
ifc, err := defaultRouteIface(filepath.Join(root, "proc/net/route"))
if err != nil {
return 0, err
}
sys := filepath.Join(root, "sys/class/net")
ports := []string{ifc}
if ents, err := os.ReadDir(filepath.Join(sys, ifc, "brif")); err == nil {
ports = ports[:0]
for _, e := range ents {
if _, err := os.Stat(filepath.Join(sys, e.Name(), "device")); err == nil {
ports = append(ports, e.Name())
}
}
if len(ports) == 0 {
return 0, fmt.Errorf("linetest: bridge %s has no physical port", ifc)
}
}
var sum uint64
for _, p := range ports {
b, err := os.ReadFile(filepath.Join(sys, p, "statistics/tx_bytes"))
if err != nil {
return 0, err
}
n, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
if err != nil {
return 0, err
}
sum += n
}
return sum, nil
}
}
func defaultRouteIface(path string) (string, error) {
b, err := os.ReadFile(path)
if err != nil {
return "", err
}
for _, line := range strings.Split(string(b), "\n")[1:] {
f := strings.Fields(line)
if len(f) >= 8 && f[1] == "00000000" && f[7] == "00000000" {
return f[0], nil
}
}
return "", errors.New("linetest: no default route")
}
+574
View File
@@ -0,0 +1,574 @@
package linetest
import (
"context"
"encoding/binary"
"encoding/json"
"errors"
"io"
"net"
"net/netip"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// ── a fake iperf3 server (the server half of the protocol, as `iperf3 --server` drives it) ─────────────────────────
type fakeServer struct {
ln net.Listener
busy bool // answer ACCESS_DENIED
stall bool // accept the cookie, then never answer
recvSeconds float64 // what the server reports as its duration
gotIDs []int // the stream ids the client reported
gotParams map[string]any
mu sync.Mutex
received int64
}
func newFakeServer(t *testing.T) *fakeServer {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
s := &fakeServer{ln: ln, recvSeconds: 1}
t.Cleanup(func() { ln.Close() })
go s.serve()
return s
}
func (s *fakeServer) addr() string { return s.ln.Addr().String() }
func wjson(w io.Writer, v any) {
b, _ := json.Marshal(v)
var h [4]byte
binary.BigEndian.PutUint32(h[:], uint32(len(b)))
w.Write(append(h[:], b...))
}
func (s *fakeServer) serve() {
ctrl, err := s.ln.Accept()
if err != nil {
return
}
defer ctrl.Close()
cookie := make([]byte, cookieSize)
if _, err := io.ReadFull(ctrl, cookie); err != nil {
return
}
if s.busy {
ctrl.Write([]byte{0xff})
return
}
if s.stall {
time.Sleep(5 * time.Second)
return
}
ctrl.Write([]byte{stParamExchange})
var params map[string]any
if readJSON(ctrl, &params) != nil {
return
}
s.gotParams = params
n := int(params["parallel"].(float64))
ctrl.Write([]byte{stCreateStreams})
var wg sync.WaitGroup
var conns []net.Conn
for i := 0; i < n; i++ {
d, err := s.ln.Accept()
if err != nil {
return
}
c := make([]byte, cookieSize)
io.ReadFull(d, c)
conns = append(conns, d)
wg.Add(1)
go func(d net.Conn) {
defer wg.Done()
k, _ := io.Copy(io.Discard, d)
s.mu.Lock()
s.received += k
s.mu.Unlock()
}(d)
}
ctrl.Write([]byte{stTestStart, stTestRunning})
var st [1]byte
if _, err := io.ReadFull(ctrl, st[:]); err != nil || st[0] != stTestEnd {
return
}
ctrl.Write([]byte{stExchangeResults})
var theirs results
if readJSON(ctrl, &theirs) != nil {
return
}
for _, x := range theirs.Streams {
s.gotIDs = append(s.gotIDs, x.ID)
}
// The server counts what it RECEIVED; report a fixed figure so the test can tell it from the client's own count.
wjson(ctrl, results{Streams: []streamResult{{ID: 1, Bytes: 12_500_000, EndTime: s.recvSeconds}}})
ctrl.Write([]byte{stDisplayResults})
io.ReadFull(ctrl, st[:])
for _, d := range conns {
d.Close()
}
wg.Wait()
}
func TestClientSpeaksTheProtocolAndCountsTheServersBytes(t *testing.T) {
s := newFakeServer(t)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
r, err := Client{Streams: 4, Time: 300 * time.Millisecond}.Run(ctx, s.addr())
if err != nil {
t.Fatalf("Run: %v", err)
}
if r.ServerBytes != 12_500_000 || r.Mbps != 100 {
t.Fatalf("result = %+v; want the SERVER's 12.5 MB in 1 s = 100 Mbit/s", r)
}
if want := []int{1, 3, 4, 5}; len(s.gotIDs) != 4 || s.gotIDs[0] != want[0] || s.gotIDs[1] != want[1] || s.gotIDs[3] != want[3] {
t.Fatalf("stream ids sent = %v, want %v (iperf3 numbers its streams 1, 3, 4, 5)", s.gotIDs, want)
}
if s.gotParams["tcp"] != true || s.gotParams["parallel"].(float64) != 4 {
t.Fatalf("params = %v", s.gotParams)
}
}
func TestClientBusyServerIsErrBusy(t *testing.T) {
s := newFakeServer(t)
s.busy = true
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if _, err := (Client{Streams: 4, Time: time.Second}).Run(ctx, s.addr()); !errors.Is(err, ErrBusy) {
t.Fatalf("err = %v, want ErrBusy", err)
}
}
func TestClientHonoursTheDeadlineAgainstAStalledServer(t *testing.T) {
s := newFakeServer(t)
s.stall = true
ctx, cancel := context.WithTimeout(context.Background(), 400*time.Millisecond)
defer cancel()
start := time.Now()
_, err := Client{Streams: 4, Time: time.Second}.Run(ctx, s.addr())
if err == nil {
t.Fatal("a stalled server gave no error")
}
if took := time.Since(start); took > 1500*time.Millisecond {
t.Fatalf("Run took %v against a stalled server; the deadline is 400 ms", took)
}
}
func TestClientRefusesAContextWithoutDeadline(t *testing.T) {
if _, err := (Client{}).Run(context.Background(), "127.0.0.1:1"); err == nil {
t.Fatal("Run without a deadline must refuse (an unbounded test is the thing RunTimeout exists to prevent)")
}
}
// ── the schedule ────────────────────────────────────────────────────────────────────────────────────────────────
func TestEligibleWindowNeverInTheNightChain(t *testing.T) {
loc := time.FixedZone("CET", 3600)
start := time.Date(2026, 10, 19, 0, 0, 0, 0, loc) // a Monday
for m := 0; m < 7*24*60; m++ {
lt := start.Add(time.Duration(m) * time.Minute)
for _, anchorDay := range []bool{true, false} {
if !EligibleAt(lt, 0, anchorDay) {
continue
}
if h := lt.Hour(); h < StartHour || h >= EndHour || (h >= 1 && h <= 6) {
t.Fatalf("eligible at %s — outside 10:00-15:59 or inside the night chain (01:00-06:59)", lt.Format("Mon 15:04"))
}
}
}
}
func TestSlotDelaysTheFirstStartOnTheDueDayOnly(t *testing.T) {
loc := time.UTC
tue := time.Date(2026, 10, 20, 10, 30, 0, 0, loc)
if EligibleAt(tue, 45, true) {
t.Fatal("10:30 with slot 45 must wait until 10:45 on the due day")
}
if !EligibleAt(tue.Add(15*time.Minute), 45, true) {
t.Fatal("10:45 with slot 45 must be eligible")
}
if !EligibleAt(tue.Add(24*time.Hour), 45, false) {
t.Fatal("a catch-up day does not wait for the slot again")
}
if a, b := SlotMinute("demo-hp-bb76ea"), SlotMinute("demo-felhom-8363b5"); a < 0 || a >= SlotSpreadMinutes || b < 0 || b >= SlotSpreadMinutes {
t.Fatalf("slots out of range: %d %d", a, b)
}
}
func TestDueWeeklyFromTuesday(t *testing.T) {
loc := time.UTC
mon := time.Date(2026, 10, 19, 12, 0, 0, 0, loc)
tue := time.Date(2026, 10, 20, 12, 0, 0, 0, loc)
if !Due(State{}, mon) {
t.Fatal("never measured is due")
}
st := State{LastOKAt: time.Date(2026, 10, 14, 11, 0, 0, 0, loc)} // last Wednesday
if Due(st, mon) {
t.Fatal("measured after last Tuesday 10:00 is not due on Monday")
}
if !Due(st, tue) {
t.Fatal("from Tuesday 10:00 a new measurement is due")
}
}
// ── the job ─────────────────────────────────────────────────────────────────────────────────────────────────────
type fakeRunner struct {
mu sync.Mutex
calls int
out []runOut
}
type runOut struct {
mbps float64
err error
}
func (f *fakeRunner) Run(ctx context.Context, addr string) (Result, error) {
f.mu.Lock()
defer f.mu.Unlock()
i := f.calls
f.calls++
if i >= len(f.out) {
return Result{}, errors.New("no more scripted runs")
}
return Result{Mbps: f.out[i].mbps, ServerBytes: 1}, f.out[i].err
}
type fakeGate struct{ busy string }
func (g *fakeGate) TryAcquire(what string) (func(), string, bool) {
if g.busy != "" {
return nil, g.busy, false
}
return func() {}, "", true
}
type rig struct {
t *Tester
run *fakeRunner
gate *fakeGate
now time.Time
tx uint64
tunnel bool
}
// newRig: Tuesday 12:00 UTC (eligible, any slot), tunnel up, 11 quiet samples taken.
func newRig(t *testing.T, outs ...runOut) *rig {
r := &rig{run: &fakeRunner{out: outs}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
r.t.loc = time.UTC
r.t.now = func() time.Time { return r.now }
r.t.sleep = func(context.Context, time.Duration) bool { return true }
r.quietMinutes(QuietMinutes + 1)
return r
}
func (r *rig) minute(bytes uint64) {
r.now = r.now.Add(time.Minute)
r.tx += bytes
r.t.sample()
}
func (r *rig) quietMinutes(n int) {
for i := 0; i < n; i++ {
r.minute(1_000_000) // 1 MB/min = 0.13 Mbit/s
}
}
func TestTickMeasuresTheHighestOfThreeRuns(t *testing.T) {
r := newRig(t, runOut{mbps: 88}, runOut{mbps: 91.5}, runOut{mbps: 90})
r.t.Tick(context.Background())
mbps, _, ok := r.t.Value()
if !ok || mbps != 91.5 || r.run.calls != RunsPerMeasurement {
t.Fatalf("value %.1f ok=%v calls=%d; want the highest of 3 runs = 91.5", mbps, ok, r.run.calls)
}
r.t.Tick(context.Background())
if r.run.calls != RunsPerMeasurement {
t.Fatal("a measured week must not test again")
}
}
func TestBusyServerIsRetriedLaterTheSameDay(t *testing.T) {
r := newRig(t, runOut{err: ErrBusy}, runOut{mbps: 90}, runOut{mbps: 89}, runOut{mbps: 92})
r.t.Tick(context.Background())
if _, _, ok := r.t.Value(); ok || r.run.calls != 1 {
t.Fatalf("busy: ok=%v calls=%d; want no value and the measurement stopped after the busy answer", ok, r.run.calls)
}
if r.t.st.TriesToday != 1 || r.t.st.LastError == "" {
t.Fatalf("busy attempt not recorded: %+v", r.t.st)
}
r.quietMinutes(1)
r.t.Tick(context.Background())
if mbps, _, ok := r.t.Value(); !ok || mbps != 92 {
t.Fatalf("retry later the same day: value %.1f ok=%v", mbps, ok)
}
}
func TestTriesPerDayAreBounded(t *testing.T) {
outs := make([]runOut, 20)
for i := range outs {
outs[i] = runOut{err: ErrBusy}
}
r := newRig(t, outs...)
for i := 0; i < 10; i++ {
r.quietMinutes(1)
r.t.Tick(context.Background())
}
if r.run.calls != MaxTriesPerDay {
t.Fatalf("calls = %d in one day, want at most %d", r.run.calls, MaxTriesPerDay)
}
}
func TestNoTunnelNoTest(t *testing.T) {
r := newRig(t, runOut{mbps: 90})
r.tunnel = false
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("tested without an operational tunnel")
}
if _, _, ok := r.t.Value(); ok {
t.Fatal("a value without a test")
}
}
func TestBusyUplinkOrHeavyOpWaits(t *testing.T) {
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
r.minute(60_000_000) // one minute at 8 Mbit/s: the family is uploading
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("tested while the uplink was busy in the last 10 minutes")
}
r.quietMinutes(QuietMinutes)
r.gate.busy = "backup:felhom-pbs"
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("tested beside a heavy operation")
}
r.gate.busy = ""
r.t.Tick(context.Background())
if r.run.calls == 0 {
t.Fatal("never tested once the uplink was quiet and the gate free")
}
}
func TestNeverTestsInTheNightOrBeforeTuesday(t *testing.T) {
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
r.quietMinutes(QuietMinutes + 1)
r.t.Tick(context.Background())
r.t.st.LastOKAt = time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
r.now = time.Date(2026, 10, 19, 12, 0, 0, 0, time.UTC) // Monday noon, measured last week
r.quietMinutes(QuietMinutes + 1)
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatalf("tested %d times at night or when not due", r.run.calls)
}
}
func TestStateSurvivesARestartAndReportsIt(t *testing.T) {
r := newRig(t, runOut{mbps: 90.7}, runOut{err: errors.New("reset")}, runOut{mbps: 80})
r.t.Tick(context.Background())
again := New(r.run, nil, r.gate, nil, "host-x", r.t.statePath, nil)
mbps, at, ok := again.Value()
if !ok || mbps != 90.7 || at.IsZero() {
t.Fatalf("after a restart: %.1f %v %v", mbps, at, ok)
}
b, _ := json.Marshal(again.LineTestStatus(context.Background()))
var back hub.LineTestStatus
json.Unmarshal(b, &back)
if back.Mbps != 90.7 || back.LastOKAt == "" || len(back.RunsMbps) != 2 || back.Schedule == "" {
t.Fatalf("report stanza = %s", b)
}
}
// ── the uplink counter ──────────────────────────────────────────────────────────────────────────────────────────
func TestUplinkCountsTheBridgesPhysicalPortsOnly(t *testing.T) {
root := t.TempDir()
w := func(p, s string) {
os.MkdirAll(filepath.Dir(filepath.Join(root, p)), 0o755)
os.WriteFile(filepath.Join(root, p), []byte(s), 0o644)
}
w("proc/net/route", "Iface\tDestination\tGateway\tFlags\tRefCnt\tUse\tMetric\tMask\n"+
"wg-felhom\t0000004D\t00000000\t0001\t0\t0\t0\tFFFFFFFF\n"+
"vmbr0\t00000000\t0100A8C0\t0003\t0\t0\t0\t00000000\n")
w("sys/class/net/vmbr0/statistics/tx_bytes", "5\n")
w("sys/class/net/vmbr0/brif/enp1s0/.keep", "")
w("sys/class/net/vmbr0/brif/veth9201i0/.keep", "")
w("sys/class/net/enp1s0/device/.keep", "")
w("sys/class/net/enp1s0/statistics/tx_bytes", "1000\n")
w("sys/class/net/veth9201i0/statistics/tx_bytes", "777777\n")
got, err := SysUplinkTx(root)()
if err != nil || got != 1000 {
t.Fatalf("uplink tx = %d, %v; want 1000 (the physical port only — not the bridge, not the guest's veth)", got, err)
}
}
func TestQuietNeedsTenFullMinutes(t *testing.T) {
base := time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC)
var s []txSample
for i := 0; i <= QuietMinutes; i++ {
s = append(s, txSample{at: base.Add(time.Duration(i) * time.Minute), tx: uint64(i) * 1_000_000})
}
if q, _ := quiet(s); !q {
t.Fatal("ten quiet minutes must read quiet")
}
if q, _ := quiet(s[1:]); q {
t.Fatal("nine minutes of samples must not read quiet")
}
gap := append([]txSample(nil), s...)
gap[5].at = gap[5].at.Add(-10 * time.Minute)
if q, _ := quiet(gap); q {
t.Fatal("a gap in the samples must not read quiet")
}
}
// ── a test on request (R-1250): a torrent app that seeds all day ───────────────────────────────────────────────────
// seedingRig: due and eligible, but every minute of the last ten sent at the line's speed — the weekly Tick never runs.
func seedingRig(t *testing.T, outs ...runOut) *rig {
r := newRig(t, outs...)
for i := 0; i < QuietMinutes+1; i++ {
r.minute(600_000_000) // 80 Mbit/s: a seed at the line's speed
}
return r
}
func TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt(t *testing.T) {
r := seedingRig(t, runOut{mbps: 88}, runOut{mbps: 92}, runOut{mbps: 90})
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("the weekly Tick tested a seeding box")
}
if !r.t.WaitingForQuiet() {
t.Fatal("a due, eligible, seeding box must report waiting_for_quiet")
}
var slept []time.Duration
r.t.sleep = func(_ context.Context, d time.Duration) bool { slept = append(slept, d); return true }
mbps, runs, err := r.t.RunNow(context.Background()) // the controller holds the app: the counter stands still
if err != nil || mbps != 92 || len(runs) != RunsPerMeasurement || r.run.calls != RunsPerMeasurement {
t.Fatalf("RunNow = %.1f %v %v, calls %d; want the best of 3 = 92", mbps, runs, err, r.run.calls)
}
want := []time.Duration{RequestSettle, RequestQuietWindow, RequestRunGap, RequestRunGap}
if len(slept) != len(want) {
t.Fatalf("slept %v, want %v", slept, want)
}
var total time.Duration
for i := range want {
if slept[i] != want[i] {
t.Fatalf("slept %v, want %v", slept, want)
}
total += slept[i]
}
if total+RunsPerMeasurement*TestTime > 60*time.Second {
t.Fatalf("a request holds the app %v + runs; it must end inside 60 s", total)
}
if v, _, ok := r.t.Value(); !ok || v != 92 {
t.Fatalf("the requested measurement was not stored: %.1f %v", v, ok)
}
if r.t.WaitingForQuiet() {
t.Fatal("measured this week: no longer waiting")
}
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-due" {
t.Fatalf("a second request this week: %v, want not-due", err)
}
}
func refusalCode(err error) string {
var ref *Refusal
if errors.As(err, &ref) {
return ref.Code
}
return ""
}
// The held app is not the only uploader: a request still refuses to measure a busy uplink.
func TestRunNowRefusesAnUplinkThatIsStillBusy(t *testing.T) {
r := seedingRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
r.t.uplink = func() (uint64, error) { r.tx += 20_000_000; return r.tx, nil } // 20 MB per 10 s = 16 Mbit/s
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-quiet" || r.run.calls != 0 {
t.Fatalf("RunNow on a busy uplink: %v, calls %d; want not-quiet and no run", err, r.run.calls)
}
}
// Every other rule of the weekly test holds for a request.
func TestRunNowKeepsTheWeeklyRules(t *testing.T) {
r := seedingRig(t)
r.gate.busy = "backup:felhom-pbs"
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "heavy-op" {
t.Fatalf("beside a backup: %v, want heavy-op", err)
}
r.gate.busy = ""
r.tunnel = false
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "no-tunnel" {
t.Fatalf("no tunnel: %v, want no-tunnel", err)
}
if r.t.WaitingForQuiet() {
t.Fatal("no tunnel: a request could not run, so it must not be invited")
}
r.tunnel = true
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "outside-window" {
t.Fatalf("at night: %v, want outside-window", err)
}
if r.run.calls != 0 {
t.Fatalf("%d runs on refused requests", r.run.calls)
}
}
// A failed request counts as a try, so a controller cannot request more than MaxTriesPerDay tests a day.
func TestRunNowFailuresSpendTheDaysTries(t *testing.T) {
var outs []runOut
for i := 0; i < MaxTriesPerDay*RunsPerMeasurement; i++ {
outs = append(outs, runOut{err: errors.New("connection refused")})
}
r := seedingRig(t, outs...)
for i := 0; i < MaxTriesPerDay; i++ {
if _, _, err := r.t.RunNow(context.Background()); err == nil || refusalCode(err) != "" {
t.Fatalf("try %d: %v, want a failed measurement", i+1, err)
}
}
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "tries-used" {
t.Fatalf("after %d failures: %v, want tries-used", MaxTriesPerDay, err)
}
}
// A quiet box tests itself on the weekly Tick: it must not invite the controller to hold its app.
func TestQuietBoxIsNotWaiting(t *testing.T) {
r := newRig(t)
if r.t.WaitingForQuiet() {
t.Fatal("ten quiet minutes: the weekly Tick runs; waiting_for_quiet must be false")
}
}
// Live proof 2026-10-24 (demo-hp): a freshly started tester answered waiting_for_quiet before it had watched a single
// minute, so a quiet box's torrent app would be held for nothing. Waiting needs the full window, seen busy.
func TestFreshTesterIsNotWaiting(t *testing.T) {
r := &rig{run: &fakeRunner{}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
r.t.loc = time.UTC
r.t.now = func() time.Time { return r.now }
if r.t.WaitingForQuiet() {
t.Fatal("no minute watched yet: waiting_for_quiet must be false")
}
for i := 0; i < QuietMinutes; i++ {
r.minute(600_000_000)
}
if r.t.WaitingForQuiet() {
t.Fatal("10 samples are 9 minutes: still not a full window")
}
r.minute(600_000_000)
if !r.t.WaitingForQuiet() {
t.Fatal("a full busy window: waiting_for_quiet must be true")
}
}
+17 -4
View File
@@ -15,7 +15,7 @@ import (
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release() // Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
// and the gate sub-case fails. // and the gate sub-case fails.
func TestAfterPrimaryBackup(t *testing.T) { func TestAfterPrimaryBackup(t *testing.T) {
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) { run := func(t *testing.T, failErr, path string) (calls []int, gateHeld bool) {
gate := &backup.InFlight{} gate := &backup.InFlight{}
b := &fakeBackups{failErr: failErr} b := &fakeBackups{failErr: failErr}
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil) srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
@@ -34,7 +34,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
done <- struct{}{} done <- struct{}{}
}) })
h := srv.Handler() h := srv.Handler()
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted { if do(t, h, "POST", path, "A", "").Code != http.StatusAccepted {
t.Fatal("POST /backup not accepted") t.Fatal("POST /backup not accepted")
} }
select { select {
@@ -47,7 +47,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
return calls, gateHeld return calls, gateHeld
} }
t.Run("success runs the leg under the gate", func(t *testing.T) { t.Run("success runs the leg under the gate", func(t *testing.T) {
calls, held := run(t, "") calls, held := run(t, "", "/backup")
if len(calls) != 1 { if len(calls) != 1 {
t.Fatalf("the leg ran %d time(s), want 1", len(calls)) t.Fatalf("the leg ran %d time(s), want 1", len(calls))
} }
@@ -56,8 +56,21 @@ func TestAfterPrimaryBackup(t *testing.T) {
} }
}) })
t.Run("a failed backup runs nothing", func(t *testing.T) { t.Run("a failed backup runs nothing", func(t *testing.T) {
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 { if calls, _ := run(t, "vzdump exploded", "/backup"); len(calls) != 0 {
t.Fatalf("the leg ran after a FAILED backup: %v", calls) t.Fatalf("the leg ran after a FAILED backup: %v", calls)
} }
}) })
// R-899: a household press is not the night's backup — no OS leg after it. Same fake, same successful backup as
// the first sub-case; only the query differs. Red-proof: make handleBackup ignore `trigger` and this sub-case
// fails with the leg run once.
t.Run("a manual press runs nothing", func(t *testing.T) {
if calls, _ := run(t, "", "/backup?trigger=manual"); len(calls) != 0 {
t.Fatalf("the OS leg ran after a manual press: %v", calls)
}
})
t.Run("the scheduled path with the new query still runs the leg", func(t *testing.T) {
if calls, _ := run(t, "", "/backup?trigger=night"); len(calls) != 1 {
t.Fatalf("the leg ran %d time(s) after a non-manual backup, want 1", len(calls))
}
})
} }
+146
View File
@@ -0,0 +1,146 @@
package localapi
import (
"fmt"
"net/http"
"sort"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// The whole-guest backup cadence cap (R-973, security review 2026-10-12).
//
// POST /backup is reachable from inside the guest, and every successful run prunes the tier
// (`--prune-backups keep-last=N` on a local tier; ep0's server-side keep-last 2 on the off-site tier). So
// a compromised controller calling it in a loop pushed every pre-compromise whole-guest restore point out
// within minutes — the threat `07` §8 row 10 counts the off-site tier as the answer to. The agent now
// refuses a run that the schedule in `07` §6 would never ask for:
//
// - a tier whose cadence is at most 24 h (the local tier): at most 3 successful runs in any rolling
// 24 h. The schedule is one night run; a „Mentés most" press makes a LOCAL copy and, since R-899, the
// night still takes its own after it, which can land inside the same 24 h as the previous night (the
// window opens anywhere in [W+2h, W+6h)). So a normal day needs at most 3: two nights plus one press.
// A second press inside 24 h is refused — the margin is spent on the night, which must never be refused
// because of the household's presses.
// - a longer tier (the 7-day off-site tier): at most 1 successful run per max(cadence − 24 h, cadence/2)
// — 6 days for 7 days. The controller asks /backup/due first and runs only when the newest copy is older
// than the cadence, and the button never targets this tier (`07` §6, controller v0.301.0), so a
// legitimate run never comes sooner; 24 h of slack absorbs clock and archive-time differences.
//
// Only SUCCESSES count — a failed run prunes nothing and must stay retryable. The count survives an agent
// restart: it reads the success history saved on disk (LastKnownBackupStore, the R-894 file) as well as
// this process's own record. A refusal answers 429, is logged at WARN, is saved, and raises the operator
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's operator-only
// `whole_guest_backup_refused` event since hub v0.159.0, R-1150). Pinned by TestBackupCap_* (backup_cap_test.go).
// backupCapRefusalHold is how long one refusal keeps the operator signal raised.
const backupCapRefusalHold = 24 * time.Hour
// backupCadenceCap returns the rolling window and the most successful runs allowed in it for a tier.
func backupCadenceCap(cadence time.Duration) (window time.Duration, max int) {
if cadence <= 0 {
cadence = defaultBackupCadence
}
if cadence <= 24*time.Hour {
return 24 * time.Hour, 3
}
w := cadence - 24*time.Hour
if w < cadence/2 {
w = cadence / 2
}
return w, 1
}
// recentCapRuns returns the start times of the successful runs on this tier for this guest inside the
// window ending now — this process's record and the saved one, each counted once. Caller holds no lock.
func (s *Server) recentCapRuns(key backupJobKey, since time.Time) []time.Time {
seen := map[int64]bool{}
var out []time.Time
add := func(t time.Time) {
if t.Before(since) || seen[t.Unix()] {
return
}
seen[t.Unix()] = true
out = append(out, t)
}
s.capMu.Lock()
for _, t := range s.capRuns[key] {
add(t)
}
s.capMu.Unlock()
if s.lastKnown != nil {
for _, t := range s.lastKnown.RecentBackupSuccesses(key.target, key.vmid, since) {
add(t)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Before(out[j]) })
return out
}
// recordCapRun notes a successful run for the cap (the saved half is written by RecordBackupSuccess).
func (s *Server) recordCapRun(key backupJobKey, b hub.Backup) {
t, err := time.Parse(time.RFC3339, b.StartedAt)
if err != nil {
t = s.now()
}
s.capMu.Lock()
defer s.capMu.Unlock()
if s.capRuns == nil {
s.capRuns = map[backupJobKey][]time.Time{}
}
runs := append(s.capRuns[key], t.UTC())
if len(runs) > 8 {
runs = runs[len(runs)-8:]
}
s.capRuns[key] = runs
}
// capRefuses applies the cap to a new run request. It writes the 429 and returns true when refused.
func (s *Server) capRefuses(w http.ResponseWriter, key backupJobKey, tier BackupTier) bool {
window, max := backupCadenceCap(tier.Cadence)
now := s.now()
runs := s.recentCapRuns(key, now.Add(-window))
if len(runs) < max {
return false
}
next := runs[len(runs)-max].Add(window)
s.capMu.Lock()
s.capRefusal = capRefusal{target: key.target, vmid: key.vmid, at: now.UTC()}
s.capMu.Unlock()
if s.lastKnown != nil {
if err := s.lastKnown.RecordBackupRefusal(key.target, key.vmid, now); err != nil {
s.logger.Warn("local-api: could not save the backup refusal on disk (R-973)", "vmid", key.vmid, "target", key.target, "err", err)
}
}
s.logger.Warn("local-api: backup REFUSED — more whole-guest backups than the schedule allows (R-973 cadence cap; check the guest)",
"vmid", key.vmid, "target", key.target, "runs_in_window", len(runs), "allowed", max,
"window_h", int(window.Hours()), "next_allowed", next.UTC().Format(time.RFC3339))
writeStatus(w, http.StatusTooManyRequests, false, nil, fmt.Sprintf(
"too many whole-guest backups on target %q: %d successful in the last %d h, the schedule allows %d — refused (the next one is allowed after %s)",
key.target, len(runs), int(window.Hours()), max, next.UTC().Format(time.RFC3339)))
return true
}
type capRefusal struct {
target string
vmid int
at time.Time
}
// BackupCapRefusal reports the newest cadence-cap refusal still inside the 24 h hold (this process's or
// the saved one) — the source of the operator signal in the host report (R-973). ok=false = none.
func (s *Server) BackupCapRefusal() (target string, vmid int, at time.Time, ok bool) {
s.capMu.Lock()
r := s.capRefusal
s.capMu.Unlock()
if s.lastKnown != nil {
if t, v, a, found := s.lastKnown.LastBackupRefusal(); found && a.After(r.at) {
r = capRefusal{target: t, vmid: v, at: a}
}
}
if r.at.IsZero() || s.now().Sub(r.at) > backupCapRefusalHold {
return "", 0, time.Time{}, false
}
return r.target, r.vmid, r.at, true
}
+377
View File
@@ -0,0 +1,377 @@
package localapi
import (
"context"
"fmt"
"io"
"log/slog"
"net/http"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-973 — the whole-guest backup cadence cap. A compromised controller calling POST /backup in a loop
// pushed out every pre-compromise restore point: each success prunes the tier. These tests model the tier
// the way Proxmox and ep0 keep it (keep-last N, pruned after each success) and ask what is LEFT.
// keepLastTier is a whole-guest tier: every successful run adds an archive stamped with the clock and
// prunes to keepLast, as vzdump's --prune-backups keep-last (local) and ep0's keep-last 2 (off-site) do.
type keepLastTier struct {
mu sync.Mutex
clock *testClock
keepLast int
// spec, when set, is the `--prune-backups` value the agent passes (R-1149) and the tier prunes the way Proxmox
// reads it: keep-last=N → the newest N; keep-daily=N → the newest copy of each of the N newest days with one.
spec string
t *testing.T
archives []time.Time
fail bool
runs int
}
// pruneBySpec applies a Proxmox `--prune-backups` spec to archives (oldest first) and returns what is kept.
func pruneBySpec(t *testing.T, archives []time.Time, spec string) []time.Time {
var n int
var mode string
if _, err := fmt.Sscanf(spec, "keep-last=%d", &n); err == nil {
mode = "last"
} else if _, err := fmt.Sscanf(spec, "keep-daily=%d", &n); err == nil {
mode = "daily"
} else {
t.Fatalf("the model does not know the prune spec %q", spec)
}
var kept []time.Time
days := map[string]bool{}
for i := len(archives) - 1; i >= 0; i-- { // newest first, as Proxmox walks them
a := archives[i]
switch mode {
case "last":
if len(kept) < n {
kept = append([]time.Time{a}, kept...)
}
case "daily":
d := a.Format("2006-01-02")
if !days[d] && len(days) < n {
days[d] = true
kept = append([]time.Time{a}, kept...)
}
}
}
return kept
}
func (k *keepLastTier) BackupWithSnapshotHook(_ context.Context, vmid int, _ func()) (hub.Backup, error) {
k.mu.Lock()
defer k.mu.Unlock()
k.runs++
now := k.clock.Now()
if k.fail {
return hub.Backup{VMID: vmid, StartedAt: now.Format(time.RFC3339)}, fmt.Errorf("vzdump failed")
}
k.archives = append(k.archives, now)
if k.spec != "" {
k.archives = pruneBySpec(k.t, k.archives, k.spec)
} else if len(k.archives) > k.keepLast {
k.archives = k.archives[len(k.archives)-k.keepLast:]
}
return hub.Backup{VMID: vmid, Success: true, Archive: "x", StartedAt: now.Format(time.RFC3339)}, nil
}
// oldestBefore reports whether the tier still holds an archive made before t (a pre-compromise point).
func (k *keepLastTier) oldestBefore(t time.Time) bool {
k.mu.Lock()
defer k.mu.Unlock()
return len(k.archives) > 0 && k.archives[0].Before(t)
}
func (k *keepLastTier) count() (runs, kept int) {
k.mu.Lock()
defer k.mu.Unlock()
return k.runs, len(k.archives)
}
type testClock struct {
mu sync.Mutex
t time.Time
}
func (c *testClock) Now() time.Time { c.mu.Lock(); defer c.mu.Unlock(); return c.t }
func (c *testClock) Advance(d time.Duration) { c.mu.Lock(); c.t = c.t.Add(d); c.mu.Unlock() }
// capServer builds the two-tier box of `07` §6: local 24 h keep-last 3, off-site 7 d keep-last 2, the
// success history on disk at path (a second capServer on the same path is an agent restart).
func capServer(t *testing.T, path string, clock *testClock, local, offsite *keepLastTier) *Server {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local"}, {Name: "felhom-pbs"}}},
Tokens: staticTokens{"A": 8200},
BackupTiers: []BackupTier{
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: local},
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: offsite},
},
LastKnownBackups: backup.NewBackupSuccessState(path),
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.now = clock.Now
return srv
}
// runBackup POSTs one backup and waits for the job to end. Returns the HTTP status of the POST.
func runBackup(t *testing.T, srv *Server, target string) int {
t.Helper()
rr := do(t, srv.Handler(), "POST", "/backup?target="+target, "A", "")
if rr.Code == http.StatusAccepted {
waitFor(t, func() bool {
srv.jobsMu.Lock()
defer srv.jobsMu.Unlock()
j := srv.jobs[backupJobKey{vmid: 8200, target: target}]
return j != nil && !backupInFlight(j.Phase)
})
}
return rr.Code
}
// The off-site tier — the one `07` §8 row 10 counts on against ransomware. Before the cap ten calls in a
// row left ep0 holding only the attacker's two copies. Now ONE runs (the tier was due), the other nine
// are refused, and the newest pre-compromise copy survives; five days of flooding later it still does.
//
// COMPANION RED-PROOF (observed 2026-10-18): make capRefuses always return false → this fails with
// "the off-site tier lost every pre-compromise copy: 10 runs" . Restored.
func TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy(t *testing.T) {
clock := &testClock{t: testNow}
local := &keepLastTier{clock: clock, keepLast: 3}
off := &keepLastTier{clock: clock, keepLast: 2}
path := filepath.Join(t.TempDir(), "backup-success-state.json")
srv := capServer(t, path, clock, local, off)
// The honest history: two weekly copies, the newest 7 days and 1 hour old (so the tier is due).
for _, age := range []time.Duration{14*24*time.Hour + time.Hour, 7*24*time.Hour + time.Hour} {
clock.t = testNow.Add(-age)
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusAccepted {
t.Fatalf("seeding the honest weekly copy: %d", code)
}
}
clock.t = testNow
compromise := testNow
refused := 0
for i := 0; i < 10; i++ {
if runBackup(t, srv, "felhom-pbs") == http.StatusTooManyRequests {
refused++
}
clock.Advance(time.Minute)
}
runs, _ := off.count()
if !off.oldestBefore(compromise) {
t.Fatalf("the off-site tier lost every pre-compromise copy: %d runs", runs-2)
}
if runs-2 != 1 || refused != 9 {
t.Fatalf("flood: %d runs, %d refused — want 1 run (the tier was due) and 9 refused", runs-2, refused)
}
for d := 0; d < 5; d++ { // five more days of flooding, every hour
for h := 0; h < 24; h++ {
clock.Advance(time.Hour)
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusTooManyRequests {
t.Fatalf("day %d hour %d: an off-site run was allowed %v after the last one", d, h, clock.Now().Sub(compromise))
}
}
}
if !off.oldestBefore(compromise) {
t.Fatal("five days of flooding pushed out the pre-compromise off-site copy")
}
}
// The local tier: a flood runs exactly 3 in a day (the schedule's own day: two nights + one press) and
// every further request is refused. Honest about the limit: with keep-last 3 those 3 replace the 3 kept
// points — the cap slows a flush from minutes to a day; it cannot keep a point a 3-per-day pace prunes.
func TestBackupCap_LocalFloodRunsAtMostThreeADay(t *testing.T) {
clock := &testClock{t: testNow}
local := &keepLastTier{clock: clock, keepLast: 3}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
refused := 0
for i := 0; i < 20; i++ {
if runBackup(t, srv, "local") == http.StatusTooManyRequests {
refused++
}
clock.Advance(10 * time.Minute)
}
if runs, _ := local.count(); runs != 3 || refused != 17 {
t.Fatalf("local flood over ~3 h: %d runs, %d refused — want 3 and 17", runs, refused)
}
clock.Advance(24 * time.Hour) // a day later the window has moved on
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("a run a day later must be allowed: %d", code)
}
}
// The schedule itself is never refused: a night at W+6h, a press at 10:00, the next night at W+2h (all
// three inside one rolling 24 h) run; a second press that day is refused; nights keep running after it.
func TestBackupCap_TheScheduledDayIsNeverRefused(t *testing.T) {
day := time.Date(2026, 6, 10, 0, 0, 0, 0, time.UTC)
clock := &testClock{t: day.Add(8*time.Hour + 29*time.Minute)} // W=02:30 → W+6h ≈ 08:29
local := &keepLastTier{clock: clock, keepLast: 3}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
steps := []struct {
at time.Time
want int
what string
}{
{day.Add(8*time.Hour + 29*time.Minute), http.StatusAccepted, "night 1 at W+6h"},
{day.Add(10 * time.Hour), http.StatusAccepted, "a press at 10:00"},
{day.Add(24*time.Hour + 4*time.Hour + 30*time.Minute), http.StatusAccepted, "night 2 at W+2h (20 h later)"},
{day.Add(24*time.Hour + 7*time.Hour), http.StatusTooManyRequests, "a second press inside the same 24 h"},
{day.Add(48*time.Hour + 4*time.Hour + 30*time.Minute), http.StatusAccepted, "night 3"},
}
for _, s := range steps {
clock.t = s.at
if got := runBackup(t, srv, "local"); got != s.want {
t.Fatalf("%s: got %d, want %d", s.what, got, s.want)
}
}
}
// A restart must not reset the cap: the history is read back from disk.
//
// COMPANION RED-PROOF (observed 2026-10-18): make recentCapRuns ignore s.lastKnown → this fails with
// "after a restart the 4th run in a day was allowed (got 202)". Restored.
func TestBackupCap_SurvivesARestart(t *testing.T) {
clock := &testClock{t: testNow}
path := filepath.Join(t.TempDir(), "s.json")
local := &keepLastTier{clock: clock, keepLast: 3}
srv := capServer(t, path, clock, local, &keepLastTier{clock: clock, keepLast: 2})
for i := 0; i < 3; i++ {
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("run %d: %d", i+1, code)
}
clock.Advance(time.Minute)
}
srv2 := capServer(t, path, clock, local, &keepLastTier{clock: clock, keepLast: 2}) // the restart
if code := runBackup(t, srv2, "local"); code != http.StatusTooManyRequests {
t.Fatalf("after a restart the 4th run in a day was allowed (got %d)", code)
}
}
// Failed runs prune nothing and must stay retryable: they do not count.
func TestBackupCap_FailuresDoNotCount(t *testing.T) {
clock := &testClock{t: testNow}
local := &keepLastTier{clock: clock, keepLast: 3, fail: true}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
for i := 0; i < 6; i++ {
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("failed attempt %d was refused: %d", i+1, code)
}
clock.Advance(15 * time.Minute)
}
local.mu.Lock()
local.fail = false
local.mu.Unlock()
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("a success after six failures was refused: %d", code)
}
}
// A refusal raises the operator signal for 24 h, and the signal survives a restart.
func TestBackupCap_RefusalRaisesTheOperatorSignal(t *testing.T) {
clock := &testClock{t: testNow}
path := filepath.Join(t.TempDir(), "s.json")
off := &keepLastTier{clock: clock, keepLast: 2}
srv := capServer(t, path, clock, &keepLastTier{clock: clock, keepLast: 3}, off)
if _, _, _, ok := srv.BackupCapRefusal(); ok {
t.Fatal("a fresh box must not raise the signal")
}
runBackup(t, srv, "felhom-pbs")
clock.Advance(time.Hour)
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusTooManyRequests {
t.Fatalf("second off-site run an hour later: %d, want 429", code)
}
target, vmid, at, ok := srv.BackupCapRefusal()
if !ok || target != "felhom-pbs" || vmid != 8200 || !at.Equal(clock.Now()) {
t.Fatalf("signal after a refusal: ok=%v target=%q vmid=%d at=%v", ok, target, vmid, at)
}
clock.Advance(23 * time.Hour)
srv2 := capServer(t, path, clock, &keepLastTier{clock: clock, keepLast: 3}, off) // a restart
if _, _, _, ok := srv2.BackupCapRefusal(); !ok {
t.Fatal("the signal did not survive a restart")
}
clock.Advance(2 * time.Hour)
if _, _, _, ok := srv2.BackupCapRefusal(); ok {
t.Fatal("the signal must clear 24 h after the last refusal")
}
}
func TestBackupCadenceCap(t *testing.T) {
cases := []struct {
cadence time.Duration
window time.Duration
max int
}{
{0, 24 * time.Hour, 3},
{24 * time.Hour, 24 * time.Hour, 3},
{12 * time.Hour, 24 * time.Hour, 3},
{7 * 24 * time.Hour, 6 * 24 * time.Hour, 1},
{36 * time.Hour, 18 * time.Hour, 1},
{30 * 24 * time.Hour, 29 * 24 * time.Hour, 1},
}
for _, c := range cases {
if w, m := backupCadenceCap(c.cadence); w != c.window || m != c.max {
t.Errorf("cadence %v: got %v/%d, want %v/%d", c.cadence, w, m, c.window, c.max)
}
}
}
// R-1149 — the consequence of keep-daily on the local tier, with the agent's REAL prune spec (config.BackupTiers →
// PruneSpec, what runDaemon hands the runner). Three honest nights, then a compromise at 01:00 that floods
// POST /backup for the rest of the day. The cap lets 3 runs through inside the rolling 24 h (two at once, one when
// last night's success ages out). With keep-last 3 those three were the only copies left; with keep-daily 3 the two
// earlier nights survive the whole day.
//
// COMPANION RED-PROOF (observed 2026-10-19): make BackupTier.PruneSpec return keep-last for the primary → this fails
// with "after a day of flooding the local tier holds 0 copies made before the compromise (kept […]) — want the 2
// earlier nights". Restored.
func TestBackupCap_R1149_LocalFloodKeepsOlderDays(t *testing.T) {
tiers, _ := config.BackupConfig{LocalBackupTarget: "local", LocalBackupRetention: 3}.BackupTiers()
spec := tiers[0].PruneSpec()
day := time.Date(2026, 6, 10, 0, 0, 0, 0, time.UTC)
clock := &testClock{t: day}
local := &keepLastTier{clock: clock, keepLast: 3, spec: spec, t: t}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
for d := 3; d >= 1; d-- { // the nights of D-3, D-2, D-1 at 03:00
clock.t = day.Add(-time.Duration(d)*24*time.Hour + 3*time.Hour)
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("seeding night D-%d: %d", d, code)
}
}
compromise := day.Add(time.Hour)
ran := 0
for clock.t = compromise; clock.t.Before(day.Add(24 * time.Hour)); clock.Advance(10 * time.Minute) {
if runBackup(t, srv, "local") == http.StatusAccepted {
ran++
}
}
if ran != 3 {
t.Fatalf("the flood ran %d backups on day D — want 3 (the cap)", ran)
}
local.mu.Lock()
kept := append([]time.Time(nil), local.archives...)
local.mu.Unlock()
pre := 0
for _, a := range kept {
if a.Before(compromise) {
pre++
}
}
if pre != 2 {
t.Fatalf("after a day of flooding the local tier holds %d copies made before the compromise (kept %v) — want the 2 earlier nights", pre, kept)
}
}
@@ -4,7 +4,6 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"io"
"log/slog" "log/slog"
"os" "os"
"path/filepath" "path/filepath"
@@ -54,8 +53,8 @@ func (f *supExec) GuestExec(_ context.Context, vmid int, args ...string) (string
} }
return "", errors.New("supExec: unexpected args") return "", errors.New("supExec: unexpected args")
} }
func (f *supExec) GuestExecStdin(context.Context, int, io.Reader, ...string) (string, error) { func (f *supExec) WriteControllerImage(context.Context, int, string) error {
return "", errors.New("supExec: no stdin exec expected") return errors.New("supExec: no image write expected")
} }
func (f *supExec) count(vmid int) int { func (f *supExec) count(vmid int) int {
f.mu.Lock() f.mu.Lock()
@@ -250,7 +249,6 @@ func TestControllerSupervisorStanza_WireShape(t *testing.T) {
} }
} }
// ---- R-539 (operator ruling 3 of 2026-09-16): the SLOW crash loop ---------------------------------- // ---- R-539 (operator ruling 3 of 2026-09-16): the SLOW crash loop ----------------------------------
// supKillOnce kills the controller and lets the supervisor restart it (two confirming sweeps), then // supKillOnce kills the controller and lets the supervisor restart it (two confirming sweeps), then
+9 -12
View File
@@ -4,7 +4,6 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"io"
"log/slog" "log/slog"
"net/http" "net/http"
"os" "os"
@@ -41,9 +40,10 @@ func ValidControllerImage(ref string) bool { return controllerImageRe.MatchStrin
// faked in tests. The single seam the swap composes over (no hand-rolled pct). // faked in tests. The single seam the swap composes over (no hand-rolled pct).
type GuestExecutor interface { type GuestExecutor interface {
GuestExec(ctx context.Context, vmid int, args ...string) (string, error) GuestExec(ctx context.Context, vmid int, args ...string) (string, error)
// GuestExecStdin is GuestExec with the command's stdin fed from stdin — the swap write pipes the // WriteControllerImage writes the image ref into the guest's /etc/felhom-controller-image through the ROOT
// image ref into an in-guest `tee` (no shell vector). // verb `felhom-priv-apply controller-image <vmid>` (R-861 (a) A1, `09` §3 decision 165), which re-checks the
GuestExecStdin(ctx context.Context, vmid int, stdin io.Reader, args ...string) (string, error) // ref against our registry + repository + x.y.z. The agent no longer holds a `tee` grant into the guest.
WriteControllerImage(ctx context.Context, vmid int, image string) error
} }
// ControllerSwapState is the durable record of a swap (crash-safety + status). Written before the swap // ControllerSwapState is the durable record of a swap (crash-safety + status). Written before the swap
@@ -141,14 +141,11 @@ func (c *ControllerSwapper) imagePresent(ctx context.Context, vmid int, image st
} }
func (c *ControllerSwapper) writeImage(ctx context.Context, vmid int, image string) error { func (c *ControllerSwapper) writeImage(ctx context.Context, vmid int, image string) error {
// Non-root path: pipe the image ref into an in-guest `tee` over stdin — no shell, no // R-861 (a) A1: the ROOT verb writes the file (it re-checks the ref — a compromised agent cannot hand the guest's
// interpolation, no `bash -c` (the only swap vector that would have needed an arbitrary-exec // bootstrap another image). The bytes are `image\n`, byte-identical to the golden's `printf '%s\n'`; the bootstrap
// grant). The trailing "\n" makes the on-disk bytes byte-identical to the golden's // reads `IMAGE=$(cat …)` so the newline is stripped on read. image is also strict-validated (controllerImageRe)
// `printf '%s\n'`; the bootstrap reads `IMAGE=$(cat …)` so the newline is stripped on read // upstream in Swap.
// (spike SPIKE-controllerswap-narrow-grants-2026-06-29). image is strict-validated return c.exec.WriteControllerImage(ctx, vmid, image)
// (controllerImageRe) upstream in Swap; defence-in-depth, the stdin path can't smuggle anyway.
_, err := c.exec.GuestExecStdin(ctx, vmid, strings.NewReader(image+"\n"), "tee", controllerImageFile)
return err
} }
func (c *ControllerSwapper) restartBootstrap(ctx context.Context, vmid int) error { func (c *ControllerSwapper) restartBootstrap(ctx context.Context, vmid int) error {
+14 -28
View File
@@ -23,7 +23,7 @@ type fakeGuestExec struct {
present map[string]bool // images pulled into the guest present map[string]bool // images pulled into the guest
good map[string]bool // images that report healthy when running good map[string]bool // images that report healthy when running
containerImg string // image the running container currently has containerImg string // image the running container currently has
teeStdin []string // raw bytes piped into each `tee` write (the swap's write vector) teeStdin []string // image refs handed to WriteControllerImage (the root verb, R-861 (a) A1)
failRestart bool failRestart bool
noHealthBlock bool // if set, .State.Health is absent ("none") noHealthBlock bool // if set, .State.Health is absent ("none")
restartCount int // F1: .RestartCount reported by docker inspect (a crash-looper has >0) restartCount int // F1: .RestartCount reported by docker inspect (a crash-looper has >0)
@@ -75,20 +75,15 @@ func (f *fakeGuestExec) GuestExec(_ context.Context, _ int, args ...string) (str
return "", fmt.Errorf("fake: unexpected exec %v", args) return "", fmt.Errorf("fake: unexpected exec %v", args)
} }
// GuestExecStdin models the swap's write vector: `tee /etc/felhom-controller-image` with the image // WriteControllerImage models the swap's write: the root verb `felhom-priv-apply controller-image <vmid>` (R-861
// piped on stdin. It records the raw stdin bytes and sets the modeled file content (newline-stripped, // (a) A1). It records the ref and sets the modeled file content, as the bootstrap's `IMAGE=$(cat …)` would read it.
// as the bootstrap's `IMAGE=$(cat …)` read would see it). func (f *fakeGuestExec) WriteControllerImage(_ context.Context, _ int, image string) error {
func (f *fakeGuestExec) GuestExecStdin(_ context.Context, _ int, stdin io.Reader, args ...string) (string, error) {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
f.calls = append(f.calls, args) f.calls = append(f.calls, []string{"felhom-priv-apply", "controller-image", image})
b, _ := io.ReadAll(stdin) f.teeStdin = append(f.teeStdin, image+"\n")
if len(args) >= 2 && args[0] == "tee" && args[1] == controllerImageFile { f.imageFile = image
f.teeStdin = append(f.teeStdin, string(b)) return nil
f.imageFile = strings.TrimSpace(string(b))
return string(b), nil // tee echoes stdin to stdout
}
return "", fmt.Errorf("fake: unexpected exec-stdin args=%v stdin=%q", args, string(b))
} }
// wrote reports whether the image was written via the stdin `tee` vector with the exact `image\n` // wrote reports whether the image was written via the stdin `tee` vector with the exact `image\n`
@@ -162,7 +157,7 @@ func TestControllerSwap_Happy(t *testing.T) {
// The write vector must be the stdin `tee` with byte-identical `image\n` and NO shell — the // The write vector must be the stdin `tee` with byte-identical `image\n` and NO shell — the
// controllerswap.go writeImage rewrite. This would FAIL on the pre-change `bash -c "printf … >"` impl. // controllerswap.go writeImage rewrite. This would FAIL on the pre-change `bash -c "printf … >"` impl.
func TestControllerSwap_WriteViaStdinTee_NoShell(t *testing.T) { func TestControllerSwap_WriteViaRootVerb_NoShell(t *testing.T) {
fe := &fakeGuestExec{ fe := &fakeGuestExec{
imageFile: prevImg, imageFile: prevImg,
present: map[string]bool{newImg: true}, present: map[string]bool{newImg: true},
@@ -173,22 +168,15 @@ func TestControllerSwap_WriteViaStdinTee_NoShell(t *testing.T) {
t.Fatalf("state = %q, want done", st.State) t.Fatalf("state = %q, want done", st.State)
} }
if !fe.wrote(newImg) { if !fe.wrote(newImg) {
t.Errorf("expected a tee write of %q+\\n; teeStdin=%q", newImg, fe.teeStdin) t.Errorf("expected the root verb to write %q; writes=%q", newImg, fe.teeStdin)
} }
sawTee := false
for _, c := range fe.calls { for _, c := range fe.calls {
if len(c) >= 2 && c[0] == "tee" { if len(c) >= 1 && c[0] == "tee" {
sawTee = true t.Errorf("the swap still uses an in-guest tee (R-861 (a) A1 removed that grant): %v", c)
if c[1] != controllerImageFile {
t.Errorf("tee target = %q, want fixed %q", c[1], controllerImageFile)
} }
} }
}
if !sawTee {
t.Error("no tee call recorded — writeImage did not use the stdin tee vector")
}
if fe.usedShell() { if fe.usedShell() {
t.Errorf("swap used a shell vector (bash/-c/printf) — must be stdin tee only; calls=%v", fe.calls) t.Errorf("swap used a shell vector (bash/-c/printf); calls=%v", fe.calls)
} }
} }
@@ -300,9 +288,7 @@ func (s *inspectScript) GuestExec(_ context.Context, _ int, args ...string) (str
} }
return "", nil return "", nil
} }
func (s *inspectScript) GuestExecStdin(_ context.Context, _ int, _ io.Reader, _ ...string) (string, error) { func (s *inspectScript) WriteControllerImage(context.Context, int, string) error { return nil }
return "", nil
}
func fastSwapper(exec GuestExecutor) *ControllerSwapper { func fastSwapper(exec GuestExecutor) *ControllerSwapper {
s := NewControllerSwapper(exec, "", discardLogger()) s := NewControllerSwapper(exec, "", discardLogger())
+13
View File
@@ -188,6 +188,13 @@ type DiskInfo struct {
// device that exposes no SMART (USB bridge, unread) is ABSENT, not a misleading zero-value UNKNOWN; // device that exposes no SMART (USB bridge, unread) is ABSENT, not a misleading zero-value UNKNOWN;
// the controller feature-detects presence and renders "Nincs adat" when nil (never alarms). // the controller feature-detects presence and renders "Nincs adat" when nil (never alarms).
Smart *hub.SmartSummary `json:"smart,omitempty"` Smart *hub.SmartSummary `json:"smart,omitempty"`
// Physical-disk identity (v0.157.0, felhom.eu architecture/16-storage-physical-identity.md §3) — the
// same agent-computed verdict the hub report carries. Empty when not computed (an absent drive, no
// resolver). Additive; an older controller ignores it.
Kind string `json:"kind,omitempty"`
PhysicalDisks []string `json:"physical_disks,omitempty"`
Identity string `json:"identity,omitempty"`
IdentityReason string `json:"identity_reason,omitempty"`
} }
// handleDisks lists the host's drives + data-bearing flags (read-only/benign). // handleDisks lists the host's drives + data-bearing flags (read-only/benign).
@@ -227,6 +234,8 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
// really is the target. Reported here so the controller can name the drive in an // really is the target. Reported here so the controller can name the drive in an
// absent-target alarm and hide the destructive controls the agent would refuse anyway. // absent-target alarm and hide the destructive controls the agent would refuse anyway.
BackupTarget: t.Name == primaryTargetID, BackupTarget: t.Name == primaryTargetID,
// `16` §3: copied from the observer, which computed it from /sys + udev.
Kind: t.Kind, PhysicalDisks: t.PhysicalDisks, Identity: t.Identity, IdentityReason: t.IdentityReason,
} }
// Intermediary model: the stable in-guest path + whether felhom-data is bound under the parent. // Intermediary model: the stable in-guest path + whether felhom-data is bound under the parent.
// Only user-data /mnt/<name> drives have a guest path (system/backup mounts never cross in). // Only user-data /mnt/<name> drives have a guest path (system/backup mounts never cross in).
@@ -398,6 +407,10 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
di.Smart = &sm di.Smart = &sm
} }
} }
// `16` §3: a registry drive's identity, judged on its OWN mount only (never root's).
if p, kind, ok := s.ownMountIdentity(d.MountPath); ok {
di.Kind, di.PhysicalDisks, di.Identity, di.IdentityReason = kind, p.PhysicalDisks, p.Identity, p.Reason
}
// R-118: statfs ONLY while the drive's device is present. With the device gone the raw // R-118: statfs ONLY while the drive's device is present. With the device gone the raw
// mountpoint reverts to a bare directory on the ROOT filesystem, and statfs would report // mountpoint reverts to a bare directory on the ROOT filesystem, and statfs would report
// pve-root's size as this drive's (measured: a 4 GB drive advertising 46 GiB). Same trap // pve-root's size as this drive's (measured: a 4 GB drive advertising 46 GiB). Same trap
+312 -13
View File
@@ -3,11 +3,15 @@ package localapi
import ( import (
"bytes" "bytes"
"context" "context"
"crypto/rand"
"crypto/subtle"
"encoding/json" "encoding/json"
"fmt" "fmt"
"math/big"
"net/http" "net/http"
"os/exec" "os/exec"
"strconv" "strconv"
"strings"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow" "gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
@@ -24,9 +28,23 @@ import (
// be structurally incapable of carrying R). // be structurally incapable of carrying R).
// - One claim, then the holder is zeroed. Unclaimed past the TTL → zeroed + phase // - One claim, then the holder is zeroed. Unclaimed past the TTL → zeroed + phase
// unclaimed_void ("R unclaimed → ceremony void → a re-run supersedes"). // unclaimed_void ("R unclaimed → ceremony void → a re-run supersedes").
// - Crash-safety is IN-MEMORY BY DESIGN: an agent restart loses R, which is SAFE (the blob is // - Crash-safety is IN-MEMORY BY DESIGN: an agent restart loses R AND the package it opens,
// on the hub; a re-run supersedes it). Status reporting "none" after a restart is the honest // which is SAFE (R-978: the package was never uploaded, so the hub still holds the previous one
// answer. No journal, deliberately — persistence is the one property R must never have. // and the customer's previous code still opens it; a re-run starts over). Status reporting "none"
// after a restart is the honest answer. No journal, deliberately — persistence is the one
// property R must never have.
// - R-978: the NEW package reaches the hub only at the claim, and the code is handed out only after
// the hub took it. Until then the previous package stays current — so a ceremony that is started
// and abandoned (never claimed, superseded, TTL, restart) leaves the customer's written-down code
// working. The TTL and a new start both wait while a claim's upload is in flight, so a package is
// never stored without its code being handed out by that same claim.
// - R-1151: a claim that asks for it (`{"defer_upload": true}`, controller ≥ 0.332.0) hands out the code
// and two word positions, and keeps the package until POST /escrow/ceremony/confirm brings those two
// words back — the household's typed-back check, now checked HERE, not only in the browser. A code that
// was shown and never typed back (the page closed) never replaces the hub's package; the confirm window
// (escrowConfirmTTL) ends it. Only the two words are kept for the check, never R. A claim WITHOUT the
// flag (a controller that has not updated) keeps the R-978 order: upload at the claim. Pinned by
// TestEscrowCeremony_R1151_*.
// - Nothing from stdout is ever logged; stderr (log-clean, spike §2.1) is tail-captured for // - Nothing from stdout is ever logged; stderr (log-clean, spike §2.1) is tail-captured for
// failure diagnostics only. // failure diagnostics only.
@@ -50,8 +68,16 @@ type EscrowCeremonyConfig struct {
// DRConfigured answers "is the DR tier applied on this box?" (pbsdr.Manager.DRConfigured, // DRConfigured answers "is the DR tier applied on this box?" (pbsdr.Manager.DRConfigured,
// late-bound). nil → reported not-applied. // late-bound). nil → reported not-applied.
DRConfigured func() bool DRConfigured func() bool
// Upload sends a ceremony's package (CeremonyOutput.PendingUpload, the hub upload body) to this
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
Upload func(ctx context.Context, pkg []byte) error
} }
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
// ceremony incl. its upload was measured at ~2.4 s).
const escrowUploadTimeout = 30 * time.Second
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until // ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned // parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
// spike-shaped output; production is runCeremonySubprocess. // spike-shaped output; production is runCeremonySubprocess.
@@ -78,8 +104,15 @@ const (
escrowPhaseDone = "done" escrowPhaseDone = "done"
escrowPhaseFailed = "failed" escrowPhaseFailed = "failed"
escrowPhaseVoid = "unclaimed_void" escrowPhaseVoid = "unclaimed_void"
// escrowPhaseUnconfirmed (R-1151): the code was shown but its words never came back inside the confirm
// window — the package was dropped, the hub's previous package (and the previous code) stays current.
escrowPhaseUnconfirmed = "unconfirmed_void"
) )
// escrowConfirmTTL is how long a claimed code waits for its typed-back words (R-1151). Writing ten words down
// by hand takes minutes; past this the package is dropped and the household starts again.
const escrowConfirmTTL = 30 * time.Minute
// escrowCeremonyTimeout bounds the whole subprocess run. Spike-measured ceremony ≈ 2.4 s incl. // escrowCeremonyTimeout bounds the whole subprocess run. Spike-measured ceremony ≈ 2.4 s incl.
// upload — 60 s is a ≥25× margin that still absorbs WAN upload latency. // upload — 60 s is a ≥25× margin that still absorbs WAN upload latency.
const escrowCeremonyTimeout = 60 * time.Second const escrowCeremonyTimeout = 60 * time.Second
@@ -129,6 +162,9 @@ func (s *Server) tryStartEscrowCeremony(job *escrowCeremonyJob) bool {
if s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning { if s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning {
return false return false
} }
if s.escrowClaiming {
return false // R-978: a claim is uploading this job's package — its code must reach the claimer
}
s.wipeEscrowRLocked() s.wipeEscrowRLocked()
cp := *job cp := *job
s.escrowJob = &cp s.escrowJob = &cp
@@ -137,7 +173,7 @@ func (s *Server) tryStartEscrowCeremony(job *escrowCeremonyJob) bool {
// finishEscrowCeremony records the terminal phase and (on success) arms the one-shot R holder. // finishEscrowCeremony records the terminal phase and (on success) arms the one-shot R holder.
// R is stored as a byte slice so the claim/TTL paths can zero it in place. // R is stored as a byte slice so the claim/TTL paths can zero it in place.
func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string) { func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string, pending []byte) {
s.escrowMu.Lock() s.escrowMu.Lock()
defer s.escrowMu.Unlock() defer s.escrowMu.Unlock()
job.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339) job.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
@@ -145,6 +181,7 @@ func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string) {
s.escrowJob = &cp s.escrowJob = &cp
if job.Phase == escrowPhaseDone && r != "" { if job.Phase == escrowPhaseDone && r != "" {
s.escrowR = []byte(r) s.escrowR = []byte(r)
s.escrowPending = append([]byte(nil), pending...)
s.escrowRClaimed = false s.escrowRClaimed = false
s.escrowRExpiry = s.nowFn().Add(escrowClaimTTL) s.escrowRExpiry = s.nowFn().Add(escrowClaimTTL)
// Active TTL belt: zero the holder even if nobody ever polls again. The lazy check in // Active TTL belt: zero the holder even if nobody ever polls again. The lazy check in
@@ -164,6 +201,53 @@ func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string) {
func (s *Server) wipeEscrowRLocked() { func (s *Server) wipeEscrowRLocked() {
zeroBytes(s.escrowR) zeroBytes(s.escrowR)
s.escrowR = nil s.escrowR = nil
s.escrowPending = nil // R-978: the package goes with its code — never uploaded once the code is gone
s.wipeEscrowConfirmLocked()
}
// wipeEscrowConfirmLocked drops the R-1151 typed-back state: the two kept words and the wait. Caller holds escrowMu.
func (s *Server) wipeEscrowConfirmLocked() {
for i := range s.escrowConfirmWords {
zeroBytes(s.escrowConfirmWords[i])
s.escrowConfirmWords[i] = nil
}
s.escrowAwaitConfirm = false
}
// checkEscrowConfirmTTLLocked ends a typed-back wait that ran past escrowConfirmTTL (R-1151): the package is
// dropped (never uploaded) and the job reads unconfirmed_void. Not while a confirm is uploading. Caller holds escrowMu.
func (s *Server) checkEscrowConfirmTTLLocked() {
if !s.escrowAwaitConfirm || s.escrowClaiming || !s.nowFn().After(s.escrowConfirmExpiry) {
return
}
s.escrowPending = nil
s.wipeEscrowConfirmLocked()
if s.escrowJob != nil {
s.escrowJob.Phase = escrowPhaseUnconfirmed
s.escrowJob.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
s.logger.Info("local-api: escrow code was not typed back in time — the new package was dropped; the previous one stays current (R-1151)",
"job_id", s.escrowJob.JobID)
}
}
// pickConfirmPositions draws two distinct 1-based word positions out of n (n ≥ 2), smaller first.
func pickConfirmPositions(n int) ([2]int, error) {
a, err := rand.Int(rand.Reader, big.NewInt(int64(n)))
if err != nil {
return [2]int{}, err
}
b, err := rand.Int(rand.Reader, big.NewInt(int64(n-1)))
if err != nil {
return [2]int{}, err
}
i, j := int(a.Int64()), int(b.Int64())
if j >= i {
j++
}
if j < i {
i, j = j, i
}
return [2]int{i + 1, j + 1}, nil
} }
// expireEscrowRLocked applies the TTL outcome: an armed, unclaimed holder is zeroed and the job // expireEscrowRLocked applies the TTL outcome: an armed, unclaimed holder is zeroed and the job
@@ -172,6 +256,9 @@ func (s *Server) expireEscrowRLocked() {
if s.escrowJob == nil || s.escrowJob.Phase != escrowPhaseDone || s.escrowRClaimed { if s.escrowJob == nil || s.escrowJob.Phase != escrowPhaseDone || s.escrowRClaimed {
return return
} }
if s.escrowClaiming {
return // R-978: the claim is uploading the package; voiding now would store a package nobody gets the code to
}
if len(s.escrowR) == 0 { if len(s.escrowR) == 0 {
return return
} }
@@ -217,7 +304,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
UpdatedAt: s.nowFn().UTC().Format(time.RFC3339), UpdatedAt: s.nowFn().UTC().Format(time.RFC3339),
} }
if !s.tryStartEscrowCeremony(job) { if !s.tryStartEscrowCeremony(job) {
writeErr(w, http.StatusConflict, "an escrow ceremony is already running") writeErr(w, http.StatusConflict, "an escrow ceremony is already running, or its code is being claimed")
return return
} }
s.logger.Info("local-api: escrow ceremony started (controller-driven)", "vmid", vmid, "job_id", job.JobID) s.logger.Info("local-api: escrow ceremony started (controller-driven)", "vmid", vmid, "job_id", job.JobID)
@@ -226,6 +313,9 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
if base == nil { if base == nil {
base = context.Background() base = context.Background()
} }
// The reply's fields are read HERE, before the job goroutine may write job.Phase (a data race the race
// detector caught, 2026-10-13 — the reply could show the finished phase or a torn value).
jobID, phase := job.JobID, job.Phase
done := make(chan struct{}) done := make(chan struct{})
go func() { go func() {
defer close(done) defer close(done)
@@ -244,7 +334,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
detail += " | stderr: " + tail detail += " | stderr: " + tail
} }
job.Phase, job.Detail = escrowPhaseFailed, detail job.Phase, job.Detail = escrowPhaseFailed, detail
s.finishEscrowCeremony(job, "") s.finishEscrowCeremony(job, "", nil)
s.logger.Warn("local-api: escrow ceremony failed", "job_id", job.JobID, s.logger.Warn("local-api: escrow ceremony failed", "job_id", job.JobID,
"exit", exit, "duration_ms", time.Since(start).Milliseconds()) "exit", exit, "duration_ms", time.Since(start).Milliseconds())
return return
@@ -252,29 +342,39 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
var out escrow.CeremonyOutput var out escrow.CeremonyOutput
if err := json.Unmarshal(stdout, &out); err != nil { if err := json.Unmarshal(stdout, &out); err != nil {
job.Phase, job.Detail = escrowPhaseFailed, "ceremony output is not the expected JSON object" job.Phase, job.Detail = escrowPhaseFailed, "ceremony output is not the expected JSON object"
s.finishEscrowCeremony(job, "") s.finishEscrowCeremony(job, "", nil)
s.logger.Warn("local-api: escrow ceremony output unparseable (never logged)", "job_id", job.JobID) s.logger.Warn("local-api: escrow ceremony output unparseable (never logged)", "job_id", job.JobID)
return return
} }
if out.Version != escrow.CeremonyOutputVersion || out.RecoveryCode == "" { if out.Version != escrow.CeremonyOutputVersion || out.RecoveryCode == "" {
out.RecoveryCode = "" out.RecoveryCode = ""
job.Phase, job.Detail = escrowPhaseFailed, fmt.Sprintf("unexpected ceremony output (version %d)", out.Version) job.Phase, job.Detail = escrowPhaseFailed, fmt.Sprintf("unexpected ceremony output (version %d)", out.Version)
s.finishEscrowCeremony(job, "") s.finishEscrowCeremony(job, "", nil)
return
}
// R-978: the package must come back un-uploaded. An output that uploaded by itself (or carries no
// package) would mean the previous package is already gone, or the new one can never be stored.
if out.Uploaded || len(out.PendingUpload) == 0 {
out.RecoveryCode = ""
job.Phase, job.Detail = escrowPhaseFailed, "ceremony output carries no pending package"
s.finishEscrowCeremony(job, "", nil)
s.logger.Warn("local-api: escrow ceremony output has no pending package — failed (R-978)", "job_id", job.JobID,
"uploaded", out.Uploaded)
return return
} }
job.Phase = escrowPhaseDone job.Phase = escrowPhaseDone
job.KeyFingerprint = out.KeyFingerprint job.KeyFingerprint = out.KeyFingerprint
job.EntropyBits = out.EntropyBits job.EntropyBits = out.EntropyBits
job.ResticPwSealed = out.ResticPwSealed job.ResticPwSealed = out.ResticPwSealed
job.Uploaded = out.Uploaded job.Uploaded = false // R-978: uploaded by the claim
s.finishEscrowCeremony(job, out.RecoveryCode) s.finishEscrowCeremony(job, out.RecoveryCode, out.PendingUpload)
out.RecoveryCode = "" // drop the parsed reference promptly (GC caveat: best-effort) out.RecoveryCode = "" // drop the parsed reference promptly (GC caveat: best-effort)
s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot)", s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim, or at the typed-back confirm (R-1151)",
"job_id", job.JobID, "restic_pw_sealed", job.ResticPwSealed, "uploaded", job.Uploaded, "job_id", job.JobID, "restic_pw_sealed", job.ResticPwSealed, "uploaded", job.Uploaded,
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds()) "claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
}() }()
s.escrowDone = done // tests wait on it; production polls the status endpoint s.escrowDone = done // tests wait on it; production polls the status endpoint
writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": job.JobID, "phase": job.Phase}, "") writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": jobID, "phase": phase}, "")
} }
// handleEscrowCeremonyStatus is GET /escrow/ceremony/status: phase + the NON-SECRET summary. // handleEscrowCeremonyStatus is GET /escrow/ceremony/status: phase + the NON-SECRET summary.
@@ -287,6 +387,7 @@ func (s *Server) handleEscrowCeremonyStatus(w http.ResponseWriter, r *http.Reque
} }
s.escrowMu.Lock() s.escrowMu.Lock()
s.checkEscrowTTLLocked() s.checkEscrowTTLLocked()
s.checkEscrowConfirmTTLLocked()
if s.escrowJob == nil { if s.escrowJob == nil {
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeOK(w, map[string]any{"vmid": vmid, "phase": escrowPhaseNone}) writeOK(w, map[string]any{"vmid": vmid, "phase": escrowPhaseNone})
@@ -301,6 +402,13 @@ func (s *Server) handleEscrowCeremonyStatus(w http.ResponseWriter, r *http.Reque
} }
} }
claimed := s.escrowRClaimed claimed := s.escrowRClaimed
awaiting := s.escrowAwaitConfirm
confirmIn := 0
if awaiting {
if d := s.escrowConfirmExpiry.Sub(s.nowFn()); d > 0 {
confirmIn = int(d.Seconds())
}
}
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeOK(w, map[string]any{ writeOK(w, map[string]any{
@@ -309,6 +417,7 @@ func (s *Server) handleEscrowCeremonyStatus(w http.ResponseWriter, r *http.Reque
"key_fingerprint": job.KeyFingerprint, "entropy_bits": job.EntropyBits, "key_fingerprint": job.KeyFingerprint, "entropy_bits": job.EntropyBits,
"restic_pw_sealed": job.ResticPwSealed, "uploaded": job.Uploaded, "restic_pw_sealed": job.ResticPwSealed, "uploaded": job.Uploaded,
"claimable": claimable, "claimed": claimed, "claim_expires_in_sec": expiresIn, "claimable": claimable, "claimed": claimed, "claim_expires_in_sec": expiresIn,
"awaiting_confirm": awaiting, "confirm_expires_in_sec": confirmIn,
"detail": job.Detail, "detail": job.Detail,
}) })
} }
@@ -321,6 +430,14 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
writeErr(w, http.StatusServiceUnavailable, "escrow ceremony not configured on this host") writeErr(w, http.StatusServiceUnavailable, "escrow ceremony not configured on this host")
return return
} }
// R-1151: a controller ≥ 0.332.0 asks for the package to wait for the typed-back words. An older controller
// sends `{}` (or nothing) and gets the R-978 order. Unknown fields are refused (decodeBody).
var req struct {
DeferUpload bool `json:"defer_upload"`
}
if !decodeBody(w, r, &req) {
return
}
s.escrowMu.Lock() s.escrowMu.Lock()
s.checkEscrowTTLLocked() s.checkEscrowTTLLocked()
switch { switch {
@@ -336,11 +453,101 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeErr(w, http.StatusConflict, "ceremony failed — nothing to claim") writeErr(w, http.StatusConflict, "ceremony failed — nothing to claim")
return return
case s.escrowClaiming:
s.escrowMu.Unlock()
writeErr(w, http.StatusConflict, "the recovery code is being claimed — try again in a moment")
return
case s.escrowRClaimed || len(s.escrowR) == 0 || s.escrowJob.Phase == escrowPhaseVoid: case s.escrowRClaimed || len(s.escrowR) == 0 || s.escrowJob.Phase == escrowPhaseVoid:
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeErr(w, http.StatusGone, "the recovery code is no longer available (already claimed or expired) — run a new ceremony; the new code supersedes") writeErr(w, http.StatusGone, "the recovery code is no longer available (already claimed or expired) — run a new ceremony; the new code supersedes")
return return
} }
// R-1151: the deferred order — hand out the code and two word positions; the package waits for
// POST /escrow/ceremony/confirm. Needs an uploader (a confirm could never store it otherwise) and a code of ≥ 2
// words (always 10; anything else falls back to the R-978 order rather than to no check).
if req.DeferUpload && len(s.escrowPending) > 0 && s.escrowCeremony.Upload != nil {
words := strings.Split(string(s.escrowR), escrow.RecoveryCodeSep)
if len(words) >= 2 {
pos, perr := pickConfirmPositions(len(words))
if perr == nil {
s.escrowConfirmPos = pos
s.escrowConfirmWords = [2][]byte{[]byte(words[pos[0]-1]), []byte(words[pos[1]-1])}
for i := range words {
words[i] = ""
}
s.escrowAwaitConfirm = true
s.escrowConfirmExpiry = s.nowFn().Add(escrowConfirmTTL)
recovery := string(s.escrowR)
zeroBytes(s.escrowR) // R only — the package and the two words stay for the confirm
s.escrowR = nil
s.escrowRClaimed = true
jobID := s.escrowJob.JobID
s.escrowMu.Unlock()
time.AfterFunc(escrowConfirmTTL+time.Second, func() {
s.escrowMu.Lock()
defer s.escrowMu.Unlock()
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
s.checkEscrowConfirmTTLLocked()
}
})
s.logger.Info("local-api: escrow ceremony R claimed — the package waits for the typed-back words (R-1151)",
"vmid", vmid, "job_id", jobID, "confirm_ttl_s", int(escrowConfirmTTL.Seconds()))
w.Header().Set("Cache-Control", "no-store")
writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "recovery_code": recovery,
"confirm_required": true, "verify_positions": []int{pos[0], pos[1]},
"confirm_expires_in_sec": int(escrowConfirmTTL.Seconds())})
recovery = ""
_ = recovery
return
}
s.logger.Warn("local-api: escrow claim — could not draw the confirm positions; package uploads at the claim (R-978 order)", "err", perr)
}
}
// R-978: store the NEW package on the hub first; only then hand out the code that opens it. Until this
// upload succeeds the hub keeps the previous package, and the customer's previous code keeps working.
if len(s.escrowPending) > 0 {
if s.escrowCeremony.Upload == nil {
s.escrowMu.Unlock()
writeErr(w, http.StatusServiceUnavailable, "the recovery package cannot be stored on this host (no upload configured) — the previous code still works")
return
}
pkg := s.escrowPending
jobID := s.escrowJob.JobID
s.escrowClaiming = true
s.escrowMu.Unlock()
start := time.Now()
base := s.baseCtx
if base == nil {
base = context.Background()
}
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg)
cancel()
s.escrowMu.Lock()
s.escrowClaiming = false
if err != nil {
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return
}
s.escrowPending = nil
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
s.escrowJob.Uploaded = true
s.escrowJob.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
}
s.logger.Info("local-api: escrow claim — package uploaded to the hub; handing out the code (R-978)",
"vmid", vmid, "job_id", jobID, "duration_ms", time.Since(start).Milliseconds())
if len(s.escrowR) == 0 { // cannot happen (TTL and supersede wait for the claim) — never hand out nothing
s.escrowMu.Unlock()
s.logger.Error("local-api: escrow claim — code vanished during the upload (invariant broken)", "job_id", jobID)
writeErr(w, http.StatusInternalServerError, "the recovery code was lost during the claim — run a new ceremony")
return
}
}
recovery := string(s.escrowR) recovery := string(s.escrowR)
s.wipeEscrowRLocked() s.wipeEscrowRLocked()
s.escrowRClaimed = true s.escrowRClaimed = true
@@ -349,11 +556,103 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
s.logger.Info("local-api: escrow ceremony R claimed (one-shot; holder zeroed)", "vmid", vmid, "job_id", jobID) s.logger.Info("local-api: escrow ceremony R claimed (one-shot; holder zeroed)", "vmid", vmid, "job_id", jobID)
w.Header().Set("Cache-Control", "no-store") w.Header().Set("Cache-Control", "no-store")
writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "recovery_code": recovery}) writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "recovery_code": recovery, "confirm_required": false})
recovery = "" // drop the reference promptly (GC caveat: best-effort) recovery = "" // drop the reference promptly (GC caveat: best-effort)
_ = recovery _ = recovery
} }
// handleEscrowCeremonyConfirm is POST /escrow/ceremony/confirm {"words": [w1, w2]} (R-1151): the two words at the
// positions the claim named, as the household typed them back. Right → the package goes to the hub now (200); wrong →
// 422 and the wait goes on; nothing waiting → 404 (or 410 after the window ran out); a failed upload → 502 and the
// wait goes on (a retry may succeed). The words are compared in constant time, case and spaces aside.
func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Request, vmid int) {
if s.escrowCeremony == nil {
writeErr(w, http.StatusServiceUnavailable, "escrow ceremony not configured on this host")
return
}
var req struct {
Words []string `json:"words"`
}
if !decodeBody(w, r, &req) {
return
}
s.escrowMu.Lock()
s.checkEscrowConfirmTTLLocked()
switch {
case s.escrowClaiming:
s.escrowMu.Unlock()
writeErr(w, http.StatusConflict, "the recovery package is being stored — try again in a moment")
return
case !s.escrowAwaitConfirm:
gone := s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseUnconfirmed
s.escrowMu.Unlock()
if gone {
writeErr(w, http.StatusGone, "the new recovery code was not confirmed in time — the previous code still works; run a new ceremony")
return
}
writeErr(w, http.StatusNotFound, "no recovery code waits for confirmation")
return
case len(req.Words) != 2:
s.escrowMu.Unlock()
writeErr(w, http.StatusBadRequest, "two words are needed")
return
}
match := 1
for i := 0; i < 2; i++ {
typed := []byte(strings.ToLower(strings.TrimSpace(req.Words[i])))
match &= subtle.ConstantTimeCompare(typed, s.escrowConfirmWords[i])
zeroBytes(typed)
}
req.Words = nil
if match != 1 {
s.escrowMu.Unlock()
s.logger.Info("local-api: escrow confirm — the typed words do not match; still waiting (R-1151)", "vmid", vmid)
writeErr(w, http.StatusUnprocessableEntity, "the words do not match the recovery code")
return
}
if s.escrowCeremony.Upload == nil || len(s.escrowPending) == 0 { // the claim refuses to defer without both
s.escrowMu.Unlock()
writeErr(w, http.StatusServiceUnavailable, "the recovery package cannot be stored on this host")
return
}
pkg := s.escrowPending
jobID := ""
if s.escrowJob != nil {
jobID = s.escrowJob.JobID
}
s.escrowClaiming = true
s.escrowMu.Unlock()
start := time.Now()
base := s.baseCtx
if base == nil {
base = context.Background()
}
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg)
cancel()
s.escrowMu.Lock()
s.escrowClaiming = false
if err != nil {
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return
}
s.escrowPending = nil
s.wipeEscrowConfirmLocked()
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
s.escrowJob.Uploaded = true
s.escrowJob.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
}
s.escrowMu.Unlock()
s.logger.Info("local-api: escrow confirm — words typed back; the new package is on the hub (R-1151)",
"vmid", vmid, "job_id", jobID, "duration_ms", time.Since(start).Milliseconds())
writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "uploaded": true})
}
// handleEscrowPreflight is GET /escrow/preflight: the wizard's prerequisite checklist. Each item // handleEscrowPreflight is GET /escrow/preflight: the wizard's prerequisite checklist. Each item
// is {id, ok, detail}. Deliberately NOT checked: the PBS key file itself — /etc/pve/priv is 0700 // is {id, ok, detail}. Deliberately NOT checked: the PBS key file itself — /etc/pve/priv is 0700
// root and the daemon cannot stat it; a missing key fails the ceremony fast with a clear error // root and the daemon cannot stat it; a missing key fails the ceremony fast with a clear error
+76 -7
View File
@@ -2,12 +2,14 @@ package localapi
import ( import (
"context" "context"
"encoding/base64"
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http" "net/http"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"sync"
"sync/atomic" "sync/atomic"
"testing" "testing"
"time" "time"
@@ -19,25 +21,91 @@ import (
const testR = "canary-alpha-bravo-charlie-delta-echo-foxtrot-golf-hotel-india" const testR = "canary-alpha-bravo-charlie-delta-echo-foxtrot-golf-hotel-india"
// cannedCeremonyJSON is shaped exactly like the agent's --output=json object (spike §2.6 values). // cannedCeremonyJSON is shaped exactly like the agent's --output=json object (spike §2.6 values;
// version 2, R-978: the package comes back un-uploaded in pending_upload). The package is a MODEL of
// the R-wrapped blob: it "opens" with exactly the code it was sealed under (fakeEscrowHub.opensWith).
func cannedCeremonyJSON(r string) string { func cannedCeremonyJSON(r string) string {
return fmt.Sprintf(`{"version":1,"recovery_code":%q,"key_fingerprint":"f2:87:68:2a:88:50:16:01","entropy_bits":129,"blob_bytes":383,"identity_blob_bytes":450,"restic_pw_sealed":true,"uploaded":true}`, r) return fmt.Sprintf(`{"version":2,"recovery_code":%q,"key_fingerprint":"f2:87:68:2a:88:50:16:01","entropy_bits":129,"blob_bytes":383,"identity_blob_bytes":450,"restic_pw_sealed":true,"uploaded":false,"pending_upload":%s}`,
r, sealedPackage(r))
}
// sealedPackage is the model hub upload body for a ceremony that minted code r.
func sealedPackage(r string) string {
return fmt.Sprintf(`{"blob_b64":%q,"key_fingerprint":"f2:87:68:2a:88:50:16:01","posture":"zero_knowledge","created_at":"2026-06-10T12:00:00Z"}`,
base64.StdEncoding.EncodeToString([]byte("sealed-under:"+r)))
}
// fakeEscrowHub is the hub's current escrow package for this host (the PUT target). Upload replaces it,
// as the hub does when the sealed secrets are unchanged (the R-978 case: no retained copy).
type fakeEscrowHub struct {
mu sync.Mutex
current []byte
uploads int
err error // non-nil → the upload fails, nothing stored
gate chan struct{} // non-nil → the upload blocks until closed
entered chan struct{} // non-nil → closed when an upload starts
}
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
if f.entered != nil {
close(f.entered)
f.entered = nil
}
if f.gate != nil {
<-f.gate
}
f.mu.Lock()
defer f.mu.Unlock()
if f.err != nil {
return f.err
}
f.current = append([]byte(nil), pkg...)
f.uploads++
return nil
}
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
func (f *fakeEscrowHub) opensWith(r string) bool {
f.mu.Lock()
defer f.mu.Unlock()
var pkg struct {
BlobB64 string `json:"blob_b64"`
}
if json.Unmarshal(f.current, &pkg) != nil {
return false
}
raw, err := base64.StdEncoding.DecodeString(pkg.BlobB64)
return err == nil && string(raw) == "sealed-under:"+r
}
func (f *fakeEscrowHub) count() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.uploads
} }
// newEscrowTestServer builds a server with the ceremony configured and every seam faked. // newEscrowTestServer builds a server with the ceremony configured and every seam faked.
func newEscrowTestServer(t *testing.T, run ceremonyRunner) *Server { func newEscrowTestServer(t *testing.T, run ceremonyRunner) *Server {
t.Helper()
srv, _ := newEscrowTestServerHub(t, run, &fakeEscrowHub{})
return srv
}
// newEscrowTestServerHub is newEscrowTestServer with the hub's escrow record exposed (R-978).
func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub) (*Server, *fakeEscrowHub) {
t.Helper() t.Helper()
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil) srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.escrowCeremony = &EscrowCeremonyConfig{ srv.escrowCeremony = &EscrowCeremonyConfig{
PBSStorageID: "felhom-pbs", PBSStorageID: "felhom-pbs",
HubConfigured: true, HubConfigured: true,
DRConfigured: func() bool { return true }, DRConfigured: func() bool { return true },
Upload: hub.upload,
} }
srv.ceremonyRun = run srv.ceremonyRun = run
srv.escrowSudoCheck = func(context.Context) error { return nil } srv.escrowSudoCheck = func(context.Context) error { return nil }
srv.escrowLookPath = func(string) (string, error) { return "/usr/bin/age", nil } srv.escrowLookPath = func(string) (string, error) { return "/usr/bin/age", nil }
srv.statFile = func(string) bool { return true } srv.statFile = func(string) bool { return true }
return srv return srv, hub
} }
// okRunner returns the canned success output and counts invocations. // okRunner returns the canned success output and counts invocations.
@@ -160,8 +228,8 @@ func TestEscrowCeremony_OneShotClaim(t *testing.T) {
if !strings.Contains(body, `"phase":"done"`) || !strings.Contains(body, `"claimable":true`) { if !strings.Contains(body, `"phase":"done"`) || !strings.Contains(body, `"claimable":true`) {
t.Fatalf("status not done/claimable: %s", body) t.Fatalf("status not done/claimable: %s", body)
} }
if !strings.Contains(body, `"restic_pw_sealed":true`) || !strings.Contains(body, `"uploaded":true`) { if !strings.Contains(body, `"restic_pw_sealed":true`) || !strings.Contains(body, `"uploaded":false`) {
t.Fatalf("summary fields missing: %s", body) t.Fatalf("summary fields missing (R-978: not uploaded before the claim): %s", body)
} }
assertNoR(t, "status payload", body) assertNoR(t, "status payload", body)
@@ -199,7 +267,8 @@ func TestEscrowCeremony_OneShotClaim(t *testing.T) {
// Post-claim status: claimed, not claimable, still phase done. // Post-claim status: claimed, not claimable, still phase done.
st2 := do(t, h, "GET", "/escrow/ceremony/status", "A", "") st2 := do(t, h, "GET", "/escrow/ceremony/status", "A", "")
if !strings.Contains(st2.Body.String(), `"claimed":true`) || strings.Contains(st2.Body.String(), `"claimable":true`) { if !strings.Contains(st2.Body.String(), `"claimed":true`) || strings.Contains(st2.Body.String(), `"claimable":true`) ||
!strings.Contains(st2.Body.String(), `"uploaded":true`) {
t.Fatalf("post-claim status wrong: %s", st2.Body.String()) t.Fatalf("post-claim status wrong: %s", st2.Body.String())
} }
} }
@@ -344,7 +413,7 @@ func TestEscrowCeremony_FailurePaths(t *testing.T) {
}) })
t.Run("wrong version", func(t *testing.T) { t.Run("wrong version", func(t *testing.T) {
srv := newEscrowTestServer(t, func(context.Context) ([]byte, []byte, int, error) { srv := newEscrowTestServer(t, func(context.Context) ([]byte, []byte, int, error) {
return []byte(`{"version":2,"recovery_code":"` + testR + `"}`), nil, 0, nil return []byte(`{"version":1,"recovery_code":"` + testR + `","uploaded":true}`), nil, 0, nil
}) })
h := srv.Handler() h := srv.Handler()
startAndWait(t, srv, h) startAndWait(t, srv, h)
+223
View File
@@ -0,0 +1,223 @@
package localapi
import (
"encoding/json"
"errors"
"net/http"
"strings"
"testing"
"time"
)
// R-1151 — the new recovery code replaces the hub's package only after the household TYPED IT BACK. Before, the
// claim uploaded it (R-978), so a household that opened the new code and closed the page without writing it down
// had already replaced the package their old code opens. These tests ask the hub's current package the household's
// question — „does my written-down code still open it?".
type deferredClaim struct {
code int
r string
positions []int
required bool
body string
}
func claimDeferred(t *testing.T, h http.Handler) deferredClaim {
t.Helper()
w := do(t, h, "POST", "/escrow/ceremony/claim", "A", `{"defer_upload":true}`)
var env struct {
Data struct {
RecoveryCode string `json:"recovery_code"`
ConfirmRequired bool `json:"confirm_required"`
VerifyPositions []int `json:"verify_positions"`
} `json:"data"`
}
_ = json.Unmarshal(w.Body.Bytes(), &env)
return deferredClaim{w.Code, env.Data.RecoveryCode, env.Data.VerifyPositions, env.Data.ConfirmRequired, w.Body.String()}
}
// typedBack is what a household that wrote the code down types for the claim's two positions.
func (c deferredClaim) typedBack(t *testing.T) []string {
t.Helper()
words := strings.Split(c.r, "-")
if len(c.positions) != 2 {
t.Fatalf("the claim named %d positions, want 2: %s", len(c.positions), c.body)
}
return []string{words[c.positions[0]-1], words[c.positions[1]-1]}
}
func confirm(t *testing.T, h http.Handler, words []string) int {
t.Helper()
b, _ := json.Marshal(map[string]any{"words": words})
return do(t, h, "POST", "/escrow/ceremony/confirm", "A", string(b)).Code
}
// THE R-1151 CASE. Ceremony 1 is claimed and typed back (the household's code 1). Ceremony 2 is started and its code
// SHOWN (claimed) — and then the page is closed: the words never come back. Three ways it ends: the confirm window
// runs out, a new start supersedes it, the agent restarts. In every case code 1 must still open the hub's package.
//
// COMPANION RED-PROOF (observed 2026-10-19): ignore the claim's defer_upload (`if false && req.DeferUpload …`, so
// every claim takes the R-978 order and uploads at once) → all three subtests FAIL with "a code that was shown but
// never typed back replaced the package: code 1 no longer opens it (uploads=2)". Restored.
func TestEscrowCeremony_R1151_ShownButNotTypedBackKeepsFirstCode(t *testing.T) {
setup := func(t *testing.T) (*Server, *fakeEscrowHub, http.Handler, *time.Time) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR, "third-code-charlie"), &fakeEscrowHub{})
cur := testNow
srv.now = func() time.Time { return cur }
h := srv.Handler()
startAndWait(t, srv, h)
c1 := claimDeferred(t, h)
if c1.code != http.StatusOK || c1.r != firstR {
t.Fatalf("deferred claim of ceremony 1: %d %s", c1.code, c1.body)
}
if c1.required { // the household types code 1 back
if code := confirm(t, h, c1.typedBack(t)); code != http.StatusOK {
t.Fatalf("typing code 1 back: %d", code)
}
}
if !hub.opensWith(firstR) || hub.count() != 1 {
t.Fatalf("after ceremony 1 the hub must hold code 1's package (uploads=%d)", hub.count())
}
startAndWait(t, srv, h)
if c2 := claimDeferred(t, h); c2.code != http.StatusOK || c2.r != secondR {
t.Fatalf("deferred claim of ceremony 2: %d %s", c2.code, c2.body)
} // …shown, and the page is closed
return srv, hub, h, &cur
}
check := func(t *testing.T, hub *fakeEscrowHub) {
t.Helper()
if !hub.opensWith(firstR) {
t.Fatalf("a code that was shown but never typed back replaced the package: code 1 no longer opens it (uploads=%d)", hub.count())
}
if hub.opensWith(secondR) || hub.count() != 1 {
t.Fatalf("a package for the never-confirmed code 2 reached the hub (uploads=%d)", hub.count())
}
}
t.Run("the confirm window runs out", func(t *testing.T) {
_, hub, h, cur := setup(t)
*cur = cur.Add(escrowConfirmTTL + time.Minute)
check(t, hub)
st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String()
if !strings.Contains(st, `"phase":"unconfirmed_void"`) || !strings.Contains(st, `"awaiting_confirm":false`) {
t.Fatalf("after the window the job must read unconfirmed_void: %s", st)
}
if code := confirm(t, h, []string{"second", "bravo"}); code != http.StatusGone {
t.Fatalf("a confirm after the window: got %d, want 410", code)
}
})
t.Run("a new start supersedes it", func(t *testing.T) {
srv, hub, h, _ := setup(t)
startAndWait(t, srv, h)
if code := confirm(t, h, []string{"second", "bravo"}); code != http.StatusNotFound {
t.Fatalf("a confirm for the superseded code: got %d, want 404", code)
}
check(t, hub)
})
t.Run("the agent restarts", func(t *testing.T) {
_, hub, _, _ := setup(t)
srv2, _ := newEscrowTestServerHub(t, sequenceRunner("unused"), hub)
if code := confirm(t, srv2.Handler(), []string{"second", "bravo"}); code != http.StatusNotFound {
t.Fatalf("a confirm after a restart: got %d, want 404", code)
}
check(t, hub)
})
}
// The positive control and the typing: wrong words store nothing and the wait goes on; the right words (any case,
// spaces around) store the package, once.
//
// COMPANION RED-PROOF (observed 2026-10-19): skip the word check (`if match != 1 && false`) → this fails with "one
// wrong word: got 200 (uploads=1), want 422 and nothing stored". Restored.
func TestEscrowCeremony_R1151_TypedBackStoresThePackage(t *testing.T) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(secondR), &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if c.code != http.StatusOK || !c.required || hub.count() != 0 {
t.Fatalf("a deferred claim must hand out the code and store nothing yet: %d required=%v uploads=%d", c.code, c.required, hub.count())
}
right := c.typedBack(t)
if code := confirm(t, h, []string{right[0], "wrong"}); code != http.StatusUnprocessableEntity || hub.count() != 0 {
t.Fatalf("one wrong word: got %d (uploads=%d), want 422 and nothing stored", code, hub.count())
}
if code := confirm(t, h, []string{right[0]}); code != http.StatusBadRequest {
t.Fatalf("one word only: got %d, want 400", code)
}
st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String()
if !strings.Contains(st, `"awaiting_confirm":true`) || !strings.Contains(st, `"uploaded":false`) {
t.Fatalf("a wrong try must leave the wait open: %s", st)
}
assertNoRIn(t, "status while awaiting the words", st, secondR)
for _, w := range right {
assertNoRIn(t, "status while awaiting the words", st, `"`+w+`"`)
}
if code := confirm(t, h, []string{" " + strings.ToUpper(right[0]), right[1] + " "}); code != http.StatusOK {
t.Fatalf("the right words: got %d, want 200", code)
}
if !hub.opensWith(secondR) || hub.count() != 1 {
t.Fatalf("after the words came back the hub must hold code 2's package (uploads=%d)", hub.count())
}
if code := confirm(t, h, right); code != http.StatusNotFound || hub.count() != 1 {
t.Fatalf("a second confirm: got %d (uploads=%d), want 404 and no second upload", code, hub.count())
}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
t.Fatalf("status after the confirm: %s", st)
}
}
// A controller that has not updated (0.331.0 and older) claims with `{}` and never confirms: it keeps the R-978
// order — the package is stored at the claim — so its household's ceremony still completes.
func TestEscrowCeremony_R1151_OldControllerKeepsTheClaimOrder(t *testing.T) {
for _, body := range []string{"", "{}"} {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR), &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
w := do(t, h, "POST", "/escrow/ceremony/claim", "A", body)
if w.Code != http.StatusOK || !strings.Contains(w.Body.String(), firstR) || !strings.Contains(w.Body.String(), `"confirm_required":false`) {
t.Fatalf("old-shape claim %q: %d %s", body, w.Code, w.Body.String())
}
if !hub.opensWith(firstR) || hub.count() != 1 {
t.Fatalf("old-shape claim %q must store the package at once (uploads=%d)", body, hub.count())
}
if code := confirm(t, h, []string{"first", "alpha"}); code != http.StatusNotFound {
t.Fatalf("nothing waits after an old-shape claim: got %d, want 404", code)
}
}
}
// The hub failing at the confirm keeps the wait open and the previous package current; a retry succeeds.
func TestEscrowCeremony_R1151_UploadFailureAtConfirmKeepsWaiting(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if code := confirm(t, h, c.typedBack(t)); code != http.StatusBadGateway {
t.Fatalf("confirm with the hub failing: got %d, want 502", code)
}
hub.mu.Lock()
hub.err = nil
hub.mu.Unlock()
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
t.Fatalf("retry after the hub recovered: %d (opens: %v)", code, hub.opensWith(secondR))
}
}
// The two positions are distinct, in range, smaller first — over many draws every position appears.
func TestPickConfirmPositions(t *testing.T) {
seen := map[int]bool{}
for i := 0; i < 2000; i++ {
p, err := pickConfirmPositions(10)
if err != nil {
t.Fatal(err)
}
if p[0] < 1 || p[1] > 10 || p[0] >= p[1] {
t.Fatalf("bad positions %v", p)
}
seen[p[0]], seen[p[1]] = true, true
}
if len(seen) != 10 {
t.Fatalf("only positions %v were ever drawn", seen)
}
}
+219
View File
@@ -0,0 +1,219 @@
package localapi
import (
"context"
"errors"
"net/http"
"strings"
"testing"
"time"
)
// R-978 — starting the escrow ceremony again must not make the customer's written-down recovery code
// useless. The hub overwrites the current package when the sealed secrets are unchanged, so the NEW
// package may reach the hub only when its code is claimed. These tests ask the customer's question of
// the hub's current package — „does my code still open it?" — not whether some function ran.
const (
firstR = "first-ceremony-code-alpha"
secondR = "second-ceremony-code-bravo"
)
// sequenceRunner mints the given codes in order, one per ceremony.
func sequenceRunner(codes ...string) ceremonyRunner {
i := 0
return func(context.Context) ([]byte, []byte, int, error) {
r := codes[i]
if i < len(codes)-1 {
i++
}
return []byte(cannedCeremonyJSON(r)), nil, 0, nil
}
}
func claimCode(t *testing.T, h http.Handler) (int, string) {
t.Helper()
w := do(t, h, "POST", "/escrow/ceremony/claim", "A", "")
return w.Code, w.Body.String()
}
// THE R-978 CASE. Ceremony 1 is claimed (the customer writes code 1 down). Ceremony 2 is started and
// abandoned — three ways: its code expires unclaimed, a third start supersedes it, the agent restarts.
// In every case the hub's current package must still open with code 1, and nothing was uploaded for 2.
//
// COMPANION RED-PROOF (observed 2026-10-18): restore the pre-R-978 shape — the package uploaded when the
// ceremony finishes (s.escrowCeremony.Upload(base, out.PendingUpload) in the job goroutine, as the root
// subprocess did with `--upload`) and nothing left for the claim (finishEscrowCeremony(job, R, nil)) →
// all three subtests FAIL with "the abandoned second ceremony replaced the package: code 1 no longer
// opens it (uploads=2)" (uploads=3 for the superseding start). Restored.
func TestEscrowCeremony_R978_AbandonedSecondCeremonyKeepsFirstCode(t *testing.T) {
setup := func(t *testing.T) (*Server, *fakeEscrowHub, http.Handler, *time.Time) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR, "third-code-charlie"), &fakeEscrowHub{})
cur := testNow
srv.now = func() time.Time { return cur }
h := srv.Handler()
startAndWait(t, srv, h)
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("claim of ceremony 1: %d %s", code, body)
}
if !hub.opensWith(firstR) || hub.count() != 1 {
t.Fatalf("after the first claim the hub must hold code 1's package (uploads=%d)", hub.count())
}
startAndWait(t, srv, h) // ceremony 2 — about to be abandoned
return srv, hub, h, &cur
}
check := func(t *testing.T, hub *fakeEscrowHub) {
t.Helper()
if !hub.opensWith(firstR) {
t.Fatalf("the abandoned second ceremony replaced the package: code 1 no longer opens it (uploads=%d)", hub.count())
}
if hub.opensWith(secondR) || hub.count() != 1 {
t.Fatalf("a package for the never-claimed code 2 reached the hub (uploads=%d)", hub.count())
}
}
t.Run("code 2 expires unclaimed", func(t *testing.T) {
_, hub, h, cur := setup(t)
*cur = cur.Add(escrowClaimTTL + time.Minute)
if code, _ := claimCode(t, h); code != http.StatusGone {
t.Fatalf("claim after TTL: got %d, want 410", code)
}
check(t, hub)
})
t.Run("a third start supersedes code 2", func(t *testing.T) {
srv, hub, h, _ := setup(t)
startAndWait(t, srv, h) // ceremony 3, also left unclaimed
check(t, hub)
})
t.Run("the agent restarts before code 2 is claimed", func(t *testing.T) {
_, hub, _, _ := setup(t)
// A restart is a new process with an empty slot; only the hub's record survives it.
srv2, _ := newEscrowTestServerHub(t, sequenceRunner("unused"), hub)
if code, _ := claimCode(t, srv2.Handler()); code != http.StatusNotFound {
t.Fatalf("claim after a restart: got %d, want 404", code)
}
check(t, hub)
})
}
// The positive control: CLAIMING code 2 is what replaces the package — the code handed out opens the
// package the hub now holds, and the claim answers only after the hub took it.
func TestEscrowCeremony_R978_ClaimStoresThePackageBeforeHandingOutTheCode(t *testing.T) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR), &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
if hub.count() != 0 {
t.Fatalf("the package was uploaded before any claim (uploads=%d)", hub.count())
}
claimCode(t, h)
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusOK || !strings.Contains(body, secondR) {
t.Fatalf("claim of ceremony 2: %d %s", code, body)
}
if !hub.opensWith(secondR) || hub.count() != 2 {
t.Fatalf("after claiming code 2 the hub must hold its package (uploads=%d, opens with 2: %v)", hub.count(), hub.opensWith(secondR))
}
}
// A failed upload hands out NO code, keeps the code claimable and the previous package current; a retry
// that reaches the hub succeeds.
func TestEscrowCeremony_R978_UploadFailureKeepsTheCodeClaimable(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusBadGateway {
t.Fatalf("claim with the hub failing: got %d, want 502 (%s)", code, body)
}
assertNoRIn(t, "failed-upload claim payload", body, firstR)
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) || !strings.Contains(st, `"uploaded":false`) {
t.Fatalf("after a failed upload the code must stay claimable and un-uploaded: %s", st)
}
hub.mu.Lock()
hub.err = nil
hub.mu.Unlock()
code, body = claimCode(t, h)
if code != http.StatusOK || !strings.Contains(body, firstR) || !hub.opensWith(firstR) {
t.Fatalf("retry after the hub recovered: %d %s (opens: %v)", code, body, hub.opensWith(firstR))
}
}
// While the claim's upload is in flight, neither the TTL nor a new start may take the code away — else
// the hub would hold a package whose code nobody was given (the R-978 failure, reached the other way).
func TestEscrowCeremony_R978_TTLAndRestartWaitForAnInFlightClaim(t *testing.T) {
hub := &fakeEscrowHub{gate: make(chan struct{}), entered: make(chan struct{})}
entered := hub.entered
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR), hub)
var mu = make(chan struct{}, 1)
mu <- struct{}{}
cur := testNow
srv.now = func() time.Time { <-mu; defer func() { mu <- struct{}{} }(); return cur }
h := srv.Handler()
startAndWait(t, srv, h)
type res struct {
code int
body string
}
done := make(chan res, 1)
go func() { c, b := claimCode(t, h); done <- res{c, b} }()
<-entered
<-mu
cur = cur.Add(escrowClaimTTL + time.Minute) // the TTL passes mid-upload
mu <- struct{}{}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); strings.Contains(st, `"phase":"unclaimed_void"`) {
t.Fatalf("the TTL voided the code while its package was uploading: %s", st)
}
if w := do(t, h, "POST", "/escrow/ceremony", "A", ""); w.Code != http.StatusConflict {
t.Fatalf("a new start during the claim's upload: got %d, want 409", w.Code)
}
if code, _ := claimCode(t, h); code != http.StatusConflict {
t.Fatalf("a second claim during the upload: got %d, want 409", code)
}
close(hub.gate)
r := <-done
if r.code != http.StatusOK || !strings.Contains(r.body, firstR) || !hub.opensWith(firstR) {
t.Fatalf("the in-flight claim must deliver the code whose package it stored: %d %s", r.code, r.body)
}
}
// No upload configured → the claim refuses rather than hand out a code for a package the hub lacks.
func TestEscrowCeremony_R978_NoUploaderRefusesTheClaim(t *testing.T) {
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), &fakeEscrowHub{})
srv.escrowCeremony.Upload = nil
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusServiceUnavailable {
t.Fatalf("claim with no uploader: got %d, want 503 (%s)", code, body)
}
assertNoRIn(t, "no-uploader claim payload", body, firstR)
}
// An output that uploaded by itself (the pre-R-978 shape) or carries no package fails the ceremony.
func TestEscrowCeremony_R978_OutputWithoutPendingPackageFails(t *testing.T) {
srv, _ := newEscrowTestServerHub(t, func(context.Context) ([]byte, []byte, int, error) {
return []byte(`{"version":2,"recovery_code":"` + firstR + `","uploaded":true}`), nil, 0, nil
}, &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String()
if !strings.Contains(st, `"phase":"failed"`) || !strings.Contains(st, "no pending package") {
t.Fatalf("want failed: %s", st)
}
if code, _ := claimCode(t, h); code != http.StatusConflict {
t.Fatalf("claim on a failed ceremony: got %d, want 409", code)
}
}
func assertNoRIn(t *testing.T, where, body, r string) {
t.Helper()
if strings.Contains(body, r) {
t.Fatalf("%s carries the recovery code", where)
}
}
+16
View File
@@ -127,6 +127,22 @@ func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Req
"retained_has_restic_pw": match.HasResticPassword, "retained_has_restic_pw": match.HasResticPassword,
}, },
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held") "the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
// ── R-304 (2026-10-08) — NOT EVERY EARLIER PACKAGE WAS CHECKED. ─────────────────────────
//
// The current package refused the code, no retained package opened it — and at least one earlier
// package the hub holds was never tried (or the list could not be read). Saying „the code is
// wrong" here would claim a check that did not happen. 424 (Failed Dependency): the verdict
// depends on packages we could not try. The controller classifies on the status, never on this
// sentence; an older controller maps an unknown status to its neutral „we do not know why".
case errors.Is(err, escrow.ErrRetainedUnchecked):
n := -1
var ue *escrow.RetainedUncheckedError
if errors.As(err, &ue) {
n = ue.Unchecked
}
s.logger.Warn("local-api: offsite key recovery: the code did not open the current package and earlier packages were NOT all checked — not reported as a wrong code (R-304)", "vmid", vmid, "unchecked", n)
writeStatus(w, http.StatusFailedDependency, false, map[string]any{"older_unchecked": n},
"the recovery code did not open the current sealed package, and earlier packages the hub holds were not all checked — the code may belong to one of them; nothing was written")
case errors.Is(err, escrow.ErrNoResticPassword): case errors.Is(err, escrow.ErrNoResticPassword):
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid) s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)") writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
@@ -54,6 +54,13 @@ func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
wantStatus: 404, wantStatus: 404,
mustNotSay: []string{"did not open"}, mustNotSay: []string{"did not open"},
}, },
{
// R-304: earlier packages were not all tried — never „did not open the sealed bundle" (the wrong-code words).
name: "earlier packages not all checked — not a wrong code",
err: &escrow.RetainedUncheckedError{Unchecked: 2},
wantStatus: 424,
mustNotSay: []string{"did not open the sealed bundle", "could not be fetched"},
},
{ {
name: "the bundle predates the repository-password field", name: "the bundle predates the repository-password field",
err: escrow.ErrNoResticPassword, err: escrow.ErrNoResticPassword,
+31 -9
View File
@@ -3,11 +3,11 @@ package localapi
import ( import (
"context" "context"
"fmt" "fmt"
"io"
"log/slog" "log/slog"
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
) )
@@ -105,6 +105,26 @@ func (b *GuestBinder) RebootGuest(ctx context.Context, vmid int) error {
return nil return nil
} }
// ShutdownGuest stops the guest cleanly with a time limit (R-998 host power-off, step 2). `pct shutdown`
// blocks until the guest is down or the limit passes; then it fails and the guest is still running.
func (b *GuestBinder) ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error {
secs := int(timeout / time.Second)
if secs < 1 {
secs = 1
}
b.logger.Warn("host-poweroff: shutting the guest down", "vmid", vmid, "timeout_s", secs)
if err := b.run(ctx, "pct", "shutdown", strconv.Itoa(vmid), "--timeout", strconv.Itoa(secs)); err != nil {
return fmt.Errorf("host-poweroff: pct shutdown %d: %w", vmid, err)
}
return nil
}
// PowerOffHost powers the Proxmox host off (R-998, step 3). The one fixed vector `/usr/bin/systemctl poweroff`.
func (b *GuestBinder) PowerOffHost(ctx context.Context) error {
b.logger.Warn("host-poweroff: powering the host off")
return b.run(ctx, "/usr/bin/systemctl", "poweroff")
}
func (b *GuestBinder) run(ctx context.Context, name string, args ...string) error { func (b *GuestBinder) run(ctx context.Context, name string, args ...string) error {
_, stderr, err := b.runner.Run(ctx, name, args...) _, stderr, err := b.runner.Run(ctx, name, args...)
if err != nil { if err != nil {
@@ -126,14 +146,16 @@ func (b *GuestBinder) GuestExec(ctx context.Context, vmid int, args ...string) (
return string(out), nil return string(out), nil
} }
// GuestExecStdin is GuestExec with the in-guest command's stdin fed from stdin. The controller-swap // privApplyBin is the root content checker (R-861); its `controller-image` verb writes the guest's image file.
// write uses it to pipe the image ref into an in-guest `tee` (no shell vector, no interpolation), const privApplyBin = "/usr/local/sbin/felhom-priv-apply"
// through the same fenced runner so the `sudo -n` prefix stays in one place.
func (b *GuestBinder) GuestExecStdin(ctx context.Context, vmid int, stdin io.Reader, args ...string) (string, error) { // WriteControllerImage pipes `image\n` to `felhom-priv-apply controller-image <vmid>` through the same fenced runner
pctArgs := append([]string{"exec", strconv.Itoa(vmid), "--"}, args...) // (the `sudo -n` prefix stays in one place). The verb checks the ref as root and writes the guest file itself
out, stderr, err := b.runner.RunStdin(ctx, stdin, "pct", pctArgs...) // (R-861 (a) A1, `09` §3 decision 165). Pinned by TestR861_WriteControllerImageUsesTheRootVerb.
func (b *GuestBinder) WriteControllerImage(ctx context.Context, vmid int, image string) error {
_, stderr, err := b.runner.RunStdin(ctx, strings.NewReader(image+"\n"), privApplyBin, "controller-image", strconv.Itoa(vmid))
if err != nil { if err != nil {
return string(out), fmt.Errorf("pct exec %d %v: %w: %s", vmid, args, err, strings.TrimSpace(string(stderr))) return fmt.Errorf("felhom-priv-apply controller-image %d: %w: %s", vmid, err, strings.TrimSpace(string(stderr)))
} }
return string(out), nil return nil
} }
+12 -1
View File
@@ -57,6 +57,10 @@ type MemoryInfo struct {
MaxMB int64 `json:"max_mb"` MaxMB int64 `json:"max_mb"`
FloorMB int64 `json:"floor_mb"` FloorMB int64 `json:"floor_mb"`
Running bool `json:"running"` Running bool `json:"running"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up): the host's swap and this guest's swap allowance, so the controller
// can tell the household when the host has too little swap behind the guest (its install page). nil = not read.
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
} }
// memoryResizeRequest is the POST /guest/memory body. // memoryResizeRequest is the POST /guest/memory body.
@@ -94,6 +98,8 @@ type memoryBounds struct {
maxMB int64 maxMB int64
floorMB int64 floorMB int64
running bool running bool
hostSwapMB int64
guestSwapMB int64
} }
func (b memoryBounds) refusal(code string) memoryRefusal { func (b memoryBounds) refusal(code string) memoryRefusal {
@@ -104,7 +110,9 @@ func (b memoryBounds) refusal(code string) memoryRefusal {
} }
func (b memoryBounds) info(vmid int) MemoryInfo { func (b memoryBounds) info(vmid int) MemoryInfo {
hostSwap, guestSwap := b.hostSwapMB, b.guestSwapMB
return MemoryInfo{ return MemoryInfo{
HostSwapMB: &hostSwap, GuestSwapMB: &guestSwap,
VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB, VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB,
MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running, MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running,
} }
@@ -139,6 +147,8 @@ func (s *Server) readMemoryBounds(ctx context.Context, vmid int) (memoryBounds,
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max) hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
minMB: minGuestMemoryMB, minMB: minGuestMemoryMB,
running: st.Status == "running", running: st.Status == "running",
hostSwapMB: node.Swap.Total / mib, // R-1192 follow-up: the kernel's SwapTotal, as Proxmox reports it
guestSwapMB: cfg.Swap, // PVE config swap is MB
} }
b.maxMB = b.hostTotalMB - hostReserveMB b.maxMB = b.hostTotalMB - hostReserveMB
b.floorMB = b.usageMB + shrinkUsageMarginMB b.floorMB = b.usageMB + shrinkUsageMarginMB
@@ -225,7 +235,8 @@ func (s *Server) handleGuestMemoryResize(w http.ResponseWriter, r *http.Request,
} }
// Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on. // Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target)}) // R-1192: the swap moves with the memory (GuestSwapMB) — a grown guest with the old swap cap thrashes the same way.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target), "swap": fmt.Sprintf("%d", GuestSwapMB(target))})
if err != nil { if err != nil {
s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err) s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err)
writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error()) writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error())
+8 -1
View File
@@ -25,12 +25,14 @@ type fakeMemory struct {
setUPID string setUPID string
setErr error setErr error
applyReflect bool applyReflect bool
hostSwapB int64 // R-1192 follow-up: node swap total
guestSwapMB int64
} }
func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) { func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
return proxmox.GuestConfig{Memory: f.allocMB}, nil return proxmox.GuestConfig{Memory: f.allocMB, Swap: f.guestSwapMB}, nil
} }
func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) { func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) {
f.mu.Lock() f.mu.Lock()
@@ -42,6 +44,7 @@ func (f *fakeMemory) NodeStatus(_ context.Context) (proxmox.NodeStatus, error) {
defer f.mu.Unlock() defer f.mu.Unlock()
var ns proxmox.NodeStatus var ns proxmox.NodeStatus
ns.Memory.Total = f.hostTotalB ns.Memory.Total = f.hostTotalB
ns.Swap.Total = f.hostSwapB
return ns, nil return ns, nil
} }
func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) { func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) {
@@ -151,6 +154,10 @@ func TestGuestMemory_GrowAndShrink(t *testing.T) {
if f.lastParams["memory"] != "12288" { if f.lastParams["memory"] != "12288" {
t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"]) t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"])
} }
// R-1192: the swap moves with the memory.
if f.lastParams["swap"] != "12288" {
t.Errorf("SetConfig swap = %q, want 12288 (swap = RAM)", f.lastParams["swap"])
}
}) })
t.Run("shrink above floor", func(t *testing.T) { t.Run("shrink above floor", func(t *testing.T) {
f := stdFixture() // floor = max(2048, 3000+512) = 3512 f := stdFixture() // floor = max(2048, 3000+512) = 3512
+73
View File
@@ -0,0 +1,73 @@
package localapi
import (
"context"
"fmt"
"log/slog"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// ── R-1192 (2026-10-21): the guest may swap as much as its RAM ────────────────────────────────────────────────────
//
// MEASURED (felhom.eu audits/fixes-2026-10-21/P4-freeze/): on the minimum machine (8 GB host, guest 4096 MB) the
// household's first uploads made Immich load its analysis models (+1.15 GB) on top of 8 apps' 1.75 GB; the guest hit
// its 4096 MB limit with its swap capped at Proxmox's LXC default of 512 MB — and full. The kernel could then evict
// only file pages: ~360 MB was left for the program code of ~25 containers, re-read from disk without end (150 000–
// 600 000 refaults and ~1 GB of reads per 10 s; memory „full" pressure 92–95 %, load 155, the dashboard dead, 0 OOM
// kills). Raising the cap to 4096 MB live ended it in under a minute (memory „full" 96 % → 1.7 %, dashboard back
// 42 s later) while the uploads ran on. The host had 7.4 GB of swap free the whole time.
//
// The agent never set `swap` at all, so every guest had the 512 MB default. Now:
// - a new guest is created with swap = its memory (reconcile.bringup);
// - a RAM resize sets swap with the memory (handleGuestMemoryResize);
// - at agent start, every LXC in the felhom pool whose swap is below its memory is raised to it (this file).
// Raising swap is a live cgroup change (memory.swap.max), no restart. It never lowers a swap someone set higher.
// GuestSwapMB is the swap a guest of memoryMB gets: the same as its RAM.
func GuestSwapMB(memoryMB int64) int64 { return memoryMB }
// SwapOps is the Proxmox surface EnsureGuestSwap needs (satisfied by *proxmox.Client).
type SwapOps interface {
Pool(ctx context.Context, name string) (proxmox.PoolInfo, error)
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
SetConfig(ctx context.Context, vmid int, params map[string]string) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
}
// EnsureGuestSwap raises the swap of every LXC guest in pool to GuestSwapMB(memory) when it is below. Returns how
// many guests were raised. One guest's failure is logged and the others are still handled.
func EnsureGuestSwap(ctx context.Context, ops SwapOps, pool string, logger *slog.Logger) (int, error) {
p, err := ops.Pool(ctx, pool)
if err != nil {
return 0, fmt.Errorf("guest swap: read pool %s: %w", pool, err)
}
raised := 0
for _, m := range p.Members {
if m.Type != "lxc" || m.VMID <= 0 {
continue
}
cfg, err := ops.GuestConfig(ctx, m.VMID)
if err != nil {
logger.Warn("guest-swap: config read failed (left as it is)", "vmid", m.VMID, "err", err)
continue
}
want := GuestSwapMB(cfg.Memory)
if cfg.Memory <= 0 || cfg.Swap >= want {
logger.Debug("guest-swap: nothing to do", "vmid", m.VMID, "memory_mb", cfg.Memory, "swap_mb", cfg.Swap)
continue
}
upid, err := ops.SetConfig(ctx, m.VMID, map[string]string{"swap": fmt.Sprintf("%d", want)})
if err == nil && upid != "" {
_, err = ops.WaitTask(ctx, upid, proxmox.WaitOptions{})
}
if err != nil {
logger.Warn("guest-swap: raise failed (left as it is)", "vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want, "err", err)
continue
}
raised++
logger.Info("guest-swap: raised (R-1192 — a guest swap cap below its RAM turned a memory burst into a freeze)",
"vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want)
}
return raised, nil
}
+71
View File
@@ -0,0 +1,71 @@
package localapi
import (
"context"
"errors"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: the freeze measured on the minimum machine (felhom.eu audits/fixes-2026-10-21/P4-freeze/) — a guest whose
// swap is capped below its RAM evicts program code instead of idle memory. EnsureGuestSwap raises every pool guest's
// swap to its RAM, never lowers one, skips non-LXC members, and survives one guest's failure.
// COMPANION RED-PROOF: return 0 before the SetConfig call → „raised 0, want 2".
type fakeSwapOps struct {
cfg map[int]proxmox.GuestConfig
failSet map[int]bool
set map[int]map[string]string
}
func (f *fakeSwapOps) Pool(context.Context, string) (proxmox.PoolInfo, error) {
return proxmox.PoolInfo{Members: []proxmox.PoolMember{{VMID: 9201, Type: "lxc"}, {VMID: 9202, Type: "lxc"}, {VMID: 300, Type: "qemu"},
{Type: "storage"}, {VMID: 9203, Type: "lxc"}, {VMID: 9204, Type: "lxc"}}}, nil
}
func (f *fakeSwapOps) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
c, ok := f.cfg[vmid]
if !ok {
return c, errors.New("no such guest")
}
return c, nil
}
func (f *fakeSwapOps) SetConfig(_ context.Context, vmid int, p map[string]string) (string, error) {
if f.failSet[vmid] {
return "", errors.New("pve said no")
}
if f.set == nil {
f.set = map[int]map[string]string{}
}
f.set[vmid] = p
return "", nil
}
func (f *fakeSwapOps) WaitTask(context.Context, string, proxmox.WaitOptions) (proxmox.TaskStatus, error) {
return proxmox.TaskStatus{}, nil
}
func TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM(t *testing.T) {
f := &fakeSwapOps{cfg: map[int]proxmox.GuestConfig{
9201: {Memory: 4096, Swap: 512}, // the measured box: raised
9202: {Memory: 2048, Swap: 4096}, // already higher: never lowered
9203: {Memory: 6144, Swap: 0}, // none: raised
9204: {Memory: 4096, Swap: 512}, // the write fails: logged, the others still done
}, failSet: map[int]bool{9204: true}}
n, err := EnsureGuestSwap(context.Background(), f, "felhom", slog.New(slog.NewTextHandler(io.Discard, nil)))
if err != nil {
t.Fatal(err)
}
if n != 2 {
t.Fatalf("raised %d, want 2 (9201, 9203)", n)
}
if f.set[9201]["swap"] != "4096" || f.set[9203]["swap"] != "6144" {
t.Fatalf("writes = %v", f.set)
}
if _, touched := f.set[9202]; touched {
t.Fatal("a swap above the RAM was lowered")
}
if len(f.set[9201]) != 1 {
t.Fatalf("the swap raise wrote more than the swap: %v", f.set[9201])
}
}
+362
View File
@@ -0,0 +1,362 @@
package localapi
import (
"context"
"errors"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
)
// ── POST /host/poweroff (R-998, documentation/architecture/14-host-power-off.md) ──────────────────────
//
// The household's „A gép kikapcsolása": shut THIS guest down cleanly, then power the whole host off. The
// first local-API route whose effect reaches beyond the caller's own guest — a compromised controller can
// now switch the box off, and nobody can switch it on remotely. The mitigations are the limits below:
//
// - refused in the first 10 minutes after boot (no power-off loop);
// - at most ONE accepted request per boot (poweroff.State, persisted, keyed by the boot);
// - refused while a backup, restore test, OS update step, controller update, drive format, NAS verify
// or escrow ceremony runs, or a vzdump of the guest is in flight — each named in the 409;
// - refused while ANY OTHER guest or VM runs on the host (security review 2026-10-13): one guest's
// controller must not switch off another household's guest, a scratch guest or a test VM (demo-hp
// carries the Tester 1 VM). Read from /proc (`lxc-start -n N`, `kvm -id N`), fail closed;
// - the heavy-operation gate is TAKEN on acceptance and never given back (the box is going down), so no
// backup, restore test or OS step can start between the checks and the power-off (security review).
// Released only if the power-off command itself fails.
//
// Every check fails CLOSED: an unwired gate, an unreadable task list or /proc refuses.
// On acceptance the record is written FIRST, the hub is given a report carrying it (bounded wait), and
// only then: guest shutdown (time-limited), host power-off. Pinned by hostpoweroff_test.go.
// HostPowerer is the privileged half: both run as fixed sudo vectors (configs/felhom-agent.sudoers).
type HostPowerer interface {
ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error
PowerOffHost(ctx context.Context) error
}
// HostPowerOffConfig wires the route. A nil config leaves it answering 503.
type HostPowerOffConfig struct {
State *poweroff.State
Power HostPowerer
ReportNow func(ctx context.Context) bool // push a report carrying the record; true = the hub accepted it
Uptime func() (time.Duration, bool) // time since host boot; false = unknown (refuse)
Running func() ([]int, error) // vmids of every running guest and VM on the host; default /proc
MinUptime time.Duration // default 10 min
GuestStop time.Duration // pct shutdown --timeout; default 120 s
ReportMax time.Duration // longest wait for the hub; default 60 s
JobWait time.Duration // longest wait for a job that started after acceptance; default 15 min
JobPoll time.Duration // how often that wait looks again; default 10 s
Now func() time.Time
}
// Refusal codes — the controller maps each to a household sentence. Stable wire values.
const (
PowerOffTooSoon = "too_soon_after_boot"
PowerOffAlreadyAsked = "already_requested_this_boot"
PowerOffBackup = "backup_running"
PowerOffRestoreTest = "restore_test_running"
PowerOffOSUpdate = "os_update_running"
PowerOffCtrlUpdate = "controller_update_running"
PowerOffDrive = "drive_operation_running"
PowerOffEscrow = "escrow_ceremony_running"
PowerOffBusy = "host_busy"
PowerOffOtherGuests = "other_guests_running"
PowerOffUnknown = "state_unknown"
)
// SetHostPowerOff enables POST /host/poweroff.
func (s *Server) SetHostPowerOff(c HostPowerOffConfig) {
if c.MinUptime == 0 {
c.MinUptime = 10 * time.Minute
}
if c.GuestStop == 0 {
c.GuestStop = 120 * time.Second
}
if c.ReportMax == 0 {
c.ReportMax = 60 * time.Second
}
if c.Now == nil {
c.Now = time.Now
}
if c.JobWait == 0 {
c.JobWait = 15 * time.Minute
}
if c.JobPoll == 0 {
c.JobPoll = 10 * time.Second
}
if c.Running == nil {
c.Running = func() ([]int, error) { return runningGuestsFromProc("/proc") }
}
s.powerOff = &c
}
type hostPowerOffRequest struct {
VMID int `json:"vmid"`
}
func (s *Server) handleHostPowerOff(w http.ResponseWriter, r *http.Request, vmid int) {
c := s.powerOff
if c == nil || c.State == nil || c.Power == nil {
writeErr(w, http.StatusServiceUnavailable, "host power-off not configured on this host")
return
}
if r.ContentLength != 0 {
var req hostPowerOffRequest
if !decodeBody(w, r, &req) {
return
}
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
return
}
}
// One decision at a time: two requests racing must not both pass the guards before either marks.
s.powerOffMu.Lock()
defer s.powerOffMu.Unlock()
if code, detail := s.powerOffBlocker(r.Context(), vmid); code != "" {
s.logger.Info("local-api: host power-off refused", "vmid", vmid, "reason", code, "detail", detail)
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": detail}, "host power-off refused: "+code)
return
}
// Take the heavy-operation gate for good: nothing heavy may start between these checks and the power-off.
release, busy, free := s.inFlight.TryAcquire("host-poweroff")
if !free {
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": PowerOffBusy, "detail": busy}, "host power-off refused: "+PowerOffBusy)
return
}
rec, err := c.State.Mark(c.Now(), vmid)
if err != nil {
release()
code := PowerOffUnknown
if errors.Is(err, poweroff.ErrAlreadyThisBoot) {
code = PowerOffAlreadyAsked
}
s.logger.Warn("local-api: host power-off refused (record)", "vmid", vmid, "reason", code, "err", err)
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": err.Error()}, "host power-off refused: "+code)
return
}
s.logger.Warn("local-api: HOST POWER-OFF accepted (household request)", "vmid", vmid,
"requested_at", rec.RequestedAt.Format(time.RFC3339))
base := s.baseCtx
if base == nil {
base = context.Background()
}
go s.runHostPowerOff(base, vmid, release)
writeStatus(w, http.StatusAccepted, true, map[string]any{
"vmid": vmid, "powering_off": true, "requested_at": rec.RequestedAt.Format(time.RFC3339),
}, "")
}
// runHostPowerOff: hub first (bounded), then the guest (time-limited), then the host. A failed guest
// shutdown does not stop the power-off — the household asked for it, and Proxmox's own shutdown stops
// any guest still running (pve-guests) before the host goes down.
func (s *Server) runHostPowerOff(base context.Context, vmid int, release func()) {
c := s.powerOff
if c.ReportNow != nil {
ctx, cancel := context.WithTimeout(base, c.ReportMax)
ok := c.ReportNow(ctx)
cancel()
s.logger.Warn("host-poweroff: step 1/3 — hub told", "vmid", vmid, "hub_accepted", ok)
}
if !s.waitJobsClear(base, vmid, "before the guest shutdown") {
release()
return
}
start := time.Now()
ctx, cancel := context.WithTimeout(base, c.GuestStop+60*time.Second)
err := c.Power.ShutdownGuest(ctx, vmid, c.GuestStop)
cancel()
if err != nil {
s.logger.Error("host-poweroff: step 2/3 — guest shutdown failed; powering off anyway", "vmid", vmid,
"took", time.Since(start).Round(time.Second), "err", err)
} else {
s.logger.Warn("host-poweroff: step 2/3 — guest stopped", "vmid", vmid, "took", time.Since(start).Round(time.Second))
}
if !s.waitJobsClear(base, vmid, "before the host power-off") {
release()
return
}
s.logger.Warn("host-poweroff: step 3/3 — powering the host off")
ctx, cancel = context.WithTimeout(base, 60*time.Second)
defer cancel()
if err := c.Power.PowerOffHost(ctx); err != nil {
s.logger.Error("host-poweroff: host power-off command failed — heavy-operation gate released", "err", err)
release()
}
}
// powerOffBlocker returns the first reason to refuse ("" = none) and a detail for the log/operator.
func (s *Server) powerOffBlocker(ctx context.Context, vmid int) (string, string) {
c := s.powerOff
if c.State.Current() != nil {
return PowerOffAlreadyAsked, "a power-off was already accepted in this boot"
}
if c.Uptime == nil {
return PowerOffUnknown, "uptime not wired"
}
up, ok := c.Uptime()
if !ok {
return PowerOffUnknown, "cannot read the host uptime"
}
if up < c.MinUptime {
return PowerOffTooSoon, "host up " + up.Round(time.Second).String() + ", minimum " + c.MinUptime.String()
}
if s.inFlight == nil || s.staleLock == nil {
return PowerOffUnknown, "the heavy-operation gate or the Proxmox task reader is not wired"
}
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
return code, detail
}
if busy := s.inFlight.Busy(); busy != "" {
switch {
case strings.HasPrefix(busy, "backup:"):
return PowerOffBackup, busy
case busy == "restore-test":
return PowerOffRestoreTest, busy
case strings.HasPrefix(busy, "os-"):
return PowerOffOSUpdate, busy
default:
return PowerOffBusy, busy
}
}
return s.jobBlocker(ctx, vmid)
}
// jobBlocker is the part of the checks that can change after acceptance — a backup job, a controller update, a drive
// or NAS job, an escrow ceremony, a Proxmox vzdump. runHostPowerOff asks it again before each step (security review
// 2026-10-13: the checks ran only once). The heavy-operation gate is not here: the power-off holds it.
func (s *Server) jobBlocker(ctx context.Context, vmid int) (string, string) {
s.jobsMu.Lock()
for k, j := range s.jobs {
if j != nil && backupInFlight(j.Phase) {
s.jobsMu.Unlock()
return PowerOffBackup, "backup job " + j.JobID + " on target " + k.target
}
}
s.jobsMu.Unlock()
s.swapMu.Lock()
for v, on := range s.swapInFlight {
if on {
s.swapMu.Unlock()
return PowerOffCtrlUpdate, "controller swap in flight for guest " + strconv.Itoa(v)
}
}
s.swapMu.Unlock()
if s.formatJobs != nil {
if j := s.formatJobs.get(); j != nil && j.Phase == formatPhaseRunning {
return PowerOffDrive, "drive format running"
}
}
s.netVerifyMu.Lock()
nv := s.netVerifyCur != nil && s.netVerifyCur.Phase == netVerifyPhaseRunning
s.netVerifyMu.Unlock()
if nv {
return PowerOffDrive, "network storage verify running"
}
s.escrowMu.Lock()
esc := s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning
s.escrowMu.Unlock()
if esc {
return PowerOffEscrow, "escrow ceremony running"
}
vz, err := s.staleLock.BackupRunning(ctx, vmid)
if err != nil {
return PowerOffUnknown, "cannot list running Proxmox tasks: " + err.Error()
}
if vz {
return PowerOffBackup, "vzdump running for guest " + strconv.Itoa(vmid)
}
return "", ""
}
// runningGuestsFromProc lists the vmids of running containers (`lxc-start … -n N`) and VMs (`kvm -id N`) from
// the process table — readable by the non-root agent, unlike /run/qemu-server (measured on demo-hp 2026-10-13).
// An unreadable /proc is an error (fail closed); an unreadable single process is skipped (it exited).
func runningGuestsFromProc(procDir string) ([]int, error) {
ents, err := os.ReadDir(procDir)
if err != nil {
return nil, err
}
seen := map[int]bool{}
var out []int
for _, e := range ents {
if _, err := strconv.Atoi(e.Name()); err != nil {
continue
}
raw, err := os.ReadFile(filepath.Join(procDir, e.Name(), "cmdline"))
if err != nil || len(raw) == 0 {
continue
}
args := strings.Split(strings.TrimRight(string(raw), "\x00"), "\x00")
flag := ""
switch filepath.Base(args[0]) {
case "lxc-start":
flag = "-n"
case "kvm", "qemu-system-x86_64":
flag = "-id"
default:
continue
}
for i := 1; i+1 < len(args); i++ {
if args[i] == flag {
if v, err := strconv.Atoi(args[i+1]); err == nil && !seen[v] {
seen[v] = true
out = append(out, v)
}
}
}
}
return out, nil
}
// otherGuestsBlocker refuses while any guest or VM other than the caller's runs (or the list cannot be read).
func (s *Server) otherGuestsBlocker(vmid int) (string, string) {
running, err := s.powerOff.Running()
if err != nil {
return PowerOffUnknown, "cannot list the running guests: " + err.Error()
}
var others []string
for _, v := range running {
if v != vmid {
others = append(others, strconv.Itoa(v))
}
}
if len(others) > 0 {
return PowerOffOtherGuests, "other guests or VMs running: " + strings.Join(others, ", ")
}
return "", ""
}
// waitJobsClear waits until jobBlocker finds nothing, at most JobWait. false = still busy (or unreadable) at the end:
// the power-off is CANCELLED — the box stays on, the gate is released, the log says why. Better a box that did not
// switch off than a backup, update or format cut in the middle.
func (s *Server) waitJobsClear(base context.Context, vmid int, when string) bool {
c := s.powerOff
ctx, cancel := context.WithTimeout(base, c.JobWait)
defer cancel()
for {
// Another guest or VM started after acceptance (security review 2026-10-13): no waiting — cancel at once.
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
s.logger.Error("host-poweroff: CANCELLED — another guest is running now; the box stays on", "when", when,
"reason", code, "detail", detail)
return false
}
code, detail := s.jobBlocker(ctx, vmid)
if code == "" {
return true
}
s.logger.Warn("host-poweroff: waiting — a job started after the request", "when", when, "reason", code, "detail", detail)
select {
case <-ctx.Done():
s.logger.Error("host-poweroff: CANCELLED — still busy after the wait; the box stays on", "when", when,
"reason", code, "detail", detail, "waited", c.JobWait)
return false
case <-time.After(c.JobPoll):
}
}
}
+353
View File
@@ -0,0 +1,353 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"net/http"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// fakePower records the power-off steps in order.
type fakePower struct {
mu sync.Mutex
steps []string
}
func (f *fakePower) ShutdownGuest(_ context.Context, vmid int, timeout time.Duration) error {
f.mu.Lock()
defer f.mu.Unlock()
f.steps = append(f.steps, "shutdown:"+itoaT(vmid)+":"+timeout.String())
return nil
}
func (f *fakePower) PowerOffHost(context.Context) error {
f.mu.Lock()
defer f.mu.Unlock()
f.steps = append(f.steps, "poweroff")
return nil
}
func (f *fakePower) got() []string {
f.mu.Lock()
defer f.mu.Unlock()
return append([]string(nil), f.steps...)
}
func itoaT(n int) string { b, _ := json.Marshal(n); return string(b) }
type fakeVzdump struct{ running bool }
func (f fakeVzdump) Guests(context.Context) ([]proxmox.Guest, error) { return nil, nil }
func (f fakeVzdump) Lock(context.Context, int) (string, bool, error) { return "", true, nil }
func (f fakeVzdump) BackupRunning(context.Context, int) (bool, error) { return f.running, nil }
func (f fakeVzdump) Unlock(context.Context, int) error { return nil }
func (f fakeVzdump) Start(context.Context, int) error { return nil }
func (f fakeVzdump) HasVzdumpSnapshot(context.Context, int) (bool, error) { return false, nil }
func (f fakeVzdump) DeleteVzdumpSnapshot(context.Context, int) error { return nil }
type powerRig struct {
srv *Server
h http.Handler
power *fakePower
state *poweroff.State
report *[]string
}
func newPowerRig(t *testing.T, uptime time.Duration) *powerRig {
t.Helper()
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.inFlight = &backup.InFlight{}
srv.staleLock = fakeVzdump{}
p := &fakePower{}
st := poweroff.New(filepath.Join(t.TempDir(), "poweroff.json"), func() string { return "1700000000" })
var reports []string
srv.SetHostPowerOff(HostPowerOffConfig{
State: st,
Power: p,
ReportNow: func(context.Context) bool {
// The record must already exist when the hub is told — that is the point of telling it.
if st.Current() == nil {
reports = append(reports, "report-without-record")
} else if len(p.got()) != 0 {
reports = append(reports, "report-after-"+p.got()[0])
} else {
reports = append(reports, "report")
}
return true
},
Uptime: func() (time.Duration, bool) { return uptime, true },
Running: func() ([]int, error) { return []int{8200}, nil },
Now: func() time.Time { return testNow },
})
return &powerRig{srv: srv, h: srv.Handler(), power: p, state: st, report: &reports}
}
func (r *powerRig) waitSteps(t *testing.T, n int) []string {
t.Helper()
for i := 0; i < 200 && len(r.power.got()) < n; i++ {
time.Sleep(time.Millisecond)
}
return r.power.got()
}
func reasonOf(t *testing.T, body []byte) string {
t.Helper()
var resp struct {
Data struct {
Reason string `json:"reason"`
} `json:"data"`
}
if err := json.Unmarshal(body, &resp); err != nil {
t.Fatalf("body %s: %v", body, err)
}
return resp.Data.Reason
}
// The whole path: 202, the record written, the hub told FIRST, then guest shutdown (120 s), then host off.
func TestHostPowerOff_AcceptedOrder(t *testing.T) {
r := newPowerRig(t, time.Hour)
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusAccepted {
t.Fatalf("got %d want 202 (%s)", w.Code, w.Body.String())
}
steps := r.waitSteps(t, 2)
want := []string{"shutdown:8200:2m0s", "poweroff"}
if len(steps) != 2 || steps[0] != want[0] || steps[1] != want[1] {
t.Fatalf("steps = %v, want %v", steps, want)
}
// The heavy-operation gate stays taken: no backup can start between the checks and the power-off.
if _, busy, ok := r.srv.inFlight.TryAcquire("backup:primary"); ok || busy != "host-poweroff" {
t.Fatalf("after acceptance a backup could take the gate (ok=%v busy=%q)", ok, busy)
}
if len(*r.report) != 1 || (*r.report)[0] != "report" {
t.Fatalf("hub report = %v, want one report carrying the record", *r.report)
}
if rec := r.state.Current(); rec == nil || rec.VMID != 8200 {
t.Fatalf("record = %+v", rec)
}
}
// Refused during a backup — the shared heavy-operation gate held by a backup tier.
func TestHostPowerOff_RefusedDuringBackup(t *testing.T) {
r := newPowerRig(t, time.Hour)
release, _, ok := r.srv.inFlight.TryAcquire("backup:primary")
if !ok {
t.Fatal("could not hold the gate")
}
defer release()
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
}
time.Sleep(5 * time.Millisecond)
if s := r.power.got(); len(s) != 0 {
t.Fatalf("power steps ran during a backup: %v", s)
}
if r.state.Current() != nil {
t.Fatal("a refused request must not use up the boot's one request")
}
// The same guard names a restore test and an OS step.
release()
for gate, want := range map[string]string{"restore-test": PowerOffRestoreTest, "os-pve-step": PowerOffOSUpdate} {
rel, _, _ := r.srv.inFlight.TryAcquire(gate)
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
rel()
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != want {
t.Fatalf("%s: got %d %s, want 409 %s", gate, w.Code, w.Body.String(), want)
}
}
}
// A vzdump of the guest in flight (a Proxmox task, not the agent's own gate) refuses too.
func TestHostPowerOff_RefusedDuringVzdump(t *testing.T) {
r := newPowerRig(t, time.Hour)
r.srv.staleLock = fakeVzdump{running: true}
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
}
}
// One accepted request per boot: the second is refused and nothing runs twice.
func TestHostPowerOff_SecondRequestSameBootRefused(t *testing.T) {
r := newPowerRig(t, time.Hour)
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
t.Fatalf("first: %d", w.Code)
}
r.waitSteps(t, 2)
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffAlreadyAsked {
t.Fatalf("second: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffAlreadyAsked)
}
time.Sleep(5 * time.Millisecond)
if s := r.power.got(); len(s) != 2 {
t.Fatalf("steps = %v, want exactly one run", s)
}
}
// Refused in the first 10 minutes after boot; an unknown uptime refuses too (fail closed).
func TestHostPowerOff_TooSoonAfterBoot(t *testing.T) {
r := newPowerRig(t, 9*time.Minute)
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffTooSoon {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffTooSoon)
}
r.srv.powerOff.Uptime = func() (time.Duration, bool) { return 0, false }
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
t.Fatalf("unknown uptime: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
}
}
// Not configured → 503; another guest's vmid in the body → 403; no token → 401.
func TestHostPowerOff_ScopeAndConfig(t *testing.T) {
plain := newTestServer(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
if w := do(t, plain, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusServiceUnavailable {
t.Fatalf("unconfigured: %d", w.Code)
}
r := newPowerRig(t, time.Hour)
if w := do(t, r.h, "POST", "/host/poweroff", "A", `{"vmid":9300}`); w.Code != http.StatusForbidden {
t.Fatalf("cross-guest: %d", w.Code)
}
if w := do(t, r.h, "POST", "/host/poweroff", "", ""); w.Code != http.StatusUnauthorized {
t.Fatalf("no token: %d", w.Code)
}
if r.state.Current() != nil || len(r.power.got()) != 0 {
t.Fatal("a refused request changed state")
}
}
// Another guest or VM on the host refuses: one controller must not switch off someone else's guest (security
// review 2026-10-13; demo-hp runs a scratch guest and the Tester 1 VM). An unreadable process list refuses too.
func TestHostPowerOff_RefusedWhileOtherGuestsRun(t *testing.T) {
r := newPowerRig(t, time.Hour)
r.srv.powerOff.Running = func() ([]int, error) { return []int{8200, 9202, 341}, nil }
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffOtherGuests {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffOtherGuests)
}
r.srv.powerOff.Running = func() ([]int, error) { return nil, errors.New("no /proc") }
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
t.Fatalf("unreadable: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
}
if len(r.power.got()) != 0 || r.state.Current() != nil {
t.Fatal("a refused request changed state")
}
}
// Fail closed: an unwired vzdump reader refuses instead of skipping the check.
func TestHostPowerOff_FailsClosedWithoutTaskReader(t *testing.T) {
r := newPowerRig(t, time.Hour)
r.srv.staleLock = nil
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
}
}
// The /proc reader finds containers and VMs by their real command lines (as measured on demo-hp).
func TestRunningGuestsFromProc(t *testing.T) {
dir := t.TempDir()
write := func(pid, cmd string) {
_ = os.MkdirAll(filepath.Join(dir, pid), 0o755)
_ = os.WriteFile(filepath.Join(dir, pid, "cmdline"), []byte(strings.ReplaceAll(cmd, " ", "\x00")+"\x00"), 0o644)
}
write("1927", "/usr/bin/lxc-start -F -n 9201")
write("1900", "/usr/bin/lxc-start -F -n 9202")
write("1823", "/usr/bin/kvm -id 341 -name night1004-tester1 -no-shutdown")
write("974", "/usr/libexec/lxc/lxc-monitord --daemon")
write("50", "/usr/bin/bash -c echo -n 5")
_ = os.MkdirAll(filepath.Join(dir, "self"), 0o755)
got, err := runningGuestsFromProc(dir)
if err != nil {
t.Fatal(err)
}
sort.Ints(got)
if len(got) != 3 || got[0] != 341 || got[1] != 9201 || got[2] != 9202 {
t.Fatalf("got %v, want [341 9201 9202]", got)
}
if _, err := runningGuestsFromProc(filepath.Join(dir, "missing")); err == nil {
t.Fatal("an unreadable /proc must be an error")
}
}
// switchVzdump is a vzdump reader the test can flip while the power-off runs.
type switchVzdump struct {
fakeVzdump
on *atomic.Bool
}
func (f switchVzdump) BackupRunning(context.Context, int) (bool, error) { return f.on.Load(), nil }
// A job that starts AFTER acceptance (security review 2026-10-13): the power-off waits; if the job is still running
// when the wait ends, the power-off is cancelled — no guest shutdown, no host power-off, the gate given back.
func TestHostPowerOff_JobStartedAfterAcceptance(t *testing.T) {
// (a) it clears during the wait → the power-off goes ahead after it
r := newPowerRig(t, time.Hour)
var on atomic.Bool
r.srv.staleLock = switchVzdump{on: &on}
r.srv.powerOff.JobWait, r.srv.powerOff.JobPoll = 2*time.Second, 2*time.Millisecond
r.srv.powerOff.ReportNow = func(context.Context) bool { on.Store(true); return true } // a vzdump starts now
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
}
time.Sleep(30 * time.Millisecond)
if s := r.power.got(); len(s) != 0 {
t.Fatalf("a step ran while the vzdump ran: %v", s)
}
on.Store(false)
if s := r.waitSteps(t, 2); len(s) != 2 {
t.Fatalf("after the vzdump ended the steps = %v, want shutdown + poweroff", s)
}
// (b) it never clears → cancelled, nothing switched off, the heavy gate free again
r2 := newPowerRig(t, time.Hour)
var on2 atomic.Bool
r2.srv.staleLock = switchVzdump{on: &on2}
r2.srv.powerOff.JobWait, r2.srv.powerOff.JobPoll = 40*time.Millisecond, 2*time.Millisecond
r2.srv.powerOff.ReportNow = func(context.Context) bool { on2.Store(true); return true }
if w := do(t, r2.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
t.Fatalf("accept: %d", w.Code)
}
time.Sleep(150 * time.Millisecond)
if s := r2.power.got(); len(s) != 0 {
t.Fatalf("cancelled power-off still ran: %v", s)
}
if busy := r2.srv.inFlight.Busy(); busy != "" {
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
}
}
// A guest or VM started AFTER acceptance cancels the power-off at once (security review 2026-10-13).
func TestHostPowerOff_OtherGuestStartedAfterAcceptance(t *testing.T) {
r := newPowerRig(t, time.Hour)
var started atomic.Bool
r.srv.powerOff.Running = func() ([]int, error) {
if started.Load() {
return []int{8200, 341}, nil
}
return []int{8200}, nil
}
r.srv.powerOff.ReportNow = func(context.Context) bool { started.Store(true); return true } // a VM starts now
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
}
time.Sleep(50 * time.Millisecond)
if s := r.power.got(); len(s) != 0 {
t.Fatalf("power-off ran although a VM started: %v", s)
}
if busy := r.srv.inFlight.Busy(); busy != "" {
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
}
}
+207
View File
@@ -0,0 +1,207 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path"
"path/filepath"
"regexp"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// The internal data place („Belső tárhely", agent v0.158.0) — felhom.eu architecture/16-storage-physical-identity.md
// §5, §7. On a one-disk box the root helper configs/felhom-internal-place makes a second thin volume in local-lvm
// (<vg>/felhom-belso, ext4, label felhom-belso). The installer (a new box) or the signed add_internal_place job (an old
// box) then writes InternalPlaceRecord to the agent's state dir. From then on the agent treats the volume exactly like
// an enrolled drive: a Felhom mount unit by fs-UUID at /mnt/belso (EnsureMount), the `enrolled` intent, and — when the
// controller asks with its own token (POST /disks/guest-attach) — the bind at /mnt/felhom-drives/belso. So it is a
// HOST mount under the guest's mp8 parent: not in the guest config, not in vzdump, not in PBS (`16` §5).
//
// What it is NOT: a drive the household can format, wipe, eject or decommission. Its device is a device-mapper volume
// on the system disk, so the role gate (roleForMountPath → system) and the format path (ValidateBlockDevice refuses
// dm) refuse all four — pinned by internalplace_test.go. Its identity is the system disk's (build A), so a Tier-2 copy
// between it and sys_drive is never a second copy.
const (
InternalPlaceName = "belso"
InternalPlaceWhere = "/mnt/belso"
// InternalPlaceFile is the record's name inside the agent's state dir.
InternalPlaceFile = "internal-place.json"
)
var reFSUUID = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
// InternalPlaceRecord is the state file the maker writes once the volume exists and is formatted.
type InternalPlaceRecord struct {
FSUUID string `json:"fs_uuid"`
GiB int `json:"gib,omitempty"`
MadeBy string `json:"made_by,omitempty"` // "installer" | "signed-job"
MadeAt string `json:"made_at,omitempty"`
}
// ReadInternalPlace reads the record; ok=false when there is none (the common case: a box with no internal place).
func ReadInternalPlace(file string) (InternalPlaceRecord, bool, error) {
var rec InternalPlaceRecord
b, err := os.ReadFile(file)
if errors.Is(err, os.ErrNotExist) {
return rec, false, nil
}
if err != nil {
return rec, false, err
}
if err := json.Unmarshal(b, &rec); err != nil {
return rec, false, fmt.Errorf("internal place record %s: %w", file, err)
}
if !reFSUUID.MatchString(rec.FSUUID) {
return rec, false, fmt.Errorf("internal place record %s: fs_uuid %q is not a filesystem UUID", file, rec.FSUUID)
}
return rec, true, nil
}
// WriteInternalPlace writes the record atomically (0600).
func WriteInternalPlace(file string, rec InternalPlaceRecord) error {
if !reFSUUID.MatchString(rec.FSUUID) {
return fmt.Errorf("internal place: fs_uuid %q is not a filesystem UUID", rec.FSUUID)
}
b, err := json.MarshalIndent(rec, "", " ")
if err != nil {
return err
}
tmp := file + ".tmp"
if err := os.WriteFile(tmp, append(b, '\n'), 0o600); err != nil {
return err
}
return os.Rename(tmp, file)
}
// internalPlaceState memoises what EnsureInternalPlace last saw, so the 20-second tick logs a state CHANGE once and
// does no work on the steady state.
type internalPlaceState struct {
mu sync.Mutex
done string // the fs-UUID that is mounted and recorded enrolled
lastWhy string // the last reason it could not be ensured (logged once per change)
}
// internalPlaceUUID returns the configured internal place's fs-UUID, or "" when this box has none (or the record is
// unreadable — logged by EnsureInternalPlace).
func (s *Server) internalPlaceUUID() string {
if s.internalPlaceFile == "" {
return ""
}
rec, ok, err := ReadInternalPlace(s.internalPlaceFile)
if err != nil || !ok {
return ""
}
return rec.FSUUID
}
// isInternalPlace reports whether a host path is this box's internal place: its raw mount (/mnt/belso) or its bind
// under the guests' shared parent (/mnt/felhom-drives/belso — the path POST /disks/same-disk judges).
func (s *Server) isInternalPlace(hostPath string) bool {
p := path.Clean(hostPath)
return (p == InternalPlaceWhere || p == StableParentDir+"/"+InternalPlaceName) && s.internalPlaceUUID() != ""
}
// EnsureInternalPlace mounts the internal place on the host at /mnt/belso by fs-UUID and records it `enrolled`,
// when this box has one. Idempotent and cheap on the steady state; safe on the 20-second tick. It never formats,
// never unmounts, and never mounts anything at /mnt/belso that is not the recorded UUID: a household drive that
// happens to be named „belso" keeps its mount, and the internal place waits (logged once).
func (s *Server) EnsureInternalPlace(ctx context.Context) {
if s.internalPlaceFile == "" {
return
}
rec, ok, err := ReadInternalPlace(s.internalPlaceFile)
if err != nil {
s.internalPlaceNote("record unreadable: " + err.Error())
return
}
if !ok {
return
}
uuid := rec.FSUUID
st := &s.ipState
st.mu.Lock()
done := st.done == uuid
st.mu.Unlock()
// What is at /mnt/belso now, if anything?
mountedUUID, mounted := "", false
if mounts, merr := s.hostReader().Mounts(); merr == nil {
for _, m := range mounts {
if path.Clean(m.MountPoint) == InternalPlaceWhere {
mounted = true
if u, okU := s.hostReader().ResolveUUID(m.Device); okU {
mountedUUID = u
}
}
}
}
if mounted && mountedUUID != "" && mountedUUID != uuid {
s.internalPlaceNote("another filesystem (uuid " + mountedUUID + ") is mounted at " + InternalPlaceWhere + " — the internal place waits")
return
}
// A Felhom unit for /mnt/belso that names another UUID (a household drive called „belso", unplugged): never
// overwrite it — that would silently take the drive's name and mount point.
if s.driveTargets != nil {
if drives, derr := s.driveTargets.Known(ctx); derr == nil {
for _, d := range drives {
if path.Clean(d.MountPath) == InternalPlaceWhere && d.UUID != "" && d.UUID != uuid {
s.internalPlaceNote("a drive unit for " + InternalPlaceWhere + " names another uuid (" + d.UUID + ") — the internal place waits")
return
}
}
}
}
if mounted && mountedUUID == uuid {
if !done {
s.recordIntent(ctx, InternalPlaceWhere, "enrolled")
st.mu.Lock()
st.done, st.lastWhy = uuid, ""
st.mu.Unlock()
s.logger.Info("internal place: mounted and enrolled", "where", InternalPlaceWhere, "fs_uuid", uuid, "gib", rec.GiB)
}
return
}
if s.disks == nil {
s.internalPlaceNote("disk management is not configured on this host")
return
}
if s.resolveStorageDevice != nil {
if _, rerr := s.resolveStorageDevice("uuid:" + uuid); rerr != nil {
s.internalPlaceNote("its volume is not present (uuid " + uuid + ")")
return
}
}
start := time.Now()
if err := s.disks.EnsureMount(ctx, storage.MountSpec{Name: InternalPlaceName, UUID: uuid, Where: InternalPlaceWhere, FSType: "ext4"}); err != nil {
s.internalPlaceNote("mount failed: " + err.Error())
return
}
s.recordIntent(ctx, InternalPlaceWhere, "enrolled")
st.mu.Lock()
st.done, st.lastWhy = uuid, ""
st.mu.Unlock()
s.logger.Info("internal place: mounted on the host", "where", InternalPlaceWhere, "fs_uuid", uuid, "gib", rec.GiB,
"duration_ms", time.Since(start).Milliseconds())
}
func (s *Server) internalPlaceNote(why string) {
st := &s.ipState
st.mu.Lock()
changed := st.lastWhy != why
st.lastWhy, st.done = why, ""
st.mu.Unlock()
if changed {
s.logger.Warn("internal place: not ensured", "why", why)
} else {
s.logger.Debug("internal place: still not ensured", "why", why)
}
}
// InternalPlaceFileIn is the record's path inside a state dir.
func InternalPlaceFileIn(stateDir string) string { return filepath.Join(stateDir, InternalPlaceFile) }
+249
View File
@@ -0,0 +1,249 @@
package localapi
import (
"context"
"errors"
"io"
"log/slog"
"net/http"
"os/exec"
"path/filepath"
"runtime"
"sync"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// The internal data place (felhom.eu architecture/16 §5, agent v0.158.0): mounted like an enrolled drive, but never
// at the cost of a household drive, never formatted, ejected or decommissioned, and the same disk as sys_drive.
const ipUUID = "0b6e4c5a-1d2e-4f3a-9b8c-7d6e5f4a3b2c"
const ipDev = "/dev/mapper/pve-felhom--belso"
// ipHost is a HostReader whose mount table the test changes, and whose by-uuid map it controls.
type ipHost struct {
mu sync.Mutex
mounts []storage.Mount
uuids map[string]string // device → fs-UUID
}
func (h *ipHost) Mounts() ([]storage.Mount, error) {
h.mu.Lock()
defer h.mu.Unlock()
return append([]storage.Mount(nil), h.mounts...), nil
}
func (h *ipHost) ResolveUUID(dev string) (string, bool) { u, ok := h.uuids[dev]; return u, ok }
func (h *ipHost) DeviceExists(string) bool { return true }
func (h *ipHost) Rotational(string) (bool, bool) { return false, false }
func (h *ipHost) Removable(string) (bool, bool) { return false, false }
func (h *ipHost) BlockSlaves(string) ([]string, bool) { return nil, false }
func newIPHost(extra ...storage.Mount) *ipHost {
return &ipHost{
// root on a whole-disk partition (as sysOnSDA): the system disk resolves, so a USB drive is user-data.
mounts: append([]storage.Mount{{Device: "/dev/sda1", MountPoint: "/", FSType: "ext4"}}, extra...),
uuids: map[string]string{ipDev: ipUUID, "/dev/sdb1": "99999999-aaaa-bbbb-cccc-dddddddddddd"},
}
}
type ipRig struct {
srv *Server
disks *fakeDiskOps
intent *fakeIntent
}
func newIPRig(t *testing.T, host *ipHost, known []storage.KnownTarget, record bool) ipRig {
t.Helper()
dir := t.TempDir()
file := filepath.Join(dir, InternalPlaceFile)
if record {
if err := WriteInternalPlace(file, InternalPlaceRecord{FSUUID: ipUUID, GiB: 183, MadeBy: "installer"}); err != nil {
t.Fatal(err)
}
}
d, in := &fakeDiskOps{}, newFakeIntent()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
Storage: fakeStorage{}, Tokens: staticTokens{"A": 8200}, Disks: d, DiskGate: &fakeGate{},
HostReader: host, Intent: in, DriveTargets: fakeKnownTargets{drives: known},
InternalPlaceFile: file, Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.resolveStorageDevice = func(id string) (string, error) {
if id == "uuid:"+ipUUID {
return "/dev/dm-7", nil
}
return "", errors.New("absent")
}
return ipRig{srv, d, in}
}
func TestInternalPlace_NoRecord_DoesNothing(t *testing.T) {
r := newIPRig(t, newIPHost(), nil, false)
r.srv.EnsureInternalPlace(context.Background())
if len(r.disks.mountCalls) != 0 {
t.Fatalf("no record: no mount may be made, got %+v", r.disks.mountCalls)
}
}
func TestInternalPlace_Record_MountsByUUIDAtMntBelso(t *testing.T) {
r := newIPRig(t, newIPHost(), nil, true)
r.srv.EnsureInternalPlace(context.Background())
if len(r.disks.mountCalls) != 1 {
t.Fatalf("want exactly one EnsureMount, got %+v", r.disks.mountCalls)
}
got := r.disks.mountCalls[0]
if got.UUID != ipUUID || got.Where != InternalPlaceWhere || got.FSType != "ext4" || got.Name != InternalPlaceName {
t.Fatalf("EnsureMount spec: %+v", got)
}
}
func TestInternalPlace_AlreadyMounted_EnrolledOnceNoRemount(t *testing.T) {
host := newIPHost(storage.Mount{Device: ipDev, MountPoint: InternalPlaceWhere, FSType: "ext4"})
r := newIPRig(t, host, nil, true)
r.srv.EnsureInternalPlace(context.Background())
r.srv.EnsureInternalPlace(context.Background())
if len(r.disks.mountCalls) != 0 {
t.Fatalf("already mounted: no EnsureMount, got %+v", r.disks.mountCalls)
}
if r.intent.Get("uuid:"+ipUUID) != storage.IntentEnrolled {
t.Fatalf("the internal place must be recorded enrolled")
}
}
// A household drive called „belso" keeps its mount point: the internal place never takes it.
// Positive control: TestInternalPlace_Record_MountsByUUIDAtMntBelso (the same record, /mnt/belso free).
func TestInternalPlace_NeverTakesAHouseholdDriveNamedBelso(t *testing.T) {
mounted := newIPHost(storage.Mount{Device: "/dev/sdb1", MountPoint: InternalPlaceWhere, FSType: "ext4"})
r := newIPRig(t, mounted, nil, true)
r.srv.EnsureInternalPlace(context.Background())
if len(r.disks.mountCalls) != 0 || r.intent.Get("uuid:"+ipUUID) != "" {
t.Fatalf("a foreign filesystem at /mnt/belso: nothing may be mounted or recorded, got %+v", r.disks.mountCalls)
}
unplugged := []storage.KnownTarget{{Name: "belso", UUID: "99999999-aaaa-bbbb-cccc-dddddddddddd", MountPath: InternalPlaceWhere, MountBacked: true}}
r2 := newIPRig(t, newIPHost(), unplugged, true)
r2.srv.EnsureInternalPlace(context.Background())
if len(r2.disks.mountCalls) != 0 {
t.Fatalf("an unplugged household drive's unit for /mnt/belso must not be overwritten, got %+v", r2.disks.mountCalls)
}
}
func TestInternalPlace_VolumeAbsent_NoMount(t *testing.T) {
r := newIPRig(t, newIPHost(), nil, true)
r.srv.resolveStorageDevice = func(string) (string, error) { return "", errors.New("absent") }
r.srv.EnsureInternalPlace(context.Background())
if len(r.disks.mountCalls) != 0 {
t.Fatalf("absent volume: no mount, got %+v", r.disks.mountCalls)
}
}
// The household cannot eject or decommission it: its device is a device-mapper volume on the system disk.
// Positive control: the same calls for a whole-disk USB drive (/dev/sdb1) are not refused by role.
func TestInternalPlace_EjectAndDecommissionRefused(t *testing.T) {
host := newIPHost(
storage.Mount{Device: ipDev, MountPoint: InternalPlaceWhere, FSType: "ext4"},
storage.Mount{Device: "/dev/sdb1", MountPoint: "/mnt/hdd_1", FSType: "ext4"},
)
r := newIPRig(t, host, nil, true)
h := r.srv.Handler()
for _, route := range []string{"/disks/eject", "/disks/decommission"} {
if w := do(t, h, "POST", route, "A", `{"where":"/mnt/belso"}`); w.Code != http.StatusForbidden {
t.Fatalf("%s /mnt/belso: want 403, got %d %s", route, w.Code, w.Body.String())
}
}
if w := do(t, h, "POST", "/disks/eject", "A", `{"where":"/mnt/hdd_1"}`); w.Code == http.StatusForbidden {
t.Fatalf("positive control: a USB drive's eject must not be refused by role, got %d %s", w.Code, w.Body.String())
}
for _, u := range r.disks.unmountCalls {
if u == InternalPlaceWhere {
t.Fatalf("the internal place must never be unmounted, got %+v", r.disks.unmountCalls)
}
}
}
// The consequence `16` §5 needs: a copy between the internal place and sys_drive is the SAME disk (build A then never
// counts it), and the place is reported as kind internal.
type ipIdentHost struct{ fakeIdentHost }
func (h *ipIdentHost) KernelName(dev string) (string, bool) {
if dev == ipDev {
return "dm-7", true
}
return h.fakeIdentHost.KernelName(dev)
}
func (h *ipIdentHost) Slaves(n string) []string {
if n == "dm-7" {
return []string{"dm-3"} // felhom-belso → the thin pool
}
if n == "dm-3" {
return []string{"sda3"} // the pool → the system disk's LVM partition
}
return nil
}
func TestInternalPlace_IsTheSystemDiskAndKindInternal(t *testing.T) {
ih := &ipIdentHost{fakeIdentHost{
virt: "none",
parent: map[string]string{"sda3": "sda"},
props: map[string]map[string]string{"sda": {"ID_BUS": "ata", "ID_MODEL": "SanDisk", "ID_SERIAL_SHORT": "1"}},
mounts: []storage.Mount{
{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"},
{Device: "/dev/mapper/pve-root", MountPoint: "/mnt/felhom-drives", FSType: "ext4"},
{Device: ipDev, MountPoint: InternalPlaceWhere, FSType: "ext4"},
{Device: ipDev, MountPoint: "/mnt/felhom-drives/belso", FSType: "ext4"},
},
}}
fg := &fakeGuestsCfg{mounts: map[int]map[string]string{8200: {
"mp0": "local-lvm:vm-8200-disk-1,mp=/var/lib/felhom,backup=1,size=100G",
"mp8": "/mnt/felhom-drives,mp=/mnt/felhom-drives",
}}}
dir := t.TempDir()
file := filepath.Join(dir, InternalPlaceFile)
if err := WriteInternalPlace(file, InternalPlaceRecord{FSUUID: ipUUID}); err != nil {
t.Fatal(err)
}
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0", Guests: fg, Backups: &fakeBackups{}, Store: &fakeStore{},
Storage: fakeStorage{targets: []hub.StorageTarget{provenSSD}}, Tokens: staticTokens{"A": 8200},
HostReader: fakeHostReader{mounts: ih.mounts}, Identity: &storage.IdentityResolver{Host: ih},
InternalPlaceFile: file, Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
got, code := sameDisk(t, srv.Handler(), "/mnt/felhom-drives/belso/felhom-data/jellyfin", "/mnt/sys_drive")
if code != http.StatusOK || got.Different || !got.Proven || got.Reason != "same_disk" {
t.Fatalf("internal place vs sys_drive: want same disk, proven; got %d %+v", code, got)
}
if got.A.Kind != storage.PlaceKindInternal {
t.Fatalf("kind: want %q, got %q", storage.PlaceKindInternal, got.A.Kind)
}
}
func TestInternalPlaceRecord_RefusesANonUUID(t *testing.T) {
file := filepath.Join(t.TempDir(), InternalPlaceFile)
if err := WriteInternalPlace(file, InternalPlaceRecord{FSUUID: "../../etc"}); err == nil {
t.Fatal("a non-UUID must not be written")
}
}
// The root helper's own tests (configs/test_felhom_internal_place.py — the ruled size table, never formatting what
// holds anything) run with the Go suite.
func TestInternalPlaceHelperPythonSuite(t *testing.T) {
py, err := exec.LookPath("python3")
if err != nil {
t.Skip("python3 not found")
}
_, file, _, _ := runtime.Caller(0)
script := filepath.Join(filepath.Dir(file), "..", "..", "configs", "test_felhom_internal_place.py")
out, err := exec.Command(py, script).CombinedOutput()
if err != nil {
t.Fatalf("helper suite failed: %v\n%s", err, out)
}
}
+89
View File
@@ -0,0 +1,89 @@
package localapi
import (
"context"
"errors"
"net/http"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
)
// LineTestReader is the R-1247 weekly upload line test (internal/linetest.Tester) as the local API reads it.
type LineTestReader interface {
Value() (mbps float64, at time.Time, ok bool)
}
// LineTestRequester is the R-1250 half: a due test the uplink never lets start, and a test on request. The Tester
// satisfies it; a reader without it answers waiting_for_quiet:false and 503 on the POST.
type LineTestRequester interface {
WaitingForQuiet() bool
RunNow(ctx context.Context) (mbps float64, runs []float64, err error)
}
// LineTestResponse is GET /host/line-test. Present=false: never measured, or the job is not wired (no tunnel, disabled,
// or an agent before the route answers 404 — the controller reads all three as „unknown").
type LineTestResponse struct {
Present bool `json:"present"`
Mbps float64 `json:"mbps,omitempty"`
MeasuredAt string `json:"measured_at,omitempty"` // RFC3339 UTC
Method string `json:"method,omitempty"` // "iperf3-tunnel": a 10-second upload to ep0 inside the tunnel
// WaitingForQuiet: a measurement is due and could start now, but the uplink has not been quiet for 10 minutes
// (R-1250). The controller with a seeding torrent app then holds its limit and POSTs /host/line-test/run.
WaitingForQuiet bool `json:"waiting_for_quiet,omitempty"`
}
// LineTestRunResponse is POST /host/line-test/run. OK=false with Refused: nothing ran (Refused is linetest.Refusal's
// code, or "server-busy"); OK=false without it: the runs failed.
type LineTestRunResponse struct {
OK bool `json:"ok"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Refused string `json:"refused,omitempty"`
Error string `json:"error,omitempty"`
}
// SetLineTest wires the weekly line test (main.go builds it after the local API server).
func (s *Server) SetLineTest(r LineTestReader) { s.lineTest = r }
func (s *Server) handleLineTest(w http.ResponseWriter, r *http.Request, vmid int) {
if s.lineTest == nil {
writeOK(w, LineTestResponse{Present: false})
return
}
var waiting bool
if q, ok := s.lineTest.(LineTestRequester); ok {
waiting = q.WaitingForQuiet()
}
mbps, at, ok := s.lineTest.Value()
if !ok {
writeOK(w, LineTestResponse{Present: false, WaitingForQuiet: waiting})
return
}
s.logger.Debug("local-api: /host/line-test served", "vmid", vmid, "mbps", mbps, "measured_at", at.UTC().Format(time.RFC3339), "waiting_for_quiet", waiting)
writeOK(w, LineTestResponse{Present: true, Mbps: mbps, MeasuredAt: at.UTC().Format(time.RFC3339), Method: "iperf3-tunnel", WaitingForQuiet: waiting})
}
// handleLineTestRun is POST /host/line-test/run (R-1250): a test on request, synchronous (about 55 s). The answer is
// always 200 with the outcome in the body, except 503 when no line test is wired.
func (s *Server) handleLineTestRun(w http.ResponseWriter, r *http.Request, vmid int) {
q, ok := s.lineTest.(LineTestRequester)
if s.lineTest == nil || !ok {
writeErr(w, http.StatusServiceUnavailable, "line test not configured")
return
}
s.logger.Info("local-api: line test requested", "vmid", vmid)
mbps, runs, err := q.RunNow(r.Context())
var ref *linetest.Refusal
switch {
case err == nil:
writeOK(w, LineTestRunResponse{OK: true, Mbps: mbps, RunsMbps: runs})
case errors.As(err, &ref):
s.logger.Info("local-api: line test request refused", "vmid", vmid, "reason", ref.Code, "detail", ref.Msg)
writeOK(w, LineTestRunResponse{Refused: ref.Code, Error: ref.Msg})
case errors.Is(err, linetest.ErrBusy):
writeOK(w, LineTestRunResponse{Refused: "server-busy", Error: err.Error()})
default:
writeOK(w, LineTestRunResponse{Error: err.Error()})
}
}
+116
View File
@@ -0,0 +1,116 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"log/slog"
"net/http/httptest"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
)
type fixedLine struct {
mbps float64
ok bool
}
func (f fixedLine) Value() (float64, time.Time, bool) {
return f.mbps, time.Date(2026, 10, 24, 6, 31, 0, 0, time.UTC), f.ok
}
func lineTestBody(t *testing.T, s *Server) LineTestResponse {
t.Helper()
rec := httptest.NewRecorder()
s.handleLineTest(rec, httptest.NewRequest("GET", "/host/line-test", nil), 9201)
var env struct {
OK bool `json:"ok"`
Data LineTestResponse `json:"data"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &env); err != nil {
t.Fatalf("decode %q: %v", rec.Body.String(), err)
}
return env.Data
}
// R-1247: the controller sets qBittorrent's limit only from a PRESENT value; never measured and not wired must both
// read present:false, never a zero line.
func TestLineTestRoute(t *testing.T) {
s := &Server{logger: slog.Default()}
if got := lineTestBody(t, s); got.Present {
t.Fatalf("not wired: %+v, want present:false", got)
}
s.SetLineTest(fixedLine{ok: false})
if got := lineTestBody(t, s); got.Present || got.Mbps != 0 {
t.Fatalf("never measured: %+v, want present:false", got)
}
s.SetLineTest(fixedLine{mbps: 90.7, ok: true})
got := lineTestBody(t, s)
if !got.Present || got.Mbps != 90.7 || got.Method != "iperf3-tunnel" || !strings.HasPrefix(got.MeasuredAt, "2026-10-24T06:31") {
t.Fatalf("measured: %+v", got)
}
}
type requestableLine struct {
fixedLine
waiting bool
mbps float64
err error
calls int
}
func (r *requestableLine) WaitingForQuiet() bool { return r.waiting }
func (r *requestableLine) RunNow(context.Context) (float64, []float64, error) {
r.calls++
if r.err != nil {
return 0, nil, r.err
}
return r.mbps, []float64{r.mbps - 1, r.mbps}, nil
}
func lineTestRun(t *testing.T, s *Server) (int, LineTestRunResponse) {
t.Helper()
rec := httptest.NewRecorder()
s.handleLineTestRun(rec, httptest.NewRequest("POST", "/host/line-test/run", nil), 9201)
var env struct {
OK bool `json:"ok"`
Data LineTestRunResponse `json:"data"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &env)
return rec.Code, env.Data
}
// R-1250: the controller acts only on waiting_for_quiet, and reads a refusal by its code — never as a measurement.
func TestLineTestRunRoute(t *testing.T) {
s := &Server{logger: slog.Default()}
if code, _ := lineTestRun(t, s); code != 503 {
t.Fatalf("not wired: HTTP %d, want 503", code)
}
s.SetLineTest(fixedLine{ok: true, mbps: 90})
if code, _ := lineTestRun(t, s); code != 503 {
t.Fatalf("a reader without the request half: HTTP %d, want 503", code)
}
l := &requestableLine{fixedLine: fixedLine{ok: false}, waiting: true, mbps: 93.5}
s.SetLineTest(l)
if got := lineTestBody(t, s); got.Present || !got.WaitingForQuiet {
t.Fatalf("GET while waiting: %+v, want present:false waiting_for_quiet:true", got)
}
if code, got := lineTestRun(t, s); code != 200 || !got.OK || got.Mbps != 93.5 || len(got.RunsMbps) != 2 {
t.Fatalf("a measured request: HTTP %d %+v", code, got)
}
l.err = &linetest.Refusal{Code: "not-quiet", Msg: "the uplink sent 16.0 Mbit/s while held"}
if _, got := lineTestRun(t, s); got.OK || got.Refused != "not-quiet" || got.Mbps != 0 {
t.Fatalf("a refusal: %+v, want ok:false refused:not-quiet", got)
}
l.err = linetest.ErrBusy
if _, got := lineTestRun(t, s); got.OK || got.Refused != "server-busy" {
t.Fatalf("server busy: %+v, want refused:server-busy", got)
}
l.err = errors.New("dial tcp 10.77.0.1:5201: i/o timeout")
if _, got := lineTestRun(t, s); got.OK || got.Refused != "" || got.Error == "" {
t.Fatalf("a failed run: %+v, want ok:false, no refusal, the error", got)
}
}
Loaded 100 of 173 files, more files were not shown because too many files have changed in this diff. Show more