Compare commits

...

47 Commits

Author SHA1 Message Date
admin 2d4b1930df REPORT: day 2026-10-23 — agent 0.163.0 released, delivered (step bundle), vouched by the operator
gates / gates (push) Successful in 1m48s
2026-10-23 11:08:14 +02:00
admin b4721eee11 CHANGELOG: v0.163.0 released (D30, D4 root SSH by key only, R-1204)
gates / gates (push) Successful in 2m5s
2026-10-23 08:44:19 +02:00
admin 1109092def D4 (R-982): root SSH from the LAN by key only — the bundle's sshd drop-in, try-reload-or-restart, an sshd -T read-back reported when not in force, a reload in undo (the reviewed v2 patch); unreleased
gates / gates (push) Successful in 2m9s
2026-10-23 08:22:17 +02:00
admin b45c47be6c REPORT: night 2026-10-22/23 (D30 live, hardened; D4 patches v2)
gates / gates (push) Successful in 2m1s
2026-10-23 00:06:13 +02:00
admin b9b831fb90 D30: a wait stamp dated after now (a clock that jumped back) is dropped, never a block until that date — the commit's security review; red-proved; unreleased
gates / gates (push) Successful in 2m3s
2026-10-22 23:08:25 +02:00
admin 39fb9cbf6c D30 rotation hardened after review: the key file is replaced by an fsynced rename over it (never missing or empty), the first old key stays private.key.prev, later ones dated (3 kept), the once-an-hour bound survives a restart; red-proved; CONTEXT; unreleased
gates / gates (push) Successful in 2m24s
2026-10-22 23:00:05 +02:00
admin ae69ad4512 D30 (R-1204): a handshake stamped in the future for 15 min gives a new WireGuard key, registered on the existing peer path (the hub re-keys in place; its full-list push swaps the key on ep0 — nothing new on ep0); old key kept as private.key.prev; red-proved; unreleased
gates / gates (push) Successful in 2m0s
2026-10-22 20:52:52 +02:00
admin e803ad2b91 night 2026-10-22: REPORT; CHANGELOG corrected — a push of the installer to main publishes nothing (R-110, tag installer-v1.35.0); D4 stays a patch because both halves need a release
gates / gates (push) Successful in 2m24s
2026-10-22 18:13:35 +02:00
admin 7651461be2 D4 (R-982) moved out of main into a ready patch — its installer half would publish the installer (forbidden tonight); R-1204 stays; unreleased
gates / gates (push) Successful in 2m45s
2026-10-22 17:10:46 +02:00
admin a0c88e5183 R-982 (D4): root SSH from the LAN by key only — a bundle drop-in, checked and reloaded by felhom-os-apply; R-1204: the heartbeat reports a future-stamped tunnel handshake; unreleased
gates / gates (push) Successful in 2m23s
2026-10-22 16:58:45 +02:00
admin 2d5840ba11 v0.162.0: released (operator yes 2026-10-22), delivered to the 3 test boxes; CHANGELOG + REPORT
gates / gates (push) Successful in 2m10s
2026-10-22 10:39:06 +02:00
admin b310773dbf REPORT: the night of 2026-10-21 (host swap report, R-1194), unreleased
gates / gates (push) Successful in 2m7s
2026-10-22 00:06:21 +02:00
admin 0da792e670 R-1192 follow-up: report host swap vs the guests' swap allowance (log once, hub alarm, local API); R-1194: no OOB stanza before the first felhom-sshd pass; unreleased
gates / gates (push) Successful in 1m56s
2026-10-21 21:29:07 +02:00
admin 286a7e3ecf R-1192: every guest may swap as much as its RAM (bring-up, RAM resize, once at start) — the measured 8 GB freeze; unreleased
gates / gates (push) Successful in 1m39s
2026-10-21 15:43:18 +02:00
admin 4e1621c650 CHANGELOG + CONTEXT + REPORT: v0.161.0 released (operator yes) and delivered to demo-hp, Tester 1, demo-felhom; not vouched; R-1083 left out
gates / gates (push) Successful in 1m27s
2026-10-20 09:07:30 +02:00
admin 51521230c9 REPORT: the night's agent fixes (R-921, R-1070, R-1010), not released
gates / gates (push) Successful in 1m25s
2026-10-19 21:47:40 +02:00
admin 84ef5818d5 security-scan gate (shared felhom.eu/scripts/security_scan_gate.py) in the fast set and CI, warn-first, with the accepted-hits file and decoys (R-986)
gates / gates (push) Successful in 1m30s
2026-10-19 21:11:46 +02:00
admin 56aabc729c Unreleased: host_busy on GET /backup/status (R-921), a missing tunnel container is 'absent' and not-applicable after an OS update (R-1070), the janitor forces a half-removed vzdump snapshot once (R-1010)
gates / gates (push) Successful in 58s
2026-10-19 19:52:37 +02:00
admin 0cbd500154 gofmt gate in agent_gates.py with its decoys (R-1160); REPORT for the day
gates / gates (push) Successful in 49s
2026-10-19 10:26:27 +02:00
admin 1edfd8e331 gofmt -w: 14 files, formatting only, no behaviour change (R-1160) 2026-10-19 10:24:32 +02:00
admin 0bc0bc5da0 CONTEXT + REPORT: v0.160.0 released, on the three test hosts, not vouched
gates / gates (push) Successful in 48s
2026-10-19 05:41:34 +02:00
admin 45458196d1 Two texts made true after 0.160.0: the ceremony log line names the confirm order; the R-973 comments name whole_guest_backup_refused (no behaviour change)
gates / gates (push) Successful in 52s
2026-10-19 05:25:58 +02:00
admin 124164db73 CHANGELOG v0.160.0 (released: R-978, R-973, R-1093, R-1149, R-1151)
gates / gates (push) Successful in 46s
2026-10-19 03:22:00 +02:00
admin c11e3e2a87 The new escrow package reaches the hub only after the household typed two words of the code back: POST /escrow/ceremony/confirm; an old controller keeps the claim order (R-1151, unreleased)
gates / gates (push) Successful in 48s
2026-10-19 03:03:37 +02:00
admin 1b6e31675b The local whole-guest tier keeps one restore point per day for 3 days: keep-daily instead of keep-last (R-1149, operator D17, unreleased)
gates / gates (push) Successful in 47s
2026-10-19 02:59:34 +02:00
admin 1693da4d1a REPORT: the three unreleased fixes (R-1093, R-978, R-973) with the R-1093 loop-device check on demo-hp
gates / gates (push) Successful in 47s
2026-10-18 21:48:03 +02:00
admin 79ba884859 CI workflow: the failure mail links the public run page, by run id (R-1135)
gates / gates (push) Successful in 49s
The alarm step built its link from GITHUB_SERVER_URL, which on this runner
is the in-cluster Gitea address, and from the run NUMBER, which Gitea 1.26
reads as a run id first (felhom.eu /actions/runs/1116 opened run id 1116 =
run number 613). Now https://gitea.dooplex.hu + GITHUB_RUN_ID, and the link
is printed into the job log as ALARM-LINK. Same change in all four repos.
2026-10-18 20:34:59 +02:00
admin f17e1979bd A cadence cap on the local-API whole-guest backup route (R-973, unreleased)
gates / gates (push) Successful in 49s
Each successful run prunes its tier, so a compromised controller calling POST /backup in a loop
pushed out every pre-compromise restore point. A tier with a cadence of at most 24 h now allows 3
successful runs per rolling 24 h (night, a press, the next night); a longer tier 1 per
max(cadence - 24 h, cadence/2), 6 days for the weekly off-site tier. Only successes count. A
refusal is a 429, a WARN, saved, and for 24 h a critical host-report status that the hub turns
into its existing operator-only agent_capability_degraded event. The success history and the
refusal are kept in the R-894 state file, so a restart does not reset the cap. No sudoers change.
Tests TestBackupCap_* and the state, status and wiring tests, red-proved.
2026-10-18 19:53:15 +02:00
admin 0963bae3c3 The partition probe fails closed on every path (R-1093 review, unreleased)
PART_ENTRY_* comes from the parent's table, so it proves nothing about a partition's contents
unless blkid also completed: lsblk's PTTYPE is now set aside only on a blkid exit 0 (exit 2, an
I/O error or a missing binary keep "device has a partition table"). An lsblk that exits 0 with
output that does not parse into a device, or lists none, leaves the device unprobed instead of
blank (a hole that predates R-1093). Six more TestInspect_R1093_PartitionVerdicts rows, each
red-proved.
2026-10-18 19:47:15 +02:00
admin 8561dc8a5c The new escrow package reaches the hub only when its code is claimed (R-978, unreleased)
The root ceremony uploaded the new package before the new code was claimed, and the hub
overwrites the current package when the sealed secrets are unchanged: a customer who started the
ceremony again and walked away kept a written-down code that opened nothing. Json mode (the
FELHOM_ESCROW argv, unchanged) now returns the package (CeremonyOutput v2, PendingUpload); the
daemon uploads it at POST /escrow/ceremony/claim before handing out the code. A failed upload is a
502 with the code still claimable; TTL and a new start wait for an in-flight claim upload; no
uploader is a 503. Text mode still uploads at once. No sudoers change.
Tests TestEscrowCeremony_R978_* (localapi) and the cmd-side defer/delivery/wiring tests, red-proved.
2026-10-18 19:42:02 +02:00
admin f7b7308c4d An empty partition is blank: lsblk's PTTYPE on a partition is the parent's (R-1093, unreleased)
lsblk prints the disk's table type on every partition, so InspectDevice read an empty partition as
"device has a partition table" and the format path asked the typed erase confirmation. lsblk's
PTTYPE is now set aside only when blkid -p on the partition itself proved it read the device and
that it is a partition of a table of that exact type (PART_ENTRY_NUMBER + matching
PART_ENTRY_SCHEME). A filesystem or a nested table in the partition, a silent blkid, a whole disk
or a different scheme keep the data-bearing reading. Same allow-listed argv, no sudoers change.
Test TestInspect_R1093_PartitionVerdicts, red-proved against the old predicate.
2026-10-18 19:35:03 +02:00
admin 27bbdaaa22 REPORT/CHANGELOG: agent 0.159.2 delivered and proven on demo-hp
gates / gates (push) Successful in 51s
2026-10-18 14:45:47 +02:00
admin e25454ccc3 CHANGELOG: v0.159.2 released (the after-failure orphan check)
gates / gates (push) Successful in 59s
2026-10-18 12:54:28 +02:00
admin 65c1994d3b CHANGELOG (Unreleased): the live proof the next release owes for the after-failure orphan check
gates / gates (push) Successful in 47s
2026-10-17 23:54:14 +02:00
admin 6545cccff7 The orphan thin check runs only after a failed snapshot backup (R-1012 follow-up, unreleased)
gates / gates (push) Successful in 48s
v0.159.0/v0.159.1 ran the check before every snapshot-mode whole-guest backup, which takes a
metadata snapshot of the live pool. Now explainFailure (vzdump start or task failure) is its only
caller; the finding is still appended to the failed backup's error. Tests: failure = 1 call,
success = 0 calls; red-proved against the v0.159.1 runner.go (2 calls / calls=1).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 22:43:03 +02:00
admin eb38d78945 REPORT/CONTEXT: agent 0.159.1, the catalog allow-list (controller v0.328.0), its catalog gate (2026-10-17)
gates / gates (push) Successful in 58s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 17:29:50 +02:00
admin 4a1c07b288 CHANGELOG: v0.159.0 (superseded) and v0.159.1 — released, not vouched
gates / gates (push) Successful in 48s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 16:42:00 +02:00
admin 5c3b218822 The signed decommission prunes the drive from every guest's bind record (found live on demo-hp, R-1124 proof)
gates / gates (push) Successful in 47s
The self-serve decommission already dropped the drive from the guest-bind record; the operator-signed one did
not, so a forgotten drive left a record the 20-second re-assert warned about forever ("skipping non-enrolled
drive"). DecommissionExecutor.SetGuestBinds; red-proved. Code only; the CHANGELOG follows the release (0.159.1).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 16:41:12 +02:00
admin e15d1d30ea R-1124 + R-1012: forget a decommissioned drive that is gone; name an orphan thin device in a backup failure
gates / gates (push) Successful in 47s
R-1124: ForgetRetiredDrives (start-up + the 20 s tick) disables and removes the host mount unit of a
DECOMMISSIONED drive whose disk is absent and unmounted; the agent then no longer lists it. A present
decommissioned drive keeps its unit (one-click re-enroll). The file goes only through the new
felhom-priv-apply verb `unit-forget`, which re-checks marker, disabled, unmounted and absent as root.

R-1012: the read-only root helper felhom-thin-check <vg> <pool> (bundle) lists the pool's thin ids
(metadata snapshot reserved and released) next to LVM's before and after. BackupRunner.SetThinCheck runs
it before a snapshot-mode backup (log) and again after a failure, appending the finding in plain words to
the error the controller mails to the operator. Two reads, intersected; fails open; never removes.

Red-proofs: felhom.eu audits/agent-allowlist-rotation-2026-10-17/P1/. Code only; the CHANGELOG entry
follows the release.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 15:20:34 +02:00
admin 344c4f9030 CI workflow: a manual proof that the failure mail can be sent (R-1129)
gates / gates (push) Successful in 59s
workflow_dispatch with input force_alarm (default false). A dispatch-only
step prints the RESEND_API_KEY length (never the key) and, when forced,
fails so the alarm step sends its real mail, subject marked FORCED TEST.
Push runs are unchanged. No agent release.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 12:49:37 +02:00
admin c52d591851 The decoy suite is a gate of the full run (R-1116); scripts only, no release
gates / gates (push) Successful in 49s
scripts/test_gate_decoys.py is now agent_gates.py's `gate-decoys` row: the full run (which CI runs on every
push), not --fast (~20 s in the workspace, 30 cases). Declared in its own COVERS; test_agent_gates.py pins it.
Red-proved: a reuse-refs FACT case flipped to expect rc 0 -> CONVICTED: gate-decoys; restored, green.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 11:01:25 +02:00
admin 37b99d73b3 Go 1.27.2 and x/crypto v0.58.0 (R-972) — built and tested, not released
gates / gates (push) Successful in 31s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-16 18:53:33 +02:00
admin dd8b2ace02 release-agent.sh: only a clean 404 proves a version is new (R-1098)
gates / gates (push) Successful in 31s
An unreachable registry (000) or a server error passed the existing-version check as new. Test:
scripts/test_release_push_once.sh, red-proved. No agent release.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-16 17:23:46 +02:00
admin 383522450a REPORT + CONTEXT: v0.158.1 build B, vouched; R-1093 filed
gates / gates (push) Successful in 32s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 10:10:21 +02:00
admin 1a8bed11e7 CHANGELOG: v0.158.1 (security review of 0.158.0; 0.158.0 superseded, do not vouch)
gates / gates (push) Successful in 25s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 04:02:17 +02:00
admin 98f3cdd76a Security review of 912f7c1: the mkfs wrapper trusts no sibling claim; the internal-place helper fails closed
gates / gates (push) Successful in 25s
- felhom-mkfs-guarded: the 0.158.0 exemption for a sibling part mounted at /mnt/<name> + under /mnt/felhom-drives is
  removed — the agent can make that corroborating mount itself. Every mount of every node outside /mnt/felhom-drives
  refuses (kernel mount table); an unlistable disk refuses. CandidatePartition.Formattable is false for every part of a
  disk with a mounted node, so it is never offered for formatting (attach still is).
- felhom-internal-place: a failed lvs listing, a listing without the pool, or a blkid exit other than 0/2 refuses
  (they read as "nothing promised" / "no signature" before).
Every new test was seen failing against the 0.158.0 versions.

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 04:02:00 +02:00
admin 888a98571b CHANGELOG: v0.158.0 — the internal data place, partitions, add_internal_place
gates / gates (push) Successful in 28s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-16 03:56:29 +02:00
95 changed files with 5377 additions and 209 deletions
+44 -4
View File
@@ -16,7 +16,16 @@
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one # A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
# layer up. That is the last step, and it runs ONLY on failure. # layer up. That is the last step, and it runs ONLY on failure.
name: gates name: gates
on: [push] # R-1129: `workflow_dispatch` is the manual PROOF that the failure mail can be sent. A push run is
# unchanged by it: the proof step below is skipped on a push, and `force_alarm` is then empty.
on:
push:
workflow_dispatch:
inputs:
force_alarm:
description: "Fail this run on purpose so the alarm step sends its real mail (a proof, not a gate result)"
type: boolean
default: false
jobs: jobs:
gates: gates:
@@ -55,8 +64,27 @@ jobs:
env: env:
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain. # In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000 GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
FELHOM_SCAN_FETCH: "1"
run: cd ws/felhom-agent && python3 scripts/agent_gates.py run: cd ws/felhom-agent && python3 scripts/agent_gates.py
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
# Runs ONLY on a manual dispatch, never on a push. It prints the key's LENGTH, never the key,
# so "CI can see the secret" is an observable rather than an assumption. With
# force_alarm=true it then fails on purpose so the alarm step below sends its real mail;
# the commit itself stays green, because its push run skips this step.
if: github.event_name == 'workflow_dispatch'
env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
FORCE_ALARM: ${{ github.event.inputs.force_alarm }}
run: |
python3 -c 'import os;print("RESEND_API_KEY length: %d" % len(os.environ.get("RESEND_API_KEY","")))'
if [ "$FORCE_ALARM" = "true" ]; then
echo "force_alarm=true - failing on purpose so the alarm step runs (R-1129 proof)"
exit 1
fi
- name: Alarm on failure - name: Alarm on failure
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO # THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches # notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
@@ -70,6 +98,8 @@ jobs:
if: failure() if: failure()
env: env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }} RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
# Empty on a push, so a push run's mail is exactly what it was before R-1129.
FORCE_ALARM: ${{ github.event.inputs.force_alarm }}
run: | run: |
python3 - <<'PY' python3 - <<'PY'
import json, os, sys, urllib.request, urllib.error import json, os, sys, urllib.request, urllib.error
@@ -81,13 +111,23 @@ jobs:
repo = os.environ.get("GITHUB_REPOSITORY", "?") repo = os.environ.get("GITHUB_REPOSITORY", "?")
sha = os.environ.get("GITHUB_SHA", "?") sha = os.environ.get("GITHUB_SHA", "?")
run = os.environ.get("GITHUB_RUN_NUMBER", "?") # R-1135: the link is for a PERSON, so it is the public address and the run ID.
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu") # NOT GITHUB_SERVER_URL: the runner registers at the in-cluster address, so that variable
# reads http://gitea.gitea-system.svc.cluster.local:3000, which nobody outside the cluster
# can open (forced-test mail, job 1997). NOT GITHUB_RUN_NUMBER: Gitea 1.26 reads
# /actions/runs/<n> as a run id first, so a run number that is also an older run's id
# opens THAT run (measured 2026-10-18: felhom.eu /actions/runs/1116 opened the run whose
# id is 1116 = run number 613). The link is printed too, so the job log carries it.
run = os.environ.get("GITHUB_RUN_ID", "?")
srv = "https://gitea.dooplex.hu"
print("ALARM-LINK %s/%s/actions/runs/%s" % (srv, repo, run))
body = json.dumps({ body = json.dumps({
"from": "Felhom CI <monitoring@felhom.eu>", "from": "Felhom CI <monitoring@felhom.eu>",
"to": ["admin@felhom.eu"], "to": ["admin@felhom.eu"],
"subject": "[felhom CI] gates FAILED in %s" % repo, "subject": ("[felhom CI] FORCED TEST, not a real failure - gates FAILED in %s"
if os.environ.get("FORCE_ALARM") == "true"
else "[felhom CI] gates FAILED in %s") % repo,
"text": ( "text": (
"The gate entry point exited non-zero.\n\n" "The gate entry point exited non-zero.\n\n"
"Repository : %s\n" "Repository : %s\n"
+18
View File
@@ -0,0 +1,18 @@
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
gitleaks generic-api-key@1ad7cc8eb57e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46ce5a3eccc4 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46fe126846b0 internal/escrow/identity_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@56474f1754f5 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@631c53997f7e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@6e790ef41ef3 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@78334b854242 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@7b88d323a44f internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b0eeee93cf24 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b1cb2659967a internal/localapi/escrow_stage_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b66bec43cc8a internal/escrow/recover_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e0525160fe03 internal/pbsdr/wrapper_read_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e442e455c360 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f23f93f461b2 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f471b78e0f6e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
+380
View File
@@ -1,3 +1,383 @@
## v0.163.0 — the tunnel heals itself after a clock that ran ahead (D30, R-1204); root SSH from the LAN by key only (D4, R-982) (2026-10-23) — RELEASED on the operator's yes (day brief); the vouch is the operator's
Binary sha256 `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c`; config bundle sha256 `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8`. Built from `1109092` (CI 2246 green). Both sections below are in it.
### The tunnel re-key (D30) and the D4 bundle drop-in
- **D30 (operator 2026-10-22: the agent makes a new key and registers it on the existing peer path).** When the latest
handshake stays stamped in the future for 15 minutes (the R-1204 sign: ep0 keeps the future timestamp and refuses
every newer handshake from that key), the agent makes a new WireGuard key pair, keeps the old private key as
`wg/private.key.prev` (0600 — the escrow may hold it), and registers the new public key through `POST /hosts/<id>/wg`.
The hub re-keys the peer IN PLACE (same /32, generation bump) and its full-list push to ep0 drops the old key and adds
the new one — nothing new runs on ep0. A new key is a new peer for ep0, with no stored timestamp, so the next
handshake is accepted. At most once an hour; not while the tunnel unit is down; never while a re-register is already
in flight. A failed registration is retried by the existing re-key-in-place path. Logs carry public keys only.
Tests `TestD30_*` (4), red-proved. A host-loss DR that installs the escrowed (older) key registers it again, and the
hub re-keys in place back to it — the same path.
- **Review of the night's own code (felhom.eu `REPORT-night-2026-10-23.md`, N6, N7).** The rotation never leaves the box
without a readable key: the new key is written and fsynced to a temp file, the current key gets its kept name by a
hard link, then the temp file is renamed OVER `private.key` and the directory fsynced (a power cut or a kill between
two renames used to leave no key, or an empty one). `private.key.prev` keeps the FIRST old key (the escrow may hold it);
later ones are `private.key.prev.<unix>`, the newest three kept. The once-an-hour bound is saved in `wg/last-rekey`, so
an agent restart does not reset it; a start or a last re-key dated AFTER now (a clock that jumped back) is dropped,
never a block until that date (the commit's own security review). Tests `TestD30_TwoRotationsKeepTheFirstOldKey`,
`TestD30_ALastReKeyFromTheFutureDoesNotBlock`,
`TestD30_TheHourBoundSurvivesARestart`, red-proved. **Live (felhom.eu `audits/night-2026-10-23/P2-D30/`):** the D30
re-key ran on a test box (before these two changes): clock corrected → 15 min → new key registered → the box switched
at its next desired-state fetch → a handshake with the new key.
- **D4 (operator ruling 2026-10-22, R-982) — root SSH from the LAN by key only, on `main` now (2026-10-23 day brief).** The
config bundle carries `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`;
content check: exactly that line, `sshd -t -f`). After an install that wrote it: `sshd -t` (the whole config), then
`systemctl try-reload-or-restart ssh` (a stopped or socket-activated sshd no longer refuses the bundle), then `sshd -T`
is read back and a drop-in that is NOT in force (no Include, an earlier drop-in, a Match block) is said loudly and
reported as `sshd_dropin_not_in_force` — never a refusal; an undone bundle reloads sshd again. The console and the
web page keep the root password; felhom-sshd (8822) is not touched. Tests `RootKeyOnly` (6), red-proved; the
installer's uninstall removes the file (felhom.eu installer 1.36.0). Reviewed 2026-10-23 night (A1–A7).
- **Known gap, unchanged (filed):** after a reboot the kernel forgets the future stamp (latest handshake 0), so neither
this re-key nor the hub's alarm sees a box whose clock was corrected and which then restarted (R-1223).
### The future-stamped handshake is reported (R-1204) — first written on the 2026-10-22 night
- **R-982 / D4 — a ready patch on that night; on `main` since `1109092` and released in v0.163.0 (see above).** Built and proven the same night (felhom.eu
`audits/night-2026-10-22/P2-D4/`): a bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
(`PermitRootLogin prohibit-password`), a `sshd` content check (exactly `prohibit-password`; `sshd -t -f`), and after
an install that wrote it `sshd -t` + `systemctl reload ssh`; tests `RootKeyOnly` (3), red-proved. Live on a fresh
nested box from ISO 1.30.0: root password over SSH refused, root key works, the console and the web page still take
the password. It was committed (`a0c88e5`) and moved back out (`7651461`): the installer's uninstall must remove every bundle file (felhom.eu `scripts/test_hostinstall.py`), the bundle half reaches a box only with an agent release, and the installer half only with a new `installer-v…` tag (R-110 — a push to `main` alone publishes nothing) — both forbidden tonight. *(Corrected the same night: the first reason written here, „a push of `felhom-host-install.sh` publishes the installer", was wrong — the website serves `/scripts/` from the tag `installer-v1.35.0`, read live in `manifests/webpage.yaml` and the running Deployment.)* Both halves are patches: `felhom-agent-bundle-D4.patch` and `felhom.eu-installer-D4.patch` in that folder; apply them together with the agent release and the next installer tag.
- **R-1204 — `wireguard.handshake_future_s`.** A box that ran once with its clock ahead keeps a handshake stamped in the
future; ep0 refuses every newer one, and `last_handshake_age_s` (clamped at 0) read „just now" for ever. The heartbeat
now carries how many seconds the handshake lies ahead (> 60 s only); the age keeps its clamp. The hub alarms on it
(felhom.eu hub `wg_tunnel_stale`). Test `TestR1204_AFutureHandshakeIsReported`, red-proved.
## v0.162.0 — every guest may swap as much as its RAM (R-1192, the 8 GB freeze); the host's swap is reported and checked; no false „operator access degraded" on a fresh box (R-1194) (2026-10-22) — RELEASED on the operator's yes; the vouch is the operator's
Released 2026-10-22 07:40 UTC (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at
`b310773`: binary sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (the same bytes as the
night's candidate), config bundle sha256 `ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5`, tag
`v0.162.0`; no `configs/` change since v0.161.0. Delivered by one signed `agent_update` job per box to demo-hp, Tester 1
and demo-felhom (07:49–07:58 UTC): each `signed op COMPLETED`, `capabilities self-check ok=76 total=76`, `update
committed`, then `guest-swap: raised` for its 9201 (512 MB → its RAM). Evidence felhom.eu
`documentation/audits/morning-2026-10-22/release/04-agent-delivery.txt`. **Found at once:** demo-hp (host swap 8191 MB,
guest swap 25898 MB) and demo-felhom (8191 MB vs 12288 MB) logged the host-swap WARN and the hub mailed
`host_swap_short` — the check compares the host's swap with the guests' whole swap allowance, which a guest larger
than the installer's 8 GB host swap always exceeds (felhom.eu register R-1216). Soak (night of 2026-10-21, 8 GB box):
11 h, no freeze, the host's swap peaked at 2.7 GB used.
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
never set it. On the 8 GB machine (guest 4096 MB) the first uploads made Immich load its analysis models; the guest hit
its limit with the swap full and could evict only program code — 150 000–600 000 refaults and ~1 GB of disk reads per
10 s, memory „full" pressure 92–95 %, load 155, the dashboard dead for 6½ minutes, no app killed. Raising the swap cap
to 4096 MB live ended it within a minute while the uploads ran on.
- `reconcile.buildBringUpConfig`: a new guest gets `swap` = its memory.
- `POST /guest/memory` (RAM resize): `swap` moves with `memory` (`localapi.GuestSwapMB`).
- At agent start (one minute in): `localapi.EnsureGuestSwap` raises every LXC in the `felhom` pool whose swap is below
its RAM to it — a live cgroup change, no restart; never lowers a higher swap; one guest's failure is logged and the
others are still done.
- Tests `TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM`, `TestR1192_BringUpSetsSwapToTheRAM`, the resize test's
new swap assertion — each red-proved (the write removed → `raised 0, want 2`; `swap=""` on both other paths).
- Privilege: `swap` is under `VM.Config.Memory`, which the RAM resize already uses; no role, sudoers or bundle change.
Added the night of 2026-10-21 (still unreleased, the same release):
- **The host's swap is reported and checked (R-1192 follow-up).** The guest fix moves a memory burst onto the HOST's
swap; a host with none gets nothing from it. The host report carries `host.host_swap_mb` (Proxmox node status — the
kernel's SwapTotal, the same number as `/proc/meminfo`) and `host.guest_swap_mb` (the sum of the guests' `swap`; each
guest spec also carries `swap_bytes`). When the first is below the second the agent logs one WARN (and one INFO when
it is covered again); the hub raises `host_swap_short`. `GET /guest/memory` carries `host_swap_mb` and
`guest_swap_mb` for the controller's install page. The agent never creates host swap (a decision — felhom.eu
register). Measured the same night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap.
- **R-1194 — no false „operator access degraded" on a fresh box.** The first host report was sent before the
felhom-sshd loop's first pass, read a config that was never rendered („config invalid, port 0") and the hub mailed
`oob_degraded` — reproduced on a fresh install 2026-10-21 (the report at 20:34:00, the apply 0.4 s later, the daemon
healthy; the check runs `sshd -t` through sudo, so the non-root theory was false). The loop now reports no OOB stanza
until its first pass has run (a failed pass counts, so a real lasting failure still reaches the hub).
- Tests `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
`TestR1194_NoOOBStanzaBeforeTheFirstPass` — each red-proved.
## v0.161.0 — three fixes (R-921, R-1070, R-1010); two texts made true after 0.160.0 (2026-10-20) — RELEASED on the operator's yes, NOT VOUCHED
Released 2026-10-20 (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at `5152123`:
binary sha256 `ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5`, config bundle sha256
`485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47`, tag `v0.161.0`. Delivered by one signed
`agent_update` job per box to demo-hp (06:38 UTC), Tester 1 (06:54) and demo-felhom (06:55): each `signed op COMPLETED`,
`capabilities self-check ok=76 total=76`, `update committed`. **The vouch is the operator's.** Live on demo-hp: `host_busy`
answers `{"busy": true, "what": "backup:local"}` during a „Mentés most" whole-guest backup and `{"busy": false}` before
and after (read from inside 9201 with its own token; felhom.eu `audits/night2-2026-10-20/P1/`). R-1083 (a key for an
existing guest) is NOT in this release: see the R-1083 row. No sudoers, wrapper or config-bundle change.
- **R-921 — `GET /backup/status` says when the host is busy.** New field `host_busy` (`{busy, what}`): whether the
host-wide heavy-operation gate is held right now (another guest's backup, the restore test, a night OS step, the
stale-lock sweep, a power-off) — the state that makes a `POST /backup` answer 409. Served on every request; absent
when the gate is not wired. Controller ≥ 0.335.0 reads it before it stops the household's apps for a tier, so a
refused tier costs no stop (the measured case of 2026-10-08: the night OS step, all 15 apps stopped for a refusal).
A read, never a reservation. No new route. Test `TestBackupStatus_R921_HostBusy`, red-proved (assignment removed →
`free gate: host_busy = <nil>`).
- **R-1070 — a box with no tunnel set up no longer fails its host OS update.** Docker 29 prints „error: no such
object" in lower case, so the tunnel probe's case-sensitive match fell through to `unknown` („could not ask the
guest"), and the host health rule failed every box without a cloudflared container (Tester4, 2026-10-15,
`os_update_health_failed`). Now the match ignores case and a missing container is a new state, **`absent`** („no
tunnel set up on this box"); the host rule reads it as not applicable only when the reading from before the update
(Docker answered) also had no cloudflared container. A tunnel that is down, gone or unreadable still fails. The hub
alarms only on `not_running`, so `absent` never mails. Tests `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows) and
three `TestClassifyTunnel` rows (Docker 29's wording measured on demo-hp), red-proved both ways.
- **R-1010 — the stale-lock janitor clears a half-removed `vzdump` snapshot.** When the delete fails ONLY with
„Failed to find logical volume …_vzdump" (one disk's snapshot already gone — demo-hp 2026-10-14 after a backup ran
out of space), it retries once with Proxmox's own `force=1` on the same API call (`DeleteSnapshotForce`; the same
token privilege, no sudoers line, no root command — `pvesh usage … -command delete` lists `--force`). Any other error
is returned unchanged; a failed forced delete is returned, never a second force. Tests
`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone` (6 rows) and `TestDeleteSnapshotForce_R1010`, red-proved
(single plain delete → `calls [plain], want [plain force]`).
- **The security-scan gate (R-986), tooling only:** `agent_gates.py` registers the SHARED
`felhom.eu/scripts/security_scan_gate.py` (FAST set: the hook and CI) — govulncheck (called vulnerabilities) and gitleaks
on the pushed commits, **warn-first**. `.security-accepted` holds the first scan's 15 accepted hits (escrow and
local-API test constants), each with a reason. CI fetches the pinned gitleaks (`FELHOM_SCAN_FETCH=1`). Decoys: the
shared 8 plus the gate on this repo, in `test_gate_decoys.py`.
### Earlier on 2026-10-19 (texts only)
- The ceremony-complete log line now says the package uploads at the claim **or at the typed-back confirm** (R-1151);
it read „package uploads at the claim" for both orders (seen in demo-hp's journal, 2026-10-19 01:47 UTC).
- The R-973 comments in `cmd/felhom-agent/main.go` and `internal/localapi/backup_cap.go` named the hub event
`agent_capability_degraded`; since hub v0.159.0 (R-1150) a refused backup raises `whole_guest_backup_refused`.
Found by the hub helper of the day brief 2026-10-19.
- **gofmt, R-1160:** `1edfd8e` formats the 14 Go files that were not gofmt-clean (formatting only, `go test ./...`
green). New gate `scripts/gofmt_gate.py` (the controller's R-454 shape) in `agent_gates.py`, fast set; its decoys in
`scripts/test_gate_decoys.py` plant an unformatted file in a SCRATCH copy (convicted by name), the formatted twin
(passes), and the no-Go modes (CI passes out loud, a dev machine is INCONCLUSIVE). Red-proved: with the conviction
switched off the FACT case reports a LIVE HOLE. The next release carries it; no release for it.
## v0.160.0 — five fixes: an abandoned or un-typed-back escrow ceremony keeps the old code (R-978, R-1151); a whole-guest backup flood is capped (R-973) and the local tier keeps one point per day (R-1149); an empty partition is not „data" (R-1093) (2026-10-19)
Released 2026-10-19 on the operator's yes (02:48 answers, D17 „one per day, 3 days"). sha256
`7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1`, config bundle
`6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f`. The vouch is the operator's.
- **R-1093 — an EMPTY partition no longer asks the typed „data will be erased" confirmation.** `lsblk` prints the
parent disk's table type (PTTYPE) on every partition, so `SudoHostOps.InspectDevice` read a blank partition as
„device has a partition table". Now lsblk's PTTYPE is set aside only when `blkid -p` on the partition itself proved
it read the device and that it is a partition of a table of exactly that type (`PART_ENTRY_NUMBER` +
`PART_ENTRY_SCHEME` equal to lsblk's PTTYPE). The partition's own signatures still come from `blkid -p`: a
filesystem or a nested table inside it stays data-bearing; a blkid that printed nothing, a whole disk, or a scheme
that differs keeps the old, safe reading. **No new command, no sudoers or config-bundle change** (the two probes are
the same allow-listed argv). Test `TestInspect_R1093_PartitionVerdicts` (8 rows), red-proved: with the old
`if pttype != ""` both empty-partition rows fail `DataBearing=true reason="device has a partition table"`.
**Hardened after the same night's review (fail-closed on every path):** `PART_ENTRY_*` is read from the PARENT's
table, so lsblk's PTTYPE is set aside only when blkid also **exited 0** — exit 2, an I/O error or a missing binary
keep „device has a partition table"; and an `lsblk` that exits 0 with output that does not parse into a device (or
lists none) now leaves the device UNPROBED („could not be reliably inspected") instead of „blank" — that hole
predates R-1093. Six more rows in the same test, each red-proved (`blkidErr` check removed → „device is blank";
`&& parsed` removed → `Probed=true … „device is blank"`).
- **R-978 — an abandoned second escrow ceremony no longer makes the customer's recovery code useless.** The root
ceremony (the FELHOM_ESCROW argv, unchanged: `--upload --output=json`) used to upload the new package before the new
code was claimed, and the hub overwrites the current package when the sealed secrets are unchanged — so a customer
who started the ceremony again and walked away (or an agent restart in the 10-minute window) was left with a
written-down code that opened nothing. Now json mode hands the package back (`CeremonyOutput.PendingUpload`, output
**version 2**; text mode still uploads at once) and the daemon uploads it at `POST /escrow/ceremony/claim`, **before**
the code is handed out (`EscrowCeremonyConfig.Upload` → `putEscrowBody`, the same PUT and per-host key). A failed
upload answers 502 and keeps the code claimable; the TTL and a new start wait while the claim's upload is in flight;
no uploader → 503; an output that uploaded by itself or carries no package fails the ceremony. Status `uploaded`
now reads false until the claim. The controller is unchanged (it never read `uploaded`; its claim already assumed
the package is on the hub when the claim returns). **No sudoers or config-bundle change.** Tests (localapi)
`TestEscrowCeremony_R978_*` (abandon by TTL / by a new start / by a restart → the hub's package still opens with
code 1; claim stores before handing out; upload failure; in-flight TTL; no uploader; no package) and (cmd)
`TestEscrowOptsFromCLI_JSONModeDefersUpload`, `TestDeliverEscrowPackage_JSONModeSendsNothing`,
`TestR978_CeremonyUploadIsWiredIntoTheDaemon`. Red-proved: upload-at-create restored → „code 1 no longer opens it
(uploads=2)"; the in-flight TTL guard removed → „the TTL voided the code while its package was uploading";
`deferUpload: false` → „json mode must defer the upload"; the `Upload:` line removed → „has no Upload". **Residual:**
the customer's typed-back check after the claim is browser-only, so „confirmed" here means „claimed" — a customer
who opens the code and closes the page without writing it down has replaced the package.
- **R-1151 — the new recovery code replaces the hub's package only after the household typed it back.** R-978 moved
the upload to the claim; but the wizard's typed-back check was the browser's alone, so a household that opened the
new code and closed the page without writing it down had already replaced the package their old code opens. Now a
claim with `{"defer_upload": true}` (controller ≥ 0.332.0) hands out the code with two word positions
(`verify_positions`, drawn by crypto/rand) and keeps the package; the new `POST /escrow/ceremony/confirm
{"words": [w1, w2]}` checks the two words (constant time, case and spaces aside) and only then uploads it. Right → 200;
wrong → 422, still waiting; one word → 400; nothing waiting → 404; the 30-minute window ran out → 410 and the job
reads `unconfirmed_void` (the package dropped, the previous code still works); hub failure → 502, still waiting. Only
the two words are kept for the check, never R; a new start, the window and a restart drop them with the package.
Status gains `awaiting_confirm` and `confirm_expires_in_sec`. **An old controller (≤ 0.331.0) claims with `{}` and
never confirms: it keeps the R-978 order (upload at the claim)**, so its household's ceremony still completes; a new
controller on an old agent sees no `confirm_required` and needs no confirm. No sudoers or bundle change. Tests
`TestEscrowCeremony_R1151_*` (shown but not typed back — by the window, a new start, a restart — code 1 still opens
the hub's package; typing; the old controller's order; hub failure at the confirm) and `TestPickConfirmPositions`.
Red-proved: the defer flag ignored → „a code that was shown but never typed back replaced the package: code 1 no
longer opens it (uploads=2)" in all three; the word check skipped → „one wrong word: got 200 (uploads=1)".
- **R-1149 — the local whole-guest tier keeps one restore point per day for 3 days** (operator decision D17,
2026-10-19). The primary (local) tier's per-run prune is now `--prune-backups keep-daily=N` (N =
`local_backup_retention`, default 3) instead of `keep-last=N` — the same at most 3 archives, but a run of extra
backups inside one day replaces only that day's point, so with the R-973 cap a flood cannot push out the earlier
days' copies. Additional tiers keep their explicit `keep_last` (the weekly off-site „last 2" is two weeks).
`config.BackupTier.PruneSpec()` is the one place the spec is built (runDaemon used to format keep-last itself); the
„backup tier armed" log line names it. Tests `TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay` and the
consequence `TestBackupCap_R1149_LocalFloodKeepsOlderDays` (the agent's real spec, a Proxmox-shaped prune model,
three nights then a day-long flood → the two earlier nights survive). Red-proved: keep-last for the primary →
„primary tier prune = "keep-last=3", want "keep-daily=3"" and „holds 0 copies made before the compromise". No
config or sudoers change.
- **R-973 — the local-API whole-guest backup route has a cadence cap.** Every successful run prunes its tier, so a
compromised controller calling `POST /backup` in a loop pushed out every pre-compromise restore point (local
keep-last 3; off-site via ep0's keep-last 2). Now (`internal/localapi/backup_cap.go`): a tier whose cadence is ≤ 24 h
allows **3 successful runs per rolling 24 h** (the `07` §6 day: the night run, a „Mentés most" press, and the next
night, which since R-899 still runs and can land inside the same 24 h — so a press never costs the night); a longer
tier allows **1 per max(cadence − 24 h, cadence/2)** — 6 days for the 7-day off-site tier, which the controller only
runs when due and the button never targets. Only successes count (a failed run prunes nothing). A refusal answers
**429** with the next allowed time, is logged at WARN, is saved, and for 24 h raises the CRITICAL host-report status
`localapi:backup-cadence-cap` — the hub's existing operator-only `agent_capability_degraded` event, **no hub change**.
**The cap survives an agent restart:** `BackupSuccessState` (the R-894 file) now also keeps the newest 8 success
times and the last refusal per tier and guest (additive JSON fields; a pre-R-973 file counts its one known
success). Tests `TestBackupCap_*` (an off-site flood keeps a pre-compromise copy for 5+ days; a local flood runs 3
a day; the scheduled day is never refused; restart; failures; the operator signal) + `TestBackupCadenceCap`,
`TestBackupSuccessState_R973_*`, `TestBackupCapStatus_*`, `TestR973_CapRefusalIsWiredIntoTheReport`. Red-proved:
the cap disabled → „the off-site tier lost every pre-compromise copy: 10 runs"; the disk history ignored → „after a
restart the 4th run in a day was allowed"; the wiring line removed → „never binds backupCapRefusal". **Limits, stated:**
with keep-last 3 the local tier's three points can still be replaced within one day (3 per day is the schedule's own
pace plus one press); the hub's event text reads „agent privileged capability degraded" (the type is generic). No
sudoers or bundle change.
- **CI workflow (R-1135, no code change).** `.gitea/workflows/gates.yml`, the alarm step: the mail's run link is now `https://gitea.dooplex.hu/<repo>/actions/runs/<run id>`.
It used `GITHUB_SERVER_URL`, which on this runner is the in-cluster address (`http://gitea.gitea-system.svc.cluster.local:3000`,
the address the runner registers at), so nobody outside the cluster could open it. It also used the run NUMBER, and
Gitea 1.26 reads `/actions/runs/<n>` as a run id first: a number that is also an older run's id opened that other run
(measured 2026-10-18: `felhom.eu/actions/runs/1116` opened run id 1116, run number 613). It now uses `GITHUB_RUN_ID` and
prints the link (`ALARM-LINK …`) into the job log. Same change in all four repos' workflows.
## v0.159.2 — the orphan thin check runs only after a failed backup (R-1012 follow-up) (2026-10-18)
**Released 2026-10-18 (committed 2026-10-17): sha256 `4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9`, bundle `477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9`; the vouch is the operator's.** No `configs/` change since v0.159.1. The independent review of 2026-10-18 found nothing in it. v0.159.0/v0.159.1 ran the R-1012 orphan check
before EVERY snapshot-mode whole-guest backup, and that check takes a short metadata snapshot of the live thin pool.
Now `BackupRunner` runs it only when the snapshot step has failed (the vzdump start or its task) — the moment the
answer is needed. The finding is still appended in plain words to the failed backup's error, exactly as before; a
successful backup never runs the check and keeps a clean record; stop mode and an unwired runner are unchanged. The
`thin-orphan-check` capability text says „after a failed whole-guest backup". Tests
`TestBackup_SnapshotFailure_NamesTheOrphan` (now: exactly 1 call, after the failure) and
`TestBackup_ThinCheck_SilentWhenNotApplicable` (now: 0 calls on a successful backup), red-proved against the v0.159.1
`runner.go`: failure legs „thin check ran 2 times, want 1", success leg `calls=1`. No root file, sudoers or bundle
change. (Installer side, felhom.eu, same night: `felhom-host-install.sh --rotate-recovery-only`, R-1141.)
**Proven live 2026-10-18 (demo-hp):** a green snapshot backup with 0 `felhom-thin-check` calls (the 0.159.1 night backup
had 1). The failed leg stays covered by the fixture test.
## v0.159.1 — the signed decommission prunes the guest-bind record (found by v0.159.0's own live proof) (2026-10-17)
**v0.159.0 is superseded — do not vouch it.** Everything of v0.159.0 below, plus: `signedjobs.DecommissionExecutor`
drops the drive from every guest's bind record (`SetGuestBinds`, wired to the agent's `GuestBindStore`), as the
self-serve decommission always did. Without it a drive decommissioned by an operator-signed job and then forgotten
(R-1124) left a record the 20-second guest-bind re-assert warned about every tick („reconcile: skipping non-enrolled
drive") — seen on demo-hp minutes after the R-1124 proof. Test `TestDecommissionExecutor_PrunesTheGuestBindRecord`,
red-proved. Released: tag `v0.159.1`, sha256 `de8f6ad9…aff9`, bundle `2af3d27d…7cd6` (the same root files as v0.159.0).
**Not vouched.** Delivery to a box on the v0.158.1 bundle: `agent_update 0.159.1` → `agent_config_update 0.159.0-step1`
(sha `1d65edb3…d6cf`, published) → `agent_config_update 0.159.1` (the bundle adds `felhom-thin-check`, R16).
## v0.159.0 — forget a decommissioned drive that is gone (R-1124); name an orphan thin device in a backup failure (R-1012); Go 1.27.2 (R-972) (2026-10-17)
**Superseded by v0.159.1 — do not vouch.** Task: „agent release, the catalog allow-list, break-glass rotation, off-site
history check" (felhom.eu `REPORT-agent-allowlist-rotation-2026-10-17.md`).
- **Go 1.27.2 + x/crypto v0.58.0** (R-972, `37b99d7`): the first release that carries them (v0.158.1 was built with
go1.26.0).
- **R-1124 — a decommissioned drive that is gone is forgotten.** `SudoHostOps.ForgetRetiredDrives` (agent start and the
20-second storage tick): for a Felhom drive unit whose intent is `decommissioned`, whose `/dev/disk/by-uuid` link is
gone and which is not mounted, `systemctl disable` + the new root verb `felhom-priv-apply unit-forget <unit>` +
`daemon-reload`. The unit is gone, so a later boot no longer waits for the drive and the agent stops listing it (no
„absent by UUID" line every 20 s). A decommissioned drive still plugged in KEEPS its unit and raw mount, so the
one-click re-enroll (H3) still works. `unit-forget` re-checks as root: a felhom by-UUID drive unit (not a network
unit), disabled, its `Where` not mounted, its by-UUID link absent — otherwise refused. Sudoers `FELHOM_MOUNT` gains the
one verb. Tests `TestForgetRetiredDrives_*` and `configs/test_felhom_priv_apply.py` `UnitForget`, red-proved.
**Live on demo-hp:** a loop-backed test drive enrolled through the local API, decommissioned by a signed job (kept
its unit while present), unplugged → 8 s later the unit and its wants link were removed and `GET /disks` dropped it.
- **R-1012 — an orphan thin device is NAMED in the backup failure, never removed.** New READ-ONLY root helper
`felhom-thin-check <vg> <pool>` (config bundle): LVM's thin ids before and after, the pool's ids through a metadata
snapshot it reserves and releases (refuses — exit 3 — when someone else holds one; releases on every exit; sends no
other pool message). `internal/thincheck`: an orphan = in the pool and in neither LVM list, on two reads 3 s apart;
fails open. `BackupRunner.SetThinCheck`: before a snapshot-mode backup (a WARN when found) and again after a failure —
the finding is appended in plain words to the error the controller mails to the operator, pointing at the new
runbook `whole-guest-backup-orphan-thin.md`. Sudoers `FELHOM_THINCHECK`, capability `thin-orphan-check`. Tests
`internal/thincheck`, `TestBackup_SnapshotFailure_NamesTheOrphan`, `configs/test_felhom_thin_check.py`, red-proved.
**Live on demo-hp:** on a scratch thin pool (a loop file on `/mnt/hdd_1`, never `pve/data`) the helper listed a made
orphan (device 7) and refused a snapshot held by someone else; on `pve/data` it read 1, 2 = LVM's and released; a
normal whole-guest backup of 9201 ran the check first and finished green.
- Released: tag `v0.159.0`, sha256 `3cfd0a25…edfed5`, bundle `40663c4f…6d681`; step bundle `0.159.0-step1`.
## CI only — a manual proof that the failure mail can be sent (R-1129, 2026-10-17)
No agent release; no binary changes.
`.gitea/workflows/gates.yml` gains a manual `workflow_dispatch` trigger with one input, `force_alarm` (default false), and
one step that runs ONLY on a dispatch: it prints the length of `RESEND_API_KEY` (never the key) and, with
`force_alarm=true`, fails on purpose so the existing alarm step sends its real mail; the mail's subject then starts
„FORCED TEST, not a real failure". A push run is unchanged: the new step is skipped and the subject is the old one. Same
change in all four repos' workflows. Purpose: prove the failure mail now goes out after the operator set the key
(every failed run until today ended „ALARM FAILED: RESEND_API_KEY is empty").
## scripts only — the decoy suite is a gate of the full run (R-1116, 2026-10-17)
No agent release; no binary changes. `scripts/test_gate_decoys.py` is now the `gate-decoys` row of `agent_gates.py`:
the full run (which CI runs on every push), not `--fast` — it takes ~20 s in the workspace (measured: 30 cases; ~5 s in a
CI-shaped shallow clone, also green). It was in no runner, the shape that let the catalog's R-1115 fail unseen for a
night. It plants only in scratch clones and asks a fake Gitea on 127.0.0.1; it does not call the runner (no
recursion). Declared in its own `COVERS`; `test_agent_gates.py` pins it (registered once, full run only, named by
`--fast`'s SKIPPED line). Red-proved: a reuse-refs FACT case flipped to expect rc 0 turned the full run red
(`CONVICTED: gate-decoys`); restored, green. CLAUDE.md's `--fast` sentence names it.
## unreleased — Go 1.27.2 and golang.org/x/crypto v0.58.0 (R-972, 2026-10-16)
**Not released, not deployed** — agent releases wait for the operator's vouch; the next vouched release carries this.
`go.mod`: `go 1.27.2` (was 1.25.0; the shipped 0.158.1 binary was built with go1.26.0), x/crypto v0.58.0 (was v0.52.0),
x/sys v0.49.0. `go build`, `go vet`, the full `go test ./...` and `agent_gates.py --fast` green on Go 1.27.2.
`release-agent.sh` builds with the toolchain `go.mod` names (GOTOOLCHAIN=auto), so no script change.
## scripts only — `release-agent.sh` refuses when the registry cannot say the version is new (R-1098, 2026-10-16)
No agent release; no binary changes. `scripts/release-agent.sh` already refused a version whose tag exists or whose
package answers 200. It let everything else through — a registry it could not reach (curl prints `000`) or a 500 read
as „new". Now only a clean `404` proves the version is new; anything else refuses with the code it got. New
`scripts/test_release_push_once.sh` (the real script, a fake `curl` and `go` on PATH; published / absent / unreachable /
server error / tag exists), red-proved: with the 404 rule removed, the unreachable and server-error cases fail. Same rule
as the controller `build.sh` and, since today, felhom.eu `scripts/build-hub.sh`.
## v0.158.1 — security review of v0.158.0: the mkfs wrapper trusts no sibling claim; the internal-place helper fails closed (2026-10-16)
**v0.158.0 is superseded — do not vouch it.** A security review of `912f7c1` found two defects; both fixed here, each test
seen failing against the 0.158.0 script:
- **`felhom-mkfs-guarded`**: 0.158.0 let a format through when a SIBLING part was mounted at /mnt/<name> AND under
/mnt/felhom-drives („Felhom's own drive"). That corroboration is a mount the agent can make itself, so a compromised agent
could unlock a host owner's disk — the exact case this root wrapper exists for. Removed: any mount of any node outside
/mnt/felhom-drives refuses again (now read from the kernel's mount table for every node; an unlistable disk refuses).
Consequence: the free part next to an in-use Felhom drive cannot be formatted — `CandidatePartition.Formattable` is false
for every part of a disk with a mounted node, and the controller does not offer it (a part with a filesystem can still be
USED). Pinned by configs/test_felhom_mkfs_guarded.py + partitions_test.go.
- **`felhom-internal-place`**: a failed `lvs` listing read as „nothing promised" (a too-large place) and a `blkid` error read
as „no signature" (the only gate before mkfs). Now: a failed listing, a listing without the pool, or any blkid exit other
than 0/2 REFUSES. Three new tests, all failing against 0.158.0.
- Released: tag `v0.158.1` (code `98f3cdd`), sha256 `20264df6…84b76a1`, bundle `c23c3d35…cb1c41b`. **Not vouched.**
## v0.158.0 — the internal data place; partitions as places; add_internal_place (2026-10-16)
Design: felhom.eu `architecture/16-storage-physical-identity.md` §5, §6, §7, §9 (operator rulings D3–D5; the D3 sizes
ruled 2026-10-16 after the space check, felhom.eu `audits/storage-identity-buildB-2026-10-15/SPACE-CHECK.md`).
- **The internal data place („Belső tárhely")** (`internal/localapi/internalplace.go`): when the record
`/var/lib/felhom-agent/internal-place.json` names a filesystem, the agent mounts it at `/mnt/belso` by fs-UUID like an
enrolled drive (startup + every 20 s tick) and records it `enrolled`; `GET /disks` reports it with kind `internal`.
It never takes `/mnt/belso` from a household drive, and eject/decommission/format of it are refused (device-mapper on
the system disk). Its identity is the system disk's, so build A never counts a copy to `sys_drive` as a second copy.
- **Root helper `configs/felhom-internal-place`** (new in the bundle): `plan|create <planned GiB>` — the ruled size rule
(reserve max(16 GiB, 10 %) ≤ 48 GiB, made only when ≥ 20 GiB, nothing promised beyond pool − reserve), the only
lvcreate/mkfs of a system-disk volume; formats only a `felhom-belso` with no signature. Run by the installer (1.33.0)
and by the new signed op.
- **Signed op `add_internal_place`** (old one-disk boxes, `16` §9): no parameters (the size is the rule's), destructive
class (operator key). Runs the helper with `create 0`, writes the record, mounts.
- **Partitions as places** (`internal/storage/partitions.go`): `GET /disks/candidates` lists each partition of a free
disk as empty / filesystem / in_use. A data-bearing part's wipe id binds disk id + part number + fs-UUID
(`byid:<…-partN>+fsuuid:<uuid>`); both must name the same device at format time.
- **`felhom-mkfs-guarded`** (REVERTED in v0.158.1): a SIBLING part that is Felhom's own drive no longer blocked formatting
the free part next to it. First tests of the wrapper: `configs/test_felhom_mkfs_guarded.py`.
- `configs/` changed (sudoers `FELHOM_INTERNALPLACE`, the helper, the wrapper, the bundle list) — **a config-bundle job
is needed** with the agent update. **Not vouched** (the operator's act).
- Released: tag `v0.158.0` (code `912f7c1`), sha256 `a3589d1d…2caba`, bundle `17066273…fb5c76`. **Superseded by v0.158.1 —
do not vouch** (its mkfs wrapper trusted a sibling claim the agent can fake; its helper failed open).
## v0.157.0 — every storage place knows its physical disk(s); POST /disks/same-disk (2026-10-15) ## v0.157.0 — every storage place knows its physical disk(s); POST /disks/same-disk (2026-10-15)
Design: felhom.eu `architecture/16-storage-physical-identity.md` §3, §4, §10 (operator rulings D1–D6, 2026-10-15). Spike: Design: felhom.eu `architecture/16-storage-physical-identity.md` §3, §4, §10 (operator rulings D1–D6, 2026-10-15). Spike:
+2 -1
View File
@@ -56,7 +56,8 @@ This is in the core because breaching it is how this component stops being audit
gate in its `GATES` table (that table is the list); the shared ones — `reuse_refs_check`, gate in its `GATES` table (that table is the list); the shared ones — `reuse_refs_check`,
`instructions_gate`, `observations_gate` — are the copies in `felhom.eu/scripts/`, never copied into `instructions_gate`, `observations_gate` — are the copies in `felhom.eu/scripts/`, never copied into
this repo (a copy recreates the drift they detect; an absent sibling clone FAILS). `--fast` selects the this repo (a copy recreates the drift they detect; an absent sibling clone FAILS). `--fast` selects the
gates touching no network and no container runtime, and skips `published` (network), naming it. **A missing gate is a FAILURE, never a skip.** gates touching no network and no container runtime, and skips `published` (network) and `gate-decoys` (the decoy
suite, ~20 s; R-1116), naming them — CI runs the full set. **A missing gate is a FAILURE, never a skip.**
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is **The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
**per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS **per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
+42
View File
@@ -1,5 +1,47 @@
# CONTEXT — felhom-agent working state # CONTEXT — felhom-agent working state
> **2026-10-22/23 night — D30 on `main` (`ae69ad4`), unreleased; the D4 patches v2 ready.** Operator ruling D30
> (2026-10-22): a handshake stamped in the future for 15 min → a new WG key pair (`wgtunnel/rekey.go`; old key kept as
> `wg/private.key.prev`), registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the
> key on ep0). Live on a test box (felhom.eu `audits/night-2026-10-23/P2-D30/`): with ep0 free to start a handshake the
> tunnel heals by itself (no re-key, correct); with ep0's starts dropped it stays dead and the agent re-keyed at 15 min.
> The interface switches at the next desired-state fetch. Gap: a reboot forgets the future stamp (R-1223). The D4 bundle
> patch v2 (`P3-review/felhom-agent-bundle-D4-v2.patch`): `try-reload-or-restart`, an `sshd -T` read-back reported as
> `sshd_dropin_not_in_force`, a reload in undo. Next agent release = D30 + D4 v2 + R-1204's field, the operator vouches.
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
> `ff4e6dc5…99d3b5`, bundle `485f36c9…12c47`. R-1083 left out on purpose: an agent key for 9202 would make its
> controller book whole-guest backups through the agent, one tier writing to ep0 — the R-1083 row says what is needed.
> **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978,
> R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with
> `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn
> positions; only those two words are held, 30-min window; a claim without it keeps the R-978 order). Vouch: version
> 0.160.0, sha256 `7ebb547f…dfca1`, bundle `6b060901…e202f`. After it, unreleased: `4545819` (two texts). felhom.eu
> `REPORT-day-2026-10-19.md`.
> **2026-10-18 (night) — UNRELEASED on main: R-1093, R-978, R-973** (release + vouch: the operator's). R-1093 an empty
> partition is blank (lsblk's PTTYPE on a partition is the parent's; set aside only on a clean blkid exit 0 with a
> matching `PART_ENTRY_SCHEME`; an unparsable lsblk is unprobed). R-978 the escrow package reaches the hub only at the
> claim (ceremony output v2, `PendingUpload`; `EscrowCeremonyConfig.Upload`). R-973 the `POST /backup` cadence cap.
> **Decision (decided by CC unattended — operator may reverse): the cap's numbers.** Q: how many whole-guest runs may
> the guest ask for? Options: (a) 2 per 24 h on the daily tier — keeps one local point older than a day, but refuses
> the NIGHT after a press (R-899's night can fall inside the same 24 h); (b) 3 per 24 h + 1 per 6 days on the weekly
> tier — the schedule never refused, a second press in a day refused, the local tier still replaceable in a day;
> (c) none — status quo. Chose (b): the off-site tier is the one `07` §8 row 10 relies on and it is fully protected
> for ~6 days; the local half's real fix is a time-bucketed prune (keep-daily), which is a retention change for the
> operator. No sudoers or bundle change in any of the three.
> **2026-10-17 — v0.159.1 RELEASED (0.159.0 superseded, do not vouch).** Go 1.27.2; R-1124 a decommissioned drive that is
> gone is forgotten (`ForgetRetiredDrives`, root verb `unit-forget`); R-1012 the read-only `felhom-thin-check` names an
> orphan thin device in a failed backup's text; the signed decommission prunes the guest-bind record. Bundle adds a path →
> step bundle `0.159.0-step1`. Proven on demo-hp; delivered by signed jobs on the operator's yes. Vouch: the operator's.
> **2026-10-16 — v0.158.1 RELEASED and VOUCHED (storage identity build B; 0.158.0 superseded, do not vouch).** The internal
> data place (root helper `felhom-internal-place`, the record in the state dir, mounted at /mnt/belso like a drive), the
> signed `add_internal_place`, partitions in `/disks/candidates` (`formattable` only while nothing on the disk is mounted),
> the partition wipe id `byid:<…-partN>+fsuuid:<uuid>`, the strict mkfs wrapper. Delivered through step bundle
> `0.158.1-step1` (a bundle that adds a path, R16). Next: R-1093 (an empty part's probe reads the parent's partition table).
> **2026-10-15 (night) — v0.157.0 RELEASED, NOT VOUCHED (storage identity build A, felhom.eu `16` §13).** Every storage > **2026-10-15 (night) — v0.157.0 RELEASED, NOT VOUCHED (storage identity build A, felhom.eu `16` §13).** Every storage
> place carries `kind`, `physical_disks`, `identity` (+ reason) in `GET /disks` and the hub report; `POST /disks/same-disk` > place carries `kind`, `physical_disks`, `identity` (+ reason) in `GET /disks` and the hub report; `POST /disks/same-disk`
> answers the controller (v0.317.0, MinAgent 0.157.0). Computed on every read, never cached. Delivered by signed jobs to > answers the controller (v0.317.0, MinAgent 0.157.0). Computed on every read, never cached. Delivered by signed jobs to
+28 -35
View File
@@ -1,43 +1,36 @@
# REPORT — v0.157.0: every storage place knows its physical disk(s); POST /disks/same-disk (2026-10-15) # REPORT — day 2026-10-23: agent 0.163.0 released, delivered to the three test boxes, vouched by the operator
Written in ASD-STE100 Simplified Technical English. Design read first: felhom.eu `architecture/16-storage-physical-identity.md` **What:** the day brief's Part 2. Design read: `03-host-agent.md` (the tunnel, the bundle), `11` §5.4.2 (the step
(§3, §4, §8, §10; ACCEPTED 2026-10-15, §13 = what was built), `03-host-agent.md`. Task: storage identity build A (operator bundle, R-880).
rulings D1–D6). Controller counterpart: felhom-controller v0.317.0 (MinAgent 0.157.0); hub v0.154.0 shows the fields.
## What changed ## Release
- `internal/storage/identity.go`: the physical disk(s) under a place, from `/sys`, `/run/udev/data`, `/proc/mounts` only - Code: `1109092` (D4 v2: the sshd drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`,
(no block I/O). `proven`, or `unknown` with `vm` / `network` / `usb_bridge` / `weak_wwn` / `no_serial`. `PermitRootLogin prohibit-password`, `try-reload-or-restart`, an `sshd -T` read-back; RootKeyOnly 6 tests). CI 2246
- `GET /disks` rows and the hub report's `storage_targets` gain `kind`, `physical_disks`, `identity`, `identity_reason`. success. CHANGELOG `b4721ee` (v0.163.0 header, both sha256), CI 2249 success.
- `POST /disks/same-disk` `{a, b}` → `{different, proven, reason, a, b}`; scoped like every `/disks` call; a path the agent - `scripts/release-agent.sh 0.163.0`: binary sha256 `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c`,
did not mount itself is `unmapped`. config bundle sha256 `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8`; tag `v0.163.0`; verified by
- `configs/` unchanged. an independent download.
- **Vouched by the operator** 07:58:46 UTC — read back from the hub DB (with its WAL) and from the running hub's
`/configuration` page: agent 0.163.0, both sha256 as above.
## Tests and red-proofs ## Delivery — three signed jobs per box (felhom-opsign, key felhom-op-1)
Fixture `/sys` + udev trees for every spike case and five same-disk handler tests. Seen failing first: the LVM thin The D4 drop-in is a NEW bundle path, and the boxes ran bundle 0.160.0 (sha256 `6b0609…`), so: `agent_update 0.163.0`
„diamond" (tmeta/tdata under one pool) read as a failure — fixed by counting a seen node once. Broken on purpose and seen → `agent_config_update 0.163.0-step1` (built with `scripts/build-step-bundle.py` from 0.160.0: only
failing: the VM rule, the vendor-less WWN rule, the own-mount rule (an absent drive), the `sys_drive` mapping. `felhom-os-apply` replaced; sha256 `6cbc0669…`, uploaded and read back) → `agent_config_update 0.163.0`.
`python3 scripts/agent_gates.py`: all OK.
## Release and delivery | Box | agent | bundle | self-check | swap = RAM (pool guest 9201) | drop-in, `sshd -T` | sshd reloaded | key login | felhom-sshd |
|---|---|---|---|---|---|---|---|---|
| Tester 1 | 0.163.0 | 0.163.0 | ok | 4096 / 4096 | in place, `without-password` | 09:27:38 SIGHUP | ok | active, listening, `sshd -t` ok |
| demo-hp | 0.163.0 | 0.163.0 | ok | 25898 / 25898 | in place, `without-password` | 09:19:44 SIGHUP | ok | active, listening, `sshd -t` ok |
| demo-felhom | 0.163.0 | 0.163.0 | ok | 12288 / 12288 | in place, `without-password` | 09:24:42 SIGHUP | ok | active, listening, `sshd -t` ok |
- `scripts/release-agent.sh 0.157.0`: tag `v0.157.0`, sha256 `addd4041…0aef`, bundle `0604bc60…397a`. **Not proven, said plainly:** the real root password over SSH was not tried (CC does not hold it); a dummy password is
- Delivered by signed `agent_update` jobs to Tester 1, demo-hp and the N100 (the three test boxes). **NOT VOUCHED** — the refused, but OpenSSH logs that only as „Failed password", so the policy proof is `sshd -T` + the reload. felhom-sshd
operator's act. Until the vouch, the global floor cannot declare MinAgent 0.157.0. (:8822) admits only the operator peer over the tunnel, so a connection from DooPlex times out by design. The console
login is not touched by the drop-in (`passwd -S root` = P on all three); not tried at a physical console.
## Live proof (method: the agent's local API on each PVE host, with the guest's own token) Installer 1.36.0 (the uninstall removes the drop-in) was published the same day (felhom.eu). R-982 and R-1204 closed.
Evidence: felhom.eu `documentation/audits/day-2026-10-23/07-agent-0163-boxes.txt`, `10-vouch-readback.txt`,
| Box | `GET /disks` | same-disk: drive vs system data | `11-installer-1360.txt`.
|---|---|---|
| demo-hp | `local`/`local-lvm` proven (SATA SSD), NVMe drive proven (`eui:`), PBS `network` | different, proven |
| N100 | system proven (SATA SSD serial), USB drive proven (the bridge passes the real serial) | different, proven |
| Tester 1 (VM) | every place `unknown/vm` | not proven (`vm`) |
On all three: system data vs `/var/lib/docker` → same disk, proven; a drive name that is not mounted → `unmapped`. The hub
stored the same fields (read from its database with the WAL). Evidence: felhom.eu `audits/storage-identity-buildA-2026-10-15/`.
## Observations
1. FILED: R-1083 — scratch box 9202 has no `local_api` in its `bootstrap.json`, so it cannot use any agent call.
2. NOT-A-FINDING: `usb_bridge` is built from the design's rule and was not seen on a real enclosure; that is R-1077, already open.
+7 -4
View File
@@ -15,13 +15,14 @@
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called | | `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) | | `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle | | ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key \| controller-image <vmid>` (the last reads the ref on stdin, R-861 (a) A1) | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line | | `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| unit-forget <name> (v0.159.0, R-1124) \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key \| controller-image <vmid>` (the last reads the ref on stdin, R-861 (a) A1) | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control | | `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) | | `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself | | `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
| `osupdate.Leg.SendUnsent` / `lockPass` (v0.144.0, R-868) | internal/osupdate/unsent.go | `(ctx) int` | an OS-pass report the agent never sent (killed mid-pass): the wrapper keeps `report-<run>-<layer>-apply.json` beside the plan; the agent deletes it once the hub has it | any new caller that runs an apply pass must hold `lockPass` (flock, across processes) — the sender must never take a running pass's copy | | `osupdate.Leg.SendUnsent` / `lockPass` (v0.144.0, R-868) | internal/osupdate/unsent.go | `(ctx) int` | an OS-pass report the agent never sent (killed mid-pass): the wrapper keeps `report-<run>-<layer>-apply.json` beside the plan; the agent deletes it once the hub has it | any new caller that runs an apply pass must hold `lockPass` (flock, across processes) — the sender must never take a running pass's copy |
| `osupdate.LoadSavedBlock` (v0.144.0, R-866) | internal/osupdate/leg.go | `(planDir) (block, savedAt, ok)` | the hub's newest os_update block as the daemon last received it (`os-update-block.json`) | the debug pass uses it ONLY when the hub cannot be reached, and says so in its header; no saved block → no pass | | `osupdate.LoadSavedBlock` (v0.144.0, R-866) | internal/osupdate/leg.go | `(planDir) (block, savedAt, ok)` | the hub's newest os_update block as the daemon last received it (`os-update-block.json`) | the debug pass uses it ONLY when the hub cannot be reached, and says so in its header; no saved block → no pass |
| `dpkg_state()` / `DPKG_STATE_SCRIPT` (v0.145.0, R-876) | configs/felhom-os-apply | `audit, journal = self.dpkg_state()` | dpkg's state in ONE call: `--audit` AND the update journal | never gate a repair on `--audit` alone — a crash leaves only the journal (measured); keep it one call (R-845) | | `dpkg_state()` / `DPKG_STATE_SCRIPT` (v0.145.0, R-876) | configs/felhom-os-apply | `audit, journal = self.dpkg_state()` | dpkg's state in ONE call: `--audit` AND the update journal | never gate a repair on `--audit` alone — a crash leaves only the journal (measured); keep it one call (R-845) |
| `thincheck.Checker.Check` + `Describe` (v0.159.0, R-1012) | internal/thincheck/thincheck.go | `Check(ctx, vmid) []Finding` | name an ORPHAN thin device (pool holds it, LVM does not) in a whole-guest backup failure — `BackupRunner.SetThinCheck` | READ-ONLY by design: the root helper `felhom-thin-check <vg> <pool>` sends only reserve/release_metadata_snap; an orphan needs two reads; fails open (never stops a backup). The repair stays human |
| `guesthook.SnippetReady` / `Register` (v0.146.0, R-861) | internal/guesthook/install.go | `SnippetReady(path) error` | is the pre-start hook (a FIXED file from the bundle) in place — register only then | The agent never installs the hook (Proxmox runs it as root); a missing hookscript stops a guest start, so never `Register` without `SnippetReady` | | `guesthook.SnippetReady` / `Register` (v0.146.0, R-861) | internal/guesthook/install.go | `SnippetReady(path) error` | is the pre-start hook (a FIXED file from the bundle) in place — register only then | The agent never installs the hook (Proxmox runs it as root); a missing hookscript stops a guest start, so never `Register` without `SnippetReady` |
### Disk / format safety (role gates, durable IDs, format guards) ### Disk / format safety (role gates, durable IDs, format guards)
@@ -44,7 +45,7 @@
| `storage.IdentityResolver` / `Relation` / `MountFor` | internal/storage/identity.go | `HostPath(path) PlaceIdentity`, `Device(dev) PlaceIdentity`, `Relation(a, b) (different, proven, reason)` | physical-disk identity of a storage place (felhom.eu `16-storage-physical-identity.md` §3, v0.157.0) | Reads /sys + /run/udev/data + /proc/mounts only (no block I/O). Judge a drive ONLY on its own mount (`localapi.ownMountIdentity`) — never the containing fs, or an absent drive reads as the root disk. A shared disk key is "same, proven" even when unknown. Pinned by identity_test.go (fixture trees) + samedisk_test.go | | `storage.IdentityResolver` / `Relation` / `MountFor` | internal/storage/identity.go | `HostPath(path) PlaceIdentity`, `Device(dev) PlaceIdentity`, `Relation(a, b) (different, proven, reason)` | physical-disk identity of a storage place (felhom.eu `16-storage-physical-identity.md` §3, v0.157.0) | Reads /sys + /run/udev/data + /proc/mounts only (no block I/O). Judge a drive ONLY on its own mount (`localapi.ownMountIdentity`) — never the containing fs, or an absent drive reads as the root disk. A shared disk key is "same, proven" even when unknown. Pinned by identity_test.go (fixture trees) + samedisk_test.go |
| `localapi.Server.EnsureInternalPlace` / `InternalPlaceRecord` | internal/localapi/internalplace.go | `EnsureInternalPlace(ctx)` (startup + 20 s tick); `ReadInternalPlace` / `WriteInternalPlace` | the one-disk box's internal data place (felhom.eu `16` §5, v0.158.0): mounts it at /mnt/belso by fs-UUID like an enrolled drive, records it `enrolled`; the controller binds + registers it | Never takes /mnt/belso from a household drive (a foreign mount, or a unit naming another UUID, waits). Never formats/ejects (dm on the system disk → role system). The ONLY maker is the root helper `configs/felhom-internal-place` (installer + signed `add_internal_place`). Pinned by internalplace_test.go | | `localapi.Server.EnsureInternalPlace` / `InternalPlaceRecord` | internal/localapi/internalplace.go | `EnsureInternalPlace(ctx)` (startup + 20 s tick); `ReadInternalPlace` / `WriteInternalPlace` | the one-disk box's internal data place (felhom.eu `16` §5, v0.158.0): mounts it at /mnt/belso by fs-UUID like an enrolled drive, records it `enrolled`; the controller binds + registers it | Never takes /mnt/belso from a household drive (a foreign mount, or a unit naming another UUID, waits). Never formats/ejects (dm on the system disk → role system). The ONLY maker is the root helper `configs/felhom-internal-place` (installer + signed `add_internal_place`). Pinned by internalplace_test.go |
| `configs/felhom-internal-place` | configs/ (bundle → /usr/local/sbin) | `plan\|create <planned GiB>` | the ruled size rule (reserve max(16, 10 %) ≤ 48 GiB; ≥ 20 GiB or nothing) + the only lvcreate/mkfs of a system-disk volume | Formats only a felhom-belso with NO signature; refuses a foreign one; never resizes/removes. Sudoers: `^(plan\|create) 0$` only. Pinned by configs/test_felhom_internal_place.py | | `configs/felhom-internal-place` | configs/ (bundle → /usr/local/sbin) | `plan\|create <planned GiB>` | the ruled size rule (reserve max(16, 10 %) ≤ 48 GiB; ≥ 20 GiB or nothing) + the only lvcreate/mkfs of a system-disk volume | Formats only a felhom-belso with NO signature; refuses a foreign one; never resizes/removes. Sudoers: `^(plan\|create) 0$` only. Pinned by configs/test_felhom_internal_place.py |
| `storage.candidatePartitions` / `CandidatePartition` | internal/storage/partitions.go | `CandidateDisk.Partitions` | partitions as places (`16` §6): empty / filesystem / in_use per part of a free disk | A part with children or a member type is never listed; a raw /mnt mount counts as Felhom's only when corroborated (felhomOwnedMounts). Pinned by partitions_test.go | | `storage.candidatePartitions` / `CandidatePartition` | internal/storage/partitions.go | `CandidateDisk.Partitions` | partitions as places (`16` §6): empty / filesystem / in_use per part of a free disk; `Formattable` only while no node of the disk is mounted | A part with children or a member type is never listed; a raw /mnt mount counts as Felhom's only when corroborated (felhomOwnedMounts) — for LISTING only: the root mkfs wrapper trusts no such claim (v0.158.1). Pinned by partitions_test.go |
| partition wipe id `byid:<…-partN>+fsuuid:<uuid>` | internal/storage/durable_device.go | `DeviceDurableID` / `ResolveDurableDevice` | a data-bearing PART binds disk id + part number + filesystem (`16` §6) | Both halves must resolve to the SAME device or the wipe refuses. Whole disks keep the plain id. Pinned by partition_durable_test.go | | partition wipe id `byid:<…-partN>+fsuuid:<uuid>` | internal/storage/durable_device.go | `DeviceDurableID` / `ResolveDurableDevice` | a data-bearing PART binds disk id + part number + filesystem (`16` §6) | Both halves must resolve to the SAME device or the wipe refuses. Whole disks keep the plain id. Pinned by partition_durable_test.go |
| `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process | | `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process |
| `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` | | `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` |
@@ -56,6 +57,7 @@
| `SudoHostOps.EnsureMount` | internal/storage/hostops.go | `EnsureMount(ctx, MountSpec) error` | persistent by-UUID systemd .mount | Validate→render→stage→`install`→`enable --now`; idempotent | | `SudoHostOps.EnsureMount` | internal/storage/hostops.go | `EnsureMount(ctx, MountSpec) error` | persistent by-UUID systemd .mount | Validate→render→stage→`install`→`enable --now`; idempotent |
| `SudoHostOps.Unmount` | internal/storage/hostops.go | `Unmount(ctx, where) error` | detach a mount unit | DESTRUCTIVE — caller MUST have gated it; does not self-authorize | | `SudoHostOps.Unmount` | internal/storage/hostops.go | `Unmount(ctx, where) error` | detach a mount unit | DESTRUCTIVE — caller MUST have gated it; does not self-authorize |
| `SudoHostOps.ReassertEnrolledMounts` | internal/storage/hostops.go | `ReassertEnrolledMounts(ctx)` | reboot remount (re-resolve by UUID) | Re-asserts unless mounted AND enabled (`shouldReassertMount`); skips absent UUIDs | | `SudoHostOps.ReassertEnrolledMounts` | internal/storage/hostops.go | `ReassertEnrolledMounts(ctx)` | reboot remount (re-resolve by UUID) | Re-asserts unless mounted AND enabled (`shouldReassertMount`); skips absent UUIDs |
| `SudoHostOps.ForgetRetiredDrives` (v0.159.0, R-1124) | internal/storage/forget_retired.go | `ForgetRetiredDrives(ctx, IntentReader) int` | drop the host mount unit of a DECOMMISSIONED drive that is GONE (start-up + the 20 s tick) | Only decommissioned + absent + unmounted; a present decommissioned drive keeps its unit (H3 re-enroll). The file goes only through `felhom-priv-apply unit-forget`, which re-checks all three as root |
| `GuestBinder.AttachDrive` / `DetachDrive` | internal/localapi/intermediary.go | `AttachDrive(ctx, vmid, where) (guestPath, err)` | live drive hot-swap under `/mnt/felhom-drives` | Normalizes to EXACTLY ONE bind via `countHostMounts` (converges double-binds); force re-bind when guest can't see it | | `GuestBinder.AttachDrive` / `DetachDrive` | internal/localapi/intermediary.go | `AttachDrive(ctx, vmid, where) (guestPath, err)` | live drive hot-swap under `/mnt/felhom-drives` | Normalizes to EXACTLY ONE bind via `countHostMounts` (converges double-binds); force re-bind when guest can't see it |
| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness | | `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from | | `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
@@ -112,8 +114,8 @@
| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled | | `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability | | `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
| `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew | | `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew |
| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v1) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) | | `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v2, R-978: json mode returns the package in `PendingUpload` instead of uploading) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) |
| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged | | localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); **R-978: the new package reaches the hub only at the claim, before R is handed out (`EscrowCeremonyConfig.Upload`); TTL and a new start wait while that upload is in flight — an abandoned ceremony leaves the previous package current**; **R-1151: a claim with `{"defer_upload":true}` hands out R + two word positions and keeps the package until `POST /escrow/ceremony/confirm` brings those two words back (30-min window, only the two words kept, never R); a claim without it keeps the R-978 order (old controllers)**; subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` | | `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
| `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) | | `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) |
| `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker | | `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker |
@@ -172,6 +174,7 @@
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. | | `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. | | `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. | | `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
| localapi backup cadence cap (R-973) | internal/localapi/backup_cap.go | `backupCadenceCap(cadence)`, `Server.capRefuses`, `Server.BackupCapRefusal()` | POST /backup refuses (429) a run the `07` §6 schedule would never ask for: ≤24 h tier → 3 successes per rolling 24 h; longer tier → 1 per max(cadence−24 h, cadence/2) | Counts SUCCESSES only, from this process AND `BackupSuccessState.RecentBackupSuccesses` (so a restart does not reset it); refusals saved (`RecordBackupRefusal`) and raised for 24 h as the CRITICAL host-report status `localapi:backup-cadence-cap` (`backupCapStatus` in main) → the hub's existing `agent_capability_degraded`. Tests `TestBackupCap_*`. |
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. | | `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. | | `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). | | `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
+67
View File
@@ -0,0 +1,67 @@
package main
import (
"go/ast"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
)
// R-973: a cadence-cap refusal becomes a CRITICAL degraded status — what the hub's existing
// agent_capability_degraded event (operator-only) keys on — and clears when the refusal ages out.
func TestBackupCapStatus_RefusalIsCriticalDegraded(t *testing.T) {
at := time.Date(2026, 10, 18, 3, 0, 0, 0, time.UTC)
s := backupCapStatus(func() (string, int, time.Time, bool) { return "felhom-pbs", 9201, at, true })
if !s.Critical || s.Status != capability.StatusDegraded || !strings.Contains(s.Reason, "9201") || !strings.Contains(s.Reason, "felhom-pbs") {
t.Fatalf("a refusal must raise a critical degraded status naming the guest and the tier: %+v", s)
}
if s := backupCapStatus(func() (string, int, time.Time, bool) { return "", 0, time.Time{}, false }); s.Status != capability.StatusOK || !s.Critical {
t.Fatalf("no refusal → ok (and still critical, so a later refusal alerts): %+v", s)
}
if s := backupCapStatus(nil); s.Status != capability.StatusOK {
t.Fatalf("no local API → ok: %+v", s)
}
}
// Wiring (the built-but-never-wired class): runDaemon's capability probe appends backupCapStatus, and the
// late-bound source is set from the local API server.
//
// COMPANION RED-PROOF (observed 2026-10-18): delete `backupCapRefusal = localSrv.BackupCapRefusal` →
// "runDaemon never binds backupCapRefusal to the local API server". Restored.
func TestR973_CapRefusalIsWiredIntoTheReport(t *testing.T) {
_, f := parseMain(t)
var appended, bound bool
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "runDaemon" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.CallExpr:
if id, ok := x.Fun.(*ast.Ident); ok && id.Name == "backupCapStatus" && len(x.Args) == 1 {
if a, ok := x.Args[0].(*ast.Ident); ok && a.Name == "backupCapRefusal" {
appended = true
}
}
case *ast.AssignStmt:
if len(x.Lhs) == 1 && len(x.Rhs) == 1 {
if l, ok := x.Lhs[0].(*ast.Ident); ok && l.Name == "backupCapRefusal" {
if sel, ok := x.Rhs[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "BackupCapRefusal" {
bound = true
}
}
}
}
return true
})
}
if !appended {
t.Fatal("runDaemon's capability probe does not call backupCapStatus(backupCapRefusal)")
}
if !bound {
t.Fatal("runDaemon never binds backupCapRefusal to the local API server")
}
}
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"context"
"encoding/json"
"go/ast"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// R-978, the root half: the controller-driven ceremony (json mode, the FELHOM_ESCROW argv with
// `--upload`) must hand its package back instead of sending it — the daemon sends it at the claim.
//
// COMPANION RED-PROOF (observed 2026-10-18): set `deferUpload: false` in escrowOptsFromCLI → this fails
// with "json mode must defer the upload (R-978): got deferUpload=false", and the delivery test's json leg
// then sends the package at create time ("put called 1 times"). Restored.
func TestEscrowOptsFromCLI_JSONModeDefersUpload(t *testing.T) {
// The FELHOM_ESCROW argv: --selftest=escrow-create --upload --output=json.
if o := escrowOptsFromCLI("", false, false, true, "", true); !o.deferUpload || !o.upload {
t.Fatalf("json mode must defer the upload (R-978): got deferUpload=%v upload=%v", o.deferUpload, o.upload)
}
// Text mode (an operator at a terminal) keeps the immediate upload; no --upload means no upload at all.
if o := escrowOptsFromCLI("", false, false, true, "", false); o.deferUpload {
t.Fatal("text mode must upload at once")
}
if o := escrowOptsFromCLI("", false, false, false, "", true); o.deferUpload {
t.Fatal("without --upload there is nothing to defer")
}
}
func TestDeliverEscrowPackage_JSONModeSendsNothing(t *testing.T) {
cfg := config.Config{}
cfg.Hub.URL, cfg.Hub.HostID, cfg.Hub.APIKey = "https://hub.invalid", "host-1", "k"
res := escrow.CreateResult{Blob: []byte("ciphertext"), KeyFingerprint: "f2:87", Posture: escrow.PostureZeroKnowledge, IdentityBlob: []byte("id")}
for _, jsonMode := range []bool{true, false} {
calls := 0
var sent []byte
put := func(_ context.Context, _ config.Config, body []byte) error { calls++; sent = body; return nil }
o := escrowOptsFromCLI("", false, false, true, "", jsonMode)
pending, uploaded, err := deliverEscrowPackage(context.Background(), cfg, res, "abc", o.deferUpload, put)
if err != nil {
t.Fatal(err)
}
if jsonMode {
if calls != 0 || uploaded {
t.Fatalf("json mode: put called %d times (uploaded=%v), want 0 — the package must wait for the claim", calls, uploaded)
}
var m map[string]any
if json.Unmarshal(pending, &m) != nil || m["blob_b64"] == "" || m["identity_blob_b64"] == "" || m["restic_pw_sha256"] != "abc" {
t.Fatalf("json mode: the pending package is not the hub upload body: %s", pending)
}
} else if calls != 1 || !uploaded || pending != nil || len(sent) == 0 {
t.Fatalf("text mode: put called %d times, uploaded=%v, pending=%q — want one immediate upload", calls, uploaded, pending)
}
}
// A hub that is not configured fails in both modes (as the pre-R-978 upload did).
if _, _, err := deliverEscrowPackage(context.Background(), config.Config{}, res, "", true, nil); err == nil {
t.Fatal("an unconfigured hub must fail the ceremony even when the upload is deferred")
}
}
// R-978 wiring (the built-but-never-wired class): runDaemon's localapi.EscrowCeremonyConfig carries an
// Upload that calls putEscrowBody. Without it every claim of a new package answers 503 and no customer
// can finish a ceremony.
//
// COMPANION RED-PROOF (observed 2026-10-18): delete the `Upload:` line in runDaemon → this fails with
// "runDaemon's EscrowCeremonyConfig has no Upload calling putEscrowBody". Restored.
func TestR978_CeremonyUploadIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
if !callsWithin(f, "main")["runDaemon"] {
t.Fatal("main → runDaemon is broken — the path this test asserts is not the live one")
}
wired := false
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "runDaemon" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
cl, ok := n.(*ast.CompositeLit)
if !ok {
return true
}
sel, ok := cl.Type.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "EscrowCeremonyConfig" {
return true
}
for _, el := range cl.Elts {
if kv, ok := el.(*ast.KeyValueExpr); ok {
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "Upload" {
ast.Inspect(kv.Value, func(n ast.Node) bool {
if ce, ok := n.(*ast.CallExpr); ok {
if id, ok := ce.Fun.(*ast.Ident); ok && id.Name == "putEscrowBody" {
wired = true
}
}
return true
})
}
}
}
return true
})
}
if !wired {
t.Fatal("runDaemon's EscrowCeremonyConfig has no Upload calling putEscrowBody")
}
}
+118 -19
View File
@@ -59,6 +59,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate" "gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs" "gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage" "gitea.dooplex.hu/admin/felhom-agent/internal/storage"
"gitea.dooplex.hu/admin/felhom-agent/internal/thincheck"
"gitea.dooplex.hu/admin/felhom-agent/internal/vzpartial" "gitea.dooplex.hu/admin/felhom-agent/internal/vzpartial"
"gitea.dooplex.hu/admin/felhom-agent/internal/wgtunnel" "gitea.dooplex.hu/admin/felhom-agent/internal/wgtunnel"
) )
@@ -193,7 +194,7 @@ func main() {
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)") flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)") flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)")
flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))") flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))")
flag.BoolVar(&upload, "upload", false, "for --selftest=escrow-create: upload the opaque blob to the hub") flag.BoolVar(&upload, "upload", false, "for --selftest=escrow-create: upload the opaque blob to the hub (with --output=json: hand the package back in pending_upload — the agent daemon uploads it when the code is claimed, R-978)")
flag.StringVar(&blobPath, "blob", "", "for --selftest=escrow-consume: path to the R-wrapped escrow blob file") flag.StringVar(&blobPath, "blob", "", "for --selftest=escrow-consume: path to the R-wrapped escrow blob file")
flag.StringVar(&expectedFP, "fingerprint", "", "for --selftest=escrow-consume: the EXPECTED key fingerprint (the gate target)") flag.StringVar(&expectedFP, "fingerprint", "", "for --selftest=escrow-consume: the EXPECTED key fingerprint (the gate target)")
flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)") flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)")
@@ -403,6 +404,31 @@ func (r *gateRemounter) Remount(ctx context.Context, t storage.KnownTarget) {
// poolReadStatus probes the felhom-pool membership read (GET /pools/felhom) the stale-lock reaper's // poolReadStatus probes the felhom-pool membership read (GET /pools/felhom) the stale-lock reaper's
// ownership scoping depends on (v0.62.0, audit A1). Needs `Pool.Audit` at `/pool/felhom` — granted // ownership scoping depends on (v0.62.0, audit A1). Needs `Pool.Audit` at `/pool/felhom` — granted
// by host-install v1.9.0; on an older ACL this reports degraded and the reaper fail-safes (skips). // by host-install v1.9.0; on an older ACL this reports degraded and the reaper fail-safes (skips).
// backupCapStatusName is the host-report status the R-973 cadence cap raises.
const backupCapStatusName = "localapi:backup-cadence-cap"
// backupCapStatus turns the local-API backup cadence cap's newest refusal (R-973) into a CRITICAL
// capability status: degraded for 24 h after a refusal, so the hub raises its operator-only
// whole_guest_backup_refused event (hub v0.159.0, R-1150; agent_capability_degraded before). The Reason names the guest, the tier and the time, and
// says what to do. nil source (local API off) → ok.
func backupCapStatus(refusal func() (string, int, time.Time, bool)) capability.Status {
s := capability.Status{
Name: backupCapStatusName,
Feature: "whole-guest restore points: a guest asked for more whole-guest backups than its schedule allows, and the extra ones were refused (R-973)",
Critical: true,
Status: capability.StatusOK,
}
if refusal == nil {
return s
}
if target, vmid, at, ok := refusal(); ok {
s.Status = capability.StatusDegraded
s.Reason = fmt.Sprintf("refused a whole-guest backup of guest %d on target %q at %s — more than the schedule allows; the guest's controller may be misbehaving or compromised: check the guest",
vmid, target, at.UTC().Format(time.RFC3339))
}
return s
}
func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status { func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status {
s := capability.Status{ s := capability.Status{
Name: "pve:pool-read", Name: "pve:pool-read",
@@ -829,8 +855,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run, run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
log: logger, log: logger,
} }
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
// drConfigured: the local API server is built further down, before the loop starts.
var backupCapRefusal func() (string, int, time.Time, bool)
probeAll := func(ctx context.Context) []capability.Status { probeAll := func(ctx context.Context) []capability.Status {
out := append(capProber.Probe(ctx), poolReadStatus(ctx, px)) out := append(capProber.Probe(ctx), poolReadStatus(ctx, px))
out = append(out, backupCapStatus(backupCapRefusal))
return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...) return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...)
} }
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot // (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
@@ -905,6 +936,22 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// network and stays under the hub's 45-minute host_stale (its // network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged. // alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait}) go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
// after start so the guest has settled; a failure is logged and the next start retries.
go func() {
select {
case <-ctx.Done():
return
case <-time.After(time.Minute):
}
sctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
if n, err := localapi.EnsureGuestSwap(sctx, px, reconcile.DefaultPool, logger); err != nil {
logger.Warn("guest-swap: check skipped", "err", err)
} else {
logger.Info("guest-swap: checked", "raised", n)
}
}()
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue // Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving // (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
@@ -1091,6 +1138,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
decommIntent = intentStore decommIntent = intentStore
} }
decommExec := signedjobs.NewDecommissionExecutor(decommIntent, logger) decommExec := signedjobs.NewDecommissionExecutor(decommIntent, logger)
if guestBindStore != nil { // typed-nil guard, as above
decommExec.SetGuestBinds(guestBindStore) // v0.159.1: prune the bind record like the self-serve decommission
}
// Agent self-update (TASK D1): the agent_update executor downloads the operator-signed binary, // Agent self-update (TASK D1): the agent_update executor downloads the operator-signed binary,
// verifies it against the SIGNED sha256, and hands it to the root guarded wrapper (A/B flip + // verifies it against the SIGNED sha256, and hands it to the root guarded wrapper (A/B flip +
@@ -1193,6 +1243,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
return cfg.Escrow.PBSStorageID return cfg.Escrow.PBSStorageID
}, },
HubConfigured: cfg.Hub.URL != "" && cfg.Hub.HostID != "" && cfg.Hub.APIKey != "", HubConfigured: cfg.Hub.URL != "" && cfg.Hub.HostID != "" && cfg.Hub.APIKey != "",
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
// subprocess used, under the same per-host key, before the code is handed out.
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
DRConfigured: func() bool { DRConfigured: func() bool {
if drConfigured != nil { if drConfigured != nil {
return drConfigured() return drConfigured()
@@ -1202,6 +1255,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} }
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens) localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil { if localSrv != nil {
backupCapRefusal = localSrv.BackupCapRefusal // R-973
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace) internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go. // R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
pwMode := proxmox.RunnerMode(cfg.Privileged.Mode) pwMode := proxmox.RunnerMode(cfg.Privileged.Mode)
@@ -1425,6 +1479,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// re-assert below sees its unit. // re-assert below sees its unit.
localSrv.EnsureInternalPlace(ctx) localSrv.EnsureInternalPlace(ctx)
if mountReasserter != nil { if mountReasserter != nil {
// R-1124: a decommissioned drive that is gone loses its host mount unit first (start-up and every tick), so
// no later boot waits for it and the re-assert below no longer lists it.
mountReasserter.ForgetRetiredDrives(ctx, intentReader)
mountReasserter.ReassertEnrolledMounts(ctx) mountReasserter.ReassertEnrolledMounts(ctx)
} }
// F9: on startup (the host's bring-up/reconcile trigger), re-assert any enrolled guest data-drive // F9: on startup (the host's bring-up/reconcile trigger), re-assert any enrolled guest data-drive
@@ -1463,6 +1520,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// unit re-disabled at runtime), then re-assert the guest binds against the now-live mounts. // unit re-disabled at runtime), then re-assert the guest binds against the now-live mounts.
localSrv.EnsureInternalPlace(ctx) // `16` §5: a record written since the last tick (the signed job) localSrv.EnsureInternalPlace(ctx) // `16` §5: a record written since the last tick (the signed job)
if mountReasserter != nil { if mountReasserter != nil {
mountReasserter.ForgetRetiredDrives(ctx, intentReader) // R-1124: decommissioned + gone → unit removed
mountReasserter.ReassertEnrolledMounts(ctx) mountReasserter.ReassertEnrolledMounts(ctx)
} }
localSrv.ReassertGuestBinds(ctx) localSrv.ReassertGuestBinds(ctx)
@@ -1905,10 +1963,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
var tierRunners []*backup.BackupRunner var tierRunners []*backup.BackupRunner
var tierTargets []string var tierTargets []string
for _, t := range backupTiers { for _, t := range backupTiers {
prune := "" prune := t.PruneSpec() // primary: keep-daily=N (R-1149); an extra tier: keep-last=N; 0 → no prune
if t.KeepLast > 0 {
prune = fmt.Sprintf("keep-last=%d", t.KeepLast)
}
// Pruning a PBS target is allowed ONLY for an additional tier with an explicit keep_last // Pruning a PBS target is allowed ONLY for an additional tier with an explicit keep_last
// (the primary's target AND retention both default, so it could prune the DR by accident). // (the primary's target AND retention both default, so it could prune the DR by accident).
allowPBSPrune := !t.Primary && t.KeepLast > 0 allowPBSPrune := !t.Primary && t.KeepLast > 0
@@ -1926,7 +1981,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
Service: r, Service: r,
}) })
logger.Info("backup tier armed", "target", t.TargetID, "cadence", t.Cadence.String(), logger.Info("backup tier armed", "target", t.TargetID, "cadence", t.Cadence.String(),
"keep_last", t.KeepLast, "wait_timeout", t.WaitTimeout.String(), "keep_last", t.KeepLast, "prune", prune, "wait_timeout", t.WaitTimeout.String(),
"prune_pbs_allowed", allowPBSPrune, "primary", t.Primary) "prune_pbs_allowed", allowPBSPrune, "primary", t.Primary)
} }
// Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown // Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown
@@ -2061,8 +2116,12 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
}, },
Logger: logger, Logger: logger,
} }
// R-1012: an orphan thin device (a full host root during a snapshot) is NAMED in the backup failure the controller
// mails to the operator, through the read-only root helper. Never removed — the repair stays a human step.
thinChecker := &thincheck.Checker{API: px, Runner: &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, Logger: logger}
for _, r := range tierRunners { for _, r := range tierRunners {
r.SetBeforeBackup(func(ctx context.Context, storage string, vmid int) { cleaner.Clean(ctx, storage, vmid) }) r.SetBeforeBackup(func(ctx context.Context, storage string, vmid int) { cleaner.Clean(ctx, storage, vmid) })
r.SetThinCheck(func(ctx context.Context, vmid int) string { return thincheck.Describe(thinChecker.Check(ctx, vmid)) })
} }
srv.SetPartialSweep(func(ctx context.Context, vmid int) { srv.SetPartialSweep(func(ctx context.Context, vmid int) {
for _, t := range tierTargets { for _, t := range tierTargets {
@@ -2814,6 +2873,20 @@ type escrowCeremonyOpts struct {
offline bool offline bool
upload bool upload bool
identityBundlePath string identityBundlePath string
// deferUpload (R-978): with upload, build the hub upload body and hand it back in
// CeremonyOutput.PendingUpload instead of sending it. Set for json mode — the controller-driven
// ceremony, whose daemon uploads the package only when the customer claims the new code.
deferUpload bool
}
// escrowOptsFromCLI maps the escrow-create flags onto the ceremony options. json mode defers the upload
// (R-978, pinned by TestEscrowOptsFromCLI_JSONModeDefersUpload); text mode — an operator at a terminal,
// who sees R at once — keeps the immediate upload.
func escrowOptsFromCLI(storage string, paperkey, offline, upload bool, identityBundlePath string, jsonMode bool) escrowCeremonyOpts {
return escrowCeremonyOpts{
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
identityBundlePath: identityBundlePath, deferUpload: upload && jsonMode,
}
} }
// escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the // escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the
@@ -2964,16 +3037,36 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
ResticPwSealed: resticStaged, ResticPwSealed: resticStaged,
} }
if opts.upload { if opts.upload {
if err := uploadEscrowBlob(ctx, cfg, res, resticPwSHA256); err != nil { pending, uploaded, err := deliverEscrowPackage(ctx, cfg, res, resticPwSHA256, opts.deferUpload, putEscrowBody)
if err != nil {
// R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets // R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets
// the text shell keep the pre-extraction order (R surfaced, THEN the failure). // the text shell keep the pre-extraction order (R surfaced, THEN the failure).
return out, &escrowCeremonyErr{kind: "upload", err: err} return out, &escrowCeremonyErr{kind: "upload", err: err}
} }
out.Sum.Uploaded = true out.Sum.PendingUpload, out.Sum.Uploaded = pending, uploaded
} }
return out, nil return out, nil
} }
// deliverEscrowPackage builds the hub upload body and either sends it now (text mode) or returns it for
// the caller to send later (deferUpload — json mode, R-978: the daemon sends it when the customer claims
// the code). A hub that is not configured fails in both modes, as before: a package that can never be
// delivered is not a ceremony. put is the sender seam (putEscrowBody in production).
func deliverEscrowPackage(ctx context.Context, cfg config.Config, res escrow.CreateResult, resticPwSHA256 string, deferUpload bool,
put func(ctx context.Context, cfg config.Config, body []byte) error) (pending []byte, uploaded bool, err error) {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return nil, false, fmt.Errorf("hub not configured (url/host_id/api_key)")
}
body := escrowUploadBody(res, resticPwSHA256)
if deferUpload {
return body, false, nil
}
if err := put(ctx, cfg, body); err != nil {
return nil, false, err
}
return nil, true, nil
}
// printEscrowTextBanner prints the text-mode header line (shared by the success and the // printEscrowTextBanner prints the text-mode header line (shared by the success and the
// post-banner failure paths so the pre-extraction output stays byte-identical — including the // post-banner failure paths so the pre-extraction output stays byte-identical — including the
// posture: the historical banner always printed the DEFAULT posture, not the result's). // posture: the historical banner always printed the DEFAULT posture, not the result's).
@@ -3037,10 +3130,7 @@ func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slo
return 2 return 2
} }
out, cerr := escrowCeremony(ctx, cfg, logger, escrowCeremonyOpts{ out, cerr := escrowCeremony(ctx, cfg, logger, escrowOptsFromCLI(storage, paperkey, offline, upload, identityBundlePath, jsonMode))
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
identityBundlePath: identityBundlePath,
})
if cerr != nil { if cerr != nil {
switch cerr.kind { switch cerr.kind {
case "usage": case "usage":
@@ -3078,6 +3168,9 @@ func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slo
if out.Sum.Uploaded { if out.Sum.Uploaded {
fmt.Fprintln(os.Stderr, " uploaded the opaque blob(s) to the hub (host record); the hub cannot open them") fmt.Fprintln(os.Stderr, " uploaded the opaque blob(s) to the hub (host record); the hub cannot open them")
} }
if len(out.Sum.PendingUpload) > 0 {
fmt.Fprintln(os.Stderr, " package prepared, NOT uploaded: the agent uploads it when the new code is claimed (R-978)")
}
wire := out.Sum wire := out.Sum
wire.RecoveryCode = out.R wire.RecoveryCode = out.R
if err := json.NewEncoder(os.Stdout).Encode(wire); err != nil { if err := json.NewEncoder(os.Stdout).Encode(wire); err != nil {
@@ -3231,13 +3324,10 @@ type escrowUploadRequest struct {
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"` ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
} }
// uploadEscrowBlob PUTs the opaque blob (and, for 10D, the identity blob) to // escrowUploadBody is the hub upload body for one ceremony's result: the opaque blob (and, for 10D, the
// the hub, authed with the per-host key. The hub stores ciphertext + non-secret fields; no usable // identity blob) plus non-secret fields. Ciphertext only — useless without R; no usable secret leaves
// secret leaves the agent. // the agent. putEscrowBody sends it, authed with the per-host key.
func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateResult, resticPwSHA256 string) error { func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)")
}
upReq := escrowUploadRequest{ upReq := escrowUploadRequest{
BlobB64: base64.StdEncoding.EncodeToString(res.Blob), BlobB64: base64.StdEncoding.EncodeToString(res.Blob),
KeyFingerprint: res.KeyFingerprint, KeyFingerprint: res.KeyFingerprint,
@@ -3249,6 +3339,15 @@ func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateR
upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob) upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob)
} }
body, _ := json.Marshal(upReq) body, _ := json.Marshal(upReq)
return body
}
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)")
}
url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow" url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow"
req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewReader(body)) req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewReader(body))
if err != nil { if err != nil {
+7 -1
View File
@@ -25,6 +25,7 @@
Cmnd_Alias FELHOM_MOUNT = \ Cmnd_Alias FELHOM_MOUNT = \
/usr/local/sbin/felhom-priv-apply ^unit mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$, \ /usr/local/sbin/felhom-priv-apply ^unit mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$, \
/usr/local/sbin/felhom-priv-apply ^unit-forget mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl daemon-reload, \ /usr/bin/systemctl daemon-reload, \
/usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \ /usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \ /usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
@@ -101,6 +102,11 @@ Cmnd_Alias FELHOM_VZPARTIAL = \
# ruled size rule and formats ONLY that volume, ONLY when it carries no signature. The agent calls it from the signed # ruled size rule and formats ONLY that volume, ONLY when it carries no signature. The agent calls it from the signed
# `add_internal_place` job (an old box), always with 0 GiB planned: its guest's volumes already exist. `plan` changes # `add_internal_place` job (an old box), always with 0 GiB planned: its guest's volumes already exist. `plan` changes
# nothing. The installer runs the helper as root itself (a new box). # nothing. The installer runs the helper as root itself (a new box).
# R-1012 (agent v0.159.0): the READ-ONLY orphan-thin check before and after a whole-guest backup — lists the pool's thin
# device ids (through a metadata snapshot it reserves and releases) next to LVM's. It sends no other pool message; the
# repair stays a human step. Pinned by configs/test_felhom_thin_check.py.
Cmnd_Alias FELHOM_THINCHECK = \
/usr/local/sbin/felhom-thin-check ^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0\,63} [A-Za-z0-9_+][A-Za-z0-9_.+-]{0\,63}$
Cmnd_Alias FELHOM_INTERNALPLACE = \ Cmnd_Alias FELHOM_INTERNALPLACE = \
/usr/local/sbin/felhom-internal-place ^(plan|create) 0$ /usr/local/sbin/felhom-internal-place ^(plan|create) 0$
@@ -336,4 +342,4 @@ Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \ /usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \
/usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$ /usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY, FELHOM_POWEROFF, FELHOM_VZPARTIAL, FELHOM_INTERNALPLACE felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY, FELHOM_POWEROFF, FELHOM_VZPARTIAL, FELHOM_INTERNALPLACE, FELHOM_THINCHECK
+19 -4
View File
@@ -55,15 +55,19 @@ pool_gib=$(gib_floor "$pool_b")
# Every thin volume promised in the pool (its virtual size) — except the place itself and vzdump's short-lived # Every thin volume promised in the pool (its virtual size) — except the place itself and vzdump's short-lived
# snapshots (snap_*_vzdump), which exist only during a whole-box copy and are what the reserve is for. # snapshots (snap_*_vzdump), which exist only during a whole-box copy and are what the reserve is for.
promised_b=0; have_lv=false # FAIL CLOSED: a failed or empty listing must never read as „nothing promised" (that would size the place too big).
listing=$(lvs --noheadings --nosuffix --units b -o lv_name,pool_lv,lv_size "$VG" 2>/dev/null) || die "cannot list the volumes of $VG"
pool_seen=false; promised_b=0; have_lv=false
while read -r name pl size; do while read -r name pl size; do
[[ -n "${name:-}" ]] || continue [[ -n "${name:-}" ]] || continue
[[ "$name" == "$POOL" ]] && pool_seen=true
[[ "$pl" == "$POOL" ]] || continue [[ "$pl" == "$POOL" ]] || continue
if [[ "$name" == "$LV_NAME" ]]; then have_lv=true; continue; fi if [[ "$name" == "$LV_NAME" ]]; then have_lv=true; continue; fi
[[ "$name" =~ ^snap_.*_vzdump$ ]] && continue [[ "$name" =~ ^snap_.*_vzdump$ ]] && continue
[[ "$size" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$name" [[ "$size" =~ ^[0-9]+$ ]] || die "cannot read the size of $VG/$name"
promised_b=$(( promised_b + size )) promised_b=$(( promised_b + size ))
done < <(lvs --noheadings --nosuffix --units b -o lv_name,pool_lv,lv_size "$VG" 2>/dev/null) done <<< "$listing"
$pool_seen || die "the listing of $VG does not contain the pool $POOL — refusing to guess what is promised"
promised_gib=$(( (promised_b + 1073741823) / 1073741824 )) # rounded UP: never promise a partial GiB twice promised_gib=$(( (promised_b + 1073741823) / 1073741824 )) # rounded UP: never promise a partial GiB twice
reserve=$(( pool_gib / 10 )); (( reserve < 16 )) && reserve=16; (( reserve > 48 )) && reserve=48 reserve=$(( pool_gib / 10 )); (( reserve < 16 )) && reserve=16; (( reserve > 48 )) && reserve=48
@@ -79,6 +83,17 @@ is_block() { [[ "$(stat -L -c %F "$1" 2>/dev/null)" == "block special file" ]];
fs_uuid() { blkid -p -s UUID -o value "$DEV" 2>/dev/null; } fs_uuid() { blkid -p -s UUID -o value "$DEV" 2>/dev/null; }
fs_type() { blkid -p -s TYPE -o value "$DEV" 2>/dev/null; } fs_type() { blkid -p -s TYPE -o value "$DEV" 2>/dev/null; }
fs_label() { blkid -p -s LABEL -o value "$DEV" 2>/dev/null; } fs_label() { blkid -p -s LABEL -o value "$DEV" 2>/dev/null; }
# has_signature: FAIL CLOSED. blkid -p exits 0 when it found something and 2 when it found NOTHING; any other result (an
# error, an ambivalent probe, blkid missing) must never read as „empty" — that is the only gate in front of mkfs.
has_signature() {
local rc=0
blkid -p -o value "$DEV" >/dev/null 2>&1 || rc=$?
case $rc in
0) return 0 ;;
2) return 1 ;;
*) die "cannot probe $DEV for a signature (blkid exit $rc) — refusing" ;;
esac
}
lv_gib() { gib_floor "$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$LV_NAME" 2>/dev/null | tr -d ' ')"; } lv_gib() { gib_floor "$(lvs --noheadings --nosuffix --units b -o lv_size "$VG/$LV_NAME" 2>/dev/null | tr -d ' ')"; }
if $have_lv; then if $have_lv; then
@@ -89,12 +104,12 @@ if $have_lv; then
echo "existing uuid=$(fs_uuid) gib=$(lv_gib)"; exit 0 echo "existing uuid=$(fs_uuid) gib=$(lv_gib)"; exit 0
fi fi
# A felhom-belso with NO signature: a crash between lvcreate and mkfs. Nothing can be on it — format it. # A felhom-belso with NO signature: a crash between lvcreate and mkfs. Nothing can be on it — format it.
[[ -z "$(blkid -p -o value "$DEV" 2>/dev/null)" ]] || die "$DEV carries a signature blkid cannot name — left alone" has_signature && die "$DEV carries a signature blkid cannot name — left alone"
else else
(( internal >= MIN_GIB )) || { echo "skipped only $internal GiB free for it (pool $pool_gib, reserve $reserve, promised $promised_gib, planned $PLANNED; needs $MIN_GIB)"; exit 0; } (( internal >= MIN_GIB )) || { echo "skipped only $internal GiB free for it (pool $pool_gib, reserve $reserve, promised $promised_gib, planned $PLANNED; needs $MIN_GIB)"; exit 0; }
lvcreate -q -y -V "${internal}G" -T "$VG/$POOL" -n "$LV_NAME" >/dev/null || die "lvcreate failed" lvcreate -q -y -V "${internal}G" -T "$VG/$POOL" -n "$LV_NAME" >/dev/null || die "lvcreate failed"
is_block "$DEV" || die "$DEV did not appear after lvcreate" is_block "$DEV" || die "$DEV did not appear after lvcreate"
[[ -z "$(blkid -p -o value "$DEV" 2>/dev/null)" ]] || die "the new $DEV already carries a signature — left alone" has_signature && die "the new $DEV already carries a signature — left alone"
made=created made=created
fi fi
mkfs.ext4 -q -L "$LABEL" "$DEV" >/dev/null || die "mkfs.ext4 failed on $DEV" mkfs.ext4 -q -L "$LABEL" "$DEV" >/dev/null || die "mkfs.ext4 failed on $DEV"
+7 -15
View File
@@ -58,31 +58,23 @@ fi
# 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under # 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under
# /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed. # /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed.
# Agent v0.158.0 (felhom.eu architecture/16 §6, partitions as places): a SIBLING partition that is Felhom's own # Agent v0.158.0: read from the kernel's mount table for EVERY node of the disk (lsblk's MOUNTPOINT shows one mount
# enrolled drive is mounted TWICE — raw at /mnt/<name> and under /mnt/felhom-drives (R-220). That raw mount is # per node; a bind elsewhere hid behind it), and a disk whose nodes cannot be listed REFUSES (it passed before).
# allowed only when the same device is ALSO mounted under /mnt/felhom-drives, a state only Felhom's enrolment # Deliberately NO exemption for a sibling that is Felhom's own enrolled drive (raw /mnt/<name> + /mnt/felhom-drives):
# produces. The TARGET itself mounted anywhere outside /mnt/felhom-drives, or any mount elsewhere (/srv, /media, # the only corroboration available here is a mount the agent itself can make, so trusting it would let a compromised
# /mnt/a/b), still REFUSES. Read from the kernel's mount table, every node of the disk (partitions and children). # agent unlock a host owner's disk. Consequence (felhom.eu `16` §13): the free part next to an in-use Felhom drive
# cannot be formatted; the dashboard does not offer it.
PROC_MOUNTS=/proc/mounts PROC_MOUNTS=/proc/mounts
node_mounts() { awk -v d="$1" '$1 == d { print $2 }' "$PROC_MOUNTS"; } node_mounts() { awk -v d="$1" '$1 == d { print $2 }' "$PROC_MOUNTS"; }
disk_nodes="$(lsblk -nrpo NAME "$whole" 2>/dev/null || true)" disk_nodes="$(lsblk -nrpo NAME "$whole" 2>/dev/null || true)"
[[ -n "$disk_nodes" ]] || die "cannot list the nodes of $whole — refusing" [[ -n "$disk_nodes" ]] || die "cannot list the nodes of $whole — refusing"
while read -r node; do while read -r node; do
[[ -z "$node" ]] && continue [[ -z "$node" ]] && continue
own=false
while read -r mp; do
case "$mp" in /mnt/felhom-drives/*) own=true ;; esac
done < <(node_mounts "$node")
while read -r mp; do while read -r mp; do
[[ -z "$mp" ]] && continue [[ -z "$mp" ]] && continue
case "$mp" in case "$mp" in
/mnt/felhom-drives|/mnt/felhom-drives/*) : ;; /mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
*) *) die "device (or a partition) is mounted at $mp ($node)" ;;
if [[ "$node" != "$dev" && "$own" == true && "$mp" =~ ^/mnt/[A-Za-z0-9_][A-Za-z0-9_.-]*$ ]]; then
: # a sibling that is Felhom's own enrolled drive (raw mount + managed mount)
else
die "device (or a partition) is mounted at $mp ($node)"
fi ;;
esac esac
done < <(node_mounts "$node") done < <(node_mounts "$node")
done <<< "$disk_nodes" done <<< "$disk_nodes"
+43 -3
View File
@@ -165,7 +165,8 @@ KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
# dest: (source file under configs/, mode, check, policy) # dest: (source file under configs/, mode, check, policy)
# check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile; # check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
# unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident); # unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none. # agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none;
# sshd → `sshd -t -f` on the snippet alone (R-982).
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may # policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none). # have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it. # Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
@@ -184,6 +185,9 @@ BUNDLE_KEEP = 3
BUNDLE_MAX_BYTES = 4 * 1024 * 1024 BUNDLE_MAX_BYTES = 4 * 1024 * 1024
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$") BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
OOB_DIR = "/etc/felhom-sshd" OOB_DIR = "/etc/felhom-sshd"
# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
BUNDLE_FILES = [ BUNDLE_FILES = [
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"), ("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"), ("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
@@ -191,6 +195,8 @@ BUNDLE_FILES = [
("/usr/local/sbin/felhom-backup-target-apply", "felhom-backup-target-apply", 0o755, "bash", "replace"), ("/usr/local/sbin/felhom-backup-target-apply", "felhom-backup-target-apply", 0o755, "bash", "replace"),
# R-1006 (agent v0.156.0): the only delete of backup files — a dead half-written vzdump, re-checked as root. # R-1006 (agent v0.156.0): the only delete of backup files — a dead half-written vzdump, re-checked as root.
("/usr/local/sbin/felhom-vzdump-partial-clean", "felhom-vzdump-partial-clean", 0o755, "bash", "replace"), ("/usr/local/sbin/felhom-vzdump-partial-clean", "felhom-vzdump-partial-clean", 0o755, "bash", "replace"),
# R-1012 (agent v0.159.0): the read-only orphan-thin check around a whole-guest backup (names an orphan, never removes).
("/usr/local/sbin/felhom-thin-check", "felhom-thin-check", 0o755, "bash", "replace"),
# felhom.eu architecture/16 §5 (agent v0.158.0): the internal data place — the only lvcreate/mkfs of a system-disk volume. # felhom.eu architecture/16 §5 (agent v0.158.0): the internal data place — the only lvcreate/mkfs of a system-disk volume.
("/usr/local/sbin/felhom-internal-place", "felhom-internal-place", 0o755, "bash", "replace"), ("/usr/local/sbin/felhom-internal-place", "felhom-internal-place", 0o755, "bash", "replace"),
("/usr/local/sbin/felhom-os-apply", "felhom-os-apply", 0o755, "python", "replace"), ("/usr/local/sbin/felhom-os-apply", "felhom-os-apply", 0o755, "python", "replace"),
@@ -219,6 +225,9 @@ BUNDLE_FILES = [
("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"), ("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"), ("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"), ("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
# R-982, operator ruling D4 (2026-10-22): the host's own sshd (port 22, the LAN): root by key only. The console
# password stays; felhom-sshd (the operator's door) reads nothing from sshd_config.d. Reloaded in after_install.
("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "replace"),
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"), ("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"), ("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
] ]
@@ -432,7 +441,8 @@ class Runner:
def check_content(self, kind, data): def check_content(self, kind, data):
"""Run an external syntax check on a root-only temp copy. Returns (rc, message).""" """Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
import tempfile import tempfile
cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"]}[kind] cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"],
"sshd": ["/usr/sbin/sshd", "-t", "-f"]}[kind]
with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d: with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
p = os.path.join(d, "f") p = os.path.join(d, "f")
with open(p, "wb") as f: with open(p, "wb") as f:
@@ -1862,6 +1872,7 @@ class Bundle:
def __init__(self, apply): def __init__(self, apply):
self.a, self.r = apply, apply.r self.a, self.r = apply, apply.r
self.report = apply.report self.report = apply.report
self.sshd_dropin_not_in_force = False
# ---------- reading and checking ---------- # ---------- reading and checking ----------
def load_file(self, path): def load_file(self, path):
@@ -1930,10 +1941,17 @@ class Bundle:
text = content.decode("utf-8") text = content.decode("utf-8")
except UnicodeDecodeError: except UnicodeDecodeError:
raise Refused("R18", f"{dest} is not UTF-8 text") raise Refused("R18", f"{dest} is not UTF-8 text")
if check in ("sudoers", "sh", "bash", "nft"): if check in ("sudoers", "sh", "bash", "nft", "sshd"):
rc, msg = self.r.check_content(check, content) rc, msg = self.r.check_content(check, content)
if rc != 0: if rc != 0:
raise Refused("R18", f"{dest} fails its {check} check: {msg}") raise Refused("R18", f"{dest} fails its {check} check: {msg}")
if check == "sshd":
# R-982: the drop-in's PermitRootLogin must be exactly `prohibit-password`. `no` would also shut the root key
# out (Proxmox's own node-to-node SSH and the LAN key route), `yes` would undo the ruling — both refused.
# Only that line is pinned: the file is signed like every bundle file (review A7).
vals = re.findall(r"^\s*PermitRootLogin\s+(\S+)\s*$", text, re.M | re.I)
if vals != ["prohibit-password"]:
raise Refused("R18", f"{dest} must set exactly `PermitRootLogin prohibit-password` (found {vals})")
elif check == "python": elif check == "python":
try: try:
compile(text, dest, "exec") compile(text, dest, "exec")
@@ -2042,6 +2060,8 @@ class Bundle:
self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]} self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
return 3 return 3
rep["signers_created"] = False rep["signers_created"] = False
if self.sshd_dropin_not_in_force:
rep["sshd_dropin_not_in_force"] = True # R-982 A2: written and reloaded, but sshd -T shows root passwords allowed
if bootstrap and not self.r.lexists(TRUST_SIGNERS): if bootstrap and not self.r.lexists(TRUST_SIGNERS):
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644) self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
rep["signers_created"] = True rep["signers_created"] = True
@@ -2071,6 +2091,23 @@ class Bundle:
"enable the crash guard") "enable the crash guard")
if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests: if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer") self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
# R-982: the host's sshd takes the root-key-only drop-in. The WHOLE config is tested first (a failure is a
# refusal, and the bundle undo puts the old files back); `try-reload-or-restart` keeps every open session and
# does nothing on a box whose ssh.service is not running (a socket-activated or stopped sshd reads the file
# at its next start) — a failure there must not freeze the bundle route (review 2026-10-22 night, A1).
# felhom-sshd is not touched. Then the POSITIVE check: sshd's own effective view for root. A box whose
# sshd_config has no Include of sshd_config.d, an earlier drop-in or a Match block can keep the password route
# — that is said loudly in the log and the report, never silently recorded as done (A2).
if SSHD_DROPIN in written:
self.must(["/usr/sbin/sshd", "-t"], "sshd -t (the host's whole sshd config)")
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", "user=root,host=felhom-check,addr=192.0.2.1"], 60)
eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
if rc != 0 or eff != ["without-password"]:
self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force: sshd -T says "
f"permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier drop-in "
f"or a Match block (R-982)")
self.sshd_dropin_not_in_force = True
def must(self, argv, what): def must(self, argv, what):
rc, out, err = self.r.host(argv, 120) rc, out, err = self.r.host(argv, 120)
@@ -2131,6 +2168,9 @@ class Bundle:
self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}") self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
if any(d.startswith("/etc/systemd/system/") for d in back): if any(d.startswith("/etc/systemd/system/") for d in back):
self.r.host(["systemctl", "daemon-reload"], 120) self.r.host(["systemctl", "daemon-reload"], 120)
if SSHD_DROPIN in back:
# A3: after_install may already have reloaded sshd with the drop-in; the restored state takes effect now.
self.r.host(["systemctl", "try-reload-or-restart", "ssh"], 120)
rc, _, _ = self.r.host(["visudo", "-c"], 60) rc, _, _ = self.r.host(["visudo", "-c"], 60)
self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}") self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
return back return back
+79 -1
View File
@@ -13,6 +13,8 @@ longer name the destination: each verb has a fixed source and a fixed (or strict
Verbs (each one sudoers line, exact-match pattern): Verbs (each one sudoers line, exact-match pattern):
unit <name> /var/lib/felhom-agent/units/<name> -> /etc/systemd/system/<name> (.mount | .automount) unit <name> /var/lib/felhom-agent/units/<name> -> /etc/systemd/system/<name> (.mount | .automount)
unit-forget <name> REMOVES /etc/systemd/system/<name> (R-1124): only a felhom by-UUID drive .mount unit that is
disabled, not mounted, and whose drive is absent (its /dev/disk/by-uuid link is gone)
dnsmasq <tmp> <name> /tmp/felhom-resolver-<digits>.conf -> /etc/dnsmasq.d/felhom-<...>.conf dnsmasq <tmp> <name> /tmp/felhom-resolver-<digits>.conf -> /etc/dnsmasq.d/felhom-<...>.conf
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600) wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
@@ -41,7 +43,7 @@ WG_SRC, WG_DEST = STATE + "/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config" SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op" KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
MAX_BYTES = 64 * 1024 MAX_BYTES = 64 * 1024
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key", "controller-image") VERBS = ("unit", "unit-forget", "dnsmasq", "wg", "sshd-config", "sshd-key", "controller-image")
# R-861 (a) A1 (`09` §3 decision 165): the SAME pattern as the agent's controllerImageRe (internal/localapi/ # R-861 (a) A1 (`09` §3 decision 165): the SAME pattern as the agent's controllerImageRe (internal/localapi/
# controllerswap.go) — a compromised agent cannot hand the guest's bootstrap any other image. Pinned by # controllerswap.go) — a compromised agent cannot hand the guest's bootstrap any other image. Pinned by
# configs/test_felhom_priv_apply.py ControllerImage. # configs/test_felhom_priv_apply.py ControllerImage.
@@ -51,6 +53,11 @@ CONTROLLER_IMAGE_MAX = 256
VMID_RE = re.compile(r"^[0-9]{1,9}$") VMID_RE = re.compile(r"^[0-9]{1,9}$")
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$") UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
DRIVE_UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.mount$")
DRIVE_UNIT_MARKER = "Managed by felhom-agent"
NET_UNIT_MARKER = "felhom network storage"
WANTS_DIR = UNIT_DIR + "/multi-user.target.wants"
BY_UUID_DIR = "/dev/disk/by-uuid"
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$") DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
DNSMASQ_NAME_RE = re.compile(r"^felhom-[a-z0-9][a-z0-9._-]*\.conf$") DNSMASQ_NAME_RE = re.compile(r"^felhom-[a-z0-9][a-z0-9._-]*\.conf$")
SEG = r"[A-Za-z0-9_-][A-Za-z0-9_.-]*" SEG = r"[A-Za-z0-9_-][A-Za-z0-9_.-]*"
@@ -135,6 +142,36 @@ class Host:
def read_stdin(self, limit): def read_stdin(self, limit):
return sys.stdin.buffer.read(limit + 1) return sys.stdin.buffer.read(limit + 1)
# ---- unit-forget (R-1124) ----
def read_installed(self, path):
"""The installed unit: None when absent; Refused when it is not a regular file (a symlink is never followed)."""
try:
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC)
except FileNotFoundError:
return None
except OSError as e:
raise Refused("F1", f"cannot open {path}: {e.strerror}")
with os.fdopen(fd, "rb") as f:
if not stat.S_ISREG(os.fstat(f.fileno()).st_mode):
raise Refused("F1", f"{path} is not a regular file")
return f.read(MAX_BYTES + 1)
def lexists(self, path):
return os.path.lexists(path)
def mountpoints(self):
"""Every mount point in /proc/self/mountinfo (field 5, octal escapes decoded)."""
out = set()
with open("/proc/self/mountinfo", "r", encoding="utf-8", errors="replace") as f:
for line in f:
parts = line.split(" ")
if len(parts) > 4:
out.add(re.sub(r"\\([0-7]{3})", lambda m: chr(int(m.group(1), 8)), parts[4]))
return out
def remove(self, path):
os.remove(path)
def write_guest_image(self, vmid, data): def write_guest_image(self, vmid, data):
"""As root: `pct exec <vmid> -- tee <the fixed file>` with the checked ref on stdin (no shell).""" """As root: `pct exec <vmid> -- tee <the fixed file>` with the checked ref on stdin (no shell)."""
r = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", CONTROLLER_IMAGE_FILE], r = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", CONTROLLER_IMAGE_FILE],
@@ -424,6 +461,45 @@ def controller_image(rest, host):
return 0 return 0
def unit_forget(rest, host):
"""R-1124: remove ONE felhom drive mount unit that the agent has already disabled, whose drive is gone. A drive that
is still present or still mounted keeps its unit — this verb can only forget, never unmount or take a drive away."""
try:
if len(rest) != 1 or not DRIVE_UNIT_NAME_RE.match(rest[0]) or "/" in rest[0]:
raise Refused("A1", "usage: felhom-priv-apply unit-forget mnt-<escaped path>.mount")
name = rest[0]
path = os.path.join(UNIT_DIR, name)
data = host.read_installed(path)
if data is None:
host.log(f"felhom-priv-apply: ABSENT unit-forget {path} (nothing to remove)")
return 0
text = text_of(data, path)
if DRIVE_UNIT_MARKER not in text or NET_UNIT_MARKER in text:
raise Refused("F2", f"{name} is not a felhom drive unit")
check_unit(name, text)
s = parse_ini(text, name)
what, where = s["Mount"].get("What", ""), s["Mount"]["Where"]
mu = re.match(r"^/dev/disk/by-uuid/(.+)$", what)
if not mu:
raise Refused("F2", f"{name} is not a by-UUID drive unit")
if host.lexists(os.path.join(WANTS_DIR, name)):
raise Refused("F3", f"{name} is still enabled (disable it first)")
if where in host.mountpoints():
raise Refused("F4", f"{where} is mounted")
if host.lexists(os.path.join(BY_UUID_DIR, mu.group(1))):
raise Refused("F5", f"the drive of {name} is present (filesystem UUID {mu.group(1)})")
except Refused as e:
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] unit-forget {' '.join(rest)[:120]}: {e.reason}")
return 2 if e.rule == "A1" else 3
try:
host.remove(path)
except OSError as e:
host.log(f"felhom-priv-apply: FAILED unit-forget {path}: {e}")
return 4
host.log(f"felhom-priv-apply: REMOVED unit-forget {path} (where={where})")
return 0
def main(argv, host=None): def main(argv, host=None):
host = host or Host() host = host or Host()
if argv == ["--self-check"]: if argv == ["--self-check"]:
@@ -445,6 +521,8 @@ def main(argv, host=None):
return 0 return 0
if argv and argv[0] == "controller-image": if argv and argv[0] == "controller-image":
return controller_image(argv[1:], host) return controller_image(argv[1:], host)
if argv and argv[0] == "unit-forget":
return unit_forget(argv[1:], host)
try: try:
verb, src, dest, mode, checker = plan(argv) verb, src, dest, mode, checker = plan(argv)
data = host.read_source(src) data = host.read_source(src)
+7
View File
@@ -0,0 +1,7 @@
# MANAGED BY FELHOM (the agent's config bundle) — R-982, operator ruling D4 (2026-10-22).
# The Proxmox host's OWN sshd — the listener the household network reaches on port 22: root logs in with a KEY only.
# The root password still works on the console (the break-glass password: felhom.eu runbooks/break-glass.md). The
# operator's door (felhom-sshd, port 8822, /etc/felhom-sshd/sshd_config) is a separate daemon whose config includes
# nothing from here, so it is not changed. Debian's sshd_config reads this folder BEFORE its own settings, and for
# sshd the first value wins; the bundle checks the effective value with `sshd -T` after the reload.
PermitRootLogin prohibit-password
+74
View File
@@ -0,0 +1,74 @@
#!/bin/bash
# felhom-thin-check — list the thin device ids a thin pool holds next to the ones LVM knows (R-1012). READ-ONLY: it
# names an orphan, it never removes one. Installed as /usr/local/sbin/felhom-thin-check (config bundle).
#
# felhom-thin-check <vg> <pool>
#
# A full host root while vzdump takes its snapshot can leave a thin snapshot inside the pool that LVM no longer knows
# (demo-hp 2026-10-14: `thin_ls` device 4, LVM only 1 and 2); from then on every snapshot backup of that guest fails
# („Creation of new snapshot 4 of device 2 failed") and nothing said why. The repair (`dmsetup message … delete <id>`)
# stays a HUMAN step, decided by the operator — this helper only reads.
#
# Reading the live pool's device list needs a metadata snapshot: the kernel's documented way to read metadata while
# the pool is in use (`reserve_metadata_snap`, `thin_ls --metadata-snap`, `release_metadata_snap`). It changes no data
# and no mapping; it is held for the length of one `thin_ls -o DEV` (well under a second on these pools). Rules:
# 1. the arguments are an LVM name pair; <vg>/<pool> is a thin pool (lv_attr starts with `t`);
# 2. only one run at a time (a lock file); a metadata snapshot that someone ELSE holds is never touched — the run
# refuses (exit 3) and does not release it;
# 3. LVM's list is read before AND after the pool's, so a volume created in between is not reported as an orphan
# (the caller treats an id as an orphan only when it is in the pool and in neither LVM list).
# Output, one record per line: lvm-before <id> <lv> | pool <id> | lvm-after <id> <lv> | ok
# Exit: 0 read, 1 refused (arguments / not a thin pool), 3 a metadata snapshot is already held or cannot be reserved,
# 4 the pool could not be read. Pinned by configs/test_felhom_thin_check.py.
set -u
LVS=/usr/sbin/lvs
DMSETUP=/usr/sbin/dmsetup
THIN_LS=/usr/sbin/thin_ls
LOCK=/run/lock/felhom-thin-check.lock
NAME_RE='^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0,63}$'
die() { echo "felhom-thin-check: REFUSED: $1" >&2; exit "${2:-1}"; } # $1 the reason, $2 the exit code
lvm_ids() { # $1 tag → "<tag> <id> <lv>" for every thin volume of this pool
"$LVS" --noheadings --separator , -o lv_name,thin_id --select "pool_lv=$POOL" -- "$VG" 2>/dev/null |
awk -F, -v tag="$1" '{ gsub(/ /, ""); if ($2 ~ /^[0-9]+$/) print tag, $2, $1 }'
}
release() { "$DMSETUP" message "$TPOOL" 0 release_metadata_snap >/dev/null 2>&1; }
main() {
[[ $# -eq 2 ]] || die "usage: felhom-thin-check <vg> <pool>"
VG=$1 POOL=$2
local attr held ids
[[ "$VG" =~ $NAME_RE && "$POOL" =~ $NAME_RE ]] || die "not an LVM name pair"
attr=$("$LVS" --noheadings -o lv_attr -- "$VG/$POOL" 2>/dev/null | tr -d ' ')
[[ "$attr" == t* ]] || die "$VG/$POOL is not a thin pool"
DM="${VG//-/--}-${POOL//-/--}"
TPOOL="$DM-tpool"
TMETA="/dev/mapper/${DM}_tmeta"
exec 9>"$LOCK" || die "cannot open the lock file" 4
flock -w 30 9 || die "another check holds the lock" 3
# Field 7 of the thin-pool status is the held metadata root: "-" when no metadata snapshot exists.
held=$("$DMSETUP" status "$TPOOL" 2>/dev/null | awk '$3 == "thin-pool" { print $7 }')
[[ -n "$held" ]] || die "cannot read the status of $TPOOL" 4
[[ "$held" == "-" ]] || die "a metadata snapshot of $TPOOL is already held (by someone else) — not touched" 3
lvm_ids lvm-before
"$DMSETUP" message "$TPOOL" 0 reserve_metadata_snap >/dev/null 2>&1 || die "could not reserve a metadata snapshot of $TPOOL" 3
trap release EXIT
trap 'release; trap - EXIT; exit 4' INT TERM HUP
if ! ids=$("$THIN_LS" --metadata-snap --no-headers -o DEV "$TMETA" 2>/dev/null); then
die "thin_ls could not read $TMETA" 4
fi
release
trap - EXIT INT TERM HUP
awk '$1 ~ /^[0-9]+$/ { print "pool", $1 }' <<<"$ids"
lvm_ids lvm-after
echo ok
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then main "$@"; fi
+84
View File
@@ -429,6 +429,90 @@ class SelfCheckUndo(unittest.TestCase):
self.assert_restored(box, before, rep) self.assert_restored(box, before, rep)
class RootKeyOnly(unittest.TestCase):
"""R-982, operator ruling D4 (2026-10-22): the bundle gives the host's own sshd a root-key-only drop-in, tests the
whole config and reloads sshd — only when the file was written; a drop-in that says anything but
`prohibit-password` is refused before anything is written.
RED-PROOF: with the after_install block removed, test_fresh_box_takes_the_dropin_and_reloads fails (no reload);
with the content rule removed, test_a_dropin_that_says_yes_or_no_is_refused fails — seen failing 2026-10-22."""
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
def test_fresh_box_takes_the_dropin_and_reloads(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertIn(b"PermitRootLogin prohibit-password", box.files[self.DEST])
self.assertEqual(box.modes[self.DEST], 0o644)
self.assertIn(["/usr/sbin/sshd", "-t"], box.calls)
self.assertIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
self.assertFalse(any("felhom-sshd" in " ".join(c) for c in box.calls), "the operator's door must not be touched")
def test_an_unchanged_dropin_is_not_reloaded(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put(self.DEST, (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes(), 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_dropin_that_says_yes_or_no_is_refused(self):
for bad in (b"PermitRootLogin yes\n", b"PermitRootLogin no\n", b"# nothing\n",
b"PermitRootLogin prohibit-password\nPermitRootLogin yes\n"):
def edit(files, bad=bad):
for e in files:
if e["path"] == self.DEST:
e["content_b64"] = base64.b64encode(bad).decode()
data, sha = edited_bundle(edit)
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertNotEqual(rc, 0, bad)
self.assertNotIn(self.DEST, box.files, bad)
# Review 2026-10-22 night. A2: the effective value is read back with `sshd -T`; a drop-in that is not in force is
# said in the log and the report (not refused — a refusal would freeze the bundle route). A3: a bundle undone
# after the reload reloads sshd again with the old state. RED-PROOF: with the sshd -T check removed,
# test_a_dropin_not_in_force_is_reported fails; with the undo reload removed, test_an_undone_bundle_reloads_sshd
# fails — seen failing 2026-10-22 night.
def _sshd_T(self, box, value):
orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
box.calls.append(argv)
return 0, f"port 22\npermitrootlogin {value}\npasswordauthentication yes\n", ""
return orig(argv, timeout, stdin)
box.host = host
def test_the_effective_value_is_checked(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "without-password")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(any(c[:2] == ["/usr/sbin/sshd", "-T"] for c in box.calls))
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
def test_a_dropin_not_in_force_is_reported(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "yes") # e.g. no Include of sshd_config.d
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
self.assertTrue(any("NOT in force" in l for l in box.logs), box.logs)
def test_an_undone_bundle_reloads_sshd(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.visudo_fail = True # the self-check fails AFTER the install → undo
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assertNotIn(self.DEST, box.files)
reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
class TrustBootstrap(unittest.TestCase): class TrustBootstrap(unittest.TestCase):
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self): def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
data, sha = real_bundle() data, sha = real_bundle()
+40 -2
View File
@@ -14,12 +14,16 @@ GIB = 1 << 30
STUB_LVS = r'''#!/usr/bin/env python3 STUB_LVS = r'''#!/usr/bin/env python3
import json, os, sys import json, os, sys
st = json.load(open(os.environ["IP_STATE"])) st = json.load(open(os.environ["IP_STATE"]))
if st.get("lvs_fail_listing") and "lv_name,pool_lv,lv_size" in sys.argv: sys.exit(5)
args = [a for a in sys.argv[1:] if not a.startswith("-") and a not in ("b",)] args = [a for a in sys.argv[1:] if not a.startswith("-") and a not in ("b",)]
fields = sys.argv[sys.argv.index("-o") + 1].split(",") fields = sys.argv[sys.argv.index("-o") + 1].split(",")
target = args[-1] target = args[-1]
listing = target == st["vg"]
for lv in st["lvs"]: for lv in st["lvs"]:
full = st["vg"] + "/" + lv["name"] full = st["vg"] + "/" + lv["name"]
if target == st["vg"] or target == full: if listing and st.get("hide_pool_in_listing") and lv.get("pool", "") == "":
continue # a listing that silently lost the pool row (the pool's own size stays readable)
if listing or target == full:
row = {"lv_name": lv["name"], "pool_lv": lv.get("pool", ""), "lv_size": str(lv["size"])} row = {"lv_name": lv["name"], "pool_lv": lv.get("pool", ""), "lv_size": str(lv["size"])}
print(" " + " ".join(row[f] for f in fields)) print(" " + " ".join(row[f] for f in fields))
''' '''
@@ -35,15 +39,18 @@ if st.get("dirty_new"): st["fs"]["felhom-belso"] = {"TYPE": "", "RAW": "junk"}
json.dump(st, open(p, "w")) json.dump(st, open(p, "w"))
''' '''
STUB_BLKID = r'''#!/usr/bin/env python3 STUB_BLKID = r'''#!/usr/bin/env python3
# like blkid -p: exit 0 when something is found, 2 when nothing is; st["blkid_rc"] forces an error exit
import json, os, sys import json, os, sys
st = json.load(open(os.environ["IP_STATE"])) st = json.load(open(os.environ["IP_STATE"]))
if st.get("blkid_rc"): sys.exit(st["blkid_rc"])
fs = st["fs"].get(os.path.basename(sys.argv[-1]), {}) fs = st["fs"].get(os.path.basename(sys.argv[-1]), {})
if "-s" in sys.argv: if "-s" in sys.argv:
v = fs.get(sys.argv[sys.argv.index("-s") + 1], "") v = fs.get(sys.argv[sys.argv.index("-s") + 1], "")
if v: print(v) if v: print(v)
else: sys.exit(0 if v else 2)
vals = [fs.get(k, "") for k in ("TYPE", "LABEL", "UUID", "RAW") if fs.get(k)] vals = [fs.get(k, "") for k in ("TYPE", "LABEL", "UUID", "RAW") if fs.get(k)]
if vals: print("\n".join(vals)) if vals: print("\n".join(vals))
sys.exit(0 if vals else 2)
''' '''
STUB_MKFS = r'''#!/usr/bin/env python3 STUB_MKFS = r'''#!/usr/bin/env python3
import json, os, sys import json, os, sys
@@ -213,5 +220,36 @@ class Create(unittest.TestCase):
b.close() b.close()
class FailsClosed(unittest.TestCase):
"""A read that fails must never read as „empty" or „nothing promised". Positive control: Create (same setups)."""
def test_a_failed_volume_listing_creates_nothing(self):
b = Box(349, lvs_fail_listing=True)
try:
rc, out, err = b.run("create", "132")
self.assertEqual(rc, 1, out)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_a_listing_without_the_pool_creates_nothing(self):
b = Box(349, [thin("vm-9201-disk-1", 250)], hide_pool_in_listing=True)
try:
self.assertEqual(b.run("create", "132")[0], 1)
self.assertEqual(b.calls(), [])
finally:
b.close()
def test_a_blkid_error_never_reads_as_empty(self):
for setup in ({"lvs": [thin("felhom-belso", 183)], "fs": {}}, {}):
b = Box(349, **({"lvs": setup["lvs"], "fs": setup["fs"]} if setup else {}), blkid_rc=4)
try:
rc, out, err = b.run("create", "0" if setup else "132")
self.assertEqual(rc, 1, (setup, out))
self.assertNotIn("mkfs", " ".join(b.calls()), setup)
finally:
b.close()
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+13 -8
View File
@@ -1,9 +1,10 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Tests for configs/felhom-mkfs-guarded.sh — the ONLY mkfs path the agent's sudoers permits. """Tests for configs/felhom-mkfs-guarded.sh — the ONLY mkfs path the agent's sudoers permits.
First tests of this wrapper (agent v0.158.0), written with its change for partitions as places (felhom.eu First tests of this wrapper (agent v0.158.0), written for partitions as places (felhom.eu architecture/16 §6). Every
architecture/16 §6): a SIBLING partition that is Felhom's own enrolled drive (mounted raw at /mnt/<name> AND under mount of every node of the disk outside /mnt/felhom-drives refuses — INCLUDING a sibling that is Felhom's own drive
/mnt/felhom-drives) no longer blocks formatting the empty partition next to it; every other mount still refuses. (its raw /mnt/<name> mount): the agent can make that corroborating mount itself, so the root wrapper must not trust
it (security review of 912f7c1). An unlistable disk refuses.
HOW IT RUNS ANYWHERE: the wrapper is copied with five literal host paths replaced (each replacement is asserted, so a HOW IT RUNS ANYWHERE: the wrapper is copied with five literal host paths replaced (each replacement is asserted, so a
drifted wrapper fails here instead of silently testing nothing): /proc/mounts → a fake mount table, the block-device drifted wrapper fails here instead of silently testing nothing): /proc/mounts → a fake mount table, the block-device
@@ -110,8 +111,12 @@ class Partitions(unittest.TestCase):
finally: finally:
d.close() d.close()
def test_the_empty_part_next_to_felhoms_own_drive_is_formatted(self): def test_a_free_part_on_an_unmounted_disk_is_formatted(self):
self.check(OWN_SDB1) self.check([]) # positive control for every refusal below
def test_the_free_part_next_to_felhoms_own_drive_is_refused(self):
# The agent could make the „corroborating" managed mount itself: the root wrapper does not trust it.
self.check(OWN_SDB1, want_ok=False)
def test_a_sibling_mounted_raw_only_is_refused(self): def test_a_sibling_mounted_raw_only_is_refused(self):
self.check([("/dev/sdb1", "/mnt/hdd_1")], want_ok=False) self.check([("/dev/sdb1", "/mnt/hdd_1")], want_ok=False)
@@ -131,9 +136,9 @@ class Partitions(unittest.TestCase):
def test_the_catastrophic_cases_still_refuse(self): def test_the_catastrophic_cases_still_refuse(self):
self.check([("/dev/sdb1", "/")], want_ok=False) # system disk self.check([("/dev/sdb1", "/")], want_ok=False) # system disk
self.check(OWN_SDB1, nodes=disk_sdb(sdb1_fstype="LVM2_member"), want_ok=False) # a member signature self.check([], nodes=disk_sdb(sdb1_fstype="LVM2_member"), want_ok=False) # a member signature
self.check(OWN_SDB1, pvs="/dev/sdb1", want_ok=False) # an LVM PV (pvs) self.check([], pvs="/dev/sdb1", want_ok=False) # an LVM PV (pvs)
self.check(OWN_SDB1, ro="1", want_ok=False) # read-only self.check([], ro="1", want_ok=False) # read-only
if __name__ == "__main__": if __name__ == "__main__":
+62
View File
@@ -376,5 +376,67 @@ class ControllerImage(unittest.TestCase):
self.assertIn("controller-image", buf.getvalue()) self.assertIn("controller-image", buf.getvalue())
class UnitForget(unittest.TestCase):
"""R-1124: the agent forgets a decommissioned drive whose disk is gone — the root half removes its unit file, and
ONLY a felhom by-UUID drive unit that is disabled, not mounted, and whose /dev/disk/by-uuid link is absent.
RED-PROOF: on the pre-R-1124 wrapper `unit-forget` is not a verb (A1 usage, rc 2) — the removal case fails."""
NAME = "mnt-hdd_1.mount"
PATH = "/etc/systemd/system/mnt-hdd_1.mount"
UUID_LINK = "/dev/disk/by-uuid/91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae"
def host(self, text=LOCAL_UNIT, kind="file", links=(), mounted=()):
h = FakeHost()
h.installed = {} if text is None else {self.PATH: (kind, text.encode())}
h.links, h.mounted, h.removed = set(links), set(mounted), []
h.read_installed = lambda p: (None if p not in h.installed else
(_ for _ in ()).throw(pa.Refused("F1", f"{p} is not a regular file"))
if h.installed[p][0] != "file" else h.installed[p][1])
h.lexists = lambda p: p in h.links
h.mountpoints = lambda: set(h.mounted)
h.remove = lambda p: h.removed.append(p)
return h
def test_an_absent_disabled_unmounted_drive_unit_is_removed(self):
h = self.host()
self.assertEqual(run(h, "unit-forget", self.NAME), 0, h.logs)
self.assertEqual(h.removed, [self.PATH])
def test_already_gone_is_fine(self):
h = self.host(text=None)
self.assertEqual(run(h, "unit-forget", self.NAME), 0, h.logs)
self.assertEqual(h.removed, [])
def refused(self, h, rule, *argv):
rc = run(h, *(argv or ("unit-forget", self.NAME)))
self.assertIn(rc, (2, 3), h.logs)
self.assertEqual(h.removed, [], h.logs)
self.assertTrue(any(f"[{rule}]" in l for l in h.logs), h.logs)
def test_F5_a_present_drive_keeps_its_unit(self):
self.refused(self.host(links=[self.UUID_LINK]), "F5")
def test_F4_a_mounted_drive_keeps_its_unit(self):
self.refused(self.host(mounted=["/mnt/hdd_1"]), "F4")
def test_F3_an_enabled_unit_is_not_removed(self):
self.refused(self.host(links=["/etc/systemd/system/multi-user.target.wants/" + self.NAME]), "F3")
def test_F2_a_network_unit_or_a_foreign_unit_is_not_removed(self):
self.refused(self.host(text=LOCAL_UNIT.replace("Managed by felhom-agent", "Written by someone")), "F2")
net = LOCAL_UNIT.replace("# Managed by felhom-agent", "# Managed by felhom-agent felhom network storage")
self.refused(self.host(text=net), "F2")
def test_F1_a_symlinked_unit_is_not_followed(self):
self.refused(self.host(kind="symlink"), "F1")
def test_A1_names(self):
for argv in (("unit-forget", "mnt-hdd_1.automount"), ("unit-forget", "sshd.service"),
("unit-forget", "../mnt-hdd_1.mount"), ("unit-forget",), ("unit-forget", self.NAME, "x")):
self.refused(self.host(), "A1", *argv)
def test_U3_a_unit_whose_name_lies_about_where_is_not_removed(self):
self.refused(self.host(text=LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/mnt/other")), "U3")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main(verbosity=2) unittest.main(verbosity=2)
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env python3
"""Tests for configs/felhom-thin-check (R-1012): it lists the thin pool's device ids next to LVM's and NEVER removes
anything. Fake `lvs`, `dmsetup` and `thin_ls` record every call; nothing real is touched. Each refusal has a positive
control (the healthy read), so a test cannot pass because the helper refuses everything.
Run: python3 configs/test_felhom_thin_check.py (also run by internal/thincheck's Go test)
"""
import os, pathlib, re, subprocess, tempfile, unittest
HELPER = pathlib.Path(__file__).resolve().parent / "felhom-thin-check"
class Box:
def __init__(self, pool_ids=(1, 2), lvm=(("vm-9201-disk-0", 1), ("vm-9201-disk-1", 2)), attr="twi-aotz--",
held="-", reserve_rc=0, thin_ls_rc=0):
self.t = tempfile.TemporaryDirectory()
r = pathlib.Path(self.t.name)
self.log = r / "calls"
self.log.write_text("")
lvm_lines = "\\n".join(f" {n},{i}" for n, i in lvm)
pool_lines = "\\n".join(str(i) for i in pool_ids)
self.bins = {}
for name, body in {
"lvs": f'''case "$*" in *lv_attr*) echo " {attr}";; *thin_id*) printf "{lvm_lines}\\n";; esac''',
"dmsetup": f'''case "$1" in
status) echo "0 112975872 thin-pool 585 6860/262144 517080/882624 {held} rw discard_passdown queue_if_no_space - 1024";;
message) [ "$4" = reserve_metadata_snap ] && exit {reserve_rc}; exit 0;;
esac''',
"thin_ls": f'''[ {thin_ls_rc} -ne 0 ] && exit {thin_ls_rc}; printf "{pool_lines}\\n"''',
}.items():
p = r / name
p.write_text(f'#!/bin/bash\necho "{name} $*" >> "{self.log}"\n{body}\n')
p.chmod(0o755)
self.bins[name] = p
self.lock = r / "lock"
def run(self, *args):
script = (f'source "{HELPER}"; LVS="{self.bins["lvs"]}"; DMSETUP="{self.bins["dmsetup"]}"; '
f'THIN_LS="{self.bins["thin_ls"]}"; LOCK="{self.lock}"; main "$@"')
r = subprocess.run(["bash", "-c", script, "x", *args], capture_output=True, text=True)
return r.returncode, r.stdout, r.stderr
def calls(self):
return self.log.read_text().splitlines()
def messages(calls):
return [c.split()[-1] for c in calls if c.startswith("dmsetup message")]
class Reads(unittest.TestCase):
def test_healthy_pool_lists_both_sides_and_releases(self):
b = Box()
rc, out, err = b.run("pve", "data")
self.assertEqual(rc, 0, err)
self.assertEqual(out.splitlines(), ["lvm-before 1 vm-9201-disk-0", "lvm-before 2 vm-9201-disk-1",
"pool 1", "pool 2",
"lvm-after 1 vm-9201-disk-0", "lvm-after 2 vm-9201-disk-1", "ok"])
self.assertEqual(messages(b.calls()), ["reserve_metadata_snap", "release_metadata_snap"])
self.assertIn("thin_ls --metadata-snap --no-headers -o DEV /dev/mapper/pve-data_tmeta", b.calls())
def test_the_2026_10_14_orphan_is_listed(self):
rc, out, _ = Box(pool_ids=(1, 2, 4)).run("pve", "data")
self.assertEqual(rc, 0)
self.assertIn("pool 4", out.splitlines())
def test_dashed_names_map_to_the_device_mapper_name(self):
b = Box()
rc, _, err = b.run("felhom-scratch", "thin-pool")
self.assertEqual(rc, 0, err)
self.assertIn("dmsetup status felhom--scratch-thin--pool-tpool", b.calls())
class Refuses(unittest.TestCase):
def refused(self, box, rc_want, *args):
rc, out, err = box.run(*args)
self.assertEqual(rc, rc_want, (out, err))
self.assertNotIn("ok", out.splitlines())
return box.calls()
def test_arguments(self):
for args in (("pve",), ("pve", "data", "x"), ("pve;id", "data"), ("-pve", "data"), ("pve", "../data"),
("pve", "da ta")):
calls = self.refused(Box(), 1, *args)
self.assertEqual(calls, [], args)
def test_not_a_thin_pool(self):
calls = self.refused(Box(attr="Vwi-aotz--"), 1, "pve", "vm-9201-disk-1")
self.assertEqual(messages(calls), [])
def test_a_snapshot_someone_else_holds_is_never_touched(self):
b = Box(held="1234")
rc, _, err = b.run("pve", "data")
self.assertEqual(rc, 3)
self.assertEqual(messages(b.calls()), []) # neither reserve nor release
# the reason line ends with the reason, not the exit code (found live on demo-hp 2026-10-17)
self.assertTrue(err.strip().endswith("not touched"), err)
def test_reserve_failure_does_not_release(self):
calls = self.refused(Box(reserve_rc=1), 3, "pve", "data")
self.assertEqual(messages(calls), ["reserve_metadata_snap"])
def test_thin_ls_failure_still_releases(self):
calls = self.refused(Box(thin_ls_rc=1), 4, "pve", "data")
self.assertEqual(messages(calls), ["reserve_metadata_snap", "release_metadata_snap"])
class NeverDeletes(unittest.TestCase):
def test_the_only_pool_messages_are_reserve_and_release(self):
src = HELPER.read_text()
sent = re.findall(r'"\$DMSETUP" message "\$TPOOL" 0 (\S+)', src)
self.assertEqual(sorted(set(sent)), ["release_metadata_snap", "reserve_metadata_snap"])
for word in ("delete", "lvremove", "thin_repair", "create_thin", "create_snap", " rm "):
code = "\n".join(l for l in src.splitlines() if not l.lstrip().startswith("#"))
self.assertNotIn(word, code, word)
if __name__ == "__main__":
unittest.main(verbosity=2)
+5 -4
View File
@@ -1,7 +1,8 @@
module gitea.dooplex.hu/admin/felhom-agent module gitea.dooplex.hu/admin/felhom-agent
go 1.25.0 go 1.27.2
require golang.org/x/crypto v0.52.0 require (
golang.org/x/crypto v0.58.0
require golang.org/x/sys v0.45.0 // indirect golang.org/x/sys v0.49.0
)
+6 -6
View File
@@ -1,6 +1,6 @@
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988= golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc= golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4= golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk= golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
+95 -5
View File
@@ -39,13 +39,25 @@ type savedSuccess struct {
target string target string
vmid int vmid int
at time.Time at time.Time
// recent (R-973) holds the start times of the newest successes (at most maxRecentSuccesses, newest
// last) — what the local-API backup cadence cap counts, so an agent restart does not reset the cap.
recent []time.Time
// refusedAt (R-973) is the newest time the cap refused a backup of this guest on this tier.
refusedAt time.Time
} }
// backupSuccessJSON is one entry on disk. // maxRecentSuccesses bounds the per-tier success history kept for the cadence cap (R-973). The cap
// allows at most 3 runs in a window, so 8 is ample.
const maxRecentSuccesses = 8
// backupSuccessJSON is one entry on disk. Recent and LastRefusedAt are additive (R-973): a file written
// before them reads as "no history", and an older agent ignores them.
type backupSuccessJSON struct { type backupSuccessJSON struct {
Target string `json:"target"` Target string `json:"target"`
VMID int `json:"vmid"` VMID int `json:"vmid"`
StartedAt string `json:"started_at"` StartedAt string `json:"started_at"`
Recent []string `json:"recent,omitempty"`
LastRefusedAt string `json:"last_refused_at,omitempty"`
} }
func backupStateKey(target string, vmid int) string { return target + "/" + strconv.Itoa(vmid) } func backupStateKey(target string, vmid int) string { return target + "/" + strconv.Itoa(vmid) }
@@ -67,7 +79,19 @@ func NewBackupSuccessState(path string) *BackupSuccessState {
if perr != nil { if perr != nil {
continue // one unreadable entry must not lose the others continue // one unreadable entry must not lose the others
} }
s.last[backupStateKey(e.Target, e.VMID)] = savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()} ss := savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()}
for _, r := range e.Recent {
if rt, rerr := time.Parse(time.RFC3339, r); rerr == nil {
ss.recent = append(ss.recent, rt.UTC())
}
}
if len(ss.recent) == 0 {
ss.recent = []time.Time{ss.at} // a pre-R-973 file: its one known success counts
}
if rt, rerr := time.Parse(time.RFC3339, e.LastRefusedAt); rerr == nil {
ss.refusedAt = rt.UTC()
}
s.last[backupStateKey(e.Target, e.VMID)] = ss
} }
return s return s
} }
@@ -85,13 +109,72 @@ func (s *BackupSuccessState) RecordBackupSuccess(target string, b hub.Backup) er
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
k := backupStateKey(target, b.VMID) k := backupStateKey(target, b.VMID)
if old, ok := s.last[k]; ok && !t.After(old.at) { old, had := s.last[k]
if had && !t.After(old.at) {
return nil return nil
} }
s.last[k] = savedSuccess{target: target, vmid: b.VMID, at: t.UTC()} ss := savedSuccess{target: target, vmid: b.VMID, at: t.UTC(), refusedAt: old.refusedAt}
ss.recent = append(append([]time.Time(nil), old.recent...), t.UTC())
if len(ss.recent) > maxRecentSuccesses {
ss.recent = ss.recent[len(ss.recent)-maxRecentSuccesses:]
}
s.last[k] = ss
return s.saveLocked() return s.saveLocked()
} }
// RecentBackupSuccesses returns the saved success start times for this tier and guest that are not
// older than since (R-973 — the cadence cap's restart-proof count).
func (s *BackupSuccessState) RecentBackupSuccesses(target string, vmid int, since time.Time) []time.Time {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
var out []time.Time
for _, t := range s.last[backupStateKey(target, vmid)].recent {
if !t.Before(since) {
out = append(out, t)
}
}
return out
}
// RecordBackupRefusal saves that the cadence cap refused a backup of this guest on this tier at at
// (R-973), so the operator signal it raises survives an agent restart.
func (s *BackupSuccessState) RecordBackupRefusal(target string, vmid int, at time.Time) error {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
k := backupStateKey(target, vmid)
ss, ok := s.last[k]
if !ok {
return nil // nothing to cap against without a success; the cap never refuses then
}
if !at.After(ss.refusedAt) {
return nil
}
ss.refusedAt = at.UTC()
s.last[k] = ss
return s.saveLocked()
}
// LastBackupRefusal returns the newest saved cadence-cap refusal across every tier and guest.
func (s *BackupSuccessState) LastBackupRefusal() (target string, vmid int, at time.Time, ok bool) {
if s == nil {
return "", 0, time.Time{}, false
}
s.mu.Lock()
defer s.mu.Unlock()
for _, e := range s.last {
if !e.refusedAt.IsZero() && e.refusedAt.After(at) {
target, vmid, at, ok = e.target, e.vmid, e.refusedAt, true
}
}
return target, vmid, at, ok
}
// LastKnownSuccess returns the newest saved success for this tier and guest (ok=false = none on file). // LastKnownSuccess returns the newest saved success for this tier and guest (ok=false = none on file).
func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Time, bool) { func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Time, bool) {
if s == nil { if s == nil {
@@ -129,7 +212,14 @@ func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
func (s *BackupSuccessState) saveLocked() error { func (s *BackupSuccessState) saveLocked() error {
entries := make([]backupSuccessJSON, 0, len(s.last)) entries := make([]backupSuccessJSON, 0, len(s.last))
for _, e := range s.last { for _, e := range s.last {
entries = append(entries, backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)}) j := backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)}
for _, r := range e.recent {
j.Recent = append(j.Recent, r.Format(time.RFC3339))
}
if !e.refusedAt.IsZero() {
j.LastRefusedAt = e.refusedAt.Format(time.RFC3339)
}
entries = append(entries, j)
} }
// Deterministic file content (Go's map order is random). // Deterministic file content (Go's map order is random).
sort.Slice(entries, func(i, j int) bool { sort.Slice(entries, func(i, j int) bool {
+37
View File
@@ -74,3 +74,40 @@ func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
t.Fatal("uncovered_volumes must marshal as [], not null") t.Fatal("uncovered_volumes must marshal as [], not null")
} }
} }
// R-973: the cadence cap's history and refusals survive a reopen; the history is bounded; a file
// written before R-973 counts its one known success (never "no history" for a tier that has a copy).
func TestBackupSuccessState_R973_RecentHistoryAndRefusalSurviveAReopen(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
base := time.Date(2026, 10, 18, 2, 30, 0, 0, time.UTC)
for i := 0; i < 10; i++ {
at := base.Add(time.Duration(i) * time.Hour)
if err := s.RecordBackupSuccess("local", hub.Backup{VMID: 9201, Success: true, StartedAt: at.Format(time.RFC3339)}); err != nil {
t.Fatal(err)
}
}
refused := base.Add(11 * time.Hour)
if err := s.RecordBackupRefusal("local", 9201, refused); err != nil {
t.Fatal(err)
}
r := NewBackupSuccessState(path)
all := r.RecentBackupSuccesses("local", 9201, time.Time{})
if len(all) != maxRecentSuccesses || !all[len(all)-1].Equal(base.Add(9*time.Hour)) {
t.Fatalf("recent history after a reopen: %v (want the newest %d, ending at +9h)", all, maxRecentSuccesses)
}
if got := r.RecentBackupSuccesses("local", 9201, base.Add(7*time.Hour)); len(got) != 3 {
t.Fatalf("successes since +7h: %d, want 3", len(got))
}
if tg, vm, at, ok := r.LastBackupRefusal(); !ok || tg != "local" || vm != 9201 || !at.Equal(refused) {
t.Fatalf("refusal after a reopen: %q %d %v %v", tg, vm, at, ok)
}
old := filepath.Join(t.TempDir(), "old.json")
if err := os.WriteFile(old, []byte(`[{"target":"felhom-pbs","vmid":9201,"started_at":"2026-10-17T03:00:00Z"}]`), 0o600); err != nil {
t.Fatal(err)
}
if got := NewBackupSuccessState(old).RecentBackupSuccesses("felhom-pbs", 9201, time.Time{}); len(got) != 1 {
t.Fatalf("a pre-R-973 file must count its one success: %v", got)
}
}
+1 -1
View File
@@ -2,9 +2,9 @@ package backup
import ( import (
"context" "context"
"fmt"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"io" "io"
"log/slog" "log/slog"
"testing" "testing"
+2 -2
View File
@@ -93,7 +93,7 @@ func TestPrune_KeepLastClamp(t *testing.T) {
t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set) t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set)
} }
} }
if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-last=3" { if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-daily=3" {
t.Fatalf("default PruneBackupsSpec = %q, want keep-last=3", spec) t.Fatalf("default PruneBackupsSpec = %q, want keep-daily=3 (R-1149)", spec)
} }
} }
+67
View File
@@ -0,0 +1,67 @@
package backup
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1012: a snapshot backup that fails carries the orphan finding in its error — the text the controller mails to
// the operator (`whole_guest_backup_failed`). The check runs ONCE, after the failure — never before the run.
// RED-PROOF (P1/redproof-thincheck.txt): with explainFailure's append removed this test fails. RED-PROOF (2026-10-17
// night, felhom-agent REPORT.md): with the v0.159.0 pre-run check restored, calls = 2 and this test fails.
func TestBackup_SnapshotFailure_NamesTheOrphan(t *testing.T) {
const why = "probable cause: thin pool pve/data holds device 4 that LVM does not know"
for name, api := range map[string]*fakeBackupAPI{
"task fails": {vzdumpUPID: "UPID:x", waitErr: errors.New("task failed: job errors")},
"start fails": {vzdumpErr: errors.New("vzdump: 500")},
} {
t.Run(name, func(t *testing.T) {
r := NewBackupRunner(api, "local", "", "", "", quiet())
calls := 0
r.SetThinCheck(func(context.Context, int) string { calls++; return why })
rec, err := r.Backup(context.Background(), 9201)
if err == nil || rec.Success {
t.Fatalf("want a failure, got %+v", rec)
}
if !strings.Contains(rec.Error, why) {
t.Fatalf("the failure does not name the orphan: %q", rec.Error)
}
if calls != 1 {
t.Fatalf("thin check ran %d times, want 1 (after the failure only)", calls)
}
})
}
}
// A successful backup never runs the check (it takes a metadata snapshot of the live pool, and nobody needs the answer)
// and keeps a clean record; a stop-mode backup takes no snapshot, so the check never runs; an unwired runner behaves
// exactly as before. RED-PROOF (2026-10-17 night): with the v0.159.0 pre-run check restored, the success leg fails
// with calls=1.
func TestBackup_ThinCheck_SilentWhenNotApplicable(t *testing.T) {
ok := &fakeBackupAPI{vzdumpUPID: "UPID:x",
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-x", Content: "backup", VMID: 9201, Size: 100, CTime: 1}}}
r := NewBackupRunner(ok, "local", "", "", "", quiet())
calls := 0
r.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
rec, err := r.Backup(context.Background(), 9201)
if err != nil || !rec.Success || rec.Error != "" || calls != 0 {
t.Fatalf("success: err=%v rec=%+v calls=%d (want no check on a successful backup, clean record)", err, rec, calls)
}
stop := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", proxmox.ModeStop, "", "", quiet())
calls = 0
stop.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
rec, _ = stop.Backup(context.Background(), 9201)
if calls != 0 || strings.Contains(rec.Error, "probable cause") {
t.Fatalf("stop mode: calls=%d error=%q", calls, rec.Error)
}
plain := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", "", "", "", quiet())
if rec, _ := plain.Backup(context.Background(), 9201); rec.Error != "boom" {
t.Fatalf("unwired: error = %q, want the task error unchanged", rec.Error)
}
}
+25
View File
@@ -61,6 +61,11 @@ type BackupRunner struct {
// beforeBackup runs just before the space preflight (R-1006): it removes a dead half-written vzdump of this guest on // beforeBackup runs just before the space preflight (R-1006): it removes a dead half-written vzdump of this guest on
// this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing. // this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing.
beforeBackup func(ctx context.Context, storage string, vmid int) beforeBackup func(ctx context.Context, storage string, vmid int)
// thinCheck (R-1012) names an orphan thin device in the guest's pools in plain words ("" = none / not checked). It
// runs ONLY when a snapshot-mode vzdump has failed (start or task), so the failure the controller mails to the
// operator says why; a successful backup never runs it (it takes a metadata snapshot of the live pool). It never
// removes anything. nil = not wired.
thinCheck func(ctx context.Context, vmid int) string
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete // rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
@@ -110,6 +115,22 @@ func (r *BackupRunner) SetBeforeBackup(fn func(ctx context.Context, storage stri
r.beforeBackup = fn r.beforeBackup = fn
} }
// SetThinCheck wires the R-1012 orphan check (see BackupRunner.thinCheck).
func (r *BackupRunner) SetThinCheck(fn func(ctx context.Context, vmid int) string) { r.thinCheck = fn }
// explainFailure appends the orphan finding to a failed snapshot-mode backup's error — the only place the check runs, so
// it sees the pool as the failure left it. Bounded, so a stuck helper cannot hold the job.
func (r *BackupRunner) explainFailure(ctx context.Context, vmid int, rec *hub.Backup) {
if r.thinCheck == nil || r.mode != proxmox.ModeSnapshot {
return
}
cctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute)
defer cancel()
if why := r.thinCheck(cctx, vmid); why != "" {
rec.Error = rec.Error + " — " + why
}
}
// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It // localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
// applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a // applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a
// separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type // separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type
@@ -201,12 +222,15 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
return rec, fmt.Errorf("backup: %s", rec.Error) return rec, fmt.Errorf("backup: %s", rec.Error)
} }
// R-1012: no orphan check here. It takes a metadata snapshot of the live pool, and its answer is only needed when the
// snapshot step has failed — explainFailure runs it then (pinned by TestBackup_ThinCheck_SilentWhenNotApplicable).
upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{ upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes, VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune) PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
}) })
if err != nil { if err != nil {
rec.Error = err.Error() rec.Error = err.Error()
r.explainFailure(ctx, vmid, &rec)
rec.DurationSeconds = time.Since(start).Seconds() rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err) return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err)
} }
@@ -220,6 +244,7 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
} }
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil { if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil {
rec.Error = err.Error() rec.Error = err.Error()
r.explainFailure(ctx, vmid, &rec)
rec.DurationSeconds = time.Since(start).Seconds() rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err) return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
} }
+2
View File
@@ -91,9 +91,11 @@ var manifest = []Capability{
{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""}, {"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""}, {"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""}, {"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
{"thin-orphan-check", "orphan thin device check after a failed whole-guest backup (read-only, R-1012)", "/usr/local/sbin/felhom-thin-check", []string{"pve", "data"}, false, ""},
// ---- Storage mount units (watchdog re-mount) ---- // ---- Storage mount units (watchdog re-mount) ----
{"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""}, {"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-forget", "decommissioned-drive unit removal (root check, R-1124)", "/usr/local/sbin/felhom-priv-apply", []string{"unit-forget", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""}, {"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "mnt-felhom\\x2dx.mount"}, false, ""}, {"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "mnt-felhom\\x2dx.mount"}, false, ""}, {"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
+32
View File
@@ -180,3 +180,35 @@ func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers) t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
} }
} }
// R-1149 (operator decision D17, 2026-10-19): the local whole-guest tier keeps one restore point per day for N days
// (`--prune-backups keep-daily=N`), not the newest N; an additional tier keeps its explicit count.
//
// COMPANION RED-PROOF (observed 2026-10-19): make PruneSpec return keep-last for the primary → this fails with
// `primary tier prune = "keep-last=3", want "keep-daily=3"`. Restored.
func TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
LocalBackupRetention: 3,
ExtraTargets: []BackupTargetConfig{
{TargetID: "felhom-pbs", CadenceSeconds: 604800, KeepLast: 2},
{TargetID: "nas", CadenceSeconds: 86400},
},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 3 {
t.Fatalf("want 3 tiers, got %d", len(tiers))
}
for _, c := range []struct {
i int
want string
}{{0, "keep-daily=3"}, {1, "keep-last=2"}, {2, ""}} {
if got := tiers[c.i].PruneSpec(); got != c.want {
t.Fatalf("%s tier prune = %q, want %q", map[bool]string{true: "primary", false: tiers[c.i].TargetID}[tiers[c.i].Primary], got, c.want)
}
}
// The unset retention defaults to 3 days, never to „prune everything".
if got := (BackupConfig{}).PruneBackupsSpec(); got != "keep-daily=3" {
t.Fatalf("default local prune = %q, want keep-daily=3", got)
}
}
+21 -3
View File
@@ -391,7 +391,8 @@ type BackupConfig struct {
// hub-served per-guest policy is slice 10; this is the agent-local cadence. // hub-served per-guest policy is slice 10; this is the agent-local cadence.
BackupCadenceSeconds int `json:"backup_cadence_seconds"` BackupCadenceSeconds int `json:"backup_cadence_seconds"`
// LocalBackupRetention is keep-last=N for the per-run `--prune-backups` on a LOCAL vzdump target — // LocalBackupRetention is N for the per-run `--prune-backups keep-daily=N` on a LOCAL vzdump target (keep-last=N
// before R-1149) —
// so the agent's own local whole-guest backups can't pile up and refill root (the felhom-pve incident; // so the agent's own local whole-guest backups can't pile up and refill root (the felhom-pve incident;
// the host_disk + storage_fill checkers are the detectors, this is the preventive default). 0/unset → // the host_disk + storage_fill checkers are the detectors, this is the preventive default). 0/unset →
// default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup. // default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup.
@@ -538,9 +539,26 @@ func (b BackupConfig) KeepLast() int {
return b.LocalBackupRetention return b.LocalBackupRetention
} }
// PruneBackupsSpec returns the PVE `--prune-backups` value for the local vzdump (e.g. "keep-last=3"). // PruneBackupsSpec returns the PVE `--prune-backups` value for the local vzdump: "keep-daily=N" (R-1149, operator
// decision D17 2026-10-19 — one restore point per day for N days; `07` §6.1). keep-last=N had the same disk cost but
// let N extra runs on one day push out every older point; keep-daily keeps the newest copy of each of the N newest
// days that have one, so a flood inside one day replaces only that day's point. Pinned by
// TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay and, as the consequence, TestBackupCap_R1149_LocalFloodKeepsOlderDays.
func (b BackupConfig) PruneBackupsSpec() string { func (b BackupConfig) PruneBackupsSpec() string {
return fmt.Sprintf("keep-last=%d", b.KeepLast()) return fmt.Sprintf("keep-daily=%d", b.KeepLast())
}
// PruneSpec is this tier's per-run `--prune-backups` value, or "" for never prune. The PRIMARY (local) tier keeps one
// point per day (keep-daily, R-1149, from KeepLast() — always ≥1); an additional tier keeps its explicit keep_last
// count (a weekly off-site tier's „last 2" is two weeks, and PBS prunes stay opt-in per tier, R-82).
func (t BackupTier) PruneSpec() string {
if t.KeepLast <= 0 {
return ""
}
if t.Primary {
return fmt.Sprintf("keep-daily=%d", t.KeepLast)
}
return fmt.Sprintf("keep-last=%d", t.KeepLast)
} }
// BackupCadence returns the per-guest /backup/due window: positive as-is, else 24h default. // BackupCadence returns the per-guest /backup/due window: positive as-is, else 24h default.
+12 -2
View File
@@ -1,5 +1,7 @@
package escrow package escrow
import "encoding/json"
// Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by // Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by
// felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API // felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API
// re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers // re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers
@@ -43,7 +45,15 @@ type CeremonyOutput struct {
IdentityBlobBytes int `json:"identity_blob_bytes"` IdentityBlobBytes int `json:"identity_blob_bytes"`
ResticPwSealed bool `json:"restic_pw_sealed"` ResticPwSealed bool `json:"restic_pw_sealed"`
Uploaded bool `json:"uploaded"` Uploaded bool `json:"uploaded"`
// PendingUpload (version 2, R-978) is the hub upload body — the R-wrapped blobs (ciphertext), the key
// fingerprint and the sealed password's hash — that json mode hands back INSTEAD of uploading. The agent
// daemon uploads it only when the customer claims the new recovery code, so a ceremony that is started
// and abandoned never replaces the package the customer's previous code opens. Not a secret without R,
// but never logged. Uploaded is false whenever this is set.
PendingUpload json.RawMessage `json:"pending_upload,omitempty"`
} }
// CeremonyOutputVersion is the current CeremonyOutput.Version value. // CeremonyOutputVersion is the current CeremonyOutput.Version value. 2 (R-978): json mode no longer
const CeremonyOutputVersion = 1 // uploads; it returns PendingUpload. The daemon refuses any other version (a binary/daemon mismatch
// fails the ceremony instead of losing a package).
const CeremonyOutputVersion = 2
+22 -1
View File
@@ -22,6 +22,11 @@ type Loop struct {
mu sync.Mutex mu sync.Mutex
fetched bool // a desired-state document has been delivered (false = never a teardown signal) fetched bool // a desired-state document has been delivered (false = never a teardown signal)
block *hub.WireWireguard block *hub.WireWireguard
// passed (R-1194) is set once the first reconcile pass has run. Until then the instance has never been rendered,
// so its health says nothing — and the first host report, sent before that pass, used to read „config invalid,
// port 0" and raise a false oob_degraded on EVERY fresh box (rehearsal 2026-10-20; reproduced 2026-10-21: the
// report at 20:34:00 read an absent config, the apply landed 0.4 s later).
passed bool
nudge chan struct{} nudge chan struct{}
} }
@@ -85,6 +90,7 @@ func (l *Loop) reconcile(ctx context.Context) {
if !fetched { if !fetched {
applyBlock = nil applyBlock = nil
} }
defer l.markPassed()
port, err := l.mgr.Apply(ctx, applyBlock) port, err := l.mgr.Apply(ctx, applyBlock)
if err != nil { if err != nil {
return // Apply logged; a claim/exhaustion or install error — retry next tick return // Apply logged; a claim/exhaustion or install error — retry next tick
@@ -106,8 +112,23 @@ func oobPeerIP(block *hub.WireWireguard) string {
return block.OOBPeerIP return block.OOBPeerIP
} }
// OOBStatus implements the hub collector's reporter seam (Part 4). // markPassed records that a reconcile pass has run (R-1194) — after a failed Apply too, so a real, lasting failure
// is still reported from the next report on.
func (l *Loop) markPassed() {
l.mu.Lock()
l.passed = true
l.mu.Unlock()
}
// OOBStatus implements the hub collector's reporter seam (Part 4). nil — no stanza, which the hub does not evaluate —
// until the first reconcile pass has run (R-1194).
func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus { func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus {
l.mu.Lock()
passed := l.passed
l.mu.Unlock()
if !passed {
return nil
}
_, block := l.snapshot() _, block := l.snapshot()
return l.mgr.Status(ctx, block) return l.mgr.Status(ctx, block)
} }
@@ -0,0 +1,29 @@
package felhomsshd
import (
"context"
"testing"
"time"
)
// R-1194: on every fresh box the first host report — sent before the OOB loop's first pass — read „config invalid,
// port 0" (the config was not rendered yet) and the hub mailed oob_degraded. Reproduced 2026-10-21 on a fresh box
// (felhom.eu audits/night-2026-10-21/P5-checks/): the report at 20:34:00, the apply 0.4 s later, the daemon healthy.
// The loop now reports NO stanza (the hub does not evaluate a missing one) until a pass has run — a failed pass
// counts, so a real lasting failure is still reported from the next report on.
// RED-PROOF: with OOBStatus not gated on `passed`, the first assertion fails (a stanza with config_invalid).
func TestR1194_NoOOBStanzaBeforeTheFirstPass(t *testing.T) {
r := newScriptRunner()
m := NewManager(r, t.TempDir(), nil)
m.isActive = func(context.Context) bool { return false }
m.isFailed = func(context.Context) bool { return false }
m.now = func() time.Time { return time.Unix(1783270000, 0) }
l := NewLoop(m, nil, time.Minute, nil)
if st := l.OOBStatus(context.Background()); st != nil {
t.Fatalf("a stanza before the first pass reads a config that was never rendered: %+v", st)
}
l.reconcile(context.Background()) // fails or succeeds — either way a pass has run
if st := l.OOBStatus(context.Background()); st == nil {
t.Fatal("after the first pass the health must be reported (a real failure must reach the hub)")
}
}
+10 -3
View File
@@ -14,6 +14,13 @@ const (
TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED
TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which) TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which)
TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting) TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting)
// TunnelAbsent (R-1070, agent Unreleased 2026-10-19): the guest answered and has NO cloudflared container — the
// controller makes none when no tunnel token is configured (a LAN-only or test box). Not a failure by itself: the
// hub never alarms on it, and the OS-update host rule reads it as „not applicable" only when the container was
// also absent before the update (HostHealthVerdict). Docker 29 prints „error: no such object" in lower case, so
// before this state existed the case-sensitive match fell through to `unknown` — which failed the host rule on
// every box without a tunnel (Tester4, 2026-10-15).
TunnelAbsent = "absent"
) )
// CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec. // CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec.
@@ -53,7 +60,7 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
return TunnelUnknown, "no running customer guest" return TunnelUnknown, "no running customer guest"
} }
worst, wdetail := "", "" worst, wdetail := "", ""
rank := map[string]int{TunnelRunning: 0, TunnelUnknown: 1, TunnelNotRunning: 2} rank := map[string]int{TunnelRunning: 0, TunnelAbsent: 1, TunnelUnknown: 2, TunnelNotRunning: 3}
for _, v := range vmids { for _, v := range vmids {
out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared") out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared")
st, d := ClassifyTunnel(string(out), string(errOut), err) st, d := ClassifyTunnel(string(out), string(errOut), err)
@@ -71,8 +78,8 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
func ClassifyTunnel(stdout, stderr string, err error) (string, string) { func ClassifyTunnel(stdout, stderr string, err error) (string, string) {
out := strings.TrimSpace(stdout) out := strings.TrimSpace(stdout)
if err != nil || out == "" { if err != nil || out == "" {
if strings.Contains(stderr, "No such object") || strings.Contains(stderr, "No such container") { if e := strings.ToLower(stderr); strings.Contains(e, "no such object") || strings.Contains(e, "no such container") {
return TunnelNotRunning, "no cloudflared container in the guest" return TunnelAbsent, "no cloudflared container in the guest (no tunnel set up on this box)"
} }
return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err) return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err)
} }
+4 -1
View File
@@ -20,7 +20,10 @@ func TestClassifyTunnel(t *testing.T) {
{"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"}, {"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"},
{"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"}, {"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"},
{"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"}, {"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelNotRunning, "no cloudflared container"}, {"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
// R-1070: Docker 29's wording, measured on demo-hp 2026-10-19 (`docker inspect` of a missing name).
{"absent, docker 29", "", "error: no such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no tunnel set up"},
{"absent, no such container", "", "Error response from daemon: No such container: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
{"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"}, {"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"},
{"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"}, {"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"},
{"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"}, {"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"},
+7 -1
View File
@@ -11,6 +11,7 @@ import (
"os" "os"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability" "gitea.dooplex.hu/admin/felhom-agent/internal/capability"
@@ -136,6 +137,8 @@ type Collector struct {
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256 selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
// swapShortWarned (R-1192): the host-swap-short WARN is logged once per state change, not every report.
swapShortWarned atomic.Bool
} }
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is // NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
@@ -375,6 +378,8 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
} }
host := hostMetrics(c.px.Node(), ns) host := hostMetrics(c.px.Node(), ns)
guests := c.collectGuests(ctx)
c.applySwap(&host, ns, guests) // R-1192: host swap vs the guests' swap allowance
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable
report := &HostReport{ report := &HostReport{
@@ -383,7 +388,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
AgentVersion: c.agentVersion, AgentVersion: c.agentVersion,
AgentSHA256: c.agentSHA256(), AgentSHA256: c.agentSHA256(),
Host: host, Host: host,
Guests: c.collectGuests(ctx), Guests: guests,
// storage_targets populated this slice (slice 5) via the observer; the rest stay // storage_targets populated this slice (slice 5) via the observer; the rest stay
// defined-but-empty (slice 6). Non-nil so they marshal as []. // defined-but-empty (slice 6). Non-nil so they marshal as [].
StorageTargets: c.collectStorage(ctx), StorageTargets: c.collectStorage(ctx),
@@ -567,6 +572,7 @@ func (c *Collector) collectGuests(ctx context.Context) []Guest {
Cores: cfg.Cores, Cores: cfg.Cores,
MemoryBytes: g.MaxMem, MemoryBytes: g.MaxMem,
DiskBytes: g.MaxDisk, DiskBytes: g.MaxDisk,
SwapBytes: cfg.Swap * (1 << 20),
} }
} }
guests = append(guests, entry) guests = append(guests, entry)
+39
View File
@@ -0,0 +1,39 @@
package hub
import (
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the guest swap fix (every guest may swap as much as its RAM,
// localapi/guestswap.go) moves a memory burst onto the HOST's swap. Measured on demo-hp the same day: our installer
// gives an 8 GB / 256 GB machine 7.75 GB of host swap, so a 4096 MB guest's 4096 MB swap fits. A host with no swap
// (a BYO install, or swap turned off) gets nothing from the fix. So the report carries both numbers, the agent says it
// once in its log, and the hub raises host_swap_short. The agent never creates host swap itself: changing a host its
// owner set up is a decision (felhom.eu register).
// HostSwapShort reports whether the host's swap is below the guests' swap allowance. A host whose guests have no
// swap allowance (0) is never short.
func HostSwapShort(hostMB, guestMB int64) bool { return guestMB > 0 && hostMB < guestMB }
// applySwap fills the host block's swap numbers from the node status and the guests' specs, and logs the short
// state once per change.
func (c *Collector) applySwap(h *HostMetrics, ns proxmox.NodeStatus, guests []Guest) {
hostMB := ns.Swap.Total / (1 << 20)
var guestMB int64
for _, g := range guests {
if g.Spec != nil {
guestMB += g.Spec.SwapBytes / (1 << 20)
}
}
h.HostSwapMB, h.GuestSwapMB = &hostMB, &guestMB
if HostSwapShort(hostMB, guestMB) {
if c.swapShortWarned.CompareAndSwap(false, true) {
c.logger.Warn("host-swap: the host's swap is below its guests' swap allowance — a memory burst can freeze the box (R-1192); the hub is told (host_swap_short)",
"host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
return
}
if c.swapShortWarned.CompareAndSwap(true, false) {
c.logger.Info("host-swap: the host's swap covers its guests' swap allowance again", "host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
}
+42
View File
@@ -0,0 +1,42 @@
package hub
import (
"bytes"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the host report carries the host's swap and the guests' swap allowance, and the
// agent says ONCE in its log when the host's swap is below it (and once when it is covered again).
// RED-PROOF: with applySwap not called the numbers are nil; with the CompareAndSwap removed the WARN repeats.
func TestR1192_HostReportCarriesSwapAndWarnsOnce(t *testing.T) {
var buf bytes.Buffer
c := &Collector{logger: slog.New(slog.NewTextHandler(&buf, nil))}
guests := []Guest{{VMID: 9201, Spec: &GuestSpec{SwapBytes: 4096 << 20}}, {VMID: 9202, Spec: nil}}
var ns proxmox.NodeStatus // a host with no swap
for i := 0; i < 3; i++ {
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if h.HostSwapMB == nil || h.GuestSwapMB == nil || *h.HostSwapMB != 0 || *h.GuestSwapMB != 4096 {
t.Fatalf("report numbers: host %v guest %v", h.HostSwapMB, h.GuestSwapMB)
}
}
if n := strings.Count(buf.String(), "host-swap: the host's swap is below"); n != 1 {
t.Fatalf("the short WARN must be logged once, got %d:\n%s", n, buf.String())
}
ns.Swap.Total = 7935 << 20 // the installer's 8 GB host
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if *h.HostSwapMB != 7935 || HostSwapShort(*h.HostSwapMB, *h.GuestSwapMB) {
t.Fatalf("a covered host must not be short: %v / %v", *h.HostSwapMB, *h.GuestSwapMB)
}
if !strings.Contains(buf.String(), "covers its guests' swap allowance again") {
t.Fatalf("the recovery must be said once:\n%s", buf.String())
}
if HostSwapShort(0, 0) {
t.Fatal("a host whose guests have no swap allowance is never short")
}
}
+11 -1
View File
@@ -292,6 +292,10 @@ type WireguardStatus struct {
Registered bool `json:"registered"` // the registration marker exists Registered bool `json:"registered"` // the registration marker exists
Active bool `json:"active"` // wg-quick@wg-felhom is-active Active bool `json:"active"` // wg-quick@wg-felhom is-active
LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"` LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"`
// HandshakeFutureS (R-1204): seconds the latest handshake lies in the FUTURE by this box's clock — set (> 0) only
// then. A box that ran once with its clock ahead keeps a handshake stamped in the future; WireGuard on ep0 then
// refuses every newer handshake, and LastHandshakeAgeS (clamped at 0) reads „just now" for ever. The hub alarms on it.
HandshakeFutureS *int64 `json:"handshake_future_s,omitempty"`
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32" AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
} }
@@ -338,6 +342,11 @@ type HostMetrics struct {
// //
// Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift. // Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift.
WrapperSHA256 string `json:"wrapper_sha256,omitempty"` WrapperSHA256 string `json:"wrapper_sha256,omitempty"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up, night 2026-10-21): the host's swap (Proxmox node status — the
// kernel's SwapTotal) and the sum of its guests' swap allowance. The guest swap fix (swap = RAM) needs swap ON THE
// HOST behind it; the hub raises host_swap_short when the first is below the second. nil = not read (older agent).
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
} }
// Guest is one LXC. The agent reports vmid; the hub derives the guest PK // Guest is one LXC. The agent reports vmid; the hub derives the guest PK
@@ -355,11 +364,12 @@ type GuestSpec struct {
Cores int `json:"cores"` Cores int `json:"cores"`
MemoryBytes int64 `json:"memory_bytes"` MemoryBytes int64 `json:"memory_bytes"`
DiskBytes int64 `json:"disk_bytes"` DiskBytes int64 `json:"disk_bytes"`
SwapBytes int64 `json:"swap_bytes,omitempty"` // R-1192: the guest's swap allowance (PVE config `swap`, MB → bytes)
} }
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest. // Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
type Cloudflared struct { type Cloudflared struct {
Status string `json:"status"` // running | not_running | unknown (TunnelRunning …) Status string `json:"status"` // running | not_running | unknown | absent (TunnelRunning …; absent since R-1070)
Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected" Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected"
} }
+146
View File
@@ -0,0 +1,146 @@
package localapi
import (
"fmt"
"net/http"
"sort"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// The whole-guest backup cadence cap (R-973, security review 2026-10-12).
//
// POST /backup is reachable from inside the guest, and every successful run prunes the tier
// (`--prune-backups keep-last=N` on a local tier; ep0's server-side keep-last 2 on the off-site tier). So
// a compromised controller calling it in a loop pushed every pre-compromise whole-guest restore point out
// within minutes — the threat `07` §8 row 10 counts the off-site tier as the answer to. The agent now
// refuses a run that the schedule in `07` §6 would never ask for:
//
// - a tier whose cadence is at most 24 h (the local tier): at most 3 successful runs in any rolling
// 24 h. The schedule is one night run; a „Mentés most" press makes a LOCAL copy and, since R-899, the
// night still takes its own after it, which can land inside the same 24 h as the previous night (the
// window opens anywhere in [W+2h, W+6h)). So a normal day needs at most 3: two nights plus one press.
// A second press inside 24 h is refused — the margin is spent on the night, which must never be refused
// because of the household's presses.
// - a longer tier (the 7-day off-site tier): at most 1 successful run per max(cadence − 24 h, cadence/2)
// — 6 days for 7 days. The controller asks /backup/due first and runs only when the newest copy is older
// than the cadence, and the button never targets this tier (`07` §6, controller v0.301.0), so a
// legitimate run never comes sooner; 24 h of slack absorbs clock and archive-time differences.
//
// Only SUCCESSES count — a failed run prunes nothing and must stay retryable. The count survives an agent
// restart: it reads the success history saved on disk (LastKnownBackupStore, the R-894 file) as well as
// this process's own record. A refusal answers 429, is logged at WARN, is saved, and raises the operator
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's operator-only
// `whole_guest_backup_refused` event since hub v0.159.0, R-1150). Pinned by TestBackupCap_* (backup_cap_test.go).
// backupCapRefusalHold is how long one refusal keeps the operator signal raised.
const backupCapRefusalHold = 24 * time.Hour
// backupCadenceCap returns the rolling window and the most successful runs allowed in it for a tier.
func backupCadenceCap(cadence time.Duration) (window time.Duration, max int) {
if cadence <= 0 {
cadence = defaultBackupCadence
}
if cadence <= 24*time.Hour {
return 24 * time.Hour, 3
}
w := cadence - 24*time.Hour
if w < cadence/2 {
w = cadence / 2
}
return w, 1
}
// recentCapRuns returns the start times of the successful runs on this tier for this guest inside the
// window ending now — this process's record and the saved one, each counted once. Caller holds no lock.
func (s *Server) recentCapRuns(key backupJobKey, since time.Time) []time.Time {
seen := map[int64]bool{}
var out []time.Time
add := func(t time.Time) {
if t.Before(since) || seen[t.Unix()] {
return
}
seen[t.Unix()] = true
out = append(out, t)
}
s.capMu.Lock()
for _, t := range s.capRuns[key] {
add(t)
}
s.capMu.Unlock()
if s.lastKnown != nil {
for _, t := range s.lastKnown.RecentBackupSuccesses(key.target, key.vmid, since) {
add(t)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Before(out[j]) })
return out
}
// recordCapRun notes a successful run for the cap (the saved half is written by RecordBackupSuccess).
func (s *Server) recordCapRun(key backupJobKey, b hub.Backup) {
t, err := time.Parse(time.RFC3339, b.StartedAt)
if err != nil {
t = s.now()
}
s.capMu.Lock()
defer s.capMu.Unlock()
if s.capRuns == nil {
s.capRuns = map[backupJobKey][]time.Time{}
}
runs := append(s.capRuns[key], t.UTC())
if len(runs) > 8 {
runs = runs[len(runs)-8:]
}
s.capRuns[key] = runs
}
// capRefuses applies the cap to a new run request. It writes the 429 and returns true when refused.
func (s *Server) capRefuses(w http.ResponseWriter, key backupJobKey, tier BackupTier) bool {
window, max := backupCadenceCap(tier.Cadence)
now := s.now()
runs := s.recentCapRuns(key, now.Add(-window))
if len(runs) < max {
return false
}
next := runs[len(runs)-max].Add(window)
s.capMu.Lock()
s.capRefusal = capRefusal{target: key.target, vmid: key.vmid, at: now.UTC()}
s.capMu.Unlock()
if s.lastKnown != nil {
if err := s.lastKnown.RecordBackupRefusal(key.target, key.vmid, now); err != nil {
s.logger.Warn("local-api: could not save the backup refusal on disk (R-973)", "vmid", key.vmid, "target", key.target, "err", err)
}
}
s.logger.Warn("local-api: backup REFUSED — more whole-guest backups than the schedule allows (R-973 cadence cap; check the guest)",
"vmid", key.vmid, "target", key.target, "runs_in_window", len(runs), "allowed", max,
"window_h", int(window.Hours()), "next_allowed", next.UTC().Format(time.RFC3339))
writeStatus(w, http.StatusTooManyRequests, false, nil, fmt.Sprintf(
"too many whole-guest backups on target %q: %d successful in the last %d h, the schedule allows %d — refused (the next one is allowed after %s)",
key.target, len(runs), int(window.Hours()), max, next.UTC().Format(time.RFC3339)))
return true
}
type capRefusal struct {
target string
vmid int
at time.Time
}
// BackupCapRefusal reports the newest cadence-cap refusal still inside the 24 h hold (this process's or
// the saved one) — the source of the operator signal in the host report (R-973). ok=false = none.
func (s *Server) BackupCapRefusal() (target string, vmid int, at time.Time, ok bool) {
s.capMu.Lock()
r := s.capRefusal
s.capMu.Unlock()
if s.lastKnown != nil {
if t, v, a, found := s.lastKnown.LastBackupRefusal(); found && a.After(r.at) {
r = capRefusal{target: t, vmid: v, at: a}
}
}
if r.at.IsZero() || s.now().Sub(r.at) > backupCapRefusalHold {
return "", 0, time.Time{}, false
}
return r.target, r.vmid, r.at, true
}
+377
View File
@@ -0,0 +1,377 @@
package localapi
import (
"context"
"fmt"
"io"
"log/slog"
"net/http"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-973 — the whole-guest backup cadence cap. A compromised controller calling POST /backup in a loop
// pushed out every pre-compromise restore point: each success prunes the tier. These tests model the tier
// the way Proxmox and ep0 keep it (keep-last N, pruned after each success) and ask what is LEFT.
// keepLastTier is a whole-guest tier: every successful run adds an archive stamped with the clock and
// prunes to keepLast, as vzdump's --prune-backups keep-last (local) and ep0's keep-last 2 (off-site) do.
type keepLastTier struct {
mu sync.Mutex
clock *testClock
keepLast int
// spec, when set, is the `--prune-backups` value the agent passes (R-1149) and the tier prunes the way Proxmox
// reads it: keep-last=N → the newest N; keep-daily=N → the newest copy of each of the N newest days with one.
spec string
t *testing.T
archives []time.Time
fail bool
runs int
}
// pruneBySpec applies a Proxmox `--prune-backups` spec to archives (oldest first) and returns what is kept.
func pruneBySpec(t *testing.T, archives []time.Time, spec string) []time.Time {
var n int
var mode string
if _, err := fmt.Sscanf(spec, "keep-last=%d", &n); err == nil {
mode = "last"
} else if _, err := fmt.Sscanf(spec, "keep-daily=%d", &n); err == nil {
mode = "daily"
} else {
t.Fatalf("the model does not know the prune spec %q", spec)
}
var kept []time.Time
days := map[string]bool{}
for i := len(archives) - 1; i >= 0; i-- { // newest first, as Proxmox walks them
a := archives[i]
switch mode {
case "last":
if len(kept) < n {
kept = append([]time.Time{a}, kept...)
}
case "daily":
d := a.Format("2006-01-02")
if !days[d] && len(days) < n {
days[d] = true
kept = append([]time.Time{a}, kept...)
}
}
}
return kept
}
func (k *keepLastTier) BackupWithSnapshotHook(_ context.Context, vmid int, _ func()) (hub.Backup, error) {
k.mu.Lock()
defer k.mu.Unlock()
k.runs++
now := k.clock.Now()
if k.fail {
return hub.Backup{VMID: vmid, StartedAt: now.Format(time.RFC3339)}, fmt.Errorf("vzdump failed")
}
k.archives = append(k.archives, now)
if k.spec != "" {
k.archives = pruneBySpec(k.t, k.archives, k.spec)
} else if len(k.archives) > k.keepLast {
k.archives = k.archives[len(k.archives)-k.keepLast:]
}
return hub.Backup{VMID: vmid, Success: true, Archive: "x", StartedAt: now.Format(time.RFC3339)}, nil
}
// oldestBefore reports whether the tier still holds an archive made before t (a pre-compromise point).
func (k *keepLastTier) oldestBefore(t time.Time) bool {
k.mu.Lock()
defer k.mu.Unlock()
return len(k.archives) > 0 && k.archives[0].Before(t)
}
func (k *keepLastTier) count() (runs, kept int) {
k.mu.Lock()
defer k.mu.Unlock()
return k.runs, len(k.archives)
}
type testClock struct {
mu sync.Mutex
t time.Time
}
func (c *testClock) Now() time.Time { c.mu.Lock(); defer c.mu.Unlock(); return c.t }
func (c *testClock) Advance(d time.Duration) { c.mu.Lock(); c.t = c.t.Add(d); c.mu.Unlock() }
// capServer builds the two-tier box of `07` §6: local 24 h keep-last 3, off-site 7 d keep-last 2, the
// success history on disk at path (a second capServer on the same path is an agent restart).
func capServer(t *testing.T, path string, clock *testClock, local, offsite *keepLastTier) *Server {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local"}, {Name: "felhom-pbs"}}},
Tokens: staticTokens{"A": 8200},
BackupTiers: []BackupTier{
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: local},
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: offsite},
},
LastKnownBackups: backup.NewBackupSuccessState(path),
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.now = clock.Now
return srv
}
// runBackup POSTs one backup and waits for the job to end. Returns the HTTP status of the POST.
func runBackup(t *testing.T, srv *Server, target string) int {
t.Helper()
rr := do(t, srv.Handler(), "POST", "/backup?target="+target, "A", "")
if rr.Code == http.StatusAccepted {
waitFor(t, func() bool {
srv.jobsMu.Lock()
defer srv.jobsMu.Unlock()
j := srv.jobs[backupJobKey{vmid: 8200, target: target}]
return j != nil && !backupInFlight(j.Phase)
})
}
return rr.Code
}
// The off-site tier — the one `07` §8 row 10 counts on against ransomware. Before the cap ten calls in a
// row left ep0 holding only the attacker's two copies. Now ONE runs (the tier was due), the other nine
// are refused, and the newest pre-compromise copy survives; five days of flooding later it still does.
//
// COMPANION RED-PROOF (observed 2026-10-18): make capRefuses always return false → this fails with
// "the off-site tier lost every pre-compromise copy: 10 runs" . Restored.
func TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy(t *testing.T) {
clock := &testClock{t: testNow}
local := &keepLastTier{clock: clock, keepLast: 3}
off := &keepLastTier{clock: clock, keepLast: 2}
path := filepath.Join(t.TempDir(), "backup-success-state.json")
srv := capServer(t, path, clock, local, off)
// The honest history: two weekly copies, the newest 7 days and 1 hour old (so the tier is due).
for _, age := range []time.Duration{14*24*time.Hour + time.Hour, 7*24*time.Hour + time.Hour} {
clock.t = testNow.Add(-age)
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusAccepted {
t.Fatalf("seeding the honest weekly copy: %d", code)
}
}
clock.t = testNow
compromise := testNow
refused := 0
for i := 0; i < 10; i++ {
if runBackup(t, srv, "felhom-pbs") == http.StatusTooManyRequests {
refused++
}
clock.Advance(time.Minute)
}
runs, _ := off.count()
if !off.oldestBefore(compromise) {
t.Fatalf("the off-site tier lost every pre-compromise copy: %d runs", runs-2)
}
if runs-2 != 1 || refused != 9 {
t.Fatalf("flood: %d runs, %d refused — want 1 run (the tier was due) and 9 refused", runs-2, refused)
}
for d := 0; d < 5; d++ { // five more days of flooding, every hour
for h := 0; h < 24; h++ {
clock.Advance(time.Hour)
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusTooManyRequests {
t.Fatalf("day %d hour %d: an off-site run was allowed %v after the last one", d, h, clock.Now().Sub(compromise))
}
}
}
if !off.oldestBefore(compromise) {
t.Fatal("five days of flooding pushed out the pre-compromise off-site copy")
}
}
// The local tier: a flood runs exactly 3 in a day (the schedule's own day: two nights + one press) and
// every further request is refused. Honest about the limit: with keep-last 3 those 3 replace the 3 kept
// points — the cap slows a flush from minutes to a day; it cannot keep a point a 3-per-day pace prunes.
func TestBackupCap_LocalFloodRunsAtMostThreeADay(t *testing.T) {
clock := &testClock{t: testNow}
local := &keepLastTier{clock: clock, keepLast: 3}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
refused := 0
for i := 0; i < 20; i++ {
if runBackup(t, srv, "local") == http.StatusTooManyRequests {
refused++
}
clock.Advance(10 * time.Minute)
}
if runs, _ := local.count(); runs != 3 || refused != 17 {
t.Fatalf("local flood over ~3 h: %d runs, %d refused — want 3 and 17", runs, refused)
}
clock.Advance(24 * time.Hour) // a day later the window has moved on
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("a run a day later must be allowed: %d", code)
}
}
// The schedule itself is never refused: a night at W+6h, a press at 10:00, the next night at W+2h (all
// three inside one rolling 24 h) run; a second press that day is refused; nights keep running after it.
func TestBackupCap_TheScheduledDayIsNeverRefused(t *testing.T) {
day := time.Date(2026, 6, 10, 0, 0, 0, 0, time.UTC)
clock := &testClock{t: day.Add(8*time.Hour + 29*time.Minute)} // W=02:30 → W+6h ≈ 08:29
local := &keepLastTier{clock: clock, keepLast: 3}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
steps := []struct {
at time.Time
want int
what string
}{
{day.Add(8*time.Hour + 29*time.Minute), http.StatusAccepted, "night 1 at W+6h"},
{day.Add(10 * time.Hour), http.StatusAccepted, "a press at 10:00"},
{day.Add(24*time.Hour + 4*time.Hour + 30*time.Minute), http.StatusAccepted, "night 2 at W+2h (20 h later)"},
{day.Add(24*time.Hour + 7*time.Hour), http.StatusTooManyRequests, "a second press inside the same 24 h"},
{day.Add(48*time.Hour + 4*time.Hour + 30*time.Minute), http.StatusAccepted, "night 3"},
}
for _, s := range steps {
clock.t = s.at
if got := runBackup(t, srv, "local"); got != s.want {
t.Fatalf("%s: got %d, want %d", s.what, got, s.want)
}
}
}
// A restart must not reset the cap: the history is read back from disk.
//
// COMPANION RED-PROOF (observed 2026-10-18): make recentCapRuns ignore s.lastKnown → this fails with
// "after a restart the 4th run in a day was allowed (got 202)". Restored.
func TestBackupCap_SurvivesARestart(t *testing.T) {
clock := &testClock{t: testNow}
path := filepath.Join(t.TempDir(), "s.json")
local := &keepLastTier{clock: clock, keepLast: 3}
srv := capServer(t, path, clock, local, &keepLastTier{clock: clock, keepLast: 2})
for i := 0; i < 3; i++ {
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("run %d: %d", i+1, code)
}
clock.Advance(time.Minute)
}
srv2 := capServer(t, path, clock, local, &keepLastTier{clock: clock, keepLast: 2}) // the restart
if code := runBackup(t, srv2, "local"); code != http.StatusTooManyRequests {
t.Fatalf("after a restart the 4th run in a day was allowed (got %d)", code)
}
}
// Failed runs prune nothing and must stay retryable: they do not count.
func TestBackupCap_FailuresDoNotCount(t *testing.T) {
clock := &testClock{t: testNow}
local := &keepLastTier{clock: clock, keepLast: 3, fail: true}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
for i := 0; i < 6; i++ {
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("failed attempt %d was refused: %d", i+1, code)
}
clock.Advance(15 * time.Minute)
}
local.mu.Lock()
local.fail = false
local.mu.Unlock()
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("a success after six failures was refused: %d", code)
}
}
// A refusal raises the operator signal for 24 h, and the signal survives a restart.
func TestBackupCap_RefusalRaisesTheOperatorSignal(t *testing.T) {
clock := &testClock{t: testNow}
path := filepath.Join(t.TempDir(), "s.json")
off := &keepLastTier{clock: clock, keepLast: 2}
srv := capServer(t, path, clock, &keepLastTier{clock: clock, keepLast: 3}, off)
if _, _, _, ok := srv.BackupCapRefusal(); ok {
t.Fatal("a fresh box must not raise the signal")
}
runBackup(t, srv, "felhom-pbs")
clock.Advance(time.Hour)
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusTooManyRequests {
t.Fatalf("second off-site run an hour later: %d, want 429", code)
}
target, vmid, at, ok := srv.BackupCapRefusal()
if !ok || target != "felhom-pbs" || vmid != 8200 || !at.Equal(clock.Now()) {
t.Fatalf("signal after a refusal: ok=%v target=%q vmid=%d at=%v", ok, target, vmid, at)
}
clock.Advance(23 * time.Hour)
srv2 := capServer(t, path, clock, &keepLastTier{clock: clock, keepLast: 3}, off) // a restart
if _, _, _, ok := srv2.BackupCapRefusal(); !ok {
t.Fatal("the signal did not survive a restart")
}
clock.Advance(2 * time.Hour)
if _, _, _, ok := srv2.BackupCapRefusal(); ok {
t.Fatal("the signal must clear 24 h after the last refusal")
}
}
func TestBackupCadenceCap(t *testing.T) {
cases := []struct {
cadence time.Duration
window time.Duration
max int
}{
{0, 24 * time.Hour, 3},
{24 * time.Hour, 24 * time.Hour, 3},
{12 * time.Hour, 24 * time.Hour, 3},
{7 * 24 * time.Hour, 6 * 24 * time.Hour, 1},
{36 * time.Hour, 18 * time.Hour, 1},
{30 * 24 * time.Hour, 29 * 24 * time.Hour, 1},
}
for _, c := range cases {
if w, m := backupCadenceCap(c.cadence); w != c.window || m != c.max {
t.Errorf("cadence %v: got %v/%d, want %v/%d", c.cadence, w, m, c.window, c.max)
}
}
}
// R-1149 — the consequence of keep-daily on the local tier, with the agent's REAL prune spec (config.BackupTiers →
// PruneSpec, what runDaemon hands the runner). Three honest nights, then a compromise at 01:00 that floods
// POST /backup for the rest of the day. The cap lets 3 runs through inside the rolling 24 h (two at once, one when
// last night's success ages out). With keep-last 3 those three were the only copies left; with keep-daily 3 the two
// earlier nights survive the whole day.
//
// COMPANION RED-PROOF (observed 2026-10-19): make BackupTier.PruneSpec return keep-last for the primary → this fails
// with "after a day of flooding the local tier holds 0 copies made before the compromise (kept […]) — want the 2
// earlier nights". Restored.
func TestBackupCap_R1149_LocalFloodKeepsOlderDays(t *testing.T) {
tiers, _ := config.BackupConfig{LocalBackupTarget: "local", LocalBackupRetention: 3}.BackupTiers()
spec := tiers[0].PruneSpec()
day := time.Date(2026, 6, 10, 0, 0, 0, 0, time.UTC)
clock := &testClock{t: day}
local := &keepLastTier{clock: clock, keepLast: 3, spec: spec, t: t}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
for d := 3; d >= 1; d-- { // the nights of D-3, D-2, D-1 at 03:00
clock.t = day.Add(-time.Duration(d)*24*time.Hour + 3*time.Hour)
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("seeding night D-%d: %d", d, code)
}
}
compromise := day.Add(time.Hour)
ran := 0
for clock.t = compromise; clock.t.Before(day.Add(24 * time.Hour)); clock.Advance(10 * time.Minute) {
if runBackup(t, srv, "local") == http.StatusAccepted {
ran++
}
}
if ran != 3 {
t.Fatalf("the flood ran %d backups on day D — want 3 (the cap)", ran)
}
local.mu.Lock()
kept := append([]time.Time(nil), local.archives...)
local.mu.Unlock()
pre := 0
for _, a := range kept {
if a.Before(compromise) {
pre++
}
}
if pre != 2 {
t.Fatalf("after a day of flooding the local tier holds %d copies made before the compromise (kept %v) — want the 2 earlier nights", pre, kept)
}
}
+308 -12
View File
@@ -3,11 +3,15 @@ package localapi
import ( import (
"bytes" "bytes"
"context" "context"
"crypto/rand"
"crypto/subtle"
"encoding/json" "encoding/json"
"fmt" "fmt"
"math/big"
"net/http" "net/http"
"os/exec" "os/exec"
"strconv" "strconv"
"strings"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow" "gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
@@ -24,9 +28,23 @@ import (
// be structurally incapable of carrying R). // be structurally incapable of carrying R).
// - One claim, then the holder is zeroed. Unclaimed past the TTL → zeroed + phase // - One claim, then the holder is zeroed. Unclaimed past the TTL → zeroed + phase
// unclaimed_void ("R unclaimed → ceremony void → a re-run supersedes"). // unclaimed_void ("R unclaimed → ceremony void → a re-run supersedes").
// - Crash-safety is IN-MEMORY BY DESIGN: an agent restart loses R, which is SAFE (the blob is // - Crash-safety is IN-MEMORY BY DESIGN: an agent restart loses R AND the package it opens,
// on the hub; a re-run supersedes it). Status reporting "none" after a restart is the honest // which is SAFE (R-978: the package was never uploaded, so the hub still holds the previous one
// answer. No journal, deliberately — persistence is the one property R must never have. // and the customer's previous code still opens it; a re-run starts over). Status reporting "none"
// after a restart is the honest answer. No journal, deliberately — persistence is the one
// property R must never have.
// - R-978: the NEW package reaches the hub only at the claim, and the code is handed out only after
// the hub took it. Until then the previous package stays current — so a ceremony that is started
// and abandoned (never claimed, superseded, TTL, restart) leaves the customer's written-down code
// working. The TTL and a new start both wait while a claim's upload is in flight, so a package is
// never stored without its code being handed out by that same claim.
// - R-1151: a claim that asks for it (`{"defer_upload": true}`, controller ≥ 0.332.0) hands out the code
// and two word positions, and keeps the package until POST /escrow/ceremony/confirm brings those two
// words back — the household's typed-back check, now checked HERE, not only in the browser. A code that
// was shown and never typed back (the page closed) never replaces the hub's package; the confirm window
// (escrowConfirmTTL) ends it. Only the two words are kept for the check, never R. A claim WITHOUT the
// flag (a controller that has not updated) keeps the R-978 order: upload at the claim. Pinned by
// TestEscrowCeremony_R1151_*.
// - Nothing from stdout is ever logged; stderr (log-clean, spike §2.1) is tail-captured for // - Nothing from stdout is ever logged; stderr (log-clean, spike §2.1) is tail-captured for
// failure diagnostics only. // failure diagnostics only.
@@ -50,8 +68,16 @@ type EscrowCeremonyConfig struct {
// DRConfigured answers "is the DR tier applied on this box?" (pbsdr.Manager.DRConfigured, // DRConfigured answers "is the DR tier applied on this box?" (pbsdr.Manager.DRConfigured,
// late-bound). nil → reported not-applied. // late-bound). nil → reported not-applied.
DRConfigured func() bool DRConfigured func() bool
// Upload sends a ceremony's package (CeremonyOutput.PendingUpload, the hub upload body) to this
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
Upload func(ctx context.Context, pkg []byte) error
} }
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
// ceremony incl. its upload was measured at ~2.4 s).
const escrowUploadTimeout = 30 * time.Second
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until // ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned // parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
// spike-shaped output; production is runCeremonySubprocess. // spike-shaped output; production is runCeremonySubprocess.
@@ -78,8 +104,15 @@ const (
escrowPhaseDone = "done" escrowPhaseDone = "done"
escrowPhaseFailed = "failed" escrowPhaseFailed = "failed"
escrowPhaseVoid = "unclaimed_void" escrowPhaseVoid = "unclaimed_void"
// escrowPhaseUnconfirmed (R-1151): the code was shown but its words never came back inside the confirm
// window — the package was dropped, the hub's previous package (and the previous code) stays current.
escrowPhaseUnconfirmed = "unconfirmed_void"
) )
// escrowConfirmTTL is how long a claimed code waits for its typed-back words (R-1151). Writing ten words down
// by hand takes minutes; past this the package is dropped and the household starts again.
const escrowConfirmTTL = 30 * time.Minute
// escrowCeremonyTimeout bounds the whole subprocess run. Spike-measured ceremony ≈ 2.4 s incl. // escrowCeremonyTimeout bounds the whole subprocess run. Spike-measured ceremony ≈ 2.4 s incl.
// upload — 60 s is a ≥25× margin that still absorbs WAN upload latency. // upload — 60 s is a ≥25× margin that still absorbs WAN upload latency.
const escrowCeremonyTimeout = 60 * time.Second const escrowCeremonyTimeout = 60 * time.Second
@@ -129,6 +162,9 @@ func (s *Server) tryStartEscrowCeremony(job *escrowCeremonyJob) bool {
if s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning { if s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning {
return false return false
} }
if s.escrowClaiming {
return false // R-978: a claim is uploading this job's package — its code must reach the claimer
}
s.wipeEscrowRLocked() s.wipeEscrowRLocked()
cp := *job cp := *job
s.escrowJob = &cp s.escrowJob = &cp
@@ -137,7 +173,7 @@ func (s *Server) tryStartEscrowCeremony(job *escrowCeremonyJob) bool {
// finishEscrowCeremony records the terminal phase and (on success) arms the one-shot R holder. // finishEscrowCeremony records the terminal phase and (on success) arms the one-shot R holder.
// R is stored as a byte slice so the claim/TTL paths can zero it in place. // R is stored as a byte slice so the claim/TTL paths can zero it in place.
func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string) { func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string, pending []byte) {
s.escrowMu.Lock() s.escrowMu.Lock()
defer s.escrowMu.Unlock() defer s.escrowMu.Unlock()
job.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339) job.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
@@ -145,6 +181,7 @@ func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string) {
s.escrowJob = &cp s.escrowJob = &cp
if job.Phase == escrowPhaseDone && r != "" { if job.Phase == escrowPhaseDone && r != "" {
s.escrowR = []byte(r) s.escrowR = []byte(r)
s.escrowPending = append([]byte(nil), pending...)
s.escrowRClaimed = false s.escrowRClaimed = false
s.escrowRExpiry = s.nowFn().Add(escrowClaimTTL) s.escrowRExpiry = s.nowFn().Add(escrowClaimTTL)
// Active TTL belt: zero the holder even if nobody ever polls again. The lazy check in // Active TTL belt: zero the holder even if nobody ever polls again. The lazy check in
@@ -164,6 +201,53 @@ func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string) {
func (s *Server) wipeEscrowRLocked() { func (s *Server) wipeEscrowRLocked() {
zeroBytes(s.escrowR) zeroBytes(s.escrowR)
s.escrowR = nil s.escrowR = nil
s.escrowPending = nil // R-978: the package goes with its code — never uploaded once the code is gone
s.wipeEscrowConfirmLocked()
}
// wipeEscrowConfirmLocked drops the R-1151 typed-back state: the two kept words and the wait. Caller holds escrowMu.
func (s *Server) wipeEscrowConfirmLocked() {
for i := range s.escrowConfirmWords {
zeroBytes(s.escrowConfirmWords[i])
s.escrowConfirmWords[i] = nil
}
s.escrowAwaitConfirm = false
}
// checkEscrowConfirmTTLLocked ends a typed-back wait that ran past escrowConfirmTTL (R-1151): the package is
// dropped (never uploaded) and the job reads unconfirmed_void. Not while a confirm is uploading. Caller holds escrowMu.
func (s *Server) checkEscrowConfirmTTLLocked() {
if !s.escrowAwaitConfirm || s.escrowClaiming || !s.nowFn().After(s.escrowConfirmExpiry) {
return
}
s.escrowPending = nil
s.wipeEscrowConfirmLocked()
if s.escrowJob != nil {
s.escrowJob.Phase = escrowPhaseUnconfirmed
s.escrowJob.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
s.logger.Info("local-api: escrow code was not typed back in time — the new package was dropped; the previous one stays current (R-1151)",
"job_id", s.escrowJob.JobID)
}
}
// pickConfirmPositions draws two distinct 1-based word positions out of n (n ≥ 2), smaller first.
func pickConfirmPositions(n int) ([2]int, error) {
a, err := rand.Int(rand.Reader, big.NewInt(int64(n)))
if err != nil {
return [2]int{}, err
}
b, err := rand.Int(rand.Reader, big.NewInt(int64(n-1)))
if err != nil {
return [2]int{}, err
}
i, j := int(a.Int64()), int(b.Int64())
if j >= i {
j++
}
if j < i {
i, j = j, i
}
return [2]int{i + 1, j + 1}, nil
} }
// expireEscrowRLocked applies the TTL outcome: an armed, unclaimed holder is zeroed and the job // expireEscrowRLocked applies the TTL outcome: an armed, unclaimed holder is zeroed and the job
@@ -172,6 +256,9 @@ func (s *Server) expireEscrowRLocked() {
if s.escrowJob == nil || s.escrowJob.Phase != escrowPhaseDone || s.escrowRClaimed { if s.escrowJob == nil || s.escrowJob.Phase != escrowPhaseDone || s.escrowRClaimed {
return return
} }
if s.escrowClaiming {
return // R-978: the claim is uploading the package; voiding now would store a package nobody gets the code to
}
if len(s.escrowR) == 0 { if len(s.escrowR) == 0 {
return return
} }
@@ -217,7 +304,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
UpdatedAt: s.nowFn().UTC().Format(time.RFC3339), UpdatedAt: s.nowFn().UTC().Format(time.RFC3339),
} }
if !s.tryStartEscrowCeremony(job) { if !s.tryStartEscrowCeremony(job) {
writeErr(w, http.StatusConflict, "an escrow ceremony is already running") writeErr(w, http.StatusConflict, "an escrow ceremony is already running, or its code is being claimed")
return return
} }
s.logger.Info("local-api: escrow ceremony started (controller-driven)", "vmid", vmid, "job_id", job.JobID) s.logger.Info("local-api: escrow ceremony started (controller-driven)", "vmid", vmid, "job_id", job.JobID)
@@ -247,7 +334,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
detail += " | stderr: " + tail detail += " | stderr: " + tail
} }
job.Phase, job.Detail = escrowPhaseFailed, detail job.Phase, job.Detail = escrowPhaseFailed, detail
s.finishEscrowCeremony(job, "") s.finishEscrowCeremony(job, "", nil)
s.logger.Warn("local-api: escrow ceremony failed", "job_id", job.JobID, s.logger.Warn("local-api: escrow ceremony failed", "job_id", job.JobID,
"exit", exit, "duration_ms", time.Since(start).Milliseconds()) "exit", exit, "duration_ms", time.Since(start).Milliseconds())
return return
@@ -255,24 +342,34 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
var out escrow.CeremonyOutput var out escrow.CeremonyOutput
if err := json.Unmarshal(stdout, &out); err != nil { if err := json.Unmarshal(stdout, &out); err != nil {
job.Phase, job.Detail = escrowPhaseFailed, "ceremony output is not the expected JSON object" job.Phase, job.Detail = escrowPhaseFailed, "ceremony output is not the expected JSON object"
s.finishEscrowCeremony(job, "") s.finishEscrowCeremony(job, "", nil)
s.logger.Warn("local-api: escrow ceremony output unparseable (never logged)", "job_id", job.JobID) s.logger.Warn("local-api: escrow ceremony output unparseable (never logged)", "job_id", job.JobID)
return return
} }
if out.Version != escrow.CeremonyOutputVersion || out.RecoveryCode == "" { if out.Version != escrow.CeremonyOutputVersion || out.RecoveryCode == "" {
out.RecoveryCode = "" out.RecoveryCode = ""
job.Phase, job.Detail = escrowPhaseFailed, fmt.Sprintf("unexpected ceremony output (version %d)", out.Version) job.Phase, job.Detail = escrowPhaseFailed, fmt.Sprintf("unexpected ceremony output (version %d)", out.Version)
s.finishEscrowCeremony(job, "") s.finishEscrowCeremony(job, "", nil)
return
}
// R-978: the package must come back un-uploaded. An output that uploaded by itself (or carries no
// package) would mean the previous package is already gone, or the new one can never be stored.
if out.Uploaded || len(out.PendingUpload) == 0 {
out.RecoveryCode = ""
job.Phase, job.Detail = escrowPhaseFailed, "ceremony output carries no pending package"
s.finishEscrowCeremony(job, "", nil)
s.logger.Warn("local-api: escrow ceremony output has no pending package — failed (R-978)", "job_id", job.JobID,
"uploaded", out.Uploaded)
return return
} }
job.Phase = escrowPhaseDone job.Phase = escrowPhaseDone
job.KeyFingerprint = out.KeyFingerprint job.KeyFingerprint = out.KeyFingerprint
job.EntropyBits = out.EntropyBits job.EntropyBits = out.EntropyBits
job.ResticPwSealed = out.ResticPwSealed job.ResticPwSealed = out.ResticPwSealed
job.Uploaded = out.Uploaded job.Uploaded = false // R-978: uploaded by the claim
s.finishEscrowCeremony(job, out.RecoveryCode) s.finishEscrowCeremony(job, out.RecoveryCode, out.PendingUpload)
out.RecoveryCode = "" // drop the parsed reference promptly (GC caveat: best-effort) out.RecoveryCode = "" // drop the parsed reference promptly (GC caveat: best-effort)
s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot)", s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim, or at the typed-back confirm (R-1151)",
"job_id", job.JobID, "restic_pw_sealed", job.ResticPwSealed, "uploaded", job.Uploaded, "job_id", job.JobID, "restic_pw_sealed", job.ResticPwSealed, "uploaded", job.Uploaded,
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds()) "claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
}() }()
@@ -290,6 +387,7 @@ func (s *Server) handleEscrowCeremonyStatus(w http.ResponseWriter, r *http.Reque
} }
s.escrowMu.Lock() s.escrowMu.Lock()
s.checkEscrowTTLLocked() s.checkEscrowTTLLocked()
s.checkEscrowConfirmTTLLocked()
if s.escrowJob == nil { if s.escrowJob == nil {
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeOK(w, map[string]any{"vmid": vmid, "phase": escrowPhaseNone}) writeOK(w, map[string]any{"vmid": vmid, "phase": escrowPhaseNone})
@@ -304,6 +402,13 @@ func (s *Server) handleEscrowCeremonyStatus(w http.ResponseWriter, r *http.Reque
} }
} }
claimed := s.escrowRClaimed claimed := s.escrowRClaimed
awaiting := s.escrowAwaitConfirm
confirmIn := 0
if awaiting {
if d := s.escrowConfirmExpiry.Sub(s.nowFn()); d > 0 {
confirmIn = int(d.Seconds())
}
}
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeOK(w, map[string]any{ writeOK(w, map[string]any{
@@ -312,6 +417,7 @@ func (s *Server) handleEscrowCeremonyStatus(w http.ResponseWriter, r *http.Reque
"key_fingerprint": job.KeyFingerprint, "entropy_bits": job.EntropyBits, "key_fingerprint": job.KeyFingerprint, "entropy_bits": job.EntropyBits,
"restic_pw_sealed": job.ResticPwSealed, "uploaded": job.Uploaded, "restic_pw_sealed": job.ResticPwSealed, "uploaded": job.Uploaded,
"claimable": claimable, "claimed": claimed, "claim_expires_in_sec": expiresIn, "claimable": claimable, "claimed": claimed, "claim_expires_in_sec": expiresIn,
"awaiting_confirm": awaiting, "confirm_expires_in_sec": confirmIn,
"detail": job.Detail, "detail": job.Detail,
}) })
} }
@@ -324,6 +430,14 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
writeErr(w, http.StatusServiceUnavailable, "escrow ceremony not configured on this host") writeErr(w, http.StatusServiceUnavailable, "escrow ceremony not configured on this host")
return return
} }
// R-1151: a controller ≥ 0.332.0 asks for the package to wait for the typed-back words. An older controller
// sends `{}` (or nothing) and gets the R-978 order. Unknown fields are refused (decodeBody).
var req struct {
DeferUpload bool `json:"defer_upload"`
}
if !decodeBody(w, r, &req) {
return
}
s.escrowMu.Lock() s.escrowMu.Lock()
s.checkEscrowTTLLocked() s.checkEscrowTTLLocked()
switch { switch {
@@ -339,11 +453,101 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeErr(w, http.StatusConflict, "ceremony failed — nothing to claim") writeErr(w, http.StatusConflict, "ceremony failed — nothing to claim")
return return
case s.escrowClaiming:
s.escrowMu.Unlock()
writeErr(w, http.StatusConflict, "the recovery code is being claimed — try again in a moment")
return
case s.escrowRClaimed || len(s.escrowR) == 0 || s.escrowJob.Phase == escrowPhaseVoid: case s.escrowRClaimed || len(s.escrowR) == 0 || s.escrowJob.Phase == escrowPhaseVoid:
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeErr(w, http.StatusGone, "the recovery code is no longer available (already claimed or expired) — run a new ceremony; the new code supersedes") writeErr(w, http.StatusGone, "the recovery code is no longer available (already claimed or expired) — run a new ceremony; the new code supersedes")
return return
} }
// R-1151: the deferred order — hand out the code and two word positions; the package waits for
// POST /escrow/ceremony/confirm. Needs an uploader (a confirm could never store it otherwise) and a code of ≥ 2
// words (always 10; anything else falls back to the R-978 order rather than to no check).
if req.DeferUpload && len(s.escrowPending) > 0 && s.escrowCeremony.Upload != nil {
words := strings.Split(string(s.escrowR), escrow.RecoveryCodeSep)
if len(words) >= 2 {
pos, perr := pickConfirmPositions(len(words))
if perr == nil {
s.escrowConfirmPos = pos
s.escrowConfirmWords = [2][]byte{[]byte(words[pos[0]-1]), []byte(words[pos[1]-1])}
for i := range words {
words[i] = ""
}
s.escrowAwaitConfirm = true
s.escrowConfirmExpiry = s.nowFn().Add(escrowConfirmTTL)
recovery := string(s.escrowR)
zeroBytes(s.escrowR) // R only — the package and the two words stay for the confirm
s.escrowR = nil
s.escrowRClaimed = true
jobID := s.escrowJob.JobID
s.escrowMu.Unlock()
time.AfterFunc(escrowConfirmTTL+time.Second, func() {
s.escrowMu.Lock()
defer s.escrowMu.Unlock()
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
s.checkEscrowConfirmTTLLocked()
}
})
s.logger.Info("local-api: escrow ceremony R claimed — the package waits for the typed-back words (R-1151)",
"vmid", vmid, "job_id", jobID, "confirm_ttl_s", int(escrowConfirmTTL.Seconds()))
w.Header().Set("Cache-Control", "no-store")
writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "recovery_code": recovery,
"confirm_required": true, "verify_positions": []int{pos[0], pos[1]},
"confirm_expires_in_sec": int(escrowConfirmTTL.Seconds())})
recovery = ""
_ = recovery
return
}
s.logger.Warn("local-api: escrow claim — could not draw the confirm positions; package uploads at the claim (R-978 order)", "err", perr)
}
}
// R-978: store the NEW package on the hub first; only then hand out the code that opens it. Until this
// upload succeeds the hub keeps the previous package, and the customer's previous code keeps working.
if len(s.escrowPending) > 0 {
if s.escrowCeremony.Upload == nil {
s.escrowMu.Unlock()
writeErr(w, http.StatusServiceUnavailable, "the recovery package cannot be stored on this host (no upload configured) — the previous code still works")
return
}
pkg := s.escrowPending
jobID := s.escrowJob.JobID
s.escrowClaiming = true
s.escrowMu.Unlock()
start := time.Now()
base := s.baseCtx
if base == nil {
base = context.Background()
}
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg)
cancel()
s.escrowMu.Lock()
s.escrowClaiming = false
if err != nil {
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return
}
s.escrowPending = nil
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
s.escrowJob.Uploaded = true
s.escrowJob.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
}
s.logger.Info("local-api: escrow claim — package uploaded to the hub; handing out the code (R-978)",
"vmid", vmid, "job_id", jobID, "duration_ms", time.Since(start).Milliseconds())
if len(s.escrowR) == 0 { // cannot happen (TTL and supersede wait for the claim) — never hand out nothing
s.escrowMu.Unlock()
s.logger.Error("local-api: escrow claim — code vanished during the upload (invariant broken)", "job_id", jobID)
writeErr(w, http.StatusInternalServerError, "the recovery code was lost during the claim — run a new ceremony")
return
}
}
recovery := string(s.escrowR) recovery := string(s.escrowR)
s.wipeEscrowRLocked() s.wipeEscrowRLocked()
s.escrowRClaimed = true s.escrowRClaimed = true
@@ -352,11 +556,103 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
s.logger.Info("local-api: escrow ceremony R claimed (one-shot; holder zeroed)", "vmid", vmid, "job_id", jobID) s.logger.Info("local-api: escrow ceremony R claimed (one-shot; holder zeroed)", "vmid", vmid, "job_id", jobID)
w.Header().Set("Cache-Control", "no-store") w.Header().Set("Cache-Control", "no-store")
writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "recovery_code": recovery}) writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "recovery_code": recovery, "confirm_required": false})
recovery = "" // drop the reference promptly (GC caveat: best-effort) recovery = "" // drop the reference promptly (GC caveat: best-effort)
_ = recovery _ = recovery
} }
// handleEscrowCeremonyConfirm is POST /escrow/ceremony/confirm {"words": [w1, w2]} (R-1151): the two words at the
// positions the claim named, as the household typed them back. Right → the package goes to the hub now (200); wrong →
// 422 and the wait goes on; nothing waiting → 404 (or 410 after the window ran out); a failed upload → 502 and the
// wait goes on (a retry may succeed). The words are compared in constant time, case and spaces aside.
func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Request, vmid int) {
if s.escrowCeremony == nil {
writeErr(w, http.StatusServiceUnavailable, "escrow ceremony not configured on this host")
return
}
var req struct {
Words []string `json:"words"`
}
if !decodeBody(w, r, &req) {
return
}
s.escrowMu.Lock()
s.checkEscrowConfirmTTLLocked()
switch {
case s.escrowClaiming:
s.escrowMu.Unlock()
writeErr(w, http.StatusConflict, "the recovery package is being stored — try again in a moment")
return
case !s.escrowAwaitConfirm:
gone := s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseUnconfirmed
s.escrowMu.Unlock()
if gone {
writeErr(w, http.StatusGone, "the new recovery code was not confirmed in time — the previous code still works; run a new ceremony")
return
}
writeErr(w, http.StatusNotFound, "no recovery code waits for confirmation")
return
case len(req.Words) != 2:
s.escrowMu.Unlock()
writeErr(w, http.StatusBadRequest, "two words are needed")
return
}
match := 1
for i := 0; i < 2; i++ {
typed := []byte(strings.ToLower(strings.TrimSpace(req.Words[i])))
match &= subtle.ConstantTimeCompare(typed, s.escrowConfirmWords[i])
zeroBytes(typed)
}
req.Words = nil
if match != 1 {
s.escrowMu.Unlock()
s.logger.Info("local-api: escrow confirm — the typed words do not match; still waiting (R-1151)", "vmid", vmid)
writeErr(w, http.StatusUnprocessableEntity, "the words do not match the recovery code")
return
}
if s.escrowCeremony.Upload == nil || len(s.escrowPending) == 0 { // the claim refuses to defer without both
s.escrowMu.Unlock()
writeErr(w, http.StatusServiceUnavailable, "the recovery package cannot be stored on this host")
return
}
pkg := s.escrowPending
jobID := ""
if s.escrowJob != nil {
jobID = s.escrowJob.JobID
}
s.escrowClaiming = true
s.escrowMu.Unlock()
start := time.Now()
base := s.baseCtx
if base == nil {
base = context.Background()
}
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg)
cancel()
s.escrowMu.Lock()
s.escrowClaiming = false
if err != nil {
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return
}
s.escrowPending = nil
s.wipeEscrowConfirmLocked()
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
s.escrowJob.Uploaded = true
s.escrowJob.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
}
s.escrowMu.Unlock()
s.logger.Info("local-api: escrow confirm — words typed back; the new package is on the hub (R-1151)",
"vmid", vmid, "job_id", jobID, "duration_ms", time.Since(start).Milliseconds())
writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "uploaded": true})
}
// handleEscrowPreflight is GET /escrow/preflight: the wizard's prerequisite checklist. Each item // handleEscrowPreflight is GET /escrow/preflight: the wizard's prerequisite checklist. Each item
// is {id, ok, detail}. Deliberately NOT checked: the PBS key file itself — /etc/pve/priv is 0700 // is {id, ok, detail}. Deliberately NOT checked: the PBS key file itself — /etc/pve/priv is 0700
// root and the daemon cannot stat it; a missing key fails the ceremony fast with a clear error // root and the daemon cannot stat it; a missing key fails the ceremony fast with a clear error
+76 -7
View File
@@ -2,12 +2,14 @@ package localapi
import ( import (
"context" "context"
"encoding/base64"
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http" "net/http"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"sync"
"sync/atomic" "sync/atomic"
"testing" "testing"
"time" "time"
@@ -19,25 +21,91 @@ import (
const testR = "canary-alpha-bravo-charlie-delta-echo-foxtrot-golf-hotel-india" const testR = "canary-alpha-bravo-charlie-delta-echo-foxtrot-golf-hotel-india"
// cannedCeremonyJSON is shaped exactly like the agent's --output=json object (spike §2.6 values). // cannedCeremonyJSON is shaped exactly like the agent's --output=json object (spike §2.6 values;
// version 2, R-978: the package comes back un-uploaded in pending_upload). The package is a MODEL of
// the R-wrapped blob: it "opens" with exactly the code it was sealed under (fakeEscrowHub.opensWith).
func cannedCeremonyJSON(r string) string { func cannedCeremonyJSON(r string) string {
return fmt.Sprintf(`{"version":1,"recovery_code":%q,"key_fingerprint":"f2:87:68:2a:88:50:16:01","entropy_bits":129,"blob_bytes":383,"identity_blob_bytes":450,"restic_pw_sealed":true,"uploaded":true}`, r) return fmt.Sprintf(`{"version":2,"recovery_code":%q,"key_fingerprint":"f2:87:68:2a:88:50:16:01","entropy_bits":129,"blob_bytes":383,"identity_blob_bytes":450,"restic_pw_sealed":true,"uploaded":false,"pending_upload":%s}`,
r, sealedPackage(r))
}
// sealedPackage is the model hub upload body for a ceremony that minted code r.
func sealedPackage(r string) string {
return fmt.Sprintf(`{"blob_b64":%q,"key_fingerprint":"f2:87:68:2a:88:50:16:01","posture":"zero_knowledge","created_at":"2026-06-10T12:00:00Z"}`,
base64.StdEncoding.EncodeToString([]byte("sealed-under:"+r)))
}
// fakeEscrowHub is the hub's current escrow package for this host (the PUT target). Upload replaces it,
// as the hub does when the sealed secrets are unchanged (the R-978 case: no retained copy).
type fakeEscrowHub struct {
mu sync.Mutex
current []byte
uploads int
err error // non-nil → the upload fails, nothing stored
gate chan struct{} // non-nil → the upload blocks until closed
entered chan struct{} // non-nil → closed when an upload starts
}
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
if f.entered != nil {
close(f.entered)
f.entered = nil
}
if f.gate != nil {
<-f.gate
}
f.mu.Lock()
defer f.mu.Unlock()
if f.err != nil {
return f.err
}
f.current = append([]byte(nil), pkg...)
f.uploads++
return nil
}
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
func (f *fakeEscrowHub) opensWith(r string) bool {
f.mu.Lock()
defer f.mu.Unlock()
var pkg struct {
BlobB64 string `json:"blob_b64"`
}
if json.Unmarshal(f.current, &pkg) != nil {
return false
}
raw, err := base64.StdEncoding.DecodeString(pkg.BlobB64)
return err == nil && string(raw) == "sealed-under:"+r
}
func (f *fakeEscrowHub) count() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.uploads
} }
// newEscrowTestServer builds a server with the ceremony configured and every seam faked. // newEscrowTestServer builds a server with the ceremony configured and every seam faked.
func newEscrowTestServer(t *testing.T, run ceremonyRunner) *Server { func newEscrowTestServer(t *testing.T, run ceremonyRunner) *Server {
t.Helper()
srv, _ := newEscrowTestServerHub(t, run, &fakeEscrowHub{})
return srv
}
// newEscrowTestServerHub is newEscrowTestServer with the hub's escrow record exposed (R-978).
func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub) (*Server, *fakeEscrowHub) {
t.Helper() t.Helper()
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil) srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.escrowCeremony = &EscrowCeremonyConfig{ srv.escrowCeremony = &EscrowCeremonyConfig{
PBSStorageID: "felhom-pbs", PBSStorageID: "felhom-pbs",
HubConfigured: true, HubConfigured: true,
DRConfigured: func() bool { return true }, DRConfigured: func() bool { return true },
Upload: hub.upload,
} }
srv.ceremonyRun = run srv.ceremonyRun = run
srv.escrowSudoCheck = func(context.Context) error { return nil } srv.escrowSudoCheck = func(context.Context) error { return nil }
srv.escrowLookPath = func(string) (string, error) { return "/usr/bin/age", nil } srv.escrowLookPath = func(string) (string, error) { return "/usr/bin/age", nil }
srv.statFile = func(string) bool { return true } srv.statFile = func(string) bool { return true }
return srv return srv, hub
} }
// okRunner returns the canned success output and counts invocations. // okRunner returns the canned success output and counts invocations.
@@ -160,8 +228,8 @@ func TestEscrowCeremony_OneShotClaim(t *testing.T) {
if !strings.Contains(body, `"phase":"done"`) || !strings.Contains(body, `"claimable":true`) { if !strings.Contains(body, `"phase":"done"`) || !strings.Contains(body, `"claimable":true`) {
t.Fatalf("status not done/claimable: %s", body) t.Fatalf("status not done/claimable: %s", body)
} }
if !strings.Contains(body, `"restic_pw_sealed":true`) || !strings.Contains(body, `"uploaded":true`) { if !strings.Contains(body, `"restic_pw_sealed":true`) || !strings.Contains(body, `"uploaded":false`) {
t.Fatalf("summary fields missing: %s", body) t.Fatalf("summary fields missing (R-978: not uploaded before the claim): %s", body)
} }
assertNoR(t, "status payload", body) assertNoR(t, "status payload", body)
@@ -199,7 +267,8 @@ func TestEscrowCeremony_OneShotClaim(t *testing.T) {
// Post-claim status: claimed, not claimable, still phase done. // Post-claim status: claimed, not claimable, still phase done.
st2 := do(t, h, "GET", "/escrow/ceremony/status", "A", "") st2 := do(t, h, "GET", "/escrow/ceremony/status", "A", "")
if !strings.Contains(st2.Body.String(), `"claimed":true`) || strings.Contains(st2.Body.String(), `"claimable":true`) { if !strings.Contains(st2.Body.String(), `"claimed":true`) || strings.Contains(st2.Body.String(), `"claimable":true`) ||
!strings.Contains(st2.Body.String(), `"uploaded":true`) {
t.Fatalf("post-claim status wrong: %s", st2.Body.String()) t.Fatalf("post-claim status wrong: %s", st2.Body.String())
} }
} }
@@ -344,7 +413,7 @@ func TestEscrowCeremony_FailurePaths(t *testing.T) {
}) })
t.Run("wrong version", func(t *testing.T) { t.Run("wrong version", func(t *testing.T) {
srv := newEscrowTestServer(t, func(context.Context) ([]byte, []byte, int, error) { srv := newEscrowTestServer(t, func(context.Context) ([]byte, []byte, int, error) {
return []byte(`{"version":2,"recovery_code":"` + testR + `"}`), nil, 0, nil return []byte(`{"version":1,"recovery_code":"` + testR + `","uploaded":true}`), nil, 0, nil
}) })
h := srv.Handler() h := srv.Handler()
startAndWait(t, srv, h) startAndWait(t, srv, h)
+223
View File
@@ -0,0 +1,223 @@
package localapi
import (
"encoding/json"
"errors"
"net/http"
"strings"
"testing"
"time"
)
// R-1151 — the new recovery code replaces the hub's package only after the household TYPED IT BACK. Before, the
// claim uploaded it (R-978), so a household that opened the new code and closed the page without writing it down
// had already replaced the package their old code opens. These tests ask the hub's current package the household's
// question — „does my written-down code still open it?".
type deferredClaim struct {
code int
r string
positions []int
required bool
body string
}
func claimDeferred(t *testing.T, h http.Handler) deferredClaim {
t.Helper()
w := do(t, h, "POST", "/escrow/ceremony/claim", "A", `{"defer_upload":true}`)
var env struct {
Data struct {
RecoveryCode string `json:"recovery_code"`
ConfirmRequired bool `json:"confirm_required"`
VerifyPositions []int `json:"verify_positions"`
} `json:"data"`
}
_ = json.Unmarshal(w.Body.Bytes(), &env)
return deferredClaim{w.Code, env.Data.RecoveryCode, env.Data.VerifyPositions, env.Data.ConfirmRequired, w.Body.String()}
}
// typedBack is what a household that wrote the code down types for the claim's two positions.
func (c deferredClaim) typedBack(t *testing.T) []string {
t.Helper()
words := strings.Split(c.r, "-")
if len(c.positions) != 2 {
t.Fatalf("the claim named %d positions, want 2: %s", len(c.positions), c.body)
}
return []string{words[c.positions[0]-1], words[c.positions[1]-1]}
}
func confirm(t *testing.T, h http.Handler, words []string) int {
t.Helper()
b, _ := json.Marshal(map[string]any{"words": words})
return do(t, h, "POST", "/escrow/ceremony/confirm", "A", string(b)).Code
}
// THE R-1151 CASE. Ceremony 1 is claimed and typed back (the household's code 1). Ceremony 2 is started and its code
// SHOWN (claimed) — and then the page is closed: the words never come back. Three ways it ends: the confirm window
// runs out, a new start supersedes it, the agent restarts. In every case code 1 must still open the hub's package.
//
// COMPANION RED-PROOF (observed 2026-10-19): ignore the claim's defer_upload (`if false && req.DeferUpload …`, so
// every claim takes the R-978 order and uploads at once) → all three subtests FAIL with "a code that was shown but
// never typed back replaced the package: code 1 no longer opens it (uploads=2)". Restored.
func TestEscrowCeremony_R1151_ShownButNotTypedBackKeepsFirstCode(t *testing.T) {
setup := func(t *testing.T) (*Server, *fakeEscrowHub, http.Handler, *time.Time) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR, "third-code-charlie"), &fakeEscrowHub{})
cur := testNow
srv.now = func() time.Time { return cur }
h := srv.Handler()
startAndWait(t, srv, h)
c1 := claimDeferred(t, h)
if c1.code != http.StatusOK || c1.r != firstR {
t.Fatalf("deferred claim of ceremony 1: %d %s", c1.code, c1.body)
}
if c1.required { // the household types code 1 back
if code := confirm(t, h, c1.typedBack(t)); code != http.StatusOK {
t.Fatalf("typing code 1 back: %d", code)
}
}
if !hub.opensWith(firstR) || hub.count() != 1 {
t.Fatalf("after ceremony 1 the hub must hold code 1's package (uploads=%d)", hub.count())
}
startAndWait(t, srv, h)
if c2 := claimDeferred(t, h); c2.code != http.StatusOK || c2.r != secondR {
t.Fatalf("deferred claim of ceremony 2: %d %s", c2.code, c2.body)
} // …shown, and the page is closed
return srv, hub, h, &cur
}
check := func(t *testing.T, hub *fakeEscrowHub) {
t.Helper()
if !hub.opensWith(firstR) {
t.Fatalf("a code that was shown but never typed back replaced the package: code 1 no longer opens it (uploads=%d)", hub.count())
}
if hub.opensWith(secondR) || hub.count() != 1 {
t.Fatalf("a package for the never-confirmed code 2 reached the hub (uploads=%d)", hub.count())
}
}
t.Run("the confirm window runs out", func(t *testing.T) {
_, hub, h, cur := setup(t)
*cur = cur.Add(escrowConfirmTTL + time.Minute)
check(t, hub)
st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String()
if !strings.Contains(st, `"phase":"unconfirmed_void"`) || !strings.Contains(st, `"awaiting_confirm":false`) {
t.Fatalf("after the window the job must read unconfirmed_void: %s", st)
}
if code := confirm(t, h, []string{"second", "bravo"}); code != http.StatusGone {
t.Fatalf("a confirm after the window: got %d, want 410", code)
}
})
t.Run("a new start supersedes it", func(t *testing.T) {
srv, hub, h, _ := setup(t)
startAndWait(t, srv, h)
if code := confirm(t, h, []string{"second", "bravo"}); code != http.StatusNotFound {
t.Fatalf("a confirm for the superseded code: got %d, want 404", code)
}
check(t, hub)
})
t.Run("the agent restarts", func(t *testing.T) {
_, hub, _, _ := setup(t)
srv2, _ := newEscrowTestServerHub(t, sequenceRunner("unused"), hub)
if code := confirm(t, srv2.Handler(), []string{"second", "bravo"}); code != http.StatusNotFound {
t.Fatalf("a confirm after a restart: got %d, want 404", code)
}
check(t, hub)
})
}
// The positive control and the typing: wrong words store nothing and the wait goes on; the right words (any case,
// spaces around) store the package, once.
//
// COMPANION RED-PROOF (observed 2026-10-19): skip the word check (`if match != 1 && false`) → this fails with "one
// wrong word: got 200 (uploads=1), want 422 and nothing stored". Restored.
func TestEscrowCeremony_R1151_TypedBackStoresThePackage(t *testing.T) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(secondR), &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if c.code != http.StatusOK || !c.required || hub.count() != 0 {
t.Fatalf("a deferred claim must hand out the code and store nothing yet: %d required=%v uploads=%d", c.code, c.required, hub.count())
}
right := c.typedBack(t)
if code := confirm(t, h, []string{right[0], "wrong"}); code != http.StatusUnprocessableEntity || hub.count() != 0 {
t.Fatalf("one wrong word: got %d (uploads=%d), want 422 and nothing stored", code, hub.count())
}
if code := confirm(t, h, []string{right[0]}); code != http.StatusBadRequest {
t.Fatalf("one word only: got %d, want 400", code)
}
st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String()
if !strings.Contains(st, `"awaiting_confirm":true`) || !strings.Contains(st, `"uploaded":false`) {
t.Fatalf("a wrong try must leave the wait open: %s", st)
}
assertNoRIn(t, "status while awaiting the words", st, secondR)
for _, w := range right {
assertNoRIn(t, "status while awaiting the words", st, `"`+w+`"`)
}
if code := confirm(t, h, []string{" " + strings.ToUpper(right[0]), right[1] + " "}); code != http.StatusOK {
t.Fatalf("the right words: got %d, want 200", code)
}
if !hub.opensWith(secondR) || hub.count() != 1 {
t.Fatalf("after the words came back the hub must hold code 2's package (uploads=%d)", hub.count())
}
if code := confirm(t, h, right); code != http.StatusNotFound || hub.count() != 1 {
t.Fatalf("a second confirm: got %d (uploads=%d), want 404 and no second upload", code, hub.count())
}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
t.Fatalf("status after the confirm: %s", st)
}
}
// A controller that has not updated (0.331.0 and older) claims with `{}` and never confirms: it keeps the R-978
// order — the package is stored at the claim — so its household's ceremony still completes.
func TestEscrowCeremony_R1151_OldControllerKeepsTheClaimOrder(t *testing.T) {
for _, body := range []string{"", "{}"} {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR), &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
w := do(t, h, "POST", "/escrow/ceremony/claim", "A", body)
if w.Code != http.StatusOK || !strings.Contains(w.Body.String(), firstR) || !strings.Contains(w.Body.String(), `"confirm_required":false`) {
t.Fatalf("old-shape claim %q: %d %s", body, w.Code, w.Body.String())
}
if !hub.opensWith(firstR) || hub.count() != 1 {
t.Fatalf("old-shape claim %q must store the package at once (uploads=%d)", body, hub.count())
}
if code := confirm(t, h, []string{"first", "alpha"}); code != http.StatusNotFound {
t.Fatalf("nothing waits after an old-shape claim: got %d, want 404", code)
}
}
}
// The hub failing at the confirm keeps the wait open and the previous package current; a retry succeeds.
func TestEscrowCeremony_R1151_UploadFailureAtConfirmKeepsWaiting(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if code := confirm(t, h, c.typedBack(t)); code != http.StatusBadGateway {
t.Fatalf("confirm with the hub failing: got %d, want 502", code)
}
hub.mu.Lock()
hub.err = nil
hub.mu.Unlock()
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
t.Fatalf("retry after the hub recovered: %d (opens: %v)", code, hub.opensWith(secondR))
}
}
// The two positions are distinct, in range, smaller first — over many draws every position appears.
func TestPickConfirmPositions(t *testing.T) {
seen := map[int]bool{}
for i := 0; i < 2000; i++ {
p, err := pickConfirmPositions(10)
if err != nil {
t.Fatal(err)
}
if p[0] < 1 || p[1] > 10 || p[0] >= p[1] {
t.Fatalf("bad positions %v", p)
}
seen[p[0]], seen[p[1]] = true, true
}
if len(seen) != 10 {
t.Fatalf("only positions %v were ever drawn", seen)
}
}
+219
View File
@@ -0,0 +1,219 @@
package localapi
import (
"context"
"errors"
"net/http"
"strings"
"testing"
"time"
)
// R-978 — starting the escrow ceremony again must not make the customer's written-down recovery code
// useless. The hub overwrites the current package when the sealed secrets are unchanged, so the NEW
// package may reach the hub only when its code is claimed. These tests ask the customer's question of
// the hub's current package — „does my code still open it?" — not whether some function ran.
const (
firstR = "first-ceremony-code-alpha"
secondR = "second-ceremony-code-bravo"
)
// sequenceRunner mints the given codes in order, one per ceremony.
func sequenceRunner(codes ...string) ceremonyRunner {
i := 0
return func(context.Context) ([]byte, []byte, int, error) {
r := codes[i]
if i < len(codes)-1 {
i++
}
return []byte(cannedCeremonyJSON(r)), nil, 0, nil
}
}
func claimCode(t *testing.T, h http.Handler) (int, string) {
t.Helper()
w := do(t, h, "POST", "/escrow/ceremony/claim", "A", "")
return w.Code, w.Body.String()
}
// THE R-978 CASE. Ceremony 1 is claimed (the customer writes code 1 down). Ceremony 2 is started and
// abandoned — three ways: its code expires unclaimed, a third start supersedes it, the agent restarts.
// In every case the hub's current package must still open with code 1, and nothing was uploaded for 2.
//
// COMPANION RED-PROOF (observed 2026-10-18): restore the pre-R-978 shape — the package uploaded when the
// ceremony finishes (s.escrowCeremony.Upload(base, out.PendingUpload) in the job goroutine, as the root
// subprocess did with `--upload`) and nothing left for the claim (finishEscrowCeremony(job, R, nil)) →
// all three subtests FAIL with "the abandoned second ceremony replaced the package: code 1 no longer
// opens it (uploads=2)" (uploads=3 for the superseding start). Restored.
func TestEscrowCeremony_R978_AbandonedSecondCeremonyKeepsFirstCode(t *testing.T) {
setup := func(t *testing.T) (*Server, *fakeEscrowHub, http.Handler, *time.Time) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR, "third-code-charlie"), &fakeEscrowHub{})
cur := testNow
srv.now = func() time.Time { return cur }
h := srv.Handler()
startAndWait(t, srv, h)
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("claim of ceremony 1: %d %s", code, body)
}
if !hub.opensWith(firstR) || hub.count() != 1 {
t.Fatalf("after the first claim the hub must hold code 1's package (uploads=%d)", hub.count())
}
startAndWait(t, srv, h) // ceremony 2 — about to be abandoned
return srv, hub, h, &cur
}
check := func(t *testing.T, hub *fakeEscrowHub) {
t.Helper()
if !hub.opensWith(firstR) {
t.Fatalf("the abandoned second ceremony replaced the package: code 1 no longer opens it (uploads=%d)", hub.count())
}
if hub.opensWith(secondR) || hub.count() != 1 {
t.Fatalf("a package for the never-claimed code 2 reached the hub (uploads=%d)", hub.count())
}
}
t.Run("code 2 expires unclaimed", func(t *testing.T) {
_, hub, h, cur := setup(t)
*cur = cur.Add(escrowClaimTTL + time.Minute)
if code, _ := claimCode(t, h); code != http.StatusGone {
t.Fatalf("claim after TTL: got %d, want 410", code)
}
check(t, hub)
})
t.Run("a third start supersedes code 2", func(t *testing.T) {
srv, hub, h, _ := setup(t)
startAndWait(t, srv, h) // ceremony 3, also left unclaimed
check(t, hub)
})
t.Run("the agent restarts before code 2 is claimed", func(t *testing.T) {
_, hub, _, _ := setup(t)
// A restart is a new process with an empty slot; only the hub's record survives it.
srv2, _ := newEscrowTestServerHub(t, sequenceRunner("unused"), hub)
if code, _ := claimCode(t, srv2.Handler()); code != http.StatusNotFound {
t.Fatalf("claim after a restart: got %d, want 404", code)
}
check(t, hub)
})
}
// The positive control: CLAIMING code 2 is what replaces the package — the code handed out opens the
// package the hub now holds, and the claim answers only after the hub took it.
func TestEscrowCeremony_R978_ClaimStoresThePackageBeforeHandingOutTheCode(t *testing.T) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR), &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
if hub.count() != 0 {
t.Fatalf("the package was uploaded before any claim (uploads=%d)", hub.count())
}
claimCode(t, h)
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusOK || !strings.Contains(body, secondR) {
t.Fatalf("claim of ceremony 2: %d %s", code, body)
}
if !hub.opensWith(secondR) || hub.count() != 2 {
t.Fatalf("after claiming code 2 the hub must hold its package (uploads=%d, opens with 2: %v)", hub.count(), hub.opensWith(secondR))
}
}
// A failed upload hands out NO code, keeps the code claimable and the previous package current; a retry
// that reaches the hub succeeds.
func TestEscrowCeremony_R978_UploadFailureKeepsTheCodeClaimable(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusBadGateway {
t.Fatalf("claim with the hub failing: got %d, want 502 (%s)", code, body)
}
assertNoRIn(t, "failed-upload claim payload", body, firstR)
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) || !strings.Contains(st, `"uploaded":false`) {
t.Fatalf("after a failed upload the code must stay claimable and un-uploaded: %s", st)
}
hub.mu.Lock()
hub.err = nil
hub.mu.Unlock()
code, body = claimCode(t, h)
if code != http.StatusOK || !strings.Contains(body, firstR) || !hub.opensWith(firstR) {
t.Fatalf("retry after the hub recovered: %d %s (opens: %v)", code, body, hub.opensWith(firstR))
}
}
// While the claim's upload is in flight, neither the TTL nor a new start may take the code away — else
// the hub would hold a package whose code nobody was given (the R-978 failure, reached the other way).
func TestEscrowCeremony_R978_TTLAndRestartWaitForAnInFlightClaim(t *testing.T) {
hub := &fakeEscrowHub{gate: make(chan struct{}), entered: make(chan struct{})}
entered := hub.entered
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR), hub)
var mu = make(chan struct{}, 1)
mu <- struct{}{}
cur := testNow
srv.now = func() time.Time { <-mu; defer func() { mu <- struct{}{} }(); return cur }
h := srv.Handler()
startAndWait(t, srv, h)
type res struct {
code int
body string
}
done := make(chan res, 1)
go func() { c, b := claimCode(t, h); done <- res{c, b} }()
<-entered
<-mu
cur = cur.Add(escrowClaimTTL + time.Minute) // the TTL passes mid-upload
mu <- struct{}{}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); strings.Contains(st, `"phase":"unclaimed_void"`) {
t.Fatalf("the TTL voided the code while its package was uploading: %s", st)
}
if w := do(t, h, "POST", "/escrow/ceremony", "A", ""); w.Code != http.StatusConflict {
t.Fatalf("a new start during the claim's upload: got %d, want 409", w.Code)
}
if code, _ := claimCode(t, h); code != http.StatusConflict {
t.Fatalf("a second claim during the upload: got %d, want 409", code)
}
close(hub.gate)
r := <-done
if r.code != http.StatusOK || !strings.Contains(r.body, firstR) || !hub.opensWith(firstR) {
t.Fatalf("the in-flight claim must deliver the code whose package it stored: %d %s", r.code, r.body)
}
}
// No upload configured → the claim refuses rather than hand out a code for a package the hub lacks.
func TestEscrowCeremony_R978_NoUploaderRefusesTheClaim(t *testing.T) {
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), &fakeEscrowHub{})
srv.escrowCeremony.Upload = nil
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusServiceUnavailable {
t.Fatalf("claim with no uploader: got %d, want 503 (%s)", code, body)
}
assertNoRIn(t, "no-uploader claim payload", body, firstR)
}
// An output that uploaded by itself (the pre-R-978 shape) or carries no package fails the ceremony.
func TestEscrowCeremony_R978_OutputWithoutPendingPackageFails(t *testing.T) {
srv, _ := newEscrowTestServerHub(t, func(context.Context) ([]byte, []byte, int, error) {
return []byte(`{"version":2,"recovery_code":"` + firstR + `","uploaded":true}`), nil, 0, nil
}, &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String()
if !strings.Contains(st, `"phase":"failed"`) || !strings.Contains(st, "no pending package") {
t.Fatalf("want failed: %s", st)
}
if code, _ := claimCode(t, h); code != http.StatusConflict {
t.Fatalf("claim on a failed ceremony: got %d, want 409", code)
}
}
func assertNoRIn(t *testing.T, where, body, r string) {
t.Helper()
if strings.Contains(body, r) {
t.Fatalf("%s carries the recovery code", where)
}
}
+12 -1
View File
@@ -57,6 +57,10 @@ type MemoryInfo struct {
MaxMB int64 `json:"max_mb"` MaxMB int64 `json:"max_mb"`
FloorMB int64 `json:"floor_mb"` FloorMB int64 `json:"floor_mb"`
Running bool `json:"running"` Running bool `json:"running"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up): the host's swap and this guest's swap allowance, so the controller
// can tell the household when the host has too little swap behind the guest (its install page). nil = not read.
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
} }
// memoryResizeRequest is the POST /guest/memory body. // memoryResizeRequest is the POST /guest/memory body.
@@ -94,6 +98,8 @@ type memoryBounds struct {
maxMB int64 maxMB int64
floorMB int64 floorMB int64
running bool running bool
hostSwapMB int64
guestSwapMB int64
} }
func (b memoryBounds) refusal(code string) memoryRefusal { func (b memoryBounds) refusal(code string) memoryRefusal {
@@ -104,7 +110,9 @@ func (b memoryBounds) refusal(code string) memoryRefusal {
} }
func (b memoryBounds) info(vmid int) MemoryInfo { func (b memoryBounds) info(vmid int) MemoryInfo {
hostSwap, guestSwap := b.hostSwapMB, b.guestSwapMB
return MemoryInfo{ return MemoryInfo{
HostSwapMB: &hostSwap, GuestSwapMB: &guestSwap,
VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB, VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB,
MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running, MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running,
} }
@@ -139,6 +147,8 @@ func (s *Server) readMemoryBounds(ctx context.Context, vmid int) (memoryBounds,
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max) hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
minMB: minGuestMemoryMB, minMB: minGuestMemoryMB,
running: st.Status == "running", running: st.Status == "running",
hostSwapMB: node.Swap.Total / mib, // R-1192 follow-up: the kernel's SwapTotal, as Proxmox reports it
guestSwapMB: cfg.Swap, // PVE config swap is MB
} }
b.maxMB = b.hostTotalMB - hostReserveMB b.maxMB = b.hostTotalMB - hostReserveMB
b.floorMB = b.usageMB + shrinkUsageMarginMB b.floorMB = b.usageMB + shrinkUsageMarginMB
@@ -225,7 +235,8 @@ func (s *Server) handleGuestMemoryResize(w http.ResponseWriter, r *http.Request,
} }
// Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on. // Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target)}) // R-1192: the swap moves with the memory (GuestSwapMB) — a grown guest with the old swap cap thrashes the same way.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target), "swap": fmt.Sprintf("%d", GuestSwapMB(target))})
if err != nil { if err != nil {
s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err) s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err)
writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error()) writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error())
+8 -1
View File
@@ -25,12 +25,14 @@ type fakeMemory struct {
setUPID string setUPID string
setErr error setErr error
applyReflect bool applyReflect bool
hostSwapB int64 // R-1192 follow-up: node swap total
guestSwapMB int64
} }
func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) { func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
return proxmox.GuestConfig{Memory: f.allocMB}, nil return proxmox.GuestConfig{Memory: f.allocMB, Swap: f.guestSwapMB}, nil
} }
func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) { func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) {
f.mu.Lock() f.mu.Lock()
@@ -42,6 +44,7 @@ func (f *fakeMemory) NodeStatus(_ context.Context) (proxmox.NodeStatus, error) {
defer f.mu.Unlock() defer f.mu.Unlock()
var ns proxmox.NodeStatus var ns proxmox.NodeStatus
ns.Memory.Total = f.hostTotalB ns.Memory.Total = f.hostTotalB
ns.Swap.Total = f.hostSwapB
return ns, nil return ns, nil
} }
func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) { func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) {
@@ -151,6 +154,10 @@ func TestGuestMemory_GrowAndShrink(t *testing.T) {
if f.lastParams["memory"] != "12288" { if f.lastParams["memory"] != "12288" {
t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"]) t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"])
} }
// R-1192: the swap moves with the memory.
if f.lastParams["swap"] != "12288" {
t.Errorf("SetConfig swap = %q, want 12288 (swap = RAM)", f.lastParams["swap"])
}
}) })
t.Run("shrink above floor", func(t *testing.T) { t.Run("shrink above floor", func(t *testing.T) {
f := stdFixture() // floor = max(2048, 3000+512) = 3512 f := stdFixture() // floor = max(2048, 3000+512) = 3512
+73
View File
@@ -0,0 +1,73 @@
package localapi
import (
"context"
"fmt"
"log/slog"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// ── R-1192 (2026-10-21): the guest may swap as much as its RAM ────────────────────────────────────────────────────
//
// MEASURED (felhom.eu audits/fixes-2026-10-21/P4-freeze/): on the minimum machine (8 GB host, guest 4096 MB) the
// household's first uploads made Immich load its analysis models (+1.15 GB) on top of 8 apps' 1.75 GB; the guest hit
// its 4096 MB limit with its swap capped at Proxmox's LXC default of 512 MB — and full. The kernel could then evict
// only file pages: ~360 MB was left for the program code of ~25 containers, re-read from disk without end (150 000–
// 600 000 refaults and ~1 GB of reads per 10 s; memory „full" pressure 92–95 %, load 155, the dashboard dead, 0 OOM
// kills). Raising the cap to 4096 MB live ended it in under a minute (memory „full" 96 % → 1.7 %, dashboard back
// 42 s later) while the uploads ran on. The host had 7.4 GB of swap free the whole time.
//
// The agent never set `swap` at all, so every guest had the 512 MB default. Now:
// - a new guest is created with swap = its memory (reconcile.bringup);
// - a RAM resize sets swap with the memory (handleGuestMemoryResize);
// - at agent start, every LXC in the felhom pool whose swap is below its memory is raised to it (this file).
// Raising swap is a live cgroup change (memory.swap.max), no restart. It never lowers a swap someone set higher.
// GuestSwapMB is the swap a guest of memoryMB gets: the same as its RAM.
func GuestSwapMB(memoryMB int64) int64 { return memoryMB }
// SwapOps is the Proxmox surface EnsureGuestSwap needs (satisfied by *proxmox.Client).
type SwapOps interface {
Pool(ctx context.Context, name string) (proxmox.PoolInfo, error)
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
SetConfig(ctx context.Context, vmid int, params map[string]string) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
}
// EnsureGuestSwap raises the swap of every LXC guest in pool to GuestSwapMB(memory) when it is below. Returns how
// many guests were raised. One guest's failure is logged and the others are still handled.
func EnsureGuestSwap(ctx context.Context, ops SwapOps, pool string, logger *slog.Logger) (int, error) {
p, err := ops.Pool(ctx, pool)
if err != nil {
return 0, fmt.Errorf("guest swap: read pool %s: %w", pool, err)
}
raised := 0
for _, m := range p.Members {
if m.Type != "lxc" || m.VMID <= 0 {
continue
}
cfg, err := ops.GuestConfig(ctx, m.VMID)
if err != nil {
logger.Warn("guest-swap: config read failed (left as it is)", "vmid", m.VMID, "err", err)
continue
}
want := GuestSwapMB(cfg.Memory)
if cfg.Memory <= 0 || cfg.Swap >= want {
logger.Debug("guest-swap: nothing to do", "vmid", m.VMID, "memory_mb", cfg.Memory, "swap_mb", cfg.Swap)
continue
}
upid, err := ops.SetConfig(ctx, m.VMID, map[string]string{"swap": fmt.Sprintf("%d", want)})
if err == nil && upid != "" {
_, err = ops.WaitTask(ctx, upid, proxmox.WaitOptions{})
}
if err != nil {
logger.Warn("guest-swap: raise failed (left as it is)", "vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want, "err", err)
continue
}
raised++
logger.Info("guest-swap: raised (R-1192 — a guest swap cap below its RAM turned a memory burst into a freeze)",
"vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want)
}
return raised, nil
}
+71
View File
@@ -0,0 +1,71 @@
package localapi
import (
"context"
"errors"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: the freeze measured on the minimum machine (felhom.eu audits/fixes-2026-10-21/P4-freeze/) — a guest whose
// swap is capped below its RAM evicts program code instead of idle memory. EnsureGuestSwap raises every pool guest's
// swap to its RAM, never lowers one, skips non-LXC members, and survives one guest's failure.
// COMPANION RED-PROOF: return 0 before the SetConfig call → „raised 0, want 2".
type fakeSwapOps struct {
cfg map[int]proxmox.GuestConfig
failSet map[int]bool
set map[int]map[string]string
}
func (f *fakeSwapOps) Pool(context.Context, string) (proxmox.PoolInfo, error) {
return proxmox.PoolInfo{Members: []proxmox.PoolMember{{VMID: 9201, Type: "lxc"}, {VMID: 9202, Type: "lxc"}, {VMID: 300, Type: "qemu"},
{Type: "storage"}, {VMID: 9203, Type: "lxc"}, {VMID: 9204, Type: "lxc"}}}, nil
}
func (f *fakeSwapOps) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
c, ok := f.cfg[vmid]
if !ok {
return c, errors.New("no such guest")
}
return c, nil
}
func (f *fakeSwapOps) SetConfig(_ context.Context, vmid int, p map[string]string) (string, error) {
if f.failSet[vmid] {
return "", errors.New("pve said no")
}
if f.set == nil {
f.set = map[int]map[string]string{}
}
f.set[vmid] = p
return "", nil
}
func (f *fakeSwapOps) WaitTask(context.Context, string, proxmox.WaitOptions) (proxmox.TaskStatus, error) {
return proxmox.TaskStatus{}, nil
}
func TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM(t *testing.T) {
f := &fakeSwapOps{cfg: map[int]proxmox.GuestConfig{
9201: {Memory: 4096, Swap: 512}, // the measured box: raised
9202: {Memory: 2048, Swap: 4096}, // already higher: never lowered
9203: {Memory: 6144, Swap: 0}, // none: raised
9204: {Memory: 4096, Swap: 512}, // the write fails: logged, the others still done
}, failSet: map[int]bool{9204: true}}
n, err := EnsureGuestSwap(context.Background(), f, "felhom", slog.New(slog.NewTextHandler(io.Discard, nil)))
if err != nil {
t.Fatal(err)
}
if n != 2 {
t.Fatalf("raised %d, want 2 (9201, 9203)", n)
}
if f.set[9201]["swap"] != "4096" || f.set[9203]["swap"] != "6144" {
t.Fatalf("writes = %v", f.set)
}
if _, touched := f.set[9202]; touched {
t.Fatal("a swap above the RAM was lowered")
}
if len(f.set[9201]) != 1 {
t.Fatalf("the swap raise wrote more than the swap: %v", f.set[9201])
}
}
+39
View File
@@ -0,0 +1,39 @@
package localapi
import (
"encoding/json"
"testing"
)
// R-1192 follow-up (night 2026-10-21): GET /guest/memory carries the host's swap and this guest's swap allowance, so
// the controller's install page can say when the host has no swap behind the guest. 0 is a real answer (no swap);
// absent means an older agent.
// RED-PROOF: without the two MemoryInfo fields the keys are missing → fails on „host_swap_mb absent".
func TestR1192_GuestMemoryCarriesHostAndGuestSwap(t *testing.T) {
for _, c := range []struct {
name string
hostB, guestMB int64
wantHost, wantG int64
}{
{"installer host (7.75 GB swap), guest swap = RAM", 7935 * mib, 8192, 7935, 8192},
{"a host with NO swap", 0, 8192, 0, 8192},
} {
f := stdFixture()
f.hostSwapB, f.guestSwapMB = c.hostB, c.guestMB
w := do(t, memServer(t, f).Handler(), "GET", "/guest/memory", "A", "")
var m struct {
Data map[string]any `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &m); err != nil {
t.Fatal(err)
}
hs, ok := m.Data["host_swap_mb"].(float64)
if !ok {
t.Fatalf("%s: host_swap_mb absent: %s", c.name, w.Body.String())
}
gs, _ := m.Data["guest_swap_mb"].(float64)
if int64(hs) != c.wantHost || int64(gs) != c.wantG {
t.Fatalf("%s: host %v guest %v, want %d / %d", c.name, hs, gs, c.wantHost, c.wantG)
}
}
}
+47
View File
@@ -90,6 +90,10 @@ type BackupStore interface {
type LastKnownBackupStore interface { type LastKnownBackupStore interface {
RecordBackupSuccess(target string, b hub.Backup) error RecordBackupSuccess(target string, b hub.Backup) error
LastKnownSuccess(target string, vmid int) (time.Time, bool) LastKnownSuccess(target string, vmid int) (time.Time, bool)
// R-973: the cadence cap's restart-proof history and its refusals.
RecentBackupSuccesses(target string, vmid int, since time.Time) []time.Time
RecordBackupRefusal(target string, vmid int, at time.Time) error
LastBackupRefusal() (target string, vmid int, at time.Time, ok bool)
} }
// StorageView yields the host's observed storage targets (for mapping a mount's storage id → // StorageView yields the host's observed storage targets (for mapping a mount's storage id →
@@ -302,6 +306,11 @@ type Server struct {
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
// R-973 cadence cap: this process's successful runs per tier+guest, and the newest refusal.
capMu sync.Mutex
capRuns map[backupJobKey][]time.Time
capRefusal capRefusal
disks DiskOps // slice 8C (optional) disks DiskOps // slice 8C (optional)
diskGate StorageGate // slice 8C (optional) diskGate StorageGate // slice 8C (optional)
guestList GuestLister // slice 8C (optional) guestList GuestLister // slice 8C (optional)
@@ -426,8 +435,17 @@ type Server struct {
escrowMu sync.Mutex escrowMu sync.Mutex
escrowJob *escrowCeremonyJob escrowJob *escrowCeremonyJob
escrowR []byte escrowR []byte
escrowPending []byte // R-978: the ceremony's hub upload body, sent by the claim before R is handed out
escrowClaiming bool // R-978: a claim is uploading escrowPending (TTL and a new start wait)
escrowRClaimed bool escrowRClaimed bool
escrowRExpiry time.Time escrowRExpiry time.Time
// R-1151: after a claim that asked to wait for the typed-back words (defer_upload), the package stays in
// escrowPending until POST /escrow/ceremony/confirm brings the two words at escrowConfirmPos back. Only those two
// words are kept (never R), zeroed on confirm, supersede and expiry.
escrowAwaitConfirm bool
escrowConfirmWords [2][]byte
escrowConfirmPos [2]int
escrowConfirmExpiry time.Time
escrowDone <-chan struct{} // closes when the detached job finishes (tests wait on it) escrowDone <-chan struct{} // closes when the detached job finishes (tests wait on it)
// ceremonyRun executes the fixed-argv sudo self-invocation (tests inject canned JSON). // ceremonyRun executes the fixed-argv sudo self-invocation (tests inject canned JSON).
ceremonyRun ceremonyRunner ceremonyRun ceremonyRunner
@@ -603,6 +621,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /escrow/ceremony", s.withGuest(s.handleEscrowCeremonyStart)) mux.HandleFunc("POST /escrow/ceremony", s.withGuest(s.handleEscrowCeremonyStart))
mux.HandleFunc("GET /escrow/ceremony/status", s.withGuest(s.handleEscrowCeremonyStatus)) mux.HandleFunc("GET /escrow/ceremony/status", s.withGuest(s.handleEscrowCeremonyStatus))
mux.HandleFunc("POST /escrow/ceremony/claim", s.withGuest(s.handleEscrowCeremonyClaim)) mux.HandleFunc("POST /escrow/ceremony/claim", s.withGuest(s.handleEscrowCeremonyClaim))
// R-1151: the household typed two words of the new code back — only now does its package reach the hub.
mux.HandleFunc("POST /escrow/ceremony/confirm", s.withGuest(s.handleEscrowCeremonyConfirm))
// v0.83.0 observability: the agent's always-DEBUG capture ring, for the controller's // v0.83.0 observability: the agent's always-DEBUG capture ring, for the controller's
// Debug page agent tab (same auth/self-scoping wrap as every sibling route). // Debug page agent tab (same auth/self-scoping wrap as every sibling route).
@@ -878,6 +898,12 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
"a backup is already in flight on target "+busyTarget+" (job "+busyJob+") — only one backup runs at a time per guest") "a backup is already in flight on target "+busyTarget+" (job "+busyJob+") — only one backup runs at a time per guest")
return return
} }
// R-973: the cadence cap (backup_cap.go). Checked under jobsMu, after the in-flight answers, so two
// requests cannot both pass it (only one run per guest is ever in flight).
if s.capRefuses(w, key, tier) {
s.jobsMu.Unlock()
return
}
// Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers // Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers
// started inside the same nanosecond (the weekly both-due night, or any injected clock) would // started inside the same nanosecond (the weekly both-due night, or any injected clock) would
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the // otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
@@ -941,6 +967,9 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive) s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive)
} }
s.store.RecordBackup(b) s.store.RecordBackup(b)
if b.Success {
s.recordCapRun(key, b) // R-973
}
if b.Success && s.lastKnown != nil { if b.Success && s.lastKnown != nil {
if err := s.lastKnown.RecordBackupSuccess(tier.TargetID, b); err != nil { if err := s.lastKnown.RecordBackupSuccess(tier.TargetID, b); err != nil {
// Not fatal: the backup exists. Only the fallback after a restart loses this copy. // Not fatal: the backup exists. Only the fallback after a restart loses this copy.
@@ -1314,6 +1343,20 @@ type BackupStatusResponse struct {
// presence. Served on the UNTARGETED request only; additive, so an older controller reads the // presence. Served on the UNTARGETED request only; additive, so an older controller reads the
// response exactly as before. // response exactly as before.
Tiers []TierBackupState `json:"tiers,omitempty"` Tiers []TierBackupState `json:"tiers,omitempty"`
// HostBusy (R-921, agent Unreleased 2026-10-19) says whether the host-wide heavy-operation gate is held RIGHT NOW
// (another guest's backup, the restore test, a night OS step, the stale-lock sweep, a power-off) — a POST /backup
// would then be refused with 409. The controller reads it BEFORE it stops the household's apps, so a refused tier
// costs no stop. Served on every request; ABSENT (nil) = an older agent, or the gate is not wired — the controller
// then keeps its old order (stop, ask, resume at once on 409). A read, never a reservation: the gate can be taken
// between this answer and the POST, and that race keeps the old order. Pinned by TestBackupStatus_R921_HostBusy.
HostBusy *HostBusyState `json:"host_busy,omitempty"`
}
// HostBusyState is GET /backup/status's host_busy (R-921). What names the holder ("backup:local", "restore-test",
// "os-pve-step" …), empty when Busy is false.
type HostBusyState struct {
Busy bool `json:"busy"`
What string `json:"what,omitempty"`
} }
func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) { func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) {
@@ -1333,6 +1376,10 @@ func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid
resp.JobID = job.JobID resp.JobID = job.JobID
resp.Error = job.Error resp.Error = job.Error
} }
if s.inFlight != nil {
what := s.inFlight.Busy()
resp.HostBusy = &HostBusyState{Busy: what != "", What: what}
}
writeOK(w, resp) writeOK(w, resp)
} }
+45
View File
@@ -13,6 +13,7 @@ import (
"testing" "testing"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub" "gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
) )
@@ -514,3 +515,47 @@ func TestBackupStatus_FiltersToThisGuest(t *testing.T) {
t.Fatalf("status not scoped to this guest / not latest: %s", w.Body.String()) t.Fatalf("status not scoped to this guest / not latest: %s", w.Body.String())
} }
} }
// R-921: GET /backup/status says when the host-wide heavy-operation gate is held, so the controller can skip the app
// stop for a tier the POST would refuse. Free → busy false; held → busy true with the holder's name; the gate not wired
// → no field at all (the controller's „older agent" reading, never „free").
//
// COMPANION RED-PROOF (observed 2026-10-19): with the `resp.HostBusy = …` assignment removed, the held case fails with
//
// server_test.go: free gate: host_busy = <nil>, want busy=false
//
// Restored.
func TestBackupStatus_R921_HostBusy(t *testing.T) {
read := func(h http.Handler) *HostBusyState {
w := do(t, h, "GET", "/backup/status", "A", "")
var resp struct {
Data BackupStatusResponse `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("status body: %v (%s)", err, w.Body.String())
}
return resp.Data.HostBusy
}
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
if got := read(srv.Handler()); got != nil {
t.Fatalf("gate not wired: host_busy = %+v, want absent", got)
}
gate := &backup.InFlight{}
srv.inFlight = gate
h := srv.Handler()
if got := read(h); got == nil || got.Busy || got.What != "" {
t.Fatalf("free gate: host_busy = %+v, want busy=false", got)
}
release, _, ok := gate.TryAcquire("os-pve-step")
if !ok {
t.Fatal("could not take the gate")
}
if got := read(h); got == nil || !got.Busy || got.What != "os-pve-step" {
t.Fatalf("held gate: host_busy = %+v, want busy=true what=\"os-pve-step\"", got)
}
// The read never takes the gate: a POST after the holder lets go is accepted.
release()
if got := read(h); got == nil || got.Busy {
t.Fatalf("released gate: host_busy = %+v, want busy=false", got)
}
}
+46 -1
View File
@@ -2,6 +2,7 @@ package localapi
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"log/slog" "log/slog"
"os" "os"
@@ -160,6 +161,7 @@ type staleLockAPI interface {
ListSnapshots(ctx context.Context, vmid int) ([]proxmox.Snapshot, error) ListSnapshots(ctx context.Context, vmid int) ([]proxmox.Snapshot, error)
ListRunningTasks(ctx context.Context) ([]proxmox.TaskStatus, error) ListRunningTasks(ctx context.Context) ([]proxmox.TaskStatus, error)
DeleteSnapshot(ctx context.Context, vmid int, snapname string) (string, error) DeleteSnapshot(ctx context.Context, vmid int, snapname string) (string, error)
DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error)
Start(ctx context.Context, vmid int) (string, error) Start(ctx context.Context, vmid int) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error) WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
} }
@@ -334,8 +336,34 @@ func (c *staleLockController) Unlock(ctx context.Context, vmid int) error {
return nil return nil
} }
// DeleteVzdumpSnapshot removes the dangling `vzdump` snapshot. R-1010: when the delete fails ONLY because one disk's
// snapshot volume is already gone („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump" — a backup that ran
// out of space, 2026-10-14), Proxmox stops at that volume and keeps the entry and the other volumes' snapshots, and a
// plain retry fails the same way every sweep, forever. Then it retries ONCE with Proxmox's force (the hand fix that
// day was `pct delsnapshot 9201 vzdump --force`): the remaining volumes' snapshots are removed and the entry goes. Any
// other error is returned as it is — never a force on an error that might mean a volume still holds data (a full
// LVM metadata area, a busy volume). Pinned by TestStaleLock_R1010_*.
func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int) error { func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int) error {
upid, err := c.px.DeleteSnapshot(ctx, vmid, vzdumpSnapshotName) err := c.deleteSnap(ctx, vmid, false)
if err == nil || !snapshotVolumeAlreadyGone(err) {
return err
}
if c.logger != nil {
c.logger.Warn("stale-lock: a disk's vzdump snapshot volume is already gone — retrying the delete once with force (R-1010)",
"vmid", vmid, "err", err)
}
if ferr := c.deleteSnap(ctx, vmid, true); ferr != nil {
return fmt.Errorf("forced snapshot delete after %q also failed: %w", err.Error(), ferr)
}
return nil
}
func (c *staleLockController) deleteSnap(ctx context.Context, vmid int, force bool) error {
del := c.px.DeleteSnapshot
if force {
del = c.px.DeleteSnapshotForce
}
upid, err := del(ctx, vmid, vzdumpSnapshotName)
if err != nil { if err != nil {
return err return err
} }
@@ -346,6 +374,23 @@ func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int
return err return err
} }
// snapshotVolumeAlreadyGone: the error is Proxmox's „Failed to find logical volume …_vzdump" from a snapshot delete
// (in the task's exit status, its log tail, or the API error text). Narrow on purpose: the force is safe only when the
// volume it cannot remove does not exist.
func snapshotVolumeAlreadyGone(err error) bool {
text := err.Error()
var te *proxmox.TaskError
if errors.As(err, &te) {
text += "\n" + te.ExitStatus + "\n" + strings.Join(te.LogTail, "\n")
}
for _, l := range strings.Split(text, "\n") {
if strings.Contains(strings.ToLower(l), "failed to find logical volume") && strings.Contains(l, "_"+vzdumpSnapshotName) {
return true
}
}
return false
}
func (c *staleLockController) Start(ctx context.Context, vmid int) error { func (c *staleLockController) Start(ctx context.Context, vmid int) error {
upid, err := c.px.Start(ctx, vmid) upid, err := c.px.Start(ctx, vmid)
if err != nil { if err != nil {
+19 -1
View File
@@ -30,6 +30,9 @@ type fakeStaleLockAPI struct {
delsnap []int delsnap []int
started []int started []int
// R-1010: scripted delete answers, consumed in order (plain and forced share the list); calls records "plain"/"force".
delErrs []error
delCalls []string
} }
func (f *fakeStaleLockAPI) ListLXC(context.Context) ([]proxmox.Guest, error) { return f.lxc, nil } func (f *fakeStaleLockAPI) ListLXC(context.Context) ([]proxmox.Guest, error) { return f.lxc, nil }
@@ -55,7 +58,22 @@ func (f *fakeStaleLockAPI) ListRunningTasks(context.Context) ([]proxmox.TaskStat
func (f *fakeStaleLockAPI) DeleteSnapshot(_ context.Context, vmid int, snapname string) (string, error) { func (f *fakeStaleLockAPI) DeleteSnapshot(_ context.Context, vmid int, snapname string) (string, error) {
f.delsnap = append(f.delsnap, vmid) f.delsnap = append(f.delsnap, vmid)
return "", nil // synchronous — no WaitTask return "", f.nextDelErr("plain") // synchronous — no WaitTask
}
func (f *fakeStaleLockAPI) DeleteSnapshotForce(_ context.Context, vmid int, snapname string) (string, error) {
f.delsnap = append(f.delsnap, vmid)
return "", f.nextDelErr("force")
}
func (f *fakeStaleLockAPI) nextDelErr(kind string) error {
f.delCalls = append(f.delCalls, kind)
if len(f.delErrs) == 0 {
return nil
}
e := f.delErrs[0]
f.delErrs = f.delErrs[1:]
return e
} }
func (f *fakeStaleLockAPI) Start(_ context.Context, vmid int) (string, error) { func (f *fakeStaleLockAPI) Start(_ context.Context, vmid int) (string, error) {
+59
View File
@@ -0,0 +1,59 @@
package localapi
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1010: a half-removed vzdump snapshot. The error text is the one demo-hp's task printed on 2026-10-14
// (`audits/power-cut-fixes-2026-10-14/C4/hand-fix.txt`).
const r1010MissingLV = `lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to find logical volume "pve/snap_vm-9201-disk-0_vzdump"`
func r1010TaskErr(exit string) error {
return &proxmox.TaskError{UPID: "UPID:demo-hp:0001:0002:0003:vzdelsnapshot:9201:root@pam:", ExitStatus: exit,
LogTail: []string{exit}}
}
// COMPANION RED-PROOF (observed 2026-10-19): with DeleteVzdumpSnapshot back to the single plain delete, the first case
// fails with
//
// stalelock_r1010_test.go: missing volume: calls [plain], want [plain force]; err = proxmox: task … failed: exitstatus "lvremove snapshot …"
//
// Restored.
func TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone(t *testing.T) {
cases := []struct {
name string
errs []error
wantCalls []string
wantErr string
}{
{"missing volume", []error{r1010TaskErr(r1010MissingLV)}, []string{"plain", "force"}, ""},
{"missing volume, named only in the log tail", []error{&proxmox.TaskError{UPID: "x", ExitStatus: "command 'lvremove' failed: exit code 5",
LogTail: []string{r1010MissingLV}}}, []string{"plain", "force"}, ""},
{"the plain delete works", nil, []string{"plain"}, ""},
{"another error: no force", []error{r1010TaskErr("lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to update pool pve/data.")},
[]string{"plain"}, "Failed to update pool"},
{"a missing volume of ANOTHER snapshot: no force", []error{r1010TaskErr(`Failed to find logical volume "pve/snap_vm-9201-disk-0_mine"`)},
[]string{"plain"}, "_mine"},
{"the forced delete fails too: returned, never a second force", []error{r1010TaskErr(r1010MissingLV), errors.New("still broken")},
[]string{"plain", "force"}, "still broken"},
}
for _, c := range cases {
api := &fakeStaleLockAPI{delErrs: c.errs}
ctrl := NewStaleLockController(api, &recordingRunner{}, "felhom", nil)
err := ctrl.DeleteVzdumpSnapshot(context.Background(), 9201)
if strings.Join(api.delCalls, " ") != strings.Join(c.wantCalls, " ") {
t.Errorf("%s: calls %v, want %v; err = %v", c.name, api.delCalls, c.wantCalls, err)
}
switch {
case c.wantErr == "" && err != nil:
t.Errorf("%s: err = %v, want nil", c.name, err)
case c.wantErr != "" && (err == nil || !strings.Contains(err.Error(), c.wantErr)):
t.Errorf("%s: err = %v, want one naming %q", c.name, err, c.wantErr)
}
}
}
+19 -2
View File
@@ -370,7 +370,8 @@ func HealthVerdict(before, after *Health) (bool, string) {
// HostHealthVerdict is THE host health rule (`11` §8.2; pinned by TestHostHealthVerdict): the Proxmox daemons and the // HostHealthVerdict is THE host health rule (`11` §8.2; pinned by TestHostHealthVerdict): the Proxmox daemons and the
// agent are active, the customer guest still runs, the guest's own rule still passes against the start of the pass, // agent are active, the customer guest still runs, the guest's own rule still passes against the start of the pass,
// and the tunnel is `running` (R-841). // and the tunnel is `running` (R-841) — or `absent` both now and before the update: a box with no tunnel set up
// (R-1070; pinned by TestHostHealthVerdict_R1070_NoTunnelSetUp).
func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) { func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
if after == nil { if after == nil {
return false, "no health reading" return false, "no health reading"
@@ -398,12 +399,28 @@ func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
if ok, why := HealthVerdict(gb, after.Guest); !ok { if ok, why := HealthVerdict(gb, after.Guest); !ok {
return false, "guest: " + why return false, "guest: " + why
} }
if tunnel != hub.TunnelRunning { switch {
case tunnel == hub.TunnelRunning:
case tunnel == hub.TunnelAbsent && noTunnelBefore(before):
// R-1070: no cloudflared container before the update and none after — the box has no tunnel to check.
case tunnel == hub.TunnelAbsent:
return false, "the tunnel is absent (no cloudflared container now, and the reading from before the update does not show it was absent then)"
default:
return false, "the tunnel is " + tunnel return false, "the tunnel is " + tunnel
} }
return true, "" return true, ""
} }
// noTunnelBefore: the reading from the start of the pass shows the guest's Docker answered and listed NO cloudflared
// container. A missing or failed reading is not „no tunnel" (an empty list from a failed `docker ps` must not pass).
func noTunnelBefore(before *Health) bool {
if before == nil || before.Guest == nil || !before.Guest.DockerOK {
return false
}
_, had := before.Guest.Containers["cloudflared"]
return !had
}
// EngineOf is the engine version `docker version` prints for a docker-ce package version: "5:29.8.2-1~debian.13~trixie" // EngineOf is the engine version `docker version` prints for a docker-ce package version: "5:29.8.2-1~debian.13~trixie"
// → "29.8.2" (no epoch, no Debian revision). // → "29.8.2" (no epoch, no Debian revision).
func EngineOf(pkgVersion string) string { func EngineOf(pkgVersion string) string {
+39
View File
@@ -568,3 +568,42 @@ func TestDocker_OOMCheckReachesTheHubUnchanged(t *testing.T) {
t.Fatalf("no docker report posted: %s", calls(w)) t.Fatalf("no docker report posted: %s", calls(w))
} }
} }
// R-1070: a box with no tunnel set up (no cloudflared container before the update and none after) passes the host
// rule; a box WITH a tunnel that is down, gone, or unreadable still fails it.
//
// COMPANION RED-PROOF (observed 2026-10-19): with the „not applicable" branch switched off (only `running` passes, the
// pre-fix rule) the first case fails with
//
// leg_test.go: no tunnel before, none after: got false (the tunnel is absent (no cloudflared container now, …)), want true
//
// and with the pre-fix case-sensitive match TestClassifyTunnel's Docker 29 row reads `unknown` („could not ask the
// guest"), the state Tester4 failed on.
// Restored.
func TestHostHealthVerdict_R1070_NoTunnelSetUp(t *testing.T) {
withTunnel := hostOK()
withTunnel.Guest.Containers["cloudflared"] = Container{State: "running", Health: "healthy"}
dockerFailed := hostOK()
dockerFailed.Guest = &Health{DockerOK: false, Containers: map[string]Container{}}
cases := []struct {
name string
before *Health
tunnel string
want bool
why string
}{
{"no tunnel before, none after", hostOK(), hub.TunnelAbsent, true, ""},
{"a tunnel before, gone after", withTunnel, hub.TunnelAbsent, false, "cloudflared was running"},
{"a tunnel before, down after", withTunnel, hub.TunnelNotRunning, false, "cloudflared was running"},
{"no tunnel before, a stopped one after", hostOK(), hub.TunnelNotRunning, false, "not_running"},
{"no tunnel before, the probe could not ask after", hostOK(), hub.TunnelUnknown, false, "unknown"},
{"no reading from before", nil, hub.TunnelAbsent, false, "absent"},
{"docker failed before (an empty list is not proof of none)", dockerFailed, hub.TunnelAbsent, false, "absent"},
}
for _, c := range cases {
got, why := HostHealthVerdict(c.before, hostOK(), c.tunnel)
if got != c.want || !strings.Contains(why, c.why) {
t.Errorf("%s: got %v (%s), want %v (…%s…)", c.name, got, why, c.want, c.why)
}
}
}
+12
View File
@@ -180,6 +180,18 @@ func (c *Client) DeleteSnapshot(ctx context.Context, vmid int, snapname string)
return c.dataString(ctx, http.MethodDelete, path, nil) return c.dataString(ctx, http.MethodDelete, path, nil)
} }
// DeleteSnapshotForce is DeleteSnapshot with Proxmox's own `force=1` („remove from the config file, even if removing
// disk snapshots fails"): the API's flag, so the same token privilege (VM.Snapshot) and no new root command. Used by
// the stale-lock janitor ONLY after a delete failed because one disk's snapshot volume is already gone (R-1010). The
// flag rides the query string, where pveproxy reads a DELETE's parameters. Pinned by TestDeleteSnapshotForce_R1010.
func (c *Client) DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error) {
if vmid == 0 || snapname == "" {
return "", fmt.Errorf("proxmox: DeleteSnapshotForce needs vmid and snapname")
}
path := fmt.Sprintf("/nodes/%s/lxc/%d/snapshot/%s?force=1", c.node, vmid, url.PathEscape(snapname))
return c.dataString(ctx, http.MethodDelete, path, nil)
}
// SetConfig applies config changes via PUT /nodes/{node}/lxc/{vmid}/config // SetConfig applies config changes via PUT /nodes/{node}/lxc/{vmid}/config
// (e.g. memory, cores, net0, mpN with a backup flag). PVE may apply this // (e.g. memory, cores, net0, mpN with a backup flag). PVE may apply this
// synchronously (no UPID) — the returned string is empty in that case, and "" is // synchronously (no UPID) — the returned string is empty in that case, and "" is
+22
View File
@@ -3,6 +3,7 @@ package proxmox
import ( import (
"context" "context"
"io" "io"
"net/http"
"strings" "strings"
"testing" "testing"
) )
@@ -99,3 +100,24 @@ func TestDestroyScratchLXC_RefusesUnconfiguredBand(t *testing.T) {
} }
} }
} }
// R-1010: the forced snapshot delete is the API's own flag on the same DELETE (no new root command).
func TestDeleteSnapshotForce_R1010(t *testing.T) {
var got *http.Request
d := &mockDoer{fn: func(r *http.Request) (*http.Response, error) {
got = r
return jsonResp(200, `{"data":"UPID:demo-felhom:0001:0002:0003:vzdelsnapshot:9201:u@pve!t:"}`), nil
}}
upid, err := newTestClient(d).DeleteSnapshotForce(context.Background(), 9201, "vzdump")
if err != nil || upid == "" {
t.Fatalf("DeleteSnapshotForce: upid %q err %v", upid, err)
}
if got.Method != http.MethodDelete || got.URL.Path != "/api2/json/nodes/demo-felhom/lxc/9201/snapshot/vzdump" ||
got.URL.Query().Get("force") != "1" {
t.Fatalf("request = %s %s, want DELETE …/lxc/9201/snapshot/vzdump?force=1", got.Method, got.URL.String())
}
d.fn = func(r *http.Request) (*http.Response, error) { got = r; return jsonResp(200, `{"data":null}`), nil }
if _, err := newTestClient(d).DeleteSnapshot(context.Background(), 9201, "vzdump"); err != nil || got.URL.RawQuery != "" {
t.Fatalf("the plain delete must carry no force: %s (err %v)", got.URL.String(), err)
}
}
+3
View File
@@ -645,6 +645,9 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
} }
if spec.MemoryMB > 0 { if spec.MemoryMB > 0 {
params["memory"] = strconv.Itoa(spec.MemoryMB) params["memory"] = strconv.Itoa(spec.MemoryMB)
// R-1192: swap = RAM. Left unset, Proxmox gives an LXC 512 MB, and a guest that fills its RAM can then evict
// only program code — the measured freeze (localapi.GuestSwapMB carries the reasoning and the numbers).
params["swap"] = strconv.Itoa(spec.MemoryMB)
} }
for i, m := range spec.Mounts { for i, m := range spec.Mounts {
// backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED // backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED
+19
View File
@@ -0,0 +1,19 @@
package reconcile
import (
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: a new guest is created with swap = its RAM (left unset, Proxmox gives an LXC 512 MB — the measured freeze,
// felhom.eu audits/fixes-2026-10-21/P4-freeze/). COMPANION RED-PROOF: drop params["swap"] in buildBringUpConfig.
func TestR1192_BringUpSetsSwapToTheRAM(t *testing.T) {
p := buildBringUpConfig(BringUpSpec{MemoryMB: 4096}, proxmox.GuestConfig{})
if p["memory"] != "4096" || p["swap"] != "4096" {
t.Fatalf("bring-up params memory=%q swap=%q, want 4096/4096", p["memory"], p["swap"])
}
if q := buildBringUpConfig(BringUpSpec{}, proxmox.GuestConfig{}); q["swap"] != "" {
t.Fatalf("no memory in the spec → no swap written either, got %q", q["swap"])
}
}
+26
View File
@@ -39,9 +39,21 @@ type decommissionParams struct {
// none is wanted, because a drive can legitimately be decommissioned while physically ABSENT. // none is wanted, because a drive can legitimately be decommissioned while physically ABSENT.
type DecommissionExecutor struct { type DecommissionExecutor struct {
intent IntentDecommissioner intent IntentDecommissioner
binds GuestBindPruner // optional (v0.159.1): drop the drive from every guest's bind record, as the self-serve path does
logger *slog.Logger logger *slog.Logger
} }
// GuestBindPruner is the per-guest enrolled-bind record (*localapi.GuestBindStore). The self-serve decommission prunes
// the drive from it; the signed one did not, so a forgotten drive (R-1124) left a record the 20-second guest-bind
// re-assert warned about forever ("skipping non-enrolled drive"), measured on demo-hp 2026-10-17.
type GuestBindPruner interface {
Guests() map[int][]string
Remove(vmid int, durableID string) error
}
// SetGuestBinds wires the bind record (nil-safe: no record, nothing pruned).
func (d *DecommissionExecutor) SetGuestBinds(b GuestBindPruner) { d.binds = b }
// NewDecommissionExecutor wires the executor to the agent's intent store. // NewDecommissionExecutor wires the executor to the agent's intent store.
func NewDecommissionExecutor(intent IntentDecommissioner, logger *slog.Logger) *DecommissionExecutor { func NewDecommissionExecutor(intent IntentDecommissioner, logger *slog.Logger) *DecommissionExecutor {
if logger == nil { if logger == nil {
@@ -75,6 +87,20 @@ func (d *DecommissionExecutor) Execute(_ context.Context, op string, params json
if err := d.intent.SetDecommissioned(p.DurableID); err != nil { if err := d.intent.SetDecommissioned(p.DurableID); err != nil {
return fmt.Errorf("decommission: record intent for %q: %w", p.DurableID, err) return fmt.Errorf("decommission: record intent for %q: %w", p.DurableID, err)
} }
if d.binds != nil {
for vmid, ids := range d.binds.Guests() {
for _, id := range ids {
if id != p.DurableID {
continue
}
if err := d.binds.Remove(vmid, id); err != nil {
d.logger.Warn("decommission: guest-bind record not pruned", "vmid", vmid, "durable_id", id, "err", err)
} else {
d.logger.Info("decommission: drive removed from the guest's bind record", "vmid", vmid, "durable_id", id)
}
}
}
}
d.logger.Warn("decommission: operator-signed decommission complete (drive will never auto-mount)", "durable_id", p.DurableID) d.logger.Warn("decommission: operator-signed decommission complete (drive will never auto-mount)", "durable_id", p.DurableID)
return nil return nil
} }
+34
View File
@@ -89,3 +89,37 @@ func TestExecutorChain_DispatchesToOwner(t *testing.T) {
t.Errorf("want ErrNoExecutor for an unowned op, got %v", err) t.Errorf("want ErrNoExecutor for an unowned op, got %v", err)
} }
} }
type fakeBinds struct{ m map[int][]string }
func (f *fakeBinds) Guests() map[int][]string { return f.m }
func (f *fakeBinds) Remove(vmid int, id string) error {
var kept []string
for _, x := range f.m[vmid] {
if x != id {
kept = append(kept, x)
}
}
f.m[vmid] = kept
return nil
}
// v0.159.1: the signed decommission drops the drive from EVERY guest's bind record, as the self-serve path does — and
// only that drive. RED-PROOF (felhom.eu audits/agent-allowlist-rotation-2026-10-17/P1/redproof-bindprune.txt): with
// the prune loop removed, the record still holds the drive and this test fails.
func TestDecommissionExecutor_PrunesTheGuestBindRecord(t *testing.T) {
fd := &fakeDecommissioner{}
fb := &fakeBinds{m: map[int][]string{9201: {"uuid:keep", "uuid:gone"}, 9202: {"uuid:gone"}}}
ex := NewDecommissionExecutor(fd, nil)
ex.SetGuestBinds(fb)
params, _ := json.Marshal(decommissionParams{DurableID: "uuid:gone"})
if err := ex.Execute(context.Background(), opDecommission, params); err != nil {
t.Fatal(err)
}
if got := fb.m[9201]; len(got) != 1 || got[0] != "uuid:keep" {
t.Fatalf("9201 record = %v, want [uuid:keep]", got)
}
if got := fb.m[9202]; len(got) != 0 {
t.Fatalf("9202 record = %v, want empty", got)
}
}
+85
View File
@@ -0,0 +1,85 @@
package storage
import (
"context"
"os"
"path/filepath"
"strings"
)
// ForgetRetiredDrives (R-1124) removes the host mount unit of every DECOMMISSIONED drive that is physically gone, so a
// later host boot no longer waits for it and the agent stops listing it (RegistryKnownTargets and
// ReassertEnrolledMounts both enumerate the units — no unit, no entry, no „absent by UUID" line every 20 s).
//
// Only a drive that is BOTH decommissioned (the operator's or the customer's permanent removal) AND absent (its
// /dev/disk/by-uuid link does not resolve) AND not mounted is forgotten. A decommissioned drive that is still plugged in
// keeps its unit and its raw host mount — that is what makes the one-click re-enroll (H3) work, and decommission stays a
// logical retire (`03` §storage). The intent record stays `decommissioned`: a forgotten drive that comes back is a new
// candidate, never auto-mounted. The root half (`felhom-priv-apply unit-forget`) re-checks every one of these conditions
// and refuses otherwise. Called at agent start, on the 20-second storage tick, and right after a decommission. Best
// effort per unit; returns how many units it removed. Pinned by TestForgetRetiredDrives_*.
func (h *SudoHostOps) ForgetRetiredDrives(ctx context.Context, intent IntentReader) int {
if intent == nil {
return 0 // no intent store → we cannot know a drive was retired; forget nothing
}
entries, err := os.ReadDir(h.unitDir)
if err != nil {
h.logger.Warn("storage: forget retired drives — cannot read unit dir", "dir", h.unitDir, "err", err)
return 0
}
mounted := map[string]bool{}
if ms, merr := h.host.Mounts(); merr == nil {
for _, m := range ms {
mounted[m.MountPoint] = true
}
} else {
h.logger.Warn("storage: forget retired drives — mount table unreadable; forgetting nothing", "err", merr)
return 0
}
removed := 0
for _, e := range entries {
name := e.Name()
if !strings.HasSuffix(name, ".mount") {
continue
}
data, rerr := os.ReadFile(filepath.Join(h.unitDir, name))
if rerr != nil {
continue
}
spec, ok := parseFelhomMountUnit(string(data))
if !ok {
continue // not one of our by-UUID drive units (network units are refused by the parser)
}
durableID := "uuid:" + spec.UUID
if intent.Get(durableID) != IntentDecommissioned {
continue
}
if mounted[spec.Where] {
h.logger.Debug("storage: decommissioned drive still mounted — keeping its unit", "where", spec.Where)
continue
}
if _, derr := ResolveStorageDevice(durableID); derr == nil {
h.logger.Debug("storage: decommissioned drive is present — keeping its unit (re-enroll stays one click)", "where", spec.Where)
continue
}
if h.unitEnabled(name) {
if err := h.run(ctx, h.bins.Systemctl, "disable", "--", name); err != nil {
h.logger.Warn("storage: forget retired drive — disable failed", "where", spec.Where, "unit", name, "err", err)
continue
}
}
if err := h.run(ctx, h.bins.PrivApply, "unit-forget", name); err != nil {
h.logger.Warn("storage: forget retired drive — unit removal refused or failed", "where", spec.Where, "unit", name, "err", err)
continue
}
removed++
h.logger.Info("storage: forgot a decommissioned drive that is gone — its mount unit is removed",
"name", spec.Name, "where", spec.Where, "uuid", spec.UUID, "unit", name)
}
if removed > 0 {
if err := h.run(ctx, h.bins.Systemctl, "daemon-reload"); err != nil {
h.logger.Warn("storage: daemon-reload after forgetting retired drives failed", "err", err)
}
}
return removed
}
+155
View File
@@ -0,0 +1,155 @@
package storage
import (
"context"
"io"
"os"
"path/filepath"
"reflect"
"testing"
)
// forgetRunner records every privileged call and plays the root half of `felhom-priv-apply unit-forget`: the unit file
// disappears from the unit dir, so the CONSEQUENCE (the drive is no longer listed) is observable, not just the calls.
type forgetRunner struct {
unitDir string
calls [][]string
}
func (r *forgetRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func (r *forgetRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
if len(args) == 2 && args[0] == "unit-forget" {
_ = os.Remove(filepath.Join(r.unitDir, args[1]))
}
return nil, nil, nil
}
type forgetCase struct {
intent DriveIntent
present bool // the by-uuid link resolves
mounted bool
enabled bool // the wants-symlink exists
mountErr bool
}
const forgetUUID = "4ebfd36d-87d3-4b63-839e-a5a710fc799a"
func runForget(t *testing.T, c forgetCase, intent IntentReader) (int, *forgetRunner, *RegistryKnownTargets, string) {
t.Helper()
unitDir := t.TempDir()
unit := "mnt-media_a.mount"
writeUnit(t, unitDir, unit, renderMountUnit(MountSpec{Name: forgetUUID, UUID: forgetUUID, Where: "/mnt/media_a", FSType: "ext4"}))
if c.enabled {
if err := os.MkdirAll(filepath.Join(unitDir, "multi-user.target.wants"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Symlink(filepath.Join(unitDir, unit), filepath.Join(unitDir, "multi-user.target.wants", unit)); err != nil {
t.Fatal(err)
}
}
devRoot := t.TempDir()
if err := os.MkdirAll(filepath.Join(devRoot, "by-uuid"), 0o755); err != nil {
t.Fatal(err)
}
if c.present {
dev := filepath.Join(devRoot, "sdz1")
if err := os.WriteFile(dev, nil, 0o600); err != nil {
t.Fatal(err)
}
if err := os.Symlink(dev, filepath.Join(devRoot, "by-uuid", forgetUUID)); err != nil {
t.Fatal(err)
}
}
withDevDiskRoot(t, devRoot)
hr := &fakeHostReader{}
if c.mounted {
hr.mounts = []Mount{{MountPoint: "/mnt/media_a", FSType: "ext4"}}
}
if c.mountErr {
hr.mountsErr = os.ErrPermission
}
rr := &forgetRunner{unitDir: unitDir}
ops := NewSudoHostOps(SudoHostOpsConfig{Runner: rr, Bins: Binaries{}.withDefaults(), UnitDir: unitDir,
StageDir: t.TempDir(), Host: hr, Logger: quietLogger()})
if intent == nil {
intent = &fakeIntent{m: map[string]DriveIntent{"uuid:" + forgetUUID: c.intent}}
}
n := ops.ForgetRetiredDrives(context.Background(), intent)
return n, rr, NewRegistryKnownTargets(unitDir, intent, quietLogger()), unitDir
}
// The R-1124 case: decommissioned, then unplugged. The unit is disabled and removed through the root checker, systemd
// is reloaded, and the CONSEQUENCE — the agent no longer lists the drive — holds. RED-PROOF (P1/redproof-forget.txt):
// with ForgetRetiredDrives' body replaced by `return 0` this test fails (the drive is still listed).
func TestForgetRetiredDrives_DecommissionedAndGone_UnitRemovedAndNotListed(t *testing.T) {
n, rr, known, _ := runForget(t, forgetCase{intent: IntentDecommissioned, enabled: true}, nil)
if n != 1 {
t.Fatalf("removed = %d, want 1 (calls %v)", n, rr.calls)
}
pa := Binaries{}.withDefaults()
want := [][]string{
{pa.Systemctl, "disable", "--", "mnt-media_a.mount"},
{pa.PrivApply, "unit-forget", "mnt-media_a.mount"},
{pa.Systemctl, "daemon-reload"},
}
if !reflect.DeepEqual(rr.calls, want) {
t.Fatalf("calls = %v\nwant %v", rr.calls, want)
}
got, err := known.Known(context.Background())
if err != nil {
t.Fatal(err)
}
if len(got) != 0 {
t.Fatalf("the forgotten drive is still listed: %+v", got)
}
}
// Already disabled (the Tester 1 shape after a manual disable): no disable call, the removal still runs.
func TestForgetRetiredDrives_AlreadyDisabled_NoDisableCall(t *testing.T) {
n, rr, _, _ := runForget(t, forgetCase{intent: IntentDecommissioned}, nil)
pa := Binaries{}.withDefaults()
want := [][]string{{pa.PrivApply, "unit-forget", "mnt-media_a.mount"}, {pa.Systemctl, "daemon-reload"}}
if n != 1 || !reflect.DeepEqual(rr.calls, want) {
t.Fatalf("n=%d calls=%v want %v", n, rr.calls, want)
}
}
// Everything that must KEEP its unit: a decommissioned drive still plugged in (re-enroll is one click — H3), one still
// mounted, an enrolled or ejected drive that is merely absent (it comes back), a drive with no recorded intent, and an
// unreadable mount table. No privileged call at all, and the drive stays listed where it was listed.
// RED-PROOF: with the present-drive check removed, the "present" case calls unit-forget and fails.
func TestForgetRetiredDrives_KeepsEveryOtherUnit(t *testing.T) {
for name, c := range map[string]forgetCase{
"decommissioned but present": {intent: IntentDecommissioned, present: true, enabled: true},
"decommissioned but mounted": {intent: IntentDecommissioned, mounted: true},
"enrolled and absent": {intent: IntentEnrolled},
"ejected and absent": {intent: IntentEjected},
"no intent": {intent: IntentNew},
"mount table unreadable": {intent: IntentDecommissioned, mountErr: true},
} {
t.Run(name, func(t *testing.T) {
n, rr, _, unitDir := runForget(t, c, nil)
if n != 0 || len(rr.calls) != 0 {
t.Fatalf("n=%d calls=%v — this unit must be kept", n, rr.calls)
}
if _, err := os.Stat(filepath.Join(unitDir, "mnt-media_a.mount")); err != nil {
t.Fatalf("unit file gone: %v", err)
}
})
}
}
// No intent store on the host (open failure): nothing is known to be retired, so nothing is forgotten.
func TestForgetRetiredDrives_NilIntent_ForgetsNothing(t *testing.T) {
unitDir := t.TempDir()
writeUnit(t, unitDir, "mnt-media_a.mount", renderMountUnit(MountSpec{Name: "x", UUID: forgetUUID, Where: "/mnt/media_a"}))
rr := &forgetRunner{unitDir: unitDir}
ops := NewSudoHostOps(SudoHostOpsConfig{Runner: rr, UnitDir: unitDir, StageDir: t.TempDir(), Host: &fakeHostReader{}, Logger: quietLogger()})
if n := ops.ForgetRetiredDrives(context.Background(), nil); n != 0 || len(rr.calls) != 0 {
t.Fatalf("n=%d calls=%v", n, rr.calls)
}
}
+32 -10
View File
@@ -419,8 +419,9 @@ func (h *SudoHostOps) InspectDevice(ctx context.Context, device string) (DeviceP
// any TYPE/PTTYPE/USAGE line is positive data-bearing evidence. Its exit code is NOT relied // any TYPE/PTTYPE/USAGE line is positive data-bearing evidence. Its exit code is NOT relied
// on (blkid exits 2 on a blank device) — output presence is what matters. A broken/empty // on (blkid exits 2 on a blank device) — output presence is what matters. A broken/empty
// blkid simply adds no positive evidence; lsblk (below) is the read-success authority. // blkid simply adds no positive evidence; lsblk (below) is the read-success authority.
bout, _, _ := h.runner.Run(ctx, h.bins.Blkid, "-p", "-o", "export", device) bout, _, berr := h.runner.Run(ctx, h.bins.Blkid, "-p", "-o", "export", device)
for k, v := range parseBlkidExport(bout) { blk := parseBlkidExport(bout)
for k, v := range blk {
switch k { switch k {
case "TYPE": case "TYPE":
probe.HasFilesystem = true probe.HasFilesystem = true
@@ -439,11 +440,11 @@ func (h *SudoHostOps) InspectDevice(ctx context.Context, device string) (DeviceP
// lsblk -J is the READ-SUCCESS authority + the partition/mount view. It exits 0 on any valid // lsblk -J is the READ-SUCCESS authority + the partition/mount view. It exits 0 on any valid
// device (blank or not), so a clean parse means the agent reliably read the device. If lsblk // device (blank or not), so a clean parse means the agent reliably read the device. If lsblk
// fails, Probed stays false → DataBearing()=true (fail-safe — never call a device blank on a // fails, Probed stays false → DataBearing()=true (fail-safe — never call a device blank on a
// failed read). // failed read). An exit 0 whose output does not parse into a device is NOT a read either (R-1093
// review): it stays unprobed, never "blank".
lout, _, lerr := h.runner.Run(ctx, h.bins.Lsblk, "-J", "-o", "NAME,FSTYPE,PTTYPE,MOUNTPOINT", device) lout, _, lerr := h.runner.Run(ctx, h.bins.Lsblk, "-J", "-o", "NAME,FSTYPE,PTTYPE,MOUNTPOINT", device)
if lerr == nil { if hasChildren, mounted, fstype, pttype, parsed := parseLsblkDevice(lout); lerr == nil && parsed {
probe.Probed = true probe.Probed = true
hasChildren, mounted, fstype, pttype := parseLsblkDevice(lout)
if hasChildren { if hasChildren {
probe.HasPartitions = true probe.HasPartitions = true
} }
@@ -456,13 +457,33 @@ func (h *SudoHostOps) InspectDevice(ctx context.Context, device string) (DeviceP
probe.FSType = fstype probe.FSType = fstype
} }
} }
if pttype != "" { if pttype != "" && !inheritedPartitionTable(blk, berr, pttype) {
probe.HasPartitionTable = true probe.HasPartitionTable = true
} }
} }
return probe, nil return probe, nil
} }
// inheritedPartitionTable reports whether lsblk's PTTYPE for the probed node is only the PARENT disk's
// table (R-1093). lsblk prints the disk's table type on every one of its partitions, so an EMPTY
// partition read as "device has a partition table" and asked for the typed erase confirmation.
//
// The partition's OWN signatures come from `blkid -p` on the partition itself (above): a nested table
// inside it is its PTTYPE, a filesystem is its TYPE. So lsblk's value is set aside ONLY when blkid
// positively proved two things — that it completed its probe of the device (exit 0: PART_ENTRY_* alone
// comes from the PARENT's table, so output without a clean exit does not prove the partition's contents
// were read), and that the device is a partition of a table of exactly that type (PART_ENTRY_NUMBER +
// PART_ENTRY_SCHEME == lsblk's PTTYPE). Anything else keeps the old, safe reading: a blkid that failed,
// exited non-zero or printed nothing, a whole disk, or a scheme that differs from lsblk's (then lsblk sees
// a table blkid did not attribute to the parent). Pinned by TestInspect_R1093_* (hostops_disk_test.go).
func inheritedPartitionTable(blkid map[string]string, blkidErr error, lsblkPTType string) bool {
if blkidErr != nil || blkid["PART_ENTRY_NUMBER"] == "" {
return false
}
scheme := blkid["PART_ENTRY_SCHEME"]
return scheme != "" && strings.EqualFold(scheme, lsblkPTType)
}
// Format runs mkfs.<fstype> on a validated device. The caller is responsible for authorization // Format runs mkfs.<fstype> on a validated device. The caller is responsible for authorization
// (8C: only after classifying the device non-data-bearing, or via a slice-10 operator signature). // (8C: only after classifying the device non-data-bearing, or via a slice-10 operator signature).
func (h *SudoHostOps) Format(ctx context.Context, device, fstype string) error { func (h *SudoHostOps) Format(ctx context.Context, device, fstype string) error {
@@ -541,13 +562,14 @@ type lsblkDevice struct {
} }
// parseLsblkDevice parses `lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT <device>` for the top device: // parseLsblkDevice parses `lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT <device>` for the top device:
// whether it has child partitions, is mounted (itself or any child), and its fstype/pttype. // whether it has child partitions, is mounted (itself or any child), and its fstype/pttype. parsed=false
func parseLsblkDevice(out []byte) (hasChildren, mounted bool, fstype, pttype string) { // when the output is not a device listing — the caller must then treat the device as unprobed.
func parseLsblkDevice(out []byte) (hasChildren, mounted bool, fstype, pttype string, parsed bool) {
var doc struct { var doc struct {
BlockDevices []lsblkDevice `json:"blockdevices"` BlockDevices []lsblkDevice `json:"blockdevices"`
} }
if json.Unmarshal(out, &doc) != nil || len(doc.BlockDevices) == 0 { if json.Unmarshal(out, &doc) != nil || len(doc.BlockDevices) == 0 {
return false, false, "", "" return false, false, "", "", false
} }
d := doc.BlockDevices[0] d := doc.BlockDevices[0]
fstype, pttype = d.FSType, d.PTType fstype, pttype = d.FSType, d.PTType
@@ -558,7 +580,7 @@ func parseLsblkDevice(out []byte) (hasChildren, mounted bool, fstype, pttype str
mounted = true mounted = true
} }
} }
return hasChildren, mounted, fstype, pttype return hasChildren, mounted, fstype, pttype, true
} }
// NoopHostOps is the safe fallback when the privileged surface is unavailable or declined // NoopHostOps is the safe fallback when the privileged surface is unavailable or declined
+111
View File
@@ -222,3 +222,114 @@ func TestInspect_ReadsFilesystemUUID(t *testing.T) {
t.Fatalf("FSUUID = %q, want the blkid UUID", p.FSUUID) t.Fatalf("FSUUID = %q, want the blkid UUID", p.FSUUID)
} }
} }
// ---- R-1093: an EMPTY partition is blank; lsblk's PTTYPE on a partition is the PARENT disk's ----------
// blkidEmptyGPTPart is `blkid -p -o export /dev/sdb1` for an empty GPT partition: only the partition's
// own entry in the parent table (PART_ENTRY_*), no TYPE, no PTTYPE — nothing inside the partition.
const blkidEmptyGPTPart = "DEVNAME=/dev/sdb1\nPART_ENTRY_SCHEME=gpt\nPART_ENTRY_UUID=3b0c1a52-55a4-4c4e-9d0b-7a1f0e0f7c11\n" +
"PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4\nPART_ENTRY_NUMBER=1\nPART_ENTRY_OFFSET=2048\n" +
"PART_ENTRY_SIZE=2097152\nPART_ENTRY_DISK=8:16\n"
// lsblkPartInheritsGPT is `lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT /dev/sdb1`: lsblk prints the DISK's
// table type on the partition — the input that made the agent read an empty partition as data-bearing.
const lsblkPartInheritsGPT = `{"blockdevices":[{"name":"sdb1","fstype":null,"pttype":"gpt","mountpoint":null}]}`
// TestInspect_R1093_PartitionVerdicts pins the CONSEQUENCE (DataBearing + Reason, what decides whether
// the typed erase confirmation is asked) for a partition, in both directions.
//
// COMPANION RED-PROOFS (observed 2026-10-18): (1) restore the pre-fix `if pttype != "" {` (drop the
// inheritedPartitionTable call) → BOTH empty-partition rows (GPT and dos) FAIL with
// `DataBearing=true reason="device has a partition table", want blank`. (2) drop the `blkidErr != nil`
// check → the "exit 2" and "I/O error" rows FAIL with `DataBearing=false reason="device is blank"`.
// (3) drop `&& parsed` → both lsblk-exit-0 rows FAIL with `Probed=true … reason="device is blank"`
// (the unparsable-output hole predates R-1093; closed here). Restored.
func TestInspect_R1093_PartitionVerdicts(t *testing.T) {
cases := []struct {
name string
dev string
blkid string
blkidErr error
lsblk string
lsblkErr error
wantBear bool
wantReason string
unprobed bool // the verdict must be "could not be reliably inspected"
}{
{name: "empty GPT partition is blank (the R-1093 case)", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, lsblk: lsblkPartInheritsGPT,
wantBear: false, wantReason: "device is blank"},
{name: "empty MBR (dos) partition is blank", dev: "/dev/sdb2",
blkid: "DEVNAME=/dev/sdb2\nPART_ENTRY_SCHEME=dos\nPART_ENTRY_TYPE=0x83\nPART_ENTRY_NUMBER=2\nPART_ENTRY_DISK=8:16\n",
lsblk: `{"blockdevices":[{"name":"sdb2","fstype":null,"pttype":"dos","mountpoint":null}]}`,
wantBear: false, wantReason: "device is blank"},
{name: "partition with a filesystem stays data-bearing", dev: "/dev/sdb1",
blkid: "DEVNAME=/dev/sdb1\nUUID=8c1e\nTYPE=ext4\nUSAGE=filesystem\n" + blkidEmptyGPTPart[len("DEVNAME=/dev/sdb1\n"):],
lsblk: `{"blockdevices":[{"name":"sdb1","fstype":"ext4","pttype":"gpt","mountpoint":null}]}`,
wantBear: true, wantReason: "device has a ext4 filesystem"},
{name: "partition holding its OWN (nested) table stays data-bearing", dev: "/dev/sdb1",
blkid: "DEVNAME=/dev/sdb1\nPTUUID=0b8d4358\nPTTYPE=dos\n" + blkidEmptyGPTPart[len("DEVNAME=/dev/sdb1\n"):],
lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "blkid printed nothing: lsblk's PTTYPE stands (fail-safe)", dev: "/dev/sdb1",
blkid: "", blkidErr: errors.New("sudo: a password is required"), lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
// Fail-closed paths (the 2026-10-18 review): PART_ENTRY_* is read from the PARENT's table, so it
// proves nothing about the partition's contents unless blkid also completed (exit 0).
{name: "blkid exit 2 with only PART_ENTRY output: lsblk stands", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, blkidErr: errors.New("exit status 2"), lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "blkid I/O error after PART_ENTRY output: lsblk stands", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, blkidErr: errors.New("exit status 1"), lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "blkid binary missing: lsblk stands", dev: "/dev/sdb1",
blkid: "", blkidErr: errors.New(`exec: "/usr/sbin/blkid": file does not exist`), lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "lsblk exit 0 with unparsable output: unprobed", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, lsblk: `lsblk: something odd`,
wantBear: true, unprobed: true},
{name: "lsblk exit 0 with no device listed: unprobed", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, lsblk: `{"blockdevices":[]}`,
wantBear: true, unprobed: true},
{name: "lsblk failed: unprobed", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, lsblk: "", lsblkErr: errors.New("exit status 32"),
wantBear: true, unprobed: true},
{name: "scheme differs from lsblk's PTTYPE: lsblk stands", dev: "/dev/sdb1",
blkid: strings.Replace(blkidEmptyGPTPart, "PART_ENTRY_SCHEME=gpt", "PART_ENTRY_SCHEME=dos", 1),
lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "whole disk with a GPT stays data-bearing", dev: "/dev/sdb",
blkid: "DEVNAME=/dev/sdb\nPTUUID=a07f88ff\nPTTYPE=gpt\n",
lsblk: `{"blockdevices":[{"name":"sdb","fstype":null,"pttype":"gpt","mountpoint":null}]}`,
wantBear: true, wantReason: "device has a partition table"},
{name: "whole disk: blkid silent, lsblk GPT stands", dev: "/dev/sdb",
blkid: "", lsblk: `{"blockdevices":[{"name":"sdb","fstype":null,"pttype":"gpt","mountpoint":null}]}`,
wantBear: true, wantReason: "device has a partition table"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
r := &scriptedRunner{
outputs: map[string][]byte{"blkid": []byte(c.blkid), "lsblk": []byte(c.lsblk)},
errs: map[string]error{"blkid": c.blkidErr, "lsblk": c.lsblkErr},
}
p, err := newSudo(r).InspectDevice(context.Background(), c.dev)
if err != nil {
t.Fatal(err)
}
if c.unprobed {
if p.Probed || !p.DataBearing() || p.Reason() != "device could not be reliably inspected" {
t.Fatalf("Probed=%v DataBearing=%v reason=%q, want unprobed + data-bearing (probe %+v)",
p.Probed, p.DataBearing(), p.Reason(), p)
}
return
}
if !p.Probed {
t.Fatal("expected a clean probe (lsblk read cleanly)")
}
if p.DataBearing() != c.wantBear || p.Reason() != c.wantReason {
t.Fatalf("DataBearing=%v reason=%q, want DataBearing=%v reason=%q (probe %+v)",
p.DataBearing(), p.Reason(), c.wantBear, c.wantReason, p)
}
})
}
}
+20 -1
View File
@@ -13,7 +13,8 @@ import (
// A disk that passed the WHOLE-disk unclaimed filter (classifyClaim: not the system disk, no LVM/ZFS/RAID/LUKS/swap // A disk that passed the WHOLE-disk unclaimed filter (classifyClaim: not the system disk, no LVM/ZFS/RAID/LUKS/swap
// member on any node, no foreign mount on any node) also lists its partitions, each with its own state: // member on any node, no foreign mount on any node) also lists its partitions, each with its own state:
// //
// empty no filesystem, no children → „format this part" (self-serve: the agent's probe says blank) // empty no filesystem, no children → „format this part" (self-serve: the agent's probe says blank) —
// only while NO part of the disk is mounted (Formattable)
// filesystem ext4/xfs (mountable) or another fs → „use this part" when mountable; „format this part" only through // filesystem ext4/xfs (mountable) or another fs → „use this part" when mountable; „format this part" only through
// the customer-confirmed data-bearing path (the agent's own probe) // the customer-confirmed data-bearing path (the agent's own probe)
// in_use already Felhom's own drive → shown, never offered (it is a registered drive already) // in_use already Felhom's own drive → shown, never offered (it is a registered drive already)
@@ -32,6 +33,10 @@ type CandidatePartition struct {
Mountable bool `json:"mountable"` Mountable bool `json:"mountable"`
DurableID string `json:"durable_id,omitempty"` // uuid:<fs-uuid> — the attach key ("" when empty) DurableID string `json:"durable_id,omitempty"` // uuid:<fs-uuid> — the attach key ("" when empty)
MountedAt string `json:"mounted_at,omitempty"` // in_use: Felhom's own mount MountedAt string `json:"mounted_at,omitempty"` // in_use: Felhom's own mount
// Formattable is false when ANY node of the disk is mounted (another part in use): the root mkfs wrapper refuses
// every mount outside /mnt/felhom-drives on the disk and deliberately does not trust a sibling that claims to be
// Felhom's own (a compromised agent could fake that). Such a part can still be USED (attach erases nothing).
Formattable bool `json:"formattable"`
} }
const ( const (
@@ -55,6 +60,19 @@ func candidatePartitions(lsblkJSON []byte, wholeName string, sizeOf func(string)
if d.Name != wholeName { if d.Name != wholeName {
continue continue
} }
diskMounted := d.MountPoint != ""
var walk func(n lsblkDev)
walk = func(n lsblkDev) {
if n.MountPoint != "" {
diskMounted = true
}
for _, c := range n.Children {
walk(c)
}
}
for _, c := range d.Children {
walk(c)
}
for _, p := range d.Children { for _, p := range d.Children {
if len(p.Children) > 0 || memberFSTypes[p.FSType] { if len(p.Children) > 0 || memberFSTypes[p.FSType] {
continue // LVM/mapper on it, or a member: not a place (the disk filter refuses these anyway) continue // LVM/mapper on it, or a member: not a place (the disk filter refuses these anyway)
@@ -74,6 +92,7 @@ func candidatePartitions(lsblkJSON []byte, wholeName string, sizeOf func(string)
c.DurableID = "uuid:" + u c.DurableID = "uuid:" + u
} }
} }
c.Formattable = c.State != PartInUse && !diskMounted
out = append(out, c) out = append(out, c)
} }
} }
+18
View File
@@ -71,3 +71,21 @@ func TestPartitions_UncorroboratedMountIsNotOffered(t *testing.T) {
} }
} }
} }
// The root wrapper refuses any format on a disk with a mounted node (and trusts no „it is Felhom's own" claim), so a
// part next to a mounted one is never offered for formatting — only for use. Positive control: an unmounted disk.
func TestPartitions_FormattableOnlyWhenNothingOnTheDiskIsMounted(t *testing.T) {
m := listParts(map[string]bool{"/mnt/films": true})
for dev, p := range m {
if p.Formattable {
t.Fatalf("%s: a part of a disk with a mounted part must not be formattable: %+v", dev, p)
}
}
free := `{"blockdevices":[{"name":"sdc","fstype":null,"mountpoint":null,"children":[
{"name":"sdc1","fstype":null,"mountpoint":null},{"name":"sdc2","fstype":"ext4","mountpoint":null}]}]}`
got := candidatePartitions([]byte(free), "sdc", func(string) int64 { return 1 << 30 }, func(string) int { return 1 },
func(string) (string, bool) { return "u", true }, nil)
if len(got) != 2 || !got[0].Formattable || !got[1].Formattable {
t.Fatalf("parts of an unmounted disk are formattable, got %+v", got)
}
}
+243
View File
@@ -0,0 +1,243 @@
// Package thincheck finds an ORPHAN thin device: one the LVM thin pool holds that LVM itself no longer knows (R-1012).
//
// A full host root while vzdump takes its storage snapshot can leave such a device behind (demo-hp 2026-10-14: the
// pool held devices 1, 2 and 4; LVM knew 1 and 2). From then on every snapshot backup of that guest fails at the
// snapshot step („Creation of new snapshot 4 of device 2 failed") and nothing in the product said why. This package
// NAMES the orphan, in plain words, in the backup's failure — it never removes one: the repair
// (`dmsetup message <pool>-tpool 0 "delete <id>"`) stays a human step, decided by the operator (`03` whole-guest
// backup; runbook `whole-guest-backup-orphan-thin.md`).
//
// The device list comes from the root helper `felhom-thin-check <vg> <pool>` (config bundle), which reads the pool
// through a metadata snapshot and LVM's list before and after. An id is an orphan only when the pool holds it and
// NEITHER LVM list names it, on two reads a few seconds apart — so a volume being created at that moment is never
// reported.
package thincheck
import (
"context"
"fmt"
"log/slog"
"sort"
"strconv"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// HelperPath is where the config bundle installs the root helper.
const HelperPath = "/usr/local/sbin/felhom-thin-check"
// Pool names one LVM thin pool.
type Pool struct{ VG, Pool string }
func (p Pool) String() string { return p.VG + "/" + p.Pool }
// Reading is one helper run, parsed.
type Reading struct {
PoolIDs []int
LVMBefore map[int]string
LVMAfter map[int]string
}
// Parse reads the helper's output. ok=false unless the closing "ok" line is there (a cut-off run is no reading).
func Parse(out string) (Reading, bool) {
r := Reading{LVMBefore: map[int]string{}, LVMAfter: map[int]string{}}
done := false
for _, line := range strings.Split(out, "\n") {
f := strings.Fields(line)
switch {
case len(f) == 1 && f[0] == "ok":
done = true
case len(f) == 2 && f[0] == "pool":
if id, err := strconv.Atoi(f[1]); err == nil {
r.PoolIDs = append(r.PoolIDs, id)
}
case len(f) == 3 && (f[0] == "lvm-before" || f[0] == "lvm-after"):
id, err := strconv.Atoi(f[1])
if err != nil {
continue
}
if f[0] == "lvm-before" {
r.LVMBefore[id] = f[2]
} else {
r.LVMAfter[id] = f[2]
}
}
}
return r, done
}
// Orphans returns the ids the pool holds that neither LVM list names, sorted.
func (r Reading) Orphans() []int {
var out []int
for _, id := range r.PoolIDs {
_, b := r.LVMBefore[id]
_, a := r.LVMAfter[id]
if !b && !a {
out = append(out, id)
}
}
sort.Ints(out)
return out
}
// Runner is the privileged exec surface (proxmox.ExecRunner in sudo mode).
type Runner interface {
Run(ctx context.Context, name string, args ...string) ([]byte, []byte, error)
}
// StorageLister is the read the checker needs to map a guest's volumes to their thin pools (*proxmox.Client).
type StorageLister interface {
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
ListStorage(ctx context.Context) ([]proxmox.Storage, error)
}
// Checker runs the check for one guest's pools.
type Checker struct {
API StorageLister
Runner Runner
Helper string // "" → HelperPath
Pause time.Duration // between the two reads when the first finds an orphan; 0 → 3 s
Logger *slog.Logger
}
// Finding is one pool with orphans.
type Finding struct {
Pool Pool
Orphans []int
}
// PoolsOf maps the guest's volume-backed disks (rootfs, mpN) to the lvmthin pools they live on.
func (c *Checker) PoolsOf(ctx context.Context, vmid int) ([]Pool, error) {
cfg, err := c.API.GuestConfig(ctx, vmid)
if err != nil {
return nil, err
}
stores, err := c.API.ListStorage(ctx)
if err != nil {
return nil, err
}
thin := map[string]Pool{}
for _, s := range stores {
if s.Type == "lvmthin" && s.VGName != "" && s.ThinPool != "" {
thin[s.Storage] = Pool{VG: s.VGName, Pool: s.ThinPool}
}
}
vols := []string{cfg.RootFS}
mps := cfg.MountPoints()
keys := make([]string, 0, len(mps))
for k := range mps {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
vols = append(vols, mps[k])
}
seen := map[Pool]bool{}
var out []Pool
for _, v := range vols {
store, _, ok := strings.Cut(strings.Split(v, ",")[0], ":")
if !ok || strings.HasPrefix(store, "/") {
continue // a bind mount (a path) is no volume
}
if p, isThin := thin[store]; isThin && !seen[p] {
seen[p] = true
out = append(out, p)
}
}
return out, nil
}
// read runs the helper once.
func (c *Checker) read(ctx context.Context, p Pool) (Reading, error) {
helper := c.Helper
if helper == "" {
helper = HelperPath
}
out, stderr, err := c.Runner.Run(ctx, helper, p.VG, p.Pool)
if err != nil {
return Reading{}, fmt.Errorf("felhom-thin-check %s: %w: %s", p, err, strings.TrimSpace(string(stderr)))
}
r, ok := Parse(string(out))
if !ok {
return Reading{}, fmt.Errorf("felhom-thin-check %s: the output has no closing ok line", p)
}
return r, nil
}
// Check returns the pools of vmid that hold an orphan (two reads, intersected). A pool that cannot be read is logged
// and left out: the check FAILS OPEN — it explains a failure, it never stops a backup.
func (c *Checker) Check(ctx context.Context, vmid int) []Finding {
logger := c.Logger
if logger == nil {
logger = slog.Default()
}
pools, err := c.PoolsOf(ctx, vmid)
if err != nil {
logger.Warn("thincheck: cannot map the guest's disks to thin pools — not checked", "vmid", vmid, "err", err)
return nil
}
pause := c.Pause
if pause <= 0 {
pause = 3 * time.Second
}
var found []Finding
for _, p := range pools {
first, err := c.read(ctx, p)
if err != nil {
logger.Warn("thincheck: thin pool not read — not checked", "vmid", vmid, "pool", p.String(), "err", err)
continue
}
o1 := first.Orphans()
if len(o1) == 0 {
logger.Debug("thincheck: thin pool and LVM agree", "vmid", vmid, "pool", p.String(), "devices", len(first.PoolIDs))
continue
}
select {
case <-ctx.Done():
return found
case <-time.After(pause):
}
second, err := c.read(ctx, p)
if err != nil {
logger.Warn("thincheck: second read failed — not reported", "vmid", vmid, "pool", p.String(), "err", err)
continue
}
in2 := map[int]bool{}
for _, id := range second.Orphans() {
in2[id] = true
}
var both []int
for _, id := range o1 {
if in2[id] {
both = append(both, id)
}
}
if len(both) > 0 {
found = append(found, Finding{Pool: p, Orphans: both})
logger.Warn("thincheck: ORPHAN thin device(s) in the pool — LVM does not know them; NOT removed (a human step)",
"vmid", vmid, "pool", p.String(), "orphans", both)
}
}
return found
}
// Describe is the operator sentence for a backup failure. Plain words; names the pool and the ids; says nothing was
// removed and where the repair is written down.
func Describe(f []Finding) string {
if len(f) == 0 {
return ""
}
var parts []string
for _, x := range f {
ids := make([]string, len(x.Orphans))
for i, id := range x.Orphans {
ids[i] = strconv.Itoa(id)
}
parts = append(parts, fmt.Sprintf("thin pool %s holds device %s that LVM does not know", x.Pool, strings.Join(ids, ", ")))
}
return "probable cause: " + strings.Join(parts, "; ") +
" (an orphan snapshot, usually left when the host disk filled during a backup). Snapshot backups on this pool " +
"fail until an operator removes it; Felhom did not remove anything. Repair: runbook whole-guest-backup-orphan-thin.md"
}
+184
View File
@@ -0,0 +1,184 @@
package thincheck
import (
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"os/exec"
"path/filepath"
"reflect"
"runtime"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// The helper's output on demo-hp 2026-10-14, before the repair: the pool held 1, 2 and 4; LVM knew 1 and 2
// (audits/power-cut-fixes-2026-10-14/E/thin-ls.txt).
const orphanOut = `lvm-before 1 vm-9201-disk-0
lvm-before 2 vm-9201-disk-1
pool 1
pool 2
pool 4
lvm-after 1 vm-9201-disk-0
lvm-after 2 vm-9201-disk-1
ok
`
const healthyOut = `lvm-before 1 vm-9201-disk-0
lvm-before 2 vm-9201-disk-1
pool 1
pool 2
lvm-after 1 vm-9201-disk-0
lvm-after 2 vm-9201-disk-1
ok
`
func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
func TestParse_TheOrphanOf20261014(t *testing.T) {
r, ok := Parse(orphanOut)
if !ok {
t.Fatal("a complete reading was not accepted")
}
if got := r.Orphans(); !reflect.DeepEqual(got, []int{4}) {
t.Fatalf("orphans = %v, want [4]", got)
}
h, _ := Parse(healthyOut)
if got := h.Orphans(); len(got) != 0 {
t.Fatalf("a healthy pool reported orphans %v", got)
}
}
// A volume LVM created WHILE the pool was read appears only in the after-list: never an orphan.
func TestOrphans_AVolumeCreatedDuringTheReadIsNotAnOrphan(t *testing.T) {
r, _ := Parse(strings.Replace(orphanOut, "lvm-after 2 vm-9201-disk-1\n", "lvm-after 2 vm-9201-disk-1\nlvm-after 4 snap_vm-9201-disk-1_vzdump\n", 1))
if got := r.Orphans(); len(got) != 0 {
t.Fatalf("orphans = %v, want none", got)
}
}
// A run cut off before its closing "ok" (the helper died, the output is partial) is no reading at all.
func TestParse_NoClosingLineIsNoReading(t *testing.T) {
if _, ok := Parse(strings.TrimSuffix(orphanOut, "ok\n")); ok {
t.Fatal("a cut-off run was accepted as a reading")
}
}
type fakeAPI struct {
cfg proxmox.GuestConfig
stores []proxmox.Storage
}
func (f *fakeAPI) GuestConfig(context.Context, int) (proxmox.GuestConfig, error) { return f.cfg, nil }
func (f *fakeAPI) ListStorage(context.Context) ([]proxmox.Storage, error) { return f.stores, nil }
func guest9201() *fakeAPI {
extra := map[string]json.RawMessage{
"mp0": json.RawMessage(`"local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=20G"`),
"mp8": json.RawMessage(`"/mnt/felhom-drives,mp=/mnt/felhom-drives"`),
"mp9": json.RawMessage(`"local:9201/vm-9201-disk-2.raw,mp=/x"`),
}
return &fakeAPI{
cfg: proxmox.GuestConfig{RootFS: "local-lvm:vm-9201-disk-0,size=8G", Extra: extra},
stores: []proxmox.Storage{
{Storage: "local", Type: "dir", Path: "/var/lib/vz"},
{Storage: "local-lvm", Type: "lvmthin", VGName: "pve", ThinPool: "data"},
{Storage: "felhom-pbs", Type: "pbs"},
},
}
}
type seqRunner struct {
outs []string
errs []error
calls [][]string
}
func (r *seqRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
i := len(r.calls) - 1
var err error
if i < len(r.errs) {
err = r.errs[i]
}
if i < len(r.outs) {
return []byte(r.outs[i]), nil, err
}
return nil, nil, err
}
func TestPoolsOf_RootfsAndVolumeMountpointsOnly(t *testing.T) {
c := &Checker{API: guest9201(), Runner: &seqRunner{}, Logger: quiet()}
pools, err := c.PoolsOf(context.Background(), 9201)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(pools, []Pool{{VG: "pve", Pool: "data"}}) {
t.Fatalf("pools = %v, want [pve/data] once (bind mounts and dir volumes are not thin)", pools)
}
}
// The consequence: an orphan seen on BOTH reads becomes a finding whose sentence names the pool and the id, and says
// nothing was removed. The only command ever run is the read-only helper with <vg> <pool>.
// RED-PROOF (P1/redproof-thincheck.txt): with Orphans() returning nil this test fails.
func TestCheck_OrphanOnBothReads_IsNamed(t *testing.T) {
rr := &seqRunner{outs: []string{orphanOut, orphanOut}}
c := &Checker{API: guest9201(), Runner: rr, Pause: 1, Logger: quiet()}
f := c.Check(context.Background(), 9201)
if len(f) != 1 || f[0].Pool.String() != "pve/data" || !reflect.DeepEqual(f[0].Orphans, []int{4}) {
t.Fatalf("findings = %+v", f)
}
want := [][]string{{HelperPath, "pve", "data"}, {HelperPath, "pve", "data"}}
if !reflect.DeepEqual(rr.calls, want) {
t.Fatalf("calls = %v, want only the read-only helper twice", rr.calls)
}
d := Describe(f)
for _, s := range []string{"thin pool pve/data holds device 4 that LVM does not know", "Felhom did not remove anything",
"whole-guest-backup-orphan-thin.md"} {
if !strings.Contains(d, s) {
t.Fatalf("Describe = %q, missing %q", d, s)
}
}
}
// An id that is gone on the second read (a volume mid-creation) is not reported; a healthy pool is read ONCE.
func TestCheck_TransientOrHealthy_NothingReported(t *testing.T) {
rr := &seqRunner{outs: []string{orphanOut, healthyOut}}
if f := (&Checker{API: guest9201(), Runner: rr, Pause: 1, Logger: quiet()}).Check(context.Background(), 9201); len(f) != 0 {
t.Fatalf("a transient id was reported: %+v", f)
}
rr = &seqRunner{outs: []string{healthyOut}}
if f := (&Checker{API: guest9201(), Runner: rr, Logger: quiet()}).Check(context.Background(), 9201); len(f) != 0 || len(rr.calls) != 1 {
t.Fatalf("healthy: findings %+v, calls %v", f, rr.calls)
}
if Describe(nil) != "" {
t.Fatal("no finding must describe as empty")
}
}
// The helper refusing (a metadata snapshot held by someone else, exit 3) or failing: no finding, no panic — the check
// fails open; it explains a failure, it never stops a backup.
func TestCheck_HelperRefuses_FailsOpen(t *testing.T) {
rr := &seqRunner{errs: []error{errors.New("exit status 3")}}
if f := (&Checker{API: guest9201(), Runner: rr, Logger: quiet()}).Check(context.Background(), 9201); len(f) != 0 {
t.Fatalf("findings = %+v", f)
}
}
// The root helper's own tests (configs/test_felhom_thin_check.py) run with the Go suite.
func TestRootHelperPythonSuite(t *testing.T) {
py, err := exec.LookPath("python3")
if err != nil {
t.Skip("python3 not found")
}
_, file, _, _ := runtime.Caller(0)
script := filepath.Join(filepath.Dir(file), "..", "..", "configs", "test_felhom_thin_check.py")
out, err := exec.Command(py, script).CombinedOutput()
if err != nil {
t.Fatalf("helper suite failed: %v\n%s", err, out)
}
}
+35
View File
@@ -99,6 +99,7 @@ type Manager struct {
resolver Resolver resolver Resolver
staleAfter time.Duration // handshake-age threshold for the re-resolve watchdog (doc 06 §4.2) staleAfter time.Duration // handshake-age threshold for the re-resolve watchdog (doc 06 §4.2)
rekeyAfter time.Duration // D30: how long a future-stamped handshake stays before the re-key (rekey.go)
mu sync.Mutex mu sync.Mutex
nextRegAt time.Time nextRegAt time.Time
@@ -111,6 +112,10 @@ type Manager struct {
resolveFailLogged bool // DNS-failure log throttle (reset on the next success) resolveFailLogged bool // DNS-failure log throttle (reset on the next success)
staleSameIPLogged bool // "stale but IP unchanged" log throttle (endpoint-down, not re-IP) staleSameIPLogged bool // "stale but IP unchanged" log throttle (endpoint-down, not re-IP)
// D30 (rekey.go): when the latest handshake was first seen stamped in the future, and the last re-key.
futureSince time.Time
lastRekey time.Time
// conv is the CACHED tunnel-convergence snapshot (v0.90.0, R-28 fast-tick source), refreshed at // conv is the CACHED tunnel-convergence snapshot (v0.90.0, R-28 fast-tick source), refreshed at
// the end of every Apply (the tunnel's own cadence) so the fast-tick reads it WITHOUT execing // the end of every Apply (the tunnel's own cadence) so the fast-tick reads it WITHOUT execing
// `wg`/`systemctl` per tick. wgBlockDesired = a wireguard block for THIS key is desired; // `wg`/`systemctl` per tick. wgBlockDesired = a wireguard block for THIS key is desired;
@@ -156,6 +161,7 @@ func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, log
now: time.Now, now: time.Now,
resolver: netResolver{}, resolver: netResolver{},
staleAfter: 180 * time.Second, staleAfter: 180 * time.Second,
rekeyAfter: rekeyAfterDefault,
} }
} }
@@ -600,6 +606,10 @@ func (m *Manager) Watchdog(ctx context.Context, fetched bool, block *hub.WireWir
if !fetched || block == nil || !m.lastResolvedIP.IsValid() { if !fetched || block == nil || !m.lastResolvedIP.IsValid() {
return // nothing applied to watch, or no desired block return // nothing applied to watch, or no desired block
} }
// D30: a handshake stamped in the future that stays → a new key on the existing peer path (rekey.go).
if m.clockAheadRekeyLocked(ctx, block) {
return
}
if !m.handshakeStaleLocked(ctx) { if !m.handshakeStaleLocked(ctx) {
return // healthy → no DNS, no action (the load-bearing negative) return // healthy → no DNS, no action (the load-bearing negative)
} }
@@ -661,6 +671,9 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
if age, ok := parseHandshakeAge(string(out), m.now()); ok { if age, ok := parseHandshakeAge(string(out), m.now()); ok {
st.LastHandshakeAgeS = &age st.LastHandshakeAgeS = &age
} }
if fut, ok := handshakeFuture(string(out), m.now()); ok {
st.HandshakeFutureS = &fut
}
} }
} }
if st.Pubkey == "" && !st.Registered && !st.Active { if st.Pubkey == "" && !st.Registered && !st.Active {
@@ -669,6 +682,28 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
return st return st
} }
// handshakeFuture (R-1204) reports how many seconds the latest handshake lies in the future, ok only when it does
// (by more than a minute — clock jitter is not the case). parseHandshakeAge clamps that case to 0 („just now"), which
// is what made a box whose clock had run ahead look healthy while ep0 refused its tunnel. Pinned by
// TestR1204_AFutureHandshakeIsReported.
func handshakeFuture(out string, now time.Time) (int64, bool) {
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
f := strings.Fields(line)
if len(f) != 2 {
continue
}
epoch, err := strconv.ParseInt(f[1], 10, 64)
if err != nil || epoch <= 0 {
continue
}
if fut := epoch - now.Unix(); fut > 60 {
return fut, true
}
return 0, false
}
return 0, false
}
// parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>"). // parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>").
// epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0). // epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0).
func parseHandshakeAge(out string, now time.Time) (int64, bool) { func parseHandshakeAge(out string, now time.Time) (int64, bool) {
+28
View File
@@ -696,3 +696,31 @@ func TestInitialResolveFailureNoTeardown(t *testing.T) {
t.Errorf("did not recover after DNS returned: %v", rr.calls) t.Errorf("did not recover after DNS returned: %v", rr.calls)
} }
} }
// R-1204: a box that ran once with its clock ahead keeps a handshake stamped in the FUTURE — ep0 then refuses every
// newer one. The age stays clamped at 0 (unchanged contract), and the new field says how far ahead it lies; a
// handshake in the past (or within a minute of now) never sets it.
// RED-PROOF: with Status not setting HandshakeFutureS the first assertion fails — seen failing 2026-10-22.
func TestR1204_AFutureHandshakeIsReported(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
ctx := context.Background()
m.Apply(ctx, false, nil)
*active = true
m.now = func() time.Time { return time.Unix(1783107115-2*24*3600, 0) } // the clock came back two days earlier
st := m.Status(ctx)
if st.HandshakeFutureS == nil || *st.HandshakeFutureS != 2*24*3600 {
t.Fatalf("handshake_future_s = %v, want %d", st.HandshakeFutureS, 2*24*3600)
}
if st.LastHandshakeAgeS == nil || *st.LastHandshakeAgeS != 0 {
t.Fatalf("the age stays clamped at 0: %v", st.LastHandshakeAgeS)
}
m.now = func() time.Time { return time.Unix(1783107115+100, 0) }
if st = m.Status(ctx); st.HandshakeFutureS != nil {
t.Fatalf("a past handshake must not set handshake_future_s: %v", *st.HandshakeFutureS)
}
m.now = func() time.Time { return time.Unix(1783107115-30, 0) } // 30 s of jitter is not the case
if st = m.Status(ctx); st.HandshakeFutureS != nil {
t.Fatalf("clock jitter must not set handshake_future_s")
}
}
+189
View File
@@ -0,0 +1,189 @@
package wgtunnel
import (
"context"
"crypto/rand"
"encoding/base64"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// D30 (operator 2026-10-22, R-1204): the tunnel heals itself after the box ran with its clock ahead.
//
// WireGuard refuses a handshake whose timestamp is not newer than the last one it accepted from that key (replay
// protection). A box whose clock ran ahead handshook with a future timestamp; ep0 keeps it, and once the clock is
// right again ep0 refuses every new handshake from that key — until the box's clock passes the stored stamp. The
// sign on the box: its own latest handshake stays stamped in the future (R-1204's handshake_future_s).
//
// The repair: when that sign stays for rekeyAfter, the agent makes a NEW key pair and registers the new public key on
// the existing path (POST /hosts/<id>/wg → the hub re-keys the peer IN PLACE: same /32, new key, generation bump,
// and the wgsync push of the FULL peer list to ep0 drops the old key and adds the new one). A new key is a new peer
// for ep0, with no stored timestamp — the next handshake is accepted. Nothing new runs on ep0.
//
// The previous private key is kept as private.key.prev (0600): the escrow ceremony may hold it (a host-loss DR that
// installs it registers it again, and the hub re-keys in place back to it — the same path).
// Pinned by TestD30_* (rekey_test.go).
const (
// rekeyAfter: how long the latest handshake must stay in the future before the re-key. A working tunnel
// handshakes every two minutes, which ends the sign within minutes of the clock being right.
rekeyAfterDefault = 15 * time.Minute
// rekeyMinGap bounds the re-key to once an hour, whatever the clock does.
rekeyMinGap = time.Hour
prevKeyName = "private.key.prev"
)
// clockAheadRekeyLocked watches for a handshake stamped in the future and re-keys when it stays. It reports whether
// it re-keyed (the caller then skips the rest of its tick). Caller holds m.mu.
func (m *Manager) clockAheadRekeyLocked(ctx context.Context, block *hub.WireWireguard) bool {
if m.keyBroken || !m.isActive(ctx) {
m.futureSince = time.Time{}
return false
}
out, _, err := m.runner.Run(ctx, "wg", "show", Iface, "latest-handshakes")
if err != nil {
return false
}
now := m.now()
fut, ok := handshakeFuture(string(out), now)
if !ok {
if !m.futureSince.IsZero() {
m.logger.Info("wgtunnel: the handshake is no longer stamped in the future — no re-key needed")
}
m.futureSince = time.Time{}
return false
}
if m.futureSince.IsZero() {
m.futureSince = now
m.logger.Warn("wgtunnel: the latest handshake is stamped in the future — this box's clock ran ahead, and the endpoint refuses its newer handshakes; re-keying if it stays",
"future_s", fut, "rekey_after", m.rekeyAfter.String())
return false
}
// A clock that jumps backwards must not stretch either wait (commit security review 2026-10-23 night): a start or a
// last re-key stamped AFTER now is from a wrong clock and is dropped — this whole feature exists because clocks jump.
if now.Before(m.futureSince) {
m.futureSince = now
}
if now.Sub(m.futureSince) < m.rekeyAfter {
return false
}
if m.lastRekey.IsZero() {
m.lastRekey = m.loadLastRekey()
}
if now.Before(m.lastRekey) {
m.lastRekey = time.Time{}
}
if !m.lastRekey.IsZero() && now.Sub(m.lastRekey) < rekeyMinGap {
return false
}
pub, _, err := EnsureKey(m.stateDir)
if err != nil || block.Pubkey != pub {
return false // a re-register is already in flight, or the key is not readable — never re-key over that
}
newPub, err := rotateKey(m.stateDir)
if err != nil {
m.logger.Error("wgtunnel: re-key after a future handshake FAILED — the old key stays", "err", err)
m.lastRekey = now // not again before rekeyMinGap
m.saveLastRekey(now)
return false
}
m.lastRekey, m.futureSince = now, time.Time{}
m.saveLastRekey(now)
m.logger.Warn("wgtunnel: RE-KEYED after a handshake stamped in the future (D30) — registering the new key; the hub re-keys the peer in place and the endpoint takes the new key",
"old_pubkey", pub, "new_pubkey", newPub, "future_s", fut)
m.nextRegAt = time.Time{} // register now, not after an older backoff
m.registerLocked(ctx, newPub)
return true
}
// rotateKey makes a new key pair, never leaving the box without a readable key (review 2026-10-23 night, N6): the new
// key is written and fsynced to a temp file; the current key gets a second name by a HARD LINK (the escrow may hold
// it — `private.key.prev` keeps the first such key, later ones `private.key.prev.<unix>`, the newest three kept, N7);
// then the temp file is renamed OVER the current name (atomic) and the directory fsynced. At every instant
// `private.key` is a complete key. Returns the new PUBLIC key only.
func rotateKey(stateDir string) (string, error) {
dir := filepath.Join(stateDir, "wg")
cur := filepath.Join(dir, keyFileName)
if _, err := os.Stat(cur); err != nil {
return "", fmt.Errorf("no current key: %w", err)
}
priv := make([]byte, 32)
if _, err := rand.Read(priv); err != nil {
return "", fmt.Errorf("entropy: %w", err)
}
clampPrivateKey(priv)
pub, err := derivePublic(priv)
if err != nil {
return "", err
}
tmp := filepath.Join(dir, keyFileName+".new")
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
return "", fmt.Errorf("writing the new key: %w", err)
}
_, werr := f.Write([]byte(base64.StdEncoding.EncodeToString(priv) + "\n"))
serr := f.Sync()
cerr := f.Close()
if werr != nil || serr != nil || cerr != nil {
os.Remove(tmp)
return "", fmt.Errorf("writing the new key: %v %v %v", werr, serr, cerr)
}
prev := filepath.Join(dir, prevKeyName)
if _, err := os.Lstat(prev); err == nil {
prev = filepath.Join(dir, fmt.Sprintf("%s.%d", prevKeyName, rekeyNow().Unix()))
}
if err := os.Link(cur, prev); err != nil {
os.Remove(tmp)
return "", fmt.Errorf("keeping the old key: %w", err)
}
if err := os.Rename(tmp, cur); err != nil {
os.Remove(tmp)
return "", fmt.Errorf("installing the new key: %w", err)
}
if d, err := os.Open(dir); err == nil {
d.Sync()
d.Close()
}
prunePrevKeys(dir, 3)
return pub, nil
}
// rekeyNow is the clock for the kept-key names (a seam).
var rekeyNow = time.Now
// prunePrevKeys keeps the newest `keep` of the `private.key.prev.<unix>` files (`private.key.prev` itself always stays).
func prunePrevKeys(dir string, keep int) {
m, _ := filepath.Glob(filepath.Join(dir, prevKeyName+".*"))
if len(m) <= keep {
return
}
sort.Strings(m) // the same digit count until 2286: lexical = time order
for _, p := range m[:len(m)-keep] {
os.Remove(p)
}
}
// lastRekeyFile records the last re-key's time across agent restarts (N7: the once-an-hour bound was in memory only).
const lastRekeyFile = "last-rekey"
func (m *Manager) loadLastRekey() time.Time {
b, err := os.ReadFile(filepath.Join(m.wgDir(), lastRekeyFile))
if err != nil {
return time.Time{}
}
var u int64
if _, err := fmt.Sscanf(strings.TrimSpace(string(b)), "%d", &u); err != nil {
return time.Time{}
}
return time.Unix(u, 0)
}
func (m *Manager) saveLastRekey(t time.Time) {
_ = os.WriteFile(filepath.Join(m.wgDir(), lastRekeyFile), []byte(fmt.Sprintf("%d\n", t.Unix())), 0o600)
}
+227
View File
@@ -0,0 +1,227 @@
package wgtunnel
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// D30 (R-1204): a handshake stamped in the future that STAYS for rekeyAfter → a new key pair, registered on the
// existing peer path (the hub re-keys in place; its full-list push to ep0 swaps the key). The fake `wg` reports a
// handshake at 1783107115; a clock three hours behind that sees it three hours in the future.
// RED-PROOF (2026-10-22 night): with the clockAheadRekeyLocked call removed from Watchdog,
// TestD30_AFutureHandshakeThatStaysReKeys and …AFailedRegistration… fail (no new key, no registration); with the
// rekeyAfter wait removed, …ThatStaysReKeys fails (a re-key at 10 min); with the isActive check removed,
// TestD30_NoReKeyWhileTheTunnelIsDown fails.
const hs = 1783107115
func at(m *Manager, unix int64) { m.now = func() time.Time { return time.Unix(unix, 0) } }
func readKey(t *testing.T, m *Manager, name string) string {
t.Helper()
b, err := os.ReadFile(filepath.Join(m.stateDir, "wg", name))
if err != nil {
t.Fatal(err)
}
return strings.TrimSpace(string(b))
}
func TestD30_AFutureHandshakeThatStaysReKeys(t *testing.T) {
fh := &fakeHub{}
m, _, active, logs := testManager(t, fh)
oldPub := establishWG(t, m, active)
oldPriv := readKey(t, m, keyFileName)
ctx := context.Background()
calls := fh.count()
t0 := int64(hs - 3*3600) // the clock is right again; the last handshake (made while it ran ahead) is 3 h ahead
at(m, t0)
m.Watchdog(ctx, true, testBlock(oldPub))
at(m, t0+10*60)
m.Watchdog(ctx, true, testBlock(oldPub))
if fh.count() != calls || localPub(t, m) != oldPub {
t.Fatal("no re-key before the future stamp has stayed rekeyAfter")
}
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(oldPub))
newPub := localPub(t, m)
if newPub == oldPub {
t.Fatal("a future stamp that stayed 16 min must give a new key")
}
if fh.count() != calls+1 {
t.Fatalf("the new key must be registered on the existing peer path (RegisterWG calls %d → %d)", calls, fh.count())
}
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
t.Fatalf("the registration marker must carry the new key: %+v", mk)
}
if readKey(t, m, prevKeyName) != oldPriv {
t.Fatal("the old private key must be kept as private.key.prev (the escrow may hold it)")
}
if strings.Contains(logs.String(), oldPriv) || strings.Contains(logs.String(), readKey(t, m, keyFileName)) {
t.Fatal("a private key reached the log")
}
// The served block still names the OLD key until the next fetch: no second re-key over that.
at(m, t0+17*60)
m.Watchdog(ctx, true, testBlock(oldPub))
if localPub(t, m) != newPub {
t.Fatal("a second re-key ran before the hub served the new key")
}
// The hub serves the new key → the tunnel is applied with the NEW private key.
m.Apply(ctx, true, testBlock(newPub))
conf, err := os.ReadFile(m.stagedConfPath())
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(conf), readKey(t, m, keyFileName)) || strings.Contains(string(conf), oldPriv) {
t.Fatal("the applied tunnel must use the new private key, not the old one")
}
}
func TestD30_AShortFutureSpellDoesNotReKey(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(pub))
at(m, hs+100) // a handshake now lies in the past: the sign ended (a working tunnel)
m.Watchdog(ctx, true, testBlock(pub))
at(m, t0+20*60) // in the future again: a NEW window starts, it does not continue the old one
m.Watchdog(ctx, true, testBlock(pub))
if localPub(t, m) != pub {
t.Fatal("a future stamp that did not STAY for rekeyAfter must not re-key")
}
}
func TestD30_NoReKeyWhileTheTunnelIsDown(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
*active = false
ctx := context.Background()
t0 := int64(hs - 3*3600)
for _, d := range []int64{0, 20 * 60, 40 * 60} {
at(m, t0+d)
m.Watchdog(ctx, true, testBlock(pub))
}
if localPub(t, m) != pub {
t.Fatal("no re-key while the tunnel unit is not active")
}
}
func TestD30_AFailedRegistrationIsRetriedByTheExistingPath(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
oldPub := establishWG(t, m, active)
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(oldPub))
fh.mu.Lock()
fh.err = errors.New("hub down")
fh.mu.Unlock()
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(oldPub))
newPub := localPub(t, m)
if newPub == oldPub {
t.Fatal("the key rotates before the registration")
}
if mk := m.loadMarker(); mk == nil || mk.Pubkey != oldPub {
t.Fatal("a failed registration leaves the marker on the old key")
}
fh.mu.Lock()
fh.err = nil
fh.mu.Unlock()
at(m, t0+30*60) // past the backoff: Apply sees the served (old) key ≠ the local key → re-key-in-place registers
m.Apply(ctx, true, testBlock(oldPub))
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
t.Fatalf("the existing re-key-in-place path must register the new key: %+v", mk)
}
}
// Review 2026-10-23 night. N7: a second rotation keeps the FIRST old key as `private.key.prev` (the escrow may hold it)
// and the later one beside it; N6: the current key is replaced by a rename OVER it, so `private.key` is never missing.
// RED-PROOF: with the „prev exists → a dated name" branch removed, TwoRotations fails (the first key overwritten) —
// seen failing 2026-10-23 night.
func TestD30_TwoRotationsKeepTheFirstOldKey(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
establishWG(t, m, active)
first := readKey(t, m, keyFileName)
old := rekeyNow
defer func() { rekeyNow = old }()
rekeyNow = func() time.Time { return time.Unix(hs, 0) }
if _, err := rotateKey(m.stateDir); err != nil {
t.Fatal(err)
}
second := readKey(t, m, keyFileName)
rekeyNow = func() time.Time { return time.Unix(hs+7200, 0) }
if _, err := rotateKey(m.stateDir); err != nil {
t.Fatal(err)
}
if readKey(t, m, prevKeyName) != first {
t.Fatal("private.key.prev must keep the FIRST old key")
}
if readKey(t, m, fmt.Sprintf("%s.%d", prevKeyName, hs+7200)) != second {
t.Fatal("the second old key must be kept beside it")
}
if cur := readKey(t, m, keyFileName); cur == first || cur == second {
t.Fatal("the current key must be new")
}
}
// N7: the once-an-hour bound survives an agent restart (it was in memory only). RED-PROOF: with loadLastRekey removed
// from clockAheadRekeyLocked, this fails — seen failing 2026-10-23 night.
func TestD30_TheHourBoundSurvivesARestart(t *testing.T) {
fh := &fakeHub{}
m, rr, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(pub))
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(pub))
newPub := localPub(t, m)
if newPub == pub {
t.Fatal("fixture: the first re-key must happen")
}
// The agent restarts: a NEW manager on the same state; the served block now names the new key; the sign is back.
m2 := NewManager(rr, fh, m.stateDir, m.logger)
m2.isActive, m2.wgPresent, m2.resolver = m.isActive, m.wgPresent, m.resolver
m2.Apply(ctx, true, testBlock(newPub))
at(m2, t0+20*60)
m2.Watchdog(ctx, true, testBlock(newPub))
at(m2, t0+40*60)
m2.Watchdog(ctx, true, testBlock(newPub))
if localPub(t, m2) != newPub {
t.Fatal("a restart must not reset the once-an-hour bound")
}
_ = active
}
// Commit security review 2026-10-23 night: a last re-key saved while the clock ran ahead (a later date than now) must
// not block the next re-key until that date. RED-PROOF: with the „lastRekey after now → dropped" guard removed, this
// fails — seen failing 2026-10-23 night.
func TestD30_ALastReKeyFromTheFutureDoesNotBlock(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
m.saveLastRekey(time.Unix(hs+2*365*86400, 0)) // written while the clock was two years ahead
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(pub))
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(pub))
if localPub(t, m) == pub {
t.Fatal("a last re-key dated in the future must not block the re-key")
}
}
+22
View File
@@ -15,6 +15,12 @@ Gates (all must pass; **non-zero exit on any failure**):
or because someone is offline on a train. CI runs the FULL set for exactly this or because someone is offline on a train. CI runs the FULL set for exactly this
reason: it is the machine that can afford a network check, and it is the half reason: it is the machine that can afford a network check, and it is the half
that emails when something is wrong. that emails when something is wrong.
… security-scan govulncheck (called vulnerabilities) + gitleaks on the pushed commits, WARN-FIRST: it lists,
it never refuses (R-986). Shared: felhom.eu/scripts/security_scan_gate.py. In the fast set
although it reads vuln.go.dev — offline it says NOT CHECKED, so it cannot stop a push.
… gate-decoys the decoy suite (test_gate_decoys.py): every gate here still convicts its decoys
(R-1116). ~20 s in the workspace, so the full run only: not the hook, but CI
runs the full set on every push. Scratch clones and a local fake Gitea only.
WHY THIS FILE EXISTS, WITH ONE GATE (2026-08-02, R-29 leg (b)). WHY THIS FILE EXISTS, WITH ONE GATE (2026-08-02, R-29 leg (b)).
@@ -46,6 +52,9 @@ SHARED_INSTRUCTIONS = os.path.join(
# R-389 — shared, like the two above: it lives in felhom.eu/scripts/ and is never copied. # R-389 — shared, like the two above: it lives in felhom.eu/scripts/ and is never copied.
SHARED_OBSERVATIONS = os.path.join( SHARED_OBSERVATIONS = os.path.join(
os.path.dirname(ROOT), "felhom.eu", "scripts", "observations_gate.py") os.path.dirname(ROOT), "felhom.eu", "scripts", "observations_gate.py")
# R-986 — shared too: the dependency and secret scanner, warn-first.
SHARED_SECURITY = os.path.join(
os.path.dirname(ROOT), "felhom.eu", "scripts", "security_scan_gate.py")
# (label, absolute script path, args, fast) # (label, absolute script path, args, fast)
GATES = [ GATES = [
@@ -58,6 +67,19 @@ GATES = [
("release-complete", os.path.join(ROOT, "scripts", "check-release-complete.py"), [], True), ("release-complete", os.path.join(ROOT, "scripts", "check-release-complete.py"), [], True),
# R-389 — a REPORT.md observation with no register row behind it. Fast: stdlib file reads. # R-389 — a REPORT.md observation with no register row behind it. Fast: stdlib file reads.
("observations", SHARED_OBSERVATIONS, [ROOT], True), ("observations", SHARED_OBSERVATIONS, [ROOT], True),
# R-1116 (2026-10-17) — the gates' OWN decoy suite. It was in no runner, the shape that let the catalog's
# R-1115 fail unseen for a night. Full run only: ~20 s in the workspace (measured, 30 cases; ~5 s in a
# CI-shaped clone, where the shared gates have fewer siblings to search), too slow for the hook — and CI
# runs the FULL set, so every push is still judged by it. It plants only in scratch clones and asks a fake
# Gitea on 127.0.0.1 (never the network, never the real tree); it does not call this runner (no recursion).
("gate-decoys", os.path.join(ROOT, "scripts", "test_gate_decoys.py"), [], False),
# R-1160 (2026-10-19) — every Go file is gofmt-clean; the controller's gofmt_gate.py shape. Fast: one gofmt -l.
# Where the runner has no Go it says NOT CHECKED out loud and passes; the pre-push hook (with Go) enforces it.
("gofmt", os.path.join(ROOT, "scripts", "gofmt_gate.py"), [], True),
# R-986 (2026-10-19) — govulncheck + gitleaks, WARN-FIRST (exit 0 whatever it finds). Fast although it reads
# vuln.go.dev: it cannot refuse, so a network blink is a NOT CHECKED line. ~4 s here. CI: gitleaks fetched pinned
# (FELHOM_SCAN_FETCH=1), no Go, so govulncheck says NOT CHECKED there.
("security-scan", SHARED_SECURITY, [ROOT], True),
] ]
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"} VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""gofmt_gate.py — every Go file in this module is gofmt-clean (R-1160; the controller's R-454 gate, same shape).
MEASURED 2026-10-19: 14 agent files were not gofmt-clean and nothing noticed (fixed by one formatting-only commit,
1edfd8e). The controller's history below is why a gate and not a one-off clean-up.
MEASURED 2026-09-02: five internal/web test files had been gofmt-unclean for an unknown length of
time and nothing noticed — `go vet` does not check formatting and no gate did. By 2026-10-05 the five
were clean and TWELVE others were not: the count only grows while nothing looks, and every `gofmt -l`
someone runs by hand is then noise that hides a real one.
Run from the repo root: python3 scripts/gofmt_gate.py
Exit 0 clean · 1 convicted (lists the files) · 2 inconclusive (gofmt not found — never a pass).
gofmt is the Go toolchain's own formatter (not a shell utility), found on PATH or under
`go env GOROOT`/bin. Decoy: scripts/test_gate_decoys.py ("gofmt"), in a scratch copy of this script.
"""
import os
import shutil
import subprocess
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ROOTS = ["cmd", "internal"]
def find_gofmt():
p = shutil.which("gofmt")
if p:
return p
go = shutil.which("go")
if go:
try:
root = subprocess.check_output([go, "env", "GOROOT"], text=True).strip()
cand = os.path.join(root, "bin", "gofmt")
if os.path.isfile(cand):
return cand
except Exception:
pass
return None
def in_ci():
"""True on the Gitea Actions runner (act sets GITHUB_ACTIONS, Gitea sets GITEA_ACTIONS)."""
return os.environ.get("GITEA_ACTIONS") == "true" or os.environ.get("GITHUB_ACTIONS") == "true"
def main():
gofmt = find_gofmt()
if not gofmt:
# The CI runner may have no Go toolchain (felhom.eu CLAUDE.md, R-887 note). There the gate
# says so OUT LOUD and passes: the pre-push hook on the developer's machine, where Go always exists, is where
# formatting is enforced. Anywhere else a missing gofmt stays INCONCLUSIVE — never a silent pass.
# Pinned by the "gofmt: CI without Go" and "gofmt: dev without Go" decoys in test_gate_decoys.py.
if in_ci():
print("gofmt gate NOT CHECKED in CI: no Go toolchain on the runner — the pre-push hook (with Go) enforces it")
return 0
print("gofmt gate INCONCLUSIVE: gofmt not found on PATH or under `go env GOROOT`/bin")
return 2
roots = [r for r in ROOTS if os.path.isdir(os.path.join(ROOT, r))]
if not roots:
print("gofmt gate INCONCLUSIVE: none of %s exists under %s" % (ROOTS, ROOT))
return 2
p = subprocess.run([gofmt, "-l"] + roots, cwd=ROOT, capture_output=True, text=True)
if p.returncode != 0:
print("gofmt gate INCONCLUSIVE: gofmt exited %d: %s" % (p.returncode, p.stderr.strip()[:400]))
return 2
bad = [l for l in p.stdout.splitlines() if l.strip()]
if bad:
for f in bad:
print(" not gofmt-clean: %s" % f)
print("GOFMT GATE FAILED: %d file(s) — run `gofmt -w <file>` (formatting only, no behaviour change)" % len(bad))
return 1
print("gofmt gate OK — every Go file under %s is gofmt-clean" % ", ".join(roots))
return 0
if __name__ == "__main__":
sys.exit(main())
+4
View File
@@ -69,6 +69,10 @@ fi
existing="$(curl -fsS -o /dev/null -w '%{http_code}' \ existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)" "$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
[[ "$existing" != "200" ]] || die "version $VERSION is ALREADY PUBLISHED — bump the version instead" [[ "$existing" != "200" ]] || die "version $VERSION is ALREADY PUBLISHED — bump the version instead"
# R-1098 (2026-10-16): only a clean 404 proves the version is new. Until now a registry that could not be reached
# (curl prints 000) or answered anything else passed this check — „I could not ask" is not that proof.
# Test: scripts/test_release_push_once.sh.
[[ "$existing" == "404" ]] || die "could not establish whether $VERSION is published (registry answered HTTP '${existing:-none}') — refusing"
# ── 3. Build ──────────────────────────────────────────────────────────────────────────────────── # ── 3. Build ────────────────────────────────────────────────────────────────────────────────────
BIN="$(mktemp -t felhom-agent-XXXXXX)" BIN="$(mktemp -t felhom-agent-XXXXXX)"
+14
View File
@@ -57,6 +57,20 @@ class AgentGatesTest(unittest.TestCase):
finally: finally:
shutil.rmtree(tmp, ignore_errors=True) shutil.rmtree(tmp, ignore_errors=True)
def test_decoy_suite_is_a_full_run_gate(self):
"""R-1116: the decoy suite is a gate of the FULL run (which CI runs on every push), and --fast
names it as skipped (~20 s, too slow for the hook)."""
sys.dont_write_bytecode = True # no __pycache__ left in scripts/ (not gitignored here)
sys.path.insert(0, os.path.join(ROOT, "scripts"))
import agent_gates as mod
rows = [g for g in mod.GATES if g[0] == "gate-decoys"]
self.assertEqual(len(rows), 1, "gate-decoys is not registered exactly once")
self.assertEqual(os.path.basename(rows[0][1]), "test_gate_decoys.py")
self.assertFalse(rows[0][3], "gate-decoys must not be --fast")
self.assertEqual([g[0] for g in mod.GATES if not g[3]], ["published", "gate-decoys"])
self.assertIn("--fast SKIPPED (deliberate periodic runs, never in a hook): published, gate-decoys",
self.out)
def test_unknown_argument_is_rejected(self): def test_unknown_argument_is_rejected(self):
p = subprocess.run([sys.executable, ENTRY, "--nope"], cwd=ROOT, p = subprocess.run([sys.executable, ENTRY, "--nope"], cwd=ROOT,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT) stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+53
View File
@@ -66,6 +66,17 @@ COVERS = {
"inside an HTML comment (R-426)", "inside an HTML comment (R-426)",
"observations": "R-419 in THIS repo's REPORT.md: an Observations note SAYING it carries no marker - vs the " "observations": "R-419 in THIS repo's REPORT.md: an Observations note SAYING it carries no marker - vs the "
"two genuine markers (R-426)", "two genuine markers (R-426)",
"gofmt": "R-1160: an unformatted .go file planted under internal/ of a SCRATCH copy is convicted by name - vs the "
"same tree formatted (passes); with no gofmt reachable, CI passes OUT LOUD and a dev machine is "
"INCONCLUSIVE, never a silent pass",
"security-scan": "R-986: the shared gate's own decoys (felhom.eu scripts/test_security_scan_gate.py, 8 cases in "
"SCRATCH git repos: a planted token listed by rule and file, its value never printed; a second "
"secret on another line of an accepted file still listed; a called govulncheck finding listed, "
"an imported-only one not; no tool is NOT CHECKED out loud), run from here; plus the gate on THIS "
"repo's tree: exit 0 and its summary line (warn-first - the listing is the fact)",
"gate-decoys": "this suite, run as a gate of agent_gates.py's full run, which CI runs on every push (R-1116): "
"every case above is a decoy of its own gate, and ONE case that stops convicting turns the run "
"red - red-proved 2026-10-17 by flipping a reuse-refs FACT case to expect rc 0",
} }
fails = [] fails = []
@@ -288,6 +299,32 @@ def release_cases(base, ws):
# ── the shared felhom.eu scripts, against THIS repo's inputs ───────────────────────────────────── # ── the shared felhom.eu scripts, against THIS repo's inputs ─────────────────────────────────────
def gofmt_cases(ws):
"""R-1160: the gofmt gate in a scratch root (its ROOT is its own grandparent directory), never the real tree."""
root = os.path.join(ws, "gofmt-root")
os.makedirs(os.path.join(root, "scripts"))
os.makedirs(os.path.join(root, "internal", "x"))
shutil.copy(os.path.join(ROOT, "scripts", "gofmt_gate.py"), os.path.join(root, "scripts", "gofmt_gate.py"))
gate = os.path.join(root, "scripts", "gofmt_gate.py")
planted = os.path.join(root, "internal", "x", "zz_gofmt_decoy.go")
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F( ) int {return 1}\n")
if shutil.which("gofmt") or shutil.which("go"):
rc, out = run([sys.executable, gate], root)
report("gofmt: FACT: an unformatted file under internal/", rc, out, 1, ("zz_gofmt_decoy.go", "GOFMT GATE FAILED"))
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F() int { return 1 }\n")
rc, out = run([sys.executable, gate], root)
report("gofmt: GENUINE: the same file formatted", rc, out, 0, ("gofmt gate OK",))
else:
print(" -- gofmt: not runnable here (no Go toolchain) - the two cases below pin that mode")
for ci, want_rc, want, name in ((True, 0, "NOT CHECKED in CI", "gofmt: CI without Go"),
(False, 2, "INCONCLUSIVE", "gofmt: dev without Go")):
env = {"PATH": os.path.join(ws, "empty-path"), "HOME": ws}
if ci:
env["GITEA_ACTIONS"] = "true"
rc, out = run([sys.executable, gate], root, env=env)
report(name, rc, out, want_rc, (want,))
def shared_cases(ws): def shared_cases(ws):
"""A scratch WORKSPACE: a clone of this repo beside symlinks to the siblings, because the shared """A scratch WORKSPACE: a clone of this repo beside symlinks to the siblings, because the shared
scripts reach across (REUSE.md cites hub paths; instructions_gate reads the workspace CLAUDE.md).""" scripts reach across (REUSE.md cites hub paths; instructions_gate reads the workspace CLAUDE.md)."""
@@ -340,6 +377,20 @@ def shared_cases(ws):
u"the same minute.**\n", 0) u"the same minute.**\n", 0)
def security_cases():
"""R-986: the shared scanner's decoys (they build their own scratch repos), then the gate on this repo."""
suite = os.path.join(SHARED, "test_security_scan_gate.py")
gate = os.path.join(SHARED, "security_scan_gate.py")
for f in (suite, gate):
if not os.path.isfile(f):
fails.append("shared %s is MISSING beside this clone - a failure, never a skip" % f)
return
rc, out = run([sys.executable, suite], ROOT)
report("security-scan: the shared gate's 8 decoys", rc, out, 0, ("all 8 cases passed",))
rc, out = run([sys.executable, gate, ROOT], ROOT)
report("security-scan: GENUINE: this repo, warn-first", rc, out, 0, ("security-scan:", "govulncheck"))
def main(): def main():
srv = Server(("127.0.0.1", 0), Handler) srv = Server(("127.0.0.1", 0), Handler)
threading.Thread(target=srv.serve_forever, daemon=True).start() threading.Thread(target=srv.serve_forever, daemon=True).start()
@@ -350,6 +401,8 @@ def main():
published_cases(base) published_cases(base)
release_cases(base, ws) release_cases(base, ws)
shared_cases(ws) shared_cases(ws)
gofmt_cases(ws)
security_cases()
finally: finally:
srv.shutdown() srv.shutdown()
srv.server_close() srv.server_close()
+50
View File
@@ -0,0 +1,50 @@
#!/usr/bin/env bash
# test_release_push_once.sh — release-agent.sh releases a version only when the registry says it is NEW (R-1098).
#
# The REAL release-agent.sh runs from a copy inside a scratch git repo (so its tag check reads scratch tags), with
# RELEASE_ALLOW_DIRTY=1 (the clean-tree gate has its own history) and a fake `curl` and `go` first on PATH: curl answers
# the package probe as FAKE_MODE says, and `go` stops the run right after the check, so nothing is built or published.
# published (200) -> refused, "ALREADY PUBLISHED"
# absent (404) -> passes the check, reaches the build
# unreachable (000) -> refused, "could not establish"
# server error (500) -> refused, "could not establish"
# the tag already exists -> refused, "already exists"
# Run: bash scripts/test_release_push_once.sh (exit 0 all cases right · 1 a case wrong)
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
WS="$(mktemp -d)"; trap 'rm -rf "${WS:?}"' EXIT
fails=0; ran=0
mkdir -p "$WS/bin" "$WS/repo/scripts"
cp "$HERE/release-agent.sh" "$WS/repo/scripts/release-agent.sh"
(cd "$WS/repo" && git init -q -b main && git -c user.name=t -c user.email=t@t commit -q --allow-empty -m init)
cat > "$WS/bin/curl" <<'FAKE'
#!/bin/bash
case "${FAKE_MODE}" in
published) printf 200; exit 0 ;;
absent) printf 404; exit 22 ;;
down) printf 000; exit 7 ;;
error) printf 500; exit 22 ;;
esac
FAKE
printf '#!/bin/sh\necho "FAKE-GO-REACHED $*"; exit 1\n' > "$WS/bin/go"
chmod +x "$WS/bin/curl" "$WS/bin/go"
run() { # name, mode, expected rc, required fragment, forbidden fragment
local out rc
out=$(cd "$WS/repo" && FAKE_MODE="$2" RELEASE_ALLOW_DIRTY=1 GITEA_USER=u GITEA_TOKEN=t PATH="$WS/bin:$PATH" \
bash scripts/release-agent.sh 9.9.9 2>&1); rc=$?
ran=$((ran+1))
if [[ "$rc" != "$3" ]] || ! grep -qi -- "$4" <<<"$out" || { [[ -n "$5" ]] && grep -qi -- "$5" <<<"$out"; }; then
fails=$((fails+1)); echo " FAIL $1 (exit $rc)"; echo "$out" | tail -4 | sed 's/^/ /'; return
fi
echo " ok $1"
}
echo "release-agent.sh — a version is released once, and only when the registry says it is new"
run "published (200) -> refused" published 1 "ALREADY PUBLISHED" "FAKE-GO-REACHED"
run "absent (404) -> proceeds" absent 1 "FAKE-GO-REACHED" "could not establish"
run "unreachable (000) -> refused" down 1 "could not establish" "FAKE-GO-REACHED"
run "server error (500) -> refused" error 1 "could not establish" "FAKE-GO-REACHED"
(cd "$WS/repo" && git tag v9.9.9)
run "the tag exists -> refused" absent 1 "already exists" "FAKE-GO-REACHED"
echo
if [[ $fails -eq 0 ]]; then echo "all $ran cases behaved"; exit 0; fi
echo "$fails of $ran cases FAILED"; exit 1