Compare commits

...

30 Commits

Author SHA1 Message Date
admin e803ad2b91 night 2026-10-22: REPORT; CHANGELOG corrected — a push of the installer to main publishes nothing (R-110, tag installer-v1.35.0); D4 stays a patch because both halves need a release
gates / gates (push) Successful in 2m24s
2026-10-22 18:13:35 +02:00
admin 7651461be2 D4 (R-982) moved out of main into a ready patch — its installer half would publish the installer (forbidden tonight); R-1204 stays; unreleased
gates / gates (push) Successful in 2m45s
2026-10-22 17:10:46 +02:00
admin a0c88e5183 R-982 (D4): root SSH from the LAN by key only — a bundle drop-in, checked and reloaded by felhom-os-apply; R-1204: the heartbeat reports a future-stamped tunnel handshake; unreleased
gates / gates (push) Successful in 2m23s
2026-10-22 16:58:45 +02:00
admin 2d5840ba11 v0.162.0: released (operator yes 2026-10-22), delivered to the 3 test boxes; CHANGELOG + REPORT
gates / gates (push) Successful in 2m10s
2026-10-22 10:39:06 +02:00
admin b310773dbf REPORT: the night of 2026-10-21 (host swap report, R-1194), unreleased
gates / gates (push) Successful in 2m7s
2026-10-22 00:06:21 +02:00
admin 0da792e670 R-1192 follow-up: report host swap vs the guests' swap allowance (log once, hub alarm, local API); R-1194: no OOB stanza before the first felhom-sshd pass; unreleased
gates / gates (push) Successful in 1m56s
2026-10-21 21:29:07 +02:00
admin 286a7e3ecf R-1192: every guest may swap as much as its RAM (bring-up, RAM resize, once at start) — the measured 8 GB freeze; unreleased
gates / gates (push) Successful in 1m39s
2026-10-21 15:43:18 +02:00
admin 4e1621c650 CHANGELOG + CONTEXT + REPORT: v0.161.0 released (operator yes) and delivered to demo-hp, Tester 1, demo-felhom; not vouched; R-1083 left out
gates / gates (push) Successful in 1m27s
2026-10-20 09:07:30 +02:00
admin 51521230c9 REPORT: the night's agent fixes (R-921, R-1070, R-1010), not released
gates / gates (push) Successful in 1m25s
2026-10-19 21:47:40 +02:00
admin 84ef5818d5 security-scan gate (shared felhom.eu/scripts/security_scan_gate.py) in the fast set and CI, warn-first, with the accepted-hits file and decoys (R-986)
gates / gates (push) Successful in 1m30s
2026-10-19 21:11:46 +02:00
admin 56aabc729c Unreleased: host_busy on GET /backup/status (R-921), a missing tunnel container is 'absent' and not-applicable after an OS update (R-1070), the janitor forces a half-removed vzdump snapshot once (R-1010)
gates / gates (push) Successful in 58s
2026-10-19 19:52:37 +02:00
admin 0cbd500154 gofmt gate in agent_gates.py with its decoys (R-1160); REPORT for the day
gates / gates (push) Successful in 49s
2026-10-19 10:26:27 +02:00
admin 1edfd8e331 gofmt -w: 14 files, formatting only, no behaviour change (R-1160) 2026-10-19 10:24:32 +02:00
admin 0bc0bc5da0 CONTEXT + REPORT: v0.160.0 released, on the three test hosts, not vouched
gates / gates (push) Successful in 48s
2026-10-19 05:41:34 +02:00
admin 45458196d1 Two texts made true after 0.160.0: the ceremony log line names the confirm order; the R-973 comments name whole_guest_backup_refused (no behaviour change)
gates / gates (push) Successful in 52s
2026-10-19 05:25:58 +02:00
admin 124164db73 CHANGELOG v0.160.0 (released: R-978, R-973, R-1093, R-1149, R-1151)
gates / gates (push) Successful in 46s
2026-10-19 03:22:00 +02:00
admin c11e3e2a87 The new escrow package reaches the hub only after the household typed two words of the code back: POST /escrow/ceremony/confirm; an old controller keeps the claim order (R-1151, unreleased)
gates / gates (push) Successful in 48s
2026-10-19 03:03:37 +02:00
admin 1b6e31675b The local whole-guest tier keeps one restore point per day for 3 days: keep-daily instead of keep-last (R-1149, operator D17, unreleased)
gates / gates (push) Successful in 47s
2026-10-19 02:59:34 +02:00
admin 1693da4d1a REPORT: the three unreleased fixes (R-1093, R-978, R-973) with the R-1093 loop-device check on demo-hp
gates / gates (push) Successful in 47s
2026-10-18 21:48:03 +02:00
admin 79ba884859 CI workflow: the failure mail links the public run page, by run id (R-1135)
gates / gates (push) Successful in 49s
The alarm step built its link from GITHUB_SERVER_URL, which on this runner
is the in-cluster Gitea address, and from the run NUMBER, which Gitea 1.26
reads as a run id first (felhom.eu /actions/runs/1116 opened run id 1116 =
run number 613). Now https://gitea.dooplex.hu + GITHUB_RUN_ID, and the link
is printed into the job log as ALARM-LINK. Same change in all four repos.
2026-10-18 20:34:59 +02:00
admin f17e1979bd A cadence cap on the local-API whole-guest backup route (R-973, unreleased)
gates / gates (push) Successful in 49s
Each successful run prunes its tier, so a compromised controller calling POST /backup in a loop
pushed out every pre-compromise restore point. A tier with a cadence of at most 24 h now allows 3
successful runs per rolling 24 h (night, a press, the next night); a longer tier 1 per
max(cadence - 24 h, cadence/2), 6 days for the weekly off-site tier. Only successes count. A
refusal is a 429, a WARN, saved, and for 24 h a critical host-report status that the hub turns
into its existing operator-only agent_capability_degraded event. The success history and the
refusal are kept in the R-894 state file, so a restart does not reset the cap. No sudoers change.
Tests TestBackupCap_* and the state, status and wiring tests, red-proved.
2026-10-18 19:53:15 +02:00
admin 0963bae3c3 The partition probe fails closed on every path (R-1093 review, unreleased)
PART_ENTRY_* comes from the parent's table, so it proves nothing about a partition's contents
unless blkid also completed: lsblk's PTTYPE is now set aside only on a blkid exit 0 (exit 2, an
I/O error or a missing binary keep "device has a partition table"). An lsblk that exits 0 with
output that does not parse into a device, or lists none, leaves the device unprobed instead of
blank (a hole that predates R-1093). Six more TestInspect_R1093_PartitionVerdicts rows, each
red-proved.
2026-10-18 19:47:15 +02:00
admin 8561dc8a5c The new escrow package reaches the hub only when its code is claimed (R-978, unreleased)
The root ceremony uploaded the new package before the new code was claimed, and the hub
overwrites the current package when the sealed secrets are unchanged: a customer who started the
ceremony again and walked away kept a written-down code that opened nothing. Json mode (the
FELHOM_ESCROW argv, unchanged) now returns the package (CeremonyOutput v2, PendingUpload); the
daemon uploads it at POST /escrow/ceremony/claim before handing out the code. A failed upload is a
502 with the code still claimable; TTL and a new start wait for an in-flight claim upload; no
uploader is a 503. Text mode still uploads at once. No sudoers change.
Tests TestEscrowCeremony_R978_* (localapi) and the cmd-side defer/delivery/wiring tests, red-proved.
2026-10-18 19:42:02 +02:00
admin f7b7308c4d An empty partition is blank: lsblk's PTTYPE on a partition is the parent's (R-1093, unreleased)
lsblk prints the disk's table type on every partition, so InspectDevice read an empty partition as
"device has a partition table" and the format path asked the typed erase confirmation. lsblk's
PTTYPE is now set aside only when blkid -p on the partition itself proved it read the device and
that it is a partition of a table of that exact type (PART_ENTRY_NUMBER + matching
PART_ENTRY_SCHEME). A filesystem or a nested table in the partition, a silent blkid, a whole disk
or a different scheme keep the data-bearing reading. Same allow-listed argv, no sudoers change.
Test TestInspect_R1093_PartitionVerdicts, red-proved against the old predicate.
2026-10-18 19:35:03 +02:00
admin 27bbdaaa22 REPORT/CHANGELOG: agent 0.159.2 delivered and proven on demo-hp
gates / gates (push) Successful in 51s
2026-10-18 14:45:47 +02:00
admin e25454ccc3 CHANGELOG: v0.159.2 released (the after-failure orphan check)
gates / gates (push) Successful in 59s
2026-10-18 12:54:28 +02:00
admin 65c1994d3b CHANGELOG (Unreleased): the live proof the next release owes for the after-failure orphan check
gates / gates (push) Successful in 47s
2026-10-17 23:54:14 +02:00
admin 6545cccff7 The orphan thin check runs only after a failed snapshot backup (R-1012 follow-up, unreleased)
gates / gates (push) Successful in 48s
v0.159.0/v0.159.1 ran the check before every snapshot-mode whole-guest backup, which takes a
metadata snapshot of the live pool. Now explainFailure (vzdump start or task failure) is its only
caller; the finding is still appended to the failed backup's error. Tests: failure = 1 call,
success = 0 calls; red-proved against the v0.159.1 runner.go (2 calls / calls=1).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 22:43:03 +02:00
admin eb38d78945 REPORT/CONTEXT: agent 0.159.1, the catalog allow-list (controller v0.328.0), its catalog gate (2026-10-17)
gates / gates (push) Successful in 58s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 17:29:50 +02:00
admin 4a1c07b288 CHANGELOG: v0.159.0 (superseded) and v0.159.1 — released, not vouched
gates / gates (push) Successful in 48s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-17 16:42:00 +02:00
67 changed files with 3272 additions and 148 deletions
+13 -2
View File
@@ -64,6 +64,9 @@ jobs:
env: env:
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain. # In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000 GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
FELHOM_SCAN_FETCH: "1"
run: cd ws/felhom-agent && python3 scripts/agent_gates.py run: cd ws/felhom-agent && python3 scripts/agent_gates.py
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129) - name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
@@ -108,8 +111,16 @@ jobs:
repo = os.environ.get("GITHUB_REPOSITORY", "?") repo = os.environ.get("GITHUB_REPOSITORY", "?")
sha = os.environ.get("GITHUB_SHA", "?") sha = os.environ.get("GITHUB_SHA", "?")
run = os.environ.get("GITHUB_RUN_NUMBER", "?") # R-1135: the link is for a PERSON, so it is the public address and the run ID.
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu") # NOT GITHUB_SERVER_URL: the runner registers at the in-cluster address, so that variable
# reads http://gitea.gitea-system.svc.cluster.local:3000, which nobody outside the cluster
# can open (forced-test mail, job 1997). NOT GITHUB_RUN_NUMBER: Gitea 1.26 reads
# /actions/runs/<n> as a run id first, so a run number that is also an older run's id
# opens THAT run (measured 2026-10-18: felhom.eu /actions/runs/1116 opened the run whose
# id is 1116 = run number 613). The link is printed too, so the job log carries it.
run = os.environ.get("GITHUB_RUN_ID", "?")
srv = "https://gitea.dooplex.hu"
print("ALARM-LINK %s/%s/actions/runs/%s" % (srv, repo, run))
body = json.dumps({ body = json.dumps({
"from": "Felhom CI <monitoring@felhom.eu>", "from": "Felhom CI <monitoring@felhom.eu>",
+18
View File
@@ -0,0 +1,18 @@
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
gitleaks generic-api-key@1ad7cc8eb57e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46ce5a3eccc4 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46fe126846b0 internal/escrow/identity_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@56474f1754f5 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@631c53997f7e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@6e790ef41ef3 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@78334b854242 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@7b88d323a44f internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b0eeee93cf24 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b1cb2659967a internal/localapi/escrow_stage_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b66bec43cc8a internal/escrow/recover_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e0525160fe03 internal/pbsdr/wrapper_read_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e442e455c360 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f23f93f461b2 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f471b78e0f6e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
+263
View File
@@ -1,3 +1,266 @@
## Unreleased (2026-10-22 night) — the tunnel's future-stamped handshake is reported (R-1204) — COMMITTED, NOT RELEASED (no agent release tonight); root SSH by key only (R-982, D4) is a READY PATCH, not on main
- **R-982 / D4 — a ready patch, NOT on main.** Built and proven the same night (felhom.eu
`audits/night-2026-10-22/P2-D4/`): a bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
(`PermitRootLogin prohibit-password`), a `sshd` content check (exactly `prohibit-password`; `sshd -t -f`), and after
an install that wrote it `sshd -t` + `systemctl reload ssh`; tests `RootKeyOnly` (3), red-proved. Live on a fresh
nested box from ISO 1.30.0: root password over SSH refused, root key works, the console and the web page still take
the password. It was committed (`a0c88e5`) and moved back out (`7651461`): the installer's uninstall must remove every bundle file (felhom.eu `scripts/test_hostinstall.py`), the bundle half reaches a box only with an agent release, and the installer half only with a new `installer-v…` tag (R-110 — a push to `main` alone publishes nothing) — both forbidden tonight. *(Corrected the same night: the first reason written here, „a push of `felhom-host-install.sh` publishes the installer", was wrong — the website serves `/scripts/` from the tag `installer-v1.35.0`, read live in `manifests/webpage.yaml` and the running Deployment.)* Both halves are patches: `felhom-agent-bundle-D4.patch` and `felhom.eu-installer-D4.patch` in that folder; apply them together with the agent release and the next installer tag.
- **R-1204 — `wireguard.handshake_future_s`.** A box that ran once with its clock ahead keeps a handshake stamped in the
future; ep0 refuses every newer one, and `last_handshake_age_s` (clamped at 0) read „just now" for ever. The heartbeat
now carries how many seconds the handshake lies ahead (> 60 s only); the age keeps its clamp. The hub alarms on it
(felhom.eu hub `wg_tunnel_stale`). Test `TestR1204_AFutureHandshakeIsReported`, red-proved.
## v0.162.0 — every guest may swap as much as its RAM (R-1192, the 8 GB freeze); the host's swap is reported and checked; no false „operator access degraded" on a fresh box (R-1194) (2026-10-22) — RELEASED on the operator's yes; the vouch is the operator's
Released 2026-10-22 07:40 UTC (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at
`b310773`: binary sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (the same bytes as the
night's candidate), config bundle sha256 `ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5`, tag
`v0.162.0`; no `configs/` change since v0.161.0. Delivered by one signed `agent_update` job per box to demo-hp, Tester 1
and demo-felhom (07:49–07:58 UTC): each `signed op COMPLETED`, `capabilities self-check ok=76 total=76`, `update
committed`, then `guest-swap: raised` for its 9201 (512 MB → its RAM). Evidence felhom.eu
`documentation/audits/morning-2026-10-22/release/04-agent-delivery.txt`. **Found at once:** demo-hp (host swap 8191 MB,
guest swap 25898 MB) and demo-felhom (8191 MB vs 12288 MB) logged the host-swap WARN and the hub mailed
`host_swap_short` — the check compares the host's swap with the guests' whole swap allowance, which a guest larger
than the installer's 8 GB host swap always exceeds (felhom.eu register R-1216). Soak (night of 2026-10-21, 8 GB box):
11 h, no freeze, the host's swap peaked at 2.7 GB used.
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
never set it. On the 8 GB machine (guest 4096 MB) the first uploads made Immich load its analysis models; the guest hit
its limit with the swap full and could evict only program code — 150 000–600 000 refaults and ~1 GB of disk reads per
10 s, memory „full" pressure 92–95 %, load 155, the dashboard dead for 6½ minutes, no app killed. Raising the swap cap
to 4096 MB live ended it within a minute while the uploads ran on.
- `reconcile.buildBringUpConfig`: a new guest gets `swap` = its memory.
- `POST /guest/memory` (RAM resize): `swap` moves with `memory` (`localapi.GuestSwapMB`).
- At agent start (one minute in): `localapi.EnsureGuestSwap` raises every LXC in the `felhom` pool whose swap is below
its RAM to it — a live cgroup change, no restart; never lowers a higher swap; one guest's failure is logged and the
others are still done.
- Tests `TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM`, `TestR1192_BringUpSetsSwapToTheRAM`, the resize test's
new swap assertion — each red-proved (the write removed → `raised 0, want 2`; `swap=""` on both other paths).
- Privilege: `swap` is under `VM.Config.Memory`, which the RAM resize already uses; no role, sudoers or bundle change.
Added the night of 2026-10-21 (still unreleased, the same release):
- **The host's swap is reported and checked (R-1192 follow-up).** The guest fix moves a memory burst onto the HOST's
swap; a host with none gets nothing from it. The host report carries `host.host_swap_mb` (Proxmox node status — the
kernel's SwapTotal, the same number as `/proc/meminfo`) and `host.guest_swap_mb` (the sum of the guests' `swap`; each
guest spec also carries `swap_bytes`). When the first is below the second the agent logs one WARN (and one INFO when
it is covered again); the hub raises `host_swap_short`. `GET /guest/memory` carries `host_swap_mb` and
`guest_swap_mb` for the controller's install page. The agent never creates host swap (a decision — felhom.eu
register). Measured the same night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap.
- **R-1194 — no false „operator access degraded" on a fresh box.** The first host report was sent before the
felhom-sshd loop's first pass, read a config that was never rendered („config invalid, port 0") and the hub mailed
`oob_degraded` — reproduced on a fresh install 2026-10-21 (the report at 20:34:00, the apply 0.4 s later, the daemon
healthy; the check runs `sshd -t` through sudo, so the non-root theory was false). The loop now reports no OOB stanza
until its first pass has run (a failed pass counts, so a real lasting failure still reaches the hub).
- Tests `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
`TestR1194_NoOOBStanzaBeforeTheFirstPass` — each red-proved.
## v0.161.0 — three fixes (R-921, R-1070, R-1010); two texts made true after 0.160.0 (2026-10-20) — RELEASED on the operator's yes, NOT VOUCHED
Released 2026-10-20 (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at `5152123`:
binary sha256 `ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5`, config bundle sha256
`485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47`, tag `v0.161.0`. Delivered by one signed
`agent_update` job per box to demo-hp (06:38 UTC), Tester 1 (06:54) and demo-felhom (06:55): each `signed op COMPLETED`,
`capabilities self-check ok=76 total=76`, `update committed`. **The vouch is the operator's.** Live on demo-hp: `host_busy`
answers `{"busy": true, "what": "backup:local"}` during a „Mentés most" whole-guest backup and `{"busy": false}` before
and after (read from inside 9201 with its own token; felhom.eu `audits/night2-2026-10-20/P1/`). R-1083 (a key for an
existing guest) is NOT in this release: see the R-1083 row. No sudoers, wrapper or config-bundle change.
- **R-921 — `GET /backup/status` says when the host is busy.** New field `host_busy` (`{busy, what}`): whether the
host-wide heavy-operation gate is held right now (another guest's backup, the restore test, a night OS step, the
stale-lock sweep, a power-off) — the state that makes a `POST /backup` answer 409. Served on every request; absent
when the gate is not wired. Controller ≥ 0.335.0 reads it before it stops the household's apps for a tier, so a
refused tier costs no stop (the measured case of 2026-10-08: the night OS step, all 15 apps stopped for a refusal).
A read, never a reservation. No new route. Test `TestBackupStatus_R921_HostBusy`, red-proved (assignment removed →
`free gate: host_busy = <nil>`).
- **R-1070 — a box with no tunnel set up no longer fails its host OS update.** Docker 29 prints „error: no such
object" in lower case, so the tunnel probe's case-sensitive match fell through to `unknown` („could not ask the
guest"), and the host health rule failed every box without a cloudflared container (Tester4, 2026-10-15,
`os_update_health_failed`). Now the match ignores case and a missing container is a new state, **`absent`** („no
tunnel set up on this box"); the host rule reads it as not applicable only when the reading from before the update
(Docker answered) also had no cloudflared container. A tunnel that is down, gone or unreadable still fails. The hub
alarms only on `not_running`, so `absent` never mails. Tests `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows) and
three `TestClassifyTunnel` rows (Docker 29's wording measured on demo-hp), red-proved both ways.
- **R-1010 — the stale-lock janitor clears a half-removed `vzdump` snapshot.** When the delete fails ONLY with
„Failed to find logical volume …_vzdump" (one disk's snapshot already gone — demo-hp 2026-10-14 after a backup ran
out of space), it retries once with Proxmox's own `force=1` on the same API call (`DeleteSnapshotForce`; the same
token privilege, no sudoers line, no root command — `pvesh usage … -command delete` lists `--force`). Any other error
is returned unchanged; a failed forced delete is returned, never a second force. Tests
`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone` (6 rows) and `TestDeleteSnapshotForce_R1010`, red-proved
(single plain delete → `calls [plain], want [plain force]`).
- **The security-scan gate (R-986), tooling only:** `agent_gates.py` registers the SHARED
`felhom.eu/scripts/security_scan_gate.py` (FAST set: the hook and CI) — govulncheck (called vulnerabilities) and gitleaks
on the pushed commits, **warn-first**. `.security-accepted` holds the first scan's 15 accepted hits (escrow and
local-API test constants), each with a reason. CI fetches the pinned gitleaks (`FELHOM_SCAN_FETCH=1`). Decoys: the
shared 8 plus the gate on this repo, in `test_gate_decoys.py`.
### Earlier on 2026-10-19 (texts only)
- The ceremony-complete log line now says the package uploads at the claim **or at the typed-back confirm** (R-1151);
it read „package uploads at the claim" for both orders (seen in demo-hp's journal, 2026-10-19 01:47 UTC).
- The R-973 comments in `cmd/felhom-agent/main.go` and `internal/localapi/backup_cap.go` named the hub event
`agent_capability_degraded`; since hub v0.159.0 (R-1150) a refused backup raises `whole_guest_backup_refused`.
Found by the hub helper of the day brief 2026-10-19.
- **gofmt, R-1160:** `1edfd8e` formats the 14 Go files that were not gofmt-clean (formatting only, `go test ./...`
green). New gate `scripts/gofmt_gate.py` (the controller's R-454 shape) in `agent_gates.py`, fast set; its decoys in
`scripts/test_gate_decoys.py` plant an unformatted file in a SCRATCH copy (convicted by name), the formatted twin
(passes), and the no-Go modes (CI passes out loud, a dev machine is INCONCLUSIVE). Red-proved: with the conviction
switched off the FACT case reports a LIVE HOLE. The next release carries it; no release for it.
## v0.160.0 — five fixes: an abandoned or un-typed-back escrow ceremony keeps the old code (R-978, R-1151); a whole-guest backup flood is capped (R-973) and the local tier keeps one point per day (R-1149); an empty partition is not „data" (R-1093) (2026-10-19)
Released 2026-10-19 on the operator's yes (02:48 answers, D17 „one per day, 3 days"). sha256
`7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1`, config bundle
`6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f`. The vouch is the operator's.
- **R-1093 — an EMPTY partition no longer asks the typed „data will be erased" confirmation.** `lsblk` prints the
parent disk's table type (PTTYPE) on every partition, so `SudoHostOps.InspectDevice` read a blank partition as
„device has a partition table". Now lsblk's PTTYPE is set aside only when `blkid -p` on the partition itself proved
it read the device and that it is a partition of a table of exactly that type (`PART_ENTRY_NUMBER` +
`PART_ENTRY_SCHEME` equal to lsblk's PTTYPE). The partition's own signatures still come from `blkid -p`: a
filesystem or a nested table inside it stays data-bearing; a blkid that printed nothing, a whole disk, or a scheme
that differs keeps the old, safe reading. **No new command, no sudoers or config-bundle change** (the two probes are
the same allow-listed argv). Test `TestInspect_R1093_PartitionVerdicts` (8 rows), red-proved: with the old
`if pttype != ""` both empty-partition rows fail `DataBearing=true reason="device has a partition table"`.
**Hardened after the same night's review (fail-closed on every path):** `PART_ENTRY_*` is read from the PARENT's
table, so lsblk's PTTYPE is set aside only when blkid also **exited 0** — exit 2, an I/O error or a missing binary
keep „device has a partition table"; and an `lsblk` that exits 0 with output that does not parse into a device (or
lists none) now leaves the device UNPROBED („could not be reliably inspected") instead of „blank" — that hole
predates R-1093. Six more rows in the same test, each red-proved (`blkidErr` check removed → „device is blank";
`&& parsed` removed → `Probed=true … „device is blank"`).
- **R-978 — an abandoned second escrow ceremony no longer makes the customer's recovery code useless.** The root
ceremony (the FELHOM_ESCROW argv, unchanged: `--upload --output=json`) used to upload the new package before the new
code was claimed, and the hub overwrites the current package when the sealed secrets are unchanged — so a customer
who started the ceremony again and walked away (or an agent restart in the 10-minute window) was left with a
written-down code that opened nothing. Now json mode hands the package back (`CeremonyOutput.PendingUpload`, output
**version 2**; text mode still uploads at once) and the daemon uploads it at `POST /escrow/ceremony/claim`, **before**
the code is handed out (`EscrowCeremonyConfig.Upload` → `putEscrowBody`, the same PUT and per-host key). A failed
upload answers 502 and keeps the code claimable; the TTL and a new start wait while the claim's upload is in flight;
no uploader → 503; an output that uploaded by itself or carries no package fails the ceremony. Status `uploaded`
now reads false until the claim. The controller is unchanged (it never read `uploaded`; its claim already assumed
the package is on the hub when the claim returns). **No sudoers or config-bundle change.** Tests (localapi)
`TestEscrowCeremony_R978_*` (abandon by TTL / by a new start / by a restart → the hub's package still opens with
code 1; claim stores before handing out; upload failure; in-flight TTL; no uploader; no package) and (cmd)
`TestEscrowOptsFromCLI_JSONModeDefersUpload`, `TestDeliverEscrowPackage_JSONModeSendsNothing`,
`TestR978_CeremonyUploadIsWiredIntoTheDaemon`. Red-proved: upload-at-create restored → „code 1 no longer opens it
(uploads=2)"; the in-flight TTL guard removed → „the TTL voided the code while its package was uploading";
`deferUpload: false` → „json mode must defer the upload"; the `Upload:` line removed → „has no Upload". **Residual:**
the customer's typed-back check after the claim is browser-only, so „confirmed" here means „claimed" — a customer
who opens the code and closes the page without writing it down has replaced the package.
- **R-1151 — the new recovery code replaces the hub's package only after the household typed it back.** R-978 moved
the upload to the claim; but the wizard's typed-back check was the browser's alone, so a household that opened the
new code and closed the page without writing it down had already replaced the package their old code opens. Now a
claim with `{"defer_upload": true}` (controller ≥ 0.332.0) hands out the code with two word positions
(`verify_positions`, drawn by crypto/rand) and keeps the package; the new `POST /escrow/ceremony/confirm
{"words": [w1, w2]}` checks the two words (constant time, case and spaces aside) and only then uploads it. Right → 200;
wrong → 422, still waiting; one word → 400; nothing waiting → 404; the 30-minute window ran out → 410 and the job
reads `unconfirmed_void` (the package dropped, the previous code still works); hub failure → 502, still waiting. Only
the two words are kept for the check, never R; a new start, the window and a restart drop them with the package.
Status gains `awaiting_confirm` and `confirm_expires_in_sec`. **An old controller (≤ 0.331.0) claims with `{}` and
never confirms: it keeps the R-978 order (upload at the claim)**, so its household's ceremony still completes; a new
controller on an old agent sees no `confirm_required` and needs no confirm. No sudoers or bundle change. Tests
`TestEscrowCeremony_R1151_*` (shown but not typed back — by the window, a new start, a restart — code 1 still opens
the hub's package; typing; the old controller's order; hub failure at the confirm) and `TestPickConfirmPositions`.
Red-proved: the defer flag ignored → „a code that was shown but never typed back replaced the package: code 1 no
longer opens it (uploads=2)" in all three; the word check skipped → „one wrong word: got 200 (uploads=1)".
- **R-1149 — the local whole-guest tier keeps one restore point per day for 3 days** (operator decision D17,
2026-10-19). The primary (local) tier's per-run prune is now `--prune-backups keep-daily=N` (N =
`local_backup_retention`, default 3) instead of `keep-last=N` — the same at most 3 archives, but a run of extra
backups inside one day replaces only that day's point, so with the R-973 cap a flood cannot push out the earlier
days' copies. Additional tiers keep their explicit `keep_last` (the weekly off-site „last 2" is two weeks).
`config.BackupTier.PruneSpec()` is the one place the spec is built (runDaemon used to format keep-last itself); the
„backup tier armed" log line names it. Tests `TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay` and the
consequence `TestBackupCap_R1149_LocalFloodKeepsOlderDays` (the agent's real spec, a Proxmox-shaped prune model,
three nights then a day-long flood → the two earlier nights survive). Red-proved: keep-last for the primary →
„primary tier prune = "keep-last=3", want "keep-daily=3"" and „holds 0 copies made before the compromise". No
config or sudoers change.
- **R-973 — the local-API whole-guest backup route has a cadence cap.** Every successful run prunes its tier, so a
compromised controller calling `POST /backup` in a loop pushed out every pre-compromise restore point (local
keep-last 3; off-site via ep0's keep-last 2). Now (`internal/localapi/backup_cap.go`): a tier whose cadence is ≤ 24 h
allows **3 successful runs per rolling 24 h** (the `07` §6 day: the night run, a „Mentés most" press, and the next
night, which since R-899 still runs and can land inside the same 24 h — so a press never costs the night); a longer
tier allows **1 per max(cadence − 24 h, cadence/2)** — 6 days for the 7-day off-site tier, which the controller only
runs when due and the button never targets. Only successes count (a failed run prunes nothing). A refusal answers
**429** with the next allowed time, is logged at WARN, is saved, and for 24 h raises the CRITICAL host-report status
`localapi:backup-cadence-cap` — the hub's existing operator-only `agent_capability_degraded` event, **no hub change**.
**The cap survives an agent restart:** `BackupSuccessState` (the R-894 file) now also keeps the newest 8 success
times and the last refusal per tier and guest (additive JSON fields; a pre-R-973 file counts its one known
success). Tests `TestBackupCap_*` (an off-site flood keeps a pre-compromise copy for 5+ days; a local flood runs 3
a day; the scheduled day is never refused; restart; failures; the operator signal) + `TestBackupCadenceCap`,
`TestBackupSuccessState_R973_*`, `TestBackupCapStatus_*`, `TestR973_CapRefusalIsWiredIntoTheReport`. Red-proved:
the cap disabled → „the off-site tier lost every pre-compromise copy: 10 runs"; the disk history ignored → „after a
restart the 4th run in a day was allowed"; the wiring line removed → „never binds backupCapRefusal". **Limits, stated:**
with keep-last 3 the local tier's three points can still be replaced within one day (3 per day is the schedule's own
pace plus one press); the hub's event text reads „agent privileged capability degraded" (the type is generic). No
sudoers or bundle change.
- **CI workflow (R-1135, no code change).** `.gitea/workflows/gates.yml`, the alarm step: the mail's run link is now `https://gitea.dooplex.hu/<repo>/actions/runs/<run id>`.
It used `GITHUB_SERVER_URL`, which on this runner is the in-cluster address (`http://gitea.gitea-system.svc.cluster.local:3000`,
the address the runner registers at), so nobody outside the cluster could open it. It also used the run NUMBER, and
Gitea 1.26 reads `/actions/runs/<n>` as a run id first: a number that is also an older run's id opened that other run
(measured 2026-10-18: `felhom.eu/actions/runs/1116` opened run id 1116, run number 613). It now uses `GITHUB_RUN_ID` and
prints the link (`ALARM-LINK …`) into the job log. Same change in all four repos' workflows.
## v0.159.2 — the orphan thin check runs only after a failed backup (R-1012 follow-up) (2026-10-18)
**Released 2026-10-18 (committed 2026-10-17): sha256 `4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9`, bundle `477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9`; the vouch is the operator's.** No `configs/` change since v0.159.1. The independent review of 2026-10-18 found nothing in it. v0.159.0/v0.159.1 ran the R-1012 orphan check
before EVERY snapshot-mode whole-guest backup, and that check takes a short metadata snapshot of the live thin pool.
Now `BackupRunner` runs it only when the snapshot step has failed (the vzdump start or its task) — the moment the
answer is needed. The finding is still appended in plain words to the failed backup's error, exactly as before; a
successful backup never runs the check and keeps a clean record; stop mode and an unwired runner are unchanged. The
`thin-orphan-check` capability text says „after a failed whole-guest backup". Tests
`TestBackup_SnapshotFailure_NamesTheOrphan` (now: exactly 1 call, after the failure) and
`TestBackup_ThinCheck_SilentWhenNotApplicable` (now: 0 calls on a successful backup), red-proved against the v0.159.1
`runner.go`: failure legs „thin check ran 2 times, want 1", success leg `calls=1`. No root file, sudoers or bundle
change. (Installer side, felhom.eu, same night: `felhom-host-install.sh --rotate-recovery-only`, R-1141.)
**Proven live 2026-10-18 (demo-hp):** a green snapshot backup with 0 `felhom-thin-check` calls (the 0.159.1 night backup
had 1). The failed leg stays covered by the fixture test.
## v0.159.1 — the signed decommission prunes the guest-bind record (found by v0.159.0's own live proof) (2026-10-17)
**v0.159.0 is superseded — do not vouch it.** Everything of v0.159.0 below, plus: `signedjobs.DecommissionExecutor`
drops the drive from every guest's bind record (`SetGuestBinds`, wired to the agent's `GuestBindStore`), as the
self-serve decommission always did. Without it a drive decommissioned by an operator-signed job and then forgotten
(R-1124) left a record the 20-second guest-bind re-assert warned about every tick („reconcile: skipping non-enrolled
drive") — seen on demo-hp minutes after the R-1124 proof. Test `TestDecommissionExecutor_PrunesTheGuestBindRecord`,
red-proved. Released: tag `v0.159.1`, sha256 `de8f6ad9…aff9`, bundle `2af3d27d…7cd6` (the same root files as v0.159.0).
**Not vouched.** Delivery to a box on the v0.158.1 bundle: `agent_update 0.159.1` → `agent_config_update 0.159.0-step1`
(sha `1d65edb3…d6cf`, published) → `agent_config_update 0.159.1` (the bundle adds `felhom-thin-check`, R16).
## v0.159.0 — forget a decommissioned drive that is gone (R-1124); name an orphan thin device in a backup failure (R-1012); Go 1.27.2 (R-972) (2026-10-17)
**Superseded by v0.159.1 — do not vouch.** Task: „agent release, the catalog allow-list, break-glass rotation, off-site
history check" (felhom.eu `REPORT-agent-allowlist-rotation-2026-10-17.md`).
- **Go 1.27.2 + x/crypto v0.58.0** (R-972, `37b99d7`): the first release that carries them (v0.158.1 was built with
go1.26.0).
- **R-1124 — a decommissioned drive that is gone is forgotten.** `SudoHostOps.ForgetRetiredDrives` (agent start and the
20-second storage tick): for a Felhom drive unit whose intent is `decommissioned`, whose `/dev/disk/by-uuid` link is
gone and which is not mounted, `systemctl disable` + the new root verb `felhom-priv-apply unit-forget <unit>` +
`daemon-reload`. The unit is gone, so a later boot no longer waits for the drive and the agent stops listing it (no
„absent by UUID" line every 20 s). A decommissioned drive still plugged in KEEPS its unit and raw mount, so the
one-click re-enroll (H3) still works. `unit-forget` re-checks as root: a felhom by-UUID drive unit (not a network
unit), disabled, its `Where` not mounted, its by-UUID link absent — otherwise refused. Sudoers `FELHOM_MOUNT` gains the
one verb. Tests `TestForgetRetiredDrives_*` and `configs/test_felhom_priv_apply.py` `UnitForget`, red-proved.
**Live on demo-hp:** a loop-backed test drive enrolled through the local API, decommissioned by a signed job (kept
its unit while present), unplugged → 8 s later the unit and its wants link were removed and `GET /disks` dropped it.
- **R-1012 — an orphan thin device is NAMED in the backup failure, never removed.** New READ-ONLY root helper
`felhom-thin-check <vg> <pool>` (config bundle): LVM's thin ids before and after, the pool's ids through a metadata
snapshot it reserves and releases (refuses — exit 3 — when someone else holds one; releases on every exit; sends no
other pool message). `internal/thincheck`: an orphan = in the pool and in neither LVM list, on two reads 3 s apart;
fails open. `BackupRunner.SetThinCheck`: before a snapshot-mode backup (a WARN when found) and again after a failure —
the finding is appended in plain words to the error the controller mails to the operator, pointing at the new
runbook `whole-guest-backup-orphan-thin.md`. Sudoers `FELHOM_THINCHECK`, capability `thin-orphan-check`. Tests
`internal/thincheck`, `TestBackup_SnapshotFailure_NamesTheOrphan`, `configs/test_felhom_thin_check.py`, red-proved.
**Live on demo-hp:** on a scratch thin pool (a loop file on `/mnt/hdd_1`, never `pve/data`) the helper listed a made
orphan (device 7) and refused a snapshot held by someone else; on `pve/data` it read 1, 2 = LVM's and released; a
normal whole-guest backup of 9201 ran the check first and finished green.
- Released: tag `v0.159.0`, sha256 `3cfd0a25…edfed5`, bundle `40663c4f…6d681`; step bundle `0.159.0-step1`.
## CI only — a manual proof that the failure mail can be sent (R-1129, 2026-10-17) ## CI only — a manual proof that the failure mail can be sent (R-1129, 2026-10-17)
No agent release; no binary changes. No agent release; no binary changes.
+28
View File
@@ -1,5 +1,33 @@
# CONTEXT — felhom-agent working state # CONTEXT — felhom-agent working state
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
> `ff4e6dc5…99d3b5`, bundle `485f36c9…12c47`. R-1083 left out on purpose: an agent key for 9202 would make its
> controller book whole-guest backups through the agent, one tier writing to ep0 — the R-1083 row says what is needed.
> **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978,
> R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with
> `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn
> positions; only those two words are held, 30-min window; a claim without it keeps the R-978 order). Vouch: version
> 0.160.0, sha256 `7ebb547f…dfca1`, bundle `6b060901…e202f`. After it, unreleased: `4545819` (two texts). felhom.eu
> `REPORT-day-2026-10-19.md`.
> **2026-10-18 (night) — UNRELEASED on main: R-1093, R-978, R-973** (release + vouch: the operator's). R-1093 an empty
> partition is blank (lsblk's PTTYPE on a partition is the parent's; set aside only on a clean blkid exit 0 with a
> matching `PART_ENTRY_SCHEME`; an unparsable lsblk is unprobed). R-978 the escrow package reaches the hub only at the
> claim (ceremony output v2, `PendingUpload`; `EscrowCeremonyConfig.Upload`). R-973 the `POST /backup` cadence cap.
> **Decision (decided by CC unattended — operator may reverse): the cap's numbers.** Q: how many whole-guest runs may
> the guest ask for? Options: (a) 2 per 24 h on the daily tier — keeps one local point older than a day, but refuses
> the NIGHT after a press (R-899's night can fall inside the same 24 h); (b) 3 per 24 h + 1 per 6 days on the weekly
> tier — the schedule never refused, a second press in a day refused, the local tier still replaceable in a day;
> (c) none — status quo. Chose (b): the off-site tier is the one `07` §8 row 10 relies on and it is fully protected
> for ~6 days; the local half's real fix is a time-bucketed prune (keep-daily), which is a retention change for the
> operator. No sudoers or bundle change in any of the three.
> **2026-10-17 — v0.159.1 RELEASED (0.159.0 superseded, do not vouch).** Go 1.27.2; R-1124 a decommissioned drive that is
> gone is forgotten (`ForgetRetiredDrives`, root verb `unit-forget`); R-1012 the read-only `felhom-thin-check` names an
> orphan thin device in a failed backup's text; the signed decommission prunes the guest-bind record. Bundle adds a path →
> step bundle `0.159.0-step1`. Proven on demo-hp; delivered by signed jobs on the operator's yes. Vouch: the operator's.
> **2026-10-16 — v0.158.1 RELEASED and VOUCHED (storage identity build B; 0.158.0 superseded, do not vouch).** The internal > **2026-10-16 — v0.158.1 RELEASED and VOUCHED (storage identity build B; 0.158.0 superseded, do not vouch).** The internal
> data place (root helper `felhom-internal-place`, the record in the state dir, mounted at /mnt/belso like a drive), the > data place (root helper `felhom-internal-place`, the record in the state dir, mounted at /mnt/belso like a drive), the
> signed `add_internal_place`, partitions in `/disks/candidates` (`formattable` only while nothing on the disk is mounted), > signed `add_internal_place`, partitions in `/disks/candidates` (`formattable` only while nothing on the disk is mounted),
+17 -10
View File
@@ -1,12 +1,19 @@
# REPORT — the agent on Go 1.27.2 (R-972, 2026-10-16) — NOT released # REPORT — night 2026-10-22: one change on `main`, one ready patch, nothing released
Written in ASD-STE100 Simplified Technical English. Part of burndown 2 (felhom.eu `REPORT-burndown2-2026-10-16.md`). **What:** the night brief's agent parts. No agent release, no vouch (both forbidden tonight). Design read:
`03-host-agent.md` (the WireGuard tunnel, the config bundle), `06-offsite-connectivity.md`.
- **What changed:** `go.mod` `go 1.27.2` and golang.org/x/crypto v0.58.0. No code change. - **R-1204, `a0c88e5` (on `main`, unreleased).** The heartbeat's `wireguard` block gains `handshake_future_s` when the
- **Proof:** `go build ./...`, `go vet ./...`, `go test ./...` rc 0 on Go 1.27.2; `agent_gates.py --fast` all OK. last handshake lies more than 60 s in the future (`wgtunnel.handshakeFuture`) — the clock-ahead lock-out that the
- **Not done, on purpose:** no release and no deploy — agent releases wait for the operator's vouch. R-972 stays clamped `last_handshake_age_s` hid. Red-proved (`TestR1204_AFutureHandshakeIsReported`). The hub's alarm on it is
NARROWED for the agent: the next vouched release carries the new Go. hub 0.166.0 (felhom.eu, built, not deployed).
- **R-982 / D4 (operator 15:47: root SSH from the LAN by key only) — a ready patch, NOT on main (`7651461` moved it
## Observations out).** A bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`),
checked and reloaded by `felhom-os-apply`; proven on a fresh install from ISO 1.30.0 (nested VM 9343, destroyed):
1. The release script needs no change: it builds with the toolchain `go.mod` names. NOT-A-FINDING: GOTOOLCHAIN=auto on DooPlex fetched go1.27.2 for this build and will for a release build. password SSH refused, key works, the console and the PVE web login keep the password. Patches:
felhom.eu `audits/night-2026-10-22/P2-D4/felhom-agent-bundle-D4.patch` + `felhom.eu-installer-D4.patch`. They ship
with the next agent release (the operator's vouch) and the next installer tag.
- **Correction:** the CHANGELOG first said a push of the installer would publish it. Wrong: `/scripts/` is served from
the tag `installer-v1.35.0` (R-110). The decision to keep D4 as a patch stands on the other two reasons.
- **Gates:** `agent_gates.py --fast` green before each push. CI by commit: felhom.eu `REPORT-night-2026-10-22.md`.
- **Teardown:** VM 9343 destroyed on demo-hp (own id); its hub appliance (id 49) waits for the operator's Discard.
+3 -2
View File
@@ -114,8 +114,8 @@
| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled | | `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability | | `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
| `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew | | `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew |
| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v1) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) | | `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v2, R-978: json mode returns the package in `PendingUpload` instead of uploading) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) |
| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged | | localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); **R-978: the new package reaches the hub only at the claim, before R is handed out (`EscrowCeremonyConfig.Upload`); TTL and a new start wait while that upload is in flight — an abandoned ceremony leaves the previous package current**; **R-1151: a claim with `{"defer_upload":true}` hands out R + two word positions and keeps the package until `POST /escrow/ceremony/confirm` brings those two words back (30-min window, only the two words kept, never R); a claim without it keeps the R-978 order (old controllers)**; subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` | | `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
| `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) | | `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) |
| `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker | | `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker |
@@ -174,6 +174,7 @@
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. | | `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. | | `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. | | `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
| localapi backup cadence cap (R-973) | internal/localapi/backup_cap.go | `backupCadenceCap(cadence)`, `Server.capRefuses`, `Server.BackupCapRefusal()` | POST /backup refuses (429) a run the `07` §6 schedule would never ask for: ≤24 h tier → 3 successes per rolling 24 h; longer tier → 1 per max(cadence−24 h, cadence/2) | Counts SUCCESSES only, from this process AND `BackupSuccessState.RecentBackupSuccesses` (so a restart does not reset it); refusals saved (`RecordBackupRefusal`) and raised for 24 h as the CRITICAL host-report status `localapi:backup-cadence-cap` (`backupCapStatus` in main) → the hub's existing `agent_capability_degraded`. Tests `TestBackupCap_*`. |
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. | | `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. | | `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). | | `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
+67
View File
@@ -0,0 +1,67 @@
package main
import (
"go/ast"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
)
// R-973: a cadence-cap refusal becomes a CRITICAL degraded status — what the hub's existing
// agent_capability_degraded event (operator-only) keys on — and clears when the refusal ages out.
func TestBackupCapStatus_RefusalIsCriticalDegraded(t *testing.T) {
at := time.Date(2026, 10, 18, 3, 0, 0, 0, time.UTC)
s := backupCapStatus(func() (string, int, time.Time, bool) { return "felhom-pbs", 9201, at, true })
if !s.Critical || s.Status != capability.StatusDegraded || !strings.Contains(s.Reason, "9201") || !strings.Contains(s.Reason, "felhom-pbs") {
t.Fatalf("a refusal must raise a critical degraded status naming the guest and the tier: %+v", s)
}
if s := backupCapStatus(func() (string, int, time.Time, bool) { return "", 0, time.Time{}, false }); s.Status != capability.StatusOK || !s.Critical {
t.Fatalf("no refusal → ok (and still critical, so a later refusal alerts): %+v", s)
}
if s := backupCapStatus(nil); s.Status != capability.StatusOK {
t.Fatalf("no local API → ok: %+v", s)
}
}
// Wiring (the built-but-never-wired class): runDaemon's capability probe appends backupCapStatus, and the
// late-bound source is set from the local API server.
//
// COMPANION RED-PROOF (observed 2026-10-18): delete `backupCapRefusal = localSrv.BackupCapRefusal` →
// "runDaemon never binds backupCapRefusal to the local API server". Restored.
func TestR973_CapRefusalIsWiredIntoTheReport(t *testing.T) {
_, f := parseMain(t)
var appended, bound bool
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "runDaemon" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.CallExpr:
if id, ok := x.Fun.(*ast.Ident); ok && id.Name == "backupCapStatus" && len(x.Args) == 1 {
if a, ok := x.Args[0].(*ast.Ident); ok && a.Name == "backupCapRefusal" {
appended = true
}
}
case *ast.AssignStmt:
if len(x.Lhs) == 1 && len(x.Rhs) == 1 {
if l, ok := x.Lhs[0].(*ast.Ident); ok && l.Name == "backupCapRefusal" {
if sel, ok := x.Rhs[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "BackupCapRefusal" {
bound = true
}
}
}
}
return true
})
}
if !appended {
t.Fatal("runDaemon's capability probe does not call backupCapStatus(backupCapRefusal)")
}
if !bound {
t.Fatal("runDaemon never binds backupCapRefusal to the local API server")
}
}
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"context"
"encoding/json"
"go/ast"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// R-978, the root half: the controller-driven ceremony (json mode, the FELHOM_ESCROW argv with
// `--upload`) must hand its package back instead of sending it — the daemon sends it at the claim.
//
// COMPANION RED-PROOF (observed 2026-10-18): set `deferUpload: false` in escrowOptsFromCLI → this fails
// with "json mode must defer the upload (R-978): got deferUpload=false", and the delivery test's json leg
// then sends the package at create time ("put called 1 times"). Restored.
func TestEscrowOptsFromCLI_JSONModeDefersUpload(t *testing.T) {
// The FELHOM_ESCROW argv: --selftest=escrow-create --upload --output=json.
if o := escrowOptsFromCLI("", false, false, true, "", true); !o.deferUpload || !o.upload {
t.Fatalf("json mode must defer the upload (R-978): got deferUpload=%v upload=%v", o.deferUpload, o.upload)
}
// Text mode (an operator at a terminal) keeps the immediate upload; no --upload means no upload at all.
if o := escrowOptsFromCLI("", false, false, true, "", false); o.deferUpload {
t.Fatal("text mode must upload at once")
}
if o := escrowOptsFromCLI("", false, false, false, "", true); o.deferUpload {
t.Fatal("without --upload there is nothing to defer")
}
}
func TestDeliverEscrowPackage_JSONModeSendsNothing(t *testing.T) {
cfg := config.Config{}
cfg.Hub.URL, cfg.Hub.HostID, cfg.Hub.APIKey = "https://hub.invalid", "host-1", "k"
res := escrow.CreateResult{Blob: []byte("ciphertext"), KeyFingerprint: "f2:87", Posture: escrow.PostureZeroKnowledge, IdentityBlob: []byte("id")}
for _, jsonMode := range []bool{true, false} {
calls := 0
var sent []byte
put := func(_ context.Context, _ config.Config, body []byte) error { calls++; sent = body; return nil }
o := escrowOptsFromCLI("", false, false, true, "", jsonMode)
pending, uploaded, err := deliverEscrowPackage(context.Background(), cfg, res, "abc", o.deferUpload, put)
if err != nil {
t.Fatal(err)
}
if jsonMode {
if calls != 0 || uploaded {
t.Fatalf("json mode: put called %d times (uploaded=%v), want 0 — the package must wait for the claim", calls, uploaded)
}
var m map[string]any
if json.Unmarshal(pending, &m) != nil || m["blob_b64"] == "" || m["identity_blob_b64"] == "" || m["restic_pw_sha256"] != "abc" {
t.Fatalf("json mode: the pending package is not the hub upload body: %s", pending)
}
} else if calls != 1 || !uploaded || pending != nil || len(sent) == 0 {
t.Fatalf("text mode: put called %d times, uploaded=%v, pending=%q — want one immediate upload", calls, uploaded, pending)
}
}
// A hub that is not configured fails in both modes (as the pre-R-978 upload did).
if _, _, err := deliverEscrowPackage(context.Background(), config.Config{}, res, "", true, nil); err == nil {
t.Fatal("an unconfigured hub must fail the ceremony even when the upload is deferred")
}
}
// R-978 wiring (the built-but-never-wired class): runDaemon's localapi.EscrowCeremonyConfig carries an
// Upload that calls putEscrowBody. Without it every claim of a new package answers 503 and no customer
// can finish a ceremony.
//
// COMPANION RED-PROOF (observed 2026-10-18): delete the `Upload:` line in runDaemon → this fails with
// "runDaemon's EscrowCeremonyConfig has no Upload calling putEscrowBody". Restored.
func TestR978_CeremonyUploadIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
if !callsWithin(f, "main")["runDaemon"] {
t.Fatal("main → runDaemon is broken — the path this test asserts is not the live one")
}
wired := false
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "runDaemon" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
cl, ok := n.(*ast.CompositeLit)
if !ok {
return true
}
sel, ok := cl.Type.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "EscrowCeremonyConfig" {
return true
}
for _, el := range cl.Elts {
if kv, ok := el.(*ast.KeyValueExpr); ok {
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "Upload" {
ast.Inspect(kv.Value, func(n ast.Node) bool {
if ce, ok := n.(*ast.CallExpr); ok {
if id, ok := ce.Fun.(*ast.Ident); ok && id.Name == "putEscrowBody" {
wired = true
}
}
return true
})
}
}
}
return true
})
}
if !wired {
t.Fatal("runDaemon's EscrowCeremonyConfig has no Upload calling putEscrowBody")
}
}
+106 -19
View File
@@ -194,7 +194,7 @@ func main() {
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)") flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)") flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)")
flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))") flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))")
flag.BoolVar(&upload, "upload", false, "for --selftest=escrow-create: upload the opaque blob to the hub") flag.BoolVar(&upload, "upload", false, "for --selftest=escrow-create: upload the opaque blob to the hub (with --output=json: hand the package back in pending_upload — the agent daemon uploads it when the code is claimed, R-978)")
flag.StringVar(&blobPath, "blob", "", "for --selftest=escrow-consume: path to the R-wrapped escrow blob file") flag.StringVar(&blobPath, "blob", "", "for --selftest=escrow-consume: path to the R-wrapped escrow blob file")
flag.StringVar(&expectedFP, "fingerprint", "", "for --selftest=escrow-consume: the EXPECTED key fingerprint (the gate target)") flag.StringVar(&expectedFP, "fingerprint", "", "for --selftest=escrow-consume: the EXPECTED key fingerprint (the gate target)")
flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)") flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)")
@@ -404,6 +404,31 @@ func (r *gateRemounter) Remount(ctx context.Context, t storage.KnownTarget) {
// poolReadStatus probes the felhom-pool membership read (GET /pools/felhom) the stale-lock reaper's // poolReadStatus probes the felhom-pool membership read (GET /pools/felhom) the stale-lock reaper's
// ownership scoping depends on (v0.62.0, audit A1). Needs `Pool.Audit` at `/pool/felhom` — granted // ownership scoping depends on (v0.62.0, audit A1). Needs `Pool.Audit` at `/pool/felhom` — granted
// by host-install v1.9.0; on an older ACL this reports degraded and the reaper fail-safes (skips). // by host-install v1.9.0; on an older ACL this reports degraded and the reaper fail-safes (skips).
// backupCapStatusName is the host-report status the R-973 cadence cap raises.
const backupCapStatusName = "localapi:backup-cadence-cap"
// backupCapStatus turns the local-API backup cadence cap's newest refusal (R-973) into a CRITICAL
// capability status: degraded for 24 h after a refusal, so the hub raises its operator-only
// whole_guest_backup_refused event (hub v0.159.0, R-1150; agent_capability_degraded before). The Reason names the guest, the tier and the time, and
// says what to do. nil source (local API off) → ok.
func backupCapStatus(refusal func() (string, int, time.Time, bool)) capability.Status {
s := capability.Status{
Name: backupCapStatusName,
Feature: "whole-guest restore points: a guest asked for more whole-guest backups than its schedule allows, and the extra ones were refused (R-973)",
Critical: true,
Status: capability.StatusOK,
}
if refusal == nil {
return s
}
if target, vmid, at, ok := refusal(); ok {
s.Status = capability.StatusDegraded
s.Reason = fmt.Sprintf("refused a whole-guest backup of guest %d on target %q at %s — more than the schedule allows; the guest's controller may be misbehaving or compromised: check the guest",
vmid, target, at.UTC().Format(time.RFC3339))
}
return s
}
func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status { func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status {
s := capability.Status{ s := capability.Status{
Name: "pve:pool-read", Name: "pve:pool-read",
@@ -830,8 +855,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run, run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
log: logger, log: logger,
} }
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
// drConfigured: the local API server is built further down, before the loop starts.
var backupCapRefusal func() (string, int, time.Time, bool)
probeAll := func(ctx context.Context) []capability.Status { probeAll := func(ctx context.Context) []capability.Status {
out := append(capProber.Probe(ctx), poolReadStatus(ctx, px)) out := append(capProber.Probe(ctx), poolReadStatus(ctx, px))
out = append(out, backupCapStatus(backupCapRefusal))
return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...) return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...)
} }
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot // (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
@@ -906,6 +936,22 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// network and stays under the hub's 45-minute host_stale (its // network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged. // alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait}) go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
// after start so the guest has settled; a failure is logged and the next start retries.
go func() {
select {
case <-ctx.Done():
return
case <-time.After(time.Minute):
}
sctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
if n, err := localapi.EnsureGuestSwap(sctx, px, reconcile.DefaultPool, logger); err != nil {
logger.Warn("guest-swap: check skipped", "err", err)
} else {
logger.Info("guest-swap: checked", "raised", n)
}
}()
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue // Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving // (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
@@ -1197,6 +1243,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
return cfg.Escrow.PBSStorageID return cfg.Escrow.PBSStorageID
}, },
HubConfigured: cfg.Hub.URL != "" && cfg.Hub.HostID != "" && cfg.Hub.APIKey != "", HubConfigured: cfg.Hub.URL != "" && cfg.Hub.HostID != "" && cfg.Hub.APIKey != "",
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
// subprocess used, under the same per-host key, before the code is handed out.
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
DRConfigured: func() bool { DRConfigured: func() bool {
if drConfigured != nil { if drConfigured != nil {
return drConfigured() return drConfigured()
@@ -1206,6 +1255,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} }
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens) localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil { if localSrv != nil {
backupCapRefusal = localSrv.BackupCapRefusal // R-973
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace) internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go. // R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
pwMode := proxmox.RunnerMode(cfg.Privileged.Mode) pwMode := proxmox.RunnerMode(cfg.Privileged.Mode)
@@ -1913,10 +1963,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
var tierRunners []*backup.BackupRunner var tierRunners []*backup.BackupRunner
var tierTargets []string var tierTargets []string
for _, t := range backupTiers { for _, t := range backupTiers {
prune := "" prune := t.PruneSpec() // primary: keep-daily=N (R-1149); an extra tier: keep-last=N; 0 → no prune
if t.KeepLast > 0 {
prune = fmt.Sprintf("keep-last=%d", t.KeepLast)
}
// Pruning a PBS target is allowed ONLY for an additional tier with an explicit keep_last // Pruning a PBS target is allowed ONLY for an additional tier with an explicit keep_last
// (the primary's target AND retention both default, so it could prune the DR by accident). // (the primary's target AND retention both default, so it could prune the DR by accident).
allowPBSPrune := !t.Primary && t.KeepLast > 0 allowPBSPrune := !t.Primary && t.KeepLast > 0
@@ -1934,7 +1981,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
Service: r, Service: r,
}) })
logger.Info("backup tier armed", "target", t.TargetID, "cadence", t.Cadence.String(), logger.Info("backup tier armed", "target", t.TargetID, "cadence", t.Cadence.String(),
"keep_last", t.KeepLast, "wait_timeout", t.WaitTimeout.String(), "keep_last", t.KeepLast, "prune", prune, "wait_timeout", t.WaitTimeout.String(),
"prune_pbs_allowed", allowPBSPrune, "primary", t.Primary) "prune_pbs_allowed", allowPBSPrune, "primary", t.Primary)
} }
// Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown // Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown
@@ -2826,6 +2873,20 @@ type escrowCeremonyOpts struct {
offline bool offline bool
upload bool upload bool
identityBundlePath string identityBundlePath string
// deferUpload (R-978): with upload, build the hub upload body and hand it back in
// CeremonyOutput.PendingUpload instead of sending it. Set for json mode — the controller-driven
// ceremony, whose daemon uploads the package only when the customer claims the new code.
deferUpload bool
}
// escrowOptsFromCLI maps the escrow-create flags onto the ceremony options. json mode defers the upload
// (R-978, pinned by TestEscrowOptsFromCLI_JSONModeDefersUpload); text mode — an operator at a terminal,
// who sees R at once — keeps the immediate upload.
func escrowOptsFromCLI(storage string, paperkey, offline, upload bool, identityBundlePath string, jsonMode bool) escrowCeremonyOpts {
return escrowCeremonyOpts{
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
identityBundlePath: identityBundlePath, deferUpload: upload && jsonMode,
}
} }
// escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the // escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the
@@ -2976,16 +3037,36 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
ResticPwSealed: resticStaged, ResticPwSealed: resticStaged,
} }
if opts.upload { if opts.upload {
if err := uploadEscrowBlob(ctx, cfg, res, resticPwSHA256); err != nil { pending, uploaded, err := deliverEscrowPackage(ctx, cfg, res, resticPwSHA256, opts.deferUpload, putEscrowBody)
if err != nil {
// R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets // R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets
// the text shell keep the pre-extraction order (R surfaced, THEN the failure). // the text shell keep the pre-extraction order (R surfaced, THEN the failure).
return out, &escrowCeremonyErr{kind: "upload", err: err} return out, &escrowCeremonyErr{kind: "upload", err: err}
} }
out.Sum.Uploaded = true out.Sum.PendingUpload, out.Sum.Uploaded = pending, uploaded
} }
return out, nil return out, nil
} }
// deliverEscrowPackage builds the hub upload body and either sends it now (text mode) or returns it for
// the caller to send later (deferUpload — json mode, R-978: the daemon sends it when the customer claims
// the code). A hub that is not configured fails in both modes, as before: a package that can never be
// delivered is not a ceremony. put is the sender seam (putEscrowBody in production).
func deliverEscrowPackage(ctx context.Context, cfg config.Config, res escrow.CreateResult, resticPwSHA256 string, deferUpload bool,
put func(ctx context.Context, cfg config.Config, body []byte) error) (pending []byte, uploaded bool, err error) {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return nil, false, fmt.Errorf("hub not configured (url/host_id/api_key)")
}
body := escrowUploadBody(res, resticPwSHA256)
if deferUpload {
return body, false, nil
}
if err := put(ctx, cfg, body); err != nil {
return nil, false, err
}
return nil, true, nil
}
// printEscrowTextBanner prints the text-mode header line (shared by the success and the // printEscrowTextBanner prints the text-mode header line (shared by the success and the
// post-banner failure paths so the pre-extraction output stays byte-identical — including the // post-banner failure paths so the pre-extraction output stays byte-identical — including the
// posture: the historical banner always printed the DEFAULT posture, not the result's). // posture: the historical banner always printed the DEFAULT posture, not the result's).
@@ -3049,10 +3130,7 @@ func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slo
return 2 return 2
} }
out, cerr := escrowCeremony(ctx, cfg, logger, escrowCeremonyOpts{ out, cerr := escrowCeremony(ctx, cfg, logger, escrowOptsFromCLI(storage, paperkey, offline, upload, identityBundlePath, jsonMode))
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
identityBundlePath: identityBundlePath,
})
if cerr != nil { if cerr != nil {
switch cerr.kind { switch cerr.kind {
case "usage": case "usage":
@@ -3090,6 +3168,9 @@ func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slo
if out.Sum.Uploaded { if out.Sum.Uploaded {
fmt.Fprintln(os.Stderr, " uploaded the opaque blob(s) to the hub (host record); the hub cannot open them") fmt.Fprintln(os.Stderr, " uploaded the opaque blob(s) to the hub (host record); the hub cannot open them")
} }
if len(out.Sum.PendingUpload) > 0 {
fmt.Fprintln(os.Stderr, " package prepared, NOT uploaded: the agent uploads it when the new code is claimed (R-978)")
}
wire := out.Sum wire := out.Sum
wire.RecoveryCode = out.R wire.RecoveryCode = out.R
if err := json.NewEncoder(os.Stdout).Encode(wire); err != nil { if err := json.NewEncoder(os.Stdout).Encode(wire); err != nil {
@@ -3243,13 +3324,10 @@ type escrowUploadRequest struct {
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"` ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
} }
// uploadEscrowBlob PUTs the opaque blob (and, for 10D, the identity blob) to // escrowUploadBody is the hub upload body for one ceremony's result: the opaque blob (and, for 10D, the
// the hub, authed with the per-host key. The hub stores ciphertext + non-secret fields; no usable // identity blob) plus non-secret fields. Ciphertext only — useless without R; no usable secret leaves
// secret leaves the agent. // the agent. putEscrowBody sends it, authed with the per-host key.
func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateResult, resticPwSHA256 string) error { func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)")
}
upReq := escrowUploadRequest{ upReq := escrowUploadRequest{
BlobB64: base64.StdEncoding.EncodeToString(res.Blob), BlobB64: base64.StdEncoding.EncodeToString(res.Blob),
KeyFingerprint: res.KeyFingerprint, KeyFingerprint: res.KeyFingerprint,
@@ -3261,6 +3339,15 @@ func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateR
upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob) upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob)
} }
body, _ := json.Marshal(upReq) body, _ := json.Marshal(upReq)
return body
}
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)")
}
url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow" url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow"
req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewReader(body)) req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewReader(body))
if err != nil { if err != nil {
+95 -5
View File
@@ -39,13 +39,25 @@ type savedSuccess struct {
target string target string
vmid int vmid int
at time.Time at time.Time
// recent (R-973) holds the start times of the newest successes (at most maxRecentSuccesses, newest
// last) — what the local-API backup cadence cap counts, so an agent restart does not reset the cap.
recent []time.Time
// refusedAt (R-973) is the newest time the cap refused a backup of this guest on this tier.
refusedAt time.Time
} }
// backupSuccessJSON is one entry on disk. // maxRecentSuccesses bounds the per-tier success history kept for the cadence cap (R-973). The cap
// allows at most 3 runs in a window, so 8 is ample.
const maxRecentSuccesses = 8
// backupSuccessJSON is one entry on disk. Recent and LastRefusedAt are additive (R-973): a file written
// before them reads as "no history", and an older agent ignores them.
type backupSuccessJSON struct { type backupSuccessJSON struct {
Target string `json:"target"` Target string `json:"target"`
VMID int `json:"vmid"` VMID int `json:"vmid"`
StartedAt string `json:"started_at"` StartedAt string `json:"started_at"`
Recent []string `json:"recent,omitempty"`
LastRefusedAt string `json:"last_refused_at,omitempty"`
} }
func backupStateKey(target string, vmid int) string { return target + "/" + strconv.Itoa(vmid) } func backupStateKey(target string, vmid int) string { return target + "/" + strconv.Itoa(vmid) }
@@ -67,7 +79,19 @@ func NewBackupSuccessState(path string) *BackupSuccessState {
if perr != nil { if perr != nil {
continue // one unreadable entry must not lose the others continue // one unreadable entry must not lose the others
} }
s.last[backupStateKey(e.Target, e.VMID)] = savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()} ss := savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()}
for _, r := range e.Recent {
if rt, rerr := time.Parse(time.RFC3339, r); rerr == nil {
ss.recent = append(ss.recent, rt.UTC())
}
}
if len(ss.recent) == 0 {
ss.recent = []time.Time{ss.at} // a pre-R-973 file: its one known success counts
}
if rt, rerr := time.Parse(time.RFC3339, e.LastRefusedAt); rerr == nil {
ss.refusedAt = rt.UTC()
}
s.last[backupStateKey(e.Target, e.VMID)] = ss
} }
return s return s
} }
@@ -85,13 +109,72 @@ func (s *BackupSuccessState) RecordBackupSuccess(target string, b hub.Backup) er
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
k := backupStateKey(target, b.VMID) k := backupStateKey(target, b.VMID)
if old, ok := s.last[k]; ok && !t.After(old.at) { old, had := s.last[k]
if had && !t.After(old.at) {
return nil return nil
} }
s.last[k] = savedSuccess{target: target, vmid: b.VMID, at: t.UTC()} ss := savedSuccess{target: target, vmid: b.VMID, at: t.UTC(), refusedAt: old.refusedAt}
ss.recent = append(append([]time.Time(nil), old.recent...), t.UTC())
if len(ss.recent) > maxRecentSuccesses {
ss.recent = ss.recent[len(ss.recent)-maxRecentSuccesses:]
}
s.last[k] = ss
return s.saveLocked() return s.saveLocked()
} }
// RecentBackupSuccesses returns the saved success start times for this tier and guest that are not
// older than since (R-973 — the cadence cap's restart-proof count).
func (s *BackupSuccessState) RecentBackupSuccesses(target string, vmid int, since time.Time) []time.Time {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
var out []time.Time
for _, t := range s.last[backupStateKey(target, vmid)].recent {
if !t.Before(since) {
out = append(out, t)
}
}
return out
}
// RecordBackupRefusal saves that the cadence cap refused a backup of this guest on this tier at at
// (R-973), so the operator signal it raises survives an agent restart.
func (s *BackupSuccessState) RecordBackupRefusal(target string, vmid int, at time.Time) error {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
k := backupStateKey(target, vmid)
ss, ok := s.last[k]
if !ok {
return nil // nothing to cap against without a success; the cap never refuses then
}
if !at.After(ss.refusedAt) {
return nil
}
ss.refusedAt = at.UTC()
s.last[k] = ss
return s.saveLocked()
}
// LastBackupRefusal returns the newest saved cadence-cap refusal across every tier and guest.
func (s *BackupSuccessState) LastBackupRefusal() (target string, vmid int, at time.Time, ok bool) {
if s == nil {
return "", 0, time.Time{}, false
}
s.mu.Lock()
defer s.mu.Unlock()
for _, e := range s.last {
if !e.refusedAt.IsZero() && e.refusedAt.After(at) {
target, vmid, at, ok = e.target, e.vmid, e.refusedAt, true
}
}
return target, vmid, at, ok
}
// LastKnownSuccess returns the newest saved success for this tier and guest (ok=false = none on file). // LastKnownSuccess returns the newest saved success for this tier and guest (ok=false = none on file).
func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Time, bool) { func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Time, bool) {
if s == nil { if s == nil {
@@ -129,7 +212,14 @@ func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
func (s *BackupSuccessState) saveLocked() error { func (s *BackupSuccessState) saveLocked() error {
entries := make([]backupSuccessJSON, 0, len(s.last)) entries := make([]backupSuccessJSON, 0, len(s.last))
for _, e := range s.last { for _, e := range s.last {
entries = append(entries, backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)}) j := backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)}
for _, r := range e.recent {
j.Recent = append(j.Recent, r.Format(time.RFC3339))
}
if !e.refusedAt.IsZero() {
j.LastRefusedAt = e.refusedAt.Format(time.RFC3339)
}
entries = append(entries, j)
} }
// Deterministic file content (Go's map order is random). // Deterministic file content (Go's map order is random).
sort.Slice(entries, func(i, j int) bool { sort.Slice(entries, func(i, j int) bool {
+37
View File
@@ -74,3 +74,40 @@ func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
t.Fatal("uncovered_volumes must marshal as [], not null") t.Fatal("uncovered_volumes must marshal as [], not null")
} }
} }
// R-973: the cadence cap's history and refusals survive a reopen; the history is bounded; a file
// written before R-973 counts its one known success (never "no history" for a tier that has a copy).
func TestBackupSuccessState_R973_RecentHistoryAndRefusalSurviveAReopen(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
base := time.Date(2026, 10, 18, 2, 30, 0, 0, time.UTC)
for i := 0; i < 10; i++ {
at := base.Add(time.Duration(i) * time.Hour)
if err := s.RecordBackupSuccess("local", hub.Backup{VMID: 9201, Success: true, StartedAt: at.Format(time.RFC3339)}); err != nil {
t.Fatal(err)
}
}
refused := base.Add(11 * time.Hour)
if err := s.RecordBackupRefusal("local", 9201, refused); err != nil {
t.Fatal(err)
}
r := NewBackupSuccessState(path)
all := r.RecentBackupSuccesses("local", 9201, time.Time{})
if len(all) != maxRecentSuccesses || !all[len(all)-1].Equal(base.Add(9*time.Hour)) {
t.Fatalf("recent history after a reopen: %v (want the newest %d, ending at +9h)", all, maxRecentSuccesses)
}
if got := r.RecentBackupSuccesses("local", 9201, base.Add(7*time.Hour)); len(got) != 3 {
t.Fatalf("successes since +7h: %d, want 3", len(got))
}
if tg, vm, at, ok := r.LastBackupRefusal(); !ok || tg != "local" || vm != 9201 || !at.Equal(refused) {
t.Fatalf("refusal after a reopen: %q %d %v %v", tg, vm, at, ok)
}
old := filepath.Join(t.TempDir(), "old.json")
if err := os.WriteFile(old, []byte(`[{"target":"felhom-pbs","vmid":9201,"started_at":"2026-10-17T03:00:00Z"}]`), 0o600); err != nil {
t.Fatal(err)
}
if got := NewBackupSuccessState(old).RecentBackupSuccesses("felhom-pbs", 9201, time.Time{}); len(got) != 1 {
t.Fatalf("a pre-R-973 file must count its one success: %v", got)
}
}
+1 -1
View File
@@ -2,9 +2,9 @@ package backup
import ( import (
"context" "context"
"fmt"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"io" "io"
"log/slog" "log/slog"
"testing" "testing"
+2 -2
View File
@@ -93,7 +93,7 @@ func TestPrune_KeepLastClamp(t *testing.T) {
t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set) t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set)
} }
} }
if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-last=3" { if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-daily=3" {
t.Fatalf("default PruneBackupsSpec = %q, want keep-last=3", spec) t.Fatalf("default PruneBackupsSpec = %q, want keep-daily=3 (R-1149)", spec)
} }
} }
+11 -8
View File
@@ -10,8 +10,9 @@ import (
) )
// R-1012: a snapshot backup that fails carries the orphan finding in its error — the text the controller mails to // R-1012: a snapshot backup that fails carries the orphan finding in its error — the text the controller mails to
// the operator (`whole_guest_backup_failed`). The check runs once before the run and once, FRESH, after the failure. // the operator (`whole_guest_backup_failed`). The check runs ONCE, after the failure — never before the run.
// RED-PROOF (P1/redproof-thincheck.txt): with explainFailure's append removed this test fails. // RED-PROOF (P1/redproof-thincheck.txt): with explainFailure's append removed this test fails. RED-PROOF (2026-10-17
// night, felhom-agent REPORT.md): with the v0.159.0 pre-run check restored, calls = 2 and this test fails.
func TestBackup_SnapshotFailure_NamesTheOrphan(t *testing.T) { func TestBackup_SnapshotFailure_NamesTheOrphan(t *testing.T) {
const why = "probable cause: thin pool pve/data holds device 4 that LVM does not know" const why = "probable cause: thin pool pve/data holds device 4 that LVM does not know"
for name, api := range map[string]*fakeBackupAPI{ for name, api := range map[string]*fakeBackupAPI{
@@ -29,15 +30,17 @@ func TestBackup_SnapshotFailure_NamesTheOrphan(t *testing.T) {
if !strings.Contains(rec.Error, why) { if !strings.Contains(rec.Error, why) {
t.Fatalf("the failure does not name the orphan: %q", rec.Error) t.Fatalf("the failure does not name the orphan: %q", rec.Error)
} }
if calls != 2 { if calls != 1 {
t.Fatalf("thin check ran %d times, want 2 (before the run and after the failure)", calls) t.Fatalf("thin check ran %d times, want 1 (after the failure only)", calls)
} }
}) })
} }
} }
// A successful backup keeps a clean record (the pre-check only logs); a stop-mode backup takes no snapshot, so the // A successful backup never runs the check (it takes a metadata snapshot of the live pool, and nobody needs the answer)
// check never runs; an unwired runner behaves exactly as before. // and keeps a clean record; a stop-mode backup takes no snapshot, so the check never runs; an unwired runner behaves
// exactly as before. RED-PROOF (2026-10-17 night): with the v0.159.0 pre-run check restored, the success leg fails
// with calls=1.
func TestBackup_ThinCheck_SilentWhenNotApplicable(t *testing.T) { func TestBackup_ThinCheck_SilentWhenNotApplicable(t *testing.T) {
ok := &fakeBackupAPI{vzdumpUPID: "UPID:x", ok := &fakeBackupAPI{vzdumpUPID: "UPID:x",
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-x", Content: "backup", VMID: 9201, Size: 100, CTime: 1}}} content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-x", Content: "backup", VMID: 9201, Size: 100, CTime: 1}}}
@@ -45,8 +48,8 @@ func TestBackup_ThinCheck_SilentWhenNotApplicable(t *testing.T) {
calls := 0 calls := 0
r.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" }) r.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
rec, err := r.Backup(context.Background(), 9201) rec, err := r.Backup(context.Background(), 9201)
if err != nil || !rec.Success || rec.Error != "" || calls != 1 { if err != nil || !rec.Success || rec.Error != "" || calls != 0 {
t.Fatalf("success: err=%v rec=%+v calls=%d (want one pre-check, clean record)", err, rec, calls) t.Fatalf("success: err=%v rec=%+v calls=%d (want no check on a successful backup, clean record)", err, rec, calls)
} }
stop := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", proxmox.ModeStop, "", "", quiet()) stop := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", proxmox.ModeStop, "", "", quiet())
+7 -9
View File
@@ -62,8 +62,9 @@ type BackupRunner struct {
// this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing. // this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing.
beforeBackup func(ctx context.Context, storage string, vmid int) beforeBackup func(ctx context.Context, storage string, vmid int)
// thinCheck (R-1012) names an orphan thin device in the guest's pools in plain words ("" = none / not checked). It // thinCheck (R-1012) names an orphan thin device in the guest's pools in plain words ("" = none / not checked). It
// runs before a snapshot-mode backup (a WARN in the log) and again when the vzdump fails, so the failure the // runs ONLY when a snapshot-mode vzdump has failed (start or task), so the failure the controller mails to the
// controller mails to the operator says why. It never removes anything. nil = not wired. // operator says why; a successful backup never runs it (it takes a metadata snapshot of the live pool). It never
// removes anything. nil = not wired.
thinCheck func(ctx context.Context, vmid int) string thinCheck func(ctx context.Context, vmid int) string
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
@@ -117,8 +118,8 @@ func (r *BackupRunner) SetBeforeBackup(fn func(ctx context.Context, storage stri
// SetThinCheck wires the R-1012 orphan check (see BackupRunner.thinCheck). // SetThinCheck wires the R-1012 orphan check (see BackupRunner.thinCheck).
func (r *BackupRunner) SetThinCheck(fn func(ctx context.Context, vmid int) string) { r.thinCheck = fn } func (r *BackupRunner) SetThinCheck(fn func(ctx context.Context, vmid int) string) { r.thinCheck = fn }
// explainFailure appends the orphan finding to a failed snapshot-mode backup's error. A FRESH check (not the one from // explainFailure appends the orphan finding to a failed snapshot-mode backup's error — the only place the check runs, so
// before the run): the failure itself may be what made the orphan. Bounded, so a stuck helper cannot hold the job. // it sees the pool as the failure left it. Bounded, so a stuck helper cannot hold the job.
func (r *BackupRunner) explainFailure(ctx context.Context, vmid int, rec *hub.Backup) { func (r *BackupRunner) explainFailure(ctx context.Context, vmid int, rec *hub.Backup) {
if r.thinCheck == nil || r.mode != proxmox.ModeSnapshot { if r.thinCheck == nil || r.mode != proxmox.ModeSnapshot {
return return
@@ -221,11 +222,8 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
return rec, fmt.Errorf("backup: %s", rec.Error) return rec, fmt.Errorf("backup: %s", rec.Error)
} }
if r.thinCheck != nil && r.mode == proxmox.ModeSnapshot { // R-1012: no orphan check here. It takes a metadata snapshot of the live pool, and its answer is only needed when the
if why := r.thinCheck(ctx, vmid); why != "" { // snapshot step has failed — explainFailure runs it then (pinned by TestBackup_ThinCheck_SilentWhenNotApplicable).
r.logger.Warn("backup: an orphan thin device may make this snapshot backup fail (R-1012)", "vmid", vmid, "finding", why)
}
}
upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{ upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes, VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune) PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
+1 -1
View File
@@ -91,7 +91,7 @@ var manifest = []Capability{
{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""}, {"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""}, {"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""}, {"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
{"thin-orphan-check", "orphan thin device check around a whole-guest backup (read-only, R-1012)", "/usr/local/sbin/felhom-thin-check", []string{"pve", "data"}, false, ""}, {"thin-orphan-check", "orphan thin device check after a failed whole-guest backup (read-only, R-1012)", "/usr/local/sbin/felhom-thin-check", []string{"pve", "data"}, false, ""},
// ---- Storage mount units (watchdog re-mount) ---- // ---- Storage mount units (watchdog re-mount) ----
{"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""}, {"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""},
+32
View File
@@ -180,3 +180,35 @@ func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers) t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
} }
} }
// R-1149 (operator decision D17, 2026-10-19): the local whole-guest tier keeps one restore point per day for N days
// (`--prune-backups keep-daily=N`), not the newest N; an additional tier keeps its explicit count.
//
// COMPANION RED-PROOF (observed 2026-10-19): make PruneSpec return keep-last for the primary → this fails with
// `primary tier prune = "keep-last=3", want "keep-daily=3"`. Restored.
func TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
LocalBackupRetention: 3,
ExtraTargets: []BackupTargetConfig{
{TargetID: "felhom-pbs", CadenceSeconds: 604800, KeepLast: 2},
{TargetID: "nas", CadenceSeconds: 86400},
},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 3 {
t.Fatalf("want 3 tiers, got %d", len(tiers))
}
for _, c := range []struct {
i int
want string
}{{0, "keep-daily=3"}, {1, "keep-last=2"}, {2, ""}} {
if got := tiers[c.i].PruneSpec(); got != c.want {
t.Fatalf("%s tier prune = %q, want %q", map[bool]string{true: "primary", false: tiers[c.i].TargetID}[tiers[c.i].Primary], got, c.want)
}
}
// The unset retention defaults to 3 days, never to „prune everything".
if got := (BackupConfig{}).PruneBackupsSpec(); got != "keep-daily=3" {
t.Fatalf("default local prune = %q, want keep-daily=3", got)
}
}
+21 -3
View File
@@ -391,7 +391,8 @@ type BackupConfig struct {
// hub-served per-guest policy is slice 10; this is the agent-local cadence. // hub-served per-guest policy is slice 10; this is the agent-local cadence.
BackupCadenceSeconds int `json:"backup_cadence_seconds"` BackupCadenceSeconds int `json:"backup_cadence_seconds"`
// LocalBackupRetention is keep-last=N for the per-run `--prune-backups` on a LOCAL vzdump target — // LocalBackupRetention is N for the per-run `--prune-backups keep-daily=N` on a LOCAL vzdump target (keep-last=N
// before R-1149) —
// so the agent's own local whole-guest backups can't pile up and refill root (the felhom-pve incident; // so the agent's own local whole-guest backups can't pile up and refill root (the felhom-pve incident;
// the host_disk + storage_fill checkers are the detectors, this is the preventive default). 0/unset → // the host_disk + storage_fill checkers are the detectors, this is the preventive default). 0/unset →
// default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup. // default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup.
@@ -538,9 +539,26 @@ func (b BackupConfig) KeepLast() int {
return b.LocalBackupRetention return b.LocalBackupRetention
} }
// PruneBackupsSpec returns the PVE `--prune-backups` value for the local vzdump (e.g. "keep-last=3"). // PruneBackupsSpec returns the PVE `--prune-backups` value for the local vzdump: "keep-daily=N" (R-1149, operator
// decision D17 2026-10-19 — one restore point per day for N days; `07` §6.1). keep-last=N had the same disk cost but
// let N extra runs on one day push out every older point; keep-daily keeps the newest copy of each of the N newest
// days that have one, so a flood inside one day replaces only that day's point. Pinned by
// TestBackupTierPruneSpec_R1149_LocalKeepsOnePerDay and, as the consequence, TestBackupCap_R1149_LocalFloodKeepsOlderDays.
func (b BackupConfig) PruneBackupsSpec() string { func (b BackupConfig) PruneBackupsSpec() string {
return fmt.Sprintf("keep-last=%d", b.KeepLast()) return fmt.Sprintf("keep-daily=%d", b.KeepLast())
}
// PruneSpec is this tier's per-run `--prune-backups` value, or "" for never prune. The PRIMARY (local) tier keeps one
// point per day (keep-daily, R-1149, from KeepLast() — always ≥1); an additional tier keeps its explicit keep_last
// count (a weekly off-site tier's „last 2" is two weeks, and PBS prunes stay opt-in per tier, R-82).
func (t BackupTier) PruneSpec() string {
if t.KeepLast <= 0 {
return ""
}
if t.Primary {
return fmt.Sprintf("keep-daily=%d", t.KeepLast)
}
return fmt.Sprintf("keep-last=%d", t.KeepLast)
} }
// BackupCadence returns the per-guest /backup/due window: positive as-is, else 24h default. // BackupCadence returns the per-guest /backup/due window: positive as-is, else 24h default.
+12 -2
View File
@@ -1,5 +1,7 @@
package escrow package escrow
import "encoding/json"
// Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by // Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by
// felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API // felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API
// re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers // re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers
@@ -43,7 +45,15 @@ type CeremonyOutput struct {
IdentityBlobBytes int `json:"identity_blob_bytes"` IdentityBlobBytes int `json:"identity_blob_bytes"`
ResticPwSealed bool `json:"restic_pw_sealed"` ResticPwSealed bool `json:"restic_pw_sealed"`
Uploaded bool `json:"uploaded"` Uploaded bool `json:"uploaded"`
// PendingUpload (version 2, R-978) is the hub upload body — the R-wrapped blobs (ciphertext), the key
// fingerprint and the sealed password's hash — that json mode hands back INSTEAD of uploading. The agent
// daemon uploads it only when the customer claims the new recovery code, so a ceremony that is started
// and abandoned never replaces the package the customer's previous code opens. Not a secret without R,
// but never logged. Uploaded is false whenever this is set.
PendingUpload json.RawMessage `json:"pending_upload,omitempty"`
} }
// CeremonyOutputVersion is the current CeremonyOutput.Version value. // CeremonyOutputVersion is the current CeremonyOutput.Version value. 2 (R-978): json mode no longer
const CeremonyOutputVersion = 1 // uploads; it returns PendingUpload. The daemon refuses any other version (a binary/daemon mismatch
// fails the ceremony instead of losing a package).
const CeremonyOutputVersion = 2
+22 -1
View File
@@ -22,6 +22,11 @@ type Loop struct {
mu sync.Mutex mu sync.Mutex
fetched bool // a desired-state document has been delivered (false = never a teardown signal) fetched bool // a desired-state document has been delivered (false = never a teardown signal)
block *hub.WireWireguard block *hub.WireWireguard
// passed (R-1194) is set once the first reconcile pass has run. Until then the instance has never been rendered,
// so its health says nothing — and the first host report, sent before that pass, used to read „config invalid,
// port 0" and raise a false oob_degraded on EVERY fresh box (rehearsal 2026-10-20; reproduced 2026-10-21: the
// report at 20:34:00 read an absent config, the apply landed 0.4 s later).
passed bool
nudge chan struct{} nudge chan struct{}
} }
@@ -85,6 +90,7 @@ func (l *Loop) reconcile(ctx context.Context) {
if !fetched { if !fetched {
applyBlock = nil applyBlock = nil
} }
defer l.markPassed()
port, err := l.mgr.Apply(ctx, applyBlock) port, err := l.mgr.Apply(ctx, applyBlock)
if err != nil { if err != nil {
return // Apply logged; a claim/exhaustion or install error — retry next tick return // Apply logged; a claim/exhaustion or install error — retry next tick
@@ -106,8 +112,23 @@ func oobPeerIP(block *hub.WireWireguard) string {
return block.OOBPeerIP return block.OOBPeerIP
} }
// OOBStatus implements the hub collector's reporter seam (Part 4). // markPassed records that a reconcile pass has run (R-1194) — after a failed Apply too, so a real, lasting failure
// is still reported from the next report on.
func (l *Loop) markPassed() {
l.mu.Lock()
l.passed = true
l.mu.Unlock()
}
// OOBStatus implements the hub collector's reporter seam (Part 4). nil — no stanza, which the hub does not evaluate —
// until the first reconcile pass has run (R-1194).
func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus { func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus {
l.mu.Lock()
passed := l.passed
l.mu.Unlock()
if !passed {
return nil
}
_, block := l.snapshot() _, block := l.snapshot()
return l.mgr.Status(ctx, block) return l.mgr.Status(ctx, block)
} }
@@ -0,0 +1,29 @@
package felhomsshd
import (
"context"
"testing"
"time"
)
// R-1194: on every fresh box the first host report — sent before the OOB loop's first pass — read „config invalid,
// port 0" (the config was not rendered yet) and the hub mailed oob_degraded. Reproduced 2026-10-21 on a fresh box
// (felhom.eu audits/night-2026-10-21/P5-checks/): the report at 20:34:00, the apply 0.4 s later, the daemon healthy.
// The loop now reports NO stanza (the hub does not evaluate a missing one) until a pass has run — a failed pass
// counts, so a real lasting failure is still reported from the next report on.
// RED-PROOF: with OOBStatus not gated on `passed`, the first assertion fails (a stanza with config_invalid).
func TestR1194_NoOOBStanzaBeforeTheFirstPass(t *testing.T) {
r := newScriptRunner()
m := NewManager(r, t.TempDir(), nil)
m.isActive = func(context.Context) bool { return false }
m.isFailed = func(context.Context) bool { return false }
m.now = func() time.Time { return time.Unix(1783270000, 0) }
l := NewLoop(m, nil, time.Minute, nil)
if st := l.OOBStatus(context.Background()); st != nil {
t.Fatalf("a stanza before the first pass reads a config that was never rendered: %+v", st)
}
l.reconcile(context.Background()) // fails or succeeds — either way a pass has run
if st := l.OOBStatus(context.Background()); st == nil {
t.Fatal("after the first pass the health must be reported (a real failure must reach the hub)")
}
}
+10 -3
View File
@@ -14,6 +14,13 @@ const (
TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED
TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which) TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which)
TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting) TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting)
// TunnelAbsent (R-1070, agent Unreleased 2026-10-19): the guest answered and has NO cloudflared container — the
// controller makes none when no tunnel token is configured (a LAN-only or test box). Not a failure by itself: the
// hub never alarms on it, and the OS-update host rule reads it as „not applicable" only when the container was
// also absent before the update (HostHealthVerdict). Docker 29 prints „error: no such object" in lower case, so
// before this state existed the case-sensitive match fell through to `unknown` — which failed the host rule on
// every box without a tunnel (Tester4, 2026-10-15).
TunnelAbsent = "absent"
) )
// CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec. // CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec.
@@ -53,7 +60,7 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
return TunnelUnknown, "no running customer guest" return TunnelUnknown, "no running customer guest"
} }
worst, wdetail := "", "" worst, wdetail := "", ""
rank := map[string]int{TunnelRunning: 0, TunnelUnknown: 1, TunnelNotRunning: 2} rank := map[string]int{TunnelRunning: 0, TunnelAbsent: 1, TunnelUnknown: 2, TunnelNotRunning: 3}
for _, v := range vmids { for _, v := range vmids {
out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared") out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared")
st, d := ClassifyTunnel(string(out), string(errOut), err) st, d := ClassifyTunnel(string(out), string(errOut), err)
@@ -71,8 +78,8 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
func ClassifyTunnel(stdout, stderr string, err error) (string, string) { func ClassifyTunnel(stdout, stderr string, err error) (string, string) {
out := strings.TrimSpace(stdout) out := strings.TrimSpace(stdout)
if err != nil || out == "" { if err != nil || out == "" {
if strings.Contains(stderr, "No such object") || strings.Contains(stderr, "No such container") { if e := strings.ToLower(stderr); strings.Contains(e, "no such object") || strings.Contains(e, "no such container") {
return TunnelNotRunning, "no cloudflared container in the guest" return TunnelAbsent, "no cloudflared container in the guest (no tunnel set up on this box)"
} }
return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err) return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err)
} }
+4 -1
View File
@@ -20,7 +20,10 @@ func TestClassifyTunnel(t *testing.T) {
{"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"}, {"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"},
{"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"}, {"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"},
{"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"}, {"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelNotRunning, "no cloudflared container"}, {"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
// R-1070: Docker 29's wording, measured on demo-hp 2026-10-19 (`docker inspect` of a missing name).
{"absent, docker 29", "", "error: no such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no tunnel set up"},
{"absent, no such container", "", "Error response from daemon: No such container: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
{"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"}, {"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"},
{"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"}, {"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"},
{"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"}, {"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"},
+7 -1
View File
@@ -11,6 +11,7 @@ import (
"os" "os"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability" "gitea.dooplex.hu/admin/felhom-agent/internal/capability"
@@ -136,6 +137,8 @@ type Collector struct {
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256 selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
// swapShortWarned (R-1192): the host-swap-short WARN is logged once per state change, not every report.
swapShortWarned atomic.Bool
} }
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is // NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
@@ -375,6 +378,8 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
} }
host := hostMetrics(c.px.Node(), ns) host := hostMetrics(c.px.Node(), ns)
guests := c.collectGuests(ctx)
c.applySwap(&host, ns, guests) // R-1192: host swap vs the guests' swap allowance
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable
report := &HostReport{ report := &HostReport{
@@ -383,7 +388,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
AgentVersion: c.agentVersion, AgentVersion: c.agentVersion,
AgentSHA256: c.agentSHA256(), AgentSHA256: c.agentSHA256(),
Host: host, Host: host,
Guests: c.collectGuests(ctx), Guests: guests,
// storage_targets populated this slice (slice 5) via the observer; the rest stay // storage_targets populated this slice (slice 5) via the observer; the rest stay
// defined-but-empty (slice 6). Non-nil so they marshal as []. // defined-but-empty (slice 6). Non-nil so they marshal as [].
StorageTargets: c.collectStorage(ctx), StorageTargets: c.collectStorage(ctx),
@@ -567,6 +572,7 @@ func (c *Collector) collectGuests(ctx context.Context) []Guest {
Cores: cfg.Cores, Cores: cfg.Cores,
MemoryBytes: g.MaxMem, MemoryBytes: g.MaxMem,
DiskBytes: g.MaxDisk, DiskBytes: g.MaxDisk,
SwapBytes: cfg.Swap * (1 << 20),
} }
} }
guests = append(guests, entry) guests = append(guests, entry)
+39
View File
@@ -0,0 +1,39 @@
package hub
import (
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the guest swap fix (every guest may swap as much as its RAM,
// localapi/guestswap.go) moves a memory burst onto the HOST's swap. Measured on demo-hp the same day: our installer
// gives an 8 GB / 256 GB machine 7.75 GB of host swap, so a 4096 MB guest's 4096 MB swap fits. A host with no swap
// (a BYO install, or swap turned off) gets nothing from the fix. So the report carries both numbers, the agent says it
// once in its log, and the hub raises host_swap_short. The agent never creates host swap itself: changing a host its
// owner set up is a decision (felhom.eu register).
// HostSwapShort reports whether the host's swap is below the guests' swap allowance. A host whose guests have no
// swap allowance (0) is never short.
func HostSwapShort(hostMB, guestMB int64) bool { return guestMB > 0 && hostMB < guestMB }
// applySwap fills the host block's swap numbers from the node status and the guests' specs, and logs the short
// state once per change.
func (c *Collector) applySwap(h *HostMetrics, ns proxmox.NodeStatus, guests []Guest) {
hostMB := ns.Swap.Total / (1 << 20)
var guestMB int64
for _, g := range guests {
if g.Spec != nil {
guestMB += g.Spec.SwapBytes / (1 << 20)
}
}
h.HostSwapMB, h.GuestSwapMB = &hostMB, &guestMB
if HostSwapShort(hostMB, guestMB) {
if c.swapShortWarned.CompareAndSwap(false, true) {
c.logger.Warn("host-swap: the host's swap is below its guests' swap allowance — a memory burst can freeze the box (R-1192); the hub is told (host_swap_short)",
"host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
return
}
if c.swapShortWarned.CompareAndSwap(true, false) {
c.logger.Info("host-swap: the host's swap covers its guests' swap allowance again", "host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
}
+42
View File
@@ -0,0 +1,42 @@
package hub
import (
"bytes"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the host report carries the host's swap and the guests' swap allowance, and the
// agent says ONCE in its log when the host's swap is below it (and once when it is covered again).
// RED-PROOF: with applySwap not called the numbers are nil; with the CompareAndSwap removed the WARN repeats.
func TestR1192_HostReportCarriesSwapAndWarnsOnce(t *testing.T) {
var buf bytes.Buffer
c := &Collector{logger: slog.New(slog.NewTextHandler(&buf, nil))}
guests := []Guest{{VMID: 9201, Spec: &GuestSpec{SwapBytes: 4096 << 20}}, {VMID: 9202, Spec: nil}}
var ns proxmox.NodeStatus // a host with no swap
for i := 0; i < 3; i++ {
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if h.HostSwapMB == nil || h.GuestSwapMB == nil || *h.HostSwapMB != 0 || *h.GuestSwapMB != 4096 {
t.Fatalf("report numbers: host %v guest %v", h.HostSwapMB, h.GuestSwapMB)
}
}
if n := strings.Count(buf.String(), "host-swap: the host's swap is below"); n != 1 {
t.Fatalf("the short WARN must be logged once, got %d:\n%s", n, buf.String())
}
ns.Swap.Total = 7935 << 20 // the installer's 8 GB host
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if *h.HostSwapMB != 7935 || HostSwapShort(*h.HostSwapMB, *h.GuestSwapMB) {
t.Fatalf("a covered host must not be short: %v / %v", *h.HostSwapMB, *h.GuestSwapMB)
}
if !strings.Contains(buf.String(), "covers its guests' swap allowance again") {
t.Fatalf("the recovery must be said once:\n%s", buf.String())
}
if HostSwapShort(0, 0) {
t.Fatal("a host whose guests have no swap allowance is never short")
}
}
+11 -1
View File
@@ -292,6 +292,10 @@ type WireguardStatus struct {
Registered bool `json:"registered"` // the registration marker exists Registered bool `json:"registered"` // the registration marker exists
Active bool `json:"active"` // wg-quick@wg-felhom is-active Active bool `json:"active"` // wg-quick@wg-felhom is-active
LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"` LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"`
// HandshakeFutureS (R-1204): seconds the latest handshake lies in the FUTURE by this box's clock — set (> 0) only
// then. A box that ran once with its clock ahead keeps a handshake stamped in the future; WireGuard on ep0 then
// refuses every newer handshake, and LastHandshakeAgeS (clamped at 0) reads „just now" for ever. The hub alarms on it.
HandshakeFutureS *int64 `json:"handshake_future_s,omitempty"`
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32" AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
} }
@@ -338,6 +342,11 @@ type HostMetrics struct {
// //
// Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift. // Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift.
WrapperSHA256 string `json:"wrapper_sha256,omitempty"` WrapperSHA256 string `json:"wrapper_sha256,omitempty"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up, night 2026-10-21): the host's swap (Proxmox node status — the
// kernel's SwapTotal) and the sum of its guests' swap allowance. The guest swap fix (swap = RAM) needs swap ON THE
// HOST behind it; the hub raises host_swap_short when the first is below the second. nil = not read (older agent).
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
} }
// Guest is one LXC. The agent reports vmid; the hub derives the guest PK // Guest is one LXC. The agent reports vmid; the hub derives the guest PK
@@ -355,11 +364,12 @@ type GuestSpec struct {
Cores int `json:"cores"` Cores int `json:"cores"`
MemoryBytes int64 `json:"memory_bytes"` MemoryBytes int64 `json:"memory_bytes"`
DiskBytes int64 `json:"disk_bytes"` DiskBytes int64 `json:"disk_bytes"`
SwapBytes int64 `json:"swap_bytes,omitempty"` // R-1192: the guest's swap allowance (PVE config `swap`, MB → bytes)
} }
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest. // Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
type Cloudflared struct { type Cloudflared struct {
Status string `json:"status"` // running | not_running | unknown (TunnelRunning …) Status string `json:"status"` // running | not_running | unknown | absent (TunnelRunning …; absent since R-1070)
Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected" Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected"
} }
+146
View File
@@ -0,0 +1,146 @@
package localapi
import (
"fmt"
"net/http"
"sort"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// The whole-guest backup cadence cap (R-973, security review 2026-10-12).
//
// POST /backup is reachable from inside the guest, and every successful run prunes the tier
// (`--prune-backups keep-last=N` on a local tier; ep0's server-side keep-last 2 on the off-site tier). So
// a compromised controller calling it in a loop pushed every pre-compromise whole-guest restore point out
// within minutes — the threat `07` §8 row 10 counts the off-site tier as the answer to. The agent now
// refuses a run that the schedule in `07` §6 would never ask for:
//
// - a tier whose cadence is at most 24 h (the local tier): at most 3 successful runs in any rolling
// 24 h. The schedule is one night run; a „Mentés most" press makes a LOCAL copy and, since R-899, the
// night still takes its own after it, which can land inside the same 24 h as the previous night (the
// window opens anywhere in [W+2h, W+6h)). So a normal day needs at most 3: two nights plus one press.
// A second press inside 24 h is refused — the margin is spent on the night, which must never be refused
// because of the household's presses.
// - a longer tier (the 7-day off-site tier): at most 1 successful run per max(cadence − 24 h, cadence/2)
// — 6 days for 7 days. The controller asks /backup/due first and runs only when the newest copy is older
// than the cadence, and the button never targets this tier (`07` §6, controller v0.301.0), so a
// legitimate run never comes sooner; 24 h of slack absorbs clock and archive-time differences.
//
// Only SUCCESSES count — a failed run prunes nothing and must stay retryable. The count survives an agent
// restart: it reads the success history saved on disk (LastKnownBackupStore, the R-894 file) as well as
// this process's own record. A refusal answers 429, is logged at WARN, is saved, and raises the operator
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's operator-only
// `whole_guest_backup_refused` event since hub v0.159.0, R-1150). Pinned by TestBackupCap_* (backup_cap_test.go).
// backupCapRefusalHold is how long one refusal keeps the operator signal raised.
const backupCapRefusalHold = 24 * time.Hour
// backupCadenceCap returns the rolling window and the most successful runs allowed in it for a tier.
func backupCadenceCap(cadence time.Duration) (window time.Duration, max int) {
if cadence <= 0 {
cadence = defaultBackupCadence
}
if cadence <= 24*time.Hour {
return 24 * time.Hour, 3
}
w := cadence - 24*time.Hour
if w < cadence/2 {
w = cadence / 2
}
return w, 1
}
// recentCapRuns returns the start times of the successful runs on this tier for this guest inside the
// window ending now — this process's record and the saved one, each counted once. Caller holds no lock.
func (s *Server) recentCapRuns(key backupJobKey, since time.Time) []time.Time {
seen := map[int64]bool{}
var out []time.Time
add := func(t time.Time) {
if t.Before(since) || seen[t.Unix()] {
return
}
seen[t.Unix()] = true
out = append(out, t)
}
s.capMu.Lock()
for _, t := range s.capRuns[key] {
add(t)
}
s.capMu.Unlock()
if s.lastKnown != nil {
for _, t := range s.lastKnown.RecentBackupSuccesses(key.target, key.vmid, since) {
add(t)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Before(out[j]) })
return out
}
// recordCapRun notes a successful run for the cap (the saved half is written by RecordBackupSuccess).
func (s *Server) recordCapRun(key backupJobKey, b hub.Backup) {
t, err := time.Parse(time.RFC3339, b.StartedAt)
if err != nil {
t = s.now()
}
s.capMu.Lock()
defer s.capMu.Unlock()
if s.capRuns == nil {
s.capRuns = map[backupJobKey][]time.Time{}
}
runs := append(s.capRuns[key], t.UTC())
if len(runs) > 8 {
runs = runs[len(runs)-8:]
}
s.capRuns[key] = runs
}
// capRefuses applies the cap to a new run request. It writes the 429 and returns true when refused.
func (s *Server) capRefuses(w http.ResponseWriter, key backupJobKey, tier BackupTier) bool {
window, max := backupCadenceCap(tier.Cadence)
now := s.now()
runs := s.recentCapRuns(key, now.Add(-window))
if len(runs) < max {
return false
}
next := runs[len(runs)-max].Add(window)
s.capMu.Lock()
s.capRefusal = capRefusal{target: key.target, vmid: key.vmid, at: now.UTC()}
s.capMu.Unlock()
if s.lastKnown != nil {
if err := s.lastKnown.RecordBackupRefusal(key.target, key.vmid, now); err != nil {
s.logger.Warn("local-api: could not save the backup refusal on disk (R-973)", "vmid", key.vmid, "target", key.target, "err", err)
}
}
s.logger.Warn("local-api: backup REFUSED — more whole-guest backups than the schedule allows (R-973 cadence cap; check the guest)",
"vmid", key.vmid, "target", key.target, "runs_in_window", len(runs), "allowed", max,
"window_h", int(window.Hours()), "next_allowed", next.UTC().Format(time.RFC3339))
writeStatus(w, http.StatusTooManyRequests, false, nil, fmt.Sprintf(
"too many whole-guest backups on target %q: %d successful in the last %d h, the schedule allows %d — refused (the next one is allowed after %s)",
key.target, len(runs), int(window.Hours()), max, next.UTC().Format(time.RFC3339)))
return true
}
type capRefusal struct {
target string
vmid int
at time.Time
}
// BackupCapRefusal reports the newest cadence-cap refusal still inside the 24 h hold (this process's or
// the saved one) — the source of the operator signal in the host report (R-973). ok=false = none.
func (s *Server) BackupCapRefusal() (target string, vmid int, at time.Time, ok bool) {
s.capMu.Lock()
r := s.capRefusal
s.capMu.Unlock()
if s.lastKnown != nil {
if t, v, a, found := s.lastKnown.LastBackupRefusal(); found && a.After(r.at) {
r = capRefusal{target: t, vmid: v, at: a}
}
}
if r.at.IsZero() || s.now().Sub(r.at) > backupCapRefusalHold {
return "", 0, time.Time{}, false
}
return r.target, r.vmid, r.at, true
}
+377
View File
@@ -0,0 +1,377 @@
package localapi
import (
"context"
"fmt"
"io"
"log/slog"
"net/http"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-973 — the whole-guest backup cadence cap. A compromised controller calling POST /backup in a loop
// pushed out every pre-compromise restore point: each success prunes the tier. These tests model the tier
// the way Proxmox and ep0 keep it (keep-last N, pruned after each success) and ask what is LEFT.
// keepLastTier is a whole-guest tier: every successful run adds an archive stamped with the clock and
// prunes to keepLast, as vzdump's --prune-backups keep-last (local) and ep0's keep-last 2 (off-site) do.
type keepLastTier struct {
mu sync.Mutex
clock *testClock
keepLast int
// spec, when set, is the `--prune-backups` value the agent passes (R-1149) and the tier prunes the way Proxmox
// reads it: keep-last=N → the newest N; keep-daily=N → the newest copy of each of the N newest days with one.
spec string
t *testing.T
archives []time.Time
fail bool
runs int
}
// pruneBySpec applies a Proxmox `--prune-backups` spec to archives (oldest first) and returns what is kept.
func pruneBySpec(t *testing.T, archives []time.Time, spec string) []time.Time {
var n int
var mode string
if _, err := fmt.Sscanf(spec, "keep-last=%d", &n); err == nil {
mode = "last"
} else if _, err := fmt.Sscanf(spec, "keep-daily=%d", &n); err == nil {
mode = "daily"
} else {
t.Fatalf("the model does not know the prune spec %q", spec)
}
var kept []time.Time
days := map[string]bool{}
for i := len(archives) - 1; i >= 0; i-- { // newest first, as Proxmox walks them
a := archives[i]
switch mode {
case "last":
if len(kept) < n {
kept = append([]time.Time{a}, kept...)
}
case "daily":
d := a.Format("2006-01-02")
if !days[d] && len(days) < n {
days[d] = true
kept = append([]time.Time{a}, kept...)
}
}
}
return kept
}
func (k *keepLastTier) BackupWithSnapshotHook(_ context.Context, vmid int, _ func()) (hub.Backup, error) {
k.mu.Lock()
defer k.mu.Unlock()
k.runs++
now := k.clock.Now()
if k.fail {
return hub.Backup{VMID: vmid, StartedAt: now.Format(time.RFC3339)}, fmt.Errorf("vzdump failed")
}
k.archives = append(k.archives, now)
if k.spec != "" {
k.archives = pruneBySpec(k.t, k.archives, k.spec)
} else if len(k.archives) > k.keepLast {
k.archives = k.archives[len(k.archives)-k.keepLast:]
}
return hub.Backup{VMID: vmid, Success: true, Archive: "x", StartedAt: now.Format(time.RFC3339)}, nil
}
// oldestBefore reports whether the tier still holds an archive made before t (a pre-compromise point).
func (k *keepLastTier) oldestBefore(t time.Time) bool {
k.mu.Lock()
defer k.mu.Unlock()
return len(k.archives) > 0 && k.archives[0].Before(t)
}
func (k *keepLastTier) count() (runs, kept int) {
k.mu.Lock()
defer k.mu.Unlock()
return k.runs, len(k.archives)
}
type testClock struct {
mu sync.Mutex
t time.Time
}
func (c *testClock) Now() time.Time { c.mu.Lock(); defer c.mu.Unlock(); return c.t }
func (c *testClock) Advance(d time.Duration) { c.mu.Lock(); c.t = c.t.Add(d); c.mu.Unlock() }
// capServer builds the two-tier box of `07` §6: local 24 h keep-last 3, off-site 7 d keep-last 2, the
// success history on disk at path (a second capServer on the same path is an agent restart).
func capServer(t *testing.T, path string, clock *testClock, local, offsite *keepLastTier) *Server {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local"}, {Name: "felhom-pbs"}}},
Tokens: staticTokens{"A": 8200},
BackupTiers: []BackupTier{
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: local},
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: offsite},
},
LastKnownBackups: backup.NewBackupSuccessState(path),
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatal(err)
}
srv.baseCtx = context.Background()
srv.now = clock.Now
return srv
}
// runBackup POSTs one backup and waits for the job to end. Returns the HTTP status of the POST.
func runBackup(t *testing.T, srv *Server, target string) int {
t.Helper()
rr := do(t, srv.Handler(), "POST", "/backup?target="+target, "A", "")
if rr.Code == http.StatusAccepted {
waitFor(t, func() bool {
srv.jobsMu.Lock()
defer srv.jobsMu.Unlock()
j := srv.jobs[backupJobKey{vmid: 8200, target: target}]
return j != nil && !backupInFlight(j.Phase)
})
}
return rr.Code
}
// The off-site tier — the one `07` §8 row 10 counts on against ransomware. Before the cap ten calls in a
// row left ep0 holding only the attacker's two copies. Now ONE runs (the tier was due), the other nine
// are refused, and the newest pre-compromise copy survives; five days of flooding later it still does.
//
// COMPANION RED-PROOF (observed 2026-10-18): make capRefuses always return false → this fails with
// "the off-site tier lost every pre-compromise copy: 10 runs" . Restored.
func TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy(t *testing.T) {
clock := &testClock{t: testNow}
local := &keepLastTier{clock: clock, keepLast: 3}
off := &keepLastTier{clock: clock, keepLast: 2}
path := filepath.Join(t.TempDir(), "backup-success-state.json")
srv := capServer(t, path, clock, local, off)
// The honest history: two weekly copies, the newest 7 days and 1 hour old (so the tier is due).
for _, age := range []time.Duration{14*24*time.Hour + time.Hour, 7*24*time.Hour + time.Hour} {
clock.t = testNow.Add(-age)
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusAccepted {
t.Fatalf("seeding the honest weekly copy: %d", code)
}
}
clock.t = testNow
compromise := testNow
refused := 0
for i := 0; i < 10; i++ {
if runBackup(t, srv, "felhom-pbs") == http.StatusTooManyRequests {
refused++
}
clock.Advance(time.Minute)
}
runs, _ := off.count()
if !off.oldestBefore(compromise) {
t.Fatalf("the off-site tier lost every pre-compromise copy: %d runs", runs-2)
}
if runs-2 != 1 || refused != 9 {
t.Fatalf("flood: %d runs, %d refused — want 1 run (the tier was due) and 9 refused", runs-2, refused)
}
for d := 0; d < 5; d++ { // five more days of flooding, every hour
for h := 0; h < 24; h++ {
clock.Advance(time.Hour)
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusTooManyRequests {
t.Fatalf("day %d hour %d: an off-site run was allowed %v after the last one", d, h, clock.Now().Sub(compromise))
}
}
}
if !off.oldestBefore(compromise) {
t.Fatal("five days of flooding pushed out the pre-compromise off-site copy")
}
}
// The local tier: a flood runs exactly 3 in a day (the schedule's own day: two nights + one press) and
// every further request is refused. Honest about the limit: with keep-last 3 those 3 replace the 3 kept
// points — the cap slows a flush from minutes to a day; it cannot keep a point a 3-per-day pace prunes.
func TestBackupCap_LocalFloodRunsAtMostThreeADay(t *testing.T) {
clock := &testClock{t: testNow}
local := &keepLastTier{clock: clock, keepLast: 3}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
refused := 0
for i := 0; i < 20; i++ {
if runBackup(t, srv, "local") == http.StatusTooManyRequests {
refused++
}
clock.Advance(10 * time.Minute)
}
if runs, _ := local.count(); runs != 3 || refused != 17 {
t.Fatalf("local flood over ~3 h: %d runs, %d refused — want 3 and 17", runs, refused)
}
clock.Advance(24 * time.Hour) // a day later the window has moved on
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("a run a day later must be allowed: %d", code)
}
}
// The schedule itself is never refused: a night at W+6h, a press at 10:00, the next night at W+2h (all
// three inside one rolling 24 h) run; a second press that day is refused; nights keep running after it.
func TestBackupCap_TheScheduledDayIsNeverRefused(t *testing.T) {
day := time.Date(2026, 6, 10, 0, 0, 0, 0, time.UTC)
clock := &testClock{t: day.Add(8*time.Hour + 29*time.Minute)} // W=02:30 → W+6h ≈ 08:29
local := &keepLastTier{clock: clock, keepLast: 3}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
steps := []struct {
at time.Time
want int
what string
}{
{day.Add(8*time.Hour + 29*time.Minute), http.StatusAccepted, "night 1 at W+6h"},
{day.Add(10 * time.Hour), http.StatusAccepted, "a press at 10:00"},
{day.Add(24*time.Hour + 4*time.Hour + 30*time.Minute), http.StatusAccepted, "night 2 at W+2h (20 h later)"},
{day.Add(24*time.Hour + 7*time.Hour), http.StatusTooManyRequests, "a second press inside the same 24 h"},
{day.Add(48*time.Hour + 4*time.Hour + 30*time.Minute), http.StatusAccepted, "night 3"},
}
for _, s := range steps {
clock.t = s.at
if got := runBackup(t, srv, "local"); got != s.want {
t.Fatalf("%s: got %d, want %d", s.what, got, s.want)
}
}
}
// A restart must not reset the cap: the history is read back from disk.
//
// COMPANION RED-PROOF (observed 2026-10-18): make recentCapRuns ignore s.lastKnown → this fails with
// "after a restart the 4th run in a day was allowed (got 202)". Restored.
func TestBackupCap_SurvivesARestart(t *testing.T) {
clock := &testClock{t: testNow}
path := filepath.Join(t.TempDir(), "s.json")
local := &keepLastTier{clock: clock, keepLast: 3}
srv := capServer(t, path, clock, local, &keepLastTier{clock: clock, keepLast: 2})
for i := 0; i < 3; i++ {
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("run %d: %d", i+1, code)
}
clock.Advance(time.Minute)
}
srv2 := capServer(t, path, clock, local, &keepLastTier{clock: clock, keepLast: 2}) // the restart
if code := runBackup(t, srv2, "local"); code != http.StatusTooManyRequests {
t.Fatalf("after a restart the 4th run in a day was allowed (got %d)", code)
}
}
// Failed runs prune nothing and must stay retryable: they do not count.
func TestBackupCap_FailuresDoNotCount(t *testing.T) {
clock := &testClock{t: testNow}
local := &keepLastTier{clock: clock, keepLast: 3, fail: true}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
for i := 0; i < 6; i++ {
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("failed attempt %d was refused: %d", i+1, code)
}
clock.Advance(15 * time.Minute)
}
local.mu.Lock()
local.fail = false
local.mu.Unlock()
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("a success after six failures was refused: %d", code)
}
}
// A refusal raises the operator signal for 24 h, and the signal survives a restart.
func TestBackupCap_RefusalRaisesTheOperatorSignal(t *testing.T) {
clock := &testClock{t: testNow}
path := filepath.Join(t.TempDir(), "s.json")
off := &keepLastTier{clock: clock, keepLast: 2}
srv := capServer(t, path, clock, &keepLastTier{clock: clock, keepLast: 3}, off)
if _, _, _, ok := srv.BackupCapRefusal(); ok {
t.Fatal("a fresh box must not raise the signal")
}
runBackup(t, srv, "felhom-pbs")
clock.Advance(time.Hour)
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusTooManyRequests {
t.Fatalf("second off-site run an hour later: %d, want 429", code)
}
target, vmid, at, ok := srv.BackupCapRefusal()
if !ok || target != "felhom-pbs" || vmid != 8200 || !at.Equal(clock.Now()) {
t.Fatalf("signal after a refusal: ok=%v target=%q vmid=%d at=%v", ok, target, vmid, at)
}
clock.Advance(23 * time.Hour)
srv2 := capServer(t, path, clock, &keepLastTier{clock: clock, keepLast: 3}, off) // a restart
if _, _, _, ok := srv2.BackupCapRefusal(); !ok {
t.Fatal("the signal did not survive a restart")
}
clock.Advance(2 * time.Hour)
if _, _, _, ok := srv2.BackupCapRefusal(); ok {
t.Fatal("the signal must clear 24 h after the last refusal")
}
}
func TestBackupCadenceCap(t *testing.T) {
cases := []struct {
cadence time.Duration
window time.Duration
max int
}{
{0, 24 * time.Hour, 3},
{24 * time.Hour, 24 * time.Hour, 3},
{12 * time.Hour, 24 * time.Hour, 3},
{7 * 24 * time.Hour, 6 * 24 * time.Hour, 1},
{36 * time.Hour, 18 * time.Hour, 1},
{30 * 24 * time.Hour, 29 * 24 * time.Hour, 1},
}
for _, c := range cases {
if w, m := backupCadenceCap(c.cadence); w != c.window || m != c.max {
t.Errorf("cadence %v: got %v/%d, want %v/%d", c.cadence, w, m, c.window, c.max)
}
}
}
// R-1149 — the consequence of keep-daily on the local tier, with the agent's REAL prune spec (config.BackupTiers →
// PruneSpec, what runDaemon hands the runner). Three honest nights, then a compromise at 01:00 that floods
// POST /backup for the rest of the day. The cap lets 3 runs through inside the rolling 24 h (two at once, one when
// last night's success ages out). With keep-last 3 those three were the only copies left; with keep-daily 3 the two
// earlier nights survive the whole day.
//
// COMPANION RED-PROOF (observed 2026-10-19): make BackupTier.PruneSpec return keep-last for the primary → this fails
// with "after a day of flooding the local tier holds 0 copies made before the compromise (kept […]) — want the 2
// earlier nights". Restored.
func TestBackupCap_R1149_LocalFloodKeepsOlderDays(t *testing.T) {
tiers, _ := config.BackupConfig{LocalBackupTarget: "local", LocalBackupRetention: 3}.BackupTiers()
spec := tiers[0].PruneSpec()
day := time.Date(2026, 6, 10, 0, 0, 0, 0, time.UTC)
clock := &testClock{t: day}
local := &keepLastTier{clock: clock, keepLast: 3, spec: spec, t: t}
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
for d := 3; d >= 1; d-- { // the nights of D-3, D-2, D-1 at 03:00
clock.t = day.Add(-time.Duration(d)*24*time.Hour + 3*time.Hour)
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
t.Fatalf("seeding night D-%d: %d", d, code)
}
}
compromise := day.Add(time.Hour)
ran := 0
for clock.t = compromise; clock.t.Before(day.Add(24 * time.Hour)); clock.Advance(10 * time.Minute) {
if runBackup(t, srv, "local") == http.StatusAccepted {
ran++
}
}
if ran != 3 {
t.Fatalf("the flood ran %d backups on day D — want 3 (the cap)", ran)
}
local.mu.Lock()
kept := append([]time.Time(nil), local.archives...)
local.mu.Unlock()
pre := 0
for _, a := range kept {
if a.Before(compromise) {
pre++
}
}
if pre != 2 {
t.Fatalf("after a day of flooding the local tier holds %d copies made before the compromise (kept %v) — want the 2 earlier nights", pre, kept)
}
}
+308 -12
View File
@@ -3,11 +3,15 @@ package localapi
import ( import (
"bytes" "bytes"
"context" "context"
"crypto/rand"
"crypto/subtle"
"encoding/json" "encoding/json"
"fmt" "fmt"
"math/big"
"net/http" "net/http"
"os/exec" "os/exec"
"strconv" "strconv"
"strings"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow" "gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
@@ -24,9 +28,23 @@ import (
// be structurally incapable of carrying R). // be structurally incapable of carrying R).
// - One claim, then the holder is zeroed. Unclaimed past the TTL → zeroed + phase // - One claim, then the holder is zeroed. Unclaimed past the TTL → zeroed + phase
// unclaimed_void ("R unclaimed → ceremony void → a re-run supersedes"). // unclaimed_void ("R unclaimed → ceremony void → a re-run supersedes").
// - Crash-safety is IN-MEMORY BY DESIGN: an agent restart loses R, which is SAFE (the blob is // - Crash-safety is IN-MEMORY BY DESIGN: an agent restart loses R AND the package it opens,
// on the hub; a re-run supersedes it). Status reporting "none" after a restart is the honest // which is SAFE (R-978: the package was never uploaded, so the hub still holds the previous one
// answer. No journal, deliberately — persistence is the one property R must never have. // and the customer's previous code still opens it; a re-run starts over). Status reporting "none"
// after a restart is the honest answer. No journal, deliberately — persistence is the one
// property R must never have.
// - R-978: the NEW package reaches the hub only at the claim, and the code is handed out only after
// the hub took it. Until then the previous package stays current — so a ceremony that is started
// and abandoned (never claimed, superseded, TTL, restart) leaves the customer's written-down code
// working. The TTL and a new start both wait while a claim's upload is in flight, so a package is
// never stored without its code being handed out by that same claim.
// - R-1151: a claim that asks for it (`{"defer_upload": true}`, controller ≥ 0.332.0) hands out the code
// and two word positions, and keeps the package until POST /escrow/ceremony/confirm brings those two
// words back — the household's typed-back check, now checked HERE, not only in the browser. A code that
// was shown and never typed back (the page closed) never replaces the hub's package; the confirm window
// (escrowConfirmTTL) ends it. Only the two words are kept for the check, never R. A claim WITHOUT the
// flag (a controller that has not updated) keeps the R-978 order: upload at the claim. Pinned by
// TestEscrowCeremony_R1151_*.
// - Nothing from stdout is ever logged; stderr (log-clean, spike §2.1) is tail-captured for // - Nothing from stdout is ever logged; stderr (log-clean, spike §2.1) is tail-captured for
// failure diagnostics only. // failure diagnostics only.
@@ -50,8 +68,16 @@ type EscrowCeremonyConfig struct {
// DRConfigured answers "is the DR tier applied on this box?" (pbsdr.Manager.DRConfigured, // DRConfigured answers "is the DR tier applied on this box?" (pbsdr.Manager.DRConfigured,
// late-bound). nil → reported not-applied. // late-bound). nil → reported not-applied.
DRConfigured func() bool DRConfigured func() bool
// Upload sends a ceremony's package (CeremonyOutput.PendingUpload, the hub upload body) to this
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
Upload func(ctx context.Context, pkg []byte) error
} }
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
// ceremony incl. its upload was measured at ~2.4 s).
const escrowUploadTimeout = 30 * time.Second
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until // ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned // parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
// spike-shaped output; production is runCeremonySubprocess. // spike-shaped output; production is runCeremonySubprocess.
@@ -78,8 +104,15 @@ const (
escrowPhaseDone = "done" escrowPhaseDone = "done"
escrowPhaseFailed = "failed" escrowPhaseFailed = "failed"
escrowPhaseVoid = "unclaimed_void" escrowPhaseVoid = "unclaimed_void"
// escrowPhaseUnconfirmed (R-1151): the code was shown but its words never came back inside the confirm
// window — the package was dropped, the hub's previous package (and the previous code) stays current.
escrowPhaseUnconfirmed = "unconfirmed_void"
) )
// escrowConfirmTTL is how long a claimed code waits for its typed-back words (R-1151). Writing ten words down
// by hand takes minutes; past this the package is dropped and the household starts again.
const escrowConfirmTTL = 30 * time.Minute
// escrowCeremonyTimeout bounds the whole subprocess run. Spike-measured ceremony ≈ 2.4 s incl. // escrowCeremonyTimeout bounds the whole subprocess run. Spike-measured ceremony ≈ 2.4 s incl.
// upload — 60 s is a ≥25× margin that still absorbs WAN upload latency. // upload — 60 s is a ≥25× margin that still absorbs WAN upload latency.
const escrowCeremonyTimeout = 60 * time.Second const escrowCeremonyTimeout = 60 * time.Second
@@ -129,6 +162,9 @@ func (s *Server) tryStartEscrowCeremony(job *escrowCeremonyJob) bool {
if s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning { if s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning {
return false return false
} }
if s.escrowClaiming {
return false // R-978: a claim is uploading this job's package — its code must reach the claimer
}
s.wipeEscrowRLocked() s.wipeEscrowRLocked()
cp := *job cp := *job
s.escrowJob = &cp s.escrowJob = &cp
@@ -137,7 +173,7 @@ func (s *Server) tryStartEscrowCeremony(job *escrowCeremonyJob) bool {
// finishEscrowCeremony records the terminal phase and (on success) arms the one-shot R holder. // finishEscrowCeremony records the terminal phase and (on success) arms the one-shot R holder.
// R is stored as a byte slice so the claim/TTL paths can zero it in place. // R is stored as a byte slice so the claim/TTL paths can zero it in place.
func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string) { func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string, pending []byte) {
s.escrowMu.Lock() s.escrowMu.Lock()
defer s.escrowMu.Unlock() defer s.escrowMu.Unlock()
job.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339) job.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
@@ -145,6 +181,7 @@ func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string) {
s.escrowJob = &cp s.escrowJob = &cp
if job.Phase == escrowPhaseDone && r != "" { if job.Phase == escrowPhaseDone && r != "" {
s.escrowR = []byte(r) s.escrowR = []byte(r)
s.escrowPending = append([]byte(nil), pending...)
s.escrowRClaimed = false s.escrowRClaimed = false
s.escrowRExpiry = s.nowFn().Add(escrowClaimTTL) s.escrowRExpiry = s.nowFn().Add(escrowClaimTTL)
// Active TTL belt: zero the holder even if nobody ever polls again. The lazy check in // Active TTL belt: zero the holder even if nobody ever polls again. The lazy check in
@@ -164,6 +201,53 @@ func (s *Server) finishEscrowCeremony(job *escrowCeremonyJob, r string) {
func (s *Server) wipeEscrowRLocked() { func (s *Server) wipeEscrowRLocked() {
zeroBytes(s.escrowR) zeroBytes(s.escrowR)
s.escrowR = nil s.escrowR = nil
s.escrowPending = nil // R-978: the package goes with its code — never uploaded once the code is gone
s.wipeEscrowConfirmLocked()
}
// wipeEscrowConfirmLocked drops the R-1151 typed-back state: the two kept words and the wait. Caller holds escrowMu.
func (s *Server) wipeEscrowConfirmLocked() {
for i := range s.escrowConfirmWords {
zeroBytes(s.escrowConfirmWords[i])
s.escrowConfirmWords[i] = nil
}
s.escrowAwaitConfirm = false
}
// checkEscrowConfirmTTLLocked ends a typed-back wait that ran past escrowConfirmTTL (R-1151): the package is
// dropped (never uploaded) and the job reads unconfirmed_void. Not while a confirm is uploading. Caller holds escrowMu.
func (s *Server) checkEscrowConfirmTTLLocked() {
if !s.escrowAwaitConfirm || s.escrowClaiming || !s.nowFn().After(s.escrowConfirmExpiry) {
return
}
s.escrowPending = nil
s.wipeEscrowConfirmLocked()
if s.escrowJob != nil {
s.escrowJob.Phase = escrowPhaseUnconfirmed
s.escrowJob.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
s.logger.Info("local-api: escrow code was not typed back in time — the new package was dropped; the previous one stays current (R-1151)",
"job_id", s.escrowJob.JobID)
}
}
// pickConfirmPositions draws two distinct 1-based word positions out of n (n ≥ 2), smaller first.
func pickConfirmPositions(n int) ([2]int, error) {
a, err := rand.Int(rand.Reader, big.NewInt(int64(n)))
if err != nil {
return [2]int{}, err
}
b, err := rand.Int(rand.Reader, big.NewInt(int64(n-1)))
if err != nil {
return [2]int{}, err
}
i, j := int(a.Int64()), int(b.Int64())
if j >= i {
j++
}
if j < i {
i, j = j, i
}
return [2]int{i + 1, j + 1}, nil
} }
// expireEscrowRLocked applies the TTL outcome: an armed, unclaimed holder is zeroed and the job // expireEscrowRLocked applies the TTL outcome: an armed, unclaimed holder is zeroed and the job
@@ -172,6 +256,9 @@ func (s *Server) expireEscrowRLocked() {
if s.escrowJob == nil || s.escrowJob.Phase != escrowPhaseDone || s.escrowRClaimed { if s.escrowJob == nil || s.escrowJob.Phase != escrowPhaseDone || s.escrowRClaimed {
return return
} }
if s.escrowClaiming {
return // R-978: the claim is uploading the package; voiding now would store a package nobody gets the code to
}
if len(s.escrowR) == 0 { if len(s.escrowR) == 0 {
return return
} }
@@ -217,7 +304,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
UpdatedAt: s.nowFn().UTC().Format(time.RFC3339), UpdatedAt: s.nowFn().UTC().Format(time.RFC3339),
} }
if !s.tryStartEscrowCeremony(job) { if !s.tryStartEscrowCeremony(job) {
writeErr(w, http.StatusConflict, "an escrow ceremony is already running") writeErr(w, http.StatusConflict, "an escrow ceremony is already running, or its code is being claimed")
return return
} }
s.logger.Info("local-api: escrow ceremony started (controller-driven)", "vmid", vmid, "job_id", job.JobID) s.logger.Info("local-api: escrow ceremony started (controller-driven)", "vmid", vmid, "job_id", job.JobID)
@@ -247,7 +334,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
detail += " | stderr: " + tail detail += " | stderr: " + tail
} }
job.Phase, job.Detail = escrowPhaseFailed, detail job.Phase, job.Detail = escrowPhaseFailed, detail
s.finishEscrowCeremony(job, "") s.finishEscrowCeremony(job, "", nil)
s.logger.Warn("local-api: escrow ceremony failed", "job_id", job.JobID, s.logger.Warn("local-api: escrow ceremony failed", "job_id", job.JobID,
"exit", exit, "duration_ms", time.Since(start).Milliseconds()) "exit", exit, "duration_ms", time.Since(start).Milliseconds())
return return
@@ -255,24 +342,34 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
var out escrow.CeremonyOutput var out escrow.CeremonyOutput
if err := json.Unmarshal(stdout, &out); err != nil { if err := json.Unmarshal(stdout, &out); err != nil {
job.Phase, job.Detail = escrowPhaseFailed, "ceremony output is not the expected JSON object" job.Phase, job.Detail = escrowPhaseFailed, "ceremony output is not the expected JSON object"
s.finishEscrowCeremony(job, "") s.finishEscrowCeremony(job, "", nil)
s.logger.Warn("local-api: escrow ceremony output unparseable (never logged)", "job_id", job.JobID) s.logger.Warn("local-api: escrow ceremony output unparseable (never logged)", "job_id", job.JobID)
return return
} }
if out.Version != escrow.CeremonyOutputVersion || out.RecoveryCode == "" { if out.Version != escrow.CeremonyOutputVersion || out.RecoveryCode == "" {
out.RecoveryCode = "" out.RecoveryCode = ""
job.Phase, job.Detail = escrowPhaseFailed, fmt.Sprintf("unexpected ceremony output (version %d)", out.Version) job.Phase, job.Detail = escrowPhaseFailed, fmt.Sprintf("unexpected ceremony output (version %d)", out.Version)
s.finishEscrowCeremony(job, "") s.finishEscrowCeremony(job, "", nil)
return
}
// R-978: the package must come back un-uploaded. An output that uploaded by itself (or carries no
// package) would mean the previous package is already gone, or the new one can never be stored.
if out.Uploaded || len(out.PendingUpload) == 0 {
out.RecoveryCode = ""
job.Phase, job.Detail = escrowPhaseFailed, "ceremony output carries no pending package"
s.finishEscrowCeremony(job, "", nil)
s.logger.Warn("local-api: escrow ceremony output has no pending package — failed (R-978)", "job_id", job.JobID,
"uploaded", out.Uploaded)
return return
} }
job.Phase = escrowPhaseDone job.Phase = escrowPhaseDone
job.KeyFingerprint = out.KeyFingerprint job.KeyFingerprint = out.KeyFingerprint
job.EntropyBits = out.EntropyBits job.EntropyBits = out.EntropyBits
job.ResticPwSealed = out.ResticPwSealed job.ResticPwSealed = out.ResticPwSealed
job.Uploaded = out.Uploaded job.Uploaded = false // R-978: uploaded by the claim
s.finishEscrowCeremony(job, out.RecoveryCode) s.finishEscrowCeremony(job, out.RecoveryCode, out.PendingUpload)
out.RecoveryCode = "" // drop the parsed reference promptly (GC caveat: best-effort) out.RecoveryCode = "" // drop the parsed reference promptly (GC caveat: best-effort)
s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot)", s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim, or at the typed-back confirm (R-1151)",
"job_id", job.JobID, "restic_pw_sealed", job.ResticPwSealed, "uploaded", job.Uploaded, "job_id", job.JobID, "restic_pw_sealed", job.ResticPwSealed, "uploaded", job.Uploaded,
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds()) "claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
}() }()
@@ -290,6 +387,7 @@ func (s *Server) handleEscrowCeremonyStatus(w http.ResponseWriter, r *http.Reque
} }
s.escrowMu.Lock() s.escrowMu.Lock()
s.checkEscrowTTLLocked() s.checkEscrowTTLLocked()
s.checkEscrowConfirmTTLLocked()
if s.escrowJob == nil { if s.escrowJob == nil {
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeOK(w, map[string]any{"vmid": vmid, "phase": escrowPhaseNone}) writeOK(w, map[string]any{"vmid": vmid, "phase": escrowPhaseNone})
@@ -304,6 +402,13 @@ func (s *Server) handleEscrowCeremonyStatus(w http.ResponseWriter, r *http.Reque
} }
} }
claimed := s.escrowRClaimed claimed := s.escrowRClaimed
awaiting := s.escrowAwaitConfirm
confirmIn := 0
if awaiting {
if d := s.escrowConfirmExpiry.Sub(s.nowFn()); d > 0 {
confirmIn = int(d.Seconds())
}
}
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeOK(w, map[string]any{ writeOK(w, map[string]any{
@@ -312,6 +417,7 @@ func (s *Server) handleEscrowCeremonyStatus(w http.ResponseWriter, r *http.Reque
"key_fingerprint": job.KeyFingerprint, "entropy_bits": job.EntropyBits, "key_fingerprint": job.KeyFingerprint, "entropy_bits": job.EntropyBits,
"restic_pw_sealed": job.ResticPwSealed, "uploaded": job.Uploaded, "restic_pw_sealed": job.ResticPwSealed, "uploaded": job.Uploaded,
"claimable": claimable, "claimed": claimed, "claim_expires_in_sec": expiresIn, "claimable": claimable, "claimed": claimed, "claim_expires_in_sec": expiresIn,
"awaiting_confirm": awaiting, "confirm_expires_in_sec": confirmIn,
"detail": job.Detail, "detail": job.Detail,
}) })
} }
@@ -324,6 +430,14 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
writeErr(w, http.StatusServiceUnavailable, "escrow ceremony not configured on this host") writeErr(w, http.StatusServiceUnavailable, "escrow ceremony not configured on this host")
return return
} }
// R-1151: a controller ≥ 0.332.0 asks for the package to wait for the typed-back words. An older controller
// sends `{}` (or nothing) and gets the R-978 order. Unknown fields are refused (decodeBody).
var req struct {
DeferUpload bool `json:"defer_upload"`
}
if !decodeBody(w, r, &req) {
return
}
s.escrowMu.Lock() s.escrowMu.Lock()
s.checkEscrowTTLLocked() s.checkEscrowTTLLocked()
switch { switch {
@@ -339,11 +453,101 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeErr(w, http.StatusConflict, "ceremony failed — nothing to claim") writeErr(w, http.StatusConflict, "ceremony failed — nothing to claim")
return return
case s.escrowClaiming:
s.escrowMu.Unlock()
writeErr(w, http.StatusConflict, "the recovery code is being claimed — try again in a moment")
return
case s.escrowRClaimed || len(s.escrowR) == 0 || s.escrowJob.Phase == escrowPhaseVoid: case s.escrowRClaimed || len(s.escrowR) == 0 || s.escrowJob.Phase == escrowPhaseVoid:
s.escrowMu.Unlock() s.escrowMu.Unlock()
writeErr(w, http.StatusGone, "the recovery code is no longer available (already claimed or expired) — run a new ceremony; the new code supersedes") writeErr(w, http.StatusGone, "the recovery code is no longer available (already claimed or expired) — run a new ceremony; the new code supersedes")
return return
} }
// R-1151: the deferred order — hand out the code and two word positions; the package waits for
// POST /escrow/ceremony/confirm. Needs an uploader (a confirm could never store it otherwise) and a code of ≥ 2
// words (always 10; anything else falls back to the R-978 order rather than to no check).
if req.DeferUpload && len(s.escrowPending) > 0 && s.escrowCeremony.Upload != nil {
words := strings.Split(string(s.escrowR), escrow.RecoveryCodeSep)
if len(words) >= 2 {
pos, perr := pickConfirmPositions(len(words))
if perr == nil {
s.escrowConfirmPos = pos
s.escrowConfirmWords = [2][]byte{[]byte(words[pos[0]-1]), []byte(words[pos[1]-1])}
for i := range words {
words[i] = ""
}
s.escrowAwaitConfirm = true
s.escrowConfirmExpiry = s.nowFn().Add(escrowConfirmTTL)
recovery := string(s.escrowR)
zeroBytes(s.escrowR) // R only — the package and the two words stay for the confirm
s.escrowR = nil
s.escrowRClaimed = true
jobID := s.escrowJob.JobID
s.escrowMu.Unlock()
time.AfterFunc(escrowConfirmTTL+time.Second, func() {
s.escrowMu.Lock()
defer s.escrowMu.Unlock()
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
s.checkEscrowConfirmTTLLocked()
}
})
s.logger.Info("local-api: escrow ceremony R claimed — the package waits for the typed-back words (R-1151)",
"vmid", vmid, "job_id", jobID, "confirm_ttl_s", int(escrowConfirmTTL.Seconds()))
w.Header().Set("Cache-Control", "no-store")
writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "recovery_code": recovery,
"confirm_required": true, "verify_positions": []int{pos[0], pos[1]},
"confirm_expires_in_sec": int(escrowConfirmTTL.Seconds())})
recovery = ""
_ = recovery
return
}
s.logger.Warn("local-api: escrow claim — could not draw the confirm positions; package uploads at the claim (R-978 order)", "err", perr)
}
}
// R-978: store the NEW package on the hub first; only then hand out the code that opens it. Until this
// upload succeeds the hub keeps the previous package, and the customer's previous code keeps working.
if len(s.escrowPending) > 0 {
if s.escrowCeremony.Upload == nil {
s.escrowMu.Unlock()
writeErr(w, http.StatusServiceUnavailable, "the recovery package cannot be stored on this host (no upload configured) — the previous code still works")
return
}
pkg := s.escrowPending
jobID := s.escrowJob.JobID
s.escrowClaiming = true
s.escrowMu.Unlock()
start := time.Now()
base := s.baseCtx
if base == nil {
base = context.Background()
}
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg)
cancel()
s.escrowMu.Lock()
s.escrowClaiming = false
if err != nil {
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return
}
s.escrowPending = nil
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
s.escrowJob.Uploaded = true
s.escrowJob.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
}
s.logger.Info("local-api: escrow claim — package uploaded to the hub; handing out the code (R-978)",
"vmid", vmid, "job_id", jobID, "duration_ms", time.Since(start).Milliseconds())
if len(s.escrowR) == 0 { // cannot happen (TTL and supersede wait for the claim) — never hand out nothing
s.escrowMu.Unlock()
s.logger.Error("local-api: escrow claim — code vanished during the upload (invariant broken)", "job_id", jobID)
writeErr(w, http.StatusInternalServerError, "the recovery code was lost during the claim — run a new ceremony")
return
}
}
recovery := string(s.escrowR) recovery := string(s.escrowR)
s.wipeEscrowRLocked() s.wipeEscrowRLocked()
s.escrowRClaimed = true s.escrowRClaimed = true
@@ -352,11 +556,103 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
s.logger.Info("local-api: escrow ceremony R claimed (one-shot; holder zeroed)", "vmid", vmid, "job_id", jobID) s.logger.Info("local-api: escrow ceremony R claimed (one-shot; holder zeroed)", "vmid", vmid, "job_id", jobID)
w.Header().Set("Cache-Control", "no-store") w.Header().Set("Cache-Control", "no-store")
writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "recovery_code": recovery}) writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "recovery_code": recovery, "confirm_required": false})
recovery = "" // drop the reference promptly (GC caveat: best-effort) recovery = "" // drop the reference promptly (GC caveat: best-effort)
_ = recovery _ = recovery
} }
// handleEscrowCeremonyConfirm is POST /escrow/ceremony/confirm {"words": [w1, w2]} (R-1151): the two words at the
// positions the claim named, as the household typed them back. Right → the package goes to the hub now (200); wrong →
// 422 and the wait goes on; nothing waiting → 404 (or 410 after the window ran out); a failed upload → 502 and the
// wait goes on (a retry may succeed). The words are compared in constant time, case and spaces aside.
func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Request, vmid int) {
if s.escrowCeremony == nil {
writeErr(w, http.StatusServiceUnavailable, "escrow ceremony not configured on this host")
return
}
var req struct {
Words []string `json:"words"`
}
if !decodeBody(w, r, &req) {
return
}
s.escrowMu.Lock()
s.checkEscrowConfirmTTLLocked()
switch {
case s.escrowClaiming:
s.escrowMu.Unlock()
writeErr(w, http.StatusConflict, "the recovery package is being stored — try again in a moment")
return
case !s.escrowAwaitConfirm:
gone := s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseUnconfirmed
s.escrowMu.Unlock()
if gone {
writeErr(w, http.StatusGone, "the new recovery code was not confirmed in time — the previous code still works; run a new ceremony")
return
}
writeErr(w, http.StatusNotFound, "no recovery code waits for confirmation")
return
case len(req.Words) != 2:
s.escrowMu.Unlock()
writeErr(w, http.StatusBadRequest, "two words are needed")
return
}
match := 1
for i := 0; i < 2; i++ {
typed := []byte(strings.ToLower(strings.TrimSpace(req.Words[i])))
match &= subtle.ConstantTimeCompare(typed, s.escrowConfirmWords[i])
zeroBytes(typed)
}
req.Words = nil
if match != 1 {
s.escrowMu.Unlock()
s.logger.Info("local-api: escrow confirm — the typed words do not match; still waiting (R-1151)", "vmid", vmid)
writeErr(w, http.StatusUnprocessableEntity, "the words do not match the recovery code")
return
}
if s.escrowCeremony.Upload == nil || len(s.escrowPending) == 0 { // the claim refuses to defer without both
s.escrowMu.Unlock()
writeErr(w, http.StatusServiceUnavailable, "the recovery package cannot be stored on this host")
return
}
pkg := s.escrowPending
jobID := ""
if s.escrowJob != nil {
jobID = s.escrowJob.JobID
}
s.escrowClaiming = true
s.escrowMu.Unlock()
start := time.Now()
base := s.baseCtx
if base == nil {
base = context.Background()
}
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg)
cancel()
s.escrowMu.Lock()
s.escrowClaiming = false
if err != nil {
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return
}
s.escrowPending = nil
s.wipeEscrowConfirmLocked()
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
s.escrowJob.Uploaded = true
s.escrowJob.UpdatedAt = s.nowFn().UTC().Format(time.RFC3339)
}
s.escrowMu.Unlock()
s.logger.Info("local-api: escrow confirm — words typed back; the new package is on the hub (R-1151)",
"vmid", vmid, "job_id", jobID, "duration_ms", time.Since(start).Milliseconds())
writeOK(w, map[string]any{"vmid": vmid, "job_id": jobID, "uploaded": true})
}
// handleEscrowPreflight is GET /escrow/preflight: the wizard's prerequisite checklist. Each item // handleEscrowPreflight is GET /escrow/preflight: the wizard's prerequisite checklist. Each item
// is {id, ok, detail}. Deliberately NOT checked: the PBS key file itself — /etc/pve/priv is 0700 // is {id, ok, detail}. Deliberately NOT checked: the PBS key file itself — /etc/pve/priv is 0700
// root and the daemon cannot stat it; a missing key fails the ceremony fast with a clear error // root and the daemon cannot stat it; a missing key fails the ceremony fast with a clear error
+76 -7
View File
@@ -2,12 +2,14 @@ package localapi
import ( import (
"context" "context"
"encoding/base64"
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http" "net/http"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"sync"
"sync/atomic" "sync/atomic"
"testing" "testing"
"time" "time"
@@ -19,25 +21,91 @@ import (
const testR = "canary-alpha-bravo-charlie-delta-echo-foxtrot-golf-hotel-india" const testR = "canary-alpha-bravo-charlie-delta-echo-foxtrot-golf-hotel-india"
// cannedCeremonyJSON is shaped exactly like the agent's --output=json object (spike §2.6 values). // cannedCeremonyJSON is shaped exactly like the agent's --output=json object (spike §2.6 values;
// version 2, R-978: the package comes back un-uploaded in pending_upload). The package is a MODEL of
// the R-wrapped blob: it "opens" with exactly the code it was sealed under (fakeEscrowHub.opensWith).
func cannedCeremonyJSON(r string) string { func cannedCeremonyJSON(r string) string {
return fmt.Sprintf(`{"version":1,"recovery_code":%q,"key_fingerprint":"f2:87:68:2a:88:50:16:01","entropy_bits":129,"blob_bytes":383,"identity_blob_bytes":450,"restic_pw_sealed":true,"uploaded":true}`, r) return fmt.Sprintf(`{"version":2,"recovery_code":%q,"key_fingerprint":"f2:87:68:2a:88:50:16:01","entropy_bits":129,"blob_bytes":383,"identity_blob_bytes":450,"restic_pw_sealed":true,"uploaded":false,"pending_upload":%s}`,
r, sealedPackage(r))
}
// sealedPackage is the model hub upload body for a ceremony that minted code r.
func sealedPackage(r string) string {
return fmt.Sprintf(`{"blob_b64":%q,"key_fingerprint":"f2:87:68:2a:88:50:16:01","posture":"zero_knowledge","created_at":"2026-06-10T12:00:00Z"}`,
base64.StdEncoding.EncodeToString([]byte("sealed-under:"+r)))
}
// fakeEscrowHub is the hub's current escrow package for this host (the PUT target). Upload replaces it,
// as the hub does when the sealed secrets are unchanged (the R-978 case: no retained copy).
type fakeEscrowHub struct {
mu sync.Mutex
current []byte
uploads int
err error // non-nil → the upload fails, nothing stored
gate chan struct{} // non-nil → the upload blocks until closed
entered chan struct{} // non-nil → closed when an upload starts
}
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
if f.entered != nil {
close(f.entered)
f.entered = nil
}
if f.gate != nil {
<-f.gate
}
f.mu.Lock()
defer f.mu.Unlock()
if f.err != nil {
return f.err
}
f.current = append([]byte(nil), pkg...)
f.uploads++
return nil
}
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
func (f *fakeEscrowHub) opensWith(r string) bool {
f.mu.Lock()
defer f.mu.Unlock()
var pkg struct {
BlobB64 string `json:"blob_b64"`
}
if json.Unmarshal(f.current, &pkg) != nil {
return false
}
raw, err := base64.StdEncoding.DecodeString(pkg.BlobB64)
return err == nil && string(raw) == "sealed-under:"+r
}
func (f *fakeEscrowHub) count() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.uploads
} }
// newEscrowTestServer builds a server with the ceremony configured and every seam faked. // newEscrowTestServer builds a server with the ceremony configured and every seam faked.
func newEscrowTestServer(t *testing.T, run ceremonyRunner) *Server { func newEscrowTestServer(t *testing.T, run ceremonyRunner) *Server {
t.Helper()
srv, _ := newEscrowTestServerHub(t, run, &fakeEscrowHub{})
return srv
}
// newEscrowTestServerHub is newEscrowTestServer with the hub's escrow record exposed (R-978).
func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub) (*Server, *fakeEscrowHub) {
t.Helper() t.Helper()
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil) srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.escrowCeremony = &EscrowCeremonyConfig{ srv.escrowCeremony = &EscrowCeremonyConfig{
PBSStorageID: "felhom-pbs", PBSStorageID: "felhom-pbs",
HubConfigured: true, HubConfigured: true,
DRConfigured: func() bool { return true }, DRConfigured: func() bool { return true },
Upload: hub.upload,
} }
srv.ceremonyRun = run srv.ceremonyRun = run
srv.escrowSudoCheck = func(context.Context) error { return nil } srv.escrowSudoCheck = func(context.Context) error { return nil }
srv.escrowLookPath = func(string) (string, error) { return "/usr/bin/age", nil } srv.escrowLookPath = func(string) (string, error) { return "/usr/bin/age", nil }
srv.statFile = func(string) bool { return true } srv.statFile = func(string) bool { return true }
return srv return srv, hub
} }
// okRunner returns the canned success output and counts invocations. // okRunner returns the canned success output and counts invocations.
@@ -160,8 +228,8 @@ func TestEscrowCeremony_OneShotClaim(t *testing.T) {
if !strings.Contains(body, `"phase":"done"`) || !strings.Contains(body, `"claimable":true`) { if !strings.Contains(body, `"phase":"done"`) || !strings.Contains(body, `"claimable":true`) {
t.Fatalf("status not done/claimable: %s", body) t.Fatalf("status not done/claimable: %s", body)
} }
if !strings.Contains(body, `"restic_pw_sealed":true`) || !strings.Contains(body, `"uploaded":true`) { if !strings.Contains(body, `"restic_pw_sealed":true`) || !strings.Contains(body, `"uploaded":false`) {
t.Fatalf("summary fields missing: %s", body) t.Fatalf("summary fields missing (R-978: not uploaded before the claim): %s", body)
} }
assertNoR(t, "status payload", body) assertNoR(t, "status payload", body)
@@ -199,7 +267,8 @@ func TestEscrowCeremony_OneShotClaim(t *testing.T) {
// Post-claim status: claimed, not claimable, still phase done. // Post-claim status: claimed, not claimable, still phase done.
st2 := do(t, h, "GET", "/escrow/ceremony/status", "A", "") st2 := do(t, h, "GET", "/escrow/ceremony/status", "A", "")
if !strings.Contains(st2.Body.String(), `"claimed":true`) || strings.Contains(st2.Body.String(), `"claimable":true`) { if !strings.Contains(st2.Body.String(), `"claimed":true`) || strings.Contains(st2.Body.String(), `"claimable":true`) ||
!strings.Contains(st2.Body.String(), `"uploaded":true`) {
t.Fatalf("post-claim status wrong: %s", st2.Body.String()) t.Fatalf("post-claim status wrong: %s", st2.Body.String())
} }
} }
@@ -344,7 +413,7 @@ func TestEscrowCeremony_FailurePaths(t *testing.T) {
}) })
t.Run("wrong version", func(t *testing.T) { t.Run("wrong version", func(t *testing.T) {
srv := newEscrowTestServer(t, func(context.Context) ([]byte, []byte, int, error) { srv := newEscrowTestServer(t, func(context.Context) ([]byte, []byte, int, error) {
return []byte(`{"version":2,"recovery_code":"` + testR + `"}`), nil, 0, nil return []byte(`{"version":1,"recovery_code":"` + testR + `","uploaded":true}`), nil, 0, nil
}) })
h := srv.Handler() h := srv.Handler()
startAndWait(t, srv, h) startAndWait(t, srv, h)
+223
View File
@@ -0,0 +1,223 @@
package localapi
import (
"encoding/json"
"errors"
"net/http"
"strings"
"testing"
"time"
)
// R-1151 — the new recovery code replaces the hub's package only after the household TYPED IT BACK. Before, the
// claim uploaded it (R-978), so a household that opened the new code and closed the page without writing it down
// had already replaced the package their old code opens. These tests ask the hub's current package the household's
// question — „does my written-down code still open it?".
type deferredClaim struct {
code int
r string
positions []int
required bool
body string
}
func claimDeferred(t *testing.T, h http.Handler) deferredClaim {
t.Helper()
w := do(t, h, "POST", "/escrow/ceremony/claim", "A", `{"defer_upload":true}`)
var env struct {
Data struct {
RecoveryCode string `json:"recovery_code"`
ConfirmRequired bool `json:"confirm_required"`
VerifyPositions []int `json:"verify_positions"`
} `json:"data"`
}
_ = json.Unmarshal(w.Body.Bytes(), &env)
return deferredClaim{w.Code, env.Data.RecoveryCode, env.Data.VerifyPositions, env.Data.ConfirmRequired, w.Body.String()}
}
// typedBack is what a household that wrote the code down types for the claim's two positions.
func (c deferredClaim) typedBack(t *testing.T) []string {
t.Helper()
words := strings.Split(c.r, "-")
if len(c.positions) != 2 {
t.Fatalf("the claim named %d positions, want 2: %s", len(c.positions), c.body)
}
return []string{words[c.positions[0]-1], words[c.positions[1]-1]}
}
func confirm(t *testing.T, h http.Handler, words []string) int {
t.Helper()
b, _ := json.Marshal(map[string]any{"words": words})
return do(t, h, "POST", "/escrow/ceremony/confirm", "A", string(b)).Code
}
// THE R-1151 CASE. Ceremony 1 is claimed and typed back (the household's code 1). Ceremony 2 is started and its code
// SHOWN (claimed) — and then the page is closed: the words never come back. Three ways it ends: the confirm window
// runs out, a new start supersedes it, the agent restarts. In every case code 1 must still open the hub's package.
//
// COMPANION RED-PROOF (observed 2026-10-19): ignore the claim's defer_upload (`if false && req.DeferUpload …`, so
// every claim takes the R-978 order and uploads at once) → all three subtests FAIL with "a code that was shown but
// never typed back replaced the package: code 1 no longer opens it (uploads=2)". Restored.
func TestEscrowCeremony_R1151_ShownButNotTypedBackKeepsFirstCode(t *testing.T) {
setup := func(t *testing.T) (*Server, *fakeEscrowHub, http.Handler, *time.Time) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR, "third-code-charlie"), &fakeEscrowHub{})
cur := testNow
srv.now = func() time.Time { return cur }
h := srv.Handler()
startAndWait(t, srv, h)
c1 := claimDeferred(t, h)
if c1.code != http.StatusOK || c1.r != firstR {
t.Fatalf("deferred claim of ceremony 1: %d %s", c1.code, c1.body)
}
if c1.required { // the household types code 1 back
if code := confirm(t, h, c1.typedBack(t)); code != http.StatusOK {
t.Fatalf("typing code 1 back: %d", code)
}
}
if !hub.opensWith(firstR) || hub.count() != 1 {
t.Fatalf("after ceremony 1 the hub must hold code 1's package (uploads=%d)", hub.count())
}
startAndWait(t, srv, h)
if c2 := claimDeferred(t, h); c2.code != http.StatusOK || c2.r != secondR {
t.Fatalf("deferred claim of ceremony 2: %d %s", c2.code, c2.body)
} // …shown, and the page is closed
return srv, hub, h, &cur
}
check := func(t *testing.T, hub *fakeEscrowHub) {
t.Helper()
if !hub.opensWith(firstR) {
t.Fatalf("a code that was shown but never typed back replaced the package: code 1 no longer opens it (uploads=%d)", hub.count())
}
if hub.opensWith(secondR) || hub.count() != 1 {
t.Fatalf("a package for the never-confirmed code 2 reached the hub (uploads=%d)", hub.count())
}
}
t.Run("the confirm window runs out", func(t *testing.T) {
_, hub, h, cur := setup(t)
*cur = cur.Add(escrowConfirmTTL + time.Minute)
check(t, hub)
st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String()
if !strings.Contains(st, `"phase":"unconfirmed_void"`) || !strings.Contains(st, `"awaiting_confirm":false`) {
t.Fatalf("after the window the job must read unconfirmed_void: %s", st)
}
if code := confirm(t, h, []string{"second", "bravo"}); code != http.StatusGone {
t.Fatalf("a confirm after the window: got %d, want 410", code)
}
})
t.Run("a new start supersedes it", func(t *testing.T) {
srv, hub, h, _ := setup(t)
startAndWait(t, srv, h)
if code := confirm(t, h, []string{"second", "bravo"}); code != http.StatusNotFound {
t.Fatalf("a confirm for the superseded code: got %d, want 404", code)
}
check(t, hub)
})
t.Run("the agent restarts", func(t *testing.T) {
_, hub, _, _ := setup(t)
srv2, _ := newEscrowTestServerHub(t, sequenceRunner("unused"), hub)
if code := confirm(t, srv2.Handler(), []string{"second", "bravo"}); code != http.StatusNotFound {
t.Fatalf("a confirm after a restart: got %d, want 404", code)
}
check(t, hub)
})
}
// The positive control and the typing: wrong words store nothing and the wait goes on; the right words (any case,
// spaces around) store the package, once.
//
// COMPANION RED-PROOF (observed 2026-10-19): skip the word check (`if match != 1 && false`) → this fails with "one
// wrong word: got 200 (uploads=1), want 422 and nothing stored". Restored.
func TestEscrowCeremony_R1151_TypedBackStoresThePackage(t *testing.T) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(secondR), &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if c.code != http.StatusOK || !c.required || hub.count() != 0 {
t.Fatalf("a deferred claim must hand out the code and store nothing yet: %d required=%v uploads=%d", c.code, c.required, hub.count())
}
right := c.typedBack(t)
if code := confirm(t, h, []string{right[0], "wrong"}); code != http.StatusUnprocessableEntity || hub.count() != 0 {
t.Fatalf("one wrong word: got %d (uploads=%d), want 422 and nothing stored", code, hub.count())
}
if code := confirm(t, h, []string{right[0]}); code != http.StatusBadRequest {
t.Fatalf("one word only: got %d, want 400", code)
}
st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String()
if !strings.Contains(st, `"awaiting_confirm":true`) || !strings.Contains(st, `"uploaded":false`) {
t.Fatalf("a wrong try must leave the wait open: %s", st)
}
assertNoRIn(t, "status while awaiting the words", st, secondR)
for _, w := range right {
assertNoRIn(t, "status while awaiting the words", st, `"`+w+`"`)
}
if code := confirm(t, h, []string{" " + strings.ToUpper(right[0]), right[1] + " "}); code != http.StatusOK {
t.Fatalf("the right words: got %d, want 200", code)
}
if !hub.opensWith(secondR) || hub.count() != 1 {
t.Fatalf("after the words came back the hub must hold code 2's package (uploads=%d)", hub.count())
}
if code := confirm(t, h, right); code != http.StatusNotFound || hub.count() != 1 {
t.Fatalf("a second confirm: got %d (uploads=%d), want 404 and no second upload", code, hub.count())
}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
t.Fatalf("status after the confirm: %s", st)
}
}
// A controller that has not updated (0.331.0 and older) claims with `{}` and never confirms: it keeps the R-978
// order — the package is stored at the claim — so its household's ceremony still completes.
func TestEscrowCeremony_R1151_OldControllerKeepsTheClaimOrder(t *testing.T) {
for _, body := range []string{"", "{}"} {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR), &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
w := do(t, h, "POST", "/escrow/ceremony/claim", "A", body)
if w.Code != http.StatusOK || !strings.Contains(w.Body.String(), firstR) || !strings.Contains(w.Body.String(), `"confirm_required":false`) {
t.Fatalf("old-shape claim %q: %d %s", body, w.Code, w.Body.String())
}
if !hub.opensWith(firstR) || hub.count() != 1 {
t.Fatalf("old-shape claim %q must store the package at once (uploads=%d)", body, hub.count())
}
if code := confirm(t, h, []string{"first", "alpha"}); code != http.StatusNotFound {
t.Fatalf("nothing waits after an old-shape claim: got %d, want 404", code)
}
}
}
// The hub failing at the confirm keeps the wait open and the previous package current; a retry succeeds.
func TestEscrowCeremony_R1151_UploadFailureAtConfirmKeepsWaiting(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if code := confirm(t, h, c.typedBack(t)); code != http.StatusBadGateway {
t.Fatalf("confirm with the hub failing: got %d, want 502", code)
}
hub.mu.Lock()
hub.err = nil
hub.mu.Unlock()
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
t.Fatalf("retry after the hub recovered: %d (opens: %v)", code, hub.opensWith(secondR))
}
}
// The two positions are distinct, in range, smaller first — over many draws every position appears.
func TestPickConfirmPositions(t *testing.T) {
seen := map[int]bool{}
for i := 0; i < 2000; i++ {
p, err := pickConfirmPositions(10)
if err != nil {
t.Fatal(err)
}
if p[0] < 1 || p[1] > 10 || p[0] >= p[1] {
t.Fatalf("bad positions %v", p)
}
seen[p[0]], seen[p[1]] = true, true
}
if len(seen) != 10 {
t.Fatalf("only positions %v were ever drawn", seen)
}
}
+219
View File
@@ -0,0 +1,219 @@
package localapi
import (
"context"
"errors"
"net/http"
"strings"
"testing"
"time"
)
// R-978 — starting the escrow ceremony again must not make the customer's written-down recovery code
// useless. The hub overwrites the current package when the sealed secrets are unchanged, so the NEW
// package may reach the hub only when its code is claimed. These tests ask the customer's question of
// the hub's current package — „does my code still open it?" — not whether some function ran.
const (
firstR = "first-ceremony-code-alpha"
secondR = "second-ceremony-code-bravo"
)
// sequenceRunner mints the given codes in order, one per ceremony.
func sequenceRunner(codes ...string) ceremonyRunner {
i := 0
return func(context.Context) ([]byte, []byte, int, error) {
r := codes[i]
if i < len(codes)-1 {
i++
}
return []byte(cannedCeremonyJSON(r)), nil, 0, nil
}
}
func claimCode(t *testing.T, h http.Handler) (int, string) {
t.Helper()
w := do(t, h, "POST", "/escrow/ceremony/claim", "A", "")
return w.Code, w.Body.String()
}
// THE R-978 CASE. Ceremony 1 is claimed (the customer writes code 1 down). Ceremony 2 is started and
// abandoned — three ways: its code expires unclaimed, a third start supersedes it, the agent restarts.
// In every case the hub's current package must still open with code 1, and nothing was uploaded for 2.
//
// COMPANION RED-PROOF (observed 2026-10-18): restore the pre-R-978 shape — the package uploaded when the
// ceremony finishes (s.escrowCeremony.Upload(base, out.PendingUpload) in the job goroutine, as the root
// subprocess did with `--upload`) and nothing left for the claim (finishEscrowCeremony(job, R, nil)) →
// all three subtests FAIL with "the abandoned second ceremony replaced the package: code 1 no longer
// opens it (uploads=2)" (uploads=3 for the superseding start). Restored.
func TestEscrowCeremony_R978_AbandonedSecondCeremonyKeepsFirstCode(t *testing.T) {
setup := func(t *testing.T) (*Server, *fakeEscrowHub, http.Handler, *time.Time) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR, "third-code-charlie"), &fakeEscrowHub{})
cur := testNow
srv.now = func() time.Time { return cur }
h := srv.Handler()
startAndWait(t, srv, h)
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("claim of ceremony 1: %d %s", code, body)
}
if !hub.opensWith(firstR) || hub.count() != 1 {
t.Fatalf("after the first claim the hub must hold code 1's package (uploads=%d)", hub.count())
}
startAndWait(t, srv, h) // ceremony 2 — about to be abandoned
return srv, hub, h, &cur
}
check := func(t *testing.T, hub *fakeEscrowHub) {
t.Helper()
if !hub.opensWith(firstR) {
t.Fatalf("the abandoned second ceremony replaced the package: code 1 no longer opens it (uploads=%d)", hub.count())
}
if hub.opensWith(secondR) || hub.count() != 1 {
t.Fatalf("a package for the never-claimed code 2 reached the hub (uploads=%d)", hub.count())
}
}
t.Run("code 2 expires unclaimed", func(t *testing.T) {
_, hub, h, cur := setup(t)
*cur = cur.Add(escrowClaimTTL + time.Minute)
if code, _ := claimCode(t, h); code != http.StatusGone {
t.Fatalf("claim after TTL: got %d, want 410", code)
}
check(t, hub)
})
t.Run("a third start supersedes code 2", func(t *testing.T) {
srv, hub, h, _ := setup(t)
startAndWait(t, srv, h) // ceremony 3, also left unclaimed
check(t, hub)
})
t.Run("the agent restarts before code 2 is claimed", func(t *testing.T) {
_, hub, _, _ := setup(t)
// A restart is a new process with an empty slot; only the hub's record survives it.
srv2, _ := newEscrowTestServerHub(t, sequenceRunner("unused"), hub)
if code, _ := claimCode(t, srv2.Handler()); code != http.StatusNotFound {
t.Fatalf("claim after a restart: got %d, want 404", code)
}
check(t, hub)
})
}
// The positive control: CLAIMING code 2 is what replaces the package — the code handed out opens the
// package the hub now holds, and the claim answers only after the hub took it.
func TestEscrowCeremony_R978_ClaimStoresThePackageBeforeHandingOutTheCode(t *testing.T) {
srv, hub := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR), &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
if hub.count() != 0 {
t.Fatalf("the package was uploaded before any claim (uploads=%d)", hub.count())
}
claimCode(t, h)
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusOK || !strings.Contains(body, secondR) {
t.Fatalf("claim of ceremony 2: %d %s", code, body)
}
if !hub.opensWith(secondR) || hub.count() != 2 {
t.Fatalf("after claiming code 2 the hub must hold its package (uploads=%d, opens with 2: %v)", hub.count(), hub.opensWith(secondR))
}
}
// A failed upload hands out NO code, keeps the code claimable and the previous package current; a retry
// that reaches the hub succeeds.
func TestEscrowCeremony_R978_UploadFailureKeepsTheCodeClaimable(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusBadGateway {
t.Fatalf("claim with the hub failing: got %d, want 502 (%s)", code, body)
}
assertNoRIn(t, "failed-upload claim payload", body, firstR)
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) || !strings.Contains(st, `"uploaded":false`) {
t.Fatalf("after a failed upload the code must stay claimable and un-uploaded: %s", st)
}
hub.mu.Lock()
hub.err = nil
hub.mu.Unlock()
code, body = claimCode(t, h)
if code != http.StatusOK || !strings.Contains(body, firstR) || !hub.opensWith(firstR) {
t.Fatalf("retry after the hub recovered: %d %s (opens: %v)", code, body, hub.opensWith(firstR))
}
}
// While the claim's upload is in flight, neither the TTL nor a new start may take the code away — else
// the hub would hold a package whose code nobody was given (the R-978 failure, reached the other way).
func TestEscrowCeremony_R978_TTLAndRestartWaitForAnInFlightClaim(t *testing.T) {
hub := &fakeEscrowHub{gate: make(chan struct{}), entered: make(chan struct{})}
entered := hub.entered
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR, secondR), hub)
var mu = make(chan struct{}, 1)
mu <- struct{}{}
cur := testNow
srv.now = func() time.Time { <-mu; defer func() { mu <- struct{}{} }(); return cur }
h := srv.Handler()
startAndWait(t, srv, h)
type res struct {
code int
body string
}
done := make(chan res, 1)
go func() { c, b := claimCode(t, h); done <- res{c, b} }()
<-entered
<-mu
cur = cur.Add(escrowClaimTTL + time.Minute) // the TTL passes mid-upload
mu <- struct{}{}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); strings.Contains(st, `"phase":"unclaimed_void"`) {
t.Fatalf("the TTL voided the code while its package was uploading: %s", st)
}
if w := do(t, h, "POST", "/escrow/ceremony", "A", ""); w.Code != http.StatusConflict {
t.Fatalf("a new start during the claim's upload: got %d, want 409", w.Code)
}
if code, _ := claimCode(t, h); code != http.StatusConflict {
t.Fatalf("a second claim during the upload: got %d, want 409", code)
}
close(hub.gate)
r := <-done
if r.code != http.StatusOK || !strings.Contains(r.body, firstR) || !hub.opensWith(firstR) {
t.Fatalf("the in-flight claim must deliver the code whose package it stored: %d %s", r.code, r.body)
}
}
// No upload configured → the claim refuses rather than hand out a code for a package the hub lacks.
func TestEscrowCeremony_R978_NoUploaderRefusesTheClaim(t *testing.T) {
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), &fakeEscrowHub{})
srv.escrowCeremony.Upload = nil
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusServiceUnavailable {
t.Fatalf("claim with no uploader: got %d, want 503 (%s)", code, body)
}
assertNoRIn(t, "no-uploader claim payload", body, firstR)
}
// An output that uploaded by itself (the pre-R-978 shape) or carries no package fails the ceremony.
func TestEscrowCeremony_R978_OutputWithoutPendingPackageFails(t *testing.T) {
srv, _ := newEscrowTestServerHub(t, func(context.Context) ([]byte, []byte, int, error) {
return []byte(`{"version":2,"recovery_code":"` + firstR + `","uploaded":true}`), nil, 0, nil
}, &fakeEscrowHub{})
h := srv.Handler()
startAndWait(t, srv, h)
st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String()
if !strings.Contains(st, `"phase":"failed"`) || !strings.Contains(st, "no pending package") {
t.Fatalf("want failed: %s", st)
}
if code, _ := claimCode(t, h); code != http.StatusConflict {
t.Fatalf("claim on a failed ceremony: got %d, want 409", code)
}
}
func assertNoRIn(t *testing.T, where, body, r string) {
t.Helper()
if strings.Contains(body, r) {
t.Fatalf("%s carries the recovery code", where)
}
}
+12 -1
View File
@@ -57,6 +57,10 @@ type MemoryInfo struct {
MaxMB int64 `json:"max_mb"` MaxMB int64 `json:"max_mb"`
FloorMB int64 `json:"floor_mb"` FloorMB int64 `json:"floor_mb"`
Running bool `json:"running"` Running bool `json:"running"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up): the host's swap and this guest's swap allowance, so the controller
// can tell the household when the host has too little swap behind the guest (its install page). nil = not read.
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
} }
// memoryResizeRequest is the POST /guest/memory body. // memoryResizeRequest is the POST /guest/memory body.
@@ -94,6 +98,8 @@ type memoryBounds struct {
maxMB int64 maxMB int64
floorMB int64 floorMB int64
running bool running bool
hostSwapMB int64
guestSwapMB int64
} }
func (b memoryBounds) refusal(code string) memoryRefusal { func (b memoryBounds) refusal(code string) memoryRefusal {
@@ -104,7 +110,9 @@ func (b memoryBounds) refusal(code string) memoryRefusal {
} }
func (b memoryBounds) info(vmid int) MemoryInfo { func (b memoryBounds) info(vmid int) MemoryInfo {
hostSwap, guestSwap := b.hostSwapMB, b.guestSwapMB
return MemoryInfo{ return MemoryInfo{
HostSwapMB: &hostSwap, GuestSwapMB: &guestSwap,
VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB, VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB,
MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running, MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running,
} }
@@ -139,6 +147,8 @@ func (s *Server) readMemoryBounds(ctx context.Context, vmid int) (memoryBounds,
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max) hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
minMB: minGuestMemoryMB, minMB: minGuestMemoryMB,
running: st.Status == "running", running: st.Status == "running",
hostSwapMB: node.Swap.Total / mib, // R-1192 follow-up: the kernel's SwapTotal, as Proxmox reports it
guestSwapMB: cfg.Swap, // PVE config swap is MB
} }
b.maxMB = b.hostTotalMB - hostReserveMB b.maxMB = b.hostTotalMB - hostReserveMB
b.floorMB = b.usageMB + shrinkUsageMarginMB b.floorMB = b.usageMB + shrinkUsageMarginMB
@@ -225,7 +235,8 @@ func (s *Server) handleGuestMemoryResize(w http.ResponseWriter, r *http.Request,
} }
// Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on. // Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target)}) // R-1192: the swap moves with the memory (GuestSwapMB) — a grown guest with the old swap cap thrashes the same way.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target), "swap": fmt.Sprintf("%d", GuestSwapMB(target))})
if err != nil { if err != nil {
s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err) s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err)
writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error()) writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error())
+8 -1
View File
@@ -25,12 +25,14 @@ type fakeMemory struct {
setUPID string setUPID string
setErr error setErr error
applyReflect bool applyReflect bool
hostSwapB int64 // R-1192 follow-up: node swap total
guestSwapMB int64
} }
func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) { func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
return proxmox.GuestConfig{Memory: f.allocMB}, nil return proxmox.GuestConfig{Memory: f.allocMB, Swap: f.guestSwapMB}, nil
} }
func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) { func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) {
f.mu.Lock() f.mu.Lock()
@@ -42,6 +44,7 @@ func (f *fakeMemory) NodeStatus(_ context.Context) (proxmox.NodeStatus, error) {
defer f.mu.Unlock() defer f.mu.Unlock()
var ns proxmox.NodeStatus var ns proxmox.NodeStatus
ns.Memory.Total = f.hostTotalB ns.Memory.Total = f.hostTotalB
ns.Swap.Total = f.hostSwapB
return ns, nil return ns, nil
} }
func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) { func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) {
@@ -151,6 +154,10 @@ func TestGuestMemory_GrowAndShrink(t *testing.T) {
if f.lastParams["memory"] != "12288" { if f.lastParams["memory"] != "12288" {
t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"]) t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"])
} }
// R-1192: the swap moves with the memory.
if f.lastParams["swap"] != "12288" {
t.Errorf("SetConfig swap = %q, want 12288 (swap = RAM)", f.lastParams["swap"])
}
}) })
t.Run("shrink above floor", func(t *testing.T) { t.Run("shrink above floor", func(t *testing.T) {
f := stdFixture() // floor = max(2048, 3000+512) = 3512 f := stdFixture() // floor = max(2048, 3000+512) = 3512
+73
View File
@@ -0,0 +1,73 @@
package localapi
import (
"context"
"fmt"
"log/slog"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// ── R-1192 (2026-10-21): the guest may swap as much as its RAM ────────────────────────────────────────────────────
//
// MEASURED (felhom.eu audits/fixes-2026-10-21/P4-freeze/): on the minimum machine (8 GB host, guest 4096 MB) the
// household's first uploads made Immich load its analysis models (+1.15 GB) on top of 8 apps' 1.75 GB; the guest hit
// its 4096 MB limit with its swap capped at Proxmox's LXC default of 512 MB — and full. The kernel could then evict
// only file pages: ~360 MB was left for the program code of ~25 containers, re-read from disk without end (150 000–
// 600 000 refaults and ~1 GB of reads per 10 s; memory „full" pressure 92–95 %, load 155, the dashboard dead, 0 OOM
// kills). Raising the cap to 4096 MB live ended it in under a minute (memory „full" 96 % → 1.7 %, dashboard back
// 42 s later) while the uploads ran on. The host had 7.4 GB of swap free the whole time.
//
// The agent never set `swap` at all, so every guest had the 512 MB default. Now:
// - a new guest is created with swap = its memory (reconcile.bringup);
// - a RAM resize sets swap with the memory (handleGuestMemoryResize);
// - at agent start, every LXC in the felhom pool whose swap is below its memory is raised to it (this file).
// Raising swap is a live cgroup change (memory.swap.max), no restart. It never lowers a swap someone set higher.
// GuestSwapMB is the swap a guest of memoryMB gets: the same as its RAM.
func GuestSwapMB(memoryMB int64) int64 { return memoryMB }
// SwapOps is the Proxmox surface EnsureGuestSwap needs (satisfied by *proxmox.Client).
type SwapOps interface {
Pool(ctx context.Context, name string) (proxmox.PoolInfo, error)
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
SetConfig(ctx context.Context, vmid int, params map[string]string) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
}
// EnsureGuestSwap raises the swap of every LXC guest in pool to GuestSwapMB(memory) when it is below. Returns how
// many guests were raised. One guest's failure is logged and the others are still handled.
func EnsureGuestSwap(ctx context.Context, ops SwapOps, pool string, logger *slog.Logger) (int, error) {
p, err := ops.Pool(ctx, pool)
if err != nil {
return 0, fmt.Errorf("guest swap: read pool %s: %w", pool, err)
}
raised := 0
for _, m := range p.Members {
if m.Type != "lxc" || m.VMID <= 0 {
continue
}
cfg, err := ops.GuestConfig(ctx, m.VMID)
if err != nil {
logger.Warn("guest-swap: config read failed (left as it is)", "vmid", m.VMID, "err", err)
continue
}
want := GuestSwapMB(cfg.Memory)
if cfg.Memory <= 0 || cfg.Swap >= want {
logger.Debug("guest-swap: nothing to do", "vmid", m.VMID, "memory_mb", cfg.Memory, "swap_mb", cfg.Swap)
continue
}
upid, err := ops.SetConfig(ctx, m.VMID, map[string]string{"swap": fmt.Sprintf("%d", want)})
if err == nil && upid != "" {
_, err = ops.WaitTask(ctx, upid, proxmox.WaitOptions{})
}
if err != nil {
logger.Warn("guest-swap: raise failed (left as it is)", "vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want, "err", err)
continue
}
raised++
logger.Info("guest-swap: raised (R-1192 — a guest swap cap below its RAM turned a memory burst into a freeze)",
"vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want)
}
return raised, nil
}
+71
View File
@@ -0,0 +1,71 @@
package localapi
import (
"context"
"errors"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: the freeze measured on the minimum machine (felhom.eu audits/fixes-2026-10-21/P4-freeze/) — a guest whose
// swap is capped below its RAM evicts program code instead of idle memory. EnsureGuestSwap raises every pool guest's
// swap to its RAM, never lowers one, skips non-LXC members, and survives one guest's failure.
// COMPANION RED-PROOF: return 0 before the SetConfig call → „raised 0, want 2".
type fakeSwapOps struct {
cfg map[int]proxmox.GuestConfig
failSet map[int]bool
set map[int]map[string]string
}
func (f *fakeSwapOps) Pool(context.Context, string) (proxmox.PoolInfo, error) {
return proxmox.PoolInfo{Members: []proxmox.PoolMember{{VMID: 9201, Type: "lxc"}, {VMID: 9202, Type: "lxc"}, {VMID: 300, Type: "qemu"},
{Type: "storage"}, {VMID: 9203, Type: "lxc"}, {VMID: 9204, Type: "lxc"}}}, nil
}
func (f *fakeSwapOps) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
c, ok := f.cfg[vmid]
if !ok {
return c, errors.New("no such guest")
}
return c, nil
}
func (f *fakeSwapOps) SetConfig(_ context.Context, vmid int, p map[string]string) (string, error) {
if f.failSet[vmid] {
return "", errors.New("pve said no")
}
if f.set == nil {
f.set = map[int]map[string]string{}
}
f.set[vmid] = p
return "", nil
}
func (f *fakeSwapOps) WaitTask(context.Context, string, proxmox.WaitOptions) (proxmox.TaskStatus, error) {
return proxmox.TaskStatus{}, nil
}
func TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM(t *testing.T) {
f := &fakeSwapOps{cfg: map[int]proxmox.GuestConfig{
9201: {Memory: 4096, Swap: 512}, // the measured box: raised
9202: {Memory: 2048, Swap: 4096}, // already higher: never lowered
9203: {Memory: 6144, Swap: 0}, // none: raised
9204: {Memory: 4096, Swap: 512}, // the write fails: logged, the others still done
}, failSet: map[int]bool{9204: true}}
n, err := EnsureGuestSwap(context.Background(), f, "felhom", slog.New(slog.NewTextHandler(io.Discard, nil)))
if err != nil {
t.Fatal(err)
}
if n != 2 {
t.Fatalf("raised %d, want 2 (9201, 9203)", n)
}
if f.set[9201]["swap"] != "4096" || f.set[9203]["swap"] != "6144" {
t.Fatalf("writes = %v", f.set)
}
if _, touched := f.set[9202]; touched {
t.Fatal("a swap above the RAM was lowered")
}
if len(f.set[9201]) != 1 {
t.Fatalf("the swap raise wrote more than the swap: %v", f.set[9201])
}
}
+39
View File
@@ -0,0 +1,39 @@
package localapi
import (
"encoding/json"
"testing"
)
// R-1192 follow-up (night 2026-10-21): GET /guest/memory carries the host's swap and this guest's swap allowance, so
// the controller's install page can say when the host has no swap behind the guest. 0 is a real answer (no swap);
// absent means an older agent.
// RED-PROOF: without the two MemoryInfo fields the keys are missing → fails on „host_swap_mb absent".
func TestR1192_GuestMemoryCarriesHostAndGuestSwap(t *testing.T) {
for _, c := range []struct {
name string
hostB, guestMB int64
wantHost, wantG int64
}{
{"installer host (7.75 GB swap), guest swap = RAM", 7935 * mib, 8192, 7935, 8192},
{"a host with NO swap", 0, 8192, 0, 8192},
} {
f := stdFixture()
f.hostSwapB, f.guestSwapMB = c.hostB, c.guestMB
w := do(t, memServer(t, f).Handler(), "GET", "/guest/memory", "A", "")
var m struct {
Data map[string]any `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &m); err != nil {
t.Fatal(err)
}
hs, ok := m.Data["host_swap_mb"].(float64)
if !ok {
t.Fatalf("%s: host_swap_mb absent: %s", c.name, w.Body.String())
}
gs, _ := m.Data["guest_swap_mb"].(float64)
if int64(hs) != c.wantHost || int64(gs) != c.wantG {
t.Fatalf("%s: host %v guest %v, want %d / %d", c.name, hs, gs, c.wantHost, c.wantG)
}
}
}
+47
View File
@@ -90,6 +90,10 @@ type BackupStore interface {
type LastKnownBackupStore interface { type LastKnownBackupStore interface {
RecordBackupSuccess(target string, b hub.Backup) error RecordBackupSuccess(target string, b hub.Backup) error
LastKnownSuccess(target string, vmid int) (time.Time, bool) LastKnownSuccess(target string, vmid int) (time.Time, bool)
// R-973: the cadence cap's restart-proof history and its refusals.
RecentBackupSuccesses(target string, vmid int, since time.Time) []time.Time
RecordBackupRefusal(target string, vmid int, at time.Time) error
LastBackupRefusal() (target string, vmid int, at time.Time, ok bool)
} }
// StorageView yields the host's observed storage targets (for mapping a mount's storage id → // StorageView yields the host's observed storage targets (for mapping a mount's storage id →
@@ -302,6 +306,11 @@ type Server struct {
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
// R-973 cadence cap: this process's successful runs per tier+guest, and the newest refusal.
capMu sync.Mutex
capRuns map[backupJobKey][]time.Time
capRefusal capRefusal
disks DiskOps // slice 8C (optional) disks DiskOps // slice 8C (optional)
diskGate StorageGate // slice 8C (optional) diskGate StorageGate // slice 8C (optional)
guestList GuestLister // slice 8C (optional) guestList GuestLister // slice 8C (optional)
@@ -426,8 +435,17 @@ type Server struct {
escrowMu sync.Mutex escrowMu sync.Mutex
escrowJob *escrowCeremonyJob escrowJob *escrowCeremonyJob
escrowR []byte escrowR []byte
escrowPending []byte // R-978: the ceremony's hub upload body, sent by the claim before R is handed out
escrowClaiming bool // R-978: a claim is uploading escrowPending (TTL and a new start wait)
escrowRClaimed bool escrowRClaimed bool
escrowRExpiry time.Time escrowRExpiry time.Time
// R-1151: after a claim that asked to wait for the typed-back words (defer_upload), the package stays in
// escrowPending until POST /escrow/ceremony/confirm brings the two words at escrowConfirmPos back. Only those two
// words are kept (never R), zeroed on confirm, supersede and expiry.
escrowAwaitConfirm bool
escrowConfirmWords [2][]byte
escrowConfirmPos [2]int
escrowConfirmExpiry time.Time
escrowDone <-chan struct{} // closes when the detached job finishes (tests wait on it) escrowDone <-chan struct{} // closes when the detached job finishes (tests wait on it)
// ceremonyRun executes the fixed-argv sudo self-invocation (tests inject canned JSON). // ceremonyRun executes the fixed-argv sudo self-invocation (tests inject canned JSON).
ceremonyRun ceremonyRunner ceremonyRun ceremonyRunner
@@ -603,6 +621,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /escrow/ceremony", s.withGuest(s.handleEscrowCeremonyStart)) mux.HandleFunc("POST /escrow/ceremony", s.withGuest(s.handleEscrowCeremonyStart))
mux.HandleFunc("GET /escrow/ceremony/status", s.withGuest(s.handleEscrowCeremonyStatus)) mux.HandleFunc("GET /escrow/ceremony/status", s.withGuest(s.handleEscrowCeremonyStatus))
mux.HandleFunc("POST /escrow/ceremony/claim", s.withGuest(s.handleEscrowCeremonyClaim)) mux.HandleFunc("POST /escrow/ceremony/claim", s.withGuest(s.handleEscrowCeremonyClaim))
// R-1151: the household typed two words of the new code back — only now does its package reach the hub.
mux.HandleFunc("POST /escrow/ceremony/confirm", s.withGuest(s.handleEscrowCeremonyConfirm))
// v0.83.0 observability: the agent's always-DEBUG capture ring, for the controller's // v0.83.0 observability: the agent's always-DEBUG capture ring, for the controller's
// Debug page agent tab (same auth/self-scoping wrap as every sibling route). // Debug page agent tab (same auth/self-scoping wrap as every sibling route).
@@ -878,6 +898,12 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
"a backup is already in flight on target "+busyTarget+" (job "+busyJob+") — only one backup runs at a time per guest") "a backup is already in flight on target "+busyTarget+" (job "+busyJob+") — only one backup runs at a time per guest")
return return
} }
// R-973: the cadence cap (backup_cap.go). Checked under jobsMu, after the in-flight answers, so two
// requests cannot both pass it (only one run per guest is ever in flight).
if s.capRefuses(w, key, tier) {
s.jobsMu.Unlock()
return
}
// Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers // Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers
// started inside the same nanosecond (the weekly both-due night, or any injected clock) would // started inside the same nanosecond (the weekly both-due night, or any injected clock) would
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the // otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
@@ -941,6 +967,9 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive) s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive)
} }
s.store.RecordBackup(b) s.store.RecordBackup(b)
if b.Success {
s.recordCapRun(key, b) // R-973
}
if b.Success && s.lastKnown != nil { if b.Success && s.lastKnown != nil {
if err := s.lastKnown.RecordBackupSuccess(tier.TargetID, b); err != nil { if err := s.lastKnown.RecordBackupSuccess(tier.TargetID, b); err != nil {
// Not fatal: the backup exists. Only the fallback after a restart loses this copy. // Not fatal: the backup exists. Only the fallback after a restart loses this copy.
@@ -1314,6 +1343,20 @@ type BackupStatusResponse struct {
// presence. Served on the UNTARGETED request only; additive, so an older controller reads the // presence. Served on the UNTARGETED request only; additive, so an older controller reads the
// response exactly as before. // response exactly as before.
Tiers []TierBackupState `json:"tiers,omitempty"` Tiers []TierBackupState `json:"tiers,omitempty"`
// HostBusy (R-921, agent Unreleased 2026-10-19) says whether the host-wide heavy-operation gate is held RIGHT NOW
// (another guest's backup, the restore test, a night OS step, the stale-lock sweep, a power-off) — a POST /backup
// would then be refused with 409. The controller reads it BEFORE it stops the household's apps, so a refused tier
// costs no stop. Served on every request; ABSENT (nil) = an older agent, or the gate is not wired — the controller
// then keeps its old order (stop, ask, resume at once on 409). A read, never a reservation: the gate can be taken
// between this answer and the POST, and that race keeps the old order. Pinned by TestBackupStatus_R921_HostBusy.
HostBusy *HostBusyState `json:"host_busy,omitempty"`
}
// HostBusyState is GET /backup/status's host_busy (R-921). What names the holder ("backup:local", "restore-test",
// "os-pve-step" …), empty when Busy is false.
type HostBusyState struct {
Busy bool `json:"busy"`
What string `json:"what,omitempty"`
} }
func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) { func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) {
@@ -1333,6 +1376,10 @@ func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid
resp.JobID = job.JobID resp.JobID = job.JobID
resp.Error = job.Error resp.Error = job.Error
} }
if s.inFlight != nil {
what := s.inFlight.Busy()
resp.HostBusy = &HostBusyState{Busy: what != "", What: what}
}
writeOK(w, resp) writeOK(w, resp)
} }
+45
View File
@@ -13,6 +13,7 @@ import (
"testing" "testing"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub" "gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
) )
@@ -514,3 +515,47 @@ func TestBackupStatus_FiltersToThisGuest(t *testing.T) {
t.Fatalf("status not scoped to this guest / not latest: %s", w.Body.String()) t.Fatalf("status not scoped to this guest / not latest: %s", w.Body.String())
} }
} }
// R-921: GET /backup/status says when the host-wide heavy-operation gate is held, so the controller can skip the app
// stop for a tier the POST would refuse. Free → busy false; held → busy true with the holder's name; the gate not wired
// → no field at all (the controller's „older agent" reading, never „free").
//
// COMPANION RED-PROOF (observed 2026-10-19): with the `resp.HostBusy = …` assignment removed, the held case fails with
//
// server_test.go: free gate: host_busy = <nil>, want busy=false
//
// Restored.
func TestBackupStatus_R921_HostBusy(t *testing.T) {
read := func(h http.Handler) *HostBusyState {
w := do(t, h, "GET", "/backup/status", "A", "")
var resp struct {
Data BackupStatusResponse `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("status body: %v (%s)", err, w.Body.String())
}
return resp.Data.HostBusy
}
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
if got := read(srv.Handler()); got != nil {
t.Fatalf("gate not wired: host_busy = %+v, want absent", got)
}
gate := &backup.InFlight{}
srv.inFlight = gate
h := srv.Handler()
if got := read(h); got == nil || got.Busy || got.What != "" {
t.Fatalf("free gate: host_busy = %+v, want busy=false", got)
}
release, _, ok := gate.TryAcquire("os-pve-step")
if !ok {
t.Fatal("could not take the gate")
}
if got := read(h); got == nil || !got.Busy || got.What != "os-pve-step" {
t.Fatalf("held gate: host_busy = %+v, want busy=true what=\"os-pve-step\"", got)
}
// The read never takes the gate: a POST after the holder lets go is accepted.
release()
if got := read(h); got == nil || got.Busy {
t.Fatalf("released gate: host_busy = %+v, want busy=false", got)
}
}
+46 -1
View File
@@ -2,6 +2,7 @@ package localapi
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"log/slog" "log/slog"
"os" "os"
@@ -160,6 +161,7 @@ type staleLockAPI interface {
ListSnapshots(ctx context.Context, vmid int) ([]proxmox.Snapshot, error) ListSnapshots(ctx context.Context, vmid int) ([]proxmox.Snapshot, error)
ListRunningTasks(ctx context.Context) ([]proxmox.TaskStatus, error) ListRunningTasks(ctx context.Context) ([]proxmox.TaskStatus, error)
DeleteSnapshot(ctx context.Context, vmid int, snapname string) (string, error) DeleteSnapshot(ctx context.Context, vmid int, snapname string) (string, error)
DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error)
Start(ctx context.Context, vmid int) (string, error) Start(ctx context.Context, vmid int) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error) WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
} }
@@ -334,8 +336,34 @@ func (c *staleLockController) Unlock(ctx context.Context, vmid int) error {
return nil return nil
} }
// DeleteVzdumpSnapshot removes the dangling `vzdump` snapshot. R-1010: when the delete fails ONLY because one disk's
// snapshot volume is already gone („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump" — a backup that ran
// out of space, 2026-10-14), Proxmox stops at that volume and keeps the entry and the other volumes' snapshots, and a
// plain retry fails the same way every sweep, forever. Then it retries ONCE with Proxmox's force (the hand fix that
// day was `pct delsnapshot 9201 vzdump --force`): the remaining volumes' snapshots are removed and the entry goes. Any
// other error is returned as it is — never a force on an error that might mean a volume still holds data (a full
// LVM metadata area, a busy volume). Pinned by TestStaleLock_R1010_*.
func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int) error { func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int) error {
upid, err := c.px.DeleteSnapshot(ctx, vmid, vzdumpSnapshotName) err := c.deleteSnap(ctx, vmid, false)
if err == nil || !snapshotVolumeAlreadyGone(err) {
return err
}
if c.logger != nil {
c.logger.Warn("stale-lock: a disk's vzdump snapshot volume is already gone — retrying the delete once with force (R-1010)",
"vmid", vmid, "err", err)
}
if ferr := c.deleteSnap(ctx, vmid, true); ferr != nil {
return fmt.Errorf("forced snapshot delete after %q also failed: %w", err.Error(), ferr)
}
return nil
}
func (c *staleLockController) deleteSnap(ctx context.Context, vmid int, force bool) error {
del := c.px.DeleteSnapshot
if force {
del = c.px.DeleteSnapshotForce
}
upid, err := del(ctx, vmid, vzdumpSnapshotName)
if err != nil { if err != nil {
return err return err
} }
@@ -346,6 +374,23 @@ func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int
return err return err
} }
// snapshotVolumeAlreadyGone: the error is Proxmox's „Failed to find logical volume …_vzdump" from a snapshot delete
// (in the task's exit status, its log tail, or the API error text). Narrow on purpose: the force is safe only when the
// volume it cannot remove does not exist.
func snapshotVolumeAlreadyGone(err error) bool {
text := err.Error()
var te *proxmox.TaskError
if errors.As(err, &te) {
text += "\n" + te.ExitStatus + "\n" + strings.Join(te.LogTail, "\n")
}
for _, l := range strings.Split(text, "\n") {
if strings.Contains(strings.ToLower(l), "failed to find logical volume") && strings.Contains(l, "_"+vzdumpSnapshotName) {
return true
}
}
return false
}
func (c *staleLockController) Start(ctx context.Context, vmid int) error { func (c *staleLockController) Start(ctx context.Context, vmid int) error {
upid, err := c.px.Start(ctx, vmid) upid, err := c.px.Start(ctx, vmid)
if err != nil { if err != nil {
+19 -1
View File
@@ -30,6 +30,9 @@ type fakeStaleLockAPI struct {
delsnap []int delsnap []int
started []int started []int
// R-1010: scripted delete answers, consumed in order (plain and forced share the list); calls records "plain"/"force".
delErrs []error
delCalls []string
} }
func (f *fakeStaleLockAPI) ListLXC(context.Context) ([]proxmox.Guest, error) { return f.lxc, nil } func (f *fakeStaleLockAPI) ListLXC(context.Context) ([]proxmox.Guest, error) { return f.lxc, nil }
@@ -55,7 +58,22 @@ func (f *fakeStaleLockAPI) ListRunningTasks(context.Context) ([]proxmox.TaskStat
func (f *fakeStaleLockAPI) DeleteSnapshot(_ context.Context, vmid int, snapname string) (string, error) { func (f *fakeStaleLockAPI) DeleteSnapshot(_ context.Context, vmid int, snapname string) (string, error) {
f.delsnap = append(f.delsnap, vmid) f.delsnap = append(f.delsnap, vmid)
return "", nil // synchronous — no WaitTask return "", f.nextDelErr("plain") // synchronous — no WaitTask
}
func (f *fakeStaleLockAPI) DeleteSnapshotForce(_ context.Context, vmid int, snapname string) (string, error) {
f.delsnap = append(f.delsnap, vmid)
return "", f.nextDelErr("force")
}
func (f *fakeStaleLockAPI) nextDelErr(kind string) error {
f.delCalls = append(f.delCalls, kind)
if len(f.delErrs) == 0 {
return nil
}
e := f.delErrs[0]
f.delErrs = f.delErrs[1:]
return e
} }
func (f *fakeStaleLockAPI) Start(_ context.Context, vmid int) (string, error) { func (f *fakeStaleLockAPI) Start(_ context.Context, vmid int) (string, error) {
+59
View File
@@ -0,0 +1,59 @@
package localapi
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1010: a half-removed vzdump snapshot. The error text is the one demo-hp's task printed on 2026-10-14
// (`audits/power-cut-fixes-2026-10-14/C4/hand-fix.txt`).
const r1010MissingLV = `lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to find logical volume "pve/snap_vm-9201-disk-0_vzdump"`
func r1010TaskErr(exit string) error {
return &proxmox.TaskError{UPID: "UPID:demo-hp:0001:0002:0003:vzdelsnapshot:9201:root@pam:", ExitStatus: exit,
LogTail: []string{exit}}
}
// COMPANION RED-PROOF (observed 2026-10-19): with DeleteVzdumpSnapshot back to the single plain delete, the first case
// fails with
//
// stalelock_r1010_test.go: missing volume: calls [plain], want [plain force]; err = proxmox: task … failed: exitstatus "lvremove snapshot …"
//
// Restored.
func TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone(t *testing.T) {
cases := []struct {
name string
errs []error
wantCalls []string
wantErr string
}{
{"missing volume", []error{r1010TaskErr(r1010MissingLV)}, []string{"plain", "force"}, ""},
{"missing volume, named only in the log tail", []error{&proxmox.TaskError{UPID: "x", ExitStatus: "command 'lvremove' failed: exit code 5",
LogTail: []string{r1010MissingLV}}}, []string{"plain", "force"}, ""},
{"the plain delete works", nil, []string{"plain"}, ""},
{"another error: no force", []error{r1010TaskErr("lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to update pool pve/data.")},
[]string{"plain"}, "Failed to update pool"},
{"a missing volume of ANOTHER snapshot: no force", []error{r1010TaskErr(`Failed to find logical volume "pve/snap_vm-9201-disk-0_mine"`)},
[]string{"plain"}, "_mine"},
{"the forced delete fails too: returned, never a second force", []error{r1010TaskErr(r1010MissingLV), errors.New("still broken")},
[]string{"plain", "force"}, "still broken"},
}
for _, c := range cases {
api := &fakeStaleLockAPI{delErrs: c.errs}
ctrl := NewStaleLockController(api, &recordingRunner{}, "felhom", nil)
err := ctrl.DeleteVzdumpSnapshot(context.Background(), 9201)
if strings.Join(api.delCalls, " ") != strings.Join(c.wantCalls, " ") {
t.Errorf("%s: calls %v, want %v; err = %v", c.name, api.delCalls, c.wantCalls, err)
}
switch {
case c.wantErr == "" && err != nil:
t.Errorf("%s: err = %v, want nil", c.name, err)
case c.wantErr != "" && (err == nil || !strings.Contains(err.Error(), c.wantErr)):
t.Errorf("%s: err = %v, want one naming %q", c.name, err, c.wantErr)
}
}
}
+19 -2
View File
@@ -370,7 +370,8 @@ func HealthVerdict(before, after *Health) (bool, string) {
// HostHealthVerdict is THE host health rule (`11` §8.2; pinned by TestHostHealthVerdict): the Proxmox daemons and the // HostHealthVerdict is THE host health rule (`11` §8.2; pinned by TestHostHealthVerdict): the Proxmox daemons and the
// agent are active, the customer guest still runs, the guest's own rule still passes against the start of the pass, // agent are active, the customer guest still runs, the guest's own rule still passes against the start of the pass,
// and the tunnel is `running` (R-841). // and the tunnel is `running` (R-841) — or `absent` both now and before the update: a box with no tunnel set up
// (R-1070; pinned by TestHostHealthVerdict_R1070_NoTunnelSetUp).
func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) { func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
if after == nil { if after == nil {
return false, "no health reading" return false, "no health reading"
@@ -398,12 +399,28 @@ func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
if ok, why := HealthVerdict(gb, after.Guest); !ok { if ok, why := HealthVerdict(gb, after.Guest); !ok {
return false, "guest: " + why return false, "guest: " + why
} }
if tunnel != hub.TunnelRunning { switch {
case tunnel == hub.TunnelRunning:
case tunnel == hub.TunnelAbsent && noTunnelBefore(before):
// R-1070: no cloudflared container before the update and none after — the box has no tunnel to check.
case tunnel == hub.TunnelAbsent:
return false, "the tunnel is absent (no cloudflared container now, and the reading from before the update does not show it was absent then)"
default:
return false, "the tunnel is " + tunnel return false, "the tunnel is " + tunnel
} }
return true, "" return true, ""
} }
// noTunnelBefore: the reading from the start of the pass shows the guest's Docker answered and listed NO cloudflared
// container. A missing or failed reading is not „no tunnel" (an empty list from a failed `docker ps` must not pass).
func noTunnelBefore(before *Health) bool {
if before == nil || before.Guest == nil || !before.Guest.DockerOK {
return false
}
_, had := before.Guest.Containers["cloudflared"]
return !had
}
// EngineOf is the engine version `docker version` prints for a docker-ce package version: "5:29.8.2-1~debian.13~trixie" // EngineOf is the engine version `docker version` prints for a docker-ce package version: "5:29.8.2-1~debian.13~trixie"
// → "29.8.2" (no epoch, no Debian revision). // → "29.8.2" (no epoch, no Debian revision).
func EngineOf(pkgVersion string) string { func EngineOf(pkgVersion string) string {
+39
View File
@@ -568,3 +568,42 @@ func TestDocker_OOMCheckReachesTheHubUnchanged(t *testing.T) {
t.Fatalf("no docker report posted: %s", calls(w)) t.Fatalf("no docker report posted: %s", calls(w))
} }
} }
// R-1070: a box with no tunnel set up (no cloudflared container before the update and none after) passes the host
// rule; a box WITH a tunnel that is down, gone, or unreadable still fails it.
//
// COMPANION RED-PROOF (observed 2026-10-19): with the „not applicable" branch switched off (only `running` passes, the
// pre-fix rule) the first case fails with
//
// leg_test.go: no tunnel before, none after: got false (the tunnel is absent (no cloudflared container now, …)), want true
//
// and with the pre-fix case-sensitive match TestClassifyTunnel's Docker 29 row reads `unknown` („could not ask the
// guest"), the state Tester4 failed on.
// Restored.
func TestHostHealthVerdict_R1070_NoTunnelSetUp(t *testing.T) {
withTunnel := hostOK()
withTunnel.Guest.Containers["cloudflared"] = Container{State: "running", Health: "healthy"}
dockerFailed := hostOK()
dockerFailed.Guest = &Health{DockerOK: false, Containers: map[string]Container{}}
cases := []struct {
name string
before *Health
tunnel string
want bool
why string
}{
{"no tunnel before, none after", hostOK(), hub.TunnelAbsent, true, ""},
{"a tunnel before, gone after", withTunnel, hub.TunnelAbsent, false, "cloudflared was running"},
{"a tunnel before, down after", withTunnel, hub.TunnelNotRunning, false, "cloudflared was running"},
{"no tunnel before, a stopped one after", hostOK(), hub.TunnelNotRunning, false, "not_running"},
{"no tunnel before, the probe could not ask after", hostOK(), hub.TunnelUnknown, false, "unknown"},
{"no reading from before", nil, hub.TunnelAbsent, false, "absent"},
{"docker failed before (an empty list is not proof of none)", dockerFailed, hub.TunnelAbsent, false, "absent"},
}
for _, c := range cases {
got, why := HostHealthVerdict(c.before, hostOK(), c.tunnel)
if got != c.want || !strings.Contains(why, c.why) {
t.Errorf("%s: got %v (%s), want %v (…%s…)", c.name, got, why, c.want, c.why)
}
}
}
+12
View File
@@ -180,6 +180,18 @@ func (c *Client) DeleteSnapshot(ctx context.Context, vmid int, snapname string)
return c.dataString(ctx, http.MethodDelete, path, nil) return c.dataString(ctx, http.MethodDelete, path, nil)
} }
// DeleteSnapshotForce is DeleteSnapshot with Proxmox's own `force=1` („remove from the config file, even if removing
// disk snapshots fails"): the API's flag, so the same token privilege (VM.Snapshot) and no new root command. Used by
// the stale-lock janitor ONLY after a delete failed because one disk's snapshot volume is already gone (R-1010). The
// flag rides the query string, where pveproxy reads a DELETE's parameters. Pinned by TestDeleteSnapshotForce_R1010.
func (c *Client) DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error) {
if vmid == 0 || snapname == "" {
return "", fmt.Errorf("proxmox: DeleteSnapshotForce needs vmid and snapname")
}
path := fmt.Sprintf("/nodes/%s/lxc/%d/snapshot/%s?force=1", c.node, vmid, url.PathEscape(snapname))
return c.dataString(ctx, http.MethodDelete, path, nil)
}
// SetConfig applies config changes via PUT /nodes/{node}/lxc/{vmid}/config // SetConfig applies config changes via PUT /nodes/{node}/lxc/{vmid}/config
// (e.g. memory, cores, net0, mpN with a backup flag). PVE may apply this // (e.g. memory, cores, net0, mpN with a backup flag). PVE may apply this
// synchronously (no UPID) — the returned string is empty in that case, and "" is // synchronously (no UPID) — the returned string is empty in that case, and "" is
+22
View File
@@ -3,6 +3,7 @@ package proxmox
import ( import (
"context" "context"
"io" "io"
"net/http"
"strings" "strings"
"testing" "testing"
) )
@@ -99,3 +100,24 @@ func TestDestroyScratchLXC_RefusesUnconfiguredBand(t *testing.T) {
} }
} }
} }
// R-1010: the forced snapshot delete is the API's own flag on the same DELETE (no new root command).
func TestDeleteSnapshotForce_R1010(t *testing.T) {
var got *http.Request
d := &mockDoer{fn: func(r *http.Request) (*http.Response, error) {
got = r
return jsonResp(200, `{"data":"UPID:demo-felhom:0001:0002:0003:vzdelsnapshot:9201:u@pve!t:"}`), nil
}}
upid, err := newTestClient(d).DeleteSnapshotForce(context.Background(), 9201, "vzdump")
if err != nil || upid == "" {
t.Fatalf("DeleteSnapshotForce: upid %q err %v", upid, err)
}
if got.Method != http.MethodDelete || got.URL.Path != "/api2/json/nodes/demo-felhom/lxc/9201/snapshot/vzdump" ||
got.URL.Query().Get("force") != "1" {
t.Fatalf("request = %s %s, want DELETE …/lxc/9201/snapshot/vzdump?force=1", got.Method, got.URL.String())
}
d.fn = func(r *http.Request) (*http.Response, error) { got = r; return jsonResp(200, `{"data":null}`), nil }
if _, err := newTestClient(d).DeleteSnapshot(context.Background(), 9201, "vzdump"); err != nil || got.URL.RawQuery != "" {
t.Fatalf("the plain delete must carry no force: %s (err %v)", got.URL.String(), err)
}
}
+3
View File
@@ -645,6 +645,9 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
} }
if spec.MemoryMB > 0 { if spec.MemoryMB > 0 {
params["memory"] = strconv.Itoa(spec.MemoryMB) params["memory"] = strconv.Itoa(spec.MemoryMB)
// R-1192: swap = RAM. Left unset, Proxmox gives an LXC 512 MB, and a guest that fills its RAM can then evict
// only program code — the measured freeze (localapi.GuestSwapMB carries the reasoning and the numbers).
params["swap"] = strconv.Itoa(spec.MemoryMB)
} }
for i, m := range spec.Mounts { for i, m := range spec.Mounts {
// backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED // backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED
+19
View File
@@ -0,0 +1,19 @@
package reconcile
import (
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: a new guest is created with swap = its RAM (left unset, Proxmox gives an LXC 512 MB — the measured freeze,
// felhom.eu audits/fixes-2026-10-21/P4-freeze/). COMPANION RED-PROOF: drop params["swap"] in buildBringUpConfig.
func TestR1192_BringUpSetsSwapToTheRAM(t *testing.T) {
p := buildBringUpConfig(BringUpSpec{MemoryMB: 4096}, proxmox.GuestConfig{})
if p["memory"] != "4096" || p["swap"] != "4096" {
t.Fatalf("bring-up params memory=%q swap=%q, want 4096/4096", p["memory"], p["swap"])
}
if q := buildBringUpConfig(BringUpSpec{}, proxmox.GuestConfig{}); q["swap"] != "" {
t.Fatalf("no memory in the spec → no swap written either, got %q", q["swap"])
}
}
+32 -10
View File
@@ -419,8 +419,9 @@ func (h *SudoHostOps) InspectDevice(ctx context.Context, device string) (DeviceP
// any TYPE/PTTYPE/USAGE line is positive data-bearing evidence. Its exit code is NOT relied // any TYPE/PTTYPE/USAGE line is positive data-bearing evidence. Its exit code is NOT relied
// on (blkid exits 2 on a blank device) — output presence is what matters. A broken/empty // on (blkid exits 2 on a blank device) — output presence is what matters. A broken/empty
// blkid simply adds no positive evidence; lsblk (below) is the read-success authority. // blkid simply adds no positive evidence; lsblk (below) is the read-success authority.
bout, _, _ := h.runner.Run(ctx, h.bins.Blkid, "-p", "-o", "export", device) bout, _, berr := h.runner.Run(ctx, h.bins.Blkid, "-p", "-o", "export", device)
for k, v := range parseBlkidExport(bout) { blk := parseBlkidExport(bout)
for k, v := range blk {
switch k { switch k {
case "TYPE": case "TYPE":
probe.HasFilesystem = true probe.HasFilesystem = true
@@ -439,11 +440,11 @@ func (h *SudoHostOps) InspectDevice(ctx context.Context, device string) (DeviceP
// lsblk -J is the READ-SUCCESS authority + the partition/mount view. It exits 0 on any valid // lsblk -J is the READ-SUCCESS authority + the partition/mount view. It exits 0 on any valid
// device (blank or not), so a clean parse means the agent reliably read the device. If lsblk // device (blank or not), so a clean parse means the agent reliably read the device. If lsblk
// fails, Probed stays false → DataBearing()=true (fail-safe — never call a device blank on a // fails, Probed stays false → DataBearing()=true (fail-safe — never call a device blank on a
// failed read). // failed read). An exit 0 whose output does not parse into a device is NOT a read either (R-1093
// review): it stays unprobed, never "blank".
lout, _, lerr := h.runner.Run(ctx, h.bins.Lsblk, "-J", "-o", "NAME,FSTYPE,PTTYPE,MOUNTPOINT", device) lout, _, lerr := h.runner.Run(ctx, h.bins.Lsblk, "-J", "-o", "NAME,FSTYPE,PTTYPE,MOUNTPOINT", device)
if lerr == nil { if hasChildren, mounted, fstype, pttype, parsed := parseLsblkDevice(lout); lerr == nil && parsed {
probe.Probed = true probe.Probed = true
hasChildren, mounted, fstype, pttype := parseLsblkDevice(lout)
if hasChildren { if hasChildren {
probe.HasPartitions = true probe.HasPartitions = true
} }
@@ -456,13 +457,33 @@ func (h *SudoHostOps) InspectDevice(ctx context.Context, device string) (DeviceP
probe.FSType = fstype probe.FSType = fstype
} }
} }
if pttype != "" { if pttype != "" && !inheritedPartitionTable(blk, berr, pttype) {
probe.HasPartitionTable = true probe.HasPartitionTable = true
} }
} }
return probe, nil return probe, nil
} }
// inheritedPartitionTable reports whether lsblk's PTTYPE for the probed node is only the PARENT disk's
// table (R-1093). lsblk prints the disk's table type on every one of its partitions, so an EMPTY
// partition read as "device has a partition table" and asked for the typed erase confirmation.
//
// The partition's OWN signatures come from `blkid -p` on the partition itself (above): a nested table
// inside it is its PTTYPE, a filesystem is its TYPE. So lsblk's value is set aside ONLY when blkid
// positively proved two things — that it completed its probe of the device (exit 0: PART_ENTRY_* alone
// comes from the PARENT's table, so output without a clean exit does not prove the partition's contents
// were read), and that the device is a partition of a table of exactly that type (PART_ENTRY_NUMBER +
// PART_ENTRY_SCHEME == lsblk's PTTYPE). Anything else keeps the old, safe reading: a blkid that failed,
// exited non-zero or printed nothing, a whole disk, or a scheme that differs from lsblk's (then lsblk sees
// a table blkid did not attribute to the parent). Pinned by TestInspect_R1093_* (hostops_disk_test.go).
func inheritedPartitionTable(blkid map[string]string, blkidErr error, lsblkPTType string) bool {
if blkidErr != nil || blkid["PART_ENTRY_NUMBER"] == "" {
return false
}
scheme := blkid["PART_ENTRY_SCHEME"]
return scheme != "" && strings.EqualFold(scheme, lsblkPTType)
}
// Format runs mkfs.<fstype> on a validated device. The caller is responsible for authorization // Format runs mkfs.<fstype> on a validated device. The caller is responsible for authorization
// (8C: only after classifying the device non-data-bearing, or via a slice-10 operator signature). // (8C: only after classifying the device non-data-bearing, or via a slice-10 operator signature).
func (h *SudoHostOps) Format(ctx context.Context, device, fstype string) error { func (h *SudoHostOps) Format(ctx context.Context, device, fstype string) error {
@@ -541,13 +562,14 @@ type lsblkDevice struct {
} }
// parseLsblkDevice parses `lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT <device>` for the top device: // parseLsblkDevice parses `lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT <device>` for the top device:
// whether it has child partitions, is mounted (itself or any child), and its fstype/pttype. // whether it has child partitions, is mounted (itself or any child), and its fstype/pttype. parsed=false
func parseLsblkDevice(out []byte) (hasChildren, mounted bool, fstype, pttype string) { // when the output is not a device listing — the caller must then treat the device as unprobed.
func parseLsblkDevice(out []byte) (hasChildren, mounted bool, fstype, pttype string, parsed bool) {
var doc struct { var doc struct {
BlockDevices []lsblkDevice `json:"blockdevices"` BlockDevices []lsblkDevice `json:"blockdevices"`
} }
if json.Unmarshal(out, &doc) != nil || len(doc.BlockDevices) == 0 { if json.Unmarshal(out, &doc) != nil || len(doc.BlockDevices) == 0 {
return false, false, "", "" return false, false, "", "", false
} }
d := doc.BlockDevices[0] d := doc.BlockDevices[0]
fstype, pttype = d.FSType, d.PTType fstype, pttype = d.FSType, d.PTType
@@ -558,7 +580,7 @@ func parseLsblkDevice(out []byte) (hasChildren, mounted bool, fstype, pttype str
mounted = true mounted = true
} }
} }
return hasChildren, mounted, fstype, pttype return hasChildren, mounted, fstype, pttype, true
} }
// NoopHostOps is the safe fallback when the privileged surface is unavailable or declined // NoopHostOps is the safe fallback when the privileged surface is unavailable or declined
+111
View File
@@ -222,3 +222,114 @@ func TestInspect_ReadsFilesystemUUID(t *testing.T) {
t.Fatalf("FSUUID = %q, want the blkid UUID", p.FSUUID) t.Fatalf("FSUUID = %q, want the blkid UUID", p.FSUUID)
} }
} }
// ---- R-1093: an EMPTY partition is blank; lsblk's PTTYPE on a partition is the PARENT disk's ----------
// blkidEmptyGPTPart is `blkid -p -o export /dev/sdb1` for an empty GPT partition: only the partition's
// own entry in the parent table (PART_ENTRY_*), no TYPE, no PTTYPE — nothing inside the partition.
const blkidEmptyGPTPart = "DEVNAME=/dev/sdb1\nPART_ENTRY_SCHEME=gpt\nPART_ENTRY_UUID=3b0c1a52-55a4-4c4e-9d0b-7a1f0e0f7c11\n" +
"PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4\nPART_ENTRY_NUMBER=1\nPART_ENTRY_OFFSET=2048\n" +
"PART_ENTRY_SIZE=2097152\nPART_ENTRY_DISK=8:16\n"
// lsblkPartInheritsGPT is `lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT /dev/sdb1`: lsblk prints the DISK's
// table type on the partition — the input that made the agent read an empty partition as data-bearing.
const lsblkPartInheritsGPT = `{"blockdevices":[{"name":"sdb1","fstype":null,"pttype":"gpt","mountpoint":null}]}`
// TestInspect_R1093_PartitionVerdicts pins the CONSEQUENCE (DataBearing + Reason, what decides whether
// the typed erase confirmation is asked) for a partition, in both directions.
//
// COMPANION RED-PROOFS (observed 2026-10-18): (1) restore the pre-fix `if pttype != "" {` (drop the
// inheritedPartitionTable call) → BOTH empty-partition rows (GPT and dos) FAIL with
// `DataBearing=true reason="device has a partition table", want blank`. (2) drop the `blkidErr != nil`
// check → the "exit 2" and "I/O error" rows FAIL with `DataBearing=false reason="device is blank"`.
// (3) drop `&& parsed` → both lsblk-exit-0 rows FAIL with `Probed=true … reason="device is blank"`
// (the unparsable-output hole predates R-1093; closed here). Restored.
func TestInspect_R1093_PartitionVerdicts(t *testing.T) {
cases := []struct {
name string
dev string
blkid string
blkidErr error
lsblk string
lsblkErr error
wantBear bool
wantReason string
unprobed bool // the verdict must be "could not be reliably inspected"
}{
{name: "empty GPT partition is blank (the R-1093 case)", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, lsblk: lsblkPartInheritsGPT,
wantBear: false, wantReason: "device is blank"},
{name: "empty MBR (dos) partition is blank", dev: "/dev/sdb2",
blkid: "DEVNAME=/dev/sdb2\nPART_ENTRY_SCHEME=dos\nPART_ENTRY_TYPE=0x83\nPART_ENTRY_NUMBER=2\nPART_ENTRY_DISK=8:16\n",
lsblk: `{"blockdevices":[{"name":"sdb2","fstype":null,"pttype":"dos","mountpoint":null}]}`,
wantBear: false, wantReason: "device is blank"},
{name: "partition with a filesystem stays data-bearing", dev: "/dev/sdb1",
blkid: "DEVNAME=/dev/sdb1\nUUID=8c1e\nTYPE=ext4\nUSAGE=filesystem\n" + blkidEmptyGPTPart[len("DEVNAME=/dev/sdb1\n"):],
lsblk: `{"blockdevices":[{"name":"sdb1","fstype":"ext4","pttype":"gpt","mountpoint":null}]}`,
wantBear: true, wantReason: "device has a ext4 filesystem"},
{name: "partition holding its OWN (nested) table stays data-bearing", dev: "/dev/sdb1",
blkid: "DEVNAME=/dev/sdb1\nPTUUID=0b8d4358\nPTTYPE=dos\n" + blkidEmptyGPTPart[len("DEVNAME=/dev/sdb1\n"):],
lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "blkid printed nothing: lsblk's PTTYPE stands (fail-safe)", dev: "/dev/sdb1",
blkid: "", blkidErr: errors.New("sudo: a password is required"), lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
// Fail-closed paths (the 2026-10-18 review): PART_ENTRY_* is read from the PARENT's table, so it
// proves nothing about the partition's contents unless blkid also completed (exit 0).
{name: "blkid exit 2 with only PART_ENTRY output: lsblk stands", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, blkidErr: errors.New("exit status 2"), lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "blkid I/O error after PART_ENTRY output: lsblk stands", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, blkidErr: errors.New("exit status 1"), lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "blkid binary missing: lsblk stands", dev: "/dev/sdb1",
blkid: "", blkidErr: errors.New(`exec: "/usr/sbin/blkid": file does not exist`), lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "lsblk exit 0 with unparsable output: unprobed", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, lsblk: `lsblk: something odd`,
wantBear: true, unprobed: true},
{name: "lsblk exit 0 with no device listed: unprobed", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, lsblk: `{"blockdevices":[]}`,
wantBear: true, unprobed: true},
{name: "lsblk failed: unprobed", dev: "/dev/sdb1",
blkid: blkidEmptyGPTPart, lsblk: "", lsblkErr: errors.New("exit status 32"),
wantBear: true, unprobed: true},
{name: "scheme differs from lsblk's PTTYPE: lsblk stands", dev: "/dev/sdb1",
blkid: strings.Replace(blkidEmptyGPTPart, "PART_ENTRY_SCHEME=gpt", "PART_ENTRY_SCHEME=dos", 1),
lsblk: lsblkPartInheritsGPT,
wantBear: true, wantReason: "device has a partition table"},
{name: "whole disk with a GPT stays data-bearing", dev: "/dev/sdb",
blkid: "DEVNAME=/dev/sdb\nPTUUID=a07f88ff\nPTTYPE=gpt\n",
lsblk: `{"blockdevices":[{"name":"sdb","fstype":null,"pttype":"gpt","mountpoint":null}]}`,
wantBear: true, wantReason: "device has a partition table"},
{name: "whole disk: blkid silent, lsblk GPT stands", dev: "/dev/sdb",
blkid: "", lsblk: `{"blockdevices":[{"name":"sdb","fstype":null,"pttype":"gpt","mountpoint":null}]}`,
wantBear: true, wantReason: "device has a partition table"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
r := &scriptedRunner{
outputs: map[string][]byte{"blkid": []byte(c.blkid), "lsblk": []byte(c.lsblk)},
errs: map[string]error{"blkid": c.blkidErr, "lsblk": c.lsblkErr},
}
p, err := newSudo(r).InspectDevice(context.Background(), c.dev)
if err != nil {
t.Fatal(err)
}
if c.unprobed {
if p.Probed || !p.DataBearing() || p.Reason() != "device could not be reliably inspected" {
t.Fatalf("Probed=%v DataBearing=%v reason=%q, want unprobed + data-bearing (probe %+v)",
p.Probed, p.DataBearing(), p.Reason(), p)
}
return
}
if !p.Probed {
t.Fatal("expected a clean probe (lsblk read cleanly)")
}
if p.DataBearing() != c.wantBear || p.Reason() != c.wantReason {
t.Fatalf("DataBearing=%v reason=%q, want DataBearing=%v reason=%q (probe %+v)",
p.DataBearing(), p.Reason(), c.wantBear, c.wantReason, p)
}
})
}
}
+25
View File
@@ -661,6 +661,9 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
if age, ok := parseHandshakeAge(string(out), m.now()); ok { if age, ok := parseHandshakeAge(string(out), m.now()); ok {
st.LastHandshakeAgeS = &age st.LastHandshakeAgeS = &age
} }
if fut, ok := handshakeFuture(string(out), m.now()); ok {
st.HandshakeFutureS = &fut
}
} }
} }
if st.Pubkey == "" && !st.Registered && !st.Active { if st.Pubkey == "" && !st.Registered && !st.Active {
@@ -669,6 +672,28 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
return st return st
} }
// handshakeFuture (R-1204) reports how many seconds the latest handshake lies in the future, ok only when it does
// (by more than a minute — clock jitter is not the case). parseHandshakeAge clamps that case to 0 („just now"), which
// is what made a box whose clock had run ahead look healthy while ep0 refused its tunnel. Pinned by
// TestR1204_AFutureHandshakeIsReported.
func handshakeFuture(out string, now time.Time) (int64, bool) {
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
f := strings.Fields(line)
if len(f) != 2 {
continue
}
epoch, err := strconv.ParseInt(f[1], 10, 64)
if err != nil || epoch <= 0 {
continue
}
if fut := epoch - now.Unix(); fut > 60 {
return fut, true
}
return 0, false
}
return 0, false
}
// parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>"). // parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>").
// epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0). // epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0).
func parseHandshakeAge(out string, now time.Time) (int64, bool) { func parseHandshakeAge(out string, now time.Time) (int64, bool) {
+28
View File
@@ -696,3 +696,31 @@ func TestInitialResolveFailureNoTeardown(t *testing.T) {
t.Errorf("did not recover after DNS returned: %v", rr.calls) t.Errorf("did not recover after DNS returned: %v", rr.calls)
} }
} }
// R-1204: a box that ran once with its clock ahead keeps a handshake stamped in the FUTURE — ep0 then refuses every
// newer one. The age stays clamped at 0 (unchanged contract), and the new field says how far ahead it lies; a
// handshake in the past (or within a minute of now) never sets it.
// RED-PROOF: with Status not setting HandshakeFutureS the first assertion fails — seen failing 2026-10-22.
func TestR1204_AFutureHandshakeIsReported(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
ctx := context.Background()
m.Apply(ctx, false, nil)
*active = true
m.now = func() time.Time { return time.Unix(1783107115-2*24*3600, 0) } // the clock came back two days earlier
st := m.Status(ctx)
if st.HandshakeFutureS == nil || *st.HandshakeFutureS != 2*24*3600 {
t.Fatalf("handshake_future_s = %v, want %d", st.HandshakeFutureS, 2*24*3600)
}
if st.LastHandshakeAgeS == nil || *st.LastHandshakeAgeS != 0 {
t.Fatalf("the age stays clamped at 0: %v", st.LastHandshakeAgeS)
}
m.now = func() time.Time { return time.Unix(1783107115+100, 0) }
if st = m.Status(ctx); st.HandshakeFutureS != nil {
t.Fatalf("a past handshake must not set handshake_future_s: %v", *st.HandshakeFutureS)
}
m.now = func() time.Time { return time.Unix(1783107115-30, 0) } // 30 s of jitter is not the case
if st = m.Status(ctx); st.HandshakeFutureS != nil {
t.Fatalf("clock jitter must not set handshake_future_s")
}
}
+13
View File
@@ -15,6 +15,9 @@ Gates (all must pass; **non-zero exit on any failure**):
or because someone is offline on a train. CI runs the FULL set for exactly this or because someone is offline on a train. CI runs the FULL set for exactly this
reason: it is the machine that can afford a network check, and it is the half reason: it is the machine that can afford a network check, and it is the half
that emails when something is wrong. that emails when something is wrong.
… security-scan govulncheck (called vulnerabilities) + gitleaks on the pushed commits, WARN-FIRST: it lists,
it never refuses (R-986). Shared: felhom.eu/scripts/security_scan_gate.py. In the fast set
although it reads vuln.go.dev — offline it says NOT CHECKED, so it cannot stop a push.
… gate-decoys the decoy suite (test_gate_decoys.py): every gate here still convicts its decoys … gate-decoys the decoy suite (test_gate_decoys.py): every gate here still convicts its decoys
(R-1116). ~20 s in the workspace, so the full run only: not the hook, but CI (R-1116). ~20 s in the workspace, so the full run only: not the hook, but CI
runs the full set on every push. Scratch clones and a local fake Gitea only. runs the full set on every push. Scratch clones and a local fake Gitea only.
@@ -49,6 +52,9 @@ SHARED_INSTRUCTIONS = os.path.join(
# R-389 — shared, like the two above: it lives in felhom.eu/scripts/ and is never copied. # R-389 — shared, like the two above: it lives in felhom.eu/scripts/ and is never copied.
SHARED_OBSERVATIONS = os.path.join( SHARED_OBSERVATIONS = os.path.join(
os.path.dirname(ROOT), "felhom.eu", "scripts", "observations_gate.py") os.path.dirname(ROOT), "felhom.eu", "scripts", "observations_gate.py")
# R-986 — shared too: the dependency and secret scanner, warn-first.
SHARED_SECURITY = os.path.join(
os.path.dirname(ROOT), "felhom.eu", "scripts", "security_scan_gate.py")
# (label, absolute script path, args, fast) # (label, absolute script path, args, fast)
GATES = [ GATES = [
@@ -67,6 +73,13 @@ GATES = [
# runs the FULL set, so every push is still judged by it. It plants only in scratch clones and asks a fake # runs the FULL set, so every push is still judged by it. It plants only in scratch clones and asks a fake
# Gitea on 127.0.0.1 (never the network, never the real tree); it does not call this runner (no recursion). # Gitea on 127.0.0.1 (never the network, never the real tree); it does not call this runner (no recursion).
("gate-decoys", os.path.join(ROOT, "scripts", "test_gate_decoys.py"), [], False), ("gate-decoys", os.path.join(ROOT, "scripts", "test_gate_decoys.py"), [], False),
# R-1160 (2026-10-19) — every Go file is gofmt-clean; the controller's gofmt_gate.py shape. Fast: one gofmt -l.
# Where the runner has no Go it says NOT CHECKED out loud and passes; the pre-push hook (with Go) enforces it.
("gofmt", os.path.join(ROOT, "scripts", "gofmt_gate.py"), [], True),
# R-986 (2026-10-19) — govulncheck + gitleaks, WARN-FIRST (exit 0 whatever it finds). Fast although it reads
# vuln.go.dev: it cannot refuse, so a network blink is a NOT CHECKED line. ~4 s here. CI: gitleaks fetched pinned
# (FELHOM_SCAN_FETCH=1), no Go, so govulncheck says NOT CHECKED there.
("security-scan", SHARED_SECURITY, [ROOT], True),
] ]
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"} VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""gofmt_gate.py — every Go file in this module is gofmt-clean (R-1160; the controller's R-454 gate, same shape).
MEASURED 2026-10-19: 14 agent files were not gofmt-clean and nothing noticed (fixed by one formatting-only commit,
1edfd8e). The controller's history below is why a gate and not a one-off clean-up.
MEASURED 2026-09-02: five internal/web test files had been gofmt-unclean for an unknown length of
time and nothing noticed — `go vet` does not check formatting and no gate did. By 2026-10-05 the five
were clean and TWELVE others were not: the count only grows while nothing looks, and every `gofmt -l`
someone runs by hand is then noise that hides a real one.
Run from the repo root: python3 scripts/gofmt_gate.py
Exit 0 clean · 1 convicted (lists the files) · 2 inconclusive (gofmt not found — never a pass).
gofmt is the Go toolchain's own formatter (not a shell utility), found on PATH or under
`go env GOROOT`/bin. Decoy: scripts/test_gate_decoys.py ("gofmt"), in a scratch copy of this script.
"""
import os
import shutil
import subprocess
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ROOTS = ["cmd", "internal"]
def find_gofmt():
p = shutil.which("gofmt")
if p:
return p
go = shutil.which("go")
if go:
try:
root = subprocess.check_output([go, "env", "GOROOT"], text=True).strip()
cand = os.path.join(root, "bin", "gofmt")
if os.path.isfile(cand):
return cand
except Exception:
pass
return None
def in_ci():
"""True on the Gitea Actions runner (act sets GITHUB_ACTIONS, Gitea sets GITEA_ACTIONS)."""
return os.environ.get("GITEA_ACTIONS") == "true" or os.environ.get("GITHUB_ACTIONS") == "true"
def main():
gofmt = find_gofmt()
if not gofmt:
# The CI runner may have no Go toolchain (felhom.eu CLAUDE.md, R-887 note). There the gate
# says so OUT LOUD and passes: the pre-push hook on the developer's machine, where Go always exists, is where
# formatting is enforced. Anywhere else a missing gofmt stays INCONCLUSIVE — never a silent pass.
# Pinned by the "gofmt: CI without Go" and "gofmt: dev without Go" decoys in test_gate_decoys.py.
if in_ci():
print("gofmt gate NOT CHECKED in CI: no Go toolchain on the runner — the pre-push hook (with Go) enforces it")
return 0
print("gofmt gate INCONCLUSIVE: gofmt not found on PATH or under `go env GOROOT`/bin")
return 2
roots = [r for r in ROOTS if os.path.isdir(os.path.join(ROOT, r))]
if not roots:
print("gofmt gate INCONCLUSIVE: none of %s exists under %s" % (ROOTS, ROOT))
return 2
p = subprocess.run([gofmt, "-l"] + roots, cwd=ROOT, capture_output=True, text=True)
if p.returncode != 0:
print("gofmt gate INCONCLUSIVE: gofmt exited %d: %s" % (p.returncode, p.stderr.strip()[:400]))
return 2
bad = [l for l in p.stdout.splitlines() if l.strip()]
if bad:
for f in bad:
print(" not gofmt-clean: %s" % f)
print("GOFMT GATE FAILED: %d file(s) — run `gofmt -w <file>` (formatting only, no behaviour change)" % len(bad))
return 1
print("gofmt gate OK — every Go file under %s is gofmt-clean" % ", ".join(roots))
return 0
if __name__ == "__main__":
sys.exit(main())
+50
View File
@@ -66,6 +66,14 @@ COVERS = {
"inside an HTML comment (R-426)", "inside an HTML comment (R-426)",
"observations": "R-419 in THIS repo's REPORT.md: an Observations note SAYING it carries no marker - vs the " "observations": "R-419 in THIS repo's REPORT.md: an Observations note SAYING it carries no marker - vs the "
"two genuine markers (R-426)", "two genuine markers (R-426)",
"gofmt": "R-1160: an unformatted .go file planted under internal/ of a SCRATCH copy is convicted by name - vs the "
"same tree formatted (passes); with no gofmt reachable, CI passes OUT LOUD and a dev machine is "
"INCONCLUSIVE, never a silent pass",
"security-scan": "R-986: the shared gate's own decoys (felhom.eu scripts/test_security_scan_gate.py, 8 cases in "
"SCRATCH git repos: a planted token listed by rule and file, its value never printed; a second "
"secret on another line of an accepted file still listed; a called govulncheck finding listed, "
"an imported-only one not; no tool is NOT CHECKED out loud), run from here; plus the gate on THIS "
"repo's tree: exit 0 and its summary line (warn-first - the listing is the fact)",
"gate-decoys": "this suite, run as a gate of agent_gates.py's full run, which CI runs on every push (R-1116): " "gate-decoys": "this suite, run as a gate of agent_gates.py's full run, which CI runs on every push (R-1116): "
"every case above is a decoy of its own gate, and ONE case that stops convicting turns the run " "every case above is a decoy of its own gate, and ONE case that stops convicting turns the run "
"red - red-proved 2026-10-17 by flipping a reuse-refs FACT case to expect rc 0", "red - red-proved 2026-10-17 by flipping a reuse-refs FACT case to expect rc 0",
@@ -291,6 +299,32 @@ def release_cases(base, ws):
# ── the shared felhom.eu scripts, against THIS repo's inputs ───────────────────────────────────── # ── the shared felhom.eu scripts, against THIS repo's inputs ─────────────────────────────────────
def gofmt_cases(ws):
"""R-1160: the gofmt gate in a scratch root (its ROOT is its own grandparent directory), never the real tree."""
root = os.path.join(ws, "gofmt-root")
os.makedirs(os.path.join(root, "scripts"))
os.makedirs(os.path.join(root, "internal", "x"))
shutil.copy(os.path.join(ROOT, "scripts", "gofmt_gate.py"), os.path.join(root, "scripts", "gofmt_gate.py"))
gate = os.path.join(root, "scripts", "gofmt_gate.py")
planted = os.path.join(root, "internal", "x", "zz_gofmt_decoy.go")
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F( ) int {return 1}\n")
if shutil.which("gofmt") or shutil.which("go"):
rc, out = run([sys.executable, gate], root)
report("gofmt: FACT: an unformatted file under internal/", rc, out, 1, ("zz_gofmt_decoy.go", "GOFMT GATE FAILED"))
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F() int { return 1 }\n")
rc, out = run([sys.executable, gate], root)
report("gofmt: GENUINE: the same file formatted", rc, out, 0, ("gofmt gate OK",))
else:
print(" -- gofmt: not runnable here (no Go toolchain) - the two cases below pin that mode")
for ci, want_rc, want, name in ((True, 0, "NOT CHECKED in CI", "gofmt: CI without Go"),
(False, 2, "INCONCLUSIVE", "gofmt: dev without Go")):
env = {"PATH": os.path.join(ws, "empty-path"), "HOME": ws}
if ci:
env["GITEA_ACTIONS"] = "true"
rc, out = run([sys.executable, gate], root, env=env)
report(name, rc, out, want_rc, (want,))
def shared_cases(ws): def shared_cases(ws):
"""A scratch WORKSPACE: a clone of this repo beside symlinks to the siblings, because the shared """A scratch WORKSPACE: a clone of this repo beside symlinks to the siblings, because the shared
scripts reach across (REUSE.md cites hub paths; instructions_gate reads the workspace CLAUDE.md).""" scripts reach across (REUSE.md cites hub paths; instructions_gate reads the workspace CLAUDE.md)."""
@@ -343,6 +377,20 @@ def shared_cases(ws):
u"the same minute.**\n", 0) u"the same minute.**\n", 0)
def security_cases():
"""R-986: the shared scanner's decoys (they build their own scratch repos), then the gate on this repo."""
suite = os.path.join(SHARED, "test_security_scan_gate.py")
gate = os.path.join(SHARED, "security_scan_gate.py")
for f in (suite, gate):
if not os.path.isfile(f):
fails.append("shared %s is MISSING beside this clone - a failure, never a skip" % f)
return
rc, out = run([sys.executable, suite], ROOT)
report("security-scan: the shared gate's 8 decoys", rc, out, 0, ("all 8 cases passed",))
rc, out = run([sys.executable, gate, ROOT], ROOT)
report("security-scan: GENUINE: this repo, warn-first", rc, out, 0, ("security-scan:", "govulncheck"))
def main(): def main():
srv = Server(("127.0.0.1", 0), Handler) srv = Server(("127.0.0.1", 0), Handler)
threading.Thread(target=srv.serve_forever, daemon=True).start() threading.Thread(target=srv.serve_forever, daemon=True).start()
@@ -353,6 +401,8 @@ def main():
published_cases(base) published_cases(base)
release_cases(base, ws) release_cases(base, ws)
shared_cases(ws) shared_cases(ws)
gofmt_cases(ws)
security_cases()
finally: finally:
srv.shutdown() srv.shutdown()
srv.server_close() srv.server_close()