Compare commits

..

8 Commits

Author SHA1 Message Date
admin 51521230c9 REPORT: the night's agent fixes (R-921, R-1070, R-1010), not released
gates / gates (push) Successful in 1m25s
2026-10-19 21:47:40 +02:00
admin 84ef5818d5 security-scan gate (shared felhom.eu/scripts/security_scan_gate.py) in the fast set and CI, warn-first, with the accepted-hits file and decoys (R-986)
gates / gates (push) Successful in 1m30s
2026-10-19 21:11:46 +02:00
admin 56aabc729c Unreleased: host_busy on GET /backup/status (R-921), a missing tunnel container is 'absent' and not-applicable after an OS update (R-1070), the janitor forces a half-removed vzdump snapshot once (R-1010)
gates / gates (push) Successful in 58s
2026-10-19 19:52:37 +02:00
admin 0cbd500154 gofmt gate in agent_gates.py with its decoys (R-1160); REPORT for the day
gates / gates (push) Successful in 49s
2026-10-19 10:26:27 +02:00
admin 1edfd8e331 gofmt -w: 14 files, formatting only, no behaviour change (R-1160) 2026-10-19 10:24:32 +02:00
admin 0bc0bc5da0 CONTEXT + REPORT: v0.160.0 released, on the three test hosts, not vouched
gates / gates (push) Successful in 48s
2026-10-19 05:41:34 +02:00
admin 45458196d1 Two texts made true after 0.160.0: the ceremony log line names the confirm order; the R-973 comments name whole_guest_backup_refused (no behaviour change)
gates / gates (push) Successful in 52s
2026-10-19 05:25:58 +02:00
admin 124164db73 CHANGELOG v0.160.0 (released: R-978, R-973, R-1093, R-1149, R-1151)
gates / gates (push) Successful in 46s
2026-10-19 03:22:00 +02:00
37 changed files with 578 additions and 98 deletions
+3
View File
@@ -64,6 +64,9 @@ jobs:
env:
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
FELHOM_SCAN_FETCH: "1"
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
+18
View File
@@ -0,0 +1,18 @@
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
gitleaks generic-api-key@1ad7cc8eb57e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46ce5a3eccc4 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46fe126846b0 internal/escrow/identity_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@56474f1754f5 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@631c53997f7e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@6e790ef41ef3 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@78334b854242 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@7b88d323a44f internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b0eeee93cf24 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b1cb2659967a internal/localapi/escrow_stage_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b66bec43cc8a internal/escrow/recover_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e0525160fe03 internal/pbsdr/wrapper_read_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e442e455c360 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f23f93f461b2 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f471b78e0f6e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
+55 -9
View File
@@ -1,15 +1,55 @@
## Unreleased (2026-10-18) — CI workflow: the failure mail links the public run page, by run id (R-1135); no code change
## Unreleased (2026-10-19) — three fixes for the next release (R-921, R-1070, R-1010); two texts made true after 0.160.0
`.gitea/workflows/gates.yml`, the alarm step: the mail's run link is now `https://gitea.dooplex.hu/<repo>/actions/runs/<run id>`.
It used `GITHUB_SERVER_URL`, which on this runner is the in-cluster address (`http://gitea.gitea-system.svc.cluster.local:3000`,
the address the runner registers at), so nobody outside the cluster could open it. It also used the run NUMBER, and
Gitea 1.26 reads `/actions/runs/<n>` as a run id first: a number that is also an older run's id opened that other run
(measured 2026-10-18: `felhom.eu/actions/runs/1116` opened run id 1116, run number 613). It now uses `GITHUB_RUN_ID` and
prints the link (`ALARM-LINK …`) into the job log. Same change in all four repos' workflows.
Not released (the night brief of 2026-10-19: agent code + tests + commit only; the release is the operator's word). No
sudoers, wrapper or config-bundle change.
## Unreleased — three fixes from the 2026-10-12 security review and the storage build (R-1093, R-978, R-973) (2026-10-18)
- **R-921 — `GET /backup/status` says when the host is busy.** New field `host_busy` (`{busy, what}`): whether the
host-wide heavy-operation gate is held right now (another guest's backup, the restore test, a night OS step, the
stale-lock sweep, a power-off) — the state that makes a `POST /backup` answer 409. Served on every request; absent
when the gate is not wired. Controller ≥ 0.335.0 reads it before it stops the household's apps for a tier, so a
refused tier costs no stop (the measured case of 2026-10-08: the night OS step, all 15 apps stopped for a refusal).
A read, never a reservation. No new route. Test `TestBackupStatus_R921_HostBusy`, red-proved (assignment removed →
`free gate: host_busy = <nil>`).
- **R-1070 — a box with no tunnel set up no longer fails its host OS update.** Docker 29 prints „error: no such
object" in lower case, so the tunnel probe's case-sensitive match fell through to `unknown` („could not ask the
guest"), and the host health rule failed every box without a cloudflared container (Tester4, 2026-10-15,
`os_update_health_failed`). Now the match ignores case and a missing container is a new state, **`absent`** („no
tunnel set up on this box"); the host rule reads it as not applicable only when the reading from before the update
(Docker answered) also had no cloudflared container. A tunnel that is down, gone or unreadable still fails. The hub
alarms only on `not_running`, so `absent` never mails. Tests `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows) and
three `TestClassifyTunnel` rows (Docker 29's wording measured on demo-hp), red-proved both ways.
- **R-1010 — the stale-lock janitor clears a half-removed `vzdump` snapshot.** When the delete fails ONLY with
„Failed to find logical volume …_vzdump" (one disk's snapshot already gone — demo-hp 2026-10-14 after a backup ran
out of space), it retries once with Proxmox's own `force=1` on the same API call (`DeleteSnapshotForce`; the same
token privilege, no sudoers line, no root command — `pvesh usage … -command delete` lists `--force`). Any other error
is returned unchanged; a failed forced delete is returned, never a second force. Tests
`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone` (6 rows) and `TestDeleteSnapshotForce_R1010`, red-proved
(single plain delete → `calls [plain], want [plain force]`).
Committed, not released — the release and its vouch are the operator's.
- **The security-scan gate (R-986), tooling only:** `agent_gates.py` registers the SHARED
`felhom.eu/scripts/security_scan_gate.py` (FAST set: the hook and CI) — govulncheck (called vulnerabilities) and gitleaks
on the pushed commits, **warn-first**. `.security-accepted` holds the first scan's 15 accepted hits (escrow and
local-API test constants), each with a reason. CI fetches the pinned gitleaks (`FELHOM_SCAN_FETCH=1`). Decoys: the
shared 8 plus the gate on this repo, in `test_gate_decoys.py`.
### Earlier on 2026-10-19 (texts only)
- The ceremony-complete log line now says the package uploads at the claim **or at the typed-back confirm** (R-1151);
it read „package uploads at the claim" for both orders (seen in demo-hp's journal, 2026-10-19 01:47 UTC).
- The R-973 comments in `cmd/felhom-agent/main.go` and `internal/localapi/backup_cap.go` named the hub event
`agent_capability_degraded`; since hub v0.159.0 (R-1150) a refused backup raises `whole_guest_backup_refused`.
Found by the hub helper of the day brief 2026-10-19.
- **gofmt, R-1160:** `1edfd8e` formats the 14 Go files that were not gofmt-clean (formatting only, `go test ./...`
green). New gate `scripts/gofmt_gate.py` (the controller's R-454 shape) in `agent_gates.py`, fast set; its decoys in
`scripts/test_gate_decoys.py` plant an unformatted file in a SCRATCH copy (convicted by name), the formatted twin
(passes), and the no-Go modes (CI passes out loud, a dev machine is INCONCLUSIVE). Red-proved: with the conviction
switched off the FACT case reports a LIVE HOLE. The next release carries it; no release for it.
## v0.160.0 — five fixes: an abandoned or un-typed-back escrow ceremony keeps the old code (R-978, R-1151); a whole-guest backup flood is capped (R-973) and the local tier keeps one point per day (R-1149); an empty partition is not „data" (R-1093) (2026-10-19)
Released 2026-10-19 on the operator's yes (02:48 answers, D17 „one per day, 3 days"). sha256
`7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1`, config bundle
`6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f`. The vouch is the operator's.
- **R-1093 — an EMPTY partition no longer asks the typed „data will be erased" confirmation.** `lsblk` prints the
parent disk's table type (PTTYPE) on every partition, so `SudoHostOps.InspectDevice` read a blank partition as
@@ -91,6 +131,12 @@ Committed, not released — the release and its vouch are the operator's.
with keep-last 3 the local tier's three points can still be replaced within one day (3 per day is the schedule's own
pace plus one press); the hub's event text reads „agent privileged capability degraded" (the type is generic). No
sudoers or bundle change.
- **CI workflow (R-1135, no code change).** `.gitea/workflows/gates.yml`, the alarm step: the mail's run link is now `https://gitea.dooplex.hu/<repo>/actions/runs/<run id>`.
It used `GITHUB_SERVER_URL`, which on this runner is the in-cluster address (`http://gitea.gitea-system.svc.cluster.local:3000`,
the address the runner registers at), so nobody outside the cluster could open it. It also used the run NUMBER, and
Gitea 1.26 reads `/actions/runs/<n>` as a run id first: a number that is also an older run's id opened that other run
(measured 2026-10-18: `felhom.eu/actions/runs/1116` opened run id 1116, run number 613). It now uses `GITHUB_RUN_ID` and
prints the link (`ALARM-LINK …`) into the job log. Same change in all four repos' workflows.
## v0.159.2 — the orphan thin check runs only after a failed backup (R-1012 follow-up) (2026-10-18)
+6
View File
@@ -1,5 +1,11 @@
# CONTEXT — felhom-agent working state
> **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978,
> R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with
> `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn
> positions; only those two words are held, 30-min window; a claim without it keeps the R-978 order). Vouch: version
> 0.160.0, sha256 `7ebb547f…dfca1`, bundle `6b060901…e202f`. After it, unreleased: `4545819` (two texts). felhom.eu
> `REPORT-day-2026-10-19.md`.
> **2026-10-18 (night) — UNRELEASED on main: R-1093, R-978, R-973** (release + vouch: the operator's). R-1093 an empty
> partition is blank (lsblk's PTTYPE on a partition is the parent's; set aside only on a clean blkid exit 0 with a
> matching `PART_ENTRY_SCHEME`; an unparsable lsblk is unprobed). R-978 the escrow package reaches the hub only at the
+17 -37
View File
@@ -1,40 +1,20 @@
# REPORT — three agent fixes, committed and NOT released (2026-10-18 evening)
# REPORT — agent, evening/night brief 2026-10-19 (code + tests + commit only, NOT released)
Task: the evening brief of 2026-10-18, Part 4 (felhom.eu `REPORT-evening-2026-10-18.md`). Baseline: `27bbdaa` (v0.159.2
released). Commits: `f7b7308` + `0963bae` (R-1093), `8561dc8` (R-978), `f17e197` (R-973). CHANGELOG under „Unreleased";
the release and its vouch are the operator's. Gates green per commit (`go build/vet/test ./...`, `agent_gates.py --fast`);
CI job 2062 success for `f17e197`.
Commit `56aabc7` (CHANGELOG „Unreleased"), scanner wiring `84ef581`. No sudoers, wrapper or config-bundle change.
The release is the operator's word (STATUS morning list: „Release agent 0.161.0?"), then the vouch.
## R-1093 — an empty partition is blank
- **R-921 — `host_busy` on `GET /backup/status`:** whether the host-wide heavy-operation gate is held (`{busy, what}`),
the state in which `POST /backup` answers 409. Controller 0.335.0 (the floor) reads it before any stop. Test
`TestBackupStatus_R921_HostBusy`, red-proved (assignment removed → `free gate: host_busy = <nil>`).
- **R-1070 — tunnel `absent`:** Docker 29 prints „error: no such object" in lower case; the case-sensitive match read
a missing cloudflared container as `unknown`, which failed the host OS-update rule (Tester4, 2026-10-15). Now a
missing container is `absent`, and the host rule passes it only when the reading from before the step also had no
cloudflared container. Tests `TestHostHealthVerdict_R1070_NoTunnelSetUp`, `TestClassifyTunnel` (Docker 29's wording,
measured on demo-hp), red-proved both ways.
- **R-1010 — the janitor's forced snapshot delete:** on „Failed to find logical volume …_vzdump" only, one retry with
the API's `force=1` (`DeleteSnapshotForce`; `pvesh usage … -command delete` lists `--force`). Tests
`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone` (6 rows), `TestDeleteSnapshotForce_R1010`, red-proved.
- **R-986 — the security-scan gate** (tooling): the shared `felhom.eu/scripts/security_scan_gate.py` in the fast set;
`.security-accepted` holds 15 accepted test constants; decoys in `test_gate_decoys.py`.
lsblk reports the PARENT disk's PTTYPE for a partition. `InspectDevice` now ignores that PTTYPE only when `blkid -p` on the
partition exited 0, printed `PART_ENTRY_NUMBER`, and its `PART_ENTRY_SCHEME` equals lsblk's PTTYPE. Every error path
stays data-bearing: blkid exit 2 / I/O error / missing binary keep „device has a partition table"; an lsblk answer that
exits 0 but does not parse or names no device is „could not be reliably inspected" (an older fail-open, closed here after
a background security review flagged the probe). No new sudo command, no config bundle.
Test `TestInspect_R1093_PartitionVerdicts` (14 rows); three red-proofs. **Live on demo-hp** (a 1 GB loop file in `/root`,
removed after; felhom.eu `audits/agent-fixes-2026-10-18/r1093-loop-check.txt`): the empty GPT partition → lsblk
`pttype "gpt"` (the bug's input), blkid rc 0 with only `PART_ENTRY_*` (the fix's condition holds); ext4 → `TYPE=ext4`; a
dos table inside the partition → `PTTYPE=dos`. Not walked: the whole `/api/storage/init` call (the agent refuses loop
devices; it needs a VM with a spare disk running a build of this commit).
## R-978 — the new escrow package reaches the hub only at the claim
A ceremony started again no longer replaces the hub's package at create time. The package travels with the pending
ceremony and is uploaded when the code is claimed, before the code is handed out; an upload failure is 502 and the code
stays claimable; a ceremony abandoned by TTL, a new start or a restart leaves the first code opening the hub's package.
Ceremony output is version 2 (`PendingUpload`); the sudo argv is unchanged. Tests `TestEscrowCeremony_R978_*` and three
`cmd` tests; four red-proofs. **Residual (filed):** the typed-back check is the browser's, so a household that opens the
new code and closes the page without writing it down has still replaced the package.
## R-973 — a cadence cap on `POST /backup`
Per guest and tier, successful runs only: the daily (local) tier at most **3 per rolling 24 h** (the night run, one
„Mentés most", the next night — `07` §6; since R-899 a press never costs the night); the longer tier at most one per
max(cadence − 24 h, cadence / 2) = **6 days** for the weekly off-site tier. A refusal is 429 with the next allowed time,
WARN-logged, kept in `backup-success-state.json` (survives a restart), and for 24 h the host report carries the CRITICAL
degraded status `localapi:backup-cadence-cap` (the hub's operator-only `agent_capability_degraded`). Tests `TestBackupCap_*`
and others; three red-proofs. **Limits (filed):** keep-last 3 lets three local points be replaced within one day; the
off-site namespace's keep-last 2 is a count — a time-bucketed prune on ep0 is the operator's.
*Decided by CC unattended — operator may reverse:* the numbers 3 / 24 h and 6 days, and counting successes only.
Gates: `go build/vet/test ./...` green; `agent_gates.py --fast` OK; decoys 36 cases OK. CI 2103, 2110 success.
+2 -2
View File
@@ -409,7 +409,7 @@ const backupCapStatusName = "localapi:backup-cadence-cap"
// backupCapStatus turns the local-API backup cadence cap's newest refusal (R-973) into a CRITICAL
// capability status: degraded for 24 h after a refusal, so the hub raises its operator-only
// agent_capability_degraded event (no hub change). The Reason names the guest, the tier and the time, and
// whole_guest_backup_refused event (hub v0.159.0, R-1150; agent_capability_degraded before). The Reason names the guest, the tier and the time, and
// says what to do. nil source (local API off) → ok.
func backupCapStatus(refusal func() (string, int, time.Time, bool)) capability.Status {
s := capability.Status{
@@ -856,7 +856,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
log: logger,
}
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
// existing agent_capability_degraded event (operator-only) tells the operator. Late-bound like
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
// drConfigured: the local API server is built further down, before the loop starts.
var backupCapRefusal func() (string, int, time.Time, bool)
probeAll := func(ctx context.Context) []capability.Status {
+1 -1
View File
@@ -2,9 +2,9 @@ package backup
import (
"context"
"fmt"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"testing"
+1 -1
View File
@@ -48,7 +48,7 @@ func TestR689_GuestBackupArchiveShapes(t *testing.T) {
{proxmox.StorageContent{VolID: "felhom-pbs:backup/vm/300/2026-07-28T05:31:14Z", VMID: 300}, true},
{proxmox.StorageContent{VolID: "local:backup/felhom-golden-0.236.0.tar.zst"}, false},
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", VMID: 9202}, false}, // vmid disagrees with the name
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
} {
if ok, why := guestBackupArchive(c.e); ok != c.ok {
t.Errorf("%s vmid=%d: ok=%v (%s), want %v", c.e.VolID, c.e.VMID, ok, why, c.ok)
+1 -1
View File
@@ -47,7 +47,7 @@ type Runner interface {
// downgrades a HEALTHY probe to StatusInactive (broken plumbing stays degraded regardless).
type Prober struct {
Runner Runner
Exists func(path string) bool // nil → os.Stat
Exists func(path string) bool // nil → os.Stat
GateActive func(gate string) bool // nil → every gate treated active
}
+2 -2
View File
@@ -21,10 +21,10 @@ import (
// The four inputs (the consumption contract — spike findings):
// - blob: the opaque R-wrapped escrow blob (hub-served restore directive in 10D).
// - recoveryCode (R): BY HAND from the customer — NEVER hub-sourced (zero-knowledge holds: a
// hub compromise alone cannot decrypt). A secret: never logged/persisted.
// hub compromise alone cannot decrypt). A secret: never logged/persisted.
// - expectedFingerprint: hub-served — the gate target (F-C4).
// - keyDest: where the PBS restore reads the key (`--keyfile` or the default
// `$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json`).
// `$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json`).
//
// Order (spike F-C2/F-C3/F-C4/F-C6): Unwrap → fingerprint-gate → install. On ANY failure there is
// NO partial install (nothing at keyDest) and no key material leaks: the recovered key lives only
+2 -2
View File
@@ -50,8 +50,8 @@ func TestParseConfMounts(t *testing.T) {
func TestMissingBindSources(t *testing.T) {
mounts := ParseConfMounts(sampleConf)
present := map[string]bool{
"/mnt/felhom-usb/felhom-data": true, // usb attached
"/var/lib/felhom-agent/guests/9201/bootstrap": true, // bootstrap always present
"/mnt/felhom-usb/felhom-data": true, // usb attached
"/var/lib/felhom-agent/guests/9201/bootstrap": true, // bootstrap always present
// "/mnt/felhom-flash/felhom-data" is ABSENT (drive unplugged)
}
got := MissingBindSources(mounts, func(p string) bool { return present[p] })
+10 -3
View File
@@ -14,6 +14,13 @@ const (
TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED
TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which)
TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting)
// TunnelAbsent (R-1070, agent Unreleased 2026-10-19): the guest answered and has NO cloudflared container — the
// controller makes none when no tunnel token is configured (a LAN-only or test box). Not a failure by itself: the
// hub never alarms on it, and the OS-update host rule reads it as „not applicable" only when the container was
// also absent before the update (HostHealthVerdict). Docker 29 prints „error: no such object" in lower case, so
// before this state existed the case-sensitive match fell through to `unknown` — which failed the host rule on
// every box without a tunnel (Tester4, 2026-10-15).
TunnelAbsent = "absent"
)
// CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec.
@@ -53,7 +60,7 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
return TunnelUnknown, "no running customer guest"
}
worst, wdetail := "", ""
rank := map[string]int{TunnelRunning: 0, TunnelUnknown: 1, TunnelNotRunning: 2}
rank := map[string]int{TunnelRunning: 0, TunnelAbsent: 1, TunnelUnknown: 2, TunnelNotRunning: 3}
for _, v := range vmids {
out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared")
st, d := ClassifyTunnel(string(out), string(errOut), err)
@@ -71,8 +78,8 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
func ClassifyTunnel(stdout, stderr string, err error) (string, string) {
out := strings.TrimSpace(stdout)
if err != nil || out == "" {
if strings.Contains(stderr, "No such object") || strings.Contains(stderr, "No such container") {
return TunnelNotRunning, "no cloudflared container in the guest"
if e := strings.ToLower(stderr); strings.Contains(e, "no such object") || strings.Contains(e, "no such container") {
return TunnelAbsent, "no cloudflared container in the guest (no tunnel set up on this box)"
}
return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err)
}
+4 -1
View File
@@ -20,7 +20,10 @@ func TestClassifyTunnel(t *testing.T) {
{"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"},
{"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"},
{"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelNotRunning, "no cloudflared container"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
// R-1070: Docker 29's wording, measured on demo-hp 2026-10-19 (`docker inspect` of a missing name).
{"absent, docker 29", "", "error: no such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no tunnel set up"},
{"absent, no such container", "", "Error response from daemon: No such container: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
{"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"},
{"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"},
{"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"},
+4 -4
View File
@@ -77,10 +77,10 @@ func TestLoop_FailedTailPushIsReArmedByNextEnvelope(t *testing.T) {
env *ControlEnvelope
err error
}{
{env: &ControlEnvelope{LogTailRequested: true}}, // arm
{err: errors.New("hub 5xx")}, // the carrying push FAILS
{env: &ControlEnvelope{LogTailRequested: true}}, // hub still pending → re-arm
{env: &ControlEnvelope{}}, // fulfilled
{env: &ControlEnvelope{LogTailRequested: true}}, // arm
{err: errors.New("hub 5xx")}, // the carrying push FAILS
{env: &ControlEnvelope{LogTailRequested: true}}, // hub still pending → re-arm
{env: &ControlEnvelope{}}, // fulfilled
}}
l := tailLoop(rep)
ctx := context.Background()
+1 -1
View File
@@ -359,7 +359,7 @@ type GuestSpec struct {
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
type Cloudflared struct {
Status string `json:"status"` // running | not_running | unknown (TunnelRunning …)
Status string `json:"status"` // running | not_running | unknown | absent (TunnelRunning …; absent since R-1070)
Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected"
}
+2 -2
View File
@@ -31,8 +31,8 @@ import (
// Only SUCCESSES count — a failed run prunes nothing and must stay retryable. The count survives an agent
// restart: it reads the success history saved on disk (LastKnownBackupStore, the R-894 file) as well as
// this process's own record. A refusal answers 429, is logged at WARN, is saved, and raises the operator
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's existing
// `agent_capability_degraded` event, operator-only). Pinned by TestBackupCap_* (backup_cap_test.go).
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's operator-only
// `whole_guest_backup_refused` event since hub v0.159.0, R-1150). Pinned by TestBackupCap_* (backup_cap_test.go).
// backupCapRefusalHold is how long one refusal keeps the operator signal raised.
const backupCapRefusalHold = 24 * time.Hour
+1 -1
View File
@@ -369,7 +369,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
job.Uploaded = false // R-978: uploaded by the claim
s.finishEscrowCeremony(job, out.RecoveryCode, out.PendingUpload)
out.RecoveryCode = "" // drop the parsed reference promptly (GC caveat: best-effort)
s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim",
s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim, or at the typed-back confirm (R-1151)",
"job_id", job.JobID, "restic_pw_sealed", job.ResticPwSealed, "uploaded", job.Uploaded,
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
}()
+18
View File
@@ -1343,6 +1343,20 @@ type BackupStatusResponse struct {
// presence. Served on the UNTARGETED request only; additive, so an older controller reads the
// response exactly as before.
Tiers []TierBackupState `json:"tiers,omitempty"`
// HostBusy (R-921, agent Unreleased 2026-10-19) says whether the host-wide heavy-operation gate is held RIGHT NOW
// (another guest's backup, the restore test, a night OS step, the stale-lock sweep, a power-off) — a POST /backup
// would then be refused with 409. The controller reads it BEFORE it stops the household's apps, so a refused tier
// costs no stop. Served on every request; ABSENT (nil) = an older agent, or the gate is not wired — the controller
// then keeps its old order (stop, ask, resume at once on 409). A read, never a reservation: the gate can be taken
// between this answer and the POST, and that race keeps the old order. Pinned by TestBackupStatus_R921_HostBusy.
HostBusy *HostBusyState `json:"host_busy,omitempty"`
}
// HostBusyState is GET /backup/status's host_busy (R-921). What names the holder ("backup:local", "restore-test",
// "os-pve-step" …), empty when Busy is false.
type HostBusyState struct {
Busy bool `json:"busy"`
What string `json:"what,omitempty"`
}
func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) {
@@ -1362,6 +1376,10 @@ func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid
resp.JobID = job.JobID
resp.Error = job.Error
}
if s.inFlight != nil {
what := s.inFlight.Busy()
resp.HostBusy = &HostBusyState{Busy: what != "", What: what}
}
writeOK(w, resp)
}
+45
View File
@@ -13,6 +13,7 @@ import (
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
@@ -514,3 +515,47 @@ func TestBackupStatus_FiltersToThisGuest(t *testing.T) {
t.Fatalf("status not scoped to this guest / not latest: %s", w.Body.String())
}
}
// R-921: GET /backup/status says when the host-wide heavy-operation gate is held, so the controller can skip the app
// stop for a tier the POST would refuse. Free → busy false; held → busy true with the holder's name; the gate not wired
// → no field at all (the controller's „older agent" reading, never „free").
//
// COMPANION RED-PROOF (observed 2026-10-19): with the `resp.HostBusy = …` assignment removed, the held case fails with
//
// server_test.go: free gate: host_busy = <nil>, want busy=false
//
// Restored.
func TestBackupStatus_R921_HostBusy(t *testing.T) {
read := func(h http.Handler) *HostBusyState {
w := do(t, h, "GET", "/backup/status", "A", "")
var resp struct {
Data BackupStatusResponse `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("status body: %v (%s)", err, w.Body.String())
}
return resp.Data.HostBusy
}
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
if got := read(srv.Handler()); got != nil {
t.Fatalf("gate not wired: host_busy = %+v, want absent", got)
}
gate := &backup.InFlight{}
srv.inFlight = gate
h := srv.Handler()
if got := read(h); got == nil || got.Busy || got.What != "" {
t.Fatalf("free gate: host_busy = %+v, want busy=false", got)
}
release, _, ok := gate.TryAcquire("os-pve-step")
if !ok {
t.Fatal("could not take the gate")
}
if got := read(h); got == nil || !got.Busy || got.What != "os-pve-step" {
t.Fatalf("held gate: host_busy = %+v, want busy=true what=\"os-pve-step\"", got)
}
// The read never takes the gate: a POST after the holder lets go is accepted.
release()
if got := read(h); got == nil || got.Busy {
t.Fatalf("released gate: host_busy = %+v, want busy=false", got)
}
}
+46 -1
View File
@@ -2,6 +2,7 @@ package localapi
import (
"context"
"errors"
"fmt"
"log/slog"
"os"
@@ -160,6 +161,7 @@ type staleLockAPI interface {
ListSnapshots(ctx context.Context, vmid int) ([]proxmox.Snapshot, error)
ListRunningTasks(ctx context.Context) ([]proxmox.TaskStatus, error)
DeleteSnapshot(ctx context.Context, vmid int, snapname string) (string, error)
DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error)
Start(ctx context.Context, vmid int) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
}
@@ -334,8 +336,34 @@ func (c *staleLockController) Unlock(ctx context.Context, vmid int) error {
return nil
}
// DeleteVzdumpSnapshot removes the dangling `vzdump` snapshot. R-1010: when the delete fails ONLY because one disk's
// snapshot volume is already gone („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump" — a backup that ran
// out of space, 2026-10-14), Proxmox stops at that volume and keeps the entry and the other volumes' snapshots, and a
// plain retry fails the same way every sweep, forever. Then it retries ONCE with Proxmox's force (the hand fix that
// day was `pct delsnapshot 9201 vzdump --force`): the remaining volumes' snapshots are removed and the entry goes. Any
// other error is returned as it is — never a force on an error that might mean a volume still holds data (a full
// LVM metadata area, a busy volume). Pinned by TestStaleLock_R1010_*.
func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int) error {
upid, err := c.px.DeleteSnapshot(ctx, vmid, vzdumpSnapshotName)
err := c.deleteSnap(ctx, vmid, false)
if err == nil || !snapshotVolumeAlreadyGone(err) {
return err
}
if c.logger != nil {
c.logger.Warn("stale-lock: a disk's vzdump snapshot volume is already gone — retrying the delete once with force (R-1010)",
"vmid", vmid, "err", err)
}
if ferr := c.deleteSnap(ctx, vmid, true); ferr != nil {
return fmt.Errorf("forced snapshot delete after %q also failed: %w", err.Error(), ferr)
}
return nil
}
func (c *staleLockController) deleteSnap(ctx context.Context, vmid int, force bool) error {
del := c.px.DeleteSnapshot
if force {
del = c.px.DeleteSnapshotForce
}
upid, err := del(ctx, vmid, vzdumpSnapshotName)
if err != nil {
return err
}
@@ -346,6 +374,23 @@ func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int
return err
}
// snapshotVolumeAlreadyGone: the error is Proxmox's „Failed to find logical volume …_vzdump" from a snapshot delete
// (in the task's exit status, its log tail, or the API error text). Narrow on purpose: the force is safe only when the
// volume it cannot remove does not exist.
func snapshotVolumeAlreadyGone(err error) bool {
text := err.Error()
var te *proxmox.TaskError
if errors.As(err, &te) {
text += "\n" + te.ExitStatus + "\n" + strings.Join(te.LogTail, "\n")
}
for _, l := range strings.Split(text, "\n") {
if strings.Contains(strings.ToLower(l), "failed to find logical volume") && strings.Contains(l, "_"+vzdumpSnapshotName) {
return true
}
}
return false
}
func (c *staleLockController) Start(ctx context.Context, vmid int) error {
upid, err := c.px.Start(ctx, vmid)
if err != nil {
+19 -1
View File
@@ -30,6 +30,9 @@ type fakeStaleLockAPI struct {
delsnap []int
started []int
// R-1010: scripted delete answers, consumed in order (plain and forced share the list); calls records "plain"/"force".
delErrs []error
delCalls []string
}
func (f *fakeStaleLockAPI) ListLXC(context.Context) ([]proxmox.Guest, error) { return f.lxc, nil }
@@ -55,7 +58,22 @@ func (f *fakeStaleLockAPI) ListRunningTasks(context.Context) ([]proxmox.TaskStat
func (f *fakeStaleLockAPI) DeleteSnapshot(_ context.Context, vmid int, snapname string) (string, error) {
f.delsnap = append(f.delsnap, vmid)
return "", nil // synchronous — no WaitTask
return "", f.nextDelErr("plain") // synchronous — no WaitTask
}
func (f *fakeStaleLockAPI) DeleteSnapshotForce(_ context.Context, vmid int, snapname string) (string, error) {
f.delsnap = append(f.delsnap, vmid)
return "", f.nextDelErr("force")
}
func (f *fakeStaleLockAPI) nextDelErr(kind string) error {
f.delCalls = append(f.delCalls, kind)
if len(f.delErrs) == 0 {
return nil
}
e := f.delErrs[0]
f.delErrs = f.delErrs[1:]
return e
}
func (f *fakeStaleLockAPI) Start(_ context.Context, vmid int) (string, error) {
+59
View File
@@ -0,0 +1,59 @@
package localapi
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1010: a half-removed vzdump snapshot. The error text is the one demo-hp's task printed on 2026-10-14
// (`audits/power-cut-fixes-2026-10-14/C4/hand-fix.txt`).
const r1010MissingLV = `lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to find logical volume "pve/snap_vm-9201-disk-0_vzdump"`
func r1010TaskErr(exit string) error {
return &proxmox.TaskError{UPID: "UPID:demo-hp:0001:0002:0003:vzdelsnapshot:9201:root@pam:", ExitStatus: exit,
LogTail: []string{exit}}
}
// COMPANION RED-PROOF (observed 2026-10-19): with DeleteVzdumpSnapshot back to the single plain delete, the first case
// fails with
//
// stalelock_r1010_test.go: missing volume: calls [plain], want [plain force]; err = proxmox: task … failed: exitstatus "lvremove snapshot …"
//
// Restored.
func TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone(t *testing.T) {
cases := []struct {
name string
errs []error
wantCalls []string
wantErr string
}{
{"missing volume", []error{r1010TaskErr(r1010MissingLV)}, []string{"plain", "force"}, ""},
{"missing volume, named only in the log tail", []error{&proxmox.TaskError{UPID: "x", ExitStatus: "command 'lvremove' failed: exit code 5",
LogTail: []string{r1010MissingLV}}}, []string{"plain", "force"}, ""},
{"the plain delete works", nil, []string{"plain"}, ""},
{"another error: no force", []error{r1010TaskErr("lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to update pool pve/data.")},
[]string{"plain"}, "Failed to update pool"},
{"a missing volume of ANOTHER snapshot: no force", []error{r1010TaskErr(`Failed to find logical volume "pve/snap_vm-9201-disk-0_mine"`)},
[]string{"plain"}, "_mine"},
{"the forced delete fails too: returned, never a second force", []error{r1010TaskErr(r1010MissingLV), errors.New("still broken")},
[]string{"plain", "force"}, "still broken"},
}
for _, c := range cases {
api := &fakeStaleLockAPI{delErrs: c.errs}
ctrl := NewStaleLockController(api, &recordingRunner{}, "felhom", nil)
err := ctrl.DeleteVzdumpSnapshot(context.Background(), 9201)
if strings.Join(api.delCalls, " ") != strings.Join(c.wantCalls, " ") {
t.Errorf("%s: calls %v, want %v; err = %v", c.name, api.delCalls, c.wantCalls, err)
}
switch {
case c.wantErr == "" && err != nil:
t.Errorf("%s: err = %v, want nil", c.name, err)
case c.wantErr != "" && (err == nil || !strings.Contains(err.Error(), c.wantErr)):
t.Errorf("%s: err = %v, want one naming %q", c.name, err, c.wantErr)
}
}
}
+7 -7
View File
@@ -6,12 +6,12 @@
// the report stanza the hub surfaces:
//
// - /run/sshd present? — OpenSSH's SHARED privsep dir; its absence is the KEXINIT-reset
// lockout (SPIKE-felhom-sshd-2026-07-05 §8).
// lockout (SPIKE-felhom-sshd-2026-07-05 §8).
// - stock sshd listener answers? — a TCP connect to the sshd port (22 for G1; H1 passes the
// discovered felhom-sshd port later).
// discovered felhom-sshd port later).
// - did the watchdog auto-heal? — the watchdog writes an RFC3339 marker to /run when it heals;
// its presence (+ timestamp) tells the hub a clobber recurred, so
// the operator learns of a recurring cause BEFORE a lockout.
// its presence (+ timestamp) tells the hub a clobber recurred, so
// the operator learns of a recurring cause BEFORE a lockout.
//
// Read-only and dependency-light: os.Stat + os.ReadFile + a short net.Dial, no exec, no sudo. Safe to
// run every heartbeat.
@@ -44,9 +44,9 @@ const (
type Reporter struct {
privsepDir string
marker string
sshdAddr string // host:port dialed for the reachability probe
statDir func(string) bool // dir-exists check (os.Stat wrapper; test seam)
readMarker func(string) (string, bool) // marker read → (timestamp, present)
sshdAddr string // host:port dialed for the reachability probe
statDir func(string) bool // dir-exists check (os.Stat wrapper; test seam)
readMarker func(string) (string, bool) // marker read → (timestamp, present)
dialSSHD func(ctx context.Context, addr string) bool
}
+19 -2
View File
@@ -370,7 +370,8 @@ func HealthVerdict(before, after *Health) (bool, string) {
// HostHealthVerdict is THE host health rule (`11` §8.2; pinned by TestHostHealthVerdict): the Proxmox daemons and the
// agent are active, the customer guest still runs, the guest's own rule still passes against the start of the pass,
// and the tunnel is `running` (R-841).
// and the tunnel is `running` (R-841) — or `absent` both now and before the update: a box with no tunnel set up
// (R-1070; pinned by TestHostHealthVerdict_R1070_NoTunnelSetUp).
func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
if after == nil {
return false, "no health reading"
@@ -398,12 +399,28 @@ func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
if ok, why := HealthVerdict(gb, after.Guest); !ok {
return false, "guest: " + why
}
if tunnel != hub.TunnelRunning {
switch {
case tunnel == hub.TunnelRunning:
case tunnel == hub.TunnelAbsent && noTunnelBefore(before):
// R-1070: no cloudflared container before the update and none after — the box has no tunnel to check.
case tunnel == hub.TunnelAbsent:
return false, "the tunnel is absent (no cloudflared container now, and the reading from before the update does not show it was absent then)"
default:
return false, "the tunnel is " + tunnel
}
return true, ""
}
// noTunnelBefore: the reading from the start of the pass shows the guest's Docker answered and listed NO cloudflared
// container. A missing or failed reading is not „no tunnel" (an empty list from a failed `docker ps` must not pass).
func noTunnelBefore(before *Health) bool {
if before == nil || before.Guest == nil || !before.Guest.DockerOK {
return false
}
_, had := before.Guest.Containers["cloudflared"]
return !had
}
// EngineOf is the engine version `docker version` prints for a docker-ce package version: "5:29.8.2-1~debian.13~trixie"
// → "29.8.2" (no epoch, no Debian revision).
func EngineOf(pkgVersion string) string {
+39
View File
@@ -568,3 +568,42 @@ func TestDocker_OOMCheckReachesTheHubUnchanged(t *testing.T) {
t.Fatalf("no docker report posted: %s", calls(w))
}
}
// R-1070: a box with no tunnel set up (no cloudflared container before the update and none after) passes the host
// rule; a box WITH a tunnel that is down, gone, or unreadable still fails it.
//
// COMPANION RED-PROOF (observed 2026-10-19): with the „not applicable" branch switched off (only `running` passes, the
// pre-fix rule) the first case fails with
//
// leg_test.go: no tunnel before, none after: got false (the tunnel is absent (no cloudflared container now, …)), want true
//
// and with the pre-fix case-sensitive match TestClassifyTunnel's Docker 29 row reads `unknown` („could not ask the
// guest"), the state Tester4 failed on.
// Restored.
func TestHostHealthVerdict_R1070_NoTunnelSetUp(t *testing.T) {
withTunnel := hostOK()
withTunnel.Guest.Containers["cloudflared"] = Container{State: "running", Health: "healthy"}
dockerFailed := hostOK()
dockerFailed.Guest = &Health{DockerOK: false, Containers: map[string]Container{}}
cases := []struct {
name string
before *Health
tunnel string
want bool
why string
}{
{"no tunnel before, none after", hostOK(), hub.TunnelAbsent, true, ""},
{"a tunnel before, gone after", withTunnel, hub.TunnelAbsent, false, "cloudflared was running"},
{"a tunnel before, down after", withTunnel, hub.TunnelNotRunning, false, "cloudflared was running"},
{"no tunnel before, a stopped one after", hostOK(), hub.TunnelNotRunning, false, "not_running"},
{"no tunnel before, the probe could not ask after", hostOK(), hub.TunnelUnknown, false, "unknown"},
{"no reading from before", nil, hub.TunnelAbsent, false, "absent"},
{"docker failed before (an empty list is not proof of none)", dockerFailed, hub.TunnelAbsent, false, "absent"},
}
for _, c := range cases {
got, why := HostHealthVerdict(c.before, hostOK(), c.tunnel)
if got != c.want || !strings.Contains(why, c.why) {
t.Errorf("%s: got %v (%s), want %v (…%s…)", c.name, got, why, c.want, c.why)
}
}
}
+12
View File
@@ -180,6 +180,18 @@ func (c *Client) DeleteSnapshot(ctx context.Context, vmid int, snapname string)
return c.dataString(ctx, http.MethodDelete, path, nil)
}
// DeleteSnapshotForce is DeleteSnapshot with Proxmox's own `force=1` („remove from the config file, even if removing
// disk snapshots fails"): the API's flag, so the same token privilege (VM.Snapshot) and no new root command. Used by
// the stale-lock janitor ONLY after a delete failed because one disk's snapshot volume is already gone (R-1010). The
// flag rides the query string, where pveproxy reads a DELETE's parameters. Pinned by TestDeleteSnapshotForce_R1010.
func (c *Client) DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error) {
if vmid == 0 || snapname == "" {
return "", fmt.Errorf("proxmox: DeleteSnapshotForce needs vmid and snapname")
}
path := fmt.Sprintf("/nodes/%s/lxc/%d/snapshot/%s?force=1", c.node, vmid, url.PathEscape(snapname))
return c.dataString(ctx, http.MethodDelete, path, nil)
}
// SetConfig applies config changes via PUT /nodes/{node}/lxc/{vmid}/config
// (e.g. memory, cores, net0, mpN with a backup flag). PVE may apply this
// synchronously (no UPID) — the returned string is empty in that case, and "" is
+22
View File
@@ -3,6 +3,7 @@ package proxmox
import (
"context"
"io"
"net/http"
"strings"
"testing"
)
@@ -99,3 +100,24 @@ func TestDestroyScratchLXC_RefusesUnconfiguredBand(t *testing.T) {
}
}
}
// R-1010: the forced snapshot delete is the API's own flag on the same DELETE (no new root command).
func TestDeleteSnapshotForce_R1010(t *testing.T) {
var got *http.Request
d := &mockDoer{fn: func(r *http.Request) (*http.Response, error) {
got = r
return jsonResp(200, `{"data":"UPID:demo-felhom:0001:0002:0003:vzdelsnapshot:9201:u@pve!t:"}`), nil
}}
upid, err := newTestClient(d).DeleteSnapshotForce(context.Background(), 9201, "vzdump")
if err != nil || upid == "" {
t.Fatalf("DeleteSnapshotForce: upid %q err %v", upid, err)
}
if got.Method != http.MethodDelete || got.URL.Path != "/api2/json/nodes/demo-felhom/lxc/9201/snapshot/vzdump" ||
got.URL.Query().Get("force") != "1" {
t.Fatalf("request = %s %s, want DELETE …/lxc/9201/snapshot/vzdump?force=1", got.Method, got.URL.String())
}
d.fn = func(r *http.Request) (*http.Response, error) { got = r; return jsonResp(200, `{"data":null}`), nil }
if _, err := newTestClient(d).DeleteSnapshot(context.Background(), 9201, "vzdump"); err != nil || got.URL.RawQuery != "" {
t.Fatalf("the plain delete must carry no force: %s (err %v)", got.URL.String(), err)
}
}
+1 -1
View File
@@ -59,7 +59,7 @@ const DefaultPool = "felhom"
//
// SysDataGrowGB itself is NOT removed — see its field comment: the host installer still passes
// `-sysdata-grow`, and its GiB are FOLDED INTO the single volume's grow rather than dropped.
const DefaultSysDataMount = ""
const DefaultSysDataMount = ""
// Structural host-bind mountpoints every provisioned guest carries (GL-5; verdict of
// SPIKE-dr-bindmount-source-2026-07-07): the permanent drives parent bind (mp8,
+2 -2
View File
@@ -25,8 +25,8 @@ import (
// operator's signing CLI (cmd/felhom-opsign) produces it; this runner consumes it. The hub never
// forges one (it holds no signing key) and never opens it.
type Envelope struct {
OpBlobB64 string `json:"op_blob_b64"` // base64 of the canonical OpBlob JSON
SigArmored string `json:"sig_armored"` // armored SSHSIG over the op-blob (namespace felhom-op-v1)
OpBlobB64 string `json:"op_blob_b64"` // base64 of the canonical OpBlob JSON
SigArmored string `json:"sig_armored"` // armored SSHSIG over the op-blob (namespace felhom-op-v1)
}
// JobSource fetches + clears jobs. Satisfied by *hub.Client.
+3 -3
View File
@@ -14,10 +14,10 @@ func TestClassifyClaim(t *testing.T) {
nodes: []claimNode{{name: "sdd", fstype: "", mountpoint: ""}, {name: "sdd1", fstype: "ntfs", mountpoint: ""}}}
}
cases := []struct {
name string
mutate func(f *claimFacts)
name string
mutate func(f *claimFacts)
wantUnclaimed bool
reasonHas string
reasonHas string
}{
{"clean unclaimed disk (ntfs, unmounted)", func(f *claimFacts) {}, true, "unclaimed"},
{"blank disk", func(f *claimFacts) { f.nodes = []claimNode{{name: "sdd"}} }, true, "unclaimed"},
+3 -3
View File
@@ -103,12 +103,12 @@ type MountSpec struct {
// Binaries holds the absolute paths of the allow-listed binaries (overridable from config
// so the sudoers entries and the agent agree on exact paths).
type Binaries struct {
Systemctl string
Install string
Systemctl string
Install string
// PrivApply is the root content checker (R-861, agent v0.146.0): a unit file reaches /etc/systemd/system only
// through `felhom-priv-apply unit <name>`, which reads the staged copy from /var/lib/felhom-agent/units/ and
// refuses anything the renderers here never produce (a bind over /etc, a Where outside /mnt, …).
PrivApply string
PrivApply string
Smartctl string
Lvs string
Blkid string // device signature probe (8C data-bearing detection)
+4 -4
View File
@@ -17,12 +17,12 @@ import (
// Four states (3A):
// - new : durable-id NOT in the store. Never auto-mounted (the user must enroll it).
// - enrolled : desired = mounted + bound into the guest. Drift (present && unmounted) with
// this intent → reconcile.
// this intent → reconcile.
// - ejected : an intentional, temporary unmount, set ONLY via the official eject endpoint.
// Present && unmounted → left alone. CLEARED to `enrolled` when the drive goes
// physically ABSENT, so a replug auto-mounts (the replug rule, for free).
// Present && unmounted → left alone. CLEARED to `enrolled` when the drive goes
// physically ABSENT, so a replug auto-mounts (the replug rule, for free).
// - decommissioned : permanent. Never auto-mounted; SURVIVES absent/present; cleared only by an
// explicit re-commission.
// explicit re-commission.
//
// The elegant invariant: intent is recorded ONLY through the official enroll/eject/decommission
// paths. "Is it unmounted" ≠ "did someone officially eject it."
+5 -5
View File
@@ -93,11 +93,11 @@ systemd-1 /mnt/felhom-drives/live autofs rw,relatime,fd=86 0 0
where string
want bool
}{
{"/mnt/felhom-drives/live", true}, // real nfs4 (the autofs line for the same path must not confuse it)
{"/mnt/felhom-drives/smb", true}, // cifs
{"/mnt/felhom-drives/idle", false}, // autofs trigger ONLY — idle automount is NOT mounted
{"/mnt/felhom-drives/disk", false}, // a local fs at the path is not a network mount
{"/mnt/felhom-drives/nope", false}, // absent
{"/mnt/felhom-drives/live", true}, // real nfs4 (the autofs line for the same path must not confuse it)
{"/mnt/felhom-drives/smb", true}, // cifs
{"/mnt/felhom-drives/idle", false}, // autofs trigger ONLY — idle automount is NOT mounted
{"/mnt/felhom-drives/disk", false}, // a local fs at the path is not a network mount
{"/mnt/felhom-drives/nope", false}, // absent
}
for _, tc := range cases {
if got := networkMountedIn(procMounts, tc.where); got != tc.want {
+2 -2
View File
@@ -96,11 +96,11 @@ type Watchdog struct {
now func() time.Time
spawn func(func()) // spawn a background task (overridable in tests; default `go f()`)
intent IntentReader // P3: gate re-mount by drive intent (nil = ungated legacy)
intent IntentReader // P3: gate re-mount by drive intent (nil = ungated legacy)
onAbsent func(durableID string) // P3: called when a known target goes ABSENT (clears `ejected`)
mu sync.Mutex
last map[string]bool // name -> last observed present (only for seen targets)
last map[string]bool // name -> last observed present (only for seen targets)
lastFire time.Time
fired bool // lastFire is valid
pending bool // a transition is awaiting the debounce window
+13
View File
@@ -15,6 +15,9 @@ Gates (all must pass; **non-zero exit on any failure**):
or because someone is offline on a train. CI runs the FULL set for exactly this
reason: it is the machine that can afford a network check, and it is the half
that emails when something is wrong.
… security-scan govulncheck (called vulnerabilities) + gitleaks on the pushed commits, WARN-FIRST: it lists,
it never refuses (R-986). Shared: felhom.eu/scripts/security_scan_gate.py. In the fast set
although it reads vuln.go.dev — offline it says NOT CHECKED, so it cannot stop a push.
… gate-decoys the decoy suite (test_gate_decoys.py): every gate here still convicts its decoys
(R-1116). ~20 s in the workspace, so the full run only: not the hook, but CI
runs the full set on every push. Scratch clones and a local fake Gitea only.
@@ -49,6 +52,9 @@ SHARED_INSTRUCTIONS = os.path.join(
# R-389 — shared, like the two above: it lives in felhom.eu/scripts/ and is never copied.
SHARED_OBSERVATIONS = os.path.join(
os.path.dirname(ROOT), "felhom.eu", "scripts", "observations_gate.py")
# R-986 — shared too: the dependency and secret scanner, warn-first.
SHARED_SECURITY = os.path.join(
os.path.dirname(ROOT), "felhom.eu", "scripts", "security_scan_gate.py")
# (label, absolute script path, args, fast)
GATES = [
@@ -67,6 +73,13 @@ GATES = [
# runs the FULL set, so every push is still judged by it. It plants only in scratch clones and asks a fake
# Gitea on 127.0.0.1 (never the network, never the real tree); it does not call this runner (no recursion).
("gate-decoys", os.path.join(ROOT, "scripts", "test_gate_decoys.py"), [], False),
# R-1160 (2026-10-19) — every Go file is gofmt-clean; the controller's gofmt_gate.py shape. Fast: one gofmt -l.
# Where the runner has no Go it says NOT CHECKED out loud and passes; the pre-push hook (with Go) enforces it.
("gofmt", os.path.join(ROOT, "scripts", "gofmt_gate.py"), [], True),
# R-986 (2026-10-19) — govulncheck + gitleaks, WARN-FIRST (exit 0 whatever it finds). Fast although it reads
# vuln.go.dev: it cannot refuse, so a network blink is a NOT CHECKED line. ~4 s here. CI: gitleaks fetched pinned
# (FELHOM_SCAN_FETCH=1), no Go, so govulncheck says NOT CHECKED there.
("security-scan", SHARED_SECURITY, [ROOT], True),
]
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""gofmt_gate.py — every Go file in this module is gofmt-clean (R-1160; the controller's R-454 gate, same shape).
MEASURED 2026-10-19: 14 agent files were not gofmt-clean and nothing noticed (fixed by one formatting-only commit,
1edfd8e). The controller's history below is why a gate and not a one-off clean-up.
MEASURED 2026-09-02: five internal/web test files had been gofmt-unclean for an unknown length of
time and nothing noticed — `go vet` does not check formatting and no gate did. By 2026-10-05 the five
were clean and TWELVE others were not: the count only grows while nothing looks, and every `gofmt -l`
someone runs by hand is then noise that hides a real one.
Run from the repo root: python3 scripts/gofmt_gate.py
Exit 0 clean · 1 convicted (lists the files) · 2 inconclusive (gofmt not found — never a pass).
gofmt is the Go toolchain's own formatter (not a shell utility), found on PATH or under
`go env GOROOT`/bin. Decoy: scripts/test_gate_decoys.py ("gofmt"), in a scratch copy of this script.
"""
import os
import shutil
import subprocess
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ROOTS = ["cmd", "internal"]
def find_gofmt():
p = shutil.which("gofmt")
if p:
return p
go = shutil.which("go")
if go:
try:
root = subprocess.check_output([go, "env", "GOROOT"], text=True).strip()
cand = os.path.join(root, "bin", "gofmt")
if os.path.isfile(cand):
return cand
except Exception:
pass
return None
def in_ci():
"""True on the Gitea Actions runner (act sets GITHUB_ACTIONS, Gitea sets GITEA_ACTIONS)."""
return os.environ.get("GITEA_ACTIONS") == "true" or os.environ.get("GITHUB_ACTIONS") == "true"
def main():
gofmt = find_gofmt()
if not gofmt:
# The CI runner may have no Go toolchain (felhom.eu CLAUDE.md, R-887 note). There the gate
# says so OUT LOUD and passes: the pre-push hook on the developer's machine, where Go always exists, is where
# formatting is enforced. Anywhere else a missing gofmt stays INCONCLUSIVE — never a silent pass.
# Pinned by the "gofmt: CI without Go" and "gofmt: dev without Go" decoys in test_gate_decoys.py.
if in_ci():
print("gofmt gate NOT CHECKED in CI: no Go toolchain on the runner — the pre-push hook (with Go) enforces it")
return 0
print("gofmt gate INCONCLUSIVE: gofmt not found on PATH or under `go env GOROOT`/bin")
return 2
roots = [r for r in ROOTS if os.path.isdir(os.path.join(ROOT, r))]
if not roots:
print("gofmt gate INCONCLUSIVE: none of %s exists under %s" % (ROOTS, ROOT))
return 2
p = subprocess.run([gofmt, "-l"] + roots, cwd=ROOT, capture_output=True, text=True)
if p.returncode != 0:
print("gofmt gate INCONCLUSIVE: gofmt exited %d: %s" % (p.returncode, p.stderr.strip()[:400]))
return 2
bad = [l for l in p.stdout.splitlines() if l.strip()]
if bad:
for f in bad:
print(" not gofmt-clean: %s" % f)
print("GOFMT GATE FAILED: %d file(s) — run `gofmt -w <file>` (formatting only, no behaviour change)" % len(bad))
return 1
print("gofmt gate OK — every Go file under %s is gofmt-clean" % ", ".join(roots))
return 0
if __name__ == "__main__":
sys.exit(main())
+50
View File
@@ -66,6 +66,14 @@ COVERS = {
"inside an HTML comment (R-426)",
"observations": "R-419 in THIS repo's REPORT.md: an Observations note SAYING it carries no marker - vs the "
"two genuine markers (R-426)",
"gofmt": "R-1160: an unformatted .go file planted under internal/ of a SCRATCH copy is convicted by name - vs the "
"same tree formatted (passes); with no gofmt reachable, CI passes OUT LOUD and a dev machine is "
"INCONCLUSIVE, never a silent pass",
"security-scan": "R-986: the shared gate's own decoys (felhom.eu scripts/test_security_scan_gate.py, 8 cases in "
"SCRATCH git repos: a planted token listed by rule and file, its value never printed; a second "
"secret on another line of an accepted file still listed; a called govulncheck finding listed, "
"an imported-only one not; no tool is NOT CHECKED out loud), run from here; plus the gate on THIS "
"repo's tree: exit 0 and its summary line (warn-first - the listing is the fact)",
"gate-decoys": "this suite, run as a gate of agent_gates.py's full run, which CI runs on every push (R-1116): "
"every case above is a decoy of its own gate, and ONE case that stops convicting turns the run "
"red - red-proved 2026-10-17 by flipping a reuse-refs FACT case to expect rc 0",
@@ -291,6 +299,32 @@ def release_cases(base, ws):
# ── the shared felhom.eu scripts, against THIS repo's inputs ─────────────────────────────────────
def gofmt_cases(ws):
"""R-1160: the gofmt gate in a scratch root (its ROOT is its own grandparent directory), never the real tree."""
root = os.path.join(ws, "gofmt-root")
os.makedirs(os.path.join(root, "scripts"))
os.makedirs(os.path.join(root, "internal", "x"))
shutil.copy(os.path.join(ROOT, "scripts", "gofmt_gate.py"), os.path.join(root, "scripts", "gofmt_gate.py"))
gate = os.path.join(root, "scripts", "gofmt_gate.py")
planted = os.path.join(root, "internal", "x", "zz_gofmt_decoy.go")
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F( ) int {return 1}\n")
if shutil.which("gofmt") or shutil.which("go"):
rc, out = run([sys.executable, gate], root)
report("gofmt: FACT: an unformatted file under internal/", rc, out, 1, ("zz_gofmt_decoy.go", "GOFMT GATE FAILED"))
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F() int { return 1 }\n")
rc, out = run([sys.executable, gate], root)
report("gofmt: GENUINE: the same file formatted", rc, out, 0, ("gofmt gate OK",))
else:
print(" -- gofmt: not runnable here (no Go toolchain) - the two cases below pin that mode")
for ci, want_rc, want, name in ((True, 0, "NOT CHECKED in CI", "gofmt: CI without Go"),
(False, 2, "INCONCLUSIVE", "gofmt: dev without Go")):
env = {"PATH": os.path.join(ws, "empty-path"), "HOME": ws}
if ci:
env["GITEA_ACTIONS"] = "true"
rc, out = run([sys.executable, gate], root, env=env)
report(name, rc, out, want_rc, (want,))
def shared_cases(ws):
"""A scratch WORKSPACE: a clone of this repo beside symlinks to the siblings, because the shared
scripts reach across (REUSE.md cites hub paths; instructions_gate reads the workspace CLAUDE.md)."""
@@ -343,6 +377,20 @@ def shared_cases(ws):
u"the same minute.**\n", 0)
def security_cases():
"""R-986: the shared scanner's decoys (they build their own scratch repos), then the gate on this repo."""
suite = os.path.join(SHARED, "test_security_scan_gate.py")
gate = os.path.join(SHARED, "security_scan_gate.py")
for f in (suite, gate):
if not os.path.isfile(f):
fails.append("shared %s is MISSING beside this clone - a failure, never a skip" % f)
return
rc, out = run([sys.executable, suite], ROOT)
report("security-scan: the shared gate's 8 decoys", rc, out, 0, ("all 8 cases passed",))
rc, out = run([sys.executable, gate, ROOT], ROOT)
report("security-scan: GENUINE: this repo, warn-first", rc, out, 0, ("security-scan:", "govulncheck"))
def main():
srv = Server(("127.0.0.1", 0), Handler)
threading.Thread(target=srv.serve_forever, daemon=True).start()
@@ -353,6 +401,8 @@ def main():
published_cases(base)
release_cases(base, ws)
shared_cases(ws)
gofmt_cases(ws)
security_cases()
finally:
srv.shutdown()
srv.server_close()