Compare commits

...

12 Commits

Author SHA1 Message Date
admin b310773dbf REPORT: the night of 2026-10-21 (host swap report, R-1194), unreleased
gates / gates (push) Successful in 2m7s
2026-10-22 00:06:21 +02:00
admin 0da792e670 R-1192 follow-up: report host swap vs the guests' swap allowance (log once, hub alarm, local API); R-1194: no OOB stanza before the first felhom-sshd pass; unreleased
gates / gates (push) Successful in 1m56s
2026-10-21 21:29:07 +02:00
admin 286a7e3ecf R-1192: every guest may swap as much as its RAM (bring-up, RAM resize, once at start) — the measured 8 GB freeze; unreleased
gates / gates (push) Successful in 1m39s
2026-10-21 15:43:18 +02:00
admin 4e1621c650 CHANGELOG + CONTEXT + REPORT: v0.161.0 released (operator yes) and delivered to demo-hp, Tester 1, demo-felhom; not vouched; R-1083 left out
gates / gates (push) Successful in 1m27s
2026-10-20 09:07:30 +02:00
admin 51521230c9 REPORT: the night's agent fixes (R-921, R-1070, R-1010), not released
gates / gates (push) Successful in 1m25s
2026-10-19 21:47:40 +02:00
admin 84ef5818d5 security-scan gate (shared felhom.eu/scripts/security_scan_gate.py) in the fast set and CI, warn-first, with the accepted-hits file and decoys (R-986)
gates / gates (push) Successful in 1m30s
2026-10-19 21:11:46 +02:00
admin 56aabc729c Unreleased: host_busy on GET /backup/status (R-921), a missing tunnel container is 'absent' and not-applicable after an OS update (R-1070), the janitor forces a half-removed vzdump snapshot once (R-1010)
gates / gates (push) Successful in 58s
2026-10-19 19:52:37 +02:00
admin 0cbd500154 gofmt gate in agent_gates.py with its decoys (R-1160); REPORT for the day
gates / gates (push) Successful in 49s
2026-10-19 10:26:27 +02:00
admin 1edfd8e331 gofmt -w: 14 files, formatting only, no behaviour change (R-1160) 2026-10-19 10:24:32 +02:00
admin 0bc0bc5da0 CONTEXT + REPORT: v0.160.0 released, on the three test hosts, not vouched
gates / gates (push) Successful in 48s
2026-10-19 05:41:34 +02:00
admin 45458196d1 Two texts made true after 0.160.0: the ceremony log line names the confirm order; the R-973 comments name whole_guest_backup_refused (no behaviour change)
gates / gates (push) Successful in 52s
2026-10-19 05:25:58 +02:00
admin 124164db73 CHANGELOG v0.160.0 (released: R-978, R-973, R-1093, R-1149, R-1151)
gates / gates (push) Successful in 46s
2026-10-19 03:22:00 +02:00
48 changed files with 1009 additions and 103 deletions
+3
View File
@@ -64,6 +64,9 @@ jobs:
env:
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
FELHOM_SCAN_FETCH: "1"
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
+18
View File
@@ -0,0 +1,18 @@
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
gitleaks generic-api-key@1ad7cc8eb57e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46ce5a3eccc4 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@46fe126846b0 internal/escrow/identity_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@56474f1754f5 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@631c53997f7e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@6e790ef41ef3 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@78334b854242 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@7b88d323a44f internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b0eeee93cf24 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b1cb2659967a internal/localapi/escrow_stage_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@b66bec43cc8a internal/escrow/recover_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e0525160fe03 internal/pbsdr/wrapper_read_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@e442e455c360 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f23f93f461b2 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
gitleaks generic-api-key@f471b78e0f6e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
+96 -9
View File
@@ -1,15 +1,96 @@
## Unreleased (2026-10-18) — CI workflow: the failure mail links the public run page, by run id (R-1135); no code change
## Unreleased (2026-10-21) — every guest may swap as much as its RAM (R-1192, the 8 GB freeze) — COMMITTED, NOT RELEASED (the release is the operator's word)
`.gitea/workflows/gates.yml`, the alarm step: the mail's run link is now `https://gitea.dooplex.hu/<repo>/actions/runs/<run id>`.
It used `GITHUB_SERVER_URL`, which on this runner is the in-cluster address (`http://gitea.gitea-system.svc.cluster.local:3000`,
the address the runner registers at), so nobody outside the cluster could open it. It also used the run NUMBER, and
Gitea 1.26 reads `/actions/runs/<n>` as a run id first: a number that is also an older run's id opened that other run
(measured 2026-10-18: `felhom.eu/actions/runs/1116` opened run id 1116, run number 613). It now uses `GITHUB_RUN_ID` and
prints the link (`ALARM-LINK …`) into the job log. Same change in all four repos' workflows.
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
never set it. On the 8 GB machine (guest 4096 MB) the first uploads made Immich load its analysis models; the guest hit
its limit with the swap full and could evict only program code — 150 000–600 000 refaults and ~1 GB of disk reads per
10 s, memory „full" pressure 92–95 %, load 155, the dashboard dead for 6½ minutes, no app killed. Raising the swap cap
to 4096 MB live ended it within a minute while the uploads ran on.
## Unreleased — three fixes from the 2026-10-12 security review and the storage build (R-1093, R-978, R-973) (2026-10-18)
- `reconcile.buildBringUpConfig`: a new guest gets `swap` = its memory.
- `POST /guest/memory` (RAM resize): `swap` moves with `memory` (`localapi.GuestSwapMB`).
- At agent start (one minute in): `localapi.EnsureGuestSwap` raises every LXC in the `felhom` pool whose swap is below
its RAM to it — a live cgroup change, no restart; never lowers a higher swap; one guest's failure is logged and the
others are still done.
- Tests `TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM`, `TestR1192_BringUpSetsSwapToTheRAM`, the resize test's
new swap assertion — each red-proved (the write removed → `raised 0, want 2`; `swap=""` on both other paths).
- Privilege: `swap` is under `VM.Config.Memory`, which the RAM resize already uses; no role, sudoers or bundle change.
Committed, not released — the release and its vouch are the operator's.
Added the night of 2026-10-21 (still unreleased, the same release):
- **The host's swap is reported and checked (R-1192 follow-up).** The guest fix moves a memory burst onto the HOST's
swap; a host with none gets nothing from it. The host report carries `host.host_swap_mb` (Proxmox node status — the
kernel's SwapTotal, the same number as `/proc/meminfo`) and `host.guest_swap_mb` (the sum of the guests' `swap`; each
guest spec also carries `swap_bytes`). When the first is below the second the agent logs one WARN (and one INFO when
it is covered again); the hub raises `host_swap_short`. `GET /guest/memory` carries `host_swap_mb` and
`guest_swap_mb` for the controller's install page. The agent never creates host swap (a decision — felhom.eu
register). Measured the same night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap.
- **R-1194 — no false „operator access degraded" on a fresh box.** The first host report was sent before the
felhom-sshd loop's first pass, read a config that was never rendered („config invalid, port 0") and the hub mailed
`oob_degraded` — reproduced on a fresh install 2026-10-21 (the report at 20:34:00, the apply 0.4 s later, the daemon
healthy; the check runs `sshd -t` through sudo, so the non-root theory was false). The loop now reports no OOB stanza
until its first pass has run (a failed pass counts, so a real lasting failure still reaches the hub).
- Tests `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
`TestR1194_NoOOBStanzaBeforeTheFirstPass` — each red-proved.
## v0.161.0 — three fixes (R-921, R-1070, R-1010); two texts made true after 0.160.0 (2026-10-20) — RELEASED on the operator's yes, NOT VOUCHED
Released 2026-10-20 (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at `5152123`:
binary sha256 `ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5`, config bundle sha256
`485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47`, tag `v0.161.0`. Delivered by one signed
`agent_update` job per box to demo-hp (06:38 UTC), Tester 1 (06:54) and demo-felhom (06:55): each `signed op COMPLETED`,
`capabilities self-check ok=76 total=76`, `update committed`. **The vouch is the operator's.** Live on demo-hp: `host_busy`
answers `{"busy": true, "what": "backup:local"}` during a „Mentés most" whole-guest backup and `{"busy": false}` before
and after (read from inside 9201 with its own token; felhom.eu `audits/night2-2026-10-20/P1/`). R-1083 (a key for an
existing guest) is NOT in this release: see the R-1083 row. No sudoers, wrapper or config-bundle change.
- **R-921 — `GET /backup/status` says when the host is busy.** New field `host_busy` (`{busy, what}`): whether the
host-wide heavy-operation gate is held right now (another guest's backup, the restore test, a night OS step, the
stale-lock sweep, a power-off) — the state that makes a `POST /backup` answer 409. Served on every request; absent
when the gate is not wired. Controller ≥ 0.335.0 reads it before it stops the household's apps for a tier, so a
refused tier costs no stop (the measured case of 2026-10-08: the night OS step, all 15 apps stopped for a refusal).
A read, never a reservation. No new route. Test `TestBackupStatus_R921_HostBusy`, red-proved (assignment removed →
`free gate: host_busy = <nil>`).
- **R-1070 — a box with no tunnel set up no longer fails its host OS update.** Docker 29 prints „error: no such
object" in lower case, so the tunnel probe's case-sensitive match fell through to `unknown` („could not ask the
guest"), and the host health rule failed every box without a cloudflared container (Tester4, 2026-10-15,
`os_update_health_failed`). Now the match ignores case and a missing container is a new state, **`absent`** („no
tunnel set up on this box"); the host rule reads it as not applicable only when the reading from before the update
(Docker answered) also had no cloudflared container. A tunnel that is down, gone or unreadable still fails. The hub
alarms only on `not_running`, so `absent` never mails. Tests `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows) and
three `TestClassifyTunnel` rows (Docker 29's wording measured on demo-hp), red-proved both ways.
- **R-1010 — the stale-lock janitor clears a half-removed `vzdump` snapshot.** When the delete fails ONLY with
„Failed to find logical volume …_vzdump" (one disk's snapshot already gone — demo-hp 2026-10-14 after a backup ran
out of space), it retries once with Proxmox's own `force=1` on the same API call (`DeleteSnapshotForce`; the same
token privilege, no sudoers line, no root command — `pvesh usage … -command delete` lists `--force`). Any other error
is returned unchanged; a failed forced delete is returned, never a second force. Tests
`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone` (6 rows) and `TestDeleteSnapshotForce_R1010`, red-proved
(single plain delete → `calls [plain], want [plain force]`).
- **The security-scan gate (R-986), tooling only:** `agent_gates.py` registers the SHARED
`felhom.eu/scripts/security_scan_gate.py` (FAST set: the hook and CI) — govulncheck (called vulnerabilities) and gitleaks
on the pushed commits, **warn-first**. `.security-accepted` holds the first scan's 15 accepted hits (escrow and
local-API test constants), each with a reason. CI fetches the pinned gitleaks (`FELHOM_SCAN_FETCH=1`). Decoys: the
shared 8 plus the gate on this repo, in `test_gate_decoys.py`.
### Earlier on 2026-10-19 (texts only)
- The ceremony-complete log line now says the package uploads at the claim **or at the typed-back confirm** (R-1151);
it read „package uploads at the claim" for both orders (seen in demo-hp's journal, 2026-10-19 01:47 UTC).
- The R-973 comments in `cmd/felhom-agent/main.go` and `internal/localapi/backup_cap.go` named the hub event
`agent_capability_degraded`; since hub v0.159.0 (R-1150) a refused backup raises `whole_guest_backup_refused`.
Found by the hub helper of the day brief 2026-10-19.
- **gofmt, R-1160:** `1edfd8e` formats the 14 Go files that were not gofmt-clean (formatting only, `go test ./...`
green). New gate `scripts/gofmt_gate.py` (the controller's R-454 shape) in `agent_gates.py`, fast set; its decoys in
`scripts/test_gate_decoys.py` plant an unformatted file in a SCRATCH copy (convicted by name), the formatted twin
(passes), and the no-Go modes (CI passes out loud, a dev machine is INCONCLUSIVE). Red-proved: with the conviction
switched off the FACT case reports a LIVE HOLE. The next release carries it; no release for it.
## v0.160.0 — five fixes: an abandoned or un-typed-back escrow ceremony keeps the old code (R-978, R-1151); a whole-guest backup flood is capped (R-973) and the local tier keeps one point per day (R-1149); an empty partition is not „data" (R-1093) (2026-10-19)
Released 2026-10-19 on the operator's yes (02:48 answers, D17 „one per day, 3 days"). sha256
`7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1`, config bundle
`6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f`. The vouch is the operator's.
- **R-1093 — an EMPTY partition no longer asks the typed „data will be erased" confirmation.** `lsblk` prints the
parent disk's table type (PTTYPE) on every partition, so `SudoHostOps.InspectDevice` read a blank partition as
@@ -91,6 +172,12 @@ Committed, not released — the release and its vouch are the operator's.
with keep-last 3 the local tier's three points can still be replaced within one day (3 per day is the schedule's own
pace plus one press); the hub's event text reads „agent privileged capability degraded" (the type is generic). No
sudoers or bundle change.
- **CI workflow (R-1135, no code change).** `.gitea/workflows/gates.yml`, the alarm step: the mail's run link is now `https://gitea.dooplex.hu/<repo>/actions/runs/<run id>`.
It used `GITHUB_SERVER_URL`, which on this runner is the in-cluster address (`http://gitea.gitea-system.svc.cluster.local:3000`,
the address the runner registers at), so nobody outside the cluster could open it. It also used the run NUMBER, and
Gitea 1.26 reads `/actions/runs/<n>` as a run id first: a number that is also an older run's id opened that other run
(measured 2026-10-18: `felhom.eu/actions/runs/1116` opened run id 1116, run number 613). It now uses `GITHUB_RUN_ID` and
prints the link (`ALARM-LINK …`) into the job log. Same change in all four repos' workflows.
## v0.159.2 — the orphan thin check runs only after a failed backup (R-1012 follow-up) (2026-10-18)
+11
View File
@@ -1,5 +1,16 @@
# CONTEXT — felhom-agent working state
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
> `ff4e6dc5…99d3b5`, bundle `485f36c9…12c47`. R-1083 left out on purpose: an agent key for 9202 would make its
> controller book whole-guest backups through the agent, one tier writing to ep0 — the R-1083 row says what is needed.
> **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978,
> R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with
> `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn
> positions; only those two words are held, 30-min window; a claim without it keeps the R-978 order). Vouch: version
> 0.160.0, sha256 `7ebb547f…dfca1`, bundle `6b060901…e202f`. After it, unreleased: `4545819` (two texts). felhom.eu
> `REPORT-day-2026-10-19.md`.
> **2026-10-18 (night) — UNRELEASED on main: R-1093, R-978, R-973** (release + vouch: the operator's). R-1093 an empty
> partition is blank (lsblk's PTTYPE on a partition is the parent's; set aside only on a clean blkid exit 0 with a
> matching `PART_ENTRY_SCHEME`; an unparsable lsblk is unprobed). R-978 the escrow package reaches the hub only at the
+16 -38
View File
@@ -1,40 +1,18 @@
# REPORT — three agent fixes, committed and NOT released (2026-10-18 evening)
# REPORT — agent, the night of 2026-10-21: the host's swap is reported; no false emergency-door alarm on a new box — COMMITTED, NOT RELEASED
Task: the evening brief of 2026-10-18, Part 4 (felhom.eu `REPORT-evening-2026-10-18.md`). Baseline: `27bbdaa` (v0.159.2
released). Commits: `f7b7308` + `0963bae` (R-1093), `8561dc8` (R-978), `f17e197` (R-973). CHANGELOG under „Unreleased";
the release and its vouch are the operator's. Gates green per commit (`go build/vet/test ./...`, `agent_gates.py --fast`);
CI job 2062 success for `f17e197`.
**Brief:** the unattended night brief of 2026-10-21 (Parts 1, 2, 5). Architecture: felhom.eu `03-host-agent.md` §9 (the
new swap paragraph). Both join the swap fix (`286a7e3`) in the coming 0.162.0 — the operator's release and vouch (D26 = A).
## R-1093 — an empty partition is blank
lsblk reports the PARENT disk's PTTYPE for a partition. `InspectDevice` now ignores that PTTYPE only when `blkid -p` on the
partition exited 0, printed `PART_ENTRY_NUMBER`, and its `PART_ENTRY_SCHEME` equals lsblk's PTTYPE. Every error path
stays data-bearing: blkid exit 2 / I/O error / missing binary keep „device has a partition table"; an lsblk answer that
exits 0 but does not parse or names no device is „could not be reliably inspected" (an older fail-open, closed here after
a background security review flagged the probe). No new sudo command, no config bundle.
Test `TestInspect_R1093_PartitionVerdicts` (14 rows); three red-proofs. **Live on demo-hp** (a 1 GB loop file in `/root`,
removed after; felhom.eu `audits/agent-fixes-2026-10-18/r1093-loop-check.txt`): the empty GPT partition → lsblk
`pttype "gpt"` (the bug's input), blkid rc 0 with only `PART_ENTRY_*` (the fix's condition holds); ext4 → `TYPE=ext4`; a
dos table inside the partition → `PTTYPE=dos`. Not walked: the whole `/api/storage/init` call (the agent refuses loop
devices; it needs a VM with a spare disk running a build of this commit).
## R-978 — the new escrow package reaches the hub only at the claim
A ceremony started again no longer replaces the hub's package at create time. The package travels with the pending
ceremony and is uploaded when the code is claimed, before the code is handed out; an upload failure is 502 and the code
stays claimable; a ceremony abandoned by TTL, a new start or a restart leaves the first code opening the hub's package.
Ceremony output is version 2 (`PendingUpload`); the sudo argv is unchanged. Tests `TestEscrowCeremony_R978_*` and three
`cmd` tests; four red-proofs. **Residual (filed):** the typed-back check is the browser's, so a household that opens the
new code and closes the page without writing it down has still replaced the package.
## R-973 — a cadence cap on `POST /backup`
Per guest and tier, successful runs only: the daily (local) tier at most **3 per rolling 24 h** (the night run, one
„Mentés most", the next night — `07` §6; since R-899 a press never costs the night); the longer tier at most one per
max(cadence − 24 h, cadence / 2) = **6 days** for the weekly off-site tier. A refusal is 429 with the next allowed time,
WARN-logged, kept in `backup-success-state.json` (survives a restart), and for 24 h the host report carries the CRITICAL
degraded status `localapi:backup-cadence-cap` (the hub's operator-only `agent_capability_degraded`). Tests `TestBackupCap_*`
and others; three red-proofs. **Limits (filed):** keep-last 3 lets three local points be replaced within one day; the
off-site namespace's keep-last 2 is a count — a time-bucketed prune on ep0 is the operator's.
*Decided by CC unattended — operator may reverse:* the numbers 3 / 24 h and 6 days, and counting successes only.
- **Host swap (R-1192 follow-up), `0da792e`:** the host report carries `host.host_swap_mb` (Proxmox node status = the
kernel's SwapTotal) and `host.guest_swap_mb` (the guests' `swap` sum; each guest spec `swap_bytes`); one WARN when the
first is below the second, one INFO when covered again; `GET /guest/memory` carries both for the controller's install
page. The agent never creates host swap (register R-1207). Proven live on the soak box: the hub stored
`host_swap_mb 7935, guest_swap_mb 4096`.
- **R-1194, `0da792e`:** no OOB stanza until the felhom-sshd loop's first pass has run — the first report on a fresh box
read a config not yet rendered and the hub mailed `oob_degraded` (reproduced 2026-10-21 20:34 CEST on a fresh install).
- **The swap fix itself** ran as a hand-placed fixture on the 8 GB soak box: guest swap 512 → 4096 MB one minute after
start; the 2-hour household load answered every probe (felhom.eu `audits/night-2026-10-21/P1-soak/`).
- Release candidate built from a clean clone at `0da792e` with the release flags, version `0.162.0`:
sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (not published).
- Tests (red-proved): `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
`TestR1194_NoOOBStanzaBeforeTheFirstPass`. CI job 2176 green.
+18 -2
View File
@@ -409,7 +409,7 @@ const backupCapStatusName = "localapi:backup-cadence-cap"
// backupCapStatus turns the local-API backup cadence cap's newest refusal (R-973) into a CRITICAL
// capability status: degraded for 24 h after a refusal, so the hub raises its operator-only
// agent_capability_degraded event (no hub change). The Reason names the guest, the tier and the time, and
// whole_guest_backup_refused event (hub v0.159.0, R-1150; agent_capability_degraded before). The Reason names the guest, the tier and the time, and
// says what to do. nil source (local API off) → ok.
func backupCapStatus(refusal func() (string, int, time.Time, bool)) capability.Status {
s := capability.Status{
@@ -856,7 +856,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
log: logger,
}
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
// existing agent_capability_degraded event (operator-only) tells the operator. Late-bound like
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
// drConfigured: the local API server is built further down, before the loop starts.
var backupCapRefusal func() (string, int, time.Time, bool)
probeAll := func(ctx context.Context) []capability.Status {
@@ -936,6 +936,22 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
// after start so the guest has settled; a failure is logged and the next start retries.
go func() {
select {
case <-ctx.Done():
return
case <-time.After(time.Minute):
}
sctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
if n, err := localapi.EnsureGuestSwap(sctx, px, reconcile.DefaultPool, logger); err != nil {
logger.Warn("guest-swap: check skipped", "err", err)
} else {
logger.Info("guest-swap: checked", "raised", n)
}
}()
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
+1 -1
View File
@@ -2,9 +2,9 @@ package backup
import (
"context"
"fmt"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"testing"
+1 -1
View File
@@ -48,7 +48,7 @@ func TestR689_GuestBackupArchiveShapes(t *testing.T) {
{proxmox.StorageContent{VolID: "felhom-pbs:backup/vm/300/2026-07-28T05:31:14Z", VMID: 300}, true},
{proxmox.StorageContent{VolID: "local:backup/felhom-golden-0.236.0.tar.zst"}, false},
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", VMID: 9202}, false}, // vmid disagrees with the name
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
} {
if ok, why := guestBackupArchive(c.e); ok != c.ok {
t.Errorf("%s vmid=%d: ok=%v (%s), want %v", c.e.VolID, c.e.VMID, ok, why, c.ok)
+1 -1
View File
@@ -47,7 +47,7 @@ type Runner interface {
// downgrades a HEALTHY probe to StatusInactive (broken plumbing stays degraded regardless).
type Prober struct {
Runner Runner
Exists func(path string) bool // nil → os.Stat
Exists func(path string) bool // nil → os.Stat
GateActive func(gate string) bool // nil → every gate treated active
}
+2 -2
View File
@@ -21,10 +21,10 @@ import (
// The four inputs (the consumption contract — spike findings):
// - blob: the opaque R-wrapped escrow blob (hub-served restore directive in 10D).
// - recoveryCode (R): BY HAND from the customer — NEVER hub-sourced (zero-knowledge holds: a
// hub compromise alone cannot decrypt). A secret: never logged/persisted.
// hub compromise alone cannot decrypt). A secret: never logged/persisted.
// - expectedFingerprint: hub-served — the gate target (F-C4).
// - keyDest: where the PBS restore reads the key (`--keyfile` or the default
// `$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json`).
// `$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json`).
//
// Order (spike F-C2/F-C3/F-C4/F-C6): Unwrap → fingerprint-gate → install. On ANY failure there is
// NO partial install (nothing at keyDest) and no key material leaks: the recovered key lives only
+22 -1
View File
@@ -22,6 +22,11 @@ type Loop struct {
mu sync.Mutex
fetched bool // a desired-state document has been delivered (false = never a teardown signal)
block *hub.WireWireguard
// passed (R-1194) is set once the first reconcile pass has run. Until then the instance has never been rendered,
// so its health says nothing — and the first host report, sent before that pass, used to read „config invalid,
// port 0" and raise a false oob_degraded on EVERY fresh box (rehearsal 2026-10-20; reproduced 2026-10-21: the
// report at 20:34:00 read an absent config, the apply landed 0.4 s later).
passed bool
nudge chan struct{}
}
@@ -85,6 +90,7 @@ func (l *Loop) reconcile(ctx context.Context) {
if !fetched {
applyBlock = nil
}
defer l.markPassed()
port, err := l.mgr.Apply(ctx, applyBlock)
if err != nil {
return // Apply logged; a claim/exhaustion or install error — retry next tick
@@ -106,8 +112,23 @@ func oobPeerIP(block *hub.WireWireguard) string {
return block.OOBPeerIP
}
// OOBStatus implements the hub collector's reporter seam (Part 4).
// markPassed records that a reconcile pass has run (R-1194) — after a failed Apply too, so a real, lasting failure
// is still reported from the next report on.
func (l *Loop) markPassed() {
l.mu.Lock()
l.passed = true
l.mu.Unlock()
}
// OOBStatus implements the hub collector's reporter seam (Part 4). nil — no stanza, which the hub does not evaluate —
// until the first reconcile pass has run (R-1194).
func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus {
l.mu.Lock()
passed := l.passed
l.mu.Unlock()
if !passed {
return nil
}
_, block := l.snapshot()
return l.mgr.Status(ctx, block)
}
@@ -0,0 +1,29 @@
package felhomsshd
import (
"context"
"testing"
"time"
)
// R-1194: on every fresh box the first host report — sent before the OOB loop's first pass — read „config invalid,
// port 0" (the config was not rendered yet) and the hub mailed oob_degraded. Reproduced 2026-10-21 on a fresh box
// (felhom.eu audits/night-2026-10-21/P5-checks/): the report at 20:34:00, the apply 0.4 s later, the daemon healthy.
// The loop now reports NO stanza (the hub does not evaluate a missing one) until a pass has run — a failed pass
// counts, so a real lasting failure is still reported from the next report on.
// RED-PROOF: with OOBStatus not gated on `passed`, the first assertion fails (a stanza with config_invalid).
func TestR1194_NoOOBStanzaBeforeTheFirstPass(t *testing.T) {
r := newScriptRunner()
m := NewManager(r, t.TempDir(), nil)
m.isActive = func(context.Context) bool { return false }
m.isFailed = func(context.Context) bool { return false }
m.now = func() time.Time { return time.Unix(1783270000, 0) }
l := NewLoop(m, nil, time.Minute, nil)
if st := l.OOBStatus(context.Background()); st != nil {
t.Fatalf("a stanza before the first pass reads a config that was never rendered: %+v", st)
}
l.reconcile(context.Background()) // fails or succeeds — either way a pass has run
if st := l.OOBStatus(context.Background()); st == nil {
t.Fatal("after the first pass the health must be reported (a real failure must reach the hub)")
}
}
+2 -2
View File
@@ -50,8 +50,8 @@ func TestParseConfMounts(t *testing.T) {
func TestMissingBindSources(t *testing.T) {
mounts := ParseConfMounts(sampleConf)
present := map[string]bool{
"/mnt/felhom-usb/felhom-data": true, // usb attached
"/var/lib/felhom-agent/guests/9201/bootstrap": true, // bootstrap always present
"/mnt/felhom-usb/felhom-data": true, // usb attached
"/var/lib/felhom-agent/guests/9201/bootstrap": true, // bootstrap always present
// "/mnt/felhom-flash/felhom-data" is ABSENT (drive unplugged)
}
got := MissingBindSources(mounts, func(p string) bool { return present[p] })
+10 -3
View File
@@ -14,6 +14,13 @@ const (
TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED
TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which)
TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting)
// TunnelAbsent (R-1070, agent Unreleased 2026-10-19): the guest answered and has NO cloudflared container — the
// controller makes none when no tunnel token is configured (a LAN-only or test box). Not a failure by itself: the
// hub never alarms on it, and the OS-update host rule reads it as „not applicable" only when the container was
// also absent before the update (HostHealthVerdict). Docker 29 prints „error: no such object" in lower case, so
// before this state existed the case-sensitive match fell through to `unknown` — which failed the host rule on
// every box without a tunnel (Tester4, 2026-10-15).
TunnelAbsent = "absent"
)
// CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec.
@@ -53,7 +60,7 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
return TunnelUnknown, "no running customer guest"
}
worst, wdetail := "", ""
rank := map[string]int{TunnelRunning: 0, TunnelUnknown: 1, TunnelNotRunning: 2}
rank := map[string]int{TunnelRunning: 0, TunnelAbsent: 1, TunnelUnknown: 2, TunnelNotRunning: 3}
for _, v := range vmids {
out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared")
st, d := ClassifyTunnel(string(out), string(errOut), err)
@@ -71,8 +78,8 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
func ClassifyTunnel(stdout, stderr string, err error) (string, string) {
out := strings.TrimSpace(stdout)
if err != nil || out == "" {
if strings.Contains(stderr, "No such object") || strings.Contains(stderr, "No such container") {
return TunnelNotRunning, "no cloudflared container in the guest"
if e := strings.ToLower(stderr); strings.Contains(e, "no such object") || strings.Contains(e, "no such container") {
return TunnelAbsent, "no cloudflared container in the guest (no tunnel set up on this box)"
}
return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err)
}
+4 -1
View File
@@ -20,7 +20,10 @@ func TestClassifyTunnel(t *testing.T) {
{"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"},
{"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"},
{"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelNotRunning, "no cloudflared container"},
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
// R-1070: Docker 29's wording, measured on demo-hp 2026-10-19 (`docker inspect` of a missing name).
{"absent, docker 29", "", "error: no such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no tunnel set up"},
{"absent, no such container", "", "Error response from daemon: No such container: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
{"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"},
{"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"},
{"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"},
+7 -1
View File
@@ -11,6 +11,7 @@ import (
"os"
"strings"
"sync"
"sync/atomic"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
@@ -136,6 +137,8 @@ type Collector struct {
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
logger *slog.Logger
now func() time.Time
// swapShortWarned (R-1192): the host-swap-short WARN is logged once per state change, not every report.
swapShortWarned atomic.Bool
}
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
@@ -375,6 +378,8 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
}
host := hostMetrics(c.px.Node(), ns)
guests := c.collectGuests(ctx)
c.applySwap(&host, ns, guests) // R-1192: host swap vs the guests' swap allowance
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable
report := &HostReport{
@@ -383,7 +388,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
AgentVersion: c.agentVersion,
AgentSHA256: c.agentSHA256(),
Host: host,
Guests: c.collectGuests(ctx),
Guests: guests,
// storage_targets populated this slice (slice 5) via the observer; the rest stay
// defined-but-empty (slice 6). Non-nil so they marshal as [].
StorageTargets: c.collectStorage(ctx),
@@ -567,6 +572,7 @@ func (c *Collector) collectGuests(ctx context.Context) []Guest {
Cores: cfg.Cores,
MemoryBytes: g.MaxMem,
DiskBytes: g.MaxDisk,
SwapBytes: cfg.Swap * (1 << 20),
}
}
guests = append(guests, entry)
+39
View File
@@ -0,0 +1,39 @@
package hub
import (
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the guest swap fix (every guest may swap as much as its RAM,
// localapi/guestswap.go) moves a memory burst onto the HOST's swap. Measured on demo-hp the same day: our installer
// gives an 8 GB / 256 GB machine 7.75 GB of host swap, so a 4096 MB guest's 4096 MB swap fits. A host with no swap
// (a BYO install, or swap turned off) gets nothing from the fix. So the report carries both numbers, the agent says it
// once in its log, and the hub raises host_swap_short. The agent never creates host swap itself: changing a host its
// owner set up is a decision (felhom.eu register).
// HostSwapShort reports whether the host's swap is below the guests' swap allowance. A host whose guests have no
// swap allowance (0) is never short.
func HostSwapShort(hostMB, guestMB int64) bool { return guestMB > 0 && hostMB < guestMB }
// applySwap fills the host block's swap numbers from the node status and the guests' specs, and logs the short
// state once per change.
func (c *Collector) applySwap(h *HostMetrics, ns proxmox.NodeStatus, guests []Guest) {
hostMB := ns.Swap.Total / (1 << 20)
var guestMB int64
for _, g := range guests {
if g.Spec != nil {
guestMB += g.Spec.SwapBytes / (1 << 20)
}
}
h.HostSwapMB, h.GuestSwapMB = &hostMB, &guestMB
if HostSwapShort(hostMB, guestMB) {
if c.swapShortWarned.CompareAndSwap(false, true) {
c.logger.Warn("host-swap: the host's swap is below its guests' swap allowance — a memory burst can freeze the box (R-1192); the hub is told (host_swap_short)",
"host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
return
}
if c.swapShortWarned.CompareAndSwap(true, false) {
c.logger.Info("host-swap: the host's swap covers its guests' swap allowance again", "host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
}
+4 -4
View File
@@ -77,10 +77,10 @@ func TestLoop_FailedTailPushIsReArmedByNextEnvelope(t *testing.T) {
env *ControlEnvelope
err error
}{
{env: &ControlEnvelope{LogTailRequested: true}}, // arm
{err: errors.New("hub 5xx")}, // the carrying push FAILS
{env: &ControlEnvelope{LogTailRequested: true}}, // hub still pending → re-arm
{env: &ControlEnvelope{}}, // fulfilled
{env: &ControlEnvelope{LogTailRequested: true}}, // arm
{err: errors.New("hub 5xx")}, // the carrying push FAILS
{env: &ControlEnvelope{LogTailRequested: true}}, // hub still pending → re-arm
{env: &ControlEnvelope{}}, // fulfilled
}}
l := tailLoop(rep)
ctx := context.Background()
+42
View File
@@ -0,0 +1,42 @@
package hub
import (
"bytes"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the host report carries the host's swap and the guests' swap allowance, and the
// agent says ONCE in its log when the host's swap is below it (and once when it is covered again).
// RED-PROOF: with applySwap not called the numbers are nil; with the CompareAndSwap removed the WARN repeats.
func TestR1192_HostReportCarriesSwapAndWarnsOnce(t *testing.T) {
var buf bytes.Buffer
c := &Collector{logger: slog.New(slog.NewTextHandler(&buf, nil))}
guests := []Guest{{VMID: 9201, Spec: &GuestSpec{SwapBytes: 4096 << 20}}, {VMID: 9202, Spec: nil}}
var ns proxmox.NodeStatus // a host with no swap
for i := 0; i < 3; i++ {
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if h.HostSwapMB == nil || h.GuestSwapMB == nil || *h.HostSwapMB != 0 || *h.GuestSwapMB != 4096 {
t.Fatalf("report numbers: host %v guest %v", h.HostSwapMB, h.GuestSwapMB)
}
}
if n := strings.Count(buf.String(), "host-swap: the host's swap is below"); n != 1 {
t.Fatalf("the short WARN must be logged once, got %d:\n%s", n, buf.String())
}
ns.Swap.Total = 7935 << 20 // the installer's 8 GB host
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if *h.HostSwapMB != 7935 || HostSwapShort(*h.HostSwapMB, *h.GuestSwapMB) {
t.Fatalf("a covered host must not be short: %v / %v", *h.HostSwapMB, *h.GuestSwapMB)
}
if !strings.Contains(buf.String(), "covers its guests' swap allowance again") {
t.Fatalf("the recovery must be said once:\n%s", buf.String())
}
if HostSwapShort(0, 0) {
t.Fatal("a host whose guests have no swap allowance is never short")
}
}
+7 -1
View File
@@ -338,6 +338,11 @@ type HostMetrics struct {
//
// Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift.
WrapperSHA256 string `json:"wrapper_sha256,omitempty"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up, night 2026-10-21): the host's swap (Proxmox node status — the
// kernel's SwapTotal) and the sum of its guests' swap allowance. The guest swap fix (swap = RAM) needs swap ON THE
// HOST behind it; the hub raises host_swap_short when the first is below the second. nil = not read (older agent).
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
}
// Guest is one LXC. The agent reports vmid; the hub derives the guest PK
@@ -355,11 +360,12 @@ type GuestSpec struct {
Cores int `json:"cores"`
MemoryBytes int64 `json:"memory_bytes"`
DiskBytes int64 `json:"disk_bytes"`
SwapBytes int64 `json:"swap_bytes,omitempty"` // R-1192: the guest's swap allowance (PVE config `swap`, MB → bytes)
}
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
type Cloudflared struct {
Status string `json:"status"` // running | not_running | unknown (TunnelRunning …)
Status string `json:"status"` // running | not_running | unknown | absent (TunnelRunning …; absent since R-1070)
Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected"
}
+2 -2
View File
@@ -31,8 +31,8 @@ import (
// Only SUCCESSES count — a failed run prunes nothing and must stay retryable. The count survives an agent
// restart: it reads the success history saved on disk (LastKnownBackupStore, the R-894 file) as well as
// this process's own record. A refusal answers 429, is logged at WARN, is saved, and raises the operator
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's existing
// `agent_capability_degraded` event, operator-only). Pinned by TestBackupCap_* (backup_cap_test.go).
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's operator-only
// `whole_guest_backup_refused` event since hub v0.159.0, R-1150). Pinned by TestBackupCap_* (backup_cap_test.go).
// backupCapRefusalHold is how long one refusal keeps the operator signal raised.
const backupCapRefusalHold = 24 * time.Hour
+1 -1
View File
@@ -369,7 +369,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
job.Uploaded = false // R-978: uploaded by the claim
s.finishEscrowCeremony(job, out.RecoveryCode, out.PendingUpload)
out.RecoveryCode = "" // drop the parsed reference promptly (GC caveat: best-effort)
s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim",
s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim, or at the typed-back confirm (R-1151)",
"job_id", job.JobID, "restic_pw_sealed", job.ResticPwSealed, "uploaded", job.Uploaded,
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
}()
+12 -1
View File
@@ -57,6 +57,10 @@ type MemoryInfo struct {
MaxMB int64 `json:"max_mb"`
FloorMB int64 `json:"floor_mb"`
Running bool `json:"running"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up): the host's swap and this guest's swap allowance, so the controller
// can tell the household when the host has too little swap behind the guest (its install page). nil = not read.
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
}
// memoryResizeRequest is the POST /guest/memory body.
@@ -94,6 +98,8 @@ type memoryBounds struct {
maxMB int64
floorMB int64
running bool
hostSwapMB int64
guestSwapMB int64
}
func (b memoryBounds) refusal(code string) memoryRefusal {
@@ -104,7 +110,9 @@ func (b memoryBounds) refusal(code string) memoryRefusal {
}
func (b memoryBounds) info(vmid int) MemoryInfo {
hostSwap, guestSwap := b.hostSwapMB, b.guestSwapMB
return MemoryInfo{
HostSwapMB: &hostSwap, GuestSwapMB: &guestSwap,
VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB,
MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running,
}
@@ -139,6 +147,8 @@ func (s *Server) readMemoryBounds(ctx context.Context, vmid int) (memoryBounds,
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
minMB: minGuestMemoryMB,
running: st.Status == "running",
hostSwapMB: node.Swap.Total / mib, // R-1192 follow-up: the kernel's SwapTotal, as Proxmox reports it
guestSwapMB: cfg.Swap, // PVE config swap is MB
}
b.maxMB = b.hostTotalMB - hostReserveMB
b.floorMB = b.usageMB + shrinkUsageMarginMB
@@ -225,7 +235,8 @@ func (s *Server) handleGuestMemoryResize(w http.ResponseWriter, r *http.Request,
}
// Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target)})
// R-1192: the swap moves with the memory (GuestSwapMB) — a grown guest with the old swap cap thrashes the same way.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target), "swap": fmt.Sprintf("%d", GuestSwapMB(target))})
if err != nil {
s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err)
writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error())
+8 -1
View File
@@ -25,12 +25,14 @@ type fakeMemory struct {
setUPID string
setErr error
applyReflect bool
hostSwapB int64 // R-1192 follow-up: node swap total
guestSwapMB int64
}
func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
f.mu.Lock()
defer f.mu.Unlock()
return proxmox.GuestConfig{Memory: f.allocMB}, nil
return proxmox.GuestConfig{Memory: f.allocMB, Swap: f.guestSwapMB}, nil
}
func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) {
f.mu.Lock()
@@ -42,6 +44,7 @@ func (f *fakeMemory) NodeStatus(_ context.Context) (proxmox.NodeStatus, error) {
defer f.mu.Unlock()
var ns proxmox.NodeStatus
ns.Memory.Total = f.hostTotalB
ns.Swap.Total = f.hostSwapB
return ns, nil
}
func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) {
@@ -151,6 +154,10 @@ func TestGuestMemory_GrowAndShrink(t *testing.T) {
if f.lastParams["memory"] != "12288" {
t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"])
}
// R-1192: the swap moves with the memory.
if f.lastParams["swap"] != "12288" {
t.Errorf("SetConfig swap = %q, want 12288 (swap = RAM)", f.lastParams["swap"])
}
})
t.Run("shrink above floor", func(t *testing.T) {
f := stdFixture() // floor = max(2048, 3000+512) = 3512
+73
View File
@@ -0,0 +1,73 @@
package localapi
import (
"context"
"fmt"
"log/slog"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// ── R-1192 (2026-10-21): the guest may swap as much as its RAM ────────────────────────────────────────────────────
//
// MEASURED (felhom.eu audits/fixes-2026-10-21/P4-freeze/): on the minimum machine (8 GB host, guest 4096 MB) the
// household's first uploads made Immich load its analysis models (+1.15 GB) on top of 8 apps' 1.75 GB; the guest hit
// its 4096 MB limit with its swap capped at Proxmox's LXC default of 512 MB — and full. The kernel could then evict
// only file pages: ~360 MB was left for the program code of ~25 containers, re-read from disk without end (150 000–
// 600 000 refaults and ~1 GB of reads per 10 s; memory „full" pressure 92–95 %, load 155, the dashboard dead, 0 OOM
// kills). Raising the cap to 4096 MB live ended it in under a minute (memory „full" 96 % → 1.7 %, dashboard back
// 42 s later) while the uploads ran on. The host had 7.4 GB of swap free the whole time.
//
// The agent never set `swap` at all, so every guest had the 512 MB default. Now:
// - a new guest is created with swap = its memory (reconcile.bringup);
// - a RAM resize sets swap with the memory (handleGuestMemoryResize);
// - at agent start, every LXC in the felhom pool whose swap is below its memory is raised to it (this file).
// Raising swap is a live cgroup change (memory.swap.max), no restart. It never lowers a swap someone set higher.
// GuestSwapMB is the swap a guest of memoryMB gets: the same as its RAM.
func GuestSwapMB(memoryMB int64) int64 { return memoryMB }
// SwapOps is the Proxmox surface EnsureGuestSwap needs (satisfied by *proxmox.Client).
type SwapOps interface {
Pool(ctx context.Context, name string) (proxmox.PoolInfo, error)
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
SetConfig(ctx context.Context, vmid int, params map[string]string) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
}
// EnsureGuestSwap raises the swap of every LXC guest in pool to GuestSwapMB(memory) when it is below. Returns how
// many guests were raised. One guest's failure is logged and the others are still handled.
func EnsureGuestSwap(ctx context.Context, ops SwapOps, pool string, logger *slog.Logger) (int, error) {
p, err := ops.Pool(ctx, pool)
if err != nil {
return 0, fmt.Errorf("guest swap: read pool %s: %w", pool, err)
}
raised := 0
for _, m := range p.Members {
if m.Type != "lxc" || m.VMID <= 0 {
continue
}
cfg, err := ops.GuestConfig(ctx, m.VMID)
if err != nil {
logger.Warn("guest-swap: config read failed (left as it is)", "vmid", m.VMID, "err", err)
continue
}
want := GuestSwapMB(cfg.Memory)
if cfg.Memory <= 0 || cfg.Swap >= want {
logger.Debug("guest-swap: nothing to do", "vmid", m.VMID, "memory_mb", cfg.Memory, "swap_mb", cfg.Swap)
continue
}
upid, err := ops.SetConfig(ctx, m.VMID, map[string]string{"swap": fmt.Sprintf("%d", want)})
if err == nil && upid != "" {
_, err = ops.WaitTask(ctx, upid, proxmox.WaitOptions{})
}
if err != nil {
logger.Warn("guest-swap: raise failed (left as it is)", "vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want, "err", err)
continue
}
raised++
logger.Info("guest-swap: raised (R-1192 — a guest swap cap below its RAM turned a memory burst into a freeze)",
"vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want)
}
return raised, nil
}
+71
View File
@@ -0,0 +1,71 @@
package localapi
import (
"context"
"errors"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: the freeze measured on the minimum machine (felhom.eu audits/fixes-2026-10-21/P4-freeze/) — a guest whose
// swap is capped below its RAM evicts program code instead of idle memory. EnsureGuestSwap raises every pool guest's
// swap to its RAM, never lowers one, skips non-LXC members, and survives one guest's failure.
// COMPANION RED-PROOF: return 0 before the SetConfig call → „raised 0, want 2".
type fakeSwapOps struct {
cfg map[int]proxmox.GuestConfig
failSet map[int]bool
set map[int]map[string]string
}
func (f *fakeSwapOps) Pool(context.Context, string) (proxmox.PoolInfo, error) {
return proxmox.PoolInfo{Members: []proxmox.PoolMember{{VMID: 9201, Type: "lxc"}, {VMID: 9202, Type: "lxc"}, {VMID: 300, Type: "qemu"},
{Type: "storage"}, {VMID: 9203, Type: "lxc"}, {VMID: 9204, Type: "lxc"}}}, nil
}
func (f *fakeSwapOps) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
c, ok := f.cfg[vmid]
if !ok {
return c, errors.New("no such guest")
}
return c, nil
}
func (f *fakeSwapOps) SetConfig(_ context.Context, vmid int, p map[string]string) (string, error) {
if f.failSet[vmid] {
return "", errors.New("pve said no")
}
if f.set == nil {
f.set = map[int]map[string]string{}
}
f.set[vmid] = p
return "", nil
}
func (f *fakeSwapOps) WaitTask(context.Context, string, proxmox.WaitOptions) (proxmox.TaskStatus, error) {
return proxmox.TaskStatus{}, nil
}
func TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM(t *testing.T) {
f := &fakeSwapOps{cfg: map[int]proxmox.GuestConfig{
9201: {Memory: 4096, Swap: 512}, // the measured box: raised
9202: {Memory: 2048, Swap: 4096}, // already higher: never lowered
9203: {Memory: 6144, Swap: 0}, // none: raised
9204: {Memory: 4096, Swap: 512}, // the write fails: logged, the others still done
}, failSet: map[int]bool{9204: true}}
n, err := EnsureGuestSwap(context.Background(), f, "felhom", slog.New(slog.NewTextHandler(io.Discard, nil)))
if err != nil {
t.Fatal(err)
}
if n != 2 {
t.Fatalf("raised %d, want 2 (9201, 9203)", n)
}
if f.set[9201]["swap"] != "4096" || f.set[9203]["swap"] != "6144" {
t.Fatalf("writes = %v", f.set)
}
if _, touched := f.set[9202]; touched {
t.Fatal("a swap above the RAM was lowered")
}
if len(f.set[9201]) != 1 {
t.Fatalf("the swap raise wrote more than the swap: %v", f.set[9201])
}
}
+39
View File
@@ -0,0 +1,39 @@
package localapi
import (
"encoding/json"
"testing"
)
// R-1192 follow-up (night 2026-10-21): GET /guest/memory carries the host's swap and this guest's swap allowance, so
// the controller's install page can say when the host has no swap behind the guest. 0 is a real answer (no swap);
// absent means an older agent.
// RED-PROOF: without the two MemoryInfo fields the keys are missing → fails on „host_swap_mb absent".
func TestR1192_GuestMemoryCarriesHostAndGuestSwap(t *testing.T) {
for _, c := range []struct {
name string
hostB, guestMB int64
wantHost, wantG int64
}{
{"installer host (7.75 GB swap), guest swap = RAM", 7935 * mib, 8192, 7935, 8192},
{"a host with NO swap", 0, 8192, 0, 8192},
} {
f := stdFixture()
f.hostSwapB, f.guestSwapMB = c.hostB, c.guestMB
w := do(t, memServer(t, f).Handler(), "GET", "/guest/memory", "A", "")
var m struct {
Data map[string]any `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &m); err != nil {
t.Fatal(err)
}
hs, ok := m.Data["host_swap_mb"].(float64)
if !ok {
t.Fatalf("%s: host_swap_mb absent: %s", c.name, w.Body.String())
}
gs, _ := m.Data["guest_swap_mb"].(float64)
if int64(hs) != c.wantHost || int64(gs) != c.wantG {
t.Fatalf("%s: host %v guest %v, want %d / %d", c.name, hs, gs, c.wantHost, c.wantG)
}
}
}
+18
View File
@@ -1343,6 +1343,20 @@ type BackupStatusResponse struct {
// presence. Served on the UNTARGETED request only; additive, so an older controller reads the
// response exactly as before.
Tiers []TierBackupState `json:"tiers,omitempty"`
// HostBusy (R-921, agent Unreleased 2026-10-19) says whether the host-wide heavy-operation gate is held RIGHT NOW
// (another guest's backup, the restore test, a night OS step, the stale-lock sweep, a power-off) — a POST /backup
// would then be refused with 409. The controller reads it BEFORE it stops the household's apps, so a refused tier
// costs no stop. Served on every request; ABSENT (nil) = an older agent, or the gate is not wired — the controller
// then keeps its old order (stop, ask, resume at once on 409). A read, never a reservation: the gate can be taken
// between this answer and the POST, and that race keeps the old order. Pinned by TestBackupStatus_R921_HostBusy.
HostBusy *HostBusyState `json:"host_busy,omitempty"`
}
// HostBusyState is GET /backup/status's host_busy (R-921). What names the holder ("backup:local", "restore-test",
// "os-pve-step" …), empty when Busy is false.
type HostBusyState struct {
Busy bool `json:"busy"`
What string `json:"what,omitempty"`
}
func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) {
@@ -1362,6 +1376,10 @@ func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid
resp.JobID = job.JobID
resp.Error = job.Error
}
if s.inFlight != nil {
what := s.inFlight.Busy()
resp.HostBusy = &HostBusyState{Busy: what != "", What: what}
}
writeOK(w, resp)
}
+45
View File
@@ -13,6 +13,7 @@ import (
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
@@ -514,3 +515,47 @@ func TestBackupStatus_FiltersToThisGuest(t *testing.T) {
t.Fatalf("status not scoped to this guest / not latest: %s", w.Body.String())
}
}
// R-921: GET /backup/status says when the host-wide heavy-operation gate is held, so the controller can skip the app
// stop for a tier the POST would refuse. Free → busy false; held → busy true with the holder's name; the gate not wired
// → no field at all (the controller's „older agent" reading, never „free").
//
// COMPANION RED-PROOF (observed 2026-10-19): with the `resp.HostBusy = …` assignment removed, the held case fails with
//
// server_test.go: free gate: host_busy = <nil>, want busy=false
//
// Restored.
func TestBackupStatus_R921_HostBusy(t *testing.T) {
read := func(h http.Handler) *HostBusyState {
w := do(t, h, "GET", "/backup/status", "A", "")
var resp struct {
Data BackupStatusResponse `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("status body: %v (%s)", err, w.Body.String())
}
return resp.Data.HostBusy
}
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
if got := read(srv.Handler()); got != nil {
t.Fatalf("gate not wired: host_busy = %+v, want absent", got)
}
gate := &backup.InFlight{}
srv.inFlight = gate
h := srv.Handler()
if got := read(h); got == nil || got.Busy || got.What != "" {
t.Fatalf("free gate: host_busy = %+v, want busy=false", got)
}
release, _, ok := gate.TryAcquire("os-pve-step")
if !ok {
t.Fatal("could not take the gate")
}
if got := read(h); got == nil || !got.Busy || got.What != "os-pve-step" {
t.Fatalf("held gate: host_busy = %+v, want busy=true what=\"os-pve-step\"", got)
}
// The read never takes the gate: a POST after the holder lets go is accepted.
release()
if got := read(h); got == nil || got.Busy {
t.Fatalf("released gate: host_busy = %+v, want busy=false", got)
}
}
+46 -1
View File
@@ -2,6 +2,7 @@ package localapi
import (
"context"
"errors"
"fmt"
"log/slog"
"os"
@@ -160,6 +161,7 @@ type staleLockAPI interface {
ListSnapshots(ctx context.Context, vmid int) ([]proxmox.Snapshot, error)
ListRunningTasks(ctx context.Context) ([]proxmox.TaskStatus, error)
DeleteSnapshot(ctx context.Context, vmid int, snapname string) (string, error)
DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error)
Start(ctx context.Context, vmid int) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
}
@@ -334,8 +336,34 @@ func (c *staleLockController) Unlock(ctx context.Context, vmid int) error {
return nil
}
// DeleteVzdumpSnapshot removes the dangling `vzdump` snapshot. R-1010: when the delete fails ONLY because one disk's
// snapshot volume is already gone („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump" — a backup that ran
// out of space, 2026-10-14), Proxmox stops at that volume and keeps the entry and the other volumes' snapshots, and a
// plain retry fails the same way every sweep, forever. Then it retries ONCE with Proxmox's force (the hand fix that
// day was `pct delsnapshot 9201 vzdump --force`): the remaining volumes' snapshots are removed and the entry goes. Any
// other error is returned as it is — never a force on an error that might mean a volume still holds data (a full
// LVM metadata area, a busy volume). Pinned by TestStaleLock_R1010_*.
func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int) error {
upid, err := c.px.DeleteSnapshot(ctx, vmid, vzdumpSnapshotName)
err := c.deleteSnap(ctx, vmid, false)
if err == nil || !snapshotVolumeAlreadyGone(err) {
return err
}
if c.logger != nil {
c.logger.Warn("stale-lock: a disk's vzdump snapshot volume is already gone — retrying the delete once with force (R-1010)",
"vmid", vmid, "err", err)
}
if ferr := c.deleteSnap(ctx, vmid, true); ferr != nil {
return fmt.Errorf("forced snapshot delete after %q also failed: %w", err.Error(), ferr)
}
return nil
}
func (c *staleLockController) deleteSnap(ctx context.Context, vmid int, force bool) error {
del := c.px.DeleteSnapshot
if force {
del = c.px.DeleteSnapshotForce
}
upid, err := del(ctx, vmid, vzdumpSnapshotName)
if err != nil {
return err
}
@@ -346,6 +374,23 @@ func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int
return err
}
// snapshotVolumeAlreadyGone: the error is Proxmox's „Failed to find logical volume …_vzdump" from a snapshot delete
// (in the task's exit status, its log tail, or the API error text). Narrow on purpose: the force is safe only when the
// volume it cannot remove does not exist.
func snapshotVolumeAlreadyGone(err error) bool {
text := err.Error()
var te *proxmox.TaskError
if errors.As(err, &te) {
text += "\n" + te.ExitStatus + "\n" + strings.Join(te.LogTail, "\n")
}
for _, l := range strings.Split(text, "\n") {
if strings.Contains(strings.ToLower(l), "failed to find logical volume") && strings.Contains(l, "_"+vzdumpSnapshotName) {
return true
}
}
return false
}
func (c *staleLockController) Start(ctx context.Context, vmid int) error {
upid, err := c.px.Start(ctx, vmid)
if err != nil {
+19 -1
View File
@@ -30,6 +30,9 @@ type fakeStaleLockAPI struct {
delsnap []int
started []int
// R-1010: scripted delete answers, consumed in order (plain and forced share the list); calls records "plain"/"force".
delErrs []error
delCalls []string
}
func (f *fakeStaleLockAPI) ListLXC(context.Context) ([]proxmox.Guest, error) { return f.lxc, nil }
@@ -55,7 +58,22 @@ func (f *fakeStaleLockAPI) ListRunningTasks(context.Context) ([]proxmox.TaskStat
func (f *fakeStaleLockAPI) DeleteSnapshot(_ context.Context, vmid int, snapname string) (string, error) {
f.delsnap = append(f.delsnap, vmid)
return "", nil // synchronous — no WaitTask
return "", f.nextDelErr("plain") // synchronous — no WaitTask
}
func (f *fakeStaleLockAPI) DeleteSnapshotForce(_ context.Context, vmid int, snapname string) (string, error) {
f.delsnap = append(f.delsnap, vmid)
return "", f.nextDelErr("force")
}
func (f *fakeStaleLockAPI) nextDelErr(kind string) error {
f.delCalls = append(f.delCalls, kind)
if len(f.delErrs) == 0 {
return nil
}
e := f.delErrs[0]
f.delErrs = f.delErrs[1:]
return e
}
func (f *fakeStaleLockAPI) Start(_ context.Context, vmid int) (string, error) {
+59
View File
@@ -0,0 +1,59 @@
package localapi
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1010: a half-removed vzdump snapshot. The error text is the one demo-hp's task printed on 2026-10-14
// (`audits/power-cut-fixes-2026-10-14/C4/hand-fix.txt`).
const r1010MissingLV = `lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to find logical volume "pve/snap_vm-9201-disk-0_vzdump"`
func r1010TaskErr(exit string) error {
return &proxmox.TaskError{UPID: "UPID:demo-hp:0001:0002:0003:vzdelsnapshot:9201:root@pam:", ExitStatus: exit,
LogTail: []string{exit}}
}
// COMPANION RED-PROOF (observed 2026-10-19): with DeleteVzdumpSnapshot back to the single plain delete, the first case
// fails with
//
// stalelock_r1010_test.go: missing volume: calls [plain], want [plain force]; err = proxmox: task … failed: exitstatus "lvremove snapshot …"
//
// Restored.
func TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone(t *testing.T) {
cases := []struct {
name string
errs []error
wantCalls []string
wantErr string
}{
{"missing volume", []error{r1010TaskErr(r1010MissingLV)}, []string{"plain", "force"}, ""},
{"missing volume, named only in the log tail", []error{&proxmox.TaskError{UPID: "x", ExitStatus: "command 'lvremove' failed: exit code 5",
LogTail: []string{r1010MissingLV}}}, []string{"plain", "force"}, ""},
{"the plain delete works", nil, []string{"plain"}, ""},
{"another error: no force", []error{r1010TaskErr("lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to update pool pve/data.")},
[]string{"plain"}, "Failed to update pool"},
{"a missing volume of ANOTHER snapshot: no force", []error{r1010TaskErr(`Failed to find logical volume "pve/snap_vm-9201-disk-0_mine"`)},
[]string{"plain"}, "_mine"},
{"the forced delete fails too: returned, never a second force", []error{r1010TaskErr(r1010MissingLV), errors.New("still broken")},
[]string{"plain", "force"}, "still broken"},
}
for _, c := range cases {
api := &fakeStaleLockAPI{delErrs: c.errs}
ctrl := NewStaleLockController(api, &recordingRunner{}, "felhom", nil)
err := ctrl.DeleteVzdumpSnapshot(context.Background(), 9201)
if strings.Join(api.delCalls, " ") != strings.Join(c.wantCalls, " ") {
t.Errorf("%s: calls %v, want %v; err = %v", c.name, api.delCalls, c.wantCalls, err)
}
switch {
case c.wantErr == "" && err != nil:
t.Errorf("%s: err = %v, want nil", c.name, err)
case c.wantErr != "" && (err == nil || !strings.Contains(err.Error(), c.wantErr)):
t.Errorf("%s: err = %v, want one naming %q", c.name, err, c.wantErr)
}
}
}
+7 -7
View File
@@ -6,12 +6,12 @@
// the report stanza the hub surfaces:
//
// - /run/sshd present? — OpenSSH's SHARED privsep dir; its absence is the KEXINIT-reset
// lockout (SPIKE-felhom-sshd-2026-07-05 §8).
// lockout (SPIKE-felhom-sshd-2026-07-05 §8).
// - stock sshd listener answers? — a TCP connect to the sshd port (22 for G1; H1 passes the
// discovered felhom-sshd port later).
// discovered felhom-sshd port later).
// - did the watchdog auto-heal? — the watchdog writes an RFC3339 marker to /run when it heals;
// its presence (+ timestamp) tells the hub a clobber recurred, so
// the operator learns of a recurring cause BEFORE a lockout.
// its presence (+ timestamp) tells the hub a clobber recurred, so
// the operator learns of a recurring cause BEFORE a lockout.
//
// Read-only and dependency-light: os.Stat + os.ReadFile + a short net.Dial, no exec, no sudo. Safe to
// run every heartbeat.
@@ -44,9 +44,9 @@ const (
type Reporter struct {
privsepDir string
marker string
sshdAddr string // host:port dialed for the reachability probe
statDir func(string) bool // dir-exists check (os.Stat wrapper; test seam)
readMarker func(string) (string, bool) // marker read → (timestamp, present)
sshdAddr string // host:port dialed for the reachability probe
statDir func(string) bool // dir-exists check (os.Stat wrapper; test seam)
readMarker func(string) (string, bool) // marker read → (timestamp, present)
dialSSHD func(ctx context.Context, addr string) bool
}
+19 -2
View File
@@ -370,7 +370,8 @@ func HealthVerdict(before, after *Health) (bool, string) {
// HostHealthVerdict is THE host health rule (`11` §8.2; pinned by TestHostHealthVerdict): the Proxmox daemons and the
// agent are active, the customer guest still runs, the guest's own rule still passes against the start of the pass,
// and the tunnel is `running` (R-841).
// and the tunnel is `running` (R-841) — or `absent` both now and before the update: a box with no tunnel set up
// (R-1070; pinned by TestHostHealthVerdict_R1070_NoTunnelSetUp).
func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
if after == nil {
return false, "no health reading"
@@ -398,12 +399,28 @@ func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
if ok, why := HealthVerdict(gb, after.Guest); !ok {
return false, "guest: " + why
}
if tunnel != hub.TunnelRunning {
switch {
case tunnel == hub.TunnelRunning:
case tunnel == hub.TunnelAbsent && noTunnelBefore(before):
// R-1070: no cloudflared container before the update and none after — the box has no tunnel to check.
case tunnel == hub.TunnelAbsent:
return false, "the tunnel is absent (no cloudflared container now, and the reading from before the update does not show it was absent then)"
default:
return false, "the tunnel is " + tunnel
}
return true, ""
}
// noTunnelBefore: the reading from the start of the pass shows the guest's Docker answered and listed NO cloudflared
// container. A missing or failed reading is not „no tunnel" (an empty list from a failed `docker ps` must not pass).
func noTunnelBefore(before *Health) bool {
if before == nil || before.Guest == nil || !before.Guest.DockerOK {
return false
}
_, had := before.Guest.Containers["cloudflared"]
return !had
}
// EngineOf is the engine version `docker version` prints for a docker-ce package version: "5:29.8.2-1~debian.13~trixie"
// → "29.8.2" (no epoch, no Debian revision).
func EngineOf(pkgVersion string) string {
+39
View File
@@ -568,3 +568,42 @@ func TestDocker_OOMCheckReachesTheHubUnchanged(t *testing.T) {
t.Fatalf("no docker report posted: %s", calls(w))
}
}
// R-1070: a box with no tunnel set up (no cloudflared container before the update and none after) passes the host
// rule; a box WITH a tunnel that is down, gone, or unreadable still fails it.
//
// COMPANION RED-PROOF (observed 2026-10-19): with the „not applicable" branch switched off (only `running` passes, the
// pre-fix rule) the first case fails with
//
// leg_test.go: no tunnel before, none after: got false (the tunnel is absent (no cloudflared container now, …)), want true
//
// and with the pre-fix case-sensitive match TestClassifyTunnel's Docker 29 row reads `unknown` („could not ask the
// guest"), the state Tester4 failed on.
// Restored.
func TestHostHealthVerdict_R1070_NoTunnelSetUp(t *testing.T) {
withTunnel := hostOK()
withTunnel.Guest.Containers["cloudflared"] = Container{State: "running", Health: "healthy"}
dockerFailed := hostOK()
dockerFailed.Guest = &Health{DockerOK: false, Containers: map[string]Container{}}
cases := []struct {
name string
before *Health
tunnel string
want bool
why string
}{
{"no tunnel before, none after", hostOK(), hub.TunnelAbsent, true, ""},
{"a tunnel before, gone after", withTunnel, hub.TunnelAbsent, false, "cloudflared was running"},
{"a tunnel before, down after", withTunnel, hub.TunnelNotRunning, false, "cloudflared was running"},
{"no tunnel before, a stopped one after", hostOK(), hub.TunnelNotRunning, false, "not_running"},
{"no tunnel before, the probe could not ask after", hostOK(), hub.TunnelUnknown, false, "unknown"},
{"no reading from before", nil, hub.TunnelAbsent, false, "absent"},
{"docker failed before (an empty list is not proof of none)", dockerFailed, hub.TunnelAbsent, false, "absent"},
}
for _, c := range cases {
got, why := HostHealthVerdict(c.before, hostOK(), c.tunnel)
if got != c.want || !strings.Contains(why, c.why) {
t.Errorf("%s: got %v (%s), want %v (…%s…)", c.name, got, why, c.want, c.why)
}
}
}
+12
View File
@@ -180,6 +180,18 @@ func (c *Client) DeleteSnapshot(ctx context.Context, vmid int, snapname string)
return c.dataString(ctx, http.MethodDelete, path, nil)
}
// DeleteSnapshotForce is DeleteSnapshot with Proxmox's own `force=1` („remove from the config file, even if removing
// disk snapshots fails"): the API's flag, so the same token privilege (VM.Snapshot) and no new root command. Used by
// the stale-lock janitor ONLY after a delete failed because one disk's snapshot volume is already gone (R-1010). The
// flag rides the query string, where pveproxy reads a DELETE's parameters. Pinned by TestDeleteSnapshotForce_R1010.
func (c *Client) DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error) {
if vmid == 0 || snapname == "" {
return "", fmt.Errorf("proxmox: DeleteSnapshotForce needs vmid and snapname")
}
path := fmt.Sprintf("/nodes/%s/lxc/%d/snapshot/%s?force=1", c.node, vmid, url.PathEscape(snapname))
return c.dataString(ctx, http.MethodDelete, path, nil)
}
// SetConfig applies config changes via PUT /nodes/{node}/lxc/{vmid}/config
// (e.g. memory, cores, net0, mpN with a backup flag). PVE may apply this
// synchronously (no UPID) — the returned string is empty in that case, and "" is
+22
View File
@@ -3,6 +3,7 @@ package proxmox
import (
"context"
"io"
"net/http"
"strings"
"testing"
)
@@ -99,3 +100,24 @@ func TestDestroyScratchLXC_RefusesUnconfiguredBand(t *testing.T) {
}
}
}
// R-1010: the forced snapshot delete is the API's own flag on the same DELETE (no new root command).
func TestDeleteSnapshotForce_R1010(t *testing.T) {
var got *http.Request
d := &mockDoer{fn: func(r *http.Request) (*http.Response, error) {
got = r
return jsonResp(200, `{"data":"UPID:demo-felhom:0001:0002:0003:vzdelsnapshot:9201:u@pve!t:"}`), nil
}}
upid, err := newTestClient(d).DeleteSnapshotForce(context.Background(), 9201, "vzdump")
if err != nil || upid == "" {
t.Fatalf("DeleteSnapshotForce: upid %q err %v", upid, err)
}
if got.Method != http.MethodDelete || got.URL.Path != "/api2/json/nodes/demo-felhom/lxc/9201/snapshot/vzdump" ||
got.URL.Query().Get("force") != "1" {
t.Fatalf("request = %s %s, want DELETE …/lxc/9201/snapshot/vzdump?force=1", got.Method, got.URL.String())
}
d.fn = func(r *http.Request) (*http.Response, error) { got = r; return jsonResp(200, `{"data":null}`), nil }
if _, err := newTestClient(d).DeleteSnapshot(context.Background(), 9201, "vzdump"); err != nil || got.URL.RawQuery != "" {
t.Fatalf("the plain delete must carry no force: %s (err %v)", got.URL.String(), err)
}
}
+4 -1
View File
@@ -59,7 +59,7 @@ const DefaultPool = "felhom"
//
// SysDataGrowGB itself is NOT removed — see its field comment: the host installer still passes
// `-sysdata-grow`, and its GiB are FOLDED INTO the single volume's grow rather than dropped.
const DefaultSysDataMount = ""
const DefaultSysDataMount = ""
// Structural host-bind mountpoints every provisioned guest carries (GL-5; verdict of
// SPIKE-dr-bindmount-source-2026-07-07): the permanent drives parent bind (mp8,
@@ -645,6 +645,9 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
}
if spec.MemoryMB > 0 {
params["memory"] = strconv.Itoa(spec.MemoryMB)
// R-1192: swap = RAM. Left unset, Proxmox gives an LXC 512 MB, and a guest that fills its RAM can then evict
// only program code — the measured freeze (localapi.GuestSwapMB carries the reasoning and the numbers).
params["swap"] = strconv.Itoa(spec.MemoryMB)
}
for i, m := range spec.Mounts {
// backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED
+19
View File
@@ -0,0 +1,19 @@
package reconcile
import (
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: a new guest is created with swap = its RAM (left unset, Proxmox gives an LXC 512 MB — the measured freeze,
// felhom.eu audits/fixes-2026-10-21/P4-freeze/). COMPANION RED-PROOF: drop params["swap"] in buildBringUpConfig.
func TestR1192_BringUpSetsSwapToTheRAM(t *testing.T) {
p := buildBringUpConfig(BringUpSpec{MemoryMB: 4096}, proxmox.GuestConfig{})
if p["memory"] != "4096" || p["swap"] != "4096" {
t.Fatalf("bring-up params memory=%q swap=%q, want 4096/4096", p["memory"], p["swap"])
}
if q := buildBringUpConfig(BringUpSpec{}, proxmox.GuestConfig{}); q["swap"] != "" {
t.Fatalf("no memory in the spec → no swap written either, got %q", q["swap"])
}
}
+2 -2
View File
@@ -25,8 +25,8 @@ import (
// operator's signing CLI (cmd/felhom-opsign) produces it; this runner consumes it. The hub never
// forges one (it holds no signing key) and never opens it.
type Envelope struct {
OpBlobB64 string `json:"op_blob_b64"` // base64 of the canonical OpBlob JSON
SigArmored string `json:"sig_armored"` // armored SSHSIG over the op-blob (namespace felhom-op-v1)
OpBlobB64 string `json:"op_blob_b64"` // base64 of the canonical OpBlob JSON
SigArmored string `json:"sig_armored"` // armored SSHSIG over the op-blob (namespace felhom-op-v1)
}
// JobSource fetches + clears jobs. Satisfied by *hub.Client.
+3 -3
View File
@@ -14,10 +14,10 @@ func TestClassifyClaim(t *testing.T) {
nodes: []claimNode{{name: "sdd", fstype: "", mountpoint: ""}, {name: "sdd1", fstype: "ntfs", mountpoint: ""}}}
}
cases := []struct {
name string
mutate func(f *claimFacts)
name string
mutate func(f *claimFacts)
wantUnclaimed bool
reasonHas string
reasonHas string
}{
{"clean unclaimed disk (ntfs, unmounted)", func(f *claimFacts) {}, true, "unclaimed"},
{"blank disk", func(f *claimFacts) { f.nodes = []claimNode{{name: "sdd"}} }, true, "unclaimed"},
+3 -3
View File
@@ -103,12 +103,12 @@ type MountSpec struct {
// Binaries holds the absolute paths of the allow-listed binaries (overridable from config
// so the sudoers entries and the agent agree on exact paths).
type Binaries struct {
Systemctl string
Install string
Systemctl string
Install string
// PrivApply is the root content checker (R-861, agent v0.146.0): a unit file reaches /etc/systemd/system only
// through `felhom-priv-apply unit <name>`, which reads the staged copy from /var/lib/felhom-agent/units/ and
// refuses anything the renderers here never produce (a bind over /etc, a Where outside /mnt, …).
PrivApply string
PrivApply string
Smartctl string
Lvs string
Blkid string // device signature probe (8C data-bearing detection)
+4 -4
View File
@@ -17,12 +17,12 @@ import (
// Four states (3A):
// - new : durable-id NOT in the store. Never auto-mounted (the user must enroll it).
// - enrolled : desired = mounted + bound into the guest. Drift (present && unmounted) with
// this intent → reconcile.
// this intent → reconcile.
// - ejected : an intentional, temporary unmount, set ONLY via the official eject endpoint.
// Present && unmounted → left alone. CLEARED to `enrolled` when the drive goes
// physically ABSENT, so a replug auto-mounts (the replug rule, for free).
// Present && unmounted → left alone. CLEARED to `enrolled` when the drive goes
// physically ABSENT, so a replug auto-mounts (the replug rule, for free).
// - decommissioned : permanent. Never auto-mounted; SURVIVES absent/present; cleared only by an
// explicit re-commission.
// explicit re-commission.
//
// The elegant invariant: intent is recorded ONLY through the official enroll/eject/decommission
// paths. "Is it unmounted" ≠ "did someone officially eject it."
+5 -5
View File
@@ -93,11 +93,11 @@ systemd-1 /mnt/felhom-drives/live autofs rw,relatime,fd=86 0 0
where string
want bool
}{
{"/mnt/felhom-drives/live", true}, // real nfs4 (the autofs line for the same path must not confuse it)
{"/mnt/felhom-drives/smb", true}, // cifs
{"/mnt/felhom-drives/idle", false}, // autofs trigger ONLY — idle automount is NOT mounted
{"/mnt/felhom-drives/disk", false}, // a local fs at the path is not a network mount
{"/mnt/felhom-drives/nope", false}, // absent
{"/mnt/felhom-drives/live", true}, // real nfs4 (the autofs line for the same path must not confuse it)
{"/mnt/felhom-drives/smb", true}, // cifs
{"/mnt/felhom-drives/idle", false}, // autofs trigger ONLY — idle automount is NOT mounted
{"/mnt/felhom-drives/disk", false}, // a local fs at the path is not a network mount
{"/mnt/felhom-drives/nope", false}, // absent
}
for _, tc := range cases {
if got := networkMountedIn(procMounts, tc.where); got != tc.want {
+2 -2
View File
@@ -96,11 +96,11 @@ type Watchdog struct {
now func() time.Time
spawn func(func()) // spawn a background task (overridable in tests; default `go f()`)
intent IntentReader // P3: gate re-mount by drive intent (nil = ungated legacy)
intent IntentReader // P3: gate re-mount by drive intent (nil = ungated legacy)
onAbsent func(durableID string) // P3: called when a known target goes ABSENT (clears `ejected`)
mu sync.Mutex
last map[string]bool // name -> last observed present (only for seen targets)
last map[string]bool // name -> last observed present (only for seen targets)
lastFire time.Time
fired bool // lastFire is valid
pending bool // a transition is awaiting the debounce window
+13
View File
@@ -15,6 +15,9 @@ Gates (all must pass; **non-zero exit on any failure**):
or because someone is offline on a train. CI runs the FULL set for exactly this
reason: it is the machine that can afford a network check, and it is the half
that emails when something is wrong.
… security-scan govulncheck (called vulnerabilities) + gitleaks on the pushed commits, WARN-FIRST: it lists,
it never refuses (R-986). Shared: felhom.eu/scripts/security_scan_gate.py. In the fast set
although it reads vuln.go.dev — offline it says NOT CHECKED, so it cannot stop a push.
… gate-decoys the decoy suite (test_gate_decoys.py): every gate here still convicts its decoys
(R-1116). ~20 s in the workspace, so the full run only: not the hook, but CI
runs the full set on every push. Scratch clones and a local fake Gitea only.
@@ -49,6 +52,9 @@ SHARED_INSTRUCTIONS = os.path.join(
# R-389 — shared, like the two above: it lives in felhom.eu/scripts/ and is never copied.
SHARED_OBSERVATIONS = os.path.join(
os.path.dirname(ROOT), "felhom.eu", "scripts", "observations_gate.py")
# R-986 — shared too: the dependency and secret scanner, warn-first.
SHARED_SECURITY = os.path.join(
os.path.dirname(ROOT), "felhom.eu", "scripts", "security_scan_gate.py")
# (label, absolute script path, args, fast)
GATES = [
@@ -67,6 +73,13 @@ GATES = [
# runs the FULL set, so every push is still judged by it. It plants only in scratch clones and asks a fake
# Gitea on 127.0.0.1 (never the network, never the real tree); it does not call this runner (no recursion).
("gate-decoys", os.path.join(ROOT, "scripts", "test_gate_decoys.py"), [], False),
# R-1160 (2026-10-19) — every Go file is gofmt-clean; the controller's gofmt_gate.py shape. Fast: one gofmt -l.
# Where the runner has no Go it says NOT CHECKED out loud and passes; the pre-push hook (with Go) enforces it.
("gofmt", os.path.join(ROOT, "scripts", "gofmt_gate.py"), [], True),
# R-986 (2026-10-19) — govulncheck + gitleaks, WARN-FIRST (exit 0 whatever it finds). Fast although it reads
# vuln.go.dev: it cannot refuse, so a network blink is a NOT CHECKED line. ~4 s here. CI: gitleaks fetched pinned
# (FELHOM_SCAN_FETCH=1), no Go, so govulncheck says NOT CHECKED there.
("security-scan", SHARED_SECURITY, [ROOT], True),
]
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""gofmt_gate.py — every Go file in this module is gofmt-clean (R-1160; the controller's R-454 gate, same shape).
MEASURED 2026-10-19: 14 agent files were not gofmt-clean and nothing noticed (fixed by one formatting-only commit,
1edfd8e). The controller's history below is why a gate and not a one-off clean-up.
MEASURED 2026-09-02: five internal/web test files had been gofmt-unclean for an unknown length of
time and nothing noticed — `go vet` does not check formatting and no gate did. By 2026-10-05 the five
were clean and TWELVE others were not: the count only grows while nothing looks, and every `gofmt -l`
someone runs by hand is then noise that hides a real one.
Run from the repo root: python3 scripts/gofmt_gate.py
Exit 0 clean · 1 convicted (lists the files) · 2 inconclusive (gofmt not found — never a pass).
gofmt is the Go toolchain's own formatter (not a shell utility), found on PATH or under
`go env GOROOT`/bin. Decoy: scripts/test_gate_decoys.py ("gofmt"), in a scratch copy of this script.
"""
import os
import shutil
import subprocess
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ROOTS = ["cmd", "internal"]
def find_gofmt():
p = shutil.which("gofmt")
if p:
return p
go = shutil.which("go")
if go:
try:
root = subprocess.check_output([go, "env", "GOROOT"], text=True).strip()
cand = os.path.join(root, "bin", "gofmt")
if os.path.isfile(cand):
return cand
except Exception:
pass
return None
def in_ci():
"""True on the Gitea Actions runner (act sets GITHUB_ACTIONS, Gitea sets GITEA_ACTIONS)."""
return os.environ.get("GITEA_ACTIONS") == "true" or os.environ.get("GITHUB_ACTIONS") == "true"
def main():
gofmt = find_gofmt()
if not gofmt:
# The CI runner may have no Go toolchain (felhom.eu CLAUDE.md, R-887 note). There the gate
# says so OUT LOUD and passes: the pre-push hook on the developer's machine, where Go always exists, is where
# formatting is enforced. Anywhere else a missing gofmt stays INCONCLUSIVE — never a silent pass.
# Pinned by the "gofmt: CI without Go" and "gofmt: dev without Go" decoys in test_gate_decoys.py.
if in_ci():
print("gofmt gate NOT CHECKED in CI: no Go toolchain on the runner — the pre-push hook (with Go) enforces it")
return 0
print("gofmt gate INCONCLUSIVE: gofmt not found on PATH or under `go env GOROOT`/bin")
return 2
roots = [r for r in ROOTS if os.path.isdir(os.path.join(ROOT, r))]
if not roots:
print("gofmt gate INCONCLUSIVE: none of %s exists under %s" % (ROOTS, ROOT))
return 2
p = subprocess.run([gofmt, "-l"] + roots, cwd=ROOT, capture_output=True, text=True)
if p.returncode != 0:
print("gofmt gate INCONCLUSIVE: gofmt exited %d: %s" % (p.returncode, p.stderr.strip()[:400]))
return 2
bad = [l for l in p.stdout.splitlines() if l.strip()]
if bad:
for f in bad:
print(" not gofmt-clean: %s" % f)
print("GOFMT GATE FAILED: %d file(s) — run `gofmt -w <file>` (formatting only, no behaviour change)" % len(bad))
return 1
print("gofmt gate OK — every Go file under %s is gofmt-clean" % ", ".join(roots))
return 0
if __name__ == "__main__":
sys.exit(main())
+50
View File
@@ -66,6 +66,14 @@ COVERS = {
"inside an HTML comment (R-426)",
"observations": "R-419 in THIS repo's REPORT.md: an Observations note SAYING it carries no marker - vs the "
"two genuine markers (R-426)",
"gofmt": "R-1160: an unformatted .go file planted under internal/ of a SCRATCH copy is convicted by name - vs the "
"same tree formatted (passes); with no gofmt reachable, CI passes OUT LOUD and a dev machine is "
"INCONCLUSIVE, never a silent pass",
"security-scan": "R-986: the shared gate's own decoys (felhom.eu scripts/test_security_scan_gate.py, 8 cases in "
"SCRATCH git repos: a planted token listed by rule and file, its value never printed; a second "
"secret on another line of an accepted file still listed; a called govulncheck finding listed, "
"an imported-only one not; no tool is NOT CHECKED out loud), run from here; plus the gate on THIS "
"repo's tree: exit 0 and its summary line (warn-first - the listing is the fact)",
"gate-decoys": "this suite, run as a gate of agent_gates.py's full run, which CI runs on every push (R-1116): "
"every case above is a decoy of its own gate, and ONE case that stops convicting turns the run "
"red - red-proved 2026-10-17 by flipping a reuse-refs FACT case to expect rc 0",
@@ -291,6 +299,32 @@ def release_cases(base, ws):
# ── the shared felhom.eu scripts, against THIS repo's inputs ─────────────────────────────────────
def gofmt_cases(ws):
"""R-1160: the gofmt gate in a scratch root (its ROOT is its own grandparent directory), never the real tree."""
root = os.path.join(ws, "gofmt-root")
os.makedirs(os.path.join(root, "scripts"))
os.makedirs(os.path.join(root, "internal", "x"))
shutil.copy(os.path.join(ROOT, "scripts", "gofmt_gate.py"), os.path.join(root, "scripts", "gofmt_gate.py"))
gate = os.path.join(root, "scripts", "gofmt_gate.py")
planted = os.path.join(root, "internal", "x", "zz_gofmt_decoy.go")
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F( ) int {return 1}\n")
if shutil.which("gofmt") or shutil.which("go"):
rc, out = run([sys.executable, gate], root)
report("gofmt: FACT: an unformatted file under internal/", rc, out, 1, ("zz_gofmt_decoy.go", "GOFMT GATE FAILED"))
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F() int { return 1 }\n")
rc, out = run([sys.executable, gate], root)
report("gofmt: GENUINE: the same file formatted", rc, out, 0, ("gofmt gate OK",))
else:
print(" -- gofmt: not runnable here (no Go toolchain) - the two cases below pin that mode")
for ci, want_rc, want, name in ((True, 0, "NOT CHECKED in CI", "gofmt: CI without Go"),
(False, 2, "INCONCLUSIVE", "gofmt: dev without Go")):
env = {"PATH": os.path.join(ws, "empty-path"), "HOME": ws}
if ci:
env["GITEA_ACTIONS"] = "true"
rc, out = run([sys.executable, gate], root, env=env)
report(name, rc, out, want_rc, (want,))
def shared_cases(ws):
"""A scratch WORKSPACE: a clone of this repo beside symlinks to the siblings, because the shared
scripts reach across (REUSE.md cites hub paths; instructions_gate reads the workspace CLAUDE.md)."""
@@ -343,6 +377,20 @@ def shared_cases(ws):
u"the same minute.**\n", 0)
def security_cases():
"""R-986: the shared scanner's decoys (they build their own scratch repos), then the gate on this repo."""
suite = os.path.join(SHARED, "test_security_scan_gate.py")
gate = os.path.join(SHARED, "security_scan_gate.py")
for f in (suite, gate):
if not os.path.isfile(f):
fails.append("shared %s is MISSING beside this clone - a failure, never a skip" % f)
return
rc, out = run([sys.executable, suite], ROOT)
report("security-scan: the shared gate's 8 decoys", rc, out, 0, ("all 8 cases passed",))
rc, out = run([sys.executable, gate, ROOT], ROOT)
report("security-scan: GENUINE: this repo, warn-first", rc, out, 0, ("security-scan:", "govulncheck"))
def main():
srv = Server(("127.0.0.1", 0), Handler)
threading.Thread(target=srv.serve_forever, daemon=True).start()
@@ -353,6 +401,8 @@ def main():
published_cases(base)
release_cases(base, ws)
shared_cases(ws)
gofmt_cases(ws)
security_cases()
finally:
srv.shutdown()
srv.server_close()