Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1109092def | |||
| b45c47be6c | |||
| b9b831fb90 | |||
| 39fb9cbf6c | |||
| ae69ad4512 | |||
| e803ad2b91 | |||
| 7651461be2 | |||
| a0c88e5183 | |||
| 2d5840ba11 | |||
| b310773dbf | |||
| 0da792e670 | |||
| 286a7e3ecf | |||
| 4e1621c650 | |||
| 51521230c9 | |||
| 84ef5818d5 | |||
| 56aabc729c | |||
| 0cbd500154 | |||
| 1edfd8e331 | |||
| 0bc0bc5da0 | |||
| 45458196d1 | |||
| 124164db73 |
@@ -64,6 +64,9 @@ jobs:
|
||||
env:
|
||||
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
|
||||
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
|
||||
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
|
||||
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
|
||||
FELHOM_SCAN_FETCH: "1"
|
||||
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
|
||||
|
||||
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
# .security-accepted — hits security_scan_gate.py (felhom.eu/scripts, R-986) lists as ACCEPTED, not WARN.
|
||||
# One per line: <tool> <id> <where> # reason. govulncheck: OSV id + module dir; gitleaks: rule@<sha256[:12] of the line> + file.
|
||||
# The first scan's verdicts (2026-10-19): felhom.eu documentation/audits/night-2026-10-19/P7/triage.md
|
||||
gitleaks generic-api-key@1ad7cc8eb57e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@46ce5a3eccc4 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@46fe126846b0 internal/escrow/identity_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@56474f1754f5 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@631c53997f7e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@6e790ef41ef3 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@78334b854242 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@7b88d323a44f internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@b0eeee93cf24 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@b1cb2659967a internal/localapi/escrow_stage_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@b66bec43cc8a internal/escrow/recover_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@e0525160fe03 internal/pbsdr/wrapper_read_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@e442e455c360 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@f23f93f461b2 internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
gitleaks generic-api-key@f471b78e0f6e internal/escrow/recover_retained_test.go # false: a test constant or a decoy value, no live secret
|
||||
+156
-9
@@ -1,15 +1,156 @@
|
||||
## Unreleased (2026-10-18) — CI workflow: the failure mail links the public run page, by run id (R-1135); no code change
|
||||
## Unreleased (2026-10-22/23 night) — the tunnel heals itself after a clock that ran ahead (D30, R-1204) — COMMITTED, NOT RELEASED
|
||||
|
||||
`.gitea/workflows/gates.yml`, the alarm step: the mail's run link is now `https://gitea.dooplex.hu/<repo>/actions/runs/<run id>`.
|
||||
It used `GITHUB_SERVER_URL`, which on this runner is the in-cluster address (`http://gitea.gitea-system.svc.cluster.local:3000`,
|
||||
the address the runner registers at), so nobody outside the cluster could open it. It also used the run NUMBER, and
|
||||
Gitea 1.26 reads `/actions/runs/<n>` as a run id first: a number that is also an older run's id opened that other run
|
||||
(measured 2026-10-18: `felhom.eu/actions/runs/1116` opened run id 1116, run number 613). It now uses `GITHUB_RUN_ID` and
|
||||
prints the link (`ALARM-LINK …`) into the job log. Same change in all four repos' workflows.
|
||||
- **D30 (operator 2026-10-22: the agent makes a new key and registers it on the existing peer path).** When the latest
|
||||
handshake stays stamped in the future for 15 minutes (the R-1204 sign: ep0 keeps the future timestamp and refuses
|
||||
every newer handshake from that key), the agent makes a new WireGuard key pair, keeps the old private key as
|
||||
`wg/private.key.prev` (0600 — the escrow may hold it), and registers the new public key through `POST /hosts/<id>/wg`.
|
||||
The hub re-keys the peer IN PLACE (same /32, generation bump) and its full-list push to ep0 drops the old key and adds
|
||||
the new one — nothing new runs on ep0. A new key is a new peer for ep0, with no stored timestamp, so the next
|
||||
handshake is accepted. At most once an hour; not while the tunnel unit is down; never while a re-register is already
|
||||
in flight. A failed registration is retried by the existing re-key-in-place path. Logs carry public keys only.
|
||||
Tests `TestD30_*` (4), red-proved. A host-loss DR that installs the escrowed (older) key registers it again, and the
|
||||
hub re-keys in place back to it — the same path.
|
||||
- **Review of the night's own code (felhom.eu `REPORT-night-2026-10-23.md`, N6, N7).** The rotation never leaves the box
|
||||
without a readable key: the new key is written and fsynced to a temp file, the current key gets its kept name by a
|
||||
hard link, then the temp file is renamed OVER `private.key` and the directory fsynced (a power cut or a kill between
|
||||
two renames used to leave no key, or an empty one). `private.key.prev` keeps the FIRST old key (the escrow may hold it);
|
||||
later ones are `private.key.prev.<unix>`, the newest three kept. The once-an-hour bound is saved in `wg/last-rekey`, so
|
||||
an agent restart does not reset it; a start or a last re-key dated AFTER now (a clock that jumped back) is dropped,
|
||||
never a block until that date (the commit's own security review). Tests `TestD30_TwoRotationsKeepTheFirstOldKey`,
|
||||
`TestD30_ALastReKeyFromTheFutureDoesNotBlock`,
|
||||
`TestD30_TheHourBoundSurvivesARestart`, red-proved. **Live (felhom.eu `audits/night-2026-10-23/P2-D30/`):** the D30
|
||||
re-key ran on a test box (before these two changes): clock corrected → 15 min → new key registered → the box switched
|
||||
at its next desired-state fetch → a handshake with the new key.
|
||||
- **D4 (operator ruling 2026-10-22, R-982) — root SSH from the LAN by key only, on `main` now (2026-10-23 day brief).** The
|
||||
config bundle carries `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`;
|
||||
content check: exactly that line, `sshd -t -f`). After an install that wrote it: `sshd -t` (the whole config), then
|
||||
`systemctl try-reload-or-restart ssh` (a stopped or socket-activated sshd no longer refuses the bundle), then `sshd -T`
|
||||
is read back and a drop-in that is NOT in force (no Include, an earlier drop-in, a Match block) is said loudly and
|
||||
reported as `sshd_dropin_not_in_force` — never a refusal; an undone bundle reloads sshd again. The console and the
|
||||
web page keep the root password; felhom-sshd (8822) is not touched. Tests `RootKeyOnly` (6), red-proved; the
|
||||
installer's uninstall removes the file (felhom.eu installer 1.36.0). Reviewed 2026-10-23 night (A1–A7).
|
||||
- **Known gap, unchanged (filed):** after a reboot the kernel forgets the future stamp (latest handshake 0), so neither
|
||||
this re-key nor the hub's alarm sees a box whose clock was corrected and which then restarted (R-1223).
|
||||
|
||||
## Unreleased — three fixes from the 2026-10-12 security review and the storage build (R-1093, R-978, R-973) (2026-10-18)
|
||||
## Unreleased (2026-10-22 night) — the tunnel's future-stamped handshake is reported (R-1204) — COMMITTED, NOT RELEASED (no agent release tonight); root SSH by key only (R-982, D4) is a READY PATCH, not on main
|
||||
|
||||
Committed, not released — the release and its vouch are the operator's.
|
||||
- **R-982 / D4 — a ready patch, NOT on main.** Built and proven the same night (felhom.eu
|
||||
`audits/night-2026-10-22/P2-D4/`): a bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
|
||||
(`PermitRootLogin prohibit-password`), a `sshd` content check (exactly `prohibit-password`; `sshd -t -f`), and after
|
||||
an install that wrote it `sshd -t` + `systemctl reload ssh`; tests `RootKeyOnly` (3), red-proved. Live on a fresh
|
||||
nested box from ISO 1.30.0: root password over SSH refused, root key works, the console and the web page still take
|
||||
the password. It was committed (`a0c88e5`) and moved back out (`7651461`): the installer's uninstall must remove every bundle file (felhom.eu `scripts/test_hostinstall.py`), the bundle half reaches a box only with an agent release, and the installer half only with a new `installer-v…` tag (R-110 — a push to `main` alone publishes nothing) — both forbidden tonight. *(Corrected the same night: the first reason written here, „a push of `felhom-host-install.sh` publishes the installer", was wrong — the website serves `/scripts/` from the tag `installer-v1.35.0`, read live in `manifests/webpage.yaml` and the running Deployment.)* Both halves are patches: `felhom-agent-bundle-D4.patch` and `felhom.eu-installer-D4.patch` in that folder; apply them together with the agent release and the next installer tag.
|
||||
- **R-1204 — `wireguard.handshake_future_s`.** A box that ran once with its clock ahead keeps a handshake stamped in the
|
||||
future; ep0 refuses every newer one, and `last_handshake_age_s` (clamped at 0) read „just now" for ever. The heartbeat
|
||||
now carries how many seconds the handshake lies ahead (> 60 s only); the age keeps its clamp. The hub alarms on it
|
||||
(felhom.eu hub `wg_tunnel_stale`). Test `TestR1204_AFutureHandshakeIsReported`, red-proved.
|
||||
|
||||
## v0.162.0 — every guest may swap as much as its RAM (R-1192, the 8 GB freeze); the host's swap is reported and checked; no false „operator access degraded" on a fresh box (R-1194) (2026-10-22) — RELEASED on the operator's yes; the vouch is the operator's
|
||||
|
||||
Released 2026-10-22 07:40 UTC (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at
|
||||
`b310773`: binary sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (the same bytes as the
|
||||
night's candidate), config bundle sha256 `ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5`, tag
|
||||
`v0.162.0`; no `configs/` change since v0.161.0. Delivered by one signed `agent_update` job per box to demo-hp, Tester 1
|
||||
and demo-felhom (07:49–07:58 UTC): each `signed op COMPLETED`, `capabilities self-check ok=76 total=76`, `update
|
||||
committed`, then `guest-swap: raised` for its 9201 (512 MB → its RAM). Evidence felhom.eu
|
||||
`documentation/audits/morning-2026-10-22/release/04-agent-delivery.txt`. **Found at once:** demo-hp (host swap 8191 MB,
|
||||
guest swap 25898 MB) and demo-felhom (8191 MB vs 12288 MB) logged the host-swap WARN and the hub mailed
|
||||
`host_swap_short` — the check compares the host's swap with the guests' whole swap allowance, which a guest larger
|
||||
than the installer's 8 GB host swap always exceeds (felhom.eu register R-1216). Soak (night of 2026-10-21, 8 GB box):
|
||||
11 h, no freeze, the host's swap peaked at 2.7 GB used.
|
||||
|
||||
|
||||
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
|
||||
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
|
||||
never set it. On the 8 GB machine (guest 4096 MB) the first uploads made Immich load its analysis models; the guest hit
|
||||
its limit with the swap full and could evict only program code — 150 000–600 000 refaults and ~1 GB of disk reads per
|
||||
10 s, memory „full" pressure 92–95 %, load 155, the dashboard dead for 6½ minutes, no app killed. Raising the swap cap
|
||||
to 4096 MB live ended it within a minute while the uploads ran on.
|
||||
|
||||
- `reconcile.buildBringUpConfig`: a new guest gets `swap` = its memory.
|
||||
- `POST /guest/memory` (RAM resize): `swap` moves with `memory` (`localapi.GuestSwapMB`).
|
||||
- At agent start (one minute in): `localapi.EnsureGuestSwap` raises every LXC in the `felhom` pool whose swap is below
|
||||
its RAM to it — a live cgroup change, no restart; never lowers a higher swap; one guest's failure is logged and the
|
||||
others are still done.
|
||||
- Tests `TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM`, `TestR1192_BringUpSetsSwapToTheRAM`, the resize test's
|
||||
new swap assertion — each red-proved (the write removed → `raised 0, want 2`; `swap=""` on both other paths).
|
||||
- Privilege: `swap` is under `VM.Config.Memory`, which the RAM resize already uses; no role, sudoers or bundle change.
|
||||
|
||||
Added the night of 2026-10-21 (still unreleased, the same release):
|
||||
|
||||
- **The host's swap is reported and checked (R-1192 follow-up).** The guest fix moves a memory burst onto the HOST's
|
||||
swap; a host with none gets nothing from it. The host report carries `host.host_swap_mb` (Proxmox node status — the
|
||||
kernel's SwapTotal, the same number as `/proc/meminfo`) and `host.guest_swap_mb` (the sum of the guests' `swap`; each
|
||||
guest spec also carries `swap_bytes`). When the first is below the second the agent logs one WARN (and one INFO when
|
||||
it is covered again); the hub raises `host_swap_short`. `GET /guest/memory` carries `host_swap_mb` and
|
||||
`guest_swap_mb` for the controller's install page. The agent never creates host swap (a decision — felhom.eu
|
||||
register). Measured the same night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap.
|
||||
- **R-1194 — no false „operator access degraded" on a fresh box.** The first host report was sent before the
|
||||
felhom-sshd loop's first pass, read a config that was never rendered („config invalid, port 0") and the hub mailed
|
||||
`oob_degraded` — reproduced on a fresh install 2026-10-21 (the report at 20:34:00, the apply 0.4 s later, the daemon
|
||||
healthy; the check runs `sshd -t` through sudo, so the non-root theory was false). The loop now reports no OOB stanza
|
||||
until its first pass has run (a failed pass counts, so a real lasting failure still reaches the hub).
|
||||
- Tests `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
|
||||
`TestR1194_NoOOBStanzaBeforeTheFirstPass` — each red-proved.
|
||||
|
||||
## v0.161.0 — three fixes (R-921, R-1070, R-1010); two texts made true after 0.160.0 (2026-10-20) — RELEASED on the operator's yes, NOT VOUCHED
|
||||
|
||||
Released 2026-10-20 (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at `5152123`:
|
||||
binary sha256 `ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5`, config bundle sha256
|
||||
`485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47`, tag `v0.161.0`. Delivered by one signed
|
||||
`agent_update` job per box to demo-hp (06:38 UTC), Tester 1 (06:54) and demo-felhom (06:55): each `signed op COMPLETED`,
|
||||
`capabilities self-check ok=76 total=76`, `update committed`. **The vouch is the operator's.** Live on demo-hp: `host_busy`
|
||||
answers `{"busy": true, "what": "backup:local"}` during a „Mentés most" whole-guest backup and `{"busy": false}` before
|
||||
and after (read from inside 9201 with its own token; felhom.eu `audits/night2-2026-10-20/P1/`). R-1083 (a key for an
|
||||
existing guest) is NOT in this release: see the R-1083 row. No sudoers, wrapper or config-bundle change.
|
||||
|
||||
- **R-921 — `GET /backup/status` says when the host is busy.** New field `host_busy` (`{busy, what}`): whether the
|
||||
host-wide heavy-operation gate is held right now (another guest's backup, the restore test, a night OS step, the
|
||||
stale-lock sweep, a power-off) — the state that makes a `POST /backup` answer 409. Served on every request; absent
|
||||
when the gate is not wired. Controller ≥ 0.335.0 reads it before it stops the household's apps for a tier, so a
|
||||
refused tier costs no stop (the measured case of 2026-10-08: the night OS step, all 15 apps stopped for a refusal).
|
||||
A read, never a reservation. No new route. Test `TestBackupStatus_R921_HostBusy`, red-proved (assignment removed →
|
||||
`free gate: host_busy = <nil>`).
|
||||
- **R-1070 — a box with no tunnel set up no longer fails its host OS update.** Docker 29 prints „error: no such
|
||||
object" in lower case, so the tunnel probe's case-sensitive match fell through to `unknown` („could not ask the
|
||||
guest"), and the host health rule failed every box without a cloudflared container (Tester4, 2026-10-15,
|
||||
`os_update_health_failed`). Now the match ignores case and a missing container is a new state, **`absent`** („no
|
||||
tunnel set up on this box"); the host rule reads it as not applicable only when the reading from before the update
|
||||
(Docker answered) also had no cloudflared container. A tunnel that is down, gone or unreadable still fails. The hub
|
||||
alarms only on `not_running`, so `absent` never mails. Tests `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows) and
|
||||
three `TestClassifyTunnel` rows (Docker 29's wording measured on demo-hp), red-proved both ways.
|
||||
- **R-1010 — the stale-lock janitor clears a half-removed `vzdump` snapshot.** When the delete fails ONLY with
|
||||
„Failed to find logical volume …_vzdump" (one disk's snapshot already gone — demo-hp 2026-10-14 after a backup ran
|
||||
out of space), it retries once with Proxmox's own `force=1` on the same API call (`DeleteSnapshotForce`; the same
|
||||
token privilege, no sudoers line, no root command — `pvesh usage … -command delete` lists `--force`). Any other error
|
||||
is returned unchanged; a failed forced delete is returned, never a second force. Tests
|
||||
`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone` (6 rows) and `TestDeleteSnapshotForce_R1010`, red-proved
|
||||
(single plain delete → `calls [plain], want [plain force]`).
|
||||
|
||||
- **The security-scan gate (R-986), tooling only:** `agent_gates.py` registers the SHARED
|
||||
`felhom.eu/scripts/security_scan_gate.py` (FAST set: the hook and CI) — govulncheck (called vulnerabilities) and gitleaks
|
||||
on the pushed commits, **warn-first**. `.security-accepted` holds the first scan's 15 accepted hits (escrow and
|
||||
local-API test constants), each with a reason. CI fetches the pinned gitleaks (`FELHOM_SCAN_FETCH=1`). Decoys: the
|
||||
shared 8 plus the gate on this repo, in `test_gate_decoys.py`.
|
||||
|
||||
### Earlier on 2026-10-19 (texts only)
|
||||
|
||||
- The ceremony-complete log line now says the package uploads at the claim **or at the typed-back confirm** (R-1151);
|
||||
it read „package uploads at the claim" for both orders (seen in demo-hp's journal, 2026-10-19 01:47 UTC).
|
||||
- The R-973 comments in `cmd/felhom-agent/main.go` and `internal/localapi/backup_cap.go` named the hub event
|
||||
`agent_capability_degraded`; since hub v0.159.0 (R-1150) a refused backup raises `whole_guest_backup_refused`.
|
||||
Found by the hub helper of the day brief 2026-10-19.
|
||||
- **gofmt, R-1160:** `1edfd8e` formats the 14 Go files that were not gofmt-clean (formatting only, `go test ./...`
|
||||
green). New gate `scripts/gofmt_gate.py` (the controller's R-454 shape) in `agent_gates.py`, fast set; its decoys in
|
||||
`scripts/test_gate_decoys.py` plant an unformatted file in a SCRATCH copy (convicted by name), the formatted twin
|
||||
(passes), and the no-Go modes (CI passes out loud, a dev machine is INCONCLUSIVE). Red-proved: with the conviction
|
||||
switched off the FACT case reports a LIVE HOLE. The next release carries it; no release for it.
|
||||
|
||||
## v0.160.0 — five fixes: an abandoned or un-typed-back escrow ceremony keeps the old code (R-978, R-1151); a whole-guest backup flood is capped (R-973) and the local tier keeps one point per day (R-1149); an empty partition is not „data" (R-1093) (2026-10-19)
|
||||
|
||||
Released 2026-10-19 on the operator's yes (02:48 answers, D17 „one per day, 3 days"). sha256
|
||||
`7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1`, config bundle
|
||||
`6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f`. The vouch is the operator's.
|
||||
|
||||
- **R-1093 — an EMPTY partition no longer asks the typed „data will be erased" confirmation.** `lsblk` prints the
|
||||
parent disk's table type (PTTYPE) on every partition, so `SudoHostOps.InspectDevice` read a blank partition as
|
||||
@@ -91,6 +232,12 @@ Committed, not released — the release and its vouch are the operator's.
|
||||
with keep-last 3 the local tier's three points can still be replaced within one day (3 per day is the schedule's own
|
||||
pace plus one press); the hub's event text reads „agent privileged capability degraded" (the type is generic). No
|
||||
sudoers or bundle change.
|
||||
- **CI workflow (R-1135, no code change).** `.gitea/workflows/gates.yml`, the alarm step: the mail's run link is now `https://gitea.dooplex.hu/<repo>/actions/runs/<run id>`.
|
||||
It used `GITHUB_SERVER_URL`, which on this runner is the in-cluster address (`http://gitea.gitea-system.svc.cluster.local:3000`,
|
||||
the address the runner registers at), so nobody outside the cluster could open it. It also used the run NUMBER, and
|
||||
Gitea 1.26 reads `/actions/runs/<n>` as a run id first: a number that is also an older run's id opened that other run
|
||||
(measured 2026-10-18: `felhom.eu/actions/runs/1116` opened run id 1116, run number 613). It now uses `GITHUB_RUN_ID` and
|
||||
prints the link (`ALARM-LINK …`) into the job log. Same change in all four repos' workflows.
|
||||
|
||||
## v0.159.2 — the orphan thin check runs only after a failed backup (R-1012 follow-up) (2026-10-18)
|
||||
|
||||
|
||||
+19
@@ -1,5 +1,24 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-22/23 night — D30 on `main` (`ae69ad4`), unreleased; the D4 patches v2 ready.** Operator ruling D30
|
||||
> (2026-10-22): a handshake stamped in the future for 15 min → a new WG key pair (`wgtunnel/rekey.go`; old key kept as
|
||||
> `wg/private.key.prev`), registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the
|
||||
> key on ep0). Live on a test box (felhom.eu `audits/night-2026-10-23/P2-D30/`): with ep0 free to start a handshake the
|
||||
> tunnel heals by itself (no re-key, correct); with ep0's starts dropped it stays dead and the agent re-keyed at 15 min.
|
||||
> The interface switches at the next desired-state fetch. Gap: a reboot forgets the future stamp (R-1223). The D4 bundle
|
||||
> patch v2 (`P3-review/felhom-agent-bundle-D4-v2.patch`): `try-reload-or-restart`, an `sshd -T` read-back reported as
|
||||
> `sshd_dropin_not_in_force`, a reload in undo. Next agent release = D30 + D4 v2 + R-1204's field, the operator vouches.
|
||||
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
|
||||
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
|
||||
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
|
||||
> `ff4e6dc5…99d3b5`, bundle `485f36c9…12c47`. R-1083 left out on purpose: an agent key for 9202 would make its
|
||||
> controller book whole-guest backups through the agent, one tier writing to ep0 — the R-1083 row says what is needed.
|
||||
> **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978,
|
||||
> R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with
|
||||
> `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn
|
||||
> positions; only those two words are held, 30-min window; a claim without it keeps the R-978 order). Vouch: version
|
||||
> 0.160.0, sha256 `7ebb547f…dfca1`, bundle `6b060901…e202f`. After it, unreleased: `4545819` (two texts). felhom.eu
|
||||
> `REPORT-day-2026-10-19.md`.
|
||||
> **2026-10-18 (night) — UNRELEASED on main: R-1093, R-978, R-973** (release + vouch: the operator's). R-1093 an empty
|
||||
> partition is blank (lsblk's PTTYPE on a partition is the parent's; set aside only on a clean blkid exit 0 with a
|
||||
> matching `PART_ENTRY_SCHEME`; an unparsable lsblk is unprobed). R-978 the escrow package reaches the hub only at the
|
||||
|
||||
@@ -1,40 +1,23 @@
|
||||
# REPORT — three agent fixes, committed and NOT released (2026-10-18 evening)
|
||||
# REPORT — night 2026-10-22 → 23: D30 on `main`, the D4 patches v2, nothing released
|
||||
|
||||
Task: the evening brief of 2026-10-18, Part 4 (felhom.eu `REPORT-evening-2026-10-18.md`). Baseline: `27bbdaa` (v0.159.2
|
||||
released). Commits: `f7b7308` + `0963bae` (R-1093), `8561dc8` (R-978), `f17e197` (R-973). CHANGELOG under „Unreleased";
|
||||
the release and its vouch are the operator's. Gates green per commit (`go build/vet/test ./...`, `agent_gates.py --fast`);
|
||||
CI job 2062 success for `f17e197`.
|
||||
**What:** the night brief's agent parts. No release, no vouch. Design read: `03-host-agent.md` (the tunnel, the key, the
|
||||
bundle), `06-offsite-connectivity.md` (the full-list push to ep0).
|
||||
|
||||
## R-1093 — an empty partition is blank
|
||||
- **D30** (`ae69ad4`, hardened `39fb9cb`, `b9b831f`): a handshake stamped in the future for 15 min → a new key pair,
|
||||
registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the key on ep0 — nothing new
|
||||
on ep0). The key file is replaced by an fsynced rename over it (never missing or empty); the first old key stays
|
||||
`private.key.prev`, later ones dated (3 kept); the once-an-hour bound survives a restart; a wait stamp dated after now
|
||||
is dropped. Tests `TestD30_*` (7), each red-proved.
|
||||
- **Live** (felhom.eu `audits/night-2026-10-23/P2-D30/`, test box Tester4-16fdef, fixture `ae69ad4`): with ep0 free to
|
||||
start a handshake, a 2-hour clock jump healed by itself and no re-key ran (correct); with ep0's starts dropped on the
|
||||
box, the tunnel stayed dead, the agent re-keyed at 15 min, the hub took the key (`existed=false generation=3
|
||||
sync=ok`), the box switched at its next desired-state fetch and shook hands with the new key.
|
||||
- **D4 patches v2** (felhom.eu `audits/night-2026-10-23/P3-review/`), after a fresh review (A1–A7): `try-reload-or-restart`,
|
||||
an `sshd -T` read-back reported as `sshd_dropin_not_in_force`, a reload in undo, the right citation; installer: a warning
|
||||
when the reload fails. RootKeyOnly 6/6 (two new, red-proved); installer tests 40/40 with the patch. Both apply to
|
||||
tonight's mains. Live by hand on the test box: root password SSH refused, key works, web password 200.
|
||||
- **Gap filed:** a reboot after the clock was corrected forgets the future stamp (felhom.eu R-1223).
|
||||
|
||||
lsblk reports the PARENT disk's PTTYPE for a partition. `InspectDevice` now ignores that PTTYPE only when `blkid -p` on the
|
||||
partition exited 0, printed `PART_ENTRY_NUMBER`, and its `PART_ENTRY_SCHEME` equals lsblk's PTTYPE. Every error path
|
||||
stays data-bearing: blkid exit 2 / I/O error / missing binary keep „device has a partition table"; an lsblk answer that
|
||||
exits 0 but does not parse or names no device is „could not be reliably inspected" (an older fail-open, closed here after
|
||||
a background security review flagged the probe). No new sudo command, no config bundle.
|
||||
Test `TestInspect_R1093_PartitionVerdicts` (14 rows); three red-proofs. **Live on demo-hp** (a 1 GB loop file in `/root`,
|
||||
removed after; felhom.eu `audits/agent-fixes-2026-10-18/r1093-loop-check.txt`): the empty GPT partition → lsblk
|
||||
`pttype "gpt"` (the bug's input), blkid rc 0 with only `PART_ENTRY_*` (the fix's condition holds); ext4 → `TYPE=ext4`; a
|
||||
dos table inside the partition → `PTTYPE=dos`. Not walked: the whole `/api/storage/init` call (the agent refuses loop
|
||||
devices; it needs a VM with a spare disk running a build of this commit).
|
||||
|
||||
## R-978 — the new escrow package reaches the hub only at the claim
|
||||
|
||||
A ceremony started again no longer replaces the hub's package at create time. The package travels with the pending
|
||||
ceremony and is uploaded when the code is claimed, before the code is handed out; an upload failure is 502 and the code
|
||||
stays claimable; a ceremony abandoned by TTL, a new start or a restart leaves the first code opening the hub's package.
|
||||
Ceremony output is version 2 (`PendingUpload`); the sudo argv is unchanged. Tests `TestEscrowCeremony_R978_*` and three
|
||||
`cmd` tests; four red-proofs. **Residual (filed):** the typed-back check is the browser's, so a household that opens the
|
||||
new code and closes the page without writing it down has still replaced the package.
|
||||
|
||||
## R-973 — a cadence cap on `POST /backup`
|
||||
|
||||
Per guest and tier, successful runs only: the daily (local) tier at most **3 per rolling 24 h** (the night run, one
|
||||
„Mentés most", the next night — `07` §6; since R-899 a press never costs the night); the longer tier at most one per
|
||||
max(cadence − 24 h, cadence / 2) = **6 days** for the weekly off-site tier. A refusal is 429 with the next allowed time,
|
||||
WARN-logged, kept in `backup-success-state.json` (survives a restart), and for 24 h the host report carries the CRITICAL
|
||||
degraded status `localapi:backup-cadence-cap` (the hub's operator-only `agent_capability_degraded`). Tests `TestBackupCap_*`
|
||||
and others; three red-proofs. **Limits (filed):** keep-last 3 lets three local points be replaced within one day; the
|
||||
off-site namespace's keep-last 2 is a count — a time-bucketed prune on ep0 is the operator's.
|
||||
|
||||
*Decided by CC unattended — operator may reverse:* the numbers 3 / 24 h and 6 days, and counting successes only.
|
||||
CI by commit: `ae69ad4` 2229, `39fb9cb` 2235 green; `b9b831f` and this docs commit: see felhom.eu
|
||||
`REPORT-night-2026-10-23.md`. Fixture names on the test box: `0.163.0-night.1` (= `ae69ad4`), then `0.162.99`
|
||||
(= `b9b831f`; a plain number, because the controller cannot compare a pre-release string — the off-site unlock refused).
|
||||
|
||||
@@ -409,7 +409,7 @@ const backupCapStatusName = "localapi:backup-cadence-cap"
|
||||
|
||||
// backupCapStatus turns the local-API backup cadence cap's newest refusal (R-973) into a CRITICAL
|
||||
// capability status: degraded for 24 h after a refusal, so the hub raises its operator-only
|
||||
// agent_capability_degraded event (no hub change). The Reason names the guest, the tier and the time, and
|
||||
// whole_guest_backup_refused event (hub v0.159.0, R-1150; agent_capability_degraded before). The Reason names the guest, the tier and the time, and
|
||||
// says what to do. nil source (local API off) → ok.
|
||||
func backupCapStatus(refusal func() (string, int, time.Time, bool)) capability.Status {
|
||||
s := capability.Status{
|
||||
@@ -856,7 +856,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
log: logger,
|
||||
}
|
||||
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
|
||||
// existing agent_capability_degraded event (operator-only) tells the operator. Late-bound like
|
||||
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
|
||||
// drConfigured: the local API server is built further down, before the loop starts.
|
||||
var backupCapRefusal func() (string, int, time.Time, bool)
|
||||
probeAll := func(ctx context.Context) []capability.Status {
|
||||
@@ -936,6 +936,22 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// network and stays under the hub's 45-minute host_stale (its
|
||||
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
|
||||
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
|
||||
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
|
||||
// after start so the guest has settled; a failure is logged and the next start retries.
|
||||
go func() {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(time.Minute):
|
||||
}
|
||||
sctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
|
||||
defer cancel()
|
||||
if n, err := localapi.EnsureGuestSwap(sctx, px, reconcile.DefaultPool, logger); err != nil {
|
||||
logger.Warn("guest-swap: check skipped", "err", err)
|
||||
} else {
|
||||
logger.Info("guest-swap: checked", "raised", n)
|
||||
}
|
||||
}()
|
||||
|
||||
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
|
||||
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
|
||||
|
||||
+41
-3
@@ -165,7 +165,8 @@ KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
||||
# dest: (source file under configs/, mode, check, policy)
|
||||
# check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
|
||||
# unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
|
||||
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none.
|
||||
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none;
|
||||
# sshd → `sshd -t -f` on the snippet alone (R-982).
|
||||
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
|
||||
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
|
||||
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
|
||||
@@ -184,6 +185,9 @@ BUNDLE_KEEP = 3
|
||||
BUNDLE_MAX_BYTES = 4 * 1024 * 1024
|
||||
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
|
||||
OOB_DIR = "/etc/felhom-sshd"
|
||||
# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
|
||||
SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||
|
||||
BUNDLE_FILES = [
|
||||
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
|
||||
@@ -221,6 +225,9 @@ BUNDLE_FILES = [
|
||||
("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
|
||||
("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
|
||||
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
|
||||
# R-982, operator ruling D4 (2026-10-22): the host's own sshd (port 22, the LAN): root by key only. The console
|
||||
# password stays; felhom-sshd (the operator's door) reads nothing from sshd_config.d. Reloaded in after_install.
|
||||
("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "replace"),
|
||||
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
|
||||
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
|
||||
]
|
||||
@@ -434,7 +441,8 @@ class Runner:
|
||||
def check_content(self, kind, data):
|
||||
"""Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
|
||||
import tempfile
|
||||
cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"]}[kind]
|
||||
cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"],
|
||||
"sshd": ["/usr/sbin/sshd", "-t", "-f"]}[kind]
|
||||
with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
|
||||
p = os.path.join(d, "f")
|
||||
with open(p, "wb") as f:
|
||||
@@ -1864,6 +1872,7 @@ class Bundle:
|
||||
def __init__(self, apply):
|
||||
self.a, self.r = apply, apply.r
|
||||
self.report = apply.report
|
||||
self.sshd_dropin_not_in_force = False
|
||||
|
||||
# ---------- reading and checking ----------
|
||||
def load_file(self, path):
|
||||
@@ -1932,10 +1941,17 @@ class Bundle:
|
||||
text = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
raise Refused("R18", f"{dest} is not UTF-8 text")
|
||||
if check in ("sudoers", "sh", "bash", "nft"):
|
||||
if check in ("sudoers", "sh", "bash", "nft", "sshd"):
|
||||
rc, msg = self.r.check_content(check, content)
|
||||
if rc != 0:
|
||||
raise Refused("R18", f"{dest} fails its {check} check: {msg}")
|
||||
if check == "sshd":
|
||||
# R-982: the drop-in's PermitRootLogin must be exactly `prohibit-password`. `no` would also shut the root key
|
||||
# out (Proxmox's own node-to-node SSH and the LAN key route), `yes` would undo the ruling — both refused.
|
||||
# Only that line is pinned: the file is signed like every bundle file (review A7).
|
||||
vals = re.findall(r"^\s*PermitRootLogin\s+(\S+)\s*$", text, re.M | re.I)
|
||||
if vals != ["prohibit-password"]:
|
||||
raise Refused("R18", f"{dest} must set exactly `PermitRootLogin prohibit-password` (found {vals})")
|
||||
elif check == "python":
|
||||
try:
|
||||
compile(text, dest, "exec")
|
||||
@@ -2044,6 +2060,8 @@ class Bundle:
|
||||
self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
|
||||
return 3
|
||||
rep["signers_created"] = False
|
||||
if self.sshd_dropin_not_in_force:
|
||||
rep["sshd_dropin_not_in_force"] = True # R-982 A2: written and reloaded, but sshd -T shows root passwords allowed
|
||||
if bootstrap and not self.r.lexists(TRUST_SIGNERS):
|
||||
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
|
||||
rep["signers_created"] = True
|
||||
@@ -2073,6 +2091,23 @@ class Bundle:
|
||||
"enable the crash guard")
|
||||
if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
|
||||
self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
|
||||
# R-982: the host's sshd takes the root-key-only drop-in. The WHOLE config is tested first (a failure is a
|
||||
# refusal, and the bundle undo puts the old files back); `try-reload-or-restart` keeps every open session and
|
||||
# does nothing on a box whose ssh.service is not running (a socket-activated or stopped sshd reads the file
|
||||
# at its next start) — a failure there must not freeze the bundle route (review 2026-10-22 night, A1).
|
||||
# felhom-sshd is not touched. Then the POSITIVE check: sshd's own effective view for root. A box whose
|
||||
# sshd_config has no Include of sshd_config.d, an earlier drop-in or a Match block can keep the password route
|
||||
# — that is said loudly in the log and the report, never silently recorded as done (A2).
|
||||
if SSHD_DROPIN in written:
|
||||
self.must(["/usr/sbin/sshd", "-t"], "sshd -t (the host's whole sshd config)")
|
||||
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
|
||||
rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", "user=root,host=felhom-check,addr=192.0.2.1"], 60)
|
||||
eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
|
||||
if rc != 0 or eff != ["without-password"]:
|
||||
self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force: sshd -T says "
|
||||
f"permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier drop-in "
|
||||
f"or a Match block (R-982)")
|
||||
self.sshd_dropin_not_in_force = True
|
||||
|
||||
def must(self, argv, what):
|
||||
rc, out, err = self.r.host(argv, 120)
|
||||
@@ -2133,6 +2168,9 @@ class Bundle:
|
||||
self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
|
||||
if any(d.startswith("/etc/systemd/system/") for d in back):
|
||||
self.r.host(["systemctl", "daemon-reload"], 120)
|
||||
if SSHD_DROPIN in back:
|
||||
# A3: after_install may already have reloaded sshd with the drop-in; the restored state takes effect now.
|
||||
self.r.host(["systemctl", "try-reload-or-restart", "ssh"], 120)
|
||||
rc, _, _ = self.r.host(["visudo", "-c"], 60)
|
||||
self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
|
||||
return back
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
# MANAGED BY FELHOM (the agent's config bundle) — R-982, operator ruling D4 (2026-10-22).
|
||||
# The Proxmox host's OWN sshd — the listener the household network reaches on port 22: root logs in with a KEY only.
|
||||
# The root password still works on the console (the break-glass password: felhom.eu runbooks/break-glass.md). The
|
||||
# operator's door (felhom-sshd, port 8822, /etc/felhom-sshd/sshd_config) is a separate daemon whose config includes
|
||||
# nothing from here, so it is not changed. Debian's sshd_config reads this folder BEFORE its own settings, and for
|
||||
# sshd the first value wins; the bundle checks the effective value with `sshd -T` after the reload.
|
||||
PermitRootLogin prohibit-password
|
||||
@@ -429,6 +429,90 @@ class SelfCheckUndo(unittest.TestCase):
|
||||
self.assert_restored(box, before, rep)
|
||||
|
||||
|
||||
|
||||
class RootKeyOnly(unittest.TestCase):
|
||||
"""R-982, operator ruling D4 (2026-10-22): the bundle gives the host's own sshd a root-key-only drop-in, tests the
|
||||
whole config and reloads sshd — only when the file was written; a drop-in that says anything but
|
||||
`prohibit-password` is refused before anything is written.
|
||||
RED-PROOF: with the after_install block removed, test_fresh_box_takes_the_dropin_and_reloads fails (no reload);
|
||||
with the content rule removed, test_a_dropin_that_says_yes_or_no_is_refused fails — seen failing 2026-10-22."""
|
||||
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||
|
||||
def test_fresh_box_takes_the_dropin_and_reloads(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertIn(b"PermitRootLogin prohibit-password", box.files[self.DEST])
|
||||
self.assertEqual(box.modes[self.DEST], 0o644)
|
||||
self.assertIn(["/usr/sbin/sshd", "-t"], box.calls)
|
||||
self.assertIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
|
||||
self.assertFalse(any("felhom-sshd" in " ".join(c) for c in box.calls), "the operator's door must not be touched")
|
||||
|
||||
def test_an_unchanged_dropin_is_not_reloaded(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.put(self.DEST, (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes(), 0o644)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||
|
||||
def test_a_dropin_that_says_yes_or_no_is_refused(self):
|
||||
for bad in (b"PermitRootLogin yes\n", b"PermitRootLogin no\n", b"# nothing\n",
|
||||
b"PermitRootLogin prohibit-password\nPermitRootLogin yes\n"):
|
||||
def edit(files, bad=bad):
|
||||
for e in files:
|
||||
if e["path"] == self.DEST:
|
||||
e["content_b64"] = base64.b64encode(bad).decode()
|
||||
data, sha = edited_bundle(edit)
|
||||
box = Box(data, signed_job(sha))
|
||||
rc, rep = run(box)
|
||||
self.assertNotEqual(rc, 0, bad)
|
||||
self.assertNotIn(self.DEST, box.files, bad)
|
||||
|
||||
# Review 2026-10-22 night. A2: the effective value is read back with `sshd -T`; a drop-in that is not in force is
|
||||
# said in the log and the report (not refused — a refusal would freeze the bundle route). A3: a bundle undone
|
||||
# after the reload reloads sshd again with the old state. RED-PROOF: with the sshd -T check removed,
|
||||
# test_a_dropin_not_in_force_is_reported fails; with the undo reload removed, test_an_undone_bundle_reloads_sshd
|
||||
# fails — seen failing 2026-10-22 night.
|
||||
def _sshd_T(self, box, value):
|
||||
orig = box.host
|
||||
def host(argv, timeout=600, stdin=None):
|
||||
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
|
||||
box.calls.append(argv)
|
||||
return 0, f"port 22\npermitrootlogin {value}\npasswordauthentication yes\n", ""
|
||||
return orig(argv, timeout, stdin)
|
||||
box.host = host
|
||||
|
||||
def test_the_effective_value_is_checked(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
self._sshd_T(box, "without-password")
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertTrue(any(c[:2] == ["/usr/sbin/sshd", "-T"] for c in box.calls))
|
||||
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
|
||||
|
||||
def test_a_dropin_not_in_force_is_reported(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
self._sshd_T(box, "yes") # e.g. no Include of sshd_config.d
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
|
||||
self.assertTrue(any("NOT in force" in l for l in box.logs), box.logs)
|
||||
|
||||
def test_an_undone_bundle_reloads_sshd(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.visudo_fail = True # the self-check fails AFTER the install → undo
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 3, rep)
|
||||
self.assertNotIn(self.DEST, box.files)
|
||||
reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
|
||||
self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
|
||||
|
||||
class TrustBootstrap(unittest.TestCase):
|
||||
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
|
||||
data, sha = real_bundle()
|
||||
|
||||
@@ -2,9 +2,9 @@ package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"testing"
|
||||
|
||||
@@ -48,7 +48,7 @@ func TestR689_GuestBackupArchiveShapes(t *testing.T) {
|
||||
{proxmox.StorageContent{VolID: "felhom-pbs:backup/vm/300/2026-07-28T05:31:14Z", VMID: 300}, true},
|
||||
{proxmox.StorageContent{VolID: "local:backup/felhom-golden-0.236.0.tar.zst"}, false},
|
||||
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", VMID: 9202}, false}, // vmid disagrees with the name
|
||||
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
|
||||
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
|
||||
} {
|
||||
if ok, why := guestBackupArchive(c.e); ok != c.ok {
|
||||
t.Errorf("%s vmid=%d: ok=%v (%s), want %v", c.e.VolID, c.e.VMID, ok, why, c.ok)
|
||||
|
||||
@@ -47,7 +47,7 @@ type Runner interface {
|
||||
// downgrades a HEALTHY probe to StatusInactive (broken plumbing stays degraded regardless).
|
||||
type Prober struct {
|
||||
Runner Runner
|
||||
Exists func(path string) bool // nil → os.Stat
|
||||
Exists func(path string) bool // nil → os.Stat
|
||||
GateActive func(gate string) bool // nil → every gate treated active
|
||||
}
|
||||
|
||||
|
||||
@@ -21,10 +21,10 @@ import (
|
||||
// The four inputs (the consumption contract — spike findings):
|
||||
// - blob: the opaque R-wrapped escrow blob (hub-served restore directive in 10D).
|
||||
// - recoveryCode (R): BY HAND from the customer — NEVER hub-sourced (zero-knowledge holds: a
|
||||
// hub compromise alone cannot decrypt). A secret: never logged/persisted.
|
||||
// hub compromise alone cannot decrypt). A secret: never logged/persisted.
|
||||
// - expectedFingerprint: hub-served — the gate target (F-C4).
|
||||
// - keyDest: where the PBS restore reads the key (`--keyfile` or the default
|
||||
// `$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json`).
|
||||
// `$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json`).
|
||||
//
|
||||
// Order (spike F-C2/F-C3/F-C4/F-C6): Unwrap → fingerprint-gate → install. On ANY failure there is
|
||||
// NO partial install (nothing at keyDest) and no key material leaks: the recovered key lives only
|
||||
|
||||
@@ -22,6 +22,11 @@ type Loop struct {
|
||||
mu sync.Mutex
|
||||
fetched bool // a desired-state document has been delivered (false = never a teardown signal)
|
||||
block *hub.WireWireguard
|
||||
// passed (R-1194) is set once the first reconcile pass has run. Until then the instance has never been rendered,
|
||||
// so its health says nothing — and the first host report, sent before that pass, used to read „config invalid,
|
||||
// port 0" and raise a false oob_degraded on EVERY fresh box (rehearsal 2026-10-20; reproduced 2026-10-21: the
|
||||
// report at 20:34:00 read an absent config, the apply landed 0.4 s later).
|
||||
passed bool
|
||||
|
||||
nudge chan struct{}
|
||||
}
|
||||
@@ -85,6 +90,7 @@ func (l *Loop) reconcile(ctx context.Context) {
|
||||
if !fetched {
|
||||
applyBlock = nil
|
||||
}
|
||||
defer l.markPassed()
|
||||
port, err := l.mgr.Apply(ctx, applyBlock)
|
||||
if err != nil {
|
||||
return // Apply logged; a claim/exhaustion or install error — retry next tick
|
||||
@@ -106,8 +112,23 @@ func oobPeerIP(block *hub.WireWireguard) string {
|
||||
return block.OOBPeerIP
|
||||
}
|
||||
|
||||
// OOBStatus implements the hub collector's reporter seam (Part 4).
|
||||
// markPassed records that a reconcile pass has run (R-1194) — after a failed Apply too, so a real, lasting failure
|
||||
// is still reported from the next report on.
|
||||
func (l *Loop) markPassed() {
|
||||
l.mu.Lock()
|
||||
l.passed = true
|
||||
l.mu.Unlock()
|
||||
}
|
||||
|
||||
// OOBStatus implements the hub collector's reporter seam (Part 4). nil — no stanza, which the hub does not evaluate —
|
||||
// until the first reconcile pass has run (R-1194).
|
||||
func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus {
|
||||
l.mu.Lock()
|
||||
passed := l.passed
|
||||
l.mu.Unlock()
|
||||
if !passed {
|
||||
return nil
|
||||
}
|
||||
_, block := l.snapshot()
|
||||
return l.mgr.Status(ctx, block)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
package felhomsshd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1194: on every fresh box the first host report — sent before the OOB loop's first pass — read „config invalid,
|
||||
// port 0" (the config was not rendered yet) and the hub mailed oob_degraded. Reproduced 2026-10-21 on a fresh box
|
||||
// (felhom.eu audits/night-2026-10-21/P5-checks/): the report at 20:34:00, the apply 0.4 s later, the daemon healthy.
|
||||
// The loop now reports NO stanza (the hub does not evaluate a missing one) until a pass has run — a failed pass
|
||||
// counts, so a real lasting failure is still reported from the next report on.
|
||||
// RED-PROOF: with OOBStatus not gated on `passed`, the first assertion fails (a stanza with config_invalid).
|
||||
func TestR1194_NoOOBStanzaBeforeTheFirstPass(t *testing.T) {
|
||||
r := newScriptRunner()
|
||||
m := NewManager(r, t.TempDir(), nil)
|
||||
m.isActive = func(context.Context) bool { return false }
|
||||
m.isFailed = func(context.Context) bool { return false }
|
||||
m.now = func() time.Time { return time.Unix(1783270000, 0) }
|
||||
l := NewLoop(m, nil, time.Minute, nil)
|
||||
if st := l.OOBStatus(context.Background()); st != nil {
|
||||
t.Fatalf("a stanza before the first pass reads a config that was never rendered: %+v", st)
|
||||
}
|
||||
l.reconcile(context.Background()) // fails or succeeds — either way a pass has run
|
||||
if st := l.OOBStatus(context.Background()); st == nil {
|
||||
t.Fatal("after the first pass the health must be reported (a real failure must reach the hub)")
|
||||
}
|
||||
}
|
||||
@@ -50,8 +50,8 @@ func TestParseConfMounts(t *testing.T) {
|
||||
func TestMissingBindSources(t *testing.T) {
|
||||
mounts := ParseConfMounts(sampleConf)
|
||||
present := map[string]bool{
|
||||
"/mnt/felhom-usb/felhom-data": true, // usb attached
|
||||
"/var/lib/felhom-agent/guests/9201/bootstrap": true, // bootstrap always present
|
||||
"/mnt/felhom-usb/felhom-data": true, // usb attached
|
||||
"/var/lib/felhom-agent/guests/9201/bootstrap": true, // bootstrap always present
|
||||
// "/mnt/felhom-flash/felhom-data" is ABSENT (drive unplugged)
|
||||
}
|
||||
got := MissingBindSources(mounts, func(p string) bool { return present[p] })
|
||||
|
||||
@@ -14,6 +14,13 @@ const (
|
||||
TunnelRunning = "running" // the cloudflared container runs AND its readiness check says CONNECTED
|
||||
TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which)
|
||||
TunnelUnknown = "unknown" // the probe could not ask (guest down, pct/sudo error, health still starting)
|
||||
// TunnelAbsent (R-1070, agent Unreleased 2026-10-19): the guest answered and has NO cloudflared container — the
|
||||
// controller makes none when no tunnel token is configured (a LAN-only or test box). Not a failure by itself: the
|
||||
// hub never alarms on it, and the OS-update host rule reads it as „not applicable" only when the container was
|
||||
// also absent before the update (HostHealthVerdict). Docker 29 prints „error: no such object" in lower case, so
|
||||
// before this state existed the case-sensitive match fell through to `unknown` — which failed the host rule on
|
||||
// every box without a tunnel (Tester4, 2026-10-15).
|
||||
TunnelAbsent = "absent"
|
||||
)
|
||||
|
||||
// CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec.
|
||||
@@ -53,7 +60,7 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
|
||||
return TunnelUnknown, "no running customer guest"
|
||||
}
|
||||
worst, wdetail := "", ""
|
||||
rank := map[string]int{TunnelRunning: 0, TunnelUnknown: 1, TunnelNotRunning: 2}
|
||||
rank := map[string]int{TunnelRunning: 0, TunnelAbsent: 1, TunnelUnknown: 2, TunnelNotRunning: 3}
|
||||
for _, v := range vmids {
|
||||
out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared")
|
||||
st, d := ClassifyTunnel(string(out), string(errOut), err)
|
||||
@@ -71,8 +78,8 @@ func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
|
||||
func ClassifyTunnel(stdout, stderr string, err error) (string, string) {
|
||||
out := strings.TrimSpace(stdout)
|
||||
if err != nil || out == "" {
|
||||
if strings.Contains(stderr, "No such object") || strings.Contains(stderr, "No such container") {
|
||||
return TunnelNotRunning, "no cloudflared container in the guest"
|
||||
if e := strings.ToLower(stderr); strings.Contains(e, "no such object") || strings.Contains(e, "no such container") {
|
||||
return TunnelAbsent, "no cloudflared container in the guest (no tunnel set up on this box)"
|
||||
}
|
||||
return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err)
|
||||
}
|
||||
|
||||
@@ -20,7 +20,10 @@ func TestClassifyTunnel(t *testing.T) {
|
||||
{"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"},
|
||||
{"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"},
|
||||
{"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"},
|
||||
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelNotRunning, "no cloudflared container"},
|
||||
{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
|
||||
// R-1070: Docker 29's wording, measured on demo-hp 2026-10-19 (`docker inspect` of a missing name).
|
||||
{"absent, docker 29", "", "error: no such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no tunnel set up"},
|
||||
{"absent, no such container", "", "Error response from daemon: No such container: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no cloudflared container"},
|
||||
{"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"},
|
||||
{"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"},
|
||||
{"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"},
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
|
||||
@@ -136,6 +137,8 @@ type Collector struct {
|
||||
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// swapShortWarned (R-1192): the host-swap-short WARN is logged once per state change, not every report.
|
||||
swapShortWarned atomic.Bool
|
||||
}
|
||||
|
||||
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
|
||||
@@ -375,6 +378,8 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
}
|
||||
|
||||
host := hostMetrics(c.px.Node(), ns)
|
||||
guests := c.collectGuests(ctx)
|
||||
c.applySwap(&host, ns, guests) // R-1192: host swap vs the guests' swap allowance
|
||||
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
|
||||
host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable
|
||||
report := &HostReport{
|
||||
@@ -383,7 +388,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
AgentVersion: c.agentVersion,
|
||||
AgentSHA256: c.agentSHA256(),
|
||||
Host: host,
|
||||
Guests: c.collectGuests(ctx),
|
||||
Guests: guests,
|
||||
// storage_targets populated this slice (slice 5) via the observer; the rest stay
|
||||
// defined-but-empty (slice 6). Non-nil so they marshal as [].
|
||||
StorageTargets: c.collectStorage(ctx),
|
||||
@@ -567,6 +572,7 @@ func (c *Collector) collectGuests(ctx context.Context) []Guest {
|
||||
Cores: cfg.Cores,
|
||||
MemoryBytes: g.MaxMem,
|
||||
DiskBytes: g.MaxDisk,
|
||||
SwapBytes: cfg.Swap * (1 << 20),
|
||||
}
|
||||
}
|
||||
guests = append(guests, entry)
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1192 follow-up (night 2026-10-21): the guest swap fix (every guest may swap as much as its RAM,
|
||||
// localapi/guestswap.go) moves a memory burst onto the HOST's swap. Measured on demo-hp the same day: our installer
|
||||
// gives an 8 GB / 256 GB machine 7.75 GB of host swap, so a 4096 MB guest's 4096 MB swap fits. A host with no swap
|
||||
// (a BYO install, or swap turned off) gets nothing from the fix. So the report carries both numbers, the agent says it
|
||||
// once in its log, and the hub raises host_swap_short. The agent never creates host swap itself: changing a host its
|
||||
// owner set up is a decision (felhom.eu register).
|
||||
|
||||
// HostSwapShort reports whether the host's swap is below the guests' swap allowance. A host whose guests have no
|
||||
// swap allowance (0) is never short.
|
||||
func HostSwapShort(hostMB, guestMB int64) bool { return guestMB > 0 && hostMB < guestMB }
|
||||
|
||||
// applySwap fills the host block's swap numbers from the node status and the guests' specs, and logs the short
|
||||
// state once per change.
|
||||
func (c *Collector) applySwap(h *HostMetrics, ns proxmox.NodeStatus, guests []Guest) {
|
||||
hostMB := ns.Swap.Total / (1 << 20)
|
||||
var guestMB int64
|
||||
for _, g := range guests {
|
||||
if g.Spec != nil {
|
||||
guestMB += g.Spec.SwapBytes / (1 << 20)
|
||||
}
|
||||
}
|
||||
h.HostSwapMB, h.GuestSwapMB = &hostMB, &guestMB
|
||||
if HostSwapShort(hostMB, guestMB) {
|
||||
if c.swapShortWarned.CompareAndSwap(false, true) {
|
||||
c.logger.Warn("host-swap: the host's swap is below its guests' swap allowance — a memory burst can freeze the box (R-1192); the hub is told (host_swap_short)",
|
||||
"host_swap_mb", hostMB, "guest_swap_mb", guestMB)
|
||||
}
|
||||
return
|
||||
}
|
||||
if c.swapShortWarned.CompareAndSwap(true, false) {
|
||||
c.logger.Info("host-swap: the host's swap covers its guests' swap allowance again", "host_swap_mb", hostMB, "guest_swap_mb", guestMB)
|
||||
}
|
||||
}
|
||||
@@ -77,10 +77,10 @@ func TestLoop_FailedTailPushIsReArmedByNextEnvelope(t *testing.T) {
|
||||
env *ControlEnvelope
|
||||
err error
|
||||
}{
|
||||
{env: &ControlEnvelope{LogTailRequested: true}}, // arm
|
||||
{err: errors.New("hub 5xx")}, // the carrying push FAILS
|
||||
{env: &ControlEnvelope{LogTailRequested: true}}, // hub still pending → re-arm
|
||||
{env: &ControlEnvelope{}}, // fulfilled
|
||||
{env: &ControlEnvelope{LogTailRequested: true}}, // arm
|
||||
{err: errors.New("hub 5xx")}, // the carrying push FAILS
|
||||
{env: &ControlEnvelope{LogTailRequested: true}}, // hub still pending → re-arm
|
||||
{env: &ControlEnvelope{}}, // fulfilled
|
||||
}}
|
||||
l := tailLoop(rep)
|
||||
ctx := context.Background()
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1192 follow-up (night 2026-10-21): the host report carries the host's swap and the guests' swap allowance, and the
|
||||
// agent says ONCE in its log when the host's swap is below it (and once when it is covered again).
|
||||
// RED-PROOF: with applySwap not called the numbers are nil; with the CompareAndSwap removed the WARN repeats.
|
||||
func TestR1192_HostReportCarriesSwapAndWarnsOnce(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
c := &Collector{logger: slog.New(slog.NewTextHandler(&buf, nil))}
|
||||
guests := []Guest{{VMID: 9201, Spec: &GuestSpec{SwapBytes: 4096 << 20}}, {VMID: 9202, Spec: nil}}
|
||||
var ns proxmox.NodeStatus // a host with no swap
|
||||
for i := 0; i < 3; i++ {
|
||||
h := HostMetrics{}
|
||||
c.applySwap(&h, ns, guests)
|
||||
if h.HostSwapMB == nil || h.GuestSwapMB == nil || *h.HostSwapMB != 0 || *h.GuestSwapMB != 4096 {
|
||||
t.Fatalf("report numbers: host %v guest %v", h.HostSwapMB, h.GuestSwapMB)
|
||||
}
|
||||
}
|
||||
if n := strings.Count(buf.String(), "host-swap: the host's swap is below"); n != 1 {
|
||||
t.Fatalf("the short WARN must be logged once, got %d:\n%s", n, buf.String())
|
||||
}
|
||||
ns.Swap.Total = 7935 << 20 // the installer's 8 GB host
|
||||
h := HostMetrics{}
|
||||
c.applySwap(&h, ns, guests)
|
||||
if *h.HostSwapMB != 7935 || HostSwapShort(*h.HostSwapMB, *h.GuestSwapMB) {
|
||||
t.Fatalf("a covered host must not be short: %v / %v", *h.HostSwapMB, *h.GuestSwapMB)
|
||||
}
|
||||
if !strings.Contains(buf.String(), "covers its guests' swap allowance again") {
|
||||
t.Fatalf("the recovery must be said once:\n%s", buf.String())
|
||||
}
|
||||
if HostSwapShort(0, 0) {
|
||||
t.Fatal("a host whose guests have no swap allowance is never short")
|
||||
}
|
||||
}
|
||||
+12
-2
@@ -292,7 +292,11 @@ type WireguardStatus struct {
|
||||
Registered bool `json:"registered"` // the registration marker exists
|
||||
Active bool `json:"active"` // wg-quick@wg-felhom is-active
|
||||
LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"`
|
||||
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
|
||||
// HandshakeFutureS (R-1204): seconds the latest handshake lies in the FUTURE by this box's clock — set (> 0) only
|
||||
// then. A box that ran once with its clock ahead keeps a handshake stamped in the future; WireGuard on ep0 then
|
||||
// refuses every newer handshake, and LastHandshakeAgeS (clamped at 0) reads „just now" for ever. The hub alarms on it.
|
||||
HandshakeFutureS *int64 `json:"handshake_future_s,omitempty"`
|
||||
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
|
||||
}
|
||||
|
||||
// SystemInfo is the `system` stanza (see HostReport.System).
|
||||
@@ -338,6 +342,11 @@ type HostMetrics struct {
|
||||
//
|
||||
// Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift.
|
||||
WrapperSHA256 string `json:"wrapper_sha256,omitempty"`
|
||||
// HostSwapMB / GuestSwapMB (R-1192 follow-up, night 2026-10-21): the host's swap (Proxmox node status — the
|
||||
// kernel's SwapTotal) and the sum of its guests' swap allowance. The guest swap fix (swap = RAM) needs swap ON THE
|
||||
// HOST behind it; the hub raises host_swap_short when the first is below the second. nil = not read (older agent).
|
||||
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
|
||||
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
|
||||
}
|
||||
|
||||
// Guest is one LXC. The agent reports vmid; the hub derives the guest PK
|
||||
@@ -355,11 +364,12 @@ type GuestSpec struct {
|
||||
Cores int `json:"cores"`
|
||||
MemoryBytes int64 `json:"memory_bytes"`
|
||||
DiskBytes int64 `json:"disk_bytes"`
|
||||
SwapBytes int64 `json:"swap_bytes,omitempty"` // R-1192: the guest's swap allowance (PVE config `swap`, MB → bytes)
|
||||
}
|
||||
|
||||
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
|
||||
type Cloudflared struct {
|
||||
Status string `json:"status"` // running | not_running | unknown (TunnelRunning …)
|
||||
Status string `json:"status"` // running | not_running | unknown | absent (TunnelRunning …; absent since R-1070)
|
||||
Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected"
|
||||
}
|
||||
|
||||
|
||||
@@ -31,8 +31,8 @@ import (
|
||||
// Only SUCCESSES count — a failed run prunes nothing and must stay retryable. The count survives an agent
|
||||
// restart: it reads the success history saved on disk (LastKnownBackupStore, the R-894 file) as well as
|
||||
// this process's own record. A refusal answers 429, is logged at WARN, is saved, and raises the operator
|
||||
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's existing
|
||||
// `agent_capability_degraded` event, operator-only). Pinned by TestBackupCap_* (backup_cap_test.go).
|
||||
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's operator-only
|
||||
// `whole_guest_backup_refused` event since hub v0.159.0, R-1150). Pinned by TestBackupCap_* (backup_cap_test.go).
|
||||
|
||||
// backupCapRefusalHold is how long one refusal keeps the operator signal raised.
|
||||
const backupCapRefusalHold = 24 * time.Hour
|
||||
|
||||
@@ -369,7 +369,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
|
||||
job.Uploaded = false // R-978: uploaded by the claim
|
||||
s.finishEscrowCeremony(job, out.RecoveryCode, out.PendingUpload)
|
||||
out.RecoveryCode = "" // drop the parsed reference promptly (GC caveat: best-effort)
|
||||
s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim",
|
||||
s.logger.Info("local-api: escrow ceremony complete — R claimable (in-memory, one-shot); package uploads at the claim, or at the typed-back confirm (R-1151)",
|
||||
"job_id", job.JobID, "restic_pw_sealed", job.ResticPwSealed, "uploaded", job.Uploaded,
|
||||
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
|
||||
}()
|
||||
|
||||
@@ -57,6 +57,10 @@ type MemoryInfo struct {
|
||||
MaxMB int64 `json:"max_mb"`
|
||||
FloorMB int64 `json:"floor_mb"`
|
||||
Running bool `json:"running"`
|
||||
// HostSwapMB / GuestSwapMB (R-1192 follow-up): the host's swap and this guest's swap allowance, so the controller
|
||||
// can tell the household when the host has too little swap behind the guest (its install page). nil = not read.
|
||||
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
|
||||
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
|
||||
}
|
||||
|
||||
// memoryResizeRequest is the POST /guest/memory body.
|
||||
@@ -94,6 +98,8 @@ type memoryBounds struct {
|
||||
maxMB int64
|
||||
floorMB int64
|
||||
running bool
|
||||
hostSwapMB int64
|
||||
guestSwapMB int64
|
||||
}
|
||||
|
||||
func (b memoryBounds) refusal(code string) memoryRefusal {
|
||||
@@ -104,7 +110,9 @@ func (b memoryBounds) refusal(code string) memoryRefusal {
|
||||
}
|
||||
|
||||
func (b memoryBounds) info(vmid int) MemoryInfo {
|
||||
hostSwap, guestSwap := b.hostSwapMB, b.guestSwapMB
|
||||
return MemoryInfo{
|
||||
HostSwapMB: &hostSwap, GuestSwapMB: &guestSwap,
|
||||
VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB,
|
||||
MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running,
|
||||
}
|
||||
@@ -139,6 +147,8 @@ func (s *Server) readMemoryBounds(ctx context.Context, vmid int) (memoryBounds,
|
||||
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
|
||||
minMB: minGuestMemoryMB,
|
||||
running: st.Status == "running",
|
||||
hostSwapMB: node.Swap.Total / mib, // R-1192 follow-up: the kernel's SwapTotal, as Proxmox reports it
|
||||
guestSwapMB: cfg.Swap, // PVE config swap is MB
|
||||
}
|
||||
b.maxMB = b.hostTotalMB - hostReserveMB
|
||||
b.floorMB = b.usageMB + shrinkUsageMarginMB
|
||||
@@ -225,7 +235,8 @@ func (s *Server) handleGuestMemoryResize(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
|
||||
// Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on.
|
||||
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target)})
|
||||
// R-1192: the swap moves with the memory (GuestSwapMB) — a grown guest with the old swap cap thrashes the same way.
|
||||
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target), "swap": fmt.Sprintf("%d", GuestSwapMB(target))})
|
||||
if err != nil {
|
||||
s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err)
|
||||
writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error())
|
||||
|
||||
@@ -25,12 +25,14 @@ type fakeMemory struct {
|
||||
setUPID string
|
||||
setErr error
|
||||
applyReflect bool
|
||||
hostSwapB int64 // R-1192 follow-up: node swap total
|
||||
guestSwapMB int64
|
||||
}
|
||||
|
||||
func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return proxmox.GuestConfig{Memory: f.allocMB}, nil
|
||||
return proxmox.GuestConfig{Memory: f.allocMB, Swap: f.guestSwapMB}, nil
|
||||
}
|
||||
func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) {
|
||||
f.mu.Lock()
|
||||
@@ -42,6 +44,7 @@ func (f *fakeMemory) NodeStatus(_ context.Context) (proxmox.NodeStatus, error) {
|
||||
defer f.mu.Unlock()
|
||||
var ns proxmox.NodeStatus
|
||||
ns.Memory.Total = f.hostTotalB
|
||||
ns.Swap.Total = f.hostSwapB
|
||||
return ns, nil
|
||||
}
|
||||
func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) {
|
||||
@@ -151,6 +154,10 @@ func TestGuestMemory_GrowAndShrink(t *testing.T) {
|
||||
if f.lastParams["memory"] != "12288" {
|
||||
t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"])
|
||||
}
|
||||
// R-1192: the swap moves with the memory.
|
||||
if f.lastParams["swap"] != "12288" {
|
||||
t.Errorf("SetConfig swap = %q, want 12288 (swap = RAM)", f.lastParams["swap"])
|
||||
}
|
||||
})
|
||||
t.Run("shrink above floor", func(t *testing.T) {
|
||||
f := stdFixture() // floor = max(2048, 3000+512) = 3512
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// ── R-1192 (2026-10-21): the guest may swap as much as its RAM ────────────────────────────────────────────────────
|
||||
//
|
||||
// MEASURED (felhom.eu audits/fixes-2026-10-21/P4-freeze/): on the minimum machine (8 GB host, guest 4096 MB) the
|
||||
// household's first uploads made Immich load its analysis models (+1.15 GB) on top of 8 apps' 1.75 GB; the guest hit
|
||||
// its 4096 MB limit with its swap capped at Proxmox's LXC default of 512 MB — and full. The kernel could then evict
|
||||
// only file pages: ~360 MB was left for the program code of ~25 containers, re-read from disk without end (150 000–
|
||||
// 600 000 refaults and ~1 GB of reads per 10 s; memory „full" pressure 92–95 %, load 155, the dashboard dead, 0 OOM
|
||||
// kills). Raising the cap to 4096 MB live ended it in under a minute (memory „full" 96 % → 1.7 %, dashboard back
|
||||
// 42 s later) while the uploads ran on. The host had 7.4 GB of swap free the whole time.
|
||||
//
|
||||
// The agent never set `swap` at all, so every guest had the 512 MB default. Now:
|
||||
// - a new guest is created with swap = its memory (reconcile.bringup);
|
||||
// - a RAM resize sets swap with the memory (handleGuestMemoryResize);
|
||||
// - at agent start, every LXC in the felhom pool whose swap is below its memory is raised to it (this file).
|
||||
// Raising swap is a live cgroup change (memory.swap.max), no restart. It never lowers a swap someone set higher.
|
||||
|
||||
// GuestSwapMB is the swap a guest of memoryMB gets: the same as its RAM.
|
||||
func GuestSwapMB(memoryMB int64) int64 { return memoryMB }
|
||||
|
||||
// SwapOps is the Proxmox surface EnsureGuestSwap needs (satisfied by *proxmox.Client).
|
||||
type SwapOps interface {
|
||||
Pool(ctx context.Context, name string) (proxmox.PoolInfo, error)
|
||||
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
|
||||
SetConfig(ctx context.Context, vmid int, params map[string]string) (string, error)
|
||||
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
|
||||
}
|
||||
|
||||
// EnsureGuestSwap raises the swap of every LXC guest in pool to GuestSwapMB(memory) when it is below. Returns how
|
||||
// many guests were raised. One guest's failure is logged and the others are still handled.
|
||||
func EnsureGuestSwap(ctx context.Context, ops SwapOps, pool string, logger *slog.Logger) (int, error) {
|
||||
p, err := ops.Pool(ctx, pool)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("guest swap: read pool %s: %w", pool, err)
|
||||
}
|
||||
raised := 0
|
||||
for _, m := range p.Members {
|
||||
if m.Type != "lxc" || m.VMID <= 0 {
|
||||
continue
|
||||
}
|
||||
cfg, err := ops.GuestConfig(ctx, m.VMID)
|
||||
if err != nil {
|
||||
logger.Warn("guest-swap: config read failed (left as it is)", "vmid", m.VMID, "err", err)
|
||||
continue
|
||||
}
|
||||
want := GuestSwapMB(cfg.Memory)
|
||||
if cfg.Memory <= 0 || cfg.Swap >= want {
|
||||
logger.Debug("guest-swap: nothing to do", "vmid", m.VMID, "memory_mb", cfg.Memory, "swap_mb", cfg.Swap)
|
||||
continue
|
||||
}
|
||||
upid, err := ops.SetConfig(ctx, m.VMID, map[string]string{"swap": fmt.Sprintf("%d", want)})
|
||||
if err == nil && upid != "" {
|
||||
_, err = ops.WaitTask(ctx, upid, proxmox.WaitOptions{})
|
||||
}
|
||||
if err != nil {
|
||||
logger.Warn("guest-swap: raise failed (left as it is)", "vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want, "err", err)
|
||||
continue
|
||||
}
|
||||
raised++
|
||||
logger.Info("guest-swap: raised (R-1192 — a guest swap cap below its RAM turned a memory burst into a freeze)",
|
||||
"vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want)
|
||||
}
|
||||
return raised, nil
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1192: the freeze measured on the minimum machine (felhom.eu audits/fixes-2026-10-21/P4-freeze/) — a guest whose
|
||||
// swap is capped below its RAM evicts program code instead of idle memory. EnsureGuestSwap raises every pool guest's
|
||||
// swap to its RAM, never lowers one, skips non-LXC members, and survives one guest's failure.
|
||||
// COMPANION RED-PROOF: return 0 before the SetConfig call → „raised 0, want 2".
|
||||
type fakeSwapOps struct {
|
||||
cfg map[int]proxmox.GuestConfig
|
||||
failSet map[int]bool
|
||||
set map[int]map[string]string
|
||||
}
|
||||
|
||||
func (f *fakeSwapOps) Pool(context.Context, string) (proxmox.PoolInfo, error) {
|
||||
return proxmox.PoolInfo{Members: []proxmox.PoolMember{{VMID: 9201, Type: "lxc"}, {VMID: 9202, Type: "lxc"}, {VMID: 300, Type: "qemu"},
|
||||
{Type: "storage"}, {VMID: 9203, Type: "lxc"}, {VMID: 9204, Type: "lxc"}}}, nil
|
||||
}
|
||||
func (f *fakeSwapOps) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
|
||||
c, ok := f.cfg[vmid]
|
||||
if !ok {
|
||||
return c, errors.New("no such guest")
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
func (f *fakeSwapOps) SetConfig(_ context.Context, vmid int, p map[string]string) (string, error) {
|
||||
if f.failSet[vmid] {
|
||||
return "", errors.New("pve said no")
|
||||
}
|
||||
if f.set == nil {
|
||||
f.set = map[int]map[string]string{}
|
||||
}
|
||||
f.set[vmid] = p
|
||||
return "", nil
|
||||
}
|
||||
func (f *fakeSwapOps) WaitTask(context.Context, string, proxmox.WaitOptions) (proxmox.TaskStatus, error) {
|
||||
return proxmox.TaskStatus{}, nil
|
||||
}
|
||||
|
||||
func TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM(t *testing.T) {
|
||||
f := &fakeSwapOps{cfg: map[int]proxmox.GuestConfig{
|
||||
9201: {Memory: 4096, Swap: 512}, // the measured box: raised
|
||||
9202: {Memory: 2048, Swap: 4096}, // already higher: never lowered
|
||||
9203: {Memory: 6144, Swap: 0}, // none: raised
|
||||
9204: {Memory: 4096, Swap: 512}, // the write fails: logged, the others still done
|
||||
}, failSet: map[int]bool{9204: true}}
|
||||
n, err := EnsureGuestSwap(context.Background(), f, "felhom", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n != 2 {
|
||||
t.Fatalf("raised %d, want 2 (9201, 9203)", n)
|
||||
}
|
||||
if f.set[9201]["swap"] != "4096" || f.set[9203]["swap"] != "6144" {
|
||||
t.Fatalf("writes = %v", f.set)
|
||||
}
|
||||
if _, touched := f.set[9202]; touched {
|
||||
t.Fatal("a swap above the RAM was lowered")
|
||||
}
|
||||
if len(f.set[9201]) != 1 {
|
||||
t.Fatalf("the swap raise wrote more than the swap: %v", f.set[9201])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1192 follow-up (night 2026-10-21): GET /guest/memory carries the host's swap and this guest's swap allowance, so
|
||||
// the controller's install page can say when the host has no swap behind the guest. 0 is a real answer (no swap);
|
||||
// absent means an older agent.
|
||||
// RED-PROOF: without the two MemoryInfo fields the keys are missing → fails on „host_swap_mb absent".
|
||||
func TestR1192_GuestMemoryCarriesHostAndGuestSwap(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
hostB, guestMB int64
|
||||
wantHost, wantG int64
|
||||
}{
|
||||
{"installer host (7.75 GB swap), guest swap = RAM", 7935 * mib, 8192, 7935, 8192},
|
||||
{"a host with NO swap", 0, 8192, 0, 8192},
|
||||
} {
|
||||
f := stdFixture()
|
||||
f.hostSwapB, f.guestSwapMB = c.hostB, c.guestMB
|
||||
w := do(t, memServer(t, f).Handler(), "GET", "/guest/memory", "A", "")
|
||||
var m struct {
|
||||
Data map[string]any `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
hs, ok := m.Data["host_swap_mb"].(float64)
|
||||
if !ok {
|
||||
t.Fatalf("%s: host_swap_mb absent: %s", c.name, w.Body.String())
|
||||
}
|
||||
gs, _ := m.Data["guest_swap_mb"].(float64)
|
||||
if int64(hs) != c.wantHost || int64(gs) != c.wantG {
|
||||
t.Fatalf("%s: host %v guest %v, want %d / %d", c.name, hs, gs, c.wantHost, c.wantG)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1343,6 +1343,20 @@ type BackupStatusResponse struct {
|
||||
// presence. Served on the UNTARGETED request only; additive, so an older controller reads the
|
||||
// response exactly as before.
|
||||
Tiers []TierBackupState `json:"tiers,omitempty"`
|
||||
// HostBusy (R-921, agent Unreleased 2026-10-19) says whether the host-wide heavy-operation gate is held RIGHT NOW
|
||||
// (another guest's backup, the restore test, a night OS step, the stale-lock sweep, a power-off) — a POST /backup
|
||||
// would then be refused with 409. The controller reads it BEFORE it stops the household's apps, so a refused tier
|
||||
// costs no stop. Served on every request; ABSENT (nil) = an older agent, or the gate is not wired — the controller
|
||||
// then keeps its old order (stop, ask, resume at once on 409). A read, never a reservation: the gate can be taken
|
||||
// between this answer and the POST, and that race keeps the old order. Pinned by TestBackupStatus_R921_HostBusy.
|
||||
HostBusy *HostBusyState `json:"host_busy,omitempty"`
|
||||
}
|
||||
|
||||
// HostBusyState is GET /backup/status's host_busy (R-921). What names the holder ("backup:local", "restore-test",
|
||||
// "os-pve-step" …), empty when Busy is false.
|
||||
type HostBusyState struct {
|
||||
Busy bool `json:"busy"`
|
||||
What string `json:"what,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
@@ -1362,6 +1376,10 @@ func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid
|
||||
resp.JobID = job.JobID
|
||||
resp.Error = job.Error
|
||||
}
|
||||
if s.inFlight != nil {
|
||||
what := s.inFlight.Busy()
|
||||
resp.HostBusy = &HostBusyState{Busy: what != "", What: what}
|
||||
}
|
||||
writeOK(w, resp)
|
||||
}
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
@@ -514,3 +515,47 @@ func TestBackupStatus_FiltersToThisGuest(t *testing.T) {
|
||||
t.Fatalf("status not scoped to this guest / not latest: %s", w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// R-921: GET /backup/status says when the host-wide heavy-operation gate is held, so the controller can skip the app
|
||||
// stop for a tier the POST would refuse. Free → busy false; held → busy true with the holder's name; the gate not wired
|
||||
// → no field at all (the controller's „older agent" reading, never „free").
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-10-19): with the `resp.HostBusy = …` assignment removed, the held case fails with
|
||||
//
|
||||
// server_test.go: free gate: host_busy = <nil>, want busy=false
|
||||
//
|
||||
// Restored.
|
||||
func TestBackupStatus_R921_HostBusy(t *testing.T) {
|
||||
read := func(h http.Handler) *HostBusyState {
|
||||
w := do(t, h, "GET", "/backup/status", "A", "")
|
||||
var resp struct {
|
||||
Data BackupStatusResponse `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("status body: %v (%s)", err, w.Body.String())
|
||||
}
|
||||
return resp.Data.HostBusy
|
||||
}
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
if got := read(srv.Handler()); got != nil {
|
||||
t.Fatalf("gate not wired: host_busy = %+v, want absent", got)
|
||||
}
|
||||
gate := &backup.InFlight{}
|
||||
srv.inFlight = gate
|
||||
h := srv.Handler()
|
||||
if got := read(h); got == nil || got.Busy || got.What != "" {
|
||||
t.Fatalf("free gate: host_busy = %+v, want busy=false", got)
|
||||
}
|
||||
release, _, ok := gate.TryAcquire("os-pve-step")
|
||||
if !ok {
|
||||
t.Fatal("could not take the gate")
|
||||
}
|
||||
if got := read(h); got == nil || !got.Busy || got.What != "os-pve-step" {
|
||||
t.Fatalf("held gate: host_busy = %+v, want busy=true what=\"os-pve-step\"", got)
|
||||
}
|
||||
// The read never takes the gate: a POST after the holder lets go is accepted.
|
||||
release()
|
||||
if got := read(h); got == nil || got.Busy {
|
||||
t.Fatalf("released gate: host_busy = %+v, want busy=false", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
@@ -160,6 +161,7 @@ type staleLockAPI interface {
|
||||
ListSnapshots(ctx context.Context, vmid int) ([]proxmox.Snapshot, error)
|
||||
ListRunningTasks(ctx context.Context) ([]proxmox.TaskStatus, error)
|
||||
DeleteSnapshot(ctx context.Context, vmid int, snapname string) (string, error)
|
||||
DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error)
|
||||
Start(ctx context.Context, vmid int) (string, error)
|
||||
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
|
||||
}
|
||||
@@ -334,8 +336,34 @@ func (c *staleLockController) Unlock(ctx context.Context, vmid int) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteVzdumpSnapshot removes the dangling `vzdump` snapshot. R-1010: when the delete fails ONLY because one disk's
|
||||
// snapshot volume is already gone („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump" — a backup that ran
|
||||
// out of space, 2026-10-14), Proxmox stops at that volume and keeps the entry and the other volumes' snapshots, and a
|
||||
// plain retry fails the same way every sweep, forever. Then it retries ONCE with Proxmox's force (the hand fix that
|
||||
// day was `pct delsnapshot 9201 vzdump --force`): the remaining volumes' snapshots are removed and the entry goes. Any
|
||||
// other error is returned as it is — never a force on an error that might mean a volume still holds data (a full
|
||||
// LVM metadata area, a busy volume). Pinned by TestStaleLock_R1010_*.
|
||||
func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int) error {
|
||||
upid, err := c.px.DeleteSnapshot(ctx, vmid, vzdumpSnapshotName)
|
||||
err := c.deleteSnap(ctx, vmid, false)
|
||||
if err == nil || !snapshotVolumeAlreadyGone(err) {
|
||||
return err
|
||||
}
|
||||
if c.logger != nil {
|
||||
c.logger.Warn("stale-lock: a disk's vzdump snapshot volume is already gone — retrying the delete once with force (R-1010)",
|
||||
"vmid", vmid, "err", err)
|
||||
}
|
||||
if ferr := c.deleteSnap(ctx, vmid, true); ferr != nil {
|
||||
return fmt.Errorf("forced snapshot delete after %q also failed: %w", err.Error(), ferr)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *staleLockController) deleteSnap(ctx context.Context, vmid int, force bool) error {
|
||||
del := c.px.DeleteSnapshot
|
||||
if force {
|
||||
del = c.px.DeleteSnapshotForce
|
||||
}
|
||||
upid, err := del(ctx, vmid, vzdumpSnapshotName)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -346,6 +374,23 @@ func (c *staleLockController) DeleteVzdumpSnapshot(ctx context.Context, vmid int
|
||||
return err
|
||||
}
|
||||
|
||||
// snapshotVolumeAlreadyGone: the error is Proxmox's „Failed to find logical volume …_vzdump" from a snapshot delete
|
||||
// (in the task's exit status, its log tail, or the API error text). Narrow on purpose: the force is safe only when the
|
||||
// volume it cannot remove does not exist.
|
||||
func snapshotVolumeAlreadyGone(err error) bool {
|
||||
text := err.Error()
|
||||
var te *proxmox.TaskError
|
||||
if errors.As(err, &te) {
|
||||
text += "\n" + te.ExitStatus + "\n" + strings.Join(te.LogTail, "\n")
|
||||
}
|
||||
for _, l := range strings.Split(text, "\n") {
|
||||
if strings.Contains(strings.ToLower(l), "failed to find logical volume") && strings.Contains(l, "_"+vzdumpSnapshotName) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (c *staleLockController) Start(ctx context.Context, vmid int) error {
|
||||
upid, err := c.px.Start(ctx, vmid)
|
||||
if err != nil {
|
||||
|
||||
@@ -30,6 +30,9 @@ type fakeStaleLockAPI struct {
|
||||
|
||||
delsnap []int
|
||||
started []int
|
||||
// R-1010: scripted delete answers, consumed in order (plain and forced share the list); calls records "plain"/"force".
|
||||
delErrs []error
|
||||
delCalls []string
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) ListLXC(context.Context) ([]proxmox.Guest, error) { return f.lxc, nil }
|
||||
@@ -55,7 +58,22 @@ func (f *fakeStaleLockAPI) ListRunningTasks(context.Context) ([]proxmox.TaskStat
|
||||
|
||||
func (f *fakeStaleLockAPI) DeleteSnapshot(_ context.Context, vmid int, snapname string) (string, error) {
|
||||
f.delsnap = append(f.delsnap, vmid)
|
||||
return "", nil // synchronous — no WaitTask
|
||||
return "", f.nextDelErr("plain") // synchronous — no WaitTask
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) DeleteSnapshotForce(_ context.Context, vmid int, snapname string) (string, error) {
|
||||
f.delsnap = append(f.delsnap, vmid)
|
||||
return "", f.nextDelErr("force")
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) nextDelErr(kind string) error {
|
||||
f.delCalls = append(f.delCalls, kind)
|
||||
if len(f.delErrs) == 0 {
|
||||
return nil
|
||||
}
|
||||
e := f.delErrs[0]
|
||||
f.delErrs = f.delErrs[1:]
|
||||
return e
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) Start(_ context.Context, vmid int) (string, error) {
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1010: a half-removed vzdump snapshot. The error text is the one demo-hp's task printed on 2026-10-14
|
||||
// (`audits/power-cut-fixes-2026-10-14/C4/hand-fix.txt`).
|
||||
const r1010MissingLV = `lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to find logical volume "pve/snap_vm-9201-disk-0_vzdump"`
|
||||
|
||||
func r1010TaskErr(exit string) error {
|
||||
return &proxmox.TaskError{UPID: "UPID:demo-hp:0001:0002:0003:vzdelsnapshot:9201:root@pam:", ExitStatus: exit,
|
||||
LogTail: []string{exit}}
|
||||
}
|
||||
|
||||
// COMPANION RED-PROOF (observed 2026-10-19): with DeleteVzdumpSnapshot back to the single plain delete, the first case
|
||||
// fails with
|
||||
//
|
||||
// stalelock_r1010_test.go: missing volume: calls [plain], want [plain force]; err = proxmox: task … failed: exitstatus "lvremove snapshot …"
|
||||
//
|
||||
// Restored.
|
||||
func TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
errs []error
|
||||
wantCalls []string
|
||||
wantErr string
|
||||
}{
|
||||
{"missing volume", []error{r1010TaskErr(r1010MissingLV)}, []string{"plain", "force"}, ""},
|
||||
{"missing volume, named only in the log tail", []error{&proxmox.TaskError{UPID: "x", ExitStatus: "command 'lvremove' failed: exit code 5",
|
||||
LogTail: []string{r1010MissingLV}}}, []string{"plain", "force"}, ""},
|
||||
{"the plain delete works", nil, []string{"plain"}, ""},
|
||||
{"another error: no force", []error{r1010TaskErr("lvremove snapshot 'pve/snap_vm-9201-disk-0_vzdump' error: Failed to update pool pve/data.")},
|
||||
[]string{"plain"}, "Failed to update pool"},
|
||||
{"a missing volume of ANOTHER snapshot: no force", []error{r1010TaskErr(`Failed to find logical volume "pve/snap_vm-9201-disk-0_mine"`)},
|
||||
[]string{"plain"}, "_mine"},
|
||||
{"the forced delete fails too: returned, never a second force", []error{r1010TaskErr(r1010MissingLV), errors.New("still broken")},
|
||||
[]string{"plain", "force"}, "still broken"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
api := &fakeStaleLockAPI{delErrs: c.errs}
|
||||
ctrl := NewStaleLockController(api, &recordingRunner{}, "felhom", nil)
|
||||
err := ctrl.DeleteVzdumpSnapshot(context.Background(), 9201)
|
||||
if strings.Join(api.delCalls, " ") != strings.Join(c.wantCalls, " ") {
|
||||
t.Errorf("%s: calls %v, want %v; err = %v", c.name, api.delCalls, c.wantCalls, err)
|
||||
}
|
||||
switch {
|
||||
case c.wantErr == "" && err != nil:
|
||||
t.Errorf("%s: err = %v, want nil", c.name, err)
|
||||
case c.wantErr != "" && (err == nil || !strings.Contains(err.Error(), c.wantErr)):
|
||||
t.Errorf("%s: err = %v, want one naming %q", c.name, err, c.wantErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,12 +6,12 @@
|
||||
// the report stanza the hub surfaces:
|
||||
//
|
||||
// - /run/sshd present? — OpenSSH's SHARED privsep dir; its absence is the KEXINIT-reset
|
||||
// lockout (SPIKE-felhom-sshd-2026-07-05 §8).
|
||||
// lockout (SPIKE-felhom-sshd-2026-07-05 §8).
|
||||
// - stock sshd listener answers? — a TCP connect to the sshd port (22 for G1; H1 passes the
|
||||
// discovered felhom-sshd port later).
|
||||
// discovered felhom-sshd port later).
|
||||
// - did the watchdog auto-heal? — the watchdog writes an RFC3339 marker to /run when it heals;
|
||||
// its presence (+ timestamp) tells the hub a clobber recurred, so
|
||||
// the operator learns of a recurring cause BEFORE a lockout.
|
||||
// its presence (+ timestamp) tells the hub a clobber recurred, so
|
||||
// the operator learns of a recurring cause BEFORE a lockout.
|
||||
//
|
||||
// Read-only and dependency-light: os.Stat + os.ReadFile + a short net.Dial, no exec, no sudo. Safe to
|
||||
// run every heartbeat.
|
||||
@@ -44,9 +44,9 @@ const (
|
||||
type Reporter struct {
|
||||
privsepDir string
|
||||
marker string
|
||||
sshdAddr string // host:port dialed for the reachability probe
|
||||
statDir func(string) bool // dir-exists check (os.Stat wrapper; test seam)
|
||||
readMarker func(string) (string, bool) // marker read → (timestamp, present)
|
||||
sshdAddr string // host:port dialed for the reachability probe
|
||||
statDir func(string) bool // dir-exists check (os.Stat wrapper; test seam)
|
||||
readMarker func(string) (string, bool) // marker read → (timestamp, present)
|
||||
dialSSHD func(ctx context.Context, addr string) bool
|
||||
}
|
||||
|
||||
|
||||
@@ -370,7 +370,8 @@ func HealthVerdict(before, after *Health) (bool, string) {
|
||||
|
||||
// HostHealthVerdict is THE host health rule (`11` §8.2; pinned by TestHostHealthVerdict): the Proxmox daemons and the
|
||||
// agent are active, the customer guest still runs, the guest's own rule still passes against the start of the pass,
|
||||
// and the tunnel is `running` (R-841).
|
||||
// and the tunnel is `running` (R-841) — or `absent` both now and before the update: a box with no tunnel set up
|
||||
// (R-1070; pinned by TestHostHealthVerdict_R1070_NoTunnelSetUp).
|
||||
func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
|
||||
if after == nil {
|
||||
return false, "no health reading"
|
||||
@@ -398,12 +399,28 @@ func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
|
||||
if ok, why := HealthVerdict(gb, after.Guest); !ok {
|
||||
return false, "guest: " + why
|
||||
}
|
||||
if tunnel != hub.TunnelRunning {
|
||||
switch {
|
||||
case tunnel == hub.TunnelRunning:
|
||||
case tunnel == hub.TunnelAbsent && noTunnelBefore(before):
|
||||
// R-1070: no cloudflared container before the update and none after — the box has no tunnel to check.
|
||||
case tunnel == hub.TunnelAbsent:
|
||||
return false, "the tunnel is absent (no cloudflared container now, and the reading from before the update does not show it was absent then)"
|
||||
default:
|
||||
return false, "the tunnel is " + tunnel
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
// noTunnelBefore: the reading from the start of the pass shows the guest's Docker answered and listed NO cloudflared
|
||||
// container. A missing or failed reading is not „no tunnel" (an empty list from a failed `docker ps` must not pass).
|
||||
func noTunnelBefore(before *Health) bool {
|
||||
if before == nil || before.Guest == nil || !before.Guest.DockerOK {
|
||||
return false
|
||||
}
|
||||
_, had := before.Guest.Containers["cloudflared"]
|
||||
return !had
|
||||
}
|
||||
|
||||
// EngineOf is the engine version `docker version` prints for a docker-ce package version: "5:29.8.2-1~debian.13~trixie"
|
||||
// → "29.8.2" (no epoch, no Debian revision).
|
||||
func EngineOf(pkgVersion string) string {
|
||||
|
||||
@@ -568,3 +568,42 @@ func TestDocker_OOMCheckReachesTheHubUnchanged(t *testing.T) {
|
||||
t.Fatalf("no docker report posted: %s", calls(w))
|
||||
}
|
||||
}
|
||||
|
||||
// R-1070: a box with no tunnel set up (no cloudflared container before the update and none after) passes the host
|
||||
// rule; a box WITH a tunnel that is down, gone, or unreadable still fails it.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-10-19): with the „not applicable" branch switched off (only `running` passes, the
|
||||
// pre-fix rule) the first case fails with
|
||||
//
|
||||
// leg_test.go: no tunnel before, none after: got false (the tunnel is absent (no cloudflared container now, …)), want true
|
||||
//
|
||||
// and with the pre-fix case-sensitive match TestClassifyTunnel's Docker 29 row reads `unknown` („could not ask the
|
||||
// guest"), the state Tester4 failed on.
|
||||
// Restored.
|
||||
func TestHostHealthVerdict_R1070_NoTunnelSetUp(t *testing.T) {
|
||||
withTunnel := hostOK()
|
||||
withTunnel.Guest.Containers["cloudflared"] = Container{State: "running", Health: "healthy"}
|
||||
dockerFailed := hostOK()
|
||||
dockerFailed.Guest = &Health{DockerOK: false, Containers: map[string]Container{}}
|
||||
cases := []struct {
|
||||
name string
|
||||
before *Health
|
||||
tunnel string
|
||||
want bool
|
||||
why string
|
||||
}{
|
||||
{"no tunnel before, none after", hostOK(), hub.TunnelAbsent, true, ""},
|
||||
{"a tunnel before, gone after", withTunnel, hub.TunnelAbsent, false, "cloudflared was running"},
|
||||
{"a tunnel before, down after", withTunnel, hub.TunnelNotRunning, false, "cloudflared was running"},
|
||||
{"no tunnel before, a stopped one after", hostOK(), hub.TunnelNotRunning, false, "not_running"},
|
||||
{"no tunnel before, the probe could not ask after", hostOK(), hub.TunnelUnknown, false, "unknown"},
|
||||
{"no reading from before", nil, hub.TunnelAbsent, false, "absent"},
|
||||
{"docker failed before (an empty list is not proof of none)", dockerFailed, hub.TunnelAbsent, false, "absent"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, why := HostHealthVerdict(c.before, hostOK(), c.tunnel)
|
||||
if got != c.want || !strings.Contains(why, c.why) {
|
||||
t.Errorf("%s: got %v (%s), want %v (…%s…)", c.name, got, why, c.want, c.why)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -180,6 +180,18 @@ func (c *Client) DeleteSnapshot(ctx context.Context, vmid int, snapname string)
|
||||
return c.dataString(ctx, http.MethodDelete, path, nil)
|
||||
}
|
||||
|
||||
// DeleteSnapshotForce is DeleteSnapshot with Proxmox's own `force=1` („remove from the config file, even if removing
|
||||
// disk snapshots fails"): the API's flag, so the same token privilege (VM.Snapshot) and no new root command. Used by
|
||||
// the stale-lock janitor ONLY after a delete failed because one disk's snapshot volume is already gone (R-1010). The
|
||||
// flag rides the query string, where pveproxy reads a DELETE's parameters. Pinned by TestDeleteSnapshotForce_R1010.
|
||||
func (c *Client) DeleteSnapshotForce(ctx context.Context, vmid int, snapname string) (string, error) {
|
||||
if vmid == 0 || snapname == "" {
|
||||
return "", fmt.Errorf("proxmox: DeleteSnapshotForce needs vmid and snapname")
|
||||
}
|
||||
path := fmt.Sprintf("/nodes/%s/lxc/%d/snapshot/%s?force=1", c.node, vmid, url.PathEscape(snapname))
|
||||
return c.dataString(ctx, http.MethodDelete, path, nil)
|
||||
}
|
||||
|
||||
// SetConfig applies config changes via PUT /nodes/{node}/lxc/{vmid}/config
|
||||
// (e.g. memory, cores, net0, mpN with a backup flag). PVE may apply this
|
||||
// synchronously (no UPID) — the returned string is empty in that case, and "" is
|
||||
|
||||
@@ -3,6 +3,7 @@ package proxmox
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
@@ -99,3 +100,24 @@ func TestDestroyScratchLXC_RefusesUnconfiguredBand(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// R-1010: the forced snapshot delete is the API's own flag on the same DELETE (no new root command).
|
||||
func TestDeleteSnapshotForce_R1010(t *testing.T) {
|
||||
var got *http.Request
|
||||
d := &mockDoer{fn: func(r *http.Request) (*http.Response, error) {
|
||||
got = r
|
||||
return jsonResp(200, `{"data":"UPID:demo-felhom:0001:0002:0003:vzdelsnapshot:9201:u@pve!t:"}`), nil
|
||||
}}
|
||||
upid, err := newTestClient(d).DeleteSnapshotForce(context.Background(), 9201, "vzdump")
|
||||
if err != nil || upid == "" {
|
||||
t.Fatalf("DeleteSnapshotForce: upid %q err %v", upid, err)
|
||||
}
|
||||
if got.Method != http.MethodDelete || got.URL.Path != "/api2/json/nodes/demo-felhom/lxc/9201/snapshot/vzdump" ||
|
||||
got.URL.Query().Get("force") != "1" {
|
||||
t.Fatalf("request = %s %s, want DELETE …/lxc/9201/snapshot/vzdump?force=1", got.Method, got.URL.String())
|
||||
}
|
||||
d.fn = func(r *http.Request) (*http.Response, error) { got = r; return jsonResp(200, `{"data":null}`), nil }
|
||||
if _, err := newTestClient(d).DeleteSnapshot(context.Background(), 9201, "vzdump"); err != nil || got.URL.RawQuery != "" {
|
||||
t.Fatalf("the plain delete must carry no force: %s (err %v)", got.URL.String(), err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -59,7 +59,7 @@ const DefaultPool = "felhom"
|
||||
//
|
||||
// SysDataGrowGB itself is NOT removed — see its field comment: the host installer still passes
|
||||
// `-sysdata-grow`, and its GiB are FOLDED INTO the single volume's grow rather than dropped.
|
||||
const DefaultSysDataMount = ""
|
||||
const DefaultSysDataMount = ""
|
||||
|
||||
// Structural host-bind mountpoints every provisioned guest carries (GL-5; verdict of
|
||||
// SPIKE-dr-bindmount-source-2026-07-07): the permanent drives parent bind (mp8,
|
||||
@@ -645,6 +645,9 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
|
||||
}
|
||||
if spec.MemoryMB > 0 {
|
||||
params["memory"] = strconv.Itoa(spec.MemoryMB)
|
||||
// R-1192: swap = RAM. Left unset, Proxmox gives an LXC 512 MB, and a guest that fills its RAM can then evict
|
||||
// only program code — the measured freeze (localapi.GuestSwapMB carries the reasoning and the numbers).
|
||||
params["swap"] = strconv.Itoa(spec.MemoryMB)
|
||||
}
|
||||
for i, m := range spec.Mounts {
|
||||
// backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
package reconcile
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1192: a new guest is created with swap = its RAM (left unset, Proxmox gives an LXC 512 MB — the measured freeze,
|
||||
// felhom.eu audits/fixes-2026-10-21/P4-freeze/). COMPANION RED-PROOF: drop params["swap"] in buildBringUpConfig.
|
||||
func TestR1192_BringUpSetsSwapToTheRAM(t *testing.T) {
|
||||
p := buildBringUpConfig(BringUpSpec{MemoryMB: 4096}, proxmox.GuestConfig{})
|
||||
if p["memory"] != "4096" || p["swap"] != "4096" {
|
||||
t.Fatalf("bring-up params memory=%q swap=%q, want 4096/4096", p["memory"], p["swap"])
|
||||
}
|
||||
if q := buildBringUpConfig(BringUpSpec{}, proxmox.GuestConfig{}); q["swap"] != "" {
|
||||
t.Fatalf("no memory in the spec → no swap written either, got %q", q["swap"])
|
||||
}
|
||||
}
|
||||
@@ -25,8 +25,8 @@ import (
|
||||
// operator's signing CLI (cmd/felhom-opsign) produces it; this runner consumes it. The hub never
|
||||
// forges one (it holds no signing key) and never opens it.
|
||||
type Envelope struct {
|
||||
OpBlobB64 string `json:"op_blob_b64"` // base64 of the canonical OpBlob JSON
|
||||
SigArmored string `json:"sig_armored"` // armored SSHSIG over the op-blob (namespace felhom-op-v1)
|
||||
OpBlobB64 string `json:"op_blob_b64"` // base64 of the canonical OpBlob JSON
|
||||
SigArmored string `json:"sig_armored"` // armored SSHSIG over the op-blob (namespace felhom-op-v1)
|
||||
}
|
||||
|
||||
// JobSource fetches + clears jobs. Satisfied by *hub.Client.
|
||||
|
||||
@@ -14,10 +14,10 @@ func TestClassifyClaim(t *testing.T) {
|
||||
nodes: []claimNode{{name: "sdd", fstype: "", mountpoint: ""}, {name: "sdd1", fstype: "ntfs", mountpoint: ""}}}
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
mutate func(f *claimFacts)
|
||||
name string
|
||||
mutate func(f *claimFacts)
|
||||
wantUnclaimed bool
|
||||
reasonHas string
|
||||
reasonHas string
|
||||
}{
|
||||
{"clean unclaimed disk (ntfs, unmounted)", func(f *claimFacts) {}, true, "unclaimed"},
|
||||
{"blank disk", func(f *claimFacts) { f.nodes = []claimNode{{name: "sdd"}} }, true, "unclaimed"},
|
||||
|
||||
@@ -103,12 +103,12 @@ type MountSpec struct {
|
||||
// Binaries holds the absolute paths of the allow-listed binaries (overridable from config
|
||||
// so the sudoers entries and the agent agree on exact paths).
|
||||
type Binaries struct {
|
||||
Systemctl string
|
||||
Install string
|
||||
Systemctl string
|
||||
Install string
|
||||
// PrivApply is the root content checker (R-861, agent v0.146.0): a unit file reaches /etc/systemd/system only
|
||||
// through `felhom-priv-apply unit <name>`, which reads the staged copy from /var/lib/felhom-agent/units/ and
|
||||
// refuses anything the renderers here never produce (a bind over /etc, a Where outside /mnt, …).
|
||||
PrivApply string
|
||||
PrivApply string
|
||||
Smartctl string
|
||||
Lvs string
|
||||
Blkid string // device signature probe (8C data-bearing detection)
|
||||
|
||||
@@ -17,12 +17,12 @@ import (
|
||||
// Four states (3A):
|
||||
// - new : durable-id NOT in the store. Never auto-mounted (the user must enroll it).
|
||||
// - enrolled : desired = mounted + bound into the guest. Drift (present && unmounted) with
|
||||
// this intent → reconcile.
|
||||
// this intent → reconcile.
|
||||
// - ejected : an intentional, temporary unmount, set ONLY via the official eject endpoint.
|
||||
// Present && unmounted → left alone. CLEARED to `enrolled` when the drive goes
|
||||
// physically ABSENT, so a replug auto-mounts (the replug rule, for free).
|
||||
// Present && unmounted → left alone. CLEARED to `enrolled` when the drive goes
|
||||
// physically ABSENT, so a replug auto-mounts (the replug rule, for free).
|
||||
// - decommissioned : permanent. Never auto-mounted; SURVIVES absent/present; cleared only by an
|
||||
// explicit re-commission.
|
||||
// explicit re-commission.
|
||||
//
|
||||
// The elegant invariant: intent is recorded ONLY through the official enroll/eject/decommission
|
||||
// paths. "Is it unmounted" ≠ "did someone officially eject it."
|
||||
|
||||
@@ -93,11 +93,11 @@ systemd-1 /mnt/felhom-drives/live autofs rw,relatime,fd=86 0 0
|
||||
where string
|
||||
want bool
|
||||
}{
|
||||
{"/mnt/felhom-drives/live", true}, // real nfs4 (the autofs line for the same path must not confuse it)
|
||||
{"/mnt/felhom-drives/smb", true}, // cifs
|
||||
{"/mnt/felhom-drives/idle", false}, // autofs trigger ONLY — idle automount is NOT mounted
|
||||
{"/mnt/felhom-drives/disk", false}, // a local fs at the path is not a network mount
|
||||
{"/mnt/felhom-drives/nope", false}, // absent
|
||||
{"/mnt/felhom-drives/live", true}, // real nfs4 (the autofs line for the same path must not confuse it)
|
||||
{"/mnt/felhom-drives/smb", true}, // cifs
|
||||
{"/mnt/felhom-drives/idle", false}, // autofs trigger ONLY — idle automount is NOT mounted
|
||||
{"/mnt/felhom-drives/disk", false}, // a local fs at the path is not a network mount
|
||||
{"/mnt/felhom-drives/nope", false}, // absent
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := networkMountedIn(procMounts, tc.where); got != tc.want {
|
||||
|
||||
@@ -96,11 +96,11 @@ type Watchdog struct {
|
||||
now func() time.Time
|
||||
spawn func(func()) // spawn a background task (overridable in tests; default `go f()`)
|
||||
|
||||
intent IntentReader // P3: gate re-mount by drive intent (nil = ungated legacy)
|
||||
intent IntentReader // P3: gate re-mount by drive intent (nil = ungated legacy)
|
||||
onAbsent func(durableID string) // P3: called when a known target goes ABSENT (clears `ejected`)
|
||||
|
||||
mu sync.Mutex
|
||||
last map[string]bool // name -> last observed present (only for seen targets)
|
||||
last map[string]bool // name -> last observed present (only for seen targets)
|
||||
lastFire time.Time
|
||||
fired bool // lastFire is valid
|
||||
pending bool // a transition is awaiting the debounce window
|
||||
|
||||
@@ -99,6 +99,7 @@ type Manager struct {
|
||||
resolver Resolver
|
||||
|
||||
staleAfter time.Duration // handshake-age threshold for the re-resolve watchdog (doc 06 §4.2)
|
||||
rekeyAfter time.Duration // D30: how long a future-stamped handshake stays before the re-key (rekey.go)
|
||||
|
||||
mu sync.Mutex
|
||||
nextRegAt time.Time
|
||||
@@ -111,6 +112,10 @@ type Manager struct {
|
||||
resolveFailLogged bool // DNS-failure log throttle (reset on the next success)
|
||||
staleSameIPLogged bool // "stale but IP unchanged" log throttle (endpoint-down, not re-IP)
|
||||
|
||||
// D30 (rekey.go): when the latest handshake was first seen stamped in the future, and the last re-key.
|
||||
futureSince time.Time
|
||||
lastRekey time.Time
|
||||
|
||||
// conv is the CACHED tunnel-convergence snapshot (v0.90.0, R-28 fast-tick source), refreshed at
|
||||
// the end of every Apply (the tunnel's own cadence) so the fast-tick reads it WITHOUT execing
|
||||
// `wg`/`systemctl` per tick. wgBlockDesired = a wireguard block for THIS key is desired;
|
||||
@@ -156,6 +161,7 @@ func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, log
|
||||
now: time.Now,
|
||||
resolver: netResolver{},
|
||||
staleAfter: 180 * time.Second,
|
||||
rekeyAfter: rekeyAfterDefault,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -600,6 +606,10 @@ func (m *Manager) Watchdog(ctx context.Context, fetched bool, block *hub.WireWir
|
||||
if !fetched || block == nil || !m.lastResolvedIP.IsValid() {
|
||||
return // nothing applied to watch, or no desired block
|
||||
}
|
||||
// D30: a handshake stamped in the future that stays → a new key on the existing peer path (rekey.go).
|
||||
if m.clockAheadRekeyLocked(ctx, block) {
|
||||
return
|
||||
}
|
||||
if !m.handshakeStaleLocked(ctx) {
|
||||
return // healthy → no DNS, no action (the load-bearing negative)
|
||||
}
|
||||
@@ -661,6 +671,9 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
|
||||
if age, ok := parseHandshakeAge(string(out), m.now()); ok {
|
||||
st.LastHandshakeAgeS = &age
|
||||
}
|
||||
if fut, ok := handshakeFuture(string(out), m.now()); ok {
|
||||
st.HandshakeFutureS = &fut
|
||||
}
|
||||
}
|
||||
}
|
||||
if st.Pubkey == "" && !st.Registered && !st.Active {
|
||||
@@ -669,6 +682,28 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
|
||||
return st
|
||||
}
|
||||
|
||||
// handshakeFuture (R-1204) reports how many seconds the latest handshake lies in the future, ok only when it does
|
||||
// (by more than a minute — clock jitter is not the case). parseHandshakeAge clamps that case to 0 („just now"), which
|
||||
// is what made a box whose clock had run ahead look healthy while ep0 refused its tunnel. Pinned by
|
||||
// TestR1204_AFutureHandshakeIsReported.
|
||||
func handshakeFuture(out string, now time.Time) (int64, bool) {
|
||||
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) != 2 {
|
||||
continue
|
||||
}
|
||||
epoch, err := strconv.ParseInt(f[1], 10, 64)
|
||||
if err != nil || epoch <= 0 {
|
||||
continue
|
||||
}
|
||||
if fut := epoch - now.Unix(); fut > 60 {
|
||||
return fut, true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>").
|
||||
// epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0).
|
||||
func parseHandshakeAge(out string, now time.Time) (int64, bool) {
|
||||
|
||||
@@ -696,3 +696,31 @@ func TestInitialResolveFailureNoTeardown(t *testing.T) {
|
||||
t.Errorf("did not recover after DNS returned: %v", rr.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1204: a box that ran once with its clock ahead keeps a handshake stamped in the FUTURE — ep0 then refuses every
|
||||
// newer one. The age stays clamped at 0 (unchanged contract), and the new field says how far ahead it lies; a
|
||||
// handshake in the past (or within a minute of now) never sets it.
|
||||
// RED-PROOF: with Status not setting HandshakeFutureS the first assertion fails — seen failing 2026-10-22.
|
||||
func TestR1204_AFutureHandshakeIsReported(t *testing.T) {
|
||||
fh := &fakeHub{}
|
||||
m, _, active, _ := testManager(t, fh)
|
||||
ctx := context.Background()
|
||||
m.Apply(ctx, false, nil)
|
||||
*active = true
|
||||
m.now = func() time.Time { return time.Unix(1783107115-2*24*3600, 0) } // the clock came back two days earlier
|
||||
st := m.Status(ctx)
|
||||
if st.HandshakeFutureS == nil || *st.HandshakeFutureS != 2*24*3600 {
|
||||
t.Fatalf("handshake_future_s = %v, want %d", st.HandshakeFutureS, 2*24*3600)
|
||||
}
|
||||
if st.LastHandshakeAgeS == nil || *st.LastHandshakeAgeS != 0 {
|
||||
t.Fatalf("the age stays clamped at 0: %v", st.LastHandshakeAgeS)
|
||||
}
|
||||
m.now = func() time.Time { return time.Unix(1783107115+100, 0) }
|
||||
if st = m.Status(ctx); st.HandshakeFutureS != nil {
|
||||
t.Fatalf("a past handshake must not set handshake_future_s: %v", *st.HandshakeFutureS)
|
||||
}
|
||||
m.now = func() time.Time { return time.Unix(1783107115-30, 0) } // 30 s of jitter is not the case
|
||||
if st = m.Status(ctx); st.HandshakeFutureS != nil {
|
||||
t.Fatalf("clock jitter must not set handshake_future_s")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
package wgtunnel
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// D30 (operator 2026-10-22, R-1204): the tunnel heals itself after the box ran with its clock ahead.
|
||||
//
|
||||
// WireGuard refuses a handshake whose timestamp is not newer than the last one it accepted from that key (replay
|
||||
// protection). A box whose clock ran ahead handshook with a future timestamp; ep0 keeps it, and once the clock is
|
||||
// right again ep0 refuses every new handshake from that key — until the box's clock passes the stored stamp. The
|
||||
// sign on the box: its own latest handshake stays stamped in the future (R-1204's handshake_future_s).
|
||||
//
|
||||
// The repair: when that sign stays for rekeyAfter, the agent makes a NEW key pair and registers the new public key on
|
||||
// the existing path (POST /hosts/<id>/wg → the hub re-keys the peer IN PLACE: same /32, new key, generation bump,
|
||||
// and the wgsync push of the FULL peer list to ep0 drops the old key and adds the new one). A new key is a new peer
|
||||
// for ep0, with no stored timestamp — the next handshake is accepted. Nothing new runs on ep0.
|
||||
//
|
||||
// The previous private key is kept as private.key.prev (0600): the escrow ceremony may hold it (a host-loss DR that
|
||||
// installs it registers it again, and the hub re-keys in place back to it — the same path).
|
||||
// Pinned by TestD30_* (rekey_test.go).
|
||||
|
||||
const (
|
||||
// rekeyAfter: how long the latest handshake must stay in the future before the re-key. A working tunnel
|
||||
// handshakes every two minutes, which ends the sign within minutes of the clock being right.
|
||||
rekeyAfterDefault = 15 * time.Minute
|
||||
// rekeyMinGap bounds the re-key to once an hour, whatever the clock does.
|
||||
rekeyMinGap = time.Hour
|
||||
prevKeyName = "private.key.prev"
|
||||
)
|
||||
|
||||
// clockAheadRekeyLocked watches for a handshake stamped in the future and re-keys when it stays. It reports whether
|
||||
// it re-keyed (the caller then skips the rest of its tick). Caller holds m.mu.
|
||||
func (m *Manager) clockAheadRekeyLocked(ctx context.Context, block *hub.WireWireguard) bool {
|
||||
if m.keyBroken || !m.isActive(ctx) {
|
||||
m.futureSince = time.Time{}
|
||||
return false
|
||||
}
|
||||
out, _, err := m.runner.Run(ctx, "wg", "show", Iface, "latest-handshakes")
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
now := m.now()
|
||||
fut, ok := handshakeFuture(string(out), now)
|
||||
if !ok {
|
||||
if !m.futureSince.IsZero() {
|
||||
m.logger.Info("wgtunnel: the handshake is no longer stamped in the future — no re-key needed")
|
||||
}
|
||||
m.futureSince = time.Time{}
|
||||
return false
|
||||
}
|
||||
if m.futureSince.IsZero() {
|
||||
m.futureSince = now
|
||||
m.logger.Warn("wgtunnel: the latest handshake is stamped in the future — this box's clock ran ahead, and the endpoint refuses its newer handshakes; re-keying if it stays",
|
||||
"future_s", fut, "rekey_after", m.rekeyAfter.String())
|
||||
return false
|
||||
}
|
||||
// A clock that jumps backwards must not stretch either wait (commit security review 2026-10-23 night): a start or a
|
||||
// last re-key stamped AFTER now is from a wrong clock and is dropped — this whole feature exists because clocks jump.
|
||||
if now.Before(m.futureSince) {
|
||||
m.futureSince = now
|
||||
}
|
||||
if now.Sub(m.futureSince) < m.rekeyAfter {
|
||||
return false
|
||||
}
|
||||
if m.lastRekey.IsZero() {
|
||||
m.lastRekey = m.loadLastRekey()
|
||||
}
|
||||
if now.Before(m.lastRekey) {
|
||||
m.lastRekey = time.Time{}
|
||||
}
|
||||
if !m.lastRekey.IsZero() && now.Sub(m.lastRekey) < rekeyMinGap {
|
||||
return false
|
||||
}
|
||||
pub, _, err := EnsureKey(m.stateDir)
|
||||
if err != nil || block.Pubkey != pub {
|
||||
return false // a re-register is already in flight, or the key is not readable — never re-key over that
|
||||
}
|
||||
newPub, err := rotateKey(m.stateDir)
|
||||
if err != nil {
|
||||
m.logger.Error("wgtunnel: re-key after a future handshake FAILED — the old key stays", "err", err)
|
||||
m.lastRekey = now // not again before rekeyMinGap
|
||||
m.saveLastRekey(now)
|
||||
return false
|
||||
}
|
||||
m.lastRekey, m.futureSince = now, time.Time{}
|
||||
m.saveLastRekey(now)
|
||||
m.logger.Warn("wgtunnel: RE-KEYED after a handshake stamped in the future (D30) — registering the new key; the hub re-keys the peer in place and the endpoint takes the new key",
|
||||
"old_pubkey", pub, "new_pubkey", newPub, "future_s", fut)
|
||||
m.nextRegAt = time.Time{} // register now, not after an older backoff
|
||||
m.registerLocked(ctx, newPub)
|
||||
return true
|
||||
}
|
||||
|
||||
// rotateKey makes a new key pair, never leaving the box without a readable key (review 2026-10-23 night, N6): the new
|
||||
// key is written and fsynced to a temp file; the current key gets a second name by a HARD LINK (the escrow may hold
|
||||
// it — `private.key.prev` keeps the first such key, later ones `private.key.prev.<unix>`, the newest three kept, N7);
|
||||
// then the temp file is renamed OVER the current name (atomic) and the directory fsynced. At every instant
|
||||
// `private.key` is a complete key. Returns the new PUBLIC key only.
|
||||
func rotateKey(stateDir string) (string, error) {
|
||||
dir := filepath.Join(stateDir, "wg")
|
||||
cur := filepath.Join(dir, keyFileName)
|
||||
if _, err := os.Stat(cur); err != nil {
|
||||
return "", fmt.Errorf("no current key: %w", err)
|
||||
}
|
||||
priv := make([]byte, 32)
|
||||
if _, err := rand.Read(priv); err != nil {
|
||||
return "", fmt.Errorf("entropy: %w", err)
|
||||
}
|
||||
clampPrivateKey(priv)
|
||||
pub, err := derivePublic(priv)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
tmp := filepath.Join(dir, keyFileName+".new")
|
||||
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("writing the new key: %w", err)
|
||||
}
|
||||
_, werr := f.Write([]byte(base64.StdEncoding.EncodeToString(priv) + "\n"))
|
||||
serr := f.Sync()
|
||||
cerr := f.Close()
|
||||
if werr != nil || serr != nil || cerr != nil {
|
||||
os.Remove(tmp)
|
||||
return "", fmt.Errorf("writing the new key: %v %v %v", werr, serr, cerr)
|
||||
}
|
||||
prev := filepath.Join(dir, prevKeyName)
|
||||
if _, err := os.Lstat(prev); err == nil {
|
||||
prev = filepath.Join(dir, fmt.Sprintf("%s.%d", prevKeyName, rekeyNow().Unix()))
|
||||
}
|
||||
if err := os.Link(cur, prev); err != nil {
|
||||
os.Remove(tmp)
|
||||
return "", fmt.Errorf("keeping the old key: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmp, cur); err != nil {
|
||||
os.Remove(tmp)
|
||||
return "", fmt.Errorf("installing the new key: %w", err)
|
||||
}
|
||||
if d, err := os.Open(dir); err == nil {
|
||||
d.Sync()
|
||||
d.Close()
|
||||
}
|
||||
prunePrevKeys(dir, 3)
|
||||
return pub, nil
|
||||
}
|
||||
|
||||
// rekeyNow is the clock for the kept-key names (a seam).
|
||||
var rekeyNow = time.Now
|
||||
|
||||
// prunePrevKeys keeps the newest `keep` of the `private.key.prev.<unix>` files (`private.key.prev` itself always stays).
|
||||
func prunePrevKeys(dir string, keep int) {
|
||||
m, _ := filepath.Glob(filepath.Join(dir, prevKeyName+".*"))
|
||||
if len(m) <= keep {
|
||||
return
|
||||
}
|
||||
sort.Strings(m) // the same digit count until 2286: lexical = time order
|
||||
for _, p := range m[:len(m)-keep] {
|
||||
os.Remove(p)
|
||||
}
|
||||
}
|
||||
|
||||
// lastRekeyFile records the last re-key's time across agent restarts (N7: the once-an-hour bound was in memory only).
|
||||
const lastRekeyFile = "last-rekey"
|
||||
|
||||
func (m *Manager) loadLastRekey() time.Time {
|
||||
b, err := os.ReadFile(filepath.Join(m.wgDir(), lastRekeyFile))
|
||||
if err != nil {
|
||||
return time.Time{}
|
||||
}
|
||||
var u int64
|
||||
if _, err := fmt.Sscanf(strings.TrimSpace(string(b)), "%d", &u); err != nil {
|
||||
return time.Time{}
|
||||
}
|
||||
return time.Unix(u, 0)
|
||||
}
|
||||
|
||||
func (m *Manager) saveLastRekey(t time.Time) {
|
||||
_ = os.WriteFile(filepath.Join(m.wgDir(), lastRekeyFile), []byte(fmt.Sprintf("%d\n", t.Unix())), 0o600)
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
package wgtunnel
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// D30 (R-1204): a handshake stamped in the future that STAYS for rekeyAfter → a new key pair, registered on the
|
||||
// existing peer path (the hub re-keys in place; its full-list push to ep0 swaps the key). The fake `wg` reports a
|
||||
// handshake at 1783107115; a clock three hours behind that sees it three hours in the future.
|
||||
// RED-PROOF (2026-10-22 night): with the clockAheadRekeyLocked call removed from Watchdog,
|
||||
// TestD30_AFutureHandshakeThatStaysReKeys and …AFailedRegistration… fail (no new key, no registration); with the
|
||||
// rekeyAfter wait removed, …ThatStaysReKeys fails (a re-key at 10 min); with the isActive check removed,
|
||||
// TestD30_NoReKeyWhileTheTunnelIsDown fails.
|
||||
|
||||
const hs = 1783107115
|
||||
|
||||
func at(m *Manager, unix int64) { m.now = func() time.Time { return time.Unix(unix, 0) } }
|
||||
|
||||
func readKey(t *testing.T, m *Manager, name string) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join(m.stateDir, "wg", name))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
func TestD30_AFutureHandshakeThatStaysReKeys(t *testing.T) {
|
||||
fh := &fakeHub{}
|
||||
m, _, active, logs := testManager(t, fh)
|
||||
oldPub := establishWG(t, m, active)
|
||||
oldPriv := readKey(t, m, keyFileName)
|
||||
ctx := context.Background()
|
||||
calls := fh.count()
|
||||
|
||||
t0 := int64(hs - 3*3600) // the clock is right again; the last handshake (made while it ran ahead) is 3 h ahead
|
||||
at(m, t0)
|
||||
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||
at(m, t0+10*60)
|
||||
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||
if fh.count() != calls || localPub(t, m) != oldPub {
|
||||
t.Fatal("no re-key before the future stamp has stayed rekeyAfter")
|
||||
}
|
||||
at(m, t0+16*60)
|
||||
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||
newPub := localPub(t, m)
|
||||
if newPub == oldPub {
|
||||
t.Fatal("a future stamp that stayed 16 min must give a new key")
|
||||
}
|
||||
if fh.count() != calls+1 {
|
||||
t.Fatalf("the new key must be registered on the existing peer path (RegisterWG calls %d → %d)", calls, fh.count())
|
||||
}
|
||||
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
|
||||
t.Fatalf("the registration marker must carry the new key: %+v", mk)
|
||||
}
|
||||
if readKey(t, m, prevKeyName) != oldPriv {
|
||||
t.Fatal("the old private key must be kept as private.key.prev (the escrow may hold it)")
|
||||
}
|
||||
if strings.Contains(logs.String(), oldPriv) || strings.Contains(logs.String(), readKey(t, m, keyFileName)) {
|
||||
t.Fatal("a private key reached the log")
|
||||
}
|
||||
// The served block still names the OLD key until the next fetch: no second re-key over that.
|
||||
at(m, t0+17*60)
|
||||
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||
if localPub(t, m) != newPub {
|
||||
t.Fatal("a second re-key ran before the hub served the new key")
|
||||
}
|
||||
// The hub serves the new key → the tunnel is applied with the NEW private key.
|
||||
m.Apply(ctx, true, testBlock(newPub))
|
||||
conf, err := os.ReadFile(m.stagedConfPath())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(conf), readKey(t, m, keyFileName)) || strings.Contains(string(conf), oldPriv) {
|
||||
t.Fatal("the applied tunnel must use the new private key, not the old one")
|
||||
}
|
||||
}
|
||||
|
||||
func TestD30_AShortFutureSpellDoesNotReKey(t *testing.T) {
|
||||
fh := &fakeHub{}
|
||||
m, _, active, _ := testManager(t, fh)
|
||||
pub := establishWG(t, m, active)
|
||||
ctx := context.Background()
|
||||
t0 := int64(hs - 3*3600)
|
||||
at(m, t0)
|
||||
m.Watchdog(ctx, true, testBlock(pub))
|
||||
at(m, hs+100) // a handshake now lies in the past: the sign ended (a working tunnel)
|
||||
m.Watchdog(ctx, true, testBlock(pub))
|
||||
at(m, t0+20*60) // in the future again: a NEW window starts, it does not continue the old one
|
||||
m.Watchdog(ctx, true, testBlock(pub))
|
||||
if localPub(t, m) != pub {
|
||||
t.Fatal("a future stamp that did not STAY for rekeyAfter must not re-key")
|
||||
}
|
||||
}
|
||||
|
||||
func TestD30_NoReKeyWhileTheTunnelIsDown(t *testing.T) {
|
||||
fh := &fakeHub{}
|
||||
m, _, active, _ := testManager(t, fh)
|
||||
pub := establishWG(t, m, active)
|
||||
*active = false
|
||||
ctx := context.Background()
|
||||
t0 := int64(hs - 3*3600)
|
||||
for _, d := range []int64{0, 20 * 60, 40 * 60} {
|
||||
at(m, t0+d)
|
||||
m.Watchdog(ctx, true, testBlock(pub))
|
||||
}
|
||||
if localPub(t, m) != pub {
|
||||
t.Fatal("no re-key while the tunnel unit is not active")
|
||||
}
|
||||
}
|
||||
|
||||
func TestD30_AFailedRegistrationIsRetriedByTheExistingPath(t *testing.T) {
|
||||
fh := &fakeHub{}
|
||||
m, _, active, _ := testManager(t, fh)
|
||||
oldPub := establishWG(t, m, active)
|
||||
ctx := context.Background()
|
||||
t0 := int64(hs - 3*3600)
|
||||
at(m, t0)
|
||||
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||
fh.mu.Lock()
|
||||
fh.err = errors.New("hub down")
|
||||
fh.mu.Unlock()
|
||||
at(m, t0+16*60)
|
||||
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||
newPub := localPub(t, m)
|
||||
if newPub == oldPub {
|
||||
t.Fatal("the key rotates before the registration")
|
||||
}
|
||||
if mk := m.loadMarker(); mk == nil || mk.Pubkey != oldPub {
|
||||
t.Fatal("a failed registration leaves the marker on the old key")
|
||||
}
|
||||
fh.mu.Lock()
|
||||
fh.err = nil
|
||||
fh.mu.Unlock()
|
||||
at(m, t0+30*60) // past the backoff: Apply sees the served (old) key ≠ the local key → re-key-in-place registers
|
||||
m.Apply(ctx, true, testBlock(oldPub))
|
||||
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
|
||||
t.Fatalf("the existing re-key-in-place path must register the new key: %+v", mk)
|
||||
}
|
||||
}
|
||||
|
||||
// Review 2026-10-23 night. N7: a second rotation keeps the FIRST old key as `private.key.prev` (the escrow may hold it)
|
||||
// and the later one beside it; N6: the current key is replaced by a rename OVER it, so `private.key` is never missing.
|
||||
// RED-PROOF: with the „prev exists → a dated name" branch removed, TwoRotations fails (the first key overwritten) —
|
||||
// seen failing 2026-10-23 night.
|
||||
func TestD30_TwoRotationsKeepTheFirstOldKey(t *testing.T) {
|
||||
fh := &fakeHub{}
|
||||
m, _, active, _ := testManager(t, fh)
|
||||
establishWG(t, m, active)
|
||||
first := readKey(t, m, keyFileName)
|
||||
old := rekeyNow
|
||||
defer func() { rekeyNow = old }()
|
||||
rekeyNow = func() time.Time { return time.Unix(hs, 0) }
|
||||
if _, err := rotateKey(m.stateDir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second := readKey(t, m, keyFileName)
|
||||
rekeyNow = func() time.Time { return time.Unix(hs+7200, 0) }
|
||||
if _, err := rotateKey(m.stateDir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if readKey(t, m, prevKeyName) != first {
|
||||
t.Fatal("private.key.prev must keep the FIRST old key")
|
||||
}
|
||||
if readKey(t, m, fmt.Sprintf("%s.%d", prevKeyName, hs+7200)) != second {
|
||||
t.Fatal("the second old key must be kept beside it")
|
||||
}
|
||||
if cur := readKey(t, m, keyFileName); cur == first || cur == second {
|
||||
t.Fatal("the current key must be new")
|
||||
}
|
||||
}
|
||||
|
||||
// N7: the once-an-hour bound survives an agent restart (it was in memory only). RED-PROOF: with loadLastRekey removed
|
||||
// from clockAheadRekeyLocked, this fails — seen failing 2026-10-23 night.
|
||||
func TestD30_TheHourBoundSurvivesARestart(t *testing.T) {
|
||||
fh := &fakeHub{}
|
||||
m, rr, active, _ := testManager(t, fh)
|
||||
pub := establishWG(t, m, active)
|
||||
ctx := context.Background()
|
||||
t0 := int64(hs - 3*3600)
|
||||
at(m, t0)
|
||||
m.Watchdog(ctx, true, testBlock(pub))
|
||||
at(m, t0+16*60)
|
||||
m.Watchdog(ctx, true, testBlock(pub))
|
||||
newPub := localPub(t, m)
|
||||
if newPub == pub {
|
||||
t.Fatal("fixture: the first re-key must happen")
|
||||
}
|
||||
// The agent restarts: a NEW manager on the same state; the served block now names the new key; the sign is back.
|
||||
m2 := NewManager(rr, fh, m.stateDir, m.logger)
|
||||
m2.isActive, m2.wgPresent, m2.resolver = m.isActive, m.wgPresent, m.resolver
|
||||
m2.Apply(ctx, true, testBlock(newPub))
|
||||
at(m2, t0+20*60)
|
||||
m2.Watchdog(ctx, true, testBlock(newPub))
|
||||
at(m2, t0+40*60)
|
||||
m2.Watchdog(ctx, true, testBlock(newPub))
|
||||
if localPub(t, m2) != newPub {
|
||||
t.Fatal("a restart must not reset the once-an-hour bound")
|
||||
}
|
||||
_ = active
|
||||
}
|
||||
|
||||
// Commit security review 2026-10-23 night: a last re-key saved while the clock ran ahead (a later date than now) must
|
||||
// not block the next re-key until that date. RED-PROOF: with the „lastRekey after now → dropped" guard removed, this
|
||||
// fails — seen failing 2026-10-23 night.
|
||||
func TestD30_ALastReKeyFromTheFutureDoesNotBlock(t *testing.T) {
|
||||
fh := &fakeHub{}
|
||||
m, _, active, _ := testManager(t, fh)
|
||||
pub := establishWG(t, m, active)
|
||||
m.saveLastRekey(time.Unix(hs+2*365*86400, 0)) // written while the clock was two years ahead
|
||||
ctx := context.Background()
|
||||
t0 := int64(hs - 3*3600)
|
||||
at(m, t0)
|
||||
m.Watchdog(ctx, true, testBlock(pub))
|
||||
at(m, t0+16*60)
|
||||
m.Watchdog(ctx, true, testBlock(pub))
|
||||
if localPub(t, m) == pub {
|
||||
t.Fatal("a last re-key dated in the future must not block the re-key")
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,9 @@ Gates (all must pass; **non-zero exit on any failure**):
|
||||
or because someone is offline on a train. CI runs the FULL set for exactly this
|
||||
reason: it is the machine that can afford a network check, and it is the half
|
||||
that emails when something is wrong.
|
||||
… security-scan govulncheck (called vulnerabilities) + gitleaks on the pushed commits, WARN-FIRST: it lists,
|
||||
it never refuses (R-986). Shared: felhom.eu/scripts/security_scan_gate.py. In the fast set
|
||||
although it reads vuln.go.dev — offline it says NOT CHECKED, so it cannot stop a push.
|
||||
… gate-decoys the decoy suite (test_gate_decoys.py): every gate here still convicts its decoys
|
||||
(R-1116). ~20 s in the workspace, so the full run only: not the hook, but CI
|
||||
runs the full set on every push. Scratch clones and a local fake Gitea only.
|
||||
@@ -49,6 +52,9 @@ SHARED_INSTRUCTIONS = os.path.join(
|
||||
# R-389 — shared, like the two above: it lives in felhom.eu/scripts/ and is never copied.
|
||||
SHARED_OBSERVATIONS = os.path.join(
|
||||
os.path.dirname(ROOT), "felhom.eu", "scripts", "observations_gate.py")
|
||||
# R-986 — shared too: the dependency and secret scanner, warn-first.
|
||||
SHARED_SECURITY = os.path.join(
|
||||
os.path.dirname(ROOT), "felhom.eu", "scripts", "security_scan_gate.py")
|
||||
|
||||
# (label, absolute script path, args, fast)
|
||||
GATES = [
|
||||
@@ -67,6 +73,13 @@ GATES = [
|
||||
# runs the FULL set, so every push is still judged by it. It plants only in scratch clones and asks a fake
|
||||
# Gitea on 127.0.0.1 (never the network, never the real tree); it does not call this runner (no recursion).
|
||||
("gate-decoys", os.path.join(ROOT, "scripts", "test_gate_decoys.py"), [], False),
|
||||
# R-1160 (2026-10-19) — every Go file is gofmt-clean; the controller's gofmt_gate.py shape. Fast: one gofmt -l.
|
||||
# Where the runner has no Go it says NOT CHECKED out loud and passes; the pre-push hook (with Go) enforces it.
|
||||
("gofmt", os.path.join(ROOT, "scripts", "gofmt_gate.py"), [], True),
|
||||
# R-986 (2026-10-19) — govulncheck + gitleaks, WARN-FIRST (exit 0 whatever it finds). Fast although it reads
|
||||
# vuln.go.dev: it cannot refuse, so a network blink is a NOT CHECKED line. ~4 s here. CI: gitleaks fetched pinned
|
||||
# (FELHOM_SCAN_FETCH=1), no Go, so govulncheck says NOT CHECKED there.
|
||||
("security-scan", SHARED_SECURITY, [ROOT], True),
|
||||
]
|
||||
|
||||
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env python3
|
||||
"""gofmt_gate.py — every Go file in this module is gofmt-clean (R-1160; the controller's R-454 gate, same shape).
|
||||
|
||||
MEASURED 2026-10-19: 14 agent files were not gofmt-clean and nothing noticed (fixed by one formatting-only commit,
|
||||
1edfd8e). The controller's history below is why a gate and not a one-off clean-up.
|
||||
|
||||
MEASURED 2026-09-02: five internal/web test files had been gofmt-unclean for an unknown length of
|
||||
time and nothing noticed — `go vet` does not check formatting and no gate did. By 2026-10-05 the five
|
||||
were clean and TWELVE others were not: the count only grows while nothing looks, and every `gofmt -l`
|
||||
someone runs by hand is then noise that hides a real one.
|
||||
|
||||
Run from the repo root: python3 scripts/gofmt_gate.py
|
||||
Exit 0 clean · 1 convicted (lists the files) · 2 inconclusive (gofmt not found — never a pass).
|
||||
|
||||
gofmt is the Go toolchain's own formatter (not a shell utility), found on PATH or under
|
||||
`go env GOROOT`/bin. Decoy: scripts/test_gate_decoys.py ("gofmt"), in a scratch copy of this script.
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
ROOTS = ["cmd", "internal"]
|
||||
|
||||
|
||||
def find_gofmt():
|
||||
p = shutil.which("gofmt")
|
||||
if p:
|
||||
return p
|
||||
go = shutil.which("go")
|
||||
if go:
|
||||
try:
|
||||
root = subprocess.check_output([go, "env", "GOROOT"], text=True).strip()
|
||||
cand = os.path.join(root, "bin", "gofmt")
|
||||
if os.path.isfile(cand):
|
||||
return cand
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def in_ci():
|
||||
"""True on the Gitea Actions runner (act sets GITHUB_ACTIONS, Gitea sets GITEA_ACTIONS)."""
|
||||
return os.environ.get("GITEA_ACTIONS") == "true" or os.environ.get("GITHUB_ACTIONS") == "true"
|
||||
|
||||
|
||||
def main():
|
||||
gofmt = find_gofmt()
|
||||
if not gofmt:
|
||||
# The CI runner may have no Go toolchain (felhom.eu CLAUDE.md, R-887 note). There the gate
|
||||
# says so OUT LOUD and passes: the pre-push hook on the developer's machine, where Go always exists, is where
|
||||
# formatting is enforced. Anywhere else a missing gofmt stays INCONCLUSIVE — never a silent pass.
|
||||
# Pinned by the "gofmt: CI without Go" and "gofmt: dev without Go" decoys in test_gate_decoys.py.
|
||||
if in_ci():
|
||||
print("gofmt gate NOT CHECKED in CI: no Go toolchain on the runner — the pre-push hook (with Go) enforces it")
|
||||
return 0
|
||||
print("gofmt gate INCONCLUSIVE: gofmt not found on PATH or under `go env GOROOT`/bin")
|
||||
return 2
|
||||
roots = [r for r in ROOTS if os.path.isdir(os.path.join(ROOT, r))]
|
||||
if not roots:
|
||||
print("gofmt gate INCONCLUSIVE: none of %s exists under %s" % (ROOTS, ROOT))
|
||||
return 2
|
||||
p = subprocess.run([gofmt, "-l"] + roots, cwd=ROOT, capture_output=True, text=True)
|
||||
if p.returncode != 0:
|
||||
print("gofmt gate INCONCLUSIVE: gofmt exited %d: %s" % (p.returncode, p.stderr.strip()[:400]))
|
||||
return 2
|
||||
bad = [l for l in p.stdout.splitlines() if l.strip()]
|
||||
if bad:
|
||||
for f in bad:
|
||||
print(" not gofmt-clean: %s" % f)
|
||||
print("GOFMT GATE FAILED: %d file(s) — run `gofmt -w <file>` (formatting only, no behaviour change)" % len(bad))
|
||||
return 1
|
||||
print("gofmt gate OK — every Go file under %s is gofmt-clean" % ", ".join(roots))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -66,6 +66,14 @@ COVERS = {
|
||||
"inside an HTML comment (R-426)",
|
||||
"observations": "R-419 in THIS repo's REPORT.md: an Observations note SAYING it carries no marker - vs the "
|
||||
"two genuine markers (R-426)",
|
||||
"gofmt": "R-1160: an unformatted .go file planted under internal/ of a SCRATCH copy is convicted by name - vs the "
|
||||
"same tree formatted (passes); with no gofmt reachable, CI passes OUT LOUD and a dev machine is "
|
||||
"INCONCLUSIVE, never a silent pass",
|
||||
"security-scan": "R-986: the shared gate's own decoys (felhom.eu scripts/test_security_scan_gate.py, 8 cases in "
|
||||
"SCRATCH git repos: a planted token listed by rule and file, its value never printed; a second "
|
||||
"secret on another line of an accepted file still listed; a called govulncheck finding listed, "
|
||||
"an imported-only one not; no tool is NOT CHECKED out loud), run from here; plus the gate on THIS "
|
||||
"repo's tree: exit 0 and its summary line (warn-first - the listing is the fact)",
|
||||
"gate-decoys": "this suite, run as a gate of agent_gates.py's full run, which CI runs on every push (R-1116): "
|
||||
"every case above is a decoy of its own gate, and ONE case that stops convicting turns the run "
|
||||
"red - red-proved 2026-10-17 by flipping a reuse-refs FACT case to expect rc 0",
|
||||
@@ -291,6 +299,32 @@ def release_cases(base, ws):
|
||||
|
||||
|
||||
# ── the shared felhom.eu scripts, against THIS repo's inputs ─────────────────────────────────────
|
||||
def gofmt_cases(ws):
|
||||
"""R-1160: the gofmt gate in a scratch root (its ROOT is its own grandparent directory), never the real tree."""
|
||||
root = os.path.join(ws, "gofmt-root")
|
||||
os.makedirs(os.path.join(root, "scripts"))
|
||||
os.makedirs(os.path.join(root, "internal", "x"))
|
||||
shutil.copy(os.path.join(ROOT, "scripts", "gofmt_gate.py"), os.path.join(root, "scripts", "gofmt_gate.py"))
|
||||
gate = os.path.join(root, "scripts", "gofmt_gate.py")
|
||||
planted = os.path.join(root, "internal", "x", "zz_gofmt_decoy.go")
|
||||
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F( ) int {return 1}\n")
|
||||
if shutil.which("gofmt") or shutil.which("go"):
|
||||
rc, out = run([sys.executable, gate], root)
|
||||
report("gofmt: FACT: an unformatted file under internal/", rc, out, 1, ("zz_gofmt_decoy.go", "GOFMT GATE FAILED"))
|
||||
io.open(planted, "w", encoding="utf-8").write(u"package x\n\nfunc F() int { return 1 }\n")
|
||||
rc, out = run([sys.executable, gate], root)
|
||||
report("gofmt: GENUINE: the same file formatted", rc, out, 0, ("gofmt gate OK",))
|
||||
else:
|
||||
print(" -- gofmt: not runnable here (no Go toolchain) - the two cases below pin that mode")
|
||||
for ci, want_rc, want, name in ((True, 0, "NOT CHECKED in CI", "gofmt: CI without Go"),
|
||||
(False, 2, "INCONCLUSIVE", "gofmt: dev without Go")):
|
||||
env = {"PATH": os.path.join(ws, "empty-path"), "HOME": ws}
|
||||
if ci:
|
||||
env["GITEA_ACTIONS"] = "true"
|
||||
rc, out = run([sys.executable, gate], root, env=env)
|
||||
report(name, rc, out, want_rc, (want,))
|
||||
|
||||
|
||||
def shared_cases(ws):
|
||||
"""A scratch WORKSPACE: a clone of this repo beside symlinks to the siblings, because the shared
|
||||
scripts reach across (REUSE.md cites hub paths; instructions_gate reads the workspace CLAUDE.md)."""
|
||||
@@ -343,6 +377,20 @@ def shared_cases(ws):
|
||||
u"the same minute.**\n", 0)
|
||||
|
||||
|
||||
def security_cases():
|
||||
"""R-986: the shared scanner's decoys (they build their own scratch repos), then the gate on this repo."""
|
||||
suite = os.path.join(SHARED, "test_security_scan_gate.py")
|
||||
gate = os.path.join(SHARED, "security_scan_gate.py")
|
||||
for f in (suite, gate):
|
||||
if not os.path.isfile(f):
|
||||
fails.append("shared %s is MISSING beside this clone - a failure, never a skip" % f)
|
||||
return
|
||||
rc, out = run([sys.executable, suite], ROOT)
|
||||
report("security-scan: the shared gate's 8 decoys", rc, out, 0, ("all 8 cases passed",))
|
||||
rc, out = run([sys.executable, gate, ROOT], ROOT)
|
||||
report("security-scan: GENUINE: this repo, warn-first", rc, out, 0, ("security-scan:", "govulncheck"))
|
||||
|
||||
|
||||
def main():
|
||||
srv = Server(("127.0.0.1", 0), Handler)
|
||||
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
||||
@@ -353,6 +401,8 @@ def main():
|
||||
published_cases(base)
|
||||
release_cases(base, ws)
|
||||
shared_cases(ws)
|
||||
gofmt_cases(ws)
|
||||
security_cases()
|
||||
finally:
|
||||
srv.shutdown()
|
||||
srv.server_close()
|
||||
|
||||
Reference in New Issue
Block a user