Compare commits

...

16 Commits

Author SHA1 Message Date
admin 6ab1e7c56c R-861: narrow the agent's root grants — exact sudo patterns, felhom-priv-apply content checker, fixed hook/parent files in the bundle, signed self-update verified as root, escrow root reads pinned
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:00:43 +02:00
admin 61345790ed REPORT: the 2026-10-05 catch-up session
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 10:29:12 +02:00
admin 7c4b8e599f v0.145.0: CHANGELOG (released 894da35c…, bundle 78c00adc…)
gates / gates (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 09:23:48 +02:00
admin 56ef1d6655 v0.145.0 code: the OS wrapper repairs dpkg's update journal by itself after a power cut (R-876); restore-test first check 30 min after start (R-874); neutral "sent late" text (R-875)
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 09:23:23 +02:00
admin 78c890e4bb REPORT: the 2026-10-05 night-fixes session
gates / gates (push) Successful in 21s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 08:25:49 +02:00
admin d48f1bbb23 v0.144.1: CHANGELOG (released 6ccd521d…, bundle e89a9ddf…)
gates / gates (push) Successful in 21s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 07:49:29 +02:00
admin 8401a30917 v0.144.1 code: the wrapper survives a dead reader (BrokenPipe) so a killed pass still keeps its report; the agent looks for kept copies every 5 min (R-868, measured live)
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 07:49:04 +02:00
admin d1b6004458 v0.144.0: CHANGELOG (released f18093c3…, bundle 6acf42fe…)
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 07:15:07 +02:00
admin ca78c17b29 v0.144.0 code: R8 measures the real download (R-865); an OS pass's report survives a killed agent (R-868); the debug pass runs from the saved block when the hub is away (R-866)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 07:14:32 +02:00
admin c8d12f1f2a REPORT + CONTEXT: 2026-10-04 night (R-840 / R-860)
gates / gates (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 20:39:47 +02:00
admin dc9164c5af CHANGELOG v0.143.0 (R-840); build-golden.sh 3.2.0: GOLDEN_GUEST_PKGS — the approved guest release at bake time, first-night count
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 20:08:16 +02:00
admin c9fa2e717b R-840: the config bundle — a signed agent_config_update brings a box's root-owned files (sudoers, wrappers, units)
gates / gates (push) Successful in 18s
felhom-os-apply gains mode 'bundle' (signed, verified by the wrapper itself against the root-owned
signers file — or, when that file is missing, only the installer's pinned key, which it then creates)
and --install-bundle (the installer's root entry). BUNDLE_FILES is the one table of paths; every check
(visudo, sh/bash -n, python, unit sections, RuntimeDirectory guard, nft -c, the route itself) runs
before the first write; a failed write or self-check puts every previous copy back. The trust root is
never a bundle path (R17). scripts/build-config-bundle.py builds it reproducibly; release-agent.sh
publishes it beside the binary. The agent reports the bundle record in system.config_bundle.
felhom-opsign signs agent_config_update. 43 wrapper tests (22 mutants red), Go executor tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 19:36:58 +02:00
admin 0af1187e03 CHANGELOG + REPORT: v0.142.1 released (R-858)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 18:21:46 +02:00
admin 495003051b R-858: after a Docker engine step the wrapper restarts ONLY the containers that mount the docker socket (controller, traefik); the health rule fails when the controller cannot reach Docker from inside its container (ruling 95)
gates / gates (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 18:21:24 +02:00
admin 42af3ab9bc build-golden.sh 3.1.0: live-restore on in the golden (fail-closed assertion), GOLDEN_DOCKER_PKGS pins the approved Docker engine set
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 17:30:19 +02:00
admin f24dce5b95 CHANGELOG + REPORT: v0.142.0 released
gates / gates (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 16:15:15 +02:00
60 changed files with 4474 additions and 495 deletions
+149
View File
@@ -1,3 +1,152 @@
## v0.145.0 — the OS update repairs itself after a power cut; a short-session box gets restore-tested; "sent late" (R-876, R-874, R-875) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `894da35c7b9e1ac78885690b78352b634e6831e7d99b321573c75d878db8886e`,
config bundle sha256 `78c00adce662d2d966b2ac50ebde46cde1ae225f0107c6a7c02b70ec8ce80c4f`. The wrapper changed: a box
needs the signed `agent_update` AND the signed `agent_config_update`.
- **R-876.** After a crash during an install, `dpkg --audit` can read clean while dpkg's update journal
(`/var/lib/dpkg/updates/`) is not — and apt refuses every install until `dpkg --configure -a` (measured on demo-hp
2026-10-05: every later pass failed until a person typed it). The wrapper now reads `--audit` and the journal in ONE
`sh -c` call (`DPKG_STATE_SCRIPT`) — a clean pass still costs one call (R-845's speed, pinned) — and repairs when
either shows something; as a belt, when apt itself says "dpkg was interrupted", it repairs and retries the install
ONCE. `REPAIR` now logs `journal=N`; a journal still not empty after the repair refuses (R13). Tests
`CrashLeftTheJournal` (the measured shape, the speed, the belt); 3 red-proofs.
- **R-874.** The restore-test's first due-check runs 30 minutes after the agent starts (`DefaultFirstEval`), then every
interval; a box whose power-on sessions are shorter than the 6 h interval never evaluated. A crash-looping agent
restarting faster than 30 minutes still never evaluates (the earned restraint, pinned).
- **R-875.** A kept report's reason is neutral — "sent late — kept on the box until the hub could take it" — the copy
cannot tell a killed agent from an absent hub.
- Red-proofs: `felhom.eu/documentation/audits/catchup-2026-10-05/part{C,D}/`.
## v0.144.1 — a killed pass really keeps its report: the wrapper survives a dead reader; the agent looks again every 5 minutes (R-868, measured live) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `6ccd521d47e64999017e8eb5bc613d724543cfdc5ef9b13bcae9e3ea8c53b8f3`,
config bundle sha256 `e89a9ddfb767e177f8874d56f3dcd3bfd47409d157ff830d362653333bf815e8`. The wrapper changed again:
a box needs the signed `agent_update` AND the signed `agent_config_update`.
- **Found live on demo-hp 2026-10-05 05:45 UTC with v0.144.0** (the night's A5 shape: kill -9 of the pass and the
daemon while apt-get ran): apt finished all 13 packages, but the wrapper's next log line went to a stderr pipe no
process read any more → `BrokenPipeError` → the wrapper died before it saved its report copy (journal: `PLAN
upgrade=13`, then nothing; no copy; the hub got nothing). v0.144.0's mechanism was right and never reached.
`Runner.log` and the final `OSAPPLY-REPORT` line now survive a dead reader (the journal still gets every line).
Test `AgentDiesMidPass` drives the REAL `log()` into a pipe that breaks while apt-get runs.
- **Also found live:** the restarted daemon looked for kept copies ~7 s before the orphaned wrapper wrote one. The
daemon now looks at start and every 5 minutes (`Leg.SendUnsentLoop`); `TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop`.
- Red-proofs: `felhom.eu/documentation/audits/night-fixes-2026-10-05/partD/r868-brokenpipe-red-proof.txt`.
- A second agent release in one session, against "one release per repo": recorded as `09` decision 108 (operator may
reverse) — the alternative was to ship a fix proven not to work.
## v0.144.0 — R8 measures the real download; an OS pass reports even when its agent was killed; the debug pass runs with the hub away (R-865, R-868, R-866) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `f18093c3466749ec4cd47f83f97a401160a24bad1e704f1183051adad14db928`,
config bundle `felhom-config-bundle.json` sha256 `6acf42fe46df5223384d767801cb2bf73238ba4dab82debb7811ca2f790591d8`.
The wrapper `felhom-os-apply` changed, so a box needs BOTH the signed `agent_update` and the signed `agent_config_update`.
- **R-865.** `download_bytes` runs `apt-get --print-uris` WITHOUT `-s`: with `-s` apt prints the simulation and no URI
list, so R8 summed 0 B and only its 500 MB floor ever applied. `--print-uris` alone downloads nothing (measured on
9202: the archive cache and the versions unchanged). The test fake now answers like real apt (with `-s`: no URIs),
and `test_R8_counts_the_real_download` / `test_download_bytes_never_simulates` pin it.
- **R-868.** The wrapper writes every apply pass's report to `<plan dir>/report-<run>-<layer>-apply.json` before it
prints it (root writes into the agent's dir: the dir opened O_NOFOLLOW and checked to be the agent's own, the file
created O_EXCL|O_NOFOLLOW, 0600, handed to the agent). The plan now carries `run_id`, `trigger`, `ring`, echoed in
the report. The agent deletes the copy once the hub has the report; a copy left on disk (the agent was killed, or
the hub was away) is sent at the agent's start and before every pass (`Leg.SendUnsent`), then deleted. A pass lock
(flock on `pass.lock`, across the daemon and a selftest) keeps the sender off a pass that is still running.
- **R-866.** The daemon saves the hub's newest os_update block (`os-update-block.json`); `--selftest=os-update` uses it
when the hub cannot be reached and says so in its header (`block=SAVED(<time>; hub unreachable: …)`); with no hub
and nothing saved it does not run.
- Tests: `configs/test_felhom_os_apply.py` (UnsentReport, SaveReportOnDisk — real files, symlink cases), Go
`TestR868_*`, `TestR866_*`. Red-proofs: `felhom.eu/documentation/audits/night-fixes-2026-10-05/part{C,D}/`.
## v0.143.0 — the config bundle: a signed route for a box's root-owned files (R-840, decision 96) (2026-10-04)
Released by `scripts/release-agent.sh`: binary sha256 `41c0d3060013dfda795262147454248149bee0888c0935170fdb85de6e7a35da`,
config bundle `felhom-config-bundle.json` sha256 `8d7273cf5313ef62b867cb6f831c631923a436452d6f90b8ff7f0771170396ba`.
- **The bundle.** Every root-owned file the installer's step 5 writes (sudoers ×2, the five wrappers, the crash guard
and its units, the agent and rollback units, the start-limit drop-in, the mgmt watchdog, the OOB belt's files) as ONE
reproducible JSON file, built by `scripts/build-config-bundle.py` from `BUNDLE_FILES` in `configs/felhom-os-apply`
(one table) and published beside the binary. The installer (1.31.0) installs the same file.
- **The route.** A signed `agent_config_update` {agent_version, bundle_sha256} (`felhom-opsign -op agent_config_update
-bundle-sha256 …`). The agent is the courier (downloads, checks the sha, hands over); `felhom-os-apply` mode `bundle`
verifies it ITSELF: the operator signature against the root-owned `/etc/felhom/operator-signers` (or, when that file
is missing, ONLY the installer's pinned key, after which it creates the file with exactly that key), the host binding,
the window, its own nonce; the bundle sha; every path in `BUNDLE_FILES` (R16) and never a trust file (R17); every
content check before the first write (visudo, sh/bash -n, python, unit sections, the RuntimeDirectory guard, User=,
nft -c, and that the route itself survives). Atomic per file, previous copies kept under
`/var/lib/felhom-os-apply/bundle-prev/`; a self-check after (visudo -c, `sudo -l` lists the route, the new wrapper's
`--self-check`, the self-update wrapper's usage, the crash guard's status = kernel.panic); any failure puts every
previous copy back. A newly installed crash guard is started (`enable --now`: kernel.panic for this boot, no reboot).
Record `/etc/felhom/config-bundle.json`; the agent reports it as `system.config_bundle`, the facts mode adds drift.
- **Bootstrap.** A box whose `felhom-os-apply` predates 0.143.0 cannot take the first bundle by the route (nothing on it
can write a root file from a signed job): `felhom.eu/scripts/felhom-bundle-bootstrap.sh` is the one by-hand step.
- **`build-golden.sh` 3.2.0** (not part of the binary): `GOLDEN_GUEST_PKGS` brings the template to exactly the
approved guest release (only installed packages, never newer, never a removal or a new package) and prints the
first-night count.
- Tests: `configs/test_felhom_config_bundle.py` (43; 22 of 22 mutants red), `internal/osupdate/bundle_test.go`,
`internal/hub/bundle_record_test.go`. Live: `felhom.eu/documentation/audits/r840-config-bundle-2026-10-04/partB/`.
## v0.142.1 — a Docker step no longer leaves the controller and traefik blind (R-858, `09` decision 95)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.142.1` (`4950030`), sha256
> `003f882a59abfc10021a1f97a4744ab78ab1e916b1392dcef132e7067f3c62bd`, verified by download. Not vouched at release time.
**MinAgent impact:** none. A same-day patch release, by the operator's ruling 95 after a live incident.
- **The incident.** v0.142.0's Docker step on demo-felhom (14:13 UTC) restarted dockerd, which recreates
`/run/docker.sock`. live-restore kept every container running — including `felhom-controller` and `traefik`, which
bind-mount the socket FILE and so kept the deleted inode. The controller could not reach Docker for 1 h 44 min (hub:
DOWN, "docker not reachable"); its own health check stayed "healthy", so the step's health rule PASSED.
- **The fix.** After a Docker step that installed something, the wrapper restarts ONLY the containers that mount
`/var/run/docker.sock` or `/run/docker.sock` (`os-apply: SOCKET-USERS restarted=…`; the apps and the engine untouched).
The guest health reading gains `controller_docker_ok` (`docker exec felhom-controller docker version`), and the health
rule fails when it is false — the consequence, not the mechanism.
- Proven live on demo-hp BEFORE the release (signed undo to 29.7.2): the two restarted, guest / controller / traefik on
the same socket inode, healthy. Red-proofs: `felhom.eu/documentation/audits/os-docker-crash-2026-10-04/partE-incident/`.
## v0.142.0 — the Docker engine slow lane, the version report, the crash guard (`11` §5.8, §5.9; `09` decisions 87–89)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.142.0` (`b1746c2`), sha256
> `7beb32224d6495e9561acfd3ad8a48393a799520f196080011cb27fceb6d1de6`, verified by download. Not vouched at release time.
**MinAgent impact:** none. **Needs hub v0.132.0** for the System page, the Docker approval and the crash events; an older
hub stores the new `system` stanza unread. **Needs the new root files** on an installed box (R-840): the wrapper,
`/etc/felhom/os-trust.json`, `/etc/felhom/operator-signers` and the crash guard — the installer 1.30.0 writes them; the
demo boxes got them by hand.
- **Docker `live-restore` ON** (decision 87). Wrapper mode `live-restore-on`: merge `"live-restore": true` into the
guest's `/etc/docker/daemon.json` and `systemctl reload docker` — never a restart (R-835). An invalid daemon.json is
left alone (R16); a reload that does not enable it puts the old file back. The leg runs it once before a Docker step;
`--selftest=live-restore -vmid N` runs it by hand. Measured: demo-hp 24 containers, demo-felhom 5 — the same ids after.
- **The Docker engine slow lane** (`11` §5.8). Wrapper layer `docker`, lane `slow` only, the six Docker packages only,
origin `Docker CE` only (R2; a Docker package in a fast-lane plan is refused). **R3 — the wrapper checks the authority
itself**, never the agent's config: a ring-1 step or ANY undo needs a signed `os_docker_step` it verifies with
`ssh-keygen -Y verify` against the ROOT-owned `/etc/felhom/operator-signers` (namespace `felhom-op-v1`, the blob's
`key_id`), bound to `/etc/felhom/os-trust.json` `host_id`, inside its time window, never replayed (a root-owned nonce
file), with exactly the signed packages and undo flag; an unsigned ring-0 step needs that file's
`"ring0_slow_lane": true` (the demo boxes only, set by hand). **R15:** live-restore must be on. An undo may downgrade
(`--allow-downgrades`) only inside a signed job. The leg: ring 0 runs the Docker step at night after a healthy guest
and host step (`select pending-docker`); ring 1 never does — only `DockerStepExecutor` (signed job, heavy-op gate).
**Health:** the guest rule + every container running at the start has the SAME id after + the engine reports the
installed version; a changed id is `health_failed`. Measured ring 0: 29.7.x → 29.8.2 on both demo boxes, every id kept.
- **The version report** (R-852, decision 89). Wrapper mode `facts` (read-only): host Debian, running and next-boot
kernel (`next_entry` > saved default > newest installed, by dpkg order), held packages (R-848), kernel taint (oops,
warn), `kernel.panic`, the crash guard state; guest Debian, Docker engine, containerd, live-restore. The host report
gains `system {pve_version, kernel_version, vmid, facts, facts_error}`, read at most every 10 min (~2 s);
`--selftest=os-facts -vmid N`. A value nobody could read is `unknown`.
- **R-849:** the guest is scanned for "restart needed" on every pass too, so a guest restart clears it.
- **The crash guard** (decision 88, R-851): `configs/felhom-crash-guard` + `felhom-crash-guard.service` (early boot;
its ExecStop writes a clean-stop marker) + an hourly re-arm timer + `/etc/felhom/crash-guard.conf`. A boot without
the marker followed an unclean stop (a crash, a power cut or a hard reset — pstore saved nothing for a real panic on
demo-hp, so they cannot be told apart). Armed: `kernel.panic = 10`. After the 2nd unclean boot within 60 min it
TRIPS (`kernel.panic = 0`), so the 3rd crash within the hour leaves the box off; it re-arms after 24 h of normal
running or `felhom-crash-guard rearm`. State in `/var/lib/felhom-crash-guard/state.json` (0644; read by facts).
- **After the tag (main only, not shipped to boxes): `configs/build-golden.sh` 3.1.0** — the golden's `daemon.json`
carries `"live-restore": true` with a fail-closed assertion, and `GOLDEN_DOCKER_PKGS` pins the approved Docker engine
set (all six `name=version`); without it the bake log warns that the set is the newest, not an approved one.
- Tests: wrapper 76 (DockerLane, LiveRestore, Facts, RealSignatureCheck with a throwaway key), crash guard 9, Go leg +
executor; red-proofs `felhom.eu/documentation/audits/os-docker-crash-2026-10-04/partB/agent-redproofs.txt` (17 caught).
## v0.141.1 — "reboot needed" is true on the host (found live on demo-felhom, 2026-10-04)
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.141.1` (`a6bc3f1`), sha256
+8
View File
@@ -1,5 +1,13 @@
# CONTEXT — felhom-agent working state
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
> `release-agent.sh` publishes it. A box whose `felhom-os-apply` predates 0.143.0 needs ONE by-hand bootstrap
> (`felhom.eu/scripts/felhom-bundle-bootstrap.sh`) — done on both demo boxes; Tester 2 waits for the operator (R-862).
> Both demo boxes: agent 0.143.0, bundle 0.143.0 (record `/etc/felhom/config-bundle.json`). R-861 found: the sudoers is
> root-equivalent. `build-golden.sh` 3.2.0 (`GOLDEN_GUEST_PKGS`). Runbook `felhom.eu/documentation/runbooks/config-bundle.md`.
> **2026-09-25 night — v0.133.0 AND v0.134.0 DELIVERED to both demo boxes (CC-signed `agent_update`, ruling 1);
> restore test back ON (the `-1` config kept as `agent.json.night-0925-off`). v0.134.0 = R-685:** `backup/runner.go`
+12 -9
View File
@@ -1,11 +1,14 @@
# REPORT — 2026-10-04: v0.141.0 + v0.141.1, the host fast lane, the true tunnel status, the fast leg
# REPORT — agent v0.145.0 (2026-10-05, afternoon): the OS update repairs itself after a power cut
Full session report: `felhom.eu/REPORT-os-host-lane-2026-10-04.md`.
Brief: the 2026-10-05 catch-up brief (operator), Parts C (R-874, R-875) and D (R-876). Full session report:
`felhom.eu/REPORT-catchup-2026-10-05.md`. Architecture: `11-os-updates.md` §5.4.1, §8.4–8.5; `09` decisions 117–118.
- Tunnel (R-841): the agent reads the guest's cloudflared container and its health check; three states.
- Host fast lane: the wrapper gains the host layer (R12 appliance proof from the root-owned install record, R14 no
kernel/boot/firmware); the leg runs the host step after a healthy guest step; host health rule.
- Speed (R-845): one call per layer instead of one per package; measured before/after in the session report.
- Tests green; red-proofs in the audit folder.
- v0.141.1 (same day): the host "reboot needed" scan hid `lxc-start` and was never cleared by a reboot — both fixed,
found live on demo-felhom, red-proved.
| Row | Fix | Proof |
|---|---|---|
| R-876 | the wrapper reads `dpkg --audit` AND the update journal in ONE call, repairs on either; belt: repair + retry once when apt says "interrupted" | 3 tests + 3 red-proofs; **live (operator's go): crash mid-unpack on demo-hp → the next pass `REPAIR … journal=1` → `DONE rc=0 upgraded=12`, nobody touched the box** |
| R-874 | the restore-test's first due-check 30 min after start | 2 tests + red-proof; live on demo-felhom: passed restore-test at start + 30 min |
| R-875 | a kept report's reason is neutral ("sent late …") | test + red-proof |
Released by `scripts/release-agent.sh`: v0.145.0 (`894da35c…`, bundle `78c00adc…`), verified by download; signed
`agent_update` + `agent_config_update` to demo-hp, demo-felhom, tester-1 (71/71 after each bundle); vouched with
golden 0.295.0, min_agent 0.131.0. `go test ./...` rc 0, Python suites OK, `agent_gates.py` OK.
+9 -2
View File
@@ -14,8 +14,15 @@
| `Privileged` (CreateGoldenLXC/MountUSBByUUID/SMART/Sensors) | internal/proxmox/privileged.go | methods on `*Privileged` | the 3 fenced root-CLI exceptions ONLY | Do NOT add methods — fence is structural (`routing_test.go` asserts it) |
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
| `stageTemp` | internal/localapi/intermediary.go | `stageTemp(pattern, content) (path, err)` | random-named temp before a root `install` (audit B1) | Fixed /tmp names are a TOCTOU — sudoers globs expect `/tmp/felhom-*-*.ext` |
| `guesthook.InstallSnippet` / `Register` | internal/guesthook/install.go | `InstallSnippet(ctx, runner) error` | pre-start self-heal hook install (C1 net) | Same random-temp+install pattern; snippet delegates to the agent binary (no shell logic). Issues `mkdir -p /var/lib/vz/snippets` FIRST (v0.63.0, B2 — fresh boxes lack the dir; sudoers grants exactly that argv) |
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key` | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
| `osupdate.Leg.SendUnsent` / `lockPass` (v0.144.0, R-868) | internal/osupdate/unsent.go | `(ctx) int` | an OS-pass report the agent never sent (killed mid-pass): the wrapper keeps `report-<run>-<layer>-apply.json` beside the plan; the agent deletes it once the hub has it | any new caller that runs an apply pass must hold `lockPass` (flock, across processes) — the sender must never take a running pass's copy |
| `osupdate.LoadSavedBlock` (v0.144.0, R-866) | internal/osupdate/leg.go | `(planDir) (block, savedAt, ok)` | the hub's newest os_update block as the daemon last received it (`os-update-block.json`) | the debug pass uses it ONLY when the hub cannot be reached, and says so in its header; no saved block → no pass |
| `dpkg_state()` / `DPKG_STATE_SCRIPT` (v0.145.0, R-876) | configs/felhom-os-apply | `audit, journal = self.dpkg_state()` | dpkg's state in ONE call: `--audit` AND the update journal | never gate a repair on `--audit` alone — a crash leaves only the journal (measured); keep it one call (R-845) |
| `guesthook.SnippetReady` / `Register` (v0.146.0, R-861) | internal/guesthook/install.go | `SnippetReady(path) error` | is the pre-start hook (a FIXED file from the bundle) in place — register only then | The agent never installs the hook (Proxmox runs it as root); a missing hookscript stops a guest start, so never `Register` without `SnippetReady` |
### Disk / format safety (role gates, durable IDs, format guards)
+56 -7
View File
@@ -22,6 +22,7 @@ import (
"os/exec"
"os/signal"
"path/filepath"
"regexp"
"strconv"
"strings"
"sync"
@@ -868,6 +869,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
logger.Info("felhom-agent daemon starting",
"version", version, "host_id", cfg.Hub.HostID, "hub_url", cfg.Hub.URL,
"interval_s", hcfg.PollSeconds) // hub key intentionally not logged
// R-868 (v0.144.0): an OS pass whose agent was killed kept its report on disk — send it now (a pass that starts
// first sends it itself; the pass lock keeps the two apart).
// v0.144.1: and again every 5 minutes — the wrapper of a killed pass can finish AFTER the restart (measured).
go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) {
logger.Info("osupdate: sent kept report(s)", "count", n)
})
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
@@ -1101,7 +1108,18 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
return release, nil
}}
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec}, cfg.Hub.HostID, logger)
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
AfterInstall: func(ctx context.Context) {
ok, total, degraded := capability.Summarize(probeAll(ctx))
names := make([]string, 0, len(degraded))
for _, d := range degraded {
names = append(names, d.Name)
}
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
}}
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, bundleExec}, cfg.Hub.HostID, logger)
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
@@ -2680,6 +2698,9 @@ func (e *escrowCeremonyErr) Error() string { return e.err.Error() }
// restic password auto-attach), Create (R + self-verified blob), wipe the staged secret, upload
// when asked. It PRINTS NOTHING — the output-mode shells own every byte of stdout/stderr. R is
// returned for the caller to surface exactly once; escrow.Create never logs it and neither do we.
// pbsStorageIDRe is a PVE storage id (letters, digits, '-', '_', '.'; starts with a letter) — never a path (R-861).
var pbsStorageIDRe = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_.-]{0,63}$`)
func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger, opts escrowCeremonyOpts) (escrowCeremonyOutcome, *escrowCeremonyErr) {
var out escrowCeremonyOutcome
storage := opts.storage
@@ -2689,6 +2710,16 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
if storage == "" {
return out, &escrowCeremonyErr{kind: "usage", err: fmt.Errorf("selftest=escrow-create requires -storage <pbs-storage-id> (or escrow.pbs_storage_id)")}
}
// R-861 (v0.146.0): this runs as ROOT through FELHOM_ESCROW, but agent.json is owned by the agent user. So the
// paths a root run reads never come from it: the PVE secret dir and the WireGuard state dir are the fixed defaults,
// and the storage id is a plain PVE id (no slash, no dot-dot) — a crafted id or dir would read another root file.
if os.Geteuid() == 0 {
cfg.Backup.PBSSecretDir = ""
cfg.WGTunnel.StateDir = ""
}
if !pbsStorageIDRe.MatchString(storage) {
return out, &escrowCeremonyErr{kind: "usage", err: fmt.Errorf("selftest=escrow-create: storage id %q is not a plain PVE storage id", storage)}
}
out.Storage = storage
keyPath := cfg.Backup.PBSEncKeyPath(storage)
if _, err := os.Stat(keyPath); err != nil {
@@ -3564,15 +3595,15 @@ func runSelftestOSUpdate(ctx context.Context, cfg config.Config, logger *slog.Lo
return 1
}
leg := newOSLeg(cfg, client, px, logger)
resp, err := client.FetchDesiredState(ctx)
if err != nil {
fmt.Fprintln(os.Stderr, "selftest=os-update: desired state:", err)
blk, source, ok := selftestOSBlock(ctx, client, leg.PlanDir)
if !ok {
fmt.Fprintln(os.Stderr, "selftest=os-update:", source)
return 1
}
leg.SetBlock(resp.DesiredState.OSUpdate)
leg.SetBlock(blk)
b := leg.Block()
fmt.Printf("=== felhom-agent %s selftest=os-update vmid=%d ring=%d enabled=%v guest-release=%v host-release=%v appliance=%v ===\n",
version, vmid, b.Ring, b.Enabled, b.Release != nil, b.HostRelease != nil, leg.Appliance)
fmt.Printf("=== felhom-agent %s selftest=os-update vmid=%d ring=%d enabled=%v guest-release=%v host-release=%v appliance=%v block=%s ===\n",
version, vmid, b.Ring, b.Enabled, b.Release != nil, b.HostRelease != nil, leg.Appliance, source)
start := time.Now()
pass := leg.Run(ctx, vmid, "debug")
worst := pass.Guest
@@ -3598,6 +3629,24 @@ func runSelftestOSUpdate(ctx context.Context, cfg config.Config, logger *slog.Lo
return 1
}
// selftestOSBlock is the debug pass's os_update block (R-866, v0.144.0): the hub's, fetched fresh; when the hub cannot
// be reached, the block the daemon last saved — named in the selftest's first line, so a pass with the hub away can
// be exercised by hand (the daemon's own leg already ran from its last block; the selftest stopped). No saved block
// and no hub → not run (ok=false), never a guessed block.
func selftestOSBlock(ctx context.Context, f interface {
FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error)
}, planDir string) (*hub.WireOSUpdate, string, bool) {
resp, err := f.FetchDesiredState(ctx)
if err == nil {
return resp.DesiredState.OSUpdate, "hub", true
}
saved, at, ok := osupdate.LoadSavedBlock(planDir)
if !ok {
return nil, fmt.Sprintf("desired state: %v — and no saved block (the daemon saves one when the hub sends it)", err), false
}
return saved, fmt.Sprintf("SAVED(%s; hub unreachable: %v)", at.UTC().Format(time.RFC3339), err), true
}
// runSelftestFacts prints the versions the host report carries (R-852, agent v0.142.0) — read-only.
//
// sudo -u felhom-agent felhom-agent --config … --selftest=os-facts -vmid 9201
@@ -0,0 +1,31 @@
package main
import (
"context"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// R-861 (agent v0.146.0): the root escrow ceremony builds a file path from the storage id; an id that is a path is
// refused before anything is read. RED-PROOF: drop the pbsStorageIDRe check → the "../" ids reach the key stat and
// come back as a "setup" error instead of "usage".
func TestEscrowCeremony_StorageIDIsNeverAPath(t *testing.T) {
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
for _, id := range []string{"../../../etc/shadow", "a/b", "/etc/pve/priv/x", ".hidden", ""} {
cfg := config.Default()
cfg.Escrow.PBSStorageID = "" // the flag decides here
_, e := escrowCeremony(context.Background(), cfg, lg, escrowCeremonyOpts{storage: id})
if e == nil || e.kind != "usage" {
t.Errorf("storage id %q was not refused as usage (got %+v)", id, e)
}
}
cfg := config.Default()
cfg.Backup.PBSSecretDir = t.TempDir()
_, e := escrowCeremony(context.Background(), cfg, lg, escrowCeremonyOpts{storage: "felhom-pbs"})
if e == nil || e.kind != "setup" {
t.Fatalf("control: a plain id must pass the check and fail later on the missing key (setup), got %+v", e)
}
}
@@ -0,0 +1,55 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
)
type r866Fetcher struct{ err error }
func (f r866Fetcher) FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error) {
if f.err != nil {
return nil, f.err
}
r := &hub.DesiredStateResponse{}
r.DesiredState.OSUpdate = &hub.WireOSUpdate{Ring: 1, Enabled: true}
return r, nil
}
// R-866 (v0.144.0). THE NIGHT'S SHAPE (A3, Tester 1 box, hub blocked): `selftest=os-update: desired state: hub:
// transport error … connect: invalid argument` — the debug pass could not run at all. Now it runs from the block the
// daemon saved, and its header says so.
// COMPANION RED-PROOF: return at once on a fetch error in selftestOSBlock → "the pass did not run from the saved block".
func TestR866_DebugPassUsesTheSavedBlockWhenTheHubIsAway(t *testing.T) {
dir := t.TempDir()
leg := &osupdate.Leg{PlanDir: dir}
r := &hub.DesiredStateResponse{}
r.DesiredState.OSUpdate = &hub.WireOSUpdate{Ring: 0, Enabled: true}
leg.OnDesiredState(context.Background(), r) // the daemon received a block and saved it
away := r866Fetcher{err: errors.New("hub: transport error: connect: invalid argument")}
b, src, ok := selftestOSBlock(context.Background(), away, dir)
if !ok || b == nil || b.Ring != 0 {
t.Fatalf("the pass did not run from the saved block: ok=%v block=%+v src=%q", ok, b, src)
}
if !strings.HasPrefix(src, "SAVED(") || !strings.Contains(src, "hub unreachable") {
t.Fatalf("the header must say the block is the saved one: %q", src)
}
// the hub reachable: its block wins, and the header says "hub"
b, src, ok = selftestOSBlock(context.Background(), r866Fetcher{}, dir)
if !ok || b.Ring != 1 || src != "hub" {
t.Fatalf("hub block not used: %+v %q", b, src)
}
}
// No hub and nothing saved: the pass does not run on a guessed block.
func TestR866_NoHubNoSavedBlockDoesNotRun(t *testing.T) {
_, why, ok := selftestOSBlock(context.Background(), r866Fetcher{err: errors.New("down")}, t.TempDir())
if ok || !strings.Contains(why, "no saved block") {
t.Fatalf("ok=%v why=%q", ok, why)
}
}
+13 -1
View File
@@ -43,7 +43,7 @@ func main() {
func run() error {
var (
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update")
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | agent_config_update")
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
@@ -52,6 +52,7 @@ func run() error {
fstype = flag.String("fstype", "ext4", "for storage_wipe: the filesystem to mkfs after wipe")
agentVer = flag.String("agent-version", "", "for agent_update: the target agent version (e.g. 0.70.1)")
sha256Hex = flag.String("sha256", "", "for agent_update: the pinned lowercase-hex sha256 of the target binary")
bundleSHA = flag.String("bundle-sha256", "", "for agent_config_update: the pinned sha256 of felhom-config-bundle.json (R-840)")
keyFile = flag.String("key", "", "operator signing key (ssh private key / sk- key handle) for ssh-keygen -Y sign")
ttl = flag.Duration("ttl", 30*time.Minute, "validity window from now (issued_at..expires_at)")
nonce = flag.String("nonce", "", "explicit nonce (default: a fresh 128-bit random nonce)")
@@ -95,6 +96,17 @@ func run() error {
}
pj, _ := json.Marshal(map[string]string{"version": *agentVer, "sha256": *sha256Hex})
params = string(pj)
case "agent_config_update":
// R-840: the box's root-owned files. The ROOT wrapper verifies this signature itself and refuses a bundle
// whose sha256 is not exactly this one.
if *agentVer == "" || *bundleSHA == "" {
return fmt.Errorf("agent_config_update needs -agent-version and -bundle-sha256 (the pinned bundle hash)")
}
if !isHex64(*bundleSHA) {
return fmt.Errorf("agent_config_update -bundle-sha256 must be 64 lowercase hex chars (got %d)", len(*bundleSHA))
}
pj, _ := json.Marshal(map[string]string{"agent_version": *agentVer, "bundle_sha256": *bundleSHA})
params = string(pj)
default:
params = "{}"
}
+63 -3
View File
@@ -61,7 +61,7 @@ set -euo pipefail
# Script provenance — logged into every bake transcript next to the baked controller tag, so an
# archive can always be traced to the script that produced it. Bump on any behavior change.
GOLDEN_SCRIPT_VERSION="3.0.0"
GOLDEN_SCRIPT_VERSION="3.2.0"
VMID="${1:-9100}"
TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
@@ -112,7 +112,15 @@ for i in $(seq 1 30); do
if pct exec "$VMID" -- getent hosts download.docker.com >/dev/null 2>&1; then break; fi
sleep 1
done
pct exec "$VMID" -- bash -c '
# v3.1.0 (`11` §5.8): GOLDEN_DOCKER_PKGS pins the APPROVED Docker engine set (the hub's newest Docker release, all six
# "name=version"); without it the newest stable set is installed and the bake log says so.
GOLDEN_DOCKER_PKGS="${GOLDEN_DOCKER_PKGS:-}"
if [[ -n "$GOLDEN_DOCKER_PKGS" ]]; then
echo "[golden] Docker engine set PINNED to the approved release: $GOLDEN_DOCKER_PKGS"
else
echo "[golden] WARNING: GOLDEN_DOCKER_PKGS not set — installing the newest stable Docker set, not an approved one"
fi
pct exec "$VMID" -- env GOLDEN_DOCKER_PKGS="$GOLDEN_DOCKER_PKGS" bash -c '
set -e
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
@@ -122,7 +130,54 @@ pct exec "$VMID" -- bash -c '
echo "deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update -qq
apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
if [ -n "$GOLDEN_DOCKER_PKGS" ]; then
apt-get install -y -qq $GOLDEN_DOCKER_PKGS >/dev/null
else
apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
fi
dpkg-query -W containerd.io docker-buildx-plugin docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin 2>/dev/null | sed "s/^/ installed: /"
'
# v3.2.0 (Part F of the R-840 brief, `11` §5.3): GOLDEN_GUEST_PKGS = the newest APPROVED guest release, as
# "name=version …" (the hub's os_releases row for layer guest, IN FORCE — never a cancelled test approval). The bake
# brings every package the template HAS to exactly that version, under felhom-os-apply's rules: never a package the
# template lacks (--only-upgrade), never newer than approved, never a removal or a new package (a simulation is checked
# first and the bake FAILS on either). Empty = no approved guest release in force: the template's versions stay, and
# the box's first night installs whatever release is approved then. Either way the bake PRINTS the first-night count:
# how many installed packages are older than the approved version (target 0).
GOLDEN_GUEST_PKGS="${GOLDEN_GUEST_PKGS:-}"
pct exec "$VMID" -- env GOLDEN_GUEST_PKGS="$GOLDEN_GUEST_PKGS" bash -c '
set -e
export DEBIAN_FRONTEND=noninteractive
if [ -z "$GOLDEN_GUEST_PKGS" ]; then
echo "[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a"
echo "[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): $(apt list --upgradable 2>/dev/null | grep -c /)"
exit 0
fi
want=""
for nv in $GOLDEN_GUEST_PKGS; do
n=${nv%%=*}; v=${nv#*=}
cur=$(dpkg-query -W -f="\${Version}" "$n" 2>/dev/null) || continue # not in the template: never added
dpkg --compare-versions "$cur" lt "$v" && want="$want $n=$v"
done
if [ -n "$want" ]; then
sim=$(apt-get -s install --only-upgrade -o Dpkg::Options::=--force-confold $want)
if echo "$sim" | grep -q "^Remv "; then echo "[golden] FATAL: the approved guest set would REMOVE a package"; echo "$sim" | grep "^Remv "; exit 1; fi
for p in $(echo "$sim" | awk "/^Inst /{print \$2}"); do
dpkg-query -W "$p" >/dev/null 2>&1 || { echo "[golden] FATAL: the approved guest set would ADD $p - not in the template"; exit 1; }
done
apt-get install -y -qq --only-upgrade -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef $want >/dev/null
echo "[golden] approved guest release installed: $(echo $want | wc -w) package(s) brought to the approved version"
else
echo "[golden] approved guest release: the template already runs every approved version"
fi
left=0
for nv in $GOLDEN_GUEST_PKGS; do
n=${nv%%=*}; v=${nv#*=}
cur=$(dpkg-query -W -f="\${Version}" "$n" 2>/dev/null) || continue
dpkg --compare-versions "$cur" lt "$v" && { left=$((left+1)); echo " still older: $n $cur < $v"; }
done
echo "[golden] first-night count vs the approved guest release: $left (target 0)"
[ "$left" -eq 0 ] || { echo "[golden] FATAL: $left package(s) stayed older than the approved release"; exit 1; }
'
echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …"
# containerd-snapshotter (Docker 28+/29 default) keeps the IMAGE content store under
@@ -135,9 +190,12 @@ echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshott
# layout (a container's `df /` reports the single volume, phase-0 spike). Since v3.0.0 /var/lib/docker
# is a BIND of <volume>/docker rather than the mp0 mount itself, wired immediately below; data-root
# still needs no override because the path is unchanged. Log caps kill the most common runaway.
# v3.1.0: "live-restore": true (`09` decision 87) — a Docker engine update then restarts no app (`11` C5). A box made
# from this golden never needs the agent's one-time live-restore-on step. NEVER removed by a plain restart (R-835).
pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
{
"features": { "containerd-snapshotter": false },
"live-restore": true,
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
@@ -175,6 +233,8 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
# mean containerd-snapshotter is still on (the split would leave images on the rootfs).
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
# v3.1.0 ASSERTION: live-restore is ON in the running daemon (decision 87), or the bake fails closed.
pct exec "$VMID" -- bash -c 'lr=$(docker info --format "{{.LiveRestoreEnabled}}" 2>/dev/null); [ "$lr" = "true" ] && echo " live-restore: on" || { echo "[golden] FATAL: live-restore is $lr, expected true (decision 87)"; exit 1; }'
# ASSERTION 1 (RETARGETED v3.0.0, not removed). /var/lib/docker must be a real mount — now the V-c
# bind of <volume>/docker rather than the mp0 mount itself. Still fails closed on the same failure:
# if the bind did not take, Docker's data-root silently sits on the OS rootfs and the golden ships
+95 -103
View File
@@ -1,30 +1,38 @@
# felhom-agent sudoers allowlist — the NARROW host-root surface (slice 5 Phase B, doc 03 §3/§7).
# felhom-agent sudoers allowlist — the NARROW host-root surface (slice 5 Phase B, doc 03 §3/§7; narrowed R-861).
#
# Install as a drop-in: /etc/sudoers.d/felhom-agent (mode 0440, root:root), validated with
# `visudo -cf`. The agent runs as the non-root `felhom-agent` service user and shells out via
# `sudo -n` with FIXED argument vectors (no shell). The fine-grained validation is done IN
# the agent BEFORE exec (internal/storage/validate.go): UUIDs against a strict hex regex,
# mount paths confined+traversal-checked, SMART devices whitelisted to raw disks, LVM names
# charset-checked. These sudoers wildcards are the COARSE allowlist; the agent is the fine
# gate, so a wildcard can never be abused by a value the agent didn't already validate.
# Install as a drop-in: /etc/sudoers.d/felhom-agent (mode 0440, root:root), validated with `visudo -cf`. It rides the
# signed config bundle (R-840). The agent runs as the non-root `felhom-agent` user and shells out via `sudo -n` with
# FIXED argument vectors (no shell).
#
# Binary paths MUST match the agent config (privileged.systemctl/install/smartctl/lvs). Adjust
# for your distro (Debian/PVE shown). A missing/declined entry degrades the agent with a
# warning (SMART→UNKNOWN, mount→logged error), it does not crash.
# R-861 (agent v0.146.0) — EXACT PATTERNS, NOT GLOBS. A sudoers `*` in the ARGUMENTS also matches spaces, so
# `pct set [0-9]* -onboot 1` matched `pct set 100 --dev0 /dev/sda -onboot 1` (a raw host disk for the guest), and
# `mount --bind /mnt/*/felhom-data /mnt/felhom-drives/*` matched a `..` path onto /etc. Every argument list that varies
# is now a sudo regular expression (`^...$`, sudo >= 1.9.10; Debian 13 / PVE 9 ship 1.9.16): one value per slot, a
# fixed character set, no `..`, no extra argument. Lines with no variable part stay literal. The patterns are pinned
# by the capability manifest (every real call must match: TestManifestCoveredBySudoers) and by injection cases that
# must NOT match (configs/test_sudoers_patterns.py, and live with `sudo -l -U felhom-agent` on the demo boxes).
#
# R-861 — NO FILE THE AGENT WROTE IS INSTALLED WHERE ROOT READS IT. The `install` lines are gone: a mount unit, a
# dnsmasq drop-in, the WireGuard config and the OOB sshd config + key go through `felhom-priv-apply`, a root wrapper
# from the bundle that checks the CONTENT against the agent's own renderers; the guest pre-start hook and the shared
# drive parent are FIXED files that come with the bundle itself; the agent binary is replaced only by an
# operator-signed agent_update that `felhom-os-apply` verifies as root (`felhom-selfupdate-guarded apply` is no longer
# here). The two remaining root runs of agent code (FELHOM_ESCROW, the guest hook) therefore run only a signed binary.
#
# Binary paths MUST match the agent config (privileged.systemctl/install/smartctl/lvs). A missing/declined entry
# degrades the agent with a warning (SMART→UNKNOWN, mount→logged error), it does not crash; the capability probe
# reports it to the hub.
Cmnd_Alias FELHOM_MOUNT = \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/* /etc/systemd/system/*.mount, \
/usr/local/sbin/felhom-priv-apply ^unit mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$, \
/usr/bin/systemctl daemon-reload, \
/usr/bin/systemctl enable --now -- *.mount, \
/usr/bin/systemctl disable -- *.mount, \
/usr/bin/systemctl stop -- *.mount
/usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.mount$, \
/usr/bin/systemctl ^stop -- mnt-[A-Za-z0-9_.\\-]+\.mount$
Cmnd_Alias FELHOM_DISK = \
/usr/sbin/smartctl -a -j /dev/sd[a-z]*, \
/usr/sbin/smartctl -a -j /dev/nvme[0-9]*n[0-9]*, \
/usr/sbin/smartctl -a -j /dev/vd[a-z]*, \
/usr/sbin/smartctl -a -j /dev/hd[a-z]*, \
/usr/sbin/lvs --reportformat json --units b -o lv_name\,data_percent\,metadata_percent -- *, \
/usr/sbin/smartctl ^-a -j /dev/(sd[a-z]+|nvme[0-9]+n[0-9]+|vd[a-z]+|hd[a-z]+)$, \
/usr/sbin/lvs ^--reportformat json --units b -o lv_name\,data_percent\,metadata_percent -- [A-Za-z0-9_.+-]+(/[A-Za-z0-9_.+-]+)?$, \
/usr/sbin/pvs --reportformat json --noheadings -o pv_name, \
/usr/sbin/zpool status -P
@@ -35,9 +43,9 @@ Cmnd_Alias FELHOM_DISK = \
# (the wildcard only ever names a path the agent itself created), and the bootstrap file the agent
# writes there is the only thing these touch. ':' is escaped per sudoers grammar.
Cmnd_Alias FELHOM_PROVISION = \
/usr/bin/chown -R 100000\:100000 /var/lib/felhom-agent/guests/*, \
/usr/sbin/pct set [0-9]* -mp[0-9]* /var/lib/felhom-agent/guests/*, \
/usr/sbin/pct set [0-9]* -onboot 1
/usr/bin/chown ^-R 100000\:100000 /var/lib/felhom-agent/guests/[0-9]+(/bootstrap)?$, \
/usr/sbin/pct ^set [0-9]+ -mp[0-9]+ /var/lib/felhom-agent/guests/[0-9]+/bootstrap\,mp\=/[A-Za-z0-9/_.-]+(\,ro\=1)?$, \
/usr/sbin/pct ^set [0-9]+ -onboot 1$
# Disk inspection + format (slice 8C + Impl-1). blkid/lsblk read the device's data-bearing evidence
# (the agent decides data-bearing-ness from THIS, never the caller's claim). Format goes ONLY through
@@ -45,66 +53,54 @@ Cmnd_Alias FELHOM_PROVISION = \
# mkfs the OS disk — the wrapper re-checks the catastrophic cases (system disk / LVM PV / foreign mount)
# as root and refuses, and the agent's unclaimed-disk filter (claim.go) is the primary guard above it.
Cmnd_Alias FELHOM_FORMAT = \
/usr/sbin/blkid -p -o export /dev/*, \
/usr/bin/lsblk -J -o NAME\,FSTYPE\,PTTYPE\,MOUNTPOINT /dev/*, \
/usr/local/sbin/felhom-mkfs-guarded /dev/* *
/usr/sbin/blkid ^-p -o export /dev/[^ ]+$, \
/usr/bin/lsblk ^-J -o NAME\,FSTYPE\,PTTYPE\,MOUNTPOINT /dev/[^ ]+$, \
/usr/local/sbin/felhom-mkfs-guarded ^/dev/[^ ]+ (ext4|xfs)$
# LAN split-horizon resolver (internal/lanresolver): the agent manages a host-side dnsmasq that
# answers *.<customer-domain> with each guest's live LAN IP. install only ever writes felhom-*.conf
# drop-ins (from agent-written /tmp temp files); the two `pct exec` reads are FIXED command vectors
# answers *.<customer-domain> with each guest's live LAN IP. A felhom-*.conf drop-in reaches /etc/dnsmasq.d
# only through felhom-priv-apply, which allows exactly the lines the resolver renders (R-861: a `dhcp-script=` would
# run as root); the two `pct exec` reads are FIXED command vectors
# (the guest's eth0 IPv4 + the controller's pulled controller.yaml for the domain) — NOT a general
# `pct exec`. systemctl is scoped to the dnsmasq unit only. The agent never edits /etc/resolv.conf.
Cmnd_Alias FELHOM_DNSMASQ = \
/usr/bin/apt-get install -y -q dnsmasq, \
/usr/bin/install -m 0644 /tmp/felhom-resolver-*.conf /etc/dnsmasq.d/felhom-*.conf, \
/usr/local/sbin/felhom-priv-apply ^dnsmasq /tmp/felhom-resolver-[0-9]+\.conf felhom-[a-z0-9][a-z0-9._-]*\.conf$, \
/usr/bin/systemctl enable --now dnsmasq, \
/usr/bin/systemctl reload dnsmasq, \
/usr/bin/systemctl restart dnsmasq, \
/usr/bin/rm -f /etc/dnsmasq.d/felhom-*.conf, \
/usr/sbin/pct exec [0-9]* -- ip -4 -o addr show dev eth0, \
/usr/sbin/pct exec [0-9]* -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller.yaml
/usr/bin/rm ^-f /etc/dnsmasq\.d/felhom-[a-z0-9][a-z0-9._-]*\.conf$, \
/usr/sbin/pct ^exec [0-9]+ -- ip -4 -o addr show dev eth0$, \
/usr/sbin/pct ^exec [0-9]+ -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller\.yaml$
# Guest mountpoint lifecycle (intermediary-mount re-architecture + C1 net). The pre-start self-heal hook
# wrapper is installed once into the PVE snippets dir (from an agent-written /tmp file) and registered
# per-guest; decommission/eject DELETE the dead mountpoint slot so a missing bind source can't brick the
# guest at next boot (the B3 C1 fix). The agent fine-validates the vmid (numeric) + slot (mp[0-9]+) and
# the snippet path is fixed — the wildcards are the coarse allowlist. The install SOURCE is a
# random-named agent temp (os.CreateTemp, audit B1 — a fixed /tmp name was a local TOCTOU), hence the
# glob; the DESTINATION stays pinned. The `mkdir -p` creates the snippets dir on a FRESH box —
# `install` won't create parents, so without it the hook install failed silently on Day-0 boxes
# (B2, DRILL-day0-cleanroom-2026-07-03; fixed agent v0.63.0).
# Guest mountpoint lifecycle (intermediary-mount re-architecture + C1 net). The pre-start self-heal hook is a FIXED file
# from the config bundle (/var/lib/vz/snippets/felhom-guest-hook.sh — R-861: the agent no longer installs it from /tmp;
# Proxmox runs it as root at every guest start). The agent only registers it per guest, deletes a dead mountpoint slot
# (the B3 C1 fix) and reboots a guest to activate binds — each with an exact vmid / slot.
Cmnd_Alias FELHOM_GUESTHOOK = \
/usr/bin/mkdir -p /var/lib/vz/snippets, \
/usr/bin/install -m 0755 -- /tmp/felhom-guest-hook-*.sh /var/lib/vz/snippets/felhom-guest-hook.sh, \
/usr/sbin/pct set [0-9]* --hookscript local\:snippets/felhom-guest-hook.sh, \
/usr/sbin/pct set [0-9]* --delete mp[0-9]*, \
/usr/sbin/pct reboot [0-9]*
/usr/sbin/pct ^set [0-9]+ --hookscript local\:snippets/felhom-guest-hook\.sh$, \
/usr/sbin/pct ^set [0-9]+ --delete mp[0-9]+$, \
/usr/sbin/pct ^reboot [0-9]+$
# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
# /mnt/felhom-drives (self-bind + make-shared + a boot-persistence systemd unit) and binds/unbinds each
# drive's felhom-data namespace UNDERNEATH it so the change propagates into the running guest live (no
# pct, no reboot). The agent fine-validates the drive name + confines paths before any exec; the trailing
# `*` (matching the comma-laden mp spec) mirrors the existing FELHOM_PROVISION pattern.
# `lxc-info -n <vmid> -p -H` resolves the guest init PID for the GuestSeesMount / bound_under_parent check
# (a READ — the drive-gate's "is the drive live in the guest?" signal); WITHOUT it the non-root agent gets
# an empty PID and reports every drive absent (multi-drive flapping, audit 2026-06-29). `make-private`
# isolates the parent's peer group on FIRST setup only (EnsureSharedParent guards on mountpoint, so it
# never re-churns a live parent); without it the parent stays in root's group and submounts double.
# /mnt/felhom-drives (self-bind + make-shared) and binds/unbinds each drive's felhom-data namespace UNDERNEATH it so the
# change propagates into the running guest live. The boot-persistence script + unit are FIXED files from the config
# bundle (R-861: the agent no longer installs them from /tmp); the agent only enables the unit. A drive name is one
# path segment that cannot start with a dot (no `..`); `lxc-info -n <vmid> -p -H` resolves the guest init PID for the
# GuestSeesMount check; `make-private` isolates the parent's peer group on FIRST setup only.
Cmnd_Alias FELHOM_INTERMEDIARY = \
/usr/bin/mkdir -p /mnt/felhom-drives, \
/usr/bin/mkdir -p /mnt/felhom-drives/*, \
/usr/bin/mkdir -p /mnt/*/felhom-data, \
/usr/bin/chown 100000\:100000 /mnt/*/felhom-data, \
/usr/bin/mkdir ^-p /mnt/felhom-drives/[A-Za-z0-9_-][A-Za-z0-9_.-]*$, \
/usr/bin/mkdir ^-p /mnt/[A-Za-z0-9_-][A-Za-z0-9_.-]*/felhom-data$, \
/usr/bin/chown ^100000\:100000 /mnt/[A-Za-z0-9_-][A-Za-z0-9_.-]*/felhom-data$, \
/usr/bin/mount --bind /mnt/felhom-drives /mnt/felhom-drives, \
/usr/bin/mount --make-shared /mnt/felhom-drives, \
/usr/bin/mount --make-private /mnt/felhom-drives, \
/usr/bin/mount --bind /mnt/*/felhom-data /mnt/felhom-drives/*, \
/usr/bin/umount /mnt/felhom-drives/*, \
/usr/bin/install -m 0755 -- /tmp/felhom-shared-parent-*.sh /usr/local/sbin/felhom-shared-parent.sh, \
/usr/bin/install -m 0644 -- /tmp/felhom-shared-parent-*.service /etc/systemd/system/felhom-shared-parent.service, \
/usr/bin/mount ^--bind /mnt/[A-Za-z0-9_-][A-Za-z0-9_.-]*/felhom-data /mnt/felhom-drives/[A-Za-z0-9_-][A-Za-z0-9_.-]*$, \
/usr/bin/umount ^/mnt/felhom-drives/[A-Za-z0-9_-][A-Za-z0-9_.-]*$, \
/usr/bin/systemctl enable felhom-shared-parent.service, \
/usr/bin/lxc-info -n [0-9]* -p -H, \
/usr/sbin/pct set [0-9]* -mp8 /mnt/felhom-drives*
/usr/bin/lxc-info ^-n [0-9]+ -p -H$, \
/usr/sbin/pct ^set [0-9]+ -mp8 /mnt/felhom-drives\,mp\=/mnt/felhom-drives$
# Controller-swap / managed auto-update (Option A, non-root). The agent owns the in-guest controller
# image SWAP (it survives the controller being killed mid-swap): read the baked image ref, check the
@@ -119,11 +115,11 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# the agent strict-validates the ref (controllerImageRe) before the write.
# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct exec [0-9]* -- cat /etc/felhom-controller-image, \
/usr/sbin/pct exec [0-9]* -- docker image inspect *, \
/usr/sbin/pct exec [0-9]* -- docker inspect -f *, \
/usr/sbin/pct exec [0-9]* -- systemctl restart felhom-controller-bootstrap.service, \
/usr/sbin/pct exec [0-9]* -- tee /etc/felhom-controller-image
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/felhom-controller-image$, \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
@@ -131,7 +127,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
# with no API equivalent (snapshot-delete + start go through the API token); the agent fine-validates the
# vmid (numeric) before exec — the `[0-9]*` is the coarse allowlist.
Cmnd_Alias FELHOM_STALELOCK = \
/usr/sbin/pct unlock [0-9]*
/usr/sbin/pct ^unlock [0-9]+$
# Restore-test scratch teardown (F-LEAK, Campaign 8, v0.110.0). A restore-test whose restore FAILS
# leaves a scratch guest the API token CANNOT destroy: `FelhomAgentGuest` is granted at /pool/felhom and
@@ -160,8 +156,9 @@ Cmnd_Alias FELHOM_SCRATCH_TEARDOWN = \
# into the guest through the existing shared bind (an unprivileged LXC cannot mount NFS/CIFS itself).
# A NAS is NOT a drive — no durable-id, no SMART, no wipe; these grants only install/enable/remove the
# unit pair. The agent fine-validates every value (share name, server, export, uid/gid, creds path) before
# any unit is rendered (internal/storage/netmount.go ValidateNetworkMountSpec); the trailing globs are the
# COARSE allowlist. The `.mount` install/enable/disable/stop reuse FELHOM_MOUNT; this alias adds the
# any unit is rendered (internal/storage/netmount.go ValidateNetworkMountSpec); the unit FILE reaches
# /etc/systemd/system only through `felhom-priv-apply unit` (FELHOM_MOUNT), which requires nosuid,nodev on a network
# share (R-861). The `.mount` enable/disable/stop reuse FELHOM_MOUNT; this alias adds the
# `.automount` variants + the unit-file removal. The unit FILE name is the systemd-escaped mountpoint,
# which always begins `mnt-felhom` (the mountpoint is /mnt/felhom-drives/<name>), so the rm glob is scoped
# to felhom mount units only. mkdir of the mountpoint reuses FELHOM_INTERMEDIARY's /mnt/felhom-drives/*.
@@ -176,51 +173,46 @@ Cmnd_Alias FELHOM_SCRATCH_TEARDOWN = \
# behind (the campaign accumulated 10 stub-shaped leftovers). rmdir ONLY (never rm -rf): it refuses
# a non-empty dir, so unexpected data is preserved, not destroyed — a fail-safe grant.
Cmnd_Alias FELHOM_NETMOUNT = \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/* /etc/systemd/system/*.automount, \
/usr/bin/systemctl enable --now -- *.automount, \
/usr/bin/systemctl disable -- *.automount, \
/usr/bin/systemctl stop -- *.automount, \
/usr/bin/systemctl reset-failed -- mnt-felhom*, \
/usr/bin/rmdir /mnt/felhom-drives/*, \
/usr/bin/rm -f /etc/systemd/system/mnt-felhom*
/usr/bin/systemctl ^enable --now -- mnt-[A-Za-z0-9_.\\-]+\.automount$, \
/usr/bin/systemctl ^disable -- mnt-[A-Za-z0-9_.\\-]+\.automount$, \
/usr/bin/systemctl ^stop -- mnt-[A-Za-z0-9_.\\-]+\.automount$, \
/usr/bin/systemctl ^reset-failed -- mnt-felhom[A-Za-z0-9_.\\-]*\.(mount|automount)$, \
/usr/bin/rmdir ^/mnt/felhom-drives/[A-Za-z0-9_-][A-Za-z0-9_.-]*$, \
/usr/bin/rm ^-f /etc/systemd/system/mnt-felhom[A-Za-z0-9_.\\-]*\.(mount|automount)$
# Offsite WG tunnel (S3, doc 06 §3.3). The agent manages wg-quick@wg-felhom as an agent-managed
# host service (the dnsmasq/lanresolver shape): conf staged in the agent-owned StateDir (never
# /tmp), installed 0600 to the FIXED destination, unit enable/restart/disable. The ONLY wg read
# /tmp), installed 0600 to the FIXED destination by `felhom-priv-apply wg`, which refuses any key renderConf never
# writes (R-861: PostUp/PreUp run as root under wg-quick), unit enable/restart/disable. The ONLY wg read
# is `latest-handshakes` — `wg show <if> dump` is FORBIDDEN everywhere (its interface line
# carries the PRIVATE KEY; the S1 session-log incident). Both install paths are FIXED (no glob):
# the agent has exactly one tunnel conf to manage.
# carries the PRIVATE KEY; the S1 session-log incident). Source and destination are fixed in the wrapper.
Cmnd_Alias FELHOM_WG = \
/usr/bin/apt-get install -y -q wireguard-tools, \
/usr/bin/install -o root -g root -m 0600 -- /var/lib/felhom-agent/wg/wg-felhom.conf /etc/wireguard/wg-felhom.conf, \
/usr/local/sbin/felhom-priv-apply wg, \
/usr/bin/systemctl enable --now wg-quick@wg-felhom, \
/usr/bin/systemctl restart wg-quick@wg-felhom, \
/usr/bin/systemctl disable --now wg-quick@wg-felhom, \
/usr/bin/wg show wg-felhom latest-handshakes
# Agent self-update (TASK D1, SPIKE-agent-selfupdate-2026-07-05). The agent downloads the
# operator-SIGNED binary (sha256 pinned in the signed op — neither hub nor Gitea compromise can
# substitute it), verifies the sha in-process, then hands off to the guarded wrapper, which
# RE-verifies the sha as root, confines the staged path to /var/lib/felhom-agent/selfupdate/,
# performs the A/B flip (atomic same-fs rename, .prev retained) and schedules a detached restart.
# The apply args are a COARSE glob (spike S4b: sudoers fnmatch makes a [a-f0-9]* sha pattern
# first-char-only anyway) — the wrapper's own sha re-verify + path confinement is the real gate.
# `rollback` is normally run by felhom-agent-rollback.service (root, OnFailure=), not via sudo;
# granting it here keeps the verb probe-able (capability self-check) and operator-invokable.
# Agent self-update (TASK D1; R-861). The A/B flip (`felhom-selfupdate-guarded apply`) is NO LONGER the agent's: the
# agent hands the operator-SIGNED agent_update to felhom-os-apply (FELHOM_OSAPPLY, mode agent_update), which verifies
# the signature as root and only then runs the flip. Until v0.146.0 the agent passed the sha itself, so a compromised
# agent could install any binary — the binary FELHOM_ESCROW and the guest hook run as root. `commit` (clear the pending
# marker) and `rollback` (pending-guarded revert, normally run by felhom-agent-rollback.service) stay.
Cmnd_Alias FELHOM_SELFUPDATE = \
/usr/local/sbin/felhom-selfupdate-guarded apply /var/lib/felhom-agent/selfupdate/* *, \
/usr/local/sbin/felhom-selfupdate-guarded commit, \
/usr/local/sbin/felhom-selfupdate-guarded rollback
# Dedicated OOB sshd (TASK H1). The agent manages felhom-sshd like wg-felhom/dnsmasq: it RENDERS the
# config (Port from its claim) + the operator's authorized_keys, validates with `sshd -t`, and reloads
# (never restart-on-change [SF-2]). Both install SOURCES are the agent-owned staged files under
# StateDir; both DESTINATIONS are FIXED. `sshd -t/-T` are the validate/discover reads. The
# (never restart-on-change [SF-2]). Both files reach /etc/felhom-sshd only through felhom-priv-apply (R-861): the config
# must be the ONE template with only the Port varying (an AuthorizedKeysFile the agent owns + `StrictModes no` would be
# a root login), the key file one plain public key without options. `sshd -t/-T` are the validate/discover reads. The
# systemctl verbs are SCOPED to felhom-sshd only. reset-failed precedes a deliberate restart [SF-5].
# NOTHING here can touch the stock sshd, :22, or /etc/ssh.
Cmnd_Alias FELHOM_SSHD = \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/sshd_config /etc/felhom-sshd/sshd_config, \
/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op /etc/felhom-sshd/authorized_keys/felhom-op, \
/usr/local/sbin/felhom-priv-apply sshd-config, \
/usr/local/sbin/felhom-priv-apply sshd-key, \
/usr/sbin/sshd -t -f /var/lib/felhom-agent/felhom-sshd/sshd_config, \
/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config, \
/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config, \
@@ -270,8 +262,8 @@ Cmnd_Alias FELHOM_OOB = \
/usr/sbin/nft list set inet felhom_oob ssh_port, \
/usr/sbin/nft flush set inet felhom_oob operator_ips, \
/usr/sbin/nft flush set inet felhom_oob ssh_port, \
/usr/sbin/nft add element inet felhom_oob operator_ips *, \
/usr/sbin/nft add element inet felhom_oob ssh_port *
/usr/sbin/nft ^add element inet felhom_oob operator_ips \{ [0-9.]+(/[0-9]+)? \}$, \
/usr/sbin/nft ^add element inet felhom_oob ssh_port \{ [0-9]+ \}$
# Escrow ceremony (controller-driven, TASK 2026-07-13; mechanics validated by
# SPIKE-controller-escrow-2026-07-13). ONE fixed argv — sudoers matches the argument vector
@@ -307,9 +299,9 @@ Cmnd_Alias FELHOM_OSAPPLY = \
/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json
Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct exec [0-9]* -- ip route show default, \
/usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \
/usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
/usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
/usr/sbin/pct ^exec [0-9]+ -- ip route show default$, \
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/network/interfaces$, \
/usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \
/usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
# felhom-agent guest pre-start self-heal hook (C1 net). PVE calls: <script> <vmid> <phase>.
/usr/local/bin/felhom-agent guest-hook "$1" "$2" || true
exit 0
+675 -26
View File
@@ -63,6 +63,9 @@ SNAPSHOT_LIST = "/etc/apt/sources.list.d/felhom-os-snapshot.list"
APT_ENV = ["env", "DEBIAN_FRONTEND=noninteractive", "APT_LISTCHANGES_FRONTEND=none", "NEEDRESTART_MODE=l", "LC_ALL=C"]
DPKG_OPTS = ["-o", "Dpkg::Options::=--force-confold", "-o", "Dpkg::Options::=--force-confdef"]
MIN_FREE = 500 * 1024 * 1024
# R-876: dpkg's state in ONE call — `--audit`, a marker line, then the update journal's file names.
JOURNAL_MARK = "@@FELHOM-DPKG-JOURNAL@@"
DPKG_STATE_SCRIPT = "dpkg --audit; echo " + JOURNAL_MARK + "; ls -A /var/lib/dpkg/updates 2>/dev/null; true"
# The installer's ROOT-OWNED record (felhom-host-install.sh `state_set mode`); the agent cannot write it.
INSTALL_STATE = "/var/lib/felhom-install/state.json"
# Kernel, boot and firmware packages are the SLOW lane on the host whatever their origin (`11` C3, §5.2): a host
@@ -86,8 +89,92 @@ SIG_NAMESPACE = "felhom-op-v1"
SIGNED_OP = "os_docker_step"
NONCE_FILE = "/var/lib/felhom-os-apply/nonces.json"
DAEMON_JSON = "/etc/docker/daemon.json"
# R-858 (v0.142.1): a Docker engine step restarts dockerd, which RECREATES the socket file. With live-restore the
# containers keep running — and one that bind-mounts the socket FILE keeps the deleted inode: measured 2026-10-04 on
# demo-felhom, the controller and traefik were blind to Docker for 1h44m. After a step that installed something, the
# wrapper restarts exactly the containers that mount one of these paths (never the apps, never the engine).
DOCKER_SOCKETS = ("/var/run/docker.sock", "/run/docker.sock")
CRASH_GUARD_STATE = "/var/lib/felhom-crash-guard/state.json"
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
# repo's configs/ at the agent tag (scripts/build-config-bundle.py) and published beside the binary. The installer
# installs the SAME bundle through this same code (--install-bundle, root only, never reachable through sudo), so a new
# box and an updated box cannot drift. This table is the ONLY set of paths a bundle may write — a signed bundle naming
# any other path is refused (R16) — and it is also the builder's list (one table). Each entry:
# dest: (source file under configs/, mode, check, policy)
# check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
# unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none.
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
# R-861 (agent v0.146.0): the signed agent update, verified here (mode agent_update), then the A/B flip.
SELFUPDATE_OP = "agent_update"
SELFUPDATE_DIR = "/var/lib/felhom-agent/selfupdate"
SELFUPDATE_WRAPPER = "/usr/local/sbin/felhom-selfupdate-guarded"
BUNDLE_FORMAT = 1
BUNDLE_OP = "agent_config_update"
BUNDLE_RECORD = "/etc/felhom/config-bundle.json" # what the box runs (0644 root; the non-root agent reports it)
BUNDLE_PREV_DIR = "/var/lib/felhom-os-apply/bundle-prev" # the previous copies of every file a bundle replaced
BUNDLE_KEEP = 3
BUNDLE_MAX_BYTES = 4 * 1024 * 1024
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
OOB_DIR = "/etc/felhom-sshd"
BUNDLE_FILES = [
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
("/usr/local/sbin/felhom-pbs-apply", "felhom-pbs-apply", 0o755, "bash", "replace"),
("/usr/local/sbin/felhom-backup-target-apply", "felhom-backup-target-apply", 0o755, "bash", "replace"),
("/usr/local/sbin/felhom-os-apply", "felhom-os-apply", 0o755, "python", "replace"),
("/usr/local/sbin/felhom-crash-guard", "felhom-crash-guard", 0o755, "python", "replace"),
# R-861 (agent v0.146.0): the content checker for every agent-staged file a root program reads, and the two FIXED
# files the agent used to install itself from /tmp (the guest pre-start hook and the shared drive parent).
("/usr/local/sbin/felhom-priv-apply", "felhom-priv-apply", 0o755, "python", "replace"),
("/var/lib/vz/snippets/felhom-guest-hook.sh", "felhom-guest-hook.sh", 0o755, "sh", "replace"),
("/usr/local/sbin/felhom-shared-parent.sh", "felhom-shared-parent.sh", 0o755, "sh", "replace"),
("/etc/systemd/system/felhom-shared-parent.service", "felhom-shared-parent.service", 0o644, "unit", "replace"),
("/etc/systemd/system/felhom-crash-guard.service", "felhom-crash-guard.service", 0o644, "unit", "replace"),
("/etc/systemd/system/felhom-crash-guard-check.service", "felhom-crash-guard-check.service", 0o644, "unit", "replace"),
("/etc/systemd/system/felhom-crash-guard-check.timer", "felhom-crash-guard-check.timer", 0o644, "unit", "replace"),
("/etc/felhom/crash-guard.conf", "crash-guard.conf", 0o644, "plain", "if-absent"),
("/etc/systemd/system/felhom-agent.service", "felhom-agent.service", 0o644, "agent-unit", "replace"),
("/etc/systemd/system/felhom-agent-rollback.service", "felhom-agent-rollback.service", 0o644, "unit", "replace"),
("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", "felhom-agent-limits.conf", 0o644, "dropin", "replace"),
("/usr/local/sbin/felhom-mgmt-watchdog", "felhom-mgmt-watchdog.sh", 0o755, "sh", "replace"),
("/etc/tmpfiles.d/felhom-privsep.conf", "felhom-privsep.tmpfiles", 0o644, "plain", "replace"),
("/etc/systemd/system/felhom-mgmt-watchdog.service", "felhom-mgmt-watchdog.service", 0o644, "unit-nort", "replace"),
("/etc/systemd/system/felhom-mgmt-watchdog.timer", "felhom-mgmt-watchdog.timer", 0o644, "unit-nort", "replace"),
("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
]
BUNDLE_DESTS = {e[0]: e for e in BUNDLE_FILES}
# The trust root is never a bundle's to change (R17): who may sign is decided by these files, so no bundle may carry them.
TRUST_PATHS = (TRUST_FILE, TRUST_SIGNERS, BUNDLE_RECORD)
# When /etc/felhom/operator-signers is MISSING (a box installed before installer 1.30.0), the job is verified against —
# and the file is created with — exactly this key: the operational key felhom-host-install.sh pins
# (OPERATOR_KEY_OPERATIONAL_*). Never any other. Pinned equal to the installer by
# test_pinned_operator_key_equals_the_installers. Signer rotation is a separate, later act (`04` §3).
PINNED_OPERATOR_KEY_ID = "felhom-op-1"
PINNED_OPERATOR_KEY_LINE = ("ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL8z0qCNgA3x2xxAB0Qj5ro8waFjGZ8Ta/sWB63tlLw+ felhom-op-1")
# The self-check (after install, before the record): the route itself must survive — a bundle whose sudoers no longer
# lets the agent call this wrapper would cut the box off from every later bundle.
SELF_CHECK_SUDO_LINE = "/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json"
PINNED_SIGNERS = "<pinned operator key>" # verify_sig: check against the pinned key, not a file on the box
def pinned_signers_line():
"""The ssh allowed_signers line the installer writes (felhom-host-install.sh _write_slow_lane_trust), byte for byte."""
return f'{PINNED_OPERATOR_KEY_ID} namespaces="{SIG_NAMESPACE}" {PINNED_OPERATOR_KEY_LINE}\n'
def sha256_hex(data):
import hashlib
return hashlib.sha256(data).hexdigest()
class Refused(Exception):
def __init__(self, code, reason):
@@ -126,6 +213,32 @@ class Runner:
if rc != 0:
raise Refused("R7", f"could not write {path} in the guest")
def save_report(self, plan_path, report):
"""R-868: keep an apply pass's report on disk until the agent has sent it (the agent deletes it). Written
INTO the agent's own plan dir as root, so: the dir is opened with O_NOFOLLOW and must be a real directory
owned by the agent (a symlink swapped in for it is refused); the file is created O_EXCL|O_NOFOLLOW after
removing an old one, then handed to the agent (0600). Any failure only loses the copy — never the run."""
base = os.path.basename(plan_path)
name = "report-" + base[len("plan-"):]
dfd = os.open(PLAN_DIR, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
try:
st = os.fstat(dfd)
if not stat.S_ISDIR(st.st_mode) or st.st_uid != self.agent_uid():
raise OSError(f"{PLAN_DIR} is not the agent's own directory")
try:
os.unlink(name, dir_fd=dfd)
except FileNotFoundError:
pass
fd = os.open(name, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600, dir_fd=dfd)
try:
os.write(fd, (json.dumps(report, sort_keys=True) + "\n").encode())
os.fchown(fd, self.agent_uid(), -1)
finally:
os.close(fd)
finally:
os.close(dfd)
return os.path.join(PLAN_DIR, name)
def now(self):
return time.time()
@@ -139,6 +252,10 @@ class Runner:
sp = os.path.join(d, "sig")
with open(sp, "w") as f:
f.write(sig)
if signers == PINNED_SIGNERS:
signers = os.path.join(d, "allowed_signers")
with open(signers, "w") as f:
f.write(pinned_signers_line())
p = subprocess.run(["ssh-keygen", "-Y", "verify", "-f", signers, "-I", key_id, "-n", namespace, "-s", sp],
input=blob, capture_output=True, timeout=30)
return p.returncode
@@ -159,12 +276,75 @@ class Runner:
os.replace(tmp, NONCE_FILE)
def log(self, line):
print(line, file=sys.stderr, flush=True)
# R-868 (v0.144.1): the agent that reads stderr may be GONE (killed mid-pass, measured live on demo-hp
# 2026-10-05): the write then raises BrokenPipeError, and v0.144.0 died right there — after apt had installed
# everything, before the report copy was saved. A dead reader must never stop the pass; the journal still
# gets every line. Pinned by AgentDiesMidPass.
try:
print(line, file=sys.stderr, flush=True)
except OSError:
pass
try:
subprocess.run(["logger", "-t", "felhom-os-apply", line], timeout=10)
except Exception:
pass
# ---------- host files, for the config bundle (R-840). Tests replace these with an in-memory tree. ----------
def read_bytes(self, path):
with open(path, "rb") as f:
return f.read()
def lexists(self, path):
return os.path.lexists(path)
def isdir(self, path):
return os.path.isdir(path)
def put_file(self, path, data, mode):
"""Atomic: a root-owned temp file beside the target (same filesystem), fsync, then rename over it."""
d = os.path.dirname(path)
os.makedirs(d, mode=0o755, exist_ok=True)
tmp = os.path.join(d, f".{os.path.basename(path)}.felhom-new.{os.getpid()}")
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
try:
with os.fdopen(fd, "wb") as f:
f.write(data)
f.flush()
os.fchown(f.fileno(), 0, 0)
os.fchmod(f.fileno(), mode)
os.fsync(f.fileno())
os.replace(tmp, path)
except BaseException:
try:
os.remove(tmp)
except OSError:
pass
raise
def remove(self, path):
os.remove(path)
def list_dir(self, path):
try:
return sorted(os.listdir(path))
except OSError:
return []
def rmtree(self, path):
import shutil
shutil.rmtree(path, ignore_errors=True)
def check_content(self, kind, data):
"""Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
import tempfile
cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"]}[kind]
with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
p = os.path.join(d, "f")
with open(p, "wb") as f:
f.write(data)
r = subprocess.run(cmd + [p], capture_output=True, text=True, timeout=60)
return r.returncode, (r.stdout + r.stderr).strip()[-300:]
class Apply:
def __init__(self, runner, plan_path):
@@ -198,8 +378,16 @@ class Apply:
def check_plan(self, plan):
mode = plan.get("mode", "apply")
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on"):
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update"):
raise Refused("R11", f"unknown mode {mode!r}")
if mode == "agent_update":
if plan.get("layer") != "host":
raise Refused("R11", "agent_update is a host-layer mode")
return mode, "host", 0, "agent_update"
if mode == "bundle":
if plan.get("layer") != "host":
raise Refused("R11", "bundle is a host-layer mode")
return mode, "host", 0, "bundle"
layer = plan.get("layer")
if layer not in ("guest", "host", "docker"):
raise Refused("R12", f"layer {layer!r} is not guest, host or docker")
@@ -293,9 +481,12 @@ class Apply:
raise Refused("R3", f"{TRUST_FILE} names no host_id")
return t
def verify_signed(self, signed, trust):
"""R3: an operator-signed os_docker_step, checked HERE (not by the agent): signature against the root-owned
signers file, op, host binding, time window, and a root-owned nonce record (no replay)."""
def verify_signed(self, signed, trust, op_name=SIGNED_OP, signers=None, burn=True):
"""R3: an operator-signed job, checked HERE (not by the agent): signature against the root-owned signers file (or,
for a bundle on a box that has none, the PINNED key — `signers` names that temp file), op, host binding, time
window, and a root-owned nonce record (no replay). burn=False leaves the nonce for the caller to burn after its
own checks (a bundle refused for a wrong sha keeps its job usable — the operator fixes the sha, not the key).
Returns the params; with burn=False, (params, nonce, expiry)."""
import base64
if not isinstance(signed, dict) or not isinstance(signed.get("blob_b64"), str) or not isinstance(signed.get("sig"), str):
raise Refused("R3", "the signed job is malformed")
@@ -307,17 +498,19 @@ class Apply:
key_id = op.get("key_id", "")
if not isinstance(key_id, str) or not re.match(r"^[A-Za-z0-9._-]{1,64}$", key_id):
raise Refused("R3", "the signed blob names no plain key_id")
try:
st = self.r.stat(TRUST_SIGNERS)
except OSError:
raise Refused("R3", f"no {TRUST_SIGNERS} — no operator key to check a signed job against")
if st.st_uid != 0 or (st.st_mode & 0o022):
raise Refused("R3", f"{TRUST_SIGNERS} is not root-owned and root-only-writable")
rc = self.r.verify_sig(TRUST_SIGNERS, key_id, SIG_NAMESPACE, blob, signed["sig"])
if signers is None:
signers = TRUST_SIGNERS
try:
st = self.r.stat(TRUST_SIGNERS)
except OSError:
raise Refused("R3", f"no {TRUST_SIGNERS} — no operator key to check a signed job against")
if st.st_uid != 0 or (st.st_mode & 0o022):
raise Refused("R3", f"{TRUST_SIGNERS} is not root-owned and root-only-writable")
rc = self.r.verify_sig(signers, key_id, SIG_NAMESPACE, blob, signed["sig"])
if rc != 0:
raise Refused("R3", f"the operator signature does not verify (ssh-keygen rc={rc})")
if op.get("op") != SIGNED_OP:
raise Refused("R3", f"the signed op is {op.get('op')!r}, not {SIGNED_OP}")
if op.get("op") != op_name:
raise Refused("R3", f"the signed op is {op.get('op')!r}, not {op_name}")
if (op.get("target") or {}).get("host_id") != trust["host_id"]:
raise Refused("R3", "the signed job is for another host")
now = self.r.now()
@@ -331,12 +524,55 @@ class Apply:
nonce = op.get("nonce")
if not isinstance(nonce, str) or not nonce:
raise Refused("R3", "the signed job has no nonce")
if nonce in self.r.read_nonces():
raise Refused("R3", "the signed job was already used (replay)")
if not burn:
return op.get("params") or {}, nonce, exp
self.burn_nonce(nonce, exp)
return op.get("params") or {}
def agent_update(self, plan):
"""R-861 (agent v0.146.0): the agent binary is replaced ONLY by an operator-signed agent_update, checked HERE as
root — signature against the root-owned signers file, op, this host, the time window, the nonce — and only the
staged file whose sha256 the SIGNED params pin, at the one staging path. Before v0.146.0 the agent handed the
sha to `felhom-selfupdate-guarded apply` itself, so a compromised agent could install any binary — and the
binary is what FELHOM_ESCROW and the guest hook run as root. The nonce is burned only after the flip."""
trust = self.load_trust()
params, nonce, exp = self.verify_signed(plan.get("signed"), trust, op_name=SELFUPDATE_OP, burn=False)
ver, sha = params.get("version"), params.get("sha256")
if not isinstance(ver, str) or not re.match(r"^[0-9]+\.[0-9]+\.[0-9]+$", ver):
raise Refused("R19", f"the signed version {ver!r} is not bare semver")
if not isinstance(sha, str) or not re.match(r"^[0-9a-f]{64}$", sha):
raise Refused("R19", "the signed sha256 is not 64 lowercase hex")
staged = os.path.join(SELFUPDATE_DIR, "felhom-agent-" + ver)
if plan.get("staged") != staged:
raise Refused("R19", f"the staged binary must be {staged}, got {plan.get('staged')!r}")
try:
st = self.r.stat(staged)
except OSError as e:
raise Refused("R19", f"cannot stat the staged binary: {e}")
if not stat.S_ISREG(st.st_mode) or st.st_uid != self.r.agent_uid():
raise Refused("R19", "the staged binary is not a regular file owned by the agent")
got = sha256_hex(self.r.read_bytes(staged))
if got != sha:
raise Refused("R19", f"the staged binary's sha256 {got[:16]}… is not the signed {sha[:16]}…")
self.r.log(f"os-apply: AGENT-UPDATE signed by the operator: version={ver} sha={sha[:16]} — handing to the A/B wrapper")
rc, out, err = self.r.host([SELFUPDATE_WRAPPER, "apply", staged, sha], 120)
self.report["agent_update"] = {"version": ver, "sha256": sha, "wrapper_rc": rc,
"wrapper": (out + err).strip()[-300:]}
if rc != 0:
self.report["failed"] = {"rc": rc, "step": "agent_update", "reason": (out + err).strip()[-300:]}
return 3
self.burn_nonce(nonce, exp)
return 0
def burn_nonce(self, nonce, exp):
seen = self.r.read_nonces()
if nonce in seen:
raise Refused("R3", "the signed job was already used (replay)")
seen[nonce] = exp
now = self.r.now()
self.r.write_nonces({k: v for k, v in seen.items() if v > now})
return op.get("params") or {}
def docker_authority(self, plan):
"""R3 for the docker layer: returns (who, undo). A signed job binds the EXACT package list and the undo flag."""
@@ -476,6 +712,10 @@ class Apply:
for k, src in (("debian", "debian"), ("docker_engine", "engine"), ("containerd", "containerd")):
g[k] = kv.get(src, "").strip() or "unknown"
g["live_restore"] = {"true": "on", "false": "off"}.get(kv.get("live", "").strip(), "unknown")
try:
h["config_bundle"] = Bundle(self).state()
except Exception as e: # a read problem must never cost the System page its other facts
h["config_bundle"] = {"version": "unknown", "error": str(e)[:200]}
self.report["facts"] = {"host": h, "guest": g}
return 0
@@ -575,7 +815,10 @@ class Apply:
if len(p) >= 4 and p[3]:
cont[p[0]]["id"] = p[3]
nrc, _, _ = self.g(["getent", "hosts", "deb.debian.org"], timeout=30)
return {"docker_ok": rc == 0, "containers": cont,
# R-858: the controller's own health check stayed "healthy" while it could not reach Docker at all — so ask
# the consequence directly: can the controller talk to the engine from inside its container?
crc, _, _ = self.g(["docker", "exec", "felhom-controller", "docker", "version", "--format", "{{.Server.Version}}"], timeout=60)
return {"docker_ok": rc == 0, "containers": cont, "controller_docker_ok": crc == 0,
"controller": cont.get("felhom-controller", {}).get("health", "absent"),
"network_ok": nrc == 0}
@@ -650,8 +893,22 @@ class Apply:
plan = self.load_plan()
self.mode, self.layer, self.vmid, self.select = self.check_plan(plan)
self.report.update(mode=self.mode, layer=self.layer, release_id=plan.get("release_id"), vmid=self.vmid)
# R-868 (v0.144.0): the agent's run id, trigger and ring travel in the report, so a report the agent never
# received (it was killed mid-pass) can be sent later from the saved copy. Plain ids only; anything else
# is dropped, never refused (the plan's other checks decide).
rid, trig, ring = plan.get("run_id"), plan.get("trigger"), plan.get("ring")
if isinstance(rid, str) and re.match(r"^[A-Za-z0-9._-]{1,80}$", rid):
self.report["run_id"] = rid
if isinstance(trig, str) and re.match(r"^[a-z0-9_-]{1,20}$", trig):
self.report["trigger"] = trig
if ring in (0, 1) and not isinstance(ring, bool):
self.report["ring"] = ring
if self.mode == "facts":
return self.facts()
if self.mode == "bundle":
return Bundle(self).from_plan(plan)
if self.mode == "agent_update":
return self.agent_update(plan)
if self.layer == "host":
self.check_appliance()
self.check_guest(self.vmid)
@@ -699,20 +956,34 @@ class Apply:
self.report["health_after"] = self.health()
return 0
def repair(self):
rc, before, _ = self.x(["dpkg", "--audit"])
def dpkg_state(self):
"""`dpkg --audit` AND dpkg's update journal, in ONE call (R-876, agent v0.145.0). A crash in the middle of an
install can leave `/var/lib/dpkg/updates/` non-empty while `--audit` reads clean — measured on demo-hp
2026-10-05 — and that journal is exactly what apt refuses on ("dpkg was interrupted"). One `sh -c` with a
constant script keeps R-845's speed: a clean pass still costs one call here, as before."""
rc, out, _ = self.x(["sh", "-c", DPKG_STATE_SCRIPT])
audit, _, journal = out.partition(JOURNAL_MARK + "\n")
return audit, [l for l in journal.split() if l]
def repair(self, force=False):
before, journal = self.dpkg_state()
configured = len([l for l in before.splitlines() if l.startswith(" ")])
fixed = 0
after = ""
if before.strip(): # nothing half-done → nothing to run (R-845: two calls saved on every clean pass)
after, journal_after = "", []
# nothing half-done and no update journal → nothing to run (R-845: two calls saved on every clean pass).
# R-876: the JOURNAL counts too, and `force` (apt said "dpkg was interrupted") always repairs.
if before.strip() or journal or force:
self.x(APT_ENV + ["dpkg", "--configure", "-a", "--force-confold"])
rc2, out, err = self.x(APT_ENV + ["apt-get", "-f", "install", "-y", "-q"] + DPKG_OPTS)
_, after, _ = self.x(["dpkg", "--audit"])
after, journal_after = self.dpkg_state()
fixed = len(re.findall(r"^Setting up ", out, re.M))
self.report["repair"] = {"half_configured_before": configured, "fixed": fixed, "clean_after": after.strip() == ""}
self.r.log(f"os-apply: REPAIR configured={configured} fixed={fixed}")
self.report["repair"] = {"half_configured_before": configured, "journal_before": len(journal), "fixed": fixed,
"clean_after": after.strip() == "" and not journal_after}
self.r.log(f"os-apply: REPAIR configured={configured} journal={len(journal)} fixed={fixed}" + (" forced" if force else ""))
if after.strip():
raise Refused("R13", "dpkg is still broken after the repair: " + after.strip().splitlines()[0])
if journal_after:
raise Refused("R13", f"dpkg's update journal is still not empty after the repair ({len(journal_after)} file(s))")
def pending_fast(self):
"""Ring 0 (select pending-fast): every pending upgrade of an INSTALLED package whose every origin is Debian /
@@ -728,6 +999,23 @@ class Apply:
out.append({"name": p["name"], "version": p["to"], "origin": "Debian-Security" if "Debian-Security" in o else "Debian"})
return out
def restart_socket_users(self):
"""R-858: restart ONLY the containers that bind-mount the Docker socket, so they attach to the new one."""
rc, out, _ = self.g(["docker", "ps", "-q", "--no-trunc"], timeout=60)
users = []
for cid in out.split():
irc, iout, _ = self.g(["docker", "inspect", "-f", "{{.Name}}|{{range .Mounts}}{{.Destination}};{{end}}", cid], timeout=60)
if irc != 0 or "|" not in iout:
continue
name, mounts = iout.strip().split("|", 1)
if any(m in DOCKER_SOCKETS for m in mounts.split(";")):
users.append(name.lstrip("/"))
users.sort()
if users:
rrc, _, rerr = self.g(["docker", "restart"] + users, timeout=300)
self.r.log(f"os-apply: SOCKET-USERS restarted={','.join(users)} rc={rrc} (R-858: they held the old docker socket)")
return users
def pending_docker(self):
"""Ring 0 (select pending-docker): the newest pending version of each INSTALLED Docker package, Docker origin."""
rc, pend, remv, _ = self.simulate(["dist-upgrade"])
@@ -812,6 +1100,12 @@ class Apply:
raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)")
t0 = time.time()
rc, out, err = self.x(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + args)
if rc != 0 and "dpkg was interrupted" in (out + err):
# R-876 (belt): apt says dpkg was interrupted although the repair found nothing — repair and
# retry ONCE. Never a loop.
self.r.log("os-apply: INTERRUPTED apt says dpkg was interrupted — repairing and retrying once")
self.repair(force=True)
rc, out, err = self.x(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + args)
secs = time.time() - t0
# dpkg says "Installing new version of config file X" when X was NOT changed locally (the package's new
# version is taken), and "Configuration file 'X'" + "Keeping old config file" when it was (--force-confold
@@ -837,6 +1131,8 @@ class Apply:
return 3, None
self.report["upgraded"] = [{"name": n, "version": v} for n, v in upgrade]
self.report["seconds"] = round(secs, 1)
if self.layer == "docker":
self.report["socket_restarted"] = self.restart_socket_users()
procs, reboot = self.restart_needed()
self.report["restart_needed"] = procs
self.report["docker_restart_needed"] = any(p in ("dockerd", "containerd") for p in procs)
@@ -848,7 +1144,11 @@ class Apply:
self.remove_snapshot_sources()
def download_bytes(self, args):
rc, out, _ = self.x(APT_ENV + ["apt-get", "-s", "-o", "Debug::NoLocking=1", "--print-uris", "-q"] + args)
# R-865 (v0.144.0): NO `-s`. With `-s` apt prints the simulation ("Inst …") and no URI list, so this summed
# 0 B and R8 only ever applied its 500 MB floor (measured 2026-10-04: 0 URIs with -s, 3 URIs without).
# `--print-uris` alone downloads nothing — measured on 9202 2026-10-05: the archive cache and the versions
# unchanged. Pinned by test_R8_counts_the_real_download / test_download_bytes_never_simulates.
rc, out, _ = self.x(APT_ENV + ["apt-get", "-o", "Debug::NoLocking=1", "--print-uris", "-q"] + args)
total = 0
for l in out.splitlines():
m = re.match(r"^'[^']+' \S+ ([0-9]+) ", l)
@@ -875,8 +1175,346 @@ class Apply:
self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
def main(argv, runner=None):
class Bundle:
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
signed unit. Every check runs before the first write; a failed write or a failed self-check puts every previous
copy back. Refusal codes: R1 (the bundle file), R3 (authority), R16 (a path outside BUNDLE_FILES), R17 (a trust
file), R18 (a content check or a wrong sha)."""
def __init__(self, apply):
self.a, self.r = apply, apply.r
self.report = apply.report
# ---------- reading and checking ----------
def load_file(self, path):
d, base = os.path.dirname(path or ""), os.path.basename(path or "")
if d != PLAN_DIR or not BUNDLE_RE.match(base) or ".." in path:
raise Refused("R1", f"the bundle must be {PLAN_DIR}/bundle-<version>.json, got {path!r}")
try:
st = self.r.stat(path)
except OSError as e:
raise Refused("R1", f"cannot stat the bundle: {e}")
if not stat.S_ISREG(st.st_mode):
raise Refused("R1", "the bundle is not a regular file")
if st.st_uid != self.r.agent_uid():
raise Refused("R1", f"the bundle is not owned by {AGENT_USER}")
if st.st_size > BUNDLE_MAX_BYTES:
raise Refused("R1", "the bundle is larger than 4 MB")
return self.r.read_bytes(path)
def parse(self, data, want_sha, want_version=None):
"""The bundle bytes → [(dest, content, mode, check, policy)], every content check passed. Nothing is written."""
import base64
got = sha256_hex(data)
if got != want_sha:
raise Refused("R18", f"the bundle's sha256 is {got}, not the pinned {want_sha}")
try:
b = json.loads(data)
except ValueError as e:
raise Refused("R18", f"the bundle is not JSON: {e}")
if not isinstance(b, dict) or b.get("format") != BUNDLE_FORMAT:
raise Refused("R18", f"the bundle format is not {BUNDLE_FORMAT}")
ver = b.get("agent_version")
if not isinstance(ver, str) or not re.match(r"^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$", ver):
raise Refused("R18", f"the bundle names no agent version ({ver!r})")
if want_version is not None and ver != want_version:
raise Refused("R18", f"the bundle is for agent {ver}, the signed job pins {want_version}")
files = b.get("files")
if not isinstance(files, list) or not files:
raise Refused("R18", "the bundle lists no files")
out, seen = [], set()
for e in files:
if not isinstance(e, dict) or not isinstance(e.get("path"), str):
raise Refused("R18", "a bundle entry has no path")
dest = e["path"]
if dest in TRUST_PATHS:
raise Refused("R17", f"{dest} is the trust root — no bundle may add, remove or change a signer")
if dest not in BUNDLE_DESTS:
raise Refused("R16", f"{dest} is not a path a bundle may write")
if dest in seen:
raise Refused("R18", f"{dest} is named twice")
seen.add(dest)
try:
content = base64.b64decode(e.get("content_b64", ""), validate=True)
except (ValueError, TypeError):
raise Refused("R18", f"{dest}: the content is not base64")
if sha256_hex(content) != e.get("sha256"):
raise Refused("R18", f"{dest}: the content does not match its sha256")
_, _, mode, check, policy = BUNDLE_DESTS[dest]
self.check_content(dest, check, content)
out.append((dest, content, mode, check, policy))
order = [x[0] for x in BUNDLE_FILES]
out.sort(key=lambda x: order.index(x[0])) # the table's order: the sudoers files last
return ver, out
def check_content(self, dest, check, content):
try:
text = content.decode("utf-8")
except UnicodeDecodeError:
raise Refused("R18", f"{dest} is not UTF-8 text")
if check in ("sudoers", "sh", "bash", "nft"):
rc, msg = self.r.check_content(check, content)
if rc != 0:
raise Refused("R18", f"{dest} fails its {check} check: {msg}")
elif check == "python":
try:
compile(text, dest, "exec")
except SyntaxError as e:
raise Refused("R18", f"{dest} is not valid Python: {e}")
elif check in ("unit", "unit-nort", "agent-unit", "dropin"):
if not re.search(r"^\[(Unit|Service|Timer)\]\s*$", text, re.M):
raise Refused("R18", f"{dest} has no [Unit]/[Service]/[Timer] section")
if check == "unit-nort" and re.search(r"^\s*RuntimeDirectory\s*=", text, re.M | re.I):
raise Refused("R18", f"{dest} declares RuntimeDirectory= — the G1 incident; refused")
if check == "agent-unit" and not re.search(r"^User=" + AGENT_USER + r"\s*$", text, re.M):
raise Refused("R18", f"{dest} does not run the agent as {AGENT_USER}")
if check == "dropin" and not re.search(r"^\[Unit\]\s*$", text, re.M):
raise Refused("R18", f"{dest} must keep its start limits in [Unit] (SF-3)")
# The route must survive the bundle: a sudoers without this wrapper's line, or a wrapper without the bundle
# mode, would cut the box off from every later bundle.
if dest == "/etc/sudoers.d/felhom-agent" and SELF_CHECK_SUDO_LINE not in text:
raise Refused("R18", "the new sudoers no longer lets the agent call felhom-os-apply — the bundle route would end")
if dest == "/usr/local/sbin/felhom-os-apply" and f'BUNDLE_OP = "{BUNDLE_OP}"' not in text:
raise Refused("R18", "the new felhom-os-apply has no bundle mode — the bundle route would end")
def live(self, dest):
"""(bytes or None, mode or None, uid or None) of what the box has now."""
if not self.r.lexists(dest):
return None, None, None
st = self.r.stat(dest)
if not stat.S_ISREG(st.st_mode):
return b"", None, None # a symlink or a directory: always replaced by the real file
return self.r.read_bytes(dest), stat.S_IMODE(st.st_mode), st.st_uid
def plan_writes(self, files):
"""Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt)."""
oob = self.r.isdir(OOB_DIR)
plan = []
for dest, content, mode, check, policy in files:
if policy == "oob" and not oob:
plan.append((dest, content, mode, "skipped", None, None))
continue
old, old_mode, old_uid = self.live(dest)
if policy == "if-absent" and old is not None:
plan.append((dest, content, mode, "kept", old, old_mode))
elif old == content and old_mode == mode and old_uid == 0:
plan.append((dest, content, mode, "same", old, old_mode))
else:
plan.append((dest, content, mode, "write", old, old_mode))
return plan
# ---------- the two entries ----------
def from_plan(self, plan):
"""A signed agent_config_update, from the agent (sudo, the --plan line)."""
data = self.load_file(plan.get("bundle"))
trust = self.a.load_trust()
signers, bootstrap = TRUST_SIGNERS, False
if not self.r.lexists(TRUST_SIGNERS):
signers, bootstrap = PINNED_SIGNERS, True
self.r.log(f"os-apply: BUNDLE {TRUST_SIGNERS} is missing — verifying against the installer's pinned key "
f"{PINNED_OPERATOR_KEY_ID} only")
params, nonce, exp = self.a.verify_signed(plan.get("signed"), trust, op_name=BUNDLE_OP,
signers=None if not bootstrap else signers, burn=False)
sha, ver = params.get("bundle_sha256"), params.get("agent_version")
if not isinstance(sha, str) or not re.match(r"^[0-9a-f]{64}$", sha) or not isinstance(ver, str):
raise Refused("R3", "the signed job does not pin agent_version and bundle_sha256")
ver, files = self.parse(data, sha, ver)
self.a.burn_nonce(nonce, exp)
return self.install(ver, sha, files, "signed", bootstrap)
def from_installer(self, path, sha):
"""The installer (root, never through sudo): the hub manifest pinned the sha; no signature."""
try:
data = self.r.read_bytes(path)
except OSError as e:
raise Refused("R1", f"cannot read the bundle: {e}")
ver, files = self.parse(data, sha)
return self.install(ver, sha, files, "installer", False)
# ---------- install, self-check, undo ----------
def install(self, ver, sha, files, authority, bootstrap):
log = self.r.log
plan = self.plan_writes(files)
counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped")}
log(f"os-apply: BUNDLE START agent={ver} sha={sha[:16]} authority={authority} files={len(plan)} "
f"write={counts['write']} same={counts['same']} kept={counts['kept']} skipped={counts['skipped']}")
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
prev = os.path.join(BUNDLE_PREV_DIR, f"{stamp}-before-{ver}")
done = [] # (dest, old bytes or None, old mode)
rep = {"agent_version": ver, "sha256": sha, "authority": authority, "prev_dir": prev,
"written": [p[0] for p in plan if p[3] == "write"], "kept": [p[0] for p in plan if p[3] == "kept"],
"skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"]}
self.report["bundle"] = rep
try:
for dest, content, mode, action, old, old_mode in plan:
if action != "write":
continue
if old is not None:
self.r.put_file(prev + dest, old, 0o600)
done.append((dest, old, old_mode))
self.r.put_file(dest, content, mode)
log(f"os-apply: BUNDLE WROTE {dest} ({'replaced' if old is not None else 'new'})")
self.after_install(plan)
rep["self_check"] = self.self_check(plan)
except (Refused, OSError, subprocess.SubprocessError) as e:
reason = e.reason if isinstance(e, Refused) else str(e)
log(f"os-apply: BUNDLE FAILED — {reason}; putting {len(done)} previous file(s) back")
rep["rolled_back"] = self.undo(done)
rep["failed"] = reason[:300]
self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
return 3
rep["signers_created"] = False
if bootstrap and not self.r.lexists(TRUST_SIGNERS):
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
rep["signers_created"] = True
log(f"os-apply: BUNDLE created {TRUST_SIGNERS} with exactly the pinned key {PINNED_OPERATOR_KEY_ID}")
record = {"format": BUNDLE_FORMAT, "agent_version": ver, "bundle_sha256": sha, "authority": authority,
"installed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(self.r.now())),
"files": {p[0]: (sha256_hex(p[1]) if p[3] in ("write", "same") else
(sha256_hex(p[4]) if p[3] == "kept" else "skipped")) for p in plan}}
self.r.put_file(BUNDLE_RECORD, (json.dumps(record, indent=2, sort_keys=True) + "\n").encode(), 0o644)
self.prune()
log(f"os-apply: BUNDLE DONE agent={ver} written={counts['write']} same={counts['same']} "
f"self-check=ok signers-created={rep['signers_created']}")
return 0
def after_install(self, plan):
written = {p[0] for p in plan if p[3] == "write"}
dests = {p[0] for p in plan if p[3] != "skipped"}
if any(d.startswith("/etc/systemd/system/") for d in written):
self.must(["systemctl", "daemon-reload"], "systemctl daemon-reload")
if "/etc/tmpfiles.d/felhom-privsep.conf" in written:
self.must(["systemd-tmpfiles", "--create", "/etc/tmpfiles.d/felhom-privsep.conf"], "systemd-tmpfiles")
# `enable --now` starts a unit that is not running and leaves a running one alone (no restart). For the crash
# guard that runs its boot step once on a box that never had it: kernel.panic for THIS boot, as the boot unit
# would set it (`11` §5.9); with no earlier state it is the "first" boot, never counted as an unclean one.
if "/etc/systemd/system/felhom-crash-guard.service" in dests:
self.must(["systemctl", "enable", "--now", "felhom-crash-guard.service", "felhom-crash-guard-check.timer"],
"enable the crash guard")
if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
def must(self, argv, what):
rc, out, err = self.r.host(argv, 120)
if rc != 0:
raise Refused("R18", f"{what} failed (rc={rc}): {(out + err).strip()[-200:]}")
def self_check(self, plan):
"""After the install, before the record: the box still has a working route and working wrappers."""
sc = {}
self.must(["visudo", "-c"], "visudo -c (the whole sudoers)")
sc["visudo"] = "ok"
rc, out, err = self.r.host(["sudo", "-n", "-l", "-U", AGENT_USER], 60)
if rc != 0 or "/usr/local/sbin/felhom-os-apply --plan" not in out:
raise Refused("R18", f"sudo -l for {AGENT_USER} no longer lists felhom-os-apply (rc={rc})")
sc["sudo_l"] = "felhom-os-apply listed"
rc, out, err = self.r.host(["/usr/bin/python3", "/usr/local/sbin/felhom-os-apply", "--self-check"], 60)
if rc != 0 or f"bundle-format={BUNDLE_FORMAT}" not in out:
raise Refused("R18", f"the installed felhom-os-apply does not answer its self-check (rc={rc})")
sc["os_apply"] = out.strip()[:120]
rc, out, err = self.r.host(["/bin/sh", "/usr/local/sbin/felhom-selfupdate-guarded"], 60)
if rc != 2 or "usage" not in (out + err):
raise Refused("R18", f"the installed felhom-selfupdate-guarded does not answer with its usage (rc={rc})")
sc["selfupdate"] = "usage ok"
if "/usr/local/sbin/felhom-priv-apply" in {p[0] for p in plan if p[3] != "skipped"}:
rc, out, err = self.r.host(["/usr/bin/python3", "/usr/local/sbin/felhom-priv-apply", "--self-check"], 60)
if rc != 0 or "felhom-priv-apply ok" not in out:
raise Refused("R18", f"the installed felhom-priv-apply does not answer its self-check (rc={rc})")
sc["priv_apply"] = out.strip()[:80]
dests = {p[0] for p in plan if p[3] != "skipped"}
if "/usr/local/sbin/felhom-crash-guard" in dests:
rc, out, err = self.r.host(["/usr/local/sbin/felhom-crash-guard", "status"], 60)
try:
st = json.loads(out)
except ValueError:
st = None
if rc != 0 or not isinstance(st, dict):
raise Refused("R18", f"the crash guard does not answer its status (rc={rc})")
try:
panic = int(self.r.read_file("/proc/sys/kernel/panic").strip())
except (OSError, ValueError):
panic = None
if panic != st.get("kernel_panic"):
raise Refused("R18", f"kernel.panic is {panic}, the crash guard set {st.get('kernel_panic')}")
sc["crash_guard"] = {"armed": st.get("armed"), "kernel_panic": panic}
return sc
def undo(self, done):
"""Put every previous copy back (newest write first); remove files the bundle created."""
back = []
for dest, old, old_mode in reversed(done):
try:
if old is None:
self.r.remove(dest)
else:
self.r.put_file(dest, old, old_mode if old_mode is not None else 0o644)
back.append(dest)
except OSError as e:
self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
if any(d.startswith("/etc/systemd/system/") for d in back):
self.r.host(["systemctl", "daemon-reload"], 120)
rc, _, _ = self.r.host(["visudo", "-c"], 60)
self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
return back
def prune(self):
names = [n for n in self.r.list_dir(BUNDLE_PREV_DIR)]
for n in names[:-BUNDLE_KEEP]:
self.r.rmtree(os.path.join(BUNDLE_PREV_DIR, n))
# ---------- the facts mode's view ----------
def state(self):
"""What the box runs: the record, and every bundle path's live sha256 against it (drift = changed by hand)."""
rec = None
try:
rec = json.loads(self.r.read_file(BUNDLE_RECORD))
except (OSError, ValueError):
rec = None
live = {}
for dest, *_ in BUNDLE_FILES:
try:
data, _, _ = self.live(dest)
except OSError:
data = None
live[dest] = sha256_hex(data) if data is not None else "absent"
out = {"version": (rec or {}).get("agent_version", "none"), "live": live}
if rec:
out["installed_at"] = rec.get("installed_at")
out["bundle_sha256"] = rec.get("bundle_sha256")
want = rec.get("files") or {}
out["drift"] = sorted(d for d, h in want.items() if h != "skipped" and live.get(d) != h)
out["signers_present"] = self.r.lexists(TRUST_SIGNERS)
return out
def main(argv, runner=None, environ=None):
r = runner or Runner()
env = os.environ if environ is None else environ
if argv[1:] == ["--self-check"]:
# The bundle's self-check runs the NEW wrapper this way: it parses and starts. Reads nothing, changes nothing.
print(f"felhom-os-apply ok bundle-format={BUNDLE_FORMAT} files={len(BUNDLE_FILES)}")
return 0
if len(argv) == 5 and argv[1] == "--install-bundle" and argv[3] == "--sha256":
# The INSTALLER's entry (root, a fresh box). Unreachable through sudo: the sudoers line pins argv[1] to --plan,
# and a sudo caller is refused here as well (sudo always sets SUDO_UID).
a = Apply(r, "")
if "SUDO_UID" in env:
r.log("os-apply: REFUSED: R1 --install-bundle is the installer's entry, never through sudo")
print("OSAPPLY-REPORT " + json.dumps({"refused": {"code": "R1", "reason": "install-bundle via sudo"}}))
return 2
sha = argv[4]
if not re.match(r"^[0-9a-f]{64}$", sha):
print("OSAPPLY-REPORT " + json.dumps({"refused": {"code": "R18", "reason": "sha256 is not 64 lowercase hex"}}))
return 2
a.report["mode"] = "bundle"
try:
rc = Bundle(a).from_installer(argv[2], sha)
except Refused as e:
r.log(f"os-apply: REFUSED: {e.code} {e.reason}")
a.report["refused"] = {"code": e.code, "reason": e.reason}
rc = 2
print("OSAPPLY-REPORT " + json.dumps(a.report, sort_keys=True))
return rc
if len(argv) != 3 or argv[1] != "--plan":
r.log("os-apply: REFUSED: R1 usage: felhom-os-apply --plan /var/lib/felhom-agent/os/plan-<id>.json")
print("OSAPPLY-REPORT " + json.dumps({"refused": {"code": "R1", "reason": "usage"}}))
@@ -894,7 +1532,18 @@ def main(argv, runner=None):
a.report["failed"] = {"rc": 124, "timeout": str(e.cmd)[:200]}
rc = 3
a.report["pass_seconds"] = round(time.time() - t0, 1)
print("OSAPPLY-REPORT " + json.dumps(a.report, sort_keys=True))
if a.report.get("mode") == "apply":
# R-868: BEFORE the stdout line — a killed agent never reads stdout, and this copy is how its report still
# reaches the hub (the agent sends an unsent copy when it starts, and deletes it once sent).
try:
r.save_report(argv[2], a.report)
except Exception as e:
r.log(f"os-apply: the report copy could not be saved (the run is unaffected): {e}")
try:
print("OSAPPLY-REPORT " + json.dumps(a.report, sort_keys=True), flush=True)
except OSError:
# R-868: nobody reads stdout any more (the agent was killed); the copy above carries the report.
sys.stdout = open(os.devnull, "w")
return rc
+414
View File
@@ -0,0 +1,414 @@
#!/usr/bin/python3
"""felhom-priv-apply — the ROOT half of every file the agent writes into a root-read place (R-861, `03` §3.1).
Install as /usr/local/sbin/felhom-priv-apply (0755 root:root) — it rides the signed config bundle (R-840).
WHY IT EXISTS. Until agent v0.146.0 the agent's sudoers let it `install` a file it had written itself into a place a
root program reads: a systemd .mount unit (a bind mount of an agent-owned directory over /etc/sudoers.d is a root
shell), a dnsmasq drop-in (`dhcp-script=` runs as root), the WireGuard config (`PostUp=` runs as root) and the OOB
sshd config (`AuthorizedKeysFile` + `StrictModes no`). A compromised agent PROCESS was therefore root on its host.
Now the agent stages the file and this wrapper — root-owned, delivered only by an operator-signed bundle — checks
the CONTENT against the exact grammar the agent's own renderers produce, and refuses anything else. The agent can no
longer name the destination: each verb has a fixed source and a fixed (or strictly named) destination.
Verbs (each one sudoers line, exact-match pattern):
unit <name> /var/lib/felhom-agent/units/<name> -> /etc/systemd/system/<name> (.mount | .automount)
dnsmasq <tmp> <name> /tmp/felhom-resolver-<digits>.conf -> /etc/dnsmasq.d/felhom-<...>.conf
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
sshd-key /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op -> /etc/felhom-sshd/authorized_keys/felhom-op
--self-check prints "felhom-priv-apply ok verbs=..." (the bundle's self-check)
Exit codes: 0 installed (or already identical), 2 usage, 3 refused (content or source), 4 install failed.
Every refusal is logged to the journal (tag felhom-priv-apply) with its rule; file CONTENT is never logged.
Pinned by configs/test_felhom_priv_apply.py (one test per rule, red-proofs in the R-861 audit).
"""
import ipaddress
import os
import re
import stat
import subprocess
import sys
AGENT_USER = "felhom-agent"
STATE = "/var/lib/felhom-agent"
UNITS_SRC = STATE + "/units"
UNIT_DIR = "/etc/systemd/system"
DNSMASQ_DIR = "/etc/dnsmasq.d"
WG_SRC, WG_DEST = STATE + "/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
MAX_BYTES = 64 * 1024
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key")
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
DNSMASQ_NAME_RE = re.compile(r"^felhom-[a-z0-9][a-z0-9._-]*\.conf$")
SEG = r"[A-Za-z0-9_-][A-Za-z0-9_.-]*"
WHERE_RE = re.compile(r"^/mnt/(felhom-drives/)?" + SEG + r"$")
UUID_RE = re.compile(r"^[A-Fa-f0-9]{4,}(-[A-Fa-f0-9]+){0,4}$")
HOST_RE = re.compile(r"^[A-Za-z0-9._:-]{1,255}$")
NET_PATH_RE = re.compile(r"^[A-Za-z0-9._/@+-]{1,512}$")
OPT_RE = re.compile(r"^[A-Za-z0-9_.:/@+-]+(=[A-Za-z0-9_.:/@+-]+)?$")
LOCAL_TYPES = {"ext4", "xfs", "btrfs", "exfat", "vfat", "ntfs3", "ntfs"}
NET_TYPES = {"nfs", "nfs4", "cifs"}
# Options that turn a device mount into something else, or let set-uid/device files act on the host.
FORBIDDEN_OPTS = {"bind", "rbind", "move", "rmove", "remount", "suid", "dev", "user", "users", "owner", "group",
"x-mount.mkdir", "helper"}
UNIT_TOKEN_RE = re.compile(r"^[A-Za-z0-9@_.\\:-]+$")
DESC_RE = re.compile(r"^[^\x00-\x1f\x7f]{0,200}$")
WG_KEY_RE = re.compile(r"^[A-Za-z0-9+/]{42}[AEIMQUYcgkosw480]=$")
KEY_LINE_RE = re.compile(r"^(ssh-ed25519|ssh-rsa|ecdsa-sha2-nistp(256|384|521)|sk-ssh-ed25519@openssh\.com) "
r"[A-Za-z0-9+/]+={0,3}( [ -~]{0,200})?$")
class Refused(Exception):
def __init__(self, rule, reason):
super().__init__(reason)
self.rule, self.reason = rule, reason
class Host:
"""Every filesystem / process effect, so the tests can play the box in memory."""
def agent_uid(self):
import pwd
return pwd.getpwnam(AGENT_USER).pw_uid
def read_source(self, path):
"""The staged file: a REGULAR file owned by the agent, never a symlink, at most MAX_BYTES."""
try:
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC)
except OSError as e:
raise Refused("P1", f"cannot open the staged file {path}: {e.strerror}")
try:
st = os.fstat(fd)
if not stat.S_ISREG(st.st_mode):
raise Refused("P1", f"{path} is not a regular file")
if st.st_uid != self.agent_uid():
raise Refused("P1", f"{path} is not owned by {AGENT_USER}")
if st.st_size > MAX_BYTES:
raise Refused("P1", f"{path} is larger than {MAX_BYTES} bytes")
with os.fdopen(fd, "rb") as f:
fd = -1
return f.read(MAX_BYTES + 1)
finally:
if fd >= 0:
os.close(fd)
def read_dest(self, path):
try:
with open(path, "rb") as f:
return f.read()
except OSError:
return None
def install(self, dest, data, mode):
"""Atomic, root-owned: a temp file beside the destination, fsync, rename."""
d = os.path.dirname(dest)
os.makedirs(d, mode=0o755, exist_ok=True)
tmp = os.path.join(d, f".{os.path.basename(dest)}.felhom-new.{os.getpid()}")
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
try:
with os.fdopen(fd, "wb") as f:
f.write(data)
f.flush()
os.fchown(f.fileno(), 0, 0)
os.fchmod(f.fileno(), mode)
os.fsync(f.fileno())
os.replace(tmp, dest)
except BaseException:
try:
os.remove(tmp)
except OSError:
pass
raise
def log(self, line):
print(line, file=sys.stderr)
try:
subprocess.run(["logger", "-t", "felhom-priv-apply", "--", line], timeout=10, check=False)
except (OSError, subprocess.SubprocessError):
pass
def systemd_escape_path(path):
"""`systemd-escape --path`: strip the slashes at both ends, `/` -> `-`, every byte outside [A-Za-z0-9:_.] (and a
leading `.`) -> `\\xNN`."""
p = path.strip("/")
out = []
for i, ch in enumerate(p):
if ch == "/":
out.append("-")
elif (ch.isascii() and (ch.isalnum() or ch in ":_.")) and not (i == 0 and ch == "."):
out.append(ch)
else:
out.extend("\\x%02x" % b for b in ch.encode())
return "".join(out)
def text_of(data, what):
if len(data) > MAX_BYTES:
raise Refused("P1", f"{what} is too large")
try:
text = data.decode("utf-8")
except UnicodeDecodeError:
raise Refused("P2", f"{what} is not UTF-8 text")
if "\x00" in text or "\r" in text:
raise Refused("P2", f"{what} carries a NUL or CR byte")
return text
def parse_ini(text, what):
"""[Section] / Key=Value / comments / blank lines. A key outside a section or a repeated key is refused."""
sections, cur = {}, None
for n, raw in enumerate(text.split("\n"), 1):
line = raw.strip()
if not line or line.startswith("#") or line.startswith(";"):
continue
m = re.match(r"^\[([A-Za-z]+)\]$", line)
if m:
cur = m.group(1)
if cur in sections:
raise Refused("U2", f"{what}: section [{cur}] twice")
sections[cur] = {}
continue
if cur is None or "=" not in line:
raise Refused("U2", f"{what}: line {n} is not Key=Value inside a section")
k, v = line.split("=", 1)
k, v = k.strip(), v.strip()
if k in sections[cur]:
raise Refused("U2", f"{what}: {cur}.{k} given twice")
sections[cur][k] = v
return sections
# ---------- the unit verb ----------
def check_unit(name, text):
if not UNIT_NAME_RE.match(name) or "/" in name:
raise Refused("U1", f"unit name {name!r} is not mnt-<escaped path>.mount|.automount")
kind = "automount" if name.endswith(".automount") else "mount"
s = parse_ini(text, name)
body = "Automount" if kind == "automount" else "Mount"
allowed = {"Unit": {"Description", "After", "Before", "Wants", "Requires"},
body: {"Where", "TimeoutIdleSec"} if kind == "automount" else {"What", "Where", "Type", "Options"},
"Install": {"WantedBy"}}
for sec, keys in s.items():
if sec not in allowed:
raise Refused("U2", f"{name}: section [{sec}] is not allowed")
bad = set(keys) - allowed[sec]
if bad:
raise Refused("U2", f"{name}: [{sec}] key(s) {sorted(bad)} not allowed")
u = s.get("Unit", {})
if not DESC_RE.match(u.get("Description", "")):
raise Refused("U2", f"{name}: Description has control characters")
for k in ("After", "Before", "Wants", "Requires"):
if k in u and not all(UNIT_TOKEN_RE.match(t) for t in u[k].split()):
raise Refused("U2", f"{name}: {k}= names something that is not a unit")
inst = s.get("Install", {})
if inst and inst.get("WantedBy") != "multi-user.target":
raise Refused("U2", f"{name}: WantedBy must be multi-user.target")
m = s.get(body)
if not m or "Where" not in m:
raise Refused("U3", f"{name}: no [{body}] Where=")
where = m["Where"]
if not WHERE_RE.match(where):
raise Refused("U3", f"{name}: Where={where} is not /mnt/<name> or /mnt/felhom-drives/<name>")
if systemd_escape_path(where) + "." + kind != name:
raise Refused("U3", f"{name}: the unit name does not match Where={where}")
if kind == "automount":
t = m.get("TimeoutIdleSec", "")
if t and not re.match(r"^[0-9]{1,6}$", t):
raise Refused("U2", f"{name}: TimeoutIdleSec must be seconds")
return
what, typ = m.get("What", ""), m.get("Type", "")
opts = [o for o in m.get("Options", "").split(",") if o]
net = False
mu = re.match(r"^/dev/disk/by-uuid/(.+)$", what)
if mu:
if not UUID_RE.match(mu.group(1)):
raise Refused("U4", f"{name}: What= is not a filesystem UUID")
if typ and typ not in LOCAL_TYPES:
raise Refused("U4", f"{name}: Type={typ} is not a local filesystem")
else:
net = True
if typ not in NET_TYPES:
raise Refused("U4", f"{name}: What= is neither /dev/disk/by-uuid/<uuid> nor a network source with Type=nfs/nfs4/cifs")
if typ == "cifs":
mm = re.match(r"^//([^/]+)/(.+)$", what)
else:
mm = re.match(r"^([^/:][^:]*):(/.*)$", what)
if not mm or not HOST_RE.match(mm.group(1)) or not NET_PATH_RE.match(mm.group(2)) or ".." in mm.group(2).split("/"):
raise Refused("U4", f"{name}: What= is not a clean {typ} source")
if not where.startswith("/mnt/felhom-drives/"):
raise Refused("U3", f"{name}: a network share mounts only under /mnt/felhom-drives/")
for o in opts:
if not OPT_RE.match(o):
raise Refused("U5", f"{name}: mount option {o!r} has characters a mount option never needs")
if o.split("=", 1)[0].lower() in FORBIDDEN_OPTS or o.lower().startswith("x-mount."):
raise Refused("U5", f"{name}: mount option {o.split('=', 1)[0]!r} is not allowed")
if net and not {"nosuid", "nodev"} <= set(opts):
# A network server is outside the box: a set-uid file on it must never run as root here.
raise Refused("U5", f"{name}: a network share must carry nosuid,nodev")
# ---------- dnsmasq ----------
def _ip(v, v6=True):
try:
a = ipaddress.ip_address(v)
except ValueError:
return False
return v6 or a.version == 4
DOMAIN_RE = re.compile(r"^[A-Za-z0-9]([A-Za-z0-9-]{0,62})(\.[A-Za-z0-9]([A-Za-z0-9-]{0,62}))*$")
def check_dnsmasq(text):
for n, raw in enumerate(text.split("\n"), 1):
line = raw.strip()
if not line or line.startswith("#"):
continue
if line in ("bind-interfaces", "no-resolv"):
continue
k, _, v = line.partition("=")
if k == "listen-address" and _ip(v, v6=False):
continue
if k == "server" and (_ip(v) or (v.count("#") == 1 and _ip(v.split("#")[0]) and v.split("#")[1].isdigit())):
continue
m = re.match(r"^/([^/]+)/$", v)
if k == "local" and m and DOMAIN_RE.match(m.group(1)):
continue
m = re.match(r"^/([^/]+)/([^/]+)$", v)
if k == "address" and m and DOMAIN_RE.match(m.group(1)) and _ip(m.group(2), v6=False):
continue
raise Refused("D1", f"dnsmasq line {n} ({k or line[:20]!r}) is not one the resolver writes")
# ---------- WireGuard ----------
def check_wg(text):
s = parse_ini(text, "wg-felhom.conf")
if set(s) != {"Interface", "Peer"}:
raise Refused("W1", "wg-felhom.conf must hold exactly [Interface] and [Peer]")
i, p = s["Interface"], s["Peer"]
if set(i) - {"PrivateKey", "Address", "MTU"} or set(p) - {"PublicKey", "Endpoint", "AllowedIPs", "PersistentKeepalive"}:
raise Refused("W1", "wg-felhom.conf carries a key the agent never writes (PostUp/PreUp/... run as root)")
if not WG_KEY_RE.match(i.get("PrivateKey", "")) or not WG_KEY_RE.match(p.get("PublicKey", "")):
raise Refused("W2", "a WireGuard key is not 32 bytes of base64")
try:
a = ipaddress.ip_network(i.get("Address", ""), strict=False)
if a.version != 4 or a.prefixlen != 32:
raise ValueError
if not (1280 <= int(i.get("MTU", "1280")) <= 1500):
raise ValueError
host, _, port = p.get("Endpoint", "").rpartition(":")
if ipaddress.ip_address(host).version != 4 or not (1 <= int(port) <= 65535):
raise ValueError
for n in p.get("AllowedIPs", "").split(","):
if ipaddress.ip_network(n.strip(), strict=True).prefixlen != 32:
raise ValueError
if not (0 <= int(p.get("PersistentKeepalive", "25")) <= 3600):
raise ValueError
except ValueError:
raise Refused("W2", "an Address/MTU/Endpoint/AllowedIPs/PersistentKeepalive value is not what the agent renders")
# ---------- OOB sshd ----------
def render_sshd(port):
"""Byte-identical to felhomsshd.renderConfig (internal/felhomsshd/config.go) — pinned by a Go test."""
return ("# felhom OOB sshd — agent-managed (H1); DO NOT EDIT\n"
f"Port {port}\n"
"ListenAddress 0.0.0.0\n"
"ListenAddress ::\n"
"HostKey /etc/felhom-sshd/ssh_host_ed25519_key\n"
"PidFile /run/felhom-sshd.pid\n"
"AuthorizedKeysFile /etc/felhom-sshd/authorized_keys/%u\n"
"PasswordAuthentication no\n"
"PermitRootLogin prohibit-password\n"
"PubkeyAuthentication yes\n"
"KbdInteractiveAuthentication no\n"
"UsePAM yes\n"
"AllowUsers root felhom-op\n"
"X11Forwarding no\n"
"Subsystem sftp internal-sftp\n")
def check_sshd(text):
m = re.search(r"^Port ([0-9]{1,5})$", text, re.M)
if not m or not (1 <= int(m.group(1)) <= 65535) or int(m.group(1)) == 22:
raise Refused("S1", "sshd_config has no Port (or claims :22, the household's sshd)")
if text != render_sshd(int(m.group(1))):
raise Refused("S1", "sshd_config differs from the one fixed template (only the Port may vary)")
def check_key(text):
lines = [l for l in text.split("\n") if l.strip()]
if len(lines) > 1:
raise Refused("S2", "felhom-op's authorized_keys holds more than one key")
if lines and not KEY_LINE_RE.match(lines[0]):
raise Refused("S2", "the key line is not a plain public key (no options such as command= or from=)")
# ---------- main ----------
def plan(argv):
"""(verb, source, dest, mode, checker) for an argv, or Refused("A1")."""
if not argv or argv[0] not in VERBS:
raise Refused("A1", "usage: felhom-priv-apply unit <name> | dnsmasq <tmp> <name> | wg | sshd-config | sshd-key")
v, rest = argv[0], argv[1:]
if v == "unit" and len(rest) == 1:
if not UNIT_NAME_RE.match(rest[0]):
raise Refused("U1", f"unit name {rest[0]!r} is not mnt-<escaped path>.mount|.automount")
return v, os.path.join(UNITS_SRC, rest[0]), os.path.join(UNIT_DIR, rest[0]), 0o644, lambda t: check_unit(rest[0], t)
if v == "dnsmasq" and len(rest) == 2:
if not DNSMASQ_TMP_RE.match(rest[0]) or not DNSMASQ_NAME_RE.match(rest[1]):
raise Refused("D2", "dnsmasq wants /tmp/felhom-resolver-<digits>.conf and felhom-<name>.conf")
return v, rest[0], os.path.join(DNSMASQ_DIR, rest[1]), 0o644, check_dnsmasq
if v == "wg" and not rest:
return v, WG_SRC, WG_DEST, 0o600, check_wg
if v == "sshd-config" and not rest:
return v, SSHD_SRC, SSHD_DEST, 0o644, check_sshd
if v == "sshd-key" and not rest:
return v, KEY_SRC, KEY_DEST, 0o644, check_key
raise Refused("A1", f"wrong arguments for {v}")
def main(argv, host=None):
host = host or Host()
if argv == ["--self-check"]:
print("felhom-priv-apply ok verbs=" + ",".join(VERBS))
return 0
if len(argv) >= 3 and argv[0] == "--check":
# CHECK ONLY (tests and the Go contract tests): `--check <verb> [<name>] <file>` validates <file> as that
# verb would and installs nothing. Not in sudoers. Prints OK or the rule; never the content.
verb, file = argv[1], argv[-1]
try:
_, _, _, _, checker = plan([verb] + argv[2:-1] if verb != "dnsmasq" else
[verb, "/tmp/felhom-resolver-1.conf", argv[2]])
with open(file, "rb") as f:
checker(text_of(f.read(), file))
except Refused as e:
print(f"REFUSED [{e.rule}] {e.reason}")
return 3
print("OK")
return 0
try:
verb, src, dest, mode, checker = plan(argv)
data = host.read_source(src)
checker(text_of(data, src))
except Refused as e:
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] {' '.join(argv)[:160]}: {e.reason}")
return 2 if e.rule == "A1" else 3
if host.read_dest(dest) == data:
host.log(f"felhom-priv-apply: SAME {verb} {dest}")
return 0
try:
host.install(dest, data, mode)
except OSError as e:
host.log(f"felhom-priv-apply: FAILED {verb} {dest}: {e}")
return 4
host.log(f"felhom-priv-apply: INSTALLED {verb} {dest} ({len(data)} bytes)")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+13
View File
@@ -0,0 +1,13 @@
[Unit]
Description=Felhom stable drive parent (shared bind for live drive hot-swap)
After=local-fs.target
Before=pve-guests.service
ConditionPathExists=/usr/local/sbin/felhom-shared-parent.sh
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/felhom-shared-parent.sh
[Install]
WantedBy=pve-guests.service multi-user.target
+15
View File
@@ -0,0 +1,15 @@
#!/bin/sh
# felhom stable drive parent: a SHARED bind so the agent can swap backing drives underneath it and the
# guest sees the change live (no restart). MUST run before pve-guests so the guest's parent bind inherits
# the shared peer group (slave). Installed + enabled by felhom-agent. Idempotent.
set -e
mkdir -p /mnt/felhom-drives
# Isolate + share ONLY when first creating the self-bind (a fresh boot). The self-bind inherits the root
# mount's shared peer group, so make-private detaches it (else binds under it DOUBLE via the root peer),
# then make-shared gives it its own group whose only slave is the guest's parent bind. Re-running this on
# an existing parent would churn the peer-group id and orphan the guest's slave — so guard on mountpoint.
if ! mountpoint -q /mnt/felhom-drives; then
mount --bind /mnt/felhom-drives /mnt/felhom-drives
mount --make-private /mnt/felhom-drives
mount --make-shared /mnt/felhom-drives
fi
+634
View File
@@ -0,0 +1,634 @@
#!/usr/bin/env python3
"""Tests for the config bundle (R-840, `11` §5.4.2): felhom-os-apply's `bundle` mode and `--install-bundle`, and
scripts/build-config-bundle.py. An in-memory host plays the files; nothing real is written or run. Each refusal has a
test; each test names the rule it pins. Red-proof: `audits/r840-config-bundle-2026-10-04/partB/redproof.txt`.
Run: python3 configs/test_felhom_config_bundle.py (also run by internal/osupdate's Go test)
"""
import sys
sys.dont_write_bytecode = True # importing the builder must not leave scripts/__pycache__ behind
import base64
import contextlib
import hashlib
import importlib.machinery
import importlib.util
import io
import json
import os
import pathlib
import re
import stat as statmod
import unittest
HERE = pathlib.Path(__file__).resolve().parent
REPO = HERE.parent
_loader = importlib.machinery.SourceFileLoader("osapply", os.environ.get("OSAPPLY_UNDER_TEST", str(HERE / "felhom-os-apply")))
_spec = importlib.util.spec_from_loader("osapply", _loader)
osapply = importlib.util.module_from_spec(_spec)
_loader.exec_module(osapply)
_bl = importlib.machinery.SourceFileLoader("bundlebuild", str(REPO / "scripts" / "build-config-bundle.py"))
_bs = importlib.util.spec_from_loader("bundlebuild", _bl)
builder = importlib.util.module_from_spec(_bs)
_bl.exec_module(builder)
PLAN = "/var/lib/felhom-agent/os/plan-b1.json"
BUNDLE = "/var/lib/felhom-agent/os/bundle-0.143.0.json"
HOST = "demo-hp-bb76ea"
INSTALLER = REPO.parent / "felhom.eu" / "scripts" / "felhom-host-install.sh"
class St:
def __init__(self, mode, uid):
self.st_mode, self.st_uid, self.st_size = mode, uid, 100
class Box:
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
def __init__(self, bundle_bytes, signed, oob=False, signers=True):
self.files, self.modes, self.uids = {}, {}, {}
self.dirs = {osapply.OOB_DIR} if oob else set()
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
if signers:
self.put(osapply.TRUST_SIGNERS, b'felhom-op-1 namespaces="felhom-op-v1" ssh-ed25519 AAAA felhom-op-1\n', 0o644)
self.put("/proc/sys/kernel/panic", b"0\n", 0o644)
self.plan = {"release_id": "bundle-0.143.0", "layer": "host", "mode": "bundle", "bundle": BUNDLE, "signed": signed}
self.put(PLAN, json.dumps(self.plan).encode(), 0o600, uid=999)
self.put(BUNDLE, bundle_bytes, 0o600, uid=999)
self.sig_rc, self.nonces, self.clock = 0, {}, 1791115200.0 # 2026-10-04T12:00:00Z
self.calls, self.logs, self.writes = [], [], []
self.visudo_fail = False # the WHOLE sudoers (`visudo -c`) after install
self.sudo_l = " (root) NOPASSWD: /usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json\n"
self.guard = {"armed": True, "kernel_panic": 10}
def put(self, p, data, mode, uid=0):
self.files[p], self.modes[p], self.uids[p] = data, mode, uid
# Runner interface
def now(self):
return self.clock
def log(self, line):
self.logs.append(line)
def agent_uid(self):
return 999
def verify_sig(self, signers, key_id, ns, blob, sig):
self.verified = (signers, key_id, ns)
return self.sig_rc
def read_nonces(self):
return dict(self.nonces)
def write_nonces(self, d):
self.nonces = dict(d)
def read_file(self, p):
if p not in self.files:
raise OSError("no such file")
return self.files[p].decode()
def read_bytes(self, p):
if p not in self.files:
raise OSError("no such file")
return self.files[p]
def stat(self, p):
if p not in self.files:
raise OSError("no such file")
return St(statmod.S_IFREG | self.modes[p], self.uids[p])
def lexists(self, p):
return p in self.files
def isdir(self, p):
return p in self.dirs
def put_file(self, p, data, mode):
self.writes.append(p)
self.put(p, data, mode)
def remove(self, p):
self.writes.append("rm " + p)
del self.files[p]
def list_dir(self, p):
return sorted({k[len(p) + 1:].split("/")[0] for k in self.files if k.startswith(p + "/")})
def rmtree(self, p):
for k in [k for k in self.files if k.startswith(p + "/")]:
del self.files[k]
def check_content(self, kind, data):
return (1, f"{kind}: syntax error") if b"BROKEN-SYNTAX" in data else (0, "")
def host(self, argv, timeout=600, stdin=None):
self.calls.append(argv)
if argv[:2] == ["visudo", "-c"]:
return (1, "", "parse error") if self.visudo_fail else (0, "ok", "")
if argv[:2] == ["sudo", "-n"]:
return 0, self.sudo_l, ""
if argv[-2:] == ["/usr/local/sbin/felhom-priv-apply", "--self-check"]:
body = self.files.get("/usr/local/sbin/felhom-priv-apply", b"")
return (0, "felhom-priv-apply ok verbs=unit\n", "") if b"VERBS" in body else (1, "", "boom")
if argv[-1] == "--self-check":
body = self.files.get("/usr/local/sbin/felhom-os-apply", b"")
return (0, "felhom-os-apply ok bundle-format=1 files=22\n", "") if b"BUNDLE_OP" in body else (1, "", "boom")
if argv[-1] == "/usr/local/sbin/felhom-selfupdate-guarded":
return 2, "", "felhom-selfupdate-guarded: usage: ...\n"
if argv[-1] == "status" and argv[0].endswith("felhom-crash-guard"):
return 0, json.dumps(self.guard), ""
if argv[:3] == ["systemctl", "enable", "--now"] and "felhom-crash-guard.service" in argv:
self.put("/proc/sys/kernel/panic", f"{self.guard['kernel_panic']}\n".encode(), 0o644)
return 0, "", ""
def signed_job(sha, version="0.143.0", host=HOST, op="agent_config_update", nonce="b1",
issued="2026-10-04T11:50:00Z", expires="2026-10-04T12:30:00Z"):
blob = json.dumps({"expires_at": expires, "issued_at": issued, "key_id": "felhom-op-1", "nonce": nonce, "op": op,
"params": {"agent_version": version, "bundle_sha256": sha},
"target": {"guest_id": "", "host_id": host}}, sort_keys=True).encode()
return {"blob_b64": base64.b64encode(blob).decode(), "sig": "-----BEGIN SSH SIGNATURE-----\nx\n-----END SSH SIGNATURE-----\n"}
def real_bundle(version="0.143.0"):
data = builder.build(version)
return data, hashlib.sha256(data).hexdigest()
def edited_bundle(edit):
"""The real bundle, with edit(files_list) applied and every sha recomputed — a SIGNED bundle with bad content."""
b = json.loads(builder.build("0.143.0"))
edit(b["files"])
for e in b["files"]:
e["sha256"] = hashlib.sha256(base64.b64decode(e["content_b64"])).hexdigest()
data = (json.dumps(b, indent=1, sort_keys=True) + "\n").encode()
return data, hashlib.sha256(data).hexdigest()
def replace_content(files, path, fn):
for e in files:
if e["path"] == path:
e["content_b64"] = base64.b64encode(fn(base64.b64decode(e["content_b64"]))).decode()
def run(box, argv=None, environ=None):
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
rc = osapply.main(argv or ["felhom-os-apply", "--plan", PLAN], runner=box, environ=environ or {})
line = [l for l in buf.getvalue().splitlines() if l.startswith("OSAPPLY-REPORT ")][-1]
return rc, json.loads(line[len("OSAPPLY-REPORT "):])
def box_files(box):
return {p: box.files[p] for p in box.files if p in osapply.BUNDLE_DESTS}
class Install(unittest.TestCase):
def test_fresh_box_gets_every_file_and_a_record(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
b = rep["bundle"]
# every path but the four OOB ones (no belt on this box)
self.assertEqual(len(b["written"]), len(osapply.BUNDLE_FILES) - 4, b)
self.assertEqual(len(b["skipped"]), 4)
self.assertEqual(box.files["/etc/sudoers.d/felhom-agent"], (REPO / "configs" / "felhom-agent.sudoers").read_bytes())
self.assertEqual(box.modes["/etc/sudoers.d/felhom-agent"], 0o440)
rec = json.loads(box.files[osapply.BUNDLE_RECORD])
self.assertEqual((rec["agent_version"], rec["bundle_sha256"], rec["authority"]), ("0.143.0", sha, "signed"))
self.assertIn("b1", box.nonces, "the job is consumed")
self.assertEqual(b["self_check"]["crash_guard"], {"armed": True, "kernel_panic": 10})
self.assertIn(["systemctl", "daemon-reload"], box.calls)
def test_sudoers_is_written_after_every_wrapper(self):
"""Order: a referenced wrapper is in place before the sudoers line that allows it."""
data, sha = edited_bundle(lambda f: f.reverse()) # the bundle lists the sudoers FIRST; the wrapper must reorder
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
w = [p for p in box.writes if p in osapply.BUNDLE_DESTS]
self.assertEqual(w[-1], "/etc/sudoers.d/felhom-agent")
self.assertLess(w.index("/usr/local/sbin/felhom-os-apply"), w.index("/etc/sudoers.d/felhom-agent"))
def test_identical_box_writes_nothing(self):
"""The demo boxes' case: hand-copied files equal to the release → 0 written, all 'same'."""
data, sha = real_bundle()
box = Box(data, signed_job(sha))
for dest, src, mode, _, policy in osapply.BUNDLE_FILES:
if policy != "oob":
box.put(dest, (REPO / "configs" / src).read_bytes(), mode)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["bundle"]["written"], [])
self.assertEqual(rep["bundle"]["same"], len(osapply.BUNDLE_FILES) - 5) # 4 oob skipped + crash-guard.conf kept
self.assertEqual(rep["bundle"]["kept"], ["/etc/felhom/crash-guard.conf"])
def test_a_wrong_mode_is_rewritten(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put("/etc/sudoers.d/felhom-agent", (REPO / "configs" / "felhom-agent.sudoers").read_bytes(), 0o644)
rc, rep = run(box)
self.assertIn("/etc/sudoers.d/felhom-agent", rep["bundle"]["written"])
self.assertEqual(box.modes["/etc/sudoers.d/felhom-agent"], 0o440)
def test_tuned_crash_guard_conf_is_kept(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put("/etc/felhom/crash-guard.conf", b"LIMIT=5\n", 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files["/etc/felhom/crash-guard.conf"], b"LIMIT=5\n")
def test_oob_files_only_on_a_box_with_the_belt(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), oob=True)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertIn("/etc/sudoers.d/felhom-op", rep["bundle"]["written"])
self.assertEqual(rep["bundle"]["skipped"], [])
def test_previous_copies_are_kept(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put("/usr/local/sbin/felhom-pbs-apply", b"#!/bin/bash\necho old\n", 0o755)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files[rep["bundle"]["prev_dir"] + "/usr/local/sbin/felhom-pbs-apply"], b"#!/bin/bash\necho old\n")
class Refusals(unittest.TestCase):
"""Each: refused, and NOTHING on the box changed."""
def refused(self, box, code):
before = dict(box_files(box))
rc, rep = run(box)
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
self.assertEqual(box_files(box), before, "a refusal changed a file")
self.assertNotIn(osapply.BUNDLE_RECORD, box.files)
return rep
def test_wrong_sha_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job("0" * 64))
self.refused(box, "R18")
self.assertEqual(box.nonces, {}, "a wrong sha must not burn the job")
def test_bad_signature_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.sig_rc = 255
self.refused(box, "R3")
def test_other_op_is_refused(self):
data, sha = real_bundle()
self.refused(Box(data, signed_job(sha, op="agent_update")), "R3")
def test_other_host_is_refused(self):
data, sha = real_bundle()
self.refused(Box(data, signed_job(sha, host="demo-felhom-8363b5")), "R3")
def test_expired_job_is_refused(self):
data, sha = real_bundle()
self.refused(Box(data, signed_job(sha, expires="2026-10-04T11:55:00Z")), "R3")
def test_replayed_job_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.nonces = {"b1": box.clock + 600}
self.refused(box, "R3")
def test_version_mismatch_is_refused(self):
data, sha = real_bundle()
self.refused(Box(data, signed_job(sha, version="0.142.1")), "R18")
def test_sudoers_failing_visudo_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/sudoers.d/felhom-agent", lambda c: c + b"BROKEN-SYNTAX\n"))
self.refused(Box(data, signed_job(sha)), "R18")
def test_sudoers_dropping_the_route_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/sudoers.d/felhom-agent",
lambda c: c.replace(b"/usr/local/sbin/felhom-os-apply --plan", b"/bin/true --plan")))
self.refused(Box(data, signed_job(sha)), "R18")
def test_wrapper_without_bundle_mode_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/usr/local/sbin/felhom-os-apply",
lambda c: c.replace(b'BUNDLE_OP = "agent_config_update"', b'BUNDLE_OPX = 1')))
self.refused(Box(data, signed_job(sha)), "R18")
def test_python_syntax_error_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/usr/local/sbin/felhom-crash-guard", lambda c: c + b"\ndef (\n"))
self.refused(Box(data, signed_job(sha)), "R18")
def test_unit_with_runtime_directory_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/systemd/system/felhom-mgmt-watchdog.service",
lambda c: c + b"RuntimeDirectory=sshd\n"))
self.refused(Box(data, signed_job(sha)), "R18")
def test_agent_unit_not_as_the_agent_user_is_refused(self):
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/systemd/system/felhom-agent.service",
lambda c: c.replace(b"User=felhom-agent", b"User=root")))
self.refused(Box(data, signed_job(sha)), "R18")
def test_a_bundle_that_changes_a_signer_is_refused(self):
"""R17: the trust root is not a bundle's to change — not even a signed one."""
def add(f):
f.append({"path": osapply.TRUST_SIGNERS, "content_b64": base64.b64encode(b"evil-key\n").decode()})
data, sha = edited_bundle(add)
box = Box(data, signed_job(sha))
self.refused(box, "R17")
self.assertIn(b"felhom-op-1", box.files[osapply.TRUST_SIGNERS])
def test_a_path_outside_the_table_is_refused(self):
def add(f):
f.append({"path": "/etc/shadow", "content_b64": base64.b64encode(b"root::0:0\n").decode()})
data, sha = edited_bundle(add)
self.refused(Box(data, signed_job(sha)), "R16")
def test_content_not_matching_its_sha_is_refused(self):
b = json.loads(builder.build("0.143.0"))
b["files"][0]["content_b64"] = base64.b64encode(b"#!/bin/bash\nexit 0\n").decode()
data = json.dumps(b).encode()
self.refused(Box(data, signed_job(hashlib.sha256(data).hexdigest())), "R18")
def test_bundle_outside_the_plan_dir_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.plan["bundle"] = "/tmp/bundle-0.143.0.json"
box.files[PLAN] = json.dumps(box.plan).encode()
self.refused(box, "R1")
def test_bundle_not_owned_by_the_agent_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.uids[BUNDLE] = 0
self.refused(box, "R1")
def test_no_trust_file_is_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
del box.files[osapply.TRUST_FILE]
self.refused(box, "R3")
class SelfCheckUndo(unittest.TestCase):
def assert_restored(self, box, before, rep):
self.assertTrue(rep["bundle"]["rolled_back"], rep)
self.assertEqual(box_files(box), before, "the previous files must be back, byte for byte")
self.assertNotIn(osapply.BUNDLE_RECORD, box.files)
def with_old_files(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put("/usr/local/sbin/felhom-pbs-apply", b"#!/bin/bash\necho old\n", 0o755)
box.put("/etc/sudoers.d/felhom-agent", b"# old sudoers\n", 0o440)
return box
def test_route_missing_after_install_puts_everything_back(self):
box = self.with_old_files()
before = dict(box_files(box))
box.sudo_l = " (root) NOPASSWD: /bin/true\n"
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assert_restored(box, before, rep)
self.assertEqual(box.calls[-1], ["visudo", "-c"], "the undo re-checks the whole sudoers")
def test_visudo_failing_after_install_puts_everything_back(self):
box = self.with_old_files()
before = dict(box_files(box))
box.visudo_fail = True
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assert_restored(box, before, rep)
def test_crash_guard_disagreeing_with_kernel_panic_puts_everything_back(self):
box = self.with_old_files()
before = dict(box_files(box))
box.guard = {"armed": True, "kernel_panic": 10}
box.host_orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:3] == ["systemctl", "enable", "--now"]:
box.calls.append(argv)
return 0, "", "" # the unit "started" but kernel.panic stayed 0
return box.host_orig(argv, timeout, stdin)
box.host = host
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assert_restored(box, before, rep)
class TrustBootstrap(unittest.TestCase):
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), signers=False)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.verified[0], osapply.PINNED_SIGNERS, "verified against the pinned key, nothing else")
self.assertTrue(rep["bundle"]["signers_created"])
self.assertEqual(box.files[osapply.TRUST_SIGNERS], osapply.pinned_signers_line().encode())
self.assertEqual(box.modes[osapply.TRUST_SIGNERS], 0o644)
def test_missing_signers_and_a_job_the_pinned_key_did_not_sign(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), signers=False)
box.sig_rc = 255
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"))
self.assertNotIn(osapply.TRUST_SIGNERS, box.files)
def test_present_signers_are_never_touched(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put(osapply.TRUST_SIGNERS, b'felhom-op-2 namespaces="felhom-op-v1" ssh-ed25519 BBBB felhom-op-2\n', 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.verified[0], osapply.TRUST_SIGNERS)
self.assertFalse(rep["bundle"]["signers_created"])
self.assertIn(b"felhom-op-2", box.files[osapply.TRUST_SIGNERS])
def test_agent_writable_signers_are_refused(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.uids[osapply.TRUST_SIGNERS] = 999
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"))
def test_pinned_operator_key_equals_the_installers(self):
"""The bootstrap key is exactly the one felhom-host-install.sh pins (OPERATOR_KEY_OPERATIONAL_*)."""
text = INSTALLER.read_text()
kid = re.search(r'^OPERATOR_KEY_OPERATIONAL_ID="([^"]+)"', text, re.M).group(1)
line = re.search(r'^OPERATOR_KEY_OPERATIONAL_LINE="([^"]+)"', text, re.M).group(1)
self.assertEqual((osapply.PINNED_OPERATOR_KEY_ID, osapply.PINNED_OPERATOR_KEY_LINE), (kid, line))
# and the file format is the installer's printf, byte for byte
self.assertIn("printf '%s namespaces=\"felhom-op-v1\" %s\\n'", text)
class InstallerEntry(unittest.TestCase):
def test_installer_installs_without_a_signature(self):
data, sha = real_bundle()
box = Box(data, None)
box.put("/root/bundle.json", data, 0o600)
rc, rep = run(box, ["felhom-os-apply", "--install-bundle", "/root/bundle.json", "--sha256", sha])
self.assertEqual(rc, 0, rep)
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["authority"], "installer")
def test_installer_entry_checks_the_sha(self):
data, sha = real_bundle()
box = Box(data, None)
box.put("/root/bundle.json", data, 0o600)
rc, rep = run(box, ["felhom-os-apply", "--install-bundle", "/root/bundle.json", "--sha256", "1" * 64])
self.assertEqual((rc, rep["refused"]["code"]), (2, "R18"))
def test_installer_entry_is_refused_through_sudo(self):
data, sha = real_bundle()
box = Box(data, None)
box.put("/root/bundle.json", data, 0o600)
rc, rep = run(box, ["felhom-os-apply", "--install-bundle", "/root/bundle.json", "--sha256", sha], {"SUDO_UID": "999"})
self.assertEqual((rc, rep["refused"]["code"]), (2, "R1"))
self.assertNotIn(osapply.BUNDLE_RECORD, box.files)
def test_self_check_answers(self):
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
rc = osapply.main(["felhom-os-apply", "--self-check"], runner=Box(b"", None))
self.assertEqual(rc, 0)
self.assertIn("bundle-format=1", buf.getvalue())
class Facts(unittest.TestCase):
def test_state_reports_drift_against_the_record(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
run(box)
box.put("/usr/local/sbin/felhom-pbs-apply", b"#!/bin/bash\necho by hand\n", 0o755)
a = osapply.Apply(box, PLAN)
st = osapply.Bundle(a).state()
self.assertEqual(st["version"], "0.143.0")
self.assertEqual(st["drift"], ["/usr/local/sbin/felhom-pbs-apply"])
self.assertTrue(st["signers_present"])
def test_state_without_a_record_says_none(self):
box = Box(b"", None)
st = osapply.Bundle(osapply.Apply(box, PLAN)).state()
self.assertEqual(st["version"], "none")
self.assertNotIn("drift", st)
self.assertEqual(st["live"]["/etc/sudoers.d/felhom-agent"], "absent")
class Builder(unittest.TestCase):
def test_reproducible(self):
self.assertEqual(builder.build("0.143.0"), builder.build("0.143.0"))
def test_every_source_exists_and_every_dest_is_unique(self):
dests = [e[0] for e in osapply.BUNDLE_FILES]
self.assertEqual(len(dests), len(set(dests)))
for _, src, *_ in osapply.BUNDLE_FILES:
self.assertTrue((REPO / "configs" / src).is_file(), src)
def test_every_root_file_the_installer_writes_is_in_the_bundle(self):
"""One source of truth: a felhom root-owned path the installer names must be a bundle path, a trust file, or a
path the AGENT itself writes at run time (named here, with why). Scope is a regex over the WHOLE installer."""
text = INSTALLER.read_text()
found = set(re.findall(r"(/usr/local/sbin/felhom-[a-z-]+|/etc/systemd/system/felhom-[a-z.-]+|"
r"/etc/sudoers\.d/felhom-[a-z-]+|/etc/felhom-oob\.nft|/etc/tmpfiles\.d/felhom-[a-z.-]+|"
r"/etc/felhom/[a-z.-]+)", text))
agent_writes = {"/usr/local/sbin/felhom-shared-parent", "/etc/systemd/system/felhom-shared-parent.service"}
trust = {osapply.TRUST_FILE, osapply.TRUST_SIGNERS, osapply.TRUST_SIGNERS + ".tmp", osapply.BUNDLE_RECORD}
missing = sorted(p for p in found if p not in osapply.BUNDLE_DESTS and p not in agent_writes | trust)
self.assertEqual(missing, [], "the installer writes these root files, but the bundle does not carry them")
# the limits drop-in is named through $AGENT_UNIT in the installer
self.assertIn("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", osapply.BUNDLE_DESTS)
# ---------- R-861 (agent v0.146.0): the agent binary only by an operator-signed agent_update, checked as root ----------
STAGED = "/var/lib/felhom-agent/selfupdate/felhom-agent-0.146.0"
NEW_BIN = b"\x7fELF the new agent"
def update_job(sha, version="0.146.0", op="agent_update", nonce="u1", host=HOST):
blob = json.dumps({"expires_at": "2026-10-04T12:30:00Z", "issued_at": "2026-10-04T11:50:00Z", "key_id": "felhom-op-1",
"nonce": nonce, "op": op, "params": {"sha256": sha, "version": version},
"target": {"guest_id": "", "host_id": host}}, sort_keys=True).encode()
return {"blob_b64": base64.b64encode(blob).decode(), "sig": "-----BEGIN SSH SIGNATURE-----\nx\n-----END SSH SIGNATURE-----\n"}
def update_box(job, staged=STAGED, content=NEW_BIN):
box = Box(b"{}", None)
box.put(PLAN, json.dumps({"release_id": "agent-0.146.0", "layer": "host", "mode": "agent_update",
"signed": job, "staged": staged}).encode(), 0o600, uid=999)
box.put(STAGED, content, 0o755, uid=999)
return box
def wrapper_calls(box):
return [c for c in box.calls if c and c[0] == osapply.SELFUPDATE_WRAPPER and c[1:2] == ["apply"]]
class AgentUpdate(unittest.TestCase):
"""RED-PROOF: make agent_update skip verify_signed → test_a_bad_signature_never_reaches_the_wrapper fails."""
def test_signed_update_flips_and_burns_the_nonce(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(wrapper_calls(box), [[osapply.SELFUPDATE_WRAPPER, "apply", STAGED, sha]])
self.assertIn("u1", box.nonces)
self.assertEqual(rep["agent_update"]["version"], "0.146.0")
def test_a_bad_signature_never_reaches_the_wrapper(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha))
box.sig_rc = 1
rc, rep = run(box)
self.assertTrue(rep.get("refused"), f"a job whose signature does not verify was NOT refused: {rep}")
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"), rep)
self.assertEqual(wrapper_calls(box), [])
def test_a_staged_binary_the_signature_does_not_pin_is_refused(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha), content=b"\x7fELF something the agent put there")
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R19"), rep)
self.assertEqual(wrapper_calls(box), [])
self.assertNotIn("u1", box.nonces, "a refused job keeps its nonce (the operator fixes the file, not the key)")
def test_another_staging_path_is_refused(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha), staged="/tmp/felhom-agent-0.146.0")
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R19"), rep)
self.assertEqual(wrapper_calls(box), [])
def test_a_bundle_job_is_not_an_agent_update(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha, op="agent_config_update"))
rc, rep = run(box)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"), rep)
def test_another_hosts_job_and_a_replay_are_refused(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha, host="tester-1-d70be4"))
self.assertEqual(run(box)[1]["refused"]["code"], "R3")
box2 = update_box(update_job(sha))
box2.nonces["u1"] = 1999999999
self.assertEqual(run(box2)[1]["refused"]["code"], "R3")
self.assertEqual(wrapper_calls(box) + wrapper_calls(box2), [])
def test_a_failed_flip_keeps_the_nonce(self):
sha = hashlib.sha256(NEW_BIN).hexdigest()
box = update_box(update_job(sha))
orig = box.host
box.host = lambda argv, timeout=600, stdin=None: (1, "", "refusing apply: sha mismatch") if argv[:1] == [osapply.SELFUPDATE_WRAPPER] else orig(argv, timeout, stdin)
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assertNotIn("u1", box.nonces)
if __name__ == "__main__":
unittest.main()
+250 -5
View File
@@ -73,6 +73,7 @@ class Fake:
self.sig_rc = 0
self.nonces = {}
self.clock = 1791115200.0 # 2026-10-04T12:00:00Z
self.saved_reports = [] # R-868: (plan path, report) the wrapper kept on disk
self.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
self.stats[osapply.TRUST_FILE] = St(mode=statmod.S_IFREG | 0o644, uid=0)
self.files[osapply.TRUST_SIGNERS] = 'felhom-op-1 namespaces="felhom-op-v1" ssh-ed25519 AAAA\n'
@@ -113,6 +114,10 @@ class Fake:
def log(self, line):
self.logs.append(line)
def save_report(self, plan_path, report):
self.saved_reports.append((plan_path, json.loads(json.dumps(report))))
return plan_path.replace("/plan-", "/report-")
def host(self, argv, timeout=600, stdin=None):
self.calls.append(("host", argv))
if argv[0] == "/usr/sbin/pct" and argv[1] == "status":
@@ -149,6 +154,8 @@ class Fake:
if cmd == "dpkg" and a[1] == "--audit":
return 0, self.dpkg_audit, ""
if cmd == "dpkg" and a[1] == "--configure":
self.configured_calls = getattr(self, "configured_calls", 0) + 1
self.dpkg_journal = [] # `dpkg --configure -a` replays and empties the update journal
return 0, "", ""
if cmd == "fuser":
return (0, " 123", "") if self.lock_held else (1, "", "")
@@ -168,9 +175,12 @@ class Fake:
if "-f" in a:
self.dpkg_audit = ""
return 0, "Setting up x (1) ...\n" if getattr(self, "repaired", False) else "", ""
if "-s" in a:
return self.sim(a)
if "--print-uris" in a or "-s" in a:
return self.sim(a) # --print-uris prints and installs nothing, with or without -s (9202, 2026-10-05)
if "install" in a:
if getattr(self, "dpkg_journal", []):
# real apt (demo-hp 2026-10-05): a non-empty update journal refuses every install
return 100, "", "E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.\n"
if self.install_rc:
return self.install_rc, "", "E: boom"
for x in a:
@@ -186,6 +196,14 @@ class Fake:
return 0, self.engine + "\n", ""
if cmd == "docker" and a[1:3] == ["ps", "-q"]:
return 0, "".join(i + "\n" for i in self.ids), ""
if cmd == "docker" and a[1] == "inspect":
mounts = {"aaa111": "/felhom-controller|/var/run/docker.sock;/app/data;", "bbb222": "/app|/data;"}
return 0, mounts.get(a[-1], "/other|;") + "\n", ""
if cmd == "docker" and a[1] == "restart":
self.restarted_containers = a[2:]
return 0, "", ""
if cmd == "docker" and a[1] == "exec":
return (1, "", "Cannot connect to the Docker daemon") if getattr(self, "controller_blind", False) else (0, self.engine + "\n", "")
if cmd == "docker":
ids = self.ids + ["x"] * 2
return 0, f"felhom-controller\trunning\tUp 1 hour (healthy)\t{ids[0]}\napp\trunning\tUp 1 hour (healthy)\t{ids[1]}\n", ""
@@ -207,6 +225,8 @@ class Fake:
return 0, "", ""
if cmd == "getent":
return 0, "1.2.3.4 deb.debian.org\n", ""
if cmd == "sh" and a[2] == osapply.DPKG_STATE_SCRIPT:
return 0, self.dpkg_audit + osapply.JOURNAL_MARK + "\n" + "".join(j + "\n" for j in getattr(self, "dpkg_journal", [])), ""
if cmd == "sh":
if "vmlinuz" in a[2]:
return 0, "/boot/vmlinuz-7.0.2-6-pve\n/boot/vmlinuz-7.0.14-20-pve\n", ""
@@ -224,7 +244,14 @@ class Fake:
def sim(self, a):
if "--print-uris" in a:
return 0, "'http://x/libc6.deb' libc6.deb 4000000 SHA256:x\n", ""
if "-s" in a:
# real apt (measured 9202 2026-10-05): with -s it prints the SIMULATION, no URI list
return 0, "Inst libc6 [2.41-12+deb13u4] (2.41-12+deb13u4 Debian:13.7/stable [amd64])\n", ""
# real apt's line shape, verbatim from 9202 2026-10-05 (audits/night-fixes-2026-10-05/partC/)
return 0, ("Need to get 4347 kB of archives.\n"
"'http://deb.debian.org/debian/pool/main/b/bash/bash_5.2.37-2%2bb10_amd64.deb' bash_5.2.37-2+b10_amd64.deb 1500792 MD5Sum:27b11721fea83d73b96e0f7023863771\n"
"'http://deb.debian.org/debian/pool/main/g/glibc/libc6_2.41-12%2bdeb13u4_amd64.deb' libc6_2.41-12+deb13u4_amd64.deb 2846580 MD5Sum:5559581916477ef1f57ea9f82cecf22e\n"
+ getattr(self, "extra_uris", "")), ""
if "dist-upgrade" in a:
if getattr(self, "pending_sim", None) is not None and not getattr(self, "_pending_used", False):
self._pending_used = True
@@ -265,7 +292,7 @@ class Happy(unittest.TestCase):
self.assertEqual(rep["pending"][0]["name"], "bash")
self.assertTrue(any(l.startswith("os-apply: REPAIR ") for l in f.logs), "the repair line must always print")
self.assertTrue(any(l.startswith("os-apply: DONE rc=0") for l in f.logs))
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2] and "--print-uris" not in c[2]]
self.assertTrue(inst and "Dpkg::Options::=--force-confold" in inst[0][2], "must keep existing config files")
def test_already_current_is_a_no_op(self):
@@ -335,7 +362,7 @@ class Refusals(unittest.TestCase):
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
self.assertTrue(any(l.startswith(f"os-apply: REFUSED: {code} ") for l in f.logs), f.logs)
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2] and "--print-uris" not in c[2]]
self.assertEqual(inst, [], "a refusal must install nothing")
return rep
@@ -446,6 +473,24 @@ class Refusals(unittest.TestCase):
f.free = 100 * 1024 * 1024
self.refused(f, "R8")
# R-865: the download is the real one. 2 GB of URIs, 5 GB free: 5 GB < 3 x 2 GB -> R8, with the size in the line.
# COMPANION RED-PROOF: put "-s" back into download_bytes -> 0 B -> no refusal -> this test fails.
def test_R8_counts_the_real_download(self):
f = Fake()
f.free = 5 * 1024 ** 3
f.extra_uris = "'http://deb.debian.org/debian/pool/main/b/big/big_1_amd64.deb' big_1_amd64.deb 2000000000 MD5Sum:x\n"
rep = self.refused(f, "R8")
self.assertIn("download 2004347372 B", str(rep))
def test_download_bytes_never_simulates(self):
f = Fake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
calls = [c[2] for c in f.calls if c[0] == "guest" and "--print-uris" in c[2]]
self.assertTrue(calls, "download_bytes was never called")
for c in calls:
self.assertNotIn("-s", c, f"--print-uris with -s prints no URIs: {c}")
def test_R9_guest_locked_by_a_backup(self):
f = Fake()
f.files["/etc/pve/lxc/9201.conf"] = CONF_OK + "lock: backup\n"
@@ -804,6 +849,26 @@ class DockerLane(unittest.TestCase):
rc, rep = run(f)
self.assertEqual(rep["plan"]["upgrade"], 0, "an older version on an unsigned step is 'already', never installed")
def test_step_restarts_only_the_socket_users(self):
# R-858: after an engine step, ONLY the container that mounts the docker socket is restarted (here the controller)
f = docker_fake(signed=signed_job())
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.restarted_containers, ["felhom-controller"])
self.assertEqual(rep["socket_restarted"], ["felhom-controller"])
def test_no_install_restarts_nothing(self):
f = docker_fake(signed=signed_job())
f.installed.update({"docker-ce": "5:29.8.2-1~debian.13~trixie", "containerd.io": "2.3.6-1~debian.13~trixie"})
rc, rep = run(f)
self.assertFalse(hasattr(f, "restarted_containers"), "nothing installed -> no container restart")
def test_health_says_when_the_controller_cannot_reach_docker(self):
f = docker_fake(signed=signed_job())
f.controller_blind = True
rc, rep = run(f)
self.assertFalse(rep["health_after"]["controller_docker_ok"])
def test_health_carries_container_ids(self):
f = docker_fake(signed=signed_job())
rc, rep = run(f)
@@ -934,3 +999,183 @@ class RealSignatureCheck(unittest.TestCase):
if __name__ == "__main__":
unittest.main()
class UnsentReport(unittest.TestCase):
"""R-868 (v0.144.0): an apply pass keeps its report on disk until the agent has sent it.
COMPANION RED-PROOF: drop the r.save_report call in main() -> test_apply_keeps_a_copy fails."""
def test_apply_keeps_a_copy_with_the_agents_ids(self):
f = Fake()
f.plan.update(run_id="20261005T0257-ab12", trigger="debug", ring=0)
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(len(f.saved_reports), 1, "an apply pass must keep its report on disk")
path, saved = f.saved_reports[0]
self.assertEqual(path, PLAN)
self.assertEqual(saved, rep, "the copy is the report the agent would have read")
self.assertEqual((saved["run_id"], saved["trigger"], saved["ring"]), ("20261005T0257-ab12", "debug", 0))
def test_a_refusal_is_kept_too(self):
f = Fake()
f.free = 1
rc, rep = run(f)
self.assertEqual(rc, 2)
self.assertEqual(f.saved_reports[0][1]["refused"]["code"], "R8")
def test_other_modes_keep_nothing(self):
f = Fake()
f.plan["mode"] = "health"
run(f)
self.assertEqual(f.saved_reports, [])
def test_odd_ids_are_dropped_not_trusted(self):
f = Fake()
f.plan.update(run_id="../../etc/x", trigger="Night; rm", ring=True)
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
for k in ("run_id", "trigger", "ring"):
self.assertNotIn(k, rep)
class SaveReportOnDisk(unittest.TestCase):
"""The real Runner.save_report on a temp dir: root writes into the AGENT's directory, so a symlink must never
be followed — neither for the directory nor for the file name."""
def setUp(self):
import tempfile
self.tmp = tempfile.mkdtemp()
self.dir = os.path.join(self.tmp, "os")
os.mkdir(self.dir)
self.prev = osapply.PLAN_DIR
osapply.PLAN_DIR = self.dir
self.r = osapply.Runner()
self.r.agent_uid = lambda: os.getuid()
def tearDown(self):
import shutil
osapply.PLAN_DIR = self.prev
shutil.rmtree(self.tmp)
def test_writes_0600_next_to_the_plan(self):
p = self.r.save_report(os.path.join(self.dir, "plan-r1-guest-apply.json"), {"mode": "apply"})
self.assertEqual(p, os.path.join(self.dir, "report-r1-guest-apply.json"))
st = os.stat(p)
self.assertEqual(statmod.S_IMODE(st.st_mode), 0o600)
with open(p) as fh:
self.assertEqual(json.load(fh), {"mode": "apply"})
def test_a_symlink_at_the_name_is_replaced_not_followed(self):
victim = os.path.join(self.tmp, "victim")
with open(victim, "w") as fh:
fh.write("untouched")
os.symlink(victim, os.path.join(self.dir, "report-r2-guest-apply.json"))
self.r.save_report(os.path.join(self.dir, "plan-r2-guest-apply.json"), {"mode": "apply"})
with open(victim) as fh:
self.assertEqual(fh.read(), "untouched")
self.assertFalse(os.path.islink(os.path.join(self.dir, "report-r2-guest-apply.json")))
def test_a_symlinked_directory_is_refused(self):
real = os.path.join(self.tmp, "elsewhere")
os.mkdir(real)
link = os.path.join(self.tmp, "linked")
os.symlink(real, link)
osapply.PLAN_DIR = link
with self.assertRaises(OSError):
self.r.save_report(os.path.join(link, "plan-r3-guest-apply.json"), {"mode": "apply"})
self.assertEqual(os.listdir(real), [])
class AgentDiesMidPass(unittest.TestCase):
"""R-868, MEASURED LIVE 2026-10-05 05:45 UTC on demo-hp (agent v0.144.0): the agent was kill -9-ed while apt-get
ran; apt finished all 13 packages, but the wrapper's next log line went to a stderr pipe nobody reads any more ->
BrokenPipeError -> the wrapper died before save_report: no DONE in the journal, no kept copy, no report.
COMPANION RED-PROOF: let Runner.log write to stderr unguarded -> this test fails (BrokenPipeError)."""
def test_a_dead_reader_does_not_stop_the_report_copy(self):
import io, sys, contextlib
class DeadPipe(io.TextIOBase):
dead = False
def write(self, s):
if DeadPipe.dead:
raise BrokenPipeError(32, "Broken pipe")
return len(s)
def flush(self):
if DeadPipe.dead:
raise BrokenPipeError(32, "Broken pipe")
class DyingFake(Fake):
def emulate(self, argv):
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
if a and a[0] == "apt-get" and "install" in a and "-s" not in a and "--print-uris" not in a:
DeadPipe.dead = True # the agent is killed while apt-get runs
return super().emulate(argv)
f = DyingFake()
journal = []
f.log = lambda line: osapply.Runner.log(f, line) # the REAL log(): stderr, then the journal
prev_run = osapply.subprocess.run
osapply.subprocess.run = lambda argv, *a, **kw: (journal.append(argv[-1]) if argv[0] == "logger"
else prev_run(argv, *a, **kw)) # never the real `logger`
prev_err, prev_out = sys.stderr, sys.stdout
sys.stderr, sys.stdout = DeadPipe(), DeadPipe()
try:
rc = osapply.main(["felhom-os-apply", "--plan", PLAN], runner=f)
finally:
sys.stderr, sys.stdout = prev_err, prev_out
osapply.subprocess.run = prev_run
DeadPipe.dead = False
self.assertEqual(rc, 0)
self.assertEqual(len(f.saved_reports), 1, "the kept copy is the only way this report reaches the hub")
self.assertEqual(len(f.saved_reports[0][1]["upgraded"]), 2)
self.assertTrue(any(l.startswith("os-apply: DONE") for l in journal), "the journal must still get the DONE line")
class CrashLeftTheJournal(unittest.TestCase):
"""R-876 — THE MEASURED SHAPE (demo-hp 2026-10-05, a crash while dpkg unpacked): `dpkg --audit` CLEAN, but
/var/lib/dpkg/updates holds 3 files; apt refuses every install ("dpkg was interrupted"). v0.144.1 logged
`REPAIR configured=0 fixed=0`, then `FAILED rc=100`, every pass, until a person ran `dpkg --configure -a`.
COMPANION RED-PROOFS: drop `or journal` from repair()'s condition -> test 1 fails; drop the interrupted-retry
-> test 3 fails; make repair() always run -> test 2 fails (R-845's speed)."""
def test_the_next_pass_repairs_by_itself_and_finishes(self):
f = Fake()
f.dpkg_audit = ""
f.dpkg_journal = ["0000", "0001", "0002"]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["repair"]["journal_before"], 3)
self.assertTrue(rep["repair"]["clean_after"])
self.assertEqual(len(rep["upgraded"]), 2)
cfg = next(i for i, c in enumerate(f.calls) if c[0] == "guest" and "--configure" in c[2])
inst = next(i for i, c in enumerate(f.calls) if c[0] == "guest" and "install" in c[2] and "-s" not in c[2]
and "-f" not in c[2] and "--print-uris" not in c[2])
self.assertLess(cfg, inst, "the repair must run before the install")
self.assertFalse(any("INTERRUPTED" in l for l in f.logs), "the journal check must catch it BEFORE apt refuses")
def test_a_clean_pass_still_costs_one_state_call(self):
f = Fake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
state = [c for c in f.calls if c[0] == "guest" and c[2][:2] == ["sh", "-c"] and c[2][2] == osapply.DPKG_STATE_SCRIPT]
self.assertEqual(len(state), 1, "R-845: a clean pass reads dpkg's state ONCE and runs no repair")
self.assertEqual(getattr(f, "configured_calls", 0), 0)
def test_apt_interrupted_is_repaired_and_retried_once(self):
"""The belt: the journal probe saw nothing (a race, an odd layout), apt still says interrupted."""
f = Fake()
orig = f.emulate
state = {"first": True}
def emulate(argv):
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
if a[0] == "apt-get" and "install" in a and "-s" not in a and "-f" not in a and "--print-uris" not in a and state["first"]:
state["first"] = False
return 100, "", "E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.\n"
return orig(argv)
f.emulate = emulate
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertTrue(any("INTERRUPTED" in l for l in f.logs), f.logs)
self.assertTrue(any(l.startswith("os-apply: REPAIR ") and l.endswith("forced") for l in f.logs), f.logs)
+309
View File
@@ -0,0 +1,309 @@
#!/usr/bin/env python3
"""Tests for felhom-priv-apply (R-861, `03` §3.1). An in-memory host plays the files; nothing real is written or run.
Each refusal rule has a test that feeds it the ATTACK it exists for; each accepted shape is a file the agent really
renders (the Go contract tests feed the live renderers too). Red-proof: audits/hub-safety-2026-10-05/partF/.
Run: python3 configs/test_felhom_priv_apply.py (also run by internal/privapply's Go test)
"""
import sys
sys.dont_write_bytecode = True
import importlib.machinery
import importlib.util
import os
import pathlib
import unittest
HERE = pathlib.Path(__file__).resolve().parent
_loader = importlib.machinery.SourceFileLoader("privapply", os.environ.get("PRIVAPPLY_UNDER_TEST", str(HERE / "felhom-priv-apply")))
_spec = importlib.util.spec_from_loader("privapply", _loader)
pa = importlib.util.module_from_spec(_spec)
_loader.exec_module(pa)
AGENT_UID = 999
class FakeHost:
def __init__(self):
self.src, self.dest, self.logs, self.writes = {}, {}, [], []
self.owner, self.kind = {}, {}
def stage(self, path, text, uid=AGENT_UID, kind="file"):
self.src[path] = text.encode() if isinstance(text, str) else text
self.owner[path], self.kind[path] = uid, kind
def agent_uid(self):
return AGENT_UID
def read_source(self, path):
# mirrors Host.read_source's refusals: absent, not a regular file (a symlink is refused by O_NOFOLLOW), owner, size
if path not in self.src:
raise pa.Refused("P1", f"cannot open the staged file {path}")
if self.kind[path] != "file":
raise pa.Refused("P1", f"{path} is not a regular file")
if self.owner[path] != AGENT_UID:
raise pa.Refused("P1", f"{path} is not owned by felhom-agent")
if len(self.src[path]) > pa.MAX_BYTES:
raise pa.Refused("P1", f"{path} is larger than {pa.MAX_BYTES} bytes")
return self.src[path]
def read_dest(self, path):
return self.dest.get(path)
def install(self, dest, data, mode):
self.writes.append((dest, mode))
self.dest[dest] = data
def log(self, line):
self.logs.append(line)
LOCAL_UNIT = """# Managed by felhom-agent — do not edit by hand.
[Unit]
Description=Felhom storage mount 91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
After=local-fs-pre.target
[Mount]
What=/dev/disk/by-uuid/91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
Where=/mnt/hdd_1
Type=ext4
[Install]
WantedBy=multi-user.target
"""
NET_MOUNT = """# felhom network storage — do not edit by hand.
[Unit]
Description=Felhom network storage media (nfs)
[Mount]
What=nas.lan:/volume1/media
Where=/mnt/felhom-drives/media
Type=nfs4
Options=vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev,retry=0,nosuid,nodev
"""
NET_AUTOMOUNT = """# felhom network storage — do not edit by hand.
[Unit]
Description=Felhom network storage automount media (nfs)
[Automount]
Where=/mnt/felhom-drives/media
TimeoutIdleSec=600
[Install]
WantedBy=multi-user.target
"""
NET_NAME = "mnt-felhom\\x2ddrives-media.mount"
DNS_BASE = """# felhom split-horizon resolver — host base config (agent-managed; DO NOT EDIT)
bind-interfaces
listen-address=192.168.0.104
listen-address=127.0.0.1
no-resolv
server=1.1.1.1
server=9.9.9.9
"""
DNS_GUEST = """# felhom split-horizon DNS — customer demo-hp (agent-managed; DO NOT EDIT)
local=/enkisfelhom.hu/
address=/enkisfelhom.hu/192.168.0.138
"""
K = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=" # base64 of 32 bytes
WG = f"""# felhom offsite tunnel — agent-managed (S3); DO NOT EDIT
[Interface]
PrivateKey = {K}
Address = 10.77.0.3/32
MTU = 1280
[Peer]
PublicKey = {K}
Endpoint = 49.12.1.2:51820
AllowedIPs = 10.77.0.1/32, 10.77.0.250/32
PersistentKeepalive = 25
"""
KEYLINE = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL8z0qCNgA3x2xxAB0Qj5ro8waFjGZ8Ta/sWB63tlLw+ felhom-op-1\n"
def run(host, *argv):
return pa.main(list(argv), host=host)
class Accepts(unittest.TestCase):
"""What the agent really writes is accepted and installed, root-owned, at the fixed destination."""
def test_local_unit_installed(self):
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/mnt-hdd_1.mount", LOCAL_UNIT)
self.assertEqual(run(h, "unit", "mnt-hdd_1.mount"), 0)
self.assertEqual(h.writes, [("/etc/systemd/system/mnt-hdd_1.mount", 0o644)])
def test_local_unit_without_type(self): # the N100's unit has no Type= (autodetect)
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/mnt-hdd_1.mount", LOCAL_UNIT.replace("Type=ext4\n", ""))
self.assertEqual(run(h, "unit", "mnt-hdd_1.mount"), 0)
def test_network_pair(self):
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/" + NET_NAME, NET_MOUNT)
h.stage("/var/lib/felhom-agent/units/mnt-felhom\\x2ddrives-media.automount", NET_AUTOMOUNT)
self.assertEqual(run(h, "unit", NET_NAME), 0)
self.assertEqual(run(h, "unit", "mnt-felhom\\x2ddrives-media.automount"), 0)
def test_identical_is_not_rewritten(self):
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/mnt-hdd_1.mount", LOCAL_UNIT)
h.dest["/etc/systemd/system/mnt-hdd_1.mount"] = LOCAL_UNIT.encode()
self.assertEqual(run(h, "unit", "mnt-hdd_1.mount"), 0)
self.assertEqual(h.writes, [])
def test_dnsmasq_both_dropins(self):
h = FakeHost()
h.stage("/tmp/felhom-resolver-123.conf", DNS_BASE)
self.assertEqual(run(h, "dnsmasq", "/tmp/felhom-resolver-123.conf", "felhom-resolver-base.conf"), 0)
h.stage("/tmp/felhom-resolver-124.conf", DNS_GUEST)
self.assertEqual(run(h, "dnsmasq", "/tmp/felhom-resolver-124.conf", "felhom-demo-hp.conf"), 0)
self.assertIn(("/etc/dnsmasq.d/felhom-demo-hp.conf", 0o644), h.writes)
def test_wg_installed_0600(self):
h = FakeHost()
h.stage(pa.WG_SRC, WG)
self.assertEqual(run(h, "wg"), 0)
self.assertEqual(h.writes, [(pa.WG_DEST, 0o600)])
def test_sshd_template_and_key(self):
h = FakeHost()
h.stage(pa.SSHD_SRC, pa.render_sshd(8822))
h.stage(pa.KEY_SRC, KEYLINE)
self.assertEqual(run(h, "sshd-config"), 0)
self.assertEqual(run(h, "sshd-key"), 0)
h.stage(pa.KEY_SRC, "") # clearing the operator login is allowed
self.assertEqual(run(h, "sshd-key"), 0)
def test_escape_matches_systemd(self):
self.assertEqual(pa.systemd_escape_path("/mnt/felhom-drives/media"), "mnt-felhom\\x2ddrives-media")
self.assertEqual(pa.systemd_escape_path("/mnt/hdd_1"), "mnt-hdd_1")
self.assertEqual(pa.systemd_escape_path("/mnt/.x"), "mnt-.x") # a dot is escaped only at the very start
class Refuses(unittest.TestCase):
"""Each rule, with the attack it exists for. Nothing is written on a refusal."""
def refused(self, h, argv, rule):
rc = run(h, *argv)
self.assertIn(rc, (2, 3), f"{argv} was accepted")
self.assertEqual(h.writes, [], f"{argv} wrote something")
self.assertTrue(any(f"[{rule}]" in l for l in h.logs), f"{argv}: rule {rule} not logged: {h.logs}")
def unit(self, text, name="mnt-hdd_1.mount"):
h = FakeHost()
h.stage("/var/lib/felhom-agent/units/" + name, text)
return h, ["unit", name]
def test_U1_name_outside_mnt(self):
h = FakeHost()
self.refused(h, ["unit", "etc-sudoers.d.mount"], "U1")
self.refused(h, ["unit", "../../etc/x.mount"], "U1")
self.refused(h, ["unit", "mnt-x.service"], "U1")
def test_U2_service_section(self):
self.refused(*self.unit(LOCAL_UNIT + "\n[Service]\nExecStart=/bin/sh -c id\n"), "U2")
def test_U2_unknown_key(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Type=ext4", "Type=ext4\nDirectoryMode=0777")), "U2")
def test_U3_bind_over_sudoers_dir(self):
# the R-861 attack: mount an agent-owned directory over /etc/sudoers.d
t = LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/etc/sudoers.d")
self.refused(*self.unit(t, "mnt-hdd_1.mount"), "U3")
def test_U3_name_must_match_where(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/mnt/other")), "U3")
def test_U3_traversal_in_where(self):
# the name passes U1 and equals the escaped Where — ONLY the Where rule stops a mount at /mnt/../etc = /etc
self.assertEqual(pa.systemd_escape_path("/mnt/../etc") + ".mount", "mnt-..-etc.mount")
self.refused(*self.unit(LOCAL_UNIT.replace("Where=/mnt/hdd_1", "Where=/mnt/../etc"), "mnt-..-etc.mount"), "U3")
def test_U4_what_is_an_agent_directory(self):
t = LOCAL_UNIT.replace("What=/dev/disk/by-uuid/91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae", "What=/var/lib/felhom-agent/evil")
self.refused(*self.unit(t), "U4")
def test_U4_tmpfs(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Type=ext4", "Type=tmpfs")), "U4")
def test_U5_bind_option(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Type=ext4", "Type=ext4\nOptions=bind")), "U5")
def test_U5_network_without_nosuid(self):
t = NET_MOUNT.replace(",nosuid,nodev", "")
self.refused(*self.unit(t, NET_NAME), "U5")
def test_U5_suid_option(self):
self.refused(*self.unit(LOCAL_UNIT.replace("Type=ext4", "Type=ext4\nOptions=suid,dev")), "U5")
def test_U3_network_outside_drives(self):
t = NET_MOUNT.replace("/mnt/felhom-drives/media", "/mnt/media")
self.refused(*self.unit(t, "mnt-media.mount"), "U3")
def test_D1_dhcp_script(self): # runs as root
h = FakeHost()
h.stage("/tmp/felhom-resolver-1.conf", DNS_BASE + "dhcp-script=/var/lib/felhom-agent/x.sh\n")
self.refused(h, ["dnsmasq", "/tmp/felhom-resolver-1.conf", "felhom-x.conf"], "D1")
def test_D1_conf_dir_and_log_file(self):
for extra in ("conf-dir=/var/lib/felhom-agent\n", "log-facility=/etc/sudoers.d/x\n", "user=root\n"):
h = FakeHost()
h.stage("/tmp/felhom-resolver-1.conf", DNS_GUEST + extra)
self.refused(h, ["dnsmasq", "/tmp/felhom-resolver-1.conf", "felhom-x.conf"], "D1")
def test_D2_paths(self):
h = FakeHost()
self.refused(h, ["dnsmasq", "/etc/shadow", "felhom-x.conf"], "D2")
self.refused(h, ["dnsmasq", "/tmp/felhom-resolver-1.conf", "../sudoers.d/x.conf"], "D2")
def test_W1_postup(self): # wg-quick runs PostUp as root
h = FakeHost()
h.stage(pa.WG_SRC, WG.replace("MTU = 1280", "MTU = 1280\nPostUp = /bin/sh -c id"))
self.refused(h, ["wg"], "W1")
def test_W2_values(self):
h = FakeHost()
h.stage(pa.WG_SRC, WG.replace("AllowedIPs = 10.77.0.1/32, 10.77.0.250/32", "AllowedIPs = 0.0.0.0/0"))
self.refused(h, ["wg"], "W2")
def test_S1_sshd_strictmodes(self): # an AuthorizedKeysFile the agent owns + StrictModes no = root login
h = FakeHost()
h.stage(pa.SSHD_SRC, pa.render_sshd(8822) + "StrictModes no\n")
self.refused(h, ["sshd-config"], "S1")
h2 = FakeHost()
h2.stage(pa.SSHD_SRC, pa.render_sshd(8822).replace("/etc/felhom-sshd/authorized_keys/%u", "/var/lib/felhom-agent/k"))
self.refused(h2, ["sshd-config"], "S1")
def test_S1_port_22(self):
h = FakeHost()
h.stage(pa.SSHD_SRC, pa.render_sshd(22))
self.refused(h, ["sshd-config"], "S1")
def test_S2_key_options_and_two_keys(self):
h = FakeHost()
h.stage(pa.KEY_SRC, 'command="/bin/sh" ' + KEYLINE)
self.refused(h, ["sshd-key"], "S2")
h2 = FakeHost()
h2.stage(pa.KEY_SRC, KEYLINE + KEYLINE)
self.refused(h2, ["sshd-key"], "S2")
def test_P1_symlink_owner_size(self):
h = FakeHost()
h.stage(pa.WG_SRC, WG, kind="symlink")
self.refused(h, ["wg"], "P1")
h2 = FakeHost()
h2.stage(pa.WG_SRC, WG, uid=0)
self.refused(h2, ["wg"], "P1")
h3 = FakeHost()
h3.stage(pa.WG_SRC, "#" * (pa.MAX_BYTES + 1))
self.refused(h3, ["wg"], "P1")
def test_A1_usage(self):
h = FakeHost()
self.refused(h, ["install", "/etc/shadow"], "A1")
self.refused(h, ["wg", "/etc/shadow"], "A1")
if __name__ == "__main__":
unittest.main(verbosity=2)
+63
View File
@@ -0,0 +1,63 @@
package backup
import (
"context"
"fmt"
"sync/atomic"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// R-874 (v0.145.0). THE MEASURED SHAPE (Part F spike, Tester 2): power-on sessions of ~1.5 h and ~5 min against a
// 6 h evaluation ticker that restarts at every start — no restore-test ever evaluated. Now the first evaluation runs
// FirstEval after start.
// COMPANION RED-PROOF: drop the first-evaluation timer in Run (back to the bare ticker) → "no evaluation within".
func TestR874_FirstEvaluationAfterStart(t *testing.T) {
var picks int32
s := NewScheduler(SchedulerOptions{
Runner: &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}},
Pick: func(context.Context) (string, error) {
atomic.AddInt32(&picks, 1)
return fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", atomic.LoadInt32(&picks)), nil
},
Store: NewStore(), Spec: (&specSpy{}).build,
Cadence: 6 * time.Hour, FirstEval: 30 * time.Millisecond, Logger: quiet(),
})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { _ = s.Run(ctx); close(done) }()
deadline := time.Now().Add(3 * time.Second)
for atomic.LoadInt32(&picks) == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
cancel()
<-done
if atomic.LoadInt32(&picks) == 0 {
t.Fatal("no evaluation within 3 s of start (FirstEval 30 ms) — a box with short sessions never gets a restore-test")
}
}
// The earned restraint stays: an agent that restarts before FirstEval never evaluates (a crash loop does not
// hammer a failing tier).
func TestR874_CrashLoopNeverEvaluates(t *testing.T) {
var picks int32
for i := 0; i < 5; i++ { // five quick "restarts"
s := NewScheduler(SchedulerOptions{
Runner: &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true}},
Pick: func(context.Context) (string, error) { atomic.AddInt32(&picks, 1); return "x", nil },
Store: NewStore(), Spec: (&specSpy{}).build,
Cadence: 6 * time.Hour, FirstEval: 200 * time.Millisecond, Logger: quiet(),
})
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
_ = s.Run(ctx)
cancel()
}
if n := atomic.LoadInt32(&picks); n != 0 {
t.Fatalf("a restart before FirstEval evaluated %d time(s)", n)
}
if DefaultFirstEval != 30*time.Minute {
t.Fatalf("DefaultFirstEval = %s, the documented 30 min", DefaultFirstEval)
}
}
+34 -6
View File
@@ -60,12 +60,21 @@ type Scheduler struct {
// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
// (single tier via `pick`), which keeps every existing caller and test working untouched.
tiers []string // configured tier target ids, primary first
tierPick TierPicker // newest archive on a named tier
rtState *RestoreTestState // persisted last-successful-per-tier (drives oldest-first)
inFlight *InFlight // shared with the backup path — Scenario F
tiers []string // configured tier target ids, primary first
tierPick TierPicker // newest archive on a named tier
rtState *RestoreTestState // persisted last-successful-per-tier (drives oldest-first)
inFlight *InFlight // shared with the backup path — Scenario F
firstEval time.Duration // R-874: the first evaluation after start
}
// DefaultFirstEval (R-874): the first due-ness evaluation runs 30 minutes after the agent starts, then every
// cadence. MEASURED need (2026-10-05 Part F spike): a box whose power-on sessions are all shorter than the 6 h
// interval (Tester 2: ~1.5 h and ~5 min) NEVER evaluated, because the ticker restarts at each start. 30 minutes
// keeps the earned restraint below — a crash-looping agent restarts far more often than that and still never
// evaluates — while a box that stays on for half an hour gets its due test. Pinned by
// TestR874_FirstEvaluationAfterStart and TestR874_CrashLoopNeverEvaluates.
const DefaultFirstEval = 30 * time.Minute
// SchedulerOptions configures a Scheduler.
type SchedulerOptions struct {
Runner RestoreTestRunner
@@ -81,6 +90,8 @@ type SchedulerOptions struct {
// 0 → no settle requirement (any archive is a candidate).
Settle time.Duration
Logger *slog.Logger
// FirstEval (R-874, v0.145.0) is when the FIRST evaluation runs after start; 0 → DefaultFirstEval.
FirstEval time.Duration
// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
@@ -110,6 +121,12 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
tierPick: opts.TierPick,
rtState: opts.State,
inFlight: opts.InFlight,
firstEval: func() time.Duration {
if opts.FirstEval > 0 {
return opts.FirstEval
}
return DefaultFirstEval
}(),
}
}
@@ -120,8 +137,9 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
// trigger any more: its phase is the process's uptime, and agent deploys reset it, which is exactly
// the defect R-86 removes. What decides that a test happens is `EvaluateDue`.
//
// It still does NOT evaluate immediately on start — the first evaluation is one interval in. That
// is an EARNED restraint, kept deliberately: a restore is heavy, agent restarts are routine, and a
// It still does NOT evaluate immediately on start. v0.145.0 (R-874): the first evaluation is
// firstEval (30 min) in, then every interval — it was one full interval in, which a box with short
// power-on sessions never reached. The restraint itself is EARNED and kept: a restore is heavy, agent restarts are routine, and a
// crash-loop that evaluated at start would hammer a permanently-failing tier as fast as it could
// restart. Due-ness does not expire while we wait, so the only cost is up to one interval of
// latency on a tier that just became due. On-demand runs use `--selftest=restore-test`.
@@ -135,6 +153,16 @@ func (s *Scheduler) Run(ctx context.Context) error {
}
s.logger.Info("backup: restore-test scheduler starting (per-archive due-check)",
"eval_interval", s.cadence, "settle", s.settle)
first := time.NewTimer(s.firstEval)
defer first.Stop()
select {
case <-ctx.Done():
s.logger.Info("backup: restore-test scheduler shutting down", "reason", ctx.Err())
return nil
case <-first.C:
s.logger.Info("backup: restore-test first evaluation after start (R-874)", "after", s.firstEval)
s.tick(ctx)
}
t := time.NewTicker(s.cadence)
defer t.Stop()
for {
+8 -12
View File
@@ -81,10 +81,8 @@ var manifest = []Capability{
{"drives-mkdir-sub", "per-drive stable dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives/felhom-usb"}, false, ""},
{"drives-mkdir-data", "felhom-data namespace create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-usb/felhom-data"}, false, ""},
{"drives-chown-data", "felhom-data guest-root chown", "/usr/bin/chown", []string{"100000:100000", "/mnt/felhom-usb/felhom-data"}, false, ""},
{"parent-script-install", "shared-parent boot script install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-shared-parent-123456789.sh", "/usr/local/sbin/felhom-shared-parent.sh"}, false, ""},
{"parent-unit-install", "shared-parent boot unit install", "/usr/bin/install", []string{"-m", "0644", "--", "/tmp/felhom-shared-parent-123456789.service", "/etc/systemd/system/felhom-shared-parent.service"}, false, ""},
{"parent-unit-enable", "shared-parent boot-persistence enable", "/usr/bin/systemctl", []string{"enable", "felhom-shared-parent.service"}, false, ""},
{"parent-bind-mp8", "parent bind into guest at provision", "/usr/sbin/pct", []string{"set", "9201", "-mp8", "/mnt/felhom-drives"}, false, ""},
{"parent-bind-mp8", "parent bind into guest at provision", "/usr/sbin/pct", []string{"set", "9201", "-mp8", "/mnt/felhom-drives,mp=/mnt/felhom-drives"}, false, ""},
// ---- Disk inspect / format gate (Critical: the data-bearing classifier + format) ----
{"disk-blkid", "disk data-bearing classify (format gate)", "/usr/sbin/blkid", []string{"-p", "-o", "export", "/dev/sda"}, true, ""},
@@ -95,11 +93,11 @@ var manifest = []Capability{
{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
// ---- Storage mount units (watchdog re-mount) ----
{"mount-unit-install", "fs-UUID mount unit install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0644", "--", "/var/lib/felhom-agent/units/felhom-x.mount", "/etc/systemd/system/felhom-x.mount"}, false, ""},
{"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "felhom-x.mount"}, false, ""},
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "felhom-x.mount"}, false, ""},
{"mount-unit-stop", "mount unit stop", "/usr/bin/systemctl", []string{"stop", "--", "felhom-x.mount"}, false, ""},
{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
{"mount-unit-stop", "mount unit stop", "/usr/bin/systemctl", []string{"stop", "--", "mnt-felhom\\x2dx.mount"}, false, ""},
// ---- Network storage re-arm + cleanup (CAMPAIGN-3 F10/F1) ----
{"netmount-reset-failed", "NAS automount re-arm after start-limit (F10)", "/usr/bin/systemctl", []string{"reset-failed", "--", "mnt-felhom\\x2ddrives-media.automount"}, false, ""},
@@ -110,18 +108,17 @@ var manifest = []Capability{
// ---- Provisioning back-half ----
{"provision-chown", "bootstrap mount guest-root chown", "/usr/bin/chown", []string{"-R", "100000:100000", "/var/lib/felhom-agent/guests/9201"}, false, ""},
{"provision-config-mount", "bootstrap config bind mount", "/usr/sbin/pct", []string{"set", "9201", "-mp0", "/var/lib/felhom-agent/guests/9201"}, false, ""},
{"provision-config-mount", "bootstrap config bind mount", "/usr/sbin/pct", []string{"set", "9201", "-mp0", "/var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1"}, false, ""},
{"provision-onboot", "customer guest autostart (onboot)", "/usr/sbin/pct", []string{"set", "9201", "-onboot", "1"}, false, ""},
// ---- Pre-start self-heal hook + guest lifecycle ----
{"guesthook-install", "pre-start hook snippet install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-guest-hook-123456789.sh", "/var/lib/vz/snippets/felhom-guest-hook.sh"}, false, ""},
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
// ---- LAN split-horizon resolver (dnsmasq) ----
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
{"dnsmasq-write", "dnsmasq drop-in write (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"dnsmasq", "/tmp/felhom-resolver-123456789.conf", "felhom-x.conf"}, false, ""},
{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
// ---- OS updates, guest fast lane (`11` §5.4.1; the wrapper holds every rule) ----
@@ -160,7 +157,7 @@ var manifest = []Capability{
// (one-time bootstrap) and disable (revocation, a deliberate teardown) stay non-critical. The
// handshake read is the ONLY wg invocation (never `dump`). ----
{"wg-tools-install", "wireguard-tools package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "wireguard-tools"}, false, ""},
{"wg-conf-install", "wg-felhom conf install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0600", "--", "/var/lib/felhom-agent/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"}, true, ""},
{"wg-conf-install", "wg-felhom conf install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"wg"}, true, ""},
{"wg-enable", "wg-quick@wg-felhom enable", "/usr/bin/systemctl", []string{"enable", "--now", "wg-quick@wg-felhom"}, true, ""},
{"wg-restart", "wg-quick@wg-felhom restart (conf change)", "/usr/bin/systemctl", []string{"restart", "wg-quick@wg-felhom"}, true, ""},
{"wg-disable", "wg-quick@wg-felhom disable (revocation)", "/usr/bin/systemctl", []string{"disable", "--now", "wg-quick@wg-felhom"}, false, ""},
@@ -192,7 +189,6 @@ var manifest = []Capability{
// operator-driven op, not a steady-state serving path — a degraded grant means "can't
// self-update" (fall back to a manual SSH deploy), not a serving outage. The apply repr uses a
// staging-dir path + a placeholder sha (list-mode never runs it). ----
{"selfupdate-apply", "agent self-update apply (A/B flip)", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"apply", "/var/lib/felhom-agent/selfupdate/felhom-agent-0.0.0", "0000000000000000000000000000000000000000000000000000000000000000"}, false, ""},
{"selfupdate-commit", "agent self-update commit", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"commit"}, false, ""},
{"selfupdate-rollback", "agent self-update rollback", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"rollback"}, false, ""},
}
+19 -3
View File
@@ -56,8 +56,10 @@ func parseSudoersEntries(t *testing.T, text string) []string {
var cur strings.Builder
for i := 0; i < len(raw); i++ {
c := raw[i]
if c == '\\' && i+1 < len(raw) {
cur.WriteByte(raw[i+1]) // unescape: keep the next char literally (\, → , ; \: → :)
if c == '\\' && i+1 < len(raw) && strings.IndexByte(",:=", raw[i+1]) >= 0 {
// unescape the sudoers grammar escapes only (\, → , ; \: → : ; \= → =). Every other backslash is kept: in
// a regex entry (R-861) `\.` `\{` `\\` mean exactly what sudo's regex engine reads.
cur.WriteByte(raw[i+1])
i++
continue
}
@@ -108,10 +110,24 @@ func globToRegex(pat string) *regexp.Regexp {
return regexp.MustCompile(b.String())
}
// entryRegex compiles one sudoers entry. R-861 (v0.146.0): an entry whose ARGUMENTS are a sudo regular expression
// (`^...$`, sudo >= 1.9.10) is matched as one — the binary literally, then a space, then the arguments joined by spaces
// (sudo's own rule). Every other entry is an fnmatch glob (globToRegex). The parser has already undone the sudoers
// escapes (`\,` `\:` `\=` `\\`), which leaves a valid RE2 pattern.
func entryRegex(e string) *regexp.Regexp {
if i := strings.IndexByte(e, ' '); i > 0 {
bin, args := e[:i], e[i+1:]
if strings.HasPrefix(args, "^") && strings.HasSuffix(args, "$") {
return regexp.MustCompile("^" + regexp.QuoteMeta(bin) + " " + args[1:])
}
}
return globToRegex(e)
}
// matchesAny reports whether cmdline matches at least one sudoers entry pattern.
func matchesAny(cmdline string, entries []string) bool {
for _, e := range entries {
if globToRegex(e).MatchString(cmdline) {
if entryRegex(e).MatchString(cmdline) {
return true
}
}
@@ -0,0 +1,65 @@
package capability
import (
"os"
"testing"
)
// R-861 (agent v0.146.0): the attacks the old globs let through, each as the exact argv a compromised agent would
// send. NONE may match any entry of the new sudoers. The same list ran against the REAL sudo 1.9.16 (a throwaway
// container, and `sudo -l -U felhom-agent` on both demo boxes after the bundle): audits/hub-safety-2026-10-05/partF/.
//
// RED-PROOF: run this list against the v0.145.0 sudoers (git show v0.145.0:configs/felhom-agent.sudoers) → most of
// these MATCH (recorded in the same evidence folder).
var r861Injections = []string{
// a raw host disk for a guest (pct options smuggled through a vmid glob)
"/usr/sbin/pct set 9201 --dev0 /dev/sda -onboot 1",
"/usr/sbin/pct set 9201 --dev0 /dev/sda -mp8 /mnt/felhom-drives",
"/usr/sbin/pct set 9201 --delete mp0 --dev0 /dev/sda",
"/usr/sbin/pct set 9201 -mp0 /var/lib/felhom-agent/guests/9201/bootstrap,mp=/x --dev0 /dev/sda",
// a bind mount over /etc through traversal
"/usr/bin/mount --bind /mnt/../var/lib/felhom-agent/x/felhom-data /mnt/felhom-drives/x",
"/usr/bin/mount --bind /mnt/a/felhom-data /mnt/felhom-drives/../../etc/sudoers.d",
"/usr/bin/umount /mnt/felhom-drives/x /",
"/usr/bin/chown 100000:100000 /mnt/a/felhom-data /etc/shadow",
"/usr/bin/mkdir -p /mnt/felhom-drives/x /etc/systemd/system/evil.mount",
// root-read files the agent writes: gone as `install` lines
"/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/x.mount /etc/systemd/system/etc-sudoers.d.mount",
"/usr/bin/install -m 0755 -- /tmp/felhom-guest-hook-1.sh /var/lib/vz/snippets/felhom-guest-hook.sh",
"/usr/bin/install -m 0755 -- /tmp/felhom-shared-parent-1.sh /usr/local/sbin/felhom-shared-parent.sh",
"/usr/bin/install -m 0644 /tmp/felhom-resolver-1.conf /etc/dnsmasq.d/felhom-x.conf",
"/usr/bin/install -o root -g root -m 0600 -- /var/lib/felhom-agent/wg/wg-felhom.conf /etc/wireguard/wg-felhom.conf",
"/usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/sshd_config /etc/felhom-sshd/sshd_config",
// the unsigned binary flip
"/usr/local/sbin/felhom-selfupdate-guarded apply /var/lib/felhom-agent/selfupdate/felhom-agent-9.9.9 0000000000000000000000000000000000000000000000000000000000000000",
// enabling or removing anything that is not ours
"/usr/bin/systemctl enable --now -- mnt-hdd_1.mount evil.service",
"/usr/bin/systemctl enable --now -- etc-sudoers.d.mount",
"/usr/bin/rm -f /etc/systemd/system/mnt-felhomx /etc/passwd",
"/usr/bin/rm -f /etc/dnsmasq.d/felhom-x.conf /etc/shadow",
"/usr/bin/rmdir /mnt/felhom-drives/x /etc",
// nftables commands chained after a set element
"/usr/sbin/nft add element inet felhom_oob operator_ips { 10.77.0.250 } ; flush ruleset",
// extra options to read-only tools
"/usr/sbin/smartctl -a -j /dev/sda -s off",
"/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data --config x",
"/usr/sbin/pct exec 9201 --keep-env -- docker inspect -f x felhom-controller",
"/usr/sbin/pct unlock 9201 --whatever",
// the checker with a path it must never take
"/usr/local/sbin/felhom-priv-apply unit ../../etc/x.mount",
"/usr/local/sbin/felhom-priv-apply dnsmasq /etc/shadow felhom-x.conf",
"/usr/local/sbin/felhom-priv-apply wg /etc/shadow",
}
func TestSudoersRefusesTheR861Injections(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatal(err)
}
entries := parseSudoersEntries(t, string(data))
for _, c := range r861Injections {
if matchesAny(c, entries) {
t.Errorf("the sudoers still allows: %s", c)
}
}
}
+28 -2
View File
@@ -7,9 +7,11 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"syscall"
)
// Slice 10D.1 — IDENTITY escrow. The K-escrow (above) wraps the PBS *encryption key* via the
@@ -71,7 +73,7 @@ func HashResticPassword(pw string) string {
// (non-empty) but NEVER logged by callers — log the field NAME only (mirrors AttachWGKey). A missing file
// is a clean no-attach (pre-fork-4 behavior, byte-compatible bundle).
func AttachResticPassword(b *IdentityBundle, stagePath string) (bool, error) {
raw, err := os.ReadFile(stagePath)
raw, err := readStagedNoFollow(stagePath)
if err != nil {
if os.IsNotExist(err) {
return false, nil
@@ -92,7 +94,7 @@ func AttachResticPassword(b *IdentityBundle, stagePath string) (bool, error) {
// no-attach (pre-S3 behavior, byte-compatible bundle); a corrupt one is an error (the operator
// should know their escrow would silently lack a live identity).
func AttachWGKey(b *IdentityBundle, keyPath string) (bool, error) {
raw, err := os.ReadFile(keyPath)
raw, err := readStagedNoFollow(keyPath)
if err != nil {
if os.IsNotExist(err) {
return false, nil
@@ -180,3 +182,27 @@ func UnwrapIdentityBundle(ctx context.Context, blob []byte, recoveryCode string)
}
return b, nil
}
// readStagedNoFollow reads a file the AGENT staged, for the escrow ceremony that runs as ROOT (FELHOM_ESCROW). R-861
// (agent v0.146.0): both files live in the agent's own directory, so a compromised agent could put a SYMLINK there
// (to any root-only file) and the root ceremony would seal that file into the blob and hand the agent R — a root file
// read. So: no symlink (O_NOFOLLOW), a regular file, at most 4 KiB. A missing file keeps its os.IsNotExist meaning.
// Pinned by TestAttach_RefusesASymlink.
func readStagedNoFollow(path string) ([]byte, error) {
f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NOFOLLOW, 0)
if err != nil {
return nil, err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("%s is not a regular file", path)
}
if fi.Size() > 4096 {
return nil, fmt.Errorf("%s is larger than 4 KiB", path)
}
return io.ReadAll(io.LimitReader(f, 4097))
}
+38
View File
@@ -0,0 +1,38 @@
package escrow
import (
"os"
"path/filepath"
"testing"
)
// R-861 (agent v0.146.0): the root escrow ceremony reads two files from the AGENT's directory. A symlink there to a
// root-only file must never be read (it would be sealed under R and handed to the agent).
// RED-PROOF (audits/hub-safety-2026-10-05/partF/red-proof.txt): use os.ReadFile in readStagedNoFollow → this fails.
func TestAttach_RefusesASymlink(t *testing.T) {
d := t.TempDir()
secret := filepath.Join(d, "root-only")
if err := os.WriteFile(secret, []byte("ROOT-ONLY-CANARY"), 0o600); err != nil {
t.Fatal(err)
}
link := filepath.Join(d, "restic_repo_password")
if err := os.Symlink(secret, link); err != nil {
t.Fatal(err)
}
var b IdentityBundle
if ok, err := AttachResticPassword(&b, link); err == nil || ok || b.ResticRepoPassword != "" {
t.Fatalf("a symlinked staged file was read: ok=%v err=%v value-set=%v", ok, err, b.ResticRepoPassword != "")
}
if ok, err := AttachWGKey(&b, link); err == nil || ok {
t.Fatalf("a symlinked wg key was read: ok=%v err=%v", ok, err)
}
// control: the real staged file is still read; a missing one is still a clean no-attach
real := filepath.Join(d, "real")
_ = os.WriteFile(real, []byte("pw\n"), 0o600)
if ok, err := AttachResticPassword(&b, real); err != nil || !ok || b.ResticRepoPassword != "pw" {
t.Fatalf("control: the plain staged file was not read: %v %v", ok, err)
}
if ok, err := AttachResticPassword(&b, filepath.Join(d, "absent")); err != nil || ok {
t.Fatalf("control: a missing file must stay a clean no-attach: %v %v", ok, err)
}
}
+2
View File
@@ -27,6 +27,8 @@ const (
PortFile = ConfDir + "/port"
// PidFile is the instance pidfile (NOT a RuntimeDirectory — that is the G1 incident cause).
PidFile = "/run/felhom-sshd.pid"
// PrivApply is the root content checker that installs the config and felhom-op's key (R-861, agent v0.146.0).
PrivApply = "/usr/local/sbin/felhom-priv-apply"
// Unit is the systemd unit name.
Unit = "felhom-sshd"
// OperatorUser is the default operator login (scoped sudo; key in AuthKeysDir only).
+5 -2
View File
@@ -143,7 +143,8 @@ func (m *Manager) Apply(ctx context.Context, block *hub.WireWireguard) (int, err
m.logger.Error("felhomsshd: staged config failed sshd -t — NOT installing", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return port, err
}
if _, errOut, err := m.runner.Run(ctx, "install", "-o", "root", "-g", "root", "-m", "0644", "--", m.stagedConfPath(), ConfPath); err != nil {
// R-861 (v0.146.0): the root checker installs it, and only if it is renderConfig's template for some port.
if _, errOut, err := m.runner.Run(ctx, PrivApply, "sshd-config"); err != nil {
m.logger.Error("felhomsshd: config install failed", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return port, err
}
@@ -181,7 +182,9 @@ func (m *Manager) applyAuthorizedKeys(ctx context.Context, sshKey string) {
m.logger.Error("felhomsshd: staging authorized_keys", "err", err)
return
}
if _, errOut, err := m.runner.Run(ctx, "install", "-o", "root", "-g", "root", "-m", "0644", "--", staged, AuthKeysUserPath); err != nil {
// R-861: the root checker installs it — one plain public key, no options (command=, from=, …), or empty.
_ = staged
if _, errOut, err := m.runner.Run(ctx, PrivApply, "sshd-key"); err != nil {
m.logger.Error("felhomsshd: authorized_keys install failed", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return
}
@@ -0,0 +1,26 @@
package felhomsshd
import (
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/privapplytest"
)
// R-861: renderConfig is byte-identical to the checker's template (only the Port varies); a changed line is refused.
// RED-PROOF: change one directive in renderConfig → this fails (the two templates drifted).
func TestPrivApply_AcceptsTheRenderedConfig(t *testing.T) {
for _, port := range []int{2222, 8822, 60022} {
conf, err := renderConfig(port)
if err != nil {
t.Fatal(err)
}
if got := privapplytest.Check(t, "sshd-config", "", conf); got != "OK" {
t.Errorf("port %d: %s", port, got)
}
}
conf, _ := renderConfig(8822)
if got := privapplytest.Check(t, "sshd-config", "", conf+"StrictModes no\n"); !strings.HasPrefix(got, "REFUSED") {
t.Fatalf("control: an extra directive was not refused: %s", got)
}
}
+16 -25
View File
@@ -38,35 +38,26 @@ const snippetBody = `#!/bin/sh
exit 0
`
// InstallSnippet writes the pre-start hook wrapper into the PVE snippets dir (idempotent, root-owned,
// executable). The agent runs as a non-root service user, so it writes an agent-writable temp file then
// `install`s it host-root (same pattern as the bootstrap mount + dnsmasq drop-ins). Safe to call repeatedly.
// The temp file is a RANDOM-named os.CreateTemp (audit B1): a fixed, predictable /tmp name could be
// pre-created by another local user and rewritten between our write and root's install (TOCTOU into a
// root-executed hookscript). The final mode comes from `install -m`, so the 0600 temp is fine.
func InstallSnippet(ctx context.Context, runner proxmox.Runner) error {
f, err := os.CreateTemp("", "felhom-guest-hook-*.sh")
// SnippetReady (R-861, agent v0.146.0) reports whether the pre-start hook is in place: a regular file at path whose
// content is exactly snippetBody. The hook is a FIXED, ROOT-OWNED file that arrives with the signed config bundle
// (configs/felhom-guest-hook.sh, pinned byte-identical by TestSnippetEqualsTheBundle). The agent no longer installs it:
// until v0.146.0 it `install`ed it from /tmp, and Proxmox runs a hookscript as root at every guest start — so the
// install grant was a root shell for a compromised agent. A missing or different hook is an error the caller logs; it
// then does NOT register the hook (a guest whose hookscript is missing does not start).
func SnippetReady(path string) error {
fi, err := os.Lstat(path)
if err != nil {
return fmt.Errorf("guesthook: create temp snippet: %w", err)
return fmt.Errorf("guesthook: %s is missing — it arrives with the signed config bundle (agent_config_update): %w", path, err)
}
tmp := f.Name()
defer os.Remove(tmp)
if _, err := f.WriteString(snippetBody); err != nil {
f.Close()
return fmt.Errorf("guesthook: write temp snippet: %w", err)
if !fi.Mode().IsRegular() {
return fmt.Errorf("guesthook: %s is not a regular file", path)
}
if err := f.Close(); err != nil {
return fmt.Errorf("guesthook: close temp snippet: %w", err)
b, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("guesthook: read %s: %w", path, err)
}
// Ensure the snippets dir exists FIRST (B2, DRILL-day0-cleanroom-2026-07-03): a fresh PVE has
// no /var/lib/vz/snippets, and `install` (without -D) won't create the parent — the whole
// hook install silently failed on a freshly-bootstrapped box. Fenced root op like the install
// itself; idempotent.
if _, stderr, err := runner.Run(ctx, "mkdir", "-p", SnippetDir); err != nil {
return fmt.Errorf("guesthook: ensure snippets dir %s: %w: %s", SnippetDir, err, string(stderr))
}
if _, stderr, err := runner.Run(ctx, "install", "-m", "0755", "--", tmp, SnippetPath); err != nil {
return fmt.Errorf("guesthook: install snippet to %s: %w: %s", SnippetPath, err, string(stderr))
if string(b) != snippetBody {
return fmt.Errorf("guesthook: %s differs from this agent's hook — the next config bundle replaces it", path)
}
return nil
}
+24 -90
View File
@@ -4,7 +4,7 @@ import (
"context"
"io"
"os"
"regexp"
"path/filepath"
"testing"
)
@@ -29,100 +29,34 @@ func (r *recordingRunner) RunStdin(ctx context.Context, _ io.Reader, name string
return r.Run(ctx, name, args...)
}
// TestInstallSnippet_RandomTempName is the audit-B1 negative test: the staged install SOURCE must be a
// RANDOM os.CreateTemp name (felhom-guest-hook-<random>.sh), never the fixed, pre-creatable
// /tmp/felhom-guest-hook.sh (a local TOCTOU into a root-executed hookscript), and two consecutive
// installs must stage through DIFFERENT paths.
func TestInstallSnippet_RandomTempName(t *testing.T) {
r := &recordingRunner{}
if err := InstallSnippet(context.Background(), r); err != nil {
t.Fatalf("InstallSnippet #1: %v", err)
// R-861 (agent v0.146.0): the hook file comes with the signed config bundle; the agent never installs it, only checks.
// RED-PROOF (audits/hub-safety-2026-10-05/partF/red-proof.txt): make SnippetReady accept any content → the
// "differs" case fails.
func TestSnippetReady(t *testing.T) {
d := t.TempDir()
p := filepath.Join(d, "felhom-guest-hook.sh")
if err := SnippetReady(p); err == nil {
t.Fatal("a missing hook read as ready — the guest would get a hookscript that does not exist")
}
if err := InstallSnippet(context.Background(), r); err != nil {
t.Fatalf("InstallSnippet #2: %v", err)
_ = os.WriteFile(p, []byte("#!/bin/sh\nid > /tmp/x\n"), 0o755)
if err := SnippetReady(p); err == nil {
t.Fatal("a hook with other content read as ready")
}
var installs [][]string
for _, call := range r.calls {
if call[0] == "install" {
installs = append(installs, call)
}
_ = os.WriteFile(p, []byte(snippetBody), 0o755)
if err := SnippetReady(p); err != nil {
t.Fatalf("the bundle's hook was not accepted: %v", err)
}
if len(installs) != 2 {
t.Fatalf("expected 2 install calls, got %d: %v", len(installs), r.calls)
}
randomName := regexp.MustCompile(`felhom-guest-hook-[^/\\]+\.sh$`)
fixedName := regexp.MustCompile(`felhom-guest-hook\.sh$`)
var srcs []string
for i, call := range installs {
// install -m 0755 -- <src> <dest>
if len(call) != 6 {
t.Fatalf("call %d: unexpected vector %v", i, call)
}
src, dest := call[4], call[5]
if dest != SnippetPath {
t.Errorf("call %d: dest = %q, want %q", i, dest, SnippetPath)
}
if !randomName.MatchString(src) {
t.Errorf("call %d: source %q does not match the random felhom-guest-hook-*.sh pattern", i, src)
}
if fixedName.MatchString(src) {
t.Errorf("call %d: source %q is the FIXED predictable temp name (B1 TOCTOU)", i, src)
}
srcs = append(srcs, src)
}
if srcs[0] == srcs[1] {
t.Errorf("two consecutive installs staged through the SAME source path %q — must be random per call", srcs[0])
}
// Non-hollow: the staged file must actually carry the snippet body at install time.
for i, c := range r.srcContent {
if c != snippetBody {
t.Errorf("call %d: staged content is not the snippet body (got %d bytes)", i, len(c))
}
}
// And the temp is cleaned up after.
for _, src := range srcs {
if _, err := os.Stat(src); err == nil {
t.Errorf("staged temp %q left behind (defer os.Remove missing)", src)
}
l := filepath.Join(d, "link.sh")
_ = os.Symlink(p, l)
if err := SnippetReady(l); err == nil {
t.Fatal("a symlink read as the hook")
}
}
// B2 Scenario D (DRILL-day0-cleanroom-2026-07-03): on a fresh PVE, /var/lib/vz/snippets does not
// exist and `install` (no -D) cannot create it — the drill saw
// `install: cannot create regular file … No such file or directory` and the guest silently got no
// pre-start self-heal hook. InstallSnippet must therefore issue a `mkdir -p <SnippetDir>` fenced op
// BEFORE the `install` op. Pre-fix wrong outcome: no mkdir call at all — only the doomed install.
func TestInstallSnippet_EnsuresSnippetsDirFirst(t *testing.T) {
r := &recordingRunner{}
if err := InstallSnippet(context.Background(), r); err != nil {
t.Fatalf("InstallSnippet: %v", err)
}
mkdirIdx, installIdx := -1, -1
for i, call := range r.calls {
switch call[0] {
case "mkdir":
if mkdirIdx == -1 {
mkdirIdx = i
want := []string{"mkdir", "-p", SnippetDir}
if len(call) != 3 || call[1] != want[1] || call[2] != want[2] {
t.Errorf("mkdir vector = %v, want %v (the sudoers fence matches exactly this argv)", call, want)
}
}
case "install":
if installIdx == -1 {
installIdx = i
}
}
}
if mkdirIdx == -1 {
t.Fatalf("no `mkdir -p %s` op issued — on a fresh box the snippet install fails ENOENT (B2); calls: %v", SnippetDir, r.calls)
}
if installIdx == -1 {
t.Fatalf("no install op issued; calls: %v", r.calls)
}
if mkdirIdx > installIdx {
t.Fatalf("mkdir (call %d) must PRECEDE install (call %d) — order: %v", mkdirIdx, installIdx, r.calls)
// The bundle's copy is byte-identical to the body the agent checks against.
func TestSnippetEqualsTheBundle(t *testing.T) {
got, err := os.ReadFile(filepath.Join("..", "..", "configs", "felhom-guest-hook.sh"))
if err != nil || string(got) != snippetBody {
t.Fatalf("configs/felhom-guest-hook.sh differs from snippetBody (err %v)", err)
}
}
+25
View File
@@ -0,0 +1,25 @@
package hub
import (
"os"
"path/filepath"
"testing"
)
// R-840: the agent reports the bundle record itself — "none" when no bundle ever reached the box (so an old wrapper
// cannot hide that), "unknown" when it cannot be read, else the version and sha the root wrapper recorded.
func TestReadBundleRecord(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "config-bundle.json")
if got := string(readBundleRecord(p)); got != `{"version":"none"}` {
t.Fatalf("absent: %s", got)
}
os.WriteFile(p, []byte("{broken"), 0o644)
if got := string(readBundleRecord(p)); got != `{"version":"unknown"}` {
t.Fatalf("broken: %s", got)
}
os.WriteFile(p, []byte(`{"format":1,"agent_version":"0.143.0","bundle_sha256":"abc","installed_at":"2026-10-04T20:00:00Z","authority":"signed","files":{"/x":"y"}}`), 0o644)
if got := string(readBundleRecord(p)); got != `{"authority":"signed","bundle_sha256":"abc","installed_at":"2026-10-04T20:00:00Z","version":"0.143.0"}` {
t.Fatalf("record: %s", got)
}
}
+53 -24
View File
@@ -92,30 +92,31 @@ type GuestNetReporter interface {
// Collector builds a HostReport from read-only sources. All deps are behind narrow
// interfaces for unit testing.
type Collector struct {
px proxmoxReader
cf CloudflaredProber
storage StorageObserver
backups BackupReporter
restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter
pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
system SystemReporter // R-852: the box versions (nil → API fields only)
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
hostID string
agentVersion string
logger *slog.Logger
now func() time.Time
px proxmoxReader
cf CloudflaredProber
storage StorageObserver
backups BackupReporter
restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter
pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
system SystemReporter // R-852: the box versions (nil → API fields only)
bundleRecordPath string // R-840: test seam; "" = BundleRecordPath
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
hostID string
agentVersion string
logger *slog.Logger
now func() time.Time
}
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
@@ -262,6 +263,33 @@ func (c *Collector) SetSystemReporter(r SystemReporter) *Collector {
return c
}
// BundleRecordPath is the root-owned record felhom-os-apply writes after a config bundle installs (R-840).
const BundleRecordPath = "/etc/felhom/config-bundle.json"
// readBundleRecord returns the record's summary: version/sha/installed_at, "none" when the file is absent, "unknown"
// when it cannot be read or parsed (never a guess).
func readBundleRecord(path string) json.RawMessage {
if path == "" {
path = BundleRecordPath
}
b, err := os.ReadFile(path)
if os.IsNotExist(err) {
return json.RawMessage(`{"version":"none"}`)
}
var rec struct {
AgentVersion string `json:"agent_version"`
BundleSHA256 string `json:"bundle_sha256"`
InstalledAt string `json:"installed_at"`
Authority string `json:"authority"`
}
if err != nil || json.Unmarshal(b, &rec) != nil || rec.AgentVersion == "" {
return json.RawMessage(`{"version":"unknown"}`)
}
out, _ := json.Marshal(map[string]string{"version": rec.AgentVersion, "bundle_sha256": rec.BundleSHA256,
"installed_at": rec.InstalledAt, "authority": rec.Authority})
return out
}
func unknownIfEmpty(s string) string {
if strings.TrimSpace(s) == "" {
return "unknown"
@@ -273,6 +301,7 @@ func unknownIfEmpty(s string) string {
func (c *Collector) systemInfo(ctx context.Context, ns proxmox.NodeStatus) *SystemInfo {
si := &SystemInfo{PVEVersion: unknownIfEmpty(ns.PVEVersion), KernelVersion: unknownIfEmpty(ns.KVersion),
ReadAt: c.now().Format(time.RFC3339)}
si.ConfigBundle = readBundleRecord(c.bundleRecordPath)
if c.system == nil {
si.FactsError = "no facts reader wired"
return si
+4
View File
@@ -255,6 +255,10 @@ type SystemInfo struct {
Facts json.RawMessage `json:"facts,omitempty"`
FactsError string `json:"facts_error,omitempty"`
ReadAt string `json:"read_at"`
// ConfigBundle is the box's root-owned config bundle record (R-840, agent v0.143.0), read by the agent itself from
// /etc/felhom/config-bundle.json (0644): {"version":"none"} on a box no bundle reached, so an OLD wrapper cannot
// hide it. The wrapper's facts carry the same record plus the drift (files changed by hand).
ConfigBundle json.RawMessage `json:"config_bundle,omitempty"`
}
// HostMetrics is the host block, sourced from proxmox NodeStatus.
+9 -1
View File
@@ -33,6 +33,8 @@ const (
DropinDir = "/etc/dnsmasq.d"
// BaseDropin holds the host-wide listen/upstream config (one per host).
BaseDropin = "felhom-resolver-base.conf"
// PrivApply is the root content checker that installs a drop-in (R-861).
PrivApply = "/usr/local/sbin/felhom-priv-apply"
)
// RenderBase returns the host-wide dnsmasq drop-in: bind to the host LAN IP (+ loopback), no-resolv,
@@ -263,7 +265,13 @@ func (m *Manager) writeFileIfChanged(ctx context.Context, path, content, mode st
return false, fmt.Errorf("write temp: %w", err)
}
tmp.Close()
if _, errOut, err := m.runner.Run(ctx, "install", "-m", mode, tmpName, path); err != nil {
// R-861 (v0.146.0): a dnsmasq drop-in reaches /etc/dnsmasq.d only through the root checker, which allows exactly
// the lines RenderBase/RenderGuestDropin write (a `dhcp-script=` would run as root). Mode is fixed (0644) there.
_ = mode
if filepath.Dir(path) != DropinDir {
return false, fmt.Errorf("drop-in %s is not in %s", path, DropinDir)
}
if _, errOut, err := m.runner.Run(ctx, PrivApply, "dnsmasq", tmpName, filepath.Base(path)); err != nil {
return false, fmt.Errorf("install %s: %s: %w", path, strings.TrimSpace(string(errOut)), err)
}
return true, nil
@@ -0,0 +1,22 @@
package lanresolver
import (
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/privapplytest"
)
// R-861: both drop-ins the resolver renders are accepted by the root checker; a dhcp-script line is not.
func TestPrivApply_AcceptsTheRenderedDropins(t *testing.T) {
if got := privapplytest.Check(t, "dnsmasq", BaseDropin, RenderBase("192.168.0.104", []string{"1.1.1.1", "8.8.8.8"})); got != "OK" {
t.Errorf("base drop-in: %s", got)
}
if got := privapplytest.Check(t, "dnsmasq", DropinName("demo-hp"), RenderGuestDropin("demo-hp", "enkisfelhom.hu", "192.168.0.138")); got != "OK" {
t.Errorf("guest drop-in: %s", got)
}
evil := RenderGuestDropin("demo-hp", "enkisfelhom.hu", "192.168.0.138") + "dhcp-script=/var/lib/felhom-agent/x\n"
if got := privapplytest.Check(t, "dnsmasq", "felhom-x.conf", evil); !strings.HasPrefix(got, "REFUSED") {
t.Fatalf("control: dhcp-script was not refused: %s", got)
}
}
+20 -50
View File
@@ -121,32 +121,33 @@ func (b *GuestBinder) EnsureSharedParent(ctx context.Context) error {
// only the script (the unit was unchanged), so the earlier unit-only gate never redeployed it —
// leaving hosts running the pre-fix script (no make-private), whose self-bind stays in root's shared
// peer group and DOUBLES every drive bind. Comparing both files closes that deploy gap.
if sharedParentInstallStale(sharedParentUnitPath, sharedParentScriptPath) {
if ierr := b.installSharedParentUnit(ctx); ierr != nil {
b.logger.Warn("shared-parent: boot-persistence (re)install failed (live setup OK; survives until host reboot)", "err", ierr)
}
if err := b.ensureSharedParentBoot(ctx, sharedParentUnitPath, sharedParentScriptPath, sharedParentWantsLink); err != nil {
b.logger.Warn("shared-parent: boot persistence not enabled (live setup OK; survives until host reboot)", "err", err)
}
return nil
}
// stageTemp writes content to a fresh random-named temp file (os.CreateTemp pattern — `*` is replaced
// by a random string) and returns its path. Caller removes it after the privileged `install`.
func stageTemp(pattern, content string) (string, error) {
f, err := os.CreateTemp("", pattern)
if err != nil {
return "", err
// sharedParentWantsLink exists once `systemctl enable felhom-shared-parent.service` ran (WantedBy=pve-guests.service).
const sharedParentWantsLink = "/etc/systemd/system/pve-guests.service.wants/felhom-shared-parent.service"
// ensureSharedParentBoot (R-861, agent v0.146.0) — the boot script and its unit are ROOT-OWNED FIXED files that arrive
// with the signed config bundle (configs/felhom-shared-parent.{sh,service}, byte-identical to the constants above,
// pinned by TestSharedParentFilesEqualTheBundle). The agent NEVER installs them any more: until v0.146.0 it installed
// them from /tmp, and a script a root unit runs at boot was a root shell for a compromised agent. Here it only checks
// them, and enables the unit (a fixed sudoers line) when the bundle has put it in place but nothing has enabled it — a
// fresh install. Missing or different files: one warning, nothing run (the next bundle brings them).
func (b *GuestBinder) ensureSharedParentBoot(ctx context.Context, unitPath, scriptPath, wantsLink string) error {
if sharedParentInstallStale(unitPath, scriptPath) {
return fmt.Errorf("%s or %s is missing or differs from this agent's — it arrives with the signed config bundle (agent_config_update); the agent does not install it (R-861)", scriptPath, unitPath)
}
name := f.Name()
if _, err := f.WriteString(content); err != nil {
f.Close()
os.Remove(name)
return "", err
if _, err := os.Lstat(wantsLink); err == nil {
return nil
}
if err := f.Close(); err != nil {
os.Remove(name)
return "", err
if err := b.run(ctx, "systemctl", "enable", "felhom-shared-parent.service"); err != nil {
return fmt.Errorf("enable unit: %w", err)
}
return name, nil
b.logger.Info("shared-parent: boot-persistence unit enabled (files from the config bundle)")
return nil
}
// sharedParentInstallStale reports whether the on-disk boot script OR unit is missing or differs from
@@ -162,37 +163,6 @@ func sharedParentInstallStale(unitPath, scriptPath string) bool {
return false
}
// installSharedParentUnit writes the script + unit (from agent-written temps) and enables the unit so the
// shared parent is re-established on every host boot before pve-guests. Idempotent. The temps are
// RANDOM-named os.CreateTemp files (audit B1): a fixed, predictable /tmp name could be pre-created by
// another local user and rewritten between our write and root's install (TOCTOU into a root-executed
// boot script). The final modes come from `install -m`, so the 0600 temps are fine.
func (b *GuestBinder) installSharedParentUnit(ctx context.Context) error {
tmpScript, err := stageTemp("felhom-shared-parent-*.sh", sharedParentScript)
if err != nil {
return fmt.Errorf("write temp script: %w", err)
}
defer os.Remove(tmpScript)
if err := b.run(ctx, "install", "-m", "0755", "--", tmpScript, sharedParentScriptPath); err != nil {
return fmt.Errorf("install script: %w", err)
}
tmpUnit, err := stageTemp("felhom-shared-parent-*.service", sharedParentUnit)
if err != nil {
return fmt.Errorf("write temp unit: %w", err)
}
defer os.Remove(tmpUnit)
if err := b.run(ctx, "install", "-m", "0644", "--", tmpUnit, sharedParentUnitPath); err != nil {
return fmt.Errorf("install unit: %w", err)
}
if err := b.run(ctx, "systemctl", "daemon-reload"); err != nil {
return fmt.Errorf("daemon-reload: %w", err)
}
if err := b.run(ctx, "systemctl", "enable", "felhom-shared-parent.service"); err != nil {
return fmt.Errorf("enable unit: %w", err)
}
return nil
}
// AttachDrive binds a drive's felhom-data namespace under the stable parent so it appears live in the
// guest at the returned stable path (via propagation — no pct, no reboot). `where` is the drive's RAW
// host PVE mount (/mnt/<name>); only `<where>/felhom-data` crosses into the guest (confinement). The
+58 -70
View File
@@ -4,94 +4,82 @@ import (
"context"
"io"
"os"
"regexp"
"path/filepath"
"testing"
)
// stagingRecorderRunner is a fake proxmox.Runner recording every call vector and snapshotting each
// install SOURCE file's content at call time (the deferred os.Remove erases it afterwards).
type stagingRecorderRunner struct {
calls [][]string
srcContent map[string]string // install dest → staged source content
}
// R-861 (agent v0.146.0): the shared-parent boot script and unit arrive with the signed config bundle; the agent never
// installs them. It checks them and enables the unit when nothing has.
//
// RED-PROOF (audits/hub-safety-2026-10-05/partF/red-proof.txt): put the old installSharedParentUnit call back (an
// `install` of a /tmp file) → TestSharedParentBoot_NeverInstalls fails.
func (r *stagingRecorderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
type callRecorder struct{ calls [][]string }
func (r *callRecorder) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
if name == "install" && len(args) >= 2 {
src, dest := args[len(args)-2], args[len(args)-1]
if r.srcContent == nil {
r.srcContent = map[string]string{}
}
b, _ := os.ReadFile(src)
r.srcContent[dest] = string(b)
}
return nil, nil, nil
}
func (r *stagingRecorderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
func (r *callRecorder) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
// installSources returns the install-call source paths keyed by destination.
func (r *stagingRecorderRunner) installSources() map[string][]string {
out := map[string][]string{}
for _, c := range r.calls {
if c[0] == "install" && len(c) >= 3 {
src, dest := c[len(c)-2], c[len(c)-1]
out[dest] = append(out[dest], src)
}
func bootFiles(t *testing.T, script, unit string) (string, string, string) {
t.Helper()
d := t.TempDir()
sp, up := filepath.Join(d, "felhom-shared-parent.sh"), filepath.Join(d, "felhom-shared-parent.service")
if script != "" {
_ = os.WriteFile(sp, []byte(script), 0o755)
}
return out
if unit != "" {
_ = os.WriteFile(up, []byte(unit), 0o644)
}
return sp, up, filepath.Join(d, "wants-link")
}
// TestInstallSharedParent_RandomTempName is the audit-B1 negative test for the shared-parent boot
// persistence install: both staged install SOURCES (script + unit) must be RANDOM os.CreateTemp names
// (felhom-shared-parent-<random>.sh / .service), never the fixed, pre-creatable /tmp names (a local
// TOCTOU into a root-executed boot script), and two consecutive installs must use DIFFERENT paths.
func TestInstallSharedParent_RandomTempName(t *testing.T) {
r := &stagingRecorderRunner{}
func TestSharedParentBoot_NeverInstalls(t *testing.T) {
for _, c := range []struct{ name, script, unit string }{
{"both missing", "", ""},
{"script differs", "#!/bin/sh\necho old\n", sharedParentUnit},
{"unit missing", sharedParentScript, ""},
} {
r := &callRecorder{}
b := NewGuestBinder(r, nil)
sp, up, link := bootFiles(t, c.script, c.unit)
if err := b.ensureSharedParentBoot(context.Background(), up, sp, link); err == nil {
t.Errorf("%s: no error — the operator would not learn the bundle is missing", c.name)
}
if len(r.calls) != 0 {
t.Errorf("%s: the agent ran %v — it must install nothing (R-861)", c.name, r.calls)
}
}
}
func TestSharedParentBoot_EnablesOnceTheBundleBroughtTheFiles(t *testing.T) {
r := &callRecorder{}
b := NewGuestBinder(r, nil)
if err := b.installSharedParentUnit(context.Background()); err != nil {
t.Fatalf("installSharedParentUnit #1: %v", err)
sp, up, link := bootFiles(t, sharedParentScript, sharedParentUnit)
if err := b.ensureSharedParentBoot(context.Background(), up, sp, link); err != nil {
t.Fatal(err)
}
if err := b.installSharedParentUnit(context.Background()); err != nil {
t.Fatalf("installSharedParentUnit #2: %v", err)
if len(r.calls) != 1 || len(r.calls[0]) != 3 || r.calls[0][0] != "systemctl" || r.calls[0][1] != "enable" ||
r.calls[0][2] != "felhom-shared-parent.service" {
t.Fatalf("want exactly `systemctl enable felhom-shared-parent.service`, got %v", r.calls)
}
_ = os.Symlink(up, link)
r.calls = nil
if err := b.ensureSharedParentBoot(context.Background(), up, sp, link); err != nil || len(r.calls) != 0 {
t.Fatalf("already enabled: want no calls, got %v (%v)", r.calls, err)
}
}
srcs := r.installSources()
cases := []struct {
dest string
random *regexp.Regexp
fixed *regexp.Regexp
content string
}{
{sharedParentScriptPath, regexp.MustCompile(`felhom-shared-parent-[^/\\]+\.sh$`),
regexp.MustCompile(`felhom-shared-parent\.sh$`), sharedParentScript},
{sharedParentUnitPath, regexp.MustCompile(`felhom-shared-parent-[^/\\]+\.service$`),
regexp.MustCompile(`felhom-shared-parent\.service$`), sharedParentUnit},
}
for _, tc := range cases {
got := srcs[tc.dest]
if len(got) != 2 {
t.Fatalf("dest %s: expected 2 install calls, got %d (%v)", tc.dest, len(got), got)
}
for i, src := range got {
if !tc.random.MatchString(src) {
t.Errorf("dest %s call %d: source %q does not match the random temp pattern", tc.dest, i, src)
}
if tc.fixed.MatchString(src) {
t.Errorf("dest %s call %d: source %q is the FIXED predictable temp name (B1 TOCTOU)", tc.dest, i, src)
}
if _, err := os.Stat(src); err == nil {
t.Errorf("dest %s: staged temp %q left behind (defer os.Remove missing)", tc.dest, src)
}
}
if got[0] == got[1] {
t.Errorf("dest %s: two consecutive installs staged through the SAME source %q — must be random per call", tc.dest, got[0])
}
// Non-hollow: the staged file carried the real content at install time.
if r.srcContent[tc.dest] != tc.content {
t.Errorf("dest %s: staged content mismatch (got %d bytes, want %d)", tc.dest, len(r.srcContent[tc.dest]), len(tc.content))
// The bundle's copies are byte-identical to the constants the agent compares against.
func TestSharedParentFilesEqualTheBundle(t *testing.T) {
for file, want := range map[string]string{"felhom-shared-parent.sh": sharedParentScript, "felhom-shared-parent.service": sharedParentUnit} {
got, err := os.ReadFile(filepath.Join("..", "..", "configs", file))
if err != nil || string(got) != want {
t.Errorf("configs/%s differs from the agent's constant (err %v)", file, err)
}
}
}
+157
View File
@@ -0,0 +1,157 @@
package osupdate
import (
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"regexp"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// OpConfigUpdate is the signed op that brings a box's ROOT-OWNED files (sudoers, wrappers, units) to a release's config
// bundle (R-840, `11` §5.4.2). The params pin the agent version and the bundle's sha256; the root wrapper re-verifies
// the signature, the host binding, the nonce and the sha ITSELF — this executor is only the courier.
const OpConfigUpdate = "agent_config_update"
// BundleFileName is the bundle's name beside the binary in the Gitea generic package felhom-agent/<version>/.
const BundleFileName = "felhom-config-bundle.json"
var (
bundleVersionRe = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$`)
bundleSHARe = regexp.MustCompile(`^[0-9a-f]{64}$`)
)
// ConfigUpdateParams are the signed params (the wrapper reads the same two names out of the signed blob).
type ConfigUpdateParams struct {
AgentVersion string `json:"agent_version"`
BundleSHA256 string `json:"bundle_sha256"`
}
// BundleURL derives the bundle's URL from the agent binary's URL template (".../felhom-agent/{version}/felhom-agent").
func BundleURL(binaryTemplate, version string) (string, error) {
if !strings.HasSuffix(binaryTemplate, "/felhom-agent") || !strings.Contains(binaryTemplate, "{version}") {
return "", fmt.Errorf("cannot derive the bundle URL from %q (want …/{version}/felhom-agent)", binaryTemplate)
}
t := strings.TrimSuffix(binaryTemplate, "felhom-agent") + BundleFileName
return strings.ReplaceAll(t, "{version}", version), nil
}
// ConfigUpdateExecutor runs a verified agent_config_update (signedjobs.Executor).
type ConfigUpdateExecutor struct {
Leg *Leg
URLTemplate string // the agent binary's template (config.SelfUpdate.URLTemplate)
Username string
Token string
HTTPClient *http.Client
// AfterInstall runs after a bundle installed (the capability re-probe; nil = none).
AfterInstall func(ctx context.Context)
}
// Execute implements signedjobs.Executor.
func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
if op != OpConfigUpdate {
return signedjobs.ErrNoExecutor
}
so, ok := signedjobs.SignedOpFrom(ctx)
if !ok {
return fmt.Errorf("agent_config_update: no signed envelope in the context — the wrapper could not verify it")
}
var p ConfigUpdateParams
if err := json.Unmarshal(params, &p); err != nil {
return fmt.Errorf("agent_config_update: bad params: %w", err)
}
if !bundleVersionRe.MatchString(p.AgentVersion) || !bundleSHARe.MatchString(p.BundleSHA256) {
return fmt.Errorf("agent_config_update: params must pin agent_version (semver) and bundle_sha256 (64 hex)")
}
lg := e.Leg.log().With("op", OpConfigUpdate, "agent_version", p.AgentVersion)
url, err := BundleURL(e.URLTemplate, p.AgentVersion)
if err != nil {
return fmt.Errorf("agent_config_update: %w", err)
}
dir := e.Leg.PlanDir
if dir == "" {
dir = DefaultPlanDir
}
if err := os.MkdirAll(dir, 0o700); err != nil {
return fmt.Errorf("agent_config_update: plan dir: %w", err)
}
path := filepath.Join(dir, "bundle-"+p.AgentVersion+".json")
start := time.Now()
got, err := e.download(ctx, url, path)
if err != nil {
_ = os.Remove(path)
return fmt.Errorf("agent_config_update: download %s: %w", url, err)
}
defer os.Remove(path)
// The agent's own check is a courtesy (an early, clear error); the wrapper's is the gate.
if got != p.BundleSHA256 {
return fmt.Errorf("agent_config_update: the downloaded bundle's sha256 is %s, the signed job pins %s — nothing installed", got, p.BundleSHA256)
}
lg.Info("osupdate: config bundle downloaded; handing it to the root wrapper", "sha256", got[:16], "duration_ms", time.Since(start).Milliseconds())
runID := "bundle-" + e.Leg.now().UTC().Format("20060102T150405Z")
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-" + p.AgentVersion, "layer": LayerHost,
"mode": "bundle", "bundle": path,
"signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(so.Blob), "sig": string(so.Sig)}})
if err != nil {
return fmt.Errorf("agent_config_update: %w", err)
}
if wr.refused() {
lg.Warn("osupdate: config bundle REFUSED by the wrapper — nothing changed", "refused", string(wr.Refused))
return fmt.Errorf("agent_config_update: refused: %s", wr.Refused)
}
if wr.failed() {
lg.Error("osupdate: config bundle FAILED — the wrapper put the previous files back", "failed", string(wr.Failed), "bundle", string(wr.Bundle))
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
}
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
if e.AfterInstall != nil {
e.AfterInstall(ctx)
}
return nil
}
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
hc := e.HTTPClient
if hc == nil {
hc = &http.Client{Timeout: 2 * time.Minute}
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return "", err
}
if e.Username != "" || e.Token != "" {
req.SetBasicAuth(e.Username, e.Token)
}
resp, err := hc.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", fmt.Errorf("HTTP %d", resp.StatusCode)
}
f, err := os.OpenFile(dest, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
if err != nil {
return "", err
}
h := sha256.New()
// 4 MB is the wrapper's own limit; read one byte more so an oversized bundle fails there, visibly.
if _, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 4*1024*1024+1)); err != nil {
f.Close()
return "", err
}
if err := f.Close(); err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
+155
View File
@@ -0,0 +1,155 @@
package osupdate
import (
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"io"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// bundleWrapper plays felhom-os-apply for mode "bundle": it records the plan and the bundle file's bytes AT CALL TIME
// (the executor deletes the file afterwards), and answers with rep.
type bundleWrapper struct {
t *testing.T
rep string
plans []map[string]any
bodies [][]byte
}
func (b *bundleWrapper) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
if name != WrapperPath || len(args) != 2 || args[0] != "--plan" {
b.t.Fatalf("unexpected command %s %v", name, args)
}
raw, err := os.ReadFile(args[1])
if err != nil {
b.t.Fatal(err)
}
var plan map[string]any
_ = json.Unmarshal(raw, &plan)
b.plans = append(b.plans, plan)
body, _ := os.ReadFile(plan["bundle"].(string))
b.bodies = append(b.bodies, body)
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
}
func serveBundle(t *testing.T, body []byte) (*httptest.Server, string) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/felhom-agent/0.143.0/"+BundleFileName {
http.NotFound(w, r)
return
}
_, _ = w.Write(body)
}))
t.Cleanup(srv.Close)
sum := sha256.Sum256(body)
return srv, hex.EncodeToString(sum[:])
}
func bundleExec(t *testing.T, w *bundleWrapper, srvURL string) (ConfigUpdateExecutor, *bool) {
l, _ := newLeg(t, &fakeWrapper{t: t}, nil)
l.Runner = w
called := false
return ConfigUpdateExecutor{Leg: l, URLTemplate: srvURL + "/felhom-agent/{version}/felhom-agent",
AfterInstall: func(context.Context) { called = true }}, &called
}
func signedCtx() context.Context {
return signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"agent_config_update"}`), Sig: []byte("SIG")})
}
func params(v, sha string) json.RawMessage {
p, _ := json.Marshal(ConfigUpdateParams{AgentVersion: v, BundleSHA256: sha})
return p
}
// The courier hands the wrapper the bundle bytes it downloaded and the RAW signed envelope (the wrapper verifies both
// itself), then runs the capability probe. Red-proof: drop "signed" from the plan → the plan check below fails.
func TestConfigUpdate_PassesBundleAndEnvelopeToTheWrapper(t *testing.T) {
body := []byte(`{"format":1,"agent_version":"0.143.0","files":[]}`)
srv, sha := serveBundle(t, body)
w := &bundleWrapper{t: t, rep: `{"mode":"bundle","bundle":{"agent_version":"0.143.0","written":["/etc/sudoers.d/felhom-agent"]}}`}
e, called := bundleExec(t, w, srv.URL)
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err != nil {
t.Fatal(err)
}
p := w.plans[0]
sg, _ := p["signed"].(map[string]any)
if p["mode"] != "bundle" || p["layer"] != "host" || sg == nil || sg["sig"] != "SIG" ||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"agent_config_update"}`)) {
t.Fatalf("plan = %v", p)
}
if string(w.bodies[0]) != string(body) || !strings.HasSuffix(p["bundle"].(string), "/bundle-0.143.0.json") {
t.Fatalf("the wrapper got %q at %v", w.bodies[0], p["bundle"])
}
if !*called {
t.Fatal("the capability probe must run after an install")
}
if _, err := os.Stat(p["bundle"].(string)); !os.IsNotExist(err) {
t.Fatal("the downloaded bundle must be removed after the call")
}
}
func TestConfigUpdate_WrongShaNeverReachesTheWrapper(t *testing.T) {
srv, _ := serveBundle(t, []byte("tampered"))
w := &bundleWrapper{t: t}
e, called := bundleExec(t, w, srv.URL)
err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", strings.Repeat("a", 64)))
if err == nil || len(w.plans) != 0 || *called {
t.Fatalf("err=%v plans=%d", err, len(w.plans))
}
}
func TestConfigUpdate_RefusedAndFailedAreErrors(t *testing.T) {
for _, rep := range []string{`{"refused":{"code":"R17","reason":"trust root"}}`, `{"failed":{"rc":3},"bundle":{"rolled_back":["/x"]}}`} {
srv, sha := serveBundle(t, []byte("{}"))
w := &bundleWrapper{t: t, rep: rep}
e, called := bundleExec(t, w, srv.URL)
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err == nil || *called {
t.Fatalf("%s: err=%v called=%v", rep, err, *called)
}
}
}
func TestConfigUpdate_GuardsBeforeAnyDownload(t *testing.T) {
w := &bundleWrapper{t: t}
e, _ := bundleExec(t, w, "http://127.0.0.1:1")
if err := e.Execute(signedCtx(), "agent_update", nil); !errors.Is(err, signedjobs.ErrNoExecutor) {
t.Fatalf("another op must pass through the chain: %v", err)
}
if err := e.Execute(context.Background(), OpConfigUpdate, params("0.143.0", strings.Repeat("a", 64))); err == nil {
t.Fatal("no envelope must refuse")
}
for _, p := range []json.RawMessage{params("0.143", strings.Repeat("a", 64)), params("0.143.0", "ABC"), json.RawMessage(`nope`)} {
if err := e.Execute(signedCtx(), OpConfigUpdate, p); err == nil {
t.Fatalf("bad params %s must refuse", p)
}
}
if len(w.plans) != 0 {
t.Fatal("no wrapper call on a refusal")
}
}
func TestBundleURL(t *testing.T) {
u, err := BundleURL("https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/{version}/felhom-agent", "0.143.0")
if err != nil || u != "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.143.0/felhom-config-bundle.json" {
t.Fatalf("%s %v", u, err)
}
if _, err := BundleURL("https://example/felhom-agent-{version}.bin", "0.143.0"); err == nil {
t.Fatal("an underivable template must refuse")
}
}
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return b.Run(ctx, name, args...)
}
+3
View File
@@ -73,6 +73,9 @@ func (e DockerStepExecutor) Execute(ctx context.Context, op string, params json.
// RunDockerSigned is one signed Docker step (ring 1 or an undo): live-restore first (decision 87, a no-op when on),
// then the docker layer with the signed envelope, which the wrapper verifies itself.
func (l *Leg) RunDockerSigned(ctx context.Context, vmid int, p DockerStepParams, blob []byte, sig string) Report {
unlock := l.lockPass(true)
defer unlock()
l.sendUnsentLocked(ctx) // R-868
runID := l.now().UTC().Format("20060102T150405Z")
lg := l.log().With("run", runID, "vmid", vmid, "trigger", "signed", "release", p.ReleaseID, "undo", p.Undo)
if err := l.EnsureLiveRestore(ctx, runID, vmid); err != nil {
+90 -15
View File
@@ -71,13 +71,16 @@ type Container struct {
// Health is one health reading. Guest layer: DockerOK..Containers. Host layer: HostServices, GuestRunning and the
// guest's own reading in Guest.
type Health struct {
DockerOK bool `json:"docker_ok"`
NetworkOK bool `json:"network_ok"`
Controller string `json:"controller"`
Containers map[string]Container `json:"containers"`
HostServices map[string]string `json:"host_services,omitempty"`
GuestRunning *bool `json:"guest_running,omitempty"`
Guest *Health `json:"guest,omitempty"`
DockerOK bool `json:"docker_ok"`
NetworkOK bool `json:"network_ok"`
Controller string `json:"controller"`
Containers map[string]Container `json:"containers"`
// ControllerDockerOK: the controller reaches the engine from INSIDE its container (R-858, wrapper ≥ v0.142.1;
// nil from an older wrapper = not checked). Its own health check stayed "healthy" while it was blind.
ControllerDockerOK *bool `json:"controller_docker_ok,omitempty"`
HostServices map[string]string `json:"host_services,omitempty"`
GuestRunning *bool `json:"guest_running,omitempty"`
Guest *Health `json:"guest,omitempty"`
}
// WrapperReport is the wrapper's OSAPPLY-REPORT object.
@@ -102,6 +105,14 @@ type WrapperReport struct {
Undo bool `json:"undo"`
LiveRestore json.RawMessage `json:"live_restore"`
Facts json.RawMessage `json:"facts"`
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
// agent process that started the pass. ReleaseID / VMID were always in the report.
RunID string `json:"run_id"`
Trigger string `json:"trigger"`
Ring *int `json:"ring"`
ReleaseID string `json:"release_id"`
VMID int `json:"vmid"`
}
func (w WrapperReport) refused() bool { return len(w.Refused) > 0 && string(w.Refused) != "null" }
@@ -132,6 +143,8 @@ type Report struct {
DockerEngine string `json:"docker_engine,omitempty"` // docker layer: the engine after the step
Authority string `json:"authority,omitempty"` // docker layer: ring0 | signed
Undo bool `json:"undo,omitempty"` // docker layer: a signed undo (downgrade)
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
}
// Reporter posts a report to the hub (*hub.Client).
@@ -158,14 +171,65 @@ type Leg struct {
block *hub.WireOSUpdate
}
// planFile / reportFile: the plan the agent writes and the copy of the report the wrapper keeps beside it (R-868).
func planFile(dir, runID, layer, mode string) string {
return filepath.Join(dir, fmt.Sprintf("plan-%s-%s-%s.json", runID, layer, mode))
}
func reportFile(dir, runID, layer, mode string) string {
return filepath.Join(dir, fmt.Sprintf("report-%s-%s-%s.json", runID, layer, mode))
}
func (l *Leg) planDir() string {
if l.PlanDir == "" {
return DefaultPlanDir
}
return l.PlanDir
}
// OnDesiredState stores the hub's os_update block (desired.RawConsumer — store only, never block).
func (l *Leg) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
if resp == nil {
return
}
l.mu.Lock()
defer l.mu.Unlock()
l.block = resp.DesiredState.OSUpdate
l.mu.Unlock()
l.saveBlock(resp.DesiredState.OSUpdate)
}
// SavedBlockFile is the hub's newest os_update block as the daemon last received it (R-866, v0.144.0): the debug
// pass falls back to it when the hub cannot be reached, and says so.
const SavedBlockFile = "os-update-block.json"
type savedBlock struct {
SavedAt time.Time `json:"saved_at"`
Block *hub.WireOSUpdate `json:"block"`
}
func (l *Leg) saveBlock(b *hub.WireOSUpdate) {
dir := l.planDir()
if err := os.MkdirAll(dir, 0o700); err != nil {
return
}
body, _ := json.Marshal(savedBlock{SavedAt: l.now().UTC(), Block: b})
tmp := filepath.Join(dir, SavedBlockFile+".tmp")
if err := os.WriteFile(tmp, body, 0o600); err == nil {
_ = os.Rename(tmp, filepath.Join(dir, SavedBlockFile))
}
}
// LoadSavedBlock reads the block the daemon saved (R-866). ok=false: none saved yet.
func LoadSavedBlock(dir string) (b *hub.WireOSUpdate, savedAt time.Time, ok bool) {
raw, err := os.ReadFile(filepath.Join(dir, SavedBlockFile))
if err != nil {
return nil, time.Time{}, false
}
var s savedBlock
if json.Unmarshal(raw, &s) != nil {
return nil, time.Time{}, false
}
return s.Block, s.SavedAt, true
}
// Block returns the newest os_update block. No block (an older hub, or nothing fetched yet) = ring 1, ON, no
@@ -240,6 +304,9 @@ func HealthVerdict(before, after *Health) (bool, string) {
if after.Controller != "healthy" {
return false, "the controller is " + after.Controller
}
if after.ControllerDockerOK != nil && !*after.ControllerDockerOK {
return false, "the controller cannot reach Docker (it holds an old socket — R-858)"
}
if before == nil {
return true, ""
}
@@ -345,15 +412,12 @@ func DockerHealthVerdict(before, after *Health, wantEngine, gotEngine string) (b
// call writes the plan and runs the wrapper once.
func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (WrapperReport, error) {
dir := l.PlanDir
if dir == "" {
dir = DefaultPlanDir
}
dir := l.planDir()
if err := os.MkdirAll(dir, 0o700); err != nil {
return WrapperReport{}, fmt.Errorf("osupdate: plan dir: %w", err)
}
b, _ := json.Marshal(plan)
path := filepath.Join(dir, fmt.Sprintf("plan-%s-%s-%s.json", runID, plan["layer"], plan["mode"]))
path := planFile(dir, runID, fmt.Sprint(plan["layer"]), fmt.Sprint(plan["mode"]))
if err := os.WriteFile(path, b, 0o600); err != nil {
return WrapperReport{}, fmt.Errorf("osupdate: write plan: %w", err)
}
@@ -387,6 +451,9 @@ type Pass struct {
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
// only, after good earlier steps) the Docker engine set. trigger is "night" or "debug".
func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
unlock := l.lockPass(true)
defer unlock()
l.sendUnsentLocked(ctx) // R-868: a report a killed agent never sent goes first
g, h := l.runFast(ctx, vmid, trigger)
p := Pass{Guest: g, Host: h}
if g.Outcome == "skipped" {
@@ -514,7 +581,8 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
lg.Info("osupdate: START", "enabled", blk.Enabled, "release", rel.ID)
plan := map[string]any{"release_id": rel.ID, "layer": layer, "lane": lane, "vmid": vmid, "snapshot": rel.Snapshot,
"packages": []Package{}, "mode": "apply", "select": "listed"}
"packages": []Package{}, "mode": "apply", "select": "listed",
"run_id": runID, "trigger": trigger, "ring": blk.Ring} // R-868: echoed into the wrapper's kept copy
if rel.ID == "" {
plan["release_id"] = "none"
}
@@ -547,6 +615,9 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
}
rep.Mode = plan["mode"].(string)
wr, err := l.call(ctx, runID, plan)
if rep.Mode == "apply" {
rep.unsent = reportFile(l.planDir(), runID, layer, rep.Mode) // the wrapper kept a copy (R-868)
}
switch {
case err != nil:
rep.Outcome, rep.HealthReason = "failed", err.Error()
@@ -677,7 +748,11 @@ func (l *Leg) finish(ctx context.Context, lg *slog.Logger, rep Report) Report {
rctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
defer cancel()
if err := l.Hub.PostOSReport(rctx, body); err != nil {
lg.Warn("osupdate: reporting to the hub failed (the run itself is done)", "err", err)
lg.Warn("osupdate: reporting to the hub failed (the run itself is done; the kept copy is sent at the next start or pass)", "err", err)
} else if rep.unsent != "" {
if rerr := os.Remove(rep.unsent); rerr != nil && !os.IsNotExist(rerr) {
lg.Warn("osupdate: could not delete the sent report's kept copy (it may be sent twice)", "path", rep.unsent, "err", rerr)
}
}
}
return rep
+40 -1
View File
@@ -3,6 +3,7 @@ package osupdate
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"os/exec"
@@ -21,6 +22,7 @@ type fakeWrapper struct {
applyRep map[string]WrapperReport // per layer
healthSeq map[string][]*Health // per layer: answers to successive "health" calls
plans []map[string]any
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
}
func yes() *bool { b := true; return &b }
@@ -72,6 +74,22 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
rep.Health = ok
}
}
if f.keep && plan["mode"] == "apply" {
kept := rep
kept.Layer, kept.RunID, _ = layer, fmt.Sprint(plan["run_id"]), 0
if tr, ok := plan["trigger"].(string); ok {
kept.Trigger = tr
}
if r, ok := plan["ring"].(float64); ok {
ri := int(r)
kept.Ring = &ri
}
kb, _ := json.Marshal(kept)
dst := filepath.Join(filepath.Dir(args[1]), "report-"+strings.TrimPrefix(filepath.Base(args[1]), "plan-"))
if err := os.WriteFile(dst, kb, 0o600); err != nil {
f.t.Fatal(err)
}
}
out, _ := json.Marshal(rep)
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
}
@@ -346,7 +364,7 @@ func TestWrapperSuite(t *testing.T) {
t.Skip("python3 not available")
}
// the OS wrapper and (agent v0.142.0) the crash guard — both root programs in configs/ with their own suites
for _, suite := range []string{"../../configs/test_felhom_os_apply.py", "../../configs/test_felhom_crash_guard.py"} {
for _, suite := range []string{"../../configs/test_felhom_os_apply.py", "../../configs/test_felhom_crash_guard.py", "../../configs/test_felhom_config_bundle.py", "../../configs/test_felhom_priv_apply.py"} {
cmd := exec.Command(py, "-B", suite)
out, err := cmd.CombinedOutput()
if err != nil {
@@ -465,3 +483,24 @@ func TestDocker_ChangedIDIsHealthFailed(t *testing.T) {
t.Fatalf("docker = %+v", p.Docker)
}
}
// R-858: a controller that cannot reach Docker fails the health rule even though its own check says healthy.
// Red-proof: drop the ControllerDockerOK check in HealthVerdict and this fails.
func TestHealthVerdict_ControllerBlindToDockerFails(t *testing.T) {
no, yes2 := false, true
after := guestOK()
after.ControllerDockerOK = &no
if ok, why := HealthVerdict(guestOK(), after); ok || !strings.Contains(why, "R-858") {
t.Fatalf("a blind controller passed: %v %q", ok, why)
}
if ok, _ := DockerHealthVerdict(guestOK(), after, "", ""); ok {
t.Fatal("the Docker rule passed a blind controller")
}
after.ControllerDockerOK = &yes2
if ok, why := HealthVerdict(guestOK(), after); !ok {
t.Fatalf("a seeing controller failed: %s", why)
}
if ok, _ := HealthVerdict(guestOK(), guestOK()); !ok {
t.Fatal("an older wrapper (no field) must not fail")
}
}
+197
View File
@@ -0,0 +1,197 @@
package osupdate
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"syscall"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// ── R-868 (v0.144.0): a pass whose agent was killed still reports ─────────────────────────────────────
//
// MEASURED 2026-10-05 02:57 UTC on demo-hp (night drill A5): the debug pass and the agent daemon were kill -9-ed while
// apt-get ran. The root wrapper (its own process under sudo) finished all six packages, but the agent that would have
// read its stdout and posted the report was gone; the hub never learned what the pass installed.
//
// THE MECHANISM: the wrapper writes its apply report to <plan dir>/report-<run>-<layer>-apply.json BEFORE printing
// it (configs/felhom-os-apply save_report). The agent deletes that copy once the hub has the report (finish). A copy
// still on disk is a report nobody sent: SendUnsent posts it — at the agent's start, and before every pass — and then
// deletes it. A pass lock (flock on <plan dir>/pass.lock, released by the kernel when a process dies) keeps the
// sender from picking up the copy of a pass that is still running, also across the daemon and a selftest process.
// Pinned by TestR868_* (unsent_test.go).
// lockPass takes the pass lock. block=false returns ok=false at once when another pass holds it. A lock that cannot
// be opened at all (no plan dir yet) does not stop a pass: the unlock is then a no-op.
func (l *Leg) lockPass(block bool) (unlock func()) {
u, _ := l.tryLockPass(block)
return u
}
func (l *Leg) tryLockPass(block bool) (unlock func(), ok bool) {
dir := l.planDir()
_ = os.MkdirAll(dir, 0o700)
f, err := os.OpenFile(filepath.Join(dir, "pass.lock"), os.O_CREATE|os.O_RDWR, 0o600)
if err != nil {
l.log().Warn("osupdate: pass lock unavailable — continuing without it", "err", err)
return func() {}, true
}
how := syscall.LOCK_EX
if !block {
how |= syscall.LOCK_NB
}
if err := syscall.Flock(int(f.Fd()), how); err != nil {
f.Close()
return func() {}, false
}
return func() { _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN); f.Close() }, true
}
// SendUnsent posts every report a pass kept on disk and nobody sent (R-868). It skips when a pass runs now (that
// pass sends them first). Called at the agent's start.
func (l *Leg) SendUnsent(ctx context.Context) int {
unlock, ok := l.tryLockPass(false)
if !ok {
l.log().Info("osupdate: a pass is running — its start sends any kept report")
return 0
}
defer unlock()
return l.sendUnsentLocked(ctx)
}
// SendUnsentLoop runs SendUnsent now and then every `every` until ctx ends (v0.144.1). MEASURED live on demo-hp
// 2026-10-05: after a kill -9 the daemon restarted in ~5 s, while the orphaned root wrapper was still installing — its
// copy appeared ~7 s AFTER the start-time sender had looked. One look at start is therefore not enough. A glob of the
// plan dir every few minutes costs nothing; the pass lock keeps it off a running pass. Pinned by
// TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop.
func (l *Leg) SendUnsentLoop(ctx context.Context, every time.Duration, onSent func(int)) {
for {
if n := l.SendUnsent(ctx); n > 0 && onSent != nil {
onSent(n)
}
select {
case <-ctx.Done():
return
case <-time.After(every):
}
}
}
func (l *Leg) sendUnsentLocked(ctx context.Context) int {
if l.Hub == nil {
return 0 // nobody to send to: keep the copies for a process that has the hub
}
files, _ := filepath.Glob(filepath.Join(l.planDir(), "report-*.json"))
sent := 0
for _, f := range files {
b, err := os.ReadFile(f)
if err != nil {
l.log().Warn("osupdate: a kept report cannot be read", "path", f, "err", err)
continue
}
var wr WrapperReport
if err := json.Unmarshal(b, &wr); err != nil || wr.Layer == "" {
l.log().Warn("osupdate: a kept report is not a report — moved aside", "path", f, "err", err)
_ = os.Rename(f, f+".bad")
continue
}
rep := l.reportFromKept(ctx, wr, f)
lg := l.log().With("run", rep.RunID, "layer", rep.Layer, "vmid", rep.VMID, "ring", rep.Ring, "trigger", rep.Trigger)
lg.Info("osupdate: sending a kept report late (the agent was stopped mid-pass or the hub was away, R-868)", "path", f)
before := rep.unsent
_ = l.finish(ctx, lg, rep)
if _, err := os.Stat(before); os.IsNotExist(err) {
sent++
// the pass's plan file is left behind too when the agent was killed inside call()
_ = os.Remove(filepath.Join(filepath.Dir(f), "plan-"+strings.TrimPrefix(filepath.Base(f), "report-")))
}
}
return sent
}
// reportFromKept builds the hub report from a kept wrapper report, as runLayer would have. Health: the copy's own
// before/after reading; when that is not healthy after an install, one fresh reading (services restart after an
// install, and the pass that would have waited for them is gone).
func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string) Report {
ring := 1
if wr.Ring != nil {
ring = *wr.Ring
}
runID, trigger := wr.RunID, wr.Trigger
if runID == "" {
runID = strings.TrimSuffix(strings.TrimPrefix(filepath.Base(path), "report-"), ".json")
}
if trigger == "" {
trigger = "unknown"
}
rep := Report{RunID: runID, Layer: wr.Layer, Trigger: trigger, Mode: wr.Mode, Ring: ring, ReleaseID: wr.ReleaseID,
VMID: wr.VMID, unsent: path}
prefix := "sent late — kept on the box until the hub could take it (R-868, R-875)"
switch {
case wr.refused():
rep.Outcome, rep.Refused, rep.HealthReason = "refused", wr.Refused, prefix
return rep
case wr.failed():
rep.Outcome, rep.Refused = "failed", wr.Failed
case len(wr.Upgraded) == 0:
rep.Outcome = "nothing"
default:
rep.Outcome = "applied"
}
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
wantEngine := ""
for _, u := range wr.Upgraded {
if u.Name == "docker-ce" {
wantEngine = EngineOf(u.Version)
}
}
verdict := func(h *Health) (bool, string) {
switch wr.Layer {
case LayerDocker:
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
case LayerHost:
t := hub.TunnelUnknown
if l.Tunnel != nil {
t, _ = l.Tunnel.Status(ctx)
}
return HostHealthVerdict(wr.HealthBefore, h, t)
}
return HealthVerdict(wr.HealthBefore, h)
}
ok, why := verdict(wr.HealthAfter)
if !ok && len(wr.Upgraded) > 0 && wr.VMID > 0 {
lane := "fast"
if wr.Layer == LayerDocker {
lane = "slow"
}
if hr, err := l.call(ctx, "kept-"+runID, map[string]any{"release_id": "kept", "layer": wr.Layer, "lane": lane,
"vmid": wr.VMID, "mode": "health", "packages": []Package{}}); err == nil && hr.Health != nil {
ok, why = verdict(hr.Health)
}
}
rep.Healthy, rep.HealthReason = ok, prefix
if why != "" {
rep.HealthReason = prefix + ": " + why
}
if !ok && rep.Outcome == "applied" {
rep.Outcome = "health_failed"
}
rep.Installed, rep.Pending = wr.Installed, wr.Pending
rep.RestartNeeded, rep.DockerRestartNeeded, rep.RebootNeeded = wr.RestartNeeded, wr.DockerRestartNeeded, wr.RebootNeeded
rep.RebootScanned = wr.RebootScanned
if wr.Layer == LayerDocker {
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
} else {
planned := map[string]bool{}
for _, u := range wr.Upgraded {
planned[u.Name] = true
}
rep.NotCovered = notCovered(wr.Pending, ring, planned)
}
return rep
}
+152
View File
@@ -0,0 +1,152 @@
package osupdate
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-868 (v0.144.0). THE NIGHT'S SHAPE (A5, demo-hp 2026-10-05 02:57 UTC): the wrapper finished six packages, the agent
// was kill -9-ed before it read the report; the hub never got it. Here: the wrapper's kept copy and the plan file are
// on disk, a NEW agent process starts — it must send exactly one `applied` report and delete both files.
// COMPANION RED-PROOF: drop the SendUnsent body (return 0) → "the hub got no report".
func TestR868_KilledPassIsReportedAtStart(t *testing.T) {
w := &fakeWrapper{t: t}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
ring := 0
kept := WrapperReport{Mode: "apply", Layer: LayerGuest, RunID: "20261005T025700Z", Trigger: "debug", Ring: &ring, VMID: 9201,
ReleaseID: "ring0-20261005T025700Z", HealthBefore: guestOK(), HealthAfter: guestOK(),
Upgraded: []Package{{Name: "libc6", Version: "u4"}, {Name: "openssl", Version: "u3"}}}
b, _ := json.Marshal(kept)
rp := reportFile(l.PlanDir, kept.RunID, LayerGuest, "apply")
pp := planFile(l.PlanDir, kept.RunID, LayerGuest, "apply")
must(t, os.WriteFile(rp, b, 0o600))
must(t, os.WriteFile(pp, []byte("{}"), 0o600))
if n := l.SendUnsent(context.Background()); n != 1 {
t.Fatalf("sent %d, want 1", n)
}
if len(h.reports) != 1 {
t.Fatalf("the hub got no report (or several): %+v", h.reports)
}
r := h.reports[0]
if r.Outcome != "applied" || !r.Healthy || r.Trigger != "debug" || r.RunID != kept.RunID || r.Ring != 0 || len(r.Upgraded) != 2 || r.VMID != 9201 {
t.Fatalf("report = %+v", r)
}
// R-875 (v0.145.0): neutral — the copy cannot tell a killed agent from an absent hub.
if !strings.HasPrefix(r.HealthReason, "sent late") || strings.Contains(r.HealthReason, "stopped mid-pass") {
t.Fatalf("health_reason = %q, want the neutral \"sent late …\"", r.HealthReason)
}
for _, p := range []string{rp, pp} {
if _, err := os.Stat(p); !os.IsNotExist(err) {
t.Fatalf("%s still on disk after the hub got it", filepath.Base(p))
}
}
// a second start sends nothing again — no duplicate report
if n := l.SendUnsent(context.Background()); n != 0 || len(h.reports) != 1 {
t.Fatalf("sent again: %d, reports %d", n, len(h.reports))
}
}
// A normal pass: the hub gets ONE report per layer and the kept copies are gone after it (nothing resent later).
// COMPANION RED-PROOF: drop the os.Remove(rep.unsent) in finish → the next pass resends → "2 guest reports".
func TestR868_NormalPassLeavesNoCopyAndNoDuplicate(t *testing.T) {
w := &fakeWrapper{t: t, keep: true, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "u4"}}}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l.Run(context.Background(), 9201, "debug")
left, _ := filepath.Glob(filepath.Join(l.PlanDir, "report-*.json"))
if len(left) != 0 {
t.Fatalf("kept copies left after the hub got them: %v", left)
}
l.Run(context.Background(), 9201, "debug") // the next pass sends kept copies first
guest := 0
for _, r := range h.reports {
if r.Layer == LayerGuest && r.Outcome == "applied" {
guest++
}
}
if guest != 2 {
t.Fatalf("%d guest reports for 2 passes (a duplicate or a loss)", guest)
}
}
type failingHub struct{ n int }
func (h *failingHub) PostOSReport(context.Context, []byte) error {
h.n++
return errors.New("hub away")
}
// The hub away: the copy stays, and goes at the next chance.
func TestR868_HubAwayKeepsTheCopy(t *testing.T) {
w := &fakeWrapper{t: t, keep: true, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "u4"}}}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l.Appliance = false
l.Hub = &failingHub{}
l.Run(context.Background(), 9201, "night")
left, _ := filepath.Glob(filepath.Join(l.PlanDir, "report-*.json"))
if len(left) != 2 { // ring 0: the guest step and the Docker step each kept one
t.Fatalf("the copies must stay while the hub is away: %v", left)
}
h := &fakeHub{}
l.Hub = h
if n := l.SendUnsent(context.Background()); n != 2 {
t.Fatalf("sent %d: %+v", n, h.reports)
}
for _, r := range h.reports {
if r.Trigger != "night" || r.Ring != 0 {
t.Fatalf("the kept report lost its ids: %+v", r)
}
}
}
// A pass in progress holds the lock: the sender at start must not take that pass's copy (it would be sent twice).
func TestR868_RunningPassKeepsTheSenderOff(t *testing.T) {
w := &fakeWrapper{t: t}
l, h := newLeg(t, w, nil)
must(t, os.WriteFile(reportFile(l.PlanDir, "r1", LayerGuest, "apply"), []byte(`{"mode":"apply","layer":"guest"}`), 0o600))
unlock := l.lockPass(true)
if n := l.SendUnsent(context.Background()); n != 0 || len(h.reports) != 0 {
t.Fatalf("sent while a pass ran: %d", n)
}
unlock()
if n := l.SendUnsent(context.Background()); n != 1 {
t.Fatalf("not sent after the pass: %d", n)
}
}
// v0.144.1 — THE LIVE SHAPE (demo-hp 2026-10-05 05:45 UTC): the restarted daemon looked at 05:45:09, the orphaned
// wrapper wrote its copy at ~05:45:16. The loop must still send it.
// COMPANION RED-PROOF: make SendUnsentLoop return after the first look → "the late copy was never sent".
func TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop(t *testing.T) {
w := &fakeWrapper{t: t}
l, h := newLeg(t, w, nil)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
sent := make(chan int, 4)
go l.SendUnsentLoop(ctx, 20*time.Millisecond, func(n int) { sent <- n })
time.Sleep(50 * time.Millisecond) // the start-time look found nothing
must(t, os.WriteFile(reportFile(l.PlanDir, "late", LayerGuest, "apply"),
[]byte(`{"mode":"apply","layer":"guest","run_id":"late","trigger":"debug","ring":0,"vmid":9201,"upgraded":[{"name":"openssl","version":"u3"}]}`), 0o600))
select {
case n := <-sent:
if n != 1 || len(h.reports) != 1 || h.reports[0].RunID != "late" || h.reports[0].Outcome == "" {
t.Fatalf("sent %d: %+v", n, h.reports)
}
case <-time.After(3 * time.Second):
t.Fatal("the late copy was never sent")
}
}
func must(t *testing.T, err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
+34
View File
@@ -0,0 +1,34 @@
// Package privapplytest runs configs/felhom-priv-apply in CHECK-ONLY mode from Go tests (R-861): each renderer's
// real output must be accepted by the root checker, so the two can never drift apart unnoticed. Test-only helper.
package privapplytest
import (
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"testing"
)
// Check writes content to a temp file and runs `felhom-priv-apply --check <verb> [name] <file>`. It returns the
// checker's verdict line ("OK" or "REFUSED [rule] …"). Skips when python3 is absent.
func Check(t *testing.T, verb, name, content string) string {
t.Helper()
py, err := exec.LookPath("python3")
if err != nil {
t.Skip("python3 not available")
}
_, here, _, _ := runtime.Caller(0)
wrapper := filepath.Join(filepath.Dir(here), "..", "..", "configs", "felhom-priv-apply")
f := filepath.Join(t.TempDir(), "staged")
if err := os.WriteFile(f, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
args := []string{"-B", wrapper, "--check", verb}
if name != "" {
args = append(args, name)
}
out, _ := exec.Command(py, append(args, f)...).CombinedOutput()
return strings.TrimSpace(string(out))
}
+3 -2
View File
@@ -186,8 +186,9 @@ func (b *BackHalf) Provision(ctx context.Context, in Input) (Result, error) {
// 6. Install + register the pre-start self-heal hook (C1 net): if a data drive is absent at a future
// boot, the hook creates a placeholder for its missing bind source so the guest still starts.
// Best-effort + non-fatal — it's defense-in-depth; a provision must not fail over the hook.
if err := guesthook.InstallSnippet(ctx, b.runner); err != nil {
b.logger.Warn("provision: pre-start hook snippet install failed (non-fatal)", "vmid", in.VMID, "err", err)
// R-861 (v0.146.0): the hook FILE comes with the signed config bundle; the agent only checks it and registers it.
if err := guesthook.SnippetReady(guesthook.SnippetPath); err != nil {
b.logger.Warn("provision: pre-start hook not in place — not registering it (non-fatal)", "vmid", in.VMID, "err", err)
} else if err := guesthook.Register(ctx, b.runner, in.VMID); err != nil {
b.logger.Warn("provision: pre-start hook registration failed (non-fatal)", "vmid", in.VMID, "err", err)
}
+5 -1
View File
@@ -51,6 +51,10 @@ const (
// A Docker engine step in the customer guest (agent v0.142.0, `11` §5.8) — ring 1, and every undo. Destructive-class
// (signed, operational key) like agent_update; the root wrapper re-verifies the same signature itself.
ClassOSDockerStep OpClass = "os_docker_step"
// The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units).
// Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself.
ClassAgentConfigUpdate OpClass = "agent_config_update"
)
// Disposition is the classifier verdict.
@@ -119,7 +123,7 @@ func Classify(class OpClass, prov Provenance) Disposition {
return Destructive
case ClassKeyRotation:
return Destructive
case ClassAgentUpdate, ClassOSDockerStep:
case ClassAgentUpdate, ClassOSDockerStep, ClassAgentConfigUpdate:
// Never benign — no agent-internal provenance can make replacing the agent binary
// unsigned-safe (a compromised process must not be able to self-bless an update).
return Destructive
+63 -8
View File
@@ -3,6 +3,7 @@ package selfupdate
import (
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"fmt"
@@ -11,6 +12,7 @@ import (
"net/http"
"os"
"path/filepath"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
@@ -47,6 +49,7 @@ type Executor struct {
username string
token string
stateDir string
planDir string // felhom-os-apply's plan dir; "" = defaultPlanDir (tests set a temp dir)
runner WrapperRunner
httpClient *http.Client
logger *slog.Logger
@@ -126,18 +129,70 @@ func (e *Executor) Execute(ctx context.Context, op string, params json.RawMessag
return fmt.Errorf("agent_update: chmod staged: %w", err)
}
// Hand off to the root wrapper. It re-verifies the sha, flips A/B, writes the pending marker,
// and schedules the detached restart. After this the new binary starts; the commit is the
// Manager's job once it has dwelled cleanly.
e.logger.Warn("agent_update: handing staged binary to the guarded wrapper", "staged", staged, "version", p.Version)
stdout, stderr, err := e.runner.Run(ctx, wrapperPath, "apply", staged, p.SHA256)
if err != nil {
return fmt.Errorf("agent_update: wrapper apply failed: %w (stderr: %s)", err, string(stderr))
// R-861 (v0.146.0): hand the SIGNED job to the root wrapper felhom-os-apply (mode agent_update). It verifies the
// operator's signature itself (root-owned signers file, this host, the window, the nonce), re-hashes the staged
// file against the SIGNED sha, and only then runs the A/B flip (felhom-selfupdate-guarded apply, no longer in the
// agent's sudoers). Until v0.146.0 the agent passed the sha to the flip itself, so a compromised agent could
// install any binary — and the binary is what the escrow ceremony and the guest hook run as root.
so, ok := signedjobs.SignedOpFrom(ctx)
if !ok {
return fmt.Errorf("agent_update: no signed envelope in the context — the root wrapper could not verify it")
}
e.logger.Warn("agent_update: apply handed off; restart scheduled", "version", p.Version, "wrapper", trim(stdout))
planDir := e.planDir
if planDir == "" {
planDir = defaultPlanDir
}
if err := os.MkdirAll(planDir, 0o700); err != nil {
return fmt.Errorf("agent_update: plan dir: %w", err)
}
plan, _ := json.Marshal(map[string]any{"release_id": "agent-" + p.Version, "layer": "host", "mode": "agent_update",
"staged": staged, "signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(so.Blob), "sig": string(so.Sig)}})
planPath := filepath.Join(planDir, "plan-agentupdate-"+p.Version+".json")
if err := os.WriteFile(planPath, plan, 0o600); err != nil {
return fmt.Errorf("agent_update: write plan: %w", err)
}
defer os.Remove(planPath)
e.logger.Warn("agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)", "staged", staged, "version", p.Version)
stdout, stderr, err := e.runner.Run(ctx, osApplyPath, "--plan", planPath)
rep := parseOSApplyReport(stdout)
var r struct {
Refused json.RawMessage `json:"refused"`
Failed json.RawMessage `json:"failed"`
AgentUpdate json.RawMessage `json:"agent_update"`
}
jerr := json.Unmarshal([]byte(rep), &r)
refused := len(r.Refused) > 0 && string(r.Refused) != "null"
if err != nil || jerr != nil || refused || len(r.Failed) > 0 || len(r.AgentUpdate) == 0 {
return fmt.Errorf("agent_update: the root wrapper did not apply it: %v (report: %s; stderr: %s)", err, trimStr(rep), trim(stderr))
}
e.logger.Warn("agent_update: signed update verified as root and handed off; restart scheduled", "version", p.Version, "report", trimStr(rep))
return nil
}
// osApplyPath is the root wrapper that verifies the signed agent_update (R-861). Fixed, never config-overridable.
const osApplyPath = "/usr/local/sbin/felhom-os-apply"
// defaultPlanDir is felhom-os-apply's ONLY plan directory (PLAN_DIR there; osupdate.DefaultPlanDir here).
const defaultPlanDir = "/var/lib/felhom-agent/os"
// parseOSApplyReport returns the JSON after the wrapper's last "OSAPPLY-REPORT " line ("" when there is none).
func parseOSApplyReport(stdout []byte) string {
rep := ""
for _, line := range strings.Split(string(stdout), "\n") {
if strings.HasPrefix(line, "OSAPPLY-REPORT ") {
rep = strings.TrimPrefix(line, "OSAPPLY-REPORT ")
}
}
return rep
}
func trimStr(s string) string {
if len(s) > 400 {
return s[:400] + "…"
}
return s
}
// download streams url → dest (0644, fsync'd) and returns the lowercase-hex sha256 of the bytes.
func (e *Executor) download(ctx context.Context, url, dest string) (string, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+69 -11
View File
@@ -15,34 +15,56 @@ import (
"sync"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// fakeWrapper records the verbs the executor/manager shell out, and returns a configurable error.
// fakeWrapper records the verbs the executor/manager shell out, and returns a configurable error. For the os-apply
// call it snapshots the plan file at call time (the executor removes it afterwards) and answers with report.
type fakeWrapper struct {
mu sync.Mutex
calls [][]string
err error
mu sync.Mutex
calls [][]string
err error
plans []map[string]any
report string // the OSAPPLY-REPORT JSON; "" = a successful agent_update
}
func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.calls = append(f.calls, append([]string{name}, args...))
if name == osApplyPath && len(args) == 2 && args[0] == "--plan" {
var p map[string]any
b, _ := os.ReadFile(args[1])
_ = json.Unmarshal(b, &p)
f.plans = append(f.plans, p)
rep := f.report
if rep == "" {
rep = `{"agent_update": {"version": "x", "wrapper_rc": 0}, "mode": "agent_update", "refused": null}`
}
return []byte("OSAPPLY-REPORT " + rep + "\n"), nil, f.err
}
return []byte("ok"), nil, f.err
}
// applyCalls are the hand-offs to the root wrapper (felhom-os-apply --plan …). Since v0.146.0 the agent never calls
// `felhom-selfupdate-guarded apply` itself.
func (f *fakeWrapper) applyCalls() [][]string {
f.mu.Lock()
defer f.mu.Unlock()
var out [][]string
for _, c := range f.calls {
if len(c) >= 2 && c[1] == "apply" {
if c[0] == osApplyPath || (len(c) >= 2 && c[1] == "apply") {
out = append(out, c)
}
}
return out
}
func signedCtx() context.Context {
return signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"agent_update"}`), Sig: []byte("SIG")})
}
func sha256Of(b []byte) string {
h := sha256.Sum256(b)
return hex.EncodeToString(h[:])
@@ -65,6 +87,7 @@ func newExec(t *testing.T, srv *httptest.Server, wrap WrapperRunner) (*Executor,
Runner: wrap,
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
e.planDir = t.TempDir()
return e, stateDir
}
@@ -84,7 +107,7 @@ func TestExecutor_HappyPath(t *testing.T) {
e, stateDir := newExec(t, srv, wrap)
sha := sha256Of(body)
if err := e.Execute(context.Background(), "agent_update", updateParamsJSON(t, "0.70.1", sha)); err != nil {
if err := e.Execute(signedCtx(), "agent_update", updateParamsJSON(t, "0.70.1", sha)); err != nil {
t.Fatalf("execute: %v", err)
}
staged := filepath.Join(stateDir, "selfupdate", "felhom-agent-0.70.1")
@@ -93,11 +116,46 @@ func TestExecutor_HappyPath(t *testing.T) {
t.Fatalf("staged binary missing/mismatch: %v", err)
}
calls := wrap.applyCalls()
if len(calls) != 1 {
t.Fatalf("apply invoked %d times, want 1 (%v)", len(calls), wrap.calls)
if len(calls) != 1 || calls[0][0] != osApplyPath || calls[0][1] != "--plan" {
t.Fatalf("want exactly one hand-off to felhom-os-apply --plan, got %v", wrap.calls)
}
if calls[0][2] != staged || calls[0][3] != sha {
t.Errorf("apply args = %v, want [.. apply %s %s]", calls[0], staged, sha)
// R-861: the plan carries the SIGNED envelope and the staged path; the wrapper, not the agent, decides.
p := wrap.plans[0]
sg, _ := p["signed"].(map[string]any)
if p["mode"] != "agent_update" || p["layer"] != "host" || p["staged"] != staged || sg["sig"] != "SIG" || sg["blob_b64"] == "" {
t.Errorf("plan = %v, want mode agent_update + the staged path + the signed envelope", p)
}
if _, err := os.Stat(calls[0][2]); !os.IsNotExist(err) {
t.Error("the plan file was left behind")
}
}
// R-861: without the signed envelope nothing reaches the root wrapper.
// RED-PROOF (audits/hub-safety-2026-10-05/partF/red-proof.txt): call `felhom-selfupdate-guarded apply` directly again
// → the happy path's "exactly one hand-off to felhom-os-apply" fails.
func TestExecutor_NoEnvelopeNoHandOff(t *testing.T) {
body := []byte("good bytes")
srv := artifactServer(t, body)
defer srv.Close()
wrap := &fakeWrapper{}
e, _ := newExec(t, srv, wrap)
if err := e.Execute(context.Background(), "agent_update", updateParamsJSON(t, "0.70.1", sha256Of(body))); err == nil {
t.Fatal("an update with no signed envelope was handed on")
}
if len(wrap.applyCalls()) != 0 {
t.Fatalf("the root wrapper was called without an envelope: %v", wrap.calls)
}
}
// A refusal in the wrapper's report is a failure, even when its exit code reads 0.
func TestExecutor_WrapperRefusalSurfaces(t *testing.T) {
body := []byte("good bytes")
srv := artifactServer(t, body)
defer srv.Close()
wrap := &fakeWrapper{report: `{"mode": "agent_update", "refused": {"code": "R3", "reason": "the operator signature does not verify"}}`}
e, _ := newExec(t, srv, wrap)
if err := e.Execute(signedCtx(), "agent_update", updateParamsJSON(t, "0.70.1", sha256Of(body))); err == nil {
t.Fatal("a refused signed update read as applied")
}
}
@@ -158,7 +216,7 @@ func TestExecutor_WrapperFailureSurfaces(t *testing.T) {
defer srv.Close()
wrap := &fakeWrapper{err: errors.New("wrapper refused: sha mismatch")}
e, _ := newExec(t, srv, wrap)
if err := e.Execute(context.Background(), "agent_update", updateParamsJSON(t, "0.70.1", sha256Of(body))); err == nil {
if err := e.Execute(signedCtx(), "agent_update", updateParamsJSON(t, "0.70.1", sha256Of(body))); err == nil {
t.Fatal("wrapper failure must surface")
}
}
+10 -3
View File
@@ -101,6 +101,10 @@ type MountSpec struct {
type Binaries struct {
Systemctl string
Install string
// PrivApply is the root content checker (R-861, agent v0.146.0): a unit file reaches /etc/systemd/system only
// through `felhom-priv-apply unit <name>`, which reads the staged copy from /var/lib/felhom-agent/units/ and
// refuses anything the renderers here never produce (a bind over /etc, a Where outside /mnt, …).
PrivApply string
Smartctl string
Lvs string
Blkid string // device signature probe (8C data-bearing detection)
@@ -119,6 +123,9 @@ func (b Binaries) withDefaults() Binaries {
if b.Install == "" {
b.Install = "/usr/bin/install"
}
if b.PrivApply == "" {
b.PrivApply = "/usr/local/sbin/felhom-priv-apply"
}
if b.Smartctl == "" {
b.Smartctl = "/usr/sbin/smartctl"
}
@@ -246,9 +253,9 @@ func (h *SudoHostOps) EnsureMount(ctx context.Context, spec MountSpec) error {
return fmt.Errorf("storage: staging unit: %w", err)
}
dest := filepath.Join(h.unitDir, unitName)
// install as root (atomic copy with fixed mode/owner) — fixed arg vector.
if err := h.run(ctx, h.bins.Install, "-o", "root", "-g", "root", "-m", "0644", "--", stagePath, dest); err != nil {
// R-861: the root checker installs it (fixed source dir, fixed destination dir, content checked) — never a
// plain `install` of a file the agent wrote.
if err := h.run(ctx, h.bins.PrivApply, "unit", unitName); err != nil {
return fmt.Errorf("storage: installing unit %s: %w", unitName, err)
}
if err := h.run(ctx, h.bins.Systemctl, "daemon-reload"); err != nil {
+3 -2
View File
@@ -55,8 +55,9 @@ func TestHostOps_MountLifecycle(t *testing.T) {
if len(rr.calls) != 3 {
t.Fatalf("expected 3 commands, got %d: %v", len(rr.calls), rr.calls)
}
if rr.calls[0][0] != "/usr/bin/install" || !contains(rr.calls[0], "0644") {
t.Errorf("call[0] not the install: %v", rr.calls[0])
// R-861: the unit is installed by the root content checker, named — never a plain `install` of a staged path.
if rr.calls[0][0] != "/usr/local/sbin/felhom-priv-apply" || len(rr.calls[0]) != 3 || rr.calls[0][1] != "unit" {
t.Errorf("call[0] not the checker's unit install: %v", rr.calls[0])
}
if !contains(rr.calls[1], "daemon-reload") {
t.Errorf("call[1] not daemon-reload: %v", rr.calls[1])
+1 -1
View File
@@ -100,7 +100,7 @@ func TestMigrateNetworkUnits_RewritesDriftedOnceIdempotent(t *testing.T) {
if strings.Contains(joined, "daemon-reload") {
reloads++
}
if strings.Contains(joined, "install") {
if strings.Contains(joined, "felhom-priv-apply unit") {
installs++
}
}
+7 -3
View File
@@ -236,9 +236,13 @@ func (s NetworkMountSpec) mountOptions() string {
"file_mode=0664",
"dir_mode=0775",
"_netdev",
"nosuid",
"nodev",
}, ",")
}
return "vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev,retry=0"
// R-861 (v0.146.0): nosuid,nodev — a set-uid file or a device node on a server outside the box must never act on
// the host. felhom-priv-apply refuses a network unit without them.
return "vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev,retry=0,nosuid,nodev"
}
// renderNetworkMountUnit builds the .mount unit (triggered by the .automount; deliberately NO [Install]
@@ -409,8 +413,8 @@ func (h *SudoHostOps) installUnit(ctx context.Context, unitName, content string)
if err := os.WriteFile(stagePath, []byte(content), 0o644); err != nil {
return fmt.Errorf("netmount: staging unit %s: %w", unitName, err)
}
dest := filepath.Join(h.unitDir, unitName)
if err := h.run(ctx, h.bins.Install, "-o", "root", "-g", "root", "-m", "0644", "--", stagePath, dest); err != nil {
// R-861: through the root checker (felhom-priv-apply unit), never a plain install of an agent-written file.
if err := h.run(ctx, h.bins.PrivApply, "unit", unitName); err != nil {
return fmt.Errorf("netmount: installing unit %s: %w", unitName, err)
}
return nil
+1 -1
View File
@@ -264,7 +264,7 @@ func TestEnsureNetworkMount_Commands(t *testing.T) {
switch {
case strings.Contains(joined, "mkdir") && strings.Contains(joined, "/mnt/felhom-drives/media"):
sawMkdir = true
case strings.Contains(joined, "install"):
case strings.Contains(joined, "felhom-priv-apply unit"):
installs++
case strings.Contains(joined, "daemon-reload"):
sawReload = true
@@ -0,0 +1,43 @@
package storage
import (
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/privapplytest"
)
// R-861: every unit the agent renders is one the root checker installs — the name it computes, the Where, the
// options. RED-PROOF: drop "nosuid","nodev" from mountOptions → the network cases are REFUSED [U5].
func TestPrivApply_AcceptsTheRenderedUnits(t *testing.T) {
local := MountSpec{Name: "x", UUID: "91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae", Where: "/mnt/hdd_1", FSType: "ext4"}
name, _ := UnitNameForMount(local.Where)
if got := privapplytest.Check(t, "unit", name, renderMountUnit(local)); got != "OK" {
t.Errorf("local unit %s: %s", name, got)
}
local.FSType = ""
if got := privapplytest.Check(t, "unit", name, renderMountUnit(local)); got != "OK" {
t.Errorf("local unit without Type: %s", got)
}
for _, spec := range []NetworkMountSpec{
{Name: "media", Protocol: ProtocolNFS, Server: "10.0.0.5", Export: "/srv/media", MappedUID: 1000, MappedGID: 1000},
{Name: "photos", Protocol: ProtocolSMB, Server: "nas.lan", Export: "photos", CredsRef: "/etc/felhom/netmount/photos.cred", MappedUID: 1000, MappedGID: 1000},
} {
mu, err := UnitNameForMount(spec.Where())
if err != nil {
t.Fatal(err)
}
if got := privapplytest.Check(t, "unit", mu, renderNetworkMountUnit(spec)); got != "OK" {
t.Errorf("%s .mount: %s", spec.Name, got)
}
au := strings.TrimSuffix(mu, ".mount") + ".automount"
if got := privapplytest.Check(t, "unit", au, renderNetworkAutomountUnit(spec)); got != "OK" {
t.Errorf("%s .automount: %s", spec.Name, got)
}
}
// control: the checker is really looking — a unit over /etc is refused
evil := strings.Replace(renderMountUnit(MountSpec{UUID: local.UUID, Where: "/mnt/hdd_1"}), "Where=/mnt/hdd_1", "Where=/etc/sudoers.d", 1)
if got := privapplytest.Check(t, "unit", name, evil); !strings.HasPrefix(got, "REFUSED") {
t.Fatalf("control: a unit over /etc/sudoers.d was not refused: %s", got)
}
}
+5 -2
View File
@@ -29,6 +29,8 @@ const (
Iface = "wg-felhom"
// confDest is the installed conf path — must match the FELHOM_WG sudoers entry EXACTLY.
confDest = "/etc/wireguard/wg-felhom.conf"
// privApply is the root content checker that installs confDest (R-861).
privApply = "/usr/local/sbin/felhom-priv-apply"
// unit is the systemd unit the sudoers allowlists.
unit = "wg-quick@wg-felhom"
@@ -507,8 +509,9 @@ func (m *Manager) ensureTunnelLocked(ctx context.Context, block *hub.WireWiregua
m.logger.Error("wgtunnel: staging conf", "err", err)
return
}
// argv matches the FELHOM_WG sudoers entry exactly (fixed source + dest).
if _, errOut, err := m.runner.Run(ctx, "install", "-o", "root", "-g", "root", "-m", "0600", "--", m.stagedConfPath(), confDest); err != nil {
// R-861 (v0.146.0): the root checker installs the staged conf (fixed source /var/lib/felhom-agent/wg/, fixed
// destination, 0600) and refuses any key renderConf never writes — PostUp/PreUp run as root under wg-quick.
if _, errOut, err := m.runner.Run(ctx, privApply, "wg"); err != nil {
m.logger.Error("wgtunnel: conf install failed", "err", err, "stderr", strings.TrimSpace(string(errOut)))
return
}
+4 -4
View File
@@ -286,7 +286,7 @@ func TestScenarioA_RegisterThenApplyOrdering(t *testing.T) {
// Block arrives → conf staged + installed + enabled.
pub := localPub(t, m)
m.Apply(ctx, true, testBlock(pub))
if rr.count("install") != 1 || rr.count("systemctl") != 1 {
if rr.count(privApply) != 1 || rr.count("systemctl") != 1 {
t.Fatalf("apply execs = %v", rr.calls)
}
if got := rr.lastSystemctl(); strings.Join(got, " ") != "systemctl enable --now wg-quick@wg-felhom" {
@@ -419,7 +419,7 @@ func TestAdoptLostMarkerWithKnownKey(t *testing.T) {
if mk == nil || mk.Pubkey != pub || mk.AssignedIP != "10.77.0.2/32" {
t.Fatalf("adoption marker = %+v", mk)
}
if rr.count("install") != 1 {
if rr.count(privApply) != 1 {
t.Errorf("adopt did not proceed to conf apply: %v", rr.calls)
}
}
@@ -686,13 +686,13 @@ func TestInitialResolveFailureNoTeardown(t *testing.T) {
m.Apply(ctx, false, nil) // register (no DNS)
pub := localPub(t, m)
m.Apply(ctx, true, testBlock(pub)) // resolve fails → cannot apply
if rr.count("install") != 0 || rr.count("systemctl") != 0 {
if rr.count(privApply) != 0 || rr.count("systemctl") != 0 {
t.Errorf("applied/tore-down despite a resolve failure: %v", rr.calls)
}
// DNS returns → the tunnel comes up on the next tick.
fr.set(netip.MustParseAddr("167.233.158.164"))
m.Apply(ctx, true, testBlock(pub))
if rr.count("install") != 1 {
if rr.count(privApply) != 1 {
t.Errorf("did not recover after DNS returned: %v", rr.calls)
}
}
@@ -0,0 +1,27 @@
package wgtunnel
import (
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/privapplytest"
)
// R-861: the conf renderConf writes (with and without the operator OOB peer) is accepted; a PostUp line is not.
func TestPrivApply_AcceptsTheRenderedConf(t *testing.T) {
priv := "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8="
b := testBlock("CQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQk=")
for _, oob := range []string{"", "10.77.0.250"} {
b.OOBPeerIP = oob
conf, err := renderConf(b, priv, "49.12.1.2")
if err != nil {
t.Fatal(err)
}
if got := privapplytest.Check(t, "wg", "", conf); got != "OK" {
t.Errorf("rendered conf (oob=%q): %s", oob, got)
}
if got := privapplytest.Check(t, "wg", "", strings.Replace(conf, "MTU = 1280", "MTU = 1280\nPostUp = id", 1)); !strings.HasPrefix(got, "REFUSED") {
t.Fatalf("control: PostUp was not refused: %s", got)
}
}
}
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""build-config-bundle.py — build the agent's CONFIG BUNDLE (R-840, `11` §5.4.2).
Usage: python3 scripts/build-config-bundle.py <agent-version> <out.json> (prints the bundle's sha256)
The bundle is every root-owned file the installer's step 5 writes for the agent (sudoers, wrappers, units), as ONE
JSON file published beside the binary (felhom-agent/<version>/felhom-config-bundle.json). A box takes it by a signed
`agent_config_update` job; a new box takes the SAME file from the installer. The list of files is NOT kept here: it is
`BUNDLE_FILES` in configs/felhom-os-apply, the root wrapper that installs it — one table, so the builder cannot put in a
path the wrapper would refuse, nor leave out one it expects.
Reproducible by construction: no timestamps, sorted keys, the table's order. The same source at the same version gives
the same sha256 every time (pinned by configs/test_felhom_config_bundle.py).
"""
import base64
import hashlib
import importlib.machinery
import importlib.util
import json
import pathlib
import re
import sys
REPO = pathlib.Path(__file__).resolve().parent.parent
CONFIGS = REPO / "configs"
def load_wrapper(configs=CONFIGS):
loader = importlib.machinery.SourceFileLoader("osapply_for_bundle", str(configs / "felhom-os-apply"))
spec = importlib.util.spec_from_loader("osapply_for_bundle", loader)
mod = importlib.util.module_from_spec(spec)
loader.exec_module(mod)
return mod
def build(version, configs=CONFIGS):
if not re.match(r"^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$", version):
raise SystemExit(f"build-config-bundle: version {version!r} is not semver")
w = load_wrapper(configs)
files = []
for dest, src, mode, check, policy in w.BUNDLE_FILES:
data = (configs / src).read_bytes()
files.append({"path": dest, "source": f"configs/{src}", "mode": oct(mode), "check": check, "policy": policy,
"sha256": hashlib.sha256(data).hexdigest(), "content_b64": base64.b64encode(data).decode()})
body = {"format": w.BUNDLE_FORMAT, "agent_version": version, "files": files}
return (json.dumps(body, indent=1, sort_keys=True) + "\n").encode()
def main(argv):
if len(argv) != 3:
print(__doc__, file=sys.stderr)
return 2
data = build(argv[1])
pathlib.Path(argv[2]).write_bytes(data)
print(hashlib.sha256(data).hexdigest())
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
+21
View File
@@ -100,6 +100,15 @@ built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
log "built ok: sha256 $BUILT_SHA"
# ── 3b. The config bundle (R-840) ───────────────────────────────────────────────────────────────
# Every root-owned file the installer's step 5 writes (sudoers, wrappers, units), as ONE file beside the binary. A box
# takes it by a signed agent_config_update; a new box from the installer. Reproducible: same source → same sha.
BUNDLE="$(mktemp -t felhom-config-bundle-XXXXXX)"
trap 'rm -f "$BIN" "$BUNDLE"' EXIT
BUNDLE_SHA="$(python3 "$REPO_ROOT/scripts/build-config-bundle.py" "$VERSION" "$BUNDLE")" || die "config bundle build failed"
[[ "$BUNDLE_SHA" =~ ^[0-9a-f]{64}$ ]] || die "config bundle build printed no sha256"
log "config bundle built: sha256 $BUNDLE_SHA"
# ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ──────────────────────────────
#
# THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03).
@@ -153,6 +162,12 @@ if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then
die "publish failed"
fi
# ── 5b. Publish the config bundle beside the binary (same package version, same credentials) ──
BURL="$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-config-bundle.json"
bcode="$(curl -sS -o /dev/null -w '%{http_code}' -u "${GITEA_USER}:${GITEA_TOKEN}" -X PUT --upload-file "$BUNDLE" "$BURL")"
[[ "$bcode" == "201" || "$bcode" == "200" ]] || die "config bundle upload failed: HTTP $bcode (the binary IS published; re-run only the bundle upload)"
log "config bundle published (HTTP $bcode)"
# ── 6. Push the tag, now that the package exists ────────────────────────────────────────────────
# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v<version>/` fetch
# the installer makes. It runs last of the two so CI can never see a tag whose package is not there.
@@ -193,6 +208,11 @@ DL_SHA="$(sha256sum "$DL" | awk '{print $1}')"
[[ "$DL_SHA" == "$BUILT_SHA" ]] \
|| die "published sha $DL_SHA != built sha $BUILT_SHA — the artifact is not what was built"
BDL="$(mktemp -t felhom-config-bundle-dl-XXXXXX)"
trap 'rm -f "$BIN" "$DL" "$BUNDLE" "$BDL"' EXIT
curl -fsS -o "$BDL" "$BURL" || die "round-trip GET of the config bundle failed"
[[ "$(sha256sum "$BDL" | awk '{print $1}')" == "$BUNDLE_SHA" ]] || die "published config bundle sha != built sha"
# The tag must also serve the configs the installer will fetch from it.
cfg_code="$(curl -fsS -o /dev/null -w '%{http_code}' \
"$GITEA_BASE/$GITEA_OWNER/felhom-agent/raw/tag/$TAG/configs/felhom-agent.service" 2>/dev/null || true)"
@@ -206,6 +226,7 @@ cat <<EOF
version : $VERSION
tag : $TAG
sha256 : $BUILT_SHA
bundle : $BUNDLE_SHA (felhom-config-bundle.json — vouch it with the agent)
NOT VOUCHED. Vouching is what points machines at this version and stays your deliberate act:
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.