Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
ca78c17b29 | ||
|
admin
|
c8d12f1f2a | ||
|
admin
|
dc9164c5af | ||
|
admin
|
c9fa2e717b | ||
|
admin
|
0af1187e03 | ||
|
admin
|
495003051b | ||
|
admin
|
42af3ab9bc | ||
|
admin
|
f24dce5b95 |
@@ -1,3 +1,92 @@
|
||||
## v0.143.0 — the config bundle: a signed route for a box's root-owned files (R-840, decision 96) (2026-10-04)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `41c0d3060013dfda795262147454248149bee0888c0935170fdb85de6e7a35da`,
|
||||
config bundle `felhom-config-bundle.json` sha256 `8d7273cf5313ef62b867cb6f831c631923a436452d6f90b8ff7f0771170396ba`.
|
||||
|
||||
- **The bundle.** Every root-owned file the installer's step 5 writes (sudoers ×2, the five wrappers, the crash guard
|
||||
and its units, the agent and rollback units, the start-limit drop-in, the mgmt watchdog, the OOB belt's files) as ONE
|
||||
reproducible JSON file, built by `scripts/build-config-bundle.py` from `BUNDLE_FILES` in `configs/felhom-os-apply`
|
||||
(one table) and published beside the binary. The installer (1.31.0) installs the same file.
|
||||
- **The route.** A signed `agent_config_update` {agent_version, bundle_sha256} (`felhom-opsign -op agent_config_update
|
||||
-bundle-sha256 …`). The agent is the courier (downloads, checks the sha, hands over); `felhom-os-apply` mode `bundle`
|
||||
verifies it ITSELF: the operator signature against the root-owned `/etc/felhom/operator-signers` (or, when that file
|
||||
is missing, ONLY the installer's pinned key, after which it creates the file with exactly that key), the host binding,
|
||||
the window, its own nonce; the bundle sha; every path in `BUNDLE_FILES` (R16) and never a trust file (R17); every
|
||||
content check before the first write (visudo, sh/bash -n, python, unit sections, the RuntimeDirectory guard, User=,
|
||||
nft -c, and that the route itself survives). Atomic per file, previous copies kept under
|
||||
`/var/lib/felhom-os-apply/bundle-prev/`; a self-check after (visudo -c, `sudo -l` lists the route, the new wrapper's
|
||||
`--self-check`, the self-update wrapper's usage, the crash guard's status = kernel.panic); any failure puts every
|
||||
previous copy back. A newly installed crash guard is started (`enable --now`: kernel.panic for this boot, no reboot).
|
||||
Record `/etc/felhom/config-bundle.json`; the agent reports it as `system.config_bundle`, the facts mode adds drift.
|
||||
- **Bootstrap.** A box whose `felhom-os-apply` predates 0.143.0 cannot take the first bundle by the route (nothing on it
|
||||
can write a root file from a signed job): `felhom.eu/scripts/felhom-bundle-bootstrap.sh` is the one by-hand step.
|
||||
- **`build-golden.sh` 3.2.0** (not part of the binary): `GOLDEN_GUEST_PKGS` brings the template to exactly the
|
||||
approved guest release (only installed packages, never newer, never a removal or a new package) and prints the
|
||||
first-night count.
|
||||
- Tests: `configs/test_felhom_config_bundle.py` (43; 22 of 22 mutants red), `internal/osupdate/bundle_test.go`,
|
||||
`internal/hub/bundle_record_test.go`. Live: `felhom.eu/documentation/audits/r840-config-bundle-2026-10-04/partB/`.
|
||||
|
||||
## v0.142.1 — a Docker step no longer leaves the controller and traefik blind (R-858, `09` decision 95)
|
||||
|
||||
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.142.1` (`4950030`), sha256
|
||||
> `003f882a59abfc10021a1f97a4744ab78ab1e916b1392dcef132e7067f3c62bd`, verified by download. Not vouched at release time.
|
||||
|
||||
**MinAgent impact:** none. A same-day patch release, by the operator's ruling 95 after a live incident.
|
||||
|
||||
- **The incident.** v0.142.0's Docker step on demo-felhom (14:13 UTC) restarted dockerd, which recreates
|
||||
`/run/docker.sock`. live-restore kept every container running — including `felhom-controller` and `traefik`, which
|
||||
bind-mount the socket FILE and so kept the deleted inode. The controller could not reach Docker for 1 h 44 min (hub:
|
||||
DOWN, "docker not reachable"); its own health check stayed "healthy", so the step's health rule PASSED.
|
||||
- **The fix.** After a Docker step that installed something, the wrapper restarts ONLY the containers that mount
|
||||
`/var/run/docker.sock` or `/run/docker.sock` (`os-apply: SOCKET-USERS restarted=…`; the apps and the engine untouched).
|
||||
The guest health reading gains `controller_docker_ok` (`docker exec felhom-controller docker version`), and the health
|
||||
rule fails when it is false — the consequence, not the mechanism.
|
||||
- Proven live on demo-hp BEFORE the release (signed undo to 29.7.2): the two restarted, guest / controller / traefik on
|
||||
the same socket inode, healthy. Red-proofs: `felhom.eu/documentation/audits/os-docker-crash-2026-10-04/partE-incident/`.
|
||||
|
||||
## v0.142.0 — the Docker engine slow lane, the version report, the crash guard (`11` §5.8, §5.9; `09` decisions 87–89)
|
||||
|
||||
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.142.0` (`b1746c2`), sha256
|
||||
> `7beb32224d6495e9561acfd3ad8a48393a799520f196080011cb27fceb6d1de6`, verified by download. Not vouched at release time.
|
||||
|
||||
**MinAgent impact:** none. **Needs hub v0.132.0** for the System page, the Docker approval and the crash events; an older
|
||||
hub stores the new `system` stanza unread. **Needs the new root files** on an installed box (R-840): the wrapper,
|
||||
`/etc/felhom/os-trust.json`, `/etc/felhom/operator-signers` and the crash guard — the installer 1.30.0 writes them; the
|
||||
demo boxes got them by hand.
|
||||
|
||||
- **Docker `live-restore` ON** (decision 87). Wrapper mode `live-restore-on`: merge `"live-restore": true` into the
|
||||
guest's `/etc/docker/daemon.json` and `systemctl reload docker` — never a restart (R-835). An invalid daemon.json is
|
||||
left alone (R16); a reload that does not enable it puts the old file back. The leg runs it once before a Docker step;
|
||||
`--selftest=live-restore -vmid N` runs it by hand. Measured: demo-hp 24 containers, demo-felhom 5 — the same ids after.
|
||||
- **The Docker engine slow lane** (`11` §5.8). Wrapper layer `docker`, lane `slow` only, the six Docker packages only,
|
||||
origin `Docker CE` only (R2; a Docker package in a fast-lane plan is refused). **R3 — the wrapper checks the authority
|
||||
itself**, never the agent's config: a ring-1 step or ANY undo needs a signed `os_docker_step` it verifies with
|
||||
`ssh-keygen -Y verify` against the ROOT-owned `/etc/felhom/operator-signers` (namespace `felhom-op-v1`, the blob's
|
||||
`key_id`), bound to `/etc/felhom/os-trust.json` `host_id`, inside its time window, never replayed (a root-owned nonce
|
||||
file), with exactly the signed packages and undo flag; an unsigned ring-0 step needs that file's
|
||||
`"ring0_slow_lane": true` (the demo boxes only, set by hand). **R15:** live-restore must be on. An undo may downgrade
|
||||
(`--allow-downgrades`) only inside a signed job. The leg: ring 0 runs the Docker step at night after a healthy guest
|
||||
and host step (`select pending-docker`); ring 1 never does — only `DockerStepExecutor` (signed job, heavy-op gate).
|
||||
**Health:** the guest rule + every container running at the start has the SAME id after + the engine reports the
|
||||
installed version; a changed id is `health_failed`. Measured ring 0: 29.7.x → 29.8.2 on both demo boxes, every id kept.
|
||||
- **The version report** (R-852, decision 89). Wrapper mode `facts` (read-only): host Debian, running and next-boot
|
||||
kernel (`next_entry` > saved default > newest installed, by dpkg order), held packages (R-848), kernel taint (oops,
|
||||
warn), `kernel.panic`, the crash guard state; guest Debian, Docker engine, containerd, live-restore. The host report
|
||||
gains `system {pve_version, kernel_version, vmid, facts, facts_error}`, read at most every 10 min (~2 s);
|
||||
`--selftest=os-facts -vmid N`. A value nobody could read is `unknown`.
|
||||
- **R-849:** the guest is scanned for "restart needed" on every pass too, so a guest restart clears it.
|
||||
- **The crash guard** (decision 88, R-851): `configs/felhom-crash-guard` + `felhom-crash-guard.service` (early boot;
|
||||
its ExecStop writes a clean-stop marker) + an hourly re-arm timer + `/etc/felhom/crash-guard.conf`. A boot without
|
||||
the marker followed an unclean stop (a crash, a power cut or a hard reset — pstore saved nothing for a real panic on
|
||||
demo-hp, so they cannot be told apart). Armed: `kernel.panic = 10`. After the 2nd unclean boot within 60 min it
|
||||
TRIPS (`kernel.panic = 0`), so the 3rd crash within the hour leaves the box off; it re-arms after 24 h of normal
|
||||
running or `felhom-crash-guard rearm`. State in `/var/lib/felhom-crash-guard/state.json` (0644; read by facts).
|
||||
- **After the tag (main only, not shipped to boxes): `configs/build-golden.sh` 3.1.0** — the golden's `daemon.json`
|
||||
carries `"live-restore": true` with a fail-closed assertion, and `GOLDEN_DOCKER_PKGS` pins the approved Docker engine
|
||||
set (all six `name=version`); without it the bake log warns that the set is the newest, not an approved one.
|
||||
- Tests: wrapper 76 (DockerLane, LiveRestore, Facts, RealSignatureCheck with a throwaway key), crash guard 9, Go leg +
|
||||
executor; red-proofs `felhom.eu/documentation/audits/os-docker-crash-2026-10-04/partB/agent-redproofs.txt` (17 caught).
|
||||
|
||||
## v0.141.1 — "reboot needed" is true on the host (found live on demo-felhom, 2026-10-04)
|
||||
|
||||
> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.141.1` (`a6bc3f1`), sha256
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
|
||||
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
|
||||
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
|
||||
> `release-agent.sh` publishes it. A box whose `felhom-os-apply` predates 0.143.0 needs ONE by-hand bootstrap
|
||||
> (`felhom.eu/scripts/felhom-bundle-bootstrap.sh`) — done on both demo boxes; Tester 2 waits for the operator (R-862).
|
||||
> Both demo boxes: agent 0.143.0, bundle 0.143.0 (record `/etc/felhom/config-bundle.json`). R-861 found: the sudoers is
|
||||
> root-equivalent. `build-golden.sh` 3.2.0 (`GOLDEN_GUEST_PKGS`). Runbook `felhom.eu/documentation/runbooks/config-bundle.md`.
|
||||
|
||||
|
||||
> **2026-09-25 night — v0.133.0 AND v0.134.0 DELIVERED to both demo boxes (CC-signed `agent_update`, ruling 1);
|
||||
> restore test back ON (the `-1` config kept as `agent.json.night-0925-off`). v0.134.0 = R-685:** `backup/runner.go`
|
||||
|
||||
@@ -1,11 +1,19 @@
|
||||
# REPORT — 2026-10-04: v0.141.0 + v0.141.1, the host fast lane, the true tunnel status, the fast leg
|
||||
# REPORT — agent v0.143.0: the config bundle (R-840) — 2026-10-04
|
||||
|
||||
Full session report: `felhom.eu/REPORT-os-host-lane-2026-10-04.md`.
|
||||
**What:** a signed route for a box's root-owned files. `felhom-os-apply` gained mode `bundle` (signed
|
||||
`agent_config_update`, verified by the wrapper itself) and `--install-bundle` (the installer's root entry);
|
||||
`BUNDLE_FILES` is the one table of 22 paths; `scripts/build-config-bundle.py` builds it reproducibly;
|
||||
`release-agent.sh` publishes it beside the binary; the agent reports `system.config_bundle`; `felhom-opsign` signs it.
|
||||
Also `build-golden.sh` 3.2.0 (`GOLDEN_GUEST_PKGS`).
|
||||
|
||||
- Tunnel (R-841): the agent reads the guest's cloudflared container and its health check; three states.
|
||||
- Host fast lane: the wrapper gains the host layer (R12 appliance proof from the root-owned install record, R14 no
|
||||
kernel/boot/firmware); the leg runs the host step after a healthy guest step; host health rule.
|
||||
- Speed (R-845): one call per layer instead of one per package; measured before/after in the session report.
|
||||
- Tests green; red-proofs in the audit folder.
|
||||
- v0.141.1 (same day): the host "reboot needed" scan hid `lxc-start` and was never cleared by a reboot — both fixed,
|
||||
found live on demo-felhom, red-proved.
|
||||
**Released:** v0.143.0, binary `41c0d306…`, bundle `8d7273cf…` (reproducible), vouched in the hub with golden 0.293.0.
|
||||
**Delivered:** demo-hp and demo-felhom (signed `agent_update`), then the one-file bootstrap, then the bundle by the
|
||||
signed route (0 written of 22; probe 71/71). demo-hp also: a wrong sha refused, a one-line change and its undo, a replay
|
||||
rejected. Tester 2: the signed `agent_update` queued (operator ruling 97); its bundle waits for the operator's
|
||||
bootstrap (R-862).
|
||||
|
||||
**Tests:** `configs/test_felhom_config_bundle.py` 43 (22 of 22 mutants red), Go `internal/osupdate/bundle_test.go`,
|
||||
`internal/hub/bundle_record_test.go`; `go vet ./... && go test ./...` rc=0; gates green.
|
||||
**Found:** R-861 — the sudoers already lets the agent user reach root (read, not exploited).
|
||||
Evidence and the full report: `felhom.eu/documentation/audits/r840-config-bundle-2026-10-04/`,
|
||||
`felhom.eu/REPORT-r840-config-bundle-2026-10-04.md`.
|
||||
|
||||
@@ -15,6 +15,10 @@
|
||||
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
|
||||
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
|
||||
| `stageTemp` | internal/localapi/intermediary.go | `stageTemp(pattern, content) (path, err)` | random-named temp before a root `install` (audit B1) | Fixed /tmp names are a TOCTOU — sudoers globs expect `/tmp/felhom-*-*.ext` |
|
||||
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
|
||||
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
|
||||
| `osupdate.Leg.SendUnsent` / `lockPass` (v0.144.0, R-868) | internal/osupdate/unsent.go | `(ctx) int` | an OS-pass report the agent never sent (killed mid-pass): the wrapper keeps `report-<run>-<layer>-apply.json` beside the plan; the agent deletes it once the hub has it | any new caller that runs an apply pass must hold `lockPass` (flock, across processes) — the sender must never take a running pass's copy |
|
||||
| `osupdate.LoadSavedBlock` (v0.144.0, R-866) | internal/osupdate/leg.go | `(planDir) (block, savedAt, ok)` | the hub's newest os_update block as the daemon last received it (`os-update-block.json`) | the debug pass uses it ONLY when the hub cannot be reached, and says so in its header; no saved block → no pass |
|
||||
| `guesthook.InstallSnippet` / `Register` | internal/guesthook/install.go | `InstallSnippet(ctx, runner) error` | pre-start self-heal hook install (C1 net) | Same random-temp+install pattern; snippet delegates to the agent binary (no shell logic). Issues `mkdir -p /var/lib/vz/snippets` FIRST (v0.63.0, B2 — fresh boxes lack the dir; sudoers grants exactly that argv) |
|
||||
|
||||
### Disk / format safety (role gates, durable IDs, format guards)
|
||||
|
||||
@@ -868,6 +868,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
logger.Info("felhom-agent daemon starting",
|
||||
"version", version, "host_id", cfg.Hub.HostID, "hub_url", cfg.Hub.URL,
|
||||
"interval_s", hcfg.PollSeconds) // hub key intentionally not logged
|
||||
// R-868 (v0.144.0): an OS pass whose agent was killed kept its report on disk — send it now (a pass that starts
|
||||
// first sends it itself; the pass lock keeps the two apart).
|
||||
go func() {
|
||||
if n := osLeg.SendUnsent(ctx); n > 0 {
|
||||
logger.Info("osupdate: sent kept report(s) at start", "count", n)
|
||||
}
|
||||
}()
|
||||
|
||||
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
|
||||
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
|
||||
@@ -1101,7 +1108,18 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
return release, nil
|
||||
}}
|
||||
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec}, cfg.Hub.HostID, logger)
|
||||
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
|
||||
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
|
||||
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
|
||||
AfterInstall: func(ctx context.Context) {
|
||||
ok, total, degraded := capability.Summarize(probeAll(ctx))
|
||||
names := make([]string, 0, len(degraded))
|
||||
for _, d := range degraded {
|
||||
names = append(names, d.Name)
|
||||
}
|
||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||
}}
|
||||
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, bundleExec}, cfg.Hub.HostID, logger)
|
||||
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
|
||||
|
||||
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
|
||||
@@ -3564,15 +3582,15 @@ func runSelftestOSUpdate(ctx context.Context, cfg config.Config, logger *slog.Lo
|
||||
return 1
|
||||
}
|
||||
leg := newOSLeg(cfg, client, px, logger)
|
||||
resp, err := client.FetchDesiredState(ctx)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest=os-update: desired state:", err)
|
||||
blk, source, ok := selftestOSBlock(ctx, client, leg.PlanDir)
|
||||
if !ok {
|
||||
fmt.Fprintln(os.Stderr, "selftest=os-update:", source)
|
||||
return 1
|
||||
}
|
||||
leg.SetBlock(resp.DesiredState.OSUpdate)
|
||||
leg.SetBlock(blk)
|
||||
b := leg.Block()
|
||||
fmt.Printf("=== felhom-agent %s selftest=os-update vmid=%d ring=%d enabled=%v guest-release=%v host-release=%v appliance=%v ===\n",
|
||||
version, vmid, b.Ring, b.Enabled, b.Release != nil, b.HostRelease != nil, leg.Appliance)
|
||||
fmt.Printf("=== felhom-agent %s selftest=os-update vmid=%d ring=%d enabled=%v guest-release=%v host-release=%v appliance=%v block=%s ===\n",
|
||||
version, vmid, b.Ring, b.Enabled, b.Release != nil, b.HostRelease != nil, leg.Appliance, source)
|
||||
start := time.Now()
|
||||
pass := leg.Run(ctx, vmid, "debug")
|
||||
worst := pass.Guest
|
||||
@@ -3598,6 +3616,24 @@ func runSelftestOSUpdate(ctx context.Context, cfg config.Config, logger *slog.Lo
|
||||
return 1
|
||||
}
|
||||
|
||||
// selftestOSBlock is the debug pass's os_update block (R-866, v0.144.0): the hub's, fetched fresh; when the hub cannot
|
||||
// be reached, the block the daemon last saved — named in the selftest's first line, so a pass with the hub away can
|
||||
// be exercised by hand (the daemon's own leg already ran from its last block; the selftest stopped). No saved block
|
||||
// and no hub → not run (ok=false), never a guessed block.
|
||||
func selftestOSBlock(ctx context.Context, f interface {
|
||||
FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error)
|
||||
}, planDir string) (*hub.WireOSUpdate, string, bool) {
|
||||
resp, err := f.FetchDesiredState(ctx)
|
||||
if err == nil {
|
||||
return resp.DesiredState.OSUpdate, "hub", true
|
||||
}
|
||||
saved, at, ok := osupdate.LoadSavedBlock(planDir)
|
||||
if !ok {
|
||||
return nil, fmt.Sprintf("desired state: %v — and no saved block (the daemon saves one when the hub sends it)", err), false
|
||||
}
|
||||
return saved, fmt.Sprintf("SAVED(%s; hub unreachable: %v)", at.UTC().Format(time.RFC3339), err), true
|
||||
}
|
||||
|
||||
// runSelftestFacts prints the versions the host report carries (R-852, agent v0.142.0) — read-only.
|
||||
//
|
||||
// sudo -u felhom-agent felhom-agent --config … --selftest=os-facts -vmid 9201
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
|
||||
)
|
||||
|
||||
type r866Fetcher struct{ err error }
|
||||
|
||||
func (f r866Fetcher) FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error) {
|
||||
if f.err != nil {
|
||||
return nil, f.err
|
||||
}
|
||||
r := &hub.DesiredStateResponse{}
|
||||
r.DesiredState.OSUpdate = &hub.WireOSUpdate{Ring: 1, Enabled: true}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// R-866 (v0.144.0). THE NIGHT'S SHAPE (A3, Tester 1 box, hub blocked): `selftest=os-update: desired state: hub:
|
||||
// transport error … connect: invalid argument` — the debug pass could not run at all. Now it runs from the block the
|
||||
// daemon saved, and its header says so.
|
||||
// COMPANION RED-PROOF: return at once on a fetch error in selftestOSBlock → "the pass did not run from the saved block".
|
||||
func TestR866_DebugPassUsesTheSavedBlockWhenTheHubIsAway(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
leg := &osupdate.Leg{PlanDir: dir}
|
||||
r := &hub.DesiredStateResponse{}
|
||||
r.DesiredState.OSUpdate = &hub.WireOSUpdate{Ring: 0, Enabled: true}
|
||||
leg.OnDesiredState(context.Background(), r) // the daemon received a block and saved it
|
||||
away := r866Fetcher{err: errors.New("hub: transport error: connect: invalid argument")}
|
||||
b, src, ok := selftestOSBlock(context.Background(), away, dir)
|
||||
if !ok || b == nil || b.Ring != 0 {
|
||||
t.Fatalf("the pass did not run from the saved block: ok=%v block=%+v src=%q", ok, b, src)
|
||||
}
|
||||
if !strings.HasPrefix(src, "SAVED(") || !strings.Contains(src, "hub unreachable") {
|
||||
t.Fatalf("the header must say the block is the saved one: %q", src)
|
||||
}
|
||||
// the hub reachable: its block wins, and the header says "hub"
|
||||
b, src, ok = selftestOSBlock(context.Background(), r866Fetcher{}, dir)
|
||||
if !ok || b.Ring != 1 || src != "hub" {
|
||||
t.Fatalf("hub block not used: %+v %q", b, src)
|
||||
}
|
||||
}
|
||||
|
||||
// No hub and nothing saved: the pass does not run on a guessed block.
|
||||
func TestR866_NoHubNoSavedBlockDoesNotRun(t *testing.T) {
|
||||
_, why, ok := selftestOSBlock(context.Background(), r866Fetcher{err: errors.New("down")}, t.TempDir())
|
||||
if ok || !strings.Contains(why, "no saved block") {
|
||||
t.Fatalf("ok=%v why=%q", ok, why)
|
||||
}
|
||||
}
|
||||
@@ -43,7 +43,7 @@ func main() {
|
||||
|
||||
func run() error {
|
||||
var (
|
||||
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update")
|
||||
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | agent_config_update")
|
||||
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
|
||||
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
|
||||
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
|
||||
@@ -52,6 +52,7 @@ func run() error {
|
||||
fstype = flag.String("fstype", "ext4", "for storage_wipe: the filesystem to mkfs after wipe")
|
||||
agentVer = flag.String("agent-version", "", "for agent_update: the target agent version (e.g. 0.70.1)")
|
||||
sha256Hex = flag.String("sha256", "", "for agent_update: the pinned lowercase-hex sha256 of the target binary")
|
||||
bundleSHA = flag.String("bundle-sha256", "", "for agent_config_update: the pinned sha256 of felhom-config-bundle.json (R-840)")
|
||||
keyFile = flag.String("key", "", "operator signing key (ssh private key / sk- key handle) for ssh-keygen -Y sign")
|
||||
ttl = flag.Duration("ttl", 30*time.Minute, "validity window from now (issued_at..expires_at)")
|
||||
nonce = flag.String("nonce", "", "explicit nonce (default: a fresh 128-bit random nonce)")
|
||||
@@ -95,6 +96,17 @@ func run() error {
|
||||
}
|
||||
pj, _ := json.Marshal(map[string]string{"version": *agentVer, "sha256": *sha256Hex})
|
||||
params = string(pj)
|
||||
case "agent_config_update":
|
||||
// R-840: the box's root-owned files. The ROOT wrapper verifies this signature itself and refuses a bundle
|
||||
// whose sha256 is not exactly this one.
|
||||
if *agentVer == "" || *bundleSHA == "" {
|
||||
return fmt.Errorf("agent_config_update needs -agent-version and -bundle-sha256 (the pinned bundle hash)")
|
||||
}
|
||||
if !isHex64(*bundleSHA) {
|
||||
return fmt.Errorf("agent_config_update -bundle-sha256 must be 64 lowercase hex chars (got %d)", len(*bundleSHA))
|
||||
}
|
||||
pj, _ := json.Marshal(map[string]string{"agent_version": *agentVer, "bundle_sha256": *bundleSHA})
|
||||
params = string(pj)
|
||||
default:
|
||||
params = "{}"
|
||||
}
|
||||
|
||||
+62
-2
@@ -61,7 +61,7 @@ set -euo pipefail
|
||||
|
||||
# Script provenance — logged into every bake transcript next to the baked controller tag, so an
|
||||
# archive can always be traced to the script that produced it. Bump on any behavior change.
|
||||
GOLDEN_SCRIPT_VERSION="3.0.0"
|
||||
GOLDEN_SCRIPT_VERSION="3.2.0"
|
||||
|
||||
VMID="${1:-9100}"
|
||||
TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
|
||||
@@ -112,7 +112,15 @@ for i in $(seq 1 30); do
|
||||
if pct exec "$VMID" -- getent hosts download.docker.com >/dev/null 2>&1; then break; fi
|
||||
sleep 1
|
||||
done
|
||||
pct exec "$VMID" -- bash -c '
|
||||
# v3.1.0 (`11` §5.8): GOLDEN_DOCKER_PKGS pins the APPROVED Docker engine set (the hub's newest Docker release, all six
|
||||
# "name=version"); without it the newest stable set is installed and the bake log says so.
|
||||
GOLDEN_DOCKER_PKGS="${GOLDEN_DOCKER_PKGS:-}"
|
||||
if [[ -n "$GOLDEN_DOCKER_PKGS" ]]; then
|
||||
echo "[golden] Docker engine set PINNED to the approved release: $GOLDEN_DOCKER_PKGS"
|
||||
else
|
||||
echo "[golden] WARNING: GOLDEN_DOCKER_PKGS not set — installing the newest stable Docker set, not an approved one"
|
||||
fi
|
||||
pct exec "$VMID" -- env GOLDEN_DOCKER_PKGS="$GOLDEN_DOCKER_PKGS" bash -c '
|
||||
set -e
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
@@ -122,7 +130,54 @@ pct exec "$VMID" -- bash -c '
|
||||
echo "deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable" \
|
||||
> /etc/apt/sources.list.d/docker.list
|
||||
apt-get update -qq
|
||||
if [ -n "$GOLDEN_DOCKER_PKGS" ]; then
|
||||
apt-get install -y -qq $GOLDEN_DOCKER_PKGS >/dev/null
|
||||
else
|
||||
apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
|
||||
fi
|
||||
dpkg-query -W containerd.io docker-buildx-plugin docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin 2>/dev/null | sed "s/^/ installed: /"
|
||||
'
|
||||
# v3.2.0 (Part F of the R-840 brief, `11` §5.3): GOLDEN_GUEST_PKGS = the newest APPROVED guest release, as
|
||||
# "name=version …" (the hub's os_releases row for layer guest, IN FORCE — never a cancelled test approval). The bake
|
||||
# brings every package the template HAS to exactly that version, under felhom-os-apply's rules: never a package the
|
||||
# template lacks (--only-upgrade), never newer than approved, never a removal or a new package (a simulation is checked
|
||||
# first and the bake FAILS on either). Empty = no approved guest release in force: the template's versions stay, and
|
||||
# the box's first night installs whatever release is approved then. Either way the bake PRINTS the first-night count:
|
||||
# how many installed packages are older than the approved version (target 0).
|
||||
GOLDEN_GUEST_PKGS="${GOLDEN_GUEST_PKGS:-}"
|
||||
pct exec "$VMID" -- env GOLDEN_GUEST_PKGS="$GOLDEN_GUEST_PKGS" bash -c '
|
||||
set -e
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
if [ -z "$GOLDEN_GUEST_PKGS" ]; then
|
||||
echo "[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a"
|
||||
echo "[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): $(apt list --upgradable 2>/dev/null | grep -c /)"
|
||||
exit 0
|
||||
fi
|
||||
want=""
|
||||
for nv in $GOLDEN_GUEST_PKGS; do
|
||||
n=${nv%%=*}; v=${nv#*=}
|
||||
cur=$(dpkg-query -W -f="\${Version}" "$n" 2>/dev/null) || continue # not in the template: never added
|
||||
dpkg --compare-versions "$cur" lt "$v" && want="$want $n=$v"
|
||||
done
|
||||
if [ -n "$want" ]; then
|
||||
sim=$(apt-get -s install --only-upgrade -o Dpkg::Options::=--force-confold $want)
|
||||
if echo "$sim" | grep -q "^Remv "; then echo "[golden] FATAL: the approved guest set would REMOVE a package"; echo "$sim" | grep "^Remv "; exit 1; fi
|
||||
for p in $(echo "$sim" | awk "/^Inst /{print \$2}"); do
|
||||
dpkg-query -W "$p" >/dev/null 2>&1 || { echo "[golden] FATAL: the approved guest set would ADD $p - not in the template"; exit 1; }
|
||||
done
|
||||
apt-get install -y -qq --only-upgrade -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef $want >/dev/null
|
||||
echo "[golden] approved guest release installed: $(echo $want | wc -w) package(s) brought to the approved version"
|
||||
else
|
||||
echo "[golden] approved guest release: the template already runs every approved version"
|
||||
fi
|
||||
left=0
|
||||
for nv in $GOLDEN_GUEST_PKGS; do
|
||||
n=${nv%%=*}; v=${nv#*=}
|
||||
cur=$(dpkg-query -W -f="\${Version}" "$n" 2>/dev/null) || continue
|
||||
dpkg --compare-versions "$cur" lt "$v" && { left=$((left+1)); echo " still older: $n $cur < $v"; }
|
||||
done
|
||||
echo "[golden] first-night count vs the approved guest release: $left (target 0)"
|
||||
[ "$left" -eq 0 ] || { echo "[golden] FATAL: $left package(s) stayed older than the approved release"; exit 1; }
|
||||
'
|
||||
echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …"
|
||||
# containerd-snapshotter (Docker 28+/29 default) keeps the IMAGE content store under
|
||||
@@ -135,9 +190,12 @@ echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshott
|
||||
# layout (a container's `df /` reports the single volume, phase-0 spike). Since v3.0.0 /var/lib/docker
|
||||
# is a BIND of <volume>/docker rather than the mp0 mount itself, wired immediately below; data-root
|
||||
# still needs no override because the path is unchanged. Log caps kill the most common runaway.
|
||||
# v3.1.0: "live-restore": true (`09` decision 87) — a Docker engine update then restarts no app (`11` C5). A box made
|
||||
# from this golden never needs the agent's one-time live-restore-on step. NEVER removed by a plain restart (R-835).
|
||||
pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
|
||||
{
|
||||
"features": { "containerd-snapshotter": false },
|
||||
"live-restore": true,
|
||||
"log-driver": "json-file",
|
||||
"log-opts": { "max-size": "10m", "max-file": "3" }
|
||||
}
|
||||
@@ -175,6 +233,8 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
|
||||
# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
|
||||
# mean containerd-snapshotter is still on (the split would leave images on the rootfs).
|
||||
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
|
||||
# v3.1.0 ASSERTION: live-restore is ON in the running daemon (decision 87), or the bake fails closed.
|
||||
pct exec "$VMID" -- bash -c 'lr=$(docker info --format "{{.LiveRestoreEnabled}}" 2>/dev/null); [ "$lr" = "true" ] && echo " live-restore: on" || { echo "[golden] FATAL: live-restore is $lr, expected true (decision 87)"; exit 1; }'
|
||||
# ASSERTION 1 (RETARGETED v3.0.0, not removed). /var/lib/docker must be a real mount — now the V-c
|
||||
# bind of <volume>/docker rather than the mp0 mount itself. Still fails closed on the same failure:
|
||||
# if the bind did not take, Docker's data-root silently sits on the OS rootfs and the golden ships
|
||||
|
||||
+570
-11
@@ -86,8 +86,82 @@ SIG_NAMESPACE = "felhom-op-v1"
|
||||
SIGNED_OP = "os_docker_step"
|
||||
NONCE_FILE = "/var/lib/felhom-os-apply/nonces.json"
|
||||
DAEMON_JSON = "/etc/docker/daemon.json"
|
||||
# R-858 (v0.142.1): a Docker engine step restarts dockerd, which RECREATES the socket file. With live-restore the
|
||||
# containers keep running — and one that bind-mounts the socket FILE keeps the deleted inode: measured 2026-10-04 on
|
||||
# demo-felhom, the controller and traefik were blind to Docker for 1h44m. After a step that installed something, the
|
||||
# wrapper restarts exactly the containers that mount one of these paths (never the apps, never the engine).
|
||||
DOCKER_SOCKETS = ("/var/run/docker.sock", "/run/docker.sock")
|
||||
CRASH_GUARD_STATE = "/var/lib/felhom-crash-guard/state.json"
|
||||
|
||||
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
|
||||
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
|
||||
# repo's configs/ at the agent tag (scripts/build-config-bundle.py) and published beside the binary. The installer
|
||||
# installs the SAME bundle through this same code (--install-bundle, root only, never reachable through sudo), so a new
|
||||
# box and an updated box cannot drift. This table is the ONLY set of paths a bundle may write — a signed bundle naming
|
||||
# any other path is refused (R16) — and it is also the builder's list (one table). Each entry:
|
||||
# dest: (source file under configs/, mode, check, policy)
|
||||
# check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
|
||||
# unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
|
||||
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none.
|
||||
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
|
||||
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
|
||||
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
|
||||
BUNDLE_FORMAT = 1
|
||||
BUNDLE_OP = "agent_config_update"
|
||||
BUNDLE_RECORD = "/etc/felhom/config-bundle.json" # what the box runs (0644 root; the non-root agent reports it)
|
||||
BUNDLE_PREV_DIR = "/var/lib/felhom-os-apply/bundle-prev" # the previous copies of every file a bundle replaced
|
||||
BUNDLE_KEEP = 3
|
||||
BUNDLE_MAX_BYTES = 4 * 1024 * 1024
|
||||
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
|
||||
OOB_DIR = "/etc/felhom-sshd"
|
||||
BUNDLE_FILES = [
|
||||
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
|
||||
("/usr/local/sbin/felhom-pbs-apply", "felhom-pbs-apply", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-backup-target-apply", "felhom-backup-target-apply", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-os-apply", "felhom-os-apply", 0o755, "python", "replace"),
|
||||
("/usr/local/sbin/felhom-crash-guard", "felhom-crash-guard", 0o755, "python", "replace"),
|
||||
("/etc/systemd/system/felhom-crash-guard.service", "felhom-crash-guard.service", 0o644, "unit", "replace"),
|
||||
("/etc/systemd/system/felhom-crash-guard-check.service", "felhom-crash-guard-check.service", 0o644, "unit", "replace"),
|
||||
("/etc/systemd/system/felhom-crash-guard-check.timer", "felhom-crash-guard-check.timer", 0o644, "unit", "replace"),
|
||||
("/etc/felhom/crash-guard.conf", "crash-guard.conf", 0o644, "plain", "if-absent"),
|
||||
("/etc/systemd/system/felhom-agent.service", "felhom-agent.service", 0o644, "agent-unit", "replace"),
|
||||
("/etc/systemd/system/felhom-agent-rollback.service", "felhom-agent-rollback.service", 0o644, "unit", "replace"),
|
||||
("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", "felhom-agent-limits.conf", 0o644, "dropin", "replace"),
|
||||
("/usr/local/sbin/felhom-mgmt-watchdog", "felhom-mgmt-watchdog.sh", 0o755, "sh", "replace"),
|
||||
("/etc/tmpfiles.d/felhom-privsep.conf", "felhom-privsep.tmpfiles", 0o644, "plain", "replace"),
|
||||
("/etc/systemd/system/felhom-mgmt-watchdog.service", "felhom-mgmt-watchdog.service", 0o644, "unit-nort", "replace"),
|
||||
("/etc/systemd/system/felhom-mgmt-watchdog.timer", "felhom-mgmt-watchdog.timer", 0o644, "unit-nort", "replace"),
|
||||
("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
|
||||
("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
|
||||
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
|
||||
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
|
||||
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
|
||||
]
|
||||
BUNDLE_DESTS = {e[0]: e for e in BUNDLE_FILES}
|
||||
# The trust root is never a bundle's to change (R17): who may sign is decided by these files, so no bundle may carry them.
|
||||
TRUST_PATHS = (TRUST_FILE, TRUST_SIGNERS, BUNDLE_RECORD)
|
||||
# When /etc/felhom/operator-signers is MISSING (a box installed before installer 1.30.0), the job is verified against —
|
||||
# and the file is created with — exactly this key: the operational key felhom-host-install.sh pins
|
||||
# (OPERATOR_KEY_OPERATIONAL_*). Never any other. Pinned equal to the installer by
|
||||
# test_pinned_operator_key_equals_the_installers. Signer rotation is a separate, later act (`04` §3).
|
||||
PINNED_OPERATOR_KEY_ID = "felhom-op-1"
|
||||
PINNED_OPERATOR_KEY_LINE = ("ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL8z0qCNgA3x2xxAB0Qj5ro8waFjGZ8Ta/sWB63tlLw+ felhom-op-1")
|
||||
# The self-check (after install, before the record): the route itself must survive — a bundle whose sudoers no longer
|
||||
# lets the agent call this wrapper would cut the box off from every later bundle.
|
||||
SELF_CHECK_SUDO_LINE = "/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json"
|
||||
PINNED_SIGNERS = "<pinned operator key>" # verify_sig: check against the pinned key, not a file on the box
|
||||
|
||||
|
||||
def pinned_signers_line():
|
||||
"""The ssh allowed_signers line the installer writes (felhom-host-install.sh _write_slow_lane_trust), byte for byte."""
|
||||
return f'{PINNED_OPERATOR_KEY_ID} namespaces="{SIG_NAMESPACE}" {PINNED_OPERATOR_KEY_LINE}\n'
|
||||
|
||||
|
||||
def sha256_hex(data):
|
||||
import hashlib
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
class Refused(Exception):
|
||||
def __init__(self, code, reason):
|
||||
@@ -126,6 +200,32 @@ class Runner:
|
||||
if rc != 0:
|
||||
raise Refused("R7", f"could not write {path} in the guest")
|
||||
|
||||
def save_report(self, plan_path, report):
|
||||
"""R-868: keep an apply pass's report on disk until the agent has sent it (the agent deletes it). Written
|
||||
INTO the agent's own plan dir as root, so: the dir is opened with O_NOFOLLOW and must be a real directory
|
||||
owned by the agent (a symlink swapped in for it is refused); the file is created O_EXCL|O_NOFOLLOW after
|
||||
removing an old one, then handed to the agent (0600). Any failure only loses the copy — never the run."""
|
||||
base = os.path.basename(plan_path)
|
||||
name = "report-" + base[len("plan-"):]
|
||||
dfd = os.open(PLAN_DIR, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
|
||||
try:
|
||||
st = os.fstat(dfd)
|
||||
if not stat.S_ISDIR(st.st_mode) or st.st_uid != self.agent_uid():
|
||||
raise OSError(f"{PLAN_DIR} is not the agent's own directory")
|
||||
try:
|
||||
os.unlink(name, dir_fd=dfd)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
fd = os.open(name, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600, dir_fd=dfd)
|
||||
try:
|
||||
os.write(fd, (json.dumps(report, sort_keys=True) + "\n").encode())
|
||||
os.fchown(fd, self.agent_uid(), -1)
|
||||
finally:
|
||||
os.close(fd)
|
||||
finally:
|
||||
os.close(dfd)
|
||||
return os.path.join(PLAN_DIR, name)
|
||||
|
||||
def now(self):
|
||||
return time.time()
|
||||
|
||||
@@ -139,6 +239,10 @@ class Runner:
|
||||
sp = os.path.join(d, "sig")
|
||||
with open(sp, "w") as f:
|
||||
f.write(sig)
|
||||
if signers == PINNED_SIGNERS:
|
||||
signers = os.path.join(d, "allowed_signers")
|
||||
with open(signers, "w") as f:
|
||||
f.write(pinned_signers_line())
|
||||
p = subprocess.run(["ssh-keygen", "-Y", "verify", "-f", signers, "-I", key_id, "-n", namespace, "-s", sp],
|
||||
input=blob, capture_output=True, timeout=30)
|
||||
return p.returncode
|
||||
@@ -165,6 +269,62 @@ class Runner:
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ---------- host files, for the config bundle (R-840). Tests replace these with an in-memory tree. ----------
|
||||
def read_bytes(self, path):
|
||||
with open(path, "rb") as f:
|
||||
return f.read()
|
||||
|
||||
def lexists(self, path):
|
||||
return os.path.lexists(path)
|
||||
|
||||
def isdir(self, path):
|
||||
return os.path.isdir(path)
|
||||
|
||||
def put_file(self, path, data, mode):
|
||||
"""Atomic: a root-owned temp file beside the target (same filesystem), fsync, then rename over it."""
|
||||
d = os.path.dirname(path)
|
||||
os.makedirs(d, mode=0o755, exist_ok=True)
|
||||
tmp = os.path.join(d, f".{os.path.basename(path)}.felhom-new.{os.getpid()}")
|
||||
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
try:
|
||||
with os.fdopen(fd, "wb") as f:
|
||||
f.write(data)
|
||||
f.flush()
|
||||
os.fchown(f.fileno(), 0, 0)
|
||||
os.fchmod(f.fileno(), mode)
|
||||
os.fsync(f.fileno())
|
||||
os.replace(tmp, path)
|
||||
except BaseException:
|
||||
try:
|
||||
os.remove(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
def remove(self, path):
|
||||
os.remove(path)
|
||||
|
||||
def list_dir(self, path):
|
||||
try:
|
||||
return sorted(os.listdir(path))
|
||||
except OSError:
|
||||
return []
|
||||
|
||||
def rmtree(self, path):
|
||||
import shutil
|
||||
shutil.rmtree(path, ignore_errors=True)
|
||||
|
||||
def check_content(self, kind, data):
|
||||
"""Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
|
||||
import tempfile
|
||||
cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"]}[kind]
|
||||
with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
|
||||
p = os.path.join(d, "f")
|
||||
with open(p, "wb") as f:
|
||||
f.write(data)
|
||||
r = subprocess.run(cmd + [p], capture_output=True, text=True, timeout=60)
|
||||
return r.returncode, (r.stdout + r.stderr).strip()[-300:]
|
||||
|
||||
|
||||
class Apply:
|
||||
def __init__(self, runner, plan_path):
|
||||
@@ -198,8 +358,12 @@ class Apply:
|
||||
|
||||
def check_plan(self, plan):
|
||||
mode = plan.get("mode", "apply")
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on"):
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle"):
|
||||
raise Refused("R11", f"unknown mode {mode!r}")
|
||||
if mode == "bundle":
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "bundle is a host-layer mode")
|
||||
return mode, "host", 0, "bundle"
|
||||
layer = plan.get("layer")
|
||||
if layer not in ("guest", "host", "docker"):
|
||||
raise Refused("R12", f"layer {layer!r} is not guest, host or docker")
|
||||
@@ -293,9 +457,12 @@ class Apply:
|
||||
raise Refused("R3", f"{TRUST_FILE} names no host_id")
|
||||
return t
|
||||
|
||||
def verify_signed(self, signed, trust):
|
||||
"""R3: an operator-signed os_docker_step, checked HERE (not by the agent): signature against the root-owned
|
||||
signers file, op, host binding, time window, and a root-owned nonce record (no replay)."""
|
||||
def verify_signed(self, signed, trust, op_name=SIGNED_OP, signers=None, burn=True):
|
||||
"""R3: an operator-signed job, checked HERE (not by the agent): signature against the root-owned signers file (or,
|
||||
for a bundle on a box that has none, the PINNED key — `signers` names that temp file), op, host binding, time
|
||||
window, and a root-owned nonce record (no replay). burn=False leaves the nonce for the caller to burn after its
|
||||
own checks (a bundle refused for a wrong sha keeps its job usable — the operator fixes the sha, not the key).
|
||||
Returns the params; with burn=False, (params, nonce, expiry)."""
|
||||
import base64
|
||||
if not isinstance(signed, dict) or not isinstance(signed.get("blob_b64"), str) or not isinstance(signed.get("sig"), str):
|
||||
raise Refused("R3", "the signed job is malformed")
|
||||
@@ -307,17 +474,19 @@ class Apply:
|
||||
key_id = op.get("key_id", "")
|
||||
if not isinstance(key_id, str) or not re.match(r"^[A-Za-z0-9._-]{1,64}$", key_id):
|
||||
raise Refused("R3", "the signed blob names no plain key_id")
|
||||
if signers is None:
|
||||
signers = TRUST_SIGNERS
|
||||
try:
|
||||
st = self.r.stat(TRUST_SIGNERS)
|
||||
except OSError:
|
||||
raise Refused("R3", f"no {TRUST_SIGNERS} — no operator key to check a signed job against")
|
||||
if st.st_uid != 0 or (st.st_mode & 0o022):
|
||||
raise Refused("R3", f"{TRUST_SIGNERS} is not root-owned and root-only-writable")
|
||||
rc = self.r.verify_sig(TRUST_SIGNERS, key_id, SIG_NAMESPACE, blob, signed["sig"])
|
||||
rc = self.r.verify_sig(signers, key_id, SIG_NAMESPACE, blob, signed["sig"])
|
||||
if rc != 0:
|
||||
raise Refused("R3", f"the operator signature does not verify (ssh-keygen rc={rc})")
|
||||
if op.get("op") != SIGNED_OP:
|
||||
raise Refused("R3", f"the signed op is {op.get('op')!r}, not {SIGNED_OP}")
|
||||
if op.get("op") != op_name:
|
||||
raise Refused("R3", f"the signed op is {op.get('op')!r}, not {op_name}")
|
||||
if (op.get("target") or {}).get("host_id") != trust["host_id"]:
|
||||
raise Refused("R3", "the signed job is for another host")
|
||||
now = self.r.now()
|
||||
@@ -331,12 +500,20 @@ class Apply:
|
||||
nonce = op.get("nonce")
|
||||
if not isinstance(nonce, str) or not nonce:
|
||||
raise Refused("R3", "the signed job has no nonce")
|
||||
if nonce in self.r.read_nonces():
|
||||
raise Refused("R3", "the signed job was already used (replay)")
|
||||
if not burn:
|
||||
return op.get("params") or {}, nonce, exp
|
||||
self.burn_nonce(nonce, exp)
|
||||
return op.get("params") or {}
|
||||
|
||||
def burn_nonce(self, nonce, exp):
|
||||
seen = self.r.read_nonces()
|
||||
if nonce in seen:
|
||||
raise Refused("R3", "the signed job was already used (replay)")
|
||||
seen[nonce] = exp
|
||||
now = self.r.now()
|
||||
self.r.write_nonces({k: v for k, v in seen.items() if v > now})
|
||||
return op.get("params") or {}
|
||||
|
||||
def docker_authority(self, plan):
|
||||
"""R3 for the docker layer: returns (who, undo). A signed job binds the EXACT package list and the undo flag."""
|
||||
@@ -476,6 +653,10 @@ class Apply:
|
||||
for k, src in (("debian", "debian"), ("docker_engine", "engine"), ("containerd", "containerd")):
|
||||
g[k] = kv.get(src, "").strip() or "unknown"
|
||||
g["live_restore"] = {"true": "on", "false": "off"}.get(kv.get("live", "").strip(), "unknown")
|
||||
try:
|
||||
h["config_bundle"] = Bundle(self).state()
|
||||
except Exception as e: # a read problem must never cost the System page its other facts
|
||||
h["config_bundle"] = {"version": "unknown", "error": str(e)[:200]}
|
||||
self.report["facts"] = {"host": h, "guest": g}
|
||||
return 0
|
||||
|
||||
@@ -575,7 +756,10 @@ class Apply:
|
||||
if len(p) >= 4 and p[3]:
|
||||
cont[p[0]]["id"] = p[3]
|
||||
nrc, _, _ = self.g(["getent", "hosts", "deb.debian.org"], timeout=30)
|
||||
return {"docker_ok": rc == 0, "containers": cont,
|
||||
# R-858: the controller's own health check stayed "healthy" while it could not reach Docker at all — so ask
|
||||
# the consequence directly: can the controller talk to the engine from inside its container?
|
||||
crc, _, _ = self.g(["docker", "exec", "felhom-controller", "docker", "version", "--format", "{{.Server.Version}}"], timeout=60)
|
||||
return {"docker_ok": rc == 0, "containers": cont, "controller_docker_ok": crc == 0,
|
||||
"controller": cont.get("felhom-controller", {}).get("health", "absent"),
|
||||
"network_ok": nrc == 0}
|
||||
|
||||
@@ -650,8 +834,20 @@ class Apply:
|
||||
plan = self.load_plan()
|
||||
self.mode, self.layer, self.vmid, self.select = self.check_plan(plan)
|
||||
self.report.update(mode=self.mode, layer=self.layer, release_id=plan.get("release_id"), vmid=self.vmid)
|
||||
# R-868 (v0.144.0): the agent's run id, trigger and ring travel in the report, so a report the agent never
|
||||
# received (it was killed mid-pass) can be sent later from the saved copy. Plain ids only; anything else
|
||||
# is dropped, never refused (the plan's other checks decide).
|
||||
rid, trig, ring = plan.get("run_id"), plan.get("trigger"), plan.get("ring")
|
||||
if isinstance(rid, str) and re.match(r"^[A-Za-z0-9._-]{1,80}$", rid):
|
||||
self.report["run_id"] = rid
|
||||
if isinstance(trig, str) and re.match(r"^[a-z0-9_-]{1,20}$", trig):
|
||||
self.report["trigger"] = trig
|
||||
if ring in (0, 1) and not isinstance(ring, bool):
|
||||
self.report["ring"] = ring
|
||||
if self.mode == "facts":
|
||||
return self.facts()
|
||||
if self.mode == "bundle":
|
||||
return Bundle(self).from_plan(plan)
|
||||
if self.layer == "host":
|
||||
self.check_appliance()
|
||||
self.check_guest(self.vmid)
|
||||
@@ -728,6 +924,23 @@ class Apply:
|
||||
out.append({"name": p["name"], "version": p["to"], "origin": "Debian-Security" if "Debian-Security" in o else "Debian"})
|
||||
return out
|
||||
|
||||
def restart_socket_users(self):
|
||||
"""R-858: restart ONLY the containers that bind-mount the Docker socket, so they attach to the new one."""
|
||||
rc, out, _ = self.g(["docker", "ps", "-q", "--no-trunc"], timeout=60)
|
||||
users = []
|
||||
for cid in out.split():
|
||||
irc, iout, _ = self.g(["docker", "inspect", "-f", "{{.Name}}|{{range .Mounts}}{{.Destination}};{{end}}", cid], timeout=60)
|
||||
if irc != 0 or "|" not in iout:
|
||||
continue
|
||||
name, mounts = iout.strip().split("|", 1)
|
||||
if any(m in DOCKER_SOCKETS for m in mounts.split(";")):
|
||||
users.append(name.lstrip("/"))
|
||||
users.sort()
|
||||
if users:
|
||||
rrc, _, rerr = self.g(["docker", "restart"] + users, timeout=300)
|
||||
self.r.log(f"os-apply: SOCKET-USERS restarted={','.join(users)} rc={rrc} (R-858: they held the old docker socket)")
|
||||
return users
|
||||
|
||||
def pending_docker(self):
|
||||
"""Ring 0 (select pending-docker): the newest pending version of each INSTALLED Docker package, Docker origin."""
|
||||
rc, pend, remv, _ = self.simulate(["dist-upgrade"])
|
||||
@@ -837,6 +1050,8 @@ class Apply:
|
||||
return 3, None
|
||||
self.report["upgraded"] = [{"name": n, "version": v} for n, v in upgrade]
|
||||
self.report["seconds"] = round(secs, 1)
|
||||
if self.layer == "docker":
|
||||
self.report["socket_restarted"] = self.restart_socket_users()
|
||||
procs, reboot = self.restart_needed()
|
||||
self.report["restart_needed"] = procs
|
||||
self.report["docker_restart_needed"] = any(p in ("dockerd", "containerd") for p in procs)
|
||||
@@ -848,7 +1063,11 @@ class Apply:
|
||||
self.remove_snapshot_sources()
|
||||
|
||||
def download_bytes(self, args):
|
||||
rc, out, _ = self.x(APT_ENV + ["apt-get", "-s", "-o", "Debug::NoLocking=1", "--print-uris", "-q"] + args)
|
||||
# R-865 (v0.144.0): NO `-s`. With `-s` apt prints the simulation ("Inst …") and no URI list, so this summed
|
||||
# 0 B and R8 only ever applied its 500 MB floor (measured 2026-10-04: 0 URIs with -s, 3 URIs without).
|
||||
# `--print-uris` alone downloads nothing — measured on 9202 2026-10-05: the archive cache and the versions
|
||||
# unchanged. Pinned by test_R8_counts_the_real_download / test_download_bytes_never_simulates.
|
||||
rc, out, _ = self.x(APT_ENV + ["apt-get", "-o", "Debug::NoLocking=1", "--print-uris", "-q"] + args)
|
||||
total = 0
|
||||
for l in out.splitlines():
|
||||
m = re.match(r"^'[^']+' \S+ ([0-9]+) ", l)
|
||||
@@ -875,8 +1094,341 @@ class Apply:
|
||||
self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||
|
||||
|
||||
def main(argv, runner=None):
|
||||
class Bundle:
|
||||
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
|
||||
signed unit. Every check runs before the first write; a failed write or a failed self-check puts every previous
|
||||
copy back. Refusal codes: R1 (the bundle file), R3 (authority), R16 (a path outside BUNDLE_FILES), R17 (a trust
|
||||
file), R18 (a content check or a wrong sha)."""
|
||||
|
||||
def __init__(self, apply):
|
||||
self.a, self.r = apply, apply.r
|
||||
self.report = apply.report
|
||||
|
||||
# ---------- reading and checking ----------
|
||||
def load_file(self, path):
|
||||
d, base = os.path.dirname(path or ""), os.path.basename(path or "")
|
||||
if d != PLAN_DIR or not BUNDLE_RE.match(base) or ".." in path:
|
||||
raise Refused("R1", f"the bundle must be {PLAN_DIR}/bundle-<version>.json, got {path!r}")
|
||||
try:
|
||||
st = self.r.stat(path)
|
||||
except OSError as e:
|
||||
raise Refused("R1", f"cannot stat the bundle: {e}")
|
||||
if not stat.S_ISREG(st.st_mode):
|
||||
raise Refused("R1", "the bundle is not a regular file")
|
||||
if st.st_uid != self.r.agent_uid():
|
||||
raise Refused("R1", f"the bundle is not owned by {AGENT_USER}")
|
||||
if st.st_size > BUNDLE_MAX_BYTES:
|
||||
raise Refused("R1", "the bundle is larger than 4 MB")
|
||||
return self.r.read_bytes(path)
|
||||
|
||||
def parse(self, data, want_sha, want_version=None):
|
||||
"""The bundle bytes → [(dest, content, mode, check, policy)], every content check passed. Nothing is written."""
|
||||
import base64
|
||||
got = sha256_hex(data)
|
||||
if got != want_sha:
|
||||
raise Refused("R18", f"the bundle's sha256 is {got}, not the pinned {want_sha}")
|
||||
try:
|
||||
b = json.loads(data)
|
||||
except ValueError as e:
|
||||
raise Refused("R18", f"the bundle is not JSON: {e}")
|
||||
if not isinstance(b, dict) or b.get("format") != BUNDLE_FORMAT:
|
||||
raise Refused("R18", f"the bundle format is not {BUNDLE_FORMAT}")
|
||||
ver = b.get("agent_version")
|
||||
if not isinstance(ver, str) or not re.match(r"^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$", ver):
|
||||
raise Refused("R18", f"the bundle names no agent version ({ver!r})")
|
||||
if want_version is not None and ver != want_version:
|
||||
raise Refused("R18", f"the bundle is for agent {ver}, the signed job pins {want_version}")
|
||||
files = b.get("files")
|
||||
if not isinstance(files, list) or not files:
|
||||
raise Refused("R18", "the bundle lists no files")
|
||||
out, seen = [], set()
|
||||
for e in files:
|
||||
if not isinstance(e, dict) or not isinstance(e.get("path"), str):
|
||||
raise Refused("R18", "a bundle entry has no path")
|
||||
dest = e["path"]
|
||||
if dest in TRUST_PATHS:
|
||||
raise Refused("R17", f"{dest} is the trust root — no bundle may add, remove or change a signer")
|
||||
if dest not in BUNDLE_DESTS:
|
||||
raise Refused("R16", f"{dest} is not a path a bundle may write")
|
||||
if dest in seen:
|
||||
raise Refused("R18", f"{dest} is named twice")
|
||||
seen.add(dest)
|
||||
try:
|
||||
content = base64.b64decode(e.get("content_b64", ""), validate=True)
|
||||
except (ValueError, TypeError):
|
||||
raise Refused("R18", f"{dest}: the content is not base64")
|
||||
if sha256_hex(content) != e.get("sha256"):
|
||||
raise Refused("R18", f"{dest}: the content does not match its sha256")
|
||||
_, _, mode, check, policy = BUNDLE_DESTS[dest]
|
||||
self.check_content(dest, check, content)
|
||||
out.append((dest, content, mode, check, policy))
|
||||
order = [x[0] for x in BUNDLE_FILES]
|
||||
out.sort(key=lambda x: order.index(x[0])) # the table's order: the sudoers files last
|
||||
return ver, out
|
||||
|
||||
def check_content(self, dest, check, content):
|
||||
try:
|
||||
text = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
raise Refused("R18", f"{dest} is not UTF-8 text")
|
||||
if check in ("sudoers", "sh", "bash", "nft"):
|
||||
rc, msg = self.r.check_content(check, content)
|
||||
if rc != 0:
|
||||
raise Refused("R18", f"{dest} fails its {check} check: {msg}")
|
||||
elif check == "python":
|
||||
try:
|
||||
compile(text, dest, "exec")
|
||||
except SyntaxError as e:
|
||||
raise Refused("R18", f"{dest} is not valid Python: {e}")
|
||||
elif check in ("unit", "unit-nort", "agent-unit", "dropin"):
|
||||
if not re.search(r"^\[(Unit|Service|Timer)\]\s*$", text, re.M):
|
||||
raise Refused("R18", f"{dest} has no [Unit]/[Service]/[Timer] section")
|
||||
if check == "unit-nort" and re.search(r"^\s*RuntimeDirectory\s*=", text, re.M | re.I):
|
||||
raise Refused("R18", f"{dest} declares RuntimeDirectory= — the G1 incident; refused")
|
||||
if check == "agent-unit" and not re.search(r"^User=" + AGENT_USER + r"\s*$", text, re.M):
|
||||
raise Refused("R18", f"{dest} does not run the agent as {AGENT_USER}")
|
||||
if check == "dropin" and not re.search(r"^\[Unit\]\s*$", text, re.M):
|
||||
raise Refused("R18", f"{dest} must keep its start limits in [Unit] (SF-3)")
|
||||
# The route must survive the bundle: a sudoers without this wrapper's line, or a wrapper without the bundle
|
||||
# mode, would cut the box off from every later bundle.
|
||||
if dest == "/etc/sudoers.d/felhom-agent" and SELF_CHECK_SUDO_LINE not in text:
|
||||
raise Refused("R18", "the new sudoers no longer lets the agent call felhom-os-apply — the bundle route would end")
|
||||
if dest == "/usr/local/sbin/felhom-os-apply" and f'BUNDLE_OP = "{BUNDLE_OP}"' not in text:
|
||||
raise Refused("R18", "the new felhom-os-apply has no bundle mode — the bundle route would end")
|
||||
|
||||
def live(self, dest):
|
||||
"""(bytes or None, mode or None, uid or None) of what the box has now."""
|
||||
if not self.r.lexists(dest):
|
||||
return None, None, None
|
||||
st = self.r.stat(dest)
|
||||
if not stat.S_ISREG(st.st_mode):
|
||||
return b"", None, None # a symlink or a directory: always replaced by the real file
|
||||
return self.r.read_bytes(dest), stat.S_IMODE(st.st_mode), st.st_uid
|
||||
|
||||
def plan_writes(self, files):
|
||||
"""Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt)."""
|
||||
oob = self.r.isdir(OOB_DIR)
|
||||
plan = []
|
||||
for dest, content, mode, check, policy in files:
|
||||
if policy == "oob" and not oob:
|
||||
plan.append((dest, content, mode, "skipped", None, None))
|
||||
continue
|
||||
old, old_mode, old_uid = self.live(dest)
|
||||
if policy == "if-absent" and old is not None:
|
||||
plan.append((dest, content, mode, "kept", old, old_mode))
|
||||
elif old == content and old_mode == mode and old_uid == 0:
|
||||
plan.append((dest, content, mode, "same", old, old_mode))
|
||||
else:
|
||||
plan.append((dest, content, mode, "write", old, old_mode))
|
||||
return plan
|
||||
|
||||
# ---------- the two entries ----------
|
||||
def from_plan(self, plan):
|
||||
"""A signed agent_config_update, from the agent (sudo, the --plan line)."""
|
||||
data = self.load_file(plan.get("bundle"))
|
||||
trust = self.a.load_trust()
|
||||
signers, bootstrap = TRUST_SIGNERS, False
|
||||
if not self.r.lexists(TRUST_SIGNERS):
|
||||
signers, bootstrap = PINNED_SIGNERS, True
|
||||
self.r.log(f"os-apply: BUNDLE {TRUST_SIGNERS} is missing — verifying against the installer's pinned key "
|
||||
f"{PINNED_OPERATOR_KEY_ID} only")
|
||||
params, nonce, exp = self.a.verify_signed(plan.get("signed"), trust, op_name=BUNDLE_OP,
|
||||
signers=None if not bootstrap else signers, burn=False)
|
||||
sha, ver = params.get("bundle_sha256"), params.get("agent_version")
|
||||
if not isinstance(sha, str) or not re.match(r"^[0-9a-f]{64}$", sha) or not isinstance(ver, str):
|
||||
raise Refused("R3", "the signed job does not pin agent_version and bundle_sha256")
|
||||
ver, files = self.parse(data, sha, ver)
|
||||
self.a.burn_nonce(nonce, exp)
|
||||
return self.install(ver, sha, files, "signed", bootstrap)
|
||||
|
||||
def from_installer(self, path, sha):
|
||||
"""The installer (root, never through sudo): the hub manifest pinned the sha; no signature."""
|
||||
try:
|
||||
data = self.r.read_bytes(path)
|
||||
except OSError as e:
|
||||
raise Refused("R1", f"cannot read the bundle: {e}")
|
||||
ver, files = self.parse(data, sha)
|
||||
return self.install(ver, sha, files, "installer", False)
|
||||
|
||||
# ---------- install, self-check, undo ----------
|
||||
def install(self, ver, sha, files, authority, bootstrap):
|
||||
log = self.r.log
|
||||
plan = self.plan_writes(files)
|
||||
counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped")}
|
||||
log(f"os-apply: BUNDLE START agent={ver} sha={sha[:16]} authority={authority} files={len(plan)} "
|
||||
f"write={counts['write']} same={counts['same']} kept={counts['kept']} skipped={counts['skipped']}")
|
||||
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
|
||||
prev = os.path.join(BUNDLE_PREV_DIR, f"{stamp}-before-{ver}")
|
||||
done = [] # (dest, old bytes or None, old mode)
|
||||
rep = {"agent_version": ver, "sha256": sha, "authority": authority, "prev_dir": prev,
|
||||
"written": [p[0] for p in plan if p[3] == "write"], "kept": [p[0] for p in plan if p[3] == "kept"],
|
||||
"skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"]}
|
||||
self.report["bundle"] = rep
|
||||
try:
|
||||
for dest, content, mode, action, old, old_mode in plan:
|
||||
if action != "write":
|
||||
continue
|
||||
if old is not None:
|
||||
self.r.put_file(prev + dest, old, 0o600)
|
||||
done.append((dest, old, old_mode))
|
||||
self.r.put_file(dest, content, mode)
|
||||
log(f"os-apply: BUNDLE WROTE {dest} ({'replaced' if old is not None else 'new'})")
|
||||
self.after_install(plan)
|
||||
rep["self_check"] = self.self_check(plan)
|
||||
except (Refused, OSError, subprocess.SubprocessError) as e:
|
||||
reason = e.reason if isinstance(e, Refused) else str(e)
|
||||
log(f"os-apply: BUNDLE FAILED — {reason}; putting {len(done)} previous file(s) back")
|
||||
rep["rolled_back"] = self.undo(done)
|
||||
rep["failed"] = reason[:300]
|
||||
self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
|
||||
return 3
|
||||
rep["signers_created"] = False
|
||||
if bootstrap and not self.r.lexists(TRUST_SIGNERS):
|
||||
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
|
||||
rep["signers_created"] = True
|
||||
log(f"os-apply: BUNDLE created {TRUST_SIGNERS} with exactly the pinned key {PINNED_OPERATOR_KEY_ID}")
|
||||
record = {"format": BUNDLE_FORMAT, "agent_version": ver, "bundle_sha256": sha, "authority": authority,
|
||||
"installed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(self.r.now())),
|
||||
"files": {p[0]: (sha256_hex(p[1]) if p[3] in ("write", "same") else
|
||||
(sha256_hex(p[4]) if p[3] == "kept" else "skipped")) for p in plan}}
|
||||
self.r.put_file(BUNDLE_RECORD, (json.dumps(record, indent=2, sort_keys=True) + "\n").encode(), 0o644)
|
||||
self.prune()
|
||||
log(f"os-apply: BUNDLE DONE agent={ver} written={counts['write']} same={counts['same']} "
|
||||
f"self-check=ok signers-created={rep['signers_created']}")
|
||||
return 0
|
||||
|
||||
def after_install(self, plan):
|
||||
written = {p[0] for p in plan if p[3] == "write"}
|
||||
dests = {p[0] for p in plan if p[3] != "skipped"}
|
||||
if any(d.startswith("/etc/systemd/system/") for d in written):
|
||||
self.must(["systemctl", "daemon-reload"], "systemctl daemon-reload")
|
||||
if "/etc/tmpfiles.d/felhom-privsep.conf" in written:
|
||||
self.must(["systemd-tmpfiles", "--create", "/etc/tmpfiles.d/felhom-privsep.conf"], "systemd-tmpfiles")
|
||||
# `enable --now` starts a unit that is not running and leaves a running one alone (no restart). For the crash
|
||||
# guard that runs its boot step once on a box that never had it: kernel.panic for THIS boot, as the boot unit
|
||||
# would set it (`11` §5.9); with no earlier state it is the "first" boot, never counted as an unclean one.
|
||||
if "/etc/systemd/system/felhom-crash-guard.service" in dests:
|
||||
self.must(["systemctl", "enable", "--now", "felhom-crash-guard.service", "felhom-crash-guard-check.timer"],
|
||||
"enable the crash guard")
|
||||
if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
|
||||
self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
|
||||
|
||||
def must(self, argv, what):
|
||||
rc, out, err = self.r.host(argv, 120)
|
||||
if rc != 0:
|
||||
raise Refused("R18", f"{what} failed (rc={rc}): {(out + err).strip()[-200:]}")
|
||||
|
||||
def self_check(self, plan):
|
||||
"""After the install, before the record: the box still has a working route and working wrappers."""
|
||||
sc = {}
|
||||
self.must(["visudo", "-c"], "visudo -c (the whole sudoers)")
|
||||
sc["visudo"] = "ok"
|
||||
rc, out, err = self.r.host(["sudo", "-n", "-l", "-U", AGENT_USER], 60)
|
||||
if rc != 0 or "/usr/local/sbin/felhom-os-apply --plan" not in out:
|
||||
raise Refused("R18", f"sudo -l for {AGENT_USER} no longer lists felhom-os-apply (rc={rc})")
|
||||
sc["sudo_l"] = "felhom-os-apply listed"
|
||||
rc, out, err = self.r.host(["/usr/bin/python3", "/usr/local/sbin/felhom-os-apply", "--self-check"], 60)
|
||||
if rc != 0 or f"bundle-format={BUNDLE_FORMAT}" not in out:
|
||||
raise Refused("R18", f"the installed felhom-os-apply does not answer its self-check (rc={rc})")
|
||||
sc["os_apply"] = out.strip()[:120]
|
||||
rc, out, err = self.r.host(["/bin/sh", "/usr/local/sbin/felhom-selfupdate-guarded"], 60)
|
||||
if rc != 2 or "usage" not in (out + err):
|
||||
raise Refused("R18", f"the installed felhom-selfupdate-guarded does not answer with its usage (rc={rc})")
|
||||
sc["selfupdate"] = "usage ok"
|
||||
dests = {p[0] for p in plan if p[3] != "skipped"}
|
||||
if "/usr/local/sbin/felhom-crash-guard" in dests:
|
||||
rc, out, err = self.r.host(["/usr/local/sbin/felhom-crash-guard", "status"], 60)
|
||||
try:
|
||||
st = json.loads(out)
|
||||
except ValueError:
|
||||
st = None
|
||||
if rc != 0 or not isinstance(st, dict):
|
||||
raise Refused("R18", f"the crash guard does not answer its status (rc={rc})")
|
||||
try:
|
||||
panic = int(self.r.read_file("/proc/sys/kernel/panic").strip())
|
||||
except (OSError, ValueError):
|
||||
panic = None
|
||||
if panic != st.get("kernel_panic"):
|
||||
raise Refused("R18", f"kernel.panic is {panic}, the crash guard set {st.get('kernel_panic')}")
|
||||
sc["crash_guard"] = {"armed": st.get("armed"), "kernel_panic": panic}
|
||||
return sc
|
||||
|
||||
def undo(self, done):
|
||||
"""Put every previous copy back (newest write first); remove files the bundle created."""
|
||||
back = []
|
||||
for dest, old, old_mode in reversed(done):
|
||||
try:
|
||||
if old is None:
|
||||
self.r.remove(dest)
|
||||
else:
|
||||
self.r.put_file(dest, old, old_mode if old_mode is not None else 0o644)
|
||||
back.append(dest)
|
||||
except OSError as e:
|
||||
self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
|
||||
if any(d.startswith("/etc/systemd/system/") for d in back):
|
||||
self.r.host(["systemctl", "daemon-reload"], 120)
|
||||
rc, _, _ = self.r.host(["visudo", "-c"], 60)
|
||||
self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
|
||||
return back
|
||||
|
||||
def prune(self):
|
||||
names = [n for n in self.r.list_dir(BUNDLE_PREV_DIR)]
|
||||
for n in names[:-BUNDLE_KEEP]:
|
||||
self.r.rmtree(os.path.join(BUNDLE_PREV_DIR, n))
|
||||
|
||||
# ---------- the facts mode's view ----------
|
||||
def state(self):
|
||||
"""What the box runs: the record, and every bundle path's live sha256 against it (drift = changed by hand)."""
|
||||
rec = None
|
||||
try:
|
||||
rec = json.loads(self.r.read_file(BUNDLE_RECORD))
|
||||
except (OSError, ValueError):
|
||||
rec = None
|
||||
live = {}
|
||||
for dest, *_ in BUNDLE_FILES:
|
||||
try:
|
||||
data, _, _ = self.live(dest)
|
||||
except OSError:
|
||||
data = None
|
||||
live[dest] = sha256_hex(data) if data is not None else "absent"
|
||||
out = {"version": (rec or {}).get("agent_version", "none"), "live": live}
|
||||
if rec:
|
||||
out["installed_at"] = rec.get("installed_at")
|
||||
out["bundle_sha256"] = rec.get("bundle_sha256")
|
||||
want = rec.get("files") or {}
|
||||
out["drift"] = sorted(d for d, h in want.items() if h != "skipped" and live.get(d) != h)
|
||||
out["signers_present"] = self.r.lexists(TRUST_SIGNERS)
|
||||
return out
|
||||
|
||||
|
||||
def main(argv, runner=None, environ=None):
|
||||
r = runner or Runner()
|
||||
env = os.environ if environ is None else environ
|
||||
if argv[1:] == ["--self-check"]:
|
||||
# The bundle's self-check runs the NEW wrapper this way: it parses and starts. Reads nothing, changes nothing.
|
||||
print(f"felhom-os-apply ok bundle-format={BUNDLE_FORMAT} files={len(BUNDLE_FILES)}")
|
||||
return 0
|
||||
if len(argv) == 5 and argv[1] == "--install-bundle" and argv[3] == "--sha256":
|
||||
# The INSTALLER's entry (root, a fresh box). Unreachable through sudo: the sudoers line pins argv[1] to --plan,
|
||||
# and a sudo caller is refused here as well (sudo always sets SUDO_UID).
|
||||
a = Apply(r, "")
|
||||
if "SUDO_UID" in env:
|
||||
r.log("os-apply: REFUSED: R1 --install-bundle is the installer's entry, never through sudo")
|
||||
print("OSAPPLY-REPORT " + json.dumps({"refused": {"code": "R1", "reason": "install-bundle via sudo"}}))
|
||||
return 2
|
||||
sha = argv[4]
|
||||
if not re.match(r"^[0-9a-f]{64}$", sha):
|
||||
print("OSAPPLY-REPORT " + json.dumps({"refused": {"code": "R18", "reason": "sha256 is not 64 lowercase hex"}}))
|
||||
return 2
|
||||
a.report["mode"] = "bundle"
|
||||
try:
|
||||
rc = Bundle(a).from_installer(argv[2], sha)
|
||||
except Refused as e:
|
||||
r.log(f"os-apply: REFUSED: {e.code} {e.reason}")
|
||||
a.report["refused"] = {"code": e.code, "reason": e.reason}
|
||||
rc = 2
|
||||
print("OSAPPLY-REPORT " + json.dumps(a.report, sort_keys=True))
|
||||
return rc
|
||||
if len(argv) != 3 or argv[1] != "--plan":
|
||||
r.log("os-apply: REFUSED: R1 usage: felhom-os-apply --plan /var/lib/felhom-agent/os/plan-<id>.json")
|
||||
print("OSAPPLY-REPORT " + json.dumps({"refused": {"code": "R1", "reason": "usage"}}))
|
||||
@@ -894,6 +1446,13 @@ def main(argv, runner=None):
|
||||
a.report["failed"] = {"rc": 124, "timeout": str(e.cmd)[:200]}
|
||||
rc = 3
|
||||
a.report["pass_seconds"] = round(time.time() - t0, 1)
|
||||
if a.report.get("mode") == "apply":
|
||||
# R-868: BEFORE the stdout line — a killed agent never reads stdout, and this copy is how its report still
|
||||
# reaches the hub (the agent sends an unsent copy when it starts, and deletes it once sent).
|
||||
try:
|
||||
r.save_report(argv[2], a.report)
|
||||
except Exception as e:
|
||||
r.log(f"os-apply: the report copy could not be saved (the run is unaffected): {e}")
|
||||
print("OSAPPLY-REPORT " + json.dumps(a.report, sort_keys=True))
|
||||
return rc
|
||||
|
||||
|
||||
@@ -0,0 +1,545 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for the config bundle (R-840, `11` §5.4.2): felhom-os-apply's `bundle` mode and `--install-bundle`, and
|
||||
scripts/build-config-bundle.py. An in-memory host plays the files; nothing real is written or run. Each refusal has a
|
||||
test; each test names the rule it pins. Red-proof: `audits/r840-config-bundle-2026-10-04/partB/redproof.txt`.
|
||||
|
||||
Run: python3 configs/test_felhom_config_bundle.py (also run by internal/osupdate's Go test)
|
||||
"""
|
||||
import sys
|
||||
sys.dont_write_bytecode = True # importing the builder must not leave scripts/__pycache__ behind
|
||||
import base64
|
||||
import contextlib
|
||||
import hashlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import stat as statmod
|
||||
import unittest
|
||||
|
||||
HERE = pathlib.Path(__file__).resolve().parent
|
||||
REPO = HERE.parent
|
||||
_loader = importlib.machinery.SourceFileLoader("osapply", os.environ.get("OSAPPLY_UNDER_TEST", str(HERE / "felhom-os-apply")))
|
||||
_spec = importlib.util.spec_from_loader("osapply", _loader)
|
||||
osapply = importlib.util.module_from_spec(_spec)
|
||||
_loader.exec_module(osapply)
|
||||
_bl = importlib.machinery.SourceFileLoader("bundlebuild", str(REPO / "scripts" / "build-config-bundle.py"))
|
||||
_bs = importlib.util.spec_from_loader("bundlebuild", _bl)
|
||||
builder = importlib.util.module_from_spec(_bs)
|
||||
_bl.exec_module(builder)
|
||||
|
||||
PLAN = "/var/lib/felhom-agent/os/plan-b1.json"
|
||||
BUNDLE = "/var/lib/felhom-agent/os/bundle-0.143.0.json"
|
||||
HOST = "demo-hp-bb76ea"
|
||||
INSTALLER = REPO.parent / "felhom.eu" / "scripts" / "felhom-host-install.sh"
|
||||
|
||||
|
||||
class St:
|
||||
def __init__(self, mode, uid):
|
||||
self.st_mode, self.st_uid, self.st_size = mode, uid, 100
|
||||
|
||||
|
||||
class Box:
|
||||
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
|
||||
|
||||
def __init__(self, bundle_bytes, signed, oob=False, signers=True):
|
||||
self.files, self.modes, self.uids = {}, {}, {}
|
||||
self.dirs = {osapply.OOB_DIR} if oob else set()
|
||||
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
|
||||
if signers:
|
||||
self.put(osapply.TRUST_SIGNERS, b'felhom-op-1 namespaces="felhom-op-v1" ssh-ed25519 AAAA felhom-op-1\n', 0o644)
|
||||
self.put("/proc/sys/kernel/panic", b"0\n", 0o644)
|
||||
self.plan = {"release_id": "bundle-0.143.0", "layer": "host", "mode": "bundle", "bundle": BUNDLE, "signed": signed}
|
||||
self.put(PLAN, json.dumps(self.plan).encode(), 0o600, uid=999)
|
||||
self.put(BUNDLE, bundle_bytes, 0o600, uid=999)
|
||||
self.sig_rc, self.nonces, self.clock = 0, {}, 1791115200.0 # 2026-10-04T12:00:00Z
|
||||
self.calls, self.logs, self.writes = [], [], []
|
||||
self.visudo_fail = False # the WHOLE sudoers (`visudo -c`) after install
|
||||
self.sudo_l = " (root) NOPASSWD: /usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json\n"
|
||||
self.guard = {"armed": True, "kernel_panic": 10}
|
||||
|
||||
def put(self, p, data, mode, uid=0):
|
||||
self.files[p], self.modes[p], self.uids[p] = data, mode, uid
|
||||
|
||||
# Runner interface
|
||||
def now(self):
|
||||
return self.clock
|
||||
|
||||
def log(self, line):
|
||||
self.logs.append(line)
|
||||
|
||||
def agent_uid(self):
|
||||
return 999
|
||||
|
||||
def verify_sig(self, signers, key_id, ns, blob, sig):
|
||||
self.verified = (signers, key_id, ns)
|
||||
return self.sig_rc
|
||||
|
||||
def read_nonces(self):
|
||||
return dict(self.nonces)
|
||||
|
||||
def write_nonces(self, d):
|
||||
self.nonces = dict(d)
|
||||
|
||||
def read_file(self, p):
|
||||
if p not in self.files:
|
||||
raise OSError("no such file")
|
||||
return self.files[p].decode()
|
||||
|
||||
def read_bytes(self, p):
|
||||
if p not in self.files:
|
||||
raise OSError("no such file")
|
||||
return self.files[p]
|
||||
|
||||
def stat(self, p):
|
||||
if p not in self.files:
|
||||
raise OSError("no such file")
|
||||
return St(statmod.S_IFREG | self.modes[p], self.uids[p])
|
||||
|
||||
def lexists(self, p):
|
||||
return p in self.files
|
||||
|
||||
def isdir(self, p):
|
||||
return p in self.dirs
|
||||
|
||||
def put_file(self, p, data, mode):
|
||||
self.writes.append(p)
|
||||
self.put(p, data, mode)
|
||||
|
||||
def remove(self, p):
|
||||
self.writes.append("rm " + p)
|
||||
del self.files[p]
|
||||
|
||||
def list_dir(self, p):
|
||||
return sorted({k[len(p) + 1:].split("/")[0] for k in self.files if k.startswith(p + "/")})
|
||||
|
||||
def rmtree(self, p):
|
||||
for k in [k for k in self.files if k.startswith(p + "/")]:
|
||||
del self.files[k]
|
||||
|
||||
def check_content(self, kind, data):
|
||||
return (1, f"{kind}: syntax error") if b"BROKEN-SYNTAX" in data else (0, "")
|
||||
|
||||
def host(self, argv, timeout=600, stdin=None):
|
||||
self.calls.append(argv)
|
||||
if argv[:2] == ["visudo", "-c"]:
|
||||
return (1, "", "parse error") if self.visudo_fail else (0, "ok", "")
|
||||
if argv[:2] == ["sudo", "-n"]:
|
||||
return 0, self.sudo_l, ""
|
||||
if argv[-1] == "--self-check":
|
||||
body = self.files.get("/usr/local/sbin/felhom-os-apply", b"")
|
||||
return (0, "felhom-os-apply ok bundle-format=1 files=22\n", "") if b"BUNDLE_OP" in body else (1, "", "boom")
|
||||
if argv[-1] == "/usr/local/sbin/felhom-selfupdate-guarded":
|
||||
return 2, "", "felhom-selfupdate-guarded: usage: ...\n"
|
||||
if argv[-1] == "status" and argv[0].endswith("felhom-crash-guard"):
|
||||
return 0, json.dumps(self.guard), ""
|
||||
if argv[:3] == ["systemctl", "enable", "--now"] and "felhom-crash-guard.service" in argv:
|
||||
self.put("/proc/sys/kernel/panic", f"{self.guard['kernel_panic']}\n".encode(), 0o644)
|
||||
return 0, "", ""
|
||||
|
||||
|
||||
def signed_job(sha, version="0.143.0", host=HOST, op="agent_config_update", nonce="b1",
|
||||
issued="2026-10-04T11:50:00Z", expires="2026-10-04T12:30:00Z"):
|
||||
blob = json.dumps({"expires_at": expires, "issued_at": issued, "key_id": "felhom-op-1", "nonce": nonce, "op": op,
|
||||
"params": {"agent_version": version, "bundle_sha256": sha},
|
||||
"target": {"guest_id": "", "host_id": host}}, sort_keys=True).encode()
|
||||
return {"blob_b64": base64.b64encode(blob).decode(), "sig": "-----BEGIN SSH SIGNATURE-----\nx\n-----END SSH SIGNATURE-----\n"}
|
||||
|
||||
|
||||
def real_bundle(version="0.143.0"):
|
||||
data = builder.build(version)
|
||||
return data, hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def edited_bundle(edit):
|
||||
"""The real bundle, with edit(files_list) applied and every sha recomputed — a SIGNED bundle with bad content."""
|
||||
b = json.loads(builder.build("0.143.0"))
|
||||
edit(b["files"])
|
||||
for e in b["files"]:
|
||||
e["sha256"] = hashlib.sha256(base64.b64decode(e["content_b64"])).hexdigest()
|
||||
data = (json.dumps(b, indent=1, sort_keys=True) + "\n").encode()
|
||||
return data, hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def replace_content(files, path, fn):
|
||||
for e in files:
|
||||
if e["path"] == path:
|
||||
e["content_b64"] = base64.b64encode(fn(base64.b64decode(e["content_b64"]))).decode()
|
||||
|
||||
|
||||
def run(box, argv=None, environ=None):
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
rc = osapply.main(argv or ["felhom-os-apply", "--plan", PLAN], runner=box, environ=environ or {})
|
||||
line = [l for l in buf.getvalue().splitlines() if l.startswith("OSAPPLY-REPORT ")][-1]
|
||||
return rc, json.loads(line[len("OSAPPLY-REPORT "):])
|
||||
|
||||
|
||||
def box_files(box):
|
||||
return {p: box.files[p] for p in box.files if p in osapply.BUNDLE_DESTS}
|
||||
|
||||
|
||||
class Install(unittest.TestCase):
|
||||
def test_fresh_box_gets_every_file_and_a_record(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
b = rep["bundle"]
|
||||
# every path but the four OOB ones (no belt on this box)
|
||||
self.assertEqual(len(b["written"]), len(osapply.BUNDLE_FILES) - 4, b)
|
||||
self.assertEqual(len(b["skipped"]), 4)
|
||||
self.assertEqual(box.files["/etc/sudoers.d/felhom-agent"], (REPO / "configs" / "felhom-agent.sudoers").read_bytes())
|
||||
self.assertEqual(box.modes["/etc/sudoers.d/felhom-agent"], 0o440)
|
||||
rec = json.loads(box.files[osapply.BUNDLE_RECORD])
|
||||
self.assertEqual((rec["agent_version"], rec["bundle_sha256"], rec["authority"]), ("0.143.0", sha, "signed"))
|
||||
self.assertIn("b1", box.nonces, "the job is consumed")
|
||||
self.assertEqual(b["self_check"]["crash_guard"], {"armed": True, "kernel_panic": 10})
|
||||
self.assertIn(["systemctl", "daemon-reload"], box.calls)
|
||||
|
||||
def test_sudoers_is_written_after_every_wrapper(self):
|
||||
"""Order: a referenced wrapper is in place before the sudoers line that allows it."""
|
||||
data, sha = edited_bundle(lambda f: f.reverse()) # the bundle lists the sudoers FIRST; the wrapper must reorder
|
||||
box = Box(data, signed_job(sha))
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
w = [p for p in box.writes if p in osapply.BUNDLE_DESTS]
|
||||
self.assertEqual(w[-1], "/etc/sudoers.d/felhom-agent")
|
||||
self.assertLess(w.index("/usr/local/sbin/felhom-os-apply"), w.index("/etc/sudoers.d/felhom-agent"))
|
||||
|
||||
def test_identical_box_writes_nothing(self):
|
||||
"""The demo boxes' case: hand-copied files equal to the release → 0 written, all 'same'."""
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
for dest, src, mode, _, policy in osapply.BUNDLE_FILES:
|
||||
if policy != "oob":
|
||||
box.put(dest, (REPO / "configs" / src).read_bytes(), mode)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["bundle"]["written"], [])
|
||||
self.assertEqual(rep["bundle"]["same"], len(osapply.BUNDLE_FILES) - 5) # 4 oob skipped + crash-guard.conf kept
|
||||
self.assertEqual(rep["bundle"]["kept"], ["/etc/felhom/crash-guard.conf"])
|
||||
|
||||
def test_a_wrong_mode_is_rewritten(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.put("/etc/sudoers.d/felhom-agent", (REPO / "configs" / "felhom-agent.sudoers").read_bytes(), 0o644)
|
||||
rc, rep = run(box)
|
||||
self.assertIn("/etc/sudoers.d/felhom-agent", rep["bundle"]["written"])
|
||||
self.assertEqual(box.modes["/etc/sudoers.d/felhom-agent"], 0o440)
|
||||
|
||||
def test_tuned_crash_guard_conf_is_kept(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.put("/etc/felhom/crash-guard.conf", b"LIMIT=5\n", 0o644)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.files["/etc/felhom/crash-guard.conf"], b"LIMIT=5\n")
|
||||
|
||||
def test_oob_files_only_on_a_box_with_the_belt(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha), oob=True)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertIn("/etc/sudoers.d/felhom-op", rep["bundle"]["written"])
|
||||
self.assertEqual(rep["bundle"]["skipped"], [])
|
||||
|
||||
def test_previous_copies_are_kept(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.put("/usr/local/sbin/felhom-pbs-apply", b"#!/bin/bash\necho old\n", 0o755)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.files[rep["bundle"]["prev_dir"] + "/usr/local/sbin/felhom-pbs-apply"], b"#!/bin/bash\necho old\n")
|
||||
|
||||
|
||||
class Refusals(unittest.TestCase):
|
||||
"""Each: refused, and NOTHING on the box changed."""
|
||||
|
||||
def refused(self, box, code):
|
||||
before = dict(box_files(box))
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 2, rep)
|
||||
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||
self.assertEqual(box_files(box), before, "a refusal changed a file")
|
||||
self.assertNotIn(osapply.BUNDLE_RECORD, box.files)
|
||||
return rep
|
||||
|
||||
def test_wrong_sha_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job("0" * 64))
|
||||
self.refused(box, "R18")
|
||||
self.assertEqual(box.nonces, {}, "a wrong sha must not burn the job")
|
||||
|
||||
def test_bad_signature_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.sig_rc = 255
|
||||
self.refused(box, "R3")
|
||||
|
||||
def test_other_op_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
self.refused(Box(data, signed_job(sha, op="agent_update")), "R3")
|
||||
|
||||
def test_other_host_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
self.refused(Box(data, signed_job(sha, host="demo-felhom-8363b5")), "R3")
|
||||
|
||||
def test_expired_job_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
self.refused(Box(data, signed_job(sha, expires="2026-10-04T11:55:00Z")), "R3")
|
||||
|
||||
def test_replayed_job_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.nonces = {"b1": box.clock + 600}
|
||||
self.refused(box, "R3")
|
||||
|
||||
def test_version_mismatch_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
self.refused(Box(data, signed_job(sha, version="0.142.1")), "R18")
|
||||
|
||||
def test_sudoers_failing_visudo_is_refused(self):
|
||||
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/sudoers.d/felhom-agent", lambda c: c + b"BROKEN-SYNTAX\n"))
|
||||
self.refused(Box(data, signed_job(sha)), "R18")
|
||||
|
||||
def test_sudoers_dropping_the_route_is_refused(self):
|
||||
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/sudoers.d/felhom-agent",
|
||||
lambda c: c.replace(b"/usr/local/sbin/felhom-os-apply --plan", b"/bin/true --plan")))
|
||||
self.refused(Box(data, signed_job(sha)), "R18")
|
||||
|
||||
def test_wrapper_without_bundle_mode_is_refused(self):
|
||||
data, sha = edited_bundle(lambda f: replace_content(f, "/usr/local/sbin/felhom-os-apply",
|
||||
lambda c: c.replace(b'BUNDLE_OP = "agent_config_update"', b'BUNDLE_OPX = 1')))
|
||||
self.refused(Box(data, signed_job(sha)), "R18")
|
||||
|
||||
def test_python_syntax_error_is_refused(self):
|
||||
data, sha = edited_bundle(lambda f: replace_content(f, "/usr/local/sbin/felhom-crash-guard", lambda c: c + b"\ndef (\n"))
|
||||
self.refused(Box(data, signed_job(sha)), "R18")
|
||||
|
||||
def test_unit_with_runtime_directory_is_refused(self):
|
||||
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/systemd/system/felhom-mgmt-watchdog.service",
|
||||
lambda c: c + b"RuntimeDirectory=sshd\n"))
|
||||
self.refused(Box(data, signed_job(sha)), "R18")
|
||||
|
||||
def test_agent_unit_not_as_the_agent_user_is_refused(self):
|
||||
data, sha = edited_bundle(lambda f: replace_content(f, "/etc/systemd/system/felhom-agent.service",
|
||||
lambda c: c.replace(b"User=felhom-agent", b"User=root")))
|
||||
self.refused(Box(data, signed_job(sha)), "R18")
|
||||
|
||||
def test_a_bundle_that_changes_a_signer_is_refused(self):
|
||||
"""R17: the trust root is not a bundle's to change — not even a signed one."""
|
||||
def add(f):
|
||||
f.append({"path": osapply.TRUST_SIGNERS, "content_b64": base64.b64encode(b"evil-key\n").decode()})
|
||||
data, sha = edited_bundle(add)
|
||||
box = Box(data, signed_job(sha))
|
||||
self.refused(box, "R17")
|
||||
self.assertIn(b"felhom-op-1", box.files[osapply.TRUST_SIGNERS])
|
||||
|
||||
def test_a_path_outside_the_table_is_refused(self):
|
||||
def add(f):
|
||||
f.append({"path": "/etc/shadow", "content_b64": base64.b64encode(b"root::0:0\n").decode()})
|
||||
data, sha = edited_bundle(add)
|
||||
self.refused(Box(data, signed_job(sha)), "R16")
|
||||
|
||||
def test_content_not_matching_its_sha_is_refused(self):
|
||||
b = json.loads(builder.build("0.143.0"))
|
||||
b["files"][0]["content_b64"] = base64.b64encode(b"#!/bin/bash\nexit 0\n").decode()
|
||||
data = json.dumps(b).encode()
|
||||
self.refused(Box(data, signed_job(hashlib.sha256(data).hexdigest())), "R18")
|
||||
|
||||
def test_bundle_outside_the_plan_dir_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.plan["bundle"] = "/tmp/bundle-0.143.0.json"
|
||||
box.files[PLAN] = json.dumps(box.plan).encode()
|
||||
self.refused(box, "R1")
|
||||
|
||||
def test_bundle_not_owned_by_the_agent_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.uids[BUNDLE] = 0
|
||||
self.refused(box, "R1")
|
||||
|
||||
def test_no_trust_file_is_refused(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
del box.files[osapply.TRUST_FILE]
|
||||
self.refused(box, "R3")
|
||||
|
||||
|
||||
class SelfCheckUndo(unittest.TestCase):
|
||||
def assert_restored(self, box, before, rep):
|
||||
self.assertTrue(rep["bundle"]["rolled_back"], rep)
|
||||
self.assertEqual(box_files(box), before, "the previous files must be back, byte for byte")
|
||||
self.assertNotIn(osapply.BUNDLE_RECORD, box.files)
|
||||
|
||||
def with_old_files(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.put("/usr/local/sbin/felhom-pbs-apply", b"#!/bin/bash\necho old\n", 0o755)
|
||||
box.put("/etc/sudoers.d/felhom-agent", b"# old sudoers\n", 0o440)
|
||||
return box
|
||||
|
||||
def test_route_missing_after_install_puts_everything_back(self):
|
||||
box = self.with_old_files()
|
||||
before = dict(box_files(box))
|
||||
box.sudo_l = " (root) NOPASSWD: /bin/true\n"
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 3, rep)
|
||||
self.assert_restored(box, before, rep)
|
||||
self.assertEqual(box.calls[-1], ["visudo", "-c"], "the undo re-checks the whole sudoers")
|
||||
|
||||
def test_visudo_failing_after_install_puts_everything_back(self):
|
||||
box = self.with_old_files()
|
||||
before = dict(box_files(box))
|
||||
box.visudo_fail = True
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 3, rep)
|
||||
self.assert_restored(box, before, rep)
|
||||
|
||||
def test_crash_guard_disagreeing_with_kernel_panic_puts_everything_back(self):
|
||||
box = self.with_old_files()
|
||||
before = dict(box_files(box))
|
||||
box.guard = {"armed": True, "kernel_panic": 10}
|
||||
box.host_orig = box.host
|
||||
|
||||
def host(argv, timeout=600, stdin=None):
|
||||
if argv[:3] == ["systemctl", "enable", "--now"]:
|
||||
box.calls.append(argv)
|
||||
return 0, "", "" # the unit "started" but kernel.panic stayed 0
|
||||
return box.host_orig(argv, timeout, stdin)
|
||||
box.host = host
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 3, rep)
|
||||
self.assert_restored(box, before, rep)
|
||||
|
||||
|
||||
class TrustBootstrap(unittest.TestCase):
|
||||
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha), signers=False)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.verified[0], osapply.PINNED_SIGNERS, "verified against the pinned key, nothing else")
|
||||
self.assertTrue(rep["bundle"]["signers_created"])
|
||||
self.assertEqual(box.files[osapply.TRUST_SIGNERS], osapply.pinned_signers_line().encode())
|
||||
self.assertEqual(box.modes[osapply.TRUST_SIGNERS], 0o644)
|
||||
|
||||
def test_missing_signers_and_a_job_the_pinned_key_did_not_sign(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha), signers=False)
|
||||
box.sig_rc = 255
|
||||
rc, rep = run(box)
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"))
|
||||
self.assertNotIn(osapply.TRUST_SIGNERS, box.files)
|
||||
|
||||
def test_present_signers_are_never_touched(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.put(osapply.TRUST_SIGNERS, b'felhom-op-2 namespaces="felhom-op-v1" ssh-ed25519 BBBB felhom-op-2\n', 0o644)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.verified[0], osapply.TRUST_SIGNERS)
|
||||
self.assertFalse(rep["bundle"]["signers_created"])
|
||||
self.assertIn(b"felhom-op-2", box.files[osapply.TRUST_SIGNERS])
|
||||
|
||||
def test_agent_writable_signers_are_refused(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
box.uids[osapply.TRUST_SIGNERS] = 999
|
||||
rc, rep = run(box)
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R3"))
|
||||
|
||||
def test_pinned_operator_key_equals_the_installers(self):
|
||||
"""The bootstrap key is exactly the one felhom-host-install.sh pins (OPERATOR_KEY_OPERATIONAL_*)."""
|
||||
text = INSTALLER.read_text()
|
||||
kid = re.search(r'^OPERATOR_KEY_OPERATIONAL_ID="([^"]+)"', text, re.M).group(1)
|
||||
line = re.search(r'^OPERATOR_KEY_OPERATIONAL_LINE="([^"]+)"', text, re.M).group(1)
|
||||
self.assertEqual((osapply.PINNED_OPERATOR_KEY_ID, osapply.PINNED_OPERATOR_KEY_LINE), (kid, line))
|
||||
# and the file format is the installer's printf, byte for byte
|
||||
self.assertIn("printf '%s namespaces=\"felhom-op-v1\" %s\\n'", text)
|
||||
|
||||
|
||||
class InstallerEntry(unittest.TestCase):
|
||||
def test_installer_installs_without_a_signature(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, None)
|
||||
box.put("/root/bundle.json", data, 0o600)
|
||||
rc, rep = run(box, ["felhom-os-apply", "--install-bundle", "/root/bundle.json", "--sha256", sha])
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["authority"], "installer")
|
||||
|
||||
def test_installer_entry_checks_the_sha(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, None)
|
||||
box.put("/root/bundle.json", data, 0o600)
|
||||
rc, rep = run(box, ["felhom-os-apply", "--install-bundle", "/root/bundle.json", "--sha256", "1" * 64])
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R18"))
|
||||
|
||||
def test_installer_entry_is_refused_through_sudo(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, None)
|
||||
box.put("/root/bundle.json", data, 0o600)
|
||||
rc, rep = run(box, ["felhom-os-apply", "--install-bundle", "/root/bundle.json", "--sha256", sha], {"SUDO_UID": "999"})
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R1"))
|
||||
self.assertNotIn(osapply.BUNDLE_RECORD, box.files)
|
||||
|
||||
def test_self_check_answers(self):
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
rc = osapply.main(["felhom-os-apply", "--self-check"], runner=Box(b"", None))
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertIn("bundle-format=1", buf.getvalue())
|
||||
|
||||
|
||||
class Facts(unittest.TestCase):
|
||||
def test_state_reports_drift_against_the_record(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
run(box)
|
||||
box.put("/usr/local/sbin/felhom-pbs-apply", b"#!/bin/bash\necho by hand\n", 0o755)
|
||||
a = osapply.Apply(box, PLAN)
|
||||
st = osapply.Bundle(a).state()
|
||||
self.assertEqual(st["version"], "0.143.0")
|
||||
self.assertEqual(st["drift"], ["/usr/local/sbin/felhom-pbs-apply"])
|
||||
self.assertTrue(st["signers_present"])
|
||||
|
||||
def test_state_without_a_record_says_none(self):
|
||||
box = Box(b"", None)
|
||||
st = osapply.Bundle(osapply.Apply(box, PLAN)).state()
|
||||
self.assertEqual(st["version"], "none")
|
||||
self.assertNotIn("drift", st)
|
||||
self.assertEqual(st["live"]["/etc/sudoers.d/felhom-agent"], "absent")
|
||||
|
||||
|
||||
class Builder(unittest.TestCase):
|
||||
def test_reproducible(self):
|
||||
self.assertEqual(builder.build("0.143.0"), builder.build("0.143.0"))
|
||||
|
||||
def test_every_source_exists_and_every_dest_is_unique(self):
|
||||
dests = [e[0] for e in osapply.BUNDLE_FILES]
|
||||
self.assertEqual(len(dests), len(set(dests)))
|
||||
for _, src, *_ in osapply.BUNDLE_FILES:
|
||||
self.assertTrue((REPO / "configs" / src).is_file(), src)
|
||||
|
||||
def test_every_root_file_the_installer_writes_is_in_the_bundle(self):
|
||||
"""One source of truth: a felhom root-owned path the installer names must be a bundle path, a trust file, or a
|
||||
path the AGENT itself writes at run time (named here, with why). Scope is a regex over the WHOLE installer."""
|
||||
text = INSTALLER.read_text()
|
||||
found = set(re.findall(r"(/usr/local/sbin/felhom-[a-z-]+|/etc/systemd/system/felhom-[a-z.-]+|"
|
||||
r"/etc/sudoers\.d/felhom-[a-z-]+|/etc/felhom-oob\.nft|/etc/tmpfiles\.d/felhom-[a-z.-]+|"
|
||||
r"/etc/felhom/[a-z.-]+)", text))
|
||||
agent_writes = {"/usr/local/sbin/felhom-shared-parent", "/etc/systemd/system/felhom-shared-parent.service"}
|
||||
trust = {osapply.TRUST_FILE, osapply.TRUST_SIGNERS, osapply.TRUST_SIGNERS + ".tmp", osapply.BUNDLE_RECORD}
|
||||
missing = sorted(p for p in found if p not in osapply.BUNDLE_DESTS and p not in agent_writes | trust)
|
||||
self.assertEqual(missing, [], "the installer writes these root files, but the bundle does not carry them")
|
||||
# the limits drop-in is named through $AGENT_UNIT in the installer
|
||||
self.assertIn("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", osapply.BUNDLE_DESTS)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -73,6 +73,7 @@ class Fake:
|
||||
self.sig_rc = 0
|
||||
self.nonces = {}
|
||||
self.clock = 1791115200.0 # 2026-10-04T12:00:00Z
|
||||
self.saved_reports = [] # R-868: (plan path, report) the wrapper kept on disk
|
||||
self.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
self.stats[osapply.TRUST_FILE] = St(mode=statmod.S_IFREG | 0o644, uid=0)
|
||||
self.files[osapply.TRUST_SIGNERS] = 'felhom-op-1 namespaces="felhom-op-v1" ssh-ed25519 AAAA\n'
|
||||
@@ -113,6 +114,10 @@ class Fake:
|
||||
def log(self, line):
|
||||
self.logs.append(line)
|
||||
|
||||
def save_report(self, plan_path, report):
|
||||
self.saved_reports.append((plan_path, json.loads(json.dumps(report))))
|
||||
return plan_path.replace("/plan-", "/report-")
|
||||
|
||||
def host(self, argv, timeout=600, stdin=None):
|
||||
self.calls.append(("host", argv))
|
||||
if argv[0] == "/usr/sbin/pct" and argv[1] == "status":
|
||||
@@ -168,8 +173,8 @@ class Fake:
|
||||
if "-f" in a:
|
||||
self.dpkg_audit = ""
|
||||
return 0, "Setting up x (1) ...\n" if getattr(self, "repaired", False) else "", ""
|
||||
if "-s" in a:
|
||||
return self.sim(a)
|
||||
if "--print-uris" in a or "-s" in a:
|
||||
return self.sim(a) # --print-uris prints and installs nothing, with or without -s (9202, 2026-10-05)
|
||||
if "install" in a:
|
||||
if self.install_rc:
|
||||
return self.install_rc, "", "E: boom"
|
||||
@@ -186,6 +191,14 @@ class Fake:
|
||||
return 0, self.engine + "\n", ""
|
||||
if cmd == "docker" and a[1:3] == ["ps", "-q"]:
|
||||
return 0, "".join(i + "\n" for i in self.ids), ""
|
||||
if cmd == "docker" and a[1] == "inspect":
|
||||
mounts = {"aaa111": "/felhom-controller|/var/run/docker.sock;/app/data;", "bbb222": "/app|/data;"}
|
||||
return 0, mounts.get(a[-1], "/other|;") + "\n", ""
|
||||
if cmd == "docker" and a[1] == "restart":
|
||||
self.restarted_containers = a[2:]
|
||||
return 0, "", ""
|
||||
if cmd == "docker" and a[1] == "exec":
|
||||
return (1, "", "Cannot connect to the Docker daemon") if getattr(self, "controller_blind", False) else (0, self.engine + "\n", "")
|
||||
if cmd == "docker":
|
||||
ids = self.ids + ["x"] * 2
|
||||
return 0, f"felhom-controller\trunning\tUp 1 hour (healthy)\t{ids[0]}\napp\trunning\tUp 1 hour (healthy)\t{ids[1]}\n", ""
|
||||
@@ -224,7 +237,14 @@ class Fake:
|
||||
|
||||
def sim(self, a):
|
||||
if "--print-uris" in a:
|
||||
return 0, "'http://x/libc6.deb' libc6.deb 4000000 SHA256:x\n", ""
|
||||
if "-s" in a:
|
||||
# real apt (measured 9202 2026-10-05): with -s it prints the SIMULATION, no URI list
|
||||
return 0, "Inst libc6 [2.41-12+deb13u4] (2.41-12+deb13u4 Debian:13.7/stable [amd64])\n", ""
|
||||
# real apt's line shape, verbatim from 9202 2026-10-05 (audits/night-fixes-2026-10-05/partC/)
|
||||
return 0, ("Need to get 4347 kB of archives.\n"
|
||||
"'http://deb.debian.org/debian/pool/main/b/bash/bash_5.2.37-2%2bb10_amd64.deb' bash_5.2.37-2+b10_amd64.deb 1500792 MD5Sum:27b11721fea83d73b96e0f7023863771\n"
|
||||
"'http://deb.debian.org/debian/pool/main/g/glibc/libc6_2.41-12%2bdeb13u4_amd64.deb' libc6_2.41-12+deb13u4_amd64.deb 2846580 MD5Sum:5559581916477ef1f57ea9f82cecf22e\n"
|
||||
+ getattr(self, "extra_uris", "")), ""
|
||||
if "dist-upgrade" in a:
|
||||
if getattr(self, "pending_sim", None) is not None and not getattr(self, "_pending_used", False):
|
||||
self._pending_used = True
|
||||
@@ -265,7 +285,7 @@ class Happy(unittest.TestCase):
|
||||
self.assertEqual(rep["pending"][0]["name"], "bash")
|
||||
self.assertTrue(any(l.startswith("os-apply: REPAIR ") for l in f.logs), "the repair line must always print")
|
||||
self.assertTrue(any(l.startswith("os-apply: DONE rc=0") for l in f.logs))
|
||||
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
|
||||
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2] and "--print-uris" not in c[2]]
|
||||
self.assertTrue(inst and "Dpkg::Options::=--force-confold" in inst[0][2], "must keep existing config files")
|
||||
|
||||
def test_already_current_is_a_no_op(self):
|
||||
@@ -335,7 +355,7 @@ class Refusals(unittest.TestCase):
|
||||
self.assertEqual(rc, 2, rep)
|
||||
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||
self.assertTrue(any(l.startswith(f"os-apply: REFUSED: {code} ") for l in f.logs), f.logs)
|
||||
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
|
||||
inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2] and "--print-uris" not in c[2]]
|
||||
self.assertEqual(inst, [], "a refusal must install nothing")
|
||||
return rep
|
||||
|
||||
@@ -446,6 +466,24 @@ class Refusals(unittest.TestCase):
|
||||
f.free = 100 * 1024 * 1024
|
||||
self.refused(f, "R8")
|
||||
|
||||
# R-865: the download is the real one. 2 GB of URIs, 5 GB free: 5 GB < 3 x 2 GB -> R8, with the size in the line.
|
||||
# COMPANION RED-PROOF: put "-s" back into download_bytes -> 0 B -> no refusal -> this test fails.
|
||||
def test_R8_counts_the_real_download(self):
|
||||
f = Fake()
|
||||
f.free = 5 * 1024 ** 3
|
||||
f.extra_uris = "'http://deb.debian.org/debian/pool/main/b/big/big_1_amd64.deb' big_1_amd64.deb 2000000000 MD5Sum:x\n"
|
||||
rep = self.refused(f, "R8")
|
||||
self.assertIn("download 2004347372 B", str(rep))
|
||||
|
||||
def test_download_bytes_never_simulates(self):
|
||||
f = Fake()
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
calls = [c[2] for c in f.calls if c[0] == "guest" and "--print-uris" in c[2]]
|
||||
self.assertTrue(calls, "download_bytes was never called")
|
||||
for c in calls:
|
||||
self.assertNotIn("-s", c, f"--print-uris with -s prints no URIs: {c}")
|
||||
|
||||
def test_R9_guest_locked_by_a_backup(self):
|
||||
f = Fake()
|
||||
f.files["/etc/pve/lxc/9201.conf"] = CONF_OK + "lock: backup\n"
|
||||
@@ -804,6 +842,26 @@ class DockerLane(unittest.TestCase):
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rep["plan"]["upgrade"], 0, "an older version on an unsigned step is 'already', never installed")
|
||||
|
||||
def test_step_restarts_only_the_socket_users(self):
|
||||
# R-858: after an engine step, ONLY the container that mounts the docker socket is restarted (here the controller)
|
||||
f = docker_fake(signed=signed_job())
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.restarted_containers, ["felhom-controller"])
|
||||
self.assertEqual(rep["socket_restarted"], ["felhom-controller"])
|
||||
|
||||
def test_no_install_restarts_nothing(self):
|
||||
f = docker_fake(signed=signed_job())
|
||||
f.installed.update({"docker-ce": "5:29.8.2-1~debian.13~trixie", "containerd.io": "2.3.6-1~debian.13~trixie"})
|
||||
rc, rep = run(f)
|
||||
self.assertFalse(hasattr(f, "restarted_containers"), "nothing installed -> no container restart")
|
||||
|
||||
def test_health_says_when_the_controller_cannot_reach_docker(self):
|
||||
f = docker_fake(signed=signed_job())
|
||||
f.controller_blind = True
|
||||
rc, rep = run(f)
|
||||
self.assertFalse(rep["health_after"]["controller_docker_ok"])
|
||||
|
||||
def test_health_carries_container_ids(self):
|
||||
f = docker_fake(signed=signed_job())
|
||||
rc, rep = run(f)
|
||||
@@ -934,3 +992,88 @@ class RealSignatureCheck(unittest.TestCase):
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
class UnsentReport(unittest.TestCase):
|
||||
"""R-868 (v0.144.0): an apply pass keeps its report on disk until the agent has sent it.
|
||||
COMPANION RED-PROOF: drop the r.save_report call in main() -> test_apply_keeps_a_copy fails."""
|
||||
|
||||
def test_apply_keeps_a_copy_with_the_agents_ids(self):
|
||||
f = Fake()
|
||||
f.plan.update(run_id="20261005T0257-ab12", trigger="debug", ring=0)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(len(f.saved_reports), 1, "an apply pass must keep its report on disk")
|
||||
path, saved = f.saved_reports[0]
|
||||
self.assertEqual(path, PLAN)
|
||||
self.assertEqual(saved, rep, "the copy is the report the agent would have read")
|
||||
self.assertEqual((saved["run_id"], saved["trigger"], saved["ring"]), ("20261005T0257-ab12", "debug", 0))
|
||||
|
||||
def test_a_refusal_is_kept_too(self):
|
||||
f = Fake()
|
||||
f.free = 1
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 2)
|
||||
self.assertEqual(f.saved_reports[0][1]["refused"]["code"], "R8")
|
||||
|
||||
def test_other_modes_keep_nothing(self):
|
||||
f = Fake()
|
||||
f.plan["mode"] = "health"
|
||||
run(f)
|
||||
self.assertEqual(f.saved_reports, [])
|
||||
|
||||
def test_odd_ids_are_dropped_not_trusted(self):
|
||||
f = Fake()
|
||||
f.plan.update(run_id="../../etc/x", trigger="Night; rm", ring=True)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
for k in ("run_id", "trigger", "ring"):
|
||||
self.assertNotIn(k, rep)
|
||||
|
||||
|
||||
class SaveReportOnDisk(unittest.TestCase):
|
||||
"""The real Runner.save_report on a temp dir: root writes into the AGENT's directory, so a symlink must never
|
||||
be followed — neither for the directory nor for the file name."""
|
||||
|
||||
def setUp(self):
|
||||
import tempfile
|
||||
self.tmp = tempfile.mkdtemp()
|
||||
self.dir = os.path.join(self.tmp, "os")
|
||||
os.mkdir(self.dir)
|
||||
self.prev = osapply.PLAN_DIR
|
||||
osapply.PLAN_DIR = self.dir
|
||||
self.r = osapply.Runner()
|
||||
self.r.agent_uid = lambda: os.getuid()
|
||||
|
||||
def tearDown(self):
|
||||
import shutil
|
||||
osapply.PLAN_DIR = self.prev
|
||||
shutil.rmtree(self.tmp)
|
||||
|
||||
def test_writes_0600_next_to_the_plan(self):
|
||||
p = self.r.save_report(os.path.join(self.dir, "plan-r1-guest-apply.json"), {"mode": "apply"})
|
||||
self.assertEqual(p, os.path.join(self.dir, "report-r1-guest-apply.json"))
|
||||
st = os.stat(p)
|
||||
self.assertEqual(statmod.S_IMODE(st.st_mode), 0o600)
|
||||
with open(p) as fh:
|
||||
self.assertEqual(json.load(fh), {"mode": "apply"})
|
||||
|
||||
def test_a_symlink_at_the_name_is_replaced_not_followed(self):
|
||||
victim = os.path.join(self.tmp, "victim")
|
||||
with open(victim, "w") as fh:
|
||||
fh.write("untouched")
|
||||
os.symlink(victim, os.path.join(self.dir, "report-r2-guest-apply.json"))
|
||||
self.r.save_report(os.path.join(self.dir, "plan-r2-guest-apply.json"), {"mode": "apply"})
|
||||
with open(victim) as fh:
|
||||
self.assertEqual(fh.read(), "untouched")
|
||||
self.assertFalse(os.path.islink(os.path.join(self.dir, "report-r2-guest-apply.json")))
|
||||
|
||||
def test_a_symlinked_directory_is_refused(self):
|
||||
real = os.path.join(self.tmp, "elsewhere")
|
||||
os.mkdir(real)
|
||||
link = os.path.join(self.tmp, "linked")
|
||||
os.symlink(real, link)
|
||||
osapply.PLAN_DIR = link
|
||||
with self.assertRaises(OSError):
|
||||
self.r.save_report(os.path.join(link, "plan-r3-guest-apply.json"), {"mode": "apply"})
|
||||
self.assertEqual(os.listdir(real), [])
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-840: the agent reports the bundle record itself — "none" when no bundle ever reached the box (so an old wrapper
|
||||
// cannot hide that), "unknown" when it cannot be read, else the version and sha the root wrapper recorded.
|
||||
func TestReadBundleRecord(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "config-bundle.json")
|
||||
if got := string(readBundleRecord(p)); got != `{"version":"none"}` {
|
||||
t.Fatalf("absent: %s", got)
|
||||
}
|
||||
os.WriteFile(p, []byte("{broken"), 0o644)
|
||||
if got := string(readBundleRecord(p)); got != `{"version":"unknown"}` {
|
||||
t.Fatalf("broken: %s", got)
|
||||
}
|
||||
os.WriteFile(p, []byte(`{"format":1,"agent_version":"0.143.0","bundle_sha256":"abc","installed_at":"2026-10-04T20:00:00Z","authority":"signed","files":{"/x":"y"}}`), 0o644)
|
||||
if got := string(readBundleRecord(p)); got != `{"authority":"signed","bundle_sha256":"abc","installed_at":"2026-10-04T20:00:00Z","version":"0.143.0"}` {
|
||||
t.Fatalf("record: %s", got)
|
||||
}
|
||||
}
|
||||
@@ -111,6 +111,7 @@ type Collector struct {
|
||||
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
|
||||
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
|
||||
system SystemReporter // R-852: the box versions (nil → API fields only)
|
||||
bundleRecordPath string // R-840: test seam; "" = BundleRecordPath
|
||||
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
|
||||
hostID string
|
||||
agentVersion string
|
||||
@@ -262,6 +263,33 @@ func (c *Collector) SetSystemReporter(r SystemReporter) *Collector {
|
||||
return c
|
||||
}
|
||||
|
||||
// BundleRecordPath is the root-owned record felhom-os-apply writes after a config bundle installs (R-840).
|
||||
const BundleRecordPath = "/etc/felhom/config-bundle.json"
|
||||
|
||||
// readBundleRecord returns the record's summary: version/sha/installed_at, "none" when the file is absent, "unknown"
|
||||
// when it cannot be read or parsed (never a guess).
|
||||
func readBundleRecord(path string) json.RawMessage {
|
||||
if path == "" {
|
||||
path = BundleRecordPath
|
||||
}
|
||||
b, err := os.ReadFile(path)
|
||||
if os.IsNotExist(err) {
|
||||
return json.RawMessage(`{"version":"none"}`)
|
||||
}
|
||||
var rec struct {
|
||||
AgentVersion string `json:"agent_version"`
|
||||
BundleSHA256 string `json:"bundle_sha256"`
|
||||
InstalledAt string `json:"installed_at"`
|
||||
Authority string `json:"authority"`
|
||||
}
|
||||
if err != nil || json.Unmarshal(b, &rec) != nil || rec.AgentVersion == "" {
|
||||
return json.RawMessage(`{"version":"unknown"}`)
|
||||
}
|
||||
out, _ := json.Marshal(map[string]string{"version": rec.AgentVersion, "bundle_sha256": rec.BundleSHA256,
|
||||
"installed_at": rec.InstalledAt, "authority": rec.Authority})
|
||||
return out
|
||||
}
|
||||
|
||||
func unknownIfEmpty(s string) string {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return "unknown"
|
||||
@@ -273,6 +301,7 @@ func unknownIfEmpty(s string) string {
|
||||
func (c *Collector) systemInfo(ctx context.Context, ns proxmox.NodeStatus) *SystemInfo {
|
||||
si := &SystemInfo{PVEVersion: unknownIfEmpty(ns.PVEVersion), KernelVersion: unknownIfEmpty(ns.KVersion),
|
||||
ReadAt: c.now().Format(time.RFC3339)}
|
||||
si.ConfigBundle = readBundleRecord(c.bundleRecordPath)
|
||||
if c.system == nil {
|
||||
si.FactsError = "no facts reader wired"
|
||||
return si
|
||||
|
||||
@@ -255,6 +255,10 @@ type SystemInfo struct {
|
||||
Facts json.RawMessage `json:"facts,omitempty"`
|
||||
FactsError string `json:"facts_error,omitempty"`
|
||||
ReadAt string `json:"read_at"`
|
||||
// ConfigBundle is the box's root-owned config bundle record (R-840, agent v0.143.0), read by the agent itself from
|
||||
// /etc/felhom/config-bundle.json (0644): {"version":"none"} on a box no bundle reached, so an OLD wrapper cannot
|
||||
// hide it. The wrapper's facts carry the same record plus the drift (files changed by hand).
|
||||
ConfigBundle json.RawMessage `json:"config_bundle,omitempty"`
|
||||
}
|
||||
|
||||
// HostMetrics is the host block, sourced from proxmox NodeStatus.
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
)
|
||||
|
||||
// OpConfigUpdate is the signed op that brings a box's ROOT-OWNED files (sudoers, wrappers, units) to a release's config
|
||||
// bundle (R-840, `11` §5.4.2). The params pin the agent version and the bundle's sha256; the root wrapper re-verifies
|
||||
// the signature, the host binding, the nonce and the sha ITSELF — this executor is only the courier.
|
||||
const OpConfigUpdate = "agent_config_update"
|
||||
|
||||
// BundleFileName is the bundle's name beside the binary in the Gitea generic package felhom-agent/<version>/.
|
||||
const BundleFileName = "felhom-config-bundle.json"
|
||||
|
||||
var (
|
||||
bundleVersionRe = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$`)
|
||||
bundleSHARe = regexp.MustCompile(`^[0-9a-f]{64}$`)
|
||||
)
|
||||
|
||||
// ConfigUpdateParams are the signed params (the wrapper reads the same two names out of the signed blob).
|
||||
type ConfigUpdateParams struct {
|
||||
AgentVersion string `json:"agent_version"`
|
||||
BundleSHA256 string `json:"bundle_sha256"`
|
||||
}
|
||||
|
||||
// BundleURL derives the bundle's URL from the agent binary's URL template (".../felhom-agent/{version}/felhom-agent").
|
||||
func BundleURL(binaryTemplate, version string) (string, error) {
|
||||
if !strings.HasSuffix(binaryTemplate, "/felhom-agent") || !strings.Contains(binaryTemplate, "{version}") {
|
||||
return "", fmt.Errorf("cannot derive the bundle URL from %q (want …/{version}/felhom-agent)", binaryTemplate)
|
||||
}
|
||||
t := strings.TrimSuffix(binaryTemplate, "felhom-agent") + BundleFileName
|
||||
return strings.ReplaceAll(t, "{version}", version), nil
|
||||
}
|
||||
|
||||
// ConfigUpdateExecutor runs a verified agent_config_update (signedjobs.Executor).
|
||||
type ConfigUpdateExecutor struct {
|
||||
Leg *Leg
|
||||
URLTemplate string // the agent binary's template (config.SelfUpdate.URLTemplate)
|
||||
Username string
|
||||
Token string
|
||||
HTTPClient *http.Client
|
||||
// AfterInstall runs after a bundle installed (the capability re-probe; nil = none).
|
||||
AfterInstall func(ctx context.Context)
|
||||
}
|
||||
|
||||
// Execute implements signedjobs.Executor.
|
||||
func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
|
||||
if op != OpConfigUpdate {
|
||||
return signedjobs.ErrNoExecutor
|
||||
}
|
||||
so, ok := signedjobs.SignedOpFrom(ctx)
|
||||
if !ok {
|
||||
return fmt.Errorf("agent_config_update: no signed envelope in the context — the wrapper could not verify it")
|
||||
}
|
||||
var p ConfigUpdateParams
|
||||
if err := json.Unmarshal(params, &p); err != nil {
|
||||
return fmt.Errorf("agent_config_update: bad params: %w", err)
|
||||
}
|
||||
if !bundleVersionRe.MatchString(p.AgentVersion) || !bundleSHARe.MatchString(p.BundleSHA256) {
|
||||
return fmt.Errorf("agent_config_update: params must pin agent_version (semver) and bundle_sha256 (64 hex)")
|
||||
}
|
||||
lg := e.Leg.log().With("op", OpConfigUpdate, "agent_version", p.AgentVersion)
|
||||
url, err := BundleURL(e.URLTemplate, p.AgentVersion)
|
||||
if err != nil {
|
||||
return fmt.Errorf("agent_config_update: %w", err)
|
||||
}
|
||||
dir := e.Leg.PlanDir
|
||||
if dir == "" {
|
||||
dir = DefaultPlanDir
|
||||
}
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return fmt.Errorf("agent_config_update: plan dir: %w", err)
|
||||
}
|
||||
path := filepath.Join(dir, "bundle-"+p.AgentVersion+".json")
|
||||
start := time.Now()
|
||||
got, err := e.download(ctx, url, path)
|
||||
if err != nil {
|
||||
_ = os.Remove(path)
|
||||
return fmt.Errorf("agent_config_update: download %s: %w", url, err)
|
||||
}
|
||||
defer os.Remove(path)
|
||||
// The agent's own check is a courtesy (an early, clear error); the wrapper's is the gate.
|
||||
if got != p.BundleSHA256 {
|
||||
return fmt.Errorf("agent_config_update: the downloaded bundle's sha256 is %s, the signed job pins %s — nothing installed", got, p.BundleSHA256)
|
||||
}
|
||||
lg.Info("osupdate: config bundle downloaded; handing it to the root wrapper", "sha256", got[:16], "duration_ms", time.Since(start).Milliseconds())
|
||||
runID := "bundle-" + e.Leg.now().UTC().Format("20060102T150405Z")
|
||||
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-" + p.AgentVersion, "layer": LayerHost,
|
||||
"mode": "bundle", "bundle": path,
|
||||
"signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(so.Blob), "sig": string(so.Sig)}})
|
||||
if err != nil {
|
||||
return fmt.Errorf("agent_config_update: %w", err)
|
||||
}
|
||||
if wr.refused() {
|
||||
lg.Warn("osupdate: config bundle REFUSED by the wrapper — nothing changed", "refused", string(wr.Refused))
|
||||
return fmt.Errorf("agent_config_update: refused: %s", wr.Refused)
|
||||
}
|
||||
if wr.failed() {
|
||||
lg.Error("osupdate: config bundle FAILED — the wrapper put the previous files back", "failed", string(wr.Failed), "bundle", string(wr.Bundle))
|
||||
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
|
||||
}
|
||||
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
|
||||
if e.AfterInstall != nil {
|
||||
e.AfterInstall(ctx)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
|
||||
hc := e.HTTPClient
|
||||
if hc == nil {
|
||||
hc = &http.Client{Timeout: 2 * time.Minute}
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if e.Username != "" || e.Token != "" {
|
||||
req.SetBasicAuth(e.Username, e.Token)
|
||||
}
|
||||
resp, err := hc.Do(req)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return "", fmt.Errorf("HTTP %d", resp.StatusCode)
|
||||
}
|
||||
f, err := os.OpenFile(dest, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
h := sha256.New()
|
||||
// 4 MB is the wrapper's own limit; read one byte more so an oversized bundle fails there, visibly.
|
||||
if _, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 4*1024*1024+1)); err != nil {
|
||||
f.Close()
|
||||
return "", err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), nil
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
)
|
||||
|
||||
// bundleWrapper plays felhom-os-apply for mode "bundle": it records the plan and the bundle file's bytes AT CALL TIME
|
||||
// (the executor deletes the file afterwards), and answers with rep.
|
||||
type bundleWrapper struct {
|
||||
t *testing.T
|
||||
rep string
|
||||
plans []map[string]any
|
||||
bodies [][]byte
|
||||
}
|
||||
|
||||
func (b *bundleWrapper) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
if name != WrapperPath || len(args) != 2 || args[0] != "--plan" {
|
||||
b.t.Fatalf("unexpected command %s %v", name, args)
|
||||
}
|
||||
raw, err := os.ReadFile(args[1])
|
||||
if err != nil {
|
||||
b.t.Fatal(err)
|
||||
}
|
||||
var plan map[string]any
|
||||
_ = json.Unmarshal(raw, &plan)
|
||||
b.plans = append(b.plans, plan)
|
||||
body, _ := os.ReadFile(plan["bundle"].(string))
|
||||
b.bodies = append(b.bodies, body)
|
||||
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
|
||||
}
|
||||
|
||||
func serveBundle(t *testing.T, body []byte) (*httptest.Server, string) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/felhom-agent/0.143.0/"+BundleFileName {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write(body)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
sum := sha256.Sum256(body)
|
||||
return srv, hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
func bundleExec(t *testing.T, w *bundleWrapper, srvURL string) (ConfigUpdateExecutor, *bool) {
|
||||
l, _ := newLeg(t, &fakeWrapper{t: t}, nil)
|
||||
l.Runner = w
|
||||
called := false
|
||||
return ConfigUpdateExecutor{Leg: l, URLTemplate: srvURL + "/felhom-agent/{version}/felhom-agent",
|
||||
AfterInstall: func(context.Context) { called = true }}, &called
|
||||
}
|
||||
|
||||
func signedCtx() context.Context {
|
||||
return signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"agent_config_update"}`), Sig: []byte("SIG")})
|
||||
}
|
||||
|
||||
func params(v, sha string) json.RawMessage {
|
||||
p, _ := json.Marshal(ConfigUpdateParams{AgentVersion: v, BundleSHA256: sha})
|
||||
return p
|
||||
}
|
||||
|
||||
// The courier hands the wrapper the bundle bytes it downloaded and the RAW signed envelope (the wrapper verifies both
|
||||
// itself), then runs the capability probe. Red-proof: drop "signed" from the plan → the plan check below fails.
|
||||
func TestConfigUpdate_PassesBundleAndEnvelopeToTheWrapper(t *testing.T) {
|
||||
body := []byte(`{"format":1,"agent_version":"0.143.0","files":[]}`)
|
||||
srv, sha := serveBundle(t, body)
|
||||
w := &bundleWrapper{t: t, rep: `{"mode":"bundle","bundle":{"agent_version":"0.143.0","written":["/etc/sudoers.d/felhom-agent"]}}`}
|
||||
e, called := bundleExec(t, w, srv.URL)
|
||||
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := w.plans[0]
|
||||
sg, _ := p["signed"].(map[string]any)
|
||||
if p["mode"] != "bundle" || p["layer"] != "host" || sg == nil || sg["sig"] != "SIG" ||
|
||||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"agent_config_update"}`)) {
|
||||
t.Fatalf("plan = %v", p)
|
||||
}
|
||||
if string(w.bodies[0]) != string(body) || !strings.HasSuffix(p["bundle"].(string), "/bundle-0.143.0.json") {
|
||||
t.Fatalf("the wrapper got %q at %v", w.bodies[0], p["bundle"])
|
||||
}
|
||||
if !*called {
|
||||
t.Fatal("the capability probe must run after an install")
|
||||
}
|
||||
if _, err := os.Stat(p["bundle"].(string)); !os.IsNotExist(err) {
|
||||
t.Fatal("the downloaded bundle must be removed after the call")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigUpdate_WrongShaNeverReachesTheWrapper(t *testing.T) {
|
||||
srv, _ := serveBundle(t, []byte("tampered"))
|
||||
w := &bundleWrapper{t: t}
|
||||
e, called := bundleExec(t, w, srv.URL)
|
||||
err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", strings.Repeat("a", 64)))
|
||||
if err == nil || len(w.plans) != 0 || *called {
|
||||
t.Fatalf("err=%v plans=%d", err, len(w.plans))
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigUpdate_RefusedAndFailedAreErrors(t *testing.T) {
|
||||
for _, rep := range []string{`{"refused":{"code":"R17","reason":"trust root"}}`, `{"failed":{"rc":3},"bundle":{"rolled_back":["/x"]}}`} {
|
||||
srv, sha := serveBundle(t, []byte("{}"))
|
||||
w := &bundleWrapper{t: t, rep: rep}
|
||||
e, called := bundleExec(t, w, srv.URL)
|
||||
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err == nil || *called {
|
||||
t.Fatalf("%s: err=%v called=%v", rep, err, *called)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigUpdate_GuardsBeforeAnyDownload(t *testing.T) {
|
||||
w := &bundleWrapper{t: t}
|
||||
e, _ := bundleExec(t, w, "http://127.0.0.1:1")
|
||||
if err := e.Execute(signedCtx(), "agent_update", nil); !errors.Is(err, signedjobs.ErrNoExecutor) {
|
||||
t.Fatalf("another op must pass through the chain: %v", err)
|
||||
}
|
||||
if err := e.Execute(context.Background(), OpConfigUpdate, params("0.143.0", strings.Repeat("a", 64))); err == nil {
|
||||
t.Fatal("no envelope must refuse")
|
||||
}
|
||||
for _, p := range []json.RawMessage{params("0.143", strings.Repeat("a", 64)), params("0.143.0", "ABC"), json.RawMessage(`nope`)} {
|
||||
if err := e.Execute(signedCtx(), OpConfigUpdate, p); err == nil {
|
||||
t.Fatalf("bad params %s must refuse", p)
|
||||
}
|
||||
}
|
||||
if len(w.plans) != 0 {
|
||||
t.Fatal("no wrapper call on a refusal")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBundleURL(t *testing.T) {
|
||||
u, err := BundleURL("https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/{version}/felhom-agent", "0.143.0")
|
||||
if err != nil || u != "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.143.0/felhom-config-bundle.json" {
|
||||
t.Fatalf("%s %v", u, err)
|
||||
}
|
||||
if _, err := BundleURL("https://example/felhom-agent-{version}.bin", "0.143.0"); err == nil {
|
||||
t.Fatal("an underivable template must refuse")
|
||||
}
|
||||
}
|
||||
|
||||
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return b.Run(ctx, name, args...)
|
||||
}
|
||||
@@ -73,6 +73,9 @@ func (e DockerStepExecutor) Execute(ctx context.Context, op string, params json.
|
||||
// RunDockerSigned is one signed Docker step (ring 1 or an undo): live-restore first (decision 87, a no-op when on),
|
||||
// then the docker layer with the signed envelope, which the wrapper verifies itself.
|
||||
func (l *Leg) RunDockerSigned(ctx context.Context, vmid int, p DockerStepParams, blob []byte, sig string) Report {
|
||||
unlock := l.lockPass(true)
|
||||
defer unlock()
|
||||
l.sendUnsentLocked(ctx) // R-868
|
||||
runID := l.now().UTC().Format("20060102T150405Z")
|
||||
lg := l.log().With("run", runID, "vmid", vmid, "trigger", "signed", "release", p.ReleaseID, "undo", p.Undo)
|
||||
if err := l.EnsureLiveRestore(ctx, runID, vmid); err != nil {
|
||||
|
||||
@@ -75,6 +75,9 @@ type Health struct {
|
||||
NetworkOK bool `json:"network_ok"`
|
||||
Controller string `json:"controller"`
|
||||
Containers map[string]Container `json:"containers"`
|
||||
// ControllerDockerOK: the controller reaches the engine from INSIDE its container (R-858, wrapper ≥ v0.142.1;
|
||||
// nil from an older wrapper = not checked). Its own health check stayed "healthy" while it was blind.
|
||||
ControllerDockerOK *bool `json:"controller_docker_ok,omitempty"`
|
||||
HostServices map[string]string `json:"host_services,omitempty"`
|
||||
GuestRunning *bool `json:"guest_running,omitempty"`
|
||||
Guest *Health `json:"guest,omitempty"`
|
||||
@@ -102,6 +105,14 @@ type WrapperReport struct {
|
||||
Undo bool `json:"undo"`
|
||||
LiveRestore json.RawMessage `json:"live_restore"`
|
||||
Facts json.RawMessage `json:"facts"`
|
||||
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
|
||||
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
|
||||
// agent process that started the pass. ReleaseID / VMID were always in the report.
|
||||
RunID string `json:"run_id"`
|
||||
Trigger string `json:"trigger"`
|
||||
Ring *int `json:"ring"`
|
||||
ReleaseID string `json:"release_id"`
|
||||
VMID int `json:"vmid"`
|
||||
}
|
||||
|
||||
func (w WrapperReport) refused() bool { return len(w.Refused) > 0 && string(w.Refused) != "null" }
|
||||
@@ -132,6 +143,8 @@ type Report struct {
|
||||
DockerEngine string `json:"docker_engine,omitempty"` // docker layer: the engine after the step
|
||||
Authority string `json:"authority,omitempty"` // docker layer: ring0 | signed
|
||||
Undo bool `json:"undo,omitempty"` // docker layer: a signed undo (downgrade)
|
||||
|
||||
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
|
||||
}
|
||||
|
||||
// Reporter posts a report to the hub (*hub.Client).
|
||||
@@ -158,14 +171,65 @@ type Leg struct {
|
||||
block *hub.WireOSUpdate
|
||||
}
|
||||
|
||||
// planFile / reportFile: the plan the agent writes and the copy of the report the wrapper keeps beside it (R-868).
|
||||
func planFile(dir, runID, layer, mode string) string {
|
||||
return filepath.Join(dir, fmt.Sprintf("plan-%s-%s-%s.json", runID, layer, mode))
|
||||
}
|
||||
|
||||
func reportFile(dir, runID, layer, mode string) string {
|
||||
return filepath.Join(dir, fmt.Sprintf("report-%s-%s-%s.json", runID, layer, mode))
|
||||
}
|
||||
|
||||
func (l *Leg) planDir() string {
|
||||
if l.PlanDir == "" {
|
||||
return DefaultPlanDir
|
||||
}
|
||||
return l.PlanDir
|
||||
}
|
||||
|
||||
// OnDesiredState stores the hub's os_update block (desired.RawConsumer — store only, never block).
|
||||
func (l *Leg) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
|
||||
if resp == nil {
|
||||
return
|
||||
}
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.block = resp.DesiredState.OSUpdate
|
||||
l.mu.Unlock()
|
||||
l.saveBlock(resp.DesiredState.OSUpdate)
|
||||
}
|
||||
|
||||
// SavedBlockFile is the hub's newest os_update block as the daemon last received it (R-866, v0.144.0): the debug
|
||||
// pass falls back to it when the hub cannot be reached, and says so.
|
||||
const SavedBlockFile = "os-update-block.json"
|
||||
|
||||
type savedBlock struct {
|
||||
SavedAt time.Time `json:"saved_at"`
|
||||
Block *hub.WireOSUpdate `json:"block"`
|
||||
}
|
||||
|
||||
func (l *Leg) saveBlock(b *hub.WireOSUpdate) {
|
||||
dir := l.planDir()
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(savedBlock{SavedAt: l.now().UTC(), Block: b})
|
||||
tmp := filepath.Join(dir, SavedBlockFile+".tmp")
|
||||
if err := os.WriteFile(tmp, body, 0o600); err == nil {
|
||||
_ = os.Rename(tmp, filepath.Join(dir, SavedBlockFile))
|
||||
}
|
||||
}
|
||||
|
||||
// LoadSavedBlock reads the block the daemon saved (R-866). ok=false: none saved yet.
|
||||
func LoadSavedBlock(dir string) (b *hub.WireOSUpdate, savedAt time.Time, ok bool) {
|
||||
raw, err := os.ReadFile(filepath.Join(dir, SavedBlockFile))
|
||||
if err != nil {
|
||||
return nil, time.Time{}, false
|
||||
}
|
||||
var s savedBlock
|
||||
if json.Unmarshal(raw, &s) != nil {
|
||||
return nil, time.Time{}, false
|
||||
}
|
||||
return s.Block, s.SavedAt, true
|
||||
}
|
||||
|
||||
// Block returns the newest os_update block. No block (an older hub, or nothing fetched yet) = ring 1, ON, no
|
||||
@@ -240,6 +304,9 @@ func HealthVerdict(before, after *Health) (bool, string) {
|
||||
if after.Controller != "healthy" {
|
||||
return false, "the controller is " + after.Controller
|
||||
}
|
||||
if after.ControllerDockerOK != nil && !*after.ControllerDockerOK {
|
||||
return false, "the controller cannot reach Docker (it holds an old socket — R-858)"
|
||||
}
|
||||
if before == nil {
|
||||
return true, ""
|
||||
}
|
||||
@@ -345,15 +412,12 @@ func DockerHealthVerdict(before, after *Health, wantEngine, gotEngine string) (b
|
||||
|
||||
// call writes the plan and runs the wrapper once.
|
||||
func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (WrapperReport, error) {
|
||||
dir := l.PlanDir
|
||||
if dir == "" {
|
||||
dir = DefaultPlanDir
|
||||
}
|
||||
dir := l.planDir()
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return WrapperReport{}, fmt.Errorf("osupdate: plan dir: %w", err)
|
||||
}
|
||||
b, _ := json.Marshal(plan)
|
||||
path := filepath.Join(dir, fmt.Sprintf("plan-%s-%s-%s.json", runID, plan["layer"], plan["mode"]))
|
||||
path := planFile(dir, runID, fmt.Sprint(plan["layer"]), fmt.Sprint(plan["mode"]))
|
||||
if err := os.WriteFile(path, b, 0o600); err != nil {
|
||||
return WrapperReport{}, fmt.Errorf("osupdate: write plan: %w", err)
|
||||
}
|
||||
@@ -387,6 +451,9 @@ type Pass struct {
|
||||
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
|
||||
// only, after good earlier steps) the Docker engine set. trigger is "night" or "debug".
|
||||
func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
|
||||
unlock := l.lockPass(true)
|
||||
defer unlock()
|
||||
l.sendUnsentLocked(ctx) // R-868: a report a killed agent never sent goes first
|
||||
g, h := l.runFast(ctx, vmid, trigger)
|
||||
p := Pass{Guest: g, Host: h}
|
||||
if g.Outcome == "skipped" {
|
||||
@@ -514,7 +581,8 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
||||
lg.Info("osupdate: START", "enabled", blk.Enabled, "release", rel.ID)
|
||||
|
||||
plan := map[string]any{"release_id": rel.ID, "layer": layer, "lane": lane, "vmid": vmid, "snapshot": rel.Snapshot,
|
||||
"packages": []Package{}, "mode": "apply", "select": "listed"}
|
||||
"packages": []Package{}, "mode": "apply", "select": "listed",
|
||||
"run_id": runID, "trigger": trigger, "ring": blk.Ring} // R-868: echoed into the wrapper's kept copy
|
||||
if rel.ID == "" {
|
||||
plan["release_id"] = "none"
|
||||
}
|
||||
@@ -547,6 +615,9 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
||||
}
|
||||
rep.Mode = plan["mode"].(string)
|
||||
wr, err := l.call(ctx, runID, plan)
|
||||
if rep.Mode == "apply" {
|
||||
rep.unsent = reportFile(l.planDir(), runID, layer, rep.Mode) // the wrapper kept a copy (R-868)
|
||||
}
|
||||
switch {
|
||||
case err != nil:
|
||||
rep.Outcome, rep.HealthReason = "failed", err.Error()
|
||||
@@ -677,7 +748,11 @@ func (l *Leg) finish(ctx context.Context, lg *slog.Logger, rep Report) Report {
|
||||
rctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
|
||||
defer cancel()
|
||||
if err := l.Hub.PostOSReport(rctx, body); err != nil {
|
||||
lg.Warn("osupdate: reporting to the hub failed (the run itself is done)", "err", err)
|
||||
lg.Warn("osupdate: reporting to the hub failed (the run itself is done; the kept copy is sent at the next start or pass)", "err", err)
|
||||
} else if rep.unsent != "" {
|
||||
if rerr := os.Remove(rep.unsent); rerr != nil && !os.IsNotExist(rerr) {
|
||||
lg.Warn("osupdate: could not delete the sent report's kept copy (it may be sent twice)", "path", rep.unsent, "err", rerr)
|
||||
}
|
||||
}
|
||||
}
|
||||
return rep
|
||||
|
||||
@@ -3,6 +3,7 @@ package osupdate
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
@@ -21,6 +22,7 @@ type fakeWrapper struct {
|
||||
applyRep map[string]WrapperReport // per layer
|
||||
healthSeq map[string][]*Health // per layer: answers to successive "health" calls
|
||||
plans []map[string]any
|
||||
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
|
||||
}
|
||||
|
||||
func yes() *bool { b := true; return &b }
|
||||
@@ -72,6 +74,22 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
|
||||
rep.Health = ok
|
||||
}
|
||||
}
|
||||
if f.keep && plan["mode"] == "apply" {
|
||||
kept := rep
|
||||
kept.Layer, kept.RunID, _ = layer, fmt.Sprint(plan["run_id"]), 0
|
||||
if tr, ok := plan["trigger"].(string); ok {
|
||||
kept.Trigger = tr
|
||||
}
|
||||
if r, ok := plan["ring"].(float64); ok {
|
||||
ri := int(r)
|
||||
kept.Ring = &ri
|
||||
}
|
||||
kb, _ := json.Marshal(kept)
|
||||
dst := filepath.Join(filepath.Dir(args[1]), "report-"+strings.TrimPrefix(filepath.Base(args[1]), "plan-"))
|
||||
if err := os.WriteFile(dst, kb, 0o600); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
out, _ := json.Marshal(rep)
|
||||
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
|
||||
}
|
||||
@@ -346,7 +364,7 @@ func TestWrapperSuite(t *testing.T) {
|
||||
t.Skip("python3 not available")
|
||||
}
|
||||
// the OS wrapper and (agent v0.142.0) the crash guard — both root programs in configs/ with their own suites
|
||||
for _, suite := range []string{"../../configs/test_felhom_os_apply.py", "../../configs/test_felhom_crash_guard.py"} {
|
||||
for _, suite := range []string{"../../configs/test_felhom_os_apply.py", "../../configs/test_felhom_crash_guard.py", "../../configs/test_felhom_config_bundle.py"} {
|
||||
cmd := exec.Command(py, "-B", suite)
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
@@ -465,3 +483,24 @@ func TestDocker_ChangedIDIsHealthFailed(t *testing.T) {
|
||||
t.Fatalf("docker = %+v", p.Docker)
|
||||
}
|
||||
}
|
||||
|
||||
// R-858: a controller that cannot reach Docker fails the health rule even though its own check says healthy.
|
||||
// Red-proof: drop the ControllerDockerOK check in HealthVerdict and this fails.
|
||||
func TestHealthVerdict_ControllerBlindToDockerFails(t *testing.T) {
|
||||
no, yes2 := false, true
|
||||
after := guestOK()
|
||||
after.ControllerDockerOK = &no
|
||||
if ok, why := HealthVerdict(guestOK(), after); ok || !strings.Contains(why, "R-858") {
|
||||
t.Fatalf("a blind controller passed: %v %q", ok, why)
|
||||
}
|
||||
if ok, _ := DockerHealthVerdict(guestOK(), after, "", ""); ok {
|
||||
t.Fatal("the Docker rule passed a blind controller")
|
||||
}
|
||||
after.ControllerDockerOK = &yes2
|
||||
if ok, why := HealthVerdict(guestOK(), after); !ok {
|
||||
t.Fatalf("a seeing controller failed: %s", why)
|
||||
}
|
||||
if ok, _ := HealthVerdict(guestOK(), guestOK()); !ok {
|
||||
t.Fatal("an older wrapper (no field) must not fail")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// ── R-868 (v0.144.0): a pass whose agent was killed still reports ─────────────────────────────────────
|
||||
//
|
||||
// MEASURED 2026-10-05 02:57 UTC on demo-hp (night drill A5): the debug pass and the agent daemon were kill -9-ed while
|
||||
// apt-get ran. The root wrapper (its own process under sudo) finished all six packages, but the agent that would have
|
||||
// read its stdout and posted the report was gone; the hub never learned what the pass installed.
|
||||
//
|
||||
// THE MECHANISM: the wrapper writes its apply report to <plan dir>/report-<run>-<layer>-apply.json BEFORE printing
|
||||
// it (configs/felhom-os-apply save_report). The agent deletes that copy once the hub has the report (finish). A copy
|
||||
// still on disk is a report nobody sent: SendUnsent posts it — at the agent's start, and before every pass — and then
|
||||
// deletes it. A pass lock (flock on <plan dir>/pass.lock, released by the kernel when a process dies) keeps the
|
||||
// sender from picking up the copy of a pass that is still running, also across the daemon and a selftest process.
|
||||
// Pinned by TestR868_* (unsent_test.go).
|
||||
|
||||
// lockPass takes the pass lock. block=false returns ok=false at once when another pass holds it. A lock that cannot
|
||||
// be opened at all (no plan dir yet) does not stop a pass: the unlock is then a no-op.
|
||||
func (l *Leg) lockPass(block bool) (unlock func()) {
|
||||
u, _ := l.tryLockPass(block)
|
||||
return u
|
||||
}
|
||||
|
||||
func (l *Leg) tryLockPass(block bool) (unlock func(), ok bool) {
|
||||
dir := l.planDir()
|
||||
_ = os.MkdirAll(dir, 0o700)
|
||||
f, err := os.OpenFile(filepath.Join(dir, "pass.lock"), os.O_CREATE|os.O_RDWR, 0o600)
|
||||
if err != nil {
|
||||
l.log().Warn("osupdate: pass lock unavailable — continuing without it", "err", err)
|
||||
return func() {}, true
|
||||
}
|
||||
how := syscall.LOCK_EX
|
||||
if !block {
|
||||
how |= syscall.LOCK_NB
|
||||
}
|
||||
if err := syscall.Flock(int(f.Fd()), how); err != nil {
|
||||
f.Close()
|
||||
return func() {}, false
|
||||
}
|
||||
return func() { _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN); f.Close() }, true
|
||||
}
|
||||
|
||||
// SendUnsent posts every report a pass kept on disk and nobody sent (R-868). It skips when a pass runs now (that
|
||||
// pass sends them first). Called at the agent's start.
|
||||
func (l *Leg) SendUnsent(ctx context.Context) int {
|
||||
unlock, ok := l.tryLockPass(false)
|
||||
if !ok {
|
||||
l.log().Info("osupdate: a pass is running — its start sends any kept report")
|
||||
return 0
|
||||
}
|
||||
defer unlock()
|
||||
return l.sendUnsentLocked(ctx)
|
||||
}
|
||||
|
||||
func (l *Leg) sendUnsentLocked(ctx context.Context) int {
|
||||
if l.Hub == nil {
|
||||
return 0 // nobody to send to: keep the copies for a process that has the hub
|
||||
}
|
||||
files, _ := filepath.Glob(filepath.Join(l.planDir(), "report-*.json"))
|
||||
sent := 0
|
||||
for _, f := range files {
|
||||
b, err := os.ReadFile(f)
|
||||
if err != nil {
|
||||
l.log().Warn("osupdate: a kept report cannot be read", "path", f, "err", err)
|
||||
continue
|
||||
}
|
||||
var wr WrapperReport
|
||||
if err := json.Unmarshal(b, &wr); err != nil || wr.Layer == "" {
|
||||
l.log().Warn("osupdate: a kept report is not a report — moved aside", "path", f, "err", err)
|
||||
_ = os.Rename(f, f+".bad")
|
||||
continue
|
||||
}
|
||||
rep := l.reportFromKept(ctx, wr, f)
|
||||
lg := l.log().With("run", rep.RunID, "layer", rep.Layer, "vmid", rep.VMID, "ring", rep.Ring, "trigger", rep.Trigger)
|
||||
lg.Info("osupdate: sending a report the agent never sent (the agent stopped mid-pass, R-868)", "path", f)
|
||||
before := rep.unsent
|
||||
_ = l.finish(ctx, lg, rep)
|
||||
if _, err := os.Stat(before); os.IsNotExist(err) {
|
||||
sent++
|
||||
// the pass's plan file is left behind too when the agent was killed inside call()
|
||||
_ = os.Remove(filepath.Join(filepath.Dir(f), "plan-"+strings.TrimPrefix(filepath.Base(f), "report-")))
|
||||
}
|
||||
}
|
||||
return sent
|
||||
}
|
||||
|
||||
// reportFromKept builds the hub report from a kept wrapper report, as runLayer would have. Health: the copy's own
|
||||
// before/after reading; when that is not healthy after an install, one fresh reading (services restart after an
|
||||
// install, and the pass that would have waited for them is gone).
|
||||
func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string) Report {
|
||||
ring := 1
|
||||
if wr.Ring != nil {
|
||||
ring = *wr.Ring
|
||||
}
|
||||
runID, trigger := wr.RunID, wr.Trigger
|
||||
if runID == "" {
|
||||
runID = strings.TrimSuffix(strings.TrimPrefix(filepath.Base(path), "report-"), ".json")
|
||||
}
|
||||
if trigger == "" {
|
||||
trigger = "unknown"
|
||||
}
|
||||
rep := Report{RunID: runID, Layer: wr.Layer, Trigger: trigger, Mode: wr.Mode, Ring: ring, ReleaseID: wr.ReleaseID,
|
||||
VMID: wr.VMID, unsent: path}
|
||||
prefix := "sent after the agent stopped mid-pass (R-868)"
|
||||
switch {
|
||||
case wr.refused():
|
||||
rep.Outcome, rep.Refused, rep.HealthReason = "refused", wr.Refused, prefix
|
||||
return rep
|
||||
case wr.failed():
|
||||
rep.Outcome, rep.Refused = "failed", wr.Failed
|
||||
case len(wr.Upgraded) == 0:
|
||||
rep.Outcome = "nothing"
|
||||
default:
|
||||
rep.Outcome = "applied"
|
||||
}
|
||||
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
|
||||
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
|
||||
wantEngine := ""
|
||||
for _, u := range wr.Upgraded {
|
||||
if u.Name == "docker-ce" {
|
||||
wantEngine = EngineOf(u.Version)
|
||||
}
|
||||
}
|
||||
verdict := func(h *Health) (bool, string) {
|
||||
switch wr.Layer {
|
||||
case LayerDocker:
|
||||
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
|
||||
case LayerHost:
|
||||
t := hub.TunnelUnknown
|
||||
if l.Tunnel != nil {
|
||||
t, _ = l.Tunnel.Status(ctx)
|
||||
}
|
||||
return HostHealthVerdict(wr.HealthBefore, h, t)
|
||||
}
|
||||
return HealthVerdict(wr.HealthBefore, h)
|
||||
}
|
||||
ok, why := verdict(wr.HealthAfter)
|
||||
if !ok && len(wr.Upgraded) > 0 && wr.VMID > 0 {
|
||||
lane := "fast"
|
||||
if wr.Layer == LayerDocker {
|
||||
lane = "slow"
|
||||
}
|
||||
if hr, err := l.call(ctx, "kept-"+runID, map[string]any{"release_id": "kept", "layer": wr.Layer, "lane": lane,
|
||||
"vmid": wr.VMID, "mode": "health", "packages": []Package{}}); err == nil && hr.Health != nil {
|
||||
ok, why = verdict(hr.Health)
|
||||
}
|
||||
}
|
||||
rep.Healthy, rep.HealthReason = ok, prefix
|
||||
if why != "" {
|
||||
rep.HealthReason = prefix + ": " + why
|
||||
}
|
||||
if !ok && rep.Outcome == "applied" {
|
||||
rep.Outcome = "health_failed"
|
||||
}
|
||||
rep.Installed, rep.Pending = wr.Installed, wr.Pending
|
||||
rep.RestartNeeded, rep.DockerRestartNeeded, rep.RebootNeeded = wr.RestartNeeded, wr.DockerRestartNeeded, wr.RebootNeeded
|
||||
rep.RebootScanned = wr.RebootScanned
|
||||
if wr.Layer == LayerDocker {
|
||||
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
|
||||
} else {
|
||||
planned := map[string]bool{}
|
||||
for _, u := range wr.Upgraded {
|
||||
planned[u.Name] = true
|
||||
}
|
||||
rep.NotCovered = notCovered(wr.Pending, ring, planned)
|
||||
}
|
||||
return rep
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-868 (v0.144.0). THE NIGHT'S SHAPE (A5, demo-hp 2026-10-05 02:57 UTC): the wrapper finished six packages, the agent
|
||||
// was kill -9-ed before it read the report; the hub never got it. Here: the wrapper's kept copy and the plan file are
|
||||
// on disk, a NEW agent process starts — it must send exactly one `applied` report and delete both files.
|
||||
// COMPANION RED-PROOF: drop the SendUnsent body (return 0) → "the hub got no report".
|
||||
func TestR868_KilledPassIsReportedAtStart(t *testing.T) {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
ring := 0
|
||||
kept := WrapperReport{Mode: "apply", Layer: LayerGuest, RunID: "20261005T025700Z", Trigger: "debug", Ring: &ring, VMID: 9201,
|
||||
ReleaseID: "ring0-20261005T025700Z", HealthBefore: guestOK(), HealthAfter: guestOK(),
|
||||
Upgraded: []Package{{Name: "libc6", Version: "u4"}, {Name: "openssl", Version: "u3"}}}
|
||||
b, _ := json.Marshal(kept)
|
||||
rp := reportFile(l.PlanDir, kept.RunID, LayerGuest, "apply")
|
||||
pp := planFile(l.PlanDir, kept.RunID, LayerGuest, "apply")
|
||||
must(t, os.WriteFile(rp, b, 0o600))
|
||||
must(t, os.WriteFile(pp, []byte("{}"), 0o600))
|
||||
if n := l.SendUnsent(context.Background()); n != 1 {
|
||||
t.Fatalf("sent %d, want 1", n)
|
||||
}
|
||||
if len(h.reports) != 1 {
|
||||
t.Fatalf("the hub got no report (or several): %+v", h.reports)
|
||||
}
|
||||
r := h.reports[0]
|
||||
if r.Outcome != "applied" || !r.Healthy || r.Trigger != "debug" || r.RunID != kept.RunID || r.Ring != 0 || len(r.Upgraded) != 2 || r.VMID != 9201 {
|
||||
t.Fatalf("report = %+v", r)
|
||||
}
|
||||
for _, p := range []string{rp, pp} {
|
||||
if _, err := os.Stat(p); !os.IsNotExist(err) {
|
||||
t.Fatalf("%s still on disk after the hub got it", filepath.Base(p))
|
||||
}
|
||||
}
|
||||
// a second start sends nothing again — no duplicate report
|
||||
if n := l.SendUnsent(context.Background()); n != 0 || len(h.reports) != 1 {
|
||||
t.Fatalf("sent again: %d, reports %d", n, len(h.reports))
|
||||
}
|
||||
}
|
||||
|
||||
// A normal pass: the hub gets ONE report per layer and the kept copies are gone after it (nothing resent later).
|
||||
// COMPANION RED-PROOF: drop the os.Remove(rep.unsent) in finish → the next pass resends → "2 guest reports".
|
||||
func TestR868_NormalPassLeavesNoCopyAndNoDuplicate(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, keep: true, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "u4"}}}}}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
l.Run(context.Background(), 9201, "debug")
|
||||
left, _ := filepath.Glob(filepath.Join(l.PlanDir, "report-*.json"))
|
||||
if len(left) != 0 {
|
||||
t.Fatalf("kept copies left after the hub got them: %v", left)
|
||||
}
|
||||
l.Run(context.Background(), 9201, "debug") // the next pass sends kept copies first
|
||||
guest := 0
|
||||
for _, r := range h.reports {
|
||||
if r.Layer == LayerGuest && r.Outcome == "applied" {
|
||||
guest++
|
||||
}
|
||||
}
|
||||
if guest != 2 {
|
||||
t.Fatalf("%d guest reports for 2 passes (a duplicate or a loss)", guest)
|
||||
}
|
||||
}
|
||||
|
||||
type failingHub struct{ n int }
|
||||
|
||||
func (h *failingHub) PostOSReport(context.Context, []byte) error {
|
||||
h.n++
|
||||
return errors.New("hub away")
|
||||
}
|
||||
|
||||
// The hub away: the copy stays, and goes at the next chance.
|
||||
func TestR868_HubAwayKeepsTheCopy(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, keep: true, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "u4"}}}}}
|
||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
l.Appliance = false
|
||||
l.Hub = &failingHub{}
|
||||
l.Run(context.Background(), 9201, "night")
|
||||
left, _ := filepath.Glob(filepath.Join(l.PlanDir, "report-*.json"))
|
||||
if len(left) != 2 { // ring 0: the guest step and the Docker step each kept one
|
||||
t.Fatalf("the copies must stay while the hub is away: %v", left)
|
||||
}
|
||||
h := &fakeHub{}
|
||||
l.Hub = h
|
||||
if n := l.SendUnsent(context.Background()); n != 2 {
|
||||
t.Fatalf("sent %d: %+v", n, h.reports)
|
||||
}
|
||||
for _, r := range h.reports {
|
||||
if r.Trigger != "night" || r.Ring != 0 {
|
||||
t.Fatalf("the kept report lost its ids: %+v", r)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A pass in progress holds the lock: the sender at start must not take that pass's copy (it would be sent twice).
|
||||
func TestR868_RunningPassKeepsTheSenderOff(t *testing.T) {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, h := newLeg(t, w, nil)
|
||||
must(t, os.WriteFile(reportFile(l.PlanDir, "r1", LayerGuest, "apply"), []byte(`{"mode":"apply","layer":"guest"}`), 0o600))
|
||||
unlock := l.lockPass(true)
|
||||
if n := l.SendUnsent(context.Background()); n != 0 || len(h.reports) != 0 {
|
||||
t.Fatalf("sent while a pass ran: %d", n)
|
||||
}
|
||||
unlock()
|
||||
if n := l.SendUnsent(context.Background()); n != 1 {
|
||||
t.Fatalf("not sent after the pass: %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func must(t *testing.T, err error) {
|
||||
t.Helper()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -51,6 +51,10 @@ const (
|
||||
// A Docker engine step in the customer guest (agent v0.142.0, `11` §5.8) — ring 1, and every undo. Destructive-class
|
||||
// (signed, operational key) like agent_update; the root wrapper re-verifies the same signature itself.
|
||||
ClassOSDockerStep OpClass = "os_docker_step"
|
||||
|
||||
// The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units).
|
||||
// Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself.
|
||||
ClassAgentConfigUpdate OpClass = "agent_config_update"
|
||||
)
|
||||
|
||||
// Disposition is the classifier verdict.
|
||||
@@ -119,7 +123,7 @@ func Classify(class OpClass, prov Provenance) Disposition {
|
||||
return Destructive
|
||||
case ClassKeyRotation:
|
||||
return Destructive
|
||||
case ClassAgentUpdate, ClassOSDockerStep:
|
||||
case ClassAgentUpdate, ClassOSDockerStep, ClassAgentConfigUpdate:
|
||||
// Never benign — no agent-internal provenance can make replacing the agent binary
|
||||
// unsigned-safe (a compromised process must not be able to self-bless an update).
|
||||
return Destructive
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""build-config-bundle.py — build the agent's CONFIG BUNDLE (R-840, `11` §5.4.2).
|
||||
|
||||
Usage: python3 scripts/build-config-bundle.py <agent-version> <out.json> (prints the bundle's sha256)
|
||||
|
||||
The bundle is every root-owned file the installer's step 5 writes for the agent (sudoers, wrappers, units), as ONE
|
||||
JSON file published beside the binary (felhom-agent/<version>/felhom-config-bundle.json). A box takes it by a signed
|
||||
`agent_config_update` job; a new box takes the SAME file from the installer. The list of files is NOT kept here: it is
|
||||
`BUNDLE_FILES` in configs/felhom-os-apply, the root wrapper that installs it — one table, so the builder cannot put in a
|
||||
path the wrapper would refuse, nor leave out one it expects.
|
||||
|
||||
Reproducible by construction: no timestamps, sorted keys, the table's order. The same source at the same version gives
|
||||
the same sha256 every time (pinned by configs/test_felhom_config_bundle.py).
|
||||
"""
|
||||
import base64
|
||||
import hashlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
CONFIGS = REPO / "configs"
|
||||
|
||||
|
||||
def load_wrapper(configs=CONFIGS):
|
||||
loader = importlib.machinery.SourceFileLoader("osapply_for_bundle", str(configs / "felhom-os-apply"))
|
||||
spec = importlib.util.spec_from_loader("osapply_for_bundle", loader)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def build(version, configs=CONFIGS):
|
||||
if not re.match(r"^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$", version):
|
||||
raise SystemExit(f"build-config-bundle: version {version!r} is not semver")
|
||||
w = load_wrapper(configs)
|
||||
files = []
|
||||
for dest, src, mode, check, policy in w.BUNDLE_FILES:
|
||||
data = (configs / src).read_bytes()
|
||||
files.append({"path": dest, "source": f"configs/{src}", "mode": oct(mode), "check": check, "policy": policy,
|
||||
"sha256": hashlib.sha256(data).hexdigest(), "content_b64": base64.b64encode(data).decode()})
|
||||
body = {"format": w.BUNDLE_FORMAT, "agent_version": version, "files": files}
|
||||
return (json.dumps(body, indent=1, sort_keys=True) + "\n").encode()
|
||||
|
||||
|
||||
def main(argv):
|
||||
if len(argv) != 3:
|
||||
print(__doc__, file=sys.stderr)
|
||||
return 2
|
||||
data = build(argv[1])
|
||||
pathlib.Path(argv[2]).write_bytes(data)
|
||||
print(hashlib.sha256(data).hexdigest())
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv))
|
||||
@@ -100,6 +100,15 @@ built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
|
||||
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
|
||||
log "built ok: sha256 $BUILT_SHA"
|
||||
|
||||
# ── 3b. The config bundle (R-840) ───────────────────────────────────────────────────────────────
|
||||
# Every root-owned file the installer's step 5 writes (sudoers, wrappers, units), as ONE file beside the binary. A box
|
||||
# takes it by a signed agent_config_update; a new box from the installer. Reproducible: same source → same sha.
|
||||
BUNDLE="$(mktemp -t felhom-config-bundle-XXXXXX)"
|
||||
trap 'rm -f "$BIN" "$BUNDLE"' EXIT
|
||||
BUNDLE_SHA="$(python3 "$REPO_ROOT/scripts/build-config-bundle.py" "$VERSION" "$BUNDLE")" || die "config bundle build failed"
|
||||
[[ "$BUNDLE_SHA" =~ ^[0-9a-f]{64}$ ]] || die "config bundle build printed no sha256"
|
||||
log "config bundle built: sha256 $BUNDLE_SHA"
|
||||
|
||||
# ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ──────────────────────────────
|
||||
#
|
||||
# THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03).
|
||||
@@ -153,6 +162,12 @@ if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then
|
||||
die "publish failed"
|
||||
fi
|
||||
|
||||
# ── 5b. Publish the config bundle beside the binary (same package version, same credentials) ──
|
||||
BURL="$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-config-bundle.json"
|
||||
bcode="$(curl -sS -o /dev/null -w '%{http_code}' -u "${GITEA_USER}:${GITEA_TOKEN}" -X PUT --upload-file "$BUNDLE" "$BURL")"
|
||||
[[ "$bcode" == "201" || "$bcode" == "200" ]] || die "config bundle upload failed: HTTP $bcode (the binary IS published; re-run only the bundle upload)"
|
||||
log "config bundle published (HTTP $bcode)"
|
||||
|
||||
# ── 6. Push the tag, now that the package exists ────────────────────────────────────────────────
|
||||
# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v<version>/` fetch
|
||||
# the installer makes. It runs last of the two so CI can never see a tag whose package is not there.
|
||||
@@ -193,6 +208,11 @@ DL_SHA="$(sha256sum "$DL" | awk '{print $1}')"
|
||||
[[ "$DL_SHA" == "$BUILT_SHA" ]] \
|
||||
|| die "published sha $DL_SHA != built sha $BUILT_SHA — the artifact is not what was built"
|
||||
|
||||
BDL="$(mktemp -t felhom-config-bundle-dl-XXXXXX)"
|
||||
trap 'rm -f "$BIN" "$DL" "$BUNDLE" "$BDL"' EXIT
|
||||
curl -fsS -o "$BDL" "$BURL" || die "round-trip GET of the config bundle failed"
|
||||
[[ "$(sha256sum "$BDL" | awk '{print $1}')" == "$BUNDLE_SHA" ]] || die "published config bundle sha != built sha"
|
||||
|
||||
# The tag must also serve the configs the installer will fetch from it.
|
||||
cfg_code="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/$GITEA_OWNER/felhom-agent/raw/tag/$TAG/configs/felhom-agent.service" 2>/dev/null || true)"
|
||||
@@ -206,6 +226,7 @@ cat <<EOF
|
||||
version : $VERSION
|
||||
tag : $TAG
|
||||
sha256 : $BUILT_SHA
|
||||
bundle : $BUNDLE_SHA (felhom-config-bundle.json — vouch it with the agent)
|
||||
|
||||
NOT VOUCHED. Vouching is what points machines at this version and stays your deliberate act:
|
||||
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
|
||||
|
||||
Reference in New Issue
Block a user