Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| b1bc63dd32 | |||
| e66a48f2b7 | |||
| b890b57540 |
+33
-3
@@ -1,4 +1,34 @@
|
||||
## 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
|
||||
## Unreleased (next: v0.167.0) — the BYO SSH clean-up in the SAME update (R-1285) (2026-10-26) — committed, not released
|
||||
|
||||
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
|
||||
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
|
||||
second later — so 0.166.0's pass never ran on the bundle that delivered it.
|
||||
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
|
||||
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
|
||||
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
|
||||
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
|
||||
agent-only — holding the bundle job until the commit would need new hub queue logic.
|
||||
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
|
||||
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
|
||||
removed → commit only, no pass — the demo-hp symptom).
|
||||
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
|
||||
0.165.0 box passes even without this fix.
|
||||
|
||||
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
|
||||
|
||||
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
|
||||
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
|
||||
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
|
||||
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
|
||||
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
|
||||
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
|
||||
|
||||
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
|
||||
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
|
||||
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
|
||||
„BUNDLE POLICY DONE install_mode=appliance removed=0".
|
||||
|
||||
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
|
||||
|
||||
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
|
||||
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
|
||||
@@ -13,7 +43,7 @@
|
||||
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
|
||||
`documentation/audits/night-2026-10-25/redproof-r1285.txt`.
|
||||
|
||||
## 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
|
||||
### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
|
||||
|
||||
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
|
||||
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
|
||||
@@ -25,7 +55,7 @@
|
||||
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
|
||||
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
|
||||
|
||||
## 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
|
||||
### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
|
||||
|
||||
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
|
||||
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
|
||||
|
||||
@@ -1,26 +1,26 @@
|
||||
# REPORT — D50 retention gate, R-1280, R-1285: committed, not released (2026-10-25 night)
|
||||
# REPORT — agent v0.166.0 released and vouched (2026-10-26, attended release run)
|
||||
|
||||
**Plain words first.** Three agent changes are on `main`. None is released. The release (v0.166.0) is Monday's and
|
||||
waits for the operator's vouch (felhom.eu `STATUS.md`, the Monday list).
|
||||
**Plain words first.** Agent 0.166.0 runs on Tester 1, demo-hp and demo-felhom, and the operator vouched it at 10:04:55
|
||||
CET, so new installs get it (a fresh install today fetched it). A signed job's window is now at most 24 hours — seen
|
||||
live. One part is not done: on a BYO host the SSH clean-up still comes one release late (R-1285, back to OPEN).
|
||||
|
||||
## Each change, before → after
|
||||
## Release
|
||||
|
||||
| Row | Commit | Before | After | Proof |
|
||||
|---|---|---|---|---|
|
||||
| R-1290 (D50 = A) | `feaaa0e` | the published-versions gate knew no keep rule | it asserts the newest 5 released versions plus the pinned MinAgent floor (0.158.1) and the day-0 agent (0.165.0) in `scripts/retention-policy.json`; a pin with no tag fails | decoys seen failing; CI 2339 ✓ |
|
||||
| R-1280 | `d912307` | a signed job's window could be any length | at most 24 h — the verifier (`ErrWindowTooLong`), the root wrapper (`MAX_WINDOW_S`) and `felhom-opsign -ttl`; exactly 24 h is accepted | red-proof; unit tests; not live |
|
||||
| R-1285 | `abcc1e6` | a BYO host upgraded from an old wrapper kept our SSH drop-in until its NEXT bundle (the old wrapper planned the new bundle) | after every bundle install the agent runs the new wrapper's `bundle-policy` pass; the file is removed in one update; a failing `sshd -t` puts it back | red-proof; live on a BYO test VM: one signed run gave `BUNDLE POLICY REMOVED`; the test package was deleted after (204, then 404) |
|
||||
- From a clean clone at `aa69114` (CI 2351 ✓): tag `v0.166.0`; binary sha256
|
||||
`3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`; bundle
|
||||
`093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`; verified by independent download.
|
||||
- Delivered by signed `agent_update` + `agent_config_update`: each box COMPLETED, bundle INSTALLED, self-check 76/76,
|
||||
update committed.
|
||||
- D50 pin `day0_agent` → 0.166.0 (`b890b57`, CI 2356 ✓); the published gate green.
|
||||
|
||||
## CI (by commit hash, `jobs` endpoint)
|
||||
## Live checks
|
||||
|
||||
`feaaa0e` 2339 ✓ · `d912307` 2344 ✓ · `abcc1e6` 2345 ✓ — all `run_attempt` 1.
|
||||
- **R-1280 (closed):** a 25 h signed job on demo-hp → „REJECTED signed op — executor not called … op validity window too
|
||||
long (25h0m0s)"; a 30 min job ran. `felhom-opsign -ttl 25h` refuses (rc 2); 24 h accepted.
|
||||
- **R-1285 (back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD 0.165.0 process (demo-hp: bundle
|
||||
09:03:07–09:03:09 CET, restart into 0.166.0 at 09:03:10), so no policy pass ran on that bundle. Under 0.166.0 a later
|
||||
bundle logged „BUNDLE POLICY DONE install_mode=appliance removed=0" — the pass is wired; its delivery order is not.
|
||||
No BYO host is in the fleet. Fix options in the row.
|
||||
- **R-1289 (hub):** the vouch logged „behind the fleet's 0.344.0" — a release version.
|
||||
|
||||
## Teardown
|
||||
|
||||
The BYO test VM (demo-hp 9372) was uninstalled through the product and destroyed. Its hub host record needs the
|
||||
operator's delete (felhom.eu `STATUS.md`).
|
||||
|
||||
## Evidence
|
||||
|
||||
felhom.eu `documentation/audits/night-2026-10-25/` (`redproof-d50.txt`, `redproof-r1280.txt`, `redproof-r1285.txt`,
|
||||
`rig/r1285-live.txt`).
|
||||
Evidence: felhom.eu `documentation/audits/release-monday-2026-10-26/agent-0166/`.
|
||||
|
||||
@@ -1216,6 +1216,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||
}}
|
||||
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
|
||||
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
|
||||
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
|
||||
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
|
||||
// to the local API server built below).
|
||||
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
|
||||
@@ -4040,3 +4043,9 @@ func agentStateDir(cfg config.Config) string {
|
||||
}
|
||||
return "/var/lib/felhom-agent"
|
||||
}
|
||||
|
||||
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
|
||||
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
|
||||
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
|
||||
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
|
||||
}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
|
||||
)
|
||||
|
||||
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
|
||||
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
|
||||
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
|
||||
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
|
||||
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
|
||||
|
||||
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
|
||||
type orderRunner struct {
|
||||
mu sync.Mutex
|
||||
order []string
|
||||
}
|
||||
|
||||
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
|
||||
var p map[string]any
|
||||
b, _ := os.ReadFile(args[1])
|
||||
_ = json.Unmarshal(b, &p)
|
||||
mode, _ := p["mode"].(string)
|
||||
r.order = append(r.order, "os-apply:"+mode)
|
||||
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
|
||||
}
|
||||
if len(args) >= 1 {
|
||||
r.order = append(r.order, filepath.Base(name)+":"+args[0])
|
||||
}
|
||||
return []byte("ok"), nil, nil
|
||||
}
|
||||
|
||||
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
|
||||
t.Helper()
|
||||
stateDir := t.TempDir()
|
||||
if marker != nil {
|
||||
dir := filepath.Join(stateDir, "selfupdate")
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := json.Marshal(marker)
|
||||
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
r := &orderRunner{}
|
||||
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
|
||||
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
|
||||
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
|
||||
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
|
||||
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
|
||||
mgr.MaybeCommit(context.Background())
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return append([]string(nil), r.order...)
|
||||
}
|
||||
|
||||
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
|
||||
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
|
||||
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
|
||||
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
|
||||
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
|
||||
}
|
||||
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -126,6 +126,19 @@ func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params jso
|
||||
// appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run
|
||||
// itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal:
|
||||
// the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it.
|
||||
// PolicyPassAfterUpdate runs the wrapper's bundle-policy pass once after a self-update commits (R-1285). In the normal
|
||||
// delivery the box gets agent_update and agent_config_update in ONE pass: the bundle job runs in the OLD process, which
|
||||
// has no policy pass, and the restart into the new agent comes a second later (measured 2026-10-26 on demo-hp). By the
|
||||
// commit the wrapper on disk is the bundle's own, so the pass removes our appliance-only files from a BYO host in the
|
||||
// SAME update. Idempotent; on an appliance it removes nothing.
|
||||
func (e ConfigUpdateExecutor) PolicyPassAfterUpdate(ctx context.Context, lg *slog.Logger, ver string) {
|
||||
if lg == nil {
|
||||
lg = slog.Default()
|
||||
}
|
||||
lg.Info("osupdate: running the bundle policy pass after the self-update commit (R-1285)", "agent_version", ver)
|
||||
e.policyPass(ctx, lg, ver)
|
||||
}
|
||||
|
||||
func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) {
|
||||
runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z")
|
||||
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"})
|
||||
|
||||
@@ -18,8 +18,14 @@ type Manager struct {
|
||||
logger *slog.Logger
|
||||
now func() time.Time // injectable for tests
|
||||
sleep func(context.Context, time.Duration)
|
||||
// onCommitted runs once, right after a successful commit (R-1285: the bundle policy pass of the NEW agent — in the
|
||||
// normal two-job delivery the bundle was installed by the OLD process, which has no such pass). nil = none.
|
||||
onCommitted func(context.Context)
|
||||
}
|
||||
|
||||
// SetOnCommitted wires the after-commit hook (R-1285). INIT-ONLY: call it before MaybeCommit's goroutine starts.
|
||||
func (m *Manager) SetOnCommitted(fn func(context.Context)) { m.onCommitted = fn }
|
||||
|
||||
// ManagerConfig wires the commit manager.
|
||||
type ManagerConfig struct {
|
||||
StateDir string
|
||||
@@ -119,4 +125,7 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
|
||||
return
|
||||
}
|
||||
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
|
||||
if m.onCommitted != nil {
|
||||
m.onCommitted(ctx)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,10 +24,10 @@
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||
],
|
||||
"recorded": "2026-10-25",
|
||||
"recorded_by": "CC 2026-10-25, D50 = A (operator); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"recorded": "2026-10-26",
|
||||
"recorded_by": "CC 2026-10-26: day0_agent 0.166.0 (operator vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"pinned": {
|
||||
"hub_min_agent": "0.158.1",
|
||||
"day0_agent": "0.165.0"
|
||||
"day0_agent": "0.166.0"
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user