Compare commits

..
13 Commits
Author SHA1 Message Date
admin 91bd96c755 REPORT + CONTEXT: R-528 and R-1254 on main, not released (night 2026-10-28)
gates / gates (push) Successful in 33s
2026-10-29 02:51:00 +01:00
admin 6f933d6ee3 R-528: GET /guest/oom-kills — the kernel's per-container OOM counter of the token's guest, read on the host (no sudo)
gates / gates (push) Successful in 34s
Tests TestR528_* (fake cgroup tree), red-proved by mutation. Commit only; the agent release needs the vouch.
2026-10-28 23:22:40 +01:00
admin 5ddd124cfc R-1254 (ceremony half, HUNT-3 H2): after a failed escrow upload, ask the hub which package it holds before wording the result
gates / gates (push) Successful in 44s
The hub holds the new package -> success; another package -> 502 'previous code still works' (now true);
cannot be asked -> 504, no promise. Tests TestR1254_* red before the fix. Commit only; the agent release needs the vouch.
2026-10-28 22:51:08 +01:00
admin 3055bc54c1 D50 pin: day0_agent 0.167.0 (the operator's vouch 2026-10-27, read from the hub)
gates / gates (push) Successful in 1m4s
2026-10-27 20:45:25 +01:00
admin 007199342f v0.167.0 RELEASED (not vouched yet): CHANGELOG header, REPORT
gates / gates (push) Successful in 1m20s
2026-10-27 15:49:30 +01:00
admin 54d3a1057b CHANGELOG: the unreleased header names R-972 (Go 1.27.3)
gates / gates (push) Successful in 59s
2026-10-26 22:21:47 +01:00
admin fbbb168d1f build(R-972): Go 1.27.3 (GO-2026-6809, crypto/x509) — govulncheck 1 → 0 called
gates / gates (push) Successful in 1m22s
2026-10-26 20:34:25 +01:00
admin 3911cc6f52 fix(R-1232): the tier due-check counts only this box's archives (own key) — a reinstalled box's first whole-guest copy is not a week late
gates / gates (push) Successful in 1m20s
Red-proved (the old box's archive was counted). Commit only.
2026-10-26 20:15:39 +01:00
admin ad80ee749a docs: CONTEXT — R-1285 state and Tuesday's line test
gates / gates (push) Successful in 1m15s
2026-10-26 18:11:24 +01:00
admin 0683cc2dab REPORT: the 2026-10-26 afternoon run. Docs only.
gates / gates (push) Successful in 1m48s
2026-10-26 14:38:15 +01:00
admin b1bc63dd32 R-1285: the new agent runs the bundle policy pass once after its self-update commits — in the normal two-job delivery the old process installs the bundle. Test in the real order (commit, then one bundle-policy plan); red-proved. Commit only, not released.
gates / gates (push) Successful in 2m15s
2026-10-26 12:44:25 +01:00
admin e66a48f2b7 REPORT: the 2026-10-26 attended release run. Docs only.
gates / gates (push) Successful in 1m41s
2026-10-26 10:38:03 +01:00
admin b890b57540 v0.166.0 released and vouched by the operator (10:04:55 CET): CHANGELOG header (R-1280 live, R-1285 back to OPEN — the bundle job runs in the old process); D50 day0_agent pin → 0.166.0. Docs only.
gates / gates (push) Successful in 1m52s
2026-10-26 10:07:15 +01:00
18 changed files with 782 additions and 28 deletions

No files matched your search

+72 -3
View File
@@ -1,4 +1,73 @@
## 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released ## Unreleased (2026-10-28 night) — R-528: GET /guest/oom-kills, the kernel's OOM counter of each of the guest's containers, read on the host — commit only; the next agent release (needs the operator's vouch)
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
could not fire. The kernel counts every kill per container cgroup, and the host can read it.
- **`internal/localapi/guest_oom.go`:** `GET /guest/oom-kills` (withGuest — the token's guest only; a cross-guest
`?vmid=` is 403) reads `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex>.scope/memory.events` (and
`ns/docker/<id>/` for the cgroupfs driver), line `oom_kill N`, and answers `{vmid, available, containers: {id: n}}`.
A guest with no cgroup folder → `available: false`. Read-only, no sudo: the files are 0444 and the non-root
felhom-agent user reads them (measured on Tester 1's host 2026-10-28). A container's scope is new at each start, so the
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
## Unreleased (2026-10-28 night) — R-1254 (the ceremony half, review round 2 HUNT-3 H2): the agent asks the hub which package it holds before it words a failed upload — commit only; the next agent release (needs the operator's vouch)
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
hub's current package opens only with the NEW code.
- **Now (`internal/localapi/escrow_ceremony.go`):** after a failed upload the agent asks the hub (`HubHolds`, 15 s,
outside the lock, the slot still marked claiming): the hub holds THIS package → a success (the code is handed out /
the confirm completes; a WARN line names the lost answer); the hub holds another package → 502 and the old sentence,
now true; the hub cannot be asked → **504** with a sentence that promises nothing about either code, and the wait goes
on (a retry decides). `cmd/felhom-agent`: `HubHolds` compares the pending package's identity blob with the one the hub
serves back (`GET /api/v1/hosts/<id>/escrow`, the read-back the agent already uses; ciphertext only, nothing logged);
a package without an identity blob cannot be compared → unknown. Tests `TestR1254_*` (localapi, red before the fix:
502 while the hub held the new package) and `TestR1254_HubHoldsComparesTheIdentityBlob` (red-proved by mutation).
- Needs controller ≥ the R-1254 controller commit (2026-10-28 night) to word 504 for the household; an older controller
shows its generic „A kód lekérése nem sikerült." (claim) / its 502 sentence (confirm) for it.
## v0.167.0 (2026-10-27) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — RELEASED (release-agent.sh from a clean clone; agent sha256 28bdd282…d826f0, bundle 9d034416…ed1268; signed updates committed on demo-felhom, Tester 1, demo-hp; VOUCHED by the operator 2026-10-27, Day-0 pin set)
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
newest. Full `go build/vet/test` green. The controller and hub move the same night.
- **R-1232 (P2):** `backup.BackupRunner.NewestArchiveTime` — the tier due-check — counts only archives written with THIS
storage's key on an encrypted storage (the R-727 discriminator; the archive carries no host id). Before: a reinstalled
box shares the customer's PBS namespace and guest number with the old box, the old box's archive read as „the last
backup", and the whole-guest tier waited up to its 168-h cadence (the 2026-10-23 test night). A key lookup failure is
an error (the due-check reads UNKNOWN), never a foreign age. Tests `TestR1232_*` (foreign only → none; own → its time;
an unencrypted dir → unfiltered), red-proved (case 1 returned the old box's time).
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
second later — so 0.166.0's pass never ran on the bundle that delivered it.
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
agent-only — holding the bundle job until the commit would need new hub queue logic.
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
removed → commit only, no pass — the demo-hp symptom).
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
0.165.0 box passes even without this fix.
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
„BUNDLE POLICY DONE install_mode=appliance removed=0".
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by - **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
@@ -13,7 +82,7 @@
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu `ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
`documentation/audits/night-2026-10-25/redproof-r1285.txt`. `documentation/audits/night-2026-10-25/redproof-r1285.txt`.
## 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's) ### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused - `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays (`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
@@ -25,7 +94,7 @@
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals `test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`. seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
## 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change) ### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and - **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned` the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
+11
View File
@@ -1,5 +1,16 @@
# CONTEXT — felhom-agent working state # CONTEXT — felhom-agent working state
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
> on demo-hp (felhom.eu R-1250, read, not forced).
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd > **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written, > drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is > our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
+23 -19
View File
@@ -1,26 +1,30 @@
# REPORT — D50 retention gate, R-1280, R-1285: committed, not released (2026-10-25 night) # REPORT — two agent fixes on main, NOT released (night 2026-10-28 → 29, unattended)
**Plain words first.** Three agent changes are on `main`. None is released. The release (v0.166.0) is Monday's and **Plain words first.** Two changes are on main and tested. Neither is released: an agent release needs the operator's
waits for the operator's vouch (felhom.eu `STATUS.md`, the Monday list). vouch, and the night brief allowed no release. Both go on the morning list.
## Each change, before → after ## What changed
| Row | Commit | Before | After | Proof | - **R-528 — the OOM counter (`6f933d6`, CI run id 2473 / #284 green).** New local-API route `GET /guest/oom-kills`. It
|---|---|---|---|---| reads the kernel's `oom_kill` count of each Docker container of the token's own guest, on the host, from
| R-1290 (D50 = A) | `feaaa0e` | the published-versions gate knew no keep rule | it asserts the newest 5 released versions plus the pinned MinAgent floor (0.158.1) and the day-0 agent (0.165.0) in `scripts/retention-policy.json`; a pin with no tag fails | decoys seen failing; CI 2339 ✓ | `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events` (mode 0444, readable by the non-root
| R-1280 | `d912307` | a signed job's window could be any length | at most 24 h — the verifier (`ErrWindowTooLong`), the root wrapper (`MAX_WINDOW_S`) and `felhom-opsign -ttl`; exactly 24 h is accepted | red-proof; unit tests; not live | agent user). Docker inside an LXC guest does not see these kills (`OOMKilled` stays false). The controller reads the
| R-1285 | `abcc1e6` | a BYO host upgraded from an old wrapper kept our SSH drop-in until its NEXT bundle (the old wrapper planned the new bundle) | after every bundle install the agent runs the new wrapper's `bundle-policy` pass; the file is removed in one update; a failing `sshd -t` puts it back | red-proof; live on a BYO test VM: one signed run gave `BUNDLE POLICY REMOVED`; the test package was deleted after (204, then 404) | route every 30 s (controller `eed23e7`). Tests `TestR528_*`, red first by mutation (another guest's container served;
`oom` parsed for `oom_kill`).
- **R-1254 — the escrow ceremony half (`5ddd124`, CI run id 2471 / #283 green).** After an escrow upload that fails, the
agent asks the hub which package it holds (the identity blob compared): the new one → success; another one → 502 and
„the previous code still works"; no answer → 504, which promises nothing. Tests `TestR1254_*`, red first.
## CI (by commit hash, `jobs` endpoint) ## Live proof
`feaaa0e` 2339 ✓ · `d912307` 2344 ✓ · `abcc1e6` 2345 ✓ — all `run_attempt` 1. - R-528, 2026-10-28 23:27 UTC on Tester 1 with a test build of this agent (put back after: 0.167.0, sha256 `28bdd282…`).
A Radicale capped at 128 MB, a memory hog inside it killed: the host counter 0 → 1, one `app_oom` „seen by
docker,cgroup", one mail to the operator with `"source":"docker,cgroup"`. Evidence: felhom.eu
`documentation/audits/night-2026-10-28/r528-oom/`.
- R-1254: no live proof. It needs a hub answer lost on purpose; that is a hub change, and no hub deploy runs at night.
## Teardown ## Next
The BYO test VM (demo-hp 9372) was uninstalled through the product and destroyed. Its hub host record needs the The agent release (the operator's vouch), then `pinned.day0_agent` in `scripts/retention-policy.json` (D50).
operator's delete (felhom.eu `STATUS.md`). The controller needs no MinAgent change for R-528: an old agent answers 404, and the controller then waits 10 minutes
before it asks again.
## Evidence
felhom.eu `documentation/audits/night-2026-10-25/` (`redproof-d50.txt`, `redproof-r1280.txt`, `redproof-r1285.txt`,
`rig/r1285-live.txt`).
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
// passes as held → FAILS.
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
body := func(id string) []byte {
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
return b
}
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
if err != nil {
return nil, err
}
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
}
}
ctx := context.Background()
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
}
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
}
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
t.Fatal("a package without an identity blob cannot be compared — an error")
}
}
+43
View File
@@ -1216,6 +1216,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} }
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ",")) logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
}} }}
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound // v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
// to the local API server built below). // to the local API server built below).
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner, internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
@@ -1247,6 +1250,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root // R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
// subprocess used, under the same per-host key, before the code is handed out. // subprocess used, under the same per-host key, before the code is handed out.
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) }, Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
// R-1254 H2: after a failed upload, the hub is asked which package it holds before the household is told
// anything — the same per-host GET the identity read-back uses.
HubHolds: func(ctx context.Context, pkg []byte) (bool, error) {
if client == nil {
return false, fmt.Errorf("hub client not configured")
}
return hubHoldsEscrowPackage(ctx, client.FetchIdentityEscrow, pkg)
},
DRConfigured: func() bool { DRConfigured: func() bool {
if drConfigured != nil { if drConfigured != nil {
return drConfigured() return drConfigured()
@@ -3363,6 +3374,32 @@ func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the // putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978). // text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
// hubHoldsEscrowPackage (R-1254 H2) reports whether the hub's CURRENT package is the upload body pkg, by its identity
// blob — the part the hub serves back (GET /api/v1/hosts/<id>/escrow). Ciphertext only; nothing is logged. A package
// without an identity blob cannot be compared: an error (the caller then promises nothing).
func hubHoldsEscrowPackage(ctx context.Context, fetch func(context.Context) (*hub.IdentityEscrowResponse, error), pkg []byte) (bool, error) {
var up escrowUploadRequest
if err := json.Unmarshal(pkg, &up); err != nil {
return false, fmt.Errorf("reading the pending package: %w", err)
}
mine, err := base64.StdEncoding.DecodeString(up.IdentityBlobB64)
if err != nil || len(mine) == 0 {
return false, fmt.Errorf("the pending package carries no identity blob to compare")
}
resp, err := fetch(ctx)
if err != nil {
return false, err
}
if !resp.Present || resp.IdentityEscrowB64 == "" {
return false, nil
}
theirs, err := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
if err != nil {
return false, fmt.Errorf("decoding the hub's package: %w", err)
}
return bytes.Equal(mine, theirs), nil
}
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error { func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" { if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)") return fmt.Errorf("hub not configured (url/host_id/api_key)")
@@ -4040,3 +4077,9 @@ func agentStateDir(cfg config.Config) string {
} }
return "/var/lib/felhom-agent" return "/var/lib/felhom-agent"
} }
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
}
@@ -0,0 +1,91 @@
package main
import (
"context"
"encoding/json"
"io"
"log/slog"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
)
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
type orderRunner struct {
mu sync.Mutex
order []string
}
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
var p map[string]any
b, _ := os.ReadFile(args[1])
_ = json.Unmarshal(b, &p)
mode, _ := p["mode"].(string)
r.order = append(r.order, "os-apply:"+mode)
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
}
if len(args) >= 1 {
r.order = append(r.order, filepath.Base(name)+":"+args[0])
}
return []byte("ok"), nil, nil
}
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
t.Helper()
stateDir := t.TempDir()
if marker != nil {
dir := filepath.Join(stateDir, "selfupdate")
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
b, _ := json.Marshal(marker)
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
t.Fatal(err)
}
}
r := &orderRunner{}
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
mgr.MaybeCommit(context.Background())
r.mu.Lock()
defer r.mu.Unlock()
return append([]string(nil), r.order...)
}
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
}
}
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
}
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 0 {
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
}
}
+1 -1
View File
@@ -1,6 +1,6 @@
module gitea.dooplex.hu/admin/felhom-agent module gitea.dooplex.hu/admin/felhom-agent
go 1.27.2 go 1.27.3
require ( require (
golang.org/x/crypto v0.58.0 golang.org/x/crypto v0.58.0
@@ -0,0 +1,55 @@
package backup
import (
"context"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1232 — the 2026-10-23 test night: a reinstalled box (new install, new PBS key, same customer namespace, same guest
// number 9201) found the OLD box's archive (2.6 days old) when the due-check asked „when did this tier last land?", so
// the whole-guest tier read „not due" for up to its 168-h cadence and the ep0 copy stayed the OLD box's. The due-check
// now counts only archives written with THIS storage's key (the R-727 discriminator; the archive carries no host id).
// The CONSEQUENCE asserted: with only a foreign archive the answer is „none found" (the tier is due); with the box's
// own archive, that archive's time. COMPANION RED-PROOF: drop the key filter in NewestArchiveTime → case 1 returns the
// old box's time.
func TestR1232_TheDueCheckCountsOnlyThisBoxsArchives(t *testing.T) {
oldBox := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-20T21:10:00Z", Content: "backup", VMID: 9201,
Size: 3490689830, CTime: time.Date(2026, 10, 20, 21, 10, 0, 0, time.UTC).Unix(), Encrypted: earlierBox1}
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbsdr", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{oldBox},
}
r := NewBackupRunner(api, "felhom-pbsdr", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
// 1. The fresh box has written nothing yet: the old box's archive must not count.
when, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatal(err)
}
if found {
t.Fatalf("the due-check counted ANOTHER box's archive (%s) as this tier's last backup — the tier would wait a week", when)
}
// 2. After the box's own first copy, that copy is the answer.
own := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-24T02:30:00Z", Content: "backup", VMID: 9201,
Size: 3600000000, CTime: time.Date(2026, 10, 24, 2, 30, 0, 0, time.UTC).Unix(), Encrypted: thisBoxKey}
api.content = append(api.content, own)
when, found, err = r.NewestArchiveTime(context.Background(), 9201)
if err != nil || !found || when.Unix() != own.CTime {
t.Fatalf("want this box's own archive time %d, got %v found=%v err=%v", own.CTime, when, found, err)
}
}
// An unencrypted storage (a local dir) holds only this box's vzdumps: no key filter, the newest counts.
func TestR1232_UnencryptedStorageCountsEveryArchive(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_10_25-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1793050025}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, found, err := r.NewestArchiveTime(context.Background(), 9201); err != nil || !found {
t.Fatalf("an unencrypted local archive was not counted: found=%v err=%v", found, err)
}
}
+12
View File
@@ -609,11 +609,23 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
if err != nil { if err != nil {
return time.Time{}, false, err return time.Time{}, false, err
} }
// R-1232: on an ENCRYPTED storage only archives written with THIS storage's key are this box's (the R-727
// discriminator — the archive carries no host id). A reinstalled box shares the customer's namespace and guest
// number with the old box, and the old box's archive made the tier read „not due" for a whole cadence (168 h).
// A key lookup failure is returned: the due-check reads UNKNOWN, never a foreign archive's age.
ownKey, err := r.storageKeyFingerprint(ctx, r.target)
if err != nil {
return time.Time{}, false, fmt.Errorf("reading the key fingerprint of storage %s: %w", r.target, err)
}
var best int64 = -1 var best int64 = -1
for _, e := range contents { for _, e := range contents {
if e.Content != "backup" || e.VMID != vmid { if e.Content != "backup" || e.VMID != vmid {
continue continue
} }
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this tier's last backup (R-1232)", shortFP(e.Encrypted), shortFP(ownKey)))
continue
}
if ok, why := archivePlausiblyComplete(e); !ok { if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why) r.warnRejectedArchiveOnce(e, why)
continue continue
+76 -2
View File
@@ -72,12 +72,56 @@ type EscrowCeremonyConfig struct {
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of // host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
// a pending package is refused (503): a code must never be handed out for a package the hub lacks. // a pending package is refused (503): a code must never be handed out for a package the hub lacks.
Upload func(ctx context.Context, pkg []byte) error Upload func(ctx context.Context, pkg []byte) error
// HubHolds (R-1254 H2) asks the hub whether its CURRENT package is exactly pkg — asked after a failed upload,
// whose answer may have been lost AFTER the hub stored it. nil or an error → the outcome is unknown.
HubHolds func(ctx context.Context, pkg []byte) (bool, error)
} }
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole // escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
// ceremony incl. its upload was measured at ~2.4 s). // ceremony incl. its upload was measured at ~2.4 s).
const escrowUploadTimeout = 30 * time.Second const escrowUploadTimeout = 30 * time.Second
// escrowHoldTimeout bounds the hub question after a failed upload (R-1254 H2).
const escrowHoldTimeout = 15 * time.Second
// uploadOutcome is what a claim or confirm upload did, as the HUB tells it (R-1254 H2): an upload whose answer was lost
// or late may have been stored all the same.
type uploadOutcome int
const (
uploadStored uploadOutcome = iota // the hub holds the new package
uploadNotStored // the hub answered and holds another package: the previous code still works
uploadUnknown // the hub cannot be asked: nothing may be promised about either code
)
// escrowUnknownOutcomeMsg is the answer when the hub cannot say which package it holds — never „the previous code
// still works", which is false when the lost answer came after the store.
const escrowUnknownOutcomeMsg = "the hub did not answer in time — it is not known whether it stored the new recovery package; try again (until a try succeeds, do not rely on the previous code alone)"
// outcomeAfterFailedUpload asks the hub whether it holds pkg after Upload returned an error. Called with the slot
// marked claiming (escrowClaiming), outside escrowMu — it is a network call.
func (s *Server) outcomeAfterFailedUpload(pkg []byte) uploadOutcome {
if s.escrowCeremony.HubHolds == nil {
return uploadUnknown
}
base := s.baseCtx
if base == nil {
base = context.Background()
}
ctx, cancel := context.WithTimeout(base, escrowHoldTimeout)
defer cancel()
held, err := s.escrowCeremony.HubHolds(ctx, pkg)
switch {
case err != nil:
s.logger.Warn("local-api: escrow — the hub cannot say which package it holds after a failed upload (R-1254)", "err", err)
return uploadUnknown
case held:
return uploadStored
default:
return uploadNotStored
}
}
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until // ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned // parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
// spike-shaped output; production is runCeremonySubprocess. // spike-shaped output; production is runCeremonySubprocess.
@@ -524,15 +568,30 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout) uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg) err := s.escrowCeremony.Upload(uctx, pkg)
cancel() cancel()
outcome := uploadStored
if err != nil {
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2: the answer may be lost AFTER the store
}
s.escrowMu.Lock() s.escrowMu.Lock()
s.escrowClaiming = false s.escrowClaiming = false
if err != nil { switch outcome {
case uploadNotStored:
s.escrowMu.Unlock() s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)", s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds()) "vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again") writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return return
case uploadUnknown:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — upload failed and the hub cannot say which package it holds; code kept claimable (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
return
}
if err != nil {
s.logger.Warn("local-api: escrow claim — the upload's answer was lost, but the hub holds the new package; handing out the code (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err)
} }
s.escrowPending = nil s.escrowPending = nil
if s.escrowJob != nil && s.escrowJob.JobID == jobID { if s.escrowJob != nil && s.escrowJob.JobID == jobID {
@@ -631,15 +690,30 @@ func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Requ
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout) uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg) err := s.escrowCeremony.Upload(uctx, pkg)
cancel() cancel()
outcome := uploadStored
if err != nil {
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2
}
s.escrowMu.Lock() s.escrowMu.Lock()
s.escrowClaiming = false s.escrowClaiming = false
if err != nil { switch outcome {
case uploadNotStored:
s.escrowMu.Unlock() s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)", s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds()) "vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again") writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return return
case uploadUnknown:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — upload failed and the hub cannot say which package it holds; still waiting (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
return
}
if err != nil {
s.logger.Warn("local-api: escrow confirm — the upload's answer was lost, but the hub holds the new package (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err)
} }
s.escrowPending = nil s.escrowPending = nil
s.wipeEscrowConfirmLocked() s.wipeEscrowConfirmLocked()
+19
View File
@@ -42,6 +42,10 @@ type fakeEscrowHub struct {
current []byte current []byte
uploads int uploads int
err error // non-nil → the upload fails, nothing stored err error // non-nil → the upload fails, nothing stored
// R-1254 H2: lost → the upload STORES the package and then fails (the answer is lost on the way back);
// holdsErr → the hub cannot be asked which package it holds.
lost bool
holdsErr error
gate chan struct{} // non-nil → the upload blocks until closed gate chan struct{} // non-nil → the upload blocks until closed
entered chan struct{} // non-nil → closed when an upload starts entered chan struct{} // non-nil → closed when an upload starts
} }
@@ -61,9 +65,23 @@ func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
} }
f.current = append([]byte(nil), pkg...) f.current = append([]byte(nil), pkg...)
f.uploads++ f.uploads++
if f.lost {
return context.DeadlineExceeded
}
return nil return nil
} }
// holds answers „is the hub's current package exactly pkg?" — the R-1254 question (production compares the identity
// blob the hub serves back).
func (f *fakeEscrowHub) holds(_ context.Context, pkg []byte) (bool, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.holdsErr != nil {
return false, f.holdsErr
}
return string(f.current) == string(pkg), nil
}
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question. // opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
func (f *fakeEscrowHub) opensWith(r string) bool { func (f *fakeEscrowHub) opensWith(r string) bool {
f.mu.Lock() f.mu.Lock()
@@ -100,6 +118,7 @@ func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub
HubConfigured: true, HubConfigured: true,
DRConfigured: func() bool { return true }, DRConfigured: func() bool { return true },
Upload: hub.upload, Upload: hub.upload,
HubHolds: hub.holds,
} }
srv.ceremonyRun = run srv.ceremonyRun = run
srv.escrowSudoCheck = func(context.Context) error { return nil } srv.escrowSudoCheck = func(context.Context) error { return nil }
+98
View File
@@ -0,0 +1,98 @@
package localapi
import (
"errors"
"net/http"
"strings"
"testing"
)
// R-1254 (the ceremony half, review round 2 HUNT-3 H2): the upload's answer can be lost or late (30 s) AFTER the hub
// stored the new package. The household was then told „the previous code still works" — false: the hub now holds the
// package the NEW code opens. The agent asks the hub which package it holds before it words the result:
// - the hub holds the new package → success (the code is handed out / the confirm completes);
// - the hub holds another package → the old sentence, now true;
// - the hub cannot be asked → 504 and a sentence that does NOT promise the previous code; the wait goes on.
// RED-PROOF: remove the hub question from the claim's failure branch → 502 „the previous code still works" while the
// hub holds the new package → FAILS.
func TestR1254_ALostAnswerAtTheClaimIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
hub := &fakeEscrowHub{lost: true}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if !hub.opensWith(firstR) {
t.Fatal("fixture: the hub must hold the new package (the answer was lost after the store)")
}
if code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("claim after a lost answer, the hub holding the new package: %d %s — want 200 with the code", code, body)
}
if strings.Contains(body, "previous code still works") {
t.Fatalf("the household was told the previous code still works: %s", body)
}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
t.Fatalf("the job must read uploaded: %s", st)
}
}
// The hub cannot be asked: no promise about the previous code, 504, the code stays claimable (a retry decides).
// RED-PROOF: as above → 502 with „the previous code still works" → FAILS.
func TestR1254_AnUnknownOutcomeAtTheClaimPromisesNothing(t *testing.T) {
hub := &fakeEscrowHub{lost: true, holdsErr: errors.New("hub unreachable")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusGatewayTimeout {
t.Fatalf("claim with an unknown outcome: %d %s — want 504", code, body)
}
if strings.Contains(body, "previous code still works") {
t.Fatalf("an unknown outcome promised the previous code: %s", body)
}
assertNoRIn(t, "unknown-outcome claim payload", body, firstR)
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) {
t.Fatalf("the code must stay claimable: %s", st)
}
// The hub answers again: the retry finds the new package already stored and hands the code out.
hub.mu.Lock()
hub.holdsErr = nil
hub.mu.Unlock()
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("retry: %d %s", code, body)
}
}
// The hub holds ANOTHER package (the upload really failed): the old sentence stays, and is now true.
func TestR1254_AFailedUploadTheHubDidNotStoreKeepsTheOldSentence(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
if code, body := claimCode(t, h); code != http.StatusBadGateway || !strings.Contains(body, "previous code still works") {
t.Fatalf("a failed upload: %d %s — want 502 and the previous code named", code, body)
}
}
// The same at the typed-back confirm (R-1151).
// RED-PROOF: remove the hub question from the confirm's failure branch → 502 → FAILS.
func TestR1254_ALostAnswerAtTheConfirmIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
hub := &fakeEscrowHub{lost: true}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
t.Fatalf("confirm after a lost answer: %d (opens: %v) — want 200", code, hub.opensWith(secondR))
}
hub.mu.Lock()
hub.lost, hub.holdsErr = true, errors.New("hub unreachable")
hub.mu.Unlock()
srv2, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h2 := srv2.Handler()
startAndWait(t, srv2, h2)
c2 := claimDeferred(t, h2)
if code := confirm(t, h2, c2.typedBack(t)); code != http.StatusGatewayTimeout {
t.Fatalf("confirm with an unknown outcome: %d — want 504", code)
}
}
+102
View File
@@ -0,0 +1,102 @@
package localapi
import (
"bufio"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
)
// R-528 — the OOM signal from the host side.
//
// MEASURED 2026-09-15 (scratch 9202, Docker 29.8.0) and 2026-09-16 (a fresh install in a nested VM): an app killed by
// its memory limit inside a Felhom LXC guest left Docker's State.OOMKilled false and sent no `oom` event, so the
// controller's OOM line (R-514) could not fire. The kernel's own counter does count every kill, and the host can read
// it: /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events, line `oom_kill N` (the
// systemd cgroup driver; `ns/docker/<id>/` under the cgroupfs driver). Read on Tester 1's host 2026-10-28: the files
// are mode 0444, readable by the non-root felhom-agent user — no sudo, no new root path.
//
// A container's scope is new at each start, so the counter is per container RUN: a count above 0 means this run had a
// kill. The controller asks every 30 s and raises its existing app_oom line (once per run) for any such container.
// readGuestOOMKills returns container id → oom_kill for one guest. available=false (no error) when the guest has no
// cgroup folder (a VM guest, or a container that is not running).
func readGuestOOMKills(root string, vmid int) (map[string]int64, bool, error) {
if root == "" {
root = "/sys/fs/cgroup"
}
base := filepath.Join(root, "lxc", strconv.Itoa(vmid), "ns")
if fi, err := os.Stat(base); err != nil || !fi.IsDir() {
return nil, false, nil
}
out := map[string]int64{}
for _, pattern := range []string{
filepath.Join(base, "system.slice", "docker-*.scope", "memory.events"),
filepath.Join(base, "docker", "*", "memory.events"),
} {
files, err := filepath.Glob(pattern)
if err != nil {
return nil, true, err
}
for _, f := range files {
id := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(filepath.Dir(f)), "docker-"), ".scope")
if !isContainerID(id) {
continue
}
n, err := readOOMKill(f)
if err != nil {
continue // a container that stopped between the glob and the read
}
out[id] = n
}
}
return out, true, nil
}
func isContainerID(s string) bool {
if len(s) != 64 {
return false
}
for _, c := range s {
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
return false
}
}
return true
}
func readOOMKill(path string) (int64, error) {
f, err := os.Open(path)
if err != nil {
return 0, err
}
defer f.Close()
sc := bufio.NewScanner(f)
for sc.Scan() {
k, v, ok := strings.Cut(strings.TrimSpace(sc.Text()), " ")
if ok && k == "oom_kill" {
return strconv.ParseInt(strings.TrimSpace(v), 10, 64)
}
}
if err := sc.Err(); err != nil {
return 0, err
}
return 0, fmt.Errorf("no oom_kill line in %s", path)
}
// handleGuestOOMKills is GET /guest/oom-kills: {"vmid", "available", "containers": {id: oom_kill}}.
func (s *Server) handleGuestOOMKills(w http.ResponseWriter, r *http.Request, vmid int) {
kills, avail, err := readGuestOOMKills(s.cgroupRoot, vmid)
if err != nil {
s.logger.Warn("local-api: guest oom-kills read failed (R-528)", "vmid", vmid, "err", err)
writeErr(w, http.StatusInternalServerError, "the guest's OOM counters could not be read")
return
}
if kills == nil {
kills = map[string]int64{}
}
writeOK(w, map[string]any{"vmid": vmid, "available": avail, "containers": kills})
}
+101
View File
@@ -0,0 +1,101 @@
package localapi
import (
"encoding/json"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
)
// R-528: Docker does not always report an OOM kill inside a Felhom LXC guest (OOMKilled stays false, no `oom` event —
// measured 2026-09-15/16 on two boxes), but the kernel's own counter does count it, and it is readable from the host:
// /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events, line `oom_kill N`. GET /guest/oom-kills
// serves the asking guest's counters, and only that guest's.
func fakeCgroup(t *testing.T, root string, vmid, id, events string) {
t.Helper()
dir := filepath.Join(root, "lxc", vmid, "ns", "system.slice", "docker-"+id+".scope")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o444); err != nil {
t.Fatal(err)
}
}
const idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
const idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
const idOther = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
// RED-PROOF: serve the counters of every guest under lxc/ (drop the vmid from the glob) → the other guest's container
// appears → FAILS; parse `oom ` instead of `oom_kill` → idA reads 9 → FAILS.
func TestR528_GuestOOMKillsServesThisGuestsCountersOnly(t *testing.T) {
root := t.TempDir()
fakeCgroup(t, root, "100", idA, "low 0\nhigh 0\nmax 4\noom 9\noom_kill 3\noom_group_kill 0\n")
fakeCgroup(t, root, "100", idB, "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\n")
fakeCgroup(t, root, "200", idOther, "oom_kill 7\n")
// a scope that is not a container (no 64-hex id) and a cgroupfs-driver layout container
if err := os.MkdirAll(filepath.Join(root, "lxc", "100", "ns", "system.slice", "cron.service"), 0o755); err != nil {
t.Fatal(err)
}
dfDir := filepath.Join(root, "lxc", "100", "ns", "docker", idOther[:8]+strings.Repeat("d", 56))
if err := os.MkdirAll(dfDir, 0o755); err != nil {
t.Fatal(err)
}
_ = os.WriteFile(filepath.Join(dfDir, "memory.events"), []byte("oom_kill 2\n"), 0o444)
got, avail, err := readGuestOOMKills(root, 100)
if err != nil || !avail {
t.Fatalf("read: avail=%v err=%v", avail, err)
}
want := map[string]int64{idA: 3, idB: 0, idOther[:8] + strings.Repeat("d", 56): 2}
if len(got) != len(want) {
t.Fatalf("got %v, want %v", got, want)
}
for k, v := range want {
if got[k] != v {
t.Fatalf("got %v, want %v", got, want)
}
}
if _, leaked := got[idOther]; leaked {
t.Fatal("another guest's container leaked into this guest's answer")
}
// A guest with no cgroup folder (a VM, or not running): not available, no error.
if _, avail, err := readGuestOOMKills(root, 300); avail || err != nil {
t.Fatalf("a guest without a cgroup folder: avail=%v err=%v", avail, err)
}
}
// Through the route, with the guest's own token.
func TestR528_TheRouteAnswersTheTokensGuest(t *testing.T) {
root := t.TempDir()
fakeCgroup(t, root, "8200", idA, "oom_kill 5\n")
fakeCgroup(t, root, "9300", idB, "oom_kill 1\n") // token B's guest: never in A's answer
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.cgroupRoot = root
w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "A", "")
if w.Code != http.StatusOK {
t.Fatalf("route: %d %s", w.Code, w.Body.String())
}
var env struct {
Data struct {
VMID int `json:"vmid"`
Available bool `json:"available"`
Containers map[string]int64 `json:"containers"`
} `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
t.Fatal(err)
}
if env.Data.VMID != 8200 || !env.Data.Available || len(env.Data.Containers) != 1 || env.Data.Containers[idA] != 5 {
t.Fatalf("answer %s — want guest 8200's one container with 5", w.Body.String())
}
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
t.Fatalf("a cross-guest ask: %d, want 403", w.Code)
}
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "", ""); w.Code != http.StatusUnauthorized {
t.Fatalf("no token: %d", w.Code)
}
}
+5
View File
@@ -459,6 +459,8 @@ type Server struct {
escrowLookPath func(file string) (string, error) escrowLookPath func(file string) (string, error)
// statFile reports whether a path exists (preflight's staged-secret item; tests inject). // statFile reports whether a path exists (preflight's staged-secret item; tests inject).
statFile func(path string) bool statFile func(path string) bool
// cgroupRoot is where the host's cgroup tree is mounted (R-528, GET /guest/oom-kills); "" = /sys/fs/cgroup. Tests set it.
cgroupRoot string
baseCtx context.Context // for fire-and-forget backups; set in Run baseCtx context.Context // for fire-and-forget backups; set in Run
} }
@@ -576,6 +578,9 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard)) mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch // R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
// read it. Read-only; never measured (or the job not wired) = present:false. // read it. Read-only; never measured (or the job not wired) = present:false.
// R-528: the kernel's OOM-kill counter of each of the guest's containers, read on the host (Docker's own flag and
// event can stay silent inside an LXC guest). Read-only; the asking guest's containers only.
mux.HandleFunc("GET /guest/oom-kills", s.withGuest(s.handleGuestOOMKills))
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest)) mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate. // Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
+13
View File
@@ -126,6 +126,19 @@ func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params jso
// appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run // appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run
// itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal: // itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal:
// the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it. // the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it.
// PolicyPassAfterUpdate runs the wrapper's bundle-policy pass once after a self-update commits (R-1285). In the normal
// delivery the box gets agent_update and agent_config_update in ONE pass: the bundle job runs in the OLD process, which
// has no policy pass, and the restart into the new agent comes a second later (measured 2026-10-26 on demo-hp). By the
// commit the wrapper on disk is the bundle's own, so the pass removes our appliance-only files from a BYO host in the
// SAME update. Idempotent; on an appliance it removes nothing.
func (e ConfigUpdateExecutor) PolicyPassAfterUpdate(ctx context.Context, lg *slog.Logger, ver string) {
if lg == nil {
lg = slog.Default()
}
lg.Info("osupdate: running the bundle policy pass after the self-update commit (R-1285)", "agent_version", ver)
e.policyPass(ctx, lg, ver)
}
func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) { func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) {
runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z") runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z")
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"}) wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"})
+9
View File
@@ -18,8 +18,14 @@ type Manager struct {
logger *slog.Logger logger *slog.Logger
now func() time.Time // injectable for tests now func() time.Time // injectable for tests
sleep func(context.Context, time.Duration) sleep func(context.Context, time.Duration)
// onCommitted runs once, right after a successful commit (R-1285: the bundle policy pass of the NEW agent — in the
// normal two-job delivery the bundle was installed by the OLD process, which has no such pass). nil = none.
onCommitted func(context.Context)
} }
// SetOnCommitted wires the after-commit hook (R-1285). INIT-ONLY: call it before MaybeCommit's goroutine starts.
func (m *Manager) SetOnCommitted(fn func(context.Context)) { m.onCommitted = fn }
// ManagerConfig wires the commit manager. // ManagerConfig wires the commit manager.
type ManagerConfig struct { type ManagerConfig struct {
StateDir string StateDir string
@@ -119,4 +125,7 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
return return
} }
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout)) m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
if m.onCommitted != nil {
m.onCommitted(ctx)
}
} }
+3 -3
View File
@@ -24,10 +24,10 @@
"readers": [ "readers": [
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version" "scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
], ],
"recorded": "2026-10-25", "recorded": "2026-10-27",
"recorded_by": "CC 2026-10-25, D50 = A (operator); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)", "recorded_by": "CC 2026-10-27: day0_agent 0.167.0 (operator vouch 2026-10-27, read from the hub: version + agent and bundle sha256 match the release); earlier 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"pinned": { "pinned": {
"hub_min_agent": "0.158.1", "hub_min_agent": "0.158.1",
"day0_agent": "0.165.0" "day0_agent": "0.167.0"
} }
} }