Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
91bd96c755 | ||
|
|
6f933d6ee3 | ||
|
|
5ddd124cfc | ||
|
|
3055bc54c1 | ||
|
|
007199342f |
No files matched your search
+30
-1
@@ -1,4 +1,33 @@
|
||||
## Unreleased (next: v0.167.0) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — committed, not released
|
||||
## Unreleased (2026-10-28 night) — R-528: GET /guest/oom-kills, the kernel's OOM counter of each of the guest's containers, read on the host — commit only; the next agent release (needs the operator's vouch)
|
||||
|
||||
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
|
||||
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
|
||||
could not fire. The kernel counts every kill per container cgroup, and the host can read it.
|
||||
- **`internal/localapi/guest_oom.go`:** `GET /guest/oom-kills` (withGuest — the token's guest only; a cross-guest
|
||||
`?vmid=` is 403) reads `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex>.scope/memory.events` (and
|
||||
`ns/docker/<id>/` for the cgroupfs driver), line `oom_kill N`, and answers `{vmid, available, containers: {id: n}}`.
|
||||
A guest with no cgroup folder → `available: false`. Read-only, no sudo: the files are 0444 and the non-root
|
||||
felhom-agent user reads them (measured on Tester 1's host 2026-10-28). A container's scope is new at each start, so the
|
||||
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
|
||||
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
|
||||
|
||||
## Unreleased (2026-10-28 night) — R-1254 (the ceremony half, review round 2 HUNT-3 H2): the agent asks the hub which package it holds before it words a failed upload — commit only; the next agent release (needs the operator's vouch)
|
||||
|
||||
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
|
||||
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
|
||||
hub's current package opens only with the NEW code.
|
||||
- **Now (`internal/localapi/escrow_ceremony.go`):** after a failed upload the agent asks the hub (`HubHolds`, 15 s,
|
||||
outside the lock, the slot still marked claiming): the hub holds THIS package → a success (the code is handed out /
|
||||
the confirm completes; a WARN line names the lost answer); the hub holds another package → 502 and the old sentence,
|
||||
now true; the hub cannot be asked → **504** with a sentence that promises nothing about either code, and the wait goes
|
||||
on (a retry decides). `cmd/felhom-agent`: `HubHolds` compares the pending package's identity blob with the one the hub
|
||||
serves back (`GET /api/v1/hosts/<id>/escrow`, the read-back the agent already uses; ciphertext only, nothing logged);
|
||||
a package without an identity blob cannot be compared → unknown. Tests `TestR1254_*` (localapi, red before the fix:
|
||||
502 while the hub held the new package) and `TestR1254_HubHoldsComparesTheIdentityBlob` (red-proved by mutation).
|
||||
- Needs controller ≥ the R-1254 controller commit (2026-10-28 night) to word 504 for the household; an older controller
|
||||
shows its generic „A kód lekérése nem sikerült." (claim) / its 502 sentence (confirm) for it.
|
||||
|
||||
## v0.167.0 (2026-10-27) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — RELEASED (release-agent.sh from a clean clone; agent sha256 28bdd282…d826f0, bundle 9d034416…ed1268; signed updates committed on demo-felhom, Tester 1, demo-hp; VOUCHED by the operator 2026-10-27, Day-0 pin set)
|
||||
|
||||
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
|
||||
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
|
||||
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
|
||||
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
|
||||
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
|
||||
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
|
||||
|
||||
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
|
||||
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
|
||||
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
|
||||
|
||||
@@ -1,21 +1,30 @@
|
||||
# REPORT — R-1285: the BYO SSH clean-up in the same update — committed, NOT released (2026-10-26 afternoon)
|
||||
# REPORT — two agent fixes on main, NOT released (night 2026-10-28 → 29, unattended)
|
||||
|
||||
**Plain words first.** A BYO box coming from 0.164.0 kept our SSH file one release too long: in the normal update the
|
||||
OLD agent installs the config bundle, so the new agent's clean-up pass never ran on it. Now the new agent runs that pass
|
||||
once, right after its self-update commits. Not released today (operator: „not today"); its live proof needs a BYO box on
|
||||
0.164.0, which rides the next agent release.
|
||||
**Plain words first.** Two changes are on main and tested. Neither is released: an agent release needs the operator's
|
||||
vouch, and the night brief allowed no release. Both go on the morning list.
|
||||
|
||||
## The fix (`b1bc63d`, CI 2363 ✓)
|
||||
## What changed
|
||||
|
||||
- `internal/selfupdate/commit.go`: `Manager.SetOnCommitted` — a hook run once after „update committed".
|
||||
- `internal/osupdate/bundle.go`: `ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's `bundle-policy` mode, host
|
||||
layer, no envelope); `cmd/felhom-agent/main.go` `wireBundlePolicyAfterCommit` hooks it on.
|
||||
- The smaller of the two fixes: agent-only — holding the bundle job until the commit would need new hub queue logic.
|
||||
- **R-528 — the OOM counter (`6f933d6`, CI run id 2473 / #284 green).** New local-API route `GET /guest/oom-kills`. It
|
||||
reads the kernel's `oom_kill` count of each Docker container of the token's own guest, on the host, from
|
||||
`/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events` (mode 0444, readable by the non-root
|
||||
agent user). Docker inside an LXC guest does not see these kills (`OOMKilled` stays false). The controller reads the
|
||||
route every 30 s (controller `eed23e7`). Tests `TestR528_*`, red first by mutation (another guest's container served;
|
||||
`oom` parsed for `oom_kill`).
|
||||
- **R-1254 — the escrow ceremony half (`5ddd124`, CI run id 2471 / #283 green).** After an escrow upload that fails, the
|
||||
agent asks the hub which package it holds (the identity blob compared): the new one → success; another one → 502 and
|
||||
„the previous code still works"; no answer → 504, which promises nothing. Tests `TestR1254_*`, red first.
|
||||
|
||||
## Proof
|
||||
## Live proof
|
||||
|
||||
- `cmd/felhom-agent/r1285_policy_after_commit_test.go`, in the real order: a pending marker for the running version →
|
||||
`commit`, then exactly one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
|
||||
removed → commit only, no pass — the demo-hp symptom of 2026-10-26).
|
||||
- **Not live:** only a box coming FROM 0.164.0 can show it (0.165.0's wrapper already applies D43 at plan time), which
|
||||
needs a signed downgrade chain on a BYO box.
|
||||
- R-528, 2026-10-28 23:27 UTC on Tester 1 with a test build of this agent (put back after: 0.167.0, sha256 `28bdd282…`).
|
||||
A Radicale capped at 128 MB, a memory hog inside it killed: the host counter 0 → 1, one `app_oom` „seen by
|
||||
docker,cgroup", one mail to the operator with `"source":"docker,cgroup"`. Evidence: felhom.eu
|
||||
`documentation/audits/night-2026-10-28/r528-oom/`.
|
||||
- R-1254: no live proof. It needs a hub answer lost on purpose; that is a hub change, and no hub deploy runs at night.
|
||||
|
||||
## Next
|
||||
|
||||
The agent release (the operator's vouch), then `pinned.day0_agent` in `scripts/retention-policy.json` (D50).
|
||||
The controller needs no MinAgent change for R-528: an old agent answers 404, and the controller then waits 10 minutes
|
||||
before it asks again.
|
||||
@@ -0,0 +1,48 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
|
||||
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
|
||||
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
|
||||
// passes as held → FAILS.
|
||||
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
|
||||
body := func(id string) []byte {
|
||||
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
|
||||
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
|
||||
return b
|
||||
}
|
||||
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
|
||||
}
|
||||
}
|
||||
ctx := context.Background()
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
|
||||
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
|
||||
}
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
|
||||
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
|
||||
}
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
|
||||
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
|
||||
}
|
||||
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
|
||||
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
|
||||
}
|
||||
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
|
||||
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
|
||||
t.Fatal("a package without an identity blob cannot be compared — an error")
|
||||
}
|
||||
}
|
||||
@@ -1250,6 +1250,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
|
||||
// subprocess used, under the same per-host key, before the code is handed out.
|
||||
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
|
||||
// R-1254 H2: after a failed upload, the hub is asked which package it holds before the household is told
|
||||
// anything — the same per-host GET the identity read-back uses.
|
||||
HubHolds: func(ctx context.Context, pkg []byte) (bool, error) {
|
||||
if client == nil {
|
||||
return false, fmt.Errorf("hub client not configured")
|
||||
}
|
||||
return hubHoldsEscrowPackage(ctx, client.FetchIdentityEscrow, pkg)
|
||||
},
|
||||
DRConfigured: func() bool {
|
||||
if drConfigured != nil {
|
||||
return drConfigured()
|
||||
@@ -3366,6 +3374,32 @@ func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
|
||||
|
||||
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
|
||||
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
|
||||
// hubHoldsEscrowPackage (R-1254 H2) reports whether the hub's CURRENT package is the upload body pkg, by its identity
|
||||
// blob — the part the hub serves back (GET /api/v1/hosts/<id>/escrow). Ciphertext only; nothing is logged. A package
|
||||
// without an identity blob cannot be compared: an error (the caller then promises nothing).
|
||||
func hubHoldsEscrowPackage(ctx context.Context, fetch func(context.Context) (*hub.IdentityEscrowResponse, error), pkg []byte) (bool, error) {
|
||||
var up escrowUploadRequest
|
||||
if err := json.Unmarshal(pkg, &up); err != nil {
|
||||
return false, fmt.Errorf("reading the pending package: %w", err)
|
||||
}
|
||||
mine, err := base64.StdEncoding.DecodeString(up.IdentityBlobB64)
|
||||
if err != nil || len(mine) == 0 {
|
||||
return false, fmt.Errorf("the pending package carries no identity blob to compare")
|
||||
}
|
||||
resp, err := fetch(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !resp.Present || resp.IdentityEscrowB64 == "" {
|
||||
return false, nil
|
||||
}
|
||||
theirs, err := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("decoding the hub's package: %w", err)
|
||||
}
|
||||
return bytes.Equal(mine, theirs), nil
|
||||
}
|
||||
|
||||
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
|
||||
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
|
||||
return fmt.Errorf("hub not configured (url/host_id/api_key)")
|
||||
|
||||
@@ -72,12 +72,56 @@ type EscrowCeremonyConfig struct {
|
||||
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
|
||||
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
|
||||
Upload func(ctx context.Context, pkg []byte) error
|
||||
// HubHolds (R-1254 H2) asks the hub whether its CURRENT package is exactly pkg — asked after a failed upload,
|
||||
// whose answer may have been lost AFTER the hub stored it. nil or an error → the outcome is unknown.
|
||||
HubHolds func(ctx context.Context, pkg []byte) (bool, error)
|
||||
}
|
||||
|
||||
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
|
||||
// ceremony incl. its upload was measured at ~2.4 s).
|
||||
const escrowUploadTimeout = 30 * time.Second
|
||||
|
||||
// escrowHoldTimeout bounds the hub question after a failed upload (R-1254 H2).
|
||||
const escrowHoldTimeout = 15 * time.Second
|
||||
|
||||
// uploadOutcome is what a claim or confirm upload did, as the HUB tells it (R-1254 H2): an upload whose answer was lost
|
||||
// or late may have been stored all the same.
|
||||
type uploadOutcome int
|
||||
|
||||
const (
|
||||
uploadStored uploadOutcome = iota // the hub holds the new package
|
||||
uploadNotStored // the hub answered and holds another package: the previous code still works
|
||||
uploadUnknown // the hub cannot be asked: nothing may be promised about either code
|
||||
)
|
||||
|
||||
// escrowUnknownOutcomeMsg is the answer when the hub cannot say which package it holds — never „the previous code
|
||||
// still works", which is false when the lost answer came after the store.
|
||||
const escrowUnknownOutcomeMsg = "the hub did not answer in time — it is not known whether it stored the new recovery package; try again (until a try succeeds, do not rely on the previous code alone)"
|
||||
|
||||
// outcomeAfterFailedUpload asks the hub whether it holds pkg after Upload returned an error. Called with the slot
|
||||
// marked claiming (escrowClaiming), outside escrowMu — it is a network call.
|
||||
func (s *Server) outcomeAfterFailedUpload(pkg []byte) uploadOutcome {
|
||||
if s.escrowCeremony.HubHolds == nil {
|
||||
return uploadUnknown
|
||||
}
|
||||
base := s.baseCtx
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(base, escrowHoldTimeout)
|
||||
defer cancel()
|
||||
held, err := s.escrowCeremony.HubHolds(ctx, pkg)
|
||||
switch {
|
||||
case err != nil:
|
||||
s.logger.Warn("local-api: escrow — the hub cannot say which package it holds after a failed upload (R-1254)", "err", err)
|
||||
return uploadUnknown
|
||||
case held:
|
||||
return uploadStored
|
||||
default:
|
||||
return uploadNotStored
|
||||
}
|
||||
}
|
||||
|
||||
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
|
||||
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
|
||||
// spike-shaped output; production is runCeremonySubprocess.
|
||||
@@ -524,15 +568,30 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
|
||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||
cancel()
|
||||
outcome := uploadStored
|
||||
if err != nil {
|
||||
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2: the answer may be lost AFTER the store
|
||||
}
|
||||
|
||||
s.escrowMu.Lock()
|
||||
s.escrowClaiming = false
|
||||
if err != nil {
|
||||
switch outcome {
|
||||
case uploadNotStored:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||
return
|
||||
case uploadUnknown:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow claim — upload failed and the hub cannot say which package it holds; code kept claimable (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: escrow claim — the upload's answer was lost, but the hub holds the new package; handing out the code (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err)
|
||||
}
|
||||
s.escrowPending = nil
|
||||
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
|
||||
@@ -631,15 +690,30 @@ func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Requ
|
||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||
cancel()
|
||||
outcome := uploadStored
|
||||
if err != nil {
|
||||
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2
|
||||
}
|
||||
|
||||
s.escrowMu.Lock()
|
||||
s.escrowClaiming = false
|
||||
if err != nil {
|
||||
switch outcome {
|
||||
case uploadNotStored:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||
return
|
||||
case uploadUnknown:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow confirm — upload failed and the hub cannot say which package it holds; still waiting (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: escrow confirm — the upload's answer was lost, but the hub holds the new package (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err)
|
||||
}
|
||||
s.escrowPending = nil
|
||||
s.wipeEscrowConfirmLocked()
|
||||
|
||||
@@ -41,9 +41,13 @@ type fakeEscrowHub struct {
|
||||
mu sync.Mutex
|
||||
current []byte
|
||||
uploads int
|
||||
err error // non-nil → the upload fails, nothing stored
|
||||
gate chan struct{} // non-nil → the upload blocks until closed
|
||||
entered chan struct{} // non-nil → closed when an upload starts
|
||||
err error // non-nil → the upload fails, nothing stored
|
||||
// R-1254 H2: lost → the upload STORES the package and then fails (the answer is lost on the way back);
|
||||
// holdsErr → the hub cannot be asked which package it holds.
|
||||
lost bool
|
||||
holdsErr error
|
||||
gate chan struct{} // non-nil → the upload blocks until closed
|
||||
entered chan struct{} // non-nil → closed when an upload starts
|
||||
}
|
||||
|
||||
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
||||
@@ -61,9 +65,23 @@ func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
||||
}
|
||||
f.current = append([]byte(nil), pkg...)
|
||||
f.uploads++
|
||||
if f.lost {
|
||||
return context.DeadlineExceeded
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// holds answers „is the hub's current package exactly pkg?" — the R-1254 question (production compares the identity
|
||||
// blob the hub serves back).
|
||||
func (f *fakeEscrowHub) holds(_ context.Context, pkg []byte) (bool, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.holdsErr != nil {
|
||||
return false, f.holdsErr
|
||||
}
|
||||
return string(f.current) == string(pkg), nil
|
||||
}
|
||||
|
||||
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
|
||||
func (f *fakeEscrowHub) opensWith(r string) bool {
|
||||
f.mu.Lock()
|
||||
@@ -100,6 +118,7 @@ func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub
|
||||
HubConfigured: true,
|
||||
DRConfigured: func() bool { return true },
|
||||
Upload: hub.upload,
|
||||
HubHolds: hub.holds,
|
||||
}
|
||||
srv.ceremonyRun = run
|
||||
srv.escrowSudoCheck = func(context.Context) error { return nil }
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1254 (the ceremony half, review round 2 HUNT-3 H2): the upload's answer can be lost or late (30 s) AFTER the hub
|
||||
// stored the new package. The household was then told „the previous code still works" — false: the hub now holds the
|
||||
// package the NEW code opens. The agent asks the hub which package it holds before it words the result:
|
||||
// - the hub holds the new package → success (the code is handed out / the confirm completes);
|
||||
// - the hub holds another package → the old sentence, now true;
|
||||
// - the hub cannot be asked → 504 and a sentence that does NOT promise the previous code; the wait goes on.
|
||||
|
||||
// RED-PROOF: remove the hub question from the claim's failure branch → 502 „the previous code still works" while the
|
||||
// hub holds the new package → FAILS.
|
||||
func TestR1254_ALostAnswerAtTheClaimIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
code, body := claimCode(t, h)
|
||||
if !hub.opensWith(firstR) {
|
||||
t.Fatal("fixture: the hub must hold the new package (the answer was lost after the store)")
|
||||
}
|
||||
if code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||
t.Fatalf("claim after a lost answer, the hub holding the new package: %d %s — want 200 with the code", code, body)
|
||||
}
|
||||
if strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("the household was told the previous code still works: %s", body)
|
||||
}
|
||||
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
|
||||
t.Fatalf("the job must read uploaded: %s", st)
|
||||
}
|
||||
}
|
||||
|
||||
// The hub cannot be asked: no promise about the previous code, 504, the code stays claimable (a retry decides).
|
||||
// RED-PROOF: as above → 502 with „the previous code still works" → FAILS.
|
||||
func TestR1254_AnUnknownOutcomeAtTheClaimPromisesNothing(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true, holdsErr: errors.New("hub unreachable")}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
code, body := claimCode(t, h)
|
||||
if code != http.StatusGatewayTimeout {
|
||||
t.Fatalf("claim with an unknown outcome: %d %s — want 504", code, body)
|
||||
}
|
||||
if strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("an unknown outcome promised the previous code: %s", body)
|
||||
}
|
||||
assertNoRIn(t, "unknown-outcome claim payload", body, firstR)
|
||||
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) {
|
||||
t.Fatalf("the code must stay claimable: %s", st)
|
||||
}
|
||||
// The hub answers again: the retry finds the new package already stored and hands the code out.
|
||||
hub.mu.Lock()
|
||||
hub.holdsErr = nil
|
||||
hub.mu.Unlock()
|
||||
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||
t.Fatalf("retry: %d %s", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The hub holds ANOTHER package (the upload really failed): the old sentence stays, and is now true.
|
||||
func TestR1254_AFailedUploadTheHubDidNotStoreKeepsTheOldSentence(t *testing.T) {
|
||||
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
if code, body := claimCode(t, h); code != http.StatusBadGateway || !strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("a failed upload: %d %s — want 502 and the previous code named", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The same at the typed-back confirm (R-1151).
|
||||
// RED-PROOF: remove the hub question from the confirm's failure branch → 502 → FAILS.
|
||||
func TestR1254_ALostAnswerAtTheConfirmIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
c := claimDeferred(t, h)
|
||||
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
|
||||
t.Fatalf("confirm after a lost answer: %d (opens: %v) — want 200", code, hub.opensWith(secondR))
|
||||
}
|
||||
hub.mu.Lock()
|
||||
hub.lost, hub.holdsErr = true, errors.New("hub unreachable")
|
||||
hub.mu.Unlock()
|
||||
srv2, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h2 := srv2.Handler()
|
||||
startAndWait(t, srv2, h2)
|
||||
c2 := claimDeferred(t, h2)
|
||||
if code := confirm(t, h2, c2.typedBack(t)); code != http.StatusGatewayTimeout {
|
||||
t.Fatalf("confirm with an unknown outcome: %d — want 504", code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// R-528 — the OOM signal from the host side.
|
||||
//
|
||||
// MEASURED 2026-09-15 (scratch 9202, Docker 29.8.0) and 2026-09-16 (a fresh install in a nested VM): an app killed by
|
||||
// its memory limit inside a Felhom LXC guest left Docker's State.OOMKilled false and sent no `oom` event, so the
|
||||
// controller's OOM line (R-514) could not fire. The kernel's own counter does count every kill, and the host can read
|
||||
// it: /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events, line `oom_kill N` (the
|
||||
// systemd cgroup driver; `ns/docker/<id>/` under the cgroupfs driver). Read on Tester 1's host 2026-10-28: the files
|
||||
// are mode 0444, readable by the non-root felhom-agent user — no sudo, no new root path.
|
||||
//
|
||||
// A container's scope is new at each start, so the counter is per container RUN: a count above 0 means this run had a
|
||||
// kill. The controller asks every 30 s and raises its existing app_oom line (once per run) for any such container.
|
||||
|
||||
// readGuestOOMKills returns container id → oom_kill for one guest. available=false (no error) when the guest has no
|
||||
// cgroup folder (a VM guest, or a container that is not running).
|
||||
func readGuestOOMKills(root string, vmid int) (map[string]int64, bool, error) {
|
||||
if root == "" {
|
||||
root = "/sys/fs/cgroup"
|
||||
}
|
||||
base := filepath.Join(root, "lxc", strconv.Itoa(vmid), "ns")
|
||||
if fi, err := os.Stat(base); err != nil || !fi.IsDir() {
|
||||
return nil, false, nil
|
||||
}
|
||||
out := map[string]int64{}
|
||||
for _, pattern := range []string{
|
||||
filepath.Join(base, "system.slice", "docker-*.scope", "memory.events"),
|
||||
filepath.Join(base, "docker", "*", "memory.events"),
|
||||
} {
|
||||
files, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil, true, err
|
||||
}
|
||||
for _, f := range files {
|
||||
id := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(filepath.Dir(f)), "docker-"), ".scope")
|
||||
if !isContainerID(id) {
|
||||
continue
|
||||
}
|
||||
n, err := readOOMKill(f)
|
||||
if err != nil {
|
||||
continue // a container that stopped between the glob and the read
|
||||
}
|
||||
out[id] = n
|
||||
}
|
||||
}
|
||||
return out, true, nil
|
||||
}
|
||||
|
||||
func isContainerID(s string) bool {
|
||||
if len(s) != 64 {
|
||||
return false
|
||||
}
|
||||
for _, c := range s {
|
||||
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func readOOMKill(path string) (int64, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
k, v, ok := strings.Cut(strings.TrimSpace(sc.Text()), " ")
|
||||
if ok && k == "oom_kill" {
|
||||
return strconv.ParseInt(strings.TrimSpace(v), 10, 64)
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return 0, fmt.Errorf("no oom_kill line in %s", path)
|
||||
}
|
||||
|
||||
// handleGuestOOMKills is GET /guest/oom-kills: {"vmid", "available", "containers": {id: oom_kill}}.
|
||||
func (s *Server) handleGuestOOMKills(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
kills, avail, err := readGuestOOMKills(s.cgroupRoot, vmid)
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: guest oom-kills read failed (R-528)", "vmid", vmid, "err", err)
|
||||
writeErr(w, http.StatusInternalServerError, "the guest's OOM counters could not be read")
|
||||
return
|
||||
}
|
||||
if kills == nil {
|
||||
kills = map[string]int64{}
|
||||
}
|
||||
writeOK(w, map[string]any{"vmid": vmid, "available": avail, "containers": kills})
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-528: Docker does not always report an OOM kill inside a Felhom LXC guest (OOMKilled stays false, no `oom` event —
|
||||
// measured 2026-09-15/16 on two boxes), but the kernel's own counter does count it, and it is readable from the host:
|
||||
// /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events, line `oom_kill N`. GET /guest/oom-kills
|
||||
// serves the asking guest's counters, and only that guest's.
|
||||
|
||||
func fakeCgroup(t *testing.T, root string, vmid, id, events string) {
|
||||
t.Helper()
|
||||
dir := filepath.Join(root, "lxc", vmid, "ns", "system.slice", "docker-"+id+".scope")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o444); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
const idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
const idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||
const idOther = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
|
||||
|
||||
// RED-PROOF: serve the counters of every guest under lxc/ (drop the vmid from the glob) → the other guest's container
|
||||
// appears → FAILS; parse `oom ` instead of `oom_kill` → idA reads 9 → FAILS.
|
||||
func TestR528_GuestOOMKillsServesThisGuestsCountersOnly(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
fakeCgroup(t, root, "100", idA, "low 0\nhigh 0\nmax 4\noom 9\noom_kill 3\noom_group_kill 0\n")
|
||||
fakeCgroup(t, root, "100", idB, "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\n")
|
||||
fakeCgroup(t, root, "200", idOther, "oom_kill 7\n")
|
||||
// a scope that is not a container (no 64-hex id) and a cgroupfs-driver layout container
|
||||
if err := os.MkdirAll(filepath.Join(root, "lxc", "100", "ns", "system.slice", "cron.service"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dfDir := filepath.Join(root, "lxc", "100", "ns", "docker", idOther[:8]+strings.Repeat("d", 56))
|
||||
if err := os.MkdirAll(dfDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dfDir, "memory.events"), []byte("oom_kill 2\n"), 0o444)
|
||||
|
||||
got, avail, err := readGuestOOMKills(root, 100)
|
||||
if err != nil || !avail {
|
||||
t.Fatalf("read: avail=%v err=%v", avail, err)
|
||||
}
|
||||
want := map[string]int64{idA: 3, idB: 0, idOther[:8] + strings.Repeat("d", 56): 2}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
for k, v := range want {
|
||||
if got[k] != v {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
if _, leaked := got[idOther]; leaked {
|
||||
t.Fatal("another guest's container leaked into this guest's answer")
|
||||
}
|
||||
// A guest with no cgroup folder (a VM, or not running): not available, no error.
|
||||
if _, avail, err := readGuestOOMKills(root, 300); avail || err != nil {
|
||||
t.Fatalf("a guest without a cgroup folder: avail=%v err=%v", avail, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Through the route, with the guest's own token.
|
||||
func TestR528_TheRouteAnswersTheTokensGuest(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
fakeCgroup(t, root, "8200", idA, "oom_kill 5\n")
|
||||
fakeCgroup(t, root, "9300", idB, "oom_kill 1\n") // token B's guest: never in A's answer
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
srv.cgroupRoot = root
|
||||
w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "A", "")
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("route: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
var env struct {
|
||||
Data struct {
|
||||
VMID int `json:"vmid"`
|
||||
Available bool `json:"available"`
|
||||
Containers map[string]int64 `json:"containers"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if env.Data.VMID != 8200 || !env.Data.Available || len(env.Data.Containers) != 1 || env.Data.Containers[idA] != 5 {
|
||||
t.Fatalf("answer %s — want guest 8200's one container with 5", w.Body.String())
|
||||
}
|
||||
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("a cross-guest ask: %d, want 403", w.Code)
|
||||
}
|
||||
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "", ""); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: %d", w.Code)
|
||||
}
|
||||
}
|
||||
@@ -459,6 +459,8 @@ type Server struct {
|
||||
escrowLookPath func(file string) (string, error)
|
||||
// statFile reports whether a path exists (preflight's staged-secret item; tests inject).
|
||||
statFile func(path string) bool
|
||||
// cgroupRoot is where the host's cgroup tree is mounted (R-528, GET /guest/oom-kills); "" = /sys/fs/cgroup. Tests set it.
|
||||
cgroupRoot string
|
||||
|
||||
baseCtx context.Context // for fire-and-forget backups; set in Run
|
||||
}
|
||||
@@ -576,6 +578,9 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
|
||||
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
|
||||
// read it. Read-only; never measured (or the job not wired) = present:false.
|
||||
// R-528: the kernel's OOM-kill counter of each of the guest's containers, read on the host (Docker's own flag and
|
||||
// event can stay silent inside an LXC guest). Read-only; the asking guest's containers only.
|
||||
mux.HandleFunc("GET /guest/oom-kills", s.withGuest(s.handleGuestOOMKills))
|
||||
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
|
||||
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
|
||||
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
||||
|
||||
@@ -24,10 +24,10 @@
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||
],
|
||||
"recorded": "2026-10-26",
|
||||
"recorded_by": "CC 2026-10-26: day0_agent 0.166.0 (operator vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"recorded": "2026-10-27",
|
||||
"recorded_by": "CC 2026-10-27: day0_agent 0.167.0 (operator vouch 2026-10-27, read from the hub: version + agent and bundle sha256 match the release); earlier 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"pinned": {
|
||||
"hub_min_agent": "0.158.1",
|
||||
"day0_agent": "0.166.0"
|
||||
"day0_agent": "0.167.0"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user