Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
637c92c762 | ||
|
|
63e63ee8bb | ||
|
|
6b0b093199 | ||
|
|
f4e63a55da | ||
|
|
91bd96c755 | ||
|
|
6f933d6ee3 | ||
|
|
5ddd124cfc | ||
|
|
3055bc54c1 | ||
|
|
007199342f | ||
|
|
54d3a1057b | ||
|
|
fbbb168d1f | ||
|
|
3911cc6f52 | ||
|
|
ad80ee749a | ||
|
|
0683cc2dab | ||
|
|
b1bc63dd32 | ||
|
|
e66a48f2b7 | ||
|
|
b890b57540 |
No files matched your search
+97
-3
@@ -1,4 +1,98 @@
|
||||
## 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
|
||||
## Unreleased (2026-10-29, afternoon) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — commit only; the next agent release (needs the operator's vouch)
|
||||
|
||||
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
|
||||
carry `operator_actions: [{id, action, arg}]` (`hub.ControlEnvelope.OperatorActions`); the agent acts ONCE per id and
|
||||
answers on the next report, `operator_action_results: [{id, outcome, message}]` (`hub.HostReport`), which it requests
|
||||
at once through the cap-1 report trigger. CLOSED list: `restart_controller <vmid>` only — anything else refused, nothing
|
||||
called. The act is the supervisor's (`systemctl restart felhom-controller-bootstrap.service` in the guest, existing
|
||||
grants — no sudoers change, no config bundle) with its guards: provisioned guest, guest running, the guest list and the
|
||||
vzdump state readable, not parked, no swap, not locked, no vzdump. The thrash brake counts operator restarts with the
|
||||
supervisor's: 3 in 15 minutes → the press is refused, and presses stay refused for 30 minutes. The result says how long
|
||||
the controller took to answer (Docker's health check: healthy), or `failed` when it did not within 3 minutes. One
|
||||
`audit: operator action` line per decision. Works with the controller stopped. Tests `TestOpActions_*` (closed list,
|
||||
unknown/bad arg refused, restart + answer time + report trigger + once per id + re-send until dropped, stopped
|
||||
controller, every guard, the brake across supervisor and operator restarts, no healthy answer = failed) and
|
||||
`TestMainWiresTheAgentOperatorActions` (AST) — each seen failing by a mutation.
|
||||
- **R-1344 — a controller that RUNS but does not answer: ALARM ONLY.** Each sweep also reads Docker's health status for a
|
||||
running controller (the swap's `docker inspect -f` grant); 6 consecutive „unhealthy" sweeps (3 minutes, on top of
|
||||
Docker's own ~90 s) start an episode: one ERROR line, the stanza's `unresponsive` / `unresponsive_since` (the hub raises
|
||||
`controller_unresponsive`, operator-only). NOT restarted for it: the brief allows the restart only after a measurement
|
||||
shows a busy night never reads like this, and that measurement runs through tonight (felhom.eu R-1344). Not counted
|
||||
during a swap, a parked or locked guest, or a vzdump; healthy / starting / no health check / an unreadable status reset
|
||||
the count. Tests `TestSupervisor_Unresponsive*` (red by mutation: the vzdump guard, the threshold).
|
||||
|
||||
## v0.168.0 (2026-10-29) — R-528: the host's per-container OOM counter (`GET /guest/oom-kills`); R-1254: a lost or late escrow answer is asked again before it is worded — RELEASED (release-agent.sh from a clean clone of `91bd96c`; agent sha256 1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca, bundle 3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8; no configs/ change since v0.167.0 → binary-only signed updates to demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 10:21:56 CET**; Day-0 pin set (D50)
|
||||
|
||||
### R-528
|
||||
|
||||
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
|
||||
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
|
||||
could not fire. The kernel counts every kill per container cgroup, and the host can read it.
|
||||
- **`internal/localapi/guest_oom.go`:** `GET /guest/oom-kills` (withGuest — the token's guest only; a cross-guest
|
||||
`?vmid=` is 403) reads `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex>.scope/memory.events` (and
|
||||
`ns/docker/<id>/` for the cgroupfs driver), line `oom_kill N`, and answers `{vmid, available, containers: {id: n}}`.
|
||||
A guest with no cgroup folder → `available: false`. Read-only, no sudo: the files are 0444 and the non-root
|
||||
felhom-agent user reads them (measured on Tester 1's host 2026-10-28). A container's scope is new at each start, so the
|
||||
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
|
||||
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
|
||||
|
||||
### R-1254 (the ceremony half, review round 2 HUNT-3 H2)
|
||||
|
||||
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
|
||||
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
|
||||
hub's current package opens only with the NEW code.
|
||||
- **Now (`internal/localapi/escrow_ceremony.go`):** after a failed upload the agent asks the hub (`HubHolds`, 15 s,
|
||||
outside the lock, the slot still marked claiming): the hub holds THIS package → a success (the code is handed out /
|
||||
the confirm completes; a WARN line names the lost answer); the hub holds another package → 502 and the old sentence,
|
||||
now true; the hub cannot be asked → **504** with a sentence that promises nothing about either code, and the wait goes
|
||||
on (a retry decides). `cmd/felhom-agent`: `HubHolds` compares the pending package's identity blob with the one the hub
|
||||
serves back (`GET /api/v1/hosts/<id>/escrow`, the read-back the agent already uses; ciphertext only, nothing logged);
|
||||
a package without an identity blob cannot be compared → unknown. Tests `TestR1254_*` (localapi, red before the fix:
|
||||
502 while the hub held the new package) and `TestR1254_HubHoldsComparesTheIdentityBlob` (red-proved by mutation).
|
||||
- Needs controller ≥ the R-1254 controller commit (2026-10-28 night) to word 504 for the household; an older controller
|
||||
shows its generic „A kód lekérése nem sikerült." (claim) / its 502 sentence (confirm) for it.
|
||||
|
||||
## v0.167.0 (2026-10-27) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — RELEASED (release-agent.sh from a clean clone; agent sha256 28bdd282…d826f0, bundle 9d034416…ed1268; signed updates committed on demo-felhom, Tester 1, demo-hp; VOUCHED by the operator 2026-10-27, Day-0 pin set)
|
||||
|
||||
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
|
||||
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
|
||||
newest. Full `go build/vet/test` green. The controller and hub move the same night.
|
||||
- **R-1232 (P2):** `backup.BackupRunner.NewestArchiveTime` — the tier due-check — counts only archives written with THIS
|
||||
storage's key on an encrypted storage (the R-727 discriminator; the archive carries no host id). Before: a reinstalled
|
||||
box shares the customer's PBS namespace and guest number with the old box, the old box's archive read as „the last
|
||||
backup", and the whole-guest tier waited up to its 168-h cadence (the 2026-10-23 test night). A key lookup failure is
|
||||
an error (the due-check reads UNKNOWN), never a foreign age. Tests `TestR1232_*` (foreign only → none; own → its time;
|
||||
an unencrypted dir → unfiltered), red-proved (case 1 returned the old box's time).
|
||||
|
||||
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
|
||||
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
|
||||
second later — so 0.166.0's pass never ran on the bundle that delivered it.
|
||||
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
|
||||
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
|
||||
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
|
||||
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
|
||||
agent-only — holding the bundle job until the commit would need new hub queue logic.
|
||||
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
|
||||
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
|
||||
removed → commit only, no pass — the demo-hp symptom).
|
||||
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
|
||||
0.165.0 box passes even without this fix.
|
||||
|
||||
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
|
||||
|
||||
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
|
||||
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
|
||||
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
|
||||
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
|
||||
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
|
||||
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
|
||||
|
||||
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
|
||||
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
|
||||
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
|
||||
„BUNDLE POLICY DONE install_mode=appliance removed=0".
|
||||
|
||||
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
|
||||
|
||||
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
|
||||
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
|
||||
@@ -13,7 +107,7 @@
|
||||
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
|
||||
`documentation/audits/night-2026-10-25/redproof-r1285.txt`.
|
||||
|
||||
## 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
|
||||
### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
|
||||
|
||||
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
|
||||
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
|
||||
@@ -25,7 +119,7 @@
|
||||
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
|
||||
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
|
||||
|
||||
## 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
|
||||
### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
|
||||
|
||||
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
|
||||
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
|
||||
|
||||
+17
@@ -1,5 +1,22 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
|
||||
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
|
||||
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
|
||||
> R-528 proven on the release (Tester 1: one `app_oom` „seen by docker,cgroup", host counter 1). R-1254: the happy path ran
|
||||
> live (a ceremony on 0.168.0 + controller 0.347.0); the lost-answer branch has tests only.
|
||||
|
||||
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
|
||||
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
|
||||
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
|
||||
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
|
||||
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
|
||||
|
||||
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
|
||||
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
|
||||
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
|
||||
> on demo-hp (felhom.eu R-1250, read, not forced).
|
||||
|
||||
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
|
||||
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
|
||||
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
|
||||
|
||||
@@ -1,26 +1,18 @@
|
||||
# REPORT — D50 retention gate, R-1280, R-1285: committed, not released (2026-10-25 night)
|
||||
# REPORT — v0.168.0 released and vouched (2026-10-29, attended)
|
||||
|
||||
**Plain words first.** Three agent changes are on `main`. None is released. The release (v0.166.0) is Monday's and
|
||||
waits for the operator's vouch (felhom.eu `STATUS.md`, the Monday list).
|
||||
**Plain words first.** The agent 0.168.0 is out, on the three test boxes, and vouched by the operator — new boxes install
|
||||
it. It carries R-528 (the host's per-container OOM count) and R-1254's ceremony half.
|
||||
|
||||
## Each change, before → after
|
||||
- **Release:** `release-agent.sh` from a clean clone of `91bd96c` (CI 2479 green): agent sha256
|
||||
`1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca`, bundle
|
||||
`3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8`, tag `v0.168.0`, verified by independent download.
|
||||
- **Delivery:** no configs/ change since v0.167.0 → one signed `agent_update` per box: demo-hp 09:56, Tester 1 09:57,
|
||||
demo-felhom 10:04 CET — each „signed op COMPLETED", self-check 76/76, „update committed"; the binary's sha256 matches on
|
||||
all three.
|
||||
- **Proof:** on Tester 1 (controller 0.347.0) a 128 MB Gokapi had a memory hog killed: host counter 1, one `app_oom`
|
||||
„seen by docker,cgroup", one operator mail. R-1254: a ceremony ran on 0.168.0 (the happy path); the lost-answer branch
|
||||
has tests only.
|
||||
- **Vouch:** the operator, 10:21:56 CET (hub log: agent=0.168.0, the bundle sha above). `pinned.day0_agent` = 0.168.0
|
||||
(`6b0b093`).
|
||||
|
||||
| Row | Commit | Before | After | Proof |
|
||||
|---|---|---|---|---|
|
||||
| R-1290 (D50 = A) | `feaaa0e` | the published-versions gate knew no keep rule | it asserts the newest 5 released versions plus the pinned MinAgent floor (0.158.1) and the day-0 agent (0.165.0) in `scripts/retention-policy.json`; a pin with no tag fails | decoys seen failing; CI 2339 ✓ |
|
||||
| R-1280 | `d912307` | a signed job's window could be any length | at most 24 h — the verifier (`ErrWindowTooLong`), the root wrapper (`MAX_WINDOW_S`) and `felhom-opsign -ttl`; exactly 24 h is accepted | red-proof; unit tests; not live |
|
||||
| R-1285 | `abcc1e6` | a BYO host upgraded from an old wrapper kept our SSH drop-in until its NEXT bundle (the old wrapper planned the new bundle) | after every bundle install the agent runs the new wrapper's `bundle-policy` pass; the file is removed in one update; a failing `sshd -t` puts it back | red-proof; live on a BYO test VM: one signed run gave `BUNDLE POLICY REMOVED`; the test package was deleted after (204, then 404) |
|
||||
|
||||
## CI (by commit hash, `jobs` endpoint)
|
||||
|
||||
`feaaa0e` 2339 ✓ · `d912307` 2344 ✓ · `abcc1e6` 2345 ✓ — all `run_attempt` 1.
|
||||
|
||||
## Teardown
|
||||
|
||||
The BYO test VM (demo-hp 9372) was uninstalled through the product and destroyed. Its hub host record needs the
|
||||
operator's delete (felhom.eu `STATUS.md`).
|
||||
|
||||
## Evidence
|
||||
|
||||
felhom.eu `documentation/audits/night-2026-10-25/` (`redproof-d50.txt`, `redproof-r1280.txt`, `redproof-r1285.txt`,
|
||||
`rig/r1285-live.txt`).
|
||||
Evidence: felhom.eu `documentation/audits/d63-2026-10-29/` (`opsign-agent.txt`, `agent-delivery*.txt`, `run-p4-oom.log`).
|
||||
@@ -89,6 +89,7 @@
|
||||
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
|
||||
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
|
||||
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
|
||||
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller` only, `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
|
||||
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
|
||||
|
||||
### Proxmox client / hub / PBS / provisioning
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
|
||||
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
|
||||
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
|
||||
// passes as held → FAILS.
|
||||
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
|
||||
body := func(id string) []byte {
|
||||
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
|
||||
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
|
||||
return b
|
||||
}
|
||||
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
|
||||
}
|
||||
}
|
||||
ctx := context.Background()
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
|
||||
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
|
||||
}
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
|
||||
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
|
||||
}
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
|
||||
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
|
||||
}
|
||||
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
|
||||
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
|
||||
}
|
||||
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
|
||||
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
|
||||
t.Fatal("a package without an identity blob cannot be compared — an error")
|
||||
}
|
||||
}
|
||||
@@ -1216,6 +1216,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||
}}
|
||||
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
|
||||
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
|
||||
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
|
||||
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
|
||||
// to the local API server built below).
|
||||
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
|
||||
@@ -1247,6 +1250,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
|
||||
// subprocess used, under the same per-host key, before the code is handed out.
|
||||
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
|
||||
// R-1254 H2: after a failed upload, the hub is asked which package it holds before the household is told
|
||||
// anything — the same per-host GET the identity read-back uses.
|
||||
HubHolds: func(ctx context.Context, pkg []byte) (bool, error) {
|
||||
if client == nil {
|
||||
return false, fmt.Errorf("hub client not configured")
|
||||
}
|
||||
return hubHoldsEscrowPackage(ctx, client.FetchIdentityEscrow, pkg)
|
||||
},
|
||||
DRConfigured: func() bool {
|
||||
if drConfigured != nil {
|
||||
return drConfigured()
|
||||
@@ -1256,6 +1267,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localSrv != nil {
|
||||
// 2026-10-29 (operator brief, Part 1): the operator's actions for THIS agent ride the host report's reply — the
|
||||
// closed list (localapi/opactions.go). The server joins the observer chain; its results ride the next report,
|
||||
// which it requests at once through the same cap-1 trigger the storage watchdog uses.
|
||||
localSrv.SetOperatorActionTrigger(func() {
|
||||
select {
|
||||
case storageTrigger <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
})
|
||||
collector.SetOperatorActionResults(localSrv.OperatorActionResults)
|
||||
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner, localSrv))
|
||||
backupCapRefusal = localSrv.BackupCapRefusal // R-973
|
||||
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
|
||||
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
|
||||
@@ -3363,6 +3385,32 @@ func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
|
||||
|
||||
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
|
||||
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
|
||||
// hubHoldsEscrowPackage (R-1254 H2) reports whether the hub's CURRENT package is the upload body pkg, by its identity
|
||||
// blob — the part the hub serves back (GET /api/v1/hosts/<id>/escrow). Ciphertext only; nothing is logged. A package
|
||||
// without an identity blob cannot be compared: an error (the caller then promises nothing).
|
||||
func hubHoldsEscrowPackage(ctx context.Context, fetch func(context.Context) (*hub.IdentityEscrowResponse, error), pkg []byte) (bool, error) {
|
||||
var up escrowUploadRequest
|
||||
if err := json.Unmarshal(pkg, &up); err != nil {
|
||||
return false, fmt.Errorf("reading the pending package: %w", err)
|
||||
}
|
||||
mine, err := base64.StdEncoding.DecodeString(up.IdentityBlobB64)
|
||||
if err != nil || len(mine) == 0 {
|
||||
return false, fmt.Errorf("the pending package carries no identity blob to compare")
|
||||
}
|
||||
resp, err := fetch(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !resp.Present || resp.IdentityEscrowB64 == "" {
|
||||
return false, nil
|
||||
}
|
||||
theirs, err := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("decoding the hub's package: %w", err)
|
||||
}
|
||||
return bytes.Equal(mine, theirs), nil
|
||||
}
|
||||
|
||||
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
|
||||
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
|
||||
return fmt.Errorf("hub not configured (url/host_id/api_key)")
|
||||
@@ -4040,3 +4088,9 @@ func agentStateDir(cfg config.Config) string {
|
||||
}
|
||||
return "/var/lib/felhom-agent"
|
||||
}
|
||||
|
||||
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
|
||||
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
|
||||
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
|
||||
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 2026-10-29 (operator brief, Part 1): the agent's operator actions are unit-green in localapi; main.go must also put
|
||||
// the server in the report loop's observer chain, give the collector its results and give the server the report
|
||||
// trigger — an inert seam otherwise (the „seam built but never wired" class). Asserted from the AST.
|
||||
// RED-PROOF: seen failing before the wiring existed (all three false).
|
||||
func TestMainWiresTheAgentOperatorActions(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
var observed, results, trigger bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
fn, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
switch fn.Sel.Name {
|
||||
case "MultiObserver":
|
||||
for _, a := range call.Args {
|
||||
if id, ok := a.(*ast.Ident); ok && id.Name == "localSrv" {
|
||||
observed = true
|
||||
}
|
||||
}
|
||||
case "SetOperatorActionResults":
|
||||
if len(call.Args) == 1 {
|
||||
if sel, ok := call.Args[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "OperatorActionResults" {
|
||||
results = true
|
||||
}
|
||||
}
|
||||
case "SetOperatorActionTrigger":
|
||||
trigger = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !observed || !results || !trigger {
|
||||
t.Fatalf("operator actions wiring: observer=%v results=%v trigger=%v — all three must be wired in main.go", observed, results, trigger)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
|
||||
)
|
||||
|
||||
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
|
||||
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
|
||||
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
|
||||
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
|
||||
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
|
||||
|
||||
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
|
||||
type orderRunner struct {
|
||||
mu sync.Mutex
|
||||
order []string
|
||||
}
|
||||
|
||||
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
|
||||
var p map[string]any
|
||||
b, _ := os.ReadFile(args[1])
|
||||
_ = json.Unmarshal(b, &p)
|
||||
mode, _ := p["mode"].(string)
|
||||
r.order = append(r.order, "os-apply:"+mode)
|
||||
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
|
||||
}
|
||||
if len(args) >= 1 {
|
||||
r.order = append(r.order, filepath.Base(name)+":"+args[0])
|
||||
}
|
||||
return []byte("ok"), nil, nil
|
||||
}
|
||||
|
||||
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
|
||||
t.Helper()
|
||||
stateDir := t.TempDir()
|
||||
if marker != nil {
|
||||
dir := filepath.Join(stateDir, "selfupdate")
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := json.Marshal(marker)
|
||||
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
r := &orderRunner{}
|
||||
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
|
||||
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
|
||||
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
|
||||
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
|
||||
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
|
||||
mgr.MaybeCommit(context.Background())
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return append([]string(nil), r.order...)
|
||||
}
|
||||
|
||||
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
|
||||
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
|
||||
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
|
||||
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
|
||||
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
|
||||
}
|
||||
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
module gitea.dooplex.hu/admin/felhom-agent
|
||||
|
||||
go 1.27.2
|
||||
go 1.27.3
|
||||
|
||||
require (
|
||||
golang.org/x/crypto v0.58.0
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1232 — the 2026-10-23 test night: a reinstalled box (new install, new PBS key, same customer namespace, same guest
|
||||
// number 9201) found the OLD box's archive (2.6 days old) when the due-check asked „when did this tier last land?", so
|
||||
// the whole-guest tier read „not due" for up to its 168-h cadence and the ep0 copy stayed the OLD box's. The due-check
|
||||
// now counts only archives written with THIS storage's key (the R-727 discriminator; the archive carries no host id).
|
||||
// The CONSEQUENCE asserted: with only a foreign archive the answer is „none found" (the tier is due); with the box's
|
||||
// own archive, that archive's time. COMPANION RED-PROOF: drop the key filter in NewestArchiveTime → case 1 returns the
|
||||
// old box's time.
|
||||
func TestR1232_TheDueCheckCountsOnlyThisBoxsArchives(t *testing.T) {
|
||||
oldBox := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-20T21:10:00Z", Content: "backup", VMID: 9201,
|
||||
Size: 3490689830, CTime: time.Date(2026, 10, 20, 21, 10, 0, 0, time.UTC).Unix(), Encrypted: earlierBox1}
|
||||
api := &fakeBackupAPI{
|
||||
storages: []proxmox.Storage{{Storage: "felhom-pbsdr", Type: "pbs", EncryptionKey: thisBoxKey}},
|
||||
content: []proxmox.StorageContent{oldBox},
|
||||
}
|
||||
r := NewBackupRunner(api, "felhom-pbsdr", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
|
||||
// 1. The fresh box has written nothing yet: the old box's archive must not count.
|
||||
when, found, err := r.NewestArchiveTime(context.Background(), 9201)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if found {
|
||||
t.Fatalf("the due-check counted ANOTHER box's archive (%s) as this tier's last backup — the tier would wait a week", when)
|
||||
}
|
||||
// 2. After the box's own first copy, that copy is the answer.
|
||||
own := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-24T02:30:00Z", Content: "backup", VMID: 9201,
|
||||
Size: 3600000000, CTime: time.Date(2026, 10, 24, 2, 30, 0, 0, time.UTC).Unix(), Encrypted: thisBoxKey}
|
||||
api.content = append(api.content, own)
|
||||
when, found, err = r.NewestArchiveTime(context.Background(), 9201)
|
||||
if err != nil || !found || when.Unix() != own.CTime {
|
||||
t.Fatalf("want this box's own archive time %d, got %v found=%v err=%v", own.CTime, when, found, err)
|
||||
}
|
||||
}
|
||||
|
||||
// An unencrypted storage (a local dir) holds only this box's vzdumps: no key filter, the newest counts.
|
||||
func TestR1232_UnencryptedStorageCountsEveryArchive(t *testing.T) {
|
||||
api := &fakeBackupAPI{
|
||||
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
|
||||
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_10_25-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1793050025}},
|
||||
}
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
if _, found, err := r.NewestArchiveTime(context.Background(), 9201); err != nil || !found {
|
||||
t.Fatalf("an unencrypted local archive was not counted: found=%v err=%v", found, err)
|
||||
}
|
||||
}
|
||||
@@ -609,11 +609,23 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
|
||||
if err != nil {
|
||||
return time.Time{}, false, err
|
||||
}
|
||||
// R-1232: on an ENCRYPTED storage only archives written with THIS storage's key are this box's (the R-727
|
||||
// discriminator — the archive carries no host id). A reinstalled box shares the customer's namespace and guest
|
||||
// number with the old box, and the old box's archive made the tier read „not due" for a whole cadence (168 h).
|
||||
// A key lookup failure is returned: the due-check reads UNKNOWN, never a foreign archive's age.
|
||||
ownKey, err := r.storageKeyFingerprint(ctx, r.target)
|
||||
if err != nil {
|
||||
return time.Time{}, false, fmt.Errorf("reading the key fingerprint of storage %s: %w", r.target, err)
|
||||
}
|
||||
var best int64 = -1
|
||||
for _, e := range contents {
|
||||
if e.Content != "backup" || e.VMID != vmid {
|
||||
continue
|
||||
}
|
||||
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
|
||||
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this tier's last backup (R-1232)", shortFP(e.Encrypted), shortFP(ownKey)))
|
||||
continue
|
||||
}
|
||||
if ok, why := archivePlausiblyComplete(e); !ok {
|
||||
r.warnRejectedArchiveOnce(e, why)
|
||||
continue
|
||||
|
||||
@@ -128,6 +128,7 @@ type Collector struct {
|
||||
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
|
||||
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
|
||||
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
||||
opResults func() []OperatorActionResult // 2026-10-29: the operator actions' answers (nil → none)
|
||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
||||
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
|
||||
@@ -235,6 +236,12 @@ type ControllerSupervisorReporter interface {
|
||||
ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus
|
||||
}
|
||||
|
||||
// SetOperatorActionResults wires the source of the operator actions' answers (2026-10-29; nil → none sent).
|
||||
func (c *Collector) SetOperatorActionResults(f func() []OperatorActionResult) *Collector {
|
||||
c.opResults = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe).
|
||||
func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector {
|
||||
c.ctrlSup = r
|
||||
@@ -432,6 +439,9 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
if c.ctrlSup != nil {
|
||||
report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx)
|
||||
}
|
||||
if c.opResults != nil {
|
||||
report.OperatorActionResults = c.opResults()
|
||||
}
|
||||
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
|
||||
if c.guestNet != nil {
|
||||
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
|
||||
|
||||
@@ -172,6 +172,24 @@ type HostReport struct {
|
||||
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
|
||||
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
|
||||
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
|
||||
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
|
||||
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
|
||||
// when there is nothing to answer, so the golden stays byte-stable.
|
||||
OperatorActionResults []OperatorActionResult `json:"operator_action_results,omitempty"`
|
||||
}
|
||||
|
||||
// OperatorAction is ONE entry of the reply's operator_actions (hub store.OperatorActionDirective, field-exact).
|
||||
type OperatorAction struct {
|
||||
ID int64 `json:"id"`
|
||||
Action string `json:"action"`
|
||||
Arg string `json:"arg,omitempty"`
|
||||
}
|
||||
|
||||
// OperatorActionResult is one answer (hub api.hostOperatorActionResult, field-exact): outcome done | refused | failed.
|
||||
type OperatorActionResult struct {
|
||||
ID int64 `json:"id"`
|
||||
Outcome string `json:"outcome"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// ControllerSupervisorStatus is the R-523 stanza. Carries no secret.
|
||||
@@ -195,6 +213,11 @@ type ControllerSupervisorGuest struct {
|
||||
Restarts24h int `json:"restarts_24h"`
|
||||
SlowCrashloop bool `json:"slow_crashloop"`
|
||||
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
|
||||
// R-1344 (2026-10-29) — the controller RUNS but Docker's health check has read „unhealthy" on several consecutive
|
||||
// sweeps. ALARM ONLY: the agent does not restart it for this (the measurement must first show a busy night never
|
||||
// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
|
||||
Unresponsive bool `json:"unresponsive"`
|
||||
UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
|
||||
}
|
||||
|
||||
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
|
||||
@@ -631,6 +654,9 @@ type ControlEnvelope struct {
|
||||
// NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror).
|
||||
// Absent/false on an old hub → nothing happens.
|
||||
LogTailRequested bool `json:"log_tail_requested"`
|
||||
// OperatorActions (2026-10-29): the operator's pending actions for THIS host's agent — a CLOSED list on both sides
|
||||
// (`04` §6.1). Absent on an older hub.
|
||||
OperatorActions []OperatorAction `json:"operator_actions,omitempty"`
|
||||
}
|
||||
|
||||
// LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail).
|
||||
|
||||
@@ -83,6 +83,14 @@ const (
|
||||
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
|
||||
// the parked marker.
|
||||
controllerSlowCounterFile = "controller-restarts-24h.json"
|
||||
|
||||
// R-1344 (operator brief 2026-10-29, Part 2) — a controller that RUNS but does not answer. Docker's own HEALTHCHECK
|
||||
// (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) reads „unhealthy" only after 3 failed probes (~90 s);
|
||||
// controllerUnresponsiveConfirm more sweeps (3 minutes) make it an episode: logged, carried by the stanza, mailed by
|
||||
// the hub (controller_unresponsive). ALARM ONLY — it is NOT restarted for this until the measurement (R-1344) shows a
|
||||
// busy night (backups, updates) never reads unhealthy that long. Not counted during a swap, a parked guest, a locked
|
||||
// guest or a vzdump.
|
||||
controllerUnresponsiveConfirm = 6
|
||||
)
|
||||
|
||||
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
|
||||
@@ -103,6 +111,14 @@ type controllerSupState struct {
|
||||
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
|
||||
restarts24h []time.Time
|
||||
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
|
||||
|
||||
// opBrakeUntil (2026-10-29): an operator restart refused by the brake refuses every press until then (30 min).
|
||||
opBrakeUntil time.Time
|
||||
|
||||
// R-1344: consecutive running-but-unhealthy sweeps, and the current episode (since kept after it ends).
|
||||
unhealthySeen int
|
||||
unresponsive bool
|
||||
unresponsiveSince time.Time
|
||||
}
|
||||
|
||||
type controllerSupervisor struct {
|
||||
@@ -293,6 +309,7 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
||||
if running {
|
||||
st.notRunningSeen = 0
|
||||
s.ctrlSup.mu.Unlock()
|
||||
s.observeControllerHealth(ctx, vmid, parked)
|
||||
return true
|
||||
}
|
||||
st.notRunningSeen++
|
||||
@@ -392,6 +409,52 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
||||
return false
|
||||
}
|
||||
|
||||
// observeControllerHealth is R-1344's count for a RUNNING controller: Docker's health status, one read per sweep over
|
||||
// the swap's `docker inspect -f` grant. Anything but „unhealthy" — healthy, starting, no health check, a read that
|
||||
// fails — resets the count; so do a swap, a parked guest, a locked guest and a vzdump (each is someone else's moment).
|
||||
func (s *Server) observeControllerHealth(ctx context.Context, vmid int, parked bool) {
|
||||
s.swapMu.Lock()
|
||||
swapping := s.swapInFlight[vmid]
|
||||
s.swapMu.Unlock()
|
||||
unhealthy := false
|
||||
if !parked && !swapping {
|
||||
if out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer); err == nil {
|
||||
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||
unhealthy = len(f) == 2 && f[0] == "true" && f[1] == "unhealthy"
|
||||
}
|
||||
}
|
||||
if unhealthy {
|
||||
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil || lock != "" {
|
||||
unhealthy = false
|
||||
} else if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
|
||||
unhealthy = false
|
||||
}
|
||||
}
|
||||
now := s.clock()
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
if !unhealthy {
|
||||
was := st.unresponsive
|
||||
st.unhealthySeen, st.unresponsive = 0, false
|
||||
s.ctrlSup.mu.Unlock()
|
||||
if was {
|
||||
s.logger.Info("controller-supervisor: the controller answers its health check again", "vmid", vmid)
|
||||
}
|
||||
return
|
||||
}
|
||||
st.unhealthySeen++
|
||||
seen := st.unhealthySeen
|
||||
start := seen >= controllerUnresponsiveConfirm && !st.unresponsive
|
||||
if start {
|
||||
st.unresponsive, st.unresponsiveSince = true, now
|
||||
}
|
||||
s.ctrlSup.mu.Unlock()
|
||||
if start {
|
||||
s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; NOT restarting it (R-1344: alarm only until measured)",
|
||||
"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String())
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) resetNotRunning(vmid int) {
|
||||
s.ctrlSup.mu.Lock()
|
||||
defer s.ctrlSup.mu.Unlock()
|
||||
@@ -446,6 +509,10 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
|
||||
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
|
||||
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
|
||||
}
|
||||
g.Unresponsive = st.unresponsive
|
||||
if !st.unresponsiveSince.IsZero() {
|
||||
g.UnresponsiveSince = st.unresponsiveSince.UTC().Format(time.RFC3339)
|
||||
}
|
||||
out.Guests = append(out.Guests, g)
|
||||
}
|
||||
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
|
||||
|
||||
@@ -72,12 +72,56 @@ type EscrowCeremonyConfig struct {
|
||||
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
|
||||
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
|
||||
Upload func(ctx context.Context, pkg []byte) error
|
||||
// HubHolds (R-1254 H2) asks the hub whether its CURRENT package is exactly pkg — asked after a failed upload,
|
||||
// whose answer may have been lost AFTER the hub stored it. nil or an error → the outcome is unknown.
|
||||
HubHolds func(ctx context.Context, pkg []byte) (bool, error)
|
||||
}
|
||||
|
||||
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
|
||||
// ceremony incl. its upload was measured at ~2.4 s).
|
||||
const escrowUploadTimeout = 30 * time.Second
|
||||
|
||||
// escrowHoldTimeout bounds the hub question after a failed upload (R-1254 H2).
|
||||
const escrowHoldTimeout = 15 * time.Second
|
||||
|
||||
// uploadOutcome is what a claim or confirm upload did, as the HUB tells it (R-1254 H2): an upload whose answer was lost
|
||||
// or late may have been stored all the same.
|
||||
type uploadOutcome int
|
||||
|
||||
const (
|
||||
uploadStored uploadOutcome = iota // the hub holds the new package
|
||||
uploadNotStored // the hub answered and holds another package: the previous code still works
|
||||
uploadUnknown // the hub cannot be asked: nothing may be promised about either code
|
||||
)
|
||||
|
||||
// escrowUnknownOutcomeMsg is the answer when the hub cannot say which package it holds — never „the previous code
|
||||
// still works", which is false when the lost answer came after the store.
|
||||
const escrowUnknownOutcomeMsg = "the hub did not answer in time — it is not known whether it stored the new recovery package; try again (until a try succeeds, do not rely on the previous code alone)"
|
||||
|
||||
// outcomeAfterFailedUpload asks the hub whether it holds pkg after Upload returned an error. Called with the slot
|
||||
// marked claiming (escrowClaiming), outside escrowMu — it is a network call.
|
||||
func (s *Server) outcomeAfterFailedUpload(pkg []byte) uploadOutcome {
|
||||
if s.escrowCeremony.HubHolds == nil {
|
||||
return uploadUnknown
|
||||
}
|
||||
base := s.baseCtx
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(base, escrowHoldTimeout)
|
||||
defer cancel()
|
||||
held, err := s.escrowCeremony.HubHolds(ctx, pkg)
|
||||
switch {
|
||||
case err != nil:
|
||||
s.logger.Warn("local-api: escrow — the hub cannot say which package it holds after a failed upload (R-1254)", "err", err)
|
||||
return uploadUnknown
|
||||
case held:
|
||||
return uploadStored
|
||||
default:
|
||||
return uploadNotStored
|
||||
}
|
||||
}
|
||||
|
||||
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
|
||||
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
|
||||
// spike-shaped output; production is runCeremonySubprocess.
|
||||
@@ -524,15 +568,30 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
|
||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||
cancel()
|
||||
outcome := uploadStored
|
||||
if err != nil {
|
||||
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2: the answer may be lost AFTER the store
|
||||
}
|
||||
|
||||
s.escrowMu.Lock()
|
||||
s.escrowClaiming = false
|
||||
if err != nil {
|
||||
switch outcome {
|
||||
case uploadNotStored:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||
return
|
||||
case uploadUnknown:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow claim — upload failed and the hub cannot say which package it holds; code kept claimable (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: escrow claim — the upload's answer was lost, but the hub holds the new package; handing out the code (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err)
|
||||
}
|
||||
s.escrowPending = nil
|
||||
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
|
||||
@@ -631,15 +690,30 @@ func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Requ
|
||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||
cancel()
|
||||
outcome := uploadStored
|
||||
if err != nil {
|
||||
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2
|
||||
}
|
||||
|
||||
s.escrowMu.Lock()
|
||||
s.escrowClaiming = false
|
||||
if err != nil {
|
||||
switch outcome {
|
||||
case uploadNotStored:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||
return
|
||||
case uploadUnknown:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow confirm — upload failed and the hub cannot say which package it holds; still waiting (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: escrow confirm — the upload's answer was lost, but the hub holds the new package (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err)
|
||||
}
|
||||
s.escrowPending = nil
|
||||
s.wipeEscrowConfirmLocked()
|
||||
|
||||
@@ -41,9 +41,13 @@ type fakeEscrowHub struct {
|
||||
mu sync.Mutex
|
||||
current []byte
|
||||
uploads int
|
||||
err error // non-nil → the upload fails, nothing stored
|
||||
gate chan struct{} // non-nil → the upload blocks until closed
|
||||
entered chan struct{} // non-nil → closed when an upload starts
|
||||
err error // non-nil → the upload fails, nothing stored
|
||||
// R-1254 H2: lost → the upload STORES the package and then fails (the answer is lost on the way back);
|
||||
// holdsErr → the hub cannot be asked which package it holds.
|
||||
lost bool
|
||||
holdsErr error
|
||||
gate chan struct{} // non-nil → the upload blocks until closed
|
||||
entered chan struct{} // non-nil → closed when an upload starts
|
||||
}
|
||||
|
||||
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
||||
@@ -61,9 +65,23 @@ func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
||||
}
|
||||
f.current = append([]byte(nil), pkg...)
|
||||
f.uploads++
|
||||
if f.lost {
|
||||
return context.DeadlineExceeded
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// holds answers „is the hub's current package exactly pkg?" — the R-1254 question (production compares the identity
|
||||
// blob the hub serves back).
|
||||
func (f *fakeEscrowHub) holds(_ context.Context, pkg []byte) (bool, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.holdsErr != nil {
|
||||
return false, f.holdsErr
|
||||
}
|
||||
return string(f.current) == string(pkg), nil
|
||||
}
|
||||
|
||||
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
|
||||
func (f *fakeEscrowHub) opensWith(r string) bool {
|
||||
f.mu.Lock()
|
||||
@@ -100,6 +118,7 @@ func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub
|
||||
HubConfigured: true,
|
||||
DRConfigured: func() bool { return true },
|
||||
Upload: hub.upload,
|
||||
HubHolds: hub.holds,
|
||||
}
|
||||
srv.ceremonyRun = run
|
||||
srv.escrowSudoCheck = func(context.Context) error { return nil }
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1254 (the ceremony half, review round 2 HUNT-3 H2): the upload's answer can be lost or late (30 s) AFTER the hub
|
||||
// stored the new package. The household was then told „the previous code still works" — false: the hub now holds the
|
||||
// package the NEW code opens. The agent asks the hub which package it holds before it words the result:
|
||||
// - the hub holds the new package → success (the code is handed out / the confirm completes);
|
||||
// - the hub holds another package → the old sentence, now true;
|
||||
// - the hub cannot be asked → 504 and a sentence that does NOT promise the previous code; the wait goes on.
|
||||
|
||||
// RED-PROOF: remove the hub question from the claim's failure branch → 502 „the previous code still works" while the
|
||||
// hub holds the new package → FAILS.
|
||||
func TestR1254_ALostAnswerAtTheClaimIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
code, body := claimCode(t, h)
|
||||
if !hub.opensWith(firstR) {
|
||||
t.Fatal("fixture: the hub must hold the new package (the answer was lost after the store)")
|
||||
}
|
||||
if code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||
t.Fatalf("claim after a lost answer, the hub holding the new package: %d %s — want 200 with the code", code, body)
|
||||
}
|
||||
if strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("the household was told the previous code still works: %s", body)
|
||||
}
|
||||
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
|
||||
t.Fatalf("the job must read uploaded: %s", st)
|
||||
}
|
||||
}
|
||||
|
||||
// The hub cannot be asked: no promise about the previous code, 504, the code stays claimable (a retry decides).
|
||||
// RED-PROOF: as above → 502 with „the previous code still works" → FAILS.
|
||||
func TestR1254_AnUnknownOutcomeAtTheClaimPromisesNothing(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true, holdsErr: errors.New("hub unreachable")}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
code, body := claimCode(t, h)
|
||||
if code != http.StatusGatewayTimeout {
|
||||
t.Fatalf("claim with an unknown outcome: %d %s — want 504", code, body)
|
||||
}
|
||||
if strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("an unknown outcome promised the previous code: %s", body)
|
||||
}
|
||||
assertNoRIn(t, "unknown-outcome claim payload", body, firstR)
|
||||
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) {
|
||||
t.Fatalf("the code must stay claimable: %s", st)
|
||||
}
|
||||
// The hub answers again: the retry finds the new package already stored and hands the code out.
|
||||
hub.mu.Lock()
|
||||
hub.holdsErr = nil
|
||||
hub.mu.Unlock()
|
||||
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||
t.Fatalf("retry: %d %s", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The hub holds ANOTHER package (the upload really failed): the old sentence stays, and is now true.
|
||||
func TestR1254_AFailedUploadTheHubDidNotStoreKeepsTheOldSentence(t *testing.T) {
|
||||
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
if code, body := claimCode(t, h); code != http.StatusBadGateway || !strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("a failed upload: %d %s — want 502 and the previous code named", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The same at the typed-back confirm (R-1151).
|
||||
// RED-PROOF: remove the hub question from the confirm's failure branch → 502 → FAILS.
|
||||
func TestR1254_ALostAnswerAtTheConfirmIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
c := claimDeferred(t, h)
|
||||
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
|
||||
t.Fatalf("confirm after a lost answer: %d (opens: %v) — want 200", code, hub.opensWith(secondR))
|
||||
}
|
||||
hub.mu.Lock()
|
||||
hub.lost, hub.holdsErr = true, errors.New("hub unreachable")
|
||||
hub.mu.Unlock()
|
||||
srv2, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h2 := srv2.Handler()
|
||||
startAndWait(t, srv2, h2)
|
||||
c2 := claimDeferred(t, h2)
|
||||
if code := confirm(t, h2, c2.typedBack(t)); code != http.StatusGatewayTimeout {
|
||||
t.Fatalf("confirm with an unknown outcome: %d — want 504", code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// R-528 — the OOM signal from the host side.
|
||||
//
|
||||
// MEASURED 2026-09-15 (scratch 9202, Docker 29.8.0) and 2026-09-16 (a fresh install in a nested VM): an app killed by
|
||||
// its memory limit inside a Felhom LXC guest left Docker's State.OOMKilled false and sent no `oom` event, so the
|
||||
// controller's OOM line (R-514) could not fire. The kernel's own counter does count every kill, and the host can read
|
||||
// it: /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events, line `oom_kill N` (the
|
||||
// systemd cgroup driver; `ns/docker/<id>/` under the cgroupfs driver). Read on Tester 1's host 2026-10-28: the files
|
||||
// are mode 0444, readable by the non-root felhom-agent user — no sudo, no new root path.
|
||||
//
|
||||
// A container's scope is new at each start, so the counter is per container RUN: a count above 0 means this run had a
|
||||
// kill. The controller asks every 30 s and raises its existing app_oom line (once per run) for any such container.
|
||||
|
||||
// readGuestOOMKills returns container id → oom_kill for one guest. available=false (no error) when the guest has no
|
||||
// cgroup folder (a VM guest, or a container that is not running).
|
||||
func readGuestOOMKills(root string, vmid int) (map[string]int64, bool, error) {
|
||||
if root == "" {
|
||||
root = "/sys/fs/cgroup"
|
||||
}
|
||||
base := filepath.Join(root, "lxc", strconv.Itoa(vmid), "ns")
|
||||
if fi, err := os.Stat(base); err != nil || !fi.IsDir() {
|
||||
return nil, false, nil
|
||||
}
|
||||
out := map[string]int64{}
|
||||
for _, pattern := range []string{
|
||||
filepath.Join(base, "system.slice", "docker-*.scope", "memory.events"),
|
||||
filepath.Join(base, "docker", "*", "memory.events"),
|
||||
} {
|
||||
files, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil, true, err
|
||||
}
|
||||
for _, f := range files {
|
||||
id := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(filepath.Dir(f)), "docker-"), ".scope")
|
||||
if !isContainerID(id) {
|
||||
continue
|
||||
}
|
||||
n, err := readOOMKill(f)
|
||||
if err != nil {
|
||||
continue // a container that stopped between the glob and the read
|
||||
}
|
||||
out[id] = n
|
||||
}
|
||||
}
|
||||
return out, true, nil
|
||||
}
|
||||
|
||||
func isContainerID(s string) bool {
|
||||
if len(s) != 64 {
|
||||
return false
|
||||
}
|
||||
for _, c := range s {
|
||||
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func readOOMKill(path string) (int64, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
k, v, ok := strings.Cut(strings.TrimSpace(sc.Text()), " ")
|
||||
if ok && k == "oom_kill" {
|
||||
return strconv.ParseInt(strings.TrimSpace(v), 10, 64)
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return 0, fmt.Errorf("no oom_kill line in %s", path)
|
||||
}
|
||||
|
||||
// handleGuestOOMKills is GET /guest/oom-kills: {"vmid", "available", "containers": {id: oom_kill}}.
|
||||
func (s *Server) handleGuestOOMKills(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
kills, avail, err := readGuestOOMKills(s.cgroupRoot, vmid)
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: guest oom-kills read failed (R-528)", "vmid", vmid, "err", err)
|
||||
writeErr(w, http.StatusInternalServerError, "the guest's OOM counters could not be read")
|
||||
return
|
||||
}
|
||||
if kills == nil {
|
||||
kills = map[string]int64{}
|
||||
}
|
||||
writeOK(w, map[string]any{"vmid": vmid, "available": avail, "containers": kills})
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-528: Docker does not always report an OOM kill inside a Felhom LXC guest (OOMKilled stays false, no `oom` event —
|
||||
// measured 2026-09-15/16 on two boxes), but the kernel's own counter does count it, and it is readable from the host:
|
||||
// /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events, line `oom_kill N`. GET /guest/oom-kills
|
||||
// serves the asking guest's counters, and only that guest's.
|
||||
|
||||
func fakeCgroup(t *testing.T, root string, vmid, id, events string) {
|
||||
t.Helper()
|
||||
dir := filepath.Join(root, "lxc", vmid, "ns", "system.slice", "docker-"+id+".scope")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o444); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
const idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
const idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||
const idOther = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
|
||||
|
||||
// RED-PROOF: serve the counters of every guest under lxc/ (drop the vmid from the glob) → the other guest's container
|
||||
// appears → FAILS; parse `oom ` instead of `oom_kill` → idA reads 9 → FAILS.
|
||||
func TestR528_GuestOOMKillsServesThisGuestsCountersOnly(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
fakeCgroup(t, root, "100", idA, "low 0\nhigh 0\nmax 4\noom 9\noom_kill 3\noom_group_kill 0\n")
|
||||
fakeCgroup(t, root, "100", idB, "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\n")
|
||||
fakeCgroup(t, root, "200", idOther, "oom_kill 7\n")
|
||||
// a scope that is not a container (no 64-hex id) and a cgroupfs-driver layout container
|
||||
if err := os.MkdirAll(filepath.Join(root, "lxc", "100", "ns", "system.slice", "cron.service"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dfDir := filepath.Join(root, "lxc", "100", "ns", "docker", idOther[:8]+strings.Repeat("d", 56))
|
||||
if err := os.MkdirAll(dfDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dfDir, "memory.events"), []byte("oom_kill 2\n"), 0o444)
|
||||
|
||||
got, avail, err := readGuestOOMKills(root, 100)
|
||||
if err != nil || !avail {
|
||||
t.Fatalf("read: avail=%v err=%v", avail, err)
|
||||
}
|
||||
want := map[string]int64{idA: 3, idB: 0, idOther[:8] + strings.Repeat("d", 56): 2}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
for k, v := range want {
|
||||
if got[k] != v {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
if _, leaked := got[idOther]; leaked {
|
||||
t.Fatal("another guest's container leaked into this guest's answer")
|
||||
}
|
||||
// A guest with no cgroup folder (a VM, or not running): not available, no error.
|
||||
if _, avail, err := readGuestOOMKills(root, 300); avail || err != nil {
|
||||
t.Fatalf("a guest without a cgroup folder: avail=%v err=%v", avail, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Through the route, with the guest's own token.
|
||||
func TestR528_TheRouteAnswersTheTokensGuest(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
fakeCgroup(t, root, "8200", idA, "oom_kill 5\n")
|
||||
fakeCgroup(t, root, "9300", idB, "oom_kill 1\n") // token B's guest: never in A's answer
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
srv.cgroupRoot = root
|
||||
w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "A", "")
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("route: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
var env struct {
|
||||
Data struct {
|
||||
VMID int `json:"vmid"`
|
||||
Available bool `json:"available"`
|
||||
Containers map[string]int64 `json:"containers"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if env.Data.VMID != 8200 || !env.Data.Available || len(env.Data.Containers) != 1 || env.Data.Containers[idA] != 5 {
|
||||
t.Fatalf("answer %s — want guest 8200's one container with 5", w.Body.String())
|
||||
}
|
||||
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("a cross-guest ask: %d, want 403", w.Code)
|
||||
}
|
||||
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "", ""); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: %d", w.Code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// The operator's actions for THIS agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).
|
||||
//
|
||||
// THE DOOR IS THE HOST REPORT'S REPLY, the controller's shape: the operator presses a button on the hub's host page, the
|
||||
// hub stores a row and pokes the box (the poke's out-of-cycle report gets the reply in seconds; without the tunnel the
|
||||
// 15-minute cycle carries it), the reply lists `operator_actions: [{id, action, arg}]`, this agent acts ONCE per id, and
|
||||
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
|
||||
// The result is re-sent until the reply stops listing the id.
|
||||
//
|
||||
// THE LIST IS CLOSED: one verb, restart_controller <vmid>. Anything else — a verb, a vmid that does not parse, a guest
|
||||
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
|
||||
//
|
||||
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
|
||||
// the guest over the existing grants, with the supervisor's guards — not during a controller swap, not parked, not on a
|
||||
// stopped, locked or vzdump-busy guest, not when the guest list or the vzdump state cannot be read — and the thrash brake
|
||||
// counts these restarts too: 3 restarts (the supervisor's or the operator's) inside 15 minutes refuse the press, and
|
||||
// presses stay refused for 30 minutes. It works with the controller stopped or stuck: the agent does it, not the
|
||||
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
|
||||
|
||||
const (
|
||||
// OpRestartController is the agent's one operator verb.
|
||||
OpRestartController = "restart_controller"
|
||||
|
||||
opOutcomeDone = "done"
|
||||
opOutcomeRefused = "refused"
|
||||
opOutcomeFailed = "failed"
|
||||
|
||||
// opBrakeRefusal is how long presses stay refused after the brake refused one.
|
||||
opBrakeRefusal = 30 * time.Minute
|
||||
// opForgetAfter drops a started id the reply no longer lists.
|
||||
opForgetAfter = 24 * time.Hour
|
||||
|
||||
defaultOpHealthPoll = 2 * time.Second
|
||||
// defaultOpHealthWait covers the bootstrap's pull-less rm/run (seconds), Docker's start period (10 s) and its first
|
||||
// probe (30 s interval) with room: the controller's own HEALTHCHECK is the answer.
|
||||
defaultOpHealthWait = 3 * time.Minute
|
||||
// opHealthTemplate reads running + health over the swap's `docker inspect -f` grant.
|
||||
opHealthTemplate = "{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}"
|
||||
)
|
||||
|
||||
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
|
||||
var operatorActionNames = []string{OpRestartController}
|
||||
|
||||
// OperatorActionNames returns the closed list (sorted copy).
|
||||
func OperatorActionNames() []string {
|
||||
out := append([]string(nil), operatorActionNames...)
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
type opActionState struct {
|
||||
mu sync.Mutex
|
||||
started map[int64]time.Time
|
||||
results map[int64]hub.OperatorActionResult
|
||||
}
|
||||
|
||||
// SetOperatorActionTrigger wires the report loop's out-of-band trigger (the result goes at once, not in 15 minutes).
|
||||
func (s *Server) SetOperatorActionTrigger(f func()) { s.opTrigger = f }
|
||||
|
||||
// OnEnvelope is the report loop's hook (hub.EnvelopeObserver): it starts each newly listed action once and forgets
|
||||
// the ids the reply no longer lists (the hub has their results).
|
||||
func (s *Server) OnEnvelope(_ context.Context, env *hub.ControlEnvelope) {
|
||||
if env == nil {
|
||||
return
|
||||
}
|
||||
listed := map[int64]bool{}
|
||||
var start []hub.OperatorAction
|
||||
now := time.Now()
|
||||
s.opActs.mu.Lock()
|
||||
if s.opActs.started == nil {
|
||||
s.opActs.started = map[int64]time.Time{}
|
||||
s.opActs.results = map[int64]hub.OperatorActionResult{}
|
||||
}
|
||||
for _, a := range env.OperatorActions {
|
||||
listed[a.ID] = true
|
||||
if _, ok := s.opActs.started[a.ID]; ok {
|
||||
continue
|
||||
}
|
||||
s.opActs.started[a.ID] = now
|
||||
start = append(start, a)
|
||||
}
|
||||
for id, at := range s.opActs.started {
|
||||
if listed[id] {
|
||||
continue
|
||||
}
|
||||
if _, answered := s.opActs.results[id]; answered || now.Sub(at) > opForgetAfter {
|
||||
delete(s.opActs.started, id)
|
||||
delete(s.opActs.results, id)
|
||||
}
|
||||
}
|
||||
s.opActs.mu.Unlock()
|
||||
for _, a := range start {
|
||||
go s.runOperatorAction(a) // a restart waits for the health answer — never block the report loop
|
||||
}
|
||||
}
|
||||
|
||||
// OperatorActionResults is the report's answer list, oldest id first.
|
||||
func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
|
||||
s.opActs.mu.Lock()
|
||||
defer s.opActs.mu.Unlock()
|
||||
out := make([]hub.OperatorActionResult, 0, len(s.opActs.results))
|
||||
for _, r := range s.opActs.results {
|
||||
out = append(out, r)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *Server) runOperatorAction(a hub.OperatorAction) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), s.opWait()+time.Minute)
|
||||
defer cancel()
|
||||
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller only", a.Action)
|
||||
if a.Action == OpRestartController {
|
||||
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
|
||||
}
|
||||
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
|
||||
"outcome", outcome, "message", msg)
|
||||
s.opActs.mu.Lock()
|
||||
if _, still := s.opActs.started[a.ID]; still {
|
||||
s.opActs.results[a.ID] = hub.OperatorActionResult{ID: a.ID, Outcome: outcome, Message: msg}
|
||||
}
|
||||
s.opActs.mu.Unlock()
|
||||
if s.opTrigger != nil {
|
||||
s.opTrigger()
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) opPoll() time.Duration {
|
||||
if s.opHealthPoll > 0 {
|
||||
return s.opHealthPoll
|
||||
}
|
||||
return defaultOpHealthPoll
|
||||
}
|
||||
|
||||
func (s *Server) opWait() time.Duration {
|
||||
if s.opHealthWait > 0 {
|
||||
return s.opHealthWait
|
||||
}
|
||||
return defaultOpHealthWait
|
||||
}
|
||||
|
||||
// operatorRestartController is the press: the supervisor's guards, the brake, the restart, then the wait for the
|
||||
// controller's healthy answer. Returns the outcome and the sentence the host page shows.
|
||||
func (s *Server) operatorRestartController(ctx context.Context, id int64, arg string) (string, string) {
|
||||
vmid, err := strconv.Atoi(arg)
|
||||
if err != nil || strconv.Itoa(vmid) != arg {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
|
||||
}
|
||||
if s.staleLock == nil || s.guestExec == nil {
|
||||
return opOutcomeRefused, "refused: the controller supervisor is not wired on this agent"
|
||||
}
|
||||
if !s.provisionedGuest(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
|
||||
}
|
||||
status := ""
|
||||
for _, g := range guests {
|
||||
if g.VMID == vmid {
|
||||
status = g.Status
|
||||
}
|
||||
}
|
||||
if status != "running" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
|
||||
}
|
||||
if s.controllerParked(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
|
||||
}
|
||||
s.swapMu.Lock()
|
||||
swapping := s.swapInFlight[vmid]
|
||||
s.swapMu.Unlock()
|
||||
if swapping {
|
||||
return opOutcomeRefused, "refused: a controller swap (update) is in flight — it owns the controller"
|
||||
}
|
||||
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
|
||||
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
|
||||
} else if lock != "" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
|
||||
}
|
||||
if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
|
||||
return opOutcomeRefused, "refused: a whole-guest backup (vzdump) may be in flight for the guest"
|
||||
}
|
||||
|
||||
// The brake: every restart counts — the supervisor's and the operator's.
|
||||
now := s.clock()
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
if now.Before(st.opBrakeUntil) {
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
|
||||
if len(st.restarts) >= controllerCrashloopMax {
|
||||
st.opBrakeUntil = now.Add(opBrakeRefusal)
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
reason := fmt.Sprintf("the operator via the hub (action #%d)", id)
|
||||
st.restarts = append(st.restarts, now)
|
||||
st.restartsTotal++
|
||||
st.lastRestartAt = now
|
||||
st.lastReason = reason
|
||||
st.notRunningSeen = 0
|
||||
s.ctrlSup.mu.Unlock()
|
||||
|
||||
s.logger.Warn("controller-supervisor: RESTARTING the controller on the operator's press", "vmid", vmid, "action_id", id, "unit", bootstrapUnit)
|
||||
began := time.Now()
|
||||
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
||||
return opOutcomeFailed, fmt.Sprintf("the restart of guest %d's controller failed: %v", vmid, err)
|
||||
}
|
||||
for time.Since(began) < s.opWait() {
|
||||
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
|
||||
if err == nil {
|
||||
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
|
||||
secs := int(time.Since(began).Round(time.Second) / time.Second)
|
||||
return opOutcomeDone, fmt.Sprintf("restarted; the controller answered after %d s (Docker's health check: healthy)", secs)
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return opOutcomeFailed, "restarted, but the wait for its answer was cut"
|
||||
case <-time.After(s.opPoll()):
|
||||
}
|
||||
}
|
||||
return opOutcomeFailed, fmt.Sprintf("restarted, but no healthy answer within %s — look at the box", s.opWait().Round(time.Second))
|
||||
}
|
||||
|
||||
func orDash(s string) string {
|
||||
if s == "" {
|
||||
return "not listed"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// Operator brief 2026-10-29, Part 1: „Restart controller" from the hub. The agent's closed list holds one verb; the act
|
||||
// is the supervisor's own restart with the supervisor's guards, and the brake counts operator restarts too.
|
||||
|
||||
// opExec answers the supervisor's fake plus the health read the operator restart waits on.
|
||||
type opExec struct {
|
||||
*supExec
|
||||
health string // what {{.State.Health.Status}} reads after the restart
|
||||
}
|
||||
|
||||
func (f *opExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
|
||||
if len(args) >= 5 && args[0] == "docker" && args[1] == "inspect" && args[3] == opHealthTemplate {
|
||||
f.mu.Lock()
|
||||
st := f.status[vmid]
|
||||
f.mu.Unlock()
|
||||
if st == "" {
|
||||
return "", errors.New("pct exec: exit status 1: Error: No such object: felhom-controller")
|
||||
}
|
||||
return map[bool]string{true: "true", false: "false"}[st == "running"] + "|" + f.health + "\n", nil
|
||||
}
|
||||
return f.supExec.GuestExec(ctx, vmid, args...)
|
||||
}
|
||||
|
||||
func opServer(t *testing.T, status string) (*Server, *opExec, *fakeGuestPowerCtl, *supClock, string, *atomic.Int32) {
|
||||
t.Helper()
|
||||
ex := &opExec{supExec: &supExec{status: map[int]string{9201: status}, onRestart: "running"}, health: "healthy"}
|
||||
ctl := runningGuest(9201)
|
||||
s, clk, dir := supServer(t, nil, ctl, 9201)
|
||||
s.guestExec = ex
|
||||
s.opHealthPoll = time.Millisecond
|
||||
s.opHealthWait = 2 * time.Second
|
||||
var triggered atomic.Int32
|
||||
s.SetOperatorActionTrigger(func() { triggered.Add(1) })
|
||||
return s, ex, ctl, clk, dir, &triggered
|
||||
}
|
||||
|
||||
func env(acts ...hub.OperatorAction) *hub.ControlEnvelope {
|
||||
return &hub.ControlEnvelope{OperatorActions: acts}
|
||||
}
|
||||
|
||||
// waitResult waits for the action's result (the act runs in the background).
|
||||
func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
for _, r := range s.OperatorActionResults() {
|
||||
if r.ID == id {
|
||||
return r
|
||||
}
|
||||
}
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("no result for #%d", id)
|
||||
return hub.OperatorActionResult{}
|
||||
}
|
||||
|
||||
// RED-PROOF: seen failing before the list existed (build failure, then the names).
|
||||
func TestOpActions_AgentClosedList(t *testing.T) {
|
||||
if got, want := OperatorActionNames(), []string{"restart_controller"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Anything outside the list — and a bad argument — is refused and NOTHING is called.
|
||||
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "running")
|
||||
s.OnEnvelope(context.Background(), env(
|
||||
hub.OperatorAction{ID: 1, Action: "reboot_guest", Arg: "9201"},
|
||||
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
|
||||
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
|
||||
))
|
||||
for _, id := range []int64{1, 2, 3} {
|
||||
if r := waitResult(t, s, id); r.Outcome != "refused" {
|
||||
t.Errorf("#%d: %+v, want refused", id, r)
|
||||
}
|
||||
}
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("a refused action restarted the controller (%d)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The press restarts ONCE, says how long the controller took to answer, asks for a report at once, and re-sends the
|
||||
// result until the hub stops listing the id. RED-PROOF: drop the GuestExec restart → restarts 0 → FAILS.
|
||||
func TestOpActions_RestartReportsTheAnswerTime(t *testing.T) {
|
||||
s, ex, _, _, _, trig := opServer(t, "running")
|
||||
a := hub.OperatorAction{ID: 7, Action: "restart_controller", Arg: "9201"}
|
||||
s.OnEnvelope(context.Background(), env(a))
|
||||
r := waitResult(t, s, 7)
|
||||
if r.Outcome != "done" || !strings.Contains(r.Message, "answered after") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("restarts = %d, want 1", n)
|
||||
}
|
||||
if trig.Load() == 0 {
|
||||
t.Fatal("the agent did not ask for a report — the result would wait 15 minutes")
|
||||
}
|
||||
s.OnEnvelope(context.Background(), env(a)) // the hub still lists it: re-send, never act again
|
||||
time.Sleep(100 * time.Millisecond) // a wrong second act runs in the background — give it time to show
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("a re-listed id acted again (%d)", n)
|
||||
}
|
||||
if len(s.OperatorActionResults()) != 1 {
|
||||
t.Fatal("the result must be re-sent while listed")
|
||||
}
|
||||
s.OnEnvelope(context.Background(), env()) // the hub recorded it
|
||||
if left := s.OperatorActionResults(); len(left) != 0 {
|
||||
t.Fatalf("a recorded result is still sent: %+v", left)
|
||||
}
|
||||
}
|
||||
|
||||
// The point of the agent doing it: a STOPPED controller is restarted too (the controller cannot restart itself).
|
||||
func TestOpActions_WorksWithTheControllerStopped(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "exited")
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 8, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 8); r.Outcome != "done" || ex.count(9201) != 1 {
|
||||
t.Fatalf("result %+v, restarts %d", r, ex.count(9201))
|
||||
}
|
||||
}
|
||||
|
||||
// Each supervisor guard refuses the press, with the reason. RED-PROOF: drop the lock check → the locked case
|
||||
// restarts → FAILS (each guard seen failing by its own mutation).
|
||||
func TestOpActions_GuardsRefuse(t *testing.T) {
|
||||
cases := []struct {
|
||||
name, want string
|
||||
setup func(s *Server, ctl *fakeGuestPowerCtl, dir string)
|
||||
}{
|
||||
{"parked", "parked", func(_ *Server, _ *fakeGuestPowerCtl, dir string) {
|
||||
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}},
|
||||
{"swap", "swap", func(s *Server, _ *fakeGuestPowerCtl, _ string) { s.swapInFlight[9201] = true }},
|
||||
{"locked", "locked", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.locks[9201] = "backup" }},
|
||||
{"vzdump", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupRun = map[int]bool{9201: true} }},
|
||||
{"vzdump unknown", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupErr = errors.New("tasks unreadable") }},
|
||||
{"guest stopped", "not running", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guests[0].Status = "stopped" }},
|
||||
{"guest list", "guest list", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guestsErr = errors.New("api down") }},
|
||||
}
|
||||
for i, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, ctl, _, dir, _ := opServer(t, "running")
|
||||
tc.setup(s, ctl, dir)
|
||||
id := int64(100 + i)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||
r := waitResult(t, s, id)
|
||||
if r.Outcome != "refused" || !strings.Contains(strings.ToLower(r.Message), tc.want) {
|
||||
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
|
||||
}
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("guard %q did not hold: restarted %d", tc.name, n)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The thrash brake counts operator restarts AND the supervisor's: 3 in 15 minutes → the next press is refused, and
|
||||
// presses stay refused for 30 minutes. RED-PROOF: drop the brake → the 4th press restarts → FAILS.
|
||||
func TestOpActions_TheBrakeCountsEveryRestart(t *testing.T) {
|
||||
s, ex, _, clk, _, _ := opServer(t, "running")
|
||||
// one restart by the supervisor itself
|
||||
ex.mu.Lock()
|
||||
ex.status[9201] = "exited"
|
||||
ex.mu.Unlock()
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
if ex.count(9201) != 1 {
|
||||
t.Fatalf("setup: the supervisor did not restart (%d)", ex.count(9201))
|
||||
}
|
||||
for id := int64(1); id <= 2; id++ {
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, id); r.Outcome != "done" {
|
||||
t.Fatalf("press %d: %+v", id, r)
|
||||
}
|
||||
}
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9201"}))
|
||||
r := waitResult(t, s, 3)
|
||||
if r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
|
||||
t.Fatalf("the 4th restart in 15 minutes: %+v", r)
|
||||
}
|
||||
if n := ex.count(9201); n != 3 {
|
||||
t.Fatalf("restarts = %d, want 3", n)
|
||||
}
|
||||
clk.t = clk.t.Add(20 * time.Minute) // the window has passed, but the 30-minute refusal has not
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 4); r.Outcome != "refused" {
|
||||
t.Fatalf("inside the 30-minute refusal: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(11 * time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 5, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 5); r.Outcome != "done" {
|
||||
t.Fatalf("after the refusal: %+v", r)
|
||||
}
|
||||
}
|
||||
|
||||
// A controller that does not come back healthy is a FAILED press, said plainly.
|
||||
func TestOpActions_NoHealthyAnswerIsFailed(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "running")
|
||||
ex.health = "unhealthy"
|
||||
s.opHealthWait = 30 * time.Millisecond
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 9, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 9); r.Outcome != "failed" || !strings.Contains(r.Message, "no healthy answer") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
}
|
||||
@@ -411,6 +411,13 @@ type Server struct {
|
||||
guestsDir string
|
||||
ctrlSup controllerSupervisor
|
||||
|
||||
// 2026-10-29: the operator's actions from the hub's reply (opactions.go). opHealthPoll/opHealthWait bound the wait
|
||||
// for a restarted controller's healthy answer (tests shorten them; zero → the defaults).
|
||||
opActs opActionState
|
||||
opTrigger func()
|
||||
opHealthPoll time.Duration
|
||||
opHealthWait time.Duration
|
||||
|
||||
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
|
||||
// detached pipeline runs through (tests inject; production defaults set in NewServer).
|
||||
netVerifyMu sync.Mutex
|
||||
@@ -459,6 +466,8 @@ type Server struct {
|
||||
escrowLookPath func(file string) (string, error)
|
||||
// statFile reports whether a path exists (preflight's staged-secret item; tests inject).
|
||||
statFile func(path string) bool
|
||||
// cgroupRoot is where the host's cgroup tree is mounted (R-528, GET /guest/oom-kills); "" = /sys/fs/cgroup. Tests set it.
|
||||
cgroupRoot string
|
||||
|
||||
baseCtx context.Context // for fire-and-forget backups; set in Run
|
||||
}
|
||||
@@ -576,6 +585,9 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
|
||||
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
|
||||
// read it. Read-only; never measured (or the job not wired) = present:false.
|
||||
// R-528: the kernel's OOM-kill counter of each of the guest's containers, read on the host (Docker's own flag and
|
||||
// event can stay silent inside an LXC guest). Read-only; the asking guest's containers only.
|
||||
mux.HandleFunc("GET /guest/oom-kills", s.withGuest(s.handleGuestOOMKills))
|
||||
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
|
||||
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
|
||||
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
||||
|
||||
@@ -256,8 +256,10 @@ func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
if callers != 5 {
|
||||
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
|
||||
// 2026-10-29: +2 deliberate callers — the operator's restart_controller (opactions.go) and R-1344's unresponsive count
|
||||
// (controllersupervisor.go observeControllerHealth). Both through s.staleLock, the one helper.
|
||||
if callers != 7 {
|
||||
t.Errorf("expected the seven known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup, the operator restart, the unresponsive count), found %d", callers)
|
||||
}
|
||||
b, _ := os.ReadFile("stalelock.go")
|
||||
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1344 (operator brief 2026-10-29, Part 2): a controller that RUNS but whose Docker health check reads „unhealthy" on
|
||||
// controllerUnresponsiveConfirm consecutive sweeps is reported unresponsive (the stanza; the hub raises
|
||||
// controller_unresponsive). ALARM ONLY: it is never restarted for it until the measurement licenses that.
|
||||
// RED-PROOF: seen failing before the stanza carried it.
|
||||
|
||||
func unrespServer(t *testing.T, health string) (*Server, *opExec, *fakeGuestPowerCtl) {
|
||||
t.Helper()
|
||||
s, ex, ctl, _, _, _ := opServer(t, "running")
|
||||
ex.health = health
|
||||
return s, ex, ctl
|
||||
}
|
||||
|
||||
func sweep(s *Server, n int) {
|
||||
for i := 0; i < n; i++ {
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSupervisor_UnresponsiveIsAlarmedNeverRestarted(t *testing.T) {
|
||||
s, ex, _ := unrespServer(t, "unhealthy")
|
||||
sweep(s, controllerUnresponsiveConfirm-1)
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) != 1 || g[0].Unresponsive {
|
||||
t.Fatalf("unresponsive before %d sweeps: %+v", controllerUnresponsiveConfirm, g)
|
||||
}
|
||||
sweep(s, 1)
|
||||
g := s.ControllerSupervisorStatus(context.Background()).Guests
|
||||
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveSince == "" {
|
||||
t.Fatalf("after %d unhealthy sweeps: %+v", controllerUnresponsiveConfirm, g)
|
||||
}
|
||||
since := g[0].UnresponsiveSince
|
||||
sweep(s, 10)
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("an unresponsive controller was RESTARTED (%d) — the restart waits for the measurement", n)
|
||||
}
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].UnresponsiveSince != since {
|
||||
t.Fatal("unresponsive_since moved while it stayed unresponsive — the hub would mail again")
|
||||
}
|
||||
// it answers again
|
||||
ex.health = "healthy"
|
||||
sweep(s, 1)
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].Unresponsive || g[0].UnresponsiveSince != since {
|
||||
t.Fatalf("after it answered: %+v (unresponsive must clear; since stays as the record)", g[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Slow but answering (healthy / starting), a swap, a vzdump or a parked guest never counts.
|
||||
func TestSupervisor_UnresponsiveGuards(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
setup func(s *Server, ex *opExec, ctl *fakeGuestPowerCtl)
|
||||
}{
|
||||
{"healthy", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
|
||||
{"starting", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
|
||||
{"swap", func(s *Server, _ *opExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
|
||||
{"vzdump", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
|
||||
{"locked", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, ctl := unrespServer(t, "unhealthy")
|
||||
tc.setup(s, ex, ctl)
|
||||
sweep(s, controllerUnresponsiveConfirm+2)
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) == 1 && g[0].Unresponsive {
|
||||
t.Fatalf("%s counted as unresponsive: %+v", tc.name, g[0])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -126,6 +126,19 @@ func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params jso
|
||||
// appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run
|
||||
// itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal:
|
||||
// the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it.
|
||||
// PolicyPassAfterUpdate runs the wrapper's bundle-policy pass once after a self-update commits (R-1285). In the normal
|
||||
// delivery the box gets agent_update and agent_config_update in ONE pass: the bundle job runs in the OLD process, which
|
||||
// has no policy pass, and the restart into the new agent comes a second later (measured 2026-10-26 on demo-hp). By the
|
||||
// commit the wrapper on disk is the bundle's own, so the pass removes our appliance-only files from a BYO host in the
|
||||
// SAME update. Idempotent; on an appliance it removes nothing.
|
||||
func (e ConfigUpdateExecutor) PolicyPassAfterUpdate(ctx context.Context, lg *slog.Logger, ver string) {
|
||||
if lg == nil {
|
||||
lg = slog.Default()
|
||||
}
|
||||
lg.Info("osupdate: running the bundle policy pass after the self-update commit (R-1285)", "agent_version", ver)
|
||||
e.policyPass(ctx, lg, ver)
|
||||
}
|
||||
|
||||
func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) {
|
||||
runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z")
|
||||
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"})
|
||||
|
||||
@@ -18,8 +18,14 @@ type Manager struct {
|
||||
logger *slog.Logger
|
||||
now func() time.Time // injectable for tests
|
||||
sleep func(context.Context, time.Duration)
|
||||
// onCommitted runs once, right after a successful commit (R-1285: the bundle policy pass of the NEW agent — in the
|
||||
// normal two-job delivery the bundle was installed by the OLD process, which has no such pass). nil = none.
|
||||
onCommitted func(context.Context)
|
||||
}
|
||||
|
||||
// SetOnCommitted wires the after-commit hook (R-1285). INIT-ONLY: call it before MaybeCommit's goroutine starts.
|
||||
func (m *Manager) SetOnCommitted(fn func(context.Context)) { m.onCommitted = fn }
|
||||
|
||||
// ManagerConfig wires the commit manager.
|
||||
type ManagerConfig struct {
|
||||
StateDir string
|
||||
@@ -119,4 +125,7 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
|
||||
return
|
||||
}
|
||||
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
|
||||
if m.onCommitted != nil {
|
||||
m.onCommitted(ctx)
|
||||
}
|
||||
}
|
||||
@@ -24,10 +24,10 @@
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||
],
|
||||
"recorded": "2026-10-25",
|
||||
"recorded_by": "CC 2026-10-25, D50 = A (operator); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"recorded": "2026-10-29",
|
||||
"recorded_by": "CC 2026-10-29: day0_agent 0.168.0 (operator vouch 2026-10-29 10:21:56 CET, read from the hub log: agent=0.168.0, bundle 3415c034… = the release); earlier 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"pinned": {
|
||||
"hub_min_agent": "0.158.1",
|
||||
"day0_agent": "0.165.0"
|
||||
"day0_agent": "0.168.0"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user