Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
91bd96c755 | ||
|
|
6f933d6ee3 | ||
|
|
5ddd124cfc | ||
|
|
3055bc54c1 | ||
|
|
007199342f | ||
|
|
54d3a1057b | ||
|
|
fbbb168d1f | ||
|
|
3911cc6f52 | ||
|
|
ad80ee749a | ||
|
|
0683cc2dab | ||
|
|
b1bc63dd32 | ||
|
|
e66a48f2b7 | ||
|
|
b890b57540 | ||
|
|
aa69114d18 | ||
|
|
abcc1e6833 | ||
|
|
d912307bb7 | ||
|
|
feaaa0e8ff | ||
|
|
6b6ba0c17f | ||
|
|
4697f8ac9e |
No files matched your search
+120
-9
@@ -1,4 +1,115 @@
|
||||
## Unreleased (next: v0.165.0) — D43 = A: BYO hosts keep their own SSH settings (R-1257) (2026-10-24 night) — committed, NOT released (the release + the operator's vouch are Monday's)
|
||||
## Unreleased (2026-10-28 night) — R-528: GET /guest/oom-kills, the kernel's OOM counter of each of the guest's containers, read on the host — commit only; the next agent release (needs the operator's vouch)
|
||||
|
||||
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
|
||||
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
|
||||
could not fire. The kernel counts every kill per container cgroup, and the host can read it.
|
||||
- **`internal/localapi/guest_oom.go`:** `GET /guest/oom-kills` (withGuest — the token's guest only; a cross-guest
|
||||
`?vmid=` is 403) reads `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex>.scope/memory.events` (and
|
||||
`ns/docker/<id>/` for the cgroupfs driver), line `oom_kill N`, and answers `{vmid, available, containers: {id: n}}`.
|
||||
A guest with no cgroup folder → `available: false`. Read-only, no sudo: the files are 0444 and the non-root
|
||||
felhom-agent user reads them (measured on Tester 1's host 2026-10-28). A container's scope is new at each start, so the
|
||||
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
|
||||
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
|
||||
|
||||
## Unreleased (2026-10-28 night) — R-1254 (the ceremony half, review round 2 HUNT-3 H2): the agent asks the hub which package it holds before it words a failed upload — commit only; the next agent release (needs the operator's vouch)
|
||||
|
||||
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
|
||||
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
|
||||
hub's current package opens only with the NEW code.
|
||||
- **Now (`internal/localapi/escrow_ceremony.go`):** after a failed upload the agent asks the hub (`HubHolds`, 15 s,
|
||||
outside the lock, the slot still marked claiming): the hub holds THIS package → a success (the code is handed out /
|
||||
the confirm completes; a WARN line names the lost answer); the hub holds another package → 502 and the old sentence,
|
||||
now true; the hub cannot be asked → **504** with a sentence that promises nothing about either code, and the wait goes
|
||||
on (a retry decides). `cmd/felhom-agent`: `HubHolds` compares the pending package's identity blob with the one the hub
|
||||
serves back (`GET /api/v1/hosts/<id>/escrow`, the read-back the agent already uses; ciphertext only, nothing logged);
|
||||
a package without an identity blob cannot be compared → unknown. Tests `TestR1254_*` (localapi, red before the fix:
|
||||
502 while the hub held the new package) and `TestR1254_HubHoldsComparesTheIdentityBlob` (red-proved by mutation).
|
||||
- Needs controller ≥ the R-1254 controller commit (2026-10-28 night) to word 504 for the household; an older controller
|
||||
shows its generic „A kód lekérése nem sikerült." (claim) / its 502 sentence (confirm) for it.
|
||||
|
||||
## v0.167.0 (2026-10-27) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — RELEASED (release-agent.sh from a clean clone; agent sha256 28bdd282…d826f0, bundle 9d034416…ed1268; signed updates committed on demo-felhom, Tester 1, demo-hp; VOUCHED by the operator 2026-10-27, Day-0 pin set)
|
||||
|
||||
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
|
||||
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
|
||||
newest. Full `go build/vet/test` green. The controller and hub move the same night.
|
||||
- **R-1232 (P2):** `backup.BackupRunner.NewestArchiveTime` — the tier due-check — counts only archives written with THIS
|
||||
storage's key on an encrypted storage (the R-727 discriminator; the archive carries no host id). Before: a reinstalled
|
||||
box shares the customer's PBS namespace and guest number with the old box, the old box's archive read as „the last
|
||||
backup", and the whole-guest tier waited up to its 168-h cadence (the 2026-10-23 test night). A key lookup failure is
|
||||
an error (the due-check reads UNKNOWN), never a foreign age. Tests `TestR1232_*` (foreign only → none; own → its time;
|
||||
an unencrypted dir → unfiltered), red-proved (case 1 returned the old box's time).
|
||||
|
||||
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
|
||||
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
|
||||
second later — so 0.166.0's pass never ran on the bundle that delivered it.
|
||||
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
|
||||
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
|
||||
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
|
||||
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
|
||||
agent-only — holding the bundle job until the commit would need new hub queue logic.
|
||||
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
|
||||
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
|
||||
removed → commit only, no pass — the demo-hp symptom).
|
||||
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
|
||||
0.165.0 box passes even without this fix.
|
||||
|
||||
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
|
||||
|
||||
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
|
||||
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
|
||||
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
|
||||
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
|
||||
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
|
||||
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
|
||||
|
||||
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
|
||||
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
|
||||
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
|
||||
„BUNDLE POLICY DONE install_mode=appliance removed=0".
|
||||
|
||||
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
|
||||
|
||||
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
|
||||
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
|
||||
drop-in until its next bundle. A change inside the wrapper alone cannot fix the run that delivers it.
|
||||
- **The fix — the new AGENT asks the new wrapper:** `internal/osupdate/bundle.go` runs a policy pass right after every
|
||||
bundle install (`mode: "bundle-policy"`, host layer, no envelope); the wrapper on disk is the bundle's own by then.
|
||||
`configs/felhom-os-apply` `Bundle.policy_pass`: on a root-owned BYO record it removes the appliance-only files WE
|
||||
wrote (our header), keeps the previous copy beside the bundle's, `sshd -t` then reloads ssh — a failing `sshd -t` puts
|
||||
the file back. It writes nothing, so it needs no signature; an appliance and the owner's own file are left alone. Never
|
||||
fatal for the agent: an older wrapper refuses the unknown mode and the next bundle does it. Report key `bundle_policy`.
|
||||
- Tests: `TestR1285_AnInstalledBundleIsFollowedByThePolicyPass` (a refused bundle runs no pass);
|
||||
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
|
||||
`documentation/audits/night-2026-10-25/redproof-r1285.txt`.
|
||||
|
||||
### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
|
||||
|
||||
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
|
||||
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
|
||||
usable that long, and the nonce store no longer keeps its record that long.
|
||||
- `configs/felhom-os-apply`: the root wrapper holds the same 24 h (`MAX_WINDOW_S`, refusal R3 „the signed job's window
|
||||
is longer than 24 h").
|
||||
- `cmd/felhom-opsign`: `-ttl` above 24 h (or ≤ 0) is refused before anything is signed.
|
||||
- Tests: `TestVerify_RejectsPerStage/window_longer_than_24_h` and `…/window_of_exactly_24_h_is_accepted`;
|
||||
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
|
||||
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
|
||||
|
||||
### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
|
||||
|
||||
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
|
||||
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
|
||||
(`hub_min_agent` 0.158.1, `day0_agent` 0.165.0 — copies of hub state, because CI holds no hub secret; the session that
|
||||
records a vouch or a new MinAgent updates them in the same commit). `check-published-versions.py` asserts the window
|
||||
plus every pin; a pin with no tag FAILS. CI was red since the operator's hand prune of 0.159.0–0.160.0 (job 2334).
|
||||
- **Decoys** (`test_gate_decoys.py`, via `FELHOM_RETENTION_POLICY`, test only): a pinned version older than the window is
|
||||
asserted; its 404 convicts; a pin with no tag convicts. Seen failing on the old gate (two LIVE HOLES): felhom.eu
|
||||
`documentation/audits/night-2026-10-25/redproof-d50.txt`.
|
||||
|
||||
## v0.165.0 (2026-10-25) — D43 = A: BYO hosts keep their own SSH settings (R-1257); the line test for a box that seeds all day (R-1250) — released in the attended release run; **VOUCHED by the operator 2026-10-25 18:38:49 CET** (binary `e9d04a47…`, bundle `8dea1f34…`)
|
||||
|
||||
**Found at the release (felhom.eu R-1285, P4):** on a BYO host upgraded from 0.164.0 the drop-in is removed only on the bundle run AFTER the one that installs this wrapper — the first run is planned by the old wrapper. Proven on a BYO test host (VM 9371 on demo-hp): the second `agent_config_update` logged `BUNDLE REMOVED … (ours, on a BYO host — D43)`.
|
||||
|
||||
### BYO hosts keep their own SSH settings (R-1257, D43 = A) — 2026-10-24 night
|
||||
|
||||
- **The root-key-only sshd drop-in (`/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`, R-982 / D4) goes only to an
|
||||
appliance** — operator ruling D43 = A (2026-10-24): a BYO host belongs to its owner (`11` §1). The table entry's policy
|
||||
@@ -17,14 +128,7 @@
|
||||
is; a LAN `Match` block is not in force; every address key-only passes; a failing `sshd -t` undoes only the SSH step.
|
||||
The fake box now carries an appliance record by default (every earlier test unchanged).
|
||||
|
||||
## 2026-10-24 (review round 2, unattended) — `.claude/rules/unprompted-work.md` aligned with its four siblings
|
||||
|
||||
- The copy lacked item 6's paragraph „A version number is pushed once …" that the felhom.eu and felhom-controller copies
|
||||
have carried since 2026-10-15; the file's header says „change all five or none". The paragraph states a fact
|
||||
(`felhom-controller/controller/build.sh` refuses an existing tag and an unreachable registry). Now byte-identical to
|
||||
the other four (md5 b1891bd4). No code change. Record: felhom.eu `documentation/audits/review-round-2-2026-10-24/EDITS.md`.
|
||||
|
||||
## Unreleased (next: v0.165.0) — the line test for a box that seeds all day (R-1250) (2026-10-24) — committed, not released
|
||||
### The line test for a box that seeds all day (R-1250) — 2026-10-24
|
||||
|
||||
Attended brief 2026-10-24, Part 4 (commit only; the release rides the next normal train). The weekly test (v0.164.0)
|
||||
starts only after 10 quiet minutes; a torrent app that seeds all day never gives them, so that box never got a tested
|
||||
@@ -47,6 +151,13 @@ line and qBittorrent never got its 50 % limit. Now:
|
||||
|
||||
**No sudoers or config-bundle change. MinController: none** — an older controller never calls the route.
|
||||
|
||||
## 2026-10-24 (review round 2, unattended) — `.claude/rules/unprompted-work.md` aligned with its four siblings
|
||||
|
||||
- The copy lacked item 6's paragraph „A version number is pushed once …" that the felhom.eu and felhom-controller copies
|
||||
have carried since 2026-10-15; the file's header says „change all five or none". The paragraph states a fact
|
||||
(`felhom-controller/controller/build.sh` refuses an existing tag and an unreachable registry). Now byte-identical to
|
||||
the other four (md5 b1891bd4). No code change. Record: felhom.eu `documentation/audits/review-round-2-2026-10-24/EDITS.md`.
|
||||
|
||||
## v0.164.0 — the weekly upload line test (R-1247) (2026-10-24) — RELEASED on the operator's ruling (attended release run); vouched by the operator 09:44:58 CEST
|
||||
|
||||
Binary sha256 `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`; config bundle sha256
|
||||
|
||||
+11
@@ -1,5 +1,16 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
|
||||
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
|
||||
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
|
||||
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
|
||||
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
|
||||
|
||||
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
|
||||
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
|
||||
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
|
||||
> on demo-hp (felhom.eu R-1250, read, not forced).
|
||||
|
||||
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
|
||||
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
|
||||
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
|
||||
|
||||
@@ -1,16 +1,30 @@
|
||||
# REPORT — D43 = A: BYO hosts keep their own SSH settings (R-1257), 2026-10-24 night (unattended; commit only)
|
||||
# REPORT — two agent fixes on main, NOT released (night 2026-10-28 → 29, unattended)
|
||||
|
||||
Brief: the restore-holes run, Part 4. Starting the run was the operator's ruling **D43 = A**. Previous report (R-1250's
|
||||
line hold, attended, same day): `git show f5c3d77:REPORT.md`.
|
||||
**Plain words first.** Two changes are on main and tested. Neither is released: an agent release needs the operator's
|
||||
vouch, and the night brief allowed no release. Both go on the morning list.
|
||||
|
||||
- **`configs/felhom-os-apply` (commit `c8d3c17`):** the root-key-only sshd drop-in is an `appliance`-policy file —
|
||||
written only when the root-owned install record says `appliance`; on `byo` nothing is written and a copy we wrote
|
||||
(exact path, our header line) is removed (the previous copy kept for the undo, then `sshd -t` and a reload); a box
|
||||
whose record proves nothing is left as it is. **X3:** the in-force check asks `sshd -T` for each of the host's own
|
||||
global IPv4 addresses. **X4:** a failing `sshd -t` puts back only the SSH file (`sshd_step_undone` in the report);
|
||||
the rest of the bundle stays.
|
||||
- **Tests:** `configs/test_felhom_config_bundle.py` class `ByoKeepsItsSSH` (7), each rule seen failing with it taken out
|
||||
(policy back to `replace`; the probe back to 192.0.2.1; `sshd -t` back to `must`). The fake box now carries an
|
||||
appliance record by default; all 68 bundle tests and 146 os-apply tests pass; `agent_gates.py --fast` OK.
|
||||
- **Not done here (Monday):** the agent release v0.165.0 and the operator's vouch; a signed bundle on a BYO test host.
|
||||
- Docs: felhom.eu `03` (the D4 paragraph) and `11` §1 carry the ruling; register R-1257 → VERIFY.
|
||||
## What changed
|
||||
|
||||
- **R-528 — the OOM counter (`6f933d6`, CI run id 2473 / #284 green).** New local-API route `GET /guest/oom-kills`. It
|
||||
reads the kernel's `oom_kill` count of each Docker container of the token's own guest, on the host, from
|
||||
`/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events` (mode 0444, readable by the non-root
|
||||
agent user). Docker inside an LXC guest does not see these kills (`OOMKilled` stays false). The controller reads the
|
||||
route every 30 s (controller `eed23e7`). Tests `TestR528_*`, red first by mutation (another guest's container served;
|
||||
`oom` parsed for `oom_kill`).
|
||||
- **R-1254 — the escrow ceremony half (`5ddd124`, CI run id 2471 / #283 green).** After an escrow upload that fails, the
|
||||
agent asks the hub which package it holds (the identity blob compared): the new one → success; another one → 502 and
|
||||
„the previous code still works"; no answer → 504, which promises nothing. Tests `TestR1254_*`, red first.
|
||||
|
||||
## Live proof
|
||||
|
||||
- R-528, 2026-10-28 23:27 UTC on Tester 1 with a test build of this agent (put back after: 0.167.0, sha256 `28bdd282…`).
|
||||
A Radicale capped at 128 MB, a memory hog inside it killed: the host counter 0 → 1, one `app_oom` „seen by
|
||||
docker,cgroup", one mail to the operator with `"source":"docker,cgroup"`. Evidence: felhom.eu
|
||||
`documentation/audits/night-2026-10-28/r528-oom/`.
|
||||
- R-1254: no live proof. It needs a hub answer lost on purpose; that is a hub change, and no hub deploy runs at night.
|
||||
|
||||
## Next
|
||||
|
||||
The agent release (the operator's vouch), then `pinned.day0_agent` in `scripts/retention-policy.json` (D50).
|
||||
The controller needs no MinAgent change for R-528: an old agent answers 404, and the controller then waits 10 minutes
|
||||
before it asks again.
|
||||
@@ -0,0 +1,48 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
|
||||
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
|
||||
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
|
||||
// passes as held → FAILS.
|
||||
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
|
||||
body := func(id string) []byte {
|
||||
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
|
||||
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
|
||||
return b
|
||||
}
|
||||
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
|
||||
}
|
||||
}
|
||||
ctx := context.Background()
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
|
||||
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
|
||||
}
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
|
||||
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
|
||||
}
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
|
||||
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
|
||||
}
|
||||
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
|
||||
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
|
||||
}
|
||||
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
|
||||
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
|
||||
t.Fatal("a package without an identity blob cannot be compared — an error")
|
||||
}
|
||||
}
|
||||
@@ -1216,6 +1216,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||
}}
|
||||
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
|
||||
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
|
||||
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
|
||||
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
|
||||
// to the local API server built below).
|
||||
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
|
||||
@@ -1247,6 +1250,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
|
||||
// subprocess used, under the same per-host key, before the code is handed out.
|
||||
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
|
||||
// R-1254 H2: after a failed upload, the hub is asked which package it holds before the household is told
|
||||
// anything — the same per-host GET the identity read-back uses.
|
||||
HubHolds: func(ctx context.Context, pkg []byte) (bool, error) {
|
||||
if client == nil {
|
||||
return false, fmt.Errorf("hub client not configured")
|
||||
}
|
||||
return hubHoldsEscrowPackage(ctx, client.FetchIdentityEscrow, pkg)
|
||||
},
|
||||
DRConfigured: func() bool {
|
||||
if drConfigured != nil {
|
||||
return drConfigured()
|
||||
@@ -3363,6 +3374,32 @@ func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
|
||||
|
||||
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
|
||||
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
|
||||
// hubHoldsEscrowPackage (R-1254 H2) reports whether the hub's CURRENT package is the upload body pkg, by its identity
|
||||
// blob — the part the hub serves back (GET /api/v1/hosts/<id>/escrow). Ciphertext only; nothing is logged. A package
|
||||
// without an identity blob cannot be compared: an error (the caller then promises nothing).
|
||||
func hubHoldsEscrowPackage(ctx context.Context, fetch func(context.Context) (*hub.IdentityEscrowResponse, error), pkg []byte) (bool, error) {
|
||||
var up escrowUploadRequest
|
||||
if err := json.Unmarshal(pkg, &up); err != nil {
|
||||
return false, fmt.Errorf("reading the pending package: %w", err)
|
||||
}
|
||||
mine, err := base64.StdEncoding.DecodeString(up.IdentityBlobB64)
|
||||
if err != nil || len(mine) == 0 {
|
||||
return false, fmt.Errorf("the pending package carries no identity blob to compare")
|
||||
}
|
||||
resp, err := fetch(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !resp.Present || resp.IdentityEscrowB64 == "" {
|
||||
return false, nil
|
||||
}
|
||||
theirs, err := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("decoding the hub's package: %w", err)
|
||||
}
|
||||
return bytes.Equal(mine, theirs), nil
|
||||
}
|
||||
|
||||
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
|
||||
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
|
||||
return fmt.Errorf("hub not configured (url/host_id/api_key)")
|
||||
@@ -4040,3 +4077,9 @@ func agentStateDir(cfg config.Config) string {
|
||||
}
|
||||
return "/var/lib/felhom-agent"
|
||||
}
|
||||
|
||||
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
|
||||
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
|
||||
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
|
||||
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
|
||||
)
|
||||
|
||||
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
|
||||
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
|
||||
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
|
||||
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
|
||||
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
|
||||
|
||||
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
|
||||
type orderRunner struct {
|
||||
mu sync.Mutex
|
||||
order []string
|
||||
}
|
||||
|
||||
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
|
||||
var p map[string]any
|
||||
b, _ := os.ReadFile(args[1])
|
||||
_ = json.Unmarshal(b, &p)
|
||||
mode, _ := p["mode"].(string)
|
||||
r.order = append(r.order, "os-apply:"+mode)
|
||||
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
|
||||
}
|
||||
if len(args) >= 1 {
|
||||
r.order = append(r.order, filepath.Base(name)+":"+args[0])
|
||||
}
|
||||
return []byte("ok"), nil, nil
|
||||
}
|
||||
|
||||
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
|
||||
t.Helper()
|
||||
stateDir := t.TempDir()
|
||||
if marker != nil {
|
||||
dir := filepath.Join(stateDir, "selfupdate")
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := json.Marshal(marker)
|
||||
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
r := &orderRunner{}
|
||||
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
|
||||
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
|
||||
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
|
||||
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
|
||||
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
|
||||
mgr.MaybeCommit(context.Background())
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return append([]string(nil), r.order...)
|
||||
}
|
||||
|
||||
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
|
||||
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
|
||||
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
|
||||
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
|
||||
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
|
||||
}
|
||||
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -123,6 +123,10 @@ func run() error {
|
||||
|
||||
now := time.Now().UTC()
|
||||
issued := now
|
||||
if *ttl <= 0 || *ttl > authz.MaxWindow {
|
||||
fmt.Fprintf(os.Stderr, "felhom-opsign: -ttl %s refused: a box accepts a window of at most %s (R-1280)\n", *ttl, authz.MaxWindow)
|
||||
os.Exit(2)
|
||||
}
|
||||
expires := now.Add(*ttl)
|
||||
|
||||
// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
|
||||
|
||||
+48
-1
@@ -187,6 +187,8 @@ BUNDLE_PREV_DIR = "/var/lib/felhom-os-apply/bundle-prev" # the previous copies
|
||||
BUNDLE_KEEP = 3
|
||||
BUNDLE_MAX_BYTES = 4 * 1024 * 1024
|
||||
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
|
||||
# R-1280: the longest validity window a signed job may carry — the agent's authz.MaxWindow, the same 24 h.
|
||||
MAX_WINDOW_S = 24 * 3600
|
||||
OOB_DIR = "/etc/felhom-sshd"
|
||||
# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
|
||||
SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||
@@ -489,7 +491,8 @@ class Apply:
|
||||
|
||||
def check_plan(self, plan):
|
||||
mode = plan.get("mode", "apply")
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
|
||||
"oom-check") \
|
||||
+ KERNEL_MODES:
|
||||
raise Refused("R11", f"unknown mode {mode!r}")
|
||||
if mode == "agent_update":
|
||||
@@ -500,6 +503,10 @@ class Apply:
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "bundle is a host-layer mode")
|
||||
return mode, "host", 0, "bundle"
|
||||
if mode == "bundle-policy":
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "bundle-policy is a host-layer mode")
|
||||
return mode, "host", 0, "bundle-policy"
|
||||
layer = plan.get("layer")
|
||||
if layer not in ("guest", "host", "docker", "pve", "kernel"):
|
||||
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
|
||||
@@ -661,6 +668,8 @@ class Apply:
|
||||
iss = calendar.timegm(time.strptime(op["issued_at"], "%Y-%m-%dT%H:%M:%SZ"))
|
||||
except (KeyError, ValueError, TypeError):
|
||||
raise Refused("R3", "the signed job has no readable time window")
|
||||
if exp - iss > MAX_WINDOW_S:
|
||||
raise Refused("R3", "the signed job's window is longer than 24 h (R-1280)")
|
||||
if now > exp or now < iss - 120:
|
||||
raise Refused("R3", "the signed job is expired or not yet valid")
|
||||
nonce = op.get("nonce")
|
||||
@@ -1070,6 +1079,8 @@ class Apply:
|
||||
return self.facts()
|
||||
if self.mode == "bundle":
|
||||
return Bundle(self).from_plan(plan)
|
||||
if self.mode == "bundle-policy":
|
||||
return Bundle(self).policy_pass()
|
||||
if self.mode == "agent_update":
|
||||
return self.agent_update(plan)
|
||||
if self.layer in ("host", "pve", "kernel"):
|
||||
@@ -2060,6 +2071,42 @@ class Bundle:
|
||||
ver, files = self.parse(data, sha)
|
||||
return self.install(ver, sha, files, "installer", False)
|
||||
|
||||
def policy_pass(self):
|
||||
"""R-1285 (D43 = A): the appliance-only files' policy, applied by THIS wrapper. A bundle that brings a new wrapper
|
||||
is planned by the OLD one, which did not know the policy — so on a BYO host upgraded from 0.164.0 our SSH drop-in
|
||||
stayed until the next bundle. The agent runs this pass right after every bundle install (the wrapper on disk is
|
||||
then the new one). It only REMOVES a file WE wrote (our header) from a box whose root-owned install record says
|
||||
byo, keeps the previous copy beside the bundle's, and reloads sshd; it writes nothing, so it needs no signature.
|
||||
A failing `sshd -t` puts the file back (R-1257 X4's rule). Report key `bundle_policy`."""
|
||||
log = self.r.log
|
||||
install_mode = self.install_mode()
|
||||
removed = []
|
||||
if install_mode == "byo":
|
||||
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
|
||||
prev = os.path.join(BUNDLE_PREV_DIR, f"{stamp}-policy")
|
||||
for dest, _src, _mode, _check, policy in BUNDLE_FILES:
|
||||
if policy != "appliance":
|
||||
continue
|
||||
old, old_mode, _ = self.live(dest)
|
||||
if old is None or not old.startswith(SSHD_DROPIN_HEADER):
|
||||
continue
|
||||
self.r.put_file(prev + dest, old, 0o600)
|
||||
self.r.remove(dest)
|
||||
removed.append((dest, old, old_mode))
|
||||
log(f"os-apply: BUNDLE POLICY REMOVED {dest} (ours, on a BYO host — D43, R-1285)")
|
||||
if any(d == SSHD_DROPIN for d, _, _ in removed):
|
||||
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
|
||||
if rc != 0:
|
||||
for d, old, old_mode in removed:
|
||||
self.r.put_file(d, old, old_mode)
|
||||
log(f"os-apply: BUNDLE POLICY WARNING sshd -t failed without our drop-in (rc={rc}) — put back")
|
||||
removed = []
|
||||
else:
|
||||
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
|
||||
self.report["bundle_policy"] = {"install_mode": install_mode, "removed": [d for d, _, _ in removed]}
|
||||
log(f"os-apply: BUNDLE POLICY DONE install_mode={install_mode} removed={len(removed)}")
|
||||
return 0
|
||||
|
||||
# ---------- install, self-check, undo ----------
|
||||
def install(self, ver, sha, files, authority, bootstrap):
|
||||
log = self.r.log
|
||||
|
||||
@@ -588,6 +588,47 @@ class ByoKeepsItsSSH(unittest.TestCase):
|
||||
return orig(argv, timeout, stdin)
|
||||
box.host = host
|
||||
|
||||
# R-1285: the policy pass the agent runs right after every bundle install, with the wrapper NOW on disk. No
|
||||
# signature (it only removes a file we wrote). RED-PROOF: make policy_pass return before the loop → our copy stays.
|
||||
POLICY_PLAN = {"release_id": "bundle-policy-0.166.0", "layer": "host", "mode": "bundle-policy"}
|
||||
|
||||
def test_the_policy_pass_removes_our_copy_from_a_byo_host(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, None, install_mode="byo")
|
||||
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
|
||||
box.put(self.DEST, self.OURS, 0o644) # the old wrapper's bundle run left it
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertNotIn(self.DEST, box.files)
|
||||
self.assertEqual(rep["bundle_policy"], {"install_mode": "byo", "removed": [self.DEST]}, rep)
|
||||
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files))
|
||||
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
|
||||
|
||||
def test_the_policy_pass_leaves_an_appliance_and_the_owners_file(self):
|
||||
for mode, content in (("appliance", None), ("byo", b"# my own\nPermitRootLogin yes\n")):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, None, install_mode=mode)
|
||||
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
|
||||
body = content if content is not None else self.OURS
|
||||
box.put(self.DEST, body, 0o644)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.files.get(self.DEST), body, mode)
|
||||
self.assertEqual(rep["bundle_policy"]["removed"], [], mode)
|
||||
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, mode)
|
||||
|
||||
def test_the_policy_pass_puts_our_copy_back_when_sshd_t_fails(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, None, install_mode="byo")
|
||||
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
|
||||
box.put(self.DEST, self.OURS, 0o644)
|
||||
self._fake(box, sshd_t_rc=1)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.files.get(self.DEST), self.OURS)
|
||||
self.assertEqual(rep["bundle_policy"]["removed"], [])
|
||||
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||
|
||||
def test_a_match_block_for_the_lan_is_not_in_force(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
|
||||
@@ -833,6 +833,14 @@ class DockerLane(unittest.TestCase):
|
||||
def test_expired_signed_job_is_refused(self):
|
||||
self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3")
|
||||
|
||||
def test_a_window_longer_than_24h_is_refused(self):
|
||||
# R-1280. RED-PROOF: drop the MAX_WINDOW_S check → the job (current, signed, 24 h + 1 s) runs.
|
||||
self.refused(docker_fake(signed=signed_job(issued="2026-10-03T11:59:59Z", expires="2026-10-04T12:00:00Z")), "R3")
|
||||
|
||||
def test_a_window_of_24h_is_accepted(self):
|
||||
rc, rep = run(docker_fake(signed=signed_job(issued="2026-10-03T12:30:00Z", expires="2026-10-04T12:30:00Z")))
|
||||
self.assertNotEqual((rep.get("refused") or {}).get("code"), "R3", rep)
|
||||
|
||||
def test_replayed_signed_job_is_refused(self):
|
||||
f = docker_fake(signed=signed_job())
|
||||
f.nonces = {"n1": f.clock + 600}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
module gitea.dooplex.hu/admin/felhom-agent
|
||||
|
||||
go 1.27.2
|
||||
go 1.27.3
|
||||
|
||||
require (
|
||||
golang.org/x/crypto v0.58.0
|
||||
|
||||
@@ -22,6 +22,8 @@ var (
|
||||
ErrExpired = errors.New("authz: op expired")
|
||||
// ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
|
||||
ErrNotYetValid = errors.New("authz: op not yet valid")
|
||||
// ErrWindowTooLong: expires_at - issued_at > MaxWindow (R-1280).
|
||||
ErrWindowTooLong = errors.New("authz: op validity window too long")
|
||||
// ErrReplay: the nonce was already recorded in the window.
|
||||
ErrReplay = errors.New("authz: replay (nonce already seen)")
|
||||
)
|
||||
@@ -20,6 +20,11 @@ const Namespace = "felhom-op-v1"
|
||||
// the validity window stays an honest upper bound.
|
||||
const DefaultClockSkew = 2 * time.Minute
|
||||
|
||||
// MaxWindow is the longest validity window a signed op may carry (R-1280, D44 #22, operator 2026-10-25): a job signed
|
||||
// with a long window by mistake would stay usable that long, and the nonce store would keep its record as long. The
|
||||
// root wrapper (configs/felhom-os-apply, MAX_WINDOW_S) and felhom-opsign (-ttl) hold the same 24 h.
|
||||
const MaxWindow = 24 * time.Hour
|
||||
|
||||
// KeyRole tags a pinned operator key (doc 04 §3 two-key model).
|
||||
type KeyRole string
|
||||
|
||||
@@ -150,6 +155,9 @@ func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
|
||||
if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
|
||||
return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
|
||||
}
|
||||
if w := op.ExpiresAt.Sub(op.IssuedAt); w > MaxWindow {
|
||||
return nil, fmt.Errorf("%w: issued_at=%s expires_at=%s (%s, the most is %s)", ErrWindowTooLong, op.IssuedAt, op.ExpiresAt, w, MaxWindow)
|
||||
}
|
||||
if now.After(op.ExpiresAt) {
|
||||
return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
|
||||
}
|
||||
|
||||
@@ -117,6 +117,26 @@ func TestVerify_RejectsPerStage(t *testing.T) {
|
||||
rejects(t, err, ErrTarget)
|
||||
})
|
||||
|
||||
// R-1280: a window longer than 24 h is refused even while it is current; exactly 24 h is accepted.
|
||||
// RED-PROOF: drop the MaxWindow check → the 24 h + 1 s case verifies.
|
||||
t.Run("window longer than 24 h", func(t *testing.T) {
|
||||
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-1", "{}",
|
||||
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow+time.Second))
|
||||
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
|
||||
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
|
||||
_, err := v.Verify(blob, sig)
|
||||
rejects(t, err, ErrWindowTooLong)
|
||||
})
|
||||
t.Run("window of exactly 24 h is accepted", func(t *testing.T) {
|
||||
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-2", "{}",
|
||||
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow))
|
||||
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
|
||||
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
|
||||
if _, err := v.Verify(blob, sig); err != nil {
|
||||
t.Fatalf("a 24 h window must verify: %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("expired", func(t *testing.T) {
|
||||
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
|
||||
refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1232 — the 2026-10-23 test night: a reinstalled box (new install, new PBS key, same customer namespace, same guest
|
||||
// number 9201) found the OLD box's archive (2.6 days old) when the due-check asked „when did this tier last land?", so
|
||||
// the whole-guest tier read „not due" for up to its 168-h cadence and the ep0 copy stayed the OLD box's. The due-check
|
||||
// now counts only archives written with THIS storage's key (the R-727 discriminator; the archive carries no host id).
|
||||
// The CONSEQUENCE asserted: with only a foreign archive the answer is „none found" (the tier is due); with the box's
|
||||
// own archive, that archive's time. COMPANION RED-PROOF: drop the key filter in NewestArchiveTime → case 1 returns the
|
||||
// old box's time.
|
||||
func TestR1232_TheDueCheckCountsOnlyThisBoxsArchives(t *testing.T) {
|
||||
oldBox := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-20T21:10:00Z", Content: "backup", VMID: 9201,
|
||||
Size: 3490689830, CTime: time.Date(2026, 10, 20, 21, 10, 0, 0, time.UTC).Unix(), Encrypted: earlierBox1}
|
||||
api := &fakeBackupAPI{
|
||||
storages: []proxmox.Storage{{Storage: "felhom-pbsdr", Type: "pbs", EncryptionKey: thisBoxKey}},
|
||||
content: []proxmox.StorageContent{oldBox},
|
||||
}
|
||||
r := NewBackupRunner(api, "felhom-pbsdr", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
|
||||
// 1. The fresh box has written nothing yet: the old box's archive must not count.
|
||||
when, found, err := r.NewestArchiveTime(context.Background(), 9201)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if found {
|
||||
t.Fatalf("the due-check counted ANOTHER box's archive (%s) as this tier's last backup — the tier would wait a week", when)
|
||||
}
|
||||
// 2. After the box's own first copy, that copy is the answer.
|
||||
own := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-24T02:30:00Z", Content: "backup", VMID: 9201,
|
||||
Size: 3600000000, CTime: time.Date(2026, 10, 24, 2, 30, 0, 0, time.UTC).Unix(), Encrypted: thisBoxKey}
|
||||
api.content = append(api.content, own)
|
||||
when, found, err = r.NewestArchiveTime(context.Background(), 9201)
|
||||
if err != nil || !found || when.Unix() != own.CTime {
|
||||
t.Fatalf("want this box's own archive time %d, got %v found=%v err=%v", own.CTime, when, found, err)
|
||||
}
|
||||
}
|
||||
|
||||
// An unencrypted storage (a local dir) holds only this box's vzdumps: no key filter, the newest counts.
|
||||
func TestR1232_UnencryptedStorageCountsEveryArchive(t *testing.T) {
|
||||
api := &fakeBackupAPI{
|
||||
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
|
||||
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_10_25-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1793050025}},
|
||||
}
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
if _, found, err := r.NewestArchiveTime(context.Background(), 9201); err != nil || !found {
|
||||
t.Fatalf("an unencrypted local archive was not counted: found=%v err=%v", found, err)
|
||||
}
|
||||
}
|
||||
@@ -609,11 +609,23 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
|
||||
if err != nil {
|
||||
return time.Time{}, false, err
|
||||
}
|
||||
// R-1232: on an ENCRYPTED storage only archives written with THIS storage's key are this box's (the R-727
|
||||
// discriminator — the archive carries no host id). A reinstalled box shares the customer's namespace and guest
|
||||
// number with the old box, and the old box's archive made the tier read „not due" for a whole cadence (168 h).
|
||||
// A key lookup failure is returned: the due-check reads UNKNOWN, never a foreign archive's age.
|
||||
ownKey, err := r.storageKeyFingerprint(ctx, r.target)
|
||||
if err != nil {
|
||||
return time.Time{}, false, fmt.Errorf("reading the key fingerprint of storage %s: %w", r.target, err)
|
||||
}
|
||||
var best int64 = -1
|
||||
for _, e := range contents {
|
||||
if e.Content != "backup" || e.VMID != vmid {
|
||||
continue
|
||||
}
|
||||
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
|
||||
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this tier's last backup (R-1232)", shortFP(e.Encrypted), shortFP(ownKey)))
|
||||
continue
|
||||
}
|
||||
if ok, why := archivePlausiblyComplete(e); !ok {
|
||||
r.warnRejectedArchiveOnce(e, why)
|
||||
continue
|
||||
|
||||
@@ -72,12 +72,56 @@ type EscrowCeremonyConfig struct {
|
||||
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
|
||||
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
|
||||
Upload func(ctx context.Context, pkg []byte) error
|
||||
// HubHolds (R-1254 H2) asks the hub whether its CURRENT package is exactly pkg — asked after a failed upload,
|
||||
// whose answer may have been lost AFTER the hub stored it. nil or an error → the outcome is unknown.
|
||||
HubHolds func(ctx context.Context, pkg []byte) (bool, error)
|
||||
}
|
||||
|
||||
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
|
||||
// ceremony incl. its upload was measured at ~2.4 s).
|
||||
const escrowUploadTimeout = 30 * time.Second
|
||||
|
||||
// escrowHoldTimeout bounds the hub question after a failed upload (R-1254 H2).
|
||||
const escrowHoldTimeout = 15 * time.Second
|
||||
|
||||
// uploadOutcome is what a claim or confirm upload did, as the HUB tells it (R-1254 H2): an upload whose answer was lost
|
||||
// or late may have been stored all the same.
|
||||
type uploadOutcome int
|
||||
|
||||
const (
|
||||
uploadStored uploadOutcome = iota // the hub holds the new package
|
||||
uploadNotStored // the hub answered and holds another package: the previous code still works
|
||||
uploadUnknown // the hub cannot be asked: nothing may be promised about either code
|
||||
)
|
||||
|
||||
// escrowUnknownOutcomeMsg is the answer when the hub cannot say which package it holds — never „the previous code
|
||||
// still works", which is false when the lost answer came after the store.
|
||||
const escrowUnknownOutcomeMsg = "the hub did not answer in time — it is not known whether it stored the new recovery package; try again (until a try succeeds, do not rely on the previous code alone)"
|
||||
|
||||
// outcomeAfterFailedUpload asks the hub whether it holds pkg after Upload returned an error. Called with the slot
|
||||
// marked claiming (escrowClaiming), outside escrowMu — it is a network call.
|
||||
func (s *Server) outcomeAfterFailedUpload(pkg []byte) uploadOutcome {
|
||||
if s.escrowCeremony.HubHolds == nil {
|
||||
return uploadUnknown
|
||||
}
|
||||
base := s.baseCtx
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(base, escrowHoldTimeout)
|
||||
defer cancel()
|
||||
held, err := s.escrowCeremony.HubHolds(ctx, pkg)
|
||||
switch {
|
||||
case err != nil:
|
||||
s.logger.Warn("local-api: escrow — the hub cannot say which package it holds after a failed upload (R-1254)", "err", err)
|
||||
return uploadUnknown
|
||||
case held:
|
||||
return uploadStored
|
||||
default:
|
||||
return uploadNotStored
|
||||
}
|
||||
}
|
||||
|
||||
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
|
||||
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
|
||||
// spike-shaped output; production is runCeremonySubprocess.
|
||||
@@ -524,15 +568,30 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
|
||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||
cancel()
|
||||
outcome := uploadStored
|
||||
if err != nil {
|
||||
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2: the answer may be lost AFTER the store
|
||||
}
|
||||
|
||||
s.escrowMu.Lock()
|
||||
s.escrowClaiming = false
|
||||
if err != nil {
|
||||
switch outcome {
|
||||
case uploadNotStored:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||
return
|
||||
case uploadUnknown:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow claim — upload failed and the hub cannot say which package it holds; code kept claimable (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: escrow claim — the upload's answer was lost, but the hub holds the new package; handing out the code (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err)
|
||||
}
|
||||
s.escrowPending = nil
|
||||
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
|
||||
@@ -631,15 +690,30 @@ func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Requ
|
||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||
cancel()
|
||||
outcome := uploadStored
|
||||
if err != nil {
|
||||
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2
|
||||
}
|
||||
|
||||
s.escrowMu.Lock()
|
||||
s.escrowClaiming = false
|
||||
if err != nil {
|
||||
switch outcome {
|
||||
case uploadNotStored:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||
return
|
||||
case uploadUnknown:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow confirm — upload failed and the hub cannot say which package it holds; still waiting (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: escrow confirm — the upload's answer was lost, but the hub holds the new package (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err)
|
||||
}
|
||||
s.escrowPending = nil
|
||||
s.wipeEscrowConfirmLocked()
|
||||
|
||||
@@ -41,9 +41,13 @@ type fakeEscrowHub struct {
|
||||
mu sync.Mutex
|
||||
current []byte
|
||||
uploads int
|
||||
err error // non-nil → the upload fails, nothing stored
|
||||
gate chan struct{} // non-nil → the upload blocks until closed
|
||||
entered chan struct{} // non-nil → closed when an upload starts
|
||||
err error // non-nil → the upload fails, nothing stored
|
||||
// R-1254 H2: lost → the upload STORES the package and then fails (the answer is lost on the way back);
|
||||
// holdsErr → the hub cannot be asked which package it holds.
|
||||
lost bool
|
||||
holdsErr error
|
||||
gate chan struct{} // non-nil → the upload blocks until closed
|
||||
entered chan struct{} // non-nil → closed when an upload starts
|
||||
}
|
||||
|
||||
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
||||
@@ -61,9 +65,23 @@ func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
||||
}
|
||||
f.current = append([]byte(nil), pkg...)
|
||||
f.uploads++
|
||||
if f.lost {
|
||||
return context.DeadlineExceeded
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// holds answers „is the hub's current package exactly pkg?" — the R-1254 question (production compares the identity
|
||||
// blob the hub serves back).
|
||||
func (f *fakeEscrowHub) holds(_ context.Context, pkg []byte) (bool, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.holdsErr != nil {
|
||||
return false, f.holdsErr
|
||||
}
|
||||
return string(f.current) == string(pkg), nil
|
||||
}
|
||||
|
||||
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
|
||||
func (f *fakeEscrowHub) opensWith(r string) bool {
|
||||
f.mu.Lock()
|
||||
@@ -100,6 +118,7 @@ func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub
|
||||
HubConfigured: true,
|
||||
DRConfigured: func() bool { return true },
|
||||
Upload: hub.upload,
|
||||
HubHolds: hub.holds,
|
||||
}
|
||||
srv.ceremonyRun = run
|
||||
srv.escrowSudoCheck = func(context.Context) error { return nil }
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1254 (the ceremony half, review round 2 HUNT-3 H2): the upload's answer can be lost or late (30 s) AFTER the hub
|
||||
// stored the new package. The household was then told „the previous code still works" — false: the hub now holds the
|
||||
// package the NEW code opens. The agent asks the hub which package it holds before it words the result:
|
||||
// - the hub holds the new package → success (the code is handed out / the confirm completes);
|
||||
// - the hub holds another package → the old sentence, now true;
|
||||
// - the hub cannot be asked → 504 and a sentence that does NOT promise the previous code; the wait goes on.
|
||||
|
||||
// RED-PROOF: remove the hub question from the claim's failure branch → 502 „the previous code still works" while the
|
||||
// hub holds the new package → FAILS.
|
||||
func TestR1254_ALostAnswerAtTheClaimIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
code, body := claimCode(t, h)
|
||||
if !hub.opensWith(firstR) {
|
||||
t.Fatal("fixture: the hub must hold the new package (the answer was lost after the store)")
|
||||
}
|
||||
if code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||
t.Fatalf("claim after a lost answer, the hub holding the new package: %d %s — want 200 with the code", code, body)
|
||||
}
|
||||
if strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("the household was told the previous code still works: %s", body)
|
||||
}
|
||||
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
|
||||
t.Fatalf("the job must read uploaded: %s", st)
|
||||
}
|
||||
}
|
||||
|
||||
// The hub cannot be asked: no promise about the previous code, 504, the code stays claimable (a retry decides).
|
||||
// RED-PROOF: as above → 502 with „the previous code still works" → FAILS.
|
||||
func TestR1254_AnUnknownOutcomeAtTheClaimPromisesNothing(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true, holdsErr: errors.New("hub unreachable")}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
code, body := claimCode(t, h)
|
||||
if code != http.StatusGatewayTimeout {
|
||||
t.Fatalf("claim with an unknown outcome: %d %s — want 504", code, body)
|
||||
}
|
||||
if strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("an unknown outcome promised the previous code: %s", body)
|
||||
}
|
||||
assertNoRIn(t, "unknown-outcome claim payload", body, firstR)
|
||||
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) {
|
||||
t.Fatalf("the code must stay claimable: %s", st)
|
||||
}
|
||||
// The hub answers again: the retry finds the new package already stored and hands the code out.
|
||||
hub.mu.Lock()
|
||||
hub.holdsErr = nil
|
||||
hub.mu.Unlock()
|
||||
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||
t.Fatalf("retry: %d %s", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The hub holds ANOTHER package (the upload really failed): the old sentence stays, and is now true.
|
||||
func TestR1254_AFailedUploadTheHubDidNotStoreKeepsTheOldSentence(t *testing.T) {
|
||||
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
if code, body := claimCode(t, h); code != http.StatusBadGateway || !strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("a failed upload: %d %s — want 502 and the previous code named", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The same at the typed-back confirm (R-1151).
|
||||
// RED-PROOF: remove the hub question from the confirm's failure branch → 502 → FAILS.
|
||||
func TestR1254_ALostAnswerAtTheConfirmIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
c := claimDeferred(t, h)
|
||||
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
|
||||
t.Fatalf("confirm after a lost answer: %d (opens: %v) — want 200", code, hub.opensWith(secondR))
|
||||
}
|
||||
hub.mu.Lock()
|
||||
hub.lost, hub.holdsErr = true, errors.New("hub unreachable")
|
||||
hub.mu.Unlock()
|
||||
srv2, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h2 := srv2.Handler()
|
||||
startAndWait(t, srv2, h2)
|
||||
c2 := claimDeferred(t, h2)
|
||||
if code := confirm(t, h2, c2.typedBack(t)); code != http.StatusGatewayTimeout {
|
||||
t.Fatalf("confirm with an unknown outcome: %d — want 504", code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// R-528 — the OOM signal from the host side.
|
||||
//
|
||||
// MEASURED 2026-09-15 (scratch 9202, Docker 29.8.0) and 2026-09-16 (a fresh install in a nested VM): an app killed by
|
||||
// its memory limit inside a Felhom LXC guest left Docker's State.OOMKilled false and sent no `oom` event, so the
|
||||
// controller's OOM line (R-514) could not fire. The kernel's own counter does count every kill, and the host can read
|
||||
// it: /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events, line `oom_kill N` (the
|
||||
// systemd cgroup driver; `ns/docker/<id>/` under the cgroupfs driver). Read on Tester 1's host 2026-10-28: the files
|
||||
// are mode 0444, readable by the non-root felhom-agent user — no sudo, no new root path.
|
||||
//
|
||||
// A container's scope is new at each start, so the counter is per container RUN: a count above 0 means this run had a
|
||||
// kill. The controller asks every 30 s and raises its existing app_oom line (once per run) for any such container.
|
||||
|
||||
// readGuestOOMKills returns container id → oom_kill for one guest. available=false (no error) when the guest has no
|
||||
// cgroup folder (a VM guest, or a container that is not running).
|
||||
func readGuestOOMKills(root string, vmid int) (map[string]int64, bool, error) {
|
||||
if root == "" {
|
||||
root = "/sys/fs/cgroup"
|
||||
}
|
||||
base := filepath.Join(root, "lxc", strconv.Itoa(vmid), "ns")
|
||||
if fi, err := os.Stat(base); err != nil || !fi.IsDir() {
|
||||
return nil, false, nil
|
||||
}
|
||||
out := map[string]int64{}
|
||||
for _, pattern := range []string{
|
||||
filepath.Join(base, "system.slice", "docker-*.scope", "memory.events"),
|
||||
filepath.Join(base, "docker", "*", "memory.events"),
|
||||
} {
|
||||
files, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil, true, err
|
||||
}
|
||||
for _, f := range files {
|
||||
id := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(filepath.Dir(f)), "docker-"), ".scope")
|
||||
if !isContainerID(id) {
|
||||
continue
|
||||
}
|
||||
n, err := readOOMKill(f)
|
||||
if err != nil {
|
||||
continue // a container that stopped between the glob and the read
|
||||
}
|
||||
out[id] = n
|
||||
}
|
||||
}
|
||||
return out, true, nil
|
||||
}
|
||||
|
||||
func isContainerID(s string) bool {
|
||||
if len(s) != 64 {
|
||||
return false
|
||||
}
|
||||
for _, c := range s {
|
||||
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func readOOMKill(path string) (int64, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
k, v, ok := strings.Cut(strings.TrimSpace(sc.Text()), " ")
|
||||
if ok && k == "oom_kill" {
|
||||
return strconv.ParseInt(strings.TrimSpace(v), 10, 64)
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return 0, fmt.Errorf("no oom_kill line in %s", path)
|
||||
}
|
||||
|
||||
// handleGuestOOMKills is GET /guest/oom-kills: {"vmid", "available", "containers": {id: oom_kill}}.
|
||||
func (s *Server) handleGuestOOMKills(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
kills, avail, err := readGuestOOMKills(s.cgroupRoot, vmid)
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: guest oom-kills read failed (R-528)", "vmid", vmid, "err", err)
|
||||
writeErr(w, http.StatusInternalServerError, "the guest's OOM counters could not be read")
|
||||
return
|
||||
}
|
||||
if kills == nil {
|
||||
kills = map[string]int64{}
|
||||
}
|
||||
writeOK(w, map[string]any{"vmid": vmid, "available": avail, "containers": kills})
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-528: Docker does not always report an OOM kill inside a Felhom LXC guest (OOMKilled stays false, no `oom` event —
|
||||
// measured 2026-09-15/16 on two boxes), but the kernel's own counter does count it, and it is readable from the host:
|
||||
// /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events, line `oom_kill N`. GET /guest/oom-kills
|
||||
// serves the asking guest's counters, and only that guest's.
|
||||
|
||||
func fakeCgroup(t *testing.T, root string, vmid, id, events string) {
|
||||
t.Helper()
|
||||
dir := filepath.Join(root, "lxc", vmid, "ns", "system.slice", "docker-"+id+".scope")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o444); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
const idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
const idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||
const idOther = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
|
||||
|
||||
// RED-PROOF: serve the counters of every guest under lxc/ (drop the vmid from the glob) → the other guest's container
|
||||
// appears → FAILS; parse `oom ` instead of `oom_kill` → idA reads 9 → FAILS.
|
||||
func TestR528_GuestOOMKillsServesThisGuestsCountersOnly(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
fakeCgroup(t, root, "100", idA, "low 0\nhigh 0\nmax 4\noom 9\noom_kill 3\noom_group_kill 0\n")
|
||||
fakeCgroup(t, root, "100", idB, "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\n")
|
||||
fakeCgroup(t, root, "200", idOther, "oom_kill 7\n")
|
||||
// a scope that is not a container (no 64-hex id) and a cgroupfs-driver layout container
|
||||
if err := os.MkdirAll(filepath.Join(root, "lxc", "100", "ns", "system.slice", "cron.service"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dfDir := filepath.Join(root, "lxc", "100", "ns", "docker", idOther[:8]+strings.Repeat("d", 56))
|
||||
if err := os.MkdirAll(dfDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dfDir, "memory.events"), []byte("oom_kill 2\n"), 0o444)
|
||||
|
||||
got, avail, err := readGuestOOMKills(root, 100)
|
||||
if err != nil || !avail {
|
||||
t.Fatalf("read: avail=%v err=%v", avail, err)
|
||||
}
|
||||
want := map[string]int64{idA: 3, idB: 0, idOther[:8] + strings.Repeat("d", 56): 2}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
for k, v := range want {
|
||||
if got[k] != v {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
if _, leaked := got[idOther]; leaked {
|
||||
t.Fatal("another guest's container leaked into this guest's answer")
|
||||
}
|
||||
// A guest with no cgroup folder (a VM, or not running): not available, no error.
|
||||
if _, avail, err := readGuestOOMKills(root, 300); avail || err != nil {
|
||||
t.Fatalf("a guest without a cgroup folder: avail=%v err=%v", avail, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Through the route, with the guest's own token.
|
||||
func TestR528_TheRouteAnswersTheTokensGuest(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
fakeCgroup(t, root, "8200", idA, "oom_kill 5\n")
|
||||
fakeCgroup(t, root, "9300", idB, "oom_kill 1\n") // token B's guest: never in A's answer
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
srv.cgroupRoot = root
|
||||
w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "A", "")
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("route: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
var env struct {
|
||||
Data struct {
|
||||
VMID int `json:"vmid"`
|
||||
Available bool `json:"available"`
|
||||
Containers map[string]int64 `json:"containers"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if env.Data.VMID != 8200 || !env.Data.Available || len(env.Data.Containers) != 1 || env.Data.Containers[idA] != 5 {
|
||||
t.Fatalf("answer %s — want guest 8200's one container with 5", w.Body.String())
|
||||
}
|
||||
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("a cross-guest ask: %d, want 403", w.Code)
|
||||
}
|
||||
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "", ""); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: %d", w.Code)
|
||||
}
|
||||
}
|
||||
@@ -459,6 +459,8 @@ type Server struct {
|
||||
escrowLookPath func(file string) (string, error)
|
||||
// statFile reports whether a path exists (preflight's staged-secret item; tests inject).
|
||||
statFile func(path string) bool
|
||||
// cgroupRoot is where the host's cgroup tree is mounted (R-528, GET /guest/oom-kills); "" = /sys/fs/cgroup. Tests set it.
|
||||
cgroupRoot string
|
||||
|
||||
baseCtx context.Context // for fire-and-forget backups; set in Run
|
||||
}
|
||||
@@ -576,6 +578,9 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
|
||||
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
|
||||
// read it. Read-only; never measured (or the job not wired) = present:false.
|
||||
// R-528: the kernel's OOM-kill counter of each of the guest's containers, read on the host (Docker's own flag and
|
||||
// event can stay silent inside an LXC guest). Read-only; the asking guest's containers only.
|
||||
mux.HandleFunc("GET /guest/oom-kills", s.withGuest(s.handleGuestOOMKills))
|
||||
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
|
||||
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
|
||||
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -114,12 +115,43 @@ func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params jso
|
||||
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
|
||||
}
|
||||
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
|
||||
e.policyPass(ctx, lg, p.AgentVersion)
|
||||
if e.AfterInstall != nil {
|
||||
e.AfterInstall(ctx)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// policyPass (R-1285) asks the wrapper NOW on disk — after this install, the bundle's own — to apply the
|
||||
// appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run
|
||||
// itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal:
|
||||
// the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it.
|
||||
// PolicyPassAfterUpdate runs the wrapper's bundle-policy pass once after a self-update commits (R-1285). In the normal
|
||||
// delivery the box gets agent_update and agent_config_update in ONE pass: the bundle job runs in the OLD process, which
|
||||
// has no policy pass, and the restart into the new agent comes a second later (measured 2026-10-26 on demo-hp). By the
|
||||
// commit the wrapper on disk is the bundle's own, so the pass removes our appliance-only files from a BYO host in the
|
||||
// SAME update. Idempotent; on an appliance it removes nothing.
|
||||
func (e ConfigUpdateExecutor) PolicyPassAfterUpdate(ctx context.Context, lg *slog.Logger, ver string) {
|
||||
if lg == nil {
|
||||
lg = slog.Default()
|
||||
}
|
||||
lg.Info("osupdate: running the bundle policy pass after the self-update commit (R-1285)", "agent_version", ver)
|
||||
e.policyPass(ctx, lg, ver)
|
||||
}
|
||||
|
||||
func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) {
|
||||
runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z")
|
||||
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"})
|
||||
switch {
|
||||
case err != nil:
|
||||
lg.Warn("osupdate: the bundle policy pass did not run", "err", err)
|
||||
case wr.refused():
|
||||
lg.Info("osupdate: the bundle policy pass was refused (a wrapper before 0.166.0?) — the next bundle applies it", "refused", string(wr.Refused))
|
||||
default:
|
||||
lg.Info("osupdate: bundle policy pass done", "bundle_policy", string(wr.BundlePolicy))
|
||||
}
|
||||
}
|
||||
|
||||
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
|
||||
hc := e.HTTPClient
|
||||
if hc == nil {
|
||||
|
||||
@@ -38,6 +38,9 @@ func (b *bundleWrapper) Run(_ context.Context, name string, args ...string) ([]b
|
||||
var plan map[string]any
|
||||
_ = json.Unmarshal(raw, &plan)
|
||||
b.plans = append(b.plans, plan)
|
||||
if plan["mode"] == "bundle-policy" { // R-1285: the pass after an install carries no bundle
|
||||
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":[]}}` + "\n"), nil, nil
|
||||
}
|
||||
body, _ := os.ReadFile(plan["bundle"].(string))
|
||||
b.bodies = append(b.bodies, body)
|
||||
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
|
||||
@@ -153,3 +156,27 @@ func TestBundleURL(t *testing.T) {
|
||||
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return b.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
// R-1285: right after an install the agent asks the wrapper now on disk for the appliance-only policy pass — the bundle
|
||||
// run itself was planned by the OLD wrapper (on a BYO host upgraded from 0.164.0 our SSH drop-in stayed one bundle
|
||||
// longer). A refused or failed bundle runs no pass.
|
||||
// RED-PROOF: drop the policyPass call → one plan only.
|
||||
func TestR1285_AnInstalledBundleIsFollowedByThePolicyPass(t *testing.T) {
|
||||
body := []byte(`{"format":1,"agent_version":"0.143.0","files":[]}`)
|
||||
srv, sha := serveBundle(t, body)
|
||||
w := &bundleWrapper{t: t, rep: `{"mode":"bundle","bundle":{"agent_version":"0.143.0","written":["/usr/local/sbin/felhom-os-apply"]}}`}
|
||||
e, _ := bundleExec(t, w, srv.URL)
|
||||
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(w.plans) != 2 || w.plans[1]["mode"] != "bundle-policy" || w.plans[1]["layer"] != "host" || w.plans[1]["signed"] != nil {
|
||||
t.Fatalf("after the install the agent must run the policy pass (host layer, no envelope): %v", w.plans)
|
||||
}
|
||||
|
||||
w2 := &bundleWrapper{t: t, rep: `{"mode":"bundle","refused":{"code":"R3","reason":"x"}}`}
|
||||
e2, _ := bundleExec(t, w2, srv.URL)
|
||||
_ = e2.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha))
|
||||
if len(w2.plans) != 1 {
|
||||
t.Fatalf("a refused bundle must run no policy pass: %v", w2.plans)
|
||||
}
|
||||
}
|
||||
@@ -123,7 +123,8 @@ type WrapperReport struct {
|
||||
Undo bool `json:"undo"`
|
||||
LiveRestore json.RawMessage `json:"live_restore"`
|
||||
Facts json.RawMessage `json:"facts"`
|
||||
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
|
||||
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
|
||||
BundlePolicy json.RawMessage `json:"bundle_policy"` // the appliance-only policy pass (R-1285, mode "bundle-policy")
|
||||
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
|
||||
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
|
||||
OOMCheck json.RawMessage `json:"oom_check"`
|
||||
|
||||
@@ -18,8 +18,14 @@ type Manager struct {
|
||||
logger *slog.Logger
|
||||
now func() time.Time // injectable for tests
|
||||
sleep func(context.Context, time.Duration)
|
||||
// onCommitted runs once, right after a successful commit (R-1285: the bundle policy pass of the NEW agent — in the
|
||||
// normal two-job delivery the bundle was installed by the OLD process, which has no such pass). nil = none.
|
||||
onCommitted func(context.Context)
|
||||
}
|
||||
|
||||
// SetOnCommitted wires the after-commit hook (R-1285). INIT-ONLY: call it before MaybeCommit's goroutine starts.
|
||||
func (m *Manager) SetOnCommitted(fn func(context.Context)) { m.onCommitted = fn }
|
||||
|
||||
// ManagerConfig wires the commit manager.
|
||||
type ManagerConfig struct {
|
||||
StateDir string
|
||||
@@ -119,4 +125,7 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
|
||||
return
|
||||
}
|
||||
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
|
||||
if m.onCommitted != nil {
|
||||
m.onCommitted(ctx)
|
||||
}
|
||||
}
|
||||
@@ -98,7 +98,8 @@ TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
|
||||
# THE retention number, read from the one file that owns it. A check and the policy it enforces
|
||||
# must read the same number from the same place, or they drift and the drift looks like a defect
|
||||
# in something else — which is exactly what happened on 2026-08-08/09 (R-287).
|
||||
RETENTION_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json")
|
||||
RETENTION_FILE = os.environ.get("FELHOM_RETENTION_POLICY") or \
|
||||
os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json") # the env: decoy tests only
|
||||
|
||||
|
||||
def retention_kept():
|
||||
@@ -114,6 +115,21 @@ def retention_kept():
|
||||
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
|
||||
return n
|
||||
|
||||
|
||||
def retention_pinned():
|
||||
"""D50 = A: the versions kept besides the newest N — the hub's MinAgent floor and the day-0 agent. Copies of hub
|
||||
state the vouching session writes into the policy file (CI holds no hub secret). Missing key = no pins."""
|
||||
with open(RETENTION_FILE, encoding="utf-8") as fh:
|
||||
pins = json.load(fh).get("pinned") or {}
|
||||
if not isinstance(pins, dict):
|
||||
raise ValueError("pinned must be an object, got %r" % (pins,))
|
||||
out = {}
|
||||
for name, v in pins.items():
|
||||
if not isinstance(v, str) or not re.match(r"^\d+\.\d+\.\d+$", v):
|
||||
raise ValueError("pinned %s must be X.Y.Z, got %r" % (name, v))
|
||||
out[name] = v
|
||||
return out
|
||||
|
||||
tried = []
|
||||
|
||||
|
||||
@@ -211,6 +227,7 @@ def main():
|
||||
all_versions = versions
|
||||
try:
|
||||
keep = retention_kept()
|
||||
pins = retention_pinned()
|
||||
except Exception as e:
|
||||
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
|
||||
|
||||
@@ -218,10 +235,15 @@ def main():
|
||||
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
|
||||
def _key(v):
|
||||
return tuple(int(x) for x in v.split("."))
|
||||
versions = sorted(all_versions, key=_key)[-keep:]
|
||||
window = sorted(all_versions, key=_key)[-keep:]
|
||||
# D50 = A: the pinned versions are asserted wherever they sit. A pin with no tag cannot be installed at all —
|
||||
# a failure, never a skip.
|
||||
unpinned_tags = sorted(("%s=%s" % (k, v)) for k, v in pins.items() if v not in all_versions)
|
||||
versions = sorted(set(window) | set(v for v in pins.values() if v in all_versions), key=_key)
|
||||
dropped = [v for v in all_versions if v not in versions]
|
||||
|
||||
print(" %d released version(s); retention policy keeps the newest %d" % (len(all_versions), keep))
|
||||
print(" %d released version(s); retention policy keeps the newest %d plus the pinned %s" % (
|
||||
len(all_versions), keep, ", ".join("%s=%s" % kv for kv in sorted(pins.items())) or "(none)"))
|
||||
print(" verifying: %s" % ", ".join(versions))
|
||||
if dropped:
|
||||
# NEVER silent. A bounded check that does not say what it stopped covering is how a
|
||||
@@ -232,6 +254,9 @@ def main():
|
||||
print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
|
||||
|
||||
bad = []
|
||||
for pin in unpinned_tags:
|
||||
bad.append((pin, ["the pinned version has NO git tag — nothing can install it (D50 = A)"]))
|
||||
print(" FAIL pinned %s: no git tag v%s" % (pin, pin.split("=")[1]))
|
||||
for v in versions:
|
||||
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
|
||||
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
|
||||
|
||||
@@ -1,41 +1,33 @@
|
||||
{
|
||||
"_comment": [
|
||||
"THE retention number for published agent artifacts. One file, read by everything that",
|
||||
"depends on it, because a check and the policy it enforces must read the same number from the",
|
||||
"same place or they drift — and the drift looks like a defect in something else.",
|
||||
"THE retention rule for published agent artifacts. One file, read by everything that depends on it, because a check",
|
||||
"and the policy it enforces must read the same rule from the same place or they drift — and the drift looks like a",
|
||||
"defect in something else (R-287, 2026-08-08/09; again R-1290, 2026-10-25).",
|
||||
"",
|
||||
"WHAT WENT WRONG WITHOUT IT (2026-08-08/09). The registry stopped serving felhom-agent",
|
||||
"0.120.0 and older, while scripts/check-published-versions.py demanded that EVERY git tag",
|
||||
"still be downloadable. Both rules are individually sensible; together they are impossible.",
|
||||
"CI went red at a commit whose own run had been green the day before, on a true finding that",
|
||||
"no one could act on. The red will return at the next publish unless the two read one number.",
|
||||
"D50 = A (operator ruling 2026-10-25, felhom.eu `09` §3 decision 262): the registry keeps the NEWEST 5 released agent",
|
||||
"versions, PLUS the hub's MinAgent floor, PLUS the version the hub's day-0 artifact manifest names. The operator prunes",
|
||||
"by hand to that rule (on 2026-10-25 he removed 0.159.0–0.160.0 and kept 0.158.1, the floor). CI asserts exactly these",
|
||||
"and never more. Before D50 the number was 10 (the R-267 newest-10 prune, R-287/R-291).",
|
||||
"",
|
||||
"HOW THE NUMBER WAS ARRIVED AT. generic_versions_kept is 10 because that is what the registry",
|
||||
"keeps: the deleter was ESTABLISHED on 2026-08-10 (R-287) — the newest-10 prune of generic packages",
|
||||
"run under R-267 (felhom-agent and felhom-golden generic held exactly 10 afterwards). Until then this",
|
||||
"file called the 10 an observed state with no located ruling; that is superseded (corrected",
|
||||
"2026-10-05, R-291). Container packages are not pruned by that rule.",
|
||||
"THE PINS ARE COPIES OF HUB STATE, because CI cannot read the hub without a secret it must not hold. The session that",
|
||||
"records an agent vouch (day0_agent) or declares a new MinAgent with a floor (hub_min_agent) updates `pinned` in the",
|
||||
"same commit (felhom-build-deploy skill, agent release). A pin that names a version with no tag is a FAILURE, never a",
|
||||
"skip — a pin nobody can install is the defect the pin exists to catch.",
|
||||
"",
|
||||
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest",
|
||||
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the",
|
||||
"operator should confirm or replace the number — at which point this file changes and both",
|
||||
"readers follow it in the same commit.",
|
||||
"SO THIS FILE IS A FLOOR, NOT A LICENCE: it says what CI may assume is still downloadable. It does not authorise",
|
||||
"deleting anything else.",
|
||||
"",
|
||||
"THE DEEPER BOUND, recorded so a future session does not have to re-derive it: the principled",
|
||||
"limit is the hub's vouched min_agent floor (0.127.0 on 2026-08-09). Nothing can install an",
|
||||
"agent below it — the hub refuses to vouch one and boxes update to the floor — so a released",
|
||||
"version below the floor being un-downloadable costs nothing real. Bounding on the floor would",
|
||||
"be better than bounding on a count, and it needs the gate to read the hub, which is network",
|
||||
"the gate does not have today. Filed as the follow-up in R-287.",
|
||||
"",
|
||||
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config",
|
||||
"files from raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that",
|
||||
"version at all — a strictly worse act than an un-downloadable binary."
|
||||
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config files from",
|
||||
"raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that version at all."
|
||||
],
|
||||
"generic_versions_kept": 10,
|
||||
"generic_versions_kept": 5,
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — bounds its assertion to the newest N versions"
|
||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||
],
|
||||
"recorded": "2026-08-09",
|
||||
"recorded_by": "CC 2026-08-09; the number's source (the R-267 newest-10 prune, established 2026-08-10 by R-287) recorded 2026-10-05 (R-291)"
|
||||
"recorded": "2026-10-27",
|
||||
"recorded_by": "CC 2026-10-27: day0_agent 0.167.0 (operator vouch 2026-10-27, read from the hub: version + agent and bundle sha256 match the release); earlier 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"pinned": {
|
||||
"hub_min_agent": "0.158.1",
|
||||
"day0_agent": "0.167.0"
|
||||
}
|
||||
}
|
||||
@@ -179,13 +179,24 @@ def published_cases(base):
|
||||
TAGS = ["v0.150.%d" % i for i in range(3)] # inside any retention window >= 3
|
||||
VERS = [t[1:] for t in TAGS]
|
||||
|
||||
# D50 = A: the gate also asserts the PINNED versions. The decoys run against their own policy file (the real pins
|
||||
# name versions the fake Gitea does not hold); PINS is what each case writes there.
|
||||
pol_dir = tempfile.mkdtemp(prefix="retention-decoy-")
|
||||
pol = os.path.join(pol_dir, "retention-policy.json")
|
||||
PINS = {}
|
||||
|
||||
def case(name, setup, expect_rc, must=()):
|
||||
FAKE.reset()
|
||||
FAKE.tags = list(TAGS)
|
||||
FAKE.packages = set(VERS)
|
||||
FAKE.raw = set(VERS)
|
||||
PINS.clear()
|
||||
setup()
|
||||
rc, out = run([sys.executable, gate], ROOT, child_env(base))
|
||||
with io.open(pol, "w", encoding="utf-8") as fh:
|
||||
json.dump({"generic_versions_kept": keep, "pinned": dict(PINS)}, fh)
|
||||
env = child_env(base)
|
||||
env["FELHOM_RETENTION_POLICY"] = pol
|
||||
rc, out = run([sys.executable, gate], ROOT, env)
|
||||
if not FAKE.hits:
|
||||
fails.append("published/%s: the gate never asked the fake Gitea - the seam is not wired" % name)
|
||||
report("published: " + name, rc, out, expect_rc, must)
|
||||
@@ -220,6 +231,25 @@ def published_cases(base):
|
||||
case("BY DESIGN: a version older than the retention window is not asserted", retired, 0,
|
||||
("NOT ASSERTED", "0.9.0"))
|
||||
|
||||
def pinned_old_ok():
|
||||
FAKE.tags = ["v0.8.0"] + ["v0.10.%d" % i for i in range(keep)]
|
||||
FAKE.packages = set(t[1:] for t in FAKE.tags)
|
||||
FAKE.raw = set(t[1:] for t in FAKE.tags)
|
||||
PINS["hub_min_agent"] = "0.8.0"
|
||||
case("GENUINE (D50): a pinned version older than the window is asserted and downloads", pinned_old_ok, 0,
|
||||
("verifying: 0.8.0", "ALL RELEASED VERSIONS INSTALLABLE"))
|
||||
|
||||
def pinned_old_gone():
|
||||
pinned_old_ok()
|
||||
FAKE.packages.discard("0.8.0")
|
||||
case("FACT (D50): a pinned version older than the window whose package 404s", pinned_old_gone, 1,
|
||||
("FAIL v0.8.0", "binary NOT downloadable"))
|
||||
|
||||
def pinned_no_tag():
|
||||
PINS["day0_agent"] = "0.150.9"
|
||||
case("FACT (D50): a pin that names a version with no tag", pinned_no_tag, 1,
|
||||
("FAIL pinned day0_agent=0.150.9", "no git tag"))
|
||||
|
||||
def five_hundred():
|
||||
FAKE.tags_status = 500
|
||||
case("INCONCLUSIVE: the tags api answers 500", five_hundred, 2, ("INCONCLUSIVE",))
|
||||
|
||||
Reference in new issue
Block a user