Compare commits

...
19 Commits
Author SHA1 Message Date
admin 91bd96c755 REPORT + CONTEXT: R-528 and R-1254 on main, not released (night 2026-10-28)
gates / gates (push) Successful in 35s
2026-10-29 02:51:00 +01:00
admin 6f933d6ee3 R-528: GET /guest/oom-kills — the kernel's per-container OOM counter of the token's guest, read on the host (no sudo)
gates / gates (push) Successful in 34s
Tests TestR528_* (fake cgroup tree), red-proved by mutation. Commit only; the agent release needs the vouch.
2026-10-28 23:22:40 +01:00
admin 5ddd124cfc R-1254 (ceremony half, HUNT-3 H2): after a failed escrow upload, ask the hub which package it holds before wording the result
gates / gates (push) Successful in 44s
The hub holds the new package -> success; another package -> 502 'previous code still works' (now true);
cannot be asked -> 504, no promise. Tests TestR1254_* red before the fix. Commit only; the agent release needs the vouch.
2026-10-28 22:51:08 +01:00
admin 3055bc54c1 D50 pin: day0_agent 0.167.0 (the operator's vouch 2026-10-27, read from the hub)
gates / gates (push) Successful in 1m4s
2026-10-27 20:45:25 +01:00
admin 007199342f v0.167.0 RELEASED (not vouched yet): CHANGELOG header, REPORT
gates / gates (push) Successful in 1m20s
2026-10-27 15:49:30 +01:00
admin 54d3a1057b CHANGELOG: the unreleased header names R-972 (Go 1.27.3)
gates / gates (push) Successful in 59s
2026-10-26 22:21:47 +01:00
admin fbbb168d1f build(R-972): Go 1.27.3 (GO-2026-6809, crypto/x509) — govulncheck 1 → 0 called
gates / gates (push) Successful in 1m22s
2026-10-26 20:34:25 +01:00
admin 3911cc6f52 fix(R-1232): the tier due-check counts only this box's archives (own key) — a reinstalled box's first whole-guest copy is not a week late
gates / gates (push) Successful in 1m20s
Red-proved (the old box's archive was counted). Commit only.
2026-10-26 20:15:39 +01:00
admin ad80ee749a docs: CONTEXT — R-1285 state and Tuesday's line test
gates / gates (push) Successful in 1m15s
2026-10-26 18:11:24 +01:00
admin 0683cc2dab REPORT: the 2026-10-26 afternoon run. Docs only.
gates / gates (push) Successful in 1m48s
2026-10-26 14:38:15 +01:00
admin b1bc63dd32 R-1285: the new agent runs the bundle policy pass once after its self-update commits — in the normal two-job delivery the old process installs the bundle. Test in the real order (commit, then one bundle-policy plan); red-proved. Commit only, not released.
gates / gates (push) Successful in 2m15s
2026-10-26 12:44:25 +01:00
admin e66a48f2b7 REPORT: the 2026-10-26 attended release run. Docs only.
gates / gates (push) Successful in 1m41s
2026-10-26 10:38:03 +01:00
admin b890b57540 v0.166.0 released and vouched by the operator (10:04:55 CET): CHANGELOG header (R-1280 live, R-1285 back to OPEN — the bundle job runs in the old process); D50 day0_agent pin → 0.166.0. Docs only.
gates / gates (push) Successful in 1m52s
2026-10-26 10:07:15 +01:00
admin aa69114d18 REPORT: the 2026-10-25 night — committed fixes, CI by hash, teardown. Docs only.
gates / gates (push) Successful in 1m43s
2026-10-25 23:54:07 +01:00
admin abcc1e6833 R-1285: after every bundle install the agent asks the wrapper now on disk for the appliance-only policy pass, so a BYO host upgraded from an older wrapper loses our SSH drop-in in one update. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:46:38 +01:00
admin d912307bb7 R-1280: a signed job's validity window is at most 24 h — the verifier, the root wrapper and felhom-opsign. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:42:18 +01:00
admin feaaa0e8ff D50 = A (R-1290): the published gate asserts the newest 5 released versions plus the pinned MinAgent (0.158.1) and day-0 agent (0.165.0); a pin with no tag fails; decoys red-proved. No binary change.
gates / gates (push) Successful in 1m41s
2026-10-25 20:20:30 +01:00
admin 6b6ba0c17f v0.165.0 vouched by the operator (18:38:49 CET); REPORT: release, delivery to four hosts, the BYO proof (removal one bundle late, felhom.eu R-1285). Docs only.
gates / gates (push) Failing after 1m49s
2026-10-25 19:02:48 +01:00
admin 4697f8ac9e v0.165.0: CHANGELOG names the release (R-1257 D43 = A, R-1250); released from a clean clone, sha256 e9d04a47…, bundle 8dea1f34…; not vouched
gates / gates (push) Successful in 1m38s
2026-10-25 17:00:02 +01:00
29 changed files with 1096 additions and 66 deletions

No files matched your search

+120 -9
View File
@@ -1,4 +1,115 @@
## Unreleased (next: v0.165.0) — D43 = A: BYO hosts keep their own SSH settings (R-1257) (2026-10-24 night) — committed, NOT released (the release + the operator's vouch are Monday's) ## Unreleased (2026-10-28 night) — R-528: GET /guest/oom-kills, the kernel's OOM counter of each of the guest's containers, read on the host — commit only; the next agent release (needs the operator's vouch)
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
could not fire. The kernel counts every kill per container cgroup, and the host can read it.
- **`internal/localapi/guest_oom.go`:** `GET /guest/oom-kills` (withGuest — the token's guest only; a cross-guest
`?vmid=` is 403) reads `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex>.scope/memory.events` (and
`ns/docker/<id>/` for the cgroupfs driver), line `oom_kill N`, and answers `{vmid, available, containers: {id: n}}`.
A guest with no cgroup folder → `available: false`. Read-only, no sudo: the files are 0444 and the non-root
felhom-agent user reads them (measured on Tester 1's host 2026-10-28). A container's scope is new at each start, so the
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
## Unreleased (2026-10-28 night) — R-1254 (the ceremony half, review round 2 HUNT-3 H2): the agent asks the hub which package it holds before it words a failed upload — commit only; the next agent release (needs the operator's vouch)
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
hub's current package opens only with the NEW code.
- **Now (`internal/localapi/escrow_ceremony.go`):** after a failed upload the agent asks the hub (`HubHolds`, 15 s,
outside the lock, the slot still marked claiming): the hub holds THIS package → a success (the code is handed out /
the confirm completes; a WARN line names the lost answer); the hub holds another package → 502 and the old sentence,
now true; the hub cannot be asked → **504** with a sentence that promises nothing about either code, and the wait goes
on (a retry decides). `cmd/felhom-agent`: `HubHolds` compares the pending package's identity blob with the one the hub
serves back (`GET /api/v1/hosts/<id>/escrow`, the read-back the agent already uses; ciphertext only, nothing logged);
a package without an identity blob cannot be compared → unknown. Tests `TestR1254_*` (localapi, red before the fix:
502 while the hub held the new package) and `TestR1254_HubHoldsComparesTheIdentityBlob` (red-proved by mutation).
- Needs controller ≥ the R-1254 controller commit (2026-10-28 night) to word 504 for the household; an older controller
shows its generic „A kód lekérése nem sikerült." (claim) / its 502 sentence (confirm) for it.
## v0.167.0 (2026-10-27) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — RELEASED (release-agent.sh from a clean clone; agent sha256 28bdd282…d826f0, bundle 9d034416…ed1268; signed updates committed on demo-felhom, Tester 1, demo-hp; VOUCHED by the operator 2026-10-27, Day-0 pin set)
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
newest. Full `go build/vet/test` green. The controller and hub move the same night.
- **R-1232 (P2):** `backup.BackupRunner.NewestArchiveTime` — the tier due-check — counts only archives written with THIS
storage's key on an encrypted storage (the R-727 discriminator; the archive carries no host id). Before: a reinstalled
box shares the customer's PBS namespace and guest number with the old box, the old box's archive read as „the last
backup", and the whole-guest tier waited up to its 168-h cadence (the 2026-10-23 test night). A key lookup failure is
an error (the due-check reads UNKNOWN), never a foreign age. Tests `TestR1232_*` (foreign only → none; own → its time;
an unencrypted dir → unfiltered), red-proved (case 1 returned the old box's time).
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
second later — so 0.166.0's pass never ran on the bundle that delivered it.
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
agent-only — holding the bundle job until the commit would need new hub queue logic.
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
removed → commit only, no pass — the demo-hp symptom).
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
0.165.0 box passes even without this fix.
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
„BUNDLE POLICY DONE install_mode=appliance removed=0".
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
drop-in until its next bundle. A change inside the wrapper alone cannot fix the run that delivers it.
- **The fix — the new AGENT asks the new wrapper:** `internal/osupdate/bundle.go` runs a policy pass right after every
bundle install (`mode: "bundle-policy"`, host layer, no envelope); the wrapper on disk is the bundle's own by then.
`configs/felhom-os-apply` `Bundle.policy_pass`: on a root-owned BYO record it removes the appliance-only files WE
wrote (our header), keeps the previous copy beside the bundle's, `sshd -t` then reloads ssh — a failing `sshd -t` puts
the file back. It writes nothing, so it needs no signature; an appliance and the owner's own file are left alone. Never
fatal for the agent: an older wrapper refuses the unknown mode and the next bundle does it. Report key `bundle_policy`.
- Tests: `TestR1285_AnInstalledBundleIsFollowedByThePolicyPass` (a refused bundle runs no pass);
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
`documentation/audits/night-2026-10-25/redproof-r1285.txt`.
### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
usable that long, and the nonce store no longer keeps its record that long.
- `configs/felhom-os-apply`: the root wrapper holds the same 24 h (`MAX_WINDOW_S`, refusal R3 „the signed job's window
is longer than 24 h").
- `cmd/felhom-opsign`: `-ttl` above 24 h (or ≤ 0) is refused before anything is signed.
- Tests: `TestVerify_RejectsPerStage/window_longer_than_24_h` and `…/window_of_exactly_24_h_is_accepted`;
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
(`hub_min_agent` 0.158.1, `day0_agent` 0.165.0 — copies of hub state, because CI holds no hub secret; the session that
records a vouch or a new MinAgent updates them in the same commit). `check-published-versions.py` asserts the window
plus every pin; a pin with no tag FAILS. CI was red since the operator's hand prune of 0.159.0–0.160.0 (job 2334).
- **Decoys** (`test_gate_decoys.py`, via `FELHOM_RETENTION_POLICY`, test only): a pinned version older than the window is
asserted; its 404 convicts; a pin with no tag convicts. Seen failing on the old gate (two LIVE HOLES): felhom.eu
`documentation/audits/night-2026-10-25/redproof-d50.txt`.
## v0.165.0 (2026-10-25) — D43 = A: BYO hosts keep their own SSH settings (R-1257); the line test for a box that seeds all day (R-1250) — released in the attended release run; **VOUCHED by the operator 2026-10-25 18:38:49 CET** (binary `e9d04a47…`, bundle `8dea1f34…`)
**Found at the release (felhom.eu R-1285, P4):** on a BYO host upgraded from 0.164.0 the drop-in is removed only on the bundle run AFTER the one that installs this wrapper — the first run is planned by the old wrapper. Proven on a BYO test host (VM 9371 on demo-hp): the second `agent_config_update` logged `BUNDLE REMOVED … (ours, on a BYO host — D43)`.
### BYO hosts keep their own SSH settings (R-1257, D43 = A) — 2026-10-24 night
- **The root-key-only sshd drop-in (`/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`, R-982 / D4) goes only to an - **The root-key-only sshd drop-in (`/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`, R-982 / D4) goes only to an
appliance** — operator ruling D43 = A (2026-10-24): a BYO host belongs to its owner (`11` §1). The table entry's policy appliance** — operator ruling D43 = A (2026-10-24): a BYO host belongs to its owner (`11` §1). The table entry's policy
@@ -17,14 +128,7 @@
is; a LAN `Match` block is not in force; every address key-only passes; a failing `sshd -t` undoes only the SSH step. is; a LAN `Match` block is not in force; every address key-only passes; a failing `sshd -t` undoes only the SSH step.
The fake box now carries an appliance record by default (every earlier test unchanged). The fake box now carries an appliance record by default (every earlier test unchanged).
## 2026-10-24 (review round 2, unattended) — `.claude/rules/unprompted-work.md` aligned with its four siblings ### The line test for a box that seeds all day (R-1250) — 2026-10-24
- The copy lacked item 6's paragraph „A version number is pushed once …" that the felhom.eu and felhom-controller copies
have carried since 2026-10-15; the file's header says „change all five or none". The paragraph states a fact
(`felhom-controller/controller/build.sh` refuses an existing tag and an unreachable registry). Now byte-identical to
the other four (md5 b1891bd4). No code change. Record: felhom.eu `documentation/audits/review-round-2-2026-10-24/EDITS.md`.
## Unreleased (next: v0.165.0) — the line test for a box that seeds all day (R-1250) (2026-10-24) — committed, not released
Attended brief 2026-10-24, Part 4 (commit only; the release rides the next normal train). The weekly test (v0.164.0) Attended brief 2026-10-24, Part 4 (commit only; the release rides the next normal train). The weekly test (v0.164.0)
starts only after 10 quiet minutes; a torrent app that seeds all day never gives them, so that box never got a tested starts only after 10 quiet minutes; a torrent app that seeds all day never gives them, so that box never got a tested
@@ -47,6 +151,13 @@ line and qBittorrent never got its 50 % limit. Now:
**No sudoers or config-bundle change. MinController: none** — an older controller never calls the route. **No sudoers or config-bundle change. MinController: none** — an older controller never calls the route.
## 2026-10-24 (review round 2, unattended) — `.claude/rules/unprompted-work.md` aligned with its four siblings
- The copy lacked item 6's paragraph „A version number is pushed once …" that the felhom.eu and felhom-controller copies
have carried since 2026-10-15; the file's header says „change all five or none". The paragraph states a fact
(`felhom-controller/controller/build.sh` refuses an existing tag and an unreachable registry). Now byte-identical to
the other four (md5 b1891bd4). No code change. Record: felhom.eu `documentation/audits/review-round-2-2026-10-24/EDITS.md`.
## v0.164.0 — the weekly upload line test (R-1247) (2026-10-24) — RELEASED on the operator's ruling (attended release run); vouched by the operator 09:44:58 CEST ## v0.164.0 — the weekly upload line test (R-1247) (2026-10-24) — RELEASED on the operator's ruling (attended release run); vouched by the operator 09:44:58 CEST
Binary sha256 `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`; config bundle sha256 Binary sha256 `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`; config bundle sha256
+11
View File
@@ -1,5 +1,16 @@
# CONTEXT — felhom-agent working state # CONTEXT — felhom-agent working state
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
> on demo-hp (felhom.eu R-1250, read, not forced).
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd > **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written, > drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is > our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
+28 -14
View File
@@ -1,16 +1,30 @@
# REPORT — D43 = A: BYO hosts keep their own SSH settings (R-1257), 2026-10-24 night (unattended; commit only) # REPORT — two agent fixes on main, NOT released (night 2026-10-28 → 29, unattended)
Brief: the restore-holes run, Part 4. Starting the run was the operator's ruling **D43 = A**. Previous report (R-1250's **Plain words first.** Two changes are on main and tested. Neither is released: an agent release needs the operator's
line hold, attended, same day): `git show f5c3d77:REPORT.md`. vouch, and the night brief allowed no release. Both go on the morning list.
- **`configs/felhom-os-apply` (commit `c8d3c17`):** the root-key-only sshd drop-in is an `appliance`-policy file — ## What changed
written only when the root-owned install record says `appliance`; on `byo` nothing is written and a copy we wrote
(exact path, our header line) is removed (the previous copy kept for the undo, then `sshd -t` and a reload); a box - **R-528 — the OOM counter (`6f933d6`, CI run id 2473 / #284 green).** New local-API route `GET /guest/oom-kills`. It
whose record proves nothing is left as it is. **X3:** the in-force check asks `sshd -T` for each of the host's own reads the kernel's `oom_kill` count of each Docker container of the token's own guest, on the host, from
global IPv4 addresses. **X4:** a failing `sshd -t` puts back only the SSH file (`sshd_step_undone` in the report); `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events` (mode 0444, readable by the non-root
the rest of the bundle stays. agent user). Docker inside an LXC guest does not see these kills (`OOMKilled` stays false). The controller reads the
- **Tests:** `configs/test_felhom_config_bundle.py` class `ByoKeepsItsSSH` (7), each rule seen failing with it taken out route every 30 s (controller `eed23e7`). Tests `TestR528_*`, red first by mutation (another guest's container served;
(policy back to `replace`; the probe back to 192.0.2.1; `sshd -t` back to `must`). The fake box now carries an `oom` parsed for `oom_kill`).
appliance record by default; all 68 bundle tests and 146 os-apply tests pass; `agent_gates.py --fast` OK. - **R-1254 — the escrow ceremony half (`5ddd124`, CI run id 2471 / #283 green).** After an escrow upload that fails, the
- **Not done here (Monday):** the agent release v0.165.0 and the operator's vouch; a signed bundle on a BYO test host. agent asks the hub which package it holds (the identity blob compared): the new one → success; another one → 502 and
- Docs: felhom.eu `03` (the D4 paragraph) and `11` §1 carry the ruling; register R-1257 → VERIFY. „the previous code still works"; no answer → 504, which promises nothing. Tests `TestR1254_*`, red first.
## Live proof
- R-528, 2026-10-28 23:27 UTC on Tester 1 with a test build of this agent (put back after: 0.167.0, sha256 `28bdd282…`).
A Radicale capped at 128 MB, a memory hog inside it killed: the host counter 0 → 1, one `app_oom` „seen by
docker,cgroup", one mail to the operator with `"source":"docker,cgroup"`. Evidence: felhom.eu
`documentation/audits/night-2026-10-28/r528-oom/`.
- R-1254: no live proof. It needs a hub answer lost on purpose; that is a hub change, and no hub deploy runs at night.
## Next
The agent release (the operator's vouch), then `pinned.day0_agent` in `scripts/retention-policy.json` (D50).
The controller needs no MinAgent change for R-528: an old agent answers 404, and the controller then waits 10 minutes
before it asks again.
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
// passes as held → FAILS.
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
body := func(id string) []byte {
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
return b
}
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
if err != nil {
return nil, err
}
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
}
}
ctx := context.Background()
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
}
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
}
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
t.Fatal("a package without an identity blob cannot be compared — an error")
}
}
+43
View File
@@ -1216,6 +1216,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} }
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ",")) logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
}} }}
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound // v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
// to the local API server built below). // to the local API server built below).
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner, internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
@@ -1247,6 +1250,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root // R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
// subprocess used, under the same per-host key, before the code is handed out. // subprocess used, under the same per-host key, before the code is handed out.
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) }, Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
// R-1254 H2: after a failed upload, the hub is asked which package it holds before the household is told
// anything — the same per-host GET the identity read-back uses.
HubHolds: func(ctx context.Context, pkg []byte) (bool, error) {
if client == nil {
return false, fmt.Errorf("hub client not configured")
}
return hubHoldsEscrowPackage(ctx, client.FetchIdentityEscrow, pkg)
},
DRConfigured: func() bool { DRConfigured: func() bool {
if drConfigured != nil { if drConfigured != nil {
return drConfigured() return drConfigured()
@@ -3363,6 +3374,32 @@ func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the // putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978). // text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
// hubHoldsEscrowPackage (R-1254 H2) reports whether the hub's CURRENT package is the upload body pkg, by its identity
// blob — the part the hub serves back (GET /api/v1/hosts/<id>/escrow). Ciphertext only; nothing is logged. A package
// without an identity blob cannot be compared: an error (the caller then promises nothing).
func hubHoldsEscrowPackage(ctx context.Context, fetch func(context.Context) (*hub.IdentityEscrowResponse, error), pkg []byte) (bool, error) {
var up escrowUploadRequest
if err := json.Unmarshal(pkg, &up); err != nil {
return false, fmt.Errorf("reading the pending package: %w", err)
}
mine, err := base64.StdEncoding.DecodeString(up.IdentityBlobB64)
if err != nil || len(mine) == 0 {
return false, fmt.Errorf("the pending package carries no identity blob to compare")
}
resp, err := fetch(ctx)
if err != nil {
return false, err
}
if !resp.Present || resp.IdentityEscrowB64 == "" {
return false, nil
}
theirs, err := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
if err != nil {
return false, fmt.Errorf("decoding the hub's package: %w", err)
}
return bytes.Equal(mine, theirs), nil
}
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error { func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" { if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)") return fmt.Errorf("hub not configured (url/host_id/api_key)")
@@ -4040,3 +4077,9 @@ func agentStateDir(cfg config.Config) string {
} }
return "/var/lib/felhom-agent" return "/var/lib/felhom-agent"
} }
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
}
@@ -0,0 +1,91 @@
package main
import (
"context"
"encoding/json"
"io"
"log/slog"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
)
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
type orderRunner struct {
mu sync.Mutex
order []string
}
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
var p map[string]any
b, _ := os.ReadFile(args[1])
_ = json.Unmarshal(b, &p)
mode, _ := p["mode"].(string)
r.order = append(r.order, "os-apply:"+mode)
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
}
if len(args) >= 1 {
r.order = append(r.order, filepath.Base(name)+":"+args[0])
}
return []byte("ok"), nil, nil
}
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
t.Helper()
stateDir := t.TempDir()
if marker != nil {
dir := filepath.Join(stateDir, "selfupdate")
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
b, _ := json.Marshal(marker)
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
t.Fatal(err)
}
}
r := &orderRunner{}
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
mgr.MaybeCommit(context.Background())
r.mu.Lock()
defer r.mu.Unlock()
return append([]string(nil), r.order...)
}
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
}
}
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
}
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 0 {
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
}
}
+4
View File
@@ -123,6 +123,10 @@ func run() error {
now := time.Now().UTC() now := time.Now().UTC()
issued := now issued := now
if *ttl <= 0 || *ttl > authz.MaxWindow {
fmt.Fprintf(os.Stderr, "felhom-opsign: -ttl %s refused: a box accepts a window of at most %s (R-1280)\n", *ttl, authz.MaxWindow)
os.Exit(2)
}
expires := now.Add(*ttl) expires := now.Add(*ttl)
// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over. // Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
+48 -1
View File
@@ -187,6 +187,8 @@ BUNDLE_PREV_DIR = "/var/lib/felhom-os-apply/bundle-prev" # the previous copies
BUNDLE_KEEP = 3 BUNDLE_KEEP = 3
BUNDLE_MAX_BYTES = 4 * 1024 * 1024 BUNDLE_MAX_BYTES = 4 * 1024 * 1024
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$") BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
# R-1280: the longest validity window a signed job may carry — the agent's authz.MaxWindow, the same 24 h.
MAX_WINDOW_S = 24 * 3600
OOB_DIR = "/etc/felhom-sshd" OOB_DIR = "/etc/felhom-sshd"
# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back). # R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf" SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
@@ -489,7 +491,8 @@ class Apply:
def check_plan(self, plan): def check_plan(self, plan):
mode = plan.get("mode", "apply") mode = plan.get("mode", "apply")
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \ if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
"oom-check") \
+ KERNEL_MODES: + KERNEL_MODES:
raise Refused("R11", f"unknown mode {mode!r}") raise Refused("R11", f"unknown mode {mode!r}")
if mode == "agent_update": if mode == "agent_update":
@@ -500,6 +503,10 @@ class Apply:
if plan.get("layer") != "host": if plan.get("layer") != "host":
raise Refused("R11", "bundle is a host-layer mode") raise Refused("R11", "bundle is a host-layer mode")
return mode, "host", 0, "bundle" return mode, "host", 0, "bundle"
if mode == "bundle-policy":
if plan.get("layer") != "host":
raise Refused("R11", "bundle-policy is a host-layer mode")
return mode, "host", 0, "bundle-policy"
layer = plan.get("layer") layer = plan.get("layer")
if layer not in ("guest", "host", "docker", "pve", "kernel"): if layer not in ("guest", "host", "docker", "pve", "kernel"):
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel") raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
@@ -661,6 +668,8 @@ class Apply:
iss = calendar.timegm(time.strptime(op["issued_at"], "%Y-%m-%dT%H:%M:%SZ")) iss = calendar.timegm(time.strptime(op["issued_at"], "%Y-%m-%dT%H:%M:%SZ"))
except (KeyError, ValueError, TypeError): except (KeyError, ValueError, TypeError):
raise Refused("R3", "the signed job has no readable time window") raise Refused("R3", "the signed job has no readable time window")
if exp - iss > MAX_WINDOW_S:
raise Refused("R3", "the signed job's window is longer than 24 h (R-1280)")
if now > exp or now < iss - 120: if now > exp or now < iss - 120:
raise Refused("R3", "the signed job is expired or not yet valid") raise Refused("R3", "the signed job is expired or not yet valid")
nonce = op.get("nonce") nonce = op.get("nonce")
@@ -1070,6 +1079,8 @@ class Apply:
return self.facts() return self.facts()
if self.mode == "bundle": if self.mode == "bundle":
return Bundle(self).from_plan(plan) return Bundle(self).from_plan(plan)
if self.mode == "bundle-policy":
return Bundle(self).policy_pass()
if self.mode == "agent_update": if self.mode == "agent_update":
return self.agent_update(plan) return self.agent_update(plan)
if self.layer in ("host", "pve", "kernel"): if self.layer in ("host", "pve", "kernel"):
@@ -2060,6 +2071,42 @@ class Bundle:
ver, files = self.parse(data, sha) ver, files = self.parse(data, sha)
return self.install(ver, sha, files, "installer", False) return self.install(ver, sha, files, "installer", False)
def policy_pass(self):
"""R-1285 (D43 = A): the appliance-only files' policy, applied by THIS wrapper. A bundle that brings a new wrapper
is planned by the OLD one, which did not know the policy — so on a BYO host upgraded from 0.164.0 our SSH drop-in
stayed until the next bundle. The agent runs this pass right after every bundle install (the wrapper on disk is
then the new one). It only REMOVES a file WE wrote (our header) from a box whose root-owned install record says
byo, keeps the previous copy beside the bundle's, and reloads sshd; it writes nothing, so it needs no signature.
A failing `sshd -t` puts the file back (R-1257 X4's rule). Report key `bundle_policy`."""
log = self.r.log
install_mode = self.install_mode()
removed = []
if install_mode == "byo":
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
prev = os.path.join(BUNDLE_PREV_DIR, f"{stamp}-policy")
for dest, _src, _mode, _check, policy in BUNDLE_FILES:
if policy != "appliance":
continue
old, old_mode, _ = self.live(dest)
if old is None or not old.startswith(SSHD_DROPIN_HEADER):
continue
self.r.put_file(prev + dest, old, 0o600)
self.r.remove(dest)
removed.append((dest, old, old_mode))
log(f"os-apply: BUNDLE POLICY REMOVED {dest} (ours, on a BYO host — D43, R-1285)")
if any(d == SSHD_DROPIN for d, _, _ in removed):
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
if rc != 0:
for d, old, old_mode in removed:
self.r.put_file(d, old, old_mode)
log(f"os-apply: BUNDLE POLICY WARNING sshd -t failed without our drop-in (rc={rc}) — put back")
removed = []
else:
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
self.report["bundle_policy"] = {"install_mode": install_mode, "removed": [d for d, _, _ in removed]}
log(f"os-apply: BUNDLE POLICY DONE install_mode={install_mode} removed={len(removed)}")
return 0
# ---------- install, self-check, undo ---------- # ---------- install, self-check, undo ----------
def install(self, ver, sha, files, authority, bootstrap): def install(self, ver, sha, files, authority, bootstrap):
log = self.r.log log = self.r.log
+41
View File
@@ -588,6 +588,47 @@ class ByoKeepsItsSSH(unittest.TestCase):
return orig(argv, timeout, stdin) return orig(argv, timeout, stdin)
box.host = host box.host = host
# R-1285: the policy pass the agent runs right after every bundle install, with the wrapper NOW on disk. No
# signature (it only removes a file we wrote). RED-PROOF: make policy_pass return before the loop → our copy stays.
POLICY_PLAN = {"release_id": "bundle-policy-0.166.0", "layer": "host", "mode": "bundle-policy"}
def test_the_policy_pass_removes_our_copy_from_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644) # the old wrapper's bundle run left it
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertEqual(rep["bundle_policy"], {"install_mode": "byo", "removed": [self.DEST]}, rep)
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files))
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
def test_the_policy_pass_leaves_an_appliance_and_the_owners_file(self):
for mode, content in (("appliance", None), ("byo", b"# my own\nPermitRootLogin yes\n")):
data, sha = real_bundle()
box = Box(data, None, install_mode=mode)
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
body = content if content is not None else self.OURS
box.put(self.DEST, body, 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), body, mode)
self.assertEqual(rep["bundle_policy"]["removed"], [], mode)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, mode)
def test_the_policy_pass_puts_our_copy_back_when_sshd_t_fails(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644)
self._fake(box, sshd_t_rc=1)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), self.OURS)
self.assertEqual(rep["bundle_policy"]["removed"], [])
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_match_block_for_the_lan_is_not_in_force(self): def test_a_match_block_for_the_lan_is_not_in_force(self):
data, sha = real_bundle() data, sha = real_bundle()
box = Box(data, signed_job(sha)) box = Box(data, signed_job(sha))
+8
View File
@@ -833,6 +833,14 @@ class DockerLane(unittest.TestCase):
def test_expired_signed_job_is_refused(self): def test_expired_signed_job_is_refused(self):
self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3") self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3")
def test_a_window_longer_than_24h_is_refused(self):
# R-1280. RED-PROOF: drop the MAX_WINDOW_S check → the job (current, signed, 24 h + 1 s) runs.
self.refused(docker_fake(signed=signed_job(issued="2026-10-03T11:59:59Z", expires="2026-10-04T12:00:00Z")), "R3")
def test_a_window_of_24h_is_accepted(self):
rc, rep = run(docker_fake(signed=signed_job(issued="2026-10-03T12:30:00Z", expires="2026-10-04T12:30:00Z")))
self.assertNotEqual((rep.get("refused") or {}).get("code"), "R3", rep)
def test_replayed_signed_job_is_refused(self): def test_replayed_signed_job_is_refused(self):
f = docker_fake(signed=signed_job()) f = docker_fake(signed=signed_job())
f.nonces = {"n1": f.clock + 600} f.nonces = {"n1": f.clock + 600}
+1 -1
View File
@@ -1,6 +1,6 @@
module gitea.dooplex.hu/admin/felhom-agent module gitea.dooplex.hu/admin/felhom-agent
go 1.27.2 go 1.27.3
require ( require (
golang.org/x/crypto v0.58.0 golang.org/x/crypto v0.58.0
+2
View File
@@ -22,6 +22,8 @@ var (
ErrExpired = errors.New("authz: op expired") ErrExpired = errors.New("authz: op expired")
// ErrNotYetValid: now < issued_at (minus clock-skew tolerance). // ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
ErrNotYetValid = errors.New("authz: op not yet valid") ErrNotYetValid = errors.New("authz: op not yet valid")
// ErrWindowTooLong: expires_at - issued_at > MaxWindow (R-1280).
ErrWindowTooLong = errors.New("authz: op validity window too long")
// ErrReplay: the nonce was already recorded in the window. // ErrReplay: the nonce was already recorded in the window.
ErrReplay = errors.New("authz: replay (nonce already seen)") ErrReplay = errors.New("authz: replay (nonce already seen)")
) )
+8
View File
@@ -20,6 +20,11 @@ const Namespace = "felhom-op-v1"
// the validity window stays an honest upper bound. // the validity window stays an honest upper bound.
const DefaultClockSkew = 2 * time.Minute const DefaultClockSkew = 2 * time.Minute
// MaxWindow is the longest validity window a signed op may carry (R-1280, D44 #22, operator 2026-10-25): a job signed
// with a long window by mistake would stay usable that long, and the nonce store would keep its record as long. The
// root wrapper (configs/felhom-os-apply, MAX_WINDOW_S) and felhom-opsign (-ttl) hold the same 24 h.
const MaxWindow = 24 * time.Hour
// KeyRole tags a pinned operator key (doc 04 §3 two-key model). // KeyRole tags a pinned operator key (doc 04 §3 two-key model).
type KeyRole string type KeyRole string
@@ -150,6 +155,9 @@ func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
if now.Before(op.IssuedAt.Add(-v.ClockSkew)) { if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now) return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
} }
if w := op.ExpiresAt.Sub(op.IssuedAt); w > MaxWindow {
return nil, fmt.Errorf("%w: issued_at=%s expires_at=%s (%s, the most is %s)", ErrWindowTooLong, op.IssuedAt, op.ExpiresAt, w, MaxWindow)
}
if now.After(op.ExpiresAt) { if now.After(op.ExpiresAt) {
return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now) return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
} }
+20
View File
@@ -117,6 +117,26 @@ func TestVerify_RejectsPerStage(t *testing.T) {
rejects(t, err, ErrTarget) rejects(t, err, ErrTarget)
}) })
// R-1280: a window longer than 24 h is refused even while it is current; exactly 24 h is accepted.
// RED-PROOF: drop the MaxWindow check → the 24 h + 1 s case verifies.
t.Run("window longer than 24 h", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-1", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow+time.Second))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
_, err := v.Verify(blob, sig)
rejects(t, err, ErrWindowTooLong)
})
t.Run("window of exactly 24 h is accepted", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-2", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
if _, err := v.Verify(blob, sig); err != nil {
t.Fatalf("a 24 h window must verify: %v", err)
}
})
t.Run("expired", func(t *testing.T) { t.Run("expired", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}", blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
refNow.Add(-2*time.Hour), refNow.Add(-time.Hour)) refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
@@ -0,0 +1,55 @@
package backup
import (
"context"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1232 — the 2026-10-23 test night: a reinstalled box (new install, new PBS key, same customer namespace, same guest
// number 9201) found the OLD box's archive (2.6 days old) when the due-check asked „when did this tier last land?", so
// the whole-guest tier read „not due" for up to its 168-h cadence and the ep0 copy stayed the OLD box's. The due-check
// now counts only archives written with THIS storage's key (the R-727 discriminator; the archive carries no host id).
// The CONSEQUENCE asserted: with only a foreign archive the answer is „none found" (the tier is due); with the box's
// own archive, that archive's time. COMPANION RED-PROOF: drop the key filter in NewestArchiveTime → case 1 returns the
// old box's time.
func TestR1232_TheDueCheckCountsOnlyThisBoxsArchives(t *testing.T) {
oldBox := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-20T21:10:00Z", Content: "backup", VMID: 9201,
Size: 3490689830, CTime: time.Date(2026, 10, 20, 21, 10, 0, 0, time.UTC).Unix(), Encrypted: earlierBox1}
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbsdr", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{oldBox},
}
r := NewBackupRunner(api, "felhom-pbsdr", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
// 1. The fresh box has written nothing yet: the old box's archive must not count.
when, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatal(err)
}
if found {
t.Fatalf("the due-check counted ANOTHER box's archive (%s) as this tier's last backup — the tier would wait a week", when)
}
// 2. After the box's own first copy, that copy is the answer.
own := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-24T02:30:00Z", Content: "backup", VMID: 9201,
Size: 3600000000, CTime: time.Date(2026, 10, 24, 2, 30, 0, 0, time.UTC).Unix(), Encrypted: thisBoxKey}
api.content = append(api.content, own)
when, found, err = r.NewestArchiveTime(context.Background(), 9201)
if err != nil || !found || when.Unix() != own.CTime {
t.Fatalf("want this box's own archive time %d, got %v found=%v err=%v", own.CTime, when, found, err)
}
}
// An unencrypted storage (a local dir) holds only this box's vzdumps: no key filter, the newest counts.
func TestR1232_UnencryptedStorageCountsEveryArchive(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_10_25-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1793050025}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, found, err := r.NewestArchiveTime(context.Background(), 9201); err != nil || !found {
t.Fatalf("an unencrypted local archive was not counted: found=%v err=%v", found, err)
}
}
+12
View File
@@ -609,11 +609,23 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
if err != nil { if err != nil {
return time.Time{}, false, err return time.Time{}, false, err
} }
// R-1232: on an ENCRYPTED storage only archives written with THIS storage's key are this box's (the R-727
// discriminator — the archive carries no host id). A reinstalled box shares the customer's namespace and guest
// number with the old box, and the old box's archive made the tier read „not due" for a whole cadence (168 h).
// A key lookup failure is returned: the due-check reads UNKNOWN, never a foreign archive's age.
ownKey, err := r.storageKeyFingerprint(ctx, r.target)
if err != nil {
return time.Time{}, false, fmt.Errorf("reading the key fingerprint of storage %s: %w", r.target, err)
}
var best int64 = -1 var best int64 = -1
for _, e := range contents { for _, e := range contents {
if e.Content != "backup" || e.VMID != vmid { if e.Content != "backup" || e.VMID != vmid {
continue continue
} }
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this tier's last backup (R-1232)", shortFP(e.Encrypted), shortFP(ownKey)))
continue
}
if ok, why := archivePlausiblyComplete(e); !ok { if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why) r.warnRejectedArchiveOnce(e, why)
continue continue
+76 -2
View File
@@ -72,12 +72,56 @@ type EscrowCeremonyConfig struct {
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of // host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
// a pending package is refused (503): a code must never be handed out for a package the hub lacks. // a pending package is refused (503): a code must never be handed out for a package the hub lacks.
Upload func(ctx context.Context, pkg []byte) error Upload func(ctx context.Context, pkg []byte) error
// HubHolds (R-1254 H2) asks the hub whether its CURRENT package is exactly pkg — asked after a failed upload,
// whose answer may have been lost AFTER the hub stored it. nil or an error → the outcome is unknown.
HubHolds func(ctx context.Context, pkg []byte) (bool, error)
} }
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole // escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
// ceremony incl. its upload was measured at ~2.4 s). // ceremony incl. its upload was measured at ~2.4 s).
const escrowUploadTimeout = 30 * time.Second const escrowUploadTimeout = 30 * time.Second
// escrowHoldTimeout bounds the hub question after a failed upload (R-1254 H2).
const escrowHoldTimeout = 15 * time.Second
// uploadOutcome is what a claim or confirm upload did, as the HUB tells it (R-1254 H2): an upload whose answer was lost
// or late may have been stored all the same.
type uploadOutcome int
const (
uploadStored uploadOutcome = iota // the hub holds the new package
uploadNotStored // the hub answered and holds another package: the previous code still works
uploadUnknown // the hub cannot be asked: nothing may be promised about either code
)
// escrowUnknownOutcomeMsg is the answer when the hub cannot say which package it holds — never „the previous code
// still works", which is false when the lost answer came after the store.
const escrowUnknownOutcomeMsg = "the hub did not answer in time — it is not known whether it stored the new recovery package; try again (until a try succeeds, do not rely on the previous code alone)"
// outcomeAfterFailedUpload asks the hub whether it holds pkg after Upload returned an error. Called with the slot
// marked claiming (escrowClaiming), outside escrowMu — it is a network call.
func (s *Server) outcomeAfterFailedUpload(pkg []byte) uploadOutcome {
if s.escrowCeremony.HubHolds == nil {
return uploadUnknown
}
base := s.baseCtx
if base == nil {
base = context.Background()
}
ctx, cancel := context.WithTimeout(base, escrowHoldTimeout)
defer cancel()
held, err := s.escrowCeremony.HubHolds(ctx, pkg)
switch {
case err != nil:
s.logger.Warn("local-api: escrow — the hub cannot say which package it holds after a failed upload (R-1254)", "err", err)
return uploadUnknown
case held:
return uploadStored
default:
return uploadNotStored
}
}
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until // ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned // parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
// spike-shaped output; production is runCeremonySubprocess. // spike-shaped output; production is runCeremonySubprocess.
@@ -524,15 +568,30 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout) uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg) err := s.escrowCeremony.Upload(uctx, pkg)
cancel() cancel()
outcome := uploadStored
if err != nil {
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2: the answer may be lost AFTER the store
}
s.escrowMu.Lock() s.escrowMu.Lock()
s.escrowClaiming = false s.escrowClaiming = false
if err != nil { switch outcome {
case uploadNotStored:
s.escrowMu.Unlock() s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)", s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds()) "vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again") writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return return
case uploadUnknown:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — upload failed and the hub cannot say which package it holds; code kept claimable (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
return
}
if err != nil {
s.logger.Warn("local-api: escrow claim — the upload's answer was lost, but the hub holds the new package; handing out the code (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err)
} }
s.escrowPending = nil s.escrowPending = nil
if s.escrowJob != nil && s.escrowJob.JobID == jobID { if s.escrowJob != nil && s.escrowJob.JobID == jobID {
@@ -631,15 +690,30 @@ func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Requ
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout) uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg) err := s.escrowCeremony.Upload(uctx, pkg)
cancel() cancel()
outcome := uploadStored
if err != nil {
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2
}
s.escrowMu.Lock() s.escrowMu.Lock()
s.escrowClaiming = false s.escrowClaiming = false
if err != nil { switch outcome {
case uploadNotStored:
s.escrowMu.Unlock() s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)", s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds()) "vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again") writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return return
case uploadUnknown:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — upload failed and the hub cannot say which package it holds; still waiting (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
return
}
if err != nil {
s.logger.Warn("local-api: escrow confirm — the upload's answer was lost, but the hub holds the new package (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err)
} }
s.escrowPending = nil s.escrowPending = nil
s.wipeEscrowConfirmLocked() s.wipeEscrowConfirmLocked()
+22 -3
View File
@@ -41,9 +41,13 @@ type fakeEscrowHub struct {
mu sync.Mutex mu sync.Mutex
current []byte current []byte
uploads int uploads int
err error // non-nil → the upload fails, nothing stored err error // non-nil → the upload fails, nothing stored
gate chan struct{} // non-nil → the upload blocks until closed // R-1254 H2: lost → the upload STORES the package and then fails (the answer is lost on the way back);
entered chan struct{} // non-nil → closed when an upload starts // holdsErr → the hub cannot be asked which package it holds.
lost bool
holdsErr error
gate chan struct{} // non-nil → the upload blocks until closed
entered chan struct{} // non-nil → closed when an upload starts
} }
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error { func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
@@ -61,9 +65,23 @@ func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
} }
f.current = append([]byte(nil), pkg...) f.current = append([]byte(nil), pkg...)
f.uploads++ f.uploads++
if f.lost {
return context.DeadlineExceeded
}
return nil return nil
} }
// holds answers „is the hub's current package exactly pkg?" — the R-1254 question (production compares the identity
// blob the hub serves back).
func (f *fakeEscrowHub) holds(_ context.Context, pkg []byte) (bool, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.holdsErr != nil {
return false, f.holdsErr
}
return string(f.current) == string(pkg), nil
}
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question. // opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
func (f *fakeEscrowHub) opensWith(r string) bool { func (f *fakeEscrowHub) opensWith(r string) bool {
f.mu.Lock() f.mu.Lock()
@@ -100,6 +118,7 @@ func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub
HubConfigured: true, HubConfigured: true,
DRConfigured: func() bool { return true }, DRConfigured: func() bool { return true },
Upload: hub.upload, Upload: hub.upload,
HubHolds: hub.holds,
} }
srv.ceremonyRun = run srv.ceremonyRun = run
srv.escrowSudoCheck = func(context.Context) error { return nil } srv.escrowSudoCheck = func(context.Context) error { return nil }
+98
View File
@@ -0,0 +1,98 @@
package localapi
import (
"errors"
"net/http"
"strings"
"testing"
)
// R-1254 (the ceremony half, review round 2 HUNT-3 H2): the upload's answer can be lost or late (30 s) AFTER the hub
// stored the new package. The household was then told „the previous code still works" — false: the hub now holds the
// package the NEW code opens. The agent asks the hub which package it holds before it words the result:
// - the hub holds the new package → success (the code is handed out / the confirm completes);
// - the hub holds another package → the old sentence, now true;
// - the hub cannot be asked → 504 and a sentence that does NOT promise the previous code; the wait goes on.
// RED-PROOF: remove the hub question from the claim's failure branch → 502 „the previous code still works" while the
// hub holds the new package → FAILS.
func TestR1254_ALostAnswerAtTheClaimIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
hub := &fakeEscrowHub{lost: true}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if !hub.opensWith(firstR) {
t.Fatal("fixture: the hub must hold the new package (the answer was lost after the store)")
}
if code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("claim after a lost answer, the hub holding the new package: %d %s — want 200 with the code", code, body)
}
if strings.Contains(body, "previous code still works") {
t.Fatalf("the household was told the previous code still works: %s", body)
}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
t.Fatalf("the job must read uploaded: %s", st)
}
}
// The hub cannot be asked: no promise about the previous code, 504, the code stays claimable (a retry decides).
// RED-PROOF: as above → 502 with „the previous code still works" → FAILS.
func TestR1254_AnUnknownOutcomeAtTheClaimPromisesNothing(t *testing.T) {
hub := &fakeEscrowHub{lost: true, holdsErr: errors.New("hub unreachable")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusGatewayTimeout {
t.Fatalf("claim with an unknown outcome: %d %s — want 504", code, body)
}
if strings.Contains(body, "previous code still works") {
t.Fatalf("an unknown outcome promised the previous code: %s", body)
}
assertNoRIn(t, "unknown-outcome claim payload", body, firstR)
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) {
t.Fatalf("the code must stay claimable: %s", st)
}
// The hub answers again: the retry finds the new package already stored and hands the code out.
hub.mu.Lock()
hub.holdsErr = nil
hub.mu.Unlock()
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("retry: %d %s", code, body)
}
}
// The hub holds ANOTHER package (the upload really failed): the old sentence stays, and is now true.
func TestR1254_AFailedUploadTheHubDidNotStoreKeepsTheOldSentence(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
if code, body := claimCode(t, h); code != http.StatusBadGateway || !strings.Contains(body, "previous code still works") {
t.Fatalf("a failed upload: %d %s — want 502 and the previous code named", code, body)
}
}
// The same at the typed-back confirm (R-1151).
// RED-PROOF: remove the hub question from the confirm's failure branch → 502 → FAILS.
func TestR1254_ALostAnswerAtTheConfirmIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
hub := &fakeEscrowHub{lost: true}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
t.Fatalf("confirm after a lost answer: %d (opens: %v) — want 200", code, hub.opensWith(secondR))
}
hub.mu.Lock()
hub.lost, hub.holdsErr = true, errors.New("hub unreachable")
hub.mu.Unlock()
srv2, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h2 := srv2.Handler()
startAndWait(t, srv2, h2)
c2 := claimDeferred(t, h2)
if code := confirm(t, h2, c2.typedBack(t)); code != http.StatusGatewayTimeout {
t.Fatalf("confirm with an unknown outcome: %d — want 504", code)
}
}
+102
View File
@@ -0,0 +1,102 @@
package localapi
import (
"bufio"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
)
// R-528 — the OOM signal from the host side.
//
// MEASURED 2026-09-15 (scratch 9202, Docker 29.8.0) and 2026-09-16 (a fresh install in a nested VM): an app killed by
// its memory limit inside a Felhom LXC guest left Docker's State.OOMKilled false and sent no `oom` event, so the
// controller's OOM line (R-514) could not fire. The kernel's own counter does count every kill, and the host can read
// it: /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events, line `oom_kill N` (the
// systemd cgroup driver; `ns/docker/<id>/` under the cgroupfs driver). Read on Tester 1's host 2026-10-28: the files
// are mode 0444, readable by the non-root felhom-agent user — no sudo, no new root path.
//
// A container's scope is new at each start, so the counter is per container RUN: a count above 0 means this run had a
// kill. The controller asks every 30 s and raises its existing app_oom line (once per run) for any such container.
// readGuestOOMKills returns container id → oom_kill for one guest. available=false (no error) when the guest has no
// cgroup folder (a VM guest, or a container that is not running).
func readGuestOOMKills(root string, vmid int) (map[string]int64, bool, error) {
if root == "" {
root = "/sys/fs/cgroup"
}
base := filepath.Join(root, "lxc", strconv.Itoa(vmid), "ns")
if fi, err := os.Stat(base); err != nil || !fi.IsDir() {
return nil, false, nil
}
out := map[string]int64{}
for _, pattern := range []string{
filepath.Join(base, "system.slice", "docker-*.scope", "memory.events"),
filepath.Join(base, "docker", "*", "memory.events"),
} {
files, err := filepath.Glob(pattern)
if err != nil {
return nil, true, err
}
for _, f := range files {
id := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(filepath.Dir(f)), "docker-"), ".scope")
if !isContainerID(id) {
continue
}
n, err := readOOMKill(f)
if err != nil {
continue // a container that stopped between the glob and the read
}
out[id] = n
}
}
return out, true, nil
}
func isContainerID(s string) bool {
if len(s) != 64 {
return false
}
for _, c := range s {
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
return false
}
}
return true
}
func readOOMKill(path string) (int64, error) {
f, err := os.Open(path)
if err != nil {
return 0, err
}
defer f.Close()
sc := bufio.NewScanner(f)
for sc.Scan() {
k, v, ok := strings.Cut(strings.TrimSpace(sc.Text()), " ")
if ok && k == "oom_kill" {
return strconv.ParseInt(strings.TrimSpace(v), 10, 64)
}
}
if err := sc.Err(); err != nil {
return 0, err
}
return 0, fmt.Errorf("no oom_kill line in %s", path)
}
// handleGuestOOMKills is GET /guest/oom-kills: {"vmid", "available", "containers": {id: oom_kill}}.
func (s *Server) handleGuestOOMKills(w http.ResponseWriter, r *http.Request, vmid int) {
kills, avail, err := readGuestOOMKills(s.cgroupRoot, vmid)
if err != nil {
s.logger.Warn("local-api: guest oom-kills read failed (R-528)", "vmid", vmid, "err", err)
writeErr(w, http.StatusInternalServerError, "the guest's OOM counters could not be read")
return
}
if kills == nil {
kills = map[string]int64{}
}
writeOK(w, map[string]any{"vmid": vmid, "available": avail, "containers": kills})
}
+101
View File
@@ -0,0 +1,101 @@
package localapi
import (
"encoding/json"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
)
// R-528: Docker does not always report an OOM kill inside a Felhom LXC guest (OOMKilled stays false, no `oom` event —
// measured 2026-09-15/16 on two boxes), but the kernel's own counter does count it, and it is readable from the host:
// /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events, line `oom_kill N`. GET /guest/oom-kills
// serves the asking guest's counters, and only that guest's.
func fakeCgroup(t *testing.T, root string, vmid, id, events string) {
t.Helper()
dir := filepath.Join(root, "lxc", vmid, "ns", "system.slice", "docker-"+id+".scope")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o444); err != nil {
t.Fatal(err)
}
}
const idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
const idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
const idOther = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
// RED-PROOF: serve the counters of every guest under lxc/ (drop the vmid from the glob) → the other guest's container
// appears → FAILS; parse `oom ` instead of `oom_kill` → idA reads 9 → FAILS.
func TestR528_GuestOOMKillsServesThisGuestsCountersOnly(t *testing.T) {
root := t.TempDir()
fakeCgroup(t, root, "100", idA, "low 0\nhigh 0\nmax 4\noom 9\noom_kill 3\noom_group_kill 0\n")
fakeCgroup(t, root, "100", idB, "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\n")
fakeCgroup(t, root, "200", idOther, "oom_kill 7\n")
// a scope that is not a container (no 64-hex id) and a cgroupfs-driver layout container
if err := os.MkdirAll(filepath.Join(root, "lxc", "100", "ns", "system.slice", "cron.service"), 0o755); err != nil {
t.Fatal(err)
}
dfDir := filepath.Join(root, "lxc", "100", "ns", "docker", idOther[:8]+strings.Repeat("d", 56))
if err := os.MkdirAll(dfDir, 0o755); err != nil {
t.Fatal(err)
}
_ = os.WriteFile(filepath.Join(dfDir, "memory.events"), []byte("oom_kill 2\n"), 0o444)
got, avail, err := readGuestOOMKills(root, 100)
if err != nil || !avail {
t.Fatalf("read: avail=%v err=%v", avail, err)
}
want := map[string]int64{idA: 3, idB: 0, idOther[:8] + strings.Repeat("d", 56): 2}
if len(got) != len(want) {
t.Fatalf("got %v, want %v", got, want)
}
for k, v := range want {
if got[k] != v {
t.Fatalf("got %v, want %v", got, want)
}
}
if _, leaked := got[idOther]; leaked {
t.Fatal("another guest's container leaked into this guest's answer")
}
// A guest with no cgroup folder (a VM, or not running): not available, no error.
if _, avail, err := readGuestOOMKills(root, 300); avail || err != nil {
t.Fatalf("a guest without a cgroup folder: avail=%v err=%v", avail, err)
}
}
// Through the route, with the guest's own token.
func TestR528_TheRouteAnswersTheTokensGuest(t *testing.T) {
root := t.TempDir()
fakeCgroup(t, root, "8200", idA, "oom_kill 5\n")
fakeCgroup(t, root, "9300", idB, "oom_kill 1\n") // token B's guest: never in A's answer
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.cgroupRoot = root
w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "A", "")
if w.Code != http.StatusOK {
t.Fatalf("route: %d %s", w.Code, w.Body.String())
}
var env struct {
Data struct {
VMID int `json:"vmid"`
Available bool `json:"available"`
Containers map[string]int64 `json:"containers"`
} `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
t.Fatal(err)
}
if env.Data.VMID != 8200 || !env.Data.Available || len(env.Data.Containers) != 1 || env.Data.Containers[idA] != 5 {
t.Fatalf("answer %s — want guest 8200's one container with 5", w.Body.String())
}
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
t.Fatalf("a cross-guest ask: %d, want 403", w.Code)
}
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "", ""); w.Code != http.StatusUnauthorized {
t.Fatalf("no token: %d", w.Code)
}
}
+5
View File
@@ -459,6 +459,8 @@ type Server struct {
escrowLookPath func(file string) (string, error) escrowLookPath func(file string) (string, error)
// statFile reports whether a path exists (preflight's staged-secret item; tests inject). // statFile reports whether a path exists (preflight's staged-secret item; tests inject).
statFile func(path string) bool statFile func(path string) bool
// cgroupRoot is where the host's cgroup tree is mounted (R-528, GET /guest/oom-kills); "" = /sys/fs/cgroup. Tests set it.
cgroupRoot string
baseCtx context.Context // for fire-and-forget backups; set in Run baseCtx context.Context // for fire-and-forget backups; set in Run
} }
@@ -576,6 +578,9 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard)) mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch // R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
// read it. Read-only; never measured (or the job not wired) = present:false. // read it. Read-only; never measured (or the job not wired) = present:false.
// R-528: the kernel's OOM-kill counter of each of the guest's containers, read on the host (Docker's own flag and
// event can stay silent inside an LXC guest). Read-only; the asking guest's containers only.
mux.HandleFunc("GET /guest/oom-kills", s.withGuest(s.handleGuestOOMKills))
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest)) mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate. // Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
+32
View File
@@ -8,6 +8,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"io" "io"
"log/slog"
"net/http" "net/http"
"os" "os"
"path/filepath" "path/filepath"
@@ -114,12 +115,43 @@ func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params jso
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed) return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
} }
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds) lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
e.policyPass(ctx, lg, p.AgentVersion)
if e.AfterInstall != nil { if e.AfterInstall != nil {
e.AfterInstall(ctx) e.AfterInstall(ctx)
} }
return nil return nil
} }
// policyPass (R-1285) asks the wrapper NOW on disk — after this install, the bundle's own — to apply the
// appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run
// itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal:
// the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it.
// PolicyPassAfterUpdate runs the wrapper's bundle-policy pass once after a self-update commits (R-1285). In the normal
// delivery the box gets agent_update and agent_config_update in ONE pass: the bundle job runs in the OLD process, which
// has no policy pass, and the restart into the new agent comes a second later (measured 2026-10-26 on demo-hp). By the
// commit the wrapper on disk is the bundle's own, so the pass removes our appliance-only files from a BYO host in the
// SAME update. Idempotent; on an appliance it removes nothing.
func (e ConfigUpdateExecutor) PolicyPassAfterUpdate(ctx context.Context, lg *slog.Logger, ver string) {
if lg == nil {
lg = slog.Default()
}
lg.Info("osupdate: running the bundle policy pass after the self-update commit (R-1285)", "agent_version", ver)
e.policyPass(ctx, lg, ver)
}
func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) {
runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z")
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"})
switch {
case err != nil:
lg.Warn("osupdate: the bundle policy pass did not run", "err", err)
case wr.refused():
lg.Info("osupdate: the bundle policy pass was refused (a wrapper before 0.166.0?) — the next bundle applies it", "refused", string(wr.Refused))
default:
lg.Info("osupdate: bundle policy pass done", "bundle_policy", string(wr.BundlePolicy))
}
}
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) { func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
hc := e.HTTPClient hc := e.HTTPClient
if hc == nil { if hc == nil {
+27
View File
@@ -38,6 +38,9 @@ func (b *bundleWrapper) Run(_ context.Context, name string, args ...string) ([]b
var plan map[string]any var plan map[string]any
_ = json.Unmarshal(raw, &plan) _ = json.Unmarshal(raw, &plan)
b.plans = append(b.plans, plan) b.plans = append(b.plans, plan)
if plan["mode"] == "bundle-policy" { // R-1285: the pass after an install carries no bundle
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":[]}}` + "\n"), nil, nil
}
body, _ := os.ReadFile(plan["bundle"].(string)) body, _ := os.ReadFile(plan["bundle"].(string))
b.bodies = append(b.bodies, body) b.bodies = append(b.bodies, body)
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
@@ -153,3 +156,27 @@ func TestBundleURL(t *testing.T) {
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) { func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return b.Run(ctx, name, args...) return b.Run(ctx, name, args...)
} }
// R-1285: right after an install the agent asks the wrapper now on disk for the appliance-only policy pass — the bundle
// run itself was planned by the OLD wrapper (on a BYO host upgraded from 0.164.0 our SSH drop-in stayed one bundle
// longer). A refused or failed bundle runs no pass.
// RED-PROOF: drop the policyPass call → one plan only.
func TestR1285_AnInstalledBundleIsFollowedByThePolicyPass(t *testing.T) {
body := []byte(`{"format":1,"agent_version":"0.143.0","files":[]}`)
srv, sha := serveBundle(t, body)
w := &bundleWrapper{t: t, rep: `{"mode":"bundle","bundle":{"agent_version":"0.143.0","written":["/usr/local/sbin/felhom-os-apply"]}}`}
e, _ := bundleExec(t, w, srv.URL)
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err != nil {
t.Fatal(err)
}
if len(w.plans) != 2 || w.plans[1]["mode"] != "bundle-policy" || w.plans[1]["layer"] != "host" || w.plans[1]["signed"] != nil {
t.Fatalf("after the install the agent must run the policy pass (host layer, no envelope): %v", w.plans)
}
w2 := &bundleWrapper{t: t, rep: `{"mode":"bundle","refused":{"code":"R3","reason":"x"}}`}
e2, _ := bundleExec(t, w2, srv.URL)
_ = e2.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha))
if len(w2.plans) != 1 {
t.Fatalf("a refused bundle must run no policy pass: %v", w2.plans)
}
}
+2 -1
View File
@@ -123,7 +123,8 @@ type WrapperReport struct {
Undo bool `json:"undo"` Undo bool `json:"undo"`
LiveRestore json.RawMessage `json:"live_restore"` LiveRestore json.RawMessage `json:"live_restore"`
Facts json.RawMessage `json:"facts"` Facts json.RawMessage `json:"facts"`
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle") Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
BundlePolicy json.RawMessage `json:"bundle_policy"` // the appliance-only policy pass (R-1285, mode "bundle-policy")
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event, // OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it. // exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
OOMCheck json.RawMessage `json:"oom_check"` OOMCheck json.RawMessage `json:"oom_check"`
+9
View File
@@ -18,8 +18,14 @@ type Manager struct {
logger *slog.Logger logger *slog.Logger
now func() time.Time // injectable for tests now func() time.Time // injectable for tests
sleep func(context.Context, time.Duration) sleep func(context.Context, time.Duration)
// onCommitted runs once, right after a successful commit (R-1285: the bundle policy pass of the NEW agent — in the
// normal two-job delivery the bundle was installed by the OLD process, which has no such pass). nil = none.
onCommitted func(context.Context)
} }
// SetOnCommitted wires the after-commit hook (R-1285). INIT-ONLY: call it before MaybeCommit's goroutine starts.
func (m *Manager) SetOnCommitted(fn func(context.Context)) { m.onCommitted = fn }
// ManagerConfig wires the commit manager. // ManagerConfig wires the commit manager.
type ManagerConfig struct { type ManagerConfig struct {
StateDir string StateDir string
@@ -119,4 +125,7 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
return return
} }
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout)) m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
if m.onCommitted != nil {
m.onCommitted(ctx)
}
} }
+28 -3
View File
@@ -98,7 +98,8 @@ TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
# THE retention number, read from the one file that owns it. A check and the policy it enforces # THE retention number, read from the one file that owns it. A check and the policy it enforces
# must read the same number from the same place, or they drift and the drift looks like a defect # must read the same number from the same place, or they drift and the drift looks like a defect
# in something else — which is exactly what happened on 2026-08-08/09 (R-287). # in something else — which is exactly what happened on 2026-08-08/09 (R-287).
RETENTION_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json") RETENTION_FILE = os.environ.get("FELHOM_RETENTION_POLICY") or \
os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json") # the env: decoy tests only
def retention_kept(): def retention_kept():
@@ -114,6 +115,21 @@ def retention_kept():
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,)) raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
return n return n
def retention_pinned():
"""D50 = A: the versions kept besides the newest N — the hub's MinAgent floor and the day-0 agent. Copies of hub
state the vouching session writes into the policy file (CI holds no hub secret). Missing key = no pins."""
with open(RETENTION_FILE, encoding="utf-8") as fh:
pins = json.load(fh).get("pinned") or {}
if not isinstance(pins, dict):
raise ValueError("pinned must be an object, got %r" % (pins,))
out = {}
for name, v in pins.items():
if not isinstance(v, str) or not re.match(r"^\d+\.\d+\.\d+$", v):
raise ValueError("pinned %s must be X.Y.Z, got %r" % (name, v))
out[name] = v
return out
tried = [] tried = []
@@ -211,6 +227,7 @@ def main():
all_versions = versions all_versions = versions
try: try:
keep = retention_kept() keep = retention_kept()
pins = retention_pinned()
except Exception as e: except Exception as e:
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e)) inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
@@ -218,10 +235,15 @@ def main():
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end. # lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
def _key(v): def _key(v):
return tuple(int(x) for x in v.split(".")) return tuple(int(x) for x in v.split("."))
versions = sorted(all_versions, key=_key)[-keep:] window = sorted(all_versions, key=_key)[-keep:]
# D50 = A: the pinned versions are asserted wherever they sit. A pin with no tag cannot be installed at all —
# a failure, never a skip.
unpinned_tags = sorted(("%s=%s" % (k, v)) for k, v in pins.items() if v not in all_versions)
versions = sorted(set(window) | set(v for v in pins.values() if v in all_versions), key=_key)
dropped = [v for v in all_versions if v not in versions] dropped = [v for v in all_versions if v not in versions]
print(" %d released version(s); retention policy keeps the newest %d" % (len(all_versions), keep)) print(" %d released version(s); retention policy keeps the newest %d plus the pinned %s" % (
len(all_versions), keep, ", ".join("%s=%s" % kv for kv in sorted(pins.items())) or "(none)"))
print(" verifying: %s" % ", ".join(versions)) print(" verifying: %s" % ", ".join(versions))
if dropped: if dropped:
# NEVER silent. A bounded check that does not say what it stopped covering is how a # NEVER silent. A bounded check that does not say what it stopped covering is how a
@@ -232,6 +254,9 @@ def main():
print(" their configs resolve; what is no longer asserted is the BINARY's presence.") print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
bad = [] bad = []
for pin in unpinned_tags:
bad.append((pin, ["the pinned version has NO git tag — nothing can install it (D50 = A)"]))
print(" FAIL pinned %s: no git tag v%s" % (pin, pin.split("=")[1]))
for v in versions: for v in versions:
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG) pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG) raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
+23 -31
View File
@@ -1,41 +1,33 @@
{ {
"_comment": [ "_comment": [
"THE retention number for published agent artifacts. One file, read by everything that", "THE retention rule for published agent artifacts. One file, read by everything that depends on it, because a check",
"depends on it, because a check and the policy it enforces must read the same number from the", "and the policy it enforces must read the same rule from the same place or they drift — and the drift looks like a",
"same place or they drift — and the drift looks like a defect in something else.", "defect in something else (R-287, 2026-08-08/09; again R-1290, 2026-10-25).",
"", "",
"WHAT WENT WRONG WITHOUT IT (2026-08-08/09). The registry stopped serving felhom-agent", "D50 = A (operator ruling 2026-10-25, felhom.eu `09` §3 decision 262): the registry keeps the NEWEST 5 released agent",
"0.120.0 and older, while scripts/check-published-versions.py demanded that EVERY git tag", "versions, PLUS the hub's MinAgent floor, PLUS the version the hub's day-0 artifact manifest names. The operator prunes",
"still be downloadable. Both rules are individually sensible; together they are impossible.", "by hand to that rule (on 2026-10-25 he removed 0.159.0–0.160.0 and kept 0.158.1, the floor). CI asserts exactly these",
"CI went red at a commit whose own run had been green the day before, on a true finding that", "and never more. Before D50 the number was 10 (the R-267 newest-10 prune, R-287/R-291).",
"no one could act on. The red will return at the next publish unless the two read one number.",
"", "",
"HOW THE NUMBER WAS ARRIVED AT. generic_versions_kept is 10 because that is what the registry", "THE PINS ARE COPIES OF HUB STATE, because CI cannot read the hub without a secret it must not hold. The session that",
"keeps: the deleter was ESTABLISHED on 2026-08-10 (R-287) — the newest-10 prune of generic packages", "records an agent vouch (day0_agent) or declares a new MinAgent with a floor (hub_min_agent) updates `pinned` in the",
"run under R-267 (felhom-agent and felhom-golden generic held exactly 10 afterwards). Until then this", "same commit (felhom-build-deploy skill, agent release). A pin that names a version with no tag is a FAILURE, never a",
"file called the 10 an observed state with no located ruling; that is superseded (corrected", "skip — a pin nobody can install is the defect the pin exists to catch.",
"2026-10-05, R-291). Container packages are not pruned by that rule.",
"", "",
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest", "SO THIS FILE IS A FLOOR, NOT A LICENCE: it says what CI may assume is still downloadable. It does not authorise",
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the", "deleting anything else.",
"operator should confirm or replace the number — at which point this file changes and both",
"readers follow it in the same commit.",
"", "",
"THE DEEPER BOUND, recorded so a future session does not have to re-derive it: the principled", "NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config files from",
"limit is the hub's vouched min_agent floor (0.127.0 on 2026-08-09). Nothing can install an", "raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that version at all."
"agent below it — the hub refuses to vouch one and boxes update to the floor — so a released",
"version below the floor being un-downloadable costs nothing real. Bounding on the floor would",
"be better than bounding on a count, and it needs the gate to read the hub, which is network",
"the gate does not have today. Filed as the follow-up in R-287.",
"",
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config",
"files from raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that",
"version at all — a strictly worse act than an un-downloadable binary."
], ],
"generic_versions_kept": 10, "generic_versions_kept": 5,
"readers": [ "readers": [
"scripts/check-published-versions.py — bounds its assertion to the newest N versions" "scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
], ],
"recorded": "2026-08-09", "recorded": "2026-10-27",
"recorded_by": "CC 2026-08-09; the number's source (the R-267 newest-10 prune, established 2026-08-10 by R-287) recorded 2026-10-05 (R-291)" "recorded_by": "CC 2026-10-27: day0_agent 0.167.0 (operator vouch 2026-10-27, read from the hub: version + agent and bundle sha256 match the release); earlier 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"pinned": {
"hub_min_agent": "0.158.1",
"day0_agent": "0.167.0"
}
} }
+31 -1
View File
@@ -179,13 +179,24 @@ def published_cases(base):
TAGS = ["v0.150.%d" % i for i in range(3)] # inside any retention window >= 3 TAGS = ["v0.150.%d" % i for i in range(3)] # inside any retention window >= 3
VERS = [t[1:] for t in TAGS] VERS = [t[1:] for t in TAGS]
# D50 = A: the gate also asserts the PINNED versions. The decoys run against their own policy file (the real pins
# name versions the fake Gitea does not hold); PINS is what each case writes there.
pol_dir = tempfile.mkdtemp(prefix="retention-decoy-")
pol = os.path.join(pol_dir, "retention-policy.json")
PINS = {}
def case(name, setup, expect_rc, must=()): def case(name, setup, expect_rc, must=()):
FAKE.reset() FAKE.reset()
FAKE.tags = list(TAGS) FAKE.tags = list(TAGS)
FAKE.packages = set(VERS) FAKE.packages = set(VERS)
FAKE.raw = set(VERS) FAKE.raw = set(VERS)
PINS.clear()
setup() setup()
rc, out = run([sys.executable, gate], ROOT, child_env(base)) with io.open(pol, "w", encoding="utf-8") as fh:
json.dump({"generic_versions_kept": keep, "pinned": dict(PINS)}, fh)
env = child_env(base)
env["FELHOM_RETENTION_POLICY"] = pol
rc, out = run([sys.executable, gate], ROOT, env)
if not FAKE.hits: if not FAKE.hits:
fails.append("published/%s: the gate never asked the fake Gitea - the seam is not wired" % name) fails.append("published/%s: the gate never asked the fake Gitea - the seam is not wired" % name)
report("published: " + name, rc, out, expect_rc, must) report("published: " + name, rc, out, expect_rc, must)
@@ -220,6 +231,25 @@ def published_cases(base):
case("BY DESIGN: a version older than the retention window is not asserted", retired, 0, case("BY DESIGN: a version older than the retention window is not asserted", retired, 0,
("NOT ASSERTED", "0.9.0")) ("NOT ASSERTED", "0.9.0"))
def pinned_old_ok():
FAKE.tags = ["v0.8.0"] + ["v0.10.%d" % i for i in range(keep)]
FAKE.packages = set(t[1:] for t in FAKE.tags)
FAKE.raw = set(t[1:] for t in FAKE.tags)
PINS["hub_min_agent"] = "0.8.0"
case("GENUINE (D50): a pinned version older than the window is asserted and downloads", pinned_old_ok, 0,
("verifying: 0.8.0", "ALL RELEASED VERSIONS INSTALLABLE"))
def pinned_old_gone():
pinned_old_ok()
FAKE.packages.discard("0.8.0")
case("FACT (D50): a pinned version older than the window whose package 404s", pinned_old_gone, 1,
("FAIL v0.8.0", "binary NOT downloadable"))
def pinned_no_tag():
PINS["day0_agent"] = "0.150.9"
case("FACT (D50): a pin that names a version with no tag", pinned_no_tag, 1,
("FAIL pinned day0_agent=0.150.9", "no git tag"))
def five_hundred(): def five_hundred():
FAKE.tags_status = 500 FAKE.tags_status = 500
case("INCONCLUSIVE: the tags api answers 500", five_hundred, 2, ("INCONCLUSIVE",)) case("INCONCLUSIVE: the tags api answers 500", five_hundred, 2, ("INCONCLUSIVE",))