Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 53d047a6c1 | |||
| 28ba8593b8 | |||
| 6981450110 |
@@ -1,3 +1,78 @@
|
|||||||
|
### Gates only — 2026-08-09 (no release, no version bump, no binary published)
|
||||||
|
|
||||||
|
**Two guards, both owed since the 2026-08-09 install outage (R-273/R-287). Nothing that runs on a
|
||||||
|
customer's box changed; `scripts/` only, and the agent stays v0.128.0.**
|
||||||
|
|
||||||
|
- **`scripts/retention-policy.json` — THE retention number, in one file.** The registry stopped
|
||||||
|
serving `felhom-agent` 0.120.0 and older while `check-published-versions.py` demanded that every
|
||||||
|
tag still be downloadable. Both rules are sensible; together they are impossible, and CI went red
|
||||||
|
at a commit whose own run had been green the day before. The check now **reads the number from
|
||||||
|
this file** and bounds its assertion to the newest N generic versions.
|
||||||
|
**What CI no longer covers, said plainly rather than left to be discovered:** a released version
|
||||||
|
older than the retention window is **no longer asserted downloadable**. Its git tag and its configs
|
||||||
|
are still asserted — only the binary's presence is dropped. The check **prints exactly which
|
||||||
|
versions it stopped covering** on every run, so the narrowing cannot become permanent by accident.
|
||||||
|
**The number is an OBSERVED state, not a located ruling** — see the file's own header and R-287.
|
||||||
|
A missing or unreadable policy file is **INCONCLUSIVE (exit 2), never silently unbounded.**
|
||||||
|
- **`scripts/check-release-complete.py` — the tag half, as a machine.** Asserts that the version at
|
||||||
|
the head of `CHANGELOG.md` is tagged, that the tag points into this history, and that its package
|
||||||
|
is published. `release-agent.sh` already warned about this in as many words and the step was still
|
||||||
|
missed on 2026-08-08, which is why this is a gate and not a reminder. Legs 1–2 need no network and
|
||||||
|
therefore run in `--fast`, so the pre-push hook catches a missing tag at the earliest moment.
|
||||||
|
Registered in `agent_gates.py`; red-proved by pointing the CHANGELOG head at an unreleased
|
||||||
|
v0.129.0 — both legs convicted and each named its fix command.
|
||||||
|
|
||||||
|
## v0.128.0 — the escrow seed is asserted every tick, not remembered once (2026-08-08, R-221)
|
||||||
|
|
||||||
|
**A rebuilt box could not run the escrow ceremony at all, and there was no way forward from inside
|
||||||
|
the product.** The preflight refuses on `escrow.pbs_storage_id`; the pbsdr bridge writes that key;
|
||||||
|
and it wrote it from exactly one place — `finishConverged`, reached only on the paths that actually
|
||||||
|
converge.
|
||||||
|
|
||||||
|
**The two things live in different places and die at different times.** The convergence marker is
|
||||||
|
host-side (`<agent-state>/pbsdr/marker.json`). The key it seeds is in `agent.json`, which
|
||||||
|
`step_agent_config` renders from `base = {}` unless an explicit `--preserve-from` is passed
|
||||||
|
(`felhom.eu/scripts/felhom-host-install.sh:2396`, the render at `:2449`, the `O_TRUNC` write at
|
||||||
|
`:2579`; the flag at `:1246`, defaulting empty at `:256`) — **and the render never writes an
|
||||||
|
`escrow` section at all.** So a rebuild keeps the marker and takes the key: same descriptor, same
|
||||||
|
hash, early return, and the seed never runs again into a config that no longer has it.
|
||||||
|
|
||||||
|
**Fixed by asserting rather than remembering.** `Apply` now re-asserts the seed *before* the
|
||||||
|
idempotent early return. `seedEscrowStorageID` is unchanged and still never clobbers a different
|
||||||
|
existing value — an operator's own choice outranks the descriptor's, with a warning naming both.
|
||||||
|
|
||||||
|
**The early return is KEPT.** It exists so a converged box does not re-run Proxmox operations every
|
||||||
|
60 s, and `TestSeedReasserted_OnConvergedTick_WithZeroProxmoxCalls` asserts **zero** recorded runner
|
||||||
|
calls on that tick — so a "fix" that simply deleted the return would fail. Cost of the re-assert: one
|
||||||
|
small file read plus a JSON parse per tick, no exec, no network, and an early return once the value
|
||||||
|
matches.
|
||||||
|
|
||||||
|
**A seed failure can never un-converge the box:** Warn plus a message on the published status,
|
||||||
|
exactly as `finishConverged` does it — no marker write, no state change. Pinned by
|
||||||
|
`TestSeedReassertFailure_DoesNotUnconverge`.
|
||||||
|
|
||||||
|
Tests drive the real `Apply` with a real temp-dir `agent.json` and a call-recording runner; calling
|
||||||
|
`seedEscrowStorageID` directly cannot see the early return, which IS the defect. The production
|
||||||
|
wiring (`pbsdr.NewManager(..., cfg.SourcePath, ...)`) is asserted by walking `main.go`'s **AST**, not
|
||||||
|
by `strings.Contains`, which a commented-out call also satisfies.
|
||||||
|
|
||||||
|
Red-proofs: removing the new call makes Scenario A fail on today's tree; removing the early return
|
||||||
|
makes the zero-Proxmox-calls assertion fail.
|
||||||
|
|
||||||
|
## (no version bump) — a comment that claimed the hub reads a field it has no field for (2026-08-08, R-260)
|
||||||
|
|
||||||
|
Comment-only; no behaviour, no wire change, nothing to rebuild.
|
||||||
|
|
||||||
|
`HostReport.SelfUpdatePending` / `SelfUpdatePendingVersion` carried the sentence *"The hub reads an
|
||||||
|
absent field as pending=false, the correct default."* **The hub has no field for either**, so it reads
|
||||||
|
nothing — present or absent — and `encoding/json` discards them on arrival. The sentence described an
|
||||||
|
intent rather than the code and read as settled for long enough that a class sweep had to find it.
|
||||||
|
|
||||||
|
The emission is correct and stays: the agent reports the truth, and the fault is entirely in the
|
||||||
|
receiving. The missing consumer is tracked as **R-264** (OPEN), and
|
||||||
|
`felhom.eu/scripts/wire_contract_gate.py` now refuses any NEW field of this shape while recording the
|
||||||
|
existing ones as explicit, reasoned allowlist entries rather than silence.
|
||||||
|
|
||||||
## v0.127.0 — a mount Felhom itself made is not "something else" (2026-08-06, R-220)
|
## v0.127.0 — a mount Felhom itself made is not "something else" (2026-08-06, R-220)
|
||||||
|
|
||||||
**After a rebuild the customer's own drives could not be re-attached, and the refusal named an action
|
**After a rebuild the customer's own drives could not be re-attached, and the refusal named an action
|
||||||
|
|||||||
@@ -1,41 +1,63 @@
|
|||||||
# REPORT — felhom-agent v0.127.0: a mount Felhom made is not foreign (R-220)
|
# REPORT — felhom-agent, 2026-08-09 (gates only)
|
||||||
|
|
||||||
**Scope: the host half of R-220.** The customer-facing refusal message is the controller's half and
|
**No release. No version bump. No binary published. `scripts/` only** — nothing that runs on a
|
||||||
ships as felhom-controller v0.203.0.
|
customer's machine changed, and the agent stays **v0.128.0** at `28ba8593b8`.
|
||||||
|
|
||||||
## What changed
|
## What changed
|
||||||
|
|
||||||
| File | Change |
|
| file | why |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `internal/storage/claim.go` | `claimFacts.felhomOwnedMounts`; `classifyClaim` forgives a non-managed mountpoint **only when corroborated**; `felhomOwnedMounts()` + `procMounts()` |
|
| `scripts/retention-policy.json` **(new)** | THE retention number, in one place, with its reasoning and its honesty about where the number came from |
|
||||||
| `internal/storage/hostops.go` | `mountTable` seam (nil ⇒ real `/proc/mounts`) |
|
| `scripts/check-published-versions.py` | reads that number; bounds its assertion to the newest N; **prints what it stopped covering** |
|
||||||
| `internal/storage/claim_r220_test.go` | new — the own-drive case, the fence, and the corroboration's four edges |
|
| `scripts/check-release-complete.py` **(new)** | asserts the CHANGELOG-head version is tagged, placed in this history, and published |
|
||||||
|
| `scripts/agent_gates.py` | registers the new gate; legs 1–2 are offline so it runs in `--fast` too |
|
||||||
|
|
||||||
## The shape chosen, and why (§7.3)
|
## The coupling defect, and the fix
|
||||||
|
|
||||||
**Candidate (b): the claimed check distinguishes a mount Felhom made from a foreign one** — the task
|
The prune keeps the newest N; the published-versions check demanded that **every** tag be
|
||||||
called it "nearer the truth" and it is, because the host and its knowledge survive the rebuild while
|
downloadable. Nothing connected them, so CI went red at `28ba8593b8` — a commit whose own run had
|
||||||
the guest's registry does not. Candidate (a) — having the rebuild path clear the raw mounts — would
|
been **green the day before** — and would have gone red again at the next publish when `0.121.0` was
|
||||||
have made correctness depend on a cleanup step running, and a cleanup that does not run leaves exactly
|
evicted. Both now read `generic_versions_kept` from one file.
|
||||||
today's defect.
|
|
||||||
|
|
||||||
**The discriminator is corroboration, not a path prefix**: the same device must ALSO be mounted under
|
**What CI no longer covers:** a released version **older than the retention window** is no longer
|
||||||
`/mnt/felhom-drives`. Only enrolment produces that pairing.
|
asserted downloadable. Its git tag and its config tree are still asserted; only the binary's presence
|
||||||
|
is dropped. The check names the dropped versions on every run.
|
||||||
|
|
||||||
**`/proc/mounts` rather than `lsblk MOUNTPOINTS`**, because the lsblk invocation is pinned verbatim in
|
**The number is not a located ruling.** `generic_versions_kept: 10` is what the registry demonstrably
|
||||||
the sudoers file; changing it would have coupled this fix to a config rollout. `/proc/mounts` is
|
holds; no register row records a prune, and container packages hold 19 each. The file says so in its
|
||||||
world-readable and needs neither.
|
own header. The principled bound is the hub's vouched `min_agent` floor — nothing can install below
|
||||||
|
it — and that is recorded as the follow-up.
|
||||||
|
|
||||||
|
## Controls, all three run
|
||||||
|
|
||||||
|
| control | expected | got |
|
||||||
|
|---|---|---|
|
||||||
|
| live run, policy = 10 | green, and it names `0.120.0` as not asserted | **exit 0**, and it did |
|
||||||
|
| widen policy to 11 | `0.120.0` re-enters the window and convicts | **exit 1**, `FAIL v0.120.0` |
|
||||||
|
| policy file removed | INCONCLUSIVE, never silently unbounded | **exit 2**, naming the path it tried |
|
||||||
|
|
||||||
|
## Red-proof of the new gate
|
||||||
|
|
||||||
|
Mutation: `CHANGELOG.md` head repointed to `## v0.129.0` — never tagged, never published. Asserted
|
||||||
|
applied (`grep -c '^## v0.129.0'` → 1). Result **exit 1**, both legs convicting:
|
||||||
|
|
||||||
|
```
|
||||||
|
- TAG v0.129.0 DOES NOT EXIST. … without the tag every install 404s mid-run, as root.
|
||||||
|
Fix: git tag -a v0.129.0 <released-commit> && git push origin v0.129.0
|
||||||
|
- PACKAGE 0.129.0 IS NOT PUBLISHED (HTTP 404 …).
|
||||||
|
Fix: bash scripts/release-agent.sh 0.129.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Reverted; `git status` clean on `CHANGELOG.md`.
|
||||||
|
|
||||||
## Green gate
|
## Green gate
|
||||||
|
|
||||||
`go build` · `go vet` clean · `go test ./...` → **29 packages ok** · `agent_gates.py --fast` → all OK.
|
`python3 scripts/agent_gates.py` — `reuse-refs OK · instructions OK · published OK ·
|
||||||
|
release-complete OK · all agent gates OK`.
|
||||||
|
|
||||||
| Red-proof | Result |
|
## Not done here
|
||||||
|---|---|
|
|
||||||
| remove the `felhomOwnedMounts` exemption | **FAILS** — "device is mounted at /mnt/adatok (sdb)", the pre-fix refusal |
|
|
||||||
| over-widen the exemption to any `/mnt/*` | **FAILS** — "/mnt/someone-elses-disk was offered for formatting" |
|
|
||||||
|
|
||||||
## Not changed
|
The deleter of `0.120.0` is **still not established** and a second attempt failed — Gitea keeps no
|
||||||
|
package-deletion trail, its container log no longer reaches the window, and the activity feed carries
|
||||||
No sudoers, no allowlisted command, no PVE surface, no format path. Every other claim signal
|
no package operation. Recorded in R-287, including the withdrawal of my own earlier over-claim that
|
||||||
(system disk, read-only, LVM PV, ZFS member, member FSTYPEs, empty-topology backstop) is untouched.
|
the router logs showed no DELETE: they do not cover the window, so they never said anything.
|
||||||
|
|||||||
@@ -68,8 +68,14 @@ type HostReport struct {
|
|||||||
// report is stored opaquely hub-side, so these additive fields need no hub-schema change.
|
// report is stored opaquely hub-side, so these additive fields need no hub-schema change.
|
||||||
// Both are `omitempty` (the Wireguard precedent): in the steady state (no update in flight)
|
// Both are `omitempty` (the Wireguard precedent): in the steady state (no update in flight)
|
||||||
// they are absent — which keeps the cross-repo host-report golden contract byte-stable without
|
// they are absent — which keeps the cross-repo host-report golden contract byte-stable without
|
||||||
// a hub change. They appear only while an update is pending. The hub reads an absent field as
|
// a hub change. They appear only while an update is pending.
|
||||||
// pending=false, the correct default.
|
//
|
||||||
|
// ⚠ CORRECTED 2026-08-08 (R-260). This comment used to end "The hub reads an absent field as
|
||||||
|
// pending=false, the correct default." THE HUB HAS NO FIELD FOR EITHER OF THESE, so it reads
|
||||||
|
// nothing — present or absent — and encoding/json discards them on arrival. The sentence
|
||||||
|
// described an intent, not the code, and it read as settled for long enough that a sweep had to
|
||||||
|
// find it. The emission is correct and stays; the missing consumer is tracked as R-264, and
|
||||||
|
// `felhom.eu/scripts/wire_contract_gate.py` now refuses any NEW field of this shape.
|
||||||
SelfUpdatePending bool `json:"selfupdate_pending,omitempty"`
|
SelfUpdatePending bool `json:"selfupdate_pending,omitempty"`
|
||||||
SelfUpdatePendingVersion string `json:"selfupdate_pending_version,omitempty"`
|
SelfUpdatePendingVersion string `json:"selfupdate_pending_version,omitempty"`
|
||||||
|
|
||||||
|
|||||||
@@ -283,7 +283,36 @@ func (m *Manager) Apply(ctx context.Context, fetched bool, block *hub.WirePBSDR)
|
|||||||
h := descriptorHash(block)
|
h := descriptorHash(block)
|
||||||
cf := m.loadConsumedFailed()
|
cf := m.loadConsumedFailed()
|
||||||
if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) {
|
if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) {
|
||||||
m.setStatus(&hub.PBSDRStatus{State: mk.State, StorageID: block.StorageID, Namespace: block.Namespace, AppliedAt: mk.AppliedAt})
|
// R-221: RE-ASSERT THE SEED, DO NOT REMEMBER IT. The marker records that this descriptor
|
||||||
|
// converged; it says nothing about whether the file the seed writes still exists.
|
||||||
|
//
|
||||||
|
// The two live in different places and die at different times. The marker is host-side
|
||||||
|
// (`<agent-state>/pbsdr/`, markerPath above); the seed's target is `agent.json`, and the
|
||||||
|
// installer's `step_agent_config` renders that file from `base = {}` unless an explicit
|
||||||
|
// `--preserve-from` is passed — it NEVER writes an `escrow` section — then replaces it with
|
||||||
|
// O_TRUNC (felhom-host-install.sh:2396, :2449, :2579; the flag is :1246, defaulting empty at
|
||||||
|
// :256). So a rebuild leaves the marker and takes the seed, the hash still matches, this
|
||||||
|
// branch returns, and `escrow.pbs_storage_id` is never written again. The customer then
|
||||||
|
// cannot run the escrow ceremony AT ALL: handleEscrowPreflight fails the `pbs_storage_id`
|
||||||
|
// row and the wizard refuses, with no way forward from inside the product.
|
||||||
|
//
|
||||||
|
// A rebuild is only the case that was measured. The same hole opens for a hand-edited or
|
||||||
|
// restored config, which is the honest reason this is a seam fix rather than an installer
|
||||||
|
// fix — the seed must be a thing the loop asserts, not a thing it did once.
|
||||||
|
//
|
||||||
|
// COST: this runs on the converged path, i.e. every tick (60 s) forever. It is one small
|
||||||
|
// file read plus a JSON parse — no exec, no network, no Proxmox call — and seedEscrowStorageID
|
||||||
|
// returns early once the value matches. That is the whole reason it is affordable here.
|
||||||
|
//
|
||||||
|
// IT MUST NEVER UN-CONVERGE THE BOX: a failure is a Warn plus a message on the published
|
||||||
|
// status, exactly as finishConverged does it. No marker write, no state change, no retry
|
||||||
|
// storm — the early return below still happens either way.
|
||||||
|
msg := ""
|
||||||
|
if err := m.seedEscrowStorageID(block.StorageID); err != nil {
|
||||||
|
msg = "escrow.pbs_storage_id seed failed: " + err.Error() + " (set it manually before the ceremony)"
|
||||||
|
m.logger.Warn("pbsdr: " + msg)
|
||||||
|
}
|
||||||
|
m.setStatus(&hub.PBSDRStatus{State: mk.State, StorageID: block.StorageID, Namespace: block.Namespace, AppliedAt: mk.AppliedAt, Message: msg})
|
||||||
return // idempotent: this exact descriptor already converged
|
return // idempotent: this exact descriptor already converged
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,232 @@
|
|||||||
|
package pbsdr
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"go/ast"
|
||||||
|
"go/parser"
|
||||||
|
"go/printer"
|
||||||
|
"go/token"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-221 — the escrow seed must be ASSERTED on every converged tick, not remembered.
|
||||||
|
//
|
||||||
|
// These tests drive the real Apply() with a real temp-dir agent.json and a call-recording runner.
|
||||||
|
// Calling seedEscrowStorageID directly would prove nothing: the defect IS the early return in
|
||||||
|
// Apply, and a test that steps around it cannot see it.
|
||||||
|
|
||||||
|
// convergedMarker writes a marker whose hash matches the block, i.e. puts the manager on exactly
|
||||||
|
// the idempotent path where the seed used to be skipped.
|
||||||
|
func convergedMarker(t *testing.T, m *Manager, block *hub.WirePBSDR, state string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := m.writeState(m.markerPath(), marker{
|
||||||
|
Hash: descriptorHash(block), State: state, AppliedAt: "2026-08-08T00:00:00Z",
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("write marker: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func escrowStorageID(t *testing.T, cfgPath string) string {
|
||||||
|
t.Helper()
|
||||||
|
raw, err := os.ReadFile(cfgPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read config: %v", err)
|
||||||
|
}
|
||||||
|
var doc struct {
|
||||||
|
Escrow struct {
|
||||||
|
PBSStorageID string `json:"pbs_storage_id"`
|
||||||
|
} `json:"escrow"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||||
|
t.Fatalf("parse config: %v", err)
|
||||||
|
}
|
||||||
|
return doc.Escrow.PBSStorageID
|
||||||
|
}
|
||||||
|
|
||||||
|
// drBlock mirrors the existing valid fixture (manager_test.go descriptor()) so that validate()
|
||||||
|
// passes and Apply actually reaches the marker check — the branch these tests are about.
|
||||||
|
func drBlock(storageID string) *hub.WirePBSDR {
|
||||||
|
return &hub.WirePBSDR{
|
||||||
|
Enabled: true, StorageID: storageID, PBSTunnelIP: "10.77.0.1",
|
||||||
|
Datastore: "felhom-offsite", Namespace: "peti", TokenID: "felhom@pbs!peti",
|
||||||
|
Fingerprint: testFP,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SCENARIO A — the seed is re-asserted on a converged box, and nothing else happens.
|
||||||
|
//
|
||||||
|
// THIS IS THE TEST THAT MATTERS. It must fail against the pre-R-221 tree; if it passes there, it is
|
||||||
|
// not testing the defect and that is the finding.
|
||||||
|
func TestSeedReasserted_OnConvergedTick_WithZeroProxmoxCalls(t *testing.T) {
|
||||||
|
r := &fakeRunner{}
|
||||||
|
st := &fakeStorage{found: true, active: []bool{true}}
|
||||||
|
c := &fakeConsumer{}
|
||||||
|
m, cfgPath := newTestManager(t, r, st, c)
|
||||||
|
|
||||||
|
block := drBlock("felhom-pbs-dr")
|
||||||
|
convergedMarker(t, m, block, "applied")
|
||||||
|
|
||||||
|
// the shape a rebuild leaves behind: escrow section present, pbs_storage_id GONE
|
||||||
|
if got := escrowStorageID(t, cfgPath); got != "" {
|
||||||
|
t.Fatalf("precondition: config already carries a storage id %q", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
m.Apply(context.Background(), true, block)
|
||||||
|
|
||||||
|
if got := escrowStorageID(t, cfgPath); got != "felhom-pbs-dr" {
|
||||||
|
t.Errorf("the converged tick did not re-assert the seed: escrow.pbs_storage_id = %q, want %q.\n"+
|
||||||
|
"This is R-221: the marker survives a rebuild, the descriptor hash still matches, the early "+
|
||||||
|
"return fires and the seed never runs into the config that no longer has it — so the customer "+
|
||||||
|
"cannot run the escrow ceremony at all.", got, "felhom-pbs-dr")
|
||||||
|
}
|
||||||
|
|
||||||
|
// ...and the idempotent path is STILL idempotent. This assertion is not decorative: without it
|
||||||
|
// a "fix" that simply deletes the early return would pass the line above.
|
||||||
|
if calls := r.recorded(); len(calls) != 0 {
|
||||||
|
t.Errorf("a converged tick must execute ZERO Proxmox commands; got %d: %+v", len(calls), calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SCENARIO B — an operator's own different value survives, and the warning names both.
|
||||||
|
func TestSeedReasserted_NeverClobbersAnOperatorValue(t *testing.T) {
|
||||||
|
r := &fakeRunner{}
|
||||||
|
m, cfgPath := newTestManager(t, r, &fakeStorage{found: true, active: []bool{true}}, &fakeConsumer{})
|
||||||
|
|
||||||
|
if err := os.WriteFile(cfgPath, []byte(
|
||||||
|
`{"log_level":"info","escrow":{"posture":"zero_knowledge","pbs_storage_id":"operator-chosen"},`+
|
||||||
|
`"custom_unknown":{"keep":1}}`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
block := drBlock("hub-chosen")
|
||||||
|
convergedMarker(t, m, block, "applied")
|
||||||
|
|
||||||
|
m.Apply(context.Background(), true, block)
|
||||||
|
|
||||||
|
if got := escrowStorageID(t, cfgPath); got != "operator-chosen" {
|
||||||
|
t.Errorf("a value a person put there was overwritten by the descriptor: got %q, want %q", got, "operator-chosen")
|
||||||
|
}
|
||||||
|
// unknown keys must still round-trip
|
||||||
|
raw, _ := os.ReadFile(cfgPath)
|
||||||
|
if !strings.Contains(string(raw), "custom_unknown") {
|
||||||
|
t.Error("an unknown config key was dropped by the re-assert")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SCENARIO C — the ceremony preflight's live read sees the re-asserted value with NO restart.
|
||||||
|
//
|
||||||
|
// The preflight itself lives in internal/localapi and reads the file through config.Load; what this
|
||||||
|
// asserts is the half that belongs to this package: after a converged tick, THE FILE ON DISK carries
|
||||||
|
// the id, so any live re-read is green. The daemon is never restarted in this test because it is
|
||||||
|
// never started — which is the point.
|
||||||
|
func TestSeedReasserted_IsVisibleOnDiskImmediately(t *testing.T) {
|
||||||
|
r := &fakeRunner{}
|
||||||
|
m, cfgPath := newTestManager(t, r, &fakeStorage{found: true, active: []bool{true}}, &fakeConsumer{})
|
||||||
|
block := drBlock("felhom-pbs-dr")
|
||||||
|
convergedMarker(t, m, block, "applied")
|
||||||
|
|
||||||
|
// preflight's predicate BEFORE: storageID == "" → the row is NOT OK
|
||||||
|
if escrowStorageID(t, cfgPath) != "" {
|
||||||
|
t.Fatal("precondition")
|
||||||
|
}
|
||||||
|
m.Apply(context.Background(), true, block)
|
||||||
|
// preflight's predicate AFTER, from the same file the ceremony subprocess loads
|
||||||
|
if id := escrowStorageID(t, cfgPath); id == "" {
|
||||||
|
t.Error("the preflight row would still be NOT OK after a converged tick")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A seed failure must NEVER un-converge the box: no marker rewrite, no state change, and the status
|
||||||
|
// still reports the marker's converged state — with the failure surfaced as a message.
|
||||||
|
func TestSeedReassertFailure_DoesNotUnconverge(t *testing.T) {
|
||||||
|
r := &fakeRunner{}
|
||||||
|
m, cfgPath := newTestManager(t, r, &fakeStorage{found: true, active: []bool{true}}, &fakeConsumer{})
|
||||||
|
block := drBlock("felhom-pbs-dr")
|
||||||
|
convergedMarker(t, m, block, "applied")
|
||||||
|
markerBefore, err := os.ReadFile(m.markerPath())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// make the seed fail in a way it cannot recover from: unparseable config
|
||||||
|
if err := os.WriteFile(cfgPath, []byte(`{ this is not json`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
m.Apply(context.Background(), true, block)
|
||||||
|
|
||||||
|
after, err := os.ReadFile(m.markerPath())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("the marker was removed by a seed failure: %v", err)
|
||||||
|
}
|
||||||
|
if string(after) != string(markerBefore) {
|
||||||
|
t.Error("a seed failure rewrote the convergence marker — it must not touch state")
|
||||||
|
}
|
||||||
|
if calls := r.recorded(); len(calls) != 0 {
|
||||||
|
t.Errorf("a seed failure must not trigger Proxmox work; got %+v", calls)
|
||||||
|
}
|
||||||
|
st := m.Status()
|
||||||
|
if st == nil || st.State != "applied" {
|
||||||
|
t.Errorf("a seed failure must leave the box converged; status = %+v", st)
|
||||||
|
}
|
||||||
|
if st != nil && !strings.Contains(st.Message, "seed failed") {
|
||||||
|
t.Errorf("a seed failure must be surfaced on the status, got message %q", st.Message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SEAM WIRING — production must construct the manager with the live config path, or the whole seed
|
||||||
|
// leg is inert. Three shipped defects in this project were fully green while their seam was never
|
||||||
|
// wired, so this walks main.go's AST for the actual call rather than grepping: a commented-out call
|
||||||
|
// satisfies strings.Contains, and an AST walk cannot see a comment.
|
||||||
|
func TestProductionWiring_NewManagerGetsTheLiveConfigPath(t *testing.T) {
|
||||||
|
path := filepath.Join("..", "..", "cmd", "felhom-agent", "main.go")
|
||||||
|
fset := token.NewFileSet()
|
||||||
|
f, err := parser.ParseFile(fset, path, nil, 0) // comments not even collected
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse main.go: %v", err)
|
||||||
|
}
|
||||||
|
found := false
|
||||||
|
ast.Inspect(f, func(n ast.Node) bool {
|
||||||
|
call, ok := n.(*ast.CallExpr)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||||
|
if !ok || sel.Sel.Name != "NewManager" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if pkg, ok := sel.X.(*ast.Ident); !ok || pkg.Name != "pbsdr" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
// signature: (runner, px, hubc, stateDir, secretDir, configPath, logger)
|
||||||
|
if len(call.Args) < 6 {
|
||||||
|
t.Errorf("pbsdr.NewManager called with %d args, expected >= 6", len(call.Args))
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
var buf strings.Builder
|
||||||
|
if err := printNode(&buf, fset, call.Args[5]); err != nil {
|
||||||
|
t.Fatalf("print arg: %v", err)
|
||||||
|
}
|
||||||
|
got := buf.String()
|
||||||
|
if !strings.Contains(got, "SourcePath") {
|
||||||
|
t.Errorf("pbsdr.NewManager's configPath argument is %q, which is not the live config path.\n"+
|
||||||
|
"With an empty or wrong path seedEscrowStorageID returns nil immediately and the entire "+
|
||||||
|
"R-221 fix is inert while every test above still passes.", got)
|
||||||
|
}
|
||||||
|
found = true
|
||||||
|
return false
|
||||||
|
})
|
||||||
|
if !found {
|
||||||
|
t.Error("no pbsdr.NewManager call found in main.go — the manager is not constructed in production")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func printNode(w io.Writer, fset *token.FileSet, n ast.Node) error {
|
||||||
|
return printer.Fprint(w, fset, n)
|
||||||
|
}
|
||||||
@@ -49,6 +49,10 @@ GATES = [
|
|||||||
("reuse-refs", SHARED_REUSE, [ROOT], True),
|
("reuse-refs", SHARED_REUSE, [ROOT], True),
|
||||||
("instructions", SHARED_INSTRUCTIONS, [ROOT], True),
|
("instructions", SHARED_INSTRUCTIONS, [ROOT], True),
|
||||||
("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False),
|
("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False),
|
||||||
|
# R-273: the tag half of a release. Legs 1-2 need no network, so it runs in --fast too — the
|
||||||
|
# missing TAG is what actually broke every install, and the pre-push hook is the earliest place
|
||||||
|
# that can catch it.
|
||||||
|
("release-complete", os.path.join(ROOT, "scripts", "check-release-complete.py"), [], True),
|
||||||
]
|
]
|
||||||
|
|
||||||
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
|
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
|
||||||
|
|||||||
@@ -95,6 +95,25 @@ PROBE_CONFIG = "configs/felhom-agent.service"
|
|||||||
|
|
||||||
TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
|
TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
|
||||||
|
|
||||||
|
# THE retention number, read from the one file that owns it. A check and the policy it enforces
|
||||||
|
# must read the same number from the same place, or they drift and the drift looks like a defect
|
||||||
|
# in something else — which is exactly what happened on 2026-08-08/09 (R-287).
|
||||||
|
RETENTION_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json")
|
||||||
|
|
||||||
|
|
||||||
|
def retention_kept():
|
||||||
|
"""How many of the newest generic versions the registry is expected to still serve.
|
||||||
|
|
||||||
|
Fails CLOSED and LOUD: a missing or unreadable policy file makes the check INCONCLUSIVE
|
||||||
|
rather than silently unbounded. An unbounded check would re-create the red this fixed; a
|
||||||
|
silently-bounded one would be worse.
|
||||||
|
"""
|
||||||
|
with open(RETENTION_FILE, encoding="utf-8") as fh:
|
||||||
|
n = json.load(fh)["generic_versions_kept"]
|
||||||
|
if not isinstance(n, int) or n < 1:
|
||||||
|
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
|
||||||
|
return n
|
||||||
|
|
||||||
tried = []
|
tried = []
|
||||||
|
|
||||||
|
|
||||||
@@ -189,7 +208,28 @@ def main():
|
|||||||
print(" no v<semver> tags in this repo yet — nothing to check, and nothing proven")
|
print(" no v<semver> tags in this repo yet — nothing to check, and nothing proven")
|
||||||
print("\ncheck-published-versions: NOTHING TO CHECK")
|
print("\ncheck-published-versions: NOTHING TO CHECK")
|
||||||
return 0
|
return 0
|
||||||
print(" %d released version(s) to verify: %s" % (len(versions), ", ".join(versions)))
|
all_versions = versions
|
||||||
|
try:
|
||||||
|
keep = retention_kept()
|
||||||
|
except Exception as e:
|
||||||
|
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
|
||||||
|
|
||||||
|
# Bound the assertion to what the registry is expected to still hold. Sorted by SEMVER, not
|
||||||
|
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
|
||||||
|
def _key(v):
|
||||||
|
return tuple(int(x) for x in v.split("."))
|
||||||
|
versions = sorted(all_versions, key=_key)[-keep:]
|
||||||
|
dropped = [v for v in all_versions if v not in versions]
|
||||||
|
|
||||||
|
print(" %d released version(s); retention policy keeps the newest %d" % (len(all_versions), keep))
|
||||||
|
print(" verifying: %s" % ", ".join(versions))
|
||||||
|
if dropped:
|
||||||
|
# NEVER silent. A bounded check that does not say what it stopped covering is how a
|
||||||
|
# narrowing becomes permanent by accident.
|
||||||
|
print(" NOT ASSERTED (older than the retention window, and therefore not expected to be")
|
||||||
|
print(" downloadable): %s" % ", ".join(dropped))
|
||||||
|
print(" ^ these versions still have git TAGS and are still installable in the sense that")
|
||||||
|
print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
|
||||||
|
|
||||||
bad = []
|
bad = []
|
||||||
for v in versions:
|
for v in versions:
|
||||||
|
|||||||
@@ -0,0 +1,124 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""check-release-complete.py — the version at the head of CHANGELOG.md is a COMPLETE release.
|
||||||
|
|
||||||
|
THE DEFECT THIS IS A MACHINE FOR (2026-08-08/09, R-273). Agent v0.128.0 was built, tested,
|
||||||
|
CHANGELOG'd and published to the package registry — and its git tag was never pushed. The hub then
|
||||||
|
vouched it, and because felhom-host-install.sh fetches an agent's config files from
|
||||||
|
`raw/tag/v<version>/configs/`, EVERY fresh install and every reinstall died at step 5 of 8, as root,
|
||||||
|
on a virgin machine, for the better part of a day.
|
||||||
|
|
||||||
|
`scripts/release-agent.sh` already warns about exactly this, in as many words:
|
||||||
|
"a released version without a git tag 404s a box mid-install, as root"
|
||||||
|
The warning was there, it was correct, and the step was still missed. **So the fix is a machine and
|
||||||
|
not a reminder** — that is the whole point of this file.
|
||||||
|
|
||||||
|
WHAT IT ASSERTS, for the newest `## vX.Y.Z` in CHANGELOG.md:
|
||||||
|
1. a git tag `vX.Y.Z` EXISTS, and
|
||||||
|
2. it points at a commit that is an ANCESTOR OF (or equal to) the tip it was released from — a tag
|
||||||
|
parked on an unrelated commit is not a release, and
|
||||||
|
3. the generic package for X.Y.Z is DOWNLOADABLE.
|
||||||
|
|
||||||
|
(3) needs the network. (1) and (2) do not, and they are the half that actually failed — so this gate
|
||||||
|
is useful offline and says so rather than going quiet.
|
||||||
|
|
||||||
|
EXIT CODES, matching this repo's other gates: 0 clean, 1 convicted, 2 inconclusive. An unreachable
|
||||||
|
registry is INCONCLUSIVE for leg 3 only; legs 1 and 2 still run and can still convict.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
GITEA_BASE = os.environ.get("GITEA_BASE", "https://gitea.dooplex.hu").rstrip("/")
|
||||||
|
OWNER, PKG = "admin", "felhom-agent"
|
||||||
|
HEAD_RE = re.compile(r"^##\s+v?(\d+\.\d+\.\d+)\b", re.M)
|
||||||
|
|
||||||
|
|
||||||
|
def git(*args):
|
||||||
|
return subprocess.run(("git",) + args, cwd=ROOT, capture_output=True, text=True)
|
||||||
|
|
||||||
|
|
||||||
|
def head_version():
|
||||||
|
ch = os.path.join(ROOT, "CHANGELOG.md")
|
||||||
|
if not os.path.exists(ch):
|
||||||
|
return None
|
||||||
|
m = HEAD_RE.search(open(ch, encoding="utf-8").read())
|
||||||
|
return m.group(1) if m else None
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
print("check-release-complete — the newest CHANGELOG version is a complete release")
|
||||||
|
v = head_version()
|
||||||
|
if not v:
|
||||||
|
print(" no '## vX.Y.Z' heading in CHANGELOG.md — nothing to check, and nothing proven")
|
||||||
|
return 0
|
||||||
|
tag = "v" + v
|
||||||
|
print(" newest CHANGELOG version: %s" % tag)
|
||||||
|
|
||||||
|
problems, inconclusive = [], []
|
||||||
|
|
||||||
|
# ---- leg 1 + 2: the tag, and where it points. Offline-capable. -----------------------------
|
||||||
|
r = git("rev-parse", "-q", "--verify", "refs/tags/%s^{commit}" % tag)
|
||||||
|
if r.returncode != 0:
|
||||||
|
# A shallow CI clone has no tags of its own; ask the remote before convicting, so this
|
||||||
|
# gate does not fire on a clone shape rather than on a real defect.
|
||||||
|
ls = git("ls-remote", "--tags", "origin", "refs/tags/%s" % tag)
|
||||||
|
if ls.returncode != 0:
|
||||||
|
inconclusive.append("cannot reach origin to look for tag %s: %s"
|
||||||
|
% (tag, ls.stderr.strip()[:120]))
|
||||||
|
elif not ls.stdout.strip():
|
||||||
|
problems.append(
|
||||||
|
"TAG %s DOES NOT EXIST. The installer fetches this version's configs from\n"
|
||||||
|
" %s/%s/felhom-agent/raw/tag/%s/configs/ — without the tag every install\n"
|
||||||
|
" 404s mid-run, as root. Fix: git tag -a %s <released-commit> && git push origin %s"
|
||||||
|
% (tag, GITEA_BASE, OWNER, tag, tag, tag))
|
||||||
|
else:
|
||||||
|
print(" ok tag %s exists on origin (not in this shallow clone)" % tag)
|
||||||
|
else:
|
||||||
|
sha = r.stdout.strip()
|
||||||
|
anc = git("merge-base", "--is-ancestor", sha, "HEAD")
|
||||||
|
if anc.returncode == 0:
|
||||||
|
print(" ok tag %s -> %s, an ancestor of HEAD" % (tag, sha[:10]))
|
||||||
|
else:
|
||||||
|
problems.append("tag %s points at %s, which is NOT an ancestor of HEAD — a tag parked "
|
||||||
|
"on an unrelated commit is not a release" % (tag, sha[:10]))
|
||||||
|
|
||||||
|
# ---- leg 3: the package. Needs the network. ------------------------------------------------
|
||||||
|
url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
|
||||||
|
req = urllib.request.Request(url, method="HEAD")
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=25) as resp:
|
||||||
|
if resp.status == 200:
|
||||||
|
print(" ok package %s is downloadable" % v)
|
||||||
|
else:
|
||||||
|
problems.append("package %s returned HTTP %s at %s" % (v, resp.status, url))
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
if e.code == 404:
|
||||||
|
problems.append("PACKAGE %s IS NOT PUBLISHED (HTTP 404 at %s).\n"
|
||||||
|
" Fix: bash scripts/release-agent.sh %s" % (v, url, v))
|
||||||
|
else:
|
||||||
|
inconclusive.append("registry returned HTTP %s for %s" % (e.code, v))
|
||||||
|
except Exception as e:
|
||||||
|
inconclusive.append("registry unreachable (%s) — leg 3 not checked; legs 1-2 still ran" % e)
|
||||||
|
|
||||||
|
if problems:
|
||||||
|
print("\ncheck-release-complete: INCOMPLETE RELEASE")
|
||||||
|
for p in problems:
|
||||||
|
print(" - " + p)
|
||||||
|
return 1
|
||||||
|
if inconclusive:
|
||||||
|
print("\ncheck-release-complete: INCONCLUSIVE — an undetermined result is never a pass")
|
||||||
|
for i in inconclusive:
|
||||||
|
print(" - " + i)
|
||||||
|
return 2
|
||||||
|
print("\ncheck-release-complete: %s is tagged, placed and published." % tag)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
{
|
||||||
|
"_comment": [
|
||||||
|
"THE retention number for published agent artifacts. One file, read by everything that",
|
||||||
|
"depends on it, because a check and the policy it enforces must read the same number from the",
|
||||||
|
"same place or they drift — and the drift looks like a defect in something else.",
|
||||||
|
"",
|
||||||
|
"WHAT WENT WRONG WITHOUT IT (2026-08-08/09). The registry stopped serving felhom-agent",
|
||||||
|
"0.120.0 and older, while scripts/check-published-versions.py demanded that EVERY git tag",
|
||||||
|
"still be downloadable. Both rules are individually sensible; together they are impossible.",
|
||||||
|
"CI went red at a commit whose own run had been green the day before, on a true finding that",
|
||||||
|
"no one could act on. The red will return at the next publish unless the two read one number.",
|
||||||
|
"",
|
||||||
|
"HOW THE NUMBER WAS ARRIVED AT — stated honestly, because it is weaker than it looks.",
|
||||||
|
"generic_versions_kept is 10 because that is what the registry demonstrably holds today",
|
||||||
|
"(felhom-agent 0.121.0..0.128.0 = 10 versions, queried 2026-08-09). It is an OBSERVED state,",
|
||||||
|
"NOT a ruling anyone has been able to locate: no register row records a package prune, R-210",
|
||||||
|
"is WAITING-ON-OPERATOR and says 'Nothing was deleted; this is a list, not an action', and it",
|
||||||
|
"concerns local Docker images rather than this registry. Container packages currently hold 19",
|
||||||
|
"each, so there is no uniform ten-per-package cap visible either. See R-287.",
|
||||||
|
"",
|
||||||
|
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest",
|
||||||
|
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the",
|
||||||
|
"operator should confirm or replace the number — at which point this file changes and both",
|
||||||
|
"readers follow it in the same commit.",
|
||||||
|
"",
|
||||||
|
"THE DEEPER BOUND, recorded so a future session does not have to re-derive it: the principled",
|
||||||
|
"limit is the hub's vouched min_agent floor (0.127.0 on 2026-08-09). Nothing can install an",
|
||||||
|
"agent below it — the hub refuses to vouch one and boxes update to the floor — so a released",
|
||||||
|
"version below the floor being un-downloadable costs nothing real. Bounding on the floor would",
|
||||||
|
"be better than bounding on a count, and it needs the gate to read the hub, which is network",
|
||||||
|
"the gate does not have today. Filed as the follow-up in R-287.",
|
||||||
|
"",
|
||||||
|
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config",
|
||||||
|
"files from raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that",
|
||||||
|
"version at all — a strictly worse act than an un-downloadable binary."
|
||||||
|
],
|
||||||
|
"generic_versions_kept": 10,
|
||||||
|
"readers": [
|
||||||
|
"scripts/check-published-versions.py — bounds its assertion to the newest N versions",
|
||||||
|
"documentation/runbooks/registry-retention.md (felhom.eu) — the prune procedure"
|
||||||
|
],
|
||||||
|
"recorded": "2026-08-09",
|
||||||
|
"recorded_by": "CC, from the registry's observed state; NOT from a located operator ruling"
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user