Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| b228b44f0e | |||
| 91b9405c81 | |||
| 4c69c25b48 | |||
| c9013bb47d |
+30
-1
@@ -1,4 +1,33 @@
|
|||||||
## Unreleased — part of v0.153.0: ring 0 stages exactly the told kernel (R-898; `09` §3 decision 176) (2026-10-07)
|
## Unreleased (2026-10-08) — no OS leg after a household press; the after-boot kernel report carries the real ring (R-899); no „wrong code" when older packages were not checked (R-304) — ships with tomorrow's release
|
||||||
|
|
||||||
|
**Delivery: the agent binary only** — no root file changed.
|
||||||
|
|
||||||
|
- **R-899 (operator ruling 2026-10-08, option A):** `POST /backup?trigger=manual` (a „Mentés most" press, sent by the
|
||||||
|
controller from its next release) runs no OS leg after it: the leg belongs to the night, after the night's own copy.
|
||||||
|
Before, a press ran the leg at once (in the day; on 2026-10-07 08:51 demo-hp skipped it only by the 20-hour rule). A
|
||||||
|
request without the parameter (an older controller, the scheduled path) behaves as before.
|
||||||
|
`internal/localapi/server.go`; `TestAfterPrimaryBackup` gained two sub-cases (red-proved: ignoring `trigger`, the leg
|
||||||
|
ran once after a press).
|
||||||
|
- **R-304 — „wrong code" only when every earlier package was tried.** `POST /escrow/recover-offsite-password` answered
|
||||||
|
400 („the recovery code did not open the sealed bundle") whenever the current package and the TRIED earlier packages
|
||||||
|
refused the code — even when the hub withheld earlier packages (rows with no key material, rows over its serve cap), a
|
||||||
|
served package was malformed, the 6-attempt cap stopped the loop, or the retained list could not be read at all. Now
|
||||||
|
those cases answer **424** with `older_unchecked` (the count, -1 = unknown) and a sentence that does not call the code
|
||||||
|
wrong. `escrow.ErrRetainedUnchecked` / `RetainedUncheckedError`; the retained fetcher's second value is now every
|
||||||
|
withheld package (unopenable + truncated + malformed). Tests `TestR304_*` (real age crypto; red-proved: without the
|
||||||
|
check, four cases returned the wrong-code error) and a 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus`.
|
||||||
|
An older controller maps the unknown 424 to its neutral „we do not know why" sentence.
|
||||||
|
- **The „ring 1" label after a boot:** in the first second after a reboot the agent has not fetched the hub's block, and
|
||||||
|
the after-boot kernel reports (`judging`, `good`, `revert`, `fell_back`…) said ring 1 on a ring-0 box (demo-felhom,
|
||||||
|
2026-10-08 night). Now they read the fetched block, else the block the daemon saved on disk before the reboot (R-866),
|
||||||
|
else ring 1 as before. A label only — the hub's approval reads its own ring list. `kernelReportRing`,
|
||||||
|
`TestKernelReportRing_BeforeFirstFetch` (red-proved: ring 1, want 0).
|
||||||
|
|
||||||
|
## v0.153.0 — ring 0 stages exactly the told kernel (R-898; `09` §3 decision 176) (2026-10-07)
|
||||||
|
|
||||||
|
Released by `scripts/release-agent.sh`: binary sha256 `b204ebe6d65944f43b70dcfa4ae0dfb90da38c92e2ba6a38a1826e488eeb6630`, bundle
|
||||||
|
`6db216275eb0dd39188d93481a2045998a69e8cb7838ad908a58d65d9a6a9b57` (tag `v0.153.0` = `2d1e5d0`). Delivered (binary only — the
|
||||||
|
bundle files are unchanged from 0.152.0) to demo-hp, demo-felhom, Tester 1 on 2026-10-07 19:03.
|
||||||
|
|
||||||
**Delivery: the agent binary only** — no root file changed (the wrapper is unchanged; its tests gained two cases).
|
**Delivery: the agent binary only** — no root file changed (the wrapper is unchanged; its tests gained two cases).
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,10 @@
|
|||||||
# CONTEXT — felhom-agent working state
|
# CONTEXT — felhom-agent working state
|
||||||
|
|
||||||
|
> **2026-10-08 (day) — UNRELEASED on main, ships with tomorrow's release (decision 178).** R-899: `POST /backup?trigger=manual`
|
||||||
|
> runs no OS leg (`localapi/server.go`); after-boot kernel reports carry the saved block's ring (`kernelReportRing`). R-304:
|
||||||
|
> `escrow.ErrRetainedUnchecked` → HTTP 424 `older_unchecked` when not every earlier package was tried (withheld, caps,
|
||||||
|
> malformed, unreadable list). Binary-only delivery; no wrapper or root file changed.
|
||||||
|
|
||||||
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
|
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
|
||||||
|
|
||||||
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
|
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
|
||||||
|
|||||||
@@ -1,18 +1,16 @@
|
|||||||
# REPORT — agent v0.152.0: the kernel lane (2026-10-07)
|
# REPORT — agent, 2026-10-08 (day): R-899 and R-304 on main, unreleased
|
||||||
|
|
||||||
**What:** R-836, `09` §3 decision 172 — a new kernel boots ONCE through a flag on the ESP; a crash falls back to the old
|
**No release, no delivery today** (kernel night 8→9; `09` §3 decision 178). Binary-only; ships with tomorrow's release.
|
||||||
kernel by itself; a healthy boot (host health rule + the hub reached, 20 min) makes it the default; a booted-but-unhealthy
|
|
||||||
one is reverted ONCE. Design: `felhom.eu/documentation/architecture/11-os-updates.md` §5.11.
|
|
||||||
|
|
||||||
- Wrapper `configs/felhom-os-apply`: layer `kernel` (stage, kernel-reboot, kernel-boot, kernel-good, kernel-revert,
|
| Item | Commit | Tests | Red-proof |
|
||||||
kernel-cancel, kernel-status; R20–R23). Bundle: `/etc/grub.d/01_felhom_oneshot`, `/etc/grub.d/42_felhom_oneshot`.
|
|---|---|---|---|
|
||||||
- Agent `internal/osupdate/kernel.go`: the night step (told nights only), `KernelAfterBoot`, `KernelStepExecutor`.
|
| R-899 — `trigger=manual` (a press) runs no OS leg | `4c69c25` | `TestAfterPrimaryBackup` + 2 sub-cases | ignoring `trigger` → the leg ran once after a press (`[8200]`) |
|
||||||
- Tests: `KernelLane` (27), `KernelStepCannotLeaveTheBoxOff` (3), `TestKernel*` (13); red-proofs
|
| „ring 1" label in after-boot kernel reports (seen 2026-10-08 night, demo-felhom) | `4c69c25` | `TestKernelReportRing_BeforeFirstFetch` | `Block().Ring` → `ring 1, want 0` |
|
||||||
`felhom.eu/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`.
|
| R-304 — 424 `older_unchecked` instead of „did not open the sealed bundle" when earlier packages were not all tried | `91b9405` | `TestR304_*` (real age crypto), 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus` | check off → four cases returned the wrong-code error |
|
||||||
- Released `v0.152.0` (`d03ab7f`; binary `95ff4220…`, bundle `f0c2cec3…`) + step bundle `0.152.0-step1` (`0b71d32b…`).
|
|
||||||
Delivered by signed jobs to Tester 1, demo-hp, demo-felhom (binary → step bundle → bundle). Not vouched (the golden is
|
**Read before the change (read-only, demo-hp):** on 2026-10-07 08:51 the morning press DID reach the OS leg
|
||||||
behind; waiver to 2026-10-13).
|
(`osupdate: skipped — already ran tonight`, saved by the 20-hour rule only); `os-update-block.json` on demo-hp holds
|
||||||
- **Proven on the Tester 1 box** (`felhom.eu/documentation/audits/kernel-lane-2026-10-07/E/RESULT.md`): forced panic →
|
`ring 0` (the fallback the label fix reads).
|
||||||
fell_back by itself; held guest → one self-revert after 20 min; healthy → 7.0.14-22 the default.
|
|
||||||
- **Open:** the ring-0 night run (R-836 says where it stopped); R-898 (ring 0 stages the pending kernel, not exactly the
|
**Gates:** `go build/vet/test ./...` rc 0; `agent_gates.py --fast` rc 0. **CI:** job 1529 (`4c69c25`) success; job 1530
|
||||||
told one); R-897 (post-reboot drive re-bind races the first backup).
|
(`91b9405`) success.
|
||||||
|
|||||||
@@ -1904,11 +1904,15 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
|||||||
return nil, 0, ferr
|
return nil, 0, ferr
|
||||||
}
|
}
|
||||||
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
|
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
|
||||||
|
// R-304: every package the hub holds and the code will NOT be tried against — no key material,
|
||||||
|
// over the hub's cap, or malformed here. A refusal may call the code wrong only when this is 0.
|
||||||
|
withheld := resp.UnopenableCount + resp.TruncatedCount
|
||||||
for _, p := range resp.Packages {
|
for _, p := range resp.Packages {
|
||||||
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
|
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
|
||||||
if derr != nil || len(blob) == 0 {
|
if derr != nil || len(blob) == 0 {
|
||||||
// One malformed package must not sink the rest — the customer's code may open a
|
// One malformed package must not sink the rest — the customer's code may open a
|
||||||
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
|
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
|
||||||
|
withheld++
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
out = append(out, escrow.RetainedBlob{
|
out = append(out, escrow.RetainedBlob{
|
||||||
@@ -1918,7 +1922,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
|||||||
Index: p.Index,
|
Index: p.Index,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return out, resp.UnopenableCount, nil
|
return out, withheld, nil
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
srv, err := localapi.NewServer(localapi.Options{
|
srv, err := localapi.NewServer(localapi.Options{
|
||||||
|
|||||||
+50
-10
@@ -59,8 +59,26 @@ var (
|
|||||||
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
|
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
|
||||||
// which is the one fact the customer needs to recognise it.
|
// which is the one fact the customer needs to recognise it.
|
||||||
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
|
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
|
||||||
|
// ErrRetainedUnchecked — the code did NOT open the current package, and NOT every earlier package the hub
|
||||||
|
// holds for this host was tried (R-304, 2026-10-08): the retained list could not be fetched, the hub
|
||||||
|
// withheld rows (over its serve cap, or rows with no key material), a package was malformed, or the
|
||||||
|
// attempt cap stopped the loop. So „the code is wrong" is NOT known — it may be right for a package
|
||||||
|
// nobody tried. Distinct from a mistype for exactly the R-224 reason: never accuse the customer of
|
||||||
|
// something we did not check.
|
||||||
|
ErrRetainedUnchecked = errors.New("escrow: the recovery code did not open the current sealed package, and some earlier packages were NOT checked")
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// RetainedUncheckedError wraps ErrRetainedUnchecked with how many earlier packages went unchecked
|
||||||
|
// (-1 = unknown: the retained list itself could not be read). No secret.
|
||||||
|
type RetainedUncheckedError struct {
|
||||||
|
Unchecked int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *RetainedUncheckedError) Error() string {
|
||||||
|
return fmt.Sprintf("%s (unchecked=%d)", ErrRetainedUnchecked.Error(), e.Unchecked)
|
||||||
|
}
|
||||||
|
func (e *RetainedUncheckedError) Unwrap() error { return ErrRetainedUnchecked }
|
||||||
|
|
||||||
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
|
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
|
||||||
// carries no secret — not the code, not the bundle, not the repository password.
|
// carries no secret — not the code, not the bundle, not the repository password.
|
||||||
type RetainedMatch struct {
|
type RetainedMatch struct {
|
||||||
@@ -102,8 +120,10 @@ type RetainedBlob struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
|
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
|
||||||
// slice is a clean "none". R-311.
|
// slice is a clean "none". R-311. `withheld` counts the earlier packages the hub holds that are NOT in
|
||||||
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error)
|
// blobs — rows with no key material, rows over the hub's serve cap, and packages dropped as malformed
|
||||||
|
// (R-304): each is a package the code was never tried against.
|
||||||
|
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, withheld int, err error)
|
||||||
|
|
||||||
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
||||||
type OffsiteKeyRecoverer struct {
|
type OffsiteKeyRecoverer struct {
|
||||||
@@ -154,9 +174,14 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
|
|||||||
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
|
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
|
||||||
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
|
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
|
||||||
// about our own incuriosity, read by the customer as a statement about their code.
|
// about our own incuriosity, read by the customer as a statement about their code.
|
||||||
if m, ok := r.tryRetained(ctx, recoveryCode); ok {
|
m, ok, unchecked := r.tryRetained(ctx, recoveryCode)
|
||||||
|
if ok {
|
||||||
return "", &RetainedOpenedError{Match: m}
|
return "", &RetainedOpenedError{Match: m}
|
||||||
}
|
}
|
||||||
|
// R-304: only when EVERY earlier package the hub holds was tried may this stay a wrong code.
|
||||||
|
if unchecked != 0 {
|
||||||
|
return "", &RetainedUncheckedError{Unchecked: unchecked}
|
||||||
|
}
|
||||||
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
||||||
}
|
}
|
||||||
if bundle.ResticRepoPassword == "" {
|
if bundle.ResticRepoPassword == "" {
|
||||||
@@ -173,23 +198,38 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
|
|||||||
// function breaking is the behaviour we had before it existed.
|
// function breaking is the behaviour we had before it existed.
|
||||||
//
|
//
|
||||||
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
|
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
|
||||||
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) {
|
//
|
||||||
|
// R-304 (2026-10-08): it also returns how many earlier packages were NOT tried — `withheld` from the hub, plus
|
||||||
|
// the ones past the attempt cap — or -1 when the retained list could not be read at all. A nil FetchRetained
|
||||||
|
// (an agent wired without the lookup) reports 0: the pre-R-311 refusal, unchanged.
|
||||||
|
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool, int) {
|
||||||
if r.FetchRetained == nil {
|
if r.FetchRetained == nil {
|
||||||
return RetainedMatch{}, false
|
return RetainedMatch{}, false, 0
|
||||||
}
|
}
|
||||||
blobs, _, err := r.FetchRetained(ctx)
|
blobs, withheld, err := r.FetchRetained(ctx)
|
||||||
if err != nil || len(blobs) == 0 {
|
if err != nil {
|
||||||
return RetainedMatch{}, false
|
return RetainedMatch{}, false, -1
|
||||||
|
}
|
||||||
|
if withheld < 0 {
|
||||||
|
withheld = 0
|
||||||
|
}
|
||||||
|
if len(blobs) == 0 {
|
||||||
|
return RetainedMatch{}, false, withheld
|
||||||
}
|
}
|
||||||
limit := r.MaxRetainedTried
|
limit := r.MaxRetainedTried
|
||||||
if limit <= 0 {
|
if limit <= 0 {
|
||||||
limit = defaultMaxRetainedTried
|
limit = defaultMaxRetainedTried
|
||||||
}
|
}
|
||||||
|
unchecked := withheld
|
||||||
|
if len(blobs) > limit {
|
||||||
|
unchecked += len(blobs) - limit
|
||||||
|
}
|
||||||
for i, rb := range blobs {
|
for i, rb := range blobs {
|
||||||
if i >= limit {
|
if i >= limit {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
if len(rb.Blob) == 0 {
|
if len(rb.Blob) == 0 {
|
||||||
|
unchecked++
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
|
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
|
||||||
@@ -204,7 +244,7 @@ func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode strin
|
|||||||
// correct and must be told so — but the history behind it still cannot be reopened, and
|
// correct and must be told so — but the history behind it still cannot be reopened, and
|
||||||
// saying otherwise would be a promise this path cannot keep.
|
// saying otherwise would be a promise this path cannot keep.
|
||||||
HasResticPassword: bundle.ResticRepoPassword != "",
|
HasResticPassword: bundle.ResticRepoPassword != "",
|
||||||
}, true
|
}, true, 0
|
||||||
}
|
}
|
||||||
return RetainedMatch{}, false
|
return RetainedMatch{}, false, unchecked
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -115,8 +115,9 @@ func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The
|
// FAIL-SAFE — if the retained lookup itself fails, the lookup's error never reaches the customer and no
|
||||||
// worst outcome of this feature breaking is the behaviour we had before it.
|
// retained package is claimed. R-304 (2026-10-08) changed what stands instead: not the wrong-code refusal (the
|
||||||
|
// earlier packages were never tried, so „wrong" is not known) but ErrRetainedUnchecked with Unchecked = -1.
|
||||||
//
|
//
|
||||||
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
|
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
|
||||||
// gets a new, unexplained failure mode → this FAILS.
|
// gets a new, unexplained failure mode → this FAILS.
|
||||||
@@ -140,6 +141,10 @@ func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
|
|||||||
if containsStr(err.Error(), "hub exploded") {
|
if containsStr(err.Error(), "hub exploded") {
|
||||||
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
|
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
|
||||||
}
|
}
|
||||||
|
var ue *RetainedUncheckedError
|
||||||
|
if !errors.As(err, &ue) || ue.Unchecked != -1 {
|
||||||
|
t.Errorf("err = %v, want RetainedUncheckedError{-1} — the earlier packages were never tried (R-304)", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
|
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
|
||||||
@@ -228,3 +233,72 @@ func containsStr(hay, needle string) bool {
|
|||||||
return false
|
return false
|
||||||
})()
|
})()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── R-304 (2026-10-08) — „wrong code" only when every earlier package was tried ──────────────────────
|
||||||
|
//
|
||||||
|
// The consequence asserted: is the refusal a WRONG CODE (the only error the screen may answer with „check your
|
||||||
|
// typing")? It may be only when the hub withheld nothing and every served package was tried.
|
||||||
|
//
|
||||||
|
// RED-PROOF: make tryRetained return 0 for `unchecked` → the three „unchecked" cases return the plain refusal
|
||||||
|
// → they FAIL; the all-tried case keeps passing.
|
||||||
|
func TestR304_WrongCodeOnlyWhenEveryEarlierPackageWasTried(t *testing.T) {
|
||||||
|
ensureAge(t)
|
||||||
|
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "4444567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||||
|
other := sealBundle(t, IdentityBundle{ResticRepoPassword: "5555567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||||
|
const code = "a code that opens nothing whatsoever in this test"
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
blobs int
|
||||||
|
withheld int
|
||||||
|
limit int
|
||||||
|
unchecked int // 0 = must be a plain wrong code
|
||||||
|
}{
|
||||||
|
{"all tried, nothing withheld", 2, 0, 6, 0},
|
||||||
|
{"the hub withheld rows (no key material / over its cap)", 1, 3, 6, 3},
|
||||||
|
{"only withheld rows, nothing served", 0, 2, 6, 2},
|
||||||
|
{"the attempt cap stopped the loop", 5, 0, 2, 3},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
blobs := make([]RetainedBlob, 0, c.blobs)
|
||||||
|
for i := 0; i < c.blobs; i++ {
|
||||||
|
blobs = append(blobs, RetainedBlob{Blob: other, SupersededAt: "2026-08-01 00:00:00", Index: i})
|
||||||
|
}
|
||||||
|
_, err := OffsiteKeyRecoverer{
|
||||||
|
Fetch: fetcherFor(current),
|
||||||
|
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
|
||||||
|
return blobs, c.withheld, nil
|
||||||
|
},
|
||||||
|
MaxRetainedTried: c.limit,
|
||||||
|
}.RecoverOffsiteRepoPassword(context.Background(), code)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("a code that opens nothing succeeded")
|
||||||
|
}
|
||||||
|
var ue *RetainedUncheckedError
|
||||||
|
isUnchecked := errors.As(err, &ue)
|
||||||
|
if c.unchecked == 0 {
|
||||||
|
if isUnchecked {
|
||||||
|
t.Fatalf("every earlier package was tried — this IS a wrong code, got %v", err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !isUnchecked || ue.Unchecked != c.unchecked || !errors.Is(err, ErrRetainedUnchecked) {
|
||||||
|
t.Fatalf("err = %v, want RetainedUncheckedError{%d} — never call the code wrong when packages were not tried", err, c.unchecked)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A malformed (empty) served package counts as not tried.
|
||||||
|
func TestR304_EmptyServedPackageCountsAsUnchecked(t *testing.T) {
|
||||||
|
ensureAge(t)
|
||||||
|
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "6666567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||||
|
_, err := OffsiteKeyRecoverer{
|
||||||
|
Fetch: fetcherFor(current),
|
||||||
|
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: nil, SupersededAt: "2026-08-01 00:00:00"}),
|
||||||
|
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
|
||||||
|
var ue *RetainedUncheckedError
|
||||||
|
if !errors.As(err, &ue) || ue.Unchecked != 1 {
|
||||||
|
t.Fatalf("err = %v, want RetainedUncheckedError{1}", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ import (
|
|||||||
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
|
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
|
||||||
// and the gate sub-case fails.
|
// and the gate sub-case fails.
|
||||||
func TestAfterPrimaryBackup(t *testing.T) {
|
func TestAfterPrimaryBackup(t *testing.T) {
|
||||||
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
|
run := func(t *testing.T, failErr, path string) (calls []int, gateHeld bool) {
|
||||||
gate := &backup.InFlight{}
|
gate := &backup.InFlight{}
|
||||||
b := &fakeBackups{failErr: failErr}
|
b := &fakeBackups{failErr: failErr}
|
||||||
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
|
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
|
||||||
@@ -34,7 +34,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
|||||||
done <- struct{}{}
|
done <- struct{}{}
|
||||||
})
|
})
|
||||||
h := srv.Handler()
|
h := srv.Handler()
|
||||||
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
|
if do(t, h, "POST", path, "A", "").Code != http.StatusAccepted {
|
||||||
t.Fatal("POST /backup not accepted")
|
t.Fatal("POST /backup not accepted")
|
||||||
}
|
}
|
||||||
select {
|
select {
|
||||||
@@ -47,7 +47,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
|||||||
return calls, gateHeld
|
return calls, gateHeld
|
||||||
}
|
}
|
||||||
t.Run("success runs the leg under the gate", func(t *testing.T) {
|
t.Run("success runs the leg under the gate", func(t *testing.T) {
|
||||||
calls, held := run(t, "")
|
calls, held := run(t, "", "/backup")
|
||||||
if len(calls) != 1 {
|
if len(calls) != 1 {
|
||||||
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
|
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
|
||||||
}
|
}
|
||||||
@@ -56,8 +56,21 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
t.Run("a failed backup runs nothing", func(t *testing.T) {
|
t.Run("a failed backup runs nothing", func(t *testing.T) {
|
||||||
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
|
if calls, _ := run(t, "vzdump exploded", "/backup"); len(calls) != 0 {
|
||||||
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
|
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
// R-899: a household press is not the night's backup — no OS leg after it. Same fake, same successful backup as
|
||||||
|
// the first sub-case; only the query differs. Red-proof: make handleBackup ignore `trigger` and this sub-case
|
||||||
|
// fails with the leg run once.
|
||||||
|
t.Run("a manual press runs nothing", func(t *testing.T) {
|
||||||
|
if calls, _ := run(t, "", "/backup?trigger=manual"); len(calls) != 0 {
|
||||||
|
t.Fatalf("the OS leg ran after a manual press: %v", calls)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("the scheduled path with the new query still runs the leg", func(t *testing.T) {
|
||||||
|
if calls, _ := run(t, "", "/backup?trigger=night"); len(calls) != 1 {
|
||||||
|
t.Fatalf("the leg ran %d time(s) after a non-manual backup, want 1", len(calls))
|
||||||
|
}
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -127,6 +127,22 @@ func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Req
|
|||||||
"retained_has_restic_pw": match.HasResticPassword,
|
"retained_has_restic_pw": match.HasResticPassword,
|
||||||
},
|
},
|
||||||
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
|
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
|
||||||
|
// ── R-304 (2026-10-08) — NOT EVERY EARLIER PACKAGE WAS CHECKED. ─────────────────────────
|
||||||
|
//
|
||||||
|
// The current package refused the code, no retained package opened it — and at least one earlier
|
||||||
|
// package the hub holds was never tried (or the list could not be read). Saying „the code is
|
||||||
|
// wrong" here would claim a check that did not happen. 424 (Failed Dependency): the verdict
|
||||||
|
// depends on packages we could not try. The controller classifies on the status, never on this
|
||||||
|
// sentence; an older controller maps an unknown status to its neutral „we do not know why".
|
||||||
|
case errors.Is(err, escrow.ErrRetainedUnchecked):
|
||||||
|
n := -1
|
||||||
|
var ue *escrow.RetainedUncheckedError
|
||||||
|
if errors.As(err, &ue) {
|
||||||
|
n = ue.Unchecked
|
||||||
|
}
|
||||||
|
s.logger.Warn("local-api: offsite key recovery: the code did not open the current package and earlier packages were NOT all checked — not reported as a wrong code (R-304)", "vmid", vmid, "unchecked", n)
|
||||||
|
writeStatus(w, http.StatusFailedDependency, false, map[string]any{"older_unchecked": n},
|
||||||
|
"the recovery code did not open the current sealed package, and earlier packages the hub holds were not all checked — the code may belong to one of them; nothing was written")
|
||||||
case errors.Is(err, escrow.ErrNoResticPassword):
|
case errors.Is(err, escrow.ErrNoResticPassword):
|
||||||
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
||||||
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
||||||
|
|||||||
@@ -54,6 +54,13 @@ func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
|
|||||||
wantStatus: 404,
|
wantStatus: 404,
|
||||||
mustNotSay: []string{"did not open"},
|
mustNotSay: []string{"did not open"},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
// R-304: earlier packages were not all tried — never „did not open the sealed bundle" (the wrong-code words).
|
||||||
|
name: "earlier packages not all checked — not a wrong code",
|
||||||
|
err: &escrow.RetainedUncheckedError{Unchecked: 2},
|
||||||
|
wantStatus: 424,
|
||||||
|
mustNotSay: []string{"did not open the sealed bundle", "could not be fetched"},
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "the bundle predates the repository-password field",
|
name: "the bundle predates the repository-password field",
|
||||||
err: escrow.ErrNoResticPassword,
|
err: escrow.ErrNoResticPassword,
|
||||||
|
|||||||
@@ -825,6 +825,10 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
key := backupJobKey{vmid: vmid, target: tier.TargetID}
|
key := backupJobKey{vmid: vmid, target: tier.TargetID}
|
||||||
|
// R-899 (operator ruling 2026-10-08): a household press („Mentés most") is not the night's backup. A controller
|
||||||
|
// that knows sends `trigger=manual`; then the OS leg does not follow (it belongs to the night, after the night's
|
||||||
|
// own copy). An older controller sends nothing and keeps the old behaviour.
|
||||||
|
manual := r.URL.Query().Get("trigger") == "manual"
|
||||||
|
|
||||||
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
|
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
|
||||||
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
|
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
|
||||||
@@ -926,7 +930,9 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
|||||||
}
|
}
|
||||||
s.finishJob(key, jobID, b)
|
s.finishJob(key, jobID, b)
|
||||||
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
|
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
|
||||||
if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
|
if b.Success && tier.Primary && s.afterPrimaryBackup != nil && manual {
|
||||||
|
s.logger.Info("local-api: no OS leg after a manual backup — it follows the night's own backup (R-899)", "vmid", vmid, "job", jobID)
|
||||||
|
} else if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
|
||||||
s.afterPrimaryBackup(base, vmid)
|
s.afterPrimaryBackup(base, vmid)
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|||||||
@@ -51,6 +51,24 @@ type KernelView struct {
|
|||||||
VMID int `json:"vmid"` // the customer guest the step was staged for (the health rule's guest)
|
VMID int `json:"vmid"` // the customer guest the step was staged for (the health rule's guest)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// kernelReportRing is the ring an after-boot report carries. In the first second after a boot the agent has not
|
||||||
|
// fetched the hub's block yet, and Block() then answers ring 1 — so a ring-0 box's „judging" report said ring 1
|
||||||
|
// (seen on demo-felhom, 2026-10-08 night, `audits/kernel-night-2026-10-07/readback/`). Order: the fetched block; the
|
||||||
|
// block the daemon saved on disk before the reboot (R-866); ring 1 as before. A label only: the hub's approval reads its
|
||||||
|
// own ring list. Pinned by TestKernelReportRing_BeforeFirstFetch.
|
||||||
|
func (l *Leg) kernelReportRing() int {
|
||||||
|
l.mu.Lock()
|
||||||
|
fetched := l.block
|
||||||
|
l.mu.Unlock()
|
||||||
|
if fetched != nil {
|
||||||
|
return fetched.Ring
|
||||||
|
}
|
||||||
|
if b, _, ok := LoadSavedBlock(l.planDir()); ok && b != nil {
|
||||||
|
return b.Ring
|
||||||
|
}
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
func parseKernel(raw json.RawMessage) KernelView {
|
func parseKernel(raw json.RawMessage) KernelView {
|
||||||
var v KernelView
|
var v KernelView
|
||||||
_ = json.Unmarshal(raw, &v)
|
_ = json.Unmarshal(raw, &v)
|
||||||
@@ -190,7 +208,7 @@ func (l *Leg) KernelAfterBoot(ctx context.Context, vmid int, j KernelJudge) Repo
|
|||||||
if vmid <= 0 {
|
if vmid <= 0 {
|
||||||
vmid = v.VMID // after a boot the guest may not run yet — the step's own record names it
|
vmid = v.VMID // after a boot the guest may not run yet — the step's own record names it
|
||||||
}
|
}
|
||||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, Mode: "kernel-boot",
|
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-boot",
|
||||||
ReleaseID: v.To, Kernel: rawOrNil(wr.Kernel)}
|
ReleaseID: v.To, Kernel: rawOrNil(wr.Kernel)}
|
||||||
switch wr.KernelEvent {
|
switch wr.KernelEvent {
|
||||||
case "fell_back":
|
case "fell_back":
|
||||||
@@ -240,7 +258,7 @@ func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelV
|
|||||||
for {
|
for {
|
||||||
if !hubReached && l.Hub != nil {
|
if !hubReached && l.Hub != nil {
|
||||||
// the hub's reachability IS this report reaching it (and the operator sees the box is back on the new kernel)
|
// the hub's reachability IS this report reaching it (and the operator sees the box is back on the new kernel)
|
||||||
body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid,
|
body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||||
Mode: "kernel-boot", ReleaseID: v.To, Outcome: "judging", Kernel: mustRaw(v)})
|
Mode: "kernel-boot", ReleaseID: v.To, Outcome: "judging", Kernel: mustRaw(v)})
|
||||||
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||||
if err := l.Hub.PostOSReport(rctx, body); err == nil {
|
if err := l.Hub.PostOSReport(rctx, body); err == nil {
|
||||||
@@ -280,7 +298,7 @@ func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelV
|
|||||||
return Report{}
|
return Report{}
|
||||||
}
|
}
|
||||||
// not healthy by the deadline: tell the hub (best effort), then ONE self-revert into the old kernel
|
// not healthy by the deadline: tell the hub (best effort), then ONE self-revert into the old kernel
|
||||||
rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid,
|
rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||||
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "health_failed", HealthReason: why + " — reverting to " + v.From,
|
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "health_failed", HealthReason: why + " — reverting to " + v.From,
|
||||||
Kernel: mustRaw(v)})
|
Kernel: mustRaw(v)})
|
||||||
lg.Error("osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel", "reason", why,
|
lg.Error("osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel", "reason", why,
|
||||||
@@ -289,7 +307,7 @@ func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelV
|
|||||||
if err != nil || wr.refused() || wr.failed() {
|
if err != nil || wr.refused() || wr.failed() {
|
||||||
lg.Error("osupdate: kernel self-revert did not start — the box stays on the new kernel; the operator decides",
|
lg.Error("osupdate: kernel self-revert did not start — the box stays on the new kernel; the operator decides",
|
||||||
"err", err, "refused", string(firstRaw(wr.Refused, wr.Failed)))
|
"err", err, "refused", string(firstRaw(wr.Refused, wr.Failed)))
|
||||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid,
|
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||||
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "revert_failed", Refused: firstRaw(wr.Refused, wr.Failed),
|
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "revert_failed", Refused: firstRaw(wr.Refused, wr.Failed),
|
||||||
HealthReason: "the self-revert did not start"})
|
HealthReason: "the self-revert did not start"})
|
||||||
}
|
}
|
||||||
@@ -298,7 +316,7 @@ func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelV
|
|||||||
|
|
||||||
func (l *Leg) kernelGood(ctx context.Context, runID string, vmid int, v KernelView, start time.Time, lg *slog.Logger) Report {
|
func (l *Leg) kernelGood(ctx context.Context, runID string, vmid int, v KernelView, start time.Time, lg *slog.Logger) Report {
|
||||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-good", vmid, nil))
|
wr, err := l.call(ctx, runID, kernelPlan("kernel-good", vmid, nil))
|
||||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, Mode: "kernel-good",
|
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-good",
|
||||||
ReleaseID: v.To}
|
ReleaseID: v.To}
|
||||||
switch {
|
switch {
|
||||||
case err != nil:
|
case err != nil:
|
||||||
|
|||||||
@@ -0,0 +1,32 @@
|
|||||||
|
package osupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// After a boot the agent has not fetched the hub's block yet; the after-boot kernel report must still carry the
|
||||||
|
// box's real ring (seen 2026-10-08: a ring-0 box's „judging" report said ring 1). Red-proof: return Block().Ring
|
||||||
|
// from kernelReportRing and the saved-block case fails with 1.
|
||||||
|
func TestKernelReportRing_BeforeFirstFetch(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
l := &Leg{PlanDir: dir}
|
||||||
|
if got := l.kernelReportRing(); got != 1 {
|
||||||
|
t.Fatalf("nothing fetched, nothing saved: ring %d, want 1 (the old default)", got)
|
||||||
|
}
|
||||||
|
l.saveBlock(&hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
l2 := &Leg{PlanDir: dir} // a fresh daemon after the reboot: no block fetched yet
|
||||||
|
if got := l2.kernelReportRing(); got != 0 {
|
||||||
|
t.Fatalf("ring-0 block saved before the reboot: ring %d, want 0", got)
|
||||||
|
}
|
||||||
|
l2.SetBlock(&hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||||
|
if got := l2.kernelReportRing(); got != 1 {
|
||||||
|
t.Fatalf("a fetched block wins over the saved one: ring %d, want 1", got)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(filepath.Join(dir, SavedBlockFile)); err != nil {
|
||||||
|
t.Fatalf("saved block missing: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user