Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
776025c824 | ||
|
|
41d5eb8051 | ||
|
|
8bd137787e | ||
|
|
39db03065a | ||
|
|
ba409de4a2 | ||
|
|
a4f1280399 | ||
|
|
ecbb0cc479 | ||
|
|
50eeccc986 | ||
|
|
808cdb36ad | ||
|
|
84726c9f27 |
No files matched your search
+55
-1
@@ -1,4 +1,58 @@
|
||||
## Unreleased — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED
|
||||
## Unreleased (2026-11-10) — R-1368 part 2: the self-update folder keeps what it needs and no more — COMMITTED, NOT RELEASED (no version, no build, no vouch; it waits for a release session)
|
||||
|
||||
- **The defect (measured 2026-11-10, space list place H12).** `Execute` downloads each new agent to
|
||||
`<state dir>/selfupdate/felhom-agent-<version>` and removed that file only when the download or its checksum failed.
|
||||
A good update left it for ever: 49 / 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1.
|
||||
- **What the folder is needed for (read first: `configs/felhom-selfupdate-guarded`).** The root wrapper reads the staged
|
||||
file once, copies it to its own root-owned folder and flips from that copy. The rollback needs `pending.json` (in the
|
||||
folder) and `/usr/local/bin/felhom-agent.prev` (outside it); the commit needs `pending.json`. No staged binary.
|
||||
- **The fix (`internal/selfupdate/prune.go`).** After the wrapper's `commit` returned success, `MaybeCommit` removes
|
||||
every staged binary but the running version's. Nothing is removed while a pending marker exists or cannot be read,
|
||||
at a plain start, after a rollback (the failed version's binary stays to be read), or when the running version is
|
||||
not a version this code can name a file for. Only a regular file with exactly the name `Execute` writes is removed,
|
||||
one `os.Remove` per name; a failed removal is logged and never fails the commit. A download and a prune take turns
|
||||
(`stagingMu`).
|
||||
- **Tests `TestPrune_*` (6).** After six updates the folder holds one binary and every bystander (unknown names, a
|
||||
folder and a symlink that carry a staged name); a pending or unreadable marker refuses; only a healthy commit prunes;
|
||||
the name rule follows `versionRe`; the wrapper's `commit` and `rollback` blocks name no staged binary; and the
|
||||
wrapper's REAL `commit` and `rollback` verbs, run from a sandboxed copy, still put the previous binary back after
|
||||
four pruned updates. **Red seen three times:** no prune call; no pending refusal; no regular-file check.
|
||||
- **Not reached.** `/usr/local/bin/felhom-agent.bak-*` (place H13): root-owned, in a folder the agent cannot write, made
|
||||
by a person. R-1368 part 4 stays open. A box gets the clean-up at its first good update AFTER the one that brings
|
||||
this code; nothing was run on a box.
|
||||
|
||||
## v0.172.1 (2026-11-07) — proxmox-grub joins the boot-chain lane: the nightly pve step stops being refused R6 — RELEASED and VOUCHED 2026-11-07 (release-agent.sh from `39db030`; agent sha256 505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8, bundle 3e7acaee4500893bfaf567b1d427790ef89bf41f08cb6a1419ee528aeb40ddc3)
|
||||
|
||||
- **The defect (measured on demo-felhom, 2026-11-05, 11-06, 11-07).** Proxmox published `proxmox-grub` 2.12-9+pmx4. It
|
||||
depends on `grub-efi-amd64` of the same version. `HOST_SLOW_RE` matches `grub` only at the START of a name, so
|
||||
`proxmox-grub` landed in the ring-0 pve plan; the simulation then pulled `grub-efi-amd64`, which is not in the plan,
|
||||
and `felhom-os-apply` refused the WHOLE pve step (R6 „the plan would touch grub-efi-amd64") every night — 34 Proxmox
|
||||
userspace upgrades held with it, and an `os_update_failed` error mail each morning. The same class as the 2026-10-09
|
||||
`proxmox-secure-boot-support` fix. Read-only proof on the box: `apt-get -s install --only-upgrade proxmox-grub` →
|
||||
`Inst grub-efi-amd64 …pmx4`.
|
||||
- **The fix.** `proxmox-grub` joins `HOST_SLOW_RE`: GRUB's meta-package now waits with the rest of the boot chain (the
|
||||
slow lane, the operator's signed step), and the pve step runs without it.
|
||||
- **Test** `test_ring0_pending_pve_leaves_proxmox_grub_out`; red seen before the fix (proxmox-grub entered the plan), green
|
||||
after; the wrapper suite 149 of 149.
|
||||
- **Delivered 2026-11-07 (operator, signed jobs):** demo-felhom, demo-hp, Tester 1 — `agent_update` committed
|
||||
(23:05/23:13/23:13 CET), then the bundle: `BUNDLE DONE agent=0.172.1 written=1 same=30 self-check=ok`, capability probe
|
||||
77/77 on each. `pinned.day0_agent` = 0.172.1 (D50).
|
||||
- **Delivery.** `configs/` changed, so a box needs BOTH signed jobs: `agent_update` 0.172.1, then `agent_config_update`
|
||||
0.172.1 with the bundle sha. Not vouched by CC. Bundle vs 0.170.0 (what the boxes run): no path added or removed, only
|
||||
`felhom-os-apply` changed — so no step bundle is needed (R-880). Both downloads re-checked by sha.
|
||||
|
||||
## v0.172.0 (2026-11-03) — R-1302: the first grant of a tier is no loss — RELEASED and VOUCHED 10:18:10 CET (release-agent.sh from a clean clone of `ecbb0cc`; agent sha256 2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7, config bundle cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce; configs/ unchanged since 0.171.0, so the signed delivery was agent_update alone)
|
||||
|
||||
- **R-1302 — the first grant of a tier is no loss.** A fresh install arms the whole-guest tier before its storage exists, so the
|
||||
first probe found no grant, repaired it and logged ERROR „GRANT WAS MISSING … investigate the loss (R-190)" plus a DEGRADED
|
||||
capability, on every new box (VM 9375, 2026-10-26). The agent now remembers which tiers have EVER held their grant
|
||||
(`<state dir>/store-grant-seen.json`, written by a healthy probe). A first grant logs INFO „granted" and reports healthy; a
|
||||
grant that existed and vanished keeps R-190's ERROR and its one degraded cycle. A doubt keeps the ERROR: an unreadable or
|
||||
corrupt file, or no file path. A box updated to this agent records its tiers at the first healthy probe, so a loss in the
|
||||
one probe before that would read as a first grant. Tests `TestR1302_*`; red seen with the first-grant branch removed and on
|
||||
three more mutants.
|
||||
|
||||
## v0.171.0 (2026-10-30) — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED — RELEASED (release-agent.sh from a clean clone of `bd18347`; agent sha256 c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5, config bundle 95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b; binary-only signed agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-30 10:47:36 CET; `pinned.day0_agent` = 0.171.0). Live on Tester 1: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the restart 09:31:22 → healthy 09:31:24; the hub mailed `controller_unresponsive` and `controller_unresponsive_restart` (hub 0.178.0), not `controller_restarted_by_agent`
|
||||
|
||||
- **The measurement (operator brief 2026-10-29 Part 2, read 2026-10-30).** Docker's own health record of the controller
|
||||
on Tester 1 and demo-hp, every probe from 2026-10-29 11:12 to 2026-10-30 07:00 UTC: 4 733 probes through the night
|
||||
|
||||
+17
-1
@@ -1,12 +1,28 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: the self-update folder cleans itself (R-1368 part 2).** After a healthy `commit` the agent removes every staged binary but the running version's (`internal/selfupdate/prune.go`; tests `TestPrune_*`, the wrapper's real `rollback` verb run in a sandbox). No version bump, no build, no vouch: it rides the next agent release. A box cleans its folder at the first good update AFTER that release. Not reached: `/usr/local/bin/felhom-agent.bak-*` (root-owned, R-1368 part 4). The old-kernel clean-up is a design only (felhom.eu `11-os-updates.md` §5.11.1, proven by hand on demo-hp); nothing of it is in this repo.
|
||||
|
||||
> **2026-11-07 (attended) — v0.172.1 RELEASED and VOUCHED; on the three test boxes (agent + bundle, 77/77).** `proxmox-grub` joins `HOST_SLOW_RE` (the N100's ring-0 pve step was refused R6 three nights: proxmox-grub pmx4 pulls grub-efi-amd64). sha `505a0594…`, bundle `3e7acaee…` (only `felhom-os-apply` changed vs 0.170.0, no step bundle). Done: vouch, both signed jobs on the three boxes, `pinned.day0_agent` = 0.172.1. Next: the N100's pve step on its next night (positive line: `os-apply: DONE` for `layer=pve:9201`, no R6).
|
||||
|
||||
> **2026-11-03 (day, attended) — v0.172.0 RELEASED and VOUCHED (10:18:10 CET), R-1302.** `2c0222cd…`, bundle `cc34b0bc…`, `pinned.day0_agent` = 0.172.0; on the three test boxes by signed `agent_update`. The appliance path pre-positions the grant, so a plain first start logs neither line; staged live both ways (felhom.eu `audits/release-2026-11-03/part3/`).
|
||||
>
|
||||
> **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
|
||||
> but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
|
||||
> alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
|
||||
> refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
|
||||
> (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
|
||||
> carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
|
||||
> `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
|
||||
> updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
|
||||
> `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
|
||||
|
||||
> **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
|
||||
> `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
|
||||
> sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
|
||||
> (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
|
||||
> `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
|
||||
> (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
|
||||
> **R-1344 (the stuck-controller restart) is still ALARM ONLY** — the night's health log is read on 2026-10-30.
|
||||
> **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
|
||||
|
||||
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
|
||||
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
|
||||
|
||||
@@ -1,36 +1,23 @@
|
||||
# REPORT — v0.170.0 released and vouched: „Reboot guest" from the hub (D65 = A, 2026-10-29 evening, attended)
|
||||
# REPORT — the self-update folder cleans itself (R-1368 part 2) — committed, NOT released (2026-11-10, operator brief)
|
||||
|
||||
**Plain words first.** The operator can now reboot a box's guest from the hub's host page, beside „Restart controller".
|
||||
The agent does it (`pct reboot` of that guest — never the host). It refuses during a backup, a restore, an update or a
|
||||
controller swap, more than once an hour, and under the restart brake. The result says when the guest and the controller
|
||||
were back. The controller that starts after the reboot tells the household for a day that Felhom support restarted
|
||||
the box.
|
||||
**Plain words first.** Each agent update left its downloaded file on the box for ever: 49 files and 740 MB on demo-hp.
|
||||
The agent now removes the old files after an update that ended well. It keeps the file of the version that runs.
|
||||
This is a commit only. No version, no build, no vouch. No box has it. It waits for the next agent release.
|
||||
|
||||
## Commits
|
||||
The full report of the run is in `felhom.eu`: `REPORT-endings-kernels-fixes-2026-11-10.md`.
|
||||
|
||||
| Commit | CI (job id) | What |
|
||||
|---|---|---|
|
||||
| `ca15a31` | 2520, 2524 green | `reboot_guest` in the closed list (`opreboot.go`): guards (provisioned, running, not locked, not parked; the heavy-operation gate held until the guest is up; `jobBlocker`; the controller's busy file, fresh ≤ 90 s, fail closed on any read error but „no file"; one per hour per guest, persisted; the restart brake counts it); `rebootInFlight` keeps the guest-power watchdog and the supervisor away; the result with both times; `GET /guest/operator-reboot` for the household line; one new read-only sudo vector (`cat` of the controller's busy file) + capability `controller-busy-read`. 19 mutants, each killed (one survivor found and fixed: a duplicate swap check). |
|
||||
| `6578a8b` | 2525 green | v0.170.0 CHANGELOG heading; `pinned.day0_agent` = 0.170.0 (D50), after the vouch. |
|
||||
| | |
|
||||
|---|---|
|
||||
| Change | `internal/selfupdate/prune.go` (new); the call in `MaybeCommit` (`commit.go`); one lock in `Execute` (`executor.go`) |
|
||||
| Rule | After the wrapper's `commit` returned success: remove every staged binary but the running version's |
|
||||
| Never removed | anything while `pending.json` exists or cannot be read; anything at a plain start or after a rollback; `pending.json`; a folder; a symlink; a name the code does not know |
|
||||
| Rollback | Needs `pending.json` and `/usr/local/bin/felhom-agent.prev`. It reads no staged binary (`configs/felhom-selfupdate-guarded`, read first) |
|
||||
| Tests | `TestPrune_*`, 6 tests. One runs the wrapper's real `commit` and `rollback` verbs from a sandboxed copy |
|
||||
| Red seen | 3 times: no prune call; no pending refusal; no regular-file check |
|
||||
| Gates | `python3 scripts/agent_gates.py --fast`: all OK |
|
||||
| Not reached | `/usr/local/bin/felhom-agent.bak-*`: root-owned, made by a person. R-1368 part 4 stays open |
|
||||
| Not proven | On a box. A box cleans its folder at the first good update AFTER the release that brings this code |
|
||||
| Machines | None touched by this change |
|
||||
|
||||
## Release
|
||||
|
||||
- `release-agent.sh 0.170.0` from a clean clone of `ca15a31`: sha256 `da8d12a05ef607c338b9ba307e7a81b18253a3188721bbd8421ba36658a22404`,
|
||||
bundle `03f2ff6442fc3ca600b1f25f26767ce4da466a770a258f3e362cf95f6e76b543`, tag `v0.170.0`, verified by download.
|
||||
- configs/ changed, so per box a signed `agent_config_update` AND `agent_update` (Tester 1, demo-hp, demo-felhom): all
|
||||
six COMPLETED; each box „config bundle INSTALLED", capability probe 76/76 ok, „update committed".
|
||||
- Vouched by the operator 2026-10-29 18:57:48 CET (the save took 38.3 s — felhom.eu R-1346).
|
||||
|
||||
## Live proofs (Tester 1, through the hub endpoint the button posts, CC's `cc-hosts` token; controller 0.349.0-rc.1)
|
||||
|
||||
- **Refused during an off-site run:** `offsite_backup_now` pressed 18:03:36 UTC → the controller's busy file 11 s later →
|
||||
„Reboot guest" → „refused: the controller is busy — a backup, a restore or an off-site run is running"; the guest kept
|
||||
its 13 h uptime.
|
||||
- **The reboot:** pressed 18:05:08 UTC → „done — rebooted; the guest was up after 67 s, the controller answered after
|
||||
79 s", on the hub 88 s after the press. Box walk after: 10 containers, none unhealthy, 0 errors, login and file manager
|
||||
200. The household dashboard: „A Felhom ügyfélszolgálata 19:05-kor újraindította a dobozt."
|
||||
- **A second press inside the hour:** „refused: guest 9201 was rebooted from the hub at 18:05 UTC — at most one reboot
|
||||
per hour (next from 19:05 UTC)".
|
||||
- The release-build reboot: see felhom.eu `REPORT-release-2026-10-29c.md`.
|
||||
|
||||
Evidence: felhom.eu `documentation/audits/release-2026-10-29c/`.
|
||||
**Next.** A release session builds and vouches it. After two updates on a test box, read the folder: it must hold one
|
||||
binary.
|
||||
@@ -35,7 +35,7 @@
|
||||
| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
|
||||
| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
|
||||
| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
|
||||
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
|
||||
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()`. **The staging folder cleans itself** (`prune.go`, R-1368): after a healthy `commit` only the running version's staged binary stays; a new file kept in `<state dir>/selfupdate/` must NOT be named `felhom-agent-<version>` or the prune removes it |
|
||||
| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
|
||||
| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
|
||||
| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
|
||||
|
||||
@@ -521,6 +521,66 @@ type storeGrantRepairer struct {
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time // target id → last ATTEMPT (success or failure)
|
||||
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
|
||||
// seenPath (R-1302) is the file that remembers which tiers have EVER held the grant: only those can LOSE it. "" = no
|
||||
// memory, so every missing grant reads as a loss — the old, loud behaviour.
|
||||
seenPath string
|
||||
}
|
||||
|
||||
// everHeld reports whether this tier's grant was ever seen on this box (R-1302). The direction of a doubt is „yes": an
|
||||
// unreadable or corrupt file keeps the loss ERROR, so a real loss is never quietened by a broken record.
|
||||
func (r *storeGrantRepairer) everHeld(target string) bool {
|
||||
if r == nil || r.seenPath == "" {
|
||||
return true
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
seen, err := readGrantSeen(r.seenPath)
|
||||
return err != nil || seen[target]
|
||||
}
|
||||
|
||||
// noteHeld records that this tier holds the grant. A write that fails is logged and costs one false „first grant" at most
|
||||
// after a restart — the old file, or none, stands.
|
||||
func (r *storeGrantRepairer) noteHeld(target string) {
|
||||
if r == nil || r.seenPath == "" {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
seen, err := readGrantSeen(r.seenPath)
|
||||
if err != nil {
|
||||
seen = map[string]bool{} // a corrupt file is replaced by what is true now
|
||||
}
|
||||
if seen[target] {
|
||||
return
|
||||
}
|
||||
seen[target] = true
|
||||
b, _ := json.Marshal(seen)
|
||||
tmp := r.seenPath + ".tmp"
|
||||
if werr := os.WriteFile(tmp, b, 0o600); werr == nil {
|
||||
werr = os.Rename(tmp, r.seenPath)
|
||||
if werr == nil {
|
||||
return
|
||||
}
|
||||
err = werr
|
||||
} else {
|
||||
err = werr
|
||||
}
|
||||
repairLogger(r).Warn("store-grant: could not record that this tier holds its grant (R-1302)", "target", target, "err", err)
|
||||
}
|
||||
|
||||
func readGrantSeen(path string) (map[string]bool, error) {
|
||||
b, err := os.ReadFile(path)
|
||||
if os.IsNotExist(err) {
|
||||
return map[string]bool{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
if err := json.Unmarshal(b, &seen); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return seen, nil
|
||||
}
|
||||
|
||||
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
|
||||
@@ -631,6 +691,7 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
||||
// host-report has certainly carried it. Without this latch the repairing probe may be a
|
||||
// log-only one and the hub never learns anything happened (measured live, see the window's
|
||||
// comment).
|
||||
repair.noteHeld(targetID) // R-1302: a tier that holds its grant can lose it from now on
|
||||
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
|
||||
}
|
||||
|
||||
@@ -679,8 +740,25 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
||||
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
|
||||
// change, no hub change, no new event type. The `Feature` text carries the explanation because
|
||||
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
|
||||
return storeGrantAfterRepair(targetID, critical, repair, s)
|
||||
}
|
||||
|
||||
// storeGrantAfterRepair is the verdict once a repair is confirmed (split out for the tests). R-1302: the FIRST grant of a
|
||||
// tier that never held one is no loss — on a fresh install the whole-guest tier is armed before its storage exists, so
|
||||
// every new box logged „investigate the loss" and a DEGRADED capability at its first start, which teaches the reader to
|
||||
// skip the line. That grant logs INFO and reports healthy. A tier seen holding its grant, or one this box cannot tell
|
||||
// about, keeps R-190's ERROR and its one degraded cycle. Pinned by TestR1302_*.
|
||||
func storeGrantAfterRepair(targetID string, critical bool, repair *storeGrantRepairer, healthy capability.Status) capability.Status {
|
||||
if !repair.everHeld(targetID) {
|
||||
repair.noteHeld(targetID)
|
||||
repairLogger(repair).Info("store-grant: granted — the first grant of this tier, nothing was lost (R-1302)",
|
||||
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||
healthy.Status, healthy.Reason = capability.StatusOK, ""
|
||||
return healthy
|
||||
}
|
||||
repair.noteRepaired(targetID, time.Now())
|
||||
s = storeGrantRepairedVerdict(targetID, critical)
|
||||
s := storeGrantRepairedVerdict(targetID, critical)
|
||||
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
|
||||
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||
@@ -853,8 +931,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
|
||||
// invoked through the privileged path, which prepends sudo itself.
|
||||
grantRepairer := &storeGrantRepairer{
|
||||
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||
log: logger,
|
||||
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||
log: logger,
|
||||
seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json"), // R-1302
|
||||
}
|
||||
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
|
||||
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"go/ast"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -415,3 +418,101 @@ func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
|
||||
"be missed entirely", storeGrantRepairReportWindow)
|
||||
}
|
||||
}
|
||||
|
||||
// ── R-1302 — the first grant of a tier is no loss ──────────────────────────────────────
|
||||
//
|
||||
// A fresh install arms the whole-guest tier before its storage exists, so its first probe found no grant, repaired it and
|
||||
// logged „GRANT WAS MISSING … investigate the loss" with a DEGRADED capability — on every new box (VM 9375, 2026-10-26).
|
||||
//
|
||||
// RED-PROOF (2026-11-02): seen failing before storeGrantAfterRepair existed (build), and with its first-grant branch removed
|
||||
// TestR1302_TheFirstGrantIsNoLoss fails on „a first grant must not report a loss".
|
||||
func r1302Repairer(t *testing.T, path string) (*storeGrantRepairer, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
return &storeGrantRepairer{run: (&fakeRepairRunner{}).Run, seenPath: path,
|
||||
log: slog.New(slog.NewTextHandler(&buf, nil))}, &buf
|
||||
}
|
||||
|
||||
func TestR1302_TheFirstGrantIsNoLoss(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
r, buf := r1302Repairer(t, path)
|
||||
healthy := capability.Status{Name: "pve:store-grant:felhom-pbs", Status: capability.StatusDegraded, Reason: "missing", Critical: true}
|
||||
s := storeGrantAfterRepair("felhom-pbs", true, r, healthy)
|
||||
if s.Status != capability.StatusOK || s.Reason != "" {
|
||||
t.Fatalf("a first grant must not report a loss: status %q reason %q", s.Status, s.Reason)
|
||||
}
|
||||
got := buf.String()
|
||||
if strings.Contains(got, "investigate the loss") || strings.Contains(got, "level=ERROR") {
|
||||
t.Fatalf("a first grant must not log the loss ERROR: %s", got)
|
||||
}
|
||||
if !strings.Contains(got, "level=INFO") || !strings.Contains(got, "granted") || !strings.Contains(got, "felhom-pbs") {
|
||||
t.Fatalf("a first grant says so at INFO: %s", got)
|
||||
}
|
||||
if r.recentlyRepaired("felhom-pbs", time.Now()) {
|
||||
t.Fatal("a first grant must not arm the loss report's latch")
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1302_AGrantThatExistedAndVanishedKeepsTheError(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
first, _ := r1302Repairer(t, path)
|
||||
first.noteHeld("felhom-pbs") // a healthy probe, in an earlier run of the agent
|
||||
|
||||
r, buf := r1302Repairer(t, path) // the agent restarted: the memory is the file, not the process
|
||||
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||
if s.Status != capability.StatusDegraded || !strings.Contains(s.Feature, "R-190") {
|
||||
t.Fatalf("a grant that existed and vanished must keep its degraded cycle; got %q / %q", s.Status, s.Feature)
|
||||
}
|
||||
if got := buf.String(); !strings.Contains(got, "level=ERROR") || !strings.Contains(got, "investigate the loss") {
|
||||
t.Fatalf("a grant that existed and vanished must keep its ERROR: %s", got)
|
||||
}
|
||||
if !r.recentlyRepaired("felhom-pbs", time.Now()) {
|
||||
t.Fatal("the loss must arm the report's latch, or the hub may never hear it")
|
||||
}
|
||||
}
|
||||
|
||||
// After a first grant the tier counts as held: the NEXT disappearance is a loss.
|
||||
func TestR1302_AfterTheFirstGrantALossIsALoss(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
r, _ := r1302Repairer(t, path)
|
||||
storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||
if s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("the second disappearance is a loss; got %q", s.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The direction of a doubt: a broken record, a repairer with no record, or no repairer at all keep the loud way.
|
||||
func TestR1302_ADoubtKeepsTheError(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, _ := r1302Repairer(t, path)
|
||||
if s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("a corrupt record must not quieten a loss; got %q", s.Status)
|
||||
}
|
||||
noMemory, _ := r1302Repairer(t, "")
|
||||
if s := storeGrantAfterRepair("felhom-pbs", true, noMemory, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("no record must not quieten a loss; got %q", s.Status)
|
||||
}
|
||||
if s := storeGrantAfterRepair("felhom-pbs", true, nil, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("no repairer must not quieten a loss; got %q", s.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The wiring, read from the source: production names the file, and a healthy probe records the tier.
|
||||
func TestR1302_TheMemoryIsWired(t *testing.T) {
|
||||
b, err := os.ReadFile("main.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
src := string(b)
|
||||
if !strings.Contains(src, `seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json")`) {
|
||||
t.Fatal("main.go must give the repairer its file, or every first grant reads as a loss")
|
||||
}
|
||||
start := strings.Index(src, "func storeGrantStatus(")
|
||||
if start < 0 || !strings.Contains(src[start:start+3000], "repair.noteHeld(targetID)") {
|
||||
t.Fatal("a healthy probe must record the tier, or a real loss would read as a first grant")
|
||||
}
|
||||
}
|
||||
@@ -97,9 +97,12 @@ OOMCHECK_SETTLE = 2
|
||||
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
|
||||
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
|
||||
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
|
||||
# proxmox-grub is GRUB's Proxmox meta-package (it depends on grub-efi-amd64 of the same version); the `grub` above is
|
||||
# anchored at the start, so it missed it. Measured 2026-11-05..07 on demo-felhom: its upgrade pulled grub-efi-amd64 and
|
||||
# the whole pve step was refused R6 every night. Pinned by test_ring0_pending_pve_leaves_proxmox_grub_out.
|
||||
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
|
||||
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
|
||||
r"proxmox-secure-boot-support)")
|
||||
r"proxmox-secure-boot-support|proxmox-grub)")
|
||||
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
|
||||
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
|
||||
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
|
||||
|
||||
@@ -1494,6 +1494,24 @@ class PVELane(unittest.TestCase):
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
|
||||
|
||||
# 2026-11-05..07, demo-felhom (EFI): proxmox-grub 2.12-9+pmx2 -> pmx4 depends on grub-efi-amd64 of the same version;
|
||||
# HOST_SLOW_RE's `grub` is anchored at the start, so proxmox-grub fell into the pve plan, the simulation pulled
|
||||
# grub-efi-amd64 and the whole pve step was refused R6 every night. It is a boot-chain package: left out.
|
||||
# RED-PROOF: drop proxmox-grub from HOST_SLOW_RE -> it is in the plan -> this test fails.
|
||||
def test_ring0_pending_pve_leaves_proxmox_grub_out(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["select"], f.plan["packages"] = "pending-pve", []
|
||||
f.installed.update({"proxmox-grub": "2.12-9+pmx2"})
|
||||
f.pending_sim = [
|
||||
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst proxmox-grub [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst grub-efi-amd64 [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
|
||||
]
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||
self.assertEqual(f.installed["proxmox-grub"], "2.12-9+pmx2")
|
||||
|
||||
def test_select_pending_pve_needs_the_pve_layer(self):
|
||||
f = Fake()
|
||||
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
|
||||
|
||||
@@ -128,4 +128,11 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
|
||||
if m.onCommitted != nil {
|
||||
m.onCommitted(ctx)
|
||||
}
|
||||
// R-1368 part 2: the update ended healthy — the folder keeps the running version's binary and no other
|
||||
// (prune.go). A refusal or a failed removal is logged and changes nothing about the commit.
|
||||
if removed, err := pruneStaged(m.stateDir, m.runningVersion, m.logger); err != nil {
|
||||
m.logger.Warn("selfupdate: old staged binaries NOT removed after the commit", "err", err)
|
||||
} else if len(removed) > 0 {
|
||||
m.logger.Info("selfupdate: old staged binaries removed after the commit", "count", len(removed), "kept", "felhom-agent-"+m.runningVersion)
|
||||
}
|
||||
}
|
||||
@@ -103,6 +103,9 @@ func (e *Executor) Execute(ctx context.Context, op string, params json.RawMessag
|
||||
}
|
||||
e.logger.Debug("agent_update: op invariants passed (semver + sha format)", "version", p.Version)
|
||||
|
||||
// The staged file is this call's from the download to the hand-off: a prune (prune.go) waits for it.
|
||||
stagingMu.Lock()
|
||||
defer stagingMu.Unlock()
|
||||
dir := stagingDir(e.stateDir)
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
return fmt.Errorf("agent_update: staging dir: %w", err)
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// The self-update folder keeps what it needs and no more (R-1368 part 2; space list 2026-11-10, place H12).
|
||||
//
|
||||
// WHAT WAS WRONG. Execute downloads each new agent to <StateDir>/selfupdate/felhom-agent-<version> and removes that
|
||||
// file only when the download or its checksum fails. A good update left it there for ever: measured 2026-11-10, 49 /
|
||||
// 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1 — one for each version the box ever got.
|
||||
//
|
||||
// WHAT THE FOLDER IS NEEDED FOR. The root wrapper (felhom-os-apply, mode agent_update) reads the staged file ONCE,
|
||||
// copies it to its own root-owned folder and flips from that copy. After that hand-off nothing reads the staged file:
|
||||
// - the rollback (felhom-selfupdate-guarded `rollback`) needs two files and neither is a staged binary:
|
||||
// `pending.json` in this folder, and `/usr/local/bin/felhom-agent.prev` outside it;
|
||||
// - the commit needs `pending.json` only.
|
||||
//
|
||||
// So a staged binary is dead weight as soon as its update ended. Pinned by TestPrune_RollbackReadsNoStagedBinary (the
|
||||
// wrapper's text) and TestPrune_RealRollbackStillWorksAfterNUpdates (the wrapper's rollback verb, run for real).
|
||||
//
|
||||
// THE RULE. After an update ended HEALTHY (the wrapper's `commit` returned success), remove every staged binary but
|
||||
// the one of the running version. Nothing is removed:
|
||||
// - while a pending marker exists or cannot be read — an update is in flight, and that is the only time a rollback
|
||||
// can fire;
|
||||
// - after a rollback, or at a plain start — only the commit path calls this, so the binary of a version that failed
|
||||
// stays for the operator to look at until the next good update;
|
||||
// - that is not a regular file with exactly the name Execute gives a staged binary. `pending.json`, a folder, a
|
||||
// symlink, and any name this code does not know are never touched.
|
||||
//
|
||||
// NOT REACHED BY THIS: `/usr/local/bin/felhom-agent.bak-*` (space list H13). Those files are root-owned, in a folder
|
||||
// the agent cannot write, and no code of the agent makes them (a person's copy before a hand deploy). R-1368 part 4.
|
||||
|
||||
// stagedNameRe is the exact name Execute gives a staged binary: "felhom-agent-" + a version that versionRe accepts.
|
||||
// Kept in step with versionRe by TestPrune_NameShapeFollowsTheVersionRule.
|
||||
var stagedNameRe = regexp.MustCompile(`^felhom-agent-([0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?)$`)
|
||||
|
||||
// stagingMu makes a download and a prune take turns. Without it a prune that runs while Execute downloads the NEXT
|
||||
// version would remove that half-written file (its name is a staged binary's and it is not the running version).
|
||||
var stagingMu sync.Mutex
|
||||
|
||||
// pruneStaged removes the staged binaries the folder no longer needs and returns the names it removed, sorted. It
|
||||
// refuses (removes nothing, returns an error) while an update is in flight. A file it cannot remove is logged and
|
||||
// skipped: a failed clean-up must never fail the commit that came before it.
|
||||
func pruneStaged(stateDir, runningVersion string, logger *slog.Logger) ([]string, error) {
|
||||
logger = orDefaultLogger(logger)
|
||||
stagingMu.Lock()
|
||||
defer stagingMu.Unlock()
|
||||
|
||||
if !versionRe.MatchString(runningVersion) {
|
||||
return nil, fmt.Errorf("selfupdate prune: refusing — running version %q is not a version this code knows; it could not name the file to keep", runningVersion)
|
||||
}
|
||||
pm, err := readPending(stateDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("selfupdate prune: refusing — the pending marker cannot be read: %w", err)
|
||||
}
|
||||
if pm != nil {
|
||||
return nil, fmt.Errorf("selfupdate prune: refusing — an update to %s is pending (a rollback may still fire)", pm.NewVersion)
|
||||
}
|
||||
dir := stagingDir(stateDir)
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, fmt.Errorf("selfupdate prune: read %s: %w", dir, err)
|
||||
}
|
||||
keep := "felhom-agent-" + runningVersion
|
||||
var removed []string
|
||||
for _, ent := range entries {
|
||||
name := ent.Name()
|
||||
if name == keep || !stagedNameRe.MatchString(name) {
|
||||
continue // the running version's binary, pending.json, or a name this code does not know
|
||||
}
|
||||
if !ent.Type().IsRegular() {
|
||||
logger.Warn("selfupdate prune: a staged-binary name that is not a regular file — left alone", "name", name, "type", ent.Type().String())
|
||||
continue
|
||||
}
|
||||
// name came from ReadDir and matched stagedNameRe, so it holds no path separator: this is one exact file.
|
||||
if err := os.Remove(filepath.Join(dir, name)); err != nil {
|
||||
logger.Warn("selfupdate prune: cannot remove an old staged binary — left for the next good update", "name", name, "err", err)
|
||||
continue
|
||||
}
|
||||
logger.Debug("selfupdate prune: removed an old staged binary", "name", name)
|
||||
removed = append(removed, name)
|
||||
}
|
||||
sort.Strings(removed)
|
||||
logger.Info("selfupdate prune: the self-update folder holds what it needs", "kept", keep, "removed", len(removed))
|
||||
return removed, nil
|
||||
}
|
||||
@@ -0,0 +1,373 @@
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1368 part 2 — the self-update folder keeps what it needs and no more (prune.go).
|
||||
//
|
||||
// RED-PROOFS (each observed 2026-11-10, then restored):
|
||||
// - drop the pruneStaged call from MaybeCommit → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with
|
||||
// "after update 1 (0.170.0) the folder holds [… felhom-agent-0.169.0 felhom-agent-0.170.0 …]", and
|
||||
// TestPrune_RealRollbackStillWorksAfterNUpdates fails too;
|
||||
// - drop the pending-marker refusal from pruneStaged → TestPrune_RefusesWhileAnUpdateIsPending fails with
|
||||
// "removed [...] while an update was pending";
|
||||
// - drop the IsRegular check → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with "a symlink with a
|
||||
// staged-binary name was removed".
|
||||
|
||||
// committingWrapper is the guarded wrapper's `commit` as far as this package can see it: it clears the pending marker.
|
||||
type committingWrapper struct {
|
||||
stateDir string
|
||||
calls int
|
||||
}
|
||||
|
||||
func (w *committingWrapper) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
|
||||
if len(args) == 1 && args[0] == "commit" {
|
||||
w.calls++
|
||||
if err := os.Remove(pendingPath(w.stateDir)); err != nil {
|
||||
return nil, []byte(err.Error()), err
|
||||
}
|
||||
}
|
||||
return []byte("ok"), nil, nil
|
||||
}
|
||||
|
||||
func stage(t *testing.T, stateDir, version string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(stagingDir(stateDir), 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(stagingDir(stateDir), "felhom-agent-"+version), []byte("binary "+version), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func folder(t *testing.T, stateDir string) []string {
|
||||
t.Helper()
|
||||
ents, err := os.ReadDir(stagingDir(stateDir))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out []string
|
||||
for _, e := range ents {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// The consequence: after N good updates the folder holds ONE staged binary — the running version's — and every file
|
||||
// that is not a staged binary is still there.
|
||||
func TestPrune_AfterNUpdatesTheFolderHoldsOneBinary(t *testing.T) {
|
||||
stateDir := t.TempDir()
|
||||
dir := stagingDir(stateDir)
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Bystanders: names this code does not know, and two things that carry a staged binary's NAME but are not files.
|
||||
outside := filepath.Join(t.TempDir(), "precious")
|
||||
if err := os.WriteFile(outside, []byte("not ours"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, n := range []string{"notes.txt", "felhom-agent-0.1.0.partial", "felhom-agent.bak-0.139.0"} {
|
||||
if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.Mkdir(filepath.Join(dir, "felhom-agent-9.9.9"), 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink(outside, filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
bystanders := []string{"felhom-agent-0.1.0.partial", "felhom-agent-8.8.8", "felhom-agent-9.9.9", "felhom-agent.bak-0.139.0", "notes.txt"}
|
||||
|
||||
versions := []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1", "0.173.0-rc.1", "0.173.0"}
|
||||
prev := "0.169.0"
|
||||
stage(t, stateDir, prev) // what an old agent left behind
|
||||
for i, v := range versions {
|
||||
stage(t, stateDir, v) // Execute downloaded it
|
||||
writePending(t, stateDir, PendingMarker{OldVersion: prev, NewVersion: v, SHA256: "ab", AppliedAt: "t"}) // the wrapper flipped
|
||||
wrap := &committingWrapper{stateDir: stateDir}
|
||||
m := newMgr(t, stateDir, v, wrap, nil)
|
||||
m.MaybeCommit(context.Background())
|
||||
if wrap.calls != 1 {
|
||||
t.Fatalf("update %d (%s): commit calls = %d, want 1", i+1, v, wrap.calls)
|
||||
}
|
||||
want := append([]string{"felhom-agent-" + v}, bystanders...)
|
||||
sort.Strings(want)
|
||||
got := folder(t, stateDir)
|
||||
if strings.Join(got, " ") != strings.Join(want, " ") {
|
||||
if _, err := os.Lstat(filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
|
||||
t.Fatalf("a symlink with a staged-binary name was removed: %v", err)
|
||||
}
|
||||
t.Fatalf("after update %d (%s) the folder holds %v, want %v", i+1, v, got, want)
|
||||
}
|
||||
prev = v
|
||||
}
|
||||
if b, err := os.ReadFile(outside); err != nil || string(b) != "not ours" {
|
||||
t.Fatalf("the file behind the symlink changed: %q %v", b, err)
|
||||
}
|
||||
}
|
||||
|
||||
// While an update is pending — the only time a rollback can fire — nothing is removed.
|
||||
func TestPrune_RefusesWhileAnUpdateIsPending(t *testing.T) {
|
||||
stateDir := t.TempDir()
|
||||
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
|
||||
stage(t, stateDir, v)
|
||||
}
|
||||
writePending(t, stateDir, PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0", SHA256: "ab", AppliedAt: "t"})
|
||||
before := folder(t, stateDir)
|
||||
|
||||
removed, err := pruneStaged(stateDir, "0.172.0", nil)
|
||||
if err == nil || len(removed) != 0 {
|
||||
t.Fatalf("removed %v while an update was pending (err=%v) — want a refusal", removed, err)
|
||||
}
|
||||
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
|
||||
t.Fatalf("removed %v while an update was pending; folder now %v", before, got)
|
||||
}
|
||||
|
||||
// The same through the commit path: a `commit` that reports success but leaves the marker (fakeWrapper does) must
|
||||
// not lead to a prune.
|
||||
m := newMgr(t, stateDir, "0.172.0", &fakeWrapper{}, nil)
|
||||
m.MaybeCommit(context.Background())
|
||||
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
|
||||
t.Fatalf("the commit path pruned with the marker still there; folder now %v", got)
|
||||
}
|
||||
|
||||
// A marker nobody can read is a refusal too.
|
||||
if err := os.WriteFile(pendingPath(stateDir), []byte("{not json"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if removed, err := pruneStaged(stateDir, "0.172.0", nil); err == nil || len(removed) != 0 {
|
||||
t.Fatalf("an unreadable marker must refuse; removed %v err=%v", removed, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Only a commit prunes: a plain start, a version mismatch and a failed commit remove nothing — so the binary of a
|
||||
// version that was rolled back stays for the operator.
|
||||
func TestPrune_OnlyAHealthyCommitPrunes(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
pending *PendingMarker
|
||||
running string
|
||||
wrap WrapperRunner
|
||||
}{
|
||||
{"plain start, no marker (also the state after a rollback)", nil, "0.171.0", &fakeWrapper{}},
|
||||
{"marker names another version", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.171.0", &fakeWrapper{}},
|
||||
{"commit fails", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.172.0", &fakeWrapper{err: fmt.Errorf("sudo: no")}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
stateDir := t.TempDir()
|
||||
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
|
||||
stage(t, stateDir, v)
|
||||
}
|
||||
if c.pending != nil {
|
||||
writePending(t, stateDir, *c.pending)
|
||||
}
|
||||
before := folder(t, stateDir)
|
||||
newMgr(t, stateDir, c.running, c.wrap, nil).MaybeCommit(context.Background())
|
||||
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
|
||||
t.Fatalf("folder changed without a healthy commit: %v → %v", before, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
// A running version this code cannot name a file for: refuse, do not guess.
|
||||
stateDir := t.TempDir()
|
||||
stage(t, stateDir, "0.170.0")
|
||||
if removed, err := pruneStaged(stateDir, "dev", nil); err == nil || len(removed) != 0 {
|
||||
t.Fatalf("running version \"dev\" must refuse; removed %v err=%v", removed, err)
|
||||
}
|
||||
}
|
||||
|
||||
// The name rule and the version rule say the same thing: every name Execute can write is a name the prune knows, and
|
||||
// nothing else is.
|
||||
func TestPrune_NameShapeFollowsTheVersionRule(t *testing.T) {
|
||||
for _, v := range []string{"0.70.1", "0.172.1", "10.0.0", "0.70.2-crash", "0.173.0-rc.1"} {
|
||||
if !versionRe.MatchString(v) {
|
||||
t.Fatalf("test input %q is not a version versionRe accepts", v)
|
||||
}
|
||||
if !stagedNameRe.MatchString("felhom-agent-" + v) {
|
||||
t.Errorf("Execute can stage felhom-agent-%s but the prune does not know that name", v)
|
||||
}
|
||||
}
|
||||
for _, n := range []string{"pending.json", "felhom-agent", "felhom-agent-", "felhom-agent-1.2", "felhom-agent-0.1.0.partial",
|
||||
"felhom-agent.bak-0.139.0", "felhom-agent.prev", "felhom-agent-0.1.0/x", "../felhom-agent-0.1.0", "felhom-agent-0.1.0 ", "xfelhom-agent-0.1.0"} {
|
||||
if stagedNameRe.MatchString(n) {
|
||||
t.Errorf("the prune would treat %q as a staged binary", n)
|
||||
}
|
||||
}
|
||||
m := stagedNameRe.FindStringSubmatch("felhom-agent-0.173.0-rc.1")
|
||||
if m == nil || !versionRe.MatchString(m[1]) {
|
||||
t.Fatalf("the version part of a staged name must satisfy versionRe; got %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func wrapperText(t *testing.T) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join("..", "..", "configs", "felhom-selfupdate-guarded"))
|
||||
if err != nil {
|
||||
t.Fatalf("the guarded wrapper is the other half of this proof and must be readable: %v", err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// The stated dependency, pinned: the wrapper's `rollback` and `commit` read the pending marker and `.prev`, and no
|
||||
// staged binary. If either verb starts to read the staging folder, this fails and prune.go's rule must be looked at.
|
||||
func TestPrune_RollbackReadsNoStagedBinary(t *testing.T) {
|
||||
s := wrapperText(t)
|
||||
for _, want := range []string{"\nBIN=/usr/local/bin/felhom-agent\n", "\nPREV=$BIN.prev\n", "\nSTAGING=/var/lib/felhom-agent/selfupdate\n", "\nPENDING=$STAGING/pending.json\n"} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Fatalf("the wrapper no longer holds the line %q — the folder's rule (prune.go) was written against it", strings.TrimSpace(want))
|
||||
}
|
||||
}
|
||||
for _, verb := range []string{"commit", "rollback"} {
|
||||
i := strings.Index(s, "\n"+verb+")\n")
|
||||
if i < 0 {
|
||||
t.Fatalf("no %s) block in the wrapper", verb)
|
||||
}
|
||||
block := s[i:]
|
||||
j := strings.Index(block, "\n ;;\n")
|
||||
if j < 0 {
|
||||
t.Fatalf("no end of the %s) block", verb)
|
||||
}
|
||||
block = block[:j]
|
||||
if !strings.Contains(block, "$PENDING") {
|
||||
t.Fatalf("%s) does not read $PENDING any more", verb)
|
||||
}
|
||||
code := regexp.MustCompile(`(?m)^\s*#.*$`).ReplaceAllString(block, "")
|
||||
for _, bad := range []string{"STAGING", "staged", "felhom-agent-"} {
|
||||
if strings.Contains(code, bad) {
|
||||
t.Fatalf("%s) now mentions %q — it may read a staged binary, and the prune removes those", verb, bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
if stagedNameRe.MatchString("pending.json") || stagedNameRe.MatchString("felhom-agent.prev") {
|
||||
t.Fatal("the prune would remove a file the rollback needs")
|
||||
}
|
||||
}
|
||||
|
||||
// The consequence, with the wrapper's own code: after N good updates (each committed by the REAL `commit` verb and
|
||||
// pruned), update N+1 goes bad and the REAL `rollback` verb puts the previous binary back. The wrapper runs from a copy
|
||||
// whose four fixed paths point into a temp folder; systemctl, logger and chown are stubs.
|
||||
func TestPrune_RealRollbackStillWorksAfterNUpdates(t *testing.T) {
|
||||
sh, err := exec.LookPath("sh")
|
||||
if err != nil {
|
||||
t.Skip("no sh on this machine")
|
||||
}
|
||||
root := t.TempDir()
|
||||
stateDir := filepath.Join(root, "state")
|
||||
binDir := filepath.Join(root, "bin")
|
||||
stubs := filepath.Join(root, "stubs")
|
||||
for _, d := range []string{stagingDir(stateDir), binDir, stubs, filepath.Join(root, "rootstaging")} {
|
||||
if err := os.MkdirAll(d, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
live := filepath.Join(binDir, "felhom-agent")
|
||||
s := wrapperText(t)
|
||||
for old, repl := range map[string]string{
|
||||
"\nBIN=/usr/local/bin/felhom-agent\n": "\nBIN=" + live + "\n",
|
||||
"\nSTAGING=/var/lib/felhom-agent/selfupdate\n": "\nSTAGING=" + stagingDir(stateDir) + "\n",
|
||||
"\nROOT_STAGING=/var/lib/felhom-os-apply/agent-update\n": "\nROOT_STAGING=" + filepath.Join(root, "rootstaging") + "\n",
|
||||
} {
|
||||
if strings.Count(s, old) != 1 {
|
||||
t.Fatalf("the wrapper's path line %q is not there exactly once — this test cannot sandbox it", strings.TrimSpace(old))
|
||||
}
|
||||
s = strings.Replace(s, old, repl, 1)
|
||||
}
|
||||
// Safety: no assignment in the copy may still point at a real system path (PREV and PENDING are built from BIN and
|
||||
// STAGING, so they follow).
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
if regexp.MustCompile(`^[A-Z_]+=/`).MatchString(line) && !strings.Contains(line, root) {
|
||||
t.Fatalf("the sandboxed wrapper still holds a real path: %q", line)
|
||||
}
|
||||
}
|
||||
script := filepath.Join(root, "wrapper.sh")
|
||||
if err := os.WriteFile(script, []byte(s), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stubLog := filepath.Join(root, "stub.log")
|
||||
for _, n := range []string{"systemctl", "logger", "chown", "systemd-run"} {
|
||||
body := "#!/bin/sh\necho \"" + n + " $*\" >> '" + stubLog + "'\nexit 0\n"
|
||||
if err := os.WriteFile(filepath.Join(stubs, n), []byte(body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
run := func(verb string) (string, error) {
|
||||
cmd := exec.Command(sh, script, verb)
|
||||
cmd.Env = []string{"PATH=" + stubs + ":/usr/bin:/bin"}
|
||||
out, err := cmd.CombinedOutput()
|
||||
return string(out), err
|
||||
}
|
||||
// applied is what the wrapper's `apply` leaves behind (it needs root, so the test writes the same four things).
|
||||
applied := func(oldV, newV string) {
|
||||
cur, err := os.ReadFile(live)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(live+".prev", cur, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(live, []byte("binary "+newV), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writePending(t, stateDir, PendingMarker{OldVersion: oldV, NewVersion: newV, SHA256: "ab", AppliedAt: "t"})
|
||||
}
|
||||
if err := os.WriteFile(live, []byte("binary 0.169.0"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
prev := "0.169.0"
|
||||
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1"} {
|
||||
stage(t, stateDir, v)
|
||||
applied(prev, v)
|
||||
m := newMgr(t, stateDir, v, runnerFunc(func(args ...string) ([]byte, []byte, error) {
|
||||
out, err := run(args[0])
|
||||
return []byte(out), nil, err
|
||||
}), nil)
|
||||
m.MaybeCommit(context.Background()) // the REAL commit verb, then the prune
|
||||
if got := folder(t, stateDir); len(got) != 1 || got[0] != "felhom-agent-"+v {
|
||||
t.Fatalf("after the good update to %s the folder holds %v", v, got)
|
||||
}
|
||||
prev = v
|
||||
}
|
||||
|
||||
// Update N+1 is applied and goes bad: systemd's OnFailure runs the wrapper's rollback.
|
||||
stage(t, stateDir, "0.173.0")
|
||||
applied("0.172.1", "0.173.0")
|
||||
out, err := run("rollback")
|
||||
if err != nil {
|
||||
t.Fatalf("the real rollback verb failed after the prunes: %v\n%s", err, out)
|
||||
}
|
||||
if b, _ := os.ReadFile(live); string(b) != "binary 0.172.1" {
|
||||
t.Fatalf("after the rollback the live binary is %q, want the previous version's", b)
|
||||
}
|
||||
if p, _ := readPending(stateDir); p != nil {
|
||||
t.Fatal("the rollback left the pending marker")
|
||||
}
|
||||
if b, _ := os.ReadFile(stubLog); !bytes.Contains(b, []byte("systemctl restart felhom-agent.service")) {
|
||||
t.Fatalf("the rollback did not restart the agent; stub log:\n%s", b)
|
||||
}
|
||||
// The rolled-back agent starts: no marker, so no commit and no prune — the bad version's binary stays to be read.
|
||||
newMgr(t, stateDir, "0.172.1", &fakeWrapper{}, nil).MaybeCommit(context.Background())
|
||||
if got := folder(t, stateDir); strings.Join(got, " ") != "felhom-agent-0.172.1 felhom-agent-0.173.0" {
|
||||
t.Fatalf("after the rollback the folder holds %v, want the running version and the one that failed", got)
|
||||
}
|
||||
}
|
||||
|
||||
// runnerFunc adapts a function to WrapperRunner; it gets the verb arguments only.
|
||||
type runnerFunc func(args ...string) ([]byte, []byte, error)
|
||||
|
||||
func (f runnerFunc) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
|
||||
return f(args...)
|
||||
}
|
||||
@@ -24,10 +24,10 @@
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||
],
|
||||
"recorded": "2026-10-29",
|
||||
"recorded_by": "CC 2026-10-29: day0_agent 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"recorded": "2026-11-03",
|
||||
"recorded_by": "CC 2026-11-07: day0_agent 0.172.1 (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"pinned": {
|
||||
"hub_min_agent": "0.158.1",
|
||||
"day0_agent": "0.170.0"
|
||||
"day0_agent": "0.172.1"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user