Compare commits

..
10 Commits
Author SHA1 Message Date
admin 776025c824 selfupdate: the staging folder keeps what it needs and no more (R-1368 part 2) — not released
gates / gates (push) Successful in 35s
Each good update left its staged binary in <state dir>/selfupdate for ever
(49 files / 740 MB on demo-hp). After the wrapper's commit returned success,
MaybeCommit now removes every staged binary but the running version's.

Nothing is removed while a pending marker exists or cannot be read, at a
plain start, or after a rollback. Only a regular file with the exact name
Execute writes is removed. The rollback reads pending.json and
felhom-agent.prev, no staged binary: pinned by a test on the wrapper's text
and by a test that runs the wrapper's real commit and rollback verbs in a
sandbox. Red seen three times.

No version bump, no build, no vouch. /usr/local/bin/felhom-agent.bak-* is
not reached (root-owned; R-1368 part 4).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 14:10:00 +01:00
adminandClaude Opus 5.5 41d5eb8051 v0.172.1 vouched and delivered to the three test boxes (agent + bundle, 77/77); pinned.day0_agent = 0.172.1 (D50)
gates / gates (push) Successful in 44s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-07 23:29:24 +01:00
adminandClaude Opus 5.5 8bd137787e v0.172.1 released (not vouched): CHANGELOG, REPORT with the operator's PowerShell delivery, CONTEXT
gates / gates (push) Successful in 33s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-07 22:43:34 +01:00
adminandClaude Opus 5.5 39db03065a os-apply: proxmox-grub joins the boot-chain lane (pve step refused R6 nightly on demo-felhom)
gates / gates (push) Successful in 36s
proxmox-grub 2.12-9+pmx4 depends on grub-efi-amd64 of the same version. HOST_SLOW_RE
anchors `grub` at the start of a name, so proxmox-grub landed in the ring-0 pve plan,
the simulation pulled grub-efi-amd64, and the whole pve step was refused R6 every
night since 2026-11-05. Same class as the 2026-10-09 proxmox-secure-boot-support fix.

Test test_ring0_pending_pve_leaves_proxmox_grub_out: red before the fix, green after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-07 22:41:05 +01:00
admin ba409de4a2 v0.172.0 vouched: pinned.day0_agent = 0.172.0 (D50); CHANGELOG heading; REPORT + CONTEXT
gates / gates (push) Successful in 34s
2026-11-03 10:29:51 +01:00
admin a4f1280399 v0.172.0 released, not yet vouched: CHANGELOG heading (R-1302; agent 2c0222cd, bundle cc34b0bc)
gates / gates (push) Successful in 36s
2026-11-03 09:08:07 +01:00
admin ecbb0cc479 REPORT: 2026-11-02 night, R-1302, commit only
gates / gates (push) Successful in 36s
2026-11-03 04:13:52 +01:00
admin 50eeccc986 R-1302: the first grant of a tier logs INFO granted; a grant that existed and vanished keeps the R-190 ERROR. Commit only.
gates / gates (push) Successful in 36s
2026-11-02 23:16:10 +01:00
admin 808cdb36ad REPORT + CONTEXT: v0.171.0 released and vouched; R-1344 restart proven live
gates / gates (push) Successful in 33s
2026-10-30 11:09:48 +01:00
admin 84726c9f27 v0.171.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.171.0 (D50)
gates / gates (push) Successful in 33s
2026-10-30 10:49:46 +01:00
13 changed files with 779 additions and 42 deletions

No files matched your search

+55 -1
View File
@@ -1,4 +1,58 @@
## Unreleased — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED
## Unreleased (2026-11-10) — R-1368 part 2: the self-update folder keeps what it needs and no more — COMMITTED, NOT RELEASED (no version, no build, no vouch; it waits for a release session)
- **The defect (measured 2026-11-10, space list place H12).** `Execute` downloads each new agent to
`<state dir>/selfupdate/felhom-agent-<version>` and removed that file only when the download or its checksum failed.
A good update left it for ever: 49 / 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1.
- **What the folder is needed for (read first: `configs/felhom-selfupdate-guarded`).** The root wrapper reads the staged
file once, copies it to its own root-owned folder and flips from that copy. The rollback needs `pending.json` (in the
folder) and `/usr/local/bin/felhom-agent.prev` (outside it); the commit needs `pending.json`. No staged binary.
- **The fix (`internal/selfupdate/prune.go`).** After the wrapper's `commit` returned success, `MaybeCommit` removes
every staged binary but the running version's. Nothing is removed while a pending marker exists or cannot be read,
at a plain start, after a rollback (the failed version's binary stays to be read), or when the running version is
not a version this code can name a file for. Only a regular file with exactly the name `Execute` writes is removed,
one `os.Remove` per name; a failed removal is logged and never fails the commit. A download and a prune take turns
(`stagingMu`).
- **Tests `TestPrune_*` (6).** After six updates the folder holds one binary and every bystander (unknown names, a
folder and a symlink that carry a staged name); a pending or unreadable marker refuses; only a healthy commit prunes;
the name rule follows `versionRe`; the wrapper's `commit` and `rollback` blocks name no staged binary; and the
wrapper's REAL `commit` and `rollback` verbs, run from a sandboxed copy, still put the previous binary back after
four pruned updates. **Red seen three times:** no prune call; no pending refusal; no regular-file check.
- **Not reached.** `/usr/local/bin/felhom-agent.bak-*` (place H13): root-owned, in a folder the agent cannot write, made
by a person. R-1368 part 4 stays open. A box gets the clean-up at its first good update AFTER the one that brings
this code; nothing was run on a box.
## v0.172.1 (2026-11-07) — proxmox-grub joins the boot-chain lane: the nightly pve step stops being refused R6 — RELEASED and VOUCHED 2026-11-07 (release-agent.sh from `39db030`; agent sha256 505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8, bundle 3e7acaee4500893bfaf567b1d427790ef89bf41f08cb6a1419ee528aeb40ddc3)
- **The defect (measured on demo-felhom, 2026-11-05, 11-06, 11-07).** Proxmox published `proxmox-grub` 2.12-9+pmx4. It
depends on `grub-efi-amd64` of the same version. `HOST_SLOW_RE` matches `grub` only at the START of a name, so
`proxmox-grub` landed in the ring-0 pve plan; the simulation then pulled `grub-efi-amd64`, which is not in the plan,
and `felhom-os-apply` refused the WHOLE pve step (R6 „the plan would touch grub-efi-amd64") every night — 34 Proxmox
userspace upgrades held with it, and an `os_update_failed` error mail each morning. The same class as the 2026-10-09
`proxmox-secure-boot-support` fix. Read-only proof on the box: `apt-get -s install --only-upgrade proxmox-grub` →
`Inst grub-efi-amd64 …pmx4`.
- **The fix.** `proxmox-grub` joins `HOST_SLOW_RE`: GRUB's meta-package now waits with the rest of the boot chain (the
slow lane, the operator's signed step), and the pve step runs without it.
- **Test** `test_ring0_pending_pve_leaves_proxmox_grub_out`; red seen before the fix (proxmox-grub entered the plan), green
after; the wrapper suite 149 of 149.
- **Delivered 2026-11-07 (operator, signed jobs):** demo-felhom, demo-hp, Tester 1 — `agent_update` committed
(23:05/23:13/23:13 CET), then the bundle: `BUNDLE DONE agent=0.172.1 written=1 same=30 self-check=ok`, capability probe
77/77 on each. `pinned.day0_agent` = 0.172.1 (D50).
- **Delivery.** `configs/` changed, so a box needs BOTH signed jobs: `agent_update` 0.172.1, then `agent_config_update`
0.172.1 with the bundle sha. Not vouched by CC. Bundle vs 0.170.0 (what the boxes run): no path added or removed, only
`felhom-os-apply` changed — so no step bundle is needed (R-880). Both downloads re-checked by sha.
## v0.172.0 (2026-11-03) — R-1302: the first grant of a tier is no loss — RELEASED and VOUCHED 10:18:10 CET (release-agent.sh from a clean clone of `ecbb0cc`; agent sha256 2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7, config bundle cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce; configs/ unchanged since 0.171.0, so the signed delivery was agent_update alone)
- **R-1302 — the first grant of a tier is no loss.** A fresh install arms the whole-guest tier before its storage exists, so the
first probe found no grant, repaired it and logged ERROR „GRANT WAS MISSING … investigate the loss (R-190)" plus a DEGRADED
capability, on every new box (VM 9375, 2026-10-26). The agent now remembers which tiers have EVER held their grant
(`<state dir>/store-grant-seen.json`, written by a healthy probe). A first grant logs INFO „granted" and reports healthy; a
grant that existed and vanished keeps R-190's ERROR and its one degraded cycle. A doubt keeps the ERROR: an unreadable or
corrupt file, or no file path. A box updated to this agent records its tiers at the first healthy probe, so a loss in the
one probe before that would read as a first grant. Tests `TestR1302_*`; red seen with the first-grant branch removed and on
three more mutants.
## v0.171.0 (2026-10-30) — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED — RELEASED (release-agent.sh from a clean clone of `bd18347`; agent sha256 c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5, config bundle 95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b; binary-only signed agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-30 10:47:36 CET; `pinned.day0_agent` = 0.171.0). Live on Tester 1: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the restart 09:31:22 → healthy 09:31:24; the hub mailed `controller_unresponsive` and `controller_unresponsive_restart` (hub 0.178.0), not `controller_restarted_by_agent`
- **The measurement (operator brief 2026-10-29 Part 2, read 2026-10-30).** Docker's own health record of the controller
on Tester 1 and demo-hp, every probe from 2026-10-29 11:12 to 2026-10-30 07:00 UTC: 4 733 probes through the night
+17 -1
View File
@@ -1,12 +1,28 @@
# CONTEXT — felhom-agent working state
> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: the self-update folder cleans itself (R-1368 part 2).** After a healthy `commit` the agent removes every staged binary but the running version's (`internal/selfupdate/prune.go`; tests `TestPrune_*`, the wrapper's real `rollback` verb run in a sandbox). No version bump, no build, no vouch: it rides the next agent release. A box cleans its folder at the first good update AFTER that release. Not reached: `/usr/local/bin/felhom-agent.bak-*` (root-owned, R-1368 part 4). The old-kernel clean-up is a design only (felhom.eu `11-os-updates.md` §5.11.1, proven by hand on demo-hp); nothing of it is in this repo.
> **2026-11-07 (attended) — v0.172.1 RELEASED and VOUCHED; on the three test boxes (agent + bundle, 77/77).** `proxmox-grub` joins `HOST_SLOW_RE` (the N100's ring-0 pve step was refused R6 three nights: proxmox-grub pmx4 pulls grub-efi-amd64). sha `505a0594…`, bundle `3e7acaee…` (only `felhom-os-apply` changed vs 0.170.0, no step bundle). Done: vouch, both signed jobs on the three boxes, `pinned.day0_agent` = 0.172.1. Next: the N100's pve step on its next night (positive line: `os-apply: DONE` for `layer=pve:9201`, no R6).
> **2026-11-03 (day, attended) — v0.172.0 RELEASED and VOUCHED (10:18:10 CET), R-1302.** `2c0222cd…`, bundle `cc34b0bc…`, `pinned.day0_agent` = 0.172.0; on the three test boxes by signed `agent_update`. The appliance path pre-positions the grant, so a plain first start logs neither line; staged live both ways (felhom.eu `audits/release-2026-11-03/part3/`).
>
> **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
> but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
> alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
> refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
> (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
> carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
> `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
> updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
> `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
> **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
> `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
> sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
> (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
> `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
> (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
> **R-1344 (the stuck-controller restart) is still ALARM ONLY** — the night's health log is read on 2026-10-30.
> **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
+19 -32
View File
@@ -1,36 +1,23 @@
# REPORT — v0.170.0 released and vouched: „Reboot guest" from the hub (D65 = A, 2026-10-29 evening, attended)
# REPORT — the self-update folder cleans itself (R-1368 part 2) — committed, NOT released (2026-11-10, operator brief)
**Plain words first.** The operator can now reboot a box's guest from the hub's host page, beside „Restart controller".
The agent does it (`pct reboot` of that guest — never the host). It refuses during a backup, a restore, an update or a
controller swap, more than once an hour, and under the restart brake. The result says when the guest and the controller
were back. The controller that starts after the reboot tells the household for a day that Felhom support restarted
the box.
**Plain words first.** Each agent update left its downloaded file on the box for ever: 49 files and 740 MB on demo-hp.
The agent now removes the old files after an update that ended well. It keeps the file of the version that runs.
This is a commit only. No version, no build, no vouch. No box has it. It waits for the next agent release.
## Commits
The full report of the run is in `felhom.eu`: `REPORT-endings-kernels-fixes-2026-11-10.md`.
| Commit | CI (job id) | What |
|---|---|---|
| `ca15a31` | 2520, 2524 green | `reboot_guest` in the closed list (`opreboot.go`): guards (provisioned, running, not locked, not parked; the heavy-operation gate held until the guest is up; `jobBlocker`; the controller's busy file, fresh ≤ 90 s, fail closed on any read error but „no file"; one per hour per guest, persisted; the restart brake counts it); `rebootInFlight` keeps the guest-power watchdog and the supervisor away; the result with both times; `GET /guest/operator-reboot` for the household line; one new read-only sudo vector (`cat` of the controller's busy file) + capability `controller-busy-read`. 19 mutants, each killed (one survivor found and fixed: a duplicate swap check). |
| `6578a8b` | 2525 green | v0.170.0 CHANGELOG heading; `pinned.day0_agent` = 0.170.0 (D50), after the vouch. |
| | |
|---|---|
| Change | `internal/selfupdate/prune.go` (new); the call in `MaybeCommit` (`commit.go`); one lock in `Execute` (`executor.go`) |
| Rule | After the wrapper's `commit` returned success: remove every staged binary but the running version's |
| Never removed | anything while `pending.json` exists or cannot be read; anything at a plain start or after a rollback; `pending.json`; a folder; a symlink; a name the code does not know |
| Rollback | Needs `pending.json` and `/usr/local/bin/felhom-agent.prev`. It reads no staged binary (`configs/felhom-selfupdate-guarded`, read first) |
| Tests | `TestPrune_*`, 6 tests. One runs the wrapper's real `commit` and `rollback` verbs from a sandboxed copy |
| Red seen | 3 times: no prune call; no pending refusal; no regular-file check |
| Gates | `python3 scripts/agent_gates.py --fast`: all OK |
| Not reached | `/usr/local/bin/felhom-agent.bak-*`: root-owned, made by a person. R-1368 part 4 stays open |
| Not proven | On a box. A box cleans its folder at the first good update AFTER the release that brings this code |
| Machines | None touched by this change |
## Release
- `release-agent.sh 0.170.0` from a clean clone of `ca15a31`: sha256 `da8d12a05ef607c338b9ba307e7a81b18253a3188721bbd8421ba36658a22404`,
bundle `03f2ff6442fc3ca600b1f25f26767ce4da466a770a258f3e362cf95f6e76b543`, tag `v0.170.0`, verified by download.
- configs/ changed, so per box a signed `agent_config_update` AND `agent_update` (Tester 1, demo-hp, demo-felhom): all
six COMPLETED; each box „config bundle INSTALLED", capability probe 76/76 ok, „update committed".
- Vouched by the operator 2026-10-29 18:57:48 CET (the save took 38.3 s — felhom.eu R-1346).
## Live proofs (Tester 1, through the hub endpoint the button posts, CC's `cc-hosts` token; controller 0.349.0-rc.1)
- **Refused during an off-site run:** `offsite_backup_now` pressed 18:03:36 UTC → the controller's busy file 11 s later →
„Reboot guest" → „refused: the controller is busy — a backup, a restore or an off-site run is running"; the guest kept
its 13 h uptime.
- **The reboot:** pressed 18:05:08 UTC → „done — rebooted; the guest was up after 67 s, the controller answered after
79 s", on the hub 88 s after the press. Box walk after: 10 containers, none unhealthy, 0 errors, login and file manager
200. The household dashboard: „A Felhom ügyfélszolgálata 19:05-kor újraindította a dobozt."
- **A second press inside the hour:** „refused: guest 9201 was rebooted from the hub at 18:05 UTC — at most one reboot
per hour (next from 19:05 UTC)".
- The release-build reboot: see felhom.eu `REPORT-release-2026-10-29c.md`.
Evidence: felhom.eu `documentation/audits/release-2026-10-29c/`.
**Next.** A release session builds and vouches it. After two updates on a test box, read the folder: it must hold one
binary.
+1 -1
View File
@@ -35,7 +35,7 @@
| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()`. **The staging folder cleans itself** (`prune.go`, R-1368): after a healthy `commit` only the running version's staged binary stays; a new file kept in `<state dir>/selfupdate/` must NOT be named `felhom-agent-<version>` or the prune removes it |
| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
+82 -3
View File
@@ -521,6 +521,66 @@ type storeGrantRepairer struct {
mu sync.Mutex
last map[string]time.Time // target id → last ATTEMPT (success or failure)
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
// seenPath (R-1302) is the file that remembers which tiers have EVER held the grant: only those can LOSE it. "" = no
// memory, so every missing grant reads as a loss — the old, loud behaviour.
seenPath string
}
// everHeld reports whether this tier's grant was ever seen on this box (R-1302). The direction of a doubt is „yes": an
// unreadable or corrupt file keeps the loss ERROR, so a real loss is never quietened by a broken record.
func (r *storeGrantRepairer) everHeld(target string) bool {
if r == nil || r.seenPath == "" {
return true
}
r.mu.Lock()
defer r.mu.Unlock()
seen, err := readGrantSeen(r.seenPath)
return err != nil || seen[target]
}
// noteHeld records that this tier holds the grant. A write that fails is logged and costs one false „first grant" at most
// after a restart — the old file, or none, stands.
func (r *storeGrantRepairer) noteHeld(target string) {
if r == nil || r.seenPath == "" {
return
}
r.mu.Lock()
defer r.mu.Unlock()
seen, err := readGrantSeen(r.seenPath)
if err != nil {
seen = map[string]bool{} // a corrupt file is replaced by what is true now
}
if seen[target] {
return
}
seen[target] = true
b, _ := json.Marshal(seen)
tmp := r.seenPath + ".tmp"
if werr := os.WriteFile(tmp, b, 0o600); werr == nil {
werr = os.Rename(tmp, r.seenPath)
if werr == nil {
return
}
err = werr
} else {
err = werr
}
repairLogger(r).Warn("store-grant: could not record that this tier holds its grant (R-1302)", "target", target, "err", err)
}
func readGrantSeen(path string) (map[string]bool, error) {
b, err := os.ReadFile(path)
if os.IsNotExist(err) {
return map[string]bool{}, nil
}
if err != nil {
return nil, err
}
seen := map[string]bool{}
if err := json.Unmarshal(b, &seen); err != nil {
return nil, err
}
return seen, nil
}
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
@@ -631,6 +691,7 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
// host-report has certainly carried it. Without this latch the repairing probe may be a
// log-only one and the hub never learns anything happened (measured live, see the window's
// comment).
repair.noteHeld(targetID) // R-1302: a tier that holds its grant can lose it from now on
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
}
@@ -679,8 +740,25 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
// change, no hub change, no new event type. The `Feature` text carries the explanation because
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
return storeGrantAfterRepair(targetID, critical, repair, s)
}
// storeGrantAfterRepair is the verdict once a repair is confirmed (split out for the tests). R-1302: the FIRST grant of a
// tier that never held one is no loss — on a fresh install the whole-guest tier is armed before its storage exists, so
// every new box logged „investigate the loss" and a DEGRADED capability at its first start, which teaches the reader to
// skip the line. That grant logs INFO and reports healthy. A tier seen holding its grant, or one this box cannot tell
// about, keeps R-190's ERROR and its one degraded cycle. Pinned by TestR1302_*.
func storeGrantAfterRepair(targetID string, critical bool, repair *storeGrantRepairer, healthy capability.Status) capability.Status {
if !repair.everHeld(targetID) {
repair.noteHeld(targetID)
repairLogger(repair).Info("store-grant: granted — the first grant of this tier, nothing was lost (R-1302)",
"target", targetID, "privilege", storeGrantRequiredPriv,
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
healthy.Status, healthy.Reason = capability.StatusOK, ""
return healthy
}
repair.noteRepaired(targetID, time.Now())
s = storeGrantRepairedVerdict(targetID, critical)
s := storeGrantRepairedVerdict(targetID, critical)
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
"target", targetID, "privilege", storeGrantRequiredPriv,
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
@@ -853,8 +931,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
// invoked through the privileged path, which prepends sudo itself.
grantRepairer := &storeGrantRepairer{
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
log: logger,
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
log: logger,
seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json"), // R-1302
}
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
+101
View File
@@ -1,11 +1,14 @@
package main
import (
"bytes"
"context"
"errors"
"go/ast"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -415,3 +418,101 @@ func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
"be missed entirely", storeGrantRepairReportWindow)
}
}
// ── R-1302 — the first grant of a tier is no loss ──────────────────────────────────────
//
// A fresh install arms the whole-guest tier before its storage exists, so its first probe found no grant, repaired it and
// logged „GRANT WAS MISSING … investigate the loss" with a DEGRADED capability — on every new box (VM 9375, 2026-10-26).
//
// RED-PROOF (2026-11-02): seen failing before storeGrantAfterRepair existed (build), and with its first-grant branch removed
// TestR1302_TheFirstGrantIsNoLoss fails on „a first grant must not report a loss".
func r1302Repairer(t *testing.T, path string) (*storeGrantRepairer, *bytes.Buffer) {
t.Helper()
var buf bytes.Buffer
return &storeGrantRepairer{run: (&fakeRepairRunner{}).Run, seenPath: path,
log: slog.New(slog.NewTextHandler(&buf, nil))}, &buf
}
func TestR1302_TheFirstGrantIsNoLoss(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
r, buf := r1302Repairer(t, path)
healthy := capability.Status{Name: "pve:store-grant:felhom-pbs", Status: capability.StatusDegraded, Reason: "missing", Critical: true}
s := storeGrantAfterRepair("felhom-pbs", true, r, healthy)
if s.Status != capability.StatusOK || s.Reason != "" {
t.Fatalf("a first grant must not report a loss: status %q reason %q", s.Status, s.Reason)
}
got := buf.String()
if strings.Contains(got, "investigate the loss") || strings.Contains(got, "level=ERROR") {
t.Fatalf("a first grant must not log the loss ERROR: %s", got)
}
if !strings.Contains(got, "level=INFO") || !strings.Contains(got, "granted") || !strings.Contains(got, "felhom-pbs") {
t.Fatalf("a first grant says so at INFO: %s", got)
}
if r.recentlyRepaired("felhom-pbs", time.Now()) {
t.Fatal("a first grant must not arm the loss report's latch")
}
}
func TestR1302_AGrantThatExistedAndVanishedKeepsTheError(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
first, _ := r1302Repairer(t, path)
first.noteHeld("felhom-pbs") // a healthy probe, in an earlier run of the agent
r, buf := r1302Repairer(t, path) // the agent restarted: the memory is the file, not the process
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
if s.Status != capability.StatusDegraded || !strings.Contains(s.Feature, "R-190") {
t.Fatalf("a grant that existed and vanished must keep its degraded cycle; got %q / %q", s.Status, s.Feature)
}
if got := buf.String(); !strings.Contains(got, "level=ERROR") || !strings.Contains(got, "investigate the loss") {
t.Fatalf("a grant that existed and vanished must keep its ERROR: %s", got)
}
if !r.recentlyRepaired("felhom-pbs", time.Now()) {
t.Fatal("the loss must arm the report's latch, or the hub may never hear it")
}
}
// After a first grant the tier counts as held: the NEXT disappearance is a loss.
func TestR1302_AfterTheFirstGrantALossIsALoss(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
r, _ := r1302Repairer(t, path)
storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
if s.Status != capability.StatusDegraded {
t.Fatalf("the second disappearance is a loss; got %q", s.Status)
}
}
// The direction of a doubt: a broken record, a repairer with no record, or no repairer at all keep the loud way.
func TestR1302_ADoubtKeepsTheError(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
t.Fatal(err)
}
r, _ := r1302Repairer(t, path)
if s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{}); s.Status != capability.StatusDegraded {
t.Fatalf("a corrupt record must not quieten a loss; got %q", s.Status)
}
noMemory, _ := r1302Repairer(t, "")
if s := storeGrantAfterRepair("felhom-pbs", true, noMemory, capability.Status{}); s.Status != capability.StatusDegraded {
t.Fatalf("no record must not quieten a loss; got %q", s.Status)
}
if s := storeGrantAfterRepair("felhom-pbs", true, nil, capability.Status{}); s.Status != capability.StatusDegraded {
t.Fatalf("no repairer must not quieten a loss; got %q", s.Status)
}
}
// The wiring, read from the source: production names the file, and a healthy probe records the tier.
func TestR1302_TheMemoryIsWired(t *testing.T) {
b, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
src := string(b)
if !strings.Contains(src, `seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json")`) {
t.Fatal("main.go must give the repairer its file, or every first grant reads as a loss")
}
start := strings.Index(src, "func storeGrantStatus(")
if start < 0 || !strings.Contains(src[start:start+3000], "repair.noteHeld(targetID)") {
t.Fatal("a healthy probe must record the tier, or a real loss would read as a first grant")
}
}
+4 -1
View File
@@ -97,9 +97,12 @@ OOMCHECK_SETTLE = 2
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
# proxmox-grub is GRUB's Proxmox meta-package (it depends on grub-efi-amd64 of the same version); the `grub` above is
# anchored at the start, so it missed it. Measured 2026-11-05..07 on demo-felhom: its upgrade pulled grub-efi-amd64 and
# the whole pve step was refused R6 every night. Pinned by test_ring0_pending_pve_leaves_proxmox_grub_out.
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
r"proxmox-secure-boot-support)")
r"proxmox-secure-boot-support|proxmox-grub)")
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
+18
View File
@@ -1494,6 +1494,24 @@ class PVELane(unittest.TestCase):
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
# 2026-11-05..07, demo-felhom (EFI): proxmox-grub 2.12-9+pmx2 -> pmx4 depends on grub-efi-amd64 of the same version;
# HOST_SLOW_RE's `grub` is anchored at the start, so proxmox-grub fell into the pve plan, the simulation pulled
# grub-efi-amd64 and the whole pve step was refused R6 every night. It is a boot-chain package: left out.
# RED-PROOF: drop proxmox-grub from HOST_SLOW_RE -> it is in the plan -> this test fails.
def test_ring0_pending_pve_leaves_proxmox_grub_out(self):
f = pve_fake(ring0=True)
f.plan["select"], f.plan["packages"] = "pending-pve", []
f.installed.update({"proxmox-grub": "2.12-9+pmx2"})
f.pending_sim = [
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-grub [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
"Inst grub-efi-amd64 [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
self.assertEqual(f.installed["proxmox-grub"], "2.12-9+pmx2")
def test_select_pending_pve_needs_the_pve_layer(self):
f = Fake()
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
+7
View File
@@ -128,4 +128,11 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
if m.onCommitted != nil {
m.onCommitted(ctx)
}
// R-1368 part 2: the update ended healthy — the folder keeps the running version's binary and no other
// (prune.go). A refusal or a failed removal is logged and changes nothing about the commit.
if removed, err := pruneStaged(m.stateDir, m.runningVersion, m.logger); err != nil {
m.logger.Warn("selfupdate: old staged binaries NOT removed after the commit", "err", err)
} else if len(removed) > 0 {
m.logger.Info("selfupdate: old staged binaries removed after the commit", "count", len(removed), "kept", "felhom-agent-"+m.runningVersion)
}
}
+3
View File
@@ -103,6 +103,9 @@ func (e *Executor) Execute(ctx context.Context, op string, params json.RawMessag
}
e.logger.Debug("agent_update: op invariants passed (semver + sha format)", "version", p.Version)
// The staged file is this call's from the download to the hand-off: a prune (prune.go) waits for it.
stagingMu.Lock()
defer stagingMu.Unlock()
dir := stagingDir(e.stateDir)
if err := os.MkdirAll(dir, 0o750); err != nil {
return fmt.Errorf("agent_update: staging dir: %w", err)
+96
View File
@@ -0,0 +1,96 @@
package selfupdate
import (
"fmt"
"log/slog"
"os"
"path/filepath"
"regexp"
"sort"
"sync"
)
// The self-update folder keeps what it needs and no more (R-1368 part 2; space list 2026-11-10, place H12).
//
// WHAT WAS WRONG. Execute downloads each new agent to <StateDir>/selfupdate/felhom-agent-<version> and removes that
// file only when the download or its checksum fails. A good update left it there for ever: measured 2026-11-10, 49 /
// 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1 — one for each version the box ever got.
//
// WHAT THE FOLDER IS NEEDED FOR. The root wrapper (felhom-os-apply, mode agent_update) reads the staged file ONCE,
// copies it to its own root-owned folder and flips from that copy. After that hand-off nothing reads the staged file:
// - the rollback (felhom-selfupdate-guarded `rollback`) needs two files and neither is a staged binary:
// `pending.json` in this folder, and `/usr/local/bin/felhom-agent.prev` outside it;
// - the commit needs `pending.json` only.
//
// So a staged binary is dead weight as soon as its update ended. Pinned by TestPrune_RollbackReadsNoStagedBinary (the
// wrapper's text) and TestPrune_RealRollbackStillWorksAfterNUpdates (the wrapper's rollback verb, run for real).
//
// THE RULE. After an update ended HEALTHY (the wrapper's `commit` returned success), remove every staged binary but
// the one of the running version. Nothing is removed:
// - while a pending marker exists or cannot be read — an update is in flight, and that is the only time a rollback
// can fire;
// - after a rollback, or at a plain start — only the commit path calls this, so the binary of a version that failed
// stays for the operator to look at until the next good update;
// - that is not a regular file with exactly the name Execute gives a staged binary. `pending.json`, a folder, a
// symlink, and any name this code does not know are never touched.
//
// NOT REACHED BY THIS: `/usr/local/bin/felhom-agent.bak-*` (space list H13). Those files are root-owned, in a folder
// the agent cannot write, and no code of the agent makes them (a person's copy before a hand deploy). R-1368 part 4.
// stagedNameRe is the exact name Execute gives a staged binary: "felhom-agent-" + a version that versionRe accepts.
// Kept in step with versionRe by TestPrune_NameShapeFollowsTheVersionRule.
var stagedNameRe = regexp.MustCompile(`^felhom-agent-([0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?)$`)
// stagingMu makes a download and a prune take turns. Without it a prune that runs while Execute downloads the NEXT
// version would remove that half-written file (its name is a staged binary's and it is not the running version).
var stagingMu sync.Mutex
// pruneStaged removes the staged binaries the folder no longer needs and returns the names it removed, sorted. It
// refuses (removes nothing, returns an error) while an update is in flight. A file it cannot remove is logged and
// skipped: a failed clean-up must never fail the commit that came before it.
func pruneStaged(stateDir, runningVersion string, logger *slog.Logger) ([]string, error) {
logger = orDefaultLogger(logger)
stagingMu.Lock()
defer stagingMu.Unlock()
if !versionRe.MatchString(runningVersion) {
return nil, fmt.Errorf("selfupdate prune: refusing — running version %q is not a version this code knows; it could not name the file to keep", runningVersion)
}
pm, err := readPending(stateDir)
if err != nil {
return nil, fmt.Errorf("selfupdate prune: refusing — the pending marker cannot be read: %w", err)
}
if pm != nil {
return nil, fmt.Errorf("selfupdate prune: refusing — an update to %s is pending (a rollback may still fire)", pm.NewVersion)
}
dir := stagingDir(stateDir)
entries, err := os.ReadDir(dir)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, fmt.Errorf("selfupdate prune: read %s: %w", dir, err)
}
keep := "felhom-agent-" + runningVersion
var removed []string
for _, ent := range entries {
name := ent.Name()
if name == keep || !stagedNameRe.MatchString(name) {
continue // the running version's binary, pending.json, or a name this code does not know
}
if !ent.Type().IsRegular() {
logger.Warn("selfupdate prune: a staged-binary name that is not a regular file — left alone", "name", name, "type", ent.Type().String())
continue
}
// name came from ReadDir and matched stagedNameRe, so it holds no path separator: this is one exact file.
if err := os.Remove(filepath.Join(dir, name)); err != nil {
logger.Warn("selfupdate prune: cannot remove an old staged binary — left for the next good update", "name", name, "err", err)
continue
}
logger.Debug("selfupdate prune: removed an old staged binary", "name", name)
removed = append(removed, name)
}
sort.Strings(removed)
logger.Info("selfupdate prune: the self-update folder holds what it needs", "kept", keep, "removed", len(removed))
return removed, nil
}
+373
View File
@@ -0,0 +1,373 @@
package selfupdate
import (
"bytes"
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
"testing"
)
// R-1368 part 2 — the self-update folder keeps what it needs and no more (prune.go).
//
// RED-PROOFS (each observed 2026-11-10, then restored):
// - drop the pruneStaged call from MaybeCommit → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with
// "after update 1 (0.170.0) the folder holds [… felhom-agent-0.169.0 felhom-agent-0.170.0 …]", and
// TestPrune_RealRollbackStillWorksAfterNUpdates fails too;
// - drop the pending-marker refusal from pruneStaged → TestPrune_RefusesWhileAnUpdateIsPending fails with
// "removed [...] while an update was pending";
// - drop the IsRegular check → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with "a symlink with a
// staged-binary name was removed".
// committingWrapper is the guarded wrapper's `commit` as far as this package can see it: it clears the pending marker.
type committingWrapper struct {
stateDir string
calls int
}
func (w *committingWrapper) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
if len(args) == 1 && args[0] == "commit" {
w.calls++
if err := os.Remove(pendingPath(w.stateDir)); err != nil {
return nil, []byte(err.Error()), err
}
}
return []byte("ok"), nil, nil
}
func stage(t *testing.T, stateDir, version string) {
t.Helper()
if err := os.MkdirAll(stagingDir(stateDir), 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(stagingDir(stateDir), "felhom-agent-"+version), []byte("binary "+version), 0o755); err != nil {
t.Fatal(err)
}
}
func folder(t *testing.T, stateDir string) []string {
t.Helper()
ents, err := os.ReadDir(stagingDir(stateDir))
if err != nil {
t.Fatal(err)
}
var out []string
for _, e := range ents {
out = append(out, e.Name())
}
sort.Strings(out)
return out
}
// The consequence: after N good updates the folder holds ONE staged binary — the running version's — and every file
// that is not a staged binary is still there.
func TestPrune_AfterNUpdatesTheFolderHoldsOneBinary(t *testing.T) {
stateDir := t.TempDir()
dir := stagingDir(stateDir)
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
// Bystanders: names this code does not know, and two things that carry a staged binary's NAME but are not files.
outside := filepath.Join(t.TempDir(), "precious")
if err := os.WriteFile(outside, []byte("not ours"), 0o644); err != nil {
t.Fatal(err)
}
for _, n := range []string{"notes.txt", "felhom-agent-0.1.0.partial", "felhom-agent.bak-0.139.0"} {
if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
}
if err := os.Mkdir(filepath.Join(dir, "felhom-agent-9.9.9"), 0o750); err != nil {
t.Fatal(err)
}
if err := os.Symlink(outside, filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
t.Fatal(err)
}
bystanders := []string{"felhom-agent-0.1.0.partial", "felhom-agent-8.8.8", "felhom-agent-9.9.9", "felhom-agent.bak-0.139.0", "notes.txt"}
versions := []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1", "0.173.0-rc.1", "0.173.0"}
prev := "0.169.0"
stage(t, stateDir, prev) // what an old agent left behind
for i, v := range versions {
stage(t, stateDir, v) // Execute downloaded it
writePending(t, stateDir, PendingMarker{OldVersion: prev, NewVersion: v, SHA256: "ab", AppliedAt: "t"}) // the wrapper flipped
wrap := &committingWrapper{stateDir: stateDir}
m := newMgr(t, stateDir, v, wrap, nil)
m.MaybeCommit(context.Background())
if wrap.calls != 1 {
t.Fatalf("update %d (%s): commit calls = %d, want 1", i+1, v, wrap.calls)
}
want := append([]string{"felhom-agent-" + v}, bystanders...)
sort.Strings(want)
got := folder(t, stateDir)
if strings.Join(got, " ") != strings.Join(want, " ") {
if _, err := os.Lstat(filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
t.Fatalf("a symlink with a staged-binary name was removed: %v", err)
}
t.Fatalf("after update %d (%s) the folder holds %v, want %v", i+1, v, got, want)
}
prev = v
}
if b, err := os.ReadFile(outside); err != nil || string(b) != "not ours" {
t.Fatalf("the file behind the symlink changed: %q %v", b, err)
}
}
// While an update is pending — the only time a rollback can fire — nothing is removed.
func TestPrune_RefusesWhileAnUpdateIsPending(t *testing.T) {
stateDir := t.TempDir()
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
stage(t, stateDir, v)
}
writePending(t, stateDir, PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0", SHA256: "ab", AppliedAt: "t"})
before := folder(t, stateDir)
removed, err := pruneStaged(stateDir, "0.172.0", nil)
if err == nil || len(removed) != 0 {
t.Fatalf("removed %v while an update was pending (err=%v) — want a refusal", removed, err)
}
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
t.Fatalf("removed %v while an update was pending; folder now %v", before, got)
}
// The same through the commit path: a `commit` that reports success but leaves the marker (fakeWrapper does) must
// not lead to a prune.
m := newMgr(t, stateDir, "0.172.0", &fakeWrapper{}, nil)
m.MaybeCommit(context.Background())
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
t.Fatalf("the commit path pruned with the marker still there; folder now %v", got)
}
// A marker nobody can read is a refusal too.
if err := os.WriteFile(pendingPath(stateDir), []byte("{not json"), 0o644); err != nil {
t.Fatal(err)
}
if removed, err := pruneStaged(stateDir, "0.172.0", nil); err == nil || len(removed) != 0 {
t.Fatalf("an unreadable marker must refuse; removed %v err=%v", removed, err)
}
}
// Only a commit prunes: a plain start, a version mismatch and a failed commit remove nothing — so the binary of a
// version that was rolled back stays for the operator.
func TestPrune_OnlyAHealthyCommitPrunes(t *testing.T) {
cases := []struct {
name string
pending *PendingMarker
running string
wrap WrapperRunner
}{
{"plain start, no marker (also the state after a rollback)", nil, "0.171.0", &fakeWrapper{}},
{"marker names another version", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.171.0", &fakeWrapper{}},
{"commit fails", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.172.0", &fakeWrapper{err: fmt.Errorf("sudo: no")}},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
stateDir := t.TempDir()
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
stage(t, stateDir, v)
}
if c.pending != nil {
writePending(t, stateDir, *c.pending)
}
before := folder(t, stateDir)
newMgr(t, stateDir, c.running, c.wrap, nil).MaybeCommit(context.Background())
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
t.Fatalf("folder changed without a healthy commit: %v → %v", before, got)
}
})
}
// A running version this code cannot name a file for: refuse, do not guess.
stateDir := t.TempDir()
stage(t, stateDir, "0.170.0")
if removed, err := pruneStaged(stateDir, "dev", nil); err == nil || len(removed) != 0 {
t.Fatalf("running version \"dev\" must refuse; removed %v err=%v", removed, err)
}
}
// The name rule and the version rule say the same thing: every name Execute can write is a name the prune knows, and
// nothing else is.
func TestPrune_NameShapeFollowsTheVersionRule(t *testing.T) {
for _, v := range []string{"0.70.1", "0.172.1", "10.0.0", "0.70.2-crash", "0.173.0-rc.1"} {
if !versionRe.MatchString(v) {
t.Fatalf("test input %q is not a version versionRe accepts", v)
}
if !stagedNameRe.MatchString("felhom-agent-" + v) {
t.Errorf("Execute can stage felhom-agent-%s but the prune does not know that name", v)
}
}
for _, n := range []string{"pending.json", "felhom-agent", "felhom-agent-", "felhom-agent-1.2", "felhom-agent-0.1.0.partial",
"felhom-agent.bak-0.139.0", "felhom-agent.prev", "felhom-agent-0.1.0/x", "../felhom-agent-0.1.0", "felhom-agent-0.1.0 ", "xfelhom-agent-0.1.0"} {
if stagedNameRe.MatchString(n) {
t.Errorf("the prune would treat %q as a staged binary", n)
}
}
m := stagedNameRe.FindStringSubmatch("felhom-agent-0.173.0-rc.1")
if m == nil || !versionRe.MatchString(m[1]) {
t.Fatalf("the version part of a staged name must satisfy versionRe; got %v", m)
}
}
func wrapperText(t *testing.T) string {
t.Helper()
b, err := os.ReadFile(filepath.Join("..", "..", "configs", "felhom-selfupdate-guarded"))
if err != nil {
t.Fatalf("the guarded wrapper is the other half of this proof and must be readable: %v", err)
}
return string(b)
}
// The stated dependency, pinned: the wrapper's `rollback` and `commit` read the pending marker and `.prev`, and no
// staged binary. If either verb starts to read the staging folder, this fails and prune.go's rule must be looked at.
func TestPrune_RollbackReadsNoStagedBinary(t *testing.T) {
s := wrapperText(t)
for _, want := range []string{"\nBIN=/usr/local/bin/felhom-agent\n", "\nPREV=$BIN.prev\n", "\nSTAGING=/var/lib/felhom-agent/selfupdate\n", "\nPENDING=$STAGING/pending.json\n"} {
if !strings.Contains(s, want) {
t.Fatalf("the wrapper no longer holds the line %q — the folder's rule (prune.go) was written against it", strings.TrimSpace(want))
}
}
for _, verb := range []string{"commit", "rollback"} {
i := strings.Index(s, "\n"+verb+")\n")
if i < 0 {
t.Fatalf("no %s) block in the wrapper", verb)
}
block := s[i:]
j := strings.Index(block, "\n ;;\n")
if j < 0 {
t.Fatalf("no end of the %s) block", verb)
}
block = block[:j]
if !strings.Contains(block, "$PENDING") {
t.Fatalf("%s) does not read $PENDING any more", verb)
}
code := regexp.MustCompile(`(?m)^\s*#.*$`).ReplaceAllString(block, "")
for _, bad := range []string{"STAGING", "staged", "felhom-agent-"} {
if strings.Contains(code, bad) {
t.Fatalf("%s) now mentions %q — it may read a staged binary, and the prune removes those", verb, bad)
}
}
}
if stagedNameRe.MatchString("pending.json") || stagedNameRe.MatchString("felhom-agent.prev") {
t.Fatal("the prune would remove a file the rollback needs")
}
}
// The consequence, with the wrapper's own code: after N good updates (each committed by the REAL `commit` verb and
// pruned), update N+1 goes bad and the REAL `rollback` verb puts the previous binary back. The wrapper runs from a copy
// whose four fixed paths point into a temp folder; systemctl, logger and chown are stubs.
func TestPrune_RealRollbackStillWorksAfterNUpdates(t *testing.T) {
sh, err := exec.LookPath("sh")
if err != nil {
t.Skip("no sh on this machine")
}
root := t.TempDir()
stateDir := filepath.Join(root, "state")
binDir := filepath.Join(root, "bin")
stubs := filepath.Join(root, "stubs")
for _, d := range []string{stagingDir(stateDir), binDir, stubs, filepath.Join(root, "rootstaging")} {
if err := os.MkdirAll(d, 0o750); err != nil {
t.Fatal(err)
}
}
live := filepath.Join(binDir, "felhom-agent")
s := wrapperText(t)
for old, repl := range map[string]string{
"\nBIN=/usr/local/bin/felhom-agent\n": "\nBIN=" + live + "\n",
"\nSTAGING=/var/lib/felhom-agent/selfupdate\n": "\nSTAGING=" + stagingDir(stateDir) + "\n",
"\nROOT_STAGING=/var/lib/felhom-os-apply/agent-update\n": "\nROOT_STAGING=" + filepath.Join(root, "rootstaging") + "\n",
} {
if strings.Count(s, old) != 1 {
t.Fatalf("the wrapper's path line %q is not there exactly once — this test cannot sandbox it", strings.TrimSpace(old))
}
s = strings.Replace(s, old, repl, 1)
}
// Safety: no assignment in the copy may still point at a real system path (PREV and PENDING are built from BIN and
// STAGING, so they follow).
for _, line := range strings.Split(s, "\n") {
if regexp.MustCompile(`^[A-Z_]+=/`).MatchString(line) && !strings.Contains(line, root) {
t.Fatalf("the sandboxed wrapper still holds a real path: %q", line)
}
}
script := filepath.Join(root, "wrapper.sh")
if err := os.WriteFile(script, []byte(s), 0o755); err != nil {
t.Fatal(err)
}
stubLog := filepath.Join(root, "stub.log")
for _, n := range []string{"systemctl", "logger", "chown", "systemd-run"} {
body := "#!/bin/sh\necho \"" + n + " $*\" >> '" + stubLog + "'\nexit 0\n"
if err := os.WriteFile(filepath.Join(stubs, n), []byte(body), 0o755); err != nil {
t.Fatal(err)
}
}
run := func(verb string) (string, error) {
cmd := exec.Command(sh, script, verb)
cmd.Env = []string{"PATH=" + stubs + ":/usr/bin:/bin"}
out, err := cmd.CombinedOutput()
return string(out), err
}
// applied is what the wrapper's `apply` leaves behind (it needs root, so the test writes the same four things).
applied := func(oldV, newV string) {
cur, err := os.ReadFile(live)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(live+".prev", cur, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(live, []byte("binary "+newV), 0o755); err != nil {
t.Fatal(err)
}
writePending(t, stateDir, PendingMarker{OldVersion: oldV, NewVersion: newV, SHA256: "ab", AppliedAt: "t"})
}
if err := os.WriteFile(live, []byte("binary 0.169.0"), 0o755); err != nil {
t.Fatal(err)
}
prev := "0.169.0"
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1"} {
stage(t, stateDir, v)
applied(prev, v)
m := newMgr(t, stateDir, v, runnerFunc(func(args ...string) ([]byte, []byte, error) {
out, err := run(args[0])
return []byte(out), nil, err
}), nil)
m.MaybeCommit(context.Background()) // the REAL commit verb, then the prune
if got := folder(t, stateDir); len(got) != 1 || got[0] != "felhom-agent-"+v {
t.Fatalf("after the good update to %s the folder holds %v", v, got)
}
prev = v
}
// Update N+1 is applied and goes bad: systemd's OnFailure runs the wrapper's rollback.
stage(t, stateDir, "0.173.0")
applied("0.172.1", "0.173.0")
out, err := run("rollback")
if err != nil {
t.Fatalf("the real rollback verb failed after the prunes: %v\n%s", err, out)
}
if b, _ := os.ReadFile(live); string(b) != "binary 0.172.1" {
t.Fatalf("after the rollback the live binary is %q, want the previous version's", b)
}
if p, _ := readPending(stateDir); p != nil {
t.Fatal("the rollback left the pending marker")
}
if b, _ := os.ReadFile(stubLog); !bytes.Contains(b, []byte("systemctl restart felhom-agent.service")) {
t.Fatalf("the rollback did not restart the agent; stub log:\n%s", b)
}
// The rolled-back agent starts: no marker, so no commit and no prune — the bad version's binary stays to be read.
newMgr(t, stateDir, "0.172.1", &fakeWrapper{}, nil).MaybeCommit(context.Background())
if got := folder(t, stateDir); strings.Join(got, " ") != "felhom-agent-0.172.1 felhom-agent-0.173.0" {
t.Fatalf("after the rollback the folder holds %v, want the running version and the one that failed", got)
}
}
// runnerFunc adapts a function to WrapperRunner; it gets the verb arguments only.
type runnerFunc func(args ...string) ([]byte, []byte, error)
func (f runnerFunc) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
return f(args...)
}
+3 -3
View File
@@ -24,10 +24,10 @@
"readers": [
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
],
"recorded": "2026-10-29",
"recorded_by": "CC 2026-10-29: day0_agent 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"recorded": "2026-11-03",
"recorded_by": "CC 2026-11-07: day0_agent 0.172.1 (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"pinned": {
"hub_min_agent": "0.158.1",
"day0_agent": "0.170.0"
"day0_agent": "0.172.1"
}
}