selfupdate: the staging folder keeps what it needs and no more (R-1368 part 2) — not released

Each good update left its staged binary in <state dir>/selfupdate for ever (49 files / 740 MB on demo-hp). After the wrapper's commit returned success, MaybeCommit now removes every staged binary but the running version's. Nothing is removed while a pending marker exists or cannot be read, at a plain start, or after a rollback. Only a regular file with the exact name Execute writes is removed. The rollback reads pending.json and felhom-agent.prev, no staged binary: pinned by a test on the wrapper's text and by a test that runs the wrapper's real commit and rollback verbs in a sandbox. Red seen three times. No version bump, no build, no vouch. /usr/local/bin/felhom-agent.bak-* is not reached (root-owned; R-1368 part 4). Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
v0.172.1 vouched and delivered to the three test boxes (agent + bundle, 77/77); pinned.day0_agent = 0.172.1 (D50)
2026-11-10 14:10:00 +01:00 · 2026-11-07 23:29:24 +01:00 · 2026-11-07 22:43:34 +01:00 · 2026-11-07 22:41:05 +01:00 · 2026-11-03 10:29:51 +01:00 · 2026-11-03 09:08:07 +01:00
17 changed files with 1023 additions and 78 deletions
@@ -1,4 +1,80 @@
-## Unreleased — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list
+## Unreleased (2026-11-10) — R-1368 part 2: the self-update folder keeps what it needs and no more — COMMITTED, NOT RELEASED (no version, no build, no vouch; it waits for a release session)
 - **The defect (measured 2026-11-10, space list place H12).** `Execute` downloads each new agent to
  `<state dir>/selfupdate/felhom-agent-<version>` and removed that file only when the download or its checksum failed.
  A good update left it for ever: 49 / 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1.
 - **What the folder is needed for (read first: `configs/felhom-selfupdate-guarded`).** The root wrapper reads the staged
  file once, copies it to its own root-owned folder and flips from that copy. The rollback needs `pending.json` (in the
  folder) and `/usr/local/bin/felhom-agent.prev` (outside it); the commit needs `pending.json`. No staged binary.
 - **The fix (`internal/selfupdate/prune.go`).** After the wrapper's `commit` returned success, `MaybeCommit` removes
  every staged binary but the running version's. Nothing is removed while a pending marker exists or cannot be read,
  at a plain start, after a rollback (the failed version's binary stays to be read), or when the running version is
  not a version this code can name a file for. Only a regular file with exactly the name `Execute` writes is removed,
  one `os.Remove` per name; a failed removal is logged and never fails the commit. A download and a prune take turns
  (`stagingMu`).
 - **Tests `TestPrune_*` (6).** After six updates the folder holds one binary and every bystander (unknown names, a
  folder and a symlink that carry a staged name); a pending or unreadable marker refuses; only a healthy commit prunes;
  the name rule follows `versionRe`; the wrapper's `commit` and `rollback` blocks name no staged binary; and the
  wrapper's REAL `commit` and `rollback` verbs, run from a sandboxed copy, still put the previous binary back after
  four pruned updates. **Red seen three times:** no prune call; no pending refusal; no regular-file check.
 - **Not reached.** `/usr/local/bin/felhom-agent.bak-*` (place H13): root-owned, in a folder the agent cannot write, made
  by a person. R-1368 part 4 stays open. A box gets the clean-up at its first good update AFTER the one that brings
  this code; nothing was run on a box.
 ## v0.172.1 (2026-11-07) — proxmox-grub joins the boot-chain lane: the nightly pve step stops being refused R6 — RELEASED and VOUCHED 2026-11-07 (release-agent.sh from `39db030`; agent sha256 505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8, bundle 3e7acaee4500893bfaf567b1d427790ef89bf41f08cb6a1419ee528aeb40ddc3)
 - **The defect (measured on demo-felhom, 2026-11-05, 11-06, 11-07).** Proxmox published `proxmox-grub` 2.12-9+pmx4. It
  depends on `grub-efi-amd64` of the same version. `HOST_SLOW_RE` matches `grub` only at the START of a name, so
  `proxmox-grub` landed in the ring-0 pve plan; the simulation then pulled `grub-efi-amd64`, which is not in the plan,
  and `felhom-os-apply` refused the WHOLE pve step (R6 „the plan would touch grub-efi-amd64") every night — 34 Proxmox
  userspace upgrades held with it, and an `os_update_failed` error mail each morning. The same class as the 2026-10-09
  `proxmox-secure-boot-support` fix. Read-only proof on the box: `apt-get -s install --only-upgrade proxmox-grub` →
  `Inst grub-efi-amd64 …pmx4`.
 - **The fix.** `proxmox-grub` joins `HOST_SLOW_RE`: GRUB's meta-package now waits with the rest of the boot chain (the
  slow lane, the operator's signed step), and the pve step runs without it.
 - **Test** `test_ring0_pending_pve_leaves_proxmox_grub_out`; red seen before the fix (proxmox-grub entered the plan), green
  after; the wrapper suite 149 of 149.
 - **Delivered 2026-11-07 (operator, signed jobs):** demo-felhom, demo-hp, Tester 1 — `agent_update` committed
  (23:05/23:13/23:13 CET), then the bundle: `BUNDLE DONE agent=0.172.1 written=1 same=30 self-check=ok`, capability probe
  77/77 on each. `pinned.day0_agent` = 0.172.1 (D50).
 - **Delivery.** `configs/` changed, so a box needs BOTH signed jobs: `agent_update` 0.172.1, then `agent_config_update`
  0.172.1 with the bundle sha. Not vouched by CC. Bundle vs 0.170.0 (what the boxes run): no path added or removed, only
  `felhom-os-apply` changed — so no step bundle is needed (R-880). Both downloads re-checked by sha.
 ## v0.172.0 (2026-11-03) — R-1302: the first grant of a tier is no loss — RELEASED and VOUCHED 10:18:10 CET (release-agent.sh from a clean clone of `ecbb0cc`; agent sha256 2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7, config bundle cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce; configs/ unchanged since 0.171.0, so the signed delivery was agent_update alone)
 - **R-1302 — the first grant of a tier is no loss.** A fresh install arms the whole-guest tier before its storage exists, so the
  first probe found no grant, repaired it and logged ERROR „GRANT WAS MISSING … investigate the loss (R-190)" plus a DEGRADED
  capability, on every new box (VM 9375, 2026-10-26). The agent now remembers which tiers have EVER held their grant
  (`<state dir>/store-grant-seen.json`, written by a healthy probe). A first grant logs INFO „granted" and reports healthy; a
  grant that existed and vanished keeps R-190's ERROR and its one degraded cycle. A doubt keeps the ERROR: an unreadable or
  corrupt file, or no file path. A box updated to this agent records its tiers at the first healthy probe, so a loss in the
  one probe before that would read as a first grant. Tests `TestR1302_*`; red seen with the first-grant branch removed and on
  three more mutants.
 ## v0.171.0 (2026-10-30) — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED — RELEASED (release-agent.sh from a clean clone of `bd18347`; agent sha256 c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5, config bundle 95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b; binary-only signed agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-30 10:47:36 CET; `pinned.day0_agent` = 0.171.0). Live on Tester 1: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the restart 09:31:22 → healthy 09:31:24; the hub mailed `controller_unresponsive` and `controller_unresponsive_restart` (hub 0.178.0), not `controller_restarted_by_agent`
 - **The measurement (operator brief 2026-10-29 Part 2, read 2026-10-30).** Docker's own health record of the controller
  on Tester 1 and demo-hp, every probe from 2026-10-29 11:12 to 2026-10-30 07:00 UTC: 4 733 probes through the night
  chain (demo-hp's PBS vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test, two
  controller updates). No failed probe, no 5-s timeout; slowest 303 ms (demo-hp, during the PBS vzdump). The only
  „unhealthy" was the controller stopped by hand for a proof. A busy night never came near ONE failed probe.
 - **So the alarm becomes a restart.** At `controllerUnresponsiveConfirm` = 6 unhealthy sweeps the alarm is raised, as
  before. At `controllerUnresponsiveRestart` = 10 (2 minutes later; ~6.5 minutes after the first failed probe) the agent
  restarts the controller's bootstrap unit — unless the controller's busy file is fresh (a backup, a restore, an
  update, an export) or cannot be read, or the brake refuses (3 restarts in 15 minutes → the crash-loop pause, which
  the hub mails). A swap, a parked guest, a locked guest and a vzdump keep resetting the count, as before. A refusal
  keeps the alarm and is logged once per reason. After a restart the count starts again from zero.
 - **The stanza** carries `unresponsive_restart_after_s` (120: the hub's alarm text says when the restart comes) and
  `unresponsive_restart_at` (equals `last_restart_at` for that act; the hub mails `controller_unresponsive_restart`).
 - **Refactor, no behaviour change:** the not-running restart's brake and its record are now `supervisorBrakeAllows`
  and `recordSupervisorRestart`, shared by both restarts. The busy-file stale log line no longer says „operator reboot".
 - Tests `TestSupervisor_UnresponsiveIsAlarmedThenRestarted`, `…Guards` (now also: never restarted),
  `…RestartRefusals` (busy fresh, busy unreadable, the brake, a crash-loop pause), `…StaleBusyFileDoesNotShield` —
  red first (build failure, then 3 behaviour failures); 7 mutants, each killed (one survivor — the count reset after a
  restart — found and pinned).
 ## v0.170.0 (2026-10-29) — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list — RELEASED (release-agent.sh from a clean clone of `ca15a31`; agent sha256 da8d12a05ef607c338b9ba307e7a81b18253a3188721bbd8421ba36658a22404, config bundle 03f2ff6442fc3ca600b1f25f26767ce4da466a770a258f3e362cf95f6e76b543; signed agent_config_update + agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-29 18:57:48 CET; `pinned.day0_agent` = 0.170.0)
 - **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
  now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
@@ -1,5 +1,29 @@
 # CONTEXT — felhom-agent working state
 > **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: the self-update folder cleans itself (R-1368 part 2).** After a healthy `commit` the agent removes every staged binary but the running version's (`internal/selfupdate/prune.go`; tests `TestPrune_*`, the wrapper's real `rollback` verb run in a sandbox). No version bump, no build, no vouch: it rides the next agent release. A box cleans its folder at the first good update AFTER that release. Not reached: `/usr/local/bin/felhom-agent.bak-*` (root-owned, R-1368 part 4). The old-kernel clean-up is a design only (felhom.eu `11-os-updates.md` §5.11.1, proven by hand on demo-hp); nothing of it is in this repo.
 > **2026-11-07 (attended) — v0.172.1 RELEASED and VOUCHED; on the three test boxes (agent + bundle, 77/77).** `proxmox-grub` joins `HOST_SLOW_RE` (the N100's ring-0 pve step was refused R6 three nights: proxmox-grub pmx4 pulls grub-efi-amd64). sha `505a0594…`, bundle `3e7acaee…` (only `felhom-os-apply` changed vs 0.170.0, no step bundle). Done: vouch, both signed jobs on the three boxes, `pinned.day0_agent` = 0.172.1. Next: the N100's pve step on its next night (positive line: `os-apply: DONE` for `layer=pve:9201`, no R6).
 > **2026-11-03 (day, attended) — v0.172.0 RELEASED and VOUCHED (10:18:10 CET), R-1302.** `2c0222cd…`, bundle `cc34b0bc…`, `pinned.day0_agent` = 0.172.0; on the three test boxes by signed `agent_update`. The appliance path pre-positions the grant, so a plain first start logs neither line; staged live both ways (felhom.eu `audits/release-2026-11-03/part3/`).
 >
 > **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
 > but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
 > alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
 > refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
 > (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
 > carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
 > `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
 > updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
 > `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
 > **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
 > `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
 > sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
 > (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
 > `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
 > (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
 > **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
 > **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
 > from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
 > `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
@@ -1,29 +1,23 @@
-# REPORT — v0.169.0 released and vouched: „Restart controller" from the hub (2026-10-29, attended)
+# REPORT — the self-update folder cleans itself (R-1368 part 2) — committed, NOT released (2026-11-10, operator brief)
-**Plain words first.** The operator can now restart a box's controller from the hub's host page. The agent does it, so it
+**Plain words first.** Each agent update left its downloaded file on the box for ever: 49 files and 740 MB on demo-hp.
-works when the controller is stopped or stuck. It uses the supervisor's own guards and brake: 3 restarts in 15 minutes,
+The agent now removes the old files after an update that ended well. It keeps the file of the version that runs.
-then presses are refused for 30 minutes. A controller that runs but does not answer now raises an operator alarm
+This is a commit only. No version, no build, no vouch. No box has it. It waits for the next agent release.
 (`controller_unresponsive`); it is not restarted by itself yet — that waits for the night's measurement (R-1344).
-## Commits
+The full report of the run is in `felhom.eu`: `REPORT-endings-kernels-fixes-2026-11-10.md`.
-| Commit | CI (job id) | What |
+| | |
-|---|---|---|
+|---|---|
-| `637c92c` | 2504, 2507 green | The agent's closed list: `restart_controller <vmid>` on the host-report reply (`ControlEnvelope.OperatorActions`), the result on the next host report (`OperatorActionResults`). Guards: provisioned, running, parked, swap, lock, vzdump. The brake counts operator restarts with the supervisor's (`opBrakeUntil`). One audit line, one result per id, re-sent until the hub records it; a report is asked for at once. R-1344: 6 unhealthy sweeps → `unresponsive_since` in the stanza. Tests in `internal/localapi/opactions_test.go`, `supervisor_unresponsive_test.go`, `opactions_wiring_test.go`, each seen failing by its mutation. |
+| Change | `internal/selfupdate/prune.go` (new); the call in `MaybeCommit` (`commit.go`); one lock in `Execute` (`executor.go`) |
-| `db25eb0` | 2513 green | v0.169.0 CHANGELOG; `pinned.day0_agent` = 0.169.0 (D50 pin), after the vouch. |
+| Rule | After the wrapper's `commit` returned success: remove every staged binary but the running version's |
 | Never removed | anything while `pending.json` exists or cannot be read; anything at a plain start or after a rollback; `pending.json`; a folder; a symlink; a name the code does not know |
 | Rollback | Needs `pending.json` and `/usr/local/bin/felhom-agent.prev`. It reads no staged binary (`configs/felhom-selfupdate-guarded`, read first) |
 | Tests | `TestPrune_*`, 6 tests. One runs the wrapper's real `commit` and `rollback` verbs from a sandboxed copy |
 | Red seen | 3 times: no prune call; no pending refusal; no regular-file check |
 | Gates | `python3 scripts/agent_gates.py --fast`: all OK |
 | Not reached | `/usr/local/bin/felhom-agent.bak-*`: root-owned, made by a person. R-1368 part 4 stays open |
 | Not proven | On a box. A box cleans its folder at the first good update AFTER the release that brings this code |
 | Machines | None touched by this change |
-## Release
+**Next.** A release session builds and vouches it. After two updates on a test box, read the folder: it must hold one
-
+binary.
 - `release-agent.sh 0.169.0` from a clean clone: sha256 `fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61`,
  bundle `3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b`, tag `v0.169.0`, verified by download.
 - Signed `agent_update` to the three test boxes; each committed.
 - Vouched by the operator 14:06:09 CET (the page answered 504; the save went through — R-1346).
 ## Live proofs (Tester 1, through the hub endpoint the button posts, CC's `cc-hosts` token)
 - 13:01–13:02 CET on rc builds: press 1 → the controller answered after 9 s; press 2 with the controller stopped by hand →
  restarted 5 s after the stop; press 3 done; press 4 refused („3 restarts in 15 minutes").
 - 14:49 CET on the release builds (agent 0.169.0, controller 0.348.0, hub 0.176.1): one press → „done — restarted; the
  controller answered after 9 s", on the hub 16 s after the press; the box showed the controller „Up 12 seconds".
 Evidence: felhom.eu `documentation/audits/release-2026-10-29b/` (`p1-restart/`, `final/`).
@@ -35,7 +35,7 @@
 | `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
 | `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
 | `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
-| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
+| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()`. **The staging folder cleans itself** (`prune.go`, R-1368): after a healthy `commit` only the running version's staged binary stays; a new file kept in `<state dir>/selfupdate/` must NOT be named `felhom-agent-<version>` or the prune removes it |
 | `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
 | `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
 | `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
@@ -88,8 +88,8 @@
 | `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
 | `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
 | `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
-| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
+| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause (`supervisorBrakeAllows` — every supervisor restart passes it; `recordSupervisorRestart` puts one on the record + R-539's slow counter); R-1344 (2026-10-30): a RUNNING controller whose Docker health reads „unhealthy" is alarmed at 6 sweeps and restarted at 10 (`restartUnresponsive`: the controller's busy file, then the brake); record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
-| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller` only, `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
+| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller`, `reboot_guest` since D65 — `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
 | `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
 ### Proxmox client / hub / PBS / provisioning
@@ -521,6 +521,66 @@ type storeGrantRepairer struct {
 	mu       sync.Mutex
 	last     map[string]time.Time // target id → last ATTEMPT (success or failure)
 	repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
 	// seenPath (R-1302) is the file that remembers which tiers have EVER held the grant: only those can LOSE it. "" = no
 	// memory, so every missing grant reads as a loss — the old, loud behaviour.
 	seenPath string
 }
 // everHeld reports whether this tier's grant was ever seen on this box (R-1302). The direction of a doubt is „yes": an
 // unreadable or corrupt file keeps the loss ERROR, so a real loss is never quietened by a broken record.
 func (r *storeGrantRepairer) everHeld(target string) bool {
 	if r == nil || r.seenPath == "" {
 		return true
 	}
 	r.mu.Lock()
 	defer r.mu.Unlock()
 	seen, err := readGrantSeen(r.seenPath)
 	return err != nil || seen[target]
 }
 // noteHeld records that this tier holds the grant. A write that fails is logged and costs one false „first grant" at most
 // after a restart — the old file, or none, stands.
 func (r *storeGrantRepairer) noteHeld(target string) {
 	if r == nil || r.seenPath == "" {
 		return
 	}
 	r.mu.Lock()
 	defer r.mu.Unlock()
 	seen, err := readGrantSeen(r.seenPath)
 	if err != nil {
 		seen = map[string]bool{} // a corrupt file is replaced by what is true now
 	}
 	if seen[target] {
 		return
 	}
 	seen[target] = true
 	b, _ := json.Marshal(seen)
 	tmp := r.seenPath + ".tmp"
 	if werr := os.WriteFile(tmp, b, 0o600); werr == nil {
 		werr = os.Rename(tmp, r.seenPath)
 		if werr == nil {
 			return
 		}
 		err = werr
 	} else {
 		err = werr
 	}
 	repairLogger(r).Warn("store-grant: could not record that this tier holds its grant (R-1302)", "target", target, "err", err)
 }
 func readGrantSeen(path string) (map[string]bool, error) {
 	b, err := os.ReadFile(path)
 	if os.IsNotExist(err) {
 		return map[string]bool{}, nil
 	}
 	if err != nil {
 		return nil, err
 	}
 	seen := map[string]bool{}
 	if err := json.Unmarshal(b, &seen); err != nil {
 		return nil, err
 	}
 	return seen, nil
 }
 // noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
@@ -631,6 +691,7 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
 		// host-report has certainly carried it. Without this latch the repairing probe may be a
 		// log-only one and the hub never learns anything happened (measured live, see the window's
 		// comment).
 		repair.noteHeld(targetID) // R-1302: a tier that holds its grant can lose it from now on
 		return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
 	}
@@ -679,8 +740,25 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
 	// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
 	// change, no hub change, no new event type. The `Feature` text carries the explanation because
 	// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
 	return storeGrantAfterRepair(targetID, critical, repair, s)
 }
 // storeGrantAfterRepair is the verdict once a repair is confirmed (split out for the tests). R-1302: the FIRST grant of a
 // tier that never held one is no loss — on a fresh install the whole-guest tier is armed before its storage exists, so
 // every new box logged „investigate the loss" and a DEGRADED capability at its first start, which teaches the reader to
 // skip the line. That grant logs INFO and reports healthy. A tier seen holding its grant, or one this box cannot tell
 // about, keeps R-190's ERROR and its one degraded cycle. Pinned by TestR1302_*.
 func storeGrantAfterRepair(targetID string, critical bool, repair *storeGrantRepairer, healthy capability.Status) capability.Status {
 	if !repair.everHeld(targetID) {
 		repair.noteHeld(targetID)
 		repairLogger(repair).Info("store-grant: granted — the first grant of this tier, nothing was lost (R-1302)",
 			"target", targetID, "privilege", storeGrantRequiredPriv,
 			"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
 		healthy.Status, healthy.Reason = capability.StatusOK, ""
 		return healthy
 	}
 	repair.noteRepaired(targetID, time.Now())
-	s = storeGrantRepairedVerdict(targetID, critical)
+	s := storeGrantRepairedVerdict(targetID, critical)
 	repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
 		"target", targetID, "privilege", storeGrantRequiredPriv,
 		"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
@@ -853,8 +931,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
 	// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
 	// invoked through the privileged path, which prepends sudo itself.
 	grantRepairer := &storeGrantRepairer{
-		run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
+		run:      (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
-		log: logger,
+		log:      logger,
 		seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json"), // R-1302
 	}
 	// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
 	// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
@@ -1,11 +1,14 @@
 package main
 import (
 	"bytes"
 	"context"
 	"errors"
 	"go/ast"
 	"io"
 	"log/slog"
 	"os"
 	"path/filepath"
 	"strings"
 	"testing"
 	"time"
@@ -415,3 +418,101 @@ func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
 			"be missed entirely", storeGrantRepairReportWindow)
 	}
 }
 // ── R-1302 — the first grant of a tier is no loss ──────────────────────────────────────
 //
 // A fresh install arms the whole-guest tier before its storage exists, so its first probe found no grant, repaired it and
 // logged „GRANT WAS MISSING … investigate the loss" with a DEGRADED capability — on every new box (VM 9375, 2026-10-26).
 //
 // RED-PROOF (2026-11-02): seen failing before storeGrantAfterRepair existed (build), and with its first-grant branch removed
 // TestR1302_TheFirstGrantIsNoLoss fails on „a first grant must not report a loss".
 func r1302Repairer(t *testing.T, path string) (*storeGrantRepairer, *bytes.Buffer) {
 	t.Helper()
 	var buf bytes.Buffer
 	return &storeGrantRepairer{run: (&fakeRepairRunner{}).Run, seenPath: path,
 		log: slog.New(slog.NewTextHandler(&buf, nil))}, &buf
 }
 func TestR1302_TheFirstGrantIsNoLoss(t *testing.T) {
 	path := filepath.Join(t.TempDir(), "store-grant-seen.json")
 	r, buf := r1302Repairer(t, path)
 	healthy := capability.Status{Name: "pve:store-grant:felhom-pbs", Status: capability.StatusDegraded, Reason: "missing", Critical: true}
 	s := storeGrantAfterRepair("felhom-pbs", true, r, healthy)
 	if s.Status != capability.StatusOK || s.Reason != "" {
 		t.Fatalf("a first grant must not report a loss: status %q reason %q", s.Status, s.Reason)
 	}
 	got := buf.String()
 	if strings.Contains(got, "investigate the loss") || strings.Contains(got, "level=ERROR") {
 		t.Fatalf("a first grant must not log the loss ERROR: %s", got)
 	}
 	if !strings.Contains(got, "level=INFO") || !strings.Contains(got, "granted") || !strings.Contains(got, "felhom-pbs") {
 		t.Fatalf("a first grant says so at INFO: %s", got)
 	}
 	if r.recentlyRepaired("felhom-pbs", time.Now()) {
 		t.Fatal("a first grant must not arm the loss report's latch")
 	}
 }
 func TestR1302_AGrantThatExistedAndVanishedKeepsTheError(t *testing.T) {
 	path := filepath.Join(t.TempDir(), "store-grant-seen.json")
 	first, _ := r1302Repairer(t, path)
 	first.noteHeld("felhom-pbs") // a healthy probe, in an earlier run of the agent
 	r, buf := r1302Repairer(t, path) // the agent restarted: the memory is the file, not the process
 	s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
 	if s.Status != capability.StatusDegraded || !strings.Contains(s.Feature, "R-190") {
 		t.Fatalf("a grant that existed and vanished must keep its degraded cycle; got %q / %q", s.Status, s.Feature)
 	}
 	if got := buf.String(); !strings.Contains(got, "level=ERROR") || !strings.Contains(got, "investigate the loss") {
 		t.Fatalf("a grant that existed and vanished must keep its ERROR: %s", got)
 	}
 	if !r.recentlyRepaired("felhom-pbs", time.Now()) {
 		t.Fatal("the loss must arm the report's latch, or the hub may never hear it")
 	}
 }
 // After a first grant the tier counts as held: the NEXT disappearance is a loss.
 func TestR1302_AfterTheFirstGrantALossIsALoss(t *testing.T) {
 	path := filepath.Join(t.TempDir(), "store-grant-seen.json")
 	r, _ := r1302Repairer(t, path)
 	storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
 	s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
 	if s.Status != capability.StatusDegraded {
 		t.Fatalf("the second disappearance is a loss; got %q", s.Status)
 	}
 }
 // The direction of a doubt: a broken record, a repairer with no record, or no repairer at all keep the loud way.
 func TestR1302_ADoubtKeepsTheError(t *testing.T) {
 	path := filepath.Join(t.TempDir(), "store-grant-seen.json")
 	if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
 		t.Fatal(err)
 	}
 	r, _ := r1302Repairer(t, path)
 	if s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{}); s.Status != capability.StatusDegraded {
 		t.Fatalf("a corrupt record must not quieten a loss; got %q", s.Status)
 	}
 	noMemory, _ := r1302Repairer(t, "")
 	if s := storeGrantAfterRepair("felhom-pbs", true, noMemory, capability.Status{}); s.Status != capability.StatusDegraded {
 		t.Fatalf("no record must not quieten a loss; got %q", s.Status)
 	}
 	if s := storeGrantAfterRepair("felhom-pbs", true, nil, capability.Status{}); s.Status != capability.StatusDegraded {
 		t.Fatalf("no repairer must not quieten a loss; got %q", s.Status)
 	}
 }
 // The wiring, read from the source: production names the file, and a healthy probe records the tier.
 func TestR1302_TheMemoryIsWired(t *testing.T) {
 	b, err := os.ReadFile("main.go")
 	if err != nil {
 		t.Fatal(err)
 	}
 	src := string(b)
 	if !strings.Contains(src, `seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json")`) {
 		t.Fatal("main.go must give the repairer its file, or every first grant reads as a loss")
 	}
 	start := strings.Index(src, "func storeGrantStatus(")
 	if start < 0 || !strings.Contains(src[start:start+3000], "repair.noteHeld(targetID)") {
 		t.Fatal("a healthy probe must record the tier, or a real loss would read as a first grant")
 	}
 }
@@ -97,9 +97,12 @@ OOMCHECK_SETTLE = 2
 # so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
 # shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
 # (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
 # proxmox-grub is GRUB's Proxmox meta-package (it depends on grub-efi-amd64 of the same version); the `grub` above is
 # anchored at the start, so it missed it. Measured 2026-11-05..07 on demo-felhom: its upgrade pulled grub-efi-amd64 and
 # the whole pve step was refused R6 every night. Pinned by test_ring0_pending_pve_leaves_proxmox_grub_out.
 HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
                          r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
-                          r"proxmox-secure-boot-support)")
+                          r"proxmox-secure-boot-support|proxmox-grub)")
 # restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
 # processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
 # 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
@@ -1494,6 +1494,24 @@ class PVELane(unittest.TestCase):
        self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
        self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
    # 2026-11-05..07, demo-felhom (EFI): proxmox-grub 2.12-9+pmx2 -> pmx4 depends on grub-efi-amd64 of the same version;
    # HOST_SLOW_RE's `grub` is anchored at the start, so proxmox-grub fell into the pve plan, the simulation pulled
    # grub-efi-amd64 and the whole pve step was refused R6 every night. It is a boot-chain package: left out.
    # RED-PROOF: drop proxmox-grub from HOST_SLOW_RE -> it is in the plan -> this test fails.
    def test_ring0_pending_pve_leaves_proxmox_grub_out(self):
        f = pve_fake(ring0=True)
        f.plan["select"], f.plan["packages"] = "pending-pve", []
        f.installed.update({"proxmox-grub": "2.12-9+pmx2"})
        f.pending_sim = [
            "Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
            "Inst proxmox-grub [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
            "Inst grub-efi-amd64 [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
        ]
        rc, rep = run(f)
        self.assertEqual(rc, 0, rep)
        self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
        self.assertEqual(f.installed["proxmox-grub"], "2.12-9+pmx2")
    def test_select_pending_pve_needs_the_pve_layer(self):
        f = Fake()
        f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
@@ -218,6 +218,12 @@ type ControllerSupervisorGuest struct {
 	// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
 	Unresponsive      bool   `json:"unresponsive"`
 	UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
 	// R-1344 restart (decided 2026-10-30 from the night's measurement): UnresponsiveRestartAfterS is how long after the
 	// alarm the agent restarts a controller that still does not answer (0 / absent = an older agent: alarm only — the
 	// hub's message says which). UnresponsiveRestartAt is the last such restart; the hub keys on it MOVING
 	// (controller_unresponsive_restart) and it equals LastRestartAt for that act.
 	UnresponsiveRestartAfterS int    `json:"unresponsive_restart_after_s,omitempty"`
 	UnresponsiveRestartAt     string `json:"unresponsive_restart_at,omitempty"` // RFC3339
 }
 // PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
@@ -3,6 +3,7 @@ package localapi
 import (
 	"context"
 	"encoding/json"
 	"fmt"
 	"os"
 	"path/filepath"
 	"sort"
@@ -87,10 +88,16 @@ const (
 	// R-1344 (operator brief 2026-10-29, Part 2) — a controller that RUNS but does not answer. Docker's own HEALTHCHECK
 	// (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) reads „unhealthy" only after 3 failed probes (~90 s);
 	// controllerUnresponsiveConfirm more sweeps (3 minutes) make it an episode: logged, carried by the stanza, mailed by
-	// the hub (controller_unresponsive). ALARM ONLY — it is NOT restarted for this until the measurement (R-1344) shows a
+	// the hub (controller_unresponsive). Not counted during a swap, a parked guest, a locked guest or a vzdump.
 	// busy night (backups, updates) never reads unhealthy that long. Not counted during a swap, a parked guest, a locked
 	// guest or a vzdump.
 	controllerUnresponsiveConfirm = 6
 	// controllerUnresponsiveRestart (decided 2026-10-30 from the measurement): at this many consecutive unhealthy sweeps
 	// (5 minutes after Docker's „unhealthy", ~6.5 minutes after the first failed probe; 2 minutes after the alarm) the
 	// agent restarts the controller — the not-running restart's guards and brake, plus the controller's busy file. The
 	// numbers: Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, 4 733 probes through the night chain (a
 	// PBS and a local vzdump, the DB dumps, tier-2, the off-site push, a restore test, two controller updates): no failed
 	// probe, slowest 303 ms against the 5 s timeout. A busy night never came near even ONE failed probe; ten sweeps is
 	// the margin. Pinned by TestSupervisor_UnresponsiveIsAlarmedThenRestarted.
 	controllerUnresponsiveRestart = 10
 )
 // controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
@@ -119,6 +126,10 @@ type controllerSupState struct {
 	unhealthySeen     int
 	unresponsive      bool
 	unresponsiveSince time.Time
 	// unresponsiveRestartAt: the last restart for not answering (kept; the hub keys on it MOVING). unrespRefusal: the
 	// last reason the restart was refused, so a refusal is logged once per change, not every sweep.
 	unresponsiveRestartAt time.Time
 	unrespRefusal         string
 }
 type controllerSupervisor struct {
@@ -354,9 +365,27 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
 		return false
 	}
-	// Backoff.
+	if !s.supervisorBrakeAllows(vmid, now) {
 		return false
 	}
 	reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
 	s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
 		"vmid", vmid, "status", status, "unit", bootstrapUnit)
 	if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
 		s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
 		reason += "; restart FAILED: " + err.Error()
 	}
 	s.recordSupervisorRestart(vmid, now, reason, nil)
 	return false
 }
 // supervisorBrakeAllows is the brake every supervisor restart passes (the not-running one and R-1344's): a crash-loop
 // pause in force refuses; controllerCrashloopMax restarts inside controllerCrashloopWindow start one (logged, carried by
 // the stanza, mailed by the hub as controller_crashloop) and refuse.
 func (s *Server) supervisorBrakeAllows(vmid int, now time.Time) bool {
 	s.ctrlSup.mu.Lock()
-	st = s.supState(vmid)
+	st := s.supState(vmid)
 	if !st.crashloopSince.IsZero() {
 		if now.Sub(st.crashloopSince) < controllerCrashloopPause {
 			s.ctrlSup.mu.Unlock()
@@ -379,22 +408,23 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
 		return false
 	}
 	s.ctrlSup.mu.Unlock()
 	return true
 }
-	reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
+// recordSupervisorRestart puts a restart the supervisor performed on the record: the brake's window, the totals, the
-	s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
+// reason, and R-539's slow counter (persisted; raised at most once per 24 hours — the hub mails on the raise MOVING).
-		"vmid", vmid, "status", status, "unit", bootstrapUnit)
+// mark, when set, runs under the same lock (R-1344 stamps its own record there).
-	if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
+func (s *Server) recordSupervisorRestart(vmid int, now time.Time, reason string, mark func(st *controllerSupState)) {
 		s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
 		reason += "; restart FAILED: " + err.Error()
 	}
 	s.ctrlSup.mu.Lock()
-	st = s.supState(vmid)
+	st := s.supState(vmid)
 	st.restarts = append(st.restarts, now)
 	st.restartsTotal++
 	st.lastRestartAt = now
 	st.lastReason = reason
 	st.notRunningSeen = 0
-	// R-539: the slow counter. Raise at most once per 24 hours — the hub mails on the raise MOVING.
+	if mark != nil {
 		mark(st)
 	}
 	st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now)
 	n24 := len(st.restarts24h)
 	raised := false
@@ -410,7 +440,6 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
 		s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop",
 			"vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax)
 	}
 	return false
 }
 // observeControllerHealth is R-1344's count for a RUNNING controller: Docker's health status, one read per sweep over
@@ -452,11 +481,60 @@ func (s *Server) observeControllerHealth(ctx context.Context, vmid int, parked b
 	if start {
 		st.unresponsive, st.unresponsiveSince = true, now
 	}
 	since := st.unresponsiveSince
 	s.ctrlSup.mu.Unlock()
 	if start {
-		s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; NOT restarting it (R-1344: alarm only until measured)",
+		s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; restarting it if it stays so",
-			"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String())
+			"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String(),
 			"restart_after_sweeps", controllerUnresponsiveRestart)
 	}
 	if seen >= controllerUnresponsiveRestart {
 		s.restartUnresponsive(ctx, vmid, seen, since, now)
 	}
 }
 // restartUnresponsive is R-1344's restart (decided 2026-10-30): the controller's busy file (fresh = a backup, a
 // restore, an update or an export is running; unreadable = cannot be ruled out) refuses, then the brake every
 // supervisor restart passes. The swap, the parked marker, the lock and the vzdump never reach here — they reset the
 // count. A refusal keeps the alarm and is logged once per reason; the next sweep asks again.
 func (s *Server) restartUnresponsive(ctx context.Context, vmid, seen int, since, now time.Time) {
 	refuse := func(why string) {
 		s.ctrlSup.mu.Lock()
 		st := s.supState(vmid)
 		changed := st.unrespRefusal != why
 		st.unrespRefusal = why
 		s.ctrlSup.mu.Unlock()
 		if changed {
 			s.logger.Warn("controller-supervisor: the controller does not answer, but the restart is refused — the alarm stays",
 				"vmid", vmid, "why", why, "unhealthy_sweeps", seen)
 		}
 	}
 	busy, what, err := s.controllerBusy(ctx, vmid)
 	if err != nil {
 		refuse("the controller's busy file cannot be read: " + err.Error())
 		return
 	}
 	if busy {
 		refuse("the controller is busy: " + what)
 		return
 	}
 	if !s.supervisorBrakeAllows(vmid, now) {
 		refuse("the restart brake")
 		return
 	}
 	reason := fmt.Sprintf("the controller ran but did not answer its health check for %d sweeps (since %s)",
 		seen, since.UTC().Format(time.RFC3339))
 	s.logger.Warn("controller-supervisor: the controller does not answer — restarting the bootstrap unit",
 		"vmid", vmid, "unhealthy_sweeps", seen, "since", since.UTC().Format(time.RFC3339), "unit", bootstrapUnit)
 	if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
 		s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
 		reason += "; restart FAILED: " + err.Error()
 	}
 	s.recordSupervisorRestart(vmid, now, reason, func(st *controllerSupState) {
 		st.unresponsiveRestartAt = now
 		st.unhealthySeen = 0
 		st.unrespRefusal = ""
 	})
 }
 func (s *Server) resetNotRunning(vmid int) {
@@ -517,6 +595,10 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
 		if !st.unresponsiveSince.IsZero() {
 			g.UnresponsiveSince = st.unresponsiveSince.UTC().Format(time.RFC3339)
 		}
 		g.UnresponsiveRestartAfterS = (controllerUnresponsiveRestart - controllerUnresponsiveConfirm) * int(controllerSupervisorInterval/time.Second)
 		if !st.unresponsiveRestartAt.IsZero() {
 			g.UnresponsiveRestartAt = st.unresponsiveRestartAt.UTC().Format(time.RFC3339)
 		}
 		out.Guests = append(out.Guests, g)
 	}
 	sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
@@ -137,7 +137,8 @@ func (s *Server) rebooter() interface {
 // controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
 // a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
-// (the caller refuses — it cannot prove no backup runs).
+// (the caller refuses — it cannot prove no backup runs). Two callers: the operator's reboot (D65) and R-1344's restart
 // of a controller that does not answer (controllersupervisor.go restartUnresponsive).
 func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
 	out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
 	if err != nil {
@@ -152,7 +153,7 @@ func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, er
 		return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
 	}
 	if s.clock().Sub(at) > controllerBusyFresh {
-		s.logger.Info("operator reboot: the controller's busy file is stale — the controller stopped writing it; not busy",
+		s.logger.Info("controller-busy: the controller's busy file is stale — the controller stopped writing it; not busy",
 			"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
 		return false, "", nil
 	}
@@ -2,19 +2,25 @@ package localapi
 import (
 	"context"
 	"errors"
 	"strings"
 	"testing"
 	"time"
 )
-// R-1344 (operator brief 2026-10-29, Part 2): a controller that RUNS but whose Docker health check reads „unhealthy" on
+// R-1344 (operator brief 2026-10-29, Part 2; decided 2026-10-30 from the night's measurement): a controller that RUNS
-// controllerUnresponsiveConfirm consecutive sweeps is reported unresponsive (the stanza; the hub raises
+// but whose Docker health check reads „unhealthy" on controllerUnresponsiveConfirm consecutive sweeps is reported
-// controller_unresponsive). ALARM ONLY: it is never restarted for it until the measurement licenses that.
+// unresponsive (the stanza; the hub raises controller_unresponsive). If it still reads „unhealthy" at
-// RED-PROOF: seen failing before the stanza carried it.
+// controllerUnresponsiveRestart sweeps, the agent restarts it — with the not-running restart's guards and brake, plus
 // the controller's own busy file. RED-PROOF: the restart tests seen failing before the restart existed.
-func unrespServer(t *testing.T, health string) (*Server, *opExec, *fakeGuestPowerCtl) {
+func unrespServer(t *testing.T, health string) (*Server, *rbExec, *fakeGuestPowerCtl, *supClock) {
 	t.Helper()
-	s, ex, ctl, _, _, _ := opServer(t, "running")
+	s, ex, ctl, clk, _, _ := opServer(t, "running")
 	ex.health = health
-	return s, ex, ctl
+	rb := &rbExec{opExec: ex}
 	s.guestExec = rb
 	return s, rb, ctl, clk
 }
 func sweep(s *Server, n int) {
@@ -23,8 +29,8 @@ func sweep(s *Server, n int) {
 	}
 }
-func TestSupervisor_UnresponsiveIsAlarmedNeverRestarted(t *testing.T) {
+func TestSupervisor_UnresponsiveIsAlarmedThenRestarted(t *testing.T) {
-	s, ex, _ := unrespServer(t, "unhealthy")
+	s, ex, _, _ := unrespServer(t, "unhealthy")
 	sweep(s, controllerUnresponsiveConfirm-1)
 	if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) != 1 || g[0].Unresponsive {
 		t.Fatalf("unresponsive before %d sweeps: %+v", controllerUnresponsiveConfirm, g)
@@ -34,41 +40,117 @@ func TestSupervisor_UnresponsiveIsAlarmedNeverRestarted(t *testing.T) {
 	if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveSince == "" {
 		t.Fatalf("after %d unhealthy sweeps: %+v", controllerUnresponsiveConfirm, g)
 	}
 	if g[0].UnresponsiveRestartAfterS != (controllerUnresponsiveRestart-controllerUnresponsiveConfirm)*int(controllerSupervisorInterval/time.Second) {
 		t.Fatalf("the stanza must tell the hub when the restart comes: %+v", g[0])
 	}
 	since := g[0].UnresponsiveSince
-	sweep(s, 10)
+	sweep(s, controllerUnresponsiveRestart-controllerUnresponsiveConfirm-1)
 	if n := ex.count(9201); n != 0 {
-		t.Fatalf("an unresponsive controller was RESTARTED (%d) — the restart waits for the measurement", n)
+		t.Fatalf("restarted after %d unhealthy sweeps — the restart waits for %d", controllerUnresponsiveRestart-1, controllerUnresponsiveRestart)
 	}
 	if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].UnresponsiveSince != since {
 		t.Fatal("unresponsive_since moved while it stayed unresponsive — the hub would mail again")
 	}
-	// it answers again
+	sweep(s, 1)
-	ex.health = "healthy"
+	if n := ex.count(9201); n != 1 {
 		t.Fatalf("after %d unhealthy sweeps: %d restarts, want 1", controllerUnresponsiveRestart, n)
 	}
 	g = s.ControllerSupervisorStatus(context.Background()).Guests
 	if g[0].UnresponsiveRestartAt == "" || g[0].UnresponsiveRestartAt != g[0].LastRestartAt || g[0].Restarts24h != 1 ||
 		!strings.Contains(g[0].LastReason, "health check") {
 		t.Fatalf("the restart is not on the record the hub reads: %+v", g[0])
 	}
 	// A restart that did not help (still „unhealthy" on the next sweep) is not repeated at once: the count starts again.
 	sweep(s, 1)
 	if n := ex.count(9201); n != 1 {
 		t.Fatalf("restarted again on the sweep after a restart: %d restarts", n)
 	}
 	// The restarted controller starts, then answers: the episode ends, its record stays.
 	ex.health = "starting"
 	sweep(s, 1)
 	if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].Unresponsive || g[0].UnresponsiveSince != since {
-		t.Fatalf("after it answered: %+v (unresponsive must clear; since stays as the record)", g[0])
+		t.Fatalf("after the restart: %+v (unresponsive must clear; since stays as the record)", g[0])
 	}
 	// A second episode needs the whole count again — never a restart on the first unhealthy read after one.
 	ex.health = "unhealthy"
 	sweep(s, controllerUnresponsiveRestart-1)
 	if n := ex.count(9201); n != 1 {
 		t.Fatalf("a second episode restarted early: %d restarts", n)
 	}
 }
-// Slow but answering (healthy / starting), a swap, a vzdump or a parked guest never counts.
+// Slow but answering (healthy / starting), a swap, a vzdump, a locked or a parked guest never counts — and so is never
 // restarted.
 func TestSupervisor_UnresponsiveGuards(t *testing.T) {
 	for _, tc := range []struct {
 		name  string
-		setup func(s *Server, ex *opExec, ctl *fakeGuestPowerCtl)
+		setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl)
 	}{
-		{"healthy", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
+		{"healthy", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
-		{"starting", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
+		{"starting", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
-		{"swap", func(s *Server, _ *opExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
+		{"swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
-		{"vzdump", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
+		{"vzdump", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
-		{"locked", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
+		{"locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
 	} {
 		t.Run(tc.name, func(t *testing.T) {
-			s, ex, ctl := unrespServer(t, "unhealthy")
+			s, ex, ctl, _ := unrespServer(t, "unhealthy")
 			tc.setup(s, ex, ctl)
-			sweep(s, controllerUnresponsiveConfirm+2)
+			sweep(s, controllerUnresponsiveRestart+2)
 			if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) == 1 && g[0].Unresponsive {
 				t.Fatalf("%s counted as unresponsive: %+v", tc.name, g[0])
 			}
 			if n := ex.count(9201); n != 0 {
 				t.Fatalf("%s: restarted %d times", tc.name, n)
 			}
 		})
 	}
 }
 // At the restart moment: the controller's busy file (fresh, or unreadable) and the brake refuse it; the alarm stays.
 // RED-PROOF: each seen failing by deleting its guard.
 func TestSupervisor_UnresponsiveRestartRefusals(t *testing.T) {
 	for _, tc := range []struct {
 		name      string
 		setup     func(s *Server, ex *rbExec, clk *supClock)
 		crashloop bool
 	}{
 		{"busy file fresh", func(_ *Server, ex *rbExec, clk *supClock) {
 			ex.busy = clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\na backup, a restore or an off-site run is running\n"
 		}, false},
 		{"busy file unreadable", func(_ *Server, ex *rbExec, _ *supClock) { ex.busyErr = errors.New("pct exec: exit status 255") }, false},
 		{"brake: 3 restarts in 15 minutes", func(s *Server, _ *rbExec, clk *supClock) {
 			st := s.supState(9201)
 			st.restarts = []time.Time{clk.t.Add(-10 * time.Minute), clk.t.Add(-5 * time.Minute), clk.t.Add(-time.Minute)}
 		}, true},
 		{"crash-loop pause in force", func(s *Server, _ *rbExec, clk *supClock) {
 			s.supState(9201).crashloopSince = clk.t.Add(-5 * time.Minute)
 		}, true},
 	} {
 		t.Run(tc.name, func(t *testing.T) {
 			s, ex, _, clk := unrespServer(t, "unhealthy")
 			tc.setup(s, ex, clk)
 			sweep(s, controllerUnresponsiveRestart+3)
 			if n := ex.count(9201); n != 0 {
 				t.Fatalf("%s: restarted %d times", tc.name, n)
 			}
 			g := s.ControllerSupervisorStatus(context.Background()).Guests
 			if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveRestartAt != "" {
 				t.Fatalf("%s: the alarm must stay and no restart be recorded: %+v", tc.name, g)
 			}
 			if tc.crashloop && !g[0].Crashloop {
 				t.Fatalf("%s: the brake did not raise the crash-loop: %+v", tc.name, g[0])
 			}
 		})
 	}
 }
 // A busy file the controller stopped writing (older than controllerBusyFresh) is not „busy" — a hung controller must
 // not shield itself with its last word.
 func TestSupervisor_UnresponsiveStaleBusyFileDoesNotShield(t *testing.T) {
 	s, ex, _, clk := unrespServer(t, "unhealthy")
 	ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\nan app update is running\n"
 	sweep(s, controllerUnresponsiveRestart)
 	if n := ex.count(9201); n != 1 {
 		t.Fatalf("a stale busy file blocked the restart: %d restarts", n)
 	}
 }
@@ -128,4 +128,11 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
 	if m.onCommitted != nil {
 		m.onCommitted(ctx)
 	}
 	// R-1368 part 2: the update ended healthy — the folder keeps the running version's binary and no other
 	// (prune.go). A refusal or a failed removal is logged and changes nothing about the commit.
 	if removed, err := pruneStaged(m.stateDir, m.runningVersion, m.logger); err != nil {
 		m.logger.Warn("selfupdate: old staged binaries NOT removed after the commit", "err", err)
 	} else if len(removed) > 0 {
 		m.logger.Info("selfupdate: old staged binaries removed after the commit", "count", len(removed), "kept", "felhom-agent-"+m.runningVersion)
 	}
 }
@@ -103,6 +103,9 @@ func (e *Executor) Execute(ctx context.Context, op string, params json.RawMessag
 	}
 	e.logger.Debug("agent_update: op invariants passed (semver + sha format)", "version", p.Version)
 	// The staged file is this call's from the download to the hand-off: a prune (prune.go) waits for it.
 	stagingMu.Lock()
 	defer stagingMu.Unlock()
 	dir := stagingDir(e.stateDir)
 	if err := os.MkdirAll(dir, 0o750); err != nil {
 		return fmt.Errorf("agent_update: staging dir: %w", err)
@@ -0,0 +1,96 @@
 package selfupdate
 import (
 	"fmt"
 	"log/slog"
 	"os"
 	"path/filepath"
 	"regexp"
 	"sort"
 	"sync"
 )
 // The self-update folder keeps what it needs and no more (R-1368 part 2; space list 2026-11-10, place H12).
 //
 // WHAT WAS WRONG. Execute downloads each new agent to <StateDir>/selfupdate/felhom-agent-<version> and removes that
 // file only when the download or its checksum fails. A good update left it there for ever: measured 2026-11-10, 49 /
 // 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1 — one for each version the box ever got.
 //
 // WHAT THE FOLDER IS NEEDED FOR. The root wrapper (felhom-os-apply, mode agent_update) reads the staged file ONCE,
 // copies it to its own root-owned folder and flips from that copy. After that hand-off nothing reads the staged file:
 //   - the rollback (felhom-selfupdate-guarded `rollback`) needs two files and neither is a staged binary:
 //     `pending.json` in this folder, and `/usr/local/bin/felhom-agent.prev` outside it;
 //   - the commit needs `pending.json` only.
 //
 // So a staged binary is dead weight as soon as its update ended. Pinned by TestPrune_RollbackReadsNoStagedBinary (the
 // wrapper's text) and TestPrune_RealRollbackStillWorksAfterNUpdates (the wrapper's rollback verb, run for real).
 //
 // THE RULE. After an update ended HEALTHY (the wrapper's `commit` returned success), remove every staged binary but
 // the one of the running version. Nothing is removed:
 //   - while a pending marker exists or cannot be read — an update is in flight, and that is the only time a rollback
 //     can fire;
 //   - after a rollback, or at a plain start — only the commit path calls this, so the binary of a version that failed
 //     stays for the operator to look at until the next good update;
 //   - that is not a regular file with exactly the name Execute gives a staged binary. `pending.json`, a folder, a
 //     symlink, and any name this code does not know are never touched.
 //
 // NOT REACHED BY THIS: `/usr/local/bin/felhom-agent.bak-*` (space list H13). Those files are root-owned, in a folder
 // the agent cannot write, and no code of the agent makes them (a person's copy before a hand deploy). R-1368 part 4.
 // stagedNameRe is the exact name Execute gives a staged binary: "felhom-agent-" + a version that versionRe accepts.
 // Kept in step with versionRe by TestPrune_NameShapeFollowsTheVersionRule.
 var stagedNameRe = regexp.MustCompile(`^felhom-agent-([0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?)$`)
 // stagingMu makes a download and a prune take turns. Without it a prune that runs while Execute downloads the NEXT
 // version would remove that half-written file (its name is a staged binary's and it is not the running version).
 var stagingMu sync.Mutex
 // pruneStaged removes the staged binaries the folder no longer needs and returns the names it removed, sorted. It
 // refuses (removes nothing, returns an error) while an update is in flight. A file it cannot remove is logged and
 // skipped: a failed clean-up must never fail the commit that came before it.
 func pruneStaged(stateDir, runningVersion string, logger *slog.Logger) ([]string, error) {
 	logger = orDefaultLogger(logger)
 	stagingMu.Lock()
 	defer stagingMu.Unlock()
 	if !versionRe.MatchString(runningVersion) {
 		return nil, fmt.Errorf("selfupdate prune: refusing — running version %q is not a version this code knows; it could not name the file to keep", runningVersion)
 	}
 	pm, err := readPending(stateDir)
 	if err != nil {
 		return nil, fmt.Errorf("selfupdate prune: refusing — the pending marker cannot be read: %w", err)
 	}
 	if pm != nil {
 		return nil, fmt.Errorf("selfupdate prune: refusing — an update to %s is pending (a rollback may still fire)", pm.NewVersion)
 	}
 	dir := stagingDir(stateDir)
 	entries, err := os.ReadDir(dir)
 	if err != nil {
 		if os.IsNotExist(err) {
 			return nil, nil
 		}
 		return nil, fmt.Errorf("selfupdate prune: read %s: %w", dir, err)
 	}
 	keep := "felhom-agent-" + runningVersion
 	var removed []string
 	for _, ent := range entries {
 		name := ent.Name()
 		if name == keep || !stagedNameRe.MatchString(name) {
 			continue // the running version's binary, pending.json, or a name this code does not know
 		}
 		if !ent.Type().IsRegular() {
 			logger.Warn("selfupdate prune: a staged-binary name that is not a regular file — left alone", "name", name, "type", ent.Type().String())
 			continue
 		}
 		// name came from ReadDir and matched stagedNameRe, so it holds no path separator: this is one exact file.
 		if err := os.Remove(filepath.Join(dir, name)); err != nil {
 			logger.Warn("selfupdate prune: cannot remove an old staged binary — left for the next good update", "name", name, "err", err)
 			continue
 		}
 		logger.Debug("selfupdate prune: removed an old staged binary", "name", name)
 		removed = append(removed, name)
 	}
 	sort.Strings(removed)
 	logger.Info("selfupdate prune: the self-update folder holds what it needs", "kept", keep, "removed", len(removed))
 	return removed, nil
 }
@@ -0,0 +1,373 @@
 package selfupdate
 import (
 	"bytes"
 	"context"
 	"fmt"
 	"os"
 	"os/exec"
 	"path/filepath"
 	"regexp"
 	"sort"
 	"strings"
 	"testing"
 )
 // R-1368 part 2 — the self-update folder keeps what it needs and no more (prune.go).
 //
 // RED-PROOFS (each observed 2026-11-10, then restored):
 //   - drop the pruneStaged call from MaybeCommit → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with
 //     "after update 1 (0.170.0) the folder holds [… felhom-agent-0.169.0 felhom-agent-0.170.0 …]", and
 //     TestPrune_RealRollbackStillWorksAfterNUpdates fails too;
 //   - drop the pending-marker refusal from pruneStaged → TestPrune_RefusesWhileAnUpdateIsPending fails with
 //     "removed [...] while an update was pending";
 //   - drop the IsRegular check → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with "a symlink with a
 //     staged-binary name was removed".
 // committingWrapper is the guarded wrapper's `commit` as far as this package can see it: it clears the pending marker.
 type committingWrapper struct {
 	stateDir string
 	calls    int
 }
 func (w *committingWrapper) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
 	if len(args) == 1 && args[0] == "commit" {
 		w.calls++
 		if err := os.Remove(pendingPath(w.stateDir)); err != nil {
 			return nil, []byte(err.Error()), err
 		}
 	}
 	return []byte("ok"), nil, nil
 }
 func stage(t *testing.T, stateDir, version string) {
 	t.Helper()
 	if err := os.MkdirAll(stagingDir(stateDir), 0o750); err != nil {
 		t.Fatal(err)
 	}
 	if err := os.WriteFile(filepath.Join(stagingDir(stateDir), "felhom-agent-"+version), []byte("binary "+version), 0o755); err != nil {
 		t.Fatal(err)
 	}
 }
 func folder(t *testing.T, stateDir string) []string {
 	t.Helper()
 	ents, err := os.ReadDir(stagingDir(stateDir))
 	if err != nil {
 		t.Fatal(err)
 	}
 	var out []string
 	for _, e := range ents {
 		out = append(out, e.Name())
 	}
 	sort.Strings(out)
 	return out
 }
 // The consequence: after N good updates the folder holds ONE staged binary — the running version's — and every file
 // that is not a staged binary is still there.
 func TestPrune_AfterNUpdatesTheFolderHoldsOneBinary(t *testing.T) {
 	stateDir := t.TempDir()
 	dir := stagingDir(stateDir)
 	if err := os.MkdirAll(dir, 0o750); err != nil {
 		t.Fatal(err)
 	}
 	// Bystanders: names this code does not know, and two things that carry a staged binary's NAME but are not files.
 	outside := filepath.Join(t.TempDir(), "precious")
 	if err := os.WriteFile(outside, []byte("not ours"), 0o644); err != nil {
 		t.Fatal(err)
 	}
 	for _, n := range []string{"notes.txt", "felhom-agent-0.1.0.partial", "felhom-agent.bak-0.139.0"} {
 		if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
 			t.Fatal(err)
 		}
 	}
 	if err := os.Mkdir(filepath.Join(dir, "felhom-agent-9.9.9"), 0o750); err != nil {
 		t.Fatal(err)
 	}
 	if err := os.Symlink(outside, filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
 		t.Fatal(err)
 	}
 	bystanders := []string{"felhom-agent-0.1.0.partial", "felhom-agent-8.8.8", "felhom-agent-9.9.9", "felhom-agent.bak-0.139.0", "notes.txt"}
 	versions := []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1", "0.173.0-rc.1", "0.173.0"}
 	prev := "0.169.0"
 	stage(t, stateDir, prev) // what an old agent left behind
 	for i, v := range versions {
 		stage(t, stateDir, v)                                                                                   // Execute downloaded it
 		writePending(t, stateDir, PendingMarker{OldVersion: prev, NewVersion: v, SHA256: "ab", AppliedAt: "t"}) // the wrapper flipped
 		wrap := &committingWrapper{stateDir: stateDir}
 		m := newMgr(t, stateDir, v, wrap, nil)
 		m.MaybeCommit(context.Background())
 		if wrap.calls != 1 {
 			t.Fatalf("update %d (%s): commit calls = %d, want 1", i+1, v, wrap.calls)
 		}
 		want := append([]string{"felhom-agent-" + v}, bystanders...)
 		sort.Strings(want)
 		got := folder(t, stateDir)
 		if strings.Join(got, " ") != strings.Join(want, " ") {
 			if _, err := os.Lstat(filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
 				t.Fatalf("a symlink with a staged-binary name was removed: %v", err)
 			}
 			t.Fatalf("after update %d (%s) the folder holds %v, want %v", i+1, v, got, want)
 		}
 		prev = v
 	}
 	if b, err := os.ReadFile(outside); err != nil || string(b) != "not ours" {
 		t.Fatalf("the file behind the symlink changed: %q %v", b, err)
 	}
 }
 // While an update is pending — the only time a rollback can fire — nothing is removed.
 func TestPrune_RefusesWhileAnUpdateIsPending(t *testing.T) {
 	stateDir := t.TempDir()
 	for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
 		stage(t, stateDir, v)
 	}
 	writePending(t, stateDir, PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0", SHA256: "ab", AppliedAt: "t"})
 	before := folder(t, stateDir)
 	removed, err := pruneStaged(stateDir, "0.172.0", nil)
 	if err == nil || len(removed) != 0 {
 		t.Fatalf("removed %v while an update was pending (err=%v) — want a refusal", removed, err)
 	}
 	if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
 		t.Fatalf("removed %v while an update was pending; folder now %v", before, got)
 	}
 	// The same through the commit path: a `commit` that reports success but leaves the marker (fakeWrapper does) must
 	// not lead to a prune.
 	m := newMgr(t, stateDir, "0.172.0", &fakeWrapper{}, nil)
 	m.MaybeCommit(context.Background())
 	if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
 		t.Fatalf("the commit path pruned with the marker still there; folder now %v", got)
 	}
 	// A marker nobody can read is a refusal too.
 	if err := os.WriteFile(pendingPath(stateDir), []byte("{not json"), 0o644); err != nil {
 		t.Fatal(err)
 	}
 	if removed, err := pruneStaged(stateDir, "0.172.0", nil); err == nil || len(removed) != 0 {
 		t.Fatalf("an unreadable marker must refuse; removed %v err=%v", removed, err)
 	}
 }
 // Only a commit prunes: a plain start, a version mismatch and a failed commit remove nothing — so the binary of a
 // version that was rolled back stays for the operator.
 func TestPrune_OnlyAHealthyCommitPrunes(t *testing.T) {
 	cases := []struct {
 		name    string
 		pending *PendingMarker
 		running string
 		wrap    WrapperRunner
 	}{
 		{"plain start, no marker (also the state after a rollback)", nil, "0.171.0", &fakeWrapper{}},
 		{"marker names another version", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.171.0", &fakeWrapper{}},
 		{"commit fails", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.172.0", &fakeWrapper{err: fmt.Errorf("sudo: no")}},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
 			stateDir := t.TempDir()
 			for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
 				stage(t, stateDir, v)
 			}
 			if c.pending != nil {
 				writePending(t, stateDir, *c.pending)
 			}
 			before := folder(t, stateDir)
 			newMgr(t, stateDir, c.running, c.wrap, nil).MaybeCommit(context.Background())
 			if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
 				t.Fatalf("folder changed without a healthy commit: %v → %v", before, got)
 			}
 		})
 	}
 	// A running version this code cannot name a file for: refuse, do not guess.
 	stateDir := t.TempDir()
 	stage(t, stateDir, "0.170.0")
 	if removed, err := pruneStaged(stateDir, "dev", nil); err == nil || len(removed) != 0 {
 		t.Fatalf("running version \"dev\" must refuse; removed %v err=%v", removed, err)
 	}
 }
 // The name rule and the version rule say the same thing: every name Execute can write is a name the prune knows, and
 // nothing else is.
 func TestPrune_NameShapeFollowsTheVersionRule(t *testing.T) {
 	for _, v := range []string{"0.70.1", "0.172.1", "10.0.0", "0.70.2-crash", "0.173.0-rc.1"} {
 		if !versionRe.MatchString(v) {
 			t.Fatalf("test input %q is not a version versionRe accepts", v)
 		}
 		if !stagedNameRe.MatchString("felhom-agent-" + v) {
 			t.Errorf("Execute can stage felhom-agent-%s but the prune does not know that name", v)
 		}
 	}
 	for _, n := range []string{"pending.json", "felhom-agent", "felhom-agent-", "felhom-agent-1.2", "felhom-agent-0.1.0.partial",
 		"felhom-agent.bak-0.139.0", "felhom-agent.prev", "felhom-agent-0.1.0/x", "../felhom-agent-0.1.0", "felhom-agent-0.1.0 ", "xfelhom-agent-0.1.0"} {
 		if stagedNameRe.MatchString(n) {
 			t.Errorf("the prune would treat %q as a staged binary", n)
 		}
 	}
 	m := stagedNameRe.FindStringSubmatch("felhom-agent-0.173.0-rc.1")
 	if m == nil || !versionRe.MatchString(m[1]) {
 		t.Fatalf("the version part of a staged name must satisfy versionRe; got %v", m)
 	}
 }
 func wrapperText(t *testing.T) string {
 	t.Helper()
 	b, err := os.ReadFile(filepath.Join("..", "..", "configs", "felhom-selfupdate-guarded"))
 	if err != nil {
 		t.Fatalf("the guarded wrapper is the other half of this proof and must be readable: %v", err)
 	}
 	return string(b)
 }
 // The stated dependency, pinned: the wrapper's `rollback` and `commit` read the pending marker and `.prev`, and no
 // staged binary. If either verb starts to read the staging folder, this fails and prune.go's rule must be looked at.
 func TestPrune_RollbackReadsNoStagedBinary(t *testing.T) {
 	s := wrapperText(t)
 	for _, want := range []string{"\nBIN=/usr/local/bin/felhom-agent\n", "\nPREV=$BIN.prev\n", "\nSTAGING=/var/lib/felhom-agent/selfupdate\n", "\nPENDING=$STAGING/pending.json\n"} {
 		if !strings.Contains(s, want) {
 			t.Fatalf("the wrapper no longer holds the line %q — the folder's rule (prune.go) was written against it", strings.TrimSpace(want))
 		}
 	}
 	for _, verb := range []string{"commit", "rollback"} {
 		i := strings.Index(s, "\n"+verb+")\n")
 		if i < 0 {
 			t.Fatalf("no %s) block in the wrapper", verb)
 		}
 		block := s[i:]
 		j := strings.Index(block, "\n    ;;\n")
 		if j < 0 {
 			t.Fatalf("no end of the %s) block", verb)
 		}
 		block = block[:j]
 		if !strings.Contains(block, "$PENDING") {
 			t.Fatalf("%s) does not read $PENDING any more", verb)
 		}
 		code := regexp.MustCompile(`(?m)^\s*#.*$`).ReplaceAllString(block, "")
 		for _, bad := range []string{"STAGING", "staged", "felhom-agent-"} {
 			if strings.Contains(code, bad) {
 				t.Fatalf("%s) now mentions %q — it may read a staged binary, and the prune removes those", verb, bad)
 			}
 		}
 	}
 	if stagedNameRe.MatchString("pending.json") || stagedNameRe.MatchString("felhom-agent.prev") {
 		t.Fatal("the prune would remove a file the rollback needs")
 	}
 }
 // The consequence, with the wrapper's own code: after N good updates (each committed by the REAL `commit` verb and
 // pruned), update N+1 goes bad and the REAL `rollback` verb puts the previous binary back. The wrapper runs from a copy
 // whose four fixed paths point into a temp folder; systemctl, logger and chown are stubs.
 func TestPrune_RealRollbackStillWorksAfterNUpdates(t *testing.T) {
 	sh, err := exec.LookPath("sh")
 	if err != nil {
 		t.Skip("no sh on this machine")
 	}
 	root := t.TempDir()
 	stateDir := filepath.Join(root, "state")
 	binDir := filepath.Join(root, "bin")
 	stubs := filepath.Join(root, "stubs")
 	for _, d := range []string{stagingDir(stateDir), binDir, stubs, filepath.Join(root, "rootstaging")} {
 		if err := os.MkdirAll(d, 0o750); err != nil {
 			t.Fatal(err)
 		}
 	}
 	live := filepath.Join(binDir, "felhom-agent")
 	s := wrapperText(t)
 	for old, repl := range map[string]string{
 		"\nBIN=/usr/local/bin/felhom-agent\n":                    "\nBIN=" + live + "\n",
 		"\nSTAGING=/var/lib/felhom-agent/selfupdate\n":           "\nSTAGING=" + stagingDir(stateDir) + "\n",
 		"\nROOT_STAGING=/var/lib/felhom-os-apply/agent-update\n": "\nROOT_STAGING=" + filepath.Join(root, "rootstaging") + "\n",
 	} {
 		if strings.Count(s, old) != 1 {
 			t.Fatalf("the wrapper's path line %q is not there exactly once — this test cannot sandbox it", strings.TrimSpace(old))
 		}
 		s = strings.Replace(s, old, repl, 1)
 	}
 	// Safety: no assignment in the copy may still point at a real system path (PREV and PENDING are built from BIN and
 	// STAGING, so they follow).
 	for _, line := range strings.Split(s, "\n") {
 		if regexp.MustCompile(`^[A-Z_]+=/`).MatchString(line) && !strings.Contains(line, root) {
 			t.Fatalf("the sandboxed wrapper still holds a real path: %q", line)
 		}
 	}
 	script := filepath.Join(root, "wrapper.sh")
 	if err := os.WriteFile(script, []byte(s), 0o755); err != nil {
 		t.Fatal(err)
 	}
 	stubLog := filepath.Join(root, "stub.log")
 	for _, n := range []string{"systemctl", "logger", "chown", "systemd-run"} {
 		body := "#!/bin/sh\necho \"" + n + " $*\" >> '" + stubLog + "'\nexit 0\n"
 		if err := os.WriteFile(filepath.Join(stubs, n), []byte(body), 0o755); err != nil {
 			t.Fatal(err)
 		}
 	}
 	run := func(verb string) (string, error) {
 		cmd := exec.Command(sh, script, verb)
 		cmd.Env = []string{"PATH=" + stubs + ":/usr/bin:/bin"}
 		out, err := cmd.CombinedOutput()
 		return string(out), err
 	}
 	// applied is what the wrapper's `apply` leaves behind (it needs root, so the test writes the same four things).
 	applied := func(oldV, newV string) {
 		cur, err := os.ReadFile(live)
 		if err != nil {
 			t.Fatal(err)
 		}
 		if err := os.WriteFile(live+".prev", cur, 0o755); err != nil {
 			t.Fatal(err)
 		}
 		if err := os.WriteFile(live, []byte("binary "+newV), 0o755); err != nil {
 			t.Fatal(err)
 		}
 		writePending(t, stateDir, PendingMarker{OldVersion: oldV, NewVersion: newV, SHA256: "ab", AppliedAt: "t"})
 	}
 	if err := os.WriteFile(live, []byte("binary 0.169.0"), 0o755); err != nil {
 		t.Fatal(err)
 	}
 	prev := "0.169.0"
 	for _, v := range []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1"} {
 		stage(t, stateDir, v)
 		applied(prev, v)
 		m := newMgr(t, stateDir, v, runnerFunc(func(args ...string) ([]byte, []byte, error) {
 			out, err := run(args[0])
 			return []byte(out), nil, err
 		}), nil)
 		m.MaybeCommit(context.Background()) // the REAL commit verb, then the prune
 		if got := folder(t, stateDir); len(got) != 1 || got[0] != "felhom-agent-"+v {
 			t.Fatalf("after the good update to %s the folder holds %v", v, got)
 		}
 		prev = v
 	}
 	// Update N+1 is applied and goes bad: systemd's OnFailure runs the wrapper's rollback.
 	stage(t, stateDir, "0.173.0")
 	applied("0.172.1", "0.173.0")
 	out, err := run("rollback")
 	if err != nil {
 		t.Fatalf("the real rollback verb failed after the prunes: %v\n%s", err, out)
 	}
 	if b, _ := os.ReadFile(live); string(b) != "binary 0.172.1" {
 		t.Fatalf("after the rollback the live binary is %q, want the previous version's", b)
 	}
 	if p, _ := readPending(stateDir); p != nil {
 		t.Fatal("the rollback left the pending marker")
 	}
 	if b, _ := os.ReadFile(stubLog); !bytes.Contains(b, []byte("systemctl restart felhom-agent.service")) {
 		t.Fatalf("the rollback did not restart the agent; stub log:\n%s", b)
 	}
 	// The rolled-back agent starts: no marker, so no commit and no prune — the bad version's binary stays to be read.
 	newMgr(t, stateDir, "0.172.1", &fakeWrapper{}, nil).MaybeCommit(context.Background())
 	if got := folder(t, stateDir); strings.Join(got, " ") != "felhom-agent-0.172.1 felhom-agent-0.173.0" {
 		t.Fatalf("after the rollback the folder holds %v, want the running version and the one that failed", got)
 	}
 }
 // runnerFunc adapts a function to WrapperRunner; it gets the verb arguments only.
 type runnerFunc func(args ...string) ([]byte, []byte, error)
 func (f runnerFunc) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
 	return f(args...)
 }
@@ -24,10 +24,10 @@
  "readers": [
    "scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
  ],
-  "recorded": "2026-10-29",
+  "recorded": "2026-11-03",
-  "recorded_by": "CC 2026-10-29: day0_agent 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
+  "recorded_by": "CC 2026-11-07: day0_agent 0.172.1 (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
  "pinned": {
    "hub_min_agent": "0.158.1",
-    "day0_agent": "0.169.0"
+    "day0_agent": "0.172.1"
  }
 }
Author	SHA1	Message	Date
admin	776025c824	selfupdate: the staging folder keeps what it needs and no more (R-1368 part 2) — not released gates / gates (push) Successful in 35s Details Each good update left its staged binary in <state dir>/selfupdate for ever (49 files / 740 MB on demo-hp). After the wrapper's commit returned success, MaybeCommit now removes every staged binary but the running version's. Nothing is removed while a pending marker exists or cannot be read, at a plain start, or after a rollback. Only a regular file with the exact name Execute writes is removed. The rollback reads pending.json and felhom-agent.prev, no staged binary: pinned by a test on the wrapper's text and by a test that runs the wrapper's real commit and rollback verbs in a sandbox. Red seen three times. No version bump, no build, no vouch. /usr/local/bin/felhom-agent.bak-* is not reached (root-owned; R-1368 part 4). Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS	2026-11-10 14:10:00 +01:00
adminandClaude Opus 5.5	41d5eb8051	v0.172.1 vouched and delivered to the three test boxes (agent + bundle, 77/77); pinned.day0_agent = 0.172.1 (D50) gates / gates (push) Successful in 44s Details Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS	2026-11-07 23:29:24 +01:00
adminandClaude Opus 5.5	8bd137787e	v0.172.1 released (not vouched): CHANGELOG, REPORT with the operator's PowerShell delivery, CONTEXT gates / gates (push) Successful in 33s Details Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS	2026-11-07 22:43:34 +01:00
adminandClaude Opus 5.5	39db03065a	os-apply: proxmox-grub joins the boot-chain lane (pve step refused R6 nightly on demo-felhom) gates / gates (push) Successful in 36s Details proxmox-grub 2.12-9+pmx4 depends on grub-efi-amd64 of the same version. HOST_SLOW_RE anchors `grub` at the start of a name, so proxmox-grub landed in the ring-0 pve plan, the simulation pulled grub-efi-amd64, and the whole pve step was refused R6 every night since 2026-11-05. Same class as the 2026-10-09 proxmox-secure-boot-support fix. Test test_ring0_pending_pve_leaves_proxmox_grub_out: red before the fix, green after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS	2026-11-07 22:41:05 +01:00
admin	ba409de4a2	v0.172.0 vouched: pinned.day0_agent = 0.172.0 (D50); CHANGELOG heading; REPORT + CONTEXT gates / gates (push) Successful in 34s Details	2026-11-03 10:29:51 +01:00
admin	a4f1280399	v0.172.0 released, not yet vouched: CHANGELOG heading (R-1302; agent 2c0222cd, bundle cc34b0bc) gates / gates (push) Successful in 36s Details	2026-11-03 09:08:07 +01:00
admin	ecbb0cc479	REPORT: 2026-11-02 night, R-1302, commit only gates / gates (push) Successful in 36s Details	2026-11-03 04:13:52 +01:00
admin	50eeccc986	R-1302: the first grant of a tier logs INFO granted; a grant that existed and vanished keeps the R-190 ERROR. Commit only. gates / gates (push) Successful in 36s Details	2026-11-02 23:16:10 +01:00
admin	808cdb36ad	REPORT + CONTEXT: v0.171.0 released and vouched; R-1344 restart proven live gates / gates (push) Successful in 33s Details	2026-10-30 11:09:48 +01:00
admin	84726c9f27	v0.171.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.171.0 (D50) gates / gates (push) Successful in 33s Details	2026-10-30 10:49:46 +01:00
admin	bd18347c1c	R-1344 decided from the night's measurement: a controller that runs but does not answer is restarted at 10 unhealthy sweeps (alarm at 6) — busy file + the brake refuse; stanza carries unresponsive_restart_after_s / unresponsive_restart_at gates / gates (push) Successful in 33s Details	2026-10-30 09:20:37 +01:00
admin	98bd1b71e9	REPORT + CONTEXT: v0.170.0 released and vouched; reboot_guest proven on the release builds gates / gates (push) Successful in 39s Details	2026-10-29 20:24:29 +01:00
admin	6578a8b91e	v0.170.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.170.0 (D50) gates / gates (push) Successful in 36s Details	2026-10-29 18:59:05 +01:00