CHANGELOG (Unreleased): the live proof the next release owes for the after-failure orphan check

v0.159.0/v0.159.1 ran the check before every snapshot-mode whole-guest backup, which takes a
metadata snapshot of the live pool. Now explainFailure (vzdump start or task failure) is its only
caller; the finding is still appended to the failed backup's error. Tests: failure = 1 call,
success = 0 calls; red-proved against the v0.159.1 runner.go (2 calls / calls=1).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

CHANGELOG.md

@@ -1,3 +1,59 @@
+## Unreleased — the orphan thin check runs only after a failed backup (R-1012 follow-up) (2026-10-17)
+
+**Not released, not vouched; the next vouched agent release carries it.** v0.159.0/v0.159.1 ran the R-1012 orphan check
+before EVERY snapshot-mode whole-guest backup, and that check takes a short metadata snapshot of the live thin pool.
+Now `BackupRunner` runs it only when the snapshot step has failed (the vzdump start or its task) — the moment the
+answer is needed. The finding is still appended in plain words to the failed backup's error, exactly as before; a
+successful backup never runs the check and keeps a clean record; stop mode and an unwired runner are unchanged. The
+`thin-orphan-check` capability text says „after a failed whole-guest backup". Tests
+`TestBackup_SnapshotFailure_NamesTheOrphan` (now: exactly 1 call, after the failure) and
+`TestBackup_ThinCheck_SilentWhenNotApplicable` (now: 0 calls on a successful backup), red-proved against the v0.159.1
+`runner.go`: failure legs „thin check ran 2 times, want 1", success leg `calls=1`. No root file, sudoers or bundle
+change. (Installer side, felhom.eu, same night: `felhom-host-install.sh --rotate-recovery-only`, R-1141.)
+**Owed by the release that carries it (not yet proven live):** one green snapshot backup with NO `felhom-thin-check`
+call in the journal, and one failed snapshot backup whose error carries the finding.
+
+## v0.159.1 — the signed decommission prunes the guest-bind record (found by v0.159.0's own live proof) (2026-10-17)
+
+**v0.159.0 is superseded — do not vouch it.** Everything of v0.159.0 below, plus: `signedjobs.DecommissionExecutor`
+drops the drive from every guest's bind record (`SetGuestBinds`, wired to the agent's `GuestBindStore`), as the
+self-serve decommission always did. Without it a drive decommissioned by an operator-signed job and then forgotten
+(R-1124) left a record the 20-second guest-bind re-assert warned about every tick („reconcile: skipping non-enrolled
+drive") — seen on demo-hp minutes after the R-1124 proof. Test `TestDecommissionExecutor_PrunesTheGuestBindRecord`,
+red-proved. Released: tag `v0.159.1`, sha256 `de8f6ad9…aff9`, bundle `2af3d27d…7cd6` (the same root files as v0.159.0).
+**Not vouched.** Delivery to a box on the v0.158.1 bundle: `agent_update 0.159.1` → `agent_config_update 0.159.0-step1`
+(sha `1d65edb3…d6cf`, published) → `agent_config_update 0.159.1` (the bundle adds `felhom-thin-check`, R16).
+
+## v0.159.0 — forget a decommissioned drive that is gone (R-1124); name an orphan thin device in a backup failure (R-1012); Go 1.27.2 (R-972) (2026-10-17)
+
+**Superseded by v0.159.1 — do not vouch.** Task: „agent release, the catalog allow-list, break-glass rotation, off-site
+history check" (felhom.eu `REPORT-agent-allowlist-rotation-2026-10-17.md`).
+
+- **Go 1.27.2 + x/crypto v0.58.0** (R-972, `37b99d7`): the first release that carries them (v0.158.1 was built with
+  go1.26.0).
+- **R-1124 — a decommissioned drive that is gone is forgotten.** `SudoHostOps.ForgetRetiredDrives` (agent start and the
+  20-second storage tick): for a Felhom drive unit whose intent is `decommissioned`, whose `/dev/disk/by-uuid` link is
+  gone and which is not mounted, `systemctl disable` + the new root verb `felhom-priv-apply unit-forget <unit>` +
+  `daemon-reload`. The unit is gone, so a later boot no longer waits for the drive and the agent stops listing it (no
+  „absent by UUID" line every 20 s). A decommissioned drive still plugged in KEEPS its unit and raw mount, so the
+  one-click re-enroll (H3) still works. `unit-forget` re-checks as root: a felhom by-UUID drive unit (not a network
+  unit), disabled, its `Where` not mounted, its by-UUID link absent — otherwise refused. Sudoers `FELHOM_MOUNT` gains the
+  one verb. Tests `TestForgetRetiredDrives_*` and `configs/test_felhom_priv_apply.py` `UnitForget`, red-proved.
+  **Live on demo-hp:** a loop-backed test drive enrolled through the local API, decommissioned by a signed job (kept
+  its unit while present), unplugged → 8 s later the unit and its wants link were removed and `GET /disks` dropped it.
+- **R-1012 — an orphan thin device is NAMED in the backup failure, never removed.** New READ-ONLY root helper
+  `felhom-thin-check <vg> <pool>` (config bundle): LVM's thin ids before and after, the pool's ids through a metadata
+  snapshot it reserves and releases (refuses — exit 3 — when someone else holds one; releases on every exit; sends no
+  other pool message). `internal/thincheck`: an orphan = in the pool and in neither LVM list, on two reads 3 s apart;
+  fails open. `BackupRunner.SetThinCheck`: before a snapshot-mode backup (a WARN when found) and again after a failure —
+  the finding is appended in plain words to the error the controller mails to the operator, pointing at the new
+  runbook `whole-guest-backup-orphan-thin.md`. Sudoers `FELHOM_THINCHECK`, capability `thin-orphan-check`. Tests
+  `internal/thincheck`, `TestBackup_SnapshotFailure_NamesTheOrphan`, `configs/test_felhom_thin_check.py`, red-proved.
+  **Live on demo-hp:** on a scratch thin pool (a loop file on `/mnt/hdd_1`, never `pve/data`) the helper listed a made
+  orphan (device 7) and refused a snapshot held by someone else; on `pve/data` it read 1, 2 = LVM's and released; a
+  normal whole-guest backup of 9201 ran the check first and finished green.
+- Released: tag `v0.159.0`, sha256 `3cfd0a25…edfed5`, bundle `40663c4f…6d681`; step bundle `0.159.0-step1`.
+
 ## CI only — a manual proof that the failure mail can be sent (R-1129, 2026-10-17)
 
 No agent release; no binary changes.

CONTEXT.md

@@ -1,5 +1,10 @@
 # CONTEXT — felhom-agent working state
 
+> **2026-10-17 — v0.159.1 RELEASED (0.159.0 superseded, do not vouch).** Go 1.27.2; R-1124 a decommissioned drive that is
+> gone is forgotten (`ForgetRetiredDrives`, root verb `unit-forget`); R-1012 the read-only `felhom-thin-check` names an
+> orphan thin device in a failed backup's text; the signed decommission prunes the guest-bind record. Bundle adds a path →
+> step bundle `0.159.0-step1`. Proven on demo-hp; delivered by signed jobs on the operator's yes. Vouch: the operator's.
+
 > **2026-10-16 — v0.158.1 RELEASED and VOUCHED (storage identity build B; 0.158.0 superseded, do not vouch).** The internal
 > data place (root helper `felhom-internal-place`, the record in the state dir, mounted at /mnt/belso like a drive), the
 > signed `add_internal_place`, partitions in `/disks/candidates` (`formattable` only while nothing on the disk is mounted),

REPORT.md

@@ -1,12 +1,41 @@
-# REPORT — the agent on Go 1.27.2 (R-972, 2026-10-16) — NOT released
+# REPORT — the orphan thin check runs only after a failed backup (unreleased) (2026-10-17, night)
 
-Written in ASD-STE100 Simplified Technical English. Part of burndown 2 (felhom.eu `REPORT-burndown2-2026-10-16.md`).
+Written in ASD-STE100 Simplified Technical English. Architecture read: felhom.eu `documentation/architecture/03-host-agent.md`
+(the `FELHOM_THINCHECK` row), `runbooks/whole-guest-backup-orphan-thin.md`, R-1012's closing note.
 
-- **What changed:** `go.mod` `go 1.27.2` and golang.org/x/crypto v0.58.0. No code change.
-- **Proof:** `go build ./...`, `go vet ./...`, `go test ./...` rc 0 on Go 1.27.2; `agent_gates.py --fast` all OK.
-- **Not done, on purpose:** no release and no deploy — agent releases wait for the operator's vouch. R-972 stays
-  NARROWED for the agent: the next vouched release carries the new Go.
+## In plain words
 
-## Observations
+- Before tonight the agent looked for an orphan thin device before **every** whole-guest backup. To look, it takes a
+  short metadata snapshot of the live disk pool. Now it looks **only when the backup's snapshot step has failed** — the
+  only moment somebody needs the answer. The operator's failure mail still names the orphan, in the same words.
+- **Not released. Not vouched.** The next agent release that the operator vouches carries it. Nothing was copied to a
+  box.
 
-1. The release script needs no change: it builds with the toolchain `go.mod` names. NOT-A-FINDING: GOTOOLCHAIN=auto on DooPlex fetched go1.27.2 for this build and will for a release build.
+## What changed
+
+- `internal/backup/runner.go`: the pre-run check (and its WARN line) is gone; `explainFailure` (after a vzdump start
+  failure or a task failure, snapshot mode only) is now the only caller. Comments say so.
+- `internal/capability/manifest.go`: the `thin-orphan-check` text says „after a failed whole-guest backup".
+- `internal/backup/r1012_thincheck_test.go`: failure → exactly ONE call and the finding in the error; success → ZERO
+  calls and a clean record; stop mode → zero calls; unwired → error unchanged.
+
+## Red-proof
+
+The new tests, run against the v0.159.1 `runner.go` (`git show HEAD:…`), then the fix restored:
+
+| test | failed with |
+|---|---|
+| `TestBackup_SnapshotFailure_NamesTheOrphan/{task_fails,start_fails}` | `thin check ran 2 times, want 1 (after the failure only)` |
+| `TestBackup_ThinCheck_SilentWhenNotApplicable` | `success: … Success:true … calls=1 (want no check on a successful backup, clean record)` |
+
+Restored → both PASS; `git diff` on `runner.go` = the fix only.
+
+## Gates
+
+`go build ./... && go vet ./... && go test ./...` and `python3 scripts/agent_gates.py` — results in the session report.
+
+## Not live-validated
+
+No agent was built or delivered (no release tonight). The changed path is the failure path; the next release's live
+proof should make one snapshot backup succeed (helper NOT called: no `felhom-thin-check` line in the agent journal
+and none in the sudo log for that run) and one fail (the finding in the failure text).

internal/backup/r1012_thincheck_test.go

@@ -10,8 +10,9 @@ import (
 )
 
 // R-1012: a snapshot backup that fails carries the orphan finding in its error — the text the controller mails to
-// the operator (`whole_guest_backup_failed`). The check runs once before the run and once, FRESH, after the failure.
-// RED-PROOF (P1/redproof-thincheck.txt): with explainFailure's append removed this test fails.
+// the operator (`whole_guest_backup_failed`). The check runs ONCE, after the failure — never before the run.
+// RED-PROOF (P1/redproof-thincheck.txt): with explainFailure's append removed this test fails. RED-PROOF (2026-10-17
+// night, felhom-agent REPORT.md): with the v0.159.0 pre-run check restored, calls = 2 and this test fails.
 func TestBackup_SnapshotFailure_NamesTheOrphan(t *testing.T) {
 	const why = "probable cause: thin pool pve/data holds device 4 that LVM does not know"
 	for name, api := range map[string]*fakeBackupAPI{
@@ -29,15 +30,17 @@ func TestBackup_SnapshotFailure_NamesTheOrphan(t *testing.T) {
 			if !strings.Contains(rec.Error, why) {
 				t.Fatalf("the failure does not name the orphan: %q", rec.Error)
 			}
-			if calls != 2 {
-				t.Fatalf("thin check ran %d times, want 2 (before the run and after the failure)", calls)
+			if calls != 1 {
+				t.Fatalf("thin check ran %d times, want 1 (after the failure only)", calls)
 			}
 		})
 	}
 }
 
-// A successful backup keeps a clean record (the pre-check only logs); a stop-mode backup takes no snapshot, so the
-// check never runs; an unwired runner behaves exactly as before.
+// A successful backup never runs the check (it takes a metadata snapshot of the live pool, and nobody needs the answer)
+// and keeps a clean record; a stop-mode backup takes no snapshot, so the check never runs; an unwired runner behaves
+// exactly as before. RED-PROOF (2026-10-17 night): with the v0.159.0 pre-run check restored, the success leg fails
+// with calls=1.
 func TestBackup_ThinCheck_SilentWhenNotApplicable(t *testing.T) {
 	ok := &fakeBackupAPI{vzdumpUPID: "UPID:x",
 		content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-x", Content: "backup", VMID: 9201, Size: 100, CTime: 1}}}
@@ -45,8 +48,8 @@ func TestBackup_ThinCheck_SilentWhenNotApplicable(t *testing.T) {
 	calls := 0
 	r.SetThinCheck(func(context.Context, int) string { calls++; return "probable cause: x" })
 	rec, err := r.Backup(context.Background(), 9201)
-	if err != nil || !rec.Success || rec.Error != "" || calls != 1 {
-		t.Fatalf("success: err=%v rec=%+v calls=%d (want one pre-check, clean record)", err, rec, calls)
+	if err != nil || !rec.Success || rec.Error != "" || calls != 0 {
+		t.Fatalf("success: err=%v rec=%+v calls=%d (want no check on a successful backup, clean record)", err, rec, calls)
 	}
 
 	stop := NewBackupRunner(&fakeBackupAPI{vzdumpUPID: "UPID:x", waitErr: errors.New("boom")}, "local", proxmox.ModeStop, "", "", quiet())

internal/backup/runner.go

@@ -62,8 +62,9 @@ type BackupRunner struct {
 	// this target, so a leftover from a power cut cannot hold the space this backup needs. nil = nothing.
 	beforeBackup func(ctx context.Context, storage string, vmid int)
 	// thinCheck (R-1012) names an orphan thin device in the guest's pools in plain words ("" = none / not checked). It
-	// runs before a snapshot-mode backup (a WARN in the log) and again when the vzdump fails, so the failure the
-	// controller mails to the operator says why. It never removes anything. nil = not wired.
+	// runs ONLY when a snapshot-mode vzdump has failed (start or task), so the failure the controller mails to the
+	// operator says why; a successful backup never runs it (it takes a metadata snapshot of the live pool). It never
+	// removes anything. nil = not wired.
 	thinCheck func(ctx context.Context, vmid int) string
 	logger    *slog.Logger
 	now       func() time.Time
@@ -117,8 +118,8 @@ func (r *BackupRunner) SetBeforeBackup(fn func(ctx context.Context, storage stri
 // SetThinCheck wires the R-1012 orphan check (see BackupRunner.thinCheck).
 func (r *BackupRunner) SetThinCheck(fn func(ctx context.Context, vmid int) string) { r.thinCheck = fn }
 
-// explainFailure appends the orphan finding to a failed snapshot-mode backup's error. A FRESH check (not the one from
-// before the run): the failure itself may be what made the orphan. Bounded, so a stuck helper cannot hold the job.
+// explainFailure appends the orphan finding to a failed snapshot-mode backup's error — the only place the check runs, so
+// it sees the pool as the failure left it. Bounded, so a stuck helper cannot hold the job.
 func (r *BackupRunner) explainFailure(ctx context.Context, vmid int, rec *hub.Backup) {
 	if r.thinCheck == nil || r.mode != proxmox.ModeSnapshot {
 		return
@@ -221,11 +222,8 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
 		return rec, fmt.Errorf("backup: %s", rec.Error)
 	}
 
-	if r.thinCheck != nil && r.mode == proxmox.ModeSnapshot {
-		if why := r.thinCheck(ctx, vmid); why != "" {
-			r.logger.Warn("backup: an orphan thin device may make this snapshot backup fail (R-1012)", "vmid", vmid, "finding", why)
-		}
-	}
+	// R-1012: no orphan check here. It takes a metadata snapshot of the live pool, and its answer is only needed when the
+	// snapshot step has failed — explainFailure runs it then (pinned by TestBackup_ThinCheck_SilentWhenNotApplicable).
 	upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
 		VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
 		PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)

internal/capability/manifest.go

@@ -91,7 +91,7 @@ var manifest = []Capability{
 	{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
 	{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
 	{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
-	{"thin-orphan-check", "orphan thin device check around a whole-guest backup (read-only, R-1012)", "/usr/local/sbin/felhom-thin-check", []string{"pve", "data"}, false, ""},
+	{"thin-orphan-check", "orphan thin device check after a failed whole-guest backup (read-only, R-1012)", "/usr/local/sbin/felhom-thin-check", []string{"pve", "data"}, false, ""},
 
 	// ---- Storage mount units (watchdog re-mount) ----
 	{"mount-unit-install", "fs-UUID mount unit install (root content check, R-861)", "/usr/local/sbin/felhom-priv-apply", []string{"unit", "mnt-felhom\\x2dx.mount"}, false, ""},