hub v0.157.0: app_definition_refused (R-971), the vault time moves on a re-vault (R-954), the hub's own snapshot count (R-895 slice); uninstall knows felhom-thin-check

- hub: `app_definition_refused` in allowedEventTypes + operatorOnlyEvents (controller v0.328.0's catalog
  allow-list refusal); SaveHostRecoveryCredential moves set_at on a re-vault (found live: both rotated hosts
  kept their first vault's time); the daily key audit also lists the box's restic snapshots/ read-only over the
  login the hub already holds and logs count/newest/oldest (no alarm yet).
- installer (main only, rides the next tag): --uninstall removes /usr/local/sbin/felhom-thin-check (agent
  v0.159.0's bundle file); test_hostinstall.py knows it.
- runbook whole-guest-backup-orphan-thin.md: the human repair the agent's new orphan finding points to.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

documentation/runbooks/whole-guest-backup-orphan-thin.md

@@ -0,0 +1,42 @@
+# RUNBOOK — an orphan thin device stops a box's whole-guest backups (R-1012)
+
+**When:** the operator mail „whole-guest backup failed" ends with „probable cause: thin pool `<vg>/<pool>` holds device
+`<N>` that LVM does not know". The agent found it (agent ≥ v0.159.0); **it removed nothing.** The repair below is a
+human step on purpose: it deletes a thin device, and only a person decides that.
+
+**Why it happens:** the host's root filled up while vzdump took its storage snapshot; the kernel made the thin
+snapshot, LVM could not write its own metadata, and the device stayed in the pool with no LV. From then on LVM's next
+snapshot asks for the same device number and fails („Creation of new snapshot <N> of device <M> failed"). First seen
+on demo-hp 2026-10-14 (`audits/power-cut-fixes-2026-10-14/E/`). Agent v0.156.1's leftover cleanup keeps the root from
+filling; this is the repair when it happened anyway.
+
+## 1. Confirm (read-only)
+
+On the PVE host, as root:
+
+```bash
+felhom-thin-check pve data          # the agent's own read-only helper; prints pool ids and LVM's, ends with "ok"
+lvs -o lv_name,thin_id,pool_lv pve  # LVM's view: every thin LV with its id
+```
+
+The orphan is a `pool <N>` line with no `lvm-before <N>` and no `lvm-after <N>`. If the helper refuses with
+„a metadata snapshot … is already held", someone else is reading the pool; wait and run it again.
+
+**Stop here if the id belongs to an LV, or if a backup or restore is running on the host** (`pct list` shows a lock,
+or `ps -ef | grep vzdump`). Write down the id and the LVM list in the evidence folder first (R-320).
+
+## 2. Repair (deletes ONE thin device — the operator decides)
+
+```bash
+dmsetup message pve-data-tpool 0 "delete <N>"
+felhom-thin-check pve data          # now the pool's ids equal LVM's
+```
+
+(`pve-data-tpool` is `<vg>-<pool>-tpool`, with every `-` in a name doubled.) Then start one whole-guest backup of the
+guest from the controller („Mentés most") and read that it made its snapshot and finished. Done this way on demo-hp
+2026-10-14 12:02 UTC (`audits/power-cut-fixes-2026-10-14/E/repair.txt`, `E/next-backup.txt`).
+
+## 3. Afterwards
+
+- The register row for the box, with the id and the evidence path.
+- Why the root filled: `df -h /`, `/var/lib/vz/dump` — the cleanup (R-1006) should have prevented it.

hub/CHANGELOG.md

@@ -1,3 +1,20 @@
+## v0.157.0 — the allow-list's operator event, the vault time moves on a re-vault, the hub's own snapshot count (2026-10-17)
+
+- **`app_definition_refused`** (R-971, operator decision D2 = A): controller v0.328.0 refuses, before it writes any app
+  compose, a catalog definition that asks for container rights not on the box's compiled allow-list, and reports it.
+  Registered in `allowedEventTypes` AND `operatorOnlyEvents` in this commit (the household can do nothing; its app
+  page says the app keeps its last good definition). Not added to the per-app cooldown list — that list is fenced
+  (`TestR389_TheAllowListHasExactlyOneMember`); the controller already sends one event per refused definition.
+- **The Console access card's „last vaulted" time moves on a re-vault** (R-954 follow-up). `SaveHostRecoveryCredential`
+  kept the FIRST vault's `set_at` on every later upsert, so after the 2026-10-17 rotation demo-hp read „2026-07-21"
+  and Tester 1 „2026-10-04" over passwords changed minutes before. `set_at` now moves; pinned by
+  `TestHostRecovery_ReVaultMovesSetAt` (red-proved: the old upsert fails it).
+- **R-895, the measurement slice:** the daily off-site key audit also lists the box's restic `snapshots/` over the
+  login the hub already holds (the sealed sub-account password — no new credential) and LOGS the count, the newest and
+  the oldest upload time. Read-only (one `ls -l`), no alarm yet: the next slice alarms on files no box run explains,
+  once this log line has shown the provider's listing on real boxes. `internal/offsitekeys/snapshots.go`, pinned by
+  `TestListSnapshots_CountsTheBoxSnapshotFilesReadOnly`.
+
 ## v0.156.0 — a supported Go: 1.27.2, x/crypto v0.58.0, alpine 3.24.2 (R-972) (2026-10-16)
 
 - Built on Go 1.27.2 (was 1.24.13 — no security fixes left) with golang.org/x/crypto v0.58.0 (was v0.31.0) and

hub/internal/api/handler.go

@@ -2345,6 +2345,10 @@ var allowedEventTypes = map[string]bool{
 	// and repaired after an unclean host boot — COUNTS ONLY. Operator-only (notify.operatorOnlyEvents) — registered in
 	// BOTH in the same commit; the household's side is the app page's lost-files note on its own box.
 	"app_repaired_after_crash": true,
+	// R-971 (hub v0.157.0, controller v0.328.0, operator decision D2 = A): the box's catalog allow-list REFUSED an app's
+	// definition (a correction, an update, a restore, a settings change) — nothing was written, the app runs on its last
+	// good compose. Operator-only (notify.operatorOnlyEvents) — registered in BOTH in the same commit.
+	"app_definition_refused": true,
 	// Special
 	"test": true,
 }

hub/internal/notify/dispatcher.go

@@ -871,6 +871,10 @@ var operatorOnlyEvents = map[string]bool{
 	// of empty files found / repaired / still broken / originals lost. The household learns of a lost photo on its own
 	// app page (name and date, on its box); THESE counts are the operator's. Registered in the same commit that mints it.
 	"app_repaired_after_crash": true,
+	// R-971 (v0.157.0, controller v0.328.0, operator decision D2 = A). The box's catalog allow-list refused an app's
+	// definition: someone pushed a catalog change asking for container rights the box does not give. The household can
+	// do nothing; its app page says the app keeps its last good definition. Registered in the same commit that mints it.
+	"app_definition_refused": true,
 }
 
 // IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the

hub/internal/offsitekeys/service.go

@@ -215,6 +215,15 @@ func (s *Service) AuditAll(ctx context.Context, windowOpen WindowOpenFunc) []Aud
 		default:
 			s.logf("[INFO] offsitekeys: audit %s: %d line(s), all pinned append-only", c.CustomerID, o.Result.Lines)
 		}
+		// R-895 measurement slice: the hub's OWN count of the box's snapshot files (read-only, logged, no alarm yet).
+		if terr == nil {
+			if l, lerr := s.Reg.ListSnapshots(ctx, t, pw, time.Now().UTC()); lerr != nil {
+				s.logf("[WARN] offsitekeys: snapshots %s: the hub's own listing failed: %v", c.CustomerID, lerr)
+			} else {
+				s.logf("[INFO] offsitekeys: snapshots %s: the hub's own listing — %d snapshot file(s), %d other line(s), newest %s, oldest %s (R-895 measurement; no alarm yet)",
+					c.CustomerID, l.Files, l.Other, l.Newest.Format(time.RFC3339), l.Oldest.Format(time.RFC3339))
+			}
+		}
 	}
 	return out
 }

hub/internal/offsitekeys/snapshots.go

@@ -0,0 +1,82 @@
+package offsitekeys
+
+import (
+	"context"
+	"fmt"
+	"regexp"
+	"strings"
+	"time"
+)
+
+// R-895, THE MEASUREMENT SLICE (hub v0.157.0; design `audits/agent-allowlist-rotation-2026-10-17/P4/design-R-895.md`).
+//
+// The clean-up window's drop check trusts the snapshot counts the box sends. Option B of the design: the hub counts the
+// box's restic `snapshots/` files ITSELF, over the login it already holds (the sub-account password, sealed — no new
+// credential), and later alarms on files no box run explains. This slice only LISTS and LOGS — count, newest and
+// oldest upload time — once a day beside the authorized_keys audit, so the listing's shape on the provider is measured
+// before anything alarms on it. READ-ONLY: one `ls -l`; it writes, moves and deletes nothing.
+
+// SnapshotListing is the hub's own read of one repository's snapshots/ directory.
+type SnapshotListing struct {
+	Files  int       // entries whose name is a restic snapshot id (64 hex)
+	Other  int       // lines that are not (a header, a temp file, an unparsable line)
+	Newest time.Time // zero when no time could be read
+	Oldest time.Time
+}
+
+var snapshotID = regexp.MustCompile(`^[0-9a-f]{64}$`)
+
+// ListSnapshots lists <repo>/snapshots with `ls -l` in the restricted shell. now anchors a year-less date.
+func (r *Registrar) ListSnapshots(ctx context.Context, t Target, password string, now time.Time) (SnapshotListing, error) {
+	sh, err := r.open(ctx, t, password)
+	if err != nil {
+		return SnapshotListing{}, err
+	}
+	defer sh.Close()
+	out, err := sh.Run(ctx, "ls -l "+strings.TrimRight(t.RepoPath, "/")+"/snapshots", nil)
+	if err != nil {
+		return SnapshotListing{}, fmt.Errorf("offsitekeys: list snapshots: %w", err)
+	}
+	return parseLsL(string(out), now), nil
+}
+
+// parseLsL reads `ls -l` lines: mode links owner group size MON DD (HH:MM|YYYY) name.
+func parseLsL(out string, now time.Time) SnapshotListing {
+	var l SnapshotListing
+	for _, line := range strings.Split(out, "\n") {
+		f := strings.Fields(line)
+		if len(f) == 0 || strings.HasPrefix(line, "total ") {
+			continue
+		}
+		if len(f) < 9 || !snapshotID.MatchString(f[len(f)-1]) {
+			l.Other++
+			continue
+		}
+		l.Files++
+		if ts, ok := lsTime(f[len(f)-4], f[len(f)-3], f[len(f)-2], now); ok {
+			if l.Newest.IsZero() || ts.After(l.Newest) {
+				l.Newest = ts
+			}
+			if l.Oldest.IsZero() || ts.Before(l.Oldest) {
+				l.Oldest = ts
+			}
+		}
+	}
+	return l
+}
+
+// lsTime reads ls's two date forms: "Oct 17 03:12" (within six months, year implied) and "Oct 17 2025".
+func lsTime(mon, day, hm string, now time.Time) (time.Time, bool) {
+	if strings.Contains(hm, ":") {
+		ts, err := time.Parse("Jan 2 15:04 2006", fmt.Sprintf("%s %s %s %d", mon, day, hm, now.Year()))
+		if err != nil {
+			return time.Time{}, false
+		}
+		if ts.After(now.Add(24 * time.Hour)) { // December's file read in January
+			ts = ts.AddDate(-1, 0, 0)
+		}
+		return ts, true
+	}
+	ts, err := time.Parse("Jan 2 2006", fmt.Sprintf("%s %s %s", mon, day, hm))
+	return ts, err == nil
+}

hub/internal/offsitekeys/snapshots_test.go

@@ -0,0 +1,67 @@
+package offsitekeys
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+)
+
+type lsShell struct {
+	out  string
+	cmds []string
+}
+
+func (s *lsShell) Run(_ context.Context, cmd string, _ []byte) ([]byte, error) {
+	s.cmds = append(s.cmds, cmd)
+	return []byte(s.out), nil
+}
+func (s *lsShell) Close() error { return nil }
+
+type lsDialer struct{ sh *lsShell }
+
+func (d lsDialer) Dial(context.Context, Target, string) (Shell, error) { return d.sh, nil }
+
+const id1 = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9"
+const id2 = "ffeeddccbbaa99887766554433221100ffeeddccbbaa99887766554433221100"
+
+// R-895 measurement slice: the hub's own listing reads the snapshot files and their upload times from `ls -l`, counts
+// only restic snapshot ids, and issues exactly ONE read-only command. RED-PROOF
+// (audits/agent-allowlist-rotation-2026-10-17/P4/redproof-listing.txt): with parseLsL counting every line, Files is 3
+// and this test fails.
+func TestListSnapshots_CountsTheBoxSnapshotFilesReadOnly(t *testing.T) {
+	sh := &lsShell{out: "total 8\n" +
+		"-rw------- 1 u1-sub4 u1-sub4 324 Oct 17 03:12 " + id1 + "\n" +
+		"-rw------- 1 u1-sub4 u1-sub4 331 Mar  2  2026 " + id2 + "\n" +
+		"-rw------- 1 u1-sub4 u1-sub4  12 Oct 17 03:13 .tmp-upload\n"}
+	r := &Registrar{Dialer: lsDialer{sh}}
+	now := time.Date(2026, 10, 17, 14, 0, 0, 0, time.UTC)
+	l, err := r.ListSnapshots(context.Background(), tgt, "pw", now)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if l.Files != 2 || l.Other != 1 {
+		t.Fatalf("files=%d other=%d, want 2 and 1", l.Files, l.Other)
+	}
+	if !l.Newest.Equal(time.Date(2026, 10, 17, 3, 12, 0, 0, time.UTC)) || !l.Oldest.Equal(time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC)) {
+		t.Fatalf("newest=%v oldest=%v", l.Newest, l.Oldest)
+	}
+	if len(sh.cmds) != 1 || sh.cmds[0] != "ls -l /home/felhom-repo/snapshots" {
+		t.Fatalf("commands = %v — the listing must be one read-only ls", sh.cmds)
+	}
+	for _, c := range sh.cmds {
+		for _, w := range []string{"rm", "mv", "dd", "chmod", "mkdir", "touch", "cp", "unlink"} {
+			if strings.HasPrefix(c, w+" ") {
+				t.Fatalf("a write verb in the listing: %q", c)
+			}
+		}
+	}
+}
+
+// A December file read in January is last year's, not next year's.
+func TestLsTime_YearBoundary(t *testing.T) {
+	ts, ok := lsTime("Dec", "30", "23:10", time.Date(2027, 1, 2, 0, 0, 0, 0, time.UTC))
+	if !ok || ts.Year() != 2026 {
+		t.Fatalf("got %v %v", ts, ok)
+	}
+}

hub/internal/store/host_recovery.go

@@ -18,7 +18,9 @@ type HostRecoveryCredential struct {
 }
 
 // SaveHostRecoveryCredential upserts a host's break-glass credential (last-write-wins: day-0 sets it,
-// --rotate re-sets). R-133 (hub v0.135.0): the secret is SEALED at rest with the same key and the same
+// --rotate re-sets). set_at is the time THIS secret was vaulted — a re-vault moves it (R-954 follow-up, hub v0.157.0:
+// it used to keep the FIRST vault's time, so the Console access card said „last vaulted 2026-07-21" over a password
+// rotated minutes before; pinned by TestHostRecovery_ReVaultMovesSetAt). R-133 (hub v0.135.0): the secret is SEALED at rest with the same key and the same
 // helpers as the off-site sub-account passwords (offsite_seal.go, OFFSITE_SECRET_KEY) — a copy of hub.db
 // alone no longer holds any box's console password. No key → the save is REFUSED (ErrNoSealKey): a hub that
 // cannot seal must not fall back to plaintext. The hub NEVER logs the secret and only ever returns it over
@@ -34,6 +36,7 @@ func (s *Store) SaveHostRecoveryCredential(hostID, username, secret string) erro
 		ON CONFLICT(host_id) DO UPDATE SET
 			username   = excluded.username,
 			secret     = excluded.secret,
+			set_at     = datetime('now'),
 			updated_at = datetime('now')`,
 		hostID, username, sealed)
 	return err

hub/internal/store/r954_revault_set_at_test.go

@@ -0,0 +1,37 @@
+package store
+
+import (
+	"testing"
+	"time"
+)
+
+// R-954 follow-up (hub v0.157.0): a re-vault (`--rotate-recovery`, the operator's own set-and-vault) MOVES set_at — the
+// time the Console access card shows as „last vaulted". Found live 2026-10-17: demo-hp's and Tester 1's rotated
+// passwords both still read set_at = their FIRST vault (2026-07-21, 2026-10-04). RED-PROOF
+// (felhom.eu audits/agent-allowlist-rotation-2026-10-17/P3/redproof-set-at.txt): without `set_at = datetime('now')` in
+// the upsert this test fails on both the meta and the credential read.
+func TestHostRecovery_ReVaultMovesSetAt(t *testing.T) {
+	st := sealTestStore(t)
+	if err := st.SaveHostRecoveryCredential("h1", "root@pam", "first-Pw-1"); err != nil {
+		t.Fatal(err)
+	}
+	// Age the first vault by a year, as a real day-0 vault would be.
+	if _, err := st.db.Exec(`UPDATE host_recovery SET set_at = '2025-10-17 10:00:00' WHERE host_id = 'h1'`); err != nil {
+		t.Fatal(err)
+	}
+	before := time.Now().UTC().Add(-time.Minute)
+	if err := st.SaveHostRecoveryCredential("h1", "root@pam", "second-Pw-2"); err != nil {
+		t.Fatal(err)
+	}
+	c, err := st.GetHostRecoveryCredential("h1")
+	if err != nil || c == nil || c.Secret != "second-Pw-2" {
+		t.Fatalf("credential = %+v, %v", c, err)
+	}
+	if c.SetAt.Before(before) {
+		t.Fatalf("credential set_at = %v — the re-vault kept the first vault's time", c.SetAt)
+	}
+	m, err := st.GetHostRecoveryMeta("h1")
+	if err != nil || m == nil || m.SetAt.Before(before) {
+		t.Fatalf("meta set_at = %+v, %v — the card would show the first vault's time", m, err)
+	}
+}

scripts/CHANGELOG.md

@@ -1,3 +1,10 @@
+## 2026-10-17 — the uninstall removes agent v0.159.0's new root helper (R-1012); not yet tagged
+
+`felhom-host-install.sh --uninstall` removes `/usr/local/sbin/felhom-thin-check`, the read-only orphan-thin check the
+agent's config bundle installs since v0.159.0; `test_hostinstall.py` names it among the bundle files (its
+`test_bundle_list_is_current` fails when the bundle gains a file the uninstall does not know). Pushing to `main`
+publishes nothing — this rides the next `installer-v…` tag. SCRIPT_VERSION unchanged.
+
 ## 2026-10-17 — `~/.config/credentials` deleted (R-1134)
 
 The operator's go, after every key in it was checked equal in the CC store. `read_credential.py`, `fb_probe.py` and

scripts/felhom-host-install.sh

@@ -1357,6 +1357,8 @@ run_uninstall() {
     if [[ -f /usr/local/sbin/felhom-backup-target-apply ]]; then run rm -f /usr/local/sbin/felhom-backup-target-apply; else log_skip "  felhom-backup-target-apply already absent"; fi
     # agent v0.156.1 (R-1006): the dead half-written vzdump remover the config bundle installs.
     if [[ -f /usr/local/sbin/felhom-vzdump-partial-clean ]]; then run rm -f /usr/local/sbin/felhom-vzdump-partial-clean; else log_skip "  felhom-vzdump-partial-clean already absent"; fi
+    # agent v0.159.0 (R-1012): the read-only orphan-thin check the config bundle installs.
+    if [[ -f /usr/local/sbin/felhom-thin-check ]]; then run rm -f /usr/local/sbin/felhom-thin-check; else log_skip "  felhom-thin-check already absent"; fi
     # 1.33.0 (`16` §5): the internal-place helper. Its VOLUME (pve/felhom-belso) holds household data and is KEPT.
     if [[ -f /usr/local/sbin/felhom-internal-place ]]; then run rm -f /usr/local/sbin/felhom-internal-place; else log_skip "  felhom-internal-place already absent"; fi
     if [[ -f /usr/local/sbin/felhom-os-apply ]]; then run rm -f /usr/local/sbin/felhom-os-apply; else log_skip "  felhom-os-apply already absent"; fi

scripts/test_hostinstall.py

@@ -36,6 +36,7 @@ BUNDLE_DESTS = [
     "/usr/local/sbin/felhom-pbs-apply",
     "/usr/local/sbin/felhom-backup-target-apply",
     "/usr/local/sbin/felhom-vzdump-partial-clean",   # agent v0.156.1 (R-1006)
+    "/usr/local/sbin/felhom-thin-check",             # agent v0.159.0 (R-1012)
     "/usr/local/sbin/felhom-internal-place",         # agent v0.158.0 (felhom.eu `16` §5)
     "/usr/local/sbin/felhom-os-apply",
     "/usr/local/sbin/felhom-crash-guard",