diff --git a/documentation/runbooks/whole-guest-backup-orphan-thin.md b/documentation/runbooks/whole-guest-backup-orphan-thin.md new file mode 100644 index 00000000..fa86c2fa --- /dev/null +++ b/documentation/runbooks/whole-guest-backup-orphan-thin.md @@ -0,0 +1,42 @@ +# RUNBOOK — an orphan thin device stops a box's whole-guest backups (R-1012) + +**When:** the operator mail „whole-guest backup failed" ends with „probable cause: thin pool `/` holds device +`` that LVM does not know". The agent found it (agent ≥ v0.159.0); **it removed nothing.** The repair below is a +human step on purpose: it deletes a thin device, and only a person decides that. + +**Why it happens:** the host's root filled up while vzdump took its storage snapshot; the kernel made the thin +snapshot, LVM could not write its own metadata, and the device stayed in the pool with no LV. From then on LVM's next +snapshot asks for the same device number and fails („Creation of new snapshot of device failed"). First seen +on demo-hp 2026-10-14 (`audits/power-cut-fixes-2026-10-14/E/`). Agent v0.156.1's leftover cleanup keeps the root from +filling; this is the repair when it happened anyway. + +## 1. Confirm (read-only) + +On the PVE host, as root: + +```bash +felhom-thin-check pve data # the agent's own read-only helper; prints pool ids and LVM's, ends with "ok" +lvs -o lv_name,thin_id,pool_lv pve # LVM's view: every thin LV with its id +``` + +The orphan is a `pool ` line with no `lvm-before ` and no `lvm-after `. If the helper refuses with +„a metadata snapshot … is already held", someone else is reading the pool; wait and run it again. + +**Stop here if the id belongs to an LV, or if a backup or restore is running on the host** (`pct list` shows a lock, +or `ps -ef | grep vzdump`). Write down the id and the LVM list in the evidence folder first (R-320). + +## 2. Repair (deletes ONE thin device — the operator decides) + +```bash +dmsetup message pve-data-tpool 0 "delete " +felhom-thin-check pve data # now the pool's ids equal LVM's +``` + +(`pve-data-tpool` is `--tpool`, with every `-` in a name doubled.) Then start one whole-guest backup of the +guest from the controller („Mentés most") and read that it made its snapshot and finished. Done this way on demo-hp +2026-10-14 12:02 UTC (`audits/power-cut-fixes-2026-10-14/E/repair.txt`, `E/next-backup.txt`). + +## 3. Afterwards + +- The register row for the box, with the id and the evidence path. +- Why the root filled: `df -h /`, `/var/lib/vz/dump` — the cleanup (R-1006) should have prevented it. diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 858a8faf..94d80fb9 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,20 @@ +## v0.157.0 — the allow-list's operator event, the vault time moves on a re-vault, the hub's own snapshot count (2026-10-17) + +- **`app_definition_refused`** (R-971, operator decision D2 = A): controller v0.328.0 refuses, before it writes any app + compose, a catalog definition that asks for container rights not on the box's compiled allow-list, and reports it. + Registered in `allowedEventTypes` AND `operatorOnlyEvents` in this commit (the household can do nothing; its app + page says the app keeps its last good definition). Not added to the per-app cooldown list — that list is fenced + (`TestR389_TheAllowListHasExactlyOneMember`); the controller already sends one event per refused definition. +- **The Console access card's „last vaulted" time moves on a re-vault** (R-954 follow-up). `SaveHostRecoveryCredential` + kept the FIRST vault's `set_at` on every later upsert, so after the 2026-10-17 rotation demo-hp read „2026-07-21" + and Tester 1 „2026-10-04" over passwords changed minutes before. `set_at` now moves; pinned by + `TestHostRecovery_ReVaultMovesSetAt` (red-proved: the old upsert fails it). +- **R-895, the measurement slice:** the daily off-site key audit also lists the box's restic `snapshots/` over the + login the hub already holds (the sealed sub-account password — no new credential) and LOGS the count, the newest and + the oldest upload time. Read-only (one `ls -l`), no alarm yet: the next slice alarms on files no box run explains, + once this log line has shown the provider's listing on real boxes. `internal/offsitekeys/snapshots.go`, pinned by + `TestListSnapshots_CountsTheBoxSnapshotFilesReadOnly`. + ## v0.156.0 — a supported Go: 1.27.2, x/crypto v0.58.0, alpine 3.24.2 (R-972) (2026-10-16) - Built on Go 1.27.2 (was 1.24.13 — no security fixes left) with golang.org/x/crypto v0.58.0 (was v0.31.0) and diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index fbcbf755..10f892f5 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -2345,6 +2345,10 @@ var allowedEventTypes = map[string]bool{ // and repaired after an unclean host boot — COUNTS ONLY. Operator-only (notify.operatorOnlyEvents) — registered in // BOTH in the same commit; the household's side is the app page's lost-files note on its own box. "app_repaired_after_crash": true, + // R-971 (hub v0.157.0, controller v0.328.0, operator decision D2 = A): the box's catalog allow-list REFUSED an app's + // definition (a correction, an update, a restore, a settings change) — nothing was written, the app runs on its last + // good compose. Operator-only (notify.operatorOnlyEvents) — registered in BOTH in the same commit. + "app_definition_refused": true, // Special "test": true, } diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index 1f1834d7..5cd74b8c 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -871,6 +871,10 @@ var operatorOnlyEvents = map[string]bool{ // of empty files found / repaired / still broken / originals lost. The household learns of a lost photo on its own // app page (name and date, on its box); THESE counts are the operator's. Registered in the same commit that mints it. "app_repaired_after_crash": true, + // R-971 (v0.157.0, controller v0.328.0, operator decision D2 = A). The box's catalog allow-list refused an app's + // definition: someone pushed a catalog change asking for container rights the box does not give. The household can + // do nothing; its app page says the app keeps its last good definition. Registered in the same commit that mints it. + "app_definition_refused": true, } // IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the diff --git a/hub/internal/offsitekeys/service.go b/hub/internal/offsitekeys/service.go index 42256e8a..3116c4ba 100644 --- a/hub/internal/offsitekeys/service.go +++ b/hub/internal/offsitekeys/service.go @@ -215,6 +215,15 @@ func (s *Service) AuditAll(ctx context.Context, windowOpen WindowOpenFunc) []Aud default: s.logf("[INFO] offsitekeys: audit %s: %d line(s), all pinned append-only", c.CustomerID, o.Result.Lines) } + // R-895 measurement slice: the hub's OWN count of the box's snapshot files (read-only, logged, no alarm yet). + if terr == nil { + if l, lerr := s.Reg.ListSnapshots(ctx, t, pw, time.Now().UTC()); lerr != nil { + s.logf("[WARN] offsitekeys: snapshots %s: the hub's own listing failed: %v", c.CustomerID, lerr) + } else { + s.logf("[INFO] offsitekeys: snapshots %s: the hub's own listing — %d snapshot file(s), %d other line(s), newest %s, oldest %s (R-895 measurement; no alarm yet)", + c.CustomerID, l.Files, l.Other, l.Newest.Format(time.RFC3339), l.Oldest.Format(time.RFC3339)) + } + } } return out } diff --git a/hub/internal/offsitekeys/snapshots.go b/hub/internal/offsitekeys/snapshots.go new file mode 100644 index 00000000..233283da --- /dev/null +++ b/hub/internal/offsitekeys/snapshots.go @@ -0,0 +1,82 @@ +package offsitekeys + +import ( + "context" + "fmt" + "regexp" + "strings" + "time" +) + +// R-895, THE MEASUREMENT SLICE (hub v0.157.0; design `audits/agent-allowlist-rotation-2026-10-17/P4/design-R-895.md`). +// +// The clean-up window's drop check trusts the snapshot counts the box sends. Option B of the design: the hub counts the +// box's restic `snapshots/` files ITSELF, over the login it already holds (the sub-account password, sealed — no new +// credential), and later alarms on files no box run explains. This slice only LISTS and LOGS — count, newest and +// oldest upload time — once a day beside the authorized_keys audit, so the listing's shape on the provider is measured +// before anything alarms on it. READ-ONLY: one `ls -l`; it writes, moves and deletes nothing. + +// SnapshotListing is the hub's own read of one repository's snapshots/ directory. +type SnapshotListing struct { + Files int // entries whose name is a restic snapshot id (64 hex) + Other int // lines that are not (a header, a temp file, an unparsable line) + Newest time.Time // zero when no time could be read + Oldest time.Time +} + +var snapshotID = regexp.MustCompile(`^[0-9a-f]{64}$`) + +// ListSnapshots lists /snapshots with `ls -l` in the restricted shell. now anchors a year-less date. +func (r *Registrar) ListSnapshots(ctx context.Context, t Target, password string, now time.Time) (SnapshotListing, error) { + sh, err := r.open(ctx, t, password) + if err != nil { + return SnapshotListing{}, err + } + defer sh.Close() + out, err := sh.Run(ctx, "ls -l "+strings.TrimRight(t.RepoPath, "/")+"/snapshots", nil) + if err != nil { + return SnapshotListing{}, fmt.Errorf("offsitekeys: list snapshots: %w", err) + } + return parseLsL(string(out), now), nil +} + +// parseLsL reads `ls -l` lines: mode links owner group size MON DD (HH:MM|YYYY) name. +func parseLsL(out string, now time.Time) SnapshotListing { + var l SnapshotListing + for _, line := range strings.Split(out, "\n") { + f := strings.Fields(line) + if len(f) == 0 || strings.HasPrefix(line, "total ") { + continue + } + if len(f) < 9 || !snapshotID.MatchString(f[len(f)-1]) { + l.Other++ + continue + } + l.Files++ + if ts, ok := lsTime(f[len(f)-4], f[len(f)-3], f[len(f)-2], now); ok { + if l.Newest.IsZero() || ts.After(l.Newest) { + l.Newest = ts + } + if l.Oldest.IsZero() || ts.Before(l.Oldest) { + l.Oldest = ts + } + } + } + return l +} + +// lsTime reads ls's two date forms: "Oct 17 03:12" (within six months, year implied) and "Oct 17 2025". +func lsTime(mon, day, hm string, now time.Time) (time.Time, bool) { + if strings.Contains(hm, ":") { + ts, err := time.Parse("Jan 2 15:04 2006", fmt.Sprintf("%s %s %s %d", mon, day, hm, now.Year())) + if err != nil { + return time.Time{}, false + } + if ts.After(now.Add(24 * time.Hour)) { // December's file read in January + ts = ts.AddDate(-1, 0, 0) + } + return ts, true + } + ts, err := time.Parse("Jan 2 2006", fmt.Sprintf("%s %s %s", mon, day, hm)) + return ts, err == nil +} diff --git a/hub/internal/offsitekeys/snapshots_test.go b/hub/internal/offsitekeys/snapshots_test.go new file mode 100644 index 00000000..3c2fdd31 --- /dev/null +++ b/hub/internal/offsitekeys/snapshots_test.go @@ -0,0 +1,67 @@ +package offsitekeys + +import ( + "context" + "strings" + "testing" + "time" +) + +type lsShell struct { + out string + cmds []string +} + +func (s *lsShell) Run(_ context.Context, cmd string, _ []byte) ([]byte, error) { + s.cmds = append(s.cmds, cmd) + return []byte(s.out), nil +} +func (s *lsShell) Close() error { return nil } + +type lsDialer struct{ sh *lsShell } + +func (d lsDialer) Dial(context.Context, Target, string) (Shell, error) { return d.sh, nil } + +const id1 = "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9" +const id2 = "ffeeddccbbaa99887766554433221100ffeeddccbbaa99887766554433221100" + +// R-895 measurement slice: the hub's own listing reads the snapshot files and their upload times from `ls -l`, counts +// only restic snapshot ids, and issues exactly ONE read-only command. RED-PROOF +// (audits/agent-allowlist-rotation-2026-10-17/P4/redproof-listing.txt): with parseLsL counting every line, Files is 3 +// and this test fails. +func TestListSnapshots_CountsTheBoxSnapshotFilesReadOnly(t *testing.T) { + sh := &lsShell{out: "total 8\n" + + "-rw------- 1 u1-sub4 u1-sub4 324 Oct 17 03:12 " + id1 + "\n" + + "-rw------- 1 u1-sub4 u1-sub4 331 Mar 2 2026 " + id2 + "\n" + + "-rw------- 1 u1-sub4 u1-sub4 12 Oct 17 03:13 .tmp-upload\n"} + r := &Registrar{Dialer: lsDialer{sh}} + now := time.Date(2026, 10, 17, 14, 0, 0, 0, time.UTC) + l, err := r.ListSnapshots(context.Background(), tgt, "pw", now) + if err != nil { + t.Fatal(err) + } + if l.Files != 2 || l.Other != 1 { + t.Fatalf("files=%d other=%d, want 2 and 1", l.Files, l.Other) + } + if !l.Newest.Equal(time.Date(2026, 10, 17, 3, 12, 0, 0, time.UTC)) || !l.Oldest.Equal(time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC)) { + t.Fatalf("newest=%v oldest=%v", l.Newest, l.Oldest) + } + if len(sh.cmds) != 1 || sh.cmds[0] != "ls -l /home/felhom-repo/snapshots" { + t.Fatalf("commands = %v — the listing must be one read-only ls", sh.cmds) + } + for _, c := range sh.cmds { + for _, w := range []string{"rm", "mv", "dd", "chmod", "mkdir", "touch", "cp", "unlink"} { + if strings.HasPrefix(c, w+" ") { + t.Fatalf("a write verb in the listing: %q", c) + } + } + } +} + +// A December file read in January is last year's, not next year's. +func TestLsTime_YearBoundary(t *testing.T) { + ts, ok := lsTime("Dec", "30", "23:10", time.Date(2027, 1, 2, 0, 0, 0, 0, time.UTC)) + if !ok || ts.Year() != 2026 { + t.Fatalf("got %v %v", ts, ok) + } +} diff --git a/hub/internal/store/host_recovery.go b/hub/internal/store/host_recovery.go index f5a659b6..dee9fba1 100644 --- a/hub/internal/store/host_recovery.go +++ b/hub/internal/store/host_recovery.go @@ -18,7 +18,9 @@ type HostRecoveryCredential struct { } // SaveHostRecoveryCredential upserts a host's break-glass credential (last-write-wins: day-0 sets it, -// --rotate re-sets). R-133 (hub v0.135.0): the secret is SEALED at rest with the same key and the same +// --rotate re-sets). set_at is the time THIS secret was vaulted — a re-vault moves it (R-954 follow-up, hub v0.157.0: +// it used to keep the FIRST vault's time, so the Console access card said „last vaulted 2026-07-21" over a password +// rotated minutes before; pinned by TestHostRecovery_ReVaultMovesSetAt). R-133 (hub v0.135.0): the secret is SEALED at rest with the same key and the same // helpers as the off-site sub-account passwords (offsite_seal.go, OFFSITE_SECRET_KEY) — a copy of hub.db // alone no longer holds any box's console password. No key → the save is REFUSED (ErrNoSealKey): a hub that // cannot seal must not fall back to plaintext. The hub NEVER logs the secret and only ever returns it over @@ -34,6 +36,7 @@ func (s *Store) SaveHostRecoveryCredential(hostID, username, secret string) erro ON CONFLICT(host_id) DO UPDATE SET username = excluded.username, secret = excluded.secret, + set_at = datetime('now'), updated_at = datetime('now')`, hostID, username, sealed) return err diff --git a/hub/internal/store/r954_revault_set_at_test.go b/hub/internal/store/r954_revault_set_at_test.go new file mode 100644 index 00000000..32b55b5c --- /dev/null +++ b/hub/internal/store/r954_revault_set_at_test.go @@ -0,0 +1,37 @@ +package store + +import ( + "testing" + "time" +) + +// R-954 follow-up (hub v0.157.0): a re-vault (`--rotate-recovery`, the operator's own set-and-vault) MOVES set_at — the +// time the Console access card shows as „last vaulted". Found live 2026-10-17: demo-hp's and Tester 1's rotated +// passwords both still read set_at = their FIRST vault (2026-07-21, 2026-10-04). RED-PROOF +// (felhom.eu audits/agent-allowlist-rotation-2026-10-17/P3/redproof-set-at.txt): without `set_at = datetime('now')` in +// the upsert this test fails on both the meta and the credential read. +func TestHostRecovery_ReVaultMovesSetAt(t *testing.T) { + st := sealTestStore(t) + if err := st.SaveHostRecoveryCredential("h1", "root@pam", "first-Pw-1"); err != nil { + t.Fatal(err) + } + // Age the first vault by a year, as a real day-0 vault would be. + if _, err := st.db.Exec(`UPDATE host_recovery SET set_at = '2025-10-17 10:00:00' WHERE host_id = 'h1'`); err != nil { + t.Fatal(err) + } + before := time.Now().UTC().Add(-time.Minute) + if err := st.SaveHostRecoveryCredential("h1", "root@pam", "second-Pw-2"); err != nil { + t.Fatal(err) + } + c, err := st.GetHostRecoveryCredential("h1") + if err != nil || c == nil || c.Secret != "second-Pw-2" { + t.Fatalf("credential = %+v, %v", c, err) + } + if c.SetAt.Before(before) { + t.Fatalf("credential set_at = %v — the re-vault kept the first vault's time", c.SetAt) + } + m, err := st.GetHostRecoveryMeta("h1") + if err != nil || m == nil || m.SetAt.Before(before) { + t.Fatalf("meta set_at = %+v, %v — the card would show the first vault's time", m, err) + } +} diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 5a8aa1ef..98a579fa 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,10 @@ +## 2026-10-17 — the uninstall removes agent v0.159.0's new root helper (R-1012); not yet tagged + +`felhom-host-install.sh --uninstall` removes `/usr/local/sbin/felhom-thin-check`, the read-only orphan-thin check the +agent's config bundle installs since v0.159.0; `test_hostinstall.py` names it among the bundle files (its +`test_bundle_list_is_current` fails when the bundle gains a file the uninstall does not know). Pushing to `main` +publishes nothing — this rides the next `installer-v…` tag. SCRIPT_VERSION unchanged. + ## 2026-10-17 — `~/.config/credentials` deleted (R-1134) The operator's go, after every key in it was checked equal in the CC store. `read_credential.py`, `fb_probe.py` and diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh index dbca22d6..2345a7ae 100644 --- a/scripts/felhom-host-install.sh +++ b/scripts/felhom-host-install.sh @@ -1357,6 +1357,8 @@ run_uninstall() { if [[ -f /usr/local/sbin/felhom-backup-target-apply ]]; then run rm -f /usr/local/sbin/felhom-backup-target-apply; else log_skip " felhom-backup-target-apply already absent"; fi # agent v0.156.1 (R-1006): the dead half-written vzdump remover the config bundle installs. if [[ -f /usr/local/sbin/felhom-vzdump-partial-clean ]]; then run rm -f /usr/local/sbin/felhom-vzdump-partial-clean; else log_skip " felhom-vzdump-partial-clean already absent"; fi + # agent v0.159.0 (R-1012): the read-only orphan-thin check the config bundle installs. + if [[ -f /usr/local/sbin/felhom-thin-check ]]; then run rm -f /usr/local/sbin/felhom-thin-check; else log_skip " felhom-thin-check already absent"; fi # 1.33.0 (`16` §5): the internal-place helper. Its VOLUME (pve/felhom-belso) holds household data and is KEPT. if [[ -f /usr/local/sbin/felhom-internal-place ]]; then run rm -f /usr/local/sbin/felhom-internal-place; else log_skip " felhom-internal-place already absent"; fi if [[ -f /usr/local/sbin/felhom-os-apply ]]; then run rm -f /usr/local/sbin/felhom-os-apply; else log_skip " felhom-os-apply already absent"; fi diff --git a/scripts/test_hostinstall.py b/scripts/test_hostinstall.py index 18426e7d..3f7088a6 100644 --- a/scripts/test_hostinstall.py +++ b/scripts/test_hostinstall.py @@ -36,6 +36,7 @@ BUNDLE_DESTS = [ "/usr/local/sbin/felhom-pbs-apply", "/usr/local/sbin/felhom-backup-target-apply", "/usr/local/sbin/felhom-vzdump-partial-clean", # agent v0.156.1 (R-1006) + "/usr/local/sbin/felhom-thin-check", # agent v0.159.0 (R-1012) "/usr/local/sbin/felhom-internal-place", # agent v0.158.0 (felhom.eu `16` §5) "/usr/local/sbin/felhom-os-apply", "/usr/local/sbin/felhom-crash-guard",