From b52cb6ec9470a37603b0b9ca3634cd5579b1d69a Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 27 Sep 2026 14:15:01 +0200 Subject: [PATCH] version-travel D1: agent 0.136.0 + 0.137.0 found and fixed on the box; records updated Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- STATUS.md | 6 +++--- .../audits/DRILL-version-travel-2026-09-26.md | 11 +++++++---- .../D1/D1-selftest-due-0137-pre.txt | 10 ++++++++++ .../D1/D1-selftest-due-after-0136.txt | 11 +++++++++++ .../D1/D1-selftest-due-after-0137.txt | 6 ++++++ .../D1/D1-selftest-due-demo-hp.txt | 9 +++++++++ .../D1/D1-signed-delivery.txt | 11 +++++++++++ .../version-travel-2026-09-26/D1/README.md | 18 +++++++++++++----- .../D1/RP-r689-acl.txt | 6 ++++++ .../D1/RP-r689-deleted-guest.txt | 6 ++++++ documentation/backlog/CLOSED-ITEMS.md | 2 +- 11 files changed, 83 insertions(+), 13 deletions(-) create mode 100644 documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-0137-pre.txt create mode 100644 documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-after-0136.txt create mode 100644 documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-after-0137.txt create mode 100644 documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-demo-hp.txt create mode 100644 documentation/audits/version-travel-2026-09-26/D1/RP-r689-acl.txt create mode 100644 documentation/audits/version-travel-2026-09-26/D1/RP-r689-deleted-guest.txt diff --git a/STATUS.md b/STATUS.md index 25d64d4c..07850f67 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-09-27. Both demo boxes run controller 0.275.0 and host agent 0.135.0. Hub 0.125.0.** +**Updated 2026-09-27. Both demo boxes run controller 0.275.0 and host agent 0.137.0. Hub 0.125.0.** **Decisions I took on my own** (you may reverse each): 1. **paperless-ngx goes to PostgreSQL 18, tandoor to 17.** Each follows what its own makers ship. paperless's makers use 18. tandoor's use 16, and 17 is the newest its framework supports. @@ -11,7 +11,7 @@ - **The box tested this by itself in the night.** It updated docmost's database at 02:15. Two minutes later the backup kept the matching old settings. The restore the next morning worked. - **Two more apps move to a new database version: paperless-ngx and tandoor.** Each was tested twice: on a throwaway test machine and on the scratch box through the normal update. Every account came back. - **adventurelog moves to its new version.** It keeps its world-map download. If that file was cut off, the box sets it aside and downloads it again. Tested both ways. -- **The HP box's restore test now tests real backups only**, not the golden template file. +- **The HP box's restore test now tests real backups of guests that exist**, not the golden template file and not an old backup of a deleted guest (two more small agent releases, each found when I checked the box). - **Small fixes:** the file browser no longer re-creates an empty kept folder; the "Kept data" name follows the box's language; after a load, the app page no longer shows a password that does not work. **What broke, or is not done.** @@ -32,6 +32,6 @@ **What needs you.** 1. **D4** above. 2. **The weekly golden bake is due** (last one a week ago, 21 controller releases behind). The brief said no golden, so I renewed the waiver for 7 days only (to 4 October). If you do nothing, the checks go red again on 4 October and nothing can be pushed to felhom.eu until a bake or a new waiver. -3. **Vouch agent 0.135.0** for new installs (hub → Configs → Day-0 artifacts). If you do nothing, new boxes install 0.134.0 and get 0.135.0 only by a signed update. +3. **Vouch agent 0.137.0** for new installs (hub → Configs → Day-0 artifacts). If you do nothing, new boxes install 0.134.0 and get the newer one only by a signed update. 4. **The image-copy question** (a maker deletes an old version): keep as is, or copy installed versions into our registry. If you do nothing, nothing changes. 5. **From before:** Peti's box in the Claude project text; Cloudflare leftovers of Peti's domain; the old Storage Box `PBS-storage-1`. If you do nothing, they stay as they are. diff --git a/documentation/audits/DRILL-version-travel-2026-09-26.md b/documentation/audits/DRILL-version-travel-2026-09-26.md index 2b9f8682..e0e4a13d 100644 --- a/documentation/audits/DRILL-version-travel-2026-09-26.md +++ b/documentation/audits/DRILL-version-travel-2026-09-26.md @@ -17,8 +17,11 @@ decisions 13–42, §5.3, §6.4 parts 4, 5, 10. Evidence: `audits/version-travel (measured). It rides the image move in the same commit, as asked. - **Off-site restore NOT exercised live** (9202 has no off-site target; the boxes that do are fenced). Unit-tested. - **The conversion-copy RELEASE on a real 18 dump not observed** — due on 9202 after tonight's 00:30 data run. -- **D1's scheduled restore-test cycle on demo-hp**: due ~18:28 CEST, read into `D1/D1-cycle-demo-hp.txt` if the session is - still alive; otherwise it is the first thing to read. +- **D1 took THREE agent releases.** Instead of waiting 6 h for the scheduled cycle, the scheduler's own read-only verdict + (`-selftest=restore-test-due`) was read on demo-hp: the golden was skipped, but the pick fell to a leftover archive of + guest 9100, deleted in August. Fixed in **v0.136.0** (the guest must exist) — whose lookup then met PVE's 403 for a guest outside the agent's pool and made the local + tier UNKNOWN (my regression, read the same way); fixed in **v0.137.0**, verified read-only on demo-hp with the pre-release + binary before the release. Each red-proofed and signed-delivered to both hosts. - **Part E (night watch) not done.** - D4 is **not ruled**: option A is built and in force (STATUS). @@ -47,7 +50,7 @@ whole at 0.95.0, "No pending database migrations"; climb 46 s). A7: 42 of 42 lad ## B, C, D B: `B/README.md` — paperless-ngx → 18 and tandoor → 17, both venues, engine gate ALLOWED, catalog `46affe0`, `53b84ca`. -C: `C/README.md` — adventurelog v0.13.0, catalog `06ea7da`. D1 agent v0.135.0 (signed delivery to both hosts); D2 R-695; +C: `C/README.md` — adventurelog v0.13.0, catalog `06ea7da`. D1 agent v0.135.0 → v0.137.0 (signed delivery to both hosts); D2 R-695; D3 R-691 (1); D4 R-694 (per-app measurement, 6 of 7 keep the login in their data). ## Teardown @@ -57,7 +60,7 @@ D3 R-691 (1); D4 R-694 (per-app measurement, 6 of 7 keep the login in their data 16 copy waiting for the release; back on the LIVE catalog (`controller.yaml.pre-vt0926`), controller 0.275.0. Drill repo reset to live `main` (`f1a7d6c`). - **Host:** demo-hp `pct list` 9201 + 9202 only; `local` and `nvme-scratch` as before the bench. -- **Hub:** global floor 0.275.0 (declared MinAgent 0.131.0); two signed `agent_update` jobs consumed; no customer or +- **Hub:** global floor 0.275.0 (declared MinAgent 0.131.0); six signed `agent_update` jobs (0.135.0, 0.136.0, 0.137.0 × 2 hosts); no customer or appliance created. ## Register diff --git a/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-0137-pre.txt b/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-0137-pre.txt new file mode 100644 index 00000000..398f0fef --- /dev/null +++ b/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-0137-pre.txt @@ -0,0 +1,10 @@ +eval_interval=6h0m0s settle=24h0m0s +time=2026-09-27T14:00:31.969+02:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/felhom-golden-0.236.0.tar.zst size_bytes=654115664 reason="not a backup of a guest (the storage reports no vmid)" +time=2026-09-27T14:00:31.975+02:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst size_bytes=656970239 reason="guest 9100 does not exist on this node or is not one this agent manages" +tier=felhom-pbs due=true archive="felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z" landed=2026-09-24T20:06:25Z proven="felhom-pbs:backup/ct/9201/2026-09-22T04:12:45Z" + reason: newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different one) +tier=local due=false archive="" landed=- proven="local:backup/vzdump-lxc-9201-2026_09_22-06_52_48.tar.zst" + reason: no settled archive yet — nothing to prove (newborn or still settling) +cost tier=felhom-pbs one_lookup=288ms +cost tier=local one_lookup=44ms +cost all_tiers=332ms diff --git a/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-after-0136.txt b/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-after-0136.txt new file mode 100644 index 00000000..d007225e --- /dev/null +++ b/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-after-0136.txt @@ -0,0 +1,11 @@ +felhom-agent 0.136.0 +eval_interval=6h0m0s settle=24h0m0s +time=2026-09-27T13:59:47.174+02:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/felhom-golden-0.236.0.tar.zst size_bytes=654115664 reason="not a backup of a guest (the storage reports no vmid)" +tier=felhom-pbs due=true archive="felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z" landed=2026-09-24T20:06:25Z proven="felhom-pbs:backup/ct/9201/2026-09-22T04:12:45Z" + reason: newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different one) +tier=local due=false archive="" landed=- proven="local:backup/vzdump-lxc-9201-2026_09_22-06_52_48.tar.zst" + reason: candidate lookup FAILED (checking whether guest 9100 still exists: proxmox: GET /nodes/demo-hp/lxc/9100/config -> HTTP 403: permission denied at /vms/9100 (missing privilege VM.Audit)) — tier is unknown this evaluation, not proven and not dismissed + ERROR: checking whether guest 9100 still exists: proxmox: GET /nodes/demo-hp/lxc/9100/config -> HTTP 403: permission denied at /vms/9100 (missing privilege VM.Audit) +cost tier=felhom-pbs one_lookup=298ms +cost tier=local one_lookup=38ms +cost all_tiers=342ms diff --git a/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-after-0137.txt b/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-after-0137.txt new file mode 100644 index 00000000..f6dbabd7 --- /dev/null +++ b/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-after-0137.txt @@ -0,0 +1,6 @@ +time=2026-09-27T14:14:33.011+02:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/felhom-golden-0.236.0.tar.zst size_bytes=654115664 reason="not a backup of a guest (the storage reports no v +time=2026-09-27T14:14:33.017+02:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst size_bytes=656970239 reason="guest 9100 does not exist on this n +tier=felhom-pbs due=true archive="felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z" landed=2026-09-24T20:06:25Z proven="felhom-pbs:backup/ct/9201/2026-09-22T04:12:45Z" + reason: newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different one) +tier=local due=false archive="" landed=- proven="local:backup/vzdump-lxc-9201-2026_09_22-06_52_48.tar.zst" + reason: no settled archive yet — nothing to prove (newborn or still settling) diff --git a/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-demo-hp.txt b/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-demo-hp.txt new file mode 100644 index 00000000..2958fecb --- /dev/null +++ b/documentation/audits/version-travel-2026-09-26/D1/D1-selftest-due-demo-hp.txt @@ -0,0 +1,9 @@ +eval_interval=6h0m0s settle=24h0m0s +time=2026-09-27T13:44:23.229+02:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/felhom-golden-0.236.0.tar.zst size_bytes=654115664 reason="not a backup of a guest (the storage reports no vmid)" +tier=felhom-pbs due=true archive="felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z" landed=2026-09-24T20:06:25Z proven="felhom-pbs:backup/ct/9201/2026-09-22T04:12:45Z" + reason: newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different one) +tier=local due=true archive="local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst" landed=2026-08-21T15:59:15Z proven="local:backup/vzdump-lxc-9201-2026_09_22-06_52_48.tar.zst" + reason: newest settled archive (landed 2026-08-21T15:59:15Z) has not been proven (last proven archive was a different one) +cost tier=felhom-pbs one_lookup=295ms +cost tier=local one_lookup=33ms +cost all_tiers=332ms diff --git a/documentation/audits/version-travel-2026-09-26/D1/D1-signed-delivery.txt b/documentation/audits/version-travel-2026-09-26/D1/D1-signed-delivery.txt index 01887b19..302b85dd 100644 --- a/documentation/audits/version-travel-2026-09-26/D1/D1-signed-delivery.txt +++ b/documentation/audits/version-travel-2026-09-26/D1/D1-signed-delivery.txt @@ -15,3 +15,14 @@ Sep 27 12:27:58 demo-hp felhom-agent[995644]: time=2026-09-27T12:27:58.210+02:00 Sep 27 12:28:02 demo-hp felhom-agent[998480]: time=2026-09-27T12:28:02.239+02:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.135.0 prev=0.134.0 dwell=1m0s Sep 27 12:29:02 demo-hp sudo[1003468]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-selfupdate-guarded commit Sep 27 12:29:02 demo-hp felhom-agent[998480]: time=2026-09-27T12:29:02.260+02:00 level=WARN msg="selfupdate: update committed" version=0.135.0 wrapper="" +== 0.136.0 demo-hp-bb76ea 2026-09-27T11:46:11Z +uploaded signed op to the hub jobs queue +== 0.136.0 demo-felhom-8363b5 2026-09-27T11:46:11Z +uploaded signed op to the hub jobs queue +== demo-hp 11:59:28 felhom-agent 0.136.0 Sep 27 13:59:10 demo-hp felhom-agent[1319775]: time=2026-09-27T13:59:10.428+02:00 level=WARN msg="selfupdate: update committed" version=0.136.0 wrapper="" +== felhom-pve 11:59:28 felhom-agent 0.136.0 Sep 27 13:53:01 demo-felhom felhom-agent[2865091]: time=2026-09-27T13:53:01.204+02:00 level=WARN msg="selfupdate: update committed" version=0.136.0 wrapper="" +== 0.137.0 demo-hp-bb76ea 2026-09-27T12:01:26Z +uploaded signed op to the hub jobs queue +== 0.137.0 demo-felhom-8363b5 2026-09-27T12:01:26Z +uploaded signed op to the hub jobs queue +12:14:32 demo-hp: felhom-agent 0.137.0 committed=1 | demo-felhom: felhom-agent 0.137.0 committed=1 diff --git a/documentation/audits/version-travel-2026-09-26/D1/README.md b/documentation/audits/version-travel-2026-09-26/D1/README.md index 8dd642d3..8625074f 100644 --- a/documentation/audits/version-travel-2026-09-26/D1/README.md +++ b/documentation/audits/version-travel-2026-09-26/D1/README.md @@ -1,6 +1,14 @@ -# D1 — R-689, agent v0.135.0 +# D1 — R-689, agent v0.135.0 → v0.137.0 -Fix: `backup/runner.go` `guestBackupArchive`. Red-proof `RP-r689-golden-picked.txt`. Released by `scripts/release-agent.sh` -(tag `v0.135.0` at `d4be12c`, sha256 `ad4e75f1…fd4c3`, verified by download; NOT vouched). Delivered by signed `agent_update` -to both demo hosts (`D1-signed-delivery.txt`): demo-felhom committed 12:22:53 CEST, demo-hp 12:29:02 CEST. The first -scheduled restore-test evaluation on demo-hp after the update is due ~18:28 CEST: `D1-cycle-demo-hp.txt` (read later). +Instead of waiting 6 h for the scheduled cycle, the scheduler's own read-only verdict (`felhom-agent +-selftest=restore-test-due`, the daemon's construction path) was read on demo-hp after each release. + +| release | what the verdict showed on demo-hp | file | +|---|---|---| +| 0.135.0 (`d4be12c`) — only `vzdump--` / PBS `ct/…` or `vm/…` with a vmid | the golden SKIPPED; the local pick fell to `vzdump-lxc-9100-2026_08_21…`, a guest deleted in August | `D1-selftest-due-demo-hp.txt` | +| 0.136.0 (`16dbc83`) — the guest must exist (`GuestConfig`) | PVE answered **403 permission denied** (9100 is outside the agent's pool), read as a failure → local tier UNKNOWN | `D1-selftest-due-after-0136.txt` | +| 0.137.0 (`3ef095f`) — 403 = not a guest this agent manages | both leftovers skipped; off-site tier due on 9201; local tier waits for today's 9201 archive to settle (~04:43 tomorrow) | `D1-selftest-due-0137-pre.txt` (pre-release binary), `D1-selftest-due-after-0137.txt` | + +Red-proofs: `RP-r689-golden-picked.txt`, `RP-r689-deleted-guest.txt`, `RP-r689-acl.txt`. Each released by +`scripts/release-agent.sh`, NOT vouched, and signed-delivered to both hosts (`D1-signed-delivery.txt`). Not observed: an +actual scheduled restore-test run (read-only verdicts only). diff --git a/documentation/audits/version-travel-2026-09-26/D1/RP-r689-acl.txt b/documentation/audits/version-travel-2026-09-26/D1/RP-r689-acl.txt new file mode 100644 index 00000000..5b5cafeb --- /dev/null +++ b/documentation/audits/version-travel-2026-09-26/D1/RP-r689-acl.txt @@ -0,0 +1,6 @@ +# R-689 red-proof 3 — 403 treated as a lookup failure (v0.136.0) + r689_golden_test.go:110: picked "" err=checking whether guest 9100 still exists: proxmox: GET /nodes/n/lxc/9100/config -> HTTP 403: permission denied at /vms/9100 (missing privilege VM.Audit) — want nothing and no error (the only settled archive is not ours) +--- FAIL: TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-agent/internal/backup 0.008s +FAIL diff --git a/documentation/audits/version-travel-2026-09-26/D1/RP-r689-deleted-guest.txt b/documentation/audits/version-travel-2026-09-26/D1/RP-r689-deleted-guest.txt new file mode 100644 index 00000000..24642471 --- /dev/null +++ b/documentation/audits/version-travel-2026-09-26/D1/RP-r689-deleted-guest.txt @@ -0,0 +1,6 @@ +# R-689 red-proof 2 — without the known-guest check (v0.135.0) + r689_golden_test.go:75: picked "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst" err= — a deleted guest's archive proves nothing about this box +--- FAIL: TestR689_AnArchiveOfADeletedGuestIsNeverPicked (0.00s) +FAIL +FAIL gitea.dooplex.hu/admin/felhom-agent/internal/backup 0.008s +FAIL diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 0ef7f8c9..5a3be786 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -417,5 +417,5 @@ Full original text of each row: `git show ^: | **R-699** | **The update's precondition accepted a unit holding no data (a just-installed app's) as a fresh copy (P2).** Found live on 9202 this session (tandoor). v0.275.0: such a unit stays listed, never a precondition copy on Tier 1/2; the update backs up first (seen live: adventurelog's fresh install got `backing-up`). | CLOSED 2026-09-27 — controller v0.275.0 | filed and closed this session; `A5-redproofs/RP6-*` | | **R-695** | **Two kept-data Deletes in one second could leave a self-perpetuating empty kept folder (P3).** v0.275.0: the file-browser sync is single-flight; an empty dated kept folder is never listed or bound. Red-proofed. | CLOSED 2026-09-27 — controller v0.275.0 | `audits/version-travel-2026-09-26/D2/` | | **R-694** | **A load/restore regenerated a withheld login and the page showed it as the password (P3).** Measured per app from each entrypoint: 6 of 7 keep the login in their data (code-server is the exception). v0.275.0: `restored_logins`; the page shows no value and says to use the password valid at the backup. | CLOSED 2026-09-27 — controller v0.275.0 | `audits/version-travel-2026-09-26/D4/` | -| **R-689** | **demo-hp's restore test picked the golden template in `local:backup/` and failed every 6 h (P3).** Agent v0.135.0: only `vzdump--` files or PBS `ct/…` or `vm/…` snapshots with a reported vmid. Delivered to both demo hosts by signed `agent_update`. | CLOSED 2026-09-27 — agent v0.135.0 | `audits/version-travel-2026-09-26/D1/` | +| **R-689** | **demo-hp's restore test picked the golden template in `local:backup/` and failed every 6 h (P3).** Agent v0.135.0 + v0.136.0 + v0.137.0: only `vzdump--` files or PBS `ct/…` or `vm/…` snapshots with a reported vmid, of a guest that still EXISTS on the node (v0.136.0 — the first half, read live with `-selftest=restore-test-due`, fell to a leftover archive of a guest deleted in August; v0.137.0 — PVE answers 403, not "does not exist", for a guest outside the agent's pool, which 0.136.0 read as a lookup failure). Delivered to both demo hosts by signed `agent_update`. | CLOSED 2026-09-27 — agent v0.137.0 | `audits/version-travel-2026-09-26/D1/` | | **R-655** | **adventurelog v0.13.0 could not become healthy in the catalog's template (P2).** Operator ruling decision 41 (keep the download). Catalog `06ea7da`: the frontend override dropped; a cut-off world-data file set aside before start (a pending import crash-looped 10× without it). Proven on the bench and on 9202 (204 s under the default 5-min wait). | CLOSED 2026-09-27 — catalog `06ea7da` | `audits/version-travel-2026-09-26/C/` |