Release 2026-11-03 close-out: every box on 0.350.0 with Traefik v3.7.17; golden 0.350.0 + agent 0.172.0 vouched; hub 0.179.0; catalog f4c5c79

Closed: R-1354, R-1347, R-1348, R-1339, R-1351, R-1323, R-1302, R-1352, R-1331, R-1356, R-1359 (235 -> 224). R-548 and R-1361 carry today's readings.
2026-11-03 10:26:23 +01:00
1 parent 01a44d5626
commit a7e86fd273
@@ -0,0 +1,7 @@
+cache HEAD f4c5c79 Four big steps on the operator's yes: paperless-ngx 3.4.1 (felhom.eu R-
+2026/11/03 08:06:50 [INFO] [sync] Updated termix/.felhom.yml
+2026/11/03 08:06:50 [INFO] [sync] catalog moved c760fca93898 -> f4c5c797d9d0 (0 new, 4 stack dir(s) changed)
+2026/11/03 08:06:51 [INFO] [sync] Catalog sync complete
+
+paperless-ngx {'hu': [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}], 'en': [{'title': 'A newer version of this app is available. Select 
+bookstack 
@@ -0,0 +1,25 @@
+# Part 2 — golden 0.350.0, one fresh install, 2026-11-03
+
+The bake: `documentation/tests/golden-0.350.0-2026-11-03/`. VM 9383 `release1103-box` on demo-hp (6 cores, 10 GiB, raw on `nvme-scratch`, never the thin pool), from the public ISO 1.30.0 (sha256 = the published one), graphical installer driven by `qm sendkey` (`install/`).
+
+**The path, exactly.** A hub-offered install refuses any golden but the vouched one (R-297), and the brief wanted the proof before the vouch. So: the public installer `felhom-host-install.sh` 1.36.1, fetched from felhom.eu (sha `e18cdd17…`), with **one added line** (`installer-the-one-line.diff.txt`) that replaces the manifest's agent + golden with 0.172.0 + 0.350.0, run by hand as root in the DIRECT shape of `felhom-bootstrap.sh` (`--customer-id Tester4 --mode appliance --hub-url … --passphrase-file …`), not through the bind page. Everything else is the real path: the hub's enroll, the fetch from Gitea, the sha check, the provision.
+
+**Three tries, said plainly.**
+1. A unit drop-in set `FELHOM_INSTALL_URL`, but the bootstrap blanks that variable and reads only `/etc/felhom/bootstrap.env`, so the box bound (bind page, 07:30:55 UTC) and installed the VOUCHED pair: golden 0.344.0, agent 0.171.0, moved by the floor to 0.350.0 about 2 minutes later (`first-run-vouched-manifest-journal.txt`). It proves the normal path and the floor, not the new golden. CC's mistake; read only at the end. Removed with `--uninstall` (`uninstall-first-run.txt`; the typed confirmation fed through `ssh -tt` on CC's own test VM).
+2. The hub refused the reinstall: 12 tries (CC expected a 45-minute limit; there is none: once a controller has reported, the passphrase stays closed until the operator deletes the host, R-1159). The operator deleted the host at 09:08:12 UTC.
+3. `install-log.txt`: day-0 SUCCESS 09:11:41 UTC, host `Tester4-e286d3`.
+
+| Check | Result (`end-state.txt`, `claim.txt`, `controller.log.gz`, `agent-journal.log.gz`) |
+|---|---|
+| golden on the box | sha256 `082e2e04…` = the bake and the registry |
+| controller | 0.350.0 from the golden: `controller_started` with no `controller_updated`; 0 ERROR lines |
+| Traefik | v3.7.17, image id `a72cb2f4…`; no v3.7.16 image on the box |
+| hub | enrolled, reports, events arrive |
+| claim | the mailed setup code, 302 `/` (the box then offers the recovery page for Tester4's sealed package of 2026-10-24; not entered) |
+| one app | BentoPDF: deploy 202, running, healthy, route 200 |
+
+**The hub's own acts on this customer:** the fresh installs made the hub re-issue Tester4's PBS token on ep0 (`pbsdr F-14 auto re-issue`, gen 2) and its off-site key, twice. That is the product's normal hands-free cascade; CC touched nothing on ep0 by hand.
+
+**Teardown** (`teardown.txt`): VM 9383 destroyed with its disks; the typed passwords shredded. **Left for the operator:** delete host `Tester4-e286d3` on the Hosts page, from about 11:00 CET (the 45-minute rule).
+
+**Vouch:** the operator, 10:18:10 CET: golden 0.350.0, agent 0.172.0, bundle `cc34b0bc…`, min_agent 0.158.1 (`vouch-readback.txt`: the hub's log and its database with the WAL agree).
@@ -0,0 +1,12 @@
+GET /claim HTTP/2 200 
+csrf_len 64 cookie_present yes
+POST /claim HTTP/2 302  location: /
+deploy 403
+deploy 202
+running True {'bentopdf': 'ghcr.io/alam00000/bentopdf:v2.8.10'}
+bentopdf ghcr.io/alam00000/bentopdf:v2.8.10 Up 18 seconds (healthy)
+filebrowser gtstef/filebrowser:1.5.8-stable Up 3 minutes (healthy)
+felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0 Up 3 minutes (healthy)
+traefik traefik:v3.7.17 Up 3 minutes
+pdf route 200
+Host(`pdf.tester4.test`)
@@ -0,0 +1,12 @@
+== ends 2026-11-03T09:14:41Z
+gitea.dooplex.hu/admin/felhom-controller:0.350.0
+bentopdf ghcr.io/alam00000/bentopdf:v2.8.10 Up 37 seconds (healthy)
+filebrowser gtstef/filebrowser:1.5.8-stable Up 3 minutes (healthy)
+felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0 Up 3 minutes (healthy)
+traefik traefik:v3.7.17 Up 3 minutes
+felhom-agent 0.172.0
+082e2e040de5615300da3ec8bc48fe6761ee0e6375dcc732e5cba82086e5e06b
+2026/11/03 10:10:51 [INFO] Event from Tester4: controller_started (info) — Controller elindult (0.350.0)
+2026/11/03 10:11:14 [INFO] Event from Tester4: controller_started (info) — Controller elindult (0.350.0)
+2026/11/03 10:13:51 [INFO] Event from Tester4: app_deploy_started (info) — Alkalmazás telepítése elindult: BentoPDF
+2026/11/03 10:14:04 [INFO] Event from Tester4: app_deployed (info) — Alkalmazás telepítve: BentoPDF
@@ -0,0 +1,145 @@
+[INFO]   auto-sized guest RAM: 5847 MiB (host 9943 MiB; clamp(host-4096, min 4096, max host-2048), ceiling host-1024)
+[INFO]   auto-sized guest cores: 5 (host 6 cores; host-1, min 2)
+
+[INFO] felhom-host-install v1.36.1 — mode=appliance customer=Tester4 vmid=9201
+
+[STEP] 1/8 pre-flight
+[INFO]   pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
+[INFO]   node: box1103 (auto)
+[INFO]   agent config: /etc/felhom-agent/agent.json
+[INFO]   agent: not installed yet — will be fetched + installed in step 5/8
+[INFO]   local-lvm free: ~19 GiB
+[WARN] local-lvm free ~19 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately
+[INFO]   free RAM: ~8151 MiB
+[INFO]   existing guests on this host: 0 (pct+qm)
+[INFO]   archive storage 'local' present
+[INFO]   acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately
+[INFO]   dnsmasq: not present before Felhom — recorded; uninstall will remove it again if we install it
+[INFO]   hub reachable (https://hub.felhom.eu)
+[OK]   customer 'Tester4' exists + passphrase valid
+[INFO]   golden: none local — will fetch + verify from Gitea in step 7/8
+[OK] pre-flight passed
+[STEP] 2/8 Proxmox API token
+[OK]   token minted (secret captured, not logged)
+[OK]   scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
+[SKIP]   old broad role FelhomAgent already absent
+[STEP] 3/8 compute volume grows
+[INFO]   auto-computed from a 19 GiB thin pool (16 §7: 50/100/130 GiB)
+[INFO]   grows: rootfs +0G (->32G), data +26G (->50G, ONE volume)
+[STEP] 4/8 host enrollment (POST /host-enroll)
+[OK]   host MINTED (first enroll)
+[INFO]   host_id: Tester4-e286d3 (api_key captured, not logged)
+[STEP] 4b/8 break-glass credential (root@pam console password → hub vault)
+[INFO]   recovery password vaulted at the hub (stored) — now setting it on root@pam
+[OK]   root@pam password vaulted, set, and the PVE login takes it (retrieve via the operator /admin path; never logged here)
+[WARN]   NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006).
+[WARN]         Retrieve the new one at hub → host page (vaulted recovery credential).
+[STEP] 5/8 agent install (fetch + verify + install)
+[OK]   apt repos aligned to no-subscription (already aligned; apt-get update OK)
+[WARN] TEST 2026-11-03: the manifest's agent + golden replaced by the unvouched 0.172.0 + 0.350.0
+[WARN]   no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
+[INFO]   manifest: agent v0.172.0 (sha 2c0222cd01713f65…), golden v0.350.0
+[INFO]   fetching agent binary v0.172.0 from Gitea …
+[OK]   verified sha256 2c0222cd01713f65… matches the hub manifest
+[OK]   installed /usr/local/bin/felhom-agent (felhom-agent 0.172.0)
+[OK]   created service user felhom-agent
+[OK]   added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
+[OK]   verified sha256 cc34b0bc3346ffd6… matches the hub manifest
+  os-apply: BUNDLE START agent=0.172.0 sha=cc34b0bc3346ffd6 authority=installer files=32 write=31 same=1 kept=0 skipped=0
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new)
+  os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new)
+  os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new)
+  os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new)
+  os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new)
+  os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new)
+  os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-sshd.service (new)
+  os-apply: BUNDLE WROTE /etc/felhom-oob.nft (new)
+  os-apply: BUNDLE WROTE /etc/systemd/system/felhom-oob-nft.service (new)
+  os-apply: BUNDLE WROTE /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (new)
+  os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-op (new)
+  os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new)
+  os-apply: BUNDLE DONE agent=0.172.0 written=31 same=1 self-check=ok signers-created=False
+[OK]   installed every root-owned file from config bundle v0.172.0 (sha cc34b0bc3346ffd6…; checked, previous copies kept)
+[OK]   OOB felhom-sshd instance + static belt from the bundle (agent renders config + fills sets once oob.enabled)
+[STEP] 5b/8 internal data place (Belső tárhely)
+[INFO]   no internal place: only -79 GiB free for it (pool 19, reserve 16, promised 0, planned 82; needs 20)
+[STEP] 6/8 agent config + service
+[WARN]   backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
+[WARN]   It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
+[INFO]   island bridge vmbr9 already present — leaving it
+[INFO]   R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=192.168.0.126
+[INFO]   node=box1103  local_api=169.254.253.1:8443  tls_fp=DE:30:AC:4C:D2:A4…
+[OK]   wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
+[OK]   agent --selftest (read-only) passed
+[OK]   felhom-agent service active (non-root felhom-agent reads the config OK)
+[OK]   wrote /etc/felhom/os-trust.json + operator-signers (felhom-op-1) — the root-owned slow-lane anchors
+[STEP] 7/8 golden archive
+[WARN]   no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
+[INFO]   fetching golden v0.350.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_11_03-10_09_43.tar.zst
+[OK]   verified sha256 082e2e040de56153… matches the hub manifest
+[OK]   golden imported + verified: local:backup/vzdump-lxc-9100-2026_11_03-10_09_43.tar.zst
+[STEP] 8/8 provision guest 9201
+[SKIP]   pool felhom already exists
+=== felhom-agent 0.172.0 selftest=provision (vmid=9201 customer=Tester4 hostname=tester4) ===
+  --- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_11_03-10_09_43.tar.zst → vmid 9201 ---
+time=2026-11-03T10:10:43.826+01:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom
+  [OK] front half: vmid 9201 up (boot+running) in 49s; MAC=BC:24:11:B8:6D:F6
+  --- back half: mint per-guest token + populate bootstrap config mount ---
+time=2026-11-03T10:10:49.198+01:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443
+  [OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap)
+  local-api endpoint 169.254.253.1:8443 · leaf fp <long> · token: minted (not printed)
+=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) ===
+    next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
+    which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
+[OK]   provision completed
+[INFO]   rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount
+[STEP] verify
+[OK]   pct status: running
+[OK]   onboot: 1
+    mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G
+    mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
+    rootfs: local-lvm:vm-9201-disk-0,size=32G
+[OK]   pool: guest 9201 is a member of felhom
+[OK]   acl: FelhomAgentBase@/ present (user+token)
+[OK]   acl: FelhomAgentGuest@/pool/felhom present (user+token)
+[OK]   acl: FelhomAgentStore@/storage/local present (user+token)
+[OK]   acl: FelhomAgentStore@/storage/local-lvm present (user+token)
+[OK]   acl: FelhomAgentStore@/storage/felhom-pbs present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990000 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990001 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990002 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990003 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990004 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990005 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990006 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990007 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990008 present (user+token)
+[OK]   acl: FelhomAgentGuest@/vms/990009 present (user+token)
+[OK]   authz signers: 2 (operator-signed self-update armed)
+[OK]   controller: Up 26 seconds (healthy) (after ~0s)
+[INFO]   controller image: gitea.dooplex.hu/admin/felhom-controller:0.350.0
+[WARN]   cloudflared not visible yet
+[INFO]   (confirm in the hub UI that host Tester4-e286d3 reports guest 9201)
+
+[OK] Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-e286d3 customer=Tester4 golden=local:backup/vzdump-lxc-9100-2026_11_03-10_09_43.tar.zst
+[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works).
+rc=0
@@ -0,0 +1,7 @@
+== teardown 2026-11-03T09:14:43Z
+name: release1103-box
+purging VM 9383 from related configurations..
+      VMID NAME                 STATUS     MEM(MB)
+       341 night1004-tester1    running    8192   
+ls: cannot access '/mnt/hdd_1/images/9383': No such file or directory
+Mem:              29          14           6           0           9          15
@@ -0,0 +1,21 @@
+2026/11/03 10:09:29 [INFO] Artifact manifest served for customer Tester4 (agent=0.171.0 golden=0.344.0)
+2026/11/03 10:18:10 [INFO] Artifact manifest set: agent=0.172.0 golden=0.350.0 min_agent="0.158.1" wrapper_sha=false bundle_sha="cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce"
+hub DB, newest report 2026-11-03 09:18:15 now 2026-11-03 09:19:09
+('artifact_agent_version', '0.172.0', '2026-11-03 09:18:10')
+('artifact_agent_sha256', '2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7', '2026-11-03 09:18:10')
+('artifact_golden_version', '0.350.0', '2026-11-03 09:18:10')
+('artifact_golden_sha256', '082e2e040de5615300da3ec8bc48fe6761ee0e6375dcc732e5cba82086e5e06b', '2026-11-03 09:18:10')
+('min_controller_version', '0.350.0', '2026-11-03 07:18:16')
+('artifact_min_agent', '0.158.1', '2026-11-03 09:18:10')
+('artifact_wrapper_sha256', '', '2026-11-03 09:18:10')
+('min_controller_version_declared_min_agent', '0.350.0=0.158.1', '2026-11-03 07:18:16')
+('artifact_bundle_sha256', 'cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce', '2026-11-03 09:18:10')
+('demo-felhom-8363b5', 'demo-felhom', '0.172.0', '2026-11-03 09:18:15')
+('demo-hp-bb76ea', 'demo-hp', '0.172.0', '2026-11-03 09:18:14')
+('tester-1-d70be4', 'tester-1', '0.172.0', '2026-11-03 09:18:14')
+('Tester4-e286d3', 'Tester4', '0.172.0', '2026-11-03 09:12:56')
+('Tester-2', '2026-10-04 18:06:24')
+('Tester4', '2026-11-03 09:13:54')
+('demo-felhom', '2026-11-03 09:04:36')
+('demo-hp', '2026-11-03 09:04:39')
+('tester-1', '2026-11-03 09:04:39')
@@ -0,0 +1,11 @@
+# Part 3 — agent 0.172.0 (R-1302), 2026-11-03
+
+`release-agent.sh` from a clean clone of `ecbb0cc` (CI 2563 green): agent sha256 `2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7`, bundle `cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce`, tag `v0.172.0`. `configs/` unchanged since 0.171.0, so the signed delivery was `agent_update` alone.
+
+**Delivery** (`agent-0172-sign.txt`, `agent-0172-delivery.txt`): demo-felhom, demo-hp, Tester 1: signed op COMPLETED, self-check 77 of 77, `selfupdate: update committed` by 07:58:34 UTC.
+
+**The fix live** (`r1302-live.txt`, `r1302_live.sh`, on the throwaway box). **A plain first start does not reach this code on the appliance path:** the installer pre-positions the grant, and the control run on agent 0.171.0 (Part 2's first try) logged no R-190 line either. The ERROR was seen on BYO boxes. So it was staged:
+- A, a true first grant: no seen file, the grant taken away, the agent started → INFO „store-grant: granted — the first grant of this tier, nothing was lost (R-1302)", self-check degraded=0, 0 R-190 lines.
+- B, a loss: the seen file there, the grant taken away, the agent started → ERROR „GRANT WAS MISSING … (R-190)", degraded=1.
+
+**Vouch:** the operator, 10:18:10 CET, with golden 0.350.0. `pinned.day0_agent` = 0.172.0 (D50).
@@ -0,0 +1,18 @@
+== tester-1
+felhom-agent 0.172.0
+Nov 03 08:57:30 felhom felhom-agent[2609044]: time=2026-11-03T08:57:30.595+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=e7c906876dcfdb8f op=agent_update
+Nov 03 08:57:33 felhom felhom-agent[2004286]: time=2026-11-03T08:57:33.706+01:00 level=INFO msg="capabilities self-check" ok=77 total=77 degraded=0 inactive=0
+Nov 03 08:58:34 felhom felhom-agent[2004286]: time=2026-11-03T08:58:34.488+01:00 level=WARN msg="selfupdate: update committed" version=0.172.0 wrapper=""
+active
+== demo-hp
+felhom-agent 0.172.0
+Nov 03 08:56:42 demo-hp felhom-agent[1819483]: time=2026-11-03T08:56:42.162+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=55bc053ef33f6cf2 op=agent_update
+Nov 03 08:56:45 demo-hp felhom-agent[279304]: time=2026-11-03T08:56:45.265+01:00 level=INFO msg="capabilities self-check" ok=77 total=77 degraded=0 inactive=0
+Nov 03 08:57:46 demo-hp felhom-agent[279304]: time=2026-11-03T08:57:46.334+01:00 level=WARN msg="selfupdate: update committed" version=0.172.0 wrapper=""
+active
+== demo-felhom
+felhom-agent 0.172.0
+Nov 03 08:49:36 demo-felhom felhom-agent[2017068]: time=2026-11-03T08:49:36.224+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=37085279fd83f188 op=agent_update
+Nov 03 08:49:38 demo-felhom felhom-agent[81417]: time=2026-11-03T08:49:38.944+01:00 level=INFO msg="capabilities self-check" ok=77 total=77 degraded=0 inactive=0
+Nov 03 08:50:40 demo-felhom felhom-agent[81417]: time=2026-11-03T08:50:40.320+01:00 level=WARN msg="selfupdate: update committed" version=0.172.0 wrapper=""
+active
@@ -0,0 +1,13 @@
+version: felhom-agent 0.172.0
+seen before: {"felhom-pbs":true,"local":true}
+== A: first grant
+time=2026-11-03T10:12:24.251+01:00 level=INFO msg="store-grant: granted — the first grant of this tier, nothing was lost (R-1302)" target=felhom-pbs privilege=Datastore.AllocateSpace action="felhom-backup-target-apply grant felh
+time=2026-11-03T10:12:24.251+01:00 level=INFO msg="capabilities self-check" ok=72 total=77 degraded=0 inactive=5
+ERROR R-190 lines: 0
+seen after: {"felhom-pbs":true,"local":true}
+== B: a loss
+time=2026-11-03T10:12:50.661+01:00 level=ERROR msg="store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" target=felhom-pbs privilege=Datastore.AllocateSpace action="felhom-backup-target-apply
+time=2026-11-03T10:12:50.661+01:00 level=INFO msg="capabilities self-check" ok=76 total=77 degraded=1 inactive=0
+time=2026-11-03T10:12:50.661+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:felhom-pbs feature="backup tier felhom-pbs: the agent's storage grant 
+INFO granted lines: 0
+grant back: 2
@@ -0,0 +1,12 @@
+#!/bin/bash
+# R-1302 staged on the throwaway box (VM 9383, agent 0.172.0). The appliance path pre-positions the grant, so a plain first start never reaches the repair (read in the install log).
+# A: a true first grant: no seen file, the grant taken away, the agent started. B: a loss: the seen file there, the grant taken away, the agent started.
+del() { for k in "--users felhom-agent@pve" "--tokens felhom-agent@pve!agent"; do pveum acl delete /storage/felhom-pbs $k --roles FelhomAgentStore; done; }
+show() { journalctl -u felhom-agent --since "$1" --no-pager | grep -E 'store-grant|capabilities self-check' | sed -E 's/^.*felhom-agent\[[0-9]+\]: //' | cut -c1-230; }
+echo "version: $(/usr/local/bin/felhom-agent --version)"
+echo "seen before: $(cat /var/lib/felhom-agent/store-grant-seen.json)"
+echo "== A: first grant"; systemctl stop felhom-agent; rm -f /var/lib/felhom-agent/store-grant-seen.json; del; T=$(date '+%F %T'); systemctl start felhom-agent; sleep 25; show "$T"
+echo "ERROR R-190 lines: $(journalctl -u felhom-agent --since "$T" --no-pager | grep -c 'R-190')"; echo "seen after: $(cat /var/lib/felhom-agent/store-grant-seen.json)"
+echo "== B: a loss"; systemctl stop felhom-agent; del; T=$(date '+%F %T'); systemctl start felhom-agent; sleep 25; show "$T"
+echo "INFO granted lines: $(journalctl -u felhom-agent --since "$T" --no-pager | grep -c 'store-grant: granted')"
+pveum acl list --output-format json | python3 -c "import json,sys;print('grant back:',sum(1 for a in json.load(sys.stdin) if a['path']=='/storage/felhom-pbs'))"
@@ -0,0 +1,5 @@
+# Part 4 — hub 0.179.0 (R-1352), 2026-11-03
+
+Tests green (`go build`, `vet`, `test`); image `sha256:9c52c1c3…` from a clean clone of `57d46201` (CI 2570 green). The operator's yes; manifest `01a44d56`; only `Deployment/hub` synced, 08:05:10 UTC. Running 0.179.0, Healthy; ArgoCD's `OutOfSync` is the known `Ingress/contact-mailer` line (the live object matches git field by field, as on 2026-10-30). `watch-after-deploy.txt`: 50 minutes, 0 ERROR lines, 0 restarts, no new event from the three boxes.
+
+R-1352 itself did not fire in that time: it needs a controller that stops answering. Proof by test only (`TestR1352_*`).
@@ -0,0 +1,11 @@
+08:05:10
+deployment "hub" successfully rolled out
+gitea.dooplex.hu/admin/felhom-hub:0.178.0
+hub-7dc9bf895d-wbsjh Running gitea.dooplex.hu/admin/felhom-hub@sha256:9c15c5106c1e3482bf33e7fb680ff1af02465ac6a17aafb6c84df3f553bd925f
+sync OutOfSync 01a44d5626 health Progressing op Succeeded
+Ingress contact-mailer OutOfSync
+08:05:43
+gitea.dooplex.hu/admin/felhom-hub:0.179.0
+hub-7b8fb5f8bf-mtkm9 Running gitea.dooplex.hu/admin/felhom-hub:0.179.0 gitea.dooplex.hu/admin/felhom-hub@sha256:9c52c1c3427dcc493e007ec94d021802eab7dd267ea2a199639ff047b2
+sync=OutOfSync health=Healthy
+2026/11/03 09:05:14 [INFO] felhom-hub 0.179.0 starting
@@ -0,0 +1,4 @@
+## hub lines 08:05-08:55 UTC, the fleet's boxes
+ERROR lines: 0
+restarts: 0
+gitea.dooplex.hu/admin/felhom-hub:0.179.0