Golden 0.289.1 recorded (baked, round-trip, vouched; floor 0.289.1 served), STATUS, session REPORT, runbook pveam note

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

REPORT-offsite-lock-build-2026-10-03.md

@@ -0,0 +1,56 @@
+# REPORT — off-site backups a box cannot delete: built and live (decisions 68–70) — 2026-10-03 (evening)
+
+Architecture read: `07-backup-architecture.md` (custody, threat rows 9–12, §D), `06-offsite-connectivity.md` §3/§5,
+`09` §3. Baselines (re-verified): controller `09453325d1b2` (0.288.0), agent `d766666ff8cf` (0.138.0), felhom.eu
+`9268d9933b8f` (hub 0.126.0 deployed), catalog `917a779cca67`. Register 327 rows, highest R-822. Rulings recorded
+first as decisions 68–70 (`5188dbd`). Evidence: `documentation/audits/offsite-lock-build-2026-10-03/`.
+
+## The Part table
+
+| Part | Result | Notes |
+|---|---|---|
+| A — migration spike | **done, passed** | An sftp-written repo is listed, extended, restored from (bytes identical), `check`ed and `check --read-data`ed through the pinned `rclone:` key with restic 0.14.0; a delete is refused (403). The same measurements also settled several facts: one key on two lines → the **first** line wins (so the window = prepend a deleting line); an absolute pinned path works; a probe signal (exit 0 + rclone output = pinned; exit 8 = unpinned); the restricted shell's `dd`/`mv`/`cat` (no `test`). |
+| B — hub registrar + sealed password | **done** — hub v0.127.0 deployed | `internal/offsitekeys`; `consume-password` → 410; password AES-256-GCM at rest (4 live rows sealed, read back as `enc:v1:`); daily key check 07:10 + on demand. 3 red-proofs. |
+| C — box on the locked key | **done** — controller v0.289.0, then **v0.289.1** | registrar client, pinned probe, `rclone:` transport (hub tier only — the household NAS stays sftp), all four deleting features off the box, window client + fake-snapshot guard. 4 red-proofs. **v0.289.1 fixes a defect v0.289.0 put live (below).** |
+| D — live, both demo boxes | **done** | demo-felhom 11→13, demo-hp 91→100 (history kept); a delete from each box refused (403), count unchanged; one-file restore and `check` through the pinned key on each; the old endpoint answers 410 to demo-hp's own key. The hand-run key check is clean for both. **Changed:** the "unprefixed test line on a demo sub-account" decoy ran on tester-1's account instead, which already held 3 unpinned lines. The demo passwords are now sealed in the hub, and the hub DB is the only route to them. |
+| — stop point | **passed** | |
+| E — window + guard | **mechanics done; a real prune NOT done** | Window 1 on demo-hp ran live: the hub opened it (deleting line first), the guard refused, the window closed in 3 s, the operator was mailed, and the key file read back clean. The refusal is a **design defect** (R-824): any manual run makes the plan remove a same-day snapshot younger than 8 days. Weekly windows stay OFF. |
+| F — ep0 → DooPlex copy | **done** | ep0: one read-only token (the only change there). DooPlex: an SSH forward (`felhom-ep0-pbs-tunnel.service` — **operator ruling in-session**, because ep0's PBS listens on `wg0` only), PBS remote, datastore `ep0-copy`, nightly pull 05:00 with `remove-vanished false`, Saturday verify, failures to admin@ via Resend (the test mail arrived). First pull: 201 s, 12 GB, 4 of 4 snapshots, matching ep0. Runbook: `runbooks/ep0-datastore-copy.md`. |
+| Golden + floor | **done** | Golden 0.289.1 baked (subagent, runbook §4.1, token-leak grep 0 with a positive control), round-trip sha matches, vouched (agent 0.138.0, min_agent 0.131.0), floor 0.289.1 SERVED to both boxes. |
+
+## Claims in the brief that turned out wrong
+
+1. **"An sftp-written repo reads through rclone"** — confirmed: it was expected, and now it is measured.
+2. **"The box stores no password today"** — true on disk (it was only in an env var during install), but the box could fetch the password at will; that route is closed now.
+3. **"One authorized_keys can hold two lines for the same key"** — it can, but only the **first** line counts. That is what makes the window possible with a single key.
+4. **"The integrity check works through the forced key"** — confirmed (`check`, `check --read-data`, the exclusive lock).
+5. **"DooPlex has room and a PBS that can pull from ep0"** — it has the room (5.5 TB) and a PBS, but it **cannot reach** ep0's PBS (open on `wg0` only). An SSH forward was added on an operator ruling.
+6. **"Every deleting feature leaves the box"** — only on the hub tier. The same code serves the household's own SFTP NAS, which keeps box-side retention. A `Transport` flag separates the two.
+7. **"Abort when the plan exceeds a week's removal"** — that would never prune after the interim. The box takes the oldest snapshots up to the cap instead (disagreement recorded in the code and the CHANGELOG).
+8. **The guard as written is too strict** (R-824). It also cannot see past-dated poisoning (R-822, residual).
+
+## Found and fixed in-session
+
+- **R-825 (v0.289.1):** the provider's rclone prints a NOTICE line on every connection, and restic forwards it into the output. Every `--json` parse failed, so demo-felhom recorded **0 snapshots as measured**, and the hub mailed a **false** `offsite_snapshots_dropped` (11→0) at 17:17. The fix went live 15 min later. It strips the notice, and an unreadable count is never a measured zero. Red-proved.
+- A shadowed `newPath` in the NAS move-aside path was caught by the existing suite before release.
+- Stale **unpinned keys** sat in the sub-accounts: 4 on demo-felhom's and 5 on demo-hp's (every reinstall added one). The registrar removed them. tester-1's 3 remain, and the daily check alarms on them (R-826).
+
+## Deviations, stated
+
+- **Two controller releases**, against the one-release rule: v0.289.1 fixes a false zero that v0.289.0 put live.
+- The hub has a **test-only commit after the release** (window-sweep test + a test helper in the store). The deployed v0.127.0 image does not contain it; behaviour is unchanged.
+- I read tester-1's sealed-era password from the hub DB again for Part A (operator ruling from the morning session; the copy was deleted, the value never printed).
+- A Hetzner storage API token was printed into this session's transcript while I read `manifests/storagebox.secret.yaml` (a gitignored file; the redaction regex missed the quoted value). **Rotate `HETZNER_TOKEN`** — it is in Secret/storagebox.
+- The window's red-proofs ran in unit tests and on one live window. A live real prune did not happen (R-824).
+
+## Records
+
+- Closed: **R-820, R-821, R-342** (+ **R-825** opened and closed). Narrowed: **R-95, R-822**. Opened: **R-823, R-824, R-826, R-827, R-828, R-830**. Register **327 → 330**.
+- Decisions 68–70 in `09` §3, CONTEXT, `07`, `06`. `07` threat rows 9/10/12 and `06` §3.6 carry `[FACT]` lines.
+- Runbooks: `ep0-datastore-copy.md` (new), `secrets.md` (offsite key, DooPlex PBS secrets), `RUNBOOK-manual-build.md` (`pveam update`).
+
+## Teardown, three layers
+
+- **Machines:** helper scripts were removed from both demo guests, their containers and hosts (0 left). tester-1's sub-account is back to `.ssh`, `felhom-repo`, with `authorized_keys` byte-identical to the start (sha256 `795e7153…`). The scratch dirs `spike-r436`, `spike-migrate` and the rclone `.config` were removed. The drill VM is reverted to `virgin`, build guest 9100 destroyed, the bake token shredded.
+- **Host (DooPlex):** **kept on purpose:** `felhom-ep0-pbs-tunnel.service`, the PBS remote/datastore/jobs/notification target (Part F). The scratchpad secret files (sub4 password, ep0 token, Resend key copy) were shredded.
+- **Hub:** **kept:** v0.127.0, Secret/offsite-secret-key, floor 0.289.1, golden 0.289.1 vouched. Weekly windows OFF. The one-shot grant for demo-hp was consumed.

STATUS.md

@@ -2,8 +2,27 @@
 
 **Ready for the first real tester (Tester-2): yes. You confirmed the tunnel route and the connect mails (2026-09-30).**
 
-**Updated 2026-10-03 (off-site backup safety, step 1: measured on Hetzner). Versions unchanged since 2026-10-02 (afternoon). Both demo boxes run controller 0.288.0 and host agent 0.138.0. Hub 0.126.0. New
+**Updated 2026-10-03 (evening): off-site backups a box cannot delete — built and live. Both demo boxes run controller
-installs get golden 0.288.0 with agent 0.138.0.**
+0.289.1 and host agent 0.138.0. Hub 0.127.0. New installs get golden 0.289.1 with agent 0.138.0; every box's floor is 0.289.1.**
+
+## Today (2026-10-03, evening): your choices A and A — built
+
+- **A box can no longer delete its off-site backups.** Both demo boxes now use a key that can only add. I tried a
+  delete from each box: refused. Old backups all stayed (demo-felhom 11 → 13, demo-hp 91 → 100).
+- **No box gets the storage password any more.** The box gives the hub only its public key; the hub puts it in the
+  storage account. I asked for the password with demo-hp's own login: refused.
+- **The hub keeps the passwords locked (encrypted).** A copy of the hub database no longer reveals them.
+- **Every day the hub checks each storage account's key file.** It found old unlocked keys from earlier boxes:
+  4 on demo-felhom's account and 5 on demo-hp's — now removed. tester-1's account still has 3 (its box is gone); you
+  get a daily alarm for it until they go.
+- **The weekly clean-up window works, but it is switched OFF.** I opened one window on demo-hp by hand: it opened,
+  the fake-backup check refused, and it closed in 3 seconds. The check refused because I had made a manual backup
+  today — it is too strict. I fix that next; until then nothing deletes old backups (there is plenty of room).
+- **ep0's whole-box backups are copied to DooPlex every night.** First copy: 12 GB in 3 minutes, all 4 backups.
+  DooPlex cannot read them (encrypted per household). A failed copy mails you; the test mail arrived.
+- **One bug, found and fixed live:** the first new box version misread its backup count as 0, and you got one
+  false alarm mail ("demo-felhom: fell from 11 to 0"). **Ignore that mail.** Fixed 15 minutes later (0.289.1).
+- **Rows:** 3 closed, 6 opened, 1 opened and closed the same day. The list went from 327 to 330.
 
 ## Today (2026-10-03, later): off-site backup safety, step 1 — measured, nothing built
 
@@ -68,23 +87,14 @@ Your licence decisions are recorded: Emby, Plex and n8n stay. recipe-importer ne
 
 ## What needs you
 
-0. **Who may delete old off-site backups, once boxes can only add?** (Details: the design in the
+0. **Off-site clean-up window: nothing to decide now.** It stays OFF until the next session fixes the too-strict check.
-   2026-10-03 off-site audit folder.)
+   **If you do nothing:** no old off-site backup is deleted; storage grows slowly (each household uses under 1 GB).
-   - **A — the box, in a short weekly window the hub opens** (recommended). The backup password stays only on
+0b. **How much history should DooPlex's ep0 copy keep?** Today it keeps everything and grows every night.
-     the box, as we promise today. Cost: during the window a broken-into box could delete; the hub checks the
+   - **A — keep the last 8 weekly copies** (recommended): undo up to 2 months; about 4 times ep0's size (ep0 keeps 2).
-     count before and after.
+   - **B — keep everything:** never loses anything; DooPlex's disk slowly fills (5.5 TB free today).
-   - **B — a Felhom machine does it for every box.** Cost: that machine must hold every household's backup
+   - **If you do nothing:** B — it grows; nothing breaks for months.
-     password, so it could read every household's backups. That changes a promise to the customer, and adds a
+0c. **tester-1's old keys:** say "remove them" and I clean that storage account's key file through the hub. **If you do
-     new always-on machine.
+   nothing:** one alarm mail a day for tester-1.
-   - **If you say nothing:** nothing is built. Boxes keep the key that can delete (today's risk stays).
-   - Either way, first: the hub installs the box's key, so the box never gets the storage password.
-0b. **ep0's backup disk has no copy of its own. Which safeguard?**
-   - **A — DooPlex copies it every night** (recommended). €0 a month, about 1–2 hours to set up. Protects against
-     losing the disk and losing Hetzner. The copy is encrypted per household, so DooPlex cannot read it. Cost: a
-     new job on DooPlex.
-   - **B — accept the risk in writing,** and copy the disk off by hand before any risky work on ep0.
-   - **If you say nothing:** the disk stays unprotected; a bad day on ep0 loses every household's whole-box
-     off-site copy.
 1. **plant-it:** keep the hidden template as it is, or remove it entirely (its image no longer exists). **If you say
    nothing:** it stays hidden; nothing runs it.
 2. **Send the SparkyFitness request, and ask the Tandoor authors** (the "Before the first paying customer" list).

documentation/runbooks/RUNBOOK-manual-build.md

@@ -151,7 +151,8 @@ qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 8192 \
 ### 4.1 Bake + publish
 
 1. Revert + boot per §4.0.
-2. The debian template is **absent on `virgin`** and **the exact point release rots** — list the
+2. Run **`pveam update` first** — the `virgin` snapshot's template INDEX is stale too, and a stale index fails as a bogus
+   `400 no such template` (0.289.1 bake, 2026-10-03). The debian template is **absent on `virgin`** and **the exact point release rots** — list the
    current one (`pveam available --section system | grep 'debian-13-standard_.*_amd64'`) and `pveam download local <that>`.
    **Filter on `_amd64`:** the index also lists an `_arm64` build of the same point release, and a version sort
    picks it (2026-09-28: the 0.276.0 bake's first attempt did, and was stopped before `pct create` finished; the

documentation/tests/golden-0.289.1-2026-10-03/02-round-trip.txt

@@ -0,0 +1,3 @@
+## round trip 2026-10-03T19:27Z
+curl https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.289.1/golden.tar.zst | sha256sum
+59fa7beadbb17fbcd30524c47e34806d3122f57fb8c5e2618727b6781e874512  (== GOLDEN_SHA256 printed by the bake)

documentation/tests/golden-0.289.1-2026-10-03/README.md

@@ -0,0 +1,51 @@
+# Golden 0.289.1 bake (2026-10-03)
+
+Baked in the drill VM on DooPlex per `documentation/runbooks/RUNBOOK-manual-build.md` §4.0 and §4.1
+steps 1–4. Step 5 (vouching in the hub) was NOT done here.
+
+- Controller image: `gitea.dooplex.hu/admin/felhom-controller:0.289.1`
+- Build script: `felhom-agent/configs/build-golden.sh` v3.0.0 at agent `main` d766666 (clean tree,
+  equal to origin/main); sha256 `e4c9ede772e777efacdbc6a2bfb3b15d181d2e96bb82b40cee557b5bbe068834`,
+  identical on DooPlex and inside the VM.
+- Template: `debian-13-standard_13.6-1_amd64.tar.zst` (from `pveam available`, filtered on `_amd64`).
+- Drill VM: `pve-manager/9.2.2`.
+
+**GOLDEN_VERSION=0.289.1**
+**GOLDEN_SHA256=59fa7beadbb17fbcd30524c47e34806d3122f57fb8c5e2618727b6781e874512**
+Package: `https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.289.1/golden.tar.zst`
+(652929857 bytes).
+
+## Pass markers, quoted from `bake.log`
+
+```
+82:  docker OK (overlay2; data-root /var/lib/docker)
+313:INFO: including mount point rootfs ('/') in backup
+314:INFO: including mount point mp0 ('/var/lib/felhom') in backup
+319:[golden]   pre-delete existing: HTTP 404 (404/204 expected)
+320:[golden]   upload OK (HTTP 201)
+```
+
+`grep -c -E 'excluding|FATAL' bake.log` = `0`. No `mp1` line (none expected since v3.0.0).
+
+## Token handling
+
+- Token copied file → file (`scp`); the bake ran from a runner script inside the VM that reads the
+  token itself (`systemd-run --unit=golden-bake --collect`).
+- `systemctl show golden-bake -p Environment -p ExecStart | grep -c -F <token>` = `0`;
+  positive control (same output with the token appended) = `1`.
+- Leak grep on THIS saved `bake.log` (the committed copy): `0`. Positive control (a throwaway copy
+  with the token appended) = `1`; the copy was `shred -u`'d.
+
+## Teardown state
+
+- `pct destroy 9100 --purge` — rc 0, both LVs removed.
+- `/root/.gitea-token`, `/root/bake-run.sh`, `/root/bake.log` in the VM — `shred -u`, confirmed absent.
+- VM powered off; qemu process gone (no `qemu-system-x86` in `ps`).
+- `qemu-img snapshot -a virgin drill.qcow2` — OK; snapshot `virgin` still listed.
+
+## Deviations from the runbook
+
+- `pveam update` was run before `pveam available` (the virgin snapshot's template index is stale);
+  the runbook's step 2 does not list it.
+- The published package was not re-downloaded to verify its sha256; the evidence is the script's
+  `upload OK (HTTP 201)` and its printed `GOLDEN_SHA256`.

documentation/tests/golden-0.289.1-2026-10-03/bake.log

@@ -0,0 +1,324 @@
+[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.289.1
+[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
+  Logical volume "vm-9100-disk-0" created.
+  Logical volume pve/vm-9100-disk-0 changed.
+Creating filesystem with 8388608 4k blocks and 2097152 inodes
+Filesystem UUID: 293c00c4-bc27-4bd1-950d-abe767af4ac9
+Superblock backups stored on blocks: 
+	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
+	4096000, 7962624
+  Logical volume "vm-9100-disk-1" created.
+  Logical volume pve/vm-9100-disk-1 changed.
+Creating filesystem with 6291456 4k blocks and 1572864 inodes
+Filesystem UUID: 60370228-027f-488c-a513-3edf4e995a38
+Superblock backups stored on blocks: 
+	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
+extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
+Total bytes read: 553512960 (528MiB, 118MiB/s)
+Detected container architecture: amd64
+Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
+done: SHA256:WnMOBLV+R80E8SXtI9qdZF7uZEMq0HTlppa5eBrYT1c root@felhom-golden
+Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
+done: SHA256:jlNHFJRnzleEHjJTCqBVLl24x9IDoeaoqL1hAZd58bc root@felhom-golden
+Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
+done: SHA256:FhN6dlu6kDKjEbzaMd11oRS9n2qAT6u4lGaBQcIjG/M root@felhom-golden
+[golden] starting + installing Docker (official repo, trixie channel) …
+apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
+perl: warning: Setting locale failed.
+perl: warning: Please check that your locale settings:
+	LANGUAGE = (unset),
+	LC_ALL = (unset),
+	LC_CTYPE = (unset),
+	LC_NUMERIC = (unset),
+	LC_COLLATE = (unset),
+	LC_TIME = (unset),
+	LC_MESSAGES = (unset),
+	LC_MONETARY = (unset),
+	LC_ADDRESS = (unset),
+	LC_IDENTIFICATION = (unset),
+	LC_MEASUREMENT = (unset),
+	LC_PAPER = (unset),
+	LC_TELEPHONE = (unset),
+	LC_NAME = (unset),
+	LANG = "en_US.UTF-8"
+    are supported and installed on your system.
+perl: warning: Falling back to the standard locale ("C").
+locale: Cannot set LC_CTYPE to default locale: No such file or directory
+locale: Cannot set LC_MESSAGES to default locale: No such file or directory
+locale: Cannot set LC_ALL to default locale: No such file or directory
+apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
+perl: warning: Setting locale failed.
+perl: warning: Please check that your locale settings:
+	LANGUAGE = (unset),
+	LC_ALL = (unset),
+	LC_CTYPE = (unset),
+	LC_NUMERIC = (unset),
+	LC_COLLATE = (unset),
+	LC_TIME = (unset),
+	LC_MESSAGES = (unset),
+	LC_MONETARY = (unset),
+	LC_ADDRESS = (unset),
+	LC_IDENTIFICATION = (unset),
+	LC_MEASUREMENT = (unset),
+	LC_PAPER = (unset),
+	LC_TELEPHONE = (unset),
+	LC_NAME = (unset),
+	LANG = "en_US.UTF-8"
+    are supported and installed on your system.
+perl: warning: Falling back to the standard locale ("C").
+locale: Cannot set LC_CTYPE to default locale: No such file or directory
+locale: Cannot set LC_MESSAGES to default locale: No such file or directory
+locale: Cannot set LC_ALL to default locale: No such file or directory
+[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
+[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
+[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
+Unable to find image 'hello-world:latest' locally
+latest: Pulling from library/hello-world
+4f55086f7dd0: Pulling fs layer
+4f55086f7dd0: Download complete
+4f55086f7dd0: Pull complete
+Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8
+Status: Downloaded newer image for hello-world:latest
+  docker OK (overlay2; data-root /var/lib/docker)
+  /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
+  /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
+  both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
+[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.289.1 (no registry cred at deploy) …
+
+WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
+Configure a credential helper to remove this warning. See
+https://docs.docker.com/go/credential-store/
+
+0.289.1: Pulling from admin/felhom-controller
+774043ccc8cc: Pulling fs layer
+ab6b448d4be9: Pulling fs layer
+23a5bfa58353: Pulling fs layer
+862a57157567: Pulling fs layer
+01426dd1a8b3: Pulling fs layer
+dbbfe8b6858f: Pulling fs layer
+862a57157567: Waiting
+01426dd1a8b3: Waiting
+dbbfe8b6858f: Waiting
+774043ccc8cc: Verifying Checksum
+774043ccc8cc: Download complete
+862a57157567: Verifying Checksum
+862a57157567: Download complete
+23a5bfa58353: Verifying Checksum
+23a5bfa58353: Download complete
+01426dd1a8b3: Verifying Checksum
+01426dd1a8b3: Download complete
+dbbfe8b6858f: Verifying Checksum
+dbbfe8b6858f: Download complete
+ab6b448d4be9: Verifying Checksum
+ab6b448d4be9: Download complete
+774043ccc8cc: Pull complete
+ab6b448d4be9: Pull complete
+23a5bfa58353: Pull complete
+862a57157567: Pull complete
+01426dd1a8b3: Pull complete
+dbbfe8b6858f: Pull complete
+Digest: sha256:97cb56a99b67cc5b9ab5df2e7bcc6bc6bfd98277cd918013b6d64fad46cb99f5
+Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.289.1
+gitea.dooplex.hu/admin/felhom-controller:0.289.1
+[golden] asking the controller which infra images it manages …
+[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 …
+v3.6.7: Pulling from library/traefik
+589002ba0eae: Pulling fs layer
+ef63511ea6cc: Pulling fs layer
+0738e5cb835e: Pulling fs layer
+3e6813f70c64: Pulling fs layer
+3e6813f70c64: Waiting
+ef63511ea6cc: Verifying Checksum
+ef63511ea6cc: Download complete
+589002ba0eae: Verifying Checksum
+589002ba0eae: Download complete
+3e6813f70c64: Verifying Checksum
+3e6813f70c64: Download complete
+0738e5cb835e: Verifying Checksum
+0738e5cb835e: Download complete
+589002ba0eae: Pull complete
+ef63511ea6cc: Pull complete
+0738e5cb835e: Pull complete
+3e6813f70c64: Pull complete
+Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a
+Status: Downloaded newer image for traefik:v3.6.7
+docker.io/library/traefik:v3.6.7
+2026.6.0: Pulling from cloudflare/cloudflared
+47de5dd0b812: Pulling fs layer
+c172f21841df: Pulling fs layer
+99515e7b4d35: Pulling fs layer
+99ba982a9142: Pulling fs layer
+d6b1b89eccac: Pulling fs layer
+2780920e5dbf: Pulling fs layer
+7c12895b777b: Pulling fs layer
+3214acf345c0: Pulling fs layer
+52630fc75a18: Pulling fs layer
+dd64bf2dd177: Pulling fs layer
+b839dfae01f6: Pulling fs layer
+ebddc55facdc: Pulling fs layer
+bdfd7f7e5bf6: Pulling fs layer
+2d4d7adf6272: Pulling fs layer
+40008157d8d2: Pulling fs layer
+bd8962e29291: Pulling fs layer
+cac2ae0193cb: Pulling fs layer
+74d1dac84ecc: Pulling fs layer
+dd64bf2dd177: Waiting
+b839dfae01f6: Waiting
+ebddc55facdc: Waiting
+bdfd7f7e5bf6: Waiting
+2d4d7adf6272: Waiting
+40008157d8d2: Waiting
+bd8962e29291: Waiting
+cac2ae0193cb: Waiting
+74d1dac84ecc: Waiting
+2780920e5dbf: Waiting
+99ba982a9142: Waiting
+d6b1b89eccac: Waiting
+7c12895b777b: Waiting
+52630fc75a18: Waiting
+3214acf345c0: Waiting
+47de5dd0b812: Download complete
+c172f21841df: Verifying Checksum
+c172f21841df: Download complete
+47de5dd0b812: Pull complete
+99515e7b4d35: Verifying Checksum
+99515e7b4d35: Download complete
+99ba982a9142: Verifying Checksum
+99ba982a9142: Download complete
+d6b1b89eccac: Verifying Checksum
+d6b1b89eccac: Download complete
+2780920e5dbf: Download complete
+7c12895b777b: Verifying Checksum
+7c12895b777b: Download complete
+3214acf345c0: Verifying Checksum
+3214acf345c0: Download complete
+52630fc75a18: Verifying Checksum
+52630fc75a18: Download complete
+dd64bf2dd177: Verifying Checksum
+dd64bf2dd177: Download complete
+c172f21841df: Pull complete
+b839dfae01f6: Verifying Checksum
+b839dfae01f6: Download complete
+ebddc55facdc: Verifying Checksum
+ebddc55facdc: Download complete
+bdfd7f7e5bf6: Verifying Checksum
+bdfd7f7e5bf6: Download complete
+2d4d7adf6272: Verifying Checksum
+2d4d7adf6272: Download complete
+bd8962e29291: Verifying Checksum
+bd8962e29291: Download complete
+40008157d8d2: Verifying Checksum
+40008157d8d2: Download complete
+cac2ae0193cb: Verifying Checksum
+cac2ae0193cb: Download complete
+99515e7b4d35: Pull complete
+74d1dac84ecc: Verifying Checksum
+74d1dac84ecc: Download complete
+99ba982a9142: Pull complete
+d6b1b89eccac: Pull complete
+2780920e5dbf: Pull complete
+7c12895b777b: Pull complete
+3214acf345c0: Pull complete
+52630fc75a18: Pull complete
+dd64bf2dd177: Pull complete
+b839dfae01f6: Pull complete
+ebddc55facdc: Pull complete
+bdfd7f7e5bf6: Pull complete
+2d4d7adf6272: Pull complete
+40008157d8d2: Pull complete
+bd8962e29291: Pull complete
+cac2ae0193cb: Pull complete
+74d1dac84ecc: Pull complete
+Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f
+Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0
+docker.io/cloudflare/cloudflared:2026.6.0
+1.3.3-stable: Pulling from gtstef/filebrowser
+6a0ac1617861: Pulling fs layer
+ef8806083e82: Pulling fs layer
+b74107c861c7: Pulling fs layer
+adc935def003: Pulling fs layer
+4f4fb700ef54: Pulling fs layer
+18695ccc900a: Pulling fs layer
+45d119d5c397: Pulling fs layer
+dac52db4fc51: Pulling fs layer
+6d598f86b2f2: Pulling fs layer
+8aa349c8396c: Pulling fs layer
+18695ccc900a: Waiting
+45d119d5c397: Waiting
+dac52db4fc51: Waiting
+6d598f86b2f2: Waiting
+8aa349c8396c: Waiting
+adc935def003: Waiting
+4f4fb700ef54: Waiting
+6a0ac1617861: Verifying Checksum
+6a0ac1617861: Download complete
+b74107c861c7: Verifying Checksum
+b74107c861c7: Download complete
+4f4fb700ef54: Verifying Checksum
+4f4fb700ef54: Download complete
+6a0ac1617861: Pull complete
+adc935def003: Verifying Checksum
+adc935def003: Download complete
+18695ccc900a: Verifying Checksum
+18695ccc900a: Download complete
+45d119d5c397: Verifying Checksum
+45d119d5c397: Download complete
+6d598f86b2f2: Verifying Checksum
+6d598f86b2f2: Download complete
+dac52db4fc51: Verifying Checksum
+dac52db4fc51: Download complete
+8aa349c8396c: Verifying Checksum
+8aa349c8396c: Download complete
+ef8806083e82: Verifying Checksum
+ef8806083e82: Download complete
+ef8806083e82: Pull complete
+b74107c861c7: Pull complete
+adc935def003: Pull complete
+4f4fb700ef54: Pull complete
+18695ccc900a: Pull complete
+45d119d5c397: Pull complete
+dac52db4fc51: Pull complete
+6d598f86b2f2: Pull complete
+8aa349c8396c: Pull complete
+Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c
+Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable
+docker.io/gtstef/filebrowser:1.3.3-stable
+1.1.0: Pulling from admin/felhom-samba
+897d797d2723: Pulling fs layer
+3051591aa250: Pulling fs layer
+ce57a3f93416: Pulling fs layer
+fb94eeec2fe1: Pulling fs layer
+fb94eeec2fe1: Waiting
+ce57a3f93416: Download complete
+fb94eeec2fe1: Verifying Checksum
+fb94eeec2fe1: Download complete
+897d797d2723: Download complete
+3051591aa250: Verifying Checksum
+3051591aa250: Download complete
+897d797d2723: Pull complete
+3051591aa250: Pull complete
+ce57a3f93416: Pull complete
+fb94eeec2fe1: Pull complete
+Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
+Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
+gitea.dooplex.hu/admin/felhom-samba:1.1.0
+[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
+Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
+[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
+Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
+[golden] baking the first-boot SSH host-key regeneration unit (F3) …
+Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
+[golden] identity-clean + minimize …
+[golden] stop + archive …
+INFO: including mount point rootfs ('/') in backup
+INFO: including mount point mp0 ('/var/lib/felhom') in backup
+INFO: archive file size: 622MB
+INFO: Finished Backup of VM 9100 (00:00:30)
+[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_10_03-21_24_48.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
+[golden] publishing golden (652929857 bytes, sha256 59fa7beadbb17fbc…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.289.1/golden.tar.zst
+[golden]   pre-delete existing: HTTP 404 (404/204 expected)
+[golden]   upload OK (HTTP 201)
+GOLDEN_VERSION=0.289.1
+GOLDEN_SHA256=59fa7beadbb17fbcd30524c47e34806d3122f57fb8c5e2618727b6781e874512
+[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.289.1 / 59fa7beadbb17fbcd30524c47e34806d3122f57fb8c5e2618727b6781e874512
+[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)

documentation/tests/golden-0.289.1-2026-10-03/vouch-and-floor.txt

@@ -0,0 +1,9 @@
+## vouch 2026-10-03T19:27:51Z
+HTTP/1.1 303 See Other
+Location: /configuration?flash=artifacts_set
+## floor
+HTTP/1.1 303 See Other
+Location: /configuration?flash=floor_set
+2026/10/03 21:27:31 [INFO] artifact versions (felhom-golden): showing newest 20 of 25
+2026/10/03 21:28:19 [INFO] Artifact manifest set: agent=0.138.0 golden=0.289.1 min_agent="0.131.0" wrapper_sha=false
+2026/10/03 21:28:20 [INFO] Global controller-version floor set to "0.289.1" (declared MinAgent "")