diff --git a/REPORT-offsite-lock-build-2026-10-03.md b/REPORT-offsite-lock-build-2026-10-03.md new file mode 100644 index 00000000..f94561af --- /dev/null +++ b/REPORT-offsite-lock-build-2026-10-03.md @@ -0,0 +1,56 @@ +# REPORT — off-site backups a box cannot delete: built and live (decisions 68–70) — 2026-10-03 (evening) + +Architecture read: `07-backup-architecture.md` (custody, threat rows 9–12, §D), `06-offsite-connectivity.md` §3/§5, +`09` §3. Baselines (re-verified): controller `09453325d1b2` (0.288.0), agent `d766666ff8cf` (0.138.0), felhom.eu +`9268d9933b8f` (hub 0.126.0 deployed), catalog `917a779cca67`. Register 327 rows, highest R-822. Rulings recorded +first as decisions 68–70 (`5188dbd`). Evidence: `documentation/audits/offsite-lock-build-2026-10-03/`. + +## The Part table + +| Part | Result | Notes | +|---|---|---| +| A — migration spike | **done, passed** | An sftp-written repo is listed, extended, restored from (bytes identical), `check`ed and `check --read-data`ed through the pinned `rclone:` key with restic 0.14.0; a delete is refused (403). The same measurements also settled several facts: one key on two lines → the **first** line wins (so the window = prepend a deleting line); an absolute pinned path works; a probe signal (exit 0 + rclone output = pinned; exit 8 = unpinned); the restricted shell's `dd`/`mv`/`cat` (no `test`). | +| B — hub registrar + sealed password | **done** — hub v0.127.0 deployed | `internal/offsitekeys`; `consume-password` → 410; password AES-256-GCM at rest (4 live rows sealed, read back as `enc:v1:`); daily key check 07:10 + on demand. 3 red-proofs. | +| C — box on the locked key | **done** — controller v0.289.0, then **v0.289.1** | registrar client, pinned probe, `rclone:` transport (hub tier only — the household NAS stays sftp), all four deleting features off the box, window client + fake-snapshot guard. 4 red-proofs. **v0.289.1 fixes a defect v0.289.0 put live (below).** | +| D — live, both demo boxes | **done** | demo-felhom 11→13, demo-hp 91→100 (history kept); a delete from each box refused (403), count unchanged; one-file restore and `check` through the pinned key on each; the old endpoint answers 410 to demo-hp's own key. The hand-run key check is clean for both. **Changed:** the "unprefixed test line on a demo sub-account" decoy ran on tester-1's account instead, which already held 3 unpinned lines. The demo passwords are now sealed in the hub, and the hub DB is the only route to them. | +| — stop point | **passed** | | +| E — window + guard | **mechanics done; a real prune NOT done** | Window 1 on demo-hp ran live: the hub opened it (deleting line first), the guard refused, the window closed in 3 s, the operator was mailed, and the key file read back clean. The refusal is a **design defect** (R-824): any manual run makes the plan remove a same-day snapshot younger than 8 days. Weekly windows stay OFF. | +| F — ep0 → DooPlex copy | **done** | ep0: one read-only token (the only change there). DooPlex: an SSH forward (`felhom-ep0-pbs-tunnel.service` — **operator ruling in-session**, because ep0's PBS listens on `wg0` only), PBS remote, datastore `ep0-copy`, nightly pull 05:00 with `remove-vanished false`, Saturday verify, failures to admin@ via Resend (the test mail arrived). First pull: 201 s, 12 GB, 4 of 4 snapshots, matching ep0. Runbook: `runbooks/ep0-datastore-copy.md`. | +| Golden + floor | **done** | Golden 0.289.1 baked (subagent, runbook §4.1, token-leak grep 0 with a positive control), round-trip sha matches, vouched (agent 0.138.0, min_agent 0.131.0), floor 0.289.1 SERVED to both boxes. | + +## Claims in the brief that turned out wrong + +1. **"An sftp-written repo reads through rclone"** — confirmed: it was expected, and now it is measured. +2. **"The box stores no password today"** — true on disk (it was only in an env var during install), but the box could fetch the password at will; that route is closed now. +3. **"One authorized_keys can hold two lines for the same key"** — it can, but only the **first** line counts. That is what makes the window possible with a single key. +4. **"The integrity check works through the forced key"** — confirmed (`check`, `check --read-data`, the exclusive lock). +5. **"DooPlex has room and a PBS that can pull from ep0"** — it has the room (5.5 TB) and a PBS, but it **cannot reach** ep0's PBS (open on `wg0` only). An SSH forward was added on an operator ruling. +6. **"Every deleting feature leaves the box"** — only on the hub tier. The same code serves the household's own SFTP NAS, which keeps box-side retention. A `Transport` flag separates the two. +7. **"Abort when the plan exceeds a week's removal"** — that would never prune after the interim. The box takes the oldest snapshots up to the cap instead (disagreement recorded in the code and the CHANGELOG). +8. **The guard as written is too strict** (R-824). It also cannot see past-dated poisoning (R-822, residual). + +## Found and fixed in-session + +- **R-825 (v0.289.1):** the provider's rclone prints a NOTICE line on every connection, and restic forwards it into the output. Every `--json` parse failed, so demo-felhom recorded **0 snapshots as measured**, and the hub mailed a **false** `offsite_snapshots_dropped` (11→0) at 17:17. The fix went live 15 min later. It strips the notice, and an unreadable count is never a measured zero. Red-proved. +- A shadowed `newPath` in the NAS move-aside path was caught by the existing suite before release. +- Stale **unpinned keys** sat in the sub-accounts: 4 on demo-felhom's and 5 on demo-hp's (every reinstall added one). The registrar removed them. tester-1's 3 remain, and the daily check alarms on them (R-826). + +## Deviations, stated + +- **Two controller releases**, against the one-release rule: v0.289.1 fixes a false zero that v0.289.0 put live. +- The hub has a **test-only commit after the release** (window-sweep test + a test helper in the store). The deployed v0.127.0 image does not contain it; behaviour is unchanged. +- I read tester-1's sealed-era password from the hub DB again for Part A (operator ruling from the morning session; the copy was deleted, the value never printed). +- A Hetzner storage API token was printed into this session's transcript while I read `manifests/storagebox.secret.yaml` (a gitignored file; the redaction regex missed the quoted value). **Rotate `HETZNER_TOKEN`** — it is in Secret/storagebox. +- The window's red-proofs ran in unit tests and on one live window. A live real prune did not happen (R-824). + +## Records + +- Closed: **R-820, R-821, R-342** (+ **R-825** opened and closed). Narrowed: **R-95, R-822**. Opened: **R-823, R-824, R-826, R-827, R-828, R-830**. Register **327 → 330**. +- Decisions 68–70 in `09` §3, CONTEXT, `07`, `06`. `07` threat rows 9/10/12 and `06` §3.6 carry `[FACT]` lines. +- Runbooks: `ep0-datastore-copy.md` (new), `secrets.md` (offsite key, DooPlex PBS secrets), `RUNBOOK-manual-build.md` (`pveam update`). + +## Teardown, three layers + +- **Machines:** helper scripts were removed from both demo guests, their containers and hosts (0 left). tester-1's sub-account is back to `.ssh`, `felhom-repo`, with `authorized_keys` byte-identical to the start (sha256 `795e7153…`). The scratch dirs `spike-r436`, `spike-migrate` and the rclone `.config` were removed. The drill VM is reverted to `virgin`, build guest 9100 destroyed, the bake token shredded. +- **Host (DooPlex):** **kept on purpose:** `felhom-ep0-pbs-tunnel.service`, the PBS remote/datastore/jobs/notification target (Part F). The scratchpad secret files (sub4 password, ep0 token, Resend key copy) were shredded. +- **Hub:** **kept:** v0.127.0, Secret/offsite-secret-key, floor 0.289.1, golden 0.289.1 vouched. Weekly windows OFF. The one-shot grant for demo-hp was consumed. diff --git a/STATUS.md b/STATUS.md index d3ee22b4..e952cfc4 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,8 +2,27 @@ **Ready for the first real tester (Tester-2): yes. You confirmed the tunnel route and the connect mails (2026-09-30).** -**Updated 2026-10-03 (off-site backup safety, step 1: measured on Hetzner). Versions unchanged since 2026-10-02 (afternoon). Both demo boxes run controller 0.288.0 and host agent 0.138.0. Hub 0.126.0. New -installs get golden 0.288.0 with agent 0.138.0.** +**Updated 2026-10-03 (evening): off-site backups a box cannot delete — built and live. Both demo boxes run controller +0.289.1 and host agent 0.138.0. Hub 0.127.0. New installs get golden 0.289.1 with agent 0.138.0; every box's floor is 0.289.1.** + +## Today (2026-10-03, evening): your choices A and A — built + +- **A box can no longer delete its off-site backups.** Both demo boxes now use a key that can only add. I tried a + delete from each box: refused. Old backups all stayed (demo-felhom 11 → 13, demo-hp 91 → 100). +- **No box gets the storage password any more.** The box gives the hub only its public key; the hub puts it in the + storage account. I asked for the password with demo-hp's own login: refused. +- **The hub keeps the passwords locked (encrypted).** A copy of the hub database no longer reveals them. +- **Every day the hub checks each storage account's key file.** It found old unlocked keys from earlier boxes: + 4 on demo-felhom's account and 5 on demo-hp's — now removed. tester-1's account still has 3 (its box is gone); you + get a daily alarm for it until they go. +- **The weekly clean-up window works, but it is switched OFF.** I opened one window on demo-hp by hand: it opened, + the fake-backup check refused, and it closed in 3 seconds. The check refused because I had made a manual backup + today — it is too strict. I fix that next; until then nothing deletes old backups (there is plenty of room). +- **ep0's whole-box backups are copied to DooPlex every night.** First copy: 12 GB in 3 minutes, all 4 backups. + DooPlex cannot read them (encrypted per household). A failed copy mails you; the test mail arrived. +- **One bug, found and fixed live:** the first new box version misread its backup count as 0, and you got one + false alarm mail ("demo-felhom: fell from 11 to 0"). **Ignore that mail.** Fixed 15 minutes later (0.289.1). +- **Rows:** 3 closed, 6 opened, 1 opened and closed the same day. The list went from 327 to 330. ## Today (2026-10-03, later): off-site backup safety, step 1 — measured, nothing built @@ -68,23 +87,14 @@ Your licence decisions are recorded: Emby, Plex and n8n stay. recipe-importer ne ## What needs you -0. **Who may delete old off-site backups, once boxes can only add?** (Details: the design in the - 2026-10-03 off-site audit folder.) - - **A — the box, in a short weekly window the hub opens** (recommended). The backup password stays only on - the box, as we promise today. Cost: during the window a broken-into box could delete; the hub checks the - count before and after. - - **B — a Felhom machine does it for every box.** Cost: that machine must hold every household's backup - password, so it could read every household's backups. That changes a promise to the customer, and adds a - new always-on machine. - - **If you say nothing:** nothing is built. Boxes keep the key that can delete (today's risk stays). - - Either way, first: the hub installs the box's key, so the box never gets the storage password. -0b. **ep0's backup disk has no copy of its own. Which safeguard?** - - **A — DooPlex copies it every night** (recommended). €0 a month, about 1–2 hours to set up. Protects against - losing the disk and losing Hetzner. The copy is encrypted per household, so DooPlex cannot read it. Cost: a - new job on DooPlex. - - **B — accept the risk in writing,** and copy the disk off by hand before any risky work on ep0. - - **If you say nothing:** the disk stays unprotected; a bad day on ep0 loses every household's whole-box - off-site copy. +0. **Off-site clean-up window: nothing to decide now.** It stays OFF until the next session fixes the too-strict check. + **If you do nothing:** no old off-site backup is deleted; storage grows slowly (each household uses under 1 GB). +0b. **How much history should DooPlex's ep0 copy keep?** Today it keeps everything and grows every night. + - **A — keep the last 8 weekly copies** (recommended): undo up to 2 months; about 4 times ep0's size (ep0 keeps 2). + - **B — keep everything:** never loses anything; DooPlex's disk slowly fills (5.5 TB free today). + - **If you do nothing:** B — it grows; nothing breaks for months. +0c. **tester-1's old keys:** say "remove them" and I clean that storage account's key file through the hub. **If you do + nothing:** one alarm mail a day for tester-1. 1. **plant-it:** keep the hidden template as it is, or remove it entirely (its image no longer exists). **If you say nothing:** it stays hidden; nothing runs it. 2. **Send the SparkyFitness request, and ask the Tandoor authors** (the "Before the first paying customer" list). diff --git a/documentation/runbooks/RUNBOOK-manual-build.md b/documentation/runbooks/RUNBOOK-manual-build.md index f6b5ae42..b3f4ea9e 100644 --- a/documentation/runbooks/RUNBOOK-manual-build.md +++ b/documentation/runbooks/RUNBOOK-manual-build.md @@ -151,7 +151,8 @@ qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 8192 \ ### 4.1 Bake + publish 1. Revert + boot per §4.0. -2. The debian template is **absent on `virgin`** and **the exact point release rots** — list the +2. Run **`pveam update` first** — the `virgin` snapshot's template INDEX is stale too, and a stale index fails as a bogus + `400 no such template` (0.289.1 bake, 2026-10-03). The debian template is **absent on `virgin`** and **the exact point release rots** — list the current one (`pveam available --section system | grep 'debian-13-standard_.*_amd64'`) and `pveam download local `. **Filter on `_amd64`:** the index also lists an `_arm64` build of the same point release, and a version sort picks it (2026-09-28: the 0.276.0 bake's first attempt did, and was stopped before `pct create` finished; the diff --git a/documentation/tests/golden-0.289.1-2026-10-03/02-round-trip.txt b/documentation/tests/golden-0.289.1-2026-10-03/02-round-trip.txt new file mode 100644 index 00000000..55257afa --- /dev/null +++ b/documentation/tests/golden-0.289.1-2026-10-03/02-round-trip.txt @@ -0,0 +1,3 @@ +## round trip 2026-10-03T19:27Z +curl https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.289.1/golden.tar.zst | sha256sum +59fa7beadbb17fbcd30524c47e34806d3122f57fb8c5e2618727b6781e874512 (== GOLDEN_SHA256 printed by the bake) diff --git a/documentation/tests/golden-0.289.1-2026-10-03/README.md b/documentation/tests/golden-0.289.1-2026-10-03/README.md new file mode 100644 index 00000000..0e6463b2 --- /dev/null +++ b/documentation/tests/golden-0.289.1-2026-10-03/README.md @@ -0,0 +1,51 @@ +# Golden 0.289.1 bake (2026-10-03) + +Baked in the drill VM on DooPlex per `documentation/runbooks/RUNBOOK-manual-build.md` §4.0 and §4.1 +steps 1–4. Step 5 (vouching in the hub) was NOT done here. + +- Controller image: `gitea.dooplex.hu/admin/felhom-controller:0.289.1` +- Build script: `felhom-agent/configs/build-golden.sh` v3.0.0 at agent `main` d766666 (clean tree, + equal to origin/main); sha256 `e4c9ede772e777efacdbc6a2bfb3b15d181d2e96bb82b40cee557b5bbe068834`, + identical on DooPlex and inside the VM. +- Template: `debian-13-standard_13.6-1_amd64.tar.zst` (from `pveam available`, filtered on `_amd64`). +- Drill VM: `pve-manager/9.2.2`. + +**GOLDEN_VERSION=0.289.1** +**GOLDEN_SHA256=59fa7beadbb17fbcd30524c47e34806d3122f57fb8c5e2618727b6781e874512** +Package: `https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.289.1/golden.tar.zst` +(652929857 bytes). + +## Pass markers, quoted from `bake.log` + +``` +82: docker OK (overlay2; data-root /var/lib/docker) +313:INFO: including mount point rootfs ('/') in backup +314:INFO: including mount point mp0 ('/var/lib/felhom') in backup +319:[golden] pre-delete existing: HTTP 404 (404/204 expected) +320:[golden] upload OK (HTTP 201) +``` + +`grep -c -E 'excluding|FATAL' bake.log` = `0`. No `mp1` line (none expected since v3.0.0). + +## Token handling + +- Token copied file → file (`scp`); the bake ran from a runner script inside the VM that reads the + token itself (`systemd-run --unit=golden-bake --collect`). +- `systemctl show golden-bake -p Environment -p ExecStart | grep -c -F ` = `0`; + positive control (same output with the token appended) = `1`. +- Leak grep on THIS saved `bake.log` (the committed copy): `0`. Positive control (a throwaway copy + with the token appended) = `1`; the copy was `shred -u`'d. + +## Teardown state + +- `pct destroy 9100 --purge` — rc 0, both LVs removed. +- `/root/.gitea-token`, `/root/bake-run.sh`, `/root/bake.log` in the VM — `shred -u`, confirmed absent. +- VM powered off; qemu process gone (no `qemu-system-x86` in `ps`). +- `qemu-img snapshot -a virgin drill.qcow2` — OK; snapshot `virgin` still listed. + +## Deviations from the runbook + +- `pveam update` was run before `pveam available` (the virgin snapshot's template index is stale); + the runbook's step 2 does not list it. +- The published package was not re-downloaded to verify its sha256; the evidence is the script's + `upload OK (HTTP 201)` and its printed `GOLDEN_SHA256`. diff --git a/documentation/tests/golden-0.289.1-2026-10-03/bake.log b/documentation/tests/golden-0.289.1-2026-10-03/bake.log new file mode 100644 index 00000000..d517280e --- /dev/null +++ b/documentation/tests/golden-0.289.1-2026-10-03/bake.log @@ -0,0 +1,324 @@ +[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.289.1 +[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) … + Logical volume "vm-9100-disk-0" created. + Logical volume pve/vm-9100-disk-0 changed. +Creating filesystem with 8388608 4k blocks and 2097152 inodes +Filesystem UUID: 293c00c4-bc27-4bd1-950d-abe767af4ac9 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624 + Logical volume "vm-9100-disk-1" created. + Logical volume pve/vm-9100-disk-1 changed. +Creating filesystem with 6291456 4k blocks and 1572864 inodes +Filesystem UUID: 60370228-027f-488c-a513-3edf4e995a38 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, +extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' +Total bytes read: 553512960 (528MiB, 118MiB/s) +Detected container architecture: amd64 +Creating SSH host key 'ssh_host_rsa_key' - this may take some time ... +done: SHA256:WnMOBLV+R80E8SXtI9qdZF7uZEMq0HTlppa5eBrYT1c root@felhom-golden +Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ... +done: SHA256:jlNHFJRnzleEHjJTCqBVLl24x9IDoeaoqL1hAZd58bc root@felhom-golden +Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ... +done: SHA256:FhN6dlu6kDKjEbzaMd11oRS9n2qAT6u4lGaBQcIjG/M root@felhom-golden +[golden] starting + installing Docker (official repo, trixie channel) … +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation … +[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds … +[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) … +Unable to find image 'hello-world:latest' locally +latest: Pulling from library/hello-world +4f55086f7dd0: Pulling fs layer +4f55086f7dd0: Download complete +4f55086f7dd0: Pull complete +Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8 +Status: Downloaded newer image for hello-world:latest + docker OK (overlay2; data-root /var/lib/docker) + /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4 + /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4 + both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576 +[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.289.1 (no registry cred at deploy) … + +WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. +Configure a credential helper to remove this warning. See +https://docs.docker.com/go/credential-store/ + +0.289.1: Pulling from admin/felhom-controller +774043ccc8cc: Pulling fs layer +ab6b448d4be9: Pulling fs layer +23a5bfa58353: Pulling fs layer +862a57157567: Pulling fs layer +01426dd1a8b3: Pulling fs layer +dbbfe8b6858f: Pulling fs layer +862a57157567: Waiting +01426dd1a8b3: Waiting +dbbfe8b6858f: Waiting +774043ccc8cc: Verifying Checksum +774043ccc8cc: Download complete +862a57157567: Verifying Checksum +862a57157567: Download complete +23a5bfa58353: Verifying Checksum +23a5bfa58353: Download complete +01426dd1a8b3: Verifying Checksum +01426dd1a8b3: Download complete +dbbfe8b6858f: Verifying Checksum +dbbfe8b6858f: Download complete +ab6b448d4be9: Verifying Checksum +ab6b448d4be9: Download complete +774043ccc8cc: Pull complete +ab6b448d4be9: Pull complete +23a5bfa58353: Pull complete +862a57157567: Pull complete +01426dd1a8b3: Pull complete +dbbfe8b6858f: Pull complete +Digest: sha256:97cb56a99b67cc5b9ab5df2e7bcc6bc6bfd98277cd918013b6d64fad46cb99f5 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.289.1 +gitea.dooplex.hu/admin/felhom-controller:0.289.1 +[golden] asking the controller which infra images it manages … +[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 … +v3.6.7: Pulling from library/traefik +589002ba0eae: Pulling fs layer +ef63511ea6cc: Pulling fs layer +0738e5cb835e: Pulling fs layer +3e6813f70c64: Pulling fs layer +3e6813f70c64: Waiting +ef63511ea6cc: Verifying Checksum +ef63511ea6cc: Download complete +589002ba0eae: Verifying Checksum +589002ba0eae: Download complete +3e6813f70c64: Verifying Checksum +3e6813f70c64: Download complete +0738e5cb835e: Verifying Checksum +0738e5cb835e: Download complete +589002ba0eae: Pull complete +ef63511ea6cc: Pull complete +0738e5cb835e: Pull complete +3e6813f70c64: Pull complete +Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a +Status: Downloaded newer image for traefik:v3.6.7 +docker.io/library/traefik:v3.6.7 +2026.6.0: Pulling from cloudflare/cloudflared +47de5dd0b812: Pulling fs layer +c172f21841df: Pulling fs layer +99515e7b4d35: Pulling fs layer +99ba982a9142: Pulling fs layer +d6b1b89eccac: Pulling fs layer +2780920e5dbf: Pulling fs layer +7c12895b777b: Pulling fs layer +3214acf345c0: Pulling fs layer +52630fc75a18: Pulling fs layer +dd64bf2dd177: Pulling fs layer +b839dfae01f6: Pulling fs layer +ebddc55facdc: Pulling fs layer +bdfd7f7e5bf6: Pulling fs layer +2d4d7adf6272: Pulling fs layer +40008157d8d2: Pulling fs layer +bd8962e29291: Pulling fs layer +cac2ae0193cb: Pulling fs layer +74d1dac84ecc: Pulling fs layer +dd64bf2dd177: Waiting +b839dfae01f6: Waiting +ebddc55facdc: Waiting +bdfd7f7e5bf6: Waiting +2d4d7adf6272: Waiting +40008157d8d2: Waiting +bd8962e29291: Waiting +cac2ae0193cb: Waiting +74d1dac84ecc: Waiting +2780920e5dbf: Waiting +99ba982a9142: Waiting +d6b1b89eccac: Waiting +7c12895b777b: Waiting +52630fc75a18: Waiting +3214acf345c0: Waiting +47de5dd0b812: Download complete +c172f21841df: Verifying Checksum +c172f21841df: Download complete +47de5dd0b812: Pull complete +99515e7b4d35: Verifying Checksum +99515e7b4d35: Download complete +99ba982a9142: Verifying Checksum +99ba982a9142: Download complete +d6b1b89eccac: Verifying Checksum +d6b1b89eccac: Download complete +2780920e5dbf: Download complete +7c12895b777b: Verifying Checksum +7c12895b777b: Download complete +3214acf345c0: Verifying Checksum +3214acf345c0: Download complete +52630fc75a18: Verifying Checksum +52630fc75a18: Download complete +dd64bf2dd177: Verifying Checksum +dd64bf2dd177: Download complete +c172f21841df: Pull complete +b839dfae01f6: Verifying Checksum +b839dfae01f6: Download complete +ebddc55facdc: Verifying Checksum +ebddc55facdc: Download complete +bdfd7f7e5bf6: Verifying Checksum +bdfd7f7e5bf6: Download complete +2d4d7adf6272: Verifying Checksum +2d4d7adf6272: Download complete +bd8962e29291: Verifying Checksum +bd8962e29291: Download complete +40008157d8d2: Verifying Checksum +40008157d8d2: Download complete +cac2ae0193cb: Verifying Checksum +cac2ae0193cb: Download complete +99515e7b4d35: Pull complete +74d1dac84ecc: Verifying Checksum +74d1dac84ecc: Download complete +99ba982a9142: Pull complete +d6b1b89eccac: Pull complete +2780920e5dbf: Pull complete +7c12895b777b: Pull complete +3214acf345c0: Pull complete +52630fc75a18: Pull complete +dd64bf2dd177: Pull complete +b839dfae01f6: Pull complete +ebddc55facdc: Pull complete +bdfd7f7e5bf6: Pull complete +2d4d7adf6272: Pull complete +40008157d8d2: Pull complete +bd8962e29291: Pull complete +cac2ae0193cb: Pull complete +74d1dac84ecc: Pull complete +Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f +Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0 +docker.io/cloudflare/cloudflared:2026.6.0 +1.3.3-stable: Pulling from gtstef/filebrowser +6a0ac1617861: Pulling fs layer +ef8806083e82: Pulling fs layer +b74107c861c7: Pulling fs layer +adc935def003: Pulling fs layer +4f4fb700ef54: Pulling fs layer +18695ccc900a: Pulling fs layer +45d119d5c397: Pulling fs layer +dac52db4fc51: Pulling fs layer +6d598f86b2f2: Pulling fs layer +8aa349c8396c: Pulling fs layer +18695ccc900a: Waiting +45d119d5c397: Waiting +dac52db4fc51: Waiting +6d598f86b2f2: Waiting +8aa349c8396c: Waiting +adc935def003: Waiting +4f4fb700ef54: Waiting +6a0ac1617861: Verifying Checksum +6a0ac1617861: Download complete +b74107c861c7: Verifying Checksum +b74107c861c7: Download complete +4f4fb700ef54: Verifying Checksum +4f4fb700ef54: Download complete +6a0ac1617861: Pull complete +adc935def003: Verifying Checksum +adc935def003: Download complete +18695ccc900a: Verifying Checksum +18695ccc900a: Download complete +45d119d5c397: Verifying Checksum +45d119d5c397: Download complete +6d598f86b2f2: Verifying Checksum +6d598f86b2f2: Download complete +dac52db4fc51: Verifying Checksum +dac52db4fc51: Download complete +8aa349c8396c: Verifying Checksum +8aa349c8396c: Download complete +ef8806083e82: Verifying Checksum +ef8806083e82: Download complete +ef8806083e82: Pull complete +b74107c861c7: Pull complete +adc935def003: Pull complete +4f4fb700ef54: Pull complete +18695ccc900a: Pull complete +45d119d5c397: Pull complete +dac52db4fc51: Pull complete +6d598f86b2f2: Pull complete +8aa349c8396c: Pull complete +Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c +Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable +docker.io/gtstef/filebrowser:1.3.3-stable +1.1.0: Pulling from admin/felhom-samba +897d797d2723: Pulling fs layer +3051591aa250: Pulling fs layer +ce57a3f93416: Pulling fs layer +fb94eeec2fe1: Pulling fs layer +fb94eeec2fe1: Waiting +ce57a3f93416: Download complete +fb94eeec2fe1: Verifying Checksum +fb94eeec2fe1: Download complete +897d797d2723: Download complete +3051591aa250: Verifying Checksum +3051591aa250: Download complete +897d797d2723: Pull complete +3051591aa250: Pull complete +ce57a3f93416: Pull complete +fb94eeec2fe1: Pull complete +Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0 +gitea.dooplex.hu/admin/felhom-samba:1.1.0 +[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'. +[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'. +[golden] baking the first-boot SSH host-key regeneration unit (F3) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'. +[golden] identity-clean + minimize … +[golden] stop + archive … +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +INFO: archive file size: 622MB +INFO: Finished Backup of VM 9100 (00:00:30) +[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_10_03-21_24_48.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive) +[golden] publishing golden (652929857 bytes, sha256 59fa7beadbb17fbc…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.289.1/golden.tar.zst +[golden] pre-delete existing: HTTP 404 (404/204 expected) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.289.1 +GOLDEN_SHA256=59fa7beadbb17fbcd30524c47e34806d3122f57fb8c5e2618727b6781e874512 +[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.289.1 / 59fa7beadbb17fbcd30524c47e34806d3122f57fb8c5e2618727b6781e874512 +[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge) diff --git a/documentation/tests/golden-0.289.1-2026-10-03/vouch-and-floor.txt b/documentation/tests/golden-0.289.1-2026-10-03/vouch-and-floor.txt new file mode 100644 index 00000000..a8c174a5 --- /dev/null +++ b/documentation/tests/golden-0.289.1-2026-10-03/vouch-and-floor.txt @@ -0,0 +1,9 @@ +## vouch 2026-10-03T19:27:51Z +HTTP/1.1 303 See Other +Location: /configuration?flash=artifacts_set +## floor +HTTP/1.1 303 See Other +Location: /configuration?flash=floor_set +2026/10/03 21:27:31 [INFO] artifact versions (felhom-golden): showing newest 20 of 25 +2026/10/03 21:28:19 [INFO] Artifact manifest set: agent=0.138.0 golden=0.289.1 min_agent="0.131.0" wrapper_sha=false +2026/10/03 21:28:20 [INFO] Global controller-version floor set to "0.289.1" (declared MinAgent "")