Off-site safety finished: Parts A-F evidence, decision 74, golden 0.290.0 recorded (vouched, floor 0.290.0), restore walked from the DooPlex copy, register 330 -> 328 (R-823/824/826/827/828/830 closed; R-833, R-834 opened)
gates / gates (push) Successful in 30s
gates / gates (push) Successful in 30s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
+1
-1
@@ -16,7 +16,7 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **Rulings 2026-10-04 (morning) — recorded before the work.** `09` §3 decisions **71** (ep0's DooPlex copy keeps 8 weekly copies; a third place later, R-832), **72** (tester-1's unpinned keys removed via the registrar), **73** (the transcript-exposed Hetzner storage token is not rotated now; R-831).
|
||||
> **Rulings 2026-10-04 (morning) — recorded before the work.** `09` §3 decisions **71** (ep0's DooPlex copy keeps 8 weekly copies; a third place later, R-832), **72** (tester-1's unpinned keys removed via the registrar), **73** (the transcript-exposed Hetzner storage token is not rotated now; R-831), and **74** (the set-aside deletion is the hub's, after a 7-day wait — from the brief's Part E).
|
||||
|
||||
> **Rulings 2026-10-03 (afternoon) — recorded before the work (off-site lock build).** `09` §3 decisions **68** (the box prunes only inside a weekly hub-opened window; option 2 rejected; no box-side retention in the interim), **69** (the hub is the key registrar; the box never receives the sub-account password; the hub stores it encrypted with a key outside the database; daily `authorized_keys` check) and **70** (nightly PBS pull-sync of ep0's `felhom-offsite` to DooPlex; the fence opens for that brief's Part F acts only).
|
||||
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# REPORT — off-site safety finished (decisions 71–74) — 2026-10-04
|
||||
|
||||
Architecture: `07-backup-architecture.md` (custody block, threat rows 9/10), `06` §3.6, `09` §3 decisions 68–74.
|
||||
Baselines (re-verified): controller `c4bf7306371a` (0.289.1), agent `d766666ff8cf`, felhom.eu `710a2505f9b4` (hub
|
||||
0.127.0). Register 330, highest R-830. Rulings recorded first (decisions 71–73, R-831, R-832: `697c2a7`).
|
||||
Evidence: `documentation/audits/offsite-finish-2026-10-04/`.
|
||||
|
||||
## The Part table
|
||||
|
||||
| Part | Result | Notes |
|
||||
|---|---|---|
|
||||
| A — guard fixed, one real window | **done; a window that removes something NOT yet observed** | Controller v0.290.0: a young snapshot superseded the same day is excluded instead of refusing; future-dated / newer-than-hub / above-the-week's-cap still refuse. 3 red-proofs (the demo-hp shape runs; the 13 future fakes refused; above-cap refused). Live: window 2 on demo-hp opened, guard ran with no refusal, closed in 3 s, **127→127 removed nothing**, because every candidate was a young same-day copy. The key file is clean and the hub's before/after check is quiet. Weekly windows **ON** (fleet switch). Next scheduled windows: demo-felhom at its next night run (never had one); demo-hp at the first night run after 2026-10-10 17:36 UTC. The first real removals are expected around 2026-10-11. |
|
||||
| B — copy keeps 8 weekly | **done** | `prune-ep0-copy` keep-weekly 8, all namespaces, daily 07:30 (sync 05:00 — ran OK today); GC Sundays 08:30; `remove-vanished` false. Dry-run **by reasoning**, because PBS has no CLI dry-run for a prune job: nothing to remove (2 snapshots per group, 2 different weeks). 12 GB used. |
|
||||
| C — tester-1's keys | **done** | Through the hub (`POST /offsite/remove-unpinned/tester-1` → `changed: 3`); the check reads 0 lines and raises no alarm. |
|
||||
| D — restore from the copy | **done** | demo-hp, scratch VMID 9299 on `nvme-scratch`: list 2 s, restore **186 s** (15 GB logical, 14 GB on disk), data read by `pct mount` (not started — starting it would run a second demo-hp controller against the hub). Torn down: VMID, storage entry, DooPlex temporary token + ACL. **Trap found → R-834.** |
|
||||
| E — set-aside deletion via the hub | **done** | Hub v0.128.0 + controller v0.290.0. Red-proofs: no deletion before the delay; a cancelled request deletes nothing; a recovery that does not cancel at the hub fails its test. Live on tester-1: naming the live repo was refused; a planted set-aside dir was deleted after the delay and read back absent. The delay was shortened to 3 min for that test only, by manifest config, logged at start-up, and reverted (the new pod logs no override). |
|
||||
| F — releases, floor, golden | **done** | Hub v0.128.0 deployed. Controller v0.290.0 on both demo boxes. Golden 0.290.0 baked (subagent), round-trip sha matches, leak grep 0 with a control, vouched (agent 0.138.0, min_agent 0.131.0). Floor 0.290.0 SERVED. Golden gate OK. |
|
||||
|
||||
## Claims in the brief that turned out wrong
|
||||
|
||||
1. **"The young superseded copies are the only cause of the refusal"** — right for 2026-10-03. But the brief's own "refuse above the weekly cap" would also have refused every honest window: the hub's cap was 40% and an honest week removes about 41%. I raised the hub cap to half (red-proved), and the cap refusal can still wedge after a long gap (R-833).
|
||||
2. **"PBS can prune `ep0-copy` without touching the sync"** — true. They are separate jobs at separate times. PBS has no dry-run for a prune JOB, so the dry-run was done by reasoning.
|
||||
3. **"A demo box's whole-guest backup is in the copy and restorable with its own key"** — true (demo-hp's own `felhom-pbs.enc`). Two things the brief did not expect: `pvesm add pbs` without `--password` fails, and on failure it **deletes** the key files you placed; and the restored config is the production one (`onboot: 1`, the real drive binds) — R-834.
|
||||
4. **"The household's page still names a deletion date"** — it did, during the countdown. After the date (since v0.289) the page showed nothing while nothing was deleted. It now shows the hub's date, and that date is true.
|
||||
5. A live window that removes snapshots could not be shown today. Nothing was old enough. I did not fake the history.
|
||||
|
||||
## Rows
|
||||
|
||||
Closed: **R-823, R-824, R-826, R-827, R-828, R-830**. Opened: **R-831** (the token, waiting on the operator), **R-832**
|
||||
(roadmap P4), **R-833**, **R-834**. R-95 narrowed further. Register **330 → 328**.
|
||||
|
||||
## Teardown, three layers
|
||||
|
||||
- **Machines:** demo-hp has no VMID 9299, no `tmp-dooplex-copy` entry, and only its own `felhom-pbs.*` priv files. tester-1's sub-account holds `.ssh` (an empty key file) and `felhom-repo`; the planted dir was deleted by the hub. Helper scripts were removed from demo-hp. The drill VM is back on `virgin`.
|
||||
- **Host (DooPlex):** **kept on purpose:** the prune job and GC schedule (Part B). Removed: the temporary restore token and its ACL. Shredded in the scratchpad: tester-1's password, its API key, the seal key copy, the restore token.
|
||||
- **Hub:** v0.128.0 at the 7-day delay (the test override was reverted). Weekly windows ON. Floor 0.290.0, golden 0.290.0 vouched.
|
||||
@@ -2,8 +2,27 @@
|
||||
|
||||
**Ready for the first real tester (Tester-2): yes. You confirmed the tunnel route and the connect mails (2026-09-30).**
|
||||
|
||||
**Updated 2026-10-03 (evening): off-site backups a box cannot delete — built and live. Both demo boxes run controller
|
||||
0.289.1 and host agent 0.138.0. Hub 0.127.0. New installs get golden 0.289.1 with agent 0.138.0; every box's floor is 0.289.1.**
|
||||
**Updated 2026-10-04: off-site safety finished. Both demo boxes run controller 0.290.0 and host agent 0.138.0. Hub
|
||||
0.128.0. New installs get golden 0.290.0 with agent 0.138.0; every box's floor is 0.290.0.**
|
||||
|
||||
## Today (2026-10-04): off-site safety finished
|
||||
|
||||
- **The weekly clean-up is ON and no longer stops itself.** The fake-backup check now skips young copies that a manual
|
||||
backup replaced the same day, instead of refusing. I ran one window on demo-hp: no refusal, nothing removed
|
||||
(127 → 127), because every candidate was still young. The first real removals come when those copies are older than
|
||||
8 days — around 11 October. demo-felhom gets its first window at its next night run.
|
||||
- **DooPlex keeps 8 weekly copies of ep0** (your choice). Old copies go weekly; anything ep0 deletes still never reaches
|
||||
the copy by itself. Today's nightly copy ran fine.
|
||||
- **tester-1's 3 old keys are gone.** The daily alarm for tester-1 stopped. (You got one last alarm mail this morning,
|
||||
sent just before I removed them.)
|
||||
- **A restore from the DooPlex copy works.** I restored demo-hp's whole box onto a scratch machine in about 3 minutes,
|
||||
read its data, then deleted it. **One trap found:** a restored box starts automatically and points at the real
|
||||
drives. Run beside the original, that would be two copies of the same box. I switched it off in time; the runbook
|
||||
now warns, and a row asks to make it safe by default.
|
||||
- **"Delete my old set-aside backups" works again.** The hub does it, 7 days after the household's request; the
|
||||
household or you can cancel in that time. Tested on tester-1 with a planted test folder.
|
||||
- **Your answers are recorded**, including that you keep the current Hetzner key for now (a row holds the 3 steps).
|
||||
- **Rows:** 6 closed, 4 opened. The list went from 330 to 328.
|
||||
|
||||
## Today (2026-10-03, evening): your choices A and A — built
|
||||
|
||||
@@ -87,14 +106,8 @@ Your licence decisions are recorded: Emby, Plex and n8n stay. recipe-importer ne
|
||||
|
||||
## What needs you
|
||||
|
||||
0. **Off-site clean-up window: nothing to decide now.** It stays OFF until the next session fixes the too-strict check.
|
||||
**If you do nothing:** no old off-site backup is deleted; storage grows slowly (each household uses under 1 GB).
|
||||
0b. **How much history should DooPlex's ep0 copy keep?** Today it keeps everything and grows every night.
|
||||
- **A — keep the last 8 weekly copies** (recommended): undo up to 2 months; about 4 times ep0's size (ep0 keeps 2).
|
||||
- **B — keep everything:** never loses anything; DooPlex's disk slowly fills (5.5 TB free today).
|
||||
- **If you do nothing:** B — it grows; nothing breaks for months.
|
||||
0c. **tester-1's old keys:** say "remove them" and I clean that storage account's key file through the hub. **If you do
|
||||
nothing:** one alarm mail a day for tester-1.
|
||||
0. **Nothing new from the off-site work.** The Hetzner key change stays your call whenever you want it (3 steps, in
|
||||
the list).
|
||||
1. **plant-it:** keep the hidden template as it is, or remove it entirely (its image no longer exists). **If you say
|
||||
nothing:** it stays hidden; nothing runs it.
|
||||
2. **Send the SparkyFitness request, and ask the Tandoor authors** (the "Before the first paying customer" list).
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -737,6 +737,10 @@ its length, and both fixes cost something the household would notice — operato
|
||||
2026-10-04 (morning).*
|
||||
73. **The Hetzner storage API token shown in the 2026-10-03 session transcript is NOT rotated now — the operator's
|
||||
choice** (R-831 holds the rotation steps). *Operator ruling 2026-10-04 (morning).*
|
||||
74. **A household's "delete my earlier off-site backups" is carried out by the HUB, after a hub-enforced wait of 7
|
||||
days from the box's request (R-823)**; the household (a recovery on the box) or the operator can cancel in that
|
||||
time; the hub deletes only `<repo>.orphaned-<…>`, never the live repository; every request, cancel and deletion is
|
||||
an operator event. *Operator brief 2026-10-04 (Part E).* Built hub v0.128.0 + controller v0.290.0.
|
||||
|
||||
### 2026-09-30 (day) — operator notes, recorded before the work
|
||||
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
## Part A live window on demo-hp 2026-10-04T05:31:42Z
|
||||
grant: {"ok":true}
|
||||
BEFORE
|
||||
count: 118 snapshots
|
||||
plan (remove times, by minute):
|
||||
6 2026-10-03T15:24
|
||||
3 2026-10-03T15:25
|
||||
HTTP 202
|
||||
2026/10/04 05:33:13 controller_image_retention.go:184: [INFO] [stacks] controller image retention: deleted gitea.dooplex.hu/admin/felhom-controller:0.288.0 (3e9d251e1ab7, 410MB) — older than the previous controller and no container uses it (decision 56)
|
||||
2026/10/04 05:33:13 controller_image_retention.go:115: [INFO] [stacks] controller image retention: pass over 3 controller image(s) — running 0.290.0, previous "0.289.1" (by version order (no swap record names one present)), 1 candidate(s), 1 deleted, the rest kept
|
||||
2026/10/04 05:36:16 offbox_window.go:232: [INFO] [offbox] clean-up window 2: the policy removes nothing
|
||||
2026/10/04 05:36:22 offbox.go:1234: [INFO] [offbox] backup OK: 9 app(s) backed up, 127 snapshot(s), 2m24s
|
||||
2026/10/04 05:36:22 night_chain.go:93: [INFO] [night-chain] finished in 4m21s
|
||||
AFTER
|
||||
count: 127 snapshots
|
||||
plan (remove times, by minute):
|
||||
7 2026-10-04T02:16
|
||||
2 2026-10-04T02:17
|
||||
2026/10/04 07:36:14 [WARN] offsitekeys: clean-up window 2 OPENED for demo-hp (key SHA256:16GpHoTF0WHLbj15Pog0mpmDMit4/BLFLRfIqmHJQc8, 127 snapshot(s), max 63 removed, closes by 2026-10-04T05:56:14Z, one-shot=true)
|
||||
2026/10/04 07:36:17 [INFO] offsitekeys: clean-up window 2 CLOSED for demo-hp: outcome=nothing, 127 -> 127 (drop 0, allowed 63)
|
||||
## key check after window 2:
|
||||
Tester-2 lines 0 pinned 0 findings 0
|
||||
demo-felhom lines 1 pinned 1 findings 0
|
||||
demo-hp lines 1 pinned 1 findings 0
|
||||
tester-1 lines 0 pinned 0 findings 0
|
||||
## hub alarms for the window (drop/guard/failed) in the last 15 min: 0
|
||||
## weekly windows ON (fleet switch): {"ok":true}
|
||||
@@ -0,0 +1,22 @@
|
||||
## Part B 2026-10-04T05:14:08Z
|
||||
Prune job created: prune-ep0-copy
|
||||
created
|
||||
+================+=========+==========+====+==========+===========+===========+=============+============+=============+==============+=============+
|
||||
| id | disable | store | ns | schedule | max-depth | keep-last | keep-hourly | keep-daily | keep-weekly | keep-monthly | keep-yearly |
|
||||
+================+=========+==========+====+==========+===========+===========+=============+============+=============+==============+=============+
|
||||
| prune-ep0-copy | | ep0-copy | | 07:30 | | | | | 8 | | |
|
||||
+================+=========+==========+====+==========+===========+===========+=============+============+=============+==============+=============+
|
||||
| id | sync-direction | store | remote | remote-store | schedule | group-filter | rate-in | comment |
|
||||
| ep0-felhom-offsite | | ep0-copy | ep0 | felhom-offsite | 05:00 | all | | decision 70: nightly copy of ep0 felhom-offsite; never removes what ep0 removed |
|
||||
gc-schedule: sun 08:30
|
||||
|
||||
## the copy NOW (what a keep-weekly 8 prune would judge)
|
||||
ns/demo-felhom/ct/9201/2026-09-22T04:12:20Z
|
||||
ns/demo-felhom/ct/9201/2026-09-29T04:16:43Z
|
||||
ns/demo-hp/ct/9201/2026-09-24T20:06:25Z
|
||||
ns/demo-hp/ct/9201/2026-10-01T20:15:29Z
|
||||
dry-run, by reasoning (PBS has no CLI dry-run for a prune JOB): every group holds 2 snapshots in 2 different ISO weeks; keep-weekly 8 keeps the newest of each of the last 8 weeks that have one → both kept → NOTHING would be removed.
|
||||
|
||||
## space
|
||||
12G /mnt/5_hdd/backup/ep0-copy
|
||||
/dev/sda1 9.1T 3.2T 5.5T 37% /mnt/5_hdd
|
||||
@@ -0,0 +1,16 @@
|
||||
## Part C 2026-10-04T05:07:52Z
|
||||
before:
|
||||
Tester-2 lines 0 pinned 0 findings 0
|
||||
demo-felhom lines 1 pinned 1 findings 0
|
||||
demo-hp lines 1 pinned 1 findings 0
|
||||
tester-1 lines 3 pinned 0 findings 3
|
||||
remove-unpinned tester-1: {"changed":3}
|
||||
after (the daily check, by hand):
|
||||
Tester-2 lines 0 pinned 0 findings 0
|
||||
demo-felhom lines 1 pinned 1 findings 0
|
||||
demo-hp lines 1 pinned 1 findings 0
|
||||
tester-1 lines 0 pinned 0 findings 0
|
||||
2026/10/04 07:07:54 [ERROR] offsitekeys: audit tester-1: 3 line(s) can delete off-site history: unpinned SHA256:3UoXpMIvo9gat9AL1380UK39UGAtO2T2CBBllo8n3Mg; unpinned SHA256:Jri1gf2AGHCTj8cJrFSpRj8+BxdY64OQ5Rnms/tbOZI; unpinned SHA256:gAhxqeDkxAPTOOeKQDHBDNe/AYpARkxI8h7Rrc8pLD8
|
||||
2026/10/04 07:07:55 [INFO] Operator email sent for tester-1/offsite_key_unlocked
|
||||
2026/10/04 07:07:56 [INFO] offsitekeys: removed 3 unpinned key line(s) from tester-1's sub-account (u629488-sub4) on the operator's request
|
||||
2026/10/04 07:07:59 [INFO] offsitekeys: audit tester-1: 0 line(s), all pinned append-only
|
||||
@@ -0,0 +1,71 @@
|
||||
## Part D on demo-hp 2026-10-04T05:14:55Z
|
||||
create storage failed: tmp-dooplex-copy: error fetching datastores - 401 Unauthorized
|
||||
(pvesm add refused: it validates with --password on argv; the entry is written to storage.cfg instead, the token already sits in /etc/pve/priv/storage/tmp-dooplex-copy.pw 0600)
|
||||
total 1
|
||||
-rw------- felhom-pbs.enc
|
||||
-rw------- felhom-pbs.pw
|
||||
entry added in 0 s
|
||||
tmp-dooplex-copy: error fetching datastores - 401 Unauthorized
|
||||
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||
tmp-dooplex-copy pbs inactive 0 0 0 0.00%
|
||||
tmp-dooplex-copy: error fetching datastores - 401 Unauthorized
|
||||
listed in 8 s
|
||||
total 2
|
||||
-rw------- felhom-pbs.enc
|
||||
-rw------- felhom-pbs.pw
|
||||
-rw------- tmp-dooplex-copy.enc
|
||||
-rw------- tmp-dooplex-copy.pw
|
||||
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||
tmp-dooplex-copy pbs active 9688301348 3345981544 5853981680 34.54%
|
||||
Volid Format Type Size VMID
|
||||
tmp-dooplex-copy:backup/ct/9201/2026-09-24T20:06:25Z pbs-ct backup 23236593218 9201
|
||||
tmp-dooplex-copy:backup/ct/9201/2026-10-01T20:15:29Z pbs-ct backup 15223817296 9201
|
||||
listed in 2 s
|
||||
Formatting '/mnt/hdd_1/images/9299/vm-9299-disk-1.raw', fmt=raw size=75161927680 preallocation=off
|
||||
Creating filesystem with 18350080 4k blocks and 4587520 inodes
|
||||
Filesystem UUID: 44f4df92-7e77-4da9-9ff0-43f79ad951d3
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
4096000, 7962624, 11239424
|
||||
restoring 'tmp-dooplex-copy:backup/ct/9201/2026-10-01T20:15:29Z' now..
|
||||
Using encryption key from file descriptor..
|
||||
Fingerprint: dd:d1:d8:53:44:62:5e:0b
|
||||
merging backed-up and given configuration..
|
||||
Using encryption key from file descriptor..
|
||||
Fingerprint: dd:d1:d8:53:44:62:5e:0b
|
||||
restore rc=0 in 186 s
|
||||
hostname: demo-hp
|
||||
memory: 25898
|
||||
mp0: nvme-scratch:9299/vm-9299-disk-1.raw,mp=/var/lib/felhom,backup=1,size=70G
|
||||
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
|
||||
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:F8:33:C2,ip=dhcp,type=veth
|
||||
onboot: 1
|
||||
rootfs: nvme-scratch:9299/vm-9299-disk-0.raw,size=32G
|
||||
mp0: nvme-scratch:9299/vm-9299-disk-1.raw,mp=/var/lib/felhom,backup=1,size=70G
|
||||
onboot: 0
|
||||
status: stopped
|
||||
mounted in 1 s at /var/lib/lxc/9299/rootfs
|
||||
rootfs: demo-hp / PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
|
||||
/var/lib/felhom: docker lost+found sys_drive
|
||||
controller data volume: /var/lib/felhom/docker/volumes/felhom-controller-data/_data
|
||||
settings.json mtime: 2026-10-01T20:05:20Z
|
||||
controller image: gitea.dooplex.hu/admin/felhom-controller:0.286.1
|
||||
felhom data size: 14G
|
||||
## live 9201 for comparison:
|
||||
unmounted
|
||||
/opt/docker/stacks -> /opt/docker/stacks
|
||||
docker volumes: 913
|
||||
## Part D teardown 2026-10-04T05:19:41Z
|
||||
purging CT 9299 from related configurations..
|
||||
9299 destroyed in 2 s
|
||||
leftover 9299 image dirs: 0
|
||||
storage entry removed
|
||||
storage.cfg mentions: 0
|
||||
felhom-pbs.enc felhom-pbs.pw
|
||||
felhom-pbs.enc (the box key) untouched
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
DooPlex token + ACL removed
|
||||
tokens left named demohp: 0
|
||||
@@ -0,0 +1,13 @@
|
||||
## Part E live 2026-10-04T05:08:37Z — plant a set-aside dir on tester-1 (u629488-sub4)
|
||||
home: . .. .ssh felhom-repo felhom-repo.orphaned-20261004-test
|
||||
## refusal: the LIVE repository named
|
||||
HTTP 502 registrar failed: offsitekeys: "/home/felhom-repo" is not a set-aside copy of /home/felhom-repo
|
||||
|
||||
## the request (as the box sends it)
|
||||
{"state":"pending","path":"/home/felhom-repo.orphaned-20261004-test","due_at":"2026-10-04T05:11:40Z"}
|
||||
HTTP 200
|
||||
## +70 s (before the 3-min delay): status {"state":"pending","path":"/home/felhom-repo.orphaned-20261004-test","due_at":"2026-10-04T05:11:40Z"}; dir: marker.txt
|
||||
## +220 s (after the delay): status {"state":"deleted","path":"/home/felhom-repo.orphaned-20261004-test","due_at":"2026-10-04T05:11:40Z"}; dir: /usr/bin/ls: cannot access '/home/felhom-repo.orphaned-20261004-test': No such file or directory
|
||||
home: . .. .ssh felhom-repo
|
||||
2026/10/04 07:12:02 [WARN] offsitekeys: DELETED tester-1's set-aside off-site copy /home/felhom-repo.orphaned-20261004-test (requested 2026-10-04T05:08:40Z, due 2026-10-04T05:11:40Z)
|
||||
## revert 2026-10-04T05:13:54Z: manifest commit 7a0d0c0 removed OFFSITE_ABANDON_DELAY; new pod started 05:12:51Z logs NO "OFFSITE_ABANDON_DELAY" line (= 7-day default); ArgoCD Synced at 7a0d0c0
|
||||
@@ -26,6 +26,21 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-04 — off-site safety finished (hub v0.128.0, controller v0.290.0, decisions 71–74)
|
||||
|
||||
> Evidence: `audits/offsite-finish-2026-10-04/`.
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-824** | **The fake-snapshot guard refused every window after a manual run.** Fixed controller v0.290.0: a young snapshot superseded the same day in its group is EXCLUDED (removed later, once old); any other young removal, a future date or a plan above the week's cap still refuses. Live: window 2 on demo-hp ran without refusal (outcome `nothing`, 127→127, the only candidates were young same-day copies). Weekly windows ON. **Reasoning kept: a guard that refuses the honest case is switched off within a fortnight — exclude the benign shape, keep refusing the poisoning shape.** | CLOSED 2026-10-04 — FIXED controller v0.290.0, live window without refusal; a window that removes something not yet observed (R-95) | `partA-live-window.txt`, `red-proofs-controller.txt` RPC1–2 |
|
||||
| **R-823** | **The household's set-aside deletion stopped happening on the append-only tier.** Built (decision 74): the box hands the due request to the hub; the hub deletes only `<repo>.orphaned-*` after 7 days unless the household or operator cancels; the page keeps a dated, cancellable deletion. Live on tester-1: the live repo named → refused; a planted set-aside dir deleted after the (test-shortened, logged, reverted) delay, read back absent. **Reasoning kept: the only deletion a broken-into box can trigger now waits a week under operator mails.** | CLOSED 2026-10-04 — BUILT hub v0.128.0 + controller v0.290.0, proven live | `partE-live-tester1.txt`, `red-proofs-hub.txt` RPH1–2, `red-proofs-controller.txt` RPC3 |
|
||||
| **R-826** | **Sub-accounts kept every earlier box's key unpinned.** tester-1's 3 lines removed through the hub registrar (decision 72, `POST /offsite/remove-unpinned`); the daily check reads 0 lines, no alarm. Demo boxes' stale lines went on 2026-10-03. | CLOSED 2026-10-04 | `partC-tester1-keys.txt` |
|
||||
| **R-827** | **The daily key check created `.ssh` when absent.** Fixed hub v0.128.0: only the write path creates it. | CLOSED 2026-10-04 — FIXED hub v0.128.0 (red-proved) | `red-proofs-hub.txt` RPH3 |
|
||||
| **R-828** | **The ep0 copy grew without bound.** Decision 71: prune job `prune-ep0-copy` keep-weekly 8 (all namespaces) daily 07:30 after the 05:00 pull; GC Sundays 08:30; `remove-vanished` stays false. Dry-run by reasoning: nothing to remove yet (2 snapshots per group, 2 weeks). | CLOSED 2026-10-04 | `partB-copy-retention.txt`; `runbooks/ep0-datastore-copy.md` |
|
||||
| **R-830** | **The restore route from the DooPlex copy had never been walked.** Walked on demo-hp with a scratch VMID: list 2 s, restore 186 s, data read via `pct mount`, all torn down. DooPlex PBS is LAN-only; the options for another network are written. **Reasoning kept: a copy that was never restored is not proven — and the restore found a trap (R-834).** | CLOSED 2026-10-04 — WALKED (route 1); route 2 not walked | `partD-restore-walk.txt`; `runbooks/ep0-datastore-copy.md` |
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-03 (evening) — the off-site lock built (hub v0.127.0, controller v0.289.0/0.289.1, decisions 68–70)
|
||||
|
||||
> Evidence: `audits/offsite-lock-build-2026-10-03/`.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -14,6 +14,8 @@ household's whole-box backups are encrypted with that household's own `encryptio
|
||||
| DooPlex PBS | datastore `ep0-copy` at `/mnt/5_hdd/backup/ep0-copy` | the copy |
|
||||
| DooPlex PBS | sync job `ep0-felhom-offsite`, daily 05:00, `remove-vanished false` | the nightly pull (ep0's prune runs 03:30). **It never removes what ep0 removed** — a deletion on ep0 does not reach the copy |
|
||||
| DooPlex PBS | verify job `verify-ep0-copy`, Saturdays 06:30 | reads the copy back |
|
||||
| DooPlex PBS | prune job `prune-ep0-copy`, daily 07:30, **keep-weekly 8**, all namespaces (decision 71) | keeps the last 8 weekly copies per group; runs after the 05:00 pull, never during it |
|
||||
| DooPlex PBS | garbage collection on `ep0-copy`, Sundays 08:30 | frees the chunks the prune released |
|
||||
| DooPlex PBS | notification target `felhom-operator` (SMTP via Resend → admin@felhom.eu) + matcher `felhom-operator-errors` (every error) | a failed pull or verify reaches the operator. Proven 2026-10-03 with a test mail |
|
||||
|
||||
Secrets, all out of git: the ep0 token secret in `/etc/proxmox-backup/remote.cfg` (root:backup 0640, base64 —
|
||||
@@ -31,20 +33,46 @@ sudo find /mnt/5_hdd/backup/ep0-copy/ns -mindepth 4 -maxdepth 4 -type d | sort #
|
||||
## If ep0 is lost — restore a household's whole box from the DooPlex copy
|
||||
|
||||
The copy is a normal PBS datastore. Two routes, both needing the household's PBS `encryption-key` (escrowed,
|
||||
recovered with the household's recovery code — the same as restoring from ep0):
|
||||
recovered with the household's recovery code — the same as restoring from ep0).
|
||||
|
||||
1. **Point the box's host at DooPlex instead of ep0.** On the household's Proxmox host, add a PBS storage for
|
||||
DooPlex's PBS (`ep0-copy`, namespace = the customer id) with the household's key, then restore the CT from it as
|
||||
from ep0. DooPlex's PBS must be reachable from the host (it is not public today — the operator decides the route
|
||||
at the time: a temporary tunnel, or a new endpoint).
|
||||
2. **Rebuild the endpoint.** Provision a new ep0 (06 §5), then pull back: on the new ep0 add DooPlex as a remote
|
||||
and run `proxmox-backup-manager pull <dooplex-remote> ep0-copy felhom-offsite`. Every box then reconnects as before.
|
||||
### Route 1 — the host reads DooPlex directly. **WALKED 2026-10-04 on demo-hp** (`audits/offsite-finish-2026-10-04/partD-restore-walk.txt`)
|
||||
|
||||
Do **not** prune or garbage-collect the copy tighter than ep0's own retention. The copy has no prune job today and
|
||||
grows with every nightly backup (R-828).
|
||||
1. **On DooPlex:** a read-only token for the restore (`proxmox-backup-manager user generate-token root@pam <name>`, then
|
||||
`acl update /datastore/ep0-copy DatastoreReader --auth-id 'root@pam!<name>'`). Keep the secret in a file only.
|
||||
2. **On the host:** put the token in `/etc/pve/priv/storage/<id>.pw` (0600) and the household's PBS key in
|
||||
`/etc/pve/priv/storage/<id>.enc`, then append the storage entry to `/etc/pve/storage.cfg`:
|
||||
`pbs: <id>` / `datastore ep0-copy` / `server <DooPlex>` / `content backup` / `fingerprint <DooPlex PBS cert>` /
|
||||
`namespace <customer>` / `username root@pam!<name>`.
|
||||
**Do not use `pvesm add pbs` without `--password`:** it validates with the password from its command line, fails 401,
|
||||
and on failure DELETES the `.pw`/`.enc` files you placed (measured). Passing `--password` puts the token on argv.
|
||||
3. `pvesm list <id>` → the household's snapshots (measured: 2 s). `pct restore <scratch VMID> <id>:backup/ct/<vmid>/<time>
|
||||
--storage <dir storage> --unique 1` (measured: **186 s for a 15 GB-logical / 14 GB-on-disk backup** over the LAN, key
|
||||
fingerprint printed by the restore).
|
||||
4. **⚠ BEFORE ANYTHING ELSE — the restored config is the PRODUCTION one:** `onboot: 1`, `mp8` bound to the host's REAL
|
||||
household drives (`/mnt/felhom-drives`) and `mp9` to the original guest's bootstrap. Starting it, or a host reboot,
|
||||
runs a second controller for the same household against the same drives. On a restore BESIDE the original:
|
||||
`pct set <vmid> --onboot 0 --delete mp8,mp9` immediately (R-834). On a true replacement host, where the original is
|
||||
gone, the binds are what you want.
|
||||
5. Read the data without starting it: `pct mount <vmid>` → `/var/lib/lxc/<vmid>/rootfs` (measured: 1 s; rootfs, the
|
||||
controller data volume and `/var/lib/felhom` present, `settings.json` dated 10 min before the backup) → `pct unmount`.
|
||||
6. Teardown: `pct destroy <vmid> --purge`; `pvesm remove <id>` (removes the entry and its priv files — never the
|
||||
household's own `felhom-pbs.enc`); on DooPlex delete the token and its ACL.
|
||||
|
||||
**Reachability.** DooPlex's PBS (`:8007`) is reachable on DooPlex's LAN only. For a host on another network — a
|
||||
household's home — the options, none built (the operator decides at the time, R-832 is the long-term answer):
|
||||
(a) a temporary SSH forward from the host to DooPlex, as DooPlex already does to ep0 (needs an SSH key on DooPlex for
|
||||
that host); (b) a WireGuard peer on DooPlex's existing tailscale/k3s network for the duration; (c) route 2 below —
|
||||
rebuild an endpoint and pull back, so every host reconnects the usual way.
|
||||
|
||||
### Route 2 — rebuild the endpoint. Not walked.
|
||||
|
||||
Provision a new ep0 (06 §5), then on it add DooPlex as a remote and run
|
||||
`proxmox-backup-manager pull <dooplex-remote> ep0-copy felhom-offsite`. Every box then reconnects as before.
|
||||
|
||||
Do **not** prune the copy tighter than decision 71 (8 weekly copies); never tighter than ep0's own retention.
|
||||
|
||||
## Remove
|
||||
|
||||
`sudo proxmox-backup-manager sync-job remove ep0-felhom-offsite; … verify-job remove verify-ep0-copy;`
|
||||
`sudo proxmox-backup-manager sync-job remove ep0-felhom-offsite; … verify-job remove verify-ep0-copy; … prune-job remove prune-ep0-copy;`
|
||||
`sudo systemctl disable --now felhom-ep0-pbs-tunnel.service`; on ep0
|
||||
`proxmox-backup-manager user delete-token root@pam dooplex-sync`. The datastore's bytes stay until removed by hand.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
Round-trip of the published golden 0.290.0 (2026-10-04, from DooPlex, anonymous GET)
|
||||
URL: https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.290.0/golden.tar.zst
|
||||
HTTP: 200, size_download=652274410 bytes
|
||||
sha256 (downloaded): bbc1ba619710d67432a802787f00f23ccfb167f3fe1113de45c8159001ddea32
|
||||
GOLDEN_SHA256 (bake.log line 322): bbc1ba619710d67432a802787f00f23ccfb167f3fe1113de45c8159001ddea32
|
||||
Result: MATCH
|
||||
@@ -0,0 +1,6 @@
|
||||
## vouch + floor 2026-10-04T05:37:43Z
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /configuration?flash=artifacts_set
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /configuration?flash=floor_set
|
||||
2026/10/04 07:38:12 [INFO] managed floor SERVED for demo-felhom: floor 0.290.0, agent requirement "0.131.0" from manifest (golden 0.290.0)
|
||||
@@ -0,0 +1,57 @@
|
||||
# Golden 0.290.0 — bake + publish, 2026-10-04
|
||||
|
||||
Procedure: `documentation/runbooks/RUNBOOK-manual-build.md` §4.0 and §4.1 steps 1–4, in the drill VM
|
||||
on DooPlex. Step 5 (vouching in the hub) was **not** done; it is the operator's act.
|
||||
|
||||
- Controller image: `gitea.dooplex.hu/admin/felhom-controller:0.290.0`
|
||||
- Build script: `felhom-agent/configs/build-golden.sh` v3.0.0, agent repo `main` = `d766666ff8cf`
|
||||
(tree clean, HEAD == origin/main); sha256 of the copy in the VM matched the repo file
|
||||
(`e4c9ede772e7…`).
|
||||
- Drill VM: reverted to `virgin`, cold-booted per §4.0; `pveversion` = `pve-manager/9.2.2`.
|
||||
- Step 2: `pveam update` → `update successful`; template `debian-13-standard_13.6-1_amd64.tar.zst`
|
||||
(the only `_amd64` debian-13 entry), downloaded with checksum verified.
|
||||
- Pre-gate: `GET …/generic/felhom-golden/0.290.0/golden.tar.zst` → **404** before the bake.
|
||||
- Token: copied file → file (`scp`); launched via the in-VM runner script as transient unit
|
||||
`golden-bake`. `systemctl show golden-bake -p Environment -p ExecStart | grep -c -F <token>` = **0**
|
||||
(control: same output with the token appended = **1**).
|
||||
|
||||
## Result
|
||||
|
||||
```
|
||||
GOLDEN_VERSION=0.290.0
|
||||
GOLDEN_SHA256=bbc1ba619710d67432a802787f00f23ccfb167f3fe1113de45c8159001ddea32
|
||||
```
|
||||
|
||||
## Pass markers (quoted verbatim from `bake.log`)
|
||||
|
||||
```
|
||||
82: docker OK (overlay2; data-root /var/lib/docker)
|
||||
313:INFO: including mount point rootfs ('/') in backup
|
||||
314:INFO: including mount point mp0 ('/var/lib/felhom') in backup
|
||||
319:[golden] pre-delete existing: HTTP 404 (404/204 expected)
|
||||
320:[golden] upload OK (HTTP 201)
|
||||
```
|
||||
|
||||
`grep -E 'excluding|FATAL' bake.log` → no matches.
|
||||
|
||||
## Token-leak grep
|
||||
|
||||
On the copy in this directory (the one that would be committed):
|
||||
`grep -c -F "$(cat ~/.gitea-token)" bake.log` = **0**.
|
||||
Positive control: a throwaway copy with the token appended → **1**; the copy was `shred -u`'d.
|
||||
Same 0 / 1 result on the scratch copy pulled straight from the VM.
|
||||
|
||||
## Round trip
|
||||
|
||||
See `02-round-trip.txt`: the published package downloaded anonymously (HTTP 200, 652274410 bytes)
|
||||
hashes to `bbc1ba619710d67432a802787f00f23ccfb167f3fe1113de45c8159001ddea32` — **matches**
|
||||
GOLDEN_SHA256.
|
||||
|
||||
## Teardown state
|
||||
|
||||
- `pct destroy 9100 --purge` → rc 0 (both LVs removed); `pct list` empty.
|
||||
- `/root/.gitea-token`, `/root/bake-run.sh`, `/root/bake.log` in the VM: `shred -u`, confirmed absent.
|
||||
- VM powered off; no `qemu-system-x86` process remained.
|
||||
- `qemu-img snapshot -a virgin drill.qcow2` → OK; snapshot list shows only `virgin`.
|
||||
- Hub, k3s, demo boxes, ep0, PBS, Storage Box: not touched.
|
||||
- `df -h`: `/mnt/5_hdd` 37 %, `/` 53 % (before and after).
|
||||
@@ -0,0 +1,324 @@
|
||||
[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.290.0
|
||||
[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
|
||||
Logical volume "vm-9100-disk-0" created.
|
||||
Logical volume pve/vm-9100-disk-0 changed.
|
||||
Creating filesystem with 8388608 4k blocks and 2097152 inodes
|
||||
Filesystem UUID: ee244db5-91fd-4dd5-9688-ae517c9adec8
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
4096000, 7962624
|
||||
Logical volume "vm-9100-disk-1" created.
|
||||
Logical volume pve/vm-9100-disk-1 changed.
|
||||
Creating filesystem with 6291456 4k blocks and 1572864 inodes
|
||||
Filesystem UUID: 9734adf2-89c8-4872-80bd-6d4b972de454
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
|
||||
Total bytes read: 553512960 (528MiB, 94MiB/s)
|
||||
Detected container architecture: amd64
|
||||
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
|
||||
done: SHA256:oQ/6nh/OvChwUIQyik7Lu/ZFPGRL4aBRFI5E7i6dOZs root@felhom-golden
|
||||
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
|
||||
done: SHA256:JaK7jqkkauImanKg39RSDa10j731klU1DxIv0OJ3G9o root@felhom-golden
|
||||
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
|
||||
done: SHA256:NXfZqPe+ERU32XuPvri80+W6rOpUfnZloxuAoxwUUuE root@felhom-golden
|
||||
[golden] starting + installing Docker (official repo, trixie channel) …
|
||||
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = (unset),
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to the standard locale ("C").
|
||||
locale: Cannot set LC_CTYPE to default locale: No such file or directory
|
||||
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
|
||||
locale: Cannot set LC_ALL to default locale: No such file or directory
|
||||
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = (unset),
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to the standard locale ("C").
|
||||
locale: Cannot set LC_CTYPE to default locale: No such file or directory
|
||||
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
|
||||
locale: Cannot set LC_ALL to default locale: No such file or directory
|
||||
[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
|
||||
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
|
||||
[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
|
||||
Unable to find image 'hello-world:latest' locally
|
||||
latest: Pulling from library/hello-world
|
||||
4f55086f7dd0: Pulling fs layer
|
||||
4f55086f7dd0: Download complete
|
||||
4f55086f7dd0: Pull complete
|
||||
Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8
|
||||
Status: Downloaded newer image for hello-world:latest
|
||||
docker OK (overlay2; data-root /var/lib/docker)
|
||||
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
|
||||
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
|
||||
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
|
||||
[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.290.0 (no registry cred at deploy) …
|
||||
|
||||
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
|
||||
Configure a credential helper to remove this warning. See
|
||||
https://docs.docker.com/go/credential-store/
|
||||
|
||||
0.290.0: Pulling from admin/felhom-controller
|
||||
774043ccc8cc: Pulling fs layer
|
||||
ab6b448d4be9: Pulling fs layer
|
||||
23a5bfa58353: Pulling fs layer
|
||||
862a57157567: Pulling fs layer
|
||||
abd1be1c6419: Pulling fs layer
|
||||
44a501e919e4: Pulling fs layer
|
||||
862a57157567: Waiting
|
||||
abd1be1c6419: Waiting
|
||||
44a501e919e4: Waiting
|
||||
23a5bfa58353: Verifying Checksum
|
||||
23a5bfa58353: Download complete
|
||||
862a57157567: Verifying Checksum
|
||||
862a57157567: Download complete
|
||||
774043ccc8cc: Verifying Checksum
|
||||
774043ccc8cc: Download complete
|
||||
abd1be1c6419: Verifying Checksum
|
||||
abd1be1c6419: Download complete
|
||||
44a501e919e4: Verifying Checksum
|
||||
44a501e919e4: Download complete
|
||||
ab6b448d4be9: Verifying Checksum
|
||||
ab6b448d4be9: Download complete
|
||||
774043ccc8cc: Pull complete
|
||||
ab6b448d4be9: Pull complete
|
||||
23a5bfa58353: Pull complete
|
||||
862a57157567: Pull complete
|
||||
abd1be1c6419: Pull complete
|
||||
44a501e919e4: Pull complete
|
||||
Digest: sha256:51231c330050c27e97628bcac69db1a709a9347dbb011ace8a9cedf8c47f1b4d
|
||||
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.290.0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.290.0
|
||||
[golden] asking the controller which infra images it manages …
|
||||
[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 …
|
||||
v3.6.7: Pulling from library/traefik
|
||||
589002ba0eae: Pulling fs layer
|
||||
ef63511ea6cc: Pulling fs layer
|
||||
0738e5cb835e: Pulling fs layer
|
||||
3e6813f70c64: Pulling fs layer
|
||||
3e6813f70c64: Waiting
|
||||
ef63511ea6cc: Download complete
|
||||
589002ba0eae: Verifying Checksum
|
||||
589002ba0eae: Download complete
|
||||
3e6813f70c64: Verifying Checksum
|
||||
3e6813f70c64: Download complete
|
||||
0738e5cb835e: Verifying Checksum
|
||||
0738e5cb835e: Download complete
|
||||
589002ba0eae: Pull complete
|
||||
ef63511ea6cc: Pull complete
|
||||
0738e5cb835e: Pull complete
|
||||
3e6813f70c64: Pull complete
|
||||
Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a
|
||||
Status: Downloaded newer image for traefik:v3.6.7
|
||||
docker.io/library/traefik:v3.6.7
|
||||
2026.6.0: Pulling from cloudflare/cloudflared
|
||||
47de5dd0b812: Pulling fs layer
|
||||
c172f21841df: Pulling fs layer
|
||||
99515e7b4d35: Pulling fs layer
|
||||
99ba982a9142: Pulling fs layer
|
||||
d6b1b89eccac: Pulling fs layer
|
||||
2780920e5dbf: Pulling fs layer
|
||||
7c12895b777b: Pulling fs layer
|
||||
3214acf345c0: Pulling fs layer
|
||||
52630fc75a18: Pulling fs layer
|
||||
dd64bf2dd177: Pulling fs layer
|
||||
b839dfae01f6: Pulling fs layer
|
||||
ebddc55facdc: Pulling fs layer
|
||||
bdfd7f7e5bf6: Pulling fs layer
|
||||
2d4d7adf6272: Pulling fs layer
|
||||
40008157d8d2: Pulling fs layer
|
||||
bd8962e29291: Pulling fs layer
|
||||
cac2ae0193cb: Pulling fs layer
|
||||
74d1dac84ecc: Pulling fs layer
|
||||
7c12895b777b: Waiting
|
||||
3214acf345c0: Waiting
|
||||
52630fc75a18: Waiting
|
||||
dd64bf2dd177: Waiting
|
||||
b839dfae01f6: Waiting
|
||||
ebddc55facdc: Waiting
|
||||
bdfd7f7e5bf6: Waiting
|
||||
2d4d7adf6272: Waiting
|
||||
40008157d8d2: Waiting
|
||||
bd8962e29291: Waiting
|
||||
cac2ae0193cb: Waiting
|
||||
74d1dac84ecc: Waiting
|
||||
99ba982a9142: Waiting
|
||||
d6b1b89eccac: Waiting
|
||||
2780920e5dbf: Waiting
|
||||
47de5dd0b812: Verifying Checksum
|
||||
c172f21841df: Verifying Checksum
|
||||
c172f21841df: Download complete
|
||||
99515e7b4d35: Verifying Checksum
|
||||
99515e7b4d35: Download complete
|
||||
99ba982a9142: Verifying Checksum
|
||||
99ba982a9142: Download complete
|
||||
d6b1b89eccac: Verifying Checksum
|
||||
d6b1b89eccac: Download complete
|
||||
47de5dd0b812: Pull complete
|
||||
2780920e5dbf: Verifying Checksum
|
||||
2780920e5dbf: Download complete
|
||||
7c12895b777b: Verifying Checksum
|
||||
7c12895b777b: Download complete
|
||||
3214acf345c0: Verifying Checksum
|
||||
3214acf345c0: Download complete
|
||||
52630fc75a18: Verifying Checksum
|
||||
52630fc75a18: Download complete
|
||||
dd64bf2dd177: Verifying Checksum
|
||||
dd64bf2dd177: Download complete
|
||||
b839dfae01f6: Verifying Checksum
|
||||
b839dfae01f6: Download complete
|
||||
ebddc55facdc: Verifying Checksum
|
||||
ebddc55facdc: Download complete
|
||||
c172f21841df: Pull complete
|
||||
bdfd7f7e5bf6: Verifying Checksum
|
||||
bdfd7f7e5bf6: Download complete
|
||||
40008157d8d2: Verifying Checksum
|
||||
40008157d8d2: Download complete
|
||||
bd8962e29291: Verifying Checksum
|
||||
bd8962e29291: Download complete
|
||||
2d4d7adf6272: Verifying Checksum
|
||||
2d4d7adf6272: Download complete
|
||||
cac2ae0193cb: Verifying Checksum
|
||||
cac2ae0193cb: Download complete
|
||||
74d1dac84ecc: Verifying Checksum
|
||||
74d1dac84ecc: Download complete
|
||||
99515e7b4d35: Pull complete
|
||||
99ba982a9142: Pull complete
|
||||
d6b1b89eccac: Pull complete
|
||||
2780920e5dbf: Pull complete
|
||||
7c12895b777b: Pull complete
|
||||
3214acf345c0: Pull complete
|
||||
52630fc75a18: Pull complete
|
||||
dd64bf2dd177: Pull complete
|
||||
b839dfae01f6: Pull complete
|
||||
ebddc55facdc: Pull complete
|
||||
bdfd7f7e5bf6: Pull complete
|
||||
2d4d7adf6272: Pull complete
|
||||
40008157d8d2: Pull complete
|
||||
bd8962e29291: Pull complete
|
||||
cac2ae0193cb: Pull complete
|
||||
74d1dac84ecc: Pull complete
|
||||
Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f
|
||||
Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0
|
||||
docker.io/cloudflare/cloudflared:2026.6.0
|
||||
1.3.3-stable: Pulling from gtstef/filebrowser
|
||||
6a0ac1617861: Pulling fs layer
|
||||
ef8806083e82: Pulling fs layer
|
||||
b74107c861c7: Pulling fs layer
|
||||
adc935def003: Pulling fs layer
|
||||
4f4fb700ef54: Pulling fs layer
|
||||
18695ccc900a: Pulling fs layer
|
||||
45d119d5c397: Pulling fs layer
|
||||
dac52db4fc51: Pulling fs layer
|
||||
6d598f86b2f2: Pulling fs layer
|
||||
8aa349c8396c: Pulling fs layer
|
||||
dac52db4fc51: Waiting
|
||||
6d598f86b2f2: Waiting
|
||||
8aa349c8396c: Waiting
|
||||
adc935def003: Waiting
|
||||
4f4fb700ef54: Waiting
|
||||
18695ccc900a: Waiting
|
||||
45d119d5c397: Waiting
|
||||
6a0ac1617861: Download complete
|
||||
adc935def003: Verifying Checksum
|
||||
adc935def003: Download complete
|
||||
4f4fb700ef54: Verifying Checksum
|
||||
4f4fb700ef54: Download complete
|
||||
b74107c861c7: Verifying Checksum
|
||||
b74107c861c7: Download complete
|
||||
45d119d5c397: Verifying Checksum
|
||||
45d119d5c397: Download complete
|
||||
6a0ac1617861: Pull complete
|
||||
dac52db4fc51: Download complete
|
||||
18695ccc900a: Verifying Checksum
|
||||
18695ccc900a: Download complete
|
||||
ef8806083e82: Verifying Checksum
|
||||
ef8806083e82: Download complete
|
||||
6d598f86b2f2: Verifying Checksum
|
||||
6d598f86b2f2: Download complete
|
||||
8aa349c8396c: Verifying Checksum
|
||||
8aa349c8396c: Download complete
|
||||
ef8806083e82: Pull complete
|
||||
b74107c861c7: Pull complete
|
||||
adc935def003: Pull complete
|
||||
4f4fb700ef54: Pull complete
|
||||
18695ccc900a: Pull complete
|
||||
45d119d5c397: Pull complete
|
||||
dac52db4fc51: Pull complete
|
||||
6d598f86b2f2: Pull complete
|
||||
8aa349c8396c: Pull complete
|
||||
Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c
|
||||
Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable
|
||||
docker.io/gtstef/filebrowser:1.3.3-stable
|
||||
1.1.0: Pulling from admin/felhom-samba
|
||||
897d797d2723: Pulling fs layer
|
||||
3051591aa250: Pulling fs layer
|
||||
ce57a3f93416: Pulling fs layer
|
||||
fb94eeec2fe1: Pulling fs layer
|
||||
fb94eeec2fe1: Waiting
|
||||
ce57a3f93416: Verifying Checksum
|
||||
ce57a3f93416: Download complete
|
||||
fb94eeec2fe1: Verifying Checksum
|
||||
fb94eeec2fe1: Download complete
|
||||
897d797d2723: Verifying Checksum
|
||||
897d797d2723: Download complete
|
||||
3051591aa250: Verifying Checksum
|
||||
3051591aa250: Download complete
|
||||
897d797d2723: Pull complete
|
||||
3051591aa250: Pull complete
|
||||
ce57a3f93416: Pull complete
|
||||
fb94eeec2fe1: Pull complete
|
||||
Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
|
||||
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
|
||||
gitea.dooplex.hu/admin/felhom-samba:1.1.0
|
||||
[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
|
||||
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
|
||||
[golden] baking the first-boot SSH host-key regeneration unit (F3) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
|
||||
[golden] identity-clean + minimize …
|
||||
[golden] stop + archive …
|
||||
INFO: including mount point rootfs ('/') in backup
|
||||
INFO: including mount point mp0 ('/var/lib/felhom') in backup
|
||||
INFO: archive file size: 622MB
|
||||
INFO: Finished Backup of VM 9100 (00:00:29)
|
||||
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_10_04-07_34_22.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
|
||||
[golden] publishing golden (652274410 bytes, sha256 bbc1ba619710d674…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.290.0/golden.tar.zst
|
||||
[golden] pre-delete existing: HTTP 404 (404/204 expected)
|
||||
[golden] upload OK (HTTP 201)
|
||||
GOLDEN_VERSION=0.290.0
|
||||
GOLDEN_SHA256=bbc1ba619710d67432a802787f00f23ccfb167f3fe1113de45c8159001ddea32
|
||||
[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.290.0 / bbc1ba619710d67432a802787f00f23ccfb167f3fe1113de45c8159001ddea32
|
||||
[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
|
||||
Reference in New Issue
Block a user