R-232/R-173: hub restored into a throwaway k3s (runbook §3 steps 4–5 proven, corrected); R-173, R-861, R-518 closed; R-921, R-922 filed; STATUS, capability map, report
gates / gates (push) Successful in 5m23s
gates / gates (push) Successful in 5m23s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
## 2026-10-09T08:42:44Z fetch the k3s image + airgap set (bench network, before isolation)
|
||||
docker.io/rancher/k3s:v1.33.6-k3s1
|
||||
total 140296
|
||||
drwxr-xr-x 2 root root 4096 Oct 9 08:36 .
|
||||
drwxr-xr-x 3 root root 4096 Oct 9 08:36 ..
|
||||
-rw-r--r-- 1 root root 143649404 Oct 9 08:36 k3s-airgap-images-amd64.tar.zst
|
||||
pd-net internal=true
|
||||
## after ~15 s
|
||||
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
|
||||
pd-k3s Ready control-plane,master 4s v1.33.6+k3s1 172.30.0.10 <none> K3s v1.33.6+k3s1 7.0.14-20-pve containerd://2.1.5-k3s1.33
|
||||
No resources found
|
||||
time="2026-10-09T08:42:48Z" level=error msg="Sending HTTP/1.1 503 response to 127.0.0.1:48848: runtime core not ready"
|
||||
## 2026-10-09T08:44:46Z fetch the k3s image + airgap set (bench network, before isolation)
|
||||
docker.io/rancher/k3s:v1.33.6-k3s1
|
||||
total 140296
|
||||
drwxr-xr-x 2 root root 4096 Oct 9 08:36 .
|
||||
drwxr-xr-x 3 root root 4096 Oct 9 08:44 ..
|
||||
-rw-r--r-- 1 root root 143649404 Oct 9 08:36 k3s-airgap-images-amd64.tar.zst
|
||||
pd-net internal=true
|
||||
pd-k3s Up 55 seconds
|
||||
## after ~15 s
|
||||
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
|
||||
pd-k3s Ready control-plane,master 49s v1.33.6+k3s1 172.30.0.10 <none> K3s v1.33.6+k3s1 7.0.14-20-pve containerd://2.1.5-k3s1.33
|
||||
NAMESPACE NAME READY STATUS RESTARTS AGE
|
||||
kube-system coredns-6d668d687-bvh4d 1/1 Running 0 43s
|
||||
kube-system local-path-provisioner-869c44bfbd-rf27x 1/1 Running 0 43s
|
||||
time="2026-10-09T08:44:50Z" level=error msg="Sending HTTP/1.1 503 response to 127.0.0.1:57484: runtime core not ready"
|
||||
@@ -0,0 +1,19 @@
|
||||
## 2026-10-09T08:43:12Z hub DB restore on DooPlex, read-only token (as felhom-hub-db-restore-test)
|
||||
newest: host/dooplex-hub/2026-10-09T00:31:46Z
|
||||
restore complete (380.047 MiB processed in 4.5s, average 84.169 MiB/s)
|
||||
-rw------- 1 root root 398508032 Oct 9 02:31 /var/lib/felhom-hub-backup/partd/out/hub.db
|
||||
integrity: ok
|
||||
Error: in prepare, no such table: customers
|
||||
hosts: 4 customers:
|
||||
Error: in prepare, no such table: customers
|
||||
|
||||
Error: in prepare, no such column: id
|
||||
select id from hosts order by id
|
||||
^--- error here
|
||||
|
||||
sealed console passwords: 4
|
||||
seal key file bytes: 64 (from Secret/offsite-secret-key, the value the operator saved 2026-10-05; not printed)
|
||||
## (corrected queries: no customers table; customers live in customer_configs)
|
||||
customer_configs: 4
|
||||
customer ids: Tester-2 demo-felhom demo-hp tester-1
|
||||
hosts: Tester-2-be8404→Tester-2 demo-felhom-8363b5→demo-felhom demo-hp-bb76ea→demo-hp tester-1-d70be4→tester-1
|
||||
@@ -0,0 +1,5 @@
|
||||
## 2026-10-09T08:44:01Z transfer to bench 9401
|
||||
-rw------- 1 root root 26974208 Oct 9 08:44 /root/pd/hub-image.tar
|
||||
95b67d69afdeddaf
|
||||
DooPlex side: 95b67d69afdeddaf
|
||||
64
|
||||
@@ -0,0 +1,49 @@
|
||||
## 2026-10-09T08:45:51Z system pods
|
||||
coredns-6d668d687-bvh4d Running
|
||||
local-path-provisioner-869c44bfbd-rf27x Running
|
||||
## import the hub image (by file; k3s cannot pull)
|
||||
Importing elapsed: 1.7 s total: 0.0 B (0.0 B/s)
|
||||
helper image: docker.io/rancher/mirrored-library-busybox:1.36.1
|
||||
## secrets: the REAL seal key (step 4: same value); throwaway dummies for mail, report API, registry
|
||||
secret/offsite-secret-key created
|
||||
secret/resend-api created
|
||||
secret/report-api created
|
||||
secret/gitea-creds created
|
||||
persistentvolumeclaim/hub-data created
|
||||
configmap/hub-config created
|
||||
deployment.apps/hub created
|
||||
service/hub created
|
||||
## first start on an EMPTY volume (after ~20 s)
|
||||
hub-7f775576fd-fmddj 1/1 Running 0 16s
|
||||
## step 5: scale to 0, copy the restored hub.db in with a helper pod, delete -wal/-shm, scale to 1
|
||||
deployment.apps/hub scaled
|
||||
pod/hub-7f775576fd-fmddj condition met
|
||||
pod/hubdb-copy created
|
||||
pod/hubdb-copy condition met
|
||||
total 412
|
||||
drwxrwxrwx 4 root root 4096 Oct 9 08:46 .
|
||||
drwxr-xr-x 1 root root 4096 Oct 9 08:46 ..
|
||||
drwxr-xr-x 2 root root 20480 Oct 9 08:46 assets
|
||||
-rw-r--r-- 1 root root 389120 Oct 9 08:46 hub.db
|
||||
drwx------ 2 root root 4096 Oct 9 08:46 snapshots
|
||||
total 389200
|
||||
drwxrwxrwx 4 root root 4096 Oct 9 08:46 .
|
||||
drwxr-xr-x 1 root root 4096 Oct 9 08:46 ..
|
||||
drwxr-xr-x 2 root root 20480 Oct 9 08:46 assets
|
||||
-rw------- 1 root root 398508032 Oct 9 08:46 hub.db
|
||||
drwx------ 2 root root 4096 Oct 9 08:46 snapshots
|
||||
95b67d69afdeddaf
|
||||
bench copy: 95b67d69afdeddaf
|
||||
pod "hubdb-copy" deleted
|
||||
deployment.apps/hub scaled
|
||||
## started on the restored DB (after ~20 s)
|
||||
hub-7f775576fd-cp9nj 1/1 Running 0 16s
|
||||
## start-up log (sealed / version / errors)
|
||||
2026/10/09 10:46:52 [INFO] off-site secrets sealed at rest (0 legacy plaintext row(s) sealed now)
|
||||
2026/10/09 10:46:52 [INFO] console passwords sealed at rest (0 legacy plaintext row(s) sealed now)
|
||||
2026/10/09 10:46:52 [INFO] box secrets sealed at rest (0 legacy plaintext value(s) sealed now)
|
||||
2026/10/09 10:46:52 [INFO] Default controller-version floor: 0.120.0
|
||||
2026/10/09 10:46:52 [INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via http://gitea.gitea-system.svc.cluster.local:3000
|
||||
2026/10/09 10:46:52 [INFO] Registry version checker started (every 6h)
|
||||
2026/10/09 10:46:52 [INFO] Listening on :8080
|
||||
2026/10/09 10:46:56 [WARN] Registry version check failed: HTTP request failed: Get "https://gitea.dooplex.hu/v2/admin/felhom-controller/tags/list": dial tcp: lookup gitea.dooplex.hu on 10.43.0.10:53: server misbehaving
|
||||
@@ -0,0 +1,26 @@
|
||||
## LIVE hub (GET only; /configs + /hosts) 2026-10-09T08:48:00Z
|
||||
customers: 4 Tester-2 demo-felhom demo-hp tester-1
|
||||
hosts: 4 Tester-2-be8404 demo-felhom-8363b5 demo-hp-bb76ea tester-1-d70be4
|
||||
## THROWAWAY hub on the bench k3s 2026-10-09T08:48:12Z (port-forward inside pd-k3s; same queries + a reveal per host)
|
||||
customers: 4 Tester-2 demo-felhom demo-hp tester-1
|
||||
hosts: 4 Tester-2-be8404 demo-felhom-8363b5 demo-hp-bb76ea tester-1-d70be4
|
||||
reveal Tester-2-be8404: http 200, password 32 chars, sealed-form=False (value not printed)
|
||||
reveal demo-felhom-8363b5: http 200, password 32 chars, sealed-form=False (value not printed)
|
||||
reveal demo-hp-bb76ea: http 200, password 32 chars, sealed-form=False (value not printed)
|
||||
reveal tester-1-d70be4: http 200, password 32 chars, sealed-form=False (value not printed)
|
||||
|
||||
## the throwaway hub log for the reveals
|
||||
2026/10/09 10:46:56 [WARN] Registry version check failed: HTTP request failed: Get "https://gitea.dooplex.hu/v2/admin/felhom-controller/tags/list": dial tcp: lookup gitea.dooplex.hu on 10.43.0.10:53:
|
||||
2026/10/09 10:47:54 [ERROR] kernel notice (7.0.14-22-pve) to customer demo-felhom failed: sending request: Post "https://api.resend.com/emails": dial tcp: lookup api.resend.com on 10.43.0.10:53: serve
|
||||
2026/10/09 10:47:58 [ERROR] kernel notice (7.0.14-22-pve) to customer demo-hp failed: sending request: Post "https://api.resend.com/emails": dial tcp: lookup api.resend.com on 10.43.0.10:53: server mi
|
||||
2026/10/09 10:48:17 [INFO] operator revealed break-glass console credential for host Tester-2-be8404 (user=root@pam, secret 32 chars)
|
||||
2026/10/09 10:48:17 [INFO] operator revealed break-glass console credential for host demo-felhom-8363b5 (user=root@pam, secret 32 chars)
|
||||
2026/10/09 10:48:18 [INFO] operator revealed break-glass console credential for host demo-hp-bb76ea (user=root@pam, secret 32 chars)
|
||||
2026/10/09 10:48:18 [INFO] operator revealed break-glass console credential for host tester-1-d70be4 (user=root@pam, secret 32 chars)
|
||||
## hubdb-check on DooPlex (second channel), on a COPY of the restored DB 2026-10-09T08:48:30Z
|
||||
hosts=4 console_passwords_opened=4 failed=0 absent=0
|
||||
rc=0
|
||||
control (a random key):
|
||||
hosts=4 console_passwords_opened=0 failed=4 absent=0
|
||||
hubdb-check: FAILED: not every console password opened with this key
|
||||
rc=1
|
||||
@@ -0,0 +1,175 @@
|
||||
## 2026-10-09T08:48:46Z teardown — bench 9401
|
||||
pd-k3s volumes: c57fb08341e5b6fa44a389ddec08f866774ad7acde602870e2e574d044ab33d4 403bfbca4a80250b0e49a8131844b17e59552746513dac0d01f409f942a6e95d d8c1d37af50fe0473724a093c518ad86ec878c2ed96018c8a2ba908346f97daf 585175f25b48d2f86c1a782ee532cac86f099c94c83fe7e84cfd9da986251698
|
||||
pd-k3s
|
||||
c57fb08341e5b6fa44a389ddec08f866774ad7acde602870e2e574d044ab33d4
|
||||
403bfbca4a80250b0e49a8131844b17e59552746513dac0d01f409f942a6e95d
|
||||
d8c1d37af50fe0473724a093c518ad86ec878c2ed96018c8a2ba908346f97daf
|
||||
585175f25b48d2f86c1a782ee532cac86f099c94c83fe7e84cfd9da986251698
|
||||
pd-net
|
||||
image-removed
|
||||
containers=0 volumes=8 networks=bridge host none traefik-public
|
||||
.bashrc
|
||||
.profile
|
||||
.ssh
|
||||
/dev/loop2 59G 3.8G 53G 7% /
|
||||
are supported and installed on your system.
|
||||
are supported and installed on your system.
|
||||
status: stopped
|
||||
## teardown — DooPlex
|
||||
.cache
|
||||
.kube
|
||||
stage
|
||||
-home-kisfenyo-git-homelab-manifests
|
||||
-home-kisfenyo-git-jarr
|
||||
-mnt-5-hdd-felhom-eu-build-wt-kept
|
||||
-mnt-5-hdd-felhom-eu-drill-app-catalog-drill
|
||||
-mnt-5-hdd-felhom-eu-git
|
||||
-mnt-5-hdd-felhom-eu-git-app-catalog-felhom-eu
|
||||
-mnt-5-hdd-felhom-eu-git-felhom-agent
|
||||
-mnt-5-hdd-felhom-eu-git-felhom-controller
|
||||
-mnt-5-hdd-felhom-eu-git-felhom-eu
|
||||
-mnt-5-hdd-felhom-eu-git-wt-catalog
|
||||
-mnt-5-hdd-felhom-eu-worktrees-ctrl-c-msgref
|
||||
-mnt-5-hdd-felhom-eu-worktrees-ctrl-d4-sessions
|
||||
-mnt-5-hdd-felhom-eu-worktrees-ctrl-integ
|
||||
-mnt-5-hdd-felhom-eu-worktrees-hub-cf-mail-drop
|
||||
ag.txt
|
||||
au_tests.py
|
||||
backups_degraded.old
|
||||
backups_empty.old
|
||||
backups_full.old
|
||||
backups_interrupted_run.old
|
||||
backups_nobackup_yet.old
|
||||
backups_tier_due.old
|
||||
bash-edit-diff
|
||||
bb
|
||||
bh.bak
|
||||
bh.new
|
||||
bookstackfix.py
|
||||
cache-break-state-35d4e820-990d-465a-b5ee-c7b72649d29a.json
|
||||
cache-break-state-55a70710-3ae0-4d71-b946-d9db725be672.json
|
||||
cache-break-state-9a980473-7466-41a7-9c4c-d5f6c1785531.json
|
||||
cache-break-state-c6c2aaef-766c-4184-8225-3610d20724ca.json
|
||||
cache-break-state-e6fbe918-56ae-4fc3-af26-4a8a10574e30.json
|
||||
cache-break-state-f6c29d80-39ec-4c54-960a-646ec49d5e71.json
|
||||
cat_gates.txt
|
||||
cfg.bak
|
||||
cg.txt
|
||||
cg2.txt
|
||||
cg3.txt
|
||||
cgi.txt
|
||||
cgi2.txt
|
||||
ci-1594.log
|
||||
ci-ctl.log
|
||||
ci.py
|
||||
ci1357.log
|
||||
ci1358.log
|
||||
ci1361.log
|
||||
ciwait.sh
|
||||
claperfix.py
|
||||
cmd_controller_main.go.bak
|
||||
configs.go.bak
|
||||
crg.bak
|
||||
ctl-build.txt
|
||||
ctlbuild.txt
|
||||
ctrlgates.txt
|
||||
deb.I8Wk
|
||||
deb2.y8f3
|
||||
deb3.XEFL
|
||||
docs45.py
|
||||
dr.bak
|
||||
drafts
|
||||
dump_test.go
|
||||
dumpgo
|
||||
en.old
|
||||
felhom-opsign
|
||||
g.txt
|
||||
g1.txt
|
||||
g2.txt
|
||||
g3.txt
|
||||
gates-b.txt
|
||||
gates.txt
|
||||
gg.bak
|
||||
gr.txt
|
||||
h.bak
|
||||
hb.new
|
||||
hc.bak
|
||||
hr.bak
|
||||
hu.old
|
||||
hubbak
|
||||
hubbuild.txt
|
||||
hubgate.txt
|
||||
hubrepogates.txt
|
||||
internal_backup_backup.go.bak
|
||||
internal_i18n_locales_en.json.bak
|
||||
internal_i18n_locales_hu.json.bak
|
||||
isogate.5cI4
|
||||
m.go
|
||||
main.bak
|
||||
main.go.bak
|
||||
mk.cur
|
||||
mk.new
|
||||
mk.old
|
||||
mnt-hdd_1.mount
|
||||
newrules.txt
|
||||
o.bak
|
||||
off.go
|
||||
ok.bak
|
||||
op.bak
|
||||
opa.bak
|
||||
osa.bak
|
||||
pa.bak
|
||||
partC-ids.json
|
||||
partC.md
|
||||
partd.py
|
||||
partd2.py
|
||||
partd3.py
|
||||
promtest
|
||||
r.bak
|
||||
r262.bak
|
||||
r705a.py
|
||||
r706.py
|
||||
rec.bak
|
||||
reg3.py
|
||||
rel147.txt
|
||||
rg.bak
|
||||
rg.txt
|
||||
rgi.txt
|
||||
rgi2.txt
|
||||
rp
|
||||
rp.bak
|
||||
rp2
|
||||
rp2.bak
|
||||
rr.txt
|
||||
server.go.bak
|
||||
service.go.bak
|
||||
sg.bak
|
||||
snap.bak
|
||||
srv.bak
|
||||
ss.bak
|
||||
st.bak
|
||||
states.py
|
||||
stg.bak
|
||||
sudoers.v0145
|
||||
sudotest
|
||||
t.bak
|
||||
t.go
|
||||
t.txt
|
||||
test_bb.py
|
||||
uf.bak
|
||||
unchecked-table.md
|
||||
v279docs.py
|
||||
vik.bak
|
||||
w.bak
|
||||
x.go
|
||||
## 2026-10-09T08:49:18Z leftover volumes from the two failed pd-k3s starts (08:36, 08:42)
|
||||
9bf479a381ed84a255645772d9424cc60a73d024bfc537485536b4e897831ac1 2026-10-09T08:36:57Z map[com.docker.volume.anonymous:]
|
||||
88e8070b44ca866db658800c9c013b7d6095880b86033d2200d99c91ea97ce0e 2026-10-09T08:42:45Z map[com.docker.volume.anonymous:]
|
||||
7119dcb4fa0e1ada468bd477c3bf7f00bda498d3cf8d984f8ff903d674bf5fb5 2026-10-09T08:42:45Z map[com.docker.volume.anonymous:]
|
||||
54334eaba740d0cd68c33610c767f361b5882b2a1711ab6abbd6de22a246f416 2026-10-09T08:42:45Z map[com.docker.volume.anonymous:]
|
||||
71831f601ff91a65952659dc60d05c5b7c6712158505d20c9665b3e0561bb5df 2026-10-09T08:36:58Z map[com.docker.volume.anonymous:]
|
||||
a7d0645cf17a95b33018daeb558756bb01f54ab3a59d8553e09b4b8933649d78 2026-10-09T08:36:58Z map[com.docker.volume.anonymous:]
|
||||
b922c6e0612a135afcdc7d9eb49a964682e4940b45b4fefe311b2258967e275d 2026-10-09T08:36:58Z map[com.docker.volume.anonymous:]
|
||||
cfabb1005614015b82c12090ad7520c6ba877a4dcfafba317803dae13c0079ef 2026-10-09T08:42:45Z map[com.docker.volume.anonymous:]
|
||||
volumes=0 containers=0
|
||||
status: stopped
|
||||
Reference in New Issue
Block a user