Peti's box retired (operator ruling 2026-09-25): hub customer + Storage Box sub-account removed, ep0 held nothing; protected list = DooPlex + ep0; decision 34 (no leg resume, R-686 closed); R-688
gates / gates (push) Successful in 25s

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-09-25 11:04:52 +02:00
parent 6b2176e480
commit eb1c56a981
28 changed files with 398 additions and 28 deletions
@@ -0,0 +1,81 @@
# RETIRE — Peti's box (`peti-felhom`), 2026-09-25
Operator ruling 2026-09-25: Peti's box is retired (the tester wiped his server; it will not return); its off-site
backup holds no user data and is to be deleted; it leaves the protected list. Architecture read first:
`runbooks/target-selection.md`, `05-hub-architecture.md` §14 (what a customer delete deprovisions),
`06-offsite-connectivity.md`, `04-control-plane-authorization.md`. Evidence: `audits/retire-peti-2026-09-25/`.
## Not done, or changed
- **Nothing of Peti's was on ep0.** No PBS namespace (`ns`: demo-felhom, demo-hp, tester-1), no WireGuard peer in
the live `wg show`, no config naming him. His only off-site item was a Storage Box **sub-account**
(`u629488-sub2`, id 269130, label `felhom-customer=peti-felhom`, home `felhom-peti-felhom`) on the pool box.
- **The "backup" was never a backup.** The sub-account's folder held ONE file: `.ssh/authorized_keys`, 81 bytes. No
restic repository was ever created — all 482 of his reports (2026-07-10 … 07-15) show 0 off-site snapshots and
0 bytes; his escrow stayed pending. The operator's "no user data" is confirmed by the listing, not only by word.
- **To list and empty the folder I reset the sub-account's password through the Hetzner API** (the product's own
`reset_subaccount_password` action, scoped to id 269130 after re-reading its label). The key file was removed from
inside the folder, then the hub deleted the sub-account. Whether Hetzner deletes a sub-account's folder with it
is NOT measured here — so the folder was emptied first, and nothing of his could remain either way.
- **Cloudflare:** his customer config carried a Cloudflare tunnel token and an API token (for `sajatfelhom.hu`).
They were purged with the customer record. The hub's delete dialog says "tunnel/zone removed", but no leg of the
cascade calls Cloudflare — the tunnel and DNS records on Cloudflare's side, if any remain, were NOT inventoried or
removed (a row: R-688).
- **A second, older Storage Box** (`PBS-storage-1`, u629193, box 611421) once held a folder `felhom-peti-spike/`
(spike leftovers, per `RUNBOOK-ep0-datastore-volume-2026-07-27.md`). Its mount on ep0 is gone, and the box is
not visible to either API token (404). The register row "delete the box" is the operator's. Not touched.
- **Code comments that name Peti were left** (hub `rollup.go`, `appliances.go`, `notify/templates.go`; agent pbsdr
tests). They explain why code is shaped as it is; no behaviour depends on Peti.
## A1 — inventory (before anything was removed)
| where | item | how found |
|---|---|---|
| hub | customer `peti-felhom` („Peti Proxmox", domain `sajatfelhom.hu`, dr_tier 0) with 482 reports, 123 events, 1,036 telemetry rows, DR recipe, one-time secret, claim, 1 log tail | copy of the hub DB (with its -wal), every table's `customer_id` / `host_id` |
| hub | host `peti-felhom-86d37d` — already DELETED 2026-07-15 08:56:22 (`host_deletions`, escrow_acked 0); no host, guest, escrow, recovery or PBS-secret rows | same |
| hub | WireGuard peer — none in `wg_peers` | same |
| Storage Box | sub-account id 269130 `u629488-sub2`, home `felhom-peti-felhom`, label `felhom-customer=peti-felhom`, created 2026-07-10 | Hetzner API with the hub's own token, matched on the LABEL |
| ep0 | nothing (namespaces, peers, `/etc`, `/root`, `/srv` grep — one false hit, "competing" in `lvm.conf`) | ssh read-only |
| documents | 48 non-history files + workspace rules + 21 memory notes (whole-word pattern; positive control target-selection.md 7 hits, negative control catalog templates 0) | `A1-documents.txt` |
**Control, before:** pool box `size_data 3,120,562,176`, sub-accounts sub1 (demo-felhom), sub3 (demo-hp), sub4
(tester-1); ep0 namespaces demo-felhom 2 snapshots / 220K, demo-hp 2 / 540K, tester-1 2 / 368K; demo boxes'
off-site `last_status ok` (11 and 90 snapshots, runs 02:15:46Z / 02:18:41Z).
## A3 — removal
1. `POST …/subaccounts/269130/actions/reset_subaccount_password` → success (password kept 0600 in the scratchpad,
deleted afterwards).
2. As `u629488-sub2`: `rm .ssh/authorized_keys`, `rmdir .ssh` → `ls -la` empty, `du -s .` 1.
3. Hub preview `GET /configs/peti-felhom/delete` — 0 hosts, off-site `u629488-sub2`, residue 1,519. Then
`POST /configs/peti-felhom/delete` (ack_hosts, ack_reset, ack_purge, confirm_id, expect_hosts=0) → 303. The hub's
log: sub-account 269130 deprovisioned; PBS tenancy `existed=false`; claim reset; residue purged (reports 482,
telemetry 1,036, log tails 1); cascade COMPLETE (journal #20).
## A4 — nothing else moved
| item | before | after |
|---|---|---|
| sub-accounts | sub1, sub2 (Peti), sub3, sub4 | sub1, sub3, sub4 — labels and homes unchanged |
| pool box `size_data` | 3,120,562,176 | 3,120,562,176 |
| login as `u629488-sub2` | worked | "Permission denied" |
| ep0 namespaces (snapshots / du) | demo-felhom 2/220K, demo-hp 2/540K, tester-1 2/368K | identical |
| ep0 WireGuard peers | .2 .3 .4 .250 | identical |
| hub hosts | demo-felhom, demo-hp, drill-r50 | identical |
| hub customers | demo-felhom, demo-hp, drill-r50, peti-felhom, tester-1 | without peti-felhom |
| hub rows naming peti | — | events 124, notification_log 80, host_deletions 1, customer_resets 1 (audit, by design), app_log_issues 8 (R-244) |
## A5 — documents changed
`runbooks/target-selection.md` (Tier 2 = DooPlex + ep0, ep0's reason rewritten, Peti's section retired); the
unprompted-work rule (4 identical copies); `03`, `06`, `00-capability-map`; runbooks `TASK-identity-only-escrow`
(obsolete note), `RUNBOOK-vzdump-target-move` (row 9), `RUNBOOK-island-migration`; retired banners on
`pilot/PETI-tester-agreement.md`, `RUNBOOK-peti-return`, `RUNBOOK-peti-pbsdr`; register (PETI closed as retired,
R-530, R-244, R-600 annotated); CONTEXT; STATUS; memory notes. Historic audits, tests and CHANGELOGs keep their text.
## Claims in the brief
1. *Peti's backup is on ep0* — **WRONG.** ep0 held nothing of his; the item was a Storage Box sub-account.
2. *The hub's host delete removes the WireGuard peer* — **not testable here:** the host was deleted in July and
ep0 carries no peer for it now; whether that delete removed one is not recorded (R-600 annotated).
3. *The backup holds no user data* — **TRUE, and stronger:** it held no backup at all (one 81-byte key file).
@@ -0,0 +1,4 @@
# A1 control BEFORE — demo boxes' off-site status from their latest hub report
demo-felhom 2026-09-25 08:44:15 {'snapshot_count': 11, 'repo_size_bytes': 147483, 'last_run': '2026-09-25T02:15:46Z', 'last_status': 'ok', 'state': None, 'escrow_state': 'escrowed'}
demo-hp 2026-09-25 08:44:17 {'snapshot_count': 90, 'repo_size_bytes': 215007449, 'last_run': '2026-09-25T02:18:41Z', 'last_status': 'ok', 'state': None, 'escrow_state': 'escrowed'}
tester-1 2026-09-17 00:28:20 {'snapshot_count': 0, 'repo_size_bytes': 0, 'last_run': '2026-09-16T21:09:00Z', 'last_status': 'error', 'state': None, 'escrow_state': 'escrowed'}
@@ -0,0 +1,75 @@
# A1 documents — whole-word pattern: \bpeti\b|peti-felhom|\bPETI\b|Peti's|Petinek|Petié|petis — 2026-09-25T08:54:25+00:00
positive control (pattern on a known line: target-selection.md): 7
negative control (same pattern on app-catalog templates): 0
== non-history files (excluding audits/, tests/ evidence, CHANGELOGs, REPORT-*.md, pilot/ runbooks of the past):
felhom.eu/STATUS.md
felhom.eu/CONTEXT.md
felhom.eu/documentation/architecture/06-offsite-connectivity.md
felhom.eu/documentation/architecture/07-backup-architecture.md
felhom.eu/documentation/architecture/03-host-agent.md
felhom.eu/documentation/architecture/_recovery-inventory-2026-07-28.md
felhom.eu/documentation/architecture/10-localisation.md
felhom.eu/documentation/runbooks/RUNBOOK-byo-deployment.md
felhom.eu/documentation/runbooks/RUNBOOK-island-migration.md
felhom.eu/documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md
felhom.eu/documentation/runbooks/publish-train-rules.md
felhom.eu/documentation/runbooks/RUNBOOK-onboarding-draft-v4.md
felhom.eu/documentation/runbooks/RUNBOOK-manual-build.md
felhom.eu/documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md
felhom.eu/documentation/architecture/00-capability-map.md
felhom.eu/documentation/runbooks/TASK-identity-only-escrow.md
felhom.eu/documentation/runbooks/target-selection.md
felhom.eu/documentation/pilot/RUNBOOK-peti-pbsdr-2026-07-11.md
felhom.eu/documentation/pilot/RUNBOOK-publish-0.85-0.120-2026-07-12.md
felhom.eu/documentation/pilot/RUNBOOK-publish-0.79-0.110-2026-07-10.md
felhom.eu/documentation/pilot/PETI-tester-agreement.md
felhom.eu/documentation/pilot/RUNBOOK-peti-return-2026-07-13.md
felhom.eu/documentation/pilot/RUNBOOK-publish-0.90-0.143-2026-07-18.md
felhom.eu/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md
felhom.eu/documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md
felhom.eu/documentation/pilot/GO-LIVE-PACKAGE.md
felhom.eu/documentation/pilot/DRILL-GL6-2026-07-08.md
felhom.eu/.claude/rules/unprompted-work.md
felhom.eu/documentation/backlog/ROADMAP.md
felhom.eu/documentation/backlog/CLOSED-ITEMS.md
felhom.eu/hub/internal/notify/templates.go
felhom.eu/documentation/backlog/OPEN-ITEMS.md
felhom.eu/hub/internal/tenantsync/client_test.go
felhom.eu/hub/internal/monitor/deadline_unbound_test.go
felhom.eu/hub/internal/web/rollup.go
felhom.eu/hub/internal/web/rollup_test.go
felhom.eu/hub/internal/web/pbsdr_generation_test.go
felhom.eu/hub/internal/web/pbsdr_poke_test.go
felhom.eu/hub/internal/web/pbsdr_test.go
felhom.eu/hub/internal/web/render_test.go
felhom.eu/hub/internal/web/appliances.go
felhom-controller/.claude/rules/unprompted-work.md
felhom-controller/CONTEXT.md
felhom-agent/CONTEXT.md
felhom-agent/internal/pbsdr/seed_reassert_test.go
felhom-agent/internal/pbsdr/manager_test.go
felhom-agent/internal/pbsdr/rearm_test.go
app-catalog-felhom.eu/.claude/rules/unprompted-work.md
== workspace level:
.claude/rules/unprompted-work.md
.claude-memory/campaign3-nightrun-2026-07-11.md
.claude-memory/agent-update-needs-signed-job.md
.claude-memory/current-state-2026-07-11.md
.claude-memory/current-state-2026-07-21.md
.claude-memory/hub-closing-bundle-2026-07-13.md
.claude-memory/pbs-tier-provisioning-spike.md
.claude-memory/drtier-by-default-2026-07-12.md
.claude-memory/peti-return-p1-stop-2026-07-13.md
.claude-memory/spike-lan-discovery-r6-2026-07-18.md
.claude-memory/restore-test-fills-thin-pool.md
.claude-memory/gl2-byo-install-profile-shipped.md
.claude-memory/customer-claim-arc-2026-07-12.md
.claude-memory/polish-batch-2026-07-13.md
.claude-memory/observability-pass-shipped.md
.claude-memory/r50-island-bridge-go-2026-07-25.md
.claude-memory/fleet-identity-peti-vs-tester1.md
.claude-memory/power-outage-audit-2026-07-22.md
.claude-memory/hub-offsite-provisioning-e2e.md
.claude-memory/nas-network-storage.md
.claude-memory/remote-applog-diagnostics-shipped.md
.claude-memory/MEMORY.md
@@ -0,0 +1,35 @@
# A1 ep0 read-only — 2026-09-25T08:57:45+00:00
felhom-hetzner
== PBS namespaces (live store /mnt/pbs-datastore):
demo-felhom
demo-hp
tester-1
demo-felhom: 220K snapshots=2
demo-hp: 540K snapshots=2
tester-1: 368K snapshots=2
== grep peti anywhere in PBS config:
== wg show (peers):
peer: snkgWxlcN7zXxjy1rG/jefe6uysGq/27E736CVPkbBc=
allowed ips: 10.77.0.3/32
latest handshake: 57 seconds ago
peer: KNaFXHiY9l08UhPO4gxUBFQz2WF3RYWyUGL8k+hzDVo=
allowed ips: 10.77.0.2/32
latest handshake: 1 minute, 39 seconds ago
peer: yNVpWt9Ekid9p459/VLHbmCDHIi8xTUYHbgazUp5S00=
allowed ips: 10.77.0.250/32
latest handshake: 18 hours, 59 minutes, 18 seconds ago
peer: kdhOHMyADRpaTaFmh8ypwsWngvBpIyMyon80DJK+00M=
allowed ips: 10.77.0.4/32
latest handshake: 43 days, 2 hours, 41 minutes, 56 seconds ago
== wg config peers:
8:PublicKey = KNaFXHiY9l08UhPO4gxUBFQz2WF3RYWyUGL8k+hzDVo=
9:AllowedIPs = 10.77.0.2/32
12:PublicKey = yNVpWt9Ekid9p459/VLHbmCDHIi8xTUYHbgazUp5S00=
13:AllowedIPs = 10.77.0.250/32
16:PublicKey = snkgWxlcN7zXxjy1rG/jefe6uysGq/27E736CVPkbBc=
17:AllowedIPs = 10.77.0.3/32
20:PublicKey = kdhOHMyADRpaTaFmh8ypwsWngvBpIyMyon80DJK+00M=
21:AllowedIPs = 10.77.0.4/32
== grep peti in /etc and /root, /srv:
/etc/lvm/lvm.conf
1053:# When there are competing read-only and rea
@@ -0,0 +1,7 @@
# A1 Hetzner (the hub's own token, read-only) — 2026-09-25T08:57:06.131163+00:00
pool box id=611714 name=storage-box-pool-1 user=u629488 stats: size=3233808384 size_data=3120562176 size_snapshots=113246208
sub id=273581 user=u629488-sub1 home=felhom-demo-felhom labels={'felhom-customer': 'demo-felhom'} created=2026-07-18T16:53:43Z desc=felhom offsite demo-felhom
sub id=269130 user=u629488-sub2 home=felhom-peti-felhom labels={'felhom-customer': 'peti-felhom'} created=2026-07-10T07:29:44Z desc=felhom offsite peti-felhom
sub id=275124 user=u629488-sub3 home=felhom-demo-hp labels={'felhom-customer': 'demo-hp'} created=2026-07-21T16:01:40Z desc=felhom offsite demo-hp
sub id=311327 user=u629488-sub4 home=felhom-tester-1 labels={'felhom-customer': 'tester-1'} created=2026-09-16T15:03:56Z desc=felhom offsite tester-1
all boxes visible to this token: [(611714, 'storage-box-pool-1')]
@@ -0,0 +1,10 @@
# A1b ep0 second storage box mount — 2026-09-25T09:02:23+00:00
inactive
ls: cannot access '/mnt/pbs-storagebox': No such file or directory
== felhom-peti-spike: du: cannot access '/mnt/pbs-storagebox/felhom-peti-spike': No such file or directory
ls: cannot access '/mnt/pbs-storagebox/felhom-peti-spike': No such file or directory
== felhom-demo: du: cannot access '/mnt/pbs-storagebox/felhom-demo': No such file or directory
ls: cannot access '/mnt/pbs-storagebox/felhom-demo': No such file or directory
== spike-sub: du: cannot access '/mnt/pbs-storagebox/spike-sub': No such file or directory
ls: cannot access '/mnt/pbs-storagebox/spike-sub': No such file or directory
== is it a PBS datastore anywhere?
@@ -0,0 +1,34 @@
# A2 listing of Peti's sub-account home (u629488-sub2), read-only — 2026-09-25T08:59:31+00:00
$ pwd
Warning: Permanently added '[u629488-sub2.your-storagebox.de]:23' (ED25519) to the list of known hosts.
/home
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
u629488-sub2 1073631360 3047552 1070583808 1% /home
$ du -s .
2 .
$ ls -la
total 3
drwxr-xr-x 3 u629488-sub2 1006 3 Jul 10 09:49 .
dr-x--x--x 7 root root 11 Jul 10 07:30 ..
drwx------ 2 u629488-sub2 1006 3 Jul 10 09:49 .ssh
$ ls -la felhom-repo
/usr/bin/ls: cannot access 'felhom-repo': No such file or directory
$ du -s felhom-repo
/usr/bin/du: cannot access 'felhom-repo': No such file or directory
$ tree -a -L 2 felhom-repo
felhom-repo [error opening dir]
0 directories, 0 files
$ ls -la felhom-repo/snapshots
/usr/bin/ls: cannot access 'felhom-repo/snapshots': No such file or directory
$ ls -la felhom-repo/keys
/usr/bin/ls: cannot access 'felhom-repo/keys': No such file or directory
$ ls -la .ssh
total 2
drwx------ 2 u629488-sub2 1006 3 Jul 10 09:49 .
drwxr-xr-x 3 u629488-sub2 1006 3 Jul 10 09:49 ..
-rw------- 1 u629488-sub2 1006 81 Jul 10 09:49 authorized_keys
$ stat -c %n .ssh/authorized_keys .ssh/nonexistent-control-zq
.ssh/authorized_keys
/usr/bin/stat: cannot statx '.ssh/nonexistent-control-zq': No such file or directory
@@ -0,0 +1,10 @@
# A2 Peti's LAST report to the hub: id 12564 received 2026-07-15 08:39:00 UTC; controller 0.115.0
deployed apps: ['rallly']
offsite status: {"enabled": true, "escrow_state": "pending", "snapshot_count": 0, "repo_size_bytes": 0, "quota_gb": 50}
backup section keys: ['enabled', 'last_db_dump', 'snapshot_count', 'repo_size_mb', 'integrity_ok']
storage: [('/', 'SSD')]
newest report WITH an offsite object: 12564 2026-07-15 08:39:00 {"enabled": true, "escrow_state": "pending", "snapshot_count": 0, "repo_size_bytes": 0, "quota_gb": 50}
== all 482 Peti reports (2026-07-10 09:49:09 .. 2026-07-15 08:39:00 UTC): 482 carry an offsite object; MAX offsite snapshot_count=0, MAX repo_size_bytes=0
apps ever deployed (report count): {'rallly': 481, 'calibre-web': 73}
offsite-related events for Peti: 1
@@ -0,0 +1,3 @@
# A3-1 2026-09-25T08:59:13.275582+00:00 TARGET CHECK: u629488-sub2 felhom-peti-felhom {'felhom-customer': 'peti-felhom'} access: {'samba_enabled': False, 'ssh_enabled': True, 'webdav_enabled': False, 'reachable_externally': True, 'readonly': False}
action 657507966278067 reset_subaccount_password running
action status: success (password stored out-of-band in the session scratchpad, 0600)
@@ -0,0 +1,9 @@
# A3-2 empty Peti's home (target: u629488-sub2 = sub 269130, label peti-felhom; its home is /home) — 2026-09-25T08:59:59+00:00
$ rm .ssh/authorized_keys
$ rmdir .ssh
$ ls -la
total 2
drwxr-xr-x 2 u629488-sub2 1006 2 Sep 25 09:00 .
dr-x--x--x 7 root root 11 Jul 10 07:30 ..
$ du -s .
1 .
@@ -0,0 +1,28 @@
# A3-3 hub delete PREVIEW for peti-felhom — 2026-09-25T09:00:09+00:00
{
"claim_present": true,
"customer_id": "peti-felhom",
"customer_name": "Peti Proxmox",
"dr_recipe_present": true,
"has_config": true,
"host_count": 0,
"hosts": [],
"offsite_enabled": true,
"offsite_identifier": "u629488-sub2",
"offsite_type": "shared",
"one_time_secret": true,
"online_host_present": false,
"pbs_tenancy_configured": true,
"pending_journal": null,
"residue": {
"app_log_tails": 1,
"app_telemetry": 1036,
"appliance_registrations": 0,
"log_tail_requests": 0,
"notification_prefs": 0,
"reports": 482,
"selfbind_tokens": 0
},
"residue_total": 1519,
"superseded_blobs": 0
}
@@ -0,0 +1,12 @@
# A3-4 hub customer DELETE cascade for peti-felhom — 2026-09-25T09:00:21Z
POST /configs/peti-felhom/delete -> 303 Location: /configs?flash=deleted
== the hub's own log:
2026/09/25 11:00:21 [INFO] customer DELETE cascade started for peti-felhom (journal #20, 0 host(s))
2026/09/25 11:00:29 [offsite] deprovisioned shared sub-account 269130 for peti-felhom (repo data destroyed)
2026/09/25 11:00:29 [INFO] reset peti-felhom: offsite deprovisioned (repo data destroyed)
2026/09/25 11:00:29 [INFO] tenantsync: deprovision ok for peti-felhom (ns=peti-felhom, existed=false)
2026/09/25 11:00:29 [INFO] reset peti-felhom: PBS tenancy deprovisioned
2026/09/25 11:00:29 [INFO] [claim] reset to unclaimed for peti-felhom (customer RESET) — next onboarding mints a fresh code
2026/09/25 11:00:33 [INFO] delete peti-felhom: residue purged (reports=482 app_telemetry=1036 app_log_tails=1 log_tail_requests=0 notif_prefs=0 selfbind_tokens=0 appliance_registrations=0)
2026/09/25 11:00:33 [INFO] customer DELETE cascade COMPLETE for peti-felhom (journal #20) — full teardown
@@ -0,0 +1,28 @@
# A4 AFTER — 2026-09-25T09:00:49.893123+00:00
pool box 611714 stats: size=3233808384 size_data=3120562176 size_snapshots=113246208
sub id=273581 user=u629488-sub1 home=felhom-demo-felhom labels={'felhom-customer': 'demo-felhom'}
sub id=275124 user=u629488-sub3 home=felhom-demo-hp labels={'felhom-customer': 'demo-hp'}
sub id=311327 user=u629488-sub4 home=felhom-tester-1 labels={'felhom-customer': 'tester-1'}
== login as the deleted sub-account must now FAIL:
Permission denied, please try again.
login_rc=0
== ep0 after:
demo-felhom
demo-hp
tester-1
demo-felhom snapshots=2 du=220K
demo-hp snapshots=2 du=540K
tester-1 snapshots=2 du=368K
KNaFXHiY9l08UhPO4gxUBFQz2WF3RYWyUGL8k+hzDVo= 10.77.0.2/32
yNVpWt9Ekid9p459/VLHbmCDHIi8xTUYHbgazUp5S00= 10.77.0.250/32
kdhOHMyADRpaTaFmh8ypwsWngvBpIyMyon80DJK+00M= 10.77.0.4/32
snkgWxlcN7zXxjy1rG/jefe6uysGq/27E736CVPkbBc= 10.77.0.3/32
== hub census AFTER: every table, every text column, rows matching 'peti'
notification_log: 80
events: 124
app_log_issues: 8
host_deletions: 1
customer_resets: 1
control — demo rows still present: {'reports': 15337, 'customer_configs': 2, 'events': 2262}
hosts: ['demo-felhom-8363b5', 'demo-hp-bb76ea', 'drill-r50-0a4f9a']
customer_configs: ['demo-felhom', 'demo-hp', 'drill-r50', 'tester-1']