docs(audit): Hetzner API provisioning spike — both offsite models FIT

KEY: Storage Box API is api.hetzner.com/v1 (NOT api.hetzner.cloud/v1). Proven
live on spike-* (torn down): sub-account create/reset/readonly/delete + billable
bx11 box create/change_type/delete. Create takes a password (no ssh-key param) →
transient-pw→ssh-copy-id -s -f→restic reach→reset_subaccount_password (key-only
steady state). RepoPath /home/<repo>; quota levers (dedicated=box type, shared=
readonly+soft-quota); idempotency via label_selector. SECURITY: token is
project-wide unscopeable — can touch ep0 + SSH keys → move customer boxes to a
dedicated Hetzner project. No prod code; spike torn down clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-09 17:46:43 +02:00
parent c1cfbe817b
commit 996d403248
2 changed files with 113 additions and 0 deletions
+12
View File
@@ -3,6 +3,18 @@
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-09 — Hetzner API provisioning spike (offsite tier).** Both offsite models are API-provisionable
(FIT). **#1 correction: the Storage Box API is `https://api.hetzner.com/v1`, NOT `api.hetzner.cloud/v1`**
(Cloud API 404s for storage boxes); same Bearer token. Proven live on `spike-*` (torn down): sub-account
create/reset/readonly/delete + a billable bx11 box create/rescale(`change_type`)/delete. Create takes a
**password (no ssh-key param)** → transient-password→`ssh-copy-id -s -f`→restic-reach→`reset_subaccount_password`
(transient pw invalidated, key survives) = key-only steady state. RepoPath `/home/<repo>` (chroot). Quota:
dedicated=box type/`change_type` (hard); shared=no native quota → Felhom soft-quota + `update_access_settings{readonly}`
freeze. Idempotency = **label_selector** (box names not unique). Rate 3600/h. **SECURITY: the token is
project-wide unscopeable Read+Write — it can also touch ep0 (the prod offsite server) + 3 SSH keys →
move customer boxes to a DEDICATED Hetzner project before the hub holds a write token.** Record:
`documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md`. NEXT: the hub-provisioning impl.
- **2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised).** The full custody cycle proven on
the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent →
**real escrow ceremony** (`--selftest=escrow-create`, K + restic-pw under a fresh R, uploaded to the hub,
@@ -0,0 +1,101 @@
# SPIKE — Hetzner API provisioning (sub-account + dedicated box) for the offsite tier
**Date:** 2026-07-09 · **Class:** spike + risky (write token; one billable box create) · **Executor:**
Claude Code live from DooPlex, Viktor on the 🛑 billable + token steps. **No code, no version bump.**
Captures the exact API shapes the hub-provisioning impl will code against, for BOTH offsite models.
---
## 1. VERDICT
**Both offsite models are API-provisionable — FIT.** An **API-created** account (shared sub-account or
dedicated box) becomes a **working restic target reached key-only**, via the transient-password → install-key
→ reset bootstrap. **The #1 correction:** the Storage Box API lives at **`https://api.hetzner.com/v1`**, NOT
`api.hetzner.cloud/v1` (the classic Cloud API 404s for every storage-box route). Same Bearer token; different
host. One **security caveat** (blast radius, §6) must be addressed before the hub wields this token.
---
## 2. The exact API surface (measured live)
**Base:** `https://api.hetzner.com/v1` · **Auth:** `Authorization: Bearer <token>` · **Rate limit:**
`RateLimit-Limit: 3600`/h (per-token; `RateLimit-Remaining`/`-Reset` headers). Async writes return an
**action object** `{id, command, status:"running"|"success"|"error", progress, error}` — poll to `success`.
### Shared sub-account (non-billable; on an existing box)
| Op | Method + path | Request | Result |
|----|---------------|---------|--------|
| List | `GET /storage_boxes/{box}/subaccounts` | `?label_selector=k%3Dv` supported | `{subaccounts:[…], meta.pagination}` |
| **Create** | `POST /storage_boxes/{box}/subaccounts` | `{home_directory, password, access_settings:{ssh_enabled,reachable_externally,samba_enabled,webdav_enabled,readonly}, labels, description}` | **201** `{action(create_subaccount), subaccount:{id, storage_box}}`; action ~**0 s** |
| Get | `GET /storage_boxes/{box}/subaccounts/{id}` | — | `{id, storage_box, name, username (uXXXXXX-subN), server (…-subN.your-storagebox.de), home_directory, access_settings, description, labels, created}` |
| **Reset pw** | `POST /storage_boxes/{box}/subaccounts/{id}/actions/reset_subaccount_password` | `{password}` | 201 action `reset_subaccount_password` |
| **Access/quota** | `POST /storage_boxes/{box}/subaccounts/{id}/actions/update_access_settings` | `{ssh_enabled,reachable_externally,samba_enabled,webdav_enabled,readonly}` | 201; `readonly:true` freezes writes |
| Delete | `DELETE /storage_boxes/{box}/subaccounts/{id}` | — | 201 action `delete_subaccount` (async) |
### Dedicated box (billable)
| Op | Method + path | Request | Result |
|----|---------------|---------|--------|
| **Create** | `POST /storage_boxes` | `{name, storage_box_type:"bx11", location:"fsn1", password, access_settings, labels}`**no `ssh_keys` param** (create takes only a password) | **201** `{action(create), storage_box:{id, status:"initializing"}}`; action **~26 s** → `active`, username `uXXXXXX`, server `uXXXXXX.your-storagebox.de` |
| **Rescale** | `POST /storage_boxes/{id}/actions/change_type` | `{storage_box_type:"bx21"}` | 201 action `change_type` (Hetzner allows downscale only if usage fits) |
| Delete | `DELETE /storage_boxes/{id}` | — | 201 action `delete` (**succeeded immediately even mid-resize** → billing stops) |
| Types | `GET /storage_box_types` | — | `bx11`(id 1333,1 TB), `bx21`(1334,5 TB), `bx31`(1335,10 TB), `bx41`… each `subaccounts_limit:100` |
**Errors:** invalid input → **422** `{error:{code:"invalid_input", message, details:{fields:[{name,messages}]}}}`;
not-found → **404** `{error:{code:"not_found"}}`. **Password policy:** must contain upper + lower + digit + a
special char (a 422 on the field otherwise). **Box `name` is NOT unique** (a dup-name create only errored on
the missing password) → idempotency must be **label-based**, not name-based.
## 3. Confirmed (the bootstrap + conventions)
- **Create takes a password; there is no ssh-key param** for boxes or sub-accounts → the
transient-password→install-key path is the only way in. **Proven live:** generate password → API-create →
`ssh-copy-id -p 23 -s -f` (the `-s -f` flags are mandatory on Storage Boxes) → passwordless key auth →
`restic init`+backup+restore round-trip **integrity OK** (chown-immune on an API-created account).
- **`reset_subaccount_password` invalidates the transient password** while the installed key keeps working —
proven (old password → "Permission denied", key → `pwd:/home`). This is the "key-only steady state" the hub
should reach: create → deliver password once → controller installs key → hub resets/discards the password.
- **RepoPath convention:** the account is chrooted to its `home_directory`, presented as `/home` → RepoPath
**`/home/<repo>`** for both models (sub-account and dedicated box `/home` are both writable). Confirmed by
the existing `u629193-sub1` (`home_directory:"felhom-demo/"`) and both spike accounts.
## 4. Quota levers
- **Dedicated:** the **box type** (`change_type`/rescale, Hetzner-enforced hard quota) — bx11 1 TB → bx21 5 TB → …
- **Shared:** **no native per-sub-account quota.** The only API lever is `readonly:true` (a hard freeze,
proven). So a shared-model customer needs a **Felhom soft-quota** (track usage → flip `readonly` when over).
## 5. Impl blueprint (for the hub-provisioning spec)
- **Cloud-API client:** base `https://api.hetzner.com/v1`, Bearer token, poll actions to `success`; structs
mirror §2's field names. Idempotency + teardown via **label_selector** (e.g. `felhom-customer=<id>`).
- **Customer config:** `offsite_enabled` (bool), `offsite_type` (`shared`|`dedicated`), `capacity`
(dedicated → box type; shared → soft-quota GB), `shared_box_id` (for the shared model).
- **Provision flow (per model):** API-create (password generated hub-side; label with the customer id) →
poll action → **deliver the password to the controller once** (one-time-consumable) → controller installs
its key → hub **`reset_subaccount_password`** (discard) → set the non-secret target descriptor
(Host=`<user>.your-storagebox.de` or `<user>-subN…`, User, Port 23, RepoPath `/home/<repo>`, the pinned
host-key line) in the served config → the controller's fork-4 enable→stage→escrow→**auto-confirm** runs.
- **Quota:** dedicated → `change_type`; shared → soft-quota monitor → `update_access_settings{readonly}`.
- **Teardown/offboarding:** `DELETE` the sub-account/box by label.
## 6. Security recommendation (load-bearing)
Hetzner API tokens are **project-scoped and NOT further scopeable** (no per-resource-type tokens). The
`storage-felhom.eu` token (project `15217960`) therefore has **Read+Write over everything in that project**
measured live: **1 cloud server (this is ep0, the production offsite WG endpoint `167.235.97.81`), 2 primary
IPs, 3 SSH keys**. A storage-box-provisioning token can today **delete ep0.** **Recommendation:** move
customer Storage Boxes into a **dedicated Hetzner project** whose only contents are storage boxes, so the
provisioning token's project scope IS the blast-radius boundary. The hub should hold that scoped token, not
one that can touch ep0/servers.
## 7. Teardown state (clean)
Every `spike-*` deleted: the dedicated box `spike-dedicated` (id 611689, `u629463`) DELETED (billing stopped
after a few minutes ≈ cents; delete succeeded even mid-resize); the sub-account `spike-sub` (id 268917,
`u629193-sub2`) DELETED (async, confirmed 0 by label). Only the pre-existing real resources remain: box
`611421` (`PBS-storage-1`) + sub-account `u629193-sub1` (the standing demo offsite) — **untouched**. All
throwaway keys/passwords/request-bodies `shred`ded on DooPlex; the token was only ever read from Viktor's
0600 file (never logged/committed). *(Left in place: `sshpass` installed on DooPlex — a standard tool.)*
## 8. NOT done / next
- The **hub-provisioning impl** (customer config + automated label-tagged create + one-time password → controller
apply + quota + auto-confirm escrow) — this spike hands it the exact shapes.
- **Decide** (impl spec, not here): dedicated-box-per-customer vs pooled sub-accounts on a shared box, per plan
tier — bx11 allows 100 sub-accounts; shared needs the Felhom soft-quota; dedicated gives Hetzner-hard quota.
- The **dedicated-project migration** (§6) before the hub gets a write token.
- Host-key pinning at provision time (ssh-keyscan + cross-check; one host key across the box's sub-accounts).