doc(03-host-agent): slice-7 scope, scenario-specific identity-reset, PBS escrow (§8a)
- §9 rewritten: bring-up is a shared FRONT HALF only; identity-reset policy is scenario-specific (provision = fresh everything; guest-loss DR = preserve restic/tunnel/hub continuity, reset only collision-prone host-local identity). Added the slice 7/8/10 mapping table. - NEW §8a: PBS recovery-code escrow (zero-knowledge) — live key on box; agent-generated recovery code R; PBS-native passphrase-wrap of K under R escrowed to hub; consumption slice 10; irreducible-residual + rotation != key-rotation stated. - §13 updated (resolved: provision/DR slice boundary + escrow design; open: identity-reset set, hub-side escrow storage + restore-mode serving). Doc-only; no version bump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -192,38 +192,114 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
|||||||
necessity, not just convenience. Integrity-verify (cheap, ciphertext-level) runs more often
|
necessity, not just convenience. Integrity-verify (cheap, ciphertext-level) runs more often
|
||||||
as the lighter check.
|
as the lighter check.
|
||||||
|
|
||||||
|
### 8a. PBS recovery-code escrow (zero-knowledge offsite-key recovery)
|
||||||
|
|
||||||
|
The DR substrate is the PBS offsite tier, and it is client-side encrypted (zero-knowledge): if the
|
||||||
|
box dies, restoring the offsite backups requires the **PBS client encryption key `K`**, which died
|
||||||
|
with the box. The escrow is how `K` comes back **without** Felhom ever being able to read customer
|
||||||
|
data. Design (decisions, with the rationale that pins them):
|
||||||
|
|
||||||
|
- **Live key unencrypted on the box** (`0600`, root): the agent backs up *and* runs restore-tests
|
||||||
|
unattended — no passphrase prompt on the management path. The privilege concentration this
|
||||||
|
implies is the whole argument for §3 root-minimization + a small auditable agent.
|
||||||
|
- **Wrap mechanism — PBS-native, not custom crypto.** At enrollment the agent generates a
|
||||||
|
high-entropy **recovery code `R`** and produces a **passphrase-protected copy of `K` under `R`**
|
||||||
|
using PBS's own key passphrase KDF (`proxmox-backup-client key` family). *Decision: lean on PBS's
|
||||||
|
documented, battle-tested key+passphrase path; do not roll a bespoke AEAD wrap.* Host/customer
|
||||||
|
binding is provided at the hub-storage layer (blob keyed by host-id), not by custom crypto.
|
||||||
|
- **Agent-side generation.** `R` is generated **on the box** (it already holds `K` and does the
|
||||||
|
wrapping), so `R` never touches the hub even in transit — zero-knowledge by construction.
|
||||||
|
- **Escrow = the `R`-wrapped blob → hub.** The hub stores opaque ciphertext bound to the
|
||||||
|
host/customer. Without `R` it is undecryptable; the operator cannot read customer data. (Hub-side
|
||||||
|
storage schema for the blob is a slice-10 / doc-05 item.)
|
||||||
|
- **Recovery code custody.** `R` is shown to the customer **once** at enrollment (printed/displayed)
|
||||||
|
and **never stored by Felhom in recoverable form**. Format: a grouped/word-list code (≥128-bit
|
||||||
|
entropy) — it is transcribed off paper by a non-technical household, so raw base32 invites typos.
|
||||||
|
- **Consumption (slice 10, host-loss).** New box re-enrolls in restore mode → hub ships the escrow
|
||||||
|
blob → customer enters `R` → box unwraps `K` → PBS restores proceed.
|
||||||
|
- **Optional belt-and-suspenders (product decision, default OFF).** A PBS **paperkey** (the raw key,
|
||||||
|
for a safe) gives the customer a recovery path that survives *both* box loss *and* recovery-code
|
||||||
|
loss, at the cost of a higher-value secret (raw key on paper, no second factor). Default is
|
||||||
|
hub-escrow + `R` only; offer the paperkey as an opt-in "advanced" path.
|
||||||
|
|
||||||
|
**Properties stated for honesty (these go to the customer at enrollment):**
|
||||||
|
- **Irreducible residual:** losing `R` *and* the box (and, if not opted in, having no paperkey) =
|
||||||
|
the offsite backups are **unrecoverable, by anyone, including Felhom.** This is the cost of
|
||||||
|
genuine zero-knowledge and must be communicated, not buried.
|
||||||
|
- **Rotation ≠ key rotation:** rotating `R` re-wraps the escrow blob (and re-shows the customer a
|
||||||
|
new code) but does **not** re-encrypt existing PBS data — that data stays keyed by `K`. Changing
|
||||||
|
`K` itself is a separate, heavier operation (new key → new backups; old backups still need old
|
||||||
|
`K`) and is out of scope for routine recovery-code rotation.
|
||||||
|
|
||||||
## 9. Provisioning & DR flows
|
## 9. Provisioning & DR flows
|
||||||
|
|
||||||
**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs
|
**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs
|
||||||
the `keyctl=1` feature flag, which Proxmox permits only for `root@pam` (Phase 3, B3) — not the
|
the `keyctl=1` feature flag, which Proxmox permits only for `root@pam` (Phase 3, B3) — not the
|
||||||
scoped token. But a token-authorized **restore preserves `keyctl`** (Phase 3, B3), so the agent
|
scoped token. But a token-authorized **restore preserves `keyctl`** (Phase 3, B3, empirically:
|
||||||
provisions **by restoring a golden base image**, never by `pct create` on the per-customer path:
|
a token `vzrestore` of a keyctl archive produced a guest that kept `features:
|
||||||
|
nesting=1,keyctl=1,unprivileged:1`), so the agent provisions **by restoring a golden base
|
||||||
|
image**, never by `pct create` on the per-customer path.
|
||||||
|
|
||||||
- A **golden base archive** — minimal Debian + Docker, `nesting=1,keyctl=1`, overlayfs — is
|
**Golden base image.** A **golden base archive** — minimal Debian + Docker, `nesting=1,keyctl=1`,
|
||||||
built once as `root@pam` **at enrollment** (when the agent legitimately holds root to mint its
|
overlayfs — is built once as `root@pam` **at enrollment** (when the agent legitimately holds root
|
||||||
Proxmox token) and refreshed on a maintenance cadence. This is the one place `keyctl`/root
|
to mint its Proxmox token) and refreshed on a maintenance cadence. This is the one place
|
||||||
provisioning lives — off the per-customer path.
|
`keyctl`/root provisioning lives — off the per-customer path. Refresh cadence + fleet versioning
|
||||||
- To provision guest G: restore the golden archive → new VMID (token-covered: `VM.Allocate` +
|
remain an operational open item (§13).
|
||||||
`Datastore.AllocateSpace`; `keyctl` preserved) → reset identity (MAC/hostname) → size the guest
|
|
||||||
(CPU/mem config + `pct resize` rootfs, token-covered) → attach storage mounts per the manifest
|
|
||||||
→ deploy the controller → hand it bootstrap config. A mid-flight failure is journaled and
|
|
||||||
compensating-rolled-back (destroy the just-restored guest — allowed without a signature per §4,
|
|
||||||
same-transaction provenance).
|
|
||||||
|
|
||||||
**Unified bring-up primitive.** Provisioning and DR-restore share the same token-covered front
|
**Unified bring-up primitive (shared *front half* — NOT shared identity policy).** Provisioning
|
||||||
half — *restore an archive → reset identity* — and differ only in the archive and the back half:
|
and DR-restore share one token-covered front-half code path:
|
||||||
provisioning restores the **golden base** then deploys a fresh controller; DR-restore restores
|
|
||||||
the **customer's backup** (already containing controller + data), brings it up, and reattaches
|
|
||||||
external storage. One code path, exercised by every restore-test (§8).
|
|
||||||
|
|
||||||
**Guest loss.** Agent restores G from the fastest surviving tier and resets identity
|
> restore an archive → **reset identity** → size the guest (CPU/mem config + `pct resize`
|
||||||
(MAC/hostname) so the restored guest rejoins cleanly — this *is* the unified restore primitive
|
> rootfs, token-covered) → attach storage mounts per the manifest
|
||||||
above (customer-backup archive, DR back half).
|
|
||||||
|
|
||||||
**Host/hardware loss.** Re-enroll the new host in **restore mode**; the hub — the durable
|
run as a **journaled reconcile job**; a mid-flight failure is compensating-rolled-back (destroy
|
||||||
source of truth that survives box death — hands the new agent the existing identity, PBS
|
the just-restored guest — allowed unsigned per §4, same-transaction provenance). They diverge in
|
||||||
namespace, tunnel token, storage manifest, and a restore directive. Tunnel is reused from
|
the *archive* and the *back half*, **and in identity policy** (below).
|
||||||
the hub record, so DNS stays intact.
|
|
||||||
|
**Identity reset is scenario-specific — this is a correctness boundary, not a detail.** "Reset
|
||||||
|
identity" is shorthand for two different operations:
|
||||||
|
|
||||||
|
- **Provision (golden base) → fresh identity, everything.** A provisioned guest is new: regenerate
|
||||||
|
MAC, hostname, **`/etc/machine-id`** (a duplicate breaks journald/DHCP/systemd), **SSH host
|
||||||
|
keys**, and it receives a **fresh** controller identity (host-id, local token, hub channel),
|
||||||
|
**fresh restic repo identity**, and a fresh tunnel association — all minted in the back half.
|
||||||
|
- **Guest-loss DR (customer backup) → preserve continuity identity, reset only what would
|
||||||
|
collide.** The restored guest must *continue* the customer's world: **keep** the restic repo
|
||||||
|
identity (resetting it orphans the existing backup chain — a silent data-continuity bug), the
|
||||||
|
tunnel/DNS association, and the hub host/customer binding. Reset only collision-prone host-local
|
||||||
|
identity (`machine-id`, SSH host keys, hostname as needed). **MAC is reset only when a source
|
||||||
|
guest may still be live** (e.g. partial loss, or the restore-*test* which boots link-down for
|
||||||
|
exactly this reason); in a true total guest-loss the original is gone, so the MAC can be kept to
|
||||||
|
preserve DHCP reservations. The agent decides MAC handling from the scenario, not a fixed rule.
|
||||||
|
|
||||||
|
The exact reset set is being pinned empirically by the slice-7 bring-up spike (live, link-up,
|
||||||
|
which the slice-6 restore-test never did — it boots link-down precisely because identity reset is
|
||||||
|
slice 7).
|
||||||
|
|
||||||
|
**Guest loss (slice 7).** Agent restores G from the fastest surviving tier (snapshot → local →
|
||||||
|
PBS) and applies the **DR identity policy** above so the restored guest rejoins cleanly. The
|
||||||
|
customer backup already contains the controller + data, so there is **no controller deploy** in
|
||||||
|
this path — bring up + reattach external storage and it is whole. This is fully in slice 7.
|
||||||
|
|
||||||
|
### Slice mapping (what is built where — keep this current)
|
||||||
|
|
||||||
|
| Capability | Slice | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| Golden base image build (root@pam, at enrollment) | **7** | spike → build |
|
||||||
|
| Unified bring-up **front half** (restore→reset identity→size→attach storage), journaled + compensating rollback | **7** | spike → spec → implement |
|
||||||
|
| **Guest-loss DR** (front half + DR identity policy; no controller deploy) | **7** | in scope |
|
||||||
|
| PBS recovery-code escrow **creation** (§8a) | **7** | designed (§8a); implement |
|
||||||
|
| Provisioning **back half** — deploy controller, hand bootstrap config, mint per-guest local token | **8** | deferred — needs the controller-deploy path + agent↔controller local API (§6) |
|
||||||
|
| **Host/hardware loss** DR — re-enroll in "restore mode"; hub serves identity / PBS namespace / tunnel token / storage manifest / restore directive | **10** | deferred — needs hub desired-state serving; hub store today holds only `{host_id, customer_id, api_key}` (slice 3) |
|
||||||
|
| PBS escrow **consumption** (recover `K` on a new box) | **10** | deferred — exercised by host-loss DR |
|
||||||
|
| Golden base refresh cadence + fleet versioning | post-launch | operational, non-blocking (§13) |
|
||||||
|
|
||||||
|
**Host/hardware loss (design intent — slice 10).** Re-enroll the new host in **restore mode**;
|
||||||
|
the hub — the durable source of truth that survives box death — hands the new agent the existing
|
||||||
|
identity, PBS namespace, tunnel token, storage manifest, a restore directive, and the **escrow
|
||||||
|
blob** (§8a) for the customer to unlock with their recovery code. Tunnel is reused from the hub
|
||||||
|
record, so DNS stays intact. This depends on hub desired-state serving (slice 10) and is not
|
||||||
|
buildable until then; recorded here so the front-half built in slice 7 lands ready for it.
|
||||||
|
|
||||||
## 10. Concurrency, crash-safety, idempotency
|
## 10. Concurrency, crash-safety, idempotency
|
||||||
|
|
||||||
@@ -263,15 +339,21 @@ argument for §3's root-minimization and a small, auditable agent.
|
|||||||
|
|
||||||
Resolved here: tunnel placement (host, agent-managed, own systemd service), the
|
Resolved here: tunnel placement (host, agent-managed, own systemd service), the
|
||||||
reconcile-vs-jobs fork (hybrid, gated by reversibility), agent process model, self-update
|
reconcile-vs-jobs fork (hybrid, gated by reversibility), agent process model, self-update
|
||||||
ownership, the local-API surface, the storage-manifest schema, **provision-by-restore**, and
|
ownership, the local-API surface, the storage-manifest schema, **provision-by-restore**, the
|
||||||
the **root-vs-API boundary** (Phase 3, B3).
|
**provision/DR slice boundary** (7 front-half + guest-loss DR + escrow creation; 8 provisioning
|
||||||
|
back-half; 10 host-loss DR + escrow consumption — §9 table), the **PBS recovery-code escrow
|
||||||
|
design** (§8a), and the **root-vs-API boundary** (Phase 3, B3).
|
||||||
|
|
||||||
Still open:
|
Still open:
|
||||||
|
|
||||||
- Multi-tenant **resource fairness** on a shared host (per-guest cgroup limits, noisy-neighbor) — deferred to the company-case pass.
|
- Multi-tenant **resource fairness** on a shared host (per-guest cgroup limits, noisy-neighbor) — deferred to the company-case pass.
|
||||||
- Operator-side **signing tooling** — where the operator signing key lives operationally and how a destructive op gets signed without undue friction (offline key vs. a small signing service; the security floor is "not in the hub").
|
- Operator-side **signing tooling** — where the operator signing key lives operationally and how a destructive op gets signed without undue friction (offline key vs. a small signing service; the security floor is "not in the hub").
|
||||||
- Hub-side **desired-state editing UX** and the host-domain report schema details — belong to the hub architecture doc.
|
- Hub-side **desired-state editing UX** and the host-domain report schema details — belong to the hub architecture doc.
|
||||||
- **Golden base image** refresh cadence + fleet versioning — who triggers a rebuild, how the per-host image version is tracked (operational detail, not blocking; §9).
|
- **Golden base image** refresh cadence + fleet versioning — operational, non-blocking (§9).
|
||||||
|
- **Identity-reset set** (live, link-up) — pinned empirically by the slice-7 bring-up spike; the
|
||||||
|
scenario-specific policy is settled in §9, the exact field list is the spike's deliverable.
|
||||||
|
- **Hub-side escrow storage + restore-mode serving** — the blob's hub schema and the restore-mode
|
||||||
|
desired-state handover are slice-10 / doc-05 (§8a, §9 host-loss).
|
||||||
|
|
||||||
This doc hands the implementation three contracts it was waiting on:
|
This doc hands the implementation three contracts it was waiting on:
|
||||||
|
|
||||||
@@ -283,6 +365,17 @@ This doc hands the implementation three contracts it was waiting on:
|
|||||||
|
|
||||||
## Changelog — design-review + Phase-3 fold-in (2026-06-08)
|
## Changelog — design-review + Phase-3 fold-in (2026-06-08)
|
||||||
|
|
||||||
|
### Slice-7 scope + escrow design (2026-06-09)
|
||||||
|
- §9 rewritten: the bring-up primitive is a **shared front half only** — identity-reset policy is
|
||||||
|
**scenario-specific** (provision = fresh everything; guest-loss DR = preserve restic/tunnel/hub
|
||||||
|
continuity identity, reset only collision-prone host-local identity). Added the **slice 7/8/10
|
||||||
|
mapping table** (front half + guest-loss DR + escrow creation in 7; provisioning back-half in 8;
|
||||||
|
host-loss DR + escrow consumption in 10).
|
||||||
|
- NEW §8a: **PBS recovery-code escrow** — live key unencrypted on box for unattended ops; agent
|
||||||
|
generates recovery code `R`; PBS-native passphrase-wrap of `K` under `R` escrowed to the hub
|
||||||
|
(zero-knowledge); consumption is slice 10. Irreducible-residual + rotation≠key-rotation stated.
|
||||||
|
- §13 updated accordingly.
|
||||||
|
|
||||||
- **NEW provision-by-restore** (§9): the agent provisions by **restoring a golden base image**
|
- **NEW provision-by-restore** (§9): the agent provisions by **restoring a golden base image**
|
||||||
(token-covered, preserves `keyctl`), never `pct create` on the per-customer path; one unified
|
(token-covered, preserves `keyctl`), never `pct create` on the per-customer path; one unified
|
||||||
restore primitive shared with DR. §2 responsibility + §3 boundary updated.
|
restore primitive shared with DR. §2 responsibility + §3 boundary updated.
|
||||||
|
|||||||
Reference in New Issue
Block a user