|
|
|
@@ -1,78 +1,57 @@
|
|
|
|
|
# REPORT — Slice 7 Phase 1: unified bring-up reconcile job (v0.8.0) (2026-06-09)
|
|
|
|
|
# REPORT — Slice 7 close-out: PBS recovery-code escrow creation (v0.9.0) (2026-06-10)
|
|
|
|
|
|
|
|
|
|
> Overwrite-latest report (most recent significant work only). Cumulative history lives in
|
|
|
|
|
> [CHANGELOG.md](CHANGELOG.md). Implements `TASK — Slice 7 Phase 1: unified bring-up reconcile job`.
|
|
|
|
|
> **Agent-only — no hub/wire change** (the new guest auto-appears in the host-report via `ListLXC`).
|
|
|
|
|
> [CHANGELOG.md](CHANGELOG.md). Implements the agent half of `TASK — Slice 7 close-out: PBS
|
|
|
|
|
> recovery-code escrow creation`. The hub opaque-storage half is in `felhom.eu` (hub v0.8.0).
|
|
|
|
|
|
|
|
|
|
## Outcome
|
|
|
|
|
|
|
|
|
|
The shared **front half** of provision and guest-loss DR shipped as a journaled reconcile job
|
|
|
|
|
(`internal/reconcile/bringup.go`), mirroring the slice-6 restore-test's crash-safety but KEEPING
|
|
|
|
|
the guest on success and applying a **scenario-specific identity policy**. Built from the slice-7
|
|
|
|
|
spike findings (commit `3342993`): F1 (MAC reset unconditional on provision), F3 (host keys via a
|
|
|
|
|
baked golden first-boot unit, not an agent guest-internal op), F4 (transient config-lock retry).
|
|
|
|
|
The first code that touches the PBS client encryption key `K` and introduces the customer recovery
|
|
|
|
|
code `R`. Default posture is **zero-knowledge**: Felhom holds an opaque `R`-wrapped blob it cannot
|
|
|
|
|
open; the customer holds `R`. The wrap→recover→restore round-trip was proven on a fenced throwaway
|
|
|
|
|
before any code (`felhom.eu/documentation/tests/slice7-escrow-spike-findings.md`) — the `R`-recovered
|
|
|
|
|
key restored a real encrypted snapshot.
|
|
|
|
|
|
|
|
|
|
## What landed
|
|
|
|
|
## Secret discipline (the overriding rule — held throughout)
|
|
|
|
|
|
|
|
|
|
- **`RunBringUp(BringUpSpec) BringUpResult`** — restore → identity reset → size → attach mounts →
|
|
|
|
|
start LINK-UP, each mutation preceded by journaling the owning entry. **Verdict is liveness**
|
|
|
|
|
(`waitRunning`), never the start exitstatus (reuses the v0.7.0 WARNINGS surface). **Success keeps
|
|
|
|
|
the guest** (the key difference from the restore-test).
|
|
|
|
|
- **Identity policy (doc 03 §9):** *provision* resets MAC unconditionally (`PUT net0`, hwaddr
|
|
|
|
|
omitted → PVE regenerates; F1) + hostname, host-side via the token; machine-id + SSH host keys
|
|
|
|
|
regenerate guest-side on first boot (systemd + the baked unit) — the agent never touches guest
|
|
|
|
|
internals. *dr_guest_loss* preserves continuity (keep hostname; keep MAC unless `KeepMAC=false`);
|
|
|
|
|
never resets restic/tunnel/hub identity.
|
|
|
|
|
- **Compensating rollback:** any mid-flight failure destroys the just-created guest
|
|
|
|
|
(`ClassGuestDestroy` benign via `Provenance{SameTxnCreated:true}`, gated). New journal flag
|
|
|
|
|
`Rollback` + `Recover.recoverBringUp` reap a half-built guest from a mid-job crash (idempotent,
|
|
|
|
|
via `ListLXC`) — distinct from the scratch path's audit label, same destroy machinery.
|
|
|
|
|
- **F4:** identity+sizing+mounts coalesced into ONE `PUT config`; rootfs grow kept separate;
|
|
|
|
|
`setConfigWithLockRetry` retries ONLY the transient PVE config-lock 500 (`pveConfigLock`), never
|
|
|
|
|
a real error.
|
|
|
|
|
- **`--selftest=bring-up`** (`-mode provision|dr -archive -vmid -hostname [-keep]`) — runs the real
|
|
|
|
|
job after a `Recover`, then tears the guest down unless `-keep`.
|
|
|
|
|
- **`configs/build-golden.sh`** — the validated golden recipe incl. the F3 first-boot
|
|
|
|
|
`felhom-regen-hostkeys.service` (Condition-gated: fires on provision, no-ops on DR). The spike's
|
|
|
|
|
golden archive (no unit) is superseded.
|
|
|
|
|
- `R`: `crypto/rand`, ≥128 bits, surfaced **exactly once** (selftest stdout), **never** logged,
|
|
|
|
|
persisted, committed, or in this report. The wrap pty's echoed passphrase is **discarded** (F-A2).
|
|
|
|
|
- `K`: read by location, **never modified** (Wrap operates on a copy; the live key file is
|
|
|
|
|
byte-unchanged — asserted live by sha256), never logged.
|
|
|
|
|
- The blob is opaque ciphertext; the hub stores bytes and never decrypts.
|
|
|
|
|
|
|
|
|
|
## Tests (assert the effect)
|
|
|
|
|
## What landed (`internal/escrow`)
|
|
|
|
|
|
|
|
|
|
`go test ./...` green; `-race` green on the build server. Provision happy path (fresh MAC = net0
|
|
|
|
|
without hwaddr, hostname, coalesced sizing+mount, rootfs-grow separate, started, **guest NOT
|
|
|
|
|
destroyed**); compensating rollback injected at each step (restore / config / start-task /
|
|
|
|
|
waitRunning → asserts the guest **was** destroyed); DR continuity (MAC kept, hostname not reset) +
|
|
|
|
|
DR `KeepMAC=false` resets MAC; liveness verdict (warnings+running pass / not-running fail); F4
|
|
|
|
|
(lock-500 → retry → proceed; non-lock 500 → fail without retry); owning entry journaled **before**
|
|
|
|
|
restore; reserved/existing VMID refused; `Recover` rolls back / clean.
|
|
|
|
|
- **`Create`** → generate `R` (10 EFF-wordlist words ≈ 129 bits) → **Wrap** `K` under `R` via the
|
|
|
|
|
PBS-native `proxmox-backup-client key change-passphrase --kdf scrypt`, driven over a **stdlib pty**
|
|
|
|
|
(`x/sys/unix`; the command is TTY-only, spike F-A1) with **output discarded** (F-A2) → **self-verify**
|
|
|
|
|
the blob unwraps back to `K` (fingerprint match) before shipping → return `R` SEPARATELY (so it
|
|
|
|
|
can't be logged via the result) + the opaque blob.
|
|
|
|
|
- **Opt-in outputs:** (b) `R`-wrapped offline copy (two-factor, no extra trust); (a) raw paperkey
|
|
|
|
|
(single-factor, unrevocable — loud caveat).
|
|
|
|
|
- **`--selftest=escrow-create`** (`-storage`/`-paperkey`/`-offline`/`-upload`): surfaces `R` once,
|
|
|
|
|
prints the opaque blob's size/fingerprint/posture, and with `-upload` PUTs it to the hub
|
|
|
|
|
(`/api/v1/hosts/{host_id}/escrow`, per-host key).
|
|
|
|
|
- Config: `escrow{posture, pbs_storage_id}` (zero_knowledge default); `PBSEncKeyPath` (the `<id>.enc`).
|
|
|
|
|
- Runtime dependency: the `proxmox-backup-client` CLI.
|
|
|
|
|
|
|
|
|
|
## Live validation (demo-felhom) — all green; caught + fixed one real bug
|
|
|
|
|
## Tests
|
|
|
|
|
|
|
|
|
|
Golden built via `configs/build-golden.sh` → `local:backup/vzdump-lxc-9100-2026_06_09-21_32_58.tar.zst`
|
|
|
|
|
(Docker `overlayfs`, host-key unit baked + enabled). All runs via `--selftest=bring-up`.
|
|
|
|
|
`go test ./...` green; `-race` green on the build server (pure-Go parts). `R` entropy ≥128 /
|
|
|
|
|
10-word format / 200-draw uniqueness; the agent→hub wire-contract key-set (mirrors the hub). The
|
|
|
|
|
integration round-trip (wrap→unwrap fingerprint match, **wrong-`R` fails**, **live `K` byte-unchanged**,
|
|
|
|
|
blob ≠ plaintext) is guarded to linux+`proxmox-backup-client` and exercised by the live selftest.
|
|
|
|
|
|
|
|
|
|
- **Bug caught live (then fixed + re-validated):** the first golden's `felhom-regen-hostkeys`
|
|
|
|
|
unit used `ExecStart=/usr/sbin/ssh-keygen` (as drafted in the task) → `status=203/EXEC` ("No such
|
|
|
|
|
file"); on Debian 13 `ssh-keygen` is at `/usr/bin/ssh-keygen`. Fixed in `build-golden.sh`,
|
|
|
|
|
rebuilt the golden, re-validated. (The unit tests mock the guest, so only the live run could
|
|
|
|
|
surface this — exactly why the slice runs live.)
|
|
|
|
|
- **provision** (vmid 8100): restore → fresh MAC `BC:24:11:4A:CF:3C` (≠ golden's) → hostname
|
|
|
|
|
`felhom-prov-8100` → link-up, 8s. Verified: `felhom-regen-hostkeys` **active** and SSH host keys
|
|
|
|
|
present (`root@felhom-prov-8100`) — **regenerated by the baked unit, the agent issued no
|
|
|
|
|
ssh-keygen**; `ssh.service` active; machine-id unique (`103a0325…`); hostname propagated; clean
|
|
|
|
|
DHCP lease `192.168.0.114`; `docker run hello-world` OK → torn down.
|
|
|
|
|
- **dr** (vmid 8101, from a 9999 customer backup): continuity preserved — hostname stayed
|
|
|
|
|
`felhom-selftest-scratch` (NOT reset) and SSH host keys **preserved** (not regenerated). The
|
|
|
|
|
source had no net0, so MAC-keep was a no-op live (covered by the unit test instead).
|
|
|
|
|
- **Recover** (vmid 8102): killed a provision mid-restore (`timeout -s KILL`); the server-side
|
|
|
|
|
restore left an orphan guest; the re-run's `Recover` logged `rolled back half-built bring-up
|
|
|
|
|
guest vmid=8102` (`BringUpRolledBack:1`), then re-provisioned cleanly → torn down. Idempotent.
|
|
|
|
|
## Live validation (demo-felhom)
|
|
|
|
|
|
|
|
|
|
Teardown: no spike guests linger; the validated golden is retained as the artifact; the service
|
|
|
|
|
runs v0.8.0.
|
|
|
|
|
`--selftest=escrow-create -storage felhom-pbs`: `R` (10 words) surfaced **once** to stdout; blob
|
|
|
|
|
**383 B opaque** (key fingerprint `01:36:e9…`, posture `zero_knowledge`, ~129-bit `R`);
|
|
|
|
|
**self-verify** confirmed recoverability; the live `K` (`felhom-pbs.enc`) **sha256 byte-unchanged**;
|
|
|
|
|
the exact `R` was **absent** from stderr and the agent journal. *(Upload to the hub validated after
|
|
|
|
|
hub v0.8.0 deploy — see felhom.eu REPORT.)*
|
|
|
|
|
|
|
|
|
|
## Deferred (stated, not built)
|
|
|
|
|
|
|
|
|
|
Provisioning BACK HALF (controller deploy, bootstrap, per-guest token mint) → slice 8; host-loss
|
|
|
|
|
DR + escrow consumption → slice 10; the SOURCE of a `BringUpSpec` (hub desired-state) → slice 10
|
|
|
|
|
(`GuestMount` defined minimally, no hub coupling). No secrets committed.
|
|
|
|
|
Escrow **consumption / restore-mode serving** → slice 10 (host-loss DR; the spike pre-validated it).
|
|
|
|
|
The **Felhom-holds-a-key** posture (the one dangerous topology cell) is documented (doc 03 §8a) and
|
|
|
|
|
implemented only on an explicit customer opt-in. No secrets committed.
|
|
|
|
|
