docs: DR-tier-by-default batch — decisions 1-5 in CONTEXT, day0 ACL advice reversed, ceremony decision-4 policy block, WG disclosure (tester agreement + GO-LIVE D3), drill F-disposition table, REPORT

Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
2026-07-12 20:43:25 +02:00
parent 448a68237a
commit fda783bdaf
7 changed files with 126 additions and 68 deletions
@@ -129,16 +129,16 @@ tier (ep0)** — which converted the drill into a full rehearsal of Peti's pendi

 | # | Sev | Phase | Finding | Disposition |
 |---|---|---|---|---|
-| F-1 | LOW | 3 | Version-string mismatches: `-h` v1.14.0 vs run banner v1.13.0 vs hub Setup-tab copy "1.12.0" | fix strings (installer + hub template) |
-| F-2 | COSMETIC | 3 | dry-run prints `curl -u <git>` on the anonymous-fetch branch | fix placeholder |
-| F-3 | MEDIUM | 4 | Root-run provision leaves `/var/lib/felhom-agent/guests{,/9201}` root:root 0700 inside the agent-owned state dir → non-root agent lanresolver "permission denied". LIVE-FIXED (chown the two parent dirs; the guest-root-owned bootstrap subtree untouched) | agent/installer: create parents agent-owned at provision; **check demo/felhom-pve for the same latent state** |
+| F-1 | LOW | 3 | Version-string mismatches: `-h` v1.14.0 vs run banner v1.13.0 vs hub Setup-tab copy "1.12.0" | **SHIPPED 2026-07-12 (installer v1.15.0 + hub v0.51.0):** single `SCRIPT_VERSION` source (header carries no literal; `-h`/banners read it; hub `hostInstallVersion`=1.15.0) — drift now gated by `scripts/hostinstall_gates.py` (asserts installer == hub const) |
+| F-2 | COSMETIC | 3 | dry-run prints `curl -u <git>` on the anonymous-fetch branch | **SHIPPED (installer v1.15.0):** dry-run fetch lines print the real auth mode (`curl (anonymous — no git credential)` vs `-u <git>`) |
+| F-3 | MEDIUM | 4 | Root-run provision leaves `/var/lib/felhom-agent/guests{,/9201}` root:root 0700 inside the agent-owned state dir → non-root agent lanresolver "permission denied". LIVE-FIXED (chown the two parent dirs; the guest-root-owned bootstrap subtree untouched) | **SHIPPED (agent v0.86.0):** a root-run provision chowns the `guests/`+`guests/<vmid>/` PARENTS to the state-dir owner (`chown --reference`, non-recursive; bootstrap leaf stays guest-root) — red-proofed. **felhom-pve latent state CONFIRMED + FIXED live 2026-07-12** (same chown); Peti's host unreachable — his check deferred to the D/E/G visit |
 | **F-4** | **HIGH** | 5 | **No operator-set dashboard password path exists anywhere** (hub has no UI/API for per-customer `password_hash`; controller open-state page defers to the operator; Day-0 preseeded path skips the wizard's password form) → **G10 unclosable; every fresh box's dashboard stays OPEN on the internet** | **RESOLVED 2026-07-12 — customer-claim arc (controller v0.122.0 + hub v0.50.0).** The customer OWNS the password, set via a one-time claim code the hub emails to the registered address; an unclaimed box serves only the claim page (no open window). The operator-set premise was correctly abandoned. Live-proven on the drill box (gate ON via the real edge). |
 | F-5 | MEDIUM | 5 | Geo-restriction APPLY is customer-dashboard-side; hub only disables. Runbook premise stale; also: the open dashboard (F-4) exposes the geo toggle unauthenticated | **RESOLVED with F-4** — the geo toggle (and every other route) is now behind the claim gate on an unclaimed box. The apply-is-customer-side design is accepted (documented). |
-| **F-6** | **HIGH** | 6 | **Identity-only escrow ceremony was never implemented** (`escrow-create` hard-requires PBS storage + key; the ≥0.80.0 runbook claim was false) → on a no-PBS box (the documented appliance standard!) the offsite escrow chain can never complete | agent feature task (identity-only mode); ceremony runbook corrected in this commit |
-| F-7 | MEDIUM | 6 | host-install ships the FELHOM_PBSDR sudoers alias but NOT the `felhom-pbs-apply` binary → pbsdr capabilities born DEGRADED on every fresh box | installer: ship the wrapper (like mkfs/selfupdate wrappers) |
-| F-8 | LOW/UX | 3 | Step 4b rotates root@pam + vaults silently — operator surprised by 401 at the PVE GUI (live: Viktor) | installer: print "root@pam rotated + vaulted — retrieve at hub → host page" |
-| F-9 | NOTE | 6 | Installer never sets `wg_tunnel.enabled`; WG registration itself is hands-free once enabled | decide appliance default alongside the F-6 spec |
-| F-10 | MEDIUM | 6 | `age` (ceremony identity-wrap dependency) not installed by host-install — fresh-box ceremony dies; demo host masked it (spike-era install) | installer: add `age` to package set; runbook prereq added in this commit |
+| **F-6** | **HIGH** | 6 | **Identity-only escrow ceremony was never implemented** (`escrow-create` hard-requires PBS storage + key; the ≥0.80.0 runbook claim was false) → on a no-PBS box (the documented appliance standard!) the offsite escrow chain can never complete | **CLOSED BY POLICY 2026-07-12 (DR-tier-by-default decisions 3/4):** identity-only escrow is formally PARKED; every install is DR-capable (installer v1.15.0), activation = the hub per-customer `dr_tier` flag (default ON, hub v0.51.0), and offsite REQUIRES the tier (guard: "Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)"). The no-PBS shape can no longer reach the offsite arc. Ceremony runbook carries the decision block |
+| F-7 | MEDIUM | 6 | host-install ships the FELHOM_PBSDR sudoers alias but NOT the `felhom-pbs-apply` binary → pbsdr capabilities born DEGRADED on every fresh box | **SHIPPED (installer v1.15.0):** wrapper fetched + `bash -n` + installed 0755 (the mkfs/selfupdate pattern); uninstall removes it; gate-asserted. Companion: agent v0.86.0 adds the `inactive` capability state (DR off = neutral chip, NEVER degraded; missing binary STAYS degraded + hub migration hint) |
+| F-8 | LOW/UX | 3 | Step 4b rotates root@pam + vaults silently — operator surprised by 401 at the PVE GUI (live: Viktor) | **SHIPPED (installer v1.15.0):** loud WARN at 4b ("the root@pam password just CHANGED… retrieve at hub → host page") + repeated in the final summary; pointer only, never the credential |
+| F-9 | NOTE | 6 | Installer never sets `wg_tunnel.enabled`; WG registration itself is hands-free once enabled | **DECIDED + SHIPPED (decision 5; installer v1.15.0):** WG is BASE infrastructure — `wg_tunnel.enabled: true` rendered in BOTH modes (setdefault; deliberate preserved-off respected). byo wg-off assert retired; consent = typed disclosure ack + tester-agreement line. Hub v0.51.0 auto-provisions the PBS-DR descriptor on the peer registration (hands-free cascade) |
+| F-10 | MEDIUM | 6 | `age` (ceremony identity-wrap dependency) not installed by host-install — fresh-box ceremony dies; demo host masked it (spike-era install) | **SHIPPED (installer v1.15.0):** `age` installed idempotently in step 5 (both modes); gate-asserted; runbook prereq updated |
 | F-11 | LOW | 6 | Native `confirm()` on the offbox restore-verify form: blocks browser automation, easy to miss (operator missed it twice live; meanwhile launched the full tier-1 restore from the adjacent form — the two restore controls invite confusion) | convert to the design-system inline confirm pattern; consider renaming |

 Observations (not F-numbered): hub "REGISTRY LATEST v0.120.0 — up to date" vs the dashboard's own
@@ -21,7 +21,7 @@ catalog `2ebe082a` · felhom.eu `1a1e42ad`/hub v0.35.0 · host-install v1.9.1 ·
 |---|------|----------|
 | D1 | Pool model | **One shared `felhom` pool.** No multi-tenant boxes yet; the pool separates Felhom guests from the host owner's own workloads. Effort goes to preflight pool-membership assertion + local-API per-guest token scoping (verified structural, see G10). |
 | D2 | Pilot topology | **BYO host.** Peti installs Felhom onto his OWN Proxmox server, already running his own LXCs. This reframes containment: not customer-vs-customer, but *Felhom-as-a-guest on a host the operator does not own*. See §2. |
-| D3 | Backup posture | **Local-only for the pilot.** `wg_tunnel.enabled` and `oob.enabled` stay FALSE. Offsite (Tier-1/Tier-2 split + production WG endpoint) is a fast-follow, and the limitation is stated honestly in the tester agreement. |
+| D3 | Backup posture | ~~Local-only for the pilot; `wg_tunnel.enabled` and `oob.enabled` stay FALSE.~~ **SUPERSEDED 2026-07-12 (DR-tier-by-default, decisions 1/5):** every install ships the full DR plumbing with `wg_tunnel.enabled=true` — WG is BASE infrastructure (outbound-only, like the CF tunnel; also the future OOB transport); activation of the PBS-DR tier is the hub's per-customer flag (default ON). `oob.enabled` stays FALSE on BYO. Consent moves to the installer's typed disclosure ack + the tester-agreement WG disclosure line (§2). |
 | D4 | Uninstall drill | **Full provision → uninstall → re-provision drill on the demo box BEFORE Peti's day-0.** Peti's box is never the first live test of anything destructive. |
 | D5 | DR bind-override | **Fix now** (small, spike-validated known-constant override per `SPIKE-dr-bindmount-source-2026-07-07.md`), not deferred to the intermediary-mount re-architecture. |
 | D6 | Packaging | **This one tracking doc + a sequenced set of CC tasks** (GL-1…GL-7). |
@@ -44,12 +44,17 @@ mechanisms:
  credential to the hub. Peti's root password is his own and is never seen or changed.
 - **Out-of-band operator SSH is OFF** (`oob.enabled=false`). Felhom opens no operator tunnel into
  his box.
- **The offsite WireGuard tunnel is OFF** (`wg_tunnel.enabled=false`). No Felhom-side connection
-  reaches back into his network.
+- **WireGuard tunnel DISCLOSURE (changed 2026-07-12, DR-tier-by-default decision 5):** the install
+  DOES enable an **outbound** WireGuard tunnel (`wg_tunnel.enabled=true`) — it is base
+  infrastructure like the Cloudflare tunnel, carries the offsite/DR backup traffic, and is the
+  future out-of-band management transport. It is strictly box-dials-out (no Felhom-side connection
+  reaches back into the network; the peer has no inbound route), it is listed in the installer's
+  typed host-mutation acknowledgement, and it can be disabled per-customer hub-side on request —
+  with the honest consequence that the DR tier and offsite backup stop working without it.

-These are asserted at install time (the byo config-write asserts `oob`/`wg_tunnel`/`lan_resolver`
-are all off before the install is marked complete) and were live-validated on the demo host during
-the GL-6 drill.
+These are asserted at install time (the byo config-write asserts `oob`/`lan_resolver` are off
+before the install is marked complete — `wg_tunnel` moved to the disclosed-default list in
+installer v1.15.0) and were live-validated on the demo host during the GL-6 drill.

 ---

@@ -19,14 +19,20 @@
 ## Prerequisites (check BEFORE scheduling with the customer)
 - **Agent version:** ≥ v0.79.0 (the ceremony records `restic_pw_sha256` — older agents produce a blob
  auto-confirm can never match).
- **⚠ No-PBS hosts: the ceremony CANNOT run.** The previously documented "identity-only mode
-  (≥ v0.80.0)" was NEVER implemented — v0.80.0's actual feature was seeding `escrow.pbs_storage_id`
-  on PBS hosts. `escrow-create` hard-requires a PBS storage id + its key file (drill-proven
-  2026-07-12, finding F-6 of DRILL-day0-vm-2026-07-12.md). Until identity-only ships, a box MUST
-  have the PBS DR tier (which itself requires the WG tunnel peer first) before any escrow/offsite
-  arc can complete.
- **Host packages:** `age` must be installed (identity wrap dependency; NOT installed by
-  host-install as of v1.14.0 — drill finding F-10). `apt-get install -y age`.
+- **⚠ No-PBS hosts: the ceremony CANNOT run — and that is now POLICY, not a gap.** The previously
+  documented "identity-only mode (≥ v0.80.0)" was NEVER implemented — v0.80.0's actual feature was
+  seeding `escrow.pbs_storage_id` on PBS hosts. `escrow-create` hard-requires a PBS storage id +
+  its key file (drill-proven 2026-07-12, finding F-6 of DRILL-day0-vm-2026-07-12.md).
+  **Operator decision 2026-07-12 (DR-tier-by-default, decision 4): identity-only escrow is
+  formally PARKED — F-6 is closed by policy, not by implementation.** Every install ships the
+  full DR plumbing (host-install ≥ v1.15.0); activation is the hub's per-customer DR-tier flag
+  (default ON), and offsite backup REQUIRES that flag (hub v0.51.0 enforces the coupling:
+  "Offsite backup requires the DR tier"). A box therefore always reaches the ceremony THROUGH
+  the tier: WG peer → PBS-DR descriptor applied → ceremony. Revisit identity-only ONLY if a
+  customer class demands lean-tier offsite — this line is here so a future session knows the
+  absence is a decision, not an oversight.
+- **Host packages:** `age` must be installed (identity wrap dependency; installed by host-install
+  since v1.15.0 — drill finding F-10 closed. Pre-v1.15.0 boxes: `apt-get install -y age`).
 - **K gate (PBS hosts):** `escrow.pbs_storage_id` set and the key file present
  (`cfg.Backup.PBSEncKeyPath(<id>)`).
 - **Staged secret (offsite):** offsite enabled → `EscrowState="pending"` on the controller and the staged
@@ -124,13 +124,16 @@ Notes:
 - vmid picking: the agent's token is pool-scoped, so the hub's guest inventory only shows
  Felhom-pool guests — other guests on a shared box are invisible there. Always pick from
  `pct list` + `qm list` on the box (`documentation/runbooks/provisioning.md`).
- Offsite backup (`felhom-pbs`) is NOT part of a go-live today: the PBS host is LAN-only. The
-  transport is DECIDED and DESIGNED — plain WireGuard, host-side agent-managed, box-dials-out to
-  a public endpoint VM running the offsite PBS. Evidence:
-  `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`; design-of-record + slice
-  roadmap: `documentation/architecture/06-offsite-connectivity.md` (MVP = slices S1→S2→S3, then
-  S4). Go-live standard = **local backups only** (that is what the Part C command configures);
-  offsite DR is retrofitted later via `--rescope-acl` / `--acl-storages` without reinstalling.
+- **DR-tier-by-default (v1.15.0, 2026-07-12):** every install deploys the FULL DR plumbing
+  uniformly — the `felhom-pbs-apply` wrapper, `age`, `wg_tunnel.enabled: true` (WG is base
+  infrastructure like cloudflared), and the default ACL incl. the pre-positioned
+  `/storage/felhom-pbs` grant. **Capability is baked; ACTIVATION is the hub's per-customer
+  "DR tier" flag** (default ON for new customers). With the flag ON the cascade converges
+  hands-free after install: WG peer registers → the hub auto-provisions the ep0 tenancy → the
+  agent applies the storage entry → the escrow ceremony becomes possible. Flag OFF = zero
+  Felhom-side cost; the pbsdr capabilities report a neutral "inactive". Offsite app backup
+  REQUIRES the tier (the ceremony depends on the PBS key — drill F-6, closed by policy).
+  Design-of-record: `documentation/architecture/06-offsite-connectivity.md`.

 ---

@@ -141,7 +144,7 @@ Notes:
 ```bash
 curl -fsSO https://felhom.eu/scripts/felhom-host-install.sh
 chmod +x felhom-host-install.sh
-./felhom-host-install.sh -h | head -3      # sanity: must print v1.10.0 (or newer) — older scripts don't know --mode
+./felhom-host-install.sh -h | head -3      # sanity: must print v1.15.0 (or newer) — DR-tier-by-default ships the full plumbing
 ```

 ### C.2 Preview (recommended)
@@ -152,7 +155,7 @@ read-only and asks for the passphrase):
 ```bash
 ./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode appliance --vmid <VMID> \
    --cores 2 --memory 4096 \
-    --force-gitea-golden --acl-storages "local local-lvm" \
+    --force-gitea-golden \
    --dry-run
 ```

@@ -161,7 +164,7 @@ read-only and asks for the passphrase):
 ```bash
 ./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode appliance --vmid <VMID> \
    --cores 2 --memory 4096 \
-    --force-gitea-golden --acl-storages "local local-lvm"
+    --force-gitea-golden
 ```

 Placeholders and flags — what and why:
@@ -173,7 +176,7 @@ Placeholders and flags — what and why:
 | `--vmid` | the free vmid from Part B | ALWAYS pass it explicitly — deterministic, and recorded in the install state for a later `--uninstall` |
 | `--cores 2 --memory 4096` | the appliance cap | protects the other guests on a shared box; size to roughly half the host if the box is small (cap ≤ host resources) |
 | `--force-gitea-golden` | — | a fresh box has no local golden archive; fetch it from Gitea and verify its sha256 against the hub manifest (this is the normal customer path) |
-| `--acl-storages "local local-lvm"` | — | grants the agent's token write access on exactly these storages. The default set includes `felhom-pbs` (offsite), which does not exist on a customer box — granting it is harmless but misleading; scope to what is real. |
+| `--acl-storages` | **do NOT pass it** | the DEFAULT set (`local local-lvm felhom-pbs`) is correct — the `felhom-pbs` grant is PRE-POSITIONED for the PBS-DR tier (the storage appears when the tier applies). Narrowing it 403'd the apply-bridge live (DRILL-day0-vm-2026-07-12: `Datastore.Allocate` on `/storage/felhom-pbs`) and needed a retrofit dual-grant. Only override on a box with genuinely different storage names. |
 | `--node <NAME>` | only on a **cluster** | the script refuses to auto-pick a node on a multi-node cluster |

 When prompted, type the **retrieval passphrase** (no echo; never on the command line). Alternative
@@ -213,8 +216,10 @@ guest. The installer therefore (a) never touches the owner's credentials — the
 workloads as the thing to protect — the CPU/RAM caps are mandatory, they are the only
 noisy-neighbor containment there; (c) refuses anything host-invasive that the appliance profile
 allows (`--enable-oob`, `--rotate-recovery`, unvalidated PVE majors, ACL grants on storages the box
-doesn't have); and (d) asserts the written agent config keeps `lan_resolver`/`wg_tunnel`/`oob` OFF
-— the agent must not take over the owner's DNS or open tunnels. Recovery on a BYO box = the owner's
+doesn't have); and (d) asserts the written agent config keeps `lan_resolver`/`oob` OFF
+— the agent must not take over the owner's DNS or start a second sshd. (`wg_tunnel` is ON in
+BOTH modes since v1.15.0 — WG is base infrastructure; byo consent = the typed disclosure ack,
+which now lists the outbound tunnel, + the tester-agreement disclosure line.) Recovery on a BYO box = the owner's
 own console access, by design.

 **Recommended first run — preflight only** (all step-1 checks incl. the byo gates, an explicit
@@ -223,7 +228,7 @@ PASS/FAIL verdict, exit 0/1, writes nothing — re-runnable any number of times)
 ```bash
 ./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode byo --vmid <VMID> \
    --cores <N> --memory <MiB> \
-    --force-gitea-golden --acl-storages "local local-lvm" \
+    --force-gitea-golden \
    --preflight-only
 ```

@@ -232,14 +237,15 @@ PASS/FAIL verdict, exit 0/1, writes nothing — re-runnable any number of times)
 ```bash
 ./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode byo --vmid <VMID> \
    --cores <N> --memory <MiB> \
-    --force-gitea-golden --acl-storages "local local-lvm"
+    --force-gitea-golden
 ```

 What byo does differently (everything else matches C.4's eight steps):

 - `--cores` AND `--memory` are **required** — size them WITH the owner (the guest must never starve
-  the owner's guests). `--acl-storages` must name storages that actually exist on the box, or the
-  preflight dies naming the missing ones.
+  the owner's guests). `--acl-storages` (when overridden) must name storages that actually
+  exist on the box, or the preflight dies naming the missing ones — EXCEPT `felhom-pbs`, which
+  is expected absent pre-tier (the PBS-DR apply creates it; the grant is pre-positioned).
 - **The host's `:53` must be free** (v1.12.0/F6): Felhom publishes the guest by name on the LAN, so a
  resolver already bound to `:53` blocks it. byo preflight **REFUSES** (does not touch the owner's
  service — it is a refusal, not a change) with: *"a resolver is already bound to :53 … stop or
@@ -334,8 +340,11 @@ golden older than 0.86.0? See Part F troubleshooting.)

 - **Hosts**: the new host reports (status OK, fresh last-report timestamp).
 - The host's guest list shows `<VMID>` running.
- **Capabilities**: 0 degraded. On a reduced-storage box (no `felhom-pbs`) the offsite-backup
-  related capability is expected degraded — that is the documented go-live standard, not a fault.
+- **Capabilities** (rendered on the host page since hub v0.51.0): **0 degraded** — v1.15.0 ships
+  the full DR plumbing, so the drill's "3 pbsdr-* born DEGRADED" is gone. With the customer's DR
+  tier flag OFF the pbsdr capabilities show a neutral **inactive** ("disabled by configuration")
+  chip — deliberate, not a fault. A pbsdr `degraded — binary not found` means the box predates
+  v1.15.0; the host page shows the retrofit one-liner (wrapper + `age` + `wg_tunnel.enabled`).
 - The controller version shown for the guest is the golden's baked (current) version — it matches
  what D.1b's check reported.