felhom.eu/documentation/runbooks/day0-install.md

# Runbook — Day-0 install: first-time deployment of a Felhom host

> **The complete, step-by-step first-time deployment guide** for putting Felhom on a customer's
> Proxmox box — hub-side onboarding, box prerequisites, the install run, post-install verification,
> first app deploy, and rollback/uninstall. Every command in this guide was executed verbatim during
> the clean-room drill (see the header line below); placeholders are marked `<LIKE-THIS>`.
>
> **Drill-validated: 2026-07-03** on a clean-room nested PVE 9.2.2 VM (qemu/KVM on the build
> server) — install → verify → app deploy → uninstall → residue-diff → reinstall, all executed
> verbatim. Versions: script **v1.9.1**, agent **v0.63.0**, golden **v0.98.3** (controller current
> on first boot — no manual update step). Evidence:
> `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` (install/uninstall path) +
> `documentation/audits/DRILL-golden-098-2026-07-03.md` (current-golden rebuild).
>
> Scope: a **shared** Proxmox box (the colleague's-box model) — a box that already runs, or will run,
> non-Felhom guests. A dedicated box is the same procedure minus the sizing caution.
>
> **v1.10.0 (GL-2): every install now REQUIRES `--mode appliance|byo`** — there is no default. Use
> `appliance` for a box Felhom owns/manages end-to-end (this guide's drilled path, unchanged apart
> from the flag); use `byo` for a host the operator does NOT own (the pilot's own PVE) — see §C.5.
> The byo profile is implemented + statically validated but **not yet drill-validated** (that is the
> supervised GL-6 run).

## Who does what

| Actor | Where | Does |
|---|---|---|
| **Operator** | HQ (hub UI + Cloudflare) | Part A (customer onboarding), go/no-go, on call during install |
| **Operator (or trusted colleague with root SSH)** | the customer's Proxmox box | Parts B–E |
| **Customer** | — | confirms the Part B prerequisites, receives the dashboard URL |

One secret crosses to the box: the customer's **retrieval passphrase** (5 Hungarian words, generated
by the hub in Part A). Everything else the box needs, it fetches and verifies itself.

---

## Part A — hub-side customer onboarding (operator, at HQ)

### A.1 Create the Cloudflare tunnel (per customer domain)

The customer's apps are published through a Cloudflare Tunnel; the in-guest controller runs
`cloudflared` with a per-customer token. In the Cloudflare dashboard (Zero Trust → Networks →
Tunnels):

1. Create a tunnel named after the customer (e.g. `<customer-id>`). Connector type: Cloudflared.
2. Copy the **tunnel token** (the long base64 string from the `cloudflared service install <token>`
   command) — this goes into the hub customer form in A.2.
3. In the customer domain's DNS zone, the controller manages per-app hostnames itself via the
   tunnel; make sure the domain is on Cloudflare (nameservers active).

Also have ready (optional but recommended): a **Cloudflare API token** with Zone edit rights for the
customer's zone — the hub uses it for geo-restriction management.

### A.2 Create the customer in the hub

Hub UI (`https://hub.felhom.eu`, operator password) → **Customers → New**:

| Field | Value | Notes |
|---|---|---|
| Customer ID | e.g. `sajatfelhom` | letters, numbers, dots, hyphens only; **immutable** — it keys everything |
| Customer name | display name | |
| Domain | e.g. `sajatfelhom.hu` | the customer's app domain (on Cloudflare) |
| Email | customer's email | used for customer-tier notifications (Hungarian) |
| CF tunnel token | from A.1 | → `infrastructure.cf_tunnel_token` |
| CF API token | from A.1 (optional) | → `infrastructure.cf_api_token` (geo rules) |
| Git username | Gitea read account | → `git.username` — **optional** (only for a private app catalog) |
| Git token | Gitea read token | → `git.token` — **optional** since installer v1.11.2 (the G3 ruling): artifacts are world-readable and fetched ANONYMOUSLY, sha256-verified against the hub manifest. Empty is the normal customer shape; expect the installer's "fetching artifacts ANONYMOUSLY" warn line in steps 5/8 (drill-verified 2026-07-12 on the VM Day-0 drill). |

On save the hub generates two credentials:

- **Retrieval passphrase** (5 Hungarian words) — the ONE secret the installer carries to the box.
  Dictate or hand it to whoever runs Part C. Treat it like a password.
- **Customer API key** — internal (baked into the generated controller.yaml); never handled manually.

### A.3 Verify the Day-0 artifact manifest

Hub UI → **Configuration → Day-0 artifacts**. The manifest must vouch an **agent version** and a
**golden version** (with their sha256s — auto-resolved from Gitea). The install script refuses to
install anything the manifest doesn't vouch. If the vouched versions lag the fleet, bump them here
first (dropdowns; the sha is fetched authoritatively). Current drilled-known-good: agent
**0.63.0** + golden **0.98.3**. Vouch a golden ≥ 0.98.3 — an older golden bakes a pre-floor
controller and re-introduces the manual update step (Part F troubleshooting).

### A.4 Check the global controller-version floor

Hub UI → **Configuration**. Know the current `min_controller_version` floor. With golden ≥ 0.98.3
vouched (the structural fix for the old OQ-2 finding — drill-proven 2026-07-03), a fresh box lands
on a **current, floor-honoring** controller and self-manages from first boot; the floor's job is
keeping it current afterwards. Recommended (1 minute, records drift-protection): raise the global
floor to the current controller version whenever the golden is rebuilt.

---

## Part B — box prerequisites (confirm BEFORE install day)

Collect these from the customer / the box ahead of time. Every line is checkable in two minutes; a
"no" on any of them turns install day into debugging day.

```text
[ ] Proxmox VE 9.x installed and reachable over SSH as root
    pveversion                      # want pve-manager/9.x
[ ] Single node or cluster?
    pvesh get /nodes --output-format json | python3 -c 'import json,sys; ns=json.load(sys.stdin); print(len(ns), [n["node"] for n in ns])'
    → more than one node: the install command MUST carry --node <name>
[ ] local-lvm free space ≥ 120 GiB
    lvs /dev/pve/data -o lv_size,data_percent
[ ] Free vmid chosen from the FULL guest list (never from the hub view)
    pct list; qm list               # pick a vmid not in either list; avoid 9999 and 990000-990009
[ ] Host cores + RAM known (to size the appliance cap)
    nproc; free -m
    → shared box: cap the guest so the other guests keep breathing (Part C uses --cores/--memory)
[ ] Outbound HTTPS reachability from the box:
    curl -fsSo /dev/null -w '%{http_code}\n' https://hub.felhom.eu            # want 302 (login redirect)
    curl -fsSo /dev/null -w '%{http_code}\n' https://gitea.dooplex.hu         # want 200
    curl -fsSo /dev/null -w '%{http_code}\n' https://felhom.eu                # want 200
[ ] python3 present (PVE 9 default: yes)
    command -v python3
[ ] The operator has the retrieval passphrase from Part A at hand
```

Notes:
- The box needs **no inbound** reachability — hub, Gitea and the Cloudflare tunnel are all
  outbound-only.
- vmid picking: the agent's token is pool-scoped, so the hub's guest inventory only shows
  Felhom-pool guests — other guests on a shared box are invisible there. Always pick from
  `pct list` + `qm list` on the box (`documentation/runbooks/provisioning.md`).
- Offsite backup (`felhom-pbs`) is NOT part of a go-live today: the PBS host is LAN-only. The
  transport is DECIDED and DESIGNED — plain WireGuard, host-side agent-managed, box-dials-out to
  a public endpoint VM running the offsite PBS. Evidence:
  `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`; design-of-record + slice
  roadmap: `documentation/architecture/06-offsite-connectivity.md` (MVP = slices S1→S2→S3, then
  S4). Go-live standard = **local backups only** (that is what the Part C command configures);
  offsite DR is retrofitted later via `--rescope-acl` / `--acl-storages` without reinstalling.

---

## Part C — the install run (on the box, as root)

### C.1 Fetch the installer

```bash
curl -fsSO https://felhom.eu/scripts/felhom-host-install.sh
chmod +x felhom-host-install.sh
./felhom-host-install.sh -h | head -3      # sanity: must print v1.10.0 (or newer) — older scripts don't know --mode
```

### C.2 Preview (recommended)

`--dry-run` prints every mutating command without executing anything (it still contacts the hub
read-only and asks for the passphrase):

```bash
./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode appliance --vmid <VMID> \
    --cores 2 --memory 4096 \
    --force-gitea-golden --acl-storages "local local-lvm" \
    --dry-run
```

### C.3 The canonical shared-box install command

```bash
./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode appliance --vmid <VMID> \
    --cores 2 --memory 4096 \
    --force-gitea-golden --acl-storages "local local-lvm"
```

Placeholders and flags — what and why:

| Flag | Value | Why |
|---|---|---|
| `--customer-id` | the Part A customer ID | keys every hub call |
| `--mode` | `appliance` or `byo` | REQUIRED, no default. `appliance` = a Felhom-owned/managed box (this section). `byo` = a host the operator does not own — use §C.5's command instead |
| `--vmid` | the free vmid from Part B | ALWAYS pass it explicitly — deterministic, and recorded in the install state for a later `--uninstall` |
| `--cores 2 --memory 4096` | the appliance cap | protects the other guests on a shared box; size to roughly half the host if the box is small (cap ≤ host resources) |
| `--force-gitea-golden` | — | a fresh box has no local golden archive; fetch it from Gitea and verify its sha256 against the hub manifest (this is the normal customer path) |
| `--acl-storages "local local-lvm"` | — | grants the agent's token write access on exactly these storages. The default set includes `felhom-pbs` (offsite), which does not exist on a customer box — granting it is harmless but misleading; scope to what is real. |
| `--node <NAME>` | only on a **cluster** | the script refuses to auto-pick a node on a multi-node cluster |

When prompted, type the **retrieval passphrase** (no echo; never on the command line). Alternative
for remote/scripted runs: put it in a 0600 file and pass `--passphrase-file <path>`.

### C.4 What the script does (so you can follow the output)

Eight steps, all idempotent/resumable (`--resume` skips completed steps after a mid-run failure):

1. **pre-flight** — root/PVE/hub checks, cluster refusal without `--node`, LVM/RAM floors,
   passphrase validated read-only against the hub, vmid guard.
2. **Proxmox API token** — `felhom-agent@pve!agent` minted (privsep), the `felhom` pool created, and
   the pool-scoped 3-role ACL applied (blast-radius containment: the token can only touch
   Felhom-pool guests + the granted storages).
3. **volume grows** — computed from free space (Docker-data/user-data sizing).
4. **host enrollment** — `POST /host-enroll` (passphrase-authed, idempotent) mints/returns the
   per-host hub credential.
5. **agent install** — binary fetched from Gitea, **sha256-verified against the hub manifest**,
   installed with the non-root service user, sudoers (visudo-validated), guarded-mkfs wrapper,
   systemd unit.
6. **agent config + service** — config written 0600, read-only selftest must pass, daemon started.
7. **golden archive** — fetched from Gitea + sha256-verified (because `--force-gitea-golden`).
8. **provision** — the agent restores the golden into vmid `<VMID>` (into the `felhom` pool),
   grows volumes, renders `bootstrap.json`, sets onboot, starts the guest. Then a verify pass.

Expected duration: minutes; dominated by the two Gitea downloads (agent ~20 MB, golden ~1-2 GB) on
the customer's uplink.

If a step fails: read the error (they are specific), fix the cause, re-run the SAME command with
`--resume` appended (with the SAME `--mode` — the script refuses a mode flip on a half-done install).

### C.5 BYO hosts (`--mode byo`) — installing on a Proxmox server the operator does NOT own

**Trust model.** On a BYO host the roles invert: the box owner is root and stays root; Felhom is the
guest. The installer therefore (a) never touches the owner's credentials — the break-glass step
(4b/8, root@pam reset + hub vault) is skipped entirely, nothing is vaulted; (b) treats the owner's
workloads as the thing to protect — the CPU/RAM caps are mandatory, they are the only
noisy-neighbor containment there; (c) refuses anything host-invasive that the appliance profile
allows (`--enable-oob`, `--rotate-recovery`, unvalidated PVE majors, ACL grants on storages the box
doesn't have); and (d) asserts the written agent config keeps `lan_resolver`/`wg_tunnel`/`oob` OFF
— the agent must not take over the owner's DNS or open tunnels. Recovery on a BYO box = the owner's
own console access, by design.

**Recommended first run — preflight only** (all step-1 checks incl. the byo gates, an explicit
PASS/FAIL verdict, exit 0/1, writes nothing — re-runnable any number of times):

```bash
./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode byo --vmid <VMID> \
    --cores <N> --memory <MiB> \
    --force-gitea-golden --acl-storages "local local-lvm" \
    --preflight-only
```

**The install command** (same flags, minus `--preflight-only`):

```bash
./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode byo --vmid <VMID> \
    --cores <N> --memory <MiB> \
    --force-gitea-golden --acl-storages "local local-lvm"
```

What byo does differently (everything else matches C.4's eight steps):

- `--cores` AND `--memory` are **required** — size them WITH the owner (the guest must never starve
  the owner's guests). `--acl-storages` must name storages that actually exist on the box, or the
  preflight dies naming the missing ones.
- **The host's `:53` must be free** (v1.12.0/F6): Felhom publishes the guest by name on the LAN, so a
  resolver already bound to `:53` blocks it. byo preflight **REFUSES** (does not touch the owner's
  service — it is a refusal, not a change) with: *"a resolver is already bound to :53 … stop or
  reconfigure it, or point your LAN DNS at the guest's address, then re-run."* Clear `:53` (or point
  the owner's DNS at the guest) and re-run. (Appliance mode is different — there Felhom's own
  lan_resolver owns `:53`.)
- **A fresh install on a populated BYO host Just Works** (v1.12.0/F3): the leaf-regeneration guard
  now fires only when a **Felhom** guest is already present — the owner's own non-Felhom guests no
  longer trip it, so **`--allow-new-leaf` is NOT needed** on a first install. (You still need
  `--preserve-state-from` / `--allow-new-leaf` on a reinstall over an existing *Felhom* guest, to
  protect its pinned fingerprint — unchanged.)
- Preflight also prints the existing-guest count.
- At the end of preflight the script prints the **complete host-mutation list** (pveum
  roles/user/token/ACL/pool, agent user/binary/unit/sudoers/config/state, the watchdog + guarded
  wrappers, the guest + golden) and requires typing the host's **short hostname** to proceed —
  show this list to the box owner; it is the consent artifact.
- Step 4b/8 (break-glass) is skipped; the verify step asserts pool membership + the scoped ACL
  grants landed (both modes do this from v1.10.0).

### C.5b Operator key pin — self-update authority (v1.11.0)

The agent's self-update only acts on **operator-signed** update ops; the verifying public keys are
pinned in the agent config (`authz.signers`). A fresh install with **no key pinned runs self-update
DORMANT** (the safe default — the verify step prints a WARN, not an error). To arm it at install
time, either:

- fill the `OPERATOR_KEY_*` constants near the top of the script (one commit, after the offline
  key-generation ceremony), or
- pass `--operator-pubkey-file <path>` — one key per line, `operational <authorized_keys line>` or
  `recovery <authorized_keys line>`; the authorized_keys **comment field is the key_id and is
  required**. The file overrides the constants.

Rules: PUBLIC keys only (private keys never touch the box or this script); a reinstall with no
keys resolved **keeps** an already-pinned config's signers (`--preserve-from`) — pinned boxes are
never silently un-pinned; providing keys over a preserved config replaces them (pin rotation,
logged). byo installs show the operational key_id in the disclosure block.

### C.6 Post-hoc mode note for pre-v1.10.0 installs

Boxes installed by ≤ v1.9.1 have no recorded mode; their state file simply predates it. `--resume`
on such a box accepts whichever `--mode` you pass — pass the mode that matches how the box is
actually operated (existing Felhom-owned boxes: `appliance`).

---

## Part D — post-install verification (on the box + hub)

Run through ALL of these before calling it done.

### D.1 On the box

```bash
# 1. The script's own final verdict said: "Day-0 provision SUCCESS — vmid=… host_id=…"

# 2. Agent service up, running non-root
systemctl is-active felhom-agent            # active
systemctl show felhom-agent -p User --value # felhom-agent

# 3. Agent selftest — all capabilities, including the pool read (stale-lock reaper)
felhom-agent --config /etc/felhom-agent/agent.json --selftest 2>&1 | tail -25
#    want: no FAILs; the "pve:pool-read" line ok (agent >= v0.62.0)

# 4. Guest up + onboot
pct status <VMID>                           # status: running
pct config <VMID> | grep -E '^(onboot|rootfs|mp0|mp1)'

# 5. Controller container healthy in-guest
pct exec <VMID> -- docker ps --filter name=felhom-controller --format '{{.Status}} {{.Image}}'

# 6. Dashboard answers in-guest (the controller sits BEHIND traefik — probe with the Host header;
#    the controller's own :8080 is internal to the Docker network and not directly reachable)
pct exec <VMID> -- curl -ksSo /dev/null -w '%{http_code}\n' \
    -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/          # want 200
```

### D.1b Controller version check (one line — the old manual update step is RETIRED)

The golden (≥ 0.98.3) bakes a **current, floor-honoring** controller — a fresh install is current on
first boot and the hub floor manages it from here. Just verify:

```bash
# controller version current on first boot; the floor manages it from here (want: update_available false)
pct exec <VMID> -- curl -ksS -X POST -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/api/selfupdate/check
```

If this reports the baked version as BEHIND (a newer controller shipped after the golden was baked),
the same endpoint's `/api/selfupdate/update` — or the dashboard's Beállítások → "Frissítés" button —
converges it; that is the normal self-update path, not an install-day requirement. (Installed from a
golden older than 0.86.0? See Part F troubleshooting.)

### D.2 In the hub UI

- **Hosts**: the new host reports (status OK, fresh last-report timestamp).
- The host's guest list shows `<VMID>` running.
- **Capabilities**: 0 degraded. On a reduced-storage box (no `felhom-pbs`) the offsite-backup
  related capability is expected degraded — that is the documented go-live standard, not a fault.
- The controller version shown for the guest is the golden's baked (current) version — it matches
  what D.1b's check reported.

### D.3 Customer-visible

- `https://felhom.<customer-domain>` loads the dashboard through the tunnel (cloudflared up
  in-guest: `pct exec <VMID> -- docker ps --filter name=cloudflared`).

### D.4 First app deploy (smoke)

In the dashboard, deploy one small catalog app end-to-end (pick something dependency-free). The app
reaches "running" and its URL answers. This proves the full customer-visible chain: catalog →
controller → Docker → tunnel → browser.

---

## Part E — rollback / uninstall

`--uninstall` cleanly reverts an install — no hub contact, no passphrase needed:

```bash
./felhom-host-install.sh --uninstall --vmid <VMID>
```

Guardrails (all verified in the drill):

- **Ours-check**: refuses to destroy a guest without the `/etc/felhom-bootstrap` mount (i.e. not
  Felhom-provisioned) unless `--force`.
- **Typed confirmation**: you must type the vmid back before anything is destroyed.
- **Shared-box safety**: if OTHER Felhom guests remain on the host, only the named guest is
  removed; the agent/token/ACL stay (guest-only mode). `--force` overrides.
- The `felhom` pool is deleted only if empty.
- `--vmid` is optional when the install state file recorded the provisioned vmid (a fresh
  v1.9.x install records it); passing it explicitly is still the recommended, deterministic form.

What it removes (v1.9.1, residue-diff-proven — a full-filesystem diff against the pre-install
baseline showed **zero Felhom-named leftovers**): the guest, the agent (unit/sudoers/binary/
state dir/config/service user), the shared-parent unit + wrapper + `/mnt/felhom-drives`, the
guarded-mkfs wrapper, the guest-hook snippet, the lan-resolver dnsmasq snippets, the pveum
roles/ACL/token/user, the pool (if empty), the install state file.

v1.11.0 additions: the teardown also removes the **self-update artifacts** (guarded wrapper, A/B
slots, rollback unit, limits drop-in), **unmounts enrolled drives** under `/mnt/felhom-drives/`
(plain umount only — a busy drive gets a warning and guidance, never a forced unmount; the data is
never touched and the drives are physically removable afterwards), and both modes end with an
explicit **KEPT-vs-WIPED statement** — read it before pulling drives or closing the customer out
(it lists what lives on: drive data, PBS backups, the hub record, the escrow blob, the vaulted
recovery credential).

**Expected remnants** (documented, not residue):

- The **hub host record** (+ its report/guest history) — the hub currently has **no host-delete**;
  removing it is a direct-DB operator action (recorded follow-up).
- The `sudo` and `dnsmasq` packages (system packages installed as dependencies; inert).
- The **golden vzdump** on the archive storage — pass `--remove-golden` to delete it too.
- System noise: LVM archive metadata, dpkg lists, PVE's own rrd/`/var/lib/lxc` scratch entries.

---

## Part F — troubleshooting (from the drill + live history)

| Symptom | Cause | Fix |
|---|---|---|
| dies immediately: "--mode is required" | v1.10.0+ has no default profile | add `--mode appliance` (Felhom-owned box) or `--mode byo` (owner's box, §C.5) |
| dies immediately: "byo mode requires explicit --cores and --memory" | byo caps are mandatory | size the caps with the box owner and pass both |
| dies: "install started as X; resume with --mode X" | `--resume` with the other mode | resume with the recorded mode, or `--uninstall` and start over |
| byo dies: "acl storage(s) not found on this box" | `--acl-storages` names a storage the box lacks | pass the box's real storages (check `pvesm status`) |
| step 1 dies: "this is a N-node cluster" | multi-node cluster | re-run with `--node <name>` |
| step 5 dies: "hub artifact manifest has no agent version" | Day-0 manifest unset/incomplete | Part A.3 — set it in the operator UI |
| step 5 warns: "no git credential — fetching artifacts ANONYMOUSLY" | customer has no git credentials — the NORMAL shape since installer v1.11.2 | nothing to do; sha256 verification is unchanged. (Pre-v1.11.2 installers die here instead — upgrade the script.) |
| step 1: passphrase REJECTED (401) | typo / wrong customer | re-check with the hub UI's printed curl command |
| step 8 fails: "CT <vmid> already exists" | vmid collision with a hub-invisible guest | pick from `pct list` + `qm list` (Part B); the agent destroys nothing on collision — re-run with a free vmid and `--resume` |
| controller container missing in-guest after provision (docker ps empty) | golden older than 0.98.3 (no baked path unit) + pre-v1.9.1 script: the controller-bootstrap unit's boot-time condition lost the race with the bootstrap-mount attach | `pct reboot <VMID>` — the unit runs on the next boot (v1.9.1 reboots itself; goldens ≥ 0.98.3 bake a path unit that starts the controller on the mount hot-plug, no reboot needed) |
| fresh box runs an OLD controller and never self-updates (hub floor ignored) | a golden **older than 0.86.0** was vouched in the Day-0 manifest — its baked controller predates the floor-honoring code | one-time manual update (the retired old D.1b): dashboard Beállítások → "Frissítés", or `POST /api/selfupdate/check` + `/api/selfupdate/update` via the traefik Host-header probe; if the agent is < v0.63.0, `systemctl restart felhom-agent` FIRST (pre-v0.63.0 agents 401 the swap call). Structural fix: re-vouch a golden ≥ 0.98.3 (A.3) |
| self-update fails: "Agent swap request failed: HTTP 401" (or dashboard agent calls 401) | agent < v0.63.0: the daemon's token index missed a token minted by the provisioner after it started (FIXED in agent v0.63.0 — reload-on-miss) | `systemctl restart felhom-agent`, then retry (unnecessary from agent v0.63.0) |
| `local-lvm free < 120 GiB` warning | small/shared box | proceed only if you sized the grows deliberately; otherwise free space first |
| cloudflared restarting in-guest | bad/missing tunnel token | re-check A.1/A.2 token, regenerate config, `docker restart cloudflared` in-guest |

(Extended during the drill with every real hiccup encountered.)

---

## Related

- vmid picking + reserved bands: `documentation/runbooks/provisioning.md`
- The Day-0 trust chain (why the box can verify what it fetches):
  `documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`
- Pool-scoped ACL model: `documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md`
- Secrets handling: `documentation/runbooks/secrets.md`