felhom.eu/documentation/architecture/07-backup-architecture.md

# 07 — The recovery model

> | | |
> |---|---|
> | **Status** | **NOT RATIFIED.** Ratification is Viktor's review, not an editor's. |
> | **Written** | 2026-07-28 (full rewrite; supersedes the 2026-07-14 DRAFT entirely) |
> | **Verified against** | controller **v0.183.0** · agent **v0.110.0** · hub **v0.80.0** · catalog `4252121` · repo HEADs `felhom.eu ff050cf`, `felhom-controller fd50a73`, `felhom-agent d5c7691` |
> | **Live fleet at verification** | demo-felhom + demo-hp, both guest 9201, both on the versions above |
> | **Freshness** | **CURRENT** as of 2026-07-28. Per standing ruling **S-2**, mark this STALE the moment it is more than a few trains behind. The document it replaces was DRAFT since 2026-07-14, verified against controller v0.132.0 — **fifty-one versions stale** — and was cited as authoritative throughout that time. |
>
> **How to read this document.** Two kinds of statement appear, and they are always labelled:
>
> - **[DESIGN]** — a decision taken in the 2026-07-28 architecture discussion (D1–D6, §3–§5, §9).
>   Not derived from code; the code may not implement it yet. Where it does not, §10 says so.
> - **[FACT]** — an observed property, carrying a `file:line`, a live command output, or a citation
>   to `_recovery-inventory-2026-07-28.md` (below: **INV**).
>
> Where the model is silent, this document says **OPEN** rather than filling the gap.
>
> **Primary input:** `_recovery-inventory-2026-07-28.md` (read-only inventory, 2026-07-28) — cited
> throughout as **INV Part n**. Every number in §6, §8 and §11 traces back to it.

---

## 1. Purpose and scope

This document describes **how a Felhom customer gets their system back**, and who can do it.

It replaces a document that described **where copies are written**. That was the wrong frame, and
§7 explains why: a set of copies is not a set of recovery routes, and the previous document's tier
table could be entirely satisfied while a real recovery was impossible.

**In scope:** the recovery model — trust boundary, lanes, the three-part model, the tiers as
*inputs to recovery*, the dependency chain between them, the failure→recovery matrix, and the
encryption policy.

**Out of scope, deliberately:** implementation specs (they live in task specs), the capture-set
algorithm (`internal/appbackup/captureset.go` and its tests are the source of truth), and per-tier
operational runbooks (`documentation/runbooks/`).

**Authority split.** This document is authoritative for the **failure→recovery matrix** (§8). The
capability map (`00-capability-map.md`) stays authoritative for **per-capability status**. Neither
restates the other; §8 rows are cited from the map, not copied into it.

---

## 2. The trust model

**[DESIGN] The operator holds root SSH on every box.** That is a fact of the product — the agent is
operator-tier, the host is operator-managed, and break-glass exists precisely so the operator can
get in when nothing else works. **"The operator cannot read customer data" was never the security
property**, and no part of this document may be read as claiming it.

**[FACT]** The mechanics that make this concrete:

- The operator's OOB SSH public key is pushed to every box from the hub
  (`hub_settings.oob_operator_ssh_pubkey` → `/var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op`,
  93 bytes, LIVE on both hosts — INV Part C, row 21).
- The break-glass `root@pam` console password for every host is stored in the hub and retrievable
  with the operator's global key (`documentation/runbooks/break-glass.md:42-48`; LIVE:
  `host_recovery` holds 3 rows — INV Part D2.1).
- Guest data is reachable from the host by definition: `pct exec`, and the data drives are bind
  mounts on the host (`mp8 /mnt/felhom-drives`, LIVE `pct config 9201` on both hosts).

**[DESIGN] What R (the customer's recovery code) does provide, and must keep providing: the hub
alone is not enough.** A compromised hub yields blobs nobody can open. This property holds **only
while the operator's own key is not stored in the hub** — which is why §11-A is an open decision and
why its recommendation on record is "operator key held offline and never in the hub".

**[FACT]** The escrow is genuinely zero-knowledge today: `host_escrow` rows carry
`posture = zero_knowledge`, a 383-byte blob and a 572-byte identity blob, and the hub holds only a
`restic_pw_sha256` **hash** beside them (LIVE, both hosts — INV Part D2.1). R exists in **zero**
system copies by design (INV Part C, row 24).

So the honest statement of the property is:

> **The hub cannot open the escrow. The operator can reach a live box. Neither fact substitutes for
> the other, and R is what keeps the first true.**

---

## 3. The two lanes (D1)

**[DESIGN] Recovery is split into two lanes with different owners. This is a product decision, not a
limitation.**

### Lane 1 — the customer, unassisted: files and app data

The customer owns their data and can get it back themselves, through the „Visszaállítás" surfaces,
with nothing but their dashboard password. No operator, no ticket, no scheduling.

**[FACT] What Lane 1 contains today** (INV Part A.1 — seven paths, all behind the controller's
`RequireAuth` gate, which is the **customer-owned** password: `internal/web/auth.go:34-35` puts
`settings.json → password_hash` ahead of the operator-provisioned `controller.yaml` value):

| # | Surface (HU) | Endpoint | Semantics |
|---|---|---|---|
| 1 | „Visszaállítás indítása" | `POST /backup/restore` | destructive — rebuilds the app from its Tier-1 unit |
| 2 | „Fájlok visszaállítása" | `POST /backup/tier2/restore` | additive, missing-only |
| 3 | „Visszaállítás a távoli tárolóból" | `POST /backup/offbox/restore` | non-destructive, to a verification copy |
| 4 | „Helyreállítás az élő adatok közé" | `POST /backup/offbox/place` | additive, missing-only |
| 5 | „Teljes visszaállítás (fájlok + adatbázis)" | `POST /backup/offbox/reconstitute` | destructive to files + DB, never deleting |
| 6 | „Megosztások visszaállítása" + place | `POST /backup/shares/{restore,place}` | additive |
| 7 | `.fab` import | `POST` → `apiImportStart` | destructive re-import of one app |

**[FACT] What is proven in Lane 1** (INV Part G.1): paths 2 and 6 are proven live end-to-end;
path 5 is proven live through a destructive drill (photos deleted, trash emptied, restored, timeline
confirmed); path 3 is proven for **bytes** only; path 7 is proven for the drive-to-drive circle but
its browser-upload leg is not; path 1 is proven to **execute** but its **content recovery after real
loss has never been demonstrated** — the single most valuable unproven item in the system
(`audits/CAMPAIGN-9-restore-proof-2026-07-28.md:825-827`); path 4 has never been exercised as a
distinct action (`audits/CAMPAIGN-8-backup-restore-2026-07-27.md:520`).

**[FACT] One honest residual on the whole lane:** every proven run was performed by the **operator**,
not by a customer. `00-capability-map.md:75` still carries "A customer (not the operator) performs a
restore via UI alone" as **MISSING as evidence** — by absence of the run, not by a product gap.

### Lane 2 — the operator: guest and host recovery

Rebuilding an LXC guest, rebuilding a host, and re-establishing a box's identity are **operator
work, by design and by contract**. They are not customer-facing and are not going to be.

**[FACT] What Lane 2 contains** (INV Part A.2): the scheduled restore-test, `pct restore`, raw
`proxmox-backup-client restore`, raw `restic restore`, the agent's DR bring-up
(`--selftest=bring-up -mode dr`), the host-loss plan builder, and the escrow-consume ceremony. Each
needs root on the host or a CLI flag; none is reachable from any customer surface.

**[FACT] What is proven in Lane 2** (INV Parts G.1, G.3): whole-guest `pct restore` from both tiers
is proven with exact mount parity; the unattended restore-test is proven and currently running on
both boxes; a corrupted snapshot is proven to fail cleanly. **The DR bring-up path has never been
executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by construction**
(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
(INV Part D1).

### Why the split is right, stated once

**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
"restore the LXC and re-attach mp8 by durable_id without binding another guest's bootstrap
credentials" — a hazard serious enough to have its own runbook page
(`runbooks/RUNBOOK-manual-guest-restore.md:45-48`, F-OPS). Putting Lane 2 behind the operator is
what lets Lane 1 be a button instead of a procedure.

---

## 4. The three-part model (D4)

**[DESIGN] Recovery needs three things, and they live in three different places.** Losing one is a
different problem from losing another, and the matrix in §8 is organised around that.

| Part | Holds | Where | Lost when |
|---|---|---|---|
| **Recipe** | scaffolding — guest sizing, drive inventory, PVE storage definitions, PBS coordinates, app inventory + storage bindings. **No secrets, no bytes.** | the hub | the hub is lost |
| **Escrow** | the host identity key material and the restic repo password | the hub, R-wrapped | the hub is lost, **or** R is lost |
| **Bytes** | Tier-1 / Tier-2 / Tier-3 / whole-guest | the tiers | the relevant medium is lost |

**[FACT] The Recipe exists and is current.** `dr_recipe` holds 6 rows; demo-felhom's was updated
`2026-07-28 17:31:13` and demo-hp's `17:25:31` — i.e. within one report cycle (LIVE, INV Part D2).
It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_bindings`.

**[FACT] The Escrow exists and is zero-knowledge.** `host_escrow`: 2 rows, `posture` `zero_knowledge`;
the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}`
(`felhom-agent/internal/escrow/identity.go:26-39`).

**[FACT] Three parts of the Recipe are empty or wrong on the live fleet**, and they are exactly the
parts a host-loss recovery would read (INV Part D2.3): `hosts.dr_record_json` is `{}` on all three
hosts; `host_escrow.directive_json` is `{}` on both escrowed hosts; `dr_recipe.host_half.drives` is
`[]` on every customer including two with enrolled data drives; and `dr_recipe.host_half.pbs.namespace`
reads `"root"` while the real namespaces are `demo-felhom` / `demo-hp`. → **R-105**, **R-106**.

---

## 5. Encryption policy (D2)

**[DESIGN] Encryption follows the boundary, not the tier.**

### On the customer's own premises: plaintext

Tier-1 recovery units, Tier-2 cross-drive copies, the app data on the drives, and the local vzdump
archives on the host are **unencrypted, deliberately**.

The reasoning, recorded so nobody "hardens" this later:

1. **It buys nothing against a real threat.** Someone who can take the second drive can take the
   first. Local encryption defends against a threat model — physical theft of *only* the backup
   medium — that does not describe a home server where both media sit in the same box.
2. **It adds a key-loss path that turns a working backup into a brick.** Every local encryption key
   is one more thing that must survive the disaster it exists for, and the system already has one
   such dependency it is trying to reduce (§7).
3. **It would break browsing, which is a feature.** FileBrowser and SMB let the household see and
   use their own files. An encrypted local copy is not browsable, and the „Megosztás" and
   FileBrowser surfaces are part of the product, not an accident.

**[FACT]** The local plaintext posture is real and verifiable: `dir: local` in `/etc/pve/storage.cfg`
carries no `encryption-key` (LIVE, both hosts), so the daily whole-guest archive is a plain
`.tar.zst` — 5.93 GB on demo-felhom, 1.68 GB on demo-hp (LIVE, INV Part B.3). **That archive is the
single highest-value object on the box**: it contains `encryption.key`, the restic `repo_password`,
the offbox `ssh_key`, `settings.json` and `controller.yaml` (all under `/var/lib/docker` = `mp0`,
`backup=1`). Naming that plainly is part of the policy, not an argument against it.

### Leaving the premises: encrypted, and the provider must not be able to read it

**[FACT]** Both offsite tiers encrypt client-side:

- **restic (Tier-3):** repo password is a 256-bit hex value generated once on the box and never
  logged (`internal/backup/offbox.go:392-410`); restic is invoked with
  `RESTIC_PASSWORD_FILE` (`:608`).
- **PBS (whole-guest offsite):** `/etc/pve/storage.cfg` carries a **per-customer** `encryption-key`
  and the fingerprints differ between demo-felhom and demo-hp (LIVE). The live restore command line
  observed on demo-felhom carries `--crypt-mode=encrypt` (INV Part A.2.3). Per-tenant encryption is
  why cross-customer dedup is impossible on the shared datastore — a cost consequence, recorded, not
  a defect.

### The one exception, stated because it is not covered by either rule

**[FACT]** The `.fab` bundle writes `app.yaml` with **decrypted plaintext secrets, deliberately**,
and its password is **optional** (`internal/appexport/export.go:484,506-511`; the generated file's
first line is literally `# Exported by felhom-controller — plaintext secrets`, `:531`;
`Encrypted: req.Password != ""`, `:307`). A `.fab` can also be written to a registered **network**
share, because `storageDriveList()` does not filter network paths
(`internal/web/handler_export.go:377-387`). This is a portability artifact, not a tier — but it is
the one place where a customer action can put every secret of one app onto a NAS in plaintext.
Recorded here so the encryption policy is not read as covering it. → **R-108** (same root cause).

---

## 6. The tiers — what each captures

The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what they can actually do.

### 6.1 The four tiers, as configured on the live fleet

| Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted |
|---|---|---|---|---|---|
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` | **yes** (per-customer key) |

**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
(`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`.

**[FACT] What the whole-guest tiers do NOT carry.** `mp8 /mnt/felhom-drives` and
`mp9 /etc/felhom-bootstrap` are **host bind mounts** and are out of `vzdump` scope entirely (LIVE
`pct config 9201`, both hosts). So the whole-guest tiers carry the guest and **none of the customer's
data drives** — 916 GB on demo-felhom, 938 GB on demo-hp.

### 6.2 Coverage per app class — and an unresolved count

**[FACT]** Of 53 catalog templates, **52 keep data in Docker named volumes**; exactly **13** carry a
`backup:` block, and those 13 are exactly the templates that bind `${HDD_PATH}` / `${USERDATA_PATH}` /
`${IMPORT_PATH}` at all; **14** have a database service (INV Part B.1, computed against catalog
`4252121`).

Applying the classifier's documented two-level default (`internal/appbackup/classify.go`: explicit
entry wins; else a `:ro` reader is *excluded*; else a writable bind is *mandatory*):

| tier filter | templates with at least one file leg |
|---|---|
| **Tier-3** (`mandatory` only) | **4** — calibre-web, immich, nextcloud, paperless-ngx |
| **Tier-2** (`mandatory + optional`) | **7** — the 4 above plus audiobookshelf, komga, romm |
| legacy resolver path (apps with no block) | **0** — no non-block template binds a namespace path |

> ### ⚠️ UNRESOLVED — two counts of the same thing disagree
>
> | source | count |
> |---|---|
> | C9-F1 Phase 0, as shipped | **A = 9 / B = 43 / C = 1** — `felhom.eu/REPORT.md:17-24`, restated `backlog/OPEN-ITEMS.md:31` |
> | INV Part B.1, independent enumeration at catalog `4252121` | **A = 7 / B = 45 / C = 1** |
>
> Both use the same definition ("templates whose Tier-2 copy can hold a readable file leg"). The
> difference is two apps and **neither number is adopted here**. The Phase-0 enumeration is described
> in prose but the script is not committed, so the two methods cannot be diffed from the repo.
> **This must be resolved before either figure is used to size anything.**

**[FACT] The class-B consequence is real regardless of which count is right.** For an app whose data
lives entirely in named volumes, the Tier-2 copy holds a full `recovery-unit/` and **no readable
leg** — LIVE on demo-felhom, `backups/secondary/bookstack/` and `.../docmost/` contain
`recovery-unit` and nothing else, at 156 MB and 86 MB (INV Part B.2). Since v0.183.0 the button
refuses **before** stopping the app and names the action that works.

### 6.3 Where capture and restore disagree

**[FACT]** Three asymmetries, each source-cited:

| tier | captured | read back by that tier's restore | gap |
|---|---|---|---|
| Tier-1 | unit incl. volume tars + DB dumps | all of it | none |
| Tier-2 | unit mirror **+** file legs | `hdd/` and `userdata/` **only** (`tier2_restore.go:101-104`) | **the unit mirror is read by nothing** — `RecoveryUnitPath` resolves to `backups/primary/` (`appbackup/paths.go:46-48`) → **R-102** |
| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay; the unit is **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **no offsite action unpacks the named-volume tars it captures** → **R-107** |

**[FACT]** Tier-2's gap is the sharper one because of *when* it bites: Tier-2 exists for the case
where the primary drive is lost — and in exactly that case the primary unit is gone while this
mirror survives on the second drive, unreachable by any customer action.

---

## 7. The recovery chain (D3) — the reason this document exists

**[DESIGN] 3-2-1 describes copies. It does not describe recovery.**

Three copies on two media with one offsite is a statement about *bytes surviving*. It says nothing
about whether the bytes can be turned back into a working system, and a system can satisfy 3-2-1
completely while having no executable recovery route for a given failure. That is not a hypothetical
here — §8 has rows where it is the actual state.

### 7.1 The dependency graph

**[FACT]** Every app-tier restore depends on the guest, for one of two reasons:

```
                       ┌──────────────────────────────────────────┐
                       │  the LIVE GUEST                          │
                       │   · settings.json (tier-2 destination)   │
                       │   · encryption.key  (32 B)               │
                       │   · app.yaml  (ENC: under that key)      │
                       │   · the deployed app itself              │
                       └───────────┬──────────────────────────────┘
                                   │ required by
        ┌──────────────────────────┼──────────────────────────┬─────────────────────┐
        │                          │                          │                     │
   Tier-1 restore            Tier-2 restore            Tier-3 reconstitute    Tier-3 scratch
   (rebuild the app)         (fill in files)           (overwrite + replay)   (verification copy)
        │                          │                          │                     │
   needs SECRETS              needs the app             needs the app          needs only the
   from app.yaml              running + the             deployed + a DB        repo password
   (restore_unit.go           recorded dest             service identifiable   (also in the guest)
    :130-132)                 (tier2_restore.go
                               :114-116)
        │                          │                          │                     │
        └──────────────────────────┴──────────────────────────┴─────────────────────┘
                                   │
                       ┌───────────▼──────────────────────────────┐
                       │  WHOLE-GUEST TIER  (local vzdump / PBS)  │   ← Lane 2, operator only
                       └───────────┬──────────────────────────────┘
                                   │ required by
                       ┌───────────▼──────────────────────────────┐
                       │  THE HOST  (agent.json, bootstrap.json,  │   ← in NO backup at all
                       │   vmbr9, sudoers, /etc/pve/priv, WG)     │      (INV Part D1)
                       └──────────────────────────────────────────┘
```

**[FACT] The two legs of the dependency, precisely:**

1. **Secrets.** The recovery unit is secret-free by design — *"It NEVER writes a secret value"*
   (`internal/backup/recovery_unit.go:73`). `RestoreFromRecoveryUnit` recovers secrets **from the
   guest, never from the unit** (`restore_unit.go:130-132`), and the policy is stated outright at
   `:18-22`: *"Regenerate NOTHING… A missing DATA-ENCRYPTING key is FATAL: regenerating it would
   render the restored data unreadable, so we refuse and tell the operator to do a PBS whole-guest
   restore."* Those secrets are encrypted under `encryption.key`, a 32-byte file that exists **only
   inside the guest** (LIVE, both boxes — INV Part C, row 9).
2. **The living app.** Tier-2's restore reads the destination recorded in the guest's `settings.json`
   (`tier2_restore.go:114-116`), and Tier-3's reconstitution refuses outright when the app is not
   deployed — *„a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat"*
   (`offbox_reconstitute.go:198-201`).

**[FACT] So all three app tiers are conditioned on the whole-guest tier**, which is Lane 2. The
customer's own recovery lane rests on an operator-only tier, and today nothing tells the customer
that.

### 7.2 A tier whose prerequisites cannot be met in the failure it exists for

**[FACT]** Two instances, both current:

- **Tier-2 vs primary-drive loss.** Tier-2's stated purpose is surviving the loss of the primary
  drive. In that failure the primary recovery unit is gone; the surviving mirror on the second drive
  is `backups/secondary/<app>/recovery-unit/`, which **no code path reads** (§6.3). For the 45-or-43
  class-B apps the restore is a guaranteed no-op in exactly its designed scenario. → **R-102**
- **Tier-3 vs guest loss.** Tier-3 holds the volume tars and the DB dump. Reconstitution requires
  the app to be deployed and skips the unit; the tars are unpacked only by the Tier-1 path, which
  requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest.
  → **R-107**

### 7.3 What D5 would change — and why it is blocked

**[DESIGN, TARGET — BLOCKED]** The intended fix is to make app secrets travel with the **local**
recovery unit, so Tier-1 and Tier-2 restore work **without the guest and without R**. Offsite
already encrypts everything, so secrets travelling offsite would be covered by the escrowed repo
password. R would then be required for **offsite recovery and host identity only** — losing R would
cost the offsite route, not local recovery.

**This is not adopted.** §2 of the task that produced this document required the premise to be
established, not assumed: the backup tree must be unreachable from every browsing, download and
export surface. **It is not.** The verification and the exposure are in §10.1. Until that is closed
(**R-108**), D5 stays a target and §7.1's chain stands as the model.

---

## 8. The failure → recovery matrix

**This is the core artifact.** One row per failure. It is authoritative for recovery routes;
`00-capability-map.md` stays authoritative for per-capability status.

**How to read the numbers.**

- **RTO** — **only measured durations** from INV Part F appear here. A blank cell means *nobody has
  ever measured it*, and a blank is a finding, not an omission.
- **RPO** — **no RPO has ever been measured from an incident.** These cells carry the **configured
  cadence that bounds RPO**, read live from the box, labelled `(cadence)`. A blank means no cadence
  governs the row.
- **Status** — `PROVEN` (live, cited) · `PARTIAL` (some legs proven) · `IMPLEMENTED` (code + tests,
  never exercised) · `NONE` (no route exists).

| # | Failure | What survives | Recovery route | Invocable by | RTO (measured) | RPO (cadence) | Status | Evidence |
|---|---|---|---|---|---|---|---|---|
| 1 | **Customer deletes files** | everything else | „Fájlok visszaállítása" — Tier-2 additive merge | **customer** | **39 s** (6 files) | 24 h | **PROVEN** | CAMPAIGN-9 A1 — byte-identical returns, both non-destruction promises kept, bytes re-served by paperless's own API |
| 2 | **An app's data directory is destroyed** | the guest, the other tiers | same route | **customer** | **46 s** (43 files) | 24 h | **PROVEN** | CAMPAIGN-9 A3 — loss verified by a positive observable (doc download 200 → 404), then 16/16 documents usable |
| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) | 24 h | **PARTIAL** | CAMPAIGN-9 A2 proves the path executes and the app survives; content recovery after real loss **never demonstrated** (`CAMPAIGN-9…:825-827`) |
| 3b | *same, for a class-B app via Tier-2* | — | **no route** — Tier-2 never reads the unit mirror | — | | | **NONE** | §6.3; **R-102** |
| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB; **the volume tars in either copy are unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102**, **R-107** |
| 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go:359-393` |
| 6 | **Guest lost or corrupted** | the host, both whole-guest tiers, the data drives (they are host binds) | `pct restore` from `local:` or `felhom-pbs:` | **operator** (SSH) | **84–112 s** local · **1101 s** PBS (both = restore-test into a scratch guest, boot + verify + teardown) | 24 h local · 7 d offsite | **PROVEN** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C (exact mount parity, `unprivileged: 1` preserved); LIVE restore-tests on both boxes this session |
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) |
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) |
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D |
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B |
| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |

### 8.1 The blank cells, listed explicitly

Per the rule that a blank is a finding, here they are:

| row | blank | why |
|---|---|---|
| 3b | RTO, RPO | no route exists to time |
| 4 | RTO | no drive-loss recovery has ever been timed |
| 5 | RTO | never timed; the rebuild is a normal Tier-2 run |
| 6 | — | RTO present, but it is a **restore into a scratch guest on the same host**; a restore *to a different host* has never been timed |
| 8 | RTO | a host has never been rebuilt as itself (INV Part D1) |
| 9 | RTO | the composed whole-box path has never been run |
| 10 | RTO | no ransomware-shaped recovery has ever been run |
| 11 | RTO | a hub restore has never been performed |
| 12 | RTO, RPO | no provider-loss recovery has ever been run |
| 13 | RTO, RPO | not a timed recovery; a capability loss |
| 14 | RTO | the break-glass path is proven but was never timed |
| 15 | RTO | the manual unlock was performed but not timed |

**Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB
from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS
incremental; `.fab` import wall-clock; and time-to-first-byte for a customer restore over a home
uplink — **no customer has ever driven a restore**.

---

## 9. What the model implies for the tiers (recorded, not new design)

**[DESIGN]** Three consequences follow from §3–§7 and are stated so they are not re-derived:

1. **Tier-2 is a *drive-loss* tier, not a second chance at Tier-1.** Its job is to survive one drive
   dying. That is why it mirrors the unit at all — and why the unit mirror being unreadable (§6.3)
   defeats the tier rather than degrading it.
2. **Tier-3 is a *premises-loss* tier.** It is the only copy that survives fire, theft and
   ransomware-with-host-access, which is why it is the only tier whose credential exposure is ranked
   as the largest open data risk (R-95).
3. **The whole-guest tiers are the *system* tier, and they sit under everything else** (§7.1).
   Treating them as "one more copy" is precisely the 3-2-1 framing this document rejects.

---

## 10. Known gaps

Every divergence between the model above and the system as it is, each with an ID.

### 10.1 D5 is BLOCKED — the backup tree is reachable from a browsing surface

**[FACT] The verification and its result.** Every surface that can read a file was checked:

| surface | can it reach `backups/`? | evidence |
|---|---|---|
| SMB share creation | **NO** | `sharingResolvePath` (`internal/web/sharing_handlers.go:52-81`) resolves symlinks *before* containment, then refuses any path within `SharingDeniedRoots(root)`; that set covers `<root>/backups` **and** the legacy `<root>/felhom-data` + `<root>/felhom-data/backups` (`internal/stacks/samba.go:46-65`, derived from `ProtectedHDDPaths`, `delete.go:59-78`) — **both namespace shapes** |
| SMB browse (folder picker) | **NO** | same deny set applied per child (`sharing_handlers.go:521-537`) |
| SMB `ensureImportShare` (the store-direct bypass) | **NO** | writes one controller-generated constant, `GetImportRoot()` = `<system ns>/userdata/import` (`sharing_handlers.go:564-587`) |
| FileBrowser — **local drives** | **NO** | the bind is `appbackup.UserdataDir(sp.Path)` only, and the comment says why (`internal/web/handlers.go:2450-2460`) |
| **FileBrowser — network shares** | **YES** | the bind is the share **ROOT**: `- %s:/srv/%s:rslave` (`handlers.go:2432`), and the path joins the config source list (`:2433`) |
| `.fab` import path validation | **NO** | confined to `<root>/exports` (`handler_export.go:400-408`, `estimate.go:215-217`) |
| `.fab` browser download | **NO** | name-pattern + parent-must-be-the-staging-dir double guard (`handler_export_download.go:36-45,120-140`) |
| `/api/debug/*` | **NO** | no file-serving branch (`handler_debug.go:46-92`) |
| `http.ServeFile` (3 sites) | **NO** | assets only, `filepath.Base`-normalised (`server.go:720-771`) |
| log bundles | **NO producer found** | no filesystem-walk bundle producer exists on the box; searched `felhom-controller/internal`, `felhom-agent/internal`, `cmd/` |
| registering the backup dir as a drive | **NO** | the manual add requires `system.IsMountPoint(path)` (`handlers.go:2091-2095`); `<drive>/backups` is not a mount point |

**The exposure, end to end.** All six links are source-cited and the precondition is live today:

1. A NAS share is registerable as a storage path and lands `Schedulable: true` — **LIVE on demo-hp**:
   `{"path":"/mnt/felhom-drives/Felhom-Share","kind":"network","schedulable":true}`.
2. `GetSchedulableStoragePaths()` has **no `IsNetwork()` filter** (`internal/settings/settings.go:904-914`),
   so that share appears in the **deploy** dropdown (`handlers.go:462-473`).
3. The **per-app migrate** target list filters only current / decommissioned / disconnected /
   schedulable — also no network filter (`handlers.go:674-679`).
4. `handleStorageMigrateApp` does **not** call `refuseNetworkLifecycle`, unlike its whole-namespace
   sibling which does (`storage_handlers.go:397` vs `:410-424`), and `startMigration` has no guard
   either (`internal/stacks/migrate.go:214-255`).
5. With `HDD_PATH` on the share, `namespaceRoot()` returns it as-is under Model A
   (`internal/backup/backup.go:262-263`), so the app's Tier-1 unit is written to
   `<share>/backups/primary/<app>/compose/app.yaml`.
6. FileBrowser binds that share at its **root** and serves it with `download: true`
   (`internal/infra/infra.go:326`).

**Observed live on demo-hp, in the generated compose — the asymmetry is visible, not inferred:**

```
- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb            <- drive:   userdata-scoped
- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave      <- network: ROOT-bound
- /mnt/sys_drive/felhom-data/userdata/import:/srv/beolvasas
```

**Today this is not a secret leak**, because the unit's `app.yaml` is secret-stripped
(`recovery_unit.go:73`). **D5 would make it one.** That is exactly the test §2 set, and D5 therefore
does **not** hold as written. → **R-108**

### 10.2 The gap register

| ID | Gap | Consequence |
|---|---|---|
| **R-102** | Tier-2 writes a full `recovery-unit/` mirror on every run and **no code path reads it** | Tier-2 is defeated in the drive-loss scenario it exists for (§7.2). Was C9-F4 |
| **R-103** | The Tier-2 no-coverage refusal **names** the working action but does not **route** to it | 45-or-43 of 53 apps dead-end at a message. Was C9-F1b |
| **R-104** | An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach, reported as *„ismeretlen okból"* | the offsite tier stays dead until a human unlocks. Was C9-F3 |
| **R-105** | Three hub-held DR records are empty on the whole live fleet: `hosts.dr_record_json`, `host_escrow.directive_json`, `dr_recipe.host_half.drives` | the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field |
| **R-106** | `dr_recipe.host_half.pbs.namespace` records `"root"` on every box | the recorded restore coordinate is wrong; real namespaces are per-customer |
| **R-107** | No offsite action unpacks the named-volume tars Tier-3 captures on every run | offsite alone cannot rebuild a named-volume app (§7.2) |
| **R-108** | Network storage can host an app's namespace, and FileBrowser binds a network share at its **root** | **blocks D5** (§10.1); today it also lets a `.fab` with plaintext secrets be exported to a NAS (§5) |
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
| R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written |
| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |

### 10.3 Divergences that are documented elsewhere and are not re-opened here

**[FACT]** `06-offsite-connectivity.md:19-21` describes the operator's public edge as a Cloudflare
Tunnel and states DooPlex has no public IP. Live DNS resolves `hub.felhom.eu` through a no-ip
DynDNS CNAME straight to DooPlex's own public address (INV Part E.3). That is a topology-document
issue, not a recovery-model one; recorded so the discrepancy is not lost.

---

## 11. Open decisions — for the operator

**Recorded, deliberately not answered.**

**A. Escrow custody.** Split custody (R **or** an offline operator key) versus a 2-of-3 threshold
across customer / hub / box drives.
*Recommendation on record:* **split custody, operator key held offline and never in the hub.* Note
the constraint from §2: the moment the operator key lives in the hub, "a compromised hub yields
blobs nobody can open" stops being true.

**B. Lost-R policy.** Under split custody the operator **can** recover. Is that the stated policy —
and if so, what identity check gates it? Today matrix row 13 has no route at all, and the customer
is not told that losing R costs them the host-loss route.

**C. RTO / RPO targets per scenario.** **None have ever been stated.** Without them §8 cannot judge
whether, for example, offsite-only recovery is acceptable for drive loss, or whether the 7-day PBS
cadence is adequate for row 8. The measured column is the input; the target column does not exist.

**D. Hetzner as a single failure domain.** restic (Storage Box `u629488`, sub-accounts per customer)
and PBS (Cloud server ep0 + a Cloud Volume) are both Hetzner. Whether they share an **account,
login and payment method** is unverified (INV Unknown U-1) — the question was deliberately not
answered by calling the provider API with the production token. Accept explicitly, or mitigate.

**E. `local` vzdump shares a physical device with the guest it backs up.** LIVE on both hosts:
`/var/lib/vz` (the archive target) and `local-lvm` (the guest's rootfs and both `backup=1`
mountpoints) are both on `/dev/sda3` → VG `pve`. The tier therefore protects against **corruption
and operator error only**, never against disk failure. Accept and name it honestly in the customer-
facing description, or move the target.

**F (added by §10.1, not in the original list).** D5 cannot be adopted until R-108 closes. Is
closing R-108 the intended path, or is D5 withdrawn?

---

## 12. Evidence index

| Claim | Grade | Source |
|---|---|---|
| Tier-2 file restore, gap-fill and after total loss | **PROVEN-LIVE** | CAMPAIGN-9 A1/A3 |
| Tier-2 refuses without an outage for a no-coverage app | **PROVEN-LIVE** | v0.183.0 replay, `felhom.eu/REPORT.md:60` |
| Tier-1 unit restore **executes** | **PROVEN-LIVE** | CAMPAIGN-9 A2 |
| Tier-1 **content recovery after loss** | **UNPROVEN** | `CAMPAIGN-9…:825-827` |
| restic restore of app data (bytes) | **PROVEN-LIVE** | CAMPAIGN-8 R-87, 155.5 MB, 6/7 byte-identical |
| offsite reconstitution of a DB-indexed app | **PROVEN-LIVE** | destructive immich drill 2026-07-20, `00-capability-map.md:66,75` |
| offsite **place-to-live** as a distinct action | **UNPROVEN** | `CAMPAIGN-8…:520` |
| shares restore (files + definitions + credential) | **PROVEN-LIVE** | 2026-07-18, `00-capability-map.md:96` |
| `.fab` drive-to-drive round trip | **PROVEN-LIVE** | CAMPAIGN-6D P-FAB, 1.7 GB |
| `.fab` **browser upload** leg | **UNPROVEN** | `00-capability-map.md:67` |
| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
| unattended restore-test across tiers | **IMPLEMENTED** (rotation not observed across consecutive cadences) | `00-capability-map.md:41`; LIVE per-tier timestamps this session |
| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
| agent DR bring-up (`ModeDRGuestLoss`) | **NEVER EXECUTED** | `CAMPAIGN-8…:522` |
| host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` |
| host rebuilt as its former self | **NEVER DONE** | INV Part D1 |
| escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` |
| hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 |
| a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` |

---

## 13. What this document deliberately does not do

- It does not restate capability status — §8 cites the map, the map cites §8.
- It does not resolve the 7/53-vs-9/43/1 count (§6.2). Both are on the record; neither is adopted.
- It does not estimate a single RTO or RPO. Every blank in §8 is a real gap.
- It does not answer §11. Those are the operator's.
- It does not claim ratification.