felhom.eu/documentation/architecture/07-backup-architecture.md

07 — The recovery model

Status	NOT RATIFIED. Ratification is Viktor's review, not an editor's.
Written	2026-07-28 (full rewrite; supersedes the 2026-07-14 DRAFT entirely)
Verified against	controller v0.183.0 · agent v0.110.0 · hub v0.80.0 · catalog 4252121 · repo HEADs felhom.eu ff050cf, felhom-controller fd50a73, felhom-agent d5c7691
Live fleet at verification	demo-felhom + demo-hp, both guest 9201, both on the versions above
Freshness	CURRENT as of 2026-07-28. Per standing ruling S-2, mark this STALE the moment it is more than a few trains behind. The document it replaces was DRAFT since 2026-07-14, verified against controller v0.132.0 — fifty-one versions stale — and was cited as authoritative throughout that time.

How to read this document. Two kinds of statement appear, and they are always labelled:

• [DESIGN] — a decision taken in the 2026-07-28 architecture discussion (D1–D6, §3–§5, §9). Not derived from code; the code may not implement it yet. Where it does not, §10 says so.
• [FACT] — an observed property, carrying a file:line, a live command output, or a citation to _recovery-inventory-2026-07-28.md (below: INV).

Where the model is silent, this document says OPEN rather than filling the gap.

Primary input: _recovery-inventory-2026-07-28.md (read-only inventory, 2026-07-28) — cited throughout as INV Part n. Every number in §6, §8 and §11 traces back to it.

---

1. Purpose and scope

This document describes how a Felhom customer gets their system back, and who can do it.

It replaces a document that described where copies are written. That was the wrong frame, and §7 explains why: a set of copies is not a set of recovery routes, and the previous document's tier table could be entirely satisfied while a real recovery was impossible.

In scope: the recovery model — trust boundary, lanes, the three-part model, the tiers as inputs to recovery, the dependency chain between them, the failure→recovery matrix, and the encryption policy.

Out of scope, deliberately: implementation specs (they live in task specs), the capture-set algorithm (internal/appbackup/captureset.go and its tests are the source of truth), and per-tier operational runbooks (documentation/runbooks/).

Authority split. This document is authoritative for the failure→recovery matrix (§8). The capability map (00-capability-map.md) stays authoritative for per-capability status. Neither restates the other; §8 rows are cited from the map, not copied into it.

---

2. The trust model

[DESIGN] The operator holds root SSH on every box. That is a fact of the product — the agent is operator-tier, the host is operator-managed, and break-glass exists precisely so the operator can get in when nothing else works. "The operator cannot read customer data" was never the security property, and no part of this document may be read as claiming it.

[FACT] The mechanics that make this concrete:

• The operator's OOB SSH public key is pushed to every box from the hub (hub_settings.oob_operator_ssh_pubkey → /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op, 93 bytes, LIVE on both hosts — INV Part C, row 21).
• The break-glass root@pam console password for every host is stored in the hub and retrievable with the operator's global key (documentation/runbooks/break-glass.md:42-48; LIVE: host_recovery holds 3 rows — INV Part D2.1).
• Guest data is reachable from the host by definition: pct exec, and the data drives are bind mounts on the host (mp8 /mnt/felhom-drives, LIVE pct config 9201 on both hosts).

[DESIGN] What R (the customer's recovery code) does provide, and must keep providing: the hub alone is not enough. A compromised hub yields blobs nobody can open. This property holds only while the operator's own key is not stored in the hub — which is why §11-A is an open decision and why its recommendation on record is "operator key held offline and never in the hub".

[FACT] The escrow is genuinely zero-knowledge today: host_escrow rows carry posture = zero_knowledge, a 383-byte blob and a 572-byte identity blob, and the hub holds only a restic_pw_sha256 hash beside them (LIVE, both hosts — INV Part D2.1). R exists in zero system copies by design (INV Part C, row 24).

So the honest statement of the property is:

The hub cannot open the escrow. The operator can reach a live box. Neither fact substitutes for the other, and R is what keeps the first true.

---

3. The two lanes (D1)

[DESIGN] Recovery is split into two lanes with different owners. This is a product decision, not a limitation.

Lane 1 — the customer, unassisted: files and app data

The customer owns their data and can get it back themselves, through the „Visszaállítás" surfaces, with nothing but their dashboard password. No operator, no ticket, no scheduling.

[FACT] What Lane 1 contains today (INV Part A.1 — seven paths, all behind the controller's RequireAuth gate, which is the customer-owned password: internal/web/auth.go:34-35 puts settings.json → password_hash ahead of the operator-provisioned controller.yaml value):

#	Surface (HU)	Endpoint	Semantics
1	„Visszaállítás indítása"	POST /backup/restore	destructive — rebuilds the app from its Tier-1 unit
2	„Fájlok visszaállítása"	POST /backup/tier2/restore	additive, missing-only
3	„Visszaállítás a távoli tárolóból"	POST /backup/offbox/restore	non-destructive, to a verification copy
4	„Helyreállítás az élő adatok közé"	POST /backup/offbox/place	additive, missing-only
5	„Teljes visszaállítás (fájlok + adatbázis)"	POST /backup/offbox/reconstitute	destructive to files + DB, never deleting
6	„Megosztások visszaállítása" + place	POST /backup/shares/{restore,place}	additive
7	.fab import	POST → apiImportStart	destructive re-import of one app

[FACT] What is proven in Lane 1 (INV Part G.1): paths 2 and 6 are proven live end-to-end; path 5 is proven live through a destructive drill (photos deleted, trash emptied, restored, timeline confirmed); path 3 is proven for bytes only; path 7 is proven for the drive-to-drive circle but its browser-upload leg is not; path 1 is proven to execute but its content recovery after real loss has never been demonstrated — the single most valuable unproven item in the system (audits/CAMPAIGN-9-restore-proof-2026-07-28.md:825-827); path 4 has never been exercised as a distinct action (audits/CAMPAIGN-8-backup-restore-2026-07-27.md:520).

[FACT] One honest residual on the whole lane: every proven run was performed by the operator, not by a customer. 00-capability-map.md:75 still carries "A customer (not the operator) performs a restore via UI alone" as MISSING as evidence — by absence of the run, not by a product gap.

Lane 2 — the operator: guest and host recovery

Rebuilding an LXC guest, rebuilding a host, and re-establishing a box's identity are operator work, by design and by contract. They are not customer-facing and are not going to be.

[FACT] What Lane 2 contains (INV Part A.2): the scheduled restore-test, pct restore, raw proxmox-backup-client restore, raw restic restore, the agent's DR bring-up (--selftest=bring-up -mode dr), the host-loss plan builder, and the escrow-consume ceremony. Each needs root on the host or a CLI flag; none is reachable from any customer surface.

[FACT] What is proven in Lane 2 (INV Parts G.1, G.3): whole-guest pct restore from both tiers is proven with exact mount parity; the unattended restore-test is proven and currently running on both boxes; a corrupted snapshot is proven to fail cleanly. The DR bring-up path has never been executed (CAMPAIGN-8…:522), the host-loss plan executes nothing by construction (felhom-agent/internal/dr/plan.go:1-4), and no host has ever been rebuilt as its former self (INV Part D1).

Why the split is right, stated once

[DESIGN] A customer can reason about "my photos are gone". A customer cannot reason about "restore the LXC and re-attach mp8 by durable_id without binding another guest's bootstrap credentials" — a hazard serious enough to have its own runbook page (runbooks/RUNBOOK-manual-guest-restore.md:45-48, F-OPS). Putting Lane 2 behind the operator is what lets Lane 1 be a button instead of a procedure.

---

4. The three-part model (D4)

[DESIGN] Recovery needs three things, and they live in three different places. Losing one is a different problem from losing another, and the matrix in §8 is organised around that.

Part	Holds	Where	Lost when
Recipe	scaffolding — guest sizing, drive inventory, PVE storage definitions, PBS coordinates, app inventory + storage bindings. No secrets, no bytes.	the hub	the hub is lost
Escrow	the host identity key material and the restic repo password	the hub, R-wrapped	the hub is lost, or R is lost
Bytes	Tier-1 / Tier-2 / Tier-3 / whole-guest	the tiers	the relevant medium is lost

[FACT] The Recipe exists and is current. dr_recipe holds 6 rows; demo-felhom's was updated 2026-07-28 17:31:13 and demo-hp's 17:25:31 — i.e. within one report cycle (LIVE, INV Part D2). It carries guest sizing, pve_storage[], and an app half with per-app storage_bindings.

[FACT] The Escrow exists and is zero-knowledge. host_escrow: 2 rows, posture zero_knowledge; the identity bundle's shape is {tunnel_token, pbs_token, wg_private_key, restic_repo_password} (felhom-agent/internal/escrow/identity.go:26-39).

[FACT] Three parts of the Recipe are empty or wrong on the live fleet, and they are exactly the parts a host-loss recovery would read (INV Part D2.3): hosts.dr_record_json is {} on all three hosts; host_escrow.directive_json is {} on both escrowed hosts; dr_recipe.host_half.drives is [] on every customer including two with enrolled data drives; and dr_recipe.host_half.pbs.namespace reads "root" while the real namespaces are demo-felhom / demo-hp. → R-105, R-106.

---

5. Encryption policy (D2)

[DESIGN] Encryption follows the boundary, not the tier.

On the customer's own premises: plaintext

Tier-1 recovery units, Tier-2 cross-drive copies, the app data on the drives, and the local vzdump archives on the host are unencrypted, deliberately.

The reasoning, recorded so nobody "hardens" this later:

1. It buys nothing against a real threat. Someone who can take the second drive can take the first. Local encryption defends against a threat model — physical theft of only the backup medium — that does not describe a home server where both media sit in the same box.
2. It adds a key-loss path that turns a working backup into a brick. Every local encryption key is one more thing that must survive the disaster it exists for, and the system already has one such dependency it is trying to reduce (§7).
3. It would break browsing, which is a feature. FileBrowser and SMB let the household see and use their own files. An encrypted local copy is not browsable, and the „Megosztás" and FileBrowser surfaces are part of the product, not an accident.

[FACT] The local plaintext posture is real and verifiable: dir: local in /etc/pve/storage.cfg carries no encryption-key (LIVE, both hosts), so the daily whole-guest archive is a plain .tar.zst — 5.93 GB on demo-felhom, 1.68 GB on demo-hp (LIVE, INV Part B.3). That archive is the single highest-value object on the box: it contains encryption.key, the restic repo_password, the offbox ssh_key, settings.json and controller.yaml (all under /var/lib/docker = mp0, backup=1). Naming that plainly is part of the policy, not an argument against it.

Leaving the premises: encrypted, and the provider must not be able to read it

[FACT] Both offsite tiers encrypt client-side:

• restic (Tier-3): repo password is a 256-bit hex value generated once on the box and never logged (internal/backup/offbox.go:392-410); restic is invoked with RESTIC_PASSWORD_FILE (:608).
• PBS (whole-guest offsite): /etc/pve/storage.cfg carries a per-customer encryption-key and the fingerprints differ between demo-felhom and demo-hp (LIVE). The live restore command line observed on demo-felhom carries --crypt-mode=encrypt (INV Part A.2.3). Per-tenant encryption is why cross-customer dedup is impossible on the shared datastore — a cost consequence, recorded, not a defect.

The one exception, stated because it is not covered by either rule

[FACT] The .fab bundle writes app.yaml with decrypted plaintext secrets, deliberately, and its password is optional (internal/appexport/export.go:484,506-511; the generated file's first line is literally # Exported by felhom-controller — plaintext secrets, :531; Encrypted: req.Password != "", :307). A .fab can also be written to a registered network share, because storageDriveList() does not filter network paths (internal/web/handler_export.go:377-387). This is a portability artifact, not a tier — but it is the one place where a customer action can put every secret of one app onto a NAS in plaintext. Recorded here so the encryption policy is not read as covering it. → R-108 (same root cause).

---

6. The tiers — what each captures

The tiers are inputs to recovery, not recovery routes. §7 and §8 say what they can actually do.

6.1 The four tiers, as configured on the live fleet

Tier	Location	Captures	Cadence (LIVE)	Retention (LIVE)	Encrypted
Tier-1 recovery unit	<nsRoot>/backups/primary/<app>/ on the app's own drive	compose + .felhom.yml + secret-stripped app.yaml, db-dumps/*.sql, volume-dumps/*.tar, manifest.json	nightly at W, plus a checksum-gated refresh on the periodic status pass	one point per app (restore_points.go:14-18)	no
Tier-2 cross-drive	<target nsRoot>/backups/secondary/<app>/	always a full mirror of the unit (tier2.go:368-369) plus mandatory + optional file legs, v2 layout hdd/<rel> + userdata/<rel>	nightly at W+60m	mirror (rsync)	no
Tier-3 restic offsite	Hetzner Storage Box over SFTP, one multi-path snapshot per app per run	the unit plus mandatory legs only; a separate _shares snapshot	nightly at W+105m	--keep-daily 7 --keep-weekly 4 --keep-monthly 6	yes (restic)
Plane-2 whole-guest, local	local: → /var/lib/vz/dump on the host	rootfs + mp0 /var/lib/docker + mp1 /mnt/sys_drive	24 h (backup_cadence_seconds: 0)	local_backup_retention: 3	no
Plane-2 whole-guest, offsite	felhom-pbs: → ep0 datastore felhom-offsite, per-customer namespace, over WireGuard	same contents	7 days (604800)	server-side prune on ep0, keep-last 2 at 03:30	yes (per-customer key)

[FACT] The three nightly legs derive from one customer-settable window start W at fixed offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered (cmd/controller/main.go:604-607). Both boxes run W = 02:30.

[FACT] What the whole-guest tiers do NOT carry. mp8 /mnt/felhom-drives and mp9 /etc/felhom-bootstrap are host bind mounts and are out of vzdump scope entirely (LIVE pct config 9201, both hosts). So the whole-guest tiers carry the guest and none of the customer's data drives — 916 GB on demo-felhom, 938 GB on demo-hp.

6.2 Coverage per app class — and an unresolved count

[FACT] Of 53 catalog templates, 52 keep data in Docker named volumes; exactly 13 carry a backup: block, and those 13 are exactly the templates that bind ${HDD_PATH} / ${USERDATA_PATH} / ${IMPORT_PATH} at all; 14 have a database service (INV Part B.1, computed against catalog 4252121).

Applying the classifier's documented two-level default (internal/appbackup/classify.go: explicit entry wins; else a :ro reader is excluded; else a writable bind is mandatory):

tier filter	templates with at least one file leg
Tier-3 (mandatory only)	4 — calibre-web, immich, nextcloud, paperless-ngx
Tier-2 (mandatory + optional)	7 — the 4 above plus audiobookshelf, komga, romm
legacy resolver path (apps with no block)	0 — no non-block template binds a namespace path

⚠️ UNRESOLVED — two counts of the same thing disagree

source	count
C9-F1 Phase 0, as shipped	A = 9 / B = 43 / C = 1 — felhom.eu/REPORT.md:17-24, restated backlog/OPEN-ITEMS.md:31
INV Part B.1, independent enumeration at catalog 4252121	A = 7 / B = 45 / C = 1

Both use the same definition ("templates whose Tier-2 copy can hold a readable file leg"). The difference is two apps and neither number is adopted here. The Phase-0 enumeration is described in prose but the script is not committed, so the two methods cannot be diffed from the repo. This must be resolved before either figure is used to size anything.

[FACT] The class-B consequence is real regardless of which count is right. For an app whose data lives entirely in named volumes, the Tier-2 copy holds a full recovery-unit/ and no readable leg — LIVE on demo-felhom, backups/secondary/bookstack/ and .../docmost/ contain recovery-unit and nothing else, at 156 MB and 86 MB (INV Part B.2). Since v0.183.0 the button refuses before stopping the app and names the action that works.

6.3 Where capture and restore disagree

[FACT] Three asymmetries, each source-cited:

tier	captured	read back by that tier's restore	gap
Tier-1	unit incl. volume tars + DB dumps	all of it	none
Tier-2	unit mirror + file legs	hdd/ and userdata/ only (tier2_restore.go:101-104)	the unit mirror is read by nothing — RecoveryUnitPath resolves to backups/primary/ (appbackup/paths.go:46-48) → R-102
Tier-3	unit (incl. volume tars) + mandatory legs	files + DB replay; the unit is skipped on the way to live (offbox_reconstitute.go:284-289; placed only if the live unit is absent, offbox_restore.go:352-356)	no offsite action unpacks the named-volume tars it captures → R-107

[FACT] Tier-2's gap is the sharper one because of when it bites: Tier-2 exists for the case where the primary drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action.

---

7. The recovery chain (D3) — the reason this document exists

[DESIGN] 3-2-1 describes copies. It does not describe recovery.

Three copies on two media with one offsite is a statement about bytes surviving. It says nothing about whether the bytes can be turned back into a working system, and a system can satisfy 3-2-1 completely while having no executable recovery route for a given failure. That is not a hypothetical here — §8 has rows where it is the actual state.

7.1 The dependency graph

[FACT] Every app-tier restore depends on the guest, for one of two reasons:

                       ┌──────────────────────────────────────────┐
                       │  the LIVE GUEST                          │
                       │   · settings.json (tier-2 destination)   │
                       │   · encryption.key  (32 B)               │
                       │   · app.yaml  (ENC: under that key)      │
                       │   · the deployed app itself              │
                       └───────────┬──────────────────────────────┘
                                   │ required by
        ┌──────────────────────────┼──────────────────────────┬─────────────────────┐
        │                          │                          │                     │
   Tier-1 restore            Tier-2 restore            Tier-3 reconstitute    Tier-3 scratch
   (rebuild the app)         (fill in files)           (overwrite + replay)   (verification copy)
        │                          │                          │                     │
   needs SECRETS              needs the app             needs the app          needs only the
   from app.yaml              running + the             deployed + a DB        repo password
   (restore_unit.go           recorded dest             service identifiable   (also in the guest)
    :130-132)                 (tier2_restore.go
                               :114-116)
        │                          │                          │                     │
        └──────────────────────────┴──────────────────────────┴─────────────────────┘
                                   │
                       ┌───────────▼──────────────────────────────┐
                       │  WHOLE-GUEST TIER  (local vzdump / PBS)  │   ← Lane 2, operator only
                       └───────────┬──────────────────────────────┘
                                   │ required by
                       ┌───────────▼──────────────────────────────┐
                       │  THE HOST  (agent.json, bootstrap.json,  │   ← in NO backup at all
                       │   vmbr9, sudoers, /etc/pve/priv, WG)     │      (INV Part D1)
                       └──────────────────────────────────────────┘

[FACT] The two legs of the dependency, precisely:

1. Secrets. The recovery unit is secret-free by design — "It NEVER writes a secret value" (internal/backup/recovery_unit.go:73). RestoreFromRecoveryUnit recovers secrets from the guest, never from the unit (restore_unit.go:130-132), and the policy is stated outright at :18-22: "Regenerate NOTHING… A missing DATA-ENCRYPTING key is FATAL: regenerating it would render the restored data unreadable, so we refuse and tell the operator to do a PBS whole-guest restore." Those secrets are encrypted under encryption.key, a 32-byte file that exists only inside the guest (LIVE, both boxes — INV Part C, row 9).
2. The living app. Tier-2's restore reads the destination recorded in the guest's settings.json (tier2_restore.go:114-116), and Tier-3's reconstitution refuses outright when the app is not deployed — „a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat" (offbox_reconstitute.go:198-201).

[FACT] So all three app tiers are conditioned on the whole-guest tier, which is Lane 2. The customer's own recovery lane rests on an operator-only tier, and today nothing tells the customer that.

7.2 A tier whose prerequisites cannot be met in the failure it exists for

[FACT] Two instances, both current:

• Tier-2 vs primary-drive loss. Tier-2's stated purpose is surviving the loss of the primary drive. In that failure the primary recovery unit is gone; the surviving mirror on the second drive is backups/secondary/<app>/recovery-unit/, which no code path reads (§6.3). For the 45-or-43 class-B apps the restore is a guaranteed no-op in exactly its designed scenario. → R-102
• Tier-3 vs guest loss. Tier-3 holds the volume tars and the DB dump. Reconstitution requires the app to be deployed and skips the unit; the tars are unpacked only by the Tier-1 path, which requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest. → R-107

7.3 What D5 would change — and why it is blocked

[DESIGN, TARGET — BLOCKED] The intended fix is to make app secrets travel with the local recovery unit, so Tier-1 and Tier-2 restore work without the guest and without R. Offsite already encrypts everything, so secrets travelling offsite would be covered by the escrowed repo password. R would then be required for offsite recovery and host identity only — losing R would cost the offsite route, not local recovery.

This is not adopted. §2 of the task that produced this document required the premise to be established, not assumed: the backup tree must be unreachable from every browsing, download and export surface. It is not. The verification and the exposure are in §10.1. Until that is closed (R-108), D5 stays a target and §7.1's chain stands as the model.

---

8. The failure → recovery matrix

This is the core artifact. One row per failure. It is authoritative for recovery routes; 00-capability-map.md stays authoritative for per-capability status.

How to read the numbers.

• RTO — only measured durations from INV Part F appear here. A blank cell means nobody has ever measured it, and a blank is a finding, not an omission.
• RPO — no RPO has ever been measured from an incident. These cells carry the configured cadence that bounds RPO, read live from the box, labelled (cadence). A blank means no cadence governs the row.
• Status — PROVEN (live, cited) · PARTIAL (some legs proven) · IMPLEMENTED (code + tests, never exercised) · NONE (no route exists).

#	Failure	What survives	Recovery route	Invocable by	RTO (measured)	RPO (cadence)	Status	Evidence
1	Customer deletes files	everything else	„Fájlok visszaállítása" — Tier-2 additive merge	customer	39 s (6 files)	24 h	PROVEN	CAMPAIGN-9 A1 — byte-identical returns, both non-destruction promises kept, bytes re-served by paperless's own API
2	An app's data directory is destroyed	the guest, the other tiers	same route	customer	46 s (43 files)	24 h	PROVEN	CAMPAIGN-9 A3 — loss verified by a positive observable (doc download 200 → 404), then 16/16 documents usable
3	An app's DB and named volumes are lost	the guest, the unit	„Visszaállítás indítása" — Tier-1 unit restore (the only path that unpacks volume tars)	customer	18.25 s (path execution)	24 h	PARTIAL	CAMPAIGN-9 A2 proves the path executes and the app survives; content recovery after real loss never demonstrated (CAMPAIGN-9…:825-827)
3b	same, for a class-B app via Tier-2	—	no route — Tier-2 never reads the unit mirror	—			NONE	§6.3; R-102
4	Primary drive dies	Tier-2 copy on the second drive; Tier-3 offsite; the guest	Tier-2 for file legs (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB; the volume tars in either copy are unreachable	customer (both)		24 h	PARTIAL	§7.2; R-102, R-107
5	Secondary drive dies	everything the customer uses	none needed — Tier-2 is a derived copy, rebuilt on the next run (07 §8 migration rule: "Migration = rebuild, not preserve")	automatic		24 h	PROVEN (by construction)	tier2 v2 layout marker + rebuild, internal/backup/tier2.go:359-393
6	Guest lost or corrupted	the host, both whole-guest tiers, the data drives (they are host binds)	pct restore from local: or felhom-pbs:	operator (SSH)	84–112 s local · 1101 s PBS (both = restore-test into a scratch guest, boot + verify + teardown)	24 h local · 7 d offsite	PROVEN	CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C (exact mount parity, unprivileged: 1 preserved); LIVE restore-tests on both boxes this session
7	Guest stopped and does not come back	everything	guest-power watchdog (60 s, onboot as the deliberate-stop discriminator)	automatic	120 s		PROVEN	agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human
8	Host dies (hardware), drives intact	the data drives; ep0's PBS namespace; the hub's Recipe + Escrow	install a new host, then --selftest=bring-up -mode dr per guest, then re-attach drives by durable_id	operator (SSH)		7 d	IMPLEMENTED	bring-up code exists and has never been executed (CAMPAIGN-8…:522); the drive half of the plan is empty on every box (R-105)
9	Whole box lost (fire/theft) — host and drives gone	ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow	new hardware → day-0 → escrow-consume with R → restore guests from PBS → app data from Tier-3	operator + customer (R)		7 d (guest) · 24 h (app data)	IMPLEMENTED / UNPROVEN	every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (06-offsite-connectivity.md:327)
10	Ransomware / malicious deletion inside the guest	PBS offsite (the box cannot delete its own snapshots); the restic repo is NOT protected the same way	whole-guest restore from PBS to a point before the event	operator (SSH)		7 d	PARTIAL	R-89 proved the box is refused when deleting its own PBS snapshot (CAMPAIGN-8…:514-515). R-95 (open, ranked #1): the restic credential can delete — readonly=False, forget --prune runs from the box, and SFTP cannot express append-only (OPEN-ITEMS.md:13)
11	Hub lost	every box's data plane, every tier, every Lane-1 route	none needed for recovery of a box; the hub itself restores from its Longhorn volume backup	operator (kubectl)		24 h + weekly (Longhorn RETAIN 1 each)	UNPROVEN	a hub restore has never been performed. The backup target is nfs://192.168.0.180 — DooPlex itself — and exactly 2 restore points exist (LIVE, INV Part D2.2)
11b	consequences while the hub is gone	—	—	—			[FACT]	day: nothing customer-visible breaks; events queue (settings.go:1466-1485). week: the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. permanently: escrow custody and break-glass credentials are gone (INV Part D2.4)
12	Offsite provider lost (Hetzner)	everything on-premises: both drives, both whole-guest tiers	none needed — on-premises recovery is unaffected. Re-provision a new offsite target.	operator			[FACT]	restic and PBS share the provider (INV Part E.1). Whether they share an account and payment method is UNKNOWN → §11-D
13	Customer loses R	every byte, every tier, the Recipe	on-premises recovery is unaffected — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish host identity and to open the escrowed offsite key material after a host loss	—			NONE for host-loss	R exists in zero system copies by design (INV Part C, row 24). Whether the operator should be able to recover is open → §11-A, §11-B
14	Host SSH management plane dies	everything	3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted root@pam on the PVE console	operator			PROVEN	runbooks/break-glass.md; the original incident and its fix
15	Interrupted offsite run leaves a stale lock	everything	manual restic unlock --remove-all	operator (SSH)			DEFECT	the self-heal exists (offbox.go:634-648) but the probe fails first and classifyResticProbe has no lock case → fail-fast, operator told „ismeretlen okból" → R-104

8.1 The blank cells, listed explicitly

Per the rule that a blank is a finding, here they are:

row	blank	why
3b	RTO, RPO	no route exists to time
4	RTO	no drive-loss recovery has ever been timed
5	RTO	never timed; the rebuild is a normal Tier-2 run
6	—	RTO present, but it is a restore into a scratch guest on the same host; a restore to a different host has never been timed
8	RTO	a host has never been rebuilt as itself (INV Part D1)
9	RTO	the composed whole-box path has never been run
10	RTO	no ransomware-shaped recovery has ever been run
11	RTO	a hub restore has never been performed
12	RTO, RPO	no provider-loss recovery has ever been run
13	RTO, RPO	not a timed recovery; a capability loss
14	RTO	the break-glass path is proven but was never timed
15	RTO	the manual unlock was performed but not timed

Also unmeasured, and not representable as a row (INV Part F.3): any restore larger than 155.5 MB from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS incremental; .fab import wall-clock; and time-to-first-byte for a customer restore over a home uplink — no customer has ever driven a restore.

---

9. What the model implies for the tiers (recorded, not new design)

[DESIGN] Three consequences follow from §3–§7 and are stated so they are not re-derived:

1. Tier-2 is a drive-loss tier, not a second chance at Tier-1. Its job is to survive one drive dying. That is why it mirrors the unit at all — and why the unit mirror being unreadable (§6.3) defeats the tier rather than degrading it.
2. Tier-3 is a premises-loss tier. It is the only copy that survives fire, theft and ransomware-with-host-access, which is why it is the only tier whose credential exposure is ranked as the largest open data risk (R-95).
3. The whole-guest tiers are the system tier, and they sit under everything else (§7.1). Treating them as "one more copy" is precisely the 3-2-1 framing this document rejects.

---

10. Known gaps

Every divergence between the model above and the system as it is, each with an ID.

10.1 D5 is BLOCKED — the backup tree is reachable from a browsing surface

[FACT] The verification and its result. Every surface that can read a file was checked:

surface	can it reach backups/?	evidence
SMB share creation	NO	sharingResolvePath (internal/web/sharing_handlers.go:52-81) resolves symlinks before containment, then refuses any path within SharingDeniedRoots(root); that set covers <root>/backups and the legacy <root>/felhom-data + <root>/felhom-data/backups (internal/stacks/samba.go:46-65, derived from ProtectedHDDPaths, delete.go:59-78) — both namespace shapes
SMB browse (folder picker)	NO	same deny set applied per child (sharing_handlers.go:521-537)
SMB ensureImportShare (the store-direct bypass)	NO	writes one controller-generated constant, GetImportRoot() = <system ns>/userdata/import (sharing_handlers.go:564-587)
FileBrowser — local drives	NO	the bind is appbackup.UserdataDir(sp.Path) only, and the comment says why (internal/web/handlers.go:2450-2460)
FileBrowser — network shares	YES	the bind is the share ROOT: - %s:/srv/%s:rslave (handlers.go:2432), and the path joins the config source list (:2433)
.fab import path validation	NO	confined to <root>/exports (handler_export.go:400-408, estimate.go:215-217)
.fab browser download	NO	name-pattern + parent-must-be-the-staging-dir double guard (handler_export_download.go:36-45,120-140)
/api/debug/*	NO	no file-serving branch (handler_debug.go:46-92)
http.ServeFile (3 sites)	NO	assets only, filepath.Base-normalised (server.go:720-771)
log bundles	NO producer found	no filesystem-walk bundle producer exists on the box; searched felhom-controller/internal, felhom-agent/internal, cmd/
registering the backup dir as a drive	NO	the manual add requires system.IsMountPoint(path) (handlers.go:2091-2095); <drive>/backups is not a mount point

The exposure, end to end. All six links are source-cited and the precondition is live today:

1. A NAS share is registerable as a storage path and lands Schedulable: true — LIVE on demo-hp: {"path":"/mnt/felhom-drives/Felhom-Share","kind":"network","schedulable":true}.
2. GetSchedulableStoragePaths() has no IsNetwork() filter (internal/settings/settings.go:904-914), so that share appears in the deploy dropdown (handlers.go:462-473).
3. The per-app migrate target list filters only current / decommissioned / disconnected / schedulable — also no network filter (handlers.go:674-679).
4. handleStorageMigrateApp does not call refuseNetworkLifecycle, unlike its whole-namespace sibling which does (storage_handlers.go:397 vs :410-424), and startMigration has no guard either (internal/stacks/migrate.go:214-255).
5. With HDD_PATH on the share, namespaceRoot() returns it as-is under Model A (internal/backup/backup.go:262-263), so the app's Tier-1 unit is written to <share>/backups/primary/<app>/compose/app.yaml.
6. FileBrowser binds that share at its root and serves it with download: true (internal/infra/infra.go:326).

Observed live on demo-hp, in the generated compose — the asymmetry is visible, not inferred:

- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb            <- drive:   userdata-scoped
- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave      <- network: ROOT-bound
- /mnt/sys_drive/felhom-data/userdata/import:/srv/beolvasas

Today this is not a secret leak, because the unit's app.yaml is secret-stripped (recovery_unit.go:73). D5 would make it one. That is exactly the test §2 set, and D5 therefore does not hold as written. → R-108

10.2 The gap register

ID	Gap	Consequence
R-102	Tier-2 writes a full recovery-unit/ mirror on every run and no code path reads it	Tier-2 is defeated in the drive-loss scenario it exists for (§7.2). Was C9-F4
R-103	The Tier-2 no-coverage refusal names the working action but does not route to it	45-or-43 of 53 apps dead-end at a message. Was C9-F1b
R-104	An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach, reported as „ismeretlen okból"	the offsite tier stays dead until a human unlocks. Was C9-F3
R-105	Three hub-held DR records are empty on the whole live fleet: hosts.dr_record_json, host_escrow.directive_json, dr_recipe.host_half.drives	the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field
R-106	dr_recipe.host_half.pbs.namespace records "root" on every box	the recorded restore coordinate is wrong; real namespaces are per-customer
R-107	No offsite action unpacks the named-volume tars Tier-3 captures on every run	offsite alone cannot rebuild a named-volume app (§7.2)
R-108	Network storage can host an app's namespace, and FileBrowser binds a network share at its root	blocks D5 (§10.1); today it also lets a .fab with plaintext secrets be exported to a NAS (§5)
R-95 (open)	The restic offsite credential can delete — the box can forget --prune its own repo	the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10)
R-86 (open)	Restore-tests are interval-scheduled, not backup-aligned	a tier's proof cadence is unrelated to when its archives are written
R-87 (open)	The restic tier is never restore-tested	matrix row 4's route has no unattended proof

10.3 Divergences that are documented elsewhere and are not re-opened here

[FACT] 06-offsite-connectivity.md:19-21 describes the operator's public edge as a Cloudflare Tunnel and states DooPlex has no public IP. Live DNS resolves hub.felhom.eu through a no-ip DynDNS CNAME straight to DooPlex's own public address (INV Part E.3). That is a topology-document issue, not a recovery-model one; recorded so the discrepancy is not lost.

---

11. Open decisions — for the operator

Recorded, deliberately not answered.

A. Escrow custody. Split custody (R or an offline operator key) versus a 2-of-3 threshold across customer / hub / box drives. Recommendation on record: *split custody, operator key held offline and never in the hub. Note the constraint from §2: the moment the operator key lives in the hub, "a compromised hub yields blobs nobody can open" stops being true.

B. Lost-R policy. Under split custody the operator can recover. Is that the stated policy — and if so, what identity check gates it? Today matrix row 13 has no route at all, and the customer is not told that losing R costs them the host-loss route.

C. RTO / RPO targets per scenario. None have ever been stated. Without them §8 cannot judge whether, for example, offsite-only recovery is acceptable for drive loss, or whether the 7-day PBS cadence is adequate for row 8. The measured column is the input; the target column does not exist.

D. Hetzner as a single failure domain. restic (Storage Box u629488, sub-accounts per customer) and PBS (Cloud server ep0 + a Cloud Volume) are both Hetzner. Whether they share an account, login and payment method is unverified (INV Unknown U-1) — the question was deliberately not answered by calling the provider API with the production token. Accept explicitly, or mitigate.

E. local vzdump shares a physical device with the guest it backs up. LIVE on both hosts: /var/lib/vz (the archive target) and local-lvm (the guest's rootfs and both backup=1 mountpoints) are both on /dev/sda3 → VG pve. The tier therefore protects against corruption and operator error only, never against disk failure. Accept and name it honestly in the customer- facing description, or move the target.

F (added by §10.1, not in the original list). D5 cannot be adopted until R-108 closes. Is closing R-108 the intended path, or is D5 withdrawn?

---

12. Evidence index

Claim	Grade	Source
Tier-2 file restore, gap-fill and after total loss	PROVEN-LIVE	CAMPAIGN-9 A1/A3
Tier-2 refuses without an outage for a no-coverage app	PROVEN-LIVE	v0.183.0 replay, felhom.eu/REPORT.md:60
Tier-1 unit restore executes	PROVEN-LIVE	CAMPAIGN-9 A2
Tier-1 content recovery after loss	UNPROVEN	CAMPAIGN-9…:825-827
restic restore of app data (bytes)	PROVEN-LIVE	CAMPAIGN-8 R-87, 155.5 MB, 6/7 byte-identical
offsite reconstitution of a DB-indexed app	PROVEN-LIVE	destructive immich drill 2026-07-20, 00-capability-map.md:66,75
offsite place-to-live as a distinct action	UNPROVEN	CAMPAIGN-8…:520
shares restore (files + definitions + credential)	PROVEN-LIVE	2026-07-18, 00-capability-map.md:96
.fab drive-to-drive round trip	PROVEN-LIVE	CAMPAIGN-6D P-FAB, 1.7 GB
.fab browser upload leg	UNPROVEN	00-capability-map.md:67
whole-guest restore, local and PBS, exact mount parity	PROVEN-LIVE	CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C
corrupted PBS snapshot fails cleanly	PROVEN-LIVE	CAMPAIGN-8 fault 17
the box cannot delete its own PBS snapshots	PROVEN-LIVE	CAMPAIGN-8, R-89
unattended restore-test across tiers	IMPLEMENTED (rotation not observed across consecutive cadences)	00-capability-map.md:41; LIVE per-tier timestamps this session
guest-power watchdog	PROVEN-LIVE	agent v0.107.0, 120 s
quiesce crash recovery	PROVEN-LIVE	CAMPAIGN-8 fault 10, 1 s, by SIGKILL
break-glass	PROVEN-LIVE	runbooks/break-glass.md
agent DR bring-up (ModeDRGuestLoss)	NEVER EXECUTED	CAMPAIGN-8…:522
host-loss plan → an actual restore	EXECUTES NOTHING BY CONSTRUCTION	felhom-agent/internal/dr/plan.go:1-4
host rebuilt as its former self	NEVER DONE	INV Part D1
escrow consume in a real recovery	SPIKE-LEVEL ONLY	06-offsite-connectivity.md:327
hub DB restore from its Longhorn backup	NEVER DONE	INV Part D2.2
a customer performing a restore unassisted	MISSING AS EVIDENCE	00-capability-map.md:75

---

13. What this document deliberately does not do

• It does not restate capability status — §8 cites the map, the map cites §8.
• It does not resolve the 7/53-vs-9/43/1 count (§6.2). Both are on the record; neither is adopted.
• It does not estimate a single RTO or RPO. Every blank in §8 is a real gap.
• It does not answer §11. Those are the operator's.
• It does not claim ratification.