# 07 — The recovery model > | | | > |---|---| > | **Status** | **NOT RATIFIED.** Ratification is Viktor's review, not an editor's. | > | **Written** | 2026-07-28 (full rewrite; supersedes the 2026-07-14 DRAFT entirely) | > | **Verified against** | controller **v0.183.0** · agent **v0.110.0** · hub **v0.80.0** · catalog `4252121` · repo HEADs `felhom.eu ff050cf`, `felhom-controller fd50a73`, `felhom-agent d5c7691` | > | **Live fleet at verification** | demo-felhom + demo-hp, both guest 9201, both on the versions above | > | **Freshness** | **CURRENT** as of 2026-07-28. Per standing ruling **S-2**, mark this STALE the moment it is more than a few trains behind. The document it replaces was DRAFT since 2026-07-14, verified against controller v0.132.0 — **fifty-one versions stale** — and was cited as authoritative throughout that time. | > > **How to read this document.** Two kinds of statement appear, and they are always labelled: > > - **[DESIGN]** — a decision taken in the 2026-07-28 architecture discussion (D1–D6, §3–§5, §9). > Not derived from code; the code may not implement it yet. Where it does not, §10 says so. > - **[FACT]** — an observed property, carrying a `file:line`, a live command output, or a citation > to `_recovery-inventory-2026-07-28.md` (below: **INV**). > > Where the model is silent, this document says **OPEN** rather than filling the gap. > > **Primary input:** `_recovery-inventory-2026-07-28.md` (read-only inventory, 2026-07-28) — cited > throughout as **INV Part n**. Every number in §6, §8 and §11 traces back to it. --- ## 1. Purpose and scope This document describes **how a Felhom customer gets their system back**, and who can do it. It replaces a document that described **where copies are written**. That was the wrong frame, and §7 explains why: a set of copies is not a set of recovery routes, and the previous document's tier table could be entirely satisfied while a real recovery was impossible. **In scope:** the recovery model — trust boundary, lanes, the three-part model, the tiers as *inputs to recovery*, the dependency chain between them, the failure→recovery matrix, and the encryption policy. **Out of scope, deliberately:** implementation specs (they live in task specs), the capture-set algorithm (`internal/appbackup/captureset.go` and its tests are the source of truth), and per-tier operational runbooks (`documentation/runbooks/`). **Authority split.** This document is authoritative for the **failure→recovery matrix** (§8). The capability map (`00-capability-map.md`) stays authoritative for **per-capability status**. Neither restates the other; §8 rows are cited from the map, not copied into it. --- ## 2. The trust model **[DESIGN] The operator holds root SSH on every box.** That is a fact of the product — the agent is operator-tier, the host is operator-managed, and break-glass exists precisely so the operator can get in when nothing else works. **"The operator cannot read customer data" was never the security property**, and no part of this document may be read as claiming it. **[FACT]** The mechanics that make this concrete: - The operator's OOB SSH public key is pushed to every box from the hub (`hub_settings.oob_operator_ssh_pubkey` → `/var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op`, 93 bytes, LIVE on both hosts — INV Part C, row 21). - The break-glass `root@pam` console password for every host is stored in the hub and retrievable with the operator's global key (`documentation/runbooks/break-glass.md:42-48`; LIVE: `host_recovery` holds 3 rows — INV Part D2.1). - Guest data is reachable from the host by definition: `pct exec`, and the data drives are bind mounts on the host (`mp8 /mnt/felhom-drives`, LIVE `pct config 9201` on both hosts). **[DESIGN] What R (the customer's recovery code) does provide, and must keep providing: the hub alone is not enough.** A compromised hub yields blobs nobody can open. This property holds **only while the operator's own key is not stored in the hub** — which is why §11-A is an open decision and why its recommendation on record is "operator key held offline and never in the hub". **[FACT]** The escrow is genuinely zero-knowledge today: `host_escrow` rows carry `posture = zero_knowledge`, a 383-byte blob and a 572-byte identity blob, and the hub holds only a `restic_pw_sha256` **hash** beside them (LIVE, both hosts — INV Part D2.1). R exists in **zero** system copies by design (INV Part C, row 24). So the honest statement of the property is: > **The hub cannot open the escrow. The operator can reach a live box. Neither fact substitutes for > the other, and R is what keeps the first true.** --- ## 3. The two lanes (D1) **[DESIGN] Recovery is split into two lanes with different owners. This is a product decision, not a limitation.** ### Lane 1 — the customer, unassisted: files and app data The customer owns their data and can get it back themselves, through the „Visszaállítás" surfaces, with nothing but their dashboard password. No operator, no ticket, no scheduling. **[FACT] What Lane 1 contains today** (INV Part A.1 — seven paths, all behind the controller's `RequireAuth` gate, which is the **customer-owned** password: `internal/web/auth.go:34-35` puts `settings.json → password_hash` ahead of the operator-provisioned `controller.yaml` value): | # | Surface (HU) | Endpoint | Semantics | |---|---|---|---| | 1 | „Visszaállítás indítása" | `POST /backup/restore` | destructive — rebuilds the app from its Tier-1 unit | | 2 | „Fájlok visszaállítása" | `POST /backup/tier2/restore` | additive, missing-only | | 3 | „Visszaállítás a távoli tárolóból" | `POST /backup/offbox/restore` | non-destructive, to a verification copy | | 4 | „Helyreállítás az élő adatok közé" | `POST /backup/offbox/place` | additive, missing-only | | 5 | „Teljes visszaállítás (fájlok + adatbázis)" | `POST /backup/offbox/reconstitute` | destructive to files + DB, never deleting | | 6 | „Megosztások visszaállítása" + place | `POST /backup/shares/{restore,place}` | additive | | 7 | `.fab` import | `POST` → `apiImportStart` | destructive re-import of one app | **[FACT] What is proven in Lane 1** (INV Part G.1): paths 2 and 6 are proven live end-to-end; path 5 is proven live through a destructive drill (photos deleted, trash emptied, restored, timeline confirmed); path 3 is proven for **bytes** only; path 7 is proven for the drive-to-drive circle but its browser-upload leg is not; path 1 is proven to **execute** but its **content recovery after real loss has never been demonstrated** — the single most valuable unproven item in the system (`audits/CAMPAIGN-9-restore-proof-2026-07-28.md:825-827`); path 4 has never been exercised as a distinct action (`audits/CAMPAIGN-8-backup-restore-2026-07-27.md:520`). **[FACT] One honest residual on the whole lane:** every proven run was performed by the **operator**, not by a customer. `00-capability-map.md:75` still carries "A customer (not the operator) performs a restore via UI alone" as **MISSING as evidence** — by absence of the run, not by a product gap. ### Lane 2 — the operator: guest and host recovery Rebuilding an LXC guest, rebuilding a host, and re-establishing a box's identity are **operator work, by design and by contract**. They are not customer-facing and are not going to be. **[FACT] What Lane 2 contains** (INV Part A.2): the scheduled restore-test, `pct restore`, raw `proxmox-backup-client restore`, raw `restic restore`, the agent's DR bring-up (`--selftest=bring-up -mode dr`), the host-loss plan builder, and the escrow-consume ceremony. Each needs root on the host or a CLI flag; none is reachable from any customer surface. **[FACT] What is proven in Lane 2** (INV Parts G.1, G.3): whole-guest `pct restore` from both tiers is proven with exact mount parity; the unattended restore-test is proven and currently running on both boxes; a corrupted snapshot is proven to fail cleanly. **The DR bring-up path has never been executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by construction** (`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self** (INV Part D1). ### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03) **[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to fire on an interval started at daemon start. It no longer does. The rule is: > Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h). > The tier is **DUE** when **A** exists and **A has not already been proven**. So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather than the process's uptime: | tier rhythm | what is proved, and when | |---|---| | daily (host tier) | yesterday's archive, once a day | | weekly (offsite tier) | last week's archive, once a week | | newborn (no archive yet) | nothing — **UNKNOWN, never a fault** | **The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive age to zero long before it reaches the lag — so the literal reading silently switches restore-testing off for the tier that matters most. What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN). **The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite retention), and falling back to the tier's declared rhythm when history is too short to observe one. ### Why the split is right, stated once **[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about "restore the LXC and re-attach mp8 by durable_id without binding another guest's bootstrap credentials" — a hazard serious enough to have its own runbook page (`runbooks/RUNBOOK-manual-guest-restore.md:45-48`, F-OPS). Putting Lane 2 behind the operator is what lets Lane 1 be a button instead of a procedure. **[FACT] And as of D5 (v0.188.0) the split is real, not just intended.** Until then Lane 1 only *looked* independent: an app's files sat on the customer's drive but could not be brought back without secrets that lived solely in the guest, so the fast customer lane was silently conditioned on the slow operator lane (§7.1 leg 1). The unit now carries those secrets, so **Lane 1 needs the drive and nothing else** for app rebuild (§7.4). The one honest caveat: Lane 1's *file-merge* surface (Tier-2 additive) still reads its destination from the guest's `settings.json`, so that surface remains guest-conditioned — D5 removed the secrets leg, not the living-app leg. --- ## 4. The three-part model (D4) **[DESIGN] Recovery needs three things, and they live in three different places.** Losing one is a different problem from losing another, and the matrix in §8 is organised around that. | Part | Holds | Where | Lost when | |---|---|---|---| | **Recipe** | scaffolding — guest sizing, drive inventory, PVE storage definitions, PBS coordinates, app inventory + storage bindings. **No secrets, no bytes.** | the hub | the hub is lost | | **Escrow** | the host identity key material and the restic repo password | the hub, R-wrapped | the hub is lost, **or** R is lost | | **Bytes** | Tier-1 / Tier-2 / Tier-3 / whole-guest | the tiers | the relevant medium is lost | **[FACT] The Recipe exists and is current.** `dr_recipe` holds 6 rows; demo-felhom's was updated `2026-07-28 17:31:13` and demo-hp's `17:25:31` — i.e. within one report cycle (LIVE, INV Part D2). It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_bindings`. **[FACT] The Escrow exists and is zero-knowledge.** `host_escrow`: 2 rows, `posture` `zero_knowledge`; the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}` (`felhom-agent/internal/escrow/identity.go:26-39`). **[FACT] Three parts of the Recipe are empty or wrong on the live fleet**, and they are exactly the parts a host-loss recovery would read (INV Part D2.3): `hosts.dr_record_json` is `{}` on all three hosts; `host_escrow.directive_json` is `{}` on both escrowed hosts; `dr_recipe.host_half.drives` is `[]` on every customer including two with enrolled data drives; and `dr_recipe.host_half.pbs.namespace` reads `"root"` while the real namespaces are `demo-felhom` / `demo-hp`. → **R-105**, **R-106**. --- ## 5. Encryption policy (D2) **[DESIGN] Encryption follows the boundary, not the tier.** ### On the customer's own premises: plaintext Tier-1 recovery units, Tier-2 cross-drive copies, the app data on the drives, and the local vzdump archives on the host are **unencrypted, deliberately**. The reasoning, recorded so nobody "hardens" this later: 1. **It buys nothing against a real threat.** Someone who can take the second drive can take the first. Local encryption defends against a threat model — physical theft of *only* the backup medium — that does not describe a home server where both media sit in the same box. 2. **It adds a key-loss path that turns a working backup into a brick.** Every local encryption key is one more thing that must survive the disaster it exists for, and the system already has one such dependency it is trying to reduce (§7). 3. **It would break browsing, which is a feature.** FileBrowser and SMB let the household see and use their own files. An encrypted local copy is not browsable, and the „Megosztás" and FileBrowser surfaces are part of the product, not an accident. **[FACT]** The local plaintext posture is real and verifiable: `dir: local` in `/etc/pve/storage.cfg` carries no `encryption-key` (LIVE, both hosts), so the daily whole-guest archive is a plain `.tar.zst` — 5.93 GB on demo-felhom, 1.68 GB on demo-hp (LIVE, INV Part B.3). **That archive is the single highest-value object on the box**: it contains `encryption.key`, the restic `repo_password`, the offbox `ssh_key`, `settings.json` and `controller.yaml` (all under `/var/lib/docker` = `mp0`, `backup=1`). Naming that plainly is part of the policy, not an argument against it. ### Leaving the premises: encrypted, and the provider must not be able to read it **[FACT]** Both offsite tiers encrypt client-side: - **restic (Tier-3):** repo password is a 256-bit hex value generated once on the box and never logged (`internal/backup/offbox.go:392-410`); restic is invoked with `RESTIC_PASSWORD_FILE` (`:608`). - **PBS (whole-guest offsite):** `/etc/pve/storage.cfg` carries a **per-customer** `encryption-key` and the fingerprints differ between demo-felhom and demo-hp (LIVE). The live restore command line observed on demo-felhom carries `--crypt-mode=encrypt` (INV Part A.2.3). Per-tenant encryption is why cross-customer dedup is impossible on the shared datastore — a cost consequence, recorded, not a defect. ### The one exception, stated because it is not covered by either rule **[FACT]** The `.fab` bundle writes `app.yaml` with **decrypted plaintext secrets, deliberately**, and its password is **optional** (`internal/appexport/export.go:484,506-511`; the generated file's first line is literally `# Exported by felhom-controller — plaintext secrets`, `:531`; `Encrypted: req.Password != ""`, `:307`). A `.fab` can also be written to a registered **network** share, because `storageDriveList()` does not filter network paths (`internal/web/handler_export.go:377-387`). This is a portability artifact, not a tier — but it is the one place where a customer action can put every secret of one app onto a NAS in plaintext. Recorded here so the encryption policy is not read as covering it. → **R-108** (same root cause). --- ## 6. The tiers — what each captures The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what they can actually do. ### 6.1 The four tiers, as configured on the live fleet | Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted | |---|---|---|---|---|---| | **Tier-1** recovery unit | `/backups/primary//` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no | | **Tier-2** cross-drive | `/backups/secondary//` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/` + `userdata/` | nightly at W+60m | mirror (rsync) | no | | **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) | | **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** | | **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` — **and the box asks for none** (R-191) | **yes** (per-customer key) | > **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does. **[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered (`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`. **[FACT] What the whole-guest tiers do NOT carry.** `mp8 /mnt/felhom-drives` and `mp9 /etc/felhom-bootstrap` are **host bind mounts** and are out of `vzdump` scope entirely (LIVE `pct config 9201`, both hosts). So the whole-guest tiers carry the guest and **none of the customer's data drives** — 916 GB on demo-felhom, 938 GB on demo-hp. ### 6.2 Coverage per app class — and an unresolved count **[FACT]** Of 53 catalog templates, **52 keep data in Docker named volumes**; exactly **13** carry a `backup:` block, and those 13 are exactly the templates that bind `${HDD_PATH}` / `${USERDATA_PATH}` / `${IMPORT_PATH}` at all; **14** have a database service (INV Part B.1, computed against catalog `4252121`). Applying the classifier's documented two-level default (`internal/appbackup/classify.go`: explicit entry wins; else a `:ro` reader is *excluded*; else a writable bind is *mandatory*): | tier filter | templates with at least one file leg | |---|---| | **Tier-3** (`mandatory` only) | **4** — calibre-web, immich, nextcloud, paperless-ngx | | **Tier-2** (`mandatory + optional`) | **7** — the 4 above plus audiobookshelf, komga, romm | | legacy resolver path (apps with no block) | **0** — no non-block template binds a namespace path | > ### ⚠️ UNRESOLVED — two counts of the same thing disagree > > | source | count | > |---|---| > | C9-F1 Phase 0, as shipped | **A = 9 / B = 43 / C = 1** — `felhom.eu/REPORT.md:17-24`, restated `backlog/OPEN-ITEMS.md:31` | > | INV Part B.1, independent enumeration at catalog `4252121` | **A = 7 / B = 45 / C = 1** | > > Both use the same definition ("templates whose Tier-2 copy can hold a readable file leg"). The > difference is two apps and **neither number is adopted here**. The Phase-0 enumeration is described > in prose but the script is not committed, so the two methods cannot be diffed from the repo. > **This must be resolved before either figure is used to size anything.** **[FACT] The class-B consequence is real regardless of which count is right.** For an app whose data lives entirely in named volumes, the Tier-2 copy holds a full `recovery-unit/` and **no readable leg** — LIVE on demo-felhom, `backups/secondary/bookstack/` and `.../docmost/` contain `recovery-unit` and nothing else, at 156 MB and 86 MB (INV Part B.2). Since v0.183.0 the button refuses **before** stopping the app and names the action that works. ### 6.3 Where capture and restore disagree **[FACT]** Three asymmetries, each source-cited: | tier | captured | read back by that tier's restore | gap | |---|---|---|---| | Tier-1 | unit incl. volume tars + DB dumps | all of it | none | | Tier-2 | unit mirror **+** file legs | `hdd/` and `userdata/` **only** (`tier2_restore.go:101-104`) | **the unit mirror is read by nothing** — `RecoveryUnitPath` resolves to `backups/primary/` (`appbackup/paths.go:46-48`) → **R-102** | | Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay; the unit is **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **no offsite action unpacks the named-volume tars it captures** → **R-107** | **[FACT]** Tier-2's gap is the sharper one because of *when* it bites: Tier-2 exists for the case where the primary drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action. --- ## 7. The recovery chain (D3) — the reason this document exists **[DESIGN] 3-2-1 describes copies. It does not describe recovery.** Three copies on two media with one offsite is a statement about *bytes surviving*. It says nothing about whether the bytes can be turned back into a working system, and a system can satisfy 3-2-1 completely while having no executable recovery route for a given failure. That is not a hypothetical here — §8 has rows where it is the actual state. ### 7.1 The dependency graph > **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below > (**secrets**) is **no longer true for Tier-1 and Tier-2**: the local recovery unit now carries the > portable secret class, so those two tiers need the **drive and nothing else**. The graph and leg 1 > are kept as written because they are the model everything downstream was derived from, and because > leg 2 (**the living app**) is unchanged and still binds Tier-2's file leg and Tier-3. Read §7.4 for > the current chain; the corrected rows are 3, 4 and 9 in §8. **[FACT — as of the R-108 survey, before D5]** Every app-tier restore depends on the guest, for one of two reasons: ``` ┌──────────────────────────────────────────┐ │ the LIVE GUEST │ │ · settings.json (tier-2 destination) │ │ · encryption.key (32 B) │ │ · app.yaml (ENC: under that key) │ │ · the deployed app itself │ └───────────┬──────────────────────────────┘ │ required by ┌──────────────────────────┼──────────────────────────┬─────────────────────┐ │ │ │ │ Tier-1 restore Tier-2 restore Tier-3 reconstitute Tier-3 scratch (rebuild the app) (fill in files) (overwrite + replay) (verification copy) │ │ │ │ needs SECRETS needs the app needs the app needs only the from app.yaml running + the deployed + a DB repo password (restore_unit.go recorded dest service identifiable (also in the guest) :130-132) (tier2_restore.go :114-116) │ │ │ │ └──────────────────────────┴──────────────────────────┴─────────────────────┘ │ ┌───────────▼──────────────────────────────┐ │ WHOLE-GUEST TIER (local vzdump / PBS) │ ← Lane 2, operator only └───────────┬──────────────────────────────┘ │ required by ┌───────────▼──────────────────────────────┐ │ THE HOST (agent.json, bootstrap.json, │ ← in NO backup at all │ vmbr9, sudoers, /etc/pve/priv, WG) │ (INV Part D1) └──────────────────────────────────────────┘ ``` **[FACT] The two legs of the dependency, precisely:** 1. **Secrets.** The recovery unit is secret-free by design — *"It NEVER writes a secret value"* (`internal/backup/recovery_unit.go:73`). `RestoreFromRecoveryUnit` recovers secrets **from the guest, never from the unit** (`restore_unit.go:130-132`), and the policy is stated outright at `:18-22`: *"Regenerate NOTHING… A missing DATA-ENCRYPTING key is FATAL: regenerating it would render the restored data unreadable, so we refuse and tell the operator to do a PBS whole-guest restore."* Those secrets are encrypted under `encryption.key`, a 32-byte file that exists **only inside the guest** (LIVE, both boxes — INV Part C, row 9). 2. **The living app.** Tier-2's restore reads the destination recorded in the guest's `settings.json` (`tier2_restore.go:114-116`), and Tier-3's reconstitution refuses outright when the app is not deployed — *„a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat"* (`offbox_reconstitute.go:198-201`). **[FACT] So all three app tiers are conditioned on the whole-guest tier**, which is Lane 2. The customer's own recovery lane rests on an operator-only tier, and today nothing tells the customer that. — **This sentence was the reason D5 exists. It is now false for Tier-1 and Tier-2; see §7.4.** ### 7.4 The chain AFTER D5 (controller v0.188.0, 2026-07-30) — the current model **[FACT] Tier-1 and Tier-2 no longer depend on the whole-guest tier.** The unit's `compose/app.yaml` (mode 0600) carries the **portable secret class** — every catalog `type: secret` field, i.e. the declared `data_key`s, the 18 DB/root passwords and the internal signing secrets — so neither `encryption.key` nor the guest's `app.yaml` is required to rebuild an app: ``` Tier-1 restore Tier-2 restore Tier-3 reconstitute Tier-3 scratch (rebuild the app) (fill in files) (overwrite + replay) (verification copy) │ │ │ │ needs ONLY THE DRIVE needs the app needs the app needs only the (unit carries the running + the deployed + a DB repo password secrets; guest is recorded dest service identifiable consulted only for the withheld class) └────────── still conditioned on the guest (leg 2, unchanged) ──────┘ │ ✅ INDEPENDENT of the whole-guest tier ``` **[FACT] What still needs the guest, precisely** — so this is not read as more than it is: 1. **Leg 2 is untouched.** Tier-2's *file* restore still reads the destination from the guest's `settings.json` (`tier2_restore.go:114-116`) and Tier-3's reconstitution still refuses when the app is not deployed (`offbox_reconstitute.go:198-201`). D5 removed the **secrets** leg, not the **living-app** leg. So "Tier-2 no longer depends on the guest" is true of the *unit-restore* path it shares with Tier-1, and **false** of its additive file-merge path. 2. **The withheld class.** The 7 `type: password` admin logins and `vaultwarden/ADMIN_TOKEN` are deliberately NOT on the drive; they come from the guest, or are regenerated (O4). Their absence costs a credential reset, never data. 3. **The fail-closed gate remains.** A `data_key` in neither the unit nor the guest still refuses the restore outright. D5 makes it normally present; it does not soften the gate. **[DESIGN] Why plaintext is the right answer here, and what it is coupled to.** Every travelling secret either decrypts data on the *same drive* or authenticates to a container on an internal compose network with no external listener — so possessing it adds nothing to possessing the drive, which is exactly D2's argument for keeping the DATA plaintext. That argument holds **only because** the internet-reachable class is withheld. **The exclusion is what licenses the plaintext; the two must not be relaxed independently.** `stacks.PortableSecretEnvVars` is the single boundary, with a code-level `nonPortableSecrets` register (not a catalog flag — a boundary a catalog push can move is not a boundary, cf. R-97a). **[FACT] Precedence, because two sources now exist.** The **unit wins** over the guest. Not "newest wins": the unit's secrets are captured in the same run as the dumps beside them, so the unit's value matches *the data being restored*, whereas the guest's is merely the most recent. Preferring the guest is the data-loss bug — a rotated data key does not decrypt data encrypted with the old one, and a rotated DB password does not match the hash inside the restored data directory. **[FACT] Proven live 2026-07-30**, on a scratch drill guest, through the real endpoints: AdventureLog (`SECRET_KEY` data_key + `DB_PASSWORD`) restored with the guest's `app.yaml` **moved aside** — `secrets recovered=2/2`, 27.6 s — and then **the application itself read the seeded customer row over TCP with its own credential**, which is the observable that matters (a restore that returns success onto unreadable data is a failure that looks like a pass). The discriminator held: the pre-backup row came back, a row added *after* the backup was gone, so the volume tar was genuinely restored. There was **no `.sql` dump** in that unit, so the DB came back from the volume tar — the exact case where a regenerated password would have failed. Separately, Grafana's `type: password` admin login was proven **withheld**: the sentinel value was live in the container and encrypted in the guest, and appeared in **0 files** anywhere under the backup namespace. Full record: `audits/D5-drive-alone-restore-2026-07-30.md`. ### 7.2 A tier whose prerequisites cannot be met in the failure it exists for **[FACT]** Two instances, both current: - **Tier-2 vs primary-drive loss.** Tier-2's stated purpose is surviving the loss of the primary drive. In that failure the primary recovery unit is gone; the surviving mirror on the second drive is `backups/secondary//recovery-unit/`, which **no code path reads** (§6.3). For the 45-or-43 class-B apps the restore is a guaranteed no-op in exactly its designed scenario. → **R-102** - **Tier-3 vs guest loss.** Tier-3 holds the volume tars and the DB dump. Reconstitution requires the app to be deployed and skips the unit; the tars are unpacked only by the Tier-1 path, which requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest. → **R-107** ### 7.3 ~~What D5 would change — and why it was blocked~~ — **D5 SHIPPED 2026-07-30 (v0.188.0)** > **This section is history.** D5 is implemented and proven live; the current chain is **§7.4**. Kept > because the reasoning below is why the precondition was required, and because the last paragraph > (`.fab` / R-126) is still open and still not part of D5. > > **One correction to the target as stated below.** It assumed the class that must travel is the > `data_key`-flagged secrets. Part 0 of the D5 task tested that and it did not survive: the flag is > unreliable (four encryption keys the catalog itself labels as such are unflagged → **R-127**), and a > DB password is **not resettable in practice** — `POSTGRES_PASSWORD` is ignored once PGDATA is > non-empty, so a regenerated value leaves the app unable to authenticate against its own restored rows > while the dump replay still reports success. The shipped boundary is therefore *all* `type: secret` > minus the register, not `data_key` alone. **[DESIGN, TARGET — SUPERSEDED BY §7.4]** The intended fix is to make app secrets travel with the **local** recovery unit, so Tier-1 and Tier-2 restore work **without the guest and without R**. Offsite already encrypts everything, so secrets travelling offsite would be covered by the escrowed repo password. R would then be required for **offsite recovery and host identity only** — losing R would cost the offsite route, not local recovery. **The premise D5 rests on is now established.** §2 of the task that produced this document required it to be proven, not assumed: the backup tree must be unreachable from every browsing, download and export surface. When this document was written it was **not** — the FileBrowser network-share bind reached it. **R-108 closed that on 2026-07-30** (controller v0.187.0) by refusing app namespaces on network storage, so no `backups/` tree can exist under the share-root bind; every other surface was already clear (§10.1's table). **D5's precondition is therefore MET and D5 may be adopted.** **Still true, and not part of D5's precondition:** a `.fab` bundle carries plaintext secrets by design with an optional password, and `storageDriveList()` does not filter network paths, so a bundle can be **exported onto** a NAS (§5, → **R-126**). That is an export destination the customer chooses explicitly, not a browsing surface reaching a backup tree, and it is unchanged by D5 — D5 moves secrets into the local recovery unit, not into `.fab`. It is tracked separately rather than folded in. Until D5 is actually implemented, §7.1's chain stands as the model. --- ### 7.5 Where a recovery unit is RETAINED — and the size bound it puts on Lane 1 Added 2026-08-02 from `audits/SPIKE-recovery-unit-space-2026-08-02.md` and `audits/CAMPAIGN-10-closeout-2026-08-02.md`. §6.1 says a Tier-1 unit lives *"on the app's own drive"*, which is true and is the whole story for a drive-resident app. **It is not the whole story for an app with no data drive, and that case was undocumented until now.** **The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go:245-255`) returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what `internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit after it is copied onward: the only prune is F5 (`backup.go:1053-1112`), which removes residue on an **old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy indefinitely. **What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`): `compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains `mp8` / `HDD_PATH` userdata** — a photo library on a 1 TB drive is not in a unit, and cannot make one overflow. **The resulting mismatch, on a default appliance:** | | size | holds | |---|---|---| | `mp0` `/var/lib/docker` | **50 G** | every app's live volumes — the thing a unit copies | | `mp1` `/mnt/sys_drive` | **20 G** | the retained units of every **driveless** app | A DB-backed app's unit is up to **~2×** its data, because it carries the volume tar **and** the SQL dump (measured: 21.1 GB of app data → a **40.2 GB** unit). A file-only app's is **1.00×** (measured: homebox 2305 MB → 2305 MB). `--sysdata-grow` (`felhom-agent/cmd/felhom-agent/main.go:178`) defaults to **0**, is not derived from the physical drive, and no observed box passes it — demo-hp's production guest 9201 ships `mp0 50G / mp1 20G`. **`mp1` gates the entire app-data chain, not just Tier 1.** Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too, so a unit that cannot be written to `mp1` leaves Tier-2 and Tier-3 with nothing to copy. **The bound this places on Lane 1 (§3).** Lane 1's promise is that a customer can restore an app from the drive alone, without the operator and without the guest — D5 (§7.4) made that true by putting the portable secrets in the unit. **That independence is bounded by app size**, and the bound is: > On a default appliance, an app can be restored by Lane 1 only while its recovery unit still fits the > retained space — **≈ 19 GB of app data for a file-only app, ≈ 10 GB for a DB-backed one.** > Past that the unit cannot be captured, and the app falls back to Lane 2's operator-driven > whole-guest route. **AS OF 2026-08-02 SOMETHING NOW WARNS, AND THE ALERTING IS PART OF THIS CONTRACT (R-167 / R-158, decision D-c; controller v0.191.x + hub v0.89.0).** The last sentence of this section used to end "nothing warns when an app crosses the line". Two signals now exist and both are PROVEN-LIVE: - **To the CUSTOMER, before anything fails** — `internal/fillwatch` warns per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) on **whichever trips first, used ≥ 85% or free < 5 GiB**, critical at 95% / 2 GiB, clearing at 75% / 7 GiB. **Two terms, because a percentage alone lies at both ends of the range this section itself documents:** 85% of a 20 G `mp1` leaves 3 G — less than one DB-backed app's unit — while 85% of a 4 TB drive leaves 600 G. It watches the app-data volume, the system-data volume **and** every registered drive, which the previous `health_degraded` signal did not. Edge-triggered against persisted state; the hub owns cooldown. - **To the OPERATOR, when a capture actually fails** — `recovery_unit_capture_failed`, per app, with the target filesystem's used/free bytes at the moment of failure, so the *why* needs no login. It is **operator-tier** (`notify.operatorOnlyEvents`) and deliberately not `backup_failed`: a customer can take no action on a capture failure. ### 7.5.1 — THE CEILING THIS SECTION DESCRIBES HAS BEEN REMOVED (2026-08-03, R-165 / decision D-a) **Everything above describes the SPLIT layout, which is now the legacy shape.** A golden built by `build-golden.sh` **v3.0.0** ships **one** data volume; `mp1` does not exist. Both consumer paths are binds of subdirectories of it (variant **V-c**): ``` mp0 -> /var/lib/felhom ├─ docker/ --bind--> /var/lib/docker └─ sys_drive/ --bind--> /mnt/sys_drive ``` **So the size bound below no longer applies to a box built from that golden.** A driveless app's recovery unit is limited by the box's actual free space, not by a partition set at build time. The mismatch table above (`mp0` 50 G vs `mp1` 20 G) describes what a merged box no longer has. **R-175, fixed here rather than left standing.** The bound below was stated as the fleet's and was **one box's**: it is derived from `mp1 = 20 G`, which is demo-hp exactly and never was demo-felhom (`mp0 200G / mp1 50G`, where the same arithmetic gives ≈ 49 GB / ≈ 24 GB), nor the golden (`16 G / 8 G` before provision grew them). **Read it as a function of `mp1`, and only for a box still on the split layout.** Measured: `audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1. **What replaced the partition's second job — the reserve.** `mp1` was also a BULKHEAD: an overflow was refused per app with the last good unit byte-identical, and it **could not reach `/var/lib/docker`**, because that was a different filesystem. On a merged box it can. Decision **B2**, shipped in controller **v0.192.0**, is that bulkhead made deliberate — a two-term reserve (97% used or 1 GiB free) in `internal/fillwatch`'s shape, sitting beyond its critical band so the customer is always warned first. It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could only destroy a different app's only local copy. **WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).** The two are deliberately different mechanisms, because conflating them is how seven failures went missing on 2026-08-03 without leaving a trace. | | Record | Notification | |---|---|---| | what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN | | when | every time, unconditionally | at the end of a run, **only if something failed** | | gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown | | where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox | - **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's REPORT freshness, independent of any mail the box sends. That check is load-bearing for this design; weakening it re-opens a silent-failure path. - **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it. Deciding not to tell someone is itself an event worth recording. - **Deliberate skips are not failures** and never appear in the digest — a disconnected or decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the operator stops reading. - **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within the hour, because someone pressing the button is actively trying to get a backup. The periodic capture sweep is capped by the ordinary hourly cooldown. **THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a **per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the volume dump and the recovery-unit capture**. Those three write under one per-app root (`backups/primary/`), which is what makes one verdict able to cover them honestly. - **What it guarantees.** A refused app has **nothing written for it in that run**, its previous unit is **byte-identical**, it is **not stopped**, nothing anywhere is deleted, and the operator gets **exactly one** alert naming the app, the term that bound and the disk figures. - **Two terms, two questions.** *Headroom*: is the filesystem already below the reserve? *Size*: would THIS app's write take it below? The size estimate is the app's previous `.sql` + `.tar` on disk; with no history the decision degrades to headroom alone, deliberately — otherwise the first backup is the one that can never happen. - **Why it is decided lazily and not once per run.** Space changes during a run: app A's dump can put app B under the reserve, so a verdict taken at run start reads a disk that no longer exists. - **Why it is never re-decided between an app's own legs.** That is precisely the shape v0.192.0 had — the two dump legs unguarded and only the capture refused — under which the reserve was consumed by the very write it exists to bound, and the refusal's *"the previous unit is untouched"* was measured false. Proven live on demo-hp 2026-08-03 (R-181), fixed the same day, and re-proven by filling the box for each of the two terms. - **It sits ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it would already have bounced the app it is refusing to back up. **Status caveat, deliberately explicit:** every box in the field that has not been reinstalled is still on the split layout and everything above still describes them exactly. This subsection describes what a box built from golden ≥ 0.192.0 gets. Both demo boxes were reinstalled from it on 2026-08-03 (R-178). Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling** (**R-163**) — this section states the constraint, not a number. And **the same-device placement is intended, not a defect**: a driveless app's unit sits on the same SSD as its volumes, but Tier-2's cross-drive mirror, the whole-guest tier (which covers `rootfs + mp0 + mp1`, §6.1) and Tier-3 all cover device loss. What was missing was that this case existed at all, and that nothing warns when an app crosses the line — **R-158**. ## 8. The failure → recovery matrix **This is the core artifact.** One row per failure. It is authoritative for recovery routes; `00-capability-map.md` stays authoritative for per-capability status. **How to read the numbers.** - **RTO** — **only measured durations** from INV Part F appear here. A blank cell means *nobody has ever measured it*, and a blank is a finding, not an omission. - **RPO** — **no RPO has ever been measured from an incident.** These cells carry the **configured cadence that bounds RPO**, read live from the box, labelled `(cadence)`. A blank means no cadence governs the row. - **Status** — `PROVEN` (live, cited) · `PARTIAL` (some legs proven) · `IMPLEMENTED` (code + tests, never exercised) · `NONE` (no route exists). | # | Failure | What survives | Recovery route | Invocable by | RTO (measured) | RPO (cadence) | Status | Evidence | |---|---|---|---|---|---|---|---|---| | 1 | **Customer deletes files** | everything else | „Fájlok visszaállítása" — Tier-2 additive merge | **customer** | **39 s** (6 files) | 24 h | **PROVEN** | CAMPAIGN-9 A1 — byte-identical returns, both non-destruction promises kept, bytes re-served by paperless's own API | | 2 | **An app's data directory is destroyed** | the guest, the other tiers | same route | **customer** | **46 s** (43 files) | 24 h | **PROVEN** | CAMPAIGN-9 A3 — loss verified by a positive observable (doc download 200 → 404), then 16/16 documents usable | | 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar | | 3c | *same, with the GUEST GONE (secrets unavailable)* | the drive | Tier-1 unit restore — **the unit carries the portable secrets** | **customer** | **27.6 s** | 24 h | **PROVEN** | D5, §7.4. Before v0.188.0 this row was **NONE**: the data-key gate refused and the customer's own copy of their own data was not a recovery | | 3b | *same, for a class-B app via Tier-2* | — | **no route** — Tier-2 never reads the unit mirror | — | | | **NONE** | §6.3; **R-102** | | 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB; **the volume tars in either copy are unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102**, **R-107** | | 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go:359-393` | | 6 | **Guest lost or corrupted** | the host, both whole-guest tiers, the data drives (they are host binds) | `pct restore` from `local:` or `felhom-pbs:` | **operator** (SSH) | **84–112 s** local · **1101 s** PBS (both = restore-test into a scratch guest, boot + verify + teardown) | 24 h local · 7 d offsite | **PROVEN** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C (exact mount parity, `unprivileged: 1` preserved); LIVE restore-tests on both boxes this session | | 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human | | 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) | | 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) | | 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) | | 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) | | 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) | | 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D | | 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) | | 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix | | 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** | ### 8.1 The blank cells, listed explicitly Per the rule that a blank is a finding, here they are: | row | blank | why | |---|---|---| | 3b | RTO, RPO | no route exists to time | | 4 | RTO | no drive-loss recovery has ever been timed | | 5 | RTO | never timed; the rebuild is a normal Tier-2 run | | 6 | — | RTO present, but it is a **restore into a scratch guest on the same host**; a restore *to a different host* has never been timed | | 8 | RTO | a host has never been rebuilt as itself (INV Part D1) | | 9 | RTO | the composed whole-box path has never been run | | 10 | RTO | no ransomware-shaped recovery has ever been run | | 11 | RTO | a hub restore has never been performed | | 12 | RTO, RPO | no provider-loss recovery has ever been run | | 13 | RTO, RPO | not a timed recovery; a capability loss | | 14 | RTO | the break-glass path is proven but was never timed | | 15 | RTO | the manual unlock was performed but not timed | **Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS incremental; `.fab` import wall-clock; and time-to-first-byte for a customer restore over a home uplink — **no customer has ever driven a restore**. --- ## 9. What the model implies for the tiers (recorded, not new design) **[DESIGN]** Three consequences follow from §3–§7 and are stated so they are not re-derived: 1. **Tier-2 is a *drive-loss* tier, not a second chance at Tier-1.** Its job is to survive one drive dying. That is why it mirrors the unit at all — and why the unit mirror being unreadable (§6.3) defeats the tier rather than degrading it. 2. **Tier-3 is a *premises-loss* tier.** It is the only copy that survives fire, theft and ransomware-with-host-access, which is why it is the only tier whose credential exposure is ranked as the largest open data risk (R-95). 3. **The whole-guest tiers are the *system* tier, and they sit under everything else** (§7.1). Treating them as "one more copy" is precisely the 3-2-1 framing this document rejects. --- ## 10. Known gaps Every divergence between the model above and the system as it is, each with an ID. ### 10.1 ~~D5 is BLOCKED~~ — precondition CLOSED by R-108 (v0.187.0); **D5 ITSELF SHIPPED (v0.188.0)** > **D5 IS DONE — 2026-07-30, controller v0.188.0.** The precondition below was met by R-108, and D5 was > then implemented and proven live the same day: Tier-1/Tier-2 no longer depend on the whole-guest tier, > and a customer needs **the drive and nothing else**. The current recovery chain is **§7.4**; the > corrected matrix rows are 3, 3c and 13. This section's remaining value is the *precondition* analysis, > which is why the table below is kept. > **D5's PRECONDITION IS MET.** An app's data namespace can no longer be placed on network storage, so > no `backups/` tree can exist inside FileBrowser's share-root bind, and the browsing surface therefore > cannot reach the backup tree on ANY storage class. Every other read surface in the table below was > already NO. **D5 may be adopted** — nothing in this section blocks it. > > **The fix inverted the obvious one, and that is the durable lesson here.** The share-root bind was > not narrowed, because it **cannot** be: (a) the `:rslave` share-ROOT bind is load-bearing — a > Phase-0 probe (2026-07-22) proved an in-container access through it wakes the idle automount > trigger, so narrowing it breaks NAS access itself; (b) there is no `userdata/` layer to scope to, > since apps on a share store at `/`; and (c) creating one would write Felhom's directory > convention onto a customer's own NAS, which R-67 forbids outright. The browsing surface being > immovable is precisely *why* the backup tree must never be placed under it. Tier 2 had already > reached the same conclusion for its own targets (`F-6C-1`); R-108 closes the PRIMARY namespace, > which was the last remaining route. > > **Operator ruling 2026-07-30: refuse the placement, keep the browse bind.** `RefuseAsAppNamespace` > (`internal/settings/settings.go`) is the single predicate; all placement surfaces consult it. It > **fails closed** — `/mnt/felhom-drives` holds both storage kinds in-guest, so a path prefix cannot > classify and `Kind` exists only on a REGISTERED path; an unregistered path under that root is > therefore un-classifiable and is refused rather than assumed to be a drive. > > **Nothing was stranded:** zero apps on network storage across all six hub customers including Peti. > R-67's browse capability is byte-for-byte unchanged (verified by diffing demo-hp's generated > compose before and after the deploy). This also **supersedes** the controller README's "NAS backup > locality — decision A" (v0.118.0), which deliberately kept a NAS-resident app's Tier-1 artifacts on > the NAS: that case can no longer arise. > > Evidence: `audits/R108-network-app-namespace-2026-07-30.md`. The pre-fix analysis below is retained > verbatim as the record of what was wrong. #### 10.1 (historical) The exposure as it stood before v0.187.0 **[FACT] The verification and its result.** Every surface that can read a file was checked: | surface | can it reach `backups/`? | evidence | |---|---|---| | SMB share creation | **NO** | `sharingResolvePath` (`internal/web/sharing_handlers.go:52-81`) resolves symlinks *before* containment, then refuses any path within `SharingDeniedRoots(root)`; that set covers `/backups` **and** the legacy `/felhom-data` + `/felhom-data/backups` (`internal/stacks/samba.go:46-65`, derived from `ProtectedHDDPaths`, `delete.go:59-78`) — **both namespace shapes** | | SMB browse (folder picker) | **NO** | same deny set applied per child (`sharing_handlers.go:521-537`) | | SMB `ensureImportShare` (the store-direct bypass) | **NO** | writes one controller-generated constant, `GetImportRoot()` = `/userdata/import` (`sharing_handlers.go:564-587`) | | FileBrowser — **local drives** | **NO** | the bind is `appbackup.UserdataDir(sp.Path)` only, and the comment says why (`internal/web/handlers.go:2450-2460`) | | **FileBrowser — network shares** | ~~**YES**~~ → **NO** (R-108, v0.187.0) | the bind is still the share **ROOT** (`- %s:/srv/%s:rslave`, `handlers.go:2437`) and deliberately so — but **no app namespace, hence no `backups/` tree, can exist on a share**, so the root bind reaches only the customer's own files. The reachability is closed at the PLACEMENT, not at the bind | | `.fab` import path validation | **NO** | confined to `/exports` (`handler_export.go:400-408`, `estimate.go:215-217`) | | `.fab` browser download | **NO** | name-pattern + parent-must-be-the-staging-dir double guard (`handler_export_download.go:36-45,120-140`) | | `/api/debug/*` | **NO** | no file-serving branch (`handler_debug.go:46-92`) | | `http.ServeFile` (3 sites) | **NO** | assets only, `filepath.Base`-normalised (`server.go:720-771`) | | log bundles | **NO producer found** | no filesystem-walk bundle producer exists on the box; searched `felhom-controller/internal`, `felhom-agent/internal`, `cmd/` | | registering the backup dir as a drive | **NO** | the manual add requires `system.IsMountPoint(path)` (`handlers.go:2091-2095`); `/backups` is not a mount point | **The exposure, end to end.** All six links are source-cited and the precondition is live today: 1. A NAS share is registerable as a storage path and lands `Schedulable: true` — **LIVE on demo-hp**: `{"path":"/mnt/felhom-drives/Felhom-Share","kind":"network","schedulable":true}`. 2. `GetSchedulableStoragePaths()` has **no `IsNetwork()` filter** (`internal/settings/settings.go:904-914`), so that share appears in the **deploy** dropdown (`handlers.go:462-473`). 3. The **per-app migrate** target list filters only current / decommissioned / disconnected / schedulable — also no network filter (`handlers.go:674-679`). 4. `handleStorageMigrateApp` does **not** call `refuseNetworkLifecycle`, unlike its whole-namespace sibling which does (`storage_handlers.go:397` vs `:410-424`), and `startMigration` has no guard either (`internal/stacks/migrate.go:214-255`). 5. With `HDD_PATH` on the share, `namespaceRoot()` returns it as-is under Model A (`internal/backup/backup.go:262-263`), so the app's Tier-1 unit is written to `/backups/primary//compose/app.yaml`. 6. FileBrowser binds that share at its **root** and serves it with `download: true` (`internal/infra/infra.go:326`). **Observed live on demo-hp, in the generated compose — the asymmetry is visible, not inferred:** ``` - /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb <- drive: userdata-scoped - /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave <- network: ROOT-bound - /mnt/sys_drive/felhom-data/userdata/import:/srv/beolvasas ``` **Today this is not a secret leak**, because the unit's `app.yaml` is secret-stripped (`recovery_unit.go:73`). **D5 would make it one.** That is exactly the test §2 set, and D5 therefore does **not** hold as written. → **R-108** > **CLOSED (v0.187.0).** Links 2, 3 and 4 of the chain above are now guarded, and a **fifth** surface > the chain did not list was found and guarded too: `handleStorageDecommission` mode=`migrate` checked > only `req.Where` (the SOURCE) via `refuseNetworkLifecycle`, so a whole namespace could be > decommissioned ONTO a NAS. Link 2's framing also understated the problem — the deploy **dropdown** is > only a UI list; the boundary is the deploy **POST** (`internal/api/router.go`), which accepts any > caller-supplied `HDD_PATH` and whose only other validation is `os.Stat` existence > (`internal/stacks/deploy.go`). Filtering the list alone would have left the surface open. Links 5 and > 6 are unchanged and still true — they simply can no longer be reached. ### 10.2 The gap register | ID | Gap | Consequence | |---|---|---| | **R-102** | Tier-2 writes a full `recovery-unit/` mirror on every run and **no code path reads it** | Tier-2 is defeated in the drive-loss scenario it exists for (§7.2). Was C9-F4 | | **R-103** | The Tier-2 no-coverage refusal **names** the working action but does not **route** to it | 45-or-43 of 53 apps dead-end at a message. Was C9-F1b | | **R-104** | An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach, reported as *„ismeretlen okból"* | the offsite tier stays dead until a human unlocks. Was C9-F3 | | **R-105** | Three hub-held DR records are empty on the whole live fleet: `hosts.dr_record_json`, `host_escrow.directive_json`, `dr_recipe.host_half.drives` | the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field | | **R-106** | `dr_recipe.host_half.pbs.namespace` records `"root"` on every box | the recorded restore coordinate is wrong; real namespaces are per-customer | | **R-107** | No offsite action unpacks the named-volume tars Tier-3 captures on every run | offsite alone cannot rebuild a named-volume app (§7.2) | | ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` | | **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) | | R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) | | ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) | | R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof | ### 10.3 Divergences that are documented elsewhere and are not re-opened here **[FACT]** `06-offsite-connectivity.md:19-21` describes the operator's public edge as a Cloudflare Tunnel and states DooPlex has no public IP. Live DNS resolves `hub.felhom.eu` through a no-ip DynDNS CNAME straight to DooPlex's own public address (INV Part E.3). That is a topology-document issue, not a recovery-model one; recorded so the discrepancy is not lost. --- ## 11. Open decisions — for the operator **Recorded, deliberately not answered.** **A. Escrow custody.** Split custody (R **or** an offline operator key) versus a 2-of-3 threshold across customer / hub / box drives. *Recommendation on record:* **split custody, operator key held offline and never in the hub.* Note the constraint from §2: the moment the operator key lives in the hub, "a compromised hub yields blobs nobody can open" stops being true. **B. Lost-R policy.** Under split custody the operator **can** recover. Is that the stated policy — and if so, what identity check gates it? Today matrix row 13 has no route at all, and the customer is not told that losing R costs them the host-loss route. **C. RTO / RPO targets per scenario.** **None have ever been stated.** Without them §8 cannot judge whether, for example, offsite-only recovery is acceptable for drive loss, or whether the 7-day PBS cadence is adequate for row 8. The measured column is the input; the target column does not exist. **D. Hetzner as a single failure domain.** restic (Storage Box `u629488`, sub-accounts per customer) and PBS (Cloud server ep0 + a Cloud Volume) are both Hetzner. Whether they share an **account, login and payment method** is unverified (INV Unknown U-1) — the question was deliberately not answered by calling the provider API with the production token. Accept explicitly, or mitigate. **E. `local` vzdump shares a physical device with the guest it backs up.** LIVE on both hosts: `/var/lib/vz` (the archive target) and `local-lvm` (the guest's rootfs and both `backup=1` mountpoints) are both on `/dev/sda3` → VG `pve`. The tier therefore protects against **corruption and operator error only**, never against disk failure. Accept and name it honestly in the customer- facing description, or move the target. **F (added by §10.1, not in the original list). — ANSWERED 2026-07-30.** D5 could not be adopted until R-108 closed. Closing R-108 *was* the intended path, and it is done (controller v0.187.0): the operator ruled to refuse app namespaces on network storage rather than narrow the browsing surface, because the share-root bind is load-bearing and cannot be scoped. **D5 is no longer blocked.** Whether to now *implement* D5 remains an open scheduling decision, not a blocked one. --- ## 12. Evidence index | Claim | Grade | Source | |---|---|---| | Tier-2 file restore, gap-fill and after total loss | **PROVEN-LIVE** | CAMPAIGN-9 A1/A3 | | Tier-2 refuses without an outage for a no-coverage app | **PROVEN-LIVE** | v0.183.0 replay, `felhom.eu/REPORT.md:60` | | Tier-1 unit restore **executes** | **PROVEN-LIVE** | CAMPAIGN-9 A2 | | Tier-1 **content recovery after loss** | **UNPROVEN** | `CAMPAIGN-9…:825-827` | | restic restore of app data (bytes) | **PROVEN-LIVE** | CAMPAIGN-8 R-87, 155.5 MB, 6/7 byte-identical | | offsite reconstitution of a DB-indexed app | **PROVEN-LIVE** | destructive immich drill 2026-07-20, `00-capability-map.md:66,75` | | offsite **place-to-live** as a distinct action | **UNPROVEN** | `CAMPAIGN-8…:520` | | shares restore (files + definitions + credential) | **PROVEN-LIVE** | 2026-07-18, `00-capability-map.md:96` | | `.fab` drive-to-drive round trip | **PROVEN-LIVE** | CAMPAIGN-6D P-FAB, 1.7 GB | | `.fab` **browser upload** leg | **UNPROVEN** | `00-capability-map.md:67` | | whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C | | corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 | | the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 | | unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) | | guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s | | quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL | | break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` | | agent DR bring-up (`ModeDRGuestLoss`) | **NEVER EXECUTED** | `CAMPAIGN-8…:522` | | host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` | | host rebuilt as its former self | **NEVER DONE** | INV Part D1 | | escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` | | hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 | | a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` | --- ## 13. What this document deliberately does not do - It does not restate capability status — §8 cites the map, the map cites §8. - It does not resolve the 7/53-vs-9/43/1 count (§6.2). Both are on the record; neither is adopted. - It does not estimate a single RTO or RPO. Every blank in §8 is a real gap. - It does not answer §11. Those are the operator's. - It does not claim ratification.