admin
0dbd954fec
gates / gates (push) Successful in 7s
- OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and its dependency stated. Header restates that R-202, the 1.2 GB ciphertext deletion and R-198's still-unit-proven retention all REMAIN OPEN. - capability map: the recovery row keeps its 'with a person present' qualifier, names which crutch remains, and cites the three now gone. - 07-backup-architecture: new 7.0 - what a customer can and cannot do ALONE, the four steps in a table with status. This is the section a future reader will use to answer that question. - CONTEXT: standing ruling S-32, superseding S-31 steps 2-5. - STATUS: rewritten to one screen per its own header; removes a corrupted half-overwritten section left from the drill session. - ROADMAP: R-196 and R-204 collapsed.
967 lines
75 KiB
Markdown
967 lines
75 KiB
Markdown
# 07 — The recovery model
|
||
|
||
> | | |
|
||
> |---|---|
|
||
> | **Status** | **NOT RATIFIED.** Ratification is Viktor's review, not an editor's. |
|
||
> | **Written** | 2026-07-28 (full rewrite; supersedes the 2026-07-14 DRAFT entirely) |
|
||
> | **Verified against** | controller **v0.183.0** · agent **v0.110.0** · hub **v0.80.0** · catalog `4252121` · repo HEADs `felhom.eu ff050cf`, `felhom-controller fd50a73`, `felhom-agent d5c7691` |
|
||
> | **Live fleet at verification** | demo-felhom + demo-hp, both guest 9201, both on the versions above |
|
||
> | **Freshness** | **CURRENT** as of 2026-07-28. Per standing ruling **S-2**, mark this STALE the moment it is more than a few trains behind. The document it replaces was DRAFT since 2026-07-14, verified against controller v0.132.0 — **fifty-one versions stale** — and was cited as authoritative throughout that time. |
|
||
>
|
||
> **How to read this document.** Two kinds of statement appear, and they are always labelled:
|
||
>
|
||
> - **[DESIGN]** — a decision taken in the 2026-07-28 architecture discussion (D1–D6, §3–§5, §9).
|
||
> Not derived from code; the code may not implement it yet. Where it does not, §10 says so.
|
||
> - **[FACT]** — an observed property, carrying a `file:line`, a live command output, or a citation
|
||
> to `_recovery-inventory-2026-07-28.md` (below: **INV**).
|
||
>
|
||
> Where the model is silent, this document says **OPEN** rather than filling the gap.
|
||
>
|
||
> **Primary input:** `_recovery-inventory-2026-07-28.md` (read-only inventory, 2026-07-28) — cited
|
||
> throughout as **INV Part n**. Every number in §6, §8 and §11 traces back to it.
|
||
|
||
---
|
||
|
||
## 1. Purpose and scope
|
||
|
||
This document describes **how a Felhom customer gets their system back**, and who can do it.
|
||
|
||
It replaces a document that described **where copies are written**. That was the wrong frame, and
|
||
§7 explains why: a set of copies is not a set of recovery routes, and the previous document's tier
|
||
table could be entirely satisfied while a real recovery was impossible.
|
||
|
||
**In scope:** the recovery model — trust boundary, lanes, the three-part model, the tiers as
|
||
*inputs to recovery*, the dependency chain between them, the failure→recovery matrix, and the
|
||
encryption policy.
|
||
|
||
**Out of scope, deliberately:** implementation specs (they live in task specs), the capture-set
|
||
algorithm (`internal/appbackup/captureset.go` and its tests are the source of truth), and per-tier
|
||
operational runbooks (`documentation/runbooks/`).
|
||
|
||
**Authority split.** This document is authoritative for the **failure→recovery matrix** (§8). The
|
||
capability map (`00-capability-map.md`) stays authoritative for **per-capability status**. Neither
|
||
restates the other; §8 rows are cited from the map, not copied into it.
|
||
|
||
---
|
||
|
||
## 2. The trust model
|
||
|
||
**[DESIGN] The operator holds root SSH on every box.** That is a fact of the product — the agent is
|
||
operator-tier, the host is operator-managed, and break-glass exists precisely so the operator can
|
||
get in when nothing else works. **"The operator cannot read customer data" was never the security
|
||
property**, and no part of this document may be read as claiming it.
|
||
|
||
**[FACT]** The mechanics that make this concrete:
|
||
|
||
- The operator's OOB SSH public key is pushed to every box from the hub
|
||
(`hub_settings.oob_operator_ssh_pubkey` → `/var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op`,
|
||
93 bytes, LIVE on both hosts — INV Part C, row 21).
|
||
- The break-glass `root@pam` console password for every host is stored in the hub and retrievable
|
||
with the operator's global key (`documentation/runbooks/break-glass.md:42-48`; LIVE:
|
||
`host_recovery` holds 3 rows — INV Part D2.1).
|
||
- Guest data is reachable from the host by definition: `pct exec`, and the data drives are bind
|
||
mounts on the host (`mp8 /mnt/felhom-drives`, LIVE `pct config 9201` on both hosts).
|
||
|
||
**[DESIGN] What R (the customer's recovery code) does provide, and must keep providing: the hub
|
||
alone is not enough.** A compromised hub yields blobs nobody can open. This property holds **only
|
||
while the operator's own key is not stored in the hub** — which is why §11-A is an open decision and
|
||
why its recommendation on record is "operator key held offline and never in the hub".
|
||
|
||
**[FACT]** The escrow is genuinely zero-knowledge today: `host_escrow` rows carry
|
||
`posture = zero_knowledge`, a 383-byte blob and a 572-byte identity blob, and the hub holds only a
|
||
`restic_pw_sha256` **hash** beside them (LIVE, both hosts — INV Part D2.1). R exists in **zero**
|
||
system copies by design (INV Part C, row 24).
|
||
|
||
So the honest statement of the property is:
|
||
|
||
> **The hub cannot open the escrow. The operator can reach a live box. Neither fact substitutes for
|
||
> the other, and R is what keeps the first true.**
|
||
|
||
---
|
||
|
||
## 3. The two lanes (D1)
|
||
|
||
**[DESIGN] Recovery is split into two lanes with different owners. This is a product decision, not a
|
||
limitation.**
|
||
|
||
### Lane 1 — the customer, unassisted: files and app data
|
||
|
||
The customer owns their data and can get it back themselves, through the „Visszaállítás" surfaces,
|
||
with nothing but their dashboard password. No operator, no ticket, no scheduling.
|
||
|
||
**[FACT] What Lane 1 contains today** (INV Part A.1 — seven paths, all behind the controller's
|
||
`RequireAuth` gate, which is the **customer-owned** password: `internal/web/auth.go:34-35` puts
|
||
`settings.json → password_hash` ahead of the operator-provisioned `controller.yaml` value):
|
||
|
||
| # | Surface (HU) | Endpoint | Semantics |
|
||
|---|---|---|---|
|
||
| 1 | „Visszaállítás indítása" | `POST /backup/restore` | destructive — rebuilds the app from its Tier-1 unit |
|
||
| 2 | „Fájlok visszaállítása" | `POST /backup/tier2/restore` | additive, missing-only |
|
||
| 3 | „Visszaállítás a távoli tárolóból" | `POST /backup/offbox/restore` | non-destructive, to a verification copy |
|
||
| 4 | „Helyreállítás az élő adatok közé" | `POST /backup/offbox/place` | additive, missing-only |
|
||
| 5 | „Teljes visszaállítás (fájlok + adatbázis)" | `POST /backup/offbox/reconstitute` | destructive to files + DB, never deleting |
|
||
| 6 | „Megosztások visszaállítása" + place | `POST /backup/shares/{restore,place}` | additive |
|
||
| 7 | `.fab` import | `POST` → `apiImportStart` | destructive re-import of one app |
|
||
|
||
**[FACT] What is proven in Lane 1** (INV Part G.1): paths 2 and 6 are proven live end-to-end;
|
||
path 5 is proven live through a destructive drill (photos deleted, trash emptied, restored, timeline
|
||
confirmed); path 3 is proven for **bytes** only; path 7 is proven for the drive-to-drive circle but
|
||
its browser-upload leg is not; path 1 is proven to **execute** but its **content recovery after real
|
||
loss has never been demonstrated** — the single most valuable unproven item in the system
|
||
(`audits/CAMPAIGN-9-restore-proof-2026-07-28.md:825-827`); path 4 has never been exercised as a
|
||
distinct action (`audits/CAMPAIGN-8-backup-restore-2026-07-27.md:520`).
|
||
|
||
**[FACT] One honest residual on the whole lane:** every proven run was performed by the **operator**,
|
||
not by a customer. `00-capability-map.md:75` still carries "A customer (not the operator) performs a
|
||
restore via UI alone" as **MISSING as evidence** — by absence of the run, not by a product gap.
|
||
|
||
### Lane 2 — the operator: guest and host recovery
|
||
|
||
Rebuilding an LXC guest, rebuilding a host, and re-establishing a box's identity are **operator
|
||
work, by design and by contract**. They are not customer-facing and are not going to be.
|
||
|
||
**[FACT] What Lane 2 contains** (INV Part A.2): the scheduled restore-test, `pct restore`, raw
|
||
`proxmox-backup-client restore`, raw `restic restore`, the agent's DR bring-up
|
||
(`--selftest=bring-up -mode dr`), the host-loss plan builder, and the escrow-consume ceremony. Each
|
||
needs root on the host or a CLI flag; none is reachable from any customer surface.
|
||
|
||
**[FACT] What is proven in Lane 2** (INV Parts G.1, G.3): whole-guest `pct restore` from both tiers
|
||
is proven with exact mount parity; the unattended restore-test is proven and currently running on
|
||
both boxes; a corrupted snapshot is proven to fail cleanly. **The DR bring-up path has never been
|
||
executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by construction**
|
||
(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
|
||
(INV Part D1).
|
||
|
||
### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03)
|
||
|
||
**[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to
|
||
fire on an interval started at daemon start. It no longer does. The rule is:
|
||
|
||
> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
|
||
> The tier is **DUE** when **A** exists and **A has not already been proven**.
|
||
|
||
So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather
|
||
than the process's uptime:
|
||
|
||
| tier rhythm | what is proved, and when |
|
||
|---|---|
|
||
| daily (host tier) | yesterday's archive, once a day |
|
||
| weekly (offsite tier) | last week's archive, once a week |
|
||
| newborn (no archive yet) | nothing — **UNKNOWN, never a fault** |
|
||
|
||
**The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest
|
||
archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive
|
||
age to zero long before it reaches the lag — so the literal reading silently switches restore-testing
|
||
off for the tier that matters most.
|
||
|
||
What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its
|
||
journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on
|
||
success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker
|
||
remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one
|
||
due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN).
|
||
|
||
**The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very
|
||
cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval
|
||
just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay
|
||
tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed
|
||
archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite
|
||
retention), and falling back to the tier's declared rhythm when history is too short to observe one.
|
||
|
||
### Why the split is right, stated once
|
||
|
||
**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
|
||
"restore the LXC and re-attach mp8 by durable_id without binding another guest's bootstrap
|
||
credentials" — a hazard serious enough to have its own runbook page
|
||
(`runbooks/RUNBOOK-manual-guest-restore.md:45-48`, F-OPS). Putting Lane 2 behind the operator is
|
||
what lets Lane 1 be a button instead of a procedure.
|
||
|
||
**[FACT] And as of D5 (v0.188.0) the split is real, not just intended.** Until then Lane 1 only *looked*
|
||
independent: an app's files sat on the customer's drive but could not be brought back without secrets
|
||
that lived solely in the guest, so the fast customer lane was silently conditioned on the slow
|
||
operator lane (§7.1 leg 1). The unit now carries those secrets, so **Lane 1 needs the drive and nothing
|
||
else** for app rebuild (§7.4). The one honest caveat: Lane 1's *file-merge* surface (Tier-2 additive)
|
||
still reads its destination from the guest's `settings.json`, so that surface remains guest-conditioned
|
||
— D5 removed the secrets leg, not the living-app leg.
|
||
|
||
---
|
||
|
||
## 4. The three-part model (D4)
|
||
|
||
**[DESIGN] Recovery needs three things, and they live in three different places.** Losing one is a
|
||
different problem from losing another, and the matrix in §8 is organised around that.
|
||
|
||
| Part | Holds | Where | Lost when |
|
||
|---|---|---|---|
|
||
| **Recipe** | scaffolding — guest sizing, drive inventory, PVE storage definitions, PBS coordinates, app inventory + storage bindings. **No secrets, no bytes.** | the hub | the hub is lost |
|
||
| **Escrow** | the host identity key material and the restic repo password | the hub, R-wrapped | the hub is lost, **or** R is lost |
|
||
| **Bytes** | Tier-1 / Tier-2 / Tier-3 / whole-guest | the tiers | the relevant medium is lost |
|
||
|
||
**[FACT] The Recipe exists and is current.** `dr_recipe` holds 6 rows; demo-felhom's was updated
|
||
`2026-07-28 17:31:13` and demo-hp's `17:25:31` — i.e. within one report cycle (LIVE, INV Part D2).
|
||
It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_bindings`.
|
||
|
||
**[FACT] The Escrow exists and is zero-knowledge.** `host_escrow`: 2 rows, `posture` `zero_knowledge`;
|
||
the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}`
|
||
(`felhom-agent/internal/escrow/identity.go:26-39`).
|
||
|
||
**[FACT] Three parts of the Recipe are empty or wrong on the live fleet**, and they are exactly the
|
||
parts a host-loss recovery would read (INV Part D2.3): `hosts.dr_record_json` is `{}` on all three
|
||
hosts; `host_escrow.directive_json` is `{}` on both escrowed hosts; `dr_recipe.host_half.drives` is
|
||
`[]` on every customer including two with enrolled data drives; and `dr_recipe.host_half.pbs.namespace`
|
||
reads `"root"` while the real namespaces are `demo-felhom` / `demo-hp`. → **R-105**, **R-106**.
|
||
|
||
---
|
||
|
||
## 5. Encryption policy (D2)
|
||
|
||
**[DESIGN] Encryption follows the boundary, not the tier.**
|
||
|
||
### On the customer's own premises: plaintext
|
||
|
||
Tier-1 recovery units, Tier-2 cross-drive copies, the app data on the drives, and the local vzdump
|
||
archives on the host are **unencrypted, deliberately**.
|
||
|
||
The reasoning, recorded so nobody "hardens" this later:
|
||
|
||
1. **It buys nothing against a real threat.** Someone who can take the second drive can take the
|
||
first. Local encryption defends against a threat model — physical theft of *only* the backup
|
||
medium — that does not describe a home server where both media sit in the same box.
|
||
2. **It adds a key-loss path that turns a working backup into a brick.** Every local encryption key
|
||
is one more thing that must survive the disaster it exists for, and the system already has one
|
||
such dependency it is trying to reduce (§7).
|
||
3. **It would break browsing, which is a feature.** FileBrowser and SMB let the household see and
|
||
use their own files. An encrypted local copy is not browsable, and the „Megosztás" and
|
||
FileBrowser surfaces are part of the product, not an accident.
|
||
|
||
**[FACT]** The local plaintext posture is real and verifiable: `dir: local` in `/etc/pve/storage.cfg`
|
||
carries no `encryption-key` (LIVE, both hosts), so the daily whole-guest archive is a plain
|
||
`.tar.zst` — 5.93 GB on demo-felhom, 1.68 GB on demo-hp (LIVE, INV Part B.3). **That archive is the
|
||
single highest-value object on the box**: it contains `encryption.key`, the restic `repo_password`,
|
||
the offbox `ssh_key`, `settings.json` and `controller.yaml` (all under `/var/lib/docker` = `mp0`,
|
||
`backup=1`). Naming that plainly is part of the policy, not an argument against it.
|
||
|
||
### Leaving the premises: encrypted, and the provider must not be able to read it
|
||
|
||
**[FACT]** Both offsite tiers encrypt client-side:
|
||
|
||
- **restic (Tier-3):** repo password is a 256-bit hex value generated once on the box and never
|
||
logged (`internal/backup/offbox.go:392-410`); restic is invoked with
|
||
`RESTIC_PASSWORD_FILE` (`:608`).
|
||
- **PBS (whole-guest offsite):** `/etc/pve/storage.cfg` carries a **per-customer** `encryption-key`
|
||
and the fingerprints differ between demo-felhom and demo-hp (LIVE). The live restore command line
|
||
observed on demo-felhom carries `--crypt-mode=encrypt` (INV Part A.2.3). Per-tenant encryption is
|
||
why cross-customer dedup is impossible on the shared datastore — a cost consequence, recorded, not
|
||
a defect.
|
||
|
||
### The one exception, stated because it is not covered by either rule
|
||
|
||
**[FACT]** The `.fab` bundle writes `app.yaml` with **decrypted plaintext secrets, deliberately**,
|
||
and its password is **optional** (`internal/appexport/export.go:484,506-511`; the generated file's
|
||
first line is literally `# Exported by felhom-controller — plaintext secrets`, `:531`;
|
||
`Encrypted: req.Password != ""`, `:307`). A `.fab` can also be written to a registered **network**
|
||
share, because `storageDriveList()` does not filter network paths
|
||
(`internal/web/handler_export.go:377-387`). This is a portability artifact, not a tier — but it is
|
||
the one place where a customer action can put every secret of one app onto a NAS in plaintext.
|
||
Recorded here so the encryption policy is not read as covering it. → **R-108** (same root cause).
|
||
|
||
---
|
||
|
||
## 6. The tiers — what each captures
|
||
|
||
The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what they can actually do.
|
||
|
||
### 6.1 The four tiers, as configured on the live fleet
|
||
|
||
| Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted |
|
||
|---|---|---|---|---|---|
|
||
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
|
||
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
||
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
||
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
|
||
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` — **and the box asks for none** (R-191) | **yes** (per-customer key) |
|
||
|
||
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
|
||
|
||
|
||
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
||
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
||
(`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`.
|
||
|
||
**[FACT] What the whole-guest tiers do NOT carry.** `mp8 /mnt/felhom-drives` and
|
||
`mp9 /etc/felhom-bootstrap` are **host bind mounts** and are out of `vzdump` scope entirely (LIVE
|
||
`pct config 9201`, both hosts). So the whole-guest tiers carry the guest and **none of the customer's
|
||
data drives** — 916 GB on demo-felhom, 938 GB on demo-hp.
|
||
|
||
### 6.2 Coverage per app class — and an unresolved count
|
||
|
||
**[FACT]** Of 53 catalog templates, **52 keep data in Docker named volumes**; exactly **13** carry a
|
||
`backup:` block, and those 13 are exactly the templates that bind `${HDD_PATH}` / `${USERDATA_PATH}` /
|
||
`${IMPORT_PATH}` at all; **14** have a database service (INV Part B.1, computed against catalog
|
||
`4252121`).
|
||
|
||
Applying the classifier's documented two-level default (`internal/appbackup/classify.go`: explicit
|
||
entry wins; else a `:ro` reader is *excluded*; else a writable bind is *mandatory*):
|
||
|
||
| tier filter | templates with at least one file leg |
|
||
|---|---|
|
||
| **Tier-3** (`mandatory` only) | **4** — calibre-web, immich, nextcloud, paperless-ngx |
|
||
| **Tier-2** (`mandatory + optional`) | **7** — the 4 above plus audiobookshelf, komga, romm |
|
||
| legacy resolver path (apps with no block) | **0** — no non-block template binds a namespace path |
|
||
|
||
> ### ⚠️ UNRESOLVED — two counts of the same thing disagree
|
||
>
|
||
> | source | count |
|
||
> |---|---|
|
||
> | C9-F1 Phase 0, as shipped | **A = 9 / B = 43 / C = 1** — `felhom.eu/REPORT.md:17-24`, restated `backlog/OPEN-ITEMS.md:31` |
|
||
> | INV Part B.1, independent enumeration at catalog `4252121` | **A = 7 / B = 45 / C = 1** |
|
||
>
|
||
> Both use the same definition ("templates whose Tier-2 copy can hold a readable file leg"). The
|
||
> difference is two apps and **neither number is adopted here**. The Phase-0 enumeration is described
|
||
> in prose but the script is not committed, so the two methods cannot be diffed from the repo.
|
||
> **This must be resolved before either figure is used to size anything.**
|
||
|
||
**[FACT] The class-B consequence is real regardless of which count is right.** For an app whose data
|
||
lives entirely in named volumes, the Tier-2 copy holds a full `recovery-unit/` and **no readable
|
||
leg** — LIVE on demo-felhom, `backups/secondary/bookstack/` and `.../docmost/` contain
|
||
`recovery-unit` and nothing else, at 156 MB and 86 MB (INV Part B.2). Since v0.183.0 the button
|
||
refuses **before** stopping the app and names the action that works.
|
||
|
||
### 6.3 Where capture and restore disagree
|
||
|
||
**[FACT]** Three asymmetries, each source-cited:
|
||
|
||
| tier | captured | read back by that tier's restore | gap |
|
||
|---|---|---|---|
|
||
| Tier-1 | unit incl. volume tars + DB dumps | all of it | none |
|
||
| Tier-2 | unit mirror **+** file legs | `hdd/` and `userdata/` **only** (`tier2_restore.go:101-104`) | **the unit mirror is read by nothing** — `RecoveryUnitPath` resolves to `backups/primary/` (`appbackup/paths.go:46-48`) → **R-102** |
|
||
| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay; the unit is **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **no offsite action unpacks the named-volume tars it captures** → **R-107** |
|
||
|
||
**[FACT]** Tier-2's gap is the sharper one because of *when* it bites: Tier-2 exists for the case
|
||
where the primary drive is lost — and in exactly that case the primary unit is gone while this
|
||
mirror survives on the second drive, unreachable by any customer action.
|
||
|
||
---
|
||
|
||
## 7. The recovery chain (D3) — the reason this document exists
|
||
|
||
**[DESIGN] 3-2-1 describes copies. It does not describe recovery.**
|
||
|
||
Three copies on two media with one offsite is a statement about *bytes surviving*. It says nothing
|
||
about whether the bytes can be turned back into a working system, and a system can satisfy 3-2-1
|
||
completely while having no executable recovery route for a given failure. That is not a hypothetical
|
||
here — §8 has rows where it is the actual state.
|
||
|
||
### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
|
||
|
||
> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
|
||
> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
|
||
> byte-identical. **It passed with an operator present, and four manual interventions stood between
|
||
> "the key is recoverable" and "the file is back" — none of which was in any design document.** They
|
||
> are recorded here because this is the section a future reader will use to answer *"can the customer
|
||
> do this alone?"*, and until 2026-08-05 the answer was no for reasons nothing wrote down.
|
||
|
||
| # | The step | Why it stopped a customer | Status |
|
||
|---|---|---|---|
|
||
| 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. |
|
||
| 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed` → `OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. |
|
||
| 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. |
|
||
| 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** is required. | **OPEN → R-193.** This is a design decision (the one-shot credential model), not a defect, and it needs an operator ruling. **It is the reason the capability-map row keeps its "with a person present" qualifier.** |
|
||
|
||
> **[FACT] The honest current answer:** three of the four are gone. **A customer alone still cannot
|
||
> complete a rebuild recovery**, because step 4 requires the operator. Everything after step 4 —
|
||
> claim, restore, verify — is now self-service.
|
||
|
||
### 7.1 The dependency graph
|
||
|
||
> **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below
|
||
> (**secrets**) is **no longer true for Tier-1 and Tier-2**: the local recovery unit now carries the
|
||
> portable secret class, so those two tiers need the **drive and nothing else**. The graph and leg 1
|
||
> are kept as written because they are the model everything downstream was derived from, and because
|
||
> leg 2 (**the living app**) is unchanged and still binds Tier-2's file leg and Tier-3. Read §7.4 for
|
||
> the current chain; the corrected rows are 3, 4 and 9 in §8.
|
||
|
||
**[FACT — as of the R-108 survey, before D5]** Every app-tier restore depends on the guest, for one of
|
||
two reasons:
|
||
|
||
```
|
||
┌──────────────────────────────────────────┐
|
||
│ the LIVE GUEST │
|
||
│ · settings.json (tier-2 destination) │
|
||
│ · encryption.key (32 B) │
|
||
│ · app.yaml (ENC: under that key) │
|
||
│ · the deployed app itself │
|
||
└───────────┬──────────────────────────────┘
|
||
│ required by
|
||
┌──────────────────────────┼──────────────────────────┬─────────────────────┐
|
||
│ │ │ │
|
||
Tier-1 restore Tier-2 restore Tier-3 reconstitute Tier-3 scratch
|
||
(rebuild the app) (fill in files) (overwrite + replay) (verification copy)
|
||
│ │ │ │
|
||
needs SECRETS needs the app needs the app needs only the
|
||
from app.yaml running + the deployed + a DB repo password
|
||
(restore_unit.go recorded dest service identifiable (also in the guest)
|
||
:130-132) (tier2_restore.go
|
||
:114-116)
|
||
│ │ │ │
|
||
└──────────────────────────┴──────────────────────────┴─────────────────────┘
|
||
│
|
||
┌───────────▼──────────────────────────────┐
|
||
│ WHOLE-GUEST TIER (local vzdump / PBS) │ ← Lane 2, operator only
|
||
└───────────┬──────────────────────────────┘
|
||
│ required by
|
||
┌───────────▼──────────────────────────────┐
|
||
│ THE HOST (agent.json, bootstrap.json, │ ← in NO backup at all
|
||
│ vmbr9, sudoers, /etc/pve/priv, WG) │ (INV Part D1)
|
||
└──────────────────────────────────────────┘
|
||
```
|
||
|
||
**[FACT] The two legs of the dependency, precisely:**
|
||
|
||
1. **Secrets.** The recovery unit is secret-free by design — *"It NEVER writes a secret value"*
|
||
(`internal/backup/recovery_unit.go:73`). `RestoreFromRecoveryUnit` recovers secrets **from the
|
||
guest, never from the unit** (`restore_unit.go:130-132`), and the policy is stated outright at
|
||
`:18-22`: *"Regenerate NOTHING… A missing DATA-ENCRYPTING key is FATAL: regenerating it would
|
||
render the restored data unreadable, so we refuse and tell the operator to do a PBS whole-guest
|
||
restore."* Those secrets are encrypted under `encryption.key`, a 32-byte file that exists **only
|
||
inside the guest** (LIVE, both boxes — INV Part C, row 9).
|
||
2. **The living app.** Tier-2's restore reads the destination recorded in the guest's `settings.json`
|
||
(`tier2_restore.go:114-116`), and Tier-3's reconstitution refuses outright when the app is not
|
||
deployed — *„a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat"*
|
||
(`offbox_reconstitute.go:198-201`).
|
||
|
||
**[FACT] So all three app tiers are conditioned on the whole-guest tier**, which is Lane 2. The
|
||
customer's own recovery lane rests on an operator-only tier, and today nothing tells the customer
|
||
that. — **This sentence was the reason D5 exists. It is now false for Tier-1 and Tier-2; see §7.4.**
|
||
|
||
### 7.4 The chain AFTER D5 (controller v0.188.0, 2026-07-30) — the current model
|
||
|
||
**[FACT] Tier-1 and Tier-2 no longer depend on the whole-guest tier.** The unit's `compose/app.yaml`
|
||
(mode 0600) carries the **portable secret class** — every catalog `type: secret` field, i.e. the
|
||
declared `data_key`s, the 18 DB/root passwords and the internal signing secrets — so neither
|
||
`encryption.key` nor the guest's `app.yaml` is required to rebuild an app:
|
||
|
||
```
|
||
Tier-1 restore Tier-2 restore Tier-3 reconstitute Tier-3 scratch
|
||
(rebuild the app) (fill in files) (overwrite + replay) (verification copy)
|
||
│ │ │ │
|
||
needs ONLY THE DRIVE needs the app needs the app needs only the
|
||
(unit carries the running + the deployed + a DB repo password
|
||
secrets; guest is recorded dest service identifiable
|
||
consulted only for
|
||
the withheld class) └────────── still conditioned on the guest (leg 2, unchanged) ──────┘
|
||
│
|
||
✅ INDEPENDENT of the whole-guest tier
|
||
```
|
||
|
||
**[FACT] What still needs the guest, precisely** — so this is not read as more than it is:
|
||
|
||
1. **Leg 2 is untouched.** Tier-2's *file* restore still reads the destination from the guest's
|
||
`settings.json` (`tier2_restore.go:114-116`) and Tier-3's reconstitution still refuses when the app
|
||
is not deployed (`offbox_reconstitute.go:198-201`). D5 removed the **secrets** leg, not the
|
||
**living-app** leg. So "Tier-2 no longer depends on the guest" is true of the *unit-restore* path it
|
||
shares with Tier-1, and **false** of its additive file-merge path.
|
||
2. **The withheld class.** The 7 `type: password` admin logins and `vaultwarden/ADMIN_TOKEN` are
|
||
deliberately NOT on the drive; they come from the guest, or are regenerated (O4). Their absence
|
||
costs a credential reset, never data.
|
||
3. **The fail-closed gate remains.** A `data_key` in neither the unit nor the guest still refuses the
|
||
restore outright. D5 makes it normally present; it does not soften the gate.
|
||
|
||
**[DESIGN] Why plaintext is the right answer here, and what it is coupled to.** Every travelling secret
|
||
either decrypts data on the *same drive* or authenticates to a container on an internal compose network
|
||
with no external listener — so possessing it adds nothing to possessing the drive, which is exactly D2's
|
||
argument for keeping the DATA plaintext. That argument holds **only because** the internet-reachable
|
||
class is withheld. **The exclusion is what licenses the plaintext; the two must not be relaxed
|
||
independently.** `stacks.PortableSecretEnvVars` is the single boundary, with a code-level
|
||
`nonPortableSecrets` register (not a catalog flag — a boundary a catalog push can move is not a
|
||
boundary, cf. R-97a).
|
||
|
||
**[FACT] Precedence, because two sources now exist.** The **unit wins** over the guest. Not
|
||
"newest wins": the unit's secrets are captured in the same run as the dumps beside them, so the unit's
|
||
value matches *the data being restored*, whereas the guest's is merely the most recent. Preferring the
|
||
guest is the data-loss bug — a rotated data key does not decrypt data encrypted with the old one, and a
|
||
rotated DB password does not match the hash inside the restored data directory.
|
||
|
||
**[FACT] Proven live 2026-07-30**, on a scratch drill guest, through the real endpoints: AdventureLog
|
||
(`SECRET_KEY` data_key + `DB_PASSWORD`) restored with the guest's `app.yaml` **moved aside** —
|
||
`secrets recovered=2/2`, 27.6 s — and then **the application itself read the seeded customer row over
|
||
TCP with its own credential**, which is the observable that matters (a restore that returns success onto
|
||
unreadable data is a failure that looks like a pass). The discriminator held: the pre-backup row came
|
||
back, a row added *after* the backup was gone, so the volume tar was genuinely restored. There was **no
|
||
`.sql` dump** in that unit, so the DB came back from the volume tar — the exact case where a regenerated
|
||
password would have failed. Separately, Grafana's `type: password` admin login was proven **withheld**:
|
||
the sentinel value was live in the container and encrypted in the guest, and appeared in **0 files**
|
||
anywhere under the backup namespace. Full record: `audits/D5-drive-alone-restore-2026-07-30.md`.
|
||
|
||
### 7.2 A tier whose prerequisites cannot be met in the failure it exists for
|
||
|
||
**[FACT]** Two instances, both current:
|
||
|
||
- **Tier-2 vs primary-drive loss.** Tier-2's stated purpose is surviving the loss of the primary
|
||
drive. In that failure the primary recovery unit is gone; the surviving mirror on the second drive
|
||
is `backups/secondary/<app>/recovery-unit/`, which **no code path reads** (§6.3). For the 45-or-43
|
||
class-B apps the restore is a guaranteed no-op in exactly its designed scenario. → **R-102**
|
||
- **Tier-3 vs guest loss.** Tier-3 holds the volume tars and the DB dump. Reconstitution requires
|
||
the app to be deployed and skips the unit; the tars are unpacked only by the Tier-1 path, which
|
||
requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest.
|
||
→ **R-107**
|
||
|
||
### 7.3 ~~What D5 would change — and why it was blocked~~ — **D5 SHIPPED 2026-07-30 (v0.188.0)**
|
||
|
||
> **This section is history.** D5 is implemented and proven live; the current chain is **§7.4**. Kept
|
||
> because the reasoning below is why the precondition was required, and because the last paragraph
|
||
> (`.fab` / R-126) is still open and still not part of D5.
|
||
>
|
||
> **One correction to the target as stated below.** It assumed the class that must travel is the
|
||
> `data_key`-flagged secrets. Part 0 of the D5 task tested that and it did not survive: the flag is
|
||
> unreliable (four encryption keys the catalog itself labels as such are unflagged → **R-127**), and a
|
||
> DB password is **not resettable in practice** — `POSTGRES_PASSWORD` is ignored once PGDATA is
|
||
> non-empty, so a regenerated value leaves the app unable to authenticate against its own restored rows
|
||
> while the dump replay still reports success. The shipped boundary is therefore *all* `type: secret`
|
||
> minus the register, not `data_key` alone.
|
||
|
||
**[DESIGN, TARGET — SUPERSEDED BY §7.4]** The intended fix is to make app secrets travel with the **local**
|
||
recovery unit, so Tier-1 and Tier-2 restore work **without the guest and without R**. Offsite
|
||
already encrypts everything, so secrets travelling offsite would be covered by the escrowed repo
|
||
password. R would then be required for **offsite recovery and host identity only** — losing R would
|
||
cost the offsite route, not local recovery.
|
||
|
||
**The premise D5 rests on is now established.** §2 of the task that produced this document required
|
||
it to be proven, not assumed: the backup tree must be unreachable from every browsing, download and
|
||
export surface. When this document was written it was **not** — the FileBrowser network-share bind
|
||
reached it. **R-108 closed that on 2026-07-30** (controller v0.187.0) by refusing app namespaces on
|
||
network storage, so no `backups/` tree can exist under the share-root bind; every other surface was
|
||
already clear (§10.1's table). **D5's precondition is therefore MET and D5 may be adopted.**
|
||
|
||
**Still true, and not part of D5's precondition:** a `.fab` bundle carries plaintext secrets by
|
||
design with an optional password, and `storageDriveList()` does not filter network paths, so a bundle
|
||
can be **exported onto** a NAS (§5, → **R-126**). That is an export destination the customer chooses
|
||
explicitly, not a browsing surface reaching a backup tree, and it is unchanged by D5 — D5 moves
|
||
secrets into the local recovery unit, not into `.fab`. It is tracked separately rather than folded in.
|
||
|
||
Until D5 is actually implemented, §7.1's chain stands as the model.
|
||
|
||
---
|
||
|
||
### 7.5 Where a recovery unit is RETAINED — and the size bound it puts on Lane 1
|
||
|
||
Added 2026-08-02 from `audits/SPIKE-recovery-unit-space-2026-08-02.md` and
|
||
`audits/CAMPAIGN-10-closeout-2026-08-02.md`. §6.1 says a Tier-1 unit lives *"on the app's own drive"*,
|
||
which is true and is the whole story for a drive-resident app. **It is not the whole story for an app
|
||
with no data drive, and that case was undocumented until now.**
|
||
|
||
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go:245-255`)
|
||
returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what
|
||
`internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit
|
||
after it is copied onward: the only prune is F5 (`backup.go:1053-1112`), which removes residue on an
|
||
**old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy
|
||
indefinitely.
|
||
|
||
**What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`):
|
||
`compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains
|
||
`mp8` / `HDD_PATH` userdata** — a photo library on a 1 TB drive is not in a unit, and cannot make one
|
||
overflow.
|
||
|
||
**The resulting mismatch, on a default appliance:**
|
||
|
||
| | size | holds |
|
||
|---|---|---|
|
||
| `mp0` `/var/lib/docker` | **50 G** | every app's live volumes — the thing a unit copies |
|
||
| `mp1` `/mnt/sys_drive` | **20 G** | the retained units of every **driveless** app |
|
||
|
||
A DB-backed app's unit is up to **~2×** its data, because it carries the volume tar **and** the SQL
|
||
dump (measured: 21.1 GB of app data → a **40.2 GB** unit). A file-only app's is **1.00×** (measured:
|
||
homebox 2305 MB → 2305 MB). `--sysdata-grow` (`felhom-agent/cmd/felhom-agent/main.go:178`) defaults to
|
||
**0**, is not derived from the physical drive, and no observed box passes it — demo-hp's production
|
||
guest 9201 ships `mp0 50G / mp1 20G`.
|
||
|
||
**`mp1` gates the entire app-data chain, not just Tier 1.** Tier-2 mirrors the unit *"(always)"* from
|
||
`RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too, so a unit that
|
||
cannot be written to `mp1` leaves Tier-2 and Tier-3 with nothing to copy.
|
||
|
||
**The bound this places on Lane 1 (§3).** Lane 1's promise is that a customer can restore an app from
|
||
the drive alone, without the operator and without the guest — D5 (§7.4) made that true by putting the
|
||
portable secrets in the unit. **That independence is bounded by app size**, and the bound is:
|
||
|
||
> On a default appliance, an app can be restored by Lane 1 only while its recovery unit still fits the
|
||
> retained space — **≈ 19 GB of app data for a file-only app, ≈ 10 GB for a DB-backed one.**
|
||
> Past that the unit cannot be captured, and the app falls back to Lane 2's operator-driven
|
||
> whole-guest route.
|
||
|
||
**AS OF 2026-08-02 SOMETHING NOW WARNS, AND THE ALERTING IS PART OF THIS CONTRACT (R-167 / R-158,
|
||
decision D-c; controller v0.191.x + hub v0.89.0).** The last sentence of this section used to end
|
||
"nothing warns when an app crosses the line". Two signals now exist and both are PROVEN-LIVE:
|
||
|
||
- **To the CUSTOMER, before anything fails** — `internal/fillwatch` warns per FILESYSTEM (never per
|
||
app: one full disk holding ten apps would fire ten times) on **whichever trips first, used ≥ 85% or
|
||
free < 5 GiB**, critical at 95% / 2 GiB, clearing at 75% / 7 GiB. **Two terms, because a percentage
|
||
alone lies at both ends of the range this section itself documents:** 85% of a 20 G `mp1` leaves
|
||
3 G — less than one DB-backed app's unit — while 85% of a 4 TB drive leaves 600 G. It watches the
|
||
app-data volume, the system-data volume **and** every registered drive, which the previous
|
||
`health_degraded` signal did not. Edge-triggered against persisted state; the hub owns cooldown.
|
||
- **To the OPERATOR, when a capture actually fails** — `recovery_unit_capture_failed`, per app, with
|
||
the target filesystem's used/free bytes at the moment of failure, so the *why* needs no login. It is
|
||
**operator-tier** (`notify.operatorOnlyEvents`) and deliberately not `backup_failed`: a customer can
|
||
take no action on a capture failure.
|
||
|
||
### 7.5.1 — THE CEILING THIS SECTION DESCRIBES HAS BEEN REMOVED (2026-08-03, R-165 / decision D-a)
|
||
|
||
**Everything above describes the SPLIT layout, which is now the legacy shape.** A golden built by
|
||
`build-golden.sh` **v3.0.0** ships **one** data volume; `mp1` does not exist. Both consumer paths are
|
||
binds of subdirectories of it (variant **V-c**):
|
||
|
||
```
|
||
mp0 -> /var/lib/felhom ├─ docker/ --bind--> /var/lib/docker
|
||
└─ sys_drive/ --bind--> /mnt/sys_drive
|
||
```
|
||
|
||
**So the size bound below no longer applies to a box built from that golden.** A driveless app's
|
||
recovery unit is limited by the box's actual free space, not by a partition set at build time. The
|
||
mismatch table above (`mp0` 50 G vs `mp1` 20 G) describes what a merged box no longer has.
|
||
|
||
**R-175, fixed here rather than left standing.** The bound below was stated as the fleet's and was
|
||
**one box's**: it is derived from `mp1 = 20 G`, which is demo-hp exactly and never was demo-felhom
|
||
(`mp0 200G / mp1 50G`, where the same arithmetic gives ≈ 49 GB / ≈ 24 GB), nor the golden (`16 G / 8 G`
|
||
before provision grew them). **Read it as a function of `mp1`, and only for a box still on the split
|
||
layout.** Measured: `audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1.
|
||
|
||
**What replaced the partition's second job — the reserve.** `mp1` was also a BULKHEAD: an overflow was
|
||
refused per app with the last good unit byte-identical, and it **could not reach `/var/lib/docker`**,
|
||
because that was a different filesystem. On a merged box it can. Decision **B2**, shipped in controller
|
||
**v0.192.0**, is that bulkhead made deliberate — a two-term reserve (97% used or 1 GiB free) in
|
||
`internal/fillwatch`'s shape, sitting beyond its critical band so the customer is always warned first.
|
||
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
|
||
only destroy a different app's only local copy.
|
||
|
||
**WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
|
||
The two are deliberately different mechanisms, because conflating them is how seven failures went
|
||
missing on 2026-08-03 without leaving a trace.
|
||
|
||
| | Record | Notification |
|
||
|---|---|---|
|
||
| what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
|
||
| when | every time, unconditionally | at the end of a run, **only if something failed** |
|
||
| gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
|
||
| where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
|
||
|
||
- **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
|
||
is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
|
||
REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
|
||
design; weakening it re-opens a silent-failure path.
|
||
- **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
|
||
Deciding not to tell someone is itself an event worth recording.
|
||
- **Deliberate skips are not failures** and never appear in the digest — a disconnected or
|
||
decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
|
||
operator stops reading.
|
||
- **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
|
||
the hour, because someone pressing the button is actively trying to get a backup. The periodic
|
||
capture sweep is capped by the ordinary hourly cooldown.
|
||
|
||
**THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
|
||
**per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
|
||
before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
|
||
volume dump and the recovery-unit capture**. Those three write under one per-app root
|
||
(`backups/primary/<app>`), which is what makes one verdict able to cover them honestly.
|
||
|
||
- **What it guarantees.** A refused app has **nothing written for it in that run**, its previous unit
|
||
is **byte-identical**, it is **not stopped**, nothing anywhere is deleted, and the operator gets
|
||
**exactly one** alert naming the app, the term that bound and the disk figures.
|
||
- **Two terms, two questions.** *Headroom*: is the filesystem already below the reserve? *Size*: would
|
||
THIS app's write take it below? The size estimate is the app's previous `.sql` + `.tar` on disk;
|
||
with no history the decision degrades to headroom alone, deliberately — otherwise the first backup
|
||
is the one that can never happen.
|
||
- **Why it is decided lazily and not once per run.** Space changes during a run: app A's dump can put
|
||
app B under the reserve, so a verdict taken at run start reads a disk that no longer exists.
|
||
- **Why it is never re-decided between an app's own legs.** That is precisely the shape v0.192.0 had —
|
||
the two dump legs unguarded and only the capture refused — under which the reserve was consumed by
|
||
the very write it exists to bound, and the refusal's *"the previous unit is untouched"* was measured
|
||
false. Proven live on demo-hp 2026-08-03 (R-181), fixed the same day, and re-proven by filling the
|
||
box for each of the two terms.
|
||
- **It sits ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal
|
||
decided inside it would already have bounced the app it is refusing to back up.
|
||
|
||
**Status caveat, deliberately explicit:** every box in the field that has not been reinstalled is still
|
||
on the split layout and everything above still describes them exactly. This subsection describes what a
|
||
box built from golden ≥ 0.192.0 gets. Both demo boxes were reinstalled from it on 2026-08-03 (R-178).
|
||
|
||
Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling**
|
||
(**R-163**) — this section states the constraint, not a number. And **the same-device placement is
|
||
intended, not a defect**: a driveless app's unit sits on the same SSD as its volumes, but Tier-2's
|
||
cross-drive mirror, the whole-guest tier (which covers `rootfs + mp0 + mp1`, §6.1) and Tier-3 all cover
|
||
device loss. What was missing was that this case existed at all, and that nothing warns when an app
|
||
crosses the line — **R-158**.
|
||
|
||
## 8. The failure → recovery matrix
|
||
|
||
**This is the core artifact.** One row per failure. It is authoritative for recovery routes;
|
||
`00-capability-map.md` stays authoritative for per-capability status.
|
||
|
||
**How to read the numbers.**
|
||
|
||
- **RTO** — **only measured durations** from INV Part F appear here. A blank cell means *nobody has
|
||
ever measured it*, and a blank is a finding, not an omission.
|
||
- **RPO** — **no RPO has ever been measured from an incident.** These cells carry the **configured
|
||
cadence that bounds RPO**, read live from the box, labelled `(cadence)`. A blank means no cadence
|
||
governs the row.
|
||
- **Status** — `PROVEN` (live, cited) · `PARTIAL` (some legs proven) · `IMPLEMENTED` (code + tests,
|
||
never exercised) · `NONE` (no route exists).
|
||
|
||
| # | Failure | What survives | Recovery route | Invocable by | RTO (measured) | RPO (cadence) | Status | Evidence |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| 1 | **Customer deletes files** | everything else | „Fájlok visszaállítása" — Tier-2 additive merge | **customer** | **39 s** (6 files) | 24 h | **PROVEN** | CAMPAIGN-9 A1 — byte-identical returns, both non-destruction promises kept, bytes re-served by paperless's own API |
|
||
| 2 | **An app's data directory is destroyed** | the guest, the other tiers | same route | **customer** | **46 s** (43 files) | 24 h | **PROVEN** | CAMPAIGN-9 A3 — loss verified by a positive observable (doc download 200 → 404), then 16/16 documents usable |
|
||
| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar |
|
||
| 3c | *same, with the GUEST GONE (secrets unavailable)* | the drive | Tier-1 unit restore — **the unit carries the portable secrets** | **customer** | **27.6 s** | 24 h | **PROVEN** | D5, §7.4. Before v0.188.0 this row was **NONE**: the data-key gate refused and the customer's own copy of their own data was not a recovery |
|
||
| 3b | *same, for a class-B app via Tier-2* | — | **no route** — Tier-2 never reads the unit mirror | — | | | **NONE** | §6.3; **R-102** |
|
||
| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB; **the volume tars in either copy are unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102**, **R-107** |
|
||
| 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go:359-393` |
|
||
| 6 | **Guest lost or corrupted** | the host, both whole-guest tiers, the data drives (they are host binds) | `pct restore` from `local:` or `felhom-pbs:` | **operator** (SSH) | **84–112 s** local · **1101 s** PBS (both = restore-test into a scratch guest, boot + verify + teardown) | 24 h local · 7 d offsite | **PROVEN** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C (exact mount parity, `unprivileged: 1` preserved); LIVE restore-tests on both boxes this session |
|
||
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
|
||
| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
|
||
| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) |
|
||
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) |
|
||
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
|
||
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
|
||
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D |
|
||
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) |
|
||
| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
|
||
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
|
||
|
||
### 8.1 The blank cells, listed explicitly
|
||
|
||
Per the rule that a blank is a finding, here they are:
|
||
|
||
| row | blank | why |
|
||
|---|---|---|
|
||
| 3b | RTO, RPO | no route exists to time |
|
||
| 4 | RTO | no drive-loss recovery has ever been timed |
|
||
| 5 | RTO | never timed; the rebuild is a normal Tier-2 run |
|
||
| 6 | — | RTO present, but it is a **restore into a scratch guest on the same host**; a restore *to a different host* has never been timed |
|
||
| 8 | RTO | a host has never been rebuilt as itself (INV Part D1) |
|
||
| 9 | RTO | the composed whole-box path has never been run |
|
||
| 10 | RTO | no ransomware-shaped recovery has ever been run |
|
||
| 11 | RTO | a hub restore has never been performed |
|
||
| 12 | RTO, RPO | no provider-loss recovery has ever been run |
|
||
| 13 | RTO, RPO | not a timed recovery; a capability loss |
|
||
| 14 | RTO | the break-glass path is proven but was never timed |
|
||
| 15 | RTO | the manual unlock was performed but not timed |
|
||
|
||
**Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB
|
||
from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS
|
||
incremental; `.fab` import wall-clock; and time-to-first-byte for a customer restore over a home
|
||
uplink — **no customer has ever driven a restore**.
|
||
|
||
---
|
||
|
||
## 9. What the model implies for the tiers (recorded, not new design)
|
||
|
||
**[DESIGN]** Three consequences follow from §3–§7 and are stated so they are not re-derived:
|
||
|
||
1. **Tier-2 is a *drive-loss* tier, not a second chance at Tier-1.** Its job is to survive one drive
|
||
dying. That is why it mirrors the unit at all — and why the unit mirror being unreadable (§6.3)
|
||
defeats the tier rather than degrading it.
|
||
2. **Tier-3 is a *premises-loss* tier.** It is the only copy that survives fire, theft and
|
||
ransomware-with-host-access, which is why it is the only tier whose credential exposure is ranked
|
||
as the largest open data risk (R-95).
|
||
3. **The whole-guest tiers are the *system* tier, and they sit under everything else** (§7.1).
|
||
Treating them as "one more copy" is precisely the 3-2-1 framing this document rejects.
|
||
|
||
---
|
||
|
||
## 10. Known gaps
|
||
|
||
Every divergence between the model above and the system as it is, each with an ID.
|
||
|
||
### 10.1 ~~D5 is BLOCKED~~ — precondition CLOSED by R-108 (v0.187.0); **D5 ITSELF SHIPPED (v0.188.0)**
|
||
|
||
> **D5 IS DONE — 2026-07-30, controller v0.188.0.** The precondition below was met by R-108, and D5 was
|
||
> then implemented and proven live the same day: Tier-1/Tier-2 no longer depend on the whole-guest tier,
|
||
> and a customer needs **the drive and nothing else**. The current recovery chain is **§7.4**; the
|
||
> corrected matrix rows are 3, 3c and 13. This section's remaining value is the *precondition* analysis,
|
||
> which is why the table below is kept.
|
||
|
||
> **D5's PRECONDITION IS MET.** An app's data namespace can no longer be placed on network storage, so
|
||
> no `backups/` tree can exist inside FileBrowser's share-root bind, and the browsing surface therefore
|
||
> cannot reach the backup tree on ANY storage class. Every other read surface in the table below was
|
||
> already NO. **D5 may be adopted** — nothing in this section blocks it.
|
||
>
|
||
> **The fix inverted the obvious one, and that is the durable lesson here.** The share-root bind was
|
||
> not narrowed, because it **cannot** be: (a) the `:rslave` share-ROOT bind is load-bearing — a
|
||
> Phase-0 probe (2026-07-22) proved an in-container access through it wakes the idle automount
|
||
> trigger, so narrowing it breaks NAS access itself; (b) there is no `userdata/` layer to scope to,
|
||
> since apps on a share store at `<share>/<app>`; and (c) creating one would write Felhom's directory
|
||
> convention onto a customer's own NAS, which R-67 forbids outright. The browsing surface being
|
||
> immovable is precisely *why* the backup tree must never be placed under it. Tier 2 had already
|
||
> reached the same conclusion for its own targets (`F-6C-1`); R-108 closes the PRIMARY namespace,
|
||
> which was the last remaining route.
|
||
>
|
||
> **Operator ruling 2026-07-30: refuse the placement, keep the browse bind.** `RefuseAsAppNamespace`
|
||
> (`internal/settings/settings.go`) is the single predicate; all placement surfaces consult it. It
|
||
> **fails closed** — `/mnt/felhom-drives` holds both storage kinds in-guest, so a path prefix cannot
|
||
> classify and `Kind` exists only on a REGISTERED path; an unregistered path under that root is
|
||
> therefore un-classifiable and is refused rather than assumed to be a drive.
|
||
>
|
||
> **Nothing was stranded:** zero apps on network storage across all six hub customers including Peti.
|
||
> R-67's browse capability is byte-for-byte unchanged (verified by diffing demo-hp's generated
|
||
> compose before and after the deploy). This also **supersedes** the controller README's "NAS backup
|
||
> locality — decision A" (v0.118.0), which deliberately kept a NAS-resident app's Tier-1 artifacts on
|
||
> the NAS: that case can no longer arise.
|
||
>
|
||
> Evidence: `audits/R108-network-app-namespace-2026-07-30.md`. The pre-fix analysis below is retained
|
||
> verbatim as the record of what was wrong.
|
||
|
||
#### 10.1 (historical) The exposure as it stood before v0.187.0
|
||
|
||
**[FACT] The verification and its result.** Every surface that can read a file was checked:
|
||
|
||
| surface | can it reach `backups/`? | evidence |
|
||
|---|---|---|
|
||
| SMB share creation | **NO** | `sharingResolvePath` (`internal/web/sharing_handlers.go:52-81`) resolves symlinks *before* containment, then refuses any path within `SharingDeniedRoots(root)`; that set covers `<root>/backups` **and** the legacy `<root>/felhom-data` + `<root>/felhom-data/backups` (`internal/stacks/samba.go:46-65`, derived from `ProtectedHDDPaths`, `delete.go:59-78`) — **both namespace shapes** |
|
||
| SMB browse (folder picker) | **NO** | same deny set applied per child (`sharing_handlers.go:521-537`) |
|
||
| SMB `ensureImportShare` (the store-direct bypass) | **NO** | writes one controller-generated constant, `GetImportRoot()` = `<system ns>/userdata/import` (`sharing_handlers.go:564-587`) |
|
||
| FileBrowser — **local drives** | **NO** | the bind is `appbackup.UserdataDir(sp.Path)` only, and the comment says why (`internal/web/handlers.go:2450-2460`) |
|
||
| **FileBrowser — network shares** | ~~**YES**~~ → **NO** (R-108, v0.187.0) | the bind is still the share **ROOT** (`- %s:/srv/%s:rslave`, `handlers.go:2437`) and deliberately so — but **no app namespace, hence no `backups/` tree, can exist on a share**, so the root bind reaches only the customer's own files. The reachability is closed at the PLACEMENT, not at the bind |
|
||
| `.fab` import path validation | **NO** | confined to `<root>/exports` (`handler_export.go:400-408`, `estimate.go:215-217`) |
|
||
| `.fab` browser download | **NO** | name-pattern + parent-must-be-the-staging-dir double guard (`handler_export_download.go:36-45,120-140`) |
|
||
| `/api/debug/*` | **NO** | no file-serving branch (`handler_debug.go:46-92`) |
|
||
| `http.ServeFile` (3 sites) | **NO** | assets only, `filepath.Base`-normalised (`server.go:720-771`) |
|
||
| log bundles | **NO producer found** | no filesystem-walk bundle producer exists on the box; searched `felhom-controller/internal`, `felhom-agent/internal`, `cmd/` |
|
||
| registering the backup dir as a drive | **NO** | the manual add requires `system.IsMountPoint(path)` (`handlers.go:2091-2095`); `<drive>/backups` is not a mount point |
|
||
|
||
**The exposure, end to end.** All six links are source-cited and the precondition is live today:
|
||
|
||
1. A NAS share is registerable as a storage path and lands `Schedulable: true` — **LIVE on demo-hp**:
|
||
`{"path":"/mnt/felhom-drives/Felhom-Share","kind":"network","schedulable":true}`.
|
||
2. `GetSchedulableStoragePaths()` has **no `IsNetwork()` filter** (`internal/settings/settings.go:904-914`),
|
||
so that share appears in the **deploy** dropdown (`handlers.go:462-473`).
|
||
3. The **per-app migrate** target list filters only current / decommissioned / disconnected /
|
||
schedulable — also no network filter (`handlers.go:674-679`).
|
||
4. `handleStorageMigrateApp` does **not** call `refuseNetworkLifecycle`, unlike its whole-namespace
|
||
sibling which does (`storage_handlers.go:397` vs `:410-424`), and `startMigration` has no guard
|
||
either (`internal/stacks/migrate.go:214-255`).
|
||
5. With `HDD_PATH` on the share, `namespaceRoot()` returns it as-is under Model A
|
||
(`internal/backup/backup.go:262-263`), so the app's Tier-1 unit is written to
|
||
`<share>/backups/primary/<app>/compose/app.yaml`.
|
||
6. FileBrowser binds that share at its **root** and serves it with `download: true`
|
||
(`internal/infra/infra.go:326`).
|
||
|
||
**Observed live on demo-hp, in the generated compose — the asymmetry is visible, not inferred:**
|
||
|
||
```
|
||
- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb <- drive: userdata-scoped
|
||
- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave <- network: ROOT-bound
|
||
- /mnt/sys_drive/felhom-data/userdata/import:/srv/beolvasas
|
||
```
|
||
|
||
**Today this is not a secret leak**, because the unit's `app.yaml` is secret-stripped
|
||
(`recovery_unit.go:73`). **D5 would make it one.** That is exactly the test §2 set, and D5 therefore
|
||
does **not** hold as written. → **R-108**
|
||
|
||
> **CLOSED (v0.187.0).** Links 2, 3 and 4 of the chain above are now guarded, and a **fifth** surface
|
||
> the chain did not list was found and guarded too: `handleStorageDecommission` mode=`migrate` checked
|
||
> only `req.Where` (the SOURCE) via `refuseNetworkLifecycle`, so a whole namespace could be
|
||
> decommissioned ONTO a NAS. Link 2's framing also understated the problem — the deploy **dropdown** is
|
||
> only a UI list; the boundary is the deploy **POST** (`internal/api/router.go`), which accepts any
|
||
> caller-supplied `HDD_PATH` and whose only other validation is `os.Stat` existence
|
||
> (`internal/stacks/deploy.go`). Filtering the list alone would have left the surface open. Links 5 and
|
||
> 6 are unchanged and still true — they simply can no longer be reached.
|
||
|
||
### 10.2 The gap register
|
||
|
||
| ID | Gap | Consequence |
|
||
|---|---|---|
|
||
| **R-102** | Tier-2 writes a full `recovery-unit/` mirror on every run and **no code path reads it** | Tier-2 is defeated in the drive-loss scenario it exists for (§7.2). Was C9-F4 |
|
||
| **R-103** | The Tier-2 no-coverage refusal **names** the working action but does not **route** to it | 45-or-43 of 53 apps dead-end at a message. Was C9-F1b |
|
||
| **R-104** | An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach, reported as *„ismeretlen okból"* | the offsite tier stays dead until a human unlocks. Was C9-F3 |
|
||
| **R-105** | Three hub-held DR records are empty on the whole live fleet: `hosts.dr_record_json`, `host_escrow.directive_json`, `dr_recipe.host_half.drives` | the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field |
|
||
| **R-106** | `dr_recipe.host_half.pbs.namespace` records `"root"` on every box | the recorded restore coordinate is wrong; real namespaces are per-customer |
|
||
| **R-107** | No offsite action unpacks the named-volume tars Tier-3 captures on every run | offsite alone cannot rebuild a named-volume app (§7.2) |
|
||
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
|
||
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
|
||
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
|
||
| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
|
||
| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
|
||
|
||
### 10.3 Divergences that are documented elsewhere and are not re-opened here
|
||
|
||
**[FACT]** `06-offsite-connectivity.md:19-21` describes the operator's public edge as a Cloudflare
|
||
Tunnel and states DooPlex has no public IP. Live DNS resolves `hub.felhom.eu` through a no-ip
|
||
DynDNS CNAME straight to DooPlex's own public address (INV Part E.3). That is a topology-document
|
||
issue, not a recovery-model one; recorded so the discrepancy is not lost.
|
||
|
||
---
|
||
|
||
## 11. Open decisions — for the operator
|
||
|
||
**Recorded, deliberately not answered.**
|
||
|
||
**A. Escrow custody.** Split custody (R **or** an offline operator key) versus a 2-of-3 threshold
|
||
across customer / hub / box drives.
|
||
*Recommendation on record:* **split custody, operator key held offline and never in the hub.* Note
|
||
the constraint from §2: the moment the operator key lives in the hub, "a compromised hub yields
|
||
blobs nobody can open" stops being true.
|
||
|
||
**B. Lost-R policy.** Under split custody the operator **can** recover. Is that the stated policy —
|
||
and if so, what identity check gates it? Today matrix row 13 has no route at all, and the customer
|
||
is not told that losing R costs them the host-loss route.
|
||
|
||
**C. RTO / RPO targets per scenario.** **None have ever been stated.** Without them §8 cannot judge
|
||
whether, for example, offsite-only recovery is acceptable for drive loss, or whether the 7-day PBS
|
||
cadence is adequate for row 8. The measured column is the input; the target column does not exist.
|
||
|
||
**D. Hetzner as a single failure domain.** restic (Storage Box `u629488`, sub-accounts per customer)
|
||
and PBS (Cloud server ep0 + a Cloud Volume) are both Hetzner. Whether they share an **account,
|
||
login and payment method** is unverified (INV Unknown U-1) — the question was deliberately not
|
||
answered by calling the provider API with the production token. Accept explicitly, or mitigate.
|
||
|
||
**E. `local` vzdump shares a physical device with the guest it backs up.** LIVE on both hosts:
|
||
`/var/lib/vz` (the archive target) and `local-lvm` (the guest's rootfs and both `backup=1`
|
||
mountpoints) are both on `/dev/sda3` → VG `pve`. The tier therefore protects against **corruption
|
||
and operator error only**, never against disk failure. Accept and name it honestly in the customer-
|
||
facing description, or move the target.
|
||
|
||
**F (added by §10.1, not in the original list). — ANSWERED 2026-07-30.** D5 could not be adopted
|
||
until R-108 closed. Closing R-108 *was* the intended path, and it is done (controller v0.187.0): the
|
||
operator ruled to refuse app namespaces on network storage rather than narrow the browsing surface,
|
||
because the share-root bind is load-bearing and cannot be scoped. **D5 is no longer blocked.** Whether
|
||
to now *implement* D5 remains an open scheduling decision, not a blocked one.
|
||
|
||
---
|
||
|
||
## 12. Evidence index
|
||
|
||
| Claim | Grade | Source |
|
||
|---|---|---|
|
||
| Tier-2 file restore, gap-fill and after total loss | **PROVEN-LIVE** | CAMPAIGN-9 A1/A3 |
|
||
| Tier-2 refuses without an outage for a no-coverage app | **PROVEN-LIVE** | v0.183.0 replay, `felhom.eu/REPORT.md:60` |
|
||
| Tier-1 unit restore **executes** | **PROVEN-LIVE** | CAMPAIGN-9 A2 |
|
||
| Tier-1 **content recovery after loss** | **UNPROVEN** | `CAMPAIGN-9…:825-827` |
|
||
| restic restore of app data (bytes) | **PROVEN-LIVE** | CAMPAIGN-8 R-87, 155.5 MB, 6/7 byte-identical |
|
||
| offsite reconstitution of a DB-indexed app | **PROVEN-LIVE** | destructive immich drill 2026-07-20, `00-capability-map.md:66,75` |
|
||
| offsite **place-to-live** as a distinct action | **UNPROVEN** | `CAMPAIGN-8…:520` |
|
||
| shares restore (files + definitions + credential) | **PROVEN-LIVE** | 2026-07-18, `00-capability-map.md:96` |
|
||
| `.fab` drive-to-drive round trip | **PROVEN-LIVE** | CAMPAIGN-6D P-FAB, 1.7 GB |
|
||
| `.fab` **browser upload** leg | **UNPROVEN** | `00-capability-map.md:67` |
|
||
| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
|
||
| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
|
||
| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
|
||
| unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) |
|
||
| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
|
||
| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
|
||
| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
|
||
| agent DR bring-up (`ModeDRGuestLoss`) | **NEVER EXECUTED** | `CAMPAIGN-8…:522` |
|
||
| host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` |
|
||
| host rebuilt as its former self | **NEVER DONE** | INV Part D1 |
|
||
| escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` |
|
||
| hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 |
|
||
| a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` |
|
||
|
||
---
|
||
|
||
## 13. What this document deliberately does not do
|
||
|
||
- It does not restate capability status — §8 cites the map, the map cites §8.
|
||
- It does not resolve the 7/53-vs-9/43/1 count (§6.2). Both are on the record; neither is adopted.
|
||
- It does not estimate a single RTO or RPO. Every blank in §8 is a real gap.
|
||
- It does not answer §11. Those are the operator's.
|
||
- It does not claim ratification.
|