admin
3c1882a0a4
gates / gates (push) Successful in 21s
Architecture: 08 records ruling A (45 m, the round-9 arithmetic, the cost) and the two new event types with their audiences; 03 records the slow counter as built; 07 records the restore-record persistence as a REVERSED design for the restore record only; CONTEXT.md carries the day's rulings. Guide: the recovery code moves after the first apps and waits for the yellow bar. Register: R-549 and R-550 closed PROVEN-LIVE; R-546 closed on red-proofed tests with its live walk owed by R-551 (no Tier-0 box is paused AND agent-connected). R-552 filed: an interrupted-restore notice for a removed app never clears - found in my own v0.246.0 after the release was built. Evidence: Part A (hub prints 45m/1h30m), Part C delivery on HP and N100, B.4(a) live proof and its teardown. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
1273 lines
116 KiB
Markdown
1273 lines
116 KiB
Markdown
# 07 — The recovery model
|
||
|
||
> | | |
|
||
> |---|---|
|
||
> | **Status** | **NOT RATIFIED.** Ratification is Viktor's review, not an editor's. |
|
||
> | **Written** | 2026-07-28 (full rewrite; supersedes the 2026-07-14 DRAFT entirely) |
|
||
> | **Verified against** | controller **v0.183.0** · agent **v0.110.0** · hub **v0.80.0** · catalog `4252121` · repo HEADs `felhom.eu ff050cf`, `felhom-controller fd50a73`, `felhom-agent d5c7691` |
|
||
> | **Live fleet at verification** | demo-felhom + demo-hp, both guest 9201, both on the versions above |
|
||
> | **Freshness** | **CURRENT** as of 2026-07-28. Per standing ruling **S-2**, mark this STALE the moment it is more than a few trains behind. The document it replaces was DRAFT since 2026-07-14, verified against controller v0.132.0 — **fifty-one versions stale** — and was cited as authoritative throughout that time. |
|
||
>
|
||
> **How to read this document.** Two kinds of statement appear, and they are always labelled:
|
||
>
|
||
> - **[DESIGN]** — a decision taken in the 2026-07-28 architecture discussion (D1–D6, §3–§5, §9).
|
||
> Not derived from code; the code may not implement it yet. Where it does not, §10 says so.
|
||
> - **[FACT]** — an observed property, carrying a `file:line`, a live command output, or a citation
|
||
> to `_recovery-inventory-2026-07-28.md` (below: **INV**).
|
||
>
|
||
> Where the model is silent, this document says **OPEN** rather than filling the gap.
|
||
>
|
||
> **Primary input:** `_recovery-inventory-2026-07-28.md` (read-only inventory, 2026-07-28) — cited
|
||
> throughout as **INV Part n**. Every number in §6, §8 and §11 traces back to it.
|
||
|
||
---
|
||
|
||
## 1. Purpose and scope
|
||
|
||
This document describes **how a Felhom customer gets their system back**, and who can do it.
|
||
|
||
It replaces a document that described **where copies are written**. That was the wrong frame, and
|
||
§7 explains why: a set of copies is not a set of recovery routes, and the previous document's tier
|
||
table could be entirely satisfied while a real recovery was impossible.
|
||
|
||
**In scope:** the recovery model — trust boundary, lanes, the three-part model, the tiers as
|
||
*inputs to recovery*, the dependency chain between them, the failure→recovery matrix, and the
|
||
encryption policy.
|
||
|
||
**Out of scope, deliberately:** implementation specs (they live in task specs), the capture-set
|
||
algorithm (`internal/appbackup/captureset.go` and its tests are the source of truth), and per-tier
|
||
operational runbooks (`documentation/runbooks/`).
|
||
|
||
**Authority split.** This document is authoritative for the **failure→recovery matrix** (§8). The
|
||
capability map (`00-capability-map.md`) stays authoritative for **per-capability status**. Neither
|
||
restates the other; §8 rows are cited from the map, not copied into it.
|
||
|
||
---
|
||
|
||
## 2. The trust model
|
||
|
||
**[DESIGN] The operator holds root SSH on every box.** That is a fact of the product — the agent is
|
||
operator-tier, the host is operator-managed, and break-glass exists precisely so the operator can
|
||
get in when nothing else works. **"The operator cannot read customer data" was never the security
|
||
property**, and no part of this document may be read as claiming it.
|
||
|
||
**[FACT]** The mechanics that make this concrete:
|
||
|
||
- The operator's OOB SSH public key is pushed to every box from the hub
|
||
(`hub_settings.oob_operator_ssh_pubkey` → `/var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op`,
|
||
93 bytes, LIVE on both hosts — INV Part C, row 21).
|
||
- The break-glass `root@pam` console password for every host is stored in the hub and retrievable
|
||
with the operator's global key (`documentation/runbooks/break-glass.md:42-48`; LIVE:
|
||
`host_recovery` holds 3 rows — INV Part D2.1).
|
||
- Guest data is reachable from the host by definition: `pct exec`, and the data drives are bind
|
||
mounts on the host (`mp8 /mnt/felhom-drives`, LIVE `pct config 9201` on both hosts).
|
||
|
||
**[DESIGN] What R (the customer's recovery code) does provide, and must keep providing: the hub
|
||
alone is not enough.** A compromised hub yields blobs nobody can open. This property holds **only
|
||
while the operator's own key is not stored in the hub** — which is why §11-A is an open decision and
|
||
why its recommendation on record is "operator key held offline and never in the hub".
|
||
|
||
**[FACT]** The escrow is genuinely zero-knowledge today: `host_escrow` rows carry
|
||
`posture = zero_knowledge`, a 383-byte blob and a 572-byte identity blob, and the hub holds only a
|
||
`restic_pw_sha256` **hash** beside them (LIVE, both hosts — INV Part D2.1). R exists in **zero**
|
||
system copies by design (INV Part C, row 24).
|
||
|
||
So the honest statement of the property is:
|
||
|
||
> **The hub cannot open the escrow. The operator can reach a live box. Neither fact substitutes for
|
||
> the other, and R is what keeps the first true.**
|
||
|
||
**[DESIGN] Because R is the only key, the household is ASKED for it from the first login** (controller
|
||
v0.245.0, R-543). Off-site backup is enabled by default but does not RUN until the ceremony is done,
|
||
so the ask is not a nicety — it is the step that turns the default-on tier into an actual copy. The
|
||
volunteer guide asks for it immediately after the dashboard password and before the first app
|
||
(`runbooks/VOLUNTEER-first-hour.md` §6), and the product repeats the ask on every page until it is
|
||
done (§6.1).
|
||
|
||
---
|
||
|
||
## 3. The two lanes (D1)
|
||
|
||
**[DESIGN] Recovery is split into two lanes with different owners. This is a product decision, not a
|
||
limitation.**
|
||
|
||
### Lane 1 — the customer, unassisted: files and app data
|
||
|
||
The customer owns their data and can get it back themselves, through the „Visszaállítás" surfaces,
|
||
with nothing but their dashboard password. No operator, no ticket, no scheduling.
|
||
|
||
**[FACT] What Lane 1 contains today** (INV Part A.1 — seven paths, all behind the controller's
|
||
`RequireAuth` gate, which is the **customer-owned** password: `internal/web/auth.go:34-35` puts
|
||
`settings.json → password_hash` ahead of the operator-provisioned `controller.yaml` value):
|
||
|
||
| # | Surface (HU) | Endpoint | Semantics |
|
||
|---|---|---|---|
|
||
| 1 | „Visszaállítás indítása" | `POST /backup/restore` | destructive — rebuilds the app from its Tier-1 unit |
|
||
| 2 | „Fájlok visszaállítása" | `POST /backup/tier2/restore` | additive, missing-only |
|
||
| 3 | „Visszaállítás a távoli tárolóból" | `POST /backup/offbox/restore` | non-destructive, to a verification copy |
|
||
| 4 | „Helyreállítás az élő adatok közé" | `POST /backup/offbox/place` | additive, missing-only **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. `audits/SPIKE-restic-restore-test-2026-08-31.md`. |
|
||
| 5 | „Teljes visszaállítás (fájlok + adatbázis)" | `POST /backup/offbox/reconstitute` | destructive to files + DB, never deleting |
|
||
| 6 | „Megosztások visszaállítása" + place | `POST /backup/shares/{restore,place}` | additive |
|
||
| 7 | `.fab` import | `POST` → `apiImportStart` | destructive re-import of one app |
|
||
|
||
**[FACT] What is proven in Lane 1** (INV Part G.1): paths 2 and 6 are proven live end-to-end;
|
||
path 5 is proven live through a destructive drill (photos deleted, trash emptied, restored, timeline
|
||
confirmed); path 3 is proven for **bytes** only; path 7 is proven for the drive-to-drive circle but
|
||
its browser-upload leg is not; path 1 is proven to **execute** but its **content recovery after real
|
||
loss has never been demonstrated** — the single most valuable unproven item in the system
|
||
(`audits/CAMPAIGN-9-restore-proof-2026-07-28.md:825-827`); path 4 has never been exercised as a
|
||
distinct action (`audits/CAMPAIGN-8-backup-restore-2026-07-27.md:520`).
|
||
|
||
**[FACT] One honest residual on the whole lane:** every proven run was performed by the **operator**,
|
||
not by a customer. `00-capability-map.md:75` still carries "A customer (not the operator) performs a
|
||
restore via UI alone" as **MISSING as evidence** — by absence of the run, not by a product gap.
|
||
|
||
**[RULING 2026-09-17, operator — a design choice REVERSED] The restore record survives a restart
|
||
(R-550, controller v0.246.0).** The async restore status (`internal/backup/opstatus.go`) was in memory
|
||
by choice — „same precedent as notification cooldowns". Chaos night round 10 measured the cost: a
|
||
restore accepted, the box hard-reset four seconds later, and the status answering the Go zero value, so
|
||
the household could never learn whether it finished. **For the restore record only**, it is now written
|
||
atomically to `restore-status.json` in the controller's state directory at both ends of an op; at
|
||
startup a record still marked running becomes a failed, interrupted result („A visszaállítás megszakadt
|
||
(a doboz újraindult) — indítsd el újra."), shown per app on the restore page until that app's next
|
||
restore, and raised once as `restore_interrupted`. **Notification cooldowns stay in memory** — the
|
||
precedent is kept for what it was written for.
|
||
|
||
### Lane 2 — the operator: guest and host recovery
|
||
|
||
Rebuilding an LXC guest, rebuilding a host, and re-establishing a box's identity are **operator
|
||
work, by design and by contract**. They are not customer-facing and are not going to be.
|
||
|
||
**[FACT] What Lane 2 contains** (INV Part A.2): the scheduled restore-test, `pct restore`, raw
|
||
`proxmox-backup-client restore`, raw `restic restore`, the agent's DR bring-up
|
||
(`--selftest=bring-up -mode dr`), the host-loss plan builder, and the escrow-consume ceremony. Each
|
||
needs root on the host or a CLI flag; none is reachable from any customer surface.
|
||
|
||
**[FACT] What is proven in Lane 2** (INV Parts G.1, G.3): whole-guest `pct restore` from both tiers
|
||
is proven with exact mount parity; the unattended restore-test is proven and currently running on
|
||
both boxes; a corrupted snapshot is proven to fail cleanly. **The DR bring-up path has never been
|
||
executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by construction**
|
||
(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
|
||
(INV Part D1).
|
||
|
||
### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03)
|
||
|
||
**[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to
|
||
fire on an interval started at daemon start. It no longer does. The rule is:
|
||
|
||
> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
|
||
> The tier is **DUE** when **A** exists and **A has not already been proven**.
|
||
|
||
So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather
|
||
than the process's uptime:
|
||
|
||
| tier rhythm | what is proved, and when |
|
||
|---|---|
|
||
| daily (host tier) | yesterday's archive, once a day |
|
||
| weekly (offsite tier) | last week's archive, once a week |
|
||
| newborn (no archive yet) | nothing — **UNKNOWN, never a fault** |
|
||
|
||
**The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest
|
||
archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive
|
||
age to zero long before it reaches the lag — so the literal reading silently switches restore-testing
|
||
off for the tier that matters most.
|
||
|
||
What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its
|
||
journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on
|
||
success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker
|
||
remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one
|
||
due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN).
|
||
|
||
**The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very
|
||
cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval
|
||
just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay
|
||
tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed
|
||
archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite
|
||
retention), and falling back to the tier's declared rhythm when history is too short to observe one.
|
||
|
||
### Why the split is right, stated once
|
||
|
||
**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
|
||
"restore the LXC and re-attach mp8 by durable_id without binding another guest's bootstrap
|
||
credentials" — a hazard serious enough to have its own runbook page
|
||
(`runbooks/RUNBOOK-manual-guest-restore.md:45-48`, F-OPS). Putting Lane 2 behind the operator is
|
||
what lets Lane 1 be a button instead of a procedure.
|
||
|
||
**[FACT] And as of D5 (v0.188.0) the split is real, not just intended.** Until then Lane 1 only *looked*
|
||
independent: an app's files sat on the customer's drive but could not be brought back without secrets
|
||
that lived solely in the guest, so the fast customer lane was silently conditioned on the slow
|
||
operator lane (§7.1 leg 1). The unit now carries those secrets, so **Lane 1 needs the drive and nothing
|
||
else** for app rebuild (§7.4). The one honest caveat: Lane 1's *file-merge* surface (Tier-2 additive)
|
||
still reads its destination from the guest's `settings.json`, so that surface remains guest-conditioned
|
||
— D5 removed the secrets leg, not the living-app leg.
|
||
|
||
---
|
||
|
||
## 4. The three-part model (D4)
|
||
|
||
**[DESIGN] Recovery needs three things, and they live in three different places.** Losing one is a
|
||
different problem from losing another, and the matrix in §8 is organised around that.
|
||
|
||
| Part | Holds | Where | Lost when |
|
||
|---|---|---|---|
|
||
| **Recipe** | scaffolding — guest sizing, drive inventory, PVE storage definitions, PBS coordinates, app inventory + storage bindings. **No secrets, no bytes.** | the hub | the hub is lost |
|
||
| **Escrow** | the host identity key material and the restic repo password | the hub, R-wrapped | the hub is lost, **or** R is lost |
|
||
| **Bytes** | Tier-1 / Tier-2 / Tier-3 / whole-guest | the tiers | the relevant medium is lost |
|
||
|
||
**[FACT] The Recipe exists and is current.** `dr_recipe` holds 6 rows; demo-felhom's was updated
|
||
`2026-07-28 17:31:13` and demo-hp's `17:25:31` — i.e. within one report cycle (LIVE, INV Part D2).
|
||
It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_bindings`.
|
||
|
||
**[FACT] The Escrow exists and is zero-knowledge.** `host_escrow`: 2 rows, `posture` `zero_knowledge`;
|
||
the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}`
|
||
(`felhom-agent/internal/escrow/identity.go:26-39`).
|
||
|
||
**[FACT] Three parts of the Recipe are empty or wrong on the live fleet**, and they are exactly the
|
||
parts a host-loss recovery would read (INV Part D2.3): `hosts.dr_record_json` is `{}` on all three
|
||
hosts; `host_escrow.directive_json` is `{}` on both escrowed hosts; `dr_recipe.host_half.drives` is
|
||
`[]` on every customer including two with enrolled data drives; and `dr_recipe.host_half.pbs.namespace`
|
||
reads `"root"` while the real namespaces are `demo-felhom` / `demo-hp`. → **R-105**, **R-106**.
|
||
|
||
---
|
||
|
||
## 5. Encryption policy (D2)
|
||
|
||
**[DESIGN] Encryption follows the boundary, not the tier.**
|
||
|
||
### On the customer's own premises: plaintext
|
||
|
||
Tier-1 recovery units, Tier-2 cross-drive copies, the app data on the drives, and the local vzdump
|
||
archives on the host are **unencrypted, deliberately**.
|
||
|
||
The reasoning, recorded so nobody "hardens" this later:
|
||
|
||
1. **It buys nothing against a real threat.** Someone who can take the second drive can take the
|
||
first. Local encryption defends against a threat model — physical theft of *only* the backup
|
||
medium — that does not describe a home server where both media sit in the same box.
|
||
2. **It adds a key-loss path that turns a working backup into a brick.** Every local encryption key
|
||
is one more thing that must survive the disaster it exists for, and the system already has one
|
||
such dependency it is trying to reduce (§7).
|
||
3. **It would break browsing, which is a feature.** FileBrowser and SMB let the household see and
|
||
use their own files. An encrypted local copy is not browsable, and the „Megosztás" and
|
||
FileBrowser surfaces are part of the product, not an accident.
|
||
|
||
**[FACT]** The local plaintext posture is real and verifiable: `dir: local` in `/etc/pve/storage.cfg`
|
||
carries no `encryption-key` (LIVE, both hosts), so the daily whole-guest archive is a plain
|
||
`.tar.zst` — 5.93 GB on demo-felhom, 1.68 GB on demo-hp (LIVE, INV Part B.3). **That archive is the
|
||
single highest-value object on the box**: it contains `encryption.key`, the restic `repo_password`,
|
||
the offbox `ssh_key`, `settings.json` and `controller.yaml` (all under `/var/lib/docker` = `mp0`,
|
||
`backup=1`). Naming that plainly is part of the policy, not an argument against it.
|
||
|
||
### Leaving the premises: encrypted, and the provider must not be able to read it
|
||
|
||
**[FACT]** Both offsite tiers encrypt client-side:
|
||
|
||
- **restic (Tier-3):** repo password is a 256-bit hex value generated once on the box and never
|
||
logged (`internal/backup/offbox.go:392-410`); restic is invoked with
|
||
`RESTIC_PASSWORD_FILE` (`:608`).
|
||
- **PBS (whole-guest offsite):** `/etc/pve/storage.cfg` carries a **per-customer** `encryption-key`
|
||
and the fingerprints differ between demo-felhom and demo-hp (LIVE). The live restore command line
|
||
observed on demo-felhom carries `--crypt-mode=encrypt` (INV Part A.2.3). Per-tenant encryption is
|
||
why cross-customer dedup is impossible on the shared datastore — a cost consequence, recorded, not
|
||
a defect.
|
||
|
||
### The one exception, stated because it is not covered by either rule
|
||
|
||
**[FACT]** The `.fab` bundle writes `app.yaml` with **decrypted plaintext secrets, deliberately**,
|
||
and its password is **optional** (`internal/appexport/export.go:484,506-511`; the generated file's
|
||
first line is literally `# Exported by felhom-controller — plaintext secrets`, `:531`;
|
||
`Encrypted: req.Password != ""`, `:307`). A `.fab` can also be written to a registered **network**
|
||
share, because `storageDriveList()` does not filter network paths
|
||
(`internal/web/handler_export.go:377-387`). This is a portability artifact, not a tier — but it is
|
||
the one place where a customer action can put every secret of one app onto a NAS in plaintext.
|
||
Recorded here so the encryption policy is not read as covering it. → **R-108** (same root cause).
|
||
|
||
---
|
||
|
||
## 6. The tiers — what each captures
|
||
|
||
The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what they can actually do.
|
||
|
||
**[RULING 2026-09-16, operator] Tier 3 (off-site) is ON for every new customer — shared, 100 GB soft
|
||
quota.** Opting a customer out stays the per-customer exception, as DR-tier-by-default has been since
|
||
2026-07-12. **The reason is this document's own [FACT] two sections down:** the whole-guest tiers do
|
||
not carry `mp8 /mnt/felhom-drives`, and a Tier-1 unit has no file leg — so for the four class-A apps
|
||
a one-drive box with Tier 3 off keeps the household's files in NO tier at all. That was measured on a
|
||
fresh box the same day: five photos put into Nextcloud, deleted, restored from the box's own backup,
|
||
and none of them opened (`audits/DRILL-prove-fixes-0243-2026-09-16.md`, R-537/R-538). Implemented in
|
||
hub v0.116.0 (`handleConfigNewForm`). **What this ruling does NOT change: what any tier captures.**
|
||
|
||
|
||
**The app update's safety precondition accepts ANY tier** (operator ruling 2026-09-13, controller
|
||
v0.239.0, R-475): the first fresh copy in the order Tier 2, Tier 1, Tier 3; with none, it backs up
|
||
first. Design: `09-update-architecture.md` §3 decision 8. **What a Tier-1 route back restores is only
|
||
what the unit holds** — for an app whose data is a bind mount that is the definition, not the data
|
||
(R-479). **Removal and the tiers (controller v0.240.0):** removing an app with its backups KEPT keeps
|
||
the unit, the Tier-2 mirror AND the Tier-2 record, so „Teljes visszaállítás" still works afterwards
|
||
(R-486); „Mentési adatok törlése" deletes the unit, every mirror and the app's backup preferences, and
|
||
never touches off-site snapshots (R-474). A removed app whose unit was kept is listed on both local backup pages with its restore since controller v0.242.0 (R-487): **the local lists are keyed on the drives, not on what is deployed** — the rule R-237 set for the off-site list — and the restore opens the unit where it sits, a data drive included. **For a bind-data app the update's tier order is second drive → off-site → own unit (R-479, v0.241.0), because the unit does not hold the files.**
|
||
|
||
### 6.1 The four tiers, as configured on the live fleet
|
||
|
||
| Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted |
|
||
|---|---|---|---|---|---|
|
||
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
|
||
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
||
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
||
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
|
||
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` — **and the box asks for none** (R-191) | **yes** (per-customer key) |
|
||
|
||
> **[FACT] Tier-3 has a fifth state the table above does not show: PAUSED (R-543, controller
|
||
> v0.245.0).** Off-site is ON by default from hub v0.116.0, and a run does not start until the
|
||
> household has performed the escrow ceremony — `tier3State` calls this `escrow_pending` and the page
|
||
> says „Kulcsletétre vár". **This is the design, not a defect:** the escrow is zero-knowledge (§2),
|
||
> the household's recovery code is the only key, and a run started without one would write a copy
|
||
> nobody could ever open. What was wrong until v0.245.0 is that **nothing asked the household for the
|
||
> code**, so a fresh box could sit paused indefinitely while its Tier-1 row promised that the off-site
|
||
> copy protected the app's files. Since v0.245.0 every dashboard page carries the reminder (the R-241
|
||
> bar, second instance) and the Tier-1 sentence renders by state — „védené … szünetel" while paused.
|
||
> Measured on a fresh box 2026-09-16: zero snapshots, and the page said the files were protected.
|
||
|
||
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
|
||
|
||
|
||
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
||
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
||
(`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`.
|
||
|
||
**[FACT] What the whole-guest tiers do NOT carry.** `mp8 /mnt/felhom-drives` and
|
||
`mp9 /etc/felhom-bootstrap` are **host bind mounts** and are out of `vzdump` scope entirely (LIVE
|
||
`pct config 9201`, both hosts). So the whole-guest tiers carry the guest and **none of the customer's
|
||
data drives** — 916 GB on demo-felhom, 938 GB on demo-hp.
|
||
|
||
### 6.2 Coverage per app class — and an unresolved count
|
||
|
||
**[FACT]** Of 53 catalog templates, **52 keep data in Docker named volumes**; exactly **13** carry a
|
||
`backup:` block, and those 13 are exactly the templates that bind `${HDD_PATH}` / `${USERDATA_PATH}` /
|
||
`${IMPORT_PATH}` at all; **14** have a database service (INV Part B.1, computed against catalog
|
||
`4252121`).
|
||
|
||
Applying the classifier's documented two-level default (`internal/appbackup/classify.go`: explicit
|
||
entry wins; else a `:ro` reader is *excluded*; else a writable bind is *mandatory*):
|
||
|
||
| tier filter | templates with at least one file leg |
|
||
|---|---|
|
||
| **Tier-3** (`mandatory` only) | **4** — calibre-web, immich, nextcloud, paperless-ngx |
|
||
| **Tier-2** (`mandatory + optional`) | **7** — the 4 above plus audiobookshelf, komga, romm |
|
||
| legacy resolver path (apps with no block) | **0** — no non-block template binds a namespace path |
|
||
|
||
> ### ✔ RESOLVED 2026-08-31 (controller v0.229.0) — **A = 7 · B = 45 · C = 1**
|
||
>
|
||
> | source | count | verdict |
|
||
> |---|---|---|
|
||
> | C9-F1 Phase 0, as shipped | **A = 9 / B = 43 / C = 1** — `felhom.eu/REPORT.md:17-24` (since overwritten), restated `backlog/OPEN-ITEMS.md:31` | **WRONG by two apps** |
|
||
> | INV Part B.1, independent enumeration at catalog `4252121` | **A = 7 / B = 45 / C = 1** | **CORRECT** |
|
||
> | Measured at catalogue `459766cb16395fd1d1a66282f5cc6da59ead5924`, 2026-08-31 | **A = 7 / B = 45 / C = 1** | adopted |
|
||
>
|
||
> **The method, so it can be re-run rather than re-argued.** A throwaway `main` inside the controller
|
||
> module drove the PRODUCTION rule over all 53 template directories — `stacks.LoadMetadata` (the single
|
||
> validation choke point, so a rejected `backup:` block degrades to legacy exactly as it does live) →
|
||
> `stacks.ParseComposeClassifiableBinds` → `appbackup.ClassifyBinds` → `appbackup.ComputeCaptureSet` at
|
||
> `TierSecondary`, with the legacy branch falling back to `AppDataBindsPresent` + `AppDataDirNames` as
|
||
> `backup.tier2CaptureSet` does. **A = at least one leg survives that pipeline.** 13 templates carry a
|
||
> valid `backup:` block; 40 are legacy and none of them binds a namespace path, so all 40 are B or C.
|
||
>
|
||
> **A (7):** audiobookshelf, calibre-web, immich, komga, nextcloud, paperless-ngx, romm.
|
||
> **C (1):** bentopdf — it declares no `volumes:` key and no `${…_PATH}` bind at all.
|
||
> **B (45):** everything else.
|
||
>
|
||
> **How the earlier disagreement arose, established rather than guessed.** Phase 0's own write-up
|
||
> (controller `CHANGELOG.md`, v0.183.0) says four apps — plex, jellyfin, emby, navidrome — are in B
|
||
> "only because their single bind is a `:ro` media mount, which `ClassifyBinds` correctly excludes". It
|
||
> applied the `:ro` **default** rule. The two apps it therefore missed are **radarr and sonarr**: their
|
||
> `${USERDATA_PATH}/media/*` and `${USERDATA_PATH}/downloads` binds are **writable**, so the `:ro` rule
|
||
> does not reach them, and they are excluded by an **explicit** `class: excluded` entry instead. 9 − 2 =
|
||
> 7 and 43 + 2 = 45, which is exactly the gap. No catalogue file was changed; this is a measurement.
|
||
|
||
**[FACT] The class-B consequence is real regardless of which count is right.** For an app whose data
|
||
lives entirely in named volumes, the Tier-2 copy holds a full `recovery-unit/` and **no readable
|
||
leg** — LIVE on demo-felhom, `backups/secondary/bookstack/` and `.../docmost/` contain
|
||
`recovery-unit` and nothing else, at 156 MB and 86 MB (INV Part B.2). Since v0.183.0 the button
|
||
refuses **before** stopping the app and names the action that works.
|
||
|
||
### 6.3 Where capture and restore disagree
|
||
|
||
**[FACT]** Three asymmetries, each source-cited:
|
||
|
||
| tier | captured | read back by that tier's restore | gap |
|
||
|---|---|---|---|
|
||
| Tier-1 | unit incl. volume tars + DB dumps | all of it | none |
|
||
| Tier-2 | unit mirror **+** file legs | the file restore reads `hdd/` and `userdata/`; **since controller v0.218.0's Tier-3 sibling and now v0.229.0, a SECOND action reads the unit mirror itself** (`RestoreTier2Unit` → `RestoreFromRecoveryUnitAt`, `tier2_restore.go`) | **CLOSED — R-102** |
|
||
| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay **+ the named-volume tars, replayed from the scratch unit** (`offbox_reconstitute.go` `volReplay`, controller **v0.218.0**); the unit itself is still **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **CLOSED — R-107** |
|
||
|
||
**[FACT] 2026-08-22 — the Tier-3 row above was corrected; the Tier-2 row was NOT.** Until controller
|
||
**v0.218.0** this table said *"no offsite action unpacks the named-volume tars it captures"*, and that
|
||
was true from the day Tier-3 shipped until 2026-08-21. **R-107 closed in v0.218.0**: `volReplay`
|
||
(`offbox_reconstitute.go`) replays the scratch unit's `volume-dumps/` into the live named volumes,
|
||
proven live on `demo-hp`. The old sentence is kept here, in the past tense, because a correction that
|
||
erases what was believed leaves the next reader no way to tell a fixed gap from one that was never
|
||
noticed.
|
||
|
||
**[FACT] 2026-08-31 — the Tier-2 half is now closed too (R-102, controller v0.229.0), and the old
|
||
sentence is kept here in the past tense for the reason the paragraph above gives.** Until v0.229.0 this
|
||
table said of Tier-2: *"the unit mirror is read by nothing — `RecoveryUnitPath` resolves to
|
||
`backups/primary/` (`appbackup/paths.go:46-48`)"*, and that was true from the day Tier-2 shipped until
|
||
2026-08-31. The mechanism was a hard-coded `primary` segment: every reader of a recovery unit could
|
||
only name a path under it. `appbackup` now also exposes four **unit-directory-relative** primitives,
|
||
`Manager.RestoreFromRecoveryUnitAt(stack, unitDir)` holds the restore body, and `RestoreTier2Unit`
|
||
points it at `<dest>/backups/secondary/<app>/recovery-unit/`. Proven live on `demo-hp` **with the
|
||
primary unit moved aside** — `audits/DRILL-r102-tier2-unit-2026-08-31/`.
|
||
|
||
**Two things did NOT change, and both are load-bearing.** THE SOURCE MOVED; THE DESTINATION DID NOT —
|
||
data still lands in the live named volumes and the live database container. And the FILE restore's
|
||
reach is unchanged: `CanRestore()` still answers only "are there file legs?", and
|
||
`tier2UnitNotCoveredMsg` is still appended where that restore runs, so a clean file result never reads
|
||
as a clean bill of health for the database.
|
||
|
||
**[FACT]** Tier-2's gap was the sharper one because of *when* it bit: Tier-2 exists for the case
|
||
where the primary drive is lost — and in exactly that case the primary unit is gone while this
|
||
mirror survives on the second drive. Until v0.229.0 it was unreachable by any customer action.
|
||
|
||
**[FACT] 2026-08-23 — taking the undo copy used to DESTROY the app's own database backup (R-361).**
|
||
`writeSafetyDump` called `DumpOne` into the app's own unit directory and renamed the result to
|
||
`pre-restore-*` afterwards. `DumpOne` writes `<stack>-<dbtype>.sql` — the app's canonical dump, the
|
||
name the replay loop matches exactly — so **every safety dump overwrote the app's real backup and then
|
||
moved it away**, leaving the app with no database backup of its own until the next nightly run. A local
|
||
restore-from-unit in that window finds no `.sql` and tells the customer the app never had a database.
|
||
The comment beside it asserted the rename meant it *"can never overwrite the app's real dump"*; it was
|
||
false as written and stood for four months. **Measured before the fix on `demo-hp` 2026-08-22:**
|
||
`docmost` and `bookstack` each held only `pre-restore-*` files and no canonical dump.
|
||
|
||
Fixed in controller **v0.221.0**: `DumpOneTo` takes an explicit final path and derives its own `.tmp`
|
||
from it, so neither the destination nor the scratch file can collide with a nightly dump running
|
||
beside it. **Proven the only way it can be** — the canonical dump's sha256, unchanged across a
|
||
restore, on both engines.
|
||
|
||
**[DESIGN] 2026-08-23 — `db_dumps` lists the app's OWN dumps, not the undo copies.** They are local
|
||
material for a restore that went wrong, not part of the app's recovery set: nothing reads them from
|
||
the manifest, and three per app were being pushed off-site permanently for no recovery value. The
|
||
files are neither deleted nor hidden — their visibility is a separate recorded decision and it stands.
|
||
**One consequence, recorded because it bit within minutes:** a stable `db_dumps` lets
|
||
`CaptureRecoveryUnit`'s already-current early return fire, so anything that must happen on every
|
||
capture — such as bounding the undo copies — has to sit ABOVE that check, not after it.
|
||
|
||
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
|
||
Recorded here rather than only in a closed register row, because a decision that survives only inside
|
||
a closed work item is a decision nobody will find.
|
||
|
||
1. **Replay.** The snapshot's `.sql` is imported into the app's database, with only the DB service up
|
||
(R-47). A pre-restore copy of the LIVE database was taken first and is on disk (R-43); the restore
|
||
refuses outright if it could not be taken.
|
||
2. **Rollback (controller v0.220.0, R-379).** If the replay fails, the product re-applies that undo
|
||
copy itself. **The whole set for this run** — an app with two databases gets two undo files, and
|
||
restoring only the first would leave the other half-written — matched on **the run's own stamp**,
|
||
never on the `pre-restore-` prefix, because several runs' copies coexist in the same directory. It
|
||
runs with the DB service still up and before any restart, so the app never observes the half
|
||
state, and into a **re-discovered** container: the DB-only start re-creates it, so the id captured
|
||
at dump time is dead by rollback time (v0.220.1, found by the first live run). The app then starts
|
||
and the customer is told **both** that the restore failed and that their data is as it was.
|
||
3. **Hold (v0.220.0, operator ruling 2026-08-22).** If the rollback ALSO fails, the app is **held
|
||
stopped**, not started. A running app on a half-written database lets the customer type into it and
|
||
makes the damage permanent. The hold is persisted, every start path refuses it with a reason and a
|
||
route, the app-stop marker is ended so nothing auto-restarts it at the next boot, and the app reads
|
||
**red** rather than green. An operator clears it with `--clear-restore-hold`, which requires a
|
||
controller restart.
|
||
|
||
**Why a rollback and not an engine flag.** Postgres gained `--single-transaction` in the same release
|
||
and that does make its replay all-or-nothing — but **MariaDB's DDL is not transactional**, so a
|
||
partial apply there is unavoidable at the engine. Measured 2026-08-22: the same truncated dump left
|
||
Postgres emptied and crash-looping, and left MariaDB with its user data intact, its schema-version
|
||
table wiped to zero rows, and the app reporting `health=healthy, running=true, restarts=0`. The flag
|
||
is a belt; the rollback is the fix. **R-379, R-380.** Evidence:
|
||
`audits/DRILL-r379-rollback-2026-08-22/`.
|
||
|
||
**[DESIGN] 2026-08-22 — the restore destination is resolved by the same rule as the capture
|
||
destination.** The drive if the app declares one (`HDD_PATH`), the system data path otherwise —
|
||
`Manager.GetAppDrivePath`, one expression, used by `CaptureRecoveryUnit` and, since controller
|
||
**v0.219.0**, by `ReconstituteFromOffsite` and `PlaceOffsiteRestore` too.
|
||
|
||
**[FACT] 2026-09-01 — the rule now has a FOURTH consumer and the "one expression" sentence is
|
||
true (R-414, controller v0.232.0).** `offboxRestoreScratchDir` — where a restore is UNPACKED, as
|
||
distinct from where it LANDS — never consulted `systemDataPath`, so on a box with no registered
|
||
storage path it refused, and the nightly proof could not run at all. **It was not excluded on
|
||
purpose:** R-356's own commit (`08eb1a6`) records in its tests that *"the prepared scratch still
|
||
resolves to the registered storage path … only the DESTINATION moves"* — it was out of scope, and
|
||
every fixture assumed a registered path exists. The one comment about a `systemDataPath` fallback
|
||
belonged to `PlaceOffsiteRestore`, concerned bulk **userdata**, and R-356 deliberately overruled it.
|
||
|
||
**But the fallback is SCOPED, and the scoping is the point.** The two callers ask different
|
||
questions, and one predicate answering both is R-356's own defect: a **unit-only** restore (the
|
||
R-87 proof) may fall back to the system data path, because §7 records as `[FACT]` that a driveless
|
||
app's unit already lives there indefinitely and that the same-device placement is *"intended, not
|
||
a defect"*; a **full** restore keeps the R-252 refusal, because it pulls the app's bulk userdata
|
||
onto what §2.2 makes a **state-only** tier.
|
||
|
||
The refusal that protects a drive app from being restored onto the wrong disk (R-253, R-351) applies
|
||
to apps that **have a drive to get wrong**. It used to be reached by testing `HDD_PATH == ""`, which
|
||
also answered "is this app installed?" — one predicate for two questions. Measured in the catalogue at
|
||
`459766cb1639`: **53 templates, 13 declare `needs_hdd: true`, 40 declare `false`**, so for 40 apps that
|
||
test was permanently true and the off-site restore refused them forever, while they were running,
|
||
with a message telling the customer to reinstall them "in the same place" — a place those apps never
|
||
offer. **An app with no drive is not misconfigured** (§8 of `01-topology-and-trust.md` carries the
|
||
`[DESIGN]` marker); it is the majority case.
|
||
|
||
Since v0.219.0 the two questions are asked separately: *installed?* of `ListDeployedStacks()`, failing
|
||
CLOSED when there is no provider to ask; *where?* of `GetAppDrivePath`. A third refusal, with its own
|
||
sentence, covers installed-but-no-resolvable-data-root. **R-356**; reasoning also recorded in
|
||
`felhom-controller/CONTEXT.md`.
|
||
|
||
---
|
||
|
||
### 6.4 The customer's page speaks per tier, and a tier without storage is skipped (2026-09-15)
|
||
|
||
**What the page may claim** (controller v0.243.0 + agent v0.131.0, R-517). The whole-system tile used
|
||
to show the agent's single latest record — so a failed 0-byte PBS attempt read „Naprakész" and ticked
|
||
„Távoli rendszermentés — külön hardveren" (BIGNIGHT 2026-09-14). Now `GET /backup/status` returns per
|
||
tier the newest **success**, the last **attempt** kept apart, and whether the tier's storage exists.
|
||
The page shows each tier's newest success; a failed attempt under it as „sikertelen"; a tier whose
|
||
storage does not exist as „nincs beállítva". „Naprakész" and the remote tick are computed from
|
||
successes only. After an agent restart the success is read back from the tier's storage (size unknown).
|
||
|
||
**What a run may do** (R-518, cheap half). A tier the agent reports `storage: absent` is dropped before
|
||
anything is stopped, logged, and reported once as `backup_tier_skipped`; `unknown` is never skipped.
|
||
**Still open:** quiescing per tier, so a slow second tier does not keep every app down.
|
||
|
||
## 7. The recovery chain (D3) — the reason this document exists
|
||
|
||
**[DESIGN] 3-2-1 describes copies. It does not describe recovery.**
|
||
|
||
Three copies on two media with one offsite is a statement about *bytes surviving*. It says nothing
|
||
about whether the bytes can be turned back into a working system, and a system can satisfy 3-2-1
|
||
completely while having no executable recovery route for a given failure. That is not a hypothetical
|
||
here — §8 has rows where it is the actual state.
|
||
|
||
### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
|
||
|
||
> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
|
||
> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
|
||
> byte-identical. **It passed with an operator present, and four manual interventions stood between
|
||
> "the key is recoverable" and "the file is back" — none of which was in any design document.** They
|
||
> are recorded here because this is the section a future reader will use to answer *"can the customer
|
||
> do this alone?"*, and until 2026-08-05 the answer was no for reasons nothing wrote down.
|
||
|
||
| # | The step | Why it stopped a customer | Status |
|
||
|---|---|---|---|
|
||
| 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. |
|
||
| 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed` → `OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. |
|
||
| 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. |
|
||
| 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** was required. | **CLOSED — controller v0.199.0 + hub v0.96.0** (2026-08-05, operator ruling: automate it, and **the trigger is a state the BOX DECLARES**). The box now reports `offsite.state=needs_credential` when two local facts hold together — a fresh data area AND a hub-held recovery package — and the hub's `internal/offsiteheal` re-arms the stored one-time secret, minting only when there is nothing to re-arm. **Deliberately NOT automated: the escrow ceremony.** A credential is replaceable; the recovery code is not. |
|
||
|
||
> **[FACT] The honest current answer, updated 2026-08-05 (controller v0.200.0): all four are gone AND
|
||
> the customer is now offered the recovery.** A full-page screen meets the owner of a rebuilt box while
|
||
> the hub holds a sealed package the box cannot open: it explains the situation, says plainly that
|
||
> **nobody can replace a lost recovery code**, takes the code, opens the repository and **lists what is
|
||
> in it** — apps, dates, sizes. No step requires an operator, and no step requires a command line.
|
||
>
|
||
> **WHERE THAT STOPS, and it is a real stop.** The screen **unlocks and only unlocks**. It restores
|
||
> nothing. Putting files back is per-app, lives in the backups area, and the step after the listing —
|
||
> a customer seeing what would change before anything is overwritten — is **not built** (→ R-213). A
|
||
> screen that unlocks and then offers to overwrite is two decisions wearing one button, which is why
|
||
> the operator ruled them separate.
|
||
>
|
||
> **What is still owed as evidence:** the final unlock has never been driven with a CORRECT code
|
||
> through the page (no code was kept for demo-felhom's orphaned history; demo-hp's is operator-held),
|
||
> so the live run exercised handler → agent → hub fetch → age KDF and stopped at the unseal. **And the
|
||
> whole journey has not been re-walked end to end since these fixes** — the closures are proven
|
||
> individually, not as one uninterrupted run. That re-walk is one more drill.
|
||
>
|
||
> **WHY THE TRIGGER FOR STEP 4 IS A DECLARATION, recorded here because it is the design and not an
|
||
> implementation detail:** from the hub, an ABSENT off-site object means *never configured*,
|
||
> *mid-restart*, *a transient config read failure* OR *rebuilt and stranded*, and the hub cannot
|
||
> distinguish them. The BOX can, from two local facts it holds with certainty. So the box states its
|
||
> condition and the hub acts on a stated request — never on a silence. Both facts are required:
|
||
> freshness alone is a box that never had off-site backups, and an escrow alone is a healthy box.
|
||
>
|
||
> **The split that is deliberate and must not be widened: CREDENTIAL AUTOMATIC, KEY CUSTOMER-PRESENT.**
|
||
> A credential is transport and is replaceable; the recovery code is not, because only the customer
|
||
> holds it. Nothing in this chain runs, or asks for, an escrow ceremony.
|
||
|
||
### 7.1 The dependency graph
|
||
|
||
> **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below
|
||
> (**secrets**) is **no longer true for Tier-1 and Tier-2**: the local recovery unit now carries the
|
||
> portable secret class, so those two tiers need the **drive and nothing else**. The graph and leg 1
|
||
> are kept as written because they are the model everything downstream was derived from, and because
|
||
> leg 2 (**the living app**) is unchanged and still binds Tier-2's file leg and Tier-3. Read §7.4 for
|
||
> the current chain; the corrected rows are 3, 4 and 9 in §8.
|
||
|
||
**[FACT — as of the R-108 survey, before D5]** Every app-tier restore depends on the guest, for one of
|
||
two reasons:
|
||
|
||
```
|
||
┌──────────────────────────────────────────┐
|
||
│ the LIVE GUEST │
|
||
│ · settings.json (tier-2 destination) │
|
||
│ · encryption.key (32 B) │
|
||
│ · app.yaml (ENC: under that key) │
|
||
│ · the deployed app itself │
|
||
└───────────┬──────────────────────────────┘
|
||
│ required by
|
||
┌──────────────────────────┼──────────────────────────┬─────────────────────┐
|
||
│ │ │ │
|
||
Tier-1 restore Tier-2 restore Tier-3 reconstitute Tier-3 scratch
|
||
(rebuild the app) (fill in files) (overwrite + replay) (verification copy)
|
||
│ │ │ │
|
||
needs SECRETS needs the app needs the app needs only the
|
||
from app.yaml running + the deployed + a DB repo password
|
||
(restore_unit.go recorded dest service identifiable (also in the guest)
|
||
:130-132) (tier2_restore.go
|
||
:114-116)
|
||
│ │ │ │
|
||
└──────────────────────────┴──────────────────────────┴─────────────────────┘
|
||
│
|
||
┌───────────▼──────────────────────────────┐
|
||
│ WHOLE-GUEST TIER (local vzdump / PBS) │ ← Lane 2, operator only
|
||
└───────────┬──────────────────────────────┘
|
||
│ required by
|
||
┌───────────▼──────────────────────────────┐
|
||
│ THE HOST (agent.json, bootstrap.json, │ ← in NO backup at all
|
||
│ vmbr9, sudoers, /etc/pve/priv, WG) │ (INV Part D1)
|
||
└──────────────────────────────────────────┘
|
||
```
|
||
|
||
**[FACT] The two legs of the dependency, precisely:**
|
||
|
||
1. **Secrets.** The recovery unit is secret-free by design — *"It NEVER writes a secret value"*
|
||
(`internal/backup/recovery_unit.go:73`). `RestoreFromRecoveryUnit` recovers secrets **from the
|
||
guest, never from the unit** (`restore_unit.go:130-132`), and the policy is stated outright at
|
||
`:18-22`: *"Regenerate NOTHING… A missing DATA-ENCRYPTING key is FATAL: regenerating it would
|
||
render the restored data unreadable, so we refuse and tell the operator to do a PBS whole-guest
|
||
restore."* Those secrets are encrypted under `encryption.key`, a 32-byte file that exists **only
|
||
inside the guest** (LIVE, both boxes — INV Part C, row 9).
|
||
2. **The living app.** Tier-2's restore reads the destination recorded in the guest's `settings.json`
|
||
(`tier2_restore.go:114-116`), and Tier-3's reconstitution refuses outright when the app is not
|
||
deployed — *„a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat"*
|
||
(`offbox_reconstitute.go:198-201`).
|
||
|
||
**[FACT] So all three app tiers are conditioned on the whole-guest tier**, which is Lane 2. The
|
||
customer's own recovery lane rests on an operator-only tier, and today nothing tells the customer
|
||
that. — **This sentence was the reason D5 exists. It is now false for Tier-1 and Tier-2; see §7.4.**
|
||
|
||
### 7.4 The chain AFTER D5 (controller v0.188.0, 2026-07-30) — the current model
|
||
|
||
**[FACT] Tier-1 and Tier-2 no longer depend on the whole-guest tier.** The unit's `compose/app.yaml`
|
||
(mode 0600) carries the **portable secret class** — every catalog `type: secret` field, i.e. the
|
||
declared `data_key`s, the 18 DB/root passwords and the internal signing secrets — so neither
|
||
`encryption.key` nor the guest's `app.yaml` is required to rebuild an app:
|
||
|
||
```
|
||
Tier-1 restore Tier-2 restore Tier-3 reconstitute Tier-3 scratch
|
||
(rebuild the app) (fill in files) (overwrite + replay) (verification copy)
|
||
│ │ │ │
|
||
needs ONLY THE DRIVE needs the app needs the app needs only the
|
||
(unit carries the running + the deployed + a DB repo password
|
||
secrets; guest is recorded dest service identifiable
|
||
consulted only for
|
||
the withheld class) └────────── still conditioned on the guest (leg 2, unchanged) ──────┘
|
||
│
|
||
✅ INDEPENDENT of the whole-guest tier
|
||
```
|
||
|
||
**[FACT] What still needs the guest, precisely** — so this is not read as more than it is:
|
||
|
||
1. **Leg 2 is untouched.** Tier-2's *file* restore still reads the destination from the guest's
|
||
`settings.json` (`tier2_restore.go:114-116`) and Tier-3's reconstitution still refuses when the app
|
||
is not deployed (`offbox_reconstitute.go:198-201`). D5 removed the **secrets** leg, not the
|
||
**living-app** leg. So "Tier-2 no longer depends on the guest" is true of the *unit-restore* path it
|
||
shares with Tier-1, and **false** of its additive file-merge path.
|
||
2. **The withheld class.** The 7 `type: password` admin logins and `vaultwarden/ADMIN_TOKEN` are
|
||
deliberately NOT on the drive; they come from the guest, or are regenerated (O4). Their absence
|
||
costs a credential reset, never data.
|
||
3. **The fail-closed gate remains.** A `data_key` in neither the unit nor the guest still refuses the
|
||
restore outright. D5 makes it normally present; it does not soften the gate.
|
||
|
||
**[DESIGN] Why plaintext is the right answer here, and what it is coupled to.** Every travelling secret
|
||
either decrypts data on the *same drive* or authenticates to a container on an internal compose network
|
||
with no external listener — so possessing it adds nothing to possessing the drive, which is exactly D2's
|
||
argument for keeping the DATA plaintext. That argument holds **only because** the internet-reachable
|
||
class is withheld. **The exclusion is what licenses the plaintext; the two must not be relaxed
|
||
independently.** `stacks.PortableSecretEnvVars` is the single boundary, with a code-level
|
||
`nonPortableSecrets` register (not a catalog flag — a boundary a catalog push can move is not a
|
||
boundary, cf. R-97a).
|
||
|
||
**[FACT] Precedence, because two sources now exist.** The **unit wins** over the guest. Not
|
||
"newest wins": the unit's secrets are captured in the same run as the dumps beside them, so the unit's
|
||
value matches *the data being restored*, whereas the guest's is merely the most recent. Preferring the
|
||
guest is the data-loss bug — a rotated data key does not decrypt data encrypted with the old one, and a
|
||
rotated DB password does not match the hash inside the restored data directory.
|
||
|
||
**[FACT] Proven live 2026-07-30**, on a scratch drill guest, through the real endpoints: AdventureLog
|
||
(`SECRET_KEY` data_key + `DB_PASSWORD`) restored with the guest's `app.yaml` **moved aside** —
|
||
`secrets recovered=2/2`, 27.6 s — and then **the application itself read the seeded customer row over
|
||
TCP with its own credential**, which is the observable that matters (a restore that returns success onto
|
||
unreadable data is a failure that looks like a pass). The discriminator held: the pre-backup row came
|
||
back, a row added *after* the backup was gone, so the volume tar was genuinely restored. There was **no
|
||
`.sql` dump** in that unit, so the DB came back from the volume tar — the exact case where a regenerated
|
||
password would have failed. Separately, Grafana's `type: password` admin login was proven **withheld**:
|
||
the sentinel value was live in the container and encrypted in the guest, and appeared in **0 files**
|
||
anywhere under the backup namespace. Full record: `audits/D5-drive-alone-restore-2026-07-30.md`.
|
||
|
||
### 7.2 A tier whose prerequisites cannot be met in the failure it exists for
|
||
|
||
**[FACT]** Two instances, both current:
|
||
|
||
- **~~Tier-2 vs primary-drive loss~~ — CLOSED 2026-08-31, controller v0.229.0 (R-102).** Tier-2's
|
||
stated purpose is surviving the loss of the primary drive. It was true until v0.229.0 that in that
|
||
failure the primary recovery unit is gone while the surviving mirror on the second drive —
|
||
`backups/secondary/<app>/recovery-unit/` — was read by **no code path** (§6.3), so for the **45**
|
||
class-B apps (§6.2, count settled the same day) the restore was a guaranteed no-op in exactly its
|
||
designed scenario.
|
||
**Tier-2 can now meet its prerequisite in the failure it exists for**, and that is stated plainly
|
||
because it is the whole point: the restore was run on `demo-hp` **with the primary unit moved aside**
|
||
and returned 3 volumes of 3 and 1 database of 1 in 28.65 s, byte-for-byte, with the app then reading
|
||
its own row over TCP with its own credential — and again with the guest's `app.yaml` also moved
|
||
aside, `secrets recovered=2/2` from the mirrored unit. Evidence:
|
||
`audits/DRILL-r102-tier2-unit-2026-08-31/`.
|
||
**What the drill did NOT cover, stated per §8 below:** the full drive-loss journey — a genuinely
|
||
absent or replaced physical drive — was not run. Only the recovery-unit half was.
|
||
- **Tier-3 vs guest loss — HALF of this closed.** Tier-3 holds the volume tars and the DB dump. It
|
||
was true until controller v0.218.0 that the tars were unpacked only by the Tier-1 path; since
|
||
v0.218.0 the reconstitution replays them itself (`volReplay`) → **R-107 CLOSED 2026-08-22**. What
|
||
is **still** true, and is the part that was never R-107: reconstitution requires the app to be
|
||
**deployed** and skips the unit, so offsite alone cannot rebuild an app onto a fresh guest. That
|
||
requirement is a deliberate decision (R-253) — the restore does not choose a customer's drive for
|
||
them — and since **v0.219.0** it means only what it says: an app that is not deployed. It no longer
|
||
catches the 40 driveless apps → **R-356**.
|
||
|
||
### 7.3 ~~What D5 would change — and why it was blocked~~ — **D5 SHIPPED 2026-07-30 (v0.188.0)**
|
||
|
||
> **This section is history.** D5 is implemented and proven live; the current chain is **§7.4**. Kept
|
||
> because the reasoning below is why the precondition was required, and because the last paragraph
|
||
> (`.fab` / R-126) is still open and still not part of D5.
|
||
>
|
||
> **One correction to the target as stated below.** It assumed the class that must travel is the
|
||
> `data_key`-flagged secrets. Part 0 of the D5 task tested that and it did not survive: the flag is
|
||
> unreliable (four encryption keys the catalog itself labels as such are unflagged → **R-127**), and a
|
||
> DB password is **not resettable in practice** — `POSTGRES_PASSWORD` is ignored once PGDATA is
|
||
> non-empty, so a regenerated value leaves the app unable to authenticate against its own restored rows
|
||
> while the dump replay still reports success. The shipped boundary is therefore *all* `type: secret`
|
||
> minus the register, not `data_key` alone.
|
||
|
||
**[DESIGN, TARGET — SUPERSEDED BY §7.4]** The intended fix is to make app secrets travel with the **local**
|
||
recovery unit, so Tier-1 and Tier-2 restore work **without the guest and without R**. Offsite
|
||
already encrypts everything, so secrets travelling offsite would be covered by the escrowed repo
|
||
password. R would then be required for **offsite recovery and host identity only** — losing R would
|
||
cost the offsite route, not local recovery.
|
||
|
||
**The premise D5 rests on is now established.** §2 of the task that produced this document required
|
||
it to be proven, not assumed: the backup tree must be unreachable from every browsing, download and
|
||
export surface. When this document was written it was **not** — the FileBrowser network-share bind
|
||
reached it. **R-108 closed that on 2026-07-30** (controller v0.187.0) by refusing app namespaces on
|
||
network storage, so no `backups/` tree can exist under the share-root bind; every other surface was
|
||
already clear (§10.1's table). **D5's precondition is therefore MET and D5 may be adopted.**
|
||
|
||
**Still true, and not part of D5's precondition:** a `.fab` bundle carries plaintext secrets by
|
||
design with an optional password, and `storageDriveList()` does not filter network paths, so a bundle
|
||
can be **exported onto** a NAS (§5, → **R-126**). That is an export destination the customer chooses
|
||
explicitly, not a browsing surface reaching a backup tree, and it is unchanged by D5 — D5 moves
|
||
secrets into the local recovery unit, not into `.fab`. It is tracked separately rather than folded in.
|
||
|
||
Until D5 is actually implemented, §7.1's chain stands as the model.
|
||
|
||
---
|
||
|
||
### 7.5 Where a recovery unit is RETAINED — and the size bound it puts on Lane 1
|
||
|
||
Added 2026-08-02 from `audits/SPIKE-recovery-unit-space-2026-08-02.md` and
|
||
`audits/CAMPAIGN-10-closeout-2026-08-02.md`. §6.1 says a Tier-1 unit lives *"on the app's own drive"*,
|
||
which is true and is the whole story for a drive-resident app. **It is not the whole story for an app
|
||
with no data drive, and that case was undocumented until now.**
|
||
|
||
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go:245-255`)
|
||
returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what
|
||
`internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit
|
||
after it is copied onward: the only prune is F5 (`backup.go:1053-1112`), which removes residue on an
|
||
**old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy
|
||
indefinitely.
|
||
|
||
**What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`):
|
||
`compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains
|
||
`mp8` / `HDD_PATH` userdata** — a photo library on a 1 TB drive is not in a unit, and cannot make one
|
||
overflow.
|
||
|
||
**The resulting mismatch, on a default appliance:**
|
||
|
||
| | size | holds |
|
||
|---|---|---|
|
||
| `mp0` `/var/lib/docker` | **50 G** | every app's live volumes — the thing a unit copies |
|
||
| `mp1` `/mnt/sys_drive` | **20 G** | the retained units of every **driveless** app |
|
||
|
||
A DB-backed app's unit is up to **~2×** its data, because it carries the volume tar **and** the SQL
|
||
dump (measured: 21.1 GB of app data → a **40.2 GB** unit). A file-only app's is **1.00×** (measured:
|
||
homebox 2305 MB → 2305 MB). `--sysdata-grow` (`felhom-agent/cmd/felhom-agent/main.go:178`) defaults to
|
||
**0**, is not derived from the physical drive, and no observed box passes it — demo-hp's production
|
||
guest 9201 ships `mp0 50G / mp1 20G`.
|
||
|
||
**`mp1` gates the entire app-data chain, not just Tier 1.** Tier-2 mirrors the unit *"(always)"* from
|
||
`RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too, so a unit that
|
||
cannot be written to `mp1` leaves Tier-2 and Tier-3 with nothing to copy.
|
||
|
||
**The bound this places on Lane 1 (§3).** Lane 1's promise is that a customer can restore an app from
|
||
the drive alone, without the operator and without the guest — D5 (§7.4) made that true by putting the
|
||
portable secrets in the unit. **That independence is bounded by app size**, and the bound is:
|
||
|
||
> On a default appliance, an app can be restored by Lane 1 only while its recovery unit still fits the
|
||
> retained space — **≈ 19 GB of app data for a file-only app, ≈ 10 GB for a DB-backed one.**
|
||
> Past that the unit cannot be captured, and the app falls back to Lane 2's operator-driven
|
||
> whole-guest route.
|
||
|
||
**AS OF 2026-08-02 SOMETHING NOW WARNS, AND THE ALERTING IS PART OF THIS CONTRACT (R-167 / R-158,
|
||
decision D-c; controller v0.191.x + hub v0.89.0).** The last sentence of this section used to end
|
||
"nothing warns when an app crosses the line". Two signals now exist and both are PROVEN-LIVE:
|
||
|
||
- **To the CUSTOMER, before anything fails** — `internal/fillwatch` warns per FILESYSTEM (never per
|
||
app: one full disk holding ten apps would fire ten times) on **whichever trips first, used ≥ 85% or
|
||
free < 5 GiB**, critical at 95% / 2 GiB, clearing at 75% / 7 GiB. **Two terms, because a percentage
|
||
alone lies at both ends of the range this section itself documents:** 85% of a 20 G `mp1` leaves
|
||
3 G — less than one DB-backed app's unit — while 85% of a 4 TB drive leaves 600 G. It watches the
|
||
app-data volume, the system-data volume **and** every registered drive, which the previous
|
||
`health_degraded` signal did not. Edge-triggered against persisted state; the hub owns cooldown.
|
||
- **To the OPERATOR, when a capture actually fails** — `recovery_unit_capture_failed`, per app, with
|
||
the target filesystem's used/free bytes at the moment of failure, so the *why* needs no login. It is
|
||
**operator-tier** (`notify.operatorOnlyEvents`) and deliberately not `backup_failed`: a customer can
|
||
take no action on a capture failure.
|
||
|
||
### 7.5.1 — THE CEILING THIS SECTION DESCRIBES HAS BEEN REMOVED (2026-08-03, R-165 / decision D-a)
|
||
|
||
**Everything above describes the SPLIT layout, which is now the legacy shape.** A golden built by
|
||
`build-golden.sh` **v3.0.0** ships **one** data volume; `mp1` does not exist. Both consumer paths are
|
||
binds of subdirectories of it (variant **V-c**):
|
||
|
||
```
|
||
mp0 -> /var/lib/felhom ├─ docker/ --bind--> /var/lib/docker
|
||
└─ sys_drive/ --bind--> /mnt/sys_drive
|
||
```
|
||
|
||
**So the size bound below no longer applies to a box built from that golden.** A driveless app's
|
||
recovery unit is limited by the box's actual free space, not by a partition set at build time. The
|
||
mismatch table above (`mp0` 50 G vs `mp1` 20 G) describes what a merged box no longer has.
|
||
|
||
**R-175, fixed here rather than left standing.** The bound below was stated as the fleet's and was
|
||
**one box's**: it is derived from `mp1 = 20 G`, which is demo-hp exactly and never was demo-felhom
|
||
(`mp0 200G / mp1 50G`, where the same arithmetic gives ≈ 49 GB / ≈ 24 GB), nor the golden (`16 G / 8 G`
|
||
before provision grew them). **Read it as a function of `mp1`, and only for a box still on the split
|
||
layout.** Measured: `audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1.
|
||
|
||
**What replaced the partition's second job — the reserve.** `mp1` was also a BULKHEAD: an overflow was
|
||
refused per app with the last good unit byte-identical, and it **could not reach `/var/lib/docker`**,
|
||
because that was a different filesystem. On a merged box it can. Decision **B2**, shipped in controller
|
||
**v0.192.0**, is that bulkhead made deliberate — a two-term reserve (97% used or 1 GiB free) in
|
||
`internal/fillwatch`'s shape, sitting beyond its critical band so the customer is always warned first.
|
||
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
|
||
only destroy a different app's only local copy.
|
||
|
||
**WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
|
||
The two are deliberately different mechanisms, because conflating them is how seven failures went
|
||
missing on 2026-08-03 without leaving a trace.
|
||
|
||
| | Record | Notification |
|
||
|---|---|---|
|
||
| what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
|
||
| when | every time, unconditionally | at the end of a run, **only if something failed** |
|
||
| gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
|
||
| where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
|
||
|
||
- **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
|
||
is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
|
||
REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
|
||
design; weakening it re-opens a silent-failure path.
|
||
- **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
|
||
Deciding not to tell someone is itself an event worth recording.
|
||
- **Deliberate skips are not failures** and never appear in the digest — a disconnected or
|
||
decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
|
||
operator stops reading.
|
||
- **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
|
||
the hour, because someone pressing the button is actively trying to get a backup. The periodic
|
||
capture sweep is capped by the ordinary hourly cooldown.
|
||
|
||
**THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
|
||
**per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
|
||
before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
|
||
volume dump and the recovery-unit capture**. Those three write under one per-app root
|
||
(`backups/primary/<app>`), which is what makes one verdict able to cover them honestly.
|
||
|
||
- **What it guarantees.** A refused app has **nothing written for it in that run**, its previous unit
|
||
is **byte-identical**, it is **not stopped**, nothing anywhere is deleted, and the operator gets
|
||
**exactly one** alert naming the app, the term that bound and the disk figures.
|
||
- **Two terms, two questions.** *Headroom*: is the filesystem already below the reserve? *Size*: would
|
||
THIS app's write take it below? The size estimate is the app's previous `.sql` + `.tar` on disk;
|
||
with no history the decision degrades to headroom alone, deliberately — otherwise the first backup
|
||
is the one that can never happen.
|
||
- **Why it is decided lazily and not once per run.** Space changes during a run: app A's dump can put
|
||
app B under the reserve, so a verdict taken at run start reads a disk that no longer exists.
|
||
- **Why it is never re-decided between an app's own legs.** That is precisely the shape v0.192.0 had —
|
||
the two dump legs unguarded and only the capture refused — under which the reserve was consumed by
|
||
the very write it exists to bound, and the refusal's *"the previous unit is untouched"* was measured
|
||
false. Proven live on demo-hp 2026-08-03 (R-181), fixed the same day, and re-proven by filling the
|
||
box for each of the two terms.
|
||
- **It sits ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal
|
||
decided inside it would already have bounced the app it is refusing to back up.
|
||
|
||
**Status caveat, deliberately explicit:** every box in the field that has not been reinstalled is still
|
||
on the split layout and everything above still describes them exactly. This subsection describes what a
|
||
box built from golden ≥ 0.192.0 gets. Both demo boxes were reinstalled from it on 2026-08-03 (R-178).
|
||
|
||
Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling**
|
||
(**R-163**) — this section states the constraint, not a number. And **the same-device placement is
|
||
intended, not a defect**: a driveless app's unit sits on the same SSD as its volumes, but Tier-2's
|
||
cross-drive mirror, the whole-guest tier (which covers `rootfs + mp0 + mp1`, §6.1) and Tier-3 all cover
|
||
device loss. What was missing was that this case existed at all, and that nothing warns when an app
|
||
crosses the line — **R-158**.
|
||
|
||
## 8. The failure → recovery matrix
|
||
|
||
> ### THE ARC IS CLOSED FOR BETA — read this before the blanks below (2026-09-01)
|
||
>
|
||
> **Closed at controller v0.232.0 / hub v0.111.1.** Everything a **customer** does for themselves is
|
||
> finished and proven live — rows **1, 2, 3, 3b, 3c, 6, 7, 14**. Everything only an **operator** does
|
||
> is **deliberately deferred until after beta** — rows **4, 8, 9, 10, 11 (+11b), 12**, each carrying
|
||
> the marker **`[BETA-DEFERRED]`** in its status cell so the set greps as a group.
|
||
>
|
||
> **The blanks in the RTO column below are now blank ON PURPOSE, and that is the whole difference.**
|
||
> Eleven rows carry a blank RTO — 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15 — and only six of those are
|
||
> deferred work. **Row 5** has no recovery to time (a derived copy), **row 13** has no route by
|
||
> design, **row 14** is proven and merely never stopwatched, **row 11b** is a note not a row, and
|
||
> **row 15 is an open DEFECT (R-104) that this line does NOT cover.**
|
||
>
|
||
> **NO STATUS MOVED ON THE DAY THIS WAS WRITTEN, because nothing was proven that day.** A stopping
|
||
> line that promotes a row is a stopping line that lies. Full text, and the conditions that reopen
|
||
> this: `documentation/backlog/OPEN-ITEMS.md`, *"DECIDED — the backup and restore arc is CLOSED FOR
|
||
> BETA"*.
|
||
|
||
**This is the core artifact.** One row per failure. It is authoritative for recovery routes;
|
||
`00-capability-map.md` stays authoritative for per-capability status.
|
||
|
||
**How to read the numbers.**
|
||
|
||
- **RTO** — **only measured durations** from INV Part F appear here. A blank cell means *nobody has
|
||
ever measured it*, and a blank is a finding, not an omission.
|
||
- **RPO** — **no RPO has ever been measured from an incident.** These cells carry the **configured
|
||
cadence that bounds RPO**, read live from the box, labelled `(cadence)`. A blank means no cadence
|
||
governs the row.
|
||
- **Status** — `PROVEN` (live, cited) · `PARTIAL` (some legs proven) · `IMPLEMENTED` (code + tests,
|
||
never exercised) · `NONE` (no route exists).
|
||
|
||
| # | Failure | What survives | Recovery route | Invocable by | RTO (measured) | RPO (cadence) | Status | Evidence |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| 1 | **Customer deletes files** | everything else | „Fájlok visszaállítása" — Tier-2 additive merge | **customer** | **39 s** (6 files) | 24 h | **PROVEN** | CAMPAIGN-9 A1 — byte-identical returns, both non-destruction promises kept, bytes re-served by paperless's own API |
|
||
| 2 | **An app's data directory is destroyed** | the guest, the other tiers | same route | **customer** | **46 s** (43 files) | 24 h | **PROVEN** | CAMPAIGN-9 A3 — loss verified by a positive observable (doc download 200 → 404), then 16/16 documents usable |
|
||
| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar. **2026-08-30 — this row KEEPS its PROVEN status and the reason is worth stating: R-353 was a defect in the MESSAGE, not in the mechanism.** The restore really did return what the unit held, every time; what it could not do was say so, because the count was discarded one call deep. Controller v0.226.0 fixed the sentence and changed nothing about the recovery path. A status that measures whether data comes back must not move because a status line was wrong |
|
||
| 3c | *same, with the GUEST GONE (secrets unavailable)* | the drive | Tier-1 unit restore — **the unit carries the portable secrets** | **customer** | **27.6 s** | 24 h | **PROVEN** | D5, §7.4. Before v0.188.0 this row was **NONE**: the data-key gate refused and the customer's own copy of their own data was not a recovery |
|
||
| 3b | *same, for a class-B app via Tier-2* | the Tier-2 copy on the second drive | „Teljes visszaállítás a másolatból" — **Tier-2 unit restore** (`POST /backup/tier2/unit-restore` → `RestoreTier2Unit` → `RestoreFromRecoveryUnitAt`), controller **v0.229.0** | **customer** | **28.65 s** (3 volumes, 1 database, 114.5 MB unit) | 24 h | **PROVEN** (2026-08-31) | `audits/DRILL-r102-tier2-unit-2026-08-31/`. docmost — a class-B app whose Tier-2 run reports **0 leg(s)** — restored **with the primary unit moved aside**: 3 volumes of 3 and 1 database of 1, from `/mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit`. **The observable is the DATA:** an accented Hungarian filename returned byte-for-byte (verified as hex, R-364) and the app read its own row **over TCP with its own credential**; the post-backup discriminator was **gone**, so the tar was genuinely replayed. Repeated with the guest's `app.yaml` also aside → `secrets recovered=2/2` from the mirrored unit. **R-102 CLOSED** |
|
||
| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 of 53 apps) **and, since controller v0.229.0, for the unit mirror — its volume tars and DB dump, i.e. the whole of what the other 45 own**; Tier-3 reconstitute for files + DB **+ the named-volume tars since v0.218.0** (`volReplay`) | **customer** (all) | | 24 h | **PARTIAL** **`[BETA-DEFERRED]`** | §7.2. **Both unreachability gaps are now closed — R-107 (v0.218.0) and R-102 (v0.229.0).** This row stays **PARTIAL** deliberately: what is proven is the ROUTE (row 3b, live, primary unit absent), not the JOURNEY. **No drive has ever actually died or been replaced under this recovery** — the drill removed a unit directory, not a disk, so drive re-attachment by `durable_id`, the agent's enrolment of a replacement, and a Tier-2 copy read from a drive that is the ONLY surviving one are all still unexercised. Promoting this row to PROVEN needs that journey, not another unit restore |
|
||
| 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go`. **The derived-copy rule is UNCHANGED by R-403 (controller v0.230.0) and the single exception is stated in §8.2 below — read it before "fixing" a skip you find in the code** |
|
||
| 6 | **Guest lost or corrupted** | the host, both whole-guest tiers, the data drives (they are host binds) | `pct restore` from `local:` or `felhom-pbs:` | **operator** (SSH) | **84–112 s** local · **1101 s** PBS (both = restore-test into a scratch guest, boot + verify + teardown) | 24 h local · 7 d offsite | **PROVEN** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C (exact mount parity, `unprivileged: 1` preserved); LIVE restore-tests on both boxes this session |
|
||
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
|
||
| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
|
||
| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) |
|
||
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recovered, so nothing was timed.** `audits/evidence-drill-r95-recovery-2026-09-01/`. |
|
||
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
|
||
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
|
||
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D |
|
||
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) |
|
||
| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
|
||
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
|
||
|
||
### 8.2 The ONE exception to row 5's derived-copy rule (R-403, controller v0.230.0)
|
||
|
||
**[DESIGN] Row 5 stands: the secondary IS a derived copy and it IS rebuilt on the next run.** Nothing
|
||
below weakens that, and a future reader who finds `RunTier2` skipping a leg and does not find this
|
||
section will "fix" it back — which is why it is here and not only in a register row.
|
||
|
||
**[FACT] What was measured, on the shipped v0.229.0, on `demo-hp` 2026-08-31.** An app's Tier-2 copy
|
||
went from **120 082 104 B** (4 database dumps + 3 named-volume tars) to **7 036 B** (none of either)
|
||
in one nightly run, and the run recorded itself a success. The mechanism was three individually
|
||
correct lines: `RunTier2` guarded the unit leg with `os.Stat` alone — *does the folder exist* —
|
||
`rsyncMirror` is `rsync -a --delete`, and nothing between them compared source to destination. **An
|
||
empty recovery unit is a folder that exists.** Evidence:
|
||
`audits/DRILL-r403-tier2-delete-2026-08-31/`.
|
||
|
||
**Why it bites harder since 2026-08-30:** R-102 made that mirror a LIVE recovery route (§6.3, §8 row
|
||
3b). Deleting it used to cost a copy nobody could open; it now costs the route itself.
|
||
|
||
**[DESIGN] The exception, stated exactly.** The unit leg — and ONLY the unit leg — is skipped when the
|
||
SOURCE unit carries no data and the DESTINATION unit does. Everything else is unchanged:
|
||
|
||
| source unit | destination unit | behaviour |
|
||
|---|---|---|
|
||
| complete | complete | mirror, with `--delete`, as before |
|
||
| complete | hollow or absent | mirror (the normal first copy) |
|
||
| hollow | hollow | mirror — both sides agree, nothing is at risk |
|
||
| **hollow** | **complete** | **skip the unit leg, preserve the destination, warn, record for the surface** |
|
||
|
||
**"Hollow" is a MANIFEST question, never a size question** — the manifest lists no database dump and
|
||
no volume tar; absent or unparseable counts as hollow, fail closed. A unit with a fat compose capture
|
||
and no dumps is the dangerous shape; a 360-byte unit belonging to a tiny app is healthy.
|
||
|
||
**The data legs are NOT guarded and must not be.** A classified app's copy legitimately shrinks as
|
||
`export` drops out of its class set (`tier2.go` header), and fencing that would be calling this row's
|
||
own decision a defect.
|
||
|
||
**[DESIGN] And the CAUSE is closed at the other end.** The hollow primary was written by the 5-minute
|
||
capture job **two seconds** after a Tier-2 unit restore. Since v0.230.0 `RestoreTier2Unit` refills an
|
||
absent or hollow primary unit from the mirror **inside the call, before returning**, so no capture can
|
||
observe the hollow state. **The capture itself is deliberately not guarded:** a capture that describes
|
||
an empty drive as empty is correct, and guarding it would make the manifest lie.
|
||
|
||
### 8.1 The blank cells, listed explicitly
|
||
|
||
Per the rule that a blank is a finding, here they are:
|
||
|
||
| row | blank | why |
|
||
|---|---|---|
|
||
| 4 | RTO | no drive-loss recovery has ever been timed. **The Tier-2 unit restore inside it now is** — 28.65 s, row 3b, 2026-08-31 — but that is the route, not the journey: no drive has been removed or replaced under a recovery |
|
||
| 5 | RTO | never timed; the rebuild is a normal Tier-2 run |
|
||
| 6 | — | RTO present, but it is a **restore into a scratch guest on the same host**; a restore *to a different host* has never been timed |
|
||
| 8 | RTO | a host has never been rebuilt as itself (INV Part D1) |
|
||
| 9 | RTO | the composed whole-box path has never been run |
|
||
| 10 | RTO | no ransomware-shaped recovery has ever been run |
|
||
| 11 | RTO | a hub restore has never been performed |
|
||
| 12 | RTO, RPO | no provider-loss recovery has ever been run |
|
||
| 13 | RTO, RPO | not a timed recovery; a capability loss |
|
||
| 14 | RTO | the break-glass path is proven but was never timed |
|
||
| 15 | RTO | the manual unlock was performed but not timed |
|
||
|
||
**Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB
|
||
from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS
|
||
incremental; `.fab` import wall-clock; and time-to-first-byte for a customer restore over a home
|
||
uplink — **no customer has ever driven a restore**.
|
||
|
||
---
|
||
|
||
## 9. What the model implies for the tiers (recorded, not new design)
|
||
|
||
**[DESIGN]** Three consequences follow from §3–§7 and are stated so they are not re-derived:
|
||
|
||
1. **Tier-2 is a *drive-loss* tier, not a second chance at Tier-1.** Its job is to survive one drive
|
||
dying. That is why it mirrors the unit at all — and why the unit mirror being unreadable (§6.3)
|
||
defeats the tier rather than degrading it.
|
||
2. **Tier-3 is a *premises-loss* tier.** It is the only copy that survives fire, theft and
|
||
ransomware-with-host-access, which is why it is the only tier whose credential exposure is ranked
|
||
as the largest open data risk (R-95).
|
||
3. **The whole-guest tiers are the *system* tier, and they sit under everything else** (§7.1).
|
||
Treating them as "one more copy" is precisely the 3-2-1 framing this document rejects.
|
||
|
||
---
|
||
|
||
## 10. Known gaps
|
||
|
||
Every divergence between the model above and the system as it is, each with an ID.
|
||
|
||
### 10.1 ~~D5 is BLOCKED~~ — precondition CLOSED by R-108 (v0.187.0); **D5 ITSELF SHIPPED (v0.188.0)**
|
||
|
||
> **D5 IS DONE — 2026-07-30, controller v0.188.0.** The precondition below was met by R-108, and D5 was
|
||
> then implemented and proven live the same day: Tier-1/Tier-2 no longer depend on the whole-guest tier,
|
||
> and a customer needs **the drive and nothing else**. The current recovery chain is **§7.4**; the
|
||
> corrected matrix rows are 3, 3c and 13. This section's remaining value is the *precondition* analysis,
|
||
> which is why the table below is kept.
|
||
|
||
> **D5's PRECONDITION IS MET.** An app's data namespace can no longer be placed on network storage, so
|
||
> no `backups/` tree can exist inside FileBrowser's share-root bind, and the browsing surface therefore
|
||
> cannot reach the backup tree on ANY storage class. Every other read surface in the table below was
|
||
> already NO. **D5 may be adopted** — nothing in this section blocks it.
|
||
>
|
||
> **The fix inverted the obvious one, and that is the durable lesson here.** The share-root bind was
|
||
> not narrowed, because it **cannot** be: (a) the `:rslave` share-ROOT bind is load-bearing — a
|
||
> Phase-0 probe (2026-07-22) proved an in-container access through it wakes the idle automount
|
||
> trigger, so narrowing it breaks NAS access itself; (b) there is no `userdata/` layer to scope to,
|
||
> since apps on a share store at `<share>/<app>`; and (c) creating one would write Felhom's directory
|
||
> convention onto a customer's own NAS, which R-67 forbids outright. The browsing surface being
|
||
> immovable is precisely *why* the backup tree must never be placed under it. Tier 2 had already
|
||
> reached the same conclusion for its own targets (`F-6C-1`); R-108 closes the PRIMARY namespace,
|
||
> which was the last remaining route.
|
||
>
|
||
> **Operator ruling 2026-07-30: refuse the placement, keep the browse bind.** `RefuseAsAppNamespace`
|
||
> (`internal/settings/settings.go`) is the single predicate; all placement surfaces consult it. It
|
||
> **fails closed** — `/mnt/felhom-drives` holds both storage kinds in-guest, so a path prefix cannot
|
||
> classify and `Kind` exists only on a REGISTERED path; an unregistered path under that root is
|
||
> therefore un-classifiable and is refused rather than assumed to be a drive.
|
||
>
|
||
> **Nothing was stranded:** zero apps on network storage across all six hub customers including Peti.
|
||
> R-67's browse capability is byte-for-byte unchanged (verified by diffing demo-hp's generated
|
||
> compose before and after the deploy). This also **supersedes** the controller README's "NAS backup
|
||
> locality — decision A" (v0.118.0), which deliberately kept a NAS-resident app's Tier-1 artifacts on
|
||
> the NAS: that case can no longer arise.
|
||
>
|
||
> Evidence: `audits/R108-network-app-namespace-2026-07-30.md`. The pre-fix analysis below is retained
|
||
> verbatim as the record of what was wrong.
|
||
|
||
#### 10.1 (historical) The exposure as it stood before v0.187.0
|
||
|
||
**[FACT] The verification and its result.** Every surface that can read a file was checked:
|
||
|
||
| surface | can it reach `backups/`? | evidence |
|
||
|---|---|---|
|
||
| SMB share creation | **NO** | `sharingResolvePath` (`internal/web/sharing_handlers.go:52-81`) resolves symlinks *before* containment, then refuses any path within `SharingDeniedRoots(root)`; that set covers `<root>/backups` **and** the legacy `<root>/felhom-data` + `<root>/felhom-data/backups` (`internal/stacks/samba.go:46-65`, derived from `ProtectedHDDPaths`, `delete.go:59-78`) — **both namespace shapes** |
|
||
| SMB browse (folder picker) | **NO** | same deny set applied per child (`sharing_handlers.go:521-537`) |
|
||
| SMB `ensureImportShare` (the store-direct bypass) | **NO** | writes one controller-generated constant, `GetImportRoot()` = `<system ns>/userdata/import` (`sharing_handlers.go:564-587`) |
|
||
| FileBrowser — **local drives** | **NO** | the bind is `appbackup.UserdataDir(sp.Path)` only, and the comment says why (`internal/web/handlers.go:2450-2460`) |
|
||
| **FileBrowser — network shares** | ~~**YES**~~ → **NO** (R-108, v0.187.0) | the bind is still the share **ROOT** (`- %s:/srv/%s:rslave`, `handlers.go:2437`) and deliberately so — but **no app namespace, hence no `backups/` tree, can exist on a share**, so the root bind reaches only the customer's own files. The reachability is closed at the PLACEMENT, not at the bind |
|
||
| `.fab` import path validation | **NO** | confined to `<root>/exports` (`handler_export.go:400-408`, `estimate.go:215-217`) |
|
||
| `.fab` browser download | **NO** | name-pattern + parent-must-be-the-staging-dir double guard (`handler_export_download.go:36-45,120-140`) |
|
||
| `/api/debug/*` | **NO** | no file-serving branch (`handler_debug.go:46-92`) |
|
||
| `http.ServeFile` (3 sites) | **NO** | assets only, `filepath.Base`-normalised (`server.go:720-771`) |
|
||
| log bundles | **NO producer found** | no filesystem-walk bundle producer exists on the box; searched `felhom-controller/internal`, `felhom-agent/internal`, `cmd/` |
|
||
| registering the backup dir as a drive | **NO** | the manual add requires `system.IsMountPoint(path)` (`handlers.go:2091-2095`); `<drive>/backups` is not a mount point |
|
||
|
||
**The exposure, end to end.** All six links are source-cited and the precondition is live today:
|
||
|
||
1. A NAS share is registerable as a storage path and lands `Schedulable: true` — **LIVE on demo-hp**:
|
||
`{"path":"/mnt/felhom-drives/Felhom-Share","kind":"network","schedulable":true}`.
|
||
2. `GetSchedulableStoragePaths()` has **no `IsNetwork()` filter** (`internal/settings/settings.go:904-914`),
|
||
so that share appears in the **deploy** dropdown (`handlers.go:462-473`).
|
||
3. The **per-app migrate** target list filters only current / decommissioned / disconnected /
|
||
schedulable — also no network filter (`handlers.go:674-679`).
|
||
4. `handleStorageMigrateApp` does **not** call `refuseNetworkLifecycle`, unlike its whole-namespace
|
||
sibling which does (`storage_handlers.go:397` vs `:410-424`), and `startMigration` has no guard
|
||
either (`internal/stacks/migrate.go:214-255`).
|
||
5. With `HDD_PATH` on the share, `namespaceRoot()` returns it as-is under Model A
|
||
(`internal/backup/backup.go:262-263`), so the app's Tier-1 unit is written to
|
||
`<share>/backups/primary/<app>/compose/app.yaml`.
|
||
6. FileBrowser binds that share at its **root** and serves it with `download: true`
|
||
(`internal/infra/infra.go:326`).
|
||
|
||
**Observed live on demo-hp, in the generated compose — the asymmetry is visible, not inferred:**
|
||
|
||
```
|
||
- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb <- drive: userdata-scoped
|
||
- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave <- network: ROOT-bound
|
||
- /mnt/sys_drive/felhom-data/userdata/import:/srv/beolvasas
|
||
```
|
||
|
||
**Today this is not a secret leak**, because the unit's `app.yaml` is secret-stripped
|
||
(`recovery_unit.go:73`). **D5 would make it one.** That is exactly the test §2 set, and D5 therefore
|
||
does **not** hold as written. → **R-108**
|
||
|
||
> **CLOSED (v0.187.0).** Links 2, 3 and 4 of the chain above are now guarded, and a **fifth** surface
|
||
> the chain did not list was found and guarded too: `handleStorageDecommission` mode=`migrate` checked
|
||
> only `req.Where` (the SOURCE) via `refuseNetworkLifecycle`, so a whole namespace could be
|
||
> decommissioned ONTO a NAS. Link 2's framing also understated the problem — the deploy **dropdown** is
|
||
> only a UI list; the boundary is the deploy **POST** (`internal/api/router.go`), which accepts any
|
||
> caller-supplied `HDD_PATH` and whose only other validation is `os.Stat` existence
|
||
> (`internal/stacks/deploy.go`). Filtering the list alone would have left the surface open. Links 5 and
|
||
> 6 are unchanged and still true — they simply can no longer be reached.
|
||
|
||
### 10.2 The gap register
|
||
|
||
| ID | Gap | Consequence |
|
||
|---|---|---|
|
||
| **R-102** | Tier-2 writes a full `recovery-unit/` mirror on every run and **no code path reads it** | Tier-2 is defeated in the drive-loss scenario it exists for (§7.2). Was C9-F4 |
|
||
| **R-103** | The Tier-2 no-coverage refusal **names** the working action but does not **route** to it | 45-or-43 of 53 apps dead-end at a message. Was C9-F1b |
|
||
| **R-104** | An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach, reported as *„ismeretlen okból"* | the offsite tier stays dead until a human unlocks. Was C9-F3 |
|
||
| **R-105** | Three hub-held DR records are empty on the whole live fleet: `hosts.dr_record_json`, `host_escrow.directive_json`, `dr_recipe.host_half.drives` | the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field |
|
||
| **R-106** | `dr_recipe.host_half.pbs.namespace` records `"root"` on every box | the recorded restore coordinate is wrong; real namespaces are per-customer |
|
||
| **R-107** | ~~No offsite action unpacks the named-volume tars Tier-3 captures on every run~~ — **CLOSED, controller v0.218.0, 2026-08-22** (`volReplay`, proven live on `demo-hp`). True from the day Tier-3 shipped until 2026-08-21. | was: offsite alone cannot rebuild a named-volume app (§7.2). Now: the tars replay; what remains is that the app must be **deployed** (R-253), which is not R-107 |
|
||
| **R-356** | The off-site restore resolved its destination with the raw `HDD_PATH` and read an empty answer as "not installed" — **CLOSED, controller v0.219.0, 2026-08-22** | 40 of 53 apps were refused permanently while running (§6.3 `[DESIGN]`) |
|
||
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
|
||
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
|
||
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo, from **two** call sites (`offbox.go:1388` retention and `offbox.go:1759` over-quota) | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10). **SPIKE 2026-09-01:** prevention needs a transport change (restic 0.14.0 does speak `rest:` — measured; append-only is a rest-server flag, not a restic one), because the sub-account API cannot express write-without-delete. **Detection is nearly free and is recommended first** — `snapshot_count` already reaches the hub and the hub appends reports, so the comparison needs no box change. `audits/SPIKE-r95-offsite-delete-2026-09-01.md`. **RE-SCOPED 2026-09-01:** the box can delete the LIVE repository but **cannot write to the daily snapshots of it** (measured, both boxes) — so the exposure is at most one day's data plus an operator-driven per-file recovery, not open-ended loss. Detection shipped hub v0.111.0 (R-431). **DRILL 2026-09-01: the re-scope's recovery clause is WITHDRAWN — no snapshot is reachable from a sub-account by any name (R-433, 777,600 names, controlled), so the exposure is NOT bounded by an operator-driven per-file recovery. R-436 is the new cheap lead: the provider already offers `rclone serve restic --stdio` server-side and restic 0.14.0 speaks `rclone:` (measured) — but the client supplies the server command line, so ask the vendor whether `--append-only` is pinned BEFORE building anything.** |
|
||
| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
|
||
| ~~R-353~~ | ~~A local unit restore reported a bare completion whether it returned an entire dataset or nothing~~ | **CLOSED 2026-08-30, controller v0.226.0.** The volume count already existed (`restoreDockerVolumesFrom`) and was discarded by a one-line wrapper, so the surface was structurally unable to say what came back. `RestoreFromRecoveryUnit` now returns `UnitRestoreResult` and the sentence has three cases, keyed on replayed-vs-**listed** — zero-replayed has two causes that are opposite news. Every sentence is a claim about the BACKUP, never the app: this path has no `SafetyDump` discriminator, and §6.3 is why that is not pedantry. Proven live on `demo-hp` |
|
||
| ~~R-357~~ | ~~The destructive reconstitute had no free-space gate; all three that existed guarded non-destructive paths~~ | **CLOSED 2026-08-30, controller v0.226.0.** The gate sits before `writeSafetyDump` and `StopStack`, so a refusal costs no outage — the test asserts the StopStack call count, not the sentence. No headroom multiplier (a measured tree, not a predicted download); fail-closed when either probe reads ≤ 0, which was a real fail-open hole. **Not live-validated** — filling a filesystem is a drill step |
|
||
| ~~R-358~~ | ~~`OffboxFullScratchReady` asked "non-empty directory", which is exactly what a failed restic run leaves~~ | **CLOSED 2026-08-30, controller v0.226.0.** A completion marker written only after restic returns nil, stale ones cleared before it starts, both orders pinned by an AST test. Both handlers refuse server-side — a hidden button is not a guard. Proven live on `demo-hp`. **See R-396: the bad state needed only a SUCCESSFUL safe restore, not a failed one** |
|
||
| ~~R-360~~ | ~~The verification-copy delete gated on the concurrency flag, which a verification restore never holds~~ | **CLOSED 2026-08-30, controller v0.226.0.** `restoreOpBlocked()`, matching the five sibling handlers. The doc comment that claimed it already did this is corrected in place — that sentence is why nobody looked. Proven live on `demo-hp` in the exact flag state that produced it |
|
||
| ~~R-396~~ | ~~A unit-only verification restore unlocked the destructive full restore~~ | **CLOSED 2026-08-30, controller v0.226.0** (by R-358's marker). Found while answering R-358's open question. Both restore modes write the SAME scratch directory, and one boolean (`ScratchReady`) drove three different intents — "is there a scratch", "may we place", "may we destructively restore". The safest action on the page unlocked the most dangerous one |
|
||
| ~~R-359~~ | ~~The off-site restic store is never verified by anything, ever~~ | **CLOSED 2026-08-30, controller v0.227.0/v0.227.1.** A daily `offsite-integrity` job on **due-ness, not a weekday**; it takes the single-writer flag and SKIPS rather than waits (`resticStep` escalates to `unlock --remove-all` and is only safe while that flag is held). Three outcomes — skipped / unreachable / failed — because 'I could not look' is not 'I looked and it is broken'. **⚠ The depth that ships ON does NOT catch silent corruption:** measured, a pack corrupted without a size change returned `no errors were found`, exit 0; only `--read-data*` caught it. Choosing the depth is **R-399** |
|
||
| ~~R-397~~ | ~~`NotifyIntegrityOK`/`NotifyIntegrityFailed` had no caller and the product advertised a weekly check that did not exist~~ | **CLOSED 2026-08-30, controller v0.227.0.** Sixth built-but-never-wired instance: hub allowlist, Hungarian text, settings checkbox and debug button all existed; only the caller did not. `ok` is severity `info` and mails nobody by design |
|
||
| ~~R-399~~ | ~~The check reads the catalogue and never the data~~ | **CLOSED 2026-08-31, controller v0.228.0.** `monitoring.integrity.read_data_subset` now defaults to **`100%`**, so the weekly check downloads and re-hashes every stored byte. **The fact that made it necessary, and the sentence that should stop anyone turning it back down to save four seconds: the structure check PASSED a size-preserving pack corruption.** Measured on `demo-hp` 2026-08-30 — plain `restic check` reported `no errors were found` and exited 0 over a pack damaged without a size change; every read-data form caught it. Cost on that 134 MB store: 35.0 s structure vs 39.2 s at 100%. `off` (any case) returns a box to structure depth; an empty value means *not configured*, therefore the default; a malformed value WARNs and falls back to the DEFAULT, never to structure. A completed check over 5 minutes logs an operator WARN naming R-401 — **one data point, on one 134 MB store, so no rotation schedule, size threshold or bandwidth budget was invented from it.** Proven live at both depths 2026-08-31 with the restic argv observed from the guest |
|
||
| ~~**R-87**~~ **— SHIPPED 2026-08-31 (controller v0.231.0), RE-SCOPED. AND IT IS STILL NOT R-359** | ~~The restic tier is never restore-TESTED~~ **The box now PROVES its own off-site copy still HOLDS something.** **THE DISTINCTION, STATED ONCE AND PLAINLY BECAUSE THE GREEN TICK INVITES THE OTHER READING: this proves the snapshot CONTAINS a recoverable unit; it does NOT prove a restore puts data back into a running app.** The proof restores to a throwaway folder, judges it against the unit's own captured compose, and deletes it — it never touches a live app. Putting data back is drill work. **§8 matrix row 4 is deliberately NOT moved.** Nightly at 05:30, one app, due-ness per SNAPSHOT (R-86's model). Evidence `tests/r87-offsite-proof-2026-08-31/`; the reasoning is `audits/SPIKE-restic-restore-test-2026-08-31.md`. | **SPIKE VERDICT, `audits/SPIKE-restic-restore-test-2026-08-31.md`:** build the NARROW version, not the row as written. **Measured:** a scratch restore of all 8 apps costs **25 s / ≤213 MB scratch**, LESS than the 40.3 s weekly check beside it; but restic 0.14.0's `--verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving corruption passed clean, red-proofed), `restic ls --json` carries no content hash, and the unit manifest hashes **4 918 B of a 213 231 242 B unit** — so **no reference for "correct" exists** (R-409). **Of the five drill-found restore defects R-353/354/356/358/403, an unattended scratch-restore would have caught ONE (R-356).** The value is elsewhere and the weekly check structurally cannot reach it: `check` proves the stored bytes are the stored bytes, never that we stored the RIGHT thing — a hollow unit backs up, checks and restores cleanly and recovers nothing (R-403, measured 2026-08-31). **Proposed re-scope, Viktor's call:** *prove the off-site snapshot still CONTAINS a recoverable unit* — one app a night, restored to scratch, checked against its own `manifest.json` via the existing `unitCarriesData`. Must use `--no-lock` and skip `unlockStale` (R-95's constraint is otherwise violated — R-407/R-408 record what the path writes today) and must take `acquireRunning`, which `RestoreOffboxScratch` does not. | matrix row 4's route has no unattended proof. **Stated explicitly because the two rows sit next to each other and are easy to conflate: a `check` proves the STORE IS READABLE; a restore-test proves DATA COMES BACK OUT.** v0.227.0 did the first and nothing else. This row is untouched by it |
|
||
|
||
### 10.3 Divergences that are documented elsewhere and are not re-opened here
|
||
|
||
**[FACT]** `06-offsite-connectivity.md:19-21` describes the operator's public edge as a Cloudflare
|
||
Tunnel and states DooPlex has no public IP. Live DNS resolves `hub.felhom.eu` through a no-ip
|
||
DynDNS CNAME straight to DooPlex's own public address (INV Part E.3). That is a topology-document
|
||
issue, not a recovery-model one; recorded so the discrepancy is not lost.
|
||
|
||
---
|
||
|
||
## 11. Open decisions — for the operator
|
||
|
||
**Recorded, deliberately not answered.**
|
||
|
||
**A. Escrow custody.** Split custody (R **or** an offline operator key) versus a 2-of-3 threshold
|
||
across customer / hub / box drives.
|
||
*Recommendation on record:* **split custody, operator key held offline and never in the hub.* Note
|
||
the constraint from §2: the moment the operator key lives in the hub, "a compromised hub yields
|
||
blobs nobody can open" stops being true.
|
||
|
||
**B. Lost-R policy.** Under split custody the operator **can** recover. Is that the stated policy —
|
||
and if so, what identity check gates it? Today matrix row 13 has no route at all, and the customer
|
||
is not told that losing R costs them the host-loss route.
|
||
|
||
**C. RTO / RPO targets per scenario.** **None have ever been stated.** Without them §8 cannot judge
|
||
whether, for example, offsite-only recovery is acceptable for drive loss, or whether the 7-day PBS
|
||
cadence is adequate for row 8. The measured column is the input; the target column does not exist.
|
||
|
||
**D. Hetzner as a single failure domain.** restic (Storage Box `u629488`, sub-accounts per customer)
|
||
and PBS (Cloud server ep0 + a Cloud Volume) are both Hetzner. Whether they share an **account,
|
||
login and payment method** is unverified (INV Unknown U-1) — the question was deliberately not
|
||
answered by calling the provider API with the production token. Accept explicitly, or mitigate.
|
||
|
||
> **The fence was reached again, and held, 2026-09-01 (SPIKE R-95).** The R-95 spike needed one
|
||
> provider-side fact — **do daily snapshots exist on the Storage Box, and how many** — because the
|
||
> register calls that mitigation ARMED and the whole ranking of R-95's options turns on it. It is
|
||
> readable from the box: measured on BOTH machines over their own SFTP credential, with positive
|
||
> and negative controls, **no `.snapshots` is visible to either sub-account**, and the account is
|
||
> jailed at `/`. The register's own confirming field, `size_snapshots`, is an API field. **So the
|
||
> spike stopped here rather than answering it by acting**, and left it as R-429 — ten minutes in
|
||
> the Storage Box panel, and it re-ranks R-95. `audits/SPIKE-r95-offsite-delete-2026-09-01.md` §Q1.
|
||
|
||
**E. `local` vzdump shares a physical device with the guest it backs up.** LIVE on both hosts:
|
||
`/var/lib/vz` (the archive target) and `local-lvm` (the guest's rootfs and both `backup=1`
|
||
mountpoints) are both on `/dev/sda3` → VG `pve`. The tier therefore protects against **corruption
|
||
and operator error only**, never against disk failure. Accept and name it honestly in the customer-
|
||
facing description, or move the target.
|
||
|
||
**F (added by §10.1, not in the original list). — ANSWERED 2026-07-30.** D5 could not be adopted
|
||
until R-108 closed. Closing R-108 *was* the intended path, and it is done (controller v0.187.0): the
|
||
operator ruled to refuse app namespaces on network storage rather than narrow the browsing surface,
|
||
because the share-root bind is load-bearing and cannot be scoped. **D5 is no longer blocked.** Whether
|
||
to now *implement* D5 remains an open scheduling decision, not a blocked one.
|
||
|
||
---
|
||
|
||
## 12. Evidence index
|
||
|
||
| Claim | Grade | Source |
|
||
|---|---|---|
|
||
| Tier-2 file restore, gap-fill and after total loss | **PROVEN-LIVE** | CAMPAIGN-9 A1/A3 |
|
||
| Tier-2 refuses without an outage for a no-coverage app | **PROVEN-LIVE** | v0.183.0 replay, `felhom.eu/REPORT.md:60` |
|
||
| Tier-1 unit restore **executes** | **PROVEN-LIVE** | CAMPAIGN-9 A2 |
|
||
| Tier-1 **content recovery after loss** | **UNPROVEN** | `CAMPAIGN-9…:825-827` |
|
||
| restic restore of app data (bytes) | **PROVEN-LIVE** | CAMPAIGN-8 R-87, 155.5 MB, 6/7 byte-identical |
|
||
| offsite reconstitution of a DB-indexed app | **PROVEN-LIVE** | destructive immich drill 2026-07-20, `00-capability-map.md:66,75` |
|
||
| offsite **place-to-live** as a distinct action | **UNPROVEN** | `CAMPAIGN-8…:520` |
|
||
| shares restore (files + definitions + credential) | **PROVEN-LIVE** | 2026-07-18, `00-capability-map.md:96` |
|
||
| `.fab` drive-to-drive round trip | **PROVEN-LIVE** | CAMPAIGN-6D P-FAB, 1.7 GB |
|
||
| `.fab` **browser upload** leg | **UNPROVEN** | `00-capability-map.md:67` |
|
||
| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
|
||
| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
|
||
| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
|
||
| unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) |
|
||
| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
|
||
| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
|
||
| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
|
||
| agent DR bring-up (`ModeDRGuestLoss`) | **NEVER EXECUTED** | `CAMPAIGN-8…:522` |
|
||
| host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` |
|
||
| host rebuilt as its former self | **NEVER DONE** | INV Part D1 |
|
||
| escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` |
|
||
| hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 |
|
||
| a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` |
|
||
|
||
---
|
||
|
||
## 13. What this document deliberately does not do
|
||
|
||
- It does not restate capability status — §8 cites the map, the map cites §8.
|
||
- It does not resolve the 7/53-vs-9/43/1 count (§6.2). Both are on the record; neither is adopted.
|
||
- It does not estimate a single RTO or RPO. Every blank in §8 is a real gap.
|
||
- It does not answer §11. Those are the operator's.
|
||
- It does not claim ratification.
|
||
| 5 | **A customer had no way to BEGIN — the only route was a command line** | Everything above was self-service, and nothing told the owner of a rebuilt box that a sealed package was waiting or how to open it. | **CLOSED — controller v0.200.0** (R-193). A full-page recovery screen, shown while the hub holds a package this box cannot open; it unlocks and lists, and **restores nothing** (→ R-213 for the put-back). |
|