From 070b0ce2dc987c9cc2213ce643c8bdbb98a4ec5b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 28 Jul 2026 20:50:41 +0200 Subject: [PATCH] 07-backup-architecture: rewrite as the recovery model (D1-D6); D5 BLOCKED by R-108 Replaces the 2026-07-14 DRAFT entirely - it was verified against controller v0.132.0, 51 versions stale, while being cited as authoritative. Now verified against controller 0.183.0 / agent 0.110.0 / hub 0.80.0 / catalog 4252121. NOT ratified. Every statement is labelled [DESIGN] (a decision from the architecture discussion) or [FACT] (file:line, live output, or a citation to _recovery-inventory-2026-07-28.md). Phase 0 verified D5's premise and it FAILED. Ten read surfaces were checked; nine are clean (SMB under both namespace shapes, FileBrowser for local drives, .fab import and download, /api/debug/*, all three ServeFile sites, registering the backup dir as a drive). One is not: FileBrowser binds a NETWORK share at its ROOT while local drives are userdata-scoped, and no IsNetwork() filter guards the deploy dropdown, the per-app migrate targets, or handleStorageMigrateApp. So an app namespace - and under D5 its plaintext secrets - can land somewhere a browsing surface serves with download:true. Not a leak today (the unit's app.yaml is secret-stripped); it becomes one under D5. D5 is therefore written as a BLOCKED target, not as settled. -> R-108 Core artifact is the failure -> recovery matrix (15 rows). RTO cells carry only measured durations; 12 are blank and each blank is listed explicitly as a finding. RPO cells carry the configured cadence that bounds RPO, labelled as such - no RPO has ever been measured from an incident and none is invented. The 7/53 vs 9/43/1 Tier-2 coverage counts are both recorded, unresolved; neither adopted. Coupling (S-1): capability map section C now cites the matrix instead of restating routes; ROADMAP gains R-102..R-108; OPEN-ITEMS gains only the in-flight R-108 plus C9-F1b/F3/F4 -> R-n cross-references; CONTEXT records D1-D6 as standing ruling S-3. No code, no version bump, no deploy, no box touched. --- CONTEXT.md | 39 + .../architecture/00-capability-map.md | 24 +- .../architecture/07-backup-architecture.md | 928 ++++++++++-------- documentation/backlog/OPEN-ITEMS.md | 7 +- documentation/backlog/ROADMAP.md | 18 + 5 files changed, 610 insertions(+), 406 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 4caad30..da3a348 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -25,6 +25,45 @@ reader meets the warning before the content, not after acting on it. Origin: versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying or retiring it is → **R-83**. +**S-3 — the recovery model: six decisions, 2026-07-28.** Taken in an architecture discussion and +expressed in the `07-backup-architecture.md` full rewrite (which replaces the 2026-07-14 DRAFT +entirely — that doc was verified against controller v0.132.0, **51 versions stale**, while being +cited as authoritative). They are **decisions, not observations**; the rewrite labels every statement +`[DESIGN]` or `[FACT]` so a reader can tell which is which. + +- **D1 — two lanes, different owners.** *Customer, unassisted:* files and app data, through the + „Visszaállítás" surfaces. *Operator:* LXC guest and host recovery. This is a deliberate product + decision, not a limitation to apologise for. +- **D2 — encryption follows the boundary, not the tier.** Data on the customer's own drives is + **plaintext**, deliberately: someone who can take the second drive can take the first, local + encryption adds a key-loss path that turns a working backup into a brick, and it would break + FileBrowser and SMB browsing, which is a feature. Data **leaving the premises** is encrypted + (restic, PBS) — the provider must not be able to read it. +- **D3 — recovery is a chain, not a set of tiers.** Every app-tier restore is conditioned on the + whole-guest tier: Tier-1 needs the guest's secrets (`restore_unit.go:17-22`, *"Regenerate + NOTHING"*), and Tier-2/Tier-3 need the app still deployed. **3-2-1 describes copies; it does not + describe recovery.** This is the reason the rewrite exists. +- **D4 — three-part model.** *Recipe* (scaffolding, hub) · *Escrow* (identity key + restic repo + password, hub, R-wrapped) · *Bytes* (the tiers). Losing one is a different problem from losing + another, and the failure→recovery matrix is organised around that. +- **D5 — R's role narrows. CONDITIONAL, and currently BLOCKED.** The target is app secrets travelling + with the LOCAL recovery unit, so Tier-1/Tier-2 restore work without the guest and without R, + leaving R needed for offsite recovery and host identity only. **Not adopted:** the premise + (the backup tree unreachable from every browsing/download surface) was tested and failed — see + **R-108**. Every other surface verified clean; the exposure is FileBrowser's root-bind of network + shares combined with the absence of an `IsNetwork()` filter on the deploy and per-app-migrate + paths. +- **D6 — the trust model, stated honestly.** The operator holds root SSH on every box; *"the + operator cannot read customer data"* was never the actual security property and no document may + imply it. What R **does** provide, and must keep providing: **the hub alone is not enough** — a + compromised hub yields blobs nobody can open, *provided the operator's key is never stored in the + hub*. That proviso is why escrow custody is an open decision (`07` §11-A). + +Five decisions were deliberately **left open for the operator** and are recorded, unanswered, in +`07` §11: escrow custody · lost-R policy · per-scenario RTO/RPO targets (**none have ever been +stated**) · Hetzner as a single failure domain · and `local` vzdump sharing a physical device with +the guest it backs up. Gaps minted the same session: **R-102 … R-108**. + - **2026-07-27 — R-85: the DR tier is restore-tested UNATTENDED, and its failure is HEARD.** agent v0.104.0 + hub v0.77.0. R-82 proved a PBS restore round-trip **by hand**; three defects kept the unattended path from covering it. (a) `buildRestoreTestScheduler` built ONE runner on diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index af1f417..2a48ba9 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -58,21 +58,33 @@ ## C. Protection & recovery (the product promise) +> **Coupling (2026-07-28, S-1).** The **failure → recovery matrix** in +> `07-backup-architecture.md` §8 is authoritative for *which failure has which recovery route, who +> can invoke it, and what its measured RTO is*. This section stays authoritative for *per-capability +> status*. **Neither restates the other** — rows below carry a `→ 07 §8 row n` pointer instead of +> repeating the route. Where a row's status and the matrix's status differ in wording, the matrix is +> about the **failure** and the row is about the **mechanism**; that is not a contradiction, and both +> cite the same evidence. +> +> The matrix's blank RTO/RPO cells are deliberate: no number is estimated anywhere. Two counts of +> Tier-2 app coverage disagree (9/43/1 vs 7/45/1) and are **both** recorded there, unresolved — +> do not adopt either from this page. + | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| -| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 | -| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | | -| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` | -| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** (2026-07-20) | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* **2026-07-20: the clean run HAPPENED** — endpoint-level supervised reconstitute of immich from snapshot `49e7cb46` (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no `already exists`, operation reported SUCCESS, immich's own DatabaseService logged `No schema drift detected` twice, 11 assets `active`, 4/4 containers healthy, 231 `public` indexes. **Operator confirmed the immich timeline renders correctly after the reconstitute** (screenshot held, 2026-07-20). Evidence: `felhom-controller/REPORT.md` §4b. **2026-07-20, LATER THE SAME DAY — the destructive drill RAN and the row now earns PROVEN-LIVE.** The operator deleted the photos in immich own UI **and emptied the trash** (the step whose absence makes a drill prove nothing — the round-1 lesson), then restored through the customer-facing UI. **`40 file(s) placed`** against the 6 of the earlier non-destructive run — the files were really gone and really came back — plus 1 DB dump replayed rc-0, 11 assets `active`, `No schema drift detected`, timeline confirmed by the operator. This is the destroy-then-recover proof the 6D downgrade asked for, and it was taken through the customer own buttons, not endpoint shortcuts. Evidence: `felhom-controller/REPORT.md` 4e | +| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 **DB replay route → `07-backup-architecture.md` §8 row 3** | +| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | **Route + RTO → `07-backup-architecture.md` §8 rows 1, 2, 4, 5.** The matrix records that the copy's `recovery-unit/` mirror is read by no path (→ R-102) | +| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect) | +| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** (2026-07-20) | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* **2026-07-20: the clean run HAPPENED** — endpoint-level supervised reconstitute of immich from snapshot `49e7cb46` (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no `already exists`, operation reported SUCCESS, immich's own DatabaseService logged `No schema drift detected` twice, 11 assets `active`, 4/4 containers healthy, 231 `public` indexes. **Operator confirmed the immich timeline renders correctly after the reconstitute** (screenshot held, 2026-07-20). Evidence: `felhom-controller/REPORT.md` §4b. **2026-07-20, LATER THE SAME DAY — the destructive drill RAN and the row now earns PROVEN-LIVE.** The operator deleted the photos in immich own UI **and emptied the trash** (the step whose absence makes a drill prove nothing — the round-1 lesson), then restored through the customer-facing UI. **`40 file(s) placed`** against the 6 of the earlier non-destructive run — the files were really gone and really came back — plus 1 DB dump replayed rc-0, 11 assets `active`, `No schema drift detected`, timeline confirmed by the operator. This is the destroy-then-recover proof the 6D downgrade asked for, and it was taken through the customer own buttons, not endpoint shortcuts. Evidence: `felhom-controller/REPORT.md` 4e **Route + RTO → `07-backup-architecture.md` §8 rows 3, 4** — the matrix also records that no offsite action unpacks the named-volume tars it captures (→ R-107) | | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D | -| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill | +| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill **Route + RTO → `07-backup-architecture.md` §8 rows 6, 8, 9** — measured 84–112 s local / 1101 s PBS into a scratch guest; a restore to a DIFFERENT host is unmeasured | | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending | | Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0, agent v0.92.1 | **PROVEN-LIVE** (2026-07-21) | All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed | **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21** (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (`compose down` → 0 containers), calibre-web and bookstack left running, then `pct reboot 9201`: the gate recreated calibre-web and logged `1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`; immich came back **stopped**, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The **static-guest** half of the network leg stays deliberately out of scope → **R-50** | | Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) | | An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 | | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped | | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | | -| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count | +| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise | ## D. Storage & devices diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 6b7eeb8..4f1f806 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -1,477 +1,611 @@ -# 07 — Backup architecture: tiers × classes × targets +# 07 — The recovery model -> **Status: DRAFT — awaiting Viktor's review (async one-line vetoes on the §10 list).** -> **NOT ratified. Ratification is Viktor's review, not an editor's.** -> > | | | > |---|---| -> | **Written** | 2026-07-14 (architecture-doc-first gate, Viktor ruling #5) | -> | **Body verified against** | felhom-controller `95f3180` **v0.132.0**, agent `c040c18` **v0.88.0**, hub **v0.54.0**, catalog `21e8df1` | -> | **PBS/whole-guest tier (§2 row, §9.1) verified against** | agent **v0.102.0**, controller **v0.175.0**, hub **v0.76.0**, host-install **1.20.0** (2026-07-26, R-82) | -> | **Verdict** | ⚠️ **STALE outside the PBS tier.** The controller-side tiers (1/2/3) were last verified 41 controller versions ago (live is v0.175.0). Treat §0–§8 as historical intent, not current fact, until re-verified. | +> | **Status** | **NOT RATIFIED.** Ratification is Viktor's review, not an editor's. | +> | **Written** | 2026-07-28 (full rewrite; supersedes the 2026-07-14 DRAFT entirely) | +> | **Verified against** | controller **v0.183.0** · agent **v0.110.0** · hub **v0.80.0** · catalog `4252121` · repo HEADs `felhom.eu ff050cf`, `felhom-controller fd50a73`, `felhom-agent d5c7691` | +> | **Live fleet at verification** | demo-felhom + demo-hp, both guest 9201, both on the versions above | +> | **Freshness** | **CURRENT** as of 2026-07-28. Per standing ruling **S-2**, mark this STALE the moment it is more than a few trains behind. The document it replaces was DRAFT since 2026-07-14, verified against controller v0.132.0 — **fifty-one versions stale** — and was cited as authoritative throughout that time. | > -> Per CONTEXT ruling **S-2**, this header exists so a reader meets the staleness warning *before* the -> content. R-82 re-verified only the rows it touched (the PBS whole-guest tier); it did **not** -> re-verify the restic/tier-2 material, and says so rather than implying a freshness it did not earn. -> Line numbers are landmarks — reconfirm before editing. +> **How to read this document.** Two kinds of statement appear, and they are always labelled: > -> Inputs: `SPIKE-backup-classification-2026-07-14.md` (`790ec84`), `CAMPAIGN-6C-2026-07-14.md` -> (`deacee11`, F-6C-1), Viktor's locked rulings of 2026-07-14. This document DECIDES; it does not -> survey. Where a decision rests on an unproven restic mechanism, it is marked **SPIKE** and listed -> in §6.4 — per the spike-first gate, those spikes precede the Task 3a spec. +> - **[DESIGN]** — a decision taken in the 2026-07-28 architecture discussion (D1–D6, §3–§5, §9). +> Not derived from code; the code may not implement it yet. Where it does not, §10 says so. +> - **[FACT]** — an observed property, carrying a `file:line`, a live command output, or a citation +> to `_recovery-inventory-2026-07-28.md` (below: **INV**). +> +> Where the model is silent, this document says **OPEN** rather than filling the gap. +> +> **Primary input:** `_recovery-inventory-2026-07-28.md` (read-only inventory, 2026-07-28) — cited +> throughout as **INV Part n**. Every number in §6, §8 and §11 traces back to it. --- -## 0. Verified baseline — what the code does today (source, not memory) +## 1. Purpose and scope -| Surface | Behavior | Evidence (v0.132.0) | -|---|---|---| -| Recovery unit (tier-1) | `compose/` (docker-compose.yml, `.felhom.yml`, secret-stripped app.yaml) + enumerated `db-dumps/*.sql` + `volume-dumps/*.tar` + `manifest.json` (SchemaVersion **1**). No userdata, no HDD appdata. | `recovery_unit.go:91-106,131` | -| Offsite (tier-3) | One `restic backup --tag felhom-offbox --tag ` per toggled app; unit dir discovered across nsRoots, newest-manifest tiebreak. Retention `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune`, **default grouping = host+paths**. | `offbox.go:504-552` (discover), `:575` (backup), `:590-597` (forget) | -| Offsite quota | Soft gate on `RepoSizeBytes` from `restic stats --json` with **no mode flag → default restore-size over all snapshots**; ≥100% refuses new backups (prune still runs), ≥80% warns. | `offbox.go:407-415,630-645,700-708` | -| Offsite restore | `restic restore latest --tag --target /offbox-restore/` — scratch, non-destructive, whole snapshot. **DataDir default = `/opt/docker/felhom-controller/data` — the guest rootfs.** | `web/offbox_handlers.go:236-246`, `config/config.go:310` | -| Tier-2 | rsync mirror of unit → `backups/secondary//recovery-unit` + resolver-driven appdata dir → `.../appdata` (flat; **refuses N>1**). `rsync -a --delete`. Target pick: pinned > other data drive > SSD (headroom-gated); **zero fs-type awareness** (F-6C-1). Size for headroom = `dirSizeBytes(unit)+dirSizeBytes(appdata)`. | `tier2.go:170-249` (run), `:92-155` (select), `:437-445` (rsync), `:198` (size) | -| Tier-2 restore | Missing-files-only merge into live appdata: `rsync -a --ignore-existing`; refuses N>1. | `tier2_restore.go:15-44,127-137` | -| Manual `.fab` | Config + DB dump + volume tars + `ExportDataMounts` = `${HDD_PATH}` binds ∪ the **whole userdata root** as one tar (SQ6 over-capture). | `stacks/delete.go:575`, `appexport/export.go` (v0.130.0) | -| Classification (INERT) | `stacks.Manager.ClassifiedBinds(name)` → `([]ClassifiedBind, hasClassification)`; whole-block-reject at `LoadMetadata`; two-level default (explicit > `:ro`-reader > writable-mandatory; **no block → legacy**). Wired seam: `StackDataProvider.GetStackClassifiedBinds` (adapter main.go ~L1106). 13 catalog apps carry blocks (`21e8df1`). | `metadata.go:285-294`, `appbackup/classify.go:97,165` | -| Network storage metadata | `settings.StoragePath.Kind == "network"` + `IsNetwork()`; `Protocol` nfs\|smb. Drive lifecycle already refuses network paths. | `settings.go:181-219,1057` | +This document describes **how a Felhom customer gets their system back**, and who can do it. -Two **new findings** from this verification pass (not in the spike or 6C): +It replaces a document that described **where copies are written**. That was the wrong frame, and +§7 explains why: a set of copies is not a set of recovery routes, and the previous document's tier +table could be entirely satisfied while a real recovery was impossible. -- **F-A1 (HIGH for 3a): the offsite restore scratch dir is a rootfs-filler once snapshots carry - userdata.** `offbox-restore/` lives under `cfg.Paths.DataDir`, default - `/opt/docker/felhom-controller/data` on the ~8 GB guest rootfs. Today snapshots are unit-sized - (MB–1 GB); with mandatory userdata (nextcloud's data dir, immich's upload library) a - `restore latest` can fill the rootfs and take the guest down. §7 decides the fix. -- **F-A2 (MEDIUM for 3a): quota accounting mode is wrong-by-design for the new shape.** - `restic stats` default restore-size aggregates across **all retained snapshots** (~17 per app at - full retention). Exact cross-snapshot counting semantics are undocumented enough to matter: - if a 50 GB mandatory library counts once per snapshot, the gate trips at ~17× reality. §9 decides - the candidate fix (`--mode raw-data`); **SPIKE SP-1** proves it. -- Minor: `tier2.go:167` (RunTier2 doc comment) still claims "recovery unit + **userdata**" — the - last surviving F-S1 stale comment (fourth stale-comment instance this arc). 3b makes it true. +**In scope:** the recovery model — trust boundary, lanes, the three-part model, the tiers as +*inputs to recovery*, the dependency chain between them, the failure→recovery matrix, and the +encryption policy. + +**Out of scope, deliberately:** implementation specs (they live in task specs), the capture-set +algorithm (`internal/appbackup/captureset.go` and its tests are the source of truth), and per-tier +operational runbooks (`documentation/runbooks/`). + +**Authority split.** This document is authoritative for the **failure→recovery matrix** (§8). The +capability map (`00-capability-map.md`) stays authoritative for **per-capability status**. Neither +restates the other; §8 rows are cited from the map, not copied into it. --- -## 1. The class model (normative recap) +## 2. The trust model -A class belongs to a **bind (host path)**, not to an app (SQ2 rule, locked): +**[DESIGN] The operator holds root SSH on every box.** That is a fact of the product — the agent is +operator-tier, the host is operator-managed, and break-glass exists precisely so the operator can +get in when nothing else works. **"The operator cannot read customer data" was never the security +property**, and no part of this document may be read as claiming it. -- **mandatory** — referentially COUPLED to app state (DB rows reference the content; SQ3: restore - without it = broken-not-empty). Never deselectable at any tier that carries userdata. -- **optional** — DECOUPLED but precious (hand-curated, not re-acquirable). Customer-selectable. -- **excluded** — DECOUPLED bulk/transient/re-derivable. Never captured automatically; `.fab` opt-in - only, behind the two-number warning. -- **legacy** (origin, not a class) — app has no `backup:` block → every tier behaves byte-identically - to v0.132.0. The safety net fails toward *today's cost*, never toward a quota blow-up (SQ5). +**[FACT]** The mechanics that make this concrete: -Roots: `${HDD_PATH}` (`hdd:` list; appdata stores) and `${USERDATA_PATH}` (`userdata:` list; -browsable tree). RelPaths are `${VAR}`-relative and hddPath-invariant (survive migration flips — -the Task 1 lesson). Resolution to absolute paths happens at capture time against the app's live -`HDD_PATH` (Model A: HDD_PATH == nsRoot). +- The operator's OOB SSH public key is pushed to every box from the hub + (`hub_settings.oob_operator_ssh_pubkey` → `/var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op`, + 93 bytes, LIVE on both hosts — INV Part C, row 21). +- The break-glass `root@pam` console password for every host is stored in the hub and retrievable + with the operator's global key (`documentation/runbooks/break-glass.md:42-48`; LIVE: + `host_recovery` holds 3 rows — INV Part D2.1). +- Guest data is reachable from the host by definition: `pct exec`, and the data drives are bind + mounts on the host (`mp8 /mnt/felhom-drives`, LIVE `pct config 9201` on both hosts). + +**[DESIGN] What R (the customer's recovery code) does provide, and must keep providing: the hub +alone is not enough.** A compromised hub yields blobs nobody can open. This property holds **only +while the operator's own key is not stored in the hub** — which is why §11-A is an open decision and +why its recommendation on record is "operator key held offline and never in the hub". + +**[FACT]** The escrow is genuinely zero-knowledge today: `host_escrow` rows carry +`posture = zero_knowledge`, a 383-byte blob and a 572-byte identity blob, and the hub holds only a +`restic_pw_sha256` **hash** beside them (LIVE, both hosts — INV Part D2.1). R exists in **zero** +system copies by design (INV Part C, row 24). + +So the honest statement of the property is: + +> **The hub cannot open the escrow. The operator can reach a live box. Neither fact substitutes for +> the other, and R is what keeps the first true.** --- -## 2. The decision table: tier × class × target +## 3. The two lanes (D1) -| Tier | legacy (no block) | mandatory | optional | excluded | Allowed targets | Copy mechanism | Restore path | -|---|---|---|---|---|---|---|---| -| **Tier-1** recovery unit | config + DB dumps + volume tars | *unchanged* — the unit never carries userdata; classes ride inside it via the captured `.felhom.yml` | — | — | app's own drive (`backups/primary/`) | direct writes | `RestoreApp` (existing) | -| **Tier-2** cross-drive | unit + resolver-appdata (byte-identical to v0.131.0) | unit + mandatory binds | + optional binds | never | **real local drives only** (network paths excluded per F-6C-1 ruling — both auto and pinned); SSD fallback = unit + mandatory **iff** headroom fits, optional skipped with honest reason | `rsync -a --delete` per capture-set path | missing-only merge (`--ignore-existing`), per-path from the new layout (§8) | -| **Tier-3** offsite restic | unit only (today's shape) | unit + mandatory binds — **not deselectable** | never | never | Hetzner Storage Box (SFTP) | one multi-path restic snapshot per app per run (§6) | staged scratch on a data drive → missing-only merge to live (§7) | -| **Manual `.fab`** | v0.130.0 full-root capture | locked-in (not deselectable) | checkbox, **pre-selected** | opt-in, behind the two-number size warning + FileBrowser pointer | download / chosen drive | tar, **exclusion-scoped** root (SQ5 verdict; manifest v1 unchanged) | existing import (old controllers import new bundles correctly) | -| **PBS** whole-guest (R-82) | rootfs + `/var/lib/docker` + `/mnt/sys_drive`; bind mounts (`/mnt/felhom-drives`, `/etc/felhom-bootstrap`) out of reach | *unchanged* | | | **`felhom-pbs` → datastore `felhom-offsite` on ep0 (Hetzner), per-customer namespace, reached over `wg-felhom`** — NOT "PBS on DooPlex" (that was the 2026-07 spike store) | vzdump, **WEEKLY** (`cadence_seconds: 604800`), retention **keep_last=2** (two weeks, operator ruling 2026-07-26) | whole-guest restore | +**[DESIGN] Recovery is split into two lanes with different owners. This is a product decision, not a +limitation.** -**Restore-proof per tier (R-85, 2026-07-27).** A tier is not proven by having archives; it is proven -by one of them restoring into a bootable, mount-complete guest. Both tiers are now restore-tested -**unattended**, rotating oldest-proven-first at the restore-test cadence, and each tier's last -successful proof is reported. The hub raises two DISTINCT operator signals: `restore_test_failed` -(a run did not pass — broken now) and `restore_test_stale` (not proven within ~7 days — *unverified*, -which is not the same claim). Before this the scheduler could only ever see the primary tier, so the -PBS row above was scheduled-but-never-verified. See `03-host-agent.md` §8. +### Lane 1 — the customer, unassisted: files and app data +The customer owns their data and can get it back themselves, through the „Visszaállítás" surfaces, +with nothing but their dashboard password. No operator, no ticket, no scheduling. -Row-level decisions folded in: +**[FACT] What Lane 1 contains today** (INV Part A.1 — seven paths, all behind the controller's +`RequireAuth` gate, which is the **customer-owned** password: `internal/web/auth.go:34-35` puts +`settings.json → password_hash` ahead of the operator-provisioned `controller.yaml` value): -1. **Classified apps' tier-2 appdata leg becomes fully class-driven** — per-bind paths, not the - whole appdata dir. Consequence: paperless-ngx's tier-2 copy *shrinks* (`appdata/paperless/media` - mandatory in, `appdata/paperless/export` excluded out) — correct per the model, and the N>1 - refusal disappears for classified apps because per-bind capture never needed the single-dir - assumption. The F-S2 resolver remains **only** for legacy apps. -2. **The SSD fallback is a state-only tier.** Its existing Hungarian reason already says so - ("csak az adatbázis/konfiguráció fér a belső SSD-re"); the engine now enforces it: unit + - mandatory if `tier2FitsHeadroom` passes for that reduced set; optional never goes to the SSD. -3. **Offsite never carries optional.** Optional is the customer's local-copy tier; offsite cost is - shared-model quota. (Viktor ruling #1, restated for the matrix.) -4. **Undeployed apps** (unit discovered on a drive, stack not deployed): offsite pushes the - **legacy unit-only shape** + WARN. Rationale: mandatory-path resolution requires the live - HDD_PATH; guessing it from a stripped app.yaml risks capturing a stale or foreign tree. No - protection regression vs today. -5. **Declared-but-absent mandatory path** (compose/classes declare it, dir missing on disk): the - F-S2 pattern — skip that path, **loud WARN**, capture the rest. The stat-filter before the - restic invocation is MANDATORY and is the ONLY detection point: restic 0.14.0 (the pinned - production binary) does NOT error on a nonexistent source path — it skips with a stderr - warning, exits 0, and silently writes a partial snapshot - (SPIKE-restic-snapshot-shape-2026-07-14.md, SP-3.4). 3a must never rely on a nonzero restic - exit to catch a missing mandatory path; the controller stat-filters to detect the absence - itself and raises the WARN from its own check. (If the container's restic is ever bumped, - SP-3.4 must be re-run — later releases changed this exit behavior.) +| # | Surface (HU) | Endpoint | Semantics | +|---|---|---|---| +| 1 | „Visszaállítás indítása" | `POST /backup/restore` | destructive — rebuilds the app from its Tier-1 unit | +| 2 | „Fájlok visszaállítása" | `POST /backup/tier2/restore` | additive, missing-only | +| 3 | „Visszaállítás a távoli tárolóból" | `POST /backup/offbox/restore` | non-destructive, to a verification copy | +| 4 | „Helyreállítás az élő adatok közé" | `POST /backup/offbox/place` | additive, missing-only | +| 5 | „Teljes visszaállítás (fájlok + adatbázis)" | `POST /backup/offbox/reconstitute` | destructive to files + DB, never deleting | +| 6 | „Megosztások visszaállítása" + place | `POST /backup/shares/{restore,place}` | additive | +| 7 | `.fab` import | `POST` → `apiImportStart` | destructive re-import of one app | + +**[FACT] What is proven in Lane 1** (INV Part G.1): paths 2 and 6 are proven live end-to-end; +path 5 is proven live through a destructive drill (photos deleted, trash emptied, restored, timeline +confirmed); path 3 is proven for **bytes** only; path 7 is proven for the drive-to-drive circle but +its browser-upload leg is not; path 1 is proven to **execute** but its **content recovery after real +loss has never been demonstrated** — the single most valuable unproven item in the system +(`audits/CAMPAIGN-9-restore-proof-2026-07-28.md:825-827`); path 4 has never been exercised as a +distinct action (`audits/CAMPAIGN-8-backup-restore-2026-07-27.md:520`). + +**[FACT] One honest residual on the whole lane:** every proven run was performed by the **operator**, +not by a customer. `00-capability-map.md:75` still carries "A customer (not the operator) performs a +restore via UI alone" as **MISSING as evidence** — by absence of the run, not by a product gap. + +### Lane 2 — the operator: guest and host recovery + +Rebuilding an LXC guest, rebuilding a host, and re-establishing a box's identity are **operator +work, by design and by contract**. They are not customer-facing and are not going to be. + +**[FACT] What Lane 2 contains** (INV Part A.2): the scheduled restore-test, `pct restore`, raw +`proxmox-backup-client restore`, raw `restic restore`, the agent's DR bring-up +(`--selftest=bring-up -mode dr`), the host-loss plan builder, and the escrow-consume ceremony. Each +needs root on the host or a CLI flag; none is reachable from any customer surface. + +**[FACT] What is proven in Lane 2** (INV Parts G.1, G.3): whole-guest `pct restore` from both tiers +is proven with exact mount parity; the unattended restore-test is proven and currently running on +both boxes; a corrupted snapshot is proven to fail cleanly. **The DR bring-up path has never been +executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by construction** +(`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self** +(INV Part D1). + +### Why the split is right, stated once + +**[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about +"restore the LXC and re-attach mp8 by durable_id without binding another guest's bootstrap +credentials" — a hazard serious enough to have its own runbook page +(`runbooks/RUNBOOK-manual-guest-restore.md:45-48`, F-OPS). Putting Lane 2 behind the operator is +what lets Lane 1 be a button instead of a procedure. --- -## 3. Capture-set computation (Task 3-core) +## 4. The three-part model (D4) -One pure function, leaf package `appbackup`, consumed by all three engines (as-built, v0.133.0): +**[DESIGN] Recovery needs three things, and they live in three different places.** Losing one is a +different problem from losing another, and the matrix in §8 is organised around that. -``` -ComputeCaptureSet(binds []ClassifiedBind, hasClassification bool, - tier CaptureTier, hddPath string) CaptureSet - → CaptureSet{HasClassification bool; Paths []CapturePath; Skipped []SkippedPath} -``` +| Part | Holds | Where | Lost when | +|---|---|---|---| +| **Recipe** | scaffolding — guest sizing, drive inventory, PVE storage definitions, PBS coordinates, app inventory + storage bindings. **No secrets, no bytes.** | the hub | the hub is lost | +| **Escrow** | the host identity key material and the restic repo password | the hub, R-wrapped | the hub is lost, **or** R is lost | +| **Bytes** | Tier-1 / Tier-2 / Tier-3 / whole-guest | the tiers | the relevant medium is lost | -- Resolves each bind's `(Root, RelPath)` against `hddPath` (HDD) / `hddPath+"/userdata"` - (USERDATA) into absolute paths (slash algebra — `path.Join`, never `filepath`); filters by the - tier column of §2 (`TierOffsite` = mandatory only; `TierSecondary` = mandatory + optional; - excluded is silently dropped). Each `CapturePath` carries `{Abs, Root, RelPath, Class}` (the - relpath is what the tier-2 layout and restore need). -- The former `UnitOnly` field is replaced by **`HasClassification`** (it mirrors the classifier's - existing bool); engines derive unit-only as `!HasClassification || len(Paths)==0`. When - `hasClassification == false` the function short-circuits to `{HasClassification:false}` — nothing - resolved, `Paths`/`Skipped` nil — so the legacy path stays byte-identical and testable (the SQ5 - cost-regression guard: an unmigrated app never resolves a bind into an automatic tier). -- **`Skipped`** carries would-be captures the tier filter selected but that are structurally unsafe, - each with an English `Reason`, so the pure function stays log-free (engines log a skipped - **mandatory** as a loud capture GAP). Three structural guards, evaluated after the tier filter: - traversal (a `..` path segment or an absolute RelPath — **load-bearing because the compose parser - path.Cleans but does NOT reject `..`, and `ValidateBackupSpec` only vets *spec* entries, so an - unlisted writable `${HDD_PATH}/../x` bind arrives classed mandatory**), bare HDD drive-root - (RelPath `""` — would nest `/backups` into the capture), and the reserved `backups/` - zone; a bare **userdata** root (`/userdata`) is allowed (it does not nest `backups/`). -- After filtering+guards: an equal-`Abs` collision from two binds collapses to one entry with - **mandatory winning** over optional (mandatory semantics never degrade); then **containment - dedup** drops any path whose ancestor is already in the set (keep the ancestor); output is sorted - by `Abs` (deterministic despite map/slice input). -- `CrossAppOverlaps(map[app]CaptureSet) []Overlap` is shipped here as a **pure** function (exact-Abs - match across ≥2 apps' `Paths`; cross-app *containment* is legitimate per §4 and is NOT an overlap). - Its WARN **wiring** lands in 3a/3b, not in 3-core — no log call sites exist yet. -- Pure: no FS, no docker, no logging (the Tasks 1/2 leaf-package discipline). The stat-filter - (decision 2.5) lives in the engines, not here. -- Companion red-proofs: (a) sonarr with no block must produce unit-only (`HasClassification:false`, - empty `Paths`) for offsite (the SQ5 cost-regression guard — the test that fails if the two-level - default is miswired); (b) immich's explicit-optional `:ro` external library must appear in - tier-2's set and NOT in offsite's; (c) paperless's `export` must appear in no automatic tier. +**[FACT] The Recipe exists and is current.** `dr_recipe` holds 6 rows; demo-felhom's was updated +`2026-07-28 17:31:13` and demo-hp's `17:25:31` — i.e. within one report cycle (LIVE, INV Part D2). +It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_bindings`. -The engines consume the existing wired seam `GetStackClassifiedBinds` (main.go ~L1106) — no new -seam. Per the F-S3 lesson, 3-core still ships an end-to-end wiring test through a real Manager. +**[FACT] The Escrow exists and is zero-knowledge.** `host_escrow`: 2 rows, `posture` `zero_knowledge`; +the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}` +(`felhom-agent/internal/escrow/identity.go:26-39`). + +**[FACT] Three parts of the Recipe are empty or wrong on the live fleet**, and they are exactly the +parts a host-loss recovery would read (INV Part D2.3): `hosts.dr_record_json` is `{}` on all three +hosts; `host_escrow.directive_json` is `{}` on both escrowed hosts; `dr_recipe.host_half.drives` is +`[]` on every customer including two with enrolled data drives; and `dr_recipe.host_half.pbs.namespace` +reads `"root"` while the real namespaces are `demo-felhom` / `demo-hp`. → **R-105**, **R-106**. --- -## 4. Cross-app dedup and writing authority +## 5. Encryption policy (D2) -**Capture attribution rule (refined).** SQ2's "the writing app is the classification authority" -covers apps that write a path — but immich's external library (`media/photos:ro`, explicit -*optional*) is populated by the **customer via FileBrowser**, not by any app. The generalized rule: -**capture attribution = the app whose classified capture set contains the path.** Writing authority -remains the *catalog* rule for who gets to class a path non-excluded. +**[DESIGN] Encryption follows the boundary, not the tier.** -**Dedup decision: the capture engine dedups nothing across apps.** Ground truth from the SQ2 -inventory: the overlap set for non-excluded classes is **empty today** — radarr+sonarr's shared -`downloads/` is excluded on both; the streamers' whole-`media` binds are RO-excluded readers; -calibre-web's mandatory `media/books` merely sits *inside* those excluded reader binds -(containment, not conflict). Engineering cross-app dedup for an empty case buys risk, not value. -Instead: +### On the customer's own premises: plaintext -1. **Catalog convention (normative):** at most one app may class a given host path non-excluded. - Two writers to one path is a catalog bug. -2. **Advisory guard (3-core, log-only):** a cheap cross-app pass at capture time WARNs if two - deployed apps' sets contain the same absolute path — the convention's tripwire, no behavior. -3. Cost note if the convention is ever violated: offsite double-capture is nearly free in real repo - bytes (restic content-defined chunking dedups across snapshots); tier-2 pays disk twice. The WARN - makes it visible before it costs. +Tier-1 recovery units, Tier-2 cross-drive copies, the app data on the drives, and the local vzdump +archives on the host are **unencrypted, deliberately**. -**Restore side.** Per-app restore restores *that app's* captured paths to their recorded relpaths — -attribution is positional, so the containment case is trivially correct: `media/books` restores -under calibre-web's set whether or not plex exists, and restoring plex restores nothing of -`media/` (it captured nothing). A path captured by exactly one app is restored by exactly that app. -The missing-only merge (§7) makes overlapping restores idempotent even in the violation case. +The reasoning, recorded so nobody "hardens" this later: + +1. **It buys nothing against a real threat.** Someone who can take the second drive can take the + first. Local encryption defends against a threat model — physical theft of *only* the backup + medium — that does not describe a home server where both media sit in the same box. +2. **It adds a key-loss path that turns a working backup into a brick.** Every local encryption key + is one more thing that must survive the disaster it exists for, and the system already has one + such dependency it is trying to reduce (§7). +3. **It would break browsing, which is a feature.** FileBrowser and SMB let the household see and + use their own files. An encrypted local copy is not browsable, and the „Megosztás" and + FileBrowser surfaces are part of the product, not an accident. + +**[FACT]** The local plaintext posture is real and verifiable: `dir: local` in `/etc/pve/storage.cfg` +carries no `encryption-key` (LIVE, both hosts), so the daily whole-guest archive is a plain +`.tar.zst` — 5.93 GB on demo-felhom, 1.68 GB on demo-hp (LIVE, INV Part B.3). **That archive is the +single highest-value object on the box**: it contains `encryption.key`, the restic `repo_password`, +the offbox `ssh_key`, `settings.json` and `controller.yaml` (all under `/var/lib/docker` = `mp0`, +`backup=1`). Naming that plainly is part of the policy, not an argument against it. + +### Leaving the premises: encrypted, and the provider must not be able to read it + +**[FACT]** Both offsite tiers encrypt client-side: + +- **restic (Tier-3):** repo password is a 256-bit hex value generated once on the box and never + logged (`internal/backup/offbox.go:392-410`); restic is invoked with + `RESTIC_PASSWORD_FILE` (`:608`). +- **PBS (whole-guest offsite):** `/etc/pve/storage.cfg` carries a **per-customer** `encryption-key` + and the fingerprints differ between demo-felhom and demo-hp (LIVE). The live restore command line + observed on demo-felhom carries `--crypt-mode=encrypt` (INV Part A.2.3). Per-tenant encryption is + why cross-customer dedup is impossible on the shared datastore — a cost consequence, recorded, not + a defect. + +### The one exception, stated because it is not covered by either rule + +**[FACT]** The `.fab` bundle writes `app.yaml` with **decrypted plaintext secrets, deliberately**, +and its password is **optional** (`internal/appexport/export.go:484,506-511`; the generated file's +first line is literally `# Exported by felhom-controller — plaintext secrets`, `:531`; +`Encrypted: req.Password != ""`, `:307`). A `.fab` can also be written to a registered **network** +share, because `storageDriveList()` does not filter network paths +(`internal/web/handler_export.go:377-387`). This is a portability artifact, not a tier — but it is +the one place where a customer action can put every secret of one app onto a NAS in plaintext. +Recorded here so the encryption policy is not read as covering it. → **R-108** (same root cause). --- -## 5. Ownership/permission fidelity per mechanism, and the NAS-as-target stance +## 6. The tiers — what each captures -The motivating failure is F-6C-1: `rsync -a` (`-o -g`) → chown on a root_squash NFS export → -exit 23 → the whole tier-2 run fails. Fidelity matters because userdata carries the setgid -convention (`SharedContentGID=1000`, mode 2775 — `appbackup/userdata.go:19-24`) and appdata stores -carry app-specific uids; a wrong-owner restore into a uid-sensitive app is a *silently broken* -restore. +The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what they can actually do. -| Mechanism | uid/gid | mode incl. setgid | xattrs | Fails on root_squash target? | Verdict | +### 6.1 The four tiers, as configured on the live fleet + +| Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted | |---|---|---|---|---|---| -| `rsync -a --delete` (tier-2) | preserved (`-og`) | preserved (`-p`) | no (`-X` not set — acceptable, nothing in the tree needs xattrs) | **yes** (the F-6C-1 chown) | keep for local drives; never point it at network fs | -| restic backup/restore (offsite) | stored in metadata; restored when run as root | preserved | yes | no (metadata lives in the repo, not on the target fs) | correct for the enlarged offsite shape as-is | -| tar (`.fab`) | preserved in archive; `--same-owner` applies on root extract | preserved | GNU tar: with flags | no | correct; Task 4 verifies the extract flags at spec time | +| **Tier-1** recovery unit | `/backups/primary//` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no | +| **Tier-2** cross-drive | `/backups/secondary//` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/` + `userdata/` | nightly at W+60m | mirror (rsync) | no | +| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) | +| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** | +| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` | **yes** (per-customer key) | -**NAS-as-tier-2-target stance (ruling #4, recorded):** excluded for now — auto AND pinned — with an -honest Hungarian card reason. Mechanism: filter `sp.IsNetwork()` in `selectTier2Target` (both the -pinned-validity check and the auto loop) — the metadata already exists (`settings.go:219`); no -fs-probing. Dropping `-o/-g` was rejected because it converts a loud failure into a silently -wrong-owner restore. **Revisit condition:** a tar-based tier-2 leg (ownership inside the archive) -would make NAS viable; that is a separate future task with its own restore round-trip proof, not a -Task 3 rider. +**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed +offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered +(`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`. + +**[FACT] What the whole-guest tiers do NOT carry.** `mp8 /mnt/felhom-drives` and +`mp9 /etc/felhom-bootstrap` are **host bind mounts** and are out of `vzdump` scope entirely (LIVE +`pct config 9201`, both hosts). So the whole-guest tiers carry the guest and **none of the customer's +data drives** — 916 GB on demo-felhom, 938 GB on demo-hp. + +### 6.2 Coverage per app class — and an unresolved count + +**[FACT]** Of 53 catalog templates, **52 keep data in Docker named volumes**; exactly **13** carry a +`backup:` block, and those 13 are exactly the templates that bind `${HDD_PATH}` / `${USERDATA_PATH}` / +`${IMPORT_PATH}` at all; **14** have a database service (INV Part B.1, computed against catalog +`4252121`). + +Applying the classifier's documented two-level default (`internal/appbackup/classify.go`: explicit +entry wins; else a `:ro` reader is *excluded*; else a writable bind is *mandatory*): + +| tier filter | templates with at least one file leg | +|---|---| +| **Tier-3** (`mandatory` only) | **4** — calibre-web, immich, nextcloud, paperless-ngx | +| **Tier-2** (`mandatory + optional`) | **7** — the 4 above plus audiobookshelf, komga, romm | +| legacy resolver path (apps with no block) | **0** — no non-block template binds a namespace path | + +> ### ⚠️ UNRESOLVED — two counts of the same thing disagree +> +> | source | count | +> |---|---| +> | C9-F1 Phase 0, as shipped | **A = 9 / B = 43 / C = 1** — `felhom.eu/REPORT.md:17-24`, restated `backlog/OPEN-ITEMS.md:31` | +> | INV Part B.1, independent enumeration at catalog `4252121` | **A = 7 / B = 45 / C = 1** | +> +> Both use the same definition ("templates whose Tier-2 copy can hold a readable file leg"). The +> difference is two apps and **neither number is adopted here**. The Phase-0 enumeration is described +> in prose but the script is not committed, so the two methods cannot be diffed from the repo. +> **This must be resolved before either figure is used to size anything.** + +**[FACT] The class-B consequence is real regardless of which count is right.** For an app whose data +lives entirely in named volumes, the Tier-2 copy holds a full `recovery-unit/` and **no readable +leg** — LIVE on demo-felhom, `backups/secondary/bookstack/` and `.../docmost/` contain +`recovery-unit` and nothing else, at 156 MB and 86 MB (INV Part B.2). Since v0.183.0 the button +refuses **before** stopping the app and names the action that works. + +### 6.3 Where capture and restore disagree + +**[FACT]** Three asymmetries, each source-cited: + +| tier | captured | read back by that tier's restore | gap | +|---|---|---|---| +| Tier-1 | unit incl. volume tars + DB dumps | all of it | none | +| Tier-2 | unit mirror **+** file legs | `hdd/` and `userdata/` **only** (`tier2_restore.go:101-104`) | **the unit mirror is read by nothing** — `RecoveryUnitPath` resolves to `backups/primary/` (`appbackup/paths.go:46-48`) → **R-102** | +| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay; the unit is **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **no offsite action unpacks the named-volume tars it captures** → **R-107** | + +**[FACT]** Tier-2's gap is the sharper one because of *when* it bites: Tier-2 exists for the case +where the primary drive is lost — and in exactly that case the primary unit is gone while this +mirror survives on the second drive, unreachable by any customer action. --- -## 6. Offsite snapshot shape +## 7. The recovery chain (D3) — the reason this document exists -**Decision: one multi-path snapshot per app per run.** -`restic backup … --tag felhom-offbox --tag `. +**[DESIGN] 3-2-1 describes copies. It does not describe recovery.** -Why this shape over two-snapshots-per-app (unit vs userdata split): +Three copies on two media with one offsite is a statement about *bytes surviving*. It says nothing +about whether the bytes can be turned back into a working system, and a system can satisfy 3-2-1 +completely while having no executable recovery route for a given failure. That is not a hypothetical +here — §8 has rows where it is the actual state. -- Preserves the invariant every consumer relies on today: *latest `--tag ` = one consistent - app-level point* (`offbox.go:732` restore, SnapshotCount, the hub report). -- restic chunk-dedup makes unchanged media across daily runs cheap; the incremental cost of a run - is the delta, not the library. -- The snapshot's own `paths` metadata (visible in `snapshots --json`) records exactly what was - captured — the restore side introspects shape from there, so **no recovery-unit manifest change - and no SchemaVersion bump** (spike rec #6 considered; answered: the unit's content and shape are - unchanged, classes already ride in the captured `.felhom.yml`, and snapshot-paths metadata is a - better shape record than a manifest field that can drift from it). +### 7.1 The dependency graph -**Consequences that must be proven before the 3a spec (spike-first gate — these are unvalidated -restic mechanisms):** - -- **SP-1 — quota stats semantics.** Measure `restic stats` default (restore-size, all snapshots) - vs `--mode raw-data` on a repo holding N retained snapshots of a large path. Expected: raw-data ≈ - actual Storage Box disk; restore-size multiplies. Decides §9's accounting switch. -- **SP-2 — retention grouping across a shape change.** Default `forget` grouping is host+**paths**; - the first enlarged push changes the path set → old-shape snapshots strand in their own group and - its keep-daily/weekly/monthly floor retains them **forever** (no new snapshots ever age them - out) — zombie quota cost. Candidate fix: `--group-by host,tags` (the `` tag groups old and - new shape together so old shape ages out naturally). Prove: seed old-shape snapshots, push - new-shape, run forget with the flag, verify old-shape aging. -- **SP-3 — multi-path restore shape.** Verify the `--target` tree layout for a multi-path snapshot - (absolute source paths reconstructed under target), and that `--include ` gives a - unit-only selective restore. Feeds §7. - -All three fit one nested-VM spike session against a scratch restic repo (no Hetzner needed — -a local SFTP or even a local repo reproduces the semantics). - ---- - -## 7. Offsite restore path (3a first-class scope) - -Today's restore is dump-to-scratch-and-inspect. With userdata in snapshots it needs three changes: - -1. **Scratch relocation + headroom gate (fixes F-A1).** The scratch target moves from - `DataDir/offbox-restore` (rootfs) to a data-drive location - (`/backups/offsite-restore/` on the app's drive, or the tier-2 target-pick logic - reused to find room). Before restoring, read the snapshot's logical size (restic - `stats ` / `snapshots --json`) and refuse with an honest Hungarian reason if the target - lacks headroom — the `SizeUnknown`-never-renders-as-fits guard philosophy applies. -2. **Selective restore.** Default operator action restores the **unit only** (`--include` the unit - path — SP-3 proves the flag shape); "full restore" (unit + userdata) is an explicit second - action showing the size first. Keeps the daily case cheap, makes the big case deliberate. -3. **Restore-to-live = staged missing-only merge.** For the SQ3 acceptance ("immich restorable - end-to-end from offsite alone"): restore to scratch → place mandatory paths into their live - locations with the `tier2_restore.go` merge pattern (`rsync -a --ignore-existing`, - never-delete) → then the existing unit restore (`RestoreApp`) brings the app up. Data before - app start — the DB references the paths, so the paths must exist when the app first scans. - Full-overwrite restore stays out of scope (PBS + DR tier own catastrophic recovery). - ---- - -## 8. Tier-2 destination layout rework + migration - -**New layout (v2), relpath-mirroring:** +**[FACT]** Every app-tier restore depends on the guest, for one of two reasons: ``` -backups/secondary// - .felhom-tier2-layout # marker: "2" - recovery-unit/ # unchanged leg - hdd// # e.g. hdd/appdata/paperless/media/ - userdata// # e.g. userdata/media/books/ + ┌──────────────────────────────────────────┐ + │ the LIVE GUEST │ + │ · settings.json (tier-2 destination) │ + │ · encryption.key (32 B) │ + │ · app.yaml (ENC: under that key) │ + │ · the deployed app itself │ + └───────────┬──────────────────────────────┘ + │ required by + ┌──────────────────────────┼──────────────────────────┬─────────────────────┐ + │ │ │ │ + Tier-1 restore Tier-2 restore Tier-3 reconstitute Tier-3 scratch + (rebuild the app) (fill in files) (overwrite + replay) (verification copy) + │ │ │ │ + needs SECRETS needs the app needs the app needs only the + from app.yaml running + the deployed + a DB repo password + (restore_unit.go recorded dest service identifiable (also in the guest) + :130-132) (tier2_restore.go + :114-116) + │ │ │ │ + └──────────────────────────┴──────────────────────────┴─────────────────────┘ + │ + ┌───────────▼──────────────────────────────┐ + │ WHOLE-GUEST TIER (local vzdump / PBS) │ ← Lane 2, operator only + └───────────┬──────────────────────────────┘ + │ required by + ┌───────────▼──────────────────────────────┐ + │ THE HOST (agent.json, bootstrap.json, │ ← in NO backup at all + │ vmbr9, sudoers, /etc/pve/priv, WG) │ (INV Part D1) + └──────────────────────────────────────────┘ ``` -- Relpath mirroring represents N>1 appdata dirs and nested binds natively (the v0.131.0 refusal is - lifted structurally, not by special-casing) and makes restore position-derivable: dest relpath → - live path under the app's current HDD_PATH — hddPath-invariant like the classifier itself. -- **Legacy apps use the same layout**: the resolver's appdata dir(s) map to - `hdd/appdata//`. Capture SET stays byte-identical (the SQ5 promise governs footprint and - cost, not the internal arrangement of a derived copy); one layout means one restore reader. -- **Migration = rebuild, not preserve.** Tier-2 copies are fully derived from live data. On the - first v2 run per app: marker absent → delete the old flat `appdata/` (and the old - `recovery-unit/` stays — it's layout-identical) → mirror fresh into v2 → write the marker last - ("floor field LAST" family: the marker asserts a complete layout, so it is written only after all - legs succeed). -- **Reconcile step (answers the v0.131.0 deferred pruning question):** after mirroring, dest - subdirs under `hdd/`/`userdata/` not present in the current capture set are removed — a bind - removed from the compose (or re-classed excluded) stops occupying the secondary drive within one - run. -- **Restore reads v2 only.** Marker absent → honest refusal: "futtass előbb egy másodlagos - mentést" — acceptable because tier-2 restore is missing-files recovery, and the source of truth - (live data) still exists in that scenario; pre-migration copies stay on disk untouched until the - first v2 run replaces them. -- Headroom math: `unitSize` becomes unit + Σ(capture-set path sizes) via the existing `du -sb` - loop; the SSD fallback computes it over the reduced (unit+mandatory) set per §2.2. +**[FACT] The two legs of the dependency, precisely:** + +1. **Secrets.** The recovery unit is secret-free by design — *"It NEVER writes a secret value"* + (`internal/backup/recovery_unit.go:73`). `RestoreFromRecoveryUnit` recovers secrets **from the + guest, never from the unit** (`restore_unit.go:130-132`), and the policy is stated outright at + `:18-22`: *"Regenerate NOTHING… A missing DATA-ENCRYPTING key is FATAL: regenerating it would + render the restored data unreadable, so we refuse and tell the operator to do a PBS whole-guest + restore."* Those secrets are encrypted under `encryption.key`, a 32-byte file that exists **only + inside the guest** (LIVE, both boxes — INV Part C, row 9). +2. **The living app.** Tier-2's restore reads the destination recorded in the guest's `settings.json` + (`tier2_restore.go:114-116`), and Tier-3's reconstitution refuses outright when the app is not + deployed — *„a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat"* + (`offbox_reconstitute.go:198-201`). + +**[FACT] So all three app tiers are conditioned on the whole-guest tier**, which is Lane 2. The +customer's own recovery lane rests on an operator-only tier, and today nothing tells the customer +that. + +### 7.2 A tier whose prerequisites cannot be met in the failure it exists for + +**[FACT]** Two instances, both current: + +- **Tier-2 vs primary-drive loss.** Tier-2's stated purpose is surviving the loss of the primary + drive. In that failure the primary recovery unit is gone; the surviving mirror on the second drive + is `backups/secondary//recovery-unit/`, which **no code path reads** (§6.3). For the 45-or-43 + class-B apps the restore is a guaranteed no-op in exactly its designed scenario. → **R-102** +- **Tier-3 vs guest loss.** Tier-3 holds the volume tars and the DB dump. Reconstitution requires + the app to be deployed and skips the unit; the tars are unpacked only by the Tier-1 path, which + requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest. + → **R-107** + +### 7.3 What D5 would change — and why it is blocked + +**[DESIGN, TARGET — BLOCKED]** The intended fix is to make app secrets travel with the **local** +recovery unit, so Tier-1 and Tier-2 restore work **without the guest and without R**. Offsite +already encrypts everything, so secrets travelling offsite would be covered by the escrowed repo +password. R would then be required for **offsite recovery and host identity only** — losing R would +cost the offsite route, not local recovery. + +**This is not adopted.** §2 of the task that produced this document required the premise to be +established, not assumed: the backup tree must be unreachable from every browsing, download and +export surface. **It is not.** The verification and the exposure are in §10.1. Until that is closed +(**R-108**), D5 stays a target and §7.1's chain stands as the model. --- -## 9. Quota & cost model +## 8. The failure → recovery matrix -- **Accounting switch (pending SP-1): quota compares `restic stats --mode raw-data` — actual - deduped+compressed repo bytes, i.e. what the customer's Storage Box really fills** — instead of - today's all-snapshots restore-size. Without this, ~17 retained snapshots of a mandatory library - would multiply the measured usage and permanently trip the ≥100% gate. The existing - stale-but-safe last-known-value behavior stays. -- **Pre-push gate (ruling #2 mechanized).** Before an app's **first enlarged push** (and whenever - its mandatory-set size estimate grows materially), compute the mandatory-set size with the - existing per-mount `du` loop (`estimate.go` reuse — zero new measurement plumbing, SQ4). If - last-known repo size + estimate crosses the quota: **the enlarged push is blocked and the - customer is notified with the two numbers** — but the **unit-only push continues** (blocking it - too would *regress* existing protection; the gate governs the enlargement, not the tier). - The notification is the "size surfacing before the first enlarged push" from the ruling. -- The existing ≥80% warn / ≥100% refuse-new-backups run gate stays as the backstop, re-based on - raw-data numbers. -- **Tier-2 cost:** headroom per §8; the two-number estimate (state+coupled vs +bulk) reuses the - classified `du` split for the `.fab` warning (Task 4) and the tier-2 config panel display. -- Hetzner reality check: BX11 = 1 TB shared-model. Mandatory ≈ DB-coupled stores (immich uploads, - nextcloud data, paperless documents) — bounded per SQ4's model, but immich uploads are the one - class member that grows like a media library. The pre-push gate + notification is what makes - that growth a conversation instead of a surprise. +**This is the core artifact.** One row per failure. It is authoritative for recovery routes; +`00-capability-map.md` stays authoritative for per-capability status. -### 9.1 PBS whole-guest tier — measured capacity state (R-82 Phase 0, 2026-07-26) +**How to read the numbers.** -> ### 🟢 SUPERSEDED 2026-07-27 — the datastore was grown; the P0.3 STOP is CLEARED -> -> The "grown later" in the paragraph below **has happened.** `felhom-offsite` was relocated off -> ep0's 40 GB root disk onto a dedicated **100 GB Hetzner Cloud Volume** -> (`scsi-0HC_Volume_106469259` → `/mnt/pbs-datastore`, ext4 `-m 0`). Evidence and method: -> `runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`. -> -> | Fact | Before (2026-07-26) | After (2026-07-27) | -> |---|---|---| -> | Datastore total | 37.2 GB | **98 GB** | -> | Used | 10.8 GB (**28.9 %**) | 13 GB (**13 %**) | -> | 80 % warn threshold | 29.8 GB | **78.4 GB** | -> | Headroom to warn | 19.0 GB | **≈ 65 GB** | -> | Additional customers before warn (at ≈5–10 GB each) | **≈ 2** | **≈ 7–13** | -> -> **What this does NOT change:** point 1 below still holds in full — per-tenant encryption still -> precludes cross-customer dedup, so cost still scales linearly per customer. The volume buys -> runway, not a better cost model. Point 3 also still holds: `pvesm` still cannot see this -> datastore's fill. -> -> **Re-measure rather than trust this table** once a second weekly snapshot exists — the weekly -> incremental size is still **UNMEASURED**, and it, not the total, decides the real slope. -> -> ✅ **The hub's PBS-DR gauge tracked the move correctly** (verified 2026-07-27, Offsite → PBS DR): -> `felhom-offsite (ep0)` — **capacity 97.9 GB, used 12.6 GB, 13 % full**. It follows the datastore's -> configured path, so the relocation needed no hub-side change. The feared "gauge reads the wrong -> filesystem" bug **does not exist**. +- **RTO** — **only measured durations** from INV Part F appear here. A blank cell means *nobody has + ever measured it*, and a blank is a finding, not an omission. +- **RPO** — **no RPO has ever been measured from an incident.** These cells carry the **configured + cadence that bounds RPO**, read live from the box, labelled `(cadence)`. A blank means no cadence + governs the row. +- **Status** — `PROVEN` (live, cited) · `PARTIAL` (some legs proven) · `IMPLEMENTED` (code + tests, + never exercised) · `NONE` (no route exists). -Recorded per the operator's 2026-07-26 ruling: **the datastore will be grown later; R-82 proceeds -meanwhile.** These are measurements, not projections-of-record — re-measure before relying on them. -Full method + evidence: `audits/SPIKE-r82-phase0-2026-07-26.md`. +| # | Failure | What survives | Recovery route | Invocable by | RTO (measured) | RPO (cadence) | Status | Evidence | +|---|---|---|---|---|---|---|---|---| +| 1 | **Customer deletes files** | everything else | „Fájlok visszaállítása" — Tier-2 additive merge | **customer** | **39 s** (6 files) | 24 h | **PROVEN** | CAMPAIGN-9 A1 — byte-identical returns, both non-destruction promises kept, bytes re-served by paperless's own API | +| 2 | **An app's data directory is destroyed** | the guest, the other tiers | same route | **customer** | **46 s** (43 files) | 24 h | **PROVEN** | CAMPAIGN-9 A3 — loss verified by a positive observable (doc download 200 → 404), then 16/16 documents usable | +| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) | 24 h | **PARTIAL** | CAMPAIGN-9 A2 proves the path executes and the app survives; content recovery after real loss **never demonstrated** (`CAMPAIGN-9…:825-827`) | +| 3b | *same, for a class-B app via Tier-2* | — | **no route** — Tier-2 never reads the unit mirror | — | | | **NONE** | §6.3; **R-102** | +| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB; **the volume tars in either copy are unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102**, **R-107** | +| 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go:359-393` | +| 6 | **Guest lost or corrupted** | the host, both whole-guest tiers, the data drives (they are host binds) | `pct restore` from `local:` or `felhom-pbs:` | **operator** (SSH) | **84–112 s** local · **1101 s** PBS (both = restore-test into a scratch guest, boot + verify + teardown) | 24 h local · 7 d offsite | **PROVEN** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C (exact mount parity, `unprivileged: 1` preserved); LIVE restore-tests on both boxes this session | +| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human | +| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) | +| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) | +| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) | +| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) | +| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) | +| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D | +| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B | +| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix | +| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** | -| Fact | Value | Source | +### 8.1 The blank cells, listed explicitly + +Per the rule that a blank is a finding, here they are: + +| row | blank | why | |---|---|---| -| `felhom-offsite` datastore total (ep0) | **37.2 GB** | ep0 `df` via the hub `usage` op (`scripts/felhom-tenantsync.sh` v1.2.0) | -| Used | **10.8 GB (28.9%)** | same, 2026-07-26 12:02 UTC | -| Hub alert thresholds | warn **80%** (29.8 GB) / crit **90%** | `monitor.PBSDRBoxChecker` | -| Snapshots present | **1** — demo-felhom `ct/9201/2026-07-18T18:31:06Z`, 9.74 GB logical, `verify ok`. demo-hp namespace exists with **zero** | PBS API | -| First-snapshot compression | ≈ **1:1** (9.74 GB logical ≈ 10.8 GB on disk) | source is already-compressed Docker layers | -| Weekly incremental size | **UNMEASURED** — no guest has ever had a second PBS snapshot | measure at the first Slice-D weekly cycle | +| 3b | RTO, RPO | no route exists to time | +| 4 | RTO | no drive-loss recovery has ever been timed | +| 5 | RTO | never timed; the rebuild is a normal Tier-2 run | +| 6 | — | RTO present, but it is a **restore into a scratch guest on the same host**; a restore *to a different host* has never been timed | +| 8 | RTO | a host has never been rebuilt as itself (INV Part D1) | +| 9 | RTO | the composed whole-box path has never been run | +| 10 | RTO | no ransomware-shaped recovery has ever been run | +| 11 | RTO | a hub restore has never been performed | +| 12 | RTO, RPO | no provider-loss recovery has ever been run | +| 13 | RTO, RPO | not a timed recovery; a capability loss | +| 14 | RTO | the break-glass path is proven but was never timed | +| 15 | RTO | the manual unlock was performed but not timed | -**Three facts that govern the cost model here — all differ from the restic tier above:** - -1. **No cross-customer dedup.** Backups are client-side encrypted with a per-customer key - (`encryption-key` in `storage.cfg`), so PBS derives chunk digests under that key and chunks - never dedup between tenants. Every customer's snapshots cost their full independent size. There - is no fleet-scale dividend — the opposite of the shared-model assumption in §9 above. -2. **The datastore is small relative to the fleet.** At weekly `keep-last` retention the current - three boxes project to ≈15–21 GB (40–57%); each additional customer costs ≈5–10 GB, so the 80% - warn is reached at roughly the **second additional customer**. This is the open constraint the - operator deferred, not a solved problem. -3. **PVE cannot see this datastore's fill.** `pvesm status felhom-pbs` reports **0/0/0 KiB while - active** — PBS answers the status call with HTTP 200 and zeroed usage because the per-customer - token holds `DatastoreBackup` on its namespace, not `Datastore.Audit` on the datastore root. - **Operators must read PBS fill from the hub's PBS-DR gauge, never from `pvesm`/the PVE UI.** - Cosmetic, not a fault: writes work (the 07-18 snapshot is owned by that exact token). - -**What the PBS tier does and does not carry** (confirmed live against `pct config 9201` and the -agent's own `uncovered_volumes`): rootfs + `/var/lib/docker` (mp0, `backup=1`) + `/mnt/sys_drive` -(mp1, `backup=1`) are **in**; `/mnt/felhom-drives` (mp8) and `/etc/felhom-bootstrap` (mp9) are bind -mounts and are **out**. The §2 tier table's "bind-mounted drives out of reach" is correct. - -**Why weekly is sufficient (R-82 P0.1 verdict).** The only state exposed by a 7-day-old snapshot is -the non-SMB half of `settings.json` — `storage_paths`, `app_backup` toggles, notification prefs, -`password_hash`, `launcher_share_token` — all recoverable (drives are physical and re-enrollable; -the password has a hub claim-reset path). Critically, `encryption.key` and the offbox credentials -are **stable files unchanged since first boot**, so a week-old copy is byte-identical. Everything -referentially coupled to app state rides the DAILY tiers. - -> ⚠️ **The verdict is conditional on Tier-3 offsite being enabled and healthy.** On a box without it -> (drill-r50 reports `offsite: null`), PBS-weekly is the ONLY DR tier and the 7-day window would -> then cover app data and definitions too. Such a box needs offsite enabled first, or a shorter PBS -> cadence. +**Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB +from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS +incremental; `.fab` import wall-clock; and time-to-first-byte for a customer restore over a home +uplink — **no customer has ever driven a restore**. --- -## 10. Decisions requiring Viktor confirmation (one-line vetoes) +## 9. What the model implies for the tiers (recorded, not new design) -The locked rulings are not re-litigated; these are **consequences the doc surfaced**: +**[DESIGN]** Three consequences follow from §3–§7 and are stated so they are not re-derived: -1. **Blocked-enlargement semantics (§9):** when the quota gate blocks an app's enlarged push, its - unit-only push CONTINUES (no protection regression). Confirm. <--- Confirmed -2. **`.fab` optional default = pre-selected** (spike rec #2 wording; ruling #1 said "checkboxes" - without a default). Confirm pre-selected. <-- Conrifmed -3. **Offsite restore default = unit-only; full (unit+userdata) is an explicit second action with - the size shown first (§7.2).** Confirm. <-- Conrifmed -4. **Undeployed apps stay legacy unit-only offsite + WARN (§2.4).** Confirm. <-- Conrifmed -5. **Tier-2 v2 layout migration = delete-and-rebuild of the derived secondary copy (§8),** old - copies replaced on the first v2 run per app. Confirm. <-- Conrifmed -6. **Quota accounting switches to raw-data bytes (§9), pending SP-1.** Confirm direction. <-- Conrifmed -7. **SSD fallback becomes explicitly state-only (unit + mandatory-if-fits, never optional) - (§2.2).** Confirm. <-- Conrifmed +1. **Tier-2 is a *drive-loss* tier, not a second chance at Tier-1.** Its job is to survive one drive + dying. That is why it mirrors the unit at all — and why the unit mirror being unreadable (§6.3) + defeats the tier rather than degrading it. +2. **Tier-3 is a *premises-loss* tier.** It is the only copy that survives fire, theft and + ransomware-with-host-access, which is why it is the only tier whose credential exposure is ranked + as the largest open data risk (R-95). +3. **The whole-guest tiers are the *system* tier, and they sit under everything else** (§7.1). + Treating them as "one more copy" is precisely the 3-2-1 framing this document rejects. -## 11. Explicitly out of scope (noted, not decided here) +--- -- Cluster-mode / HA implications (Peti's two-node roadmap item): tier-2 target selection and - drive discovery assume a single host's drive set; agent-follows-guest will revisit. -- `.fab` exclusion-scoping implementation details (Task 4 owns them; the SQ5 verdict — root tar - minus excluded subtrees, manifest v1 unchanged — stands and is assumed by §2's `.fab` row). -- NAS as a tier-2 target via a tar-based leg (§5 revisit condition). -- Hub-side surfaces: nothing here forces a hub field; if the 3a report shape adds one (e.g. - enlarged-push-blocked state), the spec flags it explicitly. +## 10. Known gaps -## 12. Observations (documented, not acted on) +Every divergence between the model above and the system as it is, each with an ID. -- `tier2.go:167` RunTier2 doc comment still says "recovery unit + userdata" (stale F-S1 family) — - 3b corrects it as a side effect of making it true. -- `offboxRecordStats` reuses one probe-timeout context (`sctx`) for both `snapshots` and `stats` - (`offbox.go:686-702`) — with a large repo the second call inherits whatever budget remains; worth - a per-call timeout when 3a touches the file. Not a bug today. -- The offsite restore handler backgrounds a 30-minute context (`offbox_handlers.go:230-233` per the - F4 fix); enlarged restores may need a bigger ceiling — spec-time decision in 3a, sized by SP-3 - timing evidence. +### 10.1 D5 is BLOCKED — the backup tree is reachable from a browsing surface -## Appendix — verified symbols Task 3 builds on +**[FACT] The verification and its result.** Every surface that can read a file was checked: -| Symbol | File (landmark) | Note | +| surface | can it reach `backups/`? | evidence | |---|---|---| -| `Manager.ClassifiedBinds` | `stacks/metadata.go:285` | the seam; validated via `LoadMetadata` | -| `appbackup.ClassifyBinds` / `ValidateBackupSpec` | `appbackup/classify.go:165/:97` | two-level default; whole-block-reject | -| `appbackup.AppDataDirNames` | `appbackup/paths.go:90` | legacy resolver (stays for no-block apps) | -| `StackDataProvider.GetStackClassifiedBinds` | adapter `main.go:~1106` | wired end-to-end (Task 2) | -| `Manager.RunTier2` / `selectTier2Target` / `rsyncMirror` | `tier2.go:170/:92/:438` | engine to rework; `-a --delete` | -| `settings.StoragePath.IsNetwork` | `settings/settings.go:219` | the F-6C-1 filter, metadata-only | -| `Manager.RunOffboxBackup` / `runOffboxInternal` / `discoverOffboxUnit` | `offbox.go:376/:559/:508` | push engine; unit discovery | -| `Manager.RestoreOffbox` | `offbox.go:722` | restore-to-scratch; §7 rework | -| `offboxQuotaState` / `offboxRecordStats` | `offbox.go:632/:685` | quota gate; stats-mode switch | -| `restoreTier2Files` merge pattern | `tier2_restore.go:127-137` | `--ignore-existing` never-delete merge | -| `Exporter.EstimateExport` / `duBytes` | `appexport/estimate.go:33/:126` | the two-number split reuse | -| `RecoveryManifest` (SchemaVersion 1) | `recovery_unit.go:31,131` | unchanged by this design | +| SMB share creation | **NO** | `sharingResolvePath` (`internal/web/sharing_handlers.go:52-81`) resolves symlinks *before* containment, then refuses any path within `SharingDeniedRoots(root)`; that set covers `/backups` **and** the legacy `/felhom-data` + `/felhom-data/backups` (`internal/stacks/samba.go:46-65`, derived from `ProtectedHDDPaths`, `delete.go:59-78`) — **both namespace shapes** | +| SMB browse (folder picker) | **NO** | same deny set applied per child (`sharing_handlers.go:521-537`) | +| SMB `ensureImportShare` (the store-direct bypass) | **NO** | writes one controller-generated constant, `GetImportRoot()` = `/userdata/import` (`sharing_handlers.go:564-587`) | +| FileBrowser — **local drives** | **NO** | the bind is `appbackup.UserdataDir(sp.Path)` only, and the comment says why (`internal/web/handlers.go:2450-2460`) | +| **FileBrowser — network shares** | **YES** | the bind is the share **ROOT**: `- %s:/srv/%s:rslave` (`handlers.go:2432`), and the path joins the config source list (`:2433`) | +| `.fab` import path validation | **NO** | confined to `/exports` (`handler_export.go:400-408`, `estimate.go:215-217`) | +| `.fab` browser download | **NO** | name-pattern + parent-must-be-the-staging-dir double guard (`handler_export_download.go:36-45,120-140`) | +| `/api/debug/*` | **NO** | no file-serving branch (`handler_debug.go:46-92`) | +| `http.ServeFile` (3 sites) | **NO** | assets only, `filepath.Base`-normalised (`server.go:720-771`) | +| log bundles | **NO producer found** | no filesystem-walk bundle producer exists on the box; searched `felhom-controller/internal`, `felhom-agent/internal`, `cmd/` | +| registering the backup dir as a drive | **NO** | the manual add requires `system.IsMountPoint(path)` (`handlers.go:2091-2095`); `/backups` is not a mount point | -**Do NOT reuse** `rsyncMirror` for any restore leg (`--delete` — the named trap); restore merges -use the `--ignore-existing` pattern only. \ No newline at end of file +**The exposure, end to end.** All six links are source-cited and the precondition is live today: + +1. A NAS share is registerable as a storage path and lands `Schedulable: true` — **LIVE on demo-hp**: + `{"path":"/mnt/felhom-drives/Felhom-Share","kind":"network","schedulable":true}`. +2. `GetSchedulableStoragePaths()` has **no `IsNetwork()` filter** (`internal/settings/settings.go:904-914`), + so that share appears in the **deploy** dropdown (`handlers.go:462-473`). +3. The **per-app migrate** target list filters only current / decommissioned / disconnected / + schedulable — also no network filter (`handlers.go:674-679`). +4. `handleStorageMigrateApp` does **not** call `refuseNetworkLifecycle`, unlike its whole-namespace + sibling which does (`storage_handlers.go:397` vs `:410-424`), and `startMigration` has no guard + either (`internal/stacks/migrate.go:214-255`). +5. With `HDD_PATH` on the share, `namespaceRoot()` returns it as-is under Model A + (`internal/backup/backup.go:262-263`), so the app's Tier-1 unit is written to + `/backups/primary//compose/app.yaml`. +6. FileBrowser binds that share at its **root** and serves it with `download: true` + (`internal/infra/infra.go:326`). + +**Observed live on demo-hp, in the generated compose — the asymmetry is visible, not inferred:** + +``` +- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb <- drive: userdata-scoped +- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave <- network: ROOT-bound +- /mnt/sys_drive/felhom-data/userdata/import:/srv/beolvasas +``` + +**Today this is not a secret leak**, because the unit's `app.yaml` is secret-stripped +(`recovery_unit.go:73`). **D5 would make it one.** That is exactly the test §2 set, and D5 therefore +does **not** hold as written. → **R-108** + +### 10.2 The gap register + +| ID | Gap | Consequence | +|---|---|---| +| **R-102** | Tier-2 writes a full `recovery-unit/` mirror on every run and **no code path reads it** | Tier-2 is defeated in the drive-loss scenario it exists for (§7.2). Was C9-F4 | +| **R-103** | The Tier-2 no-coverage refusal **names** the working action but does not **route** to it | 45-or-43 of 53 apps dead-end at a message. Was C9-F1b | +| **R-104** | An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach, reported as *„ismeretlen okból"* | the offsite tier stays dead until a human unlocks. Was C9-F3 | +| **R-105** | Three hub-held DR records are empty on the whole live fleet: `hosts.dr_record_json`, `host_escrow.directive_json`, `dr_recipe.host_half.drives` | the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field | +| **R-106** | `dr_recipe.host_half.pbs.namespace` records `"root"` on every box | the recorded restore coordinate is wrong; real namespaces are per-customer | +| **R-107** | No offsite action unpacks the named-volume tars Tier-3 captures on every run | offsite alone cannot rebuild a named-volume app (§7.2) | +| **R-108** | Network storage can host an app's namespace, and FileBrowser binds a network share at its **root** | **blocks D5** (§10.1); today it also lets a `.fab` with plaintext secrets be exported to a NAS (§5) | +| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) | +| R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written | +| R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof | + +### 10.3 Divergences that are documented elsewhere and are not re-opened here + +**[FACT]** `06-offsite-connectivity.md:19-21` describes the operator's public edge as a Cloudflare +Tunnel and states DooPlex has no public IP. Live DNS resolves `hub.felhom.eu` through a no-ip +DynDNS CNAME straight to DooPlex's own public address (INV Part E.3). That is a topology-document +issue, not a recovery-model one; recorded so the discrepancy is not lost. + +--- + +## 11. Open decisions — for the operator + +**Recorded, deliberately not answered.** + +**A. Escrow custody.** Split custody (R **or** an offline operator key) versus a 2-of-3 threshold +across customer / hub / box drives. +*Recommendation on record:* **split custody, operator key held offline and never in the hub.* Note +the constraint from §2: the moment the operator key lives in the hub, "a compromised hub yields +blobs nobody can open" stops being true. + +**B. Lost-R policy.** Under split custody the operator **can** recover. Is that the stated policy — +and if so, what identity check gates it? Today matrix row 13 has no route at all, and the customer +is not told that losing R costs them the host-loss route. + +**C. RTO / RPO targets per scenario.** **None have ever been stated.** Without them §8 cannot judge +whether, for example, offsite-only recovery is acceptable for drive loss, or whether the 7-day PBS +cadence is adequate for row 8. The measured column is the input; the target column does not exist. + +**D. Hetzner as a single failure domain.** restic (Storage Box `u629488`, sub-accounts per customer) +and PBS (Cloud server ep0 + a Cloud Volume) are both Hetzner. Whether they share an **account, +login and payment method** is unverified (INV Unknown U-1) — the question was deliberately not +answered by calling the provider API with the production token. Accept explicitly, or mitigate. + +**E. `local` vzdump shares a physical device with the guest it backs up.** LIVE on both hosts: +`/var/lib/vz` (the archive target) and `local-lvm` (the guest's rootfs and both `backup=1` +mountpoints) are both on `/dev/sda3` → VG `pve`. The tier therefore protects against **corruption +and operator error only**, never against disk failure. Accept and name it honestly in the customer- +facing description, or move the target. + +**F (added by §10.1, not in the original list).** D5 cannot be adopted until R-108 closes. Is +closing R-108 the intended path, or is D5 withdrawn? + +--- + +## 12. Evidence index + +| Claim | Grade | Source | +|---|---|---| +| Tier-2 file restore, gap-fill and after total loss | **PROVEN-LIVE** | CAMPAIGN-9 A1/A3 | +| Tier-2 refuses without an outage for a no-coverage app | **PROVEN-LIVE** | v0.183.0 replay, `felhom.eu/REPORT.md:60` | +| Tier-1 unit restore **executes** | **PROVEN-LIVE** | CAMPAIGN-9 A2 | +| Tier-1 **content recovery after loss** | **UNPROVEN** | `CAMPAIGN-9…:825-827` | +| restic restore of app data (bytes) | **PROVEN-LIVE** | CAMPAIGN-8 R-87, 155.5 MB, 6/7 byte-identical | +| offsite reconstitution of a DB-indexed app | **PROVEN-LIVE** | destructive immich drill 2026-07-20, `00-capability-map.md:66,75` | +| offsite **place-to-live** as a distinct action | **UNPROVEN** | `CAMPAIGN-8…:520` | +| shares restore (files + definitions + credential) | **PROVEN-LIVE** | 2026-07-18, `00-capability-map.md:96` | +| `.fab` drive-to-drive round trip | **PROVEN-LIVE** | CAMPAIGN-6D P-FAB, 1.7 GB | +| `.fab` **browser upload** leg | **UNPROVEN** | `00-capability-map.md:67` | +| whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C | +| corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 | +| the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 | +| unattended restore-test across tiers | **IMPLEMENTED** (rotation not observed across consecutive cadences) | `00-capability-map.md:41`; LIVE per-tier timestamps this session | +| guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s | +| quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL | +| break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` | +| agent DR bring-up (`ModeDRGuestLoss`) | **NEVER EXECUTED** | `CAMPAIGN-8…:522` | +| host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` | +| host rebuilt as its former self | **NEVER DONE** | INV Part D1 | +| escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` | +| hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 | +| a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` | + +--- + +## 13. What this document deliberately does not do + +- It does not restate capability status — §8 cites the map, the map cites §8. +- It does not resolve the 7/53-vs-9/43/1 count (§6.2). Both are on the record; neither is adopted. +- It does not estimate a single RTO or RPO. Every blank in §8 is a real gap. +- It does not answer §11. Those are the operator's. +- It does not claim ratification. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 68b3485..ab80940 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -30,9 +30,10 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-101** | ~~Tier-2 `LastRun` is written on failure and rendered to the customer as „Legutóbbi másolat" — including in the restore confirm dialog~~ | **SHIPPED + PROVEN-LIVE** (controller v0.182.0, 2026-07-28) | — | `CrossDriveBackup.LastSuccess` + `SuccessTracked`; the dialog names the last **successful** copy and discloses a failed newest attempt. Legacy rows migrate truthfully on first touch (an `ok` row adopts its time; an `error` row seeds nothing) — without the marker all 7 fleet rows would have flipped to „Még nincs sikeres másolat" on deploy. **Part 2:** the three `record*` sites rebuilt the whole struct; replaced by `tier2Update` (copy-and-overlay, safe by construction) — the naive fix would have had `recordTier2Failure` CLEAR the anchor. Live on demo-hp, rendered dialog read in both states | — | | **C9-F1** | ~~Tier-2 „Fájlok visszaállítása" is offered for apps whose copy has no restorable file leg; stops the app, restores 0 files, reports „Nincs hiányzó fájl — minden fájl megvan a helyén."~~ | **SHIPPED + PROVEN-LIVE** (controller v0.183.0, 2026-07-28) | — | **Phase 0 sized it: 43 of 53 catalog apps read NOTHING, 9 read file legs but never their DB/volumes, 1 stateless.** Honesty half shipped: `Tier2RestoreCoverage` refuses UP FRONT without stopping the app and NAMES the working action; a run that proceeds claims only what it **examined** and discloses that the database and volumes are not covered. Live on demo-felhom: bookstack refused, uptime stayed „Up About an hour" (was „Up 25 seconds"); paperless A1 re-run still byte-identical, 16/16 docs clean | — | | **C9-F2** | ~~An app in a Docker crash loop never alarms on any channel; `StateRestarting` is in no down-set~~ | **SHIPPED + PROVEN-LIVE** (controller v0.183.0, 2026-07-28) | — | `StateRestarting` deliberately NOT added to `IsDownState` (that alarms on every deploy fleet-wide); a SUSTAINED run becomes down after `crashLoopAfter`=5m, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace. Dashboard counter uses the same predicate so it no longer contradicts the alarm. Red-proof that matters: the naive `IsDownState` change fails the brief-restart test | — | -| **C9-F3** | An **interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach**: `resticStep` (`offbox.go:634-648`) has `unlock --remove-all`, but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`offbox.go:77-93`) has no lock case → `"other"` → fail-fast. Tier dead until a human unlocks; `ClassifyOffsiteFailure` likewise has no lock case so the operator is told **„A távoli mentés ismeretlen okból nem sikerült"** for a precisely-known, self-healable condition | **READY (MEDIUM)** | — | Add a lock case to both classifiers and let the probe path escalate to `unlock --remove-all`. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (`ok`, 1m35s). Reachable by any interruption — container restart, OOM, **host reboot mid-backup** | CC | -| **C9-F1b** | Tier-2's restore cannot cover 43 of 53 apps; the action that CAN is the keep-side unit restore (`POST /backup/restore` → `RestoreFromRecoveryUnit`, replays volume tars + DB dumps). v0.183.0 NAMES it in the refusal text but does not route to it | **READY** | — | Put the working action in the card the customer already opened. **Deliberately its own task:** it places a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference — the reason it was not folded into v0.183.0 | CC | -| **C9-F4** | **Nothing reads the Tier-2 copy's `recovery-unit/` mirror.** It is written by EVERY Tier-2 run (`tier2.go:369`, „Unit leg (always)") and read by no code path: `RecoveryUnitPath` resolves to `backups/**primary**/` (`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`, which reads `hdd/`+`userdata/` only | **READY (potentially > C9-F1)** | — | Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action, leaving offsite as the only route. Verified by enumeration: 6 references to `"secondary"` in the tree, one writer, one reader, one wipe-warning lister | CC | +| **C9-F3** → **R-104** | An **interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach**: `resticStep` (`offbox.go:634-648`) has `unlock --remove-all`, but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`offbox.go:77-93`) has no lock case → `"other"` → fail-fast. Tier dead until a human unlocks; `ClassifyOffsiteFailure` likewise has no lock case so the operator is told **„A távoli mentés ismeretlen okból nem sikerült"** for a precisely-known, self-healable condition | **READY (MEDIUM)** | — | Add a lock case to both classifiers and let the probe path escalate to `unlock --remove-all`. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (`ok`, 1m35s). Reachable by any interruption — container restart, OOM, **host reboot mid-backup** | CC | +| **C9-F1b** → **R-103** | Tier-2's restore cannot cover 43 of 53 apps; the action that CAN is the keep-side unit restore (`POST /backup/restore` → `RestoreFromRecoveryUnit`, replays volume tars + DB dumps). v0.183.0 NAMES it in the refusal text but does not route to it | **READY** | — | Put the working action in the card the customer already opened. **Deliberately its own task:** it places a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference — the reason it was not folded into v0.183.0 | CC | +| **C9-F4** → **R-102** | **Nothing reads the Tier-2 copy's `recovery-unit/` mirror.** It is written by EVERY Tier-2 run (`tier2.go:369`, „Unit leg (always)") and read by no code path: `RecoveryUnitPath` resolves to `backups/**primary**/` (`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`, which reads `hdd/`+`userdata/` only | **READY (potentially > C9-F1)** | — | Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action, leaving offsite as the only route. Verified by enumeration: 6 references to `"secondary"` in the tree, one writer, one reader, one wipe-warning lister | CC | +| **R-108** | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT.** Local drives are `userdata`-scoped (`web/handlers.go:2450-2460`); network paths are bound at the share root (`:2432`) and served with `download: true`. No `IsNetwork()` filter guards the deploy dropdown (`settings.go:904-914`), the per-app migrate targets (`handlers.go:674-679`), or `handleStorageMigrateApp` (`storage_handlers.go:410-424` — its whole-namespace sibling DOES refuse, `:397`) | **READY — BLOCKS an architectural target** | — | **This is why D5 was not adopted** in the 2026-07-28 `07-backup-architecture.md` rewrite: D5 moves app secrets into the local recovery unit so Tier-1/Tier-2 restore stop needing the guest, and that is safe only if no browsing surface can reach the backup tree. Every other surface was verified clean (SMB both namespace shapes, FileBrowser for drives, `.fab` import + download, `/api/debug/*`, all three `ServeFile` sites, storage-path add) — `07` §10.1 has the full sweep. **Not a leak today** (the unit's `app.yaml` is secret-stripped). Verified LIVE in demo-hp's generated FileBrowser compose | CC | | **F-DIAG** | ~~Four distinct offsite failure causes collapse into two operator-visible strings~~ | **SHIPPED** (controller v0.182.0, 2026-07-28) | — | `ClassifyOffsiteFailure` → quota / orphaned / no_repo / no_units / transport / **unknown**, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. **Secrets:** the old message was a raw `err.Error()` passthrough carrying `sftp:@:`; redaction is now by the target's **actual** host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; **not** yet exercised by a live offsite failure of each class | — | | **F-OPS** | ~~A manual `pct restore` inherits the source guest's bind mounts — during a real DR, on a different host, under pressure~~ | **DOCUMENTED** (2026-07-28) | — | `documentation/runbooks/RUNBOOK-manual-guest-restore.md`: which `mpN` are volumes vs host binds, the `mp9` source-VMID trap (it can bind **another guest's bootstrap credentials**), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift | — | | **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 386222b..12aa472 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -86,6 +86,24 @@ | R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | **SHIPPED — (a)+(c); (b) rejected-by-design.** **(a) SHIPPED (controller v0.162.0, 2026-07-24): the apply-bridge settle-gate.** `offsiteapply.SettleProvider.SettleState()` + `SettleFunc` adapter over the updater's OWN `GetFloor()`/`IsUpdateRunning()` (no second floor path); `Bridge.AwaitSettle` defers the consume while `updateRunning` OR below the known floor (the auto-floor update is imminent), releasing at/above floor on the first poll with zero added latency (B′). Bounds 10 s poll / 90 s floor sub-bound / 5 min overall — both bounds GO+WARN (a hub that can't serve a floor can't serve a consume → no burn; R-71c is the belt). Ordering-only — the consume/persist order and the 404-no-oracle contract are UNTOUCHED. Finding: the floor is in-memory (report-ACK ~5–10 s), not persisted → unknown until the first ACK on any restart; the 90 s sub-bound is sized to that. 5 scenarios (A–E) + 4 red-proofs all observed FAIL then restored. Deferral paths NOT live-fired — their precondition is now structurally prevented by the v1.25.0 golden≥floor build gate (the point); live leg = the B′ first-poll GO line on both above-floor boxes. **(c) SHIPPED (hub v0.72.0, 2026-07-23): self-heal restage via the EXISTING Re-issue path (`monitor.OffsiteDeliveryChecker`), consumed ≥1h + ≥4 offbox-less reports + zero offbox evidence, one restage/customer/24h, `offsite_credential_restaged` warning; R-39(a) guard red-proofed.** **(b) two-phase consume REJECTED-BY-DESIGN** — it would touch the 404-no-oracle contract; (a)+build-gate+(c) cover the race without it. **Layering: v1.25.0 build gate PREVENTS, (a) DEFERS, (c) HEALS.** | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. | | R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `.` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) | +### Recovery-model gaps (2026-07-28, `07-backup-architecture.md` §10.2) + +> Minted when `07-backup-architecture.md` was rewritten as the recovery model. Every one of these is +> a divergence between that model and the system as it is, and each is cited there. **They are filed +> at P2 as the neutral default, not ranked** — ranking them needs the per-scenario RTO/RPO targets +> that `07` §11-C records as never having been stated. Flips: `00-capability-map.md` §C rows, which +> now cite the matrix rather than restating the route. + +| ID | Item | Size | Status | Notes / map rows flipped | +|----|------|------|--------|--------------------------| +| R-102 | **Tier-2 writes a full `recovery-unit/` mirror on every run and no code path reads it.** Written at `internal/backup/tier2.go:368-369` ("Unit leg (always)"); `RecoveryUnitPath` resolves to `backups/**primary**/` (`internal/appbackup/paths.go:46-48`) and the only reader of the secondary tree is `internal/backup/tier2_restore.go`, which reads `hdd/`+`userdata/` only (`:101-104`) | M | READY — 2026-07-28 | **Was C9-F4** (`OPEN-ITEMS.md`). The sharp edge is *when* it bites: Tier-2 exists for primary-drive loss, and in exactly that failure the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action — leaving offsite as the only route. LIVE: demo-felhom's `backups/secondary/{bookstack,docmost}/` hold `recovery-unit` and nothing else, at 156 MB and 86 MB. Flips: the Tier-2 row in map §C; `07` §6.3, §7.2 | +| R-103 | **The Tier-2 no-coverage refusal names the working action but does not route to it.** v0.183.0 refuses up front without stopping the app and tells the customer to use „Visszaállítás indítása" on the other page; it does not take them there | S | READY — 2026-07-28 | **Was C9-F1b.** Deliberately its own item: it puts a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference. Flips: nothing until shipped; `07` §10.2 | +| R-104 | **An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach.** `resticStep` has `unlock --remove-all` (`internal/backup/offbox.go:634-648`) but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`:77-93`) has no lock case → `"other"` → fail-fast; `ClassifyOffsiteFailure` likewise, so the operator is told *„A távoli mentés ismeretlen okból nem sikerült"* for a precisely-known, self-healable condition | S | READY — 2026-07-28 | **Was C9-F3.** Reachable by any interruption — container restart, OOM, network drop, host reboot mid-backup. The tier stays dead until a human runs `restic unlock --remove-all`. Flips: the offsite row in map §C; `07` §8 row 15 | +| R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. Flips: `07` §4 | +| R-106 | **The DR recipe records the PBS namespace as `"root"` on every box** while the real namespaces are `demo-felhom` / `demo-hp` | XS | READY — 2026-07-28 | Traced to source: `Snapshot.Namespace` decodes a `ns` field (`felhom-agent/internal/pbs/client.go:97`) that PBS does not echo per item when the request is already namespace-scoped via `?ns=` (`:117-120`), so it is empty and `ToHub` normalises empty → `"root"` (`internal/pbs/report.go:23-27`), which `latestPBSCoord` writes into the recipe (`internal/hub/dr_recipe.go:149`). `latest_snapshot_id: "9201"` is **not** a defect — it is `BackupID`, documented as a coordinate (`:56`). Verified against ep0's real namespaces and a live `pct restore` command line carrying `--ns demo-felhom`. Flips: `07` §4 | +| R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 | +| R-108 | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5.** For local drives FileBrowser binds `/userdata` only (`internal/web/handlers.go:2450-2460`); for network paths it binds the share **root** with `:rslave` (`:2432`) and serves it with `download: true` (`internal/infra/infra.go:326`). Nothing stops an app's namespace landing there: `GetSchedulableStoragePaths()` has no `IsNetwork()` filter (`internal/settings/settings.go:904-914`, feeds the deploy dropdown at `handlers.go:462-473`), the per-app migrate target list has none (`handlers.go:674-679`), and `handleStorageMigrateApp` does not call `refuseNetworkLifecycle` though its whole-namespace sibling does (`internal/web/storage_handlers.go:397` vs `:410-424`) | M | **READY — blocks an architectural target** — 2026-07-28 | **Today this is not a secret leak** — the recovery unit's `app.yaml` is secret-stripped (`internal/backup/recovery_unit.go:73`). It becomes one under **D5**, which would put app secrets in the local unit precisely so Tier-1/Tier-2 restore stop needing the guest. Verified LIVE in demo-hp's generated compose, where the asymmetry is visible rather than inferred: `- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb` beside `- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave`. **Second effect, independent of D5:** `.fab` bundles already carry **plaintext** secrets by design with an OPTIONAL password (`internal/appexport/export.go:484,506-511,307`) and `storageDriveList()` does not filter network paths (`internal/web/handler_export.go:377-387`), so one can be exported onto a NAS today. Full sweep of every read surface: `07-backup-architecture.md` §10.1. Flips: `07` §7.3 (D5 target → adopted) once closed | + ## P3 — post-alpha | ID | Item | Size | Status | Notes |