R-356 docs: correct R-107 in the architecture, record the design, refresh STATUS, compress the register
gates / gates (push) Failing after 17s
gates / gates (push) Failing after 17s
07-backup-architecture.md: three places said no offsite action unpacks the named-volume tars. R-107 closed in controller v0.218.0; all three corrected with a dated [FACT], the old sentence kept in the past tense. R-102 is NOT closed and the correction says so explicitly. New [DESIGN] paragraph in 6.3: the restore destination is resolved by the same rule as the capture destination, and the wrong-disk refusal applies to apps that have a drive to get wrong. Carries the 13/40 measurement. STATUS.md was internally contradictory - nothing waiting, and one decision waiting, for something the same page recorded as shipped. 218 -> 102 lines; the deciding section now says what happens if nothing is done. R-356 compressed into CLOSED-ITEMS.md; OPEN-ITEMS 327109 -> 325236 bytes. Drill record and 16 evidence files for the live walk on demo-hp.
This commit is contained in:
@@ -334,12 +334,43 @@ refuses **before** stopping the app and names the action that works.
|
||||
|---|---|---|---|
|
||||
| Tier-1 | unit incl. volume tars + DB dumps | all of it | none |
|
||||
| Tier-2 | unit mirror **+** file legs | `hdd/` and `userdata/` **only** (`tier2_restore.go:101-104`) | **the unit mirror is read by nothing** — `RecoveryUnitPath` resolves to `backups/primary/` (`appbackup/paths.go:46-48`) → **R-102** |
|
||||
| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay; the unit is **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **no offsite action unpacks the named-volume tars it captures** → **R-107** |
|
||||
| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay **+ the named-volume tars, replayed from the scratch unit** (`offbox_reconstitute.go` `volReplay`, controller **v0.218.0**); the unit itself is still **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **CLOSED — R-107** |
|
||||
|
||||
**[FACT] 2026-08-22 — the Tier-3 row above was corrected; the Tier-2 row was NOT.** Until controller
|
||||
**v0.218.0** this table said *"no offsite action unpacks the named-volume tars it captures"*, and that
|
||||
was true from the day Tier-3 shipped until 2026-08-21. **R-107 closed in v0.218.0**: `volReplay`
|
||||
(`offbox_reconstitute.go`) replays the scratch unit's `volume-dumps/` into the live named volumes,
|
||||
proven live on `demo-hp`. The old sentence is kept here, in the past tense, because a correction that
|
||||
erases what was believed leaves the next reader no way to tell a fixed gap from one that was never
|
||||
noticed.
|
||||
|
||||
**R-102 — the Tier-2 half — is NOT closed and nothing in this correction touches it.** The Tier-2 row
|
||||
above stands exactly as written: the secondary unit mirror is still read by nothing. Do not read
|
||||
"R-107 closed" as covering both; they were always two register rows, and only one of them moved.
|
||||
|
||||
**[FACT]** Tier-2's gap is the sharper one because of *when* it bites: Tier-2 exists for the case
|
||||
where the primary drive is lost — and in exactly that case the primary unit is gone while this
|
||||
mirror survives on the second drive, unreachable by any customer action.
|
||||
|
||||
**[DESIGN] 2026-08-22 — the restore destination is resolved by the same rule as the capture
|
||||
destination.** The drive if the app declares one (`HDD_PATH`), the system data path otherwise —
|
||||
`Manager.GetAppDrivePath`, one expression, used by `CaptureRecoveryUnit` and, since controller
|
||||
**v0.219.0**, by `ReconstituteFromOffsite` and `PlaceOffsiteRestore` too.
|
||||
|
||||
The refusal that protects a drive app from being restored onto the wrong disk (R-253, R-351) applies
|
||||
to apps that **have a drive to get wrong**. It used to be reached by testing `HDD_PATH == ""`, which
|
||||
also answered "is this app installed?" — one predicate for two questions. Measured in the catalogue at
|
||||
`459766cb1639`: **53 templates, 13 declare `needs_hdd: true`, 40 declare `false`**, so for 40 apps that
|
||||
test was permanently true and the off-site restore refused them forever, while they were running,
|
||||
with a message telling the customer to reinstall them "in the same place" — a place those apps never
|
||||
offer. **An app with no drive is not misconfigured** (§8 of `01-topology-and-trust.md` carries the
|
||||
`[DESIGN]` marker); it is the majority case.
|
||||
|
||||
Since v0.219.0 the two questions are asked separately: *installed?* of `ListDeployedStacks()`, failing
|
||||
CLOSED when there is no provider to ask; *where?* of `GetAppDrivePath`. A third refusal, with its own
|
||||
sentence, covers installed-but-no-resolvable-data-root. **R-356**; reasoning also recorded in
|
||||
`felhom-controller/CONTEXT.md`.
|
||||
|
||||
---
|
||||
|
||||
## 7. The recovery chain (D3) — the reason this document exists
|
||||
@@ -525,10 +556,14 @@ anywhere under the backup namespace. Full record: `audits/D5-drive-alone-restore
|
||||
drive. In that failure the primary recovery unit is gone; the surviving mirror on the second drive
|
||||
is `backups/secondary/<app>/recovery-unit/`, which **no code path reads** (§6.3). For the 45-or-43
|
||||
class-B apps the restore is a guaranteed no-op in exactly its designed scenario. → **R-102**
|
||||
- **Tier-3 vs guest loss.** Tier-3 holds the volume tars and the DB dump. Reconstitution requires
|
||||
the app to be deployed and skips the unit; the tars are unpacked only by the Tier-1 path, which
|
||||
requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest.
|
||||
→ **R-107**
|
||||
- **Tier-3 vs guest loss — HALF of this closed.** Tier-3 holds the volume tars and the DB dump. It
|
||||
was true until controller v0.218.0 that the tars were unpacked only by the Tier-1 path; since
|
||||
v0.218.0 the reconstitution replays them itself (`volReplay`) → **R-107 CLOSED 2026-08-22**. What
|
||||
is **still** true, and is the part that was never R-107: reconstitution requires the app to be
|
||||
**deployed** and skips the unit, so offsite alone cannot rebuild an app onto a fresh guest. That
|
||||
requirement is a deliberate decision (R-253) — the restore does not choose a customer's drive for
|
||||
them — and since **v0.219.0** it means only what it says: an app that is not deployed. It no longer
|
||||
catches the 40 driveless apps → **R-356**.
|
||||
|
||||
### 7.3 ~~What D5 would change — and why it was blocked~~ — **D5 SHIPPED 2026-07-30 (v0.188.0)**
|
||||
|
||||
@@ -737,7 +772,7 @@ crosses the line — **R-158**.
|
||||
| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar |
|
||||
| 3c | *same, with the GUEST GONE (secrets unavailable)* | the drive | Tier-1 unit restore — **the unit carries the portable secrets** | **customer** | **27.6 s** | 24 h | **PROVEN** | D5, §7.4. Before v0.188.0 this row was **NONE**: the data-key gate refused and the customer's own copy of their own data was not a recovery |
|
||||
| 3b | *same, for a class-B app via Tier-2* | — | **no route** — Tier-2 never reads the unit mirror | — | | | **NONE** | §6.3; **R-102** |
|
||||
| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB; **the volume tars in either copy are unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102**, **R-107** |
|
||||
| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB **+ the named-volume tars since controller v0.218.0** (`volReplay`); **the Tier-2 copy's volume tars remain unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102** (open); **R-107 CLOSED 2026-08-22, v0.218.0** |
|
||||
| 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go:359-393` |
|
||||
| 6 | **Guest lost or corrupted** | the host, both whole-guest tiers, the data drives (they are host binds) | `pct restore` from `local:` or `felhom-pbs:` | **operator** (SSH) | **84–112 s** local · **1101 s** PBS (both = restore-test into a scratch guest, boot + verify + teardown) | 24 h local · 7 d offsite | **PROVEN** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C (exact mount parity, `unprivileged: 1` preserved); LIVE restore-tests on both boxes this session |
|
||||
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
|
||||
@@ -899,7 +934,8 @@ does **not** hold as written. → **R-108**
|
||||
| **R-104** | An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach, reported as *„ismeretlen okból"* | the offsite tier stays dead until a human unlocks. Was C9-F3 |
|
||||
| **R-105** | Three hub-held DR records are empty on the whole live fleet: `hosts.dr_record_json`, `host_escrow.directive_json`, `dr_recipe.host_half.drives` | the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field |
|
||||
| **R-106** | `dr_recipe.host_half.pbs.namespace` records `"root"` on every box | the recorded restore coordinate is wrong; real namespaces are per-customer |
|
||||
| **R-107** | No offsite action unpacks the named-volume tars Tier-3 captures on every run | offsite alone cannot rebuild a named-volume app (§7.2) |
|
||||
| **R-107** | ~~No offsite action unpacks the named-volume tars Tier-3 captures on every run~~ — **CLOSED, controller v0.218.0, 2026-08-22** (`volReplay`, proven live on `demo-hp`). True from the day Tier-3 shipped until 2026-08-21. | was: offsite alone cannot rebuild a named-volume app (§7.2). Now: the tars replay; what remains is that the app must be **deployed** (R-253), which is not R-107 |
|
||||
| **R-356** | The off-site restore resolved its destination with the raw `HDD_PATH` and read an empty answer as "not installed" — **CLOSED, controller v0.219.0, 2026-08-22** | 40 of 53 apps were refused permanently while running (§6.3 `[DESIGN]`) |
|
||||
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
|
||||
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
|
||||
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
|
||||
|
||||
Reference in New Issue
Block a user