D5 SHIPPED: Tier-1/2 restore no longer depends on the whole-guest tier
Records controller v0.188.0 across the four coupled artifacts. 07-backup-architecture.md is the owning doc: - new 7.4 = the recovery chain AFTER D5 (7.1 leg 1 superseded; leg 2, the living-app dependency, explicitly unchanged so this is not read as more than it is) - 7.3 collapsed to history, with the correction that the target as written (data_key-only) was tested in Part 0 and rejected - 3 records that the two-lane split is now real, not just intended - matrix rows 3 / 3c (new) / 13; 10.1 D5 itself shipped Also: new capability-map row, D5 collapsed in ROADMAP + OPEN-ITEMS, and R-127 filed in both (data_key flag unreliable; O4 can regenerate a DB password that no longer matches the restored data directory). The audit is named D5-drive-alone-restore rather than "...secrets..." because .gitignore blocks *secret* -- a guard worth respecting, not forcing past.
This commit is contained in:
@@ -96,6 +96,7 @@
|
||||
| Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-5 (scope=app round-trip, byte-identical); `storage-lifecycle-acceptance-2026-06-15` (two migrate-all runs via dashboard UI, sha256 byte-identical) | (Cited `CAMPAIGN-2` T-STG-MIGRATE-* were auth-hollow.) "crash-safe" is design-level (copy→verify→remove) — no clean live crash-during-migration PASS |
|
||||
| NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | `SPIKE-nas-verify-2026-07-11`, `SPIKE-nas-dsm-2026-07-11`, `CAMPAIGN-3-2026-07-11` (boot/reassert fixes) | |
|
||||
| **Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace** | controller v0.187.0 | **PROVEN-LIVE** (2026-07-30) | `audits/R108-network-app-namespace-2026-07-30.md`. Same-box before/after on demo-felhom through the exact endpoint the UI invokes (`POST /api/storage/migrate-app`, authenticated + CSRF): **pre-fix v0.186.0** the target was never examined — both a NAS-shaped and an unregistered path passed straight into `MigrateApp` and failed only on the app name (409); **post-fix v0.187.0** both are refused 400 with a Hungarian reason, while a real local drive still reaches `MigrateApp` (409) proving the guard is not over-broad. On demo-hp (the box with a REGISTERED share) the network-specific refusal fires. Non-effect verified in the registry: no `migrated_to`, nothing decommissioned, app `HDD_PATH` unchanged, no `backups/` on the share | **Closes R-108 and UNBLOCKS D5** (`07-backup-architecture.md` §7.3, §10.1). The share-root `:rslave` FileBrowser bind is deliberately UNCHANGED — load-bearing for automount wake, and unscopable — verified byte-identical by diffing demo-hp's generated compose before/after. Fails closed: `/mnt/felhom-drives` holds both kinds, so an unregistered path under it is un-classifiable and refused. **Not exercised:** the deploy POST and decommission-migrate refusals are unit-tested (non-effect, nil `stackMgr`) but were NOT live-fired — only migrate-app was. `.fab`-export-onto-NAS remains open (→ R-126) |
|
||||
| **A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5)** | controller v0.188.0 | **PROVEN-LIVE** (2026-07-30) | `audits/D5-drive-alone-restore-2026-07-30.md`, `07-backup-architecture.md` §7.4. On a scratch drill guest on felhom-pve, through the real endpoints (`POST /api/stacks/{app}/deploy` → `POST /api/backup/run` → `POST /backup/restore`): AdventureLog (`SECRET_KEY` **data_key** + `DB_PASSWORD`) restored with the guest's `app.yaml` **moved aside** → `secrets recovered=2/2`, **27.6 s**, `Restore-from-unit completed`. **The observable is the DATA, not the exit code:** the app itself then read the seeded customer row **over TCP with its own credential** (`connected_as=adventurelog over_TCP=True`), 51 Django tables intact, and the discriminator held — the pre-backup row returned while a row added AFTER the backup was gone, so the volume tar was genuinely restored. The unit held **no `.sql` dump**, so the DB came back from the volume tar, which is exactly the case a regenerated password would have broken. **The withheld half is proven too:** Grafana's `type: password` admin login was live in its container and `ENC:` in the guest, yet appeared in **0 files** anywhere under the backup namespace, and the unit's app.yaml header names it as withheld | **Ruling (operator, 2026-07-30): `type: secret` travels, `type: password` NEVER does, minus the `nonPortableSecrets` code register (`vaultwarden/ADMIN_TOKEN`).** Plaintext on the drive, like the data — defensible **only because** the internet-reachable class is withheld; the two are coupled and must not be relaxed independently. **The brief's own proposal (data_key-only) was tested and rejected:** the flag is unreliable (4 encryption keys the catalog itself labels as such are unflagged → **R-127**) and a DB password is not resettable in practice (`POSTGRES_PASSWORD` is ignored once PGDATA is non-empty). Precedence: **the unit wins** over the guest, because the unit's secrets match the data being restored. The fail-closed data-key gate is UNCHANGED. **Not exercised live:** the withheld-class O4 regeneration on restore, and Tier-2's own cross-drive copy of a secret-bearing unit (both unit-tested only). Venue was a **fixture-class scratch guest**, correct per `runbooks/target-selection.md` since D5's claim is about restore CODE, not the install path |
|
||||
| USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E4 (yanked-while-running → agent auto-rebind) + E10 (re-enroll, data intact); `CAMPAIGN-4`/`6A` (3 USB re-establish across device-letter reshuffle) | (Cited `RUNBOOK-usb` could NOT complete a wizard enrollment; `CAMPAIGN-2` legs were auth-hollow.) Fresh-USB **wizard enrollment** specifically still unproven |
|
||||
| Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E9 (decommission-anyway → bind detached, parent mp untouched, reboot-safe) + E12 (eject drive holding all apps) | (Cited `CAMPAIGN-2` T-STG-DECOM-* were auth-hollow; `SPIKE-decommission` was report-only, button still vestigial.) |
|
||||
| Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | `CAMPAIGN-4-2026-07-13` (F12 fix HOLDS: demo-host reboot + 5-boot storm, 0 ordering cycles, caps 63/63, WG re-handshake) + `CAMPAIGN-6A-2026-07-14` 1D (re-arm reboot-survival across 9 guest + 1 host reboots) | (`CAMPAIGN-3` F10/F11/F12 were the CRITICAL/HIGH *failures*; fixes shipped in agent v0.85 and were re-validated live in 4/6A — cite the validation, not the finding.) Residual: `skip-active` on `pct reboot` carried by the heal path; a NAS outage spanning a guest reboot can strand the share until agent restart (6A) |
|
||||
|
||||
@@ -140,6 +140,14 @@ credentials" — a hazard serious enough to have its own runbook page
|
||||
(`runbooks/RUNBOOK-manual-guest-restore.md:45-48`, F-OPS). Putting Lane 2 behind the operator is
|
||||
what lets Lane 1 be a button instead of a procedure.
|
||||
|
||||
**[FACT] And as of D5 (v0.188.0) the split is real, not just intended.** Until then Lane 1 only *looked*
|
||||
independent: an app's files sat on the customer's drive but could not be brought back without secrets
|
||||
that lived solely in the guest, so the fast customer lane was silently conditioned on the slow
|
||||
operator lane (§7.1 leg 1). The unit now carries those secrets, so **Lane 1 needs the drive and nothing
|
||||
else** for app rebuild (§7.4). The one honest caveat: Lane 1's *file-merge* surface (Tier-2 additive)
|
||||
still reads its destination from the guest's `settings.json`, so that surface remains guest-conditioned
|
||||
— D5 removed the secrets leg, not the living-app leg.
|
||||
|
||||
---
|
||||
|
||||
## 4. The three-part model (D4)
|
||||
@@ -307,7 +315,15 @@ here — §8 has rows where it is the actual state.
|
||||
|
||||
### 7.1 The dependency graph
|
||||
|
||||
**[FACT]** Every app-tier restore depends on the guest, for one of two reasons:
|
||||
> **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below
|
||||
> (**secrets**) is **no longer true for Tier-1 and Tier-2**: the local recovery unit now carries the
|
||||
> portable secret class, so those two tiers need the **drive and nothing else**. The graph and leg 1
|
||||
> are kept as written because they are the model everything downstream was derived from, and because
|
||||
> leg 2 (**the living app**) is unchanged and still binds Tier-2's file leg and Tier-3. Read §7.4 for
|
||||
> the current chain; the corrected rows are 3, 4 and 9 in §8.
|
||||
|
||||
**[FACT — as of the R-108 survey, before D5]** Every app-tier restore depends on the guest, for one of
|
||||
two reasons:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
@@ -357,7 +373,66 @@ here — §8 has rows where it is the actual state.
|
||||
|
||||
**[FACT] So all three app tiers are conditioned on the whole-guest tier**, which is Lane 2. The
|
||||
customer's own recovery lane rests on an operator-only tier, and today nothing tells the customer
|
||||
that.
|
||||
that. — **This sentence was the reason D5 exists. It is now false for Tier-1 and Tier-2; see §7.4.**
|
||||
|
||||
### 7.4 The chain AFTER D5 (controller v0.188.0, 2026-07-30) — the current model
|
||||
|
||||
**[FACT] Tier-1 and Tier-2 no longer depend on the whole-guest tier.** The unit's `compose/app.yaml`
|
||||
(mode 0600) carries the **portable secret class** — every catalog `type: secret` field, i.e. the
|
||||
declared `data_key`s, the 18 DB/root passwords and the internal signing secrets — so neither
|
||||
`encryption.key` nor the guest's `app.yaml` is required to rebuild an app:
|
||||
|
||||
```
|
||||
Tier-1 restore Tier-2 restore Tier-3 reconstitute Tier-3 scratch
|
||||
(rebuild the app) (fill in files) (overwrite + replay) (verification copy)
|
||||
│ │ │ │
|
||||
needs ONLY THE DRIVE needs the app needs the app needs only the
|
||||
(unit carries the running + the deployed + a DB repo password
|
||||
secrets; guest is recorded dest service identifiable
|
||||
consulted only for
|
||||
the withheld class) └────────── still conditioned on the guest (leg 2, unchanged) ──────┘
|
||||
│
|
||||
✅ INDEPENDENT of the whole-guest tier
|
||||
```
|
||||
|
||||
**[FACT] What still needs the guest, precisely** — so this is not read as more than it is:
|
||||
|
||||
1. **Leg 2 is untouched.** Tier-2's *file* restore still reads the destination from the guest's
|
||||
`settings.json` (`tier2_restore.go:114-116`) and Tier-3's reconstitution still refuses when the app
|
||||
is not deployed (`offbox_reconstitute.go:198-201`). D5 removed the **secrets** leg, not the
|
||||
**living-app** leg. So "Tier-2 no longer depends on the guest" is true of the *unit-restore* path it
|
||||
shares with Tier-1, and **false** of its additive file-merge path.
|
||||
2. **The withheld class.** The 7 `type: password` admin logins and `vaultwarden/ADMIN_TOKEN` are
|
||||
deliberately NOT on the drive; they come from the guest, or are regenerated (O4). Their absence
|
||||
costs a credential reset, never data.
|
||||
3. **The fail-closed gate remains.** A `data_key` in neither the unit nor the guest still refuses the
|
||||
restore outright. D5 makes it normally present; it does not soften the gate.
|
||||
|
||||
**[DESIGN] Why plaintext is the right answer here, and what it is coupled to.** Every travelling secret
|
||||
either decrypts data on the *same drive* or authenticates to a container on an internal compose network
|
||||
with no external listener — so possessing it adds nothing to possessing the drive, which is exactly D2's
|
||||
argument for keeping the DATA plaintext. That argument holds **only because** the internet-reachable
|
||||
class is withheld. **The exclusion is what licenses the plaintext; the two must not be relaxed
|
||||
independently.** `stacks.PortableSecretEnvVars` is the single boundary, with a code-level
|
||||
`nonPortableSecrets` register (not a catalog flag — a boundary a catalog push can move is not a
|
||||
boundary, cf. R-97a).
|
||||
|
||||
**[FACT] Precedence, because two sources now exist.** The **unit wins** over the guest. Not
|
||||
"newest wins": the unit's secrets are captured in the same run as the dumps beside them, so the unit's
|
||||
value matches *the data being restored*, whereas the guest's is merely the most recent. Preferring the
|
||||
guest is the data-loss bug — a rotated data key does not decrypt data encrypted with the old one, and a
|
||||
rotated DB password does not match the hash inside the restored data directory.
|
||||
|
||||
**[FACT] Proven live 2026-07-30**, on a scratch drill guest, through the real endpoints: AdventureLog
|
||||
(`SECRET_KEY` data_key + `DB_PASSWORD`) restored with the guest's `app.yaml` **moved aside** —
|
||||
`secrets recovered=2/2`, 27.6 s — and then **the application itself read the seeded customer row over
|
||||
TCP with its own credential**, which is the observable that matters (a restore that returns success onto
|
||||
unreadable data is a failure that looks like a pass). The discriminator held: the pre-backup row came
|
||||
back, a row added *after* the backup was gone, so the volume tar was genuinely restored. There was **no
|
||||
`.sql` dump** in that unit, so the DB came back from the volume tar — the exact case where a regenerated
|
||||
password would have failed. Separately, Grafana's `type: password` admin login was proven **withheld**:
|
||||
the sentinel value was live in the container and encrypted in the guest, and appeared in **0 files**
|
||||
anywhere under the backup namespace. Full record: `audits/D5-drive-alone-restore-2026-07-30.md`.
|
||||
|
||||
### 7.2 A tier whose prerequisites cannot be met in the failure it exists for
|
||||
|
||||
@@ -372,9 +447,21 @@ that.
|
||||
requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest.
|
||||
→ **R-107**
|
||||
|
||||
### 7.3 What D5 would change — and why it was blocked (PRECONDITION NOW MET, 2026-07-30)
|
||||
### 7.3 ~~What D5 would change — and why it was blocked~~ — **D5 SHIPPED 2026-07-30 (v0.188.0)**
|
||||
|
||||
**[DESIGN, TARGET — UNBLOCKED]** The intended fix is to make app secrets travel with the **local**
|
||||
> **This section is history.** D5 is implemented and proven live; the current chain is **§7.4**. Kept
|
||||
> because the reasoning below is why the precondition was required, and because the last paragraph
|
||||
> (`.fab` / R-126) is still open and still not part of D5.
|
||||
>
|
||||
> **One correction to the target as stated below.** It assumed the class that must travel is the
|
||||
> `data_key`-flagged secrets. Part 0 of the D5 task tested that and it did not survive: the flag is
|
||||
> unreliable (four encryption keys the catalog itself labels as such are unflagged → **R-127**), and a
|
||||
> DB password is **not resettable in practice** — `POSTGRES_PASSWORD` is ignored once PGDATA is
|
||||
> non-empty, so a regenerated value leaves the app unable to authenticate against its own restored rows
|
||||
> while the dump replay still reports success. The shipped boundary is therefore *all* `type: secret`
|
||||
> minus the register, not `data_key` alone.
|
||||
|
||||
**[DESIGN, TARGET — SUPERSEDED BY §7.4]** The intended fix is to make app secrets travel with the **local**
|
||||
recovery unit, so Tier-1 and Tier-2 restore work **without the guest and without R**. Offsite
|
||||
already encrypts everything, so secrets travelling offsite would be covered by the escrowed repo
|
||||
password. R would then be required for **offsite recovery and host identity only** — losing R would
|
||||
@@ -416,7 +503,8 @@ Until D5 is actually implemented, §7.1's chain stands as the model.
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | **Customer deletes files** | everything else | „Fájlok visszaállítása" — Tier-2 additive merge | **customer** | **39 s** (6 files) | 24 h | **PROVEN** | CAMPAIGN-9 A1 — byte-identical returns, both non-destruction promises kept, bytes re-served by paperless's own API |
|
||||
| 2 | **An app's data directory is destroyed** | the guest, the other tiers | same route | **customer** | **46 s** (43 files) | 24 h | **PROVEN** | CAMPAIGN-9 A3 — loss verified by a positive observable (doc download 200 → 404), then 16/16 documents usable |
|
||||
| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) | 24 h | **PARTIAL** | CAMPAIGN-9 A2 proves the path executes and the app survives; content recovery after real loss **never demonstrated** (`CAMPAIGN-9…:825-827`) |
|
||||
| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar |
|
||||
| 3c | *same, with the GUEST GONE (secrets unavailable)* | the drive | Tier-1 unit restore — **the unit carries the portable secrets** | **customer** | **27.6 s** | 24 h | **PROVEN** | D5, §7.4. Before v0.188.0 this row was **NONE**: the data-key gate refused and the customer's own copy of their own data was not a recovery |
|
||||
| 3b | *same, for a class-B app via Tier-2* | — | **no route** — Tier-2 never reads the unit mirror | — | | | **NONE** | §6.3; **R-102** |
|
||||
| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB; **the volume tars in either copy are unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102**, **R-107** |
|
||||
| 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go:359-393` |
|
||||
@@ -428,7 +516,7 @@ Until D5 is actually implemented, §7.1's chain stands as the model.
|
||||
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
|
||||
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
|
||||
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D |
|
||||
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B |
|
||||
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) |
|
||||
| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
|
||||
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
|
||||
|
||||
@@ -477,7 +565,13 @@ uplink — **no customer has ever driven a restore**.
|
||||
|
||||
Every divergence between the model above and the system as it is, each with an ID.
|
||||
|
||||
### 10.1 ~~D5 is BLOCKED~~ — CLOSED 2026-07-30 by R-108 (controller v0.187.0)
|
||||
### 10.1 ~~D5 is BLOCKED~~ — precondition CLOSED by R-108 (v0.187.0); **D5 ITSELF SHIPPED (v0.188.0)**
|
||||
|
||||
> **D5 IS DONE — 2026-07-30, controller v0.188.0.** The precondition below was met by R-108, and D5 was
|
||||
> then implemented and proven live the same day: Tier-1/Tier-2 no longer depend on the whole-guest tier,
|
||||
> and a customer needs **the drive and nothing else**. The current recovery chain is **§7.4**; the
|
||||
> corrected matrix rows are 3, 3c and 13. This section's remaining value is the *precondition* analysis,
|
||||
> which is why the table below is kept.
|
||||
|
||||
> **D5's PRECONDITION IS MET.** An app's data namespace can no longer be placed on network storage, so
|
||||
> no `backups/` tree can exist inside FileBrowser's share-root bind, and the browsing surface therefore
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
# D5 — an app restore works from the drive alone (controller v0.188.0, 2026-07-30)
|
||||
|
||||
**Verdict: SHIPPED + PROVEN-LIVE.** Tier-1/Tier-2 no longer depend on the whole-guest tier. A customer
|
||||
needs **the drive and nothing else** to restore an app.
|
||||
|
||||
Owning design doc: `architecture/07-backup-architecture.md` §7.4 (new), with §3, §7.1, §7.3, §8 rows
|
||||
3/3c/13 and §10.1 corrected. Capability map: new row in §D.
|
||||
|
||||
---
|
||||
|
||||
## 1. Part 0 — the rulings, and why the brief's own recommendation was rejected
|
||||
|
||||
The task proposed that **only `data_key`-flagged secrets travel**, and asked for that to be tested
|
||||
rather than obeyed. It did not survive.
|
||||
|
||||
### 1.1 `data_key: true` is not a trustworthy classification → R-127
|
||||
|
||||
Only **5 fields across 4 apps** set it (`adventurelog/SECRET_KEY`, `homebox/HBOX_AUTH_API_KEY_PEPPER`,
|
||||
`papra/AUTH_SECRET`, `sparkyfitness/{API_ENCRYPTION_KEY,BETTER_AUTH_SECRET}`). But the catalog's own
|
||||
Hungarian labels contradict the flag elsewhere:
|
||||
|
||||
| Field | Catalog's own label | `data_key` |
|
||||
|---|---|---|
|
||||
| `n8n/N8N_ENCRYPTION_KEY` | „Titkosítási kulcs" | **false** |
|
||||
| `wanderer/POCKETBASE_ENCRYPTION_KEY` | „Adatbázis titkosítási kulcs" | **false** |
|
||||
| `calcom/CALENDSO_ENCRYPTION_KEY` | „Titkosítási kulcs" | **false** |
|
||||
| `bookstack/APP_KEY` | „Alkalmazás kulcs" | **false** |
|
||||
| `adventurelog/SECRET_KEY` | „Titkosítási kulcs" | true |
|
||||
|
||||
Same label, opposite flag. Travelling "only data keys" would have omitted real data keys, and the
|
||||
fail-closed gate (`restore_unit.go`) would not fire for them — a restore that **succeeds onto
|
||||
unreadable data**.
|
||||
|
||||
### 1.2 A DB password is not resettable in practice — proven, not argued
|
||||
|
||||
`DumpAppVolumes` (`internal/backup/backup.go:549`) dumps **every** compose named volume with no DB
|
||||
exclusion, so a postgres data volume is captured and restored. Probe on a throwaway
|
||||
`postgres:16-alpine` (seed with pw A → drop container, keep volume → redeploy with regenerated pw B,
|
||||
which is exactly the O4 path):
|
||||
|
||||
| Leg | Result |
|
||||
|---|---|
|
||||
| **replay** — `docker exec psql -U app -d app`, no password (verbatim `appbackup.ImportDump`) | **SUCCEEDS** — the image's local socket is `trust` |
|
||||
| **app path** — pw B over the compose network | `FATAL: password authentication failed for user "app"` |
|
||||
| **app path** — pw A (the hash still in the restored PGDATA) | **SUCCEEDS** |
|
||||
|
||||
`POSTGRES_PASSWORD` is ignored once PGDATA is non-empty (initdb skipped). So the restore reports
|
||||
success, the dump replays, the rows are there, **and the application cannot reach them** — the §4
|
||||
"looks like a pass" failure. **18 DB/root-password fields** affected.
|
||||
|
||||
> A first attempt tested `-h 127.0.0.1` and got a **false pass**: the postgres image ships a `trust`
|
||||
> line for localhost. Only the compose-network path is the app's real path. Recorded because it is the
|
||||
> same class of false observable this task exists to catch.
|
||||
|
||||
MariaDB fails *louder*: `getMariaDBPassword` (`appbackup/dbdump.go:737-754`) reads the regenerated value
|
||||
from container env against a datadir holding the old hash → Access denied, so the replay itself fails
|
||||
(`nextcloud`, `romm`).
|
||||
|
||||
### 1.3 The rulings (operator, 2026-07-30)
|
||||
|
||||
Classifying all 53 templates mechanically produced a boundary the brief's framing missed:
|
||||
|
||||
- **TRAVELS — every `type: secret` field (45):** 5 declared data keys + 18 DB/root passwords + 22
|
||||
internal signing/encryption secrets. Each decrypts data on the **same drive** or authenticates to a
|
||||
container on an internal compose network with **no external listener** — so possessing it adds nothing
|
||||
to possessing the drive, which is exactly D2's argument for plaintext DATA.
|
||||
- **WITHHELD — every `type: password` field (7)** plus the `nonPortableSecrets` register
|
||||
(`vaultwarden/ADMIN_TOKEN`, which gates the app's public `/admin` panel). These authenticate against
|
||||
published services, so their blast radius is **not** bounded by the drive.
|
||||
- **Plaintext**, as the data already is.
|
||||
|
||||
**Excluding the withheld class is what licenses the plaintext ruling. The two are coupled and must not
|
||||
be relaxed independently.**
|
||||
|
||||
**What a customer must possess to complete a Tier-1/2 restore after this change: the drive.**
|
||||
|
||||
---
|
||||
|
||||
## 2. Implementation
|
||||
|
||||
One place per side, no parallel path.
|
||||
|
||||
| Concern | Where |
|
||||
|---|---|
|
||||
| **The boundary** | `stacks.PortableSecretEnvVars` + the `nonPortableSecrets` register (`internal/stacks/deploy.go`) — **code, not a catalog flag**: a boundary a catalog push can silently move is not a boundary (R-97a) |
|
||||
| Decrypt for capture | `GetStackRecoveryInfo` (`cmd/controller/main.go`), via the **same** `LoadAppConfigDecrypted` the restore side uses |
|
||||
| Write the split | `buildUnitAppYaml` (was `buildStrippedAppYaml`) — 0600, and it **names the withheld class in the header** so an operator sees WHY a credential is absent rather than suspecting a capture bug |
|
||||
| Read it back | `readUnitEnv` — split driven by the **manifest's** portable names, never guessed from key names |
|
||||
| Reconcile | `reconcileRestoreSecrets` stays a **pure function**; the new source arrives as an **argument** |
|
||||
| Manifest | **schema 2** + `portable_secret_env_vars` (NAMES only — the manifest is 0644) |
|
||||
|
||||
**Precedence: the UNIT WINS.** Not "newest wins". The unit's secrets are captured in the same run as
|
||||
the dumps beside them (`runVolumeDumps` → `captureAllRecoveryUnits`), so the unit's value matches *the
|
||||
data about to be restored*; the guest's is merely the most recent. A rotated data key does not decrypt
|
||||
data encrypted with the old one; a rotated DB password does not match the hash in the restored data
|
||||
directory. An undefined precedence between two sources of a decryption key is a data-loss bug waiting
|
||||
for its first disagreement.
|
||||
|
||||
**The fail-closed gate is unchanged.** A data key in **neither** source still refuses outright. D5 makes
|
||||
it normally present; "normally" is not a reason to soften a gate.
|
||||
|
||||
**Backward compatible.** A schema-1 unit carries no secrets and still restores from the guest; the next
|
||||
capture rewrites it (the app.yaml checksum changes).
|
||||
|
||||
**Three comments that asserted invariants D5 makes false were corrected** rather than left to read as
|
||||
settled (`CaptureRecoveryUnit` *"NEVER writes a secret value"*, `RestoreFromRecoveryUnit` *"no secret is
|
||||
read from the unit"*, `appbackup/paths.go` + `appdata.go` *"secret-free"*), and the O4 WARN that claimed
|
||||
*"stored data is unaffected"* for every non-data-key secret was scoped to what §1.2 shows is true.
|
||||
|
||||
---
|
||||
|
||||
## 3. Tests
|
||||
|
||||
Suite **green, `rc=0`**, read on its own (never combined with a commit).
|
||||
|
||||
| Test | Asserts |
|
||||
|---|---|
|
||||
| `TestRestoreFromRecoveryUnitWithGuestAbsent` | **D5's claim as a test:** the restore SUCCEEDS with the guest absent |
|
||||
| `TestRestoreFromRecoveryUnitGuestAbsentStillFailsClosed` | data key in neither source ⇒ refuse, nothing mutated, app not even stopped |
|
||||
| `TestReconcileRestoreSecretsPrecedence` | unit-wins; guest fills in when the unit is silent or empty; a secret never shadowed by stray plain config |
|
||||
| `TestCaptureRecoveryUnitCarriesPortableSecretsOnly` | portable present + 0600 + schema 2; **the withheld value appears NOWHERE in the unit** (wrong-outcome check) |
|
||||
| `TestReadUnitEnvSplitsByManifest` | manifest-driven split; schema-1 degrades without losing entries |
|
||||
| `TestPortableSecretEnvVars` | the boundary itself, both directions, incl. the register being slug-scoped |
|
||||
| `…Orchestration/schema-1 unit still restores` | no regression for pre-D5 units |
|
||||
|
||||
**Seam (R-125):** `Manager.stackProvider` **only** — the docker/compose ops and the guest app.yaml
|
||||
decrypt. `RecoverStackSecrets` returning nil **is** the guest being gone: it is what the real adapter
|
||||
returns when the stack or its app.yaml cannot be read. Everything else is production code, and fixtures
|
||||
come from a unit written by the **real** `CaptureRecoveryUnit`, so capture and restore meet at real
|
||||
bytes rather than a test agreeing with itself.
|
||||
|
||||
**Four red-proofs, each mutation verified present in the working file before the run:**
|
||||
|
||||
| # | Mutation | Result |
|
||||
|---|---|---|
|
||||
| 1 | drop the `PortableSecrets` merge in `buildUnitAppYaml` | consequence + capture tests FAIL (`data-encrypting key did NOT travel`) |
|
||||
| 2 | `if false &&` the fail-closed gate | **4** failures across orchestration + pure-function tests |
|
||||
| 3 | flip precedence to guest-first | `both sources disagree — the UNIT wins` FAILS (`got "guest-key"`) |
|
||||
| 4 | widen the class to include `type: password` | the boundary test FAILS (admin password becomes portable) |
|
||||
|
||||
---
|
||||
|
||||
## 4. Live proof — the data, not the exit code
|
||||
|
||||
**Venue: a scratch drill LXC (9301) on felhom-pve**, destroyed after the run.
|
||||
`runbooks/target-selection.md` makes this the **correct** class: D5's claim is about **restore code
|
||||
behaviour**, not the install path / golden / first-boot state, so a fixture-class venue applies and a
|
||||
fresh day-0 is not required. Guest 9201 (both hosts), `drill-r50` (VM 300) and demo-hp's `local-lvm`
|
||||
were all fenced by the task and untouched. demo-hp could not host the guest at all: its `local` storage
|
||||
has no `rootdir` content type and `local-lvm` was forbidden.
|
||||
|
||||
Image identity confirmed: the guest pulled digest `sha256:f297fbf0…` — byte-identical to what
|
||||
`build.sh 0.188.0 --push` produced.
|
||||
|
||||
Path exercised through the **real endpoints** the UI invokes:
|
||||
`POST /api/stacks/{app}/deploy` → `POST /api/backup/run` → `POST /backup/restore`.
|
||||
|
||||
### 4.1 The portable half — AdventureLog (`SECRET_KEY` data_key + `DB_PASSWORD`)
|
||||
|
||||
[backup] Recovery unit captured for adventurelog → …/backups/primary/adventurelog
|
||||
(images=3, secrets-referenced=2, data_keys=1, portable-carried=2/2, withheld=0)
|
||||
|
||||
Unit `compose/app.yaml`: **mode 0600**, both secrets present in plaintext, `DB_PASSWORD` fingerprint
|
||||
**identical** to the password the running postgres was built with. The guest's own app.yaml holds the
|
||||
same names `ENC:`-encrypted. Manifest: `schema_version: 2`,
|
||||
`portable_secret_env_vars: [SECRET_KEY, DB_PASSWORD]`.
|
||||
|
||||
The guest's `app.yaml` was then **moved aside**, and the restore run:
|
||||
|
||||
[backup] Restoring adventurelog from recovery unit: images=3, secrets recovered=2/2, data_keys=1
|
||||
[backup] Restore-from-unit completed: adventurelog (27.6 s)
|
||||
|
||||
**`2/2` with the guest gone.** Pre-D5 this was `0/2` and the data-key gate refused.
|
||||
|
||||
**The positive observable — the application reading its own data:**
|
||||
|
||||
APP-READ-OK rows= ['D5-CUSTOMER-DATA-2026-07-30']
|
||||
connected_as= adventurelog over_TCP= True row_count= 1
|
||||
|
||||
Read through **Django's own connection**, as the app, **over TCP** — not the localhost trust socket that
|
||||
produced §1.2's false pass. Plus `manage.py check --database default` → *"System check identified no
|
||||
issues"*, and 51 `public` tables intact.
|
||||
|
||||
**The discriminator held**, which is what makes this a restore and not a no-op: the row seeded *before*
|
||||
the backup came back, and a row inserted *after* the backup was **gone**. And the unit contained **no
|
||||
`.sql` dump** (`db_dumps: None`) — the DB came back from the **volume tar**, i.e. exactly the case where
|
||||
a regenerated password would have failed silently.
|
||||
|
||||
Guest `app.yaml` was recreated from the unit with both secrets re-encrypted at rest.
|
||||
|
||||
### 4.2 The withheld half — Grafana (`GF_SECURITY_ADMIN_PASSWORD`, `type: password`)
|
||||
|
||||
[backup] Recovery unit captured for grafana → … (images=1, secrets-referenced=1, data_keys=0,
|
||||
portable-carried=0/0, withheld=1)
|
||||
|
||||
A distinctive sentinel was supplied as the admin password. It was **live in the container** (1 env
|
||||
match) and `ENC:` in the guest app.yaml — so the absence below is **not vacuous** — and it appeared in:
|
||||
|
||||
- **0 files** in grafana's recovery unit;
|
||||
- **0 files** anywhere under the whole backup namespace.
|
||||
|
||||
The unit's app.yaml header names it explicitly:
|
||||
`# WITHHELD (internet-reachable logins — stay in the guest, regenerated on restore): GF_SECURITY_ADMIN_PASSWORD`
|
||||
|
||||
### 4.3 Not exercised live
|
||||
|
||||
- **O4 regeneration of a withheld secret on restore** (unit-tested only).
|
||||
- **Tier-2's cross-drive copy of a secret-bearing unit** (unit-tested only; no code changed).
|
||||
- The offsite leg. **Recorded consequence, not a change:** the unit already travels to Tier-2 (another
|
||||
customer drive, plaintext, same reasoning) and offsite via restic (**encrypted at rest** under the
|
||||
customer-owned repo password). No tier code was touched — the secrets simply travel inside the unit
|
||||
those tiers already carried.
|
||||
|
||||
---
|
||||
|
||||
## 5. R-120's gate does not apply to this task
|
||||
|
||||
The gate lives in `hub/internal/web/configs.go` `handleSetArtifacts` — the golden **vouch** form, the
|
||||
sole UI path to `SetArtifactManifest`. It compares a submitted golden against
|
||||
`store.NewestReportedControllerVersion()`. It therefore **never runs on a controller image deploy**, and
|
||||
did not block anything here.
|
||||
|
||||
Re-baking the golden (currently 0.186.0) is a **follow-on so that FRESH installs get D5** — not a
|
||||
prerequisite. It is its own ceremony (bake fixture, upload, vouch, live day-0 proof) and bundling it
|
||||
into D5 would have put two unrelated risky ceremonies in one session.
|
||||
|
||||
---
|
||||
|
||||
## 6. Teardown — all three layers
|
||||
|
||||
| Layer | Disposition |
|
||||
|---|---|
|
||||
| **Guest** | LXC **9301 destroyed** (`pct destroy --purge`); `vm-9301-disk-0` removed. `pct list` shows only 9201, untouched |
|
||||
| **Host** (felhom-pve) | no leftover drill files in `/tmp`; thin pool back to **27.88 %**, identical to pre-drill, only 9201's volumes remain |
|
||||
| **Hub** | **nothing to dispose** — the drill ran `hub.enabled: false` and never registered. Verified positively, not by assuming: `/hosts` and `/configs` both loaded (9880 / 10160 bytes) and returned **0** occurrences of `d5drill` |
|
||||
|
||||
**Secret-bearing files shredded** (`shred -u -z`) inside the guest *before* the volume was destroyed:
|
||||
both units' `compose/app.yaml`, both stack `app.yaml`s, `encryption.key`, the hidden guest app.yaml, and
|
||||
every transported helper script — plus the host-side tarballs and the local scratchpad scripts.
|
||||
|
||||
**One self-inflicted slip, recorded rather than smoothed over:** an inspection script's redaction regex
|
||||
assumed 2-space YAML indentation while the unit's app.yaml uses 4, so the drill's `SECRET_KEY` and
|
||||
`DB_PASSWORD` values printed to the session transcript once. They were throwaway values in a guest that
|
||||
no longer exists, and they reached **no committed file** — but the guard was mine to get right, and the
|
||||
lesson is that a redaction pattern is itself a claim that needs checking against the real format.
|
||||
|
||||
---
|
||||
|
||||
## 7. Findings
|
||||
|
||||
- **R-127 (NEW, READY):** `data_key: true` is unreliable (leg a) and O4 can regenerate a DB password
|
||||
that no longer matches the restored data directory (leg b). Filed, **not fixed** — leg (a) is a
|
||||
catalog change plus a flag-vs-label gate; leg (b) wants either fail-closed treatment for DB passwords
|
||||
or an `ALTER USER` after the volume restore. Both are outside D5's ruling and neither is what D5 was
|
||||
asked to do.
|
||||
|
||||
D5's own defect surface — the false *"stored data is unaffected"* WARN and the three stale
|
||||
invariant-asserting comments — was corrected in place, since that is the defect this task was inside.
|
||||
|
||||
## 8. Arc status
|
||||
|
||||
**The arc's definition of done is MET.** Its four named components — **R-106**, **R-109** (both agent
|
||||
v0.118.1 / hub v0.83.0), **R-108** (controller v0.187.0) and **D5** (controller v0.188.0) — are all
|
||||
SHIPPED + PROVEN-LIVE.
|
||||
|
||||
Still open, none of them part of that definition: **R-105** (three hub-held DR records empty
|
||||
fleet-wide), **R-121** (installed-vs-vouched agent drift), **R-123** (PARTLY CLOSED — registry
|
||||
hygiene), **R-124** (PBS root namespace spelled `"root"` vs the API's `""`), **R-126** (`.fab` export
|
||||
onto a NAS), **R-117g** (no guided recovery for an aborted fs), **R-117h** (`StablePathForRaw` hardcodes
|
||||
the live parent, so the stale-bind repair is not test-seamable), and **R-127** (new, above).
|
||||
|
||||
**R-119 does not exist** — it appears in neither `OPEN-ITEMS.md` nor `ROADMAP.md`. Reported as absent
|
||||
rather than given an invented status.
|
||||
@@ -48,7 +48,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **C9-F1b** → **R-103** | Tier-2's restore cannot cover 43 of 53 apps; the action that CAN is the keep-side unit restore (`POST /backup/restore` → `RestoreFromRecoveryUnit`, replays volume tars + DB dumps). v0.183.0 NAMES it in the refusal text but does not route to it | **READY** | — | Put the working action in the card the customer already opened. **Deliberately its own task:** it places a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference — the reason it was not folded into v0.183.0 | CC |
|
||||
| **C9-F4** → **R-102** | **Nothing reads the Tier-2 copy's `recovery-unit/` mirror.** It is written by EVERY Tier-2 run (`tier2.go:369`, „Unit leg (always)") and read by no code path: `RecoveryUnitPath` resolves to `backups/**primary**/` (`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`, which reads `hdd/`+`userdata/` only | **READY (potentially > C9-F1)** | — | Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action, leaving offsite as the only route. Verified by enumeration: 6 references to `"secondary"` in the tree, one writer, one reader, one wipe-warning lister | CC |
|
||||
| **R-108** | ~~Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT~~ | **SHIPPED + PROVEN-LIVE** (controller v0.187.0, 2026-07-30) | — | **D5 IS UNBLOCKED — that is this item's whole point.** `RefuseAsAppNamespace` (`internal/settings/settings.go`) is one fail-closed predicate consulted by every placement surface, so no `backups/` tree can exist inside the share-root bind. **The fix inverted the obvious one:** the bind was NOT narrowed, because it cannot be — the `:rslave` share-ROOT bind is load-bearing (a 2026-07-22 probe proved an in-container access through it wakes the idle automount trigger), there is no `userdata/` layer to scope to (apps on a share store at `<share>/<app>`), and creating one would write Felhom convention onto a customer's own NAS, which R-67 forbids. Operator ruling: refuse the placement, keep the browse bind. **FIVE surfaces, not the four this row named** — `handleStorageDecommission` mode=migrate guarded only its SOURCE, so a whole namespace could be decommissioned ONTO a NAS; and the *boundary* is the deploy **POST**, not the dropdown (it takes any caller-supplied `HDD_PATH`, validated only by `os.Stat`). **Nothing stranded:** zero apps on network storage across all six customers incl. Peti. R-67's bind verified byte-identical before/after. Supersedes the controller README's "NAS backup locality — decision A". `audits/R108-network-app-namespace-2026-07-30.md` | — |
|
||||
| **D5** | **Move app secrets into the LOCAL recovery unit** so Tier-1/Tier-2 restore stop needing the guest and stop needing R (R would then be required for offsite recovery + host identity only) | **READY — UNBLOCKED 2026-07-30** (was blocked by R-108) | — | **The precondition is now met and stated in `07-backup-architecture.md` §7.3 + §10.1:** no browsing surface can reach the backup tree on any storage class. Every other read surface was already clear in the §10.1 sweep (SMB both namespace shapes, FileBrowser for drives, `.fab` import + download, `/api/debug/*`, all three `ServeFile` sites, storage-path add); the network-share bind was the one exception and R-108 closed it at the PLACEMENT rather than at the bind. **Registered as its own row 2026-07-30** — it previously existed only inside R-108's and E-2's prose, which is the R-123 thread-loss pattern. **NOT started:** unblocking is not implementing, and the task that closed R-108 was explicitly fenced from starting D5. Scope note: `.fab` bundles carry plaintext secrets independently of D5 (→ **R-126**) and are not part of its precondition | CC |
|
||||
| **D5** | ~~**Move app secrets into the LOCAL recovery unit** so Tier-1/Tier-2 restore stop needing the guest~~ | **SHIPPED + PROVEN-LIVE** (controller v0.188.0, 2026-07-30) | — | **CLOSED — the arc's architectural centrepiece is done, and Tier-1/2 no longer depend on the whole-guest tier.** A customer now needs **the drive and nothing else**. **Part 0 overturned the brief's own recommendation, on evidence gathered before any code** — that is the substantive part of this row. It proposed that only `data_key`-flagged secrets travel; two findings killed that: (1) the flag is **unreliable** — only 5 fields across 4 apps carry it, yet `n8n/N8N_ENCRYPTION_KEY` („Titkosítási kulcs"), `wanderer/POCKETBASE_ENCRYPTION_KEY` („Adatbázis titkosítási kulcs"), `calcom/CALENDSO_ENCRYPTION_KEY` and `bookstack/APP_KEY` carry the SAME labels as flagged `adventurelog/SECRET_KEY` and are unflagged (→ **R-127**), so data-keys-only would omit real data keys and the fail-closed gate would not fire for them; (2) a **DB password is not resettable in practice** — proven on a throwaway `postgres:16-alpine`: with PGDATA restored from the volume tar, `POSTGRES_PASSWORD` is **ignored** (initdb skipped), so a regenerated value fails over the compose network (`FATAL: password authentication failed`) while the old one still works AND the dump replay still SUCCEEDS via the container's local trust socket — a restore that reports success onto data the app cannot reach. 18 DB/root-password fields affected; MariaDB fails louder (`getMariaDBPassword` reads the new value against a datadir holding the old hash → Access denied). **Operator ruling 2026-07-30: `type: secret` travels (45 fields), `type: password` NEVER (7) plus a code register (`vaultwarden/ADMIN_TOKEN`); plaintext.** The exclusion is what LICENSES the plaintext — coupled, not independent. `stacks.PortableSecretEnvVars` is the single boundary; the register is **code, not a catalog flag** (a boundary a catalog push can move is not a boundary — R-97a). **Precedence: the UNIT WINS** over the guest, because the unit's secrets were captured in the same run as the dumps beside them and therefore match the data being restored; pinned both directions. **Fail-closed data-key gate UNCHANGED.** Manifest → schema 2 + `portable_secret_env_vars` (names only); schema-1 units still restore from the guest. **Live proof** on a scratch drill guest through the real endpoints: AdventureLog restored with the guest `app.yaml` moved aside → `secrets recovered=2/2`, 27.6 s, then **the app read the seeded row over TCP with its own credential** (the observable that matters), pre-backup row back / post-backup row gone, **no `.sql` dump** so the DB came from the volume tar. Withheld half proven with Grafana: sentinel live in the container, `ENC:` in the guest, **0 files** under the whole backup namespace. 4 red-proofs, each verified to land. `audits/D5-drive-alone-restore-2026-07-30.md` Flips `07` §3, §7.1, §7.3, §7.4 (new), §8 rows 3/3c/13, §10.1; new capability-map row. **Consequence recorded, not changed:** the unit already travels to Tier-2 (another customer drive, plaintext, same reasoning) and offsite via restic (encrypted at rest under the customer-owned repo password) — no tier code touched | — |
|
||||
| **R-127** | **The catalog's `data_key: true` flag is UNRELIABLE — at least four data-encrypting keys the catalog itself labels as encryption keys are unflagged; and the O4 restore path can regenerate a DB password that then does not match the restored data directory** | **READY (S/M)** | — | **Found by D5's Part 0, and it is why D5's boundary is `type: secret` rather than `data_key`.** Two separable legs. **(a) The misclassification.** Only 5 fields across 4 apps set `data_key: true` (`adventurelog/SECRET_KEY`, `homebox/HBOX_AUTH_API_KEY_PEPPER`, `papra/AUTH_SECRET`, `sparkyfitness/{API_ENCRYPTION_KEY,BETTER_AUTH_SECRET}`), yet `n8n/N8N_ENCRYPTION_KEY` („Titkosítási kulcs"), `wanderer/POCKETBASE_ENCRYPTION_KEY` („Adatbázis titkosítási kulcs"), `calcom/CALENDSO_ENCRYPTION_KEY` and `bookstack/APP_KEY` are unflagged — the catalog's own Hungarian labels contradict the flag. **D5 makes this non-urgent but not harmless:** everything `type: secret` now travels, so the keys DO reach the drive; what stays wrong is the **fail-closed gate**, which only refuses for `data_key` names — so if one of these is missing from both sources the restore proceeds onto data it cannot decrypt instead of refusing. Fix = flag them (`app-catalog-felhom.eu`, a catalog-only change) + a gate/test that the flag set and the label set agree. **(b) The regenerated-DB-password trap.** `internal/backup/restore_unit.go` O4 generates a replacement for any missing non-data-key secret. Proven on `postgres:16-alpine`: with PGDATA restored from the volume tar, `POSTGRES_PASSWORD` is ignored (initdb skipped), so the app fails over the compose network while the dump replay still succeeds through the container's local **trust** socket — success reported, data unreachable. v0.188.0 corrected the WARN's false claim that *"stored data is unaffected"* and scoped it, but did **not** add a guard: D5 shrinks this to the rare case (the secret was empty at capture AND absent from the guest). Real fix = either treat a DB password as fail-closed like a data key, or `ALTER USER` to the regenerated value after the volume restore. 18 DB/root-password fields are in scope; MariaDB fails loudly instead (Access denied), which is the safer half | CC |
|
||||
| **R-126** | **A `.fab` bundle — plaintext secrets, OPTIONAL password — can be exported ONTO a NAS.** `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | READY (S) | — | Split out of R-108, which closed without it: this is an explicit customer-chosen **export destination**, not a browsing surface reaching a backup tree, so it was never part of D5's precondition (`07` §7.3 records that reasoning). Was recorded inside R-108's row as its "second effect, independent of D5"; promoted to its own row so it does not vanish with R-108's closure. Fix = filter network paths out of the export destination list, or require the bundle password when the destination is a share | CC |
|
||||
| **F-DIAG** | ~~Four distinct offsite failure causes collapse into two operator-visible strings~~ | **SHIPPED** (controller v0.182.0, 2026-07-28) | — | `ClassifyOffsiteFailure` → quota / orphaned / no_repo / no_units / transport / **unknown**, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. **Secrets:** the old message was a raw `err.Error()` passthrough carrying `sftp:<user>@<host>:<path>`; redaction is now by the target's **actual** host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; **not** yet exercised by a live offsite failure of each class | — |
|
||||
| **F-OPS** | ~~A manual `pct restore` inherits the source guest's bind mounts — during a real DR, on a different host, under pressure~~ | **DOCUMENTED** (2026-07-28) | — | `documentation/runbooks/RUNBOOK-manual-guest-restore.md`: which `mpN` are volumes vs host binds, the `mp9` source-VMID trap (it can bind **another guest's bootstrap credentials**), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift | — |
|
||||
|
||||
@@ -109,6 +109,8 @@
|
||||
| R-106 | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | XS | **SHIPPED** — agent v0.118.1, 2026-07-30 | Namespace resolves from the pbs STORAGE (storage.cfg `namespace`) — the field `vzdump --storage <pbs>` makes PVE read — not from the listed snapshot, whose `ns` PBS omits once the list is `?ns=`-scoped. `namespace_state` distinguishes a genuine root namespace from not knowing. Live on both boxes. Follow-on: **R-124** (the `"root"` spelling is not `--ns`-pasteable). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 |
|
||||
| R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 |
|
||||
| R-108 | ~~Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5~~ | M | **SHIPPED** — controller v0.187.0, 2026-07-30 | **D5 UNBLOCKED.** One fail-closed predicate (`Settings.RefuseAsAppNamespace`) consulted by every placement surface, so no `backups/` tree can exist inside the share-root bind. The bind was deliberately NOT narrowed — it is load-bearing (`:rslave` automount wake, 2026-07-22 probe) and unscopable (apps on a share store at `<share>/<app>`; creating a `userdata/` layer would write Felhom convention onto a customer's NAS). **Five surfaces, not the four this row named** — decommission-with-migrate guarded only its SOURCE; and the boundary is the deploy POST, not the dropdown. Zero apps on network storage fleet-wide, so nothing stranded. The `.fab`-onto-NAS "second effect" is split out as **R-126**. Flipped: `07` §7.3 (D5 → unblocked), §10.1 (CLOSED), `00-capability-map.md` §D (new PROVEN-LIVE row). `audits/R108-network-app-namespace-2026-07-30.md` |
|
||||
| R-127 | **`data_key: true` is unreliable (4+ encryption keys unflagged, contradicting the catalog's own labels), and O4 can regenerate a DB password that no longer matches the restored data directory** | S/M | READY — NEW 2026-07-30 | Found by **D5's Part 0**, and the reason D5's boundary became `type: secret` rather than `data_key`. Leg (a): flag the missing keys (catalog-only) + pin flag-vs-label agreement; the residual risk after D5 is that the **fail-closed gate** keys on `data_key`, so an unflagged key missing from both sources lets the restore proceed onto undecryptable data. Leg (b): a regenerated DB password is silently wrong — `POSTGRES_PASSWORD` is ignored once PGDATA is non-empty, so the app cannot authenticate while the replay still succeeds over the local trust socket (proven live on `postgres:16-alpine`). v0.188.0 corrected the false "stored data is unaffected" WARN but added no guard. Flips: `07` §7.4 |
|
||||
| D5 | ~~**Move app secrets into the LOCAL recovery unit** so Tier-1/Tier-2 restore stop needing the guest and stop needing R~~ | M | **SHIPPED + PROVEN-LIVE** — controller v0.188.0, 2026-07-30 | **The arc's architectural centrepiece. Tier-1/2 no longer depend on the whole-guest tier — a customer needs the DRIVE AND NOTHING ELSE.** Part 0 tested this row's own premise and **rejected** it: data-keys-only is both insufficient and unsafe, because `data_key` is unreliable (→ **R-127**) and a DB password is not resettable in practice (`POSTGRES_PASSWORD` is ignored once PGDATA is non-empty, so a regenerated value leaves the app unable to reach its own restored rows while the dump replay still reports success — proven on `postgres:16-alpine`). **Operator ruling: `type: secret` travels (45 fields), `type: password` never (7) plus a code register (`vaultwarden/ADMIN_TOKEN`); plaintext, because withholding the internet-reachable class is what licenses it — the two are coupled.** `stacks.PortableSecretEnvVars` is the single boundary; the register is code, not a catalog flag (R-97a). **Precedence: the UNIT WINS** (its secrets match the data being restored, not merely the newest), pinned both directions. Fail-closed data-key gate UNCHANGED. Manifest schema 2; schema-1 units still restore. Proven live on a scratch drill guest: AdventureLog restored with the guest `app.yaml` moved aside (`secrets recovered=2/2`, 27.6 s) and **the app read the seeded row over TCP with its own credential**; Grafana's admin password withheld with **0 hits** across the backup namespace. 4 red-proofs each verified to land. `audits/D5-drive-alone-restore-2026-07-30.md` Flips `07` §3/§7.1/§7.3/§7.4/§8/§10.1 + a new capability-map row |
|
||||
| R-126 | **A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS.** `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | S | READY — 2026-07-30 | Split out of R-108 on its closure. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (`07` §7.3 records the reasoning). Fix = filter network paths from the export destination list, or force the bundle password when the destination is a share. Flips: `07` §5 |
|
||||
| R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.1 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 |
|
||||
| E-2 | **Drive-role machinery around the moved vzdump target.** The 2026-07-28 runbook proved the architecture change by hand on both demo boxes; this is the machinery: a **backup-target role** on `StoragePath` beside `Schedulable`/`IsDefault`/`Kind`; **assignment in the storage wizard** (suggest by attribute, refuse the absurd, never decide by transport or `removable` — on the reference hardware demo-felhom's target IS a USB HDD and BOTH drives report `removable=0`); **unassigned drives do nothing automatically**; **stickiness** (never silently retarget); `felhom-host-install.sh` creating the target with `--is_mountpoint 1` **and** issuing the `FelhomAgentStore` ACL; **absent-target policy**; **retention/space accounting** on a drive the customer shares; the honest **single-drive label**; remaining fleet migration | M | READY — 2026-07-28 | Full scope + rationale in `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Two traps already paid for live: the storage `path` must BE the mountpoint or the agent reports the target `disconnected` forever (`internal/storage/observe.go:321`), and the per-storage `FelhomAgentStore` grant is mandatory or every backup 403s. Absent-drive behaviour today is **fail-loudly, no silent retarget** (`is_mountpoint 1` proven live) — which is NOT the intended fall-back-and-alarm design. Flips: matrix row 4 |
|
||||
|
||||
Reference in New Issue
Block a user