felhom-controller/REPORT.md

# REPORT — D5: an app restore works from the drive alone (controller v0.188.0, 2026-07-30)

**Overwritten** per the standing rule. Controller **v0.187.0 → v0.188.0**, MinAgent 0.113.0 (unchanged).

**Outcome: SHIPPED + PROVEN-LIVE.** Tier-1/Tier-2 no longer depend on the whole-guest tier.
Commit `4ed938c`; image `gitea.dooplex.hu/admin/felhom-controller:0.188.0`
(digest `sha256:f297fbf0…`). Full record:
`felhom.eu/documentation/audits/D5-drive-alone-restore-2026-07-30.md`.

## 1. Baselines

| Repo | State at start |
|---|---|
| `felhom-controller` | clean, `HEAD == origin/main` at `2f27a36` (v0.187.0, R-108) |
| `app-catalog-felhom.eu` | `4252121`, 53 templates |
| Golden-baked controller | **0.186.0** — a release behind the fleet's 0.187.0 |
| R-120 vouch gate | **did NOT block.** It lives in `hub/internal/web/configs.go` `handleSetArtifacts` — the golden **vouch** form — and never runs on a controller image deploy. Re-baking is a **follow-on so fresh installs get D5**, not a prerequisite; it is its own ceremony, and bundling it here would have put two unrelated risky ceremonies in one session |

## 2. The §1.1 / §1.2 rulings — the brief's recommendation was tested and rejected

Both needed the operator; both were asked before any code was written.

**§1.1 — what travels.** The brief proposed `data_key`-only. Two findings killed it:

1. **The flag is unreliable → R-127.** Only 5 fields across 4 apps set `data_key: true`, yet
   `n8n/N8N_ENCRYPTION_KEY` („Titkosítási kulcs"), `wanderer/POCKETBASE_ENCRYPTION_KEY` („Adatbázis
   titkosítási kulcs"), `calcom/CALENDSO_ENCRYPTION_KEY` and `bookstack/APP_KEY` are unflagged — the
   catalog's own labels contradict the flag. Data-keys-only would omit real data keys, and the
   fail-closed gate would not fire for them.
2. **A DB password is not resettable in practice — proven on `postgres:16-alpine`.** With PGDATA
   restored from the volume tar, `POSTGRES_PASSWORD` is **ignored** (initdb skipped): the replay
   **succeeds** via the container's local `trust` socket, the app **fails** over the compose network
   (`FATAL: password authentication failed`), and the **old** password still works. Success reported,
   data unreachable. 18 DB/root-password fields; MariaDB fails louder (Access denied). *(A first
   attempt tested `-h 127.0.0.1` and got a false pass — the image trusts localhost. Recorded because it
   is the same class of false observable this task exists to catch.)*

**Ruling (operator):** `type: secret` travels (45 fields); `type: password` never (7) plus the
`nonPortableSecrets` code register (`vaultwarden/ADMIN_TOKEN`).

**§1.2 — plaintext**, as the data already is. Defensible on D2's own logic *because* the
internet-reachable class is withheld: every travelling secret decrypts data on the same drive or
authenticates to a container on an internal compose network with no external listener. **The exclusion
licenses the plaintext — the two are coupled and must not be relaxed independently.**

## 3. What a customer must possess to complete a Tier-1/2 restore

**The drive.** Nothing else.

## 4. Files modified — commit `4ed938c`, version 0.188.0

| File | Change |
|---|---|
| `internal/stacks/deploy.go` | **`PortableSecretEnvVars`** + the `nonPortableSecrets` register — the single boundary, in code not catalog (R-97a) |
| `cmd/controller/main.go` | `GetStackRecoveryInfo` decrypts the portable class via the **same** `LoadAppConfigDecrypted` the restore side uses |
| `internal/appbackup/appdata.go` | `RecoveryInfo` gains `PortableSecretEnvVars` + `PortableSecrets`; stale "SECRET-FREE" comment corrected |
| `internal/backup/recovery_unit.go` | `buildStrippedAppYaml` → **`buildUnitAppYaml`** (0600, names the withheld class in its header); manifest **schema 2** + `portable_secret_env_vars` |
| `internal/backup/restore_unit.go` | `readStrippedEnv` → **`readUnitEnv`** (manifest-driven split); `reconcileRestoreSecrets` takes `unitSecrets` as an **argument** and stays pure; **precedence documented** |
| `internal/appbackup/paths.go` | stale "the unit is secret-free" corrected |
| `CHANGELOG.md`, `REUSE.md`, `controller/README.md` | updated same-commit |

**Precedence: the UNIT WINS** — its secrets were captured in the same run as the dumps beside them, so
they match *the data being restored*; the guest's is merely newest. **The fail-closed gate is
unchanged.** Schema-1 units still restore from the guest.

Three comments asserting invariants D5 makes false were corrected rather than left reading as settled,
and the O4 WARN that claimed *"stored data is unaffected"* for every non-data-key secret was scoped to
what §2 shows is true.

## 5. Tests — suite green, `rc=0` (read on its own; never combined with a commit)

| Test | Asserts |
|---|---|
| `TestRestoreFromRecoveryUnitWithGuestAbsent` | **the consequence** — restore SUCCEEDS with the guest absent |
| `TestRestoreFromRecoveryUnitGuestAbsentStillFailsClosed` | data key in neither source ⇒ refuse; nothing mutated, app not even stopped |
| `TestReconcileRestoreSecretsPrecedence` | unit-wins; guest fills in when unit is silent/empty; no shadowing by stray plain config |
| `TestCaptureRecoveryUnitCarriesPortableSecretsOnly` | portable present, 0600, schema 2, **and the withheld value appears NOWHERE** |
| `TestReadUnitEnvSplitsByManifest` | manifest-driven split; schema-1 degrades losslessly |
| `TestPortableSecretEnvVars` | the boundary both directions; register is slug-scoped |
| `…Orchestration/schema-1 unit still restores` | no regression for pre-D5 units |

**Seam (R-125): `Manager.stackProvider` only** — docker/compose ops + the guest app.yaml decrypt.
`RecoverStackSecrets` returning nil **is** the guest being gone (what the real adapter returns when the
stack or its app.yaml can't be read). Fixtures come from a unit written by the **real**
`CaptureRecoveryUnit`, so capture and restore meet at real bytes rather than a test agreeing with itself.

**Four red-proofs, each mutation verified present in the working file before the run, then reverted:**

| # | Mutation | Result |
|---|---|---|
| 1 | drop the `PortableSecrets` merge | consequence + capture tests FAIL |
| 2 | `if false &&` the fail-closed gate | **4** failures |
| 3 | flip precedence to guest-first | unit-wins test FAILS (`got "guest-key"`) |
| 4 | widen the class to `type: password` | boundary test FAILS |

## 6. The live restore — with the guest's `app.yaml` unavailable

**Venue: a scratch drill LXC (9301) on felhom-pve**, destroyed after. Correct class per
`target-selection.md`: D5's claim is about **restore code**, not the install path/golden/first-boot, so
**fixture, not fresh day-0**. demo-hp could not host it — its `local` storage has no `rootdir` content
type and `local-lvm` was fenced by the task. Guest 9201 (both hosts) and `drill-r50` untouched.

Real endpoints only: `POST /api/stacks/{app}/deploy` → `POST /api/backup/run` → `POST /backup/restore`.

**Portable half — AdventureLog (`SECRET_KEY` data_key + `DB_PASSWORD`):**

    Recovery unit captured … (data_keys=1, portable-carried=2/2, withheld=0)
    Restoring adventurelog from recovery unit: secrets recovered=2/2, data_keys=1
    Restore-from-unit completed: adventurelog            (27.6 s)

`2/2` with the guest's `app.yaml` moved aside. Pre-D5 that was `0/2` and the gate refused.

**The data is readable — the observable that matters:**

    APP-READ-OK rows= ['D5-CUSTOMER-DATA-2026-07-30']
    connected_as= adventurelog  over_TCP= True  row_count= 1

Read through **Django's own connection, as the app, over TCP** — not the localhost trust socket that
produced §2's false pass. Plus `manage.py check --database default` clean and 51 `public` tables intact.

**The discriminator held:** the row seeded *before* the backup returned; a row inserted *after* it was
**gone** — so the tar was genuinely restored, not left alone. The unit held **no `.sql` dump**, so the DB
came back from the **volume tar** — exactly the case a regenerated password breaks silently.

**Withheld half — Grafana (`GF_SECURITY_ADMIN_PASSWORD`, `type: password`):** `withheld=1,
portable-carried=0/0`. A sentinel value was live in the container and `ENC:` in the guest — so the
absence is not vacuous — and appeared in **0 files** in the unit and **0 files** anywhere under the
backup namespace. The unit's app.yaml header names it as withheld.

**Not exercised live:** O4 regeneration of a withheld secret on restore, and Tier-2's cross-drive copy of
a secret-bearing unit (both unit-tested only).

## 7. Does Tier-1/2 still depend on the whole-guest tier?

**No — for the app-rebuild path.** One leg remains, named rather than glossed: **Tier-2's additive
file-merge** still reads its destination from the guest's `settings.json`
(`tier2_restore.go:114-116`), and Tier-3's reconstitution still requires the app deployed. **D5 removed
the secrets leg, not the living-app leg.**

## 8. Four-artifact update — confirmed (felhom.eu `0a9bd38`)

`architecture/07-backup-architecture.md` (owning doc: new **§7.4**, §7.1 leg 1 superseded, §7.3 → history
with the rejected-premise correction, §3, matrix rows **3 / 3c (new) / 13**, §10.1) ·
`00-capability-map.md` (new PROVEN-LIVE row) · `ROADMAP.md` (D5 collapsed, R-127 filed) ·
`OPEN-ITEMS.md` (D5 collapsed, R-127 filed) · plus the audit.

## 9. Arc status — the definition of done IS met

**R-106**, **R-109** (agent v0.118.1 / hub v0.83.0), **R-108** (v0.187.0) and **D5** (v0.188.0) are all
SHIPPED + PROVEN-LIVE.

Open, none part of that definition: **R-105** (three hub-held DR records empty fleet-wide), **R-121**
(installed-vs-vouched agent drift), **R-123** (PARTLY CLOSED), **R-124** (PBS root namespace `"root"` vs
`""`), **R-126** (`.fab` export onto NAS), **R-117g** (no guided recovery for an aborted fs),
**R-117h** (`StablePathForRaw` hardcodes the live parent), **R-127** (new).
**R-119 does not exist** in either backlog file — reported absent rather than given an invented status.

## 10. Teardown — all three layers

| Layer | Disposition |
|---|---|
| Guest | LXC **9301 destroyed** (`--purge`), `vm-9301-disk-0` removed; `pct list` shows only 9201 |
| Host | no leftover `/tmp` files; thin pool back to **27.88 %**, identical to pre-drill |
| Hub | **nothing to dispose** — drill ran `hub.enabled: false`, never registered. Verified positively: `/hosts` + `/configs` loaded (9880 / 10160 bytes) with **0** `d5drill` occurrences |

Secret-bearing files **shredded** (`shred -u -z`) in the guest before the volume was destroyed: both
units' `compose/app.yaml`, both stack `app.yaml`s, `encryption.key`, the hidden app.yaml, every
transported helper — plus host tarballs and local scratchpad scripts.

## 11. Findings

- **R-127 (NEW, READY)** — filed, not fixed: (a) `data_key` unreliable; (b) O4 can regenerate a DB
  password that no longer matches the restored data directory. Both sit outside D5's ruling.
- D5's own defect surface — the false *"stored data is unaffected"* WARN and three stale
  invariant-asserting comments — was corrected in place, being the defect this task was inside.

## 12. What did not happen, and why

- **Golden re-bake** — not done. R-120's gate does not apply (§1); it is a follow-on for fresh installs,
  and its own ceremony.
- **A fresh day-0 box** — not used. `target-selection.md` makes fixture-class correct for a restore-code
  claim; a fresh box would also have installed the pre-D5 golden.
- **R-127 not fixed** — a catalog change plus a separate guard decision, outside the ruling.
- **Escrow / whole-guest / offsite / hub role** — untouched, as fenced. The offsite copy carries the
  secrets inside the unit it already pushed (restic-encrypted under the customer-owned password); that is
  a recorded consequence, not a change.
- **One self-inflicted slip:** an inspection script's redaction regex assumed 2-space YAML indentation
  while the unit's app.yaml uses 4, so the drill's two secret values printed to the session transcript
  once. Throwaway values, in a guest that no longer exists, in **no committed file** — but the guard was
  mine to get right. A redaction pattern is itself a claim and needs checking against the real format.