felhom-controller/REPORT.md

REPORT — D5: an app restore works from the drive alone (controller v0.188.0, 2026-07-30)

Overwritten per the standing rule. Controller v0.187.0 → v0.188.0, MinAgent 0.113.0 (unchanged).

Outcome: SHIPPED + PROVEN-LIVE. Tier-1/Tier-2 no longer depend on the whole-guest tier. Commit 4ed938c; image gitea.dooplex.hu/admin/felhom-controller:0.188.0 (digest sha256:f297fbf0…). Full record: felhom.eu/documentation/audits/D5-drive-alone-restore-2026-07-30.md.

1. Baselines

Repo	State at start
felhom-controller	clean, HEAD == origin/main at 2f27a36 (v0.187.0, R-108)
app-catalog-felhom.eu	4252121, 53 templates
Golden-baked controller	0.186.0 — a release behind the fleet's 0.187.0
R-120 vouch gate	did NOT block. It lives in hub/internal/web/configs.go handleSetArtifacts — the golden vouch form — and never runs on a controller image deploy. Re-baking is a follow-on so fresh installs get D5, not a prerequisite; it is its own ceremony, and bundling it here would have put two unrelated risky ceremonies in one session

2. The §1.1 / §1.2 rulings — the brief's recommendation was tested and rejected

Both needed the operator; both were asked before any code was written.

§1.1 — what travels. The brief proposed data_key-only. Two findings killed it:

1. The flag is unreliable → R-127. Only 5 fields across 4 apps set data_key: true, yet n8n/N8N_ENCRYPTION_KEY („Titkosítási kulcs"), wanderer/POCKETBASE_ENCRYPTION_KEY („Adatbázis titkosítási kulcs"), calcom/CALENDSO_ENCRYPTION_KEY and bookstack/APP_KEY are unflagged — the catalog's own labels contradict the flag. Data-keys-only would omit real data keys, and the fail-closed gate would not fire for them.
2. A DB password is not resettable in practice — proven on postgres:16-alpine. With PGDATA restored from the volume tar, POSTGRES_PASSWORD is ignored (initdb skipped): the replay succeeds via the container's local trust socket, the app fails over the compose network (FATAL: password authentication failed), and the old password still works. Success reported, data unreachable. 18 DB/root-password fields; MariaDB fails louder (Access denied). (A first attempt tested -h 127.0.0.1 and got a false pass — the image trusts localhost. Recorded because it is the same class of false observable this task exists to catch.)

Ruling (operator): type: secret travels (45 fields); type: password never (7) plus the nonPortableSecrets code register (vaultwarden/ADMIN_TOKEN).

§1.2 — plaintext, as the data already is. Defensible on D2's own logic because the internet-reachable class is withheld: every travelling secret decrypts data on the same drive or authenticates to a container on an internal compose network with no external listener. The exclusion licenses the plaintext — the two are coupled and must not be relaxed independently.

3. What a customer must possess to complete a Tier-1/2 restore

The drive. Nothing else.

4. Files modified — commit 4ed938c, version 0.188.0

File	Change
internal/stacks/deploy.go	PortableSecretEnvVars + the nonPortableSecrets register — the single boundary, in code not catalog (R-97a)
cmd/controller/main.go	GetStackRecoveryInfo decrypts the portable class via the same LoadAppConfigDecrypted the restore side uses
internal/appbackup/appdata.go	RecoveryInfo gains PortableSecretEnvVars + PortableSecrets; stale "SECRET-FREE" comment corrected
internal/backup/recovery_unit.go	buildStrippedAppYaml → buildUnitAppYaml (0600, names the withheld class in its header); manifest schema 2 + portable_secret_env_vars
internal/backup/restore_unit.go	readStrippedEnv → readUnitEnv (manifest-driven split); reconcileRestoreSecrets takes unitSecrets as an argument and stays pure; precedence documented
internal/appbackup/paths.go	stale "the unit is secret-free" corrected
CHANGELOG.md, REUSE.md, controller/README.md	updated same-commit

Precedence: the UNIT WINS — its secrets were captured in the same run as the dumps beside them, so they match the data being restored; the guest's is merely newest. The fail-closed gate is unchanged. Schema-1 units still restore from the guest.

Three comments asserting invariants D5 makes false were corrected rather than left reading as settled, and the O4 WARN that claimed "stored data is unaffected" for every non-data-key secret was scoped to what §2 shows is true.

5. Tests — suite green, rc=0 (read on its own; never combined with a commit)

Test	Asserts
TestRestoreFromRecoveryUnitWithGuestAbsent	the consequence — restore SUCCEEDS with the guest absent
TestRestoreFromRecoveryUnitGuestAbsentStillFailsClosed	data key in neither source ⇒ refuse; nothing mutated, app not even stopped
TestReconcileRestoreSecretsPrecedence	unit-wins; guest fills in when unit is silent/empty; no shadowing by stray plain config
TestCaptureRecoveryUnitCarriesPortableSecretsOnly	portable present, 0600, schema 2, and the withheld value appears NOWHERE
TestReadUnitEnvSplitsByManifest	manifest-driven split; schema-1 degrades losslessly
TestPortableSecretEnvVars	the boundary both directions; register is slug-scoped
…Orchestration/schema-1 unit still restores	no regression for pre-D5 units

Seam (R-125): Manager.stackProvider only — docker/compose ops + the guest app.yaml decrypt. RecoverStackSecrets returning nil is the guest being gone (what the real adapter returns when the stack or its app.yaml can't be read). Fixtures come from a unit written by the real CaptureRecoveryUnit, so capture and restore meet at real bytes rather than a test agreeing with itself.

Four red-proofs, each mutation verified present in the working file before the run, then reverted:

#	Mutation	Result
1	drop the PortableSecrets merge	consequence + capture tests FAIL
2	if false && the fail-closed gate	4 failures
3	flip precedence to guest-first	unit-wins test FAILS (got "guest-key")
4	widen the class to type: password	boundary test FAILS

6. The live restore — with the guest's app.yaml unavailable

Venue: a scratch drill LXC (9301) on felhom-pve, destroyed after. Correct class per target-selection.md: D5's claim is about restore code, not the install path/golden/first-boot, so fixture, not fresh day-0. demo-hp could not host it — its local storage has no rootdir content type and local-lvm was fenced by the task. Guest 9201 (both hosts) and drill-r50 untouched.

Real endpoints only: POST /api/stacks/{app}/deploy → POST /api/backup/run → POST /backup/restore.

Portable half — AdventureLog (SECRET_KEY data_key + DB_PASSWORD):

Recovery unit captured … (data_keys=1, portable-carried=2/2, withheld=0)
Restoring adventurelog from recovery unit: secrets recovered=2/2, data_keys=1
Restore-from-unit completed: adventurelog            (27.6 s)

2/2 with the guest's app.yaml moved aside. Pre-D5 that was 0/2 and the gate refused.

The data is readable — the observable that matters:

APP-READ-OK rows= ['D5-CUSTOMER-DATA-2026-07-30']
connected_as= adventurelog  over_TCP= True  row_count= 1

Read through Django's own connection, as the app, over TCP — not the localhost trust socket that produced §2's false pass. Plus manage.py check --database default clean and 51 public tables intact.

The discriminator held: the row seeded before the backup returned; a row inserted after it was gone — so the tar was genuinely restored, not left alone. The unit held no .sql dump, so the DB came back from the volume tar — exactly the case a regenerated password breaks silently.

Withheld half — Grafana (GF_SECURITY_ADMIN_PASSWORD, type: password): withheld=1, portable-carried=0/0. A sentinel value was live in the container and ENC: in the guest — so the absence is not vacuous — and appeared in 0 files in the unit and 0 files anywhere under the backup namespace. The unit's app.yaml header names it as withheld.

Not exercised live: O4 regeneration of a withheld secret on restore, and Tier-2's cross-drive copy of a secret-bearing unit (both unit-tested only).

7. Does Tier-1/2 still depend on the whole-guest tier?

No — for the app-rebuild path. One leg remains, named rather than glossed: Tier-2's additive file-merge still reads its destination from the guest's settings.json (tier2_restore.go:114-116), and Tier-3's reconstitution still requires the app deployed. D5 removed the secrets leg, not the living-app leg.

8. Four-artifact update — confirmed (felhom.eu 0a9bd38)

architecture/07-backup-architecture.md (owning doc: new §7.4, §7.1 leg 1 superseded, §7.3 → history with the rejected-premise correction, §3, matrix rows 3 / 3c (new) / 13, §10.1) · 00-capability-map.md (new PROVEN-LIVE row) · ROADMAP.md (D5 collapsed, R-127 filed) · OPEN-ITEMS.md (D5 collapsed, R-127 filed) · plus the audit.

9. Arc status — the definition of done IS met

R-106, R-109 (agent v0.118.1 / hub v0.83.0), R-108 (v0.187.0) and D5 (v0.188.0) are all SHIPPED + PROVEN-LIVE.

Open, none part of that definition: R-105 (three hub-held DR records empty fleet-wide), R-121 (installed-vs-vouched agent drift), R-123 (PARTLY CLOSED), R-124 (PBS root namespace "root" vs ""), R-126 (.fab export onto NAS), R-117g (no guided recovery for an aborted fs), R-117h (StablePathForRaw hardcodes the live parent), R-127 (new). R-119 does not exist in either backlog file — reported absent rather than given an invented status.

10. Teardown — all three layers

Layer	Disposition
Guest	LXC 9301 destroyed (--purge), vm-9301-disk-0 removed; pct list shows only 9201
Host	no leftover /tmp files; thin pool back to 27.88 %, identical to pre-drill
Hub	nothing to dispose — drill ran hub.enabled: false, never registered. Verified positively: /hosts + /configs loaded (9880 / 10160 bytes) with 0 d5drill occurrences

Secret-bearing files shredded (shred -u -z) in the guest before the volume was destroyed: both units' compose/app.yaml, both stack app.yamls, encryption.key, the hidden app.yaml, every transported helper — plus host tarballs and local scratchpad scripts.

11. Findings

• R-127 (NEW, READY) — filed, not fixed: (a) data_key unreliable; (b) O4 can regenerate a DB password that no longer matches the restored data directory. Both sit outside D5's ruling.
• D5's own defect surface — the false "stored data is unaffected" WARN and three stale invariant-asserting comments — was corrected in place, being the defect this task was inside.

12. What did not happen, and why

• Golden re-bake — not done. R-120's gate does not apply (§1); it is a follow-on for fresh installs, and its own ceremony.
• A fresh day-0 box — not used. target-selection.md makes fixture-class correct for a restore-code claim; a fresh box would also have installed the pre-D5 golden.
• R-127 not fixed — a catalog change plus a separate guard decision, outside the ruling.
• Escrow / whole-guest / offsite / hub role — untouched, as fenced. The offsite copy carries the secrets inside the unit it already pushed (restic-encrypted under the customer-owned password); that is a recorded consequence, not a change.
• One self-inflicted slip: an inspection script's redaction regex assumed 2-space YAML indentation while the unit's app.yaml uses 4, so the drill's two secret values printed to the session transcript once. Throwaway values, in a guest that no longer exists, in no committed file — but the guard was mine to get right. A redaction pattern is itself a claim and needs checking against the real format.