REPORT: D5 — restore from the drive alone (v0.188.0), proven live
This commit is contained in:
@@ -1,109 +1,190 @@
|
|||||||
# REPORT — R-108: network storage may not host an app's data namespace (v0.187.0, 2026-07-30)
|
# REPORT — D5: an app restore works from the drive alone (controller v0.188.0, 2026-07-30)
|
||||||
|
|
||||||
**Overwritten** per the standing rule. Controller **v0.186.0 → v0.187.0**, MinAgent 0.113.0 (unchanged).
|
**Overwritten** per the standing rule. Controller **v0.187.0 → v0.188.0**, MinAgent 0.113.0 (unchanged).
|
||||||
|
|
||||||
**The §7 sentence, up front: D5's precondition IS now met.** No app namespace can be placed on network
|
**Outcome: SHIPPED + PROVEN-LIVE.** Tier-1/Tier-2 no longer depend on the whole-guest tier.
|
||||||
storage, therefore no `backups/` tree can exist inside FileBrowser's share-root bind, therefore moving
|
Commit `4ed938c`; image `gitea.dooplex.hu/admin/felhom-controller:0.188.0`
|
||||||
app secrets into the local recovery unit no longer exposes them to a browsing surface. Nothing else was
|
(digest `sha256:f297fbf0…`). Full record:
|
||||||
blocking D5 — the §10.1 sweep had already cleared every other read surface.
|
`felhom.eu/documentation/audits/D5-drive-alone-restore-2026-07-30.md`.
|
||||||
|
|
||||||
## 1. The §3.1 ruling: REFUSE the placement, KEEP the browse bind
|
## 1. Baselines
|
||||||
|
|
||||||
The task offered "scope the bind" or "refuse". **Scoping is not implementable**, and establishing that
|
| Repo | State at start |
|
||||||
inverted the fix:
|
|---|---|
|
||||||
|
| `felhom-controller` | clean, `HEAD == origin/main` at `2f27a36` (v0.187.0, R-108) |
|
||||||
|
| `app-catalog-felhom.eu` | `4252121`, 53 templates |
|
||||||
|
| Golden-baked controller | **0.186.0** — a release behind the fleet's 0.187.0 |
|
||||||
|
| R-120 vouch gate | **did NOT block.** It lives in `hub/internal/web/configs.go` `handleSetArtifacts` — the golden **vouch** form — and never runs on a controller image deploy. Re-baking is a **follow-on so fresh installs get D5**, not a prerequisite; it is its own ceremony, and bundling it here would have put two unrelated risky ceremonies in one session |
|
||||||
|
|
||||||
- The share-root `:rslave` bind is **load-bearing, not an oversight**. A Phase-0 probe (2026-07-22,
|
## 2. The §1.1 / §1.2 rulings — the brief's recommendation was tested and rejected
|
||||||
recorded in the `buildFileBrowserPaths` doc comment) proved an in-container access through it wakes
|
|
||||||
the idle automount trigger. Narrow it and NAS access itself breaks.
|
|
||||||
- There is **no `userdata/` layer to scope to**. Apps on a share store at `<share>/<app>`
|
|
||||||
(`deployedAppsOnPath` doc comment, `internal/web/netstorage_handlers.go`), not under `<share>/userdata/`.
|
|
||||||
- Creating one is **forbidden**: *"Never a skeleton, never any write toward the NAS — Felhom conventions
|
|
||||||
must not be written onto a customer's own NAS."*
|
|
||||||
|
|
||||||
So the browsing surface cannot be narrowed, and the backup tree must never be placed under it. Tier 2
|
Both needed the operator; both were asked before any code was written.
|
||||||
already refuses network targets for this same class of reason (`F-6C-1`); R-108 closes the PRIMARY
|
|
||||||
namespace, the last route by which a `backups/` tree could land inside a share-root bind.
|
|
||||||
|
|
||||||
**Operator decision required and taken** (the ruling removes a capability going forward). Evidence
|
**§1.1 — what travels.** The brief proposed `data_key`-only. Two findings killed it:
|
||||||
supplied with the question: **zero apps on network storage across all six hub customers including Peti**
|
|
||||||
— nothing is stranded, and no migration is owed. R-67's browse capability is byte-identical after.
|
|
||||||
|
|
||||||
## 2. The exposure chain, confirmed at source
|
1. **The flag is unreliable → R-127.** Only 5 fields across 4 apps set `data_key: true`, yet
|
||||||
|
`n8n/N8N_ENCRYPTION_KEY` („Titkosítási kulcs"), `wanderer/POCKETBASE_ENCRYPTION_KEY` („Adatbázis
|
||||||
|
titkosítási kulcs"), `calcom/CALENDSO_ENCRYPTION_KEY` and `bookstack/APP_KEY` are unflagged — the
|
||||||
|
catalog's own labels contradict the flag. Data-keys-only would omit real data keys, and the
|
||||||
|
fail-closed gate would not fire for them.
|
||||||
|
2. **A DB password is not resettable in practice — proven on `postgres:16-alpine`.** With PGDATA
|
||||||
|
restored from the volume tar, `POSTGRES_PASSWORD` is **ignored** (initdb skipped): the replay
|
||||||
|
**succeeds** via the container's local `trust` socket, the app **fails** over the compose network
|
||||||
|
(`FATAL: password authentication failed`), and the **old** password still works. Success reported,
|
||||||
|
data unreachable. 18 DB/root-password fields; MariaDB fails louder (Access denied). *(A first
|
||||||
|
attempt tested `-h 127.0.0.1` and got a false pass — the image trusts localhost. Recorded because it
|
||||||
|
is the same class of false observable this task exists to catch.)*
|
||||||
|
|
||||||
`namespaceRoot(drivePath)` returns any non-system drive path AS-IS (`internal/backup/backup.go:262`), so
|
**Ruling (operator):** `type: secret` travels (45 fields); `type: password` never (7) plus the
|
||||||
an app's namespace root **is** its `HDD_PATH`; its recovery unit is
|
`nonPortableSecrets` code register (`vaultwarden/ADMIN_TOKEN`).
|
||||||
`<HDD_PATH>/backups/primary/<stack>/` (`internal/appbackup/paths.go`). On a NAS that sits inside the
|
|
||||||
share, which FileBrowser binds whole with `download: true`.
|
|
||||||
|
|
||||||
This was **documented, deliberate behaviour**, not an accident: `controller/README.md` carried *"NAS
|
**§1.2 — plaintext**, as the data already is. Defensible on D2's own logic *because* the
|
||||||
backup locality (v0.118.0, CAMPAIGN-3 Part 4 — decision A) — a NAS-resident app's tier-1 artifacts live
|
internet-reachable class is withheld: every travelling secret decrypts data on the same drive or
|
||||||
on the NAS itself"*. R-108 supersedes decision A; the README block now says so.
|
authenticates to a container on an internal compose network with no external listener. **The exclusion
|
||||||
|
licenses the plaintext — the two are coupled and must not be relaxed independently.**
|
||||||
|
|
||||||
## 3. §2 citation reconfirmation — all four hold, line numbers moved
|
## 3. What a customer must possess to complete a Tier-1/2 restore
|
||||||
|
|
||||||
| register citation | now at | verdict |
|
**The drive.** Nothing else.
|
||||||
|
|
||||||
|
## 4. Files modified — commit `4ed938c`, version 0.188.0
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|---|---|
|
||||||
|
| `internal/stacks/deploy.go` | **`PortableSecretEnvVars`** + the `nonPortableSecrets` register — the single boundary, in code not catalog (R-97a) |
|
||||||
|
| `cmd/controller/main.go` | `GetStackRecoveryInfo` decrypts the portable class via the **same** `LoadAppConfigDecrypted` the restore side uses |
|
||||||
|
| `internal/appbackup/appdata.go` | `RecoveryInfo` gains `PortableSecretEnvVars` + `PortableSecrets`; stale "SECRET-FREE" comment corrected |
|
||||||
|
| `internal/backup/recovery_unit.go` | `buildStrippedAppYaml` → **`buildUnitAppYaml`** (0600, names the withheld class in its header); manifest **schema 2** + `portable_secret_env_vars` |
|
||||||
|
| `internal/backup/restore_unit.go` | `readStrippedEnv` → **`readUnitEnv`** (manifest-driven split); `reconcileRestoreSecrets` takes `unitSecrets` as an **argument** and stays pure; **precedence documented** |
|
||||||
|
| `internal/appbackup/paths.go` | stale "the unit is secret-free" corrected |
|
||||||
|
| `CHANGELOG.md`, `REUSE.md`, `controller/README.md` | updated same-commit |
|
||||||
|
|
||||||
|
**Precedence: the UNIT WINS** — its secrets were captured in the same run as the dumps beside them, so
|
||||||
|
they match *the data being restored*; the guest's is merely newest. **The fail-closed gate is
|
||||||
|
unchanged.** Schema-1 units still restore from the guest.
|
||||||
|
|
||||||
|
Three comments asserting invariants D5 makes false were corrected rather than left reading as settled,
|
||||||
|
and the O4 WARN that claimed *"stored data is unaffected"* for every non-data-key secret was scoped to
|
||||||
|
what §2 shows is true.
|
||||||
|
|
||||||
|
## 5. Tests — suite green, `rc=0` (read on its own; never combined with a commit)
|
||||||
|
|
||||||
|
| Test | Asserts |
|
||||||
|
|---|---|
|
||||||
|
| `TestRestoreFromRecoveryUnitWithGuestAbsent` | **the consequence** — restore SUCCEEDS with the guest absent |
|
||||||
|
| `TestRestoreFromRecoveryUnitGuestAbsentStillFailsClosed` | data key in neither source ⇒ refuse; nothing mutated, app not even stopped |
|
||||||
|
| `TestReconcileRestoreSecretsPrecedence` | unit-wins; guest fills in when unit is silent/empty; no shadowing by stray plain config |
|
||||||
|
| `TestCaptureRecoveryUnitCarriesPortableSecretsOnly` | portable present, 0600, schema 2, **and the withheld value appears NOWHERE** |
|
||||||
|
| `TestReadUnitEnvSplitsByManifest` | manifest-driven split; schema-1 degrades losslessly |
|
||||||
|
| `TestPortableSecretEnvVars` | the boundary both directions; register is slug-scoped |
|
||||||
|
| `…Orchestration/schema-1 unit still restores` | no regression for pre-D5 units |
|
||||||
|
|
||||||
|
**Seam (R-125): `Manager.stackProvider` only** — docker/compose ops + the guest app.yaml decrypt.
|
||||||
|
`RecoverStackSecrets` returning nil **is** the guest being gone (what the real adapter returns when the
|
||||||
|
stack or its app.yaml can't be read). Fixtures come from a unit written by the **real**
|
||||||
|
`CaptureRecoveryUnit`, so capture and restore meet at real bytes rather than a test agreeing with itself.
|
||||||
|
|
||||||
|
**Four red-proofs, each mutation verified present in the working file before the run, then reverted:**
|
||||||
|
|
||||||
|
| # | Mutation | Result |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `handlers.go:2450-2460` local userdata-scoped | `:2464-2465` | **holds** |
|
| 1 | drop the `PortableSecrets` merge | consequence + capture tests FAIL |
|
||||||
| `handlers.go:2432` network share-ROOT + download | bind `:2437`, `download: true` at `infra/infra.go:328` | **holds** |
|
| 2 | `if false &&` the fail-closed gate | **4** failures |
|
||||||
| `settings.go:904-914` no `IsNetwork()` filter | `GetSchedulableStoragePaths`, `:926` | **holds** |
|
| 3 | flip precedence to guest-first | unit-wins test FAILS (`got "guest-key"`) |
|
||||||
| `handlers.go:674-679` migrate targets unfiltered | `:673-678` | **holds** |
|
| 4 | widen the class to `type: password` | boundary test FAILS |
|
||||||
| `storage_handlers.go:410-424` migrate-app unguarded | `:416-425` | **holds** |
|
|
||||||
| `storage_handlers.go:397` sibling refuses | `handleStorageMigrate`, `:404` (both source AND target) | **holds — the precedent** |
|
|
||||||
|
|
||||||
## 4. §3.2 — the set is FIVE, not four
|
## 6. The live restore — with the guest's `app.yaml` unavailable
|
||||||
|
|
||||||
Surface 4 below is **not in the register**; it was found by enumerating rather than trusting the named set.
|
**Venue: a scratch drill LXC (9301) on felhom-pve**, destroyed after. Correct class per
|
||||||
|
`target-selection.md`: D5's claim is about **restore code**, not the install path/golden/first-boot, so
|
||||||
|
**fixture, not fresh day-0**. demo-hp could not host it — its `local` storage has no `rootdir` content
|
||||||
|
type and `local-lvm` was fenced by the task. Guest 9201 (both hosts) and `drill-r50` untouched.
|
||||||
|
|
||||||
1. **deploy POST** (`internal/api/router.go`) — **the actual boundary.** The register's "the dropdown has
|
Real endpoints only: `POST /api/stacks/{app}/deploy` → `POST /api/backup/run` → `POST /backup/restore`.
|
||||||
no filter" understates it: the dropdown is a UI list, this endpoint takes any caller-supplied
|
|
||||||
`HDD_PATH`, and `DeployStack` validates only `os.Stat` existence. A list filter alone would have left
|
|
||||||
the surface open.
|
|
||||||
2. **per-app migrate target list** (`internal/web/handlers.go`) — dropped from the offer.
|
|
||||||
3. **`handleStorageMigrateApp`** — refused before `MigrateApp`.
|
|
||||||
4. **`handleStorageDecommission` mode=migrate TARGET** — `refuseNetworkLifecycle` guarded only
|
|
||||||
`req.Where` (the SOURCE); a whole namespace could be decommissioned ONTO a NAS. **NEW.**
|
|
||||||
5. **the FileBrowser bind** — deliberately unchanged, now pinned by a regression test.
|
|
||||||
|
|
||||||
Also noted, not fixed (register already records it as a separate effect): `storageDriveList()`
|
**Portable half — AdventureLog (`SECRET_KEY` data_key + `DB_PASSWORD`):**
|
||||||
(`handler_export.go`) does not filter network paths, so a `.fab` bundle — plaintext secrets, optional
|
|
||||||
password — can still be **exported onto** a NAS. That is a bundle destination, not a namespace
|
|
||||||
placement, and it is outside R-108's scope. Filed as **R-126**.
|
|
||||||
|
|
||||||
## 5. Fail-closed, and why it needed a function
|
Recovery unit captured … (data_keys=1, portable-carried=2/2, withheld=0)
|
||||||
|
Restoring adventurelog from recovery unit: secrets recovered=2/2, data_keys=1
|
||||||
|
Restore-from-unit completed: adventurelog (27.6 s)
|
||||||
|
|
||||||
`/mnt/felhom-drives` holds **both** kinds in-guest (`.../nvme-1tb` is a drive, `.../Felhom-Share` is a
|
`2/2` with the guest's `app.yaml` moved aside. Pre-D5 that was `0/2` and the gate refused.
|
||||||
NAS), so a path prefix cannot classify. `Kind` is the only discriminator and exists only on a REGISTERED
|
|
||||||
path — therefore an unregistered path under that root is un-classifiable, and un-classifiable **refuses**.
|
|
||||||
Every share is registered under that root by construction, so the network set is fully covered without
|
|
||||||
touching drives. Empty `HDD_PATH` (SSD-resident) stays allowed; a nil registry refuses.
|
|
||||||
|
|
||||||
## 6. Tests — 9 new, all non-effect; seams named (R-125)
|
**The data is readable — the observable that matters:**
|
||||||
|
|
||||||
Refusal tests run against a Server with a deliberately **nil `stackMgr`**: a guard that fails to fire
|
APP-READ-OK rows= ['D5-CUSTOMER-DATA-2026-07-30']
|
||||||
reaches the mutation and **panics** rather than passing quietly. Asserted: no job id, no `started` flag,
|
connected_as= adventurelog over_TCP= True row_count= 1
|
||||||
no `MigratedTo` written, and for decommission that the source was **not** soft-marked. Fixtures are
|
|
||||||
demo-hp's real two-class storage set.
|
|
||||||
|
|
||||||
**Seams injected:** the FileBrowser tests inject at `fbPathDeps` (`isMount`/`classify`/`ensureSkeleton`)
|
Read through **Django's own connection, as the app, over TCP** — not the localhost trust socket that
|
||||||
and run the real `buildFileBrowserPaths`; the bind-string construction — what the assertion is about —
|
produced §2's false pass. Plus `manage.py check --database default` clean and 51 `public` tables intact.
|
||||||
is NOT injected. **Not covered by that span:** `RenderFileBrowserConfig` and the compose template
|
|
||||||
downstream; closed by reading the live generated compose (§7). The handler tests inject nothing — real
|
|
||||||
handler, real settings store, `httptest`.
|
|
||||||
|
|
||||||
**Red-proofs: 4**, each mutation asserted to have landed before the run: drop the migrate-app guard →
|
**The discriminator held:** the row seeded *before* the backup returned; a row inserted *after* it was
|
||||||
panic; drop the decommission-target guard → panic; break fail-closed → 3 tests; userdata-scope the share
|
**gone** — so the tar was genuinely restored, not left alone. The unit held **no `.sql` dump**, so the DB
|
||||||
→ the R-67 regression guard fires quoting the broken bind.
|
came back from the **volume tar** — exactly the case a regenerated password breaks silently.
|
||||||
|
|
||||||
Suite rc=0, 27 packages, 0 FAIL (run separately from the commit); `go vet` rc=0;
|
**Withheld half — Grafana (`GF_SECURITY_ADMIN_PASSWORD`, `type: password`):** `withheld=1,
|
||||||
`template_id_gate.py` + `emoji_gate.py` OK.
|
portable-carried=0/0`. A sentinel value was live in the container and `ENC:` in the guest — so the
|
||||||
|
absence is not vacuous — and appeared in **0 files** in the unit and **0 files** anywhere under the
|
||||||
|
backup namespace. The unit's app.yaml header names it as withheld.
|
||||||
|
|
||||||
## 7. Live evidence
|
**Not exercised live:** O4 regeneration of a withheld secret on restore, and Tier-2's cross-drive copy of
|
||||||
|
a secret-bearing unit (both unit-tested only).
|
||||||
|
|
||||||
See `felhom.eu/documentation/audits/R108-network-app-namespace-2026-07-30.md`.
|
## 7. Does Tier-1/2 still depend on the whole-guest tier?
|
||||||
|
|
||||||
## 8. Not done
|
**No — for the app-rebuild path.** One leg remains, named rather than glossed: **Tier-2's additive
|
||||||
|
file-merge** still reads its destination from the guest's `settings.json`
|
||||||
|
(`tier2_restore.go:114-116`), and Tier-3's reconstitution still requires the app deployed. **D5 removed
|
||||||
|
the secrets leg, not the living-app leg.**
|
||||||
|
|
||||||
- **D5 not implemented** — this unblocks it only.
|
## 8. Four-artifact update — confirmed (felhom.eu `0a9bd38`)
|
||||||
- **No data migrated** — nothing needed migrating (zero apps on network storage).
|
|
||||||
- **The share-root bind unchanged** — deliberate; R-67's capability is preserved intact.
|
`architecture/07-backup-architecture.md` (owning doc: new **§7.4**, §7.1 leg 1 superseded, §7.3 → history
|
||||||
- **R-126 filed, not fixed** (`.fab` export onto a NAS).
|
with the rejected-premise correction, §3, matrix rows **3 / 3c (new) / 13**, §10.1) ·
|
||||||
|
`00-capability-map.md` (new PROVEN-LIVE row) · `ROADMAP.md` (D5 collapsed, R-127 filed) ·
|
||||||
|
`OPEN-ITEMS.md` (D5 collapsed, R-127 filed) · plus the audit.
|
||||||
|
|
||||||
|
## 9. Arc status — the definition of done IS met
|
||||||
|
|
||||||
|
**R-106**, **R-109** (agent v0.118.1 / hub v0.83.0), **R-108** (v0.187.0) and **D5** (v0.188.0) are all
|
||||||
|
SHIPPED + PROVEN-LIVE.
|
||||||
|
|
||||||
|
Open, none part of that definition: **R-105** (three hub-held DR records empty fleet-wide), **R-121**
|
||||||
|
(installed-vs-vouched agent drift), **R-123** (PARTLY CLOSED), **R-124** (PBS root namespace `"root"` vs
|
||||||
|
`""`), **R-126** (`.fab` export onto NAS), **R-117g** (no guided recovery for an aborted fs),
|
||||||
|
**R-117h** (`StablePathForRaw` hardcodes the live parent), **R-127** (new).
|
||||||
|
**R-119 does not exist** in either backlog file — reported absent rather than given an invented status.
|
||||||
|
|
||||||
|
## 10. Teardown — all three layers
|
||||||
|
|
||||||
|
| Layer | Disposition |
|
||||||
|
|---|---|
|
||||||
|
| Guest | LXC **9301 destroyed** (`--purge`), `vm-9301-disk-0` removed; `pct list` shows only 9201 |
|
||||||
|
| Host | no leftover `/tmp` files; thin pool back to **27.88 %**, identical to pre-drill |
|
||||||
|
| Hub | **nothing to dispose** — drill ran `hub.enabled: false`, never registered. Verified positively: `/hosts` + `/configs` loaded (9880 / 10160 bytes) with **0** `d5drill` occurrences |
|
||||||
|
|
||||||
|
Secret-bearing files **shredded** (`shred -u -z`) in the guest before the volume was destroyed: both
|
||||||
|
units' `compose/app.yaml`, both stack `app.yaml`s, `encryption.key`, the hidden app.yaml, every
|
||||||
|
transported helper — plus host tarballs and local scratchpad scripts.
|
||||||
|
|
||||||
|
## 11. Findings
|
||||||
|
|
||||||
|
- **R-127 (NEW, READY)** — filed, not fixed: (a) `data_key` unreliable; (b) O4 can regenerate a DB
|
||||||
|
password that no longer matches the restored data directory. Both sit outside D5's ruling.
|
||||||
|
- D5's own defect surface — the false *"stored data is unaffected"* WARN and three stale
|
||||||
|
invariant-asserting comments — was corrected in place, being the defect this task was inside.
|
||||||
|
|
||||||
|
## 12. What did not happen, and why
|
||||||
|
|
||||||
|
- **Golden re-bake** — not done. R-120's gate does not apply (§1); it is a follow-on for fresh installs,
|
||||||
|
and its own ceremony.
|
||||||
|
- **A fresh day-0 box** — not used. `target-selection.md` makes fixture-class correct for a restore-code
|
||||||
|
claim; a fresh box would also have installed the pre-D5 golden.
|
||||||
|
- **R-127 not fixed** — a catalog change plus a separate guard decision, outside the ruling.
|
||||||
|
- **Escrow / whole-guest / offsite / hub role** — untouched, as fenced. The offsite copy carries the
|
||||||
|
secrets inside the unit it already pushed (restic-encrypted under the customer-owned password); that is
|
||||||
|
a recorded consequence, not a change.
|
||||||
|
- **One self-inflicted slip:** an inspection script's redaction regex assumed 2-space YAML indentation
|
||||||
|
while the unit's app.yaml uses 4, so the drill's two secret values printed to the session transcript
|
||||||
|
once. Throwaway values, in a guest that no longer exists, in **no committed file** — but the guard was
|
||||||
|
mine to get right. A redaction pattern is itself a claim and needs checking against the real format.
|
||||||
|
|||||||
Reference in New Issue
Block a user