From e0618af438dce22a4f2dd2c0e63633509bea458d Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 13 Jul 2026 10:58:19 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20v0.125.0=20path-strand=20fix=20?= =?UTF-8?q?=E2=80=94=20CHANGELOG=20+=20REPORT=20(probe=20transcript,=20red?= =?UTF-8?q?-proofs,=20round-trip=20evidence,=20Tier-1/2=20mechanism)=20+?= =?UTF-8?q?=20CONTEXT=20+=20REUSE=20+=20README=20flows;=20hollow-bundle=20?= =?UTF-8?q?asymmetry=20recorded=20(containerized=20=E2=89=A40.124.0=20expo?= =?UTF-8?q?rts=20suspect=20=E2=80=94=20re-export);=20floor=20note=20for=20?= =?UTF-8?q?the=20next=20train?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_01GzammAMzsJTgpQHqxwM2bC --- CHANGELOG.md | 39 +++++++++++++ CONTEXT.md | 14 ++++- REPORT.md | 134 +++++++++++++++++++------------------------ REUSE.md | 1 + controller/README.md | 4 +- 5 files changed, 115 insertions(+), 77 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ee38106..c04b14e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,44 @@ ## Changelog +### v0.125.0 — .fab volume export/import: containerized path-strand data loss FIXED (IA finding 1, HIGH) (2026-07-13) — MinAgent: 0.81.0 + +Both .fab volume legs streamed via `docker run -v ` host mounts — correct +on bare metal, silently wrong under the golden containerized deployment (the daemon resolves the +`-v` host side against the GUEST filesystem): the export's tar stranded host-side while the +bundle shipped an EMPTY `data/volumes` and reported SUCCESS; the import then wiped the app's +volumes and populated them from host-side emptiness. Live-hit on demo ActualBudget (v0.124.0 +validation). No .fab format change — but note the ASYMMETRY: **any bundle exported by a +containerized controller ≤0.124.0 is suspect (hollow volume data) — re-export**; the new +import-side guard refuses such bundles loudly instead of destroying the app. + +- **`docker cp` tar-streaming both legs** (`appexport/export.go` + `restore.go`, new `dockerExec` + seam): a stopped helper container pins the volume (`docker create -v :/vol alpine true`), + the tar streams over the docker API (`docker cp :/vol/. -` out; `docker cp - :/vol` + in) — ZERO shared paths, correct in both deployment shapes. §3 live probe proved content, + subdirs, symlinks, empty files and uid/gid round-trip. Helpers are ALWAYS force-removed, error + paths included (test-asserted); 10-min/volume timeouts + truncated stderr preserved. +- **Export can no longer lie** (scenario B): a failed volume export is FATAL (was WARN+continue); + `assertBundleDataComplete` refuses to package any bundle whose manifest claims a tar that is + missing/empty (volumes AND HDD subdirs — the HDD leg also stopped pre-claiming subdirs before + the tar succeeds). Live-proven: an engine-invalid volume name failed the export naming the + volume, no bundle staged. +- **Import validates BEFORE it destroys** (scenario C): `validateBundleData` refuses a + claimed-but-absent/empty data tar in step 0 — before the app is stopped and before any volume + is removed (the pre-fix order wiped first and discovered later); the refusal names the hollow + ≤0.124.0-exporter cause and states the app is untouched. `restoreVolumeData`/`restoreHDDData` + missing-tar soft-skips became hard errors (defense in depth). +- **The class is extinct** (scenario D): `scripts/docker_run_volume_path_gate.py` — every `"-v"` + argument in non-test Go code must be allowlisted with its WHY; the Tier-1/2 volume dump/restore + entries are documented host-visible (registered-drive namespace paths under the golden + deployment's identical `/mnt` + `/opt/docker` binds), the rest are named-volume/flag usages. +- Red-proofs: assertion removed → hollow-success test fails; pre-flight disabled → the + zero-destruction assertions fail (`removedVolumes=1`); a violating `-v` line → gate exits 1. +- **Live §13**: supervised repeat of the exact failed leg on demo — export → download (bundle + now carries the 66048-byte volume tar) → drive placement → import → **volume fingerprint + byte-identical** (`ec8ea6cb…` before == after), app healthy, zero leaked helpers. +- **NOTE for the next publish train:** the golden floor must not advance past 0.124.0 without + this fix; floor may advance to 0.125.0 now that it validates. + ### v0.124.0 — backups IA restructure: four sub-pages, Felhom-offsite status card, .fab browser download (2026-07-13) — MinAgent: 0.81.0 The nine-section backups page split into four sub-pages (operator review: customers got lost); diff --git a/CONTEXT.md b/CONTEXT.md index 8ba5b62..fc063a2 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,19 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-13 (v0.124.0 — backups IA restructure) +Last updated: 2026-07-13 (v0.125.0 — .fab volume path-strand fix) + +> **2026-07-13 — v0.125.0: .FAB VOLUME PATH-STRAND DATA LOSS FIXED (IA finding 1, HIGH). +> MinAgent 0.81 unchanged; floor may advance to 0.125.0 next train (must NOT halt above 0.124.0 +> without this).** Both volume legs stream via docker cp (helper container + `dockerExec` seam — +> zero shared paths, correct bare-metal AND containerized; §3 live probe first). Export FAILS +> LOUD on any missing/empty claimed tar (`assertBundleDataComplete`); import VALIDATES BEFORE it +> destroys (`validateBundleData` in step 0 — hollow bundle → refusal, app untouched). Class +> extinguished by `scripts/docker_run_volume_path_gate.py` (every `"-v"` allowlisted with WHY; +> Tier-1/2 mounts documented host-visible). Live: the exact failed ActualBudget leg round-trips +> byte-identically (`ec8ea6cb…` before==after); engine-invalid volume → loud export failure. +> **ASYMMETRY (needs a customer-docs line):** .fab bundles exported by containerized ≤0.124.0 +> controllers are hollow — re-export; the import guard refuses them loudly. > **2026-07-13 — v0.124.0: BACKUPS IA RESTRUCTURE. MinAgent 0.81.0 + floor unchanged. > Operator decisions (2026-07-13, treat as settled):** (1) single active offsite destination per diff --git a/REPORT.md b/REPORT.md index 4f3d2b7..1067180 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,86 +1,72 @@ -# REPORT — v0.124.0: backups IA restructure (four sub-pages, Felhom-offsite status card, .fab browser download) +# REPORT — v0.125.0: .fab volume path-strand data loss FIXED (IA finding 1, HIGH) -**Date:** 2026-07-13 · **Version:** controller v0.124.0 (from v0.123.0) · **MinAgent:** 0.81.0 (UNCHANGED) · **Floor:** untouched (rides the next train) · Controller-only. +**Date:** 2026-07-13 · **Version:** controller v0.125.0 (from v0.124.0) · **MinAgent:** 0.81.0 (UNCHANGED) · **Floor:** may advance to 0.125.0 on the next train (must NOT stop at >0.124.0 without this) · Controller-only. **Disposition:** closes the v0.124.0 REPORT's finding 1. -## What shipped (commits `0ece2ba`, `aa967fb`, `753cd83`) +## §3 live probe (ran FIRST, before any code) -1. **IA split** — `/backups` (Áttekintés), `/backups/remote` (Távoli mentés), `/backups/apps` - (Alkalmazások), `/backups/restore` (Visszaállítás), sidebar children (Tárhely pattern). - Sections MOVED verbatim from the v0.123.0 page — the committed one-shot - `scripts/backups_split_move_check.py` verifies all 15 blocks against baseline `df7ad37` - (whitespace-normalized; the two allowed divergences — tier-3 anchor retarget + the - restore-to-verify relocation — are encoded explicitly). Shared handler builders extracted - (`backupsCommonData`/`backupsOffboxData`); flash redirects, tier-3 anchors and the - tier2-config back-link retargeted per page. -2. **Felhom-offsite status card** — three honest states from local data, display-only by - construction (no form/button inside the card; unit-enforced). Decision 2 honored: the card - never changes anything. -3. **.fab browser download** — existing export pipeline + staging dest + guarded streaming exit - with estimate-first UX, post-stream cleanup and a 1h TTL sweep; per-app sequential batch; - portability framing everywhere (decision 3). Import stays drive-scan. +On the drill guest: scratch volume seeded with a subdir, a file, a symlink, an empty file and a +non-root owner (1234:1234) → `docker create -v fabprobe:/vol alpine true` → `docker cp +:/vol/. -` → tar (5120 B) → volume wiped → `docker cp - :/vol` → **everything +byte-identical, symlink and uid/gid preserved**. Zero surprises; transcript in the session +record. Green light for implementation. -## Tests + red-proofs (run → FAIL → restored) +## What shipped (commit `466f427`) -- **Move check red-proof:** one moved block reworded → exit 1 ("1 block(s) rewritten"); restored → 0. -- **Traversal guard red-proof:** guard loosened to raw prefix-matching → `../decoy.fab` came back - **200 serving the decoy** — the test caught it (got 200, want 400). Restored → green. -- **Round-trip + corruption (data-mutation red-proof):** real `executeExport` → bundle → - real `executeImport` → byte-identical restored content; a mid-file-corrupted copy is REFUSED - (job fails, app not started, nothing restored). Integrity = gzip CRC + manifest segment - validation — there is NO per-file checksum (documented; the spec's "manifest checksum" wording - adapted to reality). -- Nav/state tests (marker per section, present on exactly one page; `.Page` ids), status-card - state tests, TTL sweep test (fake mtimes/now). Full green gate + all five template gates green - per commit. +1. **Streaming legs** — `dockerExec` seam + `withVolumeHelper` (create → fn → ALWAYS `rm -f`, + error paths included) + `exportVolumeTar` (`cp :/vol/. -` → file) and `importVolumeTar` + (file → `cp - :/vol`). Zero shared paths: correct on bare metal AND containerized. +2. **Scenario B** — volume-export failure is FATAL; `assertBundleDataComplete` blocks packaging + any bundle whose manifest claims a missing/empty tar (volumes + HDD; the HDD leg also stopped + pre-claiming subdirs before the tar succeeds). +3. **Scenario C** — `validateBundleData` runs in import step 0, BEFORE the app is stopped or any + volume removed; hollow bundles are refused with copy naming the ≤0.124.0-containerized-export + cause and stating the app is untouched. Missing-tar soft-skips in both restore legs are hard + errors now. +4. **Scenario D** — `scripts/docker_run_volume_path_gate.py`: every `"-v"` arg in non-test Go + needs an allowlist entry WITH ITS WHY. Gate output: 7 allowlisted usages, 0 violations. + **Tier-1/Tier-2 mechanism, proven by inspection:** `backup.go` (dump) and `backup/restore.go` + mount `dumpDir` — always a registered-drive namespace path (`/mnt/**`, `/opt/docker/**`), + which the golden deployment bind-mounts into the controller container at IDENTICAL paths + (container-inspect verified), so the daemon resolves them correctly; their volume-side mounts + are named volumes (daemon-side). They are NOT in the strand class; unchanged. -## §13 live validation +## Tests + the three red-proofs (run → FAIL → restored) -- **Deployed:** 0.124.0 on the drill guest (qm300) AND demo 9201, both healthy. -- **Four-page walk (drill, screenshots in session record):** every moved control clicked once — - tier-2 Beállítás (+ retargeted back-link), tier-3 toggle (off→on), restore-to-verify from its - NEW home (`/backups/restore` flash), manual-target form open/close. Old `/backups` bookmark → - Áttekintés on both boxes; sidebar active states correct. -- **Status card:** state 3 (no card) and state 2 ("Aktív — nincs kijelölt alkalmazás") both - live-rendered on the drill box; state 1 by unit render test (demo has a managed target — spec - says do NOT unconfigure). -- **.fab download:** drill ActualBudget — estimate 3.6 KB shown first, bundle 2102 B streamed, - staging bundle removed (server log). Demo ActualBudget — downloaded to the operator machine, - sha256-identical after placement on `felhom-usb/exports` (`e2a1233f…` both ends). -- **Supervised import (Viktor GO):** ON DEMO (see finding 2 for why not drill): scan found the - bundle, manifest read (exported-by 0.124.0), "Importálás kész! Az alkalmazás sikeresen - visszaállítva.", ActualBudget Up (healthy). **The round-trip surfaced finding 1.** +- **B:** `TestExport_HollowVolumeTarAbortsExport` (cp "succeeds" writing nothing → export fails + naming vol1, no .fab). Red-proof: assertion disabled → "a hollow volume tar must FAIL the + export — got success". +- **C:** `TestImport_HollowBundleRefusedBeforeDestroy` (handcrafted hollow bundle → refused; + asserts stopped=0, removedVolumes=0, zero docker calls, app not started). Red-proof: pre-flight + disabled → **`refusal happened AFTER destruction: removedVolumes=1`** — the exact pre-fix + disaster shape. +- **D:** violating `docker run -v /tmp/x:/out` line added → gate exit 1; restored → 0. +- Plus command-shape tests (create/cp/rm arg vectors pinned to the §3-probed shapes; import cp + streams stdin; helper removed on cp failure; no partial tar left). Full green gate + all seven + gates green. -## FINDINGS (both pre-existing, surfaced by this task's first real containerized round-trip) +## §13 live validation (0.125.0 on drill + demo, both healthy) -1. **(HIGH — data-loss trap) Containerized .fab export ships EMPTY volume data.** - `exportVolumeData` runs `docker run -v :/out alpine tar …` where `` is the - controller-CONTAINER's `os.MkdirTemp` path — the guest's docker daemon resolves it against the - HOST filesystem, so the tar lands stranded on the guest host (`/tmp/felhom-export-*/…`, found - with ActualBudget's 66 KB tar in it) and the bundle's `data/volumes` is empty while the - manifest claims volume data and the export reports SUCCESS. Import then wipes the app's - volumes and brings it up EMPTY. Affects every golden/bootstrap (containerized) box for - volume-backed apps; HDD-data apps are unaffected (direct FS copy). Demo's ActualBudget was - repaired from the stranded tar; both temp dir and staged bundle cleaned. **Fix direction:** - stage the docker-run output under a host-visible bind (e.g. the controller data volume) or use - `docker cp`/stdout streaming; plus a post-export assertion that manifest-claimed volume tars - exist in the bundle (fail LOUD). -2. **(MEDIUM — agent-side, out of this controller-only task) Drive wizard offers zero candidates - on a legacy-boot PVE.** With LVM root and no ESP mount, every system mount resolves to - device-mapper, `storage.SystemDisks` finds no raw disk → `sysKnown=false` → `isSystemBacked` - fail-safes EVERYTHING to system → `ListCandidateDisks` is always empty (drill box; a hot-added - blank 5G disk stayed invisible). The fail-safe is right to exist but overreaches here — needs - an agent ruling (e.g. resolve device-mapper parents to their PVs' disks). +- **Supervised round-trip (Viktor GO), the EXACT leg that failed yesterday:** demo ActualBudget + export → browser download (`actualbudget_20260713-085217.fab`, 3762 B — the bundle now carries + the **66048-byte volume tar**; yesterday's was 2097 B with an EMPTY data/volumes) → sha256- + identical after placement on felhom-usb/exports → import → "Importálás kész" → **volume + fingerprint after == before: `ec8ea6cb…` (2 files)** → app Up (healthy) → zero leaked alpine + helpers → bundle cleaned from the drive. +- **Scenario B live:** a scratch stack whose compose declared the engine-invalid volume + `"bad vol"` (docker auto-creates *nonexistent* names, so engine-invalid is the honest live + trigger) → export failed LOUD: "Kötet mentése sikertelen: volume scratchvol_bad vol export + failed … invalid characters …" — zero bundles staged, zero leftover containers; scratch stack + removed after. ## Observations -- **.fab browser-UPLOAD** remains the noted follow-up (import is drive-scan only — downloaded - bundles return via a drive/share, exactly as exercised here). -- **Combined mega-zip** stays parked (decision; per-app sequential batch shipped instead). -- **appexport packages LIVE state** (fresh config copy + DB dump + live data), NOT the tier-1 - recovery unit — §8's "stale unit" concern is moot; no "futtass előbb mentést" needed. -- The import page's drive-label mojibake ("TÃ˘rhely (felhom-usb)") is the KNOWN open fix-5 item. -- `NoUserDataBackupWarning` is referenced by the per-app section but never set by any handler - (dead conditional, pre-existing); `DBDumpTotalBytes` was computed but never rendered — dropped - in the split. -- Drill box left as found: hot-added disk detached + LV removed; ActualBudget re-toggled; demo - exports dir + stray export temp cleaned. +- **Hollow-bundle asymmetry (customer-facing docs line needed):** any .fab exported by a + CONTAINERIZED controller ≤0.124.0 has empty volume data while claiming it — treat as suspect + and RE-EXPORT on ≥0.125.0. The import guard now refuses them loudly (app untouched) instead of + silently importing emptiness. Bare-metal/felhotest exports were never affected; HDD-data apps + were never affected. +- Spec §7B's "point a volume at a nonexistent volume" live construction doesn't fail under + docker (named volumes auto-create; an empty volume exports as an honest small tar — correct + behavior) — the live loud-fail used an engine-invalid name instead; the missing/empty-tar + refusal is unit-pinned with red-proof. +- The one Windows-runner nuance: none — the docker seam keeps unit tests fully offline. diff --git a/REUSE.md b/REUSE.md index 32010b5..2d8ac6e 100644 --- a/REUSE.md +++ b/REUSE.md @@ -208,6 +208,7 @@ Cross-repo edges: - **New background job**: `sched.Every`/`sched.Daily` registration block in controller/cmd/controller/main.go. - **New template function**: `Server.templateFuncMap` (controller/internal/web/funcmap.go) — obey v2 state-suffix vocabulary. - **New page/nav item**: `baseData` + sidebar in controller/internal/web/templates/ (nested sub-links pattern `.nav-links-nested`); must pass `controller/scripts/template_id_gate.py` + `controller/scripts/emoji_gate.py` + `controller/scripts/native_confirm_gate.py` + `controller/scripts/offbox_rename_gate.py`. +- **Docker volume tar streaming (v0.125.0)**: `appexport.dockerExec` (seam, package var) + `withVolumeHelper`/`exportVolumeTar`/`importVolumeTar` — stream volume content via `docker cp` through a stopped helper container. NEVER `docker run -v ` — the daemon resolves `-v` host-side and strands the data when the controller is containerized (the v0.124.0 HIGH finding); `controller/scripts/docker_run_volume_path_gate.py` enforces (every `"-v"` allowlisted with its WHY). - **Guarded file download (v0.124.0)**: `handler_export_download.go` — the canonical shape for streaming a server-side file to the browser: accept a BASENAME only (shape regexp + no separators/`..`), `filepath.Join` then assert `filepath.Dir(path) == dir`, `io.Copy` (never ReadAll), `Content-Disposition: attachment`, remove after a successful stream, TTL sweep (`sweepFabDownloads(dir, now, maxAge, logger)` — now injected for tests). Red-proof the guard by loosening to prefix-matching (the `..` case must fail). - **Backups sub-page data**: `backupsCommonData(page, title, r)` + `backupsOffboxData(data)` (handlers.go) — the ONLY builders for the four `/backups*` pages; a new backups section extends these, never re-derives in a page handler. Section moves are gated by `scripts/backups_split_move_check.py` (one-shot vs df7ad37). - **Consequential-action confirm (LIGHT)**: `felhomConfirm(el, question, onYes)` in layout.html (v0.123.0) — the trigger swaps in place to "kérdés + Igen/Mégse"; form buttons opt in with `data-confirm="…"` (delegated listener, `requestSubmit` keeps formaction/name-value). NEVER native `confirm()`/`prompt()` (OS-modals freeze browser automation — drill F-11; `native_confirm_gate.py` enforces). Heavy destructive flows keep the `.confirm-overlay` `openDialog` pattern. diff --git a/controller/README.md b/controller/README.md index 801eb1e..58117a9 100644 --- a/controller/README.md +++ b/controller/README.md @@ -337,9 +337,9 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte **Encryption:** Optional AES-256-CTR + HMAC-SHA256 with scrypt key derivation (N=32768). Format: `"FABE"` magic header + salt + IV + encrypted tar.gz + HMAC tag. Streaming for multi-GB files. -**Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped. +**Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data (docker volumes stream via `docker cp` through a stopped helper container — v0.125.0; never a `docker run -v` host mount, which strands data on containerized controllers) → **fail-loud bundle assertion** (every manifest-claimed tar must exist non-empty — no hollow bundles) → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped. -**Import flow:** Decrypt if needed → extract → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD or volumes) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI. +**Import flow:** Decrypt if needed → extract → **validate-before-destroy** (v0.125.0: every manifest-claimed data tar must be present non-empty BEFORE the app is stopped or any volume removed — hollow bundles from containerized ≤0.124.0 exporters are refused with the app untouched) → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD tars; volumes populate via `docker cp` streaming) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI. **Architecture:** `internal/appexport/` package with `ExportStackProvider` adapter interface (same pattern as `backup.StackDataProvider`). `exportAdapter` in `main.go` bridges `stacks.Manager` to the provider.