docs: v0.125.0 path-strand fix — CHANGELOG + REPORT (probe transcript, red-proofs, round-trip evidence, Tier-1/2 mechanism) + CONTEXT + REUSE + README flows; hollow-bundle asymmetry recorded (containerized ≤0.124.0 exports suspect — re-export); floor note for the next train
Claude-Session: https://claude.ai/code/session_01GzammAMzsJTgpQHqxwM2bC
This commit is contained in:
@@ -1,5 +1,44 @@
|
||||
## Changelog
|
||||
|
||||
### v0.125.0 — .fab volume export/import: containerized path-strand data loss FIXED (IA finding 1, HIGH) (2026-07-13) — MinAgent: 0.81.0
|
||||
|
||||
Both .fab volume legs streamed via `docker run -v <controller-temp-path>` host mounts — correct
|
||||
on bare metal, silently wrong under the golden containerized deployment (the daemon resolves the
|
||||
`-v` host side against the GUEST filesystem): the export's tar stranded host-side while the
|
||||
bundle shipped an EMPTY `data/volumes` and reported SUCCESS; the import then wiped the app's
|
||||
volumes and populated them from host-side emptiness. Live-hit on demo ActualBudget (v0.124.0
|
||||
validation). No .fab format change — but note the ASYMMETRY: **any bundle exported by a
|
||||
containerized controller ≤0.124.0 is suspect (hollow volume data) — re-export**; the new
|
||||
import-side guard refuses such bundles loudly instead of destroying the app.
|
||||
|
||||
- **`docker cp` tar-streaming both legs** (`appexport/export.go` + `restore.go`, new `dockerExec`
|
||||
seam): a stopped helper container pins the volume (`docker create -v <vol>:/vol alpine true`),
|
||||
the tar streams over the docker API (`docker cp <cid>:/vol/. -` out; `docker cp - <cid>:/vol`
|
||||
in) — ZERO shared paths, correct in both deployment shapes. §3 live probe proved content,
|
||||
subdirs, symlinks, empty files and uid/gid round-trip. Helpers are ALWAYS force-removed, error
|
||||
paths included (test-asserted); 10-min/volume timeouts + truncated stderr preserved.
|
||||
- **Export can no longer lie** (scenario B): a failed volume export is FATAL (was WARN+continue);
|
||||
`assertBundleDataComplete` refuses to package any bundle whose manifest claims a tar that is
|
||||
missing/empty (volumes AND HDD subdirs — the HDD leg also stopped pre-claiming subdirs before
|
||||
the tar succeeds). Live-proven: an engine-invalid volume name failed the export naming the
|
||||
volume, no bundle staged.
|
||||
- **Import validates BEFORE it destroys** (scenario C): `validateBundleData` refuses a
|
||||
claimed-but-absent/empty data tar in step 0 — before the app is stopped and before any volume
|
||||
is removed (the pre-fix order wiped first and discovered later); the refusal names the hollow
|
||||
≤0.124.0-exporter cause and states the app is untouched. `restoreVolumeData`/`restoreHDDData`
|
||||
missing-tar soft-skips became hard errors (defense in depth).
|
||||
- **The class is extinct** (scenario D): `scripts/docker_run_volume_path_gate.py` — every `"-v"`
|
||||
argument in non-test Go code must be allowlisted with its WHY; the Tier-1/2 volume dump/restore
|
||||
entries are documented host-visible (registered-drive namespace paths under the golden
|
||||
deployment's identical `/mnt` + `/opt/docker` binds), the rest are named-volume/flag usages.
|
||||
- Red-proofs: assertion removed → hollow-success test fails; pre-flight disabled → the
|
||||
zero-destruction assertions fail (`removedVolumes=1`); a violating `-v` line → gate exits 1.
|
||||
- **Live §13**: supervised repeat of the exact failed leg on demo — export → download (bundle
|
||||
now carries the 66048-byte volume tar) → drive placement → import → **volume fingerprint
|
||||
byte-identical** (`ec8ea6cb…` before == after), app healthy, zero leaked helpers.
|
||||
- **NOTE for the next publish train:** the golden floor must not advance past 0.124.0 without
|
||||
this fix; floor may advance to 0.125.0 now that it validates.
|
||||
|
||||
### v0.124.0 — backups IA restructure: four sub-pages, Felhom-offsite status card, .fab browser download (2026-07-13) — MinAgent: 0.81.0
|
||||
|
||||
The nine-section backups page split into four sub-pages (operator review: customers got lost);
|
||||
|
||||
+13
-1
@@ -7,7 +7,19 @@
|
||||
>
|
||||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||||
|
||||
Last updated: 2026-07-13 (v0.124.0 — backups IA restructure)
|
||||
Last updated: 2026-07-13 (v0.125.0 — .fab volume path-strand fix)
|
||||
|
||||
> **2026-07-13 — v0.125.0: .FAB VOLUME PATH-STRAND DATA LOSS FIXED (IA finding 1, HIGH).
|
||||
> MinAgent 0.81 unchanged; floor may advance to 0.125.0 next train (must NOT halt above 0.124.0
|
||||
> without this).** Both volume legs stream via docker cp (helper container + `dockerExec` seam —
|
||||
> zero shared paths, correct bare-metal AND containerized; §3 live probe first). Export FAILS
|
||||
> LOUD on any missing/empty claimed tar (`assertBundleDataComplete`); import VALIDATES BEFORE it
|
||||
> destroys (`validateBundleData` in step 0 — hollow bundle → refusal, app untouched). Class
|
||||
> extinguished by `scripts/docker_run_volume_path_gate.py` (every `"-v"` allowlisted with WHY;
|
||||
> Tier-1/2 mounts documented host-visible). Live: the exact failed ActualBudget leg round-trips
|
||||
> byte-identically (`ec8ea6cb…` before==after); engine-invalid volume → loud export failure.
|
||||
> **ASYMMETRY (needs a customer-docs line):** .fab bundles exported by containerized ≤0.124.0
|
||||
> controllers are hollow — re-export; the import guard refuses them loudly.
|
||||
|
||||
> **2026-07-13 — v0.124.0: BACKUPS IA RESTRUCTURE. MinAgent 0.81.0 + floor unchanged.
|
||||
> Operator decisions (2026-07-13, treat as settled):** (1) single active offsite destination per
|
||||
|
||||
@@ -1,86 +1,72 @@
|
||||
# REPORT — v0.124.0: backups IA restructure (four sub-pages, Felhom-offsite status card, .fab browser download)
|
||||
# REPORT — v0.125.0: .fab volume path-strand data loss FIXED (IA finding 1, HIGH)
|
||||
|
||||
**Date:** 2026-07-13 · **Version:** controller v0.124.0 (from v0.123.0) · **MinAgent:** 0.81.0 (UNCHANGED) · **Floor:** untouched (rides the next train) · Controller-only.
|
||||
**Date:** 2026-07-13 · **Version:** controller v0.125.0 (from v0.124.0) · **MinAgent:** 0.81.0 (UNCHANGED) · **Floor:** may advance to 0.125.0 on the next train (must NOT stop at >0.124.0 without this) · Controller-only. **Disposition:** closes the v0.124.0 REPORT's finding 1.
|
||||
|
||||
## What shipped (commits `0ece2ba`, `aa967fb`, `753cd83`)
|
||||
## §3 live probe (ran FIRST, before any code)
|
||||
|
||||
1. **IA split** — `/backups` (Áttekintés), `/backups/remote` (Távoli mentés), `/backups/apps`
|
||||
(Alkalmazások), `/backups/restore` (Visszaállítás), sidebar children (Tárhely pattern).
|
||||
Sections MOVED verbatim from the v0.123.0 page — the committed one-shot
|
||||
`scripts/backups_split_move_check.py` verifies all 15 blocks against baseline `df7ad37`
|
||||
(whitespace-normalized; the two allowed divergences — tier-3 anchor retarget + the
|
||||
restore-to-verify relocation — are encoded explicitly). Shared handler builders extracted
|
||||
(`backupsCommonData`/`backupsOffboxData`); flash redirects, tier-3 anchors and the
|
||||
tier2-config back-link retargeted per page.
|
||||
2. **Felhom-offsite status card** — three honest states from local data, display-only by
|
||||
construction (no form/button inside the card; unit-enforced). Decision 2 honored: the card
|
||||
never changes anything.
|
||||
3. **.fab browser download** — existing export pipeline + staging dest + guarded streaming exit
|
||||
with estimate-first UX, post-stream cleanup and a 1h TTL sweep; per-app sequential batch;
|
||||
portability framing everywhere (decision 3). Import stays drive-scan.
|
||||
On the drill guest: scratch volume seeded with a subdir, a file, a symlink, an empty file and a
|
||||
non-root owner (1234:1234) → `docker create -v fabprobe:/vol alpine true` → `docker cp
|
||||
<cid>:/vol/. -` → tar (5120 B) → volume wiped → `docker cp - <cid>:/vol` → **everything
|
||||
byte-identical, symlink and uid/gid preserved**. Zero surprises; transcript in the session
|
||||
record. Green light for implementation.
|
||||
|
||||
## Tests + red-proofs (run → FAIL → restored)
|
||||
## What shipped (commit `466f427`)
|
||||
|
||||
- **Move check red-proof:** one moved block reworded → exit 1 ("1 block(s) rewritten"); restored → 0.
|
||||
- **Traversal guard red-proof:** guard loosened to raw prefix-matching → `../decoy.fab` came back
|
||||
**200 serving the decoy** — the test caught it (got 200, want 400). Restored → green.
|
||||
- **Round-trip + corruption (data-mutation red-proof):** real `executeExport` → bundle →
|
||||
real `executeImport` → byte-identical restored content; a mid-file-corrupted copy is REFUSED
|
||||
(job fails, app not started, nothing restored). Integrity = gzip CRC + manifest segment
|
||||
validation — there is NO per-file checksum (documented; the spec's "manifest checksum" wording
|
||||
adapted to reality).
|
||||
- Nav/state tests (marker per section, present on exactly one page; `.Page` ids), status-card
|
||||
state tests, TTL sweep test (fake mtimes/now). Full green gate + all five template gates green
|
||||
per commit.
|
||||
1. **Streaming legs** — `dockerExec` seam + `withVolumeHelper` (create → fn → ALWAYS `rm -f`,
|
||||
error paths included) + `exportVolumeTar` (`cp <cid>:/vol/. -` → file) and `importVolumeTar`
|
||||
(file → `cp - <cid>:/vol`). Zero shared paths: correct on bare metal AND containerized.
|
||||
2. **Scenario B** — volume-export failure is FATAL; `assertBundleDataComplete` blocks packaging
|
||||
any bundle whose manifest claims a missing/empty tar (volumes + HDD; the HDD leg also stopped
|
||||
pre-claiming subdirs before the tar succeeds).
|
||||
3. **Scenario C** — `validateBundleData` runs in import step 0, BEFORE the app is stopped or any
|
||||
volume removed; hollow bundles are refused with copy naming the ≤0.124.0-containerized-export
|
||||
cause and stating the app is untouched. Missing-tar soft-skips in both restore legs are hard
|
||||
errors now.
|
||||
4. **Scenario D** — `scripts/docker_run_volume_path_gate.py`: every `"-v"` arg in non-test Go
|
||||
needs an allowlist entry WITH ITS WHY. Gate output: 7 allowlisted usages, 0 violations.
|
||||
**Tier-1/Tier-2 mechanism, proven by inspection:** `backup.go` (dump) and `backup/restore.go`
|
||||
mount `dumpDir` — always a registered-drive namespace path (`/mnt/**`, `/opt/docker/**`),
|
||||
which the golden deployment bind-mounts into the controller container at IDENTICAL paths
|
||||
(container-inspect verified), so the daemon resolves them correctly; their volume-side mounts
|
||||
are named volumes (daemon-side). They are NOT in the strand class; unchanged.
|
||||
|
||||
## §13 live validation
|
||||
## Tests + the three red-proofs (run → FAIL → restored)
|
||||
|
||||
- **Deployed:** 0.124.0 on the drill guest (qm300) AND demo 9201, both healthy.
|
||||
- **Four-page walk (drill, screenshots in session record):** every moved control clicked once —
|
||||
tier-2 Beállítás (+ retargeted back-link), tier-3 toggle (off→on), restore-to-verify from its
|
||||
NEW home (`/backups/restore` flash), manual-target form open/close. Old `/backups` bookmark →
|
||||
Áttekintés on both boxes; sidebar active states correct.
|
||||
- **Status card:** state 3 (no card) and state 2 ("Aktív — nincs kijelölt alkalmazás") both
|
||||
live-rendered on the drill box; state 1 by unit render test (demo has a managed target — spec
|
||||
says do NOT unconfigure).
|
||||
- **.fab download:** drill ActualBudget — estimate 3.6 KB shown first, bundle 2102 B streamed,
|
||||
staging bundle removed (server log). Demo ActualBudget — downloaded to the operator machine,
|
||||
sha256-identical after placement on `felhom-usb/exports` (`e2a1233f…` both ends).
|
||||
- **Supervised import (Viktor GO):** ON DEMO (see finding 2 for why not drill): scan found the
|
||||
bundle, manifest read (exported-by 0.124.0), "Importálás kész! Az alkalmazás sikeresen
|
||||
visszaállítva.", ActualBudget Up (healthy). **The round-trip surfaced finding 1.**
|
||||
- **B:** `TestExport_HollowVolumeTarAbortsExport` (cp "succeeds" writing nothing → export fails
|
||||
naming vol1, no .fab). Red-proof: assertion disabled → "a hollow volume tar must FAIL the
|
||||
export — got success".
|
||||
- **C:** `TestImport_HollowBundleRefusedBeforeDestroy` (handcrafted hollow bundle → refused;
|
||||
asserts stopped=0, removedVolumes=0, zero docker calls, app not started). Red-proof: pre-flight
|
||||
disabled → **`refusal happened AFTER destruction: removedVolumes=1`** — the exact pre-fix
|
||||
disaster shape.
|
||||
- **D:** violating `docker run -v /tmp/x:/out` line added → gate exit 1; restored → 0.
|
||||
- Plus command-shape tests (create/cp/rm arg vectors pinned to the §3-probed shapes; import cp
|
||||
streams stdin; helper removed on cp failure; no partial tar left). Full green gate + all seven
|
||||
gates green.
|
||||
|
||||
## FINDINGS (both pre-existing, surfaced by this task's first real containerized round-trip)
|
||||
## §13 live validation (0.125.0 on drill + demo, both healthy)
|
||||
|
||||
1. **(HIGH — data-loss trap) Containerized .fab export ships EMPTY volume data.**
|
||||
`exportVolumeData` runs `docker run -v <volDir>:/out alpine tar …` where `<volDir>` is the
|
||||
controller-CONTAINER's `os.MkdirTemp` path — the guest's docker daemon resolves it against the
|
||||
HOST filesystem, so the tar lands stranded on the guest host (`/tmp/felhom-export-*/…`, found
|
||||
with ActualBudget's 66 KB tar in it) and the bundle's `data/volumes` is empty while the
|
||||
manifest claims volume data and the export reports SUCCESS. Import then wipes the app's
|
||||
volumes and brings it up EMPTY. Affects every golden/bootstrap (containerized) box for
|
||||
volume-backed apps; HDD-data apps are unaffected (direct FS copy). Demo's ActualBudget was
|
||||
repaired from the stranded tar; both temp dir and staged bundle cleaned. **Fix direction:**
|
||||
stage the docker-run output under a host-visible bind (e.g. the controller data volume) or use
|
||||
`docker cp`/stdout streaming; plus a post-export assertion that manifest-claimed volume tars
|
||||
exist in the bundle (fail LOUD).
|
||||
2. **(MEDIUM — agent-side, out of this controller-only task) Drive wizard offers zero candidates
|
||||
on a legacy-boot PVE.** With LVM root and no ESP mount, every system mount resolves to
|
||||
device-mapper, `storage.SystemDisks` finds no raw disk → `sysKnown=false` → `isSystemBacked`
|
||||
fail-safes EVERYTHING to system → `ListCandidateDisks` is always empty (drill box; a hot-added
|
||||
blank 5G disk stayed invisible). The fail-safe is right to exist but overreaches here — needs
|
||||
an agent ruling (e.g. resolve device-mapper parents to their PVs' disks).
|
||||
- **Supervised round-trip (Viktor GO), the EXACT leg that failed yesterday:** demo ActualBudget
|
||||
export → browser download (`actualbudget_20260713-085217.fab`, 3762 B — the bundle now carries
|
||||
the **66048-byte volume tar**; yesterday's was 2097 B with an EMPTY data/volumes) → sha256-
|
||||
identical after placement on felhom-usb/exports → import → "Importálás kész" → **volume
|
||||
fingerprint after == before: `ec8ea6cb…` (2 files)** → app Up (healthy) → zero leaked alpine
|
||||
helpers → bundle cleaned from the drive.
|
||||
- **Scenario B live:** a scratch stack whose compose declared the engine-invalid volume
|
||||
`"bad vol"` (docker auto-creates *nonexistent* names, so engine-invalid is the honest live
|
||||
trigger) → export failed LOUD: "Kötet mentése sikertelen: volume scratchvol_bad vol export
|
||||
failed … invalid characters …" — zero bundles staged, zero leftover containers; scratch stack
|
||||
removed after.
|
||||
|
||||
## Observations
|
||||
|
||||
- **.fab browser-UPLOAD** remains the noted follow-up (import is drive-scan only — downloaded
|
||||
bundles return via a drive/share, exactly as exercised here).
|
||||
- **Combined mega-zip** stays parked (decision; per-app sequential batch shipped instead).
|
||||
- **appexport packages LIVE state** (fresh config copy + DB dump + live data), NOT the tier-1
|
||||
recovery unit — §8's "stale unit" concern is moot; no "futtass előbb mentést" needed.
|
||||
- The import page's drive-label mojibake ("TÃ˘rhely (felhom-usb)") is the KNOWN open fix-5 item.
|
||||
- `NoUserDataBackupWarning` is referenced by the per-app section but never set by any handler
|
||||
(dead conditional, pre-existing); `DBDumpTotalBytes` was computed but never rendered — dropped
|
||||
in the split.
|
||||
- Drill box left as found: hot-added disk detached + LV removed; ActualBudget re-toggled; demo
|
||||
exports dir + stray export temp cleaned.
|
||||
- **Hollow-bundle asymmetry (customer-facing docs line needed):** any .fab exported by a
|
||||
CONTAINERIZED controller ≤0.124.0 has empty volume data while claiming it — treat as suspect
|
||||
and RE-EXPORT on ≥0.125.0. The import guard now refuses them loudly (app untouched) instead of
|
||||
silently importing emptiness. Bare-metal/felhotest exports were never affected; HDD-data apps
|
||||
were never affected.
|
||||
- Spec §7B's "point a volume at a nonexistent volume" live construction doesn't fail under
|
||||
docker (named volumes auto-create; an empty volume exports as an honest small tar — correct
|
||||
behavior) — the live loud-fail used an engine-invalid name instead; the missing/empty-tar
|
||||
refusal is unit-pinned with red-proof.
|
||||
- The one Windows-runner nuance: none — the docker seam keeps unit tests fully offline.
|
||||
|
||||
@@ -208,6 +208,7 @@ Cross-repo edges:
|
||||
- **New background job**: `sched.Every`/`sched.Daily` registration block in controller/cmd/controller/main.go.
|
||||
- **New template function**: `Server.templateFuncMap` (controller/internal/web/funcmap.go) — obey v2 state-suffix vocabulary.
|
||||
- **New page/nav item**: `baseData` + sidebar in controller/internal/web/templates/ (nested sub-links pattern `.nav-links-nested`); must pass `controller/scripts/template_id_gate.py` + `controller/scripts/emoji_gate.py` + `controller/scripts/native_confirm_gate.py` + `controller/scripts/offbox_rename_gate.py`.
|
||||
- **Docker volume tar streaming (v0.125.0)**: `appexport.dockerExec` (seam, package var) + `withVolumeHelper`/`exportVolumeTar`/`importVolumeTar` — stream volume content via `docker cp` through a stopped helper container. NEVER `docker run -v <controller-local path>` — the daemon resolves `-v` host-side and strands the data when the controller is containerized (the v0.124.0 HIGH finding); `controller/scripts/docker_run_volume_path_gate.py` enforces (every `"-v"` allowlisted with its WHY).
|
||||
- **Guarded file download (v0.124.0)**: `handler_export_download.go` — the canonical shape for streaming a server-side file to the browser: accept a BASENAME only (shape regexp + no separators/`..`), `filepath.Join` then assert `filepath.Dir(path) == dir`, `io.Copy` (never ReadAll), `Content-Disposition: attachment`, remove after a successful stream, TTL sweep (`sweepFabDownloads(dir, now, maxAge, logger)` — now injected for tests). Red-proof the guard by loosening to prefix-matching (the `..` case must fail).
|
||||
- **Backups sub-page data**: `backupsCommonData(page, title, r)` + `backupsOffboxData(data)` (handlers.go) — the ONLY builders for the four `/backups*` pages; a new backups section extends these, never re-derives in a page handler. Section moves are gated by `scripts/backups_split_move_check.py` (one-shot vs df7ad37).
|
||||
- **Consequential-action confirm (LIGHT)**: `felhomConfirm(el, question, onYes)` in layout.html (v0.123.0) — the trigger swaps in place to "kérdés + Igen/Mégse"; form buttons opt in with `data-confirm="…"` (delegated listener, `requestSubmit` keeps formaction/name-value). NEVER native `confirm()`/`prompt()` (OS-modals freeze browser automation — drill F-11; `native_confirm_gate.py` enforces). Heavy destructive flows keep the `.confirm-overlay` `openDialog` pattern.
|
||||
|
||||
@@ -337,9 +337,9 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte
|
||||
|
||||
**Encryption:** Optional AES-256-CTR + HMAC-SHA256 with scrypt key derivation (N=32768). Format: `"FABE"` magic header + salt + IV + encrypted tar.gz + HMAC tag. Streaming for multi-GB files.
|
||||
|
||||
**Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped.
|
||||
**Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data (docker volumes stream via `docker cp` through a stopped helper container — v0.125.0; never a `docker run -v` host mount, which strands data on containerized controllers) → **fail-loud bundle assertion** (every manifest-claimed tar must exist non-empty — no hollow bundles) → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped.
|
||||
|
||||
**Import flow:** Decrypt if needed → extract → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD or volumes) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI.
|
||||
**Import flow:** Decrypt if needed → extract → **validate-before-destroy** (v0.125.0: every manifest-claimed data tar must be present non-empty BEFORE the app is stopped or any volume removed — hollow bundles from containerized ≤0.124.0 exporters are refused with the app untouched) → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD tars; volumes populate via `docker cp` streaming) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI.
|
||||
|
||||
**Architecture:** `internal/appexport/` package with `ExportStackProvider` adapter interface (same pattern as `backup.StackDataProvider`). `exportAdapter` in `main.go` bridges `stacks.Manager` to the provider.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user