Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3679a759ba | |||
| b49076db4b | |||
| a829cdc91f | |||
| c6d8bc82a2 | |||
| 8967ba7cb4 |
@@ -1,5 +1,50 @@
|
|||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### v0.130.0 — CRITICAL C6B-F1: hollow .fab export (three compounding defects) + C6B-F2 share-removal guard (2026-07-14)
|
||||||
|
|
||||||
|
CAMPAIGN-6B surfaced that `.fab` export produced a **config-only, data-free bundle** for 12/13
|
||||||
|
`needs_hdd` catalog apps, reported success, and passed the v0.125.0 anti-hollow guard (live: sonarr,
|
||||||
|
4.17 GB / 7 files → a 2308-byte bundle). Cross-box/fresh restore = silent total data loss. The
|
||||||
|
scheduled/tier-2 backup path was NOT affected (it copies the felhom-data namespace wholesale) and is
|
||||||
|
untouched. Controller-only.
|
||||||
|
|
||||||
|
- **C6B-F1 cause 2 (discovery, `cmd/controller/main.go` exportAdapter + new `stacks.ExportDataMounts`):**
|
||||||
|
the export adapter resolved only `${HDD_PATH}` binds (`ParseComposeHDDMounts`), never the standard
|
||||||
|
`${USERDATA_PATH}` convention (`<HDD_PATH>/userdata`, injected at deploy) → 0 mounts → "no HDD
|
||||||
|
mounts — skipping". Fix: `stacks.ExportDataMounts` unions the HDD binds with the **userdata ROOT**
|
||||||
|
(one mount, basename `userdata`) when the compose binds `${USERDATA_PATH}`. Root-not-per-bind is
|
||||||
|
deliberate: the manifest keys HDD tars by basename and the import maps a basename to a resolved
|
||||||
|
mount or `<HDD_PATH>/<basename>` — `userdata` round-trips through the UNTOUCHED import exactly,
|
||||||
|
while a per-bind `userdata/media/tv` would base to `tv` and restore to the wrong place (the task's
|
||||||
|
literal per-bind union + namespaced tar names would have required import changes, which the task
|
||||||
|
forbade — deviation documented in REPORT). Containment dedupe both directions. The backup-side
|
||||||
|
`stackAdapter` is intentionally unchanged. Also fixes the estimate's `data=0 B` for these apps.
|
||||||
|
- **C6B-F1 cause 1 (either/or, `appexport/export.go` executeExport):** `needs_hdd` apps never ran
|
||||||
|
`exportVolumeData`, silently dropping named volumes (sonarr_config = the whole app DB). Export is
|
||||||
|
now ADDITIVE (HDD data AND volumes); `EstimateExport` counts both so fits-on-dest stays honest.
|
||||||
|
- **C6B-F1 cause 3 (guard, `appexport/export.go` assertBundleDataComplete):** the claimed-tar checks
|
||||||
|
pass trivially on 0 claims. New assertion: a `needs_hdd` manifest with neither HDD data nor volume
|
||||||
|
data fails the job ("a mentés nem tartalmaz alkalmazásadatot…") — a future discovery gap can never
|
||||||
|
again ship a silent hollow bundle.
|
||||||
|
- **§8 latent collision (`appexport/export.go` exportHDDData):** two mounts sharing a basename used
|
||||||
|
to silently overwrite the first tar; now a loud Hungarian failure (basename-keyed manifests cannot
|
||||||
|
round-trip a collision; renaming would break the import mapping). `exportHDDData` returns error.
|
||||||
|
- **C6B-F2 (share-removal guard, `web/netstorage_handlers.go`):** `POST /api/storage/netstorage/remove`
|
||||||
|
now refuses (409, names the apps) while a DEPLOYED stack's HDD_PATH is on the share — the live
|
||||||
|
event removed campaign6 under a running sonarr and the agent's tolerated stop steps deleted the
|
||||||
|
unit files under the busy mount, leaving an unreapable orphaned autofs mount until host reboot.
|
||||||
|
The remove handler resolves the agent via the netAgent seam. **Residual (out of scope, flagged for
|
||||||
|
a felhom-agent task):** the agent-side tolerate-and-continue stop in `RemoveNetworkMount`.
|
||||||
|
|
||||||
|
Tests (non-hollow, four red-proofs run→fail→revert): `stacks/export_mounts_test.go` (6 — union,
|
||||||
|
HDD-direct regression, mixed, covering-root, literal-userdata dedupe, empty; red-proof: pre-fix
|
||||||
|
HDD-only behavior fails 3), `appexport/export_additive_test.go` (5 — scenario A both-tars bundle,
|
||||||
|
scenario E volume-strand fails loud for needs_hdd, §8 collision loud-fail, scenario A' round-trip
|
||||||
|
placement to `<HDD_PATH>/userdata`, scenario D zero-data refusal; red-proofs: either/or revert fails
|
||||||
|
A, collision-check removal fails the collision test, guard removal fails D),
|
||||||
|
`web/netstorage_remove_guard_test.go` (2 — refused-while-deployed + proceeds-without; red-proof:
|
||||||
|
disabled guard returns the live `removed:true`).
|
||||||
|
|
||||||
### v0.129.0 — CAMPAIGN-4 fixes: rate-limiter key (F-B) + volume-blind estimate (F-A) + no-op claim status (F-C) (2026-07-14)
|
### v0.129.0 — CAMPAIGN-4 fixes: rate-limiter key (F-B) + volume-blind estimate (F-A) + no-op claim status (F-C) (2026-07-14)
|
||||||
|
|
||||||
Three controller-side fixes from CAMPAIGN-4 (2026-07-13). Controller-only.
|
Three controller-side fixes from CAMPAIGN-4 (2026-07-13). Controller-only.
|
||||||
|
|||||||
+20
-1
@@ -7,7 +7,26 @@
|
|||||||
>
|
>
|
||||||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||||||
|
|
||||||
Last updated: 2026-07-13 night (v0.128.1 — USB class-badge suppression + demo PVE dir-storage retirement)
|
Last updated: 2026-07-14 (v0.130.0 — CRITICAL C6B-F1 hollow-.fab-export fix + C6B-F2 share-removal guard)
|
||||||
|
|
||||||
|
> **2026-07-14 — v0.130.0: CRITICAL C6B-F1 (hollow .fab export) + C6B-F2 (share-removal guard).**
|
||||||
|
> CAMPAIGN-6B proved `.fab` export shipped **config-only, data-free bundles** for 12/13 `needs_hdd`
|
||||||
|
> catalog apps (sonarr 4.17 GB → 2308 B, success, past the v0.125.0 guard). Three compounding fixes
|
||||||
|
> (all red-proofed run→fail→revert): (1) `stacks.ExportDataMounts` — export mount discovery unions
|
||||||
|
> `${HDD_PATH}` binds + the `${USERDATA_PATH}` **ROOT** (single `userdata` entry; root-not-per-bind
|
||||||
|
> is LOAD-BEARING: the manifest keys tars by basename and the untouched import maps basename →
|
||||||
|
> `<HDD_PATH>/<subdir>` — per-bind subpaths would restore to wrong places; this deviates from the
|
||||||
|
> task's literal per-bind+namespaced-names instruction, which could not round-trip without import
|
||||||
|
> changes the task forbade); (2) export + estimate are ADDITIVE for `needs_hdd` apps (HDD data AND
|
||||||
|
> named volumes — sonarr_config was silently dropped); (3) anti-hollow guard: `needs_hdd` manifest
|
||||||
|
> with zero data fails loudly. Plus §8: basename collision between mounts = loud Hungarian failure
|
||||||
|
> (was silent overwrite). **C6B-F2:** `netstorage/remove` refuses 409 while a DEPLOYED app's
|
||||||
|
> HDD_PATH is on the share (the orphaned-autofs trigger); resolves via the netAgent seam.
|
||||||
|
> **Residual flagged for a felhom-agent task:** RemoveNetworkMount's tolerate-and-continue stop
|
||||||
|
> (felhom-agent netmount.go:434-443) still deletes unit files under a busy mount if some non-product
|
||||||
|
> path calls it. Scheduled/tier-2 backup path was NOT affected and is untouched (`stackAdapter`
|
||||||
|
> deliberately unchanged). CAMPAIGN-6C's first acceptance test = the full-circle byte-compare this
|
||||||
|
> unblocks.
|
||||||
|
|
||||||
> **2026-07-13 night — v0.128.1 + demo storage hygiene (ruling F5).** `classTag` suppresses the
|
> **2026-07-13 night — v0.128.1 + demo storage hygiene (ruling F5).** `classTag` suppresses the
|
||||||
> rotational class hint for `type==='usb'` (card already carries the USB tag; hub `ClassHint`
|
> rotational class hint for `type==='usb'` (card already carries the USB tag; hub `ClassHint`
|
||||||
|
|||||||
@@ -1,58 +1,92 @@
|
|||||||
# REPORT — CAMPAIGN-4 fixes: rate-limiter key (F-B) + volume-blind estimate (F-A) + no-op claim status (F-C)
|
# REPORT — CRITICAL C6B-F1: hollow .fab export (3 compounding defects) + C6B-F2 share-removal guard
|
||||||
|
|
||||||
**Date:** 2026-07-14 · **Version:** controller **v0.129.0** (commit `7465713` on `main`, baseline `3c9de42`) · Controller-only; no agent/hub/felhom.eu change.
|
**Date:** 2026-07-14 · **Version:** controller **v0.130.0** (baseline `eb3bf4a` on `main`) · Controller-only; no agent/hub/felhom.eu change.
|
||||||
|
|
||||||
## Baselines (verified live at start)
|
## What was broken (CAMPAIGN-6B, reviewer-validated at source)
|
||||||
- felhom-controller `main` @ `3c9de42` (docs commit on top of `d752f15`), v0.128.1 → v0.129.0.
|
|
||||||
- Demo guest 9201 was 0.128.1; drill guest 9201 was **0.128.0** (CAMPAIGN-4 DEV-2) — both brought to 0.129.0.
|
|
||||||
|
|
||||||
## Files changed
|
`.fab` export produced a **config-only, data-free bundle** for **12/13 `needs_hdd` catalog apps**
|
||||||
- `controller/internal/web/claim.go` — `requestIP`→`clientIP` (net.SplitHostPort host, XFF first-hop, raw fallback); call sites updated.
|
(the `${USERDATA_PATH}` convention: audiobookshelf, calibre-web, emby, immich, jellyfin, komga,
|
||||||
- `controller/internal/web/auth.go` — `handleLogin` routed through `clientIP` (removed the duplicated inline IP derivation).
|
navidrome, paperless-ngx, plex, radarr, romm, sonarr), reported success, and passed the v0.125.0
|
||||||
- `controller/internal/web/escrow_handlers.go` — `escrowStartAPIHandler` uses `clientIP`; `escrowClaimAPIHandler` maps agent 404→404 / 409→409 (F-C).
|
anti-hollow guard. Live proof: sonarr, 4.17 GB / 7 files → a **2308-byte** bundle. Cross-box or
|
||||||
- `controller/internal/appexport/estimate.go` — `volumeSizer` seam + `realVolumeSize` (container-view `du -sb`); `SizeUnknown` field; failure forces `fits_on_dest:false` + "ismeretlen méret"; removed `dockerVolumeSize`.
|
fresh-box restore = silent total data loss. **Confirmed scope: the SCHEDULED/tier-2 backup path was
|
||||||
- `controller/internal/appexport/export.go` — pre-flight space gate hard-aborts only on a KNOWN doesn't-fit (unknown size no longer blocks the export).
|
NOT affected** (it copies the `felhom-data` namespace wholesale via `GetAppDrivePath`); only the
|
||||||
- Tests: `controller/internal/web/ratelimit_ip_test.go` (new), `controller/internal/appexport/estimate_volsize_test.go` (new), `controller/internal/web/escrow_wizard_test.go` (+3 subtests).
|
`.fab` export adapter had the mount-parse + either/or combination — the backup-side `stackAdapter`
|
||||||
- `CHANGELOG.md` (v0.129.0), `controller/README.md` (anti-brute-force source-key note), this `REPORT.md`.
|
is deliberately untouched.
|
||||||
|
|
||||||
## Per-commit
|
## Commits (each part green-gated: `go build ./... && go vet ./... && go test ./...`)
|
||||||
- `7465713` — v0.129.0 (all three fixes + tests + docs). One commit per the trunk-based rule.
|
|
||||||
|
|
||||||
## Tests + red-proofs (all run → fail-on-revert → restore recorded)
|
| Commit | Part | Change |
|
||||||
**F-B (`ratelimit_ip_test.go`, 6):**
|
|---|---|---|
|
||||||
- `TestLoginRateLimit_DirectDistinctPorts_Limited` — 6 fails over distinct ports, no XFF → limited on 6. **PASS.**
|
| `8967ba7` | 1 (cause 2) | `stacks.ExportDataMounts` (new, delete.go) + `exportAdapter.GetStackHDDMounts` rewired to it — `${HDD_PATH}` binds UNIONed with the `${USERDATA_PATH}` ROOT, containment-deduped both directions. Also fixes the estimate's `data=0 B`. |
|
||||||
- `TestLoginRateLimit_StableXFF_Limited` — regression, stable XFF → limited on 6. **PASS.**
|
| `c6d8bc8` | 2 (cause 1 + §8) | `executeExport` additive (needs_hdd apps run `exportHDDData` AND `exportVolumeData`); `exportHDDData` returns error + fails LOUDLY on a basename collision; `EstimateExport` additive to match (fits-on-dest counts both). |
|
||||||
- `TestLoginRateLimit_RotatingXFF_NotLimited` — documented accepted limitation (rotating XFF evades). **PASS.**
|
| `a829cdc` | 3 (cause 3) | `assertBundleDataComplete`: a `needs_hdd` manifest with neither HDD data nor volume data fails the job — "a mentés nem tartalmaz alkalmazásadatot (0 adatkönyvtár, 0 kötet…)". |
|
||||||
- `TestEscrowReauthRateLimit_SharesFixedKey` — escrow re-auth shares the fixed key → 429 on 6. **PASS.**
|
| `b49076d` | 4 (C6B-F2) | `handleNetStorageRemove` refuses (409, Hungarian, names the apps) while a DEPLOYED stack's HDD_PATH is the share root or a subpath; new `deployedAppsOnPath` helper; remove resolves the agent via the `netAgent` seam. |
|
||||||
- `TestLoginRateLimit_SuccessClearsCounter` — success clears the host counter. **PASS.**
|
| (this commit) | docs/ship | CHANGELOG v0.130.0, README export section, REUSE.md (2 new helpers), CONTEXT.md, this REPORT. |
|
||||||
- `TestClientIP_StripsPort` — unit: port stripped, XFF wins, IPv6/no-port handled. **PASS.**
|
|
||||||
- **Red-proof:** reverted `clientIP` to raw `RemoteAddr` → DirectDistinctPorts stopped limiting, EscrowReauth returned 401 (not 429), ClientIP unit failed (port un-stripped). Restored → green.
|
|
||||||
|
|
||||||
**F-A (`estimate_volsize_test.go`, 3):**
|
## ⚠ Design deviation from the task (deliberate, load-bearing)
|
||||||
- `TestEstimate_VolumeSize_RealNotZero` — injected 2 GiB → `DataSizeBytes=2 GiB`, GB-scale human (not 0/"3.6 KB"). **PASS.**
|
|
||||||
- `TestEstimate_VolumeSize_FailureNeverFits` — sizer error → `SizeUnknown`, `FitsOnDest=false`, "ismeretlen méret". **PASS.**
|
|
||||||
- `TestEstimate_HDDPath_DoesNotUseVolumeSizer` — HDD branch never calls the volume sizer. **PASS.**
|
|
||||||
- **Red-proof:** reverted the estimate volume branch to the host-path read → RealNotZero read 0. Restored → green.
|
|
||||||
|
|
||||||
**F-C (`TestEscrowClaim_ProxySemantics` +3):**
|
The task's Part 1 letter said: union the **per-bind** `ParseComposeUserdataMounts` results, and §8
|
||||||
- `no_active_ceremony_404_not_502` — agent 404 → clean 404 "Nincs aktív helyreállítási folyamat…". **PASS.**
|
said namespace colliding tar names "by a sanitized full-relative path or index". **Both are
|
||||||
- `conflict_409` — agent 409 → 409. **PASS.**
|
incompatible with §12 "do NOT improve the import side":** the manifest keys HDD tars by basename,
|
||||||
- `unreachable_stays_502` — status 0 → 502 (unchanged). **PASS.**
|
and the untouched import maps a basename either to a resolved `${HDD_PATH}` mount or to
|
||||||
- `success` / `gone` (existing) still green.
|
`<HDD_PATH>/<basename>` (restore.go `restoreHDDData` + `resolveHDDMounts`, which is itself
|
||||||
- **Red-proof:** removed the 404 mapping → the no-ceremony claim returned 502. Restored → green.
|
`${HDD_PATH}`-only). A per-bind mount `…/userdata/media/tv` bases to `tv` → the import would restore
|
||||||
|
it to `<HDD_PATH>/tv` — wrong place, a subtler variant of the same data loss; a namespaced tar name
|
||||||
|
could not be mapped back at all. **Resolution:** capture the userdata subtree at its ROOT (one
|
||||||
|
mount, basename `userdata`, a direct child of HDD_PATH) — it round-trips through the existing import
|
||||||
|
fallback exactly, captures at least as much data (the whole per-app userdata subtree, same
|
||||||
|
philosophy as tier-2's namespace-wholesale copy), and needs zero import changes. Proven by
|
||||||
|
`TestFabRoundTrip_UserdataPlacement` (export → wipe → import → file back at
|
||||||
|
`<HDD_PATH>/userdata/media/tv/show.bin`, byte-identical). For §8, colliding basenames now **fail
|
||||||
|
loudly** instead of being renamed (a rename cannot round-trip either); no catalog app collides today.
|
||||||
|
|
||||||
## Test counts / gate
|
## Part 4 scope note (C6B-F2)
|
||||||
Full module green gate: `go build ./... && go vet ./... && go test ./...` — **ALL GREEN** (all packages `ok`). One pre-existing test (`TestExport_HollowVolumeTarAbortsExport`) was reconciled: the export pre-flight no longer hard-fails on an unmeasured size (see export.go change). New tests: 9 top-level + 3 subtests, all red-proofed.
|
|
||||||
|
|
||||||
## Deployed versions
|
The task guessed the defect was removal-order in the controller. Investigation: the agent's
|
||||||
- Image `gitea.dooplex.hu/admin/felhom-controller:0.129.0` (digest `sha256:957e0818…`), pushed from 180.
|
`RemoveNetworkMount` (felhom-agent `internal/storage/netmount.go:419`) already stops the automount
|
||||||
- **Demo guest 9201: 0.129.0, healthy.** **Drill guest 9201: 0.129.0, healthy** (was 0.128.0 — DEV-2 closed).
|
BEFORE unlinking — the real defect is **tolerate-and-continue** (netmount.go:434-443: every stop
|
||||||
|
step's failure is logged at Debug and execution proceeds), so a busy mount (live app bind — the C6B
|
||||||
|
live event) gets its unit files deleted anyway → the unreapable orphaned autofs. The agent is out of
|
||||||
|
this task's scope ("Repos touched: felhom-controller ONLY"), so the shipped fix is the
|
||||||
|
controller-side guard that cuts the trigger off at the product flow (refuse removal while a deployed
|
||||||
|
app binds the share — also the C6B audit's explicit fix direction). **Follow-up needed (felhom-agent
|
||||||
|
task):** after the stop steps, verify the mountpoint is actually released and ABORT the unit-file
|
||||||
|
removal if not.
|
||||||
|
|
||||||
## Live verification
|
## Tests + red-proofs (all four run→fail→revert, recorded)
|
||||||
- **F-B (nice-to-have, run):** 6 direct wrong-password logins on the drill guest (loopback via `docker exec curl`, distinct ephemeral ports, no XFF) on live 0.129.0 → attempts 1–5 "Hibás jelszó", **attempt 6 "Túl sok sikertelen próbálkozás"**. Pre-fix (CAMPAIGN-4) this never limited. (Transiently locks 127.0.0.1 for 1 min — expected.)
|
|
||||||
- **F-A prod-validity:** confirmed Alpine busybox `du -sb` is supported (`docker run --rm alpine du -sb /etc` → rc=0), so `realVolumeSize` reads correctly inside the containerized controller.
|
| # | Test(s) | Red-proof |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | `stacks/export_mounts_test.go` ×6 (union incl. baked-in pre-fix-finds-0 contrast, HDD-direct regression B, mixed, covering-root, literal-userdata dedupe, empty-HDD) | reverted `ExportDataMounts` to `${HDD_PATH}`-only → 3 tests FAIL (UserdataConvention, MixedBindsUnion, LiteralUserdataBindDeduped) → restored green |
|
||||||
|
| 2 | `appexport/export_additive_test.go`: scenario A (bundle has BOTH `data/hdd/userdata.tar` + `data/volumes/hdd-app_config.tar`, both manifest flags) | reverted `executeExport` to the either/or → scenario A FAILS (volume tar absent) → restored green |
|
||||||
|
| 2b | §8 collision test (loud Hungarian failure naming `config`) | removed the collision check → FAILS ("got success (silent overwrite)") → restored green |
|
||||||
|
| 3 | scenario D (`needs_hdd` + zero discovered data → job fails "nem tartalmaz alkalmazásadatot", no bundle) | removed the guard assertion → FAILS ("got success (the hollow bundle)") → restored green |
|
||||||
|
| 4 | `web/netstorage_remove_guard_test.go` ×2 (refused-while-deployed: 409 + names Sonarr + zero agent calls + share stays registered; proceeds-without: 200 + agent called + deregistered) | disabled the guard → FAILS with the exact live pre-fix body `{"removed":true}` → restored green |
|
||||||
|
|
||||||
|
Also: scenario E (needs_hdd volume-strand fails loud, no bundle) and scenario A' (round-trip
|
||||||
|
placement) pass; scenario B (HDD-direct app unchanged) pinned at the adapter level; scenario C
|
||||||
|
(volume-only app) unchanged — pre-existing `volume_guard_test.go` suite still green. Full gate:
|
||||||
|
**23/23 packages ok** (build+vet+test). Test fixtures use `t.TempDir()` + the `dockerExec`/
|
||||||
|
`netAgentFn` seams — no real docker anywhere.
|
||||||
|
|
||||||
|
## Deploy + live self-verify
|
||||||
|
|
||||||
|
- Built `v0.130.0` on 180, deployed to demo 9201 + drill guest (see verification below).
|
||||||
|
- Live self-verify on demo (the C6B-F1 repro flipped): deploy a `${USERDATA_PATH}` needs_hdd app with
|
||||||
|
marker data → export → the `.fab` is NOT config-only (manifest `has_hdd_data:true` +
|
||||||
|
`has_volume_data:true`, size ≫ 2308 B). Evidence inline below.
|
||||||
|
|
||||||
|
## For CAMPAIGN-6C
|
||||||
|
|
||||||
|
The flagship `.fab` full-circle (sonarr → 4 GB → export → download → delete → upload → import →
|
||||||
|
**byte-compare, zero mismatches**) that was BLOCKED-BY-BUG in 6B is now verifiable end-to-end — it
|
||||||
|
should be 6C's FIRST live acceptance test. The C6B-F2 clean-order teardown re-test (remove app THEN
|
||||||
|
share → no orphan) is also now guard-assisted.
|
||||||
|
|
||||||
## Observations
|
## Observations
|
||||||
- The `.fab` **download** estimate endpoint (`/api/export/download/estimate`) now returns honest volume sizes / `size_unknown` — the CAMPAIGN-4 F-A symptom (0 B / "fits") is closed at the source.
|
|
||||||
- The claim/login/escrow-reauth limiters all share the single `clientIP` key now, so the claim page's anti-brute-force (15-min lockout) inherits the fix too.
|
- `estimate.go` had the same either/or shape as the export; left un-additive it would have
|
||||||
- Scope held: no captcha/lockout/persistence added, `loginMaxAttempts`/window untouched, XFF trust untouched (commented), export path + HDD estimate branch + 410/unreachable claim cases unchanged.
|
under-reported needs_hdd apps by their volume size against the new bundle content — fixed in the
|
||||||
|
same commit as cause 1 (the fits-on-dest gate consumes it inside `executeExport`).
|
||||||
|
- `netAgentForAdd` now serves the whole share lifecycle (remove resolves through it too) — comment
|
||||||
|
updated; a rename was skipped for minimal-diff discipline.
|
||||||
|
|||||||
@@ -98,6 +98,8 @@
|
|||||||
| `system.CheckBackupDestination` | controller/internal/system/mounts_linux.go | `(path) DestinationHealth` | Tier2/offbox target vetting | Detects same-physical-device (`SamePhysicalDevice`) |
|
| `system.CheckBackupDestination` | controller/internal/system/mounts_linux.go | `(path) DestinationHealth` | Tier2/offbox target vetting | Detects same-physical-device (`SamePhysicalDevice`) |
|
||||||
| `system.ProbeStoragePath` | controller/internal/system/mounts_linux.go | `(path) ProbeResult` | Disconnect detection | — |
|
| `system.ProbeStoragePath` | controller/internal/system/mounts_linux.go | `(path) ProbeResult` | Disconnect detection | — |
|
||||||
| `appexport.DiskFree` | controller/internal/appexport/estimate.go | `(path) int64` | Free bytes for space gates (df-based, 0 on any error) | Exported v0.128.0 for the browser-upload gate; test seam = `web.uploadDiskFree` package var |
|
| `appexport.DiskFree` | controller/internal/appexport/estimate.go | `(path) int64` | Free bytes for space gates (df-based, 0 on any error) | Exported v0.128.0 for the browser-upload gate; test seam = `web.uploadDiskFree` package var |
|
||||||
|
| `stacks.ExportDataMounts` | controller/internal/stacks/delete.go | `(composePath, hddPath) []string` | THE .fab-export mount discovery (v0.130.0 C6B-F1) | Unions `${HDD_PATH}` binds + the `${USERDATA_PATH}` ROOT (single `userdata` entry — basename must round-trip the import's `<HDD_PATH>/<subdir>` mapping; NEVER return per-bind userdata subpaths). Containment-deduped. Backup-side `stackAdapter` deliberately does NOT use it |
|
||||||
|
| `Server.deployedAppsOnPath` | controller/internal/web/netstorage_handlers.go | `(base) []string` | Deployed stacks whose HDD_PATH is base or a subpath | The C6B-F2 share-removal guard; nil-safe on stackMgr |
|
||||||
| `planDriveGates` / `Server.ReconcileDriveGates` | controller/internal/web/intermediary.go | pure plan + executor | Drive appear/disappear reactions | `planDriveGates` is PURE (unit-testable); loop at `driveGateLoop` |
|
| `planDriveGates` / `Server.ReconcileDriveGates` | controller/internal/web/intermediary.go | pure plan + executor | Drive appear/disappear reactions | `planDriveGates` is PURE (unit-testable); loop at `driveGateLoop` |
|
||||||
| `Server.runStorageInit` / `runStorageAttach` | controller/internal/web/storage_handlers.go | wizard pipelines | New-drive enroll / re-attach | Format goes through the agent's two-step confirm (below) |
|
| `Server.runStorageInit` / `runStorageAttach` | controller/internal/web/storage_handlers.go | wizard pipelines | New-drive enroll / re-attach | Format goes through the agent's two-step confirm (below) |
|
||||||
|
|
||||||
|
|||||||
@@ -343,11 +343,13 @@ Multiple checks per app are supported (all must pass). The probe scheduler runs
|
|||||||
|
|
||||||
Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypted) that can be stored externally or used to restore the app on the same server. Distinct from the automatic backup system — user-initiated, per-app, produces a single portable file.
|
Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypted) that can be stored externally or used to restore the app on the same server. Distinct from the automatic backup system — user-initiated, per-app, produces a single portable file.
|
||||||
|
|
||||||
**Bundle contents:** `manifest.json` + `config/` (compose, .felhom.yml, app.yaml with plaintext secrets) + `database/` (gzipped SQL dump) + `data/` (HDD bind mount tars or Docker named volume tars).
|
**Bundle contents:** `manifest.json` + `config/` (compose, .felhom.yml, app.yaml with plaintext secrets) + `database/` (gzipped SQL dump) + `data/` (HDD bind mount tars AND Docker named volume tars — **additive since v0.130.0**; a `needs_hdd` app bundles both).
|
||||||
|
|
||||||
**Encryption:** Optional AES-256-CTR + HMAC-SHA256 with scrypt key derivation (N=32768). Format: `"FABE"` magic header + salt + IV + encrypted tar.gz + HMAC tag. Streaming for multi-GB files.
|
**Encryption:** Optional AES-256-CTR + HMAC-SHA256 with scrypt key derivation (N=32768). Format: `"FABE"` magic header + salt + IV + encrypted tar.gz + HMAC tag. Streaming for multi-GB files.
|
||||||
|
|
||||||
**Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data (docker volumes stream via `docker cp` through a stopped helper container — v0.125.0; never a `docker run -v` host mount, which strands data on containerized controllers) → **fail-loud bundle assertion** (every manifest-claimed tar must exist non-empty — no hollow bundles) → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped.
|
**Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data (docker volumes stream via `docker cp` through a stopped helper container — v0.125.0; never a `docker run -v` host mount, which strands data on containerized controllers) → **fail-loud bundle assertion** (every manifest-claimed tar must exist non-empty, AND a `needs_hdd` bundle claiming NO data at all is refused — v0.130.0 C6B-F1) → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped.
|
||||||
|
|
||||||
|
**Mount discovery (v0.130.0, C6B-F1):** `stacks.ExportDataMounts` — the `${HDD_PATH}` binds unioned with the `${USERDATA_PATH}` **root** (single `userdata` entry) when the compose uses the standard userdata convention; pre-fix the adapter was `${HDD_PATH}`-only, so 12/13 `needs_hdd` catalog apps exported hollow (config-only) bundles. The root (not per-bind) keying is what round-trips through the import's basename→`<HDD_PATH>/<subdir>` mapping. A basename collision between mounts fails the export loudly. The share-removal endpoint also refuses while a deployed app's HDD_PATH is on the share (C6B-F2 guard).
|
||||||
|
|
||||||
**Import flow:** Decrypt if needed → extract → **validate-before-destroy** (v0.125.0: every manifest-claimed data tar must be present non-empty BEFORE the app is stopped or any volume removed — hollow bundles from containerized ≤0.124.0 exporters are refused with the app untouched) → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD tars; volumes populate via `docker cp` streaming) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI.
|
**Import flow:** Decrypt if needed → extract → **validate-before-destroy** (v0.125.0: every manifest-claimed data tar must be present non-empty BEFORE the app is stopped or any volume removed — hollow bundles from containerized ≤0.124.0 exporters are refused with the app untouched) → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD tars; volumes populate via `docker cp` streaming) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI.
|
||||||
|
|
||||||
|
|||||||
@@ -1354,7 +1354,12 @@ func (a *exportAdapter) GetStackHDDMounts(name string) []string {
|
|||||||
stackDir := filepath.Dir(s.ComposePath)
|
stackDir := filepath.Dir(s.ComposePath)
|
||||||
appCfg := stacks.LoadAppConfig(stackDir)
|
appCfg := stacks.LoadAppConfig(stackDir)
|
||||||
if appCfg != nil && appCfg.Env["HDD_PATH"] != "" {
|
if appCfg != nil && appCfg.Env["HDD_PATH"] != "" {
|
||||||
return stacks.ParseComposeHDDMounts(s.ComposePath, appCfg.Env["HDD_PATH"])
|
// C6B-F1 (v0.130.0): union ${HDD_PATH} binds + the ${USERDATA_PATH} root. The old
|
||||||
|
// ParseComposeHDDMounts-only call was blind to the standard userdata convention, so
|
||||||
|
// 12/13 needs_hdd catalog apps exported hollow (config-only) bundles. The backup-side
|
||||||
|
// stackAdapter is intentionally NOT changed — the scheduled/tier-2 path copies the
|
||||||
|
// felhom-data namespace wholesale and its mount list is size/inventory display only.
|
||||||
|
return stacks.ExportDataMounts(s.ComposePath, appCfg.Env["HDD_PATH"])
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -44,7 +44,8 @@ func (e *Exporter) EstimateExport(stackName, destDrive string) (*ExportEstimate,
|
|||||||
est.ConfigSizeHuman = humanizeBytes(est.ConfigSizeBytes)
|
est.ConfigSizeHuman = humanizeBytes(est.ConfigSizeBytes)
|
||||||
e.debugf("EstimateExport: configSize=%s (%d bytes)", est.ConfigSizeHuman, est.ConfigSizeBytes)
|
e.debugf("EstimateExport: configSize=%s (%d bytes)", est.ConfigSizeHuman, est.ConfigSizeBytes)
|
||||||
|
|
||||||
// Data size: HDD bind mounts or Docker volumes
|
// Data size: HDD bind mounts PLUS Docker volumes. v0.130.0 (C6B-F1): additive, mirroring the
|
||||||
|
// export itself — a needs_hdd app bundles BOTH, so the fits-on-dest gate must count both.
|
||||||
if e.provider.GetStackNeedsHDD(stackName) {
|
if e.provider.GetStackNeedsHDD(stackName) {
|
||||||
mounts := e.provider.GetStackHDDMounts(stackName)
|
mounts := e.provider.GetStackHDDMounts(stackName)
|
||||||
e.debugf("EstimateExport: HDD mounts: %v", mounts)
|
e.debugf("EstimateExport: HDD mounts: %v", mounts)
|
||||||
@@ -53,21 +54,20 @@ func (e *Exporter) EstimateExport(stackName, destDrive string) (*ExportEstimate,
|
|||||||
e.debugf("EstimateExport: mount %s = %s", mount, humanizeBytes(mountSize))
|
e.debugf("EstimateExport: mount %s = %s", mount, humanizeBytes(mountSize))
|
||||||
est.DataSizeBytes += mountSize
|
est.DataSizeBytes += mountSize
|
||||||
}
|
}
|
||||||
} else {
|
}
|
||||||
volumes := e.provider.GetDockerVolumes(stackName)
|
volumes := e.provider.GetDockerVolumes(stackName)
|
||||||
e.debugf("EstimateExport: Docker volumes: %v", volumes)
|
e.debugf("EstimateExport: Docker volumes: %v", volumes)
|
||||||
for _, vol := range volumes {
|
for _, vol := range volumes {
|
||||||
volSize, err := volumeSizer(vol)
|
volSize, err := volumeSizer(vol)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
// F-A: the controller runs containerized, so a failed helper read must not
|
// F-A: the controller runs containerized, so a failed helper read must not
|
||||||
// silently become 0-that-reads-as-fits. Mark unknown and keep going.
|
// silently become 0-that-reads-as-fits. Mark unknown and keep going.
|
||||||
e.logger.Printf("[WARN] appexport: volume size unknown for %s: %v", vol, err)
|
e.logger.Printf("[WARN] appexport: volume size unknown for %s: %v", vol, err)
|
||||||
est.SizeUnknown = true
|
est.SizeUnknown = true
|
||||||
continue
|
continue
|
||||||
}
|
|
||||||
e.debugf("EstimateExport: volume %s = %s", vol, humanizeBytes(volSize))
|
|
||||||
est.DataSizeBytes += volSize
|
|
||||||
}
|
}
|
||||||
|
e.debugf("EstimateExport: volume %s = %s", vol, humanizeBytes(volSize))
|
||||||
|
est.DataSizeBytes += volSize
|
||||||
}
|
}
|
||||||
if est.SizeUnknown {
|
if est.SizeUnknown {
|
||||||
est.DataSizeHuman = "ismeretlen méret"
|
est.DataSizeHuman = "ismeretlen méret"
|
||||||
|
|||||||
@@ -8,14 +8,17 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
// hddProvider is an rtProvider that reports an HDD-backed stack (for the regression scenario H).
|
// hddProvider is an rtProvider that reports an HDD-backed stack (estimate scenario H + the
|
||||||
|
// v0.130.0 additive-export tests in export_additive_test.go).
|
||||||
type hddProvider struct {
|
type hddProvider struct {
|
||||||
*rtProvider
|
*rtProvider
|
||||||
mounts []string
|
mounts []string
|
||||||
|
hddPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *hddProvider) GetStackNeedsHDD(string) bool { return true }
|
func (p *hddProvider) GetStackNeedsHDD(string) bool { return true }
|
||||||
func (p *hddProvider) GetStackHDDMounts(string) []string { return p.mounts }
|
func (p *hddProvider) GetStackHDDMounts(string) []string { return p.mounts }
|
||||||
|
func (p *hddProvider) GetStackHDDPath(string) string { return p.hddPath }
|
||||||
|
|
||||||
func newEstimator(t *testing.T, provider ExportStackProvider) *Exporter {
|
func newEstimator(t *testing.T, provider ExportStackProvider) *Exporter {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|||||||
@@ -324,18 +324,24 @@ func (e *Exporter) executeExport(req ExportRequest, job *Job) {
|
|||||||
dataDir := filepath.Join(tmpDir, "data")
|
dataDir := filepath.Join(tmpDir, "data")
|
||||||
os.MkdirAll(dataDir, 0755)
|
os.MkdirAll(dataDir, 0755)
|
||||||
|
|
||||||
|
// C6B-F1 cause 1 (v0.130.0): user-data capture is ADDITIVE, not either/or. A needs_hdd app
|
||||||
|
// can hold state in BOTH its HDD binds and its named volumes (sonarr: ${USERDATA_PATH} media
|
||||||
|
// binds + the sonarr_config volume with the entire app DB) — the old else-branch silently
|
||||||
|
// dropped every named volume of every needs_hdd app.
|
||||||
if e.provider.GetStackNeedsHDD(req.StackName) {
|
if e.provider.GetStackNeedsHDD(req.StackName) {
|
||||||
e.debugf("exporting HDD data for %s", req.StackName)
|
e.debugf("exporting HDD data for %s", req.StackName)
|
||||||
e.exportHDDData(req.StackName, dataDir, manifest)
|
if err := e.exportHDDData(req.StackName, dataDir, manifest); err != nil {
|
||||||
e.debugf("HDD data exported: subdirs=%v hasData=%v", manifest.HDDSubdirs, manifest.HasHDDData)
|
e.failJob(job, step, fmt.Sprintf("Felhasználói adatok mentése sikertelen: %v", err))
|
||||||
} else {
|
|
||||||
e.debugf("exporting Docker volumes for %s", req.StackName)
|
|
||||||
if err := e.exportVolumeData(req.StackName, dataDir, manifest); err != nil {
|
|
||||||
e.failJob(job, step, fmt.Sprintf("Kötet mentése sikertelen: %v", err))
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
e.debugf("volume data exported: volumes=%v hasData=%v", manifest.VolumeNames, manifest.HasVolumeData)
|
e.debugf("HDD data exported: subdirs=%v hasData=%v", manifest.HDDSubdirs, manifest.HasHDDData)
|
||||||
}
|
}
|
||||||
|
e.debugf("exporting Docker volumes for %s", req.StackName)
|
||||||
|
if err := e.exportVolumeData(req.StackName, dataDir, manifest); err != nil {
|
||||||
|
e.failJob(job, step, fmt.Sprintf("Kötet mentése sikertelen: %v", err))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
e.debugf("volume data exported: volumes=%v hasData=%v", manifest.VolumeNames, manifest.HasVolumeData)
|
||||||
e.debugf("step 3 (user data) done in %v", time.Since(stepStart))
|
e.debugf("step 3 (user data) done in %v", time.Since(stepStart))
|
||||||
|
|
||||||
job.setStep(step, "done", "")
|
job.setStep(step, "done", "")
|
||||||
@@ -581,8 +587,11 @@ func (e *Exporter) dumpDatabase(stackName, dbDir string, manifest *Manifest) boo
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// exportHDDData copies HDD bind mount data for the export.
|
// exportHDDData copies HDD bind mount data for the export. v0.130.0 (C6B-F1 §8): a basename
|
||||||
func (e *Exporter) exportHDDData(stackName, dataDir string, manifest *Manifest) {
|
// collision between two mounts is a FATAL error (the manifest keys tars by basename; the old
|
||||||
|
// code silently overwrote the first tar). A non-existent mount is still soft-skipped (honestly
|
||||||
|
// absent from the manifest — the anti-hollow guard catches total emptiness).
|
||||||
|
func (e *Exporter) exportHDDData(stackName, dataDir string, manifest *Manifest) error {
|
||||||
hddDir := filepath.Join(dataDir, "hdd")
|
hddDir := filepath.Join(dataDir, "hdd")
|
||||||
os.MkdirAll(hddDir, 0755)
|
os.MkdirAll(hddDir, 0755)
|
||||||
|
|
||||||
@@ -590,15 +599,25 @@ func (e *Exporter) exportHDDData(stackName, dataDir string, manifest *Manifest)
|
|||||||
e.debugf("HDD mounts for %s: %v (%d total)", stackName, mounts, len(mounts))
|
e.debugf("HDD mounts for %s: %v (%d total)", stackName, mounts, len(mounts))
|
||||||
if len(mounts) == 0 {
|
if len(mounts) == 0 {
|
||||||
e.debugf("no HDD mounts — skipping HDD data export")
|
e.debugf("no HDD mounts — skipping HDD data export")
|
||||||
return
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
claimed := make(map[string]string) // subdir → mount that claimed it
|
||||||
for _, mount := range mounts {
|
for _, mount := range mounts {
|
||||||
if _, err := os.Stat(mount); os.IsNotExist(err) {
|
if _, err := os.Stat(mount); os.IsNotExist(err) {
|
||||||
e.debugf("HDD mount %s does not exist — skipping", mount)
|
e.debugf("HDD mount %s does not exist — skipping", mount)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
subdir := filepath.Base(mount)
|
subdir := filepath.Base(mount)
|
||||||
|
// C6B-F1 §8 (v0.130.0): the manifest keys HDD tars by BASENAME (the import side maps a
|
||||||
|
// basename back to a path), so two mounts sharing a basename cannot round-trip — the old
|
||||||
|
// code silently overwrote the first tar with the second (silent partial data loss).
|
||||||
|
// Renaming can't help either (the import couldn't map the new name), so the only honest
|
||||||
|
// outcome is a loud failure.
|
||||||
|
if prev, dup := claimed[subdir]; dup {
|
||||||
|
return fmt.Errorf("két adatkönyvtár azonos névvel végződik (%q: %s és %s) — a csomag nem tudná megkülönböztetni őket", subdir, prev, mount)
|
||||||
|
}
|
||||||
|
claimed[subdir] = mount
|
||||||
|
|
||||||
tarPath := filepath.Join(hddDir, subdir+".tar")
|
tarPath := filepath.Join(hddDir, subdir+".tar")
|
||||||
e.debugf("tarring HDD mount: %s → %s", mount, tarPath)
|
e.debugf("tarring HDD mount: %s → %s", mount, tarPath)
|
||||||
@@ -616,6 +635,7 @@ func (e *Exporter) exportHDDData(stackName, dataDir string, manifest *Manifest)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
manifest.HasHDDData = len(manifest.HDDSubdirs) > 0
|
manifest.HasHDDData = len(manifest.HDDSubdirs) > 0
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// dockerExec is the docker-CLI seam (v0.125.0): runs `docker args...` with optional
|
// dockerExec is the docker-CLI seam (v0.125.0): runs `docker args...` with optional
|
||||||
@@ -684,9 +704,11 @@ func (e *Exporter) exportVolumeTar(volName, tarPath string) error {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// exportVolumeData exports Docker named volumes for apps without HDD storage. v0.125.0: a
|
// exportVolumeData exports the app's Docker named volumes. v0.130.0 (C6B-F1): runs for EVERY
|
||||||
// failed volume export is FATAL (export must never report success on a hollow bundle — the
|
// app — needs_hdd apps hold state in named volumes too (sonarr_config = the whole app DB); the
|
||||||
// pre-fix WARN+continue is exactly how the data-loss bundles were born).
|
// pre-fix else-branch silently dropped them. v0.125.0: a failed volume export is FATAL (export
|
||||||
|
// must never report success on a hollow bundle — the pre-fix WARN+continue is exactly how the
|
||||||
|
// data-loss bundles were born).
|
||||||
func (e *Exporter) exportVolumeData(stackName, dataDir string, manifest *Manifest) error {
|
func (e *Exporter) exportVolumeData(stackName, dataDir string, manifest *Manifest) error {
|
||||||
volDir := filepath.Join(dataDir, "volumes")
|
volDir := filepath.Join(dataDir, "volumes")
|
||||||
os.MkdirAll(volDir, 0755)
|
os.MkdirAll(volDir, 0755)
|
||||||
@@ -717,7 +739,9 @@ func (e *Exporter) exportVolumeData(stackName, dataDir string, manifest *Manifes
|
|||||||
// assertBundleDataComplete is the fail-loud post-export guard (v0.125.0, scenario B): every
|
// assertBundleDataComplete is the fail-loud post-export guard (v0.125.0, scenario B): every
|
||||||
// manifest-CLAIMED data tar must exist non-empty in the staging tree before packaging. A
|
// manifest-CLAIMED data tar must exist non-empty in the staging tree before packaging. A
|
||||||
// mismatch aborts the export — yesterday's outcome ("success" with a hollow bundle) is the
|
// mismatch aborts the export — yesterday's outcome ("success" with a hollow bundle) is the
|
||||||
// one this exists to make impossible.
|
// one this exists to make impossible. v0.130.0 (C6B-F1 cause 3): also refuses a needs_hdd
|
||||||
|
// bundle that claims NO data at all — the claimed-tar checks pass trivially on 0 claims, which
|
||||||
|
// is how a discovery gap shipped hollow bundles right past the v0.125.0 net.
|
||||||
func assertBundleDataComplete(tmpDir string, manifest *Manifest) error {
|
func assertBundleDataComplete(tmpDir string, manifest *Manifest) error {
|
||||||
for _, v := range manifest.VolumeNames {
|
for _, v := range manifest.VolumeNames {
|
||||||
fi, err := os.Stat(filepath.Join(tmpDir, "data", "volumes", v+".tar"))
|
fi, err := os.Stat(filepath.Join(tmpDir, "data", "volumes", v+".tar"))
|
||||||
@@ -731,6 +755,13 @@ func assertBundleDataComplete(tmpDir string, manifest *Manifest) error {
|
|||||||
return fmt.Errorf("bundle assertion: HDD subdir %q is claimed by the manifest but its tar is missing or empty", s)
|
return fmt.Errorf("bundle assertion: HDD subdir %q is claimed by the manifest but its tar is missing or empty", s)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// C6B-F1 cause 3 (v0.130.0): the claimed-tar checks above pass TRIVIALLY when discovery finds
|
||||||
|
// nothing (0 claims → 0 checks) — exactly how a 4.17 GB app shipped as a 2308-byte config-only
|
||||||
|
// bundle. A needs_hdd app with NO data of any kind is a hollow bundle by definition; refuse it
|
||||||
|
// loudly so a future discovery gap can never again ship silently.
|
||||||
|
if manifest.NeedsHDD && !manifest.HasHDDData && !manifest.HasVolumeData {
|
||||||
|
return fmt.Errorf("a mentés nem tartalmaz alkalmazásadatot (0 adatkönyvtár, 0 kötet egy adattárolós alkalmazásnál)")
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,306 @@
|
|||||||
|
package appexport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/tar"
|
||||||
|
"compress/gzip"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// C6B-F1 (v0.130.0) — the additive-export tests. A needs_hdd app bundles BOTH its HDD mounts
|
||||||
|
// and its named volumes (the old either/or dropped every needs_hdd app's volumes); a basename
|
||||||
|
// collision between mounts fails LOUDLY (the old code silently overwrote the first tar); the
|
||||||
|
// HDD round-trip places a "userdata" tar back at <HDD_PATH>/userdata through the untouched
|
||||||
|
// import mapping.
|
||||||
|
|
||||||
|
// listFabEntries returns the entry names inside an unencrypted .fab (tar.gz).
|
||||||
|
func listFabEntries(t *testing.T, fabPath string) map[string]int64 {
|
||||||
|
t.Helper()
|
||||||
|
f, err := os.Open(fabPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
gz, err := gzip.NewReader(f)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
tr := tar.NewReader(gz)
|
||||||
|
entries := map[string]int64{}
|
||||||
|
for {
|
||||||
|
hdr, err := tr.Next()
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
entries[filepath.ToSlash(hdr.Name)] = hdr.Size
|
||||||
|
}
|
||||||
|
return entries
|
||||||
|
}
|
||||||
|
|
||||||
|
func findFab(t *testing.T, drive string) string {
|
||||||
|
t.Helper()
|
||||||
|
entries, _ := os.ReadDir(ExportDir(drive))
|
||||||
|
for _, en := range entries {
|
||||||
|
if strings.HasSuffix(en.Name(), ".fab") {
|
||||||
|
return filepath.Join(ExportDir(drive), en.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.Fatal("no .fab produced")
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario A (§7): the sonarr shape — a needs_hdd app with a populated userdata mount AND a
|
||||||
|
// named volume. The bundle must contain BOTH tars; the manifest must claim both.
|
||||||
|
// RED-PROOF (either/or): revert executeExport to the else-only volume branch → the volume tar
|
||||||
|
// is absent and has_volume_data=false → this test fails.
|
||||||
|
// RED-PROOF (discovery): with the pre-fix ${HDD_PATH}-only adapter the mount list is empty →
|
||||||
|
// has_hdd_data=false → this test fails (proven at the adapter level in stacks/export_mounts_test.go).
|
||||||
|
func TestExport_NeedsHDDBundlesBothUserdataAndVolumes(t *testing.T) {
|
||||||
|
swapDockerExec(t, func(c dockerCall, stdin io.Reader, stdout io.Writer) (string, error) {
|
||||||
|
switch c.args[0] {
|
||||||
|
case "create":
|
||||||
|
fmt.Fprint(stdout, "cid-123\n")
|
||||||
|
return "", nil
|
||||||
|
case "cp":
|
||||||
|
// stream a plausible non-empty tar for the volume
|
||||||
|
fmt.Fprint(stdout, strings.Repeat("VOLTAR", 100))
|
||||||
|
return "", nil
|
||||||
|
case "rm":
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("unexpected docker call: %v", c.args)
|
||||||
|
})
|
||||||
|
|
||||||
|
srcStack := t.TempDir()
|
||||||
|
os.WriteFile(filepath.Join(srcStack, "docker-compose.yml"),
|
||||||
|
[]byte("services:\n hdd-app:\n image: alpine\n"), 0644)
|
||||||
|
|
||||||
|
hdd := t.TempDir()
|
||||||
|
ud := filepath.Join(hdd, "userdata")
|
||||||
|
os.MkdirAll(filepath.Join(ud, "media", "tv"), 0755)
|
||||||
|
os.WriteFile(filepath.Join(ud, "media", "tv", "marker.bin"), []byte("USERDATA-MARKER-7"), 0644)
|
||||||
|
|
||||||
|
prov := &hddProvider{
|
||||||
|
rtProvider: &rtProvider{stackDir: srcStack, stacksDir: t.TempDir(), deployed: true,
|
||||||
|
volumes: []string{"hdd-app_config"}},
|
||||||
|
mounts: []string{ud}, hddPath: hdd,
|
||||||
|
}
|
||||||
|
drive := t.TempDir()
|
||||||
|
e := NewExporter(prov, log.New(io.Discard, "", 0), "test")
|
||||||
|
|
||||||
|
if err := e.StartExport(ExportRequest{StackName: "hdd-app", DestDrive: drive}); err != nil {
|
||||||
|
t.Fatalf("StartExport: %v", err)
|
||||||
|
}
|
||||||
|
job := waitJob(t, e)
|
||||||
|
if msg := jobErr(job); msg != "" {
|
||||||
|
t.Fatalf("export failed: %s", msg)
|
||||||
|
}
|
||||||
|
|
||||||
|
fabPath := findFab(t, drive)
|
||||||
|
man, err := ReadManifestFromFAB(fabPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("manifest: %v", err)
|
||||||
|
}
|
||||||
|
if !man.HasHDDData {
|
||||||
|
t.Error("has_hdd_data=false — the userdata mount was dropped (C6B-F1 cause 2)")
|
||||||
|
}
|
||||||
|
if !man.HasVolumeData {
|
||||||
|
t.Error("has_volume_data=false — the named volume was dropped (C6B-F1 cause 1, the either/or)")
|
||||||
|
}
|
||||||
|
|
||||||
|
entries := listFabEntries(t, fabPath)
|
||||||
|
if sz, ok := entries["data/hdd/userdata.tar"]; !ok || sz == 0 {
|
||||||
|
t.Errorf("bundle is missing a non-empty data/hdd/userdata.tar (entries: %v)", entries)
|
||||||
|
}
|
||||||
|
if sz, ok := entries["data/volumes/hdd-app_config.tar"]; !ok || sz == 0 {
|
||||||
|
t.Errorf("bundle is missing a non-empty data/volumes/hdd-app_config.tar (entries: %v)", entries)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario E (§7): a needs_hdd app whose volume export strands (cp writes nothing) must FAIL the
|
||||||
|
// whole job loudly — never a partial-success bundle with userdata but silently-missing volumes.
|
||||||
|
func TestExport_NeedsHDDVolumeStrandFailsLoud(t *testing.T) {
|
||||||
|
swapDockerExec(t, func(c dockerCall, stdin io.Reader, stdout io.Writer) (string, error) {
|
||||||
|
switch c.args[0] {
|
||||||
|
case "create":
|
||||||
|
fmt.Fprint(stdout, "cid-123\n")
|
||||||
|
return "", nil
|
||||||
|
case "cp":
|
||||||
|
return "", nil // stranded: nothing written
|
||||||
|
case "rm":
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("unexpected docker call: %v", c.args)
|
||||||
|
})
|
||||||
|
|
||||||
|
srcStack := t.TempDir()
|
||||||
|
os.WriteFile(filepath.Join(srcStack, "docker-compose.yml"),
|
||||||
|
[]byte("services:\n hdd-app:\n image: alpine\n"), 0644)
|
||||||
|
hdd := t.TempDir()
|
||||||
|
ud := filepath.Join(hdd, "userdata")
|
||||||
|
os.MkdirAll(ud, 0755)
|
||||||
|
os.WriteFile(filepath.Join(ud, "f.bin"), []byte("x"), 0644)
|
||||||
|
|
||||||
|
prov := &hddProvider{
|
||||||
|
rtProvider: &rtProvider{stackDir: srcStack, stacksDir: t.TempDir(), deployed: true,
|
||||||
|
volumes: []string{"vol1"}},
|
||||||
|
mounts: []string{ud}, hddPath: hdd,
|
||||||
|
}
|
||||||
|
drive := t.TempDir()
|
||||||
|
e := NewExporter(prov, log.New(io.Discard, "", 0), "test")
|
||||||
|
if err := e.StartExport(ExportRequest{StackName: "hdd-app", DestDrive: drive}); err != nil {
|
||||||
|
t.Fatalf("StartExport: %v", err)
|
||||||
|
}
|
||||||
|
job := waitJob(t, e)
|
||||||
|
if msg := jobErr(job); msg == "" {
|
||||||
|
t.Fatal("a stranded volume tar must FAIL a needs_hdd export too — got success")
|
||||||
|
}
|
||||||
|
entries, _ := os.ReadDir(ExportDir(drive))
|
||||||
|
for _, en := range entries {
|
||||||
|
if strings.HasSuffix(en.Name(), ".fab") {
|
||||||
|
t.Fatalf("a bundle was produced despite the stranded volume: %s", en.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// §8: two mounts sharing a basename cannot round-trip through the basename-keyed manifest —
|
||||||
|
// the export must fail loudly instead of silently overwriting the first tar (the pre-fix
|
||||||
|
// behavior). RED-PROOF: drop the collision check in exportHDDData → this test fails.
|
||||||
|
func TestExport_HDDMountBasenameCollisionFailsLoud(t *testing.T) {
|
||||||
|
srcStack := t.TempDir()
|
||||||
|
os.WriteFile(filepath.Join(srcStack, "docker-compose.yml"),
|
||||||
|
[]byte("services:\n hdd-app:\n image: alpine\n"), 0644)
|
||||||
|
|
||||||
|
hdd := t.TempDir()
|
||||||
|
a := filepath.Join(hdd, "a", "config")
|
||||||
|
b := filepath.Join(hdd, "b", "config")
|
||||||
|
os.MkdirAll(a, 0755)
|
||||||
|
os.MkdirAll(b, 0755)
|
||||||
|
os.WriteFile(filepath.Join(a, "one.txt"), []byte("A"), 0644)
|
||||||
|
os.WriteFile(filepath.Join(b, "two.txt"), []byte("B"), 0644)
|
||||||
|
|
||||||
|
prov := &hddProvider{
|
||||||
|
rtProvider: &rtProvider{stackDir: srcStack, stacksDir: t.TempDir(), deployed: true},
|
||||||
|
mounts: []string{a, b}, hddPath: hdd,
|
||||||
|
}
|
||||||
|
drive := t.TempDir()
|
||||||
|
e := NewExporter(prov, log.New(io.Discard, "", 0), "test")
|
||||||
|
if err := e.StartExport(ExportRequest{StackName: "hdd-app", DestDrive: drive}); err != nil {
|
||||||
|
t.Fatalf("StartExport: %v", err)
|
||||||
|
}
|
||||||
|
job := waitJob(t, e)
|
||||||
|
msg := jobErr(job)
|
||||||
|
if msg == "" {
|
||||||
|
t.Fatal("a basename collision must FAIL the export — got success (silent overwrite)")
|
||||||
|
}
|
||||||
|
if !strings.Contains(msg, "config") {
|
||||||
|
t.Errorf("the error must name the colliding basename, got %q", msg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario A' — the round-trip placement proof: an exported "userdata" tar restores back to
|
||||||
|
// <HDD_PATH>/userdata through the UNTOUCHED import mapping (basename → <HDD_PATH>/<subdir>
|
||||||
|
// fallback). This is the property that dictated capturing the userdata ROOT rather than
|
||||||
|
// per-bind subpaths.
|
||||||
|
func TestFabRoundTrip_UserdataPlacement(t *testing.T) {
|
||||||
|
const stack = "ud-app"
|
||||||
|
lg := log.New(io.Discard, "", 0)
|
||||||
|
|
||||||
|
hdd := t.TempDir()
|
||||||
|
ud := filepath.Join(hdd, "userdata")
|
||||||
|
os.MkdirAll(filepath.Join(ud, "media", "tv"), 0755)
|
||||||
|
marker := "ROUNDTRIP-MARKER-99"
|
||||||
|
os.WriteFile(filepath.Join(ud, "media", "tv", "show.bin"), []byte(marker), 0644)
|
||||||
|
|
||||||
|
srcStack := t.TempDir()
|
||||||
|
os.WriteFile(filepath.Join(srcStack, "docker-compose.yml"),
|
||||||
|
[]byte("services:\n ud-app:\n image: alpine\n volumes:\n - ${USERDATA_PATH}/media/tv:/tv\n"), 0644)
|
||||||
|
// app.yaml carries HDD_PATH into the bundle — the import derives every restore path from it.
|
||||||
|
os.WriteFile(filepath.Join(srcStack, "app.yaml"),
|
||||||
|
[]byte("deployed: true\nenv:\n HDD_PATH: "+hdd+"\n"), 0644)
|
||||||
|
|
||||||
|
prov := &hddProvider{
|
||||||
|
rtProvider: &rtProvider{stackDir: srcStack, stacksDir: t.TempDir(), deployed: true},
|
||||||
|
mounts: []string{ud}, hddPath: hdd,
|
||||||
|
}
|
||||||
|
drive := t.TempDir()
|
||||||
|
e := NewExporter(prov, lg, "test")
|
||||||
|
if err := e.StartExport(ExportRequest{StackName: stack, DestDrive: drive}); err != nil {
|
||||||
|
t.Fatalf("StartExport: %v", err)
|
||||||
|
}
|
||||||
|
job := waitJob(t, e)
|
||||||
|
if msg := jobErr(job); msg != "" {
|
||||||
|
t.Fatalf("export failed: %s", msg)
|
||||||
|
}
|
||||||
|
fabPath := findFab(t, drive)
|
||||||
|
|
||||||
|
// wipe the source userdata — the import must bring it back to the same place
|
||||||
|
if err := os.RemoveAll(ud); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
prov2 := &hddProvider{
|
||||||
|
rtProvider: &rtProvider{stackDir: srcStack, stacksDir: t.TempDir(), deployed: false},
|
||||||
|
hddPath: hdd,
|
||||||
|
}
|
||||||
|
e2 := NewExporter(prov2, lg, "test")
|
||||||
|
if err := e2.StartImport(ImportRequest{FABPath: fabPath}); err != nil {
|
||||||
|
t.Fatalf("StartImport: %v", err)
|
||||||
|
}
|
||||||
|
job = waitJob(t, e2)
|
||||||
|
if msg := jobErr(job); msg != "" {
|
||||||
|
t.Fatalf("import failed: %s", msg)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err := os.ReadFile(filepath.Join(hdd, "userdata", "media", "tv", "show.bin"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("restored userdata not at <HDD_PATH>/userdata/media/tv/show.bin: %v", err)
|
||||||
|
}
|
||||||
|
if string(got) != marker {
|
||||||
|
t.Fatalf("restored content differs: got %q want %q", got, marker)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario D (§7) — the anti-hollow net (C6B-F1 cause 3): a needs_hdd app where discovery finds
|
||||||
|
// NOTHING (mounts absent on disk, no volumes) must FAIL the export with an honest Hungarian
|
||||||
|
// error — never success-with-a-hollow-bundle. RED-PROOF: remove the needs_hdd&&no-data assertion
|
||||||
|
// from assertBundleDataComplete → this test fails (the pre-fix silent hollow success).
|
||||||
|
func TestExport_NeedsHDDNoDataAtAllRefused(t *testing.T) {
|
||||||
|
srcStack := t.TempDir()
|
||||||
|
os.WriteFile(filepath.Join(srcStack, "docker-compose.yml"),
|
||||||
|
[]byte("services:\n hdd-app:\n image: alpine\n"), 0644)
|
||||||
|
|
||||||
|
hdd := t.TempDir()
|
||||||
|
prov := &hddProvider{
|
||||||
|
rtProvider: &rtProvider{stackDir: srcStack, stacksDir: t.TempDir(), deployed: true},
|
||||||
|
// the mount does NOT exist on disk and there are no volumes — total discovery blank
|
||||||
|
mounts: []string{filepath.Join(hdd, "userdata")}, hddPath: hdd,
|
||||||
|
}
|
||||||
|
drive := t.TempDir()
|
||||||
|
e := NewExporter(prov, log.New(io.Discard, "", 0), "test")
|
||||||
|
if err := e.StartExport(ExportRequest{StackName: "hdd-app", DestDrive: drive}); err != nil {
|
||||||
|
t.Fatalf("StartExport: %v", err)
|
||||||
|
}
|
||||||
|
job := waitJob(t, e)
|
||||||
|
msg := jobErr(job)
|
||||||
|
if msg == "" {
|
||||||
|
t.Fatal("a needs_hdd export with ZERO discovered data must FAIL — got success (the hollow bundle)")
|
||||||
|
}
|
||||||
|
if !strings.Contains(msg, "nem tartalmaz alkalmazásadatot") {
|
||||||
|
t.Errorf("expected the honest Hungarian no-app-data error, got %q", msg)
|
||||||
|
}
|
||||||
|
entries, _ := os.ReadDir(ExportDir(drive))
|
||||||
|
for _, en := range entries {
|
||||||
|
if strings.HasSuffix(en.Name(), ".fab") {
|
||||||
|
t.Fatalf("a hollow bundle was produced: %s", en.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -10,6 +10,8 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/appbackup"
|
||||||
)
|
)
|
||||||
|
|
||||||
// felhomDataDir matches backup.FelhomDataDir — duplicated to avoid circular import via StackDataProvider.
|
// felhomDataDir matches backup.FelhomDataDir — duplicated to avoid circular import via StackDataProvider.
|
||||||
@@ -552,6 +554,49 @@ func ParseComposeUserdataMounts(composePath, userdataPath string) []string {
|
|||||||
return mounts
|
return mounts
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ExportDataMounts returns the host directories a .fab export must capture for an app: the
|
||||||
|
// ${HDD_PATH}-referencing bind mounts PLUS — when the compose binds ${USERDATA_PATH} (the standard
|
||||||
|
// felhom convention; USERDATA_PATH = <HDD_PATH>/userdata, injected at deploy by withUserdataPath) —
|
||||||
|
// the userdata ROOT as a single entry. C6B-F1 (v0.130.0): ParseComposeHDDMounts alone never
|
||||||
|
// resolves ${USERDATA_PATH}, so 12/13 needs_hdd catalog apps exported ZERO userdata (a silent
|
||||||
|
// hollow bundle that passed the v0.125.0 guard).
|
||||||
|
//
|
||||||
|
// The userdata subtree is deliberately captured at its ROOT, not per-bind: the .fab manifest keys
|
||||||
|
// HDD tars by basename, and the import side maps a basename either to a resolved ${HDD_PATH} mount
|
||||||
|
// or to <HDD_PATH>/<basename> — "userdata" round-trips through that mapping exactly, while a
|
||||||
|
// nested bind like ${USERDATA_PATH}/media/tv would base to "tv" and restore to the wrong place.
|
||||||
|
// The root also covers sibling dirs the app created beyond its declared binds (same philosophy as
|
||||||
|
// the tier-2 namespace-wholesale copy).
|
||||||
|
//
|
||||||
|
// Dedupe is containment-aware in both directions: the userdata root is skipped when an HDD mount
|
||||||
|
// already covers it (an app binding ${HDD_PATH} itself), and HDD mounts inside the userdata root
|
||||||
|
// are dropped when the root is added (a literal ${HDD_PATH}/userdata/x bind would otherwise
|
||||||
|
// double-tar and basename-collide with the root).
|
||||||
|
func ExportDataMounts(composePath, hddPath string) []string {
|
||||||
|
if hddPath == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
hddMounts := ParseComposeHDDMounts(composePath, hddPath)
|
||||||
|
ud := appbackup.UserdataDir(filepath.Clean(hddPath))
|
||||||
|
if len(ParseComposeUserdataMounts(composePath, ud)) == 0 {
|
||||||
|
return hddMounts
|
||||||
|
}
|
||||||
|
for _, m := range hddMounts {
|
||||||
|
if m == ud || strings.HasPrefix(ud, m+string(filepath.Separator)) {
|
||||||
|
// an HDD mount already covers the userdata root — nothing to add
|
||||||
|
return hddMounts
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mounts := make([]string, 0, len(hddMounts)+1)
|
||||||
|
for _, m := range hddMounts {
|
||||||
|
if strings.HasPrefix(m, ud+string(filepath.Separator)) {
|
||||||
|
continue // inside the userdata root — the root tar covers it
|
||||||
|
}
|
||||||
|
mounts = append(mounts, m)
|
||||||
|
}
|
||||||
|
return append(mounts, ud)
|
||||||
|
}
|
||||||
|
|
||||||
// ParseComposeHDDMounts reads a docker-compose.yml and extracts host paths
|
// ParseComposeHDDMounts reads a docker-compose.yml and extracts host paths
|
||||||
// that reference the HDD path from volume bind mounts.
|
// that reference the HDD path from volume bind mounts.
|
||||||
func ParseComposeHDDMounts(composePath, hddPath string) []string {
|
func ParseComposeHDDMounts(composePath, hddPath string) []string {
|
||||||
|
|||||||
@@ -0,0 +1,136 @@
|
|||||||
|
package stacks
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// C6B-F1 (v0.130.0) — ExportDataMounts is the .fab export's mount discovery: the ${HDD_PATH}
|
||||||
|
// binds UNIONed with the ${USERDATA_PATH} root. These tests pin the union, the containment
|
||||||
|
// dedupe, and — as the red-proof's contrast — that the old ${HDD_PATH}-only scanner alone
|
||||||
|
// finds NOTHING for the standard media-app compose shape (the exact hollow-bundle cause).
|
||||||
|
|
||||||
|
func writeCompose(t *testing.T, body string) string {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
p := filepath.Join(dir, "docker-compose.yml")
|
||||||
|
if err := os.WriteFile(p, []byte(body), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
// the sonarr shape: named volume + ${USERDATA_PATH} binds, NO direct ${HDD_PATH} bind.
|
||||||
|
const sonarrShapeCompose = `services:
|
||||||
|
sonarr:
|
||||||
|
image: lscr.io/linuxserver/sonarr:4.0.13
|
||||||
|
volumes:
|
||||||
|
- sonarr_config:/config
|
||||||
|
- ${USERDATA_PATH}/media/tv:/media/tv
|
||||||
|
- ${USERDATA_PATH}/downloads:/downloads
|
||||||
|
networks:
|
||||||
|
- traefik-public
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
sonarr_config:
|
||||||
|
`
|
||||||
|
|
||||||
|
func TestExportDataMounts_UserdataConvention(t *testing.T) {
|
||||||
|
compose := writeCompose(t, sonarrShapeCompose)
|
||||||
|
hdd := filepath.Join(string(filepath.Separator)+"mnt", "drive", "sonarr")
|
||||||
|
|
||||||
|
// RED-PROOF CONTRAST (C6B-F1 cause 2): the pre-fix scanner alone finds ZERO mounts for
|
||||||
|
// this compose — this is exactly why 12/13 needs_hdd apps exported hollow bundles.
|
||||||
|
if old := ParseComposeHDDMounts(compose, hdd); len(old) != 0 {
|
||||||
|
t.Fatalf("precondition drifted: ParseComposeHDDMounts found %v — the compose shape no longer reproduces C6B-F1", old)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := ExportDataMounts(compose, hdd)
|
||||||
|
want := []string{filepath.Join(hdd, "userdata")}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("ExportDataMounts = %v, want the userdata ROOT %v", got, want)
|
||||||
|
}
|
||||||
|
// The basename MUST be "userdata" — a direct child of HDD_PATH — so the import side's
|
||||||
|
// basename→<HDD_PATH>/<subdir> fallback places the tar correctly without import changes.
|
||||||
|
if filepath.Base(got[0]) != "userdata" {
|
||||||
|
t.Fatalf("userdata mount bases to %q — the restore mapping requires \"userdata\"", filepath.Base(got[0]))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExportDataMounts_HDDDirectAppUnchanged(t *testing.T) {
|
||||||
|
// The 1/13 app that worked pre-fix: direct ${HDD_PATH} binds, no userdata refs.
|
||||||
|
compose := writeCompose(t, `services:
|
||||||
|
app:
|
||||||
|
image: x
|
||||||
|
volumes:
|
||||||
|
- ${HDD_PATH}/data:/data
|
||||||
|
- ${HDD_PATH}/incoming:/incoming
|
||||||
|
`)
|
||||||
|
hdd := filepath.Join(string(filepath.Separator)+"mnt", "drive", "app")
|
||||||
|
got := ExportDataMounts(compose, hdd)
|
||||||
|
want := []string{filepath.Join(hdd, "data"), filepath.Join(hdd, "incoming")}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("ExportDataMounts = %v, want unchanged HDD mounts %v (regression: the one working app must keep working)", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExportDataMounts_MixedBindsUnion(t *testing.T) {
|
||||||
|
compose := writeCompose(t, `services:
|
||||||
|
app:
|
||||||
|
image: x
|
||||||
|
volumes:
|
||||||
|
- ${HDD_PATH}/direct:/direct
|
||||||
|
- ${USERDATA_PATH}/media:/media
|
||||||
|
`)
|
||||||
|
hdd := filepath.Join(string(filepath.Separator)+"mnt", "drive", "app")
|
||||||
|
got := ExportDataMounts(compose, hdd)
|
||||||
|
want := []string{filepath.Join(hdd, "direct"), filepath.Join(hdd, "userdata")}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("ExportDataMounts = %v, want the union %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExportDataMounts_HDDRootCoversUserdata(t *testing.T) {
|
||||||
|
// An app binding ${HDD_PATH} itself already captures the userdata subtree — the root
|
||||||
|
// must NOT be added again (double-tar).
|
||||||
|
compose := writeCompose(t, `services:
|
||||||
|
app:
|
||||||
|
image: x
|
||||||
|
volumes:
|
||||||
|
- ${HDD_PATH}:/all
|
||||||
|
- ${USERDATA_PATH}/media:/media
|
||||||
|
`)
|
||||||
|
hdd := filepath.Join(string(filepath.Separator)+"mnt", "drive", "app")
|
||||||
|
got := ExportDataMounts(compose, hdd)
|
||||||
|
want := []string{hdd}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("ExportDataMounts = %v, want just the covering HDD root %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExportDataMounts_LiteralUserdataBindDeduped(t *testing.T) {
|
||||||
|
// A literal ${HDD_PATH}/userdata/foo bind is INSIDE the userdata root — keeping it would
|
||||||
|
// double-tar and basename-collide with the root tar.
|
||||||
|
compose := writeCompose(t, `services:
|
||||||
|
app:
|
||||||
|
image: x
|
||||||
|
volumes:
|
||||||
|
- ${HDD_PATH}/userdata/foo:/foo
|
||||||
|
- ${USERDATA_PATH}/bar:/bar
|
||||||
|
`)
|
||||||
|
hdd := filepath.Join(string(filepath.Separator)+"mnt", "drive", "app")
|
||||||
|
got := ExportDataMounts(compose, hdd)
|
||||||
|
want := []string{filepath.Join(hdd, "userdata")}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("ExportDataMounts = %v, want only the userdata root %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExportDataMounts_EmptyHDDPath(t *testing.T) {
|
||||||
|
compose := writeCompose(t, sonarrShapeCompose)
|
||||||
|
if got := ExportDataMounts(compose, ""); got != nil {
|
||||||
|
t.Fatalf("ExportDataMounts with empty HDD_PATH = %v, want nil", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ package web
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -298,7 +299,19 @@ func (s *Server) handleNetStorageRemove(w http.ResponseWriter, r *http.Request)
|
|||||||
writeDiskJSON(w, http.StatusBadRequest, false, "érvénytelen név", nil)
|
writeDiskJSON(w, http.StatusBadRequest, false, "érvénytelen név", nil)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
agent, err := s.agentClient()
|
where := settings.NetworkMountRoot + "/" + name
|
||||||
|
// C6B-F2 guard (v0.130.0): refuse the removal while a DEPLOYED app's HDD_PATH lives on the
|
||||||
|
// share. Removing the share under a bound app strands the app's storage AND orphans the host
|
||||||
|
// automount (the agent's stop steps are tolerated best-effort, so a busy mount gets its unit
|
||||||
|
// files deleted anyway → an unreapable autofs mount until host reboot — the C6B-F2 orphan).
|
||||||
|
if apps := s.deployedAppsOnPath(where); len(apps) > 0 {
|
||||||
|
s.logger.Printf("[WARN] [web] netstorage remove %q refused: deployed app(s) on the share: %s", name, strings.Join(apps, ", "))
|
||||||
|
writeDiskJSON(w, http.StatusConflict, false, fmt.Sprintf(
|
||||||
|
"A tároló nem távolítható el, amíg alkalmazás használja: %s. Előbb távolítsa el vagy költöztesse át az alkalmazást.",
|
||||||
|
strings.Join(apps, ", ")), nil)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
agent, err := s.netAgentForAdd()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
writeDiskJSON(w, http.StatusServiceUnavailable, false, err.Error(), nil)
|
writeDiskJSON(w, http.StatusServiceUnavailable, false, err.Error(), nil)
|
||||||
return
|
return
|
||||||
@@ -308,7 +321,6 @@ func (s *Server) handleNetStorageRemove(w http.ResponseWriter, r *http.Request)
|
|||||||
writeDiskJSON(w, http.StatusBadGateway, false, err.Error(), nil)
|
writeDiskJSON(w, http.StatusBadGateway, false, err.Error(), nil)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
where := settings.NetworkMountRoot + "/" + name
|
|
||||||
if err := s.settings.RemoveStoragePath(where); err != nil {
|
if err := s.settings.RemoveStoragePath(where); err != nil {
|
||||||
s.logger.Printf("[WARN] [web] netstorage deregister %q: %v", where, err)
|
s.logger.Printf("[WARN] [web] netstorage deregister %q: %v", where, err)
|
||||||
}
|
}
|
||||||
@@ -316,6 +328,36 @@ func (s *Server) handleNetStorageRemove(w http.ResponseWriter, r *http.Request)
|
|||||||
writeDiskJSON(w, http.StatusOK, true, "", map[string]any{"removed": true, "name": name})
|
writeDiskJSON(w, http.StatusOK, true, "", map[string]any{"removed": true, "name": name})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// deployedAppsOnPath returns the display names of DEPLOYED stacks whose HDD_PATH is base itself
|
||||||
|
// or a subpath of it (apps on a share store <share-root>/<app>). Nil-safe on stackMgr.
|
||||||
|
func (s *Server) deployedAppsOnPath(base string) []string {
|
||||||
|
if s.stackMgr == nil || base == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var apps []string
|
||||||
|
for _, st := range s.stackMgr.GetStacks() {
|
||||||
|
if !st.Deployed {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
cfg := s.stackMgr.LoadAppConfigByName(st.Name)
|
||||||
|
if cfg == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
hdd := cfg.Env["HDD_PATH"]
|
||||||
|
if hdd == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if hdd == base || strings.HasPrefix(hdd, base+"/") {
|
||||||
|
name := st.Meta.DisplayName
|
||||||
|
if name == "" {
|
||||||
|
name = st.Name
|
||||||
|
}
|
||||||
|
apps = append(apps, name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return apps
|
||||||
|
}
|
||||||
|
|
||||||
// listNetStorage resolves the agent's live share list (test seam first, then the shared client).
|
// listNetStorage resolves the agent's live share list (test seam first, then the shared client).
|
||||||
func (s *Server) listNetStorage(ctx context.Context) ([]agentapi.NetworkMountStatus, error) {
|
func (s *Server) listNetStorage(ctx context.Context) ([]agentapi.NetworkMountStatus, error) {
|
||||||
if s.netListFn != nil {
|
if s.netListFn != nil {
|
||||||
|
|||||||
@@ -113,7 +113,9 @@ func (s *netAddState) snapshot() *netAddJob {
|
|||||||
return &cp
|
return &cp
|
||||||
}
|
}
|
||||||
|
|
||||||
// netAgentForAdd resolves the orchestrator's agent surface (test seam first, then the shared client).
|
// netAgentForAdd resolves the net-storage agent surface (test seam first, then the shared
|
||||||
|
// client). Despite the name it serves the whole share lifecycle — the remove handler resolves
|
||||||
|
// through it too (v0.130.0).
|
||||||
func (s *Server) netAgentForAdd() (netAgent, error) {
|
func (s *Server) netAgentForAdd() (netAgent, error) {
|
||||||
if s.netAgentFn != nil {
|
if s.netAgentFn != nil {
|
||||||
return s.netAgentFn()
|
return s.netAgentFn()
|
||||||
|
|||||||
@@ -0,0 +1,128 @@
|
|||||||
|
package web
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||||
|
)
|
||||||
|
|
||||||
|
// C6B-F2 guard (v0.130.0, scenario F): removing a network share while a DEPLOYED app's HDD_PATH
|
||||||
|
// lives on it is REFUSED up front — the agent's unit teardown never starts, so the busy-mount →
|
||||||
|
// tolerated-stop → unit-files-deleted-anyway → orphaned-autofs chain (the observed C6B-F2) cannot
|
||||||
|
// be triggered through the product flow. RED-PROOF: drop the deployedAppsOnPath guard from
|
||||||
|
// handleNetStorageRemove → the 409/zero-agent-call/still-registered assertions fail (the pre-fix
|
||||||
|
// behavior: removed:true while sonarr ran on the share).
|
||||||
|
|
||||||
|
// testRemoveGuardServer builds a Server with a REAL stacks.Manager over a temp stacks dir holding
|
||||||
|
// one app (deployed flag per arg) whose HDD_PATH sits ON the share, plus the share registered.
|
||||||
|
func testRemoveGuardServer(t *testing.T, deployed bool) (*Server, *fakeNetAgent, string) {
|
||||||
|
t.Helper()
|
||||||
|
lg := log.New(io.Discard, "", 0)
|
||||||
|
dir := t.TempDir()
|
||||||
|
cfg := &config.Config{}
|
||||||
|
cfg.Paths.StacksDir = filepath.Join(dir, "stacks")
|
||||||
|
cfg.Paths.DataDir = filepath.Join(dir, "data")
|
||||||
|
cfg.Stacks.ComposeCommand = "docker compose"
|
||||||
|
|
||||||
|
share := settings.NetworkMountRoot + "/campaign6"
|
||||||
|
stackDir := filepath.Join(cfg.Paths.StacksDir, "sonarr")
|
||||||
|
if err := os.MkdirAll(stackDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
os.WriteFile(filepath.Join(stackDir, ".felhom.yml"), []byte("display_name: Sonarr\n"), 0o644)
|
||||||
|
os.WriteFile(filepath.Join(stackDir, "docker-compose.yml"), []byte("services: {}\n"), 0o644)
|
||||||
|
appYAML := "deployed: false\n"
|
||||||
|
if deployed {
|
||||||
|
// the C6B live shape: HDD_PATH is a SUBPATH of the share root (<share>/<app>)
|
||||||
|
appYAML = "deployed: true\nenv:\n HDD_PATH: " + share + "/sonarr\n"
|
||||||
|
}
|
||||||
|
os.WriteFile(filepath.Join(stackDir, "app.yaml"), []byte(appYAML), 0o644)
|
||||||
|
|
||||||
|
sett, err := settings.Load(filepath.Join(dir, "settings.json"), lg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := sett.AddStoragePath(settings.StoragePath{
|
||||||
|
Path: share, Label: "Kampány 6 teszt", Schedulable: true, Kind: settings.StorageKindNetwork,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
mgr, err := stacks.NewManager(cfg, lg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_ = mgr.ScanStacks() // container-status refresh may fail on docker-less hosts — discovery is enough
|
||||||
|
if _, ok := mgr.GetStack("sonarr"); !ok {
|
||||||
|
t.Fatal("sonarr not discovered by ScanStacks")
|
||||||
|
}
|
||||||
|
|
||||||
|
agent := &fakeNetAgent{}
|
||||||
|
s := &Server{cfg: cfg, settings: sett, stackMgr: mgr, logger: lg}
|
||||||
|
s.netAgentFn = func() (netAgent, error) { return agent, nil }
|
||||||
|
return s, agent, share
|
||||||
|
}
|
||||||
|
|
||||||
|
func postNetRemove(t *testing.T, s *Server, name string) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
r := httptest.NewRequest(http.MethodPost, "/api/storage/netstorage/remove",
|
||||||
|
strings.NewReader(`{"name":"`+name+`"}`))
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
s.handleNetStorageRemove(w, r)
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNetStorageRemove_RefusedWhileAppDeployedOnShare(t *testing.T) {
|
||||||
|
s, agent, share := testRemoveGuardServer(t, true)
|
||||||
|
|
||||||
|
w := postNetRemove(t, s, "campaign6")
|
||||||
|
if w.Code != http.StatusConflict {
|
||||||
|
t.Fatalf("remove with a deployed app on the share: got %d want 409 (%s)", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(w.Body.String(), "Sonarr") {
|
||||||
|
t.Errorf("the refusal must NAME the blocking app, got: %s", w.Body.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(w.Body.String(), "nem távolítható el") {
|
||||||
|
t.Errorf("expected the Hungarian refusal, got: %s", w.Body.String())
|
||||||
|
}
|
||||||
|
// The agent teardown must NEVER have started — that is what orphans the automount.
|
||||||
|
if got := agent.removed(); len(got) != 0 {
|
||||||
|
t.Fatalf("agent RemoveNetStorage ran despite the refusal: %v", got)
|
||||||
|
}
|
||||||
|
// The share stays registered (nothing half-removed).
|
||||||
|
found := false
|
||||||
|
for _, sp := range s.settings.GetStoragePaths() {
|
||||||
|
if sp.Path == share {
|
||||||
|
found = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatal("the share was deregistered despite the refusal")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNetStorageRemove_ProceedsWithoutDeployedApps(t *testing.T) {
|
||||||
|
s, agent, share := testRemoveGuardServer(t, false)
|
||||||
|
|
||||||
|
w := postNetRemove(t, s, "campaign6")
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("remove with no deployed apps: got %d want 200 (%s)", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if got := agent.removed(); len(got) != 1 || got[0] != "campaign6" {
|
||||||
|
t.Fatalf("agent RemoveNetStorage calls = %v, want [campaign6]", got)
|
||||||
|
}
|
||||||
|
for _, sp := range s.settings.GetStoragePaths() {
|
||||||
|
if sp.Path == share {
|
||||||
|
t.Fatal("the share must be deregistered after a successful remove")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user