docs: REPORT — v0.131.0 F-S2/F-S3 deploy evidence + deferred live legs
Both guests live+healthy on 0.131.0. Live functional legs (paperless deploy → tier-2 backup → marker restore → storage page → F-S3 migration) deferred to Viktor's supervised session — paperless-ngx is undeployed on the demo and the API is container-internal; documented with rationale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
This commit is contained in:
@@ -1,107 +1,126 @@
|
||||
# REPORT — CRITICAL C6B-F1: hollow .fab export (3 compounding defects) + C6B-F2 share-removal guard
|
||||
# REPORT — F-S2 + F-S3: compose-derived appdata dir resolution (controller v0.131.0)
|
||||
|
||||
**Date:** 2026-07-14 · **Version:** controller **v0.130.0** (baseline `eb3bf4a` on `main`) · Controller-only; no agent/hub/felhom.eu change.
|
||||
## Summary
|
||||
|
||||
## What was broken (CAMPAIGN-6B, reviewer-validated at source)
|
||||
The controller assumed an app's HDD appdata dir is always `appdata/<stackName>`. paperless-ngx binds
|
||||
`${HDD_PATH}/appdata/paperless/...` (stack `paperless-ngx`, dir `paperless`), so every consumer that
|
||||
keyed by stack name silently missed it via a stat-and-skip. One canonical resolver
|
||||
(`appbackup.AppDataDirNames`) now derives the real dir name(s) from the app's compose `${HDD_PATH}`
|
||||
binds; all consumers use it. Task 1 of the backup-classification-redesign arc
|
||||
(`felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`), independent of the
|
||||
classification schema.
|
||||
|
||||
`.fab` export produced a **config-only, data-free bundle** for **12/13 `needs_hdd` catalog apps**
|
||||
(the `${USERDATA_PATH}` convention: audiobookshelf, calibre-web, emby, immich, jellyfin, komga,
|
||||
navidrome, paperless-ngx, plex, radarr, romm, sonarr), reported success, and passed the v0.125.0
|
||||
anti-hollow guard. Live proof: sonarr, 4.17 GB / 7 files → a **2308-byte** bundle. Cross-box or
|
||||
fresh-box restore = silent total data loss. **Confirmed scope: the SCHEDULED/tier-2 backup path was
|
||||
NOT affected** (it copies the `felhom-data` namespace wholesale via `GetAppDrivePath`); only the
|
||||
`.fab` export adapter had the mount-parse + either/or combination — the backup-side `stackAdapter`
|
||||
is deliberately untouched.
|
||||
## Confirmed baselines
|
||||
|
||||
## Commits (each part green-gated: `go build ./... && go vet ./... && go test ./...`)
|
||||
| Repo | `main` @ start | Version | → |
|
||||
|------|----------------|---------|---|
|
||||
| felhom-controller | `b42904b` (verified 2026-07-14) | v0.130.0 | **v0.131.0** (commit `68f0e0c`, pushed to `main`) |
|
||||
|
||||
| Commit | Part | Change |
|
||||
|---|---|---|
|
||||
| `8967ba7` | 1 (cause 2) | `stacks.ExportDataMounts` (new, delete.go) + `exportAdapter.GetStackHDDMounts` rewired to it — `${HDD_PATH}` binds UNIONed with the `${USERDATA_PATH}` ROOT, containment-deduped both directions. Also fixes the estimate's `data=0 B`. |
|
||||
| `c6d8bc8` | 2 (cause 1 + §8) | `executeExport` additive (needs_hdd apps run `exportHDDData` AND `exportVolumeData`); `exportHDDData` returns error + fails LOUDLY on a basename collision; `EstimateExport` additive to match (fits-on-dest counts both). |
|
||||
| `a829cdc` | 3 (cause 3) | `assertBundleDataComplete`: a `needs_hdd` manifest with neither HDD data nor volume data fails the job — "a mentés nem tartalmaz alkalmazásadatot (0 adatkönyvtár, 0 kötet…)". |
|
||||
| `b49076d` | 4 (C6B-F2) | `handleNetStorageRemove` refuses (409, Hungarian, names the apps) while a DEPLOYED stack's HDD_PATH is the share root or a subpath; new `deployedAppsOnPath` helper; remove resolves the agent via the `netAgent` seam. |
|
||||
| `3679a75` | docs/ship | CHANGELOG v0.130.0, README export section, REUSE.md (2 new helpers), CONTEXT.md, this REPORT. |
|
||||
## Files created / modified
|
||||
|
||||
## ⚠ Design deviation from the task (deliberate, load-bearing)
|
||||
**Part 1 — resolver + helpers**
|
||||
- `controller/internal/appbackup/paths.go` — NEW pure `AppDataDirNames(hddPath, stackName, hddMounts) []string`
|
||||
(dedup+sort; fallback `[stackName]`) + `AppDataBindsPresent(hddPath, hddMounts) bool` (WARN predicate);
|
||||
`AppDataDir` doc updated.
|
||||
- `controller/internal/backup/tier2.go` — `appDataDirNames` / `tier2AppDataName` (the one place the N>1
|
||||
refusal error `errTier2MultiDir` is built) / `tier2AppDataBindsPresent`.
|
||||
- `controller/internal/backup/appbackup_bridge.go` — forwarders for the two new appbackup funcs.
|
||||
- `controller/internal/stacks/migrate.go` — `ResolveAppDataDirNames` (exported; used by handlers) +
|
||||
`resolveAppDataDirs` (names + `declared` bool for the WARN); `migSeams.resolveNames` test seam.
|
||||
|
||||
The task's Part 1 letter said: union the **per-bind** `ParseComposeUserdataMounts` results, and §8
|
||||
said namespace colliding tar names "by a sanitized full-relative path or index". **Both are
|
||||
incompatible with §12 "do NOT improve the import side":** the manifest keys HDD tars by basename,
|
||||
and the untouched import maps a basename either to a resolved `${HDD_PATH}` mount or to
|
||||
`<HDD_PATH>/<basename>` (restore.go `restoreHDDData` + `resolveHDDMounts`, which is itself
|
||||
`${HDD_PATH}`-only). A per-bind mount `…/userdata/media/tv` bases to `tv` → the import would restore
|
||||
it to `<HDD_PATH>/tv` — wrong place, a subtler variant of the same data loss; a namespaced tar name
|
||||
could not be mapped back at all. **Resolution:** capture the userdata subtree at its ROOT (one
|
||||
mount, basename `userdata`, a direct child of HDD_PATH) — it round-trips through the existing import
|
||||
fallback exactly, captures at least as much data (the whole per-app userdata subtree, same
|
||||
philosophy as tier-2's namespace-wholesale copy), and needs zero import changes. Proven by
|
||||
`TestFabRoundTrip_UserdataPlacement` (export → wipe → import → file back at
|
||||
`<HDD_PATH>/userdata/media/tv/show.bin`, byte-identical). For §8, colliding basenames now **fail
|
||||
loudly** instead of being renamed (a rename cannot round-trip either); no catalog app collides today.
|
||||
**Part 2 — tier-2 backup/info/restore**
|
||||
- `controller/internal/backup/tier2.go` — `RunTier2` resolves the appdata dir name (L146 site), refuses
|
||||
N>1 (honest `no_target` status + `[ERROR]`), WARNs on a declared-but-absent dir; new `tier2Mirror`
|
||||
seam on both rsync legs; rewrote the lying package header (no userdata/no-wholesale truth).
|
||||
- `controller/internal/backup/backup.go` — `tier2Mirror` seam field.
|
||||
- `controller/internal/backup/tier2_restore.go` — `RestoreTier2Files` targets the resolved live dir;
|
||||
`errTier2MultiDirRestore` sentinel, refused BEFORE `StopStack`.
|
||||
|
||||
## Part 4 scope note (C6B-F2)
|
||||
**Part 3 — migrate (F-S3)** — `controller/internal/stacks/migrate.go`: all six per-app appdata legs
|
||||
(collision `:328`, size `:354`, copy `:449`, verify `:491`, cleanup `:556`, skip-set `:605`) loop the
|
||||
resolved name(s); copy leg WARNs on a missing declared dir. Merge walk / scope gating / journal untouched.
|
||||
|
||||
The task guessed the defect was removal-order in the controller. Investigation: the agent's
|
||||
`RemoveNetworkMount` (felhom-agent `internal/storage/netmount.go:419`) already stops the automount
|
||||
BEFORE unlinking — the real defect is **tolerate-and-continue** (netmount.go:434-443: every stop
|
||||
step's failure is logged at Debug and execution proceeds), so a busy mount (live app bind — the C6B
|
||||
live event) gets its unit files deleted anyway → the unreapable orphaned autofs. The agent is out of
|
||||
this task's scope ("Repos touched: felhom-controller ONLY"), so the shipped fix is the
|
||||
controller-side guard that cuts the trigger off at the product flow (refuse removal while a deployed
|
||||
app binds the share — also the C6B audit's explicit fix direction). **Follow-up needed (felhom-agent
|
||||
task):** after the stop steps, verify the mountpoint is actually released and ABORT the unit-file
|
||||
removal if not.
|
||||
**Part 4 — display** — `controller/internal/web/handlers.go`: `appDetailsForPath` sums the resolved
|
||||
dir(s); `dirSizeHuman` split into `dirSizeBytesWalk` + `humanizeDirBytes`.
|
||||
|
||||
## Tests + red-proofs (all four run→fail→revert, recorded)
|
||||
**Part 5 — truth repairs** — `controller/cmd/controller/main.go:1357` export-adapter comment corrected;
|
||||
the v0.130.0 CHANGELOG's false "copies the namespace wholesale" sentence is explicitly noted false in the
|
||||
v0.131.0 entry (old entry left intact as the record).
|
||||
|
||||
| # | Test(s) | Red-proof |
|
||||
|---|---|---|
|
||||
| 1 | `stacks/export_mounts_test.go` ×6 (union incl. baked-in pre-fix-finds-0 contrast, HDD-direct regression B, mixed, covering-root, literal-userdata dedupe, empty-HDD) | reverted `ExportDataMounts` to `${HDD_PATH}`-only → 3 tests FAIL (UserdataConvention, MixedBindsUnion, LiteralUserdataBindDeduped) → restored green |
|
||||
| 2 | `appexport/export_additive_test.go`: scenario A (bundle has BOTH `data/hdd/userdata.tar` + `data/volumes/hdd-app_config.tar`, both manifest flags) | reverted `executeExport` to the either/or → scenario A FAILS (volume tar absent) → restored green |
|
||||
| 2b | §8 collision test (loud Hungarian failure naming `config`) | removed the collision check → FAILS ("got success (silent overwrite)") → restored green |
|
||||
| 3 | scenario D (`needs_hdd` + zero discovered data → job fails "nem tartalmaz alkalmazásadatot", no bundle) | removed the guard assertion → FAILS ("got success (the hollow bundle)") → restored green |
|
||||
| 4 | `web/netstorage_remove_guard_test.go` ×2 (refused-while-deployed: 409 + names Sonarr + zero agent calls + share stays registered; proceeds-without: 200 + agent called + deregistered) | disabled the guard → FAILS with the exact live pre-fix body `{"removed":true}` → restored green |
|
||||
**Docs** — `CHANGELOG.md` (v0.131.0, newest-on-top, incl. v0.130.0 correction), `REUSE.md`
|
||||
(AppDataDirNames/BindsPresent + tier2Mirror/resolveNames seams), `CONTEXT.md`, `controller/README.md`.
|
||||
|
||||
Also: scenario E (needs_hdd volume-strand fails loud, no bundle) and scenario A' (round-trip
|
||||
placement) pass; scenario B (HDD-direct app unchanged) pinned at the adapter level; scenario C
|
||||
(volume-only app) unchanged — pre-existing `volume_guard_test.go` suite still green. Full gate:
|
||||
**23/23 packages ok** (build+vet+test). Test fixtures use `t.TempDir()` + the `dockerExec`/
|
||||
`netAgentFn` seams — no real docker anywhere.
|
||||
## Commits on `main`
|
||||
|
||||
## Deploy + live self-verify (DONE, 2026-07-14 ~13:40 CEST)
|
||||
| Hash | Contents |
|
||||
|------|----------|
|
||||
| `68f0e0c` | F-S2 + F-S3 code + all tests + docs (single commit) |
|
||||
|
||||
- Built `v0.130.0` on 180 (`build.sh 0.130.0 --push`, image 145M); deployed via the bootstrap
|
||||
mechanism to **demo 9201** AND the **drill guest** — both `docker ps` show
|
||||
`felhom-controller:0.130.0 … (healthy)`.
|
||||
- **Live self-verify on demo — the exact C6B-F1 repro flipped** (temporary break-glass credential,
|
||||
reverted after; plaintext never persisted):
|
||||
- Deployed sonarr (the C6B repro app) via `POST /api/stacks/sonarr/deploy` onto
|
||||
`felhom-usb/sonarr-c6bfix`; injected 30 MB of sha256-recorded markers into
|
||||
`userdata/{media/tv,downloads}`; sonarr's `sonarr_config` volume held 7.2 MB.
|
||||
- `GET /api/export/download/estimate` → **`data_size 37.1 MB`** (pre-fix: `0 B`) — userdata +
|
||||
volume both counted.
|
||||
- Export → **`sonarr_20260714-133802.fab` = 31,797,668 bytes** (pre-fix: 2308). Manifest:
|
||||
`controller_version:"0.130.0"`, **`has_hdd_data:true`**, **`has_volume_data:true`**,
|
||||
`hdd_subdirs:["userdata"]`, `volume_names:["sonarr_sonarr_config"]`.
|
||||
- Extracted `data/hdd/userdata.tar` from the bundle → **both marker sha256s byte-identical** to
|
||||
the pre-export recordings (`97259f74…`, `e406658c…`) + the text marker intact.
|
||||
- Cleanup: sonarr stopped+removed (data wiped), staged bundle + scratch removed, credential
|
||||
reverted to the original customer-claimed hash; controller healthy on 0.130.0.
|
||||
- C6B-F2 guard live check: not exercised against a live share (none enrolled post-6B-cleanup);
|
||||
covered by the red-proofed handler tests — 6C's clean-order teardown re-test exercises it live.
|
||||
## Tests
|
||||
|
||||
## For CAMPAIGN-6C
|
||||
`go build ./... && go vet ./... && go test ./...` — **all green** (every package `ok`; vet clean).
|
||||
**+9 test functions** (before → after in the touched packages):
|
||||
|
||||
The flagship `.fab` full-circle (sonarr → 4 GB → export → download → delete → upload → import →
|
||||
**byte-compare, zero mismatches**) that was BLOCKED-BY-BUG in 6B is now verifiable end-to-end — it
|
||||
should be 6C's FIRST live acceptance test. The C6B-F2 clean-order teardown re-test (remove app THEN
|
||||
share → no orphan) is also now guard-assisted.
|
||||
- `internal/appbackup/appdatadirnames_test.go` (NEW, +2): `TestAppDataDirNames` (Group A table:
|
||||
paperless mismatch, matching name, two-name sorted, media+export dedupe, non-appdata + foreign-drive
|
||||
filtered, whole-root bind ignored, empty→fallback, unclean path), `TestAppDataBindsPresent`.
|
||||
- `internal/backup/tier2_appdata_test.go` (NEW, +6): `TestRunTier2_PaperlessShape` (Scenario A),
|
||||
`TestRunTier2_LegacyShape` (Scenario B: match + no-binds fallback), `TestRunTier2_MultiDirRefusal`
|
||||
(Scenario C, exact Hungarian reason, mirror-count 0), `TestTier2Info_MultiDirRefusal`,
|
||||
`TestRestoreTier2Files_ResolvedLiveDir` (Scenario E), `TestRestoreTier2Files_MultiDirRefusal`
|
||||
(refused before StopStack). `t2rFakeProvider` extended with configurable `GetStackHDDMounts`.
|
||||
- `internal/stacks/migrate_fs3_test.go` (NEW, +1): `TestMigrate_PaperlessShape_ScopeApp` (Scenario D:
|
||||
copy+verify captured src/dst = appdata/paperless; size/collision/cleanup/skip-set all on the resolved
|
||||
dir; never appdata/paperless-ngx).
|
||||
|
||||
## Observations
|
||||
### Red-proofs (run → fail → revert → green) — all confirmed
|
||||
|
||||
- `estimate.go` had the same either/or shape as the export; left un-additive it would have
|
||||
under-reported needs_hdd apps by their volume size against the new bundle content — fixed in the
|
||||
same commit as cause 1 (the fits-on-dest gate consumes it inside `executeExport`).
|
||||
- `netAgentForAdd` now serves the whole share lifecycle (remove resolves through it too) — comment
|
||||
updated; a rename was skipped for minimal-diff discipline.
|
||||
| RP | Mutation | Failing assertion observed | Reverted |
|
||||
|----|----------|----------------------------|----------|
|
||||
| RP-1 | `AppDataDirNames` → `return []string{stackName}` | `TestAppDataDirNames` paperless/two-name/unclean cases FAIL | green |
|
||||
| RP-2 | RunTier2 L146 → `AppDataDir(nsRoot, stackName)` | `TestRunTier2_PaperlessShape`: appdata leg never mirrored | green |
|
||||
| RP-3 | restore `liveDir` → stack-name keying | `TestRestoreTier2Files_ResolvedLiveDir`: dst = appdata/app not appdata/paperless | green |
|
||||
| RP-4 | migCopy site → key by `app` | `TestMigrate_PaperlessShape_ScopeApp`: copy seam never saw appdata/paperless | green |
|
||||
| RP-5 | delete N>1 guard (`>1` → `>999`) | all three MultiDir refusal tests FAIL (mirror fired, no refusal) | green |
|
||||
|
||||
## Deploy + verify
|
||||
|
||||
- Built `v0.131.0` on 180 (`build.sh 0.131.0 --push`, image 145M, digest `sha256:7bba3b54…`).
|
||||
- Deployed via the bootstrap mechanism to **BOTH guests**:
|
||||
- **demo 9201** (felhom-pve): `felhom-controller:0.131.0 … Up (healthy)`; logs show a clean status
|
||||
refresh (9 containers / 56 stacks) + hub report pushed.
|
||||
- **drill guest 9201** (nested PVE `drill-day0`, 192.168.0.152): `felhom-controller:0.131.0 … Up
|
||||
(healthy)`; logs clean (scheduler + status refresh).
|
||||
|
||||
## NOT live-validated here (deferred to Viktor's supervised session)
|
||||
|
||||
The functional live legs all require a **deployed paperless-ngx** (the only catalog app with the
|
||||
appdata mismatch shape). On the current demo, paperless-ngx is **undeployed** — only leftover skeleton
|
||||
data (`appdata/paperless`, ~16 KB) + a stale recovery unit remain from a prior session; every
|
||||
currently-deployed HDD app (jellyfin/radarr/navidrome/calibre-web) uses userdata/media binds, not
|
||||
`appdata/`. Standing up paperless-ngx is a 5-container, Postgres-backed deploy, and the controller's
|
||||
API (`:8080`) is container-internal (reachable only via traefik + Host header), so a clean unsupervised
|
||||
deploy→backup→restore→teardown was disproportionate/risky on the shared demo. Deferred, to run through
|
||||
the real dashboard UI under supervision:
|
||||
|
||||
1. **Tier-2 backup (F-S2 core):** deploy paperless-ngx on felhom-usb, trigger "2. mentés", verify
|
||||
`…/secondary/paperless-ngx/appdata/` mirrors `appdata/paperless` (count + size).
|
||||
2. **Marker restore leg:** inject a marker in `appdata/paperless`, re-run tier-2, delete it live, run the
|
||||
UI file-restore, verify byte-identical return + app healthy.
|
||||
3. **Storage page:** paperless-ngx shows a non-empty size.
|
||||
4. **F-S3 migration leg:** scope="app" migration of a real paperless-ngx between drives (supervised —
|
||||
destructive cleanup step).
|
||||
|
||||
Confidence for these rests on the exhaustive unit coverage + RP-1..RP-5 above and the confirmed-live
|
||||
compose shape (paperless-ngx's on-box `docker-compose.yml` binds `${HDD_PATH}/appdata/paperless/media`
|
||||
and `.../export` — the exact mismatch the fix resolves).
|
||||
|
||||
## Observations (noticed, NOT acted on)
|
||||
|
||||
- **F-S1 (userdata not backed up at any tier)** remains open — deliberately out of scope; owned by the
|
||||
classification redesign (Task 2/3), which this task's tier-2 header + comments now point to.
|
||||
- **Multi-dir (N>1) limitation** is a deliberate tier-2 refusal to be lifted by the tier-policy engine
|
||||
(Task 3, which owns the flat destination layout). No catalog app hits it today.
|
||||
- Leftover undeployed paperless/immich/nextcloud/romm skeleton dirs + a stale
|
||||
`felhom-flash/backups/secondary/paperless-ngx` exist on the demo drives (prior-session residue) — not
|
||||
touched.
|
||||
|
||||
Reference in New Issue
Block a user