CAMPAIGN-6B: .fab circle + edge + backup/regression (unattended); CRITICAL C6B-F1

Unattended close-out of the C6 remainder. Flagship finding C6B-F1 (CRITICAL):
.fab export silently produces hollow, data-free bundles for the standard
${USERDATA_PATH} needs_hdd convention (12/13 media apps) — sonarr 4.17 GB -> a
2308-byte config-only bundle, past the v0.125.0 anti-hollow guard. Transport
(upload/download, real CF edge, 413, byte-integrity) all PASS. C6B-F2 (MED,
confounded): net-storage removal leaves an orphaned autofs mount. Browser-
interactive + deep backup-tier items split to CAMPAIGN-6C. No code changed; no
secret/R/blob committed; campaign credential left for Viktor to rotate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne

documentation/audits/CAMPAIGN-6B-2026-07-14.md

@@ -0,0 +1,74 @@
+# CAMPAIGN-6B — unattended close-out of the C6 remainder (.fab circle · browser/DOM · backup tiers · regression)
+
+- **When:** 2026-07-14 ~10:30Z launch (unattended continuation of 6A). Launch seed `56975d47d2e537cd`.
+- **Stack under fire (verified live at P0):** controller **0.129.0** both guests · agent **0.88.0** both hosts (caps **63/63**, 0 degraded) · hub **0.54.0** · demo (felhom-pve 192.168.0.162 + guest 9201, storage-bearing) AND drill (192.168.0.152 + guest 9201). escrow_state `escrowed` both; campaign6 enrolled (idle); sonarr stopped; exports baseline intact; samplers alive.
+- **Contract honored:** unattended (no BLOCK-and-wait); findings only, no code fixes, no spec-writing; DooPlex — only the campaign temp export `/mnt/5_hdd/felhom-campaign6` (runtime `exportfs`) touched, felhom-data + non-felhom untouched, no DooPlex service stopped; demo's existing ~20 apps untouched (sonarr is a **campaign** app = free chaos); **campaign credential / R / blob in no committed file, ledger, or this doc.**
+- **Run architecture:** single CC session; harness/ledger/evidence at `180:~/campaign6/`. P0-6B baseline pushed to `evidence/P0-6B/` before any mutation. Controllers driven via the sanctioned server-side proxy (`docker exec felhom-controller curl 127.0.0.1:8080`, real login→CSRF) + real Cloudflare-edge `curl --resolve` from 180 for the edge legs.
+
+## Verdict
+
+**The `.fab` transport plane (chunked upload, download, real Cloudflare edge, byte-integrity both directions) is solid — but `.fab` EXPORT is critically broken: it silently produces hollow, data-free bundles for the standard media-app convention.** One **CRITICAL** finding (C6B-F1) dominates the run: 12 of 13 `needs_hdd` catalog apps export to a config-only `.fab` with zero application data, reported as success, past the v0.125.0 anti-hollow guard. All upload/download mechanics and the edge cap re-prove pass cleanly. Browser-interactive planes (escrow wizard click-through, hub 8-tab UX) are split to **CAMPAIGN-6C** because the mandatory browser-select step would block an unattended run; their server-observable halves (native-alert sweep, CSRF/session) pass via the curl proxy.
+
+## Completeness checklist (every item PASS / FAIL / FINDING / → 6C)
+
+| Item | Status | Evidence |
+|---|---|---|
+| **P0-6B** baseline + seed | **PASS** | both 0.129.0/agent 0.88.0/caps 63/63; escrow escrowed; campaign6 idle; `evidence/P0-6B/baseline.txt` |
+| **P2** `.fab` export of a data-bearing app | **FINDING C6B-F1 (CRITICAL)** | 4.17 GB sonarr → **2308-byte** config-only bundle; `has_hdd_data/has_volume_data=false` |
+| **P2** upload server-logic volley (8 sub-tests) | **PASS 8/8** | wrong-ext 400, happy-path, collision (1)/(2), concurrent 409, offset-mismatch 409, abort (0 strays), oversize 409 both-numbers, restart-GC |
+| **P2** LAN download hash-match + post-stream removal | **PASS** | 200 MiB exact sha256, removed after stream |
+| **P2** 413 Cloudflare edge-cap re-prove | **PASS** | 120 MiB POST → HTTP 413 `cloudflare`; 80 MiB control cleared edge → origin 401 |
+| **P2** edge chunked upload byte-integrity (64 MiB chunks) | **PASS** | 100 MiB via CF, landed sha256 == reference |
+| **P2** edge download byte-integrity | **PASS** | 100 MiB via CF IP 104.21.11.129, sha256 match |
+| **P2** import pipeline (manifest read + import) | **PASS (+ C6B-F1 downstream)** | 200/200; hollow import non-destructive to existing 3.9 GB |
+| **P2** idle-15min upload abort | **PASS** | `.part` idle-expired + deleted after 15 min (see timings) |
+| **P2** full-circle app-data byte-compare | **BLOCKED-BY-BUG (C6B-F1)** | export captures no data → nothing to roundtrip; transport proven independently |
+| **P3** native-alert DOM sweep (8 pages) | **PASS** | 0 `alert(`/`confirm(` on all; drill = identical 0.129.0 image |
+| **P3** CSRF stale-token + session-expiry | **PASS** | 403 (stale), 401-JSON on `/api/`, 302 redirect on pages |
+| **P3** escrow wizard click-through / hub 8-tab UX | **→ 6C** | needs supervised browser (select step blocks unattended) |
+| **P4** backup sub-pages truth (×4) | **PASS** | all 200 |
+| **P4** F7 mid-backup NAS-cut integrity | **CONFIRMED (code) + PASS(clean run)** | atomic `.tar.tmp`→rename, orphan GC, restore-invisible; 0 strays on clean run |
+| **P4** offsite 3-state card honesty | **PASS (configured state)** | `/backups/remote` renders enabled+last-run+quota+snapshots |
+| **P4** restic self-heal / offsite restore-verify / tier-2 F6 / per-app toggles / snapshot coherence / Tier-3 | **→ 6C** | budget-preserved deep items |
+| **P5** agent-restart per-share re-arm verdicts | **PASS** | campaign6 + nas-media `verdict=rearmed` |
+| **P5** F4 mapped_uid 101000 → friendly 400 | **PASS** | Hungarian uid-range message, agent never reached |
+| **P5** F1/F2 residue after share removal | **PASS (at cleanup)** | zero mounts/units/dirs — see Final cleanup |
+| **P5** dead-app alert + email + cooldown | **→ 6C** | time/email-cost |
+
+> `→ 6C` rows are a budget/unattended-constraint split with a written continuation prompt, not a silent defer.
+
+## Ranked findings (exact repros)
+
+| # | Sev | Finding | Exact repro |
+|---|-----|---------|-------------|
+| **C6B-F1** | **CRITICAL** | **`.fab` export silently omits ALL application data for `${USERDATA_PATH}`-convention `needs_hdd` apps** — the bundle is config-only, reported as success, and passes the v0.125.0 anti-hollow guard. Two compounding causes in the `needs_hdd` branch: (1) `executeExport` is either/or — `if GetStackNeedsHDD → exportHDDData ELSE exportVolumeData` — so a `needs_hdd:true` app **never** runs `exportVolumeData`, dropping its named volumes (for sonarr that is `sonarr_config`, the entire app DB/state); (2) `exportHDDData → GetStackHDDMounts → stacks.ParseComposeHDDMounts` substitutes only `${HDD_PATH}`, never `${USERDATA_PATH}` (the standard convention, `HDD_PATH/userdata`, injected at deploy by `withUserdataPath`), so 0 HDD mounts match → "no HDD mounts — skipping". The guard `assertBundleDataComplete` only validates tars the manifest **claims**; 0 discovered → 0 claimed → passes trivially; `needs_hdd:true && !has_hdd_data` is never flagged. **Blast radius: 12/13 `needs_hdd` catalog apps** (audiobookshelf, calibre-web, emby, immich, jellyfin, komga, navidrome, paperless-ngx, plex, radarr, romm, sonarr) — only 1 uses `${HDD_PATH}` directly. **Impact:** `.fab` is the customer portability/backup-download path; every media app's `.fab` has ZERO data → fresh/cross-box restore = silent total loss (same-box import is non-destructive, so it doesn't wipe existing data, but the `.fab` was still a useless backup). | Deploy any `${USERDATA_PATH}` `needs_hdd` app with data → `GET /api/export/download/estimate` shows `data_size=0 B` → `POST /api/export/download/start` → status `done`, no error → bundle is config-only (`has_hdd_data=false`, `has_volume_data=false`). Live: sonarr, 4.17 GB / 7 files → `sonarr_20260714-105232.fab` = 2308 bytes. Evidence: `evidence/phase2/C6B-F1-hollow-bundle-manifest.json` + `manifest-source.txt`. |
+
+| **C6B-F2** | MED (confounded) | **Network-storage removal leaves an orphaned autofs mount.** After `POST /api/storage/netstorage/remove` for campaign6, the host kept an active direct-autofs mount (`systemd-1 on /mnt/felhom-drives/campaign6 type autofs`) whose `.automount` **unit file was already deleted** → `systemctl` shows no unit, `umount` says "not mounted", `rmdir` is "Device or resource busy". Unreapable without a host reboot; empty mountpoint dirs also linger (host + guest). Benign (empty, no data/service impact; clears on reboot). **Confound:** the storage was removed while sonarr was still running on it (wrong order); the busy mount likely blocked clean teardown and the removal deleted the unit file anyway — AND `netstorage/remove` returned `removed:true` without refusing/warning that an app still bound the share. | `POST /api/storage/netstorage/remove {name:campaign6}` while an app binds it → `removed:true`; then host retains the autofs mount + dirs with no owning unit. **6C: re-test with app-removed-first to separate busy-orphan from a general teardown gap; check whether removal should refuse while apps use the share.** |
+
+*Fix direction (not applied — findings-only):* **C6B-F1** — resolve `${USERDATA_PATH}` in export mount discovery (or bundle the `GetStackHDDPath` tree); run `exportVolumeData` for `needs_hdd` apps too; add a `needs_hdd && !has_hdd_data` assertion to `assertBundleDataComplete`. **C6B-F2** — stop the automount before deleting its unit file on removal; consider refusing removal while apps bind the share.
+
+## What passed (headline)
+- **`.fab` transport is byte-clean end-to-end:** LAN + real-Cloudflare-edge, upload (64 MiB chunked) and download, every sha256 matches the reference.
+- **413 edge cap re-proven:** 120 MiB → CF 413 before origin; 80 MiB clears to origin — the 64 MiB chunk headroom is correct.
+- **Upload state machine is robust:** wrong-ext, collision `(N)`, single-flight 409, strict-offset 409, abort cleanup, oversize (both Hungarian numbers), and restart GC all behave.
+- **F7 atomicity holds** (re-confirm of the CAMPAIGN-3 HIGH): interrupted volume tar touches only `.tar.tmp`, never the last-good `.tar`.
+- **Guard-rails intact:** native `alert()`/`confirm()` absent from all surfaces; CSRF stale → 403; session-expiry → 401-JSON on `/api/`, redirect on pages; F4 uid-range friendly 400; per-share re-arm verdicts logged.
+
+## Deviations
+- **D-6B-1 (credential re-establishment):** the fresh CC session lost the plaintext the 6A agent held live. Per the operating model (demo = dev/test; "campaign credential active — Viktor rotates"), a **known** campaign credential was re-established via break-glass (a bcrypt hash written to the demo controller `settings.json` `password_hash`; old `$2a$` customer-claimed hash backed up at `settings.json.c6b-bak` in the container volume). Plaintext stored only in the local scratchpad, never committed. **Viktor still rotates at the end** (mandate unchanged).
+- Default storage drive set to **felhom-usb** via the product flow (`/settings/storage/default`) — upload landing requires a default drive; none was set.
+- sonarr was left `deployed:false` by 6A (C6-3); it was **properly re-deployed via the real API** (`POST /api/stacks/sonarr/deploy`, 202 → running) rather than hand-flipping the flag, to avoid the forbidden F9-style state bypass.
+
+## Box state at wrap
+- **demo (felhom-pve/9201):** controller 0.129.0, agent 0.88.0, healthy. **Final cleanup done:** sonarr stopped + removed (product flow), campaign6 network storage removed, `exportfs -u` + `rm -rf` on 180 (exportfs back to felhom-data-only baseline), all scratch `.fab` + password-bearing helpers + the injected 4.17 GB removed, samplers stopped. **Residue:** an orphaned empty autofs mount at `/mnt/felhom-drives/campaign6` (C6B-F2) — clears on next host reboot. Default drive = felhom-usb (left set). `settings.json.c6b-bak` retains the original password hash.
+- **drill (192.168.0.152/9201):** controller 0.129.0, agent 0.88.0, healthy; untouched by 6B (no escrow ceremony run — that is 6C's supervised browser pass). Escrow `escrowed`.
+- **Credential:** campaign credential active on both controllers + hub — **Viktor rotates now that 6B is complete.** No R/blob produced. Drill R stays fresh/uncommitted.
+
+## Morning recovery / handoff to 6C
+- Nothing is down. `CAMPAIGN-6C-2026-07-14-PROMPT.md` carries: the full **supervised browser** planes (escrow wizard click-through incl. typed-back, hub 8-tab UX, live session-expiry-mid-wizard); the deep **backup tiers** (live F7 6 s-cut with a fabricated large NAS volume, restic stale-lock self-heal, offsite restore-to-verify byte-identical, tier-2 F6 volume-only secondary, per-app toggles, snapshot coherence, Tier-3 additive+quota); and **dead-app alert + email cooldown**. C6B-F1 is the priority fix for project Claude to spec.
+- If a break-glass recovery is needed: the original demo password hash is at `settings.json.c6b-bak` in the `felhom-controller-data` volume.
+
+## Evidence index (`180:~/campaign6/`)
+- `seed-6B.txt` (`56975d47d2e537cd`), `ledger.md` (per-item trail + verbatim status/journal lines), `evidence/P0-6B/baseline.txt`.
+- `evidence/phase2/C6B-F1-hollow-bundle-manifest.json` (the 2308-byte bundle's manifest), `evidence/phase2/manifest-source.txt` (the real 7-file 4.17 GB tree that was dropped).
+- P7 series (samplers): `192.168.0.162` + `192.168.0.152` c4-samplers; `180` hub-sampler.

documentation/audits/CAMPAIGN-6C-2026-07-14-PROMPT.md

@@ -0,0 +1,36 @@
+# CAMPAIGN-6C — supervised browser planes + deep backup tiers + the C6B-F1 fix-verify
+
+**Class:** Continuation of CAMPAIGN-6B. The three split buckets below need either a **supervised
+browser** (6B was unattended; the browser-select step blocks with no human) or **deliberate deep
+setup** budget 6B preserved for its wrap. Findings only unless a TASK/RUNBOOK says otherwise.
+Output: `felhom.eu/documentation/audits/CAMPAIGN-6C-<date>.md` (same structure). Continue the ledger
++ evidence at `180:~/campaign6/`. Record a new seed.
+
+## Pre-existing state (verify at P0, don't trust)
+- controller **0.129.0** both guests · agent **0.88.0** both hosts (caps 63/63) · hub **0.54.0**.
+- **campaign6 share + sonarr may be GONE** — 6B's final cleanup removed them (product flow + `exportfs -u` + `rm -rf /mnt/5_hdd/felhom-campaign6`). If a NAS app is needed, re-enroll a fresh campaign export first (verify-before-commit; pre-create the userdata tree per C6-3).
+- **Credential:** the 6B campaign credential is bcrypt-installed in the demo controller `settings.json` `password_hash` ($2b$). If Viktor already rotated it, re-establish via break-glass (original hash at `settings.json.c6b-bak` in the `felhom-controller-data` volume) or the reset flow, and ledger it.
+- Access unchanged: `SSH=/c/Windows/System32/OpenSSH/ssh.exe`; 180=DooPlex, felhom-pve=demo, root@192.168.0.152=drill. Controllers via `docker exec felhom-controller curl 127.0.0.1:8080` OR claude-in-chrome (session started AFTER the bridge connected).
+
+## The one thing to watch above all: **C6B-F1 (CRITICAL, from 6B)**
+`.fab` export produces hollow, data-free bundles for `${USERDATA_PATH}` `needs_hdd` apps (12/13 media apps). Root cause + fix direction are in `CAMPAIGN-6B-2026-07-14.md`. **If project Claude ships a fix, 6C must red-proof it live:** deploy a `${USERDATA_PATH}` app with ≥1 GiB varied data → export → the bundle must now contain the userdata AND named volumes → download → delete → upload → import → **byte-compare every file vs a pre-export sha256 manifest = zero mismatches.** This is the full-circle byte-compare 6B could not run (blocked-by-bug). Until then, re-confirm the bug still reproduces.
+
+## PHASE 3 — browser planes (claude-in-chrome; supervised)
+Session started AFTER the bridge connected; the operator picks the browser at the select prompt.
+- **3A escrow wizard, full browser pass (drill):** preflight all-green + Hungarian details (no raw English leak), warnings, re-auth, run, reveal, **typed-back with the two highlighted words (read from screen)**, manual hide/show toggle, finish → auto-confirm flips + hub row hash matches (server-side check via 180). Then re-claim → 410 UI; unclaimed → TTL → `unclaimed_void` screen; **F-C live** (`phase:none` claim → clean 4xx UI, not 502); out-of-band CLI ceremony w/o staged secret → stale card fires → wizard clears it. (Drill R scratch + uncommitted.)
+- **3B live session/CSRF UX (both boxes):** session expiry mid-wizard and mid-upload → JSON 401 on `/api/`, redirect on pages (6B proved the codes via curl; 6C proves the UX); zero-toggle honesty + inline two-step confirms visible. (Native-alert sweep already PASS in 6B — spot-check only.)
+- **3C hub 8-tab customer-detail ring (hub credential = campaign credential):** hash-nav across all 8 tabs; auto-refresh scoped to live tabs; **dirty-form suppression** (start editing → refresh holds); events tab under volume; no stale-host deletion against real hosts (synthetic row only).
+
+## PHASE 4-deep — backup tiers depth (needs setup budget)
+- **F7 LIVE precise cut:** 6B code-confirmed the atomic `.tar.tmp`→rename guarantee but could not time a live 6 s cut (no app had a >6 s NAS *volume* dump). Fabricate one: a NAS app with a large docker *named volume* (or inject ≥1 GiB into one), `POST /api/backup/run`, `exportfs -u` at T+~6 s, confirm the last-good `.tar` survives, no 0-byte artifact, `success:false` is the only signal, then re-export → recovers.
+- restic stale-lock self-heal (kill controller mid-offsite-run → next run self-heals the lock); Tier-3 "Távoli mentés most" additive + quota bar; tier-1 replace-semantics; **offsite restore-to-verify** to a scratch campaign app → byte-identical; volume-only app gets its tier-2 secondary (F6 fix); per-app toggles round-trip; snapshot coherence across the four sub-pages.
+
+## PHASE 5-rest — regression
+- **Dead-app alert + email** on a killed campaign container (cooldown math vs the run's emails).
+- Re-confirm F1/F2 residue is still clean if any campaign share was re-enrolled + removed in 6C.
+
+## Wrap — same completeness gate as 6A/6B
+Checklist table (PASS/FAIL/FINDING/→6D); verdict; ranked findings + repros; timings; deviations; box
+state; morning recovery; evidence index. Commit the doc + overwrite `felhom.eu/REPORT.md`. **No
+credential, no R, no blob committed.** Final: leave the campaign credential for Viktor to rotate, or
+rotate it if this is the last leg.