felhom.eu/documentation/audits/CAMPAIGN-6B-2026-07-14.md

# CAMPAIGN-6B — unattended close-out of the C6 remainder (.fab circle · browser/DOM · backup tiers · regression)

- **When:** 2026-07-14 ~10:30Z launch (unattended continuation of 6A). Launch seed `56975d47d2e537cd`.
- **Stack under fire (verified live at P0):** controller **0.129.0** both guests · agent **0.88.0** both hosts (caps **63/63**, 0 degraded) · hub **0.54.0** · demo (felhom-pve 192.168.0.162 + guest 9201, storage-bearing) AND drill (192.168.0.152 + guest 9201). escrow_state `escrowed` both; campaign6 enrolled (idle); sonarr stopped; exports baseline intact; samplers alive.
- **Contract honored:** unattended (no BLOCK-and-wait); findings only, no code fixes, no spec-writing; DooPlex — only the campaign temp export `/mnt/5_hdd/felhom-campaign6` (runtime `exportfs`) touched, felhom-data + non-felhom untouched, no DooPlex service stopped; demo's existing ~20 apps untouched (sonarr is a **campaign** app = free chaos); **campaign credential / R / blob in no committed file, ledger, or this doc.**
- **Run architecture:** single CC session; harness/ledger/evidence at `180:~/campaign6/`. P0-6B baseline pushed to `evidence/P0-6B/` before any mutation. Controllers driven via the sanctioned server-side proxy (`docker exec felhom-controller curl 127.0.0.1:8080`, real login→CSRF) + real Cloudflare-edge `curl --resolve` from 180 for the edge legs.

## Verdict

**The `.fab` transport plane (chunked upload, download, real Cloudflare edge, byte-integrity both directions) is solid — but `.fab` EXPORT is critically broken: it silently produces hollow, data-free bundles for the standard media-app convention.** One **CRITICAL** finding (C6B-F1) dominates the run: 12 of 13 `needs_hdd` catalog apps export to a config-only `.fab` with zero application data, reported as success, past the v0.125.0 anti-hollow guard. All upload/download mechanics and the edge cap re-prove pass cleanly. Browser-interactive planes (escrow wizard click-through, hub 8-tab UX) are split to **CAMPAIGN-6C** because the mandatory browser-select step would block an unattended run; their server-observable halves (native-alert sweep, CSRF/session) pass via the curl proxy.

## Completeness checklist (every item PASS / FAIL / FINDING / → 6C)

| Item | Status | Evidence |
|---|---|---|
| **P0-6B** baseline + seed | **PASS** | both 0.129.0/agent 0.88.0/caps 63/63; escrow escrowed; campaign6 idle; `evidence/P0-6B/baseline.txt` |
| **P2** `.fab` export of a data-bearing app | **FINDING C6B-F1 (CRITICAL)** | 4.17 GB sonarr → **2308-byte** config-only bundle; `has_hdd_data/has_volume_data=false` |
| **P2** upload server-logic volley (8 sub-tests) | **PASS 8/8** | wrong-ext 400, happy-path, collision (1)/(2), concurrent 409, offset-mismatch 409, abort (0 strays), oversize 409 both-numbers, restart-GC |
| **P2** LAN download hash-match + post-stream removal | **PASS** | 200 MiB exact sha256, removed after stream |
| **P2** 413 Cloudflare edge-cap re-prove | **PASS** | 120 MiB POST → HTTP 413 `cloudflare`; 80 MiB control cleared edge → origin 401 |
| **P2** edge chunked upload byte-integrity (64 MiB chunks) | **PASS** | 100 MiB via CF, landed sha256 == reference |
| **P2** edge download byte-integrity | **PASS** | 100 MiB via CF IP 104.21.11.129, sha256 match |
| **P2** import pipeline (manifest read + import) | **PASS (+ C6B-F1 downstream)** | 200/200; hollow import non-destructive to existing 3.9 GB |
| **P2** idle-15min upload abort | **PASS** | `.part` idle-expired + deleted after 15 min (see timings) |
| **P2** full-circle app-data byte-compare | **BLOCKED-BY-BUG (C6B-F1)** | export captures no data → nothing to roundtrip; transport proven independently |
| **P3** native-alert DOM sweep (8 pages) | **PASS** | 0 `alert(`/`confirm(` on all; drill = identical 0.129.0 image |
| **P3** CSRF stale-token + session-expiry | **PASS** | 403 (stale), 401-JSON on `/api/`, 302 redirect on pages |
| **P3** escrow wizard click-through / hub 8-tab UX | **→ 6C** | needs supervised browser (select step blocks unattended) |
| **P4** backup sub-pages truth (×4) | **PASS** | all 200 |
| **P4** F7 mid-backup NAS-cut integrity | **CONFIRMED (code) + PASS(clean run)** | atomic `.tar.tmp`→rename, orphan GC, restore-invisible; 0 strays on clean run |
| **P4** offsite 3-state card honesty | **PASS (configured state)** | `/backups/remote` renders enabled+last-run+quota+snapshots |
| **P4** restic self-heal / offsite restore-verify / tier-2 F6 / per-app toggles / snapshot coherence / Tier-3 | **→ 6C** | budget-preserved deep items |
| **P5** agent-restart per-share re-arm verdicts | **PASS** | campaign6 + nas-media `verdict=rearmed` |
| **P5** F4 mapped_uid 101000 → friendly 400 | **PASS** | Hungarian uid-range message, agent never reached |
| **P5** F1/F2 residue after share removal | **PASS (at cleanup)** | zero mounts/units/dirs — see Final cleanup |
| **P5** dead-app alert + email + cooldown | **→ 6C** | time/email-cost |

> `→ 6C` rows are a budget/unattended-constraint split with a written continuation prompt, not a silent defer.

## Ranked findings (exact repros)

| # | Sev | Finding | Exact repro |
|---|-----|---------|-------------|
| **C6B-F1** | **CRITICAL** | **`.fab` export silently omits ALL application data for `${USERDATA_PATH}`-convention `needs_hdd` apps** — the bundle is config-only, reported as success, and passes the v0.125.0 anti-hollow guard. Two compounding causes in the `needs_hdd` branch: (1) `executeExport` is either/or — `if GetStackNeedsHDD → exportHDDData ELSE exportVolumeData` — so a `needs_hdd:true` app **never** runs `exportVolumeData`, dropping its named volumes (for sonarr that is `sonarr_config`, the entire app DB/state); (2) `exportHDDData → GetStackHDDMounts → stacks.ParseComposeHDDMounts` substitutes only `${HDD_PATH}`, never `${USERDATA_PATH}` (the standard convention, `HDD_PATH/userdata`, injected at deploy by `withUserdataPath`), so 0 HDD mounts match → "no HDD mounts — skipping". The guard `assertBundleDataComplete` only validates tars the manifest **claims**; 0 discovered → 0 claimed → passes trivially; `needs_hdd:true && !has_hdd_data` is never flagged. **Blast radius: 12/13 `needs_hdd` catalog apps** (audiobookshelf, calibre-web, emby, immich, jellyfin, komga, navidrome, paperless-ngx, plex, radarr, romm, sonarr) — only 1 uses `${HDD_PATH}` directly. **Impact:** `.fab` is the customer portability/backup-download path; every media app's `.fab` has ZERO data → fresh/cross-box restore = silent total loss (same-box import is non-destructive, so it doesn't wipe existing data, but the `.fab` was still a useless backup). | Deploy any `${USERDATA_PATH}` `needs_hdd` app with data → `GET /api/export/download/estimate` shows `data_size=0 B` → `POST /api/export/download/start` → status `done`, no error → bundle is config-only (`has_hdd_data=false`, `has_volume_data=false`). Live: sonarr, 4.17 GB / 7 files → `sonarr_20260714-105232.fab` = 2308 bytes. Evidence: `evidence/phase2/C6B-F1-hollow-bundle-manifest.json` + `manifest-source.txt`. |

| **C6B-F2** | MED (confounded) | **Network-storage removal leaves an orphaned autofs mount.** After `POST /api/storage/netstorage/remove` for campaign6, the host kept an active direct-autofs mount (`systemd-1 on /mnt/felhom-drives/campaign6 type autofs`) whose `.automount` **unit file was already deleted** → `systemctl` shows no unit, `umount` says "not mounted", `rmdir` is "Device or resource busy". Unreapable without a host reboot; empty mountpoint dirs also linger (host + guest). Benign (empty, no data/service impact; clears on reboot). **Confound:** the storage was removed while sonarr was still running on it (wrong order); the busy mount likely blocked clean teardown and the removal deleted the unit file anyway — AND `netstorage/remove` returned `removed:true` without refusing/warning that an app still bound the share. | `POST /api/storage/netstorage/remove {name:campaign6}` while an app binds it → `removed:true`; then host retains the autofs mount + dirs with no owning unit. **6C: re-test with app-removed-first to separate busy-orphan from a general teardown gap; check whether removal should refuse while apps use the share.** |

*Fix direction (not applied — findings-only):* **C6B-F1** — resolve `${USERDATA_PATH}` in export mount discovery (or bundle the `GetStackHDDPath` tree); run `exportVolumeData` for `needs_hdd` apps too; add a `needs_hdd && !has_hdd_data` assertion to `assertBundleDataComplete`. **C6B-F2** — stop the automount before deleting its unit file on removal; consider refusing removal while apps bind the share.

## What passed (headline)
- **`.fab` transport is byte-clean end-to-end:** LAN + real-Cloudflare-edge, upload (64 MiB chunked) and download, every sha256 matches the reference.
- **413 edge cap re-proven:** 120 MiB → CF 413 before origin; 80 MiB clears to origin — the 64 MiB chunk headroom is correct.
- **Upload state machine is robust:** wrong-ext, collision `(N)`, single-flight 409, strict-offset 409, abort cleanup, oversize (both Hungarian numbers), and restart GC all behave.
- **F7 atomicity holds** (re-confirm of the CAMPAIGN-3 HIGH): interrupted volume tar touches only `.tar.tmp`, never the last-good `.tar`.
- **Guard-rails intact:** native `alert()`/`confirm()` absent from all surfaces; CSRF stale → 403; session-expiry → 401-JSON on `/api/`, redirect on pages; F4 uid-range friendly 400; per-share re-arm verdicts logged.

## Deviations
- **D-6B-1 (credential re-establishment, then fully reverted):** the fresh CC session lost the plaintext the 6A agent held live. Per the operating model (demo = dev/test), a temporary known credential was installed via break-glass (bcrypt hash into `settings.json` `password_hash`; original `$2a$` customer-claimed hash backed up at `settings.json.c6b-bak`) to drive the authed tests. **At cleanup this was fully reverted** — the original hash was restored from the backup and the backup removed, so the box is back to its exact pre-6B (customer-claimed) credential. Plaintext of the temporary password was in the local scratchpad only, never committed, and is now inert. **Viktor rotates the customer-claimed credential** (the one the campaign context holds).
- Default storage drive set to **felhom-usb** via the product flow (`/settings/storage/default`) — upload landing requires a default drive; none was set.
- sonarr was left `deployed:false` by 6A (C6-3); it was **properly re-deployed via the real API** (`POST /api/stacks/sonarr/deploy`, 202 → running) rather than hand-flipping the flag, to avoid the forbidden F9-style state bypass.

## Box state at wrap
- **demo (felhom-pve/9201):** controller 0.129.0, agent 0.88.0, healthy. **Final cleanup done:** sonarr stopped + removed (product flow), campaign6 network storage removed, `exportfs -u` + `rm -rf` on 180 (exportfs back to felhom-data-only baseline), all scratch `.fab` + password-bearing helpers + the injected 4.17 GB removed, samplers stopped. **Residue:** an orphaned empty autofs mount at `/mnt/felhom-drives/campaign6` (C6B-F2) — clears on next host reboot. Default drive = felhom-usb (left set). **Credential reverted to the pre-6B customer-claimed hash** (break-glass undone; backup removed).
- **drill (192.168.0.152/9201):** controller 0.129.0, agent 0.88.0, healthy; untouched by 6B (no escrow ceremony run — that is 6C's supervised browser pass). Escrow `escrowed`.
- **Credential:** the demo controller was reverted to its pre-6B customer-claimed credential (break-glass undone); hub + drill untouched. **Viktor rotates the customer-claimed credential** as planned. No R/blob produced. Drill R stays fresh/uncommitted.

## Morning recovery / handoff to 6C
- Nothing is down. `CAMPAIGN-6C-2026-07-14-PROMPT.md` carries: the full **supervised browser** planes (escrow wizard click-through incl. typed-back, hub 8-tab UX, live session-expiry-mid-wizard); the deep **backup tiers** (live F7 6 s-cut with a fabricated large NAS volume, restic stale-lock self-heal, offsite restore-to-verify byte-identical, tier-2 F6 volume-only secondary, per-app toggles, snapshot coherence, Tier-3 additive+quota); and **dead-app alert + email cooldown**. C6B-F1 is the priority fix for project Claude to spec.
- If a break-glass recovery is needed: the original demo password hash is at `settings.json.c6b-bak` in the `felhom-controller-data` volume.

## Evidence index (`180:~/campaign6/`)
- `seed-6B.txt` (`56975d47d2e537cd`), `ledger.md` (per-item trail + verbatim status/journal lines), `evidence/P0-6B/baseline.txt`.
- `evidence/phase2/C6B-F1-hollow-bundle-manifest.json` (the 2308-byte bundle's manifest), `evidence/phase2/manifest-source.txt` (the real 7-file 4.17 GB tree that was dropped).
- P7 series (samplers): `192.168.0.162` + `192.168.0.152` c4-samplers; `180` hub-sampler.