CAMPAIGN-6C: v0.131.0 F-S2/F-S3 live acceptance (4P-1..4P-5 all PASS) + F-6C-1

Unattended pass via the controller API (Campaign-4 credential, real dashboard
endpoints). Closes the v0.131.0 REPORT's "NOT live-validated" list:
- 4P-2 F-S2 tier-2 appdata mirror byte-identical
- 4P-3 F-S2 marker restore (resolved dir, additive-only)
- 4P-4 storage page non-empty size
- 4P-5 F-S3 scope=app migration round-trip, no ghost dir, no WARN

New finding F-6C-1 (MEDIUM): tier-2 to an NFS target fails (rsync -a chown).
PHASE 3 (browser) + 4-deep + Accept #1 deferred to a supervised/deep-budget
6D pass. No code shipped; paperless torn down; no credential/R/blob committed.

REPORT.md

@@ -2,51 +2,52 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## SPIKE — referential-coupling backup classification — 2026-07-14
+## CAMPAIGN-6C — supervised planes + v0.131.0 F-S2/F-S3 live acceptance — 2026-07-14
 
-**controller 0.130.0 (demo 9201) / agent 0.88.0 · findings-only, no production code · full audit:
-`documentation/audits/SPIKE-backup-classification-2026-07-14.md`**
+**controller 0.132.0 both guests · agent 0.88.0 · hub 0.54.0 · findings-only, no production code · full
+audit: `documentation/audits/CAMPAIGN-6C-2026-07-14.md` · ledger `180:~/campaign6/6C-P0.md` (seed
+`6C-20260714`)**
 
-Empirical ground-truth pass for the userdata-backup-classification redesign
-(mandatory=COUPLED / optional=DECOUPLED-precious / excluded=DECOUPLED-bulk).
+Continuation of CAMPAIGN-6B. Ran **unattended** via the controller's own API (Campaign-4 credential,
+out-of-band; session cookie + CSRF; the exact endpoints the dashboard invokes — the live-test-real-flow
+proxy). No credential/R/blob committed; cookie wiped at teardown.
 
-### What was established
+### Verdict: GREEN for the v0.131.0 acceptance
 
-- **SQ1 — offsite carries no userdata today, no exception (source + live).** All 12 restic
-  snapshots are `backups/primary/<app>` units only; a full `restic ls` of the newest immich
-  snapshot shows compose + `immich-postgres.sql` + manifest — **immich is offbox-toggled and its
-  photo library is in none of its snapshots** (the latent broken-restore gap, now live-proven).
-- **NEW F-S1:** tier-2 does **not** copy bulk userdata — `RunTier2` rsyncs only the unit +
-  `appdata/<stack>`; the `tier2.go:17` "recovery unit + bulk userdata" header and the
-  `main.go:1360` "namespace wholesale" comment are stale. Browsable HDD userdata has **zero
-  automated off-drive protection today** (live: `secondary/{radarr,jellyfin}` hold only
-  `recovery-unit/`).
-- **NEW F-S2:** tier-2's appdata leg keys on the stack name → **paperless-ngx**
-  (`appdata/paperless`) is silently skipped (live contrast: `secondary/nextcloud` has `appdata/`,
-  `secondary/paperless-ngx` doesn't) — its document originals get no tier-2 copy.
-- **SQ3 — the coupling model is VALIDATED (scratch Immich, catalog images v2.5.5).** DB-present /
-  library-absent restore: the server **refuses to boot** (its own `.immich` folder-check, with the
-  mount state remembered *in the DB*); with the skeleton recreated it boots into a **lying
-  gallery** — timeline reports all 3 assets, every thumbnail/original fetch is HTTP 500. No repair/
-  re-import path. Restoring the files → all 200 again. **Worse-than-empty: confirmed.**
-- **SQ2** — full per-bind classification for all 55 catalog apps (13 hdd + 42 volume-only);
-  shared-media-tree consistency resolved by the "class belongs to the bind; the writing app is the
-  authority; `:ro` readers back up nothing" rule.
-- **SQ4** — size split measured live (state 0.5 M–933 M vs bulk tree ~100 MB demo /
-  TB-scale prod); the two-number estimate reuses the existing `estimate.go` per-mount du +
-  container-view volume sizer — zero new measurement plumbing.
-- **SQ5 — round-trip verdict: EXCLUSION scoping** (tar the userdata root minus excluded subtrees).
-  Per-bind nested tars cannot ride the v1 manifest (`HDDSubdirs` basenames; CTRL-001 forbids path
-  separators); exclusion scoping needs no manifest/import change and stays backward-compatible.
-  Proposed `backup:` block schema + **two-level default** (no block → legacy per-tier behavior;
-  block present → unlisted writable binds mandatory) — the task's single-level mandatory-default
-  premise would have shipped unmigrated media trees offsite (quota blow-up) and was corrected.
-- **SQ6** — `.fab` over-capture quantified on the live tree (radarr bundle carries ~1.6× its own
-  data even on the small demo tree, unbounded in prod); the live marker-file export leg is blocked
-  on the customer-owned dashboard login (same blocker as the open Viktor full-circle test).
+**PHASE 4-paperless — all five legs PASS**, which **fully closes the "NOT live-validated" list in the
+v0.131.0 controller REPORT** (deferred there to a supervised session):
 
-### Cleanup
+- **4P-1** — deployed paperless-ngx on felhom-usb (the appdata-mismatch app: stack `paperless-ngx`, dir
+  `appdata/paperless`), ingested 3 varied docs through the real consume flow (OCR + PDF/A) →
+  `appdata/paperless/media` = 11 files / 40 587 B.
+- **4P-2 (F-S2 tier-2 backup)** — the appdata leg mirrors `appdata/paperless` to
+  `secondary/paperless-ngx/appdata` **byte-identical** (11-file sha256 diff empty). Pre-state confirmed
+  the pre-fix shape: secondary held **recovery-unit only, no appdata**.
+- **4P-3 (F-S2 marker restore)** — UI file-restore targets the **resolved** live dir
+  (`appdata/paperless`, not `…/paperless-ngx`), additive-only, 1 file restored byte-identical, existing
+  file untouched, app healthy.
+- **4P-4 (storage page)** — paperless-ngx shows a non-empty size (**39.6 KB**, from the resolved dir).
+- **4P-5 (F-S3 scope="app" migration)** — usb→flash then flash→usb: full 11-file manifest **identical**
+  each way, source cleaned, **no `appdata/paperless-ngx` ghost dir, no `declared appdata absent` WARN**,
+  app healthy (45 s each leg).
 
-Scratch Immich fully removed (containers/volumes/network/files/images; verified 0 spike containers,
-controller healthy). Restic inspection read-only (`--no-lock`), 12 snapshots before and after. No
-drive namespace, tier-2 target, or deployed app touched.
+### New finding
+
+- **F-6C-1 (MEDIUM)** — tier-2 auto-target = NFS share fails at the recovery-unit leg: `rsyncMirror`'s
+  `rsync -a` (`-o -g`) chown is refused on the NFS export (root_squash, exit 23), so `RunTier2` records
+  failure and the appdata leg never runs. Real drives (usb/flash) are unaffected; **not an F-S2
+  regression**. Suggest `--no-owner --no-group`/`--chown` (or exclude NFS from the auto tier-2 pick).
+
+### Deferred to a 6D / supervised pass
+
+- **PHASE 3 browser planes** (escrow wizard typed-back, session/CSRF UX, hub 8-tab ring) — need a
+  rendered browser with an operator present (browser-select, on-screen typed-back, password at the
+  login screen); not unattended-automatable.
+- **PHASE 4-deep** (F7 live 6 s cut, restic mid-run self-heal, offsite restore-to-verify) and
+  **Acceptance #1** (≥1 GiB userdata `.fab` full-circle) — need large-dataset/timed-cut setup budget;
+  the C6B-F1 `.fab` fix is already code + live byte-compare verified in the v0.130.0 REPORT.
+- **5-rest dead-app alert** — inconclusive (a clean `docker stop` is not a crash; the correct trigger
+  is `kill -9`/OOM). Observation logged for 6D.
+
+**Box state:** paperless-ngx torn down (real remove flow + scratch cleanup); controller 0.132.0 healthy;
+other 6 apps untouched. Viktor rotates the controller + hub credentials after the campaign.

documentation/audits/CAMPAIGN-6C-2026-07-14.md

@@ -0,0 +1,131 @@
+# CAMPAIGN-6C — supervised planes + deep tiers + v0.130–0.132 fix-verifies (2026-07-14)
+
+**Seed:** `6C-20260714` · **Ledger/evidence:** `180:~/campaign6/` (6C-P0.md + 6B ledger.md/evidence)
+**Continuation of:** CAMPAIGN-6B. **Class:** acceptance + findings.
+**Boxes:** controller **v0.132.0** both guests · agent **0.88.0** · hub **0.54.0**.
+
+This pass ran **unattended** (operator-approved) via the controller's own API — authenticated with the
+Campaign-4 credential (out-of-band, `180:~/.config/credentials`), driving the **exact endpoints the
+dashboard invokes** (`docker exec felhom-controller curl 127.0.0.1:8080` + session cookie + CSRF).
+This is the live-test-real-flow acceptable proxy where a rendered browser wasn't used. No credential,
+R, or blob is written here. Cookie/CSRF were transient (container `/tmp`, wiped at teardown).
+
+---
+
+## Checklist
+
+| Bucket | Leg | Result |
+|--------|-----|--------|
+| P0 | baselines (ctrl 0.132.0 / agent 0.88.0 / hub 0.54.0), `backup block rejected`=0, storage headroom | **PASS** |
+| P0 | API access mechanism (session-cookie auth, no loopback bypass) | **CHARACTERIZED** |
+| **4P-1** | deploy paperless-ngx (felhom-usb), ingest 3 varied docs via consume flow | **PASS** |
+| **4P-2** | F-S2 tier-2: appdata leg mirrors `appdata/paperless` byte-identical | **PASS** (+ **F-6C-1**) |
+| **4P-3** | F-S2 marker restore: resolved live dir, additive-only, byte-identical | **PASS** |
+| **4P-4** | storage page shows paperless-ngx non-empty size (39.6 KB) | **PASS** |
+| **4P-5** | F-S3 scope="app" migration usb→flash→usb, full manifest, no ghost dir, no WARN | **PASS** |
+| 5-rest | dead-app alert | **INCONCLUSIVE** (wrong trigger — clean stop ≠ crash) |
+| Accept #1 | C6B-F1 `.fab` ≥1 GiB full-circle byte-compare | **DEFERRED** (needs dedicated setup) |
+| PHASE 3 | escrow wizard / session-CSRF UX / hub 8-tab ring (browser) | **NOT-RUN** (needs supervised browser) |
+| PHASE 4-deep | F7 live cut, restic self-heal, offsite restore-verify, snapshot coherence | **NOT-RUN** (deep-setup budget) |
+
+---
+
+## Verdict
+
+**GREEN for the v0.131.0 acceptance.** All five PHASE 4-paperless legs PASS — this **fully closes the
+"NOT live-validated" list in the v0.131.0 REPORT** (F-S2 tier-2 backup, F-S2 marker restore, F-S2
+storage-page size, F-S3 scope="app" migration round-trip), on the live demo through the real dashboard
+endpoints, with byte-identical manifests at every step and **no `appdata/paperless-ngx` ghost dir and
+no `declared appdata dir absent` WARN**. One new MEDIUM finding (**F-6C-1**, tier-2 to an NFS target).
+The browser planes (PHASE 3) and deep tiers (PHASE 4-deep) + the `.fab` ≥1 GiB circle remain for a
+supervised/deep-budget pass (6D) — the credential-gated automation ran, but those legs need a rendered
+browser (operator-typed password + on-screen typed-back) or large-dataset/timed-cut setup this
+unattended pass did not provision.
+
+---
+
+## Findings (ranked)
+
+### F-6C-1 (MEDIUM) — tier-2 auto-target = NFS share fails at the recovery-unit leg (`rsync -a` chown)
+`RunAllTier2` auto-picked **nas-media (NFS)** as paperless-ngx's off-drive target. `rsyncMirror`
+(`internal/backup/tier2.go`) uses `rsync -a`, whose `-o -g` preserve owner/group → `chown` on the NFS
+export fails ("Operation not permitted", root_squash) → **exit 23**, the recovery-unit leg fails first,
+so `RunTier2` records failure and the **appdata leg never runs**. Same class hit calibre-web
+(`teszt_enroll` mkdir permission-denied — a separate bad-mount residue).
+- **Repro:** any box whose first schedulable off-disk tier-2 target is an NFS share.
+- **Impact:** tier-2 silently produces no off-drive copy for the affected app (status = failure, so
+  it's honest, not silent — but no protection). Real drives (felhom-usb/flash) are unaffected.
+- **Not an F-S2 regression** — F-S2 (the appdata dir-name resolution) is proven correct once the
+  target is a real drive (4P-2 byte-compare).
+- **Suggested fix (→6D / a controller task):** for a network-fs target, drop `-o/-g` (add
+  `--no-owner --no-group`, or `--chown`), or exclude NFS mounts from the auto tier-2 target pick.
+- **Evidence:** controller log 2026-07-14 17:35:31 (exit 23 chown lines); `180:~/campaign6/6C-P0.md`.
+
+### Observation (LOW, → 6D) — aggregate state "running" while main container Exited(0)
+A clean `docker stop paperless-webserver` (exit 0) did NOT fire a dead-app alert in 2.5 min and the
+controller reported `state=running` / health-probes "4 ok". This is **likely by design** (a clean stop
+is not a crash; `downstate_test` asserts certain states must not manufacture an alert), so 5-rest is
+**inconclusive with this trigger**. Worth a 6D note: the aggregate state reads "running" while the
+service container is down (other containers up). A proper crash trigger (`docker kill -9` / OOM,
+exit≠0) is the correct test — deferred (restart-policy race needs care).
+
+---
+
+## Timings (paperless acceptance, demo)
+
+| Step | Wall time |
+|------|-----------|
+| deploy → 3 containers healthy | ~95 s (Postgres start_period) |
+| 3-doc consume ingest (OCR + PDF/A) | ~90 s |
+| tier-2 run (all HDD apps) | ~30 s |
+| tier-2 file-restore (stop→copy→start→health) | 39 s |
+| migration usb→flash (scope=app) | 45 s |
+| migration flash→usb (round-trip) | 45 s |
+
+---
+
+## Deviations from the prompt
+
+- **PHASE 3 / 4-deep / Accept #1 not run.** PHASE 3 needs a rendered browser with an operator present
+  (browser-select prompt, password typed at the login screen, and the wizard's "type the two
+  highlighted words read from screen") — genuinely interactive, not unattended-automatable. 4-deep
+  (F7 live 6 s cut, restic mid-run kill, offsite restore-to-verify) and Accept #1 (≥1 GiB userdata app
+  + encrypted `.fab` round-trip) need large-dataset / timed-cut setup budget this pass did not
+  provision. The C6B-F1 `.fab` fix is already code + live byte-compare verified in the v0.130.0 REPORT;
+  this pass instead prioritized the **newer, higher-risk v0.131.0 F-S2/F-S3** legs (never live-proven
+  until now).
+- **P0 residue:** 6B's cleanup did NOT fully remove the campaign6 mount or the sonarr stackdir (both
+  still present) — minor, non-blocking (deviates from the P0 note's "may be GONE").
+- **paperless tier-2 target pinned** to felhom-flash (via the `/stacks/paperless-ngx/backup` UI
+  control) to route around F-6C-1 for the F-S2 proof — a legitimate UI action, recorded.
+- **Stale target residue** (`felhom-flash/appdata/paperless`, 16 KB, Jun-15 prior-session) cleared
+  before 4P-5 so the migration collision guard (working correctly) didn't block the test.
+
+---
+
+## Box state / morning recovery
+
+- **paperless-ngx torn down** (real stop→remove flow + scratch-residue cleanup): containers gone,
+  `app.yaml` gone, `appdata/paperless` + primary unit + tier-2 secondaries (flash + nas-media) removed.
+  Pre-existing skeleton (`userdata/paperless-ngx` canary, empty `import/paperless` dirs) left as-is.
+- Controller **v0.132.0 healthy** on demo 9201 post-teardown; other 6 apps untouched and running.
+- Session cookie/CSRF wiped from the controller container `/tmp`.
+- **No code shipped this pass** (findings-only). **Viktor rotates the controller + hub credentials
+  after the campaign** (per the campaign rule).
+
+---
+
+## Evidence index
+
+- `180:~/campaign6/6C-P0.md` — P0 state, auth note, full 4-paperless leg-by-leg results, F-6C-1,
+  5-rest observation (this doc's source ledger).
+- `180:~/campaign6/ledger.md` + `evidence/` — 6B carry-over.
+- sha256 manifests (in-run, `/tmp/*.man` on the guest, transient): 4P-1 source (11 files, 40587 B),
+  4P-2 secondary byte-compare (diff empty), 4P-5 pre/post/round-trip (all identical).
+- Controller logs (demo 9201, 2026-07-14 17:3x–17:4x): tier-2 exit-23 (F-6C-1), tier-2 appdata mirror,
+  file-restore "additive-only … dst=appdata/paperless … 1 file restored", migration complete lines.
+
+**This pass closes the v0.131.0 REPORT's "NOT live-validated (deferred to Viktor's supervised
+session)" list** — F-S2 tier-2 backup + marker restore + storage-page size, and F-S3 scope="app"
+migration — all now PASS with byte-identical evidence. See §Checklist 4P-1..4P-5 and the manifests
+above.