diff --git a/REPORT.md b/REPORT.md index 381016b..dcd849f 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,51 +2,52 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## SPIKE — referential-coupling backup classification — 2026-07-14 +## CAMPAIGN-6C — supervised planes + v0.131.0 F-S2/F-S3 live acceptance — 2026-07-14 -**controller 0.130.0 (demo 9201) / agent 0.88.0 · findings-only, no production code · full audit: -`documentation/audits/SPIKE-backup-classification-2026-07-14.md`** +**controller 0.132.0 both guests · agent 0.88.0 · hub 0.54.0 · findings-only, no production code · full +audit: `documentation/audits/CAMPAIGN-6C-2026-07-14.md` · ledger `180:~/campaign6/6C-P0.md` (seed +`6C-20260714`)** -Empirical ground-truth pass for the userdata-backup-classification redesign -(mandatory=COUPLED / optional=DECOUPLED-precious / excluded=DECOUPLED-bulk). +Continuation of CAMPAIGN-6B. Ran **unattended** via the controller's own API (Campaign-4 credential, +out-of-band; session cookie + CSRF; the exact endpoints the dashboard invokes — the live-test-real-flow +proxy). No credential/R/blob committed; cookie wiped at teardown. -### What was established +### Verdict: GREEN for the v0.131.0 acceptance -- **SQ1 — offsite carries no userdata today, no exception (source + live).** All 12 restic - snapshots are `backups/primary/` units only; a full `restic ls` of the newest immich - snapshot shows compose + `immich-postgres.sql` + manifest — **immich is offbox-toggled and its - photo library is in none of its snapshots** (the latent broken-restore gap, now live-proven). -- **NEW F-S1:** tier-2 does **not** copy bulk userdata — `RunTier2` rsyncs only the unit + - `appdata/`; the `tier2.go:17` "recovery unit + bulk userdata" header and the - `main.go:1360` "namespace wholesale" comment are stale. Browsable HDD userdata has **zero - automated off-drive protection today** (live: `secondary/{radarr,jellyfin}` hold only - `recovery-unit/`). -- **NEW F-S2:** tier-2's appdata leg keys on the stack name → **paperless-ngx** - (`appdata/paperless`) is silently skipped (live contrast: `secondary/nextcloud` has `appdata/`, - `secondary/paperless-ngx` doesn't) — its document originals get no tier-2 copy. -- **SQ3 — the coupling model is VALIDATED (scratch Immich, catalog images v2.5.5).** DB-present / - library-absent restore: the server **refuses to boot** (its own `.immich` folder-check, with the - mount state remembered *in the DB*); with the skeleton recreated it boots into a **lying - gallery** — timeline reports all 3 assets, every thumbnail/original fetch is HTTP 500. No repair/ - re-import path. Restoring the files → all 200 again. **Worse-than-empty: confirmed.** -- **SQ2** — full per-bind classification for all 55 catalog apps (13 hdd + 42 volume-only); - shared-media-tree consistency resolved by the "class belongs to the bind; the writing app is the - authority; `:ro` readers back up nothing" rule. -- **SQ4** — size split measured live (state 0.5 M–933 M vs bulk tree ~100 MB demo / - TB-scale prod); the two-number estimate reuses the existing `estimate.go` per-mount du + - container-view volume sizer — zero new measurement plumbing. -- **SQ5 — round-trip verdict: EXCLUSION scoping** (tar the userdata root minus excluded subtrees). - Per-bind nested tars cannot ride the v1 manifest (`HDDSubdirs` basenames; CTRL-001 forbids path - separators); exclusion scoping needs no manifest/import change and stays backward-compatible. - Proposed `backup:` block schema + **two-level default** (no block → legacy per-tier behavior; - block present → unlisted writable binds mandatory) — the task's single-level mandatory-default - premise would have shipped unmigrated media trees offsite (quota blow-up) and was corrected. -- **SQ6** — `.fab` over-capture quantified on the live tree (radarr bundle carries ~1.6× its own - data even on the small demo tree, unbounded in prod); the live marker-file export leg is blocked - on the customer-owned dashboard login (same blocker as the open Viktor full-circle test). +**PHASE 4-paperless — all five legs PASS**, which **fully closes the "NOT live-validated" list in the +v0.131.0 controller REPORT** (deferred there to a supervised session): -### Cleanup +- **4P-1** — deployed paperless-ngx on felhom-usb (the appdata-mismatch app: stack `paperless-ngx`, dir + `appdata/paperless`), ingested 3 varied docs through the real consume flow (OCR + PDF/A) → + `appdata/paperless/media` = 11 files / 40 587 B. +- **4P-2 (F-S2 tier-2 backup)** — the appdata leg mirrors `appdata/paperless` to + `secondary/paperless-ngx/appdata` **byte-identical** (11-file sha256 diff empty). Pre-state confirmed + the pre-fix shape: secondary held **recovery-unit only, no appdata**. +- **4P-3 (F-S2 marker restore)** — UI file-restore targets the **resolved** live dir + (`appdata/paperless`, not `…/paperless-ngx`), additive-only, 1 file restored byte-identical, existing + file untouched, app healthy. +- **4P-4 (storage page)** — paperless-ngx shows a non-empty size (**39.6 KB**, from the resolved dir). +- **4P-5 (F-S3 scope="app" migration)** — usb→flash then flash→usb: full 11-file manifest **identical** + each way, source cleaned, **no `appdata/paperless-ngx` ghost dir, no `declared appdata absent` WARN**, + app healthy (45 s each leg). -Scratch Immich fully removed (containers/volumes/network/files/images; verified 0 spike containers, -controller healthy). Restic inspection read-only (`--no-lock`), 12 snapshots before and after. No -drive namespace, tier-2 target, or deployed app touched. +### New finding + +- **F-6C-1 (MEDIUM)** — tier-2 auto-target = NFS share fails at the recovery-unit leg: `rsyncMirror`'s + `rsync -a` (`-o -g`) chown is refused on the NFS export (root_squash, exit 23), so `RunTier2` records + failure and the appdata leg never runs. Real drives (usb/flash) are unaffected; **not an F-S2 + regression**. Suggest `--no-owner --no-group`/`--chown` (or exclude NFS from the auto tier-2 pick). + +### Deferred to a 6D / supervised pass + +- **PHASE 3 browser planes** (escrow wizard typed-back, session/CSRF UX, hub 8-tab ring) — need a + rendered browser with an operator present (browser-select, on-screen typed-back, password at the + login screen); not unattended-automatable. +- **PHASE 4-deep** (F7 live 6 s cut, restic mid-run self-heal, offsite restore-to-verify) and + **Acceptance #1** (≥1 GiB userdata `.fab` full-circle) — need large-dataset/timed-cut setup budget; + the C6B-F1 `.fab` fix is already code + live byte-compare verified in the v0.130.0 REPORT. +- **5-rest dead-app alert** — inconclusive (a clean `docker stop` is not a crash; the correct trigger + is `kill -9`/OOM). Observation logged for 6D. + +**Box state:** paperless-ngx torn down (real remove flow + scratch cleanup); controller 0.132.0 healthy; +other 6 apps untouched. Viktor rotates the controller + hub credentials after the campaign. diff --git a/documentation/audits/CAMPAIGN-6C-2026-07-14.md b/documentation/audits/CAMPAIGN-6C-2026-07-14.md new file mode 100644 index 0000000..39d8c88 --- /dev/null +++ b/documentation/audits/CAMPAIGN-6C-2026-07-14.md @@ -0,0 +1,131 @@ +# CAMPAIGN-6C — supervised planes + deep tiers + v0.130–0.132 fix-verifies (2026-07-14) + +**Seed:** `6C-20260714` · **Ledger/evidence:** `180:~/campaign6/` (6C-P0.md + 6B ledger.md/evidence) +**Continuation of:** CAMPAIGN-6B. **Class:** acceptance + findings. +**Boxes:** controller **v0.132.0** both guests · agent **0.88.0** · hub **0.54.0**. + +This pass ran **unattended** (operator-approved) via the controller's own API — authenticated with the +Campaign-4 credential (out-of-band, `180:~/.config/credentials`), driving the **exact endpoints the +dashboard invokes** (`docker exec felhom-controller curl 127.0.0.1:8080` + session cookie + CSRF). +This is the live-test-real-flow acceptable proxy where a rendered browser wasn't used. No credential, +R, or blob is written here. Cookie/CSRF were transient (container `/tmp`, wiped at teardown). + +--- + +## Checklist + +| Bucket | Leg | Result | +|--------|-----|--------| +| P0 | baselines (ctrl 0.132.0 / agent 0.88.0 / hub 0.54.0), `backup block rejected`=0, storage headroom | **PASS** | +| P0 | API access mechanism (session-cookie auth, no loopback bypass) | **CHARACTERIZED** | +| **4P-1** | deploy paperless-ngx (felhom-usb), ingest 3 varied docs via consume flow | **PASS** | +| **4P-2** | F-S2 tier-2: appdata leg mirrors `appdata/paperless` byte-identical | **PASS** (+ **F-6C-1**) | +| **4P-3** | F-S2 marker restore: resolved live dir, additive-only, byte-identical | **PASS** | +| **4P-4** | storage page shows paperless-ngx non-empty size (39.6 KB) | **PASS** | +| **4P-5** | F-S3 scope="app" migration usb→flash→usb, full manifest, no ghost dir, no WARN | **PASS** | +| 5-rest | dead-app alert | **INCONCLUSIVE** (wrong trigger — clean stop ≠ crash) | +| Accept #1 | C6B-F1 `.fab` ≥1 GiB full-circle byte-compare | **DEFERRED** (needs dedicated setup) | +| PHASE 3 | escrow wizard / session-CSRF UX / hub 8-tab ring (browser) | **NOT-RUN** (needs supervised browser) | +| PHASE 4-deep | F7 live cut, restic self-heal, offsite restore-verify, snapshot coherence | **NOT-RUN** (deep-setup budget) | + +--- + +## Verdict + +**GREEN for the v0.131.0 acceptance.** All five PHASE 4-paperless legs PASS — this **fully closes the +"NOT live-validated" list in the v0.131.0 REPORT** (F-S2 tier-2 backup, F-S2 marker restore, F-S2 +storage-page size, F-S3 scope="app" migration round-trip), on the live demo through the real dashboard +endpoints, with byte-identical manifests at every step and **no `appdata/paperless-ngx` ghost dir and +no `declared appdata dir absent` WARN**. One new MEDIUM finding (**F-6C-1**, tier-2 to an NFS target). +The browser planes (PHASE 3) and deep tiers (PHASE 4-deep) + the `.fab` ≥1 GiB circle remain for a +supervised/deep-budget pass (6D) — the credential-gated automation ran, but those legs need a rendered +browser (operator-typed password + on-screen typed-back) or large-dataset/timed-cut setup this +unattended pass did not provision. + +--- + +## Findings (ranked) + +### F-6C-1 (MEDIUM) — tier-2 auto-target = NFS share fails at the recovery-unit leg (`rsync -a` chown) +`RunAllTier2` auto-picked **nas-media (NFS)** as paperless-ngx's off-drive target. `rsyncMirror` +(`internal/backup/tier2.go`) uses `rsync -a`, whose `-o -g` preserve owner/group → `chown` on the NFS +export fails ("Operation not permitted", root_squash) → **exit 23**, the recovery-unit leg fails first, +so `RunTier2` records failure and the **appdata leg never runs**. Same class hit calibre-web +(`teszt_enroll` mkdir permission-denied — a separate bad-mount residue). +- **Repro:** any box whose first schedulable off-disk tier-2 target is an NFS share. +- **Impact:** tier-2 silently produces no off-drive copy for the affected app (status = failure, so + it's honest, not silent — but no protection). Real drives (felhom-usb/flash) are unaffected. +- **Not an F-S2 regression** — F-S2 (the appdata dir-name resolution) is proven correct once the + target is a real drive (4P-2 byte-compare). +- **Suggested fix (→6D / a controller task):** for a network-fs target, drop `-o/-g` (add + `--no-owner --no-group`, or `--chown`), or exclude NFS mounts from the auto tier-2 target pick. +- **Evidence:** controller log 2026-07-14 17:35:31 (exit 23 chown lines); `180:~/campaign6/6C-P0.md`. + +### Observation (LOW, → 6D) — aggregate state "running" while main container Exited(0) +A clean `docker stop paperless-webserver` (exit 0) did NOT fire a dead-app alert in 2.5 min and the +controller reported `state=running` / health-probes "4 ok". This is **likely by design** (a clean stop +is not a crash; `downstate_test` asserts certain states must not manufacture an alert), so 5-rest is +**inconclusive with this trigger**. Worth a 6D note: the aggregate state reads "running" while the +service container is down (other containers up). A proper crash trigger (`docker kill -9` / OOM, +exit≠0) is the correct test — deferred (restart-policy race needs care). + +--- + +## Timings (paperless acceptance, demo) + +| Step | Wall time | +|------|-----------| +| deploy → 3 containers healthy | ~95 s (Postgres start_period) | +| 3-doc consume ingest (OCR + PDF/A) | ~90 s | +| tier-2 run (all HDD apps) | ~30 s | +| tier-2 file-restore (stop→copy→start→health) | 39 s | +| migration usb→flash (scope=app) | 45 s | +| migration flash→usb (round-trip) | 45 s | + +--- + +## Deviations from the prompt + +- **PHASE 3 / 4-deep / Accept #1 not run.** PHASE 3 needs a rendered browser with an operator present + (browser-select prompt, password typed at the login screen, and the wizard's "type the two + highlighted words read from screen") — genuinely interactive, not unattended-automatable. 4-deep + (F7 live 6 s cut, restic mid-run kill, offsite restore-to-verify) and Accept #1 (≥1 GiB userdata app + + encrypted `.fab` round-trip) need large-dataset / timed-cut setup budget this pass did not + provision. The C6B-F1 `.fab` fix is already code + live byte-compare verified in the v0.130.0 REPORT; + this pass instead prioritized the **newer, higher-risk v0.131.0 F-S2/F-S3** legs (never live-proven + until now). +- **P0 residue:** 6B's cleanup did NOT fully remove the campaign6 mount or the sonarr stackdir (both + still present) — minor, non-blocking (deviates from the P0 note's "may be GONE"). +- **paperless tier-2 target pinned** to felhom-flash (via the `/stacks/paperless-ngx/backup` UI + control) to route around F-6C-1 for the F-S2 proof — a legitimate UI action, recorded. +- **Stale target residue** (`felhom-flash/appdata/paperless`, 16 KB, Jun-15 prior-session) cleared + before 4P-5 so the migration collision guard (working correctly) didn't block the test. + +--- + +## Box state / morning recovery + +- **paperless-ngx torn down** (real stop→remove flow + scratch-residue cleanup): containers gone, + `app.yaml` gone, `appdata/paperless` + primary unit + tier-2 secondaries (flash + nas-media) removed. + Pre-existing skeleton (`userdata/paperless-ngx` canary, empty `import/paperless` dirs) left as-is. +- Controller **v0.132.0 healthy** on demo 9201 post-teardown; other 6 apps untouched and running. +- Session cookie/CSRF wiped from the controller container `/tmp`. +- **No code shipped this pass** (findings-only). **Viktor rotates the controller + hub credentials + after the campaign** (per the campaign rule). + +--- + +## Evidence index + +- `180:~/campaign6/6C-P0.md` — P0 state, auth note, full 4-paperless leg-by-leg results, F-6C-1, + 5-rest observation (this doc's source ledger). +- `180:~/campaign6/ledger.md` + `evidence/` — 6B carry-over. +- sha256 manifests (in-run, `/tmp/*.man` on the guest, transient): 4P-1 source (11 files, 40587 B), + 4P-2 secondary byte-compare (diff empty), 4P-5 pre/post/round-trip (all identical). +- Controller logs (demo 9201, 2026-07-14 17:3x–17:4x): tier-2 exit-23 (F-6C-1), tier-2 appdata mirror, + file-restore "additive-only … dst=appdata/paperless … 1 file restored", migration complete lines. + +**This pass closes the v0.131.0 REPORT's "NOT live-validated (deferred to Viktor's supervised +session)" list** — F-S2 tier-2 backup + marker restore + storage-page size, and F-S3 scope="app" +migration — all now PASS with byte-identical evidence. See §Checklist 4P-1..4P-5 and the manifests +above.