admin
deacee1164
Unattended pass via the controller API (Campaign-4 credential, real dashboard endpoints). Closes the v0.131.0 REPORT's "NOT live-validated" list: - 4P-2 F-S2 tier-2 appdata mirror byte-identical - 4P-3 F-S2 marker restore (resolved dir, additive-only) - 4P-4 storage page non-empty size - 4P-5 F-S3 scope=app migration round-trip, no ghost dir, no WARN New finding F-6C-1 (MEDIUM): tier-2 to an NFS target fails (rsync -a chown). PHASE 3 (browser) + 4-deep + Accept #1 deferred to a supervised/deep-budget 6D pass. No code shipped; paperless torn down; no credential/R/blob committed.
132 lines
8.0 KiB
Markdown
132 lines
8.0 KiB
Markdown
# CAMPAIGN-6C — supervised planes + deep tiers + v0.130–0.132 fix-verifies (2026-07-14)
|
||
|
||
**Seed:** `6C-20260714` · **Ledger/evidence:** `180:~/campaign6/` (6C-P0.md + 6B ledger.md/evidence)
|
||
**Continuation of:** CAMPAIGN-6B. **Class:** acceptance + findings.
|
||
**Boxes:** controller **v0.132.0** both guests · agent **0.88.0** · hub **0.54.0**.
|
||
|
||
This pass ran **unattended** (operator-approved) via the controller's own API — authenticated with the
|
||
Campaign-4 credential (out-of-band, `180:~/.config/credentials`), driving the **exact endpoints the
|
||
dashboard invokes** (`docker exec felhom-controller curl 127.0.0.1:8080` + session cookie + CSRF).
|
||
This is the live-test-real-flow acceptable proxy where a rendered browser wasn't used. No credential,
|
||
R, or blob is written here. Cookie/CSRF were transient (container `/tmp`, wiped at teardown).
|
||
|
||
---
|
||
|
||
## Checklist
|
||
|
||
| Bucket | Leg | Result |
|
||
|--------|-----|--------|
|
||
| P0 | baselines (ctrl 0.132.0 / agent 0.88.0 / hub 0.54.0), `backup block rejected`=0, storage headroom | **PASS** |
|
||
| P0 | API access mechanism (session-cookie auth, no loopback bypass) | **CHARACTERIZED** |
|
||
| **4P-1** | deploy paperless-ngx (felhom-usb), ingest 3 varied docs via consume flow | **PASS** |
|
||
| **4P-2** | F-S2 tier-2: appdata leg mirrors `appdata/paperless` byte-identical | **PASS** (+ **F-6C-1**) |
|
||
| **4P-3** | F-S2 marker restore: resolved live dir, additive-only, byte-identical | **PASS** |
|
||
| **4P-4** | storage page shows paperless-ngx non-empty size (39.6 KB) | **PASS** |
|
||
| **4P-5** | F-S3 scope="app" migration usb→flash→usb, full manifest, no ghost dir, no WARN | **PASS** |
|
||
| 5-rest | dead-app alert | **INCONCLUSIVE** (wrong trigger — clean stop ≠ crash) |
|
||
| Accept #1 | C6B-F1 `.fab` ≥1 GiB full-circle byte-compare | **DEFERRED** (needs dedicated setup) |
|
||
| PHASE 3 | escrow wizard / session-CSRF UX / hub 8-tab ring (browser) | **NOT-RUN** (needs supervised browser) |
|
||
| PHASE 4-deep | F7 live cut, restic self-heal, offsite restore-verify, snapshot coherence | **NOT-RUN** (deep-setup budget) |
|
||
|
||
---
|
||
|
||
## Verdict
|
||
|
||
**GREEN for the v0.131.0 acceptance.** All five PHASE 4-paperless legs PASS — this **fully closes the
|
||
"NOT live-validated" list in the v0.131.0 REPORT** (F-S2 tier-2 backup, F-S2 marker restore, F-S2
|
||
storage-page size, F-S3 scope="app" migration round-trip), on the live demo through the real dashboard
|
||
endpoints, with byte-identical manifests at every step and **no `appdata/paperless-ngx` ghost dir and
|
||
no `declared appdata dir absent` WARN**. One new MEDIUM finding (**F-6C-1**, tier-2 to an NFS target).
|
||
The browser planes (PHASE 3) and deep tiers (PHASE 4-deep) + the `.fab` ≥1 GiB circle remain for a
|
||
supervised/deep-budget pass (6D) — the credential-gated automation ran, but those legs need a rendered
|
||
browser (operator-typed password + on-screen typed-back) or large-dataset/timed-cut setup this
|
||
unattended pass did not provision.
|
||
|
||
---
|
||
|
||
## Findings (ranked)
|
||
|
||
### F-6C-1 (MEDIUM) — tier-2 auto-target = NFS share fails at the recovery-unit leg (`rsync -a` chown)
|
||
`RunAllTier2` auto-picked **nas-media (NFS)** as paperless-ngx's off-drive target. `rsyncMirror`
|
||
(`internal/backup/tier2.go`) uses `rsync -a`, whose `-o -g` preserve owner/group → `chown` on the NFS
|
||
export fails ("Operation not permitted", root_squash) → **exit 23**, the recovery-unit leg fails first,
|
||
so `RunTier2` records failure and the **appdata leg never runs**. Same class hit calibre-web
|
||
(`teszt_enroll` mkdir permission-denied — a separate bad-mount residue).
|
||
- **Repro:** any box whose first schedulable off-disk tier-2 target is an NFS share.
|
||
- **Impact:** tier-2 silently produces no off-drive copy for the affected app (status = failure, so
|
||
it's honest, not silent — but no protection). Real drives (felhom-usb/flash) are unaffected.
|
||
- **Not an F-S2 regression** — F-S2 (the appdata dir-name resolution) is proven correct once the
|
||
target is a real drive (4P-2 byte-compare).
|
||
- **Suggested fix (→6D / a controller task):** for a network-fs target, drop `-o/-g` (add
|
||
`--no-owner --no-group`, or `--chown`), or exclude NFS mounts from the auto tier-2 target pick.
|
||
- **Evidence:** controller log 2026-07-14 17:35:31 (exit 23 chown lines); `180:~/campaign6/6C-P0.md`.
|
||
|
||
### Observation (LOW, → 6D) — aggregate state "running" while main container Exited(0)
|
||
A clean `docker stop paperless-webserver` (exit 0) did NOT fire a dead-app alert in 2.5 min and the
|
||
controller reported `state=running` / health-probes "4 ok". This is **likely by design** (a clean stop
|
||
is not a crash; `downstate_test` asserts certain states must not manufacture an alert), so 5-rest is
|
||
**inconclusive with this trigger**. Worth a 6D note: the aggregate state reads "running" while the
|
||
service container is down (other containers up). A proper crash trigger (`docker kill -9` / OOM,
|
||
exit≠0) is the correct test — deferred (restart-policy race needs care).
|
||
|
||
---
|
||
|
||
## Timings (paperless acceptance, demo)
|
||
|
||
| Step | Wall time |
|
||
|------|-----------|
|
||
| deploy → 3 containers healthy | ~95 s (Postgres start_period) |
|
||
| 3-doc consume ingest (OCR + PDF/A) | ~90 s |
|
||
| tier-2 run (all HDD apps) | ~30 s |
|
||
| tier-2 file-restore (stop→copy→start→health) | 39 s |
|
||
| migration usb→flash (scope=app) | 45 s |
|
||
| migration flash→usb (round-trip) | 45 s |
|
||
|
||
---
|
||
|
||
## Deviations from the prompt
|
||
|
||
- **PHASE 3 / 4-deep / Accept #1 not run.** PHASE 3 needs a rendered browser with an operator present
|
||
(browser-select prompt, password typed at the login screen, and the wizard's "type the two
|
||
highlighted words read from screen") — genuinely interactive, not unattended-automatable. 4-deep
|
||
(F7 live 6 s cut, restic mid-run kill, offsite restore-to-verify) and Accept #1 (≥1 GiB userdata app
|
||
+ encrypted `.fab` round-trip) need large-dataset / timed-cut setup budget this pass did not
|
||
provision. The C6B-F1 `.fab` fix is already code + live byte-compare verified in the v0.130.0 REPORT;
|
||
this pass instead prioritized the **newer, higher-risk v0.131.0 F-S2/F-S3** legs (never live-proven
|
||
until now).
|
||
- **P0 residue:** 6B's cleanup did NOT fully remove the campaign6 mount or the sonarr stackdir (both
|
||
still present) — minor, non-blocking (deviates from the P0 note's "may be GONE").
|
||
- **paperless tier-2 target pinned** to felhom-flash (via the `/stacks/paperless-ngx/backup` UI
|
||
control) to route around F-6C-1 for the F-S2 proof — a legitimate UI action, recorded.
|
||
- **Stale target residue** (`felhom-flash/appdata/paperless`, 16 KB, Jun-15 prior-session) cleared
|
||
before 4P-5 so the migration collision guard (working correctly) didn't block the test.
|
||
|
||
---
|
||
|
||
## Box state / morning recovery
|
||
|
||
- **paperless-ngx torn down** (real stop→remove flow + scratch-residue cleanup): containers gone,
|
||
`app.yaml` gone, `appdata/paperless` + primary unit + tier-2 secondaries (flash + nas-media) removed.
|
||
Pre-existing skeleton (`userdata/paperless-ngx` canary, empty `import/paperless` dirs) left as-is.
|
||
- Controller **v0.132.0 healthy** on demo 9201 post-teardown; other 6 apps untouched and running.
|
||
- Session cookie/CSRF wiped from the controller container `/tmp`.
|
||
- **No code shipped this pass** (findings-only). **Viktor rotates the controller + hub credentials
|
||
after the campaign** (per the campaign rule).
|
||
|
||
---
|
||
|
||
## Evidence index
|
||
|
||
- `180:~/campaign6/6C-P0.md` — P0 state, auth note, full 4-paperless leg-by-leg results, F-6C-1,
|
||
5-rest observation (this doc's source ledger).
|
||
- `180:~/campaign6/ledger.md` + `evidence/` — 6B carry-over.
|
||
- sha256 manifests (in-run, `/tmp/*.man` on the guest, transient): 4P-1 source (11 files, 40587 B),
|
||
4P-2 secondary byte-compare (diff empty), 4P-5 pre/post/round-trip (all identical).
|
||
- Controller logs (demo 9201, 2026-07-14 17:3x–17:4x): tier-2 exit-23 (F-6C-1), tier-2 appdata mirror,
|
||
file-restore "additive-only … dst=appdata/paperless … 1 file restored", migration complete lines.
|
||
|
||
**This pass closes the v0.131.0 REPORT's "NOT live-validated (deferred to Viktor's supervised
|
||
session)" list** — F-S2 tier-2 backup + marker restore + storage-page size, and F-S3 scope="app"
|
||
migration — all now PASS with byte-identical evidence. See §Checklist 4P-1..4P-5 and the manifests
|
||
above.
|