# CAMPAIGN-6E — P-TIER2 deep-4 + P4-DEEP timing (pre-broad-launch resilience) (2026-07-15)

**Operator:** Viktor (present). **Driver:** Claude Code (live box). **Validator:** project Claude.
Addendum to [CAMPAIGN-6D](CAMPAIGN-6D-2026-07-15.md) — closes the two deferred deep campaigns before
broad (Facebook) launch. NOT a friend-alpha gate (that is already GREEN in 6D). Operational,
supervised; **no new production code shipped**. A surfaced issue is RECORDED + ranked, never fixed
inline. Every mechanism under test is already SHIPPED + unit-tested; these legs confirm the shipped
behavior survives a real adverse event on live infra.

Baselines (E0): controller **0.137.0** (Up healthy), agent **0.88.0**, hub **0.56.0**.
Evidence sink: `180:~/campaign6/6E/evidence/`. Subject box: demo guest 9201 (`demo-felhom`).
(Sibling doc, not an append to 6D — 6E is a distinct addendum campaign.)

---

## Headline verdict

**Both deep campaigns GREEN.** The tier-2 v2 layout engine (Task 3b, v0.135.0) and the F7 atomic-write
+ restic single-writer self-heal (CAMPAIGN-3 / C2) all survive real adverse events on live infra.

- **Phase A (P-TIER2 deep-4): 4/4 PASS.** class-driven legs, the lifted N>1 refusal, deferred-prune
  reconcile, and old-flat→v2 migration all behave exactly as designed; no removal ever left
  `backups/secondary/`; markers written last.
- **Phase B (P4-DEEP timing): 4/4 PASS.** A mid-write SIGKILL never promotes a partial dump over a good
  restore point (volume tar AND DB dump); the restic C2 crash-lock self-heal fires correctly against a
  **real SFTP repo** with a genuine foreign-hostname exclusive lock; an offbox restore round-trips
  byte-identically.
- **No HIGH/MED findings.** Two LOW observations (below), one with a candidate follow-up TASK.

---

## Pre-flight (E0–E5)

| Gate | Result |
|------|--------|
| E0 baselines | ctrl 0.137.0 / agent 0.88.0 / hub 0.56.0 — all match. |
| E1 guest health | 9201 controller 0.137.0 Up (healthy). |
| E2 credential | `C4_PASSWORD` present in `180:~/.config/credentials` (single-quoted → stripped on use). |
| E3 drill-repo isolation | drill restic repo `sftp:…:/home/6e-timing-1784142113` — a dedicated path, never the real `/home/felhom-repo`. Torn down (proof below). |
| E4 evidence sink + headroom | `180:~/campaign6/6E/evidence/` created; 153 G free on 180. |
| E5 ≥2 local drives | three non-network drives registered: **felhom-usb** (default), **teszt_enroll**, **felhom-flash** (+ nas-media, network — correctly never a tier-2 target). |

---

## Phase A — P-TIER2 deep-4 (writes ONLY under `backups/secondary/`)

Driver: the real per-app endpoint `POST /stacks/{name}/backup` (`enabled=on`, `target=<pin>`, CSRF) →
`RunTier2(name)`, target pinned for determinism. Subject: **immich** (the one HDD app on the demo with a
drive registration consistent with where its data physically lives — see Finding F-6E-1). Source
felhom-usb → target teszt_enroll. Classification edits were made to the **deployed**
`/opt/docker/stacks/immich/.felhom.yml` / `docker-compose.yml` (the exact files `ClassifiedBinds` reads)
and reverted byte-identically; all RECORDED via `/root/*.orig` backups.

| Leg | Result | Evidence |
|-----|--------|----------|
| **A1** no-block legacy app | **PASS** | immich un-classified → legacy resolver path → single `hdd/appdata/immich` leg, **byte-identical to source (286/286 files)**, `.felhom-tier2-layout=2` present, log `1 leg(s)`, no `errTier2MultiDir`. The now-unclassified `userdata/media` leg was correctly pruned by reconcile. Confirms the legacy→v2 byte-identity promise (SQ5 footprint). |
| **A2** N>1 lifted refusal | **PASS** | Added a 2nd appdata bind (`appdata/immich-6e2`, classified mandatory) → run yields **two `hdd/appdata/*` legs** (`immich` + `immich-6e2`) coexisting, log `3 leg(s)` (both appdata + `userdata/media/photos`), **no `errTier2MultiDir`** — the v0.131.0 flat-appdata refusal is structurally gone. 2nd-appdata leg byte-identical. |
| **A3** re-class reconcile | **PASS** | Re-classed `immich-6e2` → `excluded` → run logs `[INFO] Tier 2 reconcile: removed stale dest dir hdd/appdata/immich-6e2`; `hdd/appdata/immich` + `userdata/media/photos` kept; marker rewritten. Removal strictly within `backups/secondary/` (`tier2SafeRemove` guard). |
| **A4** old-flat → v2 migration | **PASS** | Synthesized a genuine pre-v2 flat copy at the target (`appdata/` flat + `recovery-unit/`, **no marker**) → run: flat `appdata/` **removed**, v2 legs (`hdd/appdata/immich` + `userdata/media/photos`) rebuilt, **`recovery-unit/` content-identical (untouched by migration)**, `.felhom-tier2-layout=2` written **last**. |

**Phase A boundary check:** every reconcile/migration removal stayed within `backups/secondary/<stack>/`
(log paths + `tier2SafeRemove`); the surrounding tree (recovery-unit, sibling legs) intact each time.
Markers written last in every case.

Log evidence (`6E-phaseA-logs.txt`):
```
Tier 2 copied immich → …/teszt_enroll/backups/secondary/immich (466.4 MB, 3 leg(s), 0s)   [A2]
Tier 2 reconcile: removed stale dest dir hdd/appdata/immich-6e2                            [A3]
Tier 2 copied immich → … (466.4 MB, 2 leg(s), 0s)                                          [A3]
Tier 2 reconcile: removed stale dest dir userdata/media                                    [A1 legacy prune]
Tier 2 copied immich → … (466.4 MB, 1 leg(s), 0s)                                          [A1]
Tier 2 copied immich → … (466.4 MB, 2 leg(s), 1s)                                          [A4 migration]
```

---

## Phase B — P4-DEEP timing (process-kill only; no power-cut)

### B1 — F7 atomic tar-write (volume path) — **PASS**
Controlled real docker volume (2 GB) + the identical controller tar seam
(`docker run … alpine tar cf …/<vol>.tar.tmp`). tar **SIGKILLed mid-write** (exit 137) → left a
**partial** `.tar.tmp` (1.21 GB < 2.0 GB full); the good `.tar` was **byte-unchanged** (sha256 match).
The orphan `.tar.tmp` is invisible to restore-point scans (matches `*.tar` only). Clean re-run + the
`atomicPromoteTar` sequence (fsync + atomic rename + dir-fsync) → a new **valid, extractable** `.tar`.
→ A partial `.tar.tmp` never replaces a good restore point.

### B2 — F7 atomic write (DB-dump path) + the asymmetry check — **PASS**
Real `pg_dump` of the immich DB (read-only) into `<name>.sql.tmp`, **SIGKILLed mid-write** (exit 137) →
partial `.sql.tmp` (41 MB < 45.8 MB); the good `.sql` was **byte-unchanged**. `DumpOne`
(`appbackup/dbdump.go`) removes the tmp on error and only renames on success.
**Asymmetry observation (confirmed from code):** `DumpOne` fsyncs the **file** (`tmpFile.Sync()`,
dbdump.go:277) but **not the directory entry** after `os.Rename` — whereas `atomicPromoteTar`
(backup.go:599) additionally `dir.Sync()`s. Under a **process-kill** (this test) both are safe: the
rename is visible in the running kernel regardless of dir-fsync. The gap matters only for **power-loss**
durability parity. → candidate follow-up TASK **T-6E-1 (LOW)**, below.

### B3 — restic single-writer self-heal (drill repo only) — **PASS**
On the isolated drill repo (E3), reproduced the exact C2 condition: a **foreign-hostname exclusive
lock** created by a helper container (`--hostname felhom-old-ctnr`) whose `restic prune` was SIGKILLed
mid-run. From the controller (host `demo-felhom`):

1. plain `restic unlock` (stale-only) reports "successfully removed locks" but the foreign lock
   **remains** — restic cannot prove `felhom-old-ctnr`'s PID dead, so it is not treated as stale. This is
   exactly why the pre-run `unlockStale` is insufficient for a recreated-container crash lock.
2. `restic backup` → `unable to create lock in backend: repository is already locked exclusively by PID
   1 on felhom-old-ctnr by root` — the error `offboxLockRe` matches.
3. `restic unlock --remove-all` → locks empty `[]`.
4. retry `restic backup` **once** → `snapshot 33568faf saved`.

This is precisely `resticStep`'s self-heal path (offbox.go:349): op → `repository is already locked` →
`unlock --remove-all` → retry once. The "never loops" bound (a 2nd lock failure surfaces the original
error, offbox.go:360/363) is a code guarantee, not force-tested live. Evidence: `6E-B3-exclusive.txt`.
Real repo `/home/felhom-repo` never touched (all writes to `/home/6e-timing-…`).

### B4 — restore-to-verify-compare — **PASS**
Read-only `restic restore` of immich's mandatory `appdata/immich` subtree (snapshot `d606269e`) from the
**real** repo to a **local** scratch (`/tmp/6e-b4` on the container rootfs — not network, per Part 0.5).
sha256 file-by-file compare vs the live source: **286/286 files byte-identical**. Restore is read-only,
so this ran safely against the real repo. → "a tier is only real once a restore round-trip is proven",
as a pure integrity compare.

---

## Findings (ranked)

| ID | Sev | Finding | Disposition |
|----|-----|---------|-------------|
| **F-6E-1** | LOW | **Demo-data drive drift:** `nextcloud` (and likely `romm`) have no `app.yaml`/`HDD_PATH`, so `GetAppDrivePath` defaults to `/mnt/sys_drive` while their real appdata lives on `felhom-usb`. Tier-2 therefore WARNs `mandatory data path missing on disk` and produces no copy. **The code is correct** (it warns loudly and does not silently succeed). This is demo-data churn from app-drive migration, not a product defect — a real deployment sets `HDD_PATH`. | No code TASK. Demo-hygiene cleanup only (re-point or redeploy nextcloud/romm on the demo if a working tier-2 copy is wanted there). Optional product idea: surface "tier-2 has never produced a copy (source missing)" more prominently in the operator UI — not filed. |
| **T-6E-1** | LOW | **DB-dump dir-fsync asymmetry** (B2): `DumpOne` fsyncs the file but not the directory after `os.Rename`; `atomicPromoteTar` fsyncs both. Safe under process-kill; only power-loss durability parity would benefit. | Candidate follow-up TASK: add a best-effort `dir.Sync()` after the rename in `DumpOne` (mirror `atomicPromoteTar`). LOW — not required for the shipped safety property; do NOT ship inside this campaign. |

No HIGH or MED findings.

---

## Teardown & box health

- **Drill repo torn down (proof):** `ssh …storagebox rm -rf /home/6e-timing-1784142113` then `ls /home`
  → `felhom-repo` only. Real repo intact.
- **All temporary fixtures reverted byte-identically:** immich `docker-compose.yml` + `.felhom.yml` ==
  `/root/*.orig` (verified `diff -q`); scratch source `appdata/immich-6e2` removed; nextcloud
  classification restored. immich DB never modified (a scratch-table attempt failed on a role error and
  created nothing).
- **Tier-2 preferences:** immich + nextcloud were left at default participation (enabled, no pin,
  `user_disabled=None`). immich now carries a healthy auto v2 copy on teszt_enroll; nextcloud reverts to
  its pre-existing F-6E-1 state. teszt_enroll/secondary otherwise returned to pre-state.
- **Scratch/secrets removed:** B1/B2 scratch volume + dumps, B3/B4 scratch, and the session
  cookie-jar/CSRF/password files all deleted; no `felhom-6e*` volumes or containers remain.
- **Box healthy:** controller 0.137.0 Up (healthy); immich-server Up (healthy).

## Evidence index — `180:~/campaign6/6E/evidence/`
`6E-prestate-secondary.txt`, `6E-A2-evidence.txt`, `6E-phaseA-logs.txt`, `6E-phaseB-B1B2.txt`,
`6E-B3-out.txt`, `6E-B3-exclusive.txt`, `6E-B4-result.txt`.

## Queue delta
- **Closed by 6E:** P-TIER2 deep-4 (A1–A4) and P4-DEEP timing (B1–B4) — the two deferred deep campaigns
  from 6D. The v2 layout engine and the F7 / C2 self-heal mechanisms are live-confirmed.
- **Remains (broad-launch, non-blocking):** T-6E-1 (LOW, optional power-loss durability parity for the
  DB-dump path); demo-hygiene for F-6E-1 (nextcloud/romm drive drift). Neither blocks broad launch.