felhom.eu/documentation/audits/CAMPAIGN-6E-2026-07-15.md

CAMPAIGN-6E — P-TIER2 deep-4 + P4-DEEP timing (pre-broad-launch resilience) (2026-07-15)

Operator: Viktor (present). Driver: Claude Code (live box). Validator: project Claude. Addendum to CAMPAIGN-6D — closes the two deferred deep campaigns before broad (Facebook) launch. NOT a friend-alpha gate (that is already GREEN in 6D). Operational, supervised; no new production code shipped. A surfaced issue is RECORDED + ranked, never fixed inline. Every mechanism under test is already SHIPPED + unit-tested; these legs confirm the shipped behavior survives a real adverse event on live infra.

Baselines (E0): controller 0.137.0 (Up healthy), agent 0.88.0, hub 0.56.0. Evidence sink: 180:~/campaign6/6E/evidence/. Subject box: demo guest 9201 (demo-felhom). (Sibling doc, not an append to 6D — 6E is a distinct addendum campaign.)

---

Headline verdict

Both deep campaigns GREEN. The tier-2 v2 layout engine (Task 3b, v0.135.0) and the F7 atomic-write

• restic single-writer self-heal (CAMPAIGN-3 / C2) all survive real adverse events on live infra.

• Phase A (P-TIER2 deep-4): 4/4 PASS. class-driven legs, the lifted N>1 refusal, deferred-prune reconcile, and old-flat→v2 migration all behave exactly as designed; no removal ever left backups/secondary/; markers written last.
• Phase B (P4-DEEP timing): 4/4 PASS. A mid-write SIGKILL never promotes a partial dump over a good restore point (volume tar AND DB dump); the restic C2 crash-lock self-heal fires correctly against a real SFTP repo with a genuine foreign-hostname exclusive lock; an offbox restore round-trips byte-identically.
• No HIGH/MED findings. Two LOW observations (below), one with a candidate follow-up TASK.

---

Pre-flight (E0–E5)

Gate	Result
E0 baselines	ctrl 0.137.0 / agent 0.88.0 / hub 0.56.0 — all match.
E1 guest health	9201 controller 0.137.0 Up (healthy).
E2 credential	C4_PASSWORD present in 180:~/.config/credentials (single-quoted → stripped on use).
E3 drill-repo isolation	drill restic repo sftp:…:/home/6e-timing-1784142113 — a dedicated path, never the real /home/felhom-repo. Torn down (proof below).
E4 evidence sink + headroom	180:~/campaign6/6E/evidence/ created; 153 G free on 180.
E5 ≥2 local drives	three non-network drives registered: felhom-usb (default), teszt_enroll, felhom-flash (+ nas-media, network — correctly never a tier-2 target).

---

Phase A — P-TIER2 deep-4 (writes ONLY under backups/secondary/)

Driver: the real per-app endpoint POST /stacks/{name}/backup (enabled=on, target=<pin>, CSRF) → RunTier2(name), target pinned for determinism. Subject: immich (the one HDD app on the demo with a drive registration consistent with where its data physically lives — see Finding F-6E-1). Source felhom-usb → target teszt_enroll. Classification edits were made to the deployed /opt/docker/stacks/immich/.felhom.yml / docker-compose.yml (the exact files ClassifiedBinds reads) and reverted byte-identically; all RECORDED via /root/*.orig backups.

Leg	Result	Evidence
A1 no-block legacy app	PASS	immich un-classified → legacy resolver path → single hdd/appdata/immich leg, byte-identical to source (286/286 files), .felhom-tier2-layout=2 present, log 1 leg(s), no errTier2MultiDir. The now-unclassified userdata/media leg was correctly pruned by reconcile. Confirms the legacy→v2 byte-identity promise (SQ5 footprint).
A2 N>1 lifted refusal	PASS	Added a 2nd appdata bind (appdata/immich-6e2, classified mandatory) → run yields two hdd/appdata/* legs (immich + immich-6e2) coexisting, log 3 leg(s) (both appdata + userdata/media/photos), no errTier2MultiDir — the v0.131.0 flat-appdata refusal is structurally gone. 2nd-appdata leg byte-identical.
A3 re-class reconcile	PASS	Re-classed immich-6e2 → excluded → run logs [INFO] Tier 2 reconcile: removed stale dest dir hdd/appdata/immich-6e2; hdd/appdata/immich + userdata/media/photos kept; marker rewritten. Removal strictly within backups/secondary/ (tier2SafeRemove guard).
A4 old-flat → v2 migration	PASS	Synthesized a genuine pre-v2 flat copy at the target (appdata/ flat + recovery-unit/, no marker) → run: flat appdata/ removed, v2 legs (hdd/appdata/immich + userdata/media/photos) rebuilt, recovery-unit/ content-identical (untouched by migration), .felhom-tier2-layout=2 written last.

Phase A boundary check: every reconcile/migration removal stayed within backups/secondary/<stack>/ (log paths + tier2SafeRemove); the surrounding tree (recovery-unit, sibling legs) intact each time. Markers written last in every case.

Log evidence (6E-phaseA-logs.txt):

Tier 2 copied immich → …/teszt_enroll/backups/secondary/immich (466.4 MB, 3 leg(s), 0s)   [A2]
Tier 2 reconcile: removed stale dest dir hdd/appdata/immich-6e2                            [A3]
Tier 2 copied immich → … (466.4 MB, 2 leg(s), 0s)                                          [A3]
Tier 2 reconcile: removed stale dest dir userdata/media                                    [A1 legacy prune]
Tier 2 copied immich → … (466.4 MB, 1 leg(s), 0s)                                          [A1]
Tier 2 copied immich → … (466.4 MB, 2 leg(s), 1s)                                          [A4 migration]

---

Phase B — P4-DEEP timing (process-kill only; no power-cut)

B1 — F7 atomic tar-write (volume path) — PASS

Controlled real docker volume (2 GB) + the identical controller tar seam (docker run … alpine tar cf …/<vol>.tar.tmp). tar SIGKILLed mid-write (exit 137) → left a partial .tar.tmp (1.21 GB < 2.0 GB full); the good .tar was byte-unchanged (sha256 match). The orphan .tar.tmp is invisible to restore-point scans (matches *.tar only). Clean re-run + the atomicPromoteTar sequence (fsync + atomic rename + dir-fsync) → a new valid, extractable .tar. → A partial .tar.tmp never replaces a good restore point.

B2 — F7 atomic write (DB-dump path) + the asymmetry check — PASS

Real pg_dump of the immich DB (read-only) into <name>.sql.tmp, SIGKILLed mid-write (exit 137) → partial .sql.tmp (41 MB < 45.8 MB); the good .sql was byte-unchanged. DumpOne (appbackup/dbdump.go) removes the tmp on error and only renames on success. Asymmetry observation (confirmed from code): DumpOne fsyncs the file (tmpFile.Sync(), dbdump.go:277) but not the directory entry after os.Rename — whereas atomicPromoteTar (backup.go:599) additionally dir.Sync()s. Under a process-kill (this test) both are safe: the rename is visible in the running kernel regardless of dir-fsync. The gap matters only for power-loss durability parity. → candidate follow-up TASK T-6E-1 (LOW), below.

B3 — restic single-writer self-heal (drill repo only) — PASS

On the isolated drill repo (E3), reproduced the exact C2 condition: a foreign-hostname exclusive lock created by a helper container (--hostname felhom-old-ctnr) whose restic prune was SIGKILLed mid-run. From the controller (host demo-felhom):

1. plain restic unlock (stale-only) reports "successfully removed locks" but the foreign lock remains — restic cannot prove felhom-old-ctnr's PID dead, so it is not treated as stale. This is exactly why the pre-run unlockStale is insufficient for a recreated-container crash lock.
2. restic backup → unable to create lock in backend: repository is already locked exclusively by PID 1 on felhom-old-ctnr by root — the error offboxLockRe matches.
3. restic unlock --remove-all → locks empty [].
4. retry restic backup once → snapshot 33568faf saved.

This is precisely resticStep's self-heal path (offbox.go:349): op → repository is already locked → unlock --remove-all → retry once. The "never loops" bound (a 2nd lock failure surfaces the original error, offbox.go:360/363) is a code guarantee, not force-tested live. Evidence: 6E-B3-exclusive.txt. Real repo /home/felhom-repo never touched (all writes to /home/6e-timing-…).

B4 — restore-to-verify-compare — PASS

Read-only restic restore of immich's mandatory appdata/immich subtree (snapshot d606269e) from the real repo to a local scratch (/tmp/6e-b4 on the container rootfs — not network, per Part 0.5). sha256 file-by-file compare vs the live source: 286/286 files byte-identical. Restore is read-only, so this ran safely against the real repo. → "a tier is only real once a restore round-trip is proven", as a pure integrity compare.

---

Findings (ranked)

ID	Sev	Finding	Disposition
F-6E-1	LOW	Demo-data drive drift: nextcloud (and likely romm) have no app.yaml/HDD_PATH, so GetAppDrivePath defaults to /mnt/sys_drive while their real appdata lives on felhom-usb. Tier-2 therefore WARNs mandatory data path missing on disk and produces no copy. The code is correct (it warns loudly and does not silently succeed). This is demo-data churn from app-drive migration, not a product defect — a real deployment sets HDD_PATH.	No code TASK. Demo-hygiene cleanup only (re-point or redeploy nextcloud/romm on the demo if a working tier-2 copy is wanted there). Optional product idea: surface "tier-2 has never produced a copy (source missing)" more prominently in the operator UI — not filed.
T-6E-1	LOW	DB-dump dir-fsync asymmetry (B2): DumpOne fsyncs the file but not the directory after os.Rename; atomicPromoteTar fsyncs both. Safe under process-kill; only power-loss durability parity would benefit.	Candidate follow-up TASK: add a best-effort dir.Sync() after the rename in DumpOne (mirror atomicPromoteTar). LOW — not required for the shipped safety property; do NOT ship inside this campaign.

No HIGH or MED findings.

---

Teardown & box health

• Drill repo torn down (proof): ssh …storagebox rm -rf /home/6e-timing-1784142113 then ls /home → felhom-repo only. Real repo intact.
• All temporary fixtures reverted byte-identically: immich docker-compose.yml + .felhom.yml == /root/*.orig (verified diff -q); scratch source appdata/immich-6e2 removed; nextcloud classification restored. immich DB never modified (a scratch-table attempt failed on a role error and created nothing).
• Tier-2 preferences: immich + nextcloud were left at default participation (enabled, no pin, user_disabled=None). immich now carries a healthy auto v2 copy on teszt_enroll; nextcloud reverts to its pre-existing F-6E-1 state. teszt_enroll/secondary otherwise returned to pre-state.
• Scratch/secrets removed: B1/B2 scratch volume + dumps, B3/B4 scratch, and the session cookie-jar/CSRF/password files all deleted; no felhom-6e* volumes or containers remain.
• Box healthy: controller 0.137.0 Up (healthy); immich-server Up (healthy).

Evidence index — 180:~/campaign6/6E/evidence/

6E-prestate-secondary.txt, 6E-A2-evidence.txt, 6E-phaseA-logs.txt, 6E-phaseB-B1B2.txt, 6E-B3-out.txt, 6E-B3-exclusive.txt, 6E-B4-result.txt.

Queue delta

• Closed by 6E: P-TIER2 deep-4 (A1–A4) and P4-DEEP timing (B1–B4) — the two deferred deep campaigns from 6D. The v2 layout engine and the F7 / C2 self-heal mechanisms are live-confirmed.
• Remains (broad-launch, non-blocking): T-6E-1 (LOW, optional power-loss durability parity for the DB-dump path); demo-hygiene for F-6E-1 (nextcloud/romm drive drift). Neither blocks broad launch.