From bed9bd2c5a1be7ccef9f7933c793b0e68b07b4d1 Mon Sep 17 00:00:00 2001
From: kisfenyo <nagyfenyvesi.viktor@gmail.com>
Date: Sun, 12 Jul 2026 04:29:48 +0200
Subject: [PATCH] =?UTF-8?q?CAMPAIGN-3=20audit:=20unattended=20night=20run?=
 =?UTF-8?q?=20=E2=80=94=20data=20plane=20solid;=20NAS=20automount=20re-arm?=
 =?UTF-8?q?=20plane=20broken=20(F10=20CRITICAL,=20F9/F11/F7=20HIGH);=20dem?=
 =?UTF-8?q?o=20host=20DOWN=20after=204e=20reboot=20(HUMAN)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

30 PASS / 17 FAIL / 11 FINDING / 1 DISCREPANCY. Fix-6 ring numbers (6.5min horizon
under load, wiped per restart), backup-vs-NAS policy discovery, morning recovery
runbook. Evidence: 180:~/campaign3/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
---
 CONTEXT.md                                    |  10 ++
 REPORT.md                                     | 105 ++----------------
 documentation/audits/CAMPAIGN-3-2026-07-11.md |  96 ++++++++++++++++
 3 files changed, 117 insertions(+), 94 deletions(-)
 create mode 100644 documentation/audits/CAMPAIGN-3-2026-07-11.md

diff --git a/CONTEXT.md b/CONTEXT.md
index 5038811..cc9e3e9 100644
--- a/CONTEXT.md
+++ b/CONTEXT.md
@@ -3,6 +3,16 @@
 > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
 > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
 
+- **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; ⚠ DEMO HOST DOWN (HUMAN).** Unattended chaos campaign
+  22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/
+  restore byte-identical/EIO instant/stub badge + deploy-409 live-validated), but **felhom-pve never
+  returned from the planned host reboot** (physical power/console needed; recovery runbook in the
+  audit) and the **NAS automount re-arm plane failed**: F10 CRITICAL (start-limit never re-armed,
+  once blocked guest start), F11/F9 HIGH (post-start reassert "skip-active" skips the healing
+  automount restart; agent-restart sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots;
+  F7 HIGH (in-place dump write → 0-byte tar replaced last good backup during outage window).
+  Fix-6 numbers: ring horizon 55min idle / **6.5min under load**, wiped on every restart.
+  Full ledger/evidence: `documentation/audits/CAMPAIGN-3-2026-07-11.md` + `180:~/campaign3/`.
 - **2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED** (CHANGELOG v0.47.0; commits `9f29bf3` →
   `0daddcd` + docs). Five deliverables: **(1)** `.data-table td a:not(.btn)` button-contrast fix;
   **(2)** customer page = **8 hash tabs** (`#tab=…`, sticky summary strip, Events error badge,
diff --git a/REPORT.md b/REPORT.md
index 3172ae4..f0068b4 100644
--- a/REPORT.md
+++ b/REPORT.md
@@ -2,102 +2,19 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## Hub v0.47.0 — UI reorganization (customer tabs, Host tab, stale-host removal, offsite multi-endpoint UI, button contrast) — 2026-07-11
+## CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability) — 2026-07-11/12
 
-### 1. Baseline
+**Full report: [`documentation/audits/CAMPAIGN-3-2026-07-11.md`](documentation/audits/CAMPAIGN-3-2026-07-11.md).** Run 22:09 → 04:27 CEST against the demo box (controller 0.117.0 / agent 0.84.0), operator-unattended, real surfaces only, no code changes. Ledger: **30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY** (66+ scenario entries, evidence at `180:~/campaign3/`).
 
-felhom.eu `main` @ `8e1a3f0d82b62ac84835ddf0aa5a6fd523afb034`, hub v0.46.0 (CHANGELOG top +
-running footer + `manifests/hub.yaml` all agreed). Spec: the "Hub UI reorganization" TASK.
+### Headlines
 
-### 2. Files created / modified
+- **⚠ HUMAN, first thing:** felhom-pve **did not return** from the planned 4e host reboot (23:30) — polled every 5 min until 04:27; no ping, no SSH (22/8822), controller 530. Needs physical power/console. Morning recovery runbook is in the report (§Box state) — after boot expect F10/F11 and heal with `reset-failed` + automount restart + `docker start` of the 4 NAS apps.
+- **CRITICAL F10 + HIGH F11/F9 — the reboot/recovery plane around NAS automounts is broken:** `mount-start-limit-hit` is never re-armed by any heal path (once even blocked guest start → guest DOWN); guest reboot with an idle share leaves the autofs trigger unpropagated into the container — the post-start reassert **logs its own WARNING and then skips** the automount restart that provably heals ("skip-active" branch). Every such reboot = 4 NAS apps dead-at-boot. Reproduced on 3 of 3 guest reboots.
+- **HIGH F7 — backup dumps are written in place (no tmp+rename):** a mid-backup NAS cut left a 0-byte tar *replacing* the last good 247M dump; in that window restore = empty volume. Next run self-heals; run-level `success:false` is the only signal.
+- **The data plane held:** all 5 refusal categories ×2 correct + fast (2–5 s, retry=0), verify/rollback/single-flight/orphan flows clean, deploy-view truth holds, restore round-trips **byte-identical**, EIO same-second under outage, organic stub → badge + deploy-409 live-validated, hardlinks work on NFSv4.1.
+- **Fix-6 answered with numbers:** ring cap horizon = ~55 min idle but **~6.5 min under load**; every restart/reboot wipes both rings — persistence, not just size, is the gap.
+- **Policy discovery (docs):** tier-1 = volumes+config only (NAS media userdata excluded by design); NAS apps' tier-1 lands *on the NAS*, tier-2 is what gets it off; volume-only apps back up to sys_drive with blank drive label and **no tier-2 copy**.
 
-- `hub/internal/web/templates/style.css` — `.data-table td a:not(.btn)` contrast fix + tab/summary-strip/badge CSS
-- `hub/internal/web/templates/customer_unified.html` — 8 hash-tab panels + sticky summary strip + Events badge + Host tab (all sections preserved; hiding = JS-added body class only)
-- `hub/internal/web/templates/host_detail_body.html` — NEW: shared `{{define "host_detail_body"}}` (identity/vitals/guests/storage/diagnostics/DR + the Deletable-gated danger zone)
-- `hub/internal/web/templates/host_detail.html` — reduced to chrome + sub-template call
-- `hub/internal/web/templates/offsite.html` — endpoint cards + add/edit/delete forms + Endpoint peer column + deferral note
-- `hub/internal/web/hosts.go` — `hostDetailData` builder; `handleHostDeleteImpact` + `handleHostDelete`
-- `hub/internal/web/configs.go` — `Hosts` view models on the customer page (via `ListHostsByCustomer`)
-- `hub/internal/web/offsite.go` — multi-endpoint render + save/delete handlers + subnet/peer guards
-- `hub/internal/web/server.go` — routes: `/hosts/{id}/delete-impact|/delete` (above the catch-all), `/offsite/endpoints[...]`
-- `hub/internal/store/store.go` — `ListHostsByCustomer`, `HostArtifacts`/`CountHostArtifacts`, `DeleteHost` (one-tx cascade), `ErrHostEscrowPresent`
-- `hub/internal/store/wg.go` — `ListWGEndpoints`, `DeleteWGEndpoint`; `SetWGEndpoint` comment updated (behavior untouched)
-- Tests: `hub/internal/web/customer_tabs_test.go` (new), `hosts_delete_test.go` (new), `offsite_test.go` (amended deliberately), `hosts_test.go` (pin comment), `hub/internal/store/host_delete_test.go` (new), `wg_endpoints_test.go` (new), `host_test.go` (+`TestListHostsByCustomer`)
-- Docs: `hub/CHANGELOG.md` (v0.47.0), `hub/README.md` (tab map, host lifecycle, offsite mgmt), `CONTEXT.md`, `REUSE.md` (new §2 table), `manifests/hub.yaml` (tag 0.47.0)
+### Box state / cleanup
 
-### 3. Commits on `main`
-
-| Commit | Content |
-|---|---|
-| `9f29bf3` | Part 1+2: CSS contrast fix + customer hash tabs + tab render tests |
-| `ae950e5` | Part 3: `host_detail_body` sub-template + Host tab + `ListHostsByCustomer` |
-| `146d165` | ⚠ NOT mine — the parallel NFS session's docs commit whose `git add -A` swept my Part-4 WIP **mid-red-proof** (hosts.go landed with the RED-PROOF-2 mutation, escrow ack bypassed; everything else was in final correct state) |
-| `068427a` | Part 4 completion: restored the escrow-ack gate line (the only delta vs. the intended state) |
-| `0daddcd` | Part 5: offsite multi-endpoint UI + guards + tests |
-| `95d71fe` | Part 6: CHANGELOG/README/CONTEXT/REUSE docs + `manifests/hub.yaml` → 0.47.0 |
-| `77247f9` | this REPORT.md |
-
-### 4. Red-proofs (all five ran: mutation → FAIL with the wrong value visible → revert → green)
-
-1. **Online gate removed** (`handleHostDelete`) → `TestHostDelete_OnlineRefused` FAILED: `online delete = 303, want 409` (live host deleted). Reverted.
-2. **Escrow-ack check dropped** (`deleteEscrow := true`) → `TestHostDelete_EscrowAckRequired` FAILED: `escrow-unacked delete = 303, want 409`. Reverted. (This exact mutation is what `146d165` accidentally shipped; `068427a` is the revert on main.)
-3. **Endpoint-delete peers-in-subnet guard removed** → `TestOffsiteEndpointDelete_Guard` FAILED: `delete with peers = 303, want 409`. Reverted.
-4. **Subnet-change guard removed** → `TestOffsiteEndpointSave_SubnetChangeGuard` FAILED: `subnet change with peers = 303, want 409`. Reverted.
-5. **log_bundles scope widened** (`WHERE scope_id = ? OR 1=1`) → `TestDeleteHost_CascadeAndIsolation` FAILED: `customer-scoped log bundle count = 0, want 1` + unrelated-host rows deleted. Reverted.
-
-### 5. Tests
-
-`cd hub && go build ./... && go vet ./... && go test ./...` — **all green** at every commit.
-Test functions in `hub/`: **259 → 280** (+21). Pinned invariants kept green UNMODIFIED:
-hosts-list zero-`<button>`, host-detail exactly-2-buttons (now doubles as the delete-hidden-
-for-online proof, comment added), SECRET-LEAK assertions, pubkey-title + empty-href offsite pins.
-
-### 6. Deploy (GitOps — no `kubectl set image` used)
-
-- Image built + pushed on 180: `gitea.dooplex.hu/admin/felhom-hub:0.47.0` (25M).
-- Deploy = the `manifests/hub.yaml` bump in `95d71fe`; ArgoCD hard-refresh → deliberate sync (Application CR patch).
-- Verify: `sync=Synced health=Healthy`; `deployment "hub" successfully rolled out`; deploy image `gitea.dooplex.hu/admin/felhom-hub:0.47.0`; startup log clean (all checkers initialized, `Listening on :8080`, 2026-07-11 21:41 CEST).
-
-### 7. Live validation — ALL PASS (method: claude-in-chrome driving hub.felhom.eu with the operator's logged-in session, 2026-07-11 ~22:00)
-
-1. **Button contrast (Scenario F) — PASS.** `/hosts/demo-felhom-01` Diagnostics View/Download:
-   computed style verified `color: rgb(10,18,32)` (--bg-0) on `background: rgb(46,168,245)`
-   (--blue-bright); screenshot shows readable labels. Caveat found first: the browser served a
-   CACHED pre-fix style.css (`max-age=3600`, no cache-bust param) — `fetch('/style.css')` cached
-   copy lacked `:not(.btn)`, `{cache:'reload'}` copy had it; a hard reload fixed it (see §9).
-2. **Customer tabs — PASS.** `/customers/demo-felhom`: 8 tabs render, only the active panel
-   visible (`body.js-tabs` set); `#tab=events` deep-link activates Events; `location.reload()`
-   (the auto-refresh mechanism) preserves the hash and the Events panel stays active; Host tab
-   rendered the shared body for BOTH hosts then bound to the customer (demo-felhom-01 ONLINE +
-   dr-drill-host NO REPORT — the list design proven live) with cross-links; sticky strip pinned
-   at `top:0` at scrollY=2000 (`position: sticky` computed). Mobile: at a 390px container the
-   tab nav scrolls horizontally (scrollWidth 725 > clientWidth 342), does not wrap, and the body
-   never scrolls horizontally (window resize was blocked by the maximized window — verified via
-   container-width constraint instead).
-3. **Sanctioned deletions — PASS.** `/hosts` before: 4 hosts. `dr-drill-host` → danger-zone card
-   (host NO REPORT), Remove → impact preview "0 guest row(s), 0 host report(s), 0 agent log
-   bundle(s). Host status: pending." + escrow checkbox (both drill hosts HAD escrow rows) →
-   ticked + retyped id → 303 to `/hosts`, host gone. Repeated for `dr-drill-host3`. `/hosts`
-   after: exactly `demo-felhom-01` (ONLINE) + `peti-felhom-86d37d` (DOWN) — untouched. Neither
-   drill host had a bound wg peer (impact preview showed none), so there was no `/offsite`
-   convergence to observe; the peer table was unchanged (still the 2 real peers).
-4. **Offsite — PASS.** ep0 card renders (address, truncated pubkey w/ title, subnet
-   10.77.0.0/24, PBS 10.77.0.1:8007, **Peers in subnet: 2**), deferral note visible, peer table
-   shows the Endpoint column (ep0). Bad-CIDR probe through the real edit form (`epEdit('ep0')` +
-   subnet `10.99.0.0/240`, posted with the form's own CSRF payload) → **400 "Invalid tunnel
-   subnet (CIDR, e.g. 10.77.0.0/24) — nothing stored."**; re-fetch confirmed ep0's subnet intact
-   and the bad value absent. No real value changed; no second endpoint created.
-
-### 8. NOT live-validated (by design)
-
-- Multi-endpoint allocation/reconciler/desired-state — deferred arc, deliberately untouched (`GetWGEndpoint` semantics pinned by test).
-- Host deletion of a once-online host WITH escrow on the live hub — no such junk host exists; covered by unit tests only.
-
-### 9. Observations (documented, not acted on)
-
-- **`/style.css` is served with `max-age=3600` and no cache-bust param** — after a deploy, browsers can render with the previous CSS for up to an hour (live-observed: the contrast fix looked unfixed until a hard reload). A `?v={{hubVersion}}` on the stylesheet link would eliminate this class of false-negative.
-- **A never-reported host renders its CUSTOMER's storage targets/log bundles**: `hostDetailData` reads `GetLatestHostReportJSON(host.CustomerID)`, so dr-drill-host (no report ever) displayed demo-felhom-01's storage table and the customer-scoped controller bundle. Pre-existing `handleHostDetail` behavior (customer-keyed report lookup), merely surfaced by two hosts sharing a customer — worth a per-host report scope someday.
-- `146d165` (parallel session) demonstrates that `git add -A` doc commits in a shared repo can ship another session's mid-mutation WIP — worth a convention note (stage explicit paths).
-- `hostInstallVersion` in `web/configs.go` still says `1.12.0` while host-install is at v1.13.0 (display-only constant; out of scope here).
-- Hub web sessions live in process memory — every deploy logs every operator out; a persisted session store would remove the "revalidate after each rollout" friction.
-- The workspace-root CLAUDE.md hub-deploy wording predates GitOps (noted in CONTEXT.md).
+DooPlex NAS restored **md5-identical to baseline** (exports + smb.conf; campaign user/share/dirs/creds removed); services never touched (exportfs-only rail held). Guest 9201 as last seen healthy (11 containers, dummy data + backups intact on NAS); `privatebin` (campaign temp) removal deferred to post-recovery. Hub untouched all night.
diff --git a/documentation/audits/CAMPAIGN-3-2026-07-11.md b/documentation/audits/CAMPAIGN-3-2026-07-11.md
new file mode 100644
index 0000000..7514be6
--- /dev/null
+++ b/documentation/audits/CAMPAIGN-3-2026-07-11.md
@@ -0,0 +1,96 @@
+# CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability)
+
+- **When:** 2026-07-11 22:09 → 2026-07-12 HH:MM CEST (launch seed `2868120970`, 8h budget, wrap early — see host incident)
+- **Stack under fire:** controller **v0.117.0** · agent **v0.84.0** · demo guest 9201 on felhom-pve · DooPlex NAS (exports only) · hub untouched
+- **Contract honored:** zero operator questions; continue-on-failure; no hub/PBS/Gitea mutations; DooPlex services never stopped (exportfs on felhom paths + campaign temps only); no code fixes — findings only.
+- **Run architecture (recorded judgment):** campaign driven by the CC session with harness/ledger/evidence at `180:~/campaign3/` (180 has no root SSH to pve; evidence spans 180-API + pve-host + guest vantage points). Phase 5 sampler + Phase 6 endurance ran detached on 180.
+
+## Verdict
+
+**The data plane is solid; the reboot/recovery plane is not.**
+Refusal categories, verify-before-commit, rollback, single-flight, orphan handling, deploy-view truth, backup/restore round-trips and EIO-fast-fail all behaved. What failed is everything around **re-arming NFS automounts after disturbance**: three independent HIGH/CRITICAL findings (F9/F10/F11) mean any NAS hiccup or guest reboot with an idle share strands all NAS apps until a human intervenes — the exact gap the v0.84.0 ReassertNetworkMounts feature was meant to close. Plus one HIGH backup-integrity finding (F7) and the host itself failed to return from its reboot (physical attention needed).
+
+Ledger (final): **30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY · 10 INFO** (FAILs include endurance-loop iterations that honestly fired during reboot/outage/host-down windows).
+
+## Top findings (ranked — these become the next specs)
+
+| # | Sev | Finding | Exact repro |
+|---|-----|---------|-------------|
+| F10 | **CRITICAL** | `mount-start-limit-hit` on a NAS automount is **never re-armed** by any platform path: agent-restart sweep no-op (F9), guest-reboot hook no-op, and the **pre-start hook once blocked guest start entirely** (rc255 while the limit was live → guest DOWN). All 4 NAS apps dead until manual `systemctl reset-failed`. | unexport share on NAS; wait 60s idle-timeout; access path 5×; `pct reboot` within the limit window |
+| F11 | **HIGH** | Guest reboot with an **idle** (armed-not-mounted) share: the autofs trigger does **not** propagate into the new container namespace. The post-start reassert **detects it and logs a WARNING** ("share NOT visible in guest after reassert (skip-active)") **but does not remediate** — the skip-active branch is the bug. Manual `systemctl restart <automount>` heals instantly. Every reboot with idle share = 4 NAS apps dead-at-boot (couples with fix-3: no alerting). Reproduced on reboots #2, #3, #4. | `pct reboot 9201` while nas-media idle; watch hook WARNING + `docker ps` |
+| F7 | **HIGH** | Volume dumps are written **in place, no tmp+rename**: a mid-backup NAS cut left a **0-byte calibre tar replacing the last good 247M dump**. Until the next successful run, the only "restore point" restores an **empty volume** (tier-1 restore is replace-semantics). Run-level `success:false` is the only signal; the artifact damage itself is silent. Next run self-heals. | `POST /api/backup/run`; `exportfs -u` the backup-target share at T+6s; inspect `volume-dumps/*.tar` |
+| F9 | **HIGH** | Agent-restart startup sweep produced **zero reassert log lines** and left the automount failed/disarmed — likely checks unit enablement, not runtime state. | get F10 state; `systemctl restart felhom-agent`; grep journal |
+| F8 | **MED** | Storage health stays **`idle` + `reachable:true` through an export outage** (probe is server-level, not export-level) — even across a reboot with NFS down the share badge shows benign "Készenlét" while stacks/dashboard show stub warnings. Spec expected `unreachable`; got neither silent nor correct: **contradictory UI**. | `exportfs -u` while idle; poll `/api/storage/netstorage` 5 min |
+| F6 | LOW | Volume-only apps (no HDD_PATH) back up to **sys_drive** with `drive_label:""` (blank in UI) and get **no tier-2 secondary** — single-copy exposure. | `GET /api/backup/snapshots?stack=actualbudget` |
+| F4 | LOW | `mapped_uid` out of range (101000) passes controller validation, fails only at agent → generic `agent_error` with raw detail instead of a friendly 400. | add with `mapped_uid:101000` |
+| F5 | LOW | `backups/primary/` retains stale dirs for apps since redeployed to another drive (invisible in snapshot list, disk residue). | redeploy app to other drive; inspect old drive |
+| F2 | LOW | Failed-add rollback removes unit files but leaves systemd **runtime units in failed state** (no `reset-failed`) + leftover mountpoint dirs. | any refused add past reachability |
+| F1 | LOW | `netstorage/remove` leaves the empty mountpoint dir on host+guest (stub-shaped residue; 10 accumulated during P1). | add+remove any share |
+| F3 | DISCREPANCY | Spec expected `not_network_fs` for a no-all_squash export; actual = `not_writable` (×2, deterministic, message even hints all_squash). Correct-by-design: `not_network_fs` is the deploy/stub-time branch — which WAS live-validated via the organic stub (below). | — |
+
+**Host incident (HUMAN):** felhom-pve did **not return** from the planned 4e reboot at 23:30 (no ping, SSH 22/8822 closed, controller 530 via Cloudflare). Polled every 5 min until **04:27 — never came back**. Needs physical attention (power/console). Note this reboot was issued while the box was in a healed, settled state — the failure is host-level (BIOS/boot device/console prompt?), not something the campaign's guest chaos can explain. The 4e boot-ordering proof is therefore **NOT RUN**.
+
+## What passed (the good news)
+
+- **NAS add/remove lifecycle (~19 ops):** happy NFS routeA-style 2.6s / prod-style 2.5s / SMB 2.5s; all 5 refusal categories ×2 with correct category + Hungarian copy in 2–5s (**retry=0 holds everywhere**); single-flight = exactly one 409; SMB password lands in a root-only `credentials=` file and is **gone after remove**; registry consistent after every cycle.
+- **Ring reconstruction (S7):** every add fully reconstructable — phase transitions, `capability gate: netstorage_verify=yes (source=version)`, probe verdict, timings.
+- **Orphan flow:** controller restart mid-add → `Árva megosztás` row surfaced → removed via the real action → clean.
+- **Deploy wave:** radarr/jellyfin/navidrome onto nas-media + seerr/actualbudget (volume-only) + privatebin; all `running`; **deploy-view truth** holds for HDD apps (selected option == stored path; volume-only apps have no HDD_PATH field — N/A recorded, not a bug).
+- **Hardlink (*arr pattern) works on NFSv4.1** (`links=2`, correct uid mapping) — the Synology-era caveat does not apply to the DooPlex export.
+- **Backup/restore:** tier-1 31s / tier-2 5s / offsite restic cycle ~30s clean (quota 129.7MB/50GB); restore round-trips **byte-identical** on calibre-web (53 files, deleted+corrupted recovered, 25s) and actualbudget; tier-1 restore = replace-semantics (defined; tier-2 is the additive path).
+- **EIO behavior under outage:** active read fails **same-second** (soft,timeo=50,retrans=2), D-count=0, containers survive, re-export recovers without remount; mid-backup cut fails clean in 14s, no hang.
+- **Stub machinery:** the organic stub (outage→idle-timeout→access) fires the distinct stub badge (4 apps) AND the deploy-time 409 with the exact Hungarian refusal. The branch is now live-validated.
+- **Concurrency spice:** deploy-during-add, backup race (200/409), rapid add/remove ×5, remove-during-backup — all clean.
+
+## Backup-vs-NAS policy discovery (docs finding, not pass/fail)
+
+1. Tier-1 backs up **volumes + config only**; NAS-hosted HDD userdata (the 2.8G media tree) is **excluded by design** (jellyfin backup = 560K).
+2. NAS apps' tier-1 artifacts land **on the NAS itself** (`nas-media/backups/primary/`) — tier-2 then copies them cross-drive to felhom-usb, which is what gets them off-NAS. During a NAS outage both tier-1 (target) and the app data are on the dead device.
+3. Volume-only apps → `sys_drive/felhom-data/backups/primary/`, blank drive label, **no tier-2 copy** (F6).
+4. Outage during backup: clean failure, plus F7's truncation window.
+
+## Fix-6 ring numbers (wrap horizon, measured)
+
+- **Idle:** controller ring at 1000-cap with ~55 min horizon (21:20→22:15).
+- **Under campaign load:** ~2.6 entries/s → **cap horizon ≈ 6.5 min** (23:04→23:08 = 618 entries). A refused flow older than ~6 min under load is already unreconstructable.
+- **Restart/reboot loss:** controller restart at 22:26 wiped the ring (the interrupted orphan-add's trace was gone); both rings died on every reboot (in-memory, expected) — post-incident operators lose exactly the window that matters. The viewer's `total` field tracked entries correctly in all samples (no `1067/1000`-style overshoot reproduced; the counter reads clean).
+- Verdict for fix-6: the 1000-entry cap is **too small under load** and ring persistence across restart is the real gap.
+
+## Timings table
+
+| Operation | Measured |
+|---|---|
+| happy add (NFS/SMB) | 2.5–2.6 s |
+| refusals (all categories) | 2–5 s |
+| EIO after unexport (active read) | same second |
+| backup fail-clean after cut | 14 s |
+| deploy → running | 25–60 s |
+| tier-1 / tier-2 / offsite run | 31 s / 5 s / 30 s |
+| restore round-trip | 25 s |
+| boot→all-healthy | **never zero-touch with idle NAS share** (F11); manual heal ~1 min |
+
+## Deviations & judgment calls (all ledgered)
+
+- Reboots #2/#3 spaced 5–6 min instead of 15 (fast phase pace; state verified settled between). #4 and host reboot properly spaced.
+- Endurance-loop injections limited to campaign exports (not felhom-data) to keep the unattended loop from re-tripping F10 with no pve-side healer.
+- App-level API records skipped for seerr/actualbudget (interactive auth setup — not trivial per spec).
+- `?raw=1` on the debug ring does not exist (it belongs to per-app container logs); JSON `/api/debug/logs` used instead.
+- One offsite cycle run (of ≤2 allowed); second budget unused.
+- No midnight operator push about the host (expendable demo box, Peti unaffected); this report is the notification.
+
+## Box state at wrap
+
+- **felhom-pve: DOWN** (since the 23:30 planned reboot; HUMAN: physical power/console needed).
+- **DooPlex/NAS: fully restored** — /etc/exports and smb.conf restored from `.bak-campaign3`, **md5 == baseline** (3 exports); campaign SMB user/share/creds/dirs removed; felhom-data export live; all customer/homelab services untouched throughout (exportfs-only rail held).
+- **Guest 9201 (as last seen, 23:33, pre-host-reboot):** 11 containers healthy — controller 0.117.0, calibre-web/radarr/jellyfin/navidrome (nas-media), seerr/actualbudget (volumes), privatebin (campaign temp — **removal via the real flow was pre-empted by the host loss**; remove after recovery), filebrowser + infra. Dummy media data (2.8G) + backups intact on the NAS.
+- **Hub:** untouched all night (reports were flowing until host loss; row will show the host offline — correct).
+
+### Morning recovery runbook (operator)
+1. Power/console felhom-pve; check why boot stalled (BIOS/GRUB/fsck).
+2. After boot, **expect F10/F11**: NAS apps dead-at-boot. Heal: `systemctl reset-failed 'mnt-felhom*'`, `systemctl restart 'mnt-felhom\x2ddrives-nas\x2dmedia.automount'`, then in guest `docker start radarr jellyfin navidrome calibre-web`.
+3. `POST /api/stacks/privatebin/remove` (campaign temp app) — or leave it, it's harmless.
+4. Verify: no stub badge on /stacks, nas-media `ok`, hub row green, one manual `POST /api/backup/run` green.
+
+## Evidence
+
+`180:~/campaign3/` — `ledger.tsv` (66+ entries), `master.log` (every API call), `evidence/` (per-scenario ring+shares+stacks snapshots), `ring-samples.log`. Guest-side manifests at `9201:/root/c3/`. No secrets in any committed artifact; campaign SMB cred stored only on 180 and deleted at wrap.