felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md

# CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability)

- **When:** 2026-07-11 22:09 → 2026-07-12 HH:MM CEST (launch seed `2868120970`, 8h budget, wrap early — see host incident)
- **Stack under fire:** controller **v0.117.0** · agent **v0.84.0** · demo guest 9201 on felhom-pve · DooPlex NAS (exports only) · hub untouched
- **Contract honored:** zero operator questions; continue-on-failure; no hub/PBS/Gitea mutations; DooPlex services never stopped (exportfs on felhom paths + campaign temps only); no code fixes — findings only.
- **Run architecture (recorded judgment):** campaign driven by the CC session with harness/ledger/evidence at `180:~/campaign3/` (180 has no root SSH to pve; evidence spans 180-API + pve-host + guest vantage points). Phase 5 sampler + Phase 6 endurance ran detached on 180.

## Verdict

**The data plane is solid; the reboot/recovery plane is not.**
Refusal categories, verify-before-commit, rollback, single-flight, orphan handling, deploy-view truth, backup/restore round-trips and EIO-fast-fail all behaved. What failed is everything around **NFS automount units and their re-arming**: four independent CRITICAL/HIGH findings (F12/F10/F11/F9) mean a host reboot is a coin flip between losing networking and losing the NAS (F12), and any NAS hiccup or guest reboot with an idle share strands all NAS apps until a human intervenes — the exact gap the v0.84.0 ReassertNetworkMounts feature was meant to close. Plus one HIGH backup-integrity finding (F7). The overnight host loss was F12 in action — hardware exonerated by RCA.

Ledger (final): **31 PASS · 17 FAIL · 12 FINDING · 1 DISCREPANCY · 11 INFO** (FAILs include endurance-loop iterations that honestly fired during reboot/outage/host-down windows).

## Top findings (ranked — these become the next specs)

| # | Sev | Finding | Exact repro |
|---|-----|---------|-------------|
| F12 | **CRITICAL** | Agent network-storage **automount** unit orders `After=`+`Wants=network-online.target` → boot **ordering cycle** with `local-fs.target`; systemd deletes an arbitrary job to break it — one boot lost **networking entirely** (host dark 7 h), the next boot lost the **automount** (NAS dead). Every boot with an enrolled share is a coin flip. Fix: remove network-online ordering from the automount (`_netdev` on the `.mount` suffices). | reboot a host with any enrolled network share; `journalctl -b \| grep "ordering cycle"` |
| F10 | **CRITICAL** | `mount-start-limit-hit` on a NAS automount is **never re-armed** by any platform path: agent-restart sweep no-op (F9), guest-reboot hook no-op, and the **pre-start hook once blocked guest start entirely** (rc255 while the limit was live → guest DOWN). All 4 NAS apps dead until manual `systemctl reset-failed`. | unexport share on NAS; wait 60s idle-timeout; access path 5×; `pct reboot` within the limit window |
| F11 | **HIGH** | Guest reboot with an **idle** (armed-not-mounted) share: the autofs trigger does **not** propagate into the new container namespace. The post-start reassert **detects it and logs a WARNING** ("share NOT visible in guest after reassert (skip-active)") **but does not remediate** — the skip-active branch is the bug. Manual `systemctl restart <automount>` heals instantly. Every reboot with idle share = 4 NAS apps dead-at-boot (couples with fix-3: no alerting). Reproduced on reboots #2, #3, #4. | `pct reboot 9201` while nas-media idle; watch hook WARNING + `docker ps` |
| F7 | **HIGH** | Volume dumps are written **in place, no tmp+rename**: a mid-backup NAS cut left a **0-byte calibre tar replacing the last good 247M dump**. Until the next successful run, the only "restore point" restores an **empty volume** (tier-1 restore is replace-semantics). Run-level `success:false` is the only signal; the artifact damage itself is silent. Next run self-heals. | `POST /api/backup/run`; `exportfs -u` the backup-target share at T+6s; inspect `volume-dumps/*.tar` |
| F9 | **HIGH** | Agent-restart startup sweep produced **zero reassert log lines** and left the automount failed/disarmed — likely checks unit enablement, not runtime state. | get F10 state; `systemctl restart felhom-agent`; grep journal |
| F8 | **MED** | Storage health stays **`idle` + `reachable:true` through an export outage** (probe is server-level, not export-level) — even across a reboot with NFS down the share badge shows benign "Készenlét" while stacks/dashboard show stub warnings. Spec expected `unreachable`; got neither silent nor correct: **contradictory UI**. | `exportfs -u` while idle; poll `/api/storage/netstorage` 5 min |
| F6 | LOW | Volume-only apps (no HDD_PATH) back up to **sys_drive** with `drive_label:""` (blank in UI) and get **no tier-2 secondary** — single-copy exposure. | `GET /api/backup/snapshots?stack=actualbudget` |
| F4 | LOW | `mapped_uid` out of range (101000) passes controller validation, fails only at agent → generic `agent_error` with raw detail instead of a friendly 400. | add with `mapped_uid:101000` |
| F5 | LOW | `backups/primary/` retains stale dirs for apps since redeployed to another drive (invisible in snapshot list, disk residue). | redeploy app to other drive; inspect old drive |
| F2 | LOW | Failed-add rollback removes unit files but leaves systemd **runtime units in failed state** (no `reset-failed`) + leftover mountpoint dirs. | any refused add past reachability |
| F1 | LOW | `netstorage/remove` leaves the empty mountpoint dir on host+guest (stub-shaped residue; 10 accumulated during P1). | add+remove any share |
| F3 | DISCREPANCY | Spec expected `not_network_fs` for a no-all_squash export; actual = `not_writable` (×2, deterministic, message even hints all_squash). Correct-by-design: `not_network_fs` is the deploy/stub-time branch — which WAS live-validated via the organic stub (below). | — |

**Host incident — RCA CLOSED 2026-07-12 06:48 (= finding F12, CRITICAL).** felhom-pve was unreachable from the 4e reboot (23:30) until the operator power-cycled it at 06:45. `journalctl --list-boots` proved the box **booted fine at 23:31:04 and ran all night — with no network**: the agent's automount unit template (`After=` + `Wants=network-online.target` on the **automount** unit, which is implicitly `Before=local-fs.target`) creates the boot ordering cycle `networking → local-fs → nas-media.automount → network-online → networking`. systemd breaks such a cycle by deleting an **arbitrary** job in it: boot `-1` deleted `networking.service` (host up, no network, 7 h); boot `0` after the power cycle hit the **same cycle** and deleted the **automount** instead (network up, NAS dead — recovered manually). **Every boot of a host with an enrolled network share is a coin flip.** Hardware exonerated; the power cycle fixed nothing but the dice. Fix for the spec: drop the network-online ordering from the automount unit — `_netdev` on the `.mount` already delays the actual mount until the network is up; the trigger needs nothing. (Interim host-side mitigation if needed before the agent fix: a systemd drop-in removing the ordering on the installed automount units.) The 4e boot-ordering proof therefore **ran and caught exactly what it was designed to catch.**

## What passed (the good news)

- **NAS add/remove lifecycle (~19 ops):** happy NFS routeA-style 2.6s / prod-style 2.5s / SMB 2.5s; all 5 refusal categories ×2 with correct category + Hungarian copy in 2–5s (**retry=0 holds everywhere**); single-flight = exactly one 409; SMB password lands in a root-only `credentials=` file and is **gone after remove**; registry consistent after every cycle.
- **Ring reconstruction (S7):** every add fully reconstructable — phase transitions, `capability gate: netstorage_verify=yes (source=version)`, probe verdict, timings.
- **Orphan flow:** controller restart mid-add → `Árva megosztás` row surfaced → removed via the real action → clean.
- **Deploy wave:** radarr/jellyfin/navidrome onto nas-media + seerr/actualbudget (volume-only) + privatebin; all `running`; **deploy-view truth** holds for HDD apps (selected option == stored path; volume-only apps have no HDD_PATH field — N/A recorded, not a bug).
- **Hardlink (*arr pattern) works on NFSv4.1** (`links=2`, correct uid mapping) — the Synology-era caveat does not apply to the DooPlex export.
- **Backup/restore:** tier-1 31s / tier-2 5s / offsite restic cycle ~30s clean (quota 129.7MB/50GB); restore round-trips **byte-identical** on calibre-web (53 files, deleted+corrupted recovered, 25s) and actualbudget; tier-1 restore = replace-semantics (defined; tier-2 is the additive path).
- **EIO behavior under outage:** active read fails **same-second** (soft,timeo=50,retrans=2), D-count=0, containers survive, re-export recovers without remount; mid-backup cut fails clean in 14s, no hang.
- **Stub machinery:** the organic stub (outage→idle-timeout→access) fires the distinct stub badge (4 apps) AND the deploy-time 409 with the exact Hungarian refusal. The branch is now live-validated.
- **Concurrency spice:** deploy-during-add, backup race (200/409), rapid add/remove ×5, remove-during-backup — all clean.

## Backup-vs-NAS policy discovery (docs finding, not pass/fail)

1. Tier-1 backs up **volumes + config only**; NAS-hosted HDD userdata (the 2.8G media tree) is **excluded by design** (jellyfin backup = 560K).
2. NAS apps' tier-1 artifacts land **on the NAS itself** (`nas-media/backups/primary/`) — tier-2 then copies them cross-drive to felhom-usb, which is what gets them off-NAS. During a NAS outage both tier-1 (target) and the app data are on the dead device.
3. Volume-only apps → `sys_drive/felhom-data/backups/primary/`, blank drive label, **no tier-2 copy** (F6).
4. Outage during backup: clean failure, plus F7's truncation window.

## Fix-6 ring numbers (wrap horizon, measured)

- **Idle:** controller ring at 1000-cap with ~55 min horizon (21:20→22:15).
- **Under campaign load:** ~2.6 entries/s → **cap horizon ≈ 6.5 min** (23:04→23:08 = 618 entries). A refused flow older than ~6 min under load is already unreconstructable.
- **Restart/reboot loss:** controller restart at 22:26 wiped the ring (the interrupted orphan-add's trace was gone); both rings died on every reboot (in-memory, expected) — post-incident operators lose exactly the window that matters. The viewer's `total` field tracked entries correctly in all samples (no `1067/1000`-style overshoot reproduced; the counter reads clean).
- Verdict for fix-6: the 1000-entry cap is **too small under load** and ring persistence across restart is the real gap.

## Timings table

| Operation | Measured |
|---|---|
| happy add (NFS/SMB) | 2.5–2.6 s |
| refusals (all categories) | 2–5 s |
| EIO after unexport (active read) | same second |
| backup fail-clean after cut | 14 s |
| deploy → running | 25–60 s |
| tier-1 / tier-2 / offsite run | 31 s / 5 s / 30 s |
| restore round-trip | 25 s |
| boot→all-healthy | **never zero-touch with idle NAS share** (F11); manual heal ~1 min |

## Deviations & judgment calls (all ledgered)

- Reboots #2/#3 spaced 5–6 min instead of 15 (fast phase pace; state verified settled between). #4 and host reboot properly spaced.
- Endurance-loop injections limited to campaign exports (not felhom-data) to keep the unattended loop from re-tripping F10 with no pve-side healer.
- App-level API records skipped for seerr/actualbudget (interactive auth setup — not trivial per spec).
- `?raw=1` on the debug ring does not exist (it belongs to per-app container logs); JSON `/api/debug/logs` used instead.
- One offsite cycle run (of ≤2 allowed); second budget unused.
- No midnight operator push about the host (expendable demo box, Peti unaffected); this report is the notification.

## Box state at wrap (final, 2026-07-12 06:53)

- **felhom-pve: RECOVERED** — operator power-cycled 06:45; RCA = F12 ordering cycle (above), hardware exonerated. Automount manually started (boot 0's cycle-break had deleted its job), all apps healed.
- **Guest 9201: defined state reached** — 6 wave apps (radarr/jellyfin/navidrome on nas-media, seerr/actualbudget volumes, calibre-web) deployed + healthy with their data; `privatebin` (campaign temp) stop+removed via the real flow; verification backup run **success:true** (25 s); nas-media `ok`+mounted; stub badge 0; deployed-not-running: none.
- ⚠ **Until F12 is fixed, every reboot of this host is the coin flip** — next boot may come up without network again (interim: systemd drop-in, or fix the agent template).
- **DooPlex/NAS: fully restored** — /etc/exports and smb.conf restored from `.bak-campaign3`, **md5 == baseline** (3 exports); campaign SMB user/share/creds/dirs removed; felhom-data export live; all customer/homelab services untouched throughout (exportfs-only rail held).
- **Guest 9201 (as last seen, 23:33, pre-host-reboot):** 11 containers healthy — controller 0.117.0, calibre-web/radarr/jellyfin/navidrome (nas-media), seerr/actualbudget (volumes), privatebin (campaign temp — **removal via the real flow was pre-empted by the host loss**; remove after recovery), filebrowser + infra. Dummy media data (2.8G) + backups intact on the NAS.
- **Hub:** untouched all night (reports were flowing until host loss; row will show the host offline — correct).

### Morning recovery (EXECUTED 06:48–06:53)
1. ✔ Operator power-cycled; RCA established from `journalctl -b -1` (F12 ordering cycle; boot -1 sacrificed networking.service, boot 0 sacrificed the automount).
2. ✔ Automount started, NAS apps + all others healthy (11 containers).
3. ✔ privatebin stopped + removed via the real flow (`remove_hdd_data` + `remove_backups`).
4. ✔ Verified: stub 0, nas-media `ok`, deployed-not-running none, verification backup `success:true`.

## Evidence

`180:~/campaign3/` — `ledger.tsv` (66+ entries), `master.log` (every API call), `evidence/` (per-scenario ring+shares+stacks snapshots), `ring-samples.log`. Guest-side manifests at `9201:/root/c3/`. No secrets in any committed artifact; campaign SMB cred stored only on 180 and deleted at wrap.