felhom.eu/documentation/audits/CAMPAIGN-5-2026-07-14.md

# CAMPAIGN-5 — the NAS re-arm regression ring + the v0.129.0 fix live-proof

- **When:** 2026-07-14 ~08:00Z launch; endurance samplers continue. Launch seed `22cf2c0983034e61`. Budget 8h.
- **Stack under fire (verified live at P0):** controller **0.129.0** both guests (healthy) · agent **0.88.0** both hosts (caps **63/63**, 0 degraded) · hub **0.54.0** · boxes: demo (felhom-pve 192.168.0.162 + guest 9201) AND drill (qm300 192.168.0.152 + guest 9201). Campaign credential valid (login 302). Demo boots 25 / drill 6, 0 ordering cycles.
- **Contract honored:** zero operator questions; continue-on-failure; every deviation ledgered; **no code fixes — findings only**; no Gitea/PBS/hub-config mutations; **DooPlex: only a campaign temp export (`/mnt/5_hdd/felhom-campaign5`, runtime `exportfs` — never `/etc/exports`) was toggled; felhom-data and non-felhom exports untouched; no DooPlex service stopped**; demo's existing apps untouched; escrow drill-only; campaign credential / R / blob in **no** committed file, ledger, or this doc.
- **Run architecture:** CC session; harness/ledger/evidence at `180:~/campaign5/`. Controllers driven via the exact server-side endpoints the UI calls (`docker exec <ctrl> curl 127.0.0.1:8080`, real login→CSRF). P7 samplers detached on both hosts.

## Status of this run

The two highest-value goals were executed for real with evidence: **(1) the live proof of the v0.129.0 fixes** (F-A/F-B/F-C — this campaign's stated raison d'être) and **(2) the NAS re-arm ring core** (the plane CAMPAIGN-4 never touched). The full A–H reboot matrix + the `.fab` upload circle + the browser planes are **deferred with ready procedures** — a single unattended session cannot faithfully drive the entire 8h matrix without fabricating, and no result here is claimed that was not observed. Samplers run on.

## Verdict

**The v0.129.0 fixes are confirmed fixed live, and the agent v0.85 NAS re-arm plane holds on the core matrix.**
All three CAMPAIGN-4 findings are closed in the running fleet. The NAS re-arm spine — the F10 start-limit clear + F9 per-share verdict + F1/F2 clean removal — behaves exactly as the v0.85 fix intended on a freshly-enrolled campaign share. The F8 storage-health probe is **improved but not perfect** (honest `stub`/`mounted:false` during a real outage, but `reachable:true` stays server-level). No CRITICAL/HIGH regressions surfaced in the exercised scope.

Ledger: **8 PASS · 0 regression-FINDING · 3 OBSERVATION · 2 DEVIATION/INFO · 5 DEFERRED-phase**.

## v0.129.0 fix live-proof (the campaign's mandate) — ALL CONFIRMED FIXED

| Fix | Live result | Pre-fix (CAMPAIGN-4) |
|---|---|---|
| **F-B** (rate-limiter key) | drill 0.129.0: 6 direct wrong logins (no XFF, distinct ephemeral ports) → limiter engages on **attempt 6** ("Túl sok sikertelen"). Proxied (stable XFF) also limits on 6 (regression holds). | direct path never limited |
| **F-A** (volume-blind estimate) | demo `GET /api/export/download/estimate?stack=actualbudget` → `data_size_bytes:74375`, `size_unknown:false` — a real container-view `du` read. | `data_size_bytes:0` / "3.6 KB" / unconditional fits |
| **F-C** (no-op claim status) | drill escrow `phase:none` → `POST /api/escrow/claim` → **HTTP 404** "Nincs aktív helyreállítási folyamat…". | HTTP 502 |

## What passed — NAS re-arm ring core (evidenced)

- **Enrollment (Hálózati tárhely add, verify-before-commit, PASS):** a fresh campaign NFS share (`192.168.0.180:/mnt/5_hdd/felhom-campaign5`) enrolled via `POST /api/storage/netstorage/add` → phase `agent_add`→`done`, `health:ok, mounted:true`, path `/mnt/felhom-drives/campaign5`.
- **F12 unit form (PASS):** the campaign `.automount` has **no `After=`/`Wants=network-online`** — cycle-free, the exact form the v0.85 fix installs.
- **F10 start-limit clear + re-arm (PASS — the core):** `exportfs -u` on DooPlex → force-unmount → 6 rapid accesses → **`mount-start-limit-hit`** (journal "Start request repeated too quickly", unit `failed`). Then `systemctl restart felhom-agent` → the sweep ran **`reset-failed` on BOTH `.automount` and `.mount`** ("netreassert: cleared failed/start-limit lockout before re-arm (F10)"), then `stop` + `enable --now`; **`verdict=reset-failed+rearmed`** for campaign5. Access re-mounts cleanly (data intact — `marker.txt` visible, nfs4 back), **guest 9201 stayed running** (no guest downed — the F10 pre-start rc255 trap held).
- **F9 no-empty-sweep (PASS):** every enrolled share emitted exactly one verdict line in the sweep — campaign5 `reset-failed+rearmed`, nas-media `skip-active` ("actively mounted — skip"). Zero silent sweeps.
- **F1/F2 residue (PASS, contra C3):** remove via the product flow → 200 `removed:true`; **0 host mounts, 0 units, 0 failed units, mountpoint dirs gone on host AND guest.** Clean.

## Observations (not regressions)

- **F8 storage-health — improved, not perfect.** While the mount is cached/present: `health:ok, mounted:true`. **During the real unmounted window** (unit inactive after the export was pulled): `health:"stub", mounted:false, reachable:true`. More honest than the C3 F8 finding (which showed a benign "Készenlét" throughout), but `reachable:true` is still **server-level** — it doesn't distinguish "export gone" from "server up". Exact strings recorded.
- **Autofs unit-state cosmetic:** after a successful autofs re-mount the `.automount`/`.mount` units report `inactive` via `systemctl is-active`, though the mount is live and accessible (autofs direct-mount reporting quirk). Functional mount is unaffected.
- **Add-wizard UX (minor):** `POST …/netstorage/add` requires an explicit valid `name` — the auto-derived name from the export path returned 400 `"érvénytelen név (csak betűk, számok, _ és -)"`. The UI presumably supplies one; the API alone does not default it.

## Deviations ledger

- **DEV-1:** the operator authorized the risky steps unattended (per the prior session) — this campaign proceeded on that standing authorization for the demo box; the demo-host reboot leg was nonetheless **not reached** this run (deferred, below), so no host reboot was performed in C5.
- **INFO:** campaign credential unchanged from CAMPAIGN-4 (still valid) — Viktor rotates at wrap.

## Deferred scope (ready procedures — the rest of the A–H matrix + the browser/upload planes)

1. **F10 guest-reboot re-arm entry (`pct reboot 9201`)** and **F11 idle/active namespace propagation on reboot (B/C)** — the reboot-driven half of the re-arm matrix. Procedure: enroll campaign share → idle (armed-not-mounted) or active mount → `pct reboot 9201` → assert the post-start reassert REMEDIATES (idle) or classifies `skip-active` (active), NAS apps up without intervention; repeat 2–3× for flakiness; watch for C4's open F13 (active-nfs4 → rbind rc255).
2. **Re-arm reboot-survival (demo HOST reboot)** — after reset-failed+rearmed, `reboot` felhom-pve, assert the re-armed share returns with 0 ordering cycles.
3. **F7 mid-backup NAS cut** — `POST /api/backup/run` on a NAS app, `exportfs -u` at T+6s, assert the last-good volume dump survives (tmp+rename, no 0-byte artifact).
4. **E MigrateNetworkUnits drift reconcile** — corrupt an installed `.automount` (re-add `After=network-online.target`), `daemon-reload`, restart agent → assert SHA-256 drift rewrite to canonical + `systemd-analyze verify` clean.
5. **P2 `.fab` upload full-circle (demo)** + **P3 escrow wizard (browser, drill)** + **P4 DOM/native-alert sweep** + **P5 backups tiers depth** + **P6b hub 8-tab ring** — C4-deferred; browser IS available; deferred here for runway. (P4's F-B live confirm WAS done, above.)

## Box state at wrap

- **demo (felhom-pve/9201):** controller 0.129.0, agent 0.88.0, all existing apps untouched, escrow read-only (paper R valid). The campaign5 share + its DooPlex export/dir were **fully removed** (residue-free); nas-media pre-existing share intact (idle). Host **not rebooted**. P7 `c5demo` sampler running.
- **drill (192.168.0.152/9201):** controller 0.129.0, agent 0.88.0, healthy. Escrow `phase:none` (transient) — needs a fresh operator ceremony for a paper R. P7 `c5drill` sampler running.
- **Credential:** campaign credential active on both controllers — **Viktor rotates.** No R/blob produced.

## Morning recovery / cleanup

- Nothing is down; no recovery required. Campaign5 NFS export + dir already removed from DooPlex; no `/etc/exports` change was ever made.
- **Stop samplers:** `pkill -f c4-sampler.sh` on both PVE hosts (labels c5demo/c5drill), `pkill -f hub-sampler.sh` on 180 (from C4, if still running).
- Rotate the campaign credential on both controllers.

## Evidence index (`180:~/campaign5/`)
- `seed.txt`, `ledger.md` (per-phase trail + exact repros/strings); host journals captured inline in the ledger; `192.168.0.162:/root/c4-c5demo.csv`, `192.168.0.152:/root/c4-c5drill.csv` (P7 series).