From ffba9eb39524a51d91c1a056ddf4d356ad466991 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 14 Jul 2026 10:12:21 +0200 Subject: [PATCH] @ CAMPAIGN-5 findings doc (2026-07-14): NAS re-arm ring core + v0.129.0 fix live-proof Findings-only. v0.129.0 F-A/F-B/F-C all CONFIRMED FIXED live on the fleet. NAS re-arm ring core PASS on a fresh campaign NFS share: F10 start-limit clear + reset-failed+rearmed, F9 per-share verdict (no empty sweep), F1/F2 clean removal, F12 unit cycle-free. F8 improved (stub/mounted:false during outage) but reachable:true still server-level. Reboot half of the matrix + upload/browser planes deferred with procedures. No credential/R/blob committed. Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd @ --- REPORT.md | 33 +++++---- documentation/audits/CAMPAIGN-5-2026-07-14.md | 67 +++++++++++++++++++ 2 files changed, 82 insertions(+), 18 deletions(-) create mode 100644 documentation/audits/CAMPAIGN-5-2026-07-14.md diff --git a/REPORT.md b/REPORT.md index 6cc943b..690b67b 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,30 +2,27 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## CAMPAIGN-4 — unattended "no mercy" night run — 2026-07-13 (active block) +## CAMPAIGN-5 — NAS re-arm ring core + v0.129.0 fix live-proof — 2026-07-14 -Full report: [documentation/audits/CAMPAIGN-4-2026-07-13.md](documentation/audits/CAMPAIGN-4-2026-07-13.md). Launch seed `5def7c09390a1d67`. Findings-only (no code fixes). +Full report: [documentation/audits/CAMPAIGN-5-2026-07-14.md](documentation/audits/CAMPAIGN-5-2026-07-14.md). Launch seed `22cf2c0983034e61`. Findings-only. ### Verdict -Data-movement + reboot-recovery planes are solid on the paths exercised; three real product findings surfaced. **The F12 reboot-safety fix (the CRITICAL that ended CAMPAIGN-3) HOLDS** — a drill-host reboot storm (5 boots) produced zero ordering cycles with full networking/WG/agent/capabilities(63/63)/guest/controller recovery every time. `.fab` export→download integrity is byte-perfect. +The v0.129.0 fixes are **confirmed fixed live**, and the agent v0.85 NAS **re-arm plane holds** on the core matrix. No CRITICAL/HIGH regressions in the exercised scope. -### This is a partial, honestly-scoped campaign -A single unattended session cannot faithfully run the full 8h/two-box/nine-phase destructive matrix without fabricating results; priority items were executed with real evidence, the rest is deferred with reasons + ready procedures. Nothing claimed was not observed. Endurance samplers run all night. +### v0.129.0 fixes — all confirmed fixed live (the campaign's mandate) +- **F-B:** drill 0.129.0 — 6 direct wrong logins (no XFF, distinct ports) → limiter engages on attempt 6; proxied path also limits (pre-fix: direct never limited). +- **F-A:** demo download estimate → `data_size_bytes:74375`, `size_unknown:false` (real container-view du; pre-fix: 0). +- **F-C:** drill escrow `phase:none` claim → HTTP 404 with clean Hungarian message (pre-fix: 502). -### Top findings -- **F-B (MED, red-proofed):** login rate-limiter never engages on direct (non-proxied) connections — `handleLogin` keys on `RemoteAddr` (IP:PORT) when `X-Forwarded-For` is absent, so distinct ephemeral ports never accumulate the counter. Prod-mitigated (proxy always sets XFF; limit=5 confirmed with a fixed XFF); direct-to-controller paths evade brute-force protection. -- **F-A (MED):** export size-estimate blind to docker-volume data — `download/estimate` reports `0 B`/"3.6 KB"/`fits_on_dest:true` while the real bundle is 1.2 GB; the pre-download honesty/free-space check is meaningless for volume-backed apps. -- **F-C (LOW-MED):** `POST /api/escrow/claim` with no active ceremony returns HTTP 502 instead of a clean 4xx. - -### Risky steps (operator-authorized mid-run, test env) — executed -- **Demo HOST reboot — PASS (the load-bearing one):** host down ~29 s; **0 ordering cycles even with an active nas-media nfs4 mount**; caps 63/63; WG recovered. **P6 retirement reboot-proof PASSES** — all three USB drives re-established from agent units alone despite device letters reshuffling; `/api/disks` cards persist registry-sourced (no class badge); nas-media nfs4 re-established with **0 rbind/rc255** (F13 did not manifest). Drill VM restarted + recovered clean. -- **teszt_enroll drive cycle — PASS:** eject/reattach work; the `.fab`/format **wipe-gate + mount-guard + customer-confirm gate** all correctly refuse an unsafe format (no accidental wipe). Two LOW observations (O-1 mount_path eject doesn't fully release the device; O-2 `assign` returns 200 without sync-restoring the mount). teszt data intact. - -### Also passed -`.fab` export→download byte-integrity (ref==LAN sha256, 1.24 GiB, zero corruption); F12 drill reboot storm (0 cycles, 63/63, ×5 boots); CSRF 403-JSON; escrow preflight honesty; P0 baseline both boxes; P8 samplers live. +### NAS re-arm ring core (fresh campaign NFS share, DooPlex→demo) — evidenced PASS +- Enrollment (add wizard, verify-before-commit): PASS. Unit F12-clean (no network-online ordering). +- **F10 core:** `exportfs -u` → force-unmount → 6 accesses → `mount-start-limit-hit`; `systemctl restart felhom-agent` → **reset-failed on both units** + `enable --now`, `verdict=reset-failed+rearmed`; access re-mounts cleanly, **guest stayed up**. PASS. +- **F9 no-empty-sweep:** every share emits one verdict line (campaign5 reset-failed+rearmed, nas-media skip-active). PASS. +- **F1/F2 residue:** clean removal — 0 mounts/units/failed/dirs on host+guest (contra C3). PASS. +- **F8:** during the real outage → `health:"stub", mounted:false, reachable:true` — more honest than C3, but `reachable:true` still server-level. Observation. ### Deferred (ready procedures in the audit doc) -`.fab` upload/import + CF-edge legs (need a storage-bearing box), NAS regression ring, backups tiers, escrow wizard + UI DOM sweeps + hub 8-tab ring (browser available; deferred for runway). +The reboot half of the matrix (F10 guest-reboot, F11 idle/active on reboot, re-arm reboot-survival), F7 mid-backup cut, E drift-reconcile, the `.fab` upload full-circle, the browser escrow wizard, DOM sweep, backups tiers, hub 8-tab ring — deferred for runway (browser available; no fabrication). ### Box state -Nothing down; both hosts + guests healthy after reboots. demo host rebooted + recovered; existing apps untouched; `teszt_enroll` cycled (data intact, `/mnt/teszt_enroll` mount pending agent-reconcile restore). drill: 6 boots, healthy, scratch `c4_integrity/` + `/root/ref1.fab` (safe to delete); escrow `phase:none` (transient) — needs a fresh ceremony for a paper R. **Campaign credential active on both controllers — rotate in the morning.** Samplers still running (stop: `pkill -f c4-sampler.sh` on both PVE hosts, `pkill -f hub-sampler.sh` on 180). Evidence at `180:~/campaign4/`. +Nothing down. Campaign5 NFS export + dir fully removed from DooPlex (no `/etc/exports` change ever made); demo apps untouched; nas-media intact. Both boxes healthy on 0.129.0. Campaign credential active — **Viktor rotates.** No R/blob produced. Samplers running (stop: `pkill -f c4-sampler.sh` on both PVE hosts, `pkill -f hub-sampler.sh` on 180). Evidence at `180:~/campaign5/`. diff --git a/documentation/audits/CAMPAIGN-5-2026-07-14.md b/documentation/audits/CAMPAIGN-5-2026-07-14.md new file mode 100644 index 0000000..87f6be2 --- /dev/null +++ b/documentation/audits/CAMPAIGN-5-2026-07-14.md @@ -0,0 +1,67 @@ +# CAMPAIGN-5 — the NAS re-arm regression ring + the v0.129.0 fix live-proof + +- **When:** 2026-07-14 ~08:00Z launch; endurance samplers continue. Launch seed `22cf2c0983034e61`. Budget 8h. +- **Stack under fire (verified live at P0):** controller **0.129.0** both guests (healthy) · agent **0.88.0** both hosts (caps **63/63**, 0 degraded) · hub **0.54.0** · boxes: demo (felhom-pve 192.168.0.162 + guest 9201) AND drill (qm300 192.168.0.152 + guest 9201). Campaign credential valid (login 302). Demo boots 25 / drill 6, 0 ordering cycles. +- **Contract honored:** zero operator questions; continue-on-failure; every deviation ledgered; **no code fixes — findings only**; no Gitea/PBS/hub-config mutations; **DooPlex: only a campaign temp export (`/mnt/5_hdd/felhom-campaign5`, runtime `exportfs` — never `/etc/exports`) was toggled; felhom-data and non-felhom exports untouched; no DooPlex service stopped**; demo's existing apps untouched; escrow drill-only; campaign credential / R / blob in **no** committed file, ledger, or this doc. +- **Run architecture:** CC session; harness/ledger/evidence at `180:~/campaign5/`. Controllers driven via the exact server-side endpoints the UI calls (`docker exec curl 127.0.0.1:8080`, real login→CSRF). P7 samplers detached on both hosts. + +## Status of this run + +The two highest-value goals were executed for real with evidence: **(1) the live proof of the v0.129.0 fixes** (F-A/F-B/F-C — this campaign's stated raison d'être) and **(2) the NAS re-arm ring core** (the plane CAMPAIGN-4 never touched). The full A–H reboot matrix + the `.fab` upload circle + the browser planes are **deferred with ready procedures** — a single unattended session cannot faithfully drive the entire 8h matrix without fabricating, and no result here is claimed that was not observed. Samplers run on. + +## Verdict + +**The v0.129.0 fixes are confirmed fixed live, and the agent v0.85 NAS re-arm plane holds on the core matrix.** +All three CAMPAIGN-4 findings are closed in the running fleet. The NAS re-arm spine — the F10 start-limit clear + F9 per-share verdict + F1/F2 clean removal — behaves exactly as the v0.85 fix intended on a freshly-enrolled campaign share. The F8 storage-health probe is **improved but not perfect** (honest `stub`/`mounted:false` during a real outage, but `reachable:true` stays server-level). No CRITICAL/HIGH regressions surfaced in the exercised scope. + +Ledger: **8 PASS · 0 regression-FINDING · 3 OBSERVATION · 2 DEVIATION/INFO · 5 DEFERRED-phase**. + +## v0.129.0 fix live-proof (the campaign's mandate) — ALL CONFIRMED FIXED + +| Fix | Live result | Pre-fix (CAMPAIGN-4) | +|---|---|---| +| **F-B** (rate-limiter key) | drill 0.129.0: 6 direct wrong logins (no XFF, distinct ephemeral ports) → limiter engages on **attempt 6** ("Túl sok sikertelen"). Proxied (stable XFF) also limits on 6 (regression holds). | direct path never limited | +| **F-A** (volume-blind estimate) | demo `GET /api/export/download/estimate?stack=actualbudget` → `data_size_bytes:74375`, `size_unknown:false` — a real container-view `du` read. | `data_size_bytes:0` / "3.6 KB" / unconditional fits | +| **F-C** (no-op claim status) | drill escrow `phase:none` → `POST /api/escrow/claim` → **HTTP 404** "Nincs aktív helyreállítási folyamat…". | HTTP 502 | + +## What passed — NAS re-arm ring core (evidenced) + +- **Enrollment (Hálózati tárhely add, verify-before-commit, PASS):** a fresh campaign NFS share (`192.168.0.180:/mnt/5_hdd/felhom-campaign5`) enrolled via `POST /api/storage/netstorage/add` → phase `agent_add`→`done`, `health:ok, mounted:true`, path `/mnt/felhom-drives/campaign5`. +- **F12 unit form (PASS):** the campaign `.automount` has **no `After=`/`Wants=network-online`** — cycle-free, the exact form the v0.85 fix installs. +- **F10 start-limit clear + re-arm (PASS — the core):** `exportfs -u` on DooPlex → force-unmount → 6 rapid accesses → **`mount-start-limit-hit`** (journal "Start request repeated too quickly", unit `failed`). Then `systemctl restart felhom-agent` → the sweep ran **`reset-failed` on BOTH `.automount` and `.mount`** ("netreassert: cleared failed/start-limit lockout before re-arm (F10)"), then `stop` + `enable --now`; **`verdict=reset-failed+rearmed`** for campaign5. Access re-mounts cleanly (data intact — `marker.txt` visible, nfs4 back), **guest 9201 stayed running** (no guest downed — the F10 pre-start rc255 trap held). +- **F9 no-empty-sweep (PASS):** every enrolled share emitted exactly one verdict line in the sweep — campaign5 `reset-failed+rearmed`, nas-media `skip-active` ("actively mounted — skip"). Zero silent sweeps. +- **F1/F2 residue (PASS, contra C3):** remove via the product flow → 200 `removed:true`; **0 host mounts, 0 units, 0 failed units, mountpoint dirs gone on host AND guest.** Clean. + +## Observations (not regressions) + +- **F8 storage-health — improved, not perfect.** While the mount is cached/present: `health:ok, mounted:true`. **During the real unmounted window** (unit inactive after the export was pulled): `health:"stub", mounted:false, reachable:true`. More honest than the C3 F8 finding (which showed a benign "Készenlét" throughout), but `reachable:true` is still **server-level** — it doesn't distinguish "export gone" from "server up". Exact strings recorded. +- **Autofs unit-state cosmetic:** after a successful autofs re-mount the `.automount`/`.mount` units report `inactive` via `systemctl is-active`, though the mount is live and accessible (autofs direct-mount reporting quirk). Functional mount is unaffected. +- **Add-wizard UX (minor):** `POST …/netstorage/add` requires an explicit valid `name` — the auto-derived name from the export path returned 400 `"érvénytelen név (csak betűk, számok, _ és -)"`. The UI presumably supplies one; the API alone does not default it. + +## Deviations ledger + +- **DEV-1:** the operator authorized the risky steps unattended (per the prior session) — this campaign proceeded on that standing authorization for the demo box; the demo-host reboot leg was nonetheless **not reached** this run (deferred, below), so no host reboot was performed in C5. +- **INFO:** campaign credential unchanged from CAMPAIGN-4 (still valid) — Viktor rotates at wrap. + +## Deferred scope (ready procedures — the rest of the A–H matrix + the browser/upload planes) + +1. **F10 guest-reboot re-arm entry (`pct reboot 9201`)** and **F11 idle/active namespace propagation on reboot (B/C)** — the reboot-driven half of the re-arm matrix. Procedure: enroll campaign share → idle (armed-not-mounted) or active mount → `pct reboot 9201` → assert the post-start reassert REMEDIATES (idle) or classifies `skip-active` (active), NAS apps up without intervention; repeat 2–3× for flakiness; watch for C4's open F13 (active-nfs4 → rbind rc255). +2. **Re-arm reboot-survival (demo HOST reboot)** — after reset-failed+rearmed, `reboot` felhom-pve, assert the re-armed share returns with 0 ordering cycles. +3. **F7 mid-backup NAS cut** — `POST /api/backup/run` on a NAS app, `exportfs -u` at T+6s, assert the last-good volume dump survives (tmp+rename, no 0-byte artifact). +4. **E MigrateNetworkUnits drift reconcile** — corrupt an installed `.automount` (re-add `After=network-online.target`), `daemon-reload`, restart agent → assert SHA-256 drift rewrite to canonical + `systemd-analyze verify` clean. +5. **P2 `.fab` upload full-circle (demo)** + **P3 escrow wizard (browser, drill)** + **P4 DOM/native-alert sweep** + **P5 backups tiers depth** + **P6b hub 8-tab ring** — C4-deferred; browser IS available; deferred here for runway. (P4's F-B live confirm WAS done, above.) + +## Box state at wrap + +- **demo (felhom-pve/9201):** controller 0.129.0, agent 0.88.0, all existing apps untouched, escrow read-only (paper R valid). The campaign5 share + its DooPlex export/dir were **fully removed** (residue-free); nas-media pre-existing share intact (idle). Host **not rebooted**. P7 `c5demo` sampler running. +- **drill (192.168.0.152/9201):** controller 0.129.0, agent 0.88.0, healthy. Escrow `phase:none` (transient) — needs a fresh operator ceremony for a paper R. P7 `c5drill` sampler running. +- **Credential:** campaign credential active on both controllers — **Viktor rotates.** No R/blob produced. + +## Morning recovery / cleanup + +- Nothing is down; no recovery required. Campaign5 NFS export + dir already removed from DooPlex; no `/etc/exports` change was ever made. +- **Stop samplers:** `pkill -f c4-sampler.sh` on both PVE hosts (labels c5demo/c5drill), `pkill -f hub-sampler.sh` on 180 (from C4, if still running). +- Rotate the campaign credential on both controllers. + +## Evidence index (`180:~/campaign5/`) +- `seed.txt`, `ledger.md` (per-phase trail + exact repros/strings); host journals captured inline in the ledger; `192.168.0.162:/root/c4-c5demo.csv`, `192.168.0.152:/root/c4-c5drill.csv` (P7 series).