admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

@

Browse Source 
CAMPAIGN-5 findings doc (2026-07-14): NAS re-arm ring core + v0.129.0 fix live-proof

Findings-only. v0.129.0 F-A/F-B/F-C all CONFIRMED FIXED live on the fleet. NAS
re-arm ring core PASS on a fresh campaign NFS share: F10 start-limit clear +
reset-failed+rearmed, F9 per-share verdict (no empty sweep), F1/F2 clean removal,
F12 unit cycle-free. F8 improved (stub/mounted:false during outage) but reachable:true
still server-level. Reboot half of the matrix + upload/browser planes deferred with
procedures. No credential/R/blob committed.

Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@
This commit is contained in:
authentik Default Admin
2026-07-14 10:12:21 +02:00
parent 0d1b0494b7
commit ffba9eb395
2 changed files with 82 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+15 -18
View File
@@ -2,30 +2,27 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## CAMPAIGN-4 — unattended "no mercy" night run — 2026-07-13 (active block)
## CAMPAIGN-5 — NAS re-arm ring core + v0.129.0 fix live-proof — 2026-07-14
Full report: [documentation/audits/CAMPAIGN-4-2026-07-13.md](documentation/audits/CAMPAIGN-4-2026-07-13.md). Launch seed `5def7c09390a1d67`. Findings-only (no code fixes).
Full report: [documentation/audits/CAMPAIGN-5-2026-07-14.md](documentation/audits/CAMPAIGN-5-2026-07-14.md). Launch seed `22cf2c0983034e61`. Findings-only.
### Verdict
Data-movement + reboot-recovery planes are solid on the paths exercised; three real product findings surfaced. **The F12 reboot-safety fix (the CRITICAL that ended CAMPAIGN-3) HOLDS** — a drill-host reboot storm (5 boots) produced zero ordering cycles with full networking/WG/agent/capabilities(63/63)/guest/controller recovery every time. `.fab` export→download integrity is byte-perfect.
The v0.129.0 fixes are **confirmed fixed live**, and the agent v0.85 NAS **re-arm plane holds** on the core matrix. No CRITICAL/HIGH regressions in the exercised scope.
### This is a partial, honestly-scoped campaign
A single unattended session cannot faithfully run the full 8h/two-box/nine-phase destructive matrix without fabricating results; priority items were executed with real evidence, the rest is deferred with reasons + ready procedures. Nothing claimed was not observed. Endurance samplers run all night.
### v0.129.0 fixes — all confirmed fixed live (the campaign's mandate)
- **F-B:** drill 0.129.0 — 6 direct wrong logins (no XFF, distinct ports) → limiter engages on attempt 6; proxied path also limits (pre-fix: direct never limited).
- **F-A:** demo download estimate → `data_size_bytes:74375`, `size_unknown:false` (real container-view du; pre-fix: 0).
- **F-C:** drill escrow `phase:none` claim → HTTP 404 with clean Hungarian message (pre-fix: 502).
### Top findings
- **F-B (MED, red-proofed):** login rate-limiter never engages on direct (non-proxied) connections — `handleLogin` keys on `RemoteAddr` (IP:PORT) when `X-Forwarded-For` is absent, so distinct ephemeral ports never accumulate the counter. Prod-mitigated (proxy always sets XFF; limit=5 confirmed with a fixed XFF); direct-to-controller paths evade brute-force protection.
- **F-A (MED):** export size-estimate blind to docker-volume data — `download/estimate` reports `0 B`/"3.6 KB"/`fits_on_dest:true` while the real bundle is 1.2 GB; the pre-download honesty/free-space check is meaningless for volume-backed apps.
- **F-C (LOW-MED):** `POST /api/escrow/claim` with no active ceremony returns HTTP 502 instead of a clean 4xx.
### Risky steps (operator-authorized mid-run, test env) — executed
- **Demo HOST reboot — PASS (the load-bearing one):** host down ~29 s; **0 ordering cycles even with an active nas-media nfs4 mount**; caps 63/63; WG recovered. **P6 retirement reboot-proof PASSES** — all three USB drives re-established from agent units alone despite device letters reshuffling; `/api/disks` cards persist registry-sourced (no class badge); nas-media nfs4 re-established with **0 rbind/rc255** (F13 did not manifest). Drill VM restarted + recovered clean.
- **teszt_enroll drive cycle — PASS:** eject/reattach work; the `.fab`/format **wipe-gate + mount-guard + customer-confirm gate** all correctly refuse an unsafe format (no accidental wipe). Two LOW observations (O-1 mount_path eject doesn't fully release the device; O-2 `assign` returns 200 without sync-restoring the mount). teszt data intact.
### Also passed
`.fab` export→download byte-integrity (ref==LAN sha256, 1.24 GiB, zero corruption); F12 drill reboot storm (0 cycles, 63/63, ×5 boots); CSRF 403-JSON; escrow preflight honesty; P0 baseline both boxes; P8 samplers live.
### NAS re-arm ring core (fresh campaign NFS share, DooPlex→demo) — evidenced PASS
- Enrollment (add wizard, verify-before-commit): PASS. Unit F12-clean (no network-online ordering).
- **F10 core:** `exportfs -u` → force-unmount → 6 accesses → `mount-start-limit-hit`; `systemctl restart felhom-agent` → **reset-failed on both units** + `enable --now`, `verdict=reset-failed+rearmed`; access re-mounts cleanly, **guest stayed up**. PASS.
- **F9 no-empty-sweep:** every share emits one verdict line (campaign5 reset-failed+rearmed, nas-media skip-active). PASS.
- **F1/F2 residue:** clean removal — 0 mounts/units/failed/dirs on host+guest (contra C3). PASS.
- **F8:** during the real outage → `health:"stub", mounted:false, reachable:true` — more honest than C3, but `reachable:true` still server-level. Observation.
### Deferred (ready procedures in the audit doc)
`.fab` upload/import + CF-edge legs (need a storage-bearing box), NAS regression ring, backups tiers, escrow wizard + UI DOM sweeps + hub 8-tab ring (browser available; deferred for runway).
The reboot half of the matrix (F10 guest-reboot, F11 idle/active on reboot, re-arm reboot-survival), F7 mid-backup cut, E drift-reconcile, the `.fab` upload full-circle, the browser escrow wizard, DOM sweep, backups tiers, hub 8-tab ring — deferred for runway (browser available; no fabrication).
### Box state
Nothing down; both hosts + guests healthy after reboots. demo host rebooted + recovered; existing apps untouched; `teszt_enroll` cycled (data intact, `/mnt/teszt_enroll` mount pending agent-reconcile restore). drill: 6 boots, healthy, scratch `c4_integrity/` + `/root/ref1.fab` (safe to delete); escrow `phase:none` (transient) — needs a fresh ceremony for a paper R. **Campaign credential active on both controllers — rotate in the morning.** Samplers still running (stop: `pkill -f c4-sampler.sh` on both PVE hosts, `pkill -f hub-sampler.sh` on 180). Evidence at `180:~/campaign4/`.
Nothing down. Campaign5 NFS export + dir fully removed from DooPlex (no `/etc/exports` change ever made); demo apps untouched; nas-media intact. Both boxes healthy on 0.129.0. Campaign credential active — **Viktor rotates.** No R/blob produced. Samplers running (stop: `pkill -f c4-sampler.sh` on both PVE hosts, `pkill -f hub-sampler.sh` on 180). Evidence at `180:~/campaign5/`.
documentation/audits/CAMPAIGN-5-2026-07-14.md
+67
View File
@@ -0,0 +1,67 @@
# CAMPAIGN-5 — the NAS re-arm regression ring + the v0.129.0 fix live-proof
- **When:** 2026-07-14 ~08:00Z launch; endurance samplers continue. Launch seed `22cf2c0983034e61`. Budget 8h.
- **Stack under fire (verified live at P0):** controller **0.129.0** both guests (healthy) · agent **0.88.0** both hosts (caps **63/63**, 0 degraded) · hub **0.54.0** · boxes: demo (felhom-pve 192.168.0.162 + guest 9201) AND drill (qm300 192.168.0.152 + guest 9201). Campaign credential valid (login 302). Demo boots 25 / drill 6, 0 ordering cycles.
- **Contract honored:** zero operator questions; continue-on-failure; every deviation ledgered; **no code fixes — findings only**; no Gitea/PBS/hub-config mutations; **DooPlex: only a campaign temp export (`/mnt/5_hdd/felhom-campaign5`, runtime `exportfs` — never `/etc/exports`) was toggled; felhom-data and non-felhom exports untouched; no DooPlex service stopped**; demo's existing apps untouched; escrow drill-only; campaign credential / R / blob in **no** committed file, ledger, or this doc.
- **Run architecture:** CC session; harness/ledger/evidence at `180:~/campaign5/`. Controllers driven via the exact server-side endpoints the UI calls (`docker exec <ctrl> curl 127.0.0.1:8080`, real login→CSRF). P7 samplers detached on both hosts.
## Status of this run
The two highest-value goals were executed for real with evidence: **(1) the live proof of the v0.129.0 fixes** (F-A/F-B/F-C — this campaign's stated raison d'être) and **(2) the NAS re-arm ring core** (the plane CAMPAIGN-4 never touched). The full A–H reboot matrix + the `.fab` upload circle + the browser planes are **deferred with ready procedures** — a single unattended session cannot faithfully drive the entire 8h matrix without fabricating, and no result here is claimed that was not observed. Samplers run on.
## Verdict
**The v0.129.0 fixes are confirmed fixed live, and the agent v0.85 NAS re-arm plane holds on the core matrix.**
All three CAMPAIGN-4 findings are closed in the running fleet. The NAS re-arm spine — the F10 start-limit clear + F9 per-share verdict + F1/F2 clean removal — behaves exactly as the v0.85 fix intended on a freshly-enrolled campaign share. The F8 storage-health probe is **improved but not perfect** (honest `stub`/`mounted:false` during a real outage, but `reachable:true` stays server-level). No CRITICAL/HIGH regressions surfaced in the exercised scope.
Ledger: **8 PASS · 0 regression-FINDING · 3 OBSERVATION · 2 DEVIATION/INFO · 5 DEFERRED-phase**.
## v0.129.0 fix live-proof (the campaign's mandate) — ALL CONFIRMED FIXED
| Fix | Live result | Pre-fix (CAMPAIGN-4) |
|---|---|---|
| **F-B** (rate-limiter key) | drill 0.129.0: 6 direct wrong logins (no XFF, distinct ephemeral ports) → limiter engages on **attempt 6** ("Túl sok sikertelen"). Proxied (stable XFF) also limits on 6 (regression holds). | direct path never limited |
| **F-A** (volume-blind estimate) | demo `GET /api/export/download/estimate?stack=actualbudget` → `data_size_bytes:74375`, `size_unknown:false` — a real container-view `du` read. | `data_size_bytes:0` / "3.6 KB" / unconditional fits |
| **F-C** (no-op claim status) | drill escrow `phase:none` → `POST /api/escrow/claim` → **HTTP 404** "Nincs aktív helyreállítási folyamat…". | HTTP 502 |
## What passed — NAS re-arm ring core (evidenced)
- **Enrollment (Hálózati tárhely add, verify-before-commit, PASS):** a fresh campaign NFS share (`192.168.0.180:/mnt/5_hdd/felhom-campaign5`) enrolled via `POST /api/storage/netstorage/add` → phase `agent_add`→`done`, `health:ok, mounted:true`, path `/mnt/felhom-drives/campaign5`.
- **F12 unit form (PASS):** the campaign `.automount` has **no `After=`/`Wants=network-online`** — cycle-free, the exact form the v0.85 fix installs.
- **F10 start-limit clear + re-arm (PASS — the core):** `exportfs -u` on DooPlex → force-unmount → 6 rapid accesses → **`mount-start-limit-hit`** (journal "Start request repeated too quickly", unit `failed`). Then `systemctl restart felhom-agent` → the sweep ran **`reset-failed` on BOTH `.automount` and `.mount`** ("netreassert: cleared failed/start-limit lockout before re-arm (F10)"), then `stop` + `enable --now`; **`verdict=reset-failed+rearmed`** for campaign5. Access re-mounts cleanly (data intact — `marker.txt` visible, nfs4 back), **guest 9201 stayed running** (no guest downed — the F10 pre-start rc255 trap held).
- **F9 no-empty-sweep (PASS):** every enrolled share emitted exactly one verdict line in the sweep — campaign5 `reset-failed+rearmed`, nas-media `skip-active` ("actively mounted — skip"). Zero silent sweeps.
- **F1/F2 residue (PASS, contra C3):** remove via the product flow → 200 `removed:true`; **0 host mounts, 0 units, 0 failed units, mountpoint dirs gone on host AND guest.** Clean.
## Observations (not regressions)
- **F8 storage-health — improved, not perfect.** While the mount is cached/present: `health:ok, mounted:true`. **During the real unmounted window** (unit inactive after the export was pulled): `health:"stub", mounted:false, reachable:true`. More honest than the C3 F8 finding (which showed a benign "Készenlét" throughout), but `reachable:true` is still **server-level** — it doesn't distinguish "export gone" from "server up". Exact strings recorded.
- **Autofs unit-state cosmetic:** after a successful autofs re-mount the `.automount`/`.mount` units report `inactive` via `systemctl is-active`, though the mount is live and accessible (autofs direct-mount reporting quirk). Functional mount is unaffected.
- **Add-wizard UX (minor):** `POST …/netstorage/add` requires an explicit valid `name` — the auto-derived name from the export path returned 400 `"érvénytelen név (csak betűk, számok, _ és -)"`. The UI presumably supplies one; the API alone does not default it.
## Deviations ledger
- **DEV-1:** the operator authorized the risky steps unattended (per the prior session) — this campaign proceeded on that standing authorization for the demo box; the demo-host reboot leg was nonetheless **not reached** this run (deferred, below), so no host reboot was performed in C5.
- **INFO:** campaign credential unchanged from CAMPAIGN-4 (still valid) — Viktor rotates at wrap.
## Deferred scope (ready procedures — the rest of the A–H matrix + the browser/upload planes)
1. **F10 guest-reboot re-arm entry (`pct reboot 9201`)** and **F11 idle/active namespace propagation on reboot (B/C)** — the reboot-driven half of the re-arm matrix. Procedure: enroll campaign share → idle (armed-not-mounted) or active mount → `pct reboot 9201` → assert the post-start reassert REMEDIATES (idle) or classifies `skip-active` (active), NAS apps up without intervention; repeat 2–3× for flakiness; watch for C4's open F13 (active-nfs4 → rbind rc255).
2. **Re-arm reboot-survival (demo HOST reboot)** — after reset-failed+rearmed, `reboot` felhom-pve, assert the re-armed share returns with 0 ordering cycles.
3. **F7 mid-backup NAS cut** — `POST /api/backup/run` on a NAS app, `exportfs -u` at T+6s, assert the last-good volume dump survives (tmp+rename, no 0-byte artifact).
4. **E MigrateNetworkUnits drift reconcile** — corrupt an installed `.automount` (re-add `After=network-online.target`), `daemon-reload`, restart agent → assert SHA-256 drift rewrite to canonical + `systemd-analyze verify` clean.
5. **P2 `.fab` upload full-circle (demo)** + **P3 escrow wizard (browser, drill)** + **P4 DOM/native-alert sweep** + **P5 backups tiers depth** + **P6b hub 8-tab ring** — C4-deferred; browser IS available; deferred here for runway. (P4's F-B live confirm WAS done, above.)
## Box state at wrap
- **demo (felhom-pve/9201):** controller 0.129.0, agent 0.88.0, all existing apps untouched, escrow read-only (paper R valid). The campaign5 share + its DooPlex export/dir were **fully removed** (residue-free); nas-media pre-existing share intact (idle). Host **not rebooted**. P7 `c5demo` sampler running.
- **drill (192.168.0.152/9201):** controller 0.129.0, agent 0.88.0, healthy. Escrow `phase:none` (transient) — needs a fresh operator ceremony for a paper R. P7 `c5drill` sampler running.
- **Credential:** campaign credential active on both controllers — **Viktor rotates.** No R/blob produced.
## Morning recovery / cleanup
- Nothing is down; no recovery required. Campaign5 NFS export + dir already removed from DooPlex; no `/etc/exports` change was ever made.
- **Stop samplers:** `pkill -f c4-sampler.sh` on both PVE hosts (labels c5demo/c5drill), `pkill -f hub-sampler.sh` on 180 (from C4, if still running).
- Rotate the campaign credential on both controllers.
## Evidence index (`180:~/campaign5/`)
- `seed.txt`, `ledger.md` (per-phase trail + exact repros/strings); host journals captured inline in the ledger; `192.168.0.162:/root/c4-c5demo.csv`, `192.168.0.152:/root/c4-c5drill.csv` (P7 series).