From add5be5bd56b6fa074e74379c7f157b2f4ab8d0b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 22 Jun 2026 22:06:00 +0200 Subject: [PATCH] docs(tests): test campaign #3 'NO MERCY' findings (brutal chaos/edge, demo 9201) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 7 phases. No silent-corruption, no path-escape/exfil found. Findings: S1 settings.json crash-loop (medium, no safe-defaults), F2 restore stack_name traversal not validated upfront (medium defense-in-depth, no escape observed), S3 corrupt quiesce marker silently ignored (low). All else fail-safe held. Phase4 time-chaos SKIP (unprivileged LXC shares host clock). No code changes. Demo restored to baseline. πŸ€– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) --- .../test-campaign-3-2026-06-22-findings.md | 90 +++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 documentation/tests/test-campaign-3-2026-06-22-findings.md diff --git a/documentation/tests/test-campaign-3-2026-06-22-findings.md b/documentation/tests/test-campaign-3-2026-06-22-findings.md new file mode 100644 index 0000000..49377af --- /dev/null +++ b/documentation/tests/test-campaign-3-2026-06-22-findings.md @@ -0,0 +1,90 @@ +# Test campaign #3 β€” "NO MERCY" brutal chaos/edge β€” 2026-06-22 (demo 9201) + +> **Status: empirical findings from an unattended brutal chaos campaign (2026-06-22).** Push the +> system into unintended states; verify it handles-or-fails-safe, then recovers. Every break carried a +> timed auto-revert; the Phase-0 verified PBS backup was the floor. ctrl **v0.75.0** + agent v0.39.0. +> Demo restored to baseline (25 containers, 0 unhealthy) and healthy at the end. Per-check raw evidence: +> `felhom-controller/TEST-REPORT.md`. **No silent-corruption and no path-escape/exfil found** (the two +> highest-severity classes). **No code changes shipped** β€” findings are behaviours logged for supervised fix. + +## Result matrix +| Phase | Result | +|---|---| +| 0 baseline + verified PBS backup floor | PASS (gate open) | +| 1 resource starvation (R1 RAM 91%, R2 settings-atomicity, R3 rootfs 94%, R4 mem-gate) | PASS (fail-safe held) | +| 2 state corruption (S1 settings, S2 app.yaml, S3 quiesce marker, S4 symlink) | PASS + 1 medium (S1) | +| 3 concurrency (C1 4-way ops, C2 drive-flapΓ—5, C3 kill-agent-mid-backup) | PASS | +| 4 time chaos (clock jumps) | SKIP (architectural β€” see below) | +| 5 network partitions (N1 tunnel block, N2 PBS block mid-backup) | PASS | +| 6 security fuzzing (F1 JSON, F2 traversal, F3 hostile compose, F4 weird filenames) | PASS (no escape) + 1 medium (F2) | +| 7 brutal recovery (B1 dockerd restart, B2 kill-9 ctrl, B3 reboot+flash-detached, B4 host reboot) | PASS (B1-B3); B4 SKIP | + +## Findings, ranked by real-world severity + +### 1. [MEDIUM] Corrupt `settings.json` β†’ controller FATAL crash-loop (Phase 2 / S1) +A truncated/invalid `settings.json` makes the controller `[FATAL] Failed to load settings … unexpected +end of JSON input` and **crash-loop** (RestartCount climbs via docker's restart-manager) β€” **no +safe-defaults fallback**; the management plane is down until the file is fixed. It is *not silent* +(FATAL is logged). Apps + tunnel keep running (control/data separation). **Contrast:** per-stack +`app.yaml` corruption (S2) is handled gracefully β€” controller logs `[WARN] LoadAppConfig: yaml: …`, +marks that app, keeps serving others, no crash. **Recommend:** make the settings.json load path match +app.yaml's β€” log a WARN + load safe defaults (or boot read-only), not `[FATAL]`. + +### 2. [MEDIUM, defense-in-depth] Restore `stack_name` not validated against path traversal (Phase 6 / F2) +`POST /backup/restore` proceeds with a traversal `stack_name` (`../../../etc`) into +`RestoreFromRecoveryUnit`/`RestoreApp` (`GetAppDrivePath` returns the default `/mnt/sys_drive`). +**No escape occurred** β€” `/etc/passwd` intact, no `/opt/etc`/`/etc/passwd/` artifacts, no `/etc` writes +β€” **only because** downstream `StopStack`/`StartStack` are map-based (`stack "../../../etc" not found`) +and there's no recovery-unit/volumes for a bogus name (no-op). The security boundary held, but by +luck of downstream behaviour, not by input validation. A future code path that built a filesystem path +from the raw name could escape. **Recommend:** explicit `stack_name` validation at the restore handler +(reject `/`, `..`, NUL). (Storage `where=` IS validated β€” `gateWhere`: `path.Clean`+`HasPrefix("/mnt/")` +rejected all 5 traversals.) + +### 3. [LOW] Corrupt quiesce marker silently ignored (Phase 2 / S3) +A bad-JSON `quiesce-state.json` does **not** panic the controller at startup (good β€” stacks not +stranded), but the unparseable marker is **neither logged nor cleared/quarantined**. A real +corrupted-mid-quiesce marker would silently skip stack-recovery. **Recommend:** log + quarantine a +marker that fails to parse. + +### 4. [INFO] Boot-ordering, re-confirmed (Phase 7 / B3) +On a guest reboot with a drive detached, docker's boot-restore attempts drive-backed bind-source +`mkdir` before mounts converge β€” but the **unprivileged-LXC mapping denies it** (2 denied attempts, +**no rootfs shadow dirs**), and the v0.75 mountpoint-gate covers the controller's belt/FileBrowser. +Benign but noisy. (Tracked from the v0.75 task.) + +## What held (highlights) +- **No path-escape / no /etc exfil** anywhere: storage `where=` validation (F2), symlink-in-backup not + followed (S4 β€” rsync preserved `EVIL_ETC -> /etc`, no contents copied), restore-traversal no-op (F2). +- **Resource pressure** (R1 91% RAM, R3 94% rootfs): controller + 25 apps stayed healthy, no OOM, no + daemon restart, **settings.json stayed valid** (atomic tmp+rename, R2). +- **Concurrency**: single-flight serialized 4-way op storm (C1), drive flapΓ—5 converged with the v0.75 + gate firing 5 clean skips and **0 permission-denied** (C2), agent-kill-mid-backup unwound the quiesce + with no stack stranded (C3). +- **Partitions**: tunnel block β†’ apps serve locally + reconnect (N1); PBS block mid-backup β†’ backup + fails safe + quiesce unwinds + next backup succeeds (N2). +- **Recovery**: dockerd restart β†’ all back <8s (B1); `kill -9` controller β†’ `unless-stopped` + auto-recovers in 8s (B2, re-confirms Finding #1); reboot+flash-detached β†’ apps held, **no shadow + dirs**, reconnect recovers (B3). + +## Phase 4 SKIP β€” time chaos not safely isolable (architectural) +The guest is an **unprivileged LXC**: it cannot set its own clock (`date -s` β†’ `Operation not +permitted`) and shares the host's `CLOCK_REALTIME` (time namespaces don't isolate the wall clock). The +only way to jump the guest wall-clock is to change the **host** clock, which would risk the cloudflared +tunnel cert, the agent leaf-cert pin, and **agentβ†’DooPlex hub/PBS TLS auth** (Β§0.1 forbids degrading +DooPlex). Proper venue: a dedicated throwaway VM with its own clock, or an injectable-clock unit test. + +## Deferred / supervised / implementation +- **Fix S1** (settings.json: safe-defaults + WARN instead of FATAL crash-loop) β€” highest priority. +- **Fix F2** (explicit restore `stack_name` traversal validation). +- **Fix S3** (log + quarantine a corrupt quiesce marker). +- **Time chaos** (Phase 4) on a dedicated VM / injectable clock. +- **Host reboot** (B4) β€” supervised (no physical recovery unattended). +- `.fab` import compose-fuzzing (F3) β€” the only customer compose vector; not exercised. + +## Cleanup +9201 running, 25 containers, 0 unhealthy; controller + cloudflared healthy; agent active; both drives +mounted; rootfs 4%/29 G free. No leftover iptables rules (host 8007, guest DOCKER-USER both 0); all +campaign timed-guards killed; corrupted files restored (settings valid, app.yaml restored, quiesce +marker removed, symlinks removed); F4 test files removed; no 9300/scratch/loopback created. Demo = +Phase-0 baseline.