64 lines
8.4 KiB
Markdown
64 lines
8.4 KiB
Markdown
# TEST-REPORT — Test campaign #3: NO MERCY (N100 / guest 9201)
|
||
|
||
**Run start:** 2026-06-22 (CC, unattended). Brutal chaos/edge — push to unintended states, verify
|
||
handles-or-fails-safe, then recovers. Every break carries a timed auto-revert; Phase-0 backup is the floor.
|
||
**Legend:** PASS / FAIL / SKIP + raw evidence. Each chaos line: break→detect→recover→verify.
|
||
(Campaigns #1/#2 + diagnoses in git history + `felhom.eu/documentation/tests/`.)
|
||
|
||
## Phase 0 — Baseline + floor — **PASS (gate OPEN)**
|
||
- ctrl **v0.75.0** healthy; agent v0.39.0; **25 containers, 0 unhealthy**; rootfs `/` 4%, `/var/lib/docker` 8%, drives 1%; mem **8.8 Gi available**.
|
||
- Floor: PBS `felhom-pbs:backup/ct/9201/2026-06-22T19:01:56Z` (success, crash-consistent, verified).
|
||
|
||
## Phase 1 — Resource starvation — **PASS (fail-safe held)**
|
||
|
||
| # | Break (timed-revert) | Result | Evidence |
|
||
|---|---|---|---|
|
||
| R1 | python hog ~8 GB (~91% RAM), 75s | **PASS** | mid-stress: 10Gi used / 1.1Gi avail — controller **healthy**, 25/0 unhealthy; post: **dockerd NRestarts=0, no OOM** (alloc fit via cache eviction), recovered to 8.9Gi. Backstop pkill guard armed. |
|
||
| R2 | settings save on full disk | **PASS (code-verified)** | `settings.save()` = atomic write-`.tmp`-then-`os.Rename` (`settings.go:236-261`) → a disk-full fails at WriteFile, original untouched. Live disk-full not reproducible: settings.json is on the 252 GB docker vol. |
|
||
| R3 | fill rootfs (holds `/mnt/sys_drive` backups) to 94% (timed rm @150s) | **PASS** | controller stayed healthy (it's on the separate 252 GB vol); DB-dump + Tier-2 (8 apps, immich 155MB) **succeeded**; **settings.json stayed valid JSON** (no corruption). Reverted → rootfs 4%. (Literal-full refusal not pushed — near-0 rootfs risks wedging the guest OS unattended; the 2 GB margin had room so the gate didn't need to refuse.) |
|
||
| R4 | memory gate hard-block | **PASS (code-verified)** | `deploy.go:186` is a **hard block** (returns an error on `committed+new > usable`), using **committed-memory** accounting (sum of deployed `mem_request`), reserved 384 MB, usable 11904 MB. Live committed ≈ 4.4 GB → ~7.5 GB headroom, so tripping needs mass-deploying ~7.5 GB of requests (impractical/risky unattended); the branch + math are verified. |
|
||
## Phase 2 — State corruption [HIGH SEV] — **mostly PASS; 1 medium finding (S1)**
|
||
|
||
| # | Break (backed up first) | Result | Evidence |
|
||
|---|---|---|---|
|
||
| S1 | truncate settings.json → restart | **⚠ FINDING (medium)** | controller **crash-loops**: `[FATAL] Failed to load settings … unexpected end of JSON input`, RestartCount=7, restarting. **No safe-defaults fallback** — a corrupt settings.json takes the management plane down. *Not silent* (FATAL logged — better than the worst case). Restore → running/healthy, 3 storage_paths back. (Docker restart-manager cycling it re-confirms Finding #1.) |
|
||
| S2 | garbage in uptime-kuma `app.yaml` → rescan | **PASS** | controller stays **healthy** (25/0), logs `[WARN] LoadAppConfig: yaml: … did not find expected key` (not silent, not fatal), other apps unaffected. Restore → `running/deployed`. **Contrast with S1:** per-stack app.yaml corruption is graceful; settings.json is fatal. |
|
||
| S3 | corrupt quiesce marker (bad JSON) → restart | **PASS (minor finding)** | no panic/crash-loop, controller healthy, **stacks not stranded** (restarts=0). But the unparseable marker is **silently ignored** (no log) **and left in place** (not cleared/quarantined) — a real corrupted-mid-quiesce marker would skip recovery with no signal. |
|
||
| S4 | symlink `→ /etc` in userdata/appdata → backup | **PASS (security holds)** 🔒 | Tier-2 rsync **preserved the symlink** (`EVIL_ETC -> /etc`), did **not** follow it; **/etc contents did NOT leak** into the backup (no passwd/shadow). No path-escape/exfil. |
|
||
## Phase 3 — Concurrency storms — **PASS**
|
||
|
||
| # | Break | Result | Evidence |
|
||
|---|---|---|---|
|
||
| C1 | deploy + backup + restore + git-sync simultaneously | **PASS** | backup/run + tier2 both single-flighted ("Mentés már folyamatban"); restore mutex'd (302); sync ran independently ("nincs változás"); after settle **no stuck flag** (`running:false`), no deadlock, 25/0 healthy |
|
||
| C2 | rapid felhom-usb flap ×5 in ~10s | **PASS** | converged → felhom-usb **MOUNTED**, disconnected mark `None` (not stuck); **0 `permission denied`** during the flap; the v0.75 mountpoint-gate fired **5 clean skips** ("not mounted") in the disconnected windows; 25/0 |
|
||
| C3 | kill `felhom-agent` mid-quiesce-backup | **PASS** | quiesce stopped 14 stacks → agent killed → controller `[quiesce] unquiescing (backup start failed): restarting 14 stack(s)` (clean error + defer-unwind, **no stack stranded**); agent restarted → `/api/disks` OK; **idle sockets→8443 = 1** (v0.74 bound holds); recovered 25/0 |
|
||
## Phase 4 — Time chaos — **SKIPPED (not safely isolable)**
|
||
|
||
T1 (+25h) / T2 (−2h): the guest is an **unprivileged LXC** — it **cannot set its own clock**
|
||
(`date -s` → `Operation not permitted`, no CAP_SYS_TIME) and shares the host's `CLOCK_REALTIME` (time
|
||
namespaces don't isolate the wall clock). The only way to jump the guest's wall-clock is to change the
|
||
**host** (felhom-pve) clock, which would risk: cloudflared tunnel cert time-validation (demo down),
|
||
the agent's leaf-cert pin to PVE/PBS, and **agent→DooPlex hub/PBS TLS auth** (§0.1 forbids degrading
|
||
DooPlex). NTP is active on the host. **Not safely isolable to the guest unattended → SKIP.** Proper
|
||
venue: a dedicated throwaway VM with its own clock, or an injectable-clock unit test. **Deferred to
|
||
supervised / implementation.**
|
||
## Phase 5 — Network partitions (guest/host-side timed rules) — **PASS**
|
||
|
||
| # | Break (timed-revert) | Result | Evidence |
|
||
|---|---|---|---|
|
||
| N1 | block cloudflared egress (iptables DOCKER-USER, 130s) | **PASS** | tunnel partitioned: cloudflared `timeout: no recent network activity` + `Retrying connection` (no crash); **apps serve locally** (romm → HTTP 200); controller healthy 25/0. Unblock → `Registered tunnel connection`; no leftover rule. |
|
||
| N2 | block agent→PBS `192.168.0.180:8007` mid-quiesce-backup (N100-side OUTPUT rule per §0.1, 150s) | **PASS** | quiesce stopped 14 stacks → `[quiesce] backup job failed` (vzdump phase=failed, clean error) → `[quiesce] unquiescing (backup failed): restarting 14 stack(s)` (**fails safe, no stack stranded**). Unblock → next PBS backup `success:true`; 25/0; no leftover rule. (Rule was N100-side; DooPlex untouched.) |
|
||
## Phase 6 — Input/security fuzzing [HIGH SEV] — **PASS (no escape) + 1 medium defense-in-depth finding**
|
||
|
||
| # | Break | Result | Evidence |
|
||
|---|---|---|---|
|
||
| F1 | malformed/oversized/unknown-field JSON | **PASS** | not-json → **400**, empty → 400, 1MB → 400, nested-object-where → 400; unknown fields safely ignored (502 from a downstream non-existent mount, **no partial mutation**); controller **healthy** (no panic) |
|
||
| F2 | path traversal (storage `where`, restore `stack_name`) | **PASS (no escape) 🔒 + FINDING** | **storage:** all 5 traversals (`/etc`, `../../etc`, `/mnt/../etc`, `/mnt/felhom-drives/../../../etc`, `/etc/shadow`) **rejected** by `gateWhere` (`path.Clean`+`HasPrefix("/mnt/")`). **restore:** the boundary HELD — `/etc/passwd` **intact**, no `/opt/etc`/`/etc/passwd/` artifacts, no `/etc` writes — but **the restore handler does NOT reject a traversal `stack_name` upfront**: `RestoreFromRecoveryUnit("../../../etc")` proceeded (`GetAppDrivePath` → default `/mnt/sys_drive`), saved only by downstream **map-based** `StopStack`/`StartStack` (`stack "../../../etc" not found`) + no recovery-unit/volumes (no-op). **Defense-in-depth gap — recommend explicit `stack_name` validation at the restore handler.** (My own batched test looped these restores ×90s each, briefly holding the mutex — a test artifact that cleared when stopped, NOT a stuck-flag bug.) |
|
||
| F3 | hostile compose on a scratch app | **N/A (good property)** | the standard deploy renders only from the git-synced **catalog** — there is **no customer-facing arbitrary-compose injection vector**. (`.fab` import is the only path; not fuzzed — lower priority.) |
|
||
| F4 | weird userdata filenames (unicode, spaces, `(N)`, 200-char) | **PASS** | files created + Tier-2 backup ran clean ("Tier 2 run complete: 8 apps"), no crash/panic, 25/0. (`(N)` merge/dedup is migration-specific — `migrate.go` unit-tested, not live-run here.) |
|
||
## Phase 7 — Brutal recovery — _pending_
|
||
|
||
---
|
||
## Findings (ranked by severity) — _filled at end_
|
||
## Cleanup confirmation — _filled at end_
|