felhom-controller/TEST-REPORT.md

# TEST-REPORT — Test campaign #3: NO MERCY (N100 / guest 9201)

**Run start:** 2026-06-22 (CC, unattended). Brutal chaos/edge — push to unintended states, verify
handles-or-fails-safe, then recovers. Every break carries a timed auto-revert; Phase-0 backup is the floor.
**Legend:** PASS / FAIL / SKIP + raw evidence. Each chaos line: break→detect→recover→verify.
(Campaigns #1/#2 + diagnoses in git history + `felhom.eu/documentation/tests/`.)

## Phase 0 — Baseline + floor — **PASS (gate OPEN)**
- ctrl **v0.75.0** healthy; agent v0.39.0; **25 containers, 0 unhealthy**; rootfs `/` 4%, `/var/lib/docker` 8%, drives 1%; mem **8.8 Gi available**.
- Floor: PBS `felhom-pbs:backup/ct/9201/2026-06-22T19:01:56Z` (success, crash-consistent, verified).

## Phase 1 — Resource starvation — **PASS (fail-safe held)**

| # | Break (timed-revert) | Result | Evidence |
|---|---|---|---|
| R1 | python hog ~8 GB (~91% RAM), 75s | **PASS** | mid-stress: 10Gi used / 1.1Gi avail — controller **healthy**, 25/0 unhealthy; post: **dockerd NRestarts=0, no OOM** (alloc fit via cache eviction), recovered to 8.9Gi. Backstop pkill guard armed. |
| R2 | settings save on full disk | **PASS (code-verified)** | `settings.save()` = atomic write-`.tmp`-then-`os.Rename` (`settings.go:236-261`) → a disk-full fails at WriteFile, original untouched. Live disk-full not reproducible: settings.json is on the 252 GB docker vol. |
| R3 | fill rootfs (holds `/mnt/sys_drive` backups) to 94% (timed rm @150s) | **PASS** | controller stayed healthy (it's on the separate 252 GB vol); DB-dump + Tier-2 (8 apps, immich 155MB) **succeeded**; **settings.json stayed valid JSON** (no corruption). Reverted → rootfs 4%. (Literal-full refusal not pushed — near-0 rootfs risks wedging the guest OS unattended; the 2 GB margin had room so the gate didn't need to refuse.) |
| R4 | memory gate hard-block | **PASS (code-verified)** | `deploy.go:186` is a **hard block** (returns an error on `committed+new > usable`), using **committed-memory** accounting (sum of deployed `mem_request`), reserved 384 MB, usable 11904 MB. Live committed ≈ 4.4 GB → ~7.5 GB headroom, so tripping needs mass-deploying ~7.5 GB of requests (impractical/risky unattended); the branch + math are verified. |
## Phase 2 — State corruption [HIGH SEV] — **mostly PASS; 1 medium finding (S1)**

| # | Break (backed up first) | Result | Evidence |
|---|---|---|---|
| S1 | truncate settings.json → restart | **⚠ FINDING (medium)** | controller **crash-loops**: `[FATAL] Failed to load settings … unexpected end of JSON input`, RestartCount=7, restarting. **No safe-defaults fallback** — a corrupt settings.json takes the management plane down. *Not silent* (FATAL logged — better than the worst case). Restore → running/healthy, 3 storage_paths back. (Docker restart-manager cycling it re-confirms Finding #1.) |
| S2 | garbage in uptime-kuma `app.yaml` → rescan | **PASS** | controller stays **healthy** (25/0), logs `[WARN] LoadAppConfig: yaml: … did not find expected key` (not silent, not fatal), other apps unaffected. Restore → `running/deployed`. **Contrast with S1:** per-stack app.yaml corruption is graceful; settings.json is fatal. |
| S3 | corrupt quiesce marker (bad JSON) → restart | **PASS (minor finding)** | no panic/crash-loop, controller healthy, **stacks not stranded** (restarts=0). But the unparseable marker is **silently ignored** (no log) **and left in place** (not cleared/quarantined) — a real corrupted-mid-quiesce marker would skip recovery with no signal. |
| S4 | symlink `→ /etc` in userdata/appdata → backup | **PASS (security holds)** 🔒 | Tier-2 rsync **preserved the symlink** (`EVIL_ETC -> /etc`), did **not** follow it; **/etc contents did NOT leak** into the backup (no passwd/shadow). No path-escape/exfil. |
## Phase 3 — Concurrency storms — **PASS**

| # | Break | Result | Evidence |
|---|---|---|---|
| C1 | deploy + backup + restore + git-sync simultaneously | **PASS** | backup/run + tier2 both single-flighted ("Mentés már folyamatban"); restore mutex'd (302); sync ran independently ("nincs változás"); after settle **no stuck flag** (`running:false`), no deadlock, 25/0 healthy |
| C2 | rapid felhom-usb flap ×5 in ~10s | **PASS** | converged → felhom-usb **MOUNTED**, disconnected mark `None` (not stuck); **0 `permission denied`** during the flap; the v0.75 mountpoint-gate fired **5 clean skips** ("not mounted") in the disconnected windows; 25/0 |
| C3 | kill `felhom-agent` mid-quiesce-backup | **PASS** | quiesce stopped 14 stacks → agent killed → controller `[quiesce] unquiescing (backup start failed): restarting 14 stack(s)` (clean error + defer-unwind, **no stack stranded**); agent restarted → `/api/disks` OK; **idle sockets→8443 = 1** (v0.74 bound holds); recovered 25/0 |
## Phase 4 — Time chaos — **SKIPPED (not safely isolable)**

T1 (+25h) / T2 (−2h): the guest is an **unprivileged LXC** — it **cannot set its own clock**
(`date -s` → `Operation not permitted`, no CAP_SYS_TIME) and shares the host's `CLOCK_REALTIME` (time
namespaces don't isolate the wall clock). The only way to jump the guest's wall-clock is to change the
**host** (felhom-pve) clock, which would risk: cloudflared tunnel cert time-validation (demo down),
the agent's leaf-cert pin to PVE/PBS, and **agent→DooPlex hub/PBS TLS auth** (§0.1 forbids degrading
DooPlex). NTP is active on the host. **Not safely isolable to the guest unattended → SKIP.** Proper
venue: a dedicated throwaway VM with its own clock, or an injectable-clock unit test. **Deferred to
supervised / implementation.**
## Phase 5 — Network partitions (guest/host-side timed rules) — **PASS**

| # | Break (timed-revert) | Result | Evidence |
|---|---|---|---|
| N1 | block cloudflared egress (iptables DOCKER-USER, 130s) | **PASS** | tunnel partitioned: cloudflared `timeout: no recent network activity` + `Retrying connection` (no crash); **apps serve locally** (romm → HTTP 200); controller healthy 25/0. Unblock → `Registered tunnel connection`; no leftover rule. |
| N2 | block agent→PBS `192.168.0.180:8007` mid-quiesce-backup (N100-side OUTPUT rule per §0.1, 150s) | **PASS** | quiesce stopped 14 stacks → `[quiesce] backup job failed` (vzdump phase=failed, clean error) → `[quiesce] unquiescing (backup failed): restarting 14 stack(s)` (**fails safe, no stack stranded**). Unblock → next PBS backup `success:true`; 25/0; no leftover rule. (Rule was N100-side; DooPlex untouched.) |
## Phase 6 — Input/security fuzzing [HIGH SEV] — **PASS (no escape) + 1 medium defense-in-depth finding**

| # | Break | Result | Evidence |
|---|---|---|---|
| F1 | malformed/oversized/unknown-field JSON | **PASS** | not-json → **400**, empty → 400, 1MB → 400, nested-object-where → 400; unknown fields safely ignored (502 from a downstream non-existent mount, **no partial mutation**); controller **healthy** (no panic) |
| F2 | path traversal (storage `where`, restore `stack_name`) | **PASS (no escape) 🔒 + FINDING** | **storage:** all 5 traversals (`/etc`, `../../etc`, `/mnt/../etc`, `/mnt/felhom-drives/../../../etc`, `/etc/shadow`) **rejected** by `gateWhere` (`path.Clean`+`HasPrefix("/mnt/")`). **restore:** the boundary HELD — `/etc/passwd` **intact**, no `/opt/etc`/`/etc/passwd/` artifacts, no `/etc` writes — but **the restore handler does NOT reject a traversal `stack_name` upfront**: `RestoreFromRecoveryUnit("../../../etc")` proceeded (`GetAppDrivePath` → default `/mnt/sys_drive`), saved only by downstream **map-based** `StopStack`/`StartStack` (`stack "../../../etc" not found`) + no recovery-unit/volumes (no-op). **Defense-in-depth gap — recommend explicit `stack_name` validation at the restore handler.** (My own batched test looped these restores ×90s each, briefly holding the mutex — a test artifact that cleared when stopped, NOT a stuck-flag bug.) |
| F3 | hostile compose on a scratch app | **N/A (good property)** | the standard deploy renders only from the git-synced **catalog** — there is **no customer-facing arbitrary-compose injection vector**. (`.fab` import is the only path; not fuzzed — lower priority.) |
| F4 | weird userdata filenames (unicode, spaces, `(N)`, 200-char) | **PASS** | files created + Tier-2 backup ran clean ("Tier 2 run complete: 8 apps"), no crash/panic, 25/0. (`(N)` merge/dedup is migration-specific — `migrate.go` unit-tested, not live-run here.) |
## Phase 7 — Brutal recovery — **PASS (B1-B3); B4 skipped**

| # | Break | Result | Evidence |
|---|---|---|---|
| B1 | restart dockerd in 9201 | **PASS** | boot-restore brought all **25 containers + controller** back in <8s (drives stay mounted on a daemon-only restart → no boot-ordering issue); controller+cloudflared healthy |
| B2 | `kill -9` controller PID (real crash) | **PASS** | `unless-stopped` auto-recovered: RestartCount 0→1, **healthy in 8s** — re-confirms Finding #1 (docker kill ≠ crash) live |
| B3 | reboot 9201 with **felhom-flash detached** | **PASS** 🔑 | booted; flash **DETACHED** (disconnect intent persisted across reboot); 8 HDD apps **held** (absent, not crash-looping); **NO rootfs shadow dirs** (`…/felhom-flash/userdata` doesn't exist). Docker boot-restore made 2 bind-source mkdir attempts → **denied by the unprivileged-LXC mapping** → no escape (the v0.75 gate covers the controller's post-boot belt/FileBrowser; the unprivileged mapping covers docker's boot-restore). Reconnect → flash remounted, 8 apps recovered → 25/0. |
| B4 | host reboot of felhom-pve | **SKIPPED** | unattended risk — no physical recovery if the N100 doesn't POST/return. Deferred to supervised (consistent with #1/#2). |

---
## Findings (ranked by real-world severity)

1. **[MEDIUM] Corrupt `settings.json` → controller FATAL crash-loop (S1).** No safe-defaults fallback —
   a truncated/invalid settings.json takes the management plane down (RestartCount climbs via docker's
   restart-manager). *Not silent* (FATAL is logged — strictly better than the worst case). Apps/tunnel
   keep running (control/data separation). **Fix:** on parse failure, log a WARN + load safe defaults
   (or boot read-only) instead of `[FATAL]`. (Contrast: per-stack `app.yaml` corruption (S2) is handled
   gracefully with a WARN — settings.json should match that.)
2. **[MEDIUM, defense-in-depth] Restore `stack_name` not validated against path traversal (F2).** The
   `/backup/restore` handler proceeds with a traversal name (`../../../etc`) into
   `RestoreFromRecoveryUnit`/`RestoreApp`; **no escape occurred** (`/etc` intact, no artifacts) only
   because downstream `StopStack`/`StartStack` are map-based ("stack not found") + there's no
   recovery-unit/volumes for a bogus name. A future code path that built a filesystem path from the raw
   name could escape. **Fix:** reject `stack_name` containing `/`, `..`, NUL, etc. at the handler.
3. **[LOW] Corrupt quiesce marker silently ignored (S3).** `Recover()` doesn't panic (good) but an
   unparseable marker is **neither logged nor cleared/quarantined** — a real corrupted-mid-quiesce
   marker would skip stack-recovery with no signal. **Fix:** log + quarantine a bad marker.
4. **[INFO] Boot-ordering (re-confirmed, B3).** Docker's boot-restore attempts drive-backed bind-source
   `mkdir` before mounts converge; the **unprivileged-LXC mapping denies it** (no shadow dirs), and the
   v0.75 gate covers the controller side — benign but noisy. (Tracked from the v0.75 task.)

**Everything else fail-safe held:** R1-R4, S2, S4, C1-C3, N1-N2, F1, F3, F4, B1-B3 all PASS. No
silent-corruption and **no path-escape/exfil** found (the two highest-severity classes the runbook
prioritized). **No code changes shipped** (no broken apps surfaced; the findings are behaviours logged
for supervised fix). Phase 4 (time chaos) and B4 (host reboot) SKIPPED for architectural/unattended-risk
reasons (documented).

## Cleanup confirmation
- **9201 running, 25 containers, 0 unhealthy**; controller + cloudflared healthy; agent active.
- Drives mounted (flash sdc1, usb sdb1); rootfs back to **4% / 29 G free** (R3 fill reverted).
- **No leftover rules:** host `OUTPUT` 8007-DROP = 0 (N2), guest `DOCKER-USER` cloudflared-DROP = 0 (N1).
- **No leftover guards** (the 2 residual `sleep` procs are host system monitors, parents 747/700843 — not mine).
- Corrupted files restored (settings.json valid; app.yaml restored; quiesce marker removed; symlinks removed); F4 test files removed.
- **No 9300 / scratch guest created** this campaign; no loopbacks (Phase 4 N/A, destructive disk used live drives only reversibly).
- Pre-existing (NOT this run): 9001 (spike-lxc) + 9999 (felhom-selftest-scratch) stopped.
- Net: demo restored to baseline (= Phase-0 state).