TEST-REPORT — Test campaign #3: NO MERCY (N100 / guest 9201)
Run start: 2026-06-22 (CC, unattended). Brutal chaos/edge — push to unintended states, verify
handles-or-fails-safe, then recovers. Every break carries a timed auto-revert; Phase-0 backup is the floor.
Legend: PASS / FAIL / SKIP + raw evidence. Each chaos line: break→detect→recover→verify.
(Campaigns #1/#2 + diagnoses in git history + felhom.eu/documentation/tests/.)
Phase 0 — Baseline + floor — PASS (gate OPEN)
- ctrl v0.75.0 healthy; agent v0.39.0; 25 containers, 0 unhealthy; rootfs
/ 4%, /var/lib/docker 8%, drives 1%; mem 8.8 Gi available.
- Floor: PBS
felhom-pbs:backup/ct/9201/2026-06-22T19:01:56Z (success, crash-consistent, verified).
Phase 1 — Resource starvation — PASS (fail-safe held)
| # |
Break (timed-revert) |
Result |
Evidence |
| R1 |
python hog ~8 GB (~91% RAM), 75s |
PASS |
mid-stress: 10Gi used / 1.1Gi avail — controller healthy, 25/0 unhealthy; post: dockerd NRestarts=0, no OOM (alloc fit via cache eviction), recovered to 8.9Gi. Backstop pkill guard armed. |
| R2 |
settings save on full disk |
PASS (code-verified) |
settings.save() = atomic write-.tmp-then-os.Rename (settings.go:236-261) → a disk-full fails at WriteFile, original untouched. Live disk-full not reproducible: settings.json is on the 252 GB docker vol. |
| R3 |
fill rootfs (holds /mnt/sys_drive backups) to 94% (timed rm @150s) |
PASS |
controller stayed healthy (it's on the separate 252 GB vol); DB-dump + Tier-2 (8 apps, immich 155MB) succeeded; settings.json stayed valid JSON (no corruption). Reverted → rootfs 4%. (Literal-full refusal not pushed — near-0 rootfs risks wedging the guest OS unattended; the 2 GB margin had room so the gate didn't need to refuse.) |
| R4 |
memory gate hard-block |
PASS (code-verified) |
deploy.go:186 is a hard block (returns an error on committed+new > usable), using committed-memory accounting (sum of deployed mem_request), reserved 384 MB, usable 11904 MB. Live committed ≈ 4.4 GB → ~7.5 GB headroom, so tripping needs mass-deploying ~7.5 GB of requests (impractical/risky unattended); the branch + math are verified. |
Phase 2 — State corruption [HIGH SEV] — mostly PASS; 1 medium finding (S1)
| # |
Break (backed up first) |
Result |
Evidence |
| S1 |
truncate settings.json → restart |
⚠ FINDING (medium) |
controller crash-loops: [FATAL] Failed to load settings … unexpected end of JSON input, RestartCount=7, restarting. No safe-defaults fallback — a corrupt settings.json takes the management plane down. Not silent (FATAL logged — better than the worst case). Restore → running/healthy, 3 storage_paths back. (Docker restart-manager cycling it re-confirms Finding #1.) |
| S2 |
garbage in uptime-kuma app.yaml → rescan |
PASS |
controller stays healthy (25/0), logs [WARN] LoadAppConfig: yaml: … did not find expected key (not silent, not fatal), other apps unaffected. Restore → running/deployed. Contrast with S1: per-stack app.yaml corruption is graceful; settings.json is fatal. |
| S3 |
corrupt quiesce marker (bad JSON) → restart |
PASS (minor finding) |
no panic/crash-loop, controller healthy, stacks not stranded (restarts=0). But the unparseable marker is silently ignored (no log) and left in place (not cleared/quarantined) — a real corrupted-mid-quiesce marker would skip recovery with no signal. |
| S4 |
symlink → /etc in userdata/appdata → backup |
PASS (security holds) 🔒 |
Tier-2 rsync preserved the symlink (EVIL_ETC -> /etc), did not follow it; /etc contents did NOT leak into the backup (no passwd/shadow). No path-escape/exfil. |
Phase 3 — Concurrency storms — PASS
| # |
Break |
Result |
Evidence |
| C1 |
deploy + backup + restore + git-sync simultaneously |
PASS |
backup/run + tier2 both single-flighted ("Mentés már folyamatban"); restore mutex'd (302); sync ran independently ("nincs változás"); after settle no stuck flag (running:false), no deadlock, 25/0 healthy |
| C2 |
rapid felhom-usb flap ×5 in ~10s |
PASS |
converged → felhom-usb MOUNTED, disconnected mark None (not stuck); 0 permission denied during the flap; the v0.75 mountpoint-gate fired 5 clean skips ("not mounted") in the disconnected windows; 25/0 |
| C3 |
kill felhom-agent mid-quiesce-backup |
PASS |
quiesce stopped 14 stacks → agent killed → controller [quiesce] unquiescing (backup start failed): restarting 14 stack(s) (clean error + defer-unwind, no stack stranded); agent restarted → /api/disks OK; idle sockets→8443 = 1 (v0.74 bound holds); recovered 25/0 |
Phase 4 — Time chaos — SKIPPED (not safely isolable)
T1 (+25h) / T2 (−2h): the guest is an unprivileged LXC — it cannot set its own clock
(date -s → Operation not permitted, no CAP_SYS_TIME) and shares the host's CLOCK_REALTIME (time
namespaces don't isolate the wall clock). The only way to jump the guest's wall-clock is to change the
host (felhom-pve) clock, which would risk: cloudflared tunnel cert time-validation (demo down),
the agent's leaf-cert pin to PVE/PBS, and agent→DooPlex hub/PBS TLS auth (§0.1 forbids degrading
DooPlex). NTP is active on the host. Not safely isolable to the guest unattended → SKIP. Proper
venue: a dedicated throwaway VM with its own clock, or an injectable-clock unit test. Deferred to
supervised / implementation.
Phase 5 — Network partitions (guest/host-side timed rules) — PASS
| # |
Break (timed-revert) |
Result |
Evidence |
| N1 |
block cloudflared egress (iptables DOCKER-USER, 130s) |
PASS |
tunnel partitioned: cloudflared timeout: no recent network activity + Retrying connection (no crash); apps serve locally (romm → HTTP 200); controller healthy 25/0. Unblock → Registered tunnel connection; no leftover rule. |
| N2 |
block agent→PBS 192.168.0.180:8007 mid-quiesce-backup (N100-side OUTPUT rule per §0.1, 150s) |
PASS |
quiesce stopped 14 stacks → [quiesce] backup job failed (vzdump phase=failed, clean error) → [quiesce] unquiescing (backup failed): restarting 14 stack(s) (fails safe, no stack stranded). Unblock → next PBS backup success:true; 25/0; no leftover rule. (Rule was N100-side; DooPlex untouched.) |
Phase 6 — Input/security fuzzing [HIGH SEV] — PASS (no escape) + 1 medium defense-in-depth finding
| # |
Break |
Result |
Evidence |
| F1 |
malformed/oversized/unknown-field JSON |
PASS |
not-json → 400, empty → 400, 1MB → 400, nested-object-where → 400; unknown fields safely ignored (502 from a downstream non-existent mount, no partial mutation); controller healthy (no panic) |
| F2 |
path traversal (storage where, restore stack_name) |
PASS (no escape) 🔒 + FINDING |
storage: all 5 traversals (/etc, ../../etc, /mnt/../etc, /mnt/felhom-drives/../../../etc, /etc/shadow) rejected by gateWhere (path.Clean+HasPrefix("/mnt/")). restore: the boundary HELD — /etc/passwd intact, no /opt/etc//etc/passwd/ artifacts, no /etc writes — but the restore handler does NOT reject a traversal stack_name upfront: RestoreFromRecoveryUnit("../../../etc") proceeded (GetAppDrivePath → default /mnt/sys_drive), saved only by downstream map-based StopStack/StartStack (stack "../../../etc" not found) + no recovery-unit/volumes (no-op). Defense-in-depth gap — recommend explicit stack_name validation at the restore handler. (My own batched test looped these restores ×90s each, briefly holding the mutex — a test artifact that cleared when stopped, NOT a stuck-flag bug.) |
| F3 |
hostile compose on a scratch app |
N/A (good property) |
the standard deploy renders only from the git-synced catalog — there is no customer-facing arbitrary-compose injection vector. (.fab import is the only path; not fuzzed — lower priority.) |
| F4 |
weird userdata filenames (unicode, spaces, (N), 200-char) |
PASS |
files created + Tier-2 backup ran clean ("Tier 2 run complete: 8 apps"), no crash/panic, 25/0. ((N) merge/dedup is migration-specific — migrate.go unit-tested, not live-run here.) |
Phase 7 — Brutal recovery — PASS (B1-B3); B4 skipped
| # |
Break |
Result |
Evidence |
| B1 |
restart dockerd in 9201 |
PASS |
boot-restore brought all 25 containers + controller back in <8s (drives stay mounted on a daemon-only restart → no boot-ordering issue); controller+cloudflared healthy |
| B2 |
kill -9 controller PID (real crash) |
PASS |
unless-stopped auto-recovered: RestartCount 0→1, healthy in 8s — re-confirms Finding #1 (docker kill ≠ crash) live |
| B3 |
reboot 9201 with felhom-flash detached |
PASS 🔑 |
booted; flash DETACHED (disconnect intent persisted across reboot); 8 HDD apps held (absent, not crash-looping); NO rootfs shadow dirs (…/felhom-flash/userdata doesn't exist). Docker boot-restore made 2 bind-source mkdir attempts → denied by the unprivileged-LXC mapping → no escape (the v0.75 gate covers the controller's post-boot belt/FileBrowser; the unprivileged mapping covers docker's boot-restore). Reconnect → flash remounted, 8 apps recovered → 25/0. |
| B4 |
host reboot of felhom-pve |
SKIPPED |
unattended risk — no physical recovery if the N100 doesn't POST/return. Deferred to supervised (consistent with #1/#2). |
Findings (ranked by real-world severity)
- [MEDIUM] Corrupt
settings.json → controller FATAL crash-loop (S1). No safe-defaults fallback —
a truncated/invalid settings.json takes the management plane down (RestartCount climbs via docker's
restart-manager). Not silent (FATAL is logged — strictly better than the worst case). Apps/tunnel
keep running (control/data separation). Fix: on parse failure, log a WARN + load safe defaults
(or boot read-only) instead of [FATAL]. (Contrast: per-stack app.yaml corruption (S2) is handled
gracefully with a WARN — settings.json should match that.)
- [MEDIUM, defense-in-depth] Restore
stack_name not validated against path traversal (F2). The
/backup/restore handler proceeds with a traversal name (../../../etc) into
RestoreFromRecoveryUnit/RestoreApp; no escape occurred (/etc intact, no artifacts) only
because downstream StopStack/StartStack are map-based ("stack not found") + there's no
recovery-unit/volumes for a bogus name. A future code path that built a filesystem path from the raw
name could escape. Fix: reject stack_name containing /, .., NUL, etc. at the handler.
- [LOW] Corrupt quiesce marker silently ignored (S3).
Recover() doesn't panic (good) but an
unparseable marker is neither logged nor cleared/quarantined — a real corrupted-mid-quiesce
marker would skip stack-recovery with no signal. Fix: log + quarantine a bad marker.
- [INFO] Boot-ordering (re-confirmed, B3). Docker's boot-restore attempts drive-backed bind-source
mkdir before mounts converge; the unprivileged-LXC mapping denies it (no shadow dirs), and the
v0.75 gate covers the controller side — benign but noisy. (Tracked from the v0.75 task.)
Everything else fail-safe held: R1-R4, S2, S4, C1-C3, N1-N2, F1, F3, F4, B1-B3 all PASS. No
silent-corruption and no path-escape/exfil found (the two highest-severity classes the runbook
prioritized). No code changes shipped (no broken apps surfaced; the findings are behaviours logged
for supervised fix). Phase 4 (time chaos) and B4 (host reboot) SKIPPED for architectural/unattended-risk
reasons (documented).
Cleanup confirmation
- 9201 running, 25 containers, 0 unhealthy; controller + cloudflared healthy; agent active.
- Drives mounted (flash sdc1, usb sdb1); rootfs back to 4% / 29 G free (R3 fill reverted).
- No leftover rules: host
OUTPUT 8007-DROP = 0 (N2), guest DOCKER-USER cloudflared-DROP = 0 (N1).
- No leftover guards (the 2 residual
sleep procs are host system monitors, parents 747/700843 — not mine).
- Corrupted files restored (settings.json valid; app.yaml restored; quiesce marker removed; symlinks removed); F4 test files removed.
- No 9300 / scratch guest created this campaign; no loopbacks (Phase 4 N/A, destructive disk used live drives only reversibly).
- Pre-existing (NOT this run): 9001 (spike-lxc) + 9999 (felhom-selftest-scratch) stopped.
- Net: demo restored to baseline (= Phase-0 state).
